The time has finally come! After the massive success of our Docker and Kubernetes guides, we are now ready to see several anti-patterns for Argo CD. Anti-patterns are questionable practices that people adopt because they seem like a good idea at first glance, but in the long run, they make processes more complicated than necessary.

Several times, we have spoken with enthusiastic teams that recognize the benefits of Gitops and want to adopt Argo CD as quickly as possible. The initial adoption phase seems to go very smoothly and more and more teams are getting onboarded to Argo CD. However, after a certain point things start slowing down and developers start complaining about the new process.

Like several open source projects, Argo CD has several capabilities that can be abused if you don’t have the full picture in your mind. The end result almost always makes life really difficult for developers.

So keep your developers happy and don’t fall into the same traps. Here is the full list of the antipatterns we will see:

The order of anti-patterns follows the timeline of an organization that starts with minimal Argo CD knowledge and slowly migrates several applications to the GitOps paradigm.

Anti-pattern 1 — Not understanding the declarative setup of Argo CD

Following the GitOps principles means that Argo CD can take your Kubernetes manifests (or Helm charts or Kustomize overlays) from Git and sync them on your Kubernetes cluster. This process is well understood by teams and most people are familiar with saving Kubernetes manifests in Git.

It is important to note however that even this link between a cluster and a Git repository is a Kubernetes resource itself.

Argo CD introduces its own Custom Resource Definitions (CRDs) for several of its central concepts such as applications and projects and also reuses existing Kubernetes CRDs for clusters, secrets, etcs.

These files should also be stored in Git. That is the whole point of following GitOps. It doesn’t make sense to use Git only for some files and not store the Applications themselves similarly.

We see several teams that use the Argo CD UI or CLI to create applications that are not stored anywhere and then have difficulties understanding what is deployed where or how to recreate their Argo CD configuration from scratch.

It should be noted that the “create new app” button in the Argo CD UI is only for experiments and quick tests. You should not use it at all in a production environment as the created application is not saved anywhere.

Everything that Argo CD needs should be stored in Git. Recreating an Argo CD instance should be a simple process with minimal steps:

1. Create a new cluster with Terraform/Pulumi/Crossplane etc.
2. Install Argo CD itself using Terraform/Autopilot/Codefresh etc.
3. Point Argo CD to your ApplicationSets or root app-of-apps file
4. Finished.

Recreating your Argo CD instance is a repeatable process that can be performed in less than 5 minutes (explained in anti-pattern 27).

If you want a comprehensive guide on how to organize your Kubernetes manifests in Git see our Application Set guide.

Anti-pattern 2 — Creating Argo CD applications in a dynamic way

A related anti-pattern occurs when organizations have an existing process for creating applications and when they adopt Argo CD they simply call the Argo CD CLI or API to pass the information they already have.

Essentially there is an existing database or application configuration somewhere else and a custom CLI or other imperative tool does the following:

1. Extracts application configuration from the existing database
2. Creates an Argo CD application or Kubernetes manifest on the fly
3. Applies this file to an Argo CD instance without storing anything in Git.

You fall into this trap if the “official” way of creating Argo CD applications in your organization is something like this:

my-app-cli new-app-name | argocd app create -f -

Or several times, envsubst is used like this

envsubst < my-app-template.yaml | kubectl apply -f -n argocd

The end result is always the same. You have custom Argo CD applications that are not saved anywhere in Git. You lose all the main benefits of GitOps:

• You don’t have a declarative file for what is deployed right now
• You don’t have a history of what was deployed in the past
• Recreating the Argo CD instance is not a single-step process any more (see anti-pattern 27)

To overcome this anti-pattern you need to make sure that you use Argo CD the way GitOps works.

You either need to convert your existing database settings and make them read/write to Git or discard them completely and start using Git as the single source of truth for everything.

Then, all day 2 operations should be handled by Git.

The same is true for updating existing applications. If you want to update the configuration of an application, the process should always be the same:

1. You (or an external system) change a file in Git
2. Argo CD notices the change in Git
3. Argo CD syncs the changes to the cluster.

If you use the Argo CD API or the Kubernetes API to manually patch resources then you are not following GItOps.

Updating applications in production with any of the following commands goes against the GitOps principles.

kubectl set image deployment/my-deployment my-container=nginx:1.27.0
kubectl patch deployment <deployment-name> [.....patch here…]

Use the Kubernetes API only for experiments and local tests, never for production upgrades (see also anti-pattern 10 — disabling auto-sync).

Anti-pattern 3 — Using Argo CD parameter overrides

Yet another way of updating an Argo CD application in a manner we do NOT recommend is the following:

argocd CLI app set guestbook -p image=example/guestbook:v2.3

The guest book application was just updated to version v2.3. Argo CD syncs the version and everything looks good. But where was this action saved? Nowhere.
This command is using the parameters feature of Argo CD that allows you to override any Argo CD application with your own custom properties. Even the official documentation has a huge warning about not using this feature as it goes against the GitOps principles.

Even if you save the parameter information in the Application manifest, you have now completely destroyed local testing for developers (see anti-patterns 6 and 17).

Anti-pattern 4 — Adopting Argo CD without understanding Helm

Helm is the package manager for Kubernetes. In its original form it offers several essential features in a single platform:

• A package format for Kubernetes manifests
• A repository specification for storing Helm packages in artifact managers
• A templating system
• A comprehensive CLI
• A lifecycle definition (upgrade, install, rollback, test )

Argo CD renders all Helm charts using the template command and completely discards most other lifecycle features. Even though in theory this is a good thing as Argo CD can replace the default Helm lifecycle (e.g. Argo CD comes with its own rollback command), Argo CD assumes that you already know how Helm templates work.

If you are adopting Argo CD and want to use Helm, then make sure that you know how Helm works on its own and how all your applications can be deployed in all your different environments WITHOUT Argo CD. Trying to learn Argo CD and Helm together at the same time is a recipe for failure.

At the very least you should know how to create Helm hierarchies of values

common-values.yaml
+-----all-prod-envs.yaml
   +----specific-prod-cluster.yaml

And how Helm works when loading the hierarchy with the correct overrides

helm install ./my-chart/ — generate-name -f common.yaml -f more.yaml -f some-more.yaml

If you use Helm umbrella charts, understand how to override child values from the top chart.

In particular, pay attention to the following:

restaurant:
 menu: vegetarian

This can be a simple value setting for a Helm chart that sets the “restaurant.menu” value to “vegetarian”. It can also be an umbrella chart which has a dependency subchart called “restaurant” which itself has a property called “menu”. Understand why those two approaches are different and the advantages and disadvantages of each.
We have already written a complete guide on how to use Argo CD with Helm value hierarchies.

Anti-pattern 5 — Adopting Argo CD without understanding Kustomize

This is the same anti-pattern as the previous one but for Kustomize users. Kustomize is a powerful tool and includes several features such as

• Overlays for different environments
• Reusable component configurations
• Common globals (labels, annotations, namespace prefixes)
• Configuration generators
• Transformers/Replacements
• Remote resources

Argo CD can reuse all Kustomize features, provided that you have structured your Kustomize files correctly first.

Again make sure that your Kustomize files work on their own BEFORE bringing Argo CD into the picture. Well structured Kustomize applications are self-contained. Any developer should be able to use the kustomize (or kubectl) command to extract the configuration for an existing environment without the need for Argo CD.
We presented a full example for Kustomize configurations in our Argo CD promotion guide.

Anti-pattern 6 — Assuming that developers need to know about Argo CD

This is the corollary to the previous two antipatterns. Several teams mix Argo CD configuration data with Kubernetes configuration making the life of developers extremely difficult.

This is a big problem for developers as one of their most important tasks is to run an application locally both during development and when trying to pinpoint difficult bugs in isolation. Creating Argo CD configurations and coupling them with Kubernetes manifests prevents them from understanding how an application runs independently.

We will see more specific anti-patterns later that also contribute to this problem but it is best to know about this trap in advance when you start working with Argo CD.

Remember also that developers don’t care about Kubernetes manifests. They only care about source code features. It is one thing to ask them to learn the basics (i.e. Helm values) and a completely different thing to require Argo CD knowledge just to be able to recreate the configuration of an existing environment.

When you design your Argo CD repository, you should always consider a developer persona who is an expert on Kubernetes but knows nothing about Argo CD. Can they recreate any configuration of any application on their laptop without using Argo CD? If the answer is no, it means that you are the victim of this anti-pattern.

Find out where you have hardcoded Argo CD configurations with Kubernetes configurations and remove the tight coupling. For specific advice see anti-patterns 17 for Helm and 18 for Kustomize.

We have seen how to split Kubernetes configuration from Argo CD manifests in our Application Set guide.

Anti-pattern 7 — Grouping applications at the wrong abstraction level

As we explained in the first anti-pattern (store everything in Git), an Argo CD application is just a link between a Git repository and a Kubernetes cluster. It is not a deployment artifact or a packaging format.

We have seen teams that abuse an Argo CD application as a generic grouping mechanism, using it for microservices or even completely unrelated applications.

You need to spend some time understanding what your applications do and how tightly coupled they are. If you have a set of “micro-services” that are always deployed together and upgraded together, you might want to use an umbrella chart for them.

Argo CD applications should generally model something that requires individual deployments and updates. If you have several Argo CD applications that you always want to be managed together (but still want to deploy and update individually), then a better choice might be the app-of-apps pattern.

If several applications need to be deployed to different or similar configurations, then Application Sets are the proper recommendation.

So any time you want to group several applications, ask the following questions:

1. Are those applications always deployed and upgraded as a single unit?
2. Are those applications related in a business or technical manner?
3. Do you want to use different configurations for different clusters for these applications?
4. Is this combination of applications always the same? Do you sometimes wish to deploy a subset of them or a superset?
5. Are these applications managed by a single team or multiple teams?

If you are unsure where to start, looking at Application Sets is always the best choice. Check also anti-pattern 19 (attempting to version application CRDs).

Anti-pattern 8 — Abusing the multi-source feature of Argo CD

This is a close relative of the previous anti-pattern. The Multi-source feature of Argo CD was one of the most requested features in the project’s history. The feature was created to solve a single scenario:

1. You wish to use an external Helm chart that is not hosted by your organization
2. You want to use your own Helm values and still store them in your own Gitrepository
3. You need a way to instruct Argo CD to combine the external Helm chart with your own values.

The feature is finally implemented in Argo CD and you can finally do the following:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  sources:
  - repoURL: 'https://prometheus-community.github.io/helm-charts'
    chart: prometheus
    targetRevision: 15.7.1
    helm:
      valueFiles:
      - $values/charts/prometheus/values.yaml
  - repoURL: 'https://git.example.com/org/value-files.git'
    targetRevision: dev
    ref: values

Unfortunately, Argo CD does not limit the number of items you can place in the “sources” array. Several people have misunderstood this, abusing the feature to group multiple (often unrelated) applications.

Don’t fall into this trap. The multi-source feature was never designed to work this way and several standard Argo CD capabilities will either be broken or not work at all if you use multi-sources as a generic application grouping mechanism.
The correct way to group applications is with Application Sets. If you want to use multisource applications with Helm hierarchies, we have also written an extensive guide.

Anti-pattern 9 — Not splitting the different Git repositories

If you look at any Kubernetes application from a high level it is comprised of 3 distinct types of files

1. The source code
2. Kubernetes manifests (deployment, service, ingress etc)
3. Argo CD application manifests (or application sets)

We have already explained that as a best practice the source code should be separate than the manifests. If you are in a big organization it might also make sense to split Kubernetes manifests from Argo CD manifests.

If you keep all manifests in a single Git repository you will have issues with both CI (Continuous Integration) and CD (Continuous Deployment) phases. Your CI system will try to auto-build application code when a manifest changes, and Argo CD will try to sync applications when a developer changes the source code.

There are several workarounds for these scenarios, but why try to solve a problem that shouldn’t exist in the first place?

We have also seen several variations of the same pattern that make the deployment process even more complicated. A classic scenario to avoid is the following

1. The source code of the application is in Git repository A
2. The Helm manifest is in Git repository B
3. Only the Helm values for the different environments are also stored in Git repository A

The assumption here is that Helm values are close to the source code that developers need to change. In reality, it never makes sense to have access to Helm values without also having access to the Helm chart that uses them. Either assume that developers know about Helm and show them everything, or assume that they don’t care about Kubernetes at all and offer them a different abstraction that hides Argo CD completely for them.

Yet another problem is mixing Kubernetes manifests and Argo CD manifests in the same repository but instead of using different folders, you hardcode Kubernetes information into Argo CD information. This is described in detail in anti-patterns 17 and 18.

Anti-pattern 10 — Disabling auto-sync and self-heal

Several times, migrating to Argo CD is a big undertaking, especially for organizations that have invested a significant amount of effort in traditional pipelines. After all, it can be argued that Argo CD simply replicates what is already possible with an existing Continuous Integration system:

1. A change happens in Git in a manifest
2. A separate process picks up the Git event
3. The process uses kubectl (or a custom script) to apply the changes in the cluster.

This could not be further from the truth as Argo CD also works the other way around. It monitors changes in the cluster and compares them against what is in Git. Then you can make a choice and either review those changes or discard them altogether.

This means that Argo CD solves the configuration drift problem once and for all, something that is not possible with traditional CI solutions.

But this advantage only exists if you let Argo CD do its job. Some organizations disable the auto-sync/self-heal behavior in Argo CD in an effort to “lock-down” or fully control production systems.

This is a bad choice because production systems are exactly the kind of systems where you want to avoid configuration drift. Manual changes that happen in production (during hotfixes or other debugging sessions) are one of the biggest factors affecting failed deployments.

We recommend that you have auto-sync/self-heal for all your systems both production and non-production.

Locking down a system should not be done in Argo CD itself, but enforced on the cluster and Git level. The most obvious solution is to reject any direct commits in a Git repository that controls your production system and only allow developers to create Pull Requests which must pass several manual and automated checks before landing in the mainline branch that Argo CD monitors.

If you disable auto-sync/self-heal you are missing the number one advantage of moving to Argo CD from traditional pipelines (eliminating configuration drift).

Anti-pattern 11 — Abusing the targetRevision field

Promoting applications when adopting Argo CD is one of the biggest challenges for developer workloads. People see the targetRevision field in the Argo CD application and assume it is a promotion mechanism.
The first issue is when teams use semantic version ranges to force Argo CD to update an application automatically to a newer version. The second issue is if they continuously update the targetRevision field to different branch names or attempt to implement “preview environments” by pointing an Argo CD application to a temporary developer/feature branch.

We have written a complete guide about the issues of abusing targetRevision.

In general, we recommend you always use HEAD in the targetRevision field which is also the default value.

Anti-pattern 12 — Misunderstanding immutability for container/git tags and Helm charts

This is not an anti-pattern with Argo CD per se, but it is closely related to the targetRevision choices, as explained in the previous anti-pattern.

We have seen several cases of people adopting Argo CD without first understanding the foundations (Helm, container registries, git tags). Several times, people use a specific git tag or Helm version in Argo CD without realizing that:

1. Git tags seem to be immutable, but can be deleted and recreated with the same name
2. Helm chart versions are mutable. This is how Helm was designed
3. Container tags are mutable by default.

Let’s take these points one by one.

Container tags are mutable. You can create a container called my-app:v1.2 and then change something and push another container with the same tag. So just because you see the same container tag doesn’t mean that it is actually the same application. Some binary repository implementations don’t allow you to do this, but this is not always the default setting.

Helm chart versions are also mutable. You can change the contents of a Chart version and use the exact same version. Again, this is how Helm charts are created.

In fact Helm offers an additional property — the appVersion which can store the “application” version. So you can have a Helm chart with 3 “version” fields:

• The container image (mutable by default)
• The appVersion field (mutable)
• The Chart version (mutable)

So unless you control how developers work with code and manifests and also configure your Helm chart repository correctly, you don’t really know if a Helm chart version contains the same thing as another chart with the same version.

Git tags can also be overwritten. You can see this very easily with any GitHub repo with default settings.

git tag -a v1.2 -m "first tag"
git push --tags
echo "A change" >> README.md
git tag -d v1.2
git push origin :v1.2
git tag -a v1.2 -m "first tag"
git push --tags

Here we just pushed the same tag (v1.2) twice but with different contents. So if you were using this tag in the targetRevision field of Argo CD, you now have the “same” application without actually having the same contents.

The end result is that using tags and Helm chart versions in Argo CD doesn’t really restrict your developers unless you actively set up the rest of the ecosystem (Git repositories, Helm repos and binary artifact managers) to also work with immutable data.

Never assume you have a “locked-down” Argo CD system when the rest of the ecosystem allows developers and operators to create stuff with the same container/Helm/Git version.

Anti-pattern 13 — Giving too much power (or no power at all) to developers

When adopting Argo CD you need to make a decision about how much power and exposure you want to give to developers. On one end of the spectrum we see installations where developers have full access to the Argo CD UI and can sync/deploy their applications at will. On the other end we see locked down installations where developers are given very little power or Argo CD is completely hidden from them.

It is important to understand that despite these two extremes, there are several choices in the middle. First, you can configure the Argo CD access (and UI) to show only applications specific to each team. You can even do advanced scenarios where you show applications from other teams but in a read-only mode.

In our Argo CD RBAC guide, we have explained how the RBAC for Argo CD works and how you can show content specific only to a developer team.

There is no right or wrong answer here, but you need to balance the flexibility versus the security that you want to offer to developers.

On a related note we have already explained that developers don’t really care about Argo CD manifests and they shouldn’t also be forced to install Argo CD for local testing (see anti-pattern 6) .

So a recommended workflow would be the following:

1. Developers can test and deploy their applications locally without using Argo CD at all
2. When they have created a feature branch this should be converted to a preview/temporary environment using the Pull Request generator without any human intervention
3. Once the feature is ready, it will be deployed to production simply by merging the Pull request
4. A system designed for promotion should propagate the changes to the next environment (check anti-pattern 30).

Ideally developers should only come in contact with Argo CD in the last phase and only in the case of a failure. In the happy path scenario where everything works as planned, developers shouldn’t have to debug anything with Argo CD.

Anti-pattern 14 — Referencing dynamic information from Argo CD/ Kubernetes manifests

This is a more specialized anti-pattern related to number 2 (creating applications in a dynamic way). The second GitOps principle explains that the desired state of your system should be immutable, versionable and auditable.

This is only true if you store EVERYTHING that the application needs in Git (or your chosen storage method). In the case of Kubernetes/Argo CD manifests, this means that all values used should be static and known in advance.

It is ok if you want to post-process your manifests in some way as long as this happens in a repeatable manner OR you also save the result itself in Git.

The problem starts when your configuration is not known in advance but requires real-time access to something else.

The best example to illustrate this is the Helm lookup method which mutates the Helm chart to a different value without knowing in advance what this value is. This is problematic with Argo CD because having access to just the application manifests is not enough to run the application anymore.

Like anti-patterns 17 and 20 this also makes lives difficult for developers as they cannot run the application locally anymore (anti-pattern 6).

Note that the only exception to this rule is secrets. Even though you can store encrypted secrets in Git, it is also ok to reference them from an external source.
But make sure to understand the difference between referencing secrets from manifests versus injecting secrets into manifests.

Anti-pattern 15 — Writing applications instead of Application Sets

The Application CRD is the main entity in Argo CD that links a cluster and a GitHub repository. If you have a small number of applications or work in a homelab environment it is OK to write these files by hand.

But for any production installation of Argo CD (used in a company) you generally wouldn’t need to write Application files by hand. In fact you should not even deal with Application files at all.

The recommendation is to use directly Application Sets.

In a big organization you will rarely have to deal with a single application on its own. Almost always you want to work with a group of applications. Some examples are:

• A set of applications that go to the same cluster
• A set of applications that are managed by the same team
• A set of applications that share a configuration
• A set of applications that should be deployed/updated as a unit.

Application Sets implement this grouping and also automate the tedious YAML needed. For example if you have 4 applications and 10 clusters you don’t really want to create 40 YAML files by hand.

1. You create a single cluster generator that iterates over your clusters
2. You create 4 folders for the 4 applications
3. The Application set runs and creates automatically all 40 combinations for you.

Some people resist using Application Sets because they think it is yet another abstraction that hides their real Application manifests.

This used to be true, but in the latest Argo CD releases the CLI allows you to render an application set and see exactly what applications will be created. You can run this command either manually or in a CI pipeline (when a pull request is created) so that you have instant visibility on what will change.

In general though, as we will see later all application set files should be created only once.They are not a deployment format (anti-pattern 19) and you shouldn’t have to edit ApplicationSets for simple operations.
We have written a dedicated guide on how to use Application Sets with Argo CD.

Anti-pattern 16 — Using Helm to package Applications instead of Application Sets

This anti-pattern is the so called “Helm sandwich”. This is the case where:

1. A set of Kubernetes manifests is packaged as a Helm chart
2. The Helm chart is referenced from an Argo CD application Manifest
3. The Argo CD application manifest itself is packaged in another Helm chart

Essentially, there are two instances of Helm templating on two different levels.

Teams that adopt this anti-pattern are familiar with Helm and assume that if it works great for plain Kubernetes manifests, it would also work great for Argo CD application manifests.

So why is this approach an anti-pattern? Using Helm for applications is not an issue on its own, but it is the start for several other anti-patterns:

1. Because Helm charts have a version people try to version Applications -> Anti-pattern 19
2. Chart version numbers often result in abusing target Revision for promotions -> Anti-pattern 11
3. If you use Helm everywhere it is super easy to hard-code Helm values in Application manifests -> Anti-pattern 17
4. People miss all the Argo CD specific features of application sets -> Anti-pattern 15
5. Distributing applications to different clusters happens with snowflake servers instead of the cluster generator -> Anti-pattern 21

The biggest problem however is that it ruins again completely the developer experience. You now have 2 levels of Helm values and this is a recipe for disaster:

Even speaking for just operators/admins, packaging applications in a Helm chart creates an extra level of indirection that not only is not needed but makes your life more difficult as Argo CD was never created for this “Helm sandwich”.

Our recommendation is obviously to use Application Sets for configuring Applications, as already explained in the previous anti-pattern.

Application Sets are the preferred way to generate Applications and they support the exact same templating functions as Helm charts. Basically if you can template it with Helm, you should be able to template it with Application Sets.

Using the “Helm sandwich” pattern makes the process more complex for everybody involved in the software lifecycle.
For more information, see our Application Set guide.

Anti-pattern 17 — Hardcoding Helm data inside Argo CD applications

If you follow the advice we outlined in Antipattern 4 you should have a clear separation between your Helm charts and your Argo CD manifests.

The Helm charts can be used independently (even by developers) and contain all application settings. The Argo CD application manifest simply defines where this application runs, and only operators need to change this file.

Those two kinds of files also have a different lifecycle

• Helm values are expected to change all the time (even by developers)
• Argo CD application manifests are created once and never changed again.

With Argo CD it is possible to hardcode Helm information inside an Application CRD. But just because you can do this, doesn’t mean it is a good idea to use this feature.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-helm-override
  namespace: argocd
spec:
  project: default


  source:
    repoURL: https://github.com/example-org/example-repo.git  
    targetRevision: HEAD  
    path: my-chart  


    helm:
      # DONT DO THIS
      parameters:
      - name: "my-example-setting-1"
        value: my-value1
      - name: "my-example-setting-2"
        value: "my-value2"
        forceString: true # ensures that value is treated as a string


      # DONT DO THIS
      values: |
        ingress:
          enabled: true
          path: /
          hosts:
            - mydomain.example.com


      # DONT DO THIS
      valuesObject:
        image:
          repository: docker.io/example/my-app
          tag: 0.1
          pullPolicy: IfNotPresent
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app

The big problem with this file is that you are mixing 2 different kinds of information with 2 different lifecycles. The application CRD is something that is interesting mostly to operators/administrators while Helm information is interesting to developers and is also expected to change a lot.

By mixing this information you make manifests harder to understand for everybody.

Now you have Helm information in two places (helm values and the helm property inside the Argo CD application manifest). It is very hard to understand what settings exist where and how to audit deployment history. Argo CD application manifests now must also change all the time especially if they define container images.

This manifest mixing is also the root of several other anti-patterns such as

• Using Applications as a unit of work (anti-pattern 19)
• Abusing the targetRevision field for promotions (anti-pattern 11)
• Not understanding how Helm hierarchy works (anti-pattern 4)
• Assuming that developers need Argo CD (anti-pattern 6)

The last point is especially important for developers. If you follow this practice, you have completely destroyed local testing for developers as they cannot run the application on its own anymore.

Even though we speak only about developers and operators here, several other scenarios will make this approach difficult to use

1. Security teams will have a hard time understanding the settings for each application
2. Your CI system cannot upgrade images just on Kubernetes manifests anymore. It also needs to look at Argo CD manifests and check if image definitions exist there as well
3. It couples your Applications to specific Argo CD features

The correct solution is of course to store all Helm information in Helm values

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-helm-override
  namespace: argocd
spec:
  project: default


  source:
    repoURL: https://github.com/example-org/example-repo.git  
    targetRevision: HEAD  
    path: my-chart  


    helm:
      ## DO THIS (values in Git on their own)
      valueFiles:
      - values-production.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app

Now there is a clear separation of concern between developers and operators.

People who want to know applications’ settings can look at Helm values, while people who want to know where applications are deployed can look at Argo CD manifests. It is possible to run a Helm application without Argo CD.

For more information about the different types of manifests and how to split them see our Application Set guide.

This anti-pattern is even worse if coupled with the previous anti-pattern (the Helm sandwich).

Now you have 3 places where configuration settings can be stored:

1. The Helm values of the chart that gets deployed
2. The helm property inside the Application Manifest that references the Helm chart
3. The Helm template that renders the Application manifest before getting passed to Argo CD

The result is a nightmare for anybody who wants to understand how an application gets deployed.

Anti-pattern 18 — Hardcoding Kustomize data inside Argo CD applications

This is the same anti-pattern as the previous one but for Kustomize. Again for convenience Argo CD allows you to hardcode Kustomize information inside an Application YAML:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-kustomize-override
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/example-org/example-repo.git  
    targetRevision: HEAD  
    path: my-app  
   
    # DONT DO THIS
    kustomize:
      namePrefix: prod-
      images:
      - docker.io/example/my-app:0.2
      namespace: custom-namespace


  destination:
    server: https://kubernetes.default.svc
    namespace: my-app

It is the same problem as before where you are mixing different concerns in a single file. Kustomize information should only exist in Kustomize overlays:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-proper-kustomize-app
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/example-org/example-repo.git  
    targetRevision: HEAD  
    ## DO THIS. Save all values in the Kustomize Overlay itself
    path: my-app/overlays/prod  


  destination:
    server: https://kubernetes.default.svc
    namespace: my-app

As explained before this helps developers during local testing. A developer can simply run “kustomize build my-app/overlays/prod” and get the full configuration of how my-app runs in production. No knowledge of Argo CD is required and no local installation of Argo CD is needed.

Developers can define how an application runs (its settings), while operators can decide where (which cluster) the application is deployed.

At the same time, several supporting functions are very easy:

• Git history of the overlays is the same as the deployment history
• There is only a single source of truth for configuration (the overlays)
• Developers don’t need to know how to use Argo CD at all
• It is very easy to diff settings between environments.

A detailed example with Kustomize overlays is available in our GitOps promotion guide.

Anti-pattern 19 — Attempting to version and promote Applications/Application Sets

An Argo CD application is just a link between a cluster and a Git repository. There is NO version field in the Application CRD. An application manifest is neither a packaging format nor a deployment artifact. The same is also true for Application Sets. You never deploy application sets. You just use them to auto-generate application manifests. Application Sets have no version on their own.

The lack of a version field is not a big problem because the expectation for both Applications and Application Sets is that you create them once and then never update them again. So, a version field is unnecessary as all Argo CD Applications are considered static.

However, we see several teams that try to use Applications as the unit of work (see anti-pattern 7) or continuously update those files in the targetRevision field (see anti-pattern 11).

At this point teams try to create their own versioning on top of the Argo CD manifests and of course they fail because Argo CD was never designed this way.

The same is true for promotions. You cannot really “promote” an Argo CD application from one cluster to the next. It doesn’t work this way. You can only promote values that are referenced from one application to the next (essentially copying them).

The way promotions work in Argo CD is the following:

1. There is an Argo CD application manifest in QA that points to a Helm chart or Kustomize overlay
2. There is an Argo CD application manifest in Staging that points to different Helm values or Kustomize overlays
3. When it is time to promote you copy the Helm values or Kustomize overlay from the QA files to the Staging files.
4. The Argo CD application manifests are not affected in any way. They are exactly the same as they were before.

See also the related anti-patterns of hardcoding Helm (anti-pattern 17) or Kustomize data (anti-pattern 18) inside applications.

If you find yourself constantly updating Argo CD application manifests (or Application Sets) you have fallen into this trap. Your Argo CD manifests should be created only once and never touched again. No process is simpler than not needing a process at all (to update Application manifests).

On a related note, we have created GitOps Cloud to solve this problem with promotions and allow you to promote applications from one cluster to another.

Anti-pattern 20 — Not understanding what changes are applied to a cluster

One of the main benefits of using Argo CD is that all your Git tools work out of the box and you can reuse your code review process for your Kubernetes manifests.

The most basic capability of storing anything in GitHub is creating a Pull Request before merging any changes. This allows humans to review what will change and also run any automated tools to verify and validate the changes.

Unfortunately this review process will not really work on files such as Helm charts or Application Sets or Kustomize Overlays. Let’s assume you need to review the following change:

You need to manually run Helm in your head to understand what is happening here. Humans never want to do that. Obviously one approach would be to pre-render all your manifests so that reviews only happen in the final content. There are however several alternatives that you can consider such as having your CI system render on the fly the manifests and show you what will actually happen.

Here is the exact change as before but this time on the final chart.

We have written a full guide on how to preview and diff your Kubernetes manifests with several other approaches.
Specifically for Argo CD you should also check https://github.com/dag-andersen/argocd-diff-preview and also understand that you can use the Argo CD CLI to render your Application Sets to Application CRDs.

Anti-pattern 21 — Using ad-hoc clusters instead of cluster labels

If you have a large number of clusters that you wish to manage with Argo CD, your first question is always whether to use a single Argo CD instance or multiple ones.

Once you answer this question the next step is to understand how you can distribute different applications to different clusters. The answer for this question is Application Sets.

Unfortunately we see a lot of teams that don’t understand how the cluster generator works and instead try to create ad-hoc cluster configurations (the pet vs cattle philosophy).

Examples to avoid often appear like this:

## DO NOT DO THIS 
- merge:
      mergeKeys:
        - app
      generators:
        - list:
            elements:
              - app: external-dns
                appPath: infra/helm-charts/external-dns
                namespace: dns
              - app: argocd
                appPath: infra/helm-charts/argocd
                namespace: argocd
              - app: external-secrets
                appPath: infra/helm-charts/external-secrets
                namespace: external-secrets
              - app: kyverno
                appPath: infra/helm-charts/kyverno
                namespace: kyverno
        - list:
            elements:
              - app: external-dns
                enabled: "true"
              - app: argocd
                enabled: "true"
              - app: external-secrets
                enabled: "false"
              - app: kyverno
                enabled: "true"
    selector:
      matchLabels:
        enabled: "true"

Or this

## DO NOT DO THIS 
Version: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-staging-cluster
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - matrix:
        generators:
        - git:
            repoURL: '<url>'
            revision: HEAD
            files:
              - path: customConfig/base.yaml
              - path: customConfig/{{ .Values.domainId }}/*.yaml
        - list:
            elements:
              - appName: 'auth'
              - appName: 'search'
              - appName: ‘billing’
              - appName: ‘payments’

These kind of configurations create snowflake servers that need constant maintenance. If you have fallen into this trap try to understand how much time you will need to spend in the following scenarios:

• Creating a brand new server
• Migrating an application from one server to another
• Applying a global setting to all your servers
• Making a different configuration change for a subset of your servers.

This kind of cluster management is even more problematic for developers, as simply understanding what applications are deployed where is not easy (already explained in anti-pattern 6).

Our recommendation is to create a production-ready setup using cluster labels. We have described all the details in a dedicated guide for the cluster generator and Argo CD application sets.

Anti-pattern 22 — Attempting to use a single application set for everything

A related anti-pattern is when teams discover application sets and, for some unknown reason, try to cram all their applications into a single application set. This often results in a complex mix of different generators that is hard to understand and hard to debug.

The recommendation is to have many different application sets in your Argo CD setup. Ideally, you should have different application sets per “type” of application. This type can be anything that makes sense to you. For example:

• An application set for all staging apps
• An application set for all AWS clusters
• An application set for all infra apps
• An application set for the billing team
• An application set for the payments team

You can slice and dice your applications in different dimensions. But in the end, you will have many application sets, and any time you make a change, you should instantly know where it should happen.

• A new requirement is that all AWS clusters need to get sealed secrets -> Change the AWS application set.
• A new requirement is that the billing team add a new microservice to their setup -> Change the “billing” application set.
• All new clusters must upgrade Prometheus -> Change the “common” application set.

There is no technical limitation on the number of application sets you can have on a single Argo CD installation. So, spend some time understanding your application sets accordingly.

As always, our Argo CD application guide is the best starting point.

Anti-pattern 23 — Using Pre-sync hooks for db migrations

Argo CD phases/waves allow you to define the order of Resources synced within a single Argo CD application. The pre-sync phase, in particular, can be used to check deployment requirements or perform other checks that need to happen before the main sync phase.

We often see organizations attempting to use pre-sync hooks for database migrations. The assumption here is that the database schema must be updated just before the new application version. Unfortunately, most organizations use legacy DB migration tools, and almost always, they package a DB migration CLI tool in the pre-sync phase, which is not the proper approach for Kubernetes applications.

Using pre-sync hooks for database migrations has several issues. The most important one is that the DB CLI tool is just a black box for Argo CD. The CLI runs and never reports back to Argo CD what really happened with the database. This can leave the DB in an inconsistent state where the main sync phase will always fail.

The second problem is that Argo CD is based on continuous reconciliation, where an application might be synced for several reasons and in different time frames. Unfortunately the traditional DB CLI tools are rarely created with this scenario in mind. Most times they assume they run inside a typical CI pipeline only once.

At this point, Argo CD users are looking for several hacks to force the pre-sync hook to run ONLY in the initial application deployment and not in any subsequent sync events, as this either slows down the deployment or breaks the database completely.

The correct approach is to use a Database migration operator built specifically for Kubernetes. We have written a full guide using the AtlasGo DB operator.

Anti-pattern 24 — Mixing Infrastructure apps with developer workloads

We have seen in anti-pattern 7 several ways to group applications with GitOps (applicationsets, apps-of-apps, Helm umbrella charts).

These grouping methods should always be used for the same types of applications. They should group either infra applications (core-dns, nginx, prometheus) OR the applications that developers create.

You should never mix applications of different types as you force developers to deal with infrastructure errors.

When you create a new cluster and hand it over to developers, it should already have everything they need. The respective application sets should also have installed any infrastructure applications.

Mixing both infrastructure and developer applications might be easier for you (to control directly the deployment order) but always results in a bad user experience for developers.

If developers have access to the Argo CD UI and see a deployment error, they should know immediately that it is something they can fix themselves.

Anti-pattern 25 — Misusing Argo CD finalizers

Argo CD finalizers allow you to define what happens when an Argo CD application (or application set) is removed. You must understand how finalizers work and the impact of adding/removing a finalizer from a resource.

Several teams have accidentally deleted one or more Argo CD applications because they never understood how finalizers work. Other times several resources are “stuck” and never recreated because of a misconfiguration with finalizers.

Finalizers are also very useful when you want to migrate applications from one Argo CD instance to another.
We have written a comprehensive guide about Argo CD finalizers and how to use them.

Anti-pattern 26 — Not understanding resource tracking

This anti-pattern is related to the previous one. First of all it is important to understand how Argo CD tracks and “adopts” Kubernetes resources. You can have Kubernetes resources that are not managed by Argo CD, or Argo CD resources that “owned” by other Kubernetes controllers.

It is vital to know that the relationship between a Kubernetes resource and the Argo CD application that owns it is not always 1–1. You can have

1. Argo CD applications that no longer contain any Kubernetes resources
2. Kubernetes resources that are no longer owned by an Argo CD application

The second scenario is achieved with finalizers. This pattern is very useful when you want to migrate applications from one Argo CD instance to another without downtime. The full process is the following:

1. Argo CD instance A owns all Kubernetes resources
2. You remove all finalizers for all applications (and application Sets)
3. You delete all Argo CD applications
4. The Kubernetes resources are still running just fine. There is no downtime
5. You apply the same Argo CD applications to Argo CD instance B
6. Argo CD instance B will adopt the same Kubernetes resources as before (with no downtime)

You can try this exact scenario of moving Argo CD applications between instances without downtime in our Gitops Certification (level 3) course.

The exact process can be used to upgrade an existing Argo CD instance to a new version in the safest way possible.

Anti-pattern 27 — Creating “active-active” installations of Argo CD

This is the corollary to the previous two anti-patterns. We see several several teams that try to create “active-active” installations of Argo CD with the following requirement:

1. The main Argo CD instance is controlling all applications and deployments
2. There is a secondary Argo CD instance that is also pointed to the same cluster
3. If the main Argo CD instance “fails” the secondary instance “jumps-in”
4. When the main Argo CD instance is restored it “adopts” again all applications.

These teams are disappointed to learn that Argo CD doesn’t support this and even the centralized mode is for controlling other Kubernetes clusters and not other Argo CD instances.

This requirement doesn’t really make sense for Argo CD and teams that look for this “active-active” configuration haven’t really understood resource tracking.

First of all it is important to understand that Argo CD only deploys applications. It doesn’t really control them. If Argo CD fails, new deployments will stop but existing applications will continue to work just fine. And even if those fail for some reason, their pods will be rescheduled/restarted by the Kubernetes cluster (even if Argo CD is no longer operational).

The disaster recovery scenario for Argo CD is straightforward if your team has everything in Git (see antipattern 1). You can launch a second Argo CD cluster and point it to the same application manifests. Argo CD will then adopt the existing Kubernetes resources.

This is even possible if the central Argo CD instance has issues but still runs, as you can use finalizers (as explained in the previous section) to migrate applications to the second Argo CD instance without downtime.

Keeping a second Argo CD instance in “active-active” mode only wastes resources.

Anti-pattern 28 — Recreating Argo Rollouts with Argo CD and duct tape

Bad deployments always happen regardless of whether you are using Argo CD or not. That is a fact for any software team. So how do failed deployments work if you have adopted GitOps?

The simplest way to fix a failed deployment is to roll “forward”. Make a new release or fix the Kubernetes manifests and once you commit, Argo CD will deploy the new changes and hopefully bring back the application to a good state.

Argo CD also includes a “rollback” command which simply points the application back to a previous Git hash. This sounds great in theory but it has 2 major problems:

1. It works only if auto-sync is disabled (anti-pattern 10)
2. It breaks GitOps as your cluster doesn’t represent what is in Git anymore

At this point, teams start creating custom solutions to overcome these limitations. The most common approaches we see are:

1. Trying to detect a failed deployment, and then disable auto-sync on the fly while rolling back
2. Using notifications with external metric providers who will automatically try to revert a commit on their own, so Argo CD will sync as usual to the previous version

These custom solutions are always clunky and create more problems than they solve. You know that your team is falling into this trap if you hear people always asking the question, “How can I disable auto-sync temporarily?”

The recommended solution is to use Argo Rollouts.

Argo Rollouts is a progressive Delivery controller designed for this exact scenario — automated rollbacks when things go wrong. It also comes with its own resource (Analysis) that allows you to look at your metrics during a deployment and roll back without any human intervention.

Argo Rollouts will handle production deployments while non-production environments can still use plain Argo CD.

Anti-pattern 29 — Recreating Argo Workflows with Argo CD, sync-waves and duct tape

The sync wave feature of Argo CD allows you to execute tasks before and after the main sync phase. These tasks should ideally be idempotent and quick to finish. Some examples are:

• Sending a notification to another system
• Performing a quick smoke test
• Verifying that a dependency exists

We see teams that misuse the sync waves in Argo CD with long-running tasks that are part of a bigger process with strict requirements such as

• Automatic retries
• If/else control flows
• Dependency graphs and fan-in/fan-out configuration
• Artifact storage are retrieval

Sync waves were NEVER designed for this kind of requirement. If you try to do this, you will soon resort to custom scripts that nobody wants to maintain. Adopting Argo CD for deployments and then trying to incorporate custom scripts in the sync process is a huge step backwards.

If you have this kind of process you should use Argo Workflows which handle exactly these kinds of requirements.

Argo Workflows are Kubernetes native workflows that offer you all these features out of the box.

Therefore the whole sync process must be:

1. Run an Argo Workflow before the sync process
2. Perform the main Sync phase
3. Run another Argo Workflow after the sync process.

Argo Workflows will then handle all the heavy lifting using declarative Kubernetes resources instead of custom scripts.

Anti-pattern 30 — Abusing Argo CD as a full SDLC platform

Despite all the features and the developer-friendly UI, Argo CD is very simple at its core. It is a powerful sync engine that continuously watches what you have in Git and applies the change to your cluster. All the features are centered around this use case.

However, deploying an application is only part of the software development life cycle (SDLC). Several other requirements must be met before (the CI process) and after (observability) the main deployment.

We have seen several teams trying to expand Argo CD’s scope and make it something it never was. Most importantly, Argo CD has no visibility in your Continuous Integration (CI) process. Argo CD doesn’t know:

1. What are the new features in the container that gets deployed
2. Who made the container build
3. If the application has passed your unit and integration tests
4. If your new container has passed your security scans
5. Who approved the Pull request of the source code change

In fact, Argo CD doesn’t even know that it deploys a new container that includes commits from a source code repo. All it knows is diffing and applying Kubernetes manifests without any insights about the business features behind.

Attempting to integrate this information into Argo CD either through custom plugins or custom YAML segments is always a clunky process. We understand the need for a unified interface. Developer teams love the Argo CD UI and think they can use it as a central dashboard for everything.

That is not the role of Argo CD. You can create a developer portal that uses Argo CD behind the scenes, but Argo CD is not a developer portal itself.

If you want to use a central platform for all your Argo CD instances that also combines deployment information with the CI world, check out Codefresh GitOps cloud.

Conclusion

We hope this comprehensive guide is useful and has provided several good and bad practices when adopting GitOps. Argo CD is a great tool, but it offers several knobs and switches that can be used with undesirable results.

Some features can be abused in several ways, simply because no good documentation exists about the history of the feature, what the intended use is, and what to avoid.

Using this guide you can start your Argo CD journey in the best way possible as you now have the knowledge of what to avoid before investing a significant amount of effort into your Application manifests.

Here is a summary of all the anti-patterns and our recommendation:

1. Not understanding the declarative setup of Argo CD -> Store Application CRDs in Git.
2. Creating dynamic Argo CD applications -> Use Git as the single source of truth for application configuration.
3. Using Argo CD parameters -> Avoid using the parameters feature as it goes against GitOps principles.
4. Adopting Argo CD without understanding Helm -> Understand how Helm works independently before adopting Argo CD.
5. Adopting Argo CD without understanding Kustomize -> Ensure your Kustomize files work on their own before integrating with Argo CD.
6. Assuming that developers need to know about Argo CD -> Design your Argo CD applications so developers can recreate configurations without Argo CD knowledge.
7. Grouping applications at the wrong abstraction level -> Use Application Sets or app-of-apps pattern for proper application grouping.
8. Abusing the multi-source feature of Argo CD -> Use multi-source as a last resort and only for edge case scenarios.
9. Not splitting the different Git repositories -> Separate source code, Kubernetes manifests, and Argo CD application manifests into different Git repositories.
10. Disabling auto-sync and self-heal -> Keep auto-sync/self-heal enabled for all systems, including production.
11. Abusing the targetRevision field -> Always use HEAD in the targetRevision field.
12. Misunderstanding immutability for container/git tags and Helm charts -> Actively set up the ecosystem (Git, Helm repos, artifact managers) to work with immutable data.
13. Giving too much power (or no power at all) to developers -> Balance flexibility and security with Argo CD RBAC, and enable local testing without Argo CD.
14. Referencing dynamic information from Argo CD/ Kubernetes manifests -> Store all values used in manifests statically in Git.
15. Writing applications instead of Application Sets -> Use Application Sets to automate the creation of Application files.
16. Using Helm to package Applications instead of Application Sets -> Learn how Application Sets work and their features.
17. Hardcoding Helm data inside Argo CD applications -> Store all Helm information in Helm values, separate from Argo CD manifests.
18. Hardcoding Kustomize data inside Argo CD applications -> Store Kustomize information only in Kustomize overlays separate from Argo CD manifests
19. Attempting to version and promote Applications/Application Sets -> Promote values or overlays, not Application manifests themselves.
20. Not understanding what changes are applied to a cluster -> Use tools or CI systems to preview and diff rendered Kubernetes manifests.
21. Using ad-hoc clusters instead of cluster labels -> Use cluster labels and Application Sets to distribute applications to different clusters.
22. Attempting to use a single application set for everything -> Have many different Application Sets, each with a different purpose/scope.
23. Using Pre-sync hooks for db migrations -> Use a Database migration operator explicitly built for Kubernetes.
24. Mixing Infrastructure apps with developer workloads -> Separate infrastructure applications from developer workloads.
25. Misusing Argo CD finalizers -> Understand how finalizers work and use them correctly for application deletion and migration.
26. Not understanding resource tracking -> Understand how Argo CD tracks and adopts Kubernetes resources.
27. Creating “active-active” installations of Argo CD -> Avoid active-active setups, rely on Git and resource tracking for disaster recovery.
28. Recreating Argo Rollouts with Argo CD and duct tape -> Use Argo Rollouts for progressive delivery and automated rollbacks.
29. Recreating Argo Workflows with Argo CD, sync-waves and duct tape -> Use Argo Workflows for long-running tasks and complex process orchestration.
30. Abusing Argo CD as a full SDLC platform -> Use a different system as a developer portal or promotion orchestrator.

Let us know in the comment section if we have missed any other questionable practices!

Happy deployments!

Top 30 Argo CD Anti-Patterns to Avoid When Adopting Gitops was originally published in Container Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

In our big guide on how to use ApplicationSets for Argo CD applications, we explained the best practice of having a 3-level structure for all manifests with a clear distinction between Argo CD Application files and Kubernetes resource files.

In that article, we also outlined several anti-patterns that we have seen in the wild, meaning questionable practices that might seem ok at first glance but are problematic in the long run both for developers and for Argo CD operators.
In this guide we want to expand “Antipattern 2 — Working at the wrong abstraction level” and focus on the targetRevision field of the Argo CD application manifest.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  ## DONT DO THIS
  name: my-ever-changing-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example-org/example-repo.git  
    targetRevision: dev
    ## earlier it was "targetRevision: staging" and before that it was "targetRevision: 1.0.0",
    ## and even earlier it was "targetRevision: 1.0.0-rc"
    path: my-staging-app
    ## Previously it was "path: my-qa-app"
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app

Using the targetRevision field as a poor-man’s promotion mechanism is a big trap that impacts both usability and auditability for your Argo CD applications.

How the targetRevision field works

The targetRevision field is part of the Application specification, which is the central Argo CD construct for deploying your GitOps applications.

An Argo CD application describes a link between a Git repository and a Kubernetes cluster. At its most basic form you point Argo CD to the HEAD of a Git repository that contains all your Kubernetes manifests. For convenience, the targetRevision field allows you to define other values apart from HEAD and even select specific Helm versions if you point Argo CD to a Helm chart instead of a Git repository.

You can see all the possible options for tracking strategies at the official documentation page.

Our recommendation is to have “targetRevision: HEAD” in all your application sets and very sparingly use “targetRevision: v1.2.3” for Helm charts that define infrastructure applications that remain mostly static (unlike applications created by developers).

Here is a list of all the possible options and our recommendation:

As always, our recommendation is for production usage of Argo CD in large organizations. On a small scale (e.g. homelabs) or with a small team you can obviously get away with any approach you choose.

The main problem we see with several teams is abusing the target revision field as a promotion mechanism for developer applications. We will explain the shortcomings of this approach and the advantages of our recommendation.

Recommendation: Setting TargetRevision to Git HEAD

Before exploring all the alternative options let’s set the baseline and see why our recommendation of using the default HEAD value is the proper one. This scenario is where environments are based on Helm values and Kustomize overlays (or Git folders).

We will compare the following aspects of each approach:

• Direct changes to code and simplifying day-to-day operations
• Helping developers understand how and when an application is deployed
• Enable easy auditing (one of the main benefits of GitOps)
• Addressing break-glass scenarios and urgent production hotfixes

Setting the value to HEAD and instructing Argo CD to look at the latest version of the manifests/charts/overlays contained in that folder is the most direct and most straightforward approach for developers. It makes any deployment a single step. A developer can update the Kubernetes files and immediately see the change in any affected environment.

The same is true for hotfixes or rollbacks. Developers can change the status of an environment with the familiar git revert and git reset commands.

In fact this is the only setup where developers don’t need to know what Argo CD does at all. We have already explained that developers don’t really care about Argo CD manifests, and keeping them happy by allowing deployments without tampering with Argo CD manifests is the optimal process for them.

Auditing is also as simple as possible. Since Argo CD continuously tracks what is committed in the Git repository of the Kubernetes resources, the deployment history is the SAME as Git history.

The advantages of using Git history as deployment history cannot be overstated. In a large organization and with large numbers of environments, when something breaks the first questions everybody asks are always the same:

1. What did we change in this environment?
2. When did the change happen and who did it?
3. What was the previous version that worked correctly?

Answering these questions quickly is a considerable advantage, especially during an active incident when timing is critical.

In summary, using HEAD for targetRevision is the solution that is fully GitOps compliant (as far as auditing is concerned), easy for developers to use, and flexible enough to cover any possible edge case scenarios and urgent hotfixes.

Let’s compare it with the additional tracking options that Argo CD offers.

Avoid using environment names for targetRevision

The first approach that we can dismiss right away is using branch names for environments (dev, qa, staging, prod etc).

This is the practice where teams either point specific Argo CD applications to long-running Git branches or they use the targetRevision field to “mimic” promotion in the following way:

1. An application is first pointed to the “dev” branch of the manifests using targetRevision
2. Then the Application manifest is updated to point to the “qa” branch of manifests
3. Then, finally, the targetRevision field is set to “staging” or whichever branch is the one before production.
4. The cycle starts again

First, if you use the targetRevision field for long-running static branches, you have bigger problems than environment promotions. We have written a complete guide with all the details about the problems with the branch-per-environment approach.

The biggest problem with this approach, however, is that it completely misses all the benefits of auditing that come from GitOps.

If you constantly update the targetRevision field to different branch names it is tough (if not impossible) to reason about the history of your deployments.

In our baseline scenario of using HEAD, if you want to find what was running in a specific application last Thursday, you can go to your Git history and see which commit was active on the respective Git repository. This is a single-step process that anyone can complete in less than a minute.

If you use branch names in the targetRevision field, looking at history now becomes a multi-step process:

1. First, you need to go to the Git repository that holds the application manifest and find the git commit that was done last Thursday
2. Read the application file and see which branch was inserted in the targetRevision (let’s say it was “dev”).
3. Then you need to go to the Git repository of the Kubernetes manifests and the dev branch, and see what was committed last Thursday

This process is more complex and prone to errors, as you manually need to correlate different git repos and commits for different functionalities (application manifests vs. Kubernetes manifests). Humans should not have to do this under pressure at 3 a.m. (when an incident often happens).

Auditing deployments becomes even more chaotic if you use Application Sets. First, you would have to checkout the Git repos (with the correct revisions) and use the argocd cli to recreate how your application set looked last Thursday before you actually reach the appropriate Kubernetes resources that existed in the cluster.

What completely breaks down this process is all the “temporary fixes” that developers will make if you allow them. A widespread pattern we see is the temporary change of the targetRevision field to another branch “just for testing purposes.”

For example, a team that deploys typically its QA environment from the “qa” branch will often point the ArgoCD application to the “staging” branch to debug an issue that occurred in staging using the resources from the QA branch.

Or, several times, teams create ad-hoc preview environments by pointing different environments to feature branches from individual developers. This is even worse as developer branches can be removed at any time.

In summary, using branch names for the targetRevision field is a very complex process that presents many issues regarding auditing and deployment history. If you work in an organization that has specific financial and legal requirements, you will spend a lot of time trying to keep the auditors happy and manually reconstructing what and where was deployed in the past.

Avoid using semantic ranges in targetRevision

The targetRevision field can also work with version ranges for both Git tags and Helm versions. This sounds great in theory. You enter a value like 3.4.x and then Argo CD will automatically deploy 3.4.0, 3.4.1, 3.4.2 and so on.

You think you have solved the promotion problem with Argo CD but in reality you just introduced two major issues that are especially important to developers.

The first problem is that you have now missed all of GitOps’s auditing capabilities. If you use this pattern, you don’t really have a deployment history in the Git repository with the Argo CD manifests.

If somebody asks what you had deployed in a past moment, you cannot see this information in the Git repository anymore. You need to correlate information between versions from your artifact manager that holds Helm charts or locate the dates where each Git tag was created. This is a cumbersome process and again it is not something that you want to do during an incident.

If your organization has strict legal requirements, using version ranges in targetRevisions will complicate auditing on a magnitude of order.

But the biggest problem is that you can no longer roll back to a previous version. As Argo CD honors semantic version rules, it will only deploy newer versions of an application. This becomes a big problem when critical issues are found in production.

1. You have 3.4.x as a value in targetRevision
2. Version 3.4.2 is deployed right now in production
3. You create a new Git tag with version 3.4.3
4. Argo CD deploys it and it has a critical issue
5. You cannot really go back to 3.4.2 anymore in an automated way

You would have to manually edit the application file and change the targetRevision to 3.4.2 yourself. Then, remember to switch it back to 3.4.x when the issue with 3.4.3 is fixed and 3.4.4 is released.

With this requirement you just forced 2 extra commits that humans have to do during incidents (where you usually want to avoid manual steps).

This process also leaves room for human error. People might forget to switch the targetRevision field back to semantic versioning and wonder why new releases are not getting deployed anymore.

In summary, while version ranges look like an easy way to gain “free promotions”, the problems they create are more impactful than their benefits. The same issues apply if you use semantic versioning for Helm charts.

Avoid using Git hashes in targetRevision

This is the case when an organization wants the “safest” process possible and forces all environments to point to a specific Git hash.

This is a truly locked system, as Argo CD will not deploy anything new anymore. We see this pattern often in financial companies and other companies that want to restrict developers to the greatest extent.

First of all, let’s clarify the perceived “safety” of this approach. As the Argo CD documentation clearly states, even if you set up an Argo CD application with a specific Git Hash, parameter overrides will still take effect. We don’t recommend using parameter overrides, but this means that a person can still affect this application configuration (either by mistake or on purpose).

So don’t assume that using a specific hash in the targetRevision field is a bulletproof method for “securing” your Argo CD applications.

On the other hand, you have ruined the developer experience for all your teams. Nothing gets deployed unless somebody also changes the targetRevision. Developer self-service is not possible at all. Every time developers create a new release, another human or system must update the targetRevision field.

The experience is even worse during incidents. We already know that developers don’t care about Git hashes, so understanding what is deployed where becomes a lengthy and error-prone process. Rolling back is also super difficult as the only way to do it is by switching the targetRevision.

Git hashes are immutable. This means that it is impossible for an external system (e.g. CI) to make any changes to your manifests or understand what needs to be deployed next. Instead, the external system (or human) must always work in 2 steps:

1. First, somebody needs to commit the new version of the manifests with application updates (i.e. bumping the container image)
2. Then you also need to obtain the new Git Hash to put into targetRevision and do a separate commit

It is impossible to do both tasks in a single step as the Git hash from the first action is needed in the second one.

If you assign this responsibility to humans, you just introduced manual steps into your deployment process. If you use an external system, you just added complexity for the sake of complexity.

In summary, using Git hashes in the targetRevision field goes against all DevOps principles and significantly slows down deployments. Using this approach in non-production environments is always a sign that the organization doesn’t really trust its own developer teams.

Use (if needed) Git tags in targetRevision

The last choice for the targetRevision field is to use numbered Git Tags.

This is an acceptable practice, but we recommend it only for locking down production environments. It is better than using plain hashes, as with Git versions, developers can understand where each version is deployed. However, it still suffers from all the issues mentioned in the previous section. Developers cannot deploy or rollback on their own, and extra effort is required during incidents. Like Git hashes, Git tags are considered immutable, meaning you need special care to update them when a new release occurs.

Using Git versions as the target revision in production environments is a good practice if you want to fully control what goes into production. But don’t use this technique for non-production environments. Even if your organization is under legal restrictions, there is no point in making the lives of developers (especially for their QA/staging/dev environments) difficult.

Therefore, our recommendation is:

1. Use a specific Git Tag in the targetRevision field ONLY in production environments
2. Use the simple HEAD tracking method in every other environment.

This gives you the best of both worlds. Developers can deploy fast in non-production environments and can change versions at will. But when it comes to production, they need a human (or external system) to actually update the targetRevision field to a new version.

Note, however, that Git tags can be deleted and recreated with different contents. So, unless you set proper permissions in your Git repository, using a specific Git tag does not guarantee that your environment is locked down. If the same tag gets associated with a different Git hash, Argo CD will happily redeploy the application.

Use specific chart versions for Infrastructure charts

The targetRevision field can also accept specific versions for Helm charts. This is a good pattern to follow but only for infrastructure Helm charts (coreDNS, sealed-secrets, prometheus etc.). Basically you should pin your Helm charts only if all the following apply:

• The chart is stored in a Helm repository (and not in Git)
• The chart represents off-the-shelf software and not something your developers create
• You never “promote” these charts from one environment to another. They just represent infrastructure applications.

If you use Helm charts for your own applications (the ones your developers create) then follow the advice of the first section of this guide. Put them in Git and use HEAD in the targetRevision field. This will give you all the benefits of easy updating, history as auditing, and following the GitOps principles.

It is worth mentioning that Helm versions are mutable by default. Unless your artifact manager has specific support for this, a developer can push a different Helm chart with the same version and override the contents of the previous one. So even if you set targetRevision for a Helm chart to version 2.3.4 it doesn’t mean that it has the same contents of chart 2.3.4 as it was last week unless you configure Helm chart versions as immutable. Some developers prefer to bump only appVersion without also bumping the chart version if they have never changed anything in the chart itself.

Summary

We have now seen all the choices for the targetRevision field and examined the advantages and disadvantages.

The HEAD tracking method is the simplest, most direct, and flexible for developers. It makes incident response as painless as possible, as rolling back can be performed by anyone with simple Git commands. It also allows developers to self-serve their needs. It makes auditing straightforward. We recommend using HEAD as the value in the targetRevision field whenever possible.

For production environments, we understand if you are using a specific Git tag/version. However, employ this approach sparingly and only for systems where you want to “restrict” developers. Make sure you understand all the limitations of this approach. Tags are mutable by default, and you have introduced two extra steps in all your deployment processes.

We strongly recommend against using branch/environment names in the targetRevision field. It completely breaks GitOps history and makes auditing a nightmare.

We recommend against using version ranges. Again, you lose all the benefits of GitOps auditing.

We also recommend against using Git hashes. Especially in non-production environments, it slows down your developers.

Happy deployments!

Abusing the Target Revision Field for Argo CD Promotions was originally published in Container Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

A Kubernetes cluster with default configuration has access to only two deployment strategies:

• Recreate (causes downtime)
• Rolling Update (avoids downtime but you cannot preview or validate the next version in advance)

To get access to more advanced deployment strategies such as blue/green and canaries you need to use a dedicated Progressive Delivery controller such as Argo Rollouts.

We have previously covered several basic and advanced scenarios for Argo Rollouts in our blog. Today we answer another common question which is how you can select which of all live users will have access to the canary deployment.

As a reminder, with a canary deployment, you gradually shift traffic to your live users for the new version’s pods. The canary is finished when 100% of live users see the new pods or when something goes wrong and you revert all of them to the previous/stable version.

In the example above, we start the canary by shifting 20% of network requests to the v2 container, then 50%, and finally 100%. The key point here is that unless you do something special, the network traffic requests that go to the new version are random. Some users might even see both application versions if you are not careful.

In the real world you almost always want specific groups of people to be part of the canary process. Some examples are:

• “Only our internal users should see this new version”
• “Only French users must be part of the canary”
• “Asia should stay in the old version, the US will see the canary only”
• “Only users who have checked the ‘preview checkbox’ must see the canary”
• “The payment gateway should stay in the old version. The intranet should see the new version”

So can we still use Argo Rollouts to cover these use cases? The answer is yes, and in this guide we explain two approaches, one basic and one advanced and also compare the advantages and disadvantages.

The methods we are going to use to decide which users see the canary are

1. Static Routing with extra URLs (limited but simple to implement)
2. Header based Routing (more powerful but also more complex to implement)

If you want to try the examples on your own, all resources are available at https://github.com/kostis-codefresh/rollouts-header-routing-example

Understanding the blast radius of your deployments

The central promise of Argo Rollouts is automatic rollbacks. You deploy a new version and then within 1–2 hours (ideally 15 minutes) either the new version is promoted as stable or it is automatically reverted.

This sounds great in theory, but in practice, you need to understand who will be affected if a deployment fails. Let’s say you are doing canary deployments and need 1 hour to get good metrics to decide about the new version’s health. If the metrics fail, some users will have issues for 1 hour. Is this acceptable? Could you control which users face the disruption and who never participate in the canary?

If you read the official Argo Rollouts documentation, the assumption is that the Rollout controller only focuses on a single application.

In a big organization most services have dependencies. This is especially true for companies that have adopted microservices. So instead of looking at a single service independently, you need to understand how the application works inside the whole cluster.

A more realistic example would be the following:

Here we have an e-shop application with different kinds of users

• External partners can interact with the inventory of the application
• Internal/Intranet users provide customer support and handle the store management
• The general public accesses the public website to order/buy items.

If we choose Progressive Delivery for the “auth” service shown in the middle, we see that even though it is a single service, it is a runtime requirement for 3 other services (portal, admin, store). So even if we apply a canary approach, a failed deployment will affect all users of our application.

Therefore if you need 2 hours for a canary deployment , and you have a failure then ALL your users will be affected for 2 hours. Wouldn’t it be nice if you could control which user groups are affected and which are not?

Isolating specific users instead of random network requests

Making a decision about which users see the canary process and which do not is only one aspect of the deployment process. The other aspect is verifying whether a user is part of the canary. Then, all network requests should always be directed to the preview/canary version of the application.

A widespread misconception about Argo Rollouts is that integrating with a traffic provider allows you to send specific users to the canary version. Unfortunately, this is not true in the default configuration.

Even if you use a traffic provider, the percentage of requests that go to the canary application is completely RANDOM. If you set up Argo Rollouts with a canary step of 30%, Argo Rollouts will only guarantee that 30% of all network requests will go to the canary process. But there is no guarantee that these are from the same users.

This leads to a very common problem for several organizations: Multiple requests from the same user result in different application versions (both old/stable and preview/canary).

In the example of 30%, Argo Rollouts will indeed send 30% of the total network requests to the canary version, but if you look at the network requests for a single user, you might have the case where the first request is not part of the 30%, the next one is, the next is not and so on. This limitation can be catastrophic for applications with a Graphical Interface, as the user might see different components on screen with each subsequent network request (if the canary version also affects the application’s UI).

In the real world, companies don’t want a random percentage of requests to go to the canary version. You want to apply the percentage to individual users.

The expectation is that if you set up a canary of 30%, you expect 30% of users to see the canary and 70% of them are still on the old/stable version. If you log the network requests of a single user however, you want all of them to go to EITHER the canary version OR the stable version and never both.

So can Argo Rollout support this use case of user segmentation instead of network request segmentation?

Example application — Visualize your canary

Our example Rollout can be found at https://github.com/kostis-codefresh/rollouts-header-routing-example/

This repository includes

• The source code of the example application
• A setup for the Traefik proxy
• Example Rollout definitions
• Examples of HTTP routes from the Gateway API

The highlight of the example application is that you can see visually which requests hit one version or the other.

In the screenshot above, a canary is in progress between the application’s v1 and v2. The dashboard performs multiple requests (one for each box shown), allowing you to examine your canary networking in a very simple way.

Approach 1 — Static URL routing

Let’s start with the first use case — reducing the blast radius from a failed deployment. The solution is to create a different URL for each group of users who participate in the deployment process.

We have 3 URLs:

1. The canary/default URL where requests are routed to the canary. Users of this URL will follow the canary traffic as it increases
2. A URL that ALWAYS sends requests to the canary/preview version regardless of the defined percentage
3. A URL that ALWAYS sends requests to the stable/old version regardless of the defined percentage.

Instead of having a single URL for the canary, we can give each user group a different URL according to their risk acceptance.

In the previous example of the e-shop application we can easily accommodate the following imaginary requirements:

1. We want our external partner never to see the canary at all. They will be shown the stable version until the last possible moment
2. We want our public users to be part of the canary process as normal
3. We want our own employees to “see” the new version right away so that they can detect problems as early as possible

In this example we use different URL paths, but we could do the same thing with hostnames (e.g. canary.auth.com, preview.auth.com, stable.auth.com).

Now when a canary process is started

• Users who follow the /stable endpoint will always see the old/stable application version
• Users who follow the /preview endpoint go to the new version straight away
• Users who follow the /canary endpoint participate in the canary as usual.

Here is a timeline for each user group. Blue indicates that network requests go to the old/stable version, and green indicates that they go to the new canary version.

The end result for all groups is precisely the same. They see the new version of the application. The big difference is in failed deployments. If a deployment fails and the canary reverts, users that follow /stable (external partners in our example above) have no impact at all.

Instead of affecting everybody, we have completely isolated our external patterns and also have a different risk acceptance between the general public and our own internal users.

Implementing this approach with Argo Rollouts is straightforward. Instead of using just one network endpoint (for the canary), you create an additional one pointing at the stable service and one more for the preview service.

If you run our example, you can now access 3 URLS (/canary, /stable, /preview). If you start a canary process only the /canary will gradually move to the new version.

Users who visit /preview will see the new version right away:

Users who visit /stable will always view the stable version regardless of the state of the canary process:

This approach needs no source code changes and can be implemented quickly. It has however three significant limitations:

• It is static in the sense that you need to decide in advance which user groups will visit which URL
• You need to notify all dependent services about the new URLs if they don’t want to follow the default behaviour
• It still works at the level of network requests instead of actual users

Approach 2 — Dynamic URL routing

The main limitation of static routing is that you need to identify which user group will use which service in advance. Once you make this selection, you cannot change it after the canary has started.

We still haven’t addressed the problem of users versus requests. In the case of the canary endpoint a random number of requests can see the canary instead of real users.

Using HTTP headers instead of simple endpoints can improve network isolation. Argo Rollouts can detect optional HTTP headers and make decisions accordingly.

In the example above Argo Rollouts will send to the canary all requests that have an HTTP header “X-Canary:true”.

Now we have the capability to have canaries for users instead of just networks. We can modify our application source code to enable this header on the fly.

All requests for users with this header present will be redirected to the canary user. This user group will always see the canary, so this approach works great even for Graphic applications.

HTTP headers are fully dynamic. You can change them on the fly. There are several networking products such as load balancers, api gateways, service meshes that allow you to inject headers or modify headers in a network request.
We can activate this pattern by creating a standard HTTP route and then instructing the canary to create a second one on the fly only if a specific header exists.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: smart-rollouts-demo
spec:
  replicas: 5
  strategy:
    canary:
      canaryService: smart-canary-service
      stableService: smart-stable-service
      trafficRouting:
        managedRoutes:
        - name: always-preview
        plugins:
          argoproj-labs/gatewayAPI:
            httpRoutes:
              - name: my-smart-route
                useHeaderRoutes: true
            namespace: default
      steps:
        - setHeaderRoute:
            name: always-preview
            match:
              - headerName: X-Canary
                headerValue:
                  exact: "yes"  
        - setWeight: 25                        
        - pause: {}
        - setWeight: 100

If you launch the application and this HTTP header is not present, you will see a standard canary with both versions.

If you activate the header then all requests of this user go to the canary!

This setting is now per user. If you open another browser (to simulate a different user), you will see the standard canary behavior again.

In this simple example, the application itself controls the HTTP header. In a real application, a networking component might do this for you (for example, adding this header only to French users).

Conclusion

In this guide we have seen two additional approaches to decide which users can look at the new version during a canary instead of a random percentage of requests (default behavior for Argo Rollouts)

With these approaches

• We have complete control over the impact of a failed deployment. We can choose user groups that will always be redirected to the stable version even when a canary is in progress
• Splitting user groups according to their risk acceptance can be decided in advance or updated on the fly
• Canary behavior is now per user instead of per network request

In both cases, you might need to make source code changes to use a different endpoint or enable/disable a specific HTTP header.

Laser Focused Kubernetes Deployments Using Argo Rollouts and Header Based Routing was originally published in Container Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the previous article in this series, we explained how Argo CD application Sets work and how to use them for organizing your applications in different environments or groups. We received a lot of positive feedback from our readers, and many teams now use the associated Git repository as a starting point for their own Argo CD setup.

Even though we covered Application Sets, and more specifically the Git generator, we never explained how to assign different applications to different clusters. This is a common question from teams managing multiple clusters with different application settings per environment.

In this article, we complete the Application Set puzzle and analyze:

• How to decide which application goes to which cluster
• How to have different application settings per environment
• How to split your clusters into different groups with cluster labels
• How to combine the Argo CD Git Generator with the Cluster generator
• How you can simplify your day-to-day operations using cluster labels.

For more details, we’ve again included an example Git repository.

Managing multiple Kubernetes clusters with Argo CD

Argo CD ApplicationSets let you automate your Application manifests in Argo CD. If you adopt ApplicationSets, you no longer need to deal with individual Argo CD applications’ YAML. You can simply point Argo CD to your clusters and folders, and all the possible combinations get created on the fly for you.

We’ve already seen that you can use ApplicationSets to deploy multiple applications on a single cluster.

We’ve also seen the other dimension — how to deploy the same application to different clusters:

In this guide, we cover the most complex scenario where we have multiple applications and multiple clusters.

To achieve this scenario, we need to use the Cluster Generator of Argo CD. This means you need to connect all your clusters to a single Argo CD instance. This is the hub-and-spoke setup of Argo CD. See our Argo CD architecture guide for different configurations and the advantages and disadvantages of each one.

Using a combination of the cluster and the Git generator, we can create a 2-dimensional matrix of all the pairs (cluster-app) and have Argo CD deploy everything with a single file.

This approach is a great starting point, but in a real organization, we need 2 more capabilities:

1. The ability to enable/disable some applications for some clusters
2. The ability to have different configurations (for example, Helm values) according to the cluster the application belongs to.

The final result is not a full 2-D matrix because some applications won’t exist in all environments. We want to achieve this:

In the example above, Sealed Secrets is NOT present in Cluster C. And the Cert manager is not present in Cluster A. In addition, the “Billing Application” needs to have a different configuration for each cluster.

So, can we achieve these requirements with Application Sets?

Anti-pattern — Creating Snowflake servers with ad-hoc combinations

When faced with the problem of distributing different applications to different clusters, many teams jump straight into very complex solutions that combine multiple Application Set generators. Unfortunately, most hard code custom combinations in the application set files.
A classic example of this approach is trying to individually enable/deactivate a specific application for a particular cluster. We advise AGAINST using such Application Set structures.

 ## DO NOT DO THIS 
- merge:
      mergeKeys:
        - app
      generators:
        - list:
            elements:
              - app: external-dns
                appPath: infra/helm-charts/external-dns
                namespace: dns
              - app: argocd
                appPath: infra/helm-charts/argocd
                namespace: argocd
              - app: external-secrets
                appPath: infra/helm-charts/external-secrets
                namespace: external-secrets
              - app: kyverno
                appPath: infra/helm-charts/kyverno
                namespace: kyverno
        - list:
            elements:
              - app: external-dns
                enabled: "true"
              - app: argocd
                enabled: "true"
              - app: external-secrets
                enabled: "false"
              - app: kyverno
                enabled: "true"
    selector:
      matchLabels:
        enabled: "true"

This file creates snowflake/pet servers where you need to define exactly what they contain. The final result is brittle, requiring significant effort when any major change happens. There are several challenges with this setup:

• It works directly on individual clusters (instead of cluster groups, as we’ll see later in the guide), so it never scales as your requirements change.
• It forces you to hardcode application combinations inside Application Sets. This makes the generators your new unit of work instead of your Kubernetes manifests.
• It makes all day-2 operations lengthy and cumbersome procedures.
• It makes reasoning about your clusters super difficult. Understanding what’s deployed where is no longer trivial.

The final two points cannot be overstated. This approach might look ok at first glance, but the more clusters you have, the more complex it will become.

1. If somebody asks which clusters contain kyverno, you need to scan all individual files for the “enabled” property of the “kyverno” line.
2. Every time you add a new cluster to your setup, you need to copy/paste the list of components from another cluster and start enabling/deactivating each individual component. If you have many components and many clusters, this is an error-prone process that you should avoid at all costs.
3. If you add a new component, you need to go to all your existing files and add it to all the enabled/deactivated lists.
4. It only addresses the first requirement (enabling/deactivating applications for clusters) but not the second one (having different configurations per cluster for the same application).

There is a better way to distribute applications to Argo CD clusters. The approach we DO recommend is using cluster generator labels.

Working with cluster groups instead of individual clusters

In a large organization, you don’t really care about individual clusters. You care about cluster groups. Argo CD doesn’t model the concept of a cluster group on its own, but you can replicate it using cluster labels.

You need to spend some time thinking about the different types of clusters you have and then assign labels to them.

The labels can be anything that makes sense to your organization

• Environment types (for example, QA/staging/prod)
• Regions or countries
• Department or teams
• Cloud provider or other technical difference
• Any other special configuration that distinguishes one or more clusters from the rest

After you have those labels, you can slice and dice your cluster across any dimension and start thinking about cluster groups instead of individual cluster labels.

Because, ultimately, 99% of use cases resolve around cluster groups rather than individual clusters.

• “I want all my production clusters to have application X with Y settings.”
• “I want all my AWS clusters to have X authentication enabled.”
• “Team X will control this environment while team Y will control that environment.”
• “All European clusters need this application.”
• “Application X is installed on both US-East and US-West regions, but with different configurations.”
• “Just for our QA environment, we need this load testing app deployed.”

We’ll see in detail all the advantages when using cluster labels, but one of the easiest ways to understand the flexibility of this approach is to examine what happens for a very common scenario — adding a brand new cluster.

In most cases, a new cluster is “similar” to another cluster. A human operator needs to “clone” an existing cluster, or at the very least, define the new properties of the new cluster in the configuration file.

If you use cluster labels (as we suggest), the whole process requires zero modifications to your application set files.

1. You create the cluster with your favorite infra tool (Terraform/Pulumi/Crossplane, etc)
2. You assign the labels on this cluster (for example, it’s a new QA cluster in US East)
3. Finished!

Argo CD automatically detects this new cluster when it collects all its clusters, and deploys everything that needs to be deployed in the cluster according to its labels. There’s no configuration file to edit to “enable/deactivate” your apps. The process cannot get any easier than this.

Notice that this setup also helps with communication between developers and operators/infrastructure people. Opening a ticket for a new cluster and having several discussions about the contents of the new cluster significantly slows down development time.

In most cases, developers want a cluster that either mimics an existing one or has similar configuration to another cluster group. This makes your job very easy, as you can map directly to cluster labels what developers need.

Creating a new cluster can be a hectic process because you need to validate that it matches the expected workloads and is “similar” to your other clusters. If you use cluster labels, then Argo CD takes care of everything in minutes instead of hours.

Organizing your Argo CD clusters with different labels

Let’s see how all our use cases can work together with a semi-realistic example. You can find all Argo CD manifests at https://github.com/kostis-codefresh/multi-app-multi-value-argocd if you want to follow along.

The repository contains:

• A set of scripts to create 7 clusters running on the same machine using K3d
• Different labels for organizing those clusters into different groups
• Example Application Sets that distribute applications to those cluster groups
• Helm charts and Kustomize overlays for placeholder applications

Here are the 7 clusters that we define with K3d. In a real organization, these clusters would be created with Terraform or another similar tool.

We’ve assigned several example labels on those clusters. Notice that even before talking about applications, the clusters themselves exist in 2 dimensions:

• A promotion flow (QA -> staging -> production) on the horizontal axis
• A region setting (US/EU/Asia) on the vertical axis

The “hub” cluster contains the Argo CD instance that manages all the other clusters. In our example, this cluster only has Argo CD and no end-user applications, so it doesn’t take part in our application sets (it has a label type=hub instead of type=workload).

You can verify or change the labels of each cluster by the Cluster Secret in the main Argo CD instance. Here’s an example of a QA cluster that shows the assigned labels as created by our example GitHub repository.

apiVersion: v1
data:
  [...snip..]
kind: Secret
metadata:
  annotations:
    managed-by: argocd.argoproj.io
  labels:
    argocd.argoproj.io/secret-type: cluster
    cloud: gcp
    department: billing
    env: qa
    region: eu
    type: workload
  name: cluster-k3d-qa-eu-serverlb-1347542961
  namespace: argocd

We’re now ready to look at some typical scenarios. It’s impossible to cover all possible use cases, so we’ll see some representative scenarios for each use case.

The major question that you need to ask yourself is whether you want to deploy an application across different environments with the exact same configuration OR you want a different configuration per environment. The latter is obviously more complex and requires a good understanding of your Kustomize Overlays and Helm value hierarchies, but it’s closer to how a real organization works:

Here are the scenarios we’ll see:

Notice that in our example repository, our applications are grouped in folders by type: manifests, Kustomize, or Helm apps.

In a real organization, you might have different sub-folders for each type, but it’s simpler if you only have to manage one type of application (for example, Kustomize for your own developers and Helm charts for external applications).

Scenario 1 — Run some applications on all workload clusters

Let’s see a very simple use case. We want to deploy all the following common applications to all our clusters only, excluding the Argo CD “hub” cluster. We can take advantage of the “workload” label and point Argo CD to a folder that has all our common applications.

spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - matrix:
      generators:
        - git:
            repoURL: https://github.com/kostis-codefresh/multi-app-multi-value-argocd.git
            revision: HEAD
            directories:
            - path: simple-apps/*
        - clusters:    
            selector:
              matchLabels:
                type: "workload"

You can see the full Application Set at 01-common-apps.yml. This file instructs Argo CD to:

1. Gather all connected clusters that have the “type=workload” label
2. Gather all the Kubernetes manifests found under “simple-apps”
3. Create all the combinations between those clusters and those apps
4. Deploy the resulting Argo CD applications.

If you’re not familiar with generators, please read our Application Set Guide. If you deploy this file, you’ll see the following:

We got 18 applications (6 clusters multiplied by 3 apps) in a single step. Isn’t this cool?

Scenario 2- Choose only GCP clusters and exclude those in AWS

In the next example, we want to install all the applications under `simple-apps` folder only in our Google Cloud clusters, but those applications should not exist in our Amazon clusters. Again, we have created the appropriate labels in advance. In our imaginary organization, all non-production servers run in GCP.

The admin server also runs in AWS, and this is why it won’t get picked up by our application set. You can find the full manifest at 02-gcp-only.yml.

spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - matrix:
      generators:
        - git:
            repoURL: https://github.com/kostis-codefresh/multi-app-multi-value-argocd.git
            revision: HEAD
            directories:
            - path: simple-apps/*
        - clusters:    
            selector:
              matchLabels:
                type: "workload"    
                cloud: "gcp"

This Application Set is similar to the previous one, but now we’re matching 2 labels — one for Google Cloud and one for all our “workload” clusters.

If you apply it, you get several applications, but only to non-prod environments.

Argo CD created a list of applications for only the QA and Staging cluster groups, as they contain clusters that run on Google Cloud.

Scenario 3 — Choose only European Clusters

The big power of labels will become clear when you get requirements that need to work with clusters in an unusual or non-linear way. Let’s imagine a scenario where you need to do something specific to all European clusters because of GDRP regulations.

At this point, most teams realize that the primary way of organizing their clusters was by type (qa/staging/prod), and they modelled the region as a secondary parameter. This creates several challenges and makes people ask the same question, “Does product X support deployments in regions?”.

But when using the cluster generator, all labels are first-level constructs, allowing you to make any selection possible. We can focus on European clusters by just defining our region the same way as any other scenario.

Today, only the QA and Production environments have a European server. But tomorrow, you might add one in the Staging environment WITHOUT any modifications in your application Set.

We select all European servers by region with file 03-eu-only.yml

spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - matrix:
      generators:
        - git:
            repoURL: https://github.com/kostis-codefresh/multi-app-multi-value-argocd.git
            revision: HEAD
            directories:
            - path: simple-apps/*
        - clusters:    
            selector:
              matchLabels:
                type: "workload"    
                region: "eu"  

Deploying this application set will instruct Argo CD to place all the applications under simple-apps folder only in the European servers:

If you add a new European Region in the Staging environment, then in the next Argo CD sync, that cluster also gets all applications defined for Europe, with zero effort from the administrator.

Scenario 4 — Choose a specific cluster among a cluster group

If it wasn’t clear from the previous examples, the label selector for clusters works in an “AND” manner by default. So the more labels you add in the selector, the more specific the application set becomes.

This means that even if you really want to select a single cluster among a group, you can just define all the labels that correctly identify it.

We want to select the Asian Environment for Production (which is a specific Kubernetes cluster).

The application set that selects this cluster is at 04-specific-cluster.yml.

spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - matrix:
      generators:
        - git:
            repoURL: https://github.com/kostis-codefresh/multi-app-multi-value-argocd.git
            revision: HEAD
            directories:
            - path: simple-apps/*
        - clusters:    
            selector:
              matchLabels:
                type: "workload"  
                region: "asia"    
                env: "prod"    

The labels we have defined in the application set map only to one cluster. Argo CD will look at this application set and find all clusters that have type=workload AND region=asia AND env=prod.

Applying the file you will see the following

As expected, Argo CD deployed all the applications under simple-apps folder only to the production cluster in Asia.

Scenario 5 — Different Kustomize overlays for the QA clusters

For simplicity, in all the previous examples, all our applications use the same configuration across all clusters. So even if our cluster generator selected multiple clusters, they all used the plain manifests we defined.

While this approach can work for some trivial applications, you almost certainly want to use a different configuration per cluster. This can take the form of DNS names, database credentials, security controls, rate limiting settings, etc.

For our next example, we’ll use Kustomize overlays. For each application, we have the base configuration plus extra settings in overlays or Kustomize components.

We have covered Kustomize overlays in detail in the promotion article and explained how they work with Argo CD in our Application Set guide, so make sure you read those first if you’re not familiar with overlays.

For the cluster selector, we’ll choose the QA environment this time (which corresponds to 2 clusters).

The application set that selects the QA clusters and deploys applications with the respective configuration is at 05-my-qa-appset.yml.

spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - matrix:
      generators:
        - git:
            repoURL: https://github.com/kostis-codefresh/multi-app-multi-value-argocd.git
            revision: HEAD
            directories:
            - path: kustomize-apps/*/envs/qa  
        - clusters:    
            selector:
              matchLabels:
                type: "workload"    
                env: "qa"  

The Matrix generator selects all clusters that match the QA/Workload labels and applies only the applications that have a QA overlay.

Apply the file, and you see all QA deployments:

The important point here is that for each application, only the QA overlay is selected.

You can see in the Git repository that the “Invoices” application comes with configurations for all environments, but we appropriately employ only the QA one in our application set.

Scenario 6 — Different Kustomize settings for US and EU in production

There are many more examples we can show with this setup. Be sure to read the documentation of the cluster generator. One important point is that you can use the output of this generator as input to another generator.

As a final example with Kustomize, let’s see a scenario where we want to deploy our applications to Production Europe and Production US, but not in Asia.

Remember that by default, server labels work in “AND” mode. So if we simply list “us” and “eu” as labels, Argo CD will try to find all clusters that have both labels at the same time. We don’t want this, as no cluster matches this description.

Also, unlike the previous example where we specifically asked for the “QA” overlay, now we want to choose the overlays that match whatever the cluster type/region is (either prod-us or prod-eu).

You can find the full application set at 06-my-prod-appset.yml.

spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - matrix:
      generators:
        - clusters:    
            selector:
              matchLabels:
                type: "workload"      
                env: "prod"
              matchExpressions:
              - key: region
                operator: In
                values:
                  - "eu"
                  - "us"        
        - git:
            repoURL: https://github.com/kostis-codefresh/multi-app-multi-value-argocd.git
            revision: HEAD
            directories:
            - path: 'kustomize-apps/*/envs/{{.name}}'

The first thing to show here is the matchExpressions block. This lets you choose clusters in an “OR” manner. We want all clusters that are either EU or US AND in production.

The second point is using the output of the cluster generator as input to the Git generator. The “{{.name}}” variable will render to the name of the cluster matched, forcing the Git generator to load the respective Kustomize overlay for each environment.

Apply the file and you will see production deployment in EU and US but not in Asia:

And most importantly, you see that each server loads the configuration for its own region:

You should now understand how to select any combination of clusters and apply your exact choice of Kustomize overlays according to the “type” of each cluster.

Scenario 7 — A Helm hierarchy of values for the QA environment

Cluster labels can also work with your Helm charts and values.

As a starting example, let’s deploy our Helm charts to the two QA clusters using the same configuration for both.

You can find the full application set at 07-helm-qa-only.yml.

spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - matrix:
      generators:
        - git:
            repoURL: https://github.com/kostis-codefresh/multi-app-multi-value-argocd.git
            revision: HEAD
            directories:
            - path: charts/*
        - clusters:    
            selector:
              matchLabels:
                type: "workload"    
                env: "qa"  

The generator part of the file selects our charts and applies them to all clusters with the QA/workload label.

We have 2 example charts in the Git repository, so Argo CD created 4 applications for us (1 for each region).

Scenario 8 — Different Helm values for the European environments

Like the Kustomize example, we want to make our examples more advanced and have different value files per environment.

The same Git repository also contains a set of Helm values for each environment.

We have covered Helm value hierarchies and Argo CD applications in our Helm guide, so please read that guide first if you don’t know how to create your own value hierarchies.

Let’s deploy our Helm charts at all European servers:

This time, however, we want to specifically load the European values only instead of all values.

You can find the full application set at 08-helm-eu.yml

spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - matrix:
      generators:
        - clusters:    
            selector:
              matchLabels:
                type: "workload"      
                region: "eu"                        
        - git:
            repoURL: https://github.com/kostis-codefresh/multi-app-multi-value-argocd.git
            revision: HEAD
            directories:
            - path: charts/*  

The generator part is straightforward. It applies all charts to clusters with the EU/Workload labels. The smart selection of values happens in the “sources” section of the generated application:

sources:
  - repoURL: https://github.com/kostis-codefresh/multi-app-multi-value-argocd.git
    path: '{{.path.path}}'
    targetRevision: HEAD
    helm:
      valueFiles:
      - '$my-values/values/{{index .path.segments 1}}/common-values.yaml'  
      - '$my-values/values/{{index .path.segments 1}}/app-version/{{index .metadata.labels "env"}}-values.yaml'                
      - '$my-values/values/{{index .path.segments 1}}/regions/eu-values.yaml'              
      - '$my-values/values/{{index .path.segments 1}}/envs/{{index .metadata.labels "env"}}-eu-values.yaml'                  
  - repoURL: 'https://github.com/kostis-codefresh/multi-app-multi-value-argocd.git'
    targetRevision: HEAD
    ref: my-values

Here we apply the appropriate values according to:

1. The chart name (index .path.segments 1)
2. The environment label that exists on the cluster (index .metadata.labels “env”)

In this example, you see how you can query the cluster itself for its own metadata.

If you apply this file, you see both charts deployed in the European servers.

But most importantly, you see that each environment gets the correct values according to its type:

Notice that in both cases, we still have some common values that apply to both environments.

Scenario 9 — Different Helm values for all 3 Production regions

As a final example with Helm, let’s deploy to all production regions with the appropriate settings for each one.

We choose all 3 regions in our cluster generator.

You can find the full application set at 09-helm-prod.yml.

Like before, we select all 3 regions in an “OR” manner and apply our charts.

spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - matrix:
      generators:
        - clusters:    
            selector:
              matchLabels:
                type: "workload"      
                env: "prod"
              matchExpressions:
              - key: region
                operator: In
                values:
                  - "eu"
                  - "us"  
                  - "asia"                      
        - git:
            repoURL: https://github.com/kostis-codefresh/multi-app-multi-value-argocd.git
            revision: HEAD
            directories:
            - path: charts/* 

For each application, we query each cluster for its environments and region.

sources:
  - repoURL: https://github.com/kostis-codefresh/multi-app-multi-value-argocd.git
    path: '{{.path.path}}'
    targetRevision: HEAD
    helm:
      valueFiles:
      - '$my-values/values/{{index .path.segments 1}}/common-values.yaml'  
      - '$my-values/values/{{index .path.segments 1}}/app-version/{{index .metadata.labels "env"}}-values.yaml'              
      - '$my-values/values/{{index .path.segments 1}}/env-type/{{index .metadata.labels "env"}}-values.yaml'  
      - '$my-values/values/{{index .path.segments 1}}/regions/{{index .metadata.labels "region"}}-values.yaml'              
      - '$my-values/values/{{index .path.segments 1}}/envs/{{index .metadata.labels "env"}}-{{index .metadata.labels "region"}}-values.yaml'                  
  - repoURL: 'https://github.com/kostis-codefresh/multi-app-multi-value-argocd.git'
    targetRevision: HEAD
    ref: my-values

All charts are now deployed in all regions:

You can also verify that each environment picks the correct settings from the value hierarchy:

You have now seen how to apply value hierarchies with Application Sets and cluster labels.

Day 2 operations

We now reach the most important point of this guide. We’ve seen how cluster labels let you define exactly what goes into which cluster. You might be wondering why this is the recommended solution and how it’s better than other approaches you’ve seen.

The answer is that with cluster labels you treat your application sets in a “create-and-forget” function. After the initial set up, you shouldn’t need to touch your application sets at all. This means that maintenance effort is zero, which is always the OPTIMAL way of evaluating any architecture decision.

Let’s see some semi-realistic scenarios of using our recommendation in a real organization.

Imagine you just organized all your application sets with cluster labels. All files are committed in Git, and all applications are successfully deployed. Everything runs smoothly.

Scenario A — Removing a server

On Monday, you need to decommission the US/prod server. You remove the “us” and “prod” tags from the server. In the next sync, the cluster generator from all related appsets doesn’t pick it up and nothing gets deployed there. You don’t really care how many application sets touched this cluster. They will all stop deploying there automatically.

Changes you had to do in your Application Sets: ZERO

Scenario B — Deploying a new application

On Tuesday, a developer wants a new application in the QA environment. You commit a new overlay for QA configuration for that app. All QA application sets pick it up in the next sync and deploy it to any/all clusters that deal with QA. You don’t really care how many application sets affect QA or how many clusters are contained in QA. They will all get the new application in the same manner.

Changes you had to do in your Application Sets: ZERO

Scenario C — Adding a new Cluster

On Wednesday, you need to add a new cluster to replace the decommissioned one. You create the new cluster with Terraform/Pulumi/Crossplane/whatever and just assign it the appropriate tags (“us”,”prod”, “workload”,”aws”). All respective Application Sets see the new cluster in the next sync and deploy whatever needs to be deployed there. You don’t really care how many Application Sets touch this cluster. The cluster will get the exact same applications as it had before.

Changes you had to do in your Application Sets: ZERO

Scenario D — Copying an application

On Thursday, a developer says that a specific application that exists in staging also needs to go to QA.

You copy the staging overlay to a QA overlay for this application and ask the developer about the correct settings. In the next sync, all the QA Application Sets pick it up and deploy it. The developer doesn’t need to know anything about application sets or cluster labels. In fact, they could just do this deployment on their own if they had access to the Kustomize overlays.

Changes you had to do in your Application Sets: ZERO

Scenario E — Central cluster change

On Friday, you’re told that ALL your clusters now need sealed-secrets installed. You add a new configuration for sealed secrets in your “common” folder and commit it to Git. Then the “Common” Application Set (that applies to all clusters) picks it up and applies it to all clusters.

Changes you had to do in your Application Sets: ZERO

Essentially, the Application Sets only need to change when you need to add another dimension to your servers (i.e., new labels) for something that was not expected. If you completed a proper evaluation in the beginning, and communicated to all parties how all the servers are going to be used, then this scenario won’t happen very often. For daily operations, the Application Sets just sit in the Git repository without anybody (operators or developers) having to make any changes at all.

The other big advantage of cluster labels is that they work the same, regardless of how many servers you have. The Application Sets that work with labels will automatically update on their own, even if they manage 1, 10, or 100s of servers that you connect to the central Argo CD instance.
Let’s compare those same scenarios with the approach we do NOT recommend, where application sets explicitly enable/deactivate components/apps in each specific server.

## Do not do this
- merge:
      mergeKeys:
        - app
      generators:
        - list:
            elements:
              - app: external-dns
                appPath: infra/helm-charts/external-dns
                namespace: dns
              - app: argocd
                appPath: infra/helm-charts/argocd
                namespace: argocd
              - app: external-secrets
                appPath: infra/helm-charts/external-secrets
                namespace: external-secrets
              - app: kyverno
                appPath: infra/helm-charts/kyverno
                namespace: kyverno
        - list:
            elements:
              - app: external-dns
                enabled: "true"
              - app: argocd
                enabled: "true"
              - app: external-secrets
                enabled: "false"
              - app: kyverno
                enabled: "true"
    selector:
      matchLabels:
        enabled: "true"

What actions do you need for each scenario?

• Scenario A — Removing a server

1. You need first to locate all Application Sets that “choose” this server.
2. You then need to edit all application Sets and “deactivate” all the components they contain.
3. You need to commit and sync all changes.
4. There is a risk that you either forgot an application set or forgot to “deactivate” a line
5. The more servers you have, the more complex is the process

• Scenario B — Deploying a new application

1. You first need to locate all Application Sets that choose the servers that need this application.
2. You need to edit all those application Sets and add a new line for this application.
3. You need to commit and sync all changes.
4. There’s a risk that you either forgot an application set or forgot to add a line for the new application.
5. The more servers you have, the more complex the process.

• Scenario C — Adding a new Cluster

1. You need to understand how this cluster is “similar” to other clusters.
2. You either need to create a new Application Set for this cluster or locate all Application Sets that touch it.
3. You need to add all new lines of enabled/disabled components for this cluster.
4. There’s a risk that you either forgot an application set or forgot to add a line for the new application.

• Scenario D — Copying an application

1. You need first to locate all Application Sets that choose the news servers for this application.
2. You need to edit all those application Sets and locate the line for this application and change it to “enabled”.
3. You need to commit and sync all changes.
4. There’s a risk that you either forgot an application set or forgot to “enable” the component.
5. The more servers you have, the more complex the process.

• Scenario E — Central change

1. You need first to locate all Application Sets that you manage.
2. You need to edit all those application Sets and add a new line for this common application.
3. You need to commit and sync all changes.
4. There’s a risk that you either forgot an application set or forgot to add a line for the new application
5. The more servers you have, the more complex the process.

It shouldn’t be a surprise that having snowflake servers where you must enable/deactivate each application individually is a much more complex process than working with cluster groups identified by labels.

Developers and self-service

At the start of this guide, we talked about effective communication with developers. Another major reason that makes cluster labels the optimal solution is that they’re fully automated. At each sync, the Argo CD cluster generator detects which clusters have the appropriate labels and does whatever needs to be done (deploy or undeploy an application).

Your developers don’t need to know anything about cluster labels. In fact, they don’t even need to know about Application Sets. Developers can work with a Git repository that holds standard Helm charts/Kustomize overlays/plain manifests, and their instructions are super simple:

• If they add a new overlay in the QA folder, then that application will be deployed in the QA environments regardless of the number of servers.
• If they delete an overlay, that application will be undeployed.
• If they want a brand new application, they can just commit the new overlays or Helm values in a specific folder, and Argo CD will pick it up.

Developers can work on their own without opening any tickets or waiting for you to do something for them.

This comes in complete contrast to the anti-pattern we explained above, where you manually enable/deactivate applications in each specific cluster. If you follow this approach, all actions become a two-step process:

1. The developer adds their overlay or Helm values in a Git repository.
2. Then you MUST go to all your Application Sets and manually “enable” the new application.

Preventing developers from deploying their applications and waiting for you to do something is the fastest way to create bottlenecks in your organization.

There is no reason for this complexity when using cluster labels is a much better choice.

Conclusion

In this guide, we explained in detail how to create cluster groups with Argo CD using custom labels. We have also seen:

• How to use the cluster generator to select a cluster in an “AND” and “OR” fashion.
• How to deploy applications to multiple clusters using the same configuration.
• How to deploy Kustomize applications with different overlays per cluster.
• How to deploy Helm applications with different value sets per cluster.
• How to perform common day-2 operations with many Argo CD clusters.
• Why our recommended approach is the optimal one, as the number of clusters and developers grows in your organization.

You can find all Application Sets and manifests at https://github.com/kostis-codefresh/multi-app-multi-value-argocd.

Happy labeling 🙂

Distribute Your Argo CD Applications to Different Kubernetes Clusters Using Application Sets was originally published in Container Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

The cloud ecosystem has reached a turning point. Tools for operators/administrators are now mature and can handle most day-to-day operations that deal with Kubernetes clusters. Finally, we can turn our focus to application developers and their needs.

If you look at all the Kubernetes tools available, you’ll understand that most of them treat Kubernetes as another form of infrastructure. You can easily find tools that install Kubernetes, monitor Kubernetes, secure Kubernetes, do cost estimations for Kubernetes, etc. But how many Kubernetes tools can you find that target application developers and their day-to-day responsibilities?

Several companies even try to hide Kubernetes completely from developers by using leaky abstractions or so-called developer portals. These adoption efforts almost always fail simply because nobody asked the developers what they really need. Don’t fall into this trap.

In this article, we see some common examples of what companies “think” about developers’ needs versus what developers need in practice, in the context of application development for Kubernetes.

Confusing clusters with environments

When designing a deployment workflow, most teams center the discussion around individual clusters. You often hear people talk about direct mappings between clusters and “environments”.

• “This is our production cluster.”
• “Our staging cluster is down.”
• “We need a new cluster for QA.”

This forces all Kubernetes tools to expose clusters as first-level constructs in their user interface (UI). If your team has created any kind of dashboard for Kubernetes, I can bet that the left-panel navigation contains a “cluster” entry where people can look at individual clusters.

In reality, developers never care about individual clusters. Most times, they care about:

• Cluster groups that behave the same (e.g., prod-us, prod-asia, prod-eu).
• A set of namespaces within a big cluster (most common in shared qa/staging clusters).
• A combination of the above.

Developers have a different mindset:

• “I am ready to ship this feature to production.”
• “Oh, my new feature is failing in the QA environment.”
• “That is strange, application 1.23 works ok in QA, but presents an error in the staging environment.”

So, how many clusters are in production, QA, or staging? It does NOT matter to developers. Folks only think that developers care about environment settings and, more specifically, what differs between environment configurations.

This means that if your internal portal/developer platform looks like this:

You need to redesign it like this:

Let me repeat that again. Developers do NOT care about individual Kubernetes clusters. They mostly care about the different settings between the cluster groups that represent each “environment”.

Promotions are more important than deployments

We’ve established that developers prefer thinking about environments and not individual clusters. Let’s see another common misconception with tools that target developers. If you look at the most typical scenario of how a feature reaches production, this is the process:

1. A developer performs an initial deployment in the first environment — let’s call it QA environment.
2. After passing several tests and reviews, the feature gets promoted to the staging environment.
3. After passing several tests, the feature gets promoted to production.
4. Depending on the company, there might be several other intermediate environments where promotions happen (e.g., load testing).

Several tools promise to simplify deployments for Kubernetes developers. It turns out that developers are actually interested in promotions. There are 3 main reasons for that:

1. A deployment (where code is packaged in a brand new artifact) happens only once in the first environment. In all the subsequent environments, developers want to promote an existing image/configuration/release/artifact and not deploy anything from scratch
2. Problems with promotions have more impact. Promoting to production is always a risky process. Deploying to QA is not.
3. Promotions can often fail due to external configuration not directly controlled by developers.

The last point cannot be overstated. One of the most common cases for failed deployments in production is the difference in environment configuration. Developers dutifully tested their application in QA and staging, and everything worked fine. Then the application failed to deploy in production because of an unexpected change in production settings completely outside the scope of the application container.

Continuing our wireframe from an imaginary developer portal, most teams think that developers need this:

The UI is problematic for many reasons:

1. Developers can deploy an older version to any environment by mistake.
2. Developers need to manually correlate versions between environments and understand what was in the next/previous environment.
3. Most times, versions correspond to Docker image versions that exclude external configuration.

But as explained already, developers just want to promote. So they would prefer this:

Notice that this makes a developer’s job very easy. There are also several guardrails in place. At a minimum, the production drop-down can ONLY promote what is currently in staging and nothing else (or maybe the last 3 versions that are available in staging). You can also perform checks and present warnings in several scenarios (for example, if a developer tries to promote something to production that is clearly failing in staging).

Developers don’t care about Git hashes

Git hashes are great when it comes to source code operations. When developers perform basic merges, cherry-picks, and rebase scenarios, they do care about Git hashes. But when it comes to deployments, Git hashes mean nothing:

• Git hash names have no specific order. You cannot look at 2 hashes and understand which is newer/older.
• Git hashes can only capture the state of a source code snapshot or the specific commit for Kubernetes manifests, but never both at the same time.
• They allow for mistakes when it comes to deployments, especially when developers have to copy/paste hashes between different tools.

What developers care about instead is software versions. Version numbers are simple to read, simple to reason, and simple to understand.

Products and dashboards that expose Git hashes as a central concept are cumbersome and difficult to use. It’s ok if Git hashes are an additional piece of information for a deployment, but they should never be used for direct promotions or other day-to-day operations.

What developers really want to see are versions. Or, at least some kind of numbering where ordering is easy to understand.

So, where do these versions come from?

Here we reach one of the biggest misconceptions about container images for Kubernetes. Several tools (and dashboards) simply use a special tag with a version number on a container that gets “promoted” from each environment to the next. Using a container image version for the promotion process might make sense at first glance.

However, this approach misses 2 facts completely:

• Docker tags are mutable by default. Just because you see a container called my-image:v1.0 in one environment and another container my-image:v1.0 in another environment, that doesn’t mean they’re the same application. They might be completely different.
• Used as an application version, the container tag only works in 80% of cases. Developers sometimes need to promote a container image AND associated configuration (like configmaps and secrets). In those cases, the container tag is NOT enough to understand what gets promoted and where.

You can partially solve the first problem with tooling support. You need to instruct ALL tools that take part in the software lifecycle to treat container tags as immutable. This is also a very basic requirement for any package registry that your organization is using.

You can solve the second problem with Helm charts. Helm charts give you access to 2 additional “version” properties (in addition to your container tag). You can annotate a Helm chart with an “application” version and also have a different version for the chart itself. This way, when you promote a Helm chart, you can promote a container image PLUS additional configuration in a single step.

However, not all organizations use Helm charts. Several teams prefer to use Kustomize or even plain manifests. In this case, the versioning problem still exists.

Unfortunately, most tools assume that developers only care about container images and center their whole interface around image tags.

Developers would prefer a system that lets them promote configuration and container tags at the same time. This would cover 100% of their needs and cater even to edge case scenarios where they only promote configuration, while the container image stays the same.

Tools that expose Git hashes and assume developers only work with container tags completely miss the way Kubernetes applications work.

Don’t abuse pipelines as promotion mechanisms

Many developers see Continuous Delivery (CD) as the next evolution of Continuous Integration (CI). After all, before you deploy a container image, you need to build and test it first.

Several teams that switch to cloud-native development make their first step in CD by abusing their existing CI pipelines. Developers love to see a single pipeline that shows the whole picture for a specific feature, from the initial code commit all the way through to production.

The main problem here is that the typical CI pipeline only knows what is happening WHILE it’s running. After it finishes, it has no visibility into the actual cluster.

This leads to the classic problem of failed deployments in the following manner:

1. A developer commits a new feature (or merges something in a branch).
2. The CI pipeline starts and then builds/tests a container image with success.
3. The CI pipeline deploys the image to a Kubernetes cluster and optionally runs some checks.
4. Everything looks good, and the pipeline shows its status as “green”.
5. Ten minutes later, the application has issues (memory leaks, wrong dependencies, missing DB).
6. Developers get paged about a failed deployment, even though the pipeline STILL shows as green.

Your organization probably falls into this trap if developer teams always talk about “lack of deployment visibility”, “wasting time to troubleshoot deployments”, “not enough production access”, and similar complaints.

In reality, developers look at the CI system for the basic build. Then they need to go to another system (usually a metrics/monitoring solution) to understand what’s happening with their application.

The key takeaway here is that instead of a basic CI pipeline, you need a system that gives developers real-time information about deployments and promotions. In that system, the “green” status means that the application is healthy RIGHT NOW, not 5 minutes before.

Now developers can use a single interface for deploying/promoting, and understanding if the application is healthy. They can go to their metrics solution when things go wrong, but in the happy path scenario, a single system can tell them if the application is successfully running in a Kubernetes cluster (or a specific environment, as we saw earlier in the article).

Stop deploying and start promoting

You should now understand what developers actually need and why existing solutions aren’t designed with Kubernetes/GitOps in mind. There are several initiatives right now for investing in developer portals in big organizations, and, unfortunately, developers don’t always have a say about exactly what they need from an internal platform.

The next question is whether you actually need to create a platform like this from scratch. You might think that Argo CD is a solution that helps developers with Kubernetes deployments and that simply adopting Argo CD will make developers happy. In reality, Argo CD is a great sync engine, but doesn’t try to solve changes between applications, promotions, or environments. For example, Argo CD doesn’t have the concept of environments, instead only operating with individual clusters.
This is why we extended Argo CD to create Codefresh GitOps Cloud. We looked at existing solutions and understood that developer experience is always an afterthought, even in newer platforms that are supposedly designed with Kubernetes in mind.

Codefresh GitOps Cloud implements all the best patterns we explained that developers need:

1. It works with environments instead of individual clusters. Each environment can be one cluster, a set of clusters, a set of namespaces, application labels, or any combination of those.
2. It’s based around promotions. Developers can easily understand what’s different between 2 environments and how to move an application from one environment to the next.
3. Git hashes are there if you need them. The central construct, however, is products and their versions. A product is a new entity that includes an application along with its configuration and its container images. When you promote from one environment to another, you promote the whole application and not individual container images.
4. The graphical dashboard always shows real-time information. When you see a “green” checkmark, it means that an application is running successfully in an environment right now. Developers can detect right away what deployment was successful and what failed without going to another system.
5. Argo CD and its amazing sync engine power everything behind the scenes.

You can use Codefresh GitOps Cloud today along with your existing CI system (like Jenkins and GitHub Actions). GitOps Cloud doesn’t replace your CI solution. It makes developers happy by giving them a dedicated platform for application promotions, which is what developers really need (instead of playing deployments).

Oh, one more thing. If you already have your own Argo CD instance, you can bring it along!

Ready to start your GitOps journey with Codefresh? Try GitOps Cloud free for 45 days now.

Why Environments Beat Clusters For Dev Experience was originally published in Container Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Two of the most popular guides we’ve written are the GitOps promotion guide and the ApplicationSet guide. Used together, they explain an end-to-end solution for organizing your GitOps applications and promoting them between different environments, while keeping things DRY by using application sets.

Both of these guides use Kustomize. We offered some hints for Helm users, but this is a dedicated guide for Helm applications. Organizations that choose Helm have several other challenges to overcome when adopting GitOps. This is because you can also store Helm charts in dedicated Helm repos (apart from Git). Helm charts have versions on their own.

Unfortunately, we’ve seen teams try to replicate features in Argo CD that Helm already supports. Or, some teams try to adopt Argo CD without first understanding the basics of Helm.

In this guide, we include several Helm-related topics to consider when adopting GitOps and Argo CD in your organization.

We describe:

• The recommended Helm structure for GitOps repositories
• When to use the multi-source feature of Argo CD and when not to use it
• How to create Helm value hierarchies and why this is important
• Common Helm bad practices and misconceptions that people carry over to Argo CD

If you like our guides but wonder how our advice works with Helm and not Kustomize, then this guide is for you!

You can find the full example at https://github.com/kostis-codefresh/multi-sources-example.

How Helm values work

The original design for Helm is to have a generic chart (without any hardcoded configuration) and then use several different value files that hold environment-specific configuration. We’ve seen this scenario in detail in our guide about Helm values. It’s also possible to have default values for a chart if no external values are added.

You need to decide if your Helm chart will map to a specific application or not. One approach is to have a Helm chart for every application you develop. In this case, the Helm chart has 2 versions. One version keeps a history about the chart itself and the second version (appVersion in the Chart.yml file) keeps track of the application contained in the chart.

With the second approach, you have a reusable “single” chart that’s targeted at many applications. It doesn’t hold a specific application on its own. You always need to combine it with values that define the image/tag/configuration.

Both approaches are valid and Argo CD supports both. It makes sense, however, to select a single strategy for your organization.

Assuming the classic trinity of QA/Staging/Prod you should be able to easily install any chart with the appropriate configuration, like this:

helm install ./my-chart/ --generate-name -f values-qa.yaml
helm install ./my-chart/ --generate-name -f values-staging.yaml
helm install ./my-chart/ --generate-name -f values-production.yaml

In this very simple example, the charts are saved in Git as normal files along with their possible configurations.

Creating reusable configuration with Helm value hierarchies

The structure explained above works for most organizations. In some cases, however, the application has a complex configuration you need to define for a large number of environments. You could duplicate the values file for each environment, but this would create a cumbersome setup where you need to change common configuration between some environments.

To overcome this challenge, Helm also supports value hierarchies. This means that instead of defining a single value file when installing a chart, you can define multiple. Helm then merges all value files, and for properties defined multiple times, the last file wins.

This means that you can do:

helm install ./my-chart/ — generate-name -f common.yaml -f more.yaml -f some-more.yaml

If you have properties that are both in common.yaml and in more.yaml, the latter will prevail. In the same way, all properties defined in some-more.yaml will function as “overrides” for everything else you defined in the previous files.

The obvious advantage of this pattern is that you can create a hierarchy of Helm value files where “early” files contain common values for most environments and “late” files contain specific values for individual deployments.

A possible setup would be:

common-values.yaml
+-----all-prod-envs.yaml
    +----specific-prod-cluster.yaml

This means common-values get shared between ALL your deployments unless they’re overridden in “all-prod-envs” for all your production environments, which can also be overridden by “specific-prod-cluster.yaml” for a specific cluster.

To make this work, you need to apply all files in the expected order — remembering the last one wins.

helm install ./my-chart/ — generate-name -f common-values.yaml -f all-prod-envs.yaml -f specific-prod-cluster.yaml

If you’re using Helm for your own applications, it’s essential you create an appropriate hierarchy for all your Helm applications that balances flexibility with minimal duplication.

It’s also important to remember that Helm has these hierarchies built-in. They’re unrelated to Argo CD.

Using Helm value hierarchies with Argo CD

After spending time designing your hierarchy of Helm values, you’re now ready to use Argo CD. The good news is that Argo CD supports Helm hierarchies out-of-the-box without any additional configuration.

Here’s an example application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    chart: my-chart
    repoURL: https://github.com/my-example-app
    targetRevision: 2.43
    helm:
      valueFiles:
      - common-values.yaml
      - all-prod-envs.yaml            
      - specific-prod-cluster.yaml
  destination:
    server: "https://kubernetes.default.svc"
    namespace: default 

The valueFiles property accepts an array of value files. Like the Helm CLI, each file mentioned in the list overrides all values from the previous files. The last file wins over the previous ones.

Why is this important? We’ve seen several teams try to replicate this pattern of default/environment-specific values using several hacks and workarounds, as we described in the previous guide. Usually, the teams that use these workarounds aren’t aware how Helm value hierarchies work.

Don’t fall into the trap of adding another layer of templating to your applications or appsets just to replicate Helm value hierarchies.

Using local Helm values with charts from Helm repositories

All the previous examples assume your Helm chart and its values are in Git. That’s always our recommendation, especially for your own applications. Some organizations, however, prefer to store charts in Helm repositories or OCI registries. Argo CD supports this scenario, but you need to understand the implications:

• Helm repositories (and OCI artifacts) are mutable. This means that you might download Helm chart 1.2.3 today with different contents than what you downloaded yesterday as version 1.2.3. And, of course, this breaks the GitOps principles.
• There’s no built-in auditing or history for changes made in the chart. If you use Git you get these features for “free”.
• There are risks regarding integrity and security. Can you verify that the chart you downloaded from the registry is the one the creator uploaded? Again, if you use your own Git repo, you can monitor and optionally approve all changes that happen on the chart.

If you insist on loading charts from external repositories, you should still keep your own values in your own Git repository. Argo CD can then merge the two using multiple sources.

Multiple sources is a feature of Argo CD created specifically for this scenario. In the most basic case, you should define 2 sources. One contains your external Helm chart and the other your Helm values.

Like before, you can also have Helm hierarchies.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-fancy-app
  namespace: argocd
spec:
  project: default
  destination:
    server: https://kubernetes.default.svc
    namespace: prod-us
  sources:
    - repoURL: https://acme.github.io/my-charts/
      chart: my-app
      targetRevision: 2.43
      helm:
        valueFiles:
        - $values/common-values.yaml      
        - $values/all-prod-envs.yaml                
        - $values/specific-prod-cluster.yaml
    - repoURL: 'https://github.com/example-org/all-my-values.git'
      targetRevision: HEAD
      ref: values

Notice here the relationship between the “ref:values” field and the “$values” prefix in the helm property. Of course, you can choose any name for the reference and not just “values”.

It should be clear that with this syntax, Argo CD has built-in support for Helm value hierarchies. Argo CD will load the chart from the external source and combine it with all the value files defined in order (last one wins).

On a related note, don’t abuse the multiple source feature of Argo CD. It’s NOT a generic way to group applications. Usually, you should have just 2 sources (as shown below) of related data that form a single application. If you find yourself grouping unrelated apps or having more than 2 to 3 sources, you need to look at app-of-apps and Application Sets instead.

Using Application Sets with Helm value hierarchies

If you deploy your Helm charts as an individual Argo CD application, you should do the same for your Application Sets. We covered Application Sets in detail in our previous guide.

Again, the key point here is that Application Sets also support Helm value hierarchies. And, like everything else in the application, you can template your chart values, folder, revision, or anything else you want.

Application Sets also work with multiple sources. So we can combine everything that we have seen so far in a single Application Set, like this:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-fancy-apps
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - list:
      elements:
      - env: qa
        type: non-prod           
      - env: prod-eu 
        type: prod      
      - env: prod-us
        type: prod      
  template:      
    metadata:
      name: '{{.env}}'
    spec:
      # The project the application belongs to.
      project: default


      sources:
        - repoURL: https://acme.github.io/my-charts/
          chart: my-chart
          targetRevision: 2.43
          helm:
            valueFiles:
            - $values/common-values.yaml          
            - $values/all-{{.type}}-envs.yaml              
            - $values/{{.env}}-values.yaml  
        - repoURL: 'https://github.com/example-org/all-my-values.git'
          targetRevision: HEAD
          ref: values
     
      # Destination cluster and namespace to deploy the application
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.env}}'

This Application Set defines 3 environments (qa, prod-eu, prod-us). It uses Helm hierarchies and an external chart.

Notice that we can template the location of all value files without any additional templating solution.

We see many teams who place their application CRDs (or applicationSet CRDs) inside Helm charts to template them on a second level. This complexity isn’t needed. Application Sets on their own cover most common scenarios. Application Sets are a way of templating (the same as Helm). Using Helm to generate Application Sets (or Applications) is a recipe for disaster.

If there’s something you can’t template with Application Sets, it’s better to open an enhancement request instead of bringing yet another layer of templating into the mix.

A complete example of GitOps environments with Helm values

Let’s see how everything works with a semi-realistic example. You can find all the Kubernetes manifests and Argo CD resources at https://github.com/kostis-codefresh/multi-sources-example.

The repository contains:

• A generic Helm chart both in Git and as part of a Helm repo served by GitHub pages at https://kostis-codefresh.github.io/multi-sources-example/
• Helm values for different environments of the chart
• Example applicationsets that reference the Helm chart
• Example apps that use Helm hierarchies

The environments are similar to those defined in the promotion guide, but this time we use Helm values instead of Kustomize overlays.

We have the following environments:

• Integration testing (non-GPU and GPU-enabled)
• QA (only one instance, located in the US)
• Staging in EU, Asia, US
• Production only in EU and US

Similar to Kustomize reusable components, we also have a structure of reusable Helm values that you can use in a hierarchical manner.

The expectation is that to form a full Helm release, you need to combine multiple value files in order.

For example, to deploy the application to the “Staging” environment in the EU region, you need to do the following with plain Helm:

helm install ./my-chart/ --generate-name --create-namespace -n staging-eu \
-f my-values/common-values.yaml \
-f my-values/app-version/staging-values.yaml \
-f my-values/env-type/non-prod-values.yaml \
-f my-values/regions/eu-values.yaml \
-f my-values/envs/staging-eu-values.yaml

Remember that the last setting wins. You can always override the common settings with different configurations with files deeper in the tree.
Of course, we don’t want to use Helm, but install the same hierarchy with Argo CD. Here’s the respective Argo CD application.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: staging-eu-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io    
spec:
  project: default
  destination:
    server: https://kubernetes.default.svc
    namespace: staging-eu
  sources:
    - repoURL: https://github.com/kostis-codefresh/multi-sources-example.git
      path: my-chart
      targetRevision: HEAD
      helm:
        valueFiles:
        - $values/my-values/common-values.yaml          
        - $values/my-values/app-version/staging-values.yaml              
        - $values/my-values/env-type/non-prod-values.yaml  
        - $values/my-values/regions/eu-values.yaml                
        - $values/my-values/envs/staging-eu-values.yaml
    - repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
      targetRevision: HEAD
      ref: values
  syncPolicy:
    syncOptions:
      - CreateNamespace=true  
    automated:
      prune: true
      selfHeal: true

Notice again the hierarchy of value files. The file also follows the best practices outlined in our Application Set guide.

If you deploy this application with Argo CD, you can easily verify that the settings are correctly defined according to all the values “merged” in the correct order (the last one wins in case of duplicate properties).

That is great. But as we have said many times before, you shouldn’t deal with individual applications. We can also use Helm hierarchies with Application Sets.

There are many ways to slice and dice your apps. The following examples just show the possible combination of Helm hierarchies.

Application Set on a single dimension

Here’s a very simple Application Set that includes a list generator to model the regions for a single application.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-staging-appset
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - list:
      elements:
      - region: us
      - region: eu
      - region: asia      
  template:      
    metadata:
      name: 'staging-{{.region}}'
    spec:
      # The project the application belongs to.
      project: default


      sources:
        - repoURL: https://github.com/kostis-codefresh/multi-sources-example.git
          path: my-chart
          targetRevision: HEAD
          helm:
            valueFiles:
            - $values/my-values/common-values.yaml          
            - $values/my-values/app-version/staging-values.yaml              
            - $values/my-values/env-type/non-prod-values.yaml  
            - $values/my-values/regions/{{.region}}-values.yaml                
            - $values/my-values/envs/staging-{{.region}}-values.yaml
        - repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
          targetRevision: HEAD
          ref: values
     
      # Destination cluster and namespace to deploy the application
      destination:
        server: https://kubernetes.default.svc
        namespace: 'staging-{{.region}}'


      # Sync policy
      syncPolicy:
        syncOptions:
          - CreateNamespace=true  
        automated:
          prune: true
          selfHeal: true

Here, the Helm value locations are templated according to the application’s region. An alternative is to use the list generator for a list of apps, clusters, or environment types.Deploying this application set creates 3 applications, one for each region.

Again, you can easily verify that each deployment gets the appropriate settings according to the respective Helm value hierarchy.

Application Set with multiple templated values

In the previous example, we only looped on the “region” property of the value folders. We can use the list generator to loop over several different properties.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: all-my-envs
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - list:
      elements:
      - env: integration-gpu
        region: us
        type: non-prod
        version: qa
      - env: integration-non-gpu
        region: us
        type: non-prod
        version: qa
[...snip…]    
      - env: prod-us
        region: us
        type: prod
        version: prod        
  template:      
    metadata:
      name: '{{.env}}'
    spec:
      # The project the application belongs to.
      project: default


      sources:
        - repoURL: https://github.com/kostis-codefresh/multi-sources-example.git
          path: my-chart
          targetRevision: HEAD
          helm:
            valueFiles:
            - $values/my-values/common-values.yaml          
            - $values/my-values/app-version/{{.version}}-values.yaml              
            - $values/my-values/env-type/{{.type}}-values.yaml  
            - $values/my-values/regions/{{.region}}-values.yaml                
            - $values/my-values/envs/{{.env}}-values.yaml
        - repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
          targetRevision: HEAD
          ref: values
     
      # Destination cluster and namespace to deploy the application
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.env}}'

In the example above, each deployment depends on different factors according to the application version, if it’s production or not, its region, and so on.

This application set actually creates ALL our applications in a single step.

Again, if you look at the resources of each individual application, you see each one has a different configuration according to all Helm values “merged” in the file tree.

Using a Helm repository with a hierarchy of Helm values

In all the previous examples, we used the Helm chart from Git, which is also our recommendation. However, we know some teams like to use OCI registries or Helm repositories. You can modify the previous Application Set and refer to a Helm repository instead of a Git folder for the Helm chart.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: all-my-envs-from-repo
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - list:
      elements:
      - env: integration-gpu
        region: us
        type: non-prod
        version: qa
[...snip…]
      - env: prod-us
        region: us
        type: prod
        version: prod        
  template:      
    metadata:
      name: '{{.env}}'
    spec:
      # The project the application belongs to.
      project: default


      sources:
        - repoURL: https://kostis-codefresh.github.io/multi-sources-example
          chart: my-chart
          targetRevision: 0.1.0
          helm:
            valueFiles:
            - $values/my-values/common-values.yaml          
            - $values/my-values/app-version/{{.version}}-values.yaml              
            - $values/my-values/env-type/{{.type}}-values.yaml  
            - $values/my-values/regions/{{.region}}-values.yaml                
            - $values/my-values/envs/{{.env}}-values.yaml
        - repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
          targetRevision: HEAD
          ref: values
     
      # Destination cluster and namespace to deploy the application
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.env}}'


      # Sync policy
      syncPolicy:
        syncOptions:
          - CreateNamespace=true  
        automated:
          prune: true
          selfHeal: true

You can see that we now specifically define the 0.1.0 chart from a Helm repo hosted in GitHub pages instead of loading the chart from a Git revision.

Other than that, the application set is exactly the same. The end result after deployment is also the same as the previous example.

Saving environment configurations on their own files

The last example has a very long list generator. As you add more environments, the list will become even longer. It’s far easier to extract this useful information in individual files. We can use the Git file generator to achieve this.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: all-my-envs-from-repo-with-version
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - git:
      repoURL: https://github.com/kostis-codefresh/multi-sources-example.git
      revision: HEAD
      files:
      - path: "appsets/4-final/env-config/**/config.json"      
  template:      
    metadata:
      name: '{{.env}}'
    spec:
      # The project the application belongs to.
      project: default


      sources:
        - repoURL: https://kostis-codefresh.github.io/multi-sources-example
          chart: my-chart
          targetRevision: '{{.chart}}'
          helm:
            valueFiles:
            - $values/my-values/common-values.yaml          
            - $values/my-values/app-version/{{.version}}-values.yaml              
            - $values/my-values/env-type/{{.type}}-values.yaml  
            - $values/my-values/regions/{{.region}}-values.yaml                
            - $values/my-values/envs/{{.env}}-values.yaml
        - repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
          targetRevision: HEAD
          ref: values
     
      # Destination cluster and namespace to deploy the application
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.env}}'

In this example, we extracted all environment configuration in config.json files. We also templated the chart version, so we can define differently for each environment.

This lets you edit each environment individually. For example, if we want chart version 0.1.0 in all production environments and 0.2.0 in all other environments.

Here is the configuration for prod-us:

{
    "env": "prod-us",
    "region": "us",
    "type": "prod",
    "version": "prod",
    "chart": "0.1.0"
  }

And the one for QA:

{
    "env": "qa",
    "region": "us",
    "type": "non-prod",
    "version": "qa",
    "chart": "0.2.0"
  }

Now you have a very flexible Argo CD setup that can cover most scenarios with minimal effort.

Scenario 1 — Bootstrap all apps in a single step -> Deploy the Application Set

Scenario 2 — Add a new setting to all European deployments -> change the file my-values/regions/eu-values.yaml

Scenario 3 — Making a common change to all environments apart from production -> change the file my-values/env-type/non-prod-values.yaml

Scenario 4 — Change chart version in Production US -> change the file env-config/prod/us/config.json

Scenario 5 — Make the QA environment inherit settings from Europe region instead of US -> change the file env-config/qa/config.json

Scenario 6 — Create a brand new environment called load testing with the same settings as Integration (GPU) -> copy the folder env-config/integration/gpu/ to env-config/load-testing and edit the config.json file accordingly.

Conclusion: Start simple

In this guide, you’ve seen how to combine different configuration settings with your Helm charts and how to employ Helm value hierarchies to model your environments without duplication. You also learned the proper way to use the multi-source feature of Argo CD.

We’ve shown that plain Application Sets can go a long way toward modeling all usual cases, even with Helm charts. You don’t need to template Application Sets with an extra Helm layer, and you certainly don’t need to resort to hacks for “default” and “extra” values.

We recommend using this structure as a starting point in your organization. Only add complexity if it’s really required by your business needs.

Using Helm Hierarchies in Multi-Source Argo CD Applications for Promoting to Different GitOps… was originally published in Container Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of Argo CD’s standout features is its powerful user interface (UI) that shows the live status of all applications and the respective Kubernetes resources. Both developers and operators can quickly understand the status of their deployments by looking at the UI and drilling down into all the different views of Argo CD applications. Some teams even use the Argo CD UI as a generic Kubernetes dashboard and management interface (especially after the addition of the web-based terminal feature).

As organizations scale their Argo CD use, they understand that using a single Argo CD instance across separate teams is only possible with proper security controls. Argo CD is designed to handle multi-tenant installations and has comprehensive controls for limiting and isolating applications between developer teams. This lets operators define who has access to what application and with what limits.

In this guide, we examine the individual building blocks (application projects, security policies, user groups, and roles) and explain how you can set up a multi-tenant Argo CD installation for your organization.

Popular use cases for multi-tenant Argo CD installations

Each organization has different requirements when it comes to risk and ease of management. We’ve seen many ways that teams prefer to use (or not use) Argo CD in a multi-tenant/multi-use environment.

Firstly, we have the use case where developers have full access to the Argo CD UI and can debug/sync/edit their deployments with full power. This is great if your developers already have some Kubernetes knowledge and know the principles of GitOps.

After your team reaches a certain size, you need some restrictions in place, especially with larger developer teams. The classic scenario is where developers have full access to non-production environments, but read-only access to production environments. They can look at the status of production and possibly retrieve logs, but can’t change anything on the deployment.

The restrictions on developers can be as strict as you want. Argo CD supports isolating developers to specific namespaces and clusters, hiding completely unrelated applications from the UI, and even setting up different access for different application actions. For example, a developer might have access to take logs from an Argo CD application, but they’re restricted from triggering a re-sync of the application.

At the other end of the spectrum, some organizations choose to hide Argo CD completely from developers. Argo CD is reserved for operators only. Developers interact with their applications using an external system that often sits on top of Argo CD and abstracts several Kubernetes concepts (like an internal developer platform or portal).

In all scenarios, it’s important to decide whether you let developers to create new applications on their own or only edit existing ones. You also need to consider how to handle ephemeral environments.

Three important things to remember are:

1. Argo CD comes with its own role-based access control (RBAC) mechanisms that aren’t related to Kubernetes RBAC.
2. There are several moving parts that coordinate together (single sign-on (SSO), roles, app-projects).
3. This guide is about your internal users. If you want to achieve multi-tenancy for external users and customers, you probably need to look at other approaches.

There isn’t a one-size-fits-all solution, of course. It’s also impossible to cover all variations in a single guide. There are several designs that you will see in real life.

Consider this guide a gentle introduction to the security controls of Argo CD. We explain the theory behind the security controls and share a semi-realistic example to give you ideas on how to implement multi-tenancy in your own organization.

Argo CD security overview

There are two different aspects when it comes to multi-tenant Argo CD installations:

1. Limits over where applications get deployed. Which Kubernetes resources can people deploy and from which Git repos.
2. Security over Argo CD resources. Who can see an application, sync it, delete it etc.

These two aspects are independent and are linked together via the application project resource.

The other pieces of the puzzle are:

1. Users, groups, and SSO configuration (single sign-on)
2. Local users (optional)
3. Global security rules and per-project security policies
4. Grouping of policies into roles
5. Argo CD resource actions
6. Other associated application resources such as clusters, namespace, and Git repos

We briefly cover all these components in the next sections.

How to define access for Argo CD resources

Let’s start with Argo CD resources (i.e., applications). The RBAC model behind Argo CD resources follows the who-what-how model. Resources get secured via RBAC policies. Essentially, you can define who has access to what application and how they can access it.

Even though Argo CD can define security controls with individual objects (“allow David to edit application billing”), in reality, it’s most useful to treat similar objects in groups. Therefore, all individual parts of a policy are mostly applied between groups instead of individual entities.

After you have all groups/roles/apps defined, you can apply policies on a global or project level.
You define policies in the casbin format and follow the syntax:

p, <role/user/group>, <resource>, <action>, <object>, <effect>

The first p stands for “policy”.

Here are some examples of policies:

“p, infra:read-only, applications, get, infra/*, allow”

This policy says, “All users that belong to the infra/read-only group are allowed to view/get all applications that belong to the infra app-project”.

Another example:

“p, developers:full-access, applications, delete, prod/*, deny”

This policy says, “All users that belong to the developers:full-access group are NOT allowed to delete applications that are part of the prod app-project”.

You can find all the details for what actions and resources are available in the official Argo CD documentation.

The important point here is that you need to do some analysis and split all your Argo CD applications into app projects. Different organizations will select different meanings for the projects. A project can be a specific team of people, a department, “environment”, or anything else that makes sense in your organization for application groups. We recommend using projects for each team (both developer teams and infra teams).

After you’ve defined all your policies, you should store them declaratively in your application projects under .spec.roles.

The second important point is that contrary to other security designs, a “role” in Argo CD is a set of policy resources. You first create an app project, then apply policies to it, and finally connect policies to users.

This means you need to anticipate how your applications will get used and design appropriate roles. For example, if you follow the scenario where a group of developers can see all applications in Argo CD but in “view-only” mode, then you need to create a “view-only” security policy in all the respective application projects.

How to define access for Kubernetes resources

RBAC policies define how users can interact with the main Argo CD resources (applications). The application project has additional capabilities for securing Kubernetes resources.

Remember that an Argo CD application is, in its essence, just a link between a Git repository and a Kubernetes cluster/namespace. In a multi-tenant environment, you almost always want to restrict:

1. What clusters/namespaces developers can deploy to
2. Which Git repos/paths can be used for applications
3. What kind of Kubernetes resources can be affected
4. Optionally, when deployments can happen

The default project is an “allow-everything” project that has no restrictions at all.

spec:
  sourceRepos:
  - '*'
  destinations:
  - namespace: '*'
    server: '*'
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'

Using the default project is great for experimenting with Argo CD, but we recommend you don’t use it in a production setup and only have your own projects defined.

Application projects are very flexible when it comes to cluster resources. You can isolate developers to specific namespaces, specific types of resources, and days of deployments using sync windows.

Here is an example application project:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-revenue
  namespace: argocd

spec:
  description: Team with developers for Revenue

  sourceRepos:
  - "https://github.com/acme/revenue/*"

  destinations:
  - namespace: "revenue"
    server: https://kubernetes.default.svc


  clusterResourceBlacklist:
  - group: ""
    kind: "Namespace"


  namespaceResourceBlacklist:
  - group: ""
    kind: "ResourceQuota"
  - group: "networking.k8s.io"
    kind: "NetworkPolicy"

This project defines what the “Revenue” team can do.

1. They can only deploy from Git repos under acme/revenue.
2. They can only use the namespace “revenue” in the local cluster.
3. They cannot create new namespaces, resource quotas, and network policies.

While the argocd CLI includes several commands for managing projects, again we recommend you declare everything in the Project CRD and use a declarative setup for a production environment.

A complete example with two developer teams

We’ve seen how the application project is the centerpiece of Argo CD security in multi-tenant environments.

Let’s take a look at a full example. In our imaginary company, we have 2 developer teams (A and B) that continuously deploy applications. We also have infrastructure applications (i.e., cert-manager) handled by the operator team.

Here are our requirements:

• Admin/operator can view/edit all applications (including infra apps).
• Team B can only view/edit their own applications and nothing else.
• Team A can view/edit their own applications and view-only applications from team B.
• No developer from either team can view the infrastructure applications.
• No developer can delete any application (even their own).
• All applications (including infra) can only be synced from a specific GitHub organization/user.

You can find all Argo CD settings at https://github.com/kostis-codefresh/intro-argocd-rbac . Note that for simplicity, we used local users instead of SSO.

We have 4 applications in total — one for each team and 2 infrastructure apps. The admin of Argo CD can see all of them and can perform any action.

Developers from team B are the most constrained. They can only look at their application and nothing else.

They can sync at will, but they cannot delete the application. Pressing the delete button presents an error message.

Developers from team A can see both apps from both teams, but not the infrastructure apps.

Even though they can sync/edit their own application, they cannot affect the application of team B. If they try to perform a sync on the application of team B, again they will get an error message.

With this set up, we’ve satisfied our initial requirements. Let’s look under the hood to see how everything was implemented.

Using app-projects per team

We mentioned in the introduction using application projects to represent teams. In our example, we defined 3 projects, one for each team (A, B, and infra).

The most interesting project is the one from team B:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-b
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io  
spec:
  description: Team B
  destinations:
  - name: '*'
    namespace: '*'
    server: '*'  
  roles:
  - description: Read Only
    name: read-only
    policies:
    - p, proj:team-b:read-only, applications, get, team-b/*, allow
  - description: Read Write
    name: read-write
    policies:
    - p, proj:team-b:read-write, applications, get, team-b/*, allow
    - p, proj:team-b:read-write, applications, sync, team-b/*, allow
    - p, proj:team-b:read-write, applications, update, team-b/*, allow        
  sourceRepos:
  - https://github.com/kostis-codefresh/intro-argocd-rbac.git

Notice that we define 2 roles here (sets of policies). One is the “read-write’ for team B, and one is “read-only” for team A. Notice also that neither role includes a “delete” action, This means that no developer can delete the applications of this project. Finally, we restrict the application source repository to a specific GitHub project.

After we have AppProjects in place, we deploy all 4 applications using an application set. If you’re not familiar with application sets, please read our application sets guide first.

The applicationset will automatically assign each application to the same AppProject as its parent folder.

This concludes the initial application deployment.

Assigning users to groups/roles

For simplicity, we use local users in our example. In a real system, you’d use SSO to assign users to groups for Argo CD.

In our case, we add 2 users (one for team A and one for team B) in argocd-cm.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  accounts.jane: apiKey, login
  accounts.jane.enabled: "true"
  accounts.david: apiKey, login
  accounts.david.enabled: "true"  

You can set the password for local users with the argocd account update-password command or via declarative setup.

Finally, we assign each user to their appropriate roles in

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-rbac-cm
    app.kubernetes.io/part-of: argocd
data:
  policy.csv: |
    g, jane, proj:team-a:read-write
    g, jane, proj:team-b:read-only
    g, david, proj:team-b:read-write  

You can see here that Jane belongs to team A. She has read/write access to all apps of team A and read-only access to projects from team B. David is on team B and he can only see his own projects.

Neither Jane nor David see any of the applications from the “infra” projects (which was one of the original requirements).

This concludes the setup. If you now log in as Admin, Jane, or David in the Argo CD user interface, you’ll see only what each user is allowed to view/edit.

Where to go from here

This guide was a gentle introduction to Argo CD RBAC and how to use Argo CD in multi-user situations. There are several more things to explore, like appProject hierarchies, specific SSO configurations, complex policies, and more. For now, you should have some great ideas for your own organization and understand how all the different components work together to achieve multi-tenancy.

Security is a complex topic. Consider the present guide a starting point for your organization. You should always follow the best practices and requirements of your security team.

Securing Argo CD in a Multi-Tenant Environment with Application Projects was originally published in Container Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Argo CD is also fully open source, which means teams can freely install it on their private clouds, behind-the-firewall data centers, or even in air-gapped environments without any licensing restrictions. Argo CD’s popularity is surging right now, especially since Weaveworks (the company behind Flux — the main Argo CD competitor) ceased operations in early 2024.

It’s important to understand, however, that while Argo CD is great as a GitOps sync engine, it’s missing several features that big enterprises need when deploying at scale. If you have one or two Kubernetes clusters and a small number of applications, it’s easy to start using Argo CD. For larger installations with multiple clusters and multiple applications, using plain Argo CD takes too much manual effort. In those cases, looking at an enterprise solution like the Codefresh platform will be more beneficial as your organization scales its Kubernetes use.

In this guide, we look at the 4 stages of the Argo maturity curve. We explain how Codefresh can help you at each stage beyond what Argo CD offers out-of-the-box.

The central theme here is that enterprises have several other requirements not offered by the sync engine of Argo CD, while Codefresh is built from the ground up as an enterprise platform.

Stage 1 — Initial Argo CD installation versus using a control plane

The OSS version of Argo CD allows for 2 installation modes out of the box:

1. Installing an Argo CD instance per Kubernetes cluster.
2. Installing a central Argo CD instance that manages several other clusters.

Each method has advantages and disadvantages. We cover them in detail in our Argo CD architecture article. The takeaway is that no option is perfect, each has shortcomings.

The individual Argo CD instances are very difficult to manage on a large scale.

The biggest problem is daily cluster management. You now need to manage multiple Argo CD instances and their respective dashboards. If you have for example 6 Kubernetes clusters, you’ll also have 6 different dashboards. When searching for an application you need to remember to use the appropriate Argo CD dashboard. Upgrading all 6 Argo CD instances to the same version is also a non-trivial job.

On the other hand, you can have the central Argo CD instance that manages other clusters.

This also comes with significant challenges.

1. The central Argo CD instance is a single point of failure.
2. Communication to other clusters is outgoing. This means you need to open holes in your firewall if the other clusters aren’t in the same networking domain as the central instance.
3. The centralized instance will quickly become a bottleneck when you have a large number of applications and clusters.

Teams that choose either of these installation options will have to deal with the disadvantages on a day-to-day basis and either accept them or try to find workarounds.
With Codefresh there is a third solution — using a Control Plane.

The Codefresh Control Plane unifies multiple Argo CD instances under a single management dashboard.

The Control Plane approach offers the best of both worlds without the disadvantages

1. Communication with the Control Plane is inbound. This means you can manage Argo CD instances behind the firewall and there isn’t any need for open ports
2. There isn’t a single point of failure. Even if the Control Plane has issues, the individual Argo CD instances will continue syncing on their own.
3. There are no performance bottlenecks as you can easily mix and match multiple Argo CD instances and clusters.
4. There’s still a single application dashboard for Argo CD instances.

The last point is especially important for end users of Argo CD and, more specifically, developers. They can now view their applications easily without juggling between different dashboards.

Even at this early stage (installation of Argo CD), choosing Codefresh brings extra value to both Argo CD administrators and users.

Stage 2 — Managing multiple applications and tracing back code changes

With the initial installation out of the way, let’s examine what happens with day-2 operations. Creating Argo CD applications is a straightforward process. Out of the box, Argo CD supports your Helm charts, Kustomize overlays, and any other custom manifests you already have.

The main problem here is that Argo CD has very limited information about the applications that your developers create. The only thing that Argo CD knows about an application is:

1. The Git repository that has the Kubernetes resources
2. The container image that gets deployed
3. The target cluster where the application gets deployed
4. The Git hash of the manifests that were applied

And that’s all! Argo CD has no visibility into the things that matter for developers. Developers want to know what source code changes went into a release, plus information like unit test results, security vulnerabilities, and pull request information. Argo CD doesn’t know what happened before a container image was built.

This means that developers have to switch between multiple systems like GitHub, Jira, the CI server, and Argo CD to understand exactly what Argo CD deployed.

Several teams adopt Argo CD hoping that developers will use the Argo CD UI as a one-stop solution for all their deployment needs. Unfortunately, it quickly becomes apparent that developers can’t work with just the Argo CD UI. At this point organizations start looking at developer portals and internal developer platforms to address the daily needs of the development team.

With Codefresh you get full traceability from source code committed to deployment.

Codefresh aggregates information from several other systems, like the CI server or the ticket server (i.e. JIRA), and provides a complete timeline of what happened with an application deployment including:

• Who initiated a change
• What was the Git commit
• What features (i.e. Jira tickets) were included
• Which pull request took part in the change
• Which pipeline was used for artifact creation
• Results of security scans, unit tests, lint checks, etc
• Which container images were produced and pushed
• Which Kubernetes applications were affected
• Which individual components were updated

The timeline also gives you the overall health status of a service/application across the complete history of deployments.

Using Codefresh, developers can answer in seconds the burning questions they face every day. Some examples are:

1. What JIRA features are now deployed in Staging?
2. Who deployed release 0.3.0 last Thursday and what source code features did it contain?
3. Where is JIRA feature 4524 deployed right now? Is it in Production just yet?
4. What is the CI pipeline that created container image billing:v0.4.0 that’s been sitting in the QA environment for the last 5 weeks?
5. Do we have any critical vulnerabilities known in the artifact deployed in the Load Testing environment?
6. How many developers have just pushed their feature to QA?
7. Who approved pull request 2345 that just went into production?

If you have only Argo CD, answering these kinds of questions can take significantly longer (even hours in some cases). With Codefresh, developers can instead reason about any deployments and quickly discover the full picture for any active application.

Stage 3 — Promoting apps between environments

After you have the initial deployment for every application in place, the next step for developers is promoting applications from one environment to another.

Unfortunately, the open-source version of Argo CD can’t help with promotions. Argo CD has no concept of an “environment” and doesn’t have any information about promotions. This forces people who’ve adopted Argo CD to try implementing their own promotion mechanisms with several workarounds.

The classic pattern here is the “special” naming of applications and the hardcoding of an “environment” in the application name:

Here these 3 Argo CD applications have been named with a special pattern to try accommodating “environments”. Even though it’s clear to a human user that this is the same application deployed in 3 environments, for Argo CD it’s 3 different applications with no connection between them whatsoever.

Codefresh has full support for both grouping multiple applications (in products) as well as modeling environments the proper way:

A Codefresh environment can be a different namespace, a different cluster, or different namespaces in a different cluster. Basically, you can mark any combination of clusters/namespaces as an environment. Then Codefresh can correlate applications and their environments in the backend and reason about them. In the example above, even though the underlying applications are still named “accounts-qa, account-staging, accounts-production”, Codefresh knows that it’s the same application — accounts — deployed to 3 different environments.

In addition to environment support, Codefresh can aggregate information from the CI phase into environments (as explained in the previous section). This means that, unlike Argo CD, which only knows a container image version and nothing more, Codefresh can match other metadata into an environment, like source code commits, pull requests, security scans, and more.

Here’s another view of the environment dashboard which now has JIRA information as well:

Using this dashboard, developers can reason in seconds about their GitOps environments and answer questions about features which isn’t possible with just Argo CD.

The highlight feature, of course, is the promotion between environments. Since Codefresh has all the context about which application is deployed into which environment, promotions are a core, built-in construct in the Codefresh GitOps platform.

In the most simple scenario, you can simply let developers promote applications by drag-and-drop. Codefresh automatically creates a commit with all the required information to perform the promotion.

In a more complex scenario, you can define a complete promotion workflow with specific restrictions and pre/post checks that again isn’t possible with plain Argo CD.

In the screenshot above, we defined a basic promotion policy between 4 environments. New applications can only be deployed in “dev”. After they pass the checks in “dev”, they can move in parallel to “staging” and “qa”. Only if all checks pass in both “staging” and “qa” can the application go to production.

The beauty of the promotion policy is that you can define any pre/post checks and enforce them. Your checks can be security scans, smoke tests, database checks, etc.

If a check fails, promotion gets blocked:

In this example, checks were successful in “staging”, but something went wrong in “qa”. This means developers can’t push to production until “qa” gets fixed. You can use the friendly Codefresh UI to understand what went wrong and drill down in the logs.

Essentially, when it comes to environments and promotions, plain Argo CD can’t help you at all.

The functionality of Codefresh is essential as developers always think in terms of environments and promotions.

Stage 4 — Enterprise features

With the basic functionality out of the way, we can finally look at all the features that enterprises need as they scale.

Often, organizations start with just Argo CD and soon realize they need other features like built-in pipelines and progressive delivery (blue/green/canary). At this point, they choose to manually install Argo Workflows and Argo Rollouts. Then they spend extra time integrating those products.

Even though the Argo projects are made to work together, integrating them in a single cluster is a non-trivial process. For example, Argo Workflows have a different SSO mechanism than Argo CD, while Argo Rollouts has no SSO support at all.

Codefresh comes with this integration out-of-the-box. The Codefresh runtime is a unified installation package that contains all 4 Argo projects out-of-the-box and with security patches applied.

The unified integration dashboard we mentioned in the first section also applies to the other Argo projects. For example, by adopting the Codefresh runtime, you get free SSO support for all 4 projects with no extra effort.

Here’s an example of using progressive delivery in an application from the same Codefresh interface.

At the same time, Codefresh offers several enterprise features that would require significant effort to replicate with just Argo CD. Because Codefresh has full visibility of what happened in both the CI and CD phases, DORA metrics are always available from the Codefresh UI with zero configuration.

Finally, you can find several statistics and insights that reveal bottlenecks in your deployment process. Again, you’d need to manually create them if you only use Argo CD

This was just an overview of the extra enterprise features offered by Codefresh. There are several other features, like annotating container images with extra metadata:

Correlating images with deployments also happens out-of-the-box with Codefresh:

Often, developers need to know where a specific image is deployed. Answering this question with Codefresh literally takes seconds. Answering with just Argo CD can take hours depending on the complexity of the application.

Conclusion

We’ve seen the 4 stages for GitOps maturity and explained at every stage what you get with Argo CD and what extra features Codefresh offers.

It should now be clear that Argo CD is great as a sync engine but misses several features that enterprises need for effective deployments. With Codefresh you get a full enterprise platform that has all the missing functionality, while still using Argo CD under the hood.

The Codefresh platform is actually a superset of all 4 Argo projects and contains extra features in regards to:

1. Initial installation
2. Application deployment and correlation with the CI process
3. Full traceability for a source code change from the initial commit until it reaches production
4. Environment promotion and creation of developer guardrails
5. Enterprise security and support
6. Deployment statistics, DORA metrics, and application insights

Get started with the Codefresh GitOps platform, sign up now or schedule a demo.

The 4 Stages of the Argo Maturity Curve was originally published in Container Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Teams adopting GitOps often ask how to use secrets with Argo CD. The official Argo CD page about secrets is unopinionated by design and simply lists a set of projects that can help you with secrets.

We’ve seen several approaches to secret management. These include sealed secrets, the Argo CD Vault plugin, and the External Secret Operator.

In this post, we showcase the External Secret Operator and Hashicorp Vault and focus on 2 important aspects.

• How to avoid saving ANY secrets in Git, including tokens for fetching the application secrets
• How to refresh secrets automatically without pod restarts and application deployments

Several existing tutorials show how you don’t need to store any application secrets in Git, but never actually explain where to store the token for fetching the secrets from Vault or other secret providers. So you still have the same problem but for the token itself instead of the application secrets — where to store it?

Refreshing secrets automatically is also fundamental, as passing secrets to an application is only half the battle. You also need an easy way to rotate and revoke compromised secret information.

How to pass secrets with the External Secret Operator

The External Secret Operator (ESO) is a Kubernetes controller that supports retrieving secrets from several providers (AWS, Azure, GCP, Vault, Gitlab, etc) and converting them to plain Kubernetes secrets so any Kubernetes application can consume them.

The Controller is designed for Kubernetes installations and although it works great with Argo CD, it doesn’t need Argo CD or even follow GitOps principles. It introduces a set of custom resources (CRDs) for representing a secret source, SecretStore, and a secret from that store, ExternalSecret.

One of the best characteristics of ESO is that all secrets it controls get converted into plain Kubernetes secrets in the end (regardless of their actual source). This makes it a very flexible choice because the application code can be as minimal as possible.

In practice, your application code:

• Doesn’t know anything about the ESO controller
• Doesn’t need a special API to access secrets
• Doesn’t even need to know it’s running inside Kubernetes
• Can read secret values from files (recommended) or environment variables

This is one of the controller’s highlight features, especially in legacy applications where you can’t change the source code at all. ESO is also a great choice when you want to move secrets from one provider to another, as the application is completely oblivious to where the secrets come from.

The example secret application

Our demo application is at https://github.com/kostis-codefresh/external-secrets-gitops-example/tree/main/src.

It’s a very simple application that reads secrets from files mounted at /secrets.

We recommend reading secrets from files and not environment variables, as you can monitor files for changes. When you need to rotate or revoke secrets, it would be great to make your application consume the new secrets without any restarts or redeployments. Of course, you can also use Kubernetes secrets as environment variables.

The application needs 3 secrets:

1. Database URL
2. Database username
3. Database password

Just for demo purposes, there’s no real database. The application simply prints the secrets so you can verify visually if the values in Hashicorp Vault have passed successfully to the pod.

Notice the application clearly says where these secrets get loaded from (/secrets directory). This is a great practice that lets you know the source of your secrets, ensuring you don’t miss any during rotation. You can expose this using an API instead, or even have an organization-wide convention that specifies the source of secrets. We recommend you do this regardless of your adoption of Argo CD or GitOps.

When trying to understand how an application loads configuration and secrets, having the application tell you itself saves time spent debugging and looking at logs. This is especially helpful during an incident where time is of the essence.

Installing Hashicorp Vault and ESO in your Kubernetes cluster

Let’s start first by deploying Hashicorp Vault and ESO in your cluster. You can do it with Argo CD and simply use the public Helm charts.

In a real company, Hashicorp Vault should be already installed somewhere else by your administrator. You’d need to ask your security team for access.
For this demo, we installed Hashicorp Vault with the server.dev.enabled option as true, so that Vault runs in development mode and doesn’t need sealing/unsealing. This is only for demo purposes. The default root token is also set as “root” and you can use it to log in to the UI as well.

Fetching external secrets from Hashicorp Vault

Before we pass secrets to our application we need to explain to ESO where to get them. In our example, we use the built-in integration for Hashicorp Vault. For the integration to work, we need to create a SecretStore or ClusterSecretStore resource that defines how to authenticate against Vault to get secrets.

This is the crucial point of the whole process. Several existing tutorials about external secrets either don’t explain the integration process, or use a predefined token for accessing Vault (or any other secret provider).

If you use an authentication token, you’re introducing the chicken-and-egg question about how to store the token itself. Using external secrets avoids storing the secrets in Git. But if we need a token for accessing secrets, where should we store that token? If we choose Git, we’re back to square one. If we hard-code in a manifest and push it on the cluster, we’ve introduced a single point of failure for reliability and security reasons.

The answer to this question is provider-specific, but essentially you need to use an authentication method based on trust and not static tokens. Several ESO providers support authentication methods with this characteristic. You need to contact your security team and discuss your options with them. Make sure to check the support of the Open ID Connect protocol (OIDC) for your secret provider.

This guide isn’t about Hashicorp Vault, but since we’re using it as an example, we can easily follow this best practice and use a trust method instead of a predefined token.

For our demo, we can instruct Hashicorp Vault to use Kubernetes authentication by trusting the same Kubernetes cluster that ESO is running on. This way we don’t need to store any token at all. ESO will use a Kubernetes service account from the pod it’s running on and pass that to Vault. Vault has many more authentication options, like trusting outside Kubernetes clusters or even other Vault instances, but explaining them is outside this guide’s scope.

The result is that we didn’t hardcode any secrets or tokens anywhere, in the cluster or in Git.
To enable the Kubernetes authentication read the Vault documentation. You can use the UI or the CLI to enable it. In our example cluster, you can expose the Vault UI with:

kubectl port-forward -n vault vault-0 8200:8200

The default token is “root” because we used the “dev” installation mode.

A nice part of the default installation of Vault in Kubernetes, is that you get the Vault CLI preinstalled as well in the same pod. So enabling Kubernetes auth for a quick demo is as simple as doing the following:

kubectl exec --stdin=true --tty=true -n vault vault-0 -- /bin/sh
vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host=https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT

vault write auth/kubernetes/role/demo \
  bound_service_account_names=* \
  bound_service_account_namespaces=* \
  policies=default \
  ttl=1h

You also need to change the default policy to allow reading of secrets by adding the following snippet to the default policy:

 path "secret/*" {
  capabilities = [ "read", "list" ]
}

The rest of the policy should stay as-is.

Your dedicated security team should handle these settings for you in Vault or your preferred secret providers. The settings above are just for demo purposes to show you how external secrets work.
The last step is creating a ClusterSecretStore that tells ESO that Vault will be used for secret storage.

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://vault.vault:8200"
      path: "secret"
      # Version is the Vault KV secret engine version.
      # This can be either "v1" or "v2", defaults to "v2"
      version: "v2"
      auth:
        # points to a secret that contains a vault token
        # https://www.vaultproject.io/docs/auth/token
        kubernetes:
          mountPath: "kubernetes"
          role: "demo"

Notice there are no credentials in this file, so it can be safely stored in Git as-is.

Passing secrets from Vault to your application

With the setup done, we’re now ready to use secrets in our application. First, create a set of secrets in Vault. You can do this with the CLI, the UI, Terraform, etc.

Next, create an External Secret YAML.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-db-credentials
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: mysql-credentials
    template:
      engineVersion: v2
      data:
        credentials: |
          db_con="{{ .db_url }}"
          db_user="{{ .db_username }}"
          db_password="{{ .db_password }}"
       
  data:
  - secretKey: db_url
    remoteRef:
      key: mysql_credentials
      property: url
  - secretKey: db_username
    remoteRef:
      key: mysql_credentials
      property: username
  - secretKey: db_password
    remoteRef:
      key: mysql_credentials
      property: password

In this file, we’re using secret templating to match the file format that the application expects (a simple property file). Again, notice this file doesn’t have any confidential information. It’s simply a pointer to Hashicorp Vault. This means you can safely store it in Git (and apply it with Argo CD).
Finally, deploy the application with Argo CD. Here’s the deployment file.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitops-secrets-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gitops-secrets-app
  template:
    metadata:
      labels:
        app: gitops-secrets-app
    spec:
      containers:
      - name: gitops-secrets-app
        image: docker.io/kostiscodefresh/simple-secret-app:latest
        imagePullPolicy: Always  
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: mysql
          mountPath: "/secrets"
          readOnly: true      
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
      volumes:
      - name: mysql
        secret:
          secretName: mysql-credentials

Notice that it mounts the mysql-credentials secret as a normal file in the container filesystem at the /secrets folder.

Now launch the application and you can see it has the correct secret information. You can do that with:

kubectl port-forward svc/gitops-secrets-service 8080:8080

Success! Our secrets from Vault are accessible to our application without saving anything in Git.

If we look at the Resource Overview in Argo CD we can see that ESO automatically generated a standard secret for us with Vault information. That secret was mounted on the container.

Here’s what happened behind the scenes:

1. We applied our application manifests with Argo CD.
2. The service and deployment are standard Kubernetes resources and were applied directly by Argo CD.
3. The external secret is a CRD. It was passed to the external secret controller.
4. ESO sees that this is an external secret that mentions our Vault secret store.
5. ESO contacts Vault and asks for secret details.
6. Vault trusts ESO because Kubernetes authentication is active and they are running on the same Kubernetes cluster.
7. Vault gives the secret information to ESO.
8. ESO creates a standard Kubernetes secret using the template defined in the ExternalSecret CRD.
9. The job of ESO is over.
10. A standard Kubernetes deployment mounts the Kubernetes secret at /secrets/credentials in the filesystem of the container.
11. The application starts up. It reads /secrets/credentials like any other file without any knowledge of how this file was created.
12. The secrets appear on the web page.

Refreshing secrets without application restarts

In the introduction, we mentioned that using secrets is just one part of the equation. We also need a way to easily rotate and revoke secrets.

One of the highlight features of ESO is that it can automatically refresh our secrets when they change in the original secret storage.

Our application takes advantage of this automatically.

1. It loads the secret from a file and not an environment variable. Environment variables are fixed once a process starts and the only way to refresh them is to restart the app.
2. The source code automatically monitors the /secrets/credentials file for changes and auto-reloads it if it changes.

This means that we can change our secrets on the fly and see the application update right away. Let’s say that our database password became compromised and we had to update it. We’d make the change in Vault:

In the next refresh period (defined in the External Secret CRD with the refreshInterval property), ESO will pass the new change to the application.

The application will update the secret on its own. How nice is that? 🙂

We achieved one of the holy grails of good security practices. We can rotate and revoke secrets without restarting the application and without recreating any pods.

Conclusion

In this guide, you saw an alternative method of using secrets with GitOps, where an external source keeps the secret values. In Git, we only stored their configuration and how they’re mounted on their application. If your organization already has an external source for secrets, check if the External Secret Operator supports it.

You also saw how to prepare an application to automatically reload secrets when they change, making secret rotation simple.

We hope that now you have several choices for managing secrets with GitOps, you can choose your own level of risk or ease of use based on how closely you want to follow GitOps principles.

Happy deployments!

GitOps Secrets with Argo CD, Hashicorp Vault and the External Secret Operator was originally published in Container Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

In our big guide for Kubernetes deployments, we explained the benefits of using dynamic environments for testing.

The general idea is that each developer gets a preview environment instead of having a fixed number of testing/QA environments. The environment gets created on the fly when you open a pull request. Typically, it gets destroyed when you merge the pull request (or after a specific amount of time).

Previously, we explained how to implement this pattern using just Helm applications and an Ingress on a Kubernetes cluster with a traditional deployment pipeline.

If your organization has moved to GitOps, you might wonder if you can replicate this setup with Argo CD (or the Codefresh runtime). The answer is yes!

In this guide, you learn how to create preview environments with Argo CD by using the Pull Request Generator designed for this scenario.

Keeping developers happy

There are several approaches for creating preview environments with Argo CD. One major decision is whether you want a preview environment created for a pull request in the application source code, or a pull request on the Kubernetes manifests. As a reminder, it’s always a good idea to separate source code from manifests in two separate Git repositories.

Both approaches are valid, but in this guide, we focus on pull requests created on the source code. Developers open these pull requests when they’ve implemented a feature and want to test or share it with their team.

We don’t want to force developers to learn how Argo CD or Kubernetes works. So, as far as they’re concerned, they open a pull request as usual, and after some time, a new preview environment gets created with the contents of the pull request.

There are many ways to create preview environments for Kubernetes. For simplicity, we’ll follow the same assumptions as in the Helm article.

1. We’ll use a single cluster for all preview environments.
2. Argo CD will run inside the same cluster.
3. We’ll deploy each pull request to its own namespace.
4. The name of the namespace will match the name of the pull request.

If two developers create two pull requests named “my-billing-feature” and “fix-queue-size”, two new namespaces with the same name get created with the respective deployments. Each namespace contains the container from the respective Continuous Integration build. So both developers can test their own features in isolation.
In a future article, we’ll cover more advanced scenarios, like creating a virtual cluster per pull request.

Generating Argo CD applications automatically

The basic concept of an Argo CD installation is the application, which is a link between a Git repository and a destination cluster. We explained how to create Argo CD applications using generators in our article about Argo CD application sets.

Things are a bit different in preview environments, as we don’t really have a folder in Git that represents the application. If we did, we could use the Git generator and create preview environments as more folders get created/destroyed.

In theory, we could instruct our Continuous Integration (CI) system to commit new folders on the manifest repo for each preview application. But this process is too cumbersome and there’s no need to go down that route. The Pull Request (PR) Generator solves this problem by creating applications directly from pull requests instead of folders.

Unlike other Argo CD generators, the PR Generator is special because it can monitor any Git repository for pull requests. It doesn’t have to be the same one that has manifests. This lets you monitor the Git repository for source code while still deploying manifests from a different Git repository.

Here’s our updated architecture where we’ve expanded the “magic” section from the previous picture.

The whole process goes like this:

1. There’s one Git repository with just the source code of the application. There’s another Git repository with Kubernetes manifests (Helm, Kustomize, or plain manifests).
2. A developer creates a branch on the source Git repository and starts working on a feature.
3. When the developer finishes, they create a pull request on the branch (usually pending tests and feedback from their team).
4. Argo CD notices the pull request on the source code Git repository and creates a new application for it.
5. The Pull Request Generator takes the manifests from the infrastructure repository and templates/modifies them according to the pull request in the source code repo.
6. A new application gets created and deployed in the Kubernetes cluster.
7. The developer accesses the new application, reviews it, runs unit tests, shares it, etc.
8. When the pull request gets merged/closed, Argo CD automatically discards the preview environment.
9. The process works in parallel for all developers working on features.

This is the heart of the process. There are also several supporting factors we must consider to create a rock-solid experience for developers.

Coordinating with the Continuous Integration system

The main goal of the preview environment is to let developers test their new features in isolation. Before dealing with Argo CD deployments, we need a Continuous Integration (CI) system that covers two basic requirements:

1. It automatically builds a container every time you open a pull request.
2. It pushes the container using the Git hash of the source code as a tag.

We use Codefresh CI in our example application, but any CI system should be able to cover these requirements.

You can find the source code of the example application at https://github.com/kostis-codefresh/preview-env-source-code. All the manifests are at https://github.com/kostis-codefresh/preview-env-manifests. Note that we follow the Argo CD best practices for splitting source code from Kubernetes manifests.

This example uses Kustomize. For an example with Helm, see Piot’s guide on the same subject.

Here’s a minimal Codefresh pipeline that builds a container image with the same tag as the source code.

version: "1.0"
stages:
  - "clone"
  - "build"
steps:
  clone:
    title: "Cloning repository"
    type: "git-clone"
    repo: "kostis-codefresh/preview-env-source-code"
    revision: "${{CF_REVISION}}"
    git: "github-1"
    stage: "clone"


  build:
    title: "Building Docker image"
    type: "build"
    image_name: "kostiscodefresh/my-preview-app"
    working_directory: "${{clone}}"
    tag: "${{CF_SHORT_REVISION}}"
    dockerfile: "Dockerfile"
    stage: "build"

The important part here is the “tag” line that instructs Codefresh to tag the Docker image with the same hash as the source code. The short revision variable represents the Git hash and gets automatically injected by Codefresh with the commit that we checked out.

The second requirement is to make this pipeline run only for pull request events. This happens out of the box with Codefresh Git triggers.

We now have a pipeline that satisfies our requirements. If you create pull requests on the source code repository, Codefresh automatically builds them. Here’s an example build.

When the pipeline has finished, you can see in Docker Hub (or whichever container registry you’re using) the resulting images tagged with the same Git hash as the source code branch:

This concludes the setup for the Continuous Integration part. Let’s move on to the deployment process.

Performing the initial deployment

With the CI part in place, we can create our Pull Request Generator. Here’s the initial version:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapps
  namespace: argocd  
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - pullRequest:
      github:
        owner: kostis-codefresh
        repo: preview-env-source-code
      requeueAfterSeconds: 180
  template:
    metadata:
      name: 'myapp-{{.branch}}'
    spec:
      source:
        repoURL: 'https://github.com/kostis-codefresh/preview-env-manifests.git'
        targetRevision: HEAD
        path: kustomize-preview-app/
        kustomize:
          images:
            - 'docker.io/kostiscodefresh/my-preview-app:{{.head_short_sha_7}}'
      project: "default"
      destination:
        server: https://kubernetes.default.svc
        namespace: 'preview-{{.branch_slug}}'
      syncPolicy:
        automated:
          prune: true  
        syncOptions:
        - CreateNamespace=true

This Pull Request Generator does the following:

1. Monitors the kostis-codefresh/preview-env-source code repository for pull requests.
2. If it finds a pull request, it creates an Argo CD application named myapp-<name of branch>.
3. The Argo CD application syncs manifests found at preview-env-manifests folder “kustomize-preview-app”.
4. It deploys the manifests to the local cluster (the same cluster where Argo CD is installed) at a namespace with the same name as the branch of the pull request prepended with “preview”.
5. It sets the container tag with the same hash as the source code, so that it matches the CI pipelines of Codefresh described in the previous section.

You can test this Pull Request Generator by applying it to your local Argo CD instance.

kubectl apply -f pr-generator.yml -n argocd

Then, you can start creating pull requests on your source code repository and see applications created automatically. Isn’t this great?

Note that in our example, we instructed Argo CD to check for new pull requests every 3 minutes via the requeueAfterSeconds property. You need to fine-tune this property according to your organization’s needs.

As always, you can click on any application to see its Kubernetes resources.

That’s it!

Now every time a developer creates a new pull request, their container image gets deployed automatically in the same namespace as the branch name. You can also inspect the applications manually in the cluster with the CLI.

➜  argocd git:(main) kubectl get ns
NAME                       STATUS   AGE
argocd                     Active   26d
default                    Active   26d
kube-node-lease            Active   26d
kube-public                Active   26d
kube-system                Active   26d
preview-fix-queue-size     Active   2m14s
preview-my-billing-feature   Active   2m14s

To make things easy, we use the “preview” prefix for all namespaces. But you can choose any naming convention for your organization.

Handling subsequent commits for the same pull request

Often, a developer notices something in a preview environment, fixes it, and wants to perform a redeployment. With our setup, this happens automatically. If the developer commits again, the Pull Request Generator detects the commit and redeploys the application.

Note that in our example, we set the refresh period to 3 minutes. Our application is very simple, so usually when the Pull Request Generator is redeploying an environment, the container image is already built and pushed to our registry.

There’s also the scenario where the image build is not yet complete. Argo CD will mark the deployment with an ImagePullBackOff error. Kubernetes will automatically retry the pull later and the application will eventually succeed. But note that this will work only if your application takes less than 5 minutes to build as this is the default timeout for image pull operations.

If this is an issue for your developers, you can simply instruct the Pull Request Generator to only deploy pull requests with a special label, for example “preview-ready”. Then you have your CI system or developers add that label on the pull request after the CI build has finished. This way you guarantee the image is always there when Argo CD creates the temporary environment.

Destroying the temporary Argo CD application

Destroying an environment is straightforward. After you merge or reject/close a pull request, Argo CD detects it and automatically discards the respective environment.

Note that in our simple example, the preview namespace stays behind. To delete the namespace as well, you need to add a namespace resource in your Helm chart or Kustomize folder.

Alternatively, you can set up a job that periodically clears unused namespaces.

Passing the name of the branch to a preview environment

The setup we have works great for several common scenarios. One key point is that the application itself doesn’t know if it’s running in a preview environment.

Sometimes, however, you want to pass to the application, information about the pull request environment, and most probably give it as a parameter with the name of the branch or the Git hash. These can be used to make the application adapt in some way to the preview environment. An example scenario would be to set up an Ingress with the URL that matches the pull request. So in our example, we’d have URLs like this:

• example.com/fix-queue-size
• example.com/my-billing-app

Our example application source code already accepts the Git hash and the branch name as environment variables.

We can modify our Pull Request Generator to pass these parameters in the created application.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapps
  namespace: argocd  
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - pullRequest:
      github:
        owner: kostis-codefresh
        repo: preview-env-source-code
      requeueAfterSeconds: 180
  template:
    metadata:
      name: 'myapp-{{.branch}}'
    spec:
      source:
        repoURL: 'https://github.com/kostis-codefresh/preview-env-manifests.git'
        targetRevision: HEAD
        path: kustomize-preview-app/
        kustomize:
          images:
            - 'docker.io/kostiscodefresh/my-preview-app:{{.head_short_sha_7}}'        
          patches:
            - target:
                kind: Deployment
                name: simple-deployment
              patch: |-
                - op: replace
                  path: /spec/template/spec/containers/0/env/0
                  value:
                    name: GIT_BRANCH
                    value: '{{.branch_slug}}'
                - op: replace
                  path: /spec/template/spec/containers/0/env/1
                  value:
                    name: GIT_HASH
                    value: '{{.head_short_sha_7}}'
      project: "default"
      destination:
        server: https://kubernetes.default.svc
        namespace: 'preview-{{.branch_slug}}'
      syncPolicy:
        automated:
          prune: true  
        syncOptions:
        - CreateNamespace=true

Now, when you create a preview application, if you visit the application, you see that it knows which branch and hash it was created from.

Notice that this override syntax is something we DO NOT recommend for static and/or production environments. It mixes Argo CD resources with information that belongs to the Kubernetes manifests. We recognize, however, that some developers like this capability, so we include it in this guide for completeness.

Limitations and Future Considerations

The approach we’ve seen — monitoring the source code repository for pull requests — is very simple to set up, but there are some limitations:

1. By default, it deploys the new manifests without actually waiting for the container image to be ready. Kubernetes will automatically retry to pull it for the first 5 minutes only. So if your CI system needs more than 5 minutes to build (and test) the application, this approach won’t work for you.
2. Developers can only create a preview environment for a single application (the one that has the pull request). If your organization has adopted microservices, a developer might want to create a common preview environment with different applications from different Git repositories. This scenario is not possible with the approach in this guide.
3. There are some cases where the preview environment is not just a new container tag, but the developer wants to preview a change in configuration as well, for example, introducing a new variable or configuration file. Again, this scenario is not possible with the approach in this guide

You can solve all these limitations by changing the behavior of the Pull Request Generator to monitor the Kubernetes manifest repository for pull requests. This process is more complex, though, so we’ll cover it in a future article.

Summary

In this guide, you’ve seen how you can use Argo CD even for preview environments. Using the ApplicationSet Pull Request Generator, you can create temporary/ephemeral deployments so developers can test a feature in isolation. Updating the environment with new code when the developer performs additional commits in the pull request is also very easy to set up.

The benefits of GitOps can now apply to preview environments as well. For example, it’s now possible to quickly find configuration differences between environments or even revert back to a preview version of the environment using just a Git action.

With the power of the ApplicationSet Generator, you can also automatically delete all inactive temporary environments. This helps with cloud costs and long-term maintenance of temporary environments.

The process is fully transparent for developers. They can keep creating pull requests in the source code repository like they already do. After they merge/reject the pull request, Argo CD discards the temporary environment without any human intervention.

We hope you now have a good starting point for your preview environment strategy.

Creating Temporary Preview Environments Based On Pull Requests With Argo CD And Codefresh was originally published in Container Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.