Build your PKI and manage Kubernetes secrets with Hashicorp Vault
Managing usernames, passwords and certificate material is very cumbersome on any IT setup. In even relatively simple environments, if you have a number of applications talking to each other in a secure fashion, database passwords start to come up, along with keys and certificates for https that need to be created, stored and rotated. Certificate expiration is probably the number one cause of issues in IT today. Service meshes are somehow meant to help with the certificate hell, but are aimed at solving a different problem: how to transparently setup secure connections between microservice components. We are referring here to another problem: how to manage certificates and secrets that are explicitly (not transparently) required by applications.
Secrets managers are pieces of software in charge of making all this simpler and more secure, implementing scenarios such as generating temporary credentials to applications with limited permissions for a database, and executing the periodic automatic renewal of certificates. But don’t expect that all security and ease of use problems are solved using them. Not at all.
The centralization of sensitive information does not make everything secure automagically. If a secret is stored in a manager, it has to be itself protected somehow, with another secret. The recursivity could go on forever. So, what is the point of adding another layer?
In the end, there is no full solution to the problem, but the main idea is that there is a hierarchy of keys. Each level imposes more tightened security, trading security for ease of use. For example, ephemeral secrets for a Continuous Integration system can created on the fly with a validity of only a few minutes and stored in the secret manager with another secret that has longer validity and that does not live outside the manager. This second level secret is subject to heavier restrictions, but is not used so frequently, and so on. The final level could live in a physical safebox (not joking: this is what cloud providers to).
The second problem is that the secrets have to live necessarily outside the secret manager. At the very least, in the memory of the process using them. Very often, in files for being read by https servers, or in Kubernetes secrets. To mitigate potential damage, those pieces of sensitive information should have short validity time, to avoid the impact of being leaked as much as possible.
Getting security right is very difficult, so it is always a good idea to rely on products that offload as much of the burden as possible.
Hashicorp Vault is the secret manager of choice in on-prem environments. In this post we are going to show how to install it and configure it as a PKI infrastructure and as a store for Kubernetes secrets. The final goal is to fully avoid the need for manual management of certificates and database passwords. Although for a home environment Vault could be an overkill, and simply using Cert-Manager with a CA issuer instead of Vault could be enough for transient setups, the purpose of the activity is learning. Most of what is done here will work similarrily in enterprise environments.
The Kubernetes setup
For setting up Vault, any Kubernetes cluster with a default storage class will do. Even a single node kind is sufficient for what we are going to show. But for the purposes of this post we'll be using K3S. The only dependence with this specific distribution is the way to get hold of the CA certificate of the cluster, which anyway is easily available on any kubeconfig file.
Roadmap
Here is a list of what we intend to do, with emphasis in the limitations to be taken into account:
- We’ll install Hashicorp Vault on top of Kubernetes with a single node. Obviously this is an issue in production environments, but it is more than enough for learning purposes. Since the focus is on security, we’ll make sure from the beginning that access to Vault is secured with https. What certificate is used for that, given that the whole purpose of our work is to create a PKI? Good question. You’ll see later.
- We’ll setup Vault as a PKI, that is, a certificate issuer. In fact, it will act both as a root CA (Certification Authority) and as an intermediate CA. We’ll discuss how common is this scenario, and why the split, later in the post. The final goal of this is to integrate Vault with Cert-Manager, which will be in charge of making the certificates available in the Kubernetes Cluster.
- We’ll integrate Vault also with External-Secrets, which is a project for syncing Kubernetes secrets with other sources. External, as the name implies. It can be used also to “reflect” secrets. A typical scenario is to push to Vault some auto-generated secret, so that it is available to another software in another namespace.
In another post, we’ll show how to integrate the generated certificates with a Kubernetes Gateway, to expose https endpoints to the outside.
Vault installation
Preparation step: Generating the certificate for accessing Vault using https
We’ll follow the steps described in the official documentation https://developer.hashicorp.com/Vault/tutorials/Kubernetes/Kubernetes-minikube-tls with small changes.
We are facing here a chicken and egg dilemma, since it is our Vault instance that will generate TLS certificates, but it does not yet exist to emit a certificate for itself. Happily, any Kubernetes installation includes its own Certificate Authority for internal use. We’ll use that one. It could be thought that this is indeed a good idea, since all Kubernetes components will automatically trust the certificate, but in reality there is no Kubernetes component even knowing that Vault exists. So this is not the real reason. The only reason is convenience. That CA already exists, and it is better to avoid creating a new one.
For creating the key and certificate material for Vault we are executing the following steps:
- Create a key
- Create a Certificate Signing Request (CSR)
- Make the Kubernetes CA sign the CSR
- Generate a secret with the key and certificate
The commands to execute are as follows. Notice that we’ll create a temporary directory with the key material in $WORKDIR, and is a good idea to remove it when we are finished. $VAULT_SERVICE_NAME is the service name Vault will use for its stateful set, which will be vault-internal if nothing is customized in the installation with the Helm Chart (this step is explained later) and $CLUSTER_NAME will be typically local. Finally $VAULT_NAMESPACE is self-explanatory.
# Generate the private key
WORKDIR=$(mktemp -d)
openssl genrsa -out $WORKDIR/Vault.key 2048# Create the CSR configuration file...
cat > $WORKDIR/vault-csr.conf <<EOF
[req]
default_bits = 2048
prompt = no
encrypt_key = yes
default_md = sha256
distinguished_name = kubelet_serving
req_extensions = v3_req
[ kubelet_serving ]
O = system:nodes
CN = system:node:*.${VAULT_NAMESPACE}.svc.${CLUSTER_NAME}
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.${VAULT_SERVICE_NAME}
DNS.2 = *.${VAULT_SERVICE_NAME}.${VAULT_NAMESPACE}.svc.${CLUSTER_NAME}
DNS.3 = *.${VAULT_NAMESPACE}
DNS.4 = localhost
IP.1 = 127.0.0.1
EOF
# ... and the CSR itself
openssl req -new -key ${WORKDIR}/vault.key -out ${WORKDIR}/Vault.csr -config ${WORKDIR}/vault-csr.conf
# Create the CSR Kubernetes resource with very long expiration time
cat > ${WORKDIR}/csr.yaml <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: vault.svc
spec:
signerName: Kubernetes.io/kubelet-serving
expirationSeconds: 864000000
request: $(cat ${WORKDIR}/Vault.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF
# Send the CSR to Kubernetes
kubectl delete csr vault.svc &>/dev/null # Just to be idempotent
kubectl apply -f ${WORKDIR}/csr.yaml
# Wait for CSR created
echo "waiting for CSR to be Pending"
while ! kubectl get csr | grep vault.svc | grep Pending &>/dev/null; do echo -n "."; sleep 1; done
# Approve the certificate
kubectl certificate approve vault.svc
echo "waiting for CSR to be Approved"
while ! kubectl get csr | grep vault.svc | grep Approved &>/dev/null; do echo -n "."; sleep 1; done
# Retrieve the certificate
kubectl get csr vault.svc -o jsonpath='{.status.certificate}' | openssl base64 -d -A -out ${WORKDIR}/vault.crt
# Retrieve the Kubernetes CA
kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d > ${WORKDIR}/vault.ca
# Create secret with the certificate and key material
kubectl create secret generic vault-tls \
-n $VAULT_NAMESPACE \
--from-file=vault.key=${WORKDIR}/vault.key \
--from-file=vault.crt=${WORKDIR}/vault.crt \
--from-file=vault.ca=${WORKDIR}/vault.caPerforming the installation
We’ll install Vault using Helm.
There are a lot of tunable parameters and options, for which the official guide is the source of truth Vault Kubernetes Guide. In this exercise we’ll look at only a few of the options.
We’ll use a single replica. Configuring full ha with multiple replicas seems to be easy, but for now we’ll use server.ha.enable: false in the values file. On the other hand, we'll secure access to Vault using https, since this post is about security and because it does not make much sense to deploy a secrets manager that is accessible using plain text. We have already prepared the certificate in the section above. The interesting thing to note is that, in spite of having set server.ha.enable to false, we still need to have a server.ha.config section for configuring https. Another important thing to note is that server.standalone must be true in this setup, even if the server.ha.enable: false setting seems to imply the former. Otherwise, the pod will not come up.
The example values.yaml file is shown below. Note the recurrent reference to the file locations where the certificate and key material reside, and the name of the secret with that material that was created in the previous section. Readiness and Liveness probes have been set to use HTTP protocol.
# In this version, tls is enabled but ha is disabled
# Change server.standalone, ha.enabled, ha.replicas for HA
global:
enabled: true
tlsDisable: false
injector:
enabled: false
server:
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/Vault-tls/vault.ca
VAULT_TLSCERT: /vault/userconfig/Vault-tls/vault.crt
VAULT_TLSKEY: /vault/userconfig/Vault-tls/vault.key
volumes:
- name: userconfig-vault-tls
secret:
defaultMode: 420
secretName: vault-tls
volumeMounts:
- mountPath: /vault/userconfig/vault-tls
name: userconfig-vault-tls
readOnly: true
standalone:
enabled: true
affinity: ""
server:
readinessProbe:
scheme: HTTP
livenessProbe:
scheme: HTTP
ha:
enabled: false
replicas: 1
raft:
enabled: true
setNodeId: true
config: |
cluster_name = "vault-integrated-storage"
ui = true
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-tls/vault.crt"
tls_key_file = "/vault/userconfig/Vault-tls/vault.key"
tls_client_ca_file = "/vault/userconfig/vault-tls/vault.ca"
}
storage "raft" {
path = "/vault/data"
}
disable_mlock = true
service_registration "Kubernetes" {}With the configuration in place, we can register the repo and launch Helm to proceed with the installation.
# Enable hasicorp repo
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update# Install Vault
# Remember to define the used variables
helm upgrade --install vault hashicorp/vault --namespace $VAULT_NAMESPACE --create-namespace --version $VAULT_HELM_CHART_VERSION --values values.yamUnsealing Vault
Vault needs to get some special tokens every time it starts. This includes the case in which the node hosting the pod changes. This is a security measure, and is possibly the most cumbersome aspect of Vault operation. The process is called “unsealing”. There are ways to automate this, including using the Enterprise version or using an additional “development” instance of Vault. We’ll not dig into this issue here, and simply mention that, in our case, in order to do the unseal, we must execute a command in the vault pod.
This command has to be executed for initialization, just after installation and before anything else.
kubectl exec -it vault-0 -n $VAULT_NAMESPACE -- vault operator initThe output of the command includes the root token and five unseal tokens. This output needs to be saved. If lost, we’ll be unable to restart Vault again. In order to unseal Vault, we need to provide three of the five tokens with the following command.
kubectl exec -it vault-0 -n $VAULT_NAMESPACE -- vault operator unseal $UNSEAL_TOKEN_IOnly after that Vault will be usable.
Vault command line tool
We’ll use all the time the vault command line tool. This tool assumes that the Vault endpoint is reachable in VAULT_ADDR and, if using https (this is our case), that the ca file for the certificate of the endpoint is in VAULT_CACERT. Since we have installed Vault in our cluster, we need to expose the endpoint to the outside, or use Kubernetes port-forwarding. Since this is not the core of this post, which is already long enough, we are leaving that configuration as an exercise for the reader (hint: kubectl port-forward pod/vault-0 8200:8200 -n vault).
Kubernetes integrated Vault Authentication
Vault supports many authentication methods, many of them offloading the procedure to an external entity.
When a Vault client authenticates using any method, a token is assigned which is associated to a set of policies. In turn, a policy is a set of permissions. Policies are also part of Roles, which are contructs that group configuration settings together, to simplify operations.
When running Vault in Kubernetes, and when the clients of Vault are Kubernetes pods, it is appropriate to use Kubernetes integrated authentication. There are a few ways to do that in Vault, as explained in the official Hashicorp documentation (Kubernetes authentication) or with more clarity on the Cert-Manager documentation for integration with Vault (Cert-Manager with Valult). The one to use depends mainly on whether Vault is installed in the same Kubernetes or not.
We choose the method called “Use local token as reviewer JWT”. This assumes Vault is running on Kubernetes and has a Service Account token that can validate other tokens. A Role is defined in Vault that maps a Kubernetes Service Account to a policy. When a client tries to authenticate to Vault, it may present a Kubernetes token for that Service Account, it will be validated by Vault using its own Service Account and, if successful, a Vault token will be returned with the policy attached to the Vault Role.
In the Cert-Manager documentation (Vault Authentication Method: Use Kubernetes Auth), the Service Account used is one created specifically for this purpose, and the Cert-Manager Service Account must be associated a Role with capabilities to create tokens for that Service Account. In our case, we’ll build the Vault Role directly associated to the Cert-Manager Service Account, which is simpler.
All this may look complicated, and it is. Security is complicated, and Vault is not a trivial product to understand. In the above paragraphs we have introduced a lot of concepts. The only way to get familiar with them is to experience with different use cases, and suffer at least a little. In case it helps, consider the the following clarifications.
- Both Vault and Kubernetes want to be presented with a token for executing any operation. Vault supports many authentication methods to produce a usable token. So does Kubernetes. Vault / Kubernetes integration for authentication is about a pod being able to get a Vault token starting with a Kubernetes token. This method will avoid introducing additional intermediate entities, such as Vault client identities, that will pollute the system unnecessarily. We just need to define a Vault Kubernetes Role that is associated to a Kubernetes Service Account.
- The required configuration steps are:
- Configure Vault Kubernetes Authentication to use “Use local token as reviewer JWT”, which requires as a parameter only the address of the Kubernetes API Server.
- Define a Vault Policy with the permissions to issue Vault tokens.
- Define a Role that associates one Service Account, to a Policy (the one just created) and some other parameters such as defaults when issuing a certificate.
With that, we are prepared to configure Cert-Manager later to use Vault.
Create non root policies
We’ll create two policies for executing some configuration commands with non-root tokens, namely, for managing the Kubernetes integration (policy=Kubernetes) and for managing the PKI creation (policy=pki-mgmt). Those management operations could be perfectly performed using the root token, but we are here to learn, which requires some amount of suffering.
# Kubernetes management policy
echo '
path "sys/auth/kubernetes" {
capabilities = ["create", "update", "sudo"]
}# Work with Kubernetes secrets engine
path "auth/kubernetes*" {
capabilities = [ "create", "read", "update", "delete", "list", "sudo", "patch" ]
}
' > $TEMP_DIR/kubernetes.hcl
vault policy write kubernetes $TEMP_DIR/kubernetes.hcl
# PKI management policy
echo '
# to enable secrets engine
path "sys/mounts/*" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}
# List enabled secrets engines
path "sys/mounts" {
capabilities = [ "read", "list" ]
}
# Work with pki secrets engine
path "pki*" {
capabilities = [ "create", "read", "update", "delete", "list", "sudo", "patch" ]
}
' > $TEMP_DIR/pki-mgmt.hcl
vault policy write pki-mgmt $TEMP_DIR/pki-mgmt.hclThe following two policies will be attached to the Roles used for issuing certificates and for storing secrets. That is, for integration with Cert-Manager and with External-Secrets respectively
# PKI issuer policy
echo '
path "pki*" {
capabilities = [ "create", "read", "update", "delete", "list", "sudo", "patch" ]
}
' > $TEMP_DIR/pki-issuer.hclvault policy write pki-issuer $TEMP_DIR/pki-issuer.hcl
# Enable the secrets engine
vault secrets enable kv-v2
# Create a policy and a token for that policy
echo "creating external-secrets policy"
echo '
path "kv-v2/*" {
capabilities = ["read", "create", "update"]
}
' > $TEMP_DIR/external-secrets.hcl
echo
echo "Creating policy for external-secrets"
vault policy write external-secrets $TEMP_DIR/external-secrets.hclIn order to use tokens with any of the created policies, we need to execute (for instance, for the policy pki-mgmt)
PKI_MGMT_TOKEN=$(vault token create -policy=pki-mgmt -format json | jq -r .auth.client_token)
vault login $PKI_MGMT_TOKENBy default, tokens have a validity of 30 days.
Kubernetes configuration
In this step we’ll configure Vault Kubernetes authentication to use the method described above, one in which Vault it will use its own Service Account to validate the tokens presented by the Vault client. The only parameter needed is a pointer to the Kubernetes API server. We will get that endpoint with a kubectl config view command, looking into the .clusters[].cluster.server property.
Get Frank Cardozo’s stories in your inbox
Join Medium for free to get updates from this writer.
In order to configure Kubernetes authentication, we execute the following commands after having logged on with the root token
vault auth enable Kubernetes# Configure Vault to talk to the local Kubernetes API
# We will use the token mounted in Vault for using the Kubernetes token validation service
# This is the method called "Use local token as reviewer JWT" in https://developer.hashicorp.com/Vault/docs/auth/Kubernetes
# and does not require to specify a jwt or a certificate
API_SERVER=$(kubectl config view --minify --flatten -ojson | jq -r '.clusters[].cluster.server')
vault write auth/Kubernetes/config \
Kubernetes_host="$API_SERVER"Creating the PKI and integrating with Cert-Manager
After clearing up the basics, in this section we are going to explore how to configure Vault as a PKI. It will expose methods for issuing certificates for us, which we’ll integrate with Cert-Manager.
In an enterprise setup, there will already be a higher level certification authority. Getting a certificate for signing other, lower level certificates (that is, creating and intermediate CA) is not as easy as getting a standard certificate for a web site. For may scenarios, companies resort to self signed root certificates. In the end, this can be as simple as a person having access to the private key, and signing “Certificate Sign Requests” manually. In a big organization, this is too much work if implemented just like that. Thus, this “signing” activity (certificate issuing) is delegated to multiple lower level authorities, with more limited responsibility (e.g., only over a specific sub-domain, or some internal platform), called “intermediate certification authorities”. Those require a certificate signed by the higher level authority. As was the case explained above for key management, each delegation level trades security for ease of use. It is common to have intermediate certificate authorities that are fully automated, but operate on a very restricted level (e.g. for a Kubernetes cluster). Typically, Vault will hold one of those intermediate certificates, and will automate the process of issuing leaf certificates.
In this setup, we are going to host even the root certificate in Vault. An intermediate certification authority will also be created, and it will be this one signing the leaf certificate requests.
Creating the root PKI
The steps to create the root PKI involve enabling the PKI functionality in some path, generate the certificate material and generating a Vault Role with the bag of configuration parameters for issuing the certificates. In the following commands, VAULT_ADDR is the endpoint of Vault to be embedded in the URL for Certificate Revocation List retrieval embedded in the Certificates. ISSUER_BASE_NAME is part of the name we'll give to the root and the intermediate certification authorities, the second referencing the former.
vault path-help pki &>/dev/null || vault secrets enable pki
# Change the max-lease-ttl of the secrets engine for the pki, in order not to use the 30d default time
vault secrets tune -max-lease-ttl=87600h pki# Configure Vault to talk to the local Kubernetes API
# We will use the token mounted in Vault for using the Kubernetes token validation service
# This is the method called "Use local token as reviewer JWT" in https://developer.hashicorp.com/Vault/docs/auth/Kubernetes
# and does not require to specify a jwt or a certificate
API_SERVER=$(kubectl config view --minify --flatten -ojson | jq -r '.clusters[].cluster.server')
vault write auth/Kubernetes/config \
Kubernetes_host="$API_SERVER"Generating the Intermediate Certification Authority
The process for generating the intermediate PKI, to which we mount in pki_int, is similar to the process for the root CA, but with additional steps. We have to generate a CSR by executing a write on pki_int/intermediate/generate/internal, save it and get it signed by the root ca by executing a write to pki/root/sign-intermediate. The signed certificate is written to pki_int/intermediate/set-signed.
# Enable the pki engine. Good maximum expiration time, since this is a CA (although intermediate)
vault secrets enable -path=pki_int pki
vault secrets tune -max-lease-ttl=43800h pki_int# Configure the CA. Done here to avoid warning about AIA.
# These may be embedded in the certificates, but most times not really used
vault write pki_int/config/urls \
issuing_certificates="$VAULT_ADDR/v1/pki_int/ca" \
crl_distribution_points="$VAULT_ADDR/v1/pki_int/crl"# Generate an intermediate certification authority
# The private key is kept inside Vault
# A CSR is generated, to be signed by the root CA
vault write -format=json pki_int/intermediate/generate/internal \
common_name="$ISSUER_BASE_NAME-intermediate" \
issuer_name="$ISSUER_BASE_NAME-intermediate" \
| jq -r '.data.csr' > $INTERMEDIATE_CSR_NAME# Sign the CSR, getting the certificate material
vault write -format=json pki/root/sign-intermediate \
issuer_ref="$ISSUER_BASE_NAME-ca" \
csr="@$INTERMEDIATE_CSR_NAME" \
format=pem_bundle ttl="43800h" \
| jq -r '.data.certificate' > "$INTERMEDIATE_CERT_NAME"# Import the certificate to finalise the creation of the intermediate CA
vault write pki_int/intermediate/set-signed certificate=@$INTERMEDIATE_CERT_NAME# Generate the role for intermediate CA certificate generation
# require_cn=false. Vaults complains if a CSR does not include this, although Cert-Manager has the opposite policy
vault write pki_int/roles/"$ISSUER_BASE_NAME-intermediate" \
issuer_ref="$(Vault read -field=default pki_int/config/issuers)" \
allowed_domains="$BASE_DOMAIN" \
allow_subdomains=true \
require_cn=false \
max_ttl="$MAX_CERTIFICATES_TTL" \
ttl="5m"Manually generating a certificate
With the above steps in place, we can generate a certificate with
vault write pki_int/issue/"$ISSUER_BASE_NAME-intermediate" \
common_name="test.$BASE_DOMAIN" \
ttl="5m" -format=json &>/dev/nullThe certificate generated is inside Vault, but we can recover it using
# Will return a list of serial number for the stored certificates
vault list pki_int/certs
vault read pki_int/cert/<serial-number>This way we can check that the above steps worked correctly.
Integrating the PKI with Cert-Manager
Using the just created PKI via Cert-Manager is not trivial. We have to arrange a few things as explained below.
The key element in Cert-Manager is the object issuing certificates, which may be an Issuer or a ClusterIssuer, the latter being able to be referenced from any namespace. This is the kind we'll use in this exercise, although it is somewhat more secure to stick to an Issuer when possible.
Let’s show here the manifest to be used, although some of the elements referenced do not yet exist. We’ll explain them in detail after understanding why they are needed
apiVersion: Cert-Manager.io/v1
kind: ClusterIssuer
metadata:
name: my-cluster-issuer
spec:
Vault:
path: pki_int/sign/$ISSUER_BASE_NAME-intermediate
server: https://Vault.$VAULT_NAMESPACE:8200
caBundleSecretRef:
name: k8s-ca
key: ca.crt
auth:
Kubernetes:
role: Vault-issuer-role
serviceAccountRef:
name: Cert-Managerspec.Vault.path points to the Vault endpoint with the intermediate certification authority that we created. We are cleared here. The server property is the URL where Vault is reachable. It can and should be an internal Kubernetes URL. But here is the first issue: it is an https endpoint. We need thus to let the ClusterIssuer that it can trust the certificate shown by that endpoint. We started this exercise creating a certificate for the Vault endpoint that was signed by the internal Kubernetes CA. The certificate for this CA is available in all namespaces under a configmap with name kube-root-ca.crt in k3s. Unfortunately, ClusterIssuer requires a secret, so we generate one with the same content.
CA_CRT=$(kubectl get cm kube-root-ca.crt -o=jsonpath='{.data.ca\.crt}')
kubectl create secret generic k8s-ca -n Cert-Manager --from-literal=ca.crt="$CA_CRT"Now, the ClusterIssuer needs to authenticate in Vault. As explained, we'll use Kubernetes authentication. ClusterIssuer will present a token that will be authorized by Vault. Vault is instructed to trust tokens for some Service Accounts via a Role configuration, which also assigns policies to them. We create the Role with the following command.
Vault write auth/Kubernetes/role/Vault-issuer-role \
bound_service_account_names=Vault-issuer-sa \
bound_service_account_namespaces=Cert-Manager \
policies=pki-issuer \
audience="Vault://my-cluster-issuer" \
ttl=10mFor the audience field, just use Vault://$name_of_the_cluster_issuer.
The policy applied is our friend pki-issuer created above. We set a default TTLfor the credentials of only 10m. And finally define which Service Account will be allowed to present tokens to impersonate this Role. The Cert-Manager documentation for this (Authentication) uses an additional service-account. We are using here instead the Cert-Manager sa directly. We just need to add a Role and RoleBinding to grant cert-manager Service Account the permission to create tokens for itself, using the following manifests.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: token-issuer
namespace: Cert-Manager
rules:
- apiGroups: ['']
resources: ['serviceaccounts/token']
resourceNames: ['Cert-Manager']
verbs: ['create']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: token-issuer
namespace: Cert-Manager
subjects:
- kind: ServiceAccount
name: Cert-Manager
namespace: Cert-Manager
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: token-issuer
" | kubectl apply -f -Notice in the manifest for ClusterIssuer the values of spec.auth.Kubernetes.role pointing to the Vault Role created, and spec.auth.Kubernetes.serviceaccountref.name, pointing to the Cert-Manager Service Account.
After creating the ClusterIssuer we should see it as READY True. The object will try to authenticate in Kubernetes and will show Fail state if anything goes wrong
$ kubectl get clusterissuer
NAME READY AGE
my-cluster-issuer True 1mCreating a Certificate
We can now create a certificate declaratively. Behind the scenes, a CertificateRequest will be created also, and then deleted. It is common to have issues in the process that are better debugged looking for the CertificateRequest.
Here is an example certificate, whose contents will be written in a secret with fields ca.crt, tls.crt and tls.key.
apiVersion: Cert-Manager.io/v1
kind: Certificate
metadata:
name: my-certificate
namespace: test
spec:
secretName: app-certificate
dnsNames:
- $app.example.com
duration: 1h
renewBefore: 30m
issuerRef:
name: my-cluster-issuer
kind: ClusterIssuerChecking the renewal of certificates
One of the most interesting reasons to automate certificate management using Cert-Manager is performing automated renewal. Using he duration and (optionally) renewBefore fields of Certificate, we can tune the frequency of the renewals. If we describe the Certificate resource we should be able to see something like this
Status:
Conditions:
Last Transition Time: 2026-10-27T17:02:15Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2026-10-27T18:02:15Z
Not Before: 2026-10-27T17:01:45Z
Renewal Time: 2026-10-27T17:32:15Z
Revision: 3
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 14m Cert-Manager-certificates-trigger Renewing certificate as renewal was scheduled at 2026-10-27 16:25:12 +0000 UTC
Normal Issuing 14m (x3 over 111m) Cert-Manager-certificates-issuing The certificate has been successfully issued
Normal Generated 14m Cert-Manager-certificates-key-manager Stored new private key in temporary Secret resource "forgejo-9kfd5"
Normal Requested 14m Cert-Manager-certificates-request-manager Created new CertificateRequest resource "forgejo-3"The underlying secret is also obviously changed. One important thing to take into account is that not all applications are able to detect the change and act accordingly. Typically, providers of Ingress or Gateway API do this well and reload. Projects such as https://github.com/stakater/Reloader fill the gap, being able to monitor resources and trigger reloads when necessary.
Integrating with External-Secrets
The External-Secrets Operator (ESO) is an open-source Kubernetes operator that creates and manages Kubernetes secrets from sources such as Vault, and also performs the opposite process, uploading the Kubernetes secrets content to an external secret manager.
In any typical microservices deployment, secrets with credentials start to appear everywhere, to store database access credentials, access tokens to other applications, etc. Having all that info centralized in Vault, instead of only in Kubernetes secrets may be useful for some use cases. We’ll take that usefulness for granted here. In the end, the secret is still in Kubernetes, which is not the most secure place in the world, and we get duplication, but OK, let’s assume that we want or need to do it.
Assuming we have External-Secrets in place, we can create objects of type ClusterSecretStore, which are a kind of "device driver" for the external secret holder. They encapsulate the functionality to talk to it from Kubernetes. Again, there must be a way to authenticate the requests to Vault, and we'll use a similar mechanism to the one for integration with Cert-Manager: a Vault role will be created with the mapping from a Service Account to a set of policies and other parameters in a Vault Role. Here we are creating such a role.
vault write auth/Kubernetes/role/external-secrets-role \
bound_service_account_names=external-secrets \
bound_service_account_namespaces=external-secrets \
policies=external-secrets \
audience=external-secrets-audience \
ttl=10mFortunately, the standard service Acccount for external-secrets already has permissions to issue tokens for itself, so we do not need to go over the process of creating a Role and Rolebinding for that, as was necessary for Cert-Manager.
We need to make sure that a secrets engine is available in Vault, possibly having to execute this command to enable the kv-v2 engine in a path with the same name.
Vault secrets enable kv-v2With the Role and the secrets engine in place we can now create the manifest for our ClusterSecretStore
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: Vault-backend
spec:
provider:
Vault:
server: https://Vault.$VAULT_NAMESPACE:8200
caProvider:
# Can be Secret or ConfigMap
type: "ConfigMap"
# namespace is mandatory for ClusterSecretStore
namespace: "external-secrets"
name: "kube-root-ca.crt"
key: "ca.crt"
path: kv-v2
version: v2
auth:
# Authenticate against Vault using a Kubernetes ServiceAccount
# token stored in a Secret.
# https://www.Vaultproject.io/docs/auth/Kubernetes
Kubernetes:
# Path where the Kubernetes authentication backend is mounted in Vault
mountPath: Kubernetes
# A required field containing the Vault Role to assume.
role: external-secrets-role
# Optional Service Account field containing the name
# of a Kubernetes ServiceAccount
serviceAccountRef:
name: external-secrets
namespace: external-secrets
audiences:
- external-secrets-audienceSome of the fields are the same as for Cert-Manager. We need to declare the certificates for speaking https to Vault (caProvider), and reference the Vault Role just created (auth.Kubernetes.role). The Service Account used for authentication (serviceAccountRef.name) is external-secrets itself.
With all this in place, we can create for instance a PushSecret that references an existing secret and copy it to Vault, looking for changes at certain intervals.
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
name: some-push-secret
namespace: test
spec:
deletionPolicy: Delete
refreshInterval: 5m
secretStoreRefs:
# The name of the ClusterSecretStore created above
- name: Vault-backend
kind: ClusterSecretStore
selector:
secret:
name: some-existing-secret
data:
- match:
remoteRef:
remoteKey: key-of-the-secret-in-VaultIf we login to Vault GUI and look for the secret in the kv-v2, we must be able to get the value.
Conclusion
In this post we have examined how to install Vault in Kubernetes and integrate it with Cert-Manager and External-Secrets. Honestly, this is more of a self-documenting exercise, but I’ll be happy if it is of any help to anybody out there.
