<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vasilii Angapov on Medium]]></title>
        <description><![CDATA[Stories by Vasilii Angapov on Medium]]></description>
        <link>https://medium.com/@angapov?source=rss-c669aaff473------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*eNm8AEV8zhpLNIbjHrfpeg.jpeg</url>
            <title>Stories by Vasilii Angapov on Medium</title>
            <link>https://medium.com/@angapov?source=rss-c669aaff473------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:32:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@angapov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Running Red Hat CoreOS in OpenShift Virtualization]]></title>
            <link>https://angapov.medium.com/running-red-hat-coreos-in-openshift-virtualization-f5e209cf1814?source=rss-c669aaff473------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5e209cf1814</guid>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[openshift-virtualization]]></category>
            <category><![CDATA[red-hat-openshift]]></category>
            <category><![CDATA[coreos]]></category>
            <dc:creator><![CDATA[Vasilii Angapov]]></dc:creator>
            <pubDate>Mon, 29 Dec 2025 07:59:51 GMT</pubDate>
            <atom:updated>2025-12-29T07:59:51.618Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*03gobck3XjEbvhWE" /></figure><p><a href="https://www.redhat.com/en/technologies/cloud-computing/openshift/virtualization">Red Hat OpenShift Virtualization</a> is quickly gaining momentum in modern virtualization world. Many big companies are choosing OpenShift Virt as a platform for running both containerized and virtualized workloads.</p><p>One of the things that immediately comes to mind when running OpenShift Virt is: “can I run OpenShift in the VMs on top of OpenShift Virt?”. Yes, you absolutely can. But a funny thing is that openshift-installer <a href="https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/installation_overview/installing-preparing#supported-installation-methods-for-different-platforms">doesn’t support</a> OpenShift Virt as a first class compute platform so far. That might change in future but for now the only viable option is to use platform-agnostic <a href="https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/installing_on_any_platform/installing-platform-agnostic">user-provisioned infrastructure</a> (UPI).</p><p>So to be able to run OpenShift in OpenShift Virt we need to run <a href="https://docs.redhat.com/en/documentation/openshift_container_platform/4.19/html/architecture/architecture-rhcos">Red Hat CoreOS</a> VMs. And in order to run CoreOS you somehow need to provide a so called <a href="https://docs.fedoraproject.org/en-US/fedora-coreos/producing-ign/">ignition config</a> to it during boot time.</p><p>In the UPI method the standard way of providing ignition configs is via TFTP + HTTP. That requires us to run DHCP/TFTP/HTTP stack in the same subnet with the CoreOS VM. That is not very efficient, I would like to utilize platform capabilities of OpenShift Virt to automatically pass ignition configs to CoreOS VMs.</p><p>In order to achieve that we first need to create a DataVolume and a DataSource with Red Hat CoreOS ISO image. We download an official RHCOS ISO from <a href="https://mirror.openshift.com/">https://mirror.openshift.com</a> (in the example below it’s version 4.19). Here we use OpenStack variant of CoreOS because other variants have issues with passing ignition configs.</p><pre>apiVersion: cdi.kubevirt.io/v1beta1<br>kind: DataVolume<br>metadata:<br>  annotations:<br>    cdi.kubevirt.io/storage.bind.immediate.requested: &#39;true&#39;<br>    cdi.kubevirt.io/storage.usePopulator: &#39;true&#39;<br>  name: rhcos-4.19<br>  namespace: openshift-virtualization-os-images<br>  labels:<br>    instancetype.kubevirt.io/default-instancetype: u1.xlarge<br>    instancetype.kubevirt.io/default-preference: rhel.9<br>spec:<br>  source:<br>    http:<br>      url: &#39;https://mirror.openshift.com/pub/openshift-v4/amd64/dependencies/rhcos/4.19/latest/rhcos-openstack.x86_64.qcow2.gz&#39;<br>  storage:<br>    accessModes:<br>      - ReadWriteOnce<br>    resources:<br>      requests:<br>        storage: 20Gi<br>    volumeMode: Block<br>---<br>apiVersion: cdi.kubevirt.io/v1beta1<br>kind: DataSource<br>metadata:<br>  annotations:<br>    description: RHEL CoreOS 4.19<br>  labels:<br>    instancetype.kubevirt.io/default-instancetype: u1.xlarge<br>    instancetype.kubevirt.io/default-preference: rhel.9<br>  name: rhcos-4.19<br>  namespace: openshift-virtualization-os-images<br>spec:<br>  source:<br>    pvc:<br>      name: rhcos-4.19<br>      namespace: openshift-virtualization-os-images</pre><p>Next step is to create Secret with ignition config. In this example we’re defining SSH authorized key for core user (default user for CoreOS). The below ignition config also <a href="https://docs.fedoraproject.org/en-US/fedora-coreos/tutorial-autologin/">enables auto-login</a> on VM serial console (for troubleshooting purposes only, <strong>not for use in production!</strong>)</p><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: coreos-ignition-config<br>stringData:<br>  userdata: |<br>    {<br>      &quot;ignition&quot;: {<br>        &quot;version&quot;: &quot;3.4.0&quot;<br>      },<br>      &quot;passwd&quot;: {<br>        &quot;users&quot;: [<br>          {<br>            &quot;name&quot;: &quot;core&quot;,<br>            &quot;sshAuthorizedKeys&quot;: [<br>              &quot;ssh-rsa AaAaBbBb...ZzZz&quot;<br>            ]<br>          }<br>        ]<br>      },<br>      &quot;systemd&quot;: {<br>        &quot;units&quot;: [<br>          {<br>              &quot;dropins&quot;: [<br>                {<br>                  &quot;contents&quot;: &quot;[Service]\n# Override Execstart in main unit\nExecStart=\n# Add new Execstart with `-` prefix to ignore failure`\nExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM\n&quot;,<br>                  &quot;name&quot;: &quot;autologin-core.conf&quot;<br>                }<br>              ],<br>              &quot;name&quot;: &quot;serial-getty@ttyS0.service&quot;<br>          }<br>        ]<br>      }<br>    }</pre><p>The final step would be to run a VM using the above ignition config. In this example we run a 8 CPU/32 GB RAM machine with 100GB disk using a default storage class. VM root disk is cloned from CoreOS DataSource we created in the very beginning. Ignition config is passed as a cloudInitConfigDrive .</p><pre>apiVersion: kubevirt.io/v1<br>kind: VirtualMachine<br>metadata:<br>  name: coreos-test-1<br>spec:<br>  dataVolumeTemplates:<br>  - metadata:<br>      name: coreos-test-1<br>    spec:<br>      sourceRef:<br>        kind: DataSource<br>        name: rhcos-4.19<br>        namespace: openshift-virtualization-os-images<br>      storage:<br>        resources:<br>          requests:<br>            storage: 100Gi<br>  runStrategy: Always<br>  template:<br>    metadata:<br>      labels:<br>        network.kubevirt.io/headlessService: headless<br>    spec:<br>      domain:<br>        resources:<br>          limits:<br>            cpu: 8<br>            memory: 32Gi<br>        devices:<br>          disks:<br>          - name: rootdisk<br>            disk:<br>              bus: virtio<br>          - name: cloudinitdisk<br>            disk:<br>              bus: virtio<br>          interfaces:<br>          - name: default<br>            masquerade: {}<br>        machine:<br>          type: q35<br>      networks:<br>      - name: default<br>        pod: {}<br>      subdomain: headless<br>      architecture: amd64<br>      volumes:<br>      - name: rootdisk<br>        dataVolume:<br>          name: coreos-test-1<br>      - name: cloudinitdisk<br>        cloudInitConfigDrive:<br>          secretRef:<br>            name: coreos-ignition-config</pre><p>After a little wait we can see that the VM is Running.</p><pre>$ oc get vm<br>NAME            AGE     STATUS    READY<br>coreos-test-1   2m36s   Running   True</pre><p>In the VM console we can validate that the ignition config was applied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jPvf_hQyc3Ksb8WXoVkU0w.png" /></figure><p>If we switch to VM serial console and press Enter we can see that auto login actually worked!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ArHBTSNs1R1waU6IUeOInw.png" /></figure><p>So now we can log into VM with virtctl . That proves that SSH authorized key from ignition config was applied to core user.</p><pre>$ virtctl ssh core@coreos-test-1<br>{&quot;component&quot;:&quot;portforward&quot;,&quot;level&quot;:&quot;warning&quot;,&quot;msg&quot;:&quot;Defaulting to type &#39;vmi&#39;.\nOmitting the type in the target is deprecated.\nSpecifying the type will become mandatory with the next release.&quot;,&quot;pos&quot;:&quot;portforward.go:243&quot;,&quot;timestamp&quot;:&quot;2025-12-29T07:31:39.012152Z&quot;}<br>The authenticity of host &#39;vmi/coreos-test-1.default (&lt;no hostip for proxy command&gt;)&#39; can&#39;t be established.<br>ED25519 key fingerprint is SHA256:ycZ61uDop/4hV4oZfKhKP8BJ4Q3kLchL2NsdZWzlqyk.<br>This key is not known by any other names.<br>Are you sure you want to continue connecting (yes/no/[fingerprint])? yes<br>Last login: Mon Dec 29 07:29:37 2025<br>[core@coreos-test-1 ~]$</pre><p>Now how exactly does the OpenShift Virt pass the ignition config to the VM? As you can see below ignition config is mounted to VM as an additional 1MB cloud-init disk (/dev/vdb). We can then go ahead and mount the disk and examine the cloud-init user-data which is exactly the ignition config we want.</p><pre>[core@coreos-test-1 ~]$ sudo lsblk<br>NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS<br>loop0    7:0    0  5.7M  1 loop<br>vda    252:0    0   50G  0 disk<br>├─vda1 252:1    0    1M  0 part<br>├─vda2 252:2    0  127M  0 part<br>├─vda3 252:3    0  384M  0 part /boot<br>└─vda4 252:4    0 49.5G  0 part /var<br>                                /sysroot/ostree/deploy/rhcos/var<br>                                /etc<br>                                /sysroot<br>vdb    252:16   0    1M  0 disk<br>[core@coreos-test-1 ~]$ sudo mount /dev/vdb /mnt<br>mount: /var/mnt: WARNING: source write-protected, mounted read-only.<br>[core@coreos-test-1 ~]$ sudo cat /mnt/openstack/latest/user_data<br>{<br>  &quot;ignition&quot;: {<br>    &quot;version&quot;: &quot;3.4.0&quot;<br>  },<br>  &quot;passwd&quot;: {<br>    &quot;users&quot;: [<br>      {<br>        &quot;name&quot;: &quot;core&quot;,<br>        &quot;sshAuthorizedKeys&quot;: [<br>          &quot;ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTq7oCSqhRJcqsmvTKm/k4/jlbbY80VJx4If0JJY7l8UWmYNrXlaq1pmqBbU1b64WRki+dUJ6IgE6GFOjdLQq2ce9UCpXwfWwMg7Ti/nXufHdVO4CRh6VeSlSzFefGxeysYsi7B9qOQUT2kV+reuix75ekJGSdBsqvlZ5vecWohFbOHmIDdpNAEY6yeYagijAoF/klB9hYwI5RVhxEqp2D/7+71Pl4m2d3U6Gj6yTmCNK6cOJvlT+eVPAsQjTpAHuxJN2vaxz69ufNzan1uz0wY71mIhKWyaw1+VS2dSqIm1gTGvShtOJGB6AVoLQsn2l6dMmqCyabEGrh9V7w9kPt vasilii.angapov@li9.com&quot;<br>        ]<br>      }<br>    ]<br>...<br>}</pre><p>By the way path to the cloud-init openstack/latest/user_data is defined in the CoreOS code <a href="https://github.com/coreos/ignition/blob/ef86d6118071a0094f286a13c19f85f2d86a00bc/internal/providers/openstack/openstack.go#L46">here</a>.</p><p>Thank you for reading! Feel free to ask questions!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5e209cf1814" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Argo Workflows failed workflows notification with Prometheus]]></title>
            <link>https://angapov.medium.com/argo-workflows-failed-workflows-notification-with-prometheus-db4e409d0b38?source=rss-c669aaff473------2</link>
            <guid isPermaLink="false">https://medium.com/p/db4e409d0b38</guid>
            <category><![CDATA[argo-workflows]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[prometheus]]></category>
            <dc:creator><![CDATA[Vasilii Angapov]]></dc:creator>
            <pubDate>Wed, 06 Nov 2024 13:27:53 GMT</pubDate>
            <atom:updated>2024-11-07T12:31:32.863Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*tiPEROQMs9ogKpvbIOnWYQ.png" /></figure><p>In Argo Workflows there’s no built-in method of sending notifications. But what if you want to be notified of failed workflows? Well, you have several options:</p><ol><li>Use <a href="https://argo-workflows.readthedocs.io/en/latest/walk-through/exit-handlers/">exitHandler</a> . You can configure per-workflow or default exitHandler that will send notifications if the job fails.</li><li>Use <a href="https://argo-workflows.readthedocs.io/en/latest/workflow-events/">workflow generated</a> Kubernetes Events. Every time Argo WF starts or stops the workflow it emits Kubernetes Events. You can then listen to those events with something like <a href="https://argoproj.github.io/argo-events/eventsources/setup/resource/">Argo Events</a> and take actions in response.</li><li>Use <a href="https://argo-workflows.readthedocs.io/en/latest/metrics/#custom-metrics">custom metrics</a> in workflow. You can then scrape those metrics using Prometheus and send alerts with Alertmanager. This is the most operationally efficient method because you’ll be using the same Alertmanager you use for your Kubernetes cluster monitoring. This also removes the need to write a custom alerting logic which you would have to implement in case of options 1 and 2.</li></ol><p>So let’s see how we can implement failed workflows notification with Prometheus.</p><p>First of all we need to configure Argo Workflows to emit custom metric when any workflow fails. For that to work you need to <a href="https://argo-workflows.readthedocs.io/en/latest/metrics/">enable Prometheus metrics</a> in Argo Workflows.</p><p>Then you need to add the following snippet to your <a href="https://argo-workflows.readthedocs.io/en/latest/default-workflow-specs/">default workflow spec</a>:</p><pre>    workflowDefaults:<br>      spec:<br>        metrics:<br>          prometheus:<br>          - name: workflow_failed_count<br>            counter:<br>              value: &quot;1&quot;<br>            help: Failed workflow counter<br>            labels:<br>            - key: workflow_name<br>              value: &quot;{{ workflow.name }}&quot;<br>            - key: workflow_namespace<br>              value: &quot;{{ workflow.namespace }}&quot;<br>            - key: workflow_duration_seconds<br>              value: &quot;{{ workflow.duration }}&quot;<br>            - key: workflow_created_at<br>              value: &quot;{{ workflow.creationTimestamp }}&quot;<br>            - key: cron_workflow<br>              value: &quot;{{= sprig.default(&#39;&#39;, workflow.labels[&#39;workflows.argoproj.io/cron-workflow&#39;]) }}&quot;<br>            when: &quot;{{workflow.status}} == Failed || {{workflow.status}} == Error&quot;</pre><p>Because this is a default workflow spec it will be added to every workflow which is exactly what we want. Remember that the alert routing will be done on Alertmanager side.</p><p>Let’s take a closer look at the above code snippet:</p><ol><li>name defines the metric name as it will be seen by Prometheus.</li><li>counter is the type of Prometheus metric. In this case we’re increasing the counter each time the workflow fails. We will then create an alerting rule based on the counter value increase.</li><li>help is not mandatory but serves as a hint</li><li>labels defines the list of labels that will be added to Prometheus metric. In this case we’re setting up some useful label like workflow name, namespace, duration and creation timestamp. We’re also defining cron_workflow label which is not empty for any Workflowthat is triggered byCronWorkflow. If the workflow isn’t related to cron the label will be set to empty string.</li></ol><p>The spec above will result in the following Prometheus metric (assuming that Prometheus <a href="https://argo-workflows.readthedocs.io/en/latest/metrics/#prometheus-scraping">scrapes metrics</a> from Argo WF workflow-controller):</p><pre>argo_workflows_workflow_failed_count{cron_workflow=&quot;example-cron-failure&quot;,workflow_created_at=&quot;2024-11-06T12:40:00Z&quot;,workflow_duration_seconds=&quot;10.047688&quot;,workflow_name=&quot;example-cron-failure-1730896800&quot;,workflow_namespace=&quot;default&quot;} 1</pre><p>Finally let’s create a PrometheusRule (requires <a href="https://prometheus-operator.dev/">prometheus-operator</a>) like this:</p><pre>apiVersion: monitoring.coreos.com/v1<br>kind: PrometheusRule<br>metadata:<br>  name: argo-workflows<br>spec:<br>  groups:<br>  - name: argo-workflows<br>    rules:<br>    - alert: ArgoWorkflowFailed<br>      annotations:<br>        summary: Workflow {{ $labels.workflow_namespace }}/{{ $labels.workflow_name }} failed<br>        link: https://ARGO_WORKFLOWS_HOSTNAME/workflows/{{ $labels.workflow_namespace }}/{{ $labels.workflow_name }}<br>      expr: max_over_time(argo_workflows_workflow_failed_count[2m]) &gt; 0<br>      for: 0m<br>      labels:<br>        app: argo-workflows<br>        severity: warning</pre><p>Optionally you can have routing rules in your Alertmanager config like this:</p><pre>    route:<br>      routes:<br>      - matchers:<br>        - alertname = &quot;ArgoWorkflowFailed&quot;<br>        - workflow_namespace =~ &quot;.*-prod&quot;<br>        receiver: &quot;prod-alerts&quot;<br>        continue: true<br>      - matchers:<br>        - alertname = &quot;ArgoWorkflowFailed&quot;<br>        - workflow_namespace =~ &quot;.*-qa&quot;<br>        receiver: &quot;qa-alerts&quot;</pre><p>This configuration sends ArgoWorkflowFailed alerts to different receivers based on the workflow namespace.</p><p>That’s it! Thank you guys for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db4e409d0b38" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hashicorp Vault integration with OIDC provider using Identity Groups]]></title>
            <link>https://angapov.medium.com/hashicorp-vault-integration-with-oidc-provider-using-identity-groups-60d401bc1ec7?source=rss-c669aaff473------2</link>
            <guid isPermaLink="false">https://medium.com/p/60d401bc1ec7</guid>
            <category><![CDATA[hashicorp-vault]]></category>
            <category><![CDATA[okta]]></category>
            <category><![CDATA[openid-connect]]></category>
            <dc:creator><![CDATA[Vasilii Angapov]]></dc:creator>
            <pubDate>Mon, 04 Nov 2024 11:56:24 GMT</pubDate>
            <atom:updated>2024-11-04T12:12:46.018Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y8SMyAyaTf0nC27Hs7EmoA.png" /></figure><p>One of the most common ways to give users access to Vault is by integrating it with an external OIDC-compliant identity provider like Okta, Auth0, Azure Entra ID, Keycloak and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M0XA1M3fg3RnwFJQ" /></figure><p>If you read Vault <a href="https://developer.hashicorp.com/vault/tutorials/auth-methods/oidc-auth">documentation</a> on how to set up OIDC auth method you’ll learn the following typical steps:</p><ol><li>Create OIDC app in your identity provider. Assign “groups” claim to it and add Vault URL to the list of allowed redirect URIs.</li><li>Create client credentials (client id and secret) that will be used by Vault to verify user identities. These two steps are common between all OIDC-compliant providers.</li><li>Create Vault policy for the user/group.</li><li>Create Vault OIDC role that serves as a mapper between external user/group identity and Vault policy created at step 3. Mapping is done via bound_claims role attribute.</li></ol><p>As you can see Vault OIDC role plays a critical part in the authentication process. For example, I have Okta configured as an OIDC provider, so the users are greeted with the following screen when they want to log in to Vault UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*poaViMlzp06iHaEytBfn8A.png" /></figure><p>Here comes the problem!</p><p>If you have multiple user groups/teams you need to create multiple roles for them, right? That requires users to <strong>remember</strong> their roles, which sounds awkward! It sounds inconvenient for a user to always remember his Vault role. It becomes even more inconvenient if a user is a member of multiple groups (a common situation when a developer writes code for different applications). Because now he has to remember all his roles. That doesn’t sound very user-friendly, does it?</p><p>Thankfully, Vault has a solution for this problem. However, the solution is not at all obvious and pretty much hidden in the documentation. It took me almost two years to understand what they’re saying in the docs.</p><p>It’s called <a href="https://developer.hashicorp.com/vault/tutorials/auth-methods/identity">Identity Entities and Groups</a>. In terms of OIDC auth method it allows you to do this:</p><ol><li>Create a default OIDC role without <strong>any</strong> permissions.</li><li>Create an Identity entity/group with arbitrary name and desired policies assigned to it.</li><li>Create an Identity entity/group alias with the name that matches external user or group name.</li></ol><p>Here’s an interesting excerpt from the Vault “Create OIDC/JWT role” <a href="https://developer.hashicorp.com/vault/api-docs/auth/jwt#create-update-role">API documentation</a> that serves as a hint:</p><blockquote><a href="https://developer.hashicorp.com/vault/api-docs/auth/jwt#user_claim">user_claim</a> (string: &lt;required&gt;) - The claim to use to uniquely identify the user; this will be used as the name for the Identity entity alias created due to a successful login.<br><a href="https://developer.hashicorp.com/vault/api-docs/auth/jwt#groups_claim">groups_claim</a> (string: &lt;optional&gt;) - The claim to use to uniquely identify the set of groups to which the user belongs; this will be used as the names for the Identity group aliases created due to a successful login.</blockquote><p>Let me show you an example with Okta as an identity provider. Keep in mind you can always create a free Okta account at <a href="https://developer.okta.com/">https://developer.okta.com</a>.</p><p>Let’s say we have an Okta organization and we want to give members of Okta group Okta-Vault-Admins admin permissions to the Vault. We would also like to give members of the group Okta-Vault-Developers read-only permissions to KV secret engine mounted under kv/ path.</p><p>First of all, you need to create Okta app for Hashicorp Vault:</p><ol><li>Sign in to Okta Admin dashboard and create desired users and groups under “Directory”. You can also sync users and groups from other identity providers like Active Directory, Azure Entra ID and others.</li><li>Go to “Applications” and click “Create a new app integration”.</li><li>Choose “OIDC OpenID Connect” and “Web application”.</li><li>On the next screen specify app name (like Hashicorp Vault) in “App integration name” and enable “Client credentials” checkbox.</li><li>In Sign-in redirect URIs you need to add two URLs:<br>a. $VAULT_ADDR/ui/vault/auth/oidc/oidc/callbackwhere $VAULT_ADDR is the URL of your Vault instance.<br>b. http://localhost:8250/oidc/callbackfor Vault CLI.</li><li>Under Assignments you can decide whether to enable Vault access for all members of your Okta organization or whether to allow it only for a subset of groups.</li></ol><p>After app creation you can further adjust the OIDC groups claim (it’s a JSON field in the ID token that contains the list of groups the user belongs to). For example, you could limit the groups to those starting with Okta-Vault-. This is done under application’s “Sign On” tab in “OpenID Connect ID Token” section.</p><p>Once the app is created and configured you can copy Client ID from “Client Credentials” section and Client Secret from “Client Secrets”. You will also need to copy Okta Issuer URL from “Security” -&gt; “API”.</p><pre>export OKTA_CLIENT_ID=&lt;CLIENT_ID&gt;<br>export OKTA_CLIENT_SECRET=&lt;CLIENT_SECRET&gt;<br>export OKTA_URL=&lt;OKTA_ISSUER_URL&gt;<br>export VAULT_ADDR=&lt;VAULT_URL&gt;</pre><p>Now it’s time to set up the Vault OIDC integration. First let’s create admin and developer policies.</p><pre># Admin policy gives unlimited Vault permissions<br>cat &gt; admin.hcl &lt;&lt;EOF<br>path &quot;*&quot; {<br>    capabilities = [&quot;sudo&quot;,&quot;read&quot;,&quot;create&quot;,&quot;update&quot;,&quot;delete&quot;,&quot;list&quot;,&quot;patch&quot;]<br>}<br>EOF<br>vault policy write admin admin.hcl<br><br># Developer role allows to list and read secrets under &quot;kv/&quot; path<br>cat &gt; developer.hcl &lt;&lt;EOF<br>path &quot;kv/*&quot; {<br>    capabilities = [&quot;read&quot;, &quot;list&quot;]<br>}<br>EOF<br>vault policy write developer developer.hcl</pre><p>Next we enable OIDC auth method according with the documentation:</p><pre># Enable OIDC auth method<br>vault auth enable oidc<br><br># Add Okta connection details and set the default role<br>vault write auth/oidc/config \<br>  oidc_discovery_url=&quot;$OKTA_URL&quot; \<br>  oidc_client_id=&quot;$OKTA_CLIENT_ID&quot; \<br>  oidc_client_secret=&quot;$OKTA_CLIENT_SECRET&quot; \<br>  default_role=&quot;default&quot;<br><br># Add OIDC auth method to the UI login screen<br>vault auth tune -listing-visibility=unauth oidc</pre><p>The default_rolehere is very important since it will be the only role in the OIDC auth method. Let’s go ahead and create the defaultrole:</p><pre>vault write auth/oidc/role/default \<br>  bound_audiences=&quot;$OKTA_CLIENT_ID&quot; \<br>  allowed_redirect_uris=&quot;$VAULT_ADDR/ui/vault/auth/oidc/oidc/callback&quot; \<br>  allowed_redirect_uris=&quot;http://localhost:8250/oidc/callback&quot; \<br>  user_claim=&quot;user.email&quot; \<br>  groups_claim=&quot;groups&quot; \<br>  token_policies=&quot;default&quot;</pre><p>As you know <a href="https://developer.hashicorp.com/vault/docs/concepts/policies#default-policy">default</a> policy exists by default, cannot be removed and gives no special permissions.</p><p>Now let’s go ahead and create <a href="https://developer.hashicorp.com/vault/tutorials/auth-methods/identity#create-an-external-group">external identity groups</a> Okta-Vault-Admins and Okta-Vault-Developers. These names technically should not match with Okta group names but for the sake of operational simplicity let’s have them match.</p><pre>vault write -format=json identity/group name=&quot;Okta-Vault-Admins&quot; \<br>     policies=&quot;admin&quot; \<br>     type=&quot;external&quot; | jq -r &quot;.data.id&quot; &gt; admin_group_id.txt<br><br>vault write -format=json identity/group name=&quot;Okta-Vault-Developers&quot; \<br>     policies=&quot;developer&quot; \<br>     type=&quot;external&quot; | jq -r &quot;.data.id&quot; &gt; developer_group_id.txt</pre><p>Now let’s create group aliases that <strong>must</strong> match Okta group names.</p><pre>vault auth list -format=json | jq -r &#39;.[&quot;oidc/&quot;].accessor&#39; &gt; accessor.txt<br><br>vault write identity/group-alias name=&quot;Okta-Vault-Admins&quot; \<br>     mount_accessor=$(cat accessor.txt) \<br>     canonical_id=&quot;$(cat admin_group_id.txt)&quot;<br><br>vault write identity/group-alias name=&quot;Okta-Vault-Developers&quot; \<br>     mount_accessor=$(cat accessor.txt) \<br>     canonical_id=&quot;$(cat developer_group_id.txt)&quot;</pre><p>This is it! Now when the user logs in via OIDC auth method he gets assigned a defaultpolicy but his groups claim from Okta will be also matched against identity group aliases. The user will be assigned all the matched group policies.</p><p>Let’s test it out.</p><pre>$ vault login -method=oidc<br>Complete the login via your OIDC provider. Launching browser to:<br><br>    https://REDACTED.okta.com/oauth2/default/v1/authorize?client_id=REDACTED&amp;code_challenge=REDACTED&amp;code_challenge_method=S256&amp;nonce=n_66KsLhMrbu9KniAjA3Xn&amp;redirect_uri=http%3A%2F%2Flocalhost%3A8250%2Foidc%2Fcallback&amp;response_type=code&amp;scope=openid&amp;state=st_isH1SA1UhWo7qdpAGFuN<br><br><br>Waiting for OIDC authentication to complete...<br>WARNING! The VAULT_TOKEN environment variable is set! The value of this<br>variable will take precedence; if this is unwanted please unset VAULT_TOKEN or<br>update its value accordingly.<br><br>Success! You are now authenticated. The token information displayed below<br>is already stored in the token helper. You do NOT need to run &quot;vault login&quot;<br>again. Future Vault requests will automatically use this token.<br><br>Key                  Value<br>---                  -----<br>token                hvs.CAESIPTMBUZRVGYMs5IamLGcQM8fRGhi85BqB3NA0Q_nyaJwGh4KHGh2cy5GVjR6NG5GZ0JlTGsxZmlKSGFCTkpKTDc<br>token_accessor       DZzbTeEliGIqGA0rtkwaiUPC<br>token_duration       1h<br>token_renewable      true<br>token_policies       [&quot;default&quot;]<br>identity_policies    [&quot;admin&quot;]<br>policies             [&quot;admin&quot; &quot;default&quot;]<br>token_meta_role      default</pre><p>Here we can see that user was assigned defaultpolicy from token (we configured that on defaultrole creation) and admin policy from his Okta identity (comes from identity alias). The resulting set of policies is admin, default which is exactly what we want.</p><p>Finally let’s have a look at identity objects that were created.</p><pre>$ vault list identity/group/name<br>Keys<br>----<br>Okta-Vault-Admins<br><br>$ vault read identity/group/name/Okta-Vault-Admins<br>Key                  Value<br>---                  -----<br>alias                map[canonical_id:425da627-730d-8579-d865-d1e70400d923 creation_time:2024-11-03T20:14:00.764054964Z id:e1654fa4-2516-71b6-fe4a-fae7cc509039 last_update_time:2024-11-04T11:27:39.855948677Z merged_from_canonical_ids:&lt;nil&gt; metadata:&lt;nil&gt; mount_accessor:auth_oidc_61baf77a mount_path:auth/oidc/ mount_type:oidc name:Okta-Vault-Admins]<br>creation_time        2024-11-03T20:13:58.913899344Z<br>id                   425da627-730d-8579-d865-d1e70400d923<br>last_update_time     2024-11-04T11:27:39.85594435Z<br>member_entity_ids    [8197c3f4-682c-b36d-1107-3e0fb3cee71c]<br>member_group_ids     &lt;nil&gt;<br>metadata             map[]<br>modify_index         6<br>name                 Okta-Vault-Admins<br>namespace_id         root<br>parent_group_ids     &lt;nil&gt;<br>policies             [admin]<br>type                 external<br><br>$ vault list identity/entity/name<br>Keys<br>----<br>entity_176c43a8<br><br>$ vault read identity/entity/name/entity_176c43a8<br>Key                    Value<br>---                    -----<br>aliases                [map[canonical_id:8197c3f4-682c-b36d-1107-3e0fb3cee71c creation_time:2024-11-03T20:15:38.898546783Z custom_metadata:&lt;nil&gt; id:e18efc0b-c505-ef98-cf0b-c719a2c4fdf8 last_update_time:2024-11-03T20:15:38.898546783Z local:false merged_from_canonical_ids:&lt;nil&gt; metadata:map[role:default] mount_accessor:auth_oidc_61baf77a mount_path:auth/oidc/ mount_type:oidc name:user@example.com]]<br>creation_time          2024-11-03T20:15:38.898534584Z<br>direct_group_ids       [425da627-730d-8579-d865-d1e70400d923]<br>disabled               false<br>group_ids              [425da627-730d-8579-d865-d1e70400d923]<br>id                     8197c3f4-682c-b36d-1107-3e0fb3cee71c<br>inherited_group_ids    []<br>last_update_time       2024-11-03T20:15:38.898534584Z<br>merged_entity_ids      &lt;nil&gt;<br>metadata               &lt;nil&gt;<br>name                   entity_176c43a8<br>namespace_id           root<br>policies               []</pre><p>Thanks a lot for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60d401bc1ec7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes container images signing using Cosign, Kyverno, HashiCorp Vault and GitLab CI]]></title>
            <link>https://angapov.medium.com/kubernetes-container-images-signing-using-cosign-kyverno-hashicorp-vault-and-gitlab-ci-c4e2041d1310?source=rss-c669aaff473------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4e2041d1310</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[kyverno]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cosign]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Vasilii Angapov]]></dc:creator>
            <pubDate>Mon, 22 Apr 2024 02:19:19 GMT</pubDate>
            <atom:updated>2024-04-22T12:33:36.443Z</atom:updated>
            <content:encoded><![CDATA[<p>Container images are the crucial part of the applications running in Kubernetes. But how can we make sure that the container images that we build with CI/CD are the same images we run in Kubernetes? How can we forbid running the untrusted images?</p><p>In this article I’ll show how to achieve supply chain security using the popular open-source tools:</p><ol><li><a href="https://docs.sigstore.dev/signing/quickstart/">Sigstore Cosign</a> is a modern signing and verification tool. I will use it for signing the container images in this article. What I especially love about Cosign is that it’s just a single binary, not a service.</li><li><a href="https://kyverno.io/">Kyverno</a> is a powerful policy engine for Kubernetes. It has numerous use cases one of which is <a href="https://kyverno.io/docs/writing-policies/verify-images/sigstore/">image signature verification</a>. It allows to verify images signed with Cosign.</li><li><a href="https://www.vaultproject.io/">HashiCorp Vault</a> is a secrets management service. Its <a href="https://developer.hashicorp.com/vault/docs/secrets/transit">transit secret engine</a> (so called encryption-as-a-service) can be used by Cosign and Kyverno to sign and verify image signatures. In this case Vault acts as a trusted Key Management Service (KMS).</li></ol><p>In this article GitLab CI will be used as CI/CD tool for a container image build, although you can use any CI/CD you’re comfortable with (Github Actions, Jenkins and so on). I will also use GitLab Container Registry to store container images but you’re free to use any container registry you like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OHGCXSWuvfvwK3TIAgtf0Q.png" /></figure><p>The overall architecture looks like this:</p><ol><li>As a part of the GitLab CI pipeline wewill utilize Cosign in order to sign the container image and upload the resulting image signature to the Gitlab Container Registry. We will use Vault as a KMS provider.</li><li>We will create a Kyverno ClusterPolicy that checks that: 1) Image is pulled from the trusted container registry, 2) It is signed by trusted Vault key. If the image is not signed or signed by untrusted key, the Kyverno will block pod creation (if the ClusterPolicy mode is “Enforce”). Alternatively we can create ClusterPolicy in Audit mode in which case it will allow pod creation but generate a warning event that can be investigated later.</li><li>Each time we build a new image it’ll get new signature uploaded and then verified by Kyverno in Kubernetes. Which means the more frequently we build the image the more image signatures we will have. Ideally we need to set up periodic registry clean up to get rid of outdated image signatures.</li></ol><p>Important fact here is that everything happens transparently to the user. The user is not aware that image signature is checked. No modification to existing deployment manifests is required.</p><p>You’ll the following components to be installed.</p><ol><li>Kubernetes cluster.</li><li>HashiCorp Vault cluster: <a href="https://developer.hashicorp.com/vault/install">https://developer.hashicorp.com/vault/install</a></li><li>Kyverno: <a href="https://kyverno.io/docs/installation/">https://kyverno.io/docs/installation/</a></li></ol><h3>Vault setup</h3><p>The first thing you’ll need to do is to create Vault transit encryption key. In this example the key is named cosign :</p><pre>$ vault secrets enable transit<br>Success! Enabled the transit secrets engine at: transit/<br>$ vault write -f transit/keys/cosign<br>Success! Data written to: transit/keys/cosign</pre><p>Next thing is to define Vault policy that will allow the above key usage. I created the following policy:</p><pre>$ cat &gt; cosign.hcl &lt;&lt;EOF<br>path &quot;transit/keys/cosign&quot; {<br>  capabilities = [&quot;read&quot;]<br>}<br><br>path &quot;transit/hmac/cosign/*&quot; {<br>  capabilities = [&quot;update&quot;]<br>}<br><br>path &quot;transit/sign/cosign/*&quot; {<br>  capabilities = [&quot;update&quot;]<br>}<br><br>path &quot;transit/verify/cosign&quot; {<br>  capabilities = [&quot;create&quot;]<br>}<br><br>path &quot;transit/verify/cosign/*&quot; {<br>  capabilities = [&quot;update&quot;]<br>}<br>EOF<br><br>$ vault policy write cosign cosign.hcl<br>Success! Uploaded policy: cosign</pre><p>Both Cosign and Kyverno will need Vault roles with the above policy attached in order to sign (with Cosign) and verify (with Kyverno) image signatures. Both Cosign and Kyverno will need access to the Container Registry storing the image and its signatures.</p><p>Cosign will be used as a part of GitLab pipeline. In order to authenticate GitLab pipelines to Vault we can use GitLab JWT provider as described in <a href="https://docs.gitlab.com/ee/ci/examples/authenticating-with-hashicorp-vault/">their article</a>. I won’t cover the GitLab → Vault authentication in this article (otherwise the article will become overly huge).</p><p>The next question is how we’re gonna authenticate Kyverno to Vault. Ideally, because Kyverno runs in Kubernetes, we would naturally like to use <a href="https://developer.hashicorp.com/vault/docs/auth/kubernetes">Kubernetes auth method</a> as the most secure one. However, Kyverno doesn’t yet support Kubernetes auth method for Vault (see <a href="https://github.com/kyverno/kyverno/issues/4274">Github issue</a>). The only method available to us so far is VAULT_TOKEN environment variable for Kyverno. It’s far from ideal but it is what it is.</p><p>Now we need to create a long-lived Vault token with the cosign policy attached.</p><pre>$ vault token create -policy=cosign -orphan -ttl=8760h<br><br>Key                  Value<br>---                  -----<br>token                hvs.CAESIEGiDhSts4rDwJQw4Twre1IJACXB5h288PWJVgZFMSbcGh4KHGh2cy5NZ09icUxqdmZnYU1wd1VJZGE5M0pDV0k<br>token_accessor       RtW8rWhB7trpfyfm9IC8Fow1<br>token_duration       8760h<br>token_renewable      true<br>token_policies       [&quot;cosign&quot;]<br>identity_policies    []<br>policies             [&quot;cosign&quot;]</pre><p>In this example we created a token with a TTL of 1 year (I really hope Kyverno will start supporting Kubernetes auth method soon). The token has to be orphan in order to not to depend on parent token TTL.</p><p>Save the long token value above, we will need it on the next step.</p><p>Log in to Kubernetes and go to the namespace where Kyverno is installed (in my case it’s <strong>kyverno</strong>). Now let’s create a secret containing a Vault connection details (Vault URL and Vault token above):</p><pre>$ export VAULT_ADDR=&lt;YOUR VAULT URL&gt;<br>$ export VAULT_TOKEN=&lt;YOUR VAULT TOKEN&gt;<br>$ kubectl -n kyverno create secret generic kyverno-vault-integration --from-literal=VAULT_ADDR=$VAULT_ADDR --from-literal=VAULT_TOKEN=$VAULT_TOKEN</pre><p>Prepare a patch for Kyverno admission controller deployment and apply the patch:</p><pre>cat &gt; patch.yaml &lt;&lt;EOF<br>spec:<br>  template:<br>    spec:<br>      containers:<br>      - name: kyverno<br>        env:<br>        - name: VAULT_ADDR<br>          valueFrom:<br>            secretKeyRef:<br>              key: VAULT_ADDR<br>              name: kyverno-vault-integration<br>        - name: VAULT_TOKEN<br>          valueFrom:<br>            secretKeyRef:<br>              key: VAULT_TOKEN<br>              name: kyverno-vault-integration<br>EOF<br><br>$ kubectl -n kyverno patch deploy kyverno-admission-controller --patch-file patch.yaml<br>deployment.apps/kyverno-admission-controller patched</pre><p>Alternatively if you installed Kyverno using <a href="https://github.com/kyverno/kyverno/tree/main/charts/kyverno">Helm chart</a> (which is the recommended way) you can include the following lines into your Helm values:</p><pre>admissionController:<br>  container:<br>    extraEnvVars:<br>    - name: VAULT_ADDR<br>      valueFrom:<br>        secretKeyRef:<br>          key: VAULT_ADDR<br>          name: kyverno-vault-integration<br>    - name: VAULT_TOKEN<br>      valueFrom:<br>        secretKeyRef:<br>          key: VAULT_TOKEN<br>          name: kyverno-vault-integration</pre><p>Now Kyverno has all it needs to authenticate to Vault.</p><h3>GitLab pipeline</h3><p>Proceeding to GitLab CI pipeline. I used the following pipeline snippet to build, push and sign the container image:</p><pre>container-build:<br>  stage: build<br>  image: ghcr.io/angapov/kaniko-cosign<br>  variables:<br>    VAULT_AUTH_PATH: auth/gitlab/login<br>    VAULT_AUTH_ROLE: cosign<br>    COSIGN_KEY: hashivault://cosign<br>    KANIKO_ARGS: --context=$CI_PROJECT_DIR --dockerfile=${CI_PROJECT_DIR}/Containerfile --digest-file=./digest<br>  id_tokens:<br>    VAULT_ID_TOKEN:<br>      aud: https://gitlab.com<br>  before_script:<br>  - echo &quot;{\&quot;auths\&quot;:{\&quot;${CI_REGISTRY}\&quot;:{\&quot;auth\&quot;:\&quot;$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64 -w0)\&quot;}}}&quot; &gt; /kaniko/.docker/config.json<br>  - export VAULT_TOKEN=$(vault write --field=token $VAULT_AUTH_PATH role=$VAULT_AUTH_ROLE jwt=$VAULT_ID_TOKEN)<br>  script:<br>  - /kaniko/executor ${KANIKO_ARGS} --destination=${CI_REGISTRY_IMAGE}:${CI_COMMIT_BRANCH}<br>  - cosign sign ${CI_REGISTRY_IMAGE}@$(cat digest) --tlog-upload=false</pre><p>This snippet requires some explanations:</p><ol><li>Here we use <a href="https://github.com/GoogleContainerTools/kaniko">Kaniko</a> as an image builder. I had to build a custom image <strong>ghcr.io/angapov/kaniko-cosign</strong> (which is btw also signed by Cosign) that contains kaniko, cosign and vault binaries. Kaniko itself isn’t important here, you can replace it with Docker/Podman/Buildkit or anything you want.</li><li>VAULT_AUTH_PATH is the Vault auth path for GitLab JWT auth method (see <a href="https://docs.gitlab.com/ee/ci/examples/authenticating-with-hashicorp-vault/">documentation</a>). VAULT_AUTH_ROLE is the corresponding Vault JWT role. Same relates to id_tokens — it’s used to authenticate GitLab to Vault. <em>I didn’t explain how to set this up in the article.</em></li><li>COSIGN_KEY is the environment variable <a href="https://docs.sigstore.dev/key_management/overview/#hashicorp-vault">used by Cosign</a> to determine which key should be used for image signing. In this case it uses Vault transit key named cosign.</li><li>KANIKO_ARGS is the common set of args used by Kaniko. In this case I use --digest-file=./digest in order to write image digest to a file where it can be read by Cosign on the next line. The reason is that I build an image with named tag (like main)instead of digest. Cosign always expects to sign an image with a digest, not a tag, because the tag is mutable whereas the digest is not.</li></ol><p>The fun part begins in the last line:</p><pre>cosign sign ${CI_REGISTRY_IMAGE}@$(cat digest) --tlog-upload=false</pre><p>We’re signing the container image but we’re NOT uploading the signature record (--tlog-upload=false) into the <strong>public</strong> transparency log called <a href="https://docs.sigstore.dev/logging/overview/">Rekor</a> (<a href="https://rekor.sigstore.dev/">https://rekor.sigstore.dev</a>).</p><p>Now, what exactly is Rekor and why I don’t wanna use it in this case? Rekor by definition is an “immutable, tamper-resistant ledger of metadata generated within a software project’s supply chain”. Rekor is very similar to TLS <a href="https://certificate.transparency.dev/">Certificate Transparency log</a> (CTlog) which is the live public append-only log of all issued public TLS certs. In the context of Cosign, Rekor maintains public records of all data signed with Cosign, allowing for querying, verification, and auditing.</p><p>In my case I use Cosign to sign private container images for my company. It makes absolutely zero sense to upload private signatures to public storage. Although it’s completely possible to deploy a <a href="https://docs.sigstore.dev/logging/installation/#deploy-a-rekor-server-manually">private Rekor instance</a> and use it internally within the company.</p><p>Ok, we uploaded an image signature. But where the signature is actually stored? Here’s an example:</p><p>Image URL: <strong><em>registry.gitlab.com/testcompany/testimage@sha256:c73ce979afcfbbe03084fe77853e0e0e6b740247b0415eb503810edd9b2f5f9b</em></strong></p><p>Image<em> </em>Signature URL:<br><strong><em>registry.gitlab.com/testcompany/testimage@sha256-c73ce979afcfbbe03084fe77853e0e0e6b740247b0415eb503810edd9b2f5f9b.sig</em></strong></p><p>As you can see, essentially, the signature is stored within the same image registry as the main image. The signature carries the identical image digest, with the addition of “.sig” appended to it. Very easy to remember!</p><p>Up to this moment we uploaded the image and the image signature. What’s next? Now let’s verify it in Kubernetes using Kyverno!</p><h3>Kyverno</h3><p>Let’s create the following Kyverno ClusterPolicy:</p><pre>apiVersion: kyverno.io/v1<br>kind: ClusterPolicy<br>metadata:<br>  name: container-image-policy<br>spec:<br>  validationFailureAction: Enforce<br>  background: false<br>  webhookTimeoutSeconds: 30<br>  failurePolicy: Fail<br>  rules:<br>  - name: gitlab-container-registry-required<br>    match:<br>      resources:<br>        kinds:<br>        - Pod<br>        namespaces:<br>        - test<br>    validate:<br>      message: &quot;Image must be from registry.gitlab.com/testcompany/* container registry&quot;<br>      pattern:<br>        spec:<br>          containers:<br>          - image: &quot;registry.gitlab.com/testcompany/*&quot;<br>  - name: verify-image-signature<br>    match:<br>      resources:<br>        kinds:<br>        - Pod<br>        namespaces:<br>        - test<br>    verifyImages:<br>    - imageReferences:<br>      - &quot;registry.gitlab.com/testcompany/*&quot;<br>      attestors:<br>      - count: 1<br>        entries:<br>        - keys:<br>            rekor:<br>              ignoreTlog: true<br>              url: https://rekor.sigstore.dev<br>            kms: hashivault://cosign</pre><p>This policy has validationFailureAction: Enforce which means it will deny any resource (in this case Pod) that doesn’t satisfy the rules. It also has background: false which means it doesn’t run in the background. It also has failurePolicy: Fail so that if Kyverno for any reason fails to respond to the request from kube-apiserver the latter will consider the request at failed.</p><pre>    match:<br>      resources:<br>        kinds:<br>        - Pod<br>        namespaces:<br>        - test </pre><p>This tells Kyverno to apply policy only to Pods in the namespace test. We can omit namespaces, in this case the policy will have cluster-wide effect.</p><p>The policy defines two rules:</p><ol><li>First rule “gitlab-container-registry-required” makes sure that any Pod in “test” namespace is running container image from registry.gitlab.com/testcompany/* container registry. Any other Pod will be denied.</li><li>Any image from registry.gitlab.com/testcompany/* registry is subject to signature validation. Signature provider is kms: hashivault://cosign and with ignoreTlog: true we don’t search for log record in Rekor.</li></ol><p>Let’s test the policy by running the image from unstrusted registry:</p><pre>$ kubectl run test --image ubuntu<br>Error from server: admission webhook &quot;validate.kyverno.svc-fail&quot; denied the request: <br><br>resource Pod/test/test was blocked due to the following policies <br><br>container-image-policy:<br>  gitlab-container-registry-required: &#39;validation error: Image must be from registry.gitlab.com/testcompany/*<br>    container registry. rule gitlab-container-registry-required failed at path /spec/containers/0/image/&#39;</pre><p>Now let’s try the image from the trusted registry but without the signature:</p><pre>$ kubectl run test --image registry.gitlab.com/testcompany/testimage:test<br>Error from server: admission webhook &quot;mutate.kyverno.svc-fail&quot; denied the request: <br><br>resource Pod/test/test was blocked due to the following policies <br><br>container-image-policy:<br>  verify-image-signature: &#39;failed to verify image registry.gitlab.com/testcompany/testimage:test:<br>    .attestors[0].entries[0].keys: no matching signatures&#39;</pre><p>Finally let’s run the image that has the proper signature:</p><pre>$ kubectl run test --image registry.gitlab.com/testcompany/testimage:main<br>pod/test created</pre><p>It worked!</p><h3>Conclusion</h3><p>Cosign is a very powerful tool that enhances the supply chain security of your applications. Overall, Sigstore project offers several other cool products and features like:</p><ol><li>Cosign supports various encryption providers like self-managed keys, AWS KMS, Azure Key Vault, GCP KMS and even Kubernetes secrets!</li><li><a href="https://docs.sigstore.dev/verifying/verify/#keyless-verification-using-openid-connect">Keyless signing using OIDC</a> and Fulcio. It allows to sign the data with short-lived x509 certificates using OIDC provider like Google, Microsoft, Github and so on. The benefit is that you don’t have to store the encryption keys.</li><li>Sigstore has its own <a href="https://docs.sigstore.dev/policy-controller/overview/">policy controller</a> that can verify container images in Kubernetes. It has some small advantages over Kyverno, but Kyverno for me looks powerful enough and more universal.</li></ol><p>What I’m really missing in Kyverno is the ability to define custom image signature registry which is <a href="https://docs.sigstore.dev/system_config/registry_support/#specifying-registry">supported</a> by Cosign. That will allow to store the signatures in a dedicated registry and avoid creating a big mess in the main image registry. The support for it <a href="https://github.com/kyverno/kyverno/issues/10018">is planned to be added</a> in Kyverno 1.13 release.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4e2041d1310" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Change Service Network in running OpenShift 4 cluster]]></title>
            <link>https://angapov.medium.com/change-service-network-in-running-openshift-4-cluster-13d626d56963?source=rss-c669aaff473------2</link>
            <guid isPermaLink="false">https://medium.com/p/13d626d56963</guid>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[openshift-4]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[networking]]></category>
            <dc:creator><![CDATA[Vasilii Angapov]]></dc:creator>
            <pubDate>Wed, 04 Nov 2020 06:24:05 GMT</pubDate>
            <atom:updated>2020-11-04T06:28:00.444Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*M6JrBT9AOjJPnVBL_ePIoA.jpeg" /></figure><p>Use case: we want to change serviceNetwork in running OpenShift 4.5 cluster from 172.25.0.0/16 to 172.30.0.0/16 without recreating it from scratch. ServiceNetwork as it name stands is used by Services, not Pods. It is important to add that this operation is certainly disruptive and causes some amount of downtime for services depending on the size of your cluster.</p><p>There is no Red Hat’s documentation on how to that. Officially changing network configuration of running cluster is not possible, see <a href="https://access.redhat.com/solutions/4873171">https://access.redhat.com/solutions/4873171</a>.</p><p>However, changing serviceNetwork in OpenShift 4 can be done in 4 steps:</p><ol><li>Create new TLS certificate for kube-apiserver with its new clusterIP address (for service default/kubernetes). Remember that this service always uses the first IP address from serviceNetwork. In our case its IP will change from 172.25.0.1 to 172.30.0.1. We will create TLS certificate with both old and new addresses to ensure kube-apiserver will continue to be available in case something goes really wrong.</li><li>Change serviceNetwork using network operator.</li><li>Recreate ALL services to force their IP address change.</li><li>Update MachineConfigs with new DNS IP address.</li></ol><h3>Create new TLS certificate for kube-apiserver</h3><p>Export the certificate for internal OpenShift CA.</p><pre>oc -n openshift-kube-apiserver-operator get secret service-network-serving-signer -o=jsonpath=&#39;{.data.tls\.crt}&#39; | base64 -d &gt; ca.crt<br>oc -n openshift-kube-apiserver-operator get secret service-network-serving-signer -o=jsonpath=&#39;{.data.tls\.key}&#39; | base64 -d &gt; ca.key</pre><p>Export the certificate for kube-apiserver.</p><pre>oc -n openshift-kube-apiserver get secret service-network-serving-certkey -o=jsonpath=&#39;{.data.tls\.crt}&#39; | base64 -d &gt; tls.crt<br>oc -n openshift-kube-apiserver get secret service-network-serving-certkey -o=jsonpath=&#39;{.data.tls\.key}&#39; | base64 -d &gt; tls.key</pre><p>Inspect the apiserver certificate, make note of its X509v3 Subject Alternative Name field. You will need that for the next step.</p><pre>openssl x509 -text -noout -in tls.crt</pre><p>Create options file for openssl. Make sure that alt_names section corresponds to those in your apiserver certificate. Add new IP address to alt_names and req_distinguished_namesection.</p><pre>cat &gt; ssl.conf &lt;&lt;EOF<br>[ req ]<br>default_bits       = 4096<br>distinguished_name = req_distinguished_name<br>req_extensions     = req_ext<br>prompt             = no</pre><pre>[ req_distinguished_name ]<br>commonName = 172.30.0.1</pre><pre>[ req_ext ]<br>subjectAltName = @alt_names</pre><pre>[alt_names]<br>DNS.1  = kubernetes<br>DNS.2  = kubernetes.default<br>DNS.3  = kubernetes.default.svc<br>DNS.4  = kubernetes.default.svc.cluster.local<br>DNS.5  = openshift<br>DNS.6  = openshift.default<br>DNS.7  = openshift.default.svc<br>DNS.8  = openshift.default.svc.cluster.local<br>DNS.9  = 172.25.0.1<br>DNS.10 = 172.30.0.1<br>IP.1   = 172.25.0.1<br>IP.2   = 172.30.0.1<br>EOF</pre><p>Create new TLS certificate for apiserver.</p><pre>openssl genrsa -out new.key 4096<br>openssl req -new -sha256 -out new.csr -key new.key -config ssl.conf<br>openssl x509 -req -sha256 -days 3650 -in new.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out new.crt -extensions req_ext -extfile ssl.conf</pre><p>It can be seen that new certificate has the correct issuer and SANs.</p><pre>$ openssl x509 -text -noout -in new.crt | grep Issuer<br>        Issuer: OU=openshift, CN=kube-apiserver-service-network-signer<br>$ openssl x509 -text -noout -in new.crt | grep -A1 &quot;Subject Alternative Name&quot;<br>            X509v3 Subject Alternative Name:<br>                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:openshift, DNS:openshift.default, DNS:openshift.default.svc, DNS:openshift.default.svc.cluster.local, DNS:172.25.0.1, DNS:172.30.0.1, IP Address:172.25.0.1, IP Address:172.30.0.1</pre><p>Insert new certificates into apiserver secret.</p><pre>oc -n openshift-kube-apiserver patch secret service-network-serving-certkey --type=&#39;merge&#39; --patch &quot;{ \&quot;data\&quot;: { \&quot;tls.crt\&quot;: \&quot;$(base64 -w0 new.crt)\&quot;, \&quot;tls.key\&quot;: \&quot;$(base64 -w0 new.key)\&quot; } }&quot;</pre><p>Verify that apiserver is now working with new certificate (in this example my API endpoint is api.crc.testing:6443).</p><pre>$ echo | openssl s_client -connect api.crc.testing:6443 2&gt;/dev/null | grep subject<br>subject=/CN=172.30.0.1</pre><h3>Change serviceNetwork using network operator</h3><p>First we need to tell network operator that we are changing the network settings, otherwise it will decline any modifications to serviceNetwork.</p><pre>oc annotate network.operator cluster &#39;networkoperator.openshift.io/network-migration&#39;=&quot;&quot;</pre><p>Change serviceNetwork CIDR.</p><pre>oc patch network.config cluster --type=&#39;merge&#39; --patch &#39;{ &quot;spec&quot;: { &quot;serviceNetwork&quot;: [&quot;172.30.0.0/16&quot;] } }&#39;</pre><p>Network operator will restart kube-apiserver with new settings, there will be a brief downtime of kube API.</p><h3>Recreate all services</h3><p>Here comes the fun part. We need to recreate all services to force them to get new IP addresses.</p><p>Back up all services:</p><pre>oc get svc -A -o yaml --export | sed &#39;/clusterIP/d&#39; &gt; all_svc.yaml</pre><p>Delete all services.</p><pre>oc delete -f all_svc.yaml</pre><p>Wait until OpenShift operators recreate system services and then create all other application related.</p><pre>oc create -f all_svc.yaml</pre><p>Don’t bother all those “services *** already exists“ errors — it’s normal in this case.</p><p>Watch OpenShift operators reaching normal state.</p><pre>oc get co</pre><h3>Update MachineConfigs</h3><p>Node MachineConfigs need to be updated in order to update kubelet clusterDNS setting (clusterDNS is always 10th IP address in serviceNetwork).</p><pre>oc get machineconfig -o yaml | sed &#39;s/172.25.0.10/172.30.0.10/g&#39; | oc apply -f-</pre><p>Another way to achieve the same is via <a href="https://docs.openshift.com/container-platform/4.5/nodes/nodes/nodes-nodes-managing.html">KubeletConfig</a> CR.</p><p>Watch nodes are being rebooted to apply new config.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13d626d56963" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installing bare metal servers in OVH using PXE]]></title>
            <link>https://angapov.medium.com/installing-bare-metal-servers-in-ovh-using-pxe-489634232b06?source=rss-c669aaff473------2</link>
            <guid isPermaLink="false">https://medium.com/p/489634232b06</guid>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[ipxe]]></category>
            <category><![CDATA[ovh]]></category>
            <dc:creator><![CDATA[Vasilii Angapov]]></dc:creator>
            <pubDate>Mon, 20 Jul 2020 07:53:35 GMT</pubDate>
            <atom:updated>2020-07-20T07:53:35.389Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently we needed to use OpenShift 4 virtualization (<a href="https://kubevirt.io/">kube-virt</a>) lab which requires <a href="https://docs.openshift.com/container-platform/4.5/installing/installing_bare_metal/installing-bare-metal.html">installing OpenShift on bare metal servers</a>. Although it is possible to deploy OpenShift nodes manually I preferred to do it automatically using PXE boot. Another requirement was to have a dedicated private network between servers.</p><p>Public clouds such as AWS, Google Cloud and others don’t offer option to install physical server from PXE.</p><p>We chose <a href="https://www.ovh.ie/">OVH</a> as a good option. It offers more or less reasonable prices for dedicated servers, unmetered public traffic and has layer 2 private networking (so called vRack) for free. It is worth mentioning that private networking is available only for certain server types and its bandwidth also depends.</p><p>We purchased three Infra-1 servers (Intel Xeon-E 2274G 4c/8t, 64GB DDR4 2666 MHz, 2x960GB SSD NVME soft RAID, 1Gbps Internet and 2Gbps private network) for $129/month with one-time setup fee of $89.</p><p>OVH has a large number of ready-to-go server templates, however it appeared to be unclear how to install server using my own template. OVH documentation unfortunately is not very useful so I had to use google as usual.</p><p>I found that it is possible to install server using <a href="https://ipxe.org/scripting">iPXE script</a>. For that you need to use OVH API. Strangely there is no user friendly web interface for that. One way to explore OVH API is to use their Swagger-like UI: <a href="https://api.us.ovhcloud.com/console/">https://api.us.ovhcloud.com/console</a>. However, it is not convenient to configure each server manually using Swagger.</p><p>Luckily <a href="https://www.terraform.io/">Terraform</a> supports OVH provider, so it is possible to configure all servers at once using Terraform.</p><p>First thing you need to do is to create OVH application keys. These are long-term authentication tokens for OVH API similar to AWS access keys. Instructions are here: <a href="https://github.com/ovh/python-ovh#1-create-an-application">https://github.com/ovh/python-ovh#1-create-an-application</a>, but for me it worked only after:</p><pre>pip3 install -e git+<a href="https://github.com/ovh/python-ovh.git#egg=ovh">https://github.com/ovh/python-ovh.git#egg=ovh</a></pre><pre>pip3 install --no-deps ovhcli</pre><pre>ovh setup init</pre><p>After that you may test your credentials using command:</p><pre>ovh me info</pre><p>I installed VMware ESXi on first server and installed 1 service, 1 bootstrap and 1 master VM on it. Service VM included DHCP, DNS, NAT, Apache (for ignition configs), Haproxy (for OpenShift load balancers and Router). It is important to use first server’s NIC as a public network and second NIC as a private network in ESXi.</p><p>I used the following iPXE script to setup OpenShift nodes:</p><pre>#!ipxe</pre><pre>set base-url <a href="https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest">https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest</a></pre><pre>kernel ${base-url}/rhcos-4.4.3-x86_64-installer-kernel-x86_64 initrd=rhcos-4.4.3-x86_64-installer-initramfs.x86_64.img ip=dhcp rd.neednet=1 console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=nvme0n1 coreos.inst.image_url=${base-url}/rhcos-4.4.3-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=<a href="http://10.0.0.1:8080/baremetal/worker.ign">http://10.0.0.1:8080/baremetal/worker.ign</a></pre><pre>initrd ${base-url}/rhcos-4.4.3-x86_64-installer-initramfs.x86_64.img<br>boot</pre><p>Here <a href="http://10.0.0.1:8080/baremetal/worker.ign">http://10.0.0.1:8080/baremetal/worker.ign</a> is the location of worker node ignition config on service VM.</p><p>After that I used OVH API to make my servers boot from above iPXE script. The following Terraform manifest did the job for me:</p><pre>provider &quot;ovh&quot; {<br>  endpoint = &quot;ovh-us&quot;<br>}</pre><pre>variable &quot;server_names&quot; {<br>  type = list<br>  default = [<br>    &quot;nsXXXXX1.ip-147-135-97.us&quot;,<br>    &quot;nsXXXXX2.ip-147-135-97.us&quot;<br>  ]<br>}</pre><pre>data ovh_dedicated_server_boots &quot;ipxe&quot; {<br>  count        = length(var.server_names)<br>  service_name = var.server_names[count.index]<br>  boot_type    = &quot;ipxeCustomerScript&quot;<br>  depends_on = [<br>    ovh_me_ipxe_script.ocp4-worker<br>  ]<br>}</pre><pre>resource ovh_me_ipxe_script &quot;ocp4-worker&quot; {<br>  name   = &quot;ocp4-worker1&quot;<br>  script = file(&quot;${path.module}/ocp4-worker.ipxe&quot;)<br>}</pre><pre>resource ovh_dedicated_server_update &quot;server&quot; {<br>  count        = length(var.server_names)<br>  service_name = var.server_names[count.index]<br>  boot_id      = data.ovh_dedicated_server_boots.ipxe[count.index].result[length(data.ovh_dedicated_server_boots.ipxe[count.index].result) - 1]<br>  state        = &quot;ok&quot;<br>}</pre><p>Here variable “<strong>server_names</strong>” contains all the servers that you need to boot and <strong>ocp4-worker.ipxe</strong> is the script itself provided above.</p><p>After that all you need to do is to reboot you servers and they will start booting from your iPXE script.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=489634232b06" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Prometheus-operator with Minikube]]></title>
            <link>https://angapov.medium.com/using-prometheus-operator-with-minikube-9855c2b275f0?source=rss-c669aaff473------2</link>
            <guid isPermaLink="false">https://medium.com/p/9855c2b275f0</guid>
            <category><![CDATA[minikube]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[prometheus-operator]]></category>
            <category><![CDATA[prometheus]]></category>
            <dc:creator><![CDATA[Vasilii Angapov]]></dc:creator>
            <pubDate>Fri, 26 Jul 2019 21:53:40 GMT</pubDate>
            <atom:updated>2019-07-26T21:57:59.703Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q4SZ1u2zW3LYjJ_v_iA1qQ.png" /></figure><p>In this article I will describe how to install <a href="https://github.com/coreos/prometheus-operator">Prometheus-operator</a> into Minikube for evaluation and testing purposes. Minikube is a tiny flavor of Kubernetes and some of Prometheus exporters need special tuning to run under Minikube.</p><p>My Minikube version:</p><pre>$ minikube version<br>minikube version: v1.2.0</pre><p>First, let’s install Prometheus-operator using Helm:</p><pre>$ helm fetch --untar stable/prometheus-operator<br>$ cd prometheus-operator</pre><p>Now we need to adapt values.yaml for Minikube environment. First, let’s disable https scraping for kubelet:</p><pre>kubelet:<br>  enabled: true<br>  namespace: kube-system</pre><pre>  serviceMonitor:<br>    interval: &quot;&quot;<br>    https: <strong>false</strong></pre><p>This will make Prometheus alert saying “Kubelet target is down” go away. Easy fix.</p><p>The next issue is that Prometheus cannot scrape Etcd cluster’s metrics. Issue actually consists of two independent issues. First, due to security constraints Etcd cluster (although running in pod) is not available from pods network. It is only available using node (Minikube VM) IP address. So first, we must obtain Minikube VM IP:</p><pre>$ minikube ip<br>192.168.99.100</pre><p>Then we must explicitly define this endpoint in values.yaml:</p><pre>kubeEtcd:<br>  enabled: true<br>  endpoints:<br>  - 192.168.99.100</pre><p>Second issue is that Etcd listens on HTTPS with client SSL authentication. That means that we must provide Prometheus with set of certificates to be able to talk to /metrics endpoint of Etcd. Those certificates are stored on Minikube VM under /var/lib/minikube/certs/etcd directory. We must download those certificates and store them as Kubernetes secret.</p><pre>$ minikube ssh -- sudo cat /var/lib/minikube/certs/etcd/ca.crt &gt; ca.crt<br>$ minikube ssh -- sudo cat /var/lib/minikube/certs/etcd/healthcheck-client.key &gt; healthcheck-client.key<br>$ minikube ssh -- sudo cat /var/lib/minikube/certs/etcd/healthcheck-client.crt &gt; healthcheck-client.crt<br>$ kubectl create secret generic etcd-certs --from-file healthcheck-client.crt --from-file healthcheck-client.key --from-file ca.crt</pre><p>We then must specify created secret in our values.yaml :</p><pre>prometheusSpec:<br>  secrets:<br>  - etcd-certs</pre><p>And finally specify those certificates underkubeEtcd key:</p><pre>kubeEtcd:<br>  enabled: true<br>  endpoints:<br>  - 192.168.99.100</pre><pre>  service:<br>    port: 2379<br>    targetPort: 2379<br>    selector:<br>      component: etcd</pre><pre>  serviceMonitor:<br>    scheme: https<br>    insecureSkipVerify: false<br>    serverName: localhost<br>    caFile: /etc/prometheus/secrets/etcd-certs/ca.crt<br>    certFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.crt<br>    keyFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.key</pre><p>One more highly desired thing is to expose Prometheus dashboard using NodePort service:</p><pre>prometheus:<br>  service:<br>    type: NodePort</pre><p>And the last step is obviously to install Prometheus-operator with all our fixes:</p><pre>helm install . --name prometheus -f values.yaml</pre><p>After that we can go to Prometheus dashboard. To find out where it is located we may use minikube service list command. In my case it was <a href="http://192.168.99.100:30090/">http://192.168.99.100:30090</a>. So we go to Prometheus -&gt; Targets and here is the picture we may see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qRzZ62Jkx-DNcY2mniutqw.png" /></figure><p>It shows us that all the targets are up. Another way to verify is to use PromQL query:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fBe1YfW4H2iMngxtIrqe9w.png" /></figure><p>It also shows that all scrape jobs are running without issues.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9855c2b275f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quick installation of EFK stack on Kubernetes]]></title>
            <link>https://angapov.medium.com/quick-installation-of-efk-stack-on-kubernetes-5bff21ddcde0?source=rss-c669aaff473------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bff21ddcde0</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[efk]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Vasilii Angapov]]></dc:creator>
            <pubDate>Wed, 24 Oct 2018 11:34:00 GMT</pubDate>
            <atom:updated>2019-07-27T07:38:59.763Z</atom:updated>
            <content:encoded><![CDATA[<p>This solution describes how to very quickly install Elastisearch, Fluentd and Kibana on Kubernetes. It is taken from <a href="https://habr.com/post/426959">habr.com</a> and requires working Helm in cluster.</p><pre>helm install stable/elasticsearch --namespace logging --name es --set data.persistence.size=50Gi</pre><p>This by default will create 3 master pods, 2 data pods and 2 client pods of Elasticsearch. Wait for pods to be all 1/1 Running.</p><p>Then create elasticsearch-curator cronjob that ensures data retention in Elasticsearch equal to 7 days by default.</p><pre>helm install stable/elasticsearch-curator --name es-curator --namespace logging --set config.elasticsearch.hosts={&#39;es-elasticsearch-client&#39;}</pre><pre>helm install stable/fluentd-elasticsearch --namespace logging --name fluentd --set elasticsearch.host=es-elasticsearch-client</pre><p>This creates DaemonSet on all nodes for fluentd agent.</p><pre>helm install --namespace logging --set env.ELASTICSEARCH_URL=http://es-elasticsearch-client:9200,env.LOGGING_QUIET=true --name kibana stable/kibana</pre><p>This creates pods with Kibana. The only thing left if to create proper ingress and head to dashboard.</p><p>Next step is to configure Kibana. Go to dashboard and press “Index Patterns”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C9c-jE5E4ST3Ocyu.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TR1tGAXV2pXJXyuE.png" /></figure><p>Choose @timestamp.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RdQZ4tVMbHOoQV-2.png" /></figure><p>And here it works!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nx2cx4cuc--t2fGt.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bff21ddcde0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>