<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by cerberus on Medium]]></title>
        <description><![CDATA[Stories by cerberus on Medium]]></description>
        <link>https://medium.com/@cer6erus?source=rss-98d7ead8ce49------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by cerberus on Medium</title>
            <link>https://medium.com/@cer6erus?source=rss-98d7ead8ce49------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:02:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cer6erus/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Running podman on MacOs with Linux Backend]]></title>
            <link>https://cer6erus.medium.com/running-podman-on-macos-with-linux-backend-70695b09dcf6?source=rss-98d7ead8ce49------2</link>
            <guid isPermaLink="false">https://medium.com/p/70695b09dcf6</guid>
            <category><![CDATA[podman]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[fedora]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[cloud-native]]></category>
            <dc:creator><![CDATA[cerberus]]></dc:creator>
            <pubDate>Wed, 10 Jan 2024 23:57:13 GMT</pubDate>
            <atom:updated>2024-04-02T03:28:41.628Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*VnAvNTsjIWMXJa2VjScjTg.png" /></figure><h3>Problem Statement — Podman builds are bloody slow on Mac</h3><p>I’m big fan of multi-staging container builds, to make containers slim and contain what they need to run only. Unfortunately my poor Mac can not handle builds nicely, because podman runs in QEMU, as Mac doesn’t have native support of it. As result the multi-staging builds eat lots of resources, slowdown my machine and doesn’t allow me to do anything during the build. If you are lucky owner of a Linux Desktop or small server, you won’t notice this problem. But if you don’t, but do have your org’s servers accessible via SSH I do have a solution for you.</p><p>Podman <a href="https://docs.podman.io/en/latest/markdown/podman-system-connection.1.html">does support build</a>s on remote machines!</p><h3>Installation and configuration on remote build server</h3><p>Install podman on remote machine (in my case Fedora), login as a regular user, not root <br>dnf install -y podman</p><p>Enable socket for the system<br>sudo systemctl enable -now podman.socket</p><p>Enable socket for the regular user to let podman client talk with runC <br>systemctl — user enable — now podman.socket</p><h3>Configuration on Mac</h3><p>Enable remote daemon for podman<br>podman --debug system connection add --identity ~/.ssh/id_rsa -port 22 build-server build-user@192.168.0.12 <br>Let’s review the flags <br>--debug — enable debug to see what’s happening during <br>system connection add — instruction for podman to add remote system <br>--identity ~/.ssh/id_rsa — your private key to access remote system<br>--port 22 — SSH port <br>build-server — connection name <br>build-user@192.168.0.12 — username and IP of remote server</p><p>If operation was successful you will be able see connection in the list</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8TBVAgHtrTPCM-Yl14BfHQ.png" /><figcaption>“podman system connection list” output</figcaption></figure><p>To make connection default, just run command<br>podman system connection default &lt;connection_name&gt;</p><p>If you set everything correctly, let’s try to run an</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*FRX5EJwmfKVMXKHwl_H7dA.png" /></figure><h3><strong>Test the Build</strong></h3><p>Create a simple Dockerfile</p><pre><br>❯ cat Dockerfile<br>FROM busybox<br>LABEL DESCRIPTION=”Simple BusyBox to test remote builds”<br>CMD [“date”]</pre><p>Run build</p><pre>❯ podman build . -t i12<br>STEP 1/3: FROM busybox<br>STEP 2/3: LABEL DESCRIPTION=&quot;Simple BusyBox to test remote builds&quot;<br>--&gt; Using cache c4553ec53950935483c503053065e61bf7fdb38ad5502c5f5f5f274003cbd485<br>--&gt; c4553ec53950<br>STEP 3/3: CMD [&quot;date&quot;]<br>--&gt; Using cache 9c42cf168484f6b10c9c085d27e1276a4905d9901bede4770d66c72d3ca87865<br>COMMIT i12<br>--&gt; 9c42cf168484<br>Successfully tagged localhost/i12:latest<br>9c42cf168484f6b10c9c085d27e1276a4905d9901bede4770d66c72d3ca87865</pre><p>Run container</p><pre>❯ podman run --rm -ti i12<br>Wed Jan 10 23:47:22 UTC 2024</pre><p>Happy days</p><h3>Hints</h3><p>Podman remote uses scp to copy build context to the remote system. Therefore, use .dockerignore to make sure you do not copy nodejs go and python venvs, otherwise builds will take longer and contain unwilling files.</p><p>UPDATE</p><p>Instead of sockets, podman may expose TCP port. I find it useful when my I need to run latest version of podman on Ubuntu server:</p><pre>podman run -d --name podman-daemon \<br>--cap-add=sys_admin,mknod \<br>--device=/dev/fuse \<br>--security-opt label=disable \<br>-p 8877:8877 \<br>quay.io/podman/stable \<br>podman system service tcp:localhost:8877 --log-level=debug --time=0</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70695b09dcf6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Superset BI with Victoria Metrics]]></title>
            <link>https://cer6erus.medium.com/superset-bi-with-victoria-metrics-a109d3e91bc6?source=rss-98d7ead8ce49------2</link>
            <guid isPermaLink="false">https://medium.com/p/a109d3e91bc6</guid>
            <category><![CDATA[victoriametrics]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[starburst]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[superset]]></category>
            <dc:creator><![CDATA[cerberus]]></dc:creator>
            <pubDate>Thu, 23 Sep 2021 00:59:04 GMT</pubDate>
            <atom:updated>2021-09-23T00:59:04.390Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A very short note how to integrate Superset with Victoria Metrics.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5wmfRSNeHFbupVX7lVTRnw.png" /></figure><p>Superset it’s gaining popularity among users especially Business/Product Owners and people who would like to create Business Intelligence graphs. Unfortunately it does not support Prometheus protocol at the moment of writing, therefore can not be used for DevOps’ish or Telemetry BI. This can resolved with very popular in Data Scientists circles data aggregation framework— Trino. It allows to gather and transform data from different sources. Let’s see how it can help us extract data from VM and pass it into Superset for plotting and visualisation.</p><p>To start we need a few things ready:</p><p><strong>Kubernetes</strong>. I use OpenShift, therefore be mindful that some kinds might be different (for example route is equivalent of Ingress in vanila kube)</p><p><strong>Superset. </strong>Deployed via <a href="https://superset.apache.org/docs/installation/running-on-kubernetes/">Helm</a> Chart or can be deployed from Kustomize (be aware of version as Kustomize built from Helm template)</p><p><strong>Trino (aka Starburst)</strong>. Installed OpenShift Starburst Operator or Helm Operator. This is an enterprise version of Trino, I assume that Enterprise version is more stable than <a href="https://github.com/trinodb/trino">community</a> (1.3k open issues on github at the moment of this writing). It shipped with a few <a href="https://docs.starburst.io/latest/connector.html">free connectors</a> and one of them is Prometheus. Default CRD is hungry for resources (100GB memory for workers) so I updated resource requests in my manifests to make it run on smaller cluster (for demo purposes).</p><p><strong>Victoria Metrics. </strong>Installed OpenShift operator.</p><p>Assuming that Kubernetes is up and running, operators installed (and Superset had been deployed ) in vm namespace, apply Kustomize manifest</p><p>oc apply -k <a href="https://github.com/mancubus77/superset-trino-vm.git/base">https://github.com/mancubus77/superset-trino-vm.git/base</a></p><p>As result Victoria Metrics cluster and Trino/Starburst will be deployed in vm namespace. <br><strong>NB. </strong><em>Ingress for services not included for non OpenShift k8s, please expose services yourself.<br></em><strong>NB2. </strong><em>Sunburst (v.356) has a bug, what doesn’t work with Prometheus endpoints that have URI (eg: </em><a href="https://prometheus/"><em>https://prometheus</em></a><strong><em>/select/0/prometheus</em></strong><em>). Therefore you need to use Ingress + Rewrite rule to address it. Update your Victoria Metrics endpoint </em><a href="https://github.com/mancubus77/superset-trino-vm/blob/main/base/starburstEnterprise.yaml#L197-L202"><em>here</em></a><em>.</em></p><p>To play with Trino we might need a cli tool what can be installed from your favourite packet manager, I use brew , therefore for MacOS owners it’s brew install trino</p><p>Connect to Startburst coordinator via trino CLI (expose service via ingress if it’s not OpenShift)</p><p>./trino — server=coordinator-vm.apps.ocp4.rhpg.org:80</p><p>Now you can run Trino/Presto <a href="https://prestodb.io/docs/current/functions.html">SQL</a> to extract data from Victoria Metrics:</p><pre># Show al <br>trino&gt; SHOW SCHEMAS from Prometheus;<br>       Schema<br>--------------------<br> default<br> information_schema<br>(2 rows)</pre><pre>Query 20210920_125344_00004_xqx8m, FINISHED, 3 nodes<br>Splits: 36 total, 36 done (100.00%)<br>0.28 [2 rows, 35B] [7 rows/s, 123B/s]</pre><pre># Show all tables in default schema<br>trino&gt; SHOW TABLES FROM prometheus.default;<br>                                        Table<br>-------------------------------------------------------------------------------------<br> :kube_pod_info_node_count:<br> :node_memory_memavailable_bytes:sum<br> aggregator_openapi_v2_regeneration_count<br> aggregator_openapi_v2_regeneration_duration<br> aggregator_unavailable_apiservice<br> alertmanager_alerts<br> alertmanager_alerts_invalid_total<br> alertmanager_alerts_received_total<br> node_load15</pre><pre># Prepare SQL for explanation <br>trino&gt; PREPARE my_select1 FROM SELECT * FROM prometheus.default.node_load15;</pre><pre># All tables<br>trino&gt; DESCRIBE OUTPUT my_select1;<br> Column Name |  Catalog   | Schema  |    Table    |            Type             | Type Size | Aliased<br>-------------+------------+---------+-------------+-----------------------------+-----------+---------<br> labels      | prometheus | default | node_load15 | map(varchar, varchar)       |         0 | false<br> timestamp   | prometheus | default | node_load15 | timestamp(3) with time zone |         8 | false<br> value       | prometheus | default | node_load15 | double                      |         8 | false<br>(3 rows)</pre><pre>Query 20210920_125727_00011_xqx8m, FINISHED, 1 node<br>Splits: 1 total, 1 done (100.00%)<br>0.78 [0 rows, 0B] [0 rows/s, 0B/s]</pre><pre># Select all entries from table <br>trino&gt; SELECT timestamp, value, map_keys(labels) as insance  FROM prometheus.default.node_load15 where  element_at(labels, &#39;pod&#39;) = &#39;node-exporter-nnd4s&#39;;<br>          timestamp          | value |                                                 insance<br>-----------------------------+-------+-------------------------------------------------------------------------------------------------------<br> 2021-09-18 13:13:48.412 UTC | 22.74 | [__name__, namespace, service, job, instance, pod, container, endpoint, prometheus, prometheus_replica<br> 2021-09-18 13:14:03.393 UTC | 22.75 | [__name__, namespace, service, job, instance, pod, container, endpoint, prometheus, prometheus_replica<br> 2021-09-18 13:14:18.395 UTC | 23.03 | [__name__, namespace, service, job, instance, pod, container, endpoint, prometheus, prometheus_replica<br> 2021-09-18 13:14:33.394 UTC | 23.23 | [__name__, namespace, service, job, instance, pod, container, endpoint, prometheus, prometheus_replica<br> 2021-09-18 13:14:48.395 UTC | 23.24 | [__name__, namespace, service, job, instance, pod, container, endpoint, prometheus, prometheus_replica<br> 2021-09-18 13:15:03.390 UTC | 23.37 | [__name__, namespace, service, job, instance, pod, container, endpoint, prometheus, prometheus_replica</pre><pre># run aggregation functions <br>trino&gt; SELECT sum(value) FROM prometheus.default.node_load15 where  element_at(labels, &#39;pod&#39;) = &#39;node-exporter-nnd4s&#39;;<br>       _col0<br>-------------------<br> 433328.5800000004<br>(1 row)</pre><p>As it works, it’s time to integrate with Superset<br>Create new database:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*WKdfSp2ZhSMtDnB-mTsktg.png" /><figcaption>Connection string to Trino</figcaption></figure><p>Run test SQL via new Trino Database connection:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L4wo8XpHea5R7NhdH3eC0w.png" /><figcaption>Data explore</figcaption></figure><p>When data extracted, build a nice graphs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T9jvgwLqZrZbmcLWl-Wy0g.png" /><figcaption>Generated graph from VM dataset</figcaption></figure><p><strong>Conclusion</strong></p><p>Starburst/Trino is data scientist tool, but can be used to extract metrics from different sources for Superset BI. It’s written in Java, so if you thinking to use it in production, check sizing recommendations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a109d3e91bc6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ansible Tower and its ‘venv’ Lifecycle on OpenShift]]></title>
            <link>https://cer6erus.medium.com/ansible-tower-and-its-venv-lifecycle-on-openshift-7544ac97072b?source=rss-98d7ead8ce49------2</link>
            <guid isPermaLink="false">https://medium.com/p/7544ac97072b</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[tekton]]></category>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[openshift]]></category>
            <dc:creator><![CDATA[cerberus]]></dc:creator>
            <pubDate>Mon, 02 Aug 2021 13:23:43 GMT</pubDate>
            <atom:updated>2021-08-03T05:56:19.526Z</atom:updated>
            <content:encoded><![CDATA[<p>If you have been working with Ansible Tower or AWX you know what are virtual environments and how they are crucial for Ansible lifecycle. When you are part of a big organization managing virtual environments could be a pain, especially if you have a dozen of projects what require different modules and plugins. For sure Red Hat Ansible Automation Platform 2.x introduce lots of benefits thanks to execution environments (they can run as containers), but I believe it will take some time before organizations will move to AAP 2.x</p><p>The reason for this article is to show an example of how Tekton can be used to lifecycle Ansible Tower and Python virtual environments on OpenShift.</p><h3>Process overview</h3><p>As we live in the beautiful world of GitOps, therefore the best and easiest way to store our environments would be GIT (GitLab, GitHub, BitBucket, gitea or any other preferred SCM). Each time when developers update their dependencies they follow the standard PullRequest procedure. After changes merged in master branch, Git can trigger Tekton WebHook to launch a pipeline, or it can be triggered manually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1rWLkTGyqsG7v11Vj-APCw.png" /><figcaption>High-level Pipeline Overview</figcaption></figure><p><em>This is a simple version of the pipeline to demonstrate the approach, therefore there is no“testing” stage that is essential for [pre]</em><strong><em>production</em></strong><em> pipelines.</em></p><p>There are 4 steps in the pipeline:</p><p><strong>Get images and push to local.</strong> On this step tekton checkout images from Red Hat Container registry and push them to on-cluster registry. <br> <strong>Download environment.</strong> This step checkout virtual environments from a remote git repository and save them on local storage linked with Pipeline<br> <strong>Get Image; Customize image; Push Image. </strong>Task checkout image from on-cluster registry, run it in “<em>editable</em>” mode, install additional python environments via Buildah toolchain, saves(commit) image and push it to on-prem registry<br> <strong>Deploy Custom Image. </strong>In this step, Tekton downloads Ansible Installer from Red Hat Portal. Effectively Ansible Installer is Ansible playbook that accepts extra parameters via -e keys. Therefore Tekton builds Ansible commands with a few extra -e keys and runs deployment<strong>. This step requires namespace admin privileges.</strong></p><h3>Deploying pipeline</h3><p><em>I use OCP 4.6 with pipelines 1.2.3 and assume it’s all up and running.</em><br> Create a new project/namespace</p><pre>oc new-project tower</pre><p>Tekton creates a services account in each namespace, but as we use the tower installer, it needs extra privileges to create role bindings. Just for the demo, I add tekton SA to cluster-admin group. <strong><em>Please do not do it on prod cluster. Instead, create proper SA or tweak Tekton SA for your namespace.</em></strong></p><pre>oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:tower:pipeline</pre><p>Now we are good to go to deploy tekton pipeline and tasks via kustomize:</p><pre>$ oc apply -k <a href="https://github.com/mancubus77/tekton-ansible-lifecycle.git">https://github.com/mancubus77/tekton-ansible-lifecycle.git</a> -n tower<br><br>configmap/ansible-inventory unchanged<br>pipeline.tekton.dev/ansible-build unchanged<br>task.tekton.dev/ansible-lifecycle configured<br>task.tekton.dev/buildah configured</pre><p>Kustomize has a few manifests:<br><strong>configmap/ansible-inventory</strong> — config map for ansible installer. It’s Ansible inventory file that has all key values for deployment.</p><p><strong>pipeline.tekton.dev/ansible-build</strong>- Tekton pipeline to customize image and deploy Ansible on OpenShift</p><p><strong>task.tekton.dev/buildah</strong>- Tekton task to customize Red Hat Tower image and add python virtual environments</p><p><strong>task.tekton.dev/ansible-lifecycle- </strong>Tekton task to deploy Ansible on openshift via the installer</p><p>After manifest deployment, you should see pipeline in OpenShift UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6vxMRcMh0b8W0yAg3uixuA.png" /><figcaption>Ansible builder pipeline in OpenShift UI</figcaption></figure><p>The pipeline has a few parameters with default values:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d849fe50c0466b825be247bcf5ab576/href">https://medium.com/media/2d849fe50c0466b825be247bcf5ab576/href</a></iframe><h3>Running deployment</h3><p>There are a few ways to trigger Tekton pipeline. <strong>WebHook</strong> — most advanced method when it will be launched Git action like (merge or commit); <strong>Pipeline </strong><strong>UI</strong> in Openshift; Tekton CLI tool <strong>tkn </strong>(my preference)</p><p>In our example, first, we install Ansible Tower 3.8.1 and upgrade it to 3.8.3 later. To trigger first pipeline with default parameters simply run:</p><pre>tkn pipeline start ansible-build -w name=shared-workspace,volumeClaimTemplateFile=<a href="https://raw.githubusercontent.com/openshift/pipelines-tutorial/master/01_pipeline/03_persistent_volume_claim.yaml">https://raw.githubusercontent.com/openshift/pipelines-tutorial/master/01_pipeline/03_persistent_volume_claim.yaml</a></pre><p>I use “Persistent storage template” which created PV and PVC to store tekton artifacts. If all is good you should see the following output<br>Happy path output looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uUaW2s443jQQynf_X1-hRg.png" /><figcaption>Happy Path Output after Pipeline trigger</figcaption></figure><p>The result of execution can be seen in UI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jucvNTaM92UdVHs25AA9RA.png" /></figure><p>After pipeline completer, you should see Ansible Tower deployed in tower namespace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w6f51eO8LNf_QrRxTkeR1A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HzhsFQKLDUmXUsz_j_BEFA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*vH5KSlRkwHwrgY1qH4sDZA.png" /><figcaption>Ansible Tower version and new environments (Sure you need to apply Tower Subscription on the first login)</figcaption></figure><h3>Upgrading version or virtual environment</h3><p>As most of the parameters can be passed to Tekton as parameters on launch, we can easily re-run pipeline with a few arguments. As we agreed before, we are going to deploy a new version of Ansible Tower and keep the same virtual environments. Simply trigger new build:</p><pre>tkn pipeline start ansible-build \<br>-p tower-image=3.8.3 \<br>-p ansible-installer=<a href="https://releases.ansible.com/ansible-tower/setup_openshift/ansible-tower-openshift-setup-3.8.3-1.tar.gz">https://releases.ansible.com/ansible-tower/setup_openshift/ansible-tower-openshift-setup-3.8.3-1.tar.gz</a> \<br>-w name=shared-workspace,volumeClaimTemplateFile=<a href="https://raw.githubusercontent.com/openshift/pipelines-tutorial/master/01_pipeline/03_persistent_volume_claim.yaml">https://raw.githubusercontent.com/openshift/pipelines-tutorial/master/01_pipeline/03_persistent_volume_claim.yaml</a></pre><p>After pipeline completer, you should see Ansible Tower deployed in tower namespace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w6f51eO8LNf_QrRxTkeR1A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*UiNPB9kwVI4Fl3pEz405RA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*vH5KSlRkwHwrgY1qH4sDZA.png" /><figcaption>Ansible Tower version was upgraded</figcaption></figure><h3>Conclusion</h3><p>Tekton is a very powerful tool that allows not only manage the application lifecycle but also useful for infrastructural tasks in a GitOps fashion.</p><h3>Links</h3><p><strong>Tekton Source code:</strong> <a href="https://github.com/mancubus77/tekton-ansible-lifecycle">https://github.com/mancubus77/tekton-ansible-lifecycle</a><strong><br>Buildah:</strong> <a href="https://github.com/containers/buildah">https://github.com/containers/buildah</a><br><strong>Ansible Automation Platfrom:</strong> <a href="https://www.ansible.com/products/automation-platform">https://www.ansible.com/products/automation-platform</a><br><strong>Ansible environments git: </strong><a href="https://gitlab.com/skozlov1/ansible-tower-environments">https://gitlab.com/skozlov1/ansible-tower-environments</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7544ac97072b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zero Trust GitOps]]></title>
            <link>https://cer6erus.medium.com/zero-trust-gitops-d013d1daacd4?source=rss-98d7ead8ce49------2</link>
            <guid isPermaLink="false">https://medium.com/p/d013d1daacd4</guid>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gitops]]></category>
            <dc:creator><![CDATA[cerberus]]></dc:creator>
            <pubDate>Wed, 21 Apr 2021 00:05:00 GMT</pubDate>
            <atom:updated>2022-09-30T02:37:18.811Z</atom:updated>
            <content:encoded><![CDATA[<h3>Trust no1</h3><p>I love work in startups and greenfield projects, and one of the reasons is a free choice of tools and technology. It’s like a cradle for technology and innovation. A team defines comfortable and promising technology and lines of code starting to grow exponentially. Big companies a bit different story, teams need to follow guidelines and the WildWildWest approach won’t sustain. Systems and platforms must be supported by operations or DevOps teams and they can not support every new stack introduced by developer teams. If a new technology proposed to use, the team needs to train and prepare SOP (Standard Operating Procedures) to let other teams know how to support and troubleshoot the stack. There are many articles about People, Processes, Technology, and how they could help to deliver better applications, and one part of them is <strong>GitOps</strong>.</p><p>When you have one k8s cluster it’s fun and team (or teams) easily can work together on development, deployment, and operations. But when you have 100 projects, x10 clusters, and dev teams jump from one project to another, you might need to set some policies. If you contribute to OpenSource projects you might notice that before your change reaches a Merge Request phase, it goes through dozens of CI checks before the core team approve/reject it. Why? To avoid something that doesn’t follow the team’s guidelines or the way how core team operates. So CI is mature in software development, but let’s talk about CI for CD in Kubernetes.</p><h3>Use Case: Unified CI for OpenShift/Kubernetes deployments</h3><p>Hard to believe that organisations do not adopt clouds or Kubernetes to accelerate their business, improve operations and deliver new features. Kubernetes is great, but it’s not easy to operate at scale. You may find hundreds of stories about pitfalls and antipatterns, do not ignore them, they are like road rules, written from experience and P1 incidents by survivors.</p><p>Definitely, it will take some time before Kubernetes will be a “must-have” tool on the belt of software developers, but before that DevOps and DevSecOps need to help dev teams to setup environments. As we mentioned before, one project is fine, but what if we have 100 teams and a few Kubernetes clusters? If we will attach one DevOps to each team it will be expensive and very soon it will become a bottleneck. With help of DevSecOps, we can introduce the first frontier to check deployment files follow Kubernetes Standards.</p><h4>Project overview</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jtznkz0eiB7vvcl2EmhAPQ.png" /><figcaption>OpenShift Project Deployment</figcaption></figure><p>We have ArgoCD, which gets code from GitLab and deploys the project on OpenShift Cluster. This is sort of industry-standard nowadays, but how to ensure that code in Git is compliant and follows organisation standards? This is where we introduce GitLab CI for compliance and policy checks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OERUPj-d0SvsP02ZgVdZiw.png" /><figcaption>Schematic Approval Process</figcaption></figure><p>Based on the diagram above every commit to a new or existed branch triggers CI pipeline. If some manifest files not compliant with our standards, the commit will be rejected, so developers need to make changes until all checks pass. If the commit was successful, developers may trigger a merge request what will trigger another pipeline for Master(Production Branch). In other words, this concept moves the burden of manifest quality from DevOps to Developers. If you are OpenShift user, you can add an extra layer of compliance checks with ACM (Advanced Cluster Manager) Probably it’s a topic for another article.</p><h3>git Repository structure with CI</h3><p>As you noticed we need to use Git Version Control with CI capabilities. In this example, I will use public GitLab, but there are plenty of others on the market. If you would like to learn more about GitLab CI capabilities, have a look <a href="https://docs.gitlab.com/ee/ci/quick_start/">here</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*goL61uNWT4LlBE3Fw_IRiw.png" /><figcaption>.gitlab-ci.yml Repositories Dependencies</figcaption></figure><p>It’s very important to protect .gitlab-ci.yml and avoid changes by devs. To do that, create another project and add CI configuration (a reference to another project) in GitLab CI settings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*U0CXj6XaOW5b11mMUhOS_g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wrj5PPnQkc0L__LkxZFXgQ.png" /><figcaption>Master CI configuration</figcaption></figure><p>Setting above ignore CI instructions .gitlab-ci.yml and always use instructions from the referenced repository.</p><h3>CI Overview</h3><h4>Action</h4><p>Let’s meet developers Eric, Suren, and Meshva. They are great developers full-stack developers, but new to Kubernetes. DevOps created them repository for Kubernetes deployment with enabled <a href="https://gitlab.com/skozlov1/gitops-cicd-policy/-/blob/master/.gitlab-ci.yml">CI</a> and said whatever they push into repository be deployed (Keep in mind ArgoCD) on OpenShift cluster. Kubernetes cluster can support only Kustomize and HELM packaging, so deployment from simple files not possible/allow. In case deployment files have issues, the person committing changes receives an error report.</p><p><strong>First attempt.</strong> The team pushed deployment as a bunch of *.yaml files</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2a175IpgSHx2lWDrcOKfXg.png" /><figcaption>Error message as HELM nor KUSTOMIZE detected</figcaption></figure><p>What happened? CI tried to detect that the project contains kustomization.yaml or HELM chart. This check failed. So whoever triggered merge request or create a new branch receives an email notification (or slack message) with pipeline result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8nJkkHyHa1f3fczFzZvz0w.png" /><figcaption>Error, no Kustomize or Helm in the repo</figcaption></figure><p><strong>Second attempt. </strong>The team added kustomization.yaml file, and commit another change</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y9nrCDOf3aCRAE9IuxEjVg.png" /><figcaption>Error as KubeLinter and Yaml Linting failed</figcaption></figure><p>What happened? The previous error had been was resolved, but now checks fail on KubeLinter and YAML linting</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zMxFE1pXbDsgwGlLUTbAkA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fbpv0xkEKufGqtm050lJOw.png" /><figcaption>KubeLinter and YAML linter errors</figcaption></figure><p>The team needs to resolve these issues to proceed.</p><p>I stop here as the further discussion is something I leave you to try with your teams, but I hope you got an idea why we need to enable CI for GitOps.</p><p>Feel free to have a look at the source code of<br><strong>Master CI Pipeline</strong>: <a href="https://gitlab.com/skozlov1/gitops-cicd-policy">https://gitlab.com/skozlov1/gitops-cicd-policy</a><br><strong>Kustomize project example</strong>: <a href="https://gitlab.com/skozlov1/acm-gitops-kustomize">https://gitlab.com/skozlov1/acm-gitops-kustomize</a><br><strong>HELM project example</strong>: <a href="https://gitlab.com/skozlov1/acm-gitops">https://gitlab.com/skozlov1/acm-gitops</a></p><h3>Conclusion</h3><p>GitOps is a very powerful approach and can help to operate at scale with code development, operations, and DevSecOps teams. But to make it successful in your organisation it might require some effort and a deep understanding of the process. If you big organisation and maintain on-prem Kubernetes cluster, please invest in DevOps teams, let them do their works and talk to tenants and consider OpenShift/Kubernetes cluster as a product to plan lifecycle and features for your tenants.</p><p>[1] <a href="https://gitlab.com/skozlov1/gitops-cicd-policy/-/blob/master/.gitlab-ci.yml">https://gitlab.com/skozlov1/gitops-cicd-policy/-/blob/master/.gitlab-ci.yml</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d013d1daacd4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DevSecOps with Tekton and OpenShift]]></title>
            <link>https://cer6erus.medium.com/devsecops-with-tekton-and-openshift-cd95e299dbbd?source=rss-98d7ead8ce49------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd95e299dbbd</guid>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[openshift]]></category>
            <dc:creator><![CDATA[cerberus]]></dc:creator>
            <pubDate>Sun, 21 Mar 2021 23:30:46 GMT</pubDate>
            <atom:updated>2021-03-21T23:30:46.723Z</atom:updated>
            <content:encoded><![CDATA[<h3>Incoming call from… CISO</h3><p>If you are a project lead and you see an incoming call from CISO or his deputy it’s not a good sign. And also I hope you never paid BTC to bad boys who stolen private information of your customers because one of your devs forgot to disable <em>admin/admin</em> on public-facing application or used an old and vulnerable JS library. This is my one of my <a href="https://medium.com/hackernoon/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5">favorite stories</a> about how easy people can get access to sensitive data, just because devs use libraries without source code review. Devs are brave and creative, they want to write good and efficient software and try new things. If you forbid them to do that, you are slowing an innovation or can lose good people. But new tech might be good and dangerous at the same time. If you are a team lead and work with Kubernetes I assuming your teams use only clean <a href="https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image">UBI</a> images and well know libraries. But how to breed creativity/innovation and keep your job and reputation at the same time? There are many ways to do that, but today we will speak about DevSecOps.</p><h3>DevOps + Sec = Dev Sec Ops</h3><p>DevOps is not a buzzword anymore, you or your company do have them and they bring enormous benefits for the business. These warriors help to automate product delivery with CI/CD pipelines and let devs just push a code and get reports in Stash/Rocketchat/Telegram about build/deployment status. But how is about to add <strong>Sec</strong>urity bit to ensure that automation delivers secure products? That’s how <strong>DevSecOps</strong> appeared. I’m not very good in describing methodologies, so here is a <a href="https://www.redhat.com/en/topics/devops/what-is-devsecops#:~:text=DevSecOps%20means%20thinking%20about%20application,DevOps%20workflow%20from%20slowing%20down.">good explanation</a> from Red Hat what is DevSecOps.</p><h3><strong>Use Case</strong></h3><p>Let’s assume our team <strong>λ</strong> started to work on a simple application that has front end and backend deployed on OpenShift cluster. Front-end is internet-facing with an exposed HTTPS Route and a simple backend. Also let’s imagine that our company is adopting GitOps, so developers have limited access to the cluster (at least on Staging and Prod) namespaces. Therefore they can not deploy anything via kubectland oc commands. And to unify processes DevOps defined a structure of repositories that each team has to follow:</p><pre>.<br>├── app<br>│   ├── Dockerfile<br>│   ├── app.py<br>│   └── requirements.txt<br>└── kubernetes<br>    ├── base<br>    │   └── be<br>    │       ├── deployment-config.yaml<br>    │       ├── image-builder.yaml<br>    │       └── service.yaml<br>    └── kustomization.yaml</pre><p>Where <strong>app </strong>folder<strong> </strong>used<strong> </strong>for application source code and Dockerfile, and <strong>kubernetes</strong> for project manifest files compatible with Kustomize. We have 2 sub-teams for FrontEnd and Backend and they can make commits whenever they want without asking permission. In this case, minimum delivery pipeline (which is Tekton) might look like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8FYumqjhpzEyknF4gxpapQ.png" /><figcaption>FrondEnd and BackEnd pipeline</figcaption></figure><p><em>NB: This is a simple case, so we omit all integration tests in CI/CD, for sure in real life you will have more steps like unit tests and integration testing. Also in real life, we won’t have 2 parallel tasks in one pipeline</em></p><p>If you want to have a look at the pipeline to follow our further discussion, please feel free to apply it to your cluster</p><pre># Deploy simple Tekton pipeline<br>oc apply -k &quot;<a href="https://github.com/mancubus77/devsecops-demo-tekton.git/?ref=simple">https://github.com/mancubus77/devsecops-demo-tekton.git/?ref=simple</a>&quot;</pre><pre># Run task<br>tkn pipeline start build-and-deploy \ <br>-w name=sharedworkspace,volumeClaimTemplateFile=<a href="https://raw.githubusercontent.com/openshift/pipelines-tutorial/master/01_pipeline/03_persistent_volume_claim.yaml">https://raw.githubusercontent.com/openshift/pipelines-tutorial/master/01_pipeline/03_persistent_volume_claim.yaml</a></pre><p><em>I hope you had a look at repository content before :-)</em></p><p>So, we have delivery pipeline and each time devs make a commit git triggers CI/CD pipeline, and project will be built and deployed on the cluster. So far we solved a few challenges:<br>* We defined unified repository structure<br>* We are compatible with GitOps<br>* We can separate teams and let them work individually</p><h3>OK, what is about Security?</h3><p>Good question. Nothing stops our Devs from testing new technologies and libraries, but how <strong>do we know</strong> that they are secure and harmless? The answer is simple <strong>we do not</strong>, as we do not check repo nor code quality. So any mistake or vulnerability can jeopardize or compromise a project. To avoid this, let add a few steps for our delivery pipeline:</p><ul><li>Code scanning of backend code via Python <a href="https://github.com/PyCQA/bandit">Bandit</a></li><li>Code scanning of Node code with <a href="https://github.com/ajinabraham/njsscan">njsscan</a></li><li>Image scanning with <a href="https://github.com/aquasecurity/trivy">Trivy</a></li><li>Manifest scanning with <a href="https://github.com/stackrox/kube-linter">kube-linter</a></li></ul><p>Therefore, we not only automating CI/CD we also ensure that security vulnerabilities or dodgy code not being deployed in a staging/production environment. Let’s have a look at how the pipeline looks like with DevSecOps’ish approach:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ChuDK0ddlDs3so-l6UwbWQ.png" /><figcaption>DevSecOps pipeline execution</figcaption></figure><p>If you wish to have a look at new pipeline deploy updated pipeline on your cluster:</p><pre>oc apply -k <a href="https://github.com/mancubus77/devsecops-demo-tekton.git">https://github.com/mancubus77/devsecops-demo-tekton.git</a></pre><p>Let’s stop on each step and explain what we can mitigate while we are running it</p><pre>tkn pipeline start build-and-deploy \<br>-w name=shared-workspace,\<br>volumeClaimTemplateFile=<a href="https://raw.githubusercontent.com/openshift/pipelines-tutorial/master/01_pipeline/03_persistent_volume_claim.yaml">https://raw.githubusercontent.com/openshift/pipelines-tutorial/master/01_pipeline/03_persistent_volume_claim.yaml</a></pre><p><strong>Python-Scan — </strong>scan Python for hardcoded passwords or dodgy libraries. For example, my source code had a typical Flask pattern</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*OEfrvY1U4fc31NhEL-fhIw.png" /><figcaption>The issue with port binding</figcaption></figure><p>And Bandit didn’t allow me to proceed, because binding a port to all interfaces is not a good idea.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8R4UVpzNerucpBB02Va50Q.png" /><figcaption>Error message from Bandit about port binding</figcaption></figure><p><strong>Kube-Linter </strong>— a great product from StackRox, it checks manifest files and reveals configuration issues. Unfortunately, it doesn’t support OpenShift APIs, and as I use “DeploymentConfig” which is not recognized by Kube-Linter, so I bypassing the error. But please consider this tool if you do use other Kubernetes distro.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SEUYDovm71Q-pdJR5S5WJg.png" /><figcaption>Kube-lint error message</figcaption></figure><p>Kube-Linter could catch orphan services or absence of CPU/Memory limits.</p><p><strong>build-and-scan</strong>— This task has a few steps to build and scan containers on vulnerabilities. To make scanning in a true Cloud-Native way, instead of mounting the image to Trivy we use Docker side-car container with Trivy container to scan freshly built image.</p><p>The steps are:</p><ul><li>Build an image with buildah and push the image to Docker side-car container</li><li>Scan the new image with Trivy</li><li>Push image to the local OpenShift registry</li></ul><p>And our base python image from Docker Hub has a lot of surprises:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*A78WzdkO5N3XzCJEJEp3Rw.png" /><figcaption>Part of Dockerfile for Backend</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KGu9x1hxnKtcz8b1MdVXgg.png" /><figcaption>Trivy scan results for backend</figcaption></figure><p>Pretty interesting, right? This image has 19 critical vulnerabilities that will be working hand in hand with your code. Please be noted that the severity can be tuned and probably some warnings can be skipped, but it depends on your faith in the code or how convincing you might be in front of InfoSec</p><p>How is our Frontend container?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*1oI0OGJS4DjRSEhuNgd08Q.png" /><figcaption>Dockerfile frontend</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bBNumtd8rXmnW1l5cPMzGw.png" /><figcaption>Trivy scan result for frontend</figcaption></figure><p>Not better actually, 32 critical vulnerabilities.</p><p><strong>deploy-k8 — </strong>Is a service task that takes our manifest file and deploys them in our namespace. You do not need to create a new serviceaccount as it will be using one from Tekton.</p><h3>Remediation</h3><p>The best way to avoid all issues above is to use certified and clean images, for example <a href="https://catalog.redhat.com/software/containers/search">from Red Hat</a>. Let’s change Dockerfile to use Python from Red Hat UBI<br>registry.redhat.io/rhel8/python-38 <br>and trigger pipeline again to see how it change the situation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZRmHs-uQkKy4IwE9MZ-wPA.png" /><figcaption>Image scanning with Red Hat image</figcaption></figure><p>As you can see we still have some warnings but there are no critical issues, so we can review them and probably relax severity.</p><h3>Conclusion</h3><p>If you do not want to be at google news feed with the name of your company and the words [“Breach”, “Leak”, “Security”] in the same title or be blackmailed by hacker groups, you might want to consider DevSecOps practices in your organization. DevSecOps does not slow down or stop innovations but allows you to raise red flags to developers if something needs attention.</p><p><em>PS: There are many security scanners, I used Trivy, Bandit, and njsscan because we used them in our projects before. Feel free to make your own research.</em></p><h3>Resources</h3><p>[1] Tekton pipelne (master and simple branches) <a href="https://github.com/mancubus77/devsecops-demo-tekton">https://github.com/mancubus77/devsecops-demo-tekton</a><br>[2] App Frontend (NodeJS), <a href="https://github.com/mancubus77/devsecops-demo-fe">https://github.com/mancubus77/devsecops-demo-fe</a><br>[3] App Backend (Python), <a href="https://github.com/mancubus77/devsecops-demo-be">https://github.com/mancubus77/devsecops-demo-be</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd95e299dbbd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud-Native Database GitOps on OpenShift]]></title>
            <link>https://cer6erus.medium.com/cloud-native-database-gitops-on-openshift-90ba3768f320?source=rss-98d7ead8ce49------2</link>
            <guid isPermaLink="false">https://medium.com/p/90ba3768f320</guid>
            <category><![CDATA[percona]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[tekton]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[cerberus]]></dc:creator>
            <pubDate>Sat, 20 Feb 2021 12:31:09 GMT</pubDate>
            <atom:updated>2021-03-19T00:02:50.039Z</atom:updated>
            <content:encoded><![CDATA[<p>Cloud-Native Database GitOps on OpenShift</p><h3>Problem</h3><p>One of our teams decided to use centralised cloud-native database on OpenShift clusters and onboard all projects on it to avoid an issue with multiple operation points. For sure it’s not aligned with DevOps mantra “You build, you own”, but for big organisations “X-as-a-Service” is much important from Ops point of view. Usually, teams deploy databases per project and maintain them as part of their application, but developers, not DBAs and they assume that DB just works. Unfortunately, when something happened, devs have to escalate to DBAs who spend some time to understand setup and configuration. So that’s how an idea of centralised Cloud Native DB maintained by DevOps DBAs came from. The idea is fresh and considered as a prototype, so please take it with a grain of salt.</p><h3><strong>Simplified architecture</strong></h3><p>The database sits in one namespace and access to it controlled by roles and network policies. This picture illustrates this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oc1QwFrEM513dRvSzU-ylA.png" /><figcaption>Simplified architecture</figcaption></figure><p><a href="https://operatorhub.io/operator/percona-xtradb-cluster-operator">Percona</a> operator used to operate and manage MySQL database on OpenShift cluster. There are several worker nodes that have MySQL instances with fast attached persistent storage for redundancy. There are some optimisations enabled by DBAs to make it work right in cloud environments. For simplicity in this article, I will propose a simplified version with default settings just to illustrate the idea.</p><h3>Tekton</h3><p>Manage database and control schemas was selected Tekton, as it’s cloud-native and part of OpenShift 4.6 (TechPreview). If you are interested in Tekton, please have a look details here. It’s pretty cool and many teams within our organisations like it for customisation and template-driven operations. As you will see later all steps can be defined as YAML file, therefore the pipeline can be part of your (or your organisation) GitOps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5xVkF7tFfjkOHiLCbNHmaA.png" /><figcaption>Delivery pipeline</figcaption></figure><p>DB schemas, functions, and data stored in GIT. When Devs want to modify schema or create a new schema they commit a new schema to GIT and raise pull requests to DBA DevOps. After pull requests, GIT triggers a webhook to Tekton to modify schema. This example uses Ansible Runner and simple playbooks, but in reality, more complex methods like MySQL client can be used.</p><h3>Hands-on!</h3><p><em>At the moment of writing, ansible-runner having some </em><a href="https://github.com/ansible/ansible-runner/issues/636"><em>issues</em></a><em> with dependencies, therefore I switched </em><a href="https://github.com/mancubus77/tekton-db-ansible/tree/no-ansible-build"><em>GitHub</em></a><em> repository from “main” branch to “no-ansible-build”. Main branch builds ansible-runner from source, “no-ansible-build” branch uses image from Docker Hub.</em></p><p>Before start, we need to review the test scenario. We have a git repository with the following resources:<br>1. Ansible playbook to apply SQL schema and files<br>2. SQL Files<br>3. Tekton pipelines</p><p>The Tekton pipeline has 3 steps</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*IpPv9oEAMJMgjErw3Wdlfg.png" /></figure><p>build-runner-image — downloads ansible-runner and alters to installs MySQL client to work with MySQL database<br>fetch-repository — checks out code from git with ansible-playbook and SQL files <br>ansible-runner — runs ansible playbook on the database (with mounted secret)</p><p>I use OpenShift 4.6, so all manifest designed for this platform. Before start, you might need to install Percona Operator in <em>percona</em> namespace and Pipeline operator for OpenShift cluster. It’s easy to do via <a href="https://docs.openshift.com/container-platform/4.5/operators/admin/olm-adding-operators-to-cluster.html">UI or Manifest files</a>.</p><p>Assuming you have completed prerequisites, let’s create a new namespace (project) percona what will be our working namespace</p><pre>oc new-project percona</pre><p>Now let’s install a basic Percona cluster.</p><pre># Ensure you are in &quot;percona&quot; namespace<br>oc project percona</pre><pre># deploy Percona Secret <br>oc apply -f <a href="https://raw.githubusercontent.com/mancubus77/tekton-db-ansible/main/base/percona-secret.yaml">https://raw.githubusercontent.com/mancubus77/tekton-db-ansible/main/base/percona-secret.yaml</a></pre><pre># intitiate percona cluster<br>oc apply -f <a href="https://raw.githubusercontent.com/mancubus77/tekton-db-ansible/main/base/percona.yaml">https://raw.githubusercontent.com/mancubus77/tekton-db-ansible/main/base/percona.yaml</a></pre><p><em>Please be noted that my cluster has Storage Class, therefore PVCs will be created automatically. If your cluster doesn’t have SC, you might need to add PV and PVC manually</em></p><p>If you see this, cluster installed</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K2l-YdtqJIqPmpmsq_Kpqg.png" /><figcaption>Percona namespace pods</figcaption></figure><p>Let’s have a look what databases we have:</p><pre>oc rsh mysql-cluster-proxysql-0 mysql -h mysql-cluster-proxysql -uroot -p$(oc get secrets/my-cluster-secrets -o jsonpath=&#39;{.data.root}&#39;|base64 -D) -e &#39;show databases;&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GKdjpSgSAaDrzAS2CpNa7w.png" /><figcaption>Databases</figcaption></figure><p>Let’s deal with Tekton. To use Tekton from CLI you might need to install tkn utility, have a look <a href="https://github.com/tektoncd/cli">here</a> for details. <br>Tekton has2 kinds of tasks<br>“clustertask”</p><pre>$ tkn clustertask list                                                                           NAME                       DESCRIPTION              AGE<br>buildah                    Buildah task builds...   1 day ago<br>buildah-pr                 Buildah task builds...   1 day ago<br>buildah-pr-v0-16-3         Buildah task builds...   1 day ago<br>buildah-v0-16-3            Buildah task builds...   1 day ago<br>git-cli                    This task can be us...   1 day ago<br>git-clone                  These Tasks are Git...   1 day ago<br>git-clone-v0-16-3          These Tasks are Git...   1 day ago<br>helm-upgrade-from-repo     These tasks will in...   1 day ago<br>helm-upgrade-from-source   These tasks will in...   1 day ago<br>jib-maven                  This Task builds Ja...   1 day ago<br>kn                         This Task performs ...   1 day ago<br>kn-v0-16-3                 This Task performs ...   1 day ago</pre><p>and user-defined. We will come back to them shortly.</p><p>Let’s install our tasks and pipeline:</p><pre># install buildah task<br>oc apply -f <a href="https://raw.githubusercontent.com/mancubus77/tekton-db-ansible/no-ansible-build/tekton/buildah.yaml">https://raw.githubusercontent.com/mancubus77/tekton-db-ansible/no-ansible-build/tekton/buildah.yaml</a></pre><pre># install ansible-runner task <br>oc apply -f <a href="https://raw.githubusercontent.com/mancubus77/tekton-db-ansible/no-ansible-build/tekton/task-ansible-runner.yaml">https://raw.githubusercontent.com/mancubus77/tekton-db-ansible/no-ansible-build/tekton/task-ansible-runner.yaml</a></pre><pre># install a tekton pipeline<br>oc apply -f <a href="https://raw.githubusercontent.com/mancubus77/tekton-db-ansible/no-ansible-build/tekton/pipeline.yaml">https://raw.githubusercontent.com/mancubus77/tekton-db-ansible/no-ansible-build/tekton/pipeline.yaml</a></pre><p>If all was successful will see this output:</p><pre>bash-3.2$ tkn task list<br>NAME             DESCRIPTION              AGE<br>ansible-runner   Task to run Ansible...   8 hours ago<br>buildah          Buildah task builds...   8 hours ago</pre><pre>bash-3.2$ tkn pipeline list<br>NAME         AGE           LAST RUN            STARTED          DURATION   STATUS<br>migrate-db   8 hours ago   migrate-db-quf0ha   59 minutes ago   1 minute   Succeeded</pre><p>Now we ready to kick off our first task. Let’s do it with tkn</p><pre>tkn pipeline start migrate-db \<br> -w name=shared-workspace,volumeClaimTemplateFile=https://raw.githubusercontent.com/openshift/pipelines-tutorial/master/01_pipeline/03_persistent_volume_claim.yaml \<br> -p git-url=https://github.com/mancubus77/tekton-db-ansible.git \<br> -p git-revision=main</pre><p><em>Please be noted that I use PVC in my, define yours if you do not have StorageClass.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KuM5Pk7XA-dW_gdvo6aRyQ.gif" /><figcaption><em>Pipeline execution</em></figcaption></figure><p>After execution, you will be able to see a new table and entries from SQL file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zo5i4uoXFL_0bPQXDW1vHg.png" /><figcaption>New entries in the MySQL database</figcaption></figure><p>Sure we can go further and configure Tekton WebHooks on Git commit, but I leave it to you to play with.</p><h3>Conclusion</h3><p>Tekton is a very powerful cloud-native CI/CD tool, it fully templates driven and can be stored in git as source code, therefore it’s GitOps ready solution. GitOps Database is one of the simplest use cases to illustrate how it can be used in your projects. From my point of view, it’s a great replacement for Jenkins and can be fully integrated with Kubernetes/OpenShift. This moves you one step closer to NoOps model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90ba3768f320" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud Native Model Driven Telemetry Stack on OpenShift]]></title>
            <link>https://cer6erus.medium.com/cloud-native-model-driven-telemetry-stack-on-openshift-80712621f5bc?source=rss-98d7ead8ce49------2</link>
            <guid isPermaLink="false">https://medium.com/p/80712621f5bc</guid>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[telemetry]]></category>
            <category><![CDATA[telco]]></category>
            <category><![CDATA[victoriametrics]]></category>
            <dc:creator><![CDATA[cerberus]]></dc:creator>
            <pubDate>Fri, 08 Jan 2021 05:43:33 GMT</pubDate>
            <atom:updated>2021-01-08T05:43:33.387Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to measure a network?</h3><p>Modern networks monitoring is all about metrics, considering that Ops needs rapidly react on network changes or incidents Telco needs something really fast with low granularity (down to milliseconds). If you are in Telecom or Networking for a quite some time, your obvious choice to measure network health and performance would be SNMP. SNMP (Simple Network Management Protocol) <a href="https://tools.ietf.org/html/rfc1157">RFC 1157</a> was ratified in 1990 and thanks to simplicity it’s still on duty and helping to monitor networks of any scale. The main advantage of SNMP defined in first letter of the acronym (<strong>S</strong>)imple as it’s based on UDP and format of messages is pretty simple <em>(I do not consider SNMPv3 with encryption, form my point of view it’s a bit overkill as based on my experience it was killing devices with encryption offloading)</em> But considering current speeds and traffic volumes simple may not be sufficient. SNMP engine on routers/switches expose hardware (ASIC) counters based on OID and return value to SNMP client. Here is a simple example of SNMP Polling in action:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MFCnIcJJOWfQcoNKgxrXaQ.png" /></figure><p>Unfortunately, as more loaded a system less responsive it is. Sometimes you may get updates once in 30 or even 1 minute. In other words, even you will be requesting data every 10ms, you will get the same value and possibly overwhelm your device. Therefore your judgment about real throughput might be far away from real, it will be an average. Considering the nature of TCP traffic and micro-bursts you may not be aware of real throughput. If you have an interest in Carrier Grade monitoring system based on SNMP, have a look on <a href="https://getnoc.com/">NOC Project</a>. It’s an Open Source Network Monitoring and Fault management system.</p><h3>What is telemetry</h3><p><a href="https://blogs.cisco.com/developer/model-driven-telemetry-sandbox">Here is good explanation of telemetry from Cisco</a>. Long story short, telemetry is metrics streaming (pushing) from network elements to Telemetry receiver/server. Receivers may subscribe to specific data based on YANG model (by dial-in or dial-out mode). Therefore you may receive nearly real-time data or with minimum delay. As result, an operator may see what’s happening on the interface without overloading device with constant polling. Another advantage of Telemetry is reliability as it works over TCP protocol where message delivery guaranteed. Also, it supports gRPC that works over HTTP/2 and proved more efficient streaming and embedded flow control. Messaged decoded in GPB (Google ProtoBuf) format, have a look <a href="https://blogs.cisco.com/sp/streaming-telemetry-with-google-protocol-buffers">here</a> if you would like to know more about this format. Here is an example of metrics with sampling interval 500ms</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*95ZoC-qUIzsrJOR8qMj9Mw.png" /><figcaption>10 data points per 5 seconds</figcaption></figure><h3>Classic Telemetry Stack</h3><p>There are a couple of libraries to work with telemetry messages:<br><strong>Python </strong>Library <a href="https://pypi.org/project/cisco-mdt/">cisco_mdt </a><br><strong>Go</strong> Library (unfortunately, it’s not maintained and outdated) <a href="https://github.com/cisco/bigmuddy-network-telemetry-pipeline">pipeline</a><br><strong>Go</strong> Library based on gNMI <a href="https://github.com/cisco-ie/pipeline-gnmi">pipeline-gnmi</a></p><p>Below is recommended deployment stack by Cisco</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*utLkZqiaC_U7Qj1z.png" /><figcaption>© <a href="https://github.com/cisco/bigmuddy-network-telemetry-pipeline">https://github.com/cisco/bigmuddy-network-telemetry-pipeline</a></figcaption></figure><p>To implement this stack you might need roughly 11 VMs (Sure it depends on project sizing, but let’s assume we are building minimum HA solution)</p><pre>+------------+-------------------------------+-----+<br>|    role    |          description          | qty |<br>+------------+-------------------------------+-----+<br>| Pipeline   | Pipeline application consumer |   2 |<br>| Pipeline   | Pipeline application producer |   2 |<br>| Kafka      | Kafka Brokers                 |   3 |<br>| tsdb       | Time Series Database          |   2 |<br>| monitoring | Grafana and Prometheus        |   2 |<br>+------------+-------------------------------+-----+</pre><p>If your company do have infra departments who looks after your VMs, you should not have any issues. Order VMs, deploy applications, create an alert plan, write handbooks for operations and you are happy telemetry user. But who will be looking after all these applications? Who will be restarting services if they stuck and looks after OS level alarms? These are reasonable questions if you want to build a production ready solution, not just Proof Of Technology.</p><h3>Cloud Native</h3><p>We are happy to live in fun times and witnessing a new era in application development and operations (G’Day DevOps and DevNetOps). So presumably same stack can be built on top of Kubernetes/OpenShift platform. Sizing of Kubernetes/OpenShift cluster depends on a number of projects and your resources I leave you here. My OpenShift test cluster has 3 workers (8 x vCPU, 32GB, 100GB). There are many tweaks how to configure all OpenShift resources correctly and efficiently, but this article is just a high level overview.</p><h4>High level architecture of Telemetry cluster on OpenShift</h4><figure><img alt="High level diagram" src="https://cdn-images-1.medium.com/max/1024/1*N3m-On_I2jxtuPpVIACU5Q.png" /><figcaption>High Level Diagram</figcaption></figure><p>There are several key components in this solution</p><p><strong>Network Elements</strong><br> In my test environment it’s Cisco IOSXR ASR9k instance running on EVE-NG with enabled telemetry config</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*58luy0xsDR7P6tkC95KaHg.png" /><figcaption>IOXSR Router config</figcaption></figure><p><strong>Telemetry Service</strong><br> This is an entry-point from our network to OpenShift cluster. We can use MetalLB or Keepalived operator to expose dedicated floating IP. This service connected to Telegraf Service. Have a look on MetalLB/Keepalived <a href="https://cer6erus.medium.com/openshift-baremetal-ingress-for-telco-use-cases-b38d06c94cb9">here</a>.</p><p><strong>Telegraf telemetry receiver/Kafka producer<br> </strong>As we highlighted above we have plenty of options to receive and decode GPB messages. I would prefer to write my own telemetry server, but to make it simple I use <a href="https://github.com/influxdata/telegraf">Telegraf</a> with <a href="https://github.com/influxdata/telegraf/tree/master/plugins/inputs/cisco_telemetry_mdt">cisco_telemetry_mdt</a> plugin. Telegraf is a native collector for Infulx DB what supports multiple inputs and outputs modes. It’s pretty optimised for high performance and has low memory footprint. Telegraf instance has 3 PODs for load balancing and high availability. Cisco_telemetry_mdt plugin receives GPB messages from network elements, telegraf serialize messages to influx format and send to kafka topic. In kafka messages looks like this text (for sure you may send bare GBP in kafka). This Influx line protocol format:</p><pre>ifstats,interface_name=GigabitEthernet0/0/0/0,path=Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/generic-counters,source=ios,subscription=Sub1 packets_received=93818i,bytes_received=5421494i,packets_sent=105098i,bytes_sent=140044185i,multicast_packets_received=0i,broadcast_packets_received=6323i,multicast_packets_sent=0i,broadcast_packets_sent=1i,output_drops=0i,output_queue_drops=0i,input_drops=18i,input_queue_drops=0i,runt_packets_received=0i,giant_packets_received=0i,throttled_packets_received=0i,parity_packets_received=0i,unknown_protocol_packets_received=0i,input_errors=0i,crc_errors=0i,input_overruns=0i,framing_errors_received=0i,input_ignored_packets=0i,input_aborts=0i,output_errors=0i,output_underruns=0i,output_buffer_failures=0i,output_buffers_swapped_out=0i,applique=0i,resets=0i,carrier_transitions=0i,availability_flag=0i,last_data_time=1609257618i,seconds_since_last_clear_counters=0i,last_discontinuity_time=1609250418i,seconds_since_packet_received=0i,seconds_since_packet_sent=0i 1609257638844000000</pre><p><strong>Messaging cluster (Kafka)<br> </strong>Kafka cluster is represented by <a href="https://strimzi.io/">Strimzi</a> OpenShift/Kubernetes operator. Operators in OpenShift/K8s are super cool! They are looking after application lifecycle, fix issues, performing day2 operations and scale pods to handle increasing load.</p><p><strong>Telegraf telemetry consumer<br> </strong>Another telegraf instance with kafka consumer input and influx URL output. It subscribes to Kafka topic, reads messages and send them to configured URL. You may configure Victoria Metrics as influx endpoint as it do know and recognise this Influx line format version 1 and 2</p><p><strong>Victoria Metrics<br> </strong>This is the heart of our solution — Fast, cost-effective and scalable monitoring solution and time-series database. Victoria Metrics also has Kubernetes operator for deployment and lifecycle management. The clustered version of Victoria Metrics consist of 3 key components (there are more than 3 in the clustered version, check the documentation if you are interested)</p><figure><img alt="C" src="https://cdn-images-1.medium.com/max/1024/0*HIOmM3Whm8B7gOyd" /><figcaption>Victoria Metrics Clustered Version architecture ©</figcaption></figure><p><strong>&gt; VMInsert<br> </strong>Receives data in different formats (Influx(v1|2) Line Protocol , OpenTSDB HTTP and Telnet, Graphite, Prometheus) and send data to VMStorage<br><strong>&gt; VMStorage</strong><br> A storage engine to save, compress and select data points. VM has pretty high compress ratio, that allows to save data points more efficient and make a selection with low processing times. Cluster version support replication for high availability. If one node goes down we have same data in other node thanks to VMInsert and also replica<br><strong>&gt; VMSelect</strong><br> VMSelect receives MericsQL requests (same format as PromQL), send query to all connected VMStorages and de-duplicate datapoints. As you noticed same data written to all sources, so to avoid duplicates VM drop repeated datapoints. Downsampling is also coming ;-)</p><p><strong>Grafana<br></strong>Everyone knows Grafana, but OpenShift/Kubernetes has Grafana Operator, so you do need to bother with managing it.</p><h3>Fire in a hole!</h3><p>As I mentioned my sandpit is not very powerful, but let’s give it a bit of pressure. To test the stack I do have 2 message sources <a href="https://gist.github.com/7faa530444abf9c6ba086d8d39185916">synthetic</a> message producer(a) and EVE-NG environment(b). BTW to run the synthetic test I do use Red Hat CodeReady workspace on OpenShift. It allows you to make development on your cluster with web-based VS Code IDE. Unfortunately emulated devices can not go down to milliseconds and maximum what I was able to get is 10 seconds update (this depends on virtual instance or release). For test purpose, I ingest messages with maximum possible speed of message generator, what runs on the same cluster. To monitor cluster performance we use 3 dashboards:</p><ul><li><a href="https://grafana.com/grafana/dashboards/11831">Victoria Metrics</a></li><li><a href="https://grafana.com/grafana/dashboards/11285">Strimzi</a></li><li>OpenShift (native OCP monitoring)</li></ul><p>Let’s run test for 10 minutes and analyze data.</p><p>OK, let’s have a look on results.</p><p><strong>Kubernetes cluster health</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j0Dx2m1K8VjeX6J3aKKefg.png" /><figcaption>Cluster utilisation for 10 mins</figcaption></figure><p>As you may see overall utilisation over 3 projects didn’t exceed +3 cores. Most hungry was Kafka, because of lag and replication.</p><p><strong>Kafka Dashboards</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fiLCbLS5avp_Pp7X1hGGMQ.png" /><figcaption>Strimzi dashboard</figcaption></figure><p><strong>Ingestion rate:</strong> 120k messages per sec (peak)<br><strong>Consume rate:</strong> ~90k messages per sec (peak) As you may see consuming is steep, that’s because of magic of “Autoscaling” Our consumers scale up if (create one more copy) if one POD takes more than 60% of allocated resources. Therefore we started with 1 pod proceed with 4 pods and ended with 1 pod after all messages had been consumed<br><strong>Lag:</strong> This is a beauty of queues in action. All messages saved in queues before they being consumed and processed, therefore we can afford consumer lose for a while. Lag was growing because ingestion speed was higher than reading.</p><p><strong>Victoria Metrics</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tQskF-NSndvZgl34HJqqZg.png" /><figcaption>VM Utilisation</figcaption></figure><p><strong>Ingestion rate:</strong> As you may see maximum ingestion rate was ~100k cps (please be noted that it’s not VM possible maximum)<strong><br>CPU Needed: </strong>Writing is easy, reading is much more challenging as result you may see we do not more cores.</p><p>PS: Please ignore negative projection, I use 10 minutes interval view and it’s not enough to make prediction.</p><p><strong>Test metrics</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iC58cHMrMm9foAQjriMAhg.png" /><figcaption>Datapoints from 1 second of traffic</figcaption></figure><p>Grafana uses custom intervals, so we can not go under 10ms intervals in this example, but if you want to make API call to VM it can give you more precise data.</p><h3><strong>Conclusion</strong></h3><p>As a big fan of networking and cloud-native development, I see a lot’s of use cases for networking applications on top of Kubernetes. That was one of the examples just to illustrate some very cool features what Kubernetes can provide you for networking stuff. As I mentioned before there are heaps of things happening in CNF space. For example modern 5G Core is containerised and designed for Kubernetes. Big companies like Red Hat are working with the community to deliver telco functions on OpenShift like SR-IOV, DPDK, Infiniband, GPU Support, HTTP/2 gRPC and many others.</p><p>Please leave comments if you need source code for the projects. I need to create kustomize file or helm chart to share it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80712621f5bc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenShift Baremetal Ingress for Telco use cases]]></title>
            <link>https://cer6erus.medium.com/openshift-baremetal-ingress-for-telco-use-cases-b38d06c94cb9?source=rss-98d7ead8ce49------2</link>
            <guid isPermaLink="false">https://medium.com/p/b38d06c94cb9</guid>
            <category><![CDATA[baremetal-kubernetes]]></category>
            <category><![CDATA[telco]]></category>
            <category><![CDATA[kebernetes-telco]]></category>
            <category><![CDATA[ingress]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[cerberus]]></dc:creator>
            <pubDate>Wed, 06 Jan 2021 00:45:35 GMT</pubDate>
            <atom:updated>2021-01-06T00:53:07.986Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h3>Overview</h3><p>Kubernetes is great tool and it has got a niche in IT segment. Almost everyone talks about containers, pods, services, ingresses and can find a use case what can be solved with Kubernetes. Nevertheless, it’s relatively new in Telecom segment, but eventually it will be in same row with well industry recognized Platform-As-Service. Keeping in mind SDx (Software Defined Everything) it will happen very soon. Vendors, Open Source communities and Independent Software Vendors introduce concept of CNF (Cloud Native Functions) what can run in containers instead of VMs to replace VNFs and Baremetal installations. Kubernetes community is working hard to make it happen and develop more and more functions to make the platform efficient, performance and reliable as VIM (like Open Stack or VMWare). <br> I was working on cloud native Telemetry stack for even driven telemetry and faced a couple of challenges with OpenShift, so let’s speak about them and how to solve them. In particularly let’s speak about how traffic gets into the cluster from network elements. Kubernetes can fairly easy manage and manipulate HTTP with routes and ingress. Let’s talk about L4 TCP/UDP traffic what is common in telecom.</p><h4>Why would we need not HTTP traffic?</h4><p>Before start looking on details let’s understand why do we need get into cluster via non-HTTP traffic? We have many use cases in Telco and the simplest one is network monitoring. If we want to build cloud native high available system to measure a health of the network we need to allow access to the cluster for a bunch of non-HTTP protocols like SNMP Traps (over UDP), BGP (over TCP) for BGP Monitoring and topology tools, Telemetry (over TCP or gRPC/HTTP2) for event driven telemetry and of course &lt;put your favorite use case here&gt;.</p><h3>How to traffic can get to cluster</h3><p>If you are familiar with Kubernetes you should know a few methods how traffic getting into the cluster. Let’s put aside Ingress Controllers and speak about Services and it types. You may stop me here and object, but bear with for a while. We intentionally do not considering Ingress, because it’s something behind load-balancer and our discussion is around getting into the cluster in HA manner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*VAbORTUTFihQcAEAGoGleQ.png" /><figcaption>Entry points</figcaption></figure><p>As you see from picture above, Kubernetes will take care about traffic when it will reach ingress, but it’s not managing Global Server Load Balancer (unless you are on cloud) Meanwhile Kubernetes service allows to expose Pods port to the cluster or outside world. So what do we have here?</p><h4>ClusterIP</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*_Bgj-AVaOl2mWGxWz82fZQ.png" /><figcaption>ClusterIP</figcaption></figure><p>This is a basic method to expose pod’s port and IP. Pod’s IP (assigned by OVN) and PORT will be available within cluster via FQDN (Fully Qualified Domain Name) e.g. cluster-kafka-brokers.kafka.svc.cluster.local:9092<em> or </em>&lt;<strong>service-name</strong>&gt;.&lt;<strong>namespace-name</strong>&gt;.svc.cluster.local:&lt;<strong>port</strong>&gt; or simple &lt;pod-ip&gt;:&lt;port&gt; Within same namespace it can be called cluster-kafka-brokers<em> or </em>&lt;service_name&gt;<em>. </em>It is not possible connect to pods from outside of the cluster. This useful method to link service together but not helpful for our Telecom Application.</p><h4>NodePort</h4><p>Node port is getting us closer to connection with our cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*pjJXU4vmnskFep2Jdcwxzw.png" /></figure><p>Now we have IP and Port to establish TCP/UDP connection! But we need to be aware about a couple of caveats of Kubernetes design.<br>NodeIP is iptables entry what maps PORT to Service, therefore:<br>Ports can be only within range 30,000~32,767 (by Kubernetes Design)<br>Port will be exposed to all worker nodes in the cluster, so to access it you need to know worker IP address. In case you have baremetal installation of OpenShift, presumably you have HAProxy/NGINX or something more cool like F5 in front of your cluster. This entity forwards traffic according Red Hat’s <a href="https://www.openshift.com/blog/haproxy-highly-available-keepalived">recommendations</a> and to get to any of your nodes you might need to access your cluster. First issue that you might notice that recommended settings for HAProxy doesn’t forward custom ports (30,000~32,767), therefore even you try to resolve your cluster via <strong>WhatEvenYouWantPutHere</strong>.apps.<strong>&lt;YOUR-CLUSTER-NAME&gt;</strong>.<strong>&lt;YOUR-DOMAIN-NAME&gt;</strong>:30000<em> </em>it won’t go through default HAProxy configuration. So if you prefer to use NodeIP method you might need to update your HAProxy configuration for each application or even make mapping of &lt;YOU LIKE PORT&gt; to &lt;K8s LIKES PORT&gt; It’s good, but not what we are after</p><h4>LoadBalancer</h4><p>How is about to let Kubernetes update port mappings and nodes automatically and keep it actual despite application lifecycle? Sounds good right? You can get it, but this is available only for cloud installation (what we Telco do not have because we need to be OnPrem (on premises). For sure you can use various CIS (Custom Ingress Services) like <a href="https://clouddocs.f5.com/containers/v2/kubernetes/">F5 BIG-IP</a> platform it will do same what AWS/AZURE/GCP do and it will be integrated with your infra and GSLB/SLB. OK, but we are telco and that means we want cool stuff with more control and less vendor lock-in (Sorry F5). So far I would like to explore 2 projects within this post, unfortunately both of them not supported by Red Hat, but I believe it’s matter of time when it will be part of Red Hat OpenShift.</p><h4>Keepalived Operator</h4><p><a href="https://github.com/redhat-cop/keepalived-operator">Keepalived Operator</a> is Kubernetes operator of well know and acknowledged application within IT industry application called <a href="https://www.keepalived.org/">Keepalived</a>. Long story short, you getting one or several VIP (Virtual IP addresses or Floating IP) which can be assigned to elected node via VRRP (Virtual Router Redundancy Protocol). In case main node node become unavailable, VRRP detects it and reallocate VIP to another active node. VRRP is essential between workers as they will be negotiating who should carry IP. So, in case we would like to spin up BGP Client on your Kubernetes cluster what will accept connections from routers you might need to do following steps:</p><ul><li>Define a pool of IP addresses (sure thing within same CIDR of your worker nodes)</li><li>Deploy “keepalived operator” on your OpenShift/Kuberneres cluster.</li><li>Deploy BGP Speaker (ExaBGP, goBGP, Bird, Quagga … on K8S cluster) and expose service with “<a href="https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer">Load Balancer</a>” type and “<a href="https://kubernetes.io/docs/concepts/services-networking/service/#external-ips">External IP</a>” address.</li><li>Operator will do other things for you.</li><li>Setup BGP session with your cluster</li><li>Enjoy BGP full view copy on your Kubernetes cluster (Why? May be to create another looking glass or monitoring anomaly of your AS or load them in Graph Database)</li></ul><p><strong>Cool things about Keepalived operator:</strong></p><ul><li>It’s Kubernetes operator, therefore it’s taking care about application lifecycle</li><li>It can be assigned to particular node (Hello VRF!)</li><li>It supports cloud native monitoring</li><li>It can co-exist with OpenShift native services (be aware about support)</li></ul><p>Please be noted that keepalived operator doesn’t provide pure load balancing capabilities like keepalived virtual server, only High Availability, therefore all traffic to service will go through one node.</p><h4>MetalLB</h4><p><a href="https://metallb.universe.tf/">MetalLB</a> is new and very promising project of cloud native load balancing with support of VIP (Virtual IP) and BGP ECMP (💜💘♥). This beast supports 2 configuration methods Layer2 and BGP. Please be noted project is “Beta”.</p><p><strong>Layer2</strong><br>This method uses same concept with Keepalived, with a couple of differences in realisation. For example MetalLB L2 uses <a href="https://github.com/hashicorp/memberlist">memerlist</a> based on <a href="https://en.wikipedia.org/wiki/Gossip_protocol">gossip</a> not VRRP like keepalived, but overall concept is nearly the same.</p><p><strong>BGP<br></strong>Now we are talking! BGP is well know and beloved protocol for Telco and everyone respect it. If you are not familiar with BGP, have a look <a href="https://www.kentik.com/blog/bgp-routing-tutorial-series-part-1/">here</a>. So MetalLB can establish BGP session with BGP router and advertise virtual IP to your network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*eOHDodfkr8Y9JAFq9TAYwQ.png" /><figcaption>ECMP with MetalLB</figcaption></figure><p>Also MetalLB supports ECMP, therefore traffic will be speaded across all BGP Speakers in cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eF0xi5-SXAiLjqFkzGoOng.png" /><figcaption>source: <a href="https://metallb.universe.tf/configuration/">https://metallb.universe.tf/configuration/</a></figcaption></figure><p><strong>Cool things about MetalLB:</strong></p><ul><li>Two modes L2 and BGP</li><li>Supports Ingress</li><li>Pin L2/BGP to worker nodes</li><li>Compatible with CNI (Container Network Interface)</li><li>Load Balancing with ECMP</li><li>Almost unlimited number of L2 routers (VRRP has limit of 255)</li></ul><h4>Conclusion</h4><p>With help of methods above you easily define an IP address or FQDN with one of floatingIPs in a pool what can be used by all your network elements to send SNMP Traps or configured as destination address for Event Driven Telemetry. MetalLB or Keepalived take care about IP availability and Kubernetes make sure that service is resilient and available</p><p>OpenShift provides new capabilities for Telco and Open Source community is moving forward to make it happen. OpenShift and Kubernetes are dominating in clouds, but in next few years it will be another standard for Telco Applications. <br> In terms of load-balancing MetalLB looks very promising from my point of view and allow to make real cool stuff, meanwhile Keepalived is mature and acknowledged in the community.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b38d06c94cb9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>