<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sudhir Mohanty (DevOps &amp; DevSecOps Engineer) on Medium]]></title>
        <description><![CDATA[Stories by Sudhir Mohanty (DevOps &amp; DevSecOps Engineer) on Medium]]></description>
        <link>https://medium.com/@email2smohanty?source=rss-31e146e30744------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FoN1aYSFwRpILqiXzp8_Sg.jpeg</url>
            <title>Stories by Sudhir Mohanty (DevOps &amp;amp; DevSecOps Engineer) on Medium</title>
            <link>https://medium.com/@email2smohanty?source=rss-31e146e30744------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 17:13:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@email2smohanty/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What Happens When an Operator Bundle Gets Too Large?]]></title>
            <link>https://medium.com/@email2smohanty/what-happens-when-an-operator-bundle-gets-too-large-94e2aebc8746?source=rss-31e146e30744------2</link>
            <guid isPermaLink="false">https://medium.com/p/94e2aebc8746</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[openshift-operators]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Sudhir Mohanty (DevOps & DevSecOps Engineer)]]></dc:creator>
            <pubDate>Sat, 29 Aug 2026 17:35:10 GMT</pubDate>
            <atom:updated>2026-08-29T17:35:10.016Z</atom:updated>
            <content:encoded><![CDATA[<p>A small change in an Operator repository can sometimes cause a much bigger problem later in the release pipeline.</p><p>I recently came across this while working with a large Operator project where multiple squads contribute Operators and microservices. These components are eventually assembled into a <strong>parent Operator</strong>, whose bundle and catalog are used for installation.</p><p>The failure was not detected when the individual component was built.</p><p>It was detected much later, during Operator installation, when <strong>bundle unpacking failed because the generated installation data had become too large</strong>.</p><p>The investigation was challenging because the parent Operator was built from contributions coming from multiple squads.</p><p>In this post, I’ll explain what happened, why identifying the root cause was difficult, and what practical CI guardrails can be introduced at both the <strong>individual component level</strong> and the <strong>parent Operator level</strong>.</p><h3>The Scenario</h3><p>Our build flow roughly looked like this:</p><pre>Squad A ── Operator A ──┐<br>                       │<br>Squad B ── Operator B ──┤<br>                       │<br>Squad C ── Service C ───┤<br>                       │<br>Squad D ── Operator D ──┤<br>                       ▼<br>                Parent Operator<br>                       │<br>                       ▼<br>                 Bundle / FBC<br>                       │<br>                       ▼<br>                  Catalog<br>                       │<br>                       ▼<br>                 Installation</pre><p>Each squad owns different components and can deliver multiple Operators and microservices.</p><p>The parent Operator brings these components together and produces the final Operator bundle and catalog.</p><p>This means the team that owns the parent Operator may not immediately know which individual component caused a significant increase in the final bundle.</p><h3>What Actually Happened?</h3><p>In our case, the installation started failing during <strong>bundle unpacking</strong>.</p><p>The important observation was:</p><blockquote><em>The Operator build itself had completed, but the problem was discovered when the resulting Operator was being installed.</em></blockquote><p>The investigation eventually showed that a <strong>new Custom Resource (CR) with a relatively large definition</strong> had been added as part of one of the component changes.</p><p>That additional content increased the size of the final Operator bundle/installation data enough to hit the size limitation.</p><p>The simplified flow was:</p><pre>New large CR added<br>       │<br>       ▼<br>Individual component build succeeds<br>       │<br>       ▼<br>Parent Operator consumes component<br>       │<br>       ▼<br>Final bundle becomes larger<br>       │<br>       ▼<br>Catalog generated successfully<br>       │<br>       ▼<br>Operator installation<br>       │<br>       ▼<br>Bundle unpacking fails</pre><p>The frustrating part was that <strong>nothing obviously failed at the point where the large CR was introduced</strong>.</p><h3>Why Was Finding the Problem Difficult?</h3><p>In a small project, this could be relatively easy.</p><p>You might have:</p><pre>Operator<br> ├── CR<br> ├── CRD<br> └── CSV</pre><p>and immediately know which change caused the size increase.</p><p>Our project was different.</p><p>Multiple squads contribute to the final product:</p><pre>Parent Operator<br>                       │<br>       ┌───────────────┼───────────────┐<br>       │               │               │<br>    Squad A          Squad B         Squad C<br>       │               │               │<br>   Operator A      Operator B      Service C<br>       │               │               │<br>    changes          changes          changes</pre><p>A release could contain changes from many squads.</p><p>Therefore, when the parent Operator installation failed, the first question was:</p><blockquote><strong><em>Which component introduced the additional size?</em></strong></blockquote><p>Looking only at the final bundle wasn’t enough to make ownership obvious.</p><h3>The Guardrail Idea</h3><p>This led to an important idea:</p><blockquote><strong><em>Don’t wait until the parent Operator installation to discover bundle-size growth. Detect significant size increases earlier in the component pipelines, and then perform a final validation at the parent Operator level.</em></strong></blockquote><p>There are really two useful guardrail levels:</p><pre>Component Pipelines<br>                         │<br>             ┌───────────┼───────────┐<br>             ▼           ▼           ▼<br>          Operator     Service     Operator<br>             │           │           │<br>             └───────────┼───────────┘<br>                         │<br>                         ▼<br>                  Parent Operator<br>                         │<br>                         ▼<br>                 Final Bundle/FBC<br>                         │<br>                         ▼<br>                    Installation</pre><p>The first level helps identify <strong>who introduced the growth</strong>.</p><p>The second level verifies <strong>whether the final assembled product is still within acceptable limits</strong>.</p><h3>Guardrail 1: Individual Operator/Microservice Pipelines</h3><p>Every squad-owned component can add a lightweight size check to its own build pipeline.</p><p>The goal isn’t necessarily to predict the final InstallPlan exactly.</p><p>The goal is to detect:</p><ul><li>New large resources</li><li>Significant size increases</li><li>Unexpected bundle growth</li></ul><p>For example, suppose an Operator currently produces:</p><pre>Component bundle:<br>900 KB</pre><p>A change introduces a new CR:</p><pre>Component bundle:<br>1.5 MB</pre><p>The pipeline can report:</p><pre>Previous: 900 KB<br>Current:  1.5 MB<br>Increase: +600 KB</pre><p>That doesn’t automatically mean the change should fail.</p><p>Instead, it gives the owning squad an opportunity to review the change.</p><h3>Detect New Large Files</h3><p>One simple and practical check is to compare the files/resources between the previous and current build.</p><p>For example:</p><pre>Previous build</pre><pre>CR-A.yaml    20 KB<br>CR-B.yaml    30 KB<br>CRD-A.yaml   80 KB</pre><p>Current build:</p><pre>CR-A.yaml    20 KB<br>CR-B.yaml    30 KB<br>CR-C.yaml   500 KB   ← New<br>CRD-A.yaml   80 KB</pre><p>The pipeline can report:</p><pre>New resource detected:<br>CR-C.yaml</pre><pre>Size:<br>500 KB</pre><p>This is much more actionable than discovering the problem later from an installation failure.</p><h3>Track Size Delta</h3><p>Another useful check is to track the size of the component’s generated output.</p><p>For example:</p><pre>Component: operator-a</pre><pre>Previous build: 1.2 MB<br>Current build:  1.3 MB<br>Delta:          +100 KB</pre><p>If the increase crosses a project-defined threshold, the pipeline can:</p><pre>PASS</pre><p>or:</p><pre>WARNING / REVIEW REQUIRED</pre><p>depending on how the team wants to implement the policy.</p><p>The important part is that the threshold should be <strong>defined by the project based on observed data</strong>, rather than assuming that one universal value is appropriate.</p><h3>Make the Check Visible in CI</h3><p>A useful implementation would expose the information directly in the pipeline output.</p><p>For example:</p><pre>Component Size Report<br>--------------------------------<br>Previous size : 1.20 MB<br>Current size  : 1.80 MB<br>Increase      : +600 KB</pre><pre>New resources:<br>  large-cr.yaml : 500 KB</pre><pre>Result:<br>  REVIEW REQUIRED</pre><p>This makes the component owner aware of the change before the artifact reaches the parent Operator.</p><h3>Guardrail 2: Parent Operator Pipeline</h3><p>The individual component checks solve only part of the problem.</p><p>The parent Operator is still the final point where everything comes together:</p><pre>Operator A ──┐<br>Operator B ──┤<br>Service C ───┤<br>Operator D ──┤<br>              ▼<br>       Parent Operator<br>              │<br>              ▼<br>       Final Bundle<br>              │<br>              ▼<br>          Catalog</pre><p>Therefore, the parent pipeline should have a second, stronger guardrail.</p><p>This is where the <strong>actual final bundle used for installation</strong> should be inspected.</p><h3>Final Bundle Size Check</h3><p>The parent pipeline can calculate the size of the generated bundle before publishing the catalog.</p><p>For example:</p><pre>Parent Operator Bundle</pre><pre>Previous release:  4.8 MB<br>Current release:   5.6 MB<br>Increase:          +800 KB</pre><p>If there is an unexpected increase, the pipeline can flag it before the catalog reaches the installation environment.</p><p>This is valuable because the parent pipeline has visibility into the <strong>complete assembled artifact</strong>.</p><h3>Identify Which Component Caused the Increase</h3><p>This is where the two-level approach becomes particularly useful.</p><p>Suppose the parent bundle increased by:</p><pre>+800 KB</pre><p>The parent pipeline can compare the components contributing to the bundle:</p><pre>Component                Size Delta</pre><pre>Operator A                +20 KB<br>Operator B                +50 KB<br>Service C                +680 KB   ← Major contributor<br>Operator D                +50 KB<br>                         --------<br>Total                    +800 KB</pre><p>Now the parent pipeline can identify:</p><blockquote><em>Service C is responsible for most of the increase.</em></blockquote><p>The owning squad can then investigate the changes in that component.</p><p>This is much more useful than simply reporting:</p><pre>Bundle too large</pre><h3>A Better Build Flow</h3><p>With these guardrails, the release process can look like this:</p><pre>Squad Pipelines<br>─────────────────────────────</pre><pre>Operator A ── Size Check ──┐<br>Operator B ── Size Check ──┤<br>Service C  ── Size Check ──┤<br>Operator D ── Size Check ──┘<br>                            │<br>                            ▼<br>                    Parent Operator<br>                            │<br>                            ▼<br>                    Build Bundle<br>                            │<br>                            ▼<br>                 Final Size / Delta Check<br>                            │<br>                     ┌──────┴──────┐<br>                     │             │<br>                    PASS          FAIL<br>                     │             │<br>                     ▼             ▼<br>                  Catalog       Investigate<br>                     │<br>                     ▼<br>                Installation</pre><p>This creates two opportunities to catch the problem.</p><h3>Why Both Levels Are Needed</h3><p>It may be tempting to put the entire check only in the parent pipeline.</p><p>But that still leaves an ownership problem.</p><p>If the parent pipeline reports:</p><pre>Bundle increased by 1 MB</pre><p>someone still needs to determine which squad caused it.</p><p>Component-level checks provide <strong>early ownership and visibility</strong>.</p><p>The parent-level check provides <strong>final protection</strong>.</p><p>So they solve different problems:</p><p>LevelMain PurposeComponent pipelineDetect and report changes introduced by that componentParent pipelineValidate the final assembled artifactInstallationFinal runtime validation</p><p>The goal is to move detection as far left as practical.</p><h3>What Should Be Measured?</h3><p>The exact metrics can depend on the project, but a useful starting point is:</p><h3>At component level</h3><p>Track:</p><pre>Generated artifact size<br>New resource count<br>Size of newly added resources<br>Size delta from previous build</pre><h3>At parent level</h3><p>Track:</p><pre>Final bundle size<br>Bundle size delta<br>Large/new resources<br>Component-wise contribution</pre><p>The key is to keep the checks focused on <strong>observable build artifacts</strong> rather than trying to predict every possible installation behavior.</p><h3>Don’t Start With a Hard Failure</h3><p>One lesson from this type of problem is that immediately failing every build that increases in size may create unnecessary friction.</p><p>For example:</p><pre>+5 KB</pre><p>and:</p><pre>+800 KB</pre><p>shouldn’t necessarily receive the same treatment.</p><p>A practical rollout could start with reporting:</p><pre>INFO<br>WARNING</pre><p>and establish what normal growth looks like.</p><p>After enough data is collected, the project can introduce thresholds for builds that require review or should fail.</p><h3>What the Parent Pipeline Can Tell Us</h3><p>A particularly useful improvement would be a size report attached to the parent build.</p><p>Something like:</p><pre>Parent Operator Bundle Report<br>================================</pre><pre>Previous: 4.8 MB<br>Current : 5.6 MB<br>Delta   : +800 KB</pre><pre>Largest changes<br>--------------------------------<br>Service C<br>  +680 KB<br>  New: large-cr.yaml</pre><pre>Operator B<br>  +70 KB</pre><pre>Operator A<br>  +30 KB</pre><pre>Status: REVIEW REQUIRED</pre><p>This turns a difficult investigation into a much more direct one.</p><p>Instead of starting with:</p><pre>Installation failed.<br>Where did the extra size come from?</pre><p>the team starts with:</p><pre>Service C introduced +680 KB.<br>Let&#39;s inspect that change.</pre><h3>The Important Distinction</h3><p>There are two different questions that are easy to mix up:</p><h3>Question 1</h3><blockquote><em>Which component caused the bundle to grow?</em></blockquote><p>Component-level size tracking helps answer this.</p><h3>Question 2</h3><blockquote><em>Is the final Operator artifact safe to install?</em></blockquote><p>The parent Operator pipeline needs to answer this.</p><p>That’s why a single guardrail isn’t enough in a multi-team Operator project.</p><h3>Lessons Learned</h3><p>The main lesson from this incident was not simply:</p><blockquote><em>“Don’t add large CRs.”</em></blockquote><p>The more useful lesson was:</p><blockquote><strong><em>When many independently built components are assembled into a single Operator, size ownership and final artifact validation both matter.</em></strong></blockquote><p>A component can pass its own build successfully while still contributing to a problem that only becomes visible after everything is assembled.</p><p>The solution is therefore not just a larger build machine or a manual investigation after installation fails.</p><p>It’s better observability throughout the build pipeline:</p><pre>Component change<br>      ↓<br>Component size visibility<br>      ↓<br>Parent assembly<br>      ↓<br>Final bundle visibility<br>      ↓<br>Installation</pre><h3>Final Takeaways</h3><p>If your Operator is assembled from contributions across multiple squads, consider introducing two layers of guardrails:</p><p><strong>At each Operator/microservice pipeline:</strong></p><ul><li>Track generated artifact size.</li><li>Report significant size increases.</li><li>Identify newly added large resources.</li><li>Make the size change visible to the owning squad.</li></ul><p><strong>At the parent Operator pipeline:</strong></p><ul><li>Measure the final bundle size.</li><li>Compare it with the previous release.</li><li>Identify the largest contributors to the size delta.</li><li>Validate the final artifact before publishing it for installation.</li></ul><p>This won’t eliminate every possible installation failure, but it can move an issue that is currently discovered here:</p><pre>Operator installation<br>        ↓<br>Bundle unpacking failure</pre><p>to a much earlier point:</p><pre>Component build<br>        ↓<br>Size increase detected<br>        ↓<br>Squad investigates</pre><p>And that is the real value of a build guardrail: <strong>finding the problem closer to where it was introduced, while the context and ownership are still clear.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94e2aebc8746" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding and Fixing OPM “Multiple Channel Heads” Errors]]></title>
            <link>https://medium.com/@email2smohanty/understanding-and-fixing-opm-multiple-channel-heads-errors-782137b79d66?source=rss-31e146e30744------2</link>
            <guid isPermaLink="false">https://medium.com/p/782137b79d66</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[openshift-operators]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Sudhir Mohanty (DevOps & DevSecOps Engineer)]]></dc:creator>
            <pubDate>Sat, 29 Aug 2026 15:51:25 GMT</pubDate>
            <atom:updated>2026-08-29T15:51:25.896Z</atom:updated>
            <content:encoded><![CDATA[<p>While working with Operator Lifecycle Manager (OLM) and File-Based Catalogs (FBC), I recently came across an opm validate error that initially looked confusing:</p><pre>invalid index:<br>└── invalid package &quot;my-operator&quot;:<br>    └── invalid channel &quot;v2.0&quot;:<br>        └── multiple channel heads found in graph:<br>            my-operator.v2.12.2,<br>            my-operator.v2.14.0-20260820075739</pre><p>At first, the error suggested that there were two channel heads. But the interesting part was that the Operator releases were actually supposed to form a simple sequential upgrade path.</p><p>The real problem turned out to be an <strong>incorrect </strong><strong>replaces value in the channel definition</strong>, which caused the upgrade graph to become disconnected.</p><p>In this article, I’ll walk through the scenario, explain what “multiple channel heads” means, show how the graph became invalid, and how correcting a single replaces value resolved the issue.</p><h3>1. Our Operator Release Sequence</h3><p>Let’s start with the actual release sequence.</p><p>Our Operator releases followed this progression:</p><pre>2.12.0<br>   ↓<br>2.12.1<br>   ↓<br>2.12.2<br>   ↓<br>2.13.0<br>   ↓<br>2.14.0</pre><p>Each release normally replaces the previous release.</p><p>So the expected upgrade graph was:</p><pre>2.12.0 → 2.12.1 → 2.12.2 → 2.13.0 → 2.14.0</pre><p>However, there was an important detail around 2.14.0.</p><p>The 2.14.0 release was still under development and had not yet been GA&#39;ed. Therefore, its build identifier included a timestamp.</p><p>For example:</p><pre>2.14.0-20260820075739</pre><p>So at that point, the graph effectively looked like:</p><pre>2.12.0<br>   ↓<br>2.12.1<br>   ↓<br>2.12.2<br>   ↓<br>2.13.0<br>   ↓<br>2.14.0-20260820075739</pre><p>Everything was working as expected.</p><h3>2. The Problem Appeared During the 2.13.1 Release</h3><p>The problem surfaced later when 2.13.1 was being released as a GA version.</p><p>The intended release sequence was now:</p><pre>2.12.0<br>   ↓<br>2.12.1<br>   ↓<br>2.12.2<br>   ↓<br>2.13.0<br>   ↓<br>2.13.1</pre><p>We added the FBC for the 2.13.1 release.</p><p>After that, opm validate started failing with:</p><pre>invalid index:<br>└── invalid package &quot;my-operator&quot;:<br>    └── invalid channel &quot;v2.0&quot;:<br>        └── multiple channel heads found in graph:<br>            my-operator.v2.12.2,<br>            my-operator.v2.14.0-20260820075739</pre><p>This was confusing because 2.12.2 and 2.14.0 were not supposed to be two independent heads.</p><p>There should have been a valid upgrade path between the releases.</p><p>So the next question was:</p><blockquote><em>Why does OPM think </em><em>2.12.2 and </em><em>2.14.0-20260820075739 are both channel heads?</em></blockquote><h3>3. Understanding the Channel Graph</h3><p>To understand this error, it helps to think of an OLM channel as a directed graph.</p><p>For our releases, the expected graph was:</p><pre>Channel: v2.0</pre><pre>2.12.0<br>   │<br>   ▼<br>2.12.1<br>   │<br>   ▼<br>2.12.2<br>   │<br>   ▼<br>2.13.0<br>   │<br>   ▼<br>2.14.0-20260820075739</pre><p>The latest version at the end of the graph is the channel head:</p><pre>2.12.0 → 2.12.1 → 2.12.2 → 2.13.0 → 2.14.0<br>                                             ↑<br>                                           HEAD</pre><p>This is a valid graph because every release is connected to the previous one.</p><p>The connection is established through fields such as replaces.</p><h3>4. What Went Wrong?</h3><p>The actual root cause was introduced while adding the 2.13.1 release entry to the channel file.</p><p>Our release sequence was:</p><pre>2.12.0<br>   ↓<br>2.12.1<br>   ↓<br>2.12.2<br>   ↓<br>2.13.0<br>   ↓<br>2.13.1</pre><p>Before adding 2.13.1, the 2.13.0 entry correctly pointed to 2.12.2:</p><pre>name: my-operator.v2.13.0<br>replaces: my-operator.v2.12.2</pre><p>However, while updating the channel file for the 2.13.1 release, the replaces value of the existing 2.13.0 entry was accidentally changed:</p><pre>name: my-operator.v2.13.0<br>replaces: my-operator.v2.12.3</pre><p>The problem was that <strong>2.12.3 was never released and therefore did not exist in the channel graph</strong>.</p><p>Our actual releases were:</p><pre>2.12.0<br>2.12.1<br>2.12.2<br>2.13.0<br>2.13.1</pre><p>There was no:</p><pre>2.12.3</pre><p>As a result, the upgrade relationship between 2.12.2 and 2.13.0 was broken.</p><p>The important point here is:</p><blockquote><strong><em>Adding </em></strong><strong><em>2.13.1 was not the root cause. The root cause was accidentally modifying the existing </em></strong><strong><em>2.13.0 </em></strong><strong><em>replaces value to point to a non-existent version while adding the </em></strong><strong><em>2.13.1 entry.</em></strong></blockquote><h3>5. How Does OLM Know Which Version Replaces Which?</h3><p>The upgrade graph needs relationships describing how bundles are connected.</p><p>For our release sequence, the intended relationships were:</p><pre>2.12.0<br>   ↓<br>2.12.1<br>   ↓<br>2.12.2<br>   ↓<br>2.13.0<br>   ↓<br>2.13.1</pre><p>Conceptually, the channel entries establish relationships such as:</p><pre>2.12.1 replaces 2.12.0<br>2.12.2 replaces 2.12.1<br>2.13.0 replaces 2.12.2<br>2.13.1 replaces 2.13.0</pre><p>The important relationship in our incident was:</p><pre>2.13.0 replaces 2.12.2</pre><p>But after the accidental change, it became:</p><pre>2.13.0 replaces 2.12.3</pre><p>Since 2.12.3 did not exist, the graph could no longer connect 2.12.2 to 2.13.0.</p><h3>6. What Does “Multiple Channel Heads” Mean?</h3><p>A channel represents an upgrade graph, not simply a list of Operator versions.</p><p>Our intended graph was:</p><pre>2.12.0 → 2.12.1 → 2.12.2 → 2.13.0 → 2.13.1<br>                                             ↑<br>                                           HEAD</pre><p>There is one continuous path and therefore one channel head.</p><p>After the incorrect replaces modification, the graph became effectively disconnected:</p><pre>2.12.0 → 2.12.1 → 2.12.2<br>                       ↑<br>                     HEAD<br></pre><pre>2.13.0 → 2.12.3 ❌<br>   ↓<br>2.13.1<br>   ↑<br> HEAD</pre><p>Because 2.12.3 did not exist, OPM could not use that relationship to connect 2.13.0 back to 2.12.2.</p><p>It therefore saw two independent parts of the graph, resulting in the error:</p><pre>multiple channel heads found in graph:<br>my-operator.v2.12.2,<br>my-operator.v2.14.0-20260820075739</pre><p>The exact reported heads can initially look surprising, but the important thing is that OPM is reporting the endpoints of the disconnected graph.</p><h3>7. Why Did Adding 2.13.1 Expose the Problem?</h3><p>This is where the incident became particularly interesting.</p><p>The problem appeared when we added the FBC entry for the 2.13.1 GA release.</p><p>However, <strong>2.13.1 itself was not the problem</strong>.</p><p>While adding the new 2.13.1 entry to the channel file, we accidentally modified the existing 2.13.0 entry.</p><p>The original entry was correct:</p><pre>name: my-operator.v2.13.0<br>replaces: my-operator.v2.12.2</pre><p>But during the update, it was accidentally changed to:</p><pre>name: my-operator.v2.13.0<br>replaces: my-operator.v2.12.3</pre><p>The intended graph was:</p><pre>2.12.0<br>   ↓<br>2.12.1<br>   ↓<br>2.12.2<br>   ↓<br>2.13.0<br>   ↓<br>2.13.1</pre><p>But after the accidental change:</p><pre>2.12.0<br>   ↓<br>2.12.1<br>   ↓<br>2.12.2<br></pre><pre>2.13.0<br>   ↓<br>2.12.3 ❌</pre><p>There was no 2.12.3 bundle in the catalog because it was <strong>not a released version</strong>.</p><p>This effectively disconnected the 2.13.0 branch from the 2.12.2 branch.</p><p>The problem became visible when the updated catalog was validated:</p><pre>opm validate &lt;fbc-directory&gt;</pre><p>and OPM reported:</p><pre>multiple channel heads found in graph:<br>my-operator.v2.12.2,<br>my-operator.v2.14.0-20260820075739</pre><p>So the important distinction is:</p><pre>Adding 2.13.1<br>       │<br>       └──&gt; Channel file was modified<br>                    │<br>                    └──&gt; 2.13.0 replaces value accidentally changed<br>                                      │<br>                                      ▼<br>                              2.12.3 ❌<br>                         (does not exist)<br>                                      │<br>                                      ▼<br>                              Graph disconnected<br>                                      │<br>                                      ▼<br>                           Multiple channel heads</pre><p>Once we corrected the replaces value back to:</p><pre>replaces: my-operator.v2.12.2</pre><p>the graph was restored.</p><h3>8. How We Debugged It</h3><p>The first step was to run:</p><pre>opm validate &lt;fbc-directory&gt;</pre><p>The error provided three important pieces of information:</p><pre>Package:<br>my-operator</pre><pre>Channel:<br>v2.0</pre><pre>Heads:<br>my-operator.v2.12.2<br>my-operator.v2.14.0-20260820075739</pre><p>Rather than assuming that one of these versions was invalid, we traced the upgrade relationships in the channel file.</p><p>We compared the actual release sequence:</p><pre>2.12.0<br>2.12.1<br>2.12.2<br>2.13.0<br>2.13.1</pre><p>with the replaces chain.</p><p>The unexpected relationship was:</p><pre>2.13.0<br>   ↓<br>2.12.3</pre><p>We then checked whether 2.12.3 existed in the catalog.</p><p>It did not.</p><p>That was the key clue.</p><p>The correct relationship was:</p><pre>2.13.0<br>   ↓<br>2.12.2</pre><p>After changing it back, the graph became continuous again.</p><h3>9. The Fix</h3><p>The fix was simply to restore the correct replaces value for 2.13.0.</p><h3>Incorrect</h3><pre>name: my-operator.v2.13.0<br>replaces: my-operator.v2.12.3</pre><p>2.12.3 was not a released version and did not exist in the graph.</p><h3>Correct</h3><pre>name: my-operator.v2.13.0<br>replaces: my-operator.v2.12.2</pre><p>The graph was then restored to:</p><pre>2.12.0<br>   ↓<br>2.12.1<br>   ↓<br>2.12.2<br>   ↓<br>2.13.0<br>   ↓<br>2.13.1</pre><p>The ongoing 2.14.0 build was also correctly connected according to its upgrade relationship:</p><pre>2.12.0 → 2.12.1 → 2.12.2 → 2.13.0 → 2.14.0-20260820075739</pre><p>After correcting the FBC:</p><pre>opm validate &lt;fbc-directory&gt;</pre><p>passed successfully.</p><h3>10. The Key Lesson: replaces Must Point to the Correct Existing Bundle</h3><p>The most important lesson from this issue is that the replaces field should be treated as an important part of the upgrade graph, not just another field in the channel YAML.</p><p>For example:</p><pre>name: my-operator.v2.13.0<br>replaces: my-operator.v2.12.2</pre><p>means that the graph can establish:</p><pre>2.12.2<br>   ↓<br>2.13.0</pre><p>But if we accidentally specify:</p><pre>replaces: my-operator.v2.12.3</pre><p>while 2.12.3 doesn&#39;t exist, the expected connection cannot be established.</p><p>In our case, this was particularly easy to miss because 2.12.3 looked like a perfectly reasonable version number—it simply wasn&#39;t an actual released version.</p><p>This is an important lesson for manually maintained or automatically modified channel files:</p><blockquote><strong><em>Don’t infer the previous version from the version number alone. Verify that the referenced bundle actually exists in the channel graph.</em></strong></blockquote><h3>11. Version Sequence vs. Upgrade Graph</h3><p>Another important distinction is that version numbers themselves don’t define the upgrade graph.</p><p>Our versions were:</p><pre>2.12.0<br>2.12.1<br>2.12.2<br>2.13.0<br>2.13.1</pre><p>From the version numbers, it’s obvious that 2.13.0 follows 2.12.2.</p><p>But OLM relies on the relationships defined in the catalog to construct the upgrade graph.</p><p>So having:</p><pre>2.12.2<br>2.13.0<br>2.13.1</pre><p>doesn’t automatically guarantee:</p><pre>2.12.2 → 2.13.0 → 2.13.1</pre><p>The catalog needs to explicitly represent those relationships.</p><p>In our case, the version sequence was correct, but the relationship was accidentally changed to:</p><pre>2.13.0 → 2.12.3</pre><p>That mismatch between the <strong>actual release sequence</strong> and the <strong>declared upgrade relationship</strong> was the root cause.</p><h3>12. A Better Debugging Approach</h3><p>Looking back at this issue, a useful debugging approach for multiple channel heads errors is to reconstruct the graph from the channel file.</p><p>Start with the error:</p><pre>multiple channel heads found in graph:<br>A, B</pre><p>Then work backwards.</p><h3>Step 1 — Identify the affected channel</h3><pre>Package: my-operator<br>Channel: v2.0</pre><h3>Step 2 — Identify the reported heads</h3><pre>my-operator.v2.12.2<br>my-operator.v2.14.0-20260820075739</pre><h3>Step 3 — List the expected release sequence</h3><p>For example:</p><pre>2.12.0<br>2.12.1<br>2.12.2<br>2.13.0<br>2.13.1<br>2.14.0-20260820075739</pre><h3>Step 4 — Trace the replaces relationships</h3><p>Build the graph manually:</p><pre>2.12.1 → 2.12.0<br>2.12.2 → 2.12.1<br>2.13.0 → ???<br>2.13.1 → 2.13.0<br>2.14.0 → 2.13.1</pre><p>The missing or suspicious relationship is now much easier to spot.</p><h3>Step 5 — Verify every replaces target exists</h3><p>In our case:</p><pre>2.13.0 → 2.12.3 ❌</pre><p>Immediately stood out because:</p><pre>2.12.3</pre><p>wasn’t a released version.</p><h3>Step 6 — Correct the relationship</h3><pre>2.13.0 → 2.12.2 ✓</pre><h3>Step 7 — Validate again</h3><pre>opm validate &lt;fbc-directory&gt;</pre><p>This graph-based approach was much more useful than simply looking at the version numbers.</p><h3>13. Preventing This Type of Problem</h3><p>This incident also highlights an opportunity to improve the release process.</p><p>When adding a new release to a channel file, especially when manually editing an existing channel definition, it’s easy to accidentally modify an existing entry.</p><p>A CI validation step can help catch this before the catalog is published.</p><p>For example:</p><pre>New Release<br>     │<br>     ▼<br>Update Channel File<br>     │<br>     ▼<br>Validate Release Relationships<br>     │<br>     ▼<br>opm validate<br>     │<br>   ┌─┴─┐<br>   │   │<br> PASS FAIL<br>   │   │<br>   ▼   ▼<br>Publish  Fix Graph</pre><p>Some useful checks would be:</p><ul><li>Does every replaces target exist?</li><li>Does the channel have exactly one expected head?</li><li>Does the new bundle connect to the previous release?</li><li>Were existing channel relationships unintentionally modified?</li><li>Does the generated FBC pass opm validate?</li></ul><p>For automated release pipelines, these checks can prevent a small channel-file mistake from reaching downstream catalog generation or installation.</p><h3>14. Final Takeaways</h3><p>The multiple channel heads error can look like a version conflict, but it is better understood as an <strong>upgrade graph consistency problem</strong>.</p><p>In our case, the intended graph was:</p><pre>2.12.0 → 2.12.1 → 2.12.2 → 2.13.0 → 2.13.1</pre><p>While adding the 2.13.1 release entry, the existing 2.13.0 entry was accidentally changed from:</p><pre>replaces: my-operator.v2.12.2</pre><p>to:</p><pre>replaces: my-operator.v2.12.3</pre><p>Since 2.12.3 was <strong>not a released version and did not exist in the graph</strong>, the upgrade path was broken.</p><p>The resulting disconnected graph caused OPM to detect multiple channel heads.</p><p>The fix was simply:</p><pre>replaces: my-operator.v2.12.2</pre><p>which restored the expected graph.</p><p>The main lesson I took from this incident is:</p><blockquote><strong><em>When debugging OPM’s “multiple channel heads” error, don’t focus only on the versions reported in the error. Trace the </em></strong><strong><em>replaces relationships and verify that every referenced version actually exists in the channel graph.</em></strong></blockquote><p>A single incorrect replaces value can be enough to disconnect an otherwise valid Operator upgrade graph.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=782137b79d66" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Securing Containerized Applications: A Defense-in-Depth Approach with Real-World Examples]]></title>
            <link>https://medium.com/@email2smohanty/securing-containerized-applications-a-defense-in-depth-approach-with-real-world-examples-fb697be811e0?source=rss-31e146e30744------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb697be811e0</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kubernetes-security]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[container-security]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Sudhir Mohanty (DevOps & DevSecOps Engineer)]]></dc:creator>
            <pubDate>Wed, 24 Dec 2025 15:27:52 GMT</pubDate>
            <atom:updated>2025-12-24T15:27:52.074Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Containerization has transformed how modern applications are built, shipped, and deployed. Technologies like Docker and Kubernetes enable faster delivery, scalability, and consistency across environments. However, this flexibility also introduces new security challenges.</p><p>Security in containerized applications cannot rely on a single control. Instead, it must be applied at <strong>multiple levels</strong>, following a <em>defense-in-depth</em> strategy. This blog explores key security measures across different layers of a containerized application, supported by real-world examples.</p><h3>1. Application Code Level Security</h3><p>Security starts at the source — your application code.</p><h4>Best Practices</h4><ul><li>Validate and sanitize all user inputs</li><li>Follow secure coding standards</li><li>Avoid hard coding credentials</li><li>Keep third-party libraries up to date</li></ul><h4>Example</h4><p>Hard coding secrets is a common mistake:</p><pre>db_password = &quot;admin123&quot;</pre><p>A more secure approach:</p><pre>db_password = os.environ.get(&quot;DB_PASSWORD&quot;)</pre><p>This ensures sensitive data is injected securely at runtime.</p><h3>2. Container Image Security</h3><p>A secure application can still be compromised if the container image is insecure.</p><h4>Best Practices</h4><ul><li>Use minimal and trusted base images (Alpine, Distroless)</li><li>Remove unnecessary packages</li><li>Scan images for vulnerabilities</li><li>Avoid running containers as root</li><li>Do not embed credentials or sensitive values in Dockerfiles</li></ul><h4>Example</h4><p>Instead of using a full OS image:</p><pre>FROM ubuntu</pre><p>Use a minimal and secure image:</p><pre>FROM node:18-alpine<br>USER node</pre><p>Tools like <strong>Twistlock</strong> or <strong>A</strong>qua can automatically scan images for known vulnerabilities.</p><h3>3. Secrets Management</h3><p>Managing secrets securely is critical in containerized environments.</p><h4>Best Practices</h4><ul><li>Never store secrets in source code or images</li><li>Use secret management tools</li><li>Rotate secrets periodically</li></ul><h4>Example</h4><p>Using Kubernetes Secrets:</p><pre>env:<br>- name: API_KEY<br>  valueFrom:<br>    secretKeyRef:<br>      name: api-secret<br>      key: api_key</pre><p>This keeps sensitive data out of the container image and source control.</p><h3>4. Container Runtime Security</h3><p>Runtime security limits the damage if a container is compromised.</p><h4>Best Practices</h4><ul><li>Run containers as non-root users</li><li>Restrict privilege escalation</li><li>Apply seccomp, AppArmor, or SELinux profiles</li><li>Define CPU and memory limits</li></ul><h4>Example (Kubernetes)</h4><pre>securityContext:<br>  runAsNonRoot: true<br>  allowPrivilegeEscalation: false</pre><p>This prevents attackers from gaining elevated access.</p><h3>5. Orchestration-Level Security (Kubernetes)</h3><p>Orchestration platforms must be secured to prevent cluster-wide risks.</p><h4>Best Practices</h4><ul><li>Use RBAC to control access</li><li>Apply network policies</li><li>Enforce Pod Security Standards</li><li>Secure Kubernetes API access</li></ul><h4>Example</h4><p>A network policy allowing only frontend-to-backend communication reduces lateral movement within the cluster.</p><h3>6. CI/CD Pipeline Security</h3><p>Security checks should be integrated early in the development lifecycle (<em>shift-left security</em>).</p><h4>Best Practices</h4><ul><li>Perform static code analysis (SAST)</li><li>Scan container images during builds</li><li>Enforce security gates (e.g., BVT enforcement)</li><li>Sign container images</li></ul><h4>Example: Image Scanning Using Twistlock</h4><p>You can integrate <strong>Twistlock </strong>into your CI/CD pipeline to scan container images and fail builds when critical vulnerabilities are detected.</p><pre>twistcli images scan \<br>  --address https://&lt;twistlock-console-url&gt; \<br>  --user &lt;username&gt; \<br>  --password &lt;password&gt; \<br>  myapp:${BUILD_TAG}</pre><p>This ensures that images with high or critical vulnerabilities do not progress further in the pipeline.</p><h4>Signing Container Images:</h4><p>Signing container images helps verify image integrity and ensures that only trusted images are deployed to production environments.</p><p>To understand <strong>why code signing is important and how it works</strong>, refer to my earlier blog post here:</p><p><a href="https://medium.com/@email2smohanty/code-signing-demystified-why-it-matters-for-modern-software-security-b992657e413e">What is Code Signing &amp; Why It Matters for Modern Software Security</a></p><h3>7. Host and Infrastructure Security</h3><p>Containers share the host OS, making host security essential.</p><h4>Best Practices</h4><ul><li>Harden the host operating system</li><li>Regularly patch the OS and container runtime</li><li>Restrict SSH access</li><li>Use minimal container-focused OS distributions</li></ul><h4>Example</h4><p>Using container-optimized operating systems like <strong>Bottlerocket</strong> reduces the attack surface.</p><h3>8. Monitoring and Logging</h3><p>Visibility is key to detecting and responding to security incidents.</p><h4>Best Practices</h4><ul><li>Centralized logging</li><li>Runtime threat detection</li><li>Alerts for abnormal behavior</li></ul><h4>Example</h4><p>Tools like <strong>Falco</strong> can detect suspicious actions such as unexpected shell execution inside a container.</p><h3>Conclusion</h3><p>Securing containerized applications requires a layered approach that spans from development to production. By applying security controls at the code, image, runtime, orchestration, and infrastructure levels, organizations can significantly reduce risk and improve resilience.</p><p>Container security is not a one-time task — it is an ongoing process that evolves alongside your applications and platforms.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb697be811e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Script Execution on Linux with Cron: Setup, Verification, and Debugging]]></title>
            <link>https://medium.com/@email2smohanty/automating-script-execution-on-linux-with-cron-setup-verification-and-debugging-16457d921669?source=rss-31e146e30744------2</link>
            <guid isPermaLink="false">https://medium.com/p/16457d921669</guid>
            <category><![CDATA[scheduling]]></category>
            <category><![CDATA[crontab]]></category>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Sudhir Mohanty (DevOps & DevSecOps Engineer)]]></dc:creator>
            <pubDate>Thu, 14 Aug 2025 06:54:10 GMT</pubDate>
            <atom:updated>2025-08-14T06:54:10.012Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Cron</strong> is a powerful and widely used scheduling utility in <strong>Linux</strong> systems. It allows users to automate the execution of scripts or commands at specified times or intervals — making it ideal for tasks like backups, monitoring, data processing, and system maintenance.</p><p>In this guide, we’ll walk through:</p><ol><li>How to schedule a script using cron on Linux</li><li>How to verify that it’s running correctly</li><li>How to debug when the script doesn’t run as expected</li></ol><h3>Step 1: Prepare the Script</h3><p>Ensure your script is executable and located at a stable path:</p><pre>chmod +x /home/username/scripts/myscript.sh</pre><p>Add logging to track execution:</p><pre>#!/bin/bash<br>echo &quot;Script started at $(date)&quot; &gt;&gt; /home/username/scripts/myscript.log<br><br># Your actual script logic here<br><br>echo &quot;Script ended at $(date)&quot; &gt;&gt; /home/username/scripts/myscript.log</pre><h3>Step 2: Schedule the Script Using Cron</h3><p>Edit your crontab:</p><pre>crontab -e</pre><p>Add a cron expression for your desired schedule. For example, to run your script every day at <strong>11:25 AM (system local time)</strong>:</p><pre>25 11 * * * /home/username/scripts/myscript.sh &gt;&gt; /home/username/scripts/myscript.log 2&gt;&amp;1</pre><h4>Important:</h4><p>Cron expressions are interpreted based on the <strong>Linux system’s local timezone</strong>. You can check the current timezone using:</p><pre>timedatectl</pre><p>Look for the line:</p><blockquote>Local time: Wed 2025–08–13 23:06:17 PDT</blockquote><p>If your system uses UTC or a different timezone than your desired schedule, adjust the cron expression accordingly.</p><h3>Step 3: Verify That Cron Is Running</h3><h4>1. View Your Current Cron Jobs</h4><pre>crontab -l</pre><h4>2. Ensure Cron Daemon Is Running</h4><p>On most modern Linux distributions:</p><pre>sudo systemctl status crond</pre><p>If it’s not active, start and enable it:</p><pre>sudo systemctl start crond<br>sudo systemctl enable crond</pre><h3>Step 4: Debugging Cron Issues</h3><p>If your script doesn’t appear to be running, here’s a structured checklist to troubleshoot it:</p><h4>A. Test with a Simple Cron Job</h4><p>Edit your crontab and add:</p><pre>*/2 * * * * echo &quot;Cron test at $(date)&quot; &gt;&gt; /tmp/cron-test.log</pre><p>Check after a few minutes:</p><pre>cat /tmp/cron-test.log</pre><p>If there’s no output, cron may not be working properly.</p><h4>B. Check Cron Logs</h4><p>On RHEL/CentOS/Fedora:</p><pre>grep CRON /var/log/cron</pre><p>You should see entries like:</p><pre>Aug 13 11:25:01 hostname CROND[12345]: (username) CMD (/path/to/script.sh)</pre><p>This confirms cron <strong>attempted</strong> to run your job.</p><h4>C. Inspect Script Output Log</h4><p>If your cron job is redirecting output like this:</p><pre>&gt;&gt; /home/username/scripts/myscript.log 2&gt;&amp;1</pre><p>Check the log:</p><pre>cat /home/username/scripts/myscript.log</pre><p>You should see timestamps and execution details. If not, the script likely didn’t execute.</p><h4>D. Use Absolute Paths in Scripts</h4><p>Cron uses a <strong>minimal environment</strong>. Always use full paths for all commands.</p><p>Instead of:</p><pre>python3 script.py</pre><p>Use:</p><pre>/usr/bin/python3 /home/username/scripts/script.py</pre><p>Use which to find full paths:</p><pre>which python3<br>which curl</pre><h4>E. Check File Permissions</h4><p>Ensure your script is:</p><ul><li>Executable: chmod +x script.sh</li><li>Owned by the correct user</li><li>Not restricted by SELinux or AppArmor (see below)</li></ul><h4>F. Confirm Environment and PATH</h4><p>Cron does not load your shell environment. If your script relies on environment variables or PATH, set them explicitly at the top of the crontab:</p><pre>PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin</pre><h4>G. SELinux / AppArmor Restrictions (Advanced)</h4><p>On RHEL systems with SELinux:</p><pre>sestatus</pre><p>If enabled, look for audit denials:</p><pre>sudo ausearch -m avc -ts recent</pre><p>If cron is blocked by SELinux, you may need to update the script’s context or permissions accordingly.</p><h3>Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QRXQnKI0ehsNHjrQtRyIVw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16457d921669" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secure Code Signing at Scale with GaraSign]]></title>
            <link>https://medium.com/devsecops-ai/secure-code-signing-at-scale-with-garasign-how-it-works-and-why-we-use-it-71fc16f55e39?source=rss-31e146e30744------2</link>
            <guid isPermaLink="false">https://medium.com/p/71fc16f55e39</guid>
            <category><![CDATA[image-signing]]></category>
            <category><![CDATA[codesign]]></category>
            <category><![CDATA[software-security]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Sudhir Mohanty (DevOps & DevSecOps Engineer)]]></dc:creator>
            <pubDate>Wed, 06 Aug 2025 01:21:24 GMT</pubDate>
            <atom:updated>2025-08-12T15:10:25.887Z</atom:updated>
            <content:encoded><![CDATA[<p>This post explains how the GaraSign platform is used in a real-world code signing workflow — with focus on GPG-based image signing using skopeo, centralized key management, and client certificate authentication.</p><p>Modern DevSecOps pipelines need to balance <strong>security, control, and automation</strong>. Code signing is essential, but traditional approaches often involve developers or CI systems accessing private keys — increasing the risk of key leakage.</p><p>To address this, we use <strong>GaraSign</strong> — a centralized, HSM-backed signing platform that enables secure code signing without ever exposing private keys.</p><h4>Why GaraSign?</h4><p>GaraSign acts as a <strong>code-signing proxy</strong>, mediating all signing requests via a controlled, authenticated interface. It provides:</p><ul><li><strong>Centralized signing key control</strong> (keys never leave the HSM)</li><li><strong>Client certificate-based access control</strong></li><li><strong>Support for various signing tools</strong>, including GPG, X.509, Authenticode, and more</li><li><strong>Logging and auditability</strong> of all signing actions</li></ul><h4>Components in GaraSign-Based Signing Setup</h4><ol><li><strong>Client Certificate</strong>: Each signing client is authenticated via a TLS client certificate issued by GaraSign.</li><li><strong>GPG Key</strong>: Used for signing artifacts like container images (via skopeo) — the GPG key lives only inside the HSM.</li><li><strong>X.509 Signing Certificate</strong>: May be used for higher-trust scenarios like Windows or macOS code signing.</li><li><strong>GaraSign Server</strong>: Handles requests, enforces policies, communicates with HSM, and logs everything.</li><li><strong>HSM (Hardware Security Module)</strong>: Holds the actual private keys used for signing. GaraSign never lets the raw key leave the HSM.</li></ol><h4>Signing Flow: Image Signing with skopeo + GaraSign</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wAPuiv5H74jz4lYRI6iaJw.png" /></figure><p>Here’s how it works in pipeline:</p><ol><li>CI system or developer triggers an image signing operation using <a href="https://github.com/containers/skopeo">skopeo</a>.</li><li>skopeo uses a preconfigured <strong>GPG key fingerprint</strong>, pointing to a key managed inside GaraSign.</li><li>The signing request is sent via the GaraSign client, which authenticates using a <strong>client certificate</strong>.</li><li>GaraSign validates the request, routes it to the correct GPG key in the HSM, and returns the <strong>detached signature</strong>.</li><li>The signature is attached to the container image, which is then pushed to image registry.</li></ol><p>No system ever touches the private key — only GaraSign interacts with it securely.</p><h4>Key Advantages We Gain</h4><ul><li><strong>Zero key leakage risk</strong>: No private keys stored on developer machines or build agents</li><li><strong>Granular access control</strong>: Who can sign, what they can sign, and when</li><li><strong>Audit trails</strong>: Every signature operation is logged for forensics and compliance</li><li><strong>Flexibility</strong>: Same system can sign GPG artifacts, Docker images, code packages, and more</li></ul><h4>Bonus: Using Fingerprints vs Key IDs</h4><p>Skopeo can refer to the GPG key using its full fingerprint rather than the shorter key ID — improving accuracy and avoiding ambiguity, especially when multiple keys share similar short IDs.</p><pre>skopeo copy \<br>  --sign-by=&quot;&lt;GPG fingerprint&gt;&quot; \<br>  docker-daemon:image:latest \<br>  docker://registry/image:tag</pre><p>This ensures the signature is made using the exact key we trust — managed securely inside GaraSign and stored in an HSM.</p><h3>In Summary</h3><p>GaraSign simplifies secure code signing at scale. By abstracting away key handling and centralizing policy enforcement, it helps us achieve <strong>strong software integrity guarantees</strong> without slowing down delivery or compromising security.</p><p>If your organization is dealing with the complexity of managing multiple signing keys, signing methods, and compliance requirements — GaraSign is well worth exploring.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71fc16f55e39" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devsecops-ai/secure-code-signing-at-scale-with-garasign-how-it-works-and-why-we-use-it-71fc16f55e39">Secure Code Signing at Scale with GaraSign</a> was originally published in <a href="https://medium.com/devsecops-ai">DevSecOps &amp; AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Code Signing & Why It Matters for Modern Software Security]]></title>
            <link>https://medium.com/@email2smohanty/code-signing-demystified-why-it-matters-for-modern-software-security-b992657e413e?source=rss-31e146e30744------2</link>
            <guid isPermaLink="false">https://medium.com/p/b992657e413e</guid>
            <category><![CDATA[code-signing]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[software-security]]></category>
            <category><![CDATA[codesign]]></category>
            <dc:creator><![CDATA[Sudhir Mohanty (DevOps & DevSecOps Engineer)]]></dc:creator>
            <pubDate>Wed, 06 Aug 2025 00:42:41 GMT</pubDate>
            <atom:updated>2025-08-06T01:31:37.548Z</atom:updated>
            <content:encoded><![CDATA[<p>A foundational blog post explaining what code signing is, how it works, and why it’s critical for securing software delivery in today’s threat landscape. In a world where software is distributed through diverse channels and installed on millions of machines, ensuring that what gets deployed is <strong>authentic and untampered</strong> is more important than ever. This is where <strong>code signing</strong> plays a pivotal role.</p><h4>What Is Code Signing?</h4><p>Code signing is the process of <strong>digitally signing software artifacts</strong> — such as executables, scripts, container images, or packages — to verify their <strong>origin and integrity</strong>.</p><p>It uses <strong>cryptographic signatures</strong>, usually based on public key infrastructure (PKI), to provide assurance that:</p><ul><li>The code <strong>has not been modified</strong> since it was signed.</li><li>It was <strong>produced by a trusted developer, team, or company</strong>.</li></ul><h4>How Code Signing Works (At a High Level)</h4><ol><li>A developer or CI system signs a software artifact using a <strong>private key</strong>.</li><li>The signature, along with the corresponding <strong>public certificate</strong>, is embedded in or attached to the artifact.</li><li>When a user or system receives the artifact, it verifies the signature using the <strong>public key</strong>.</li><li>If the signature is valid and trusted (e.g., signed by a recognized certificate authority), the code is considered <strong>safe to run</strong>.</li></ol><h4>Why Code Signing Matters</h4><ul><li><strong>Tamper Protection</strong>: Confirms that the code hasn’t been modified after it was signed.</li><li><strong>Authenticity</strong>: Proves the code came from a verified source.</li><li><strong>Compliance</strong>: Required by many regulatory frameworks and security standards.</li><li><strong>Secure Software Supply Chain</strong>: Forms the foundation of zero-trust and secure software delivery.</li></ul><h4>What Can Be Code-Signed?</h4><ul><li>Applications (EXEs, DLLs)</li><li>Scripts (PowerShell, Bash, Python)</li><li>Mobile Apps (iOS/Android)</li><li>Container images (e.g., Docker, via skopeo)</li><li>Software packages (RPMs, DEBs)</li><li>Drivers and firmware</li></ul><h4>Risks of Not Code Signing</h4><ul><li>Increased risk of malware distribution</li><li>Vulnerability to supply chain attacks</li><li>Loss of user trust</li><li>Blocked installations by OS security tools</li></ul><h3>In Summary</h3><p>Code signing is not just a good-to-have — it’s a <strong>must-have</strong> for secure software development and distribution. It builds trust, ensures accountability, and forms a critical link in protecting the integrity of your software supply chain.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b992657e413e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Behind the Scenes: What Really Happens When You Install an Operator in OpenShift]]></title>
            <link>https://medium.com/@email2smohanty/behind-the-scenes-what-really-happens-when-you-install-an-operator-in-openshift-bc9a25a2c30c?source=rss-31e146e30744------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc9a25a2c30c</guid>
            <category><![CDATA[operatör]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Sudhir Mohanty (DevOps & DevSecOps Engineer)]]></dc:creator>
            <pubDate>Wed, 09 Jul 2025 05:17:16 GMT</pubDate>
            <atom:updated>2025-07-09T05:25:08.564Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey everyone! 👋 Welcome back to the channel.</p><p>If you’ve ever installed an Operator in OpenShift — maybe from OperatorHub or using YAML — you might wonder, <em>what’s actually happening behind the scenes?</em></p><p>Well, today we’re diving into the internals of <strong>Operator Lifecycle Manager (OLM)</strong> to understand exactly how OpenShift pulls, processes, and installs an Operator from a CatalogSource. Let’s break it down step by step — and trust me, it’s way more interesting than just clicking &#39;Install&#39;. 😄</p><p>Let’s get started!</p><p><strong>📦 [Part 1: It All Starts with a CatalogSource]</strong></p><p>When you install an Operator, the first component involved is the <strong>CatalogSource</strong>. Think of this as the ‘app store’ for Operators.</p><p>When you apply a CatalogSource YAML or select one from OperatorHub, OpenShift spins up a <strong>gRPC-based service</strong> (usually a Pod) that serves a list of all Operators available in that catalog.</p><p>The CatalogSource points to an <strong>index image</strong>, which is a container that holds metadata about all the available Operators, including versions, channels, and the location of their bundle images.</p><p><strong>🔍 [Part 2: OLM Queries the Catalog]</strong></p><p>Next, the <strong>Operator Lifecycle Manager</strong> — or OLM — comes into play.</p><p>It queries the CatalogSource over gRPC to retrieve all the available Operator packages and their version info.</p><p>This data is stored as <strong>PackageManifests</strong>, which you can view with:</p><pre>oc get packagemanifest</pre><p>At this point, OpenShift knows which Operators are available and which versions are offered in each channel.</p><p><strong>📥 [Part 3: Subscriptions and CSVs]</strong></p><p>When you create a <strong>Subscription</strong>, you’re telling OLM:</p><ul><li>‘Hey, I want this Operator, from this catalog, on this channel.’ *</li></ul><p>OLM uses the Subscription to:</p><ul><li>Locate the latest <strong>bundle</strong> for the Operator</li><li>Pull the Operator’s <strong>ClusterServiceVersion (CSV)</strong>, which acts like an install blueprint</li></ul><p>The CSV contains:</p><ul><li>Required CRDs</li><li>Deployment details</li><li>RBAC</li><li>Webhook config</li><li>Versioning info</li><li>And more</li></ul><p><strong>⚙️ [Part 4: Pulling the Bundle and Deploying the Operator]</strong></p><p>The <strong>Operator bundle</strong> is a container image — yes, a container! — that holds all the manifests.</p><p>OLM unpacks the bundle, creates the CRDs, applies the CSV, and deploys the actual Operator Pod into your namespace.</p><p>Now, your Operator is ready to manage its custom resources!</p><p><strong>🔁 [Part 5: Dependency Resolution &amp; Upgrades]</strong></p><p>Here’s where OLM gets really smart.</p><p>If the Operator needs other Operators — like cert-manager or service-binding-operator — and it&#39;s declared in the metadata, OLM will automatically:</p><ul><li>Resolve the dependency</li><li>Install the required Operators</li><li>Track which versions are compatible</li></ul><p>Plus, if the channel has new versions, OLM handles upgrades automatically based on your subscription strategy.</p><p><strong>👀 [Real World Tip]</strong></p><p>You can monitor all this with:</p><pre>oc get csv -n &lt;namespace&gt;<br>oc get subscription -n &lt;namespace&gt;</pre><p>And check logs for Operator Pods just like any other Kubernetes app.</p><p><strong>📌 [Recap &amp; Wrap-up]</strong></p><p>So to recap, here’s what happens when you install an Operator:</p><ol><li>CatalogSource exposes a catalog over gRPC</li><li>OLM queries and lists Operators via PackageManifests</li><li>You create a Subscription for the Operator</li><li>OLM pulls the corresponding bundle and CSV</li><li>The Operator is installed and deployed</li><li>Dependencies and upgrades are handled automatically 🎉</li></ol><p>OpenShift and OLM do a lot of heavy lifting under the hood to make Operators as powerful and hands-off as they are.</p><p><strong>🧠 [Call to Action]</strong></p><p>If you found this breakdown helpful, don’t forget to <strong>like</strong>, <strong>subscribe.</strong></p><p>Thanks for watching — and I’ll see you in the next one! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc9a25a2c30c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essential OpenShift and Kubernetes Commands for Daily Cluster Management]]></title>
            <link>https://medium.com/@email2smohanty/essential-openshift-and-kubernetes-commands-for-daily-cluster-management-f61007cc484f?source=rss-31e146e30744------2</link>
            <guid isPermaLink="false">https://medium.com/p/f61007cc484f</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[management]]></category>
            <category><![CDATA[cluster]]></category>
            <dc:creator><![CDATA[Sudhir Mohanty (DevOps & DevSecOps Engineer)]]></dc:creator>
            <pubDate>Tue, 17 Sep 2024 14:09:17 GMT</pubDate>
            <atom:updated>2024-09-17T14:09:17.195Z</atom:updated>
            <content:encoded><![CDATA[<p>This blog provides a collection of useful OpenShift commands that are essential for day-to-day activities in an OpenShift cluster. Some commands are specific to OpenShift, while others combine OpenShift and Bash commands.</p><h3><strong>Label and annotation related commands:</strong></h3><ol><li>Adding labels to resources: Below command will add label <strong>type=custom</strong> to pv <strong>volume-123</strong>.</li></ol><pre>oc label pv volume-123 type=custom</pre><p>2. Removing labels from resources: Below command will remove label <strong>type=custom</strong> from pv <strong>volume-123</strong>.</p><pre>oc label pv volume-123 type-</pre><p>3. Adding annotations to resources: Below command will add annotation <strong>type=custom</strong> to pv <strong>volume-123</strong>.</p><pre>oc annotate pv volume-123 type=custom</pre><p>4. Removing annotations from resources: Below command will remove annotation <strong>type=custom</strong> from pv <strong>volume-123</strong>.</p><pre>oc annotate pv volume-123 type-</pre><p><strong>Note: </strong>We can use label and annotate commands for other type of resources like pod, secrets etc.</p><h3><strong>Patching commands:</strong></h3><ol><li>Setting serviceAccountName in deployment: Below command will set the <strong>serviceAccountName</strong> as <strong>useroot</strong> in deployment demo-app.</li></ol><pre>oc patch deploy/demo-app \ --patch &#39;{&quot;spec&quot;:{&quot;template&quot;:{&quot;spec&quot;:{&quot;serviceAccountName&quot;: &quot;useroot&quot;}}}}&#39;</pre><p>2. Setting image value in catalogsource: Below command will set the image value as <strong>docker.io/library/python:latest</strong> in catalogsource <strong>my-app-catalog</strong>.</p><pre>oc patch CatalogSource my-app-catalog  -n openshift-marketplace   -p &#39;{&quot;spec&quot;: {&quot;image&quot;: &quot;docker.io/library/python:latest&quot;}}&#39; --type=merge</pre><p>3. Replacing an attribute value: Below command will replace value for label <strong>type</strong> in pod <strong>devops-xyz</strong>.</p><pre>oc patch po devops-xyz --type=json -p=&#39;[{&quot;op&quot;: &quot;replace&quot;, &quot;path&quot;: &quot;/metadata/labels/type&quot;, &quot;value&quot;: &quot;external&quot;}]&#39;</pre><p>4. Removing an attribute value: Below command will remove label <strong>type</strong> from pod <strong>devops-xyz</strong>.</p><pre>oc patch po devops-xyz --type=json -p=&#39;[{&quot;op&quot;: &quot;remove&quot;, &quot;path&quot;: &quot;/metadata/labels/type&quot;}]&#39;</pre><h3><strong>Mustgather commands:</strong></h3><p>The oc must-gather command in OpenShift is a diagnostic tool used to collect a comprehensive set of logs, configuration data, and resource information from an OpenShift cluster.</p><p>Below command will collect logs and other data’s of <strong>development</strong> and <strong>test</strong> namespaces into <strong>tmp</strong> directory. The timeout specified is 600 sec, it’s for limiting the amount of time for must gather operation.</p><pre>oc adm must-gather --image=quay.io/opencloudio/must-gather:4.5.1 --gather -n development,test --dest-dir=/tmp --timeout=600s</pre><p>After running oc adm must-gather, you will find a new directory or zip file under tmp directory (e.g., must-gather.local.*) that contains various logs, configuration files, and resource information organized in a structured format.</p><h3><strong>Resource get commands:</strong></h3><ol><li>Command to get ip address of nodes:</li></ol><pre>kubectl get nodes -o jsonpath=&#39;{.items[*].status.addresses[?(@.type==&quot;InternalIP&quot;)].address}&#39;</pre><p>2. Command to get pods having certain labels: Below command is for getting all pods having label <strong>app=wordpress</strong> from default namespace.</p><pre>oc get pods -n default --no-headers -l app=wordpress</pre><p><strong>Note:</strong> Same command can be used to get other resources like deployments, pvc’s, and secrets etc.</p><p>3. Getting full name of pod with partial name: Below command will get the full name of pod with partial name <strong>devops</strong> from default namespace.</p><pre>kubectl get po -n default| grep devops | cut -d &quot; &quot; -f1</pre><p><strong>Note:</strong> Same command can be used to get other resources like deployments, pvc’s, and secrets etc.</p><p>4. Checking if pod exists with some name: Below command will check if a pod named <strong>devops-pod </strong>exists in cluster or not, -A option stands for all namespaces.</p><pre>oc get pod -A | grep devops-pod</pre><p>5. Getting the storage class used by a pvc: Replace the &lt;pvc-name&gt; in below command before using.</p><pre>oc get pvc &lt;pvc-name&gt; -o jsonpath=&#39;{.spec.storageClassName}&#39;</pre><p>6. Getting the state of a pods: Below command will return the sate of all the pods for a particular namespace and its most useful if we want to check if all the pods of a namespace are in running state or not.</p><pre>oc get pods -o=jsonpath=&#39;{.items[*].status.phase}{&quot;\n&quot;}&#39;</pre><p>7. Getting the state of a particular pod: Below command will get the state of <strong>devops-xyz </strong>pod.</p><pre>oc get pod devops-xyz -o=jsonpath=&#39;{.status.phase}&#39;</pre><p>8. Getting the pod ready status: Below command is for checking if pod <strong>devops-xyz</strong> is Ready or not, if it’s ready then it will return <strong>True</strong> else <strong>False</strong>.</p><pre>oc get po devops-xyz -o=jsonpath=&#39;{.status.conditions[?(@.type==&quot;Ready&quot;)].status}&#39;</pre><p>9. Getting replica count of deployments and sts: Below command will get replica count of <strong>monitoring-layout</strong> deployment.</p><pre>kubectl get deployments monitoring-layout -o=jsonpath=&#39;{.status.replicas}&#39;</pre><p>10. Showing labels of resources: Below command will show labels of <strong>devops-xyz</strong> pod, we can use it for showing labels of other resource types as well.</p><pre>oc get po devops-xyz --show-labels</pre><p>11. Get more details about nodes:</p><pre>oc get nodes -o wide</pre><h3><strong>wait commands:</strong></h3><p>The kubectl wait command is used in Kubernetes (and OpenShift) to wait for specific conditions to be met before proceeding.</p><ol><li>Wait for a Pod to Be Ready: Below command is to wait for <strong>my-pod</strong> to be ready and the wait timeout is 60sec</li></ol><pre>kubectl wait --for=condition=Ready pod/my-pod --timeout=60s</pre><p>2. Wait for a Deployment to Be Available: Below command is to wait for <strong>my-deployment</strong> to be Available and the wait timeout is 120sec.</p><pre>kubectl wait --for=condition=Available deployment/my-deployment --timeout=120s</pre><p>3. Wait for Resource Deletion: Below command is to wait for <strong>my-pod</strong> to be deleted and the wait timeout is 60sec</p><pre>kubectl wait --for=delete pod/my-pod --timeout=60s</pre><h3><strong>cp commands:</strong></h3><p>The cp command is often used to <strong>copy files between your local machine and the running pods</strong> in the cluster.</p><ol><li>Copy a File from Local Machine to Pod: Below command copies local-file.txt from your local machine to the /tmp directory in the pod named my-pod .</li></ol><pre>oc cp ./local-file.txt my-pod:/tmp/local-file.txt</pre><p>2. Copy a Directory from Pod to Local Machine: Below command copies the /var/logs/ directory from my-pod to a local.</p><pre>oc cp my-pod:/var/logs/ ./local-logs/</pre><h3><strong>Creating resources of a namespace into different namespace:</strong></h3><ol><li>Suppose we have a requirement of creating all the secrets of namespace <strong>A</strong> in namespace <strong>B</strong> then we can make use of below command. Make sure you are replacing values <strong>A</strong> and <strong>B</strong> in below command with your actual namespace values before running the command.</li></ol><pre>for i in `kubectl get secrets | awk &#39;{print $1}&#39;`; do  kubectl get secret $1 -n A -o yaml | sed s/&quot;namespace: A&quot;/&quot;namespace: B&quot;/ | kubectl apply -n B -f -  ; done</pre><p>2. Suppose we have a requirement of creating a particular secret of namespace <strong>A</strong> in namespace <strong>B</strong> then we can make use of below command. Make sure you are replacing values <strong>&lt;secret name&gt;</strong>, <strong>A</strong> and <strong>B</strong> in below command with your actual secret and namespace values before running the command.</p><pre>kubectl get secret &lt;secret name&gt; --namespace=A -o yaml | kubectl apply --namespace=B -f -</pre><p><strong>Note:</strong> Above two commands can be used to create other type of resources like pod, pvc etc</p><h3><strong>cluster contexts commands:</strong></h3><p>When working with multiple clusters, you may often need to switch between them frequently. Instead of repeatedly using oc login to connect to different clusters, you can simplify the process by using the oc config use-context command to switch contexts, provided your kubeconfig file contains valid credentials for the target cluster. If the kubeconfig file does not have a valid token for the cluster, you will be prompted to log in to the cluster again.</p><ol><li>To see the currently active context:</li></ol><pre>kubectl config current-context</pre><p>2. List all Contexts:</p><pre>kubectl config get-contexts</pre><p>3. Set the current context to required cluster:</p><pre>kubectl config use-context &lt;context-name&gt;</pre><h3>logs commands:</h3><p>The oc logs command in OpenShift is used to retrieve the logs from containers running in pods. This command is valuable for debugging and monitoring the behavior of applications running in the cluster.</p><ol><li>Viewing the live logs of a pod: Below command will show the live logs of pod <strong>devops-xyz, </strong>if multiple containers are present in the pod and we want to see logs of a particular container then we have to specify the container name or id with<strong> </strong><strong>-c </strong>option<strong>.</strong></li></ol><pre>oc logs -f devops-xyz </pre><p>2. Viewing only the last few lines of logs: Below command will show last 5 line logs of <strong>devops-xyz pod.</strong></p><pre>oc logs -f devops-xyz --tail=5</pre><h3>Conclusion:</h3><p>There are many important OpenShift commands, but I’ve focused on those I use for my daily cluster operations. I hope these will be useful to others as well. I plan to continue adding new commands to this blog in the future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f61007cc484f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building, Packaging, and Bundling OpenShift Operators: A Step-by-Step Guide]]></title>
            <link>https://blog.devops.dev/building-packaging-and-bundling-openshift-operators-a-step-by-step-guide-a73cd2ee93d3?source=rss-31e146e30744------2</link>
            <guid isPermaLink="false">https://medium.com/p/a73cd2ee93d3</guid>
            <category><![CDATA[bundle]]></category>
            <category><![CDATA[build]]></category>
            <category><![CDATA[operator-sdk]]></category>
            <category><![CDATA[packaging]]></category>
            <category><![CDATA[openshift]]></category>
            <dc:creator><![CDATA[Sudhir Mohanty (DevOps & DevSecOps Engineer)]]></dc:creator>
            <pubDate>Sun, 18 Aug 2024 16:04:03 GMT</pubDate>
            <atom:updated>2024-08-19T17:39:46.580Z</atom:updated>
            <content:encoded><![CDATA[<p>The process of building, packaging, and bundling an OpenShift Operator typically involves several steps that integrate various tools and methodologies. Here’s a high-level overview of the process:</p><h3>1. Setup Your Development Environment</h3><p>Before you start, ensure you have the necessary tools installed:</p><ul><li><strong>Operator SDK</strong>: To scaffold, build, and manage your operator project.</li><li><strong>Go</strong>: The programming language typically used for developing operators (though other languages are supported).</li><li><strong>Docker</strong>: For containerizing your operator.</li><li><strong>kustomize: </strong>Enables you to customize untemplated YAML files, leaving the original files untouched</li></ul><h3>2. Operator Development</h3><p><strong>a. Initialize the Operator Project:</strong> Use the Operator SDK to create a new operator project.</p><pre>operator-sdk init --domain=my.domain --repo=github.com/myorg/my-operator</pre><p>This command creates a new project with a standard directory structure:</p><pre>.<br>├── config/            # Kubernetes and OLM manifests<br>├── Makefile           # Makefile having different targets like build, test etc<br>├── Dockerfile         # Dockerfile for building the operator image<br>├── go.mod             # Go module file<br>└── main.go            # Main program</pre><p><strong>b. Create API and Controller: </strong>Define the custom resource (CR) that your operator will manage by generating the API and controller code.</p><pre>operator-sdk create api --group=app --version=v1 --kind=MyApp</pre><p><strong>API</strong>: This command generates API types (in api/) and CRD manifests.</p><p><strong>Controller</strong>: It also scaffolds the controller logic (in controllers/).</p><p><strong>c. Customize the API:</strong> Open the API files generated in api/ and define the fields for your custom resource. This defines the desired and observed states of the CR</p><p>Example (api/v1/myapp_types.go):</p><pre>type MyAppSpec struct {<br>    Size int32 `json:&quot;size&quot;`<br>}<br><br>type MyAppStatus struct {<br>    Nodes []string `json:&quot;nodes&quot;`<br>}</pre><p><strong>d. Implement the Controller Logic:</strong> Open the controller file in controllers/ and implement the reconciliation logic.</p><p>Example (controllers/myapp_controller.go):</p><pre>func (r *MyAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {<br>    // Fetch the MyApp instance<br>    myApp := &amp;appv1.MyApp{}<br>    err := r.Get(ctx, req.NamespacedName, myApp)<br>    if err != nil {<br>        return ctrl.Result{}, client.IgnoreNotFound(err)<br>    }<br><br>    // Implement reconciliation logic here...<br><br>    return ctrl.Result{}, nil<br>}</pre><p>The reconciliation loop checks the desired state defined in the CR and adjusts the actual state of the cluster to match it.</p><h3>3. Build the Operator</h3><p>a. Build the go code:</p><pre>go build -a -o manager main.go</pre><p>b. Create a Dockerfile to containerize your operator:</p><pre>FROM registry.access.redhat.com/ubi8/ubi-minimal:latest<br><br>USER 65532:65532<br><br>COPY manager /usr/local/bin/manager<br><br>ENTRYPOINT [&quot;/usr/local/bin/manager&quot;]</pre><p>c. Build the operator image and push it to a container registry:</p><pre>docker build -t myorg/my-operator:v1 .<br>docker push myorg/my-operator:v1</pre><h3>4. <strong>Packaging the Operator</strong></h3><p><strong>a. Generate Bundle Manifests: </strong>Use the Operator SDK to generate bundle manifests using a command like operator-sdk generate kustomize manifests. And this will generate YAML files under the config directory, which include:</p><ul><li><strong>ClusterServiceVersion (CSV):</strong> Metadata defining how your operator is deployed. This file will be created in path config/manifests/bases .</li><li><strong>CRDs:</strong> The Custom Resource Definitions for your operator.</li></ul><p><strong>b. Set the operator image in created manifests: </strong>It can be done using command like below, here the operator image is what we had created in Build Operator stage.</p><pre>cd config/manager &amp;&amp; kustomize edit set image controller=myorg/my-operator:v1</pre><p><strong>c. Create the Operator Bundle:</strong> Use below command to package the operator code, CRDs, and metadata into an operator bundle. Basically one directory named bundle will be created and it should have two main sub directories named <strong>manifests</strong> and <strong>metadata</strong>.</p><pre>kustomize build config/manifests | operator-sdk generate bundle --version 0.0.1</pre><p>After running above command, you should see this directory structure:</p><pre>bundle/<br> ├── manifests<br> │ └── my-operator.clusterserviceversion.yaml<br> └── metadata<br> └── annotations.yaml</pre><p><strong>d. Validate the Bundle: </strong>Use operator-sdk bundle validate ./bundle to check for compliance with the Operator Lifecycle Manager (OLM) standards.</p><p><strong>e. Build the Operator Bundle Image: </strong>You need to build a Docker image containing your Operator bundle. Create a Dockerfile that copies your bundle files into the image:</p><p>Example Dockerfile:</p><pre>FROM scratch<br>COPY bundle/manifests /manifests/<br>COPY bundle/metadata /metadata/</pre><p>Build and push the bundle image:</p><pre>docker build -t myorg/my-operator-bundle:v1 .<br>docker push myorg/my-operator-bundle:v1</pre><h3>5. Create the Operator Bundle and Catalog</h3><p><strong>a. </strong>Use opm to create a catalog image from your bundle image.</p><p><strong>Command to Create the Catalog Image:</strong></p><pre>opm -u docker index add --bundles myorg/my-operator-bundle:v1 --tag myorg/my-operator-index:v1</pre><ul><li>--bundles: A comma-separated list of bundle images to include in the catalog.</li><li>--tag: The tag for the resulting catalog source image.</li></ul><p>If you have multiple bundles (e.g., different versions or different Operators), you can list them all:</p><pre>opm -u docker index add --bundles myorg/my-operator-bundle:v1,myorg/another-operator-bundle:v2 --tag myorg/custom-catalog:v1</pre><p><strong>b. Push to a Container Registry:</strong></p><pre>docker push myorg/custom-catalog:v1</pre><h3>Conclusion:</h3><p>This workflow ensures that your operator is built, packaged, and deployed according to OpenShift and Kubernetes standards, facilitating easier management, upgrades, and scaling.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a73cd2ee93d3" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/building-packaging-and-bundling-openshift-operators-a-step-by-step-guide-a73cd2ee93d3">Building, Packaging, and Bundling OpenShift Operators: A Step-by-Step Guide</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stress Testing Kubernetes (K8s) and OpenShift Nodes Using Stress-ng]]></title>
            <link>https://medium.com/@email2smohanty/stress-testing-kubernetes-k8s-and-openshift-nodes-using-stress-ng-35b6a9752c8d?source=rss-31e146e30744------2</link>
            <guid isPermaLink="false">https://medium.com/p/35b6a9752c8d</guid>
            <category><![CDATA[stress-testing]]></category>
            <category><![CDATA[load-testing]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Sudhir Mohanty (DevOps & DevSecOps Engineer)]]></dc:creator>
            <pubDate>Sun, 04 Aug 2024 08:01:10 GMT</pubDate>
            <atom:updated>2024-08-04T08:01:10.789Z</atom:updated>
            <content:encoded><![CDATA[<p>Stress testing Kubernetes (K8s) or OpenShift nodes involves pushing the node beyond its normal operational capacity to observe how it behaves under extreme conditions. This type of testing helps identify the maximum capacity and stability limits, potential bottlenecks, and areas for improvement.</p><p>In this tutorial, we’ll explore how to perform stress testing on Kubernetes and OpenShift nodes using Stress-ng. Stress-ng will stress test a computer system in various selectable ways. It was designed to exercise various physical subsystems of a computer as well as the various operating system kernel interfaces. It’s a powerful tool for stressing various components of your system, including CPU, memory, I/O, and disk. It allows to simulate different types of workloads and stress conditions.</p><p>We will generate stress on a required OpenShift or Kubernetes node by deploying a stress generation pod on that node. This pod will use the polinux/stress-ng container image, which leverages the Stress-ng tool to apply stress to the node.</p><p>Here are the steps to generate stress on a specific Kubernetes or OpenShift node:</p><h3>Step 1. <strong>Login to the required k8s or openshift cluster using cli</strong></h3><p>Since we will be generating stress by deploying a pod, you first need to log in to the OpenShift or Kubernetes cluster.</p><h3>Step 2. Prepare the stress generation pod definition</h3><p>Here is a sample definition for the stress generation pod. Before deploying the pod, you need to specify the node name in the pod definition. This name should correspond to the actual OpenShift or Kubernetes node on which you want to generate the stress.</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: strees-pod<br>spec:<br>  containers:<br>  - name: pod-stress-test<br>    image: polinux/stress-ng<br>    command: [ &quot;/bin/bash&quot;, &quot;-c&quot;, &quot;--&quot; ]<br>    args: [ &quot;stress-ng --cpu 3 --vm 3 --vm-bytes 2G --io 3&quot; ]<br>  nodeName: &lt;node name&gt;</pre><p>In the pod definition above, the command stress-ng --cpu 3 --vm 3 --vm-bytes 2G --io 3 will generate the stress using the Stress-ng utility. This Stress-ng utility is provided by the polinux/stress-ng container image. Basically we will simulate CPU, memory, and I/O stress through the used stress-ng command.</p><p>--cpu 3: Spawns three CPU stressor process.</p><p>--io 3: Spawns three I/O stressor process.</p><p>--vm 3: Spawns three virtual memory stressor process.</p><p>--vm-bytes 2G: Allocates 2G of memory per virtual memory stressor process.</p><h3>Step 3. Deploy the stress generation pod</h3><p>In the previous step, we prepared the stress generation pod. Now, to deploy the pod using that definition, you can use the following command:</p><ul><li><strong>For OpenShift</strong>:</li></ul><pre>oc create -f pod-definition.yaml</pre><ul><li><strong>For Kubernetes</strong>:</li></ul><pre>kubectl create -f pod-definition.yaml</pre><p>Make sure to replace pod-definition.yaml with the actual filename of your pod definition YAML file.</p><h3>Step 4. Analyze Node Performance</h3><p>While the stress generation pod is running, you can monitor the node’s performance using the following commands:</p><ul><li><strong>For OpenShift</strong>: Use oc adm top to view resource usage.</li></ul><pre>oc adm top nodes</pre><ul><li><strong>For Kubernetes</strong>: Use kubectl top to view resource usage.</li></ul><pre>kubectl top nodes</pre><p>These commands will display metrics for CPU, memory, and, in some setups, I/O usage. Observe how these metrics change as the stress generation pod applies load to the node. This will help you understand the impact of the stress test on node performance.</p><h3>Conclusion</h3><p>Stress testing is a powerful technique for understanding the resilience and limits of your Kubernetes or OpenShift infrastructure. By carefully planning and executing stress tests, you can improve the stability and reliability of your systems under extreme conditions. In this blog, we’ve used a simple stress-ng command as an example. However, you can utilize various other commands and options based on your specific stress scenarios. For more details about the Stress-ng tool and the polinux/stress-ng container image, you can refer to the following links:</p><ul><li><a href="https://manpages.ubuntu.com/manpages/bionic/man1/stress-ng.1.html">Stress-ng Tool Documentation</a></li><li><a href="https://hub.docker.com/r/polinux/stress-ng/dockerfile">polinux/stress-ng Container Image on Docker Hub</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35b6a9752c8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>