<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yevgeniy Ovsyannikov on Medium]]></title>
        <description><![CDATA[Stories by Yevgeniy Ovsyannikov on Medium]]></description>
        <link>https://medium.com/@281332?source=rss-c16c438002cd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ocunXahXNzE-iShhYK7bxA.jpeg</url>
            <title>Stories by Yevgeniy Ovsyannikov on Medium</title>
            <link>https://medium.com/@281332?source=rss-c16c438002cd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:42:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@281332/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ArgoCD access to AWS ECR for helm OCI (external secrets operator)]]></title>
            <link>https://medium.com/@281332/argocd-access-to-aws-ecr-for-helm-oci-external-secrets-operator-c850d3461f5f?source=rss-c16c438002cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/c850d3461f5f</guid>
            <dc:creator><![CDATA[Yevgeniy Ovsyannikov]]></dc:creator>
            <pubDate>Thu, 27 Jun 2024 06:51:18 GMT</pubDate>
            <atom:updated>2024-06-27T06:51:18.402Z</atom:updated>
            <content:encoded><![CDATA[<p>If you have been landed to this article, you probably know that connect ArgoCD to ECR its a challenge, AWS ECR by his nature generates authorization token only for 12 hours and you have to feed to ArgoCD repository connection this token over and over. Folks inventing different workarounds, like Kubernetes cron jobs etc…</p><p>I want to share more organic way I’ve observed.</p><p>Originally in our organization we already used <a href="https://external-secrets.io/latest/">external secrets operator</a> that we are installing like a base application for every cluster. This tool is providing you out of the box way for <a href="https://external-secrets.io/latest/api/generator/ecr/">connecting to ECR</a>. Their example is more POC oriented</p><p>I want to show you E2E solution with cross AWS account connectivity.</p><p>Assume you have 2 accounts the first with ECR and the second with your Argocd on top of EKS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*h3MWj1ynjoybNL05WOD8DQ.png" /></figure><p>Now you have to install External Secrets Operator on your cluster aside with your Argocd (i wont show step by step command, it could be with some kubectll apply, we wrapped it to helm)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*8AWSI-BKfXFpFpi2E4h49Q.png" /></figure><p>Next following objects should be applied (almost the same like from ESO example), I will show our helm template, it probably will help someone with escaping (lost some time on it, because of ESO templating uses go templating behind the scenes, so it wass challenge to remove https from endpoint i got, the only solution I found is using \\\ :-) )</p><pre>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  annotations:<br>    eks.amazonaws.com/role-arn: {{ .Values.ecr.role_arn }}<br>  name: eso-ecr<br>  namespace: argocd<br>---<br>apiVersion: generators.external-secrets.io/v1alpha1<br>kind: ECRAuthorizationToken<br>metadata:<br>  name: eso-ecr-gen<br>  namespace: argocd<br>spec:<br>  region: {{ .Values.ecr.region }}<br>  role: {{ .Values.ecr.role_arn }}<br>  auth:<br>    jwt:<br>      serviceAccountRef:<br>        name: &quot;eso-ecr&quot;<br>---<br>apiVersion: external-secrets.io/v1beta1<br>kind: ExternalSecret<br>metadata:<br>  name: &quot;ecr-helm-secret&quot;<br>  namespace: argocd<br>  labels:<br>    argocd.argoproj.io/secret-type: &quot;repository&quot;<br>spec:<br>  refreshInterval: &quot;180m&quot;<br>  target:<br>    name: ecr-helm-secret<br>    template:<br>      metadata:<br>        labels:<br>          argocd.argoproj.io/secret-type: &quot;repository&quot;<br>      engineVersion: v2<br>      data:<br>        enableOCI: &quot;true&quot;<br>        username: &quot;{{&quot;{{ .username }}&quot;}}&quot;<br>        password: &quot;{{&quot;{{ .password }}&quot;}}&quot;<br>        url: &quot;{{&quot;{{ .proxy_endpoint | trimPrefix \\\&quot;https://\\\&quot; }}&quot;}}&quot;<br>        type: helm<br>        name: awscloud-ecr<br>        project: default<br>  dataFrom:<br>  - sourceRef:<br>      generatorRef:<br>        apiVersion: generators.external-secrets.io/v1alpha1<br>        kind: ECRAuthorizationToken<br>        name: &quot;eso-ecr-gen&quot;</pre><p>Now once 3 those object are installed, you will probably ask, which arn should I provide to SA and ECRAuthorizationToken for assuming role.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*sq9yRreeol79tLDxhQIHrA.png" /></figure><p>That was not so obvious for me, hope it will save you some time.</p><p>On <strong>Account foo </strong>you have to create:</p><ol><li>OIDC connect provider with id of your EKS in <strong>Account bar</strong></li><li>Role with sts:AssumeRoleWithWebIdentity to this OIDC</li><li>ECR policy, or use existing AWS ECR policy like I did</li></ol><p>And nothing on <strong>Account bar</strong></p><p>Here is terraform module that can help you for automation:</p><pre>variable &quot;name&quot; {<br>  description = &quot;cluster name&quot;<br>  type        = string<br>}<br><br>variable &quot;oidc-url&quot; {<br>  description = &quot;oidc url&quot;<br>  type        = string<br>}<br><br>variable &quot;oidc-thumbprint&quot; {<br>  description = &quot;oidc thumbprint&quot;<br>  type        = string<br>}<br><br><br>data &quot;aws_caller_identity&quot; &quot;current&quot; {}<br>locals {<br>    account_id = data.aws_caller_identity.current.account_id<br>}<br><br><br>resource &quot;aws_iam_openid_connect_provider&quot; &quot;oidc_issuer&quot; {<br>  url             = var.oidc-url<br>  thumbprint_list = [var.oidc-thumbprint]<br>  client_id_list  = [&quot;sts.amazonaws.com&quot;]<br>}<br><br><br>data &quot;aws_iam_policy_document&quot; &quot;irsa&quot; {<br>  statement {<br>    actions = [&quot;sts:AssumeRoleWithWebIdentity&quot;]<br>    principals {<br>      type = &quot;Federated&quot;<br>      identifiers = [<br>        aws_iam_openid_connect_provider.oidc_issuer.arn<br>      ]<br>    }<br>  }<br>  statement {<br>    actions = [ &quot;sts:AssumeRole&quot;]<br>    principals {<br>      type = &quot;AWS&quot;<br>      identifiers = [<br>        &quot;arn:aws:sts::${local.account_id}:assumed-role/ecr-access-for-${var.name}/external-secrets-provider-aws&quot;<br>      ]<br>    }<br>  }<br>}<br><br> <br>resource &quot;aws_iam_role&quot; &quot;argocd_role&quot; {<br>  name               =  &quot;ecr-access-for-${var.name}&quot;<br>  path               = &quot;/&quot;<br>  assume_role_policy = data.aws_iam_policy_document.irsa.json<br>}<br> <br>resource &quot;aws_iam_role_policy_attachment&quot; &quot;policy_attachment&quot; {<br>  role       = aws_iam_role.argocd_role.name<br>  policy_arn = &quot;arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly&quot;<br>}</pre><p>And calling this module:</p><pre>module &quot;ecr-access&quot; {<br>  source = &quot;./module&quot;<br>  name = &quot;argocd-to-ecr&quot;<br>  oidc-url = &quot;https://oidc.eks.&lt;region&gt;.amazonaws.com/id/&lt;OIDC ID&gt;&quot; # OIDC url from your EKS cluster<br>  oidc-thumbprint = &quot;dfdsfdsfd&quot; # Thumbprint of OIDC from your EKS cluster<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c850d3461f5f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we built “Playgrounds” for our potential customers]]></title>
            <link>https://engineering.lightrun.com/how-we-built-playgrounds-for-our-potential-customers-9ddf446015e1?source=rss-c16c438002cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ddf446015e1</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[playground]]></category>
            <dc:creator><![CDATA[Yevgeniy Ovsyannikov]]></dc:creator>
            <pubDate>Sun, 21 Aug 2022 10:59:39 GMT</pubDate>
            <atom:updated>2022-08-31T07:45:05.578Z</atom:updated>
            <content:encoded><![CDATA[<h3>Intro</h3><p>I’m leading a brilliant devops team in a small startup — <a href="https://lightrun.com/">Lightrun</a> which provides observability platform for developers. TL;DR you as a developer can get any <strong><em>code</em></strong> related information which runs on live production directly from your IDE. You no need anymore wait for redeploy if you have to print some variables or get all values like you did it with local debbuger.</p><p>The simple topology is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*DuJsX2gDcg5BC-XvBPqaDA.png" /></figure><ul><li>Developer install IDE plugin</li><li>Live application runs on production aside with agent</li><li>Client communicate with agent through the server (could be saas or onprem)</li></ul><h3>Motivation</h3><p>The product is super cool and sexy, but if you only want to play or try, and discover some features which will potentially bring you some benefits, you have to spent some time for building such setup.</p><p>Initially you have to:</p><ul><li>emulate running application</li><li>install IDE plugin</li><li>signup to our platform</li><li>read documentation</li></ul><p>We wanted save for you all these steps and provide already predefined setup where you can directly start to play.</p><h3>Dream for DevOps</h3><p>I want to provide our journey from devops prospective, cause we found this project very interesting and challenging, which touches almost all areas of our magnificent field.</p><ul><li>Microservices and Serverless</li><li>CI/CD</li><li>Scale and Availability</li><li>Provisioning time and TTL</li><li>Security and Cost</li><li>Public cloud services</li><li>Monitoring, BI, Alerting</li><li>Peering with other teams</li></ul><h3>Requirements</h3><ul><li>Provide dedicated “playground” on demand based on user email</li><li>Provisioning time should aims to zero</li><li>Every “playground” should include 2 components (code-server aka vscode and some game application)</li></ul><h3>Architecture</h3><p>The main architectural challenge was related to “playground” provisioning time, we decided deploy containers on top of EKS with AWS Fargate backend (for security and cost purposes) also we had some lambdas on the way which had to register user on our platform, trig deployment of containers, containers were trig creation new Fargate nodes accordingly, container boot time, readiness probes…</p><p>So…. We realized that the most optimistic time for provisioning 1 playground is about 2 minutes... This number was not acceptable cause user had to wait all this time in front for “spinning wheel” and our frontend developer (which has limited time for this project) should implement this logic as well.</p><p>After some considerations and discussions we got the following solution:</p><ul><li>We have to have desired number of free predefined environments</li><li>User will get access to such environment instantly for 1 hour (zero seconds to wait)</li><li>Once 1 free playground was taken, watcher should automatically create 1 new</li><li>After 1 hour, used playground should be deleted</li><li>User will get the same playground within 1 hour even if the browser window was closed</li><li>We have to limit maximum playgrounds in the system</li></ul><p>So lets see how we achieved it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*FXok-5VD8ZIGQFLitgpYGg.png" /></figure><ol><li>The center of our project is DynamoDB table. And it has the following columns</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wZzQOBdaWBjOST05I1g9ig.png" /></figure><ul><li><strong>SystemEmail</strong> — with this email we have created dummy user in our system</li><li><strong>CompanyID </strong>— this id was assigned for current user, we need it for BI</li><li><strong>SessionId</strong> — uniq identifier of playground, this uuid we are using for naming and dns records</li><li><strong>Status</strong> — actual status of playground. Introduced to avoid providing not ready/booting playground to end user</li><li><strong>TimeToExist</strong> — TTL of playground, after expiration DynamoDB automatically deletes line which trig event for removal playground itself</li><li><strong>UserEmail</strong> — actual user email who requested playground</li></ul><p>2. Lambda <strong>“Watcher”</strong> is responsible for desired number of empty playgrounds and maximum available playgrounds at all.</p><p>3. Lambda <strong>“Allocator” </strong>is responsible for providing empty playground to user and if he asked within 1 hour one more time, return the same one.</p><p>4. Lambda <strong>“Creator/Remover”</strong> listens for dynamodb events (from streams) and create/remove needed resources accordingly</p><p>5. Frontend has javascript application which has signup page and 2 iframes which is hosted on S3 static website behind the AWS CloudFront</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bYF9j9bLUufcuI09faNSZg.png" /><figcaption>custom metrics</figcaption></figure><h3>UserFlow</h3><p>If you tried our playground, you can see that we also implemented some walkthrough mechanism (popups) by <a href="https://userflow.com/">https://userflow.com/</a> which explains what to do and which button should be pressed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A_lgsPWySXqju_H13MIMOg.png" /></figure><h3>The end…</h3><p>Nope, it’s just the beginning… We have some areas to improve. Our platform is fully provisioned by terraform, and potentially we can easily onboard on it different types of playgrounds with different flows.</p><p>The result of our work you can find here: <a href="https://playground.lightrun.com/">https://playground.lightrun.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ddf446015e1" width="1" height="1" alt=""><hr><p><a href="https://engineering.lightrun.com/how-we-built-playgrounds-for-our-potential-customers-9ddf446015e1">How we built “Playgrounds” for our potential customers</a> was originally published in <a href="https://engineering.lightrun.com/">Lightrun Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why my pod gets OOMKill (exit code 137) without reaching threshold of requested memory]]></title>
            <link>https://awstip.com/why-my-pod-got-oom-kill-exit-code-137-without-reaching-threshold-of-requested-memory-bd783b9ccd0f?source=rss-c16c438002cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd783b9ccd0f</guid>
            <dc:creator><![CDATA[Yevgeniy Ovsyannikov]]></dc:creator>
            <pubDate>Sun, 21 Aug 2022 10:25:53 GMT</pubDate>
            <atom:updated>2022-08-30T19:30:53.963Z</atom:updated>
            <content:encoded><![CDATA[<p>One day we realized that pod of our main service started to get bunch of restarts. Based on <strong><em>kubectl describe pod </em></strong>we saw that pod was restarted due to OOMKill with exit status 137. For production purposes we set requests of CPU and RAM equal to limits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*IuYGeEkweRSoO7NkHPVWSA.png" /></figure><p>So based on it, our pod should be killed once it reaches 16Gi. But through the monitoring we are seeing another picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*sgoo3DLy0Nw8XdDa8VP5NA.png" /></figure><p>Pod was killed while it had only about 7Gi.</p><p>That’s weird..</p><p>After some research on <a href="https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/">kubernetes docs</a>, we observed that there are few types of QoS</p><ul><li>Guaranteed</li><li>Burstable</li><li>BestEffort</li></ul><p>Which are using Linux mechanism <em>oom_score_adj </em>behind the scenes. You can read about it <a href="https://gist.github.com/makeittotop/58996c0c9e861b8d179a">here</a>.</p><p>We started to check what type of QoS our pod has with help of this command</p><pre>kubectl get pod &lt;pod name&gt; -o jsonpath=’{.status.qosClass}’</pre><p>And realized that it’s <strong>Burstable </strong>but other pods on the system are <strong>Guaranteed</strong></p><p>Be aware that QoS parameter is not configurable and kubernetes automatically understands which type to set base on request/limits configuration.</p><p>Here are requirements for <strong>Guaranteed</strong>:</p><ul><li>Every Container in the Pod must have a memory limit and a memory request.</li><li>For every Container in the Pod, the memory limit must equal the memory request.</li><li>Every Container in the Pod must have a CPU limit and a CPU request.</li><li>For every Container in the Pod, the CPU limit must equal the CPU request.</li></ul><p>Occasionally we had <strong>init containers</strong> without any preconfigured resources and kubernetes based on it set QoS <strong>Burstable.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd783b9ccd0f" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/why-my-pod-got-oom-kill-exit-code-137-without-reaching-threshold-of-requested-memory-bd783b9ccd0f">Why my pod gets OOMKill (exit code 137) without reaching threshold of requested memory</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynamic setups for PRs with help of ArgoCD and ApplicationSet Controller]]></title>
            <link>https://engineering.lightrun.com/dynamic-deployments-for-prs-with-help-of-argocd-and-applicationset-controller-994deb31e5d4?source=rss-c16c438002cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/994deb31e5d4</guid>
            <category><![CDATA[argo]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[helm]]></category>
            <dc:creator><![CDATA[Yevgeniy Ovsyannikov]]></dc:creator>
            <pubDate>Wed, 23 Mar 2022 21:07:37 GMT</pubDate>
            <atom:updated>2022-08-27T06:58:04.270Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*sAlpVUP-_IOvSK-HrJpFnQ.png" /></figure><p>…</p><h4>Intro</h4><p>We are using ArgoCD widely for our environments and it’s really great and helpful tool. The only downside from my point of view was assumption, that GitOps and specific ArgoCD is only good for “static” environments with declared state on git. But what to do with dynamic spin ups for some quick testing…</p><p>One day we got requirement, to build such setups for pull requests and kill them once pr is merged. There were a lot of considerations how to do it, from committing every-time for every such setup to git, to not implement it with GitOps approach at all.</p><p>Fortunately we have observed <a href="https://argocd-applicationset.readthedocs.io/en/stable/Generators-Pull-Request/">this feature</a> which was added officially to ArgoCD from v.2.3 and it was named — <strong>ApplicationSet Controller</strong></p><h4>Flow</h4><ul><li>ApplicationSet Controller watches for GitHub/GitLab PR with specific label</li><li>Once label is observed, controller creates application and can pass to this application some dynamic values from the PR (branch number, branch name, head_sha)</li></ul><p>Application controller manifest in our case looks this way</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: ApplicationSet<br>metadata:<br>  name: feature-branch            # name of application set<br>  namespace: argocd<br>spec:       <br>  generators:        <br>  - pullRequest:<br>      github:                     # This repo will be scanned<br>        owner: &lt;organization or user name&gt;<br>        repo: &lt;repo name&gt;<br>        tokenRef:<br>          secretName: github-token<br>          key: token<br>        labels:<br>        - setup-saas              # This label should be added to PR<br>      requeueAfterSeconds: 500    <br>  template:<br>    metadata:<br>      name: &#39;app-pr-{{number}}-saas&#39;<br>      namespace: argocd<br>      finalizers:<br>        - resources-finalizer.argocd.argoproj.io<br>    spec:<br>      syncPolicy:<br>        automated:<br>          prune: true<br>          selfHeal: true<br>      project: ondemand<br>      source:<br>        repoURL: &lt;repo where is your helm chart stored&gt;<br>        targetRevision: HEAD<br>        path: &lt;path to the chart&gt;<br>        helm:<br>          parameters:<br>          - name: &quot;image.tag&quot;<br>            value: &quot;pr-{{number}}&quot;<br>          <br>      destination:<br>        server: <a href="https://kubernetes.default.svc/">https://kubernetes.default.svc</a><br>        namespace: ondemand</pre><p>We are not building docker images per every PR, but in this case we need some image to deploy… So we got agreement that if someone needs such dynamic setup he should build image beforehand with another pipeline and add tag it like “pr-&lt;number of opened pr&gt; for example “pr-2737”.</p><p>Afterwards ArgoCD will look for this image and deploy based on helm values. Shutting down appears once label is removed or PR is merged.</p><h4>Demo</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FcsOTb5G_f4o&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcsOTb5G_f4o&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f65730aba2a5c7bc6a76fb42f1c9e8e0/href">https://medium.com/media/f65730aba2a5c7bc6a76fb42f1c9e8e0/href</a></iframe><p>Happy deployment !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=994deb31e5d4" width="1" height="1" alt=""><hr><p><a href="https://engineering.lightrun.com/dynamic-deployments-for-prs-with-help-of-argocd-and-applicationset-controller-994deb31e5d4">Dynamic setups for PRs with help of ArgoCD and ApplicationSet Controller</a> was originally published in <a href="https://engineering.lightrun.com/">Lightrun Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ArgoCD + AWS Secrets Manager]]></title>
            <link>https://engineering.lightrun.com/argocd-aws-secrets-manager-3d625aa917f7?source=rss-c16c438002cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d625aa917f7</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[dc]]></category>
            <category><![CDATA[secrets]]></category>
            <dc:creator><![CDATA[Yevgeniy Ovsyannikov]]></dc:creator>
            <pubDate>Mon, 14 Feb 2022 15:11:31 GMT</pubDate>
            <atom:updated>2022-03-31T07:44:35.887Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*i6DmaC3PzNg04w4i-a-XrQ.png" /></figure><p>GitOps in trends these days, it has pros and cons, but no doubt it gives you clear understanding what is deployed right now and makes CD process much smoother.</p><p>Once we started implementation, right away we have faced with trivial question <strong>“How to store secrets?”.</strong> GitOps approach drives you to have all stuff stored in git, but what to do with a sensitive data?</p><p>We use Kubernetes Secrets for storing secrets and somehow ArgoCD should create update them. (Keep aside that secrets are in the file system and not consumed by application in real time, this is another topic…)</p><p>Luckily Argo developers cared about it and provided flexible API for creating various cool plugins. One of them is <a href="https://argocd-vault-plugin.readthedocs.io/en/stable/">Argo Vault plugin</a> (kudos to the community). Despite of the plugin name, it could also work with <strong>AWS Secrets Manager</strong>. (we decided not couple ourselves with Hashicorp Vault, cause nowadays we a small startup and additional maintenance unneeded).</p><p>So here is the flow:</p><ul><li>Secrets stored in AWS Secrets manager</li><li>ArgoCD deploys helm template for Kubernetes Secret which has annotation with name of secret and placeholders with keys of secrets parsed by Argo Vault Plugin</li></ul><pre>kind: Secret<br>apiVersion: v1<br>metadata:<br>  name: secret<br>  annotations:<br>    avp.kubernetes.io/path: /some/test/secret<br>type: Opaque<br>stringData:<br>  password: &lt;foo&gt;</pre><ul><li>In case you change secret in ASM it’s instantly changing in you Kubernetes without any redeploy.</li></ul><p>Now more details…</p><ol><li>We have to create secret in AWS secrets manager, some basic key:value, for example here was created secret <strong><em>some/test/secret</em></strong> with foo:bar</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wxthH3TLvW8JWQbwgcVbhA.png" /></figure><p>2. Kubernetes nodes should have policy for getting secrets, something like this (modify based your security best practices, resource could be only node/nodes where argocd-server deployed):</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Sid&quot;: &quot;Stmt1644847676679&quot;,<br>      &quot;Action&quot;: [<br>        &quot;secretsmanager:DescribeSecret&quot;,<br>        &quot;secretsmanager:GetResourcePolicy&quot;,<br>        &quot;secretsmanager:GetSecretValue&quot;,<br>        &quot;secretsmanager:ListSecretVersionIds&quot;,<br>        &quot;secretsmanager:ListSecrets&quot;<br>      ],<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Resource&quot;: &quot;*&quot;<br>    }<br>  ]<br>}</pre><p>3. Patch your argocd server deployment with following command, it will add init container which download plugin + adding variables AWS_REGION and AVP_TYPE for server container.</p><p><strong><em>kubectl patch deployment argocd-repo-server -n argocd --patch-file patch.yaml</em></strong></p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: argocd-repo-server<br>spec:<br>  template:<br>    spec:<br>      containers:<br>      - name: argocd-repo-server<br>        env:<br>          - name: AWS_REGION<br>            value: us-east-1<br>          - name: AVP_TYPE<br>            value: awssecretsmanager<br>        volumeMounts:<br>        - name: custom-tools<br>          mountPath: /usr/local/bin/argocd-vault-plugin<br>          subPath: argocd-vault-plugin<br><br>      volumes:<br>      - name: custom-tools<br>        emptyDir: {}<br>      initContainers:<br>      - name: download-tools<br>        image: alpine:3.8<br>        command: [sh, -c]<br><br>        # Don&#39;t forget to update this to whatever the stable release version is<br>        # Note the lack of the `v` prefix unlike the git tag<br>        env:<br>          - name: AVP_VERSION<br>            value: &quot;1.8.0&quot;<br>        args:<br>          - &gt;-<br>            wget -O argocd-vault-plugin<br>            https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v${AVP_VERSION}/argocd-vault-plugin_${AVP_VERSION}_linux_amd64 &amp;&amp;<br>            chmod +x argocd-vault-plugin &amp;&amp;<br>            mv argocd-vault-plugin /custom-tools/<br>        volumeMounts:<br>          - mountPath: /custom-tools<br>            name: custom-tools<br><br>      # Not strictly necessary, but required for passing AVP configuration from a secret and for using Kubernetes auth to Hashicorp Vault<br>      automountServiceAccountToken: true</pre><p>4. Patch ArgoCD configmap and register new plugin (we are using option with helm)</p><p><strong><em>kubectl patch configmap argocd-cm -n argocd — patch-file patch.yaml</em></strong></p><pre>data:<br>  configManagementPlugins: |<br>    - name: argocd-vault-plugin-helm<br>      init:<br>        command: [sh, -c]<br>        args: [&quot;helm dependency build&quot;]<br>      generate:<br>        command: [&quot;sh&quot;, &quot;-c&quot;]<br>        args: [&quot;helm template $ARGOCD_APP_NAME ${helm_args} . | argocd-vault-plugin generate -&quot;]</pre><p>As you see it will parse helm template and output will be also parsed with help of argocd-vault-plugin</p><p>5. Finally lets create some ArgoCD application and bind it to plugin</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: example-app<br>  namespace: argocd<br>  finalizers:<br>    - resources-finalizer.argocd.argoproj.io<br>spec:<br>  syncPolicy:<br>    automated:<br>      prune: true<br>      selfHeal: true<br>  project: default<br>  source:<br>    repoURL: <a href="/service/mailto:git@github.com">git@github.com</a>:somerepo.git<br>    targetRevision: HEAD<br>    path: /app<br>    plugin:<br>      name: argocd-vault-plugin-helm<br>      env:<br>        - name: helm_args<br>          value: -f values.yaml<br>  destination:<br>    server: <a href="https://kubernetes.default.svc/">https://kubernetes.default.svc</a><br>    namespace: default</pre><p>Hope that aggregated information in this article will save your time for research. Cheers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d625aa917f7" width="1" height="1" alt=""><hr><p><a href="https://engineering.lightrun.com/argocd-aws-secrets-manager-3d625aa917f7">ArgoCD + AWS Secrets Manager</a> was originally published in <a href="https://engineering.lightrun.com/">Lightrun Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How collect Java heap dumps from stateless microservices running on Kubernets]]></title>
            <link>https://awstip.com/how-collect-java-heap-dumps-from-stateless-microservices-running-on-kubernets-a1aed5575128?source=rss-c16c438002cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1aed5575128</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Yevgeniy Ovsyannikov]]></dc:creator>
            <pubDate>Wed, 09 Oct 2019 13:48:17 GMT</pubDate>
            <atom:updated>2022-07-20T11:49:19.338Z</atom:updated>
            <content:encoded><![CDATA[<h3>How collect Java heap dumps from stateless microservices running on Kubernetes</h3><h3>Issue</h3><p>Microservices by definition are stateless and ephemeral, it means if some content was created inside and pod was crashed or restarted we will lose it. So it’s kind of problem if you have to gather java heap dumps and analyze them after some disaster.</p><h3>POD termination</h3><p>Let’s try to understand how POD termination works, it will help us not to lose fresh dump:</p><ol><li>A SIGTERM signal is sent to the main process (PID 1) in each container, and a “grace period” countdown starts (defaults to 30 seconds).</li><li>Upon the receive of the SIGTERM, each container should start a graceful shutdown of the running application and exit.</li><li>If a container doesn’t terminate within the grace period, a SIGKILL signal will be sent and the container violently terminated</li></ol><p>It means that by default we have just 30 seconds for creation java heap dump and upload it to the safe place. In most of cases it wont be enough. Luckily Kubernetes provides way for configuring this timer. It calls <strong><em>terminationGracePeriodSeconds </em></strong>and could be configured for your Deployment.</p><h3>Please, take me out of here</h3><p>Now when we have enough time, we will use it for collecting relevant dumps from the pod and upload it to some repository for further analysing.</p><ul><li><em>TL;DR</em></li></ul><ol><li>Pod writes his dump directly to the host (persistent place)</li><li>Every host has special container (dump collector managed by DaemonSet) which is mounted to the same host directory</li><li>Dump collector constantly synchronises all stuff to S3.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VAoy1ayH-P83MM4ozd8scg.png" /></figure><ul><li><em>More details</em></li></ul><ol><li>Every pod should have mounted volume type: <strong><em>hostPath </em></strong>(<strong><em>hostPath</em></strong> type means that volume will be mounted directly to the container’s host and data wont be destroyed after the container’s termination) Be aware that flag <strong><em>DirectoryOrCreate</em></strong> should be set if directory does not exist at the given path, an empty directory will be created by Kubelet. Don’t forget reconfigure your application to store dumps in /dumps</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*NoB8kFxKwJZeoXAqqKomJQ.png" /></figure><p>2. Now when our dump has no risk to be removed after pod’s termination, we need collect it from the host and deliver to some central repository (S3 in our case). Kubernetes provides way of deployment single pod on <strong><em>every</em></strong> node, aka DaemonSet. With help of DaemonSet we will deploy our dummy dump-collector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*UTRvjq2LkRev8l_f6_0Ynw.png" /></figure><p>Code is placed here, you can build docker and upload to your registry: <a href="https://github.com/yevgeniyo/dump-collector/blob/master/Dockerfile"><strong><em>https://github.com/yevgeniyo/dump-collector</em></strong></a></p><p>3. After DaemonSet deployment, dump-collector will sync all stuff to S3 and take care that /dumps folder on the host wont be larger than 50G. Once new dump is detected rsync process upload it to S3 bucket (be sure: 1. that your kubernetes node has permissions put object to S3, we have added it on host role level 2. configure bucket policy for archiving dumps older than one month for example)</p><h3>Conclusion</h3><p>We have achieved 2 major goals:</p><ul><li>We don’t lose our dumps due to container ephemeral character.</li><li>Developers have one-stop source for all dumps generated by every pod of microservice, no need to jump from pod to pod, node to node and look for the dump.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Mk7KlwZ4G_-FnxRbSL-3Q.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1aed5575128" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/how-collect-java-heap-dumps-from-stateless-microservices-running-on-kubernets-a1aed5575128">How collect Java heap dumps from stateless microservices running on Kubernets</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>