<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Swarup Donepudi on Medium]]></title>
        <description><![CDATA[Stories by Swarup Donepudi on Medium]]></description>
        <link>https://medium.com/@swarupdonepudi?source=rss-656d13fde0ca------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*RIq4hjCSxXx09ZYdEMY3VA.jpeg</url>
            <title>Stories by Swarup Donepudi on Medium</title>
            <link>https://medium.com/@swarupdonepudi?source=rss-656d13fde0ca------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:13:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@swarupdonepudi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI agents know the cloud. They don’t know when they’re done.]]></title>
            <link>https://medium.com/@swarupdonepudi/ai-agents-know-the-cloud-they-dont-know-when-they-re-done-d0a06378fb4b?source=rss-656d13fde0ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0a06378fb4b</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Swarup Donepudi]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 06:53:23 GMT</pubDate>
            <atom:updated>2026-11-06T06:53:23.953Z</atom:updated>
            <content:encoded><![CDATA[<h4>A new benchmark gave frontier agents 45 real AWS operations tasks. Not one of them failed for lack of knowledge.</h4><p>AI agents are starting to do real operations work on our cloud accounts. They clean up after retired projects, close access gaps, and fix things that silently stopped working. So I was curious when a new benchmark called CloudOpsBench set out to measure how well they actually do it.</p><p>The answer surprised me. The agents know the cloud really well. What they don’t know is when the job is actually done.</p><figure><img alt="Infographic: Every model knew MediaConvert has queues. None looked. Task: delete every vera2 queue. The agent deleted 2 SQS queues, found 11 other regions empty, and reported “No vera2- queue remains”, but the MediaConvert queue vera2-transcode remains. Across all models, about 80% of failures were misjudged scope, about 20% wrong actions, 0% lack of knowledge. 45 AWS tasks in a simulated account. Source: CloudOpsBench blog, 2 Nov 2026." src="https://cdn-images-1.medium.com/max/1024/1*LqlDbdsqOY1WCdhEh2Yl7g.png" /></figure><h3>What CloudOpsBench tests</h3><p>Most agent benchmarks test coding. There, a test suite tells you whether the change works. Cloud operations are harder to grade. You are changing a shared account that other teams depend on, and some changes can’t be taken back.</p><p>So the authors wrote 45 AWS tasks, the kind of tickets that fill a normal week: decommissioning a project (13 tasks), securing access (12), migrating and deploying services (7), configuring and monitoring (6), and repairing failures (7). Each task is written as intent, the way a ticket is. It doesn’t name the resources or the API calls. The agent has to find them.</p><p>Every attempt runs in a fresh, emulated AWS account, and every model is run on every task four times.</p><h3>The scores</h3><p>The top of the leaderboard is Claude Fable 5.1 in Claude Code, at a 60.6% resolution rate. GPT-6 Astra in Codex is close behind at 58.9%. So the best agent today passes about six in ten attempts at real cloud operations work.</p><p>One more thing I noticed on the leaderboard. The same model, Fable 5.1, scores 43.8% in a different setup: a different agent tool, with a lower reasoning-effort setting. That’s about 17 points from the setup around the model. I think that matters as much as which model you pick.</p><h3>Where they fail</h3><p>The authors sorted every failure into three buckets:</p><ul><li><strong>Knowing:</strong> the agent didn’t know how a cloud service behaves.</li><li><strong>Reasoning:</strong> it knew, but misjudged the task’s scope. It surveyed only part of the account, or treated a resource as out of scope.</li><li><strong>Acting:</strong> it picked the wrong action, broke something, or left the job unfinished.</li></ul><p>Not one failure came from a knowledge gap. About 80% came from misjudging the scope, and about 20% from wrong actions. In the authors’ words:</p><blockquote><em>“knowledge was never the bottleneck”</em></blockquote><p>When I thought about it, that changes what we should be working on. Giving agents more docs won’t fix this. They already know the services. What’s missing is judgment.</p><h3>The queue nobody listed</h3><p>Here is a real run from the benchmark. The task was simple: a project called vera2 is retired, so delete every one of its queues.</p><p>Most of us think of SQS when we hear “queue”. But on AWS, other services keep their own queues too. MediaConvert, the video transcoding service, has queues for its jobs.</p><p>The agent, Claude Fable 5.1, listed the SQS queues, found two and deleted both. Then it scanned eleven other regions, found nothing, and reported that no vera2 queue remained in the account. Every command worked. Its own re-check came back clean.</p><p>But it never listed the MediaConvert queues, and one was still there. The grader checked the account and failed it.</p><p>The authors say every model knew that MediaConvert has queues. Not one attempt, from any model, actually looked. I would have missed this one myself. The agent didn’t lack knowledge. It stopped looking too early, and nothing in its output said anything was missing.</p><h3>The upload that can’t come back</h3><p>The second case is worse, because it can’t be undone.</p><p>The task was a bucket migration: move data from one S3 bucket into another. In the bucket, agents found an in-progress multipart upload, a large file still being uploaded in parts, so not a complete object yet. To them it looked like leftover junk, so they aborted it. That destroyed the data, and the authors say the upload can’t be recovered.</p><p>This is why cloud work is so different from code. If code is wrong, you just revert the commit. If an agent deletes the wrong thing, there’s nothing to go back to.</p><p>So I think the rule for agents on a platform is simple. Before anything irreversible, the agent has to prove it’s safe, or stop and ask.</p><h3>How they decide “done”</h3><p>The part of this benchmark I’d copy first is how it grades.</p><p>When the agent says it’s done, nobody reads its message. A verifier checks the actual state of the cloud account and asks two questions. Did the requested change happen? And is everything outside the task exactly as it was? If the agent fixed the problem but broke something unrelated, the attempt scores zero.</p><p>I think that’s the right definition of done for agents. With my own coding agents, I had to write one down: finished means committed, pushed and proven by a check. It doesn’t mean the agent thinks it worked. At the end of the day, I’d rather check the account myself than trust the agent’s word.</p><h3>What this benchmark doesn’t tell us</h3><p>It’s only 45 tasks, on one cloud, run in a simulated AWS account. The project’s own README says the 42 most recently added tasks have not yet been run with oracle and negative controls, so the authors are still checking that most tasks grade correctly. And the claim that better docs wouldn’t help is hedged in their own words: it “would likely not” have changed the results.</p><p>None of that changes the shape of the finding for me. The gap is in scope and in knowing when to stop, not in what the agent knows.</p><h3>Your first step this week</h3><p>When an agent cleans something up for you, check one service it never listed. Then write down how you’d verify the job is done without reading the agent’s answer.</p><p>I’m curious about your side of this. If you’ve let an agent do real cloud operations work, how did it go? Does your experience match what this benchmark found?</p><p>I made a video on it for my show, Off the Pager:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FdWJWoC2TCUk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DdWJWoC2TCUk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FdWJWoC2TCUk%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/653c452a765491866087c030b8394f41/href">https://medium.com/media/653c452a765491866087c030b8394f41/href</a></iframe><h3>Sources</h3><ul><li>CloudOpsBench leaderboard (v1.0): <a href="https://cloudopsbench.ai/">https://cloudopsbench.ai/</a></li><li>How Well Do Agents Perform on DevOps Tasks? (CloudOpsBench blog, 2 Nov 2026): <a href="https://cloudopsbench.ai/blog/failure-taxonomy">https://cloudopsbench.ai/blog/failure-taxonomy</a></li><li>What cloud operations are, and what we test (CloudOpsBench blog, 13 Oct 2026): <a href="https://cloudopsbench.ai/blog/cloud-operations">https://cloudopsbench.ai/blog/cloud-operations</a></li><li>The tasks, setup scripts and verifiers (GitHub, Apache 2.0): <a href="https://github.com/CloudOpsBench/CloudOpsBench">https://github.com/CloudOpsBench/CloudOpsBench</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0a06378fb4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Flags Just Had Their OpenTelemetry Moment]]></title>
            <link>https://medium.com/@swarupdonepudi/feature-flags-just-had-their-opentelemetry-moment-940f1ac32087?source=rss-656d13fde0ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/940f1ac32087</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[feature-flags]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Swarup Donepudi]]></dc:creator>
            <pubDate>Thu, 08 Oct 2026 18:43:39 GMT</pubDate>
            <atom:updated>2026-10-08T18:45:14.530Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BHlrsbDuprQhUIh9.png" /></figure><p>About a year ago, I was reviewing a plan for a part of Planton that nobody was using yet. My only feedback was this: “This system is not in production yet, so we don’t need to plan for any type of feature flagging. Because at this point, I consider that as a complete overhead.”</p><p>And at that moment, I think I was right. When nobody is using a system, you simply change it. There is nothing to protect.</p><p>Then real users showed up.</p><p>We have built an AI assistant into Planton, the Planton Assistant, and along with it, automations that let organizations hand their workflows to AI agents, so the agents execute them automatically based on external triggers. Based on some internal feedback, we decided it is not ready yet, and we have not released it to any organization.</p><p>As we were getting ready to go to market, the first thing that started to bother me was the fact that the Planton Assistant is scattered across the product, and it was not functional at the moment. So I needed to disable the Assistant. But when I thought about it, this is going to be common from here on. We build stuff that we don’t want to enable in production yet, and only getting things out via releases is not practical. In a lot of cases, we want to enable a feature for certain organizations first.</p><p>So this is the first time we built a feature that requires a careful, slow rollout, and we wanted to have more control.</p><p>To be honest, I had never used a feature flagging system and instrumented it in a product. I didn’t know what the architectural consequences were. I didn’t know what I didn’t know.</p><blockquote><strong><em>Tip:</em></strong><em> Feature flags are overhead until the day real users arrive. From that day, a release is no longer a safe way to hold unfinished work back. If you are about to put your first unfinished feature in front of customers, that is the day to decide how you will switch things on and off, not after.</em></blockquote><p>We initially implemented it using if-else conditions, which is basically the in-house way. And then we started noticing that the code is starting to get all messed up.</p><p>Here is what that looked like. The Assistant had one switch for the whole deployment, and the only signal was whether its configuration existed or not. The comment in our own code said it plainly: “Configuration presence IS the capability switch — there is deliberately no separate enabled flag and no per-organization gating.” Then, for the desktop app, which can run without an account, we had a second, private knob: a rollout percentage that decided which devices got the hosted Assistant.</p><p>So we had two mechanisms with two meanings, and neither could answer the one question I actually had: can I turn this on for this one organization? And every new place the Assistant showed up meant one more check, written a little differently each time.</p><p>That is when it clicked for me. We already had a feature flag system. It was just a bad one. There was no single place to see who has what, no owner, no expiry date, and no clean way to delete it once the feature is done.</p><p>So why did we end up there in the first place? Looking back, I think it was the lack of experience, and the development overhead. We were the ones writing all the code, so bringing in a purpose-built feature flagging system seemed like more work, not less. So we just resorted to doing it via fields in our configuration.</p><p>That is no longer true. We no longer write all the code ourselves; coding agents do a lot of it now. Once that happened, it made sense to actually go with a purpose-built system.</p><p>And it was not just the coding agents. When I look at it now, a few things came together at the same time. Coding agents made the instrumentation cheap. OpenFeature meant I would not be wiring a vendor into my product. Open-source engines meant we could run the flag system on our own infrastructure without adding any cost. And Planton made deploying that open-source software onto Kubernetes a simple thing for us. All these factors together played a crucial role in choosing to go with a proper feature flagging system.</p><p>So I wanted to understand what the options are.</p><blockquote><strong><em>Tip:</em></strong><em> The moment you write your second hand-rolled switch, stop. You already have a feature flag system, just a bad one. And if you skipped a proper one because instrumenting it felt like too much work, do that math again. With coding agents, that work is now the cheap part.</em></blockquote><p>All we wanted to do was a gradual rollout of one single feature. So it is a fair question: why build a whole system for that?</p><p>The answer is that the Assistant is not going to be the last one. Once we have a good feature flag system in place, it becomes much easier for us to build more and more features and gradually roll them out, to provide a more optimal end-user experience. Shipping code and releasing a feature become two different things. We can merge and deploy every day, and decide separately who sees what, and when.</p><p>The first thing I learned while reading up is that “feature flag” means several different things. Pete Hodgson’s article on Martin Fowler’s site, <a href="https://martinfowler.com/articles/feature-toggles.html">“Feature Toggles”</a>, describes four kinds: release, experiment, ops and permission toggles. We built only the first kind, release flags. Which plan a customer is on is a permission, and for us that is an entitlement, a different system with different rules. If a feature eventually becomes part of a paid plan, its release flag retires and an entitlement takes over.</p><p>Then we wrote down one rule: a release flag may hide something new from organizations it is not ready for, with an expiry date and a retirement path. It never chooses between an old and a new implementation. The moment a flag starts choosing between two implementations, you are maintaining both, forever.</p><blockquote><strong><em>Tip:</em></strong><em> Before you write your first flag, write down what a flag is allowed to be. And keep release flags apart from billing plans: “is this ready?” and “did they pay for it?” are different questions.</em></blockquote><p>I already historically knew about <a href="https://launchdarkly.com/">LaunchDarkly</a>. But I knew that instrumenting the product with a vendor’s code was not something I was excited about. Every check in the product would be tied to one company’s SDK.</p><p>Then I learned about <a href="https://openfeature.dev/">OpenFeature</a>, and that was the moment that got me excited, because it felt like an OpenTelemetry moment for feature flags.</p><p>I didn’t know which one came first, so I looked it up. OpenTelemetry came first. It <a href="https://opensource.googleblog.com/2019/05/opentelemetry-merger-of-opencensus-and.html">formed in 2019</a> when OpenTracing and OpenCensus merged, under the Cloud Native Computing Foundation. OpenFeature came in 2022. It was <a href="https://www.dynatrace.com/news/blog/new-openfeature-standard-for-feature-flagging/">launched by Dynatrace</a> together with a group of feature flag vendors, and interestingly, LaunchDarkly was one of them. The <a href="https://www.cncf.io/projects/openfeature/">CNCF</a> accepted it in June 2022, and it became an incubating project in November 2023.</p><p>The OpenTelemetry part is personal for me. I was at KubeCon North America 2019 in San Diego, and that is where I heard about <a href="https://opentelemetry.io/">OpenTelemetry</a> for the first time. It was on the main keynote stage, <a href="https://redmonk.com/kfitzpatrick/2019/11/20/kubecon-north-america-2019-day-2/">with a live demo</a>, and I was in that packed hall. I was super excited, because it felt like a great moment for the observability space. Somebody was finally bringing some standardization into a chaotic space.</p><p>Six years later, at Planton, we decided we are going to adopt the OpenTelemetry framework across the board for our observability. So I already knew what it feels like when a standard sits between your code and the vendor. You instrument once, and the backend becomes a choice you can change later.</p><p>I also have a strong opinion about standards: for a specification to become the standard, there should be a bigger brand behind it. So the moment I saw a standard SDK as part of the Cloud Native Computing Foundation, with real vendors behind it, I was immediately convinced that this is the right move.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sgkZnJgOzqXKZgjc.gif" /></figure><p>The idea is simple. Our code asks one standard API a question, like “is assistant on for this organization?&quot;, and a provider plugs in whichever engine actually answers. In our control plane, exactly one small library imports OpenFeature, and the rest of the product asks that library. The engine can change, and the code asking the question does not.</p><blockquote><strong><em>Tip:</em></strong><em> Code against the standard API, not the engine, and keep even the standard behind one seam in your own code. Then swapping an engine is swapping one provider, not a refactor.</em></blockquote><p>I then explored what the actual runtime would be that my product is going to integrate with, and I had two options: <a href="https://gofeatureflag.org/">GO Feature Flag</a> and <a href="https://flagd.dev/">flagd</a>. Both speak OpenFeature, both are open source, and both run on Kubernetes.</p><p>To be honest, our first plan actually picked flagd. It is part of the OpenFeature project itself, under the CNCF, with maintainers from several companies. Then I asked one question: did we consider GO Feature Flag? When we put them side by side, GO Feature Flag was a better fit for us:</p><ul><li>It has an upstream Helm chart. With flagd, we would have owned the Kubernetes manifests ourselves.</li><li>Its flag files are readable YAML, with rules like org in [&quot;acme&quot;, &quot;globex&quot;].</li><li>It reads flags through the Kubernetes API. In our tests, a flip landed within its 2-second poll, while flagd waited about 75 seconds for the kubelet to sync a mounted file.</li></ul><p>Now, the honest part. GO Feature Flag has basically one maintainer. On <a href="https://github.com/thomaspoignant/go-feature-flag">GitHub</a>, its creator has over 1,600 commits, and the next human contributor has 13. But it felt like the project is being well maintained, even by that one developer. It had at least a hundred commits in the last three months alone, and a new release the week we adopted it.</p><p>To be honest, today, with all the coding agents, I think it is practical to believe that even a one-person team can actually keep an open-source project alive. I believe that because, with a very, very small team, we are able to operate a platform like Planton. From firsthand experience, I believe it is absolutely an okay decision to get convinced and commit to an open-source project like that.</p><p>And OpenFeature is what makes that decision safe. If GO Feature Flag ever stalls, I can swap it with flagd. Leaving means changing one provider and translating one flag file per environment. Not one product check changes.</p><blockquote><strong><em>Tip:</em></strong><em> Judge a small open-source project by your exit cost, not by its headcount. If a standard sits between you and the project, a single maintainer is a risk you can afford.</em></blockquote><p>Here is a story from long before Planton. In 2017, at a startup I was building, the very first time we needed a SQL database, I chose a managed database on AWS. That database cost more than the rest of the system combined, around 80 dollars a month, and that too just for development. On Kubernetes, I could have run a Postgres container with a quarter of a CPU and maybe 100 MB of RAM, because that infrastructure is shareable.</p><p>But I didn’t, because running it on Kubernetes was a whole lot of operational complexity. A few years later, at the end of 2020, we did start running Postgres on Kubernetes. Even then, as an operations engineer, I didn’t have a standard pattern, template or source for it. It’s not as simple as a brew install on your Mac. Helm is supposed to do that, but it&#39;s not all that simple in most cases.</p><p>That gap is exactly what <a href="https://github.com/plantonhq/planton/tree/main/catalog">Planton’s catalog</a> fills today. Every open-source system becomes a building block: a typed, validated catalog item that deploys the same way every time. And that’s how we operate: as long as an open-source project delivers what we need, we put it into the catalog and use it, so the rest of our customers can also benefit from it.</p><p>Our own environments run on close to fifty of our own <a href="https://github.com/plantonhq/planton/tree/main/catalog/kubernetes">Kubernetes catalog items</a>: Postgres with <a href="https://cloudnative-pg.io/">CloudNativePG</a>, Valkey, OpenBao for secrets, OpenFGA for authorization, Temporal for workflows, Tekton, Istio, cert-manager, external-dns, and Prometheus, Loki, Tempo, Grafana and OpenTelemetry to watch it all. Even self-hosted Planton is an item in the catalog, and our own management instance runs on it. In the same week as this work, we tore our environments down and recreated every one of them from the same files.</p><p>To be fair, we don’t run everything ourselves. We still use managed services where they clearly earn it, like hosted sign-in, object storage and, of course, the cloud itself. Open source on Kubernetes is our default, not a religion.</p><p>So, as part of our commitment to dogfooding, we added the flag engines to the catalog first. Actually, four building blocks: <a href="https://github.com/plantonhq/planton/tree/main/catalog/kubernetes/kubernetesgofeatureflag">GO Feature Flag</a> and <a href="https://github.com/plantonhq/planton/tree/main/catalog/kubernetes/kubernetesflagd">flagd</a>, and a separate flag file for each. There are nuances as to why we separated them:</p><ul><li>Flipping a switch never touches the server.</li><li>Permission to flip a switch can be given without permission to change the server.</li><li>Different teams can own different flag files against one server.</li></ul><p>Then we put GO Feature Flag into our own environments: one 65-line chart template for the engine, and a 36-line flag file per environment. From the catalog release to serving flags in production took about 16 hours.</p><p>Even though running GO Feature Flag seemed like an extra component in the stack, because of how Planton simplifies deploying open-source software on Kubernetes for us, the decision became a lot easier. I think about it this way: we shouldn’t just run away from complexity; we should tame that complexity and simplify it, right? An extra component is only expensive when deploying one is a project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xj40-quk_T5P4Jdd.gif" /></figure><p>And our customers get both engines in the catalog. We give them the choice, and we take GO Feature Flag as our choice.</p><blockquote><strong><em>Tip:</em></strong><em> Keep the switches apart from the server that serves them. Deploy the flag file on its own, so flipping a flag never redeploys the engine, and the people who flip flags don’t need the keys to the server.</em></blockquote><p>Each of our environments runs a GO Feature Flag relay next to the control plane, and its flags come from one flag file in git. When I want to turn the Assistant on for one more organization, I add its name to one line in that file and apply it. On our dev environment, we measured it: 26 seconds from the apply, in the same running pod, with zero restarts. No release, no deploy.</p><p>Three decisions make this safe:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qD-oUehWmJGxSgtJ.gif" /></figure><ul><li><strong>The server decides, always.</strong> The control plane evaluates every flag with the caller’s own organization. Browsers never talk to the flag engine.</li><li><strong>Git is the audit log and the admin UI.</strong> Every flip is a reviewed commit. There is no flag database and no second admin console.</li><li><strong>It fails closed.</strong> No flag source, or no answer yet, means everything unreleased is off. A feature showing up for a second where it shouldn’t is worse than a feature showing up a second late.</li></ul><blockquote><strong><em>Tip:</em></strong><em> Fail closed. A late reveal is cheaper than a leak. And let the server decide; a flag a browser can evaluate is a flag someone can flip from the browser.</em></blockquote><p>Most of these are not in any documentation. Each one cost us a debugging session, so hopefully it saves you one.</p><p><strong>The provider gave up after one failure.</strong> If the Java provider’s very first configuration load fails, it never tries again, so a control plane that boots before its flag engine would serve “everything off” forever. We found it by reading the provider’s source, and added a small job that re-installs it. A relay started late is now picked up in about 25 seconds, with no restart.</p><p><strong>The health check asked the wrong port.</strong> A deployed relay answers /health only on its monitoring port, not its API port. Our first version asked the API port, so it would have reported &quot;did not answer&quot; forever. We caught it live, before production depended on it, and fixed it in a release.</p><p><strong>Failures are quiet.</strong> A missing flag file still answers with an empty {} and HTTP 200, and both engines skip a malformed flag with only a log line. Either way, your code silently gets &quot;off&quot;. So our health check names every flag the engine has no entry for, and our flag files are validated against the engine&#39;s own rules before they are applied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TvZNPofBUjmWmMgl.gif" /></figure><p><strong>Hiding things leaves holes.</strong> When you hide a button or a page, something has to stand in its place. I didn’t want that to be left to whoever hides something, so anyone who puts a surface behind a flag must now declare what occupies its place. If they don’t, the code does not compile.</p><blockquote><strong><em>Tip:</em></strong><em> Read your provider’s source code, especially what happens on the first failure. And make sure “off” is never silent: a missing flag should be something your health check can name.</em></blockquote><p>This was one of my first questions before we built anything. What happens when a feature becomes permanent? Do all those checks remain in the codebase forever? What does the community do about it?</p><p>The community’s answer is that flags are inventory with a carrying cost. Pete Hodgson’s article recommends expiry dates and adding the removal task at the same time as the flag. Uber built a whole tool, <a href="https://www.uber.com/blog/piranha/">Piranha</a>, to delete stale flags automatically, and it has removed around two thousand of them.</p><p>But we are in 2026, and coding agents are fully capable of refactoring any kind of code. Deleting a flag is no longer the hard part; an agent does that in minutes. What stays hard is knowing <em>when</em> to delete it, and making sure the deletion is <em>complete</em>. So that is where we invested:</p><ul><li><strong>Every flag has an owner and an expiry,</strong> at most 90 days out, renewable with a reason. A check in our release pipeline fails on any flag with no owner or a past expiry.</li><li><strong>Checks live only at a few seams,</strong> and every flag is declared once in a typed registry. Delete it there, and the compiler hands you every site.</li><li><strong>Retirement is a written procedure an agent runs,</strong> and we rehearsed it before we needed it.</li><li><strong>Retirement takes two releases.</strong> First the consoles stop asking, then the server check goes. Our desktop app updates on its own schedule, and doing it in one release would hide a released feature from every older desktop.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*849ZwFQvjfWGOcGb.gif" /></figure><p>Every flag ends one of three ways: retired, abandoned (the feature’s code is deleted), or graduated into a paid-plan entitlement.</p><blockquote><strong><em>Tip:</em></strong><em> Give every flag an owner and an expiry the day you create it, and let CI enforce both. Keep the checks at a few seams, so retiring a flag is mechanical work an agent can finish in one go.</em></blockquote><p>Here is everything above, in one place:</p><ol><li><strong>Flags are overhead until real users arrive.</strong> Then decide how you switch things on and off.</li><li><strong>Your second hand-rolled switch is the signal.</strong> And with coding agents, a proper system is the cheap part.</li><li><strong>Write down what a flag may be.</strong> Hide something new; never choose between old and new.</li><li><strong>Keep release flags apart from billing plans.</strong></li><li><strong>Code against OpenFeature,</strong> behind one seam in your own code.</li><li><strong>Judge an open-source project by your exit cost,</strong> not its headcount.</li><li><strong>Deploy the flag file apart from the engine.</strong></li><li><strong>Let the server decide, fail closed,</strong> and keep flags in git.</li><li><strong>Read your provider’s source,</strong> and make sure a silent “off” shows up in your health checks.</li><li><strong>Give every flag an owner and an expiry,</strong> and let CI enforce both.</li></ol><p>Once this is in place, every feature after it ships dark and rolls out one organization at a time. For us, the Assistant was only the first.</p><p>Both flag engines, and their flag files, are in our open-source catalog, so you can run the same building blocks on your own infrastructure: <a href="https://github.com/plantonhq/planton/tree/main/catalog/kubernetes/kubernetesgofeatureflag">GO Feature Flag</a> and <a href="https://github.com/plantonhq/planton/tree/main/catalog/kubernetes/kubernetesflagd">flagd</a> in the Planton catalog.</p><p><em>Originally published at </em><a href="https://planton.ai/blog/feature-flags-opentelemetry-moment"><em>https://planton.ai</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=940f1ac32087" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Infrastructure as Data(IaD) — Coincidence?]]></title>
            <link>https://medium.com/@swarupdonepudi/infrastructure-as-data-iad-coincidence-2ca6c70f896e?source=rss-656d13fde0ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ca6c70f896e</guid>
            <category><![CDATA[cloud-infrastructure]]></category>
            <category><![CDATA[developer-productivity]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Swarup Donepudi]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 10:50:55 GMT</pubDate>
            <atom:updated>2024-10-08T10:50:55.255Z</atom:updated>
            <content:encoded><![CDATA[<h3>Infrastructure as Data(IaD) — Coincidence?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KN9PhXk3hUmN2WIj5hv5ig.png" /></figure><p>Recently, <a href="https://medium.com/u/ccdbe9a99319">Brian Grant</a>, the lead architect of the Kubernetes Resource Model, shared insightful thoughts on the concept of <em>Infrastructure as Data (IaD)</em> in his <a href="https://itnext.io/what-is-infrastructure-as-data-389a69f3709c">blog post</a>. He highlighted that while <strong>Infrastructure as Code (IaC)</strong> is a well-understood term, <strong>IaD</strong> isn’t as widely or consistently used. Brian explored how tools like Ansible, Idem, Crossplane, and even Kubernetes fit into the IaD paradigm by allowing users to describe infrastructure in simple, machine-readable data formats like YAML or JSON.</p><p>One of the key questions Brian raised is whether using data formats is enough to distinguish IaD from IaC. He pointed out that many IaC tools also use data serialization formats and that the line between code and data can often blur, especially when templates or parameter substitutions are involved.</p><p>It feels to me like a coincidence just when I am getting ready to write up about an open-source project I have been working on has a lot in common with the concept of IaD. It immediately resonated with me and I decided to scratch the idea of creating a well-written launch like post and just write up about it here. The <strong>ProjectPlanton</strong> is on github — <a href="https://github.com/plantoncloud/project-planton">https://github.com/plantoncloud/project-planton</a>. There is also a good write up on the background of the inspiration and design choices for the project can be explored at <a href="https://project-planton.org/about">https://project-planton.org/about</a>. The project was created with the primary goal of helping developers become more productive by simplifying the deployment process. Our aim is to provide a tool that reduces complexity and lowers the barrier to entry for deploying cloud-native applications. Now that ProjectPlanton is ready for wider adoption, we are eager to share it with the community very soon.</p><p>ProjectPlanton resonates deeply with Brian Grant’s idea of Infrastructure as Data. Here’s how:</p><p><strong>- Independent Data and Logic:</strong> In ProjectPlanton, the infrastructure is defined purely as data in YAML manifests, while the logic resides independently in Pulumi modules. This separation ensures that the <strong>data (desired state)</strong> and the<strong> logic (implementation details)</strong> are decoupled. The Pulumi modules act like Kubernetes controllers in our framework, processing the data and executing the necessary actions to achieve the desired state.</p><p><strong>- Client-Only Framework:</strong> Unlike systems like Kubernetes or Crossplane that require server-side components, ProjectPlanton intentionally avoids adding this complexity. We leverage Pulumi’s state backend to maintain the state of the infrastructure. This approach allows us to reconcile the desired state with the actual state <strong>on-demand</strong>, without needing a persistent server-side component. This design lowers the barrier to entry and simplifies the deployment process for developers.</p><p><strong>- Data-Centric Configuration:</strong> By modeling deployment components as data structures using Protobuf APIs and buf schema registry, we emphasize a clear separation between what the infrastructure should be and how it gets implemented. Users provide the “what” in simple YAML manifests, and the Pulumi modules handle the “how.”</p><p><strong>- Minimal Abstractions:</strong> This framework also avoids unnecessary abstractions that could introduce a learning curve. By keeping everything client-side and data-driven, the framework makes it easier for developers to adopt without the overhead of understanding complex systems or deploying additional infrastructure.</p><h3>Bridging the Gap Between IaD and IaC</h3><p>Brian Grant questioned whether Infrastructure as Data deserves its own term and how it differs from Infrastructure as Code. With ProjectPlanton, we believe that treating infrastructure specifications purely as data offers significant benefits:</p><p><strong>- Simplified Reconciliation:</strong> Since the infrastructure is defined as data, and the logic is encapsulated in Pulumi modules, reconciling the desired state with the actual state becomes more straightforward. Pulumi’s ability to manage state and perform updates on-demand ensures that the infrastructure remains consistent without continuous server-side monitoring.</p><p><strong>- Reduced Complexity:</strong> By eliminating the need for server-side components like Kubernetes controllers, we reduce operational complexity. Developers can focus on defining their infrastructure in data formats without worrying about managing additional services.</p><p><strong>- Enhanced Clarity and Maintainability:</strong> Separating data from logic makes the system more transparent. Changes to infrastructure configurations are easier to track and understand when they are defined as data, and the implementation logic remains consistent within the Pulumi modules.</p><h3>Conclusion</h3><p>There is little doubt that the most effective way to manage infrastructure at scale is by treating infrastructure configuration data just like any other application data. Consistency, scalability, and ease of management across complex systems are promoted by this approach. While storing these configurations as YAML files in a Git repository is a great starting point, the eventual storage of them in proper databases with querying capabilities is recommended to enhance accessibility and operational efficiency.</p><p>By embracing the principles of Infrastructure as Data, the deployment process for developers is aimed to be simplified and standardized by ProjectPlanton. The possibilities this method offers are exciting, and it is eagerly anticipated how it will evolve with community collaboration and feedback.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ca6c70f896e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Postgres Databases on Kubernetes & Save $$$]]></title>
            <link>https://medium.com/plantoncloud/postgres-databases-on-kubernetes-save-bd8238669da5?source=rss-656d13fde0ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd8238669da5</guid>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[cloud-cost-optimization]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Swarup Donepudi]]></dc:creator>
            <pubDate>Fri, 07 Apr 2023 01:45:17 GMT</pubDate>
            <atom:updated>2023-04-07T01:45:17.526Z</atom:updated>
            <content:encoded><![CDATA[<p>Are you using Postgres databases to in your application? Kubernetes is quickly becoming the go-to platform for running databases. With its scalability, reliability and ease of use, Kubernetes provides an ideal environment to run databases such as PostgreSQL in the cloud. Whether you are using Amazon Web Services (AWS) or other cloud providers, Kubernetes can help you manage your database deployments with ease. In this article, we will discuss some of the advantages of running databases on Kubernetes and how it can help make managing your database deployments simpler and more efficient.</p><p>If you’re on AWS or Google Cloud, you will probably be using something like AWS RDS or Google CloudSQL to create and manage your databases. The biggest benefit of using AWS RDS or Google Cloud SQL is that you don’t really have to be a DBA in order to manage your database. All of the DBA knowledge is already built into these services and that is the value they offer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/0*NR8EzWpOsI4ZODNb.png" /></figure><p>It’s very convenient. I’ve done it in the past. There are definitely a lot of advantages to doing so. The advantages are more obvious when compared to managing your own installation of Postgres database on a virtual machine, because you will have to deal with all the operational aspects like figuring out how to take backups and all all other operational aspects.</p><p>But there is other ways to achieve the same result. These alternatives are more modern and more Cloud Native. So if you are thinking of Cloud Native for your micro services, or if you have already been using Cloud Native ecosystem for running your micro services, you are probably be using Kubernetes. There are now Kubernetes operators that allow you to run your Postgres databases on Kubernetes itself. You can enjoy similar benefits to those offered by AWS RDS or similar services.</p><ul><li>Zalando Operator: <a href="https://github.com/zalando/postgres-operator">https://github.com/zalando/postgres-operator</a></li><li>CrunchyData Operator: <a href="https://github.com/CrunchyData/postgres-operator">https://github.com/CrunchyData/postgres-operator</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aod-w-3C2v4D3K86.png" /></figure><p>These operators take care of all the operational aspects, just like the AWS RDS would, allowing you to only focus on building your application. There are a few important advantages of using this approach that will simplify the operational overhead and infrastructure costs. First, let’s focus on the cost aspect.</p><p>Lets assume that you have 4 micro services running on your AWS EKS managed Kubernetes Cluster and a Postgres Database with 2 CPUs and 6GB of Memory(db.t3.medium) running on AWS RDS. You Kubernetes Clusters has one virtual machine(EC2 Instance) running your micro services with Cluster Autoscaler enabled. Here is a rough sketch of how the micro services would have been scheduled. Assuming that the one worker node has 8 cores of CPU and 12GB of Memory on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a4btIN3qyind7Zy1-c_OIQ.png" /><figcaption>Utilized Resources</figcaption></figure><p>Kubernetes does a really good job at bin packing to maximize the CPU and Memory resources available on the EC2 instance worker nodes of your cluster. In this allocation, we have assigned all four micro services to a single node with 8 cores of CPU and 12 GB of memory. Each micro service requires 1 core of CPU and 0.5 GB of memory, which means that the total resources required by all micro services are within the available resources on the node. After allocating the required resources to each micro service, we have the following resources left on the node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RrLFQBGuRVkjdz7EgVf-Bg.png" /><figcaption>Unused Resources</figcaption></figure><p>This means that there are 4 cores of CPU and 8 GB of memory available on the node that can be allocated to additional micro services or other applications. These unused resources are already paid. So you’re essentially paying for the compute that is unused on your Kubernetes worker nodes.</p><p>You’re also paying for the compute resources for the databases that you are running on AWS RDS. As of April 2023, for the US East (N. Virginia) region, the pricing for the db.t3.medium instance type (which has 2 vCPUs and 6 GB of memory) is $0.172 per hour or approximately $124.80 per month.</p><p>If you spend some time figuring out <a href="https://postgres-operator.readthedocs.io/en/latest/administrator/">how to run your Postgres databases on your Kubernetes clusters</a>, then you can move those databases into your Kubernetes. So these databases basically make use of the under utilized resources on your Kubernetes worker nodes. So you’re not paying money for RDS service anymore. You are better utilizing the hardware resources that you have already. You could save that $124.80 are your instant savings if you move your database to Kubernetes. The savings will start to be more visible as your database infrastructure footprint starts to grow.</p><h3>Unifed Operations</h3><p>The next benefit of running your databases on Kubernetes is that you can replace two different methods of managing your infrastructure i.e Terraform and Kubernetes Manifests into One. You might have been using TerraForm to manage your AWS RDS instances, whereas in your Kubernetes, you’re using some sort of Kubernetes API client like helm or kubectl or some other tool to manage your micro services, deployment manifests.</p><h3>Terraform for RDS Instance</h3><pre>resource &quot;aws_db_instance&quot; &quot;example&quot; {<br>  allocated_storage    = 50<br>  engine               = &quot;postgres&quot;<br>  engine_version       = &quot;13.3&quot;<br>  instance_class       = &quot;db.t3.medium&quot;<br>  name                 = &quot;example-postgres-db&quot;<br>  username             = &quot;admin&quot;<br>  password             = &quot;password&quot;<br>  db_subnet_group_name = &quot;example-subnet-group&quot;<br>  vpc_security_group_ids = [<br>    aws_security_group.example.id,<br>  ]<br>}</pre><h3>Kubernetes Manifest for Database</h3><pre>apiVersion: acid.zalan.do/v1<br>kind: postgresql<br>metadata:<br>  name: my-app-db<br>  namespace: my-app-db<br>spec:<br>  numberOfInstances: 1<br>  patroni: {}<br>  podAnnotations:<br>    postgres-cluster-id: db<br>  postgresql:<br>    parameters:<br>      max_connections: &quot;2000&quot;<br>    version: &quot;14&quot;<br>  resources:<br>    limits:<br>      cpu: 4<br>      memory: 6Gi<br>    requests:<br>      cpu: 4<br>      memory: 6Gi<br>  teamId: my-app<br>  volume:<br>    size: 10Gi</pre><p>Overall, running databases on Kubernetes offers a number of advantages in terms of cost savings and infrastructure management. By leveraging the unused resources available on your Kubernetes worker nodes, you can save money by eliminating AWS RDS costs and also reduce complexity by using one tool to manage both Terraform and Kubernetes Manifests. This approach could be especially beneficial for businesses that are looking to increase their database footprint as they will start seeing more significant cost savings over time. With all these benefits in mind, it’s worth considering if running Postgres databases on Kubernetes is right for your business.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd8238669da5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/plantoncloud/postgres-databases-on-kubernetes-save-bd8238669da5">Postgres Databases on Kubernetes &amp; Save $$$</a> was originally published in <a href="https://medium.com/plantoncloud">PlantonCloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup Splunk on Kubernetes]]></title>
            <link>https://medium.com/@swarupdonepudi/setup-splunk-on-kubernetes-94716dab0e9?source=rss-656d13fde0ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/94716dab0e9</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[enterprise-technology]]></category>
            <category><![CDATA[splunk]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Swarup Donepudi]]></dc:creator>
            <pubDate>Tue, 06 Feb 2018 09:30:27 GMT</pubDate>
            <atom:updated>2018-02-06T09:30:27.191Z</atom:updated>
            <content:encoded><![CDATA[<p>To my surprise I did not find a single medium story explaining how to set up centralized logging using managed <a href="https://www.splunk.com/en_us/resources/why-splunk-cloud.html">Splunk on cloud</a> for <a href="https://en.wikipedia.org/wiki/Microservices">micro services</a> running on <a href="https://kubernetes.io/">kubernetes</a> clusters. Our team manages close to 10 kubernetes clusters, most of which are multi-tenant while some are single-tenant clusters. We were using <a href="https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html">journald</a> as the docker log driver. Also, it was required by the development teams to make sure that the micro service that they develop has to write all of the logs that they would like to find on splunk, be written to <a href="http://www.linfo.org/standard_output.html">standard output</a>. Journald was chosen as the log driver long before this project was undertaken. Simple reason for choosing journald as the log aggregator was that we could use the powerful journalctl CLI tool to get to our logs and also jourland is being adopted by almost all of *unix platforms. We were running kubernetes on <a href="https://coreos.com/why/">CoreOS</a> operating system. Going into the project I assumed that this problem has probably been solved by a bunch of engineers already. Apparently according to <a href="https://github.com/kubernetes/kubernetes/issues/24677">this</a> github issue it is not such an easy problem after all. However, I found <a href="https://www.splunk.com/blog/2015/04/30/integrating-splunk-with-docker-coreos-and-journald.html">this</a> blog on splunk.com super helpful, which explained exactly what I was looking for. I am reusing the same pictures available on the same blog, since they perfectly represent the design of the setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*UdU8FZJWv4jrjzmZ5lwMzQ.png" /></figure><p>In the above infographic each Application corresponds to a micro service running as a <a href="https://www.docker.com/">docker</a> container on each kubernetes worker <a href="https://kubernetes.io/docs/concepts/architecture/nodes/">node</a>. Since journald is the docker logging driver, it captures all of the events written by the applications to standard output. This is all explained in the blog on splunk.com that I referenced earlier.</p><p>I am going to explain the process of setting up each of the below mentioned components required to take log events generated by docker container in a separate section. However, there are a few problems with this seemingly simple setup, which I will explain in these sections.</p><ol><li>Convert journald logs from binary to JSON format</li><li>Splunk Deployment Server</li><li>Splunk Forwarder on each node</li></ol><h4>Convert journald logs from binary to JSON format:</h4><p>Splunk can not read the default binary format of journald , you should write to a non binary file and then forward to splunk cloud. Most of the problems that I have faced while I was trying to stabilize the log aggregation originated from this conversion. I will explain the problems that I have faced in this process in my next story. Assuming that everything in life is simple, the following command should take care of constantly converting the binary journal logs and write them to a readable json file on disk.</p><pre>journalctl — no-tail -f — since “$(date — date=”3 minutes ago” +”%F %H:%M:%S”)” -o json &gt; /var/splunk/journald.json</pre><p>However, since this should run continuously, this needs to be run as a <a href="https://en.wikipedia.org/wiki/Systemd">systemd</a> unit. Here is the systemd unit that I used to set this up nicely.</p><pre>Unit file : /etc/systemd/system/journald-to-json-for-splunk.service</pre><p>Contents of the file:</p><pre>enable: true<br>command: start<br>content: |<br>[Unit]<br>Description=Write json formatted journald logs to /var/splunk/journald for splunk forwarder.<br>[Service]<br>EnvironmentFile=/etc/environment<br>ExecStartPre=/usr/bin/mkdir -p /var/splunk<br>ExecStart=/bin/bash -c ‘/usr/bin/journalctl — no-tail -f — since “$(date — date=”3 minutes ago” +”%F %H:%M:%S”)” -o json &gt; /var/splunk/journald.json’<br>Restart=always<br>RestartSec=10</pre><p>Start the systemd unit</p><pre>systemctl start journald-to-json-for-splunk</pre><p>As long as the systemd unit is running, the journald logs will be converted to json format and are written to <em>/var/splunk/journald.json</em> file.</p><h4>Setup Splunk Deployment Server</h4><p>Up until this point we have not really done anything that would take the logs from the host machine and then forward them to splunk cloud. There are two steps that need to be completed in order to forward logs from a machine to splunk cloud. One of them is setting up a <a href="http://docs.splunk.com/Documentation/Splunk/7.0.2/Updating/Deploymentserverarchitecture">Splunk Deployment Server</a>. In order for an an agent(splunk forwarder) to send logs from a host machine to splunk cloud, the agent needs the following information.</p><ol><li>What is the splunk cloud url?</li><li>Authentication required to authorize itself with splunk cloud.</li><li>What file to watch for?</li><li>What is the index on splunk cloud?</li><li>Any metadata to add to each event?</li><li>Any filters to apply before sending?</li></ol><p>I call all of this information as configuration. While some of this configuration might same for every single forwarder, some configuration is different. For instance #1, #2, #3 is same for all the forwarders. #4, #5 and #6 could be different for different forwarders. It is more likely that a group of forwarders will have same exact configuration. Managing this configuration is made easy is by a special piece of software from splunk called as <em>“Deployment Server”</em>. Running a deployment server is optional though. You can totally add this configuration on to a “Forwarder” directly. The below infographic from splunk docs, that explains how Deployment Server distributes this configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*51I0piA0cgJPIO0il0ML8w.png" /></figure><p>We dockerized Splunk Deployment server and pushed the docker image to docker registry using CI CD pipeline. All of the configuration required by deployment server is passed during the deployment. This dockerized splunk deployment server is deployed as a “Deployment” resource on kubernetes. Optionally you can connect to the web interface of this splunk deployment server. If you are planning on doing that, then do not run more than one instances of as kubernetes does not provide session stickiness, unless you configure it to.</p><h4>Setup Splunk Forwarder</h4><p>This is probably the most important of all the three components. Splunk forwarder does almost all of the heavy lifting in forwarding the logs from host machine to splunk cloud. All of the configuration required by the splunk forwarder is received from deployment server. However, when a forwarder is started, it needs to know which deployment server to talk to in order to get the configuration. We have dockerized splunk-forwarder too. The image is built and pushed to internal docker registry using CI CD process.</p><p>Every time a new forwarder connects to a deployment server, the deployment server will identify the class of the client based on the name. After identifying the group/server-class to which the forwarder belongs, then deployment server will forward that configuration to the forwarder. Here is the kubernetes manifest files that are required to make it all work.</p><pre>##########################<br># Splunk Deployment Server<br>##########################<br>apiVersion: extensions/v1beta1<br>kind: Deployment<br>metadata:<br>  name: splunk-deployment-server<br>  namespace: splunk<br>  labels:<br>    app: splunk-deployment-server<br>spec:<br>  replicas: 1<br>  revisionHistoryLimit: 10<br>  template:<br>    spec:<br>      containers:<br>      - name: splunk-deployment-server<br>        image: my-splunk-deployment-server:latest<br>        env:<br>        - name: SPLUNK_INDEX<br>          value: myindex<br>        ports:<br>          - name: splunk-api<br>            containerPort: 8089<br>            protocol: TCP<br>          - name: splunk-web<br>            containerPort: 8000<br>            protocol: TCP</pre><p>Splunk Deployment server is a simple kubernetes deployment. Splunk Forwader is a daemonset. However, setting up splunk forwarder required a complicated setup which includes adding an <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/">init container</a>. This is because, the first time a forwarder connects to a deployment server, the deployment server will send the configuration to the forwarder. However, in order for that configuration to work on forwarder, the splunk daemon in forwarder needs to be restarted. Since, splunk daemon is PID 1 in the splunk forwarder docker container, the container dies and kubernetes will spin up a new one instantly. The configuration that deployment server sent to forwarded gets destroyed along with the container. This make splunk-forwarders semi stateful. Since the configuration is not present in a newly spun up container, the forwarder will contact the deployment server and this results in a cyclic behavior. To avoid this situation we add kubernetes init container that stores the configuration between container restart/recreations.</p><pre>###############################################################################<br># Splunk Forwarder Daemonset<br>###############################################################################<br>---<br>kind: DaemonSet<br>apiVersion: extensions/v1beta1<br>metadata:<br>  name: splunk-forwarder<br>  namespace: splunk<br>spec:<br>  template:<br>      annotations:<br>        pod.beta.kubernetes.io/init-containers: &#39;[<br>          {<br>            &quot;name&quot;: &quot;wait-for-deployment-server&quot;,<br>            &quot;image&quot;: &quot;busybox&quot;,<br>            &quot;imagePullPolicy&quot;: &quot;Always&quot;,<br>            &quot;command&quot;: [&quot;sh&quot;, &quot;-c&quot;, &quot;until ping deployment-server-internal-url -c 1; do sleep 3; done;&quot;]<br>          },<br>          {<br>            &quot;name&quot;: &quot;splunk-forwarder-initializer&quot;,<br>            &quot;image&quot;: &quot;my-splunk-forwarder:latest&quot;,<br>            &quot;imagePullPolicy&quot;: &quot;Always&quot;,<br>            &quot;env&quot;:[<br>              {<br>                &quot;name&quot;: &quot;CLIENT_NAME_PREFIX&quot;,<br>                &quot;value&quot;: &quot;example-client&quot;<br>              },<br>              {<br>                &quot;name&quot;: &quot;CLIENT_NAME_SUFFIX&quot;,<br>                &quot;value&quot;: &quot;initializer&quot;<br>              },<br>              {<br>                &quot;name&quot;: &quot;DEPLOYMENT_SERVER_HOST&quot;,<br>                &quot;value&quot;: &quot;deployment-server-internal-url&quot;<br>              },<br>              {<br>                &quot;name&quot;: &quot;DEPLOYMENT_SERVER_PORT&quot;,<br>                &quot;value&quot;: &quot;32004&quot;<br>              }<br>            ],<br>            &quot;volumeMounts&quot;: [<br>                {<br>                  &quot;name&quot;: &quot;forwarder-config&quot;,<br>                  &quot;mountPath&quot;: &quot;/opt/splunk/etc&quot;<br>                },<br>                {<br>                  &quot;name&quot;: &quot;forwarder-var&quot;,<br>                  &quot;mountPath&quot;: &quot;/opt/splunk/var&quot;<br>                }<br>            ]<br>          }<br>        ]&#39;<br>    spec:<br>      hostNetwork: true<br>      volumes:<br>      - name: var-splunk-journald-json<br>        hostPath:<br>          path: /var/splunk/journald-json<br>      - name: forwarder-var<br>        emptyDir: {}<br>      - name: forwarder-config<br>        emptyDir: {}<br>      containers:<br>      - name: splunk-forwarder-container<br>        image: my-splunk-forwarder:latest<br>        env:<br>        # This variable is required to register the forwarder with deployment server<br>        - name: CLIENT_NAME_PREFIX<br>          value: example-client<br>        - name: CLIENT_NAME_SUFFIX<br>          valueFrom:<br>            fieldRef:<br>              fieldPath: spec.nodeName<br>        # This variable is required to register the forwarder with deployment server<br>        - name: DEPLOYMENT_SERVER_HOST<br>          value: deployment-server-internal-url<br>        - name: DEPLOYMENT_SERVER_PORT<br>          value: &quot;32004&quot;<br>        volumeMounts:<br>        - name: var-splunk-journald-json<br>          mountPath: /opt/splunk/journald.json<br>        - name: forwarder-config<br>          mountPath: /opt/splunk/etc<br>        - name: forwarder-var<br>          mountPath: /opt/splunk/var<br>        ports:<br>          - name: splunk-daemon<br>            containerPort: 8089<br>            protocol: TCP<br>          - name: event-collector<br>            containerPort: 8088<br>            protocol: TCP<br>          - name: nw-input<br>            containerPort: 1514<br>            protocol: TCP</pre><p>There is a lot more details associated with respect to host networking. port configurations and volume mounts. Size of this story might be blown out if every detail is documented here. I will try to write more posts around all of the several problems that I had to deal with in the process of stabilizing splunk logging.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94716dab0e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cheaper by the Dozen, not so!!]]></title>
            <link>https://medium.com/@swarupdonepudi/cheaper-by-the-dozen-not-so-248b0a5a7ee7?source=rss-656d13fde0ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/248b0a5a7ee7</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[startup-lessons]]></category>
            <category><![CDATA[startup-life]]></category>
            <dc:creator><![CDATA[Swarup Donepudi]]></dc:creator>
            <pubDate>Mon, 29 Jan 2018 09:22:53 GMT</pubDate>
            <atom:updated>2018-01-29T21:18:30.578Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently joined a group of Software Engineers to work on an idea which we all hope, would eventually turn out to be a successful startup. This group is a good mix of Application Developers, Infrastructure Automation Engineers and Devops Engineers. One very interesting aspect of this group is that it is a completely distributed team. Few engineers are in India while few others work from different states in the US. So, soon after starting to work together we started to see problems with setting up all of the tools that each of us have access to at our own work places and expect to be available easily for use. The toolset that I am alluding to includes but not limited to collaboration tools like slack or hipchat, online meeting with audio, video and screen sharing capabilities, project management tools like JIRA, wiki management tool like confluence or share point and the list goes on..So, I took it on me to set up these tools for the entire team. What we ended up with was a good tool set, all of which was procured on cloud using monthly subscriptions. It took a lot of time for me to evaluate various alternatives for each of the tools and I ended up with the ones that either have better reviews, better user interface, better integration with other tools or better pricing model. I thought that several other people with similar requirements must have been going thru the same exact process as I write this story or as you read this story. First I will list all of the tools that I would like to share and then explain the reason for choosing the toolset.</p><ol><li>G Suite</li><li>Slack</li><li>Atlassian Cloud JIRA</li><li>Atlassian Cloud Confluence</li><li>Bitbucket Cloud</li><li>AWS</li><li>Keeper Security</li><li>JumpCloud</li></ol><p>I will now explain the reason for choosing each of these tools.</p><h3>G Suite</h3><p>Probably the best professional email service available for SMBs. The pricing model for G Suite is a life saver for a small startup. We were able to setup our professional email-id by paying a mere $5 per user per month. This $5 also includes 20GB of Google Drive storage and also includes subscription for meet.google.com service, online meeting tool.</p><p>10 Users = $50 / month</p><p>11 Users = $55 / month</p><h3>Slack</h3><p>Slack is the most preferred IM tool for startup these days and rightfully so. It has a good pricing model which follows <a href="https://kopywritingkourse.com/the-three-pronged-pricing-technique/">three pronged</a> pricing technique. While the basic free plan gave us what we absolutely needed, we ended up purchasing the premium license, I will explain the reasons for doing so at the end.</p><p>10 Users = $150 / month</p><h3>Atlassian Cloud</h3><p>Atlassian has created an amazing set of tools that every organization either big or small, can fully utilize in order to support every phase of the Application Lifecycle Management. However, Atlassian has a very interesting pricing model for all of its cloud products. Atlassian charges you a flat fee of $10 per service / per month if you have 10 or below users. Which works very well for a startup of size smaller than 10 people. However, starting from the 11th user, Atlassian charges $7 /user/month. So, your monthly bill would go up from $10 to $77 when you add 11th user. So, it actually gets super expensive as you get close to the “DOZEN”. As long as you are below 10, you can use Atlassian Cloud products by paying just $10 per service. I highly recommend this product even if you have users more than 10. The $77 that you are charged is totally worth it.</p><h4>JIRA</h4><p>Every project starts of with project planning and there is probably no other tool that comes event remotely close to what JIRA can give you.</p><p>10 Users = $10 / month</p><p>11 Users = 11 * $7 /user / month = $77</p><h4>Confluence</h4><p>Every organization requires a online wiki tool to create documents and share the documents with other people in the organization. Again, I dont think there is any other tool that does as good of a job as confluence. Pricing model for confluence is exactly same as JIRA</p><p>10 Users = $10 / month</p><p>11 Users = 11 * $7 /user / month = $77</p><h4>Bitbucket</h4><p>The very next thing that every developer wants to start is to write code. Every organization will need a centralized code repository. While there are many, I chose Bitbucket Cloud purely because its ‘Bitbucket Pipelines’ feature. I do not have to setup my own continuous integration server because bitbucket provides both SCM and CI features all in one place. Also, it is pretty cheap. If you have a team of less than 5 engineers you pay nothing. You can have unlimited private repositories. Even if you have more than 5, you basically have to pay $2 per user per month which is not too bad.</p><p>Pricing for Bitbucket Pipelines is different from the money you pay for user subscription. You will have to pay for the build minutes that you use. Every team gets 50 minutes of free build time. If you run out of those 50 minutes which you will run out in 50 minutes, obviously, your credit card will be charged $10 for every 1000 minutes.</p><p>10 Users = $10 / month</p><p>11 Users = 11 * $7 /user / month = $77</p><p>First 1050 build minutes = $10</p><p>Bitbucket also announced “Deployments” in Alpha.</p><h3>Amazon Web Services</h3><p>I do not want to pretend that I will write yet another post about the awesomeness of AWS for any startup. Of course, we are starting to see more and more cloud providers are offering similar services at competitive prices, we chose AWS its service catalog is more complete and integration between multiple services is seamless. I would be lying if I promise you folks that we will be running on AWS for the rest of our lives. But that is what we are using right now.</p><h3>Keeper Security</h3><p>This is an interesting service that not many people would like to subscribe to at the beginning phases of a startup. This tool offers shared secrets management solution. Shared credentials like database passwords, AWS private keys etc can be elegantly and securely shared with other members on the team with out having to copy paste them in IM tool (Slack). The license is not very expensive when compared to other tools in the market. There is one other strong reason to choose Keeper over other tools that offer similar functionality. You will find the reason while reading about JumpCloud in this post.</p><p>10 Users = $300 / year</p><h3>JumpCloud</h3><p>I saved the best one for the last. While it is awesome to have all the tools that enable engineers to focus on what they are really required to do in order to build the product, no body wants to remember 10 different logins for 10 different systems. That is where LDAP systems like Active Directory make every ones life easier. JumpCloud is a Directory as a Service solution that gives you the features of a directory service on cloud with monthly pricing. The best part of JumpCloud is that it is completely free for up to 10 users forever. So, if your startup is less than 10 users then you can use JumpCloud’s services all for free for a life time. I used JumpCloud to configure Single Sign On for all of the tools that I have described thus far. Single Sing On made it very convenient for all of the engineers in the group to quickly get to the applications all from JumpCloud’s dashboard. The process of setting up SSO using JumpCloud is not as painful either. I plan on posting more stories detailing how to setup SSO for each service in future. The reason for paying for a premium subscription for Slack is that SSO is only available in paid subscription. Also, reason for choosing KeeperSecurity as the secrets management software is because that is the only out of the box SSO integration available on JumpCloud. Its very weird how humans make decisions, isn&#39;t it?</p><p>JumpCloud has a very simple and clean interface for logging into all the apps via SSO.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OINlGaKer0Qhd28HAf86RQ.png" /></figure><p>The pricing model for JumpCloud after 10 users is that a minimum of 10 licenses have to be purchased at $10 per user per month. However you get a total of 20 licenses if you pay for 10 licenses. So, pay for 10 users and you will have 20 users.</p><p>First 10 Users = Free</p><p>11–20 Users = $10 / user / month (min of 10 users)</p><p>You can probably imagine by now as to how much of a trouble the 11th member on your team is ;)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=248b0a5a7ee7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mac Tips : Easiest way to lock your screen]]></title>
            <link>https://medium.com/@swarupdonepudi/mac-tips-easiest-way-to-lock-your-screen-7298d3bfcc4f?source=rss-656d13fde0ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/7298d3bfcc4f</guid>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[apple]]></category>
            <category><![CDATA[mac]]></category>
            <category><![CDATA[mac-osx]]></category>
            <dc:creator><![CDATA[Swarup Donepudi]]></dc:creator>
            <pubDate>Tue, 21 Nov 2017 13:55:37 GMT</pubDate>
            <atom:updated>2017-11-21T13:55:37.970Z</atom:updated>
            <content:encoded><![CDATA[<p>Here is one of the most useful tip that I have found for using your mac. You don’t need any additional software to make this work and it only takes 2 minutes of your time to set it up. Why not right?</p><ul><li>Go to System Preferences -&gt; Desktop &amp; Screen Saver -&gt; Screen Saver</li><li>Click on ‘Hot Corners…’</li><li>You will see four drop downs. While you can choose any, I would recommend choosing the bottom right one.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vpEl-BTBgtaF8jT7-ZZORA.png" /></figure><ul><li>Select “Put Display to Sleep”. We are not done yet.</li><li>This will only turn on the screen saver. However, to make sure that your mac book is safely locked out we need to do couple more things.</li><li>Go to System Preferences -&gt; Security &amp; Privacy-&gt; General</li><li>Look for the check box followed by label “Require Box” which has a drop down right next to it. The default is set to “5 minutes”. Change this to “immediately”. You will need to enter your user account password to make this change.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1SDPY8mnyO2jG1uiHBsXPQ.png" /></figure><ul><li>Now try swiping your finger on the touchpad to the bottom right corner and voilaa your mac book is safely locked. Super easy right??</li><li>Because we changed the password policy on wake up to immediately, you will need to enter password to log back in.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7298d3bfcc4f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Step-by-Step Guide : How to setup SSM Agent to use EC2 Run Command service]]></title>
            <link>https://medium.com/@swarupdonepudi/step-by-step-guide-how-to-setup-ssm-agent-to-use-ec2-run-command-service-15b51418ecc4?source=rss-656d13fde0ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/15b51418ecc4</guid>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Swarup Donepudi]]></dc:creator>
            <pubDate>Mon, 21 Aug 2017 01:03:24 GMT</pubDate>
            <atom:updated>2017-08-21T01:03:24.993Z</atom:updated>
            <content:encoded><![CDATA[<p>I shared a <a href="https://medium.com/@swarupdonepudi/alternative-to-ssh-on-aws-ec2-instances-b1d655553d2b">story</a> earlier talking about AWS offering for an alternative to SSHing to EC2 instance to run commands. In this step-by-step guide I will walk you through the process of setting up alternative method to SSH to run commands remotely on EC2 instances.</p><ol><li>Log into your AWS Console by going to <a href="https://aws.amazon.com/">https://aws.amazon.com/</a> and clicking on the button on the top right corner of the browser window.</li><li>Create a new IAM Role by following the instructions on this step-by-step <a href="https://medium.com/@swarupdonepudi/step-by-step-guide-create-a-new-iam-role-on-aws-61d66c4bdf56">guide</a> to create a new IAM role.</li><li>Select “Amazon EC2” under “AWS Service Role” section for IAM “Role type” when “Establishing trust” as part of IAM role creation.</li><li>You should choose “AmazonSSMFullAccess” AWS managed policy when “Attaching policy” as part of IAM role creation.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15b51418ecc4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Alternative to SSH on AWS EC2 instances]]></title>
            <link>https://medium.com/@swarupdonepudi/alternative-to-ssh-on-aws-ec2-instances-b1d655553d2b?source=rss-656d13fde0ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1d655553d2b</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[ssh]]></category>
            <dc:creator><![CDATA[Swarup Donepudi]]></dc:creator>
            <pubDate>Mon, 21 Aug 2017 00:04:53 GMT</pubDate>
            <atom:updated>2017-08-21T00:09:42.427Z</atom:updated>
            <content:encoded><![CDATA[<p>The first thing that most of us do immediately after creating a new EC2 instance is log into that machine using SSH. While being able to SSH into machines is a powerful tool to have when troubleshooting system failures, there is definitely a downside to it. The downside is particularly true when that EC2 instance is a shared machine that several users use. Allowing users to SSH into machines opens up greater risk of leaving the machine in an inconsistent state when someone runs a command that they should not have. Tracing back the root cause for what caused that machine to go into an inconsistent state is always a pain in the wrong place for system administrators. So, there are definitely both advantages and disadvantages to SSH on EC2 machines.</p><p>However, there is going to be a ton of people who might hate you if you prevent them from logging into the machines. Well, you should always be ready to face some heat when you are trying to do the right thing, for the right to do most of the times will result in some sort of inconvenience and nobody like inconvenience. But AWS offers a decent alternative if you decide to do so in the form of <a href="https://aws.amazon.com/ec2/run-command/">EC2 Run Command</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*HpXnTOrTrMmka2xqWXL4xQ.png" /><figcaption>Run Command on EC2 Dashboard</figcaption></figure><p>Using EC2 Run Command as an alternative to SSH gives you a lot more visibility into “Who did what?” and also a lot more control by leveraging its integration with AWS IAM roles to define “Who can do what?”.</p><p>Please read <a href="#">this</a> story if you are looking for a step-by-step guide on how to set it all up.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1d655553d2b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>