<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aron Eidelman on Medium]]></title>
        <description><![CDATA[Stories by Aron Eidelman on Medium]]></description>
        <link>https://medium.com/@aron_60636?source=rss-36b4cd6aeae3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gQ2TEGcs_poaKN_ua-Ft1Q.png</url>
            <title>Stories by Aron Eidelman on Medium</title>
            <link>https://medium.com/@aron_60636?source=rss-36b4cd6aeae3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:40:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aron_60636/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why is test-driven development with agents so helpful for security?]]></title>
            <link>https://medium.com/google-cloud/why-is-test-driven-development-with-agents-so-helpful-for-security-841710b90430?source=rss-36b4cd6aeae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/841710b90430</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Aron Eidelman]]></dc:creator>
            <pubDate>Sat, 30 May 2026 04:06:58 GMT</pubDate>
            <atom:updated>2026-05-30T16:37:42.430Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VXWH2y3vY2mFiDZtFzpIDA.jpeg" /><figcaption>A test-driven flow shifting down and left with security, then merging at high development velocity</figcaption></figure><p><em>I’ve given variations of this article as a conference talk several times, at </em><a href="https://www.linkedin.com/article/edit/7466024630571642880/#"><em>Google Cloud Next </em></a><em>’26 in a breakout with about 200 developers and again at </em><a href="https://www.linkedin.com/article/edit/7466024630571642880/#"><em>Wiz</em></a><em>’s booth, then a couple days before publishing this article at </em><a href="https://www.linkedin.com/article/edit/7466024630571642880/#"><em>Zenity</em></a><em>’s excellent </em><a href="https://zenity.io/resources/events/ai-agent-security-summit-san-francisco#red-green-refactor-secure-the-new-loop-for-building-with-agents"><em>AI Agent Security Summit</em></a><em>. I’ve wanted to go a bit deeper on certain examples and tie together more sources, so writing it down was the natural next step. This is long, longer than the recommended “750 words,” but the good news is I wrote it quickly so you can read it quickly too. I did not use AI to write it so kindly don’t use AI to read it either.</em></p><p>Wander into any of the big security conferences over the past few months, and you’ll see every third row has a screen with a demo showing some tool catching a vulnerability and surfacing the finding under a GitHub PR. It’s easy to show, it lets people know the tool is stopping something bad before the code gets merged, and it meets most bars for having “shifted left,” which are still fairly low. But we really need to escape the idea that this is efficient or scalable beyond a single demo, or that we can’t go a bit further left and do better. We can and should eliminate many issues before the code ever even gets pushed.</p><p>Catching bugs in the local editor reduces costs compared to handling a fire drill in the pipeline. This post explains how to integrate security into the local PRGR (plan, red, green, refactor) loop with an agent specifically following TDD. Whether you are writing secure code from scratch or fixing legacy applications, I want to show how establishing consistent (and often not terribly new) software engineering practices is helpful when working with agents writing code.</p><h3>Vibe Coding Conundrums and the 70% Problem</h3><p>Ask anyone who spends time generating comments on LinkedIn under posts about vibe coding and they’ll use an LLM to remind you that vibe coding introduces stability and security risks if left without any sort of standards or guidance. And their LLM would not be entirely wrong. While a coding agent can quickly get a prototype 70% of the way to a functional application, completing the remaining 30% requires deep engineering rigor to avoid loops where fixing one bug introduces other functional regressions. This is what <a href="https://www.linkedin.com/article/edit/7466024630571642880/#">Addy Osmani</a> terms the <a href="https://addyosmani.com/agentic-engineering/the-70-percent-problem/">70% Problem</a>. Developers fall into a “two steps back pattern,” where fixing one bug introduces other functional regressions. More dangerously, vibe-coded applications are highly susceptible to security vulnerabilities; as <a href="https://beyond.addy.ie/">he notes</a>, “vibe coding is fun until you start leaking database credentials.”</p><p>Recent findings from DORA (that is, <a href="http://dora.dev/">dora.dev</a>, not the EU regulation nor the musical adventurer) show that unconstrained AI adoption for writing code can pose a few common issues. All of the findings I list below are from the <a href="https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report?e=48754805&amp;utm_campaign=CDR_0xda6e4e7f_default_b516614965&amp;utm_medium=external&amp;utm_source=blog">2025 report</a>, which you can grab for free.</p><p>Coding agents tend to make massive, sweeping changes that can quickly overwhelm developers, making it difficult for even experienced engineers to track quality and security issues unless they establish clear boundaries.</p><p>Letting an agent run without constraints can result in hundreds of lines of code being generated in a single batch, which might make for fun demos of “one-shotting” or a faux pas LinkedIn post about how you made sooo many lines of code, but actually, it increases review effort (turning into fatigue for developers or context collapse for review agents), and elevates the risk of introducing vulnerabilities that are hard for static tools to catch.</p><p>Interestingly though, DORA research also demonstrates that the assumed trade-off between speed and stability is a myth. High-performing teams excel at both delivery speed and system stability simultaneously. So not only does speed not require sacrificing quality, it usually goes hand-in-hand with quality.</p><p>Integrating security objectives directly into daily activities (from design to coding to testing) enables teams to spend 50% less time remediating security issues.</p><p>Providing the AI agent with clear team conventions, APIs, and requirements guides its scope and boosts individual effectiveness.</p><p>Enforcing small, iterative improvements minimizes human review effort and reduces the risk of introducing vulnerabilities. DORA research supports that working in small batches improves product performance and decreases friction for AI-assisted teams.</p><h3>Enforcing the PRGR (Plan, Red, Green, Refactor) Cycle</h3><p>To prevent agents from making massive, unguided changes that overwhelm developers, teams can write a persistent context file, such as GEMINI.md or ANTIGRAVITY.md. This configuration guides the agent through a structured four-stage workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*_UKQw9Hxitpu2r_L" /></figure><p><strong>Plan</strong>: The agent needs to articulate the architecture and design before writing code. This is pretty common now without configuration.</p><p><strong>Red</strong>: The agent to write a single failing test ideally from the user’s perspective. The test must fail first to prove it covers a new behavior.</p><p><strong>Green</strong>: Limit the agent’s focus exclusively to making that single test pass. Separate this step from writing the test in order to preserve the codebase’s “fitness function,” preventing the AI from modifying the test to match a flawed implementation.</p><p><strong>Refactor</strong>: Clean up the code to maintain quality and expand the persistent context file to prevent recurring issues. This is where you run a security scan, even a basic one, to catch easy issues to fix that you can immediately retest, all before you even commit.</p><p>During <strong>Plan</strong> or <strong>Refactor</strong>, you may also realize the need to divide out the functionality. In that case, you can simply update the plan and start a new flow. This prevents trying to cram too many tests and too much functionality into one commit.</p><p>This local structure ensures that we are defining outcomes and reviewing results, which replaces direct, line-by-line developer interaction.</p><h3>Encounters in the wild</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/0*dnJAoAzPDfsH4DU_" /></figure><p>In Spinoza’s philosophy, a “bad encounter” happens when two forces meet and drain each other’s power instead of building it up. He offers that view to explain why there is so much conflict in nature, but moreso to indicate that conflict between <em>reasonable</em> people can be transformed with more self-awareness and a more accurate model of the each person’s motivations. As <a href="https://www.linkedin.com/article/edit/7466024630571642880/#">Travis McPeak</a> points out, when security is treated as a late-stage gate, developers are forced to choose between shipping on time or shipping securely, then 81% of developers admit to knowingly shipping vulnerable code due to competing business priorities. The issue isn’t security or developers per se, it’s the timing and framing of each team’s goals.</p><p>To resolve this tension, modern engineering teams deploy the <a href="https://beyond.addy.ie/">paved road</a> concept. Originating from Netflix’s response to the Heartbleed situation, a paved road takes the security team’s expertise and bakes it directly into self-service tooling, making secure configurations the easiest and most natural path for developers. <a href="https://medium.com/google-cloud/shifting-security-down-early-1a42169d05a5">Shift down instead of left</a>. Instead of playing a reactive game of “whack-a-mole” against vulnerability findings, the team systematically eradicates these vulnerability classes by providing pre-configured, secure-by-default modules.</p><p>Under this model, security scanning shifts from detecting vulnerabilities to verifying whether developers are using the paved patterns. To be clear, THAT’s what delivers the real promise of shifting left, not (just) plugging in a ton of CI/CD tools. And guess what? Agents can inherit that ability to using skills and rules.</p><p>This shift to proactive security is urgent because we are facing what <a href="https://www.linkedin.com/article/edit/7466024630571642880/#">Gadi Evron</a> terms the <a href="https://www.knostic.ai/blog/the-ai-vulnerability-storm-a-cisos-playbook-for-the-mythos-era">AI Vulnerability Storm</a>. It is not (purely) hype, and typically calm, sober security minds that don’t bat an eye at the typical marketing FUD are now genuinely concerned. <a href="https://www.linkedin.com/article/edit/7466024630571642880/#">Anton Chuvakin</a> identifies this barrier as the <a href="https://medium.com/anton-on-security/breaking-the-patch-sound-barrier-your-vulnerability-remediation-will-not-keep-up-ai-exploit-speed-b41116c4e92b">“Patch Sound Barrier”</a> or the constraint where an organization’s maximum remediation velocity is permanently exceeded by AI-driven vulnerability discovery. In this storm, the median time from vulnerability disclosure to active exploitation has shrunk, with many exploits active in seconds, long before the CVE is published. We cannot patch our way out of this crisis manually.</p><p>As a developer who has seen the, shall we say, less glamorous side of security teams trying to do auto-remediation (or convince an unprepared development and operations team to adopt it), that is not comforting. There’s another side to this as well: when a vendor releases a security patch, AI can now reverse-engineer that patch, identify the vulnerability it fixes, and generate a working weaponized exploit in minutes, while enterprise testing and deployment pipelines require an average of 20 days to deploy that same patch.</p><h3>TDD as a security and stability gate</h3><p>Someone once told me the more times you add “and” to a claim, the more you set yourself up, because in logic, each “and” is a single point of failure.</p><p>I’m going to stick with the “and” in TDD for both security and stability, insofar as they both involve (1) being able to change code with confidence if needed, and (2) narrowing the code you write to what you need.</p><p>When an agent optimizes to pass a test, it finds the shortest path to do so. Writing tests after implementation causes the agent to write tests that validate what the code <em>happens</em> to do, which fails to check the intended behavior and masks underlying flaws. To prevent this, TDD must be treated as a non-negotiable software practice if you’re going to use it at all. I’m sorry! It’s extreme, I know. And I see an analogy (and the same pushback) to infrastructure-as-code-or-bust in that if only part of the team or project is using IaC, you’re still running the risk of drift and missing the control gates that IaC-directed scans and policies provide.</p><p>At Google, we have the Beyonce Rule: “If you liked it, then you should have put a test on it.” When there are deep debates online about “percentage of test coverage” it seems as misleading a metric as “lines of code.” My thought experiment is, “How unimportant is this code that you wouldn’t put a test on it?” This helps to avoid overly narrow tests of implementation details, and helps you define your unit by behavior rather than lines or blocks. If you set the boundary at behavior, at what the user is getting out of your application, you will not have coverage gaps, and you will quickly identify what can be deleted.</p><p>When adding code (and in this case that is, adding value), by forcing the agent to write a failing test (RED) before generating the minimum code to make it pass (GREEN), we establish concrete requirements that restrict the agent’s occasionally erstwhile activity and prevent logic drift. They do sometimes write entire mock libraries for APIs that don’t exist, but that’s on my system and process, not them.</p><p><a href="https://www.linkedin.com/article/edit/7466024630571642880/#">Paul Hammond</a> , whose thoughtful comments on TDD (often responding to less thoughtful posts about TDD) have shown me more about it than just about any source. I think he summed it up best when he wrote, “TDD is the reason I’m never stressed.”</p><h3>Up Next</h3><ul><li>Example of security skills, rules, plugins, and hooks in Antigravity 2.0 and CLI</li><li>Reliability practices for deploying agent-written security remediations</li><li>The ever changing state of models and static tools in the SSDLC</li></ul><h3>Recommended Technical Readings</h3><ul><li><a href="https://medium.com/google-cloud/getting-started-with-antigravity-2-0-updated-8a953f079f97">Getting Started with Antigravity 2.0</a></li><li><a href="https://medium.com/google-cloud/antigravity-cli-tutorial-series-12b46cfe3bf2">Antigravity CLI Tutorial Series</a></li><li><a href="https://www.wiz.io/reports/sdlc-security-report-2026">Wiz Research: State of SDLC Security 2026</a></li><li><a href="https://sre.google/sre-book/table-of-contents/">SRE Handbook</a></li><li><a href="https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report?e=48754805">DORA 2025 Report</a></li><li><a href="https://beyond.addy.ie/">Beyond Vibe Coding: A Guide to AI-Assisted Development</a></li><li><a href="https://addyosmani.com/blog/factory-model/">The Factory Model: How Coding Agents Changed Software Engineering</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=841710b90430" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/why-is-test-driven-development-with-agents-so-helpful-for-security-841710b90430">Why is test-driven development with agents so helpful for security?</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Shifting Security “Down,” Early]]></title>
            <link>https://medium.com/google-cloud/shifting-security-down-early-1a42169d05a5?source=rss-36b4cd6aeae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a42169d05a5</guid>
            <category><![CDATA[shiftleft]]></category>
            <category><![CDATA[gcp-security-operations]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Aron Eidelman]]></dc:creator>
            <pubDate>Wed, 28 Feb 2024 22:12:19 GMT</pubDate>
            <atom:updated>2024-03-13T21:20:08.753Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IFDyc42-KYn_WjgsNMzchw.jpeg" /><figcaption>A frustrated cat looking at security alerts on a screen, generated with the author and <a href="https://gemini.google.com/">Gemini</a></figcaption></figure><blockquote>Making plans and conceiving projects involves many agreeable sentiments. Those with the strength to be nothing but contrivers of plans all their lives would be happy people. But one must occasionally have a rest from this activity by carrying a plan into execution–and then comes anger and sobriety. — <em>Friedrich Nietzsche</em></blockquote><p>According to Ponemon’s Cost of a Data Breach 2023 report, DevSecOps adoption led to the highest cost savings for a data breach compared to any other practice or trait in a security program. While the value of “shifting left” with security is undeniable, the reality is that many teams end up hitting a wall <em>very</em> early on. At Google Cloud Next ’23 a few months ago, I had a chance to meet with security leaders who had experienced a successful DevSecOps adoption and those who were stuck or on the verge of abandoning their plans. Most of them faced the same initial challenges:</p><ul><li>A barrage of high-severity vulnerability alerts, most of which were false positives.</li><li>The conflation of “severity” with actual<em> risk</em> at all levels of the organization.</li><li>Other bottlenecks that could potentially worsen development and security instead of improving them.</li></ul><p>So, what was the difference that helped those who succeeded? Despite setbacks, they almost always had an executive sponsor committed to a long-term plan. At first, that may seem to go against the ideological valence of “cultural change” that pictures different groups of engineers independently breaking down barriers and becoming friends (although that does actually happen). They still need to have a high-level leader committed to getting to a certain end-state, willing to provide air cover, able to understand initial setbacks, and, of course, able to relate to the business what the eventual cost-benefit of DevSecOps will be.</p><p>So, what’s the <strong>opposite</strong> image of that type of leader? One who is:</p><ul><li>Counting on seeing immediate value from shifting left</li><li>Does not realize that “adding security” can, at least initially, significantly slow down all of the incredible speed of a modern CI/CD pipeline</li><li>Frames the implementation as “bottom-up,” but in a way that merely shirks commitment and leaves engineers confused about their priority. Is it speed? Frugality? Is it creating a bunch of new tools instead of buying them?</li></ul><p>The neat thing was that in cases where a leader realized they were falling into this pattern, there was still a chance to recover and learn in ways that helped them anticipate even more significant difficulties. The ones who recovered would immediately start to plan how to mitigate the burden on developers and security teams. Besides merely “shifting left,” they described what Richard Seroter generally refers to as “shifting down” in <a href="https://cloud.google.com/blog/products/application-development/richard-seroter-on-shifting-down-vs-shifting-left">The Modernization Imperative</a>. That article offers a high-level strategy to maximize developer efficiency and innovation. I wanted to say more about security at a practical level, at the very early stages of adding static testing to software development.</p><h3>From the Start</h3><blockquote>Beginning is the time for taking the most delicate care that the balances are correct. — <em>Princess Irulan, Frank Herbert’s</em> Dune</blockquote><p>The beginning hurts the most for the companies I’ve met who have attempted to integrate SAST, SCA, and other capabilities. Initially, there’s excitement about the potential to catch vulnerabilities much earlier. If we manage to shift left, there is a sense that we automatically win from the get-go. It is too easy to take for granted the value realized in this graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DWtAdFNCNk-jAiW4" /></figure><p>But then, reality hits. The slick, 15-minute-or-less build to deploy now gets mired in warnings that your team can’t possibly address in time. Security, it turns out, requires <em>deliberation</em> and applied knowledge of your application and your business that doesn’t match the speed of the rest of the development process. If you look at OWASP’s <a href="https://owasp.org/www-project-devsecops-maturity-model/">DevSecOps Maturity Model</a>, you can start to anticipate many of these issues looking at the less mature stages:</p><ol><li>Early on, you don’t break the build on failed tests–because there will be too many failures whose validity is unknown.</li><li>You enable software component analysis (SCA) and static analysis (SAST) starting from the server side, <em>without any configuration unique to your application</em>, and then slowly add breadth (to the client side) and depth (with better configuration).</li><li>You start at high-intensity testing and progressively turn tests off as you discover they are less critical.</li></ol><p>TL;DR, it’s going to be very noisy and unreliable at first. Even with the wisdom that this initial stage is only temporary, it can lead to alert fatigue, burnout, and, ironically, a <em>decline</em> in security posture and the creation of additional project risks.</p><h3>With the End in Mind</h3><p>The importance of having a security vision can’t be overstated here: everyone, from the engineer wrangling tests to the C-level describing risks to the board, should have a general sense of how their efforts, over time, are going to lead to much less painful, more valuable state in the future:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*46SI8vW8C5GYf89k" /></figure><p>But this is almost impossible if we ignore the developers’ time and cognitive load, i.e., if the security risk mitigation leads people to overlook the <em>growth</em> risk they might inadvertently create. Imagine this conversation:</p><blockquote>“So I see we added static testing, and that’s generated a lot of vulnerability notifications. It also delayed our latest release by a week; even minor iterations now take hours instead of minutes. What did we gain for it, though?”</blockquote><blockquote>— “Well, we gained <strong>security</strong>. We would otherwise be shipping software with lots of vulnerabilities. Right?”</blockquote><blockquote>“Yeah, but how bad were the vulnerabilities?”</blockquote><blockquote>— “Oh, pretty bad. 10% were CRITICAL, 15% were HIGH, and even the MEDIUM ones could be chained together, so overall, an orange-meets-crimson red, if we used a color wheel.”</blockquote><blockquote>“Okay, well, the delays cost us approximately $2 gazillion. How bad would it have been if all those CRITICALS got exploited?”</blockquote><blockquote>— “Oh, I mean, any of them was <em>likely </em>or <em>highly likely</em>; look, it said so in the dashboard with CVSS details. Just one of them would probably cost us about, I don’t know, take a wild guess–$10 bazillion?”</blockquote><blockquote>“How do you determine likelihood? Can we see all the possible scenarios, what the impact would be, and some justification for why or why not it could happen?”</blockquote><blockquote><em>— </em>“<em>Impossible to really know</em>. We go based on the score in the dashboard. There are too many issues for us to spend time getting into an academic debate about everyone. So even if it hurts a little, we trust in the process.”</blockquote><p>In the past half-decade or so, the thinking pattern in that conversation is now facing fresh criticism. (It met challenges before, but the solutions for prioritizing risk from 12 years ago <em>never</em> penetrated modern dev teams, who are now the ones playing catchup.) The challenges are becoming more apparent, especially in the past couple of years, from the organization maintaining the CVSS specification stressing <a href="https://www.first.org/cvss/v3.1/user-guide#2-1-CVSS-Measures-Severity-not-Risk">that severity cannot be mistaken as risk</a> to more data-driven descriptions of likelihood from <a href="https://www.first.org/epss/">EPSS</a> and <a href="https://www.cisa.gov/known-exploited-vulnerabilities-catalog">CISA KEV</a> to Google Cloud <a href="https://cloud.google.com/artifact-analysis/docs/create-vex">adding VEX statements</a>.</p><h3>But What are Developers Seeing?</h3><p>Despite the advances in helping organizations think about and measure likelihood, exploitability, and so forth, you have to ask what the people who are actually writing the software tend to encounter when they go to a trusted source like OWASP:</p><blockquote>Generally, identifying whether the likelihood [of a potential risk] is low, medium, or high is sufficient. — <em>OWASP</em></blockquote><p>This is precisely where the gap exists.</p><blockquote>Does [OWASP’s] phrase, stating “low, medium, or high is sufficient,” need to be taken on faith? Considering the critical nature of the decisions such methods will guide, we argue that <strong>it should not</strong>. — <em>Douglas Hubbard and Richard Seiersen, </em>How to Measure Anything in Cybersecurity Risk</blockquote><p>The reality on the ground (despite the years of many security blogs crying out in protest against it) is that dev teams are <em>still</em> sorting and patching purely based on uncritical, faith-based severity scores, with almost no attention to context-specific likelihood or impact.</p><p>Regarding risk prioritization, part of “shifting down” is <strong>not</strong> reducible to tooling. However, there are great products addressing parts of this issue, such as <a href="https://www.silk.security/">Silk Security</a>, which has a configurable risk model based on the asset and environment, and <a href="https://mondoo.com/docs/platform/infra/cloud/gcp/overview/">Mondoo</a>, which can tell a developer very early on in their IDE if a new line of code would violate compliance necessary to do business.</p><p>Shifting down occurs very early when setting requirements: it means picking goals, frameworks, and measurements that stem the need to “invent” prioritization well <em>after</em> getting inundated with tons of noise. Avoid cognitive overload by thinking about what can cause it and mitigating <em>it</em>.</p><h3>Shifting Down on False Positive Analysis</h3><blockquote>We suffer more in imagination than in reality. — <em>Seneca</em></blockquote><p>I am going to skip the litany about false positives in <em>general</em>. Last year, I was a guest on a podcast with Anton Chuvakin, Tim Peacock, and Steve McGhee in <a href="https://www.youtube.com/watch?v=zEW0dIX35hk">SRE Wisdom for Security — Alerting</a>, where we unpack this problem in detail. Those lessons, in part, address what Michele Chiburka recently described in <a href="https://postmodernsecurity.com/2024/01/25/fear-and-loathing-in-security-dashboards/">Fear and Loathing in Security Dashboards</a>–that appeals to fear often backfire–and we have the evidence.</p><p>So, what’s the practical issue with SAST and SCA? The pain point is that with many open-source and free SAST and SCA tools that teams use, there will be a ton of false positives. In a (somewhat superficial) effort to “improve security early on,” we have suddenly created an authentic and significant project risk. The content of these alerts, when it isn’t perplexing even to senior developers, still takes time to analyze.</p><p>My first question to a colleague at a large multinational company planning to implement SAST earlier in their dev pipeline was, “Who exactly does the false positive analysis?” The DevOps lead exclaimed, “Exactly!” They subtly meant that we were suddenly creating a new, vast, likely underappreciated burden, whether it was developers or security engineers.</p><p>How do we shift down here?</p><ul><li>An upstream that introduces fewer issues to begin with.<strong> </strong>Developers should not be hounding down every dependency or worrying that maintainers might neglect security testing. Using <a href="https://cloud.google.com/security/products/assured-open-source-software?hl=en">Assured OSS</a>, for example, a developer can pull in packages in which Google finds and fixes vulnerabilities, including AI/ML projects such as TensorFlow.</li><li>Software component analysis that tracks the dependencies of dependencies and focuses on alerting on what part of the package we use, <em>not</em> just the version. Endor Labs has an excellent <a href="https://www.endorlabs.com/learn/why-are-all-sca-tools-wrong">breakdown of why SCAs tend to be so noisy</a> and a way to <a href="https://cloud.google.com/blog/topics/partners/securing-software-supply-chain-with-endor-labs-dependency-management/">reduce them to a much more manageable level in Google Cloud</a>.</li><li>SAST tools integrate easily into the build process with minimal friction. Plenty of great SAST tools exist, but the integration piece alone can significantly burden a dev team. GitLab has a SAST capability using other scanners, from Kubesec to Semgrep, <a href="https://docs.gitlab.com/ee/user/application_security/sast/#supported-languages-and-frameworks">that supports many different languages</a>, can <a href="https://docs.gitlab.com/ee/user/application_security/sast/rules.html">take in rules from multiple sources</a>, and can <a href="https://about.gitlab.com/releases/2023/03/22/gitlab-15-10-released/#automatically-resolve-sast-findings-when-rules-are-disabled">automatically resolve issues that are not relevant</a>. You can add that as the initial step to <a href="https://cloud.google.com/build/docs/automating-builds/gitlab/build-repos-from-gitlab">a build process going to Google Cloud</a>.</li></ul><p>Those are just a few practical examples, but here is the guiding idea: shifting security <em>down</em> is a matter of reducing risk–not <em>just</em> security risk, but also growth risk. We need to ask: is there something we are asking developers to spend time on related to security, where we don’t know the actual value in terms of the risk we’re mitigating? Is there some aspect of testing that is producing too much noise? Have we considered more managed tools, products, or practices that can reduce it, and do we have people with the leverage to see this step as part of our vision for what we want security to look like? Find me in the Innovator’s Hive at <a href="https://cloud.withgoogle.com/next">Google Cloud Next in Las Vegas</a> this year to share your stories with me and figure out other ways to shift down.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a42169d05a5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/shifting-security-down-early-1a42169d05a5">Shifting Security “Down,” Early</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Alerting Guidelines]]></title>
            <link>https://medium.com/google-cloud/alerting-guidelines-a71fc78dfdc0?source=rss-36b4cd6aeae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a71fc78dfdc0</guid>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[it]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[operations]]></category>
            <dc:creator><![CDATA[Aron Eidelman]]></dc:creator>
            <pubDate>Wed, 15 Feb 2023 01:24:42 GMT</pubDate>
            <atom:updated>2023-02-25T06:28:24.444Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="An abstract diagram of a few “alert” icons above a surface, leading down to branching graph that ends in round nodes, similar to a root structure." src="https://cdn-images-1.medium.com/max/1024/1*HRHwX45n0Uw3ZmEePr_BAA.jpeg" /></figure><p><em>The ideas reflected in this post do not necessarily reflect the opinions, attitudes, and statements of my employer or anyone associated with me.</em></p><p>Engineering organizations need to be able to quickly and easily identify and resolve issues, but this need often takes a backseat compared to other priorities. With so many things to track, it can be challenging to know which ones need alerts. It is possible to set up too many alerts, create too much noise, and lose relevance when faced with real problems. Because of this, we need to clarify <em>why</em> we are creating an alert in two respects:</p><ul><li>the <em>relevance</em> of what we’re monitoring to our business goals</li><li>the intended <em>outcome </em>once we notify someone there’s an issue</li></ul><p>Understanding the relevance of what we’re monitoring can help us support triage in advance. When an issue occurs, we should have enough context to gauge its impact on users, potential cost, and what other problems it can take priority over. Relevance also helps us identify which signals warrant setting <em>objectives</em>. For example, latency is incredibly relevant in a customer-facing application that serves real-time data — yet irrelevant in a weekly cron job to clean up old files across systems. Relevance based on context helps us answer <em>why, what, </em>and <em>when </em>to alert.</p><p>The outcomes we intend from an alert notification vary widely, from immediate response to situational awareness. We should not be waking up our on-call engineers for an SLO breach that is outside their power to fix and affects our release velocity. We should also not be tucking away all of our alerts in emails since the more critical issues could slip by. We are determining the outcomes we want, which can help us figure out <em>who </em>to alert and <em>how</em>.</p><h3>Why should I create alerts for only some things?</h3><p>When I create an alert for something, the assumption is that someone can and should take action if notified.</p><p>Think of alerts you might experience in everyday life:</p><ul><li>A flash flood warning on our phone</li><li>A text message about a declined credit card transaction</li><li>An email about a “login from a new device”</li></ul><p>Each of these alerts could exist on a website or in a log, but we’re getting notified <em>directly</em> about each one. We often also get instructions about what to do:</p><ul><li>Avoid roads, stay at home, and prepare an emergency kit</li><li>Agree that you intended to make the transaction and try again or report it as fraud</li><li>Trust the new device, or block access and immediately change your password</li></ul><p>In some cases, even if we feel the alert isn’t relevant or are already aware of an issue, we still see the value of having the alert on <em>in general</em>. And even if we need to do some deeper digging, e.g., looking up the store’s name where the transaction took place or verifying the device matches what we’re using, the alert is a good starting point.</p><p>The zombie strategy of alerting, where too many metrics have alerts, potentially leads to a situation where the critical issues do not stand out. The saying goes, “If everything is important, nothing is important.” Imagine getting a flash flood warning for every flash flood on Earth, not just the ones in your area. Imagine a text message for every transaction you make, not just the declined ones. How about an email every time you log in, even from the same device? These would condition you to ignore the more relevant warnings or otherwise distract you with noise.</p><p>We have a starting point: we should have alerts for things insofar as we can fix, influence, or control them, and insofar as <em>not knowing</em> about the alert could be worse for us. Still, that leaves a large set of possibilities that we need to trim down, starting with <em>what</em> to alert on.</p><h3>What should I alert on?</h3><p>As we increasingly automate systems, something interesting happens: the number of things that <em>should </em>be actionable should decrease. So what stays the same?</p><p>No matter how much we automate, the constant area of focus is on <em>relevance to users</em>. In <a href="https://bootcamp.uxdesign.cc/operational-focus-why-symptoms-not-causes-e4af0e115e14">Why Focus on Symptoms, Not Causes</a>, we explored why user-facing symptoms are a better area of focus for alerting than causes.</p><p>We should alert on things that are <em>actionable</em> <strong>and</strong> <em>relevant to users</em>. A few examples:</p><ul><li><strong>Availability</strong>. 500s, unintentional 400s, hanging requests, redirects to malicious sites–all count. Whether it’s the entire site or a small third-party component, anything that disrupts the critical user journey should be considered “unavailability.”</li><li><strong>Latency</strong>. <em>Fast (</em>as long as it’s humans that are waiting).</li><li><strong>Integrity/durability.</strong> The data should always be safe. Even if the data is temporarily unavailable, it should be correct when it comes back.</li></ul><p>Starting from symptoms, you may occasionally discover that there are still “causes” you need to alert on–something deep within your system that, while its behavior may be invisible to users, can still impact them down the road. Ideally, you’ll move to a state where you can automate responses to any of these issues, but it makes sense to have a plan in place.</p><p>An essential rule of thumb to keep in mind is that alerts are only the “start” of an action or an investigation; they do not form the entirety of your strategy. Not all metrics should have a corresponding alert. If a system grows <em>and</em> becomes more automated, it would be normal to monitor more <em>and </em>alert less.</p><h3>When should I trigger an alert?</h3><p>Picking the right time to trigger an alert relates to how actionable and severe the impact is.</p><p>Suppose latency exceeds 300ms, or there is a spike in 500 errors for less than a few seconds. In each case, the anomaly may be worth looking into — but these are less severe than when high latency or error rates persist for several <em>minutes</em>. If an issue is ongoing, it needs direct intervention, whereas an issue that has already happened may require investigation. These deserve different levels of attention.</p><p>Sometimes, an alert serves as an early warning for a situation that would be too complicated to act on if it were to arrive. For example, in monitoring the quota consumption of a cloud service, it’s necessary to know well in advance if a service is approaching maximum usage so that a customer can request a limit increase, which can take time to approve.</p><h3>Who should I notify, and how?</h3><p>Once you have identified the people who need to know about an issue, you can decide how to notify them. For example, if the problem is urgent, you may need to page a person directly. You can create a ticket without paging anyone if the issue is less pressing. While this may seem painfully simple, a common problem is using only one notification channel for every severity level.</p><p>If someone gets paged all the time, even for minor issues, they get stressed out and may get distracted from the priority. If alerts go to a low-priority channel, like a group email (which tends to be the default for cloud quota consumption), people may only see them after it’s too late. <em>Only</em> notify people who need to know <em>that</em> there is a problem, and leave it to their discretion to inform others. It may take some planning, but the goal is to avoid spamming people with irrelevant notifications. Different channels enable prioritization and appropriate visibility.</p><p>If the on-call person is the only person who can resolve an urgent issue, you should page them. There are plenty of urgent situations where the person on-call can’t handle the resolution on their own, so paging them is just the first step in an urgent escalation. (See a separate discussion and examples of incident management <a href="https://sre.google/workbook/incident-response/">here</a>.) If the issue isn’t critical, the alert should generate a <em>ticket</em> in the queue. The on-call person can still work on it–just not as an “interrupt.”</p><h3>Concluding Guidelines</h3><p>From establishing the relevance and intended outcomes of alerts, we can use quick rules of thumb to help us get the most out of alerts by keeping a high signal-to-noise ratio.</p><p>What to alert on:</p><ul><li>Alerts should be actionable and relevant to users.</li><li>Some examples of things to alert on are availability, latency, and integrity/durability.</li><li>Alerts are the “start” of an action or an investigation; they may only represent a small portion of what you monitor.</li></ul><p>When to alert on things:</p><ul><li>Alerts can be more or less urgent depending on how long the issue has been going on and how severe the impact is. Consider that impact changes over time.</li><li>You can use alerts to give people enough time to act before there are consequences for issues such as quota consumption.</li></ul><p>Who to alert, and how:</p><ul><li>Only notify people you intend to <em>act</em> in response, and trust them to inform more people if needed.</li><li>Page the person on-call if the situation requires an immediate response.</li><li>Consider not paging the person on-call and creating a ticket if the issue is not urgent.</li></ul><p>If these guidelines helped you create effective alerts or you have any questions, please join us at the next <a href="https://r9y.dev/discuss">Reliability Engineering Discussion</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a71fc78dfdc0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/alerting-guidelines-a71fc78dfdc0">Alerting Guidelines</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Realistic SLAs]]></title>
            <link>https://medium.com/google-cloud/realistic-slas-73685b96f8d0?source=rss-36b4cd6aeae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/73685b96f8d0</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[certification]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Aron Eidelman]]></dc:creator>
            <pubDate>Tue, 29 Nov 2022 02:32:20 GMT</pubDate>
            <atom:updated>2023-02-25T06:28:50.144Z</atom:updated>
            <content:encoded><![CDATA[<h4>Experiences that Prepared Me for the Cloud DevOps Engineer Exam, Part II</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*qx5idZymfnTt4Tyo7q23zQ.png" /></figure><p><em>Disclosure: I am a Google employee. The ideas reflected in this post are personal and do not reflect my employer’s views. The following is Part II of my three-part blog, </em><a href="https://medium.com/@aron_60636/experiences-that-prepared-me-for-the-cloud-devops-engineer-exam-bd7edee4ccaa"><em>Experiences that Prepared Me for the Cloud DevOps Engineer Exam</em></a><em>.</em></p><p>Setting objectives is one of the most challenging topics from my experience that I encountered on the exam.</p><p>The challenges involve the motivations and ideas of the <em>people</em> setting the objectives rather than just the technology used to obtain them.</p><blockquote>“Hope is not a strategy.” - <a href="https://sre.google/sre-book/introduction/">Traditional SRE saying</a></blockquote><p>Let’s start with SLAs.</p><p>I had a situation a few years ago with a small startup I was consulting. They had an SLA on their website for “99.99% availability.” I asked what justified this number, and it turned out someone had arbitrarily picked it because it seemed like the industry norm and to coincide with the SLAs from their cloud provider.</p><p>The risk was obvious enough: if 99.99% was <em>purely</em> aspirational, there’s a good chance they’d fail to meet it, which would cost their business.</p><p>We decided to find a more realistic number.</p><p>Alex Ewerlöf has an excellent overview of how to <a href="https://alexewerlof.medium.com/calculating-composite-sla-d855eaf2c655">calculate a composite SLA</a>. Needless to say, once we looked more closely at the architecture, the SLAs of the cloud provider’s services, and observed the application itself, we ended up with a number more humble than 99.99%.</p><p>It relieved everyone that this lower SLA didn’t hurt the startup’s sales<em>.</em></p><p>What their customers cared about wasn’t the precise number of minutes a month before they get a refund, but more so what would happen <em>if </em>there were an outage.</p><p>The service could block users from logging in, so it needed to <a href="https://community.microfocus.com/cyberres/b/sws-22/posts/security-fundamentals-part-1-fail-open-vs-fail-closed">fail open</a> in case of an outage. Finding a graceful way to handle failure was a common topic of conversation, whereas the SLA was an afterthought.</p><p>Whereas any company consuming a service has probably bumped into an SLA, it’s important to note that some global services do not have or need them:</p><blockquote>Google Search is an example of an important service that doesn’t have an SLA for the public: we want everyone to use Search as fluidly and efficiently as possible, but we haven’t signed a contract with the whole world. Even so, there are still consequences if Search isn’t available — unavailability results in a hit to our reputation, as well as a drop in advertising revenue. Many other Google services, such as Google for Work, do have explicit SLAs with their users. Whether or not a particular service has an SLA, it’s valuable to define SLIs and SLOs and use them to manage the service. (<a href="https://sre.google/sre-book/service-level-objectives/">SRE Handbook, Ch. 4 — Service Level Objectives</a>)</blockquote><p>The topics covered in the exam tend to focus on SLOs, or the SLIs they’re based on. They share a similar framing to the above experience, and this experience showed me a two-way understanding between business stakeholders and engineering.</p><p>Engineering should look to business stakeholders to <strong>define what customers need</strong> at the “service level.” Otherwise, they may be <a href="https://bootcamp.uxdesign.cc/operational-focus-why-symptoms-not-causes-e4af0e115e14">setting objectives that are irrelevant to the business</a> and overcommitting resources to those ends.</p><p>In my example, we discovered a slightly lower SLA didn’t affect sales after some trial and error. Predicting this type of impact in advance is possible, even in setting internal SLOs for services that don’t have SLAs.</p><p>The understanding has to be mutual. Business stakeholders should look to engineering to understand the <a href="https://medium.com/@aron_60636/experiences-that-prepared-me-for-the-cloud-devops-engineer-exam-bd7edee4ccaa#ff58">cost and complexity tradeoffs</a> for increasing their objectives, <em>especially</em> how it affects release velocity.</p><p>In my example, suppose the business insisted on sticking to its initial 99.99% claim. How much more would that have cost, and how much would it hold up development?</p><p>The fact that the risk existed and that a more modest claim would be a safer bet was ultimately sound <em>business</em> reasoning. It required some engineers to <em>validate</em> the logic and identify the hidden tension.</p><p>Determining how a technical decision affects customers, your business, and the teams building and delivering the product will help with these topics more than narrowly focusing on how systems behave internally.</p><p>Do you have similar stories to share? Check out <a href="https://sites.google.com/view/reliability-discuss/">Reliability Engineering</a>, which has a lean coffee format where you can propose topics to discuss, and people can vote on their favorites. A discussion about SLOs in that group gave me a great mental model that helped me during the exam and helped me come up with my post on <a href="https://bootcamp.uxdesign.cc/operational-focus-why-symptoms-not-causes-e4af0e115e14">why to prioritize symptoms over causes</a>.</p><p>In my final post coming next week, we’ll look at what to do in the inevitable case of failure.</p><figure><a href="https://www.credential.net/0dcec0e8-4eaf-466f-88ea-9966aa51b7cf#gs.irxraz"><img alt="" src="https://cdn-images-1.medium.com/max/200/1*h0AUl44v2Rcz3Npb8qGhiw.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73685b96f8d0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/realistic-slas-73685b96f8d0">Realistic SLAs</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Experiences that Prepared Me for the Cloud DevOps Engineer Exam]]></title>
            <link>https://medium.com/google-cloud/experiences-that-prepared-me-for-the-cloud-devops-engineer-exam-bd7edee4ccaa?source=rss-36b4cd6aeae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd7edee4ccaa</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[certification]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Aron Eidelman]]></dc:creator>
            <pubDate>Tue, 15 Nov 2022 00:35:23 GMT</pubDate>
            <atom:updated>2023-02-25T06:29:03.650Z</atom:updated>
            <content:encoded><![CDATA[<figure><a href="https://www.credential.net/0dcec0e8-4eaf-466f-88ea-9966aa51b7cf#gs.iruubg"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Njh5E52T5iRq1dUhS8w1iw.jpeg" /></a></figure><p><em>Disclosure: I am a Google employee. The ideas reflected in this post are personal and do not reflect my employer’s views.</em></p><p>I joined Google several months ago as a Cloud Operations Advocate. As part of my ramp-up time, I prepared to take the Cloud DevOps Engineer certification since it overlapped the most with the use cases I’m focused on in my role. Without making assumptions about job titles or specific products, I want to tune into the <em>experience</em> that other engineers have on Google Cloud. I saw some of my own experiences reflected in the exam content, which supports the validity of any technical certification.</p><p>Google’s <a href="https://sre.google/sre-book/table-of-contents/">SRE handbook</a> had a good amount of bearing on the exam content, which surprised me. What I wanted to avoid more than anything was a 2-hour round of “feature and configuration trivia,” otherwise known as “multiple choice that you could ace with reference docs.” This was no such exam. It is good to know general configuration patterns, but the best mark of knowledge based on <em>experience</em> is having a deep, intuitive sense of how things can go <em>wrong</em>. I liked that the exam asked questions in this direction and that I could use my experience to reason through the possibilities.</p><blockquote>“Be warned that being an expert is more than understanding how a system is supposed to work. Expertise is gained by investigating why a system doesn’t work.” — <a href="https://sre.google/sre-book/effective-troubleshooting/">Brian Redman</a></blockquote><p>In this post, which I’ll add to over the coming weeks, I want to share several challenging experiences before joining Google that gave me a deeper understanding of <em>why </em>it makes sense to do things a certain way. (For those who came for general study tips, I’ve added some to the final section.)</p><p>I tapped into these experiences while studying new material for the exam, thinking, “How would I have had better outcomes in the past if I had done X or used Y?” I found this approach helped me integrate new information. It also helps to learn from other people’s experiences, which I benefitted from reading the SRE handbook, and which I hope some will benefit from reading here as well.</p><h4>Hidden Tradeoffs</h4><p>A few years ago, I worked with a customer on integrating a solution to prevent user account takeover. The problem ranged from bots enumerating through credentials to criminals committing account fraud. Since the activity occurred <em>within</em> an application, observing specific actions at the account level was necessary.</p><p>The developers would typically need to add the solution’s SDK to their login flow so that it could log regular attempts and intercept malicious ones. Developers didn’t love needing to write and maintain extra code around the SDK, so the solution provider came up with a “codeless” variant: a customer could add an edge function to their favorite CDN, and boom, it would <em>magically</em> zero in on the relevant requests.</p><p>In reality, there was still some configuration required. It just wasn’t the application developers who needed to do it. The edge function relied on response status codes, custom headers, or content in the response body to know if a login attempt had succeeded or failed. Since that could change dramatically from application to application, a <em>person</em> from the solution provider needed to step through the customer’s app manually, the way a regular user would, and test out various requests and responses. They would only know how that particular app represented successful and failed logins and have the information they needed to write the configuration.</p><p>To understand how “custom” this could get, keep in mind that not every development team uses the <a href="https://datatracker.ietf.org/doc/html/rfc9110">RFC for HTTP status codes</a>. Sometimes, <em>every</em> login attempt receives a 200 response. From there, the difference in responses could be very subtle. The configuration occasionally hinges on the string “error” or “denied” being included in the response body or an opaque header simply being absent for failed logins.</p><p>So what would happen if, <em>post</em>-configuration, the application developers decided to change the response for a failed login attempt?</p><p>What if they inadvertently removed the indicator necessary for the configuration to work?</p><p>In this case, the solution’s ability to detect and block malicious traffic could be at stake. And since security succeeds when nothing bad is happening, things might still appear to be working.</p><p>So the developers would be better off at least writing some tests to preserve the indicators so they’d know if they were potentially breaking the solution by making a change.</p><p>But that would entail writing code, perhaps even more than just implementing the SDK.</p><p>The other problem was that most customers only used the CDN with the edge function in <em>production</em> environments.</p><p>They had no way to justify a CDN for staging. As a result, there was no way to see whether the edge function was working, even <em>manually</em>, before production.</p><p>Suppose they bit the bullet and, in desperation, added a comment in their code, “Before changing this response, make sure to ask the solution provider to update the configuration for the edge function.” Yikes, I know, but still, would <em>that</em> work?</p><p>How would they ensure the third-party solution provider published the new edge function’s configuration simultaneously when the company deployed the latest version of their application? What if the company needed to roll back the most recent version? Because there was no automation for updating the configuration, and even the submission of the configuration file was entirely manual, it would <em>perpetually</em> create a bottleneck to any release that touched the login responses.</p><p>The likelihood that this operational gap could slip through the cracks in testing or deployment or that merely changing the people on the team could lead to this configuration being completely forgotten seemed to trade against the value of the “codeless” approach.</p><p>Where it took away some initial coding from development, it added manual work and a lack of confidence to the release process.</p><p>As a result, the reality was that the “codeless” approach <em>might </em>be nice for some cooker-cutter scenarios and proofs of concept, but most customers would be better off with the SDK.</p><p>It was a helpful scenario to remember for the exam because it reinforced the following points:</p><ul><li>If developers cannot test a feature, or if the team cannot automate a portion of the application deployment, consider how the resulting issues could affect production. How would they impact users? How long would it take to (1) realize a problem and then (2) fix it? Some key areas, such as security and availability, may be too sensitive to gamble with, even if you can’t guarantee them 100%.</li><li>Always think in terms of <em>tradeoffs</em> as opposed to pure improvements. If something seems purely good (e.g., a “codeless” add-on), question what you are bargaining away and if you can afford to do so. You might be able, but you don’t want to be surprised if you have already committed and then realize it entails manual work, higher risk, and lower release velocity.</li></ul><h4>Realistic SLAs</h4><p><a href="https://medium.com/@aron_60636/realistic-slas-73685b96f8d0">Out now!</a></p><h4>Study Resources</h4><p>My colleague, <a href="https://medium.com/@ammettw">Ammett</a>, put together a <a href="https://medium.com/google-cloud/preparing-for-the-google-cloud-professional-cloud-devops-engineer-exam-30e9d5fe07e4">great post with resources for the Cloud DevOps Exam</a>. In particular, I used the<a href="https://drive.google.com/file/d/1cCCTwulZuSBa4XmEh9bGzEwotaaOz9Wt/view"> prep sheet</a> he created to double-check that I’d covered all the necessary sections.</p><p>Another colleague, <a href="https://medium.com/@lukeschlangen">Luke</a>, had suggested closely reviewing the <a href="https://sre.google/sre-book/table-of-contents/">SRE handbook</a>. Just before the exam, he reassured me that even if it felt like it was too difficult halfway through, not to lose hope.</p><p>While I did not join a study group or work with anyone else preparing for the exam, it did help to <a href="https://cloud.google.com/certification/guides/cloud-devops-engineer">discuss the exam topics</a> with people who had direct experience in the relevant areas.</p><p>One discussion group you can join, <a href="https://sites.google.com/view/reliability-discuss/">Reliability Engineering</a>, has a lean coffee format wherein you can propose topics to discuss, and people can vote on their favorites. A discussion about SLOs in that group gave me a great mental model that helped me during the exam and helped me come up with my post on <a href="https://bootcamp.uxdesign.cc/operational-focus-why-symptoms-not-causes-e4af0e115e14">why to prioritize symptoms over causes</a>.</p><figure><a href="https://www.credential.net/0dcec0e8-4eaf-466f-88ea-9966aa51b7cf#gs.irxraz"><img alt="" src="https://cdn-images-1.medium.com/max/200/1*h0AUl44v2Rcz3Npb8qGhiw.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd7edee4ccaa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/experiences-that-prepared-me-for-the-cloud-devops-engineer-exam-bd7edee4ccaa">Experiences that Prepared Me for the Cloud DevOps Engineer Exam</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Operational Focus: Why Symptoms, not Causes?]]></title>
            <link>https://medium.com/google-cloud/operational-focus-why-symptoms-not-causes-e4af0e115e14?source=rss-36b4cd6aeae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4af0e115e14</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[operations]]></category>
            <dc:creator><![CDATA[Aron Eidelman]]></dc:creator>
            <pubDate>Thu, 03 Nov 2022 23:46:46 GMT</pubDate>
            <atom:updated>2023-03-01T04:38:39.700Z</atom:updated>
            <content:encoded><![CDATA[<p>“Users don’t care <em>why</em> something is not working, but <em>that</em> it is not working.”</p><p>How can we turn this platitude into something that helps Ops teams?</p><p>Let’s start with a traditional model, where Ops focuses on infrastructure, and we wait for customers to tell us something is wrong:</p><figure><img alt="A set of hugging-faces representing customers stand before a box titled “promises to users.” The box connects to a larger box with a quesiton mark, representing a chain of unknown causes, that all lead to a final box titled “infrastructure.” Next to the infra box is a cat, representing Ops. Above the diagram is a smily face representing the business." src="https://cdn-images-1.medium.com/max/1024/0*pJLDI4l3KAaa3SQq" /></figure><p>Let’s consider the worst-case scenario in this traditional state.</p><p>Users experience an issue: the business made a promise to users, and it isn’t coming true. But the infrastructure is fine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qmTqD8Oq62vXzV4A" /></figure><p>Users don’t care.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yYzz_4OwRB_5spfZ" /></figure><p>The Ops team may only have a small, partial view, and this partial view leads to another potential issue.</p><p>Say things are going well for the business, and more users start using their service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VCIh2rli6yyGv4pG" /></figure><p>Traditional Ops might be panicking even when something <em>good</em> is happening for users. And they might have a legitimate reason to be concerned!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FRQ8TscpKlFLmN8W" /></figure><p>Operations is ultimately a business problem, not just a technical one.</p><p>We need to be able to see the causal chain between different layers of a system.</p><p>We see a chain of dependencies surfacing differently as a mix of clear and ambiguous causes.</p><p>We also see layers of redundancy that allow for lower-level infrastructure failures without impacting users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y31PB7fO8NhA6hlR" /></figure><p>Moving from this conceptual awareness, you can think of how to identify and measure different areas of interest. Based on how apparent they are to users, we can group them into symptoms and causes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U3SX8FRtswtpNu6i" /></figure><p>Now that we have a model of the causal order, Ops can focus more on the same area of concern as the rest of the business: <strong>the users</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Olwh9HOIGKAkTfyD" /></figure><p>When issues arise, starting from a<em> few symptoms</em>, Ops can find the cause more efficiently than before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f_MYgx8ql9YbYYrF" /></figure><p>But if we know that causes precede symptoms, don’t we want to know when causes start to look wrong <strong><em>in advance</em></strong>?</p><p>Isn’t a symptoms-first approach <em>more</em> reactive and not as <em>predictive</em>, regardless of if we know a causal chain?</p><p>These are valid concerns if causes are as powerful as before and if we still need to do more to mitigate the impact of a failure deep within our system.</p><p>So suppose instead of those mitigations, we <em>alert </em>on causes.</p><p>We run a risk of being overwhelmed with causal failures. Alert fatigue and a high noise-to-signal ratio do not help us fix things faster.</p><p>Firefighting hardly seems more manageable if we’re merely <em>aware</em> of more fires.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VnFQf0y1V66efdBe" /></figure><p>How do we get out of this mess?</p><p>Ideally, we would ask, “What would it take to <em>only</em> alert on symptoms and not causes?”</p><p>We would build in layers of automation that obviate the need for alerts.</p><p>Why? Because alerts need to be <strong>actionable</strong>, we should have a system ready to handle failure.</p><p>With the ultimate goal of<em> turning off alerts</em> for causes, we automate as much as possible and progressively move closer to <em>just </em>the symptoms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HuuQuLXn8VsTmzfo" /></figure><p>Even in tossing away alerts, at no point are we turning off <em>monitoring.</em></p><p>We still need to monitor causes for troubleshooting, cost control, and so forth–but we are increasingly confident in our ability to focus on the symptoms primarily.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*43C2RSPTLYG-jQ6P" /></figure><p>Even with automation and monitoring in place, we had accepted earlier that any technical system guaranteed some failures.</p><p>Beyond the types of failures that we can prepare for, there are still unknown potential causes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TOY9VU13JPg9m1-P" /></figure><p>With a pattern for handling newly discovered causes, we avoid the need to obsess over them.</p><p>A bit of project work saves us from a lot of future toil. In a little time, we can return our focus to users. But we do it with the expectation that failure is inevitable, and we’re ready to discover future unknown causes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R6yrhk8lPbksk1vi" /></figure><p>Apply this perspective to orient discussions about expected improvements to Ops.</p><p>Think when an IT leader says, “We want <strong>complete</strong>,<strong> end-to-end</strong> visibility.”</p><p>In that case, though, what is the <em>main priority</em>?</p><p>“We want to be aware when something goes <strong>wrong</strong>.”</p><p>If you’ve designed a system to handle failure, what does it mean to “go wrong?”</p><p>There is a provocative way to get people to <em>think</em> about these issues:</p><p>“Starting tomorrow, turn off all alerts except for user-facing symptoms. Any objections?”</p><p>You will get a litany of dependencies, a lack of redundancy, and gaps in monitoring. It would be too abrupt to make this move all at once.</p><p>The point is really to ask:</p><p><em>“What will it take</em> to work <em>towards</em> that ideal state?”</p><p>It’s up to Ops to care more about <em>why </em>something isn’t working–even if users don’t. The change in perspective here isn’t merely about transitively caring about the same things; empathy is only a starting point.</p><p>Instead, what a user-centric perspective gives us is a different set of values:</p><ul><li>There are more possible causes of issues in our system than possible moves in chess; <em>accept</em> the ambiguity and focus on the <strong>most relevant</strong>.</li><li>What started as “business concerns” may result in discovering new technical issues that we didn’t previously see.</li><li>Starting with users and alerting Ops on symptoms is the sanest way to approach debugging. Alerting exclusively on symptoms should be our goal.</li><li>Automation isn’t a side project or a luxury. It’s the best means to obtain our goal confidently.</li></ul><p>Happy hunting!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4af0e115e14" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/operational-focus-why-symptoms-not-causes-e4af0e115e14">Operational Focus: Why Symptoms, not Causes?</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>