<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by David Feinblum on Medium]]></title>
        <description><![CDATA[Stories by David Feinblum on Medium]]></description>
        <link>https://medium.com/@dfeinblum?source=rss-ee31639b4146------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*0erj2Qm1-MsQT6J_h5Sopw.png</url>
            <title>Stories by David Feinblum on Medium</title>
            <link>https://medium.com/@dfeinblum?source=rss-ee31639b4146------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:12:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dfeinblum/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Change Risk Management: A Brief, Secure Addendum]]></title>
            <link>https://medium.com/jw-player-engineering/change-risk-management-a-brief-secure-addendum-c0eeab87cd67?source=rss-ee31639b4146------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0eeab87cd67</guid>
            <category><![CDATA[change-management]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[David Feinblum]]></dc:creator>
            <pubDate>Mon, 27 Jul 2020 18:30:49 GMT</pubDate>
            <atom:updated>2020-07-27T18:30:49.729Z</atom:updated>
            <content:encoded><![CDATA[<p>Back in April of this year, we shared a little bit about our newly-minted <a href="https://medium.com/jw-player-engineering/change-risk-management-at-jw-player-dccdb96471cc">Change Risk Management</a> (CRM) process that we’d just rolled out. After a quarter’s worth of risk assessments, avoided outages (H1 2020 had 30% fewer disruptions than H1 2019), and more automated tooling, we realized that there was one aspect of change that CRM was agnostic towards: security.</p><p>Now in fairness, this was somewhat intentional. While we were crafting CRM, myself and a few other engineers were working on improving our security processes separately, and the separation was conscious so that we wouldn’t get in each other’s ways. However, we came to a point last month where both processes were mature, and it was time that they be wed. In this post, I’ll explain how we folded security into CRM, as well as what we learned along the way.</p><p><strong>Championing Security</strong></p><figure><img alt="Spandex optional." src="https://cdn-images-1.medium.com/max/1024/0*ViOw-ookNSMGhZt0.jpg" /><figcaption>This picture was simply too good not to share. Borrowed from the good folks over at <a href="https://www.securedevelopment.org/handbook/security-champion/">Secure Development</a>.</figcaption></figure><p>In an organization of our size, scaling up our security efforts would’ve been quite the challenge. And without wishing to get too soap-boxy, writing secure code is as important as writing functioning code; a vulnerability should be treated exactly the same way we treat bugs. So security is everyone’s responsibility.</p><p>Bearing in mind how we democratized CRM, we applied a similar approach to security, building a volunteer-based security team that featured engineers from virtually every team in the organization. These “Security Champions” would be responsible (on a rotating basis) to review changes that involved security risks, identified by the teams making the changes.</p><p>This approach worked quite well, and as a member of this team, I’ve found that my nose for security concerns has strengthened quite a bit since we implemented this process. There are far more discussions on a regular basis surrounding security concerns, even when the Security Champion on-call isn’t notified. But now the question should be fairly obvious: how do security concerns affect the overall risk of a change?</p><p><strong>What’s Another Rubric Among Friends?</strong></p><p>Okay, so we didn’t actually implement another rubric. Instead, we simplified down all of the security questions present in our Software Development Lifecycle (SDLC) into these four questions, meant to be asked both before work begins <strong>and</strong> before a deployment:</p><ol><li>Are you accepting new data from the customer or a third party?</li><li>Are you modifying any access control or authentication mechanisms?</li><li>Are you handling any credentials, secrets, or personally identifiable information (tokens, IDs, location data) as part of this change?</li><li>Are there new dependencies or infrastructure as part of this change?</li></ol><p>If an engineer answers “yes” to any of the above questions, they have to notify the on-call Security Champion and request a code review of the change in question.</p><p>This fork of the process provided us with two main benefits. The first is that security is now entangled with risk management (as it should be), and the second is that the original CRM process could basically remain unchanged. Answering “yes” to any of the above questions doesn’t change the “rules of releasing” (again, check the <a href="https://medium.com/jw-player-engineering/change-risk-management-at-jw-player-dccdb96471cc">last post</a> for details on what these are if you’re interested), which means we didn’t need to retrain engineers or rewrite any of the automated tooling we’d already created (the release-ledger had a <em>very</em> fancy checkbox added to it to capture this data). And similarly, the preexisting security processes could continue to hum along as they were.</p><p><strong>Pulling it All Together</strong></p><p>Now that questions surrounding security are melded with CRM, we’re getting dangerously close to SDLC nirvana: a place where the path from ideation to deployment is given to you for free by automated tooling and simple processes. Undoubtedly, there will be additional forks we’ll have to add to CRM, and there will be evermore security and compliance questions we’ll want to fold in, too. But CRM has turned into quite the backbone for our deployment processes here at JW Player, and I’m excited to see what else we can do with it.</p><p>As an aside, I’d like to thank <a href="https://medium.com/u/5ba71919db4c">Matt Brennan</a> for his help and guidance throughout both of these projects. He’s been a great mentor and we at JW have a lot to thank him for!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0eeab87cd67" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jw-player-engineering/change-risk-management-a-brief-secure-addendum-c0eeab87cd67">Change Risk Management: A Brief, Secure Addendum</a> was originally published in <a href="https://medium.com/jw-player-engineering">JWX Product &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Change Risk Management at JW Player]]></title>
            <link>https://medium.com/jw-player-engineering/change-risk-management-at-jw-player-dccdb96471cc?source=rss-ee31639b4146------2</link>
            <guid isPermaLink="false">https://medium.com/p/dccdb96471cc</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[change-management]]></category>
            <dc:creator><![CDATA[David Feinblum]]></dc:creator>
            <pubDate>Wed, 15 Apr 2020 14:15:52 GMT</pubDate>
            <atom:updated>2020-04-15T14:15:52.304Z</atom:updated>
            <content:encoded><![CDATA[<p>If you do a google search for “Change Risk Management,” you’ll be besieged by a glut of wordy processes and huge matrices. You’ll also find that this phrase was born in heavy industry, not silicon valley.</p><p>Some of the earliest change management and risk assessments were created to help folks like chemical engineers make changes to their chemical processes, while ensuring their safety and the safety of the plant doing the science. And while changes to an API can’t cause physical damage, an incident in production can have negative impacts ranging from data loss to financial losses for customers.</p><p>As its name implies, Change Risk Management (CRM) is a process by which an organization can understand, measure, and mitigate risk. Here at JW Player, we’ve recently instituted such a process, and we thought it might be interesting to not only share what we came up with, but how we built it and what we hope to achieve with it.</p><p><strong>Why CRM?</strong></p><p>Back in October of 2018, we launched our Incident Management Process. The goal of this was to help mitigate the negative effects of incidents, increase inter-team communication while working the issue, and clarify external communication to our users. And while this process performed marvelously in these regards, it produced some interesting data.</p><p>A significant chunk of our incidents were “avoidable” in the sense that we did something to production which caused the problem. This made it clear that there was a great opportunity to make our changes more visible, understandable, and safe.</p><p><strong>The Change Risk Matrix</strong></p><p>At the core of any CRM process is the humble risk matrix. Implementations of these matrices vary, but they often look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/0*F-9h2cbuwSOGPI0B" /></figure><p>On the left, you can see risk factors like how much testing an application already has, how difficult it is to rollback, whether or not data loss is expected, etc. The right side of the matrix illustrates how these risk factors contribute to the overall riskiness of the change, indicated by increasing numbers in the top row.</p><p>To use a tool like this, you measure the risk factors of your change, add up the “risk points” you get in the top row, and then follow the process outlined in the bottom row. And while this sounds simple in principle, building a process around a matrix like the one shown above causes issues in practice.</p><p><strong>Democratizing CRM</strong></p><p>One of the most obvious issues with the above matrix is that it’s hideously complicated. Full disclosure: I wrote that matrix as the V0 of our CRM process back in December of last year, and I’d forgotten just how big it was when we first started.</p><p>A lot of the CRM matrices you’ll find online are designed for an entire team whose sole purpose is to perform deployments. This is a division of labor that’s quite common at large companies, but for smaller companies, you have to rely on everyone to be Site Reliability Engineers, and so your process has to be simple.</p><p>Realizing that the above matrix would require far too much overhead, we condensed it down by</p><ol><li>removing the “Extreme” category</li><li>switching to a yes/no system for determining risk</li><li>removing redundant/vague risk factors</li></ol><p>and what we ended up with was this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/0*zANxFqP5aI1UM0YR" /></figure><p>Calculating risk was now just as simple as asking yourself 5 questions. And based on the number of “yes” answers, you’d fall into one of three risk levels, each with their own process.</p><p>Obviously, there’s some loss of granularity by paring back the original matrix shown above, but that’s a small price to pay when the upshot is a matrix that’s easier to use and requires less mental overhead to understand.</p><p><strong>Winning Hearts and Minds</strong></p><p>So, let’s suppose that after reading this you’re inspired to draft up a CRM process for your own organization. It might seem that the hard work is determining the process and the risk factors that work for your specific situation. But this couldn’t be further from the truth.</p><p>For us, identifying key risks was easy because of our Incident Management process; for every incident, we had a well-written RCA. Indeed, the real difficulty was getting buy-in from engineers. And while no two organizations have the same culture, here’s how we did it at JW Player.</p><p>First, we communicated. A lot. I presented the above matrix to our engineering managers at the end of January, and then at an all-hands meeting in February. We then followed this up with a pair of one hour feedback sessions where engineers could vent their concerns and frustrations about the matrix. We added that feedback in and re-presented it during the first week of march, at which time we went into an open beta. Engineers could test out the process for a month and report back to us with other issues and experiences.</p><p>Additionally, we built an app called the release-ledger, which is a simple UI built on top of an API that allows for quick scheduling and measuring of our changes. Fill out a few fields, tick a few boxes, and the ledger will save your change into a log, while also telling you which process you need to follow based on the risks you face. Not only does this serve to lower the amount of overhead for our engineers, it also exposes all of our changes in one central location, improving communication.</p><p><strong>Next Steps</strong></p><p>Now that our CRM process is out in the wild, the natural question is: does it work? The short answer is that it’s too soon to tell, but we’ve got some important numbers we’re watching in order to determine its efficacy.</p><p>First, we are still keeping our ears open for feedback from our engineers. If the process is laborious and disruptive, no one will use it, and it won’t work (watching the number of deployments is a good stand-in for this). We’re also closely monitoring the number of incidents we have which are self-inflicted.</p><p>We will of course be continuing to hone and improve the tooling we use for CRM. There are plans to automate communication channels, spruce up and add functionality to our release-ledger, and integrate the process with our preexisting CI/CD infrastructure to streamline our changes even more. We also plan on extending the process to include considerations for security into calculating risk.</p><p>Above all else, the way to successfully institute a CRM process is to clearly outline what it is you want to achieve at the outset. Come up with some metrics you can watch to see if you’re achieving your goals. Listen to feedback, iterate often, and be willing to change things, though make sure to calculate the risk first!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dccdb96471cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jw-player-engineering/change-risk-management-at-jw-player-dccdb96471cc">Change Risk Management at JW Player</a> was originally published in <a href="https://medium.com/jw-player-engineering">JWX Product &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>