<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[True Tales from Engineering - Medium]]></title>
        <description><![CDATA[Engineers and SREs share their stories from personal experience and best practices, from xMatters in San Ramon, CA. - Medium]]></description>
        <link>https://medium.com/true-tales-from-engineering?source=rss----f16fb9eb876c---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>True Tales from Engineering - Medium</title>
            <link>https://medium.com/true-tales-from-engineering?source=rss----f16fb9eb876c---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:47:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/true-tales-from-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Training to Run a Marathon Is Like Bringing a Marathon Project to the Finish Line]]></title>
            <link>https://medium.com/true-tales-from-engineering/how-training-to-run-a-marathon-is-like-bringing-a-marathon-project-to-the-finish-line-originally-c1ad4ffd2b0f?source=rss----f16fb9eb876c---4</link>
            <guid isPermaLink="false">https://medium.com/p/c1ad4ffd2b0f</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[marathon]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[distance-running]]></category>
            <dc:creator><![CDATA[Lu Li]]></dc:creator>
            <pubDate>Wed, 18 Mar 2020 19:35:15 GMT</pubDate>
            <atom:updated>2026-06-21T17:09:40.060Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*XO6-8Q3pflTrpiGIBGFDsg.jpeg" /></figure><p><em>This article originally appeared in </em><a href="https://sdtimes.com/softwaredev/how-training-to-run-a-marathon-is-like-bringing-a-marathon-project-to-the-finish-line/"><em>SDTimes</em></a></p><p>Anyone who has run a full marathon knows about “the wall,” also known as the point where runners feel like they can’t go any further. This happens when runners hit the 20-mile mark which for many is the most challenging part of the race — their bodies send signals like muscle cramps and fatigue as a cry for them to stop. However, determined runners continue to put one foot in front of the other. Similarly, a marathon project, by definition, is a project that has been running forever and without an end in sight. Marathon projects typically result in teams who are frustrated and left with the perception that it will be impossible to deliver their project before the deadline (that’s already been pushed back three times).</p><p>One of xMatters’ high-profile projects ran into this situation last year. The project, a new feature that allows the end users to get reminders of coming on-call shifts, seemed straightforward. However, as our team began to scratch the surface, they realized the complexity of the project. This included crazy amounts of shift data, as well as the unlimited combinations of data. The project was pushed so many times, we got to a point where we couldn’t push it anymore.</p><p><strong>Test in production, a.k.a. ‘train on race day’</strong><br>After we exhausted our last resort, we made the important decision to test the project on production. Here’s how the scene unfolded:</p><p><strong>Q:</strong> “Wait a minute, did you say test in production? Or do you actually mean test in production-like data?”</p><p><strong>A:</strong> “The best place with production-like data is in production, so we test in production.”</p><p><strong>Q:</strong> “Doesn’t that sound too risky to run the project in production without proper performance testing? It’s like running a marathon without training.”</p><p><strong>A:</strong> “The risks are calculated. It’s like training for a marathon on race day, with preparation.”</p><p>Before I explain how we made this work, let’s take a look at my experience training for my first marathon. Back in 2016, not long after I accomplished my first 10K race, I decided to give a marathon a shot. As an engineer trained to approach the problem in a very logical manner, I conducted a technical spike, otherwise known as a Google search. I realized after this search that the most important training I could implement for a full marathon would be to build my endurance. This meant I needed to get to a point where I could complete anywhere from 32K-35K (roughly 20 to 22 miles) in a single run. However, I did have a constraint here — I really didn’t have the time. As a busy parent of two, I couldn’t afford to go on three-hour-long runs; not even occasionally. I came up with a plan:</p><ol><li>While I could not afford to participate in 30K runs, I would train up to 10K, but with the marathon pace.</li><li>I needed to figure out the areas I needed to work on most.</li><li>The best way to know a race is by doing it as often as possible. And I did just that.</li></ol><p>In 2017, one year after my venture into distance running, I ran my first marathon. I managed to walk/limp the last 10K across the finish line, with a 4+ hour finish time. This result was not ideal. However, I was able to get data that I wasn’t able to previously see from any of my training runs, including the following:</p><ol><li>I did the first half in 1:42, so it built my confidence that I could do a half marathon in 100 minutes.</li><li>The second half is is much more difficult than the first. I needed to ease into the first half and keep my pace lower.</li><li>Most importantly, preventing cramping needed to become my main focus. I needed to ensure I have proper hydration techniques, fuel and increased strength training.</li></ol><blockquote>Even after almost two years, if someone asked me if I was ready for another 26.2 marathon, I would say, “no.” I’m NEVER ready!</blockquote><p>Similar to my marathon experience, the marathon project we at xMatters were working on had multiple constraints. There was a hard deadline we weren’t able to push anymore, or we would risk the project being cancelled, and the features weren’t finished. Performance testing also hadn’t been done yet, as it required ample time that we just didn’t have. To meet the deadline, we decided to test in production. There were some risks with this including the potential for a bad user experience with a half-baked feature, flooded downstream services and outage, and other services on the same cluster being impacted if CPU or RAM is going out of the capacity.</p><p>To minimize the risk, we had to have risk control measures in place including:</p><ol><li><strong>Extensive monitoring: </strong>We set up extensive monitoring to ensure all matrices were in a safe range. At the same time, we gathered valuable data on traffic and usage patterns.</li><li><strong>Disabled live notification: </strong>The live notification portion of the feature was disabled. It was just micro brains doing their calculations in the back end without actually bothering end users.</li><li><strong>Feature turned on in canary fashion: </strong>The feature was turned on in the canary fashion, e.g., 5% of users in one cluster on day one, 10% on day two, etc.</li></ol><p>The result was quite eye-opening. We discovered that the usage pattern was quite different from what we expected. Some of the traffic peak was from the source of an epic data sync that we hadn’t planned for. It was clear that some of the expensive API calls caused performance problems.</p><blockquote>Compared to the first attempt, my pace improved from 9:40min/mile to 7:55min/mile.</blockquote><p>Just like my marathon training, testing in production did the trick! Identifying and knowing pain points was integral to achieving good progress. It made it easier for us to figure out which areas we needed to focus on the most. Actions like improving the performance by implementing caching were taken to resolve these problems. We wouldn’t have been able to figure out these areas just from performance testing, no matter how well it might have been defined. It helped us to have a good understanding of usage patterns, while directing us to areas that needed more work. Additionally, it uncovered a lot of unknowns and built the team’s confidence in delivering the feature.</p><p><strong>The last push</strong><br>Back to my marathon running. I didn’t make another attempt to run a marathon for quite a while. Even after almost two years, if someone asked me if I was ready for another 26.2 marathon, I would say, “no.” I’m NEVER ready! When I did decide to register for another marathon, my brain decided for me that a half would be plenty. Two weeks before race-day, I asked myself, “How many more times will you have the opportunity to run a race like this?” I guess the answer is not many. I realized, it’s important to start now; getting TO the start line is half the battle. That final push convinced me to train for another full marathon. Ready or not, I just ran it. And the result was not as bad as I originally anticipated. Compared to the first attempt, my pace improved from 9:40min/mile to 7:55min/mile.</p><blockquote>Just like my marathon training, testing in production did the trick!</blockquote><p>Similarly, in the final stage of any engineering project delivery, the team may need a final push. The last few miles are always the hardest, especially after all the preparation that has come before to get the team to where it is. Project managers need to make it clear that it’s time to get the project finalized, no matter what it takes. I have witnessed the team struggled on that daunting “wall” of a project, but I’ve also witnessed their determination to cross the finish line. When the team first started, it didn’t seem feasible to get the project out on time, especially given so many constraints. With determination, willingness to go above and beyond and time management, we accomplished our goals and completed the project.</p><p>One of the greatest marathon runners, Eliud Kipchoge, always said, “No human is limited.” A team, if empowered properly and with proper risk management, can make the impossible possible.</p><p>Learn more about xMatters and even <a href="https://www.xmatters.com/signup?medium">try it for free</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1ad4ffd2b0f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/true-tales-from-engineering/how-training-to-run-a-marathon-is-like-bringing-a-marathon-project-to-the-finish-line-originally-c1ad4ffd2b0f">How Training to Run a Marathon Is Like Bringing a Marathon Project to the Finish Line</a> was originally published in <a href="https://medium.com/true-tales-from-engineering">True Tales from Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[After a Deployment Error, Should You Fix Forward or Roll Back?]]></title>
            <link>https://medium.com/true-tales-from-engineering/after-a-deployment-error-should-you-fix-forward-or-roll-back-e573f7420a36?source=rss----f16fb9eb876c---4</link>
            <guid isPermaLink="false">https://medium.com/p/e573f7420a36</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[rollbacks]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[incident-management]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <dc:creator><![CDATA[Doug Gorley]]></dc:creator>
            <pubDate>Wed, 26 Feb 2020 20:07:57 GMT</pubDate>
            <atom:updated>2020-02-26T20:07:57.621Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*edSjiuNYJZrc2V_YFzXP8A.jpeg" /></figure><p>You open your eyes to the sound of your phone, bleating at you from your bedside table. This can’t be good, you think to yourself, as the adrenalin starts to flow. The glowing red numbers staring back at you from the clock read 4:48am.</p><p>As you settle down into your chair and unlock your laptop, the bright light from your screen seems blinding. A couple of minutes later, you’re on the conference bridge for an incident. You gather that a key component of your infrastructure is failing due to an upgrade the day before, your customers’ messages are not being processed, and they’re furious about it.</p><p>Engineers responsible for the component are testing some hypothesis, and asking for time to fix it. “Maybe if… I bet we can…” Meanwhile, senior management are on the call, and have to field tough questions from customers, and they want to pull the plug on this change and roll back, getting customers back online ASAP. Engineering isn’t so sure, saying that the nature of the change makes rollback a dicey proposition. They turn to you for your opinion.</p><p>What do you say?</p><p><strong>Understand your options</strong><br>Being stuck in this position can be daunting, and there isn’t always an easy answer. Fixing your components in-place means you can move forward after this incident; and once it’s fixed, that’s (hopefully) the end of your remediation. But, this costs time while your services are offline, leaving your customers unserved and unsatisfied.</p><p>On the other hand, rolling back the change might be quick (depending on your deployment strategy), but unless you intend to permanently freeze there, you still need to get that change out, and soon. There’s additional remediation work required here to correct the change, perform the requisite QA, and re-deploy your changes into production.</p><p>When incidents happen, the time you’ve invested and the plans you’ve made preparing for it is key. Understanding the strategies available to you, and when to pursue them, will determine whether your recovery efforts are a success.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/194/0*aGNUlfTdx9h-Kzbi.png" /></figure><p><strong>The Anna Karenina Principle of Deployments</strong><br>Leo Tolstoy’s book Anna Karenina begins:</p><blockquote><em>All happy families are alike; each unhappy family is unhappy in its own way.</em></blockquote><p>This concept maps as well to software deployments as it does to dysfunctional 19th-century Russian families. When a rollout goes as planned, it looks like every other rollout: no errors to remediate, no disrupted services to restore, just new bug fixes or features available for use. On the other hand, when a deployment goes poorly, any number of scenarios might unfold. Consider the following:</p><p>An upgrade to your messaging bus leaves consumers struggling to get messages from their queues. The messaging bus runs on VMs, which were upgraded in place.</p><p>A change to an internal data format was rolled out at 4pm Pacific Time, and everything seemed fine until 8am Eastern Time, when your largest customers began feeding traffic to your system. Too late, you realize what the computational cost of processing this new format looks like under heavy load. By now, you have 13 hours worth of data in the new format, which amounts to tens of thousands of transactions.</p><figure><img alt="Let’s take a look at how Dynatrace and xMatters remove the stress from blue-green deployments." src="https://cdn-images-1.medium.com/max/1024/1*GziemhHOV1KHcvpIGIv6CQ.png" /></figure><p>An example of how xMatters Flow Designer — and in this case Dynatrace — removes the stress from blue-green deployments.</p><p>An update to a service which was just deployed contains a bug which is destabilizing your customers’ environments. Because your CI/CD pipelines follow a blue-green deployment pattern, a rollback is technically straightforward, but there’s a hitch. The new update includes new functionality which you are contractually committed to provide to some key customers, and rolling back means missing your deadlines.</p><p>Each of these situations might require a different response from your organization, based on the options available to you. Ensure you know what those options are before deploying the change.</p><p><strong>What are your Options?<br></strong>When dealing with a failure scenario, your ability to recover is only as good as the effort you’ve invested in developing your recovery options. Careful planning and preparation give you options for how you deal with an incident, and will guide you on the path to successfully restoring your services to customers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/0*WDfKIcgMUvLX_N0B.png" /></figure><blockquote><a href="https://www.xmatters.com/self-healing-devops-ebook/?utm_source=blog&amp;utm_medium=web&amp;utm_content=organic">Read the DevOps Self-Healing DevOps eBook</a></blockquote><p><strong>Reverting a Change</strong><br>By far, the simplest solution to resolving a problem is usually to revert the change, and go back to running the previous version of the service or configuration. How this works depends on the service and the nature of the change, as well as the extent to which you’ve effectively prepared to implement a rollback. Some examples of reverting a change include:</p><ul><li>Running a rollback script against the database to remove a new index</li><li>Removing the new version of a service from a load balancer pool</li><li>Disabling a feature flag which exposed customers to new functionality</li></ul><p>With proper planning, many changes can be reverted quickly and safely, with little or no impact on your uptime and service delivery. Wherever possible, reverting a change which is causing a customer incident should be the initial plan of attack.</p><p><strong>Failover to a Secondary Environment</strong><br>In some cases, it’s just not as simple as reverting a change. An example of this is when there is, in fact, no change in play, and the incident is caused by some external factor, such as a change in your application traffic pattern, or a hardware failure in your datacenter. In these scenarios, having a secondary or DR environment available to you gives you the option to redirect application traffic away from the problem area of your application stack, and instead process traffic in a different logical or geographic area. Such a change may involve some orchestration in your application stack, such as simultaneously promoting a new leader in your database cluster while rerouting traffic at your load balancers. While somewhat complex, investing in the ability to leverage a secondary environment can be particularly advantageous when the cause of your failure is nebulous or out of your control, or the path to reverting your change is onerous and time consuming.</p><p><strong>Fix Forward</strong><br>As a last resort, you might need to forgo the rollback plans, and simply fix the issue in-place. The challenge with deciding to fix an issue in place is until you know how to fix the issue — until you can diagnose and remediate the issue, and then deploy a fix — your customers will continue to experience its effects.</p><p><strong>Developing Options for Rollback</strong><br>When planning a change in your software environment, there are several approaches which lower the barrier to rollback in the event of a catastrophic failure. We’re going to briefly discuss feature flags and dark launches, canary releases, and blue-green deployments.</p><figure><img alt="Feature flags at work" src="https://cdn-images-1.medium.com/max/822/1*rcaMdLqTySZUol4evdAJPA.png" /></figure><p><strong>Feature flags and dark launches: </strong>Deploying your changes behind a feature flag is a method of controlling the exposure of your change to your customers. In such a scenario, a new feature or change is deployed to your customer’s application environment, but is disabled by default. Such a deployment is called a dark launch, and is a popular strategy for managing the risk associated with new application functionality.</p><figure><img alt="Canary release flowchart" src="https://cdn-images-1.medium.com/max/1024/1*i-HoYvLwWJXJn-GmWkFLYQ.jpeg" /></figure><p><strong>Canary releases: </strong>A canary release involves exposing the darkly-launched functionality to a certain segment of your customers, and evaluating the results. This approach can be highly effective not only for limiting the impact of defects in code, but also for evaluating customers’ responses to changes on a small scale before exposing those changes to a larger audience.</p><figure><img alt="Blue-green deployment flowchart" src="https://cdn-images-1.medium.com/max/1024/1*-XIbVAcSt6LbjkcTf3FtMQ.jpeg" /></figure><p><strong>Blue-Green Deployments:</strong> In order to roll back a change, it’s important that you have something to roll back to. In-place upgrades to software or cloud infrastructure can obliterate the previous working version, leaving you without a viable rollback target. Blue-green deployments involve deploying the new services in parallel with the existing services, then rerouting traffic to the new services. Not unlike a canary release, blue-green deployment can be done in stages, rather than rerouting all traffic at once. A newly deployed service may initially get 2% of the traffic, then 10%, then 50%, and finally 100%, all while your team monitors error rates for unwanted patterns. Retaining the previous version of a service until the next deployment means that, at any time, traffic could be rerouted back to the old service version to mitigate the impact of a bug.</p><p><strong>When to Fix Forward</strong><br>If none of the rollback strategies described above are available to you, you may be stuck with fixing the issue in-place. Some changes simply don’t lend themselves well to rollback, such as:</p><ul><li>Changing a communication protocol between two (or more) components in your application stack, where the clients become dependent on the new protocol and are unable to revert to the previous one</li><li>Changes to a database schema which were destructive on the original data (e.g., rewriting source data in a new format), or schema changes which client services depend on for their new features to function correctly</li><li>Upgrades to components of your application stack which do not gracefully lend themselves to a blue-green deployment strategy, such as databases and message brokers</li></ul><p>Because of the perilous nature of a fix-forward strategy, it’s worth the time to identify those areas of your environment that don’t allow for rollback, and look for opportunities to redesign them, with the goal of making rollbacks simple and safe.</p><p><strong>Have an Incident Management Plan</strong><br>Having a strategy planned out before an incident occurs is your best bet for balancing the internal impact of your remediation against the service level you provide to customers. For example, your plan can include an initial response where you attempt to identify and fix the issue in place, but that effort is timeboxed to 30 minutes, after which a rollback is performed to quickly restore service. This planned response can vary based on the customers impacted, the particular component in your system that’s affected, or some other parameter; regardless of the specifics, planning in advance for failure scenarios gives you a framework to guide decisions when under pressure.</p><p><strong>Document your Strategies in Runbooks</strong><br>In an incident response scenario, your remediation strategies are only as good as your ability to execute them in the moment. Too often, individuals in an organization become subject matter experts in a service, but then fail to disseminate key knowledge for dealing the service in an incident, and become a single point of failure in the incident response. One effective method for turning this tribal knowledge into institutional knowledge is to document incident response plans in runbooks, which give newcomers to your service simple, step-by-step processes to follow to address an issue. In the case of a rollback, having instructions on how to disable a feature flag or revert to a previous version are likely to reduce the time required to correct an issue, as well as reduce the stress on the personnel tasked with performing the fix.</p><p><strong>Conclusion<br></strong>Downtime incidents are stressful for both you and your customers, and resolving them quickly and effectively depends on understanding the options available to you, as well as having a plan to implement them. Fixing an issue in place or rolling back a change each have their own set of pros and cons, and must be measured against the specifics of the incident. Creating options for rolling back changes, understanding the boundaries of a fix-forward strategy, and advance planning of how to manage an ongoing incident are the best tools you have available to restore your system within your SLAs, and keep your customers happy.</p><p>Whether you’re rolling your deployment back or fixing it forward, time matters. Some common ways to gain efficiencies include getting critical information to the right people, using templated workflows, and automation. If you’re an xMatters customer, you can use <a href="https://www.xmatters.com/features/workflow-process-automation">scenarios to develop templates and response counts for different situations</a>. You can use <a href="https://www.xmatters.com/features/workflow-process-automation">Flow Designer to build drag-and-drop workflows</a> for repeatable, scalable processes for complex situations. You can execute <a href="https://info.xmatters.com/rs/178-CPU-592/images/DevOps_Self-Healing_Dynatrace_eBook.pdf?mkt_tok=eyJpIjoiTVRCaVpUSTBOV1F3TlRJMiIsInQiOiJDMGJFZzUzVTdOTlFYWnBWN0pYenQ3SWtyRUpqT2VzUnEyWTJ0WE1LT29kaVdESSsweXRXaWtPWWxQREFxSmpNSEpYYmFWSldcL0EwcEFKVkR2aFBnR3huQ2Y4RlI2Mk8ydWpncDBCZW40VWwzYVVqK294cTI2T2R3anZnZWNsM08ifQ%3D%3D">automated remediation</a> for things like process crashes, full disk prevention, and blue-green deployment rollback.</p><p>Looking to leverage integrated tools to drive software development security? <a href="https://www.xmatters.com/signup?utm_source=med&amp;utm_medium=social&amp;utm_content=organic">Try xMatters… for Free</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e573f7420a36" width="1" height="1" alt=""><hr><p><a href="https://medium.com/true-tales-from-engineering/after-a-deployment-error-should-you-fix-forward-or-roll-back-e573f7420a36">After a Deployment Error, Should You Fix Forward or Roll Back?</a> was originally published in <a href="https://medium.com/true-tales-from-engineering">True Tales from Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Put Software Development Security First]]></title>
            <link>https://medium.com/true-tales-from-engineering/how-to-put-software-development-security-first-51c5dbb1c424?source=rss----f16fb9eb876c---4</link>
            <guid isPermaLink="false">https://medium.com/p/51c5dbb1c424</guid>
            <category><![CDATA[gdpr]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[xmatters]]></category>
            <dc:creator><![CDATA[Kit Brown-Watts]]></dc:creator>
            <pubDate>Wed, 19 Feb 2020 18:31:04 GMT</pubDate>
            <atom:updated>2020-02-19T18:31:04.169Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*4Y6TS-x9N9CuACi7S8gV6g.jpeg" /></figure><p><a href="https://www.xmatters.com/blog/devops/how-to-put-software-development-security-first/"><em>This article originally appeared on xMatters.com</em></a></p><p>Every investment decision comes with trade-offs, usually in the form of cost, quality, or speed. The CQS Matrix captures the dilemma most Product People face. If you’re a perfectionist, you may find the temptation to build 100% of a feature set irresistible. On the other side of the perfection spectrum, the temptation to “just ship it” can be equally strong.</p><p>When it comes to software development security and compliance, regardless of your product instincts and tendencies, most top-tier organizations tend to take quality very seriously and are often willing to invest lots of resources and time to meet accepted standards. Especially companies who have achieved standards like <a href="https://www.xmatters.com/security/gdpr-privacy-myth-versus-reality/">GDPR or ISO27001</a>. This has the unintended consequence of product leaders often “playing it safe” and erring on the side of security. However, the idea that security takes extra time, costs more or reduces innovation is often merely a symptom of not <a href="https://www.xmatters.com/security/devsecops-key-cost-reduction/">properly building security into the Software Development Life Cycle (SDLC) in the first place</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e8bE6BKXERlFPdRW.png" /><figcaption>Speed, quality, and cost in product development. Image source: <a href="https://m.subbu.org/cloud-optimization-circus-65f3b47d0c79">https://m.subbu.org/cloud-optimization-circus-65f3b47d0c79</a></figcaption></figure><p>Building software development security right into the product delivery process through automated build processes is frequently referred to as DevSecOps. This portmanteau of DevOps and Security plays off the DevOps mindset of continuous improvement and continuous delivery using containers. It also includes testing security compliance targets with each code commit, in addition to testing against performance and deployability metrics.</p><p>The benefits of testing early and often are well documented. The <a href="https://deepsource.io/blog/exponential-cost-of-fixing-bugs/">National Institute of Standards and Technology</a> (NIST) published well-cited data illustrating the exponential increase in cost associated with fixing issues in production instead of at the coding or integration testing phase of the delivery cycle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u-5ZQ_JCTv_wUvME.png" /><figcaption>Increase in cost associated with fixing issues in production instead of at the coding or integration testing phase. Image source: <a href="https://deepsource.io/blog/exponential-cost-of-fixing-bugs/">https://deepsource.io/blog/exponential-cost-of-fixing-bugs/</a></figcaption></figure><p>Further evidence of material gains can be found in <a href="https://services.google.com/fh/files/misc/state-of-devops-2019.pdf">Google’s 2019 State of DevOps report</a>. It identifies top-tier performers as companies who are able to recover from incidents 2,604x faster, deploy code 208x more frequently, reduce their lead times from commit to deploy by 106x, reduce their change failure rate by 7x, and meet their organizational goals 2x as often as their lower-tier counterparts.</p><p>Creating efficient automation within a team’s software delivery toolchain doesn’t happen on its own. It takes investment upfront, support from management, and dedication. It also takes tools, culture, and some common sense.</p><p><strong>Enabling tools</strong><br>Scanning code for vulnerabilities as part of the build-test-deploy pipeline is a critical enabler for CI/CD processes. Continuous, automated security scanning is so important because our development teams could never comb through every line of code or every changeset. Here’s why:</p><p>Modern applications typically rely on open source libraries, which themselves rely on other third-party dependencies. Taken together, these chains of dependencies constitute a complex web of code, owned and maintained by disparate teams distributed all over the world who are often unaware of each other’s existence or how they are using code.</p><p>Neglect security scanning at your own risk. Popular package repositories like NPM have been subject to attacks, leaving developers at the mercy of repository owners (either hijacked or otherwise).</p><p>Fortunately, we have tools to help with this, including <a href="https://www.xmatters.com/devops/socks-on-before-shoes-a-smarter-dependency-manager/">open source tools like UpGreat</a>, which make it easy to ensure complex chains of dependencies are up-to-date.</p><p><strong>Company culture</strong><br>But tools alone do not make a top-performing product development team — it’s also about company culture. <a href="https://www.scss.tcd.ie/publications/theses/diss/2017/TCD-SCSS-DISSERTATION-2017-056.pdf">Kai Song says a culture of standards compliance</a> “should be a continuous process rather than just meet the audit requirement annually.”</p><p>An effective culture can make the best possible decisions early enough in the software delivery process. No scan is perfect. We apply the scientific method to identify real threats from the red herrings, but there is still an art to it.</p><p>This is especially the case for native mobile apps. Native apps typically follow a different release cadence than web apps because they need to be updated on the device instead of simply refreshing a webpage, and they go through a more involved review process with Apple and Google. The code is much harder for potential hackers to inspect, and apps can be designed with much greater control over their runtime operation. At xMatters, we integrate continuous Static Application Security Testing with a variety of tools and services. We build vulnerability scanning into every commit through automated processes, which saves us a lot of time, most of the time.</p><p><strong>Common sense</strong><br>Automation is not a silver bullet either. Automated scanning rulesets don’t know your use case, but they still generate findings that need to be addressed.</p><p>In the late 20th century, MIT professor Ronald Rivest authored several well-known cryptographic hashing algorithms. One of the most popular, MD-5 (the 5th iteration of the Message Digest algorithm) continues to be widely used today, despite having been shown to be “<a href="https://www.scss.tcd.ie/publications/theses/diss/2017/TCD-SCSS-DISSERTATION-2017-056.pdf">cryptographically broken and unsuitable for further use</a>.” This was part of a wider realisation that the standards defined in SHA-2 (Secure Hashing Algorithm) were inadequate, giving rise to the current standard, SHA-3. Clearly, anyone still using SHA-2 algorithms must be disregarding security best practices, right?</p><p>Not necessarily.</p><p>Hashing algorithms can be used for other purposes beyond generating unbreakable cryptographic strings. They are also good at generating unique values, such as unique device identifiers — this is how it was being used. However, it would always get flagged (rightfully so) by clients’ scans showing usage of a “cryptographically insecure” algorithm.</p><p><strong>Lesson learned:</strong> Don’t use security algorithms if you’re not using them for security. Use fit-for-purpose algorithms rather than general case ones to avoid the appearance of non-compliance. In our case we’ve now switched to a hashing algorithm which is designed explicitly for performance and unique ID generation.</p><p>Striking the right balance between software development security and user experience requires plenty of critical thinking to make the best decision. Take the following example — imagine you have no other information other than the following:</p><blockquote><strong>Information Leakage:</strong> Copy Paste (Medium Class Vulnerability)</blockquote><blockquote><strong>Description:</strong> iOS/Android devices support copy/paste functionalities. Another user or malicious application can recover sensitive data copied to the pasteboard/clipboard. Since the copy/paste buffer is shared between applications, any user or application can paste the data from the buffer into another application. A malicious app can retrieve the copied information such as credit card information.</blockquote><blockquote><strong>Solution:</strong> Consider clearing sensitive data from the pasteboard/clipboard buffer whenever the application enters background mode.</blockquote><p>What might you take away from this? Observe your initial thoughts.</p><p>When you focus only on the risk in the worst-case scenario, it is easy to forget the feature’s value. Copy-Paste is a familiar, universal function that users expect to use everywhere in their digital lives. It saves time under stressful and chaotic conditions.</p><p>In our case, the recommendation was to clear the device’s clipboard when our users switch apps. Since the xMatters app helps DevOps teams get notified and handle incidents as quickly as possible, it wasn’t long before we heard from customers who were prevented from copying and sharing complex technical logs with their dev teams to get help. We quickly re-instituted Copy-Paste, to our customers’ relief and gratitude.</p><blockquote><strong>Lesson learned:</strong> Be careful when making changes to OS-wide defaults. Connect with real end-users often — or even better: become a real end-user yourself.</blockquote><p>Looking to leverage integrated tools to drive software development security? <a href="https://www.xmatters.com/signup?blog">Try xMatters… for Free</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51c5dbb1c424" width="1" height="1" alt=""><hr><p><a href="https://medium.com/true-tales-from-engineering/how-to-put-software-development-security-first-51c5dbb1c424">How to Put Software Development Security First</a> was originally published in <a href="https://medium.com/true-tales-from-engineering">True Tales from Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Socks On Before Shoes — Smarter Way to Upgrade JS Dependencies]]></title>
            <link>https://medium.com/true-tales-from-engineering/socks-on-before-shoes-smarter-way-to-upgrade-js-dependencies-23212f80f66e?source=rss----f16fb9eb876c---4</link>
            <guid isPermaLink="false">https://medium.com/p/23212f80f66e</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[xmatters]]></category>
            <category><![CDATA[dependencies]]></category>
            <category><![CDATA[dependency-manager]]></category>
            <category><![CDATA[cli]]></category>
            <dc:creator><![CDATA[Buğra Fırat]]></dc:creator>
            <pubDate>Thu, 05 Dec 2019 21:19:11 GMT</pubDate>
            <atom:updated>2019-12-06T14:58:39.541Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Socks On Before Shoes — Smarter Way to Upgrade JS Dependencies</strong></h3><figure><img alt="Socks On Before Shoes — Smarter Way to Upgrade JS Dependencies" src="https://cdn-images-1.medium.com/max/682/1*l1_MSicJu6858rGiA9BTlg.jpeg" /></figure><p><a href="https://www.xmatters.com/devops/socks-on-before-shoes-a-smarter-dependency-manager/"><em>This article first appeared on xMatters.com</em></a><em>.</em></p><p>As we work in more and more projects in a modular manner, we end up creating a number of packages which all have dependencies of their own. Updating and managing these dependencies becomes a problem after some time. Some factors that contribute to this are slow development cycles (combined with fast publishes in the JS world), avoiding scope creep, and testing burdens.</p><p>So we sometimes end up with packages which work just fine, but have some outdated dependencies we want to renovate.</p><p>There are a few solutions to this problem of updating package dependencies, but none in the wild were quite right for us. The biggest players in this space are <a href="https://greenkeeper.io/">Greenkeeper</a> and <a href="https://dependabot.com/">Dependabot</a> (now acquired by GitHub!). It’s no mystery why GitHub acquired Dependabot, as it’s very tightly integrated into GitHub itself. Greenkeeper is also a great solution and very tightly integrated with GitHub, but it’s not free for private repositories.</p><p>What to do if you’re not using GitHub at all? Or you have private packages whose dependencies you still would like to keep updated?</p><p>Well, until now, there weren’t many dependency manager options, which left most of us doing it manually.</p><figure><img alt="Until now, there weren’t many dependency manager options, which left most of us doing it manually." src="https://cdn-images-1.medium.com/max/1024/1*1c8u16y9aN0B0nlPGxSWPg.png" /><figcaption>ugh…</figcaption></figure><h4>How many Pull Requests to upgrade all dependencies?</h4><p>Another issue that we’ve had with previous solutions is that, very cleverly, they do the bare minimum of upgrading a dependency, and leave the rest to your CI/CD pipelines. This is great for most cases; but in a world where time means money, you might have to think twice about having to wait for and merge 25 pull-requests that update 25 different dependencies.</p><figure><img alt="In a world where time means money, you might have to think twice about having to wait for and merge 25 pull-requests that upd" src="https://cdn-images-1.medium.com/max/267/1*TPQyQ4Kt8uCWZKJnHWzBbQ.png" /><figcaption>found god on the 26th merged PR</figcaption></figure><p>Finally, instead of that evergreen-utopia, you might want to make conscious and well-tested upgrades that help keep your sanity in your day-to-day environment.</p><h4>upgreat ups the dependency manager game</h4><p>We developed <a href="https://github.com/xmatters/upgreat">upgreat</a> to address some of these issues and make it easier for us to manage, monitor and schedule dependency upgrades for our internal projects, and are now releasing it to the wild in hopes that it will be useful for other developers and organizations.</p><p>upgreat aims to address some of these issues and operates in a different way. You plan your upgrade path, then execute it in one go. When it’s done doing its thing, you are left with a repository that has all its dependencies upgraded. Any upgrades that cause test or build failures are rolled back, and upgraded to the latest version for which the tests still pass. Finally, you get a nice report of what was upgraded, and get a list of changelogs for those dependencies that failed the upgrade.</p><p>Another nuance here is that you can upgrade all of your devDependencies without your tests failing once. Sure, upgrade all my build tooling and run unit tests afterwards, what could go wrong? upgreat runs build scripts if you’re updating devDependencies to save you from headaches.</p><p>In a way, upgreat embraces the “manual-ness” and the human element of undertaking this task, while making things easier for the maintainer(s) at every step.</p><h4>Socks on before shoes</h4><p>On a cursory look you could think that doing such an upgrade is super simple. Get the list of all outdated dependencies, then loop over them and upgrade each one to the latest one. Done!</p><p>Or is it?</p><p>This was our first approach to the problem, but it proves to be rather fragile and too naive for any sufficiently complex package. Lets take a simple app with Webpack for example:</p><pre>// package.json<br>&quot;file-loader&quot;: &quot;0.11.1&quot;,<br>&quot;webpack&quot;: &quot;2.5.1&quot;,</pre><p>If you upgrade file-loader to 4.2.0 first, and then try to upgrade webpack to 4.41.2 afterwards, you will get into a state where your builds will be broken. Do you see why?</p><p>It’s because file-loader versions have peerDependency to webpack versions. You need to upgrade the libraries in the correct order to avoid running into issues. Just like in real life, if you put your shoes on first, and then try to put your socks on, you will likely fail.</p><p>upgreat’s solution to this problem is straightforward. Just like webpack, make, and a host of other projects, upgreat sorts the packages to be upgraded in topological order. This makes sure that any packages that others in the list depend on are upgraded first, and avoids any nasty collisions.</p><h4>Topo-what?</h4><figure><img alt="I just pretend I understand stuff." src="https://cdn-images-1.medium.com/max/384/1*hm7iiDvaQr5KLgfeuW0_TA.jpeg" /><figcaption>I just pretend i understand it perfectly</figcaption></figure><p>Scary as it might sound, topological ordering is a straightforward concept. When presented with a list of dependencies and a flow, we want to find an ordering that will acknowledge the dependencies (like file-loader@4.2.0 depending on webpack@4.41.2).</p><p>Think of it as the order in which you would put on your socks and shoes. The astute among you might realize that there could be many different topological orderings for a given dependency graph. The good thing is, they are all valid and will work just fine.</p><p>Here are some ways you can put on your socks and shoes:</p><pre>Left Sock -&gt; Right Sock -&gt; Left Shoe -&gt; Right Shoe</pre><pre>Left Sock -&gt; Right Sock -&gt; Right Shoe -&gt; Left Shoe</pre><pre>Left Sock -&gt; Left Shoe -&gt; Right Sock -&gt; Right Shoe</pre><pre>Right Sock -&gt; Left Sock -&gt; Right Shoe -&gt; Left Shoe</pre><pre>Right Sock -&gt; Left Sock -&gt; Left Shoe -&gt; Right Shoe</pre><pre>Right Sock -&gt; Right Shoe -&gt; Left Sock -&gt; Left Shoe</pre><p>and various permutations. Ultimately, the end state will be that you have worn socks and shoes on both feet. The order in which you do it (despite the heated arguments you might read on the internet about <strong>the right way</strong> of doing it) does not matter, as long as you don’t break the internal dependency between Sock -&gt; Shoe.</p><p>Similarly for upgrading dependencies, the exact order of operations doesn’t make much difference, as long as you address the inter-dependencies.</p><h4>Silver bullets and panaceas</h4><p>I don’t think I need to tell you that there is no foolproof and fully automated way to upgrade dependencies of your projects. Unless you choose to live on the wild side and not write tests, but then you could just include package@latest until everything breaks down one day, including your sanity.</p><p>After trying a few of these approaches in xMatters, we’ve found this one works okay for our purposes. It automates the most grueling parts of the library upgrade process; but after all is said and done, it leaves it up to you to follow through and commit the changes. You can pretend you did it all by yourself, I won’t tell.</p><p>As a developer it feels great to be able to give back to the community who have basically provided for my livelihood over the years with amazing open source projects. I hope to be involved in more projects as we embrace open-source in xMatters and share some of our ideas and other projects with the community from which we all benefit greatly. Check out upgreat at <a href="https://github.com/xmatters/upgreat">https://github.com/xmatters/upgreat</a>. I would love to hear your thoughts on your experiences doing library upgrades and opinions on upgreat.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23212f80f66e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/true-tales-from-engineering/socks-on-before-shoes-smarter-way-to-upgrade-js-dependencies-23212f80f66e">Socks On Before Shoes — Smarter Way to Upgrade JS Dependencies</a> was originally published in <a href="https://medium.com/true-tales-from-engineering">True Tales from Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Created a Great Data Transfer Service from a Hairball Using Google Cloud Pub/Sub]]></title>
            <link>https://medium.com/true-tales-from-engineering/how-we-created-a-great-data-transfer-service-from-a-hairball-using-google-cloud-pub-sub-baf1887b3302?source=rss----f16fb9eb876c---4</link>
            <guid isPermaLink="false">https://medium.com/p/baf1887b3302</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[google-pubsub]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Tim Thompson]]></dc:creator>
            <pubDate>Tue, 29 Oct 2019 23:45:13 GMT</pubDate>
            <atom:updated>2019-10-29T23:45:13.046Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/682/0*yqlaYojcz6gW20um.png" /></figure><p>Our customers sync a lot of data into our systems. Our original data synchronization became such a hairball I’m pretty sure the <a href="https://www.youtube.com/results?search_query=funny+cat+videos">YouTube cat cabal</a> had something to do with it. So we decided to start from scratch (pun intended) and implement a new data transfer service, which we ingeniously named the Transfer Service.</p><p>When we were thinking about the <a href="https://www.xmatters.com/features/enterprise-grade-architecture">enterprise-grade architecture</a> of this new Transfer Service, our architect helpfully supplied exactly one diagram to help us out:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f9FhdZQ4bt7rp1x4.jpg" /></figure><p>There’s plenty to unpack in that graphic, and as a UI developer I kept trying to imagine what was in all those enigmatic blue boxes. But for this article let’s focus on that “transfer job processor” object, which is a job queue enabling asynchronous processing of, well… transfer jobs. And it would also eventually help scale the service by decoupling the transfer job processor from the “transfer API.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*dIHOHuet3woq6o4W.png" /></figure><p><strong>The right tool for the jobs<br></strong>As we started thinking through the challenge, we had to make some technology choices. Other dev teams were already using Apache Kafka as a stream processor that can persist messages. Kafka provides delivery guarantees as well as other non-functional requirements such as disaster recovery and fault tolerance. However, using Kafka would also mean managing its infrastructure complexity, which was a complaint we heard from those other teams. It also requires Zookeeper as a distributed configuration management tool.</p><p>Other teams were using RabbitMQ, but they told us that the their implementations were less than ideal and they would have taken a different path if they had to do it over again. It seemed like dynamically creating queues was not easy to do in RabbitMQ. And, as with Kafka, we’d have to manage the infrastructure.</p><p>We had just started the migration of our service to <a href="https://cloud.google.com/">Google Cloud Platform</a>, so naturally we looked into what functionality Google had to offer. We first considered Google Dataflow, but concluded that it works better for self-contained processing that can be easily parallelized. While it can auto-scale workers, the actual work is done via calls to our API and would not be self-contained in a worker. In a nutshell, if DataFlow scaled to 1,000 workers, we would kill our API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*xuxzcJnqjfudHypU.png" /></figure><h4>“Our original data synchronization became such a hairball I’m pretty sure the YouTube cat cabal had something to do with it. So we decided to start from scratch.”</h4><p><strong>We (may) have a winner<br></strong>Ultimately, it looked like <a href="https://cloud.google.com/pubsub">Google Cloud Pub/Sub</a> might have the features we needed. Let’s dig into its pros and cons.</p><p>The Pros of Google Pub/Sub:</p><ul><li>Google manages the infrastructure: In the absolute sense, this is a big pro, but you should weigh financial cost. The pricing page states that it costs 0.06USD/GiB/month after 10GiB of traffic. For each transfer job initiated, we publish a small JSON object of metadata to Pub/Sub. Each metadata JSON object was about 578 Bytes, but according to Google, “The minimum billable volume of data per request (not message) is 1 KB of data.” We’d have to be processing more than a million (1GiB = 1³⁰ Bytes = 1073741824 / 1024 B = 1,048,576) jobs in a month before we started getting charged. For us, this wasn’t expensive for the approximately 1GiB/month of data we planned to store. We calculated that even if we sent all our expected jobs through Pub/Sub, we still wouldn’t be charged — of course, YMMV.</li><li>Ability to create infrastructure programmatically: We could create topics and subscriptions programmatically without having to set up any queues in advance. This makes deployments of new versions easier, as we planned on having a Pub/Sub topic for each version of the Transfer Service. We would then just need to deploy a new version of the service and it would create the Pub/Sub topic and subscription for its own version.</li><li>Ability to limit access to Pub/Sub resources: Setting up Service Accounts for the Transfer Service was fairly straightforward as we were able to create a credential key that could be passed to the correct environment (dev, test, production). Adding the Pub/Sub editor role allowed us to use and manage all topics and subscriptions.</li><li>Built-in disaster recovery: Pub/Sub will keep trying to deliver messages that have not been acknowledged, and we planned on using that to support disaster recovery. If the Transfer Service went down, we could just restart it, begin pulling messages, and start working on the same job again. If we ACK (acknowledge) messages immediately and crash, we would need to recover that message or lose the job.</li></ul><p>The Cons of Google Pub/Sub:</p><p>Of course, not everything was rosy and there were problems along the way. Here are some drawbacks and things to be aware of.</p><ul><li>Limit of 10K topics: Pub/Sub has a 10,000 topic limit per project. This might seem like more than enough, but early on we were discussing how to achieve fair consumption by having at least one topic per customer. We have <a href="https://www.xmatters.com/customers/">thousands of customers</a> and this would quickly exhaust the available topics for Pub/Sub, thereby making fair consumption impossible as “Fair queuing uses one queue per packet flow,” according to the fair consumption algorithm.</li><li>Pub/Sub SDK classes are harder to stub out: Many Google Cloud Pub/Sub classes are concrete and not interfaces, making them harder to stub out when writing unit or integration tests (for example, TopicAdminClient &amp; GrpcSubscriberStub).We could have wrapped each SDK class in our own wrapper, but it’s nicer not having to do that.</li><li>NACK (not acknowledged) semantics are unclear: Let’s say you want to discard the head message and retrieve the second in the queue. Unfortunately, the Google Cloud Pub/Sub documentation doesn’t shed much light on this scenario.</li><li>Multiple Transfer Services Can be Complicated: This may be a function of how we’re using Pub/Sub, but we wanted to run multiple Transfer Services subscribed to the same topic. Each Transfer Service would be getting the same unacknowledged message and would happily start working on the same import job. To overcome this, we implemented another check that would associate the current Transfer Service UUID with a job and only start working on available or orphaned jobs. An orphaned job has a Transfer Service UUID associated with it but is not being processed. For example, when a Transfer Service dies mid-job, it orphans the job it was working on.</li><li>Some messages cannot be retrieved: The Google Pub/Sub documentation states: “A message published before a given subscription was created will usually not be delivered. Thus, a message published to a topic with no subscription cannot be retrieved.” This means that we couldn‘t delete a subscription during a troubleshooting session. We also couldn’t <a href="https://www.xmatters.com/features/notifications/">publish a message</a> without a Transfer Service already subscribed to the topic. This can be fixed by having both the producer (for us, our API) and consumer (our Transfer Service) of the queue create a Pub/Sub subscription upon startup.</li><li>Ordering is not guaranteed: This is covered extensively in Google’s Pub/Sub documentation; but basically if ordering is a show-stopper, you can’t use Pub/Sub.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*zqRq2tG5KT_Cq2yX.png" /></figure><p><strong>Mostly a victory that bears watching<br></strong>Google’s Pub/Sub is a great, flexible, and managed service that allowed us to get up and running quickly with some code churning through jobs. Depending on your usage, the cost can be fairly reasonable and definitely beats maintaining your own queue infrastructure. We’re still not sure if we need to implement fair consumption as it depends on how production companies will use this service. If so, we may be forced to use a different solution. But until that happens, Pub/Sub supports our Transfer Service in production — with a drastic reduction in hairball sightings.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=baf1887b3302" width="1" height="1" alt=""><hr><p><a href="https://medium.com/true-tales-from-engineering/how-we-created-a-great-data-transfer-service-from-a-hairball-using-google-cloud-pub-sub-baf1887b3302">How We Created a Great Data Transfer Service from a Hairball Using Google Cloud Pub/Sub</a> was originally published in <a href="https://medium.com/true-tales-from-engineering">True Tales from Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ISO 27001 Certification | How to Achieve Certification — xMatters]]></title>
            <link>https://medium.com/true-tales-from-engineering/iso-27001-certification-how-to-achieve-certification-xmatters-8953d8cd2ea6?source=rss----f16fb9eb876c---4</link>
            <guid isPermaLink="false">https://medium.com/p/8953d8cd2ea6</guid>
            <category><![CDATA[iso-27001]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[information-assurance]]></category>
            <category><![CDATA[data-governance]]></category>
            <dc:creator><![CDATA[Karen Meohas]]></dc:creator>
            <pubDate>Tue, 15 Oct 2019 21:31:53 GMT</pubDate>
            <atom:updated>2019-10-15T21:31:53.048Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Achieve ISO 27001 Certification</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/0*aXHmqRZ_53SAawKg.jpg" /></figure><p><em>When I was a hired by xMatters I was given a very specific mission: ISO 27001 certification by March 2020. As an experienced project manager and knowing that the company already had a robust security framework in place, my obvious decision was to overcomplicate things.</em></p><p><em>Originally published at </em><a href="https://www.xmatters.com/"><em>https://www.xmatters.com</em></a><em> on September 19, 2019.</em></p><p>Just in case you don’t manage privacy and security for a living, <a href="https://www.iso.org/isoiec-27001-information-security.html">ISO/IEC 27001</a> provides requirements for an information security management system (ISMS). The ISO is a quality standard to optimize data governance and asset management, including personal identifiable information (pii), corporate security, intellectual property, employee data, and third-party vendors. An ISMS is the collection of policies, procedures, and controls for an organization’s information risk management processes.</p><p>Our customers have high standards for privacy and security, and we have always tried to exceed those standards. By doing so, we provide a better SaaS experience and give our customers a sense of confidence that their assets are safe. We also add value to our products and increase our competitive advantage. To have the best possible experience, our customers also need self-service access to important privacy and security information.</p><p>This is no small feat. It requires a careful balance between industry best practices, customer demands, and our strategic goals to create the Information Assurance Portfolio. The IA Portfolio includes two big programs: privacy and security. We chose to manage both projects simultaneously to save time and resources. This is a lot of work, so I suggest you write a project plan and be patient. Doing it well requires extreme attention to detail, and it takes time.</p><p>Here are a few suggestions for success, which you will read about below:</p><ul><li>Centralize management of processes and methods</li><li>Work with a subject matter expert</li><li>Obtain evidence of the results of your work that you can show to customers</li><li>Provide self-service access to information</li></ul><p><strong>Centralize management of processes and methods</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*_a1mkmRFJ4Ep06jDBgQlOA.jpeg" /><figcaption>Centralize management of processes and methods.</figcaption></figure><p>In case you thought privacy and security are the same thing, they are not. Privacy means doing enough to comply with legal standards for protecting identities, while security means keeping information safe and secure.</p><p>Managing many different projects that often share the same assets can get unwieldy and can lead to mistakes. So we centralized management of processes and methods for a more controlled environment and fewer errors. This is called Data Governance.</p><p>The xMatters security Program, to be finalized in March 2020, is composed of different projects with the following deliverables:</p><ul><li>ISO 27001:2013 certification</li><li>Semi-annual third party audits for Security Controls</li><li>SOC 2 Type 2 external verification program based on two principles: Security and Availability</li><li>Maintenance of EU-U.S. Privacy Shield Framework and the Swiss-U.S. Privacy</li><li>Shield for HR data and client data</li></ul><p><strong>Work with a subject matter expert</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*Vd5VAHB8F-gVO-UYCCdjTQ.jpeg" /><figcaption>Work with a subject matter expert.</figcaption></figure><p>The main objective of our privacy program is to demonstrate compliance with the most relevant privacy regulations. We hired a third-party subject matter expert to create a comprehensive privacy framework and conduct an eight-week in-depth audit of our documents, tools, website, and systems. The scope of this audit is in compliance with:</p><ul><li>General Data Protection Regulation — Europe (GDPR)</li><li>Personal Information Protection and Electronic Documents Act — Canada (Pipeda)</li><li>Australia Privacy Principles — Australia (APPs)</li><li>California Consumer Privacy Act — California, USA (CCPA)</li></ul><p>The idea is to grow the scope gradually and start the ISO 27701:2019 — Privacy Information Management Systems next year, as an extension to our ISO 27001 certification.</p><p><strong>Obtain evidence of the results of your work for customers</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*SfCtvpo3HFLqpbR9EuB8kw.jpeg" /><figcaption>Show evidence to your client.</figcaption></figure><p>When privacy verification is concluded, we will be provided an attestation letter from an independent Privacy Manager and organized information in the format of a Privacy Code of Conduct and Artifacts that map our compliance to each law.</p><p>This is beyond any requirements, and it gives our customers a sense of confidence.</p><p>A third program binds the other two together. It was named FAQ and our main objective is delivering more information internally and externally. Our Sales and Customer Success teams will have access to both privacy documents and an internal Security Q&amp;A portal that we are developing.</p><p><strong>Provide self-service access to information</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*gWA9ydvX7G78xSbs0c8zUQ.jpeg" /><figcaption>Provide self-service access to information.</figcaption></figure><p>We are also creating a public Privacy and Security page where clients will be able to find white papers, artifacts, and documents on ISO 27001 certification and other privacy and security topics.</p><p>This will speed up our ability to deliver more information and give clients self-service access. We will release white papers, articles, audit summaries, and certifications at planned intervals internally and in a new area on the xMatters website. This will give our customers the speed they need when they check on their privacy and security and show the transparency that confident, honest organizations do.</p><p>Do you have experience with ISO 27001 certification or other aspects of privacy or security governance? Let us know on social media. If you want a super secure incident management platform, <a href="https://www.xmatters.com/trial">Get xMatters Free</a> (forever).</p><p><a href="https://www.xmatters.com/trial">Create Your Free Account</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8953d8cd2ea6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/true-tales-from-engineering/iso-27001-certification-how-to-achieve-certification-xmatters-8953d8cd2ea6">ISO 27001 Certification | How to Achieve Certification — xMatters</a> was originally published in <a href="https://medium.com/true-tales-from-engineering">True Tales from Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open-Sourcing Cthulhu for Chaos Engineering]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/true-tales-from-engineering/open-sourcing-cthulhu-for-chaos-engineering-28963c1aa410?source=rss----f16fb9eb876c---4"><img src="https://cdn-images-1.medium.com/max/682/0*5xJ2PTnM8_fcLgTG.jpg" width="682"></a></p><p class="medium-feed-snippet">In a previous article, I wrote about Chaos Engineering, and how this approach to microservice design helps us at xMatters build software&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/true-tales-from-engineering/open-sourcing-cthulhu-for-chaos-engineering-28963c1aa410?source=rss----f16fb9eb876c---4">Continue reading on True Tales from Engineering »</a></p></div>]]></description>
            <link>https://medium.com/true-tales-from-engineering/open-sourcing-cthulhu-for-chaos-engineering-28963c1aa410?source=rss----f16fb9eb876c---4</link>
            <guid isPermaLink="false">https://medium.com/p/28963c1aa410</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[cthulhu]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <dc:creator><![CDATA[Gabrielle Gasse]]></dc:creator>
            <pubDate>Tue, 24 Sep 2019 21:34:38 GMT</pubDate>
            <atom:updated>2019-09-24T21:34:38.647Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How Hard Could It Be to Replace 22,000 Lines of Code?]]></title>
            <link>https://medium.com/true-tales-from-engineering/how-hard-could-it-be-to-replace-22-000-lines-of-code-637f21130131?source=rss----f16fb9eb876c---4</link>
            <guid isPermaLink="false">https://medium.com/p/637f21130131</guid>
            <category><![CDATA[swift]]></category>
            <category><![CDATA[ios-app-development]]></category>
            <category><![CDATA[objective-c]]></category>
            <dc:creator><![CDATA[James Leung]]></dc:creator>
            <pubDate>Tue, 10 Sep 2019 12:01:01 GMT</pubDate>
            <atom:updated>2019-09-10T12:01:01.335Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="How Hard Could It Be to Replace 22,000 Lines of Code?" src="https://cdn-images-1.medium.com/max/682/1*3q5-iqffyC8gpylqZ33mvw.jpeg" /></figure><p><em>When Apple introduced Swift, it promised a modern, strongly-typed, easy-to-read, safer and more secure language for iOS development. Emboldened by the promise of interoperability, we were excited to adopt Swift. So… how hard could it be to replace 22,000 lines of code written in Objective-C? We were about to find out. This blog originally appeared on </em><a href="https://www.xmatters.com/devops/blog-how-hard-could-it-be-to-replace-22000-lines-of-code/"><em>xMatters.com</em></a><em>.</em></p><figure><img alt="When Apple introduced Swift back in June 2014, most of us were excited to adopt it." src="https://cdn-images-1.medium.com/max/300/0*kfTLzKhChh41hRcM.jpg" /></figure><p>Most developers have heard the rule of thumb that at some point, the programming language becomes less important than the algorithm. That is, it’s best to limit religious debates about which language to use and focus instead on execution. I wholeheartedly agree, but sometimes you need to step back and evaluate whether the language you’re using is the right fit for the task. I doubt anyone relishes the idea of writing a mobile app in <a href="https://en.wikipedia.org/wiki/Assembly_language">Assembly</a>!</p><p>My team develops our <a href="https://apps.apple.com/ca/app/xmatters/id409162401">iOS app</a> for the <a href="https://www.xmatters.com/product/">xMatters product</a>, and when Apple introduced <a href="https://www.swift.org/">Swift</a> back in June 2014, most of us were excited to adopt it. That November, a couple of months after Swift was officially supported for iOS development, we created our first <a href="https://docs.swift.org/swift-book/LanguageGuide/ClassesAndStructures.html">Swift class</a>.</p><p><strong>How hard could it be to replace 22,000 lines of code?<br></strong>Adopting Swift was more than just chasing a shiny new object: it had been more than a year since we first released the xMatters native iOS app. From late 2012 to February 2013 when we released v2.0 of the app, the team coded in <a href="https://en.wikipedia.org/wiki/Objective-C">Objective-C</a>. However, the team never really loved that verbose-message-sending, weak-typed language.</p><p>In contrast, Swift promised a modern, strongly-typed, easy-to-read, safer and more secure language. Emboldened by the promise of interoperability, we marched forward with Swift. I mean, how hard could it be to replace 22,000 lines of Objective-C?</p><p><strong>Pretty hard, it turns out<br></strong>Fast forward about three years to November 2017, and the xMatters iOS app was less than 17% Swift: about 8,000 lines of code in the new language and — due to added functionality — 41,000 lines of Objective-C, excluding headers.</p><figure><img alt="We still had about 8,000 lines of code in Swift and 41,000 lines of Objective-C, excluding headers." src="https://cdn-images-1.medium.com/max/300/0*RD75XqmJh2Yquswy.jpg" /></figure><p>Interoperability was indeed possible, but the code change resulted in increased difficulty when debugging new class compilation issues — and the code took a turn for the worse in readability. Writing and reading blocks between the languages was a pain and it sometimes felt like we were getting the worst of both worlds.</p><p>At about this time, Apple started writing its iOS development documentation using Swift examples, and many popular third-party libraries were also written in Swift. Around the same time, we onboarded a new member to the iOS team who was very technical and skilled in iOS development but was unfamiliar with our code base. We took this as a sign, and accelerated transition to Swift (so to speak).</p><p><strong>Welcome, and good luck!<br></strong>We gave our new team member the task of migrating as much of our code base to Swift as possible during his first few months on the team. This might sound like hazing, but the idea was that he could learn our code base while reducing the burden of maintaining two languages. After some research, he proposed to migrate it in a very short timeframe in a huge focused effort.</p><p>He made his first commit toward a 100% Swift code base in January 2018, and a month later we had totally converted the code. Many things had to align for this to happen so quickly: the product owner’s trust in the team’s ability to handle quality of software in such a large migration; the audacity and technical capability of our new member to carry out a 100% migration in only three months; and, the team’s <a href="https://www.xmatters.com/use-cases/devops-sre">willingness to contribute their extra hours</a> to testing and fixing migration issues.</p><p><strong>Let the machine do (some of) the work<br></strong>The initial cut of the migration was done with an Objective-C-to-Swift converter tool called <a href="https://objectivec2swift.com/#/home">Swiftify</a> running in offline mode. The project would not have been possible without this tool. The few times we had to reach out to Swiftify for help, its CEO was quick to reply with helpful and insightful answers.</p><figure><img alt="Sometimes one person has to replace all those lines of code." src="https://cdn-images-1.medium.com/max/300/0*wyic_y_eSHYcfaIm.jpeg" /></figure><p>Sometimes one person has to replace all those lines of code.</p><p>We started — naïvely (!) — by running the conversion tool on the entire code base… which generated so many compilation errors that we had to break down the task into bite-size pieces. Lesson learned, our newbie doggedly divided and conquered the code base with 13 individual migrations, where each commit retained the ability for us to compile the project.</p><p>After those individual heroics came the team effort. Even though we could compile the app in the new Swift code base, we needed to ensure we didn’t break any functionality. Our initial round of functional testing generated 128 individual subtasks for the team to fix. Regular development had to continue, but the team took time away from their main tasks and worked together to solve the migration issues.</p><p><strong>Here be gotchas<br></strong>While the migration tool did migrate Objective-C code into Swift, in some cases the conversion wasn’t accurate. For those of you thinking of going down the same path, here are a few things to watch for:</p><ul><li>Optionals don’t translate well, as the conversion tool cannot determine our business logic. To deal with this, the team retraced the logic and fixed each instance of unwrapping.</li><li>Converted code that tested empty Strings and Arrays was incorrect. To get around this issue, the team made use ofguardandif let</li><li>There was converted code that tested equality withNSNull()when it should have been tested fornil</li><li>Clean up untyped conversion to[AnyHashable: Any!]for dictionaries</li><li>Clean up untyped elements for Arrays; for example,[Any]() versus [String]()</li><li>Fix conversions ofCalendar.componenttoCalendar.dateComponents</li></ul><p><strong>Fortune favors the Swift<br></strong>By March 2018, our investment in the 100% Swift migration started to pay off. With the new code base, our bug fixing became more efficient. Eventually we got to our first ‘zero’ bug count since we started coding the native iOS app. It was finally time for our first post-migration release of the app.</p><p>For us, the key was to stop trying to do a ‘piecemeal’ conversion when we had time, and instead to really concentrate on getting it done in a focused and efficient way. In the year since migration, we haven’t for a moment missed Objective-C’s umbrella file or interop blocks and their impact on our development timelines!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=637f21130131" width="1" height="1" alt=""><hr><p><a href="https://medium.com/true-tales-from-engineering/how-hard-could-it-be-to-replace-22-000-lines-of-code-637f21130131">How Hard Could It Be to Replace 22,000 Lines of Code?</a> was originally published in <a href="https://medium.com/true-tales-from-engineering">True Tales from Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Adopted Time-Based Monitoring with Prometheus]]></title>
            <link>https://medium.com/true-tales-from-engineering/how-we-adopted-time-based-monitoring-with-prometheus-8766fdc4383c?source=rss----f16fb9eb876c---4</link>
            <guid isPermaLink="false">https://medium.com/p/8766fdc4383c</guid>
            <category><![CDATA[alerting]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Curtis St. Pierre]]></dc:creator>
            <pubDate>Fri, 30 Aug 2019 12:01:01 GMT</pubDate>
            <atom:updated>2019-08-30T12:01:01.363Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*AgcQBGWFSo3OrSBf.jpg" /></figure><p><em>Sensu monitoring provides a simple up/down check system. When we switched to Prometheus, we got time-based monitoring and some incredible abilities. Of course, there were also a few pitfalls along the way.</em></p><p>This article originally appeared on <a href="https://www.xmatters.com/devops/blog-how-we-adopted-time-based-monitoring-with-prometheus/">xmatters.com</a>.</p><figure><img alt="Learn more about the xMLabs xMatters-Prometheus integration on GitHub." src="https://cdn-images-1.medium.com/max/253/0*a1nJNsiaeWwyb-J_.png" /><figcaption>Learn more about the xMLabs xMatters-Prometheus integration on GitHub.</figcaption></figure><p>Moving from our own data centers to <a href="https://www.cloud.google.com/">Google Cloud Platform</a> came with some amazing opportunities to optimize code, speed up deployment processes, downsize infrastructure, and even to evaluate new monitoring system options.</p><p>Our original systems were monitored using <a href="https://www.xmatters.com/integration/sensu/">Sensu</a>, an alerting system with a lot of bells and whistles around automation and extend-ability. But as far as monitoring is concerned, it’s a simple up/down check system. In contrast, <a href="https://www.xmatters.com/integration/prometheus/">Prometheus</a> represented a significant change — from a limited “trinary” signal to your ability to use monitoring data.</p><p>Sensu alerting is very simple and you have to write your own checks that return an exit code of 0, 1, or 2, depending on system health.</p><p>Prometheus allows you to write your own expressions with the data it has collected so you can check error rates over time, compare deltas, find the 99th percentile of metrics, and any number of things.</p><p>To be clear, you can use Prometheus to just replicate Sensu’s simple ‘healthy, error, warning’ style monitoring, but that doesn’t take advantage of its key benefit: time-based monitoring. Time-based monitoring is a very different paradigm from old-skool monitoring, as it gives you some incredible abilities such as alerting only if a condition is met for a specific period of time.</p><blockquote>“Lots of monitoring systems let you delay alerting. Prometheus lets you take a metric like request response speed and perform math on it.”</blockquote><p>I know what you’re thinking. “Curtis, lots of monitoring systems let you delay alerting.” Sure, but that’s not what I’m talking about here. Prometheus offers you the ability to take a metric like request response speed and perform math on it to only alert if, say, less than 99% of your requests are processed in 150ms. It also allows you to do other interesting things like monitoring the rate of increase. For example, if you usually have 10 requests per second, and suddenly the rate of requests skyrockets to 1,000 per second and stays like that for a long period of time.</p><figure><img alt="Image of tracking http requests in Prometheus" src="https://cdn-images-1.medium.com/max/1024/0*gFoa9aECM1nuCl_j.png" /><figcaption>Tracking http requests in Prometheus</figcaption></figure><p>Additionally, Prometheus allows you to tag the alert with information you can use to group alerts, services, and machines together. This is incredibly useful when you have high-availability services where you have multiple instances of the same service up; you may not care if one goes down, but you might get a bit ‘lively’ if more than 50% are down. Grouping them can be a helpful <a href="https://www.xmatters.com/features/it-event-management">IT event management</a> practice so you only get paged once for a whole series of alerts, as opposed to getting a page storm (or, as we lovingly call it, “phone melting”).</p><p>All that said, let’s go through some of the things you’ll want to pay attention to if you’re using Prometheus for your monitoring solution.</p><p><strong>Service discovery </strong><br>In our shop we use <a href="https://www.consul.io/">HashiCorp Consul</a> for our service discovery mechanism. One of the great things about Prometheus is that it has an out-of-the-box way to find your services using Consul by setting the consul_sd_config value in the configuration file. This allows you to monitor a service such as <a href="https://www.rabbitmq.com/">Rabbitmq</a> and use Consul tags to further limit your targets. This comes with a huge caveat: if your service disappears from Consul, Prometheus no longer knows about it.</p><p>Here’s an example of a problem we’ve run into with this. We run many of our services as three Kubernetes pods, and we want to run a check to ensure 2/3 of the cluster is up:</p><p>(avg(up{job=&quot;job&quot;}) BY (deployment, version, svcprom_env, branch, clusterid)) &lt; 0.67</p><p>If this comes back false, we want to send an xMatters alert to the team responsible for the service that is in an unhealthy state. The problem is if two of your pods disappear from Consul, then only one of your pods is getting scraped for metrics… and Prometheus is quite happy to carry on thinking everything is fine.</p><p>Now, you may think that since we’re running three pods, we could just validate that the up check returns &gt;2 instances. However, that stops us from being able to scale our services to, say, seven and expecting five to be up. We got around this by monitoring pod health through <a href="https://kubernetes.io/">Kubernetes</a> endpoints directly instead of relying on pods to tell us their health.</p><p><strong>Magic math </strong><br>We’ve already discussed the power of time-based monitoring, but there is a drawback. You remember all that math you had to learn in school you thought you’d never use because you’re just planning on writing code? Bad news: it’s back! And, this time there’s a compelling reason to bone up on it.</p><blockquote>“Remember all that math you had to learn in school you thought you’d never use because you’re just planning on writing code? It’s back!”</blockquote><p>Turns out in order to write meaningful time-based monitoring using services statistics, you need to know, well statistics. But more than that, you need to understand how those statistics relate to writing Prometheus alerting rules. We have some checks that use a histogram to figure out the speed of our notification resolutions. We want 99% of the notifications that go through one of our services to do so in under 125ms.</p><p>This means that after you figure out how to bucket your checks properly, you need to figure out how it works in the Prometheus expressions. Here’s an example from the Prometheus documentation:</p><p>histogram_quantile(0.95,sum(rate(http_request_duration_seconds_bucket[5m])) by (le))</p><p>So, this takes the sum of the rates of HTTP request durations over five minutes and returns the line that the 95th percentile of http_request_duration_seconds falls under.</p><p>Make sense? Perfect! You’ll find that summing a rate, delta, or some of the other functions Prometheus provides is very common. It’s the only way to do math while also being able to group by your (hopefully) incredibly useful labels.</p><p><strong>Alerting and too many metrics</strong><br>Let’s see now — what other weird things did we find? Well, here’s one: say you had something alerting, so you solved the problem (or so you thought), and then it falls over again. Would you expect Prometheus to page you again? We did, but that’s apparently not the case. This is because when you have it set to alert only one time per interval, it won’t alert you if it fails again in that interval. I guess that’s not crazy, but we didn’t expect it. You’re welcome.</p><p>We use Rabbitmq for multiple systems at xMatters and we thought, “Hey, there’s a Prometheus plugin for Rabbitmq — that’s great!” and we sort of got monkey pawed on that gift. It turned out that Rabbitmq’s exporter has so many metrics that you can crush your service under its own alerting weight.</p><blockquote>“Turns out in order to write meaningful time-based monitoring using services statistics, you need to know statistics. But more than that, you need to understand how those statistics relate to writing Prometheus alerting rules.”</blockquote><p>On the upside, Rabbitmq does give you the ability to limit the metrics you generate; the downside is that it’s not incredibly granular, and if you have a lot of queues you’re in for a bad time.</p><p>In this situation you also must be prepared for Prometheus svcprom to be more heavily loaded — which means you need to give it more resources — and you need to prepare your pod health checks for a possibly slow startup/render time. If not, expect crash loops to be part of your life (and your nightmares).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/0*gzKlM86AK3s_5v64.jpg" /><figcaption>Prometheus gave humans fire before he gave us time-based monitoring.</figcaption></figure><p><strong>Avoiding Those Hotspots </strong><br>Of course, you know that Prometheus was a mischievous Titan who stole fire from the gods and gave it to humans. Who wouldn’t want to have a drink with that guy? But you might want to know in advance what you are getting yourself into (and maybe book a separate ride home). Likewise, I hope I’ve given you a sense about some of the joys and pitfalls of implementing the Prometheus monitoring system so that you’re successful and don’t get, uh, burned (sorry, had to be done).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8766fdc4383c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/true-tales-from-engineering/how-we-adopted-time-based-monitoring-with-prometheus-8766fdc4383c">How We Adopted Time-Based Monitoring with Prometheus</a> was originally published in <a href="https://medium.com/true-tales-from-engineering">True Tales from Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Executing end-to-end tests in Kubernetes]]></title>
            <link>https://medium.com/true-tales-from-engineering/executing-end-to-end-tests-in-kubernetes-bb7742da138?source=rss----f16fb9eb876c---4</link>
            <guid isPermaLink="false">https://medium.com/p/bb7742da138</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[jaxenter]]></category>
            <category><![CDATA[gradle]]></category>
            <category><![CDATA[containerization]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Deepa Guna]]></dc:creator>
            <pubDate>Thu, 15 Aug 2019 12:48:45 GMT</pubDate>
            <atom:updated>2019-08-15T12:48:45.673Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Executing end-to-end tests in Kubernetes" src="https://cdn-images-1.medium.com/max/682/1*wB6lACZtihsjF5p0GzqWHw.jpeg" /></figure><p><em>Originally published at </em><a href="https://jaxenter.com/"><em>https://jaxenter.com</em></a><em> on March 19, 2019. Also published on </em><a href="https://www.xmatters.com/devops/blog-executing-end-to-end-tests-in-kubernetes/"><em>xMatters.com</em></a><em>.</em></p><p><em>Our infrastructure is more complex than ever and there is greater pressure to deliver quality features to customers on time. To meet these needs, automated end-to-end tests play an important role in our continuous integration and delivery process. Let’s look at how we can execute these tests in a container within a Kubernetes cluster on Google Cloud Kubernetes Engine.</em></p><p>As software applications transition towards a microservice architecture and platforms become more <a href="https://container-solutions.com/what-is-cloud-native/">cloud native</a>, these shifts have changed how development teams build and test software. To scale and manage containers, organizations are turning to orchestration platforms like <a href="https://kubernetes.io/">Kubernetes</a> to automate deployment, scaling, and management of containerized applications at scale, whether they run in a private, public, or hybrid cloud.</p><p>With increased complexity in the infrastructure and the need for timely delivery of quality features to customers, <a href="https://screenster.io/end-to-end-testing/">automated end-to-end testing</a> plays an important role in the continuous integration and delivery process. Let’s look at how we can execute these tests in a container within a Kubernetes cluster on <a href="https://cloud.google.com/kubernetes-engine/">Google Cloud Kubernetes Engine</a>.</p><h3>Building the Testing Container Image</h3><p>We start by building our tests, which are written in using <a href="https://www.seleniumhq.org/projects/webdriver/">Selenium WebDriver</a>, into a container image. The image includes all the test files, libraries, drivers, and a properties file, as well as the shell script to start the tests.</p><p>Below you’ll find some sample code that should give you a sense of how you can structure and configure your testing, including snippets from the following files:</p><h3>Dockerfile</h3><figure><img alt="Building a Dockerfile container image" src="https://cdn-images-1.medium.com/max/400/1*kezht5Uf4U2dqQg-9AbMag.jpeg" /></figure><pre>FROM centos:7.3.1611<br>RUN yum install -y \<br>    java-1.8.0-openjdk \<br>    java-1.8.0-openjdk-devel<br>ENV JAVA_HOME /usr/lib/jvm/java-1.8.0-openjdk/</pre><pre># Set local to UTF8<br>RUN localedef -i en_US -f UTF-8 en_US.UTF-8<br>ENV LANG en_US.UTF-8<br>ENV LANGUAGE en_US.UTF-8<br>ENV LC_ALL en_US.UTF-8</pre><pre># Install automation tests<br>COPY ./build/libs ${test_dir}/bin<br>COPY ./build/lib ${test_dir}/lib<br>COPY ./testfiles ${test_dir}/testfiles<br>COPY ./build/resources ${test_dir}/resources<br>COPY ./build/drivers ${test_dir}/drivers<br>RUN chmod -R 755 ${test_dir}/drivers<br>COPY ./*.properties ${test_dir}/</pre><pre>WORKDIR /opt/automation<br>USER root</pre><pre>CMD [ &quot;./run-suite.sh&quot;, &quot;TestSuite&quot; ]</pre><h3>run-suite.sh:</h3><figure><img alt="Building a run-suite.sh container image" src="https://cdn-images-1.medium.com/max/800/1*9BT-AefGcG87wgXCAsESWA.jpeg" /></figure><pre>#!/bin/bash</pre><pre># run test suite<br>fullList=&quot;&quot;<br>function listOfSuites() {<br>    for i in $(echo $1 | sed &quot;s/,/ /g&quot;)<br>    do<br>      # call your procedure/other scripts here below<br>      echo &quot;$i&quot;<br>      fullPath=`find . -type f -name &quot;$i.xml&quot;`<br>      #echo &quot;fullList=$fullList&quot;<br>      #echo &quot;fullPath=$fullPath&quot;<br>      fullList=&quot;$fullList$fullPath &quot;<br>    done<br>}<br>listOfSuites $1<br>echo $fullList</pre><pre>java -Dlog4j.configuration=resources/main/log4j.properties -cp<br>&quot;./lib/*:./bin/*:.&quot; \<br>org.testng.TestNG $fullList</pre><pre>java -cp &quot;./lib/*:./bin/*:.&quot; com.gcp.UploadTestData</pre><h3>build.gradle</h3><figure><img alt="Building a Gradle container image" src="https://cdn-images-1.medium.com/max/270/1*yBaLU7Tp9xRkaW8U9_UteQ.png" /></figure><pre>buildscript {<br>    repositories { maven { url &quot;${nexus}&quot; } }<br>    dependencies {<br>      classpath &#39;com.bmuschko:gradle-docker-plugin:3.6.2&#39;<br>    }<br>}</pre><pre>apply plugin: com.bmuschko.gradle.docker.DockerRemoteApiPlugin</pre><pre>import com.bmuschko.gradle.docker.tasks.image.DockerBuildImage</pre><pre>def gcpLocation = &#39;gcr.io&#39;<br>def gcpProject = &#39;automation&#39;<br>def dockerRegistryAndProject = &quot;${gcpLocation}/${gcpProject}&quot;</pre><p>Provide a lazily resolved $project.version to get the execution time value, which may include -SNAPSHOT.</p><pre>def projectVersionRuntime = &quot;${-&gt; project.version}-${buildNumber}&quot;<br>def projectVersionConfigurationTime = &quot;${project.version}-${buildNumber}&quot;<br>def projectRpm = &quot;${projectName}-${projectVersionConfigurationTime}.${arch}.rpm&quot;</pre><p>We want to use the branch name as part of the GCR tag. However we don’t want the raw branch name, so we strip out symbols and non-alpha-numerics. We also strip out git branch text that contains remotes/origin or origin/, since we don’t care about that.</p><pre>def sanitize = { input -&gt;<br>    return input.replaceAll(&quot;[^A-Za-z0-9.]&quot;, &quot;_&quot;).toLowerCase().replaceAll(&quot;remotes_origin_&quot;, &quot;&quot;).replaceAll(&quot;origin_&quot;, &quot;&quot;);<br>}</pre><pre>def gitbranchNameRev = &#39;git name-rev --name-only HEAD&#39;.execute().text.trim()<br>def gcpGitbranch = System.env.GIT_BRANCH ?:(project.hasProperty(&#39;gitbranch&#39;)) ? &quot;${gitbranch}&quot; : &quot;${gitbranchNameRev}&quot;<br>def gitbranchTag = sanitize(gcpGitbranch)</pre><pre>def projectVersionRuntimeTag = sanitize(&quot;${-&gt; project.version}&quot;)<br>def dockerTag = &quot;${dockerRegistryAndProject}/${projectName}:${projectVersionRuntimeTag}-${buildNumber}-${gitbranchTag}-${githash}&quot;</pre><pre>def buildType = System.env.BUILD_NUMBER ? &quot;JENKINS&quot; : &quot;LOCAL&quot;</pre><p>Create gcpBuildVersion.properties file containing build information. This is for the build environment to pass onto other upstream callers that are unable to figure out this information on their own.</p><pre>task versionProp() {<br>    onlyIf { true }<br>    doLast {<br>      new File(&quot;$project.buildDir/gcpBuildVersion.properties&quot;).text = &quot;&quot;&quot;APPLICATION=${projectName}<br>VERSION=${-&gt; project.version}<br>BUILD=${buildNumber}<br>BRANCH=${gcpGitbranch}<br>GIT_HASH=${githash}<br>TAG_FULL=${dockerTag} TAG=${projectVersionRuntimeTag}-${buildNumber}-${gitbranchTag}-${githash}<br>TIMESTAMP=${new Date().format(&#39;yyyy-MM-dd HH:mm:ss&#39;)} BUILD_TYPE=${buildType}</pre><pre>&quot;&quot;&quot;<br>    }<br>}</pre><p>Make sure the below version file generation is always run after build:</p><pre>build.finalizedBy versionProp<br>task dockerPrune(type: Exec) {<br>    description &#39;Run docker system prune --force&#39;<br>    group &#39;Docker&#39;</pre><pre>    commandLine &#39;docker&#39;, &#39;system&#39;, &#39;prune&#39;, &#39;--force&#39;<br>}</pre><pre>task buildDockerImage(type: DockerBuildImage) {<br>    description &#39;Build docker image locally&#39;<br>    group &#39;Docker&#39;<br>    dependsOn buildRpm<br>    inputDir project.buildDir</pre><pre>    buildArgs = [<br>      &#39;rpm&#39;: &quot;${projectRpm}&quot;,<br>      &#39;version&#39;: &quot;${projectVersionConfigurationTime}&quot;<br>    ]</pre><pre>    doFirst {</pre><p>Copy the Dockerfile to the build directory so we can limit the context provided to the Docker daemon.</p><pre>    copy {<br>      from &#39;Dockerfile&#39;<br>      into &quot;${project.buildDir}&quot;<br>    }<br>    copy {<br>      from &#39;docker&#39;<br>      into &quot;${project.buildDir}/docker&quot;<br>    include &quot;**/*jar&quot;<br>    }</pre><pre>    println &quot;Using the following build args: ${buildArgs}&quot;</pre><p>This block will get the execution time value of $project.version, which may include -SNAPSHOT.</p><pre>      tag &quot;${dockerTag}&quot;<br>    }<br>}</pre><pre>task publishContainerGcp(type: Exec) {<br>    description &#39;Publish docker image to GCP container registry&#39;<br>    group &#39;Google Cloud Platform&#39;<br>    dependsOn buildDockerImage</pre><pre>    commandLine &#39;docker&#39;, &#39;push&#39;, &quot;${dockerTag}&quot;<br>}</pre><h3>Selenium Grid Infrastructure Setup</h3><p>Our end-to-end tests use Selenium WebDriver to execute browser-related tests, and we have a scalable container-based Selenium grid deployed in a Kubernetes cluster (you can see <a href="https://opensource.zalando.com/zalenium/#kubernetes">setup details here</a>). You can configure the grid URL and browser in the test_common.properties file included in the test container image:</p><pre>targetUrl=http://www.mywebsite.com<br># webDriver settings webdriver_gridURL=http://${SELENIUM_GRID}/wd/hub webdriver_browserType=${BROWSER_TYPE}</pre><h3>Deploying Test Containers in Kubernetes Cluster</h3><p>Now that we have the container image built and pushed to the Google Cloud Platform, let’s deploy the container in Kubernetes. Here’s a snippet of the template for the manifest file to execute tests as a Kubernetes job:</p><pre>apiVersion: batch/v1<br>kind: Job<br>metadata:</pre><pre># Unique key of the Job instance<br>    name: run-suite-${JENKINS_JOB_INFO}-${TEST_SUITE_LOWER}<br>    labels:<br>      jobgroup: runtest<br>spec:<br>    template:<br>      metadata:<br>        name: runtest<br>        namespace : automation<br>        labels:<br>          jobgroup: runtest<br>      spec:<br>        containers:<br>        - name: testcontainer<br>          image: gcr.io/automation/${IMAGE_NAME}:${IMAGE_TAG}<br>          command: [&quot;./run-suite.sh&quot;, &quot;${TEST_SUITE}&quot;]<br>          env:<br>          - name: SELENIUM_GRID<br>            value: &quot;${SELENIUM_GRID}&quot;<br>          - name: BROWSER_TYPE<br>            value: &quot;${BROWSER_TYPE}&quot;<br>          - name: TARGET_URL<br>             value: &quot;${TARGET_URL}&quot;<br>          - name: GCP_CREDENTIALS<br>             value : &quot;${GCP_CREDENTIALS}&quot;<br>          - name: GCP_BUCKET_NAME<br>             value: &quot;${GCP_BUCKET_NAME}&quot;<br>          - name : BUCKET_FOLDER<br>             value : &quot;${BUCKET-FOLDER}&quot;<br>        restartPolicy: Never</pre><p>And, here’s the command to create the job:</p><p>kubectl apply -f ./manifest.yaml --namespace automation</p><h3>Publishing Test Results</h3><p>Once test execution is complete, you can upload test results to your Google Cloud storage bucket using a code snippet similar to this:</p><pre>public static void main(String... args) throws Exception {</pre><pre>    String gcp_credentials = readEnvVariable(GCP_CREDENTIALS);<br>    String gcp_bucket = readEnvVariable(GCP_BUCKET_NAME);<br>    String bucket_folder_name = readEnvVariable(BUCKET_FOLDER);</pre><pre>    // authentication on gcloud<br>    authExplicit(gcp_credentials);<br>    <br>    // define source folder and destination folder<br>    String source_folder = String.format(&quot;%s/%s&quot;,<br>System.getProperty(&quot;user.dir&quot;), TEST_RESULTS_FOLDER_NAME);<br>    String destination_folder = &quot;&quot;;<br>    if (!bucket_folder_name.isEmpty()) {<br>      destination_folder = bucket_folder_name;<br>    } else {<br>      String timeStamp = new    SimpleDateFormat(&quot;yyyyMMdd_HHmmss&quot;).format(Calendar.getInstance().getTime());</pre><pre>destination_folder = TEST_RESULTS_FOLDER_NAME + &quot;-&quot; + timeStamp;<br>System.out.println(&quot;destination folder: &quot; + destination_folder);<br>}<br>    // get gcp bucket for automation-data<br>    Bucket myBucket = null;<br>    if (!gcp_bucket.isEmpty()) {<br>      myBucket = storage.get(gcp_bucket);<br>    }<br>    // upload files<br>    List&lt;File&gt; files = new ArrayList&lt;File&gt;();<br>    GcpBucket qaBucket = new GcpBucket(myBucket, TEST_RESULTS_FOLDER_NAME);<br>    if (qaBucket.exists()) {       qaBucket.createBlobFromDirectory(destination_folder, source_folder, files);<br>      System.out.println(files.size() + &quot;files are uploaded to &quot; + destination_folder);<br>      }<br>    }<br>}</pre><h3>Next Steps</h3><p>We’re looking into publishing test results in a centralized result database fronted by an API service. This allows users to easily post test results data for test result monitoring and analytics, which I will cover in a future post about building a centralized test results dashboard. Until then, I hope this has helped you put all the pieces together for executing automated end-to-end testing using Kubernetes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb7742da138" width="1" height="1" alt=""><hr><p><a href="https://medium.com/true-tales-from-engineering/executing-end-to-end-tests-in-kubernetes-bb7742da138">Executing end-to-end tests in Kubernetes</a> was originally published in <a href="https://medium.com/true-tales-from-engineering">True Tales from Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>