<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by victor raton on Medium]]></title>
        <description><![CDATA[Stories by victor raton on Medium]]></description>
        <link>https://medium.com/@vraton81?source=rss-d174f40c5a40------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*igYP7KwlA4csx_N8FxiHxg.jpeg</url>
            <title>Stories by victor raton on Medium</title>
            <link>https://medium.com/@vraton81?source=rss-d174f40c5a40------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:45:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vraton81/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Chaos Engineering: How simulating adversity can help build eCommerce Resilience]]></title>
            <link>https://medium.com/adidoescode/chaos-engineering-how-simulating-adversity-can-help-build-ecommerce-resilience-4a799c8912dc?source=rss-d174f40c5a40------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a799c8912dc</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[resilience]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <dc:creator><![CDATA[victor raton]]></dc:creator>
            <pubDate>Thu, 03 Mar 2022 09:02:56 GMT</pubDate>
            <atom:updated>2022-03-03T09:02:56.380Z</atom:updated>
            <content:encoded><![CDATA[<p>“Champions aren’t made in the ring; they are merely recognized there” — Joe Frazier</p><p>A look at the lives of most sporting legends, tells us one thing. Legends are not forged in playing fields but on training grounds. What separates the mortals from legends like Leo Messi or Zidane, are the trials and tribulations they put their bodies and minds through so that they are ready and raring for whatever adversities they may face during the competition. While this sports metaphor is fitting for me, it is true in every aspect of life. Captains of ships, astronauts and firefighters will tell you that only extreme rigour and turbulences can prepare them to brave the ever-changing, ever-challenging conditions of their jobs. They must undergo innumerable practice sessions, simulations, drills of varying degrees so that they are prepared to handle adversity. Without testing their equipment for resilience, they cannot be ready to tackle situations in the field</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hO5K4dbaU5h3dHHGIMkOig.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@brett_jordan?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Brett Jordan</a> on <a href="https://unsplash.com/s/photos/chaos?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>With more and more consumers changing their shopping preferences, eCommerce business is scaling heights year on year. Every peak season, online sales shatter all previous records. Catering to these growing needs means our eCommerce systems get more and more complex every year. Additionally, the need for these systems to be robust and resilient is more important than ever. Disruptions are extremely expensive, not only in terms of revenue loss but also in terms of brand value.</p><p>Extending the earlier metaphor to the context of eCommerce makes Operations and Site Reliability Engineers captains and firefighters in their own right. How does one build resilience in a system? The simple answer- find faults and fix them. However, given the complex nature of systems today and the ever-changing environment in which they operate, the exercise is easier said than done. Unless we make systems go through turbulence, they may never reveal possible weaknesses and points of failure. Enter Chaos Engineering.</p><p><strong>What is Chaos Engineering?</strong></p><p>The most popular definition that exists is based on a statement from Netflix who were one of the earliest pioneers of this concept and it is as follows: “The discipline of performing proactive experimentation on a distributed system to build confidence in the system’s capability to withstand turbulent and malicious conditions.” The objective behind chaos engineering is basically to test the resilience of systems, people, infrastructure, etc. to ensure that the system can react to an outage and continue to provide an excellent customer experience.</p><p>Introducing Chaos engineering requires us to take care of the following building blocks:</p><p><strong>1.</strong> <strong>People, Practice and Process</strong>:</p><p>Chaos Engineering is not limited to systems alone. When you discuss the concept, you receive comments like “we don’t need to break things randomly in production” or “we are so far away from executing chaos engineering”. Therefore, it is important to break these myths and clarify to the masses what chaos engineering is and what it is not. We have to make people understand that chaos engineering is a practice that could be very useful in the early stages of the creation of applications. Running chaos in production requires a lot of time and practice and there is nothing random behind it. Firstly, it requires developing a “chaos mindset” in SRE professionals, which makes it a cultural change as well. For this purpose, organizations need to establish the right foundations by organizing workshops to propagate awareness and instill the ethos of Chaos Engineering into the minds of their people and find ways to incorporate the philosophy in their processes. We should be able to provide clear guidelines to the product teams about how to identify targets to execute chaos (architectural changes, P1, historical data..), clear technical documentation about how to use the chaos engineering tool and templates to follow to execute experiments, how to document the experiment outcomes in RCA and next steps.</p><p><strong>2.</strong> <strong>Infrastructure, Platform and Applications:</strong></p><p>Once the non-tech foundations are laid, it is important to ensure adequate infrastructure is available to practice Chaos Engineering. This calls for better capacity planning and the ability to scale. Additionally, Chaos Engineering also requires adequate platforms and applications to run effectively. For the chaos engineering execution, the identification of a steady state is one of the key points. This needs the support of an observability platform that provides a lot of information of the status of the applications based on metrics, logs, and traces.</p><p>For the execution of experiments, it is vital to use tools that can create unexpected behaviors like packets loss or network delays between services or between services and third-party solutions. Other examples are issues in infrastructure like Kubernetes pods destroyed or processes killing. In the beginning, I was surprised to see that there are a large number of chaos engineering tools like Gremlin, LitmusChaos, Chaos Toolkit, Steadybit etc. In addition, I also learned that it was possible to inject failure in applications with service mess solutions like Linkerd. The decision to select the right tool may not be easy, because we need to consider different topics like security, operational model, attacks scope, GitOps, community and documentation. But it is an important step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*wwuMHwfqz8rwo9vsfNzvKg.png" /><figcaption><strong>Figure 1: Building Blocks of Chaos Engineering</strong></figcaption></figure><p>With the fundamentals in place, let us examine the “how” aspects of Chaos Engineering by looking at the key processes by which a state-of-the-art Chaos Engineering practice can be put in place.</p><ol><li><strong>Whiteboarding and Target Identification:</strong> As the name suggests, this involves determining the potential parts of a system that can be methodically targeted. Organizations may decide to start to work with one team, wherein, with the support of the value stream champions, they can identify and create the dependency map of the services involved on the value stream, identify the soft and hard dependencies. This would provide a whole view of the value stream flow architecture and a clear view of the dependencies between services, platform and third-party solutions which would help to identify possible points of failure. The next step would be to analyze the historical data of the incidents that have happened. Finally, it would also be very important to define what is the steady-state of a system.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*FJkwH_e5iFo-HgtKoE1Hvw.png" /><figcaption><strong>Figure 2: Identification of targets</strong></figcaption></figure><p><strong>2.</strong> <strong>Creating Chaos Experiments/Scenarios:</strong> Once we have the target in place, the next step is to develop experiments that attack the identified target. An experiment is basically a hypothesis. For example, what would happen if there were a packet loss between the payments service provider and the checkout service. We should provide an expected outcome to this outage that could be a failure, or if the application was designed thinking of failures, the checkout service could provide an alternative solution to continue with this process for the end-user. If the experiment goes well, perfect! We have proved that our application is resilient to these kinds of issues. If an experiment fails, perfect again! We have identified an issue before the end user, and we will be able to create a backlog to fix this issue as soon as possible. Each experiment must have a backup and abort plan in place. The term “blast radius” is an important consideration here, as it is important to control these experiments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/1*MmdaCojbp3nFB_mWdq8mow.png" /><figcaption><strong>Figure 3: Steps to execute experiments</strong></figcaption></figure><p><strong>3.</strong> <strong>Feasibility Study:</strong> This essentially includes the selection criteria to determine which experiments can be used for what kind of targets. During the whiteboard session, organizations typically identify different possible issues to be used as a target.</p><p>a. <strong>Prior 1 incident</strong></p><p>b. <strong>Top 10 incidents</strong></p><p>c. <strong>Business KPIs</strong></p><p>d. <strong>Architectural changes</strong></p><p><strong>4.</strong> <strong>Dry Run: </strong>As the name suggests, it is a trial run to make sure that the experiment is running as expected.</p><p><strong>5.</strong> <strong>Gameday:</strong> Possibly the most important part of the whole process. It is a day dedicated for different teams to apply the different chaos engineering experiments. It is a game since the activity is happening in a test system and not in the actual production environment which creates a no-pressure scenario for the different teams involved. An outage is created, and the teams participate to ensure that the system continues to provide an excellent customer experience. The objective of the gameday is to build confidence in people, processes, and systems. The results of the gameday are logged and used to further improve the system. Typically, organizations execute a few of them, overall to engage the people into the chaos engineering practices, to gain confidence during the investigation and resolution of issues and challenge the process itself (people, process and practices). Once people are feeling more confident with these practices (together with workshops), then we can change the scope of the gameday, focus on small teams and dedicate ourselves to testing and improving the resilience of the applications.</p><p><strong>6.</strong> <strong>Reflect:</strong> The gameday backlog is reviewed and the findings obtained are used to plan for the next improvement activity. SRE will be able to create a backlog and based on the outcomes of the gameday be able to influence code changes or architectural redesign to improve the resilience of the system.</p><p>Chaos Engineering is a valuable tool to obtain an in-depth understating of one’s distributed system. It opens our eyes to hidden behaviors, improves observability by giving us a more definite set of KPIs to observe and predict failure. Not only does it help in reducing anomalies in the system, but the philosophy helps teams detect and resolve issues much faster. Chaos Engineering when done right has a significant impact on a whole range of KPIs such as mean time to detect (MTTD), mean time to resolve (MTTR), mean time between failures (MTBF), avoided incidents ratio, etc. While the words Chaos and Stability are antonyms in the field of literature, engineered chaos is a valuable ally to stability and resilience.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a799c8912dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/adidoescode/chaos-engineering-how-simulating-adversity-can-help-build-ecommerce-resilience-4a799c8912dc">Chaos Engineering: How simulating adversity can help build eCommerce Resilience</a> was originally published in <a href="https://medium.com/adidoescode">adidoescode</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>