<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Muhammad Soliman on Medium]]></title>
        <description><![CDATA[Stories by Muhammad Soliman on Medium]]></description>
        <link>https://medium.com/@muhammad-soliman?source=rss-c336f0f9e471------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3RzCAqZG4Oc0N1GGHQrxjQ.jpeg</url>
            <title>Stories by Muhammad Soliman on Medium</title>
            <link>https://medium.com/@muhammad-soliman?source=rss-c336f0f9e471------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 14:03:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@muhammad-soliman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fault injection]]></title>
            <link>https://muhammad-soliman.medium.com/fault-injection-bde62176aa4a?source=rss-c336f0f9e471------2</link>
            <guid isPermaLink="false">https://medium.com/p/bde62176aa4a</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <dc:creator><![CDATA[Muhammad Soliman]]></dc:creator>
            <pubDate>Thu, 20 Feb 2025 21:29:57 GMT</pubDate>
            <atom:updated>2025-02-20T21:34:47.073Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*flltvXr1d973CGrU4k-Oww.jpeg" /></figure><p>Fault injection is the practice of injecting artificial faults or failures in the system and watching how the system behaves. The core premise behind building a reliable distributed system is that it has a level of fault tolerance. The promised fault tolerance level varies from one system to the other, and varies with types of failures the system promises to tolerate.</p><p>The types of failures to inject are infinite and can not be all covered in one article, so I am going to talk about my preferred type of fault injection: network partitioning, and give examples on how to achieve it.</p><h3>Why network partitioning</h3><p>First reason has to do with a core limitation of distributed systems defined by the CAP theorem, which states that in any distributed system you can only achieve at most only two of the three: strong consistency, Availability and partition tolerance.</p><p>Network portioning happens when the network becomes two or more disconnected parts, where nodes in a given part are able to communicate with each other, but not with nodes in other parts. Network partitioning is inevitable, it happens a lot.</p><p>Most systems are CP or AP systems: consistent and partition tolerant or Available and partition tolerant. Partition intolerance, or a CA system, means that when a network partitioning happens the whole system stops working. Because implementing CA must include distributed synchronous transactions which are almost impossible to implement to the full theoretical correctness<strong>, </strong>which in turn means that in a CA system chances for a split brain exist<strong>, many CA systems do not continue functioning after the portioning ends by themselves</strong>, once a partitioning happens the system stops working forever until an external intervention happens: human intervention or an automated mechanism for resolving conflicts.</p><p>This is why it is rare to find CA systems, and this is why my post is focusing on CP or AP systems.</p><p>The second reason has to do with the second most important theorem in distributed systems after CAP theorem: the FLP impossibility, which states that in any distributed system where a node can crash and we cannot put a cap on response time, it is impossible to have consensus.</p><p>The idea is simple: if I send a request to a node on the network, how long should I wait before I assume the node is dead? what If I decide that after waiting x seconds that the node is dead but turns out that the node is not dead but just busy?</p><p>And If I send a request and don’t get a response, should I assume that my request didn’t reach the destination or should I assume that it reached the destination, was executed, and I just didn’t get the response? In other words did the network outage happen before or after I sent my request?</p><p>And what happens if I assume that a node in the system is dead, because it didn’t respond for a long time, and then it turns to be a live and joins the system again after the network disruption that caused it to be non responsive ended? will the system be in a sort of split brain situation? will this corrupt data?</p><p>What FLP impossibility tells us is that in reality, all distributed systems work on an assumptions. We hope that our assumptions are true and they are true most of the time.</p><p>Network portioning tests those assumptions.</p><p>In a network portioning the system becomes split, each node in a partition makes assumptions about nodes in the other partition. Those assumptions could be that they crashed while in reality they are not, they are alive, or that the write request I sent didn’t reach its destination while in fact it did and the sender just missed the confirmation response, etc.</p><p>This is why network partitioning is a more serious test than shutting down some nodes for example. Shutting down a node does not leave any space for wrong assumptions. In theoretical distributed systems all failures are classified into two major categories: crash failures and byzantine failures. Crash failures, like shutting down a node, are the ones that are easy to reason about and tolerate, byzantine failures are the nasty ones. Timeouts are byzantine failures.</p><h3>Fault injection and engineering teams dynamics</h3><blockquote>Everybody knows that you need more prevention than treatment, but few reward acts of prevention. Nassim Nicholas Taleb — The black swan.</blockquote><p>Rewarding treatment more than prevention is a human nature. In his book <strong>The Black swan</strong>, Nassim Nicholas Taleb gives various examples from different aspect of life, like rewarding the central banker who avoids a recession less than the one who “corrects” what his predecessor has done.</p><p>Site reliability engineers, and those who work on distributed systems reliability whatever title they hold, are good examples of this human nature. Usually, the more their job is well done the more difficult it is to show. If a system doesn’t fail much it is difficult to credit this to the SRE’s design, implementation and operations vs luck that caused no problems to happen or any other similar factor. However, if the system fails and the engeineer intervenes to save it and end a costly outage they get celebrated more easily.</p><p>When Booking.com changed technical leadership, the new leaders came with many new ideas and mandates, one of those mandates that resonated with me and still does is “No more heroes”. Instead of celebrating those who jump into outages across the whole engineering org, systems should be reliable, have clear ownership and clear expectations on interventions when incidents happen, and measurable reliability should be more celebrated than heroic saving of failed systems.</p><p>However from my experience I would say that this is attitude is the exception not the norm.</p><p>Fault injection lies in the golden middle ground between prevention and treatment. When you do a fault injection exercise and systems react to it with some of them failing and some not, failures become things that need treatment with all of what this word entails: they are objectively clear to everyone in terms of severity, impact and in turn priority of fixing.</p><p>Back in the days when I worked in booking.com I had a discussion with a colleague about a change I suggested that should increase the overall reliability of the system our team owned, but my colleague thought it was too complicated. One thing that helped us resolve the disagreement about such subjective matter was that the system we owned fired alerts in the last company wide fault injection exercise, so we have to implement a fix and report this fix to management. My suggested change would fix this issue, it was a <strong>treatment </strong>to a problem that happened in real world not one that we theorize could happen or not. <br>In the next quarterly review my colleague was super fair to me, mentioning that my change resulted in decreasing the system’s alerts overall by about 97% percent — I didn’t bother to measure its effect with that level of precision when I was writing my own quarterly self reflections. While the change had a positive effect on the system overall, <strong>fault injection was the thing that helped give objectivity to a subjective matter like design complexity vs gained reliability.</strong></p><p>In the same time, and assuming the fault injection exercise is performed in a non production environment, the fixes for failures that happen during a fault injection exercise are <strong>preventive </strong>actions, they prevent the occurrence of a problem that didn’t happen yet in a customer facing systems.</p><h3>How to perform a network partitioning fault injection</h3><p>Any distributed system has failure domains: groups of components of the infrastructure that could all fail together.</p><p>If we are talking about a cloud service, the largest failure domain should be a zone. This is the largest amount of components that could fail together- assuming we are talking about <a href="https://medium.com/@muhammad-soliman/what-does-it-mean-to-have-a-fail-safe-distributed-system-f9c793f84cfd">data plane not control plane component</a>.</p><p>Ideally, any distributed system should have at least 3 failure domains of the largest size because when one of them fails the system can still do consensus. This is why any cloud region has 3 zones or more.</p><p>So you should perform network portioning by disconnecting 1 of your largest failure domains from the rest, So for example if your system is deployed in 3 zones or 3 data centers you should perform the exercise by disconnecting all your infra structure components (servers, virtual machines, database replicas, etc) in one zone or one data center from the rest of the infrastructure.</p><h3>Network portioning fault injection in AWS</h3><p>AWS has a powerful tool called fault injection service (FIS). It is a tool that allows users to perform many kinds of fault injections including network portioning. It uses templates to describe fault injection experiments, where each template include one or more actions and each action has one or more targets. In case of network portioning the action is <em>aws:network:disrupt-connectivity</em>.</p><p>The targets for this action should be all the subnets in the zone you want to disconnect from the rest. Subnets are a zonal resource in AWS.</p><p>Behind the scenes, FIS works by adding access control list (ACL) rules that prevent traffic to and from the targeted subnets.</p><p>Each fault injection experiment template can be used to run the fault injection multiple times, with settings for duration of the exercise, cloud watch alerts to monitor to terminate the exercise if they fire and many more powerful features.</p><h3>Network portioning fault injection in GCP</h3><p>GCP does not have a powerful fault injection tool like AWS but you can perform fault injection manually.</p><p>Subnets are regional resources in GCP, so you cannot target full subnets to isolate resources in a given zone. So when creating resources, you should add a network tag to all resources in each zone with value equal to the zone they lie in.</p><p>Then to disconnect all resources in a given zone, in each subnet add firewall egress and ingress rules that prevent all inbound and outbound traffic. Firewall rules define targets by network tags, so the network tags of those two rules should be set to the zone you want to disconnect from the rest.</p><p>Removing or disabling those two firewall rules should stop the network partitioning.</p><h3>Network portioning fault injection in Azure</h3><p>I don’t know a way to perform network portioning fault injections in azure.</p><p>Azure’s network security group (NSG) rules, which are Azure’s alternative to access control lists at AWS or firewall rules in GCP, only affect new connections but do not affect already existing ones.</p><p>So if you have a virtual machine that has connections opened to other systems in your network or the internet or even your ssh client and you add a NSG rule that stops all traffic to and from this virtual machine, those connections will remain open. Opening new connections will not be possible but old connection will keep functioning.</p><p>So far I don’t know a way to overcome this limitation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bde62176aa4a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What does it mean to have a fail-safe distributed system]]></title>
            <link>https://muhammad-soliman.medium.com/what-does-it-mean-to-have-a-fail-safe-distributed-system-f9c793f84cfd?source=rss-c336f0f9e471------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9c793f84cfd</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Muhammad Soliman]]></dc:creator>
            <pubDate>Mon, 04 Mar 2024 11:43:46 GMT</pubDate>
            <atom:updated>2024-03-05T19:03:10.734Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nDksRULo6PK9s2MV" /><figcaption>Photo by <a href="https://unsplash.com/@ianjbattaglia?utm_source=medium&amp;utm_medium=referral">Ian Battaglia</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>If something is fail-safe, it has been designed so that if one part of it does not work, the whole thing does not become dangerous. Cambridge dictionary.</blockquote><p>Fail-safe is a common concept shared among many engineering disciplines. It means that if a component or a machine fails it leaves the system in a state that presents no danger to the users.</p><p>Let’s take the train brakes as a famous example to clarify the concept. Train brakes operate by compressed air. An intuitive idea to design such brakes is by applying compressed air pressure to brake shoes installed on each wheel when we want to stop the train. But what happens when the air pipes that transfer this compressed air are cut or have a big leak? This will cause us to have this huge mass full of people in motion with nothing to stop it. A disaster.</p><p>An improvement to this initial idea is to have a spring applying pressure by default on all brake shoes. Air pressure is used to release the brakes not to apply the brakes. This means that if the air pressure system fails for any reason like big air leakage, brakes are applied and the train stops. when the brake system fails it fails to a state where the train stops.</p><p>Another famous example of fail-safe systems can be seen in nuclear power reactors. To have a nuclear chain reaction neutrons have to hit atoms in a defined speed range. If neutrons are too fast or too slow nuclear fission will not happen. Neutrons need to be <em>moderated</em>, a term used to express reducing their speed to be in the range needed to cause nuclear fission. One design pattern of fail-safe nuclear reactors relies on reactor water or other fluid as a moderator with little moderation from other sources. In those designs, if the reactor overheated for any reason like a human error or a failure of some component, more water will turn into steam, which is described in nuclear engineering as <em>void</em>. This void means that there is nothing to moderate the neutrons, so neutrons become too fast with less water to reduce their speed, so they become too fast and cause less nuclear fission and the nuclear reaction slows or stops with no human or machine intervention. These reactors are described as having <em>a negative void coefficient</em> where the more void forms due to overheating the more the nuclear reaction slows down, void has a negative effect on the nuclear reaction. With no human or machine intervention, the failure to control the reactor leaves the reactor in a state where the nuclear reaction stops or slows down significantly.</p><p>There are numerous applications from all engineering disciplines to the fail-safe design concept.</p><p>So what does it mean to have a fail-safe software system or in particular distributed system? Knowing that software is used to coordinate or control a lot of our daily life things, what does safety mean in this context? For example, If a large critical system like the one used for air traffic control stopped working like what happened in the UK in August 2023 or the USA in January 2023, causing a cancellation or delay of thousands of flights and millions of dollars in costs, can this be considered a safe failure? And If this is a safe failure, what does an unsafe failure look like then? And if not, is there a way to make it a safe failure? What does safety mean in a distributed software systems context?</p><h3>Safety vs Liveness</h3><p>Theoretical distributed system literature classifies all properties of a distributed system, all the things that a distributed system promises to do, under two major categories: <em>safety</em> and <em>liveness</em> properties. Safety properties simply state that the algorithm should not do anything wrong ¹. Does this mean that a system that does nothing is safe? Yes, it does! Using this definition, an air traffic control system that stops working has a safe failure. An unsafe failure is a failure where it does wrong operations, like giving the wrong output causing planes to crash together for example.</p><p>And here comes the other category of properties, liveness properties. Liveness means that something happens. When the system receives a request it sends a response, this is a liveness property.</p><p>Many engineers are familiar with measuring and setting targets for how often the system fails, expressed in numerical targets like service level indicators (SLIs) and service level objectives (SLOs), like saying that at least 99.99% of requests should return a response, which is another way to say that the system will not return a response once for every 10,000 response at most, The error budget for the system is 1/10,000. Such measurements are measurements of liveness<em> </em>properties.</p><p>But besides analyzing how often the system fails, it is important to understand how the system fails, which is something done in activities like <strong>failure mode analysis</strong>, where you analyze what can lead to a system failure, what this system failure looks like, and its possible consequences. Unfortunately, it is difficult to put a numeric measure or promise on safety failures if a system is prone to them. From a theoretical distributed systems perspective, safety properties are properties that when violated at time t can never be satisfied again after that ¹. <strong>If your system is prone to doing an unsafe operation then it will forever be described as an unsafe system, it cannot be unsafe at one point in time and later become safe.</strong></p><p>Usually, safety and liveness are two competing sets of properties, and the larger the system (the more data it holds, the more requests it handles) the more difficult it is to achieve both safety and liveness properties, and compromises have to be made.</p><p>Let’s take the dynamo database as an example. Dynamo, and later the databases that had the same concepts as Cassandra, relied heavily on the idea of eventual consistency where the chance of reading the latest version of a data record increases as the time between reads and writes increases. <em>Eventually</em>, the client will get the latest version of the data. Eventual consistency is a tradeoff between availability and response latency on one side (two liveness properties) and correctness of data (a safety property) on the other side. If you ask for the latest version of a data record with strong guarantees that this is the latest version of the record, you will get a slow response and with lower chance of successfully getting a response at all. If you asked for any version of the record, optimistically hoping that you will get the latest version, you will get a quick response, and the chances of successfully getting a response increase.</p><p>Dynamo was originally built to be used inside Amazon retail, and one main component that relied on it was the shopping cart. This was before there was something called AWS. One deciding criterion for approving the idea of eventual consistency was measuring the rate of old data records returned to the shopping cart service, and in return measuring the expected rate of orders that require a fix by support personnel, like doing a reshipping of an item, and comparing it with the normal rate of human-made errors in handling of orders in fulfillment centers and shipping. It was found that the rate of errors introduced by eventual consistency was too little compared to the already existing rate of errors caused by humans so the idea was approved.</p><p>From the CAP theorem perspective, consistency is a safety property while availability and partition tolerance are liveness properties.</p><h3>Control planes and safety</h3><p>Control planes and data planes are two concepts that came from networking and found their way to distributed systems. In networking control plane controls data flow, the data plane is where the actual data flow happens. In distributed systems, the control plane is used to create and configure resources (databases, virtual machines, data buckets, etc) and the data plane is the resources themselves and the operations performed on them (query a database, delete record, run compute workload, etc).</p><p>While Dynamo did a trade-off of some safety to get more liveness, other types of applications prefer to have safety over liveness, they prefer to guarantee the most correct actions over being quick or doing anything at all. Control planes are a famous category of applications that favor safety over liveness. Let’s take control planes of AWS as an example.</p><p>In AWS, Control plane operations are not covered by SLAs! S3’s SLAs cover reading or writing data from and to an S3 Bucket but do not cover the creation of the S3 bucket. Control planes tend to be more complex with more dependencies, so they are more error-prone, and on top of that they favor being consistent partition tolerant (CP) systems, they prefer not responding or responding slowly over corrupting data or taking wrong actions. Some AWS services’ control planes even exist in a single region ². Data planes on the other hand tend to be simpler, faster, and target being available partition tolerant (AP) systems ³ and exist in multiple regions.</p><p>This is why it is strongly recommended to make your systems control plane operations rely on other systems or cloud services’ control planes, and your data plane operations rely on other systems or cloud services’ data plane, and maintain a sharp separation between the two types of operations. Making your systems’ data plane operations rely on another system’s control plane operation decreases your systems’ availability and response time.</p><p>One important consequence of control planes favoring safety over failure can be seen in auto-scaling systems. Auto-scaling adds more compute resources (pods, virtual machines, load balancers, etc) to the system when the load on the system increases and takes them away when the load decreases. Since adding resources is a control plane operation, so slow, and has no promised bounds to this slowness, auto-scaling fails in handling sharp spikes where traffic unpredictably increases significantly in a short period. People treat auto-scaling as a silver bullet and if I had a dollar for each time I had to explain that auto-scaling cannot handle sharp spikes I would have been a millionaire.</p><p>Another example of compromise between safety and liveness in control planes can be seen in Kubernetes’ scheduler. The scheduler is the component that assigns a pod to a node. If the system has a lot of nodes and pods put complex scheduling constraints (like spreading all pods from a deployment evenly among all nodes), the scheduler will have a lot of pods and nodes to consider for every scheduling decision which will result in slow scheduling. Given that the Kubernetes scheduler is sequential, where it handles one pod at a time in queued order, the slowness builds up and affects all scheduling operations.</p><p>So a tradeoff that Kubernetes scheduler offers is a configuration parameter to consider only a limited number or ratio of nodes in scheduling decisions. This is a tradeoff between making the most correct scheduling decision but slowly and making a correct enough scheduling decision but quickly. A tradeoff between safety and liveness that is configurable.</p><h3>Fail-safe vs fail-secure</h3><p>one key aspect of evaluating how a system fails is to see if it fails an open failure or it fails a closed failure.</p><p>A basic example of the contradiction between fail-open and fail-secure is what happens when you have an automatic door lock and you experience a power failure. Opening the door guarantees that everyone in the building can leave safely, particularly if the failure was caused by a disaster, but this can allow thieves in. Fail-open is a kind of fail-safe.</p><p>In distributed systems, in an ideal world, when a system fails you want all hands to be on deck, or at least all hands can be on deck, every engineer can contribute to debugging the problem and restoring the system as fast as possible, the system’s failure is open to everyone to help fix. An open failure is a safe failure. However, giving access to everyone is against the least privilege security principle, it contradicts a bunch of security and even sometimes regulatory rules (depending on the nature of the application).</p><p>Both Google ⁴ and Facebook ⁵ had incidents where security restrictions on accessing hardware and networking caused delays in resolving incidents, eventually in the case of Facebook costing the company 100 million dollars and a 5% stock price loss.</p><h3>Conclusion</h3><p>It is important to understand how the system fails, and what tradeoffs are made around different failure modes. While fail-safe sounds like a good thing, safety has a lot of tradeoffs including but not limited to latency and availability.</p><p><strong>References:</strong></p><p>1. Introduction to Reliable and Secure Distributed Programming.</p><p>Christian Cachin, Rachid Guerraoui and Luís Rodrigues.</p><p>2.<a href="https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/global-services.html">https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/global-services.html</a></p><p><a href="https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/control-plane-and-data-plane.html">3. https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/control-plane-and-data-plane.html</a></p><p><a href="https://google.github.io/building-secure-and-reliable-systems/raw/ch01.html#reliability_versus_security_design_cons">4.https://google.github.io/building-secure-and-reliable-systems/raw/ch01.html#reliability_versus_security_design_cons</a></p><p><a href="https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/">5.https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9c793f84cfd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sorry, You don’t have enough motivation to join our company or team]]></title>
            <link>https://muhammad-soliman.medium.com/sorry-you-dont-have-enough-motivation-to-join-our-company-or-team-eecd9723de0e?source=rss-c336f0f9e471------2</link>
            <guid isPermaLink="false">https://medium.com/p/eecd9723de0e</guid>
            <category><![CDATA[hiring]]></category>
            <dc:creator><![CDATA[Muhammad Soliman]]></dc:creator>
            <pubDate>Mon, 29 Jan 2024 21:29:53 GMT</pubDate>
            <atom:updated>2024-01-30T11:31:05.766Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Epo68_CCgoyo12p9OM3cUw.jpeg" /></figure><p>In 2013 google news had a main feature: it would give you different coverages from different online news sources to the same story or event. But this feature was not supported in Arabic language. So I decided to make a website that does particularly this. <br>The website was built on top of Cassandra and Elastic Search. It crawled Arabic news sources, and in particular Egyptian ones, and used Elastic Search to get similarity score between different news stories with different fuzzy matching algorithms. If the score exceeded a certain threshold then it would consider the two links covering the same news story.</p><p>At first I was expecting some errors that would require tweaking the technique, but I got a lot of correct matches from early trials. With inspection I found that the similarity score was 100% in almost all news stories. The reason is that one symptom of the deteriorating Egyptian journalism was that all news sources were copying each other without even bothering to change a single character, even with the spelling mistakes. Everyone knew that this happens, but I didn’t think that it was to this degree. There is a difference between noticing two news stories that are the same while browsing news in the morning, and having the numerical proof to this on the full scale of almost all online news sources. So the added value of the website was limited, and it had about 40 visits per day only.</p><p>After the military coup in Egypt, and like many like me I was trying to flee Egypt by applying for any job abroad. One colleague in my Egyptian company had already an offer from booking.com and forwarded my CV to a recruiter. I got an online interview that I passed and went to Amsterdam for a face to face interview.</p><p>At this time booking.com asked you to do a small presentation at the start of the interview about a topic you like. I presented the website I made and I said that I don’t have a lot of traffic so every thing (crawling, matching, web serving, etc) run on a single AWS instance, but I have plans for scaling it up. One of the interviewers said OK, lets talk about your supposed design for scaling up. That was my system design interview.</p><p>Of course I did a lot of mistakes, the design doesn’t hold water by my current standards, but I was coming from a C++ , scientific visualization, 3D image processing , computational geometry, computer guided surgery job. I had experience in web development but never on large scale. I had grown interest in large scale distributed systems in last few months before that but it was all based on reading with no real application.</p><p>The interviewer didn’t stop here and went to designing something like google news without Elastic Search to do the matching. This was a core software engineer interview, not a senior, not a staff or principal, so this exceeds the limits of such role, but I went on with him using unconventional data structures in such problems (like KD trees) that were not ideal for this problem but came from my background in computational geometry and collision detection, and went with optimization until he had to stop me saying (you cannot optimize this beyond x because this is a problem of type y and there is a mathematical proof for this limit). I said Ok. Then he realized that he almost took all the interview time without allowing the other interviewer to ask me so he left him the remaining time.</p><p>Later on after I worked at booking.com I found that he was one of the most respected and knowledgeable engineers, but he was famous for giving super difficult interviews,to the extent that some colleagues called his interviews (the goodbye kiss) because it is like the company is kissing you good bye and you are not getting the job.</p><p>After the interview my friend was waiting for me to get the updates. I told him what happened and he was angry at me for allowing the interview to go into complex areas. He wanted me to limit the scope of the interview to just pass without getting somewhere too complex to handle.</p><p>I got the job but this was more than just a job for me and for my friend. For me it was another life, escaping a country under a military coup that had less than a year ago massacred thousands of people in a public square, one of those people was a childhood friend who I knew from KG till college. Going to the interview I left behind my wife who was 7 months pregnant, I was making a major step for my family and for my unborn son. For my friend he had just moved to the Netherlands, he knew no one in the country, his wife knew no friends, and he needed a familiar face here. Later, 6 of our colleagues in the Egyptian company moved to the Netherlands in particular to be with familiar faces.</p><p>So he didn’t care for how marvelous I was in distributed systems, how smart was my design, etc. I didn’t care either but at this time I had little experience in such interviews to direct the interview. Both of us cared for something else that exceeds just getting this job.</p><p>Now I have a Dutch passport, I have worked in booking, amazon, elastic. I was an interviewer in hundreds of interviews. In the feedback meeting after the interview, weather I am the interviewer or interviewee, I hear in many cases the argument that (you / the candidate does not have enough motivation to join the company / team). Sometimes when I say that I want to move to this team or that company because it has this positive attribute, the interviewer says (but many other teams / companies have the same trait).</p><p>I still remember a candidate that I was interviewing and in the interview feedback one of the other interviewers said that he wants to work here because he wants to leave his country to get better life conditions which is OK. I felt happy to hear this and in the same time it felt strange to hear in the sea of all the hiring managers or recruiters that require the candidate to be drooling (or pretend to be drooling) to join this particular company or team. I felt that I am not the only one declaring this opinion.</p><p>Yes, your company / team is not that unique. and Yes there are other companies / teams that can fit the candidate’s requirements for the job he is looking for. And not all people sleep at night dreaming of moving to your company / team and not all of them are aware of the (pretend that you are interested) game or willing to play it. That said, I will never show to an interview without doing my due diligence, studying all available information about the company/ team, their product, their market position, their culture, their technology stack, you name it. I will do my best to be effective in my job and in the interview.</p><p>And I will do my best to deserve the salary I get, Even if the job is not optimal for me. When I joined booking.com I wanted to join core infra department because of my interest in distributed systems. Without proven record in large scale systems I didn’t have a big chance, so I spent a year and 3 months working in front facing systems and playing with distributed queues in hackathons until I got the opportunity to move there. But during this year and 3 months, when I was working on things that was not my first choice, my quarterly evaluation was exceeds expectations most of the time.</p><p>Not all people have the same motivation model that some hiring managers or recruiters want to cast everyone in, where you are only productive when you work in a company / team if you only strongly believe in this company’s / team’s particular mission and unique traits. Trying to cast everyone in this particular box deprives you from excellent people who are capable and willing to do the job because they have different motivations.</p><p>For many people a job is part of life, not all life. Not everyone gets his purpose of life or self definition from his job, and even if so, jobs imply other things: they imply salary, better working environment than the candidate’s current company or team, new learning experience, city or country to live in and even nationality. Those things are for many people more important than the job itself and the self definition value it may provide.</p><p>Not to mention that many people are old school who value doing their job to the best of what they can, regardless of what this job is or how much they like it or hate it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eecd9723de0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Uninformed opinion in a meeting is just plain noise]]></title>
            <link>https://muhammad-soliman.medium.com/uninformed-opinion-in-a-meeting-is-just-plain-noise-5a5f9f54b6ca?source=rss-c336f0f9e471------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a5f9f54b6ca</guid>
            <category><![CDATA[management]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Muhammad Soliman]]></dc:creator>
            <pubDate>Mon, 03 Jul 2023 10:29:13 GMT</pubDate>
            <atom:updated>2023-07-03T14:39:11.132Z</atom:updated>
            <content:encoded><![CDATA[<p>One feedback that I got more than once from different people in different companies is that I do my homework extensively before meetings. Believe it or not this was not given as a positive feedback! The rest of the feedback is that I give my opinion in very little topics but when I do I have already studied the problem deeply, so I know a lot and I come with a lot of relevant data points that allows me to quickly provide the counter argument to everyone’s opinion. Usually my opinion or suggestion or suggested decision is correct — this is not me talking, this is what I got in the feed backs - but people are not comfortable when their ideas are dismissed so quickly and with arguments that leave no space for maneuvering. Another third similar feedback was that I like data a lot and like to put them in arguments! and again believe it or not it was given as a point of improvement not a positive feedback. A forth similar discussion with a skip manager when I asked to control who goes into a major project discussion or agree about points participants need to read about before coming to the meeting was that “Everyone is entitled to his opinion” which I replied to with “Everyone is entitled to his <strong>informed</strong> opinion”.</p><p>I was talking about this with a friend and he told me that his manager asked him why he reports in stand ups that he had this meeting today while the meeting is only one hour? where did the rest of the hours go? My friend answered that this meeting involves discussing a document that has some suggestions. He spent time reading the document, the other documents it refers to, read books and articles about the topic and build an opinion to take to the meeting. His manager looked at him silently and astonished.</p><p>One of the very early lessons that I remember clearly as a young engineer is a lesson from Steve McConnell’s “Software Estimation” book, which recommended to never give of the cuff estimations. Whenever someone asks you how long do you think this will take, never answer him immediately, resist the urge to answer and resist the person’s pressure, take your time to do do your estimation and come back later with your estimation. Later I learned that opinions in general are very expensive and tedious to build. You don’t have an opinion, you <strong>build</strong> an opinion through reading and thinking. And since opinions are expensive you cannot have an opinion in every and each topic, <strong>you have to select topics in which you build an opinion</strong>.</p><p>This idea clearly contradicts with the amount of meetings people usually have in the software industry, particularly when those people climb high in the management or technical ladder and go to meetings that cover wide range of topics or projects. It is very common in comics to see the industry mocked for too much useless long meetings, and in my opinion it is a self feeding vicious circle where people with little knowledge of the topic go to meetings and naturally cannot reach a decision quickly so they schedule more meetings and in turn the total number of meetings everyone has to attend grows, which in turn leaves little time to do the due diligence before every meeting and so the vicious loop continues.</p><p>In Egypt we have a lot of jokes about how a layman is willing to discuss topics they have little to no information about and give confident opinions or even information, but as I reflect on the idea I am surprised at how much sophisticated people with big titles in multi billion dollar multi nationals are willing to go discussing topics they have very little knowledge about and go in circles of fruitless discussions — or worse: come with the clearly wrong decision — with no one willing to say “I don’t know so I am not going to participate or I will go learn about the topic before I come to this meeting”. I don’t know if this is part of modern entitlement culture where people are entitled to have a lot and do so little or this is something that plagues big multinationals corporations only or what exactly.</p><p>One of the basic questions in history and sociology is why the western culture dominated the world starting from the renaissance till now? what attributes did this culture have that allowed it to overcome and dominate over other cultures?. And while this is a question that will never have a definitive answer, one of the interesting differences between Europeans and other cultures like Arabs, Turks and Chinese in the era just before the western dominance started was that their maps included question marks where the new world is. They don’t know what is there, while the other cultures were confident in what lies there: edge of the world, daemons, hell, whatever. Only Europeans were able to say “We don’t know”. And the same thing continued when the Americas were discovered, so while the Dutch, English, French and Spanish fought for the new land and went exploring what lies there, the rest of the cultures, some of which had better sea navigation abilities were not interested to join the fight because they were confident in what lies there: nothing useful. Only the Europeans were able to say “We don’t know what possibilities are there in this land or what are its boundaries, lets explore”. It looks like saying “I don’t know so I will learn” does not come naturally.</p><p>Software engineering is a knowledge worker industry and hence has special attributes, where managers have position based power and engineers have knowledge based power. The effect of this on different management aspects was discussed extensively, and decision making is one of those aspects . Andrew grove was Intel’s super famous CEO, under whose management — along with Gordan Moore the creator of the famous Moore’s law as chairman- intel moved from making memory to manufacturing processors and became the giant it is now. Grove has a super famous management book called “High output management”, a book that popularized some basic managerial practices today like 1 on 1s. The book had a dedicated chapter for decision making in knowledge based industry. In this chapter, one of the many controls for making decisions was that people at the lowest (in terms of title or position) competent level be involved in the meeting. Competent was defined as <strong>not only having knowledge of the topic but also good judgement</strong>, which comes with experience. But what I notice is that as an industry, even knowledge is not a requirement in people involved in decision making processes and meetings. Note that we are talking about a book that was first published in 1983, 40 years later and we are far away from the guidelines the famous book has put for decision making in knowledge based industry.</p><p>Building an opinion has exhaustive due diligence, And giving an opinion without doing the due diligence is plain noise, and the more we have of this the more noise we have in the meeting. The happy case of this is wasting people’s time, the sad case is that this noise will contribute to making the wrong decision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a5f9f54b6ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Soft skills are not enough to be an engineering manager]]></title>
            <link>https://muhammad-soliman.medium.com/soft-skills-are-not-enough-to-be-an-engineering-manager-4eaa2b0f24d1?source=rss-c336f0f9e471------2</link>
            <guid isPermaLink="false">https://medium.com/p/4eaa2b0f24d1</guid>
            <category><![CDATA[management]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <dc:creator><![CDATA[Muhammad Soliman]]></dc:creator>
            <pubDate>Mon, 26 Jun 2023 12:07:08 GMT</pubDate>
            <atom:updated>2023-06-26T12:12:07.178Z</atom:updated>
            <content:encoded><![CDATA[<p>In your opinion, What is the starting point for someone to get a position as a junior or a graduate developer in any company? My answer is that usually the starting point is that the person spends a long time reading programming books, tutorials and videos, getting education in college or courses, applying this knowledge in some dry run environment like college projects, personal projects, hello world projects, etc. After that there is a chance for this person to start as a developer somewhere, but never before.</p><p>Now compare this to the starting point at which someone gets a chance to getting a software development management position, even the lowest management positions, what is the starting point for someone to be a manager in the software industry? In the past this move was usually started by someone who showed good technical abilities so someone higher up the management chain assumed that since he/she is a good developer then this implies that he/she will be a good development manager. That was the wrong answer and naturally we had an era of questioning if being a good engineer would imply being a good engineering manager, so the industry moved to another aspect which is the soft skills aspect: if you are an engineer with good soft skills like communication and social acceptance then this is enough for you to get your first role as an engineering manager.</p><p>Again, wrong answer.</p><p>Management is a science. There are books, courses, college programs in management: engineering management, software management, project management, you name it. And while I think that engineering interviewing and evaluation is a flawed process across the industry, I think it is 10 fold more rigid process than managers interviewing and evaluation. Try to guess or estimate how many of basic management topics (like different organizational structures, where a team lies in the company’s structure and the effect of this on the team’s operation and method of work, different companies’ business phases and its implication on the software development process, companies’ financial attributes and their effect on the engineering efforts of those companies, different ways of running engineering operations, different forms of decision making process, dual reporting, risk evaluation, etc) that a person was asked before being given his first role as a manager in our industry. Or even try to guess how many of these topics your direct manager was asked before he was hired in his interviews for his role. I doubt that the answer will be satisfactory.</p><p>One contributing factor to this is the massive expansion of software companies or engineering departments. With increasing head count the company needs more managers to manage those people which in turn means that companies lower the bar for someone to become a manager. We are not going to wait for someone to finish actually learning what it takes to be a manager and in wild expansion phases the process becomes practically anyone who raises his hand and is socially above average can get his first managerial role.</p><p>With very few exceptions, those topics mentioned above and other similar topics are sort of magic or science fiction to most managers that I have worked with, and I have been working in multi billion dollar companies in the last 10 years. Most managers I have worked with don’t even know where to start the conversation around those topics, or they learn them the hard way while on the job, one screw up at a time — some don’t learn even with screw ups. The ones who really know those topics were sort of unicorns and working with them was a totally different experience, and of course the performance of their teams was usually something from another world — at least given the surrounding environment they operated in.</p><p>like programming, Management is a science and a skill and it should be treated accordingly. You cannot just focus on the skill without the science and you cannot assume that the skill equals social skills. Social skills being easy to evaluate does not mean that it equates management skills.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4eaa2b0f24d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Components of a good postmortem]]></title>
            <link>https://muhammad-soliman.medium.com/components-of-a-good-postmortem-8d14ee52bcbd?source=rss-c336f0f9e471------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d14ee52bcbd</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[site-reliability]]></category>
            <dc:creator><![CDATA[Muhammad Soliman]]></dc:creator>
            <pubDate>Mon, 01 Aug 2022 10:50:35 GMT</pubDate>
            <atom:updated>2022-08-01T10:51:33.882Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rvy4j3wu2r5lTFzXMJY1Uw.jpeg" /></figure><p>Postmortem is a written record of an incident. Postmortem after an incident is how to ensure that this incident will not happen again in the future and that things after the incident became better than before the incident. Ideally a postmortem is discussed and reviewed then published for the engineers to learn from.</p><p>To ensure that a postmortem is useful it should answer six questions.</p><h3>What was the impact?</h3><p>The impact of an incident should be the decisive factor in starting a postmortem or not in the first place. Incidents happen frequently and it is impossible to do a postmortem for every single incident. Engineering teams should have a predefined criteria that specifies if they should do a postmortem for a given incident or not.</p><p>Examples of impact that triggers a postmortem are:</p><ul><li>consuming a defined ratio from error budget: An error budget is the other face of the SLA/SLO coin. For example, if your SLA/SLO is to serve 99.9% of requests successfully then your error budget is 1/1000 of requests, You are allowed to fail in serving 1/1000 of requests, this is your error budget. An incidents that eats more than a defined ratio of the monthly or quarterly error budget (like 20% or 30%) should be a justification to start a postmortem. <br>Ideally breaking SLOs or SLAs should have consequences or they would be meaningless. breaking SLAs usually results in compensation for external users of your service, breaking SLOs should result in a company internal consequence like stopping working on new features till the end of the month or quarter. This is why engineers should react before they consume their monthly or quarterly error budget.</li><li>If the incident exceeded a certain severity threshold: Some companies have a classification of severity of each incident. for example AWS support cases have the following severity classifications: low — General guidance, normal — System impaired, high — Production system impaired,urgent — Production system down, critical — Business-critical system down. <br>If an engineering team or a company have a severity classification of incidents then incidents exceeding a certain severity level should get a postmortem.</li></ul><p>If possible, the impact of an incident should reflect final user and business impacts, and it would be nice if this could be mapped to financial impact although this might be difficult in many cases, like when the service or component that had the incident is deep down the services hierarchy and consumed in different ways by components that have different business or user impact.</p><h3>What happened?</h3><p>This is usually the easiest part. It should explain what happened and ideally this should be in the form of a timeline that states the sequence of events since the start of the incident till it was solved. It would be helpful if the communication logs between engineers addressing the incident (like chat history) was attached to give discussions, series of solutions tried and result of each solution.</p><h3>How long did it take engineers to know about the incident and to react?</h3><p>This is a measure of how good the alerting and operational response to alerts are. If engineers were notified of the incident quickly then detection criteria were correct and alerting was well setup. If they reacted quickly then operational procedures around alerting are running well (on call rules are followed, escalation is done based on the defined escalation rules, etc).</p><p>In my system interview at amazon, and after I finished designing some large scale system, the interviewer asked me what would I monitor. I started listing various metrics that should be collected for such system but he stopped me by asking me to select a single metric , the most important one. More than five years later and long after leaving amazon I am still using this same question in system design interviews where I am the interviewer.</p><p>Ideally you should have 3 or 4 metrics that should define if a given system is working well or not and those are the ones that you should build your alerting on. Those metrics should be directly related to the business or user impact of your services. If I open a website I don’t care if the CPU usage of a the servers running the website is high, but I do care if the page loads in a long time. The google SRE book defines the <a href="https://sre.google/sre-book/monitoring-distributed-systems/#xref_monitoring_golden-signals">4 golden signals of monitoring</a>, and while they are not always the most important measures of all systems’ health as this varies from one system to the other, they are another way to say that for any system a few metrics are more important than the rest and ideally these are the ones you should alert upon.</p><p>Key metrics of a given system are not always easy to define -this is why I ask about them in system design interviews and many people don’t get them correctly- and failure to define them can result in discovering incidents late and in turn increasing the impact of the incident. Asking about time between incident start and engineers knowing about it and reacting to it can reveal such failures.</p><p>If engineers only knew about the incidents when users started calling or opening support tickets then this is a sign that something(s) in the alerting and / or operations is wrong and needs fixing, like the alerting pipeline is not well set up, alerting criteria are not correctly defined, the team is not building their alerts on the system’s key metrics, the team’s on call procedures and rules are not well defined or not well followed, etc.</p><h3>Could we have discovered it before it became an incident?</h3><p>while alerting kicks in when something goes wrong, ideally engineers should be monitoring bad trends and addressing them before they become an incident. Trends develop over long period of time, think of things like gradual decrease in free disk space or memory or gradual increase of unprocessed items in queues, etc.</p><p>A slight increase in unprocessed queue items count in a relatively short period of time is difficult to notice, but if multiple slight increases continue building up for weeks and months we end up with a very long queue that will eventually break.</p><p>It is difficult to automatically discover trends, usually engineers are the ones able to find them, not automation. To discover trends engineering teams need to develop a habit of periodically looking at monitoring dashboards looking for bad trends at a relatively long time intervals like two weeks or month. In some teams I worked in this was done daily by the on call engineer or in other teams it was done weekly in a weekly operational review meeting.</p><p>A relatively common problem when looking at dashboards searching for trends is having multiple dashboards when we have a lot of deployments and you setup a dashboard per deployment. For example if we run our system in multiple availability zones (AZs) or regions and we have a dashboard per region or AZ, or if we have separate deployment for each customer and we have a dashboard for each deployment. I used to be one of the engineers that developed and operated the automation that controlled one of the largest Cassandra fleets in the world that had 70 something clusters each one serving a different service or subsystem, holding petabytes of data in thousands of nodes, and where each cluster had a dedicated dashboard. How can we go looking at trends in each dashboard of the 70 something clusters? That can take forever. But we did it every day as part of the on call engineer’s daily routine.</p><p>The trick here is to have a dashboard that only displays the worst few metrics from all clusters for each type of metrics we monitor. So for example to discover trends in free disk space we had a graph that only plots the lowest 5 free disk space ratios of all clusters we have for the past 2 weeks. If the worst free disk space ratios are good enough and do not have bad trends then by definition the rest are more than good, we don’t need to look at the rest. The dashboard had graphs that applied the same for CPU utilization, number of unavailable nodes, etc.</p><p>The trend detection dashboard should be minimal, it is only used to quickly and easily answer a single boolean question: is everything OK or not?. This dashboard is not intended for troubleshooting or telling what is going wrong exactly. If we find that something is wrong with one particular cluster we go to the dedicated cluster dashboard to debug and troubleshoot.</p><p>So in a postmortem asking why didn’t the engineers catch the incident before it even became an incident is questioning the team’s monitoring and trend detection. It could show a missing signal that the team did not look at or could reveal things that need improvement in the team’s operational procedures.</p><h3>How can you prevent this from happening again in the future?</h3><p>This should be the action items that should prevent this incident or category of incidents from happening again in the future. It should contain solutions to address all issues discovered by the questions we asked earlier, so it could contain fixes to reliability bugs in code or changes to system design, but also it could contain fixes to alerting, fixes to monitoring, changes to operational procedures, etc.</p><p>Ideally this list of action items should be finalized after review and discussion.</p><h3>Who owns the follow ups?</h3><p>A postmortem is not closed until all action items are completed. Ideally there should be a tracking of postmortems where number of open postmortems and their age is tracked across teams, departments and the whole company.</p><p>But tracking is meaningless unless there is someone accountable for each postmortem. This is the person who is responsible to ensure that agreed upon action items are completed. Preferably he should be someone who can influence teams’ priorities like a team lead, a manger or a PO.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d14ee52bcbd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When a postmortem is not enough]]></title>
            <link>https://muhammad-soliman.medium.com/when-a-postmortem-is-not-enough-ccd5ac2bc068?source=rss-c336f0f9e471------2</link>
            <guid isPermaLink="false">https://medium.com/p/ccd5ac2bc068</guid>
            <category><![CDATA[site-reliability]]></category>
            <dc:creator><![CDATA[Muhammad Soliman]]></dc:creator>
            <pubDate>Sat, 06 Mar 2021 01:18:47 GMT</pubDate>
            <atom:updated>2021-03-06T01:29:21.588Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vgzzWpANbIBUCRIRGo1q0g.jpeg" /><figcaption>Challenger space shuttle explosion</figcaption></figure><p>When the Challenger space shuttle accident happened, Ronald Regan formed a committee to investigate the incident. I will give you a spoiler here: an O ring in a rocket booster lost responsiveness due to cold weather, so burning fuel leaked causing the explosion.</p><p>When Richard Feynman got an invitation to join the committee he wanted to say no. One of the reasons was that things will not stop at finding the direct mechanical cause of the accident. He will have to find out what is wrong with NASA as an organization, if the whole shuttle program should continue or should we move back to rockets, and then what should be the future goals in space. Yet some close people to him convinced him that he should join the committee and fortunately he did.</p><p>While everyone went looking for the mechanical reason for the explosion, and although Feynman played an important role in publicly displaying the O ring issue and putting some focus on it, Feynman went the extra mile (or miles) of questioning the organization that produced this disaster. He knew from the first moment that an incident at this scale results from a questionable managerial and technical culture. It does not stop at mechanics.</p><p>For example Feynman noticed that NASA’s managers claimed that the chance of an accident happening to a shuttle is 1/100000. This means that NASA can send a shuttle to space every day for 273 years and gets only a single accident. While he described this as fantasy, he did not stop here.He asked engineers to individually estimate the probability of rocket failure, then compared the results which were 1/100 in the best estimate to what management said, showing that there is a clear lack of communication between management and engineers. Then he went to what this entails: if we say that the shuttle’s failure possibility is 1/100000 the whole idea of measuring failure rates by multiple launches or multiple tests to components makes no sense because no one can test a component a hundred thousand times to get a single failure, so managers had to shift to other ways to prove that things are fine. So while the O ring had tons of failures in previous launches, managers deduced that since this did not result in an accident then these failures must be safe! They did things like measuring erosion, finding that it was 1/3 of the O ring, so they deduced that they had a 3x safety margin, while they had no grantees that the erosion in in next launch will stop at this ratio and they did not know what led to this erosion in the first place. No proper quantitative measures of the rate of failure, its limits, what could be deemed safe and what not, etc. The O ring was not acting as designed, no one knows why and since we abandoned the statistic measures and no accidents happened in previous launches then it must be safe. Feynman described this as playing Russian roulette pulling the trigger and when nothing happens deducing that it must is safe to pull the trigger again.</p><p>Feynman even went to inspect things that didn’t explode like the space shuttle engine and the software used to run the shuttle. We are software engineers, we know that software is the least reliable thing in the world, yet ironically the software and computers part was water tight.Feynman showed that the engine’s components had multiple failures in the past, and that they suffered from the same pattern of management behavior towards failures. Feynman even went examining the whole design process and comparing it to rocket and plane engine design process in the aviation industry and in the army’s air forces, showing that while usually people designed small components, tested them then assembled them into the full thing, NASA went by designing, assembling then testing. Of course when failures happened it was difficult to find which component is responsible, changes to a component’s design was difficult or impossible, etc.</p><p>I cannot give Feynman’s appendix to the committee’s report enough credit and trying to show just parts of it here feels like a crime, it should be read fully, it is a piece of art. However I do not have enough space.</p><p>Postmortems is part of site reliability engineering. You have and incident, you solve it, you write a postmortem, fix the process or design issues that lead to the incident and share it with the rest of the organization as a learning experience. While there is a common agreement on defining what should lead to a postmortem, like consuming a defined percent of your error budget, and unifying this across the company (see google’s first site reliability engineering book), a little attention is given to the scope of the postmortem in regard to the incident’s scale. From my experience postmortems usually focus on technical or process errors and rectifying them in the scope of a team or a group of teams or a single practice across the organization, but rarely does it extend to question the whole organization and the engineering culture that runs it.</p><p>When an incident results in a disaster, and a disaster definition here varies based on the size of your organization, nature of the service you are running, etc, the scale and scope of the postmortem should he proportional to this incident, questioning more that just the intimidate operational or technical practice that led to it. Ownership, actions to fix and follow up on those actions should come from high up the organizational ladder and should reflect deep changes.</p><p>The same should be done when you have more than a single incident, non of them alone is a disaster but their rate and size collectively qualify for being described as a disaster.</p><h3>Do not hide behind SLAs</h3><p>SLAs can be a blanket to hide the need for an organizational questioning. If have a web service that has SLA of request success rate of 3 nines, and assuming even distribution of requests across time, and then your service fails for an hour or a full day or a couple of days you break your SLA in all cases. In all cases you can just report this as (Breaking the SLA).</p><p>SLAs must be simple, one of the reasons for this is the need to be able to communicate them easily to the customer, yet this simplicity should not be used to treat all incidents equally and hide the need for organizational changes.You should not write the same level of postmortems in all cases, The major ones qualify for larger postmortems owned by people higher up the management chain and should include corrective actions on a scope that exceeds the smaller one.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccd5ac2bc068" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Service quotas and spike patterns as part of the SLAs of multi tenant services]]></title>
            <link>https://muhammad-soliman.medium.com/service-quotas-and-spike-patterns-as-part-of-the-slas-of-multi-tenant-services-57441b6e7e1d?source=rss-c336f0f9e471------2</link>
            <guid isPermaLink="false">https://medium.com/p/57441b6e7e1d</guid>
            <category><![CDATA[site-reliability]]></category>
            <dc:creator><![CDATA[Muhammad Soliman]]></dc:creator>
            <pubDate>Thu, 11 Feb 2021 15:03:06 GMT</pubDate>
            <atom:updated>2021-02-12T08:33:49.733Z</atom:updated>
            <content:encoded><![CDATA[<h3>Service quotas and spike patterns as part of multi tenant services’ SLAs</h3><p>In his famous book, “High output management”, Andrew Grove (Intel’s co founder and ex-CEO) explained that sometimes quality checks can stop the whole organization. For example, if you put a quality check on the input raw material for a factory, and due to some market changes providers became unable to provide raw materials up to these quality standards, then your whole factory stops. In this case, managers should discuss what is the accepted degradation in the product, unless reliability gets affected. Reliability is untouchable even if you stop the whole organization; the reason for this is that <strong>you can never know the consequences of low reliability</strong>. The example that he gave was a non reliable component that goes into pacemaker, it can be discovered at the manufacturer, causing cost to replace, but if it fails later after the pacemaker is implanted; then, the consequences are more than just financials.</p><p>Most industries have higher reliability than software, so it is easy to say that you cannot make compromises regarding reliability when you produce hardware, but not software. However, the basic principle of being unable to assess the overall consequences of low reliability still applies.</p><p>And here come SLAs. SLAs basically says that I, the software service provider, can provide this level of reliability and the consequences of this level of reliability should be assessed by the user because I cannot assess them.</p><p>A more explicit example is the famous severity classification of services done by many companies like Amazon. A Sev1 outage means it affected amazon.com retail operations for most or all users, the basics of buying of the site is down or major part of it. A Sev2 outage means that it affected some of the customers or some of the operations but not a major portion of them, Sev3 means that it is annoying to some customers but not causing serious problems.</p><p>Then, when you build a service you classify its severity, how reliable it is based on the accepted severity level of outages. S3 is a Sev1 service, this means that it is allowed for parts of amazon business that may generate a Sev1 outage to run on top of S3. A newly created service usually starts with low level of severity and with time (years usually) and maturity it is upgraded until it reaches Sev1.</p><p>So this is another form of shifting the reliability problem to the user. If I build a service and claim that it as Sev2 service, the developer using this service is the one responsible for classifying what types of outages he may generate and in turn use my service or not. If a developer in the team building the shopping cart for example starts using my service then it is the developer’s problem. That is another explicit form of shifting assessing the consequences of low reliability product to the user.</p><p>So If you are building a multi tenant service, a service that has many users where usage patterns of user A like excess traffic and spikes should not affect service quality perceived by user B, how can you do capacity planning? The short answer is (do rate limiting), but what should the rate limiting restriction and the service’s performance promises look like? how to describe what is accepted under what usage conditions and put this into your capacity plans and service description or SLAs.</p><p>In the cloud we find the concept of service quotas. Basically cloud providers say: for this service or that we can give you this amount of requests per minute, day, month, etc. If you want to exceed this quota you add a request. But the interesting part is that there are no guarantees on the time for processing this quota increase requests. If you ask to increase your big table quota the support engineer at google may have to contact big data team, who would do capacity testing in the regions you run your services at, add more servers, etc, before raising your quotas.</p><p>Even for auto scaling services there is a limit on spikes. You would expect that amazon’s elastic load balancer can handle any traffic spike — it has the word (elastic) in it — but even there there is a quota on the spike: by default elastic load balancers does not support a spike in traffic of more than 50% in less than 5 minutes. If you expect your traffic to go beyond this pattern you have to add a quota increase request before hand. Again no promises on how long processing your quota increase request take.</p><p>Being able to handle spikes means that you are over provisioning, there is no other way around this. The question is how to control this over provisioning to be in an accepted range.</p><p><strong>Similar to SLAs being a way to shift evaluating consequences of low reliability to the user, service quotas are a way to shift the capacity estimation problem and over provisioning to the user of your service</strong>. You give each user a given quota for their service account and you know how many service accounts you have, so you know your capacity needs. If a user asks for more you get more resources, with no promise on when you will respond to their quota increase request.</p><p>Speaking of how to implement rate limiting is beyond this article, In the cloud there are services that provide quota and rate limiting out of the box. For example AWS api gateway is facade that controls users access to your service based on the service account you give them, implementing whatever rate limiting and quota rates you specify for them. Google has similar service and Envoy can integrate with any rate limiting service that provides a specific api and there are much more ways.</p><p>However it is not only about rate limiting implementation, it is also an operational practice and agreement between you and consumers of your service.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57441b6e7e1d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>