<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Fahd Siddiqui on Medium]]></title>
        <description><![CDATA[Stories by Fahd Siddiqui on Medium]]></description>
        <link>https://medium.com/@fahdsiddiqui?source=rss-d26a5cde30dc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*1gL1UCbOpSpdZoF851KSfw.png</url>
            <title>Stories by Fahd Siddiqui on Medium</title>
            <link>https://medium.com/@fahdsiddiqui?source=rss-d26a5cde30dc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 20:54:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@fahdsiddiqui/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[CI/CD and the Wind Tunnel]]></title>
            <link>https://medium.com/@fahdsiddiqui/ci-cd-and-the-wind-tunnel-fd6fcebe6bc5?source=rss-d26a5cde30dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd6fcebe6bc5</guid>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Fahd Siddiqui]]></dc:creator>
            <pubDate>Sun, 16 Aug 2020 23:58:41 GMT</pubDate>
            <atom:updated>2020-08-18T14:54:34.480Z</atom:updated>
            <content:encoded><![CDATA[<h4>The wind tunnel was the Wright brothers’ competitive advantage, the CI/CD pipeline should be yours.</h4><p>I visited the Smithsonian National Air and Space Museum four years ago. The one thing that has stuck with me ever since is the meticulous experiments Wright brothers did to get to their aircraft design, and the methodology they adopted. I am convinced that their competitive advantage was their sublime design process, more than anything else.</p><p>What their design process gave them is what your CI/CD pipeline should give you. An easy way to experiment and deploy your designs. Test them, tweak them, perfect them — quickly and with confidence. What the wind tunnel was to the Wright brothers is what CI/CD should be to software engineers.</p><h3>Wright Brother’s CI/CD Pipeline</h3><p>The Wright brothers had their doubts about Lilienthal’s data and Smeaton’s coefficient that they were using with the widely accepted lift and drag equations. After designing two gliders that wouldn’t produce the lift those calculations predicted, they thought it was time they question the data they depended upon.</p><blockquote>The Wrights didn’t invent the wind tunnel, but they were the first to use a wind tunnel as an integral part of their aircraft design process.</blockquote><p>They focused on making a miniature wind tunnel where they could efficiently and accurately collect data for potential wing shapes. This allowed them to pick the most efficient wing for the aircraft they wanted to build. Not only were they able to check the accuracy of Lilienthal’s table of coefficients for his wing shape, but they were able to get data for dozens of other shapes. Finally, they corrected the Smeaton’s coefficient of 0.005 that was being used since the 18th century and came up with a new average value of 0.0033. Modern aerodynamics has proven this value to be accurate within a few percent. The key thing to note here is that the Wrights weren’t the inventors of the wind tunnel, but they were the first to use a wind tunnel as an integral part of their aircraft design process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*5xUXsjpexadwaiL_.JPG" /><figcaption>Wright Wind Tunnel, their CI/CD pipeline. Source: National Museum of the United States Air Force</figcaption></figure><p>We may not think of “agile” in the physical world, but that’s exactly what this setup was.</p><blockquote>CI/CD is not simply a way to package up artifacts but should offer an environment that allows prod parity with complete verifiability.</blockquote><p>Wright brothers never went to college and Orville never graduated high school. However, what they had in their wind tunnel was nothing short of what we should aspire in a sophisticated CI/CD process. CI/CD is not simply a way to package up artifacts but should offer an environment that allows prod parity with complete verifiability.</p><h3>Scientific method and CI/CD</h3><p>As engineers, we need to keep in mind the scientific method, you know the experimental technique employed by the sciences that we all probably studied as early as elementary/middle school. Let’s quickly revisit this method.</p><ol><li>Ask a question</li><li>Formulate hypothesis</li><li>Design experiment to test the hypothesis</li><li>Conduct the experiment</li><li>Compare your observations (what actually happened) with what should have happened if the hypothesis holds</li><li>Draw conclusions and accept/reject the hypothesis</li><li>Repeat and profit</li></ol><p>Now, ask yourself if your CI/CD environment equips you with the needed tools to follow the above intuitive steps of the scientific method.</p><p>CI/CD should serve us as the mechanism to lower the cost of experimentation. Just like the wind tunnel, the CI/CD environment offered to each developer pays off dividends in primarily two ways (if not more):</p><ol><li>The reduced cost of a null hypothesis or an invalid hypothesis that, otherwise, may present itself in the form of service degradation or worse, complete downtime.</li><li>The other bigger intangible benefit will come in the form of a low barrier to innovative ideas. There is no way Wright brothers would have collected data on dozens of wing shapes if the only way to do it was to manufacture gliders. It would have been foolish for them to experiment with non-traditional ideas as the cost of failure would be too high. So, make a wind tunnel for your team, and unleash the innovation by following the demonstrably successful scientific method.</li></ol><h3>A Word about Microservices and Containerization</h3><p>A personal dev environment for each developer that closely resembles the production environment is imperative to be able to cultivate a culture of following the scientific methodology. Containerization and orchestration tools such as Kubernetes bring us much closer to this reality than ever before. The complexity to spin up new environments have drastically gone down over the years so much so that a shared “dev” environment can be replaced by individual environments customized to each developer branch. The various functional, integration, and performance tests can be directed towards any custom environment to compare the results.</p><p>Depending on the graph of your microservices, your product can quickly become a complicated platform as a whole with several edges that can break things in unexpected ways. Microservices development can be both complicated and simple simultaneously. On the face of it, every microservice is independently deployable and scalable, which may decrease the development complexity. However, as the graph of microservices expands, it becomes harder to reproduce the dev environments locally. The end to end picture requires us to synchronize the deployment of multiple services. Some of these microservices would be the domain of a single team, while others may be managed by others. The dependent services that are managed by others could be consumed from a shared “integration” environment while the services under test would be deployed in their custom namespaces.</p><p>Jenkins-X (<a href="https://jenkins-x.io/">https://jenkins-x.io/</a>) is a good evolutionary step in this direction. Check it out, if you haven’t already done so. I do think there is room for improvement there, but for the most part, it should solve the following:</p><ul><li>Remote debugging tools: Easy way to open up a port that a remote debugger is able to attach to.</li><li>Rsync files or binaries to the dev environment from the local file system. Make it super easy and fast to deploy changes to your environment while sticking to your favorite IDEs locally.</li><li>Tools to kick-off long-running tests: Scripts that make it easy to kick off tests from CLI.</li></ul><h3>Conclusion</h3><p>The one thing that seems to be common among all smart engineers who seem to always solve a sticky issue is rarely extra information that they possess and others don’t. Almost always, it is the process they have with which they approach a problem. They have a knack of creating a tight iterative cycle where they can reproduce an issue and then keep tightening the loop until they are closely monitoring the root cause of the issue. This requires a fairly organized flow of thoughts. They put in logs if the data is lacking. They put in metrics and wait until they can validate all assumptions. In other words, they will always create an environment that allows them to experiment quickly and accurately. Demand that from your CI/CD environment. Take a page from history and make CI/CD your competitive advantage.</p><figure><img alt="The Wrights’ wind tunnel measurement book and graphs. Source: Smithsonian National Air and Space Museum" src="https://cdn-images-1.medium.com/max/1024/0*T5e9HRyH-C-DIg_U" /><figcaption>The Wrights’ wind tunnel measurement book and graphs. Source: Smithsonian National Air and Space Museum</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd6fcebe6bc5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cleanroom Builds]]></title>
            <link>https://medium.com/@fahdsiddiqui/cleanroom-builds-86931fc849b9?source=rss-d26a5cde30dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/86931fc849b9</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[developer-tools]]></category>
            <dc:creator><![CDATA[Fahd Siddiqui]]></dc:creator>
            <pubDate>Sun, 01 Sep 2019 23:42:42 GMT</pubDate>
            <atom:updated>2019-09-01T23:49:57.933Z</atom:updated>
            <content:encoded><![CDATA[<h4>Use a docker “builder” image that builds your deployable artifacts.</h4><p>Does this sound familiar to you?</p><ul><li>You need to install protobuf compiler, java 11, thingamajig version 7 to build the project on your machine.</li><li>The build is fine on our Jenkins Linux box, but your local mac needs other things.</li><li>You need python 3 for some scripts, but python 2.7 for some legacy stuff. Create virtual python environments on your machine to keep your sanity intact.</li></ul><p>You get the idea. The point is that your local machine almost never mirrors your actual build environment. The problem is exacerbated by more sophisticated tools that we need to build our artifacts. Some codegen tools that are now ubiquitous introduce another level of complexity if the version is not strictly controlled. Linting tools, quality checking tools, all have to be carefully synchronized among the team of developers. Can we bring about complete local/Jenkins parity? As it turns out, we can borrow yet another concept from the manufacturing and scientific research world — cleanrooms.</p><h3>What is Cleanroom?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*VyNq7tFgvUDGssssGJzgHw.png" /></figure><p>Anybody familiar with “Breaking Bad” knows that certain measures need to be taken to achieve 99.1 percent purity. They dedicated an entire episode “Fly” on just how important a cleanroom is (honestly there are wildly different interpretations of that episode, but this is what made sense to me). Typically used in manufacturing or scientific research, a cleanroom provides a controlled environment free of pollutants to maintain consistency in the manufacturing process.</p><p>A consistent environment is important to control quality in the manufacturing process. In our world, we try to do that by writing elaborate READMEs so every dev in the team can have a homogenous environment, which is a futile attempt. With no empirical evidence, I am going to claim that thousands of developer hours are wasted getting our environments just barely consistent enough to successfully make the build.</p><h3>Making of a Cleanroom for our product</h3><ul><li>Build the “builder” docker image that will build the project.</li><li>Use the above container to build the actual deployable artifact, whether on your local machine or on a Jenkins agent.</li></ul><p>The first thing that the build process would do is make the “builder image” itself. It is this image that gets to build your project. This Docker build image may look something like this:</p><pre>FROM openjdk:14-alpine</pre><pre># Install dependencies like protobuf compiler here or, better yet, you may want to use an image with all the required dependencies.</pre><pre>RUN mkdir /build</pre><pre>WORKDIR /build</pre><pre># Run the following build command in the builder container.<br>CMD [&quot;mvn&quot;, &quot;clean&quot;, &quot;package&quot;]</pre><p>A separate image for build makes it independent of the host machine. It doesn’t matter if the host machine happens to be your local box or a Jenkins agent. This “builder image” is now used to build the application, giving us complete local/Jenkins parity. The inconsistencies of our local environments matter no more. So, first let’s build this image:</p><pre>docker build -t $(BUILDER_TAG) -f $(BUILD_DOCKERFILE_NAME) .</pre><p>Now, we need to run the build in the builder container to generate the artifact. The following docker command, when executed from the project root location, will mount the project directory in the host machine to the build directory in the container. This way the build command runs in the container, aka, the cleanroom.</p><pre>docker run -t -v `pwd`:/build $(BUILDER_TAG)</pre><p>That’s it! This will create your artifacts using a “builder” container, which is the same container used by Jenkins.</p><h3>Conclusion</h3><ul><li>Remove any ambiguity from the build environment by using a “builder” docker image.</li><li>Achieve complete local / Jenkins / production parity.</li><li>You may still want to get all the dependencies on your local machine, for different reasons. But if you do, you should still use the builder container to build one final time to be confident that nothing breaks.</li><li>Last but not the least, never tell your one team-mate who is on Linux to switch to Mac because she keeps breaking everyone else’s dev environment :).</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86931fc849b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Consistent Cuts in Dynamo-style Systems]]></title>
            <link>https://medium.com/@fahdsiddiqui/consistent-cuts-in-dynamo-style-systems-2fd4dd992214?source=rss-d26a5cde30dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/2fd4dd992214</guid>
            <category><![CDATA[magento]]></category>
            <category><![CDATA[adobe]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[cosmosdb]]></category>
            <dc:creator><![CDATA[Fahd Siddiqui]]></dc:creator>
            <pubDate>Sun, 21 Jul 2019 19:27:56 GMT</pubDate>
            <atom:updated>2019-07-21T20:53:31.587Z</atom:updated>
            <content:encoded><![CDATA[<h4>Demand consistency metrics and consistent cuts from your eventually consistent systems</h4><p>Microsoft’s Azure Cosmos DB packs quite a punch with its spectrum of consistency level choices. A less celebrated feature, albeit a very powerful one, is consistency metrics, such as probabilistic bounded staleness (PBS), out of the box. These metrics answer a simple question: How eventual is your eventual consistency? In other words, how far back do we have to go to guarantee that all reads have a consistent view of the world without any gaps? Note that this is different from just replication latency.</p><h3>What is a consistent cut</h3><p>Consistency metrics are not just about monitoring the health of the system, although that is important. Being able to answer the above questions leads us to, perhaps, the most interesting consistency level choice that Cosmos DB offers — <strong>consistent prefix</strong>. A consistent prefix is defined as some prefix of all the updates, with no gaps. In distributed systems, this is the same as a <strong>consistent cut </strong>or a <strong>consistent snapshot</strong>. A consistent cut is a set of events that are causally ordered with no gaps. Formally, a consistent cut in an event-based distributed system, E, is defined as any subset F ⊆ E, such that</p><p><em>f</em> ∈ F ∩ <em>e </em>→ <em>f </em>⇒ <em>e</em> ∈ F,</p><p>For F to be a consistent cut, it should satisfy the condition: if event <em>f</em> <strong>belongs to</strong> F <strong>and</strong> event <em>e</em> <strong>happens before</strong> event <em>f</em>, then <em>e </em>also<em> </em><strong>belongs to</strong> F.</p><p>Chandy and Lamport were the first to propose an algorithm, back in 1985, to record such a consistent snapshot in a distributed system. As a side-note, Dr. Leslie Lamport is now employed by Microsoft, although I can only guess if he had anything to do with presenting consistent cut in the form of a consistency level choice.</p><h3>Probabilistic Bounded Staleness</h3><p>Peter Bailis, et al., came up with <a href="http://www.bailis.org/papers/pbs-vldb2012.pdf">Probabilistic Bounded Staleness (PBS)</a> in their paper — “Probabilistic Bounded Staleness for Practical Partial Quorums”. PBS gives you the consistency probability of any given commit in a distributed system. You can see a cool <a href="http://pbs.cs.berkeley.edu/#demo">demo</a> of PBS in action. What you get from PBS is a chart like the following:</p><figure><img alt="Source: http://pbs.cs.berkeley.edu/#demo" src="https://cdn-images-1.medium.com/max/971/1*vmaGaXSu6ohXVOU4JUJNyg.png" /><figcaption>Source: <a href="http://pbs.cs.berkeley.edu/#demo">http://pbs.cs.berkeley.edu/#demo</a></figcaption></figure><p>In the above chart, at 25 ms of a given eventual commit, the probability of reading the commit goes to 100%.</p><blockquote>You have at least a 90.64 percent chance of reading the last written version 10 ms after it commits.</blockquote><p>If we are able to get a consistent cut for our system, we can also determine a consistency metric that tells us what is the most recent consistent cut available to us. Let’s call this — full consistency lag (FCL). Assuming no clock skew, FCL can provide us with a consistent prefix guarantee. An extension of this is <strong>bounded staleness</strong> where reads may lag behind by, at most, <em>K</em> updates or <em>T </em>interval, but are guaranteed to form a consistent cut, i.e., guarantee total global order without any gaps.</p><h3>Use-cases for consistent prefix and tracking staleness in eventually consistent systems</h3><p>Tracking staleness allows you to provide guarantees to your clients that would have been otherwise hard to do. Equipped with the staleness guarantee, clients can solve issues without always resorting to strong consistency. Here are some of the common use cases that call for a consistent cut or a staleness guarantee.</p><h4>When you need total causal order</h4><p>With consistent cuts, you get the throughput of eventually consistent writes, but your reads will only show you a consistent state at the expense of it being slightly stale. This property is useful for any use case where globally consistent causal ordering matters, even if it isn’t strictly real-time.</p><p>Say I would like to track the score of a soccer game¹ and I am ok for it to have a little lag. However, I would never want it to lie to me. Now, consider the following sequence of the score: 0–0 → 1–0 → 2–0 → 2–1. While it’s ok to report that order at whatever lag, it is not ok to report something like this: 0–0 → <strong>0–1</strong> → 2–1. The final score of 2–1 is <em>eventually consistent</em> with reality, but it is unforgivable for a soccer fan to see a score of 0–1 that never happened. There was no point in the game when the score was 0–1, yet we reported it as such. This calls for a consistent snapshot or a consistent cut of a system, even if that snapshot is stale. A fan following the score on her phone may have a 10-second delay but knows that the score is always true, in the right order, and every subsequent read is increasingly up to date. To be complete, the last property is known as <strong>monotonic reads</strong>.</p><p>¹Inspired by the baseball example from Azure’s <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2011/10/ConsistencyAndBaseballReport.pdf">paper</a> and changed it to soccer for simplicity sake.</p><h4>Synchronize cross-region events</h4><p>Consider cache invalidation in multiple data centers. To invalidate a global cache after an update is made to one data center, it would have to let other data centers know to invalidate their caches. If the update is eventually consistent, it is possible to get into a race condition where a data center would invalidate its cache before the update has reached its data centers and miss the update resulting in a stale cache. If full consistency lag is known, data centers will delay invalidating their caches to make sure they have the most recent update. This way clients can keep a global cache synced up without resorting to strong consistency.</p><p>Note that the above example of invalidating caches is really an instance of synchronizing events across data-centers without taking the performance hit of strong consistency.</p><h4>Compaction use case</h4><p>If you have multi-master writes, chances are you need a consistent cut for some of your use cases.</p><blockquote>Multiple processes in any region can do this compaction in a non-blocking way only if they are assured of a consistent cut, even if it’s stale.</blockquote><p>Consider a multi-region distributed inventory system — EU and US. Both regions are selling the same product and write to their respective regions in a multi-master way. In order to have non-blocking writes, each writer simply writes an update of a new stock reservation as orders stream in. For example:</p><p>{“reserve_stock_EU”: 1}<br>{“reserve_stock_US”: 2}<br>{“reserve_stock_EU”: 3}</p><p>As the number of these records can increase pretty rapidly, the read latency for aggregations can go prohibitively high. So, we want to periodically consolidate the above, and replace the above documents with the following single “compaction” document and aggregate the counts per region:</p><p>{<br> “total_reserved_count_US”: 2,<br> “total_reserved_count_EU”: 4, <br> “<strong>compacted_updates</strong>”: 3<br>}</p><p>We store the count of all updates that are compacted in “compacted_updates” field. This helps us to choose the most recent compaction record as we will see soon.</p><p>Multiple processes in any region can do this compaction in a non-blocking way only if they are assured of a consistent cut, even if it’s stale. Since all of the compactors work with a consistent cut, the records are read in the same global order without any gaps. It is possible that one cut is more recent than the other. In the event of multiple compaction records, we simply choose the one with the greatest number of compacted_updates.</p><h3>How to implement consistent prefix guarantee in Cassandra</h3><p>Cassandra is a Dynamo-style system, meaning it follows, loosely, the concepts laid out in the <a href="https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf">Dynamo paper</a>. To provide a consistent prefix guarantee, we would need a way to guarantee that a given subset of multi-master writes are causally ordered with no gaps. This requires a transaction log of updates, so we can cut off at a place where we are certain that there are no gaps. An easy way to model this in Cassandra is to create columns where the column names are timestamps, and the values are the updates. Equipped with this model and assuming no clock skew, we have the following two options to implement a consistent prefix guarantee in Cassandra:</p><h4>Use PBS</h4><p>Once we get the lag we need that gets close to 100% consistency, we only consider column names whose timestamps are older.</p><h4>Exploit Hinted Handoffs</h4><p>Cassandra implements the notion of hinted handoffs as detailed in Amazon’s Dynamo paper as a way of handling failure. This is what Cassandra leverages for fault-tolerant replication. If a write is made to a replica node that is unavailable, then Cassandra will write a “hint” to the coordinator node and try again later in a configured amount of time.</p><p>The main idea, here, is to poll all the nodes to see if they have any pending hints for other nodes. The time when they all report zero (<em>tₐ</em>) is when we know that there are no failed writes, and the only pending writes are those that are in flight. So, subtracting the Cassandra timeout (rpc_timeout) from <em>tₐ </em>gives us our full consistency timestamp (FCT). Any column name that is older than the FCT is considered in the consistent cut.</p><h3>Conclusion</h3><p>A consistent cut or a distributed global snapshot is usually thought of in the context of backups or disaster recovery. However, for a multi-master eventually consistent system, a consistent cut or a “consistent prefix” is imperative for satisfying a lot of use cases, without compromising write throughput.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fd4dd992214" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>