<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prateek Gupta on Medium]]></title>
        <description><![CDATA[Stories by Prateek Gupta on Medium]]></description>
        <link>https://medium.com/@prateek-gupta?source=rss-71aec4827d61------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*I2J81Wv78KwWrf3KDd36Yg.jpeg</url>
            <title>Stories by Prateek Gupta on Medium</title>
            <link>https://medium.com/@prateek-gupta?source=rss-71aec4827d61------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:14:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@prateek-gupta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kafka — Practical guidelines and Monitoring]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://faun.pub/kafka-practical-guidelines-and-monitoring-d1bbbad263f9?source=rss-71aec4827d61------2"><img src="https://cdn-images-1.medium.com/max/2194/1*atF0taPCEgh8N5XpG7dJHQ.png" width="2194"></a></p><p class="medium-feed-snippet">Designing an ideal kafka architecture for an organisation, I came across various articles to understand standard&#x2026;</p><p class="medium-feed-link"><a href="https://faun.pub/kafka-practical-guidelines-and-monitoring-d1bbbad263f9?source=rss-71aec4827d61------2">Continue reading on FAUN.dev()  »</a></p></div>]]></description>
            <link>https://faun.pub/kafka-practical-guidelines-and-monitoring-d1bbbad263f9?source=rss-71aec4827d61------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1bbbad263f9</guid>
            <category><![CDATA[kafka-producer]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[monitoring-software]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kafka-consumer]]></category>
            <dc:creator><![CDATA[Prateek Gupta]]></dc:creator>
            <pubDate>Mon, 12 Aug 2024 15:02:10 GMT</pubDate>
            <atom:updated>2026-03-17T12:21:43.244Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building products as Tech Lead]]></title>
            <link>https://prateek-gupta.medium.com/building-products-as-tech-lead-f06577a17cec?source=rss-71aec4827d61------2</link>
            <guid isPermaLink="false">https://medium.com/p/f06577a17cec</guid>
            <category><![CDATA[tech-lead]]></category>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[software-design]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Prateek Gupta]]></dc:creator>
            <pubDate>Sun, 10 Sep 2023 08:10:31 GMT</pubDate>
            <atom:updated>2023-09-24T03:20:50.899Z</atom:updated>
            <content:encoded><![CDATA[<p>I have shared quite a few posts about different technologies and software architecture principles. Though I like writing about tech I learn and wish to share, I wanted to share a different post about software engineering this time. So having been more than 3 years of working as a tech lead, I decided to write about a typical journey of a large scale project as a tech lead perspective, the various aspects of it and how much it overlaps with an EM role.</p><p>This post is not aimed at sharing the technical details about the project in specific because of enterprise clauses but about the journey itself.</p><h3>Stages</h3><p>Firstly, a tech lead is owning the end to end cycle of the product. This does not mean I dont have to execute all aspects of it but involve in every aspect and drive all the engineering aspect of things along with stakeholders like devs, product managers, program managers and EM.</p><ol><li><strong>PRD (Product Requirement Doc) Review and Clarifications :</strong> This is the first phase where the EM and lead gets involved. The phase is to understand the requested requirements, vet them as much as possible with questions, identify must haves and good to haves and collaborate to get them tweaked basis existing system capabilities. <strong><em>The intention is to get clarity on requirements, drive product scoping and milestoning and to ensure documentation of as much clarifications as possible so that it becomes a golden source of all decisions between stakeholders and avoid future conflicts.</em></strong> <em>One of the ways I prefer to have here is to attract multiple eyes here so that different questions from multiple views can be asked here. Since its not decided who all will be working here yet, I prefer to align the EM with initial 1 or 2 folks who will be working so they can get an early view and seek questions.</em></li><li><strong>Tech solutioning and review :</strong> This phase intends to prepare and document the technical design. I generally start with a section called <em>Knowledge gathering</em> with to-the-point details of the requirements and the existing systems which can cater a specific functionality or can be extended to be build. <strong><em>This phase is generally driven with multiple meetings with leads of different teams owning the systems that were thought of in the knowledge gathering part.</em> </strong>The intention is to arrive at consensus about the reuse of their systems/functionality or if it needs to be extended at their end as well. Then the aim is to start puting the <em>puzzle pieces together to arrive at the HLD (High level design)</em> with different systems and their interactions together. <strong><em>In case of conflicts, we go back to the PRD and the product manager for aligning the product feature expectations OR tweak the requirement basis the capability</em></strong><em>.</em> With the HLD in place, the technical reviews can be started with other leads about the high level approach and accomodating feedbacks, and parallely working on <em>to gather the finer details of the system i.e. LLD (Low Level Design)</em>. I generally split the LLD details of components in my domain and try to <strong><em>delegate some parts of it to the initial folks</em></strong> involved in the project. This alignment with the EM <strong><em>helps me to mentor and review them to grow in tech design and be accountable for the specific smaller features</em></strong> in the overall product.<em> </em>Even during LLD phase, we come across certain conflicts which we needs to refine with the product manager. <strong><em>With the tech design nearing 60%</em></strong>, I start giving walkthrough of the details with the QA lead for him to start working on the testing strategy and efforts.</li><li><strong>Engineering efforts, Parallelisation and Staffing</strong> : The previous 2 phases generally takes 2 to 4 weeks for a big business requirement. This phase is collaboration with the EM/DOE, program manager and product manager. <strong><em>After roughly 60% — 70% tech design completion</em></strong>, is when I start working on engineering tasks breakdown and putting in the efforts required and getting the same filled in by the leads of the dependent teams. <em>I get an initial review with the EM/DOE for high level efforts needed for the product. </em><strong><em>I wish to callout that I keep some buffers in the timelines to account for unforseen circumstances or rainy days.</em></strong><em> We work to align on the availability of the resources and assignment based on the degree of work parallelisation possible.</em> Post this, I work with the program manager to get a high level Gantt view of the efforts and assignees with expected dates of dev drops. We then review the Gantt with the larger audience like EM/director, dependent teams, product team. <strong><em>Basis the effort size and feedbacks, there are scenarios to optimise efforts, product scope cutting to reduce efforts or delay deliverables to later stage, product milestoning for incremental drops or faster shipping, resource addition to increase parallelisation if possible to shrink the dates.</em></strong> <em>This nearly 1 week, is the most crucial phase of alignment across range of stakeholders and personally the most irritating for me :)</em></li><li><strong>Implementation :</strong> Now that we have the biggest barriers crossed in the previous phases, starts the implementation phase.<em> </em><strong><em>I generally keep a 3–4 days initial time for the devs onboarding to the project where I give a full walkthrough, resolve their questions and incorporate minor changes in the technical design based on their feedback.</em></strong> Then they are good to go with the development. With so many things in my plate, I still keep a specific implementation piece for myself, because at the end I wish to be hands on with the code. <strong><em>Few of the critical part during implementation are handling the dev bottlenecks like pulling of the devs to other highly critical requirement from the existing project, incorrect assumptions of the LLD when extending the implementation, handling delay in dev timelines, requirement disconnect from the PRD. There is a lot to go bad in this phase due to misses in the previous phases OR the assumptions and estimations not going on track with the reality OR just when luck is against you :).</em> </strong><em>Handling crisis, mitigating it and still delivering on the commitments with least delays is what differentiates between individual’s skills.</em></li><li><strong>Team structure :</strong> Generally how I structure the team here is to have leads for frontend and mobile which will guide the frontend folks, while I work with backend devs since I am a backend lead. I share the expectations with FE leads to ensure some common standards across project. The main daily scrum is conducted by the program team but <strong><em>I separately keep a weekly connect with all the devs about the dev issues and how can we resolve them</em></strong> or improvise on a practise for better way of resolving issues. <strong><em>Though its not my job, I conduct one-on-one with the devs of the team every 3 weeks to understand technical aspirations of the folks and to understand what is and what is not working with team in the project</em></strong><em>. </em>All this while, I maintain weekly one-on-ones with my manager to keep him updated on the positives, negatives and track of things to detect or callout early signs of problems if any.</li><li><strong>QA/Testing :—: Deployment : — : Go-To-Market strategy :</strong> As dev reach drops milestones, I work with the QA lead reviewing the testing plan, testing env setup to avoid any surprise hiccups. <strong>Deployment</strong> is where I need to plan and document the deployment checklist for the project milestones ensuring to record all DB schema changes, service deployments, migrations if any, feature flags, monitoring and alerting with time checkpoints along with the devs. <strong>Go-To-Market </strong>is where I have to be involved in understanding the prod bugs in the partial release of the feature in the GTM phase. Things start to be more relaxed but proactiveness to bugs and planning their resolution when the execution for the next set of milestones is WIP.</li></ol><p>Its worth highlighting that the stages above does not have very clear segregation and most of the times overlap since there is always some information that gets missed and needs to be checked again. The ideal way is to avoid moving back in phases but hard to achieve even with best effort estimate.</p><p>This is just a POV of how I have worked across building product. The contribution in building a product lies across stakeholders and attributed to multiple folks across business, engineering, operations and specially the target users. Every lead might be working in a different model and every journey is a learning to hear. Any suggestions and clarifications are welcome.</p><p>Thanks!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f06577a17cec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding OAuth with chatgpt]]></title>
            <link>https://prateek-gupta.medium.com/understanding-oauth-with-chatgpt-32325676f831?source=rss-71aec4827d61------2</link>
            <guid isPermaLink="false">https://medium.com/p/32325676f831</guid>
            <category><![CDATA[oauth2]]></category>
            <category><![CDATA[google-oauth]]></category>
            <category><![CDATA[authorization]]></category>
            <category><![CDATA[login]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Prateek Gupta]]></dc:creator>
            <pubDate>Sun, 25 Jun 2023 07:29:54 GMT</pubDate>
            <atom:updated>2023-07-05T03:46:32.770Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kZJhyZVC9jyzIdOhmjiGWg.png" /></figure><p>OAuth (Open Authorization) is an open standard protocol that allows secure authorization and delegation of access to resources on behalf of a user, without sharing their credentials (such as username and password) with a third-party application.</p><blockquote>I wanted to experiment with a different style of blog this time. As a conversation with chatgpt. I will post summaries of the responses from chatgpt. Here it goes!!!</blockquote><h3><strong><em>What</em></strong> is OAuth?</h3><p>OAuth provides a secure way for users to grant access to their resources on one service without sharing their sensitive information. It is widely adopted and supported by major platforms and service providers, making it a popular choice for enabling third-party integration and authentication.</p><p>The basic flow of OAuth involves three parties: the user (resource owner), the client application (consumer), and the service provider (resource server). Here’s a simplified overview of the OAuth process:</p><ol><li>User initiates an action in the client application that requires access to their resources on the service provider’s platform.</li><li>The client application requests authorization from the user and redirects them to the service provider’s authorization server.</li><li>The user is prompted to authenticate and grant permission to the client application.</li><li>If the user grants permission, the authorization server generates an access token and provides it to the client application.</li><li>The client application presents the access token to the resource server (API) of the service provider.</li><li>The resource server validates the access token and, if valid, allows the client application to access the requested resources on behalf of the user.</li><li>The client application can then perform actions or retrieve data from the resource server using the access token.</li><li>The process may involve additional steps like token refreshing, token revocation, or requesting additional permissions.</li></ol><h3>Major providers of OAuth</h3><p>OAuth is widely supported by major providers and platforms across various industries. Some of the major providers of OAuth include:</p><p>1. Google: Google offers OAuth as part of its Google Sign-In service, allowing users to sign in to third-party applications using their Google accounts. Developers can integrate Google OAuth to access user information, authenticate users, and request permissions for Google APIs.</p><p>2. Facebook: Facebook provides OAuth through its Facebook Login feature. Developers can use Facebook OAuth to allow users to sign in to their applications using their Facebook accounts and access their public profile information, friend lists, and other permissions granted by the user.</p><p>3. Twitter: Twitter offers OAuth for developers to enable users to sign in to their applications using their Twitter accounts. OAuth allows access to a user’s Twitter timeline, tweets, and other related data.</p><p>Many other popular platforms and services, such as Microsoft (Microsoft Identity Platform formerly known as Azure Active Directory), GitHub, LinkedIn(LinkedIn Sign In), Amazon(Amazon Cognito service), Instagram, Dropbox, Slack, and more also support OAuth for user authentication and access delegation.</p><h3>Sample Integration with Google OAuth</h3><p>Sure! Here’s an example of how you can integrate Google OAuth in a Go (Golang) application using the `oauth2` package:</p><p>1. Install the required packages by running the following command:</p><pre>####In command prompt####<br>go get golang.org/x/oauth2<br>go get google.golang.org/api/oauth2/v2</pre><p>2. Create a new Go file, e.g., “google_oauth_example.go”, and import the necessary packages:</p><pre>package main<br>import (<br>  &quot;fmt&quot;<br>  &quot;log&quot;<br>  &quot;net/http&quot;<br>  &quot;os&quot;<br>  &quot;golang.org/x/oauth2&quot;<br>  &quot;golang.org/x/oauth2/google&quot;<br>  &quot;google.golang.org/api/oauth2/v2&quot;<br>)</pre><p>3. Set up the OAuth configuration, including your client ID and client secret obtained from the Google API Console:</p><pre>var (<br>  googleOauthConfig *oauth2.Config<br>  oauthStateString = &quot;randomstate&quot; // random string for OAuth2 state<br>)<br><br>func init() {<br>  googleOauthConfig = &amp;oauth2.Config{<br>  RedirectURL: &quot;http://localhost:8080/callback&quot;,<br>  ClientID: &quot;YOUR_CLIENT_ID&quot;,<br>  ClientSecret: &quot;YOUR_CLIENT_SECRET&quot;,<br>  Scopes: []string{&quot;https://www.googleapis.com/auth/userinfo.email&quot;}, // scope for accessing user&#39;s email<br>  Endpoint: google.Endpoint,<br> }<br>}</pre><p>Make sure to replace “YOUR_CLIENT_ID” and “YOUR_CLIENT_SECRET” with your actual client ID and client secret obtained from the Google API Console. Check the next section for this.</p><p>4. Implement the login and callback handlers:</p><pre>func handleMain(w http.ResponseWriter, r *http.Request) {<br>  var html = `&lt;html&gt;&lt;body&gt;&lt;a href=”/login”&gt;Google Log In&lt;/a&gt;&lt;/body&gt;&lt;/html&gt;`<br>  fmt.Fprintf(w, html)<br>}<br><br>func handleLogin(w http.ResponseWriter, r *http.Request) {<br>  url := googleOauthConfig.AuthCodeURL(oauthStateString)<br>  http.Redirect(w, r, url, http.StatusTemporaryRedirect)<br>}<br><br>func handleCallback(w http.ResponseWriter, r *http.Request) {<br>  code := r.FormValue(&quot;code&quot;)<br>  token, err := googleOauthConfig.Exchange(oauth2.NoContext, code)<br>  if err != nil {<br>    log.Println(&quot;Code exchange failed with error:&quot;, err)<br>    http.Redirect(w, r, &quot;/&quot;, http.StatusTemporaryRedirect)<br>    return<br>  }<br><br>  client := oauth2.NewClient(oauth2.NoContext, googleOauthConfig.TokenSource(oauth2.NoContext, token))<br>  svc, err := oauth2.New(client)<br>  if err != nil {<br>    log.Println(&quot;Failed to create Google OAuth2 service:&quot;, err)<br>    http.Redirect(w, r, &quot;/&quot;, http.StatusTemporaryRedirect)<br>    return<br>  }<br><br>  userinfo, err := svc.Userinfo.Get().Do()<br>  if err != nil {<br>    log.Println(&quot;Failed to retrieve user information:&quot;, err)<br>    http.Redirect(w, r, &quot;/&quot;, http.StatusTemporaryRedirect)<br>    return<br>  }<br>  fmt.Fprintf(w, &quot;Email: %s&quot;, userinfo.Email)<br>}</pre><p>5. Set up the HTTP server and routes:</p><pre>func main() {<br>  http.HandleFunc(&quot;/&quot;, handleMain)<br>  http.HandleFunc(&quot;/login&quot;, handleLogin)<br>  http.HandleFunc(&quot;/callback&quot;, handleCallback)<br>  port := os.Getenv(&quot;PORT&quot;)<br>  if port == &quot;&quot; {<br>    port = &quot;8080&quot;<br>  }<br>  log.Println(&quot;Server starting on port&quot;, port)<br>    log.Fatal(http.ListenAndServe(&quot;:&quot;+port, nil))<br>  }</pre><p>6. Build and run the application:</p><pre>####In command prompt####<br>go build google_oauth_example.go<br>./google_oauth_example</pre><p>7. Access the application in your web browser at “<a href="http://localhost:8080`">http://localhost:8080</a>”. Click on the “Google Log In” link to initiate the OAuth flow.</p><h3>Generate client details in Google API console</h3><p>To generate a client ID and secret from the Google API Console, you’ll need to create a project and enable the necessary APIs. Here’s a step-by-step guide:</p><ol><li>Go to the Google API Console (<a href="https://console.developers.google.com/">https://console.developers.google.com/</a>) and sign in with your Google account.</li><li>Click on the project dropdown menu at the top of the page and select “New Project.” Enter a name for your project and click “Create.”</li><li>Once your project is created, you’ll be redirected to the project dashboard. From the left sidebar, click on “Credentials.”</li><li>On the Credentials page, click on the “Create Credentials” button and select “OAuth client ID” from the dropdown menu.</li><li>Configure the OAuth consent screen by clicking on the “Configure consent screen” button. Enter the necessary details such as the application name and authorized domains. You may also need to specify the scopes that your application requires.</li><li>After configuring the consent screen, you’ll be redirected back to the “Create OAuth client ID” page. Select the application type based on your requirements, such as Web application, Android, or iOS.</li><li>Provide the necessary information for the chosen application type, such as the authorized JavaScript origins and redirect URIs. These URIs are important as they define where the authorization flow should redirect after authentication.</li><li>Click “Create” to generate the client ID and client secret. You will see a pop-up dialog displaying your newly created credentials.</li><li>Take note of the generated client ID and client secret. These will be used in your Go application’s OAuth configuration.</li></ol><h3>Alternatives to OAuth</h3><p>While OAuth is a widely adopted standard for authorization and access delegation, there are a few alternative protocols and frameworks available. Here are a few notable alternatives to OAuth:</p><p>1. OpenID Connect (OIDC): OpenID Connect is built on top of OAuth 2.0 and provides identity layer functionality. It adds an authentication layer to the authorization capabilities of OAuth, allowing applications to obtain user identity information along with access tokens. OIDC provides a standard way to authenticate users and retrieve user profile information using ID tokens.</p><p>2. SAML (Security Assertion Markup Language): SAML is an XML-based protocol used for exchanging authentication and authorization data between an identity provider (IdP) and a service provider (SP). It is commonly used in enterprise scenarios for single sign-on (SSO) and federated identity management. SAML relies on the exchange of digitally signed XML documents to facilitate secure authentication and authorization.</p><p>3. JWT (JSON Web Tokens): JWT is a compact, self-contained token format that can be used for authentication and authorization. It consists of three parts: header, payload, and signature. JWTs are often used as access tokens in API authentication scenarios. They can be signed and encrypted, allowing for secure transmission and verification of claims between parties.</p><p>It’s important to note that while these alternatives exist, OAuth remains the most widely adopted and supported standard for authorization and access delegation. The choice of protocol depends on the specific requirements of your application and the ecosystem in which you are operating.</p><blockquote>We will be discussing the mentioned protocols in the future blogs.</blockquote><h3>Constraints</h3><p>While OAuth is a widely used and effective protocol, it does have some constraints and considerations to keep in mind:</p><p>1. Complexity: OAuth can be complex to implement. The OAuth flow involves multiple steps and interactions between different components, such as the client application, authorization server, and resource server. Understanding and correctly implementing these interactions can be challenging.</p><p>2. Lack of standardization: While OAuth provides a framework for authorization, it doesn’t standardize every aspect of the implementation. Different service providers may have variations in the OAuth implementation, such as differences in supported scopes, endpoints, and error handling. This can lead to additional complexity when integrating with multiple OAuth providers.</p><p>3. Security risks if misconfigured: OAuth requires careful configuration to ensure proper security. Misconfiguration of the client application, redirect URIs, or token storage can lead to vulnerabilities, such as unauthorized access or leakage of sensitive information. It’s crucial to follow best practices and security guidelines to mitigate these risks.</p><p>4. Limited control over user experience: With OAuth, the authentication and authorization process typically takes place on the service provider’s platform, which may be outside of the client application’s control. This can result in a fragmented user experience, as the user is redirected to the provider’s site for authentication and consent. The customization and branding options may be limited in such cases.</p><p>5. Trust dependency on third-party providers: When using OAuth, the client application relies on the security and reliability of the authorization server and resource server (API provider). If any of these components are compromised or unavailable, it can impact the functionality and security of the application. It’s important to choose reputable providers and monitor their status and security practices.</p><p>6. Scope of access: OAuth relies on predefined scopes to grant access to specific resources. This can limit the flexibility for users who might want to grant partial or customized access to their resources. Scopes may not cover all possible use cases, requiring additional customizations or negotiations with the provider.</p><h3>Ending Notes</h3><p>OAuth is a widely adopted and effective protocol for authorization and access delegation. It provides a standard framework that enables secure third-party access to protected resources without sharing credentials.</p><blockquote>Reiterating, the information that I have documented is driven from chatgpt</blockquote><p>I might have assumed a few terminologies that the readers would be aware. Please let me know in case of any discrepacy. I will try to respond and rectify.</p><p>Thanks!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32325676f831" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Patterns in Distributed Systems — Consensus for data reliability]]></title>
            <link>https://prateek-gupta.medium.com/patterns-in-distributed-systems-consensus-for-data-reliability-adcfadbd9aa8?source=rss-71aec4827d61------2</link>
            <guid isPermaLink="false">https://medium.com/p/adcfadbd9aa8</guid>
            <category><![CDATA[reliability-engineering]]></category>
            <category><![CDATA[fault-tolerance]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[consensus-algorithm]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Prateek Gupta]]></dc:creator>
            <pubDate>Sun, 26 Mar 2023 14:07:53 GMT</pubDate>
            <atom:updated>2023-03-28T04:29:37.425Z</atom:updated>
            <content:encoded><![CDATA[<h3>Patterns in Distributed Systems — Consensus for data reliability</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*RWx4BAFbBSRxeQrjgOsD3g.png" /></figure><p>Today’s enterprise architecture is full of platforms and frameworks which are distributed by nature. All these are ‘distributed’ by nature. What does it mean for a system to be distributed? There are two aspects:</p><ul><li>They run on multiple nodes. The number of nodes in a cluster can vary from as few as two servers to a few thousand servers.</li><li>They manage data. So these are inherently ‘stateful’ systems.</li></ul><p>There are several ways in which things can go wrong when multiple servers are involved in storing data. All the distributed systems need to solve those problems. The implementation of these systems have some recurring solutions to these problems. Understanding these solutions in their general form helps in understanding the implementation of the broad spectrum of these systems and can also serve as a good guidance when new systems need to be built.</p><h3>Thoughts on the system architecture</h3><p>Softwares can be arranged in different ways. Following are the few ways I have discussed.</p><p><strong>Peer to peer architecture<br></strong>In case of p2p architecture, data or information is replicated across all the nodes in the system. Every node in the system is free to take a call on what to do with the data and commit the modified information on the network for other peers to listen and update its copy of the data. <strong>In this architecture style, the inbound requests from the external systems can land on any of the participant nodes and they have to be serviced by that node in coordination with the other nodes of the system</strong>. <strong>Algorithms discussed later which are part of this are — Quorum and gossip</strong>.</p><p><strong>Leader follower architecture<br></strong>In case of a leader follower architecture, data is replicated across all the nodes in the system as well. But there is a coordinator node which is the face of the system for external systems. Its job is to coordinate consensus among the participating nodes, proceed with the decision forming based on algorithms and then propagate the decision to the participating nodes. <strong>This is node where the inbound requests from the external systems will land for their demands of the information</strong>. <strong>This is single point of failure but it somewhat reduces the overhead of a peer to peer architecture</strong>. In case of leader failure, a new leader needs to be elected among the available nodes for continuity of the system. <strong>Algorithms that will be discussed later for this type are — raft, paxos</strong></p><p>This section plays a significant role in getting an understanding of the essence of the topic of consensus and algorithms that will be discussed in the further sections.</p><h3>What is consensus?</h3><p><strong>Consensus</strong> is one of the most important and fundamental problems in distributed computing. The goal of consensus algorithms is pretty simple: get multiple machines (a.k.a nodes) to <strong>agree</strong> on something. As a simple definition, we can say that a distributed system is s set of nodes(generally computers) that are accomplishing a common task concurrently by communicating with each other over a network. In reality, this is quite hard to achieve due to various failure scenarios (e.g., network partition).</p><p><a href="https://medium.com/@isuruboyagane.16/what-is-consensus-in-distributed-system-6d51d0802b8c">What is Consensus 🤝 in Distributed System?</a></p><h3>Why consensus?</h3><p>In a distributed scenario, decisions needs to be taken when there are multiple options available, we need votes from all or atleast some parties. Basis the votes, we can come to a conclusion which is the most optimised approach to move forward with. Kind of like a democracy.</p><p>With respect to distributed systems, the major decisions that needs to be called for in such a scenario are :<br>1. Leader among the participating nodes.<br>2. Commit data to a threshold of nodes<br>3. Read most recent data from a threshold of nodes.</p><p>Consensus helps the system to provide <strong>consistency guarantees in the scenarios of partition tolerance</strong>. To arrive at a consensus, there are some well known algorithms that needs to be deployed. These algorithms are used to provide the details that a system is either strongly consistent, weakly consistent or eventually consistent.</p><p>Common examples seen in popular enterprise systems include <a href="https://zookeeper.apache.org/">Zookeeper</a>, <a href="https://etcd.io/">etcd</a> and <a href="https://www.consul.io/">Consul</a>. They implement consensus algorithms such as <a href="https://zookeeper.apache.org/doc/r3.4.13/zookeeperInternals.html#sc_atomicBroadcast">zab</a> and <a href="https://raft.github.io/">Raft</a> to provide replication and strong consistency. There are other popular algorithms to implement consensus; <a href="https://www.youtube.com/watch?v=JEpsBg0AO6o&amp;t=1920s">multi-paxos</a> which is used in Google’s <a href="https://research.google/pubs/pub27897/">Chubby</a> locking service, view stamp replication and <a href="https://www.cs.cornell.edu/ken/History.pdf">virtual-synchrony</a>. In very simple terms, Consensus refers to a set of servers which agree on stored data, the order in which the data is stored and when to make that data visible to the clients.</p><blockquote>I generally refer to <a href="https://martinfowler.com/articles/patterns-of-distributed-systems/#ProblemsAndTheirRecurringSolutions">Martin Fowler’s articles</a> for my understanding</blockquote><h3>Algorithms</h3><p>The algorithms that I am going to discuss in this post are as follows :<br>1. Two phase commit<br>2. Paxos<br>3. Quorum<br>4. Raft<br>5. Gossip</p><p>I have written medium articles previously for the above algorithms. The next section shares the same for in-depth understanding.</p><h4>2PC — Two Phase Commit</h4><p>This topic has been discussed in depth in my previous post. Link.</p><p><a href="https://medium.com/nerd-for-tech/transactions-in-software-systems-3dfeb77fd6f2">Transactions in software systems</a></p><h4>Paxos</h4><p>This topic has been discussed in depth in my previous post. Link.</p><p><a href="https://prateek-gupta.medium.com/paxos-algorithm-a71d6fe7be97">Paxos algorithm</a></p><h4>Quorum</h4><p>This topic has been discussed in depth in my previous post. Link.</p><p><a href="https://prateek-gupta.medium.com/quorum-consensus-56bc1bacb0d2">Quorum Consensus</a></p><h4>Raft</h4><p>This topic has been discussed in depth in my previous post. Link.</p><p><a href="https://prateek-gupta.medium.com/raft-consensus-algorithm-fc2de6852d9">Raft Consensus algorithm</a></p><h4>Gossip</h4><p>This topic has been discussed in depth in my previous post. Link.</p><p><a href="https://prateek-gupta.medium.com/gossip-protocol-in-distributed-systems-e2b0665c7135">Gossip Protocol in distributed systems</a></p><blockquote>Sharing an great article I have come across which explains a comparative study of major distributed algorithms. I hope your find this relevant.</blockquote><p><a href="https://yernsun.medium.com/raft-and-its-three-subproblems-b721a23a92b4">Raft and its three subproblems</a></p><p>This post is a collation of my previous posts about the consensus strategies to handle fault tolerance in distributed systems.</p><p>I hope the information that I have described has been helpful for you to grab the concepts of fault-tolerant distributed systems. I might have assumed a few terminologies that the readers would be aware. In such a case, I would suggest to please comment for any additional concepts that would be helpful so I can add a post about them as well.</p><p>Please let me know in case of any discrepacy. I will try to respond asap and rectify. Thanks!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=adcfadbd9aa8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Raft Consensus algorithm]]></title>
            <link>https://prateek-gupta.medium.com/raft-consensus-algorithm-fc2de6852d9?source=rss-71aec4827d61------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc2de6852d9</guid>
            <category><![CDATA[architecture-pattern]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[consensus-algorithm]]></category>
            <category><![CDATA[raft]]></category>
            <category><![CDATA[fault-tolerance]]></category>
            <dc:creator><![CDATA[Prateek Gupta]]></dc:creator>
            <pubDate>Sun, 26 Mar 2023 14:01:17 GMT</pubDate>
            <atom:updated>2023-03-28T04:29:39.917Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*Lmr6-F9EzX0hPyCADVonNA.png" /></figure><h3>Problem</h3><p>Raft is a consensus algorithm that is designed to be easy to understand. It’s equivalent to Paxos in fault-tolerance and performance. The shared status or data is usually a data structure supported by a replicated log. We need the system to be fully operational as long as a majority of the servers are up.</p><h3>Solution</h3><p>Raft works by electing a leader in the cluster. The leader is responsible for accepting client requests and managing the replication of the log to other servers. The data flows only in one direction: from leader to other servers.</p><p>Raft decomposes consensus into three sub-problems:</p><ul><li><strong>Leader Election </strong>: A new leader needs to be elected in case of the failure of an existing one.</li><li><strong>Log replication</strong> : The leader needs to keep the logs of all servers in sync with its own through replication.</li><li><strong>Safety </strong>: If one of the servers has committed a log entry at a particular index, no other server can apply a different log entry for that index.</li></ul><p>In normal operation there is exactly one leader and all of the other servers are followers. Followers are passive: they issue no requests on their own but simply respond to requests from leaders and candidates. The leader handles all client requests (if a client contacts a follower, the follower redirects it to the leader). The third state, candidate, is used to elect a new leader.</p><p>Raft divides time into <strong>terms<em> </em></strong>of arbitrary length, each beginning with an election. If a candidate wins the election, it remains the leader for the rest of the term. In the case of a split vote, that term ends without a leader.</p><p>The <strong>term number</strong> increases monotonically. Each server stores the <strong>current term number</strong><em> </em>which is also exchanged in every communication.</p><p>Raft makes use of two remote procedure calls (RPCs) to carry out its basic operation.</p><ul><li>RequestVotes is used by candidates during elections</li><li>AppendEntries is used by leaders for replicating log entries and also as a heartbeat (a signal to check if a server is up or not — it doesn’t contain any log entries)</li></ul><h4>Leader election</h4><p>The leader periodically sends a heartbeat to its followers to maintain authority. A leader election is triggered when a follower times out after waiting for a heartbeat from the leader. This follower transitions to the candidate state and increments its <strong>term number</strong>. After voting for itself, it issues RequestVotes RPC in parallel to others in the cluster. Three outcomes are possible:</p><ol><li>The candidate receives votes from the majority of the servers and becomes the leader. It then sends a heartbeat message to others in the cluster to establish authority.</li><li>If other candidates receive AppendEntries RPC, they check for the<em> </em>term number. If the term number is greater than their own, they accept the server as the leader and return to follower state. If the term number is smaller, they reject the RPC and still remain a candidate.</li><li>The candidate neither loses nor wins. If more than one server becomes a candidate at the same time, the vote can be split with no clear majority. In this case a new election begins after one of the candidates times out.</li></ol><h4>Log Replication</h4><p>The client requests are assumed to be write-only for now. Each request consists of a command to be executed ideally by the replicated state machines of all the servers. When a leader gets a client request, it adds it to its own log as a new entry. Each entry in a log:</p><ul><li>Contains the client specified command</li><li>Has an index to identify the position of entry in the log</li><li>Has a <strong>term number</strong> to logically identify when the entry was written</li></ul><p>It needs to replicate the entry to all the follower nodes in order to keep the logs consistent. The leader issues AppendEntries RPCs to all other servers in parallel. The leader retries this until all followers safely replicate the new entry.</p><p>When the entry is replicated to a majority of servers by the leader that created it, it is considered committed. All the previous entries, including those created by earlier leaders, are also considered committed. The leader executes the entry once it is committed and returns the result to the client.</p><p>The leader maintains the highest index it knows to be committed in its log and sends it out with the AppendEntries RPCs to its followers. Once the followers find out that the entry has been committed, it applies the entry to its state machine in order.</p><p>When sending an AppendEntries RPC, the leader includes the <strong>term number</strong> and index of the entry that immediately precedes the new entry. If the follower cannot find a match for this entry in its own log, it rejects the request to append the new entry.</p><p>This consistency check lets the leader conclude that whenever AppendEntries returns successfully from a follower, they have identical logs until the index included in the RPC.</p><h4>Safety</h4><p>Raft makes sure that the leader for a term has committed entries from all previous terms in its log. This is needed to ensure that all logs are consistent and the state machines execute the same set of commands.</p><p>During a leader election, the RequestVote RPC includes information about the candidate’s log. If the voter finds that its log it more up-to-date that the candidate, it doesn’t vote for it.</p><h3>Highlights of Raft</h3><ul><li>Raft decomposes consensus into three sub-problems — leader election, log replication and safety</li><li>Raft makes use of two remote procedure calls — RequestVotes and AppendEntries</li><li>The leader maintains a nextIndex for each follower, which is the index of the next log entry the leader will send to that follower.</li><li>Log entries are replicated to all servers</li><li>A leader does not need to take any special actions to restore log consistency when it comes to power. It just begins normal operation, and the logs automatically converge in response to failures of the Append-Entries consistency check.</li><li>A leader never overwrites or deletes entries in its own log.</li><li>Raft determines which of two logs is more up-to-date by comparing the index and term of the last entries in the logs. If the logs have last entries with different terms, then the log with the later term is more up-to-date.</li></ul><h3>Epic Visualization</h3><p>The following links are by far the best vizualization that I have come across for any distributed systems algorithm.</p><blockquote><a href="https://raft.github.io/"><strong><em>Link 1</em></strong></a><strong><em> and </em></strong><a href="http://thesecretlivesofdata.com/raft/"><strong><em>Link 2</em></strong></a><strong><em> explaining distributed raft</em></strong></blockquote><h3>Ending Notes</h3><p>Raft is a pretty great way to achieve consensus presented in the paper <a href="https://www.usenix.org/system/files/conference/atc14/atc14-paper-ongaro.pdf">In Search of An Understandable Consensus Algorithm</a> by Diego Ongaro and John Ousterhout. I have tried to collect simplistic details about the algorithm and more details can be referenced from google and from some sources below that helped me.</p><p>Let me know if there is any discrepancy in the article. I will respond and rectify asap. Thanks!!!</p><p>Credits<br><a href="https://www.freecodecamp.org/news/in-search-of-an-understandable-consensus-algorithm-a-summary-4bc294c97e0d/">Freecodecamp</a><br><a href="https://www.youtube.com/watch?v=LAqyTyNUYSY&amp;ab_channel=LinkedInEngineering">Linkedin engg</a> <br><a href="https://raft.github.io/">Visualization</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc2de6852d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quorum Consensus]]></title>
            <link>https://prateek-gupta.medium.com/quorum-consensus-56bc1bacb0d2?source=rss-71aec4827d61------2</link>
            <guid isPermaLink="false">https://medium.com/p/56bc1bacb0d2</guid>
            <category><![CDATA[quorum]]></category>
            <category><![CDATA[system-architecture]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[quorum-blockchain]]></category>
            <category><![CDATA[consensus-protocol]]></category>
            <dc:creator><![CDATA[Prateek Gupta]]></dc:creator>
            <pubDate>Sun, 26 Mar 2023 06:18:29 GMT</pubDate>
            <atom:updated>2023-03-28T04:29:46.097Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/413/1*aaHqWT7QIUV8UwMHrEYsfg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*iBy3Z4PPyxUYhTHWVu4GNg.png" /></figure><h3>Problem</h3><p>In a distributed system, whenever a server takes any action, it needs to ensure that in the event of a crash the results of the actions are available to the clients. This can be achieved by replicating the result to other servers in the cluster. But that leads to the question: how many other servers need to confirm the replication before the original server can be confident that the update is fully recognized.</p><p>If the original server waits for too many replications, then it will respond slowly. This will in turn, cause the liveness of the system impacted because any node that is suffering outage will impact the original server to not commit the transaction successfully. But if it doesn’t have enough replications, then the update could be lost. This will in turn cause failure of safety since the data committed in the system can get lost. It’s critical to balance between the overall system performance and system continuity.</p><h3>Solution</h3><p>A quorum is the minimum number of nodes in the system that have committed a transaction. In simple term, it is the majority or 1 + 1/2 of the total nodes in the system. But the quorum value can be defined by the user in the system setup basis their availability and consistency requirements.</p><p>There can be different read and write quorum values. This would again depend on the system non-functional requirements like latency, consistency and freshness of data.</p><h3>The write operation</h3><p>A write requires either one or two phases. These phases require communication with a quorum, and the phase is complete when a quorum of responses has arrived at the client side.</p><ol><li>The request node sends a read request for the version number to all the replicas, and waits for replies from a read quorum. Then it takes the biggest version number.</li><li>The request node sends a write request containing the state and new version number bigger than the one received in the first step to all the replicas and waits to receive acknowledgements from a write quorum. At that point the write operation is complete.</li></ol><blockquote>If there are concurrent writers, the one choosing the largest version number will prevail. The protocol doesn’t provide any synchronization</blockquote><h3>The Read Operation</h3><p>To perform a read requires either one or two phases.</p><ol><li>The request node sends a request to read the version number and state to all the replicas and waits for replies from a read quorum. If all version numbers it receives are the same, this is the happy case.</li><li>Else the request node sends the largest version number and the associated value to all the replicas, and waits for responses from a write quorum. Once the quorum of writes has arrived, the request node is good to respond to the caller.</li></ol><p>The second phase of read is called the “write-back” phase. It is needed when the read is concurrent with a write but not a must have since the read operation was completed in the first step itself. It is also useful to handle the case of a previously failed write.</p><h3>Example</h3><p>There are N replicas in a group. A replica group stores a collection of items. Each item has an ID and a state. Each replica stores the state for each item, plus an extra piece of information: a version number.</p><p>The idea is that if different replicas store different version numbers for an item, the state associated with a larger version number is more recent than the state associated with a smaller version number.</p><p>Reads go to a read quorum of size R and writes go to a write quorum of size W.</p><blockquote>There is a notion that read quorums must always intersect with write quorums. This will ensure that read results always reflect the result of the most recent write (because the read quorum will include at least one replica that was involved in the most recent write).</blockquote><p>For example, consider a group of 3 replicas. Then we have the following possibilities:</p><ul><li>R=3 and W=1. This improves performance for writes at the expense of reads, which is probably a bad idea since generally reads are more common than writes. In addition, this choice of quorums is bad because a write might happen at a single replica that then fails. If that replica were to lose its state, the outcome of the write would be lost. So generally we would like to have W&gt;1.</li><li>R=1 and W=3. This works very well for reads which is generally good since reads are common. But it is undesirable for writes because if one of the replicas is down or inaccessible, a write cannot complete until that replica recovers.</li><li>R=2 and W=2. This choice is a good compromise compared to the R=1 and W=3 choice, since it increases the cost of reads in return for providing reasonable availability for writes.</li></ul><blockquote><strong><em>Distributes systems examples that needs a quorum</em></strong></blockquote><blockquote><strong>Updating data in a cluster of servers<br></strong>High-Water Mark is used to ensure only data which is guaranteed to be available on the majority of servers is visible to clients.</blockquote><blockquote><strong>Leader election — A </strong>leader is selected only if it gets votes from a majority of the servers.</blockquote><h3>Highlights of Quorum</h3><ul><li>All the consensus implementations like <a href="https://zookeeper.apache.org/doc/r3.4.13/zookeeperInternals.html#sc_atomicBroadcast">Zab</a>, <a href="https://raft.github.io/">Raft</a>, <a href="https://en.wikipedia.org/wiki/Paxos_(computer_science)">Paxos</a> are quorum based.</li><li>This protocol provides atomicity. It guarantees that a read will see a state at least as recent as that produced by the most recently completed write that completed before the read started. It also guarantees that if some reader sees the results of a particular write, than any reader that starts after that reader finishes will also see a result at least that recent.</li><li>Linearizability is another name for quorum.</li><li>Even in systems which don’t use consensus, quorum is used to make sure the latest update is available to at least one server in case of failures or network partition. For instance, in databases like <a href="http://cassandra.apache.org/">Cassandra</a>, a database update can be configured to return success only after a majority of the servers have updated the record successfully.</li></ul><h3>Ending Notes</h3><p>Quorum is one of the most widely used concept that is being used in the daily softwares that many of us use but are not aware about. Every distributed systems needs some kind of Quorum for conflict resolution.</p><p>I have tried to explain it in a straightforward manner. Let me know if there seems to be discrepancy and I will try to address it asap. Always looking forward for improvements. Thanks!!!</p><p>Credits -<br><a href="https://martinfowler.com/articles/patterns-of-distributed-systems/quorum.html">Martin Fowler’ Distributed system</a><br><a href="http://web.mit.edu/6.033/2005/wwwdocs/quorum_note.html">MIT Quorum notes</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56bc1bacb0d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Paxos algorithm]]></title>
            <link>https://prateek-gupta.medium.com/paxos-algorithm-a71d6fe7be97?source=rss-71aec4827d61------2</link>
            <guid isPermaLink="false">https://medium.com/p/a71d6fe7be97</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[spanner]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[paxos]]></category>
            <dc:creator><![CDATA[Prateek Gupta]]></dc:creator>
            <pubDate>Sun, 12 Jun 2022 05:41:08 GMT</pubDate>
            <atom:updated>2022-06-19T12:09:12.418Z</atom:updated>
            <content:encoded><![CDATA[<h3>Paxos algorithm in distributes systems</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LKlp4A9YwBtMdyRVSdxiIw.png" /></figure><h3>Problem</h3><p>When multiple nodes share state, they often need to agree between themselves on a particular value. With <a href="https://martinfowler.com/articles/patterns-of-distributed-systems/leader-follower.html">Leader and Followers</a>, the leader decides and passes its value to the followers. But if there is no leader, then the nodes need to determine a value themselves. (Even with a leader-follower, they may need to do this to elect a leader.)</p><p>A leader can ensure replicas safely acquire an update by using <a href="https://martinfowler.com/articles/patterns-of-distributed-systems/two-phase-commit.html">Two Phase Commit</a>, but without a leader we can have competing nodes attempt to gather a <a href="https://martinfowler.com/articles/patterns-of-distributed-systems/quorum.html">Quorum</a>. This process is further complicated because any node may fail or disconnect. A node may achieve quorum on a value, but disconnect before it is able to communicate this value to the entire cluster.</p><h3>Solution</h3><p>The Paxos algorithm was developed by <a href="http://lamport.org/">Leslie Lamport</a>, published in his 1998 paper <a href="http://lamport.azurewebsites.net/pubs/pubs.html#lamport-paxos">The Part-Time Parliament</a>. Paxos works in three phases to make sure multiple nodes agree on the same value in spite of partial network or node failures. The first two phases act to build consensus around a value, the last phase then communicates that consensus to the remaining replicas.</p><ul><li>Prepare phase: establish the latest <a href="https://martinfowler.com/articles/patterns-of-distributed-systems/generation.html">Generation Clock</a> and gather any already accepted values.</li><li>Accept phase: propose a value for this generation for replicas to accept.</li><li>Commit Phase: let all the replicas know that a value has been chosen.</li></ul><p>In the first phase (called <strong>prepare phase</strong>), the node proposing a value (called a <strong>proposer</strong>) contacts all the nodes in the cluster (called <strong>acceptors</strong>) and asks them if they will promise to consider its value. Once a quorum of acceptors return such a promise, the proposer moves onto the second phase. In the second phase (called the <strong>accept phase</strong>) the proposer sends out a proposed value, if a quorum <a href="https://martinfowler.com/articles/patterns-of-distributed-systems/paxos.html#footnote-flexible-paxos">[1]</a> of nodes accepts this value then the value is <strong>chosen</strong>. In the final phase (called the <strong>commit phase</strong>), the proposer can then commit the chosen value to all the nodes in the cluster.</p><p>In the original paxos papers (<a href="http://lamport.azurewebsites.net/pubs/pubs.html#lamport-paxos">1998</a> and <a href="https://lamport.azurewebsites.net/pubs/paxos-simple.pdf">2001</a>), there is no mention of the commit phase,as the focus of the algorithm is to prove that only a single value is chosen and it’s enough, even if only the proposer cluster node knows about the chosen value. But in practice, all of the cluster nodes need to know about the chosen value and there is a need for a commit phase where the proposer communicates the decision to all of the cluster nodes.</p><h3>Details</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ikqzjF_VQ0o5qHggwA5i5g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WzLEhzPSMkww2W9vuiybGw.png" /><figcaption>Paxos flow</figcaption></figure><h4>Protocol Flow Example as a sequence diagram (<strong>MUST READ)</strong></h4><p><a href="https://martinfowler.com/articles/patterns-of-distributed-systems/paxos.html">Paxos algorithm in 5 distributed nodes</a>. <br>I have read through multiple links online but found this to be the most helpful simply because it explains the flow using sequence diagrams and covering multiple scenarios.</p><p>The Paxos protocol explained here, builds consensus on a single value (often called as single-decree Paxos). Most practical implementations used in mainstream products like <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/introduction">Cosmos DB</a> or <a href="https://cloud.google.com/spanner">Spanner</a> use a modification of paxos called multi-paxos which is implemented as a <a href="https://martinfowler.com/articles/patterns-of-distributed-systems/replicated-log.html">Replicated Log</a>.</p><p><a href="http://cassandra.apache.org/">[cassandra]</a> uses Paxos to implement light-weight transaction.</p><p>All the consensus algorithms like <a href="https://raft.github.io/">Raft</a>(which will be a future post WIP) use basic concepts similar to the basic Paxos. The use of <a href="https://martinfowler.com/articles/patterns-of-distributed-systems/two-phase-commit.html">Two Phase Commit</a>, <a href="https://martinfowler.com/articles/patterns-of-distributed-systems/quorum.html">Quorum</a> and <a href="https://martinfowler.com/articles/patterns-of-distributed-systems/generation.html">Generation Clock</a> is used in a similar manner.</p><h3>Problems</h3><p>There are a few problems which are associated with Paxos algorithm. To understand this, I am assuming you have spent enough time to understand the sample working of the algorithm on either Martin Fowler’s blog or some other material.</p><p><strong>Dueling leaders<br></strong>Imagine a situation where 2 or more nodes initiate proposals at the same time. This leads to a deadlock situation where 2 nodes/replicas are elected as proposers and each proposed a commit with a given number, but failed and both tried with a higher number and failed again? This can potentially be a show stopper and our replicas would potentially never converge.</p><p><strong>Livelock<br></strong>Imagine there were network failures in the cluster such that every acceptor accepted a different value. Without the ability to change their value in future rounds, no progress could ever be made and you have a “livelock”. This is another deadlock situation where replicas are unable to update their values since the proposer is unable to have a common value updated in the proposal which all nodes can agree upon.</p><h3>Multi-paxos : Another version of Paxos</h3><p>Multi-Paxos elects a leader, and the proposal is initiated by the leader. Due to zero competitions, the livelock problem is eliminated. <br>If all proposals are initiated by the leader, the preparation phase can be skipped to change the two-phase process to a one-phase process, which improves the efficiency. <br>Multi-Paxos does not assume a unique leader. Instead, it allows multiple leaders to propose requests concurrently without affecting safety. In extreme cases, it degrades to Basic Paxos.</p><h3>Ending notes</h3><p>Through this post, I have tried to provided a high level understanding of the Paxos algorithm. This is a important algorithm in the world of distributed systems. Different versions or improvised versions are being used in practise as described above i.e. cassandra, spanner and cosmos databases. A few relevant links have been provided but the online references are abundant.</p><p>Hope this helps. Let me know in the comments if you find any discrepancy in the post. Thanks for reading!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a71d6fe7be97" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gossip Protocol in distributed systems]]></title>
            <link>https://prateek-gupta.medium.com/gossip-protocol-in-distributed-systems-e2b0665c7135?source=rss-71aec4827d61------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2b0665c7135</guid>
            <category><![CDATA[peer-to-peer]]></category>
            <category><![CDATA[consensus-algorithm]]></category>
            <category><![CDATA[gossip-protocol]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[cassandra]]></category>
            <dc:creator><![CDATA[Prateek Gupta]]></dc:creator>
            <pubDate>Sat, 23 Apr 2022 14:05:36 GMT</pubDate>
            <atom:updated>2022-04-27T01:12:51.522Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sy9fDAy1KLb7mYflAP_F_A.jpeg" /></figure><h3>Problem</h3><p>In a distributed systems scenarios, the most basic problem that one can think of are :<br>1. System state : How are system nodes aware about the other system nodes state, whether they are dead or alive.<br>2. Communication : How can system nodes interact with other nodes.</p><p>For a system that wish to use a distributed system, the problems above are immaterial and all it cares about is the end result. But for a distributed system, this is a core problem for evaluating the final result be it a database, search engine, object storage, a load balancer or maybe something else.</p><p>To address the problems mentioned, there can be two possible solutions : <br>1. A <strong>centralised system which manages the system state</strong>, for example zookeeper in kafka. The problem with this is <strong>single point of failure</strong>. But the finer side is this setup is inclined to CP side of CAP theorem providing higher consistency guarantee.<br>2. A <strong>peer-to-peer</strong> solution to track system state. This solution is inclined to AP side of CAP theorem providing eventual consistency. But the solution is <strong>highly scalable and more resilient</strong>. Gossip protocol based algorithms fall under this category and is famously used in <strong>cassandra</strong>.</p><p>In a distributed system you need at least two independent sources of information to mark a node down. It’s not enough to simply say because your node can’t contact another node that the other node is down. It’s quite possible that your node is broken and the other node is fine. But if other nodes in the system also see that other node is dead then you can with some confidence conclude that that node is dead. Many complex hard to debug bugs are hidden here. How do you know what other nodes are seeing? Via a gossip protocol exchanging this kind of reachability data.</p><h3>Gossip protocol</h3><p>Gossip is a peer-to-peer<em> </em>communication protocol in which nodes periodically exchange state information about themselves and about other nodes they know about.</p><p>Gossip protocol based algorithms are one of the most robust and scalable algorithms out there for strongly eventual consistent membership list, fault detection, and can piggy back any information on top of the gossip messages.</p><p>Gossip is about letting each node send the latest information it happens to have to some set of other nodes eventually spreading that information throughout the network. It’s a way for nodes to build a global map from limited local interactions.</p><p>Gossip protocols are at best eventually consistent, but aren’t even necessarily that. And if there’s a partition, nodes in sub-partitions will still happily gossip with each other. So a request that hits one side of a partition can get totally different answers than if it hits another side.</p><h4>High level details about the protocol</h4><ul><li>In the cluster, every member keeps a list of a subset of known members and their addresses and some metadata.</li><li>Periodically, each member updates its list of neighbors’ heartbeat counters based the data emitted by different nodes and sends the updated information to some further neighbours.</li><li>Upon receipt of such a gossip message, a node merges the list in the message with its own list and adopts the max heartbeat counter for each member.</li><li>As long as heartbeat counter keeps on increasing for a node, it is guaranteed to be healthy and is considered dead if heartbeat hasn’t increased for more than some threshold time period.</li><li>It seems somewhat obvious that you would transmit node properties to other nodes. Stats like load average, free memory, etc. would allow a local node to decide where to send work, for example. If a node is idle send it work (as long as everyone doesn’t send it work at the same time). This local decision making angle is the key to scale. There’s no centralized controller. Local nodes make local decisions based on local data. This can scale as far as the gossip protocol can scale.</li></ul><h4>Relevant Links for understanding a few gossip based algorithms</h4><p><a href="https://en.wikipedia.org/wiki/Gossip_protocol">Gossip protocol wiki</a><a href="https://cwiki.apache.org/confluence/display/cassandra/ArchitectureGossip"><br>Gossip in Cassandra</a><br><a href="https://dl.acm.org/doi/10.1145/1317379.1317382">Limitations of gossip protocol</a><br><a href="https://managementfromscratch.wordpress.com/2016/04/01/introduction-to-gossip/#strength">An interesting read for low level details of the protocol</a><br><a href="https://www.baeldung.com/cs/distributed-systems-guide">Fundamentals of distributed systems</a></p><h3>Ending notes</h3><p>This post has tried to provided a high level understanding of the gossip protocol. Since there are multiple implementation of the protocol, its not possible to explain each of the variant. The way to go further in the topic is to pick up a specific algorithm and go deep in that. A few relevant links have been provided but the list is huge and is dependent on the specific software using their implementation of the protocol.</p><p>Hope this helps. Let me know in the comments if you find any discrepancy in the post. Thanks for reading!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2b0665c7135" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transactions in software systems]]></title>
            <link>https://prateek-gupta.medium.com/transactions-in-software-systems-3dfeb77fd6f2?source=rss-71aec4827d61------2</link>
            <guid isPermaLink="false">https://medium.com/p/3dfeb77fd6f2</guid>
            <category><![CDATA[saga-pattern]]></category>
            <category><![CDATA[software-translation]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[microservices-pattern]]></category>
            <category><![CDATA[distributed-computing]]></category>
            <dc:creator><![CDATA[Prateek Gupta]]></dc:creator>
            <pubDate>Sat, 19 Feb 2022 14:23:47 GMT</pubDate>
            <atom:updated>2022-03-05T14:44:18.751Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*T6DMcPKiHjyiGdOMteeBZg.jpeg" /></figure><h3>Transaction</h3><p>A simple question to start with : What is a transaction?<br>Simply put, a transaction is an exchange of good or service. Now when applied to a software system, a transaction would be define as an operation to be done on an entity.</p><h4><strong>An entity?</strong></h4><p>An entity is a fundamental unit of a real world thing in a software system. Ex — financial entity like money, asset entity like a property, etc.</p><p>When an operation needs to be done on an entity, it needs to be done with a set of criterion that needs to be met with, which would ensure that the entity state before, during and after the operation is deterministic.</p><p>We generally talk about <strong>transactions</strong> in software systems <strong>in the context of the transactions provided by persistence layer i.e. databases</strong>. This is not the only definition though. This statement is more of the time when relational DB and SQL DB were the only solution and software services were confined to a single monolithic services. The well known properties of relational DB are the ACID properties. The overview of ACID is as follows :</p><ol><li>Atomicity — Transactions either happens successfully or fails. There is no other states in between.</li><li>Consistency — This refers to the integrity of the data so that database is consistent before and after a transaction.</li><li>Isolation — Each transaction executes independently. Multiple transactions occurring do not interfere with each other.</li><li>Durability — Once a transaction is successfully completed, it stays in the system even if there is a system failure.</li></ol><p>Some well known criterion about a transaction I find interesting :</p><ol><li>For a transaction to be completed and database changes to made permanent, a transaction has to be completed in its entirety.</li><li>When a transaction completes successfully, database changes are said to be <em>committed</em>; when a transaction does not complete, changes are <a href="https://searchsqlserver.techtarget.com/definition/rollback"><em>rolled back</em></a>.</li><li>There are different levels of ACID properties that can be configured mainly for consistenct and isolation.</li><li>Transactions can have multistep process for effectively managing them.</li></ol><h3>Monolithic system</h3><p>When your service is a monolithic service using a SQL database, you can pretty much assume the ACID guarantees from the DB. The transaction management can be handled by your code using the DB transaction management and hence your entity states are managed.</p><p>The Biggest advantage for Monolithic application in transaction management is a single and common database server. Transactions can be initiated at the database level and can be committed or rolled back based on the final outcome of the transaction.</p><p>Before we discuss about transactions across distributed boundaries, lets first understand how transactions happen theoretically in SQL servers. SQL Server can operate 3 different transactions modes and these are:</p><ol><li><strong>Autocommit Transaction </strong>mode is the default transaction for the SQL Server. In this mode, each T-SQL statement is evaluated as a transaction and they are committed or rolled back according to their results. The successful statements are committed and the failed statements are rolled back immediately</li><li><strong>Implicit transaction </strong>mode<strong> </strong>enables to SQL Server to start an implicit transaction for every DML statement but we need to use the commit or rolled back commands explicitly at the end of the statements</li><li><strong>Explicit transaction </strong>mode provides to define a transaction exactly with the starting and ending points of the transaction</li></ol><h3>Distributed Systems</h3><p>But when you work in a distributed setup with multiple microservices having their own set of domain and bounded contexts. Microservices guidelines strongly recommend you to use the Single Repository Principle(SRP), which means each microservice maintains its own database and no other service should access the other service’s database directly. The entity and its shared or related data now spans over multiple microservices.</p><h4>Distributed Transaction Approaches in microservices</h4><ul><li>Two-Phase Commit Protocol</li><li>Eventual Consistency and Compensation</li></ul><h4>2PC — 2 phase commit</h4><p>A two-phase commit is a standardized protocol that ensures that a <strong>atomic transaction commit</strong> across multiple nodes is implementing in the situation where a commit operation must be broken into two separate parts. <strong>Saving data changes is known as a commit and undoing changes is known as a rollback.</strong> There are two phases in the algorithm — <strong>prepare </strong>and <strong>commit</strong>. The algorithm comprises of a coordinating node which initiates the transaction :</p><ol><li>Prepare : All the participants of a transaction in this phase will be prepared for the commit and inform the transaction coordinator/message broker that they are ready for completing the transaction</li><li>Commit or Rollback : In this phase, transaction coordinator will issue one of the commands they are a commit or a rollback to all the participants.</li></ol><p>Both the phases can be achieved comparatively easily using transaction logging when a single server is involved, but when the data is spread across geographically-diverse servers in distributed computing (i.e., each server being an independent entity with separate log records), the process can become more tricky.</p><blockquote>One really interesting use case is to implement <strong>exactly-once message processing</strong> using 2PC. Please read <a href="https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">this article: “An Overview of End-to-End Exactly-Once Processing in Apache Flink”</a> if you are interested why two phase commit algorithm must be used to achieve the exactly once semantic.</blockquote><h3>Eventual Consistency and Compensation</h3><p>This model of distributed transaction arises from the concept of relaxing C in CAP theorem. Compromising on consistency and trusting the dependent domains to take the right steps based on the transaction nature. The impact of this is that the data across nodes or domains would be updated but with some delay essentially eliminating the guarantee of the freshness of data. We need to make sure that system should be eventually consistent at some point of time in the future. This model doesn’t force to use ACID transactions across microservice but forces some mechanism for ensuring for consistency.</p><p>Each service which involved in the transaction should be responsible to update the user with the proper status of the transaction even if next consecutive services are failed to respond and should handle them whenever services are up and make sure all the scheduled transactions are completed and data in the system is consistent.</p><h3>Saga Pattern</h3><p>Coming to microservices, 2PC is not an option. We employ something called Saga pattern. Saga pattern defines how interaction between different microservices needs to be carried out for transaction propagation and transactional reversal. The two types of Saga pattern are choreography and orchestration pattern.</p><p>In a saga pattern. we implement each business transaction that spans multiple services. A saga is a sequence of local transactions. Each local transaction updates the database and publishes a message or event to trigger the next local transaction in the saga. If a local transaction fails because it violates a business rule then the saga executes a series of compensating transactions that undo the changes that were made by the preceding local transactions.</p><p><a href="https://microservices.io/patterns/data/saga.html">Microservices Pattern: Pattern: Saga</a></p><p>For understanding Saga pattern better, we will assume the case of an <br>1. Order service <br>2. Customer service<br>3. Fulfillment service.</p><h4>Choreography</h4><p>Choreography pattern says that each pair of interacting microservices is responsible for its transaction commit or rollback and contracts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*mdG_q8lGHqVe91fSegU3dQ.jpeg" /></figure><h4>Orchestration</h4><p>Orchestration pattern has a orchestrator or coordinating service which tells the involved services what is to be done and the order in which it is to be done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*m7Y3igQE8fq-IYfwx_JfXA.jpeg" /></figure><h4>Benefits</h4><ul><li>It enables an application to maintain data consistency across multiple services without using distributed transactions</li></ul><h4>Drawbacks</h4><ul><li>The programming model is more complex.There needs to be compensating transactions that explicitly undo changes made earlier in a saga.</li></ul><p>The post tries to help people gain better understanding of transactions in software systems addressing both the architectural styles. The complexity is very different in both the architectures. The approaches to manage Transaction propagation and reversal with details about the pitfalls and sample examples. The following are the non functional aspects of the microservices architecture which make distributed transactions way more complex :<br>1. The multiple point of failures in the transaction path.<br>2. Multiple modes of inter-service communication.<br>3. Availability of systems in the transaction path.<br>4. Reliability and Resilience of the transaction path.<br>5. Scalability across all points in the transaction path.</p><p>In the next post, I will be discussing about consistency management in distributed systems. As part of this, I will be discussing about some famous consensus protocols like — paxos and gossip, which are used in some famous distributed systems like kafka, cassandra db.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3dfeb77fd6f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Custom Middlewares in Go gin]]></title>
            <link>https://prateek-gupta.medium.com/custom-middlewares-in-go-gin-4a7408e4d472?source=rss-71aec4827d61------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a7408e4d472</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[gin-gonic]]></category>
            <category><![CDATA[middleware]]></category>
            <category><![CDATA[api-interception]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Prateek Gupta]]></dc:creator>
            <pubDate>Thu, 23 Dec 2021 06:41:36 GMT</pubDate>
            <atom:updated>2021-12-29T12:48:08.369Z</atom:updated>
            <content:encoded><![CDATA[<h3>Custom Middlewares in Go gin project</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*C2zST_Bn4rtZ1GLhbC5gxQ.png" /></figure><p>With one of my side projects, I explored a mechanism to move common functionality that needs to be performed for each of the incoming API calls like authentication, tokens, sessions and etc. Middlewares in Go gin framework is a technique to solve this. If I compare the middlewares to a similar thing in JAVA, it would be interceptor. The job of an interceptor is to intercept every incoming call and perform the steps. Middlewares are exactly the same.</p><p>I found this <a href="https://github.com/gin-gonic/contrib">repo</a> which has a collection of some amazing middlewares. If you need any of the available ones, just hook up this repo and start using it.</p><p>But this post is dedicating to create a custom middleware for your specific usecase. Lets start with this.</p><h3>Purpose</h3><p>My middleware has a simple twofold purpose :<br>1. Add a uuid to every incoming request. This uuid will be used for printing logs in the handler flow thus making the request unique and easing traceability and debuggability.<br>2. Capturing the time taken to serve the request.</p><p><strong>Show me the code</strong></p><p>Lets jump to the code. I am putting snippets of the code which is the application that I have mentioned in my previous posts.</p><p><strong>Raw application.go file</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07f41edd77c06a7b063dd99e71f2e405/href">https://medium.com/media/07f41edd77c06a7b063dd99e71f2e405/href</a></iframe><p>Now to add middleware in the file. Just add the code as below.</p><p><strong>applicationUpdated.go file</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46fc61bb8f4e05427cfaec6c480f38c8/href">https://medium.com/media/46fc61bb8f4e05427cfaec6c480f38c8/href</a></iframe><p>Observe the simplicity of adding the <strong>guidMiddleware()</strong> function and embedding it using <strong>router.Use()</strong> method. The middleware function has a <strong>return type of gin.HandlerFunc</strong>.</p><p>Just like you can add as many middlewares and embed it in the gin router. Thats it! This is how easy it is to add a custom middleware to your gin app. Let me know if you face any issue and find any discrepancy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a7408e4d472" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>