<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Somanshu Singla on Medium]]></title>
        <description><![CDATA[Stories by Somanshu Singla on Medium]]></description>
        <link>https://medium.com/@somanshusingla?source=rss-3f64da4146b9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ScrJ61Fsp4hw_fNykHXWsw@2x.jpeg</url>
            <title>Stories by Somanshu Singla on Medium</title>
            <link>https://medium.com/@somanshusingla?source=rss-3f64da4146b9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:39:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@somanshusingla/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Saga Design Pattern for Distributed Transactions]]></title>
            <link>https://somanshusingla.medium.com/the-saga-design-pattern-for-distributed-transactions-aeff14a0c82c?source=rss-3f64da4146b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/aeff14a0c82c</guid>
            <category><![CDATA[design-patterns]]></category>
            <category><![CDATA[saga-pattern]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Somanshu Singla]]></dc:creator>
            <pubDate>Tue, 17 Jun 2025 08:21:06 GMT</pubDate>
            <atom:updated>2025-06-17T08:21:06.790Z</atom:updated>
            <content:encoded><![CDATA[<p>As a Staff Engineer navigating the world of distributed systems, I’ve come to appreciate the elegance and practicality of design patterns that help us build resilient and scalable microservices. Today, I want to dive deep into one such pattern: the <strong>Saga pattern</strong>, particularly relevant when dealing with distributed transactions.</p><p>If you’ve ever wrestled with maintaining data consistency across multiple independent services, you know it can get tricky. Traditional ACID (Atomicity, Consistency, Isolation, Durability) transactions are often difficult to implement in a distributed environment due to the challenges of coordinating commits across different databases and services. This is where the Saga pattern comes to our rescue.</p><h3>What Exactly is a Saga? 🤔</h3><p>Simply put, a saga is a sequence of local transactions. Each transaction within the saga updates data within a single service. The key is that after each local transaction completes, it triggers the next transaction in the sequence. If a transaction fails at any point, the saga executes a series of compensating transactions that undo the changes made by the preceding transactions, aiming to maintain eventual consistency across the system.</p><p>Think of it like a choreographed dance between your microservices. Each service performs its step (local transaction), and signals the next service to perform its step. If someone stumbles, we have a rollback routine (compensating transactions) to try and get back on track.</p><h3>Why Use the Saga Pattern? 💪</h3><p>The Saga pattern offers several compelling advantages in a microservices architecture:</p><ul><li><strong>Local ACID Transactions:</strong> Each service manages its own data and uses local ACID transactions, simplifying database interactions and improving performance.</li><li><strong>Decoupling:</strong> Services remain independent and don’t need to coordinate direct two-phase commits, leading to looser coupling and increased autonomy.</li><li><strong>Resilience:</strong> If one service fails during a saga, only the changes made by that service and the preceding ones need to be compensated, rather than rolling back an entire distributed transaction.</li></ul><h4>A Deeper Dive: ACID vs. BASE</h4><p>To fully appreciate the Saga pattern, it’s essential to understand the two main data consistency models: ACID and BASE.</p><p><strong>ACID (Atomicity, Consistency, Isolation, Durability)</strong></p><p>This is the traditional model for transactions, most often associated with single relational databases (like SQL).</p><ul><li><strong>Atomicity:</strong> Guarantees that all operations within a transaction are completed successfully as a single, indivisible unit. If any part of the transaction fails, the entire transaction fails, and the database is left unchanged. It’s “all or nothing.”</li><li><strong>Consistency:</strong> Ensures that a transaction can only bring the database from one valid state to another. It prevents data corruption by enforcing all rules and constraints.</li><li><strong>Isolation:</strong> Ensures that concurrently executing transactions do not interfere with each other. The result is as if the transactions were executed sequentially.</li><li><strong>Durability:</strong> Guarantees that once a transaction has been committed, it will remain committed even in the event of a system failure (like a crash or power outage).</li></ul><p>ACID provides a strong guarantee of reliability and data integrity, but this comes at the cost of availability and scalability in a distributed system, as coordinating a two-phase commit across multiple services is slow and complex.</p><p><strong>BASE (Basically Available, Soft state, Eventual consistency)</strong></p><p>The BASE model, born from the realities of large-scale distributed systems, accepts a lesser degree of consistency to achieve higher availability and scalability.</p><ul><li><strong>Basically Available:</strong> The system guarantees availability. It will respond to requests, even if it’s with stale data or an error, rather than blocking or failing entirely.</li><li><strong>Soft State:</strong> The state of the system may change over time, even without input, due to the “eventual consistency” model.</li><li><strong>Eventual Consistency:</strong> This is the cornerstone of BASE. It guarantees that if no new updates are made to a given item, eventually all accesses to that item will return the last updated value. The data will become consistent “eventually,” but there’s no guarantee of when.</li></ul><p>The Saga pattern is an embodiment of the BASE model. It forgoes the strict, immediate consistency of ACID in favor of a workflow that ensures the system <em>eventually</em> reaches a consistent state, making it a perfect fit for microservice architectures that prioritize resilience and uptime.</p><h3>Two Main Saga Implementation Approaches 🛠️</h3><p>There are two primary ways to implement the Saga pattern:</p><ol><li><strong>Choreography-based Saga:</strong> In this approach, each service involved in the saga listens for events published by other services and reacts accordingly by executing its local transaction and publishing new events. There’s no central orchestrator; the services coordinate themselves through asynchronous event communication.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*vTd66pIjW59dJFH788Z9dA.png" /></figure><p><strong>2. Orchestration-based Saga:</strong> This approach introduces a central orchestrator service that manages the entire saga lifecycle. The orchestrator sends commands to each participating service to execute its local transaction and then listens for responses to decide the next step or whether to trigger compensating transactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*_EaKwZIfADan4XetovWCGQ.png" /></figure><h3>Considerations and Challenges 🤔🚧</h3><p>While the Saga pattern is powerful, it’s essential to be aware of its challenges:</p><ul><li><strong>Eventual Consistency:</strong> Sagas provide eventual consistency, meaning there might be a period where data is inconsistent across services until all transactions or compensating transactions are completed. This requires careful consideration for user experience and business logic.</li><li><strong>Complexity:</strong> Implementing compensating transactions can be complex and requires careful design to ensure they correctly undo the effects of the original transactions.</li><li><strong>Idempotency:</strong> Saga participants must be designed to handle the same event or command multiple times without causing unintended side effects (idempotency).</li><li><strong>Transaction Isolation:</strong> Since local transactions are independent, you might encounter issues like “dirty reads,” “fuzzy reads,” or “phantom reads” between the steps of a saga. Design strategies like semantic locks or optimistic/pessimistic locking might be needed to mitigate these.</li></ul><h4>Choreography vs. Orchestration: A Detailed Comparison</h4><p>Choosing between choreography and orchestration is a critical design decision. Here is a breakdown of their key characteristics in a points format.</p><p><strong>Choreography-based Saga</strong></p><ul><li><strong>Workflow Logic:</strong> The workflow is decentralized. Each service contains a piece of the business logic and knows its own role and which events to listen for.</li><li><strong>Service Coupling:</strong> Services are very loosely coupled. They don’t need direct knowledge of each other, only of the events they produce or consume from a message bus.</li><li><strong>Communication Style:</strong> Uses an event-driven (Publish/Subscribe) model where services react to events as they occur.</li><li><strong>Discoverability &amp; Monitoring:</strong> It is difficult to monitor the state of the overall business process because there is no single source of truth. This often requires specialized tracing and monitoring tools.</li><li><strong>Point of Failure:</strong> There is no single point of failure, which increases resilience. However, the process can fail if any service fails to correctly process or publish its event.</li><li><strong>Complexity:</strong> It’s simple to add new participants to the flow without changing existing services. However, the overall process can become hard to understand and debug as more services are added.</li><li><strong>Best For:</strong> Simpler, shorter workflows with a small number of participants where loose coupling and service autonomy are the highest priority.</li></ul><p><strong>Orchestration-based Saga</strong></p><ul><li><strong>Workflow Logic:</strong> The workflow is centralized within the orchestrator. It explicitly defines and controls every step of the process.</li><li><strong>Service Coupling:</strong> Participant services are loosely coupled from each other, but they are all tightly coupled to the central orchestrator.</li><li><strong>Communication Style:</strong> Uses a command-driven (Request/Reply) model. The orchestrator sends direct commands to services and waits for a response.</li><li><strong>Discoverability &amp; Monitoring:</strong> It is easy to monitor the state of a saga because it is explicitly managed and can be queried from the orchestrator, which greatly simplifies monitoring and debugging.</li><li><strong>Point of Failure:</strong> The orchestrator itself is a potential single point of failure and can become a performance bottleneck if not designed for high availability.</li><li><strong>Complexity:</strong> Involves more upfront design to build and maintain the orchestrator. However, it simplifies the logic within each participant service, as they become simple “workers.”</li><li><strong>Best For:</strong> Complex, long-running processes with many steps, conditional logic, or when centralized control and clear visibility of the entire workflow are required.</li></ul><h3>When to Choose the Saga Pattern? ✅</h3><p>The Saga pattern is a great fit when:</p><ul><li>You need to manage transactions that span multiple microservices.</li><li>You cannot or don’t want to use distributed transactions with two-phase commits.</li><li>Eventual consistency is acceptable for your business requirements.</li><li>You need to maintain autonomy and resilience in your microservices.</li></ul><h3>Conclusion 🎉</h3><p>The Saga pattern is a valuable tool in the arsenal of any microservices architect. By understanding its principles, implementation approaches, and potential challenges, we can build more robust and scalable distributed systems. Remember to carefully consider your specific use case and choose the implementation strategy (choreography or orchestration) that best fits your needs.</p><p>What are your experiences with the Saga pattern? Share your thoughts and insights in the comments below! 👇</p><p>#microservices #distributedtransactions #sagapattern #designpatterns #softwarearchitecture #eventualconsistency</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aeff14a0c82c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pipeline and Stage Concurrency Design Pattern in Golang]]></title>
            <link>https://somanshusingla.medium.com/pipeline-and-stage-design-pattern-in-golang-4d8e7e5a6ea2?source=rss-3f64da4146b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d8e7e5a6ea2</guid>
            <category><![CDATA[golang-concurrency]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Somanshu Singla]]></dc:creator>
            <pubDate>Fri, 06 Jun 2025 07:47:49 GMT</pubDate>
            <atom:updated>2025-06-06T07:58:16.290Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction to Pipeline and Stage Design Pattern</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*PAgMdWtWP-VPmo4aEZTnMA.png" /></figure><p>The true power of Golang lies in the concurrency patterns. Pipeline Stage Concurrency is the ultimate concurrency design pattern and Golang is my favourite language to implement it.</p><p>In this article, I would introduce you to the Pipeline Stage Design Pattern, how to implement it and how to build complex workflows using this design pattern.</p><h3>The Power of a Pipeline: True Separation of Concerns</h3><p>By structuring your data processing as a pipeline, you gain a level of abstraction that offers some significant benefits. Let’s break them down.</p><h3>Independent &amp; Composable Stages</h3><p>One of the most immediate advantages is the ability to <strong>modify stages independently</strong>. When each stage of your pipeline is a self-contained unit, you can change the internals of one stage without worrying about unintended side effects in others. This makes your code easier to reason about and safer to change.</p><p>Furthermore, you can <strong>mix and match how stages are combined</strong>. Imagine you have a set of well-defined pipeline stages: one to read data, one to filter it, and one to write it. You can easily rearrange these stages or insert new ones to create different processing flows, all without having to modify the stages themselves.</p><h3>Unlocking Concurrency</h3><p>This is where pipelines truly shine, especially in a language like Go. Each stage of the pipeline can be processed <strong>concurrently</strong> to its upstream and downstream neighbors. While one stage is processing a piece of data, the next stage can be working on the data it just received, and the previous stage can be fetching the next piece. This parallelism can lead to significant performance improvements.</p><h3>Advanced Patterns: Fan-Out and Rate-Limiting</h3><p>Pipelines also open the door to more advanced and powerful patterns. We can <strong>fan-out</strong> a particular stage, meaning we can have multiple instances of that stage running in parallel to handle a higher workload. We can also introduce <strong>rate-limiting</strong> to control the flow of data through our pipeline, preventing downstream systems from being overwhelmed. We’ll dive deeper into fan-out and fan-in in a future article, and we’ll explore rate-limiting in more detail as we work through the concepts in <em>“Concurrency in Go”</em>.</p><h3>How to implement stages?</h3><p>A stage is just something that takes data in, performs a transformation on it, and sends the data back out</p><p>Multiplier Stage:</p><pre>multiply := func(values []int, multiplier int) []int {<br>    multipliedValues := make([]int, len(values))<br>    for i, v := range values {<br>        multipliedValues[i] = v * multiplier<br>    }<br>    return multipliedValues<br>}</pre><p>Addition Stage:</p><pre>add := func(values []int, additive int) []int {<br>    addedValues := make([]int, len(values))<br>    for i, v := range values {<br>        addedValues[i] = v + additive<br>    }<br>    return addedValues<br>}</pre><p>Executing them:</p><pre>ints := []int{1, 2, 3, 4}<br>for _, v := range add(multiply(ints, 2), 1) {<br>    fmt.Println(v)<br>}</pre><p>Output</p><pre>3<br>5<br>7<br>9</pre><h3>What <em>are</em> the properties of a pipeline stage?</h3><ul><li>A stage consumes and returns the same type.</li><li>A stage must be reified by the language so that it may be passed around. Functions in Go are reified and fit this purpose nicely.</li></ul><h3>Best Practices for Constructing Pipelines</h3><p>Channels are uniquely suited to constructing pipelines in Go because they fulfill all of our basic requirements. They can receive and emit values, they can safely be used concurrently, they can be ranged over, and they are reified by the language.</p><pre>generator := func(done &lt;-chan interface{}, integers ...int) &lt;-chan int {<br>    intStream := make(chan int)<br>    go func() {<br>        defer close(intStream)<br>        for _, i := range integers {<br>            select {<br>            case &lt;-done:<br>                return<br>            case intStream &lt;- i:<br>            }<br>        }<br>    }()<br>    return intStream<br>}<br><br>multiply := func(<br>  done &lt;-chan interface{},<br>  intStream &lt;-chan int,<br>  multiplier int,<br>) &lt;-chan int {<br>    multipliedStream := make(chan int)<br>    go func() {<br>        defer close(multipliedStream)<br>        for i := range intStream {<br>            select {<br>            case &lt;-done:<br>                return<br>            case multipliedStream &lt;- i*multiplier:<br>            }<br>        }<br>    }()<br>    return multipliedStream<br>}<br><br>add := func(<br>  done &lt;-chan interface{},<br>  intStream &lt;-chan int,<br>  additive int,<br>) &lt;-chan int {<br>    addedStream := make(chan int)<br>    go func() {<br>        defer close(addedStream)<br>        for i := range intStream {<br>            select {<br>            case &lt;-done:<br>                return<br>            case addedStream &lt;- i+additive:<br>            }<br>        }<br>    }()<br>    return addedStream<br>}<br><br>done := make(chan interface{})<br>defer close(done)<br><br>intStream := generator(done, 1, 2, 3, 4)<br>pipeline := multiply(done, add(done, multiply(done, intStream, 2), 1), 2)<br><br>for v := range pipeline {<br>    fmt.Println(v)<br>}</pre><p>Output:</p><pre>6<br>10<br>14<br>18</pre><h3>How to build Pipelines for complex workflows?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a3R-N-5foKsnghfgZGAU1A.png" /></figure><h4>Representing a Stage</h4><pre>type Stage struct {<br> ctx            *gin.Context<br> Next           []*Stage<br> Name           *string<br> Callback       func([]*pipelinemodels.StageArgs) (interface{}, *pipelinemodels.StageArgs, error)<br> incomingStream chan *pipelinemodels.StageArgs<br> responseStream chan *pipelinemodels.ExecutionResponse<br>}<br><br>type StageArgs struct {<br> Input interface{}<br>}<br><br>type ExecutionResponse struct {<br> Data   map[string]interface{}<br> Errors map[string]error<br>}</pre><ul><li>Callback is the func call that gets executed as part of the stage</li><li>A stage would have a list of Next stages.</li><li>In above example, Stage 1 would have Stage 2 and Stage 3 in its Next list.</li><li>You would want to have common context across stages for controlling the failures.</li><li>incomingStream is the chan stream of inputs (buffered channel with len equivalent to count of incoming edges)</li><li>responseStream is the output of the stage (blocking channel)</li></ul><h4>Representing a Pipeline</h4><pre>type Pipeline struct {<br> stageMap           map[string]*Stage<br> stageIncomingCount map[string]int<br>}</pre><ul><li>Have reference to each stage</li><li>For each stage, store the length of incoming edges</li></ul><h4>Functions you would need to build the pipeline</h4><p>addStage(name, callback) to pipeline — similar to creating vertex in graph:</p><pre>pipeline.stageMap[name] = &amp;Stage{<br>  ctx:          ctx,<br>  Name:         &amp;name,<br>  Callback:     callback,<br>  PipelineName: pipeline.Name,<br>  Next:         make([]*Stage, 0),<br>  logger:       pipeline.logger,<br> }</pre><p>Connect Stages — similar to adding edge between 2 vertices:</p><pre>pipeline.stageMap[fromStage].Next = append(pipeline.stageMap[fromStage].Next, pipeline.stageMap[toStage])</pre><p>Post this, initialize the streams</p><pre>pipeline.stageMap[stageName].incomingStream = make(chan *pipelinemodels.StageArgs, count)<br>pipeline.stageMap[stageName].responseStream = make(chan *pipelinemodels.ExecutionResponse, 1)</pre><p>After initialization of channels, execute each of the stage in loop with goroutines:</p><pre>for args := range stage.incomingStream {<br>   if args.Ctx != nil {<br>    done := args.Ctx.Done()<br>    if done != nil {<br>     stage.safeExit(args.Ctx)<br>     return<br>    }<br>   }<br><br>   allArgs = append(allArgs, args)<br>   if len(allArgs) == cap(stage.incomingStream) {<br>    break<br>   }<br>  }<br><br>  response, nextArgs, err := stage.Callback(allArgs)</pre><p>Once you have the response and the nextArgs, propage it to the incomingStreams of connected nodes:</p><pre>for _, nextStage := range stage.Next {<br>  nextStage.incomingStream &lt;- nextArgs<br>}<br><br>executionResponse := pipelinemodels.ExecutionResponse{<br>  Data:   make(map[string]interface{}),<br>}<br><br>executionResponse.Data[*stage.Name] = response<br><br>for _, nextStage := range stage.Next {<br>  partialResponse, ok := &lt;-nextStage.responseStream<br>  if !ok || partialResponse == nil {<br>   continue<br>  }<br>  <br>  <br>  for stageName, stageExecResponse := range partialResponse.Data {<br>   executionResponse.Data[stageName] = stageExecResponse<br>  }<br>  <br>  for stageName, stageExecError := range partialResponse.Errors {<br>   executionResponse.Errors[stageName] = stageExecError<br>  }<br>}<br><br>stage.responseStream &lt;- &amp;executionResponse</pre><h3>How the input and response flow through stages</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8L0Pykv19IcnFI0-y8z82Q.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d8e7e5a6ea2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka Notes]]></title>
            <link>https://somanshusingla.medium.com/kafka-notes-f0bd721c4e21?source=rss-3f64da4146b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0bd721c4e21</guid>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Somanshu Singla]]></dc:creator>
            <pubDate>Fri, 20 Sep 2024 10:00:57 GMT</pubDate>
            <atom:updated>2024-09-20T10:00:57.173Z</atom:updated>
            <content:encoded><![CDATA[<p><em>My notes from Kafka: </em><a href="https://learning.oreilly.com/library/view/kafka-the-definitive/9781492043072/"><em>The Definitive Guide</em></a></p><p><strong>Kafka’s Other Names</strong></p><ul><li>Distributed commit log</li><li>Distributing streaming platform</li></ul><p><strong>Message</strong>: A unit of data within Kafka. It’s an array of bytes for kafka.</p><p><strong>Keys</strong>: Message metadata. Used when messages are to be written to partitions in a more controlled manner.</p><p><strong>Topic</strong>: Equivalent to Database table.</p><p>A topic typically has multiple partitions, there is no guarantee of message ordering across the entire topic, just within a single partition.</p><p><strong>Stream</strong>: A stream is considered to be a single topic of data, regardless of the number of partitions.</p><p><strong><em>Producers</em></strong> create new messages. In other publish/subscribe systems, these may be called <em>publishers</em> or <em>writers</em>. A message will be produced to a specific topic. By default, the producer will balance messages over all partitions of a topic evenly. In some cases, the producer will direct messages to specific partitions. This is typically done using the message key and a partitioner that will generate a hash of the key and map it to a specific partition. This ensures that all messages produced with a given key will get written to the same partition. The producer could also use a custom partitioner that follows other business rules for mapping messages to partitions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*T2E41NTbXcB6oqhm.png" /></figure><p><strong><em>Consumers</em></strong> read messages. In other publish/subscribe systems, these clients may be called <em>subscribers</em> or <em>readers</em>. The consumer subscribes to one or more topics and reads the messages in the order in which they were produced to each partition. The consumer keeps track of which messages it has already consumed by keeping track of the offset of messages. The <em>offset</em> — an integer value that continually increases — is another piece of metadata that Kafka adds to each message as it is produced. Each message in a given partition has a unique offset, and the following message has a greater offset (though not necessarily monotonically greater). By storing the next possible offset for each partition, typically in Kafka itself, a consumer can stop and restart without losing its place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*FZdgANfZVqtTY3qy.png" /></figure><p><strong>Broker</strong></p><ul><li>A single Kafka server is called a <em>broker</em>.</li><li>The broker receives messages from producers, assigns offsets to them, and writes the messages to storage on disk</li><li>It also services consumers, responding to fetch requests for partitions and responding with the messages that have been published.</li><li>Kafka brokers are designed to operate as part of a <em>cluster</em>.</li><li>Within a cluster of brokers, one broker will also function as the cluster <em>controller</em> (elected automatically from the live members of the cluster).</li><li>The controller is responsible for administrative operations, including assigning partitions to brokers and monitoring for broker failures.</li><li>A partition is owned by a single broker in the cluster, and that broker is called the <em>leader</em> of the partition.</li><li>A replicated partition is assigned to additional brokers, called <em>followers</em> of the partition.</li><li>All producers must connect to the leader in order to publish messages, but consumers may fetch from either the leader or one of the followers.</li></ul><p><strong>Producer</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bgwb5uLJFYmTVorf0MBreg.png" /></figure><p><strong>Replication</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*oVBdWJpol5KKm6_u.png" /></figure><p><strong>Retention</strong></p><ul><li>Kafka brokers are configured with a default retention setting for topics, either retaining messages for some period of time (e.g., 7 days) or until the partition reaches a certain size in bytes (e.g., 1 GB).</li><li>Once these limits are reached, messages are expired and deleted.</li><li>Individual topics can also be configured with their own retention settings so that messages are stored for only as long as they are useful.</li></ul><p><strong>Compaction</strong></p><p>Topics can also be configured as <em>log compacted</em>, which means that Kafka will retain only the last message produced with a specific key</p><h4>Why Kafka?</h4><p><strong>Multiple Producers</strong></p><p>Kafka is able to seamlessly handle multiple producers, whether those clients are using many topics or the same topic.</p><p>This makes the system ideal for aggregating data from many frontend systems and making it consistent.</p><p><strong>Multiple Consumers</strong></p><ul><li>Kafka is designed for multiple consumers to read any single stream of messages without interfering with each other client.</li><li>This is in contrast to many queuing systems where once a message is consumed by one client, it is not available to any other.</li><li>Multiple Kafka consumers can choose to operate as part of a group and share a stream, assuring that the entire group processes a given message only once.</li></ul><p><strong>Disk-Based Retention</strong></p><ul><li>Not only can Kafka handle multiple consumers, but durable message retention means that consumers do not always need to work in real time.</li><li>Messages are written to disk and will be stored with configurable retention rules.</li></ul><p><strong>Scalable</strong></p><ul><li>Brokers/Partitions can be increased</li><li>Expansions can be performed while the cluster is online, with no impact on the availability of the system as a whole.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0bd721c4e21" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[WIP] Preventing Software Downtimes]]></title>
            <link>https://somanshusingla.medium.com/wip-preventing-software-downtimes-18d34b20ee47?source=rss-3f64da4146b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/18d34b20ee47</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Somanshu Singla]]></dc:creator>
            <pubDate>Sun, 31 Mar 2024 07:06:01 GMT</pubDate>
            <atom:updated>2024-03-31T07:06:01.609Z</atom:updated>
            <content:encoded><![CDATA[<h4>Importance of Availablity</h4><p>How your availability goals should look like?</p><ol><li>Define Availability SLAs for your services.</li><li>For you sync APIs and customer-facing flows, try to achieve five nines availability (99.999%).</li><li>Even 99.9% availability leads to ~9hrs of downtime per year.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*-j6Ukr5mKI-ceynmehr0Vg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18d34b20ee47" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting promoted from SDE-1 to SDE-2]]></title>
            <link>https://somanshusingla.medium.com/getting-promoted-from-sde-1-to-sde-2-49bddff09820?source=rss-3f64da4146b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/49bddff09820</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Somanshu Singla]]></dc:creator>
            <pubDate>Thu, 29 Feb 2024 05:42:12 GMT</pubDate>
            <atom:updated>2024-02-29T05:48:02.079Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>What it takes to become a Senior SDE.</blockquote><h4>What matters and what to look out for?</h4><ol><li>Are you able to deliver projects independently?</li><li>How much have you influenced your team? Have you defined best practices?</li><li>How good are your estimates?</li><li>Take ownership and resolve risks.</li></ol><h4>1. How Good are your coding skills?</h4><ol><li>Coding Practices</li><li>Review Comments that you are getting on your PRs</li><li>Review Comments that you are doing on others’ PRs</li><li>How many people have reviewed your PRs</li><li>Have you contributed to the code base in the away team and followed their best practices?</li></ol><h4>2. HLD and LLD</h4><ol><li>How many designs have you worked on?</li><li>How much do you know about cloud services?</li><li>What design patterns have you used?</li><li>How many designs have you reviewed?</li></ol><h4>3. Delivering Projects and Deployments</h4><ol><li>How many of your deployments were successful and how many were rolled back/had a huge blast radius?</li><li>What was the impact and adoption of your deliverable?</li><li>How many times were your timelines missed?</li><li>How many people are available to give feedback for that project?</li></ol><h4>4. Debugging Skills</h4><ol><li>Have you handled and fixed any critical issues?</li><li>Have you handled the on-call or reliability during peak traffic days?</li><li>How is your performance during on-calls?</li><li>Have you created metrics/dashboards for your services?</li><li>Are you solving more on-call issues than other SDE-1s or SDE-2s in the team?</li></ol><h4>5. Your Disambiguation skills</h4><ol><li>Can you start the execution even before having 100% of the requirements?</li><li>Do you ask relevant questions and able to figure out the edge cases?</li><li>How vocal are you in team meetings? Have you helped in the closure of tech discussions?</li></ol><h4>6. Quality of your work</h4><ol><li>Do you add testing details to your PRs?</li><li>Do you share deployment steps with the team before deploying?</li><li>Do you get your work reviewed before releasing it to production?</li><li>What is the average number of comments or revisions on your PRs?</li><li>Have you implemented unit or integration tests for any of your deliverables?</li></ol><h4>7. Your mentoring skills</h4><ol><li>How many interns have you mentored?</li><li>How much have you contributed to hiring?</li><li>How many new folks have you onboarded to your team?</li><li>Do you bring innovation and new tech to the team?</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49bddff09820" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>