<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gururaj Narayanan on Medium]]></title>
        <description><![CDATA[Stories by Gururaj Narayanan on Medium]]></description>
        <link>https://medium.com/@4nguru?source=rss-dc3f5abce4bb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*YTkOrinbu5xMwuHq.jpg</url>
            <title>Stories by Gururaj Narayanan on Medium</title>
            <link>https://medium.com/@4nguru?source=rss-dc3f5abce4bb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:03:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@4nguru/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How do LLMs execute Tools or functions ?]]></title>
            <link>https://medium.com/@4nguru/how-do-llms-execute-tools-or-functions-11fa25553483?source=rss-dc3f5abce4bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/11fa25553483</guid>
            <category><![CDATA[llm-agent]]></category>
            <dc:creator><![CDATA[Gururaj Narayanan]]></dc:creator>
            <pubDate>Sun, 25 Oct 2026 18:42:28 GMT</pubDate>
            <atom:updated>2026-10-25T18:42:28.291Z</atom:updated>
            <content:encoded><![CDATA[<p>When working with LLMs, “invoking tools” or “invoking functions” is often abstracted away. You pass a list of functions to an SDK, the model figures out what to call, and data flows back and forth.</p><p>In a distributed system — where a model sits on a remote GPU cluster in the cloud while your code runs locally — a natural question pops up: <em>How does a cloud model (such as Gemini or GPT) trigger functions on my local machine? And why do Python and Java handle this execution loop automatically, while C++ requires manual handling? BTW, until I wrote C++ code to invoke a model, I did not realise it !</em></p><p>Let’s dive deep into the mechanics behind this magic. I’ll focus here on invoking cloud models from a local environment, but the exact same principles apply when your application code runs on a cloud server.</p><h3>1. LLM Never Executes Code Locally</h3><p>When an API provider like Google, OpenAI, or Anthropic hosts a model, that model runs inside a remote cloud data center. When a tool call is triggered, the actual sequence of steps taking place over the network between your local machine and the cloud server is as follows:</p><ol><li>Application sends a user prompt along with metadata describing your available tools.</li><li>The model analyzes the request and returns a JSON payload stating: <em>“Execute Function X with Arguments Y.”</em></li><li>Local SDK or application intercepts that JSON payload and executes the function on your local machine.</li><li>Application packages the return value into a follow-up request and sends it back to the model.</li><li>The model reads that result and generates the final natural-language response for the user.</li></ol><p>The model doesn’t run code; it only conveys intent. Your local runtime handles execution.</p><h3>2. How the LLM SDK “Reads” Your Code</h3><p>A common question is how an LLM SDK knows what a function does without parsing raw source code files.</p><p>Python treats functions as first-class objects in memory. When you pass a function into an LLM SDK method, the SDK uses the built-in inspect module to read attributes already attached to that object at runtime (such as docstrings and type annotations). The SDK translates this local runtime metadata into a JSON OpenAPI schema before transmitting it across the network to the model.</p><p>Java relies on runtime reflection (java.lang.reflect) and metadata annotations to extract method signatures, parameter names, and types from classes loaded inside the JVM.</p><p>C++ compiles directly to native machine code and lacks a native runtime reflection system. Consequently, C++ cannot automatically introspect functions at runtime. In C++, you must explicitly construct the JSON schema structures and write explicit dispatching code to handle tool execution on your local machine. The C++ LLM SDK cannot automatically execute functions behind the scenes like Python or Java SDKs.</p><h3>3. Automatic LLM SDK Execution vs. Explicit Frameworks vs. Self-Orchestration</h3><p>When designing agent applications, you generally choose among three architectural patterns for managing the execution loop:</p><h4>Pattern A: LLM SDK-Managed Automatic Execution</h4><p>In high-level SDKs (like Python or Java), you can enable automatic tool execution. The LLM SDK completely abstracts away the network round-trips. When the model emits a tool-call request, the LLM SDK catches, executes your local function, feeds the observation back to the model, and returns the final answer in a single call.</p><h4>Pattern B: Explicit Orchestration Frameworks (LangGraph, CrewAI, etc.)</h4><p>When using orchestration frameworks, the LLM SDK’s hidden loop is replaced by an explicit state machine. Tool calls are treated as nodes and edges in a directed graph. This allows you to inspect message checkpoints at every step, pause execution for human-in-the-loop approvals, or coordinate complex multi-agent workflows.</p><h4>Pattern C: Self-Orchestration (Custom Business Logic &amp; Control Flow)</h4><p>In this approach, you hand-craft the execution plumbing yourself. You handle the network request/response parsing, manage the conversation state memory, and construct the execution graph directly in application code. This pattern is mandatory in native C++ environments and preferred in high-performance enterprise systems where developers require total transparency over data flow, error handling, and security.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11fa25553483" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Split Walnut ( Brain)]]></title>
            <link>https://medium.com/@4nguru/split-walnut-brain-734d2e4f86ed?source=rss-dc3f5abce4bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/734d2e4f86ed</guid>
            <category><![CDATA[split-brain-experiment]]></category>
            <category><![CDATA[consensus]]></category>
            <dc:creator><![CDATA[Gururaj Narayanan]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 04:00:51 GMT</pubDate>
            <atom:updated>2026-10-02T04:00:51.473Z</atom:updated>
            <content:encoded><![CDATA[<p>The title was fun or click bait but the topic is serious ! Let’s jump into it.</p><p>There is a class of algorithms called self-stabilising distributed algorithms, which are used to recover systems after arbitrary failures in network and cloud environments where manual intervention is impossible or too slow. The core concept relies on each node continuously updating its own state based on predefined local rules, while constantly monitoring the states of its immediate neighbors. There is temporary chaos before the order returns to the system. The premise is to start from a failure state and recover the system. However in comparing this to consensus algorithms, they avoid failure state by driving common understanding of the state across all nodes in the systems.</p><p>Consensus algorithms are built to prevent chaos, whereas self-stabilisation is built to survive it after it happens.</p><p>Now the examples from real systems :</p><ol><li>Self- stabilising : In Cassandra, anti-entropy is used as a mechanism to ensure the proper state of a node recovering from failure. The Gossip protocol is used to exchange node-level information, allowing each node to eventually converge to the correct view of the nodes in the topology.</li><li>Consensus : tools like Raft and Paxos help distributed systems in synchronization. For example, Elastic Search &amp; Kafka use Raft for leader election, while Redis uses Raft extensions for data consistency. Meanwhile, Cassandra relies on Paxos for lightweight transactions.</li></ol><p>Now, regardless of whether systems rely on proactive consensus or reactive self-stabilization, edge cases do happen and split-brain is the failure that slips through both these guardrails.</p><p>I have to admit, I’ve rarely seen this happen in production. Most likely, infrastructure has evolved to handle these failures seamlessly in the background in managed cloud offerings across AWS, GCP, and Azure. However in continuation of this blog, lets discuss few real-world problem that consensus solved :</p><p><strong>Example 1 : </strong>Elastic search ( <a href="https://github.com/elastic/elasticsearch/issues/2488">https://github.com/elastic/elasticsearch/issues/2488</a> ) split brain issue was reported in 2012, when it was using Zen Discovery for simple consensus. Leader election in Zen discovery relies on simple quorum with rule, and is master able to reach nodes with quorum. The issue was participating nodes in quorum need not vote. This causes a split brain.</p><p>The issue occurred in a three-node cluster (node-a, node-b, and node-c, where node-b was the master) when the network connection between node-b and node-c was lost. The existing master (node-b) continued to function normally by accepting client operations. Meanwhile, node-c realized it had lost connection to the master but could still communicate with node-a, so it promoted itself to master. This resulted in a split-brain scenario, split into two hemispheres: one with node-a and node-b (the original master), and the other with node-a and node-c (the newly promoted master).</p><p>According to the bug report, one of the root causes for the connectivity loss between node-b and node-c was heavy Garbage Collection (GC) pauses. To mitigate this, temporary fixes were introduced:</p><ul><li>Increased GC latency settings so that long-running GC pauses would not be falsely interpreted as a permanent network failure.</li><li>Improved pinging logic, changing the behavior so that when a node detects it has lost connection to the master, it reports the “loss of connection” rather than immediately electing itself as the new leader.</li></ul><p>The split-brain scenario caused severe data issues, such as duplicate indices being created independently across both hemispheres. Once the network recovered, reconciling these duplicate indices proved to be extremely difficult.</p><p>Elasticsearch replaced Zen with a Raft-like consensus protocol. A core principle of Raft is: once a new state is accepted by a majority quorum, any transactions or updates based on the older state are rejected, effectively preventing split-brain scenarios.”</p><p><strong>Example 2:</strong> Split Brain in PostgreSQL (<a href="https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/41">https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/41</a> )</p><p>In PostgreSQL cluster consisting of 4 nodes pg1, pg2, pg3, pg4 where pg1 is the primary.</p><p>During a PostgreSQL upgrade, pg2 was promoted as the new primary. pg1, the former primary, was stopped and immediately restarted. It came back and claimed its previous state as a primary, creating a split-brain condition between pg1 and pg2. GitLab considered pg2 as the true primary, and the application sent traffic to pg2.</p><p>pg3 and pg4, however, were configured to follow pg1, so when pg1 returned they attempted to replicate from it. With only pg2 serving traffic, all the traffic was directed to it and system performance degraded. This issue was identified in real time by engineers and acted on.</p><p>As per the retrospective, GitLab had chosen to manually configure PostgreSQL replication instead of using repmgr. This was an important contributing factor: pg3 and pg4 did not automatically switch to the newly promoted primary and had to be manually reconfigured to follow pg2.</p><p><strong>Closing Remark:</strong> Problems do happen in spite of all the guardrails!</p><p>If you notice closely, we only talked about split-brain issues in the distributed system itself. What happens to the application when it sends and receives data from a rogue primary? It can mess up the application state too.</p><p>Like every problem, this one has a remedy too — it is called fencing token (read about it)!</p><p>Consider a situation where a downstream S3 is receiving updates from a rogue primary PgSQL. The legitimate primary has a higher fencing token. Every update sent to S3 carries this token. S3 can keep track of the latest fencing token it has seen and reject any update coming with an older token from the rogue primary.</p><p>This way, even if the rogue primary is still alive and continues sending updates, the downstream system can reject its updates. The rogue primary may still think it is the primary, but it cannot modify the downstream state anymore.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=734d2e4f86ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cost of Faster Writes]]></title>
            <link>https://medium.com/@4nguru/cost-of-faster-writes-1d05f76b5c0e?source=rss-dc3f5abce4bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d05f76b5c0e</guid>
            <dc:creator><![CDATA[Gururaj Narayanan]]></dc:creator>
            <pubDate>Sat, 26 Sep 2026 18:05:34 GMT</pubDate>
            <atom:updated>2026-09-26T18:05:34.936Z</atom:updated>
            <content:encoded><![CDATA[<p>We come across multiple systems that allow faster writes. In many systems, there is a trade-off where some of the cost of these faster writes is paid for later. In this blog, we look at Cassandra, OpenSearch, Kafka and PostgreSQL. and articulate where this cost is transferred. It is necessary to understand this because it can have a direct impact on read latency, the type of hardware that needs to be provisioned, and ultimately the infrastructure cost.</p><p>However, this is not a generalized argument that faster writes always result in higher read cost or additional computational cost. Kafka is a good counterexample, where its append-only design allows it to keep the read path relatively simple. In PostgreSQL, the cost is distributed between the write path and background processing.</p><h4>Cassandra</h4><p>In <strong><em>Cassandra</em></strong>, when a write, update or delete happens, the mutation is first written to the commit log on disk for durability and applied to an in-memory memtable. This helps Cassandra achieve low latency and high write throughput.</p><p>But there is a cost we pay for this fast write path.</p><ol><li>Once the memtable is flushed, it becomes an SSTable on disk. Each SSTable has associated Bloom filters, indexes and other metadata. These consume memory and disk space, but they help Cassandra avoid unnecessarily lookup on every SSTable during a read.</li><li>Over time, Cassandra ends up with multiple SSTables containing mutations for the same partition. Cassandra periodically runs compaction, where multiple SSTables are merged into a new SSTable. This compaction reconciles older versions of data and tombstones.<br>If compaction never happened, a read could potentially need to consult more SSTables ( read amplification ) to reconstruct the current state of a partition.</li><li>Cassandra continues serving reads and writes while compaction is running, but compaction consumes CPU, disk I/O and additional disk space. The new SSTables also need their associated indexes and other structures, and the old SSTables are retired only when they are no longer needed.</li></ol><p>Cassandra basically keeps the write path simple and fast, and moves some of the work to the background. The cost is that we now have to pay for this work through compaction, additional I/O, CPU, storage and potentially higher read amplification.</p><h4><strong>OpenSearch</strong></h4><p>In <strong>OpenSearch</strong>, when write, update or delete happens, the mutation is first written to the in-memory buffer and transaction log on disk for durability. When the buffer is full, it is flushed to disk and creates a new segment. On the disk, the segments are periodically merged to form larger segments. When new segments are formed, the associated indexes and metadata are also created based on Lucene’s segment organization.</p><p>Now what is the impact on search query ?</p><p>Before the segments are merged, a search may need to search across multiple segments and then merge the results. Lucene has indexes and other data structures within each segment to make this search efficient, so it is not simply scanning every segment. But as the number of segments increases, the query may have to do more segment-level work, which can increase CPU and query latency.</p><p>Segment merging reduces the number of segments and can make searches more efficient. But the merging itself consumes CPU, disk I/O and additional disk space because Lucene needs to read the existing segments, create a new segment with its associated indexes and metadata, and write it back to disk.</p><p>There is another trade-off with refresh. OpenSearch needs to refresh the index to make newly indexed documents visible to search. Frequent refreshes make writes searchable sooner, but can result in more segments and additional resource consumption.</p><p>So again, OpenSearch keeps the indexing path relatively simple and moves some of the work to segment creation, refresh and background segment merging. The cost is paid through CPU, disk I/O, storage and potentially additional work during reads.</p><h4>Kafka</h4><p>In <strong><em>Kafka</em></strong>, the situation is slightly different compared to Cassandra and OpenSearch. Kafka is primarily an append-only log. When a write happens, the record is appended to the partition log and is served through the OS page cache. The record is also replicated to the followers based on the replication configuration.</p><p>Because the records are append-only, Kafka does not normally have to do the same kind of reconciliation during reads that we see in Cassandra or OpenSearch. A consumer reads the records sequentially based on the offset. It does not have to look across multiple structures to reconstruct the latest value.</p><p>So in Kafka, the cost of faster writes is not normally transferred to the read path in the same way.</p><h4>PostgreSQL</h4><p>PostgreSQL takes a different path compared to Cassandra and OpenSearch. When a row has to be inserted or updated, the page containing the row needs to be loaded into PostgreSQL’s shared buffers and then modified in memory. This page-level read can be an additional cost compared to the append-oriented write paths of Cassandra and OpenSearch.</p><p>At regular intervals, checkpoints write dirty pages from the shared buffers to the data files on disk. Vacuum, on the other hand, cleans up dead row versions created by updates and deletes once they are no longer needed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d05f76b5c0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Memory for low latency reads and writes]]></title>
            <link>https://medium.com/@4nguru/memory-for-low-latency-reads-and-writes-0e4a460aa702?source=rss-dc3f5abce4bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e4a460aa702</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[memories]]></category>
            <category><![CDATA[cassandra-opensearch]]></category>
            <dc:creator><![CDATA[Gururaj Narayanan]]></dc:creator>
            <pubDate>Sat, 19 Sep 2026 05:53:10 GMT</pubDate>
            <atom:updated>2026-09-19T05:53:10.029Z</atom:updated>
            <content:encoded><![CDATA[<p>System design must leverage node memory to achieve sub-ms read/write latencies, backed by guardrails against data loss. Distributed systems prove that avoiding random disk I/O on the critical path is essential for scale. We should adopt these exact memory-first patterns in our system design:</p><h3>Cassandra</h3><p><strong>Write path →</strong> When a write request comes, it is written concurrently to the in-memory Memtable and an append-only CommitLog on disk for durability. When the Memtable fills up, it is flushed to disk as an immutable SSTable along with its Bloom filter and index metadata. Writing to RAM and appending to the log keeps write/update latency extremely low.</p><p><strong>Read path →</strong> When the node starts (or when new SSTables are flushed/compacted), Cassandra loads the Bloom filters and Index Summaries for all active SSTables into off-heap memory. Off-heap memory avoids JVM GC overhead. When a read query arrives, searching these off-heap Bloom filters and metadata makes identifying which SSTable has the data much faster. When data is fetched from disk, the OS loads full file pages into the Page Cache. If adjacent or nearby data happens to sit on the same cached page, subsequent reads hit system RAM directly, making them significantly faster.</p><h3>Kafka</h3><p><strong>Write path →</strong> When a producer sends a message, Kafka appends it to the active log segment in the OS Page Cache (RAM). Kafka skips explicit disk flushes to avoid disk latency. Writing to the Page Cache protects against Kafka process crashes (since the OS keeps RAM intact), but node failures (power loss) will wipe unflushed pages. To protect against node failure, Kafka relies on cross-node replication, confirming writes only when multiple brokers hold the message in RAM.</p><p><strong>Read path →</strong> When a consumer requests messages, Kafka reads directly from the active log segment in the Page Cache. Since active consumers read data shortly after it is produced, messages are almost always sitting in RAM. Kafka then uses Zero-Copy, transferring bytes straight from kernel Page Cache to the network socket without copying them into JVM memory. Bypassing the JVM heap avoids Garbage Collection overhead, making reads extremely fast.</p><h3>Open search</h3><p><strong>Write path →</strong> When an indexing write request comes, OpenSearch writes it directly to an in-memory Indexing Buffer and sequentially to a Transaction log on disk for durability. When the in-memory buffer fills up, it flushes data to the OS Page Cache as searchable segments and eventually commits them to disk. Writing to RAM and appending to the log gives extremely low latency for writes.</p><p><strong>Read path →</strong> OpenSearch cannot search documents while they sit in the in-memory buffer; data becomes searchable only after flushing to segments in OS Page cache. When a query runs, OpenSearch checks the OS Page Cache first. If segment files are not in RAM, the OS loads them from physical disk into the Page Cache to serve the read. Subsequent reads hit the Page Cache directly, making searches extremely fast.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e4a460aa702" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Apache Kafka is So Fast: A Simple Explanation]]></title>
            <link>https://medium.com/@4nguru/why-apache-kafka-is-so-fast-a-simple-explanation-f22517d5a09a?source=rss-dc3f5abce4bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f22517d5a09a</guid>
            <category><![CDATA[apache-kafka]]></category>
            <dc:creator><![CDATA[Gururaj Narayanan]]></dc:creator>
            <pubDate>Mon, 07 Sep 2026 20:21:58 GMT</pubDate>
            <atom:updated>2026-09-07T20:21:58.827Z</atom:updated>
            <content:encoded><![CDATA[<p>We know Kafka is extremely fast and handles huge amounts of data without slowdowns. Kafka stays simple and lets the OS do the hard work.</p><p>Here are the main reasons why Kafka gives great performance:</p><h3>1. Sequential Writes</h3><p>Hard disks are usually slow if you write data in random places. Kafka completely avoids random writes. It uses an Append-Only Log:</p><ul><li>It opens a file and adds new messages only at the end of the file.</li><li>Because the drive writes sequentially without stopping or jumping, even cheap normal hard drives can write more data / second.</li></ul><h3>2. Using OS Memory &amp; Page Cache Lock Trade-offs</h3><p>Most Java programs save cached data inside the JVM Heap. But when Java manages a lot of memory, it frequently pauses the whole system to clean up old data (Garbage Collection or GC). These pauses cause delays.</p><p>Kafka avoids this problem by using Page Cache to store messages which leads to following advantages:</p><ul><li>Kafka still performs JVM garbage collection, but keeping large amounts of log data outside the JVM heap significantly reduces GC pressure.</li><li>Reading from RAM happens almost instantly.</li><li>If Kafka restarts, the log data remains available on disk, while the OS may retain recently accessed log pages in memory.</li></ul><p><strong>The Trade-off — Page Cache Lock Contention:</strong> While the Page Cache gives Kafka great average speed, it comes with a trade-off. Because the Page Cache is managed by the kernel, multiple threads must compete for kernel locks when reading or writing heavily at the same time. Under extreme traffic, this lock contention can cause occasional worst-case latency delays.</p><h3>3. Zero-Copy Data Transfer (Direct Memory to Network)</h3><p>Normally sending a file over the network requires copying the same data four times:</p><ol><li>Disk → Page Cache</li><li>Page Cache → Java App Memory</li><li>Java App Memory → Network Buffer</li><li>Network Buffer → Network Card</li></ol><p>Copying the same data back and forth consumes significant CPU power.</p><p>Kafka uses<em> sendfile</em> to transfer data from the page cache to the socket without first copying the data into user-space memory. This significantly reduces CPU and memory-copy overhead.</p><h3>4. Fast Network Copies</h3><p>To prevent losing data if a computer breaks, Kafka sends copies of messages to other replicas in the cluster. It sends the message primary page-cache to socket and receiver reads from socket and writes to its page-cache. Because sending data between page cache and over the network is much faster.</p><h3>My Personal Experience of working with Page-cache in Embedded systems.</h3><p>A few years ago, while working on multimedia middleware that supported music playback, we wanted to ensure faster music playback at the application level. We implemented music file decoding as a kernel process, and the decoded audio for playback was stored in the page cache. This page cache was dedicated to the music file decoding process, which avoided page cache contention with other kernel processes as kernel did not allocate the page-cache to other processes, and the music application could read directly from the page cache. This ensured that we achieved better playback performance at the application level. This is not how Linux kernel works but it was custom/dedicated memory mechanism in our system to improve performance,</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f22517d5a09a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Uber’s approach to Rate Limiter: Architectural Evolution and Trade-offs]]></title>
            <link>https://medium.com/@4nguru/ubers-approach-to-rate-limiteruber-s-global-rate-limiting-architectural-evolution-and-trade-offs-8d65e46bfa54?source=rss-dc3f5abce4bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d65e46bfa54</guid>
            <category><![CDATA[rate-limiter]]></category>
            <category><![CDATA[rate-limiting]]></category>
            <dc:creator><![CDATA[Gururaj Narayanan]]></dc:creator>
            <pubDate>Fri, 27 Mar 2026 12:05:28 GMT</pubDate>
            <atom:updated>2026-03-27T12:07:14.191Z</atom:updated>
            <content:encoded><![CDATA[<p>Uber has 1000s of microservices making millions of RPCs between them and there was definitely a need for a rate limiter so that services don’t get overloaded. However there were multiple issues:</p><ol><li>Each team had their own implementations which caused overhead of maintenance and debugging during incidents. Some services even ran without rate limiting.</li><li>Redis based rate limiters caused operational overhead due to redeploying servers for configuration changes and maintenance of the Redis fleet.</li><li>Observability was an issue as different rate limiters reported metrics differently.</li></ol><p><strong>Ruled out centralised key-value store</strong> due to: Each request would need to increment and read counters remotely, introducing unacceptable latency and cross-region consistency challenges. Even with sharding and replication, hundreds of Redis clusters would be required to maintain accurate global state in real time, adding operational complexity and new failure modes. They also ruled out periodic synchronization of counters because it would reduce network overhead but produce stale data and slow reactions to sudden traffic spikes.</p><p>The goal was ambitious: Provide a unified rate-limiting service that makes it easy for any team to configure per-caller or per-procedure quotas, without code changes.</p><p><strong>Approach 1:</strong> Implement token bucket algorithm with unused token carry-forward — unused tokens at any second are retained for up to 10 seconds (configurable up to 20 seconds) to address bursty traffic. This had one significant issue: token allocation was not fair among callers, causing a noisy neighbour problem.</p><p>Why? There was no per-caller subdivision of tokens — it was one shared pool. Allocating tokens per caller is reasonable in theory, but unused tokens of a specific caller would be wasted if not utilised. Also, when a caller is unknown it would result in regular manual updates to the configuration.</p><p><strong>Approach 2 (Final Design):</strong> Uber redesigned the rate limiter to handle allocation dynamically, adjusting to real-time traffic conditions. Enforcement happens in-process on each service host via a lightweight rate limiting library — no remote call is needed per request. Two systems coordinate to keep quotas accurate:</p><ol><li><strong>Rate Limiter Configurator (RLC)</strong> — periodically analyses real-time traffic and pushes updated per-caller RPS limits down to each host. It runs frequently enough to detect new callers quickly, and unknown callers are passed through by default to avoid blocking legitimate new services.</li><li><strong>Global Rate Limiter</strong> — aggregates observed RPS across all hosts and signals a rate drop percentage when the service is approaching overload. Each host applies this signal locally without needing to coordinate with other hosts per request.</li></ol><p>As a complementary safety net, a <strong>Throttler layer</strong> runs on each host to provide immediate, host-level overload protection for sudden traffic spikes that occur faster than the control plane can react.</p><p><strong>Downsides of this approach:</strong></p><ul><li><strong>2–3 second lag</strong> — The RLC and Global Rate Limiter operate on a polling cycle, so there’s an inherent delay before quota adjustments propagate. The Throttler layer acts as a safety net for sudden spikes during this window.</li><li><strong>Sub-second bursts</strong> — These can briefly exceed quotas before the system reacts. This is an accepted tradeoff — they are extremely rare and short-lived.</li><li><strong>Control plane dependency</strong> — If the RLC becomes unavailable, hosts fail open and continue operating on the last known directive, preventing self-inflicted outages.</li><li><strong>New caller detection</strong> — New callers are passed through by default until the RLC detects and classifies them, which is intentional to avoid blocking legitimate new services.</li></ul><p>Uber explicitly accepted these downsides because the alternative — per-request coordination with the control plane — would reintroduce the latency and scalability problems they were trying to solve in the first place.</p><p>The main changes: tightened the language around “operational overhead,” fleshed out Approach 2 with how enforcement actually works end-to-end, introduced the Throttler before referencing it in the downsides, and expanded the downside bullets into readable sentences.</p><p>Reference for detailed read : <a href="https://www.uber.com/en-IN/blog/ubers-rate-limiting-system/?uclick_id=ccd675a5-2f76-409d-8c82-1f570d2c3e27">https://www.uber.com/en-IN/blog/ubers-rate-limiting-system/?uclick_id=ccd675a5-2f76-409d-8c82-1f570d2c3e27</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d65e46bfa54" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Netflix Real Time Distributed Graph — Example of how tradeoffs are done]]></title>
            <link>https://medium.com/@4nguru/netflix-real-time-distributed-graph-example-of-how-tradeoff-are-made-ff6d40c75c46?source=rss-dc3f5abce4bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff6d40c75c46</guid>
            <category><![CDATA[graph-database]]></category>
            <dc:creator><![CDATA[Gururaj Narayanan]]></dc:creator>
            <pubDate>Wed, 11 Mar 2026 15:04:33 GMT</pubDate>
            <atom:updated>2026-03-12T10:14:46.896Z</atom:updated>
            <content:encoded><![CDATA[<h3>Problem:</h3><p>On Netflix, the same member does different activities on various devices. There were requirements to know that these three activities belong to the same member, despite occurring at different times and across various devices. Discovering these relationships became very difficult due to micro service architecture ( service decomposition and data isolation ), which led to separation of data (different events landed in different tables). Stitching the data from different tables to derive relationships was an onerous task. Hence there was a need for a solution to process and store our enormous swath of interconnected data while enabling fast querying to discover insights.</p><h3>Solution:</h3><p>Three important design trade-offs to arrive on architecture that support ingestions of real-time stream of events and still offer low latency and thigh throughput:</p><ol><li>Data store — storing data in structured tables will result in expensive joins for relationship queries, expensive schema changes for new relationships, hard to identify patterns or anomaly detection. Alternative would be to use Graph that offer faster relationship queries as its mere hop on edges and nodes, easy adoption of new schemas as relationship grows, grouping and hidden relationship cycles are easier.</li><li>However using Graph databases to store graph data have their own limitations to achieve low latency on various operations. Eg. AWS Neptune has high latency due to Single writer and multiple reader mode which is limitation to ingest millions of data, neo4j has limitations on large memory needs, challenges in horizontal scaling for large real time dataset. These limitations make it not suitable for low latency scans, filtering and indexing requirements. Also graph databases are not inherently designed or optimized for the continuous, high-throughput event streaming and real-time data ingestion workloads. For a scalable solution they relied on Cassandra KV Store for records and adjacency list for graph relationship. This performed better than traditional Graph databases.</li><li>Data Processing — To make the data available in near real time, we need to process the data at low latency. So traditional batch processing and data warehouses don’t offer low latency processing hence updating the data in the graph in real time requires streaming processing. The processing is required on various topics where each topic will generate 1 million events per seconds hence for large data volume and real time processing we will adopt stream processing</li></ol><h3>Let’s explore this with visual representations:</h3><h3>Architecture:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nVkQ-xlWCh1Sis0342vQnw.png" /></figure><h3>KV DAL data representation:</h3><p>Here is how the data for the example — two users, two movies, and one shared game — would be physically organized:</p><p>1. The Metadata Namespaces (Nodes)</p><p>These namespaces store the details of the entities. They use a simple point-lookup structure.</p><p><strong>Namespace: node_member</strong></p><p>| Record ID (Key) | Item Key | Item Value (Properties) |</p><p>| user_01 | metadata | { “name”: “Alice”, “tier”: “Premium” } |</p><p>| user_02 | metadata | { “name”: “Bob”, “tier”: “Standard” } |</p><p><strong>Namespace: node_title</strong></p><p>| Record ID (Key) | Item Key | Item Value (Properties) |</p><p>| movie_top_gun | metadata | { “genre”: “Action”, “length”: “130m” } |</p><p>| movie_gladiator | metadata | { “genre”: “Drama”, “length”: “155m” } |</p><p>| game_st_1984 | metadata | { “genre”: “Adventure”, “platforms”: [“iOS”, “Android”] } |</p><p>2. The Relationship Namespaces (Edges / Adjacency Lists)</p><p>This is where the actual “watching” and “playing” connections are stored. In these namespaces, the <strong>User ID</strong> is the record_id, allowing the system to find all of a user’s activity in one request.</p><p><strong>Namespace: edge_watched</strong></p><p>| Record ID (From) | Item Key (To) | Item Value (Edge Properties) |</p><p>| <strong>user_01</strong> | movie_top_gun | { “ts”: “2026–03–10”, “progress”: “100%” } |</p><p>| <strong>user_02</strong> | movie_gladiator | { “ts”: “2026–03–11”, “progress”: “40%” } |</p><p><strong>Namespace: edge_played</strong></p><p>| Record ID (From) | Item Key (To) | Item Value (Edge Properties) |</p><p>| <strong>user_01</strong> | game_st_1984 | { “last_played”: “2026–03–11”, “level”: 12 } | | <strong>user_02</strong> | game_st_1984 | { “last_played”: “2026–03–11”, “level”: 3 } |</p><h4>Key Takeaways for this Example:</h4><ul><li><strong>The Shared Game:</strong> Notice in the edge_played namespace that game_st_1984 appears as an Item Key for two different Record IDs. This is how the graph “knows” both users play the same game without needing a complex join.</li><li><strong>Data Locality:</strong> If Alice (user_01) logs in, Netflix can fetch her entire watch history and game history by hitting two specific records in two namespaces. Because it’s a “Two-Level Map” (Record -&gt; Items), this is a single, low-latency seek in Cassandra.</li><li><strong>Independent Scaling:</strong> If everyone starts playing the shared game at once, Netflix can move the edge_played namespace to its own dedicated Cassandra cluster to handle the traffic without affecting movie playback data.</li></ul><h3>Reference to the blogs:</h3><p><a href="https://netflixtechblog.com/how-and-why-netflix-built-a-real-time-distributed-graph-part-1-ingesting-and-processing-data-80113e124acc">https://netflixtechblog.com/how-and-why-netflix-built-a-real-time-distributed-graph-part-1-ingesting-and-processing-data-80113e124acc</a></p><p><a href="https://netflixtechblog.medium.com/how-and-why-netflix-built-a-real-time-distributed-graph-part-2-building-a-scalable-storage-layer-ff4a8dbd3d1f">https://netflixtechblog.medium.com/how-and-why-netflix-built-a-real-time-distributed-graph-part-2-building-a-scalable-storage-layer-ff4a8dbd3d1f</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff6d40c75c46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Critical Role of Preprocessing in Low Latency Experiences]]></title>
            <link>https://medium.com/@4nguru/the-critical-role-of-preprocessing-in-low-latency-experiences-cd81ae647d0b?source=rss-dc3f5abce4bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd81ae647d0b</guid>
            <dc:creator><![CDATA[Gururaj Narayanan]]></dc:creator>
            <pubDate>Sun, 21 Apr 2024 05:33:13 GMT</pubDate>
            <atom:updated>2024-04-21T05:33:13.626Z</atom:updated>
            <content:encoded><![CDATA[<p>There are several common scenarios where users expect instant responses or low latency experiences:</p><ol><li>Navigation Apps: Users expect instant directions and estimated time of arrival when they input their starting point and destination. This is crucial for planning their journeys efficiently.</li><li>Video Streaming Platforms: Users want videos to start playing immediately after clicking on them, without any buffering or delays. Seamless playback is essential for an enjoyable viewing experience.</li><li>Social Media Feeds: When users open their social media apps, they expect their feeds to load quickly and smoothly, allowing them to scroll through updates from their connections without any lag.</li><li>Food Delivery Apps: In food ordering applications, users expect to see nearby restaurants and their menus loaded instantly, so they can quickly decide where to order from and what to eat.</li></ol><p>Achieving low latency experiences for users, particularly in the read path where users consume information and take action, involves various aspects. Here are two areas that are important to consider for achieving low latency experiences:</p><ol><li>Architecture of System Infrastructure: Designing the system architecture to minimise the distance between the user and the infrastructure serving the application is crucial.</li><li>Write Path Optimisation: The write path, which involves generating and storing data on the server side, plays a vital role in achieving low latency experiences. Optimising the write path entails ensuring efficient data generation, processing, and storage mechanisms.</li></ol><p>By focusing on these aspects, software developers can work towards providing users with low latency experiences, resulting in better engagement and satisfaction with the application. While low latency is crucial for providing a smooth and responsive user experience, ensuring availability takes precedence in situations of failures or system disruptions.</p><p>Lets discuss two examples of systems that do heavy duty pre-processing to provide low latency user experience:</p><p><strong>Netflix Streaming service: </strong>This is a good example of parallel processing, where multiple tasks are executed simultaneously to ensure that the video is ready to be served up to the customer instantly. We take short de-tour to understand how Netflix ensure availability during unforeseen events such as the failure of an entire region.</p><p>Upon the arrival of a new title arrives for publishing, a series of workflows are initiated. These workflows manage various tasks, including conducting inspections, encoding video in multiple resolutions, qualities, and codecs, encoding audio in various qualities and codecs, generating subtitles in numerous languages, and packaging the resulting outputs in multiple player formats. There are hierarchy of these multiple workflows to support numerous device types on which Netflix support video streaming.</p><p>Below is a representation of building block used by Netflix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*1JnMtLfZGJv--O0QX0GaTg.png" /></figure><p>This foundational element is utilised to construct a hierarchical structure aimed at achieving the desired functionality. It begins with the ingestion of content from the publishing studio and culminates in the generation of various playback formats and associated metadata. Creating concurrent workflow that are scalable based on input load, ensures every content has its own dedicated path to get published as soon as possible. Below is a representation of the hierarchy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Klvv_tBvQPm5T8ZXZpeSjg.png" /></figure><p>Another crucial step for low latency experience is to ensure that all this content is made available through OpenConnect CDNs, strategically positioned closer to the customers.</p><p><strong><em>Now the detour </em></strong>— Ensuring the resilience of the system infrastructure in the face of failures or catastrophic events such as the failure of an entire region is a critical aspect of large-scale preprocessing. Netflix addresses this challenge by replicating content across all regions, serving multiple purposes. Firstly, it facilitates load distribution, enabling the transfer of excess load from overwhelmed regions to others. Secondly, in the event of a region failure, the system seamlessly redistributes the workload to unaffected regions, guaranteeing uninterrupted service for customers. This robust infrastructure design is integral to maintaining the reliability and availability of Netflix’s platform under diverse circumstances.</p><p>As you might already realised, this could lead to thundering herd problem. This is a significant concern during load distribution, where the sudden influx of additional load to a region could potentially overwhelm it, leading to failure. This scenario underscores the importance of implementing intelligent load balancing mechanisms and capacity planning strategies to prevent such issues. To mitigate the thundering herd problem and minimise the negative impact on regions taking on additional load, two strategies are commonly adopted:</p><p>a) For small incremental load distribution, every region is over-provisioned with additional capacity. This means that each region already has extra capacity available, which is actively handling load within its current region. This preemptive over-provisioning helps to absorb sudden increases in load without overwhelming the region.</p><p>b) In the event of region failures or situations where new regions must take on significantly more load, Dark Capacity is employed. Dark capacity involves provisioning additional capacity in a region and pre-warming it so that it can be rapidly deployed when needed. This ensures that the region has the necessary resources readily available to handle unexpected surges in demand without experiencing performance degradation or failure.</p><p>This detour was to emphasise that each aspect contributes to the overall reliability, performance, and user experience of the system, emphasising the need for a comprehensive and adaptable approach to system design.</p><p>In the Netflix example, the focus was on system infrastructure and framework design to handle heavy-duty preprocessing for providing low-latency responses on the read path. Now, let’s shift gears to an example that revolves around optimising data structure design, specifically in the context of graph structures, to achieve low-latency responses, particularly in route planning scenarios with additional processing during the read path.</p><p><strong>Maps Route planning: </strong>Navigation systems, like Google Maps and Apple Maps, utilise graph representations of road networks to execute various queries, such as determining the shortest distance from point X to point Y and estimating the estimated time of arrival (ETA) for travel between the two points, among other functionalities. Recently, there has been ongoing work to integrate navigation through charging stations, aimed at alleviating range anxiety for electric cars. However, for the purpose of our discussion, we will focus specifically on calculating the shortest distance on road from point X to point Y.</p><p>A road network typically consists of intersections and roads, which are represented as vertices (V) and edges (E) on a graph. Algorithms such as Dijkstra’s or A* are commonly employed to compute the shortest path from Point X to Point Y on this graph. The time complexity of Dijkstra’s algorithm is O((V + E) log V), where each vertex is processed once and each edge is examined once.</p><p>As the number of edges grows, the time complexity of Dijkstra’s algorithm can increase, potentially resulting in slower performance. This is because the algorithm’s time complexity is influenced by both the number of vertices and the number of edges in the graph. In dense graphs, where there are many edges, the algorithm needs to process more edges during each iteration of the main loop, leading to higher time complexity.</p><p>In real-world scenarios, such as computing the distance from New Delhi to Bengaluru, which is about 2200+ kilometres, with numerous vertices and edges, using Dijkstra’s algorithm without any optimisations may result in poor user experience due to high latency. The sheer volume of edges and vertices in a dense map can significantly impact the algorithm’s execution time, making route computation slower and less responsive. Therefore, it’s essential to employ optimisation techniques to ensure efficient and timely rout planning in such scenarios.</p><p>Graph contraction is popular technique used to create sparse graphs through eliminating edges and replacing them short cut edges. Dijkrata’s algorithm will have better performance when graph is sparse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*CR2VQiY_lhzCQhZ0KF5T6w.png" /></figure><p>In the image above, we replace edges E1 and E2 with a new edge (shortcut/virtual edge) C1, and edges E3 and E4 are replaced with a new edge (shortcut/virtual edge) C2. Additionally, vertex V3 is eliminated. When creating this sparse graph, it’s essential to note that the elimination of E1, E2, E3, E4, and V3 will not impact shortest path calculations since there are no other edges or vertices dependent on them.</p><p>Applying this concept to real-time route planning on maps, we can create multiple sparse maps using contraction. Map1 can represent the full, un-contracted map, while Map2 represents a contracted graph for finding routes within a city, and Map3 represents a contracted graph for finding routes between two cities over long distances. The route planning system can query the appropriate map based on the distance between the two nodes for which route planning is requested. Referring back to the example, computing the route from New Delhi to Bengaluru, which spans about 2200+ kilometres, would be faster using the appropriate contracted map compared to the full map.</p><p><strong>Summary:</strong></p><p>This write-up emphasises the significance of low latency design and highlights the need for optimisation across various aspects of system design, including system infrastructure, data structure, and algorithm design, to deliver a seamless user experience.</p><p><strong>Reference:</strong></p><ol><li>[Netflix] <a href="https://www.youtube.com/watch?v=T_D1G42G0dE&amp;t=12s">https://www.youtube.com/watch?v=T_D1G42G0dE&amp;t=12s</a></li><li>[Netflix] <a href="https://netflixtechblog.com/the-netflix-cosmos-platform-35c14d9351ad">https://netflixtechblog.com/the-netflix-cosmos-platform-35c14d9351ad</a></li><li>[Maps] <a href="https://www.youtube.com/watch?v=-0ErpE8tQbw&amp;t=2s">https://www.youtube.com/watch?v=-0ErpE8tQbw&amp;t=2s</a></li><li>[Maps] <a href="https://www.cs.cmu.edu/afs/cs/academic/class/15210-f12/www/lectures/lecture16.pdf">https://www.cs.cmu.edu/afs/cs/academic/class/15210-f12/www/lectures/lecture16.pdf</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd81ae647d0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Replication — Chaotic Goodness]]></title>
            <link>https://medium.com/@4nguru/replication-chaotic-goodness-1ea298c27730?source=rss-dc3f5abce4bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ea298c27730</guid>
            <dc:creator><![CDATA[Gururaj Narayanan]]></dc:creator>
            <pubDate>Sat, 13 Apr 2024 18:22:52 GMT</pubDate>
            <atom:updated>2024-04-13T18:48:14.297Z</atom:updated>
            <content:encoded><![CDATA[<h3>Replication — Chaotic Goodness</h3><p>Replication is a powerful tool, offering numerous benefits. However, it’s essential for users to be aware of potential challenges and address them in their designs. During my academic years, a significant focus of my research revolved around self-stabilising distributed algorithms, which possess the remarkable ability to converge within a finite time to a configuration where their behavior is correct. An example of this is the leader election algorithm employed in single leader replication systems. However, it’s worth noting that in such systems, there’s a possibility for the emergence of two leaders, necessitating a convergence to a single leader state. Therefore, it becomes crucial for database systems employing replication to employ deterministic recovery from failure scenarios. This ensures the integrity and consistency of replicated data, thereby maintaining the reliability and effectiveness of the replication process.</p><p>Replication in databases enhances availability by creating redundancy through the distribution of data across multiple nodes. Additionally, it positively impacts the throughput of both reads and writes. It also leads to problems such as stale reads , write conflicts and so on. Furthermore, we will delve into the details of different types of replication and the mechanisms employed to resolve conflicts.</p><p>There are replication has following types:</p><p><strong>1) Single leader replication </strong>— Writes are directed to a designated leader, which then replicates the data to followers. These followers essentially mirror the leader. For instance, relational databases commonly support this replication model, ensuring strong consistency, which means deterministic values during multiple reads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/231/1*uK0sgMDV98uWp6HLgIvXsg.png" /></figure><p><strong>2) Multi leader replication </strong>— A system may have multiple leaders, each capable of accepting write operations, while reads are directed to followers of those leaders. In scenarios where there are multiple regions, each region may have its own set of leaders to enhance read-write performance. However, this setup requires the leaders to replicate data and manage conflicts accordingly. For example, a relational database deployed across multiple regions, such as Asia-Pacific and North America clusters, might have their own designated leaders. Within a region the reads will be consistent whereas across regions they will eventually get consistent for replicated tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*LW7Z2ECO1nJULMEuQ06b9A.png" /></figure><p><strong>3) Leaderless replication </strong>— In this system , each node possesses the ability to handle both read and write operations autonomously. Through continual communication and data exchange among nodes, eventual consistency is attained, ensuring that all nodes converge to a unified and up-to-date state over time. In this replication system, quorums play a crucial role in ensuring consistency and availability. A quorum is a subset of nodes that must agree on a particular operation (read or write) before it’s considered successful. Quorums are typically defined in terms of the total number of replicas in the system. This replication strategy is commonly employed in NoSQL databases like Cassandra and Riak, enabling them to scale horizontally while maintaining robustness and fault tolerance. In cases where multi-region replication, all nodes in each region need to have data consistency before replicating across regions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*mCrM7TzLUJiWHntlBSuNew.png" /></figure><p>The main problem with leader based replications is there are bottlenecked by leader node’s ability to write data faster and replicate it quickly enough across follower node. This guarantees strong consistency ( read on every node give same data but possibly stale ) and further writes can be synchronous to followers ( making it strongly consistent ) or loosen it by replicating to followers asynchronously ( leading to stale read at times ). Leaderless replication comes at the cost of faster reads and writes but compromises on data consistency ( causality of writes, eventual consistency on data across nodes and so on ).</p><p>Expanding on this, in distributed systems, failure is inherent and replication add more failure scenarios such as Leader nodes may crash, followers may fail, network glitches may prevent reads and writes from reaching their destinations, and the network can become overloaded with the volume of data to be replicated and so-on. This complexity is why I consider replication is inherently chaotic. The rest of the write-up discusses various failures scenarios and their recovery.</p><h3>How to handle failures OR divergent records in replicated systems ?</h3><p><strong>Single leader replication — </strong>When a leader fails, one of the follower node gets elected as leader. The failure is identified either through a quorum of follower reporting incommunicable leader node ( say heartbeats from leader is not received ) OR identified through coordinating services like Zookeeper. A new leader gets elected through either a consensus based algorithms ( Paxos or Raft ) OR managed through coordination services like Zookeeper. The mentioned leader resolution mechanisms also address the worst-case scenario where the system encounters a split-brain situation, resulting in two leaders. Its worth noting that ZooKeeper often incurs higher resource needs and operational overhead compared to Paxos or Raft-based systems.</p><p>While not a failure scenario, single leader systems can result in stale reads due to replication lag caused by asynchronous replication at follower nodes of the writes performed at the leader node. Although synchronous replication can solve this issue, it often leads to significantly reduced read and write throughput.</p><p><strong>2) Multi leader / Leaderless replication</strong> — More chaos reigns here, it encompasses all the challenges of single-leader replication while introducing additional issues. What we discuss here also applies to leaderless replication, with one unique consideration regarding consistency of reads and writes, which we will address separately.</p><p>When multiple leaders accept writes, determining the latest write becomes a significant challenge, especially when parallel writes occur, resulting in write conflicts. Before providing resolutions for write conflicts, we must first establish a mechanism to identify their occurrence. While a naive approach might involve taking a lock, such a method could lead to disastrous system performance.</p><p>Identifying write conflicts and implementing resolution techniques is crucial for maintaining data consistency in distributed systems with multiple leaders. Here are some mechanisms for identifying write conflicts and techniques for resolving them:</p><p><strong><em>a) Using</em></strong><em> </em><strong><em>Timestamp — </em></strong>Assigning timestamps to write operations allows systems to track the order in which writes occur. Conflicts can be detected when two writes to the same data item have conflicting timestamps. Resolution techniques may involve prioritising writes based on timestamps. However local node timestamp is not fully reliable due to — time stamps can drift over a period of time or can be tampered. Alternatively, Network Time Protocol(NTP) can be used as authoritative source. but this time is dependent on network delay for timestamp to reach the node.</p><p>While timestamps provide a reference for ordering events, they are not always entirely reliable. Consequently, resolution mechanisms like <em>Last Write Wins </em>may be utilised. However, this approach can lead to side effects such as lost writes.</p><p><strong><em>b) Using version vector</em></strong> — This mechanism is useful for identifying the following scenarios of writes: a) determining whether a write on the current leader node occurred before or after writes on other participating nodes — this is straightforward to resolve; b) resolving occurrences of parallel writes, which are harder to resolve and may require assistance from the user or a user-defined mechanism for conflict resolution. To aid in conflict resolution, the database will store siblings of each write the node has observed for itself and utilise that information for resolution</p><p>Some databases offer automatic write-conflict resolution through a) CRDT( Conflict Free Replicated data types) — these rely on data satisfying certain mathematical properties such as commutative , associative or idempotence. b) OT ( Operational Transform ) — this is popularised by conflict resolution in Google docs’s parallel editing feature. Here operation performed on the doc is sent between two nodes ( where parallel editing is happening ) and transform of operations are done before applying the operations to local node.</p><p><strong><em>c) Read repair </em></strong>is a mechanism designed to identify divergent records during read operations. When an application reads data from multiple replicas (e.g., Quorum reads in the case of leaderless replication), it updates the node that holds divergent data. If data is not read, it may never receive updates.</p><p><strong><em>d) Anti-entropy</em></strong> is a mechanism that identifies data divergence between two nodes by exchanging Merkle trees. This approach is particularly useful as it sends only divergent data to nodes, rather than the entire database, thereby reducing the load on the network. This mechanism typically runs in background and data convergence across nodes will depend on frequency of anti-entropy process.</p><p><strong><em>e)</em></strong> In leaderless systems, quorum writes are a common mechanism for achieving consistency among replicas. However, in the event of node failures where the quorum requirement cannot be met, maintaining availability becomes a priority over strict consistency. Therefore, in favour of ensuring system availability, the system may allow writes to proceed even if the quorum requirement cannot be fully satisfied . This we call as <strong><em>Sloppy quorum</em></strong> which is available as configurable feature in Riak, Cassandra and Dynamo.</p><p><strong><em>Hinted handoff</em></strong> is a mechanism where writes intended for a failed node are temporarily accepted by another available node, ensuring uninterrupted write operations. This allows the system to continue functioning even when nodes fail. However, hinted data usually comes with an expiration time limit to prevent indefinite storage. Once the failed node recovers, the temporarily holding node forwards the writes back.</p><h3>How replication data is sent across nodes ?</h3><p>The Replication Log mechanism is preferred over others for its ability to transmit actual operations on keys, ensuring accurate replication. Write-Ahead Logging (WAL) is less suitable because it specifies the address for data writes, which might not align with replication involving heterogeneous databases. Sending query statements is also impractical, as not all statements are deterministic, making it impossible to execute queries reliably at the destination database. Therefore, Replication Log is most effective method for ensuring consistent replication.</p><h3>Commercial databases and their associated support ?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FqQJeZMnRrBJJyZXYVzmkA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ea298c27730" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Big data and systems that process it.]]></title>
            <link>https://medium.com/@4nguru/big-data-and-systems-that-process-it-0a8dc969fff2?source=rss-dc3f5abce4bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a8dc969fff2</guid>
            <dc:creator><![CDATA[Gururaj Narayanan]]></dc:creator>
            <pubDate>Sat, 06 Apr 2024 02:54:38 GMT</pubDate>
            <atom:updated>2024-04-06T02:54:38.465Z</atom:updated>
            <content:encoded><![CDATA[<p>In the era of big data, there exists a vast amount of data, both structured and semi-structured, requiring efficient systems capable of analysing such massive volumes and providing insights for applications like recommendation systems, fraud detection, social media analytics, and machine learning systems, among others.</p><p>Let’s examine the needs of big data processing that cannot be easily met with regular NoSQL (or SQL) databases:</p><ol><li><strong>Processing Large Volumes of Data with Complex Operations:</strong> Big data processing often involves handling large volumes of data requiring complex filtering, sorting, and aggregation operations. While NoSQL databases excel at managing low-cardinality datasets and real-time processing, they may struggle with the scale and complexity of such operations.</li><li><strong>Performing Large Table Scans on Consistent Data: </strong>Big data processing tasks may require performing large table scans on strongly consistent data. NoSQL databases are optimised for low-cardinality records and may not be well-suited for efficiently handling such scenarios.</li><li><strong>Handling High Write Throughput:</strong> In many cases, big data processing involves handling large amounts of writes, which can pose challenges for ensuring eventual consistency in NoSQL databases. Enforcing consistency rules in such scenarios can introduce significant overhead, potentially slowing down the processing pipeline.</li></ol><p>In summary, while NoSQL databases are suitable for use cases such as real-time processing, they may face limitations when it comes to processing large volumes of data with complex operations, performing large table scans on consistent data, and handling high write throughput. In such scenarios, specialised big data processing systems like Hadoop may be more appropriate for efficiently analysing and extracting insights from massive datasets.</p><p>Let’s dive deeper into Hadoop, a popular Big data system. The diagram below illustrates the Hadoop system and its integration with external systems for data reading, processing, and writing. Components highlighted in orange represent core elements of the Hadoop system, while the remaining components signify add-ons aimed at enhancing Hadoop’s capabilities. Rather than delving extensively into general Hadoop system explanations, I will concentrate on the key attributes of Hadoop in contrast with Cassandra(NoSQL). They focus is to answer why Hadoop is more appropriate than Cassandra for large data processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*qUF_APzT7t42VKy8PWhDCQ.png" /></figure><p>We will describe Hadoop’s capabilities and deficiencies in comparison with another database called Cassandra. Both are capable of handling large data-set but considerations of CAP theorem, data processing methods sets them apart.</p><ol><li>CAP Theorem: Both Hadoop and Cassandra prioritise availability and partition tolerance over consistency. In Cassandra, consistency levels can be tuned, whereas in Hadoop, eventual consistency is achieved as partitions become consistent over time. Hadoop’s consistency can be enforced through the modelling of MapReduce workflows, which is typically determined by user requirements.</li><li>Hadoop’s suitability for real-time data edits is limited due to its reliance on HDFS, where data is stored in standard formats like Parquet or Avro. Real-time edits to specific rows in Hadoop often involve time-consuming seek operations on physical disks and may result in entire file writes. In contrast, Cassandra utilizes the LSM-Tree/SS table model, which facilitates easy writes. Additionally, Cassandra provides indexing mechanisms for faster data access, a feature lacking in Hadoop. Reading potentially involves accessing multiple SS-Tables and possibly spanning across multiple partitions in Cassandra, which can lead to slower data retrieval compared to Hadoop. In Hadoop, data is typically co-located with the processing nodes, resulting in faster data delivery throughput to processing systems such as MapReduce or Apache Spark. This co-location minimises data transfer overhead and latency, enhancing the efficiency of data processing workflows</li><li>Hadoop leverages MapReduce for large-scale data analytics through batch processing. While MapReduce was once the standard for such tasks, Apache Spark has emerged as a faster alternative, thanks to its memory-based processing. Although MapReduce is not integrated to run directly on Cassandra, Spark can operate on Cassandra data and provide analytics and batch processing capabilities akin to MapReduce. Consequently, Cassandra with Spark offers functionality similar to Hadoop with MapReduce, with Hadoop being more suitable for batch processing of large data volumes, while Spark combined with Cassandra excels in real-time data processing and analytics. Ultimately, the choice between the two depends on specific data and processing requirements, necessitating careful consideration of use case needs.</li></ol><p>With the above context, the below picture representing internals of Hadoop explains how Hadoop architecture lends itself for big data processing such as a) Processing happens on the same node (Data Node ) where files to be processed are stored b) Map-reduce for complex parallel processing for analytics c) Strong consistency through Name Node. Now picture this in comparison to Cassandra internals to visualise why Hadoop is suitable for big data batch processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*CU1i8zfY6GuB8-1ABOEmEA.png" /></figure><p>In conclusion, big data analytics often complement real-time analytics in data-driven environments. While offline analytics systems like Hadoop tend to offer more precise insights compared to the probabilistic outputs of real-time analytics, both play critical roles in user exeperince. For instance, systems recommending currently trending videos based on user history could leverage Cassandra along with probabilistic estimation to provide real-time video recommendations, ensuring users have access to the latest trending content. Simultaneously, offline analysis facilitated by Hadoop, running jobs on a daily basis (hypothetically), can refine inputs for the recommendation system. This offline analysis may involve training machine learning models that drive the recommendation system.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a8dc969fff2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>