<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yashwant Sanjay Saste on Medium]]></title>
        <description><![CDATA[Stories by Yashwant Sanjay Saste on Medium]]></description>
        <link>https://medium.com/@yashwantsaste?source=rss-384f60710a3f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BPVlShl2pdTRz8iM0XCaGw.jpeg</url>
            <title>Stories by Yashwant Sanjay Saste on Medium</title>
            <link>https://medium.com/@yashwantsaste?source=rss-384f60710a3f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:17:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yashwantsaste/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Rate Limiting Algorithms Explained]]></title>
            <link>https://medium.com/@yashwantsaste/rate-limiting-algorithms-explained-de8556ceb044?source=rss-384f60710a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/de8556ceb044</guid>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[rate-limiting]]></category>
            <category><![CDATA[high-level-design]]></category>
            <category><![CDATA[rate-limiting-algorithms]]></category>
            <category><![CDATA[information-security]]></category>
            <dc:creator><![CDATA[Yashwant Sanjay Saste]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 21:50:24 GMT</pubDate>
            <atom:updated>2026-10-17T21:50:24.536Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p6IevyjZoX2QRHy44H9a0Q.png" /></figure><h3>What Is a Rate Limiter?</h3><p>A <strong>rate limiter</strong> is a mechanism that controls how many requests a client can send to a service within a defined time interval. It tracks incoming requests against a configured limit and decides whether to allow or reject each request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mnWJvg8RC14nKqYltpHMsQ.png" /></figure><p>For example, an API may allow 100 requests per minute per client. Once the client reaches this limit, further requests may be rejected with an HTTP 429 Too Many Requests response until the configured policy allows more requests.</p><p>Rate limiting helps protect backend services from excessive traffic, prevents resource misuse, and ensures fair access to APIs.</p><h3>Where Should a Rate Limiter Be Implemented?</h3><p>A rate limiter can be implemented on the client side, on the server side, or as middleware between clients and API servers.</p><h4>Client-Side Rate Limiting</h4><p>The client controls how frequently it sends requests. However, clients can bypass these limits, so client-side rate limiting alone cannot protect an API.</p><h4>Server-Side Rate Limiting</h4><p>The server checks incoming requests against configured limits and decides whether to allow or reject them. Since the API provider controls the implementation, this approach provides reliable enforcement.</p><h4>Middleware or API Gateway</h4><p>A rate limiter can run as a separate component between clients and API servers. It checks incoming requests before forwarding allowed requests to the backend.</p><p>For example, if an API allows two requests per second per client, the first two requests are forwarded to the API servers, while the third is rejected with an HTTP 429 Too Many Requests response.</p><p>This approach is useful for applications with multiple API servers because requests can be checked at a common entry point.</p><h3>Rate Limiting Algorithms</h3><p>Rate-limiting algorithms use different approaches to track incoming requests and decide whether to allow or reject them. The following five algorithms differ in how they manage request counts, time windows, and traffic bursts.</p><h3>1. Token Bucket Algorithm</h3><p>The Token Bucket algorithm is a widely used rate-limiting algorithm. It controls incoming requests using a bucket that holds a limited number of tokens. Tokens are added at a configured rate, and each request consumes one token.</p><h4>How Does It Work?</h4><ul><li><strong>Bucket capacity:</strong> The bucket has a predefined maximum capacity. For example, if its capacity is 4, it can hold at most 4 tokens.</li><li><strong>Token refill:</strong> Tokens are added at a fixed rate, such as 2 tokens per second. If the bucket is full, additional tokens are discarded.</li><li><strong>Request handling:</strong> When a request arrives, the rate limiter checks whether a token is available. If one is available, it is consumed and the request is allowed.</li><li><strong>Request rejection:</strong> If no tokens are available, the request is rejected.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6juV0Fc6ZoJONqTIQCllig.png" /><figcaption><em>This diagram illustrates how tokens are added to the bucket and how excess tokens are discarded when the bucket reaches its capacity</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YYrQVL4pkeZ_T5ZmSnYEsg.png" /><figcaption><em>This diagram shows how requests are allowed when tokens are available and rejected when the bucket is empty.</em></figcaption></figure><p>Suppose the bucket capacity is 4 tokens and the refill rate is 4 tokens per minute.</p><ul><li>At 1:00:00, the bucket contains 4 tokens.</li><li>At 1:00:05, three requests arrive and consume 3 tokens, leaving 1 token.</li><li>At 1:00:20, another request consumes the remaining token. Any additional request is rejected until a token becomes available.</li><li>At 1:01:00, the bucket can refill up to its capacity of 4 tokens.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*idg9O1oIpOC0fijWDZj7eg.png" /><figcaption><em>This diagram illustrates how requests consume tokens and how the bucket refills over time.</em></figcaption></figure><h4>Parameters</h4><p>The Token Bucket algorithm uses two main parameters:</p><ul><li><strong>Bucket size:</strong> The maximum number of tokens the bucket can hold.</li><li><strong>Refill rate:</strong> The number of tokens added to the bucket over a specified period.</li></ul><p>The number of buckets depends on the rate-limiting rules. For example, separate buckets may be used for different API endpoints or individual IP addresses. A global bucket can be used when the system needs to limit the total number of requests across all clients.</p><h4>Advantages</h4><ul><li>Easy to implement and memory-efficient.</li><li>Allows short bursts of traffic when tokens are available.</li></ul><h4>Limitations</h4><ul><li>Choosing the right bucket size and refill rate requires careful tuning.</li></ul><h3>2. Leaky Bucket Algorithm</h3><p>The <strong>Leaky Bucket algorithm</strong> is a rate-limiting algorithm that processes requests at a fixed rate. It is usually implemented using a First-In, First-Out (FIFO) queue, where incoming requests wait until they can be processed.</p><p>Unlike the Token Bucket algorithm, which allows bursts when tokens are available, the Leaky Bucket algorithm controls the rate at which requests leave the queue.</p><h4>How Does It Work?</h4><ul><li><strong>Request arrival:</strong> When a request arrives, the system checks whether the queue has space.</li><li><strong>Queue capacity:</strong> If the queue is not full, the request is added to the queue.</li><li><strong>Request rejection:</strong> If the queue is full, the incoming request is rejected.</li><li><strong>Request processing:</strong> Requests are removed from the queue and processed at a fixed rate, one after another.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ED01Z4fVg895k3NuF5MPEg.png" /><figcaption><em>This diagram illustrates how incoming requests are queued, processed at a controlled rate, and rejected when the queue reaches its capacity.</em></figcaption></figure><p>Suppose a rate limiter has a queue capacity of 4 requests and processes 2 requests per second.</p><ul><li><strong>At 1:00:00:</strong> Four requests arrive and enter the queue.</li><li><strong>At 1:00:00.2:</strong> Another request arrives, but the queue is full, so the request is rejected.</li><li><strong>At 1:00:01:</strong> Two requests are processed, freeing space in the queue.</li><li><strong>After that:</strong> New requests can enter the queue as space becomes available, while queued requests continue to be processed at the configured rate.</li></ul><h4>Parameters</h4><p>The Leaky Bucket algorithm has two main parameters:</p><ul><li><strong>Bucket size:</strong> The maximum number of requests the queue can hold.</li><li><strong>Outflow rate:</strong> The rate at which requests are processed, such as 2 requests per second.</li></ul><h4>Advantages</h4><ul><li>Memory-efficient because the queue has a fixed capacity.</li><li>Provides a steady request-processing rate.</li><li>Helps prevent sudden traffic bursts from overwhelming backend services.</li></ul><h4>Limitations</h4><ul><li>A burst of requests can fill the queue, causing newer requests to be rejected even if they are valid.</li><li>Requests may experience delays while waiting in the queue.</li><li>Choosing the right queue size and processing rate requires careful tuning.</li></ul><h3>3. Fixed Window Counter Algorithm</h3><p>The <strong>Fixed Window Counter algorithm</strong> divides time into fixed intervals and maintains a separate request counter for each interval. It allows requests up to a configured limit within each window. Once the limit is reached, additional requests are rejected until the next window begins.</p><h4>How Does It Work?</h4><ul><li><strong>Time windows:</strong> Divide time into fixed intervals, such as 1 second or 1 minute.</li><li><strong>Request counting:</strong> Each incoming request increments the counter for the current window.</li><li><strong>Request allowance:</strong> If the counter is below the configured limit, the request is allowed.</li><li><strong>Request rejection:</strong> Once the limit is reached, additional requests are rejected until the counter resets at the start of the next window.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FaD4_RJsd4d2UzbZEU-Y0Q.png" /><figcaption><em>This diagram shows how requests are counted within each fixed time window and how requests exceeding the limit are rejected.</em></figcaption></figure><p>Suppose an API allows a maximum of 3 requests per second.</p><ul><li><strong>At 1:00:00:</strong> The counter starts at zero. The first three requests are allowed.</li><li><strong>At 1:00:00.5:</strong> Two more requests arrive. Both are rejected because the limit has already been reached.</li><li><strong>At 1:00:01:</strong> A new time window begins, and the counter resets to zero. The next three requests can be allowed.</li></ul><h4>The Boundary Problem</h4><p>One major limitation of this algorithm is that it can allow a sudden burst of requests around the boundary between two windows.</p><p>Suppose an API allows 5 requests per minute:</p><ul><li><strong>From 2:00:00 to 2:00:59:</strong> Five requests are allowed.</li><li><strong>Just after 2:01:00:</strong> The counter resets, allowing another five requests.</li></ul><p>A client could send five requests just before 2:01:00 and five more immediately after it. This means <strong>10 requests could be allowed within a very short period</strong>, even though the configured limit is 5 requests per minute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*60acrhmsY3Xj8m8M2fGB7Q.png" /><figcaption><em>This diagram illustrates how requests near a window boundary can exceed the intended limit over a short period.</em></figcaption></figure><h4>Advantages</h4><ul><li>Easy to understand and implement.</li><li>Memory-efficient because only a counter and window information need to be maintained.</li><li>Works well when limits reset at fixed intervals, such as per-minute quotas.</li></ul><h4>Limitations</h4><ul><li>Traffic bursts around window boundaries can allow more requests than intended over a short period.</li><li>The counter resets suddenly at the beginning of each new window, rather than tracking requests across a rolling time period.</li></ul><h3>4. Sliding Window Log Algorithm</h3><p>The <strong>Sliding Window Log algorithm</strong> addresses the boundary problem of the Fixed Window Counter algorithm by tracking the timestamps of individual requests. Instead of resetting a counter at fixed intervals, it considers requests within a rolling time window.</p><p>For example, if an API allows 2 requests per minute, the algorithm checks how many requests the client has made during the last 60 seconds before deciding whether to accept a new request.</p><h4>How Does It Work?</h4><ul><li><strong>Store timestamps:</strong> Maintain a log of request timestamps, often using an in-memory data store such as Redis.</li><li><strong>Remove outdated timestamps:</strong> When a new request arrives, remove timestamps that fall outside the current time window.</li><li><strong>Record the request:</strong> Add the timestamp of the incoming request to the log.</li><li><strong>Check the limit:</strong> If the number of recorded requests is within the configured limit, accept the request. Otherwise, reject it.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wpiqYEVOPiNuoENN9WSZkg.png" /><figcaption><em>This diagram illustrates how the algorithm removes expired timestamps and checks the number of requests within the current rolling window.</em></figcaption></figure><p>Suppose an API allows a maximum of 2 requests per minute.</p><ul><li><strong>At 1:00:01:</strong> The log is empty. The request timestamp is added, and the request is accepted. Log size: 1.</li><li><strong>At 1:00:30:</strong> The timestamp is added. There are now 2 requests within the window, so the request is accepted. Log size: 2.</li><li><strong>At 1:00:50:</strong> The timestamp is added, bringing the count to 3. Since this exceeds the limit, the request is rejected.</li><li><strong>At 1:01:40:</strong> The algorithm removes timestamps older than 1:00:40. The timestamps from 1:00:01 and 1:00:30 are removed, leaving the recent timestamp from 1:00:50. The new request is added and accepted because the count is now 2.</li></ul><p>The key point is that the algorithm checks a <strong>rolling 60-second window</strong>, not a fixed clock-based minute.</p><h4>Advantages</h4><ul><li>Accurate rate limiting because every request is evaluated against a rolling time window.</li><li>Avoids the boundary problem of the Fixed Window Counter algorithm.</li><li>Provides precise control over how many requests are allowed during a given period.</li></ul><h4>Limitations</h4><ul><li>Requires more memory because it stores individual request timestamps rather than just a counter.</li><li>Processing each request involves removing expired timestamps and checking the log.</li><li>Under high traffic, storing and managing large numbers of timestamps can become expensive.</li></ul><h4>When Is It Useful?</h4><p>The Sliding Window Log algorithm is useful when precise rate limiting is important, such as protecting sensitive API endpoints or enforcing strict per-client request limits. However, its memory overhead can make it less suitable for systems handling very high request volumes.</p><h3>5. Sliding Window Counter Algorithm</h3><p>The <strong>Sliding Window Counter algorithm</strong> combines the simplicity of the Fixed Window Counter algorithm with the flexibility of a sliding time window. Instead of storing every request timestamp, it uses counters from the previous and current windows to estimate the number of requests in the rolling window.</p><p>This approach reduces memory usage while helping address the traffic spikes that can occur at fixed window boundaries.</p><h4>How Does It Work?</h4><ul><li><strong>Count requests:</strong> Maintain separate counters for the previous and current fixed time windows.</li><li><strong>Calculate the overlap:</strong> Determine how much of the previous window falls within the current rolling window.</li><li><strong>Estimate the request count:</strong> Multiply the previous window’s request count by its overlap percentage, then add the current window’s count.</li><li><strong>Check the limit:</strong> Allow the incoming request if the estimated count remains within the configured limit. Otherwise, reject it.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*srMgdMpry6jvWe7WE79A5g.png" /><figcaption><em>This diagram illustrates how the algorithm combines the current window’s request count with a weighted portion of the previous window’s count.</em></figcaption></figure><p>Suppose an API allows a maximum of 7 requests per minute. At a particular moment:</p><ul><li>Previous minute: 5 requests.</li><li>Current minute: 3 requests.</li><li>Elapsed time in the current minute: 30 seconds.</li><li>Remaining portion of the previous minute included in the rolling window: 50%.</li></ul><p>The estimated request count is calculated as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8lw8ZlrXG5L4SHx9x_IhxA.png" /></figure><p>If the implementation rounds down, the estimated count is 5. Since this is below the limit of 7, the new request can be accepted.</p><p><strong>Note:</strong> This is an estimate, not an exact count, because the algorithm does not know precisely when requests occurred within the previous window.</p><h4>Advantages</h4><ul><li><strong>Memory-efficient:</strong> Stores window counters instead of individual request timestamps.</li><li><strong>Reduces boundary spikes:</strong> Considers traffic from the previous window rather than resetting the effective count abruptly.</li><li><strong>Practical for high traffic:</strong> Offers a useful balance between accuracy and resource usage.</li></ul><h4>Limitations</h4><ul><li><strong>Approximate results:</strong> Assumes requests in the previous window were distributed evenly over time, which may not reflect actual traffic.</li><li><strong>Potential inaccuracies:</strong> Some requests may be allowed or rejected incorrectly near the configured limit.</li><li><strong>Less suitable for strict limits:</strong> Systems requiring an exact count of requests in every rolling window may need a more precise algorithm.</li></ul><h4>When Is It Useful?</h4><p>The Sliding Window Counter algorithm is useful for APIs that need efficient rate limiting without storing every request timestamp. It is a practical choice when lower memory usage and smoother traffic control are more important than maintaining an exact rolling-window count.</p><h3>Conclusion</h3><p>Each rate-limiting algorithm has its own trade-offs:</p><ul><li><strong>Token Bucket:</strong> Allows short bursts while controlling the average request rate.</li><li><strong>Leaky Bucket:</strong> Processes requests at a steady rate to smooth traffic.</li><li><strong>Fixed Window Counter:</strong> Simple and memory-efficient, but can allow bursts around window boundaries.</li><li><strong>Sliding Window Log:</strong> Provides accurate rolling-window limits but requires more memory to store timestamps.</li><li><strong>Sliding Window Counter:</strong> Estimates rolling request counts with less memory, balancing efficiency and accuracy.</li></ul><p>The right choice depends on your traffic patterns, accuracy requirements, memory constraints, and system architecture. Understanding these algorithms is the first step toward building reliable rate limiting.</p><p>In the <strong>next blog</strong>, we’ll move from algorithms to a high-level system design, exploring how to build a scalable rate-limiting service, where to enforce limits, how to manage distributed counters, and how components such as Redis and API gateways fit into the architecture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de8556ceb044" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Event Streaming Platform From Scratch in Rust]]></title>
            <link>https://medium.com/@yashwantsaste/building-a-event-streaming-platform-from-scratch-in-rust-548fac6705cf?source=rss-384f60710a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/548fac6705cf</guid>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[event-streaming]]></category>
            <category><![CDATA[rust-programming-language]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <dc:creator><![CDATA[Yashwant Sanjay Saste]]></dc:creator>
            <pubDate>Tue, 04 Aug 2026 06:50:49 GMT</pubDate>
            <atom:updated>2026-08-09T19:17:41.468Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gxdpA81N-ikz9iHrDXbHWg.png" /></figure><p>I wanted to understand event streaming well enough to build one, not just configure one. So I wrote a broker, a producer, a consumer, and a CLI in Rust, all talking over a hand-rolled binary protocol, with data surviving a restart.</p><p>I’ve used Kafka before, but I wanted to better understand what happens behind the scenes. Building a simple message broker seemed like a good way to learn. Instead of only learning how to use Kafka, I wanted to understand how a message broker works internally. This project was part of that learning process.</p><p>My goal is now to build aworking Rust event streaming platform with topics split into partitions, producers that append records, consumers that read them back with consumer-group offsets, and everything persisted to disk so a restart doesn’t lose data. Not just in-memory system but a using network layer and a real storage layer, and structured as separate crates</p><blockquote>Project GitHub Repo Link: <a href="https://github.com/YashwantSaste/Event-Streaming-Proj-Rust">https://github.com/YashwantSaste/Event-Streaming-Proj-Rust</a></blockquote><h3>Concepts I Learned First</h3><ul><li><strong>Event streaming</strong> — producers append immutable records to a log; consumers read that log independently, at their own pace.</li><li><strong>Broker-topic-partition model</strong> — a broker hosts topics; each topic splits into partitions so writes parallelize and stay ordered <em>within</em> a partition.</li><li><strong>Producer-consumer flow</strong> — producers only append, consumers only read and track their own position; neither blocks the other.</li><li><strong>Offsets and consumer groups</strong> — a partition is a sequence of offsets. A group commits its last processed offset so a restarted consumer resumes instead of re-reading everything.</li><li><strong>Persistence and recovery</strong> — metadata, records, and offsets must survive a process restart, which means writing state you can also reconstruct from disk.</li><li><strong>Async networking in Rust</strong> — a broker needs to serve many connections without a thread per client, which is where tokio came in.</li></ul><h3>System Architecture I Built</h3><p>I split the project into a Cargo workspace with five crates, each with one job:</p><ul><li><strong>common</strong> — the wire protocol (encoder/decoder, request/response types), domain models (Topic, Partition, Record, ConsumerGroup), TOML/env configuration, a file-system abstraction, and per-domain errors.</li><li><strong>broker</strong> — topic management, partition management, a local storage engine, consumer-group/offset tracking, and the network layer.</li><li><strong>producer</strong> — client library and binary for appending records.</li><li><strong>consumer</strong> — client library and binary for polling records and committing offsets.</li><li><strong>cli</strong> — the entry point that ties everything together.</li></ul><p>Keeping common dependency-free (no tokio, no I/O assumptions) meant the protocol and models were reused identically by the broker and both clients — nobody re-encoded requests by hand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WRzzEJLCf8K3K-3RUQ5vow.png" /></figure><h3>Step-by-Step Implementation process</h3><ol><li><strong>Protocol first.</strong> A binary header with a fixed length, a request-type enum (Produce, Fetch, CreateTopic, ListTopics, CommitOffset), and an encoder/decoder pair — so the broker and clients could be built in parallel against the same contract.</li><li><strong>Domain models and errors.</strong> Topic, TopicPartition, Record, Offset, and per-domain error types gave every crate a shared vocabulary instead of raw strings and tuples.</li><li><strong>Storage engine.</strong> LocalStorageEngine appends records to segment files (segment-NNNNNN.log) capped at a configurable size; when a segment fills, a new one is created.</li><li><strong>Topic and partition managers.</strong> TopicManager persists topic metadata via TopicMetadataStore; PartitionManager exposes append and read by topic and partition ID.</li><li><strong>Consumer groups and offsets.</strong> ConsumerGroupManager and OffsetStore track the last committed offset per group and partition, persisted the same way as everything else.</li><li><strong>Broker runtime and dispatcher.</strong> BrokerRuntime matches a decoded Request against its payload type, calls the right manager, and wraps the result in a Response.</li><li><strong>Network layer.</strong> BrokerServer binds a TcpListener, accepts connections in a loop, and spawns a tokio::spawn task per client. Each task reads length-prefixed frames, dispatches through a shared Mutex-guarded runtime, and writes the response back.</li><li><strong>Producer and consumer clients.</strong> Thin TCP wrappers using the same codec from common, so there&#39;s no protocol drift between server and client.</li><li><strong>CLI.</strong> CommandParser turns argv into a ParsedCommand, CommandFactory maps it to a command struct (init, start-broker, create-topic, list-topics, produce, consume), and network-touching commands spin up a small tokio runtime and block_on just that call.</li></ol><h3>Demo Flow of the Final System</h3><ol><li>init — creates a workspace with a default broker.toml and data directories.</li><li>start-broker — loads configuration, recovers existing topics and partitions from disk, and starts listening.</li><li>create-topic --topic orders --partitions 3 — the broker persists metadata and creates partition directories.</li><li>list-topics — confirms the topic exists by asking the broker, not by reading local files.</li><li>produce --topic orders --partition 0 --message &quot;order-42&quot; — the broker appends the record and returns the assigned offset.(Else for live terminal use case you can just enter produce --topic orders --partition 0)</li><li>consume --topic orders --group checkout-service — fetches from the group&#39;s last committed offset, then commits the new one.</li><li>Restarting the broker and repeating step 6 proves persistence: the consumer resumes exactly where it left off.</li></ol><h3>Flow Diagram of Current state:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qiymHD4TQMfRjiEbYEzz0w.png" /></figure><h3>Rust Memory Management Decisions and Trade-offs</h3><p><strong>Length-prefixed binary framing over TCP, not HTTP.</strong> Simple to read with read_exact, avoids an HTTP stack for what&#39;s fundamentally an RPC — at the cost of writing my own encoder/decoder for every message.</p><p><strong>One shared </strong><strong>Mutex around the broker runtime.</strong> Easy to reason about — only one request dispatches at a time — at the cost of throughput. Production would shard locks per partition; I chose simplicity first.</p><p><strong>Generic storage/file-system traits, not fixed types.</strong> TopicManager&lt;F&gt; and PartitionManager&lt;S&gt; are generic over FileSystem and StorageEngine, so tests can swap in an in-memory implementation without touching broker logic, at the cost of more verbose type signatures.</p><p><strong>Segment-based append-only logs, not one growing file.</strong> Capping segments keeps files bounded and makes recovery (scanning file names for the latest segment) simple, at the cost of needing a naming scheme (SegmentNamer).</p><p><strong>A synchronous CLI wrapping an async core.</strong> Only network calls need to be async, so each command builds a small tokio runtime and block_ons just that operation — keeping the command layer easy to read.</p><h3>Challenges, Bugs, and How I Solved Them</h3><p><strong>Partial TCP reads.</strong> I assumed one read() would return a full message; TCP doesn&#39;t guarantee that. Fixed by reading the 4-byte length prefix with read_exact, then reading exactly the remaining frame length in a second call.</p><p><strong>Offsets resetting after a restart.</strong> Partition state lived only in memory, so restarting reset offsets to zero and risked overwriting segment data. Fixed by having LocalStorageEngine recover state on first access — scanning the segment directory, finding the highest segment ID, and reading its size to reconstruct next_offset.</p><p><strong>One slow client blocking everyone.</strong> Handling requests inline in the accept loop meant one client stalled all others. Fixed with tokio::spawn per connection, sharing the runtime behind an Arc&lt;Mutex&lt;_&gt;&gt; so only dispatch itself is serialized.</p><p><strong>Oversized or malformed frames.</strong> A corrupt length prefix could request a huge allocation. Fixed with a max_frame_bytes check that rejects any frame over the configured limit before allocating a buffer.</p><p><strong>Offset commits racing with fetches.</strong> I initially let a consumer commit before confirming a batch was processed, risking lost records on a crash. Fixed by making commit an explicit, separate step the caller invokes only after handling the batch.</p><h3>Future Scopes:</h3><ul><li><strong>Replication</strong> — one broker today; a event streaming system needs partition replicas for fault tolerance.</li><li><strong>Per-partition locking</strong> instead of one global runtime lock.</li><li><strong>Compaction and retention</strong> for segments instead of keeping every record forever.</li><li><strong>A versioned wire-format schema</strong> instead of hand-written encode/decode per message type.</li><li><strong>Consumer group rebalancing</strong> — nothing coordinates multiple consumers in the same group today.</li><li>Add Streaming and Data Pipelines - integrating external data platforms</li><li><strong>Better observability</strong> — structured logs and metrics instead of eprintln! on error.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=548fac6705cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Kafka Fundamentals: Events, Topics, Partitions, Producers, and Consumers]]></title>
            <link>https://medium.com/@yashwantsaste/apache-kafka-fundamentals-events-topics-partitions-producers-and-consumers-93c1b6c704b4?source=rss-384f60710a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/93c1b6c704b4</guid>
            <category><![CDATA[kafka-producer]]></category>
            <category><![CDATA[kafka-consumer]]></category>
            <category><![CDATA[kafka-partition]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[event-streaming]]></category>
            <dc:creator><![CDATA[Yashwant Sanjay Saste]]></dc:creator>
            <pubDate>Mon, 08 Jun 2026 18:28:32 GMT</pubDate>
            <atom:updated>2026-06-08T18:28:32.585Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pZ47c49MK9jgvi12gnGd9Q.png" /></figure><p>In the previous article, we focused on the fundamentals of Apache Kafka. We understood what events and event streams are, why event-driven systems are widely used, and how Kafka fits into this model as a platform for publishing, storing, and processing events.</p><p>In this article, we move one level deeper. Instead of focusing on concepts, we will look at how Kafka is built internally. We will explore its core components such as brokers, clusters, producers, and consumers, and understand how they work together to enable scalable and reliable event streaming.</p><h3>Fundamentals of Apache Kafka</h3><h3>1. Event</h3><p>In Kafka terminology, the basic unit of data is an <strong>event</strong>.</p><p>Event is nothing but represents a fact that something has occurred in the system. (For example: User Logged In, Order Created, Payment Done)It is immutable, meaning once it is written, it cannot be changed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*efGt1TYTG8PkghJf_7Evlg.png" /></figure><p>An event typically contains:</p><ul><li>Key</li><li>Value (actual data)</li><li>Timestamp</li><li>Headers (optional metadata)</li></ul><p>For example:</p><pre>{<br>  &quot;key&quot;: &quot;order-101&quot;,<br>  &quot;value&quot;: {<br>    &quot;orderId&quot;: &quot;101&quot;,<br>    &quot;amount&quot;: 500,<br>    &quot;status&quot;: &quot;CREATED&quot;<br>  },<br>  &quot;timestamp&quot;: &quot;2026-06-08T10:00:00Z&quot;<br>}</pre><p>Kafka treats each event as part of a continuous stream of data.</p><h3>2. Topic</h3><p>A <strong>topic</strong> is a logical category where events are stored.</p><blockquote>Very simplified, a topic is similar to a folder in a filesystem, and the events are the files in that folder.</blockquote><p>Every event published to Kafka is written into a topic. A topic acts like a named stream of related data.</p><p>For example:</p><ul><li>orders → all order-related events</li><li>payments → all payment-related events</li><li>user-activity → login, click, and navigation events</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f6ErZdP4nhrbYekN2RKDBw.png" /></figure><p>A topic does not store data in a single place. Instead, Kafka distributes topic data across multiple brokers for scalability.</p><p>Topics are <strong>partitioned</strong> , meaning a topic is spread over a number of “buckets” located on different Kafka brokers.</p><h3>3. Partition</h3><p>A <strong>partition</strong> is the fundamental unit of parallelism in Kafka.</p><p>Each topic is split into multiple partitions, and each partition is an ordered, immutable sequence of events.</p><p>Kafka uses partitions to scale horizontally.</p><p>If a topic has three partitions, different brokers can store different partitions, allowing Kafka to distribute load.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*89-WNXpUrcg7YuE4.png" /></figure><p>If Kafka stored all data in a single place, it would become a bottleneck. Partitions solve this by:</p><ul><li>Splitting data across multiple machines</li><li>Allowing parallel reads</li><li>Enabling high throughput</li></ul><p>Each partition maintains strict ordering, but ordering is only guaranteed within that partition.</p><p>Kafka guarantees that events inside a partition are stored in the exact order they are received.</p><p>However, ordering is not guaranteed across partitions.</p><p>This is why system design decisions (like choosing partition keys) are important.</p><h3>4. Offset</h3><p>Every event inside a partition has a unique identifier called an <strong>offset</strong>.<br>An offset represents the position of an event in the partition.</p><p>For example:</p><pre>Partition 0:<br>0 → OrderCreated  <br>1 → PaymentCompleted  <br>2 → InventoryReserved  <br>3 → ShipmentCreated</pre><p>Offsets allow Kafka to track progress and enable consumers to resume reading from where they left off.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mckZunwqeHvZzxUH_MhMiQ.png" /></figure><p>Offsets provide replay capability, fault tolerance, consumer state tracking<br>Consumers store their last processed offset so they can continue from the correct position after a failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HifSWpDVHSRD730bxudJqQ.png" /></figure><h3>5. Kafka Broker</h3><p>A <strong>Kafka Broker</strong> is a single Kafka server that stores data and serves client requests.</p><p>In simple terms, it is the <strong>worker node of Kafka</strong>.</p><p>A broker is responsible for:</p><ul><li>Receiving records from producers</li><li>Storing them in topic partitions (as logs on disk)</li><li>Serving records to consumers when they request them</li></ul><p>Each broker manages a portion of the overall data, which allows Kafka to scale horizontally.</p><p>A Kafka system is not built on one broker. It is built on <strong>multiple brokers working together</strong>.</p><h3>6. Kafka Cluster</h3><p>A <strong>Kafka Cluster</strong> is a group of Kafka brokers working together as a single system.</p><p>Instead of storing all data in one machine, Kafka distributes data across multiple brokers.</p><p>This design allows Kafka to handle large volumes of data, survive server failures (fault tolerance), scale by adding more brokers and balance load across the system</p><p>If one broker fails, the cluster continues working because data is replicated across other brokers.</p><h3>7. Kafka Producer</h3><p>A <strong>Kafka Producer</strong> is the component that sends data into Kafka. It is the entry point of all events in a Kafka system.</p><p>In a <strong>publish-subscribe model</strong>, the producer plays the role of the <strong>publisher</strong>. It creates events and publishes them to a Kafka topic. After that, Kafka takes responsibility for storing and distributing those events to consumers.</p><p>The producer does not know who will read the data. It only knows the topic where the data should go.</p><h4>How Producer Works Internally</h4><p>When an application sends data using a producer, it does not directly push data to consumers or even to a single storage location.</p><p>Instead, the process looks like this:</p><ol><li>Application creates an event</li><li>Producer converts it into a Kafka record</li><li>Record is sent to a topic</li><li>Kafka decides which partition should store it</li><li>Broker stores it in the partition log</li><li>Consumers read it later</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SjwfZObLiKb7zuGWJkIa2A.png" /></figure><p>Kafka uses a <strong>publish-subscribe model</strong>. Producers <strong>publish</strong> events while Kafka acts as the middle layer and consumers <strong>subscribe</strong> to topics</p><p>This means:</p><ul><li>One producer can send data once</li><li>Many consumers can read the same data independently</li></ul><p>For example:</p><ul><li>Order Service publishes OrderCreated</li><li>Payment, Inventory, and Notification services all consume it separately</li></ul><p>The producer does not need to know about any of these services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fhdrner4r9GX3AoLu24lcg.png" /></figure><p>Kafka Producer works in <strong>asynchronous mode by default</strong>.</p><p>When you send a record, it is not immediately written to Kafka; instead, it is first stored in a buffer and later sent in batches.</p><p>This batching mechanism improves performance because it reduces the number of network calls, allows multiple records to be sent together, and increases overall system throughput.</p><p>If every event were sent one by one, the system would become slow due to increased network overhead, and the overall throughput would drop significantly.</p><p>Batching makes Kafka suitable for high-volume systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gU7NjuNnpWRPJKUgx__deg.png" /></figure><p>Kafka allows producers to control how safe the write operation should be using <strong>acknowledgements</strong>.</p><p>ACKs define when a producer considers a message as successfully sent.</p><ul><li><strong>acks = 0</strong></li></ul><p>-&gt; Producer does not wait for confirmation<br>-&gt; Fast but unsafe</p><ul><li><strong>acks = 1</strong></li></ul><p>-&gt; Leader broker confirms write<br>-&gt; Balanced approach</p><ul><li><strong>acks = all</strong></li></ul><p>-&gt; All replicas confirm write<br>-&gt; Safest option</p><h4><strong>How to Use ACKs Wisely</strong></h4><ul><li>Use acks=all for critical data like payments</li><li>Use acks=1 for normal business events</li><li>Avoid acks=0 unless data loss is acceptable</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7QYLyQsV1GNUHiD0hjZLOw.png" /></figure><p>A Kafka Producer is designed to be:</p><ul><li>Fast (async processing)</li><li>Scalable (batching + partitioning)</li><li>Flexible (different ACK levels)</li><li>Decoupled (no knowledge of consumers)</li></ul><p>It is the <strong>starting point of every event stream in Kafka</strong>.</p><h3>8. Kafka Consumer</h3><p>A <strong>Kafka Consumer</strong> is an application that reads records from Kafka topics.</p><p>If the producer is responsible for publishing data into Kafka, the consumer is responsible for reading and processing that data.</p><p>In a publish-subscribe model, consumers subscribe to topics and receive records that producers have published.</p><p>For example, after an OrderCreated event is published to Kafka:</p><ul><li>Payment Service may process the payment</li><li>Inventory Service may reserve stock</li><li>Notification Service may send a confirmation email</li></ul><p>All of these services act as consumers.</p><p>A consumer does not wait for Kafka to send data automatically. Instead, Kafka follows a <strong>pull-based model</strong>, where consumers actively request data from Kafka.</p><p>The process looks like this:</p><ol><li>Consumer subscribes to a topic</li><li>Consumer requests records from Kafka</li><li>Kafka returns available records</li><li>Consumer processes them</li><li>Consumer keeps track of its progress using offsets</li></ol><p>This gives consumers full control over how quickly they process data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D1ibZgvKuvk-WOCtHEbEJQ.png" /></figure><h4>Pull Model vs Push Model</h4><p>Many messaging systems use a push model where messages are sent directly to consumers.</p><p>Kafka uses a pull model.</p><p>This means consumers decide:</p><ul><li>When to fetch data</li><li>How much data to fetch</li><li>How quickly to process records</li></ul><p>This approach prevents slow consumers from becoming overwhelmed and makes scaling easier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2A5idlDgbc6wRE9l4qJ61w.png" /></figure><h4>Offsets and Consumer Progress</h4><p>Kafka stores records in partitions, and every record has an offset. Consumers use offsets to track which records they have already processed.</p><p>Example:</p><pre>Partition 0<br>Offset 0 → OrderCreated<br>Offset 1 → PaymentCompleted<br>Offset 2 → InventoryReserved<br>Offset 3 → OrderShipped</pre><p>If a consumer has processed records up to offset 2, it knows the next record to read starts from offset 3.</p><h4>Why Offsets Are Important</h4><p>Offsets allow consumers to:</p><ul><li>Resume after failures</li><li>Avoid reading the same records repeatedly</li><li>Replay historical records when needed</li></ul><p>Unlike traditional queues, Kafka does not delete records after consumption. The consumer simply tracks its current position.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GYIl_rL4B6K07K0M13VWJg.png" /></figure><h4>Consumer Groups</h4><p>A single consumer can read data from a topic, but real-world systems often need more processing power.</p><p>Kafka solves this using <strong>consumer groups</strong>.</p><p>A consumer group is a collection of consumers working together to process a topic. Kafka distributes partitions among consumers in the same group.</p><p>For example:</p><pre>Topic: Orders<br>Partitions:<br>Partition 0<br>Partition 1<br>Partition 2<br><br>Consumer Group:<br>Consumer A<br>Consumer B<br>Consumer C</pre><p>Each consumer receives one partition and processes records independently.</p><p>Consumer groups provide horizontal scaling, parallel processing and fault tolerance</p><p>As data volume grows, more consumers can be added to the group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*45v88VwkNMZwOxnglVQhKw.png" /></figure><p>When consumers join or leave a group, Kafka automatically redistributes partitions. This process is called <strong>rebalancing</strong>.</p><p>Examples:</p><ul><li>New consumer joins → Kafka redistributes partitions</li><li>Consumer crashes → Remaining consumers take over its partitions</li></ul><p>This ensures all partitions continue to be processed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M6i1VolXgCvKAPwjtAyVuA.png" /></figure><h4>Auto Commit vs Manual Commit</h4><p>Consumers can decide how offsets should be committed.</p><p>For Auto Commits, Kafka periodically saves offsets automatically.</p><p>Advantages:</p><ul><li>Easy to use</li></ul><p>Disadvantages:</p><ul><li>Risk of losing track of processing state if records fail during processing</li></ul><p>For Manual Commits, Application explicitly commits offsets after successful processing.</p><p>Advantages:</p><ul><li>Better reliability</li><li>Greater control</li></ul><p>Disadvantages:</p><ul><li>Slightly more implementation effort</li></ul><p>For critical systems such as payments and financial transactions, manual commits are often preferred.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o-RFZaX4IAQ2Iss5REjmBQ.png" /></figure><h3>Conclusion</h3><p>In this article, we moved beyond the basic concepts of event streaming and explored the core building blocks that make Kafka work.</p><p>We started with events, topics, partitions, and offsets — the fundamental structures used to store and organize data in Kafka. We then looked at brokers and clusters to understand how Kafka distributes data across multiple servers to achieve scalability and fault tolerance.</p><p>Finally, we explored producers and consumers, the two components responsible for publishing and processing data. Along the way, we discussed concepts such as publish-subscribe messaging, asynchronous publishing, acknowledgements, consumer groups, offset management, and rebalancing.</p><p>Together, these components form the foundation of every Kafka-based system. Understanding how they interact is essential before moving on to writing Kafka applications.</p><p>In the next article, we will explore Kafka’s APIs and learn how applications interact with Kafka programmatically. We will cover the Producer API, Consumer API, Admin API, Connect API, and Streams API, along with practical examples of when and how they are used in real-world systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93c1b6c704b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Kafka: An Event Streaming Platform]]></title>
            <link>https://medium.com/@yashwantsaste/apache-kafka-an-event-streaming-platform-58cf8732427e?source=rss-384f60710a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/58cf8732427e</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[event-streaming]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[pub-sub]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <dc:creator><![CDATA[Yashwant Sanjay Saste]]></dc:creator>
            <pubDate>Sun, 07 Jun 2026 19:28:29 GMT</pubDate>
            <atom:updated>2026-06-07T19:28:29.882Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g2qzz4g9RjV-jJrNYjwXZQ.png" /></figure><p>Suppose a customer placing an order on an e-commerce website.</p><p>Within a few seconds, multiple things need to happen. The payment service must process the transaction, the inventory service must reserve the product, the shipping service must prepare the delivery, and the notification service must send a confirmation message.</p><p>Now for example this platform receives thousands of orders every second.</p><p>How do all these systems communicate reliably? What happens if one service is temporarily unavailable? How can new services be added without modifying existing ones?</p><p>These are the kinds of problems Apache Kafka was designed to solve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rXX1LzxbH9QzfClSFduD_w.png" /></figure><p>Apache Kafka is a distributed event streaming platform that enables applications to exchange data in real time. Instead of services communicating directly with each other, they communicate through Kafka by producing and consuming events. This approach helps build systems that are scalable, fault-tolerant, and easier to maintain.</p><p>Originally developed at LinkedIn, Kafka was created to handle massive streams of user activity data and make that data available to multiple systems simultaneously. Today, it is widely used for building event-driven architectures, real-time analytics platforms, log processing systems, and large-scale data pipelines.</p><p>In this article, we will start from the fundamentals and gradually build an understanding of how Kafka works internally. Before diving into brokers, topics, partitions, and consumers, we first need to understand the concept that Kafka is built around: <strong>event streaming</strong>.</p><h3>1. What is Event Streaming?</h3><p>Before understanding Kafka, we need to understand the concept it was built to solve: <strong>event streaming</strong>.</p><p>Kafka is not just a messaging system or a queue. At its core, Kafka is an event streaming platform. Most of Kafka’s design decisions — topics, partitions, retention, consumer groups, and replayability — make much more sense once you understand what events and event streams are.</p><h3>1.1 What is an Event?</h3><p>An event is a record of something that happened in a system.</p><p>Whenever an action occurs, a system can capture it as an event.</p><p>For example:</p><ul><li>A customer places an order.</li><li>A payment is completed.</li><li>A user logs into an application.</li><li>A package is shipped.</li><li>A sensor reports a temperature reading.</li></ul><p>An event represents a fact. It describes something that already happened and contains information about that occurrence.</p><p>Consider an online shopping application. When a customer places an order, the system may generate an event like this:</p><pre>{<br>  &quot;eventId&quot;: &quot;ORD-1001&quot;,<br>  &quot;eventType&quot;: &quot;OrderCreated&quot;,<br>  &quot;customerId&quot;: &quot;C123&quot;,<br>  &quot;amount&quot;: 499.99,<br>  &quot;timestamp&quot;: &quot;2026-06-08T10:30:00Z&quot;<br>}</pre><p>This event tells us that an order was created, who created it, how much it cost, and when it happened. Although event structures vary between organizations, most events contain three important parts:</p><h4>Event Metadata</h4><p>Metadata provides information about the event itself.</p><p>Examples include:</p><ul><li>Event ID</li><li>Event Type</li><li>Timestamp</li><li>Source System</li><li>Correlation ID</li></ul><h4>Event Payload</h4><p>The payload contains the actual business data. For an order event, the payload might contain:</p><pre>{<br>  &quot;orderId&quot;: &quot;ORD-1001&quot;,<br>  &quot;customerId&quot;: &quot;C123&quot;,<br>  &quot;amount&quot;: 499.99<br>}</pre><h4>Event Timestamp</h4><p>The timestamp records when the event occurred. This is especially important in distributed systems because events may arrive at different times than when they were originally generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u3kYAbd_x-u-Y9-q1LxDQA.png" /><figcaption>Diagram: Structure of an Event</figcaption></figure><h3>1.2 What is a Stream of Events?</h3><p>A single event is useful, but systems rarely generate just one event. Applications continuously generate events as users interact with them.</p><p>Consider an online shopping platform:</p><pre>Order Created<br>Payment Completed<br>Inventory Reserved<br>Shipment Created<br>Package Delivered</pre><p>These events occur continuously throughout the day.</p><p>A sequence of related events flowing over time is called an <strong>event stream</strong>.</p><p>Unlike traditional databases that primarily focus on the current state of data, event streams capture everything that happened over time.</p><p>For example:</p><pre>10:00 AM -&gt; Order Created<br>10:01 AM -&gt; Payment Completed<br>10:02 AM -&gt; Inventory Reserved<br>10:10 AM -&gt; Shipment Created<br>10:30 AM -&gt; Package Delivered</pre><p>This sequence forms an event stream. The stream tells the story of what happened inside the system. For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lD3MyKz7cb20wVMBl_HqBg.png" /></figure><h3>1.3 Traditional Systems vs Event-Driven Systems</h3><p>To understand why event streaming matters, consider how systems were traditionally built.</p><p>Suppose an Order Service needs to notify multiple systems after an order is placed.</p><p>The Order Service directly calls:</p><ul><li>Payment Service</li><li>Inventory Service</li><li>Shipping Service</li><li>Notification Service</li></ul><p>This creates multiple dependencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ODKcxNYQYZZFPdGkulPfGA.png" /></figure><p>If one service becomes unavailable, failures can cascade through the system.</p><p>Now consider an event-driven architecture. The Order Service simply publishes an event. Interested systems subscribe to it.</p><p>The producer does not need to know who consumes the event.</p><p>The producer publishes an event once. Multiple consumers can independently react to that event. This reduces coupling and makes systems easier to scale and evolve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kufk38ki3wI2ga5wHRBpbw.png" /></figure><h4>Why Event Streaming Matters</h4><p>Event streaming enables systems to react to information as it is generated.</p><p>Instead of waiting for scheduled jobs or batch processes, applications can process data immediately after an event occurs.</p><p>This provides several advantages:</p><ul><li>Faster processing</li><li>Better scalability</li><li>Reduced coupling</li><li>Improved fault tolerance</li><li>Real-time insights</li></ul><p>These requirements are exactly what led to the creation of Apache Kafka.</p><p>Kafka provides the infrastructure required to store, distribute, and process these event streams reliably at scale.</p><h3>3. Kafka as an Event Streaming Platform</h3><p>In the previous section, we explored the concepts of events and event streams. We learned that modern applications continuously generate events and that these events often need to be shared across multiple systems.</p><p>The next question is how these event streams are managed.</p><p>A large application may generate thousands or even millions of events every second. These events must be delivered reliably, stored safely, and made available to multiple consumers without creating direct dependencies between systems.</p><p>This is the problem Kafka solves.</p><p>Although Kafka is often described as a messaging system, that description does not fully capture its capabilities. Traditional messaging systems focus primarily on moving messages from one application to another. Kafka goes beyond simple message delivery by providing a platform where events can be published, stored, distributed, and processed continuously.</p><p>For this reason, Kafka is commonly referred to as an <strong>event streaming platform</strong>.</p><p>An event streaming platform must be able to do three things effectively:</p><ol><li>Allow applications to publish and subscribe to streams of events.</li><li>Store events reliably for future consumption.</li><li>Process events as they are generated.</li></ol><p>Let’s examine each of these capabilities.</p><h3>3.1 Publish and Subscribe</h3><p>The most visible capability of Kafka is enabling communication between applications through events.</p><p>Applications that generate events are called <strong>producers</strong>. Applications that read those events are called <strong>consumers</strong>.</p><p>Without Kafka, the Order Service would need to communicate directly with each of these systems. Every new consumer would require additional integrations, increasing complexity and coupling.</p><p>Kafka simplifies this interaction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kufk38ki3wI2ga5wHRBpbw.png" /></figure><p>Instead of sending data directly to every consumer, the Order Service publishes the event to Kafka. Any system interested in that event can subscribe and consume it independently.</p><p>The producer does not need to know who is consuming the event, and consumers do not need to know who produced it. Kafka acts as the intermediary that connects them.</p><p>The publish-subscribe model significantly reduces dependencies between services. New consumers can be introduced without changing existing producers, making systems easier to extend and maintain.</p><p>This decoupling is one of the key reasons Kafka is widely adopted in microservice architectures.</p><h3>3.2 Store Events Reliably</h3><p>Delivering an event is only part of the problem.</p><p>What happens if a consumer is temporarily unavailable?</p><p>Suppose the Notification Service is undergoing maintenance when an order event is generated. If the event exists only in memory, the notification may never be sent.</p><p>Kafka addresses this challenge by storing events durably.</p><p>When an event is published, Kafka writes it to disk and maintains it for a configurable retention period. During this period, consumers can read the event whenever they are ready.</p><p>This storage model is fundamentally different from many traditional messaging systems.</p><p>Consider the following sequence of events:</p><pre>Offset 0 → Order Created<br>Offset 1 → Payment Completed<br>Offset 2 → Inventory Reserved<br>Offset 3 → Shipment Created</pre><p>Each event is assigned a unique position known as an <strong>offset</strong>. Offsets allow Kafka to track the location of every event within a stream.</p><p>Because events are retained, consumers can revisit historical data whenever necessary. This capability enables:</p><ul><li>Failure recovery</li><li>Event replay</li><li>Historical analysis</li><li>Debugging</li><li>Building new services from existing data</li></ul><p>Rather than acting as a temporary transport mechanism, Kafka becomes a durable system of record for event streams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6n3VcgBXEg6whxwIfbfhtg.png" /></figure><p>The ability to store and replay events is one of the features that distinguishes Kafka from many traditional messaging platforms.</p><h3>3.3 Process Events in Real Time</h3><p>Modern applications increasingly depend on real-time decision making.</p><p>A fraud detection system cannot wait until tomorrow to analyze a suspicious transaction. A monitoring platform cannot wait several hours before detecting a server failure. A recommendation engine must react quickly to changing user behavior.</p><p>Kafka enables applications to process events as soon as they are generated.</p><p>As events enter Kafka, consumers can immediately begin processing them. Multiple consumers can process the same event simultaneously while performing completely different tasks.</p><p>Consider a payment transaction.</p><p>When a PaymentCompleted event is published, several systems may react:</p><ul><li>A Fraud Detection Service validates the transaction.</li><li>An Analytics Platform updates dashboards.</li><li>A Rewards System calculates loyalty points.</li><li>A Notification Service sends a confirmation message.</li></ul><p>All of these systems can process the same event independently and in parallel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OqO6RUF1dEhojLEPEYtlNg.png" /></figure><p>This model allows organizations to build responsive systems that react to business events as they happen.</p><h3>Why Kafka is More Than a Message Queue</h3><p>At first glance, Kafka may appear similar to traditional message brokers such as RabbitMQ, ActiveMQ, or IBM MQ. All of these systems allow applications to exchange information.</p><p>However, Kafka was designed with a different philosophy.</p><p>Traditional message queues focus primarily on delivering messages from producers to consumers. Once a message has been processed, it is often removed from the system.</p><p>Kafka treats events differently.</p><p>Events are retained for a configurable period and remain available even after they have been consumed. Multiple consumers can read the same event independently, and new consumers can process historical events that were generated long before they existed.</p><p>This approach transforms Kafka from a simple communication mechanism into a central event platform.</p><p>For many organizations, Kafka becomes the backbone through which business events flow across the entire enterprise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DDaxZPUTVxOa1bIWDN0P0g.png" /></figure><p>This distinction becomes increasingly important as systems grow. Instead of viewing events as temporary messages, Kafka treats them as valuable records of business activity.</p><h3>Conclusion</h3><p>Apache Kafka is often introduced through concepts such as brokers, topics, partitions, and consumer groups. However, before exploring these components, it is important to understand the problem Kafka was built to solve.</p><p>In this article, we started with the fundamentals of event streaming. We looked at what events are, how event streams represent the continuous flow of business activities, and why event-driven architectures have become a common approach for building modern distributed systems. We then explored how Kafka acts as an event streaming platform by enabling applications to publish, store, and process events in real time.</p><p>At this point, we have established the foundation needed to understand Kafka. We know what an event is, how events flow through systems, and why organizations rely on event streams to build scalable and loosely coupled applications.</p><p>The next step is to look inside Kafka itself.</p><p>In the next article, we will explore Kafka’s architecture and core building blocks, including brokers, clusters, producers, consumers, topics, and partitions. Understanding these components will help us see how Kafka delivers the scalability, durability, and high throughput that make it a key technology in modern data platforms.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58cf8732427e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Event-Driven Architecture (EDA)]]></title>
            <link>https://medium.com/@yashwantsaste/introduction-to-event-driven-architecture-eda-da11dbe367e2?source=rss-384f60710a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/da11dbe367e2</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[high-level-design]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <dc:creator><![CDATA[Yashwant Sanjay Saste]]></dc:creator>
            <pubDate>Sun, 24 May 2026 18:20:26 GMT</pubDate>
            <atom:updated>2026-05-24T18:20:26.700Z</atom:updated>
            <content:encoded><![CDATA[<h3>Event-Driven Architecture Explained for Beginners</h3><p>Every app you use is built on some kind of structure. That structure decides how different parts of the system talk to each other, share data, and respond to what users do.</p><p>When apps are small, a simple setup works fine. But as systems grow — more users, more features, more services — things get complicated fast. You need a design that can handle scale without falling apart.</p><p>Event-Driven Architecture (EDA) is one approach that solves this well. You’ve already seen it in action — every time you get a payment notification, an order update, or a Slack message, there’s a good chance EDA is running behind the scenes.</p><h3>What Is an Event?</h3><p>Before understanding the architecture, you need to understand what an “event” means here.</p><p>An event is simply <strong>something that happened</strong>. That’s it.</p><ul><li>A user signed up</li><li>An order was placed</li><li>A payment was completed</li><li>A file was uploaded</li></ul><p>Each of these is an event. It’s a record of a moment — a fact that something occurred in the system.</p><p>Think of it like YouTube notifications. When a creator uploads a video, YouTube doesn’t call each subscriber individually. It just fires off a notification, and whoever is subscribed receives it automatically. The creator doesn’t need to know who’s watching or how many people got notified.</p><p>That’s the core idea behind event-driven systems.</p><h3>What Is Event-Driven Architecture?</h3><p>In a traditional system, services call each other directly. Service A needs something from Service B, so it sends a request and waits for a response.</p><p>In an event-driven system, services don’t call each other. Instead, they produce and consume events.</p><ul><li>A <strong>producer</strong> generates an event when something happens</li><li>A <strong>consumer</strong> listens for events and reacts to them</li><li>The two sides don’t need to know about each other</li></ul><p>Here’s a simple example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zsjXZR8htI_uznnwLfLIZg.png" /></figure><p>The Order Service doesn’t call the Payment Service or the Email Service directly. It just says “an order was created” and moves on. Each other service picks that up and does its own job.</p><h3>Core Components of EDA</h3><h3>Event Producer</h3><p>The producer is whatever generates the event. When a user places an order, the Order Service is the producer. It creates an event and sends it out.</p><h3>Event Consumer</h3><p>The consumer listens for events and acts on them. The Email Service, for example, listens for an “OrderCreated” event and sends a confirmation email when it hears one.</p><h3>Event Broker</h3><p>The broker sits in the middle. It receives events from producers and delivers them to consumers. It’s basically the messaging layer.</p><p>Common brokers you’ll hear about:</p><ul><li><strong>Apache Kafka</strong> — built for high-volume event streaming</li><li><strong>RabbitMQ</strong> — reliable message queuing</li><li><strong>Amazon SQS</strong> — managed queue service on AWS</li></ul><h3>The Event Itself</h3><p>The event is just a piece of data — a payload that describes what happened. It usually includes what the event is, when it happened, and any relevant details (like an order ID or user ID).</p><h3>Why Use Event-Driven Architecture?</h3><p><strong>Scalability</strong> Each service can scale on its own. If your Payment Service is under heavy load, you can scale just that — without touching anything else.</p><p><strong>Loose Coupling</strong> Services don’t depend directly on each other. The Order Service doesn’t care how many other services are listening. Adding a new service won’t break existing ones.</p><p><strong>Faster Processing</strong> Multiple consumers can process events at the same time. The Inventory Service, Payment Service, and Email Service all work in parallel — not one after the other.</p><p><strong>Real-Time Updates</strong> EDA makes it natural to push updates as they happen. That’s why tracking your delivery or seeing a payment confirmation shows up almost instantly.</p><p><strong>Flexibility</strong> Need to add a new service later? Just subscribe it to the relevant events. No need to change the producer or anything else.</p><h3>A Real-World Comparison: E-Commerce Without and With EDA</h3><h3>Without EDA</h3><p>When a user places an order, the Order Service directly calls:</p><ul><li>The Payment Service</li><li>The Inventory Service</li><li>The Email Service</li></ul><p>One by one. In sequence.</p><p>The problems here:</p><ul><li>If the Email Service is down, the whole flow might fail</li><li>Every service is tightly connected to every other</li><li>Scaling one service affects how others behave</li><li>Adding a new step (like a loyalty points service) means changing the Order Service code</li></ul><h3>With EDA</h3><p>The Order Service publishes one event: “OrderCreated.”</p><p>Every other service reacts independently. They don’t know about each other. If the Email Service is slow or temporarily down, the other services keep running. The email will go out when it catches up.</p><p>This makes the whole system more resilient and easier to maintain.</p><h3>Common Technologies Used in EDA</h3><p>Technology What It Does Apache Kafka High-throughput event streaming RabbitMQ Message queuing Redis Pub/Sub messaging and caching AWS EventBridge Cloud-native event bus Apache Pulsar Distributed messaging at scale</p><p>If you’re just getting started, RabbitMQ is easier to learn. Kafka is worth exploring once you understand the basics.</p><h3>The Challenges — Honestly</h3><p>EDA is not a perfect solution. There are real trade-offs.</p><p><strong>Debugging is harder.</strong> In a direct call system, you can trace a request step by step. In EDA, events flow through multiple services asynchronously, which makes tracking down a bug more work.</p><p><strong>Event ordering isn’t guaranteed.</strong> In some setups, events can arrive out of order. If your system depends on strict sequencing, you need to handle that explicitly.</p><p><strong>Duplicate events can happen.</strong> Network issues or retries can cause the same event to be delivered more than once. Consumers need to handle this gracefully (this is called idempotency).</p><p><strong>Eventual consistency.</strong> Different services process events at different speeds. There’s usually a small window where the system is in an inconsistent state before everything catches up.</p><p><strong>Monitoring is more complex.</strong> Watching a distributed event-driven system requires proper tooling. Without it, you’re flying blind.</p><p>None of these are reasons to avoid EDA — but they’re things to be aware of before you build with it.</p><h3>When Should You Use EDA?</h3><p><strong>Good fit:</strong></p><ul><li>Real-time notifications and updates</li><li>Microservices that need to work independently</li><li>Analytics pipelines processing large volumes of data</li><li>IoT systems with many data sources</li><li>Any system where services need to react to changes without being tightly coupled</li></ul><p><strong>Not a good fit:</strong></p><ul><li>Small, simple applications</li><li>Basic CRUD apps with straightforward workflows</li><li>Systems where every step must happen in a strict, synchronous order</li></ul><p>EDA adds complexity. If your system doesn’t need that flexibility or scale, a simpler architecture will serve you better.</p><h3>Conclusion</h3><p>Event-Driven Architecture changes how services communicate — from direct calls to publishing and reacting to events. The result is a system that’s easier to scale, more resilient to failures, and simpler to extend over time.</p><p>It’s not the right choice for every project, but for systems that need to handle real-time data, scale independently, or support many loosely connected services, it’s a solid approach.</p><p>If you want to explore this further, start with RabbitMQ for a hands-on feel, or look into Apache Kafka if you’re dealing with high-volume data. Building even a small producer-consumer setup will make the concepts click much faster than reading about them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da11dbe367e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rust Macros: A Way for Writing Code That Writes Code]]></title>
            <link>https://medium.com/@yashwantsaste/rust-macros-a-way-for-writing-code-that-writes-code-f202c0e2b243?source=rss-384f60710a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/f202c0e2b243</guid>
            <category><![CDATA[rust-macros]]></category>
            <category><![CDATA[multithreading]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[rust-programming-language]]></category>
            <category><![CDATA[system-programming]]></category>
            <dc:creator><![CDATA[Yashwant Sanjay Saste]]></dc:creator>
            <pubDate>Sun, 12 Apr 2026 11:22:18 GMT</pubDate>
            <atom:updated>2026-04-12T11:22:18.320Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>From your first println! to building your own DSL — everything you need to understand, use, and master Rust macros.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n3qOgRXTRGtWG5ijqb-crA.png" /></figure><h3>The Exclamation Mark That Started It All</h3><p>The first time most developers encounter Rust, they type something like this:</p><pre>fn main() {<br>    println!(&quot;Hello, Rust!&quot;);<br>}</pre><p>It works. It prints. You feel good. But then, maybe a few hours later, you pause and think:Wait. Why does println have an exclamation mark? Is that a typo?</p><p>That little ! is one of the most important symbols in Rust. It&#39;s not an accident, it&#39;s not excitement, and it&#39;s definitely not a typo. It&#39;s Rust&#39;s way of telling you:</p><p><strong>“This is not a regular function. This is a macro.”</strong></p><p>And once you understand what that means — once you really <em>get</em> why that ! exists — a whole new dimension of Rust programming opens up.</p><h3>1. What Are Macros in Rust?</h3><blockquote>A <strong>macro</strong> in Rust is a way to write code that generates other code.</blockquote><p>That might sound abstract, so let’s use an analogy.</p><p>Imagine you’re a baker who makes sandwiches. A regular function is like a trained baker — you give them bread and fillings, and they make you a sandwich. The baker follows a fixed process. You can’t change <em>how</em> they work, just <em>what</em> you give them.</p><p>A macro is like a <strong>recipe template generator</strong>. Instead of making a sandwich, it writes a new recipe for you. You describe the pattern, and the macro <em>creates the actual instructions</em> that will then be followed. Before any baking (running) happens, the recipe (code) is already written.</p><p>In Rust terms:</p><ul><li><strong>Functions</strong> are compiled, called at runtime, and execute logic using values.</li><li><strong>Macros</strong> are expanded at <strong>compile time</strong>, transforming your source code into different (usually more) source code before the compiler even starts compiling it.</li></ul><pre>Your source code<br>      ↓<br>[Macro Expansion] ← happens here, before compilation<br>      ↓<br>Expanded source code<br>      ↓<br>[Compilation]<br>      ↓<br>Binary / Executable</pre><p>This is the key insight. When you write println!(&quot;Hello&quot;), the compiler doesn&#39;t just call a function named println. It first <strong>expands</strong> that macro into a block of real Rust code — a whole set of formatting instructions — and then compiles that expanded code.</p><p>You never see the expanded code unless you go looking for it. It happens silently, before your eyes.</p><p>That’s why macros feel a little magical at first. The code you write is not quite the code that gets compiled. Macros are the bridge between the two.</p><h3>2. Why Rust Needs Macros</h3><p>If Rust is so powerful, why aren’t regular functions enough?</p><p>Great question. Let’s look at some concrete limitations.</p><h3>Problem 1: Functions Can’t Accept a Variable Number of Arguments</h3><p>In many languages, you can write something like:</p><pre>print(&quot;a&quot;, &quot;b&quot;, &quot;c&quot;, &quot;d&quot;)  # Python handles this fine</pre><p>In Rust, a function must have a fixed, typed signature. You cannot define a regular function that accepts one, two, three, or ten arguments of potentially different types. Rust’s strict type system simply doesn’t allow it.</p><p>But this works perfectly fine:</p><pre>println!(&quot;{} {} {} {}&quot;, &quot;a&quot;, &quot;b&quot;, &quot;c&quot;, &quot;d&quot;);</pre><p>Why? Because println! is a macro. It doesn&#39;t have a fixed signature — it&#39;s a pattern matcher that generates the right code depending on whatever you pass it.</p><h3>Problem 2: Functions Can’t Inspect Their Own Syntax</h3><p>Macros receive <strong>tokens</strong> — the raw text of your code — before type checking. This lets them manipulate syntax itself.</p><p>Consider vec!:</p><pre>let v = vec![1, 2, 3, 4, 5];</pre><p>This creates a Vec&lt;i32&gt; initialized with five elements. To replicate this as a function, you&#39;d need something clunky:</p><pre>let v = Vec::from([1, 2, 3, 4, 5]);  // A bit verbose and less flexible</pre><p>The vec! macro is cleaner, more flexible, and can handle different types automatically. You couldn&#39;t write it as a regular function without losing clarity.</p><h3>Problem 3: Functions Can’t Reduce Boilerplate at a Structural Level</h3><p>Imagine you’re implementing a trait for twenty different types. Every implementation looks nearly identical — just the type name changes. With a macro, you can write the pattern once and stamp it out for all twenty types automatically.</p><h3>Problem 4: Functions Can’t Create Domain-Specific Languages</h3><p>Some of the most powerful uses of macros are in creating mini-languages embedded inside Rust:</p><pre>route!(&quot;/home&quot; =&gt; home_handler);<br>route!(&quot;/login&quot; =&gt; login_handler, method = POST);</pre><p>This kind of expressive, readable syntax isn’t possible with regular functions. Macros let you define your own syntax rules.</p><h3>3. Macro vs Function: A Clear Comparison</h3><p>Before diving into how to write macros, let’s make sure the macro vs. function distinction is crystal clear.</p><h3>Side-by-Side Example</h3><p>Here’s a simple add function:</p><pre>fn add(a: i32, b: i32) -&gt; i32 {<br>    a + b<br>}<br>let result = add(3, 4); // works fine</pre><p>And here’s a macro that does something similar:</p><pre>macro_rules! add {<br>    ($a:expr, $b:expr) =&gt; {<br>        $a + $b<br>    };<br>}<br>let result = add!(3, 4); // also works</pre><p>They look similar, but there are deep differences:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*EAdCrc_n-bTPa7bl1mH6Og.png" /></figure><h3>When to Use Which</h3><p>Use a <strong>function</strong> when:</p><ul><li>Your logic is self-contained and takes fixed inputs</li><li>You want clear, debuggable behavior</li><li>Type safety and IDE support matter most</li></ul><p>Use a <strong>macro</strong> when:</p><ul><li>You need variable argument counts</li><li>You’re generating repetitive code</li><li>You want custom syntax or DSL-like expressiveness</li><li>Compile-time code generation is needed</li></ul><h3>4. Declarative Macros: macro_rules!</h3><p>There are two broad types of macros in Rust:</p><ol><li><strong>Declarative macros</strong> — defined with macro_rules!</li><li><strong>Procedural macros</strong> — more advanced, defined as separate crates</li></ol><p>We’ll spend most of our time on declarative macros because they’re where every Rust developer starts. They’re powerful, they’re approachable, and they cover 80% of macro use cases.</p><h3>Your First Macro</h3><pre>macro_rules! say_hello {<br>    () =&gt; {<br>        println!(&quot;Hello from a macro!&quot;);<br>    };<br>}<br>fn main() {<br>    say_hello!(); // prints: Hello from a macro!<br>}</pre><p>Let’s dissect every symbol:</p><pre>macro_rules! say_hello {<br>│            │<br>│            └── The name of the macro (used as say_hello!)<br>└── The keyword that defines a declarative macro<br>() =&gt; {<br>    │     │<br>    │     └── The expansion block: what code gets generated<br>    └── The pattern to match against (empty here - no arguments)<br>        println!(&quot;Hello from a macro!&quot;);<br>    };<br>│<br>└── Semicolon ends this match arm (like a match arm in Rust&#39;s match expression)<br>}</pre><p>The structure of macro_rules! is essentially a series of <strong>pattern =&gt; expansion</strong> arms, very similar to Rust&#39;s match expression.</p><p>When you write say_hello!(), Rust looks at the argument list — in this case, nothing — and finds the arm whose pattern matches. It then <strong>replaces</strong> your macro call with the content of the expansion block.</p><p>After expansion, the compiler sees:</p><pre>fn main() {<br>    println!(&quot;Hello from a macro!&quot;); // ← this is what the compiler actually compiles<br>}</pre><p>You wrote say_hello!(). The compiler compiled println!(...). The macro was the transformation in between.</p><h3>Multiple Arms</h3><p>Just like match, macro_rules! can have multiple arms:</p><pre>macro_rules! greet {<br>    () =&gt; {<br>        println!(&quot;Hello, stranger!&quot;);<br>    };<br>    ($name:ident) =&gt; {<br>        println!(&quot;Hello, {}!&quot;, stringify!($name));<br>    };<br>}</pre><pre>fn main() {<br>    greet!();           // Hello, stranger!<br>    greet!(Alice);      // Hello, Alice!<br>}</pre><p>Rust checks each arm from top to bottom and uses the first one that matches.</p><h3>5. Fragment Specifiers: The Grammar of Macros</h3><p>This is one of the most important sections in this article. Read it carefully.</p><p>When you write a macro that accepts input, you need to tell Rust <em>what kind of thing</em> you’re accepting. These are called <strong>fragment specifiers</strong>, and they’re written after the : in a pattern like $x:expr.</p><p>Think of fragment specifiers as types for macro patterns. Just as i32 tells Rust &quot;this function parameter is an integer,&quot; expr tells Rust &quot;this macro parameter is an expression.&quot;</p><p>Let’s go through all the major ones.</p><h3>expr — An Expression</h3><pre>macro_rules! double {<br>    ($x:expr) =&gt; {<br>        $x * 2<br>    };<br>}</pre><pre>let result = double!(5 + 3); // expands to: (5 + 3) * 2 = 16</pre><p>Matches any Rust expression: a literal, a function call, a math operation, a closure, a block — basically anything that produces a value.</p><p><strong>Common confusion:</strong> expr captures the <em>whole</em> expression as a unit. So double!(5 + 3) expands to (5 + 3) * 2, not 5 + 3 * 2. The expression is preserved intact.</p><h3>ident — An Identifier</h3><pre>macro_rules! create_variable {<br>    ($name:ident, $value:expr) =&gt; {<br>        let $name = $value;<br>    };<br>}</pre><pre>create_variable!(my_var, 42);<br>// expands to: let my_var = 42;<br>println!(&quot;{}&quot;, my_var); // prints: 42</pre><p>Matches any valid Rust identifier: variable names, function names, type names, etc. It does <em>not</em> match keywords like fn or let.</p><p><strong>Common confusion:</strong> Beginners sometimes try to use $name:expr when they want a variable name. Use ident for names, expr for values.</p><h3>ty — A Type</h3><pre>macro_rules! create_vec_of_type {<br>    ($t:ty) =&gt; {<br>        Vec::&lt;$t&gt;::new()<br>    };<br>}</pre><pre>let v: Vec&lt;i32&gt; = create_vec_of_type!(i32);</pre><p>Matches a Rust type — i32, String, Option&lt;bool&gt;, Vec&lt;u8&gt;, etc.</p><p><strong>Use case:</strong> Generating code for specific types, like implementing traits.</p><h3>stmt — A Statement</h3><pre>macro_rules! log_and_run {<br>    ($s:stmt) =&gt; {<br>        println!(&quot;Running statement...&quot;);<br>        $s<br>    };<br>}</pre><pre>log_and_run!(let x = 5);</pre><p>Matches a complete statement: a let binding, an expression statement, etc.</p><p><strong>Use case:</strong> Wrapping statements with before/after logic (like timing or logging).</p><h3>block — A Block of Code</h3><pre>macro_rules! time_it {<br>    ($b:block) =&gt; {<br>        let start = std::time::Instant::now();<br>        $b<br>        println!(&quot;Elapsed: {:?}&quot;, start.elapsed());<br>    };<br>}</pre><pre>time_it!({<br>    let mut sum = 0;<br>    for i in 0..1000 { sum += i; }<br>    println!(&quot;Sum: {}&quot;, sum);<br>});</pre><p>Matches a { ... } block. Very useful for wrapping arbitrary code.</p><h3>tt — A Token Tree</h3><pre>macro_rules! echo {<br>    ($($t:tt)*) =&gt; {<br>        println!(&quot;{}&quot;, stringify!($($t)*))<br>    };<br>}</pre><pre>echo!(hello world this is anything);</pre><p>The most flexible specifier. tt matches a single token or a grouped token tree (anything inside (), [], or {}). Using $($t:tt)* captures <em>everything</em>.</p><p><strong>Use case:</strong> When you don’t know what the user will pass, or you want to forward tokens to another macro.</p><h3>pat — A Pattern</h3><pre>macro_rules! match_value {<br>    ($val:expr, $p:pat =&gt; $body:expr) =&gt; {<br>        match $val {<br>            $p =&gt; $body,<br>            _ =&gt; println!(&quot;no match&quot;),<br>        }<br>    };<br>}<br>match_value!(Some(42), Some(x) =&gt; println!(&quot;Got {}&quot;, x));</pre><p>Matches a Rust pattern (like the left side of a match arm).</p><h3>item — A Rust Item</h3><pre>macro_rules! export_item {<br>    ($item:item) =&gt; {<br>        pub $item<br>    };<br>}<br>export_item!(fn greet() { println!(&quot;hi&quot;); });<br>// expands to: pub fn greet() { println!(&quot;hi&quot;); }</pre><p>Matches a top-level Rust item: a function definition, struct, enum, trait, impl block, etc.</p><h3>path — A Path</h3><pre>macro_rules! use_type {<br>    ($p:path) =&gt; {<br>        let _: $p;<br>    };<br>}<br>use_type!(std::collections::HashMap::&lt;String, i32&gt;);</pre><p>Matches a Rust path like std::io::Error, MyModule::MyType, etc.</p><h3>meta — A Meta Item (Used in Attributes)</h3><pre>macro_rules! apply_attr {<br>    (#[$m:meta] $item:item) =&gt; {<br>        #[$m]<br>        $item<br>    };<br>}</pre><p>Matches the content inside an attribute like #[derive(Debug)] — the derive(Debug) part is a meta.</p><h3>lifetime — A Lifetime</h3><pre>macro_rules! with_lifetime {<br>    ($lt:lifetime, $t:ty) =&gt; {<br>        &amp;$lt $t<br>    };<br>}</pre><p>Matches a Rust lifetime like &#39;a, &#39;static, &#39;_.</p><h3>vis — A Visibility Modifier</h3><pre>macro_rules! make_field {<br>    ($v:vis $name:ident: $t:ty) =&gt; {<br>        $v $name: $t<br>    };<br>}</pre><p>Matches visibility keywords: pub, pub(crate), pub(super), or nothing (private).</p><h3>literal — A Literal Value</h3><pre>macro_rules! announce {<br>    ($lit:literal) =&gt; {<br>        println!(&quot;The value is: {}&quot;, $lit);<br>    };<br>}<br>announce!(42);<br>announce!(&quot;hello&quot;);<br>announce!(3.14);</pre><p>Matches literal values only: numbers, strings, booleans, characters. Unlike expr, it won&#39;t match expressions like 1 + 2.</p><h3>6. Macros with Parameters</h3><p>Now that you understand fragment specifiers, let’s look at how to capture and use parameters in macros.</p><h3>Basic Parameter Capture</h3><pre>macro_rules! square {<br>    ($x:expr) =&gt; {<br>        $x * $x<br>    };<br>}<br>let result = square!(5);     // 25<br>let result2 = square!(2 + 3); // (2 + 3) * (2 + 3) = 25</pre><p>The $x is a <strong>metavariable</strong> (sometimes called a macro variable). The $ prefix distinguishes it from regular Rust variables. You can reuse $x as many times as you want in the expansion.</p><h3>Multiple Parameters</h3><pre>macro_rules! max {<br>    ($a:expr, $b:expr) =&gt; {<br>        if $a &gt; $b { $a } else { $b }<br>    };<br>}<br>let m = max!(10, 20); // 20<br>let m2 = max!(3 * 4, 2 + 9); // 12 vs 11, so 12</pre><h3>Parameters with Different Types</h3><pre>macro_rules! create_function {<br>    ($func_name:ident) =&gt; {<br>        fn $func_name() {<br>            println!(&quot;Function {:?} called&quot;, stringify!($func_name));<br>        }<br>    };<br>}<br>create_function!(foo);<br>create_function!(bar);<br>fn main() {<br>    foo(); // Function &quot;foo&quot; called<br>    bar(); // Function &quot;bar&quot; called<br>}</pre><p>Here, $func_name:ident captures an identifier, which is then used as a function name in the expansion. This is genuinely code generating code — we defined two functions by calling the macro twice.</p><h3>7. Multiple Pattern Arms in Detail</h3><p>Macros support multiple arms, which lets them behave differently based on what they receive — similar to function overloading in other languages.</p><pre>macro_rules! describe {<br>    () =&gt; {<br>        println!(&quot;Nothing was passed.&quot;);<br>    };<br>    ($x:expr) =&gt; {<br>        println!(&quot;Got one thing: {:?}&quot;, $x);<br>    };<br>    ($x:expr, $y:expr) =&gt; {<br>        println!(&quot;Got two things: {:?} and {:?}&quot;, $x, $y);<br>    };<br>    ($x:expr, $y:expr, $z:expr) =&gt; {<br>        println!(&quot;Got three things: {:?}, {:?}, and {:?}&quot;, $x, $y, $z);<br>    };<br>}</pre><pre>describe!();           // Nothing was passed.<br>describe!(42);         // Got one thing: 42<br>describe!(1, 2);       // Got two things: 1 and 2<br>describe!(1, 2, 3);    // Got three things: 1, 2, and 3</pre><p>Rust checks arms in order, from top to bottom, and uses the <em>first</em> match. This means you should generally put more specific patterns before more general ones.</p><h3>Comparing to match</h3><p>This multi-arm structure mirrors Rust’s match expression:</p><pre>// Rust match<br>match value {<br>    0 =&gt; println!(&quot;zero&quot;),<br>    1 =&gt; println!(&quot;one&quot;),<br>    _ =&gt; println!(&quot;other&quot;),<br>}</pre><pre>// Macro with multiple arms<br>macro_rules! my_macro {<br>    (zero) =&gt; { println!(&quot;zero&quot;) };<br>    (one)  =&gt; { println!(&quot;one&quot;) };<br>    ($x:expr) =&gt; { println!(&quot;other: {}&quot;, $x) };<br>}</pre><p>Both are pattern-based. The key difference is that match runs at runtime and matches against values; macro_rules! arms are checked at compile time and match against token patterns.</p><h3>8. Repetition and Variadic Patterns</h3><p>This is where macros get truly powerful — and where beginners sometimes get lost. Take your time with this section.</p><p>The magic of println! accepting any number of arguments, or vec! accepting any number of elements, comes from <strong>repetition patterns</strong>.</p><h3>The Three Repetition Operators</h3><p>Operator Meaning * Zero or more + One or more ? Zero or one (optional)</p><p>These look just like regex quantifiers, and they work similarly.</p><h3>Zero or More: *</h3><pre>macro_rules! sum {<br>    ($($x:expr),*) =&gt; {<br>        {<br>            let mut total = 0;<br>            $(total += $x;)*<br>            total<br>        }<br>    };<br>}<br>let s0 = sum!();           // 0<br>let s1 = sum!(5);          // 5<br>let s2 = sum!(1, 2, 3);    // 6<br>let s3 = sum!(10, 20, 30, 40); // 100</pre><p>Let’s break down ($($x:expr),*):</p><pre>(          ← outer delimiter for this arm&#39;s pattern<br>  $(       ← start of repetition group<br>    $x:expr  ← capture one expression as $x<br>  ),       ← the separator between repetitions (a comma)<br>  *        ← repeat zero or more times<br>)</pre><p>And in the expansion, $(total += $x;)* means &quot;repeat the statement total += $x; once for each captured $x.&quot;</p><h3>One or More: +</h3><pre>macro_rules! first {<br>    ($head:expr $(, $tail:expr)+) =&gt; {<br>        $head<br>    };<br>}<br>// Must have at least 2 elements<br>let f = first!(1, 2, 3); // 1<br>// first!(); ← compile error! + requires at least one occurrence</pre><p>Use + when you need at least one element. Using * would allow empty input.</p><h3>Optional: ?</h3><pre>macro_rules! greet {<br>    ($name:expr $(, $greeting:expr)?) =&gt; {<br>        match (Some($name), ${count($greeting)}) {<br>            _ =&gt; {<br>                println!(&quot;Hello, {}!&quot;, $name);<br>            }<br>        }<br>    };<br>}</pre><p>The ? operator is a bit more subtle. It&#39;s often used for optional trailing syntax or optional configuration parameters.</p><p>A cleaner example:</p><pre>macro_rules! config {<br>    ($key:ident $(= $val:expr)?) =&gt; {<br>        // $val is either present or not<br>    };<br>}<br>config!(debug);            // no value<br>config!(timeout = 30);     // with value</pre><h3>The Separator</h3><p>In $($x:expr),*, the comma between ) and * is the <strong>separator</strong>. It appears <em>between</em> repetitions but not after the last one.</p><p>So sum!(1, 2, 3) matches correctly: 1, then , 2, then , 3. The comma is the separator, not part of the expression itself.</p><p>What about trailing commas? sum!(1, 2, 3,) — that last comma would fail! The separator appears <em>between</em> elements, not after the last one.</p><p>To handle trailing commas, you can use a common trick:</p><pre>macro_rules! sum {<br>    ($($x:expr),* $(,)?) =&gt; {<br>        {<br>            let mut total = 0;<br>            $(total += $x;)*<br>            total<br>        }<br>    };<br>}<br>let s = sum!(1, 2, 3,); // Now this works!</pre><p>The $(,)? at the end matches an optional trailing comma.</p><h3>9. Separator Handling and Edge Cases</h3><p>Edge cases are where most beginner macro writers run into trouble. Let’s go through them systematically.</p><h3>The Trailing Comma Problem</h3><pre>macro_rules! sum_basic {<br>    ($($x:expr),*) =&gt; { /* ... */ };<br>}<br>sum_basic!(1, 2, 3);   // ✅ works<br>sum_basic!(1, 2, 3,);  // ❌ compile error: unexpected token &#39;,&#39;</pre><p>The trailing comma is one of the most common issues. Fix it with $(,)?:</p><pre>macro_rules! sum_fixed {<br>    ($($x:expr),* $(,)?) =&gt; { /* ... */ };<br>}<br>sum_fixed!(1, 2, 3);   // ✅ works<br>sum_fixed!(1, 2, 3,);  // ✅ also works now</pre><p>This pattern is so common that it’s essentially considered best practice for any variadic macro.</p><h3>The Empty Invocation</h3><pre>sum!()     // uses * → expands to total = 0; returns 0 ✅<br>sum!()     // uses + → compile error! + needs at least one ❌</pre><p>Choose * if empty is valid, + if you need at least one.</p><h3>Nested Repetition</h3><p>You can have repetition inside repetition, which is powerful but tricky:</p><pre>macro_rules! matrix {<br>    ($([$($x:expr),*]),*) =&gt; {<br>        vec![$(vec![$($x),*]),*]<br>    };<br>}<br>let m = matrix!([1, 2, 3], [4, 5, 6], [7, 8, 9]);<br>// creates: vec![vec![1,2,3], vec![4,5,6], vec![7,8,9]]</pre><p>The outer $( ... ),* iterates over rows. The inner $( $x:expr ),* iterates over elements within each row.</p><p><strong>Important rule:</strong> In nested repetition, you can only use a variable in the repetition depth where it was captured. You can’t mix inner and outer variables carelessly.</p><h3>Ambiguity in Patterns</h3><p>Rust’s macro matcher can sometimes become ambiguous. For example, if you have an arm that matches $x:expr and another that matches $x:expr, $y:expr, the matcher needs to know when to stop. Rust handles this with a concept called <strong>FOLLOW sets</strong> — certain tokens can follow certain fragment types.</p><p>For instance, after $x:expr, only certain tokens are allowed (like ,, ;, =&gt;, |, etc.). If you try:</p><pre>macro_rules! bad {<br>    ($x:expr $y:expr) =&gt; { /* ... */ }; // ❌ ambiguous!<br>}</pre><p>The compiler will complain because there’s no separator between $x and $y, making it impossible to know where one ends and the other begins.</p><h3>10. Nested Macros: Macros Calling Macros</h3><p>Macros can call other macros — both standard ones like println! and custom ones you define.</p><pre>macro_rules! log {<br>    ($msg:expr) =&gt; {<br>        println!(&quot;[LOG] {}&quot;, $msg);<br>    };<br>}</pre><pre>macro_rules! debug_value {<br>    ($name:ident, $val:expr) =&gt; {<br>        log!(format!(&quot;{} = {:?}&quot;, stringify!($name), $val));<br>    };<br>}</pre><pre>fn main() {<br>    let x = 42;<br>    debug_value!(x, x); // [LOG] x = 42<br>}</pre><p>Here, debug_value! expands and the expansion itself contains a call to log!, which then gets expanded further. The expansion happens in rounds: Rust keeps expanding until no more macros remain.</p><h3>Expansion Order</h3><p>Macros are expanded from the outside in. If a! expands to b!(), then b! is expanded next. This continues until all macros are resolved.</p><h3>The stringify! Macro</h3><p>Notice stringify!($name) in the example. This is a built-in macro that converts its argument to a string literal <em>at compile time</em>. So stringify!(x) becomes &quot;x&quot;. It&#39;s incredibly useful inside other macros.</p><h3>Readability Warning</h3><p>Deeply nested macros quickly become hard to read and debug. If you find yourself with three or four levels of nesting, consider whether some of the inner logic should be a function instead.</p><h3>11. Scope and Visibility</h3><p>Unlike functions and types, macros have their own scoping rules that confuse many beginners.</p><h3>Local Scope</h3><p>By default, a macro is available from the point of definition to the end of the current scope:</p><pre>fn main() {<br>    macro_rules! local_macro {<br>        () =&gt; { println!(&quot;local!&quot;) };<br>    }<br>local_macro!(); // ✅ works here<br>    {<br>        local_macro!(); // ✅ works in inner scope too<br>    }<br>}<br>// local_macro!(); // ❌ can&#39;t use here - out of scope</pre><h3>Module Scope</h3><p>Macros defined at the module level are not automatically available in other modules, even child modules:</p><pre>mod my_module {<br>    macro_rules! inner_macro {<br>        () =&gt; { println!(&quot;inner&quot;) };<br>    }<br>pub fn use_it() {<br>        inner_macro!(); // ✅ works here<br>    }<br>}</pre><pre>fn main() {<br>    // inner_macro!(); // ❌ not accessible here<br>    my_module::use_it(); // ✅ works<br>}</pre><h3>#[macro_export]</h3><p>To make a macro available throughout your crate (and to users of your crate), mark it with #[macro_export]:</p><pre>#[macro_export]<br>macro_rules! my_public_macro {<br>    () =&gt; { println!(&quot;exported!&quot;) };<br>}</pre><p>When #[macro_export] is used, the macro is placed at the <strong>crate root</strong>, not in the module where it was defined. This is an important detail — even if you define it inside my_module, it&#39;s exported as if it were at the crate root.</p><h3>Using Macros from Other Crates</h3><p>If you depend on a crate that exports macros, you can use them with:</p><pre>use some_crate::some_macro;<br>// or to get all macros:<br>use some_crate::*;</pre><p>Since Rust 2018 edition, you no longer need #[macro_use] on external crates for most purposes (though it still exists for legacy compatibility).</p><h3>12. Macro Hygiene: The Invisible Superpower</h3><p>Hygiene is one of Rust’s most underrated macro features. It’s what prevents your macros from accidentally breaking the code around them.</p><h3>The Problem It Solves</h3><p>Imagine a macro that creates a temporary variable named result. Now imagine someone uses that macro inside a function that already has a variable called result. In many languages, the macro would accidentally shadow or corrupt the outer result.</p><p>Let’s see what could go wrong without hygiene:</p><pre>// Hypothetically, without hygiene:<br>let result = 100;<br>some_macro!(); // internally creates `let result = 42;`<br>println!(&quot;{}&quot;, result); // Would this print 100 or 42?</pre><p>This is the classic <strong>variable capture problem</strong>. It causes subtle, hard-to-debug bugs.</p><h3>How Rust Handles It</h3><p>Rust’s macro system is <strong>hygienic by default</strong>. Variables introduced inside a macro expansion exist in their own distinct scope — they cannot accidentally collide with variables in the calling code.</p><pre>macro_rules! add_one {<br>    ($x:expr) =&gt; {{<br>        let result = $x + 1; // This `result` is private to the macro<br>        result<br>    }};<br>}</pre><pre>let result = 100;             // This `result` belongs to the outer code<br>let y = add_one!(5);          // The macro&#39;s `result` is different<br>println!(&quot;{}&quot;, result);        // Still prints 100 ← hygiene!</pre><p>The two result variables live in different &quot;universes&quot; as far as the compiler is concerned. They have the same text name, but they&#39;re fundamentally separate bindings.</p><h3>When You Want to Escape Hygiene</h3><p>Sometimes you <em>want</em> a macro to define a variable that the caller can use. This requires using $name:ident patterns where the caller provides the name:</p><pre>macro_rules! assign_double {<br>    ($name:ident, $val:expr) =&gt; {<br>        let $name = $val * 2;<br>    };<br>}<br>assign_double!(my_result, 21);<br>println!(&quot;{}&quot;, my_result); // 42</pre><p>By using $name:ident and having the caller pass my_result, you&#39;re letting the <em>calling code</em> name the variable. This is intentional and hygienic — the caller chose the name.</p><h3>13. Built-in Standard Macros</h3><p>Before writing your own, it’s worth understanding the macros you use every day. They’re all built with macro_rules! (or procedural macros) and follow the same rules.</p><h3>println! and print!</h3><pre>println!(&quot;Hello&quot;);<br>println!(&quot;Hello, {}!&quot;, &quot;world&quot;);<br>println!(&quot;{:?}&quot;, vec![1, 2, 3]);<br>println!(&quot;{name} is {age}&quot;, name = &quot;Alice&quot;, age = 30);</pre><p>println! is a variadic macro. It takes a format string plus any number of arguments. Behind the scenes, it uses Rust&#39;s formatting infrastructure (std::fmt) to generate a string and write it to stdout.</p><p>The first argument must be a string literal (not a variable) because the macro inspects it at compile time to generate the right formatting code.</p><h3>format!</h3><pre>let s = format!(&quot;Hello, {}! You are {} years old.&quot;, &quot;Bob&quot;, 25);<br>// s is now a String</pre><p>Same as println! but returns a String instead of printing it. Extremely common in real-world Rust.</p><h3>vec!</h3><pre>let v1: Vec&lt;i32&gt; = vec![];           // empty<br>let v2 = vec![1, 2, 3, 4, 5];       // initialized with values<br>let v3 = vec![0; 10];               // ten zeros</pre><p>The vec! macro has multiple arms. The second form with ; uses repetition to initialize all elements with the same value — something you couldn&#39;t express as a simple function.</p><h3>assert! and assert_eq!</h3><pre>assert!(2 + 2 == 4);<br>assert!(2 + 2 == 4, &quot;Math is broken!&quot;);</pre><pre>assert_eq!(2 + 2, 4);<br>assert_eq!(2 + 2, 4, &quot;Expected {} but got {}&quot;, 4, 2 + 2);</pre><p>assert! is special because when it fails, it shows you the <em>code</em> you wrote, not just the values. That&#39;s only possible at compile time — a regular function couldn&#39;t inspect its own source.</p><p>assert_eq! goes further: when it fails, it formats both values so you see:</p><pre>thread &#39;main&#39; panicked at &#39;assertion `left == right` failed<br>  left: 5<br> right: 4&#39;</pre><h3>panic!</h3><pre>panic!(&quot;Something went terribly wrong&quot;);<br>panic!(&quot;Expected {}, got {}&quot;, expected, actual);</pre><p>Immediately terminates the current thread with a message. The macro version is needed because it supports formatting — you can include dynamic values in the panic message.</p><h3>dbg!</h3><pre>let x = 5;<br>let y = dbg!(x * 2) + 1; // prints: [src/main.rs:2] x * 2 = 10<br>// y is now 11</pre><p>A beginner’s best friend. dbg! prints the <em>source expression</em> and its value to stderr, then returns the value. Unlike println!, dbg! shows you the file and line number automatically. It&#39;s non-destructive — you can wrap any expression with it without changing the logic.</p><h3>14. Real-World Practical Use Cases</h3><p>Let’s look at some real patterns you’ll encounter in production Rust code.</p><h3>Logging System</h3><pre>macro_rules! log {<br>    (ERROR, $($msg:tt)*) =&gt; {<br>        eprintln!(&quot;[ERROR] {}&quot;, format!($($msg)*));<br>    };<br>    (WARN, $($msg:tt)*) =&gt; {<br>        eprintln!(&quot;[WARN]  {}&quot;, format!($($msg)*));<br>    };<br>    (INFO, $($msg:tt)*) =&gt; {<br>        println!(&quot;[INFO]  {}&quot;, format!($($msg)*));<br>    };<br>    (DEBUG, $($msg:tt)*) =&gt; {<br>        if cfg!(debug_assertions) {<br>            println!(&quot;[DEBUG] {}&quot;, format!($($msg)*));<br>        }<br>    };<br>}</pre><pre>log!(INFO, &quot;Server started on port {}&quot;, 8080);<br>log!(ERROR, &quot;Failed to connect: {}&quot;, error_message);<br>log!(DEBUG, &quot;Processing item #{}&quot;, item_id);</pre><p>This gives you a flexible, level-aware logging system in about 15 lines.</p><h3>Custom Assertions for Testing</h3><pre>macro_rules! assert_between {<br>    ($val:expr, $low:expr, $high:expr) =&gt; {<br>        assert!(<br>            $val &gt;= $low &amp;&amp; $val &lt;= $high,<br>            &quot;Expected {} to be between {} and {}, but it wasn&#39;t&quot;,<br>            $val, $low, $high<br>        );<br>    };<br>}</pre><pre>#[test]<br>fn test_temperature() {<br>    let temp = compute_temperature();<br>    assert_between!(temp, -10.0, 40.0);<br>}</pre><h3>Builder-Style API</h3><pre>macro_rules! request {<br>    ($method:ident $url:expr) =&gt; {<br>        Request::new(Method::$method, $url)<br>    };<br>    ($method:ident $url:expr, body = $body:expr) =&gt; {<br>        Request::new(Method::$method, $url).with_body($body)<br>    };<br>    ($method:ident $url:expr, headers = { $($key:expr =&gt; $val:expr),* }) =&gt; {<br>        {<br>            let mut req = Request::new(Method::$method, $url);<br>            $(req.add_header($key, $val);)*<br>            req<br>        }<br>    };<br>}</pre><pre>let r1 = request!(GET &quot;/api/users&quot;);<br>let r2 = request!(POST &quot;/api/users&quot;, body = json_data);<br>let r3 = request!(GET &quot;/api/users&quot;, headers = { &quot;Auth&quot; =&gt; &quot;Bearer token&quot;, &quot;Accept&quot; =&gt; &quot;application/json&quot; });</pre><h3>Testing Helpers</h3><pre>macro_rules! test_case {<br>    ($name:ident: $input:expr =&gt; $expected:expr) =&gt; {<br>        #[test]<br>        fn $name() {<br>            assert_eq!(process($input), $expected);<br>        }<br>    };<br>}</pre><pre>test_case!(empty_input: &quot;&quot; =&gt; None);<br>test_case!(single_item: &quot;hello&quot; =&gt; Some(&quot;hello&quot;.to_string()));<br>test_case!(with_spaces: &quot;  hi  &quot; =&gt; Some(&quot;hi&quot;.to_string()));</pre><p>This generates three separate test functions with clean names — no copy-paste required.</p><h3>15. DSLs Using Macros</h3><p>One of the most exciting applications of macros is building <strong>Domain-Specific Languages</strong> (DSLs) — mini-languages tailored to a specific problem.</p><p>A DSL is not a full programming language. It’s a set of readable, declarative syntax forms that make expressing solutions in a particular domain feel natural.</p><h3>Why Macros Are Perfect for DSLs</h3><p>Functions constrain you to Rust’s function-call syntax. Macros let you define your own syntax forms. This means you can create APIs that read almost like prose.</p><h3>A Routing DSL</h3><pre>macro_rules! route {<br>    ($path:literal =&gt; $handler:expr) =&gt; {<br>        Router::add(Method::GET, $path, $handler)<br>    };<br>    ($method:ident $path:literal =&gt; $handler:expr) =&gt; {<br>        Router::add(Method::$method, $path, $handler)<br>    };<br>}</pre><pre>route!(&quot;/home&quot; =&gt; home_handler);<br>route!(POST &quot;/login&quot; =&gt; login_handler);<br>route!(DELETE &quot;/users/:id&quot; =&gt; delete_user_handler);</pre><h3>An HTML Template DSL</h3><pre>macro_rules! html {<br>    (head { title: $title:literal }) =&gt; {<br>        format!(&quot;&lt;head&gt;&lt;title&gt;{}&lt;/title&gt;&lt;/head&gt;&quot;, $title)<br>    };<br>    (p { $content:expr }) =&gt; {<br>        format!(&quot;&lt;p&gt;{}&lt;/p&gt;&quot;, $content)<br>    };<br>    (div { $($child:expr)* }) =&gt; {<br>        format!(&quot;&lt;div&gt;{}&lt;/div&gt;&quot;, vec![$($child),*].join(&quot;&quot;))<br>    };<br>}</pre><pre>let page = html!(div {<br>    html!(p { &quot;Hello world&quot; })<br>    html!(p { &quot;This is Rust-generated HTML&quot; })<br>});</pre><h3>An Event System DSL</h3><pre>macro_rules! on_event {<br>    ($event:ident =&gt; $handler:block) =&gt; {<br>        EventBus::subscribe(stringify!($event), |_| $handler);<br>    };<br>    ($event:ident ($data:ident: $ty:ty) =&gt; $handler:block) =&gt; {<br>        EventBus::subscribe(stringify!($event), |raw| {<br>            let $data: $ty = raw.downcast().unwrap();<br>            $handler<br>        });<br>    };<br>}</pre><pre>on_event!(user_joined =&gt; {<br>    println!(&quot;Someone joined!&quot;);<br>});</pre><pre>on_event!(message_received(msg: String) =&gt; {<br>    println!(&quot;Got message: {}&quot;, msg);<br>});</pre><p>DSL macros make Rust code read more like specifications than implementations. They’re a key reason why Rust frameworks can feel elegant despite Rust’s verbosity in raw form.</p><h3>16. Procedural Macros: The Next Level</h3><p>Everything we’ve covered so far has been <strong>declarative macros</strong> (macro_rules!). There&#39;s a more powerful type called <strong>procedural macros</strong> that work differently.</p><p>Instead of pattern-matching token trees, procedural macros are full Rust programs that:</p><ol><li>Accept a stream of tokens as input</li><li>Process those tokens using Rust code</li><li>Return a new stream of tokens as output</li></ol><p>There are three kinds:</p><h3>1. Derive Macros</h3><p>These are attached to structs and enums to automatically implement traits.</p><pre>#[derive(Debug, Clone, PartialEq)]<br>struct Point {<br>    x: f64,<br>    y: f64,<br>}</pre><p>The #[derive(Debug)] attribute triggers a procedural macro that reads your struct&#39;s definition and generates a Debug implementation for it — something that would require many lines of boilerplate by hand.</p><p>You can write your own derive macros. The popular serde crate uses this to auto-generate serialization/deserialization code:</p><pre>#[derive(Serialize, Deserialize)]<br>struct Config {<br>    host: String,<br>    port: u16,<br>}</pre><p>With just one line, serde generates hundreds of lines of custom serialization code.</p><h3>2. Attribute Macros</h3><p>These can be placed on any item and transform it:</p><pre>#[tokio::main]<br>async fn main() {<br>    // async code here<br>}</pre><p>#[tokio::main] is an attribute macro that wraps your main function with Tokio&#39;s async runtime setup. Without it, you&#39;d need to write boilerplate to initialize the runtime yourself.</p><p>Another common one:</p><pre>#[route(GET, &quot;/&quot;)]<br>async fn index(req: HttpRequest) -&gt; impl Responder {<br>    &quot;Hello!&quot;<br>}</pre><p>Frameworks like Actix-web use attribute macros to register route handlers cleanly.</p><h3>3. Function-Like Procedural Macros</h3><p>These look like macro_rules! calls but are backed by Rust code:</p><pre>let sql = sql!(SELECT * FROM users WHERE id = 1);</pre><p>A function-like procedural macro can parse the SQL string at compile time, validate it, and generate optimized Rust code — catching SQL errors before your program even runs.</p><h3>When to Use Procedural vs. Declarative</h3><p>Use macro_rules! when:</p><ul><li>Your pattern is straightforward</li><li>You don’t need to analyze tokens beyond matching</li><li>You want something simple and self-contained</li></ul><p>Use procedural macros when:</p><ul><li>You need complex token analysis</li><li>You’re generating code based on struct fields or variants</li><li>You want to implement derive traits</li><li>You need compiler error messages with precise spans</li></ul><h3>17. Common Beginner Mistakes</h3><p>Learning from mistakes saves time. Here are the ones that trip up almost every new Rust macro writer.</p><h3>Mistake 1: Using a Macro When a Function Would Do</h3><p>Macros add complexity. If your macro doesn’t need variadic arguments, custom syntax, or compile-time code generation, write a function instead.</p><pre>// ❌ Don&#39;t do this<br>macro_rules! add_numbers {<br>    ($a:expr, $b:expr) =&gt; { $a + $b };<br>}<br>// ✅ Just write this<br>fn add_numbers(a: i32, b: i32) -&gt; i32 { a + b }</pre><h3>Mistake 2: Wrong Fragment Specifier</h3><pre>// ❌ Wrong: using expr when you need ident<br>macro_rules! define_fn {<br>    ($name:expr) =&gt; {<br>        fn $name() {} // ← error! expr can&#39;t be used as a function name<br>    };<br>}</pre><pre>// ✅ Correct: use ident<br>macro_rules! define_fn {<br>    ($name:ident) =&gt; {<br>        fn $name() {}<br>    };<br>}</pre><p>When you want a name, use ident. When you want a value, use expr. Mixing these up is the most common fragment specifier mistake.</p><h3>Mistake 3: Forgetting the Trailing Comma Pattern</h3><p>Not including $(,)? means your macro will fail for users who habitually add trailing commas:</p><pre>// ❌ Will reject: my_macro!(a, b, c,)<br>macro_rules! my_macro {<br>    ($($x:expr),*) =&gt; { /* ... */ };<br>}</pre><pre>// ✅ Accept trailing commas gracefully<br>macro_rules! my_macro {<br>    ($($x:expr),* $(,)?) =&gt; { /* ... */ };<br>}</pre><h3>Mistake 4: Unexpected Expansion Behavior with Expressions</h3><pre>macro_rules! double_call {<br>    ($x:expr) =&gt; {<br>        $x + $x // ← $x is expanded TWICE<br>    };<br>}<br>let mut n = 0;<br>let result = double_call!({ n += 1; n }); // n is incremented TWICE!<br>// result is 1 + 2 = 3, not 2!</pre><p>If $x is a side-effectful expression, expanding it multiple times causes the side effects to run multiple times. Fix this by storing in a temporary:</p><pre>macro_rules! double_call {<br>    ($x:expr) =&gt; {{<br>        let val = $x; // evaluate once<br>        val + val<br>    }};<br>}</pre><h3>Mistake 5: Infinite Recursion</h3><pre>// ❌ This will cause a compile error (recursion limit reached)<br>macro_rules! forever {<br>    () =&gt; { forever!() };<br>}</pre><p>Macros can call themselves recursively (useful for processing lists), but you must always have a base case that stops the recursion.</p><pre>// ✅ Recursive macro with base case<br>macro_rules! count_items {<br>    () =&gt; { 0 };                     // base case<br>    ($first:expr $(, $rest:expr)*) =&gt; {<br>        1 + count_items!($($rest),*)  // recursive case<br>    };<br>}<br>let c = count_items!(a, b, c, d); // 4</pre><h3>Mistake 6: Confusing Macro Scope</h3><p>Macros defined in a child module are not visible in the parent module unless exported. Beginners often define a macro in a module and wonder why it’s not accessible elsewhere.</p><h3>18. Best Practices</h3><h4>When to Use Macros</h4><ul><li>Variadic arguments (different number of inputs)</li><li>Generating repetitive code patterns</li><li>DSL-style readable syntax</li><li>Compile-time validation or code generation</li><li>Test helper infrastructure</li></ul><h4>When Not to Use Macros</h4><ul><li>Logic that could be a regular function</li><li>Simple computations</li><li>Situations where you want clear stack traces</li><li>Code others need to easily read and understand</li><li>Anything that can be done with generics or traits</li></ul><h3>Naming Conventions</h3><ul><li>Use snake_case for macro names: my_macro!, not MyMacro!</li><li>Make the name clearly describe what the macro does</li><li>If a macro generates a type or function, the name should hint at that</li></ul><h3>Documentation</h3><p>Always document macros, especially if they’re public. The /// doc comment works on macros just like on functions:</p><pre>/// Creates a HashMap from key-value pairs.<br>///<br>/// # Example<br>/// ```<br>/// let map = map!{ &quot;a&quot; =&gt; 1, &quot;b&quot; =&gt; 2 };<br>/// assert_eq!(map[&quot;a&quot;], 1);<br>/// ```<br>#[macro_export]<br>macro_rules! map {<br>    ($($key:expr =&gt; $val:expr),* $(,)?) =&gt; {<br>        {<br>            let mut m = std::collections::HashMap::new();<br>            $(m.insert($key, $val);)*<br>            m<br>        }<br>    };<br>}</pre><h3>Testing Macros</h3><p>You can test macros just like functions using #[test]:</p><pre>#[cfg(test)]<br>mod tests {<br>    use super::*;<br>#[test]<br>    fn test_sum_macro_empty() {<br>        assert_eq!(sum!(), 0);<br>    }<br>    #[test]<br>    fn test_sum_macro_multiple() {<br>        assert_eq!(sum!(1, 2, 3, 4, 5), 15);<br>    }<br>    #[test]<br>    fn test_sum_trailing_comma() {<br>        assert_eq!(sum!(1, 2, 3,), 6);<br>    }<br>}</pre><h3>Debugging with cargo expand</h3><p>The most useful tool for debugging macros is cargo-expand. Install it with:</p><pre>cargo install cargo-expand</pre><p>Then run:</p><pre>cargo expand</pre><p>This shows you your entire source file after all macros have been expanded. When a macro isn’t behaving as expected, expanding it is the fastest way to see what’s actually being generated.</p><h3>What to Practice Next</h3><ol><li><strong>Rebuild </strong><strong>vec! from scratch.</strong> Write a macro that takes comma-separated expressions and creates a Vec. Add support for trailing commas and the [value; count] form.</li><li><strong>Write a </strong><strong>hashmap! macro.</strong> Accept key =&gt; value pairs and produce a HashMap.</li><li><strong>Create a </strong><strong>test_cases! macro.</strong> Generate multiple test functions from a compact table of inputs and expected outputs.</li><li><strong>Build a simple logging macro.</strong> Support log levels (INFO, WARN, ERROR) and format strings.</li><li><strong>Explore </strong><strong>cargo expand.</strong> Take any macro from the standard library, put it in your code, and expand it to see what it generates.</li></ol><h3>Suggested Mini Projects</h3><ul><li><strong>A </strong><strong>retry! macro</strong> that retries a block of code N times on failure</li><li><strong>A </strong><strong>timed! macro</strong> that measures and prints the execution time of any block</li><li><strong>A simple query builder DSL</strong> using macros</li><li><strong>An </strong><strong>impl_trait_for_all! macro</strong> that implements a simple trait for multiple types at once</li></ul><h3>Recommended Next Topics</h3><p>Once you’re comfortable with declarative macros, the natural next step is <strong>procedural macros</strong> — writing derive macros, building your own #[derive(MyTrait)]. The syn and quote crates are the tools of choice.</p><p>From there, explore how major Rust libraries use macros in practice:</p><ul><li><strong>serde</strong> — for automatic serialization</li><li><strong>tokio</strong> — for async runtime setup</li><li><strong>diesel</strong> — for type-safe SQL queries</li></ul><p>Rust macros are one of those features that, once you understand them, you start seeing opportunities everywhere. They’re the reason Rust can be both low-level and expressive — the kind of language where you can define your own ergonomic abstractions without paying runtime costs.</p><p>The ! was never just punctuation. It was always an invitation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f202c0e2b243" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Memory Management in Rust]]></title>
            <link>https://medium.com/@yashwantsaste/memory-management-in-rust-b7125d2197f6?source=rss-384f60710a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7125d2197f6</guid>
            <category><![CDATA[rust-memory-safety]]></category>
            <category><![CDATA[rust-programming-language]]></category>
            <category><![CDATA[rustlang]]></category>
            <category><![CDATA[memory-management]]></category>
            <dc:creator><![CDATA[Yashwant Sanjay Saste]]></dc:creator>
            <pubDate>Mon, 23 Mar 2026 20:09:32 GMT</pubDate>
            <atom:updated>2026-03-23T20:09:32.028Z</atom:updated>
            <content:encoded><![CDATA[<p>Memory management is a core part of programming, especially in systems programming. Every program uses memory, and how it is managed affects performance, stability, and safety.</p><figure><img alt="Memory Management in Rust" src="https://cdn-images-1.medium.com/max/1024/1*WPck0P-6NaN4pmkALXRa0A.png" /></figure><p>Traditionally, there are two main approaches.</p><ul><li>The first is <strong>Garbage Collection (GC)</strong>, used in languages like Java and Go. It automatically frees unused memory, making development easier. However, it runs at runtime, which can cause delays and adds overhead.</li><li>The second is <strong>manual memory management</strong>, used in languages like C and C++. Here, developers control allocation and deallocation (malloc and free). This gives more control and performance, but it is error-prone and can lead to issues like memory leaks and dangling pointers.</li></ul><p>Rust takes a different approach.</p><ul><li>It uses <strong>ownership</strong>, <strong>borrowing</strong>, and <strong>compile-time checks</strong> to manage memory. This means memory safety is enforced during compilation, not at runtime. As a result, Rust does not need a garbage collector, and developers don’t need to manually free memory.</li><li>Memory is automatically cleaned up when it goes out of scope, following strict rules checked by the compiler. This prevents common bugs and ensures safe memory usage.</li></ul><p>This gives Rust a unique advantage: it combines the performance and control of low-level languages with the safety of high-level languages.</p><p>In this blog, we will explore how Rust manages memory through <strong>mutability</strong>, <strong>stack vs heap memory</strong>, <strong>ownership</strong>, and <strong>borrowing with references</strong> — the main conceptual blocks of Rust’s memory model.</p><h3>Core Concept of Memory Management in Rust</h3><p>Rust is designed with a simple goal in mind: <strong>provide both safety and performance</strong>. In many languages, you often have to choose between the two — but Rust aims to give you both without compromise.</p><p>One of the key ways Rust achieves this is through <strong>compile-time memory checks</strong>. Instead of handling memory during program execution (runtime), Rust checks everything while the code is being compiled. This means there is <strong>no extra runtime cost</strong>, and your program runs efficiently without hidden overhead.</p><p>Unlike languages that rely on a <strong>garbage collector (GC)</strong>, Rust does not have one. Garbage collectors exist to automatically free unused memory, which makes development easier. However, they can introduce <strong>latency (pauses)</strong> and <strong>performance overhead</strong>, especially in performance-critical applications.</p><p>On the other hand, manual memory management (like in C and C++) gives full control to the developer but comes with risks. Common issues include:</p><ul><li><strong>Memory leaks</strong>, where memory is never freed</li><li><strong>Dangling pointers</strong>, where memory is accessed after being freed</li></ul><p>Rust avoids both these problems with a different approach.</p><p>It uses an <strong>ownership system</strong>, where every piece of data has a clear owner. When the owner goes out of scope, the memory is automatically cleaned up. This happens without needing a garbage collector or manual free calls.</p><p>In simple terms, Rust ensures that:</p><ul><li>Memory is used safely</li><li>Memory is freed at the right time</li><li>No extra runtime system is needed</li></ul><p>This is what makes Rust’s memory management both <strong>safe and efficient</strong>.</p><h3>Mutability in Rust</h3><p>In Rust, variables are <strong>immutable by default</strong>. This means once a value is assigned, it cannot be changed unless explicitly allowed.</p><pre>let x = 5;<br>// x = 6; ❌ Error</pre><p>If you want to make a variable changeable, you must use the mut keyword:</p><pre>let mut x = 5;<br>x = 6; // ✅ Allowed</pre><p>This design forces developers to <strong>think before allowing changes</strong>, which leads to safer and more predictable code.</p><p>Immutability is important for several reasons:</p><ul><li>It <strong>prevents unintended changes</strong>, reducing bugs</li><li>It helps the <strong>compiler optimize code better</strong></li><li>It makes programs <strong>safer in concurrent environments</strong>, where multiple parts of a program run at the same time</li></ul><p>From a memory management perspective, immutability plays a key role in <strong>ensuring safety</strong>. When data cannot change unexpectedly, it becomes much easier to track and manage memory correctly.</p><p>By making mutability explicit, Rust gives you control while still protecting you from common mistakes, contributing to its overall <strong>safe memory model</strong>.</p><h3>Stack vs Heap Memory in Rust</h3><p>Understanding how Rust uses memory requires knowing the difference between <strong>stack</strong> and <strong>heap</strong>.</p><h4>What is Stack Memory</h4><p>The <strong>stack</strong> is a region of memory that follows a <strong>Last In, First Out (LIFO)</strong> order.<br>It is very fast because memory is allocated and freed automatically in a fixed pattern.</p><ul><li>Stores <strong>fixed-size data</strong> (like integers, booleans)</li><li>Very <strong>fast allocation and deallocation</strong></li><li>Managed automatically based on scope</li></ul><h4>What is Heap Memory</h4><p>The <strong>heap</strong> is used for data that can change in size or is not known at compile time.</p><ul><li>Uses <strong>dynamic allocation</strong></li><li>Slightly <strong>slower access</strong> (needs pointer dereferencing)</li><li>Used for flexible data like strings and collections</li></ul><p>Rust uses the stack for:</p><ul><li>Local variables</li><li>Function calls (stack frames)</li></ul><p>Each function creates a <strong>stack frame</strong>, and when the function ends, all its stack data is automatically removed.</p><p>For dynamic data types like String and Vec:</p><ul><li>The <strong>actual data is stored on the heap</strong></li><li>A <strong>pointer to that data is stored on the stack</strong></li></ul><p>So, the stack holds:</p><ul><li>Pointer</li><li>Length</li><li>Capacity</li></ul><p>While the heap holds:</p><ul><li>The actual content</li></ul><pre>fn main() {<br>    // Stack allocation<br>    let x = 10;// Heap allocation<br>    let s = String::from(&quot;Hello&quot;);<br>    println!(&quot;x = {}&quot;, x);<br>    println!(&quot;s = {}&quot;, s);<br>}</pre><ul><li>x is stored <strong>directly on the stack</strong> (fixed size, simple value)</li><li>s has:</li><li>A <strong>pointer on the stack</strong></li><li>Actual string data stored on the <strong>heap</strong></li></ul><p>When main ends:</p><ul><li>x is automatically removed from the stack</li><li>s goes out of scope → Rust <strong>automatically frees the heap memory</strong></li></ul><p>This clear separation between stack and heap, along with automatic cleanup, helps Rust achieve both <strong>performance and memory safety</strong>.</p><h3>Ownership</h3><p>Ownership is the <strong>core concept</strong> behind Rust’s memory management. It is a set of rules that the compiler follows to ensure memory is used safely and cleaned up correctly — without needing a garbage collector.</p><p>In simple terms, <strong>ownership means that every value in Rust has a single owner</strong>. The owner is responsible for that value, and when the owner goes out of scope, the value is automatically removed from memory.</p><h4>Rules of Ownership</h4><p>Rust enforces three main rules:</p><ol><li><strong>Each value has one owner</strong></li><li><strong>There can only be one owner at a time</strong></li><li><strong>When the owner goes out of scope, the value is dropped (memory is freed)</strong></li></ol><p>These rules ensure that memory is always managed safely and predictably.</p><h4>Move Semantics</h4><p>When working with heap data (like String), assigning a value to another variable does <strong>not copy the data</strong>. Instead, Rust <strong>moves ownership</strong>.</p><p>This avoids unnecessary memory duplication and ensures there is only one owner at a time.</p><pre>fn main() {<br>    let s1 = String::from(&quot;Hello&quot;);<br>    let s2 = s1;  // Ownership moved to s2<br>    // println!(&quot;{}&quot;, s1); ❌ Error: s1 is no longer valid<br>    println!(&quot;{}&quot;, s2);   // ✅ Works<br>}</pre><p>Here:</p><ul><li>s1 originally owns the string</li><li>Ownership is <strong>moved</strong> to s2</li><li>s1 becomes invalid to prevent double-free errors</li></ul><h4>Copy vs Move</h4><p>Not all types behave this way.</p><ul><li><strong>Heap data (like </strong><strong>String) → Move</strong></li><li><strong>Primitive types (like integers) → Copy</strong></li></ul><p>Primitive types are small and stored on the stack, so Rust simply creates a copy instead of moving ownership.</p><h4>Copy Example (Stack Data)</h4><pre>fn main() {<br>    let x = 10;<br>    let y = x;  // Value is copied<br>    println!(&quot;x = {}&quot;, x); // ✅ Works<br>    println!(&quot;y = {}&quot;, y); // ✅ Works<br>}</pre><p>Here:</p><ul><li>x is copied to y</li><li>Both variables are valid</li></ul><h4>Move Example (Heap Data)</h4><pre>fn main() {<br>    let s1 = String::from(&quot;Hello&quot;);<br>    let s2 = s1;  // Ownership moved<br>    println!(&quot;{}&quot;, s2); // ✅ Works<br>    // println!(&quot;{}&quot;, s1); ❌ Error<br>}</pre><h4>Why Ownership Matters</h4><p>Ownership helps Rust:</p><ul><li>Prevent <strong>memory leaks</strong></li><li>Avoid <strong>double-free errors</strong></li><li>Ensure <strong>safe and predictable memory usage</strong></li></ul><p>All of this happens at <strong>compile time</strong>, making Rust both <strong>safe and efficient</strong> without needing a garbage collector.</p><h3>Borrowing and References</h3><p>In Rust, sometimes you don’t want to transfer ownership of a value — you just want to <strong>use it temporarily</strong>. This is where <strong>references</strong> and <strong>borrowing</strong> come in.</p><h3>References</h3><p>A <strong>reference</strong> allows you to access a value <strong>without taking ownership</strong> of it. Instead of passing the actual value, you pass its address.</p><pre>fn main() {<br>    let s1 = String::from(&quot;Hello&quot;);<br>    let s2 = &amp;s1;<br>    println!(&quot;{}&quot;, s2);<br>    println!(&quot;{}&quot;, s1); // ✅ Still valid<br>}</pre><p>Here:</p><ul><li>s2 is a reference to s1</li><li>Ownership is <strong>not transferred</strong></li><li>Both s1 and s2 can be used</li></ul><p>References let you reuse data safely without copying or moving it.</p><h3>Borrowing</h3><p>When you pass a reference to a function, it is called <strong>borrowing</strong>. The function can use the value, but it does not own it.</p><pre>fn main() {<br>    let my_string = String::from(&quot;Hello, Rust!&quot;);<br>    takes_ownership(&amp;my_string);  // Borrowing<br>    println!(&quot;{}&quot;, my_string);    // ✅ Still valid<br>}<br><br>fn takes_ownership(some_string: &amp;String) {<br>    println!(&quot;{}&quot;, some_string);<br>}</pre><p>Here:</p><ul><li>my_string is borrowed by the function</li><li>Ownership remains with main</li><li>The value can still be used after the function call</li></ul><h3>Mutable References</h3><p>If you want to modify a value, you can use a <strong>mutable reference</strong> (&amp;mut).</p><pre>fn main() {<br>    let mut s1 = String::from(&quot;Hello&quot;);<br>    update_word(&amp;mut s1);<br>    println!(&quot;{}&quot;, s1);<br>}<br><br>fn update_word(word: &amp;mut String) {<br>    word.push_str(&quot; World&quot;);<br>}</pre><p>Here:</p><ul><li>s1 is mutable</li><li>The function borrows it and modifies the value</li></ul><h3>Rules of Borrowing</h3><p>Rust enforces strict rules to prevent bugs and unsafe behavior:</p><h4>1. Multiple immutable references are allowed</h4><pre>fn main() {<br>    let s1 = String::from(&quot;Hello&quot;);<br>    let s2 = &amp;s1;<br>    let s3 = &amp;s1;<br>    println!(&quot;{}&quot;, s1);<br>    println!(&quot;{}&quot;, s2);<br>    println!(&quot;{}&quot;, s3);<br>}</pre><p>No issues here because no one is modifying the data.</p><h4>2. Only one mutable reference at a time</h4><pre>fn main() {<br>    let mut s1 = String::from(&quot;Hello&quot;);<br>    let s2 = &amp;mut s1;<br>    update_word(&amp;mut s1); // ❌ Error<br>    println!(&quot;{}&quot;, s2);<br>}</pre><p>This is not allowed because it creates multiple mutable references.</p><h4>3. Cannot mix mutable and immutable references</h4><pre>fn main() {<br>    let mut s1 = String::from(&quot;Hello&quot;);<br>    let s2 = &amp;mut s1;<br>    let s3 = &amp;s1; // ❌ Error<br>    println!(&quot;{}&quot;, s2);<br>}</pre><p>This is not allowed because one part could modify the data while another expects it to stay the same.</p><h4>Why These Rules Exist</h4><p>These rules help Rust prevent:</p><ul><li><strong>Data races</strong></li><li><strong>Unexpected changes in data</strong></li><li><strong>Inconsistent program behavior</strong></li></ul><p>If a value is borrowed immutably, Rust guarantees it won’t change.<br>If it’s borrowed mutably, Rust ensures there is only one reference modifying it.</p><p>This system allows Rust to provide <strong>safe concurrency and memory safety</strong>, all enforced at compile time — without needing a garbage collector.</p><p>💡 <em>Note:</em> Advanced concepts like <strong>lifetimes</strong> and <strong>string slices (</strong><strong>&amp;str)</strong> build on this, but can be explored separately.</p><h3>Conclusion</h3><p>Rust offers a unique way to handle memory by combining <strong>safety and performance</strong> without using a garbage collector. Instead, it enforces rules at <strong>compile time</strong>, avoiding runtime overhead.</p><p>With concepts like <strong>immutability</strong>, <strong>stack vs heap</strong>, <strong>ownership</strong>, and <strong>borrowing</strong>, Rust ensures memory is:</p><ul><li>Used efficiently</li><li>Cleaned up automatically</li><li>Safe from issues like leaks and data races</li></ul><p>Although these rules may feel strict initially, they help you write <strong>safe, predictable, and high-performance code</strong>.</p><p>From here, concepts like <strong>lifetimes</strong> and <strong>string slices (</strong><strong>&amp;str)</strong> can further deepen your understanding of Rust’s memory model.</p><h3>References</h3><ul><li><em>The Rust Programming Language (Book)</em> — Chapter 4: Ownership<br> <a href="https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html">https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html</a></li><li><em>The Rust Programming Language (Book)</em> — References and Borrowing<br> <a href="https://doc.rust-lang.org/book/ch04-02-references-and-borrowing.html">https://doc.rust-lang.org/book/ch04-02-references-and-borrowing.html</a></li><li><em>Rust Official Documentation</em><br> <a href="https://www.rust-lang.org/learn">https://www.rust-lang.org/learn</a></li><li>Rust BootCamp — Harkirat Singh<br> <a href="https://projects.100xdevs.com/tracks/rust-bootcamp/Rust-Bootcamp-1">https://projects.100xdevs.com/tracks/rust-bootcamp/Rust-Bootcamp-1</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7125d2197f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning NGINX Properly and Setting Up a Basic Static Server with NGINX]]></title>
            <link>https://medium.com/@yashwantsaste/learning-nginx-properly-and-setting-up-a-basic-static-server-with-nginx-81bb0a8c91ad?source=rss-384f60710a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/81bb0a8c91ad</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[reverse-proxy]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[load-balancing]]></category>
            <dc:creator><![CDATA[Yashwant Sanjay Saste]]></dc:creator>
            <pubDate>Tue, 09 Dec 2025 22:17:34 GMT</pubDate>
            <atom:updated>2025-12-09T22:17:34.819Z</atom:updated>
            <content:encoded><![CDATA[<p>Learning &amp; Implementing NGINX: Reverse Proxy, Load Balancing, and Containerized Setup — Step-by-Step Tutorial</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WhoAWltKnKbaXDqn" /></figure><p>I recently finished an NGINX crash course that helped me understand the core concepts properly instead of just copying configurations. To make sure I really learned it, I rebuilt the entire demo from scratch and uploaded it <a href="https://github.com/YashwantSaste/nginx-learning">here</a>.</p><p>This blog explains the same concepts in a clear, straightforward way so anyone learning NGINX can follow along without confusion or unnecessary complexity.</p><h3>What NGINX Is and Why It Matters</h3><p>NGINX is a high-performance <em>web server</em> built to handle a large number of connections efficiently.</p><blockquote>A <strong>web server</strong> is a program that listens for requests from a browser and sends back the correct files or responses.<br> For example, when you open example.com, your browser sends an HTTP request to a machine running a web server. That server finds the right file—like index.html—and returns it.</blockquote><blockquote>A web server does two basic things:</blockquote><blockquote><strong>Accepts incoming HTTP/HTTPS requests</strong><br> (like “give me /home.css” or “POST this login form”).</blockquote><blockquote><strong>Sends the appropriate response</strong><br> (HTML, JSON, images, errors, etc.).</blockquote><blockquote>That’s it at the core.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qZSOsEiQYpuv9lEsCVzRsQ.png" /><figcaption>NGINX Core Features</figcaption></figure><p>Because of these capabilities, NGINX is widely used in high-traffic platforms, containerized environments, and cloud-native systems.</p><p>NGINX uses an event-driven, non-blocking architecture. Instead of creating many processes, a single worker process can manage thousands of active connections at the same time. This reduces memory usage and improves performance, especially for APIs and applications that deal with many concurrent users.</p><h3>Key Concepts in NGINX</h3><h4>1. nginx.conf</h4><p>nginx.conf is the <strong>main configuration file of NGINX</strong>.</p><blockquote>Every behavior of NGINX—routing, proxying, load balancing, caching, compression, security—is controlled from this file.</blockquote><p>NGINX reads this file <strong>every time it starts or reloads</strong>, and it decides how NGINX will function.</p><h4>2. Worker Processes</h4><p>worker_processes defines how many worker processes NGINX should start.</p><blockquote>A worker process is responsible for handling all incoming requests. NGINX uses an event-driven model, so even a single worker can manage thousands of active connections efficiently.</blockquote><p>Example:</p><pre>worker_processes auto;</pre><h4>How it works</h4><ul><li>Each worker is single-threaded.</li><li>Workers do not share request handling; each one independently manages its own set of connections.</li><li>The master process only manages workers — it doesn’t handle traffic.</li></ul><h4>Best practice</h4><p>Set:</p><pre>worker_processes auto;</pre><p>This automatically matches the number of CPU cores on the machine, which is the recommended setting for almost all production environments.</p><h4>3. Worker Connections</h4><p>worker_connections specifies how many simultaneous connections <strong>each</strong> worker process can handle.</p><p>Example:</p><pre>worker_connections 1024;</pre><p>This means:</p><ul><li>If you have 4 worker processes → max ~4096 connections (approx).</li><li>Actual usable connections may be slightly less due to internal connections and OS limits.</li></ul><p>Higher worker connections = more concurrent clients NGINX can support without overload.</p><h4>Best Practice</h4><ul><li>Increase worker_connections based on expected traffic</li><li>Common values: 1024, 4096, sometimes higher.</li><li>Always check OS limits using ulimit -nbecause NGINX cannot exceed the system’s file descriptor limit.</li></ul><h4>4. Directives and Contexts in NGINX</h4><h4>Directives</h4><p>A directive is a single configuration instruction.<br> Example:</p><pre>worker_processes auto;</pre><p>Every directive ends with a semicolon.</p><h4>Contexts</h4><p>Contexts (also called <em>blocks</em>) group related directives.<br> Example:</p><pre>http { ... }</pre><p>Different contexts handle different types of traffic:</p><ul><li><strong>events { }</strong> → general connection processing (how NGINX accepts connections)</li><li><strong>http { }</strong> → HTTP and HTTPS traffic (websites, APIs)</li><li><strong>mail { }</strong> → email protocols like SMTP</li><li><strong>stream { }</strong> → TCP and UDP traffic (databases, raw services)</li></ul><p>In most web applications, you mainly work with events {} and http {}.</p><h4>5. Server Block</h4><p>Inside the http context, you define one or more <strong>server blocks</strong>.<br> A server block tells NGINX how to handle requests for a particular domain or application.</p><p>Example:</p><pre>server {<br>    listen 80;<br>    server_name example.com;<br>    location / { }<br>}</pre><h4>Key Directives Inside a Server Block</h4><h4>a. listen</h4><p>Defines which port (and optionally IP) the server should serve.</p><pre>listen 80;<br>listen 443 ssl;</pre><h4>b. server_name</h4><p>Which domain(s) this block responds to.</p><pre>server_name example.com www.example.com;</pre><h4>c. location</h4><p>Controls how NGINX handles specific paths/URLs.</p><pre>location / {<br>    # rules for /<br>}<br>location /api {<br>    # rules for /api<br>}</pre><h3>Understanding the &quot;location&quot; Block and How Reverse Proxying Works</h3><p>The location block is one of the most important parts of an NGINX config.<br> It defines <strong>what NGINX should do for a specific URL path</strong>.</p><p>A location block matches part of the URL and applies specific rules to it.<br>Examples:</p><pre>location /images {<br>    # handles requests like /images/logo.png<br>}<br>location /api {<br>    # handles requests like /api/users<br>}</pre><p>NGINX looks at the request path and picks the <strong>best matching location</strong>.<br> This is how you send some routes to static files, some to your backend, and some to special rules.</p><h4>Common Uses of Location Blocks</h4><ul><li>Serving static files (HTML, CSS, JS, images)</li><li>Proxying API routes to a backend server</li><li>Adding caching rules</li><li>Blocking or restricting access</li><li>Handling custom error pages</li></ul><h4>Forwarding Requests to a Backend</h4><pre>location /api {<br>    proxy_pass http://backend;<br>}</pre><p>Here, backend refers to an upstream block that defines backend servers.</p><ul><li><strong>Upstream server:</strong> The backend server(s) that NGINX forwards requests to.</li><li><strong>Downstream server:</strong> The client or system that receives the response from NGINX after it processes or proxies the request.<br>Example:</li></ul><pre>upstream backend {<br>    server 127.0.0.1:3001;<br>    server 127.0.0.1:3002;<br>}</pre><p>This setup allows NGINX to distribute requests using round-robin by default, automatically retry failed servers, and enable horizontal scaling without changing the client-facing endpoint.</p><h4>Forwarding the Correct Headers (proxy_set_header)</h4><p>When NGINX proxies a request to your backend app, you usually want the original request details to be forwarded.<br> This is done using proxy_set_header.</p><p>Common ones:</p><pre>proxy_set_header Host $host;                        # original domain<br>proxy_set_header X-Real-IP $remote_addr;           # real client IP<br>proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  # chain of client IPs<br>proxy_set_header X-Forwarded-Proto $scheme;        # http or https</pre><p>These headers ensure the backend sees the real client IP instead of NGINX’s IP, knows whether the original request was HTTP or HTTPS, enables correct logging and security rules, and supports frameworks like Express and Django that rely on this information.<br>Here’s a full example combining location, upstream, proxy_pass, and headers:</p><pre>upstream backend {<br>    server 127.0.0.1:3001;<br>    server 127.0.0.1:3002;<br>}</pre><pre>server {<br>    listen 80;<br>    server_name example.com;</pre><pre>    location /api/ {<br>        proxy_pass <a href="http://backend;/">http://backend;</a><br>        <br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>    }<br>}</pre><h3>Setting Up a Basic Static Server with NGINX</h3><p>In this section , we’ll set up a simple Node.js app, containerize it, and use NGINX as a reverse proxy with load balancing. We’ll also explain how to safely modify NGINX configuration on your system.</p><blockquote><em>⚠️ </em><strong><em>Note:</em></strong><em> The </em><em>nginx.conf.template and </em><em>nginx.conf.ssl.template in the repo are for reference only. You </em><strong><em>cannot use them directly</em></strong><em>—you need to modify the configuration file in your local NGINX installation.</em></blockquote><h3>Step 0: Install NGINX on Your System</h3><ul><li><strong>Windows:</strong> Use <a href="http://nginx.org/en/docs/windows.html">NGINX Windows builds</a>.</li><li><strong>Linux:</strong> sudo apt install nginx (Ubuntu/Debian) or sudo yum install nginx (CentOS).</li></ul><p>Check the default configuration location:</p><ul><li><strong>Linux:</strong> /etc/nginx/nginx.conf</li><li><strong>Windows:</strong> nginx/conf/nginx.conf</li></ul><blockquote><em>You will edit </em><strong><em>your own NGINX configuration</em></strong><em>, using my templates as a reference.</em></blockquote><h3>Step 1: Run the Node.js Backend</h3><p>The repo has a simple Node.js server (server.js) that serves static files from public/ and listens on <strong>port 3000</strong> inside the container.</p><pre>git clone https://github.com/YashwantSaste/nginx-learning.git<br>cd nginx-learning<br>npm install<br>node server.js</pre><ul><li>Open http://localhost:3000 in your browser — you should see the static page.</li></ul><p>At this stage, the server is running directly on your machine.</p><h3>Step 2: Dockerize the Backend</h3><p>The Dockerfile in the repo containerizes the Node.js server:</p><pre>docker build -t nginx-learning-app .<br>docker run -d -p 3001:3000 --name app1 nginx-learning-app<br>docker run -d -p 3002:3000 --name app2 nginx-learning-app<br>docker run -d -p 3003:3000 --name app3 nginx-learning-app</pre><ul><li>Each container internally listens on <strong>3000</strong>, but maps to different <strong>host ports</strong> (3001, 3002, 3003).</li><li>Test the containers individually:<br>http://localhost:3001 → app1<br>http://localhost:3002 → app2<br>http://localhost:3003 → app3</li></ul><h3>Step 3: Configure NGINX as Reverse Proxy</h3><ol><li><strong>Locate your NGINX configuration file</strong> on your system (nginx.conf).</li><li><strong>Open the template</strong> (nginx.conf.template or nginx.conf.ssl.template) from the repo as a reference.</li><li><strong>Modify your NGINX config</strong> to include:</li></ol><pre>upstream nodejs_cluster {<br>    least_conn;<br>    server 127.0.0.1:3001;<br>    server 127.0.0.1:3002;<br>    server 127.0.0.1:3003;<br>}</pre><pre>server {<br>    listen 8080;<br>    server_name localhost;</pre><pre>    location / {<br>        proxy_pass <a href="http://nodejs_cluster;/">http://nodejs_cluster;</a><br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>    }<br>}</pre><ul><li><strong>upstream block</strong> → defines backend containers (3001, 3002, 3003).</li><li><strong>server block</strong> → listens on <strong>8080</strong> and forwards requests to the backend.</li><li><strong>proxy_set_header</strong> → ensures the backend knows original client info.</li></ul><blockquote><em>You can also refer to the SSL template to add HTTPS (443) and HTTP → HTTPS redirect (8080).</em></blockquote><h3>Step 4: Restart NGINX</h3><p>After editing your config:</p><pre># Linux<br>sudo nginx -t   # test config<br>sudo systemctl restart nginx<br><br># Windows<br>nginx -t   # test config<br>nginx -s reload</pre><ul><li>Open http://localhost:8080 — NGINX should forward requests to the backend containers.</li></ul><h3>Step 5: Run Everything with Docker Compose (Optional)</h3><p>The repo also has a docker-compose.yaml:</p><pre>docker-compose up --build</pre><ul><li>This will spin up <strong>3 backend containers</strong> (ports 3001–3003).</li><li>NGINX on 8080 handles all traffic automatically.</li><li>Only NGINX port is exposed to the host; backend containers are hidden.</li></ul><h3>Step 6: Setting Up HTTPS (SSL) with NGINX</h3><p>So far, we have NGINX forwarding HTTP traffic on <strong>8080</strong> to the backend containers. Now let’s add <strong>HTTPS</strong> for encrypted traffic.</p><blockquote><em>⚠️ This is for </em><strong><em>learning only</em></strong><em>. Self-signed certificates should </em><strong><em>not</em></strong><em> be used in production.</em></blockquote><h3>6.1 Generate a Self-Signed Certificate</h3><p>On Linux or Windows (via Git Bash or WSL), run:</p><pre># Create a directory to store certificates<br>mkdir nginx-certs<br>cd nginx-certs</pre><pre># Generate private key and self-signed certificate<br>openssl req -x509 -nodes -days 365 -newkey rsa:2048 \<br>  -keyout nginx.key \<br>  -out nginx.crt</pre><ul><li><strong>nginx.key</strong> → private key</li><li><strong>nginx.crt</strong> → self-signed certificate</li><li>Follow the prompts (Country, State, CN, etc.) — for learning, values don’t need to be real.</li></ul><h3>6.2 Modify NGINX Configuration</h3><p>Using your nginx.conf.ssl.template as a reference:</p><pre># HTTPS Server Block<br>server {<br>    listen 443 ssl;<br>    server_name localhost;</pre><pre>    ssl_certificate /path/to/ssl/nginx.crt;<br>    ssl_certificate_key /path/to/ssl/nginx.key;</pre><pre>    location / {<br>        proxy_pass <a href="http://nodejs_cluster;/">http://nodejs_cluster;</a><br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>    }<br>}</pre><pre># HTTP Server Block (Redirect to HTTPS)<br>server {<br>    listen 8080;<br>    server_name localhost;</pre><pre>    location / {<br>        return 301 <a href="https://$host$request_uri;/">https://$host$request_uri;</a><br>    }<br>}</pre><p><strong>What this does:</strong></p><ul><li><strong>443 server block</strong> → handles HTTPS requests using the self-signed certificate.</li><li><strong>8080 server block</strong> → automatically redirects all HTTP traffic to HTTPS.</li><li><strong>proxy_pass</strong> → still forwards requests to the same backend containers (3001–3003).</li></ul><h3>6.3 Restart NGINX</h3><p>After modifying your configuration:</p><pre># Reload NGINX<br>sudo systemctl restart nginx   # Linux<br>nginx -s reload               # Windows</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*-kVdtPmTwjvgzepfaoMfLQ.png" /><figcaption>What does each port do now?</figcaption></figure><h3>6.4 Test HTTPS</h3><ul><li>Open your browser: <a href="https://localhost/">https://localhost</a></li><li>Browser may warn about the self-signed certificate — this is expected for learning.</li><li>NGINX forwards requests to your Node.js containers as before.</li><li>Open http://localhost:8080 → it should automatically redirect to HTTPS.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XfvN4mUfsDJAonbx4k-D1Q.png" /><figcaption>How NGINX handling proxy and load balancing in our example</figcaption></figure><h3>Conclusion</h3><p>By following this tutorial, you now understand what NGINX is and how it works. You learned how it handles HTTP requests using an event-driven model, how worker processes and connections manage traffic, and how directives and contexts structure the configuration. You also saw how server blocks and location blocks control request handling, and how NGINX forwards requests to backend servers using <strong>proxy_pass</strong> and headers.</p><p>You also learned that NGINX can work as a reverse proxy, load balancer, cache, and HTTPS endpoint. With this knowledge, you can serve static files, proxy API requests, manage multiple backend servers, redirect HTTP to HTTPS, and use NGINX with containerized or cloud-native apps.</p><p>In short, NGINX is a tool for efficiently managing web traffic, whether for a single website or multiple backend services.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81bb0a8c91ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Model Context Protocol (MCP): An Overview of How LLMs Connect to Real-World Systems]]></title>
            <link>https://medium.com/@yashwantsaste/model-context-protocol-mcp-an-overview-of-how-llms-connect-to-real-world-systems-83380cad6e82?source=rss-384f60710a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/83380cad6e82</guid>
            <category><![CDATA[mcp-protocol]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Yashwant Sanjay Saste]]></dc:creator>
            <pubDate>Fri, 14 Nov 2025 17:19:34 GMT</pubDate>
            <atom:updated>2025-11-14T17:19:34.303Z</atom:updated>
            <content:encoded><![CDATA[<h3>An Introduction to the Model Context Protocol (MCP)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oqffKmRHKwutJr510x1VQA.png" /></figure><h3><strong>What is MCP?</strong></h3><p><strong>Model Context Protocol (MCP)</strong> is an <strong>open-source standard by Anthropic</strong> that allows <strong>Large Language Models (LLMs)</strong> to securely connect with external <strong>applications, databases, and tools</strong>.</p><blockquote>“Just as REST provides a standard protocol for communication between websites and applications, the Model Context Protocol (MCP) aims to standardize communication between large language models (LLMs) and external tools or applications.”</blockquote><h3><strong>Why MCP Was Needed?</strong></h3><p>Large Language Models (LLMs) like Claude, GPT, or Gemini are extremely capable at understanding and generating natural language. However, in their traditional form, they operate as <strong>self-contained systems</strong> — isolated from the real world.</p><p>They process text in a simple pattern:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*UAoDj03yI61fSmUcR8lmjQ.png" /><figcaption>Traditional LLM model workflow</figcaption></figure><p>Traditional LLMs operate in isolation and cannot directly:</p><ul><li>Access real-time data such as PostgreSQL logs or system metrics.</li><li>Read or modify local file systems for automation tasks.</li><li>Query internal company APIs for updated business information.</li><li>Retrieve live web content or execute code securely.</li></ul><p>They rely only on pre-training data, which is static and quickly becomes outdated. Once deployed, the model remains frozen unless retrained or manually updated.</p><h3>The Context Gap &amp; How MCP Solves This</h3><p>When asked something like <em>“Summarize today’s database errors,”</em> a traditional LLM can only speculate or provide generic advice.<br> It lacks a direct connection to live systems, preventing it from using current, relevant data.<br> This disconnect between the model’s reasoning and real-time context limits its usefulness in production environments.</p><p>The Model Context Protocol (MCP) addresses these limitations by introducing a standardized and secure bridge between LLMs and external systems.</p><p>It enables an LLM to:</p><ul><li>Request tools, data, or context dynamically from the host application.</li><li>Communicate with databases, APIs, or file systems through MCP servers.</li><li>Operate securely and modularly without hard-coded integrations.</li></ul><p>MCP effectively converts an isolated model into a connected, context-aware agent capable of performing real-world tasks safely and intelligently.</p><h3>MCP Architecture Explained</h3><p>The <strong>Model Context Protocol (MCP)</strong> introduces a structured and secure way for Large Language Models (LLMs) to communicate with the outside world. Instead of directly connecting to various applications or databases, an LLM interacts through a layered architecture. Each layer handles a specific responsibility — ensuring modularity, security, and scalability in how the model accesses external systems.</p><h3>1. AI System (LLM) — The Reasoning Engine</h3><p>At the top of this architecture lies the <strong>LLM</strong>, which serves as the system’s reasoning and decision-making engine.<br>While it can understand and generate human-like text, it does not natively have access to external data or tools.<br>When the model identifies the need for additional information — such as reading a file, running a query, or calling an API — it sends a structured <strong>request</strong> to the next layer, the Host Application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HfqaJo-uoMeiEjRp7F3xng.png" /></figure><h3>2. Host Application — The Coordinator</h3><p>The <strong>Host Application</strong> serves as the bridge between the LLM and the real world.It receives requests from the model and decides which client should handle them.<br>The host can be any environment embedding an LLM — such as Claude Desktop, a web IDE, or a custom AI platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*b4CpBjk7DjHGsFmacyEIKQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L-VFXSv89Ga7iWYVE6esFA.png" /></figure><h3>3. MCP Servers — The Resource Layer</h3><p><strong>MCP Servers</strong> are managed by service providers or organizations and represent the actual systems holding or processing data.<br> They expose well-defined endpoints that follow the MCP standard, allowing any compliant client or host to communicate securely.<br> This ensures interoperability, consistency, and fine-grained permission control.</p><p>MCP servers <strong>can be</strong>:</p><ul><li>cloud services</li><li>internal company APIs</li><li>local developer tools</li><li>local filesystem utilities</li><li>personal scripts running on a machine</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8n3Vh_W4EbMrv0fTeR_Ajg.png" /></figure><p>This architecture defines a clear chain of responsibility:</p><blockquote><strong><em>LLM requests → Host coordinates → Clients communicate → Servers execute</em></strong></blockquote><p>Through this pipeline, the LLM gains structured, secure, and scalable access to real-world systems — allowing it to act as a truly intelligent, context-aware agent rather than a static text generator.</p><h3>How MCP Enables Multi-Tool Chaining</h3><p>Traditional LLMs can only reason and generate text. They cannot interact with real-world systems such as databases, APIs, or file systems. MCP changes this by allowing an LLM to call external tools one after another, forming a structured chain of actions. This approach — called <strong>multi-tool chaining</strong> — enables the model to complete complex, real tasks that require multiple steps and multiple resources.</p><p>The process begins when the LLM receives a user query and generates an internal plan. Instead of responding directly, the model identifies what tools are required to solve the problem. For instance, a request like <em>“Analyze today’s PostgreSQL errors”</em> may require one tool to fetch logs, another to parse data, and a third to analyze patterns. The LLM expresses these steps as MCP tool requests, essentially outlining the workflow it intends to follow.</p><p>Once the LLM knows which tools are needed, the MCP Host takes over execution. The host is responsible for selecting the correct tool, running it, and collecting its output. This ensures that each tool runs securely, in the right order, and with the right permissions. The LLM does not execute anything directly; instead, it operates through the host, which acts as a controlled gateway to the external environment.</p><p>After each tool finishes its task, the result is sent back to the LLM. The model then reads the output, reasons over it, and determines the next tool to call. This create a loop where the LLM continually updates its understanding as fresh data arrives. Each tool’s output becomes context for the next step, allowing the model to adjust its plan dynamically and intelligently.</p><p>When all required tools have been used, the LLM produces the final answer — one that is grounded in real, up-to-date information. The output is no longer a guess based on training data but a carefully reasoned conclusion based on live results from multiple sources. This is what makes MCP especially powerful for real-world automation, analytics, debugging, DevOps, and multi-system workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jso1I6gYHyT_nX1mdUL0mA.png" /></figure><h3>Conclusion</h3><p>The Model Context Protocol makes it much easier for Large Language Models to work with real-world data and systems. Instead of staying isolated and relying only on their training, MCP gives them a safe and standard way to connect to tools, databases, files, and APIs. This allows an LLM to do real tasks — not just generate text — by using live information and running actions through tools. With MCP, models can fetch data, analyze logs, automate steps, and work through multi-tool workflows just like a real assistant. This makes LLMs far more useful in everyday software work, automation, and real-time applications, opening the door to smarter and more practical AI systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83380cad6e82" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forward and Reverse Proxies Explained]]></title>
            <link>https://medium.com/@yashwantsaste/forward-and-reverse-proxies-explained-04bab1cd433e?source=rss-384f60710a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/04bab1cd433e</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[cloud-networking]]></category>
            <category><![CDATA[network-security]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[proxy-server]]></category>
            <dc:creator><![CDATA[Yashwant Sanjay Saste]]></dc:creator>
            <pubDate>Sun, 28 Sep 2025 19:50:53 GMT</pubDate>
            <atom:updated>2025-09-28T19:50:53.842Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rco7oYYtsszCayk2ZWbDzQ.png" /></figure><p>Proxies are like middlemen on the internet. Instead of a client talking directly to a server, a proxy sits in between and helps with things like security, speed, and routing.</p><p>Forward proxies hide the client, reverse proxies hide the server, and database proxies simplify how apps talk to databases. In this blog, we’ll break down what each of these does, why they are used, and where you’ll see them in real life.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EDnm1_09oPjo48raUvW9vw.png" /><figcaption>What is Proxy?</figcaption></figure><h3>What is a Proxy?</h3><p>A proxy is a server (or a set of servers) that sits between two systems. Instead of connecting directly, the client and server talk through the proxy. This adds a layer of control for things like <strong>security, caching, monitoring, and routing</strong></p><h4>Types of Proxies (High-Level)</h4><ol><li><strong>Forward Proxy</strong> — Sits in front of clients and hides their identity from the internet.</li><li><strong>Reverse Proxy</strong> — Sits in front of servers and hides backend infrastructure from clients.</li><li><strong>Database Proxy</strong> — Manages how applications connect to databases, handling pooling, caching, and routing queries.</li></ol><h3>A. Forward Proxy</h3><p>A forward proxy sits <strong>between the client and the internet</strong>. The client sends requests to the proxy, and the proxy forwards them to the destination server. The server only sees the proxy’s IP, not the client’s.</p><h4>Why It’s Used</h4><ul><li><strong>Hides client identity</strong> → adds privacy and anonymity.</li><li><strong>Centralized control</strong> → admins can enforce rules and filters.</li><li><strong>Caching</strong> → speeds up access to frequently used content.</li></ul><h4>Common Use Cases</h4><ul><li><strong>Security &amp; Anonymity</strong>: Masking user IPs when browsing.</li><li><strong>Policy Enforcement</strong>: Blocking certain sites in schools or companies.</li><li><strong>Caching Content</strong>: Storing common resources (like documentation) to save bandwidth.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EfE7tljWbQhBfudnErcfYg.png" /><figcaption>Forward Proxy</figcaption></figure><h3>B. Reverse Proxy</h3><p>A reverse proxy sits <strong>between clients and backend servers</strong>. Clients send requests to the proxy, which then forwards them to the appropriate server. The client only sees a single endpoint and doesn’t know about the backend servers.</p><h4>Why It’s Used</h4><ul><li><strong>Load Balancing</strong> → distributes traffic evenly across multiple servers.</li><li><strong>Routing</strong> → directs requests to different services based on URL or rules.</li><li><strong>Caching</strong> → stores popular responses to reduce server load and speed up delivery.</li><li><strong>Infrastructure Abstraction</strong> → hides backend complexity, making scaling easier.</li></ul><h4>Common Use Cases</h4><ul><li><strong>Load Balancer</strong>: Tools like NGINX or HAProxy manage traffic across servers.</li><li><strong>API Gateway</strong>: Routes /auth/* requests to the Auth service, /payments/* to Payments service.</li><li><strong>Static Content Caching</strong>: Popular pages served quickly from the proxy cache.</li><li><strong>Backend Abstraction</strong>: Clients don’t need to know how many servers are running.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*5-Po8wRFlFj_7Q8q7OaNyQ.png" /><figcaption>Reverse Proxy</figcaption></figure><h3>C. Database Proxy</h3><p>A database proxy acts like a <strong>reverse proxy for databases</strong>. Applications send queries to the proxy, which then forwards them to the correct database server. This adds a layer of <strong>control, caching, and connection management</strong>.</p><h4>Why It’s Used</h4><ul><li><strong>Query Caching</strong> → stores results of frequent queries to reduce load on the database.</li><li><strong>Connection Pooling</strong> → manages many client connections with a limited number of database connections.</li><li><strong>Topology Abstraction</strong> → hides database structure (shards, replicas) from the client.</li></ul><h4>Common Use Cases</h4><ul><li><strong>Scaling Applications</strong>: Handles thousands of app connections efficiently.</li><li><strong>Simplifying Database Access</strong>: Clients don’t need to know which server holds which data.</li><li><strong>Improving Performance</strong>: Reduces latency for repeated queries.</li></ul><p><strong>Popular Tool:</strong> ProxySQL</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*lSGKZUPevz8VTpCtBICzaw.png" /><figcaption>Database Proxy</figcaption></figure><h3><strong>Proxy Comparisons: VPN, Firewall, and Load Balancer</strong></h3><h4>A. Proxy V/S VPN</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zzX8njOXkQbw-iD0aBLo4Q.png" /><figcaption>Difference between Proxy and VPN</figcaption></figure><h4># When to use Proxy</h4><p>Choose a proxy when you need lightweight, fast routing for specific applications or want to bypass simple restrictions without full device encryption.</p><h4># When to use VPN</h4><p>Choose a VPN when you need comprehensive security, complete privacy protection, or want to secure all traffic from your entire device or network.</p><h4>B. Proxy vs Firewall</h4><p>Proxies act as intelligent intermediaries that can understand and modify application data, while firewalls act as security barriers that filter traffic based on predetermined rules at the network level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CmteKwNaiu5aHmtdjdYpAQ.png" /><figcaption>Difference between Proxy and Firewall</figcaption></figure><h4># When to use Proxy</h4><p>Choose a proxy when you need to manage application-level traffic, cache content, provide anonymity, or route requests between clients and servers intelligently.</p><h4># When to use Firewall</h4><p>Choose a firewall when you need to secure network perimeters, block unauthorized access, prevent intrusions, and enforce security policies at the network level.</p><p>C. Proxy V/s Load Balancer</p><p>Proxies are versatile intermediaries focused on security, privacy, and content management, while load balancers are specialized traffic distributors focused purely on performance, availability, and server optimization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lrkrkVH8zTfY318pHYz81g.png" /><figcaption>Difference between Proxy and Load Balancer</figcaption></figure><h4># When to use Proxy</h4><p>Choose a proxy when you need security, privacy, content filtering, caching, or want to act as an intermediary for various purposes between clients and servers.</p><h4># When to use Load Balancer</h4><p>Choose a load balancer when you need to distribute traffic across multiple servers, ensure high availability, handle scaling, and optimize server performance.</p><blockquote>Modern reverse proxies can include load balancing features, and some load balancers can perform proxy functions. The distinction often depends on the primary use case and configuration.</blockquote><h3>Conclusion</h3><p>Proxies are versatile tools that help with security, performance, and traffic management. While they share some functions with VPNs, firewalls, and load balancers, each serves a distinct purpose in a network. Understanding these differences helps in designing systems that are secure, efficient, and scalable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04bab1cd433e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>