<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manthan Gupta on Medium]]></title>
        <description><![CDATA[Stories by Manthan Gupta on Medium]]></description>
        <link>https://medium.com/@manthangupta109?source=rss-602e6ade635c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-od6rrpuphaZJ_431AxKkQ.jpeg</url>
            <title>Stories by Manthan Gupta on Medium</title>
            <link>https://medium.com/@manthangupta109?source=rss-602e6ade635c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:06:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@manthangupta109/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Replication in Distributed Systems — Part 3]]></title>
            <link>https://medium.com/@manthangupta109/replication-in-distributed-systems-part-3-ca013a395feb?source=rss-602e6ade635c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca013a395feb</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Manthan Gupta]]></dc:creator>
            <pubDate>Tue, 13 Feb 2024 15:47:22 GMT</pubDate>
            <atom:updated>2024-02-15T08:46:46.538Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D0i4A3yxHpkn6_SgDucO_Q.png" /></figure><h3>Replication in Distributed Systems — Part 3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*LTbxt5gLaX8HxTGC.gif" /></figure><p>Welcome to the third and final part of my series on replication in distributed systems. If you have missed the first two parts of the series, I would advise you to read them first, as this comes off as a storyline. Click <a href="https://manthanguptaa.in/posts/replication_in_distributed_systems_part_1/">here</a> and <a href="https://manthanguptaa.in/posts/replication_in_distributed_systems_part_2/">here</a> for the previous blogs. In this blog, we will discuss… well, let’s dive right in without formalities!</p><h3>Leaderless Replication</h3><p>In a leaderless implementation, the client directly sends the writes to several replicas and waits for the majority of them to acknowledge, compared to a setup with one or more leaders where a leader exists to coordinate the writes.</p><h4>What happens when a node goes down?</h4><p>In a leader-based configuration with one leader and two replicas, we need to perform <em>failover</em> if we want to continue to process writes. But in a leaderless configuration, failover doesn’t exist. So when the unavailable node comes back online, it has missing writes, meaning it has <em>stale</em> data.</p><p>This problem can easily be solved by sending the read request from the client to not just one replica but to several nodes in parallel. The client may receive different versions of the data that may be up-to-date or stale. Version numbers are used to determine which value is newer.</p><p>When the unavailable node comes back online, it has stale data. There are two mechanisms to catch up on the writes</p><ol><li><strong>Read Repair</strong> — When a client makes a read request from several nodes in parallel, it can detect the node/s having stale data. The client sees the node which has stale data and writes back the newer value. This approach works well for values frequently read.</li><li><strong>Anti-Entropy</strong> — A background process runs constantly to look for differences in the data between the replicas. This approach doesn’t copy writes in the same order they are written to the up-to-date replica</li></ol><h3>Quorums Consistency</h3><p>Quorums aren’t necessarily the majority — the only thing that matters is that the sets of the nodes used by the read and write operations overlap in at least one node. If there are <em>n</em> replicas, every write must be confirmed by <em>w</em> nodes to be considered successful, and we must query at least <em>r</em> nodes for each read. As long as <strong>w + r &gt; n</strong>, we expect to get up-to-date values.</p><p>Reads and writes that obey these r and w values are called <em>quorum reads and writes</em>. The quorum condition <strong>w + r &gt; n</strong> allows the system to tolerate unavailable nodes</p><ul><li>if w &lt; n, we can still process writes if a node is unavailable</li><li>if r &lt; n, we can still process reads if a node is unavailable</li></ul><p>But even with quorum consistency, there are edge cases where stale values returned</p><ul><li>If a <em>sloppy quorum</em> is used, the w writes can end up on different nodes than the r reads, so there is no longer a guaranteed overlap between the r and w nodes.</li><li>If two writes occur concurrently, it’s not clear which one happened first. The only safe solution is to merge the concurrent writes. If a winner is picked based on the timestamp (LWW — last write wins), writes can be lost due to clock skew.</li><li>If a write happens concurrently with a read, the write may be reflected on only some of the replicas. It’s undetermined whether the reads return the old or new value.</li><li>If the write succeeds on fewer than w nodes and fails on the remaining ones, it is not rolled back on the replicas where it succeeded. It means that if a write is reported as failed, subsequent reads may or may not return the value from that write.</li></ul><h4>Sloppy Quorums &amp; Hinted Handoff</h4><p>In large clusters with significantly large n nodes, the client can likely connect to some database nodes during the network interruption, just not the nodes that needed to assemble a quorum for a particular write or read. Is it better to return errors to all requests when we can’t reach a quorum, or should we accept the writes anyway and write them to some nodes that are reachable, but aren’t among the n nodes on which the value lives?</p><p>This is called <em>sloppy quorums</em>. Writes and reads still require w and r successful responses, but those may include nodes that aren’t among the designated n “home” nodes for a value. Once the network interruption is fixed, any writes that one node temporarily accepted on behalf of another node are sent to the appropriate “home” nodes. It is called <em>hinted handoff</em>.</p><p>Sloppy quorums are particularly useful for increasing write availability. As long as any w nodes are available, the database can accept the writes. However, this means that even when w + r &gt; n, you cannot be sure to read the latest value for a key because the latest value may have been temporarily written to some nodes outside of n.</p><h3>Last Write Wins (LWW)</h3><p>Leaderless replication is designed to tolerate conflicting concurrent writes. One approach to achieving eventual convergence is to declare that each replica needs only to store the most “recent” value and allow “older” values to be overwritten. We can attach a timestamp to each write, pick the biggest timestamp as the most “recent” value, and discard any writes with an earlier timestamp. This conflict resolution algorithm is called <em>last write wins</em>.</p><p>LWW achieves the goal of eventual convergence but at the cost of durability. If there are several concurrent writes to the same key, even if they were all reported to be successful by the quorum to the client, only one of the writes will be able to survive, and the others will be silently discarded. It may cause LWW to drop writes that aren’t concurrent. Caching is one use case where lost writes are acceptable, and LWW is a good choice. But if losing data isn’t acceptable, then LWW is a poor choice for conflict resolution.</p><h3>Defining Concurrency &amp; “Happens-Before” Relationship</h3><p>An operation A happens before another operation B if B knows about A, B depends on A, or B builds upon A. The key to defining concurrency is to know whether one operation happens before another operation. In concurrency, the timestamp doesn’t matter. We call two operations concurrent if they are unaware of each other, regardless of the physical time at which they occurred.</p><h4>Capturing the Happens-Before Relationship</h4><p>The server can determine whether two operations are concurrent by looking at the version numbers.</p><ul><li>The server maintains the version number for every key and increments the version number every time that key is written. The server stores the new version along with the value written.</li><li>When a client reads a key, the server returns all the values that haven’t been overwritten with the latest version number. A client must read the key before writing.</li><li>When a client writes a key, it must include the version number from the prior read and must merge all the values it received in the prior read.</li><li>When the server receives a write with a particular version number, it can overwrite all values with that version number or below, but it must keep all values with a higher version number.</li></ul><h3>Version Vectors</h3><p>Using a single version number to capture dependencies between operations isn’t sufficient when multiple replicas are accepting writes concurrently. Instead, we need to use a version number per replica per key. Each replica increments its own version number when processing a write and keeps track of the version numbers it has seen from each of the other replicas. This information indicates which values to overwrite and which values to keep as siblings (concurrent values are called siblings). The collection of version numbers from all replicas is called <em>version vector</em>.</p><p>Version vectors are sent from the database replicas to clients when values are read and need to be sent back to the database when a value is subsequently written. It helps the database to distinguish between overwrites and concurrent writes.</p><h3>Parting Words</h3><p>The third part and final part of the blog series introduced quorum consistency for leaderless replication, the LWW concurrency model, and version vectors. This rounds up my blog series on replication in distributed systems.</p><p>If you liked the blog, don’t forget to share it and follow me on <a href="https://twitter.com/manthanguptaa">Twitter</a>, <a href="https://www.linkedin.com/in/manthangupta109/">LinkedIn</a>, <a href="https://peerlist.io/manthangupta">Peerlist</a>, and other social platforms, and tag me. You may also write to me at <a href="/service/mailto:guptaamanthan01@gmail.com">guptaamanthan01@gmail.com</a> to share your thoughts about the blog.</p><p><em>Originally published at </em><a href="https://manthanguptaa.in/posts/replication_in_distributed_systems_part_3/"><em>https://manthanguptaa.in</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca013a395feb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Replication in Distributed Systems — Part 2]]></title>
            <link>https://medium.com/@manthangupta109/replication-in-distributed-systems-part-2-67d44cc08e63?source=rss-602e6ade635c------2</link>
            <guid isPermaLink="false">https://medium.com/p/67d44cc08e63</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Manthan Gupta]]></dc:creator>
            <pubDate>Thu, 01 Feb 2024 11:05:53 GMT</pubDate>
            <atom:updated>2024-02-01T11:18:38.720Z</atom:updated>
            <content:encoded><![CDATA[<h3>Replication in Distributed Systems — Part 2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*wCs2YQKa_dkf2KkT.gif" /></figure><p>In the <a href="https://manthanguptaa.in/posts/replication_in_distributed_systems_part_1/">first part</a> of the series, we laid the foundation of replication in distributed systems. We will take this forward in the 2nd part, where we introduce the different consistency guarantees, multi-leader configuration and its comparison with single-leader configuration, conflict resolution, and more.</p><h3>Reading your own writes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DeHVTwGfSYBxP7oA.png" /></figure><p>Many applications allow the user to submit some data and then view what they have submitted, for example, comment in a discussion, buy a stock, etc. The submitted data is sent to the leader and read by the follower, but if we have asynchronous replication, it is a problem. The new data may have not reached the replica. It seems like the data is lost.</p><p>In this situation, we need <em>read-after-write consistency</em>, also known as <em>read-your-writes consistency</em>. It guarantees that if the user reloads the page, they will always see the updates that they have submitted. It doesn’t hold for other user’s updates, and it will take some time to reflect them.</p><p>But how do we implement it? The initial thought process should be to read from the leader the data that the user may have modified. For example, user info can only be edited by the user itself, so a simple rule is that the user profile is read from the leader and any other user’s profile from a follower. But if most things are editable by the user, then the benefit of read scaling will be negated. In this case, track the time of the last update and serve the reads from the leader after a minute of the last update.</p><p>Another complication arises if the user is using multiple devices to access the service. Let’s say the user comments under an Instagram photo and checks the comment from another device. In this case, we need <em>cross-device read-after-write consistency</em>. Approaches that require remembering the timestamp of the last update need to be centralized. Another issue is that the request might not be routed to the same data center for multiple devices, so there might be read inconsistencies. If your approach requires reading from the leader, we may first need to route requests from all of a user’s devices to the same data center.</p><h3>Monotonic Reads</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R7Z9T6TvGaPkEjul.png" /></figure><p>If the user makes multiple reads from different replicas as the user refreshes the web page, each request is served from a random server. The user sees time moving backward because the data hasn’t persisted on all the replicas.</p><p><em>Monotonic reads</em> is a guarantee that this kind of anomaly doesn’t happen. It’s a lesser consistency than strong consistency but a stronger guarantee than eventual consistency. One way of achieving this is by serving all the reads for a user from the same replica chosen by the hash of the userID.</p><h3>Consistent Prefix Reads</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QBTLViYzUbHfmwUsvSNA0A.png" /></figure><p>Let’s suppose you are in a WhatsApp group of 3 people, and all 3 of you are exchanging messages. Now you see the sequence of messages not making any sense. Your friend is replying to messages that haven’t arrived, and it feels like he can see the future.</p><p><em>Consistent prefix reads</em> guarantees that if a sequence of writes happens in a particular order, anyone reading those writes will see them appear in the same order. One solution to this problem is to make sure that any writes that are causally related to each other are stored in the same partition.</p><h3>Multi-Leader Replication</h3><p>Leader based replication has one major downside, it has only one leader. If we aren’t able to connect to the leader for any reason, we can’t write to the database. The obvious answer to this problem is to have more than one node that accepts writes. This is called <em>multi-leader</em> configuration, also known as <em>master-master</em> or <em>active/active replication</em></p><h3>Single-Leader v/s Multi-Leader Configuration in Multi-Data Center Deployment</h3><h4>Performance</h4><p>In a single-leader configuration, all writes go to the data center with the leader, which adds significant latency to writes, compared to a multi-leader configuration where every write is processed in the local data center and is replicated asynchronously to the other data center. The inter-data center network delay is hidden from the user, which means the perceived performance may be better.</p><h4>Tolerance of Data Center Outages</h4><p>In a single-leader configuration, if the data center with the leader fails, failover can promote a follower to the leader in another data center. In a multi-leader configuration, it continues to operate independently of the others, and replication catches up when the failed data center comes back online.</p><h4>Tolerance of Network Problems</h4><p>A single-leader configuration is very sensitive to problems in this inter-data center link because writes are made asynchronously over this link. A multi-leader configuration with asynchronous configuration can tolerate network problems better as it doesn’t prevent it from processing writes.</p><p>A big downside of multi-leader configuration is that the same data can be modified in 2 different places, causing <em>write conflicts</em>. Auto-incrementing keys, triggers, and integrity constraints can be problematic.</p><h3>Handling Write-Conflicts</h3><h4>Synchronous v/s Asynchronous Conflict Detection</h4><p>In a single-leader database, the 2nd writer is either blocked or queued, waiting for the first write to complete or aborted, forcing the user to retry the write. In a multi-leader setup, both the writes are accepted, and the conflict is only detected asynchronously at some later point in time. We could’ve made the conflict detection synchronous in a multi-leader setup i.e. wait for the first write to be replicated to all the replicas before telling the user the write was successful. But this would make us lose on the benefit of multi-leader replication to allow each replica to accept writes independently. Use single-leader replication if you want synchronous conflict detection.</p><h4>Converging Towards a Consistent State</h4><p>A single-leader database applies the writes in sequential order, meaning that if there are multiple updates to the same field, the last write determines the final value of the field. In multi-leader configuration, there is no defined ordering of writes, so it’s not clear what the final value of the field will be.</p><p>Various ways of achieving <em>convergent conflict resolution</em></p><ul><li>Give each write a unique ID, pick the write with the highest ID as the winner, and throw all the other writes. If a timestamp is used, the technique is called <em>last write wins (LWW)</em>. But this is prone to data loss.</li><li>Give each replica a unique ID, and let the writes that originated at a higher-numbered replica always take precedence over writes that originated at a lower-numbered replica. But this is also prone to data loss.</li><li>Record the conflict in an explicit data structure that preserves all information, and write application code that resolves the conflict.</li></ul><h4>Custom Conflict Resolution Logic</h4><p>Most multi-leader replication tools let us write conflict resolution logic using application code that may execute on read or on write.</p><ul><li><em>on write</em> — As soon as the system detects a conflict in the log of replicated changes, it calls the conflict handler.</li><li><em>on read</em> — When a conflict is detected, all the conflicting writes are stored. The next time data is read, multiple versions of the data are sent back to the application. The application may prompt the user to manually resolve the conflict.</li></ul><p>Conflict resolution usually applies on a row or document level and not to a complete transaction.</p><h4>Automatic Conflict Resolution</h4><ul><li><em>Conflict-free replicated datatypes (CRDTs)</em> are a family of data structures for sets, maps, ordered lists, counters, etc that can be concurrently edited by multiple users and which automatically sensibly resolve conflicts. It uses 2-way merge function</li><li><em>Mergeable persistent data structure</em> tracks history explicitly, similarly to the git version control, and uses a 3-way merge function</li><li><em>Operational transformation</em> is the conflict resolution algorithm behind collaborative editing applications such as Google Docs. It is designed particularly for concurrent editing of an ordered list of items.</li></ul><h3>Parting Words</h3><p>The second part of the blog series introduced the different consistency guarantees other than strong consistency and eventual consistency. It also gave an in-depth introduction to multi-leader configuration, where we compared it to single-leader configuration.</p><p>If you liked the blog, don’t forget to share it and follow me on <a href="https://twitter.com/manthanguptaa">Twitter</a>, <a href="https://www.linkedin.com/in/manthangupta109/">LinkedIn</a>, <a href="https://peerlist.io/manthangupta">Peerlist</a>, and other social platforms, and tag me. You may also write to me at <a href="/service/mailto:guptaamanthan01@gmail.com">guptaamanthan01@gmail.com</a> to share your thoughts about the blog.</p><p><em>Originally published at </em><a href="https://manthanguptaa.in/posts/replication_in_distributed_systems_part_2/"><em>https://manthanguptaa.in</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67d44cc08e63" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Replication in Distributed Systems — Part 1]]></title>
            <link>https://medium.com/@manthangupta109/replication-in-distributed-systems-part-1-69b8fe477a4a?source=rss-602e6ade635c------2</link>
            <guid isPermaLink="false">https://medium.com/p/69b8fe477a4a</guid>
            <category><![CDATA[replication]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Manthan Gupta]]></dc:creator>
            <pubDate>Mon, 29 Jan 2024 16:14:29 GMT</pubDate>
            <atom:updated>2024-01-29T16:14:29.788Z</atom:updated>
            <content:encoded><![CDATA[<h3>Replication in Distributed Systems — Part 1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*NxMOW3OUCumbY9Vo.gif" /></figure><p>Welcome, fellow nerds, to the 1st part of a blog series on replication. We will be discussing why we even need to distribute a database across multiple machines, what are leaders and followers, how to handle the failure of leaders and followers, etc. It will set it up nicely for our future blog in this series.</p><h3>Why distribute a database across multiple machines?</h3><ol><li><strong>Scalability</strong> — If your data volume, read load, or write load grows larger than what a single machine can handle, you can spread the load across multiple machines</li><li><strong>High availability</strong> — If your application needs to continue to serve even if one or more machines goes down, you can use multiple machines to give you redundancy/ fault tolerance.</li><li><strong>Latency</strong> — If you have users over the globe, you will want to have servers at various locations worldwide so that each user is served from a data center that is geographically closer to them.</li></ol><p>There are 2 common ways to distribute data across multiple machines (or nodes)</p><ol><li><strong>Replication</strong> — Keeping a copy of the same data on several different nodes, potentially in different locations. Replication provides redundancy.</li><li><strong>Partition</strong> — Splitting the database into smaller subsets called <em>partitions</em> so that different partitions can be assigned to different nodes (also known as <em>sharding</em>)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J_uRUX6_13Y1q0P4.png" /></figure><h3>Leaders &amp; Followers</h3><p>Every node that has a copy of the database is called a <em>replica</em>. Every write needs to be processed by every replica; otherwise, the replicas would no longer have the same data. One of the replicas is designated as the <em>leader</em>. When a client wants to write to the database, they must send their request to the leader that first writes the new data to its local storage.</p><p>The replicas other than the leader are known as <em>followers</em>. Whenever the leader writes new data to its local storage, it also sends the data change to all of its followers as part of the replication log or change stream in the same order they were processed by the leader. All the followers are read-only, while only the leader accepts the writes. So, when the client wants to read from the database, the followers or the leader can serve the read request.</p><h3>Async vs Sync Replication</h3><p>In synchronous replication, the leader waits until the follower confirms that it received the write before reporting the success to the user and before making the write visible to other clients. In asynchronous replication, the leader sends the message and doesn’t wait for the acknowledgment from the follower.</p><p>The advantage of synchronous replication is that the follower is guaranteed to have an up-to-date copy of the data that is consistent with the leader. If the leader suddenly fails, we can be sure the follower has the data available. The disadvantage is that if the follower doesn’t respond, for some reason, it has crashed, network fault, etc. The write cannot be processed. The leader will have to block all the writes and wait for the acknowledgment from the follower whenever it is available again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vlnE5vDFeO8DCB-G.png" /></figure><p>It is definitely impractical to have all the followers as synchronous. Any one node outage would cause the whole system to grind to a halt. If we have a synchronous replication setting enabled on a database, it usually means that one of the followers is synchronous, and the others are asynchronous. If the synchronous follower becomes unavailable or slow, one of the asynchronous followers is promoted to synchronous. It guarantees we have an up-to-date copy of the data on at least two nodes: the leader and one synchronous follower.</p><p>In asynchronous replication, if the leader fails and isn’t recoverable, any writes that haven’t yet been replicated to the followers are lost. It means the writes aren’t guaranteed to be durable, even if it has been confirmed to the client. However, the advantage of a fully asynchronous configuration is that the leader can continue processing writes even if all the followers have fallen behind. Weak durability may sound like a bad trade-off, but asynchronous replication is widely used, especially if there are many followers or if they are geographically distributed.</p><h3>How to add new Followers?</h3><p>Let’s say we want to add a new replica because we want to scale or replace a failed node. Simply copying data files from one node to another is not sufficient. Clients are constantly writing to the database, and the data is always in flux, so a standard file copy would see different parts of the database at different points in time. We can lock the database, making it unavailable for any writes, but that would go against our goal of high availability.</p><p>Here is how it’s done</p><ul><li>Take a consistent snapshot of the leader’s database at some point in time. If possible, without taking a lock on the database</li><li>Copy the snapshot to the new follower node</li><li>The follower connects to the leader and requests all the data changes that have happened since the snapshot was taken</li><li>When the followers have processed the backlog of data changes since the snapshot, we say it has caught up. It can now continue to process data changes from the leader as they happen</li></ul><h3>Handling Node Outages</h3><h3>Follower Failure</h3><p>Each follower stores a log of the data changes it has received from the leader on its local disk. If a follower crashes and restarts or the network between the leader and the follower is temporarily interrupted. The follower can recover from its log: it knows the last transaction that was processed before the fault occurred. Thus, the follower can connect to the leader and request all the data changes that occurred when the follower was disconnected.</p><h3>Leader Failure</h3><p>What if the leader himself goes down? TLDR; One of the followers is promoted to be the leader. Clients are reconfigured to send all the writes to the new leader, and the followers need to start consuming data from the new leader. This process is called <em>failover</em>.</p><ul><li>There is no foolproof way of detecting what has gone wrong, so most systems use a timeout. Nodes frequently bounce messages back and forth between each other, and if a node doesn’t respond for some time — say 30 seconds, it is assumed to be dead. (This doesn’t apply if the leader is taken down for planned maintenance)</li><li>Choosing a new leader is done through an election where the leader is chosen by a majority of the remaining replicas, or a new replica could be appointed by the previous leader. The best candidate is usually the replica with the most up-to-date data changes from the previous leader.</li><li>Clients are now reconfigured to send their write requests to the new leader. If the old leader comes back, it might still believe it is the leader, not realizing that the other replicas have forced it to step down. The system has to ensure that the older leader becomes a follower and recognizes the new leader.</li></ul><p>The time between when the leader is down and the next leader is getting elected. Writes are blocked or queued in the leaderless period. Consistent reads are served by the replicas if synchronous replication is enabled else there is the possibility that stale and inconsistent data is served because of eventual consistency.</p><h4>What could go wrong with Failover?</h4><p>If asynchronous replication is used, the new leader may not have received all the writes from the old leader before it failed. If the former leader rejoins the cluster after a new leader has been chosen, what should happen to the writes that the new leader hasn’t received? The new leader may receive conflicting writes in the meantime. The most common solution is to discard the unreplicated changes of the old leader. But, discarding the writes is dangerous business as it violates the client’s durability.</p><p>In certain fault situations, two nodes might believe that they are the leader. This situation is called <em>split brain</em>, and it is dangerous. If both the leaders accept writes, and there is no process for resolving conflicts, data is likely to be lost or corrupted. As a safety catch, some systems have mechanisms to shut down one node if two leaders are detected.</p><p>And finally, what should be the ideal timeout before the leader is declared dead? A longer timeout means a longer time to recover in case where the leader fails. However, if the timeout is too short, there could be unnecessary failovers.</p><h3>Implementation of Replication Logs</h3><h3>Statement Based Replication</h3><p>The leader logs every write request that it executes and sends that statement log to its followers. For a relational database, every <em>INSERT, UPDATE, or DELETE</em> statement is forwarded to followers, and each follower parses and executes that SQL statement as it has received from a client. But this approach could break down on statements that call a non-deterministic function such as <em>Now()</em> or <em>RAND()</em> that is likely to generate different values on each replica.</p><h3>Write Ahead Log (WAL)</h3><p>This log is an append-only sequence of bytes containing all writes to the database. We can use the same log to build a replica on another node. The main disadvantage is that the log describes the data on a very low level, which makes it tightly coupled to the storage engine. If the database changes its storage format from one version to another, it is typically not possible to run different versions of the database software on the leader and the followers.</p><h3>Logical Log Replication (row-based)</h3><p>A logical log is decoupled from the storage engine and uses different log formats for the storage engine and replication. For a relational database, it is usually a sequence of records describing writes to database tables at the granularity of a row. As it is decoupled from a storage engine, it can be made backward compatible, allowing the leader and the follower to run different versions of the database software or even different storage engines.</p><h3>Parting Words</h3><p>The first part of the blog series aimed at laying the foundation for future blogs in this series on replication in distributed systems. We still have a lot to discuss on this topic that we will do in the next one.</p><p>If you liked the blog, don’t forget to share it on <a href="https://twitter.com/intent/tweet?text=I%20just%20read%20this%20blog%20%22Replication%20in%20Distributed%20Systems%20-%20Part%201%22%20by%20%40manthanguptaa%20and%20%3Cshare%20positive%20words%20with%20the%20world%3E.%0AHere%20is%20the%20link%20to%20this%20awesome%20blog%3A&amp;url=https%3A%2F%2Fmanthanguptaa.in%2Fposts%2Freplication_in_distributed_systems_part_1%2F%20%20">Twitter</a>, LinkedIn, Peerlist, and other social platforms and tag me. You may also write to me at <a href="/service/mailto:guptaamanthan01@gmail.com">guptaamanthan01@gmail.com</a> to share your thoughts about the blog.</p><p><em>Originally published at </em><a href="https://manthanguptaa.in/posts/replication_in_distributed_systems_part_1/"><em>https://manthanguptaa.in</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69b8fe477a4a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Structures That Power Your Databases]]></title>
            <link>https://medium.com/@manthangupta109/data-structures-that-power-your-databases-753695643f7d?source=rss-602e6ade635c------2</link>
            <guid isPermaLink="false">https://medium.com/p/753695643f7d</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Manthan Gupta]]></dc:creator>
            <pubDate>Mon, 29 Jan 2024 16:00:39 GMT</pubDate>
            <atom:updated>2024-01-29T16:00:39.673Z</atom:updated>
            <content:encoded><![CDATA[<p>Have you ever thought about why databases are so complex internally and why we can’t use a text file to store the data? This blog post aims to go from the most basic database that writes data to text files to a more complex setting where we use LSM-tree and B-tree. We will understand why we need these data structures and more.</p><h3>World’s Simplest Key-Value Store</h3><p>What does the world’s simplest key-value store look like? 2 bash function that writes data to a text file and gets data from the text file.</p><pre>db_set(){ <br>  echo &quot;$1, $2&quot; &gt;&gt; database <br>} <br><br>db_get(){ <br>  grep &quot;^$1,&quot; database | sed -e &quot;s/^$1,//&quot; | tail -n 1 <br>}</pre><p>Let me explain both the functions.</p><ul><li>The db_set() function appends to the end of the file without any logic. That means if you update the value of a key, it will not overwrite the value but create a new key value at the end of the file.</li><li>The db_get() function gets the last occurrence of the key.</li></ul><p>What’s the problem with this implementation?</p><ul><li>Doesn’t deal with concurrency control, reclaiming disk space, handling errors, and partially written records.</li><li>db_get() function has terrible performance as it takes O(n) to lookup for a record.</li></ul><p>To efficiently find the value of a key, we need a different data structure: <strong>an index</strong>. The general idea behind indexes is to keep some additional metadata on the side, which acts as a signpost and helps you locate the data you want. But this comes with a tradeoff as it slows down writes but offers fast reads.</p><h3>Hash Indexes</h3><p>I am not planning to dive deep into the workings of a hash indexes in this particular blog. I plan on covering it in a future blog, but till then, you can refer to <a href="https://samuel-sorial.hashnode.dev/understanding-hash-indexes">this blog</a> to understand more about it.</p><p>Let’s again go back to our key-value store. The strategy to improve the lookup speed is to keep an in-memory hash map where every key is mapped to a byte offset in the file. Whenever a new key is appended to the file, the hash map is also updated to reflect the offset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wOmmY_SPmHFZerpR.png" /></figure><h4>How do we avoid running out of disk space as we are only appending to the file?</h4><p>We break the files into smaller segments and perform <strong>compaction</strong> on each segment. Compaction is throwing away the duplicate keys in the file and keeping only the most recent update for each key. We can perform compaction over multiple segments together at the same time. Segments are never modified, so the merged segment is written to a new file. It is done in the background while we serve read and write requests using the old segment files.</p><p>But there are some things to take care of with the implementation and limitations of the hash table index:</p><ul><li>Hash table must fit in memory because, on disk, it is hard to get the same performance</li><li>We can’t do range queries efficiently</li><li>If the database is restarted, the in-memory hash maps are lost. We rebuild the index by reading the entire segment file from beginning to end. It will take a lot of time. We can have a snapshot of each segment’s hash map on disk to solve this issue</li><li>Database may crash halfway through appending a record that can be resolved by including checksums</li><li>Writes are appended to the file in a strictly sequential order. A common implementation is to have only 1 write thread and multiple read threads</li><li>When merging segments, the <strong>tombstone</strong> tells the merging process to discard any previous values for the deleted key</li></ul><h3>SSTables</h3><p>A simple change to our segment files can help us get rid of maintaining the hash index with all keys. We store the key-value pairs in a sorted-by-key order. We call this format <strong>SSTable</strong></p><h4>What advantages do we gain over the hash index?</h4><p>Merging the segment files gets simple and efficient even if the files are larger than the available memory. The merge sort algorithm is used to create new merged segment files, also sorted by key. We now don’t need to maintain an index of all keys in memory. We build a sparse index where we get the keys between which our key lies. It saves disk space and I/O bandwidth.</p><h4>How is this SSTable constructed and maintained?</h4><p>When writes come in we add it to the in-memory tree <strong>memtable</strong>. When it gets bigger than a threshold, we write all the data from the memtable to the SSTable file. The new SSTable file becomes the most recent segment of the database. While we write to SSTable, writes continue to new memtable instance. When read requests come in we first search for the key in memtable then in the most recent on-disk segment, then in the next older segment and so on. And from time to time we run compaction in the background. We can also keep a seperate log on disk to which every write is immediately appended for durability purposes.</p><h3>LSM Tree</h3><p>Log-structure merge-tree or LSM tree is a clever algorithm based on the above principle of merging and compacting sorted files. LSM is append-only as we only append all the modfications to the SSTable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/0*0lMdsJhmUwP5qUrv.png" /></figure><p>The LSM tree algorithm can be slow when looking up keys that do not exist in the database. We first check the memtable and then all the segments. Storage engines use <strong>Bloom Filters</strong> to optimize the algorithm.</p><p>A bloom filter is a memory-efficient data structure for approximating the contents of a set. It can tell if a key exists in the database and thus save many unnecessary disk reads for keys that don’t exist.</p><h3>B Tree</h3><p>B Tree keeps key-value pairs sorted by keys like SSTable, which allows efficient key lookups and range queries. It breaks down the database into fixed-size blocks or pages, traditionally 4Kb in size, and reads or writes one page at a time. This design corresponds closely to the underlying hardware, as disks are also arranged in fixed-size blocks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*njwIsytDf1S4QtTP.png" /></figure><p>An append-only file where we capture all B tree modifications before applying them to the pages of the tree itself. It is a <strong>write-ahead log</strong> (WAL or redo log) that helps make the database crash-resilient. We use latches (lightweight locks) to protect our B tree when multiple threads try to access the B tree at the same time.</p><h3>Which one is better? LSM Tree vs B Tree</h3><ol><li>B Tree writes every piece of data at least twice: once to the write-ahead log and once to the tree page itself (and sometimes again when the pages split). The log-structure index rewrites data multiple times due to the repeated compaction and merging of SSTable. It results in one write to the database and multitudinous writes to the disk over the course of the database’s lifetime — known as <strong>write amplification</strong>. It is a concern on SSDs as a limited number of overwrites can perform before wearing out.</li><li>LSM trees can be compressed better, resulting in smaller files on disk than B trees.</li><li>B tree storage engine leaves some disk space unused due to fragmentation when the page is split or the row cannot fit in an existing table. LSM trees aren’t page oriented and periodically rewrite SSTables to remove fragmentation.</li><li>The compaction process can sometimes interfere with the performance of ongoing reads and writes. At high throughput, the disk’s finite bandwidth is shared between the initial write and the compaction threads running in the background. The bigger the database gets, the more disk bandwidth is required for compaction.</li><li>In B trees, each key only exists in exactly one place in the index, whereas a log-structured storage engine may have multiple copies of the same key in different segments. It makes B trees attractive in databases that want to offer strong transactional semantics.</li><li>LSM trees are typically able to sustain a higher write throughput than B trees. But B trees are thought to offer faster read throughput.</li></ol><h3>Parting Words</h3><p>There are definitely more data structures that power databases like skiplist, inverted index, R tree, suffix tree, etc. But this blog introduces to the more famous ones and to help understand the underlying working of them.</p><p>If you liked the blog don’t forget to share it on <a href="https://twitter.com/intent/tweet?text=I%20just%20read%20this%20blog%20%22Data%20Structures%20That%20Power%20Your%20Databases%22%20by%20%40manthanguptaa%20and%20%3Cshare%20positive%20words%20with%20the%20world%3E.%0AHere%20is%20the%20link%20to%20this%20awesome%20blog%3A%20&amp;url=https%3A%2F%2Fmanthanguptaa.in%2Fposts%2Fdata_structures_that_power_your_databases%2F">Twitter</a>, LinkedIn, Peerlist, and other social platforms and tag me. You may also write to me on <a href="/service/mailto:guptaamanthan01@gmail.com">guptaamanthan01@gmail.com</a> to share your thoughts about the blog.</p><p><em>Originally published at </em><a href="https://manthanguptaa.in/posts/data_structures_that_power_your_databases/"><em>https://manthanguptaa.in</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=753695643f7d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Django: The web framework for perfectionists with deadlines]]></title>
            <link>https://medium.com/@manthangupta109/django-the-web-framework-for-perfectionists-with-deadlines-894746200767?source=rss-602e6ade635c------2</link>
            <guid isPermaLink="false">https://medium.com/p/894746200767</guid>
            <category><![CDATA[django]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[framework]]></category>
            <dc:creator><![CDATA[Manthan Gupta]]></dc:creator>
            <pubDate>Sun, 01 May 2022 13:16:14 GMT</pubDate>
            <atom:updated>2022-05-01T13:16:14.453Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_tTgBPv_OXLiVm83ZHFXYg.png" /></figure><p>Hey Everyone! Today in this blog, we will cover the basics of Django. This will help someone who is just starting with Django get a primer on the web framework by reading this blog.</p><h3>What is a web framework?</h3><p>Before diving into Django web framework, we need to understand what is a web framework exactly? <br>A simple explanation of a web framework can be a piece of software that makes developers’ work smooth, efficient, and the journey of software development less taxing. A web framework is one such tool used to help in building web-based applications.</p><h3>What is django?</h3><p>Django is also a python based web framework used to create high-performance, secure and maintainable services in python at a rapid speed by removing or automating unnecessary parts.</p><h3>Why use django?</h3><ol><li>It makes the whole development cycle much faster.</li><li>When working with a machine learning algorithm built on top of python, It is easy to combine it with django at the backend.</li><li>Django provides sharing and synchronization, as well as scalability.</li><li>Django offers adequate inbuilt features, so you don’t have to put up with work from scratch.</li><li>The main advantage of using django is that it enables the app to scale smoothly.</li></ol><h3>Architecture</h3><p>Django follows an MVT architecture, also known as Model View Template architecture.</p><p><strong>Model</strong>:<br>The model helps to handle the database and works as a data access layer, which has required fields and constraints for data to be stored.</p><p><strong>View</strong>:<br>The view is used to execute the business logic and interact with the data access layer to carry the data.</p><p><strong>Template</strong>: <br>Template handles the presentation layer and is a way to dynamically generate HTML for your django project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*vMwCHJ0PpU-DEC4sVD5zhw.jpeg" /></figure><p>Thus the web application has all the three required pillars <br>1. Data <br>2. Logic <br>3. Layout<br>These work together to provide us with our desired results.</p><h3>Folder Structure</h3><p>After creating the project, the terminal will itself create some files for you:</p><ol><li>__init__.py</li><li>asgi.py</li><li>settings.py</li><li>urls.py</li><li>wsgi.py</li><li>manage.py</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/237/0*-LBKRA3_YkTfXPWx" /></figure><p>settings.py: One of the most important files. It stores project configuration and adds all the applications and middleware applications.</p><p>urls.py: Handles all the endpoints of our app.</p><p>wsgi.py: Used for deploying our application on servers like apache.</p><p>asgi.py: Similar to wsgi.py but with additional functionalities.</p><p>__init__.py: It is present only to indicate that this directory holds a django project.</p><p>manage.py: Let you interact with the app on the command line interface.</p><h3>Admin Interface</h3><p>It is the most distinctive part of django. It reads metadata from our model and quickly generates an interface where trusted users can manage content on the site to remove the need for an external content resource manager.<br>It’s practical as an interface for our data and interacts with it in a meaningful way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*U4_eZ9gDQ3m9S5XhQgFGzg.png" /></figure><p>But again, this is not all-powerful. You cannot do just everything with it. It can only perform limited tasks, so if you want to make something better or different, writing custom HTML is the way.</p><h3>Let’s make an API</h3><p>Let’s quickly jump to the main part after setting up the required environment -</p><ol><li>First of all, run migration for the directory<br><em>python manage.py migrate</em></li><li>Now we will need an admin user to interact with django<br><em>python manage.py createsuperuser — email admin@head.com — username admin</em></li><li>Now setup django rest framework API in settings inside INSTALLED_APPS as</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/0*G3id_qawEho9m2vV" /></figure><p>4. Now we will create a ModelSerializer class as per the requirements</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/0*H7xqh0s1FAbhoDm1" /></figure><p>5. Now it is time to create views for our application that would be rendered. UserList view is used for getting all the</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/0*6zwh5dHPnbKkj8c9" /></figure><p>6. The final step is to decide the endpoints for the API</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/0*KqygDhSmgmRdm6NW" /></figure><p>7. And WOOHOO our API is ready to run. Here is a sample output for one of the APIs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bkh9xmRCTXIya6Ve" /></figure><h3>Conclusion</h3><p>This marks the end of this blog, where I have tried my best to give you an overview of prime things. If you liked the blog, send some claps my way and follow me on <a href="https://twitter.com/manthanguptaa">Twitter</a> &amp; <a href="https://www.linkedin.com/in/manthangupta109/">Linkedin</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=894746200767" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Big Ideas Behind Reliable, Scalable & Maintainable Systems]]></title>
            <link>https://medium.com/@manthangupta109/the-big-ideas-behind-reliable-scalable-maintainable-systems-135e5b573d63?source=rss-602e6ade635c------2</link>
            <guid isPermaLink="false">https://medium.com/p/135e5b573d63</guid>
            <category><![CDATA[design-systems]]></category>
            <dc:creator><![CDATA[Manthan Gupta]]></dc:creator>
            <pubDate>Fri, 26 Nov 2021 19:18:25 GMT</pubDate>
            <atom:updated>2021-11-26T19:18:25.862Z</atom:updated>
            <content:encoded><![CDATA[<p>Theoretical approaches are different from ground reality. There are many tools for a single-use case, each having its characteristics. There are various approaches to building a thing, and we need to figure out which tools and approaches are appropriate for the task. It can be generally hard to combine tools when a single tool doesn’t suffice all the requirements.</p><p>When designing a data system, a lot of questions arise that spin the head. Questions like how to ensure data correctly even when things go south? How to provide good performance even at a severe load?</p><p>Many factors influence the design of a data system. In this blog, we are going to discuss the three major concerns that are important in most systems.</p><h3>Reliability</h3><p>The system should work as expected, even in adverse conditions. The system should be able to tolerate mistakes of users or usage in an unexpected way. Reliability intuitively gives away its meaning, reliable enough to perform well under the expected load.</p><p>A fault is something that has gone wrong, and systems that can anticipate these faults and cope with them are known as resilient.</p><p>Let’s discuss how to deal with different types of faults</p><ol><li>Hardware faults<br>Hard disk crashes, faulty RAM, blackout, etc can cause hardware faults. The first response is to add redundancy to individual hardware components of the system to reduce the rate of failure. When one of the hardware components dies or for any reason isn’t working anymore then the redundant component can take its place.</li><li>Software Errors<br>These types of errors are harder to anticipate and cause issues across nodes. There is no quick solution to this problem, but lots of small things can help. Thorough testing, process isolation, stress testing, monitoring, and analyzing system behavior in production, etc.</li><li>Human Errors<br>Humans design the system and keep it running. Even with the best of intentions, humans can make mistakes. So, we have to make the system more reliable to accommodate human errors. To account for these, decouple the systems where people make the most mistakes, test thoroughly at all levels, allow quick and easy recovery of the systems from human errors, and set up detailed monitoring.</li></ol><h3>Scalability</h3><p>Scalability describes a system’s ability to cope up with the increased load. This load can be defined in different ways depending on the architecture of our system. It may be requests per second to a server, the ratio of reads to writes for databases, etc.</p><p>In scenarios where we would expect that each request will be served in the same time, it doesn’t. We get the variation that may be due to random addition of latency, loss of network packet, garbage collection pause, etc. So usually, it’s a common practice to see the average response time for a service.</p><p>To cope with the load, people often talk between scaling up (moving to a more powerful machine) and scaling out (distributing load).</p><h3>Maintainability</h3><p>Once a product is built, and no more innovation is left. It is mostly maintaining the product, and the majority of the cost is spent on ongoing maintenance — fixing bugs, failure investigation, adding features.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=135e5b573d63" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Algorithmic bias in AI]]></title>
            <link>https://medium.com/@manthangupta109/algorithmic-bias-in-ai-bc82c53dbdd6?source=rss-602e6ade635c------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc82c53dbdd6</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-and-society]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Manthan Gupta]]></dc:creator>
            <pubDate>Sat, 20 Nov 2021 17:16:11 GMT</pubDate>
            <atom:updated>2021-11-20T17:16:11.423Z</atom:updated>
            <content:encoded><![CDATA[<p>Artificial Intelligence is the capability of a machine to mimic the decision-making and problem-solving skills of human-based on the training given to it. And this training can cause biases due to the skewed data that the human who has trained the machine chooses. In an attempt to make an equitable world with the help of AI, we let these small subtleties take over and create biases unknowingly.</p><h3>What is AI Bias?</h3><p>The decision-making ability of AI when performing partiality towards a group of people or a thing is known as AI bias. Racial bias, gender bias, age discrimination, etc are some of the examples of AI bias. The AI systematically takes decisions that are unfair for a group. The enormous impact of this is more concerning as compared to anyone giving heed to it.</p><h3>Some examples of AI Bias</h3><p>Algorithmic bias can take place in many situations. Here are a few of them</p><ol><li><strong>Online recruitment tools<br></strong>Amazon workforce has overall 60% male with 74% of all the managerial positions occupied by them. Their recruiting algorithm had a gender bias and, when discovered, was shut down. The training data used to build the AI system were the resumes from the past 10-years, which were predominantly white males. Trained on predominantly white male workforce data, the algorithm searched for word patterns than relevant skill sets. It caused the algorithm to penalize any resume that contained the word ‘women’ resulting, in gender bias.</li><li><strong>Word association<br></strong>Princeton University analyzed and linked 2.2Mn words, which showed that European names were perceived as more pleasant compared to African-Americans. Words like “women” and “girl” are associated with arts instead of science and maths that are most likely connected to males.</li><li><strong>Facial Recognition<br></strong>In a study by MIT, a researcher found that the algorithms powering commercially available facial recognition software systems failed to recognize darker skin complexions.<br>Another study by Georgia Tech revealed that self-driving cars powered by AI struggle in detecting dark-colored pedestrians might risk their life.</li><li><strong>Criminal Justice<br></strong>The COMPAS algorithm used by the judges was found to be biased against African-Americans. The algorithm gives a risk score to the defendant for the probability of committing a future offense based on the records. When compared to white males who had a similar probability of future offense resulted in less detention period as to African-Americans.</li></ol><p>While these examples are not exhaustive but showcase those biases aren’t just theoretical but a real-world issue.</p><h3>Causes of Bias</h3><p>There are mainly two reasons for this to happen</p><ol><li>Personal bias due to data gatherer</li><li>Environmental bias caused intentionally/ unintentionally in the process of data gathering</li></ol><p>Sourcing of training data is done by someone like us, and knowingly or unknowingly, we tend to involve our internal biases. This is projected in the data used to build machine learning models.</p><p>Environmental bias can be due to sourcing data locally for a system that is to be served globally. The AI system is not trained with enough representative data.</p><h3>Solution</h3><p>Taking steps towards solving the AI bias is crucial as its repercussions can be paid with life behind the bar for more than expected or killed as a pedestrian by a self-driving car. Here are a few solutions that might help us to take care of this:</p><ol><li>Everyone faces a unique set of risks from AI bias where it could creep into your datasets and algorithms. Determine your unique vulnerabilities and list down the bias of your AI system. Calculate the risk so that the mitigation efforts are focused on the right things.</li><li>Controlling the data is the most predominant part of solving this issue of AI bias. The traditional process to control probably isn’t sufficient enough to detect problems that can cause AI bias. Special attention is required to point out issues in historical data and data acquired via a third party.</li><li>AI is always learning from the data fed into it continuously. Governance needs to keep up with it so that there are certain rules in place already that helps a specialist to spot the problems conveniently. To build that confidence in the AI system, a systematic approach to continuous management of your AI system is critical.</li><li>Due to personal bias, sometimes we aren’t able to spot the issues that lead to AI bias. Diversifying the team welcomes a fresher perspective that will help notice different biases that some would be neglecting.</li></ol><h3>Conclusion</h3><p>As outlined in the blog, these types of algorithms should be concerning if there is not a process to control the bias that might creep in any form. If algorithms are designed responsibly, it might help in avoiding the consequences of discrimination.</p><p>Putting AI in everything isn’t necessary. Some things will be best solved by algorithms and AI tools, and some need due diligence and thoughtful consideration before making any type of decision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc82c53dbdd6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Kafka: Distributed Event Streaming Platform]]></title>
            <link>https://medium.com/theleanprogrammer/apache-kafka-distributed-event-streaming-platform-64f7f6d2f244?source=rss-602e6ade635c------2</link>
            <guid isPermaLink="false">https://medium.com/p/64f7f6d2f244</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Manthan Gupta]]></dc:creator>
            <pubDate>Mon, 20 Sep 2021 13:29:47 GMT</pubDate>
            <atom:updated>2021-09-20T13:29:47.708Z</atom:updated>
            <content:encoded><![CDATA[<h4>An introduction for beginners to Kafka</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LMJXfV_AM33X2ZaBgoA5gw.png" /></figure><h4><strong>What is Publish/Subscribe Messaging?</strong></h4><p>Before diving into Kafka, it’s crucial to get some basics right and pub/sub messaging is one of them. In a pub/sub model, any message published to a topic is immediately received by all its subscribers via a broker. The publisher somehow classifies the message that subscribers receive if they are subscribed to that class.</p><p>There are multiple use-cases to pub/sub messaging, but the most popular one would be the chat application. When a user sends a message to a chat room, it gets published on the chat room topic. Subscribers or the member of the chat room receive the message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*AqalcHhpwLTBJAD9xYnMhQ.png" /><figcaption>Source: AWS</figcaption></figure><h4>What is event streaming?</h4><p>In layman&#39;s terms, event streaming is nothing but capturing data in real-time from event sources like sensors, mobile devices, databases, cloud services, and software applications in the form of streams of events.</p><p>These events are stored durably for all types of playing around with the data. It is an implementation of pub/sub architecture with its twist, ensuring the right information is present at the right place and right time.</p><p>Event streaming has a plethora of use-cases across industries and companies. But some of them include processing financial transactions in real-time, track real-time shipments, capture data from IoT devices and analyze them, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*kwW4b4KPBA9ntkPQtipteg.png" /><figcaption>Source: Confluent</figcaption></figure><h4>But what is Apache Kafka?</h4><blockquote>According to the official website, Apache Kafka is an open-source distributed event streaming platform used for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.</blockquote><p>Kafka provides three main functions:</p><ol><li>Publish and subscribe to a stream of events</li><li>Durable and reliable storage of stream of events</li><li>Processing of streams in real-time</li></ol><p>And all this functionality is provided in a distributed, fault-tolerant, elastic, secure manner, and highly scalable.</p><p>This comes in handy when creating a data pipeline that tracks user activity in real-time. Kafka here is used to ingesting and store streaming data. Kafka is also often used as a broker that processes and mediates communication between two applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ccUPUFBXUzP5mt6uYEoQtg.png" /><figcaption>Source: Confluent</figcaption></figure><h4>The Architecture</h4><p>An <strong>event </strong>means that something has happened in the scope where it’s listening. <strong>Producers</strong> are clients that publish events to Kafka, and <strong>consumers</strong> are those that subscribe to these events. These both are fully decoupled in Kafka that helps it achieve high scalability.</p><p>Events are stored durably and reliably in <strong>topics</strong>. Topics can be thought of as a folder and events as files inside the folder. Topics are multi-producer and multi-subscriber. A topic can have 0 to many producers that write events to it as well as 0 to many subscribers to the topic.</p><p>A topic is spread over several different Kafka brokers, which is very important for scalability as it allows read/write from multiple brokers at the same time.</p><p>To make the data highly available and fault-tolerant, the topic is replicated across geo-regions and data centers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u3vTjt0z8moUbBKh3qOFtQ.jpeg" /><figcaption>Source: Apache Kafka</figcaption></figure><p>The blog works as just a primer if you want to start with Kafka. If you want to dive deep into this then I recommend you to refer to the official documentation of Apache Kafka.</p><p>If you liked what you saw and read then don’t forget to click on that clap button and support me. You can reach out to me on <a href="https://www.linkedin.com/in/manthangupta109/">LinkedIn</a> and <a href="https://twitter.com/manthanguptaa">Twitter</a> to tell me your thoughts about the blog or just to connect.</p><h4>References</h4><ul><li><a href="https://kafka.apache.org/documentation/">Apache Kafka</a></li><li><a href="https://www.confluent.io/what-is-apache-kafka/">What is Apache Kafka? | Confluent</a></li><li><a href="https://aws.amazon.com/msk/what-is-kafka/">What is Apache Kafka? | AWS</a></li></ul><p>Don’t forget to follow The Lean Programmer Publication for more such articles, and subscribe to our newsletter <a href="http://tinyletter.com/TheLeanProgrammer">tinyletter.com/TheLeanProgrammer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64f7f6d2f244" width="1" height="1" alt=""><hr><p><a href="https://medium.com/theleanprogrammer/apache-kafka-distributed-event-streaming-platform-64f7f6d2f244">Apache Kafka: Distributed Event Streaming Platform</a> was originally published in <a href="https://medium.com/theleanprogrammer">TheLeanProgrammer</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Denoising Images Using Autoencoders]]></title>
            <link>https://medium.com/theleanprogrammer/denoising-images-using-autoencoders-4a64499f10ba?source=rss-602e6ade635c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a64499f10ba</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[autoencoder]]></category>
            <dc:creator><![CDATA[Manthan Gupta]]></dc:creator>
            <pubDate>Tue, 10 Nov 2020 07:56:28 GMT</pubDate>
            <atom:updated>2021-04-11T13:23:00.764Z</atom:updated>
            <content:encoded><![CDATA[<p>Image noise may be caused by various intrinsic or extrinsic conditions which are practically hard to deal with. Denoising images is a very fundamental challenge in the field of <strong>image processing</strong> and <strong>computer vision</strong>. Therefore, it plays an important part in a wide variety of fields where getting the original image is really important for robust performance.</p><p>Some examples of how images with noise look like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*avDC5lmq6tQLA5FmS_Aa9g.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*FIyPBiJXyV2VBp37rNUO6g.jpeg" /><figcaption>Example of image with noise</figcaption></figure><h3>What is an Autoencoder?</h3><p>Autoencoder is an <strong>unsupervised artificial neural network</strong> that is trained to copy its input to output. Let’s consider that we are given an image, an autoencoder will first encode the image into a lower-dimensional representation, then decodes the representation back to the image.</p><p>With appropriate dimensionality and sparsity constraints, autoencoders can learn data projections that are far more interesting than PCA or other basic techniques.</p><p>Autoencoders are only able to <strong>compress data</strong> similar to what they have been trained on. They are also <strong>lossy in nature</strong> which means that the output will be degraded compared to the original input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Me8UpTdSuznquyF86s9xeQ.jpeg" /></figure><h3>Where else can Autoencoder be used?</h3><ol><li>Converting a black and white image into a colored image</li><li>Extract required features from an image</li><li>Dimensionality reduction</li><li>Watermark Removal</li></ol><p>The list isn&#39;t exhaustive so let me know in the comments if you find more uses of autoencoders.</p><h3>Architecture</h3><p>There are mainly 3 parts in autoencoders</p><ol><li><strong>Encoder</strong>: In this part of the architecture the model compresses the input data to represent the compressed data in a reduced dimension.</li><li><strong>Code</strong>: Also known as <strong>Bottleneck</strong> this part of the architecture represents the compressed data that is going to be fed to the decoder.</li><li><strong>Decoder</strong>: This part reconstructs the encoded data as close to the input data as possible. The output from the decoder is a lossy reconstruction of the original data.</li></ol><p>The goal of an autoencoder is to get an output that is identical to the input. The dimensionality of the input and output is similar as obviously the goal is to get the output as identical to the input we can get.</p><p>They are trained similarly to ANNs via backpropagation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*bfmT2JpqcCW-PCzW4w6Sww@2x.png" /><figcaption>Autoencoder Architecture</figcaption></figure><h3>Hyperparameters Of Autoencoders</h3><p>There are mainly 4 parameters that we need to set before training the autoencoder</p><ol><li><strong>Code size</strong>: This represents the number of nodes in the middle layer. The smaller the code size more the compression is and if you want less compression then increase the code size.</li><li><strong>Number of Layers</strong>: In the above architecture image there are only 2 layers in encoder and decoder but we can make it as deep we want it to be.</li><li><strong>The number of nodes per layer</strong>: Usually what happens is the number of nodes per layer decreases with each subsequent layer of an encoder and then starts increasing again with each subsequent layer of the decoder. The decoder is symmetric to the structure of the encoder but that’s not a requirement.</li><li><strong>Loss function</strong>: The popular choices here are <strong>mean squared error(MSE) </strong>or<strong> binary cross-entropy. </strong>If the input values are in the range [0,1] then binary cross-entropy is favored as compared to MSE. Otherwise, we just use MSE.</li></ol><p>Now, you know the basics of the autoencoder, and if this interests you then you can find some helpful links at the end of the blog which are blogs from other writers who have driven deep in this topic and also have talked about the mathematical part of the autoencoder.</p><p>Let’s dive into building our encoder which will help us to denoise the images.</p><h3>Implementation</h3><p>For implementing this we will be using the famous <strong>MNIST dataset</strong> as input. For this, we don’t need to download the dataset. We can import it from Keras library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*N-43GwxkmUqzJgbTJs3ocw.png" /></figure><p>MNIST is a dataset of black and white handwritten images of size 28x28.</p><p>Denoising is the process of removing noise. This can be an image, audio, or document. You can train an Autoencoder network to learn how to remove noise from pictures.</p><p>To train our autoencoder let’s first start by loading our data from Keras and scale down to [0,1].</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/273fffcf34a8ae53af4bfda0aa8b7199/href">https://medium.com/media/273fffcf34a8ae53af4bfda0aa8b7199/href</a></iframe><p>Let’s now move to add noise to the dataset so that it can be fed to the autoencoder as the input data. We will be adding <strong>Gaussian noise</strong> to the image by a factor of 0.5.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1692a1f5f3b433c3ce544e9e5d5188f0/href">https://medium.com/media/1692a1f5f3b433c3ce544e9e5d5188f0/href</a></iframe><p>Now, if we try to visualize the images we will get the images with white noise on it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f308f38b89d09890d15ed2b5e1e2c38c/href">https://medium.com/media/f308f38b89d09890d15ed2b5e1e2c38c/href</a></iframe><p>With this piece of code, we will be able to display the noised images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*ry4Vwtg0hsrQWjXaip0efA.png" /><figcaption>Noisy Digits</figcaption></figure><p>These noisy digits will serve as our input data to our encoder. The autoencoder isn’t fed the original images at any point in time and we are expecting the autoencoder to give us output without any noise. Let’s see how good of work the autoencoder will do and if it is able to remove the noise.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b30a3d889f1fff822fc207535d97ecce/href">https://medium.com/media/b30a3d889f1fff822fc207535d97ecce/href</a></iframe><p>In the first half of the code, we are defining the input shape to be (28, 28, 1) as the dimensions of the images are 28x28 and are grayscale which explains the input shape.</p><p>Then we have the convolution layer of kernel size (3, 3) with 32 output filters and activation function ‘Relu’. As padding is mentioned as the same the image size won’t be decreased and will remain the same.</p><p>After each convolution layer, we are using the MaxPooling layer to reduce the dimensions. The (28, 28, 32) output from the first convolution layer is decreased by a factor of 2 so it will be (14, 14, 32) and then again by a factor of 2 which gives (7, 7, 32).</p><p>This encodes the data and now if we move to the second part of the code it is the reconstruction of the original data. This is the part where the autoencoder learns to remove the noise from the image. We are using UpSampling to rebuild the images to the original dimension (28, 28).</p><p>Moving to the final part of the implementation let&#39;s fit the model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1689f8fa5eb51fb14e01ad31d563062/href">https://medium.com/media/a1689f8fa5eb51fb14e01ad31d563062/href</a></iframe><p>And that&#39;s it! You have implemented your autoencoder which will help us in denoising the images. Let’s check out the output from the autoencoder to see if it really did a good job or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*xOMGgKAYYYLj6TFMPb_5yQ@2x.png" /></figure><p>You can see the awesome job our autoencoder has done in reconstructing the image and give us the denoised images back. If I want to summarise the whole process in one image, the image below is the best for that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*kYnDIS-3yCqrIVTX8omkgQ@2x.png" /></figure><p>Pat on your back for doing an awesome job! If you liked what you saw and read then don’t forget to smash that clap button and support me. You can reach out to me on <a href="https://www.linkedin.com/in/manthangupta109/">LinkedIn</a> to tell me your thoughts about the blog or just to connect.</p><p>If you want to deep diver into this particular subject then you can read it from the below links.</p><ul><li><a href="https://blog.keras.io/building-autoencoders-in-keras.html">Building Autoencoders in Keras</a></li><li><a href="https://www.jeremyjordan.me/autoencoders/">Introduction to autoencoders.</a></li><li><a href="https://www.tensorflow.org/tutorials/generative/autoencoder">Intro to Autoencoders | TensorFlow Core</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a64499f10ba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/theleanprogrammer/denoising-images-using-autoencoders-4a64499f10ba">Denoising Images Using Autoencoders</a> was originally published in <a href="https://medium.com/theleanprogrammer">TheLeanProgrammer</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a BMI app with Flutter]]></title>
            <link>https://medium.com/theleanprogrammer/building-a-bmi-app-with-flutter-f14dfe2348?source=rss-602e6ade635c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f14dfe2348</guid>
            <category><![CDATA[app-development]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[flutter]]></category>
            <category><![CDATA[android]]></category>
            <category><![CDATA[android-app-development]]></category>
            <dc:creator><![CDATA[Manthan Gupta]]></dc:creator>
            <pubDate>Fri, 24 Apr 2020 09:10:47 GMT</pubDate>
            <atom:updated>2021-04-17T06:51:56.444Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn Flutter by making a BMI calculator app</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q3pR5eXpO7xI6Lyu6KsdMg.png" /></figure><p>Welcome to another blog where you will learn to build a simple BMI App with Flutter for beginners.</p><p>So, let’s start this!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*gT8g__5x3Id6ZdMUVS1tiA.jpeg" /></figure><p>First of all, create a flutter project in your favorite IDE. I prefer IntelliJ but you could have some other preference.</p><p>Remove all the code of the counter app which loads by default on the IDE. Then add the following code for starters.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c9558610f40f8128c19c17ac9ff181f/href">https://medium.com/media/6c9558610f40f8128c19c17ac9ff181f/href</a></iframe><p>Select a class name for your HomeScreen for the BMI app so that the home attribute could refer to that class and open the HomeScreen. Which should look like this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a45a9009154d42d51537376d886eab3b/href">https://medium.com/media/a45a9009154d42d51537376d886eab3b/href</a></iframe><p>We will create a Stateful class of Home(). After which the code should like this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/47924bdea896bf40b0971ce40f17bfa2/href">https://medium.com/media/47924bdea896bf40b0971ce40f17bfa2/href</a></iframe><p>Then instead of returning the container widget, we will have it return the Scaffold widget. We will use the app bar attribute of the Scaffold widget to create the app bar and add a title to our app. After it should look like this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91bf19731817096e01c264700db4dfcb/href">https://medium.com/media/91bf19731817096e01c264700db4dfcb/href</a></iframe><p>We will adjust the title by aligning it in the center, increasing the size of the text, changing the color of the text, and also change the color of the app bar.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3eff0585180540f2654708b22ebc8b20/href">https://medium.com/media/3eff0585180540f2654708b22ebc8b20/href</a></iframe><p>Now let’s add an icon to our app so that it gets its own identity. I am using <a href="https://www.flaticon.com/free-icon/bathroom-scale_1754237?term=bmi&amp;page=1&amp;position=1">this</a> icon but you can use any as per your choice. So the first thing now we have to do is create a new folder in our project directory and name it <strong>assets. </strong>Under this asset folder, we will create another folder and name it as <strong>logo</strong>. Keep a habit of putting everything cleanly in different folders so that it is easy for anyone to understand what is where in your project. The project directory should look like this after adding your png file in the logo folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*GOCX6EF680Sh3WFl-rLOnw.png" /></figure><p>If we want to use this asset in our app then we will have to import it in our project. For that go to your <strong>pubspec.yaml</strong> file and under the assets, attribute add the following line. Remember that the name of my png is bmi.png and it is under logo directory which is under another directory assets.</p><pre>- assets/logo/bmi.png</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*jl-2TXeyrwZSDlfIvdD35A.png" /></figure><p>And after adding this line to your pubspec.yaml file click on the pub get to import it. If there is an error then the most common error is when the assets attribute is not in the proper indentation. So take care of it.</p><p>Now you have imported your image lets use it in our app.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a37b57a030bf6ac3bcafb5c67008321e/href">https://medium.com/media/a37b57a030bf6ac3bcafb5c67008321e/href</a></iframe><p>We have the logo, we have the app bar what else do we need?</p><p>How are we going to take the input from the user? Let’s add some TextFields in there to take user input.</p><p>After adding 3 TextFields in the app the code looks like this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1bf924b8b4fefbae22b5b5855c61053/href">https://medium.com/media/e1bf924b8b4fefbae22b5b5855c61053/href</a></iframe><p>We still have some work to do with the TextFields. We will add a controller which lets us listen and control the text of the TextField as well. Then add the KeyboardType as a number because we don&#39;t want any alphabets or special characters as input. End this by adding some decorations like hintText, labelText, and an icon.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb3794f9e5061e85ba972e693de84927/href">https://medium.com/media/eb3794f9e5061e85ba972e693de84927/href</a></iframe><p>We are defining the markers for the 3 TextFields as in when the user enters the data it will be saved in these variables and these are final as we don’t want it changed throughout the code to maintain the homogeneity throughout the code. The exact job of the TextEditingController as stated in the doc is</p><blockquote>Whenever the user modifies a TextField with an associated TextEditingController, the TextField updates value and the controller notifies its listeners. Listeners can then read the text and selection properties to learn what the user has typed or how the selection has been updated.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eaed3d8650cfc2e80b226666fbc797c7/href">https://medium.com/media/eaed3d8650cfc2e80b226666fbc797c7/href</a></iframe><p>After adding the TextFields our app looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eYu8T3owC0K097AVx9b4SQ.jpeg" /></figure><p>Let’s add a button</p><p>We will have the parent widget as padding for the button to provide padding around all sides of the button. Using the Raised Button widget with text on the button as calculate, color as lime, and on getting pressed it will refer to the Calculate function which is the logic part of the app that we will build next.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0d3c0c2a5551bb1bee49b43771dd95e/href">https://medium.com/media/d0d3c0c2a5551bb1bee49b43771dd95e/href</a></iframe><p>Let’s build the logic part of the app. We will create a function Calculate and use the setState. What it does is that it notifies the framework that the internal state of the object has changed in a way that might impact the user interface, which causes the framework to schedule a build for this State object. So we use <strong>setState </strong>when the state of the widget is going to change so that the to trigger a rebuild of the view and see immediately the changes.</p><p>The next part is to convert the input given in the TextField to integer and double( age in integer and height and weight in double). Then check if all the TextFields are not empty and the entered value is greater than 0 then calculate the result. We the BMI index so we check in what category the person lies by using if else condition and provide the person with a category.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c66e095c78087b407cd5194de8ec42f/href">https://medium.com/media/2c66e095c78087b407cd5194de8ec42f/href</a></iframe><p>We have the logic part ready. Let’s print these results on the screen so that the app user gets the output for his/her input. Let’s add 2 text widgets which will display the result i.e the BMI index and the category in which the person lies.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/83158103950c15fc9851b21d21ecd4c6/href">https://medium.com/media/83158103950c15fc9851b21d21ecd4c6/href</a></iframe><p>The final app looks like this after entering some random data in the TextFields.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3CKPMw7XouXmIflFJ92AfA.jpeg" /></figure><p>I hope you all liked this blog and it helped you get started with Flutter! Don’t forget to smash that clap button and leave a comment down below.</p><p>Meet you in the next one!</p><p>Don’t forget to follow <a href="https://medium.com/theleanprogrammer">The Lean Programmer Publication</a> for more such articles, and subscribe to our newsletter <a href="https://tinyletter.com/TheLeanProgrammer">tinyletter.com/TheLeanProgrammer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f14dfe2348" width="1" height="1" alt=""><hr><p><a href="https://medium.com/theleanprogrammer/building-a-bmi-app-with-flutter-f14dfe2348">Building a BMI app with Flutter</a> was originally published in <a href="https://medium.com/theleanprogrammer">TheLeanProgrammer</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>