<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ritesh Kapoor on Medium]]></title>
        <description><![CDATA[Stories by Ritesh Kapoor on Medium]]></description>
        <link>https://medium.com/@ritesh-kapoor?source=rss-5e168bb045d9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MvgIAR2052p2eSxiPGGBhw.jpeg</url>
            <title>Stories by Ritesh Kapoor on Medium</title>
            <link>https://medium.com/@ritesh-kapoor?source=rss-5e168bb045d9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:24:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ritesh-kapoor/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding Real-Time Application Monitoring]]></title>
            <link>https://medium.com/expedia-group-tech/essential-application-monitoring-metrics-a08519ecab9d?source=rss-5e168bb045d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/a08519ecab9d</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[operations]]></category>
            <dc:creator><![CDATA[Ritesh Kapoor]]></dc:creator>
            <pubDate>Tue, 13 Jun 2023 12:01:48 GMT</pubDate>
            <atom:updated>2023-06-13T13:05:08.513Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/expedia-group-tech/software/home">EXPEDIA GROUP TECHNOLOGY — SOFTWARE</a></h4><h4>How we observe, monitor and improve the operational performance of our applications</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MB_t_dcoAS72giYj" /><figcaption>Photo by Ruth Troughton on Unsplash</figcaption></figure><h3>Introduction</h3><p>Application monitoring is the process of collecting data to track availability, bugs, resource use, and changes to performance in applications that affect the end user. Understanding the performance of applications in real-time is critical for ensuring positive experiences for our travelers. Proactive application monitoring with monitoring tools provides teams with deep visibility into application performance and enables catching critical errors before they affect end users. Application monitoring tools provide alerts to live anomaly events, and through distributed tracing provide a means of seeing which events form a causal chain that led to them across multiple services.</p><h3>Performance indicators</h3><p>Performance indicators are a quantitative measure of how effectively the application achieves the business objectives. Capturing the right metrics will give a comprehensive report and powerful insights into ways to improve your application. Not all performance indicators would be required for different metrics.</p><ul><li><strong>TP99/TP95/TP90 Percentile: </strong>TPx latency is the maximum time under which <strong>x</strong>% of requests have been served;</li><li><strong>Average</strong>: Average Time taken;</li><li><strong>Min/Max: </strong>Maximum / Minimum time taken;</li><li><strong>TPS: </strong>Overall transactions processed per second;</li></ul><h3>SLI/SLO/SLA</h3><p><em>SLI/SLO/SLA</em> should be defined and monitored for all applications.</p><ul><li><strong>SLI: </strong>An SLI (<strong>S</strong>ervice <strong>L</strong>evel <strong>I</strong>ndicator) is a quantitative measure of some aspect of the level of service that is provided. <strong>Example SLI</strong>: “<em>availability</em>”.</li><li><strong>SLO: </strong>An SLO (<strong>S</strong>ervice <strong>L</strong>evel <strong>O</strong>bjective) is a target value or range of acceptable values for a particular SLI. Not every SLI will have an associated SLO. RPS (Requests Per Second) is an example of an SLI that we can’t set an SLO for because it is determined by user load. <strong>Example SLO</strong>: “<em>availability &gt;= 99.9% for 95% of days</em>”</li><li><strong>SLA: </strong>An SLA (<strong>S</strong>ervice <strong>L</strong>evel <strong>A</strong>greement) is an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain. <strong>Example SLA</strong>: <em>“For each day Service X doesn’t achieve at least 99.9% availability, the customer will be paid $100 for not meeting SLA”</em>.</li></ul><h3>Metrics</h3><p>Metrics categories are listed in the following paragraphs, and more than one category might apply to an application. For example, if our service is not using Lambda then metrics related to Lambda Monitoring can be considered optional. Capturing all metrics under the category would enable identifying errors and performance issues during service degradations. Performance indicators for metrics can be optional and may vary for different applications and clients as required. Metrics are organised in the following way:</p><blockquote><strong>&lt;Metric Name&gt; [</strong>Filtering/Group By Criteria&#39;s, Criteria1, Criteria2 , ....<strong>]:</strong> &lt;<strong>Description&gt;</strong></blockquote><blockquote>Filtering/Group By Criteria’s can be used filter or get aggregated results for metrics</blockquote><h3>Application VM metrics</h3><p>VM metrics refer to the performance and utilization metrics associated with the Java Virtual Machine (JVM) or .NET Runtime including <a href="https://plumbr.io/handbook/garbage-collection-in-java">heap</a>, and <a href="https://www.yourkit.com/docs/kb/sizes.jsp#:~:text=Also%2C%20the%20JVM%20has%20memory,as%20well%20as%20interned%20Strings.">non-heap</a> memory. These metrics provide insights into the behaviour and efficiency of the JVM and can be used for monitoring, troubleshooting, and optimizing Java/.NET applications. Capturing these metrics would help us understand the performance of our application e.g. Reasons for the increase in response time, memory leaks, etc.</p><ul><li><strong>Heap Memory Utilisation: </strong>Overall heap memory utilisation for different heap areas like Young, Eden space, etc;</li><li><strong>Major GC Frequency Metric: </strong>The frequency at which Major GCs are triggered;</li><li><strong>Minor GC Frequency Metric: </strong>The frequency at which Minor GCs are triggered;</li><li><strong>Major GC Time Spent: </strong>Average Time spent in Major GC Cycle;</li><li><strong>Minor GC Time Spent: </strong>Average Time spent in Minor GC Cycle;</li><li><strong>Total Thread Count: </strong>Total Number of threads running within JVM;</li></ul><h3>Application API metrics</h3><p>API Metrics should be monitored in case the application exposes API to clients. Capturing these metrics would help in understanding overall traffic trends, SLA breaches, etc.</p><ul><li><strong>Response time TP [</strong>TOTAL CLIENT API<strong>]:</strong> Response time for API. Top Percentile e.g. TP99/TP95/TP90 Depending upon application requirements;</li><li><strong>API success rate TP [</strong>TOTAL CLIENT API<strong>]: </strong>Overall success rate. Top Percentile e.g. TP99/TP95/TP90 Depending upon application requirements;</li><li><strong>Request count [</strong>TOTAL CLIENT API<strong>]:</strong> Overall request count for API;</li><li><strong>Technical error status code count [</strong>TOTAL CLIENT API<strong>]: </strong>Technical Error Status Code count per API or in general. (These are errors/exceptions when there is some technical fault in the system. This is usually generated when the system has valid input and fence/conditions are expected to be processed, but due to some technical issue (like no connectivity, database schema incorrect, unable to insert data in the database, etc), the request could not be processed;</li><li><strong>Business error status code count [</strong>TOTAL CLIENT API<strong>]: </strong>Business Error Status Code count per API or in general (These are expected errors when a system receives a request which does not conform to business criteria, for which there are fences/conditions and the request could not be completed. Examples could be invalid input/request, unauthorized access to resources, or a request not aligned with business rules);</li><li><strong>SLA breach count [</strong>TOTAL CLIENT API<strong>]: </strong>Total number of requests which were not served within the SLA Threshold;</li></ul><h3>Application database response metrics</h3><p>Metrics related to applications connecting to Datastore like SQL, S3 and No-SQL. Capturing these metrics would help in understanding slow queries, DB Network issues, connection pooling issues, etc.</p><ul><li><strong>Overall client response time per API [</strong>TOTAL QUERY<strong>]:</strong> Response time while making external API calls to other services;</li><li><strong>Timeout/connection reset count</strong>: Failure count due to connectivity/timeout issue while invoking external APIs;</li><li><strong>Client status code count [</strong>TOTAL QUERY<strong>]</strong>: Overall status code count for downstream service;</li></ul><h3>Downstream clients’ metrics</h3><p>Metrics related to applications connecting other downstream services. Capturing these metrics would help in understanding downstream clients&#39; behavior, their error trends, etc.</p><ul><li><strong>Overall client response time per API [</strong>TOTAL CLIENT API<strong>]:</strong> Response time while making external API calls to other services;</li><li><strong>Timeout/connection reset count [</strong>TOTAL CLIENT API<strong>]:</strong> Failure count due to connectivity/timeout issue while invoking external APIs;</li><li><strong>Client status code count [</strong>TOTAL CLIENT API<strong>]:</strong> Overall status code count for downstream service;</li></ul><h3>Infrastructure metrics</h3><p>Infrastructure metrics (EC2/ECS/Kubernetes) where the application is deployed and executed. Capturing these metrics will help us understand the underlying bottlenecks for our application.</p><ul><li><strong>Container CPU utilisation:</strong> CPU Utilisation for the container where the application is deployed;</li><li><strong>Container memory utilisation:</strong> Memory Utilisation for the container where the application is deployed (including Main and Swap Memory);</li><li><strong>Service unavailability metrics:</strong> Capturing service unavailability;</li></ul><h3>Datastore metrics</h3><p>Datastore Metrics (SQL/NoSQL/S3, etc) are captured at the datastore side, to understand if the datastore is responding within SLA. To understand horizontal or vertical scaling requirements, if there are slow queries, or if the datastore is misused by redundant queries. Understanding query patterns and optimizing datastore accordingly.</p><ul><li><strong>Query response time [</strong>TOTAL TYPE<strong>]:</strong> Query Response time as reported by the database;</li><li><strong>Queries per second [</strong>TOTAL TYPE<strong>]:</strong> Number of Queries executed per second;</li><li><strong>CPU utilization:</strong> CPU Utilization of the host where the Database is running (if applicable);</li><li><strong>Memory utilization:</strong> Memory Utilization of the host where the Database is running (if applicable);</li><li><strong>Page fault </strong>(If Applicable);</li></ul><h3>Queue metrics</h3><p>Metrics related to queues (e.g. <em>Kafka/RabbitMQ/ActiveMQ</em>, etc.) where message producers and consumers are involved. Capturing these metrics will help us identify slow consumers.</p><ul><li><strong>Consumer consumption rate</strong>: The rate at which records are consumed by consumers;</li><li><strong>Producer consumption rate</strong>: The rate at which the producer produces records;</li><li><strong>Average message size for records</strong>: The average size of records in the queue;</li><li><strong>Producer-Consumer lag</strong>: The difference in the number of records between the consumer and producer;</li></ul><h3>Cluster metrics</h3><p>Metrics related to service cluster for both application and datastores (if applicable). Capturing these metrics will help us understand issues like incorrect autoscaling policies.</p><ul><li><strong>Overall request count [</strong>TOTAL HOST<strong>]: </strong>Overall request count observed at the cluster level;</li><li><strong>Overall response time [</strong>TOTAL API HOST<strong>]: </strong>Overall response time for different APIs observed at the cluster level;</li><li><strong>Overall healthy host count:</strong> Total number of healthy host count;</li><li><strong>Total instances stopped per unit time: </strong>The rate at which instances are stopped;</li></ul><h3>Scheduled job metrics</h3><p>Metrics to monitor scheduled jobs. <br>The primary purpose of monitoring scheduled jobs is to identify and address any issues that may arise during their execution. This includes detecting failures, errors, delays, or deviations from the expected behaviour. By closely monitoring these jobs, organizations can proactively identify and resolve any problems before they impact critical business processes or cause disruptions.</p><ul><li><strong>Completion time [</strong>TOTAL JOB<strong>]: </strong>Total time for job completion;</li><li><strong>Job frequency [</strong>TOTAL JOB<strong>]:</strong> The frequency at which the job is executed;</li><li><strong>Job failures/errors count [</strong>TOTAL JOB<strong>]:</strong> The total number of failures for jobs;</li><li><strong>Job success count:</strong> Total count of success for jobs;</li></ul><h3>FaaS/Lambda Metrics</h3><p>Metrics related to Function As A Service.</p><p>Monitoring for Lambda functions is essential for ensuring the performance, availability, and efficiency of serverless applications. Monitoring these functions involves the continuous observation and analysis of their execution to detect issues, optimize performance, and ensure reliable operation.</p><p>It will help to identify any abnormalities, such as excessive execution time or frequent errors.</p><ul><li><strong>Overall Execution Success Count: </strong>Overall success count for successful execution of Function/Lambda</li><li><strong>Overall Execution Error Count: </strong>Overall error count for successful execution of Function/Lambda</li><li><strong>Overall Execution Time: </strong>Overall execution time for Function/Lambda</li></ul><h3>Load balancer metrics</h3><p>Load Balancer Metrics ( These are different from application metrics, it is possible that there are issues between load balancers and application instances)</p><ul><li><strong>Overall response time:</strong> overall response time observed at the load balancer;</li><li><strong>Overall failure count:</strong> overall request failures observed at the load balancer;</li></ul><h3>Event traceability</h3><p>Distributed tracing is a method of tracking application requests as they flow from multiple services and databases. Distributed tracing can troubleshoot requests that exhibit high latency or errors.</p><p>Tracing should not only apply to synchronous service communication but also asynchronous communication across different microservices.</p><h3>Conclusion</h3><p>Improving operational performance is challenging. Monitoring applications with the right metrics through tools, visualization, and alarms will troubleshoot the operational problem and help act at the right time.</p><p><strong>Acknowledgments</strong></p><p>Special thanks to <a href="https://medium.com/@lucapelosi">Lucas Pelosi</a> for reviews and comments.</p><blockquote><a href="https://careers.expediagroup.com/life/?utm_source=techblog&amp;utm_medium=career&amp;utm_campaign=postfooter">Learn more about technology at Expedia Group</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a08519ecab9d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/essential-application-monitoring-metrics-a08519ecab9d">Understanding Real-Time Application Monitoring</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Expedia Reviews Engineering Is Using Event Streams as a Source Of Truth]]></title>
            <link>https://medium.com/expedia-group-tech/how-expedia-reviews-engineering-is-using-event-streams-as-a-source-of-truth-d3df616cccd8?source=rss-5e168bb045d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3df616cccd8</guid>
            <category><![CDATA[architectural-patterns]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[event-sourcing]]></category>
            <category><![CDATA[architectural-design]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Ritesh Kapoor]]></dc:creator>
            <pubDate>Tue, 25 Apr 2023 12:01:33 GMT</pubDate>
            <atom:updated>2023-05-11T04:55:05.396Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/expedia-group-tech/engineering/home">EXPEDIA GROUP TECHNOLOGY — ENGINEERING</a></h4><h4>A journey to “Event sourcing”</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eZcUC0BVcFIvGOS-" /><figcaption>Photo by Austin Distel on Unsplash</figcaption></figure><p>Travel is fun, and Expedia Group™ provides the biggest network of travel content for any type of travel seller. By collecting and sharing reviews, we help to create an engaging user experience and build trust by setting expectations for both partners and travelers. We know that there is a direct correlation between conversion and review count/ratings.</p><h3>Introduction</h3><p>Expedia Group™ is becoming the most trusted travel marketplace in the world for reviews and insights from global and local travelers. We enable this by implementing a publishing loop of flexible review solicitation ensuring coverage, differentiation and stay experiences for continuous improvement.</p><p>The Expedia Group™ Reviews platform is built on top of four foundational pillars which are :</p><ul><li><strong>Notification: </strong>Notification focuses on “<em>WHEN” </em>to collect feedback from travelers.</li><li><strong>Collection: </strong>The Collection focuses on <em>“WHAT”</em> to collect from the traveler.</li><li><strong>Moderation: </strong>After collecting feedback reviews are moderated and validated.</li><li><strong>Distribution: </strong>Once reviews are validated and checked for appropriateness they are made available through APIs via Distribution Pillar</li></ul><p>With Moderation, the moderation pipeline sends validated reviews to a Kafka topic, which acts as the source of truth for all moderated reviews. After moderation, reviews are ready to be cached and displayed on the website. These reviews can also be processed for sentiment analysis, aggregation and analytics. The reviews pipeline contains multiple processors which are Kafka consumers streaming events from the source topic, processing events per use case, and providing output to different datastore/topics. Supporting new processors or use cases involves adding a new consumer reading event from the topic, starting at the beginning and providing the necessary output.</p><blockquote>The goal is to simplify integration, reduce migrations and support future use cases with agility.</blockquote><h3>Architecture</h3><figure><img alt="High Level Expedia Group Reviews Architecture for Distribution. Illustrates different systems like Aggregation, Caching, Search, Sentiment Analysis, Tag Extraction and Analytics connected to Moderated Reviews Kafka topic, doing processing and providing results through API or through other Kafka streams" src="https://cdn-images-1.medium.com/max/494/1*EcQF3lxaYDIFpY8Hf6ptrQ.png" /><figcaption>High-Level Reviews Distribution Architecture</figcaption></figure><p>The Reviews Distribution pipeline is responsible for reviews data distribution. Moderated reviews are pushed to Kafka Topic, which acts as the source for different processors:</p><ul><li><strong>Aggregation: </strong>Aggregating reviews data and provides a high-level summary of products.</li><li><strong>Caching: </strong>All the reviews are cached to In-Memory Datastore which is exposed through APIs for fast retrieval.</li><li><strong>Searching:</strong> Storing review data to Elasticsearch for sorting, filtering and searching use cases.</li><li><strong>Sentiment Analysis: </strong>Moderated reviews are sent for sentiment analysis to understand the experience, emotion or opinion of travelers about a particular product. e.g. different amenities of a hotel</li><li><strong>Tag Extraction: </strong>Using Natural Language Processing (NLP), relevant tags are extracted from reviews and tagged.</li><li><strong>Analytics: </strong>Reviews are then sent for analytics to better understand the customer’s experience with our product.</li></ul><p>We store the complete current state of the review with reviewId as the partition key in Moderated Reviews Kafka Topic, which acts as a source of truth for different data stores.</p><h3>What to store as an event</h3><p><em>“Events as a source of truth” </em>is a simple but powerful idea to persist the state of the business entity as a sequence of state-changing events. A new event is atomically inserted into the event store whenever the state of the entity changes. Event sourcing involves state changes as an immutable sequence of “log” of events.</p><ul><li>Entities’ current state can be reconstructed by capturing all the changes to the application state as a sequence of events.</li><li>Events for every entity change can lead to a large number of events, which can be optimized by periodically persisting snapshots of the entity’s current state. To reconstruct the current state, recent snapshots and the events that have occurred since that snapshot can be used.</li><li>An entity’s complete current state can also be stored in an event whenever there is a state change. The latest event provides the final state of the entity and doesn’t depend on any other snapshot or older events.</li></ul><p>Storing complete entity state vs state changes depends on the use case and performance trade-offs. The former erodes performance at the producer if a new change needs to be merged with an older snapshot before it is pushed to the store. In later consumption, performance degrades if merging needs to happen at the consumer. Another scenario involves producing an event which is consumed as-is, leading to no performance impact.</p><h3>Why Kafka for event sourcing</h3><p><a href="https://kafka.apache.org/documentation/">Kafka</a> is a high-performance, low-latency, scalable and durable log used worldwide and battle-tested at scale. Kafka not only offers events subscriptions by multiple applications but can be used as a primary data store. Kafka provides out-of-the-box <em>Compacted Topics </em>which allows data to be retained based on the key rather than time. It can not only be used to store events but also enables rewinding/replaying events multiple times, and reprocessing data at scale.</p><h3>Storing events in Kafka</h3><p><strong>Storing in Topic “per-entity-type”: </strong>A separate topic is created for all events related to a particular entity. E.g., Events related to all products should be present within one topic. Having a separate topic per entity allows a logical grouping of events and enables scaling each entity type stream separately.</p><p><strong>Storing in Partition “per-entity”: </strong>All of the events for a single entity should be assigned a single partition. E.g., Events related to a single user or product should always be present in a single partition. Assigning events for a single entity to the same partition ensures that all the updates are applied sequentially and don’t lead to an inconsistent state.</p><figure><img alt="Diagram illustrating Product and User Topic, How events are stored in topic per “per-entity-type”" src="https://cdn-images-1.medium.com/max/461/1*EmIX-qjqt_duGKhe1V-Xiw.png" /><figcaption>Storing events in topic per <strong>“per-entity-type”</strong> for Product and User information</figcaption></figure><p>The above diagram illustrates two topics which are Product and User storing events per entity type. Within each entity topic, we have partitions and events for a single product or user that are assigned the same partitions.</p><h3>Capacity planning</h3><p>Cluster Configuration and the number of brokers depend on several factors. Different parameters like CPU, RAM, Disk, JVM Size, Number of Kafka Instances, and Replication Factor can be estimated if we know the rate of events produced, consumed, and the number of events retained for any given window within the cluster. For now, let’s focus only on how we can calculate topic size and the number of partitions within it.</p><h4>Calculating queue size</h4><p>The size of the queue depends upon the throughput of the producer. Depending upon event size (storing complete state vs all state changes) <strong><em>E </em></strong>bytes<em>,</em> with Number of Events <strong><em>N</em></strong>, we can calculate the size of the queue for a particular time window.</p><blockquote>Size of Queue (in bytes) <strong><em>S = E X N </em></strong>which is uncompressed.</blockquote><p>Queue Size also depends upon Retention Period <strong><em>R </em></strong>(which may be infinite or a few days).</p><h4>Calculating partitions</h4><p>Kafka is slightly different from traditional message queues, as consumed messages are not immediately deleted. The message will continue to live until the retention period is elapsed, which is configurable for each topic on Kafka. An increasing number of partitions leads to higher throughput.</p><p>A rough formula for picking the number of partitions is based on throughput. Measure the throughput that can be achieved on a single partition (call it <em>p</em>) and consumption (call it <em>c</em>).</p><blockquote>To achieve Target throughput <em>t</em>, at least <em>max(t/p, t/c)</em> partitions are required.</blockquote><p>If we have infinite retention for the topic, other factors like Disk Space also come into the picture.</p><p>Partitions can be increased over time. But as Kafka maps messages to partitions based on a hash of the key, this ensures messages within a partition are always delivered to the consumer. If the number of partitions changes, such a guarantee may no longer hold. To avoid this situation, a common practice is to over-partition by just a bit, as too many partitions may also have a negative impact.</p><h3>Kafka log compaction</h3><p><em>Log compaction is a mechanism to give finer-grained per-record retention, rather than coarser-grained time-based retention. </em>The idea is to selectively remove records where we have a more recent update with the same primary key. This way the log is guaranteed to have at least the last state for each key.</p><p>Kafka automatically removes messages when time or size is reached for a particular topic using a configurable retention policy. However, this behaviour can be changed to <em>key-based</em> retention by enabling Log Compaction. Kafka will remove any old event with a newer version of the event having the same key in the same partition. Compacted topics provide the ‘latest’ view of the entity.</p><figure><img alt="Illustration of events stored in Kafka Topic before compaction. We have two events with the same key K6, K4 stored inside the Kafka topic" src="https://cdn-images-1.medium.com/max/361/1*n9OWA2PNssnPOnzmHdI1xQ.png" /><figcaption>Kafka Topic Before Compaction</figcaption></figure><figure><img alt="Illustration of Kafka topic after compaction. All the events with duplicate event keys e.g. K4 and K6 are removed and only the recent one remains inside the topic." src="https://cdn-images-1.medium.com/max/241/1*g2thCOZTX4rSAWyOe8TU5Q.png" /><figcaption>Kafka Topic After Compaction</figcaption></figure><blockquote><em>“Compacted Kafka topics can be used as KV datastore for storing very large datasets. Kafka is battle-tested for production workloads serving use cases over terabytes and can be used to store the state of an entity similar to a traditional database.”</em></blockquote><h3>Migration strategy</h3><p>The below figure illustrates how we can migrate data from a traditional database and migrate to Event Source architecture in different phases.</p><figure><img alt="Migration Strategy in 4 different phases, Data first being migrated to Kafka using migration job and backfill job. Updating new events back to the database to support backward compatibility" src="https://cdn-images-1.medium.com/max/901/1*XliR9n9JUlTGdxurzRf7Jg.png" /><figcaption>Migration Strategy to Adopt Event Sourcing Architecture</figcaption></figure><p><strong>Figure existing architecture: Application writing data to database</strong></p><p>This represents an existing architecture where the application is writing data to a traditional SQL/NoSQL Database and is considered the source of truth.</p><p><strong>Figure phase II: Historical data migration to Kafka topic</strong></p><p>As the first phase for migration, the application starts writing data to a database with Record Version <strong>R(v)</strong> (which is strictly increasing). A migration job starts migrating data from a traditional database to Kafka Topic. It is also responsible for transforming data to the event, which is a snapshot of the event state having Event Version <strong>E(v)=R(v)</strong>. AWS DMS can also be considered to migrate all the data.</p><p><strong>Figure phase III: streaming latest changes to Kafka topic</strong></p><p>Once all the data is loaded/migrated to Kafka Topic, migration jobs start listening to new changes happening on the database and will stream those events to the Kafka topic. Another backfill job is also started, which writes back events to a database having Event Version <strong>E(v)</strong> <strong>&gt;</strong> <strong>R(v)</strong>. This ensures that older updates don’t override the latest updates written by the application.</p><p><strong>Figure phase IV: backfill for backward compatibility</strong></p><p>The application starts writing events to Kafka, and all the latest updates are written to the database by the backfill job. <strong>E(v)</strong> <strong>&gt;</strong> <strong>R(v) </strong>ensures that we only write the latest information to the database, so the state of the entity is always the latest, it doesn’t change interim to older value, and all the latest updates will eventually be applied to the record state.</p><blockquote><strong>Note: </strong>Not all phases might be required, as this depends on the use cases. A trade-off can be evaluated to reduce phases, depending on the acceptable percentage of data losses.</blockquote><h3>Learnings</h3><h4>Consistency</h4><blockquote>Eventual consistency is a theoretical guarantee that, provided no new updates to an entity are made, all reads of the entity will eventually return the last updated value.</blockquote><p>Achieving eventual consistency is also not a trivial task, and we must be careful that we eventually have consistent information during migration. Record versioning can be used to ensure consistency, and only the most recent updates should be applied to drop events/records with an older version.</p><p><strong>Consistency across multiple data stores</strong></p><p>Eventual Consistency across different data sources can be easily ensured by consuming events from Kafka and writing them to multiple data stores with infinite retries. The addition of a new data store to support different use cases and query patterns is made simple and easy by just consuming all the events from the beginning of the queue.</p><h4>Idempotency</h4><p>During migration and ensuring backward compatibility, we also must make sure that all operations are idempotent — there is no impact if the operation is executed more than once. Again, using version-based conflict resolution and storing the full object state to history, we can restore the final state by correctly reordering the object states and saving only the delta in persistent storage. The only restriction is that the client should use some sort of concurrency, which should be manageable by sharing object updates to a single partition.</p><h4>Scaling</h4><p>As discussed, in capacity Planning depending on producer traffic and consumer consumption rate, we can easily identify the right number of partitions. The application can be scaled to the extent that there is almost zero lag during the streaming of events.</p><h3>Conclusion</h3><p>Treating events as the primary source of truth not only provides decoupling but also supports extensibility for future use cases. This also allows organisations to end up with loosely coupled application architecture that is both resilient and scalable.</p><p><strong>Acknowledgements</strong></p><p>Special thanks to <a href="https://medium.com/@lucapelosi">Lucas Pelosi</a> and <a href="https://medium.com/@mitul.kapoor21">Mitul Kapoor</a> for their reviews and comments.</p><blockquote><a href="https://lifeatexpediagroup.com/technology?utm_source=techblog&amp;utm_medium=career&amp;utm_campaign=postfooter">Learn more about technology at Expedia Group</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3df616cccd8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/how-expedia-reviews-engineering-is-using-event-streams-as-a-source-of-truth-d3df616cccd8">How Expedia Reviews Engineering Is Using Event Streams as a Source Of Truth</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Onion Architecture]]></title>
            <link>https://medium.com/expedia-group-tech/onion-architecture-deed8a554423?source=rss-5e168bb045d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/deed8a554423</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[domain-driven-design]]></category>
            <category><![CDATA[design-principles]]></category>
            <dc:creator><![CDATA[Ritesh Kapoor]]></dc:creator>
            <pubDate>Tue, 12 Jul 2022 13:03:02 GMT</pubDate>
            <atom:updated>2023-01-06T17:35:53.278Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="http://medium.com/expedia-group-tech/engineering/home">EXPEDIA GROUP TECHNOLOGY —ENGINEERING</a></h4><h4>Let’s slice it like a Pro</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8vCPV9ltRgTN2kW-dMrokg.png" /></figure><blockquote><em>Domain-driven design (DDD) is an approach to developing software for complex needs by deeply connecting the implementation to an evolving model of the core business concepts.</em></blockquote><p>The domain is a sphere of knowledge. It refers to the business knowledge that our software is trying to model. Domain-Driven Design centres on the domain model that has a rich understanding of the processes and rules of a domain. Onion architecture implements this concept and dramatically increases code quality, reduces complexity and enables evolutionary enterprise systems.</p><figure><img alt="Onion Architecture illustrating different layers. Domain Model at the centre, enclosed by Domain Services. Domain Services enclosed by Application services and then Infrastructure Services. Application enclosing all the layers. Observability services for monitoring the application" src="https://cdn-images-1.medium.com/max/481/1*Pd6l7Za_Pl8BI-kltDQukg.png" /><figcaption>Onion Architecture</figcaption></figure><h3>Why Onion Architecture?</h3><p>Domain entities are the core and centre part. Onion architecture is built on a domain model in which layers are connected through interfaces. The idea is to keep external dependencies as far outward as possible where domain entities and business rules form the core part of the architecture.</p><ul><li>It provides flexible, sustainable and portable architecture.</li><li>Layers are not tightly coupled and have a separation of concerns.</li><li>It provides better maintainability as all the code depends on deeper layers or the centre.</li><li>Improves overall code testability as unit tests can be created for separate layers without impacting other modules.</li><li>Frameworks/technologies can be easily changed without impacting the core domain. e.g. RabbitMQ can be replaced by ActiveMQ, SQL can be replaced by MongoDB</li></ul><h3>Principles</h3><p>Onion Architecture is comprised of multiple concentric layers interfacing with each other towards the core that represents the domain. It is based on the inversion of control principle. The architecture does not focus on underlying technology or frameworks but the actual domain models. It is based on the following principles.</p><h4>Dependency</h4><p>The circles represent different layers of responsibility. In general, the deeper we dive, the closer we get to the domain and business rules. The outer circles represent mechanisms and the inner circles represent core domain logic. The outer layers depend on inner layers and the inner layers are completely unaware of outer circles. Classes, methods, variables, and source code in general belonging to the outer circle depends on the inner circle but not vice versa.</p><p>Data formats/structures may vary from layers. Outer layer data formats should not be used by inner layers. E.g. Data formats used in an API can vary from those used in a DB for persistence. Data flow can use data transfer objects. Whenever data crosses layers/boundaries, it should be in a form that is convenient for that layer. E.g. API’s can have DTO’s, DB layer can have Entity Objects depending on how objects stored in a database vary from the domain model.</p><h4>Data encapsulation</h4><p>Each layer/circle encapsulates or hides internal implementation details and exposes an interface to the outer layer. All layers also need to provide information that is conveniently consumed by inner layers. The goal is to <strong>minimize coupling between layers and maximize coupling within a vertical slice across layers. </strong>We define abstract interfaces at deeper layers and provide their concrete implementation at the outermost layer. This ensures we focus on the domain model without worrying too much about implementation details. We can also use dependency injection frameworks, like Spring, to connect interfaces with implementation at runtime. E.g. Repositories used in the domain and external services used in Application Services are implemented at the infrastructure layer.</p><figure><img alt="Concentric circles depicting an external service interface exposed to the Application layer and implemented at the Infrastructure layer. Also shows a Repository interface exposed to Domain Model and implemented at Infrastructure layer." src="https://cdn-images-1.medium.com/max/351/1*dGhChg8YsXcdx2J93CxxVg.png" /><figcaption>Data Encapsulation in Onion Architecture</figcaption></figure><h4>Separation of concerns</h4><p>Application is divided into layers where each layer has a set of responsibilities and addresses separate concerns. Each layer acts as modules/package/namespace within the application.</p><h4>Coupling</h4><p>Low coupling in which one module interacts with another module and does not need to be concerned with the other module’s internals. All the internal layers need not be concerned about internal implementation of external layers.</p><h3>Onion Architecture Layers</h3><p>Let’s understand different layers of the architecture and their responsibilities with an order creation use case. When receiving a create order request, we would like to validate the order, save the order in the database, update inventory for all order items, debit order amount and lastly send a notification to the customer about order completion.</p><figure><img alt="Package Diagram Illustrating Dependency across Layers. Application Services depends on Domain Services. Infrastructure services depending on Domain and Application Services." src="https://cdn-images-1.medium.com/max/943/1*mO7KcMMR0WdDcTEXnlX8Xg.png" /><figcaption>Package Diagram Illustrating Dependency across Layers</figcaption></figure><figure><img alt="Example Interactions Across Different Layers. Infrastructure Services like GRPC Server calling Application Services for Creating Order and orchestrating order creation across different Domain Services. Domain Services could be storing information in database, providing Interface for Inventory or Notification Service." src="https://cdn-images-1.medium.com/max/1024/1*AiplnhYu4uF9-kh2RStQSg.png" /></figure><h4>Domain Model/Entities</h4><p>Domain Entities are the fundamental building block of Domain-Driven Design and they’re used to model concepts of your Ubiquitous Language in code. Entities are Domain concepts that have a unique identity in the problem domain. Domain entities encapsulate attributes and entity behaviour. It is supposed to be independent of specific technologies like databases or web APIs. E.g. In the Orders domain. Order is an entity and has attributes like OrderId, Address, UserInfo, OrderItems, PricingInfo and behaviour like AddOrderItems, GetPricingInfo, ValidateOrder, etc.</p><figure><img alt="Order Entity Class Example illustrating Entity’s Data and Behaviour" src="https://cdn-images-1.medium.com/max/331/1*1-8HcYXPgh7X5DERL4ilfw.png" /><figcaption>Order Entity Class</figcaption></figure><h4>Domain services</h4><p>Domain services are responsible for holding domain logic and business rules. All the business logic should be implemented as a part of domain services. Domain services are orchestrated by application services to serve business use-case. They are NOT typically CRUD services and are usually standalone services. Domain services are responsible for complex business rules like computing pricing and tax information when processing order, Order repository interface for saving and updating order, Inventory Interface for updating information about items purchased, etc.</p><p><em>It consists of algorithms that are essential to its purpose and implement the use cases that are the heart of the application.</em></p><h4>Application services</h4><p>Application services also referred to as “Use Cases”, are services responsible for just orchestrating steps for requests and should not have any business logic. Application Services interact with other services to fulfil the client’s request. Let’s consider the use case to create an order with a list of items. We first need to calculate the price including tax computation/discounts, etc., save order items and send order confirmation notification to the customer. Pricing computation should be part of the domain service, but orchestration involving pricing computation, checking availability, saving order and notifying users should be part of the application service. The application services can be only invoked by Infrastructure services.</p><h4>Infrastructure services</h4><p>Infrastructure services also referred to as Infrastructure adapters are the outermost layer in onion architecture. These services are responsible for interacting with the external world and do not solve any domain problem. These services just communicate with external resources and don’t have any logic. E.g. External notification Service, GRPC Server endpoint, Kafka event stream adapter, database adapters.</p><h4>Observability services</h4><p>Observability services are responsible for monitoring the application. These services help perform tasks like :</p><ul><li>Data collection (metrics, logs, traces) — use mainly libraries/sidecars to collect various data during code execution.</li><li>Data storage — use tools that enable central storage of the collected data (sorting, indexing, etc.)</li><li>Visualisation — use tools that allow you to visualise the collected data.</li></ul><p>Few examples include Splunk, ELK, Grafana, Graphite, Datadog.</p><h3>Testing Strategy</h3><p>Different layers of onion architecture have a different set of responsibilities and accordingly, there are different testing strategies. The testing pyramid is a great framework that lays out the different types of tests. Business rules that belong to the domain model, domain services and application services should be tested via Unit Testing. As we move to the outer layer, it makes more sense to have integration tests in infrastructure services. For our application End to End testing and BDD are the most appropriate testing strategies.</p><figure><img alt="Concentric circles labeling testing practices by layer. Unit Testing for Domain Model, Domain Services and Application Services. Integration Testing for Infrastructure Services and End to End Testing for Application." src="https://cdn-images-1.medium.com/max/543/1*7vdYUJ0cP8WzuCAxcoLT4g.png" /><figcaption>Testing Strategy For Different Layers</figcaption></figure><h3>Microservices</h3><p>Onion architecture is also applicable to microservices when viewing each microservice in isolation. Each microservice has its own model, its own use cases and defines its own external interfaces for retrieving or modifying the data. These interfaces can be implemented with an adapter that connects to another microservice by exposing HTTP Rest, GRPC, Thrift Endpoints, etc. It’s a good fit for microservices, where data access layer not only comprises database, but also for example an http client, to get data from another microservice, or even from an external system.</p><h3>Application Structure &amp; Layers</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee132e1e64d08e21789be6eac1eb798d/href">https://medium.com/media/ee132e1e64d08e21789be6eac1eb798d/href</a></iframe><h3>Modularisation vs Packaging</h3><p>There are two ways to organise application source code:</p><ul><li>Either, we can have all the packages in a single module/project or</li><li>Divide the application into different modules/projects each responsible for a layer in onion architecture.</li></ul><p>It greatly depends on the complexity of the application and the size of the project to divide source code into multiple modules. In a microservice architecture, modularisation may or may not make sense depending upon the complexity and use-case.</p><h3>Frameworks, Clients and Drivers</h3><p>The infrastructure layer composes frameworks for web or servers, clients for databases, queues or external services. It is responsible for configuring and stitching all the external services and frameworks together. Onion architecture provides decoupling so that it becomes easier to swap technologies at any point in time.</p><h3>Do We Need Every Layer?</h3><p>Organising our application in layers helps in achieving separation of concerns. But do we need all the layers? Maybe, maybe not. It depends on the use cases and the complexity of the application. It is also possible to create more layers of abstractions depending on application needs. E.g. for smaller applications that don’t have a lot of business logic, it might not make sense to have domain services. Regardless of layers, dependencies should always be from outer layers to inner layers.</p><h3>Conclusion</h3><p>Onion architecture might seem hard in beginning but is <a href="https://jeffreypalermo.com/2013/08/onion-architecture-part-4-after-four-years/">widely accepted in the industry</a>. It is a powerful architecture and enables easy evolution of software. By separating the application into layers, the system becomes more testable, maintainable and portable. It helps easy adoption of new frameworks/technologies when old frameworks become obsolete. Similar to other architectural styles like Hexagonal, Layered, Clean Architecture, etc. it provides a solution for common problems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=deed8a554423" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/onion-architecture-deed8a554423">Onion Architecture</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Bidirectional Dijkstra’s Algorithm]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://ritesh-kapoor.medium.com/understanding-bidirectional-dijkstras-algorithm-db3511145b64?source=rss-5e168bb045d9------2"><img src="https://cdn-images-1.medium.com/max/2600/0*wrwFbZn54tPD-QoD" width="5645"></a></p><p class="medium-feed-snippet">Dijkstra&#x2019;s algorithm is well known shortest path finding algorithm in a graph. But there exists Bidirectional Dijkstra algorithm as well&#x2026;</p><p class="medium-feed-link"><a href="https://ritesh-kapoor.medium.com/understanding-bidirectional-dijkstras-algorithm-db3511145b64?source=rss-5e168bb045d9------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://ritesh-kapoor.medium.com/understanding-bidirectional-dijkstras-algorithm-db3511145b64?source=rss-5e168bb045d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/db3511145b64</guid>
            <category><![CDATA[dijkstra-implementation]]></category>
            <category><![CDATA[bidirectional-dijkstra]]></category>
            <category><![CDATA[dijkstras-algorithm]]></category>
            <category><![CDATA[shortest-path]]></category>
            <dc:creator><![CDATA[Ritesh Kapoor]]></dc:creator>
            <pubDate>Tue, 18 Jan 2022 13:47:38 GMT</pubDate>
            <atom:updated>2022-01-18T14:09:12.496Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Testing Automation, What are Pyramids and Diamonds?]]></title>
            <link>https://ritesh-kapoor.medium.com/testing-automation-what-are-pyramids-and-diamonds-67494fec7c55?source=rss-5e168bb045d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/67494fec7c55</guid>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[testing-strategies]]></category>
            <category><![CDATA[testing-micoservices]]></category>
            <category><![CDATA[testing-pyramid]]></category>
            <dc:creator><![CDATA[Ritesh Kapoor]]></dc:creator>
            <pubDate>Sat, 01 Jan 2022 13:05:33 GMT</pubDate>
            <atom:updated>2022-05-16T09:07:57.640Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j3f7nO44rWrPRyFS7mLsEw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@eugene_tkachenko?utm_source=medium&amp;utm_medium=referral">Eugene Tkachenko</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Testing pyramids (which are also referred to as Test Automation pyramids) is a framework that lays out the different types of tests that should be included in automated test suites. Producing high-quality software and reducing the time required to identify breaking change is not a trivial task. The testing strategy is a very important factor in deciding the agility of your team and project.</p><h3>Why Testing Pyramids are important?</h3><ul><li><strong>Quality of the product: </strong>A product free of defects and issues.</li><li><strong>The velocity of the team:</strong> How fast a feature could be delivered.</li><li><strong>Release predictability</strong>: The ability to plan and deliver with predictability.</li></ul><h3>What are different Types of Pyramids?</h3><h4>Testing Pyramid</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*F26nbm1Nk-5HB1_GtDYc4Q.png" /></figure><p>You might have probably seen the test automation pyramid something like above by now. Mike Cohn came up with this concept in his book <em>Succeeding with Agile</em>. It tells us about different testing layers and how much testing is required to be done on each layer. The different Layers are:</p><ol><li><strong>Manual Testing: </strong>We should have a test suite strong enough to avoid any manual testing which is required as a prerequisite before taking it to production. But in the practical world, there are times where we cannot completely avoid manual testing due to time or delivery constraints. We should introduce tests as early as possible to maintain hygiene. If it is practice, then it highlights something is wrong with how we are testing.</li><li><strong>End To End Testing:</strong> Automated tests, which are responsible for testing applications workflow from beginning to end for the entire software. They focus on testing user stories or an end to end functionality that simulates real-world scenarios. In a microservices architecture, it can span across different services to test a functionality End To End.</li><li><strong>Integration Testing: </strong>It takes different software modules that are integrated logically and test them as a group. In microservices architecture. It can also refer to testing service in isolation.</li><li><strong>Unit Testing: </strong>Unit tests take a small piece of the product and test that piece in isolation.</li></ol><h4>Inverted Test Pyramid</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*5n5fMwgwCJQY8oqAyiy_1g.png" /></figure><p>The Inverted test pyramid which is also referred to as Ice cream cone pyramid is another strategy in which a lot of focus is made towards Manual/E2E Testing and is least concerned about Unit Testing. The cost of testing is too high and are extremely painful and inefficient. It impacts agility and is sometimes considered as an Anti Pattern that should be avoided.</p><p>But, it works well when you are developing application prototypes or working on Proof of Concept, where the focus is more on developing a functionality rather than investing time in maintaining Test Suite. Prototypes eventually phase out to stable products.</p><h4>Diamond</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*3EHuHJVj31kk0oBaM4xg5A.png" /></figure><p>We all know that writing and maintaining a test suite is not cheap. Like source code, it should be carefully written and maintained. In the microservice world, practically speaking Integration Tests could hold more value than Unit tests. Testing how an application behaves when it interacts with other services provides more confidence and reliability on the test suite. Integration test does not mean to test application using actual service, rather mocks should represent external service so that tests run in isolation.</p><p>End to End Tests is still hard to develop and maintain and should be avoided.</p><h3>How to apply them in projects?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7zRw3ulqJ6PfxFsWvCVMdQ.png" /></figure><h3>Comparison on different Metrics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*nIiVSvvOv95JCKx9G3mfBQ.png" /></figure><h3>Final Thoughts</h3><p>The true value of the test strategy is to let the developer know that there is some problem with code changes, the issue lies in what area and how fast we can get to know about the problem. There is no right or wrong strategy, It’s all about what works for you.</p><p>Please share your thoughts and experiences.</p><h3>References</h3><ul><li><a href="https://www.linkedin.com/pulse/inverting-test-pyramid-joel-masset/">https://www.linkedin.com/pulse/inverting-test-pyramid-joel-masset/</a></li><li><a href="https://martinfowler.com/bliki/TestPyramid.html">https://martinfowler.com/bliki/TestPyramid.html</a></li><li><a href="https://martinfowler.com/articles/practical-test-pyramid.html">https://martinfowler.com/articles/practical-test-pyramid.html</a></li><li><a href="https://eason.blog/posts/2020/03/test-automation-diamond/">https://eason.blog/posts/2020/03/test-automation-diamond/</a></li><li><a href="https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html">https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html</a></li><li><a href="https://engineering.atspotify.com/2018/01/11/testing-of-microservices/">https://engineering.atspotify.com/2018/01/11/testing-of-microservices/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67494fec7c55" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PACELC Theorem & Distributed Databases]]></title>
            <link>https://ritesh-kapoor.medium.com/pacelc-theorem-and-distributed-databases-301d971deda3?source=rss-5e168bb045d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/301d971deda3</guid>
            <category><![CDATA[distributed-database]]></category>
            <category><![CDATA[cap-theorem]]></category>
            <category><![CDATA[pacelc]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Ritesh Kapoor]]></dc:creator>
            <pubDate>Thu, 23 Dec 2021 15:25:12 GMT</pubDate>
            <atom:updated>2022-01-17T06:33:01.372Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article we are going to cover <strong>PACELC</strong> Theorem which is an extension for CAP Theorem and see how how distributed databases have limitations and tradeoffs regarding Network Partition, Consistency, Availability and Latency.</p><h3>What is P.A.C.E.L.C. ?</h3><h4>Consistency</h4><p>Consistency means that all clients see the same data at the same time, regardless of node they are connected to.</p><h4>Availability</h4><p>Availability means that clients get successful response, even if nodes are down in cluster. Though the response maybe inconsistent with previous write.</p><h4>Partition Tolerance</h4><p>Partition Tolerance means that in case there is communication break within cluster (Broken Network, Delayed message transmission) The cluster is able to operate regardless of where partition happens.</p><h4>Latency</h4><p>Latency is the time taken when client initiated the request till it receives response which includes both processing time and network transmission delay.</p><h3><strong>PACELC</strong> Theorem States That :</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*yjoH_K9j7eqHqB5XccEciA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*9ZcyAUQPRURs421l1xG4kQ.png" /></figure><h3>CAP Theorem Trade-Offs</h3><p>Lets apply CAP Theorem in Master Slave Replication cluster setup for simplicity. Also, let’s imagine that network partition happens at master node in a way that it cannot communicate with rest of the nodes in cluster and with clients or master node goes down. Let’s visualize this in diagram below.</p><blockquote><em>We can also see that CAP theorem is applicable for other scenarios including but not limited to:</em></blockquote><blockquote><em>* Multi Master Replication cluster setup</em></blockquote><blockquote><em>* Cluster with more than three nodes in Master Slave Replication</em></blockquote><blockquote><em>* Sharded databases like Cassandra/Dynamodb where Key Range is owned by more than one node.</em></blockquote><blockquote><em>* Network Partitioning happening across clients and cluster nodes where they are isolated from others.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qqMbXGROp3xJV_9FD_jvow.png" /></figure><h4>Consistency and Availability Trade-Off</h4><p>As we can see that when master node is down, Client 1 has few options to retrieve information</p><p><strong>Consistency: </strong>To Achieve consistency clients can:</p><ul><li>Connect to master and the request fails.</li><li>It waits/retires till master node is up and running, which again violates availability.</li></ul><p><strong>Availability: </strong>Client can connect to other replica and access stale or inconsistent information.</p><p>This is the trade-off which client needs to make and as described through CAP Theorem, client can either choose Consistency or Availability in case of Network Partitions.</p><h3>PACELC as Extension to CAP Theorem</h3><p>Again, Let’s take previous example with Master Slave Cluster Setup. This time lets assume that the Network Partition doesn’t happen and master node is up and running. But after certain point in time its is overwhelmed by clients request and its performance degrades.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ihyLsCV9jBFS0Wv32aWJeA.png" /></figure><h4>The Latency Consistency Trade-Off</h4><p>Client 1 has few following options to retrieve information and this is the trade-off which client needs to make and as described through PACELC Theorem.</p><p><strong>Consistency: </strong>To achieve consistent information client still requests Master node but receives slow response or high latency.</p><p><strong>Latency: </strong>Client can connect to other replica and accesses stale or inconsistent information and receives response with low latency.</p><p>Off-course, PACELC Theorem is applicable to different scenarios where clients or nodes within cluster need to make a tradeoff between latency and consistency when Network Partition doesn’t happens.</p><h3>Applying PACELC Theorem to Databases</h3><ul><li>The default versions of DynamoDB, Cassandra, and Cosmos DB are PA/EL systems: if a partition occurs, they give up consistency for availability, and under normal operation they give up consistency for lower latency.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*Wl3oNerV-1p5v157lFl9RQ.png" /><figcaption><strong>Cassandra</strong> <strong><em>(PA/E</em></strong><em>lse </em><strong><em>L</em></strong><em>atency</em><strong><em>)</em></strong></figcaption></figure><ul><li>MongoDB can be classified as a PA/EC system. In the baseline case, the system guarantees reads and writes to be consistent.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*XJc_-BQ8HJ75TC5ES5YzzQ.png" /><figcaption><strong>MongoDB</strong> <strong><em>(</em></strong>PA/<strong><em>E</em></strong><em>lse </em><strong><em>C</em></strong><em>onsistency</em><strong><em>)</em></strong></figcaption></figure><ul><li>Fully ACID systems such as VoltDB/H-Store, Megastore and MySQL Cluster are PA/EC: they refuse to give up consistency, and will pay the availability and latency costs to achieve it. BigTable and related systems such as HBase are also PC/EC.</li></ul><p>Databases these days offer a way to configure these tradeoffs between Availability, Consistency and Lower Latency.</p><h3>Conclusion</h3><p><em>“Ignoring the consistency/latency trade-off of replicated systems is a major oversight [in CAP]” </em>is one important aspect addressed by PACELC Theorem in distributed database systems.</p><p><strong>Daniel J. Abadi</strong> first described PACELC theorem around 2010 in his blog and then in paper in 2012. Theorems help us carefully design platform and analyse tradeoff which everyone needs to make while architecting distributed solutions to solve real world problems.</p><h3>Coming Up Next Part-2</h3><p>Hope you enjoyed the article. Stay tuned for the Part-2 and any suggestions are most welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=301d971deda3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>