<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zehra Gül Çabuk Keskin on Medium]]></title>
        <description><![CDATA[Stories by Zehra Gül Çabuk Keskin on Medium]]></description>
        <link>https://medium.com/@gul.cabuk?source=rss-79dab65197b0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0rabvJ2cDE-KydaQEEvswA.jpeg</url>
            <title>Stories by Zehra Gül Çabuk Keskin on Medium</title>
            <link>https://medium.com/@gul.cabuk?source=rss-79dab65197b0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:22:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gul.cabuk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why should be careful about using Redisson live object service?]]></title>
            <link>https://blog.devgenius.io/why-should-be-careful-about-using-redisson-live-object-service-32fc1a3afdeb?source=rss-79dab65197b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/32fc1a3afdeb</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[performance-improvement]]></category>
            <category><![CDATA[locust]]></category>
            <category><![CDATA[redisson]]></category>
            <category><![CDATA[new-relic]]></category>
            <dc:creator><![CDATA[Zehra Gül Çabuk Keskin]]></dc:creator>
            <pubDate>Wed, 29 Jun 2022 20:25:03 GMT</pubDate>
            <atom:updated>2022-07-09T14:31:47.251Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VicBvC5CdsvJlvSv" /><figcaption>Photo by <a href="https://unsplash.com/es/@jdent?utm_source=medium&amp;utm_medium=referral">Jason Dent</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>First, I would like to start by explaining why I wrote this headline. We had the opportunity to use some features of the <a href="https://redisson.org/">Redisson</a> library at different points in our code base for a while. Until now it has reduced our development effort by giving us an easy-to-use library for Redis in situations where we needed to develop features quickly. Not long ago, the question came to our minds: Can we use Redisson to reduce the load on the database at a very heavy and performance-critical point?</p><p>To test our theory, we wanted to try it quickly and see the results. After implementation, we observed the system&#39;s behavior under load. I can say that this time <strong>Redisson couldn’t live up to our expectations</strong>. Before sharing <a href="https://locust.io/">Locust</a> load tests results and the <a href="https://newrelic.com/">New Relic</a> APM dashboards, let me give some details about Redisson.</p><p><a href="https://github.com/redisson/redisson/wiki/1.-Overview">Redisson</a> is a Redis Java client with features of In-Memory Data Grid[<a href="https://github.com/redisson/redisson/wiki/Table-of-Content">1</a>] It has distributed objects, collections, locks, synchronizers, and services. One of our favorite features of Redisson is the <strong>live object service </strong>from distributed services. A Live Redisson Object (LRO) is an instance reference that can be shared not only between threads in a single JVM but can also be <strong>shared between different JVMs across different machines</strong>. [<a href="https://github.com/redisson/redisson/wiki/9.-distributed-services/#921-introduction">2</a>]</p><p>The most meaningful and powerful aspect of <strong>Redisson live objects</strong> is that it works like a search engine and allows querying on multiple fields. The idea of being able to <strong>forward a database query that receives hundreds of requests per second to Redis</strong> and <strong>reducing the database load</strong> was very exciting for us. It is easy to create a cache model and make it queryable via Live Object Service to query an object by multiple fields, it is only needed to add <a href="http://twitter.com/RIndex">@RIndex</a> annotation to the necessary fields. This is a sample MarketCalendar live object which can be queried on all of its fields.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14138ead1307eb8a7e5dfd60a67fcfe5/href">https://medium.com/media/14138ead1307eb8a7e5dfd60a67fcfe5/href</a></iframe><p>After preparing the cache model, it’s time to write a query that finds today’s market calendar. You can find one example query below. We used a combination<strong>(Conditions.and)</strong> of the greater than<strong>(Conditions.gt)</strong> and equal<strong>(Conditions.eq)</strong> conditions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f21899d52a4ad44faadfcc2c09719021/href">https://medium.com/media/f21899d52a4ad44faadfcc2c09719021/href</a></iframe><p>There are the different types of search conditions like that: Conditions.and,Conditions.eq,Conditions.or,Conditions.in, Conditions.gt,Conditions.ge,Conditions.lt,Conditions.le</p><p>See more: <a href="https://github.com/redisson/redisson/wiki/9.-distributed-services#923-search-by-object-properties">https://github.com/redisson/redisson/wiki/9.-distributed-services#923-search-by-object-properties</a></p><p>After implementing a basic cache model and queries, we performed a load test on an endpoint that uses today’s market calendar. This is one of the load test results on a resource that has 1 pod, 2000m CPU, and 2048Mi (2.147484GB — A mebibyte equals 1.04858MB) on <a href="https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/">Kubernetes</a>. The load was 10 Workers, and 100 Users with a 10 spawn rate. The test needs 2 different select queries on market calendars to retrieve the endpoint result. You can find an example locust task below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9579ed132b9c0faf81148d47be64059d/href">https://medium.com/media/9579ed132b9c0faf81148d47be64059d/href</a></iframe><p>See more detailed locust example:<br><a href="https://performancelabus.com/load-testing-using-locust/">https://performancelabus.com/load-testing-using-locust/</a></p><p>After execution of a load test on our dev environment, it has become clear that our heavy-duty services cannot survive with the Redisson live object service. After these results, we started to search for an alternative solution. We thought that maybe an in-memory implementation might be the proper solution for this situation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lvO5im95H4jZQpHVOJM7MA.png" /><figcaption>Load Test Results on Test Environment</figcaption></figure><p>While developing an in-memory solution, we kept the Redisson solution active for a while in the development environment. In this way, we had a chance to compare the results with the in-memory method. After in-memory implementation, we were able to compare in-memory and Redisson methods by using <a href="https://newrelic.com/"><strong>New Relic</strong></a> in the development environment.</p><p>In the New Relic dashboards below, the green lines you see as “previous avg duration” is our average response times when the Redisson implementation is active. On the other hand, blue lines show the in-memory implementation which seems more appropriate for our problem here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zGoGg4RSN-zpZtyzTFjTvQ.png" /></figure><p>Our findings of the Redisson Live Object Service are as follows:</p><ul><li>Query times can <strong>get slower</strong> as the <strong>number of elements on the indexed list increases</strong>.</li><li>If you bring a large number of live objects to memory as a result of your query and do a sorting etc. on these objects; the query time might be even slower because live objects work as a proxy, and <strong>getter methods of fields retrieve the last value of the field by asking Redis, every single time</strong>.</li><li>You can get rid of this situation using <strong>detach </strong>on live object service, but it still does not provide a permanent and acceptable response time.</li></ul><p><strong>Note</strong>: I also would like to inform you about a weird bug that we encountered during tests. We observed that there is a possibility to get NullPointerException during any of the getters so the retrieve method should be protected with a Retryable annotation to prevent NPEs.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7b63d5e2fd94e666d784bea9e765677/href">https://medium.com/media/d7b63d5e2fd94e666d784bea9e765677/href</a></iframe><p>We used <a href="https://github.com/redisson/redisson/tree/master/redisson-spring-boot-starter"><strong>redisson-spring-boot-starter</strong></a><strong> </strong>and what we reported today is the results of this library. There is also <a href="https://redisson.pro/">Redisson PRO</a> library defined as ultra-fast. (I should have been suspicious when I read this sentence on their official documentation)</p><p>Our experimental work ended up with giving up the Redisson Live Object service for this problem. On the other hand, for example, we had no problems when using the <a href="https://redisson.org/articles/redis-based-java-time-series-collection.html">time series collection</a>. So, our advice can be is that<strong> not every library or part of a library is suitable for every problem and use case, so it’s very important to make sure that the method we choose is correct with load testing and observation.</strong></p><p>I hope it was a useful post. We will continue to share our experiences.</p><p><strong>References</strong></p><p>[1] <a href="https://github.com/redisson/redisson/wiki/Table-of-Content">https://github.com/redisson/redisson/wiki/Table-of-Content</a></p><p>[2] <a href="https://github.com/redisson/redisson/wiki/9.-distributed-services/#921-introduction">https://github.com/redisson/redisson/wiki/9.-distributed-services/#921-introduction</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32fc1a3afdeb" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/why-should-be-careful-about-using-redisson-live-object-service-32fc1a3afdeb">Why should be careful about using Redisson live object service?</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to test and monitor your circuit breaker implementation?]]></title>
            <link>https://engineering.getmidas.com/how-to-test-and-monitor-your-circuit-breaker-implementation-d6c365f2fcdd?source=rss-79dab65197b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6c365f2fcdd</guid>
            <category><![CDATA[spring-boot-admin]]></category>
            <category><![CDATA[resilience4j]]></category>
            <category><![CDATA[integration-testing]]></category>
            <category><![CDATA[spring-boot-actuator]]></category>
            <category><![CDATA[circuit-breaker]]></category>
            <dc:creator><![CDATA[Zehra Gül Çabuk Keskin]]></dc:creator>
            <pubDate>Sat, 18 Jun 2022 15:19:29 GMT</pubDate>
            <atom:updated>2022-06-18T21:07:51.864Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kQyecTQXS1HddFXu" /><figcaption>Photo by <a href="https://unsplash.com/@tobiastu?utm_source=medium&amp;utm_medium=referral">Tobias Tullius</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In my previous blog post, I went over why the circuit breaker pattern is important and what are the necessary steps to implement a circuit breaker for a spring boot project. I recommend you read it before this post from <a href="https://engineering.getmidas.com/a-circuit-breaker-story-how-to-save-your-valuable-services-from-burning-bf9064084ba4"><strong>here</strong></a>.</p><p>Today, I am going to explain how you can test and monitor your circuit breaker by going through the same use case study in the previous blog post.</p><h3>Test: Portfolio-API Retrieve Account Summary</h3><p>In order to test our circuit breaker implementation, <strong>the first thing</strong> we need is <strong>a component that acts as a server with a slow response time</strong>. We can achieve this by using <a href="https://wiremock.org/"><strong>wiremock</strong></a><strong>. </strong>Wiremock is a tool for creating mock APIs. It is easy to isolate yourself from flakey 3rd party APIs by using it[<a href="https://wiremock.org/">1</a>]. It allows simulating cases such as errors or slow response time when you need to test your flow end-to-end.</p><p>We stated that we need a server that acts like a third-party server that responds with a slow response time. For our own example, we can start an external account mock server with <strong>WireMockServer</strong> and <strong>stub</strong> the behavior of the relevant third-party endpoint as follows.<br><strong>See more</strong>: <a href="https://wiremock.org/docs/getting-started/">https://wiremock.org/docs/getting-started/</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7cfe3a9690e27da908f964ce1b1c6e1/href">https://medium.com/media/d7cfe3a9690e27da908f964ce1b1c6e1/href</a></iframe><p>As we have discussed in the previous post, the circuit breaker implementation makes its decision by monitoring the requests in a sliding window. Therefore, it is not enough to send only one request to our service to observe circuit breaker behavior and consequent calls to our endpoint should be simulated.</p><p>This can be implemented with the <strong><em>RepeatedTest </em></strong><em>annotation</em><strong><em> </em></strong>which is introduced in <a href="https://junit.org/junit5/docs/current/release-notes/index.html"><strong>JUnit 5</strong></a>. It enables us to execute tests more than once. Iterations of the same test could be configured differently using RepetitionInfowhich contains the number of total repetitions and the current repetition. I’m planning to configure different external service response delays for each iteration. In this way, we can observe every request (which is a part of the sliding window) behaves in a different way.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23493908498cf6040abcb0137a765f64/href">https://medium.com/media/23493908498cf6040abcb0137a765f64/href</a></iframe><p>In a simple java project, you can add the dependency below to add repeated test capability to your project. <strong>See more</strong>: <a href="https://www.baeldung.com/junit-5-repeated-test">https://www.baeldung.com/junit-5-repeated-test</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0339ccc4bf4699f171781aacf6bb1b8b/href">https://medium.com/media/0339ccc4bf4699f171781aacf6bb1b8b/href</a></iframe><p>Thanks to the <strong>spring-boot-starter-test</strong>, it already includes the JUnit 5. So we will continue with the <strong>spring-boot-starter-test </strong>dependency. These are what we need to add to our pom.xml file for both wiremock and repeated tests.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0823e9f233721a708cd19ee8e2d0fb8/href">https://medium.com/media/e0823e9f233721a708cd19ee8e2d0fb8/href</a></iframe><p>We need to configure our circuit breaker with <a href="https://resilience4j.readme.io/">Resilience4j</a> for the test case. In the previous blog post, I shared a configuration but, we need to modify that configuration for our test.</p><p>This is the configuration we will use during our integration test. Our goal is to make some of the requests slow to observe and test the state transitions of the circuit breaker. According to our configuration (because <strong>sliding-window-size:5</strong>), the circuit breaker needs at least 5 requests to be able to make a decision.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51611cdd00112a6bcdb1a2d307543cb4/href">https://medium.com/media/51611cdd00112a6bcdb1a2d307543cb4/href</a></iframe><p>Our circuit breaker test will contain 4 different groups of 5 requests each. I’m aiming to simulate the <strong>CLOSED to OPEN</strong> state transition with the first 2 groups of 5 requests each. After that, using a waiting time we will observe the <strong>OPEN to HALF-OPEN</strong> state transition. The second 2 groups of 5 requests each will show us <strong>HALF-OPEN to CLOSED</strong> state transitions.</p><p>In order to simulate the CLOSED to OPEN state transition, our test starts by mocking an external service that returns a response in 1000 or 2000 milliseconds for the first 2 groups of 5 requests each.</p><p>The first 5 requests will always access the external service since the circuit breaker needs to observe at least 5 requests (<strong>sliding-window-size:5</strong>) before functioning. Since 3 out of 5 requests exceeds the slow call duration threshold (<strong>slow-call-duration-threshold:2 seconds</strong>) and will be marked as slow calls, the sliding window that contains requests 1–5 will have slow-call-rate of 60%, which is greater than or equal to our slow call rate threshold (<strong>slow-call-rate-threshold: 60</strong>). Therefore, the state will transition to <strong>OPEN</strong> after the first 5 requests are completed.</p><p>Since the state is <strong>OPEN</strong>, all of the following 5 requests (the second group) will be <strong>short-circuited</strong> and <strong>the account value will return from the cache</strong>, which is the fallback service, instead of sending a request to the external service for nothing. (The second group doesn’t need to be 5 requests indeed. The current state will be OPEN until the time of <strong>wait-duration-in-open-state</strong> is passed.)</p><p>That’s why in the 11th iteration, we sleep the thread to provide the time to pass (<strong>wait-duration-in-open-state: 5s</strong>) from <strong>OPEN</strong> to <strong>HALF_OPEN</strong> state.</p><p>The second 2 groups will contain different delays. The 12th and 13th delay 3000ms, and the rest of them run with zero delays. Since the allowed request count is 5 in the <strong>HALF_OPEN</strong> state (<strong>permitted-number-of-calls-in-half-open-state: 5</strong>), the circuit breaker will allow the next 5 requests to pass.</p><p>While in the <strong>HALF_OPEN</strong> state(11th to 15th iterations), since only the 12th and 13th requests are slow (<strong>permitted-number-of-calls-in-half-open-state: 5</strong>), the <strong>60% slow call threshold is not exceeded</strong> and the state is decided as <strong>CLOSED</strong> again. Because the state is CLOSED again, in the next 5 requests, we can get a response from the external service with zero delays from the external service.</p><p>We can see assertions for buying power value from external service or cache according to the iteration of the test. It is expected to get buying power from the fallback method(cache) for the 6th,7th,8th,9th, and 10th iterations because the circuit breaker is in an OPEN state.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3465ce316438d545be413a2d44d9204a/href">https://medium.com/media/3465ce316438d545be413a2d44d9204a/href</a></iframe><p>In order to make it clear, I would like to add state transition logs for each iteration. <strong>CircuitBreakerStateMachine.java </strong>class of <strong><em>io.github.resilience4j.circuitbreaker.internal</em></strong> package gives us detailed state transitions. A sample <strong>logback-test.xml </strong>file looks like this to be able to see debug logs of the state machine.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d482f4660b29bab5e5e3cd9385a4188/href">https://medium.com/media/8d482f4660b29bab5e5e3cd9385a4188/href</a></iframe><p>We can see from the logs, that state transitions and events are what we exactly explained in the previous paragraph. See the logs:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba72275b486e0504bb4830d0a1f56f5d/href">https://medium.com/media/ba72275b486e0504bb4830d0a1f56f5d/href</a></iframe><p>In order to understand the logic of the circuit breaker state machine, we wrote a test for a sample config. After this test, it is possible to determine better config values by understanding the circuit breaker state transitions and the parameters in the configuration.</p><h3>Monitor: Portfolio-API Retrieve Account Summary</h3><p>Monitoring can be achieved using <a href="https://docs.spring.io/spring-boot/docs/current/actuator-api/htmlsingle/#metrics">Spring Boot Actuator Metrics</a>. In order to expose metrics, the following lines can be added to the application.yml file. The important exposed endpoint is <strong>metrics</strong> for us.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/acc90f18c112bb1a7c72e9c21326bb84/href">https://medium.com/media/acc90f18c112bb1a7c72e9c21326bb84/href</a></iframe><p>After adding the configuration above, making a GET request to /actuator/metrics/endpoint, as shown in the following curl-based example, will retrieve all available metrics.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/03914b2c3b0b60041e6281ddeaa68fcb/href">https://medium.com/media/03914b2c3b0b60041e6281ddeaa68fcb/href</a></iframe><p>/actuator/metrics/{metric.name} is also available to use.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54521a9a7ff875eff7e69a5ebc4d4177/href">https://medium.com/media/54521a9a7ff875eff7e69a5ebc4d4177/href</a></iframe><p>When metrics endpoints are exposed, <a href="https://codecentric.github.io/spring-boot-admin/current/#_what_is_spring_boot_admin"><strong>Spring Boot Admin</strong></a> allows you to manage and monitor Spring Boot applications. The applications which you would like to monitor can register with Spring Boot Admin Client or can be discovered using Spring Cloud.[<a href="https://codecentric.github.io/spring-boot-admin/current/#_what_is_spring_boot_admin">2</a>]</p><p>After exposing the metrics endpoint, the <strong>Spring Boot Admin &gt; Insights &gt; Metrics</strong> page will give detailed information about the requests count until that time and the status of your circuit breaker. This page gives you insights about your service and configuration can be made by observing these values. This is a screenshot of the Metrics page from a short-term load test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*tq1EFsqaaMKih3UlNVvrSA.png" /></figure><p>You can also use Spring Boot Admin to set the log level (<strong>Spring Boot Admin &gt; Logs</strong>). You can set CircuitBreakerStateMachine logs to DEBUG on runtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W_wsseXWWgAvGJ7s0MBxoQ.png" /></figure><p>And that’s all. Our circuit breaker case study has been completed. It’s time to monitor your circuit breaker metrics on production and tune the configuration according to the use case.</p><h3>References</h3><p>[1] Flexible API mocking for unit, integration, and performance testing<br><a href="https://wiremock.org/">https://wiremock.org/</a></p><p>[2] Spring Boot Admin<br><a href="https://codecentric.github.io/spring-boot-admin/current/#_what_is_spring_boot_admin">https://codecentric.github.io/spring-boot-admin/current/#_what_is_spring_boot_admin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6c365f2fcdd" width="1" height="1" alt=""><hr><p><a href="https://engineering.getmidas.com/how-to-test-and-monitor-your-circuit-breaker-implementation-d6c365f2fcdd">How to test and monitor your circuit breaker implementation?</a> was originally published in <a href="https://engineering.getmidas.com/">Midas Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Circuit Breaker Story — How to save your valuable services from burning?]]></title>
            <link>https://engineering.getmidas.com/a-circuit-breaker-story-how-to-save-your-valuable-services-from-burning-bf9064084ba4?source=rss-79dab65197b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf9064084ba4</guid>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[technical-book]]></category>
            <category><![CDATA[release-it]]></category>
            <category><![CDATA[resilience4j]]></category>
            <category><![CDATA[circuit-breaker]]></category>
            <dc:creator><![CDATA[Zehra Gül Çabuk Keskin]]></dc:creator>
            <pubDate>Sun, 29 May 2022 14:58:56 GMT</pubDate>
            <atom:updated>2022-05-29T14:58:56.562Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Circuit Breaker Story — How to save your valuable services from burning?</h3><p>As Midas engineering team, we have a technical book club. Every Friday, we organize a session moderated by one person, where we discuss the chapter we read that week.</p><p>We have recently completed the book <a href="https://learning.oreilly.com/library/view/release-it-2nd/9781680504552"><strong>Release It!</strong></a>, and while we were reading, we had the opportunity to talk about the issues we experienced, and we determined action items to which we should pay more attention.</p><p>The “<strong>Create Stability</strong>” part of the book focuses on patterns and antipatterns for designing robust systems. The concept which is repeatedly underlined at each step is the importance of implementing a “<strong>circuit breaker</strong>” pattern at integration points.</p><p>An integration point is a connection between services. These services might be internal or external. And all these integration points are there to destroy your system! In Midas, we also have many integration points like most of the technical teams. And they are very critical as they contain possible risks which can occur in any tiny service and can cascade failures to our entire system.</p><blockquote>A cascading failure occurs when cracks jump from one system or layer to another, usually because of insufficiently paranoid integration points [1]</blockquote><p><strong>So how can we be paranoid?</strong></p><p><strong>The Circuit Breaker pattern</strong> is the most recommended <strong>stability pattern</strong> in the book to be paranoid about integration points. I was familiar with the pattern before reading the part because my fintech background had forced me to learn it🙂</p><p>The idea of the pattern is simple. <strong>You should prevent the application from constantly re-sending requests to the service when the service is expected to fail or timeout</strong>. In other words, if you understand that one service probably will not respond to you for a while, you should stop calling it.</p><p>Well, we understood that this is a fail-fast pattern and we should stop calling the broken service for a while. In that case, how can we decide if a service has enough errors and should stop sending requests? Also, how can we know if the service we call starts responding successfully after a certain time?</p><p>A circuit breaker pattern is a state machine that allows us to get answers to our questions. Circuit Breaker includes 3 states basically. These states are similar to the states of an electrical circuit.</p><p><strong>Closed state</strong>: When everything is normal, the circuit is closed and requests continue to be sent. This is the state your services mostly be in. (hopefully🙂)</p><p><strong>Open state</strong>: When there are errors (slow or fail), this state is passed according to predetermined criteria. When this state is decided, all requests are now directly blocked from going to the external service. With a specific time limit or criteria, it is passed from this state to the half-open state.</p><p><strong>Half-open state</strong>: A batch of requests is allowed to pass to check whether the external service is back to normal, and if successful, a closed state is passed. If it still receives an error, it is passed to the open state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*u5QJ3R2_JOJZD93QB50DuA.png" /><figcaption>Circuit Breaker State Machine</figcaption></figure><p>The states are determined by recording the results of requests sent to other microservices or external services. For example, if 80 out of 100 requests are unsuccessful or timeout, most likely the next request will have the same result, so the state is updated as OPEN (stops sending requests) and the called server has time to repair itself. In addition, this fail-fast approach on the requesting side prevents unnecessary filling of the thread pool because the threads won’t waste their time waiting for a response from a request that will already fail.</p><p>We’ve understood the concept so far, now it’s time to roll up our sleeves. Today I’m going to implement the pattern for a <a href="https://spring.io/projects/spring-boot">Spring Boot</a> project by using one of the circuit breaker libraries.</p><p>Previously, in Spring projects, <a href="https://github.com/Netflix/Hystrix"><strong>Netflix’s Hystrix</strong> library</a> (Latency and Fault Tolerance for Distributed Systems) was our favorite to implement circuit breaker patterns because of Netflix&#39;s support and widely-use. However, Netflix’s switching to maintenance mode with Hystrix 1.5.18 version (2018, November) and not making any further development led us to alternative libraries. <strong>See</strong>: <a href="https://github.com/Netflix/Hystrix#hystrix-status">https://github.com/Netflix/Hystrix#hystrix-status</a></p><h3>Resilience4j</h3><p><a href="https://resilience4j.readme.io/"><strong>Resilience4j</strong></a><strong> </strong>is a lightweight fault tolerance library inspired by Netflix Hystrix but designed for functional programming. Resilience4j also provides higher-order functions (decorators) to enhance any functional interface, lambda expression, or method reference with Circuit Breaker, Rate Limiter, Retry, or Split [2].</p><p>We will focus on its circuit breaker capability today.</p><p>Resilience4j uses the <a href="https://resilience4j.readme.io/v0.17.0/docs/circuitbreaker"><strong>Ring Bit Buffer</strong></a> to track requests and states, and this is a configurable value. A successful function call is recorded as 0 bit, and a failed function call is recorded as 1 bit. (In fact, I expect the opposite) We can think that it is a sliding window to store and aggregate the outcome of calls.</p><p>There are two types of sliding windows: <strong>count-based </strong>and <strong>time-based</strong>. The count-based sliding window aggregates the outcome of the last N calls. The time-based sliding window aggregates the outcome of the calls of the last N seconds.</p><p>The state machine observes requests (N calls or N seconds) and <strong>signs every single request with success or failure</strong>. According to the threshold which is given before, it decides the state of the circuit breaker (<strong>OPEN, CLOSED, HALF-OPEN — Note: </strong>There are two more special states: DISABLED and FORCED_OPEN. In these states, no request metrics are recorded)</p><p>In order to start collecting metrics of requests and add a circuit breaker to your code, it is needed to add resilience4j dependency to your java project’s pom.xml.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71d0f073e925cf9ba83675c55c7ef780/href">https://medium.com/media/71d0f073e925cf9ba83675c55c7ef780/href</a></iframe><p>We can configure these thresholds by code. The state machine starts to collect metrics and decides state transitions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2fa3fdb18b1a47d18aaf14ea57d9fcc3/href">https://medium.com/media/2fa3fdb18b1a47d18aaf14ea57d9fcc3/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/115057dd8a3289ac31e5bc0f37070716/href">https://medium.com/media/115057dd8a3289ac31e5bc0f37070716/href</a></iframe><p>But we have a spring boot project and we would like to configure it via an application.yml file. Let’s see how we initialize and configure it.</p><h3>Spring Boot Application with Resilience4j</h3><p>We should add the following three dependencies to our project as stated in the <a href="https://resilience4j.readme.io/docs/getting-started-3"><strong>Getting started with the resilience4j-spring-boot2</strong></a> document. Spring Cloud Starter allows us to use Spring Cloud Config as a central place to manage and refresh properties at runtime.</p><blockquote>The module also expects that org.springframework.boot:spring-boot-starter-actuator and org.springframework.boot:spring-boot-starter-aop are already provided at runtime[3]</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a055a1164e1dbbaa7c5c22fdfca2cc3/href">https://medium.com/media/6a055a1164e1dbbaa7c5c22fdfca2cc3/href</a></iframe><p>Also, as stated in the <a href="https://cloud.spring.io/spring-cloud-circuitbreaker/reference/html/index.html#configuring-resilience4j-circuit-breakers"><strong>Spring Cloud Circuit Breaker</strong></a> documentation, there are two starters for the Resilience4j implementations, one for reactive applications and one for non-reactive applications.</p><blockquote>org.springframework.cloud:spring-cloud-starter-circuitbreaker-resilience4j — for non-reactive applications</blockquote><blockquote>org.springframework.cloud:spring-cloud-starter-circuitbreaker-reactor-resilience4j — for reactive applications</blockquote><p>So we need to add one of them to our project. We will go forward with the non-reactive one for this sample.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0edd3b2d31926be2380e4ed662ff591/href">https://medium.com/media/c0edd3b2d31926be2380e4ed662ff591/href</a></iframe><h3>Use Case: Retrieve Account Summary</h3><p>We have a microservice and one of its responsibility is to retrieve the account information of our members from our partner overseas. We will apply the circuit breaker pattern to prevent the thread pool of this microservice from getting stuck when our partner returns a 500 status code or when the response times increase too much. In such cases, we have a fallback mechanism and our aim is to forward requests to this fallback method immediately.</p><p>For this, the following annotation is added at the place where the request is sent to the external service.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d94dbf9c479c2a383dba98eb0980e863/href">https://medium.com/media/d94dbf9c479c2a383dba98eb0980e863/href</a></iframe><p>We use CircuitBreaker annotation here, in place of the “execute supplier” call in our previous code example. The important thing about the CircuitBreaker annotation is its <strong>name</strong>. This is used in configuration if we would like to customize some of them or use the shared one.</p><p>This circuit breaker annotation throws a <strong>CallNotPermitted</strong> exception for following requests if the state machine decides to state transition to <strong>OPEN</strong>. And the method which is wrapped up with circuit breaker annotation should consider having a fallback method to be able to return a response, even if the owned data is stale.</p><p>For our use case, the fallback data is the latest available snapshot account summary to not show an error page to our members.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53296520d5b7a69af1a482743ed5f9d3/href">https://medium.com/media/53296520d5b7a69af1a482743ed5f9d3/href</a></iframe><h3><strong>How to configure it?</strong></h3><p>The configuration file can be as follows. It is necessary to optimize these values according to the response times of the service. Firstly we need to understand the properties.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/afe57af3a952a6025b214fb6ebb6e318/href">https://medium.com/media/afe57af3a952a6025b214fb6ebb6e318/href</a></iframe><p><strong>Sliding-window-type — </strong>Can be count_based or time_based.</p><p><strong>Sliding-window-size —</strong> Determines the size of the window where the result of requests made will be recorded when Circuit Breaker is in the CLOSED state.</p><p><strong>Slow-call-rate-threshold — </strong>It detects what percentage of requests are slow on the aggregation of windows above. If the percentage of slow requests is greater than or equal to the threshold, it switches to the OPEN state and starts a short circuit.</p><p><strong>Slow-call-duration-threshold —</strong> If a request takes longer than this value, it is marked as slow.</p><p><strong>Failure-rate-threshold — </strong>When the fail rate is greater than or equal to the threshold, Circuit Breaker switches to the OPEN state and initiates a short circuit.</p><p><strong>Permitted-number-of-calls-in-half-open-state —</strong> Indicates the number of requests allowed when CircuitBreaker is HALF-OPEN.</p><p><strong>Max-wait-duration-in-half-open-state — </strong>Specifies a maximum wait time that controls the longest time a CircuitBreaker can stay in the HALF-OPEN state before switching to the OPEN state.</p><p><strong>Wait-duration-in-open-state —</strong> Specifies the amount of time to wait before going from OPEN to HALF-OPEN.</p><p><strong>Automatic-transition-from-open-to-half-open-enabled —</strong> If set to True, it means it will automatically switch from OPEN state to HALF-OPEN state and no call is required to trigger the transition.</p><p>More detailed information and default values: <a href="https://resilience4j.readme.io/docs/circuitbreaker#create-and-configure-a-circuitbreaker">https://resilience4j.readme.io/docs/circuitbreaker#create-and-configure-a-circuitbreaker</a></p><p><strong>And, here are some points from our experiences:</strong></p><ul><li>These values ​​should change for each external service and use case. While deciding on the values, we examined<strong> the average, max, and min response times</strong> of the external service over the last 30 days from <a href="https://newrelic.com/"><strong>New Relic</strong></a>. In addition, we observed <strong>slow/failure response counts</strong> in order to decide on sliding window size and type. We also took into account <strong>the number of requests</strong> we handle in a second during our decision.</li><li>A fallback method of the service should be considered in the case of the OPEN state of the circuit breaker. If you have stale data, returning it may be an option. Or the error can be quickly forwarded to the caller service. However, <strong>it should not be forgotten that the service you responded to can call your service by retrying and can overload your system</strong>.</li><li>It might not be possible to achieve a perfect configuration at first. You should keep the circuit breaker under observation, <strong>perform a load test</strong> and <strong>monitor it continuously</strong> after deploying it to the production. I will explain what you can do about it in another article.</li><li>You might consider using the circuit breaker special states: DISABLED (always allow access) and FORCED_OPEN (always deny access). Having endpoints to pass these states, especially DISABLED, may save your production from misconfigured circuit breaker hell immediately. When things go wrong with configuration, <strong>it might be a good option to be able to disable or force it to open</strong>.</li><li>It can be defined different circuit breaker definitions(names) for different services which we use from the same external provider. If one of the services you use is down but the others are working, <strong>using the same circuit breaker for all of them might cause a production problem</strong>. It will create a full circuit breaker situation and you can’t actually get service when you can normally. It is a good practice to reduce dependencies as much as possible and keep the context small to prevent unexpected side effects.</li></ul><p>I will explain how to write an integration test for a circuit breaker and monitor your circuit breaker on production in the next article. Stay tuned!</p><h3>References</h3><p>[1] Release It, 2nd Edition, Michael T. Nygard <a href="https://learning.oreilly.com/library/view/release-it-2nd/9781680504552">https://learning.oreilly.com/library/view/release-it-2nd/9781680504552</a></p><p>[2] Resilience4j - Lightweight fault tolerance library. <br><a href="https://resilience4j.readme.io/">https://resilience4j.readme.io/</a></p><p>[3] Getting started with resilience4j-spring-boot2 <br><a href="https://resilience4j.readme.io/docs/getting-started-3">https://resilience4j.readme.io/docs/getting-started-3</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf9064084ba4" width="1" height="1" alt=""><hr><p><a href="https://engineering.getmidas.com/a-circuit-breaker-story-how-to-save-your-valuable-services-from-burning-bf9064084ba4">A Circuit Breaker Story — How to save your valuable services from burning?</a> was originally published in <a href="https://engineering.getmidas.com/">Midas Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>