<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marius Manescu on Medium]]></title>
        <description><![CDATA[Stories by Marius Manescu on Medium]]></description>
        <link>https://medium.com/@marius.manescu?source=rss-29862588fbc2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*E6-6ziF0RO-8VZtKOnaXTg.jpeg</url>
            <title>Stories by Marius Manescu on Medium</title>
            <link>https://medium.com/@marius.manescu?source=rss-29862588fbc2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:26:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@marius.manescu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Simulating split-brain scenarios in Akka Cluster]]></title>
            <link>https://medium.com/ing-tech-romania/simulating-split-brain-scenarios-in-akka-cluster-ba7a1369d917?source=rss-29862588fbc2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba7a1369d917</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[toxiproxy]]></category>
            <category><![CDATA[akka]]></category>
            <category><![CDATA[akka-cluster]]></category>
            <dc:creator><![CDATA[Marius Manescu]]></dc:creator>
            <pubDate>Mon, 19 Jan 2026 09:13:40 GMT</pubDate>
            <atom:updated>2026-01-21T07:12:16.599Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article I show how to simulate &amp; test various split-brain scenarios in Akka Cluster — a process that uncovered a bug in Akka’s split-brain resolver (v2.7.0) capable of leaving two independent clusters running during a network partition and even after it heals.</p><h3>Introduction</h3><p>In a distributed system, where multiple nodes work together to provide a service, network partitions can occur.<br>A network partition happens when one or more nodes are isolated from the rest of the cluster due to network failures of different kinds. Network partitions are inevitable in distributed systems whether it’s network glitches, firewall issues, cloud provider outages, link congestion or otherwise.</p><p>If your application requires that some part of its logic to only be executed by a single node at any given time (for consistency reasons), a network partition can bring a lot of headaches.</p><h3>What is a split-brain?</h3><p>A network partition can lead to a split-brain scenario, if multiple nodes on different sides of the partition assume the leader role and more than one decides to keep running. This can potentially lead to data inconsistencies, as leaders can make different decisions, or generate duplicate data, etc.</p><p>To avoid split-brain scenarios, Akka Cluster offers a Split Brain Resolver (SBR) module, with multiple strategies that can be configured. Its purpose is to decide which side of a network partition is shut-down, so that there is only one active cluster running.</p><p>Find out more about it here: <a href="https://doc.akka.io/libraries/akka-core/current/split-brain-resolver.html">https://doc.akka.io/libraries/akka-core/current/split-brain-resolver.html</a></p><h3>Setting up the test environment</h3><p>To demonstrate how to simulate split-brain scenarios with Akka Cluster, we can create a simple test environment using Docker and Toxiproxy.</p><p>Let’s consider a cluster of 4 nodes, running in an Akka Cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Hvb8BIUKuNDZzBRmTTtJQ.png" /><figcaption>Figure 1: Four node cluster communication links</figcaption></figure><p>In case of a clean network partition, node-1 can become isolated from the rest of the cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l5-_G_AkTHa3Ugo2piG_BA.png" /><figcaption>Figure 2: Clean partition</figcaption></figure><p><strong>Simulating network partitions</strong></p><p>We’re using Toxiproxy (chaos engineering tool) to take control of the network links between the nodes, and be able to inject failures. Toxiproxy gives us fine‑grained control (full isolation, latency, bandwidth, cut link, partial drop), while <em>iptables</em> gives us the ability to simulate asymmetric link failures, where direction matters.<br>First, we create a proxy for each node in <em>setup-toxiproxy.sh</em> script, by leveraging Toxiproxy’s API.</p><pre>curl -s -X POST -d &quot;{\&quot;name\&quot;: \&quot;$name\&quot;, \&quot;listen\&quot;: \&quot;0.0.0.0:$listen_port\&quot;, \&quot;upstream\&quot;: \&quot;$upstream_host:$upstream_port\&quot;, \&quot;enabled\&quot;: true}&quot; http://localhost:8474/proxies</pre><p>We can simulate various network failures, such as:<br>- Isolating a node (cutting off all its network traffic)<br>- Block traffic between two nodes (simulate problematic network partitions)<br>- Heal links back (restoring normal communication)</p><p>Then, we create the <em>simulate-partition.sh</em> script to enable control of these failure cases by using either:<br>- Toxiproxy’s <em>toxics</em>: to completely isolate nodes, or<br>- <em>iptables</em> rules: to block / heal a specific link direction between nodes</p><p><strong>Akka Cluster application</strong></p><p>Let’s start by creating a simple Java application using Akka.<br>We add the required dependencies in our <em>pom.xml</em> and set up the main class to start an Akka Cluster node, with support for management (observability) and cluster bootstrap (automatic cluster formation).</p><pre>public static void main(String[] args) {<br>    Config config = ConfigFactory.load();<br>    ActorSystem&lt;Void&gt; system = ActorSystem.create(rootBehavior(), &quot;sbr-demo&quot;, config);<br>    AkkaManagement.get(system).start(); // Akka Management (HTTP endpoint we use for observability)<br>    ClusterBootstrap.get(system).start(); // Start Cluster Bootstrap<br>}</pre><p>Next we configure the application to use Toxiproxy host and respective proxy port for the links between the nodes, by setting the appropriate <em>akka.remote.artery.canonical.hostname</em> and <em>akka.remote.artery.canonical.port</em> in <em>application.conf</em>.</p><pre>akka {<br>  remote {<br>    artery {<br>      transport = tcp<br>      canonical.hostname = ${?ADVERTISED_HOST}<br>      canonical.port = ${?ADVERTISED_PORT}<br><br>      bind.hostname = &quot;0.0.0.0&quot;<br>      bind.port = 25520<br>    }<br>  }<br>}</pre><p>The nodes which are going to form a cluster are defined in the <em>akka.discovery.config.services</em> section, and matching <em>akka-sbr</em> service in <em>akka.management.cluster.bootstrap.contact-point-discovery.service-name</em> for automatic discovery.</p><p>We also provide a basic <em>Dockerfile</em> to containerize our application which we’ll use for running multiple nodes.</p><p><strong>Dashboard</strong></p><p>Next we create a dashboard to monitor the cluster status, see how each node perceives the cluster membership differently, and track leadership changes.<br>The <em>index.html</em> contains simple HTML and JavaScript code to fetch and display the cluster status from each node. <br>We can use <em>/cluster/members</em> endpoint the Akka Management HTTP endpoint which provides the cluster status of a node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cYcvMYulDdcU9cgzhZTDEA.png" /><figcaption>Figure 3: Cluster status view per node basis</figcaption></figure><p>We refresh the view each second, to provide near real-time status.</p><p><strong>Docker compose</strong></p><p>Finally, we create a <em>docker-compose.yml</em> file with the whole environment:<br>- 4 Akka Cluster nodes: each node environment contains the <em>ADVERTISED_HOST</em> and <em>ADVERTISED_PORT</em> variables to point to respective proxy<br>- 1 Toxiproxy<br>- 1 Dashboard</p><h3>Running tests</h3><p>To run the environment, we follow these steps:<br>1. Build the application: <em>mvn clean package</em><br>2. Start the environment: <em>docker-compose up -d — build</em><br>3. Configure Toxiproxy: <em>./scripts/setup-toxiproxy.sh</em></p><p>At this point a cluster should be formed, which can be verified through the dashboard at <a href="http://localhost:8080](http://localhost:8080).">http://localhost:8080</a>.</p><p><strong>Clean partition</strong></p><p>Let’s isolate a node <em>./scripts/simulate-partition.sh isolate-node node-1</em><br><br>Expectation: SBR <em>keep-oldest</em> strategy will down <em>node-1</em> if oldest node is part of the other partition, or will down the other partition in case <em>node-1</em> is the oldest.</p><p>That is what we observe when we look at the dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dkRvoi1CkswdoHW9FakKww.png" /><figcaption>Figure 4: Oldest node stays up (and is leader)</figcaption></figure><p>Let’s heal the partition in preparation for the next test: <em>./scripts/simulate-partition.sh heal-node node-1</em></p><p><strong>Problematic partition — partial &amp; timed link failures</strong></p><p>Network partitions aren’t always clean, sometimes only some links are broken.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AC006-EzMsFrWN_4JKYTBg.png" /><figcaption>Figure 5: Complex network failure</figcaption></figure><p>Let’s simulate that by blocking links between <em>node-1 </em>and other nodes:<br>- first blocking inbound link to <em>node-1</em> from <em>node-2</em><br>- after a few seconds (should be less than <em>akka.cluster.split-brain-resolver.down-all-when-unstable</em>) blocking the rest of the inbound links to <em>node-1</em>: from <em>node-3</em> and <em>node-4</em></p><pre>./scripts/simulate-partition.sh block-link node-2 node-1; sleep 4; ./scripts/simulate-partition.sh block-link node-3 node-1; ./scripts/simulate-partition.sh block-link node-4 node-1;</pre><p>Expectation: SBR <em>keep-oldest</em> will down <em>node-1</em> if oldest node is part of the other partition, or down the other partition in case <em>node-1</em> is the oldest.<br>Observation: split-brain occurs, with two leaders elected (one in each partition) after temporary phase, as seen in the dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_i4NYB6ZqrN-7xc3evpLgQ.png" /><figcaption>Figure 6: Split-brain: multiple clusters / leaders</figcaption></figure><p>Let’s heal the links:</p><pre>./scripts/simulate-partition.sh heal-link node-2 node-1; ./scripts/simulate-partition.sh heal-link node-3 node-1; ./scripts/simulate-partition.sh heal-link node-4 node-1;</pre><h3>Conclusion</h3><p>We’ve set up a test environment using Docker and Toxiproxy to simulate network partitions and observe the behavior of the cluster under different failure scenarios.</p><p>In one of the tests we broke SBR functionality and ended up with 2 independent Akka Clusters running, even after network partition healed. Something that will likely lead to undersired behaviour of your application.</p><p>This is applicable to <em>keep-oldest</em> (used in this article) but also can be observed with other strategies like <em>keep-majority</em> (even &amp; odd number of nodes).</p><p>For this particular case, we reached Akka and an improvement has been implemented in <em>akka-core</em> (v2.10.15), through the opt-in parameter <em>down-all-when-indirectly-connected = on </em>(<a href="https://github.com/akka/akka-core/pull/32865/files">https://github.com/akka/akka-core/pull/32865/</a>)</p><p>Feel free to clone the repository (<a href="https://github.com/mariuscristian/akka-sbr">https://github.com/mariuscristian/akka-sbr</a>) and experiment with different configurations and failure scenarios to better understand how Akka Cluster handles split-brain situations.</p><p>More importantly, check your application requirements, choose the SBR strategy that best fits your needs and test it out on the battlefield !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba7a1369d917" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ing-tech-romania/simulating-split-brain-scenarios-in-akka-cluster-ba7a1369d917">Simulating split-brain scenarios in Akka Cluster</a> was originally published in <a href="https://medium.com/ing-tech-romania">ING Hubs Romania</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>