Redundancy Is Not Quorum: How a MariaDB/MaxScale Design Lost Its Write Path
Two MaxScale nodes and four database nodes looked comfortably redundant on a diagram. The design had no obvious single point of failure, and every important component appeared to have a peer.
That did not mean the service would remain writable under every degraded condition.
I exposed the difference in a controlled laboratory test. After I removed two database members, the surviving infrastructure was still running, but MaxScale no longer maintained a writable path. The result initially looked like an availability failure. The observed behaviour was consistent with the configured fail-closed safety policy: without majority authority, the monitoring layer did not behave as though a writable primary could be trusted.
The test changed the way I reviewed high-availability designs. I stopped treating redundancy, quorum and service availability as interchangeable properties. They answer different questions:
- Redundancy: Are there alternative components?
- Quorum: Does one side still have enough authority to make a decision
- Availability: Can the application complete the operation it needs — in this case, a write?
An architecture can have redundant components while lacking quorum. It can also preserve consistency by deliberately sacrificing write availability.
This article describes a laboratory design and the failure scenarios I personally executed. It is not a claim about a production failover event, and it is not an argument that one database node count is universally correct.
The architecture looked more resilient than it was
The initial laboratory topology had two MaxScale nodes in front of four MariaDB backend nodes. The MaxScale monitors used cooperative monitoring so that only one monitor would have authority to perform topology-changing operations.
At a component level, the design looked reassuring. Application clients reached a shared service endpoint, which was presented through two MaxScale nodes. Behind them were four MariaDB members.
There were two proxy nodes. There were four database members. Losing a process or a server did not automatically remove the entire service.
The mistake was not the presence of redundant nodes. It was evaluating the design mainly by counting them.
A MaxScale monitor does more than check whether MariaDB responds on a TCP port. In a cooperative-monitoring arrangement, multiple MaxScale instances coordinate through locks on the backend database servers. One monitor becomes the primary monitor only after it obtains the required majority of locks. That authority matters because the primary monitor is the instance allowed to perform cluster-changing operations such as failover or rejoin actions.
The current MariaDB Monitor documentation describes three cooperative-lock modes: none, majority_of_all and majority_of_running. My design used the fail-closed majority_of_all model.
Under majority_of_all, the majority is calculated from all configured backend servers, including servers that are down. For four configured backends, a majority requires three locks. If only two backends remain reachable, neither MaxScale monitor can obtain that majority.
The critical path was therefore not simply “database process running, therefore service available”. It was a sequence of dependencies:
- enough backend lock targets had to remain reachable;
- one monitor had to retain majority authority;
- a writable primary had to be treated as authoritative;
- only then could the routing layer expose a write path.
The diagram showed redundant components. It did not show whether decision-making authority survived the failure model.
I tested the failure model, not just component health
I designed the laboratory test around observable service behaviour. I was not trying merely to prove that MaxScale and MariaDB processes remained alive. I wanted to know whether the application-facing database path behaved safely when the backend topology degraded.
In the initial test, I took two of the four backend members out of service. Two database members and both MaxScale nodes remained, but the monitor could no longer obtain the three-of-four lock majority required by majority_of_all.
The operational symptom was loss of the write path. MaxScale did not continue as though a writable primary remained safely established. The retained record does not include the original monitor trace, so I treat the documented cooperative-lock mechanism as the explanation consistent with the observation, rather than claiming an independently reconstructed internal state transition.
That distinction matters. If I had checked only process state, open sockets or node reachability, I could have reported that most of the platform was still running. From the application’s perspective, however, the required service was unavailable because writes could not proceed.
This is why an HA test needs an application-level assertion. Infrastructure health and service availability are related, but they are not equivalent.
The other important observation was what did not happen: I did not observe two independently authoritative write paths. The design failed closed rather than accepting the risk of competing primary ownership.
I deliberately retain the wording “did not observe”. A bounded laboratory test does not prove that split brain is impossible across every timing condition, network partition or software failure. It does show that, in the scenario I executed, the configured majority policy sacrificed the write path rather than weakening its authority rule.
Why refusing writes can be the safer result
An HA system is often expected to maximise uptime, but availability is not its only objective. A database architecture must also protect consistency and prevent independent decision-makers from making incompatible changes.
In cooperative monitoring, the MaxScale nodes need a way to decide which monitor is allowed to act as primary. If both monitors could independently decide that they had authority during a partition, they could make conflicting topology decisions or expose incompatible views of the writable primary.
majority_of_all avoids that class of risk by requiring a majority of all configured backend lock targets. The consequence is intentionally conservative: when too many backends are unavailable, the monitors may both remain secondary and the service may stay read-only or lose its writable route.
That can feel counterintuitive during an outage. Several servers are still alive; why not use them?
Because liveness is not authority.
The remaining nodes may be technically capable of processing transactions, but the monitoring layer no longer has enough shared evidence to declare that one side is authorised to control the topology. Continuing to write would exchange a visible outage for a less visible consistency risk.
MariaDB also documents majority_of_running, which calculates the majority only from servers currently considered running. That mode can retain operational authority when more configured backends are down, but the trade-off is significant: during a network partition, separate monitor groups may each calculate a majority from the subset they can see. MariaDB explicitly identifies the resulting split-brain risk.
Neither mode is “the HA setting”. They express different risk decisions.
majority_of_all — consistency and single authority. The majority is calculated from all configured backend servers. This favours a single source of authority, but write availability may be lost when too many lock targets are unavailable.
majority_of_running — availability under backend loss. The majority is calculated from the servers each monitor considers running. This can preserve authority during backend loss, but a network partition can allow separate monitor groups to calculate competing majorities.
Get Peng Zhao’s stories in your inbox
Join Medium for free to get updates from this writer.
The correct choice depends on the failure model, recovery objectives and tolerance for inconsistent writes. It should not be selected only because one option appears to keep the service online more often.
The redesign was about matching quorum to the intended failure model
My response was not simply “odd numbers are better”. That slogan hides the engineering decision.
The original four-member test removed two backends. With majority_of_all, the survivors represented only half of the configured lock targets, so they could not form a majority. The result followed directly from the policy.
I revised the laboratory design to use three database members for the intended single-node-failure model. With three configured backends, the majority requirement is two. After one backend is lost, two lock targets remain available and a monitor can still obtain majority authority.
The arithmetic is simple. With four configured backends, the majority was three. Removing two left only two lock targets, so majority authority was lost. With three configured backends, the majority was two. Removing one left two lock targets, so majority authority remained.
This comparison must not be misread as proof that three nodes tolerate more simultaneous failures than four. They do not. The two arrangements describe different tested failure conditions.
The improvement was alignment. The revised design matched its node count and majority policy to the failure scenario the service was expected to tolerate: one backend unavailable while maintaining one authoritative write path.
That is a more useful design question than “How many redundant servers do we have?”:
After the failures we claim to tolerate, do the surviving components still have enough authority to provide the application operation we promise?
I validated the write path after the redesign
After changing the topology, I repeated the relevant failure testing in the laboratory. I disconnected one database member and checked the behaviour through the service path.
The revised arrangement retained service and write availability. I also did not observe split brain during the test.
This validation was deliberately bounded:
- it demonstrated the target single-backend-loss scenario;
- it exercised the application-facing write path rather than relying only on process health;
- it observed that competing write authority did not appear in that scenario;
- it did not claim tolerance of two simultaneous backend failures;
- it did not reproduce every possible network partition;
- it did not constitute production failover acceptance.
Those limits do not weaken the test. They define what the test actually proved.
HA evidence becomes unreliable when a successful narrow test is promoted into a universal resilience claim. A useful test report should state the fault injected, the service assertion made, the unsafe state watched for and the scenarios not covered.
What I now review in quorum-controlled designs
The incident changed my HA review process. I now separate five questions that are often collapsed into one architecture diagram.
1. What is the authoritative operation?
For this service, the important operation was not “connect to a database port”. It was “complete a write through the normal application path”. Read availability and write availability needed separate assertions.
2. Which components vote or provide locks?
A node can serve data, participate in replication, provide a monitor lock, expose a VIP or route a client connection. Those are different roles. The failure calculation must use the components involved in authority, not every redundant box visible in the diagram.
3. How is majority calculated?
“Majority” is incomplete without its denominator. Is it a majority of all configured members, currently running members, a witness set or a dynamic membership view? The answer determines behaviour during node loss and partition.
4. What is the deliberate fail-closed state?
If authority cannot be established, does the system become read-only, remove the primary from routing, block failover or continue optimistically? Operators should recognise this state before an incident so that a safety response is not mistaken for an unrelated failure.
5. Does the test match the availability claim?
If the architecture promises tolerance of one backend failure, test one failure and the application operation that must survive it. If it promises site-partition tolerance, a node shutdown is not a sufficient substitute. The claim, injected fault and validation must describe the same thing.
Availability is an outcome, not a component count
The most valuable result from this laboratory was not the final node count. It was the change in reasoning.
I began with an architecture that looked redundant because it had pairs and multiple backends. The failure test showed that component redundancy did not preserve the monitor authority required for a writable service. MaxScale’s conservative behaviour exposed the mismatch: the platform remained partially alive, but the write path correctly disappeared when the configured majority could not be established.
I then revised the topology around an explicit failure model and validated the operation the application actually needed.
The broader lesson is applicable to database clusters, storage systems, orchestration control planes and distributed network services:
Count failure domains and decision-making authority, not just servers. Redundancy tells you what remains. Quorum tells you whether the survivors may act. Only an end-to-end test tells you whether the service is available.
