<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dhivya Arumugam on Medium]]></title>
        <description><![CDATA[Stories by Dhivya Arumugam on Medium]]></description>
        <link>https://medium.com/@divya4619?source=rss-10c6508696b2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*tB6dALZ1Cxf2fHSG</url>
            <title>Stories by Dhivya Arumugam on Medium</title>
            <link>https://medium.com/@divya4619?source=rss-10c6508696b2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 06:25:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@divya4619/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Troubleshooting & Recovery: MySQL InnoDB Cluster (3-Node HA Setup)]]></title>
            <link>https://medium.com/@divya4619/troubleshooting-recovery-mysql-innodb-cluster-3-node-ha-setup-9e8ed178ee82?source=rss-10c6508696b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e8ed178ee82</guid>
            <dc:creator><![CDATA[Dhivya Arumugam]]></dc:creator>
            <pubDate>Tue, 22 Apr 2025 12:06:33 GMT</pubDate>
            <atom:updated>2025-04-22T12:06:33.180Z</atom:updated>
            <content:encoded><![CDATA[<h3>Objective</h3><p>This document outlines the steps to recover a <strong>3-node MySQL InnoDB Cluster</strong> from a failure scenario. I encountered an issue where the <strong>primary node switched to read-only mode</strong> because the HA nodes failed to apply a transaction from the primary.</p><h3>Issue Summary</h3><ul><li>Primary node turned <strong>read-only</strong>.</li><li>HA nodes failed to apply a transaction from primary.</li><li>Cluster was in <strong>degraded</strong> state</li></ul><p>&quot;Cluster is NOT tolerant to any failures. 2 members are not active.&quot;</p><h3>Recovery Procedure</h3><h3>1. Stop Group Replication on All Nodes</h3><p>On <strong>each</strong> of the 3 HA nodes, run the following in the MySQL client:</p><p>STOP GROUP_REPLICATION;</p><h3>2. Reboot Cluster from Outage (from Primary Node)</h3><p>Using mysqlsh on the <strong>primary node</strong>:</p><p>dba.rebootClusterFromCompleteOutage();</p><p>If this fails (e.g., due to GTID set mismatch or missing transactions on HA nodes), retry with the force option:</p><p>mysqlsh&gt; dba.rebootClusterFromCompleteOutage(&#39;cluster&#39;, {force: true});</p><h3>3. Primary Node Comes Online</h3><p>After successful reboot with force:</p><ul><li>The <strong>primary node</strong> should come <strong>online in R/W mode</strong>.</li><li>However, the other HA nodes may <strong>fail to rejoin</strong> due to <strong>incompatible GTID sets</strong>.</li></ul><h3>4. Reconfigure the HA Nodes</h3><h3>a. Remove the Failed HA Node(s)</h3><p>Run from the <strong>primary node</strong> (mysqlsh):</p><p>dba.getCluster().removeInstance(&quot;HA_NODE_HOST:3306&quot;);</p><h3>b. Rejoin the HA Node(s)</h3><p>Ensure replication user credentials are valid.</p><p>dba.getCluster().addInstance(&quot;rep_user@HA_NODE_HOST:3306&quot;, {password: &quot;xxxxx&quot;});</p><p><strong>Tip:</strong> If the data size is large, consider using screen to keep the session alive in background.</p><h3>Verify Cluster Health</h3><p>Check the current cluster status:</p><p>dba.getCluster().status();</p><p>Ensure:</p><ul><li>All 3 nodes are <strong>ONLINE</strong></li><li><strong>Cluster is tolerant</strong> to at least 1 failure</li><li><strong>Replication lag</strong> is minimal</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e8ed178ee82" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MySQL High Availability Made Easy]]></title>
            <link>https://medium.com/@divya4619/mysql-high-availability-made-easy-55b248ef384c?source=rss-10c6508696b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/55b248ef384c</guid>
            <dc:creator><![CDATA[Dhivya Arumugam]]></dc:creator>
            <pubDate>Tue, 01 Oct 2024 07:52:13 GMT</pubDate>
            <atom:updated>2024-10-01T07:52:13.422Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In today’s digital landscape, ensuring database high availability is crucial. MySQL replication is a popular choice for ensuring redundancy and failover, but when paired with DRBD (Distributed Replicated Block Device) and PCS (Pacemaker/Corosync), you can achieve near real-time data replication and automated failover between nodes. This blog will walk you through setting up MySQL replication using DRBD and PCS, providing a robust solution for database high availability.</p><h3>Why Combine MySQL with DRBD and PCS?</h3><ul><li><strong>MySQL Replication</strong>: Enables real-time or near-real-time duplication of data across multiple nodes. It’s excellent for data redundancy, load balancing, and disaster recovery.</li><li><strong>DRBD</strong>: Works at the block level to mirror data between two servers in real-time. It provides synchronous or asynchronous replication to ensure data integrity.</li><li><strong>PCS (Pacemaker + Corosync)</strong>: Provides clustering and failover management, allowing automatic failover to a standby node when a failure occurs.</li></ul><p>Combining these tools gives you a fault-tolerant, highly available database infrastructure with automatic recovery, keeping your business-critical applications up and running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ol1ckWK-Ajn1w7cKXJHXoA.png" /></figure><h3>Prerequisites</h3><p>Before we begin, make sure you have:</p><ul><li>Two Linux servers with root access (RHEL 8).</li><li>Red Hat Developer Subscription (you can register <a href="https://developers.redhat.com/articles/getting-red-hat-developer-subscription-what-rhel-users-need-know">here</a>)</li></ul><h3>Step 1: Setting Up DRBD for Data Replication</h3><h3>1.1 Install DRBD</h3><p>Install DRBD on both nodes using the following commands:</p><pre>dnf -y install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm<br>rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org<br>yum install -y kmod-drbd84 drbd84-utils<br><br>systemctl disable drbd<br><br>mv /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.conf.bak<br><br>cat &lt;&lt; EOF &gt; /etc/drbd.d/global_common.conf<br>global {<br> usage-count no;<br>}<br>common {<br> net {<br> protocol C;<br> }<br>}<br>EOF</pre><h3>1.2 Configure DRBD Resource</h3><p>Create a configuration file for the DRBD resource:</p><pre>cat &lt;&lt; EOF &gt; /etc/drbd.d/mysql.res<br>resource mysql {<br> protocol C;<br> on node1.ec2.internal {<br>   address 10.0.0.1:7788;<br>   device /dev/drbd0;<br>   disk /dev/mapper/mysql_lvm;<br>   meta-disk internal;<br> }<br> on node2.ec2.internal {<br>   address 10.0.0.2:7788;<br>   device /dev/drbd0;<br>   disk /dev/mapper/mysql_lvm;<br>   meta-disk internal;<br> }<br>}<br>EOF</pre><p>Replace node1 and node2 with the hostnames or IP addresses of your servers, and adjust the device paths as necessary.</p><h3>1.3 Initialize and Synchronize DRBD</h3><p>On both nodes, initialize the DRBD resource:</p><pre>modprobe drbd<br>drbdadm create-md mysql<br>drbdadm up mysql<br>drbdadm status mysql</pre><p>Then, on the primary node (node1), run the following command to start the initial synchronization:</p><pre>drbdadm -- --overwrite-data-of-peer primary mysql<br>drbdadm status mysql</pre><p>Monitor the synchronization process using:</p><pre>cat /proc/drbd</pre><h3>Step 2: Installing and Configuring PCS (Pacemaker and Corosync)</h3><h3>2.1 Install PCS/Corosync and Configure Cluster</h3><p>On both nodes, install PCS/Corosync cluster software and start the required services:</p><pre>dnf update -y<br>yum install -y  pacemaker corosync pcs<br>systemctl enable pacemaker<br>systemctl enable corosync<br>mkdir –p /home/hacluster<br>chown hacluster:haclient /home/hacluster<br>echo &quot;mysupersecretpassword&quot; | passwd --stdin hacluster<br>systemctl start pcsd<br>systemctl enable pcsd<br>systemctl start corosync<br>systemctl enable corosync<br></pre><p>Authenticate PCS:</p><pre>pcs host auth node1 node2 -u hacluster</pre><p>On primary node, execute the below commands to create and start the cluster:</p><pre>sudo pcs cluster setup --name mysql_cluster node1 node2</pre><p>On both nodes, start the cluster service.</p><pre>pcs cluster start --all<br>pcs cluster enable --all</pre><p>Verify the cluster service on both nodes.</p><pre>pcs status corosync<br>corosync-cfgtool -s<br>corosync-cmapctl | grep members<br>pcs status</pre><h3>Step 3: Setting Up MySQL Replication</h3><h3>3.1 Configure MySQL on Both Nodes</h3><p>Install and configure MySQL on both nodes.</p><pre>dnf module install mysql -y<br>systemctl disable mysqld<br>mkdir -p /mysql/data<br>chown -R mysql:mysql /mysql/data/<br><br>cat &lt;&lt; EOF &gt; /mysql/data/my.cnf<br>[mysqld]<br>datadir=/mysql/data<br>socket=/var/lib/mysql/mysql.sock<br>log-error=/var/log/mysql/mysqld.log<br>pid-file=/mysql/data/mysqld.pid<br>server-id = 1<br>log-bin = mysql-bin<br>relay-log = relay-log<br>binlog_format = row<br>gtid-mode=ON<br>enforce-gtid-consistency=1<br>log_replica_updates<br>relay_log_recovery=1<br>master_info_repository=TABLE<br>relay_log_info_repository=TABLE<br>EOF<br><br>ln -s /mysql/data/my.cnf /etc/my.cnf</pre><p>On primary node, mount the drbd device.</p><pre>mount /dev/drbd0 /mysql/data</pre><p>Initialize MySQL on primary node.</p><pre>mysqld --initialize --user=mysql --datadir=&#39;/mysql/data&#39;<br>systemctl start mysqld<br>systemctl status mysqld</pre><h3>Step 4: Configure Cluster Resources to PCS</h3><p>Now that the cluster is running, configure DRBD as a resource in PCS. Add the DRBD resource to PCS on primary node:</p><pre>pcs resource create DRBD_Data ocf:linbit:drbd drbd_resource=mysql promotable promoted-max=1 promoted-node-max=1 clone-max=2 clone-node-max=1 notify=true</pre><p>Create DRBD Cluster File System as cluster resource to PCS on primary node:</p><pre>pcs resource create DRBD_FS ocf:heartbeat:Filesystem device=/dev/drbd0 directory=/mysql/data fstype=ext4</pre><p>Create MySQL service as cluster resource to PCS on primary node:</p><pre>pcs resource create mysql_db ocf:heartbeat:mysql binary=&quot;/usr/sbin/mysqld&quot; config=&quot;/etc/my.cnf&quot; user=&quot;mysql&quot; group=&quot;mysql&quot;  datadir=&quot;/mysql/data&quot; socket=&quot;/var/lib/mysql/mysql.sock&quot; pid=&quot;/mysql/data/mysqld.pid&quot; additional_parameters=&quot;--log-error=/var/log/mysql/mysqld.log&quot; op start timeout=45s on-fail=restart op stop timeout=60s op monitor </pre><p>Verify whether all the created resources are available on primary node.</p><pre>pcs resource status</pre><p>Configure Resource Constraint on primary node.</p><pre>pcs constraint colocation add DRBD_FS with DRBD_Data-clone INFINITY with-rsc-role=Master<br>pcs constraint colocation add mysql_db with DRBD_FS INFINITY<br>pcs constraint order DRBD_FS then mysql_db<br>pcs constraint order promote DRBD_Data-clone then start DRBD_FS<br><br>pcs constraint</pre><p>Verify the cluster status. With this configuration, mysql data is available in active/passive mode.</p><pre>pcs status</pre><h3>Conclusion</h3><p>By setting up MySQL with DRBD and PCS, you’ve built a highly available database solution with automatic failover and block-level replication. This solution provides real-time replication and ensures that your MySQL databases remain consistent and available in case of server failures.</p><p>This setup is highly customizable, allowing you to fine-tune performance and failover mechanisms to meet your organization’s needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55b248ef384c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>