<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ach.Chusnul Chikam on Medium]]></title>
        <description><![CDATA[Stories by Ach.Chusnul Chikam on Medium]]></description>
        <link>https://medium.com/@achchusnulchikam?source=rss-3718e4245ddf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*WKimlPk36jZrTeoHhTmAhg.png</url>
            <title>Stories by Ach.Chusnul Chikam on Medium</title>
            <link>https://medium.com/@achchusnulchikam?source=rss-3718e4245ddf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:19:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@achchusnulchikam/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Backup and Restore the Control Plane Node using Relax-and-Recover (ReaR)]]></title>
            <link>https://achchusnulchikam.medium.com/backup-and-restore-the-control-plane-node-using-relax-and-recover-rear-bf174a7b94f9?source=rss-3718e4245ddf------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf174a7b94f9</guid>
            <dc:creator><![CDATA[Ach.Chusnul Chikam]]></dc:creator>
            <pubDate>Sat, 08 Nov 2025 03:43:38 GMT</pubDate>
            <atom:updated>2025-11-08T03:43:38.840Z</atom:updated>
            <content:encoded><![CDATA[<p>Before performing an upgrade or update on <strong>Red Hat OpenStack Platform (RHOSP)</strong>, it is critical to back up the <strong>undercloud</strong> node and <strong>control plane</strong> nodes. Red Hat recommends using the <strong>Relax-and-Recover (ReaR)</strong> tool for this process.</p><p>Look how to backup and restore undercloud node here:<br><a href="https://medium.com/@achchusnulchikam/backup-and-restore-the-undercloud-nodes-using-relax-and-recover-rear-759032b6270f">https://medium.com/@achchusnulchikam/backup-and-restore-the-undercloud-nodes-using-relax-and-recover-rear-759032b6270f</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/0*pbN-C3ww7lyPllo2" /></figure><p><strong>Relax-and-Recover (ReaR)</strong> is a simple, “set up and forget” Linux bare-metal disaster recovery solution. It is designed for reliability and minimal maintenance, making it an ideal choice for system recovery. Once configured, ReaR can back up system configurations and restore them efficiently in the event of a failure or upgrade issue.</p><h4>Prerequisites:</h4><ul><li><strong>RHOSP</strong> is installed.</li><li><strong>NFS server</strong> is installed and configured on the backup node.</li><li><strong>ReaR</strong> is installed on the control plane node.</li><li><strong>Network connectivity</strong> exists between the control plane node and the backup node.</li></ul><h3>There are 2 activities:</h3><h3>- First, Backup the Control Plane Nodes</h3><p>This activity focuses on creating a full backup of the control plane node to ensure that all configurations and system states can be restored if necessary.</p><h3>- Second, Restore the Control Plane Nodes</h3><p>This activity involves restoring the control plane node from the previously created backup image in case of system failure or after an upgrade.</p><h3>1. Backup the Control Plane Nodes</h3><p>To backup the control plane node, prepare and configure the backup-node after that install the Relax-and-Recover tool on the control plane node, and then create the backup image. Create backups as a part of our regular environment maintenance or before performing updates or upgrades.</p><p>Use the backups file to restore the control plane node to its previous state if an error occurs during an update or upgrade. The backup process causes the failover of services that are managed by pacemaker. For this reason, run it during planned maintenance and expect some transactions to be lost.</p><p><strong>1.1. Configuring the backup storage location</strong></p><p>Prepare NFS server on backup-node with IP address (10.8.60.39) and validate control plane can reach and mount NFS share.</p><pre>### on backup-node with IP Adress (10.8.60.39) configure NFS server<br>[root@backup-node ~]# dnf install rpcbind nfs-utils -y<br>[root@backup-node ~]# systemctl enable --now nfs-server rpcbind<br>[root@backup-node ~]# firewall-cmd --permanent --add-service={nfs,rpc-bind,mountd}<br>[root@backup-node ~]# mkdir -p /shares/registry/<br>[root@backup-node ~]# chmod 777 -R /shares/registry/<br>[root@backup-node ~]# ls -lZ /etc/exports<br>[root@backup-node ~]# cat &lt;&lt;EOF&gt; /etc/exports<br>/shares/registry  *(rw,sync,no_subtree_check,no_root_squash,insecure)<br>EOF<br>[root@backup-node ~]# exportfs -rv<br>[root@backup-node ~]# df -h /shares/registry/<br>Filesystem             Size  Used Avail Use% Mounted on<br>/dev/mapper/rhel-root  355G  144G  212G  41% /<br>...<br><br>### on director node<br>[stack@director ~]$ sudo mount -t nfs 10.8.60.39:/shares/registry /mnt/<br>[stack@director ~]$ df -h /mnt/<br>Filesystem                   Size  Used Avail Use% Mounted on<br>10.8.60.39:/shares/registry  355G  144G  212G  41% /mnt</pre><p>Configure the backup storage location in the bar-vars.yaml environment file. This file stores the key-value parameters that pass to the backup execution.</p><pre>### on director node<br>mkdir ~/rear; cd ~/rear<br>cat &lt;&lt;EOF&gt; /home/stack/rear/bar-vars.yaml<br>tripleo_backup_and_restore_server: 10.8.60.39<br>tripleo_backup_and_restore_shared_storage_folder: /shares/registry<br>EOF</pre><p><strong>1.2. Installing and configuring an NFS server on the backup node</strong></p><p>To install and configure an NFS server on the backup-node, create an inventory file, create an SSH key, and run the openstack overcloud backup command with the NFS server options.</p><pre>### on director node<br>cat &lt;&lt;EOF&gt; /home/stack/rear/nfs-inventory.yaml<br>[BackupNode]<br>10.8.60.39 ansible_host=10.8.60.39 ansible_user=root<br>EOF<br><br>ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.8.60.39<br>source ~/stackrc<br>openstack overcloud backup --setup-nfs \<br>--extra-vars /home/stack/rear/bar-vars.yaml \<br>--inventory /home/stack/rear/nfs-inventory.yaml</pre><p><strong>1.3. Installing ReaR on the control plane node</strong></p><p>Before create a backup of the control plane node, install and configure Relax and Recover (ReaR) on the control plane. Extract the static ansible inventory file from the location in which it was saved during installation then Install ReaR on the control plane node.</p><pre>### on director node<br>cp ~/overcloud-deploy/overcloud/tripleo-ansible-inventory.yaml \<br>~/rear/tripleo-inventory.yaml<br><br>openstack overcloud backup --setup-rear \<br>--extra-vars /home/stack/rear/bar-vars.yaml \<br>--inventory /home/stack/rear/tripleo-inventory.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hM2z-AXFarZeAmSHXU9ctw.png" /><figcaption>openstack overcloud backup setup-rear on contrl plane</figcaption></figure><p>If the system uses the UEFI boot loader, perform the following steps on the control plane node:</p><pre>sudo dnf install dosfstools efibootmgr -y</pre><p>Enable UEFI backup in the ReaR configuration file located in /etc/rear/local.conf by replacing the USING_UEFI_BOOTLOADER parameter value 0 with the value 1 and adding UEFI_BOOTLOADER=/boot/efi/EFI/redhat/shimx64.efi.</p><p><strong>1.4. Creating a backup of the control plane node</strong></p><p>To create a backup of the control plane node, use the openstack overcloud backup command then use the backup to restore the control plane node to it’s previous state in case the node becomes corrupted or inaccessible. The backup of the control plane node includes the backup of the database that runs on the control plane node.</p><blockquote>Important</blockquote><blockquote>Before you back up the control plane node, stop the control plane node and check that the file system is free of corruption. For example, disk corruption on the control plane node prevents you from performing a backup of the node.</blockquote><p>Locate the config-drive partition on each control plane node</p><pre>### on control plane node<br>[root@controller-x ~]# blkid -t LABEL=&quot;config-2&quot; -odevice<br>/dev/vda5</pre><p>On each control plane node, backup the config-drive partition of each node as the root user. Replace &lt;config_drive_partition&gt; with the name of the config-drive partition that found it in previous step which is /dev/vda5</p><pre>### on control plane node<br>[root@controller-x ~]# dd if=&lt;config_drive_partition&gt; of=/mnt/config-drive</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*7b96iOl1382I7pTOi3w2mg.png" /><figcaption>backup the config-drive partition</figcaption></figure><p>Source the undercloud credentials then create a backup of the control plane node</p><pre>### on undercloud / director node<br>[stack@director ~]$ tree /home/stack/rear/<br>/home/stack/rear/<br>├── bar-vars.yaml<br>├── nfs-inventory.yaml<br>└── tripleo-inventory.yaml<br>0 directories, 3 files<br><br>[stack@director ~]$ source ~/stackrc<br>(undercloud) [stack@awal-director ~]$ openstack overcloud backup \<br>--inventory /home/stack/rear/tripleo-inventory.yaml</pre><p>In the middle of a control plane backup, controller node will be temporarily removed from the cluster and backup process will be performed one by one. To backup specific node, edit the inventory file then define only contains one controller node</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*youBBGHUwfcaOLDYE3Q50g.png" /><figcaption>pcs cluster status</figcaption></figure><p>Monitor backup process on backup-node with specific share directory, find backup files contains ISO, log and compress data. Notice the output of ReaR backup, <strong>controller-0–202511080909.iso </strong>file will used as a restore image and <strong>backup.tar.gz </strong>contains control plane data.</p><pre>### on backup-node <br>[root@backup-node ~]# ls -lh /shares/registry/controller-0/<br>total 6.2G<br>-rw-------. 1 root root 688M Nov  8 09:10 controller-0-202511080909.iso<br>-rw-------. 1 root root  76M Nov  8 09:19 backup.log<br>-rw-------. 1 root root 5.5G Nov  8 09:19 backup.tar.gz<br>-rw-------. 1 root root  202 Nov  8 09:10 README<br>-rw-------. 1 root root 152K Nov  8 09:10 rear-controller-0-202511080909.log<br>-rw-------. 1 root root    0 Nov  8 09:19 selinux.autorelabel<br>-rw-------. 1 root root  281 Nov  8 09:10 VERSION</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*TKb8SxeNgNPHPHwiKGPBJQ.png" /><figcaption>data backup control plane nodes</figcaption></figure><p>Wait until backup successfully like below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*htocUQ122sQ8oO478bCiRA.png" /><figcaption>openstack overcloud backup successfully execute on all control plane nodes</figcaption></figure><h3>2. Restore the Control Plane Nodes</h3><p>If an error occurs during an update or upgrade, restore the control plane nodes to their previous state using the backup ISO image that created using ReaR. To restore the control plane properly, restore all control plane nodes to ensure state consistency. Find the backup ISO images on the backup node. Burn the bootable ISO image to a DVD or download it to the undercloud node through Integrated Lights-Out (iLO) remote access.</p><p>2.1. Power off each control plane node. Ensure that the control plane nodes are powered off completely.</p><p>2.2. Boot the control plane node with the backup ISO image, in this case will use <strong>controller-0–202511080909.iso.</strong></p><p>2.3. When the Relax-and-Recover boot menu displays, select Recover controller-0</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*2Vm2DfoPdyerTaOQ1udoMQ.png" /><figcaption>Relax and Recover Menu Before Booting</figcaption></figure><blockquote>Notes:<br>If the system uses UEFI, select the <em>Relax-and-Recover (no Secure Boot)</em> option.</blockquote><p>2.4. Log in as the root user and restore the node, the following message displays:</p><p>Welcome to Relax-and-Recover. Run &quot;rear recover&quot; to restore your system! RESCUE &lt;control_plane_node&gt;:~ # rear recover</p><p>When the control plane node restoration process completes, the console displays the following message:</p><p>Finished recovering your system <br>Exiting rear recover <br>Running exit tasks</p><p>2.5. When the command line console is available, restore the config-drive partition of each control plane node</p><p># once completed, restore the config-drive partition (which is ISO9660)<br>RESCUE &lt;control_plane_node&gt;:~ $ dd if=/mnt/local/mnt/config-drive of=&lt;config_drive_partition&gt;</p><p>2.6. Power off the node</p><p>RESCUE director:~ # poweroff</p><p>On boot up, the node resumes its previous state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*9BPxnQv5RqnNtu68Ljl2RQ.png" /><figcaption>Controller Booting with Restoring Data</figcaption></figure><p>2.7. Set the boot sequence to the normal boot device. On boot up, the node resumes its previous state.</p><p>2.8. To ensure that the services are running correctly, check the status of pacemaker. Log in to a Controller node as the root user and enter pcs status command</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8MQSqn5pSaQP6Avo5dMeKQ.png" /><figcaption>pcs cluster status</figcaption></figure><p><a href="https://achchusnulchikam2.medium.com/table-of-contents-19564a6278df"><strong><em>See other content</em></strong></a></p><p><strong>References </strong>:</p><ul><li><a href="https://github.com/rear/rear">https://github.com/rear/rear</a></li><li><a href="https://access.redhat.com/articles/7105543">https://access.redhat.com/articles/7105543</a></li></ul><p>#OpenStack #Backup #ReaR #DisasterRecovery #CloudComputing #PrivateCloud #OpenSource</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf174a7b94f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Backup and Restore the Undercloud Nodes using Relax-and-Recover (ReaR)]]></title>
            <link>https://achchusnulchikam.medium.com/backup-and-restore-the-undercloud-nodes-using-relax-and-recover-rear-759032b6270f?source=rss-3718e4245ddf------2</link>
            <guid isPermaLink="false">https://medium.com/p/759032b6270f</guid>
            <dc:creator><![CDATA[Ach.Chusnul Chikam]]></dc:creator>
            <pubDate>Fri, 07 Nov 2025 22:59:22 GMT</pubDate>
            <atom:updated>2025-11-08T02:31:23.023Z</atom:updated>
            <content:encoded><![CDATA[<p>Before performing an upgrade or update on <strong>Red Hat OpenStack Platform (RHOSP)</strong>, it is critical to back up the <strong>undercloud</strong> node and <strong>control plane</strong> nodes. Red Hat recommends using the <strong>Relax-and-Recover (ReaR)</strong> tool for this process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/0*pbN-C3ww7lyPllo2" /></figure><p><strong>Relax-and-Recover (ReaR)</strong> is a simple, “set up and forget” Linux bare-metal disaster recovery solution. It is designed for reliability and minimal maintenance, making it an ideal choice for system recovery. Once configured, ReaR can back up system configurations and restore them efficiently in the event of a failure or upgrade issue.</p><h4>Prerequisites:</h4><ul><li><strong>RHOSP</strong> is installed.</li><li><strong>NFS server</strong> is installed and configured on the backup node.</li><li><strong>ReaR</strong> is installed on the undercloud node.</li><li><strong>Network connectivity</strong> exists between the undercloud node and the backup node.</li></ul><h3>There are 2 activities:</h3><h3>- First, Backup the Undercloud Node</h3><p>This activity focuses on creating a full backup of the undercloud node to ensure that all configurations and system states can be restored if necessary.</p><h3>- Second, Restore the Undercloud Node</h3><p>This activity involves restoring the undercloud node from the previously created backup image in case of system failure or after an upgrade.</p><h3>1. Backup the Undercloud Node</h3><p>To backup the undercloud node, prepare and configure the backup node after that install the Relax-and-Recover tool on the undercloud node, and then create the backup image. Create backups as a part of our regular environment maintenance can be option too.</p><p>In addition, backup the undercloud node before performing updates or upgrades. Use the backups file to restore the undercloud node to its previous state if an error occurs during an update or upgrade. Let’s take a look at the steps required to backup undercloud node usingReaR like the following:</p><p><strong>1.1. Configuring the backup storage location</strong></p><p>Prepare NFS server on backup-node with IP address (10.8.60.39) and validate undercloud can reach and mount NFS share.</p><pre>### on backup-node with IP Adress (10.8.60.39) configure NFS server<br>[root@backup-node ~]# dnf install rpcbind nfs-utils -y<br>[root@backup-node ~]# systemctl enable --now nfs-server rpcbind<br>[root@backup-node ~]# firewall-cmd --permanent --add-service={nfs,rpc-bind,mountd}<br>[root@backup-node ~]# mkdir -p /shares/registry/<br>[root@backup-node ~]# chmod 777 -R /shares/registry/<br>[root@backup-node ~]# ls -lZ /etc/exports<br>[root@backup-node ~]# cat &lt;&lt;EOF&gt; /etc/exports<br>/shares/registry  *(rw,sync,no_subtree_check,no_root_squash,insecure)<br>EOF<br>[root@backup-node ~]# exportfs -rv<br>[root@backup-node ~]# df -h /shares/registry/<br>Filesystem             Size  Used Avail Use% Mounted on<br>/dev/mapper/rhel-root  355G  186G  170G  53% /<br>...<br><br>### on undercloud / director node<br>[stack@director ~]$ sudo mount -t nfs 10.8.60.39:/shares/registry /mnt/<br>[stack@director ~]$ df -h /mnt/<br>Filesystem                   Size  Used Avail Use% Mounted on<br>10.8.60.39:/shares/registry  355G  186G  170G  53% /mnt</pre><p>Configure the backup storage location in the bar-vars.yaml environment file. This file stores the key-value parameters that pass to the backup execution.</p><pre>### on undercloud / director node<br>mkdir ~/rear; cd ~/rear<br>cat &lt;&lt;EOF&gt; /home/stack/rear/bar-vars.yaml<br>tripleo_backup_and_restore_server: 10.8.60.39<br>tripleo_backup_and_restore_shared_storage_folder: /shares/registry<br>EOF</pre><p><strong>1.2. Installing and configuring an NFS server on the backup node</strong></p><p>To install and configure an NFS server on the backup node, create an inventory file, create an SSH key, and run the openstack undercloud backup command with the NFS server options.</p><pre>### on undercloud / director node<br>cat &lt;&lt;EOF&gt; /home/stack/rear/nfs-inventory.yaml<br>[BackupNode]<br>10.8.60.39 ansible_host=10.8.60.39 ansible_user=root<br>EOF<br><br>ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.8.60.39<br><br>source ~/stackrc<br>openstack undercloud backup --setup-nfs \<br>--extra-vars /home/stack/rear/bar-vars.yaml \<br>--inventory /home/stack/rear/nfs-inventory.yaml</pre><p><strong>1.3. Installing ReaR on the undercloud node</strong></p><p>Before create a backup of the undercloud node, install and configure Relax and Recover (ReaR) on the undercloud. Extract the static ansible inventory file from the location in which it was saved during installation then Install ReaR on the undercloud node.</p><pre>### on undercloud / director node<br>cp ~/overcloud-deploy/overcloud/tripleo-ansible-inventory.yaml \<br>~/rear/tripleo-inventory.yaml<br><br>openstack undercloud backup --setup-rear \<br>--extra-vars /home/stack/rear/bar-vars.yaml \<br>--inventory /home/stack/rear/tripleo-inventory.yaml</pre><p>If the system uses the UEFI boot loader, perform the following steps on the undercloud node:</p><pre>sudo dnf install dosfstools efibootmgr -y</pre><p>Enable UEFI backup in the ReaR configuration file located in /etc/rear/local.conf by replacing the USING_UEFI_BOOTLOADER parameter value 0 with the value 1 and adding UEFI_BOOTLOADER=/boot/efi/EFI/redhat/shimx64.efi.</p><p><strong>1.4. Creating a backup of the undercloud node</strong></p><p>To create a backup of the undercloud node, use the openstack undercloud backup command then use the backup to restore the undercloud node to its previous state in case the node becomes corrupted or inaccessible. The backup of the undercloud node includes the backup of the database that runs on the undercloud node.</p><blockquote>Important</blockquote><blockquote>Before you back up the undercloud node, stop the undercloud node and check that the file system is free of corruption. For example, disk corruption on the undercloud node prevents you from performing a backup of the node.</blockquote><p>On the undercloud node, retrieve the MySQL root password and create a database backup of the undercloud node</p><pre>### on undercloud / director node<br>PASSWORD=$(sudo /bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)<br>sudo podman exec mysql bash -c &quot;mysqldump -uroot -p$PASSWORD --opt --all-databases&quot; | sudo tee /root/$(date +&quot;%d-%b-%Y&quot;)-undercloud-all-databases.sql</pre><p>Source the undercloud credentials then Create a backup of the undercloud node</p><pre>### on undercloud / director node<br>[stack@director ~]$ tree /home/stack/rear/<br>/home/stack/rear/<br>├── bar-vars.yaml<br>├── nfs-inventory.yaml<br>└── tripleo-inventory.yaml<br><br>0 directories, 3 files<br>[stack@director rear]$<br><br>[stack@director ~]$ source ~/stackrc<br>(undercloud) [stack@director ~]$ openstack undercloud backup \<br>--inventory /home/stack/rear/tripleo-inventory.yaml</pre><p>Monitor backup process on backup-node with specific share directory, find backup files contains ISO, log and compress data. Notice the output of ReaR backup, <strong>director-202511071055.iso </strong>file will used as a restore image and <strong>backup.tar.gz </strong>contains undercloud data.</p><pre>### on backup-node <br>[root@backup-node ~]# ls -lh /shares/registry/director/<br>total 44G<br>-rw-------. 1 root root 696M Nov  7 10:57 director-202511071055.iso<br>-rw-------. 1 root root  69M Nov  7 11:33 backup.log<br>-rw-------. 1 root root  43G Nov  7 11:33 backup.tar.gz<br>-rw-------. 1 root root  202 Nov  7 10:57 README<br>-rw-------. 1 root root 200K Nov  7 10:57 rear-director-202511071055.log<br>-rw-------. 1 root root    0 Nov  7 11:33 selinux.autorelabel<br>-rw-------. 1 root root  277 Nov  7 10:57 VERSION</pre><p>Wait until backup successfully like below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0r5LR2Qex47OELbeMcv8VA.png" /><figcaption>openstack undercloud backup successfully execute</figcaption></figure><h3>2. Restore the Undercloud Node</h3><p>Restore the undercloud node to its previous state using the backup ISO image that created using ReaR. Find the backup ISO images on the backup node. Burn the bootable ISO image to a DVD or download it to the undercloud node through Integrated Lights-Out (iLO) remote access.</p><p>2.1. Power off the undercloud node. Ensure that the undercloud node is powered off completely.</p><p>2.2. Boot the undercloud node with the backup ISO image, in this case will use <strong>director-202511071055.iso.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*K2x_JWelWB19jGcHr_dzVQ.png" /><figcaption>Setup Boot Order</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*qoPRZU3z5E_yQO-pIGU5nA.png" /><figcaption>Attach ISO File <strong>director-202511071055.iso</strong></figcaption></figure><p>2.3. When the Relax-and-Recover boot menu displays, select Recover director</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*7FbhatrtWP1LJRZD0N48nA.png" /><figcaption>Relax and Recover Menu Before Booting</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*7mDYMsPuGttK7ZTTHMahng.png" /><figcaption>Configuring Relax and Recover Rescue Option</figcaption></figure><blockquote>Notes:<br>If the system uses UEFI, select the Relax-and-Recover (no Secure Boot) option.</blockquote><p>2.4. Log in as the root user and restore the node, the following message displays:</p><p>Welcome to Relax-and-Recover. Run &quot;rear recover&quot; to restore your system! RESCUE &lt;undercloud_node&gt;:~ # rear recover</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*5PffzB6Kgzo57YpqMA4g3A.png" /><figcaption>Restoring Data Using “rear recover” command</figcaption></figure><p>When the undercloud node restoration process completes, the console displays the following message:</p><p>Finished recovering your system <br>Exiting rear recover <br>Running exit tasks</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*prhyzmGh9MXcDJzT8j6uxQ.png" /><figcaption>Recovering Data Finish</figcaption></figure><p>2.5. Power off the node</p><p>RESCUE director:~ # poweroff</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*BdA9fg7wru8e_y8CZLub7w.png" /><figcaption>Powering Off ReaR Section</figcaption></figure><p>On boot up, the node resumes its previous state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*1Q9MXZC6C-mtn-ygpkoauw.png" /><figcaption>Undercloud Booting with Restoring Data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*LIGyIrMvZsnI8kdo2vyyhg.png" /><figcaption>Undercloud Up and Running Again</figcaption></figure><p><a href="https://achchusnulchikam2.medium.com/table-of-contents-19564a6278df"><strong><em>See other content</em></strong></a></p><p><strong>References </strong>:</p><ul><li><a href="https://github.com/rear/rear">https://github.com/rear/rear</a></li><li><a href="https://access.redhat.com/articles/7105543">https://access.redhat.com/articles/7105543</a></li><li><a href="https://docs.redhat.com/en/documentation/red_hat_openstack_platform/17.1/html-single/backing_up_and_restoring_the_undercloud_and_control_plane_nodes/index#assembly_backing-up-the-undercloud-and-the-control-plane-nodes-using-the-relax-and-recover-tool_br-undercloud-ctlplane">https://docs.redhat.com/en/documentation/red_hat_openstack_platform/17.1/html-single/backing_up_and_restoring_the_undercloud_and_control_plane_nodes/index#assembly_backing-up-the-undercloud-and-the-control-plane-nodes-using-the-relax-and-recover-tool_br-undercloud-ctlplane</a></li></ul><p>#OpenStack #Backup #ReaR #DisasterRecovery #CloudComputing #PrivateCloud #OpenSource</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=759032b6270f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring OpenStack Cluster with Datadog]]></title>
            <link>https://achchusnulchikam.medium.com/monitoring-openstack-cluster-with-datadog-bcf8066ff36e?source=rss-3718e4245ddf------2</link>
            <guid isPermaLink="false">https://medium.com/p/bcf8066ff36e</guid>
            <category><![CDATA[openstack]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[datadog]]></category>
            <category><![CDATA[centralized]]></category>
            <dc:creator><![CDATA[Ach.Chusnul Chikam]]></dc:creator>
            <pubDate>Fri, 29 Aug 2025 05:00:56 GMT</pubDate>
            <atom:updated>2025-08-29T05:00:56.263Z</atom:updated>
            <content:encoded><![CDATA[<p>OpenStack is an open-source cloud platform designed to let organizations deploy and manage compute, storage, and networking resources through web-based dashboards or APIs. Beyond the traditional infrastructure-as-a-service model, OpenStack provides a variety of additional services, including orchestration, fault tolerance, and service lifecycle management. These capabilities enable enterprises to build, operate, and scale applications with high availability. As environments grow, effective monitoring becomes essential.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*JMqUXUio2PqzJVf3" /></figure><p>Datadog a cloud-based monitoring and security platform, provides unified visibility into applications and infrastructure through metrics, logs, and traces. Datadog is powerful dashboards, alerting mechanisms, and integrations with hundreds of technologies make it a versatile solution for both developers and operations teams, supporting efficient collaboration and faster incident resolution. Datadog using <a href="https://docs.datadoghq.com/integrations/openstack-controller/">OpenStack Controller Integration</a> to extends support beyond openstack services to helping operators gain deeper insights and ensure reliable OpenStack operations.</p><h3>* Prerequisites:</h3><ul><li>Existing OpenStack Cluster</li><li>Datadog Account</li><li><a href="https://ap2.datadoghq.com/account/settings/agent/latest">Datadog Agent</a> package</li><li>Credential and access to OpenStack API</li></ul><h3>*Getting Started</h3><p>The OpenStack Controller integration is designed to collect information from all compute nodes and the servers running it. The integration should be run from a single Agent to monitor OpenStack environment and can be deployed on controller node or an adjacent server that has access to the Keystone, Nova, Neutron, Cinder, Ironic, and Octavia endpoints.</p><h3>Preparing Datadog</h3><ol><li>Sign up for your free Datadog trial <a href="https://us5.datadoghq.com/signup">here</a>, until looks like this</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k7T5XhM4JvqEaGsvbUZgfw.png" /></figure><p>2. Go to Integrations menu</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eDImMPu3-OYG5tQt52Stgg.png" /></figure><p>3. Type “OpenStack” in search integration box and install OpenStack Controller</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dGH0DjH2glMoWkD8YZSw4A.png" /></figure><p>4. Install <a href="https://ap2.datadoghq.com/account/settings/agent/latest">datadog agent</a>, in this article used Linux as a platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EZ3xdFjrDtYD3pZboeuM-w.png" /></figure><p>5. Execute provided command on one of Controller node</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*-TCHXMfl9Cgkv4kYXckSeA.png" /></figure><p>6. Check status datadog agent service</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TJF12ifmSzylyR0RHF1qxw.png" /></figure><h3>Agent configuration</h3><ol><li>Edit the conf.yaml file, in the /etc/datadog-agent/conf.d/openstack_controller.d/ folder to start collecting OpenStack Controller performance data. See the <a href="https://github.com/DataDog/integrations-core/blob/master/openstack_controller/datadog_checks/openstack_controller/data/conf.yaml.example">sample openstack_controller.d/conf.yaml</a> for all available configuration options:</li></ol><pre>init_config:<br>instances:<br>  - keystone_server_url: http://XX.XX.X.X:5000 <br>    username: &quot;admin&quot;<br>    password: &#39;XXXXXXX&#39;<br>    domain_id: &quot;default&quot;<br>    use_legacy_check_version: false<br>    ironic_microversion: &#39;1.72&#39;<br>    nova_microversion: &#39;2.88&#39;<br>    cinder_microversion: volume 3.64</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*WLbZdJ8N2gE3TEvcOPhhAg.png" /></figure><p>2. Restart datadog agent service</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KORBspSWz6CwwyFaAOaeHg.png" /></figure><p>3. (Optional) Collecting logs is disabled by default in the Datadog Agent, to enable it in /etc/datadog-agent/datadog.yaml file</p><p>logs_enabled: true</p><p>Add this configuration block to your openstack_controller.d/conf.yaml file to start collecting your Openstack logs:</p><pre>logs:<br>  - type: file<br>    path: &quot;&lt;LOG_FILE_PATH&gt;&quot;<br>    source: openstack</pre><p>Change the path parameter value and configure them for your environment.</p><h3>Explore Datadog Dashboard</h3><p>Open menu Dashboard from Datadog</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*EdYhkbfngzXTAJ_HcbTrvg.png" /></figure><p>There is an “OpenStack Controller Overview” Dashboard provided by Datadog, we can also create our own dashboard using metrics from the Datadog agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jRIv3KlcYwhbYSFP_QFoVA.png" /></figure><p>The dashboard shows the status of service checks, response times, and specific metrics regarding your OpenStack components. The dashboard serves as a first point of reference for troubleshooting errors or identifying opportunities for optimization. For example, we can verify the status of your Nova service, see if any resources are in maintenance mode, review the hosts they’re running on, and pinpoint the best place to start an investigation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BQ3AdOLTRJTL-v1YjhQFeA.png" /></figure><p>This Datadog dashboard provides a comprehensive view of the OpenStack environment, enabling operators to monitor the health, usage, and performance of various OpenStack components in real time. The top-left section displays details about agents, including their host assignments, types, and availability status, which helps ensure critical services like networking and controllers are functioning properly. Below, quota information is presented by project, covering resources such as floating IPs, networks, ports, routers, and subnets, allowing administrators to quickly identify allocation and consumption across domains. The block storage panel highlights key storage metrics, including volume counts, snapshot counts, and pool usage, with detailed breakdowns by volume name, type, and size.</p><p>On the right side, system performance metrics are visualized, including disk activity (read and write requests, errors) and memory utilization (current and maximum usage). These charts provide instant visibility into workload demands and potential bottlenecks.</p><p>The dashboard template can be download here<br><a href="https://pastebin.com/raw/0VKZWsDT">https://pastebin.com/raw/0VKZWsDT</a></p><p><a href="https://achchusnulchikam2.medium.com/table-of-contents-19564a6278df"><strong><em>See other content</em></strong></a></p><p><strong>References </strong>:</p><ul><li><a href="https://docs.datadoghq.com/integrations/openstack-controller/">https://docs.datadoghq.com/integrations/openstack-controller/</a></li><li><a href="https://www.datadoghq.com/blog/openstack-controller-integration/">https://www.datadoghq.com/blog/openstack-controller-integration/</a></li></ul><p>#OpenStack #Monitoring #datadog #observability</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bcf8066ff36e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing SSL/TLS on Red Hat OpenStack’s Public Endpoints]]></title>
            <link>https://achchusnulchikam.medium.com/implementing-ssl-tls-on-red-hat-openstacks-public-endpoints-7ba042a37a01?source=rss-3718e4245ddf------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ba042a37a01</guid>
            <category><![CDATA[ssl]]></category>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[redhat-linux]]></category>
            <category><![CDATA[openstack]]></category>
            <dc:creator><![CDATA[Ach.Chusnul Chikam]]></dc:creator>
            <pubDate>Mon, 28 Jul 2025 05:05:08 GMT</pubDate>
            <atom:updated>2025-07-28T05:22:40.931Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s cloud infrastructure, securing communication between clients and services is a critical aspect of system design. Red Hat OpenStack Platform (RHOSP), being a cloud operating system that manages large pools of compute, storage, and networking resources, often exposes public endpoints to users and services. To protect data in transit and establish trust, implementing SSL/TLS encryption on these endpoints is essential.</p><p>This article focuses on implementing <strong>self-signed SSL/TLS certificates</strong> on Red Hat OpenStack’s public endpoints. While public CA-signed certificates are commonly used in production environments, self-signed certificates can serve as a practical and cost-effective alternative, especially in air-gapped deployments or testing environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*oFD-TC_hNfEJL3bW" /></figure><h3>Self-signed certificates</h3><p>Self-signed certificates provide encryption of data in transit between OpenStack services and clients, ensuring confidentiality and integrity of communication. However, unlike certificates signed by a public Certificate Authority (CA), self-signed certificates are not inherently trusted by clients, browsers, or external systems. This can lead to warning messages or blocked connections unless the certificate or internal CA is manually trusted.</p><p>In RHOSP, self-signed certificates can be useful for securing API endpoints and service-to-service communication such as between RabbitMQ, MariaDB, Keystone, Cinder, Neutron and Nova especially when deployed in isolated environments or within a private data center.</p><h3>Prerequisites:</h3><ul><li>Red Hat OpenStack Environment Installed</li><li>Network isolation to define the endpoints for the Public API</li><li>Self Sign SSL/TLS certificate</li><li>The openssl-perl package is installed</li></ul><h3>*Configuring Self-signed SSL/TLS certificates</h3><p>To use a SSL certificate with our own certificate authority, complete the following configuration steps.</p><h4>1. Initializing the signing host</h4><p>The /etc/pki/CA/index.txt file contains records of all signed certificates and /etc/pki/CA/serial file identifies the next serial number to use for the next certificate to sign.</p><pre>$ mkdir ssl; cd ssl<br>$ sudo mkdir -p /etc/pki/CA<br>$ sudo touch /etc/pki/CA/index.txt<br>$ echo &#39;1000&#39; | sudo tee /etc/pki/CA/serial</pre><h4>2. Creating a certificate authority</h4><p>Create our own certificate authority, generate a key and certificate pair to act as the certificate authority.</p><pre>$ openssl genrsa -out ca.key.pem 4096<br>$ openssl req  -key ca.key.pem -new -x509 -days 7300 -extensions v3_ca -out ca.crt.pem</pre><h4>3. Adding the certificate authority to clients</h4><p>Copy the certificate authority file to each client that requires access to your Red Hat OpenStack Platform environment then add the certificate to the certificate authority trust bundle</p><pre>$ sudo cp ca.crt.pem /etc/pki/ca-trust/source/anchors/<br>$ sudo update-ca-trust extract</pre><p>Set the certificate location as the value for the PublicTLSCAFile parameter in the enable-tls.yaml file in deployment templates. When you set the certificate location as the value for the PublicTLSCAFile parameter, you ensure that the CA certificate path is added to the clouds.yaml authentication file.</p><pre>parameter_defaults:<br>  PublicTLSCAFile: /etc/pki/ca-trust/source/anchors/ca.crt.pem</pre><h4>4. Creating a SSL/TLS Key</h4><p>Enabling SSL/TLS on an OpenStack environment requires an SSL/TLS key (server.key.pem) to generate certificates</p><pre>$ openssl genrsa -out server.key.pem 2048</pre><h4>5. Creating an SSL/TLS certificate signing request</h4><p>Copy the default OpenSSL configuration file then edit the new openssl.cnf file and configure the SSL parameters that you want to used.</p><pre>$ cp /etc/pki/tls/openssl.cnf .<br><br># Edit the new openssl.cnf file and configure the SSL parameters that you want to use for director. An example of the types of parameters to modify include:<br>...<br>[req]<br>distinguished_name = req_distinguished_name<br>req_extensions = v3_req<br><br>[req_distinguished_name]<br>countryName = Country Name (2 letter code)<br>countryName_default = ID<br>stateOrProvinceName = State or Province Name (full name)<br>stateOrProvinceName_default = Jakarta<br>localityName = Locality Name (eg, city)<br>localityName_default = Jakarta Selatan<br>organizationName  = Organization Name (eg, company)<br>organizationName_default = i3<br>organizationalUnitName = Organizational Unit Name (eg, section)<br>organizationalUnitName_default = Cloud<br>commonName = Common Name (eg, your name or your servers hostname)<br>commonName_default = kinton.lab<br>commonName_max = 64<br><br>[ v3_req ]<br># Extensions to add to a certificate request<br>basicConstraints = CA:FALSE<br>keyUsage = nonRepudiation, digitalSignature, keyEncipherment<br>subjectAltName = @alt_names<br><br>[alt_names]<br>IP.1 = 192.168.1.17<br>DNS.1 = overcloud.kinton.lab<br>DNS.2 = kinton.lab<br>...</pre><p>generate a certificate signing request server.csr.pem file, which is the certificate signing request. Use this file to create your OpenStack SSL/TLS certificate.</p><pre>$ openssl req  -key ca.key.pem -new -x509 -days 7300 -extensions v3_ca -out ca.crt.pem</pre><h4>6. Creating a SSL/TLS Key</h4><p>To generate the SSL/TLS certificate for your OpenStack environment, ensure this requirements:</p><ul><li>openssl.cnf</li><li>server.csr.pem</li><li>ca.crt.pem</li><li>ca.key.pem</li></ul><p>Run this command to creates a new certificate named server.crt.pem. Use this certificate in conjunction with your OpenStack SSL/TLS key</p><pre>$ sudo openssl ca -config openssl.cnf \<br>-extensions v3_req \<br>-days 3650 \<br>-in server.csr.pem \<br>-out server.crt.pem \<br>-cert ca.crt.pem \<br>-keyfile ca.key.pem</pre><h4>7. Adding the certificate to undercloud trust bundle</h4><p>Creates a undercloud.pem file with combination of the certificate and key:</p><pre>$ cat server.crt.pem server.key.pem &gt; undercloud.pem</pre><p>Check all SSL/TLS file requirements have been created</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*8A_Ix3nV0M14I0yv-md1RQ.png" /></figure><p>Copy the undercloud.pem file to a location within your /etc/pki directory and set the necessary SELinux context so that HAProxy can read it:</p><pre>$ sudo mkdir /etc/pki/undercloud-certs<br>$ sudo cp ~/undercloud.pem /etc/pki/undercloud-certs/.<br>$ sudo semanage fcontext -a -t etc_t &quot;/etc/pki/undercloud-certs(/.*)?&quot;<br>$ sudo restorecon -R /etc/pki/undercloud-certs<br>$ sudo update-ca-trust extract</pre><p>Add the undercloud.pem file location to the undercloud_service_certificate option in the undercloud.conf file:</p><pre>...<br>undercloud_service_certificate = /etc/pki/undercloud-certs/undercloud.pem</pre><p>Verify the certificate authority was added to the undercloud, use openssl to check the trust bundle:</p><pre>$ openssl crl2pkcs7 -nocrl -certfile /etc/pki/tls/certs/ca-bundle.crt | openssl pkcs7 -print_certs -text | grep kinton.lab -A 10 -B 10</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*bUiIeN6fhWH7FmmW6hXPQg.png" /></figure><h3>*Enabling SSL/TLS on OpenStack public endpoints</h3><p>To enable SSL/TLS in your overcloud, you must create an environment file that contains parameters for your SSL/TLS certiciates and private key.</p><h4>1. Enable SSL/TLS to Templates</h4><p>Copy the enable-tls.yaml environment file from the heat template collection</p><pre>$ cp -r /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml ~/templates/</pre><p>Edit this enable-tls.yaml file and make the following changes for these parameters:</p><p><strong>SSLCertificate<br></strong>Copy the contents of the certificate file (server.crt.pem) into the SSLCertificate parameter</p><p><strong>SSLKey<br></strong>Copy the contents of the private key (server.key.pem) into the SSLKey parameter</p><pre>parameter_defaults:<br>  SSLCertificate: |<br>    -----BEGIN CERTIFICATE-----<br>    MIIFCjCCAvKgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwgYwxCzAJBgNVBAYTAklE<br>    MRAwDgYDVQQIDAdKYWthcnRhMRgwFgYDVQQHDA9KYWthcnRhIFNlbGF0YW4xCzAJ<br>    ...<br>    WLBnyIz5lnJ95JlpsQrVlUcsNBeL/AbQYroZFNYEqkXbPV8Ke0IEO6Mqc2viNfQ9<br>    K1zkt2FIN+7+WIzmpVJj4sZ0QYL4XYlGmRucx4X/qIvIEuW47Fq6ucmaKWK3jA==<br>    -----END CERTIFICATE-----<br>  <br>  SSLKey: |<br>    -----BEGIN PRIVATE KEY-----<br>    MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDtYmtijhp4jdj3<br>    2tPKOADFG74qAiToH7D/QAkTAVJXwJQ/aUbUxR/rldG3kynfWSG4rMRJQE8Kazuc<br>    ...<br>    btv6CxHkRRz27OHmHW4c8w6dNj1CNAbk46MP+btuwfwQ+cR/1li7WXKxKO73ZiQM<br>    CMEn1+978zGHuexLNU7ueQHXmPO6JfoF8fU5S1q+JSfGNY0pXwF5YqG8/u2bMO1i<br>    e+ur2VG3pParMoPedaZ1eNHR<br>    -----END PRIVATE KEY-----</pre><h4>2. Injecting Root Certificate</h4><p>Copy the inject-trust-anchor-hiera.yaml environment file from the heat template collection:</p><pre>$ cp -r /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor-hiera.yaml ~/templates/</pre><p>Edit this inject-trust-anchor-hiera.yaml file and make the following changes for these parameters:</p><p><strong>CAMap<br></strong>The overcloud requires the CA files used to sign the certificates for both the undercloud and the overcloud. Copy the contents of the root certificate authority file (ca.crt.pem) into an entry.</p><pre>parameter_defaults:<br>  # Map containing the CA certs and information needed for deploying them.<br>  # Type: json<br>  CAMap:<br>    kinton-ca:<br>      content: |<br>        -----BEGIN CERTIFICATE-----<br>        MIIF+zCCA+OgAwIBAgIUSYcvchq33bkMQ7y50GuGdjNEUwEwDQYJKoZIhvcNAQEL<br>        BQAwgYwxCzAJBgNVBAYTAklEMRAwDgYDVQQIDAdKYWthcnRhMRgwFgYDVQQHDA9K<br>        ...<br>        9d0BYwoh9KdzLPF+FR/QJI/WhQnQmr5bYShmiZFRsB5xYKKKao8RQxnjnp6yk8CJ<br>        6HFI4iMkzopKAOAykBcaCJDjtFoFHZc20ZLQVJOj7cHYTXMvSi8NSNi7VvAvpNs=<br>        -----END CERTIFICATE-----</pre><h4>3. Configure DNS Endpoints</h4><p>Configure the host and domain names for all fields, adding parameters for custom networks if needed:</p><ul><li>CloudDomain → DNS domain for hosts.</li><li>CloudName → DNS hostname of the overcloud endpoints.</li><li>CloudNameCtlplane → DNS name of the provisioning network endpoint.</li><li>CloudNameInternal → DNS name of the Internal API endpoint.</li><li>CloudNameStorage → DNS name of the storage endpoint.</li><li>CloudNameStorageManagement → DNS name of the storage management endpoint.</li></ul><pre>$ cp -r /usr/share/openstack-tripleo-heat-templates/environments/predictable-placement/custom-domain.yaml ~/templates/configure-domain.yaml</pre><p>Edit this configure-domain.yaml file and make the following changes for these parameters:</p><pre>$ cat ~/templates/configure-domain.yaml <br>parameter_defaults:<br>  CloudDomain: overcloud.kinton.lab<br>  CloudName: overcloud.kinton.lab<br>  CloudNameCtlplane: ctlplane.overcloud.kinton.lab<br>  CloudNameInternal: internalapi.overcloud.kinton.lab<br>  CloudNameStorage: storage.overcloud.kinton.lab<br>  CloudNameStorageManagement: storagemgmt.overcloud.kinton.lab</pre><h4>4. Adding environment files during overcloud creation</h4><p>Use the -e option with the deployment command openstack overcloud deploy to include environment files in the deployment process.</p><pre>$ openstack overcloud deploy --templates \<br>...<br>-e ~/templates/enable-tls.yaml \<br>-e ~/templates/configure-domain.yaml \<br>-e ~/templates/inject-trust-anchor-hiera.yaml \<br>-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-dns.yaml</pre><p>Wait until deployment completed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*sEGuBj9XTeXZf56Z3-AC1Q.png" /></figure><p>Verify that the Public endpoint is now using SSL/TLS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*t0H5i1adndOnCks_5Ok8iA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*eugR4W94aaskDduP9N6CVw.png" /></figure><p>Check on horizon dashboard is connection secure</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tu4LUJ--zbN9PV9DMnmccQ.png" /></figure><p><a href="https://achchusnulchikam2.medium.com/table-of-contents-19564a6278df"><strong><em>See other content</em></strong></a></p><p><strong>References </strong>:</p><ul><li><a href="https://docs.openstack.org/security-guide/secure-communication/introduction-to-ssl-and-tls.html">https://docs.openstack.org/security-guide/secure-communication/introduction-to-ssl-and-tls.html</a></li><li><a href="https://docs.redhat.com/en/documentation/red_hat_openstack_platform/17.1/html/hardening_red_hat_openstack_platform/assembly_enabling-ssl-tls-on-overcloud-public-endpoints">https://docs.redhat.com/en/documentation/red_hat_openstack_platform/17.1/html/hardening_red_hat_openstack_platform/assembly_enabling-ssl-tls-on-overcloud-public-endpoints</a></li></ul><p>#OpenStack #RedHat #RHOSP #PrivateCloud #OpenSource #ssl #tls #secure</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ba042a37a01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Emulated SSD and NVMe Storage on KVM Virtual Machines]]></title>
            <link>https://achchusnulchikam.medium.com/emulated-ssd-and-nvme-storage-on-kvm-virtual-machines-0c5b652e2c58?source=rss-3718e4245ddf------2</link>
            <guid isPermaLink="false">https://medium.com/p/0c5b652e2c58</guid>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[ceph]]></category>
            <category><![CDATA[kvm]]></category>
            <category><![CDATA[ssd]]></category>
            <category><![CDATA[virtualization]]></category>
            <dc:creator><![CDATA[Ach.Chusnul Chikam]]></dc:creator>
            <pubDate>Tue, 03 Jun 2025 00:10:01 GMT</pubDate>
            <atom:updated>2025-06-03T00:10:01.032Z</atom:updated>
            <content:encoded><![CDATA[<p>When building a testbed for storage-heavy platforms like Ceph or OpenStack Swift, simulating realistic hardware conditions can significantly improve the accuracy and value of your tests. One key aspect is storage performance. While physical SSDs and NVMe devices provide high IOPS and low latency in production, it’s not always feasible or necessary to attach real hardware in a virtualized test environment. Fortunately, KVM allows us to emulate SSD and NVMe drives within guest virtual machines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/219/1*aL1FIH7yQ9TbM4HzxhIJ3Q.png" /></figure><h3>Emulate SSD Drives</h3><p>SSD drives require a slightly different setup. To emulate an SSD, add a drive to your KVM guest as usual, specifying the desired bus (e.g., SCSI or SATA). Then, use the appropriate command to set the rotational speed to indicate it’s an SSD. Note that in QEMU, you set this value to 1, which translates to 0 in Linux (indicating a non-rotational drive).</p><p>To do this, you’ll need to know the exact device name, which depends on the number and type of drives you’ve added. Typically, the naming follows a pattern like scsi0-0-0-0 for the first SCSI drive on the first SCSI controller or sata0-0-0 for SATA drives. However, it’s always a good idea to verify the exact name.</p><ul><li>Create Disk</li></ul><pre>[root@server4 ~]# qemu-img create -f qcow2 /virt/ceph2-ssd1.qcow2 20G<br>Formatting &#39;/virt/ceph2-ssd1.qcow2&#39;, fmt=qcow2 size=21474836480 cluster_size=65536 lazy_refcounts=off refcount_bits=16</pre><ul><li>Attach Disk with SCSI targetbus via CLI</li></ul><pre>[root@server4 ~]# virsh attach-disk kinton-ceph2 \<br>  --source /virt/ceph2-ssd1.qcow2 \<br>  --target sda \<br>  --driver qemu \<br>  --subdriver qcow2 \<br>  --targetbus scsi \<br>  --persistent \<br>  --config \<br>  --live</pre><p>or via GUI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*AU5dnftGG4EIfal5T1RJ9A.png" /></figure><ul><li>Confirm the device name by querying the guest using command virsh qemu-monitor-command. Notice the given id</li></ul><pre>[root@server4 ~]# virsh qemu-monitor-command kinton-ceph2 --hmp &quot;info qtree&quot; | grep scsi<br>                scsi = false<br>                scsi = false<br>          dev: virtio-scsi-pci, id &quot;scsi0&quot;<br>              dev: virtio-scsi-device, id &quot;&quot;<br>                bus: scsi0.0<br>                  dev: scsi-hd, id &quot;scsi0-0-0-0&quot;<br>                    device_id = &quot;drive-scsi0-0-0-0&quot;<br>                    scsi_version = 5 (0x5)<br>                    scsi-id = 0 (0x0)<br>                scsi = false</pre><ul><li>Shutdown VM then edit metadata</li></ul><pre>[root@server4 ~]# virsh shutdown kinton-ceph2<br>[root@server4 ~]# virsh edit kinton-ceph2</pre><pre>[root@server4 ~]# virsh edit kinton-ceph2<br>&lt;domain type=&quot;kvm&quot; xmlns:qemu=&#39;http://libvirt.org/schemas/domain/qemu/1.0&#39;&gt;<br>...<br>...<br>  &lt;qemu:commandline&gt;<br>    &lt;qemu:arg value=&#39;-set&#39;/&gt;<br>    &lt;qemu:arg value=&#39;device.scsi0-0-0-0.rotation_rate=1&#39;/&gt;<br>  &lt;/qemu:commandline&gt;<br>&lt;/domain&gt;</pre><ul><li>Start VM</li></ul><pre>[root@server4 ~]# virsh start kinton-ceph2</pre><h3>Verify a New Additional SSD</h3><p>Confirm the rotational speed with lsblk</p><pre>[root@bip-ceph-2 ~]# lsblk -d -o name,rota</pre><p>This will return either <strong>0</strong> (for rotational speed false, meaning SSD) or <strong>1</strong> (for rotating drives, meaning non-SSD). For example, here’s a bunch of drives on a KVM guest where you can see /dev/sda are SSD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*tP9IKGSL48lTvmWQODZRBg.png" /></figure><p>Check with smartctl, which will report the rotational rate as an SSD. Here’s an example on /dev/sda which is set to be an SSD in KVM guest.</p><pre>[root@bip-ceph-2 ~]# smartctl -i /dev/sda</pre><p>This shows a result like this, note Rotational Rate is Solid State Device.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*omvBhPwUL0EZlaw2NGFuXQ.png" /></figure><p>Its very useful for implement multiple device class in Ceph and OpenStack environment. Here is the example, so ceph node VM possible to have HDD and SSD device class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*48D4-1QbkYO9eXptUG49tg.png" /></figure><p><a href="https://achchusnulchikam2.medium.com/table-of-contents-19564a6278df"><strong><em>See other content</em></strong></a></p><p><strong>References </strong>:</p><ul><li><a href="https://blog.christophersmart.com/2019/12/18/kvm-guests-with-emulated-ssd-and-nvme-drives/">https://blog.christophersmart.com/2019/12/18/kvm-guests-with-emulated-ssd-and-nvme-drives/</a></li><li><a href="https://blog.frankenmichl.de/2018/02/13/add-nvme-device-to-vm/">https://blog.frankenmichl.de/2018/02/13/add-nvme-device-to-vm/</a></li></ul><p>#OpenSource #KVM #Ceph #CloudComputing #Linux #SysAdmin</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0c5b652e2c58" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migration VMware to OpenStack with vJailbreak]]></title>
            <link>https://achchusnulchikam.medium.com/migration-vmware-to-openstack-with-vjailbreak-fa58b80c9f57?source=rss-3718e4245ddf------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa58b80c9f57</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[vmware]]></category>
            <category><![CDATA[cloud-migration]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[openstack]]></category>
            <dc:creator><![CDATA[Ach.Chusnul Chikam]]></dc:creator>
            <pubDate>Thu, 24 Apr 2025 02:12:25 GMT</pubDate>
            <atom:updated>2025-04-24T09:00:52.061Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://github.com/platform9/vjailbreak">vJailbreak</a> is an open-source tool featuring a user-friendly interface designed to simplify and accelerate the migration of virtual machines (VMs) from VMware vSphere environments to any OpenStack-compliant cloud. It eliminates the complexities of cross-platform VM migration, enabling you to modernize your infrastructure with minimal disruption and a streamlined, visual workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*dXFQ6f66T5wVoJrtTDkVPQ.png" /><figcaption><em>vJailbreak by Platform9</em></figcaption></figure><p>Key Features of vJailbreak by <a href="https://platform9.com/vjailbreak/">Platform9</a>:</p><ul><li><strong>Intuitive User Interface:</strong> Manage the entire migration process through a clear, easy-to-use graphical interface — no command-line expertise required.</li><li><strong>Seamless vCenter Integration:</strong> Easily connect to your VMware vCenter to manage and migrate VMs.</li><li><strong>Effortless VM Selection:</strong> Select the virtual machines you wish to migrate with just a few clicks.</li><li><strong>Automated Disk Conversion:</strong> VM disks are automatically converted from vmdk to qcow2 format.</li><li><strong>Driver and Device Installation:</strong> Necessary virtual devices and drivers are installed to ensure smooth operation post-migration.</li><li><strong>Post-Migration Health Checks:</strong> Comprehensive health checks are performed to verify the success of the migration and the operational status of the VMs in the new environment.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AxRPprV4a-hfZYXsQhh-qg.png" /><figcaption>vJailbreak migration process flow</figcaption></figure><p>vJailbreak’s intuitive interface leverages the OpenStack &amp; VMware SDKs to interact directly with both your VMware vSphere environment and your target OpenStack cloud. The UI guides you through these key steps:</p><ol><li><strong>Connection Setup:</strong> Easily configure connections to your source VMware vSphere environment and your target OpenStack cloud.</li><li><strong>VM Selection:</strong> Select the VMs you wish to migrate from your vSphere environment.</li><li><strong>Migration Planning:</strong> Configure migration settings, such as target storage and network configurations, through interactive forms.</li><li><strong>Migration Execution:</strong> Initiate and monitor the migration process with real-time progress updates.</li><li><strong>Post-Migration Validation:</strong> Verify the successful migration and launch of your VMs in OpenStack.</li></ol><h3>* Prerequisites:</h3><ul><li>Existing VMware environment as source migration environment</li><li>Existing OpenStack Cluster as destination migration environment</li><li>Ensure that your vJailbreak VM can communicate with your OpenStack and VMware environments. This includes any setup required for VPNs, etc</li><li>Credential VMware account with the <a href="https://platform9.github.io/vjailbreak/introduction/prerequisites/">required permissions</a></li><li>Credential and access to OpenStack API</li><li>Enable <a href="https://knowledge.broadcom.com/external/article/320557">Changed Block Tracking</a> (CBT) on virtual machines</li><li>Installed <a href="https://developer.broadcom.com/sdks/vmware-virtual-disk-development-kit-vddk/8.0">VMware Virtual Disk Development Kit (VDDK) 8.0 for Linux</a></li></ul><h3>*Getting Started</h3><p>Download and install <a href="https://oras.land/docs/installation">ORAS</a>. Then, download the latest version of the vJailbreak image with the following command:</p><pre>oras pull quay.io/platform9/vjailbreak:v0.1.8</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*gLmPdFKMe82tBUaaie_KNA.png" /><figcaption>Download vJailbreak image via ORAS</figcaption></figure><p>This will download the vJailbreak qcow2 folder containing the image locally in the current directory named vjailbreak_qcow2/vjailbreak-image.qcow2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*_N2ZyaR9-Cy93hpREgyUFQ.png" /><figcaption>vJailbreak image detail</figcaption></figure><p>Upload vjailbreak-image.qcow2 to glance using this command:</p><pre>openstack image create --os-interface admin --insecure --container-format bare --disk-format qcow2 --file vjailbreak-image.qcow2 vjailbreak-image.qcow2</pre><p>Deploy a new “vJailbreak” VM from image, choosing proper flavor and a network that can reach your VMware vCenter environment. Give the VM a name, optionally assign an SSH key, assign a network security group that allows inbound and outbound traffic and set a password using cloud-init.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*dHiRdBXTUqGPMBEAWxcALQ.png" /><figcaption>vJailbreak VM running on OpenStack</figcaption></figure><p>Access vjailbreak VM, copy the latest version of the <a href="https://developer.broadcom.com/sdks/vmware-virtual-disk-development-kit-vddk/8.0">VDDK libraries</a> for Linux into /home/ubuntu of the vjailbreak VM. Untar it to a folder named vmware-vix-disklib-distrib in the /home/ubuntu directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*W10IRLMx9tv8LzhM6-jxWQ.png" /><figcaption>VDDK libraries</figcaption></figure><p>Check vjailbreak components and services</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*lV0CPTevXk7RGglWKO5vOA.png" /><figcaption>vJailbreak components and services</figcaption></figure><h3>*Setup vjailbreak</h3><p>Connect to the vJailbreak UI using the IP address assigned during VM creation, in this case <strong><em>192.168.1.233</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*5OEACl59atBXrVOhuwHouA.png" /><figcaption>vJailbreak UI Migration menu</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*xU89ONDRY0bK-pkItCKgbA.png" /><figcaption>vJailbreak Agents menu</figcaption></figure><p>Start the migration process by providing the VMware vCenter and OpenStack admin-rc credentials. First, choose <strong>Credentials</strong> menu and select “<strong>Add OpenStack Credentials</strong>” to fill OpenStack Credentials then <strong>Create Credentials</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*A53yyAhaNLZTyDYpWUjM_w.png" /><figcaption>vJailbreak Credentials menu</figcaption></figure><p>Second, choose <strong>Credentials</strong> menu and select “<strong>Add VMware Credentials</strong>” to fill VMware Credentials then <strong>Create Credentials</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*HMNYrzV5RrYYsF5Hsnw3Bg.png" /><figcaption>vJailbreak Credentials menu</figcaption></figure><p>Verify OpenStack and VMware credential already registered</p><h3>*Running Migration</h3><p>Select “Start Migration” button</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*Tt3jA9y3wptMXvGNFsD2bg.png" /><figcaption>vJailbreak Credentials menu</figcaption></figure><p>Select the VMs you wish to migrate and complete the rest of the migration form containing:</p><ol><li>Source and Destination Environments</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*OjDRIyKEvSro68nVDtFf5A.png" /></figure><p>2. Select Virtual Machines to Migrate, in this case “<strong>vm-winser-2022</strong>”. Flavor can be choosen by itself or auto assign</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*C441bjuLLIPsRYvdm-bWiw.png" /></figure><p>3. Network and Storage Mapping</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*6QBQOsqjo96zojIiX8c49g.png" /></figure><p>4. Migration Options (Optional)<br>By default data copy method is “copy live VMs, then power off”. It means source VM still running during copying data and when cutover source VM will power off.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*SRgjeZddYnSvfXOXnCvP2Q.png" /></figure><p>During the migration process, source VM in VMware is still running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*BAhkspK5XtIhVWJTrMtCHw.png" /></figure><p>When review the logs, it will looks like below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*L5T7SWdcVLeRNDWsk1levQ.png" /><figcaption>Early log entry</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*4AV6DPVZIFdb1M0UY12wTQ.png" /><figcaption>Last log entry</figcaption></figure><p>full log can be seen here <a href="https://pastebin.com/raw/CNQWWL6D">https://pastebin.com/raw/CNQWWL6D</a></p><p>Notice that VM created successfully with ID<strong> <em>611f9ce1–077b-404d-8e0a-4153c117839b</em></strong></p><p>From vjailbreak UI “<strong>vm-winser-2022</strong>” VM successfully entered in migration list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*IYpz7FZogyOqEXrdcLHJxg.png" /><figcaption>vJailbreak Migration menu</figcaption></figure><p>After the cutover process, Source VM on VMware is in powered off state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*5-uJodlVUWWn-mtgzo0U0Q.png" /><figcaption>VMware vCenter menu</figcaption></figure><p>Finally, all migration process is done. “<strong>vm-winser-2022</strong>” VM from VMware now running on OpenStack. Make sure instance ID matches with the logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*ccKOM0kNozLxg2Et0TgpVg.png" /><figcaption>List VM via CLI</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*28o4z42BFSdjhBZVv01O-g.png" /><figcaption>List VM via Horizon</figcaption></figure><p>Based on volume information below, vm-winser-2022-Hard disk 1 attached to instance <strong>vm-winser-2022</strong> as /dev/vda.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*TNjKyiyrEMfhknJjrgYXYQ.png" /><figcaption>List Volumes via Horizon</figcaption></figure><p>Check all functionality of services on VM in new environment, ensure connectivity with surrounding, data gap with source VM, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*tyR3r5VZWgxpmghjCHS_Kw.png" /><figcaption>Display Console VM via Horizon</figcaption></figure><h3>*Additional Items</h3><p>An additional component inside the vjailbreak VM is grafana. Users can explore metrics and monitoring there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*JutxLyrMFC6T2CbmG01yIA.png" /><figcaption>Grafana Kubernetes / Compute Resources (1)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*ukLQ8dbZ4-qS73ZZeDx3DA.png" /><figcaption>Grafana Kubernetes / Compute Resources (2)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*uedZ3Jr_LzQOIWRTWjD0kw.png" /><figcaption>Grafana Kubernetes / Compute Resources (3)</figcaption></figure><p>If we look at grafana, the metric is raise as the time of migration process start from copying live data at 10:05 until VM running on OpenStack at 10:25.</p><h3>Aspects to think about</h3><p>Migrating from VMware to OpenStack can be a complex process, but with the right tools, strategy, and knowledge, it is possible to streamline the transition while minimizing downtime and disruption. To ensure a successful migration, consider the following key factors:</p><ol><li><strong>Choose the right tools</strong>: Selecting the appropriate migration tools is critical to support the process effectively.</li><li><strong>Migration speed</strong>: Evaluate the speed of the migration cycle, from data replication to the final cutover phase, to minimize disruptions.</li><li><strong>Application complexity</strong>: Align the migration strategy with the complexity of the applications being migrated.</li><li><strong>Downtime and rollback planning</strong>: Prepare a detailed plan to handle downtime and ensure rollback procedures are in place in case of issues.</li><li><strong>Data integrity</strong>: Address potential data gaps during the migration process and ensure data consistency.</li><li><strong>Workload dependencies</strong>: Identify dependencies between workloads and their surrounding ecosystem to avoid service interruptions.</li></ol><p><a href="https://achchusnulchikam2.medium.com/table-of-contents-19564a6278df"><strong><em>See other content</em></strong></a></p><p><strong>References </strong>:</p><ul><li><a href="https://platform9.com/vjailbreak/">https://platform9.com/vjailbreak/</a></li><li><a href="https://www.openstack.org/vmware-migration-to-openstack/">https://www.openstack.org/vmware-migration-to-openstack/</a></li></ul><p>#VMware #Migration #OpenStack #OpenSource #vJailbreak #Platform9 #CloudComputing #PrivateCloud</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa58b80c9f57" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streamline Workload Migration: VMware to OpenStack with MigrateKit]]></title>
            <link>https://achchusnulchikam.medium.com/streamline-workload-migration-vmware-to-openstack-with-migratekit-f9c8f92b597f?source=rss-3718e4245ddf------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9c8f92b597f</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[vmware]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[openstack]]></category>
            <dc:creator><![CDATA[Ach.Chusnul Chikam]]></dc:creator>
            <pubDate>Wed, 25 Dec 2024 10:19:34 GMT</pubDate>
            <atom:updated>2024-12-27T22:17:53.253Z</atom:updated>
            <content:encoded><![CDATA[<p>In November 2023, Broadcom completed its acquisition of VMware and subsequent licensing changes have incentivized organizations around the world to re-evaluate their virtualization strategy. <a href="https://www.openstack.org/">OpenStack</a>, the open source standard for cloud infrastructure, has emerged as a leading alternative. Over 80% of <a href="https://openinfra.dev/">OpenInfra</a> members have already talked to customers about migrating workloads from VMware to OpenStack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5B0SXilprNysdmMspirzlA.png" /><figcaption>Migration VMware to OpenStack with Migratekit</figcaption></figure><p>OpenStack’s flexibility and open development enables organizations to:</p><ul><li>Implement a cost effective virtualization strategy by avoiding vendor lock-in</li><li>Modernize their infrastructure with a cloud native strategy</li><li>Customize their stack through increased integration options</li><li>Rely on a global ecosystem and diverse, active open source contributors</li><li>Maintain complete ownership of their infrastructure</li></ul><p>While these benefits are substantial, the migration process can be challenging. Choosing the right tools can make all the difference in ensuring a smooth and cost-effective transition. <a href="https://github.com/vexxhost/migratekit">MigrateKit</a>, recently open sourced by <a href="https://vexxhost.com/">VEXXHOST</a>, is a CLI tool focused on simplifying VMware to OpenStack migration with near-zero downtime. Designed to be easy to use, flexible, it supports various cloud strategies, provides seamless integration with OpenStack environments and allow user to move as much data as possible online. In fact it also uses Virt-V2V in the migration process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/0*SEPNkcMhPdTgA_QX.png" /></figure><p>Key Features of Migratekit:</p><ul><li><strong>Open-Source:</strong> Migratekit is an open-source tool enhancing cloud migration to an open-source environment ensuring seamless integration and consistent deployment.</li><li><strong>Automated Process:</strong> Automates the migration, reducing the required manual effort and minimizing human error risk.</li><li><strong>Consistent Deployments:</strong> Ensure a uniform migration across different environments.</li><li><strong>Flexibility:</strong> Supports various cloud strategies, making it a versatile tool for any migration plan.</li><li><strong>Near Zero Downtime:</strong> Migratekit is designed to move virtual machines to OpenStack with little to no downtime, maintaining business continuity.</li><li><strong>Seamless Integration:</strong> Compatible with all OpenStack deployments.</li></ul><p>Migratekit has 2 phases that it operates in to allow user to migrate virtual machine with near-zero downtime which are documented below:</p><h4>1. Migration phase</h4><p>During this phase, Migratekit proactively transfers data while the virtual machines remain operational. This phase is critical, as it ensures that the bulk of data migration is completed with the <strong><em>virtual machines still running on VMware</em></strong>, vastly reducing the need for system downtime.</p><h4>2. Cutover phase</h4><p>The final phase of the migration process entails a brief, carefully orchestrated downtime window where the remaining data is synchronized. Following this, the virtual machine instances are <strong><em>transitioned from the VMware environment to their new OpenStack</em></strong> setting, which is now ready to operate with all data and settings intact.</p><h3>* Prerequisites:</h3><ul><li>Existing VMware environment as source migration platform</li><li>Existing OpenStack Cluster as target migration platform</li><li>Linux server with docker installed (<em>In this article used Ubuntu 22.04</em>)</li><li>Have connectivity to VMware vCenter endpoint and OpenStack keystone endpoint</li><li>Enable <a href="https://knowledge.broadcom.com/external/article/320557">Changed Block Tracking</a> (CBT) on virtual machines</li><li>Installed <a href="https://developer.broadcom.com/sdks/vmware-virtual-disk-development-kit-vddk/8.0">VMware Virtual Disk Development Kit (VDDK) 8.0 for Linux</a></li><li>Credential VMware account with the <a href="https://github.com/vexxhost/migratekit?tab=readme-ov-file#configuring-account-for-migratekit">required permissions</a></li><li>Credential and access to OpenStack API</li></ul><h3>* Migration Phase</h3><p>On the first migration cycle, Migratekit will do a full copy of the virtual machine to the OpenStack. On subsequent migration cycles, Migratekit will only copy the changes that have been made to the virtual machine since the last migration cycle. The steps look something like the following:</p><h4>1. Preparing Migratekit Setup</h4><p>Check docker version and migratekit images</p><pre>docker version; docker images</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*9PU1k-Wa8gKEb3n1ryjzMQ.png" /></figure><p>Make sure VMware Disk Development Kit (VDDK) is already extract to /usr/lib64/vmware-vix-disklib/</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*1LDRjw84HuOV6Qv7D2_GJQ.png" /></figure><p>Test connection to OpenStack API</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*K3ZeiLpIzn6wPznUPQmTGw.png" /></figure><p>Test connection and access to VMware vCenter server then list all virtual machines managed by a specific host or datacenter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*W7xvJuoU6yK8d2sPXFBaCg.png" /></figure><p>From the list above, there are 2 VMs that will be migrated to OpenStack, namely <strong>vm-ubuntu24</strong> and <strong>vm-windows-server-2022</strong></p><h4>2. Running Migration Cycle with Migratekit</h4><p>Assuming that OpenStack RC file was provided to interact with OpenStack and Docker already installed on migratekit server, then use the following command to run the Migratekit Docker container:</p><pre># Used RC file in OpenStack environment to load administrative credentials<br>source admin-openrc.sh<br><br># Run Migratekit container <br>docker run -it --rm --privileged \<br>  --network host \<br>  -v /dev:/dev \<br>  -v /usr/lib64/vmware-vix-disklib/:/usr/lib64/vmware-vix-disklib:ro \<br>  --env-file &lt;(env | grep OS_) \<br>  registry.atmosphere.dev/library/migratekit:latest \<br>  migrate \<br>  --vmware-endpoint $VMWARE_ENDPOINT \<br>  --vmware-username $VMWARE_USERNAME \<br>  --vmware-password $VMWARE_PASSWORD \<br>  --vmware-path /Datacenter/vm/$VM_NAME</pre><p><strong># Migration vm-ubuntu24</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8bveR3zAOd0na8AZT5je2w.png" /><figcaption>Process of copying data to OpenStack</figcaption></figure><p>In the process above, migration cycle running against a virtual machine located at /Datacenter/vm/vm-ubuntu24 on the VMware endpoint <em>$VMWARE_ENDPOINT</em> (which can be both an ESXi host or a vCenter server) using <em>$VMWARE_USERNAME</em> and <em>$VMWARE_PASSWORD</em> to access VMware. Make sure all of required OpenStack environment variables set in system environment before running the command so that Migratekit can connect to the OpenStack. Also notice volume_id <strong>fffa863d-3e38–4513–8377–970bc10d6ae4 </strong>from the output.</p><p><strong># Migration vm-windows-server-2022</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8QkO3DlYw3zDQx5yyu_U5Q.png" /><figcaption>Process of copying data to OpenStack</figcaption></figure><p>Notice that vm-windows-server-2022 has 2 disks, with volume_id <strong>5cf66b7c-61aa-433e-8d10-a3140a2858da</strong> and<strong> </strong><strong>c4936fd6–5650–4809-bbf0–8cc36581f2b8</strong></p><p><strong># From VMware Side when Migration Phase</strong></p><p>At this point, <strong>virtual machines still running on VMware. </strong>Notice the IP of web service, it will used when VM running on OpenStack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sUtM1hEu2rskQxroj-XW5Q.png" /><figcaption>Ubuntu VM hosting a web service running on VMware</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wf4Fn3aehReN-S7qeJQYdg.png" /><figcaption>Windows Server 2022 VM hosting a web service running on VMware</figcaption></figure><p><strong># From OpenStack Side when Migration Phase</strong></p><p>All data copied from VMware is now available in OpenStack as volumes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LPuou78agynfK2b2Fd5p-Q.png" /><figcaption>List volumes in OpenStack (via Horizon)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9IgtBZQCb0QGoUKuvtWV0w.png" /><figcaption>List volumes in OpenStack (via OpenStack CLI)</figcaption></figure><h3>* Cutover Phase</h3><p>Once VM ready to cut over to the OpenStack cloud, it’s time to run the cutover phase. This will ensure all of the matching resources such as Neutron ports exist on the cloud, do a final sync of the virtual machine, power it off inside of VMware, execute a final sync, and then build a new virtual machine on the OpenStack cloud using the same volumes as the original virtual machine. Run the following command to cutover to the OpenStack cloud:</p><pre># Run Cutover<br>docker run -it --rm --privileged \<br>  --network host \<br>  -v /dev:/dev \<br>  -v /usr/lib64/vmware-vix-disklib/:/usr/lib64/vmware-vix-disklib:ro \<br>  --env-file &lt;(env | grep OS_) \<br>  registry.atmosphere.dev/library/migratekit:latest \<br>  cutover \<br>  --vmware-endpoint $VMWARE_ENDPOINT \<br>  --vmware-username $VMWARE_USERNAME \<br>  --vmware-password $VMWARE_PASSWORD \<br>  --vmware-path /Datacenter/vm/$VM_NAME \<br>  --flavor $FLAVOR_ID \<br>  --security-groups $SECURITY_GROUP_ID \<br>  --network-mapping mac=$MAC_ADDRESS,network-id=$NETWORK_ID,subnet-id=$SUBNET_ID,ip=$IP_ADDRESS</pre><p>Run it with the same arguments, however at this time specify the <strong>flavor </strong>and <strong>security group </strong>that want to use as well as the <strong>network mapping</strong> for the virtual machine on the OpenStack cloud.<br>The network mapping allows Migratekit to know what network to attach the virtual machine to on the OpenStack cloud, as well as the IP address to assign to the virtual machine since it cannot map this information from VMware. The format of the network mapping is as follows:</p><ul><li>mac: The MAC address of the virtual interface on the virtual machine that you want to map to the OpenStack network (mandatory)</li><li>network-id: The UUID of the network that you want to attach the virtual machine to on the OpenStack (mandatory)</li><li>subnet-id: The UUID of the subnet that you want to attach the virtual machine to on the OpenStack (mandatory)</li><li>ip: The IP address that you want to assign to the virtual machine on the OpenStack (optional, Neutron will assign an IP address if this is not specified).</li></ul><p>Ideally match the network mapping to the network that the virtual machine is attached to on the VMware side to ensure that the virtual machine can communicate with the network once it has been migrated to the OpenStack cloud.</p><p><strong># Cutover vm-ubuntu24 using same IP address (<em>192.168.1.158</em>) with source VM</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KYBgYYnigIgw6DBuJk-2TA.png" /><figcaption>Cutover process for VM vm-ubuntu24</figcaption></figure><p>During cutover, there is a full copy of the migration cycle and the Source VM will automatically shut down. Then a new VM created in OpenStack using volumes copy from Source VM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BTTRNHWLnn37VFETUUPBzQ.png" /><figcaption>VM on VMware was poweroff (left), VM now running on OpenStack (middle), web service running with the same IP Address (right)</figcaption></figure><p>At this point check all functionality of services on VM in new environment, IP address, firewall / security group, ensure connectivity with surrounding, data gap with source VM, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dwmTdXa5JI9WT9-Ahub6Xg.png" /><figcaption>A web service with updated content currently hosted on OpenStack</figcaption></figure><p><strong># Cutover vm-windows-server-2022 using different IP address with source VM and has 2 disks</strong></p><p>VM windows server 2022 has 2 disks, first disk for OS and a secondary disk for data. During cutover all disks will be automatically attached to the New VM in OpenStack</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5MXz4I0Up2eT9ZitxO08VQ.png" /><figcaption>Cutover process for VM Windows Server 2022</figcaption></figure><p>Check all functionality of services on VM in new environment, ensure connectivity with surrounding, data gap with source VM, etc. Secondary disk must working fine after cutover.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y6bTX1GZorhsuJ9QCsT1Jg.png" /><figcaption>VM windows on VMware was poweroff (left), VM windows now running on OpenStack (right)</figcaption></figure><p>Finally, both VMs from VMware now running on OpenStack</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*33SpYGuSbPA1vsU5OtBydQ.png" /><figcaption>List instances via horizon OpenStack</figcaption></figure><p>Based on volume information below, volume-ubuntu24–2000 attached to instance <strong>vm-ubuntu24</strong> as /dev/vda. Meanwhile instance <strong>vm-windows-server-2022</strong> has 2 volumes attached for OS and secondary disk or data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DmOg1og98BoO4Ne9YtHskw.png" /><figcaption>List volumes via horizon OpenStack</figcaption></figure><h3>Aspects to think about</h3><p>Migrating from VMware to OpenStack can be a complex process, but with the right tools, strategy, and knowledge, it is possible to streamline the transition while minimizing downtime and disruption. To ensure a successful migration, consider the following key factors:</p><ol><li><strong>Choose the right tools</strong>: Selecting the appropriate migration tools is critical to support the process effectively.</li><li><strong>Migration speed</strong>: Evaluate the speed of the migration cycle, from data replication to the final cutover phase, to minimize disruptions.</li><li><strong>Application complexity</strong>: Align the migration strategy with the complexity of the applications being migrated.</li><li><strong>Downtime and rollback planning</strong>: Prepare a detailed plan to handle downtime and ensure rollback procedures are in place in case of issues.</li><li><strong>Data integrity</strong>: Address potential data gaps during the migration process and ensure data consistency.</li><li><strong>Workload dependencies</strong>: Identify dependencies between workloads and their surrounding ecosystem to avoid service interruptions.</li></ol><p><a href="https://achchusnulchikam2.medium.com/table-of-contents-19564a6278df"><strong><em>See other content</em></strong></a></p><p><strong>References </strong>:</p><ul><li><a href="https://github.com/vexxhost/migratekit">https://github.com/vexxhost/migratekit</a></li><li><a href="https://www.openstack.org/vmware-migration-to-openstack/">https://www.openstack.org/vmware-migration-to-openstack/</a></li></ul><p>#VMware #Migration #OpenStack #OpenSource #Migratekit #CloudComputing #PrivateCloud</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9c8f92b597f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designate DNS as a Service in OpenStack]]></title>
            <link>https://achchusnulchikam.medium.com/deploy-designate-dns-as-a-service-in-openstack-d889fe2d8d16?source=rss-3718e4245ddf------2</link>
            <guid isPermaLink="false">https://medium.com/p/d889fe2d8d16</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[dns]]></category>
            <category><![CDATA[private-cloud]]></category>
            <category><![CDATA[openstack]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Ach.Chusnul Chikam]]></dc:creator>
            <pubDate>Fri, 29 Dec 2023 08:36:57 GMT</pubDate>
            <atom:updated>2023-12-29T08:40:18.108Z</atom:updated>
            <content:encoded><![CDATA[<h3>Overview Designate</h3><p>Designate is an Open Source DNS-as-a-Service implementation and a part of the OpenStack ecosystem of services for running clouds. It allows users and operators to manage DNS records, names and zones via a REST API and can configure existing DNS name servers to contain those records. Designate can also be configured by an operator to integrate with both the OpenStack Network Service (Neutron) and the Compute Service (Nova) so that records are automatically created when floating IPs and compute instances are created respectively, and uses the OpenStack Identity Service (Keystone) for user management. Designate can be thought of as similar to AWS Route 53.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/0*F82T9jrkI5umMru9" /></figure><p>Designate provides DNSaaS services for OpenStack:</p><ul><li>REST API for domain/record management</li><li>Multi-tenant</li><li>Integrated with Keystone for authentication</li><li>Framework in place to integrate with Nova and Neutron notifications (for auto-generated records)</li><li>Support for Bind9 and Infoblox out of the box</li></ul><h3>Designate Architecture</h3><p>Designate is comprised of several different services: the API, Producer, Central, Worker and Mini DNS. It uses an oslo.db compatible database to store state and data, and an oslo.messaging compatible message queue to facilitate communication between services. Multiple copies of all Designate services can be run in tandem to facilitate high availability deployments, with the API process often sitting behind load balancers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*NhBPTjfNQahvuVV2.png" /></figure><p>More about designate: <a href="https://docs.openstack.org/designate/latest/intro/index.html">https://docs.openstack.org/designate/latest/intro/index.html</a></p><h3>Prerequisites:</h3><ul><li>OpenStack Environment Installed, <br>Look here for OpenStack Deployment reference <a href="https://achchusnulchikam.medium.com/deploy-production-ready-openstack-using-kolla-ansible-9cd1d1f210f1">Deploy Multinode OpenStack</a></li><li>Interface used by<strong>dns_interface</strong> must be reachable from public network</li></ul><h3>Setup Designate on Kolla deployment:</h3><p>Let’s take a look at the steps required to configure designate on Kolla deployment. The steps look something like the following:</p><h3>1. Configuration designate service</h3><p>Enable and configure designate service in <strong>/etc/kolla/globals.yml</strong></p><pre>...<br>enable_designate: &quot;yes&quot;<br>dns_interface: &quot;ens8&quot;<br>designate_ns_record:<br>  - &quot;ns1.sample.kinton.org&quot;<br>designate_backend: &quot;bind9&quot;</pre><blockquote><em>Designate MDNS node requires interface </em><strong><em>ens8</em></strong><em> to be reachable from public network.</em></blockquote><h3>2. Install required python clients</h3><pre>pip3 install python-openstackclient python-designateclient</pre><h3>3. Enable designate service</h3><p>Ensure the environment is ready with prechecks process.<br>Then start deploy to enable designate</p><pre>### Do pre-deployment checks for hosts<br>kolla-ansible -i multinode prechecks<br><br>### Finally proceed to deploy <br>kolla-ansible -i multinode deploy</pre><p>Check designate service was already enabled</p><pre>openstack service list; openstack endpoint list --service designate</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SlnQph1VTPiXCjP3i6D9aw.png" /></figure><h3>4. Neutron and Nova Integration</h3><p>Create default designate Zone, in this case is <strong>sample.kinton.org.</strong></p><pre>openstack zone create --email admin@sample.kinton.org sample.kinton.org.<br>openstack zone list<br>+--------------------------------------+--------------------+---------+------------+--------+--------+<br>| id                                   | name               | type    |     serial | status | action |<br>+--------------------------------------+--------------------+---------+------------+--------+--------+<br>| <strong>d11f8186-a09c-4ef6-90d7-d5c1a709fa71</strong> | sample.kinton.org. | PRIMARY | 1703834168 | ACTIVE | NONE   |<br>+--------------------------------------+--------------------+---------+------------+--------+--------+<br><br>openstack recordset list sample.kinton.org.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2_e4aNTQnrM1t0TRabuS0w.png" /></figure><p>Create designate-sink custom configuration folder</p><pre>mkdir -p /etc/kolla/config/designate/</pre><p>Append designate <strong>ZONE_ID</strong> in <strong>/etc/kolla/config/designate/designate-sink.conf</strong></p><blockquote><strong>zone_id</strong> must be manually filled an ID from openstack zone list, check in file kolla-ansible/ansible/roles/designate/templates/designate.conf.j2</blockquote><pre>### cat /etc/kolla/config/designate/designate-sink.conf<br>[handler:nova_fixed]<br>zone_id = <strong>d11f8186-a09c-4ef6-90d7-d5c1a709fa71</strong><br>[handler:neutron_floatingip]<br>zone_id = <strong>d11f8186-a09c-4ef6-90d7-d5c1a709fa71</strong></pre><p>Reconfigure Designate:</p><pre>kolla-ansible -i multinode reconfigure --tags designate,neutron,nova</pre><h3>5. Verify operation</h3><p>List available networks and then associate withdns domain</p><pre>### List available network<br>openstack network list<br><br>### Associate a domain to a network<br>openstack network set &lt;NETWORK_ID&gt; --dns-domain sample.kinton.org.</pre><p>Check detail of network, change <strong>&lt;NETWORK_ID&gt; </strong>with actual value</p><pre>openstack network show &lt;NETWORK_ID&gt; --fit</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*dQjpc2xqYqiFtgBwYKu4Wg.png" /></figure><p>Create a new instance</p><pre>openstack server create \<br>  --image cirros \<br>  --flavor m1.tiny \<br>  --key-name mykey \<br>  --nic net-id=&lt;NETWORK_ID&gt; \<br>  vm-cirros<br>openstack server list</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LrsIxuWEbkbtw0JlJcmBKg.png" /></figure><p>Check DNS records in Designate</p><pre>openstack recordset list sample.kinton.org.<br>+--------------------------------------+------------------------------------+------+-------------------------------------------+--------+--------+<br>| id                                   | name                               | type | records                                   | status | action |<br>+--------------------------------------+------------------------------------+------+-------------------------------------------+--------+--------+<br>| ceec5fb2-5ffe-4b4c-b42b-94e1424a94ca | sample.kinton.org.                 | NS   | ns1.sample.kinton.org.                    | ACTIVE | NONE   |<br>| eb1cf5f2-edf4-475c-9fa4-0c23b263ea3e | sample.kinton.org.                 | SOA  | ns1.sample.kinton.org.                    | ACTIVE | NONE   |<br>|                                      |                                    |      | admin.sample.kinton.org. 1703838070 3558  |        |        |<br>|                                      |                                    |      | 600 86400 3600                            |        |        |<br>| 5e1b92ab-450c-4fb2-a690-0d5ce6dd040b | vm-cirros.admin.sample.kinton.org. | A    | 10.8.60.83                                | ACTIVE | NONE   |<br>| b4787cce-70aa-4cd5-9e17-3959b7d2e666 | vm-cirros.sample.kinton.org.       | A    | 10.8.60.83                                | ACTIVE | NONE   |<br>| ec9cad6d-01e2-40e1-9979-cfc041b1692e | 10-8-60-83.sample.kinton.org.      | A    | 10.8.60.83                                | ACTIVE | NONE   |<br>+--------------------------------------+------------------------------------+------+-------------------------------------------+--------+--------+</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gp6K_Y92Bl54zbo2iU9hKw.png" /></figure><p>Query instance DNS information to Designate <strong>dns_interface</strong> IP address. Validate that designate resolves the DNS record.</p><pre>dig +short @<strong>&lt;DNS_INTERFACE_IP&gt;</strong> vm-cirros.sample.kinton.org. A</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*Y7mQ83Fhbx0upCiRz83CtQ.png" /></figure><p>more information about how designate works, see <a href="https://docs.openstack.org/designate/latest/">Designate, a DNSaaS component for OpenStack</a>.</p><p><a href="https://achchusnulchikam2.medium.com/table-of-contents-19564a6278df"><strong><em>See other content</em></strong></a></p><p><strong>References </strong>:</p><ul><li><a href="https://docs.openstack.org/designate/latest/">https://docs.openstack.org/designate/latest/</a></li><li><a href="https://docs.openstack.org/kolla-ansible/latest/reference/networking/designate-guide.html">https://docs.openstack.org/kolla-ansible/latest/reference/networking/designate-guide.html</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d889fe2d8d16" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Skyline Console OpenStack Modern Dashboard]]></title>
            <link>https://achchusnulchikam.medium.com/skyline-console-openstack-modern-dashboard-835ff3c9190e?source=rss-3718e4245ddf------2</link>
            <guid isPermaLink="false">https://medium.com/p/835ff3c9190e</guid>
            <category><![CDATA[skyline]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[openstack]]></category>
            <category><![CDATA[private-cloud]]></category>
            <dc:creator><![CDATA[Ach.Chusnul Chikam]]></dc:creator>
            <pubDate>Tue, 26 Dec 2023 03:06:28 GMT</pubDate>
            <atom:updated>2023-12-26T05:49:21.852Z</atom:updated>
            <content:encoded><![CDATA[<h3>Skyline Console - Modern Dashboard for OpenStack</h3><p>Skyline Console is one part of OpenStack Modern Dashboard, which provides a web based user interface to OpenStack services including Nova, Swift, Keystone, etc. Skyline was optimized by UI and UE, support OpenStack Train+.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*48qQ-Q0CueXTYDbd.png" /></figure><p>Skyline’s mascot is the nine-color deer. The nine-color deer comes from Dunhuang mural “<em>the nine-color king deer</em>”, whose moral is Buddhist cause-effect and gratefulness, which is consistent with 99cloud’s philosophy of embracing and feedback community since its inception. We also hope Skyline can keep light, elegant and powerful as the nine-color deer, to provide a better dashboard for the openstack community and users.</p><p>More about Skyline: <a href="https://docs.openstack.org/skyline-console/latest/">https://docs.openstack.org/skyline-console/latest/</a></p><h3>Prerequisites:</h3><ul><li>OpenStack Environment Installed</li><li>Instance to run Skyline (image: <strong>Ubuntu 20.04</strong>)</li><li>Skyline server can reach Keystone API</li></ul><h3>Setup Skyline on Ubuntu 20.04</h3><p>Let’s take a look at the steps required to setup Skyline. The steps look something like the following:</p><h3>1. Add Skyline Account</h3><blockquote>Login to your <strong>Openstack administration node</strong> to manage OpenStack from CLI. In my case is <strong>Controller </strong>node</blockquote><pre><strong>### Source the admin credentials<br></strong>(kolla-venv) root@controller1:~# source /etc/kolla/admin-openrc.sh<br><br><strong>### Create the skyline user<br></strong>(kolla-venv) root@controller1:~# openstack user create \<br>--domain default --password-prompt skyline<br>User Password:<br>Repeat User Password:<br>+---------------------+----------------------------------+<br>| Field               | Value                            |<br>+---------------------+----------------------------------+<br>| domain_id           | default                          |<br>| enabled             | True                             |<br>| id                  | 1cdbd41bcb374603af22c2e05e835823 |<br>| name                | skyline                          |<br>| options             | {}                               |<br>| password_expires_at | None                             |<br>+---------------------+----------------------------------+<br><br><strong>### Add the admin role to the skyline user<br></strong>(kolla-venv) root@controller1:~# openstack role add \<br>--project service --user skyline admin</pre><h3>2. Configure MySQL DB</h3><blockquote>Login to your <strong>Skyline Server</strong></blockquote><pre><strong>### Install MySQL<br></strong>$ sudo -i<br># apt-get install mariadb-server -y<br><br><strong>### Create database<br></strong># mysql -u root -p<br>Enter password: <br>Welcome to the MariaDB monitor.  Commands end with ; or \g.<br>Your MariaDB connection id is 36<br>Server version: 10.3.38-MariaDB-0ubuntu0.20.04.1 Ubuntu 20.04<br><br>Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.<br><br>Type &#39;help;&#39; or &#39;\h&#39; for help. Type &#39;\c&#39; to clear the current input statement.<br><br>MariaDB [(none)]&gt; CREATE DATABASE IF NOT EXISTS skyline DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;<br>Query OK, 1 row affected (0.000 sec)<br><br><strong>### Grant proper access to the databases<br></strong>MariaDB [(none)]&gt; GRANT ALL PRIVILEGES ON skyline.* TO &#39;skyline&#39;@&#39;localhost&#39; IDENTIFIED BY &#39;Skyline123&#39;;<br>Query OK, 0 rows affected (0.003 sec)<br>MariaDB [(none)]&gt; GRANT ALL PRIVILEGES ON skyline.* TO &#39;skyline&#39;@&#39;%&#39;  IDENTIFIED BY &#39;Skyline123&#39;;<br>Query OK, 0 rows affected (0.001 sec)</pre><h3>3. Setup Docker</h3><pre><strong>### Install docker<br></strong># apt-get install docker.io -y<br># docker version<br>Client:<br> Version:           24.0.5<br> API version:       1.43<br> Go version:        go1.20.3<br> Git commit:        24.0.5-0ubuntu1~20.04.1<br> Built:             Mon Aug 21 19:50:14 2023<br> OS/Arch:           linux/amd64<br> Context:           default<br><br>Server:<br> Engine:<br>  Version:          24.0.5<br>  API version:      1.43 (minimum version 1.12)<br>  Go version:       go1.20.3<br>  Git commit:       24.0.5-0ubuntu1~20.04.1<br>  Built:            Mon Aug 21 19:50:14 2023<br>  OS/Arch:          linux/amd64<br>  Experimental:     false<br> containerd:<br>  Version:          1.7.2<br>  GitCommit:        <br> runc:<br>  Version:          1.1.7-0ubuntu1~20.04.1<br>  GitCommit:        <br> docker-init:<br>  Version:          0.19.0<br>  GitCommit:   </pre><h3>4. Setup Skyline app</h3><p>Clone skyline repository from <a href="https://opendev.org/openstack/skyline-apiserver">https://opendev.org/openstack/skyline-apiserver</a></p><pre># git clone https://opendev.org/skyline/skyline-apiserver</pre><p>Edit the /etc/skyline/skyline.yaml file in linux server. We can refer to the sample file, and modify the following parameters according to the actual environment:</p><ul><li>database_url</li><li>keystone_url</li><li>default_region</li><li>interface_type</li><li>system_project_domain</li><li>system_project</li><li>system_user_domain</li><li>system_user_name</li><li>system_user_password</li></ul><p>Configure Skyline usingskyline.yaml file</p><pre># mkdir /etc/skyline; cp skyline-apiserver/etc/skyline.yaml.sample /etc/skyline/skyline.yaml</pre><p>Below config are going to provide our openstack environment keystone_url to fetch all endpint so skyline will be able to access openstack. Here is in my case:</p><pre><strong>### cat /etc/skyline/skyline.yaml<br></strong>default:<br>  access_token_expire: 3600<br>  access_token_renew: 1800<br>  cafile: &#39;&#39;<br>  cors_allow_origins: []<br><strong><em>  database_url: mysql://skyline:Skyline123@localhost:3306/skyline</em><br></strong>  debug: false<br>  log_dir: ./log<br>  log_file: skyline.log<br>  policy_file_path: /etc/skyline/policy<br>  policy_file_suffix: policy.yaml<br>  prometheus_basic_auth_password: &#39;&#39;<br>  prometheus_basic_auth_user: &#39;&#39;<br>  prometheus_enable_basic_auth: false<br>  prometheus_endpoint: http://localhost:9091<br>  secret_key: aCtmgbcUqYUy_HNVg5BDXCaeJgJQzHJXwqbXr0Nmb2o<br>  session_name: session<br>  ssl_enabled: true<br>openstack:<br>  base_domains:<br>  - heat_user_domain<br><strong><em>  default_region: RegionOne</em><br></strong>  enforce_new_defaults: true<br>  extension_mapping:<br>    floating-ip-port-forwarding: neutron_port_forwarding<br>    fwaas_v2: neutron_firewall<br>    qos: neutron_qos<br>    vpnaas: neutron_vpn<br><strong>  <em>interface_type: public<br>  keystone_url: https://public.achikam.net:5000/v3/</em><br></strong>  nginx_prefix: /api/openstack<br>  reclaim_instance_interval: 604800<br>  service_mapping:<br>    baremetal: ironic<br>    compute: nova<br>    container: zun<br>    container-infra: magnum<br>    database: trove<br>    dns: designate<br>    identity: keystone<br>    image: glance<br>    instance-ha: masakari<br>    key-manager: barbican<br>    load-balancer: octavia<br>    network: neutron<br>    object-store: swift<br>    orchestration: heat<br>    placement: placement<br>    sharev2: manilav2<br>    volumev3: cinder<br>  sso_enabled: false<br>  sso_protocols:<br>  - openid<br>  sso_region: RegionOne<br>  system_admin_roles:<br>  - admin<br>  - system_admin<br> <strong> <em>system_project: service<br>  system_project_domain: Default</em><br>  </strong>system_reader_roles:<br>  - system_reader<strong><br>  <em>system_user_domain: Default<br>  system_user_name: skyline<br>  system_user_password: Skyline123</em></strong><br>setting:<br>  base_settings:<br>  - flavor_families<br>  - gpu_models<br>  - usb_models<br>  flavor_families:<br>  - architecture: x86_architecture<br>    categories:<br>    - name: general_purpose<br>      properties: []<br>    - name: compute_optimized<br>      properties: []<br>    - name: memory_optimized<br>      properties: []<br>    - name: high_clock_speed<br>      properties: []<br>  - architecture: heterogeneous_computing<br>    categories:<br>    - name: compute_optimized_type_with_gpu<br>      properties: []<br>    - name: visualization_compute_optimized_type_with_gpu<br>      properties: []<br>  gpu_models:<br>  - nvidia_t4<br>  usb_models:<br>  - usb_c</pre><h3>5. Deploy Skyline app</h3><p>Run the skyline_bootstrap container to bootstrap</p><pre><strong>### Run the skyline_bootstrap container to bootstrap<br></strong># rm -rf /tmp/skyline &amp;&amp; mkdir /tmp/skyline &amp;&amp; mkdir /var/log/skyline<br># docker run -d --name skyline_bootstrap -e KOLLA_BOOTSTRAP=&quot;&quot; -v /etc/skyline/skyline.yaml:/etc/skyline/skyline.yaml --net=host 99cloud/skyline:latest<br><br><strong>### Check bootstrap is normal `exit 0`<br></strong># docker ps -a<br>CONTAINER ID   IMAGE                    COMMAND              CREATED         STATUS                     PORTS     NAMES<br>0122f89e1daf   99cloud/skyline:latest   &quot;start_service.sh&quot;   7 seconds ago   Exited (0) 5 seconds ago             skyline_bootstrap<br><br># docker logs skyline_bootstrap<br>+ echo &#39;/usr/local/bin/gunicorn -c /etc/skyline/gunicorn.py skyline_apiserver.main:app&#39;<br>+ mapfile -t CMD<br>++ tail /run_command<br>++ xargs -n 1<br>+ [[ -n 0 ]]<br>+ cd /skyline-apiserver/<br>+ make db_sync<br>alembic -c skyline_apiserver/db/alembic/alembic.ini upgrade head<br>2023-12-26 01:38:33.711 | INFO     | alembic.runtime.migration:__init__:205 - Context impl MySQLImpl.<br>2023-12-26 01:38:33.712 | INFO     | alembic.runtime.migration:__init__:208 - Will assume non-transactional DDL.<br>2023-12-26 01:38:33.843 | INFO     | alembic.runtime.migration:run_migrations:619 - Running upgrade  -&gt; 000, init<br>+ exit 0</pre><p>Run the skyline service after bootstrap is complete</p><ul><li>If you need to modify skyline port, add -e LISTEN_ADDRESS=&lt;ip:port&gt; in the following command LISTEN_ADDRESS defaults to <strong>0.0.0.0:9999</strong></li><li>If you need to modify the policy rules of a service, add -v /etc/skyline/policy:/etc/skyline/policy in the following command</li></ul><p>Rename the service policy yaml file to &lt;service_name&gt;_policy.yaml, and place it in /etc/skyline/policy folder</p><pre><strong>### Remove bootstrap container<br></strong># docker rm -f skyline_bootstrap<br><br><strong>### Run skyline container using docker <br></strong># docker run -d --name skyline --restart=always -v /etc/skyline/skyline.yaml:/etc/skyline/skyline.yaml --net=host 99cloud/skyline:latest<br>4043b16b3b231e12f3b88441b79486e4122de15e0d21b9ad791d13a5466aaf96</pre><h3>6. Verify Skyline app</h3><pre><strong>### Verify skyline app was running<br></strong># docker ps<br>CONTAINER ID   IMAGE                    COMMAND              CREATED         STATUS         PORTS     NAMES<br>4043b16b3b23   99cloud/skyline:latest   &quot;start_service.sh&quot;   4 seconds ago   Up 3 seconds             skyline</pre><p>We can now access the dashboard: <a href="http://10.8.60.243:9999/base/overview-admin"><strong>http://&lt;IP_ADDRESS&gt;:9999/base/overview-admin</strong></a><br>Used OpenStack user credentials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lF8QkxO_pHF2BIyORp1LDw.png" /><figcaption>Login Page Skyline Dashboard</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4HUvhqbd0Gh-P1v0eSY2qA.png" /><figcaption>Home Overview Admin</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jJulxEsy724OZMkeAxmLPw.png" /><figcaption>Home Overview</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wwkacl737xMH1cuHbnKMTA.png" /><figcaption>Hypervisor Admin List</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LhWCl-8IrzGV6og6PMfSSw.png" /><figcaption>Instances List</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qlenHxSHN9M-IbvtOjphLw.png" /><figcaption>Volumes List</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kb0LfckoYNc10EqjEILosw.png" /><figcaption>Network Topology</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*numbh-aiVkYo26rgPAvFmQ.png" /><figcaption>Create Instance</figcaption></figure><p>Using Skyline we can <strong>set password very easy way</strong> when creating Instance (^_^)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-iLN-hCvdyCeDA3rjSdtNQ.png" /><figcaption>Form System Config when Creating Instance</figcaption></figure><p>Also using Skyline we can <strong>choose manually where Compute to running our new Instance </strong>(^_^). Continue until last form and click <strong>Confirm</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rETOrswr1nkUJuXnOblfug.png" /><figcaption>Form System Config when Creating Instance</figcaption></figure><p>Verify New Instance was successfully running</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kMKmvqrYuu9ZQtJ8N9lHNg.png" /><figcaption>Instance Detail</figcaption></figure><p>Verify from Horizon Dashboard</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QmteUcrQ-qiL2J8Jrr4odQ.png" /><figcaption>List Instance from Horizon</figcaption></figure><p>Let’s Enjoy Skyline Dashboard for your OpenStack management GUI</p><p><a href="https://achchusnulchikam2.medium.com/table-of-contents-19564a6278df"><strong><em>See other content</em></strong></a></p><p><strong>References </strong>:</p><ul><li><a href="https://opendev.org/openstack/skyline-apiserver">https://opendev.org/openstack/skyline-apiserver</a></li><li><a href="https://docs.openstack.org/skyline-console/latest/">https://docs.openstack.org/skyline-console/latest/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=835ff3c9190e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using OpenStack After Kolla Ansible Deployment]]></title>
            <link>https://achchusnulchikam.medium.com/using-openstack-after-kolla-ansible-deployment-9c9a41df9283?source=rss-3718e4245ddf------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c9a41df9283</guid>
            <dc:creator><![CDATA[Ach.Chusnul Chikam]]></dc:creator>
            <pubDate>Sun, 24 Dec 2023 03:16:57 GMT</pubDate>
            <atom:updated>2023-12-24T03:24:10.710Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Install the OpenStack CLI client</h3><pre><br>pip3 install python-openstackclient -c https://releases.openstack.org/constraints/upper/master</pre><h3>2. OpenStack requires a <strong>clouds.yaml</strong> file where credentials for the admin user are set. To generate this file:</h3><pre>kolla-ansible -i multinode post-deploy -vvv</pre><blockquote>Note:</blockquote><blockquote>The file will be generated in <strong>/etc/kolla/clouds.yaml</strong>, you can use it by copying it to <strong>/etc/openstack</strong> or <strong>~/.config/openstack</strong>, or by setting the <strong>OS_CLIENT_CONFIG_FILE</strong> environment variable.</blockquote><h3>3. Verify OpenStack cluster, see everything is great and check Ceph Cluster was integrated with OpenStack</h3><pre>source ~/kolla-venv/bin/activate; source /etc/kolla/admin-openrc.sh<br>openstack endpoint list<br>openstack service list; openstack compute service list; openstack network agent list<br>openstack volume service list; cinder get-pools</pre><h3>4. Depending on how you installed Kolla Ansible, there is a script that will create openstack resources like networks, images, router, flavor, keypair and so on.</h3><blockquote>Warning</blockquote><blockquote>You are free to use the following <strong>init-runonce</strong> script for demo purposes but note it does <strong>not</strong> have to be run in order to use your cloud. Depending on your customisations, it may not work, or it may conflict with the resources you want to create. You have been warned.</blockquote><pre>kolla-venv/share/kolla-ansible/init-runonce</pre><p>The output should be like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AA1nt-5R5gf98w_Hoff9Cg.png" /><figcaption>Create Image</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vatEH5OO5AjqPqwm0j6Gqw.png" /><figcaption>Create router</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*lBld8-xW8ZOukER6IgKFpw.png" /><figcaption>Create Tenant Network</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*sHhMj-KZoSLfP1P8123ihA.png" /><figcaption>Create Tenant Subnet</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*QTBeMXIwPH_nSr7hqBtO4w.png" /><figcaption>Create External Network</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*tVkc5jDDMvA6ccNwhfcqWQ.png" /><figcaption>Create External Subnet</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*4IHek8Xz4thsAwlf_mj8mQ.png" /><figcaption>Create Security Group rule ICMP</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*Z5zkZzBBn0QlHoIGzwYZjw.png" /><figcaption>Create Security Group rule SSH</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*zpftruDadFC_L-D2ltApPg.png" /><figcaption>Create Keypair</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*hR7y80ysVN-i8OHfPtqCKg.png" /><figcaption>Create Flavor</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*Q-x6AsXPtZcg7_t4GCu0zQ.png" /><figcaption>Create another Flavor</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*ycdf0a-hVv4DbF7Geex9Vw.png" /><figcaption>Script Done, the we got next command to create Instance using Volume</figcaption></figure><pre>openstack volume create \<br>  --image cirros \<br>  --size 3 \<br>  boot-cirros<br><br>sleep 20<br><br>openstack server create \<br> --volume boot-cirros \<br> --flavor m1.tiny \<br> --key-name mykey \<br> --network demo-net \<br> vm-cirros</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*-zv_aqqVblyQC9ZqoPG1cw.png" /><figcaption>Create Volume Bootable</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*lEXBBYTd-8F9nddJpBbO9Q.png" /><figcaption>Create Instance</figcaption></figure><p>Below is network topology from our OpenStack resource</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p6eRLLnSCVyuGaF7BKisCg.png" /></figure><p>Access console instance and test connection to internet</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*BkpzktCAwDn_sQ9OfzbYHQ.png" /></figure><h3>5. Look at Ceph Cluster so we can verify that image and volumes are store there.</h3><pre>ceph df </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bm8BBswog24ryC4pZ5-A5g.png" /></figure><p><a href="https://achchusnulchikam2.medium.com/table-of-contents-19564a6278df"><strong><em>See other content</em></strong></a></p><p><strong>References </strong>:</p><ul><li><a href="https://docs.openstack.org/kolla-ansible/latest/">https://docs.openstack.org/kolla-ansible/latest/</a></li></ul><p>#OpenStack #kolla #ansible #ceph #CloudComputing #PrivateCloud #OpenSource</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c9a41df9283" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>