<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amitabh Prasad on Medium]]></title>
        <description><![CDATA[Stories by Amitabh Prasad on Medium]]></description>
        <link>https://medium.com/@amitabhprasad?source=rss-912e2cec2888------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*QULSzEUnuy_zS-bS.</url>
            <title>Stories by Amitabh Prasad on Medium</title>
            <link>https://medium.com/@amitabhprasad?source=rss-912e2cec2888------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:37:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@amitabhprasad/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kubernetes volume backup for disaster recovery Part 2]]></title>
            <link>https://medium.com/@amitabhprasad/kubernetes-volume-backup-for-disaster-recovery-part-2-5c6c86bcbe72?source=rss-912e2cec2888------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c6c86bcbe72</guid>
            <category><![CDATA[backup-and-restore]]></category>
            <category><![CDATA[readwritemany]]></category>
            <category><![CDATA[pvc]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[disaster-recovery]]></category>
            <dc:creator><![CDATA[Amitabh Prasad]]></dc:creator>
            <pubDate>Wed, 13 Dec 2023 07:40:10 GMT</pubDate>
            <atom:updated>2023-12-14T09:58:57.061Z</atom:updated>
            <content:encoded><![CDATA[<p>Authors: <a href="https://medium.com/u/912e2cec2888">Amitabh Prasad</a>, <a href="https://medium.com/u/e99a813fbdbd">Naveen Kamagani</a></p><h3>Recovery for RWX storage in AWS</h3><p>In the previous <a href="https://medium.com/@amitabhprasad/kubernetes-volume-backup-for-disaster-recovery-56a5facee7fe">blog</a>, we looked at ways to backup and restore the volume used for RWO storage.</p><p>Since some of the workload in Kubernetes also requires RWX storage, such requirements come because of performance consideration while other could be business needs or plain legacy, we do get such requirements time and time again, especially when we think about why will someone ever need this :-)</p><p>We extended our existing solution of <a href="https://medium.com/@amitabhprasad/kubernetes-volume-backup-for-disaster-recovery-56a5facee7fe">backup &amp; recovery for RWO</a> to RWX. Some of the restore processes will be the same as that for RWO but we get different challenges with EFS ( AWS cloud uses EFS for RWX PVC for Kubernetes) read the <a href="https://aws.amazon.com/blogs/storage/persistent-storage-for-kubernetes/">AWS blog</a> on how to create RWX storage.</p><p>Once the EFS file system is provisioned (mapped for RWX PVC )we can take backup of the mapped volume and move this backup to a different region for regional failure recovery.</p><p>See the screenshots below on configuring backup and moving backup to different DR regions.</p><p>In this guide, we’ll walk you through the process of creating, backing, and restoring Amazon EFS (Elastic File System) data using AWS Backup. Follow these simple steps to recover your data seamlessly. This guide is for taking backup of RWX storage in Kubernetes/</p><p>Some of the new components used in this blog compared to previous ones are:</p><ol><li><strong>EFS (Elastic File System): </strong>EFS is like a shared drive in the cloud. It provides scalable, elastic file storage that multiple EC2 instances can access concurrently. EFS allows your applications running on different servers to share files seamlessly. It’s particularly useful for scenarios where multiple instances need access to the same data.</li><li><strong>VPC (Virtual Private Cloud): </strong>VPC helps you organize and control your virtual infrastructure. You can launch AWS resources, like servers, in your VPC in a secure and isolated manner.</li></ol><p>NOTE:</p><p>EFS does not support snapshotting, as such velero snapshot using CSI can not be used. This leaves us with two options for taking backups</p><ol><li>Take an EFS file system backup using a native backup technique from the cloud provider.</li><li>Use restic.</li></ol><p>For this blog, we used native volume backup using the recovery point of AWS backup as, that’s much more efficient compared to restic, though backup and restore are quite involved but advantages interms performance especially when dealing with gigabytes or terrabytes of data, granular RTO etc. far outweigh disadvantages.</p><p>We created a separate utility for automating backup and restoring in Kubernetes using `AWS Backup`, more on that in a separate blog</p><p>The diagram below represents the use case we will be using for this blog</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tD9QXhygX8wYG8ZyJ07CxA.png" /></figure><p>Let’s understand all the steps involved.</p><ol><li>Deploy two sample applications that use the same PVC to persist data but in different folder/mount points. Add some sample data to this volume by using two applications independently</li><li>Take a backup of volume using a volume snapshot</li><li>Copy the snapshot to another region (eu-west-2) where the second cluster is deployed</li><li>Use the copied snapshot to create VolumeSnapshotContent &amp; VolumeSnapshot in the second OCP cluster</li><li>Redeploy the same two applications that use the restored PVC and verify if the contents are restored</li></ol><p><strong>Prerequisites</strong></p><ol><li>As before, we will have two OpenShift clusters deployed in AWS, which should be in two different regions (us-east-1 &amp; eu-west-2).</li><li>Two different applications are installed and running on one of the clusters where we shall initiate application volume backup and restore it on another cluster.</li><li>A storage class compatible with CSI driver should be installed on both clusters, this is used to create PV/PVC mapping to EFS file system dynamically.</li></ol><p><a href="https://github.com/Naveen-Kamagani/pvc-restore">Link to git repo</a> that has code used in this blog</p><p>The subsequent section will walk us through the steps for backup and restoring the EFS file system, and subsequently using that to re-create PV/PVC there by preserving the state cross cluster/regions</p><h3><strong>Steps for backup</strong></h3><p><strong>Step 1. Create EFS</strong></p><ul><li>Create New EFS in AWS in `custom-vpc`. Customize the network of the EFS using subnets in the `custom-vpc` and assign security groups as needed, check the screen-shots below</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WKjvyopZMEfiW8JB42jNlg.png" /></figure><p><strong>Step 2. Deploy two sample application that uses same pvc to persist data, add sample data to this volume from both the application</strong></p><p>We deploy two simple applications with pod (test-pod-1 &amp; test-pod-2) and a PVC (pvc-001) attached to both the pod to persist data. These applications are installed on the first cluster running on AWS, us-east-1 region.</p><pre>oc apply -f csi-app-storage-class.yaml<br>oc apply -f csi-app-pvc.yaml<br>oc apply -f csi-app-pod.yaml</pre><p>Storage Class:-</p><p>Create storage class pointing to EFS created</p><pre>kind: StorageClass<br>apiVersion: storage.k8s.io/v1<br>metadata:<br>  name: efs-sc-bcdr<br>provisioner: efs.csi.aws.com<br>volumeBindingMode: Immediate<br>reclaimPolicy: Delete<br>parameters:<br>  provisioningMode: efs-ap<br>  fileSystemId: fs-0e9747b9f471e9d94<br>  directoryPerms: &#39;755&#39;<br>  basePath: /dynamic_provisioning<br>  gidRangeEnd: &#39;1001051000&#39;<br>  gidRangeStart: &#39;1001050001&#39;</pre><p>Pod definition:-</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: app1<br>  namespace: csi-app<br>spec:<br>  containers:<br>  - name: app1<br>    image: busybox<br>    command: [&quot;/bin/sh&quot;]<br>    args: [&quot;-c&quot;, &quot;sleep 1000&quot;]<br>    volumeMounts:<br>    - name: persistent-storage<br>      mountPath: /data1<br>  volumes:<br>  - name: persistent-storage<br>    persistentVolumeClaim:<br>      claimName: efs-claim<br><br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: app2<br>  namespace: csi-app<br>spec:<br>  containers:<br>  - name: app1<br>    image: busybox<br>    command: [&quot;/bin/sh&quot;]<br>    args: [&quot;-c&quot;, &quot;sleep 1000&quot;]<br>    volumeMounts:<br>    - name: persistent-storage<br>      mountPath: /data2<br>  volumes:<br>  - name: persistent-storage<br>    persistentVolumeClaim:<br>      claimName: efs-claim</pre><p>PVC definition:-</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: efs-claim<br>  namespace: csi-app<br>spec:<br>  accessModes:<br>    - ReadWriteMany<br>  storageClassName: efs-sc-bcdr<br>  resources:<br>    requests:<br>      storage: 2Gi</pre><ul><li>Exec in the pod and add data on both pods</li></ul><pre>kubectl -n csi-app exec -ti app1 bash<br>while true; do echo -n &quot;FOOBARBAZ1 &quot; &gt;&gt; /data1/foobar; done<br>cksum /data1/foobar <br>&lt;&lt;&gt;&gt; <br><br>kubectl -n csi-app exec -ti app2 bash<br>while true; do echo -n &quot;FOOBARBAZ2 &quot; &gt;&gt; /data2/foobar; done<br>cksum /data2/foobar</pre><ul><li>Make a note of cksum which can be used to verify restored data</li></ul><p><strong>Step 3</strong>: <strong>Create an On-demand backup job for the EFS file system using AWS Backup </strong><br>Start by logging into your AWS account and navigating to AWS Backup. Once there, locate the “Backup Vaults” section.</p><ul><li>Create a backup vault say bcdr-default</li><li>Create on-demand backup job using the vault created above and the EFS storage that was created in Step 1</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*1nxfng3nBm-x-0QIoqq7Aw.png" /></figure><ul><li>This backup-job will create a recovery point as shown in the screenshot below</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Sk-J0Hs-TQoa745jT4Bbg.png" /></figure><ul><li>Use the recovery point to replicate data in the DR region (eu-west-2) where the second cluster is deployed</li><li>We can copy this recovery point using the AWS console as per the following image. This requires the destination region and description on the page and this will create a recovery point in the second region which is a copy of the first one.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*ehIWftR3smOISCJdDOHJyA.png" /></figure><ul><li>This recovery point ARN we will use to restore data in the new cluster in the DR region</li><li>In case volume has to be restored in the same region, recovery point created as part of backup job can be used</li></ul><h4>Recover in different regions</h4><h4>Steps for restore</h4><p><strong>Step 1</strong>: <strong>Navigate to AWS Backup </strong><br>Start by logging into your AWS account and navigating to AWS Backup. Once there, locate the “Backup Vaults” section.</p><p><strong>Step 2: Choose Your Backup Vault </strong><br>Backup vaults are secure storage containers designed to store and manage copies of data or information for the purpose of backup and recovery in AWS. Select the specific backup vault you want to work with. In this example, we’ll use “bcdr-default.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PToP8gmw6AhWrxeKvyBCwA.png" /></figure><p><strong>Step 3: Locate Recovery Points </strong><br>In AWS Backup, recovery points represent specific instances in time when a backup of your data or resources is taken, serving as restore points that allow you to recover data to a particular state captured during the backup process. Within the chosen backup vault, find the “Recovery Points” section. Here, you can search for the desired Elastic File System (EFS) by name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-CluOHQNSBRfshmnrl3A3g.png" /></figure><p><strong>Step 4: Select the Latest Backup Recovery Point </strong><br>Identify the latest backup recovery point for your EFS. Click on it to proceed with the restoration process.</p><p><strong>Step 5: Initiate the Restore</strong><br>Once you’ve selected the desired recovery point, look for the “Restore” option. Click on it to initiate the restoration process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JyRL-CkitMoJ7GoF6jY-xw.png" /></figure><p><strong>Step 6: Access EFS in AWS Console </strong><br>Navigate to the EFS section in the AWS Console. Locate and click on the newly restored EFS file system by selecting its unique ID.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QfGRg-keSsTnZcyUBY6pjA.png" /></figure><p><strong>Step 7: Navigate to Network Settings </strong><br>Since the cluster should be able to connect to this EFS, we have to modify the network configuration. Find the “Network” option associated with your restored file system within the EFS console. Click on it to access network management.</p><p><strong>Step 8: Select “Manage” for EFS-Network </strong><br>Look for the “Manage” option within the network settings. This is where you’ll configure the connection to your network.</p><p><strong>Step 9: Choose the DR Cluster VPC </strong><br>In the network management interface, select the appropriate VPC (Virtual Private Cloud) associated with your Disaster Recovery (DR) cluster. This step establishes the connection between your file system and the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fA6Vw7r2SEWteXp3gAXsvg.png" /></figure><p><strong>Step 10: Configure Subnets and Security Groups </strong><br>Specify the public subnets and default security groups for your EFS. This ensures the necessary network configurations are in place for optimal connectivity. To facilitate access to your EFS across different availability zones, add mount targets by providing the relevant Subnet ID and Security Groups. This step ensures your file system is accessible in multiple zones, enhancing its availability and fault tolerance.</p><p><strong>Step 11: Create an EC2 Instance </strong><br>Now, create an EC2 instance within the same cluster VPC. This instance will be used to access the files and directories restored from the backup. This manual step is for AWS since data gets copied under the dynamically generated folder, backup data then have to be moved to original folder structure for example expected folder name in our case will be /dynamic_provisioning/data1…</p><p><strong>Step 12: Mount EFS on EC2 </strong><br>On the newly created EC2 instance, mount the EFS file system to access your restored files and directories. This step establishes a connection between the EC2 instance and the EFS file system, allowing for seamless file access.</p><ol><li>Use the SSH command to connect to your EC2 instance:</li></ol><pre>ssh -i &lt;Your-EC2-Key.pem&gt; ec2-user@&lt;EC2-Public-IP&gt;</pre><p>2. Mount EFS on the EC2 Instance Run the following commands on your EC2 instance, replacing &lt;NEW-FILESYSTEMID-DNS&gt; with the DNS Name from your newly restored EFS in AWS:</p><pre>mkdir ~/efs-mount-point-2<br>sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport &lt;NEW-FILESYSTEMID-DNS&gt;:/ ~/efs-mount-point-2<br>cd ~/efs-mount-point-2/</pre><p>Explore Restored Files i.e., explore the contents of your restored EFS by navigating through the directories:</p><pre>cd aws-backup-restore_2023–02–16T07–42–20–609987Z<br>ls<br>cd dynamic_provisioning<br>ls<br>pvc-480c2fa1-ec53–4f19–96cb-7c80c2659ea</pre><p>Move Files for Recreation of Mount Points<br>Run the following commands to create a directory and move files in the dynamic_provisioning directory.<br>After moving the files, use the latest path to create access points in EFS.</p><pre>sudo mkdir -p ~/efs-mount-point-2/dynamic_provisioning/restore<br>sudo mv ~/efs-mount-point-2/aws-backup-restore_2023-02-16T07-42-20-609987Z/dynamic_provisioning/* ~/efs-mount-point-2/dynamic_provisioning/restore<br>cd /home/ec2-user/efs-mount-point-2/dynamic_provisioning/restore</pre><p>5. Creating access points in the restored EFS.<br>You can create an access point in EFS by following the below example screenshots.</p><p>Replace the name and root directory path according to your cluster paths</p><p><strong>Step 5. Create application PVC in the second cluster using newly created VolumeSnapshot</strong> <strong>and verify the restore</strong></p><p>Last, create PODs representing two different application in the second OCP cluster that uses newly created PVC pointing to restored data using the same POD definition.</p><p>Log into each POD and check if data is correctly restored at the mount point using volume snapshot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y8eTgCtAkusX3uODGdEn-A.png" /></figure><p>Change the user id and owner group id according to your resource user id and group id in your cluster</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ko8-7FPsFUcts0zGf6h3lQ.png" /></figure><p>Change the Owner User Id, Group Id values according to your resource Owner user id and group id in the cluster</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wYsChzB8qqT7rh9kuNDmJw.png" /></figure><p>After configuring(modify the above fields according to your requirements) all details you can click on “create access point” option to create access points.</p><p><strong>Configuring Kubernetes Storage for Restored Amazon EFS Data</strong></p><p>After connecting your restored Amazon EFS to a network and accessing files on an EC2 instance, the next step is to integrate this data into your Kubernetes environment. Follow these straightforward YAML configurations to create a StorageClass, Persistent Volume (PV), and Persistent Volume Claim (PVC) for your EFS data in Kubernetes.</p><p><strong>Step 1</strong>: Create a StorageClass</p><p>Define a StorageClass using the following YAML configuration. Replace &lt;STORAGECLASSNAME&gt;, &lt;PATH&gt;, &lt;gidRangeStart&gt;,&lt;gidRangeEnd&gt; and &lt;FILESYSTEMID&gt; with appropriate values:</p><pre>kind: StorageClass<br>apiVersion: storage.k8s.io/v1<br>metadata:<br> name: &lt;STORAGECLASSNAME&gt;<br>provisioner: efs.csi.aws.com<br>parameters:<br> basePath: &lt;PATH&gt;<br> directoryPerms: ‘755’<br> fileSystemId: &lt;FILESYSTEMID&gt;<br> gidRangeEnd: &lt;gidRangeEnd&gt;<br> gidRangeStart: &lt;gidRangeStart&gt;<br> provisioningMode: efs-ap<br>reclaimPolicy: Delete<br>volumeBindingMode: Immediate</pre><p><strong>Note</strong>: Adjust the basePath according to the path of data directories (e.g., /dynamic_provisioning/restore) and also provide the gidRangeStart and gidRangeEndvalues according to the user range in your cluster. Ensure &lt;FILESYSTEMID&gt; is replaced with the newly restored EFS filesystem ID, and &lt;STORAGECLASSNAME&gt; is a unique name for your storage class.</p><p><strong>Step 2</strong>: Create Persistent Volume from Restored EFS</p><p>Pre-create Persistent Volumes (PV) using the following YAML configuration. Replace &lt;VOLUMENAME&gt;, &lt;FILESYSTEMID&gt;, &lt;ACCESSPOINTID&gt;, and &lt;STORAGECLASSNAME&gt;, &lt;STORAGE&gt;with appropriate values:</p><pre>kind: PersistentVolume<br>apiVersion: v1<br>metadata:<br>  name: &lt;VOLUMENAME&gt;<br>spec:<br>  capacity:<br>    storage: &lt;STORAGE&gt;<br>  csi:<br>    driver: efs.csi.aws.com<br>    volumeHandle: &#39;&lt;FILESYSTEMID&gt;::&lt;ACCESSPOINTID&gt;&#39;<br>  accessModes:<br>    - ReadWriteMany<br>persistentVolumeReclaimPolicy: Delete<br>storageClassName: &lt;STORAGECLASSNAME&gt;<br>volumeMode: Filesystem</pre><p>Ensure &lt;FILESYSTEMID&gt; corresponds to the restored volume ID or EFS ID, and &lt;ACCESSPOINTID&gt; corresponds to each access point pointing to respective folders or PVs.</p><p><strong>Step 3</strong>: Create Persistent Volume Claim</p><p>Define a Persistent Volume Claim using the following YAML configuration. Replace &lt;PVC-NAME&gt;, &lt;NAMESPACE&gt;, &lt;VOLUMENAME&gt;, and &lt;STORAGECLASSNAME&gt;,&lt;STORAGE&gt;with appropriate values:</p><pre>kind: PersistentVolumeClaim<br>apiVersion: v1<br>metadata:<br>  annotations:<br>    volume.beta.kubernetes.io/storage-provisioner: efs.csi.aws.com<br>    volume.kubernetes.io/storage-provisioner: efs.csi.aws.com<br>  name: &lt;PVC-NAME&gt;<br>  namespace: &lt;NAMESPACE&gt;<br>spec:<br>  accessModes:<br>    - ReadWriteMany<br>  resources:<br>    requests:<br>      storage: &lt;STORAGE&gt;<br>  volumeName: &lt;VOLUMENAME&gt;<br>  storageClassName: &lt;STORAGECLASSNAME&gt;<br>  volumeMode: Filesystem</pre><p>Ensure &lt;VOLUMENAME&gt; matches the name given to the PV and &lt;STORAGECLASSNAME&gt; is the same as defined in the StorageClass.</p><p><strong>Step 4</strong>: Configure PV and PVC in Deployments</p><p>Once the PV and PVC are bound, configure them in your deployments to validate the data. Ensure that the paths and configurations align with your specific use case.</p><p><strong>Step 5</strong>: Restore/redeploy pods and other related resources</p><p>We can either use velero restore or redeploy pods which should mount to restored PVC’s, exec inside these pods, and verify if the data is the same.</p><p>For this blog we deployed pods, but for complex deployment this may not be the option, in those cases we can backup/restore kubernetes objects using velero.</p><p><strong>Conclusion:</strong></p><p>In the above blog post, we demonstrated how to take a volume backup for the EFS file system, move it into a different region, and recreate the volume using that backup.</p><p>Taking EFS backup and restoring is a bit complex and involves lots of manual steps, we tried to break those down as much as possible.</p><p>We managed to automate a few of the steps and will share those in a different blog. Meanwhile, we would love to hear what others have tried and if this is useful in any ways.</p><p>For native Kubernetes, we continue to use Velero but purposefully left those details to keep our focus on EFS volume backup/restore.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c6c86bcbe72" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes volume backup for disaster recovery]]></title>
            <link>https://medium.com/@amitabhprasad/kubernetes-volume-backup-for-disaster-recovery-56a5facee7fe?source=rss-912e2cec2888------2</link>
            <guid isPermaLink="false">https://medium.com/p/56a5facee7fe</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[backup]]></category>
            <category><![CDATA[volume]]></category>
            <category><![CDATA[disaster-recovery]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Amitabh Prasad]]></dc:creator>
            <pubDate>Wed, 15 Dec 2021 10:23:21 GMT</pubDate>
            <atom:updated>2021-12-15T10:23:21.041Z</atom:updated>
            <content:encoded><![CDATA[<p>Authors: <a href="https://medium.com/u/f1ef3fe1e43f">Ranjeet Singh</a>, <a href="https://medium.com/u/912e2cec2888">Amitabh Prasad</a></p><p>Containers have always been considered a great fit for stateless applications. But most real-world use case requires some sort of data persistence. To support such use cases Kubernetes introduced the concept of stateful workloads.</p><p>Some of the cloud-native applications we recently developed uses in cluster databases (such as MongoDB deployed using their operators) to take advantage of resilience, auto-scale, and other life cycle features that made Kubernetes goto platform for cloud deployment.</p><p>When running in-cluster DB it’s equally important to take regular backups. While for native Kubernetes resources taking etcd backup is sufficient and tools such as Velero does an excellent job to do backup/restore of Kubernetes resources, the challenge comes with volume backup, though Velero does support volume backup using restic but that’s a very rudimentary way of taking backups moreover restic backup always does full backups which may result in extended RTO’s among other complexities, as volume of data grows.</p><p>In this blog post, we will show one more way of taking volume backups i.e. by using volume snapshots in the AWS cloud. Please note at the time of writing volume backups relies heavily on cloud providers and Kubernetes inherently does not support DR with volume snapshots i.e. it doesn’t support the recovery of data in a new cluster in a new region.</p><p>Thankfully most of the cloud providers do support volume snapshot, moving snapshot to recovery region and restore using those snapshots.</p><p>Here is the <a href="https://velero.io/docs/v1.5/supported-providers/">list</a> of cloud providers that supports volume snapshot.</p><p>Some of the Kubernetes components used in this blog are</p><p><a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">Persistent Volume (PV)</a></p><p>The abstraction that represents physical storage.</p><p><a href="https://kubernetes.io/docs/concepts/storage/storage-classes/">StorageClass</a></p><p>The type of storage from which a PV is created. For example a PersistenceVolumeClass can be from an AWS gp2 or io1. It can also be from a local disk with specific RAID options or an NFS partition.</p><p><a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">PersistentVolumeClaim (PVC)</a></p><p>A claim to a specific resource on a PV. The PVC consumes specific size and access modes and other characteristics of the PV. In order for a pod to mount a PV storage type it must first create a claim upon that PV. While in the course of mounting storage on k8s pods it operates as though the PVC is the volume while in fact the PVC itself does not represent any physical storage but a right to use physical storage.</p><p><a href="https://kubernetes.io/docs/concepts/storage/volume-snapshots/#volume-snapshot-contents">VolumeSnapshotContent</a></p><p>This is similar to the PV in that it represents an actual snapshot of data on some physical storage. This is typically created from a PVC but can also be pre-provisioned.</p><p><a href="https://kubernetes.io/docs/concepts/storage/volume-snapshots/">VolumeSnapshot</a></p><p>The VolumeSnapshot relationship to the VolumeSnapshotContent is akin the PVC relationship to the PV. The VolumeSnapshotContent is the physical storage while the VolumeSnapshot is the interface to it. The VolumeSnapshot represents the request for a snapshot from a PVC and it maintains the status of creating the snapshot from the data held by the PVC onto the VolumeSnapshotContent. It also can be used as a source for creating a PVC from the contents of its bound VolumeSnapshotContent.</p><p><a href="https://kubernetes.io/docs/concepts/storage/volume-snapshot-classes/">VolumeSnapshotClass</a></p><p>This is similar to the SnapshotClass in that it describes attributes of the VolumeSnapshotContent. The most important attribute in this case is the ebs.csi.aws.com driver.</p><p>The diagram below represents the use case we will be using for this blog</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*AoVV33mExAYsL4ztWCKb6g.png" /></figure><p>Let’s understand all the steps involved .</p><ol><li>Deploy sample application that uses pvc for persistency and add some sample data to this volume</li><li>Take backup of volume using volume snapshot</li><li>Copy the snapshot to another region (eu-west-2) where second cluster is deployed</li><li>Use AWS copied snapshot to create VolumeSnapshotContent &amp; VolumeSnapshot in second OCP cluster</li><li>Create application PVC in second cluster using newly created VolumeSnapshot and verify restore</li></ol><p><strong>Prerequisites</strong></p><ol><li>We have two OpenShift clusters deployed in AWS and these should be in two different regions (us-east-1 &amp; eu-west-2).</li><li>An application is installed and running on one of the clusters where we shall initiate application volume backup and restore it on another cluster.</li><li>A storage class compatible to CSI driver should be installed on both the clusters.</li><li>Ensure that Volume snapshot CRD’s &amp; volume snapshot controller are installed on these two clusters, if not please execute following steps.</li></ol><ul><li>kubectl apply -f <a href="https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml">https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml</a></li><li>kubectl apply -f <a href="https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml">https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml</a></li><li>kubectl apply -f <a href="https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml">https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml</a></li></ul><p>5. VolumeSnapshotClass should be also created in both the clusters</p><ul><li>Make sure to use `deletionPolicy` as Retain, else snapshot will be deleted from AWS as soon as volumesnapshot is deleted either by deleting application namespace or when kubernetes cluster is deleted.</li></ul><p><strong>The Solution:-</strong></p><p>We now execute all the following steps sequentially to demonstrate the backup &amp; restore of an application with its volume (PVC) using AWS snapshot.</p><p><a href="https://github.com/amitabhprasad/volume-snapshot">Link to git repo</a> that has code used in this blog</p><p><strong>Step 1. Deploy sample application that uses pvc for persistency and add some sample data to this volume</strong></p><p>We deploy a simple application with a pod (test-pod) and a PVC (pvc-002) attached to this pod for data persistency. This application is installed on first cluster running on AWS, us-east-1 region.</p><p>Pod definition:-</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: test-pod<br>  namespace: csi-app<br>  labels:<br>    app: test-pod<br>spec:<br>  containers:<br>  - image: quay.io/libpod/ubuntu<br>    name: test-pod<br>    command: [&quot;sleep&quot;,&quot;1000&quot;]<br>    volumeMounts:<br>    - mountPath: /myvolumepath<br>      name: mydata<br>  volumes:<br>  - name: mydata<br>    persistentVolumeClaim:<br>      claimName: pvc-002</pre><p>PVC definition:-</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  namespace: csi-app<br>  name: pvc-002<br>spec:<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 1Gi<br>  storageClassName: gp2-csi</pre><p>Then we add some sample data to the application to its mount point (/myvolumepath) with a sample file, sample.txt</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*-sWDh6JSJmtnNVhOY5byDg.png" /></figure><p><strong>Step 2. Take backup of volume using volume snapshot</strong></p><p>Now, we create a volume snapshot that will take backup of application PVC and this will create a snapshot in AWS. We run the following command to create a snapshot of PVC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*m5Y4E0kuSS-BRI0MsJnFCw.png" /></figure><p>VolumeSnapshot definition-</p><pre>apiVersion: snapshot.storage.k8s.io/v1beta1<br>kind: VolumeSnapshot<br>metadata:<br>  name: pvc002-snapshot<br>spec:<br>  volumeSnapshotClassName: csi-aws-vsc<br>  source:<br>    persistentVolumeClaimName: pvc-002</pre><p>Once we create a volume snapshot for PVC, a snapshot is also created in the same region, us-east-1 in the AWS environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*pQfryoAbootk0jVcFZ-ipQ.png" /></figure><p><strong>Step 3. Copy the snapshot to another AWS region (eu-west-2) where the second cluster is deployed</strong></p><p>We can copy this snapshot using the AWS console as per the following image. We need to specify the destination region and description on the page and then a snapshot will be created in the second region which is a copy of the first one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*-L7ctUCr0KCaTYjZzgvoGQ.png" /></figure><p>Copied snapshot is created in the eu-west-2 region in AWS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*YH2I2xxyewIt0MzV2DVs4g.png" /></figure><p><strong>Step 4. Use AWS copied snapshot to create VolumeSnapshotContent &amp; VolumeSnapshot in the second OCP cluster which is our recovery cluster.</strong></p><p>Now, we create a VolumeSnapshotContent in the second OCP cluster deployed in the eu-west-2 region in AWS using the newly copied snapshot.</p><p>We use the following command to create it. Notice, we mention <strong>Snapshot ID for snapshotHandle</strong> parameter in VolumeSnapshotContent definition. We can get this value (Snapshot ID starts with <strong>snap-</strong>) on the AWS console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dit8mnwlVZ-OVykJz4pdjg.png" /></figure><p>VolumeSnapshotContent definition-</p><pre>apiVersion: snapshot.storage.k8s.io/v1<br>kind: VolumeSnapshotContent<br>metadata:<br>  name: test-snapshot-content<br>spec:<br>  deletionPolicy: Delete<br>  driver: ebs.csi.aws.com<br>  source:<br>    snapshotHandle: snap-026780e8ff3bd6e74<br>  volumeSnapshotClassName: csi-aws-vsc<br>  volumeSnapshotRef:<br>    name: restore-snapshot<br>    namespace: csi-app</pre><p>Next, we create VolumeSnapshot on the same second OCP cluster using the newly created VolumeSnapshotContent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YFsNqFsWHJuVVUB2d6Zxog.png" /></figure><p>VolumeSnapshot definition:-</p><pre>apiVersion: snapshot.storage.k8s.io/v1beta1<br>kind: VolumeSnapshot<br>metadata:<br>  name: restore-snapshot<br>spec:<br>  source:<br>        volumeSnapshotContentName: test-snapshot-content</pre><p><strong>Step 5. Create application PVC in the second cluster using newly created VolumeSnapshot</strong> <strong>and verify the restore</strong></p><p>We need to specify the VolumeSnapshot name and storage class name in the spec section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*b8ghwEnftguxV9jVWch8eQ.png" /></figure><p>PVC definition:-</p><pre>---<br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  namespace: csi-app<br>  name: pvc-002<br>spec:<br>  storageClassName: gp2-csi<br>  dataSource:<br>    name: restore-snapshot<br>    kind: VolumeSnapshot<br>    apiGroup: snapshot.storage.k8s.io<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 1Gi</pre><p>Last, create POD in the second OCP cluster that uses newly created PVC pointing to restored data using the same POD definition.</p><p>Log into the POD and check if data is correctly restored at the mount point using volume snapshot.</p><p>We can see sample.txt file is present at the mount path, which has the same data that we have written in the application running on the first cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*-sWDh6JSJmtnNVhOY5byDg.png" /></figure><p><strong>Conclusion:</strong></p><p>In the above blog post, we demonstrated how we can take a volume snapshot, move it into a different region, and recreate volume using the same snapshot. Snapshot can be taken using Velero as well, but moving to different region restore in new cluster/region doesn’t work out of the box with Velero. This was one of the ways to meet our requirement for addressing DR for volumes.</p><p>For native Kubernetes, we continue to use Velero but purposefully left those details to keep our focus on volume backup/restore.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56a5facee7fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>