<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by RK Kuppala on Medium]]></title>
        <description><![CDATA[Stories by RK Kuppala on Medium]]></description>
        <link>https://medium.com/@yogirk?source=rss-b3dbb13ab1fb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8h6x8IMKwq5CXAAHTRwsxg.jpeg</url>
            <title>Stories by RK Kuppala on Medium</title>
            <link>https://medium.com/@yogirk?source=rss-b3dbb13ab1fb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:56:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yogirk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[HPC Clusters on GCP — Running LS-Dyna jobs on Slurm using Intel MPI]]></title>
            <link>https://blog.thecloudside.com/hpc-clusters-on-gcp-running-ls-dyna-jobs-on-slurm-using-intel-mpi-7e02e4410511?source=rss-b3dbb13ab1fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e02e4410511</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[slurm]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[lsdyna]]></category>
            <category><![CDATA[intel]]></category>
            <dc:creator><![CDATA[RK Kuppala]]></dc:creator>
            <pubDate>Mon, 03 Oct 2022 20:07:00 GMT</pubDate>
            <atom:updated>2022-10-26T07:55:19.883Z</atom:updated>
            <content:encoded><![CDATA[<h3>HPC Clusters on GCP — Running LS-Dyna jobs on Slurm using Intel MPI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gmI45XJl06NMTIcARDMbHQ.png" /></figure><p>We recently started helping one of our clients exploring a migration from a datacenter hosted HPC workload that uses Sun Grid Engine to a Slurm HPC cluster in cloud. Creating a slurm cluster itself is not a difficult task in GCP, thanks to the multiple options available. There is a <a href="https://console.cloud.google.com/marketplace/product/schedmd-slurm-public/schedmd-slurm-gcp?_ga=2.2676208.-1337277848.1658906472">marketplace offering</a> that lets you spin up a cluster quickly, or SchedMD’s <a href="https://github.com/SchedMD/slurm-gcp">terraform template</a> is easy to get started with. There is also <a href="https://cloud.google.com/hpc-toolkit/docs/overview">Cloud HPC Toolkit</a>, which makes deploying a highly customizable and production grade cluster easy.</p><p>An HPC cluster is not just the scheduler, but the actual software that runs on it. In this case, among other things, we had to make sure that we could run distributed <strong>LS-Dyna jobs</strong> using <strong>Intel MPI</strong>. A lot of things could go wrong while trying to make these systems work together — a Slurm cluster running on Google Compute Engine, scheduling LS-Dyna jobs with Intel message passing interface (MPI), and they did go wrong. We thought it’d be beneficial to document how we went about it for the future random internet stranger, just as we benefitted from the random internet strangers while trying to make this whole thing work. In this blog post, we will set up a Slurm cluster on GCE from scratch with Cloud filestore as the shared storage, install and configure Intel MPI, LD-Dyna and finally run a sample job.</p><h3>Deploy a Slurm cluster</h3><p>As mentioned above, there are multiple (easy) ways you can create a Slurm cluster on GCP like the marketplace, Terraform template, or HPC Toolkit. For this blog post, we will use <a href="https://github.com/SchedMD/slurm-gcp">SchedMD’s terraform template</a> to build a basic cluster. You may create a VPC, Cloud Filestore as part of the template itself, but in our example, we are going to create them separately. Let’s get started.</p><p>If you already have a VPC and subnets, feel free to skip this section. If you don’t have one already, follow along.</p><p>Configure your project and region in gcloud</p><pre>gcloud auth login <br>gcloud config set project YOUR-PROJECT-NAME<br>gcloud config set compute/region asia-south1</pre><p>We will create a VPC and one subnet in Mumbai region</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a2e0e2e5653de5f4774c229aedc869b/href">https://medium.com/media/6a2e0e2e5653de5f4774c229aedc869b/href</a></iframe><pre>gcloud compute --project=projectname firewall-rules create slurm-vpc-internal --direction=INGRESS --priority=1000 --network=slurm-vpc --action=ALLOW --rules=all --source-ranges=10.10.0.0/24</pre><p>You might also want to create a NAT gateway for this VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ml2Tk5ldYC6qgn69o9t-4g.png" /></figure><p>Go to cloud filestore service in GCP and create a basic file share choosing connectivity with the VPC we just created above. Also, configure the “private connection” under “advanced network options” to allow slurm nodes to freely mount the fileshare. I am naming this file share as slurmfs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZN0EN5R8rluJUtdVEXNOfw.png" /></figure><p>Let’s now use the terraform template to create a Slurm cluster. We will be creating a “basic” and small cluster with 2 compute nodes in debug partition. You might want to use <a href="https://cloud.google.com/shell">Cloud Shell</a> for running terraform apply.</p><pre>git clone <a href="https://github.com/SchedMD/slurm-gcp.git">https://github.com/SchedMD/slurm-gcp.git</a> &amp;&amp; cd slurm-gcp/terraform/slurm_cluster/examples/slurm_cluster/cloud/basic</pre><p>You may choose the cluster type that makes the best sense for your scenario. For the sake of this blog post, we will choose a basic cluster. Edit the example.tfvars using your favorite text editor with the project, VPC, filestore, and other necessary details.</p><p>Under network storage, fill in the cloud filestore details you have created above</p><pre>network_storage = [</pre><pre>{<br>server_ip     = &quot;10.x.x.x&quot;<br>remote_mount  = &quot;/slurmfs&quot;<br>local_mount   = &quot;/data&quot;<br>fs_type       = &quot;nfs&quot;<br>mount_options = null<br>},<br>]</pre><p>Similarly, the login</p><pre>login_network_storage = [<br>{<br>server_ip     = &quot;10.x.x.x&quot; #change this depending on your env<br>remote_mount  = &quot;/slurmfs&quot;<br>local_mount   = &quot;/data&quot;<br>fs_type       = &quot;nfs&quot;<br>mount_options = null<br>},<br>]</pre><p><a href="https://gist.github.com/rk-cloudside/923ed6c23435a17b7216085294cb4a5c">Here is</a> my Below is my example.tfvars file. You can make changes accordingly.</p><p>Run terraform</p><pre>terraform init<br>terraform apply -var-file=example.tfvars</pre><p>If everything went well, you should see the slurm cluster up and running on your GCP console. Connect to controller node (I am assuming you have the necessary firewall rules created allowing IaP access via SSH).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zFHxJHp3_1t0JaRVf-vUaA.png" /></figure><p>My slurm cluster is now up and running and just so that you can follow along, here is what my cluster looks like. I have a controller node, login node and 2 compute nodes in the debug partition</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OVCraRonSP2NVNZbqw2wRw.png" /></figure><h3>Intel MPI (mpirun) and Hostbased Passwordless SSH</h3><p>Before we get to the reason behind configuring hostbased SSH, we should understand the <a href="https://hpc-wiki.info/hpc/MPI">MPI</a> (Message Passing Interface) in <a href="https://hpc-tutorials.llnl.gov/mpi/what_is_mpi/">HPC world</a>. An MPI is a library spec for parallel computing architectures which allow communication of information between various nodes and clusters. In our scenario, we are going to run LS-Dyna jobs using <a href="https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html">Intel MPI</a> library, mpirun .</p><p>If you are new to Slurm + Intel MPI combination, you are likely to spend a ton of time trying to trigger distributed jobs and run into multiple obscure error messages. Please note that Intel MPI requires <a href="https://www.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/installation-and-prerequisites/prerequisite-steps.html">passwordless authentication</a> between nodes (not just between the controller and compute nodes). Intel provides a utility sshconnectivity.exp along with the installation files, but I was not able to get it to work. I ended up configuring hostbased authentication between the nodes.</p><p>Here is a <a href="https://hea-www.harvard.edu/~fine/Tech/ssh-host-based.html">great explanation</a> of how hostbased SSH authentication works. Also <a href="https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Host-based_Authentication">this cookbook</a> will come in handy too. We will collect the public keys of all nodes in the cluster and then distribute them to /etc/ssh/ssh_known_hosts in each node.</p><p>SSH to the login node</p><pre>gcloud compute ssh --zone &quot;asia-south1-a&quot; &quot;hpc-login-zd18txv1-001&quot;  --tunnel-through-iap --project &quot;PROJECTNAME&quot;</pre><p>Collect all the public keys using sh_keyscan and save them on a temporary file, preferably to the cloud filestore mount. In this post, it’s mounted as /data/ on each node. This will allow you to easily copy the file between the nodes.</p><pre>sudo -i<br>ssh-keyscan -t rsa hpc-controller hpc-debug-worker-0 hpc-debug-worker-1 &gt; /data/known_hosts</pre><p>You will end up with an output like this.</p><pre>hpc-controller ssh-rsa AAAAB3NzaC1yc2.........<br>hpc-debug-worker-0 ssh-rsa AAAAB3NzaC1yc2E...........<br>hpc-debug-worker-1 ssh-rsa AAAAB3NzaC1yc2..........</pre><p>To avoid running into any name resolution errors, add all variants of your hostname. Open the temporary hosts file and edit it with IP address, FQDN of your nodes. You can cat /etc/hosts to get the FQDN. After making these changes, I ended up with the following content in /data/known_hosts file</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27bacf6e2669114d46dc1800e6e0b31e/href">https://medium.com/media/27bacf6e2669114d46dc1800e6e0b31e/href</a></iframe><p>Now, copy this file on ALL nodes — <em>hpc-controller, hpc-debug-worker-0, hpc-debug-worker-1</em><strong> </strong>to /etc/ssh/ssh_known_hosts</p><pre># SSH to each VM and copy the file<br>sudo cp /data/known_hosts /etc/ssh/ssh_known_hosts</pre><p>Next, on all nodes, configure the following in /etc/ssh/ssh_config</p><pre>sudo tee -a /etc/ssh/ssh_config &lt;&lt; EOF<br>HostbasedAuthentication yes<br>EnableSSHKeysign yes<br>EOF</pre><p>Next, on each node — enable host-based authentication in /etc/ssh/<strong>sshd</strong>_config</p><pre>HostbasedAuthentication yes</pre><p>On all nodes, restart sshd — systemctl restart sshd</p><p>on all nodes, create the following file — vi /etc/ssh/shosts.equiv . In my case, I ended up with the following contents</p><pre>hpc-controller.c.PROJECTID.internal<br>hpc-debug-worker-0.c.PROJECTID.internal<br>hpc-debug-worker-1.c.PROJECTID.internal</pre><p>Finally, on each node, fix permissions</p><pre>chcon system_u:object_r:etc_t:s0 /etc/ssh/sshd_config /etc/ssh/ssh_config /etc/ssh/shosts.equiv /etc/ssh/ssh_known_hosts</pre><p>At this point, we are good to go, but I ran into ssh_keysign related errors, because the CentOS 7 image the terraform template was using, had incorrect permissions on the host keys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IO6C8TQxJB8SV4j5auQfrQ.png" /></figure><p>Fix permissions on each node</p><pre>sudo chgrp ssh_keys /etc/ssh/*_key<br>sudo chmod g+r /etc/ssh/*_key</pre><p>The updated permissions should look like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nDX5SKrs8gniG5NLKj9wWQ.png" /></figure><p>Now, test host-based authentication from the controller (or any node) ssh -vvv hpc-debug-worker-0 . Remember to use ssh as a non-root user.</p><h4>Install Intel MPI</h4><p>Now that we are done with host-based authentication, let’s setup Intel MPI on all the nodes</p><p>Download the stand-alone offline installer from <a href="https://www.intel.com/content/www/us/en/developer/articles/tool/oneapi-standalone-components.html#mpi">here</a></p><p>SSH to the hpc-controller node and download</p><pre>wget <a href="https://registrationcenter-download.intel.com/akdlm/irc_nas/18926/l_mpi_oneapi_p_2021.7.0.8711_offline.sh">https://registrationcenter-download.intel.com/akdlm/irc_nas/18926/l_mpi_oneapi_p_2021.7.0.8711_offline.sh</a></pre><pre>chmod a+x l_mpi_oneapi_p_2021.7.0.8711_offline.sh</pre><pre># start installation</pre><pre>./l_mpi_oneapi_p_2021.7.0.8711_offline.sh</pre><p>The installation is pretty much self-explanatory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O9dPR0ebzkuCQ2p_ltO_hA.png" /></figure><p>Thankfully, slurm takes care of propagating this installation to the other nodes as well.</p><p>source the env variables file and invoke the mpirun</p><pre>source intel/oneapi/setvars.sh <br>mpirun --version</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z2DCjtyZexrxVsnyWlIbuA.png" /></figure><h3>Install LS-Dyna</h3><p>LS-Dyna requires a license to run. In this blog post, we will trigger a Dyna 9.3.1 release with a network license. We assume that a license server already exists in your environment.</p><p>Download ls-dyna binary</p><pre>wget <a href="https://www.oasys-software.com/dyna/wp-content/uploads/2018/01/ls-dyna_mpp_s_R9_3_1_x64_centos65_ifort131_sse2_platformmpi.tar.gz">https://www.oasys-software.com/dyna/wp-content/uploads/2018/01/ls-dyna_mpp_d_R9_3_1_x64_centos65_ifort131_sse2_platformmpi.tar.gz</a></pre><pre>tar -xvf ls-dyna_mpp_d_R9_3_1_x64_centos65_ifort131_sse2_platformmpi.tar.gz</pre><pre>mv ls-dyna_mpp_d_R9_3_1_x64_centos65_ifort131_sse2_platformmpi lsdyna-931</pre><pre>chmod a+x lsdyna-931</pre><h3>Run LS-Dyna job using mpirun</h3><p>We can now test a distributed job using both srun and mpirun. There is a sample job on the internet that you can download and run</p><pre>wget <a href="https://hcc.unl.edu/docs/submitting_jobs/app_specific/job-examples/ls-dyna/ls-dyna.files/pendulum.k">https://hcc.unl.edu/docs/submitting_jobs/app_specific/job-examples/ls-dyna/ls-dyna.files/pendulum.k</a></pre><p>Running the distributed job usingsrun . We can invoke the job using srun, by specifying the option— mpi=pmi2 , this will make use of mpirun. You can wrap this in a SBATCH script and invoke it as well.</p><pre>source intel/oneapi/setvars.sh</pre><pre>export LSTC_LICENSE=network<br>export LSTC_LICENSE_SERVER=10.50.16.4 #this is our license server</pre><pre># use srun to invoke the job on specific nodes</pre><pre>srun --nodelist=hpc-debug-worker-0,hpc-debug-worker-1  --mpi=pmi2 lsdyna-931 i=pendulum.k</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LRnWmEIUpv_kFFtLtTesGg.png" /></figure><p>And it ran successfully!</p><p>You may also trigger the same job using mpirun directly, specifying slurm as the bootstrap option</p><pre>mpirun -bootstrap slurm -hosts hpc-debug-worker-0,hpc-debug-worker-1  ./lsdyna-931 i=pendulum.k</pre><p>Hope you find this useful! Happy high-performance computing! :)</p><p>Here are some resources I found useful</p><ul><li><a href="https://hea-www.harvard.edu/~fine/Tech/ssh-host-based.html">Understanding hostbased SSH authentication</a></li><li><a href="https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Host-based_Authentication">OpenSSH hostbased authentication cookbook</a></li><li><a href="https://wiki.fysik.dtu.dk/niflheim/SLURM">Super useful page about Slurm</a></li><li><a href="https://docs.01.org/clearlinux/latest/tutorials/hpc.html#set-up-password-less-ssh-access-and-pdsh-on-all-nodes">Building a HPC cluster — clearlinux docs</a></li><li><a href="https://slurm.schedmd.com/mpi_guide.html#intel_mpi">Intel MPI and Slurm</a></li><li><a href="https://www.hpc2n.umu.se/documentation/batchsystem/basic-submit-example-scripts">Sbatch and some samples</a></li><li><a href="https://help.rc.ufl.edu/doc/Sample_SLURM_Scripts">More Slurm examples</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e02e4410511" width="1" height="1" alt=""><hr><p><a href="https://blog.thecloudside.com/hpc-clusters-on-gcp-running-ls-dyna-jobs-on-slurm-using-intel-mpi-7e02e4410511">HPC Clusters on GCP — Running LS-Dyna jobs on Slurm using Intel MPI</a> was originally published in <a href="https://blog.thecloudside.com/">The Cloudside View</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Replicate RDS PostgreSQL to BigQuery using Datastream CDC]]></title>
            <link>https://blog.thecloudside.com/replicate-rds-postgresql-to-bigquery-using-datastream-cdc-99319375aa87?source=rss-b3dbb13ab1fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/99319375aa87</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[change-data-capture]]></category>
            <dc:creator><![CDATA[RK Kuppala]]></dc:creator>
            <pubDate>Wed, 21 Sep 2022 13:29:22 GMT</pubDate>
            <atom:updated>2022-09-21T13:29:22.395Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v306wT7h-8nG__eLQ_uTWw.png" /></figure><p>Continuing the series of posts we started with <a href="https://blog.thecloudside.com/replicate-rds-postgresql-to-bigquery-using-airbyte-cdc-9be9f294e71a">Replicate RDS PostgreSQL to BigQuery using Airbyte CDC</a>, in this post, we will cover the all-new Datastream for BigQuery.</p><p>GCP has recently released <a href="https://cloud.google.com/datastream-for-bigquery">Datastream for BigQuery</a>, which allows access to streaming data from MySQL, PostgreSQL, <a href="https://cloud.google.com/alloydb">AlloyDB</a>, and Oracle databases directly to BigQuery without needing an additional pipeline line Dataflow. This is pretty cool, and we took it for a spin with an RDS PostgreSQL source and BigQuery as the target. So let’s get started!</p><h4>How does it work?</h4><p>Datastream is a CDC tool that taps into pglogical replication for a PostgreSQL source. All you need to do is set the replica identity on the tables you are interested in replicating, create a publication and a replication slot and off you go!</p><h3>Prepare RDS PostgreSQL</h3><p>We assume that AWS and GCP VPCs have been already connected with a site-to-site VPN. Let’s create a dedicated user for Airbyte replication with minimum necessary privileges (read-only).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b2b03160ce8e572d7cb00df254aa19f3/href">https://medium.com/media/b2b03160ce8e572d7cb00df254aa19f3/href</a></iframe><h3>Publication, Replication slot, and Datastream</h3><p>We will now create a publication with all the tables we are interested in replicating to BigQuery. Before that, set the replica identity to default. Repeat this for all the tables you intend to sync</p><pre>ALTER TABLE tbl1 REPLICA IDENTITY DEFAULT;<br>.<br>.<br>ALTER TABLE tbl-n REPLICA IDENTITY DEFAULT;</pre><p>Create a publication with all the tables you intend to sync</p><pre>CREATE PUBLICATION pub_datastream <br>FOR TABLE myschema.table01 <br>,myschema.table02 <br>,myschema.table03 <br>,myschema.table04 <br>,myschema.table05;</pre><p>Create a replication slot</p><pre>SELECT pg_create_logical_replication_slot(&#39;datastream&#39;, &#39;pgoutput&#39;);</pre><p>Go to Datastream and start configuring. The options are self explanatory</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vD0DV8pEntZMGALbwNgcEA.png" /></figure><p>In the next step, you will be asked to configure a connection profile, which is basically how datastream connects to RDS. In our case, we had AWS to GCP VPN setup already so we used the internal IP address. You may use the public endpoint with firewall rules allowing access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CIOicRRCk4XxM-iMd_lUCA.png" /></figure><p>Next, fill the replication slot name and the publication name we created above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EY2FowS6DAHvIcaRIZVbWg.png" /></figure><p>Choose the tables you want to sync and similar to the source connection profile, configure a destination connection profile — BigQuery in this case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KH0Ck5iJDY_jwBMQ5x1p2Q.png" /></figure><p>Once you are done, you should see the summary screen like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Io3FYI0Uz6wi7As0AapLxA.png" /></figure><p>Click on “START” on the stream page. Sit back and relax, because it will initially do a backfill of all data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2HYr8vdxUWGHSh-HXvp-pw.png" /></figure><h3>Monitoring</h3><p>The great thing about Datastream to BigQuery integration is the simplicity of it. While being extremely simple to set up, it also has comprehensive monitoring, to track unsupported events, status of the sync, number of rows processed etc.,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z43iNwoj2LiZkVoTI0AJvQ.png" /></figure><p>You can also go to the stream dashboard and look at all the tables that are currently part of the sync and check their individual status. If needed, you can initiate a manual backfill (full load) at an individual table level</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l-J0osMSFtJIsfyQ9xOLYQ.png" /></figure><p>It also has useful data freshness tracker.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bRE85IQeyeM7_BylCIdqkw.png" /></figure><p>And best of all, the logviewer!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B7KEF2Df6hgKMH1kriFQDw.png" /></figure><p>Hope you found this useful! Happy data ingestion :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99319375aa87" width="1" height="1" alt=""><hr><p><a href="https://blog.thecloudside.com/replicate-rds-postgresql-to-bigquery-using-datastream-cdc-99319375aa87">Replicate RDS PostgreSQL to BigQuery using Datastream CDC</a> was originally published in <a href="https://blog.thecloudside.com/">The Cloudside View</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Replicate RDS PostgreSQL to BigQuery using AirByte CDC]]></title>
            <link>https://blog.thecloudside.com/replicate-rds-postgresql-to-bigquery-using-airbyte-cdc-9be9f294e71a?source=rss-b3dbb13ab1fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/9be9f294e71a</guid>
            <category><![CDATA[change-data-capture]]></category>
            <category><![CDATA[airbyte]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[RK Kuppala]]></dc:creator>
            <pubDate>Sun, 07 Aug 2022 17:19:39 GMT</pubDate>
            <atom:updated>2022-08-07T17:19:39.342Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E3B1JKxIihGQxZDtweboHA.png" /></figure><p>At Cloudside, one of the most recurring workloads that we implement for our clients is to build a data warehouse on the Google Cloud Platform using BigQuery. As a part of this, we work with multiple relational database sources like SQL Server, Oracle, MySQL, and PostgreSQL. Most of our clients would like to keep the data warehouse with the source database with minimal possible delay, so tapping into the native CDC feature would make sense.</p><p>While GCP has multiple tools and approaches that can do CDC like Datastream (currently Oracle and MySQL are supported, I hope PostgreSQL is in the works), <a href="https://cloud.google.com/data-fusion/docs/concepts/replication">Data Fusion replication</a> (currently Oracle, MySQL, and SQL Server are supported) and <a href="https://cloud.google.com/blog/products/databases/new-cloud-based-cdc-replication-across-databases">Dataflow</a>, we recently tested and implemented another opensource ETL tool, <a href="https://airbyte.com/">AirByte</a>. We like it for its simplicity, and ease of use. In a series of posts, we will look at the syncing PostgreSQL, SQL Server, and MySQL to BigQuery, starting with PostgreSQL in this post. Let’s get started!</p><h3>Deploy AirByte</h3><p>We will not go into the details of creating a VM and deploying AirByte in this post. It’s pretty straightforward, and you can take a look at the <a href="https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine/">step-by-step here</a> to spin up a GCE instance and install AirByte on it. Since AirByte doesn’t come with any authentication, I recommend deploying AirByte in an unmanaged instance group, behind the HTTP(s) Loadbalancer and enable the Identity Aware Proxy(IaP).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TLWCaGVoW2csea2c6VS8vg.png" /></figure><p><strong>IaP</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ND6sA7gWRwEKlKlnB9ZHhg.png" /></figure><h3>Prepare RDS PostgreSQL</h3><p>We assume that AWS and GCP VPCs have been already connected with a site-to-site VPN. Let’s create a dedicated user for Airbyte replication with minimum necessary privileges (readonly).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1cab99ca7e5bb1f68dc2dd80fb217103/href">https://medium.com/media/1cab99ca7e5bb1f68dc2dd80fb217103/href</a></iframe><h3>Publication, Replication Slot and AirByte Connections</h3><p>AirByte’s PostgreSQL CDC taps into native pglogical replication in PostgreSQL. At a high level, we create a publication for select tables, create a replication slot and configure a sync. Below is a representation of a sample AirByte Connection. It has a source with publication and replication slot specified, a destination, and a <strong>Connection </strong>that defines the tables that will be replicated, the frequency at which the sync runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jyJDdk5b9vfwnb58jaVL5w.png" /></figure><p>In a production environment, you are likely to sync more than a handful of tables and some of these tables might benefit from grouping — for example, sync all inventory-related tables every 30 min and sync transactions table every 5 min. I recommend splitting your source tables into a handful of categories like this so that you can have respective publications created in PostgreSQL. This helped us handle the connections better and also debug the respective pipeline instead of all tables in a single connection. Here is a representation of what it looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nknD9hf3JIJ7zBTaTIB06g.png" /></figure><p>So to summarize — we will create a separate publication in PostgreSQL for each group of tables, <strong>one </strong>BigQuery destination, and one AirByte connection each for a publication.</p><h4>Create Publication(s)</h4><blockquote><strong><em>Note</em></strong><em>: The AirByte documentation instructs you to create a replication slot first before creating publications. However, I ran into a debezium error which said “publication does not exist”. </em><a href="https://elephanttamer.net/?p=50"><em>Apparently the order matters</em></a><em>, so create publication(s) first and then create replication slo</em>t</blockquote><p>In the source RDS instance, set the replica identity to default. Repeat this for all the tables you intend to sync</p><pre>ALTER TABLE tbl1 REPLICA IDENTITY DEFAULT;<br>.<br>.<br>ALTER TABLE tbl-n REPLICA IDENTITY DEFAULT; </pre><p>Now, create publication(s)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/feab5e99c50c33d98f42f8b9fc71ce35/href">https://medium.com/media/feab5e99c50c33d98f42f8b9fc71ce35/href</a></iframe><p>Create a logical replication slot</p><pre>SELECT pg_create_logical_replication_slot(&#39;airbyte_slot&#39;, &#39;pgoutput&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OlKKdlJSjcAPHccTPRljoQ.png" /></figure><p>We can now go ahead and create AirByte sources, destinations, and connections to start the sync.</p><h3>AirByte Connections</h3><p>Log on to the AirByte web UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*veqIIrcUWzLlzHLCS4vATA.png" /></figure><p>Configure a destination first. We don’t need multiple destinations, because that will be redundant and we will anyway be writing all results to BigQuery datasets. Before we proceed further, you need to have the following</p><ul><li>A service account that has enough access to write to BigQuery and its credentials (we will use this to authenticate with BQ)</li><li>A Google Cloud Storage bucket. This will be used for staging data from the source before loading to BigQuery. The other option os writing directly to BigQuery without storing is not efficient.</li><li>HMAC credentials for the service account. Follow the instructions <a href="https://cloud.google.com/storage/docs/authentication/managing-hmackeys">here</a> to create.</li></ul><h4>Create a Destination</h4><p>Go to “Destinations” in AirByte and configure one. It’s UI and things are self-explanatory. here is my configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J_s8ylO9OAgcD-w_8PA2AQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VznaHh4Y3nICsiHHqpMIyw.png" /></figure><p>Save and close. Now, create a source for each of your publications.</p><h4>Create Data Sources</h4><p>In AirByte UI, go to sources and create. Again, the configuration is self-explanatory. Just note that the name of the publication will change for each data source you create (if you are using multiple publications). All other details remain the same. Here is a sample configuration of one of my data sources for inventory tables</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XYZvZn6J1Kry2Orp6QI3-w.png" /></figure><p>Under “Replication Method” — choose <strong>Logical Replication (CDC), </strong>and fill in the publication name, and replication slot name. Since this data source is for inventory tables, I am going to use the respective publication name. You can leave the initial waiting time in seconds to the default value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5iZReU4yAmrOQQl5CDT2ow.png" /></figure><h4>Setup a Connection</h4><p>Finally, it’s time to connect the source and the destinations we created above using a connection. In AirByte UI, go to “Connections” and configure.</p><p>In the first screen, use the existing source name that we created above</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x1zNHeE2CtIF8a8stpo5lQ.png" /></figure><p>Similarly, use the existing destination</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4nkkTTVRf4vSJjgyhO8f0Q.png" /></figure><p>Give the connection an appropriate name, and choose the frequency at which you would like to sync. Under streams, I prefer to “Mirror source structure” because it will essentially emulate schema.table to dataset.table in BigQuery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_sfnMSyTr-P9RdpBc4jClw.png" /></figure><p>You will also see a bunch of tables (the ones you added to this publication in one of the previous steps).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6rGuLkYOpwNDipGn2Fv4MQ.png" /></figure><p>PostgreSQL connector in AirByte supports <a href="https://docs.airbyte.com/integrations/sources/postgres#configuring-postgres-connector-with-change-data-capture-cdc">multiple sync modes</a> such as Full Refresh — Overwrite, Full Refresh — Append, Incremental — Append and incremental — Depuded History. In our case, we will choose Incremental + Deduped history because it will create two set of tables apart from the temporary tables. These two tables are Table_scd (a history table with all changes, its SCD type 1) and the target deduped table (similar to the source table).</p><p>That’s it. You can now repeat this process for all other publications you have created and choose the existing target BQ destination.</p><p>In my case, I ended up with 5 different sources (1 source for each publication)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-LxQUP7-a8cRgc3X82SU7w.png" /></figure><p>And 1 destination, BigQuery target used by all connections</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cNWTUQUIG3TU2IjAUDEhrA.png" /></figure><p>And 5 connections</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EChBQF7AcVZFJ2n09EurzQ.png" /></figure><p>One connection for each publication. I was able to successfully run the sync to BigQuery</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*F9IjsQqzRnutQFsgCx1Vpg.png" /></figure><p>That’s about it. I hope you found this useful. Happy replication! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9be9f294e71a" width="1" height="1" alt=""><hr><p><a href="https://blog.thecloudside.com/replicate-rds-postgresql-to-bigquery-using-airbyte-cdc-9be9f294e71a">Replicate RDS PostgreSQL to BigQuery using AirByte CDC</a> was originally published in <a href="https://blog.thecloudside.com/">The Cloudside View</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[VPN with Overlapping Networks on GCP using Aviatrix]]></title>
            <link>https://blog.thecloudside.com/vpn-with-overlapping-networks-on-gcp-using-aviatrix-462b329349c?source=rss-b3dbb13ab1fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/462b329349c</guid>
            <category><![CDATA[aviatrix]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[hybrid-cloud]]></category>
            <dc:creator><![CDATA[RK Kuppala]]></dc:creator>
            <pubDate>Sun, 23 Jan 2022 17:38:17 GMT</pubDate>
            <atom:updated>2022-01-23T18:04:49.764Z</atom:updated>
            <content:encoded><![CDATA[<p>We recently ran into an interesting scenario where we had to configure a site-to-site VPN between one of our customer’s VPC (let’s assume 10.1.0.0/24 ) and an on-premise network (again, the CIDR is 10.10.0.0/24). We all know that it’s not directly possible to configure connectivity between two networks with colliding CIDR ranges unless you use something like PNAT (Policy NAT) on both sides (or Policy twice NAT on one side). If you would like to know more about the problem of overlapping networks and possible solutions, read this excellent article from <a href="https://www.practicalnetworking.net/stand-alone/vpn-overlapping-networks/">practicalnetworking.net</a>.</p><p>After looking at multiple solutions both on and off Google Cloud Marketplace, we zeroed in on <a href="https://aviatrix.com/">Aviatrix</a>, because they were the first to respond to our queries via their chatbot on the website and it also turned out to be the right solution for us technically and cost-wise too.</p><p>What follows is a tutorial that explains how to connect two GCP VPCs of the same CIDR range (since I can’t easily emulate an on-prem network, let’s just assume one of these VPCs is on-premise) using Aviatrix.</p><h4>Prerequisites</h4><p>I assume that you are already familiar with Google Cloud Platform and have a project to try this on. This tutorial uses gcloud commands wherever possible instead of taking you through multiple screenshots of the GCP console to create VPCs etc. I also assume that you have an Aviatrix trial license. If not, you may <a href="https://a.aviatrix.com/free-trial/#:~:text=To%20start%20your%20free%2030,on%20your%20platform%20of%20preference.">request one here</a></p><h4>What will be build</h4><p>In this tutorial, we will deploy the following.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2P8sRj4fjGTOBTiNDbHHCw.png" /></figure><p>We will also configure the Aviatrix Controller, Aviatrix Gateway, VPN gateway on the on-premise VPC and a VPN tunnel. Below is the diagram that represents this setup at a high level</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NKS3PTPM39tG8AgtSV35yA.png" /><figcaption>made with Excalidraw</figcaption></figure><p>So let’s get started! :)</p><h3>Setup Aviatrix Controller</h3><p>I assume you already have gcloud setup. Authenticate and set the project</p><pre>gcloud auth login</pre><pre>gcloud config set project yourown-aviatrix-poc</pre><p>Create a management VPC and a subnet, this is where the Aviatrix controller goes</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c735cc2ea2ba6ca57c8231c8935a0466/href">https://medium.com/media/c735cc2ea2ba6ca57c8231c8935a0466/href</a></iframe><p>You need a firewall rule to allow yourself access to aviatrix GUI once it&#39;s provisioned. The below command opens access for 0.0.0.0/0 , but you should limit access to necessary IP addresses / CIDR ranges</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/61b5e559bf00487eab73a2711839db6b/href">https://medium.com/media/61b5e559bf00487eab73a2711839db6b/href</a></iframe><p>It’s a good idea to reserve a static public IP for your Aviatrix controller VM so that you can map it to a DNS record and also, you can freely start/stop the VM without having to lookup for the changed IP</p><pre>gcloud compute addresses create aviatrix-ip --region=asia-south1</pre><pre><em>//get the IP address you just created</em><br>gcloud compute addresses describe aviatrix-ip</pre><p>Create the Aviatrix controller VM image. Note that I am currently using a source as a public GCS bucket. You can get the latest source URI from Aviatrix <a href="https://docs.aviatrix.com/StartUpGuides/google-aviatrix-cloud-controller-startup-guide.html">documentation here</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/40c39e15ea707350bb747c31afe504a0/href">https://medium.com/media/40c39e15ea707350bb747c31afe504a0/href</a></iframe><p>Create a VM from this image, and use the public IP we created in the previous step.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/859330e889c1907daee747d9795d3f7b/href">https://medium.com/media/859330e889c1907daee747d9795d3f7b/href</a></iframe><p>Once the instance is up, access it on the browser using it’s public IP. The user name and password for initial login are Admin and the internal IP address of the controller VM respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sQdfOxLpvZBN4HS4yUMFcQ.png" /></figure><p>After logging in, go through the one-time onboarding process. Go to “Onboarding” on the menu and choose Google Cloud Platform</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ECFIrjpeckzKmwj50PVCkw.png" /></figure><p>Enter the customer ID here (you will receive this from your trial request, or via direct email, if you already purchased Aviatrix). Enter an account name you like, enter the GCP project name and upload a credential file of a service account that has enough access to launch instances, etc. You can check the detailed privileges needed at <a href="https://docs.aviatrix.com/HowTos/CreateGCloudAccount.html">this link</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3sG5AKMspxB7hA526OWPxQ.png" /></figure><p>We will come back to the Aviatrix Controller UI later, for now, let’s proceed with the next steps.</p><h3>Create VPCs that emulate production and on-premise networks</h3><p>We are going to create two VPCs, a “Production network” in the Mumbai region and an “On-premise network” in the Delhi region. We will create necessary firewall rules on both VPCs</p><p>Create prod-vpc-mumbai VPC , prod-subnet-mumbai subnet with the CIDR range 10.1.0.0/24 . Similarly, create onpre-vpc-delhi VPC, onprem-subnet-delhi with the same CIDR range, 10.1.0.0/24</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f993e82b83bdd298a922d5d376dae1ab/href">https://medium.com/media/f993e82b83bdd298a922d5d376dae1ab/href</a></iframe><p>We are going to create a couple of VMs: vm-01, vm-02 in “prod” and “on-prem” networks respectively to test connectivity. So it’s useful to whitelist IaP for secure SSH access. Let’s create those firewall rules</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f71361896433c68113b108163d02e05/href">https://medium.com/media/4f71361896433c68113b108163d02e05/href</a></iframe><p>Let’s go ahead and launch these VMs now.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58b2728a4e3c2f47da80eba623bd469a/href">https://medium.com/media/58b2728a4e3c2f47da80eba623bd469a/href</a></iframe><p>So now we have two VPCs with the same CIDR range. Next, we need to configure an aviatrix gateway in the “production network”.</p><h3>Configure Aviatrix spoke gateway in prod VPC</h3><p>Come back to the Aviatrix controller UI. We will now launch a spoke gateway in prod-vpc-mumbai VPC. A gateway is nothing but an appliance that runs in a VM.</p><p>Go to :<a href="https://35.200.158.220/#/transitnetwork/setup">https://your.controller.piblic.ip/#/transitnetwork/setup</a> , and scroll down to find the Spoke Gateway. Fill your respective details. You will see the account name you have created in the first step in the dropdown. We will choose “Enable SNAT” because our gateway is going to be the NAT gateway for the spoke VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QQRMdv2Xvb2xTRbIdjqefA.png" /></figure><p>Click on Create and this should launch a VM in the production VPC. It will take a few min, you can track progress here</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zPD8FT83EZG6cZwzz-qRjg.png" /></figure><p>While the gateway is being provisioned, let’s go ahead and create a VPN Gateway for our on-premise VPC.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/593b09102697edb487e912d68fa583e1/href">https://medium.com/media/593b09102697edb487e912d68fa583e1/href</a></iframe><p>The aviatrix gateway would have completed provisioning now. Take a note of its public IP address. We will go ahead and establish a tunnel from on-premise VPN gateway to the gateway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*35FJmiLClt6aPZ9PanIewA.png" /><figcaption>Note the tunnel is routebased and IKE preshared key can be generated by you.</figcaption></figure><p>Note that the remote network IP range is the virtual CIDR range of production VPC — 172.16.0.0/24. Go ahead and create. The remote peer IP address is the IP address of Aviatrix gateway we provisioned above in Prod VPC (Mumbai).</p><p>On to the final step now, we will configure a site2cloud connection from Aviatrix controller for the gateway we provisioned using mapped NAT.</p><h3>Configure Prod VPC to Onprem Tunnel (Site2Cloud connection in Aviatrix)</h3><p>Go to the controller UI once again, and navigate to site2cloud. Click on Add New under “Create New Site2Cloud connection”. Fill the options like below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hKKNtoF_zlF-xA_cgc2pnQ.png" /><figcaption>Note that remote gateway type is “generic” and Primary gateway us the one you created in the step above.</figcaption></figure><p>The remote gateway is the IP address of VPN gateway from on-prem VPC. Also, pay attention to the pre-shared key, and the virtual subnets. We are configuring the gateway to map the local real CIDR with a virtual and similarly remote on-prem subnet with a virtual CIDR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oJFO7feOPsXOsg6l_noEGQ.png" /></figure><p>Once created, you can go to diagnostics and see if the spoke connection is up</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kH20XveKDYxG5ltOeuz56A.png" /></figure><p>You will also see that the VPN Tunnel we created is also up now</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HO_1WoAzLY9h7PAvXxhLCw.png" /></figure><h3>Testing Connectivity</h3><p>We are finally ready to run some tests and see if VM-01 and VM-02 can communicate with each other. Before we do that, let’s create firewall rules to enable access. So the Prod-VPC-Mumbai should accept connections from the virtual CIDR range of Onpre-VPC-Delhi and vice versa. Let’s create those rules</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e7a72ca8ffa954c41035b06b3377308/href">https://medium.com/media/8e7a72ca8ffa954c41035b06b3377308/href</a></iframe><p>You can now connect to these respective VMs and do a ping test</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IB-qviiLJ-N_oKFH2KBeyg.png" /></figure><p>Another cool thing in GCP is you can run “connectivity tests” under network intelligence and see how the packets flow</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*q296gL4oqHuek10GjIJKNg.png" /></figure><h3>Cleanup</h3><p>If you have followed this article so far, it&#39;s likely you have created multiple VMs that will result in costs. Please clean up the respective resources.</p><p>Hope you found this useful! Happy Hybrid networking! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=462b329349c" width="1" height="1" alt=""><hr><p><a href="https://blog.thecloudside.com/vpn-with-overlapping-networks-on-gcp-using-aviatrix-462b329349c">VPN with Overlapping Networks on GCP using Aviatrix</a> was originally published in <a href="https://blog.thecloudside.com/">The Cloudside View</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Kafka Cluster with Zookeeper on GCP — The hard way]]></title>
            <link>https://blog.thecloudside.com/running-zookeeper-and-kafka-cluster-on-gcp-the-hard-way-750c40a55a97?source=rss-b3dbb13ab1fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/750c40a55a97</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[RK Kuppala]]></dc:creator>
            <pubDate>Sat, 01 Jan 2022 23:09:10 GMT</pubDate>
            <atom:updated>2022-07-14T08:46:20.773Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deploy Kafka Cluster with Zookeeper on GCP — The hard way</h3><p>If you accidentally landed on this page looking for the original (Franz) Kafka, however unlikely it may seem, I have nothing to offer to you, except this wonderful quote. 😄</p><blockquote><em>By believing passionately in something that still does not exist, we create it. The nonexistent is whatever we have not sufficiently desired</em></blockquote><blockquote><em>— Franz Kafka</em></blockquote><p>But if you came here for Apache Kafka, I assume you already know what Kafka is. For the uninitiated, here is a fun illustrated introduction to Kafka, aptly named <a href="https://www.gentlydownthe.stream/">Gently Down The Stream</a>. But don’t stop there, read a more serious introduction <a href="https://www.confluent.io/blog/apache-kafka-intro-how-kafka-works/">here</a>.</p><p>With that out of the way, let&#39;s get straight to the point of this post. We recently had the opportunity of deploying self-managed Kafka clusters at scale for our multiple customers as part of migrating them from On-premise / AWS environments to Google Cloud Platform. <em>This post is a how-to guide on deploying a highly available Kafka (version 3.0.0) cluster with Zookeeper ensemble on Compute Engine</em>.</p><h3>Kafka and It’s Keeper</h3><p>We are going to deploy this Kafka cluster along with a Zookeeper ensemble. I am sure some of you are wondering <a href="https://www.confluent.io/blog/removing-zookeeper-dependency-in-kafka/">why Kafka still needs a keeper</a>. Zookeeper is a well-known Ope source server, which is used to manage Kafka cluster’s metadata, among other things. To avoid external metadata management and simplify Kafka clusters, Kafka in the future is <a href="https://www.confluent.io/blog/kafka-without-zookeeper-a-sneak-peek/">not going to need</a> an external Zookeeper for metadata management, instead — Kafka will internally manage metadata quorum using <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum">KRaft</a>(Kafka Raft Metadata Mode). Unfortunately, KRaft is <a href="https://github.com/apache/kafka/blob/trunk/config/kraft/README.md">not production ready yet</a>, so for this post — we will go with Zookeeper.</p><h3>What will we build</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xZ5h4A6CLMpLkXPOWx5Zaw.png" /><figcaption>Made with Excalidraw.</figcaption></figure><p>We will deploy a highly available Kafka cluster in Mumbai region on GCP. We will configure a 3 node Zookeeper ensemble (or cluster) spread across 3 availability zones and configure 3 Kafka brokers (you can configure more, depending on your throughput requirements)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*G2d1Q6aPzo0HBicbpwcDyg.png" /></figure><p>Let’s get started!</p><h3>VPC, Subnets, and firewall rules</h3><p>If you already have a VPC and subnets, feel free to skip this section. If you don’t have one already, follow along.</p><p>Configure your project and region in gcloud</p><pre>gcloud auth login <br>gcloud config set project YOUR-PROJECT-NAME<br>gcloud config set compute/region asia-south1</pre><p>If you already have a VPC, you can skip this part. For the sake of this guide, let’s create a simple VPC and one subnet. In production, you would ideally provision these in a private subnet dedicated to the database layer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31fb648cb4889cdc29acecba3201f513/href">https://medium.com/media/31fb648cb4889cdc29acecba3201f513/href</a></iframe><p>We will use the network tag kafka to enable access between Kafka and zookeeper nodes. Let&#39;s create a firewall rule. This is rather permissive, you might want to open only necessary ports in production.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a65341de838b53fa7e56025594fa3dfb/href">https://medium.com/media/a65341de838b53fa7e56025594fa3dfb/href</a></iframe><h3>Configure Zookeeper</h3><p>Let’s provision a VM and configure Zookeeper on it. While launching these VMs, we will assign the network tag kafka so that the firewall rule we created above comes into effect and allows traffic between these VMs</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a65d848ccbfc3a37850eac9b5d8c19ea/href">https://medium.com/media/a65d848ccbfc3a37850eac9b5d8c19ea/href</a></iframe><p>Verify if the VM is up and running</p><pre>gcloud compute instances list --filter=&quot;tags.items=kafka&quot;</pre><p>SSH to this VM. Install and configure Zookeeper.</p><pre>gcloud beta compute ssh zk-01  \<br>--tunnel-through-iap \<br>--zone=asia-south1-a</pre><p>Install JDK. After installing JDK, it’s important to set the Java heap size in production environments. This will help avoid swapping, which will degrade Zookeeper’s performance. Refer to the Zookeeper administration guide <a href="https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_zkMulitServerSetup">here</a></p><pre>sudo apt update <br>sudo apt install default-jdk -y</pre><p>Add a user to run Zookeeper services and grant relevant permissions, we will go with zkadmin</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b644336e2ad738c9dfc23a51458733f/href">https://medium.com/media/5b644336e2ad738c9dfc23a51458733f/href</a></iframe><p>Time to configure Zookeeper. Let’s make a config file.</p><pre>vi /opt/zookeeper/conf/zoo.cfg</pre><p>Add the following configuration.</p><pre>tickTime=2000<br>dataDir=/data/zookeeper<br>clientPort=2181<br>initLimit=10<br>syncLimit=5</pre><p>We can quickly verify if everything went well so far by starting zookeeper</p><pre>cd /opt/zookeeper &amp;&amp; /opt/zookeeper/bin/zkServer.sh start</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jRRIcpBhfd0fCQZPPeHdlw.png" /></figure><p>You can connect and verify</p><pre>bin/zkCli.sh -server 127.0.0.1:2181</pre><p>You should see something like this</p><pre>WATCHER::</pre><pre>WatchedEvent state:SyncConnected type:None path:null</pre><pre><br>[zk: 127.0.0.1:2181(CONNECTED) 0]</pre><p>let’s stop Zookeeper and proceed with the rest of the setup</p><pre>sudo bin/zkServer.sh stop</pre><p>Let’s create a systemd unit file so that we can manage Zookeeper as a service — sudo vi /etc/systemd/system/zookeeper.service</p><p>Add the following</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b1f9abb89259ec689aef4ed544985ac/href">https://medium.com/media/9b1f9abb89259ec689aef4ed544985ac/href</a></iframe><p>Now that we have tested that Zookeeper on one instance, we can create an image out of it and use it to spin up 2 more nodes. Let’s make an image.</p><pre>gcloud beta compute machine-images \<br>  create zookeeper-machine-image \<br>  --source-instance zk-01 \<br>  --source-instance-zone asia-south1-a</pre><p>We can now use this image to create two more VMs — zk-02 and zk-03 in ap-south1-b and ap-south1-c zones respectively. Let&#39;s create these VMs</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aeb1fe978b88194623d9c98c479ca7d3/href">https://medium.com/media/aeb1fe978b88194623d9c98c479ca7d3/href</a></iframe><p>Once all the zookeeper nodes are up and running, We will create a myid file in each of the zookeeper nodes. The myid file consists of a single line containing only the text of that machine&#39;s id. So myid of server 1 would contain the text &quot;1&quot; and nothing else. The id must be unique within the Zookeeper ensemble and should have a value between 1 and 255.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7228e81bea9506cc8415d62df7bd3ba/href">https://medium.com/media/b7228e81bea9506cc8415d62df7bd3ba/href</a></iframe><p>Let’s now edit the zookeeper configuration file on all nodes to form the cluster quorum. We will add the hostname and ports of each node in the configuration file. You can find the FQDN of your zookeeper nodes by running the command hostname -A, usually it will be in the following format</p><pre>[SERVER-NAME].[ZONE].[PROJECT-NAME].internal</pre><p>Example: zk-01.asia-south1-a.c.cloudside-academy.internal. If your zookeeper cluster is a single zone deployment, you can simply resolve using the VM name - Ex: ping zk-03 . Since our deployment is multi-zone and you can not resolve names without using an FQDN across Zones, let&#39;s use an FQDN. If you want to keep things simpler, you may add the static internal IP of each node instead of the domain name.</p><p>SSH to each zookeeper node. On each node, change to zkadmin user - su -l zkadmin and open /opt/zookeeper/conf/zoo.cfg. The config file on all nodes should look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1387bc4fe41be3dd2a871486a8000ef/href">https://medium.com/media/d1387bc4fe41be3dd2a871486a8000ef/href</a></iframe><p>we have added 4lw.commands.whitelist=* config option to whitelist commands like stat, ruok, conf, isro.</p><h3>Start Zookeeper Cluster</h3><p>We are now ready to start the cluster. Let’s enable zookeeper.service on all nodes and also start it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e9baab54f5c1c7ab2fc0198c1a95f6df/href">https://medium.com/media/e9baab54f5c1c7ab2fc0198c1a95f6df/href</a></iframe><p>You may now connect to one of the zookeeper nodes and check the cluster status.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3dbc5d7d8548bb6b864cf0f3fd30f7ac/href">https://medium.com/media/3dbc5d7d8548bb6b864cf0f3fd30f7ac/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w5ZXRM08EbHbipbGdZXL4g.png" /></figure><p>You will see the status as either leader or follower for each node</p><h3>Setting Up Kafka</h3><p>Zookeeper ensemble is up and running, so let’s move on to configuring Kafka brokers. We will configure broker on one VM, and from it’s image, spin up two more Kafka brokers.</p><p>You might want to choose the right instance type in production and also attach an SSD disk for storage. For this guide, we will go ahead with e2-small VM. Let’s reserve three internal static IPs for our Kafka brokers</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/480c226bc0a28e6093ad59896292163e/href">https://medium.com/media/480c226bc0a28e6093ad59896292163e/href</a></iframe><p>SSH to the VM and start setting up Kafka.</p><pre>gcloud beta compute ssh kafka-broker-01 \<br>    --tunnel-through-iap \<br>    --zone=asia-south1-a</pre><p>Create a user for running Kafka services and download binaries</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fdd778dc620659e1485bfb1deb1abc3e/href">https://medium.com/media/fdd778dc620659e1485bfb1deb1abc3e/href</a></iframe><p>You can use the following configuration file, to begin with.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/83c8f7bb49c9cef265159f8c89b7945e/href">https://medium.com/media/83c8f7bb49c9cef265159f8c89b7945e/href</a></iframe><p>For this first kafka broker, we will assign broker.id as 0 and you can add more brokers and increment the id. use the internal IP of the broker VM to configure advertised.listeners, and listeners. Finally, configure zookeeper.connect to include all zookeeper nodes (use either FQDNs or IP addresses)</p><p>Move the existing existing sample config file and create a new config with the contents from above gist</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/06af4e75b8e37e6ee115195a182dd708/href">https://medium.com/media/06af4e75b8e37e6ee115195a182dd708/href</a></iframe><h3>Create a Kafka Service</h3><p>Let’s create a systemd file — sudo vi /etc/systemd/system/kafka.service</p><p>Add the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e533f4760668997b80f55fbc9718b08c/href">https://medium.com/media/e533f4760668997b80f55fbc9718b08c/href</a></iframe><p>Start the service — systemctl start kafka</p><p>You can now connect to one of the Zookeeper nodes and verify that the kafka broker with id 0 shows up.</p><pre><em>#SSH to any zookeeper node and cd /opt/zookeeper </em></pre><pre>bin/zkCli.sh -server 127.0.0.1:2181</pre><p>After connecting, run</p><pre>ls /brokers/ids</pre><p>You should see this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*hlvE5TCdEezyh0Xoh8EIwg.png" /></figure><h3>Create an Image</h3><p>We are now ready to make an image out of this VM and use it to spin up more Kafka broker nodes. Let’s make an image</p><pre>gcloud beta compute machine-images \<br>  create kafka-machine-image \<br>  --source-instance kafka-broker-01 \<br>  --source-instance-zone asia-south1-a</pre><p>Let’s now create kafka-broker-02, kafka-broker-03 VMs in asia-south1-b, asia-south1-c zones respectively.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/537df352499dbc9b565c9d2073d2559c/href">https://medium.com/media/537df352499dbc9b565c9d2073d2559c/href</a></iframe><p>Once these VMs are online, all you need to do is to SSH to these VMs and change the following properties in the configuration file — vi /data/kafka/config/server.properties</p><p>For kafka-broker-02 VM</p><pre>broker.id=1<br>listeners=PLAINTEXT://10.10.0.21:9092<br>advertised.listeners=PLAINTEXT://10.10.0.21:9092</pre><p>For kafka-broker-03 VM</p><pre>broker.id=2<br>listeners=PLAINTEXT://10.10.0.22:9092<br>advertised.listeners=PLAINTEXT://10.10.0.22:9092</pre><p>Since we are launching these VMs from an image, and there might be some data left which belonged to Broker 0, we have to cleanup the data (log) directories before we start the service.</p><pre>sudo rm -rf /data/kafka/logs/ &amp;&amp; systemctl start kafka</pre><p>You can now finally verify the status of all brokers in one of the zookeeper nodes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uqE7M4XtR2MPKpZolI3ZCA.png" /></figure><p>Congratulations! you have now set up Kafka the hard way on Google Compute Engine. Let’s clean up now.</p><h3>Cleanup</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d92192fdbc90fd58163f65770453b54/href">https://medium.com/media/0d92192fdbc90fd58163f65770453b54/href</a></iframe><p>Hope you found this guide useful. If you need help running self-managed kafka at scale on Google Compute Engine, <a href="https://thecloudside.com/">reach out to us</a>! Happy stream processing :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=750c40a55a97" width="1" height="1" alt=""><hr><p><a href="https://blog.thecloudside.com/running-zookeeper-and-kafka-cluster-on-gcp-the-hard-way-750c40a55a97">Deploying Kafka Cluster with Zookeeper on GCP — The hard way</a> was originally published in <a href="https://blog.thecloudside.com/">The Cloudside View</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We shall not cease from exploration…!]]></title>
            <link>https://blog.thecloudside.com/we-shall-not-cease-from-exploration-3a9b07363dab?source=rss-b3dbb13ab1fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a9b07363dab</guid>
            <category><![CDATA[launch]]></category>
            <dc:creator><![CDATA[RK Kuppala]]></dc:creator>
            <pubDate>Wed, 07 Jul 2021 11:33:37 GMT</pubDate>
            <atom:updated>2021-07-07T11:33:37.004Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*odyV9kK8v9yyqz_j1rOnGg.png" /><figcaption>We make cloud work for you!</figcaption></figure><p>For those of you that don’t personally know us, which we believe is the entire world barring a teeny tiny fraction of it that we had the pleasure of interacting with, here is a quick intro. We are <a href="https://www.linkedin.com/in/ramdevaraj/">Ram</a> and <a href="https://www.linkedin.com/in/yogirk/">RK</a>, engineers by trade and entrepreneurs at heart. We have been in the public cloud and data consulting space for close to a decade now — most part of this together.</p><p>Over the years, we have had the good fortune of working with some amazing people. We have witnessed the growth of cloud consulting companies at the very early stage of the public cloud. We subsequently co-founded a cloud consulting company, contributing to its growth from humble beginnings to 120+ people, and recognition of becoming a premier AWS partner, a notable Azure and GCP partner. Later at <a href="https://www.searce.com/">Searce</a>, we led the Cloud and Data engineering teams, Product initiatives, App Engineering, AI &amp; ML &amp; more specifically, loved getting the AWS consulting practice off the ground, enjoyed taking the GCP practice to the next level, &amp; doubling the cloud, data, and app engineering teams along the way.</p><p>Throughout this wonderful journey, here is what we learned and experienced.</p><ul><li>There is something immensely satisfying and fulfilling in witnessing a fresh college graduate spread his/her wings and transform into a Cloud or a Data Architect. We often built teams from scratch and it required working with a large number of fresh graduates. Being able to contribute to their growth was a great personal fortune.</li><li>Working in the services industry is like working on a hundred different products. We were fortunate to help customers of all shapes and sizes. From a two-person startup that visited our 2BHK office for their cloud cost optimization meeting, to a large manufacturing conglomerate that wanted to modernize their data stack and implement a data lake. And everything in between. Internet-scale companies, ISVs, Born In Cloud — the whole shebang.</li><li>We are still excited about tomorrow’s customer call. After multiple years, working with different customers — we still get excited about the opportunity to consult, design, deploy, scale, optimize and secure for our customers. There is no better sight than a happy customer.</li><li>We love giving back to the community. When work is a by-product of what makes you inherently happy, then work never tires you. We love contributing in whatever small way possible — blog posts, open source tools, helping fresh graduates become solutions architects/data architects.</li><li>People are most important. You prioritize your people, customers, investors, and stakeholders. In that order. People well looked after, will never let your customers down.</li></ul><p>A few months ago, when we started toying with the idea of doing a startup once again — we naturally tilted towards a Cloud-native and Data company, given our experience so far and also because of things that give us joy at work. <strong>At Searce, we witnessed hockey stick growth with rapid expansion into multiple global regions as a global MSP for Google Cloud. </strong>We loved our experience of scaling the Cloud, Data &amp; App Engineering practice at Searce and we thought let’s do <strong>zero to one, once again</strong>.</p><p>We are super excited to announce <a href="https://thecloudside.com/">Cloudside Technologies</a>, our new startup, and new passion :). As a company, there is a lot for us to do. Particularly in the wake of the world’s applications moving towards a new home, <strong>Kubernetes. </strong>Businesses are in varying stages of their cloud adoption. It’s now a matter of <strong><em>when </em></strong>and not <strong><em>if</em></strong>. We believe that we can add value to our clients by bringing a new perspective to the oft-encountered problems in Cloud and Data consulting. We intend to look at these problems from a product engineering lens and build solutions.</p><p>Searce &amp; Hardik have always been passionate about supporting tech entrepreneurs in their growth journey and have continued to invest in cloud &amp; data focused tech startups.</p><blockquote><strong>We are super excited to announce that Searce is making a strategic investment in Cloudside Technologies and expanding its cloud consulting &amp; data engineering footprint</strong></blockquote><p>As a part of this partnership, Searce and Cloudside Technologies will continue to collaborate as strategic partners to serve our customers better and to accelerate scale in the cloud consulting space.</p><p>We started assembling a team for our new adventure, and we are lucky to be able to onboard like-minded people who are ready for the startup journey and be part of our core team — <a href="https://www.linkedin.com/in/richardeliab/">Richard</a>, <a href="https://www.linkedin.com/in/umeshdotcloud/">Umesh</a>, <a href="https://www.linkedin.com/in/karthik-k-057aa314a/">Karthik</a>, and are excited to be supported by the well-wishers from our extended family at Searce. We are open for business and are currently <a href="https://thecloudside.com/careers.html">hiring</a>. Follow <a href="https://blog.thecloudside.com/">our blog</a> to keep up with our adventures in Cloud, Data and App engineering!</p><p>That’s all for now folks! Keep being awesome!</p><p><strong>Ram &amp; RK</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a9b07363dab" width="1" height="1" alt=""><hr><p><a href="https://blog.thecloudside.com/we-shall-not-cease-from-exploration-3a9b07363dab">We shall not cease from exploration…!</a> was originally published in <a href="https://blog.thecloudside.com/">The Cloudside View</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hello World!]]></title>
            <link>https://blog.thecloudside.com/hello-world-25721c192c7b?source=rss-b3dbb13ab1fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/25721c192c7b</guid>
            <dc:creator><![CDATA[RK Kuppala]]></dc:creator>
            <pubDate>Mon, 26 Apr 2021 16:39:44 GMT</pubDate>
            <atom:updated>2021-04-26T16:39:44.342Z</atom:updated>
            <content:encoded><![CDATA[<p>We shall not cease from exploration<br>And the end of all our exploring<br>Will be to arrive where we started<br>And know the place for the first time.</p><p>— T.S. Eliot, <a href="http://www.columbia.edu/itc/history/winter/w3206/edit/tseliotlittlegidding.html">Four Quartets</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25721c192c7b" width="1" height="1" alt=""><hr><p><a href="https://blog.thecloudside.com/hello-world-25721c192c7b">Hello World!</a> was originally published in <a href="https://blog.thecloudside.com/">The Cloudside View</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hello World!]]></title>
            <link>https://yogirk.medium.com/hello-world-a48ea108d696?source=rss-b3dbb13ab1fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/a48ea108d696</guid>
            <dc:creator><![CDATA[RK Kuppala]]></dc:creator>
            <pubDate>Mon, 26 Apr 2021 15:53:07 GMT</pubDate>
            <atom:updated>2021-04-26T15:53:07.551Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>We shall not cease from exploration<br>And the end of all our exploring<br>Will be to arrive where we started<br>And know the place for the first time.</em></blockquote><p><strong>T.S. Eliot, </strong><a href="https://www.goodreads.com/work/quotes/2886568"><strong>Four Quartets</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a48ea108d696" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>