<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pablo Acuña on Medium]]></title>
        <description><![CDATA[Stories by Pablo Acuña on Medium]]></description>
        <link>https://medium.com/@pacuna?source=rss-33202ee06367------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xmLFhKuS-_jU8cGy2M8aZA.jpeg</url>
            <title>Stories by Pablo Acuña on Medium</title>
            <link>https://medium.com/@pacuna?source=rss-33202ee06367------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 15:07:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pacuna/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Spark, Kubernetes and Azure]]></title>
            <link>https://medium.com/@pacuna/apache-spark-kubernetes-and-azure-c307e093c137?source=rss-33202ee06367------2</link>
            <guid isPermaLink="false">https://medium.com/p/c307e093c137</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Pablo Acuña]]></dc:creator>
            <pubDate>Sun, 05 May 2019 03:53:16 GMT</pubDate>
            <atom:updated>2019-05-05T03:53:16.879Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Spark and Kubernetes are some of my favorite tools these days. And turns out you can run Spark jobs using Kubernetes as a resource manager. It sounds cool, but due to implementation differences between cloud providers and lack of documentation, it can get a bit tricky.</p><p>I’ve worked with Amazon AWS, Google GCE and Microsoft Azure in the past. And no matter what people say, they all work fine. Of course you will find differences, but they are all professional production-ready cloud environments.</p><p>I tested this integration on GCE and Azure. And I decided to share my experience with Azure, mainly because it wasn’t so straightforward and it required extra setup.</p><p>During the process I got into some issues. Mainly around networking when running Spark jobs in client mode. One important lesson is:</p><blockquote>When using Apache Spark in client mode with Kubernetes, the pods should be able to communicate over the network with the host that is submitting the job.</blockquote><p>Having a host submitting Spark jobs in the same virtual network the cluster is running in, and that allows inbound traffic from the pods should be enough. In this case with Azure, the solution was to create a custom Virtual network and a Subnet. Once you have these resources, you can launch the AKS cluster and the VM that will submit the jobs, in the same VNet and Subnet. Then, the Kubernetes pods that act as executors, can talk to the VM that runs the driver program and the jobs can be executed with no issues.</p><p>This post contains a bunch of commands. In case you want to dig deeper, links with more details will be provided. Let’s get started.</p><h4>Local environment</h4><p>All the commands where tested using an Ubuntu bionic container. If you have your own Ubuntu machine, you can use it instead. This container is only used to create the initial Azure resources, before launching the VM that will submit the Spark jobs. If you don’t have an Ubuntu 18 machine, you can use Docker and follow along.</p><p>Let’s start a bash session in an Ubuntu bionic container:</p><pre>docker run -it ubuntu:bionic bash</pre><p>We need to install the <a href="https://docs.microsoft.com/en-us/cli/azure/?view=azure-cli-latest">Azure CLI tool</a> to interact with our Azure account along with some dependencies:</p><pre>apt update &amp;&amp; apt install curl ssh -y<br>curl -sL <a href="https://aka.ms/InstallAzureCLIDeb">https://aka.ms/InstallAzureCLIDeb</a> | bash</pre><p>You can sign in with:</p><pre>az login --use-device-code</pre><p>Follow the instructions from the login command output. You will need to authenticate with your Azure credentials using your browser.</p><h4>Create a Resource Group</h4><p>A resource group will help us to encapsulate all the resources we need for this tutorial. Once we are done, we can delete everything by just deleting the resource group.</p><pre>az group create --name sparkRG --location eastus</pre><h4>Create a Virtual Network and Subnet</h4><p>Like I mentioned in the introduction, we need a custom Virtual Network and a Subnet. Now, when you create an AKS cluster, if you don’t specify a Vnet, a default one will be created. I had some issues with this default Vnet when I tried to create a VM in it after. And we need that VM to be located along the AKS cluster. That’s why we are creating a custom one here.</p><p>You can find more details about the following commands in <a href="https://docs.microsoft.com/en-us/azure/aks/configure-kubenet">this tutorial</a>.</p><p>First, we create a Vnet in our resource group. We need to provide the network configuration for the IP ranges and the Subnet name and CIDR prefix. Take note of the Vnet and Subnet names. We are going to use them in later commands.</p><pre>az network vnet create \<br>    --resource-group sparkRG \<br>    --name myAKSVnet \<br>    --address-prefixes 10.0.0.0/8 \<br>    --subnet-name myAKSSubnet \<br>    --subnet-prefix 10.240.0.0/16</pre><p>You don’t need to use those exact values in case you wonder. Those are just the default used in the tutorial I linked previously.</p><h4>Assign permissions</h4><p>The following command will allow the AKS cluster to interact with other Azure resources. This method of managing permissions is very common when you need to integrate Azure services.</p><p>First we create a <a href="https://docs.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals">Service Principal:</a></p><pre>az ad sp create-for-rbac --skip-assignment</pre><p>Output:</p><pre>{<br>  &quot;appId&quot;: &quot;81bba314-cc5a-4232-b619-285ddce5a199&quot;,<br>  &quot;displayName&quot;: &quot;azure-cli-2019-05-04-00-40-40&quot;,<br>  &quot;name&quot;: &quot;<a href="http://azure-cli-2019-05-04-00-40-40/">http://azure-cli-2019-05-04-00-40-40</a>&quot;,<br>  &quot;password&quot;: &quot;cffc5a68-20ba-4a1d-b2db-41e75a10ffc3&quot;,<br>  &quot;tenant&quot;: &quot;47a8af16-7f6e-414a-8612-5930927bd83d&quot;<br>}</pre><p>Take note of the <em>appId </em>and <em>password </em>properties. We also need those for later use.</p><p>To assign roles, we need the resource ids of the Vnet and Subnet created previosly. We can use the az tool to run queries for these values:</p><pre>VNET_ID=$(az network vnet show --resource-group sparkRG --name myAKSVnet --query id -o tsv)</pre><pre>SUBNET_ID=$(az network vnet subnet show --resource-group sparkRG --vnet-name myAKSVnet --name myAKSSubnet --query id -o tsv)</pre><p>Now let’s create the role assignment that we will use for our cluster and that assigns the right permissions to interact with the Vnet. <strong>Replace the assignee parameter with you <em>appId </em>from the your Service Principal</strong>:</p><pre>az role assignment create --assignee 81bba314-cc5a-4232-b619–285ddce5a199 --scope $VNET_ID --role Contributor</pre><h4>Creating the AKS cluster</h4><p>Most of the parameters to launch the cluster are network related and also copied from the link posted before. Remember the cluster needs to be created in the right Vnet and Subnet. <strong>Replace the service-principal with your <em>appId </em>and the client-secret with your <em>password </em>from the Service Principal:</strong></p><pre>az aks create \<br>    --resource-group sparkRG \<br>    --name sparkAKSCluster \<br>    --node-count 3 \<br>    --network-plugin kubenet \<br>    --service-cidr 10.0.0.0/16 \<br>    --dns-service-ip 10.0.0.10 \<br>    --pod-cidr 192.168.0.0/16 \<br>    --docker-bridge-address 172.17.0.1/16 \<br>    --vnet-subnet-id $SUBNET_ID \<br>    --service-principal 81bba314-cc5a-4232-b619-285ddce5a199 \<br>    --client-secret cffc5a68-20ba-4a1d-b2db-41e75a10ffc3 \<br>    --generate-ssh-keys</pre><h4>Creating a container registry</h4><p>In order to submit Spark jobs to Kubernetes, we need to host Spark docker images somewhere. Let’s use the container registry service from Azure to create our own private registry:</p><pre>az acr create --resource-group sparkRG --name sparkimages --sku Basic</pre><p>Our AKS cluster should be able to pull images from this registry. We can also create role assignment for this (more details <a href="https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-aks">here</a>):</p><pre>CLIENT_ID=$(az aks show --resource-group sparkRG --name sparkAKSCluster --query &quot;servicePrincipalProfile.clientId&quot; --output tsv)</pre><pre>ACR_ID=$(az acr show --name sparkimages --resource-group sparkRG --query &quot;id&quot; --output tsv)</pre><pre>az role assignment create --assignee $CLIENT_ID --role acrpull --scope $ACR_ID</pre><h4>Creating the VM</h4><p>Now we can create the VM that will submit the Spark jobs using the same Vnet and Subnet the cluster was created in:</p><pre>az vm create \<br>--name sparkVM \<br>--resource-group sparkRG \<br>--image ubuntults \<br>--vnet-name myAKSVnet \<br>--subnet myAKSSubnet \<br>--generate-ssh-keys \<br>--admin-username pacuna \<br>--admin-password Xah8aruV5ciel5Weeroh</pre><p>The generated SSH keys will be stored in the ~/.ssh directory. Copy them in case you want to use them later. Remember once you exit the container, they will be gone.</p><h4>Setting up the VM</h4><p>Log into the VM (from here on you can keep using the container, or you can use the generated SSH keys and use your regular local environment):</p><pre>ssh pacuna@13.82.212.139</pre><p>Install Java and Spark:</p><pre>sudo apt update<br>sudo apt upgrade<br>sudo apt install openjdk-8-jre -y<br>wget <a href="http://mirrors.sonic.net/apache/spark/spark-2.4.2/spark-2.4.2-bin-hadoop2.7.tgz">http://mirrors.sonic.net/apache/spark/spark-2.4.2/spark-2.4.2-bin-hadoop2.7.tgz</a><br>tar -xzvf spark-2.4.2-bin-hadoop2.7.tgz<br>sudo mv spark-2.4.2-bin-hadoop2.7 /opt/spark</pre><p>Set the SPARK_HOME environmental variable, and add the Spark binaries to the path:</p><pre>export SPARK_HOME=/opt/spark<br>export PATH=$SPARK_HOME/bin:$PATH</pre><p>Let’s also install the Azure CLI tool to connect to the Kubernetes cluster from this VM:</p><pre>curl -sL <a href="https://aka.ms/InstallAzureCLIDeb">https://aka.ms/InstallAzureCLIDeb</a> | sudo bash</pre><p>Log in once again by following the instructions:</p><pre>az login --use-device-code</pre><p>Docker will be necessary to build and push the Spark images to the container registry:</p><pre>curl -fsSL https://get.docker.com -o get-docker.sh<br>sudo sh get-docker.sh<br>sudo usermod -aG docker $USER</pre><p>Log out and log in so docker can be used without sudo.</p><p>Submitting Spark applications to a Kubernetes cluster also requires to have kubectl installed and configured to communicate with the cluster:</p><pre>curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s <a href="https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl">https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl</a><br>chmod +x ./kubectl<br>sudo mv ./kubectl /usr/local/bin/kubectl</pre><p>We can configure kubectl by pulling the AKS credentials:</p><pre>az aks get-credentials --resource-group sparkRG --name sparkAKSCluster</pre><h4>Build and push Spark docker images</h4><p>Log in to the container registry so we can push our local Docker images:</p><pre>az acr login --name sparkimages</pre><p>Now we can build and push the Spark images using the <em>docker-image-tool.</em>sh that comes with Spark. <strong>The following commands need to be run from the SPARK_HOME directory</strong>. After running the commands, the remote container registry will have 3 images: <strong>spark:0.1</strong>, <strong>spark-py:0.1</strong> and <strong>spark-r:0.1:</strong></p><pre>cd $SPARK_HOME<br>docker-image-tool.sh -r sparkimages.azurecr.io -t 0.1 build<br>docker-image-tool.sh -r sparkimages.azurecr.io -t 0.1 push</pre><h4>Add <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#rbac">RBAC </a>permissions for spark</h4><pre>kubectl create serviceaccount spark<br>kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default</pre><h4>Running a Spark shell</h4><p>There are two important parameters when using client mode: <strong>spark.driver.host</strong> and <strong>spark.driver.port</strong>. These parameters tell the pods where the driver is running on. The host will be our VM’s private IP address:</p><pre>hostname --ip-address</pre><p>Which in my case is <strong>10.240.0.7. </strong>And for the port we will use <strong>7778</strong>.</p><p>We also need our Kubernetes cluster address. You can get it with:</p><pre>kubectl cluster-info</pre><p>Which in my case is <a href="https://sparkakscl-sparkrg-e659ac-e06a364e.hcp.eastus.azmk8s.io/"><strong>https://sparkakscl-sparkrg-e659ac-e06a364e.hcp.eastus.azmk8s.io:443</strong></a><strong>.</strong></p><p>Now we can start a spark-shell with the following command:</p><pre>spark-shell \<br>--master k8s://<a href="https://sparkakscl-sparkrg-e659ac-e06a364e.hcp.eastus.azmk8s.io/">https://sparkakscl-sparkrg-e659ac-e06a364e.hcp.eastus.azmk8s.io:443</a> \<br>--deploy-mode client \<br>--conf spark.driver.host=10.240.0.7 \<br>--conf spark.driver.port=7778 \<br>--conf spark.kubernetes.container.image=sparkimages.azurecr.io/spark:0.1 \<br>--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark</pre><p>We are using our Kubernetes cluster endpoint, the driver parameters, the Spark image from our private container registry, and the service account created previously.</p><p>Run a simple test:</p><pre>spark.range(10).toDF(&quot;number&quot;).show()</pre><p>You should see something like the following:</p><pre>19/05/04 02:06:48 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<br>Using Spark&#39;s default log4j profile: org/apache/spark/log4j-defaults.properties<br>Setting default log level to &quot;WARN&quot;.<br>To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).<br>Spark context Web UI available at <a href="http://10.240.0.7:4040/">http://10.240.0.7:4040</a><br>Spark context available as &#39;sc&#39; (master = k8s://<a href="https://sparkakscl-sparkrg-e659ac-e06a364e.hcp.eastus.azmk8s.io/">https://sparkakscl-sparkrg-e659ac-e06a364e.hcp.eastus.azmk8s.io:443</a>, app id = spark-application-1556935623374).<br>Spark session available as &#39;spark&#39;.<br>Welcome to<br>      ____              __<br>     / __/__  ___ _____/ /__<br>    _\ \/ _ \/ _ `/ __/  &#39;_/<br>   /___/ .__/\_,_/_/ /_/\_\   version 2.4.2<br>      /_/</pre><pre>Using Scala version 2.12.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)<br>Type in expressions to have them evaluated.<br>Type :help for more information.</pre><pre>scala&gt; spark.range(10).toDF(&quot;number&quot;).show()<br>+------+<br>|number|<br>+------+<br>|     0|<br>|     1|<br>|     2|<br>|     3|<br>|     4|<br>|     5|<br>|     6|<br>|     7|<br>|     8|<br>|     9|<br>+------+</pre><p>If you run the <em>kubectl get pods </em>command, you should see the Spark executors running as Kubernetes pods while you have this shell opened.</p><h4>Running a Python application</h4><p>TODO</p><h4>Running a Scala application</h4><p>TODO</p><h4>Running a Jupyter Notebook</h4><p>TODO</p><h4>Delete everything</h4><pre>az group delete --name sparkRG --no-wait --yes</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c307e093c137" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running a Zookeeper Ensemble on AWS]]></title>
            <link>https://medium.com/@pacuna/running-a-zookeeper-ensemble-on-aws-8025a66e0c1?source=rss-33202ee06367------2</link>
            <guid isPermaLink="false">https://medium.com/p/8025a66e0c1</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[apache-zookeeper]]></category>
            <category><![CDATA[zookeeper]]></category>
            <dc:creator><![CDATA[Pablo Acuña]]></dc:creator>
            <pubDate>Mon, 15 Oct 2018 14:56:41 GMT</pubDate>
            <atom:updated>2018-10-15T14:56:41.551Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post I’ll describe the steps to deploy a Zookeeper Ensemble on AWS. I won’t be using any tool to automate the process. Just plain old manual configuration. I’m writing this because during the process of setting up the cluster I encountered some issues that weren’t covered in other posts I saw. So if you are trying to run your own Zookeeper cluster on AWS and you want to learn how to configure it manually, this post might be useful for you.</p><h4>Prerequisites</h4><p>I’m assuming you’re comfortable working with AWS and Linux, and that you already have a AWS account to work with. I won’t be covering basic AWS stuff . Also, this is about the deployment, not about Zookeeper. So if you want to learn more about the tool itself, you can visit the <a href="https://zookeeper.apache.org/">official site</a>.</p><h4>Launching the instances</h4><p>Once thing I don’t enjoy too much is to configure Java on a new machine. There’s an Amazon Linux instance that comes with Java already installed and configured. In the instances list, make sure you find this image (there’s another one that doesn’t come with Java preinstalled):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*41X5mfmBo4blhlzKYJBxjQ.png" /><figcaption>Amazon Linux Image (with Java)</figcaption></figure><p>Select that image and choose the family you want to use. I’ll be using the m4.large instance in case you want to follow along:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tnzgS1rLdwY6n1yrsdr5kg.png" /></figure><p>Let’s launch a 3-node cluster. The recommended setting for a non-standalone Zookeeper deployment is to always use an odd number of nodes. This is because the consensus algorithm that zookeeper uses is designed to work with an odd number.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*bJ57MuVCCgVOzpsAauCwbw.png" /></figure><p>I’ll leave the default values for the rest of the configuration. You can also configure the storage but since this is just a proof of concept, I’ll stick with the 8GB EBS that comes by default, then click on <em>review and launch</em> and then <em>launch</em>.</p><h4>Installing Zookeeper</h4><p>We have to install zookeeper in each one of the nodes. The configuration will be almost the same for the 3 machines with just a minor difference.</p><p>Once the instances are up and running let’s add a name to each one so we don’t get confused. We need to be able to identify each node and their addresses. I’ll use zk-1, zk-2 and zk-3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oGJr9HAyTeA7l-MwDyyCsw.png" /></figure><p>Now let’s ssh into the first machine and start with the process:</p><pre>ssh -i zk-cluster.pem <a href="/service/mailto:ec2-user@ec2-18-222-212-204.us-east-2.compute.amazonaws.com">ec2-user@ec2–18–222–212–204.us-east-2.compute.amazonaws.com</a></pre><p>Look for the stable version of Zookeeper in the <a href="https://zookeeper.apache.org/releases.html#download">download section</a>, copy the link and download it:</p><pre>wget <a href="https://www-us.apache.org/dist/zookeeper/stable/zookeeper-3.4.12.tar.gz">https://www-us.apache.org/dist/zookeeper/stable/zookeeper-3.4.12.tar.gz</a></pre><p>Now, extract Zookeeper and install it:</p><pre>tar -xzf zookeeper-3.4.12.tar.gz<br>sudo mv zookeeper-3.4.12 /usr/local/zookeeper</pre><p>We also need to create a data directory:</p><pre>sudo mkdir /var/lib/zookeeper</pre><p>Zookeeper comes with a sample configuration file that we can use as a base:</p><pre>sudo cp /usr/local/zookeeper/conf/zoo_sample.cfg /usr/local/zookeeper/conf/zoo.cfg</pre><p>This zoo.cfg file will be used by default when launching the Jar file. Open the file and change the dataDir parameter to the data directory we just created:</p><pre>dataDir=/var/lib/zookeeper</pre><p>Also, we need to add a section that contains the other nodes information. The only information we need is their private IP addresses which you can copy from the AWS console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*oRgFbaOJh9yPqDu2btov4A.png" /></figure><p>Using the same id order we used for the name tags, add the following to the configuration file:</p><pre>server.1=172.31.47.44:2888:3888<br>server.2=172.31.39.15:2888:3888<br>server.3=172.31.46.188:2888:3888</pre><p>We are defining 3 nodes and their ids (1, 2 and 3). Then we use their private address, then the port used for communication, and finally a port that’s used for leader election.</p><p>The final configuration file should look like this:</p><pre>tickTime=2000<br>initLimit=10<br>syncLimit=5<br>dataDir=/var/lib/zookeeper<br>clientPort=2181<br>server.1=172.31.47.44:2888:3888<br>server.2=172.31.39.15:2888:3888<br>server.3=172.31.46.188:2888:3888</pre><h4><strong>Important: all the steps we’ve done so far should be replicated for each node.</strong></h4><p>Once you finished with all the nodes, the last step is to create an identifier file for each of them. This file should be located inside the data directory /var/lib/zookeeper and should be named myid. This file will contain only the id for the correspondent node.</p><p>So, for example, for the zk-1 node, create the file and add 1 to it:</p><pre>echo 1 | sudo tee -a /var/lib/zookeeper/myid</pre><p>For the second server zk-2 use 2:</p><pre>echo 2 | sudo tee -a /var/lib/zookeeper/myid</pre><p>And finally for the third server zk-3 use 3:</p><pre>echo 3| sudo tee -a /var/lib/zookeeper/myid</pre><p>And that’s it for the configuration</p><h4>Launching the cluster</h4><p>Before launching the Zookeeper servers, we need to modify the security group the machines were launch with, so they can communicate to each other through the ports we defined in the configuration. The 3 machines were launched using the same security group so let’s just add a new inbound rule for all TCP ports but restricted to the subnet we’re using:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RTiCSMPgJLaDPncQ0yx-gw.png" /></figure><p>In case you don’t want access for every port, you need at least access to ports 2181, 2888 and 3888 for all the nodes.</p><p>Now we can start the Zookeeper server for each node. It’s convenient to have an opened ssh session simultaneously in case you want to check for any problems.</p><p>Inside each node, run the following command:</p><pre>sudo /usr/local/zookeeper/bin/zkServer.sh start</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*h-Re9GePkV58WZZ94ryhqw.png" /></figure><p>After starting the servers, you can run</p><pre>sudo /usr/local/zookeeper/bin/zkServer.sh status</pre><p>To check the status of each one of the nodes. You’ll see which one was elected as the leader and which ones are followers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*NMDo_ICTEjk1c66xQkEd0w.png" /></figure><h4>Troubleshooting</h4><p>Useful things I found during the process:</p><ul><li>Use start-foreground instead of start to test if the server starts correctly (or use tail -f zookeeper.out to checkout the logs if you using the background start)</li><li>Make sure your security groups are well configured and that the CIDR you choose covers all the nodes IP addresses.</li></ul><p>And that’s it! Now you can start experimenting with your Zookeeper cluster.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8025a66e0c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Rails 5.1 System Tests with Docker]]></title>
            <link>https://medium.com/@pacuna/using-rails-5-1-system-tests-with-docker-a90c52ed0648?source=rss-33202ee06367------2</link>
            <guid isPermaLink="false">https://medium.com/p/a90c52ed0648</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[rails]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Pablo Acuña]]></dc:creator>
            <pubDate>Sun, 18 Jun 2017 02:09:39 GMT</pubDate>
            <atom:updated>2017-06-18T02:09:39.839Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been using Docker for my Rails development environment for a while now. This means I’ve had to address a lot of different issues and learn some of the good and bad practices around containerized systems.</p><p>I’m also a heavy advocate of TDD and testing in general. I think the time you <em>lose</em> writing tests is actually less than the time you spend testing your application by hand to see if your new method or feature works as expected. And of course, in the future when you need to modify your code, good luck with testing you didn’t break anything if you don’t have a decent test suite.</p><p>With that said, I’ve never been a big fan of acceptance testing. Maybe it’s because every time I’ve tried has been a painful experience. Most of this was probably due to my inexperience with them. On the other side, when I started with Unit and Functional testing, everything worked just fine. And the integration issues were always minor, whereas with acceptance testing, it was always hard.</p><p>All of that changed when I started to use new Rails 5.1 System Tests feature. I know that under the hood, thanks to the great core team, the integration with the gems is rock solid. I think convention over configuration specifically with these kind of topics is really important.</p><p>Before this version you had to add Capybara, Database cleaner and maybe other gems to start writing your acceptance tests. And then you had to modify your helpers, select a database cleaning strategy, etc. It just didn’t feel <em>clean</em> to me. That’s why I tried to avoid that kind of testing whenever I could.</p><p>At first, with this new Rails feature, I was a little bit confused about how to use it with Docker. Because you need to setup a driver such a Chrome or Firefox. And you don’t want to add all those dependencies to your Dockerfile. So I thought, hey! Probably someone already built a Docker image with some of those drives, and I found <a href="https://github.com/SeleniumHQ/docker-selenium">this</a>.</p><p>So using that image and doing some minor adjustments, you can have an independent container running your driver and not pollute your main application Dockerfile with all those dependencies.</p><p>I ended up having a docker-compose.yml file similar to this:</p><pre>version: &#39;2&#39;<br>services:<br>  chrome:<br>    image: selenium/standalone-chrome-debug<br>    volumes:<br>      - /dev/shm:/dev/shm<br>  myapp:<br>    build:<br>      context: .<br>      dockerfile: Dockerfile.development<br>    depends_on:<br>      - chrome</pre><p>And then you can configure your <em>test/application_system_test_case.rb</em> to use this chrome service instead of looking for a local instance:</p><pre>require &quot;test_helper&quot;<br>require &quot;socket&quot;</pre><pre>class ApplicationSystemTestCase &lt; ActionDispatch::SystemTestCase<br>  driven_by :selenium, using: :chrome, screen_size: [1400, 1400], options: {url: &quot;<a href="http://chrome:4444/wd/hub">http://chrome:4444/wd/hub</a>&quot;}</pre><pre>include Devise::Test::IntegrationHelpers</pre><pre>  def setup<br>    host! &quot;<a href="http://#{IPSocket.getaddress(Socket.gethostname)}">http://#{IPSocket.getaddress(Socket.gethostname)}</a>&quot;<br>    super<br>  end<br>end</pre><p>As you can see, we are telling Rails to use our other service container address for chrome, which is <em>chrome.</em></p><p>The setup part is necessary. If you don’t change your host address, you’ll basically run your tests using localhost as your base path. So for example, if you try to <em>visit</em> something with capybara, let’s say <em>articles_path</em>, capybara will try to go to http://localhost/articles instead your container’s IP address.</p><p>I’m sure there are some other ways to set the container’s IP address but this one worked for me immediately. And I have to thank to the creator of <a href="https://github.com/rails/rails/pull/28341">this</a> pull request for the implementation of these extra options and the basic idea.</p><p>So that’s it! Now you can run your system tests in a Dockerize environment:</p><pre>root@9bb7de76347b:/home/app/webapp# RAILS_ENV=test bin/rails test:system<br>Run options: --seed 10639</pre><pre># Running:</pre><pre>Puma starting in single mode...<br>* Version 3.8.2 (ruby 2.4.0-p0), codename: Sassy Salamander<br>* Min threads: 0, max threads: 1<br>* Environment: test<br>* Listening on tcp://0.0.0.0:45365<br>Use Ctrl-C to stop<br>................</pre><pre>Finished in 12.000958s, 1.3332 runs/s, 2.2498 assertions/s.<br>16 runs, 27 assertions, 0 failures, 0 errors, 0 skips<br>root@9bb7de76347b:/home/app/webapp#</pre><p>The cool thing is that I had to configure nothing else. Rails will clean the database in between runs, it’ll start the puma server to Run the tests and any other thing that was impeding me to adopt this important part of testing.</p><p>If you need help with Docker and Rails, check out <a href="http://github.com/pacuna/dckerize">my Dckerize project </a>to generate all the necessary boilerplate.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a90c52ed0648" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>