<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pavel Pokrovsky on Medium]]></title>
        <description><![CDATA[Stories by Pavel Pokrovsky on Medium]]></description>
        <link>https://medium.com/@pavelpokrovsky?source=rss-d9405198864e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*5uW5ZKunedkjrSQ-.</url>
            <title>Stories by Pavel Pokrovsky on Medium</title>
            <link>https://medium.com/@pavelpokrovsky?source=rss-d9405198864e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:13:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pavelpokrovsky/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Distributed Computing with Spark. On laptop. Part 2 (of 2).]]></title>
            <link>https://medium.com/@pavelpokrovsky/apache-spark-archlinux-on-laptop-part-2-fde8ec7367ac?source=rss-d9405198864e------2</link>
            <guid isPermaLink="false">https://medium.com/p/fde8ec7367ac</guid>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[arch-linux]]></category>
            <dc:creator><![CDATA[Pavel Pokrovsky]]></dc:creator>
            <pubDate>Tue, 16 Apr 2019 07:55:05 GMT</pubDate>
            <atom:updated>2019-04-16T07:55:05.296Z</atom:updated>
            <content:encoded><![CDATA[<h3>Preface</h3><p>In the <a href="https://medium.com/@pavelpokrovsky/apache-spark-archlinux-on-laptop-part-1-73a82527ea7">previous part</a> we successfully deployed a fully functional Spark cluster on our home laptop. Cluster consists of three nodes: master and two workers. Master primary role is to delegate tasks to workers (aka <em>slaves</em>, but we don’t call them like that), which are doing actual computations. Also master will host our interface to submit tasks. Speaking of which, as a programming language we will be using Python 3.7 and tasks will be submitted with Jupyter. There’re tons of articles and books on Jupyter. This is simply put a web interface, which allows you to write and execute code, written in different programming languages. Again, Jupyter comes from Python world, so Python support is built-in. Now let’s make our hands dirty and install programming environment to write our computational tasks.</p><h3>Installation</h3><p>We have already installed Python 3.7 and prerequisites in the previous article. Now we will install necessary modules to run Jupyter notebooks. Log in to your master host as ‘<em>spark</em>’ user and create a directory <em>~/venvs</em> — this is a directory where our Python virtual environments will reside. Python virtual environment is simply a collection of Python packages for specific application. Let’s call our application ‘<em>medium</em>’ and create our venv:</p><pre>$ virtualenv ~/venvs/medium</pre><p>This will install python executable and some basic packages, such as <em>pip</em> and <em>wheel</em>. Now let’s switch to our venv:</p><pre>$ source ~/venvs/medium/bin/activate</pre><p>It should give you a nice prefix to shell prompt with venv name in it (in our case ‘medium’)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*kGAhQYZpRcFII2Qijvq6Mg.png" /></figure><p>For package installation in Python venvs, we will be using pip utility. This is a common way to install Python packages. Let’s install jupyter:</p><pre>$ pip install jupyter</pre><p>You should not worry about current directories, while in venv. Pip is smart enough to install packages with correct paths. It will also install all necessary dependencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*CBeKDKAizkjyMvFuhb5TJg.png" /></figure><p>This is it with the installation. But before we start using Jupyter UI, we will do some additional configuration to make our life a bit easier.</p><h3>Configuration</h3><p>If you followed all the steps, you should be now in master host with active medium venv and installed Jupyter. Let’s create <em>~/apps/medium</em> directory — this is a directory where our Jupyter Notebooks will live.</p><p>Jupyter by default is started on local interface hence will not be available from your host machine. In order to fix that, we will need to do some tweaking to Jupyter configuration. First, let’s create configuration file for Jupyter:</p><pre>(medium) $ jupyter notebook –-generate-config</pre><p>Let’s set a password for Jupyter. This is purely for convenience. Default authentication option for Jupyter is a token, which is provided when Jupyter is started, but it may become tedious to copy and paste it every time, especially from virtual machine environment.</p><pre>(medium) $ jupyter notebook password</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*TWaLLIGq7nvJKbR03KGbDw.png" /></figure><p>We need to edit Jupyter configuration file to enable it running on external network interface. Edit Jupyter configuration file at <em>~/.jupyter/jupyter_notebook_config.py</em> and add following lines to it:</p><pre>c.NotebookApp.ip = &#39;192.168.56.200&#39;</pre><pre>c.NootebookApp.open_browser = False</pre><p>This will enable Jupyter to start on public network interface and will prevent it from attempting to run a local browser (you don’t have it on your virtual machine anyway). After you’ve done that, try and run jupyter notebook server:</p><pre>(medium) $ jupyter notebook</pre><p>It should give you something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*q1saAmaVTBlsnWdQiLjTNg.png" /></figure><p>It will also be available from your host machine at <a href="http://192.168.56.200:8888/">http://192.168.56.200:8888/</a>. I dare you to try it! Just log in with the password you set previously.</p><p>For now, you can safely shut down Jupyter, because this is not how we will be starting it anyway. But at least now we have configured Jupyter Notebooks server, which is great!</p><p>Now in order to integrate Python with Spark, we will use another library — <a href="https://spark.apache.org/docs/2.2.1/api/python/pyspark.html">pyspark</a>. You don’t have to install it separately, it comes pre-bundled with Spark distribution, that you installed in first part of this article. But to use it together with Jupyter, you have to run <em>pyspark</em> with extra environment variables set. Luckily it’s not that complicated, all you need to do is to run following command (all in one line):</p><pre>(medium) $ PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=&quot;notebook&quot; pyspark</pre><p>If you’ve done everything correctly, it should give you the same bunch of messages, as with normal Jupyter launch.</p><p>Congratulations! You have successfully completed configuration of your user interface for Spark! Now let’s try and write some actual code.</p><h3>Computing Pi</h3><p>In order to demonstrate a glory of distributed computation, we’re going to solve some mathematical problem. You might be familiar with Pi number — this is the sixteenth letter of the Greek alphabet, but more important it is one of the fundamental constants in our world. It is roughly equal to 3.14, it represents the ratio of the circumference of a circle to its diameter. It is irrational number which means that it cannot be expressed by a simple fraction. It is also known as ‘infinite decimal’ — after the decimal point, digits will go on forever. The difficulty of computing Pi is directly linked to a precision, a number of digits after the decimal point. We will write a program where precision will be configurable and will compare run times between standalone and cluster execution of spark depending on precision.</p><p>Let’s go and log in to our Jupyter Notebook. Open <a href="http://192.168.56.200:8888/">http://192.168.56.200:8888/</a> in your browser and create a new Python 3 notebook. This is what it should look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9QsgK7n2lUXcHWVH6J5_3A.png" /></figure><p>Now we can type some Python code into notebook and see the result. But here’s a tricky part. If you run pyspark with the command, that I gave above:</p><pre>(medium) $ PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=&quot;notebook&quot; pyspark</pre><p>This will launch pyspark in standalone mode, meaning it’s another instance of Spark which is not aware of cluster environment. This will be useful too, since we will be comparing performance between standalone and cluster mode, but in order to connect to cluster, pyspark should be run with following line:</p><pre>(medium) $ PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=&quot;notebook&quot; pyspark –master spark://master:7077</pre><p>This explicitly tells <em>pyspark</em> to connect to existing Spark master node. Other option is to stop standalone Spark context and create another with connection to Master. This is how it’s done in Jupyter Notebook:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8fe9d46e7a04e76fbfd23342bdbeac15/href">https://medium.com/media/8fe9d46e7a04e76fbfd23342bdbeac15/href</a></iframe><p>If you put this snippet in Jupyter Notebook and run it, you will see your application running in Spark UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QjsiIaUs_Lf5Bh3LeqI_vw.png" /></figure><p>Now let’s write a small program to compute Pi depending on precision. We will need to collect some execution time statistics. Luckily, Jupyter provides handy magic command for this. <em>%%timeit</em> magic macro will execute code below a few times consequently and will provide feedback on performance.</p><p>Here’s a full source code of our script:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fde49c0ccb53c4063b4f31e14ab95ac1/href">https://medium.com/media/fde49c0ccb53c4063b4f31e14ab95ac1/href</a></iframe><p>Using SAMPLES constant we can tweak precision. We will also wrap our script in <em>%%timeit </em>macro. Let’s try and run it! After few moments it will give you an output as well as execution times with some nice stats:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kXVKbRQKIye9QsgV7mEciQ.png" /></figure><p>It was really fast because we did not ask for that much precision. When we will start increasing precision, execution times will increase dramatically.</p><p>Okay, looks like we’re ready to start our measurements. First, let’s collect performance stats from standalone Spark context. Stop Jupyter notebook and start it with following command:</p><pre>(medium) $ PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=&quot;notebook&quot; pyspark</pre><p>Run notebook with each value of SAMPLES constant then stop Jupyter and start it again with this command:</p><pre>(medium) $ PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=&quot;notebook&quot; pyspark –master spark://master:7077</pre><p>Repeat the same procedure. You should end up with performance measurements for Pi calculation for every value of SAMPLES.</p><p>Below I put together a small table with my statistics, depending on required precision</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*4KvpZKpvuNgwDrG1jL6qHQ.png" /></figure><p>Interestingly, with a smaller number of samples, cluster configuration performs slower, than standalone. This is probably due to network latency time and overhead, needed for master to distribute task to workers. But with the larger amount of samples, cluster configuration performs 2–3 times better than standalone!</p><p>Let’s make our task even more interesting. Remember, we’ve configured our master as a dedicated master, so it does not perform any calculations, just delegates it to workers. It does not have to be this way, master itself is fully capable to do calculations as well. All we need to do to configure it, is to add master hostname to <em>/opt/apache-spark/conf/slaves</em> and since master is using ssh to connect to workers, we need to add master’s own public key to <em>~/.ssh/authorized_keys</em> file:</p><pre>$ cat ~/.ssh/id_rsa.pub &gt;&gt; ~/.ssh/authorized_keys</pre><p>After that simply restart your cluster and you should see three nodes up and running in your Spark UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YdkKlsefCkRKWf9BzJLkWg.png" /></figure><p>Now let’s do our same task with Pi calculation, but this time using 3 nodes in cluster.</p><p>Below is the same table with extra column, demonstrating performance of this cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*A6_-Wv-5HL65WFZwJRKLbw.png" /></figure><p>Interestingly, our cluster of three nodes with higher values of samples performs worse than cluster of two nodes. The explanation is actually simple: remember, our host machine is just a laptop with two cores CPU. Each worker runs on its own core, while third worker does not have a dedicated core and calculations has to be shared. Taking into account an overhead to manage a cluster, we end up with worse performance, than when every node performs on its own core. This basically sums up an idea behind distributed computing, made using Hadoop/Spark:</p><ol><li>It is relatively easy to deploy a cluster. Most of the operations can be automated with shell scripts and orchestrating engines, such as Kubernetes or Docker.</li><li>You don’t have to have a super computer with multiple CPUs/cores, a commodity hardware with an option to quickly add extra nodes can be quite sufficient to perform calculations.</li></ol><p>These two factors are what made Hadoop stack so popular in distributed computing. There’s also a lot to learn in Hadoop stack: HDFS, Map/Reduce and all kinds of magnificent solutions. I hope you will continue your journey to this world and I was able to give you a small glance on its wonders. Thank you for reading. I hope you enjoyed reading and doing hands-on as much as I enjoyed writing this article. Please feel free to comment and ask any questions, I will appreciate it very much.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fde8ec7367ac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Distributed Computing with Spark. On laptop. Part 1 (of 2).]]></title>
            <link>https://medium.com/@pavelpokrovsky/apache-spark-archlinux-on-laptop-part-1-73a82527ea7?source=rss-d9405198864e------2</link>
            <guid isPermaLink="false">https://medium.com/p/73a82527ea7</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[jupyter]]></category>
            <category><![CDATA[arch-linux]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[hadoop]]></category>
            <dc:creator><![CDATA[Pavel Pokrovsky]]></dc:creator>
            <pubDate>Tue, 02 Apr 2019 10:55:40 GMT</pubDate>
            <atom:updated>2019-05-13T12:06:27.093Z</atom:updated>
            <content:encoded><![CDATA[<h3>Preface</h3><p>Yes, you read it correctly. This article is about basic aspects of distributed computing with hardware, you (probably) already have. We will be using a commodity Windows laptop, where we will install <a href="https://www.virtualbox.org/">Virtualbox</a> and deploy a <a href="https://spark.apache.org/">Spark</a> cluster with three nodes, each working on its own virtual machine and performing dedicated role within Spark cluster. Spark is an analytics engine for distributed computing. It is widely used across Big Data industry and primarily known for its performance, as well as deep integration with <a href="https://hadoop.apache.org/">Hadoop</a> stack.</p><p>The reason why I decided to make this tutorial is simple: there’re many beginner-level questions on Internet on how to install and configure Spark, but I could not find a full-scale tutorial, which would go through this process step-by-step. So why not create one.</p><p>As a performance task, we will be computing <a href="https://en.wikipedia.org/wiki/Pi">Pi </a>— a mathematical constant that is defined as the ratio of a circle’s circumference to its diameter. There’re plenty of ways to compute Pi and probably even more ways to implement it in Python (yes, we will be using <a href="https://www.python.org/">Python</a>), but the idea behind this tutorial is to compare performance between stand-alone and cluster deployments of Spark, therefore we will use the same script to run in stand-alone and in cluster environments without extra optimizations, such as using specialized math libraries (<a href="http://www.numpy.org/">numpy</a> for example). This is the script that we will be running in our environment:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fde49c0ccb53c4063b4f31e14ab95ac1/href">https://medium.com/media/fde49c0ccb53c4063b4f31e14ab95ac1/href</a></iframe><p>To use this tutorial in full, you need to be familiar with VirtualBox and Linux-based operating systems. Some Python knowledge is also desirable, although not necessary.</p><p>We will be using a simple laptop with 2 core i5 CPU and eight gigs of RAM. For operating system I chose <a href="https://www.archlinux.org/">ArchLinux</a>, which is a very simple and customizable Linux distribution. It is also famous for providing latest versions of software. We will be using <a href="https://www.python.org/">Python 3.7</a> in our exercise, so it’s nice having it out of the box, rather than compiling it by hand.</p><p>This tutorial covers following steps:</p><ul><li>Installing and configuring master node of Spark cluster on virtual machine</li><li>Cloning virtual machine to create workers</li><li>Configuring and running Spark in stand-alone and cluster environments</li><li>Installing and running Jupyter for task submissions</li></ul><p>Now let’s take it to real hands-on.</p><h3>Environment</h3><p>My physical machine is a laptop with two cores of i5–8540U running at 1.6GHz with eight gigs of RAM and running Windows 10 as host operating system. It is nothing out of ordinary and used primarily to work with documents and watch movies. I tried playing <a href="https://en.bandainamcoent.eu/dark-souls/dark-souls-iii">Dark Souls 3</a> on it, but it’s way too laggy — just to give you an idea on overall performance.</p><p>I use <a href="https://www.virtualbox.org/">VirtualBox</a>, which is a freeware virtualization engine. My pseudo-cluster environment is as following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*y7Q118XjtHIU0uGbBCS9RQ.png" /><figcaption>Spark cluster on VirtualBox</figcaption></figure><p>‘<em>spark-master</em>’ node is not doing any computational jobs, serving just as a pure master node for Spark cluster. All hard computations will lie on shoulders of ‘<em>spark-worker-1</em>’ and ‘<em>spark-worker-2</em>’ machines.</p><h3>Installation</h3><p>We will skip creation of virtual machines in Virtualbox environment (it’s trivial, just don’t forget to add extra network interfaces and set it to ‘Host-only’) and proceed directly to installation of ArchLinux. It is very well detailed in <a href="https://wiki.archlinux.org/">ArchLinux Wiki</a> and you definitely should refer to it often. You can <a href="https://www.archlinux.org/download/">download</a> ArchLinux bootable ISO image ArchLinux website. I will install and do a basic configuration of ‘<em>spark-master</em>’ machine first, then I will clone it to worker nodes since general configuration is the same across all virtual machines. This aspect is particularly interesting — you can automate creation and deployment of new cluster nodes using tools such as <a href="https://kubernetes.io/">Kubernetes</a> and <a href="https://www.docker.com/">Docker</a>. So literally you can add extra nodes to cluster with just one click. I will accompany installation with screenshots when applicable.</p><p>We start with booting ArchLinux installation ISO. After you booted an installation ISO, you are getting straight to shell prompt:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*3woivBxjDmjkQl0guixcKg.png" /><figcaption>initial screen of ArchLinux bootable ISO</figcaption></figure><p>From here, you do all the installation and configuration.</p><p>First thing is to create a linux partitition.</p><pre># cfdisk /dev/sda</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*qxJ_ZZT2ykTZY_QaW2LqkA.png" /><figcaption>cfdisk screen</figcaption></figure><p>For a quick setup, select ‘<strong>dos</strong>’ as a label type and mark partition as primary bootable. We will not be creating swap partition for this tutorial. This is what it should look like after you have done basic settings. Write partition to disk and quit <em>cfdisk</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*M0ukSKV_biQFOMJc8Rnsng.png" /></figure><p>Now it is time to create ext4 filesystem on <em>/dev/sda1</em>. Simply type:</p><pre># mkfs.ext4 /dev/sda1</pre><p>and mount partition with</p><pre># mount /dev/sda1 /mnt</pre><p>You have your disk ready to install ArchLinux distro.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*jQjSmb2LcbLa2VrPdKGVDQ.png" /></figure><p>Next step is to install basic ArchLinux packages. They are provided as package groups. We need to install <em>base </em>and <em>base-devel</em> groups. Use this command:</p><pre># pacstrap /mnt base base-devel</pre><p>All the required packages will be downloaded and installed automatically. <strong>Please note</strong> that an access to Internet is required at this point and if you are behind proxy — a corresponding proxy must be configured. ArchLinux wiki will help you here.</p><p>Once you are finally done with pacstrap, you have ArchLinux core installed. Now we will do some additional package installation and configuration. First, you must bootstrap you disk partition:</p><pre># genfstab –U /mnt &gt;&gt; /mnt/etc/fstab</pre><p>and then you may safely chroot to your ArchLinux environment:</p><pre># arch-chroot /mnt</pre><p>This is what it should look like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*y4GRNi_euXCPFjswRZWSXw.png" /></figure><p>Additional packages we will need are: <em>grub, linux-header, virtualbox-guest-dkms, network-manager, python3, python-virtualenv, python-pip, vim, openssh, git</em></p><p>We will need Python because ultimately we will be using <a href="https://jupyter.org/">Jupyter</a> and our computational task will be written in Python. <a href="https://www.openssh.com/">OpenSSH</a> is how nodes in cluster communicate with each other. <a href="https://git-scm.com/">Git</a> is needed to use AUR (I will give more details below) and <a href="https://www.vim.org/">vim</a> is just handy for config files editing. Let’s do it all in one go:</p><pre># pacman –S grub linux-headers virtualbox-guest-dkms networkmanager python3 python-virtualenv python-pip vim openssh git</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*GdDOHT_yQ394v9KBik0FpA.png" /></figure><p>Now after we have successfully installed (most) of the required packages, all that is left is to install boot loader and we are ready to boot in our brand new ArchLinux environment:</p><pre># grub-install /dev/sda</pre><pre># grub-mkconfig –o /boot/grub/grub.cfg</pre><pre># reboot</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*ZSn13PX6iXft87bfV4HhMA.png" /></figure><p>Now when we have installed initial ArchLinux, we will need to do some additional configuration and package installation, including Spark and Hadoop packages. Please note that neither Spark nor Hadoop are provided as part of ArchLinux distro packages, but they exist in AUR. AUR stands for <a href="https://aur.archlinux.org/">ArchLinux User Repository</a> and it is something that also makes ArchLinux shine. You can find almost any software in AUR and it will be installed just as a normal package (after some really basic pre-configuration). We will go through these steps in a moment. But first we will create our <em>spark </em>user. Log into your <em>spark-master</em> machine as <em>root</em>. After fresh install, it won’t ask you a password, but it makes sense to set it straight after login. Now create a new user ‘<em>spark</em>’ with home in <em>/home/spark</em> directory:</p><pre># useradd –d /home/spark –m –G wheel spark</pre><p><strong>Important note</strong> — we’re adding <em>spark </em>user to group <em>wheel </em>here. This is an administrative group. I am doing that for sake of <em>sudo </em>convenience, because I will be doing a lot of <em>sudo </em>and will generally use spark user for administrative purposes. This is purely for demonstration purposes; you should never do that in production environment.<strong> I will also do another horrible thing here: </strong>I will allow <em>wheel </em>group to execute <em>sudo </em>commands without entering password. To do this, I will uncomment this string in <em>/etc/sudoers</em>:</p><pre>%wheel ALL=(ALL) NOPASSWD: ALL</pre><p>Now when we are finished unsecuring our system, we will enable and start our network services including ssh. I am already in <em>spark </em>user environment, so I am switching to <em>sudo</em>:</p><pre>$ sudo systemctl enable NetworkManager --now</pre><pre>$ sudo systemctl enable sshd --now</pre><p>This will enable and start our network interfaces and ssh daemon and you can check if they’re up and running with:</p><pre>$ ip addr</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*vDYepv7MY8rRj-Rb0Hq4IQ.png" /></figure><p>Note, that we are not assigning and static addressing to our host-only interface (<em>enp0s8</em>). We will do it, but a bit later.</p><p>It is finally time to install Hadoop and Spark. I usually create <em>~/aurs</em> directory, where I store and build packages, installed from AUR. This is purely for convenience; you can choose any directory to which you have write permissions.</p><p>First, we will git clone Hadoop and Spark manifests from respective AUR repos:</p><pre>$ cd ~/aurs</pre><pre>$ git clone <a href="https://aur.archlinux.org/hadoop-git">https://aur.archlinux.org/hadoop-git</a></pre><pre>$ git clone <a href="https://aur.archlinux.org/apache-spark.git">https://aur.archlinux.org/apache-spark.git</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*qJpMuQzbCvvvK6lrsAXHRQ.png" /></figure><p>Strictly speaking, Hadoop is not required to run Spark, but Spark package from AUR at the time of writing has some errors that can be easily fixed by using Hadoop. That is why we will be installing Hadoop as well.</p><p>Change to <em>~/aurs/hadoop</em> directory and run</p><pre>$ makepkg –si</pre><p>This command downloads, builds and installs package from AUR just like normal ArchLinux package. At first run, it will prompt you to choose JDK version. I am using JDK8 for this demo, so I am picking <em>jdk8-openjdk</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*qkXXnSlHdihikw03iiai1A.png" /></figure><p>It is the same with <em>apache-spark</em> package:</p><pre>$ cd ~/aurs/apache-spark</pre><pre>$ makepkg –si</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*oe9g2AoabfBM_-YRIFQ-Xg.png" /></figure><p>Now as you probably remember, we will be configuring Spark for cluster environment and we have another two virtual machines to install. Fret not, for you won’t have to go through all these steps again. We will use an amazing clone feature from VirtualBox, which will allow us to create our virtual machines in no time. They will, of course, require some additional configuration, such as ssh key management and network, but other than that they are ready for running Spark. Before we proceed to cloning, we will add some network information, which will allow us to save time on configuring every virtual machine in cluster. We will add following lines to <em>/etc/hosts</em>:</p><pre>192.168.56.200 master spark-master</pre><pre>192.168.56.201 worker1 spark-worker1</pre><pre>192.168.56.202 worker2 spark-worker2</pre><p>As you have guessed, this is a network topology for our cluster. We will also configure workers (or slaves, but we don’t call them that way) for our Spark master.</p><pre>$ cd /opt/apache-spark/conf</pre><pre>$ sudo cp ./slaves.template ./slaves</pre><pre>$ sudo vim ./slaves</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*T0zJulDVt11w_kAEjURUNg.png" /></figure><p>By default, <em>apache-spark</em> is installed in <em>/opt/apache-spark</em> under root ownership and it also creates working directories in <em>/var/lib/apache-spark</em>. We will change ownership for this directories to <em>spark:wheel</em></p><pre>$ sudo chown –R spark:wheel /opt/apache-spark</pre><pre>$ sudo chown –R spark:wheel /var/lib/apache-spark/work</pre><p>Again, this is not super-secure, but okay for demonstration purposes.</p><p>This is the final step in installation and we can finally switch off our spark-master machine and proceed to cloning. I will not go into much detail here, just use default settings for cloning. It is also convenient to group virtual machines in VirtualBox manager interface. This is what it should look like after cloning:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*D2QWzSivFen6NGUShAASqg.png" /></figure><p>Okay, let’s go and start our cluster!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*w_NmK_ayI5LEr6KSQAd1tA.png" /></figure><p>Detachable start in VirtualBox allows you to close windows with running virtual machines, leaving them to run in background. This is useful if you don’t want to clutter your desktop with multiple windows.</p><h3>Configuration</h3><p>We will start with network configuration. I will show how it is done on ‘<em>spark-master</em>’ machine, so you will be able to easily reproduce this step on every machine in cluster.</p><p>Start with logging in as ‘<em>spark</em>’ user and then run:</p><pre>$ sudo nmtui</pre><p>We will configure host-only interface which is <em>enp0s8 </em>and set static IP address and gateway to <em>192.168.56.200</em> and <em>192.168.56.1</em> correspondingly. For <em>worker1</em> and <em>worker2</em> IP addresses will be correspondingly 1<em>92.168.56.201</em> and <em>192.168.56.202</em> and gateway is the same <em>192.168.56.1</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*3Io5kddeoq67Vc27H7mucw.png" /></figure><p>Set hostnames for all machines:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*0_K8uAzH8Oa2Zwb44cPxww.png" /></figure><p>It should be correspondingly:</p><pre>spark-master</pre><pre>spark-worker1</pre><pre>spark-worker2</pre><p>Ultimately, you should end up with every machine successfully pinging every machine in cluster.</p><p>After you’ve done with network configuration, it is time to configure <em>ssh</em>. It is used by Spark master to communicate with other nodes in cluster (workers). By configuring SSH, you will be able to start and stop all cluster services with just one command on master. We will be using key-based authentication. Let’s configure keys at master node. Similar approach will be used on worker nodes.</p><p>As spark user, run following command on ‘<em>spark-master</em>’:</p><pre>$ ssh-keygen</pre><p>This will create <em>~/.ssh</em> directory where your private and public ssh keys are stored (<em>id_rsa </em>and <em>id_rsa.pub </em>respectively). You must do this as <em>spark </em>user on every machine in cluster. Don’t set passphrase for private key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*fJJVoRkkkFp3j9TwV9q1iw.png" /></figure><p>Now in order to use key-based authentication, you must copy-paste your master public key into <em>~/.ssh/authorized_keys</em> file on every worker machine. The easiest way to do this in this case is simply use <em>scp </em>command to copy public key to corresponding <em>authorized_keys</em> file on worker nodes from master:</p><pre>$ scp ~/.ssh/id_rsa.pub spark@worker1:~/.ssh/authorized_keys</pre><pre>$ scp ~/.ssh/id_rsa.pub spark@worker2:~/.ssh/authorized_keys</pre><p>You know if it works, when you will be able to ssh as ‘<em>spark</em>’ user from master to every worker without entering password.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*5xSh9Aedv_NkbvswRFyF5A.png" /></figure><p>Finally, we need to configure master and local IPs in Spark on our nodes to prevent weird resolving issues. On every node edit <em>/opt/apache-spark/conf/spark-env.sh</em> and add following lines:</p><pre>SPARK_MASTER_IP=192.168.56.200 # master node</pre><pre>SPARK_LOCAL_IP=&lt;IP address of host-only interface&gt;</pre><p>You’re probably wondering, what else do we have to tweak to finish configuring our cluster? But that’s it! Our cluster is ready to run and we can start it with one (okay, with two) simple commands:</p><pre><em>$ cd /opt/apache-spark/sbin</em></pre><pre><em>$ ./start-all.sh</em></pre><p>If everything is configured correctly, it should show messages on started master and two workers. It may also give <em>HADOOP_SLAVES</em> warning, but you can safely ignore it for demonstration purposes.</p><p>You can check logs in <em>/opt/apache-spark/logs</em>, it should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*ApgHFIUrJqbsANPtvUhQ0Q.png" /></figure><p><strong>Note, that workers are running on public interfaces (<em>192.168.56.*</em>) — this is important, otherwise you will run into issues with task submission.</strong></p><p>Behold! If you open <a href="http://192.168.56.200:8080/">http://192.168.56.200:8080</a> in your browser on host machine, it should present you an initial page of Spark UI, through which you are able to view current status of workers and running applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ZxzTBGf6Vu47Mh523zb0w.png" /></figure><p>This is it! That concludes our first part of <strong>Distributed Computing with Spark. On laptop. </strong>In <a href="https://medium.com/@pavelpokrovsky/apache-spark-archlinux-on-laptop-part-2-fde8ec7367ac?source=friends_link&amp;sk=04b8ba9084d9dbe970f7d3bb7208c8ae"><strong>part 2</strong></a> we will do an actual implementation of distributed computing task and will measure performance comparing to the same computing but done in a single-machine environment. I will appreciate any comments as well as advises on how to improve things.</p><h4>To move on to the next part, click <a href="https://medium.com/@pavelpokrovsky/apache-spark-archlinux-on-laptop-part-2-fde8ec7367ac?source=friends_link&amp;sk=04b8ba9084d9dbe970f7d3bb7208c8ae">here.</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73a82527ea7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>