<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[LuckSpark - Medium]]></title>
        <description><![CDATA[Spark | Scala | Python | Pandas for Beginners - Medium]]></description>
        <link>https://medium.com/luckspark?source=rss----d533d9556b8c---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>LuckSpark - Medium</title>
            <link>https://medium.com/luckspark?source=rss----d533d9556b8c---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:40:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/luckspark" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Hashing Pandas DataFrame Column with Nonce]]></title>
            <link>https://medium.com/luckspark/hashing-pandas-dataframe-column-with-nonce-763a8c23a833?source=rss----d533d9556b8c---4</link>
            <guid isPermaLink="false">https://medium.com/p/763a8c23a833</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[dataframes]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[hashing]]></category>
            <dc:creator><![CDATA[LC]]></dc:creator>
            <pubDate>Wed, 29 Aug 2018 15:30:43 GMT</pubDate>
            <atom:updated>2018-08-29T15:30:42.622Z</atom:updated>
            <content:encoded><![CDATA[<p>Your dataset can commonly contain sensitive data in one or more columns. For example, user IDs, patient numbers, or license numbers. Here I share how to create a new column containing hashed strings based on the clear-text strings of the other column of Pandas DataFrame.</p><p>Here are the properties of my code:</p><ol><li>It is designed to be unconvertible back to the original clear-text data, even for the data owner. Technically it is possible if you keep the nonces (the random numbers) used to generate the hashes. But if you simply, intentionally or accidentally, re-run the code again, the random numbers will be completely different, and identifying the clear-text from the hash would be very difficult.</li><li>It is designed to generate hashes such that <strong>the hash based on the same original clear text</strong> <strong>will be the same</strong>. That is, nonces generated for the same clear-text string are the same. Therefore, although the records are kept anonymous, readers know which rows are from the same (hashed) name/id/thing/type. For example, if the hashed column is hospital’s patient IDs, we can tell which records belongs to the same patient (without knowing who).</li></ol><p>This can be useful when you wish to conceal sensitive data before releasing the dataset to public or third party.</p><h3>The Data</h3><p>To make it practical, I create a DataFrame (DF) from a real .csv file, containing several columns. Here I use the results of English football leagues of 2014/2015 season, E0.csv, <a href="http://www.football-data.co.uk/mmz4281/1415/E0.csv">which can be downloaded here</a>.</p><p>I strip the columns down to only few and create a new hash column based on the column named HomeTeam(i.e., I suppose HomeTeam is our sensitive data). In case you are curious, here are descriptions of some columns of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b_FICzD1v2wwzHVhPuZcXg.png" /><figcaption>Some abbreviation descriptions of the E0.csv file.</figcaption></figure><h3>Hashing and Nonce</h3><p>In short, <strong>Hashes</strong> can be created by entering a clear text as a parameter to a hash function. The same clear text would generate the same hash value from the same hash function.</p><p>Suppose you have a document with hashed ID numbers. If you happen to know someone’s ID, together with the hash algorithm (there are only a handful of the well-known ones), it is easy to identify the hashed records of the known ID.</p><p>This is why we need nonces. <strong>Nonces</strong> are some random strings or numbers added to the original text. With nonces appended or mixed to the original clear-text string as an input to the hash function, knowing only the ID would not be enough to identify the records anymore.</p><h3>Algorithm</h3><p>Here is the algorithm of my code.</p><ol><li>Imports and reads the .csv file as a DF (d0). Select only few columns as d1.</li><li>Create a list of the data from the sensitive, clear text column (‘HomeTeam’ in this case)</li><li>Get a unique list of the clear text. The purpose is to generate the same nonce for the same clear text value.</li><li>Generate 2 nonces for each clear text, and added in front and behind the clear text. For example, a clear text Liverpool becomes aaaaaaaLiverpoolbbbbbbbb.</li><li>Create a new DF (as d2 in the code, just to leave the original d1 intact) and create a new hash column.</li><li>Generate hashes in hexadecimals (0–9, a-f, and A-F)</li><li>Create new DF (d3) with the original clear text column removed and replaced with the hash.</li></ol><h3>The Code</h3><pre>import pandas as pd<br>import hashlib<br>import random<br>import string</pre><pre>d0 = pd.read_csv(&#39;./E0.csv&#39;)<br>d1 = d0[[&#39;Date&#39;,&#39;HomeTeam&#39;,&#39;FTR&#39;,&#39;HTR&#39;,&#39;Referee&#39;]].dropna()<br>display(d1.head())</pre><pre># Get a unique list of the clear text, as a List<br>tmplist = list(set(d1[&#39;HomeTeam&#39;]))</pre><pre># Add some random characters before and after the team name.<br># Structured them in a Dictionary<br># Example -- Liverpool -&gt; aaaaaaaLiverpoolbbbbbbbb</pre><pre>mapping1 = {i : (&#39;&#39;.join(random.choice(string.hexdigits) for i in range(12)))+i+(&#39;&#39;.join(random.choice(string.hexdigits) for i in range(12)))  for i in tmplist}</pre><pre># Create a DF to leave the original DF intact.<br>d2 = d1.copy()</pre><pre># Create a new column containing clear_text_Nonce<br>d2[&#39;newname&#39;] = [mapping1[i] for i in d2[&#39;HomeTeam&#39;]]<br>display(d2.head())</pre><pre># Hash the clear_text+Nonce string<br>d2[&#39;hash&#39;] = [hashlib.sha1(str.encode(str(i))).hexdigest() for i in d2[&#39;newname&#39;]]<br>display(d2.head())</pre><pre># To proof that the same clear text has the same Hash string.<br>display(d2[d2[&#39;HomeTeam&#39;]==&#39;Chelsea&#39;].tail(30))</pre><pre># Create a new DF with the clear-text column removed.<br>d3 = d2[[&#39;Date&#39;,&#39;hash&#39;,&#39;FTR&#39;,&#39;Referee&#39;]].rename(columns={&#39;hash&#39;:&#39;HomeTeamHash&#39;})<br>display(d3.head())</pre><h3>The Output</h3><p>Here is the original DF (d1) with clear text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*Rq5kFeKE7BDriIfs9_Ap8g.jpeg" /><figcaption>The original DataFrame with clear text (HomeTeam) to be hashed.</figcaption></figure><p>Here is the new DF (d3) with hashed strings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JDBUFKu10uJ0hFHlJpGwVA.jpeg" /><figcaption>The new DataFrame with the new hashed column.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=763a8c23a833" width="1" height="1" alt=""><hr><p><a href="https://medium.com/luckspark/hashing-pandas-dataframe-column-with-nonce-763a8c23a833">Hashing Pandas DataFrame Column with Nonce</a> was originally published in <a href="https://medium.com/luckspark">LuckSpark</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Simple Solution to Run Multiple PySpark Instances on Multiple Jupyter Notebooks Simultaneously.]]></title>
            <link>https://medium.com/luckspark/a-simple-solution-to-run-multiple-pyspark-instances-on-multiple-jupyter-notebooks-simultaneously-3cdf28170ab9?source=rss----d533d9556b8c---4</link>
            <guid isPermaLink="false">https://medium.com/p/3cdf28170ab9</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[macos]]></category>
            <dc:creator><![CDATA[LC]]></dc:creator>
            <pubDate>Tue, 28 Aug 2018 14:03:27 GMT</pubDate>
            <atom:updated>2018-08-28T14:03:27.166Z</atom:updated>
            <content:encoded><![CDATA[<p>Jupyter Notebook is a great tool for coding PySpark. I have setup my PySpark environment on Jupyter Notebook using these configurations, <a href="https://medium.com/@GalarnykMichael/install-spark-on-mac-pyspark-453f395f240b">taken from here</a>, in my ~/.bash_profile on my MacBook.</p><pre>export PYSPARK_DRIVER_PYTHON=&quot;jupyter&quot; <br>export PYSPARK_DRIVER_PYTHON_OPTS=&quot;notebook&quot;</pre><p>So when I run &#39;pyspark&#39; on my terminal, the Jupyter Notebook opens up.</p><p>I just recently found out that I <strong>cannot</strong> run 2 PySpark’s .ipynb files concurrently. It gave out the instantiation of hive.ql.metadata.SessionHive MetaStoreClient error, caused by ERROR XSDB6: Another instance of Derby may have already booted the database /metastore_db, exactly <a href="https://stackoverflow.com/questions/42078594/how-to-run-multiple-instances-of-spark-2-0-at-once-in-multiple-jupyter-notebook">discussed here</a>.</p><p>According to some sources, e.g., <a href="https://stackoverflow.com/questions/36311185/jupyter-pyspark-how-to-run-multiple-notebooks">here</a> and <a href="https://stackoverflow.com/questions/42078594/how-to-run-multiple-instances-of-spark-2-0-at-once-in-multiple-jupyter-notebook">here</a>, this is caused by that Spark allows it’s light weight DB named Derby to serve only 1 PySpark instance at a time. In other words, once the Derby is used by an instance of PySpark, it seems to be locked and prevent another PySpark instance from sharing it. One possible solution suggested by the discussions is to use Postgres instead of Derby.</p><p>Good news, I found out that there is yet another much more simple way to let us run multiple PySpark notebooks simultaneously, which is: <strong>just put and run the </strong><strong>.ipynb files in separate folders</strong>. It is that simple.</p><p>Here is my scenario. I have DF1.ipynb and Spark DF Tutorial E0.ipynb under the sparkpython folder. Instead of placing these 2 .ipynb files directly under this folder, I create subfolders for each of them and put them there, as shown below. After being executed for the first time, each folder will have its own metastore_db and derby.log files created. That fixed the sharing problem, I guess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5n-t2w3pGlpn516vOKu7pA.jpeg" /><figcaption>My sparkpython folder has subfolders, one for each .ipynb file.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YS-hogBTitWWzkp4_Ot-9A.jpeg" /><figcaption>Each .ipynb file will have their own metastore_db and derby.log</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ob_C60pLItGpF8wD8MS5g.jpeg" /></figure><p>I am not sure if this would be a good solution for a large-scale production environment. It just works for me under a small playground environment and save me some time from Postgres installation.</p><p>The Spark version I use is 2.3.1. My machine is macOS High Sierra. My Python 3.6 and Jupyter Notebook are installed through Anaconda 5.2.</p><p>Hope it helps.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cdf28170ab9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/luckspark/a-simple-solution-to-run-multiple-pyspark-instances-on-multiple-jupyter-notebooks-simultaneously-3cdf28170ab9">A Simple Solution to Run Multiple PySpark Instances on Multiple Jupyter Notebooks Simultaneously.</a> was originally published in <a href="https://medium.com/luckspark">LuckSpark</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Analytics using Pandas: A Quick Practical Tutorial 1]]></title>
            <link>https://medium.com/luckspark/data-analytics-using-pandas-a-quick-practical-tutorial-1-1aab90ca368?source=rss----d533d9556b8c---4</link>
            <guid isPermaLink="false">https://medium.com/p/1aab90ca368</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[LC]]></dc:creator>
            <pubDate>Sat, 25 Aug 2018 13:59:24 GMT</pubDate>
            <atom:updated>2018-09-01T08:37:59.600Z</atom:updated>
            <content:encoded><![CDATA[<p>Pandas is a magnificent tool that empowers python enormously for data analytics. It is able to read and transform structured data in tons of ways. This article will take you through some practical Pandas data transformation using English Premier League season 2014/2015 results as dataset.</p><p>This practical tutorial get you straight to the functions and methods of Pandas to manipulate data and DataFrames without much of syntax and option explanations. I tried to focus on how Pandas functions work for us to get the answer we need. I leave the readers to find out a huge arrays of options and varieties of other syntax patterns from other sources by themselves.</p><p>I summarize Pandas methods/functions learnt (and what they do) for each section in <strong><em>Pandas Summary</em></strong> subsection.</p><h3>The Dataset</h3><p><a href="http://www.football-data.co.uk/mmz4281/1415/E0.csv">This is the direct link to download .csv file of English Premier League season 2014/2015 results used in this tutorial</a>.</p><p>The explanation of column names and abbreviations can be read <a href="http://www.football-data.co.uk/notes.txt">here</a>. Some abbreviations which will be used in this tutorial are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b_FICzD1v2wwzHVhPuZcXg.png" /><figcaption>Figure 0: Some abbreviation descriptions of the E0.csv file. Only these columns are used in this tutorial.</figcaption></figure><h3>Coding Environments</h3><p>I simply installed Anaconda 5.2 for MacOS High Sierra (64-bit with Python 3.6) on my system. I use Jupyter Notebook as my coding IDE.</p><p>I put the E0.csv under datafile folder, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y41fjKly20Y1raMeAJuuVw.jpeg" /><figcaption>Figure 1: File locations viewed from Jupyter Notebook.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yj2Pu0QSlRecdsT92Wtmmw.jpeg" /><figcaption>Figure 2: The E0.csv file is in the ./datafile folder.</figcaption></figure><h3>Data Preparations</h3><h3>Loading The E0.csv file</h3><p>The first step is to create a DataFrame (DF) by importing the .csv file. Simply use pd.read_csv(). In the code below, the first DF created from the E0.csv file is named d0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zTdiTo8UKePxTh7KyprSEw.png" /><figcaption>Figure 3: Import Pandas and create DF by reading from a .csv file. Display the table on Jupyter notebook.</figcaption></figure><p>The display() is Jupyter Notebook’s feature to magically display the table in very eye-pleasing way. Using print() is ok, but display() is more beautiful on Notebook.</p><p>At the end of the screen shown below, the size of the table is also shown — 381 rows × 68 columns. The last line showing 381 is the result of the print(len(d0)) command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VOGpxiUQCIUZ7IV3fVMdXg.jpeg" /><figcaption>Figure 4: Size of the DF and output of len(d0).</figcaption></figure><p><strong><em>Pandas summary:</em></strong></p><ul><li>pd.read_csv() : to create a DF by importing a csv file</li><li>len(d0) : to count the rows of the DF</li></ul><h3>Display only the First or the Last few Rows</h3><p>From the display() command above, although some rows in the middles are skipped, it is still a long output. We can choose to display only the first or the last few rows using .head() and .tail() methods. You can also specify the number of rows you want to display by entering the number of rows as a parameter, e.g., head(10) or tail(10).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eewH3jkVZWu22iSZftLeJw.png" /><figcaption>Figure 5: The .head() method displays the first few rows of the DF.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YpdJaclNl1igThPQX4G9qg.png" /><figcaption>Figure 6: The tail() method displays the last few rows of the DF.</figcaption></figure><p><strong><em>Pandas summary:</em></strong></p><ul><li>.head() and .tail() : to view only the first few or the last few rows of DF.</li></ul><h3>Checking for Missing Data</h3><p>In the image above, showing the result of tail() method, you might notice that the last line of the data shows380 NaN NaN ... NaN. The <strong><em>NaN</em></strong> indicates the <strong><em>missing of data</em></strong>. Let’s examine the file by opening up the E0.csv file using Jupyter Notebook (or on any text editor, e.g., Sublime), which looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MaCydCtjjrcwpf9p_UQjsQ.jpeg" /><figcaption>Figure 7: E0.csv screen shown on Jupyter Notebook.</figcaption></figure><p>When you scroll down to the very end, you probably see this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jEWNOv4B4FqHDz7XdmvRiw.jpeg" /><figcaption>Figure 8: The last lines of the E0.csv file.</figcaption></figure><p>The line 382 contains only an array of commas with out data in between. Therefore, the line 382 causes the <em>NaN</em> to show up when we load the file using Pandas.</p><p>Further examination on the missing of data can be done using df.info() command. The df.info() shows data types and, more importantly, the number of non-null rows (the number rows with valid data) of each column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mpOc1sR2cjjbHJJYZqTz-Q.jpeg" /><figcaption>Figure 9: df.info() shows the name, number of non-null rows, and data types of each column.</figcaption></figure><p>When scrolling down slightly and looking carefully, it can be seen that there are 3 unusual columns, namely SJH, SJD, and SJA, with only 40 non-null value (instead of 380 as in other columns).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vrB1jQ50wIGkkopcG4w6Zw.jpeg" /><figcaption>Figure 10: Notice the 40 non-null columns: SJH, SJD, and SJA. This indicates that these 3 columns has 40 non-null rows of data, and probably 380–40 = 340 rows of missing data.</figcaption></figure><p>Let’s take a look at the raw E0.csv file to see the root cause of the null data. In the file you will see some missing data in some columns, the consecutive commas, as shown in the image below. Thus, these are the missing data of the columns named SJH, SJD, and SJA summarized by the df.info() above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZzrEEucbZ08gmcwVmZg7TA.jpeg" /><figcaption>Figure 11: The missing data viewed thought the raw text editor.</figcaption></figure><p><strong><em>Pandas summary:</em></strong></p><ul><li>df.info() : shows the name, number of non-null rows, and data types of each column.</li></ul><h3>Discarding (Dropping) Missing Data</h3><p>Our objective here is to remove the last line (line 380 of Figure 6) containing the missing data. To discard the missing data, we can use df.dropna(how=’any’) . This removes all rows that contain the <em>NaN</em>.</p><p>However, dropping missing data can be trickier than you think. If we apply the .dropna() to the entire data set (the d0 here), any rows with the missing data caused by the SJH, SJD, and SJA columns will be removed too. The 380 rows of all columns will be shortened down to only 40 rows, which certainly is NOT what we want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3U2NmlX3zg8rqRlArmODDA.png" /><figcaption>Figure 12: d0.dropna() will remove all rows with missing data, ending up with only 40 rows left.</figcaption></figure><p>In our tutorial here, we will use only first few columns and not the SJH, SJA, and SJD. Therefore, we shall select the columns we want to use first, which excludes the SJH, SJA, and SJD columns, then discarding the <em>NaN</em> rows later.</p><p><strong><em>Pandas summary:</em></strong></p><ul><li>df.dropna(how=’any’) : removes all rows that contain the <em>NaN</em>.</li></ul><h3>Selecting Column(s) from DF</h3><p><strong><em>To select ONE column</em></strong> from a DF, we can use df[‘column name’] just like selecting an element from a List in Python. You can also use df.column_name format. The first is easier to remember as it is the same as List element selection syntax. It also supports column name with spaces. The latter is slightly easier to write.</p><p>The column selection will not affect the original DF (d0). We have to return the output to a new DF (d1). The d1 = d0[&#39;HomeTeam&#39;] means creating a new DF named d1 by selecting all rows of the column named &#39;HomeTeam&#39; from the DF d0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eBd77eZ16r63o29ECpRNAQ.png" /><figcaption>Figure 13: Create a new DF, d1, from HomeTeam column of d0.</figcaption></figure><p><strong><em>To select several columns</em></strong>, use double brackets — df[[&#39;name1&#39;,&#39;name2&#39;]]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P8zEg-Ajqaoph-KnvYN6ew.png" /><figcaption>Figure 14: Selecting multiple columns of a DF.</figcaption></figure><p>The code below combines the column selection and <em>dropna() </em>in one line. Notice that the line 380 with <em>NaN</em> is removed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QCufosf6jAyLcHjDo3jzpw.png" /><figcaption>Figure 15: Combine the selection of multiple columns and the dropna( ) in one line.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wFCBXYaU6DAgYPYQMgegfw.png" /><figcaption>Figure 16: The last rows of Figure 15 shows that the NaN row (row 380 or Figure 6) is removed.</figcaption></figure><p><strong><em>Pandas Summary</em></strong></p><ul><li>df[&#39;column name&#39;] or df.column_name : select only a specific column of the DF.</li><li>d0[[&#39;Date&#39;,&#39;HomeTeam&#39;,&#39;AwayTeam&#39;,&#39;FTR&#39;]] : select specific columns of the DF.</li></ul><h3>Data Analytics</h3><h3>Problem 1 : Get the name of all teams played this season</h3><p>Put simply, get the <strong>unique</strong> names of all teams. We all know that there must be exactly 20 team names. An algorithm to solve this problem is simple: get a unique team names of either HomeTeam or AwayTeam column.</p><p>There are few ways of doing this.</p><p>First, use .unique(). Thus print(d1[‘HomeTeam’].unique()) will give you the answer. Of course, print(d1.HomeTeam.unique()) will also work. The output is of type numpy.ndarray. To ensure correctness, we can use .nunique() to count the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kX6Yl5RtRLJIx8a05NlvKQ.png" /><figcaption>Figure 17: Get a list of unique elements of a column using df[‘column_name’].unique( ). Use .nunique() to count.</figcaption></figure><p>Second, exploit the uniqueness property of Python’s Set. Throw the HomeTeam names into a set and it will remove duplicates automatically for you. Thus, print(set(d1[‘HomeTeam’])) will also give you the answer. The output is a set. To count the elements of the Set, we can use len(), as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NK3YiwHY-0tlqpZjvaTHMA.png" /><figcaption>Figure 18: Get a list of unique elements of a column using Set.</figcaption></figure><p><strong><em>Pandas Summary</em></strong></p><ul><li>.unique(): get the unique elements from the column.</li><li>.nunique() : count the unique elements.</li><li>set(df[&#39;HomeTeam&#39;]) : convert DF’s column to Set.</li></ul><h3>Problem 2: How many matches Liverpool win this season?</h3><p><strong>Algorithm :</strong> the FTR column, Full Time Result, indicates which team won the match — H, D, or A. Thus when Liverpool played as a <strong>home team</strong>, we have to filter only rows with FTR=H, and when it played as an <strong>away team</strong>, we have to filter only rows that FTR=A. Then we count the number of rows of both scenarios to get the final answer.</p><p>In practice, we find the results of win games of Liverpool as a home team first. Then to get the answer of the away-team part, it is just a matter of copy and paste with some minor modifications.</p><p>In coding, there are couple of ways to achieve this.</p><p><strong><em>Solution 1</em></strong></p><p>We can perform a series of filters. We first filter Liverpool from HomeTeam. Then we filter H from FTR. Then count.</p><p>To filter Liverpool from HomeTeam, we use</p><p>d21 = d1[d1[&#39;HomeTeam&#39;] ==&#39;Liverpool&#39;]</p><p>(or d21 = d1[d1.HomeTeam==&#39;Liverpool&#39;])</p><p>Then we can filter the d21 again with FTR==&#39;H&#39; using the same syntax</p><p>d22 = d21[d21[&#39;FTR&#39;]==&#39;H&#39;]</p><p>The last step is to count the result using len(d22).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xn3kWESMVAjJbTdp8XKotQ.png" /><figcaption>Figure 19: All home wins of Liverpool this season.</figcaption></figure><p>To find all away wins, simply change HomeTeam to AwayTeam, and FTR==&#39;H&#39; to FTR==&#39;A&#39;.</p><p><strong><em>Solution 2</em></strong></p><p>In stead of applying 2 filters on different columns, we can also set on column as the index of the DF (the index is the header of the rows), then we select rows we want using the index label. This is basically another way to filter the data by row. Then, we will apply another filter on the other column. In this case, we will set HomeTeam as an index, filter the HomeTeam using index selection method, then filter the FTR column to get the answer.</p><p>In the code shown below, we first create a new DF named d23 which derived from d1 with only 2 columns, HomeTeam and FTR. I print out the d23 to show that the index is integers (0, 1, 2, …).</p><p>Then we set the HomeTeam as an index of the d23 using d23.set_index(). The inplace=True option applies the new index setting to the d23 itself. You can see from the result that the HomeTeam becomes the index, replacing the integer 1,2,3, …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SjAsAyCDt8gDrVOpay7g1Q.png" /><figcaption>Figure 20: using .set_index() to set HomeTeam as index of the DF</figcaption></figure><p>We can use .loc() to refer to specific rows of the DF using the index name (i.e., filter the rows using the index). Thus, with the HomeTeam as the index, we can simply locate the index=Liverpool by d23.loc[&#39;Liverpool&#39;].</p><p>To select the FTR==&#39;H&#39;, we can just use the same filtering syntax as in Solution 1 above. Finally we can use len() or .count() to get the final answer.</p><p>As shown in the code below, the FTR==&#39;H&#39; is applied first, followed by the .loc[] to filter Liverpool from the HomeTeam index.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wT5rQdQyapWbEiPbtO2Jzw.png" /><figcaption>Figure 21: Select FTR=H, then use df.loc[‘Liverpool’] to select only the rows with index = Liverpool.</figcaption></figure><p><strong><em>Pandas Summary</em></strong></p><ul><li>df[df.HomeTeam==&#39;Liverpool&#39;]: filter data from a specific column.</li><li>df.set_index(&#39;HomeTeam&#39;, inplace=True): set a specific column as a new index and apply to the DF.</li><li>df.loc[&#39;Liverpool&#39;]: select (filter) rows by specifying index name.</li></ul><h3>Problem 3: Count the number of HOME wins of all teams</h3><p>This problem will introduce you to a new Pandas method: .groupby(). The algorithm is quite simple — select the HomeTeam with FTR=H, then groupby team name, and then count.</p><p>To demonstrate how groupby() works, I split the code into d31, d32, d33, and d34. The actual answer to this question requires only the d31 and d34.</p><p>As shown in the code below, d31 derives from d1 with 2 columns, HomeTeam and FTR, and with FTR==&#39;H&#39; filter. d32 creates a groupby of d31 using HomeTeam. Then selecting only the first element of each group using .first(). I print out d31 and d32 to show how groupby() affects the output. One easily unnoticed impact of groupby() is that the column that we use for the groupby(), i.e., HomeTeam in our case, becomes the index of the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5lR33ZRwKDWP4ZDvmznjAg.png" /><figcaption>Figure 22: d31 derived from d1, then being grouped-by as d32. Notice that ‘HomeTeam’ becomes index of d32.</figcaption></figure><p>Let’s explain a bit more about how groupby() works. The d32 groups d31 by HomeTeam. Thus the FTR of the same team are grouped together. <em>The </em><em>groupby() method requires 3 stages: split, apply, and combine.</em> The groupby() command alone only split DF into groups and, at this stage, you cannot directly print out the group elements using print() or display(). We need an <em>‘apply’</em> method to process and combine the data of each group, and produce final output. Examples of apply methods are count(), first(), sum(), or mean(). We use .first() in d32 to obtain the first element of each group just to demonstrate the groupby() process. Again, notice that the HomeTeam column becomes the index after applying groupby() method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gIPNSv6iaksNlmNbbpjWWw.png" /><figcaption>Figure 23: Demonstration of get_group() and count() methods being applied to groupby().</figcaption></figure><p>From the code above, the d33 is also just for demonstration purpose only. Here the .get_group() is used as an apply function to take a look at elements of the group. Notice that the integer index is not replaced by the HomeTeam. The get_group() only construct DF to display the elements of the named group but has not apply any significant functions to the elements.</p><p>The d34 applies .count() to the groupby(), which counts the elements in each group. The HomeTeam becomes the index of the DF. This is the final answer for this problem.</p><p><strong><em>Pandas Summary</em></strong></p><ul><li>d31.groupby(&#39;HomeTeam&#39;) : groups the DF by the specific column</li><li>d31.groupby(&#39;HomeTeam&#39;).first(): groupby() then obtain the first elements of each group.</li><li>d31.groupby(&#39;HomeTeam&#39;).get_group(&#39;Liverpool&#39;): groupby() then create a DF that contains only elements of the specific group.</li><li>d31.groupby(&#39;HomeTeam&#39;).count() : groupby() then counts elements of each group.</li></ul><h3>Problem 4: Compute total SCORE of all teams</h3><p>A team gets 3 points if it wins a match, 1 if draws, and 0 if loose. The total score of the season of each team is a summation of these points of all matches.</p><p>This problem will introduce you to some new Pandas tricks: create a new DF from Dictionary, compose an IF-ELSE statements, and use the .append() to combine two DFs.</p><p><strong>Algorithms:</strong> First we will create a new DF with team name and points columns. The points column is a new column that does not exist in the original dataset (does not exist in the .csv file). The point computations will be separated into 2 parts, home matches and away matches since the FTR is computed differently. Specifically, on home matches, the FTR and points are computed as H=3, D=1, A=0, while on the away matches, they are H=0, D=1, A=3. When we are done with the computation of both parts, we combine both DF together and sum() the points.</p><p>The first part of codes and outputs are shown below. This parts create new DFs and convert FTR of {H, A, D} to Points {0, 1, 3}, according to the point computation of home team and away team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XGcyY4rBe7_2a4TMEw-f7Q.png" /><figcaption>Figure 24: d41 and d42 are new DFs that convert FTR result to Points.</figcaption></figure><p>Let’s explain the code in more details. The pd.DataFrame() creates a new DF. The new DF is created using Dictionary data structure, as {column_name : element_list}. In our case, the Team column is taken straightforwardly from d1. HomeTeam. The Points column is a list of either 3 or 1 or 0, according to the IF-ELSE condition based on the FTR values.</p><p>The IF-ELSE condition begins by taking elements from d1.FTR (as specified at the very end of the clause). Then put each of the element through the IF statement (‘H’ or ‘D’ or something else (which can possibly be only ‘A’ here)). An integer of either 3, 1, or 0 is returned as a result for each of the element of d1.FTR.</p><p>The code below combines d41 and d42using .append(), then performs groupby() by the Team column, and applies the sum() function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s4PP1gNXDo-tkKQVydh6kQ.png" /><figcaption>Figure 25: d43 combines d41 and d42 together. Then groupby() and sum() the elements.</figcaption></figure><p>.append() combines 2 DFs together by simply appending the rows of one DF at the end of the other. For a smooth append task, both DFs must have the same number of columns and the same column names.</p><p>Applying sum() method to groupby() add all points in the group together. If you want to see what happen before the sum() method, you can try the .get_group() like this:</p><pre>d43 = d42.append(d41).groupby(‘Tam’).get_group(‘Liverpool’)<br>display(d43)</pre><p><strong><em>Pandas Summary</em></strong></p><ul><li>pd.DataFrame(): create a new DF using Dictionary data structure.</li><li>[3 if i==&#39;A&#39; else 1 if i==&#39;D&#39; else 0 for i in d1.FTR]}: example of applying IF-ELSE statements with elements of df.column.</li><li>d43 = d42.append(d41): appends all rows of one DF to another DF.</li></ul><h3>Problem 5: Count the WIN matches only with HTR score lead (only wins that has score lead at half time)</h3><p>There is a HTR field (Half Time Result) that specifies the result at half time, H, D, or A. This problem extends the problem 3 by adding another condition, HTR must be win, on top of the FTR must be win.</p><p><strong>Algorithm:</strong> This problem actually combines problem 2 and 3 together and, thus comprising 2 main parts: filtering and grouping-by. The filtering part is similar to the solutions of the problem 2:</p><ol><li>filter the result twice using the comparison clause (the == clause) just like the solution 1 of the problem 2. Or</li><li>set one of the condition (FTR or HTR) as index then use .loc[] along with the == clause, just like the solution 2 of problem 2.</li></ol><p>The grouping-by part is similar to problem 3. I put the code of the home-team scenario below. I leave the away-team scenario as an exercise for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rKNZcL8uog4qZaqgHPuGOw.png" /><figcaption>Figure 26: Solution of problem 5 using set_index()</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TrRsvhmlbZJohtZxtwiQOA.png" /><figcaption>Figure 27: Solution of problem 5 using the comparison clause twice.</figcaption></figure><p>In the code above I have added some new tricks including</p><ul><li>select the column right within the groupby() statement. Specifically, the groupby() statement in my code looks like this:</li></ul><p>d53 = d52.groupby(&#39;HomeTeam&#39;)[&#39;FTR&#39;].count()</p><p>The d52 has 4 columns before being grouped-by, HomeTeam, AwayTeam, FTR, and HTR. However we want only 1 column after the .count(). To select specific column(s) of the result of the groupby() statement (i.e., d53), you can just simply specify the column names after the groupby() clause. Here I select FTR. If you are not quite sure what I am talking about here, try take out the [&#39;FTR&#39;] from the statement above and observe the result by yourself.</p><ul><li>sort the results of the count() method using .sort_values(). If the resulting DF has more than 1 column, you would need by=&#39;column_name&#39; as another parameter of the .sort_value() , e.g., .sort_values(by=’FTR’, ascending=False).</li></ul><p><strong><em>Pandas Summary</em></strong></p><ul><li>d53 = d52.groupby(&#39;HomeTeam&#39;)[&#39;FTR&#39;].count(): group by HomeTeam and count. The result will have HomeTeam as an index and 1 column named FTR.</li><li>.sort_values(by=’FTR’, ascending=False): sort the input by FTR column from the greatest value to the least. If only there is 1 column, the by= parameter must be removed.</li></ul><h3>Problem 6: Create a Home-Win Table</h3><p>Show a home-win table of all team that looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-GNM5ETQQGb_SOYSM8oGCA.png" /><figcaption>Figure 28: A Home-Win table.</figcaption></figure><p><strong>Algorithm:</strong> This problem add some new tricks to manipulate the groupby() method. Rather than performing sum() or count(), we <em>append</em> or <em>join</em> the data in the group together to create a nice-looking table. Before being grouped-by, we just select HomeTeam and FTR columns, and replace the D and A letters with the dash -.</p><p>Here is the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZCv3TtoF5BLkpW-ffx6bUA.png" /><figcaption>Figure 29: The code for Home-Win table problem.</figcaption></figure><p>From the code, the .replace() simply finds and replaces the text specified in the entire DF (all rows of all columns). So pick the search term carefully or the changes might be done on where they are not intended to.</p><p>The .apply(&#39; &#39;.join) joins the data in the group, FTR of each team in this case, together with a space in between.</p><p><strong><em>Pandas Summary</em></strong></p><ul><li>.replace() : search entire DF for text pattern and replace with the specified text.</li><li>.apply(&#39; &#39;.join) : concatenate the data of the group (of the groupby() function)</li></ul><p>That’s it for this tutorial. Hope it helps. Peace.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1aab90ca368" width="1" height="1" alt=""><hr><p><a href="https://medium.com/luckspark/data-analytics-using-pandas-a-quick-practical-tutorial-1-1aab90ca368">Data Analytics using Pandas: A Quick Practical Tutorial 1</a> was originally published in <a href="https://medium.com/luckspark">LuckSpark</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Tutorial — Using Filter and Count]]></title>
            <link>https://medium.com/luckspark/spark-tutorial-2-using-filter-and-count-63400604f09e?source=rss----d533d9556b8c---4</link>
            <guid isPermaLink="false">https://medium.com/p/63400604f09e</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[apache-spark-tutorial]]></category>
            <dc:creator><![CDATA[LC]]></dc:creator>
            <pubDate>Tue, 20 Mar 2018 08:25:03 GMT</pubDate>
            <atom:updated>2018-03-23T05:25:55.391Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hlev0jaC2RkVLoRWoiHAZg.png" /></figure><p>Since raw data can be very huge, one of the first common things to do when processing raw data is <strong>filtering</strong>. Data that is not relevant to the analysis shall be discarded as much as possible in the first few steps. This would make the code less prone to errors, consumes less resources, and runs faster.</p><p>This tutorial will guide you how to perform basic filtering on your data based on RDD transformations.</p><h3>Objectives</h3><p>What you will learn includes</p><ul><li>How to read data from file to an RDD using .textFile()</li><li>How to sample and print elements of an RDD using .take()</li><li>How to filter data using .filter() with .contains()</li><li>How to use NOT with .contains()</li><li>How to count the number of RDD elements using .count()</li></ul><p>Information regarding Spark setup and environment used in this tutorial are provided on this <a href="https://medium.com/luckspark/installing-spark-2-3-0-on-macos-high-sierra-276a127b8b85">Spark Installation</a> (<a href="https://medium.com/luckspark/%E0%B8%81%E0%B8%B2%E0%B8%A3%E0%B8%95%E0%B8%B4%E0%B8%94%E0%B8%95%E0%B8%B1%E0%B9%89%E0%B8%87-apache-spark-2-3-0-%E0%B8%9A%E0%B8%99-macos-high-sierra-c100e023c6b9">another version in Thai here</a>).</p><p>For absolute beginners, please also read <a href="https://medium.com/luckspark/scala-spark-tutorial-1-hello-world-7e66747faec">Hello World programming on Spark</a> if you have not.</p><p>Ok. Let’s get started.</p><h3>The data</h3><p>The first dataset that I will use throughout a number of my Spark tutorials is the results of football matches, which can be <a href="http://www.football-data.co.uk/englandm.php">found here</a>.</p><p>For this tutorial, let’s take a look at and <strong>download</strong> the <a href="http://www.football-data.co.uk/mmz4281/1617/E0.csv">English Premier League season 2016/2017</a>. The data is a .csv file. It looks like this when being opened in a text editor, Sublime in this case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7nR4SJBNLldwYzkwZjk1hg.png" /><figcaption>Match results of <a href="http://www.football-data.co.uk/mmz4281/1617/E0.csv">English Premier League season 2016/2017</a> (E0.csv), open in a Sublime.</figcaption></figure><ul><li>The data contains a number of rows, 381 to be exact, and each row contains several fields separated by commas.</li><li>The first line contains the information of the header row. It is no the actual data but rather the description of the data. <a href="http://www.football-data.co.uk/notes.txt">The full description can be found here</a>.</li><li>Here I put some descriptions along with the first dataset (the second line in the file)</li></ul><pre>Here are parts of the first data (from the second row)<br>E0,13/08/16,Burnley,Swansea,0,1,A,0,0,D,J Moss,...</pre><pre>Here are descriptions (with data)</pre><pre>Div = League Division (E0)<br>Date = Match Date (dd/mm/yy) (13/08/16)<br>HomeTeam = Home Team (Burnley)<br>AwayTeam = Away Team (Swansea)<br>FTHG and HG = Full Time Home Team Goals (0)<br>FTAG and AG = Full Time Away Team Goals (1)<br>FTR and Res = Full Time Result (H=Home Win, D=Draw, A=Away Win) (A)<br>HTHG = Half Time Home Team Goals (0)<br>HTAG = Half Time Away Team Goals (0)<br>HTR = Half Time Result (H=Home Win, D=Draw, A=Away Win) (D)<br>Referee = Match Referee (J Moss)</pre><p>The file looks more or less like this when you open it from a spreadsheet program.</p><ul><li>Notice that the program automatically put the description line as a header row.</li><li>The program, however, mistakenly separate the first column (Div) apart from the other columns.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CVDXc7gpm5mGfDLF-XybeA.png" /><figcaption>Match results of <a href="http://www.football-data.co.uk/mmz4281/1617/E0.csv">English Premier League season 2016/2017</a> (E0.csv), open in a spreadsheet application.</figcaption></figure><ul><li><strong>I rename the file (from </strong><strong>E0.csv) to </strong><strong>E02016.csv</strong>. Throughout the rest of this article, I will refer to the file using this new name.</li></ul><h3>Directory Structure</h3><p>I name this project as 002_filtering. You can name it anything as you wish. My path for this project is ~/scalaSpark/002_filtering/. I created a separate directory specifically for all raw data to be read by Spark at ~/scalaSpark/input/. Here are the summary of relevant directories:</p><ul><li>project directory : ~/scalaSpark/002_filtering/</li><li>.sbt file : ~/scalaSpark/002_filtering/002filtering.sbt</li><li>.scala file : ~/scalaSpark/002_filtering/src/main/scala/002filtering.scala</li><li>E02016.csv : ~/scalaSpark/input/E02016.csv</li></ul><h3>Setup the .sbt file</h3><p>Here are what you have to put in the sbt file.</p><pre>name := &quot;002filtering&quot;<br>version := &quot;1.0&quot;<br>scalaVersion := &quot;2.11.8&quot;<br>libraryDependencies += &quot;org.apache.spark&quot; % &quot;spark-core_2.11&quot; % &quot;2.3.0&quot;</pre><ul><li>The last line : libraryDependencies += “org.apache.spark” % “spark-core_2.11” % “2.3.0” is Spark dependency.</li><li>In this tutorial we will use only basic RDD functions, thus only spark-core is needed.</li><li>The number 2.11 refers to version of Scala, which is 2.11.x. The number 2.3.0 is Spark version.</li></ul><h3>Write the Scala code</h3><p>Now it is time to write our code to the process the E02016.csv file. We will start from short codes, then run, then add some more codes, then run, repeatedly. I will explain the codes along the way.</p><h3>1. Setup SparkContext</h3><ul><li>Create the 002filtering.scala file and add these lines to it.</li><li>Then save the file.</li></ul><pre>import org.apache.spark.SparkContext<br>import org.apache.spark.SparkConf<br>object filtering002<br>{<br>    def main(args: Array[String])<br>    {<br>        println(&quot;\n\n&gt;&gt;&gt;&gt;&gt; START OF PROGRAM &lt;&lt;&lt;&lt;&lt;\n\n&quot;);</pre><pre>        val conf = new SparkConf().setAppName(&quot;002filtering&quot;)<br>        val sc = new SparkContext(conf);</pre><pre>        println(&quot;\n\n&gt;&gt;&gt;&gt;&gt; END OF PROGRAM &lt;&lt;&lt;&lt;&lt;\n\n&quot;);<br>    }<br>}</pre><ul><li>In line 3 I named the object as filtering002. This is just because the object name cannot start with numbers.</li><li>The code begins by importing SparkContext and SparkConf. SparkConf is basically the configuration parameters of SparkContext.</li><li>SparkContext is the class for the Spark world, where your codes are managed and run. You can refer to the Spark’s <a href="https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.package">main API page</a> or <a href="https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkContext">SparkContext API page</a> for more information.</li><li>The val conf = new SparkConf() line creates the configuration for SparkContext. In this case, only 1 parameter, AppName, is defined.</li><li>The val sc = new SparkContext(conf) line creates a SparkContext object using the SparkConf defined.</li><li><strong>val</strong> is a Scala reserve word use to declare a new variable. The variables declared with <strong>val</strong> is <strong>immutable</strong>. <em>An immutable variable is a variable that its value cannot be changed later</em>. This similar to a constant in some other languages. It is common to use immutable variables in Spark. If we need to alter an immutable variable, we commonly create a new modified immutable one rather than directly modify the existing one. You will gradually learn why this is Spark’s way of coding as you proceed.</li><li>The two println lines just print out strings. These are what we expect to see on the output after running the code.</li></ul><h4>Run</h4><p>Although we have not done anything much here, but let’s run to code to make sure things go well so far.</p><pre>cd ~/scalaSpark/002_filtering       # goto project home<br>sbt package                         # compile the project</pre><pre>### Wait for the compilation.<br>### The compilation should end with &quot;Success&quot;</pre><pre>### Then run the spark-submit command below</pre><pre>spark-submit ./target/scala-2.11/002filtering_2.11-1.0.jar</pre><ul><li>Output of sbt package</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JXOrp-5CbowJThy9FMOTMg.png" /><figcaption>The beginning output lines from sbt package command.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hWtuh391jRStVahEhQ1jUA.png" /><figcaption>Then finishing output lines from sbt package command.</figcaption></figure><ul><li>Output of spark-submit</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*379ZMgJAK2oADkjsiQzKOA.png" /><figcaption>The beginning output lines from spark-submit command.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zQADHmRcwaCON4k9PCnrow.png" /><figcaption>Then finishing output lines from spark-submit command.</figcaption></figure><ul><li>From the spark-submit command, you will see lines of output.</li><li>You should clearly see the <em>START OF PROGRAM</em> and <em>END OF PROGRAM</em> lines in the output. This indicates that you have done well so far.</li></ul><p>Now, let’s read the E02016.csv file into Spark and do some interesting things to it.</p><h3>2. Read .csv file into Spark</h3><ul><li>Spark allows you to read several file formats, e.g., text, csv, xls, and turn it in into an RDD. We then apply series of operations, such as filters, count, or merge, on RDDs to obtain the final outcome.</li><li>To read the file E02016.csv into an RDD, add this line to the scala file.</li></ul><pre>val logfile = &quot;./../input/E02016.csv&quot;<br>//OR val logfile = &quot;/Users/luckspark/scalaSpark/input/E02016.csv&quot;</pre><pre>val logrdd = sc.textFile(logfile)</pre><ul><li>Note that I use the relative path “./../input/E02016.csv&quot; to refer to the data file. <strong>This is the path relative to the location that you run the spark-submit command</strong>. For example, I commonly execute the spark-submit command at the project directory, which is ~/scalaSpark/002_filtering in this case. That is, my location is ~/scalaSpark/002_filtering and the file location is ~/scalaSpark/input/E02016.csv. Therefore, the relative path from my location to the file is ./../input/E2016.csv. If I run the spark-submit command at any other location, file does not exist error would be thrown at runtime.</li><li>Alternatively, you can use an absolute path instead, as I put in the //OR comment line.</li><li>Next, The sc.textFile() reads the logfile and convert it into an RDD named logrdd.</li></ul><h3>3. Print elements of an RDD</h3><ul><li>One of the main reasons for printing out RDD elements is to see whether the data format or pattern are as what we expected. The raw data is generally huge and it is rare to print ALL elements of an RDD.</li><li>Rather than printing out all elements, using <strong>.take()</strong> to print out RDD elements is a good idea. It picks samples of elements for us. Every time the code is executed, samples taken are generally the same. This is, however, <em>not always</em>, both the data itself or the order of the data could be different.</li><li>Now that we have read all lines from E02016.csv file onto the logrdd RDD.</li><li>Let’s print the logrdd and see if the data is correctly read using a combination of take, foreach, and println, like this</li></ul><pre>logrdd.take(10).foreach(println)</pre><ul><li>The dot (.) is how you connect the methods together. Methods are processed from left to right. The output of the method on the left becomes an input of the next method on the right. Here the output of .take(10) becomes the input to the .foreach(println) method.</li><li>.take(10) take 10 sample rows from the logrdd RDD and repeatedly print each of them out.</li><li>Now, let’s recap and run the code. Below is the entire code with the new lines added.</li></ul><pre>import org.apache.spark.SparkContext<br>import org.apache.spark.SparkConf<br>object filtering002<br>{<br>    def main(args: Array[String])<br>    {<br>        println(&quot;\n\n&gt;&gt;&gt;&gt;&gt; START OF PROGRAM &lt;&lt;&lt;&lt;&lt;\n\n&quot;)</pre><pre>        val conf = new SparkConf().setAppName(&quot;002filtering&quot;)<br>        val sc = new SparkContext(conf)</pre><pre>        val logfile = &quot;./../input/E02016.csv&quot;<br>        val logrdd = sc.textFile(logfile)</pre><pre>        logrdd.take(10).foreach(println)</pre><pre>        println(&quot;\n\n&gt;&gt;&gt;&gt;&gt; END OF PROGRAM &lt;&lt;&lt;&lt;&lt;\n\n&quot;)<br>    }<br>}</pre><ul><li>Re-compile and run the code using the same commands: sbt package and then spark-submit ./target/scala-2.11/002filtering_2.11–1.0.jar</li><li>The output is shown below.</li><li>At the very end, above the END OF PROGRAM string, you shall see the contents of the E02016.csv displayed.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GRd8-bCLZT5nlG0rADiBBQ.png" /><figcaption>Read E02016.csv and print 10 sample rows.</figcaption></figure><ul><li>Let’s compare it with the data displayed in text editor.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7nR4SJBNLldwYzkwZjk1hg.png" /><figcaption>E02016.csv opened using Sublime.</figcaption></figure><h3>4. Data selection by row using <strong>.filter( )</strong></h3><p>Now, as you can see, there are too much data. Let’s filter something out to make things more meaningful.</p><p>First, let’s filter out the header row. This is obvious since it is mixed with the data and would definitely cause errors to analysis outcomes. For example, if we wish to count the total number of matches played in the season, since the data is one match per line, simply counting the number of lines would give us the answer. This would, however, mistakenly include the header line in the counting, giving out the answer of 381 rather than 380 matches per season. This is clearly undesirable. Thus, let’s discard the header row now.</p><ul><li>You can select to filter out rows by criteria using .filter()</li><li>Let’s try this code.</li></ul><pre>val f1 = logrdd.filter(s =&gt; s.contains(&quot;E0&quot;))<br>f1.take(10).foreach(println)</pre><ul><li>In human language, the val f1 = logrdd.filter(s =&gt; s.contains(“E0”)) would read, “copy every element of logrdd RDD that contains a string “E0” as new elements in a new RDD named f1”.</li><li>The second line simply print samples of the new f1 RDD (that only contains E0 in the line).</li><li>In details, the code first define a new RDD named f1.</li><li>Then it reads all elements of the logrdd RDD and applies the criterion of the filter to it.</li><li>The <strong>s</strong> in s =&gt; s.contains() represents <strong>each element of the logrdd</strong> RDD.</li><li>You can replace boths with any other letters or strings you wish. The most commonly seen in Spark world is line, i.e., .filter(line =&gt; line.contains(“E0”).</li><li>Algorithmically, I choose the string “E0” because the header line is the only line without such string. Try it yourself with other strings.</li></ul><p>Let’s recap and run the code.</p><pre>import org.apache.spark.SparkContext<br>import org.apache.spark.SparkConf<br>object filtering002<br>{<br>    def main(args: Array[String])<br>    {<br>        println(&quot;\n\n&gt;&gt;&gt;&gt;&gt; START OF PROGRAM &lt;&lt;&lt;&lt;&lt;\n\n&quot;)</pre><pre>        val conf = new SparkConf().setAppName(&quot;002filtering&quot;)<br>        val sc = new SparkContext(conf)</pre><pre>        val logfile = &quot;./../input/E02016.csv&quot;<br>        val logrdd = sc.textFile(logfile)</pre><pre>        logrdd.take(10).foreach(println)</pre><pre>        val f1 = logrdd.filter(s =&gt; s.contains(&quot;E0&quot;))<br>        f1.take(10).foreach(println)</pre><pre>        println(&quot;\n\n&gt;&gt;&gt;&gt;&gt; END OF PROGRAM &lt;&lt;&lt;&lt;&lt;\n\n&quot;)<br>    }<br>}</pre><ul><li>From the output, you will notice 2 sets of data printed out, one from logrdd and the other one from f1. Notice f1 does not contain the header line any more.</li></ul><h3>5. Using Not with .contains( )</h3><p>In the previous topic we have covered the .filter(s =&gt; s..contains()) in which we choose to keep every lines containing “E0”. How about if we wish to keep lines that “<em>do not</em> contain something”?</p><p>To do this, we add the “NOT”, which is !, into the statement. It is a bit tricky where to put the ! sign, thus I think it is worth mentioning it here.</p><ul><li>Below is a statement to keep lines that do not contain a string “FTHG”.</li></ul><pre>val f2 = logrdd.filter(s =&gt; !(s.contains(&quot;FTHG&quot;)))</pre><ul><li>Notice the position of the ! and next time you can use it correctly.</li><li>Try this line of code and you shall get the same result as in the previous topic.</li></ul><h3>6. Count the number of elements in an RDD</h3><p>From the previous section, we discarded the header line and there were no header lined printed out from the f1.take(10).foreach(println)line. However, as I explained earlier that the take(10) just takes some samples of RDD elements, is it possible that perhaps the header line is still there but just had not been sampled? Let’s count the lines to help verifying this.</p><ul><li>.count() method is Spark’s <em>action</em>. It counts the number of elements of an RDD.</li><li>It returns a Long integer. Therefore, we can simply print it out.</li></ul><pre>println(logrdd.count() + &quot; &quot; + f1.count())</pre><ul><li>Here I print the count of logrdd RDD first, add a space, then follow by the count of f1 RDD.</li><li>The entire code is shown again here (with just 1 line added from the previous one).</li></ul><pre>import org.apache.spark.SparkContext<br>import org.apache.spark.SparkConf<br>object filtering002<br>{<br>    def main(args: Array[String])<br>    {<br>        println(&quot;\n\n&gt;&gt;&gt;&gt;&gt; START OF PROGRAM &lt;&lt;&lt;&lt;&lt;\n\n&quot;)</pre><pre>        val conf = new SparkConf().setAppName(&quot;002filtering&quot;)<br>        val sc = new SparkContext(conf)</pre><pre>        val logfile = &quot;./../input/E02016.csv&quot;<br>        val logrdd = sc.textFile(logfile)</pre><pre>        logrdd.take(10).foreach(println)</pre><pre>        val f1 = logrdd.filter(s =&gt; s.contains(&quot;E0&quot;))<br>        f1.take(10).foreach(println)</pre><pre>        println(logrdd.count() + &quot; &quot; + f1.count())</pre><pre>println(&quot;\n\n&gt;&gt;&gt;&gt;&gt; END OF PROGRAM &lt;&lt;&lt;&lt;&lt;\n\n&quot;)<br>    }<br>}</pre><ul><li>After you run it, you shall see the “381 380” line, just above the END OF PROGRAM string, as shown below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*LLCIUuwkZDE7vikVhY41Mg.png" /><figcaption>Output of the .count() method</figcaption></figure><p>There you go, now you know that 1 line of data is discarded (presumably, the header line).</p><h3>7. Print all elements of an RDD</h3><p>For some, counting the number of lines of the f1 RDD may be not enough to prove that the header line has really been removed. How about printing out <em>all</em> elements of the f1 RDD?</p><ul><li>Printing out all elements of an RDD is a bit tricky.</li><li>At this stage, you can look at an RDD as it is an Array, with one important exception — <strong><em>it is distributed</em></strong>. Spark is designed to be run on a large number of machines where data are divided and distributed among them. The data could even be divided into several partitions in one machine. For example, if you have 100 rows of data, perhaps the first 10 are given to the first machine, the next 10 are given to the other machine, and so on.</li><li>When printing out, spark use machines’ stdout. Differeent machien means different stdout.</li><li>In local mode, it is ok to print out using</li><li>f1.foreach(println) OR f1.take(f1.count().toInt).foreach(println)</li><li>take() expects an Int integer as a parameter but count() returns a Long integer. Thus, the .toInt method is needed here.</li><li>If you wish to print the exact element of the RDD, e.g, something like printing out the value at the index <em>i</em> of an Array, you have to convert the RDD to a local array using .collect() method. The array returned will be stored on one machine (the driver machine), meaning that it is not a distributed dataset any more. This .collect() could cause errors if the local machine has insufficient amount of memory to hold the entire set of data.</li><li>In cluster mode where the driver and the executor machines are different, outputs of the print command will be displayed on each machine’s stdout. Therefore, you will not see all outputs at the driver node. That is, if you try to print all elements of an RDD, you will not see all outputs from the driver machine. Using collect() fixes the problem if you have sufficient memory at the driver node.</li></ul><h3>8. Try it yourself</h3><p>Let’s recap what you have learnt:</p><ul><li>read from file to RDD using .textFile()</li><li>sample and print elements of RDD using .take()</li><li>filter data using .filter() and .contains()</li><li>using NOT with .contains()</li><li>count the number of RDD elements using .count()</li></ul><p>Here are some interesting things you are able to do using methods you have learnt. Try it out yourself.</p><ul><li>Print and count only the Liverpool matches (i.e., 20 matches).</li><li>Print and count only the Liverpool vs Chelsea matches (i.e., only 2 matches).</li><li>Print and count only matches that Liverpool won.</li><li>Print only matches played in November.</li><li>Count the number of matches that Liverpool lost before the new year (of that season)</li></ul><h3>That’s all for this tutorial. Peace.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63400604f09e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/luckspark/spark-tutorial-2-using-filter-and-count-63400604f09e">Spark Tutorial — Using Filter and Count</a> was originally published in <a href="https://medium.com/luckspark">LuckSpark</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Tutorial — Hello World]]></title>
            <link>https://medium.com/luckspark/scala-spark-tutorial-1-hello-world-7e66747faec?source=rss----d533d9556b8c---4</link>
            <guid isPermaLink="false">https://medium.com/p/7e66747faec</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[LC]]></dc:creator>
            <pubDate>Mon, 12 Mar 2018 03:26:59 GMT</pubDate>
            <atom:updated>2018-03-21T01:51:51.556Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*irw5AJERr9qOdbcR20cSQA.png" /></figure><p>This tutorial will guide you to write the first Apache Spark program using Scala script, a self-contained program, and not an interactive one through the Spark shell.</p><p>Main objective is to jump-start your first Scala code on Spark platform with a very shot and simple code, i.e., the real “Hello World”. The focus is to get the reader through a complete cycle of setup, coding, compile, and run fairly quickly. The code does not even use any fancy function of Spark at all.</p><h3>Objectives</h3><p>What you will learn includes</p><ul><li>To create directory structure of Scala Spark program</li><li>To setup .sbt file</li><li>To setup and write some code in .scala file</li><li>Code — To print strings using print and println</li><li>To compile and run the Scala code on Spark platform.</li></ul><p>Let’s get started.</p><h3>0. Tutorial Environment</h3><p>The Apache Spark 2.3.0 used in this tutorial is installed based on tools and steps <a href="https://medium.com/luckspark/installing-spark-2-3-0-on-macos-high-sierra-276a127b8b85">explained in this tutorial</a>. I summarize my Spark-related system information again here.</p><ul><li>macOS High Sierra 10.13.3. I guess that the older macOS version like 10.12 or 10.11 shall be fine. This tutorial can certainly be use as guideline for other Linux-based OS too (of course with some differences in commands and environments)</li><li>Apache Spark 2.3.0, JDK 8u162, Scala 2.11.12, Sbt 0.13.17, Python 3.6.4</li></ul><p>The directory and path related to Spark installation are based on <a href="https://medium.com/luckspark/installing-spark-2-3-0-on-macos-high-sierra-276a127b8b85">this installation tutorial</a> and remain intact. Just make sure that you can run pyspark or spark-shell from your Home directory, so that we could compile and run our code in this tutorial.</p><h3>1. Essential Files</h3><p>There are 2 files that you have to write in order to run a Scala Spark program:</p><ol><li>A .sbt file, which is a configuration file, similar to an include file in C.</li><li>A .scala file. This is where your Scala code resides.</li></ol><p>These files, however, must be put in a certain directory structure explained in the next section.</p><h3>2. Directory Structure</h3><ul><li>First, you have to create your project’s directory, in this case named <strong><em>hello</em></strong>.</li><li>Right inside the project directory is where you put your .sbt file.</li><li>For the .scala file, you have to create directories src/main/scala inside the project directory. Then you put your .scala file there.</li><li>The figure below shows the files and directory structure.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*lObc6UkOyfdX14z4WLXHsw.png" /><figcaption>Directory structure of Scala Spark project. The project directory is “hello”. There must be a .sbt file inside the project directory. There must be .scala file under src/main/scala inside the project directory.</figcaption></figure><ul><li>Let me fast forward you to the directory structure <em>after</em> the Scala code is compiled, there will be 2 new directories created, target and project ,as shown in the figure below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*O9vqEU2_UtgIR75fO4NaYQ.png" /><figcaption>The new “target” and “project” directories after the compilation.</figcaption></figure><ul><li>The compiled file, .jar, used to run the project is under the target directory.</li></ul><h3>3. Create directory structure</h3><p>Now let’s create the directory structure discussed above using command line on Terminal.</p><p>Please note that I will create a directory named ‘scalaSpark’ under my Home directory. This directory will contain all Scala-based Spark project in the future. Thus, in this tutorial the main project named hello, is located at /Users/luckspark/scalaSpark/hello/ or ~/scalaSpark/hello/. You can pick any other location (path) as you wish and modify the path accordingly.</p><p>Here are step-by-step procedures.</p><ul><li>Open the Terminal program.</li><li>Make sure that you are at your Home by entering the command cd</li><li>Create a directory named scalaSpark under Home directory, and create a directory named <strong><em>hello</em></strong> under the scalaSpark directory, using mkdir -p scalaSpark/hello command.</li><li>Get inside the <em>hello</em> directory by typing cd saclaSpark/hello.</li><li>Create the src/main/scala directory inside the hello directory by typing mkdir -p ./src/main/scala</li></ul><p>Here is a summary of all commands above.</p><pre>cd                                    #change directory to HOME<br>mkdir -p ./scalaSpark/hello           #create new directories<br>cd scalaSpark/hello                   #change directory into hello<br>mkdir -p ./src/main/scala             #create new directories</pre><pre># Note that text after # is treated as comments, so it won&#39;t be run.<br># Thus, you can copy all lines including the comments and paste them on the Terminal shell prompt.</pre><p>This is how it looks like when copy and paste the lines above onto the Terminal app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v_obDWA0v6wbEUXrteKjrA.png" /><figcaption>Copy and Paste lines onto the Terminal.</figcaption></figure><p>On the Finder, the new directories shall appear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*jBYbnlRW45qembbCkKFqew.png" /></figure><h3>4. Create SBT Configuration File</h3><p>Now it is time to setup the Sbt configuration file. The location of this file is right under the project’s directory. In this case, it’s ~/scalaSpark/hello.</p><ul><li>You can create the hello.sbt using</li></ul><pre>cd ~/scalaSpark/hello            # change directory <br>touch hello.sbt                  # create hello.sbt file<br>open -a TextEdit hello.sbt       # open hello.sbt using TextEdit app</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oZ6frNdPmVPdbBp_wM61eg.png" /><figcaption>Open hello.sbt using TextEdit.</figcaption></figure><ul><li>Add these lines to hello.sbt</li></ul><pre>name := &quot;hello&quot;<br>version := &quot;1.0&quot;<br>scalaVersion := &quot;2.11.8&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S4GUzTlGGjTPaNWEPP2HQQ.png" /><figcaption>~/scalaSpark/hello/hello.sbt</figcaption></figure><ul><li>Save the file.</li></ul><p>Before we proceed, let’s explain the configuration in more detail.</p><ul><li>In this file, you simply put <strong>keys</strong> and <strong>values</strong> with := in the middle, which makes the configurations pretty much self-explained.</li><li>In the first line, the name hello, <strong>will reflect the name of the .jar file created after the compilation</strong>. For example, if you change the variable name from hello to hello1234, the jar file will be named hello1234_2.11–1.0.jar rather than hello_2.11–1.0.jar.</li></ul><h3>5. Create Scala File</h3><p>Now let’s create your Spark’s source code. This file is at ~/scalaSpark/hello/src/main/scala. The name is hello.scala.</p><ul><li>Create <strong>hello.scala</strong> using these command lines</li></ul><pre>cd ~/scalaSpark/hello/src/main/scala    # change directory<br>touch hello.scala                       # create hello.scala<br>open -a TextEdit hello.scala         # open hello.scala for editing</pre><ul><li>Add these lines to hello.scala</li></ul><pre>object hello<br>{<br>    def main(args: Array[String])<br>    {<br>        print(&quot;\n\n&gt;&gt;&gt;&gt;&gt; START OF PROGRAM &lt;&lt;&lt;&lt;&lt;\n\n&quot;);<br>        <br>        println(&quot;Hello World.&quot;)<br>        <br>        print(&quot;\n\n&gt;&gt;&gt;&gt;&gt; END OF PROGRAM &lt;&lt;&lt;&lt;&lt;\n\n&quot;);<br>    }<br>}</pre><ul><li>Save the file. It shall look like this.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EpZATKpkbiCbW8CKjYa7Qg.png" /><figcaption>~/scalaSpark/hello/src/main/scala/hello.scala</figcaption></figure><p>Let’s examine the code</p><ul><li>This code defines scala object hello, which has only one method, main.</li><li>It does not use any fancy feature of Spark at all.</li><li>It just prints out 3 messages, using print and println.</li><li>The semicolon at the end of the line is optional. If your finger is so familiar to typing it at the end of the line, just do it. Otherwise, you can ignore it.</li></ul><p>That’s it. Let’s compile and run the code.</p><h3>6. Compile and Run</h3><ul><li>To compile and run the project, you have to change directory back to the root of the project, which is ~/scalaSpark/hello in this case.</li></ul><h4>6. 1. Compile</h4><ul><li>Then you can compile the code using sbt package command.</li></ul><pre>cd ~/scalaSpark/hello   # change directory back project root<br>sbt package             # compile the code</pre><ul><li>It might take some time to compile as it has to download some dependencies.</li><li>There could also be some warnings.</li><li>It, however, should end with [success] message, as shown below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UDd8CJw2OA15zXOUv_hE-Q.png" /><figcaption>sbt package</figcaption></figure><h4>6.2. Run</h4><ul><li>After compilation, a number of new files will be created under new directories named project and target.</li><li>Among these new files, we will use the jar file under the target/scala-2.11 directory to run the code. The file is named after project’s name, Scala version, and code’s version. In this case, hello_2.11–1.0.jar.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*THq_-s3Uwg3sLqBE61225Q.png" /></figure><ul><li>Run the code using this command</li></ul><pre>spark-submit ./target/scala-2.11/hello_2.11-1.0.jar</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cDxuHOTdec_rcNGNMy6h7w.png" /></figure><ul><li>As expected, you shall see 3 lines of strings in the code.</li><li>There might be some warning, but that is fine.</li></ul><h3>That’s it. Enjoy your coding. — Peace.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e66747faec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/luckspark/scala-spark-tutorial-1-hello-world-7e66747faec">Spark Tutorial — Hello World</a> was originally published in <a href="https://medium.com/luckspark">LuckSpark</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up Spark 2.0.1 and Zeppelin 0.6.2 on macOS Sierra]]></title>
            <link>https://medium.com/luckspark/setting-up-spark-2-0-1-and-zeppelin-0-6-2-on-macos-sierra-b163db9848f3?source=rss----d533d9556b8c---4</link>
            <guid isPermaLink="false">https://medium.com/p/b163db9848f3</guid>
            <category><![CDATA[apache-zeppelin]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[macos-sierra]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[LC]]></dc:creator>
            <pubDate>Thu, 08 Mar 2018 02:46:46 GMT</pubDate>
            <atom:updated>2017-02-06T05:39:53.819Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VpLJD94amSLSRWw5r9ihNg.png" /></figure><p>At the time of writing, Zeppelin 0.6.2 is compatible with Spark 2.0.1.</p><p>Spark 2.1.0 is already released <em>but is not yet compatible with Zeppelin 0.6.2. </em>There are still errors when trying to use zeppelin with spark 2.1.0.</p><h3>Step 1: Download all Required Softwares</h3><ol><li><a href="http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html">Java JDK</a></li><li><a href="https://www.python.org/ftp/python/3.5.3/python-3.5.3-macosx10.6.pkg">Python 3.5.x</a></li><li><a href="https://dl.bintray.com/sbt/native-packages/sbt/0.13.13/sbt-0.13.13.tgz">SBT 0.13.x</a></li><li><a href="http://d3kbcqa49mib13.cloudfront.net/spark-2.0.1-bin-hadoop2.7.tgz">Spark 2.0.1</a></li><li><a href="http://www-us.apache.org/dist/zeppelin/zeppelin-0.6.2/zeppelin-0.6.2-bin-all.tgz">Zeppelin 0.6.2</a></li></ol><h3>Note on the HOME directory of this tutorial</h3><ul><li><em>Home directory</em> of this tutorial is<strong> /Users/myaccount. </strong>This directory will be referred to as <strong>HOME</strong> or <strong>~</strong> in this tutorial. (The <strong>~</strong> also means HOME in the shell (terminal))</li></ul><blockquote>In this tutorial<br>/Users/myaccount/Downloads = HOME/Downloads = ~/Downloads</blockquote><ul><li>It is import that you adjust your HOME directory according to your own HOME path for all of the following configurations.</li><li>For example, you might have to change from /Users/myaccount/Downloads to /Users/tim/Downloads.</li></ul><h3>Step 2: Extract the downloaded files</h3><ul><li>Assumed that all files are downloaded at ~/Downloads directory of your mac (i.e., /Users/myaccount/Downloads), as shown below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*B52V5jmw7BDrXEcgvufkNw.png" /><figcaption>The downloaded files saved at HOME/Downloads directory.</figcaption></figure><ul><li>Extract the .tgz files (sbt*.tgz, spark*.tgz, zeppelin*.tgz) by double click the file, which will launch the Archive Utility program and extract the files automatically.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*7OZLuN4WYMU0_OnqZkFkqQ.png" /><figcaption>Files extraction by Archive Utility</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*hI6E8h4NKpYKxKYQuaV2hQ.png" /><figcaption>sbt, spark, and zeppelin directories after the extraction.</figcaption></figure><h3>Step 3: Install JDK</h3><ul><li>Double click the jdk-8u121-macosx-x64.dmg file launch the installation process.</li><li>Double click the JDK8 Update 121.pkg icon to install.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*TjCJIXmaYc06xchasa0YYg.png" /><figcaption>Double click the box icon to start the installation.</figcaption></figure><ul><li>Follow the screen to finish the installation.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TeHAMXCXxnJiE5mgnGu7RQ.png" /><figcaption>Follow the screen to finish the installation.</figcaption></figure><h3>Step 4: Install Python 3</h3><ul><li>Double click the python-3.5.3-macosx10.6.pkg file to start Python 3 installation.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rOK1a8y6T-rTjMnI9vv-3A.png" /><figcaption>Follow the screen to finish the installation.</figcaption></figure><h3>Step 5: Configure Zeppelin</h3><ul><li>start the “Terminal” program, which is located at /Applications/Utilities.</li><li>go to <strong><em>conf</em></strong> directory of the zeppelin at <em>HOME/Downloads/zeppelin-0.6.2-bin-all/conf</em></li><li>in the <em>conf</em> directory, you shall see a file named <strong><em>zeppelin-env.sh.template</em></strong></li><li><strong>copy</strong> this file to a new name called <strong><em>zeppelin-env.sh</em></strong></li></ul><blockquote>cd Downloads/zeppelin-0.6.2-bin-all/conf<br>cp zeppelin-env.sh.template zeppelin-env.sh</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vyUzfpJSShqONthlGH3fuQ.png" /><figcaption>Go to the conf directory of the zeppelin. Then copy the zeppelin-env.sh.template to zeppelin-env.sh.</figcaption></figure><ul><li>edit the zeppelin-env.sh file by adding this line to the very top of the file.</li></ul><blockquote>export SPARK_HOME=/Users/myaccount/Downloads/spark-2.0.1-bin-hadoop2.7</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TIPpKpuExU38JB_ZWTQ0Qg.png" /><figcaption>Declare the SPARK_HOME environment for zeppelin</figcaption></figure><ul><li>Save and close the zeppelin-env.sh file.</li></ul><h3>Step 6: Setup shell environment by editing the ~/.bash_profile file</h3><ul><li>Using the terminal, change directory back to HOME</li><li>Open the ~/.bash_profile file using any text editor (e.g., nano, vi, or sublime)</li><li>Add these lines to the file</li></ul><blockquote>export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/<br>export SPARK_HOME=/Users/myaccount/Downloads/spark-2.0.1-bin-hadoop2.7<br>export SBT_HOME=/Users/myaccount/Downloads/sbt-launcher-packaging-0.13.13<br>export ZEPPELIN_HOME=/Users/myaccount/Downloads/zeppelin-0.6.2-bin-all</blockquote><blockquote>export PYSPARK_PYTHON=python3</blockquote><blockquote>export PATH=$JAVA_HOME/bin:$SPARK_HOME:$SPARK_HOME/bin:$SPARK_HOME/sbin:$SBT_HOME/bin:$ZEPPELIN_HOME/bin:$PATH</blockquote><blockquote>PATH=”/Library/Frameworks/Python.framework/Versions/3.5/bin:${PATH}”<br>export PATH</blockquote><ul><li>Your ~/.bash_profile shall look like this.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1KXwKMSbrALMO6AbpVhJMw.png" /></figure><ul><li>Save and close the file.</li><li>Quit the Terminal program. Make sure you completely quit the terminal otherwise the environment variables declared above will not be loaded.</li></ul><h3>Step 7: Test Spark</h3><ul><li>Open the Terminal.</li></ul><h4>Step 7.1: Test pyspark</h4><ul><li>Launch <strong><em>pyspark</em></strong> (Python of Spark)</li><li>Note 1: with the PATH setting in step 6, you shall be able to call pyspark from anywhere in the system.</li><li>Note 2: notice the first output line, you should get the Python 3.5.3 (not the 2.x.x).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N9TAvx4XVeELItfeWtodoQ.png" /><figcaption>pyspark interactive shell shall be started with Python 3.5.3 engine.</figcaption></figure><ul><li>type ctrl-d to exit</li></ul><h4>Step 7.2: Test spark-shell</h4><ul><li>On the terminal, launch <strong><em>spark-shell</em></strong> (Scala of Spark)</li><li>Note: you shall be able to invoke spark-shell from anywhere too.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oNFPBfxFKNEe-85wTAPAqg.png" /><figcaption>Scala-based interactive spark shell can be invoked using spark-shell command.</figcaption></figure><h3>Step 8: Test Zeppelin</h3><ul><li>In the Terminal, start the zeppelin daemon using this command</li></ul><blockquote>zeppelin-daemon.sh start</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uwuzhMrDsDwhXuM9a_2fsg.png" /></figure><ul><li>Open web browser (e.g., Safari)</li><li>Go to localhost:8080</li><li>Zeppelin home page shall appear as shown below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s450L_qWLN1Qx93VP1Sbgg.png" /></figure><ul><li>to stop the zeppelin, simply use <em>zeppelin-daemon.sh stop</em> command.</li></ul><h3>That’s it. Enjoy your Spark and Zeppelin.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b163db9848f3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/luckspark/setting-up-spark-2-0-1-and-zeppelin-0-6-2-on-macos-sierra-b163db9848f3">Setting up Spark 2.0.1 and Zeppelin 0.6.2 on macOS Sierra</a> was originally published in <a href="https://medium.com/luckspark">LuckSpark</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup password-less SSH on Ubuntu 16.04]]></title>
            <link>https://medium.com/luckspark/setup-passwordless-ssh-on-ubuntu-16-04-7ac81592fee6?source=rss----d533d9556b8c---4</link>
            <guid isPermaLink="false">https://medium.com/p/7ac81592fee6</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[ssh]]></category>
            <category><![CDATA[ubuntu]]></category>
            <dc:creator><![CDATA[LC]]></dc:creator>
            <pubDate>Thu, 08 Mar 2018 02:45:41 GMT</pubDate>
            <atom:updated>2016-07-04T06:24:10.898Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rJozCcQj9AsenzuLdvobPw.jpeg" /></figure><h4>Setup</h4><pre>sudo apt-get update<br>sudo apt-get install openssh-server</pre><pre>ssh-keygen -t rsa -P &quot;&quot;<br>cat ./.ssh/id_rsa.pub &gt;&gt; ./.ssh/authorized_keys<br>chmod 600 ~/.ssh/authorized_keys</pre><pre>## THEN ##<br>sudo service ssh start<br>-- OR --<br>sudo service ssh restart<br>-- OR --<br>sudo reboot</pre><h4>Test SSH login</h4><pre>## ssh from the server host (to itself)<br>ssh localhost<br>-- OR --<br>## ssh from the server or the client host<br>ssh 10.1.1.1   // 10.1.1.1 is the ip address of the server</pre><h4>Passwordless SSH from the client</h4><p>Concept: <strong>append</strong> <em>client’s public key (content of .ssh/id_rsa.pub)</em> <strong>TO</strong> <em>server’s .ssh/authorized_keys</em> <strong>file</strong></p><pre>### 1. use ssh-copy-id ###<br>## at CLIENT machine<br>** ubuntu only, mac does not have ssh-copy-id<br>** -- format --&gt; ssh-copy-id user@server_IP</pre><pre>ssh-copy-id luck@10.1.37.12</pre><pre>### Then TEST SSH login ###</pre><pre>-- OR --</pre><pre>### 2. manual append ###<br>** manually copy the public key content to server&#39;s auth file ** <br>** both mac and linux</pre><pre>## at CLIENT machine<br>cd<br>cat .ssh/id_rsa.pub<br>&gt;&gt; copy the key (the long text)</pre><pre>## at SERVER<br>cd<br>vi .ssh/authorized_keys<br>&gt;&gt; paste the client’s key at the end of the file</pre><pre>### Then TEST SSH login ###</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ac81592fee6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/luckspark/setup-passwordless-ssh-on-ubuntu-16-04-7ac81592fee6">Setup password-less SSH on Ubuntu 16.04</a> was originally published in <a href="https://medium.com/luckspark">LuckSpark</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[การติดตั้ง Apache Spark 2.3.0 บน macOS High Sierra]]></title>
            <link>https://medium.com/luckspark/%E0%B8%81%E0%B8%B2%E0%B8%A3%E0%B8%95%E0%B8%B4%E0%B8%94%E0%B8%95%E0%B8%B1%E0%B9%89%E0%B8%87-apache-spark-2-3-0-%E0%B8%9A%E0%B8%99-macos-high-sierra-c100e023c6b9?source=rss----d533d9556b8c---4</link>
            <guid isPermaLink="false">https://medium.com/p/c100e023c6b9</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[macos]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[LC]]></dc:creator>
            <pubDate>Thu, 08 Mar 2018 02:23:54 GMT</pubDate>
            <atom:updated>2018-03-08T02:42:40.718Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*99lgzUPQOsSKbx3xNHKo-g.png" /></figure><blockquote>Apache Spark ปล่อยเวอร์ชั่น 2.3.0 ออกมาเมื่อวันที่ 28 ก.พ. 2018. บทความนี้จะแนะนำวิธีติดตั้ง Apache Spark 2.3.0 บน macOS High Sierra</blockquote><p><a href="https://medium.com/luckspark/installing-spark-2-3-0-on-macos-high-sierra-276a127b8b85">Read English version here</a></p><h4>Step 1: ดาวน์โหลดโปรแกรมต่าง ๆ</h4><p>ใน<a href="https://spark.apache.org/docs/latest/">เว็บหลักของ Spark</a> ระบุว่าเวอร์ชั่น 2.3.0 นี้จะทำงานร่วมกับโปรแกรมอื่น ๆ ในเวอร์ชั่นต่าง ๆ ดังต่อไปนี้ครับ</p><ul><li>Java 8 ขึ้นไป</li><li>Python 2.7 ขึ้นไป หรือ 3.4 ขึ้นไป — ภาษาไพธอนมี 2 เวอร์ชั่นหลักคือ 2.x และ 3.x ซึ่งเวลาที่เราใช้งานต้องเลือกอย่างใดอย่างหนึ่ง ในการใช้งาน Spark นั้น สำหรับผู้เร่ิมต้นหรือไม่มีข้อจำกัดอะไร แนะนำให้ใช้ 3.x ครับ กรณีที่จะเลือกใช้ 2.x ก็มักจะเป็นไปตามข้อจำกัด ยกอย่างเช่น มีโค้ดเก่าหรือโค้ดอื่นที่จะมาใช้ร่วมกับ Spark ที่เป็น Python 2.x เราจึงต้องใช้ 2.x ด้วยเป็นต้น อย่างไรก็ตามหากผู้อ่านเลือกลงไปแล้ว ก็สามารถลงควบคู่กัน หรือเปลี่ยนเป็นอีกเวอร์ชั่นภายหลังได้</li><li>Scala 2.11 ในปัจจุบัน Scala มีเวอร์ชั่นล่าสุดไปถึง 2.12 แล้ว ดังนั้น ถ้าจะใช้ Spark ต้องโหลดเวอร์ชั่นย้อนหลัง แค่ 2.11 มาใช้</li></ul><p>เมื่อพร้อมแล้ว เรามาเริ่มดาวน์โหลดโปรแกรมต่าง ๆ กันเลยครับ ลิงค์ด้านล่างเป็นลิงค์ตรงที่จะโหลดไฟล์มาได้เลยโดยไม่ต้องเข้าหน้าเว็บ ยกเว้น Java อันเดียวที่ผู้อ่านต้องเข้าไปกดยอมรับเงื่อนไข และเลือกดาวน์โหลดด้วยตนเองครับ โดยผมแสดงเป็นชื่อไฟล์ไว้ให้เพื่อความชัดเจนครับ โดยโปรแกรมที่โหลดจะเวอร์ชั่นต่าง ๆ ดังนี้ครับ : Spark 2.3.0, JDK 8u162, Scala 2.11.12, Sbt 0.13.17, และ Python 3.6.4.</p><ul><li>spark-2.3.0-bin-hadoop2.7.tgz (<a href="http://www-us.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz">us link</a>, <a href="http://www-eu.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz">eu link</a>)</li><li><a href="http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html">jdk-8u162-macosx-x64.dmg</a></li><li><a href="https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz">scala-2.11.12.tgz</a></li><li><a href="https://github.com/sbt/sbt/releases/download/v0.13.17/sbt-0.13.17.tgz">sbt-0.13.17.tgz</a></li><li><a href="https://www.python.org/ftp/python/3.6.4/python-3.6.4-macosx10.6.pkg">python-3.6.4-macosx10.6.pkg</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*piQoZCjVO8yfp2cfLinDmA.png" /><figcaption>Java JDK 8 เป็นลิงค์เดียวที่ต้องเข้าไปกดเลือก โดยให้เลือกเวอร์ชั่น macOS ครับ (นามสกุล .dmg)</figcaption></figure><h3>Step 2: ตำแหน่ง Path/Folder สำหรับการติดตั้ง</h3><h4>2.1 โฟล์เดอร์ Home ในบทความนี้</h4><ul><li>ตำแหน่ง<em> Home </em>ที่ใช้ในบทความนี้คือตำแหน่ง /Users/luckspark<strong> </strong>ซึ่งสามารถใช้สัญลักษณ์ $HOME หรือ ~ แทนได้. นั่นคือ /Users/luckspark/server และ $HOME/server และ ~/server คือสิ่งเดียวกัน. โดยในบทความนี้จะใช้สลับ ๆ กันครับ</li><li>Home folder ของผู้อ่านน่าจะแตกต่างจากที่ผมใช้ในบทความนี้ค่อนข้างแน่นอน เพราะใช้ username ต่างกัน ดังนั้น <strong>เมื่อตั้งค่าในขั้นตอนต่าง ๆ หลังจากนี้ ก็อย่าลืมเปลี่ยนตามด้วยครับ </strong>เช่นว่า หากใช้ username ชื่อ tim ก็จะต้องเปลี่ยนทุกจุดที่เป็น /Users/luckspark/server ให้กลายเป็น /Users/tim/server เป็นต้นครับ</li></ul><h4>2.2 ตำแหน่งที่จะติดตั้งโปรแกรมต่าง ๆ</h4><ul><li>โปรแกรม Java และ Python จะถูกติดตั้งใน Path ของระบบ ซึ่งเป็นตำแหน่งมาตรฐานของเค้าครับ เราไม่จำเป็นต้องไปเปลี่ยนแปลงอะไร</li><li>ส่วนโปรแกรม Spark, Sbt, และ Scala เราสามารถเลือกตำแหน่ง Path/Folder ที่จะติดตั้งได้ ในบทความนี้ ผมเลือกติดตั้งที่ /Users/luckspark/server (ซึ่งก็คือ $HOME/server หรือ ~/server) ครับ</li><li>การสร้าง folder ที่ชื่อ server ที่ HOME สามารถทำได้โดย (เลือกวิธีใดวิธีหนึ่งนะครับ)</li></ul><ol><li>เปิด finder ขึ้นมา ไปที่ HOME และคลิกขวา → New Folder และตั้งชื่อว่า server</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/1*t5z2bynhzbUFXpScf5s2dg.png" /></figure><p>2. <strong>หรือ</strong> สำหรับคนที่ถนัด shell ก็ให้เปิดโปรแกรม Terminal ขึ้นมาและใช้คำสั่งต่อไปนี้ครับ</p><pre>cd<br>mkdir server</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FacmnHP9LSua2FaWoGcfTw.png" /><figcaption>สร้าง ~/server ด้วยคำสั่ง mkdir</figcaption></figure><ul><li>เมื่อใช้คำสั่ง ls -l ดู จะเห็น folder ที่ชื่อ server ปรากฎขึ้นมา</li><li>คำสั่ง cd จะเปลี่ยนโฟล์เดอร์กลับมาที่ HOME และคำสั่ง mkdir server ก็คือสร้างโฟล์เดอร์ชื่อ server ขึ้นมา</li></ul><h4>2.3 Copy ไฟล์ที่ดาวน์โหลดมาทั้งหมดไปไว้ที่ $HOME/server</h4><ul><li>เพื่อให้ง่าย ให้ผู้อ่าน copy ไฟล์ที่ดาวน์โหลดมาทั้งหมดใน Step 1 ไปไว้ที่ $HOME/server ครับ. ซึ่งเมื่อ copy เสร็จแล้วก็จะมีหน้าตาประมาณนี้</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*auTyTtAfPWLLUkMRj5f2vg.png" /><figcaption>Copy all downloaded files to <strong>~HOME/server</strong> directory</figcaption></figure><h3>Step 3: แตกไฟล์ .tgz</h3><ul><li>ไฟล์ที่โหลดมาในข้อ 1 จะเป็นไฟล์ชนิด .tgz อยู่ 3 ไฟล์ ได้แก่ Spark, Sbt, และ Scala.</li><li>เราแค่แตกไฟล์เหล่านี้ออกมา และบอกระบบว่าไฟล์เหล่านี้อยู่ที่ไหน (ซึ่งจะกล่าวใน Step 6) แค่นี้ก็จะใช้งานได้แล้วครับ</li><li>วิธีการแตกไฟล์ ก็มี 2 วิธีครับ 1) แบบง่าย ๆ กับ 2) แบบ command line. <strong>เลือกทำวิธีใดวิธีหนึ่งครับ</strong></li><li>1) แบบง่าย ๆ ก็คือ double click ที่ไฟล์นามสกุล .tgz เลยครับ โปรแกรม Archive Utility ก็จะถูกเรียกขึ้นมาทำการแตกไฟล์ให้ครับ</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V5R6_ylhhpzjd9YoN_6JwQ.png" /><figcaption>The Archive Utility program extracts the files automatically.</figcaption></figure><ul><li>2) วิธีใช้ command line บนโปรแกรม Terminal สามารถทำได้โดยใช้คำสั่งต่อไปนี้ครับ</li></ul><pre>cd<br>cd server<br>tar xzvf spark-2.3.0-bin-hadoop2.7<br>tar xzvf sbt-0.13.17.tgz<br>tar xzvf scala-2.11.12</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SHcpvLV4JBjYDct6uerQnA.png" /><figcaption>การแตกไฟล์โดยใช้คำสั่ง tar xzvf</figcaption></figure><ul><li>เมื่อแตกไฟล์เสร็จแล้ว จะได้ folder ใหม่ 3 folder ตามภาพครับ</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*BIKUCM8tdxcgB6OdgnsHAA.png" /><figcaption>$HOME/server เมื่อแตกไฟล์ .tgz เสร็จแล้ว จะมีโฟลเดอร์ใหม่เกิดขึ้นมา</figcaption></figure><h3>Step 4: ติดตั้ง JDK</h3><ul><li>Double click ไฟล์jdk-8u162-macosx-x64.dmg ตัว install จะถูกเรียกขึ้นมา</li><li>Double click ที่รูปกล่องสีน้ำตาล (JDK 8 Update 162.pkg ) หน้าจอการติดตั้งจะแสดงขึ้นมา</li><li>เลือก Continue ตามหน้าจอไปเรื่อย ๆ โดยใช้ค่ามาตรฐานที่ให้มา จนเสร็จสิ้นครับ</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*Y_U5y5fCQt5DJkVALCU-0A.png" /><figcaption>การติดตั้ง JDK. ให้ Double click ที่รูปกล่องเพื่อเริ่มติดตั้ง</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0r8-LmhmaEh1FsAZ7_GEqA.png" /><figcaption>หน้าจอการติดตั้ง กด Continue และตามหน้าจอไปเรื่อย ๆ จนเสร็จสิ้น โดยใช้ค่า default ที่ให้มา</figcaption></figure><h3>Step 5: ติดตั้ง Python 3</h3><ul><li>Double click ไฟล์ python-3.6.4-macosx10.6.pkg</li><li>กด Continue ไปเรื่อย ๆ ตนเสร็จสิ้น โดยใช้ค่า default ที่ให้มา</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3f_8-WEi8_s4eimemgF3KQ.png" /><figcaption>หน้าจอการติดตั้ง Python 3</figcaption></figure><h3>Step 6: ตั้งค่าในไฟล์ ~/.bash_profile</h3><h4>6.1 สรุปค่า Path ต่าง ๆ ที่ได้ใช้ในการติดตั้งโปรแกรมที่ผ่านมา</h4><p>ในขั้นตอน 3–5 เราได้ติดตั้งโปรแกรมต่าง ๆ ตาม Path ต่าง ๆ ดังนี้</p><ul><li>JDK: /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk</li><li>Python: /Library/Frameworks/Python.framework/Versions/3.6</li><li>Sbt: /Users/luckspark/server/sbt</li><li>Scala: /Users/luckspark/server/scala-2.11.12</li><li>Spark: /Users/luckspark/server/spark-2.3.0-bin-hadoop2.7</li></ul><p>ค่าต่าง ๆ เหล่านี้จะนำไปใช้ในขั้นตอน 6.2 ซึ่ง<strong>ขอย้ำอีกครั้งว่าต้องเปลี่ยนค่า Path จาก </strong><strong>/Users/luckspark/server ให้เป็นค่า Path ของเครื่องของผู้อ่านเสมอ</strong> ส่วน Path ของ Java กับ Python นั้นไม่ต้องเปลี่ยนครับ</p><h4>6.2 เกี่ยวกับไฟล์ .bash_profile</h4><ul><li>ไฟล์ .bash_profile เป็นไฟล์ระบบ ที่ใช้ในการกำหนดค่าตัวแปรของสภาพแวดล้อมต่าง ๆ ของระบบ (environment variables) ซึ่งในที่นี้ เราจะใช้กำหนดค่า Path เป็นหลัก <em>ซึ่งนั่นก็คือการบอกกับระบบว่าโปรแกรมที่เราได้ติดตั้งไปแล้วนั้นอยู่ที่ใดในระบบ</em> เมื่อมีการเรียกใช้ จะได้หาเจอ</li><li>ไฟล์ .bash_profile นี้ เป็นไฟล์ที่จงใจให้ขึ้นต้นด้วย “จุด” <strong>ดังนั้น เวลาเขียนต้องมีจุดข้างหน้าด้วยเสมอ ไม่เช่นนั้น จะใช้งานไม่ได้</strong></li><li>การตั้งชื่อไฟล์ด้วยการใส่จุดเข้าไปข้างหน้านั้น จะทำให้ไฟล์อยู่ในสถานะ “ถูกซ่อน” (hidden) ดังนั้นเมื่อเราใช้คำสั่ง ls จะมองไม่เห็นไฟล์นี้ หากต้องการให้มองเห็น ต้องใส่พารามิเตอร์ -a ด้วย เช่น ls -a หรือ ls -la</li><li>ตำแหน่งของไฟล์ .bash_profile นี้จะต้องอยู่ที่ตำแหน่ง HOME. นั่นคือ /Users/luckspark/.bash_profile หรือ /Users/tim/.bash_profile เป็นต้น ซึ่งก็คือ ~/.bash_profile นั่นเอง หากไปอยู่ที่อื่น จะไม่ถูกเรียกใช้งาน</li></ul><h4>6.3 ลงมือตั้งค่าของไฟล์ .bash_profile</h4><ul><li>สิ่งที่เราจะทำก็คือ 1) เปิดไฟล์ .bash_profile ขึ้นมา 2) เพิ่มข้อมูล Path ต่าง ๆ เข้าไป 3) บันทึกไฟล์</li><li>เปิดไฟล์ .bash_profile ด้วยโปรแกรมอ่านไฟล์ทั่วไป เช่น TextEdit, nano, vi, หรือ sublime.</li><li>ในที่นี้ ผมจะให้ลองใช้คำสั่งที่เปิดอ่านไฟล์ .bash_profile ด้วย TextEdit จาก command line ซึ่งสามารถทำได้โดย ไปที่ Terminal และพิมพ์คำสั่งต่อไปนี้</li></ul><pre>cd<br>touch -c .bash_profile<br>open -a TextEdit .bash_profile</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UokOErIbQm4yJ89t9u2BHQ.png" /><figcaption>เปิดไฟล์ .bash_profile ด้วย TextEdit จาก command line. ข้อมูลในไฟล์ .bash_profile ของท่าน อาจจะเป็นไฟล์เปล่าหรือแตกต่างจากที่แสดงในภาพได้</figcaption></figure><ul><li>คำสั่งข้างบน 3 คำสั่งคือ 1) เปลี่ยน directory กลับไปที่ HOME, 2) หากไม่มีไฟล์ชื่อ .bash_profile ให้สร้างไฟล์ขึ้นมา (ถ้ามีแล้ว ก็ไม่ทำอะไร), 3) เปิดไฟล์ .bash_profile ด้วย TextEdit</li><li><em>ข้อมูลในไฟล์ .bash_profile ของผู้อ่าน อาจจะเป็นไฟล์เปล่าหรือแตกต่างจากที่แสดงในภาพได้</em></li><li>จากนั้น ให้ copy ข้อมูลข้างล่างนี้ใส่ใน .bash_profile โดยอย่าลืมเปลี่ยน Path จาก /Users/luckspark ทุกอัน ให้เป็น Path ที่ผู้อ่านใช้นะครับ</li></ul><pre>export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/<br>export SPARK_HOME=/Users/luckspark/server/spark-2.3.0-bin-hadoop2.7<br>export SBT_HOME=/Users/luckspark/server/sbt<br>export SCALA_HOME=/Users/luckspark/server/scala-2.11.12</pre><pre>export PATH=$JAVA_HOME/bin:$SBT_HOME/bin:$SBT_HOME/lib:$SCALA_HOME/bin:$SCALA_HOME/lib:$PATH</pre><pre>export PATH=$JAVA_HOME/bin:$SPARK_HOME:$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH</pre><pre>export PYSPARK_PYTHON=python3</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HBJY4LrfBi-SstkAen1iyg.png" /><figcaption>ไฟล์ .bash_profile เมื่อใส่ข้อมูลแล้ว</figcaption></figure><ul><li>สังเกตว่า ข้อมูล 4 บรรทัดสุดท้ายในภาพข้างบน (ซึ่งเป็นการเซ็ต Path ของ Python) ผมไม่ได้เขียนใส่ใน config ที่ให้ copy ไปใส่ในไฟล์นะครับ ทั้งนี้เพราะข้อความ 4 บรรทัดนี้ อาจจะถูกใส่มาแล้วโดยตัวติดตั้งของ Python หากว่า .bash_profile ของผู้อ่าน ยังไม่มีข้อความดังกล่าว ก็ให้ copy ไปใส่เพิ่มเติมได้ครับ</li></ul><pre>PATH=”/Library/Frameworks/Python.framework/Versions/3.6/bin:${PATH}”<br>export PATH</pre><ul><li>ข้อสังเกต — ตัวอักษร # ใน .bash_profile เป็นการ comment. หมายความว่า บรรทัดใดที่ขึ้นต้นด้วย # จะไม่ทำงาน</li><li>Save และ ปิดไฟล์ .bash_profile ครับ</li></ul><h3>Step 7: ปิด Terminal</h3><ul><li>เนื่องจากเราเซ็ตค่าในไฟล์ .bash_profile ใหม่ ดังนั้นก็ต้อง ปิด เปิดโปรแกรม Terminal ใหม่เพื่อให้มันอ่านค่าใหม่ที่เราเพิ่งเซ็ตให้มันครับ</li><li>การปิดนี้ ต้องปิดแบบ Quit นะครับ คือปิดทิ้งทั้งโปรแกรม ไม่ใช่แค่ Close หรือแค่ปิดหน้าต่างไป ใน Mac นั้น Quit กับ Close ต่างกันนะครับ</li><li>การปิดแบบ Quit นี้คือต้องไปที่เมนู และเลือก Quit Terminal หรือกดคีย์ ⌘Q</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*AX43D-jW6DMKaL0Ql7qNVA.png" /><figcaption>Quite Terminal</figcaption></figure><h3>Step 8: ทดสอบการติดตั้ง</h3><ul><li>เปิดโปรแกรม Terminal ขึ้นมาอีกครั้ง</li></ul><h4>8.1 ทดสอบ Java</h4><ul><li>ใน Terminal ให้พิมพ์ java -version ซึ่งถ้าทุกอย่างถูกต้อง จะแสดงผลดังภาพข้างล่าง โดยจะแสดงหมายเลข version ของ java ที่เราดาวน์โหลดมาชัดเจน</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3xj6GhGr2JZuztIOkb7__A.png" /><figcaption>ผลลัพท์ของคำสั่ง java -version.</figcaption></figure><h4>8.2 ทดสอบ PySpark</h4><ul><li>PySpark คือ interactive shell ของ​ Spark ที่ใช้ภาษา Python</li><li>เมื่อเราเปิด PySpark ขึ้นมา เราสามารถป้อนคำสั่งภาษา Python เพื่อสั่งงาน Spark ได้ แต่เป็นการสั่งทีละบรรทัด และให้ผลลัพท์ของแต่ละคำสั่งทันที (จึงเรียกว่า interactive shell)</li><li>ใน Terminal พิมพ์pyspark , ซึ่งก็จะให้ผลลัพท์ดังภาพด้านล่าง. จะเห็นตัวกราฟฟิกคำว่า Spark ใหญ่ ๆ พร้อมเลขเวอร์ชั่น 2.3.0 ชัดเจน</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bbb-KIPYWAU9ZmtJCK4lWA.png" /><figcaption>Output ของคำสั่ง pyspark.</figcaption></figure><ul><li>เราลองเขียนคำสั่ง Python เล่น ๆ ได้ครับ ลองใส่คำสั่งต่อไปนี้ดู พิมพ์ลงไปทีละบรรทัดนะครับ ซึ่งก็น่าจะได้ output ตามภาพข้างล่างครับ</li></ul><pre>a=5<br>b=3<br>a+b</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kx1AtENFOP2QVU2_NViBlA.png" /><figcaption>ทดสอบการป้อนภาษา Python ไปที่ PySpark</figcaption></figure><ul><li>เมื่อจะออกจาก PySpark ให้กด CTRL-D หรือพิมพ์ exit()</li></ul><h4>8.3: ทดสอบ spark-shell</h4><ul><li>Spark-shell คือ interactive shell ของ​ Spark ที่ใช้ภาษา Scala</li><li>เมื่อเราเปิด spark-shell ขึ้นมา เราสามารถป้อนคำสั่งภาษา Scala เพื่อสั่งงาน Spark ทีละบรรทัดได้</li><li>ใน Terminal พิมพ์spark-shell , ซึ่งก็จะให้ผลลัพท์ดังภาพด้านล่าง จะเห็นตัวกราฟฟิกคำว่า Spark ใหญ่ ๆ พร้อมเลขเวอร์ชั่น 2.3.0 ชัดเจน คล้าย ๆ pyspark แต่ต่างกันตรงที่ prompt จะแสดงคำว่า scala&gt;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*11L9ghlrTImI5RhhFBshPw.png" /><figcaption>Output screen ของ spark-shell.</figcaption></figure><ul><li>เราทดลองเขียนภาษา Scala กันครับ โดยลองพิมพ์ดังนี้ครับ</li></ul><pre>val a = 5<br>val b = 3<br>a+b</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ANaOuUYOcaFLHlmh48BNwA.png" /><figcaption>Output ของการป้อนภาษา Scala ไปที่ spark-shell</figcaption></figure><ul><li>เมื่อจะออกจาก PySpark ให้กด CTRL-D</li></ul><h3>แค่นี้ครับ เสร็จเรียบร้อยครับ ^_^</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c100e023c6b9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/luckspark/%E0%B8%81%E0%B8%B2%E0%B8%A3%E0%B8%95%E0%B8%B4%E0%B8%94%E0%B8%95%E0%B8%B1%E0%B9%89%E0%B8%87-apache-spark-2-3-0-%E0%B8%9A%E0%B8%99-macos-high-sierra-c100e023c6b9">การติดตั้ง Apache Spark 2.3.0 บน macOS High Sierra</a> was originally published in <a href="https://medium.com/luckspark">LuckSpark</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installing Apache Spark 2.3.0 on macOS High Sierra]]></title>
            <link>https://medium.com/luckspark/installing-spark-2-3-0-on-macos-high-sierra-276a127b8b85?source=rss----d533d9556b8c---4</link>
            <guid isPermaLink="false">https://medium.com/p/276a127b8b85</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[macos]]></category>
            <dc:creator><![CDATA[LC]]></dc:creator>
            <pubDate>Tue, 06 Mar 2018 09:47:54 GMT</pubDate>
            <atom:updated>2018-03-09T06:13:39.571Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*99lgzUPQOsSKbx3xNHKo-g.png" /></figure><blockquote>Apache Spark 2.3.0 has been released on 28 February 2018. This tutorial guides you through its essential installation steps on macOS High Sierra.</blockquote><p><a href="https://medium.com/luckspark/%E0%B8%81%E0%B8%B2%E0%B8%A3%E0%B8%95%E0%B8%B4%E0%B8%94%E0%B8%95%E0%B8%B1%E0%B9%89%E0%B8%87-apache-spark-2-3-0-%E0%B8%9A%E0%B8%99-macos-high-sierra-c100e023c6b9">อ่านฉบับภาษาไทยได้ที่นี่ครับ</a></p><h3>Step 1: List of Downloads</h3><p>As clearly mentioned in <a href="https://spark.apache.org/docs/latest/">Spark’s documentation</a>, in order to run Apache Spark 2.3.0 you need “Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark 2.3.0 uses Scala 2.11”. The download links below are for JDK 8u162, Scala 2.11.12, Sbt 0.13.17, and Python 3.6.4.</p><ul><li>spark-2.3.0-bin-hadoop2.7.tgz (<a href="http://www-us.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz">us link</a>, <a href="http://www-eu.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz">eu link</a>)</li><li><a href="http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html">jdk-8u162-macosx-x64.dmg</a></li><li><a href="https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz">scala-2.11.12.tgz</a></li><li><a href="https://github.com/sbt/sbt/releases/download/v0.13.17/sbt-0.13.17.tgz">sbt-0.13.17.tgz</a></li><li><a href="https://www.python.org/ftp/python/3.6.4/python-3.6.4-macosx10.6.pkg">python-3.6.4-macosx10.6.pkg</a>. Although optional (as macOS has built-in Python), it is recommended to install your own Python.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*piQoZCjVO8yfp2cfLinDmA.png" /><figcaption>Java JDK 8 download page, choose the macOS version to download.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lz_d1HMbVc_GQNW8EtLYaQ.png" /><figcaption>Python download page. If you have old codes that specifically requires version 2.x, choose 2.7.x. If you are new to Python or Spark, choose 3.x (i.e., download version 3.6.4 here). You can always swap between 2.x or 3.x Python later.</figcaption></figure><h3>Step 2: Installation Preparations</h3><h4>2.1 The HOME folder of this tutorial</h4><ul><li><em>Home folder</em> of this tutorial is /Users/luckspark<strong>. </strong>This home directory can also be referred to as $HOME or ~. Therefore, /Users/luckspark/server and $HOME/server and ~/server are the same.</li><li>Your HOME folder is probably different. Therefore, it is import that you replace all /Users/luckspark/server to match your HOME directory on your machine.</li><li>For example, you might want to replace all /Users/luckspark/server to /Users/tim/server.</li></ul><h4>2.2 The installation folder of this tutorial</h4><p>In this tutorial, Sbt, Scala, and Spark, will be installed at /Users/luckspark/server (i.e., $HOME/server or ~/server). You can create the server directory under your HOME using the following commands</p><pre>cd<br>mkdir server</pre><ul><li>Note for beginners, the command cd changes the directory (from wherever it is) to HOME directory. The commands above, thus, change the directory back to HOME, then create a new directory named “server”.</li></ul><h4>2.3 Copy all downloaded files to $HOME/server folder</h4><ul><li>For simplicity, <strong>copy all downloaded files from step 1 to the </strong><strong>$HOME/server folder</strong>. Your server folder shall look like this.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*auTyTtAfPWLLUkMRj5f2vg.png" /><figcaption>Copy all downloaded files to <strong>~HOME/server</strong> directory</figcaption></figure><h3>Step 3: Extract the downloaded files</h3><ul><li>Extract the .tgz files (sbt*.tgz, spark*.tgz, and scala*.tgz) by double click each file, which will launch the Archive Utility program and extract the files automatically.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V5R6_ylhhpzjd9YoN_6JwQ.png" /><figcaption>The Archive Utility program extracts the files automatically.</figcaption></figure><ul><li>There will be 3 new folders, each of which corresponds to each .tgz files, as shown below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*BIKUCM8tdxcgB6OdgnsHAA.png" /><figcaption>$HOME/server after extracting all .tgz files.</figcaption></figure><h3>Step 4: Install JDK</h3><ul><li>Double click the jdk-8u162-macosx-x64.dmg to launch JDK installation process.</li><li>Double click the JDK 8 Update 162.pkg icon to install. The installation wizard screen will pop up.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*Y_U5y5fCQt5DJkVALCU-0A.png" /><figcaption>JDK installation. Double click the box icon to begin the installation.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0r8-LmhmaEh1FsAZ7_GEqA.png" /><figcaption>Installation wizard screens. Just follow the default options.</figcaption></figure><h3>Step 5: Install Python 3</h3><ul><li>Double click the python-3.6.4-macosx10.6.pkg file to start Python 3 installation. Follow the wizard screens with default options.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3f_8-WEi8_s4eimemgF3KQ.png" /><figcaption>Python 3 installation wizard screen. Follow the default options.</figcaption></figure><h3>Step 6: Setup shell environment by editing the ~/.bash_profile file</h3><h4>6.1 Summary of directory paths</h4><p>Here are the directory paths of the programs that we have installed so far:</p><ul><li>JDK: /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk</li><li>Python: /Library/Frameworks/Python.framework/Versions/3.6</li><li>Sbt: /Users/luckspark/server/sbt</li><li>Scala: /Users/luckspark/server/scala-2.11.12</li><li>Spark: /Users/luckspark/server/spark-2.3.0-bin-hadoop2.7</li></ul><p>These paths will be used in step 6.2. <strong>Make sure to replace the </strong>/Users/luckspark/server<strong> with your HOME path</strong>. You do not need to modify the paths of JDK and Python.</p><h4>6.2 Setup .bash_profile file</h4><ul><li>Note for beginners, this file is a special file in which the name is deliberately start with a “dot”. Therefore, <strong>make sure that you type the file name correctly, which is</strong> <strong>.bash_profile (with a “dot” in front)</strong>.</li><li>Open the .bash_profile file, which is located at your HOME directory (i.e., ~/.bash_profile), using any text editor (e.g., TextEdit, nano, vi, or sublime). For example, open the Terminal app and use these commands to open the .bash_profile with Mac’s TextEdit app.</li></ul><pre>cd<br>touch -c .bash_profile<br>open -a TextEdit .bash_profile</pre><ul><li>Note for beginners, the commands above 1) change the directory back to HOME directory, 2) If does not exist, create a file named .bash_profile<em>, if the file does already exist, this will not overwrite the file</em>, and 3) open the .bash_profile with TextEdit app.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UokOErIbQm4yJ89t9u2BHQ.png" /><figcaption>Open .bash_profile using TextEdit app. Your .bash_profile could be blank or contain different texts.</figcaption></figure><ul><li>Copy these lines to the file.</li></ul><pre>export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/<br>export SPARK_HOME=/Users/luckspark/server/spark-2.3.0-bin-hadoop2.7<br>export SBT_HOME=/Users/luckspark/server/sbt<br>export SCALA_HOME=/Users/luckspark/server/scala-2.11.12</pre><pre>export PATH=$JAVA_HOME/bin:$SBT_HOME/bin:$SBT_HOME/lib:$SCALA_HOME/bin:$SCALA_HOME/lib:$PATH</pre><pre>export PATH=$JAVA_HOME/bin:$SPARK_HOME:$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH</pre><pre>export PYSPARK_PYTHON=python3</pre><ul><li>The .bash_profile file shall look like this.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HBJY4LrfBi-SstkAen1iyg.png" /><figcaption>~/.bash_profile with environment variables configured. Again, do not forget to replace all <strong>/Users/luckspark/server</strong> to match your installation PATH.</figcaption></figure><ul><li>Note that the last 4 lines are the path for Python and could have been automatically added from the Python installation process. If not, you can copy and paste these lines manually.</li></ul><pre>PATH=”/Library/Frameworks/Python.framework/Versions/3.6/bin:${PATH}”<br>export PATH</pre><ul><li>Note for beginners, the lines starting with “#” in the .bash_profile are comment lines.</li><li>Save and close the file.</li></ul><h3>Step 7: Reload .bash_profile</h3><p>Since the .bash_profile has been changed, we have to reload it. Options are</p><ul><li>Type source ~/.bash_profile</li></ul><p>OR</p><ul><li>Quit and reopen the Terminal program. Make sure you completely quit the Terminal using menu → Quit Terminal (⌘Q), otherwise the environment variables declared above will not be loaded.</li></ul><h3>Step 8: Test the installation</h3><ul><li>Open the Terminal app.</li></ul><h4>8.1 Test Java</h4><ul><li>type java -version which shall return the screen below</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3xj6GhGr2JZuztIOkb7__A.png" /><figcaption>Output of java -version.</figcaption></figure><h4>8.2 Test PySpark</h4><ul><li>PySpark is Spark’s Python interactive shell.</li><li>At the Terminal, type pyspark , you shall get the following screen showing Spark banner with version 2.3.0.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bbb-KIPYWAU9ZmtJCK4lWA.png" /><figcaption>Output screen of pyspark.</figcaption></figure><ul><li>Type CTRL-D or exit() to exit the pyspark shell.</li></ul><h4>8.3: Test spark-shell</h4><ul><li>Spark-shell is interactive Spark shell for Scala.</li><li>At the Terminal, type spark-shell , you shall get the following screen output.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*11L9ghlrTImI5RhhFBshPw.png" /><figcaption>Output screen of spark-shell.</figcaption></figure><ul><li>Type CTRL-D to quit spark-shell.</li></ul><h3>That’s it. Enjoy your Spark.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=276a127b8b85" width="1" height="1" alt=""><hr><p><a href="https://medium.com/luckspark/installing-spark-2-3-0-on-macos-high-sierra-276a127b8b85">Installing Apache Spark 2.3.0 on macOS High Sierra</a> was originally published in <a href="https://medium.com/luckspark">LuckSpark</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>