<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Geraldo Souza Junior on Medium]]></title>
        <description><![CDATA[Stories by Geraldo Souza Junior on Medium]]></description>
        <link>https://medium.com/@geraldosouzajunior?source=rss-a3c5a66f3384------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*_MyJCOfjD3amtfgxHxPa2g.png</url>
            <title>Stories by Geraldo Souza Junior on Medium</title>
            <link>https://medium.com/@geraldosouzajunior?source=rss-a3c5a66f3384------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:53:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@geraldosouzajunior/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Image Segmentation on Apache Spark]]></title>
            <link>https://medium.com/data-science/image-segmentation-on-apache-spark-46164dd53c73?source=rss-a3c5a66f3384------2</link>
            <guid isPermaLink="false">https://medium.com/p/46164dd53c73</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[k-means-clustering]]></category>
            <category><![CDATA[image-segmentation]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Geraldo Souza Junior]]></dc:creator>
            <pubDate>Fri, 14 Feb 2020 15:06:18 GMT</pubDate>
            <atom:updated>2020-02-14T15:06:18.819Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Image Segmentation with K-means on Apache Spark and Scala</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*kje5V_qHoJtpFfh2" /><figcaption>Source: Unsplash</figcaption></figure><p>Computer Vision is one of the most exciting branches of data science. There is many possible applications for applying Machine Learning algorithms and techniques and Image Segmentation is one of the first steps into it.</p><p>Image Segmentation is one of the main developments for processing on Computer Vision. Many Computer Vision tasks requires the segmentation of an image, to understand each part and easier comprehension as a whole. Each segment contains a set of pixels, which may represent something. The segmentation is the basis for two important applications of image analysis:</p><ul><li><strong>Object Detection:</strong> identify objects in the image, cars, people or animal, etc. The goal here is to just identify that there is an object in that set of pixels, but not to classify properly which object was it.</li><li><strong>Classification</strong>: after the object detection, the objective is to classify each detected object into a class. The image below illustrates this application, with each object detected and classified.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*WZ1r40rLNd13f5k0" /><figcaption>Image Source: Towards a Meaningful 3D Map Using a 3D Lidar and a Camera. <a href="https://www.researchgate.net/publication/326875064_Towards_a_Meaningful_3D_Map_Using_a_3D_Lidar_and_a_Camera">https://www.researchgate.net/publication/326875064_Towards_a_Meaningful_3D_Map_Using_a_3D_Lidar_and_a_Camera</a></figcaption></figure><p><strong>Segmentation Techniques</strong></p><p>There is many ways to achieve the segmentation of an image, through Deep Learning, region based approaches (which will try to detect the borders of an object), through the application of clustering algorithms and so on. Each technique as its advantages and disadvantages. Deep Learning techniques has proven the best results so far, but the implementation tends to be complex and requires a large number of data to train the model. Of course, it is possible to use pre trained models available, but for use on a specific application, it will require training, which can be time and processing consuming. Another possible way, is to apply clustering algorithms. The goal is to identify groups in the data, assigning desired information from each pixel (x and y positions, color information), to classify them in one of the groups, based on feature similarity. In contrast to Deep Learning, it is less efficient, because rather than analyzing pre defined groups, clustering works iteratively to organically form groups, which will require more human interference. However, since clustering is <strong>unsupervised</strong>, it does not require too much data too train, which can reduce development time and save processing time compared to Deep Learning counterparts.</p><p><strong>K-Means Clustering</strong></p><p>Before organize the data into groups, we need to understand the concept of distance between two data points, in analytic geometry. There is many ways to compute distance, but i will focus here in one of the most commonly used methods: <strong>Euclidean distance</strong>. According to <a href="https://en.wikipedia.org/wiki/Euclidean_distance">wikipedia</a> ,the <strong>Euclidean distance</strong> between two points <strong>p</strong> and <strong>q</strong> is the length of the <a href="https://en.wikipedia.org/wiki/Line_segment">line segment</a> connecting them. If <strong>p</strong> = (<em>p</em>1, <em>p</em>2,…, <em>pn</em>) and <strong>q</strong> = (<em>q</em>1, <em>q</em>2,…, <em>qn</em>) are two points in <a href="https://en.wikipedia.org/wiki/Euclidean_space">Euclidean <em>n</em>-space</a>, the Pythagorean Theorem can be used to calculate the distance between two points:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/0*3QYP9VbFah5gT18G" /></figure><p>Let’s use a practical example. Given two points <strong>p</strong> and <strong>q</strong> , where <strong>p=(5,3) </strong>and <strong>q = (4,2). </strong>Using the image below as illustration, <em>x1</em> = 5, <em>y1</em> =3, <em>x2</em> = 4, <em>y2</em> = 2. Applying the formula, we have d(<strong>p</strong>,<strong>q</strong>) = (4–5)2+ (2–3)2 = 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/0*9ZHZcMVUu2qpRvCr" /></figure><p>Now that we have an understanding of distance, we can proceed to K-Means computation. First it is defined first the final number of clusters <strong>K</strong>. The number of cluster should be defined considering the application context of each case. After that, it is necessary to define the <strong>centroids</strong> for the clusters. A <strong>centroid </strong>is defined as the average of all data points in a given cluster. So, if a data point <strong><em>P </em></strong>is closer to a centroid K1 instead of K2, we can say that this point belongs to the group K1 . There is a lot of ways to initialize the centroids. Here it is some of then:</p><ul><li><strong>RP</strong> — <em>randomly selected points</em>. k distinct cases of the data are randomly selected to be the initial centres.</li><li><strong>KMPP</strong> — <em>random farthest points, or k-means++.</em> The first centre is selected as a random case from the dataset. The 2nd centre is selected also randomly, but the probability of selection of a case is proportional to the distance (square euclidean) of it to that (1st) centre. The 3rd centre is selected also randomly with the probability of selection proportional to the distance of a case to the nearest of those two centres, — and so on.</li><li><strong>SIMFP</strong> — <em>farthest points (simple selection).</em> The first centre is selected as a random case from the dataset. The 2nd centre is selected as the case maximally distant from that centre. The 3rd centre is selected as the case maximally distant from those two (from the nearest of the two), — and so on.</li><li><strong>RUNFP</strong> — <em>farthest points (running selection).</em> First k cases are taken as centres and then during the run through the rest of the cases of the dataset there progressively replacements among the centres are done; the aim of the replacements is to obtain in the end k points most distant from each other in the variable space. These points (cases) occupying peripheral positions in the data cloud are the produced initial centres.</li><li><strong>GREP</strong> — <em>group representative points</em>. The method idea — to collect as centres k most representative, “deputy” cases. The 1st centre is taken as the case closest to the general data cenroid. Then the rest of the centres are selected from the data points in such a way that each point is considered as to whether it is closer (and how much, in terms of squared euclidean distance) to a set of points than each one of the latter is to any of the already existing centres. I.e. each point is examined as a candidate to represent some group of points not yet well enough represented by the centres already collected. Point most representative in this respect is selected as the next centre.</li></ul><p>So, now that we have now that we have consolidated the distance and the initialization of centroids, let’s see how the K-Means algorithms works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*4w_vAgNge5JAq5R6BQokPQ.png" /></figure><p>If you still have any doubts in any step, i fully recommend this step by step calculation, wrote by <a href="http://https//www.linkedin.com/in/sunaina-shashikumar/">Sunaina</a>, on <a href="https://www.datasciencecentral.com/profiles/blogs/steps-to-calculate-centroids-in-cluster-using-k-means-clustering">datasciencecentral</a>. Also, this <a href="https://www.youtube.com/watch?v=nXY6PxAaOk0&amp;t=1s">video</a> shows a visualization of each step.</p><h3><strong>Implementing Image Segmentation with K-Mean on Spark</strong></h3><p>Ok, time to start solving the image segmentation problem with the k-means clustering algorithm on apache spark with scala. Wait, but why scala? Python is currently the most preferred language among the data scientists not just it is easy to learn and implement but also for its extensive libraries and frameworks. In data science and machine learning projects, it includes a broad range of useful libraries SciPy, NumPy, Matplolib, Pandas, among others while for more complex projects in deep learning, Python offers libraries such as Keras, Pytorch, and TensorFlow. With all these advantages, again, why scala? Well, Spark is written in Scala as it can be quite fast because it’s statically typed and it compiles in a known way to the JVM. I came from the Java world, and one of the advantages is to use the whole legacy from Java, such as libraries and frameworks.</p><p>I will not dig into the differences between languages because this is not the point of this post (there is hundreds of pos and cons available, google is your friend), but like said before, a fact is that the Python community is huge and provides many libs and frameworks to use together with Spark, a lot of tutorials available. Because of this, my main motivation with this post, is to contribute with the scala community.</p><h4>Loading Images into Spark Data Frames</h4><p>Since, the version 2.4, Spark has a new data source built in, to read compressed formats (jpg, png, etc). The image will be read with the ImageIO Java Library, and it has a special <a href="https://spark.apache.org/docs/latest/sql-programming-guide.html#datasets-and-dataframes">data frame</a> schema . Reading an image file (or several, just pointing to a direcory that contains only images) can be done by:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6b6d88e1fba62f54cd0926d6436e5c3/href">https://medium.com/media/c6b6d88e1fba62f54cd0926d6436e5c3/href</a></iframe><p>The schema contains a <a href="https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.types.StructType"><strong>StructType</strong></a> Column “Image”, that contains all the information about the read data. Inside the StructType, you can find the following columns:</p><ul><li>origin: StringType (represents the file path of the image)</li><li>height: IntegerType (height of the image)</li><li>width: IntegerType (width of the image)</li><li>nChannels: IntegerType (number of image channels)</li><li>mode: IntegerType (OpenCV-compatible type)</li><li>data: BinaryType (Image bytes in OpenCV-compatible order: row-wise BGR in most cases)</li></ul><p>The important data for us, is the array of bytes that spark generates for us, representing the raw image data. The array contains all the features that we may use for clustering.</p><p>Digital images are composed by pixels, and each pixel is composed by a combinations of colors, represented by code. One grayscale image for example, has only one channel of color. For computer displays, the most used standard, is the <strong>RGB. </strong>An RGB image has three channels, each one meaning the intensivity brightness of the respective colours: red, green and blue. So, basically, an RGB image is a combination of three images (one for each channel), each one containing information about the respective colour.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*94Wsbcz65z5gRw04XoVTvA.png" /></figure><p>Have you ever tried too look closer into old CRT tv’s screens? If so, you probably remember that you could see the three color channels. The ideia is the same here, as illustrates the image .</p><p><strong>How Spark decode the image</strong></p><p>To build our feature map, we need to understand how spark read and decode the image. Diving into spark code, we have the<strong> </strong><a href="https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/image/ImageSchema.scala#L131-L190"><strong>ImageSchema.scala</strong></a><strong> </strong>object, more specifically, the <strong>decode</strong> method.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ca417b9946bf38ef823b3baf2a64f34/href">https://medium.com/media/9ca417b9946bf38ef823b3baf2a64f34/href</a></iframe><p>This method basically checks the number of channels that the image has, and decodes based on this number. Let’s take an example: consider an 640 x 480 image, which will have 307200 pixels. If the image is grayscaled (only 1 channel), the final array size will be 307200. However, if it is an RGB image, the final array will have 307200 * 3 = 921600. The Channel order is defined as BGR (and also there is space for the alpha channel, which define transparency). With the same 640 x 480 RGB image example, the first three positions of the array will contain color information(Blue, Green and Red) from the pixel at height = 0 and width = 0. The next three position will contain information from the pixel at at height = 0 and width = 1 and so on.</p><p><strong>Creating the features Data Frame</strong></p><p>Now that we know how spark stores image information, we can read the array of data to build our features data frame, that will be used on the K-Means algorithm. For each pixel, we should have the w position, h position and each one of the color channels.</p><p>To start this task, we will use two images, as provided below, one landscape and a person(i blurred the woman’s face, just to prevent any problems).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/0*1xXIUtcmyk3N-hjw.jpg" /><figcaption>Full size image can be found on: <a href="https://ibb.co/d4xWxvw">https://ibb.co/d4xWxvw</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/0*sBDwlSX6ypLFMCVk.jpg" /><figcaption>Full sizeimage can be found on: <a href="https://ibb.co/d4xWxvw">https://ibb.co/d4xWxvw</a></figcaption></figure><p>The logic to extract the information, is to read the array following the logic on the decode method, and them transposing it into data frame columns. Let’s follow step by step:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/17e15fa1bdc32f91cb6e8c0881cd4a11/href">https://medium.com/media/17e15fa1bdc32f91cb6e8c0881cd4a11/href</a></iframe><p>We start by reading the image and extracting the array of bytes and creating a new RDD for it, applying the flatMap transformation, returning each array position as an entry on the RDD. We could work directly on the image array, but we would lost one of the advatanges of using spark: distributed data structures. By working always with RDD structures, we make sure that our code will always be distributed.</p><p>The next step is to extract informations about channels, height, width and start the accumulators. Accumulators are variables to aggregate values of all spark executors. As we are using RDD’s, we cannot use regular aggregation variables, because when Spark ships this code to every executor the variables become local to that executor and its updated value is not relayed back to the driver. So, if we want to our counters work correctly (for h and w positions and the colours offset), we need to use accumulators.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1b96616255913c80ee6b4c61712f681/href">https://medium.com/media/d1b96616255913c80ee6b4c61712f681/href</a></iframe><p>The final processing begins with the <strong>zipWithIndex</strong> transformation, which will return a new RDD of pairs, a tuple with the element and an index, that will be useful for the bytes offset. The <strong>zipWithIndex</strong> will be useful to assign a index to each entry on the RDD, that will be used for the offset. In the end, the RDD will be converted into a <strong>Data Frame</strong> with three columns, representing the color information and the x and y positions. It is important to notice, that each pixel will have the number of channels described on the schema. So, it will be important to collapse all the color information of a certain pixel into only one entry of the data set. For that, it is used an aggregation on the Data Frame, grouping by the X and Y positions of the pixel. Next, it is applied the <strong>org.apache.spark.sql.functions.collect_list</strong> function, which will take all the aggregated values from the color column and return a list containing it. Finally, it is returned a DataFrame with the X and Y positions, blue, green and red color informations.</p><p>The implementation above considers only images with three channels (RGB), since it is the most common found on images. Images with 4 channels (RGBa) uses the fourth channel for transparency information and would require little adjustment in the offset position of the pixels.</p><p><strong>The Machine Learning Pipeline on Spark</strong></p><p>In general, most of the implementations of machine learning models can be designed as a ordered sequence of some algorithms, like the following:</p><ul><li>Feature extraction, transformation and selection.</li><li>Train a predictive model based on the these vectors and label.</li><li>Make predictions using the generated model.</li><li>Evaluate model (performance and accuracy).</li></ul><p>Spark MLib provides two top level abstractions to facilitate the development of this pipeline: <strong>transformers</strong> and <strong>estimators</strong>. A transformer implements a method transform() which will convert one DataFrame into another, generally appending one or more new column. For example, a transformer will take all the columns features of each entries on the Data Frame and map it into a new column (feature vectors). The estimator will be responsible for applying the learning algorithm that fits or trains on data. It implements the method fit(), that accepts a DataFrame and produces a Model, which is a transformer. As an example, a learning algorithm such as <strong>DecisionTree</strong> is an estimator, and calling fit() trains a <strong>DecisionTreeClassificationModel, </strong>which is a Model and hence a transformer.</p><p>With our DataFrame with feature columns (b,g,r,w,h) ready, we can start our pipeline by applying the transformer <strong>VectorAssembler</strong>. This transformer combines a given list of columns into a single vector column. We use the method <strong>setInputCols</strong> to pass an array, pointing the columns that we want to combine. The method <strong>setOutputCol</strong> is used to point the column that will contain the combined vector.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2cf9b445880e508b7a72a16f99cea300/href">https://medium.com/media/2cf9b445880e508b7a72a16f99cea300/href</a></iframe><p>Initially, only the color informations will be used as features of our model. We can test and discuss the inclusion of x and y positions later. This will be the resulting DataFrame:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/1*kdxHYGv8wnsUfSS_kaGejw.png" /></figure><p>Now, we can create the Kmeans estimator and train our model. The variable K will define the number of clusters that will be used. The <strong>setSeed</strong> method, is for the random initialization of the clusters. If you do not provide the seed, spark will generate one internally, which will compromise tests comparisons, because the performance will be different for each run, since the clusters initial points will be different for each run. The method <strong>setFeaturesCol </strong>receives the column that contains the vector of features.</p><p>The model is created by calling the fit() method and passing the DataFrame that was transformed by the Vector Assembler. The model will give the results by invoking the transform() method, returning a DataFrame containing a new column, prediction.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0a51209f51b5f963a003cd816513873/href">https://medium.com/media/e0a51209f51b5f963a003cd816513873/href</a></iframe><p>Finally, the resulting DataFrame will be like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/1*SSNtAlb8gGHvU3D98DEwBQ.png" /></figure><p>The prediction column will inform which group that entry belong to. Pay attention that the cluster numbers will allways starts with zero.</p><p><strong>Writing the segmented image.</strong></p><p>Spark does not provide a way to write images, so we need to implement one. Also, it is important to decide how to represent the clusters colors on the new rendered image. On this case, i decided to use the first color found from each cluster, to represent itself.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c92ad998ef1e100895c2c795fb623e9/href">https://medium.com/media/3c92ad998ef1e100895c2c795fb623e9/href</a></iframe><p>The ideia is pretty simple. For each cluster, we filter the dataset using the cluster number and get the first entry found, we add the cluster number as the key in a map and the color channels as the value in a Scala Tuple (<a href="https://docs.scala-lang.org/tour/tuples.html">https://docs.scala-lang.org/tour/tuples.html</a>).</p><p>Now it is possible to finally write the final image. For that, we gonna use the class <strong>javax.imageio.ImageIO</strong>. First, we select the w, h and prediction columns and transform the DataFrame into an array, by using the collect() function. Then, we create a Buffered Image, passing the width, height and the image type. Finally, just iterate over the array, writing into the buffered image the pixel position and the respective colored image. Notice that the BufferedImage class only has the <strong>setRGB</strong> method, so we must change the order of the colors when using the class <strong>java.awt.Color</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e83b6b59615f596d5f6585b4b1ad865c/href">https://medium.com/media/e83b6b59615f596d5f6585b4b1ad865c/href</a></iframe><p><strong>Testing the implementation</strong></p><p>Ok, now we can test the solution with the images presented before. But first, we must decide how many clusters we want to have for the segmentation? Looking on the image below, i will use K =5, based on the number of distinct parts that i can observe on naked eye (the red dress, the skin, the ocean, the sky and the ground).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*zt5mzDRdLLcwXEzfUVDvyg.png" /></figure><p>This is the result for K = 5:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*QMu4daZ5xGEpcAOumM_xFw.png" /></figure><p>We can see that the dress and skin were segmented almost perfectly. Some parts were put into the same color of the rocks, because their spectrum are closer. The sky also is almost an unique segment. Following the next image, i will consider K = 5 also.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*B9aok9zDLnRczjDtiSgFeA.png" /></figure><p>This is the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*yfuhKiqx31gRTYJIB3WQmw.png" /></figure><p>The three persons on the beach were classified in the same cluster of the montain and part of the beach, because due to the shadows, they color spectrum are similar.</p><p>Here are other runs with different K’s. Pay attention that by increasing K, we get closer to the original image. This can be useful to compress images. Try to run with K = 128 and K = 256.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p1EgravhmqD52mdxGlu05g.png" /><figcaption>K=3,K=4 and K=6</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cEfj4dLx5_wY1bhJvP3JTw.png" /><figcaption>K=3,K=4 and K=6</figcaption></figure><p><strong>Finding the best value of K</strong></p><p>How can we determine the best value of K for a certain image? Previously, i said that i used my own perception to use a number, but we cannot rely on human perception, this is Data Science!</p><p>Here it is how we will proceed: we will define a range for K, like from 2 to 20. Then, for each K value, we will compute the cost of the K-Means function, and plot the cost on a graph. The value which is minimized with little<br>return gained thereafter, will be the best choice. For plotting, i will use the vegas-viz plot lib for scala (<a href="https://github.com/vegas-viz/Vegas">https://github.com/vegas-viz/Vegas</a>). Lets include the following dependencies in the pom.xml:</p><pre>&lt;dependency&gt;</pre><pre>   &lt;groupId&gt;org.vegas-viz&lt;/groupId&gt;</pre><pre>   &lt;artifactId&gt;vegas_2.11&lt;/artifactId&gt;</pre><pre>   &lt;version&gt;0.3.11&lt;/version&gt;</pre><pre>&lt;/dependency&gt;</pre><pre>&lt;dependency&gt;</pre><pre>   &lt;groupId&gt;org.vegas-viz&lt;/groupId&gt;</pre><pre>   &lt;artifactId&gt;vegas-spark_2.11&lt;/artifactId&gt;</pre><pre>   &lt;version&gt;0.3.11&lt;/version&gt;</pre><pre>&lt;/dependency&gt;</pre><p>We will us eMLlib ‘s KMeans() estimator in the Spark DataFrame containing our feature vectors, iterating over values of k in the range(2, 20). Then we plot the results on a html page (you can plot on a popup window, check vegas-viz documentation, but you will need the javafx package, which is not included on openjdk). Observe that we do not need to run the estimator in the entire DataFrame, taking a sample of 10% only, to save runtime execution.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/97c71fcf39d035bb3791b2603d840727/href">https://medium.com/media/97c71fcf39d035bb3791b2603d840727/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*dtSK9VWtdimgXaQIb6xnAQ.png" /></figure><p>So, as we can see on the results, after K=14 for the beach image does not change the cost too much and K=12 for the woman image. The new rendered images are below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*p2hVXUfzu1K_xbO0WdscEw.png" /><figcaption>K = 12</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*rZ_0bmEBAAR-6ujVLB9aOw.png" /><figcaption>K = 14</figcaption></figure><p>We can see that the woman skin and dress are still in one cluster for each, which can be very usefull for detecting the person on the picture. However, the sky is not just an unique cluster. In the beach image, we can see that the three persons can be identified distinctly from the reflection on the sand.</p><p><strong>Conclusion</strong></p><p>I hope that i contributed to your learning on K-Means algorithm, Apache Spark and ML. Spark is a framework designed for batch processing. It extends the MapReduce model to efficiently use it for more types of computations which includes interactive queries and stream processing.</p><p>Although Spark has many ML algorithms implementations ready to use, it does not have nothing related to Computer Vision, except for the ImageSchema, which made possible load images into DataFrames. While Python developers can find a wide range of libraries and documentations about different approaches of different problems and Mix frameworks for it, i sense that the Scala/Java community is needy about more content regarding data science. This was my motivation for this development.</p><p>You can check the full code here: <a href="https://github.com/gsjunior86/SIS.git">https://github.com/gsjunior86/SIS</a>. If you have any questions, doubts about it, please write me.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46164dd53c73" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/image-segmentation-on-apache-spark-46164dd53c73">Image Segmentation on Apache Spark</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction to Big Data Processing With Spark]]></title>
            <link>https://medium.com/an-introduction-to-big-data-processing-with-spark/an-introduction-to-big-data-processing-with-spark-a5034ef6c785?source=rss-a3c5a66f3384------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5034ef6c785</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Geraldo Souza Junior]]></dc:creator>
            <pubDate>Thu, 05 Apr 2018 19:23:04 GMT</pubDate>
            <atom:updated>2018-04-05T19:23:04.458Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*Pa7PO1v7bANI7C-eHMS_PQ.png" /></figure><p>The concept of Big Data exists long time before the term was created. Since a long time, Stock Markets has billions of transactions each day, Space Industry has a lot data to process (trajectory of rocket launches, simulations) and the Oil Industry processes a lot of data about geology. Ok, but what it is Big Data really? Doug Laney in an article published inworkflow 2001 about the challenge of the increase of data management at that time (the big data term is not even present in this article yet), described the three Vs: volume, velocity and variety. Therefore, to answer the question whether your data is big, we can translate this into the following three sub questions:</p><ul><li><strong>Volume</strong>: <em>Can you store your data in memory?</em></li><li><strong>Velocity</strong>: <em>Can you process new incoming data with a single machine?</em></li><li><strong>Variety</strong>:<em> Is your data from a single source?</em></li></ul><p>If your answered all the questions with yes, then perhaps your data is not that big, you just need to simplify your architecture’s solution.</p><p>If you have answered all the questions with no then your data is big. But if you have mixed answers, then it is complicated. Some will argue that one V is more important than another and you will not find any consensus. The exact answer depends on the problem that you are trying to solve.</p><p><strong>Big data application architecture</strong></p><p>Big data, such as documents, weblogs, social networks, sensor data, and others, are stored in a NoSQL database, such as MongoDB, or a distributed filesystem, such as HDFS. In case we deal with structured data, we can deploy database capabilities using systems such as Cassandra or HBase built atop Hadoop. Data processing follows the MapReduce paradigm, which breaks data processing problems into smaller subproblems and distributes tasks across processing nodes. Machine learning models are finally trained with machine learning libraries such as Mahout and Spark.</p><p><strong>The MapReduce Model</strong></p><p>Mapreduce is a paradigm design to process huge volumes of data. It was introduced by <a href="https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf">Jeffrey Dean and Sanjay Ghemawa at Google in 2004</a> as a model of programming to solution to process huge collections of data in clusterized computers. This paradigm is based in functional programming primitives that allows the manipulation of great volumes of data in distributed and parallel ways. A great number of real applications can be expressed in this programming model.</p><p>The model consists in the creation of a program that is formed by two basic operations: <em>map</em> and <em>reduce</em>. The map operation receives a key/value pair and produces an intermediary set of data, also in the key/value format. The reduce operation is executed to each intermediary key, with all the sets of intermediary values assigned to that key combined. In general, the map operation is used to find something and the reduce operation is used to produce the summarization of the result.</p><p>Let’s consider the problem of count the number of times that a word occurs in a big set of documents. Take a look in the pseudo code for this problem using mapreduce:</p><pre><strong>map</strong>(String input_key, String input_value):<br>// input_key: document name<br>// input_value: document contents<br>  <strong>for each</strong> word w in input_value:<br>    EmitIntermediate(w, “1”);</pre><pre><strong>reduce</strong>(String output_key, Iterator intermediate_values):<br>// output_key: a word<br>// output_values: a list of counts<br>    int result = 0;<br>    <strong>for each</strong> v <strong>in</strong> intermediate_values:<br>      result += ParseInt(v);<br>      Emit(AsString(result));</pre><p>For each word from the input, the map function emit the “1” value assigned to the key that represents the respective word. The reduce function sums all the counts emitted by same key, or the same word.</p><p><strong>Execution:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/0*Mr8-iwT6zTF6Kyui." /><figcaption>The map reduce flow</figcaption></figure><p>The map operation is distributed on multiple machines from the partitioning of the data into pieces that can be processed in parallel by different machines. The reduce operation is distributed through the partitioning of the intermediate switches. The partitioning size and function are user-supplied parameters.</p><p>Do not make confusion with Apache Map Reduce, a framework that implements the Map Reduce paradigm inside Apache Hadoop (The map reduce solution for distributed processing and file storage). By a long time, the Apache Map Reduce was the most popular framework for big data processing. Implemented in Java, it requires a lot of code to execute simple tasks, due to the verbose nature of Java. Then, in 2014 Spark first version was launched.</p><p>Spark was originally conceived at Berkeley’s AMPLab by Matei Zaharia, who went on to cofound Databricks, together with his mentor Ion Stoica, as well as Reynold Xin, Patrick Wendell, Andy Konwinski, and Ali Ghodsi. Although Spark is open source, Databricks is the main force behind Apache Spark, contributing more than 75% of Spark’s code. It also offers Databricks Cloud, a commercial product for big data analysis based on Apache Spark.</p><p>By using Spark’s elegant API and runtime architecture, you can write distributed programs in a manner similar to writing local ones. Spark’s collections abstract away the fact that they’re potentially referencing data distributed on a large number of nodes. Spark also allows the usage of functional programming methods, which are a great match for data-processing tasks, and the great advantage, 100 times faster than Apache Mapreduce.</p><p><strong>Features of Spark</strong></p><p>Spark extends MapReduce by avoiding data movement during processing, through features such as data storage in memory and near-real-time processing, performance can be several times faster than other Big Data technologies.</p><p>On-demand query validation for Big Data is also supported, helping to streamline data processing and provides a higher-level API to improve developer productivity and a consistent model for Big Data solution architect.</p><p>Spark holds intermediate results in memory instead of writing them to disk, which is very useful when you need to process the same data sets many times. Its purpose was to make it an execution mechanism that works both in memory and on disk, so Spark performs disk operations when the data no longer fits in memory. You can then use it to process data sets larger than the aggregate memory in a cluster.</p><p>Spark will store as much data as possible in memory, and then will persist them to disk. It is up to the system architect to look at their data and use cases to assess memory requirements. With this in-memory storage mechanism, the use of Spark brings performance advantages.</p><p><strong>Other features of Spark:</strong></p><ul><li>Supports more than just Map and Reduce functions;</li><li>Optimizes the use of arbitrary graph operators;</li><li>On-demand evaluation of Big Data queries contributes to optimizing the overall flow of data processing;</li><li>Provides concise and consistent APIs in Scala, Java, and Python;</li><li>Provides interactive shell for Scala and Python. The shell is not yet available in Java.</li></ul><p>Spark is written in the Scala language and runs in a Java virtual machine. It currently supports the following languages ​​for application development.</p><ul><li>Scala</li><li>Java</li><li>Python</li><li>Clojure</li><li>R</li></ul><p><strong>Spark Installation</strong></p><p>In this section, we will look into local development Spark installation. Spark is Hadoop’s sub-project. Therefore, it is better to install Spark into a Linux based system. The following steps show how to install Apache Spark.</p><ul><li><strong>Step 1: Verifying Java Installation</strong></li></ul><p>Java installation is one of the mandatory things in installing Spark. Try the following command to verify the JAVA version.</p><pre><strong>$java -version</strong></pre><p>If Java is already installed on your system, you get to see the following response −</p><pre><strong>openjdk version &quot;1.8.0_151&quot;<br>OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)<br>OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)</strong></pre><p>In case you do not have Java installed on your system, then Install Java before proceeding to next step.</p><ul><li><strong>Step 2: Verifying Scala installation</strong></li></ul><p>You should Scala language to implement Spark. So let us verify Scala installation using following command.</p><pre><strong>$scala -version</strong></pre><p>If Scala is already installed on your system, you get to see the following response −</p><pre><strong>Scala code runner version 2.11.8 — Copyright 2002–2016, LAMP/EPFL</strong></pre><p>In case you don’t have Scala installed on your system, then proceed to next step for Scala installation.</p><ul><li><strong>Step 3: Downloading Scala</strong></li></ul><p>Download the latest version of Scala by visit the following link <a href="http://www.scala-lang.org/download/">Download Scala</a>. For this tutorial, we are using scala-2.11.8 version. After downloading, you will find the Scala tar file in the download folder.</p><ul><li><strong>Step 4: Installing Scala</strong></li></ul><p>Follow the below given steps for installing Scala.<br>Extract the Scala tar file<br>Type the following command for extracting the Scala tar file.</p><pre><strong>$ tar xvf scala-2.11.8.tgz</strong></pre><p><strong>Move Scala software files</strong></p><p>Use the following commands for moving the Scala software files, to respective directory (/usr/local/scala).</p><pre><strong>$ su — <br>Password: <br># cd /home/Hadoop/Downloads/ <br># mv scala-2.11.8 /usr/local/scala <br># exit</strong></pre><p><strong>Set PATH for Scala</strong></p><p>Use the following command for setting PATH for Scala.</p><p><strong>$ export PATH = $PATH:/usr/local/scala/bin</strong></p><p><strong>Verifying Scala Installation</strong></p><p>After installation, it is better to verify it. Use the following command for verifying Scala installation.</p><p><strong>$scala -version</strong></p><p>If Scala is already installed on your system, you get to see the following response −</p><pre><strong>Scala code runner version 2.11.8 — Copyright 2002–2016, LAMP/EPFL</strong></pre><ul><li><strong>Step 5: Downloading Apache Spark</strong></li></ul><p>Download the latest version of Spark by visiting the following link <a href="https://spark.apache.org/downloads.html">Download Spark</a>. For this tutorial, we are using <strong>spark-2.2.1-bin-hadoop2.7</strong> version. After downloading it, you will find the Spark tar file in the download folder.</p><ul><li><strong>Step 6: Installing Spark</strong></li></ul><p>Follow the steps given below for installing Spark.</p><p>Extracting Spark tar</p><p>The following command for extracting the spark tar file.</p><pre><strong>$ tar xvf spark-2.2.1-bin-hadoop2.7.tgz</strong></pre><p><strong>Moving Spark software files</strong></p><p>The following commands for moving the Spark software files to respective directory (/usr/local/spark).</p><pre><strong>$ su — <br>Password: <br># cd /home/Hadoop/Downloads/ <br># mv spark-2.2.1-bin-hadoop2.7 ~/spark<br># exit</strong></pre><p>Setting up the environment for Spark</p><p>Add the following line to ~/.bashrc file. It means adding the location, where the spark software file are located to the PATH variable.</p><p><strong>export PATH = $PATH:~/spark/bin</strong></p><p>Use the following command for sourcing the ~/.bashrc file.</p><pre><strong>$ source ~/.bashrc</strong></pre><ul><li><strong>Step 7: Verifying the Spark Installation</strong></li></ul><p>Write the following command for opening Spark shell.</p><pre><strong>$spark-shell</strong></pre><p>If spark is installed successfully then you will find the following output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/0*-_vxSdz-wVNUW7rN." /></figure><p><strong>The default spark-shell is for Scala (shown in the screenshot). To start Python shell execute:</strong></p><pre><strong>$pyspark</strong></pre><p><strong>Introduction to Scala</strong></p><p>Before starting let’s review some concepts of Scala. For further information, see the<a href="http://www.scala-lang.org/old/node/198"> language manual</a> or refer to Martin Odersky book “<a href="https://www.amazon.de/Programming-Scala-3rd-Martin-Odersky/dp/0981531687">Programming in Scala 3rd edition</a>”.</p><p>Scala does type inference. You do not need to declare the variable type. In the example below a tuple with an integer and a string is created. Using getClass you can verify that Scala knows exactly the types of each variable, a tuple being a variable with a set of types.</p><pre>scala&gt; val tup1 = (0, “zero”)<br>scala&gt; tup1.getClass<br>res1: Class[_ &lt;: (Int, String)] = class scala.Tuple2scala&gt; val tup2 = (1, “um”, 1.0)<br>scala&gt; tup2.getClass<br>res2: Class[_ &lt;: (Int, String, Double)] = class scala.Tuple3scala&gt; val tup2 = (1, “um”, 1.0)<br>scala&gt; tup2.getClass<br>res2: Class[_ &lt;: (Int, String, Double)] = class scala.Tuple3</pre><p>You can access the values of each tuple using the underscore syntax (“_”).</p><pre>scala&gt; println(tup1._1)<br>0<br>scala&gt; println(tup1._2)<br>zero<br>scala&gt; println(tup2._2)<br>um<br>scala&gt; println(tup2._3)<br>1.0</pre><p>To define a function in Scala it is necessary to say the type of the parameters and the type of the return. In Scala you can define functions within functions. In addition, Scala supports function names with the characters +, ++, ~, &amp;, -, -,. Below is the sum function that performs the sum of two integers.</p><pre>scala&gt; def sum (x: Int, y: Int) : Int = { return x + y }<br>sum: (x: Int, y: Int)Int<br>scala&gt; sum (1,2)res0: Int = 3</pre><p>Functions do not necessarily need to have a name. Any unnamed function is said anonymous function or lambda function. Below is a lambda function that sums two integers.</p><pre>scala&gt; (x: Int, y: Int) =&gt; x + y<br>res1: (Int, Int) =&gt; Int = &lt;function2&lt;<br>scala&gt; res1(1,2)<br>res2: Int = 3</pre><p>What is an anonymous (lambda) function it is used for? or, how to invoke a lambda function? Many Spark functions have other functions as a parameter. Therefore, lambdas functions are important for not having to formally declare a function. In addition to simplifying and saving code.</p><p>The max function presented below returns the largest value among two integers. Note that max receives two integers, not assigning them any variables. In the body of the max function there is the declaration of a lambda function, which receives two integers, and verifies the largest integer.</p><pre>scala&gt; val max: (Int, Int) =&gt; Int = (m: Int, n: Int) =&gt; if(m &gt; n) m else n<br>max: (Int, Int) =&gt; Int = &lt;function2&lt;<br>scala&gt; max(3,4)<br>res8: Int = 4</pre><p><strong>First steps with Spark: Data Abstraction</strong></p><p>Every Spark application consists of a driver that performs a main function written by the user and also performs several operations in parallel in the cluster. Each of these operations is performed on a data collection, represented internally in Spark by a data abstraction. For now, we will focus on the basic Data Abstraction that Spark provides: the Resilient Distributed Dataset (RDD).</p><p>AnRDD is the first abstraction provided by Spark for data manipulation. It is a representation of a data distributed by the cluster nodes that can be operated in parallel. RDDs can be created from files in HDFS or Scala language collections.</p><p>There are two ways to create a RDD: (1) by parallelizing an existing collection within the program, or (2) referencing data on a system external to the program (HDFS, HBase, or any data source with a Hadoop InputFormat). Let’s start spark-shell. The spark context will be available by the variable ‘sc’.</p><p><strong>RDDs from an existing collection</strong></p><p>To parallelize a collection you must invoke the Spark Context’s parallelize method over an existing collection.</p><pre>scala&gt; val data = Array(1, 2, 3, 4, 5)<br>scala&gt; val distData = sc.parallelize(data)<br>distData: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[21] at parallelize at :29</pre><p>You can also use the makeRDD function to create an RDD.</p><pre>scala&gt; val data = Array(1, 2, 3, 4, 5)<br>scala&gt; val dataRDD = sc.makeRDD(data)<br>dataRDD: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[20] at makeRDD at :29</pre><p><em>There is no difference between makeRDD and parallelize. The makeRDD function invokes parallelize. You can check the invocation of parallelize directly in the implementation of makeRDD.</em></p><p>An important parameter for the parallel collection is the number of partitions on which the data will be divided. Spark will perform a Task for each partition in the cluster. Usually divided into 2 to 4 partitions per cluster CPU. Spark attempts to configure the number of partitions according to its cluster, but you can configure this number manually by passing the number of partitions as the second argument to parallelize.</p><p><strong>RDDs from external data</strong></p><p>Spark can create RDDs from any data source supported by Hadoop, including its local file system, HDFS, Cassandra, HBase, Amazon S3, among others. Spark supports any format supported by Hadoop InputFormat.</p><p>RDDs can be created from text files using the textFile method of SparkContext. This method takes the URI of the file (which can either be a relative path on the local file system, or a full path of an external system such as hdfs: // and s3n: //) and reads the contents of the file as a collection of lines. Let’s read the Spark readme file inside the spark root folder (you need to start the shell from root folder to be able to execute the following code).</p><pre>scala&gt; val distFile = sc.textFile(“README.md”)<br>distFile: RDD[String] = MappedRDD@1d4cee08</pre><p>Once the RDD is created, distFile can be transformed with operations called dataset operations. For example, we can add all line sizes using map and reduce operations.</p><pre>scala&gt; distFile.map(s =&gt; s.length).reduce((a, b) =&gt; a + b)</pre><p><strong>Notes: about reading files with Spark.</strong></p><ul><li>If you are reading a local file system file, this file must be accessible with the same path on all nodes. Either the same file is copied to all nodes, or the file is shared via a file system distributed as NFS.</li><li>All methods for reading files, such as textFile, support reading directories, compressed files, and regular expressions (Unix-like). For example, you can use: textFile (“/ dir”), textFile (“/ dir / *. Txt”), and textFile (“/ dir / * .gz”).</li><li>The textFile method has an additional parameter to control the number of file partitions. By default, Spark creates a partition for each block of the file (64MB, standard HDFS size), but you can configure Spark to use a larger number of partitions. You can not have fewer blocks.</li><li>In addition to text files, Spark supports several other formats:</li></ul><p>The SparkContext.wholeTextFiles method reads a directory containing several text files and returns a vector with the (file-name, file-content) tuple.</p><p>To read other formats you can use the SparkContext.hadoopRDD method. This method reads the InputFormat class from a given jobConf. InputFormats can be included into Scala project as simple maven dependencies.</p><p><strong>Saving files</strong></p><p>To save a RDD to a file you can use the saveAsObjectFile method, which saves the file using Java serialization, or saveAsTextFile which saves the file in plain text.</p><pre>scala&gt; val file = sc.textFile(“README.md”)<br>file: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[10] at textFile at :27</pre><pre>scala&gt; file.count<br>res1: Long = 95</pre><pre>scala&gt; file.saveAsObjectFile(“/tmp/readme_object”)<br>scala&gt; file.saveAsTextFile(“/tmp/readme_text”)</pre><p>The saveAsObjectFile method saves the files using Java serialization. The partitions are converted to a stream of bytes, where this stream can later be reverted to the original file.</p><pre>spark@spark-hare-1.cs.uni-paderborn.de:spark-training$ cat /tmp/readme_object/part-00000<br>SEQ!org.apache.hadoop.io.NullWritable”org.apache.hadoop.io.BytesWritable<br>[Ljava.lang.String;]{Gxpt# Apache SparkttNSpark is a fast and general cluster computing system for Big Data. It providestKhigh-level APIs in Scala, Java, Python, and R, and an</pre><p>The saveAsTextFile method saves all partitions in plain text.</p><pre>spark@spark-hare-1.cs.uni-paderborn.de:spark-training$ cat /tmp/readme_text/part-00000<br>Spark is a fast and general cluster computing system for Big Data. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a</pre><p>Note that in both methods the file is saved on several partitions (e.g., part-0000).</p><pre>spark@spark-hare-1.cs.uni-paderborn.de:spark-training$ ls /tmp/readme_object/<br>_SUCCESS part-00000 part-00001</pre><pre>spark@spark-hare-1.cs.uni-paderborn.de:spark-training$ ls /tmp/readme_text/<br>_SUCCESS part-00000 part-00001</pre><p>To retrieve the files you can read the entire contents of the directory using textFile. Note that the number of rows is the same, because the files are the same.</p><pre>scala&gt; val f = sc.textFile(“/tmp/readme_text/*”)<br>f: org.apache.spark.rdd.RDD[String] = /tmp/readme_text/ MapPartitionsRDD[10] at textFile at :27</pre><pre>scala&gt; f.count<br>res28: Long = 95</pre><pre>scala&gt; val f = sc.objectFile(“/tmp/readme_object/*”)<br>f: org.apache.spark.rdd.RDD[Nothing] = MapPartitionsRDD[7] at objectFile at :27</pre><pre>scala&gt; f.count<br>res4: Long = 95</pre><p><strong>Word Count</strong></p><p>Let’s create a simple word count using the basic operations of a RDD.</p><pre>val text_file = sc.textFile(“README.md”)<br>val counts = text_file.flatMap(lambda line: line.split(“ “)) \<br> .map(lambda word: (word, 1)) \<br> .reduceByKey(lambda a, b: a + b)<br>counts.saveAsTextFile(“count.txt”)</pre><p>The flatMap function transforms an RDD of length N into a collection of N collections, then flattens these into a single RDD of results. Map transforms an RDD of length N into another RDD of length N, in a key-value pair, initializing 1 to every word stored as key. Finally the reduceByKey function will add value to every duplicated key.</p><p><strong>Conclusions</strong></p><p>In this post, we give an introduction to simple data processing in Spark shell. . In future posts, we will advance in further features of spark for Analytics and Machine Learning. A good further reading about Spark is the books <a href="https://www.apress.com/de/book/9781484209653">Mohammed Guller’s Big Data Analytics with Spark</a> and <a href="https://www.manning.com/books/spark-in-action">Spark in Action</a>. Those two books covers a lot of all Spark’s modules and very useful to use as a reference. Feel free to comment below any questions and suggestions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5034ef6c785" width="1" height="1" alt=""><hr><p><a href="https://medium.com/an-introduction-to-big-data-processing-with-spark/an-introduction-to-big-data-processing-with-spark-a5034ef6c785">An Introduction to Big Data Processing With Spark</a> was originally published in <a href="https://medium.com/an-introduction-to-big-data-processing-with-spark">Big Data Processing With Spark</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>