<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Isham Mohamed on Medium]]></title>
        <description><![CDATA[Stories by Isham Mohamed on Medium]]></description>
        <link>https://medium.com/@isham?source=rss-54de9e9d0daf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*QTDrD2X6u54PRGzE.</url>
            <title>Stories by Isham Mohamed on Medium</title>
            <link>https://medium.com/@isham?source=rss-54de9e9d0daf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:54:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@isham/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An Abstract Understanding of Spark]]></title>
            <link>https://medium.com/@isham/an-abstract-understanding-of-spark-220d5a43f152?source=rss-54de9e9d0daf------2</link>
            <guid isPermaLink="false">https://medium.com/p/220d5a43f152</guid>
            <category><![CDATA[rdd]]></category>
            <category><![CDATA[lineage]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[gda]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Isham Mohamed]]></dc:creator>
            <pubDate>Tue, 16 Jan 2018 20:44:10 GMT</pubDate>
            <atom:updated>2019-03-18T04:15:07.849Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*Gxmi4VsWvG3L_1AooaFXwA.jpeg" /></figure><p>If you ever came across anything related to data analysis you must have heard of spark. You might have even used it. So what makes the spark better than MapReduce? How it becomes more resilient than Hadoop? What makes it “lighting” fast? In this article, I’ll give you an abstract overall picture of spark.</p><p>Apache Spark is an open-source general-purpose cluster-computing framework written in Scala, a general-purpose programming language providing support for functional programming. It provides real-time stream processing, interactive processing, graph processing, in-memory processing as well as batch processing with very fast speed, ease of use and standard interface Even though written in scala, spark provides a rich API in Java, R, and Python too. Spark can be used with HDFS, Cassandra, HBase, Hive, Tachyon as the data sources and can be run in Standalone, YARN, and Mesos cluster manager. Spark introduced an abstraction called RDD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aFBq3ic_QVEomsi1haExXQ.jpeg" /></figure><p>Resilient Distributed Dataset (RDD) is the fundamental data structure of Spark, a read-only collection of objects partitioned across a set of machines that can be rebuilt if a partition is lost. RDD is one of the key concepts that make spark more efficient and resilient. In Spark, data is stored as RDD, which is set of immutable collection of Scala objects distributed among multiple clusters. Hence the computations can be done in individual clusters, this speed ups the operations are done in spark by providing some kind of parallelism and they are fault tolerant. There are three ways to create RDD, which we will discuss in future discussions. RDDs are immutable, which means they can’t be altered once they are created. They can only be read by the interface provided by Scala since they are fundamentally Scala objects, which makes the data consistency and easy to share among clusters. RDD stores intermediate results in the memory of the clusters instead of the disk. This makes the operations even faster. The writing of RDD is always a Coarse-grained Operation instead fine-grained operation.</p><p>RDDs supports 2 types of operation, Transformations, and Actions. A “Transformation” take an RDD as an input and produces one or more RDDs. They are basically map and filter kind of operations. Transformation never alters the input RDDs since they are immutable and always produces RDDs differ from the input since spark is a cyclic data flow model(?). An “Action” is basically an RDD operation which produces non-RDD data, such as count, or aggregations. Since we have discussed RDDs, Transformations, and Action, we can discuss the next fundamental concept of Spark, DAG aka RDD Lineage.</p><p>A Directed Acyclic Graphs (DAG) is a set of vertices and edges where vertices represent RDDs and edges represents a Transformation or an Actions. DAG is submitted to DAG Scheduler which further splits the graph into the stages of the task. Whenever the user defines an action to be done with an RDD, it creates a new edge with a resultant RDD as the ending node. This is how the whole DAG is formed.This leads us to an important discussion, Lazy Evaluation, and Fault Tolerant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*d-c-a1vjpO-dAolb0jQ8Pg.jpeg" /></figure><p>The fundamental of lazy evaluation is to load the data whenever it is needed only. you might think we always need the data when we do any operation to an RDD. But we only need an output whenever we perform an Action. For example, we are not interested in the whole RDD which is a result from a map or a filter function, instead we are only interested in the results from an Action such as aggregated results or a selection result. So Spark only need to load the data when there is an Action to be performed on an RDD. This is the concept of lazy loading. Science spark represents every action in the DAG it only loads and processes the data whenever it encounters with an Action. Since the whole action is represented in this DAG, whenever there is a node failure the spark only have to process the DAG up to the failure level from the last available parent RDD of the specific cluster only. This gives a huge advantage in fault tolerance.</p><p>So there is no one magic recipe which makes the spark what it is. It’s collection of techniques used in a proper manner which leads the spark a better general purpose computing framework. The goal of the article is not to make you master of the spark, but to give an overall idea of the concepts you may come across in the sparky journey you are gonna begin. Most of the information and Images were obtained from <a href="https://data-flair.training/">data-flair</a>. I strongly recommend you to go ahead and follow the course about spark to understand furthermore. Thank you.</p><p><strong>References</strong></p><p>Matei Zaharia , Mosharaf Chowdhury , Michael J. Franklin , Scott Shenker , Ion Stoica, “Spark: cluster computing with working sets”, Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, p.10–10, June 22–25, 2010, Boston, MA</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=220d5a43f152" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[R for Machine Learning]]></title>
            <link>https://medium.com/@isham/https-medium-com-isham-r-for-machine-learning-ffa7ae37901f?source=rss-54de9e9d0daf------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffa7ae37901f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[installation]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[rstudio]]></category>
            <category><![CDATA[ubuntu]]></category>
            <dc:creator><![CDATA[Isham Mohamed]]></dc:creator>
            <pubDate>Sun, 23 Jul 2017 17:12:10 GMT</pubDate>
            <atom:updated>2017-07-24T05:23:08.041Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*G2k_zSPJDf_D2fmTK2Wmvw.png" /></figure><p>I’m sure you have come across the word “Machine Learning” a million times by now unless you have been living under the locks for past twenty years and the R plays a major role in machine learning.I just got started with R and machine learning. As I learn I’ll continue on writing about them. R is a statistical programming software favored by many academia and due to the rise in data science and availability of libraries and being open source, the industry has started using R.</p><p>We will continue on learning about Machine Learning libraries and R codes in this series, but before getting there we have a problem to worry about, that is setting R and Rstudio on you Ubuntu machine. There are many resources for on installing R on Windows but I found it hard to install them on my Ubuntu machine.</p><pre><strong>Installing R:</strong></pre><pre>1: sudo apt-key adv –keyserver keyserver.ubuntu.com –recv-keys E084DAB9</pre><pre>2: sudo add-apt-repository &#39;deb https://ftp.ussg.iu.edu/CRAN/bin/linux/ubuntu xenial/&#39;</pre><pre>3: sudo apt-get update</pre><pre>4: sudo apt-get install r-basesudo apt-get install r-base-dev</pre><pre><strong>Installing Rstudio:</strong></pre><pre>5: sudo apt-get install gdebi-core</pre><pre>6: wget https://download1.rstudio.org/rstudio-1.0.44-amd64.debsudo </pre><pre>7: gdebi rstudio-1.0.44-amd64.deb</pre><pre>8: rm rstudio-1.0.44-amd64.deb</pre><p>In case of a installation fail u may want to change the link in step 2. <a href="https://cran.r-project.org/bin/linux/ubuntu/">Follow this link</a> to get newer version.</p><p>By the end of step 8 you would have the setup done. To check the R version enter the following command in your terminal.(the R has to be in capital)</p><pre>R --version</pre><p>In case if you want to update the R base enter the following commands in your terminal.</p><pre>sudo add-apt-repository ppa:marutter/rrutter</pre><pre>sudo apt-get update</pre><pre>sudo apt full-upgrade</pre><p>Will be continued…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffa7ae37901f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>