<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vinodh Rhaj on Medium]]></title>
        <description><![CDATA[Stories by Vinodh Rhaj on Medium]]></description>
        <link>https://medium.com/@vinodhrhaj?source=rss-af9303938a58------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Vinodh Rhaj on Medium</title>
            <link>https://medium.com/@vinodhrhaj?source=rss-af9303938a58------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:51:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vinodhrhaj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Transformation in R with dplyr]]></title>
            <link>https://vinodhrhaj.medium.com/data-transformation-in-r-with-dplyr-7059bc1b3020?source=rss-af9303938a58------2</link>
            <guid isPermaLink="false">https://medium.com/p/7059bc1b3020</guid>
            <category><![CDATA[etl-pipeline]]></category>
            <category><![CDATA[dplyr]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[r]]></category>
            <dc:creator><![CDATA[Vinodh Rhaj]]></dc:creator>
            <pubDate>Sat, 06 Nov 2021 02:43:28 GMT</pubDate>
            <atom:updated>2021-11-06T02:48:19.602Z</atom:updated>
            <content:encoded><![CDATA[<p>To answer pressing business questions, we need to perform ETL operations on the given data. In most of the scenarios the given data need to be cleansed and transformed in addition to ingestion process. Here we’ll summarize key transformation techniques that are handy using R dplyr package which is inclusive as part of tidyverse library in R. We’ll be using built in Newyork Flights data for learning purpose.</p><p>Pre- Requisites:</p><p>Instalal R, Install R -Studio</p><p>Load the data sets using below :</p><p><strong>library</strong>(“nycflights13”)</p><p><strong>library</strong>(“tidyverse”)</p><p>5 useful function within dplyr package:</p><p><strong>· filter() </strong>— Pick Observations by their values</p><p>· <strong>arrange()</strong> — Reorder the rows</p><p>· <strong>select() </strong>— Variable selection</p><p>· <strong>mutate() </strong>— Create new variables using existing variables</p><p>· <strong>summarize() </strong>— Collapse values to provide summary</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*BsYkSrXgHeaT5f7jvo7JMw.png" /></figure><h3><strong>filter()</strong></h3><p>The filter operations can be used alongside with Comparison and Logical operator. The filter() operation in combination with demorgan’s law would be a good idea. For example in the given dataset ‘flights’ , finding flights with an arrival delay of 120 or more minutes is as below</p><pre>delayed_filghts<strong> &lt;- filter</strong>(flights, arr_delay &gt;= 120)</pre><p>In the above example we use logical Operation greater than or equal to to filter filter with arrival delay 2 hours. The same filter operation can be applied with demorgan’s law as follows</p><pre>delayed_filghts2<strong> &lt;- filter</strong>(flights, !(arr_delay &lt; 120))</pre><h3><strong>arrange()</strong></h3><p>You can use this function to rearrange your data columns. To arrange the flights data set with year,month, day you can apply function as below:</p><p>order_filghts <strong>&lt;- arrange</strong>(flights, year,month,day)</p><pre>arrange(flights, year, month, day)<br>#&gt; # A tibble: 336,776 x 19<br>#&gt;    year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time<br>#&gt;   &lt;int&gt; &lt;int&gt; &lt;int&gt;    &lt;int&gt;          &lt;int&gt;     &lt;dbl&gt;    &lt;int&gt;          &lt;int&gt;<br>#&gt; 1  2013     1     1      517            515         2      830            819<br>#&gt; 2  2013     1     1      533            529         4      850            830<br>#&gt; 3  2013     1     1      542            540         2      923            850<br>#&gt; 4  2013     1     1      544            545        -1     1004           1022<br>#&gt; 5  2013     1     1      554            600        -6      812            837<br>#&gt; 6  2013     1     1      554            558        -4      740            728<br>#&gt; # … with 336,770 more rows, and 11 more variables: arr_delay &lt;dbl&gt;,&gt; #   carrier &lt;chr&gt;, flight &lt;int&gt;, tailnum &lt;chr&gt;, origin &lt;chr&gt;, dest &lt;chr&gt;,<br>#&gt; #   air_time &lt;dbl&gt;, distance &lt;dbl&gt;, hour &lt;dbl&gt;, minute &lt;dbl&gt;, time_hour &lt;dttm&gt;</pre><p>You can apply descending order for columns as below:</p><p>Order_flights &lt;- filter(flights, desc(arr_delay))</p><h3><strong>select()</strong></h3><p>The select() function is similar to SQL Select statement . You can select columns needed from your data. This is particularly useful if our data ha more columns. We can use helper functions like starts_with,contains, matches, etc., within select.</p><p>For e.g. Using select with contains can be written with Regex operations as below,</p><pre><strong>select</strong>(flights, <strong>matches</strong>(&quot;^(dep|arr)_(time|delay)$&quot;))</pre><pre><em>#&gt; # A tibble: 336,776 x 4<br>#&gt;   dep_time dep_delay arr_time arr_delay<br>#&gt;      &lt;int&gt;     &lt;dbl&gt;    &lt;int&gt;     &lt;dbl&gt;</em></pre><pre><em>#&gt; 1      517         2      830        11<br>#&gt; 2      533         4      850        20<br>#&gt; 3      542         2      923        33<br>#&gt; 4      544        -1     1004       -18<br>#&gt; 5      554        -6      812       -25<br>#&gt; 6      554        -4      740        12<br>#&gt; # … with 336,770 more rows</em></pre><h3><strong>mutate()</strong></h3><p>mutate() function adds new variable at the end of the dataset. We can transform existing variables to create new variables using this function.</p><p>For e.g. we can select records from flights data and mutate them to create new variables as below</p><pre>flights_sml &lt;- select(flights,year:day,ends_with(&quot;delay&quot;),distance,air_time)<br>mutate(flights_sml,gain = dep_delay - arr_delay,speed = distance / air_time * 60)</pre><pre>#&gt; # A tibble: 336,776 x 9<br>#&gt;    year month   day dep_delay arr_delay distance air_time  gain speed<br>#&gt;   &lt;int&gt; &lt;int&gt; &lt;int&gt;     &lt;dbl&gt;     &lt;dbl&gt;    &lt;dbl&gt;    &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;<br>#&gt; 1  2013     1     1         2        11     1400      227    -9  370.<br>#&gt; 2  2013     1     1         4        20     1416      227   -16  374.<br>#&gt; 3  2013     1     1         2        33     1089      160   -31  408.<br>#&gt; 4  2013     1     1        -1       -18     1576      183    17  517.<br>#&gt; 5  2013     1     1        -6       -25      762      116    19  394.<br>#&gt; 6  2013     1     1        -4        12      719      150   -16  288.<br>#&gt; # … with 336,770 more rows</pre><p>Note that you can refer to columns that you’ve just created:</p><pre>mutate(flights_sml,gain = dep_delay - arr_delay,hours = air_time / 60,gain_per_hour = gain / hours)</pre><pre>#&gt; # A tibble: 336,776 x 10<br>#&gt;    year month   day dep_delay arr_delay distance air_time  gain hours<br>#&gt;   &lt;int&gt; &lt;int&gt; &lt;int&gt;     &lt;dbl&gt;     &lt;dbl&gt;    &lt;dbl&gt;    &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;</pre><pre>#&gt; 1  2013     1     1         2        11     1400      227    -9  3.78<br>#&gt; 2  2013     1     1         4        20     1416      227   -16  3.78<br>#&gt; 3  2013     1     1         2        33     1089      160   -31  2.67<br>#&gt; 4  2013     1     1        -1       -18     1576      183    17  3.05<br>#&gt; 5  2013     1     1        -6       -25      762      116    19  1.93<br>#&gt; 6  2013     1     1        -4        12      719      150   -16  2.5<br>#&gt; # … with 336,770 more rows, and 1 more variable: gain_per_hour &lt;dbl&gt;</pre><p><strong>transmute()</strong> is a function used to keep your new variables.</p><pre>transmute(flights,<br>gain = dep_delay - arr_delay,<br>hours = air_time / 60,<br>gain_per_hour = gain / hours)</pre><pre>#&gt; # A tibble: 336,776 x 3<br>#&gt;    gain hours gain_per_hour<br>#&gt;   &lt;dbl&gt; &lt;dbl&gt;         &lt;dbl&gt;<br>#&gt; 1    -9  3.78         -2.38<br>#&gt; 2   -16  3.78         -4.23<br>#&gt; 3   -31  2.67        -11.6<br>#&gt; 4    17  3.05          5.57<br>#&gt; 5    19  1.93          9.83<br>#&gt; 6   -16  2.5          -6.4</pre><pre>#&gt; # … with 336,770 more rows</pre><h3><strong>summarize()</strong></h3><p>As the name, used to summarize the data, as popular SQL approaches we use groupby and aggregate functions along with summarize . For example,</p><pre>delays &lt;- flights %&gt;%<br>group_by(dest) %&gt;%<br>summarise(<br>count = n(),<br>dist = <a href="https://rdrr.io/r/base/mean.html">mean</a>(distance, na.rm = TRUE),<br>delay = <a href="https://rdrr.io/r/base/mean.html">mean</a>(arr_delay, na.rm = TRUE)<br>) %&gt;%<br>f<a href="https://rdrr.io/r/stats/filter.html">ilter</a>(count &gt; 20, dest != &quot;HNL&quot;)</pre><p>Usage of pipe (%&gt;%) is to tie stages of operation before using summarize()</p><p>In Addition to above functions, you have arithmetic operators, aggregate functions like count(),rank(),cumin(), offset functions like lag(),lead() to perform advanced data transformations in R.</p><p>Reference:</p><p>“R for Data Science by Hadley Wickham and Garrett Grolemund(o’Reily). Copyright 2017 Garrett Grolemund, Hadley Wickham, 978-1-491-91039-9.”</p><p><a href="https://r4ds.had.co.nz/transform.html">https://r4ds.had.co.nz/transform.html</a></p><p><a href="https://jrnold.github.io/r4ds-exercise-solutions/transform.html#grouped-summaries-with-summarise">https://jrnold.github.io/r4ds-exercise-solutions/transform.html#grouped-summaries-with-summarise</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7059bc1b3020" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Spark  — Introduction]]></title>
            <link>https://vinodhrhaj.medium.com/learning-spark-introduction-92c1878cce2b?source=rss-af9303938a58------2</link>
            <guid isPermaLink="false">https://medium.com/p/92c1878cce2b</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[learning-to-code]]></category>
            <dc:creator><![CDATA[Vinodh Rhaj]]></dc:creator>
            <pubDate>Tue, 27 Oct 2020 03:31:13 GMT</pubDate>
            <atom:updated>2020-10-27T03:49:37.671Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learning Spark — Introduction</h3><p>When traditional storage systems and essential programming found it challenging to handle huge volume of data. Google found GFS (Google File system), MapReduce and Big table, that handled hardware, storage and parallel programming challenges.</p><p>The computational challenges and solutions in GFS, lead to HDFS and MapReduce as a framework at Yahoo, which was then donated to Apache Software Foundation. When MapReduce, HDFS with Hadoop Common and Hadoop YARN became Hadoop, it had widespread adoption outside Yahoo by large open source community. Hadoop was able to handle large scale jobs for general batch processing , it was not suitable for machine learning, streaming and interactive SQL queries.</p><p>Researches at UC Berkeley acknowledged the inefficiencies in MapReduce and the complex learning curve for beginners in MapReduce. They developed ‘Spark’ with parts from MapReduce and further <strong><em>enhanced to support — in memory storage </em></strong><em>for interactive map and reduce computation.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*FfX6mm2o8Axzxdfe6Ivg6Q.jpeg" /></figure><p>Spark is designed to make large scale distributed data processing, <em>easier, simpler and faster. </em>In Spark, intermediate results are processed in memory that limits disk I/O and drives <strong><em>speed</em></strong>. Spark offers RDD, which is a fundamental logical data structure upon which other data abstractions are constructed (e.g. spark dataframes). This makes it <strong><em>easy to use</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oSEk2_ttrpFDLkXGLP_t0Q.jpeg" /><figcaption>Apache Spark Ecosystem</figcaption></figure><p>Spark is a unified processing engine that can support multiple programming language like Scala, Java, Python, SQL and R. It offers unified libraries like SparkSQL, MLlib, GraphX etc. Spark can read data from different data sources like Apache Hive, HBase, RDBMS and cloud sources like Amazon S3/Azure Storage etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*2j-1fHesgjdRZPvAuyTQmw.jpeg" /><figcaption>Apache Spark Ecosystem of connectors</figcaption></figure><p>In Spark Architecture, the core Apache spark components are,</p><ol><li>Spark Application</li><li>Spark Driver</li><li>Spark Session</li><li>Cluster Manager</li><li>Spark Executors</li></ol><p><strong><em>Spark Application</em></strong> has a Spark driver program which can orchestrate parallel operations on Spark Cluster. The driver access Spark components through Spark Session.</p><p><strong><em>Spark Driver </em></strong>is<strong><em> </em></strong>responsible for communication with cluster manager, it requests resources from cluster manager for spark executors, transforms spark operations into DAG computations, schedules and distributes task.</p><p><strong><em>Spark Session</em></strong> consolidates all spark operations and data, it can streamline multiple entry points like Spark Context, Hive Context, SQL Context, Spark Conf and Streaming Context.</p><p><strong><em>Cluster Manager </em></strong>manages and allocates resources for cluster of nodes that runs spark application. It supports Standalone , Apache Hadoop YARN, Apache Mesos and Kubernetes.</p><p><strong><em>Spark Executor </em></strong>runs on each worker node in the cluster. They communicate with Spark Driver and executes tasks on the workers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rU7H7b25rKz53c-7nxITkw.jpeg" /><figcaption>Apache Spark components and architecture</figcaption></figure><p>Apache Spark’s easy to use APIs suits data of all shapes and sizes across Scala, Java, Python, SQL and R. Undoubtedly, Apache Spark is developer’s delight.</p><p>Sources:</p><p>“Learning Spark, 2nd Edition, by Jules S. Damji, Brooke Wenig, Tathagata Das and Denny Lee. Copyright 2020 Databricks, Inc., 978–1–492–05004–9”</p><p>“<a href="https://databricks.com/p/ebook/learning-spark-from-oreilly">https://databricks.com/p/ebook/learning-spark-from-oreilly</a>”</p><p>“<a href="https://spark.apache.org/docs/latest/cluster-overview.html">https://spark.apache.org/docs/latest/cluster-overview.html</a>”</p><p>“<a href="https://databricks.com/spark/about">https://databricks.com/spark/about</a>”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92c1878cce2b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>