<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aruna Singh on Medium]]></title>
        <description><![CDATA[Stories by Aruna Singh on Medium]]></description>
        <link>https://medium.com/@datasciencelogs?source=rss-b1488f986f7f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*c320CqxUJ0wXKJFxC1NDJw.jpeg</url>
            <title>Stories by Aruna Singh on Medium</title>
            <link>https://medium.com/@datasciencelogs?source=rss-b1488f986f7f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 19:55:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@datasciencelogs/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Consume s3 data to Redshift via AWS Glue]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/analytics-vidhya/consume-s3-data-to-redshift-via-aws-glue-505fad72c622?source=rss-b1488f986f7f------2"><img src="https://cdn-images-1.medium.com/max/1826/1*bAuZa-b73eWBvJVO6y7Fjw.png" width="1826"></a></p><p class="medium-feed-snippet">In this article, we will discover how to build an ETL pipeline by consuming data from S3 to AWS Redshift via the Glue service and&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/analytics-vidhya/consume-s3-data-to-redshift-via-aws-glue-505fad72c622?source=rss-b1488f986f7f------2">Continue reading on Analytics Vidhya »</a></p></div>]]></description>
            <link>https://medium.com/analytics-vidhya/consume-s3-data-to-redshift-via-aws-glue-505fad72c622?source=rss-b1488f986f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/505fad72c622</guid>
            <category><![CDATA[redshift]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[s3]]></category>
            <dc:creator><![CDATA[Aruna Singh]]></dc:creator>
            <pubDate>Sun, 26 Feb 2023 06:11:34 GMT</pubDate>
            <atom:updated>2023-03-01T08:18:20.501Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Machine Learning with PySpark MLlib]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://datasciencelogs.medium.com/machine-learning-with-pyspark-mllib-6f0feec7f1a3?source=rss-b1488f986f7f------2"><img src="https://cdn-images-1.medium.com/max/1278/1*G4cHTJFdwpk4d9Q6iK2CMQ.png" width="1278"></a></p><p class="medium-feed-snippet">In the last article, you learned about PySpark SQL and how to interact with it using DataFrame API and SQL query. In this article, you&#x2019;ll&#x2026;</p><p class="medium-feed-link"><a href="https://datasciencelogs.medium.com/machine-learning-with-pyspark-mllib-6f0feec7f1a3?source=rss-b1488f986f7f------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://datasciencelogs.medium.com/machine-learning-with-pyspark-mllib-6f0feec7f1a3?source=rss-b1488f986f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f0feec7f1a3</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Aruna Singh]]></dc:creator>
            <pubDate>Mon, 04 Oct 2021 08:58:33 GMT</pubDate>
            <atom:updated>2021-10-05T17:07:05.950Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[PySpark SQL and DataFrames]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.datadriveninvestor.com/pyspark-sql-and-dataframes-4c821615eafe?source=rss-b1488f986f7f------2"><img src="https://cdn-images-1.medium.com/max/1785/1*0-RcYP2bywNaZFVWmcagmQ.jpeg" width="1785"></a></p><p class="medium-feed-snippet">In the previous article, we covered Fundamentals of BIG DATA with PySpark. In this article, we will explore PySpark SQL which is Spark&#x2019;s&#x2026;</p><p class="medium-feed-link"><a href="https://medium.datadriveninvestor.com/pyspark-sql-and-dataframes-4c821615eafe?source=rss-b1488f986f7f------2">Continue reading on DataDrivenInvestor »</a></p></div>]]></description>
            <link>https://medium.datadriveninvestor.com/pyspark-sql-and-dataframes-4c821615eafe?source=rss-b1488f986f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c821615eafe</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Aruna Singh]]></dc:creator>
            <pubDate>Sat, 12 Jun 2021 23:41:41 GMT</pubDate>
            <atom:updated>2021-07-07T16:19:33.203Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Fundamentals of BIG DATA with PySpark]]></title>
            <link>https://datasciencelogs.medium.com/big-data-fundamentals-with-pyspark-939112ef2281?source=rss-b1488f986f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/939112ef2281</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Aruna Singh]]></dc:creator>
            <pubDate>Fri, 04 Jun 2021 21:02:05 GMT</pubDate>
            <atom:updated>2021-06-13T03:57:48.377Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Tpw9vFpxzGb7eanXvE2TQ.png" /><figcaption><a href="https://www.edureka.co/blog/big-data-analytics/">https://www.edureka.co/blog/big-data-analytics/</a></figcaption></figure><blockquote>This article introduces the exciting world of Big Data, as well as the various concepts and different frameworks for processing Big Data. You will understand why Apache Spark is considered the best framework for BigData</blockquote><h3><strong>Big Data concepts and Terminology</strong></h3><p>What exactly is Big Data? It is a term which refers to the study and applications of data sets that are too complex for traditional data-processing software.</p><p>There are three Vs of Big data that are used to describe its characteristics: <strong>Volume</strong> refers to the size of data, <strong>Variety</strong> refers to different sources and formats of data and <strong>Velocity</strong> is the speed at which data is generated and available for processing</p><p><em>Now, let’s take a look at some of the concepts and terminology of Big Data</em></p><blockquote><strong>Clustered computing </strong>is the pooling of resources of multiple machines to complete jobs</blockquote><blockquote><strong>Parallel computing </strong>is a type of computation in which many calculations are carried out simultaneously</blockquote><blockquote><strong>Distributed computing</strong> involves nodes or networked computers that run jobs in parallel</blockquote><blockquote><strong>Batch processing</strong> refers to the breaking data into smaller pieces and running each piece on an individual machine</blockquote><blockquote><strong>Real-time processing</strong> demands that information is processed and made ready immediately</blockquote><h3>Big Data Processing Systems</h3><p><strong>Hadoop/MapReduce : </strong>An open source and scalable framework for batch data.</p><p><strong>Apache Spark: </strong>It is also open source and is suited for both batch and real-time data processing. It is a fast and general-purpose framework for Big data processing. Apache Spark provides high-level APIs in Scala, Java, Python, and R. It runs most computations in memory and thereby provides better performance for applications such as interactive data mining.</p><p>It is a powerful alternative to Hadoop MapReduce, with rich features like machine learning, real-time stream processing, and graph computations. At the center of the ecosystem is the Spark Core which contains the basic functionality of Spark. The rest of Spark’s libraries are built on top of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KsAaANasQRWb0PEW_jHCqQ.png" /></figure><blockquote>Spark runs on two modes. The first is the local mode where you can run Spark on a single machine such as your laptop. It is very convenient for testing, debugging and demonstration purposes. The second is the cluster mode where Spark is run on a cluster. The cluster mode is mainly used for production.</blockquote><h3><strong>Spark’s version of Python: PySpark</strong></h3><p>Apache Spark is originally written in Scala programming language. To support Python with Spark, PySpark was developed with similar computation power as Scala. APIs in PySpark are similar to Pandas &amp; Scikit-learn python packages.</p><p>Spark comes with an interactive python shell in which PySpark is already installed in it. It is particularly helpful for fast interactive prototyping before running the jobs on clusters. Unlike most other shells, Spark shell allows you to interact with data that is distributed on disk or in memory across many machines, and Spark takes care of automatically distributing this processing.</p><p>Spark provides the shell in three programming languages: spark-shell for Scala, PySpark for Python and sparkR for R. PySpark. Similar to Scala Shell, Pyspark shell has been augmented to support connecting to a cluster.</p><p>In PySpark Shell , A SparkContext represents the entry point to Spark functionality. PySpark automatically creates a SparkContext for you in the PySpark shell (so you don&#39;t have to create it by yourself) and is exposed via a variable sc. You can access the SparkContext in the PySpark shell as a variable named sc. It’s like a key to your car. Without the key you cannot enter the house, similarly, without an entry point, you cannot run any PySpark jobs.</p><p>Now, let’s take a look at some of the important attributes of SparkContext.</p><blockquote><strong>sc.version</strong> shows the version of spark that you are currently running</blockquote><blockquote><strong>sc.pythonVer </strong>shows the version of Python that Spark is currently using.</blockquote><blockquote><strong>sc.master</strong> shows the URL of the cluster or “local” string to run in local mode.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0fRlionhyxK6yrMjsCbEUw.png" /></figure><p>The easiest way to demonstrate the power of PySpark’s shell is to start using it. Let’s take an example of a simple list containing numbers ranging from 1 to 100 in the PySpark shell.</p><p>The most important thing to understand here is that we are not creating any SparkContext object because PySpark automatically creates the SparkContext object named sc, by default in the PySpark shell.</p><p><strong>You can load your raw data into PySpark using SparkContext by two different methods which we will be discussed later:</strong></p><ol><li>SparkContext’s parallelize method</li><li>SparkContext’s textFile method</li></ol><h3><strong>Use of Lambda Function in </strong>Python</h3><p>Python supports the creation of anonymous functions. That is functions that are not bound to a name at runtime, using a construct called the lambda. It is used in conjunction with typical functional concepts like map and filter functions. Like def, the lambda creates a function to be called later in the program. Let’s look at some of its uses:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XdCt-ff7IgqE7qhINXfbBA.png" /></figure><p><strong><em>Use of Lambda function in python — map():</em></strong><em> The map function is called with all the items in the list and a new list is returned which contains items returned by that function for each item.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ntWipwXPSG2UhLLALaCyJg.png" /></figure><p><strong><em>Use of Lambda function in python — filter(): </em></strong><em>The function is called with all the items in the list and a new list is returned which contains items for which the function evaluates to True.</em></p><h3>Introduction to PySpark RDD</h3><p>It is simply a collection of data distributed across the cluster. RDD is the fundamental and backbone data type in PySpark.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m3gvxdeqDGQ3ZM6JTlbGVw.png" /></figure><p>Now, let’s take a look at the different features of RDD. The name RDD captures 3 important properties:</p><blockquote><strong>Resilient</strong>, which means the ability to withstand failures and recompute missing or damaged partitions.</blockquote><blockquote><strong>Distributed</strong>, which means spanning the jobs across multiple nodes in the cluster for efficient computation.</blockquote><blockquote><strong>Datasets</strong>, which is a collection of partitioned data e.g. Arrays, Tables, Tuples or other objects.</blockquote><p>There are three different methods for creating RDDs, out of them You have already seen two methods which is being mentioned before.</p><p><strong>SparkContext’s parallelize method:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k-1KJ1GayUH5eZx8ISCwcA.png" /></figure><p><strong>From external datasets (</strong>SparkContext’s textFile method<strong>):</strong></p><p>Files stored in HDFS or objects in Amazon S3 buckets or from lines in a text file stored locally and pass it to SparkContext’s textFile method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_u6e0XZjNWdhgqBOKK2fcw.png" /></figure><p><strong>From existing RDDs </strong>(mutating RDDs)<strong>:</strong></p><p>This transformation is the way to create an RDD from already existing RDD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrGoVBdwe7kClcnl6XFvMQ.png" /></figure><h3>Partitioning in PySpark</h3><p>Data partitioning is an important concept in Spark and understanding how Spark deals with partitions allow one to control parallelism. A partition in Spark is the division of the large dataset with each part being stored in multiple locations across the cluster. By default, Spark partitions the data at the time of creating RDD based on several factors such as available resources, external datasets etc, however, this behavior can be controlled by passing a second argument called minPartitions which defines the minimum number of partitions to be created for an RDD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6z7gPz0OzIYcjSSzc3A0UQ.png" /></figure><h3><strong>Introduction to RDDs in PySpark</strong></h3><p>There are the various operations that support RDDs in PySpark. RDDs in PySpark supports two different types of operations — Transformations and Actions.</p><blockquote>Transformations are operations on RDDs that return a new RDD.</blockquote><blockquote>Actions are operations that perform some computation on the RDD.</blockquote><p><strong>Lazy evaluation :</strong> The most important feature which helps RDDs in fault tolerance and optimizing resource use. Spark creates a graph from all the operations you perform on an RDD and execution of the graph starts only when an action is performed on RDD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ARnuCVc4nk-udXAyoZpfNA.png" /></figure><h4>Transformations on <strong>RDDs :</strong></h4><p><em>The </em><strong><em>map() </em></strong><em>takes in a function and applies it to each element in the RDD.</em></p><p><em>The </em><strong><em>filter()</em></strong><em> takes in a function and returns an RDD that only has elements that pass the condition.</em></p><p><em>The</em><strong><em> flatMap()</em></strong><em> is similar to map transformation except it returns multiple values for each element in the source RDD.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dICwgKyUhxiSXsipuxzKbQ.png" /></figure><p><em>The </em><strong><em>union()</em></strong><em> returns the union of one RDD with another RDD.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mvy81Fe3j1UDIWNhCf2eng.png" /></figure><h4>Actions<strong> on RDDs:</strong></h4><p><em>The </em><strong><em>collect() </em></strong><em>action returns complete list of elements from the RDD.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cCigaJPokrNgkL6k_GtLpQ.png" /></figure><p><em>The </em><strong><em>take()</em></strong><em> action print an ’N’ number of elements from the RDD.</em></p><p><em>The </em><strong><em>count()</em></strong><em> action returns the total number of rows/elements in the RDD</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kkPmZkzm8s2vgAydlLS8_Q.png" /></figure><p><em>The </em><strong><em>first() </em></strong><em>action returns the first element in an RDD</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WVyM2q7Bj55VVzPHSL2qdQ.png" /></figure><h3>Introduction to pair RDDs in PySpark</h3><p>Real world datasets are generally key/value pairs. Each row is a key that maps to one or more values. In order to deal with this kind of dataset, PySpark provides a special data structure called pair RDDs. In pair RDDs, the key refers to the identifier, whereas value refers to the data.</p><p><strong>There two most common ways of creating pair RDDs are as follows:</strong></p><blockquote>The first step in creating pair RDDs is to get the data into key/value form.</blockquote><blockquote>Next, we create a pair RDD using map function which returns tuple with key/value pairs with key being the name and age being the value.</blockquote><h4>Transformations on pair RDDs :</h4><p><strong><em>reduceByKey:</em></strong><em> It is the most popular pair RDD transformation which combines values with the same key using a function. reduceByKey runs several parallel operations, one for each key in the dataset, returns a new RDD consisting of each key and the reduced value for that key.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IktR6g1HT24TEgomgN1r4A.png" /></figure><p><strong><em>sortByKey: </em></strong><em>We can sort pair RDD as long as there is an ordering defined in the key and returns an RDD sorted by key in ascending or descending order.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ioCpdoVUtvYMgt2_dthG9g.png" /></figure><p><strong><em>groupByKey: </em></strong><em>it groups all the values with the same key in the pair RDD</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TTIoD9diy9gLsL3iv7ol1g.png" /></figure><p><strong><em>join transformation: </em></strong><em>Applying join transformation merge two pair RDDs together by grouping elements based on the same key.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BNK_uvNqcJ9d_rEwtl5YZw.png" /></figure><h4><strong>Advanced Actions on pair RDDs:</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*Z3W4liOvaYihIY4E4GJhog.png" /></figure><p><strong><em>Reduce() </em></strong><em>operates on two elements of the same type of RDD and returns a new element of the same type. The function should be commutative and associative so that it can be computed correctly in parallel.</em></p><h4><em>In many cases, it is not advisable to run collect action on RDDs because of the huge size of the data. In these cases, it’s common to write data out to a distributed storage systems.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hr1pFiXvZ8lRqiOqcm76Jg.png" /></figure><p><strong><em>saveAsTextFile()</em></strong><em> saves RDD with each partition as a separate file inside a directory by default. However, you can change it to return a new RDD that is reduced into a single partition using the coalesce method.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yVs6crCshRpZ8V1tnwcHlg.png" /></figure><p><strong><em>countByKey() </em></strong><em>is only available on RDDs of type (Key, Value). With the countByKey operation, we can count the number of elements for each key. One thing to note is that countByKey should only be used on a dataset whose size is small enough to fit in memory.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1QJ19edom30PdgxjEL4Klw.png" /></figure><p><strong><em>collectAsMap()</em></strong><em> returns the key-value pairs in the RDD to the as a dictionary. Like SparkContext’s parallelize method, collectAsMap produces the key-value pairs in the RDD as a dictionary which can be used for downstream analysis. Similar to countByKey, this action should only be used if the resulting data is expected to be small, as all the data is loaded into the memory.</em></p><p><strong>Hence, we have covered the overview of Big Data Fundamentals with PySpark, and in the process learned some useful syntax for RDD Transformations and actions. There is another article which includes SQL, Dataframes and machine learning with MLib. To understand it, you can click on this </strong><a href="https://datasciencelogs.medium.com/pyspark-sql-and-dataframes-4c821615eafe"><strong>link</strong></a><strong>.</strong></p><p><strong>Stay Connected and Enjoy Reading !</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=939112ef2281" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Dive into the Tree Based Models]]></title>
            <link>https://medium.datadriveninvestor.com/deep-dive-into-the-tree-based-models-707eedc0477b?source=rss-b1488f986f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/707eedc0477b</guid>
            <category><![CDATA[regression-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[classifcation-models]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Aruna Singh]]></dc:creator>
            <pubDate>Wed, 03 Mar 2021 21:29:33 GMT</pubDate>
            <atom:updated>2021-07-05T19:22:56.372Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sd4ECe28S7_enP7suLN86w.png" /></figure><blockquote>It is majorly used in finance, investing, and other disciplines that attempts to determine the strength and character of the relationship between one dependent variable (usually denoted by Y) and a series of other variables (known as independent variables). Tree-based models use a series of if-then rules to generate predictions from one or more decision trees. All tree-based models can be used for either regression (predicting numerical values) or classification (predicting categorical values). Hence, We’ll explore five different types of tree-based models.</blockquote><h4><strong>Supervised Learning and Unsupervised Learning</strong></h4><p>Supervised learning is the subfield of machine learning in which you train a model using input data and corresponding labels. In supervised learning, each example is a pair consisting of the input data and an output value which represents a category or label in the case of classification, or a numeric value in the case of regression. The converse is called unsupervised learning, where you learn from the input data alone.</p><h3><strong>Tree Based Models</strong></h3><p>Decision tree-based models, including tree-based ensemble models such as Random Forests and Gradient Boosting Machines (or GBMs). Tree-based models stand out from other types of machine learning models due to their unique combination of model interpretability, ease-of-use, and, when used in ensembles, excellent accuracy. They are used to make decisions, explore the data and make predictions. In this article, you will interpret and explore different use cases, build and evaluate classification and regression models tune model parameters for optimal performance.</p><p><a href="https://www.datadriveninvestor.com/2020/05/18/how-could-algorithms-put-individuals-and-communities-in-harms-way/">How could algorithms put individuals and communities in harm&#39;s way? | DataDrivenInvestor</a></p><p><strong>Introduction to Classification Tree</strong></p><p>A classification<em> </em>tree is a decision tree that performs a classification (vs regression) task.<strong> </strong>It is a hierarchical structure with nodes and directed edges. The node at the top is called the root node. The nodes at the bottom are called the leaf nodes or terminal nodes. Nodes that are neither the root node or the leaf nodes are called internal nodes. The root and internal nodes have binary test conditions associated with them and each leaf node has an associated class label.</p><blockquote><em>“rpart” is short for recursive partitioning which is a process used for training both classification and regression trees.</em></blockquote><p>Let’s train a decision tree model to understand which loan applications are at higher risk of default using a subset of the <a href="https://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29"><strong>German Credit Dataset</strong></a>. The response variable, called “default”, indicates <em>whether the loan went into a default or not</em>, which means this is a binary classification problem. Both categorical and continuous predictors are used for binary classification.</p><pre># Look at the data<br>str(creditsub)</pre><pre># Create the model<br>credit_model &lt;- rpart(formula = default ~ .,data = creditsub, method = &quot;class&quot;)</pre><pre># Display the results<br>rpart.plot(x = credit_model, yesno = 2, type = 0, extra = 0)</pre><ul><li>In R, formulas are used to model the response as a function of some set of predictors, here is <strong>default~.</strong>, which means use all columns (except the response column) as predictors.</li><li>In the rpart() function, note that you&#39;ll also have to provide the training data frame.</li><li>Using the model object that you create, plot the decision tree model using the rpart.plot() function from the rpart.plot package.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8yZiCX5rqVBR-P8tTPgkvQ.png" /><figcaption><strong>Output</strong>: structure of credit dataset</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gBdW2eOqSDyHNfr0llyb4A.png" /><figcaption>rpart.plot- Decision Tree</figcaption></figure><p>This tree predicts whether a loan applicant will default on their loan (or not). After following the path, if you will end up in a “Yes” leaf means that the model predicts that this individual <em>will default</em> on their loan, where as a “No” prediction means that they <em>will not default</em> on their loan.</p><p>Assume we have a loan applicant who:</p><ul><li>is applying for a 20-month loan</li><li>is requesting a loan amount that is 2% of their income</li><li>is 25 years old</li></ul><p><strong>Answer would be Yes, the person will default on their loan.</strong></p><p>Let’s move on to the modeling. Now, we’ll keep discussing about the restaurant dataset to create a classification tree to make decisions.</p><p>First, we will split the data into 2 pieces. The first part of the data, is called training set, can be used for building the model and the second part of the data, the test set, can be used to test the results. One common way of doing this is to use 70% of the data for a training set and 30% of the data for the test set. Of course, there can be a lot of variation possible. One way to reduce this variation is by using cross-validation.</p><pre># Total number of rows in the credit data frame<br>n &lt;- nrow(credit)</pre><pre># Number of rows for the training set (70% of the dataset)<br>n_train &lt;- round( 0.7 * n)</pre><pre># Create a vector of indices which is an 70% random sample<br>set.seed(123)<br>train_indices &lt;- sample(1:n, n_train)</pre><pre># Subset the credit data frame to training indices only<br>credit_train &lt;- credit[train_indices, ]  <br>  <br># Exclude the training indices to create the test set<br>credit_test &lt;- credit[-train_indices, ]</pre><h4>Train a classification tree model</h4><pre># Train the model (to predict &#39;default&#39;)<br>credit_model &lt;- rpart(formula = default ~ ., <br>                      data = credit_train, <br>                      method = &quot;class&quot;)</pre><pre># Look at the model output                      <br>print(credit_model)</pre><p>At this point, you have a classification tree model ready. The next steps would be to make a prediction and evaluate your predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ms9GIuhpmf-hl1YK4BHZxQ.png" /><figcaption><strong>Output</strong>: Credit Model</figcaption></figure><p>To begin with, The rpart package has a predict function where the first argument is the trained model and the second argument is the test dataset. It consists the type argument, which controls whether the function returns<strong> </strong>predicted labels or the raw predicted values. In classification problems, the model generates a raw predicted value for each class. When type equals “class”, the function will return a predicted class label and when type equals “prob”, a matrix of raw predicted values is returned instead, with one column for each class.</p><p>There are many ways to evaluate classification performance. Accuracy, confusion matrix, log-loss, and Area under the ROC Curve (or AUC) are some of the most popular metrics for binary classification problems. For now, we will introduce the confusion matrix and accuracy.</p><p><strong>Confusion Matrix: </strong>The columns of a confusion matrix correspond to the truth labels, and the rows represent the predictions. In the 2-class case, a single prediction has four different possible outcomes: The true positives and true negatives are correct classifications. A false positive is when the outcome is incorrectly predicted as positive when it is actually negative. A false negative is when the outcome is incorrectly predicted as negative when it is actually positive, the same concept can be extended to any number of classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*12QjOxXZbjDJ17QuASAmdQ.png" /></figure><p><strong>Accuracy: </strong>It measures how often the classifier predicts the class correctly. It is defined as the ratio between the number of correct predictions and the total number rows or data points in the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IcWBYxR8dF7Nlx7WlVreGw.png" /></figure><p><em>Choose </em><strong><em>Recall</em></strong><em> if the occurrence of </em><strong><em>false negatives is unaccepted/intolerable. </em></strong><em>Recall gives us an idea about when it’s actually yes, how often does it predict yes.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TclBZMUcqsfa-9EYjvyukg.png" /></figure><p><em>Choose </em><strong><em>Precision </em></strong><em>if you want to be more </em><strong><em>confident of your true positives</em></strong><em>. Precision tells us about when it predicts yes, how often is it correct.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*loFsUQDW9b0WEV8xJT_6wg.png" /></figure><p><em>Choose</em><strong><em> Specificity </em></strong><em>if you want to </em><strong><em>cover all true negatives, </em></strong><em>i.e. meaning we do not want any false alarms or false positives.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4O089b4vipO1bXdndToeaw.png" /></figure><p>To create a confusion matrix in R, we will use the confusionMatrix function from the caret package. The arguments you need to specify are: the data and the reference. The data argument is a vector of predicted class labels on a test set and the reference is a vector of the true class labels.</p><pre># Generate predicted classes using the model object<br>class_prediction &lt;- predict(object = credit_model,  <br>                        newdata = credit_test,   <br>                        type = &quot;class&quot;)  <br>                            <br># Calculate the confusion matrix for the test set<br>confusionMatrix(data = class_prediction,       <br>                reference = credit_test$default)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JIcTLlU4jvk7R7bvHtkcvA.png" /><figcaption><strong>Output</strong>: Confusion Matrix</figcaption></figure><p>We can conclude that:</p><ul><li>Accuracy value of 64% means that identification of 72 of every 200 default loan predicted values is incorrect, and 128 is correct.</li><li>Precision value of 86% means that label of 28 of every 200 default loan values is a not default, and 172 are default.</li><li>Recall value is 50% means that 100 of every 200 default loan , in reality, are missed by our model and 100 are correctly identified as defaults.</li><li>Specificity value is 86% means that 28 of every 200 default loans in reality are miss-labeled as default and 172 are correctly labeled.</li></ul><h4>Splitting criterion in trees</h4><p>A classification tree uses a split condition to predict class labels based on one or more input variables. The classification process starts from the root node of the tree and at each node, the process will check whether the input value should recursively continue to the right or left sub-branch according to the split condition. This process stops when meeting any leaf or terminal nodes.</p><p>The idea behind classification trees is to split the data into subsets where each subset belongs to only one class. This is accomplished by dividing the input space into “pure” regions, that is — regions with samples from only one class. With real data, completely pure regions may not be possible, so the decision tree will do the best it can to create regions that are as pure as possible. Boundaries separating these regions are called decision boundaries, and the decision tree model makes classification decisions based on these decision boundaries.The goal is to partition data at a node into subsets that are as pure as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Udn4H4-JgG_Ga8NWaV3WNQ.png" /></figure><p>In this example, the partition shown on the right results in more homogeneous subsets. Since these subsets contain more samples belonging to a single class than the resulting subsets shown on the left. So the partition on the right results in purer subsets and is the preferred partition.</p><p>Therefore, we need a way to measure the purity of a split in order to compare different ways to partition a set of data. It turns out that it works out better mathematically if we measure the impurity rather than the purity of a split. So the impurity measure of a node specifies how mixed the resulting subsets are.</p><p><strong>Gini Index:</strong> A common impurity measure used for determining the best split is the Gini Index. The lower the <strong>Gini Index</strong>, the higher the purity of the split. So, the decision tree will select the split that minimizes the Gini Index.</p><p>Besides the <strong>Gini</strong> <strong>Index</strong>, other impurity measures include: entropy (or information gain), and misclassification rate which we will discuss in the next article.</p><h4>Compare models with a different splitting criterion</h4><p>Train two models that use a different splitting criterion and use the validation set to choose a “best” model from this group. To do this you’ll use the parms argument of the rpart() function. This argument takes a named list that contains values of different parameters you can use to change how the model is trained. Set the parameter split to control the splitting criterion.</p><pre># Train a gini-based model, splitting the tree based on gini index.<br>credit_model1 &lt;- rpart(formula = default ~ ., <br>                       data = credit_train, <br>                       method = &quot;class&quot;,<br>                       parms = list(split = &quot;gini&quot;))</pre><pre># Train an information-based model, splitting the tree based on information index.<br>credit_model2 &lt;- rpart(formula = default ~ ., <br>                       data = credit_train, <br>                       method = &quot;class&quot;,<br>                       parms = list(split = &quot;information&quot;))</pre><pre># Generate predictions on the validation set using the gini model<br>pred1 &lt;- predict(object = credit_model1, <br>             newdata = credit_test,<br>             type = &quot;class&quot;)</pre><pre># Generate predictions on the validation set using the information model<br>pred2 &lt;- predict(object = credit_model2, <br>             newdata = credit_test,<br>             type = &quot;class&quot;)</pre><pre># Compare classification error<br>ce(actual = credit_test$default, <br>   predicted = pred1)<br>ce(actual = credit_test$default, <br>   predicted = pred2)</pre><p>Classification error is the fraction of incorrectly classified instances. Compute and compare the test set classification error of the two models by using the ce() function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6z3kBxjVM_Akfuc_IDtKyQ.png" /><figcaption><strong>Output: </strong>Comparison between GINI and INFORMATION Index</figcaption></figure><p>After getting the final result, it is concluded that the model predicted by using information based model is better than gini based model as the classification error is higher for the gini based model.</p><p>Hence, we have covered the overview of classification tree with R, and in the process learned some real insights from the german credit dataset. We have also learnt about accuracy and confusion matrix to understand more about the performance of the dataset.</p><p>Thanks for Reading and Stay Connected !</p><p><strong>Gain Access to Expert View — </strong><a href="https://datadriveninvestor.com/ddi-intel"><strong>Subscribe to DDI Intel</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=707eedc0477b" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/deep-dive-into-the-tree-based-models-707eedc0477b">Deep Dive into the Tree Based Models</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Experimentation with Tidyverse in R]]></title>
            <link>https://ai.plainenglish.io/experimentation-with-tidyverse-in-r-a52acfc15d3f?source=rss-b1488f986f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a52acfc15d3f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tidyverse]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <dc:creator><![CDATA[Aruna Singh]]></dc:creator>
            <pubDate>Thu, 21 Jan 2021 05:32:09 GMT</pubDate>
            <atom:updated>2021-03-05T02:01:18.350Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VNYNz7UhpA-VeDNgh1D--A.png" /></figure><p>In this article, you’ll learn the intertwined processes of data manipulation, extraction and visualization using the tools dplyr and ggplot2. You’ll learn to manipulate data by filtering, sorting, and summarizing a real dataset in order to answer exploratory questions. Henceforth, you’ll get a taste of the exploratory data analysis and the power of Tidyverse tools. If you have prior experience in R, you can continue with this article. Otherwise, I would recommend to have a glance in this tutorial <a href="https://rfordatascience.medium.com/">Quick Tutorial on R</a> for better understanding.</p><h3><strong>Data Wrangling</strong></h3><p>In this section, you’ll learn to do three things with a table: filter for particular observations, arrange the observations in the desired order and mutate to add or change a column. You’ll see how each of these steps allows you to answer questions about your data.</p><p>At every step, you will be analyzing the real dataset called gapminder. The gapminder tracks the economic and social indicators like gdp per capita, life expectancy of countries over time. To do this, install the package &amp; load it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*PRd6cBiSW5aiBQMO1cLjeQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*7bJoAwkfiZOKTEo6oczO-g.png" /><figcaption>Structure of the gapminder Data Set</figcaption></figure><h3><strong>Verbs in the dplyr package</strong></h3><ol><li><strong>filter() :</strong> To filter the subset of the observations based on a particular condition. It’s a common step in an analysis.</li></ol><blockquote><strong>Every time you access the verb, use pipe %&gt;% which gets feeded into the next step.</strong></blockquote><pre>library(dplyr)<br>library(gapminder)</pre><pre>gapminder %&gt;%<br>filter(year == 2007) <em>#gapminder dataset for the year 2007</em></pre><pre>gapminder %&gt;%<br>filter(year == 2007, country == &quot;United States&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*gNFVhn2AuLyfh1bjIpRA_g.png" /><figcaption>Filtered Data for year “2007”</figcaption></figure><p>2.<strong> arrange() : </strong>To. sort the data in the dataset in ascending and descending order.</p><pre>gapminder %&gt;%<br>  filter(year == 2007) %&gt;%<br>  arrange(desc(gdpPercap)) </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xZecXxH62o5Rt1_shAejJw.png" /><figcaption><em>Filter for 1957, then arrange in descending order of gdp per capita</em></figcaption></figure><p>3. <strong>mutate() : </strong>To change any variables or add a new variable in the dataset. Make sure to keep column name in one word.</p><pre>gapminder %&gt;%<br>  mutate(gdp = gdpPercap * pop)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KGIY0tUUgw33LSPhe__-jg.png" /><figcaption>Added gdp as the product of gdpPercap and population</figcaption></figure><p><strong>Combining Verbs: </strong>Let’s combine all three of the verbs you’ve learned just now, to find the countries with the highest life expectancy, in months, in the year 2007.</p><pre>gapminder %&gt;%<br>filter(year == 2007) %&gt;%<br>mutate(lifeExpMonths = 12 * lifeExp) %&gt;%<br>arrange(desc(lifeExpMonths))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qnPTOkL2ky2HhaRacf3_WQ.png" /></figure><h3>Data Visualization</h3><p>In this section, you’ll learn the essential skills of data visualization using the ggplot2 package, and you’ll see how the dplyr and ggplot2 packages work closely together to create informative graphs.</p><p>In particular, you are going to see how to create scatterplot that compares two variables on x and y axis.</p><p><strong>Variable Assignment: </strong>When you’re working with just that subset, it’s useful to save the filtered data, as a new data frame. To do this, you use the assignment operator. This is a less then and a minus sign, like an arrow facing to the left.</p><p>In this operation, you’re taking the gapminder dataset, filtering it for the observations from the year 2007, and then saving it as gapminder_2007. Now if you print the gapminder_2007 dataset, we can see that it’s another table. But this one has only 142 rows,for the year 2007.</p><pre>gapminder_2007 &lt;- gapminder %&gt;%<br>filter(year == 2007)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4KOQxXgD9w3nVlO96B75qQ.png" /></figure><p>Now that you’ve saved this variable, you can use it to create our visualization.</p><p><strong>Visualizing with ggplot2:</strong> Suppose you want to examine the relationship between a country’s wealth and its life expectancy. You could do this with a scatterplot comparing two variables in our gapminder dataset: GDP per capita on the X axis and life expectancy on the y-axis. You’ll be creating this plot using the ggplot2 package.</p><pre>library(ggplot2)</pre><pre>gapminder_1952 &lt;- gapminder %&gt;%<br>filter(year == 1952)</pre><pre>ggplot(gapminder_1952, aes(x = pop, y = lifeExp)) +<br>geom_point()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fRJzZ_QBgdPSaK_OdsHBBg.png" /><figcaption>ScatterPlot with pop on the x-axis and lifeExp on the y-axis</figcaption></figure><p>This plot communicates an interesting observation: higher income countries tend to have higher life expectancy. But there is a problem, however, a lot of countries get crammed into the leftmost part of the x-axis. This is because the distribution of GDP per capita spans several orders of magnitude, with some countries in the tens of thousands of dollars and others in the hundreds. When one of your axes has that kind of distribution, it’s useful to work with a logarithmic.</p><p><strong>Log scale: </strong>Change the existing scatter plot (code provided) to put the x-axis (representing population) on a log scale. A scale where each fixed distance represents a multiplication of the value. This is what the scatter plot looks like when x is on a log scale. This is the same data, but now each unit on the x-axis represents a change of 10 times the GDP.</p><pre>ggplot(gapminder_1952, aes(x = pop, y = lifeExp)) +<br>geom_point() + scale_x_log10()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rhR7mJ_81hMEZX5AnO7r9A.png" /><figcaption>Change this plot to put the x-axis on a log scale</figcaption></figure><p>Suppose you want to create a scatter plot with population on the x-axis and GDP per capita on the y-axis. Both population and GDP per-capita are better represented with log scales, since they vary over many orders of magnitude.</p><pre>ggplot(gapminder_1952, aes(x = pop, y = gdpPercap)) +<br>geom_point() + scale_x_log10() + scale_y_log10()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XNOwB6CN59LHapLrvHefWg.png" /><figcaption>Comparing pop and gdpPercap, with both axes on a log scale</figcaption></figure><p><strong>Adding size and color to the plot:</strong> Create a scatter plot communicating information about each country’s population, life expectancy, and continent. Now you’ll use the color and size of the points to communicate even more.</p><pre>ggplot(gapminder_1952, aes(x = pop, y = lifeExp, color = continent , size = gdpPercap)) + geom_point() + scale_x_log10()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xh3n2TiLzLQftMK5zJyCgA.png" /><figcaption>Add the color and size aesthetic to represent a country’s gdpPercap</figcaption></figure><p><strong>Faceting:</strong> It is used to divide a graph into subplots based on one of its variables, such as the continent.</p><pre>ggplot(gapminder, aes(x = gdpPercap, y = lifeExp), color = continent , size = pop) + geom_point() + scale_x_log10() + facet_wrap(~ year)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wrZn9aUBeKhVlQeaMILCtA.png" /><figcaption>Scatter plot comparing gdpPercap and lifeExp, with color representing continent and size representing population, faceted by year</figcaption></figure><h3><strong>Grouping and Summarizing</strong></h3><p>So far you’ve been answering questions about individual country-year pairs, but you may be interested in aggregations of the data, such as the average life expectancy of all countries within each year. Here you’ll understand “split-apply-combine” paradigm: split the data into groups, apply some analysis to each group, and then combine the results.</p><ol><li><strong>summarize() : </strong>To collapse each group into a single-row summary. It does this by applying an aggregating or summary function to each group.</li></ol><p>Let’s find the median life expectancy by using medium function within summarize function.</p><pre>gapminder %&gt;%<br>summarize(medianLifeExp = median(lifeExp))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*mIH7ePmlebmI5Gi3w9zYjA.png" /><figcaption>Median life expectancy</figcaption></figure><p>Rather than summarizing the entire dataset, you may want to find the median life expectancy for only one particular year. In this case, you can combine the summarize verb with filter. You can create multiple summaries at once with the summarize verb.</p><pre>gapminder %&gt;%<br>filter(year == 1957) %&gt;%<br>summarize(medianLifeExp = median(lifeExp))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*OBUcesRnOiR63NUwAKr-lw.png" /><figcaption>Median life expectancy for year 1957</figcaption></figure><blockquote><strong>In the last point, you’ve learnt to calculate the average life expectancy and the total population in the year 2007. What if we weren’t interested just in the average for the year 2007, but for each of the years in the dataset? You could rerun this code and change the year each time, but that’s very tedious. Instead, you can use the group_by verb, which tells dplyr to summarize within groups instead of summarizing the entire dataset.</strong></blockquote><p><strong>2. group_by() : </strong>It splits the data into groups. When the data is grouped then use summarize function.</p><p>Suppose you’re interested in the average life expectancy and the total population in 2007 within each continent. You can find this by first filtering for the year 2007, grouping by continent (instead of year), and then performing your summary.</p><pre>gapminder %&gt;%<br>  filter(year == 1957) %&gt;%<br>  group_by(continent) %&gt;%<br>  summarize(medianLifeExp=median(lifeExp), maxGdpPercap= max(gdpPercap))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YaTQrbHqHsPac7Tl_FnZFw.png" /></figure><h3><strong>Visualizing Summarized Data</strong></h3><p>Instead of viewing the summarized data as a table, let’s save it as an object called by_year, so you can visualize the data using ggplot2 package. You would construct the graph with the three steps of ggplot2: the data, which is by_year. The aesthetics, which puts year on the x-axis and total population on the y-axis. And the type of graph, which in this case is a scatter plot, represented by geom_point.</p><p>Here, you’ll examine the median GDP per capita instead, and see how the trend differs among continents.</p><pre>library(ggplot2) #import ggplot2</pre><pre># Summarize medianGdpPercap within each continent within each year: by_year_continent</pre><pre>by_year_continent &lt;- gapminder %&gt;%<br>group_by(continent, year) %&gt;%<br>summarize(medianGdpPercap = median(gdpPercap))</pre><pre>ggplot(by_year_continent, aes(x = year, y = medianGdpPercap, color = continent), expand_limits(y = 0)) + geom_point()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mc4BOJzBG9JJagnbx99fyQ.png" /><figcaption>Change in medianGdpPercap in each continent over time</figcaption></figure><p>Hence, we have covered the principles of transforming and visualizing data with R, and in the process learned some real insights from the Gapminder dataset. We have also learnt about ggplot2 to create more informative and customized data visualizations.</p><p>Thank you for reading the article and I am sure, it would be pretty helpful. Do let me know your inputs and follow me on LinkedIn or Twitter for more updates.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a52acfc15d3f" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/experimentation-with-tidyverse-in-r-a52acfc15d3f">Experimentation with Tidyverse in R</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intermediate R for  Beginner’s Expert]]></title>
            <link>https://ai.plainenglish.io/intermediate-r-for-beginners-expert-89ce9b20c518?source=rss-b1488f986f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/89ce9b20c518</guid>
            <dc:creator><![CDATA[Aruna Singh]]></dc:creator>
            <pubDate>Fri, 08 Jan 2021 18:10:58 GMT</pubDate>
            <atom:updated>2021-06-11T14:18:18.840Z</atom:updated>
            <content:encoded><![CDATA[<h3>Regular Expressions and Apply Functions — R for Data Science</h3><p>If you have acquired the expertise in the basics of R, you would find this tutorial quite helpful for enhancing few more concepts in R. I have tried to keep as simple as it could. So, Let’s dive into it.</p><h3><strong>Regular Expressions</strong></h3><p>A ‘<strong>regular expression</strong>’ is a pattern that describes a set of strings. Two types of <strong>regular</strong> expressions are used in <strong>R</strong>, extended <strong>regular</strong> expressions (the default) and Perl-like <strong>regular</strong> expressions used by perl = TRUE . There is also fixed = TRUE which can be considered to use a literal <strong>regular expression</strong>. Some of them are mentioned below:</p><h4><strong>grep() and grepl()</strong></h4><p>To search and matches to the argument pattern within each element of a character vector. They differ in the format and amount of detail in the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fd-pNr0ibUVFSizmvmuGAg.png" /></figure><p>For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F2E9JO0DEFzitsE2wMMmKQ.png" /></figure><h4><strong>sub() and gsub()</strong></h4><p>To perform the replacement of the first and all matches respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YYIMGNBoGBW0ifBFE-2VWg.png" /></figure><h3>Apply Functions</h3><p>Apply functions are a family of functions in base R which allow you to repetitively perform an action on multiple chunks of data like list or vector. An apply function is essentially a loop, but run faster than loops and often require less code.</p><h4>lapply, sapply, vapply, tapply and mapply</h4><p>They are all functions that will loop a function through data in a list or vector.</p><ol><li><strong>lapply:</strong> The output is equivalent to the list.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ugfjXtLVAUSP_tyNCj4KxQ.png" /></figure><p>If you have a custom function to incorporate in the lapply() then you need to pass the function name and argument value if required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GJGrudccQBHHt3IzhNfzzg.png" /></figure><p>To unlist the list retrieved from lapply function, do this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bpiWiP5t1Q6BE_p__Jfr5A.png" /></figure><p><strong>2. sapply(): </strong>It tries to simplify the list to an array. Look at the difference between lapply and sapply to understand better:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cjH1loHuJp4HylR5Bznn6w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ofJlzvHk16b1PfCPVk0Gw.png" /></figure><p><strong>3. vapply():</strong> Explicitly specify the output format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I6B_gkUF1OcChdtOjxpZ9g.png" /></figure><p><strong>Note: </strong>If you are trying to decide which of these three functions to use, because it is the simplest, I would suggest to use sapply if possible. If you do not want your results to be simplified to a vector, lapply should be used. If you want to specify the type of result you are expecting, use vapply.</p><p><strong>4. tapply()</strong></p><p>Sometimes you may want to perform the apply function on some data, but have it separated by factor. In that case, you should use tapply. Let’s take a look at the information for <strong>tapply</strong> .</p><p>The arguments for tapply are tapply(X, INDEX, FUN). The only new argument is INDEX, which is the factor you want to use to separate the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pK1mLnJ-glt4oc4CrUeaJw.png" /></figure><p>Now, let&#39;s use column 1 as the index and find the mean of column 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oP4qPZyrYN9VgelaFPhTjw.png" /></figure><p><strong>5. mapply() : </strong>Lastly, I would cover mapply(). The arguments for mapply are mapply(FUN, …, MoreArgs = NULL, SIMPLIFY = TRUE, USE.NAMES = TRUE). First you list the function, followed by the vectors you are using the rest of the arguments have default values so they don’t need to be changed for now. When you have a function that takes 2 arguments, the first vector goes into the first argument and the second vector goes into the second argument.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-pYnja0LFfzVwCPvxS2uSQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z2sRAQxH445fnw52YDqN0w.png" /></figure><p>Well, we have covered the regular expressions and apply functions in this tutorial. Do share your feedback, hope you find it useful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89ce9b20c518" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/intermediate-r-for-beginners-expert-89ce9b20c518">Intermediate R for  Beginner’s Expert</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SAS Viya: SAS Visual Analytics]]></title>
            <link>https://ai.plainenglish.io/sas-viya-sas-visual-analytics-ec87ecd63ced?source=rss-b1488f986f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec87ecd63ced</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sass]]></category>
            <dc:creator><![CDATA[Aruna Singh]]></dc:creator>
            <pubDate>Wed, 30 Dec 2020 01:43:17 GMT</pubDate>
            <atom:updated>2021-03-06T02:55:53.667Z</atom:updated>
            <content:encoded><![CDATA[<h3>SAS®<strong> Viya: </strong>SAS<strong> Visual Analytics</strong></h3><blockquote>In this course, you would understand some tasks for the demonstrations and practices in SAS Viya for Learners. SAS Viya is the latest extension of the SAS Platform. Due to the distributed nature of this flexible computing environment, results from different sessions will be similar but likely not identical.</blockquote><h3><strong>Introduction</strong></h3><p>SAS Visual Analytics coupled with the power of Cloud Analytic Services (CAS) enables you to spend less time preparing and accessing the data and more time discovering trends, developing insights, and creating visually stunning reports to showcase your data.</p><h4>1. Purpose and features of SAS Visual Analytics</h4><p>Visual Analytics uses SAS high-performance technologies to accelerate analytic computations, which helps you derive value from massive amounts of data. This gives you the power to solve difficult problems, improve business performance, predict future performance, and mitigate risk rapidly and confidently. You can use Visual Analytics to explore huge volumes of data very quickly to identify trends and patterns that you might not have noticed before. Then, you can take your analysis a step further by creating powerful statistical models on the patterns that you discovered (with Visual Statistics) and solve complex problems using machine learning algorithms (with Visual Data Mining and Machine Learning). After you identify those patterns and get a better feel for your data, you can create reports or dashboards to share with anyone, anywhere via the web or a mobile device.</p><blockquote>In traditional reporting, the output is well-defined up-front. That is, you know what you are looking at and what you need to convey to your audience.</blockquote><h4>2. SAS® Cloud Analytic Services (CAS)</h4><p>At the heart of SAS Viya is Cloud Analytic Services (CAS), a high-performance, in-memory analytics engine. CAS is designed to run in a single machine or multi-machine configuration. For both configurations, the CAS server uses multi-threaded algorithms to rapidly perform analytic processing on in-memory data of any size. In fact, the CAS server can continue processing data even when the memory capacity of the server is exceeded.</p><p><strong>Single Machine Configuration: </strong>All processing is completed on one node.</p><p><strong>Multi-Machine Configuration: </strong>It consists of one controller and one or more worker nodes and provides optimal processing capabilities. When performing analytics, CAS efficiently spreads big data processing across all nodes in the cluster, which results in very fast operations. In this configuration, CAS has a communications layer that supports fault tolerance, meaning it can continue processing requests even after losing connectivity to some nodes. It does this by distributing work to other nodes. This communications layer also enables you to dynamically delete and add nodes to the system while the server is running.</p><blockquote>In addition to using SAS Viya applications to access CAS, you can also use scripts via a command line, REST APIs, and third-party languages (such as Python, R, Lua, and Java).</blockquote><h4>3. SAS® Viya applications</h4><p>SAS Viya consists of many different applications that enable you to take advantage of the powerful CAS server.</p><ol><li><strong>SAS Drive:</strong> The entry point for SAS Viya, a collaborative interface for accessing, organizing, and sharing content. This application is the gateway to all other applications on the platform.</li><li><strong>SAS Data Studio: </strong>The data preparation application. Using SAS Data Studio, you can prepare CAS tables and create CAS tables using transforms.</li><li><strong>Visual Analytics:</strong> When your tables are ready for analysis, you can use them to visualize data, build statistical models, and create interactive reports. Reports that are created in Visual Analytics can be viewed in a browser using SAS Report Viewer or on a tablet or mobile device using the SAS Visual Analytics App.</li></ol><blockquote>SAS Theme Designer can be used to customize the look and feel of your applications and reports or use SAS Graph Builder to create custom graph objects. In addition, you can perform programming tasks using SAS Studio, view, reload and import data to CAS using SAS Data Explorer, manage the environment using SAS Environment Manager, and view and understand relationships between objects using SAS Lineage Viewer. In this instance, it’s not the data itself that is being passed to SAS Lineage, but details about the relationships between tables and objects.</blockquote><h4>4. SAS® Visual Analytics Methodology</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dto4qnK54rfNQCSs4cTrdw.png" /></figure><p>A step-by-step process that you can follow when using Visual Analytics. The methodology is divided into five phases: Access, Investigate, Prepare, Analyze, and Report.</p><ol><li><strong>Access:</strong> Identify the analysis tables that will be used in Visual Analytics and load those tables into CAS.</li><li><strong>Investigate:</strong> Inspect the tables to determine whether any changes are needed for data items due to data inconsistencies or data quality issues. In this phase, you also identify any new data items that need to be created.</li><li><strong>Prepare: </strong>Correct any data quality issues and create any new calculated items needed for analysis.</li><li><strong>Analyze: </strong>Explore the data to identify any patterns, relationships, and trends.</li><li><strong>Report: </strong>Develop interactive reports that can be shared via the web or a mobile device.</li></ol><blockquote>More often, you will step through the phases consecutively. However, it is possible to skip some phases or perform them simultaneously or out of order.</blockquote><h4>5. Different types of Visual Analytics users.</h4><p>There are five different job roles:</p><ol><li><strong>Analysts</strong>, like you, will be analyzing data and building reports, so you will use Visual Analytics.</li><li><strong>Data Scientists:</strong> Build statistical models to improve business performance, so they will use Visual Statistics and Visual Data Mining and Machine Learning, which can be accessed through Visual Analytics.</li><li><strong>Data administrators: </strong>Prepare the data for the analysts and can use SAS Data Studio to do so.</li><li><strong>Environment administrators:</strong> Manage the environment by creating folders, adding users, and making sure users can access only the content they need to perform their job. They will use the SAS Environment Manager to perform these functions.</li><li><strong>Information consumers: </strong>The users who will be viewing any reports you create. These can be executives and managers who want to better understand the current state of the organization, so they can plan for the future or it can be users with other job roles in the organization, like other analysts or data scientists. All users can view reports using the Report Viewer.</li></ol><h3>Preparing Data Using SAS® - Data Studio</h3><h4>1. Visual Analytics Methodology: Access</h4><p>In the Access phase of the Visual Analytics methodology, you identify the analysis tables that will be used in Visual Analytics and load those tables into CAS.To do so, you’ll need to discuss the intended uses for the tables with your analysts. Once you understand their goals, you can locate the tables and load them into CAS for immediate use.</p><p>Tables can be loaded into CAS in many different ways. The method you choose will depend on your role and the types of changes you need to make to the table before it is loaded.</p><p>If no changes need to be made to the table, an administrator could import it into CAS using SAS Environment Manager. Sometimes administrators configure a feature called self-service import, which enables analysts and data scientists to import the data themselves using Visual Analytics. With this feature, they can import many different types of data, including local files (like Excel spreadsheets, text files, SAS data sets, or data from the clipboard), server data (like Teradata, Oracle, Hadoop, or Impala), or social media data (like Facebook, Google Analytics, Twitter, or YouTube).</p><p>If data changes need to be made, however, other methods can be used. For example, a data administrator can use SAS Data Studio to create a CAS table after a series of transforms have been applied. In addition, any users can write code to modify a table and load the result into CAS. This code can be SAS code written in SAS Studio or SAS Enterprise Guide, or code written in any supported open languages, like Python, Lua, and Java.</p><h4><strong>2. Visual Analytics Methodology: Investigate</strong></h4><p>In the Investigate phase of the Visual Analytics methodology, you inspect the data. Specifically, you need to look at the detailed data to determine if there are any data changes that are needed. These data changes could be the result of data quality issues in the data or variables with a significant number of missing values. These could be signs of faulty data preparation and will need to be detected (and fixed) as early as possible. It also includes new data items that need to be created for the analysis. For example, if you want to analyze delivery times, but your table contains only delivery date and order date, you’ll need to calculate the number of days for delivery.</p><h4><strong>3. Visual Analytics Methodology: Prepare</strong></h4><p>In the Prepare phase of the Visual Analytics methodology, you create the tables needed for analysis.</p><p>This can be done by combining tables, correcting data quality issues, and by modifying existing or creating new data items.</p><p>If the data you need currently exists in multiple tables, you can combine them together in this phase. Tables can be combined horizontally (using joins) or vertically (using appends).</p><p>Also, in this phase, data quality issues can be corrected. These issues can be fixed by changing the case of columns, changing the data type or format, renaming columns, removing white space, or standardizing, parsing, or clustering the data. Some data quality transforms require an additional license. For more information about these transforms, see the Resource section of this lesson.</p><p>Additionally, you can also create new calculated columns in this phase by splitting, with an expression, or with custom code.</p><p>You can use SAS Data Studio to prepare tables for Visual Analytics by creating plans that apply the necessary transforms to the table. These plans can then be executed at any time or scheduled to run periodically to create your analysis tables.</p><h4>4. Data Types</h4><p>As part of the Investigate phase, you need to determine the contents of the analysis table; <strong>that is, the number of character, datetime, and numeric variables in the table.</strong></p><p>In Visual Analytics, character and datetime data items are treated as categories. These are data items whose distinct values can be used to group and aggregate measures. For example, if you have a Gender data item with six distinct values, your data might need to be cleaned before it can be used for analysis.</p><p>Numeric data items, however, are treated as measures. These are data items whose values can be used in computations. In Visual Analytics, you can see descriptive statistics for measures, including the minimum, maximum, standard deviation, skewness, and kurtosis. These descriptive statistics are useful for understanding the range of your measures and can be helpful in building models.</p><h4><strong>5. Tables and Automatic Charts</strong></h4><p>Another part of the Investigate phase is viewing your data to better understand the size, structure, and data values.</p><p>There are many report objects available for viewing and understanding your data, but perhaps the easiest object to use is the list table. The list table can be used to view the summary or detailed data. To view the data for the first time, It gives you an idea about the values for each column and the range of values you can expect to see. Viewing data in a list table is an easy way to spot obvious data quality issues.</p><p>Another great object for learning about your data is the <strong>crosstab</strong>. This object is especially useful for viewing distinct values for category data items and summary information for your measures. This can also be helpful in spotting data quality issues and for gaining a broader insight into your data. When you use a crosstab, it’s recommended that lower cardinality categories (that is, categories with few distinct values) be placed in the Columns role and higher cardinality categories (that is, categories with many distinct values) be placed in the Rows role. This enables users to quickly and easily read the table without a ton of scrolling.</p><p>Another way to get a quick view of your data is to use an <strong>automatic chart</strong>. It is a special type of report object that selects the chart type based on the data that is assigned to it. For example, if you assign a Geography data item and two measures to the automatic chart, you get a geo map where the bubble placement indicates the geographic location, the size of the bubble represents one measure, and the color of the bubble represents the other measure. Conversely, if you were to assign five measures to the automatic chart, you get a correlation matrix where the color of the cells indicates the linear relationship between those measures. This is a great chart type to use to compare data items but you’re not sure of the best way to do so. For more information about the types of charts produced by the automatic chart with specific data item assignments, see the Resource section of this lesson.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8_jOIhkku4Uf-bJrZdZjeg.png" /></figure><h3>Transforming Data Using SAS® Data Studio</h3><p>Let&#39;s have a look into Customer Data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4328Zh_7_L4QT0VTVUqm7g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x0sl3V3dDLc6GOBF436SOw.png" /></figure><h3>SAS® Data Studio</h3><p>SAS Data Studio is the data preparation application in SAS Viya. Using SAS Data Studio, you can view metrics about your tables and apply transforms to the tables by creating plans.</p><p>Before data can be prepared using SAS Data Studio, the table must first be loaded into CAS.</p><p>Then, you can view details about the table. You can see a sample of the columns and rows, generate a profile of the table to view metrics, and view metadata information.</p><p>The table sample gives you an idea about the individual data values and can be used to view changes applied to the table when transforms are added.</p><p>Profile metrics give you an idea about the structure of the table and can be used to identify data quality issues. Profiles can be run on the source table and on the table created from the plan. You can use these profiles to compare metrics over time to ensure that your data quality has improved. Some profile metrics require an additional license. For more information about these metrics, see the Resource section of this lesson.</p><p>Metadata information shows details about the columns, like the type, length, and format. This can be used to understand your data at a high level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vvnwMEv_Ox2YyXENRZacwA.png" /></figure><figure><img alt="In SAS, dates are stored as the number of days since January 1, 1960, so you can simply subtract Order_Date from Delivery_Dat" src="https://cdn-images-1.medium.com/max/1024/1*OAodDyZGknKboQscxABL7w.png" /><figcaption>In SAS, dates are stored as the number of days since January 1, 1960, so you can simply subtract Order_Date from Delivery_Date.</figcaption></figure><h3>Analyzing Data Using SAS® Visual Analytics</h3><blockquote>In this lesson, you learn how to analyze data using Visual Analytics. You’ll start by modifying data items for your analysis. Then, you’ll explore your data with charts and graphs to help you uncover hidden relationships. Next, you’ll create some data items that you need for the analysis and apply filters to focus the analysis on a specific group. Finally, you’ll add some analytics (like forecasting and fit lines) to discover trends and patterns among data items</blockquote><h4>Visual Analytics Methodology: Analyze</h4><p>In the Analyze phase of the Visual Analytics methodology, you explore the data to identify any patterns, relationships, and trends that might exist. During this phase, you might need to make changes to the data items for your analysis. This can consist of modifying data item properties (like the names, formats, and aggregations), creating new data items, and applying filters that are needed for the analysis. After meeting the specifications, you can begin to explore relationships using charts and graphs and discover trends and patterns with analytics. Lastly, you can create, test, and compare models on the patterns that you discovered using SAS Visual Statistics and SAS Visual Data Mining and Machine Learning.</p><h3>SAS Data Studio Versus SAS Visual Analytics</h3><p>Remember that SAS Data Studio uses a CAS table as input and creates a CAS table as output from the plan. Alternatively, SAS Visual Analytics uses a CAS table as input and creates a report that can be viewed in the Report Viewer or the SAS Visual Analytics app. Any changes that are made to data items in Visual Analytics are applied to the report only and do not affect the CAS table.</p><p>However, beginning with Visual Analytics 8.3, report data views can be created to save and apply data changes for a CAS table. A data view acts as a template for any data settings that are modified, including data property changes, data source filters, hierarchies, geography data items, and calculated items. These views can be applied to the data source when you create reports and can be shared with others. This means that you spend less time preparing the data and more time analyzing the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l4H2mG6I9CuyH2T0ITOX2g.png" /></figure><h3>Descriptive Graphs</h3><p>Typically, the first part of the Analyze phase is gaining a better understanding of your data items in broad terms, that is, understanding the ranges of values and totals for specific groups or subgroups.</p><p><strong>This can be accomplished with descriptive graphs (histograms, box plots, and bar charts).</strong></p><p><strong>Histograms</strong> can be used to view the distribution of a single measure. A histogram consists of a series of bars where the height represents the number of observations that fit in a specific range. Histograms can also be used to determine whether a measure is normally distributed, which can be very helpful in modeling.</p><p>Alternatively, <strong>box plots</strong> can be used to view information about the variability and extreme values of a measure for distinct values of a category data item. In the box plot, the size and location of the box indicate the range of values between the 25th and 75th percentile for that group. The diamond marker indicates the Mean value and the line inside the box indicates the Median value. You can also modify the options to display outliers in the plot. Outliers are data points whose distances from the interquartile range are more than 1.5 times the size of the interquartile range. The whiskers can indicate either the Minimum and Maximum values or the range of values that are outside the interquartile range, but close enough to not be considered outliers. Box plots are extremely useful for determining the differences between groups.</p><p>Next are the <strong>bar charts </strong>to compare the summarized data. It displays aggregated data for the distinct values of a category. Bar charts can be used to compare Nominal values (categories that have no particular order), Time series data, to show Rankings (like the top or bottom values), and to show Parts of a whole.</p><h3>Creating Data Items and Applying Filters</h3><h3>Calculated Items Versus Aggregated Measures</h3><p>In Visual Analytics, two different types of calculated items can be created: Calculated data items or Aggregated measures.</p><p><strong>Calculated Data Items: </strong>It is created by performing mathematical calculations on numeric values or by performing operations on DateTime data items or categories. All calculations are performed on unaggregated data. That is, the expression is evaluated for each row in the data source.</p><p>For example, you can create a new calculated item (New Salary) by multiplying Salary and Increase. For each row in the data source, the operation is evaluated, and the new data item is created. Then, when that data item is used in a graph (for example, with Gender) the values for New Salary are totaled for each gender.</p><p><strong>Aggregated Measures:</strong> It enables you to use aggregated values to calculate new data items. These data items are created by first aggregating values, and then by performing the operation. With aggregated measures, the calculation changes depending on other data items that are available in the graph. The dynamic nature of aggregated measures makes them extremely useful for analysis and reporting.</p><p>For example, you can create a new aggregated measure (Salary (Percent of total)) by dividing the Sum of the Salary for each group by the Sum of the Salary for all groups. In this instance, because Salary (Percent of total) is paired with Gender, salaries are first totaled for each gender. Next, salaries are totaled for all genders. Then, the operation is evaluated for each gender to show the contribution to total profit. You could take the same aggregated measure and pair it with the country to see the contribution of each country to total profit.</p><blockquote>Calculated items are created by performing operations on unaggregated data. Date, Hemisphere, and State Abbreviations can be created using unaggregated data. Aggregated measures are created by aggregating and then performing the operation. Gross Profit Margin, GDP Growth, and Number of Employees must aggregate the data before performing the operation.</blockquote><h3>Custom Categories</h3><p>Sometimes, you might need to create labels for groups of categories or measure data items. This can be accomplished with Calculated items. However, Custom categories can do the same thing but are a bit easier to create.</p><blockquote>When you create a Custom category from a measure data item, you can use intervals or distinct values to group the data. <br>When you create a Custom category from a category data item, you can use distinct values to group the data.</blockquote><h3>Filtering Data</h3><p>Many different types of filters can be created to subset data in Visual Analytics. In general, these filters are split between those that can be modified by only the Report Designer and Report Viewer as well. In this lesson, we focus on filters that can be modified by the Report Designer.</p><p>These filters can be classified into two types: Detail report filters and Post-aggregate report filters. Detail report filters look at the individual values for data items, but Post-aggregate report filters look at the aggregated, or total, values.</p><p><strong>Detail report filters </strong>can be split between those that are applied at a global level and those that are applied at a local level. At the global level, a Data source filter can be added to subset the entire report. This filter is applied to every object that uses the data source. It acts as a pre-filter by filtering the data before it is brought in to Visual Analytics. This can be seen in the updated cardinality values in the Data pane after the filter is applied. At the local level, Basic and Advanced report filters can be added to subset individual report objects. The Basic report filter subsets the object using a single data item and an equality condition, while the Advanced report filter subsets the object using any number of data items and operators in the same expression. Each of these local filters subsets the data using detail data or individual values for data items.</p><p><strong>Post-aggregate report filter </strong>subsets the report object using aggregated values, not individual values. These aggregated values are evaluated in the report object and are used to specify the Minimum and Maximum values in the selectable range.</p><blockquote>Filters can be very helpful to focus on anomalies or inconsistencies that are discovered during analysis.</blockquote><h3>Geography Graphs</h3><p>When location is a critical component of your analysis, geo maps can be very helpful. A geo map overlays data on a geographic map. Four different types of geo maps can be created in Visual Analytics: Coordinate, Region, Bubble, and Contour.</p><p>Coordinate maps (also known as dot distribution maps or dot density maps) help with detecting spatial patterns and understanding the distribution of data over a geographic region. These maps use clustered points to help reveal patterns.</p><p>Region maps (also known as choropleth maps) use colors to show variations by location. With Region maps, larger regions seem more emphasized than smaller ones, which can skew your analysis.</p><p>Geo region or Coordinate maps should be used only when there is an even distribution of values within each region.</p><p>Alternatively, geo Bubble maps can be used even when data is not evenly distributed. This is because Bubble maps can compare two measures. One is used for the size of the bubble and one is used for the color of the bubble. Bubble maps also have another advantage over Region maps. The bubble size helps with comparing regions without the size of the region itself causing distortions. Sometimes, however, the size of the bubble can overlap with other bubbles making the chart difficult to read.</p><p>Contour maps display shaded regions over a geographic region. These are best used to show very dense data.</p><h3>Analysis Graphs:</h3><h4>1. Bubble Plots and Treemaps</h4><p>As part of your analysis, you might be interested in viewing differences between groups. This can be accomplished with analysis graphs (like bubble plots and treemaps).</p><p>Bubble plots are useful for displaying three dimensions of data (horizontal location, vertical location, and size of bubble) for some group of category values. A bubble plot compares at least three measures by using markers of varying sizes in a scatter plot. The values of two measures determine the location of the bubble in the plot, and the value of the third measure determines the size of the bubble. Bubble plots can also be animated to show changes in data over time. For animated bubble plots, you want to focus on bubbles that act differently than the others. That is, bubbles that are stationary while others are moving or bubbles that are moving in a different direction from the rest of the group.</p><p>Conversely, treemaps can be used to display lots of information in a small amount of space. Treemaps can compare two measures for some group of category values. The value of the category is represented by tiles, and measures can be added to both size and color of the tiles. Typically, the size and color are used to draw attention to areas of interest (for example, top contributors). The measures that are used should mean something when they are compared, but they should not be the same.</p><h4>2. Relationship Plots</h4><p>As part of your analysis, you might be interested in viewing relationships between measures. This can be accomplished with relationship plots (like a correlation matrix, a scatter plot, or a heat map).</p><p>A correlation matrix can be used to evaluate the Linear relationship between multiple measures. It displays the degree of correlation as a matrix of rectangular cells. Each cell represents the intersection of two measures, and the color of the cell indicates the degree of correlation between those measures. The correlation values are calculated by using Pearson’s correlation coefficient. They are identified as Weak (if the absolute value of the correlation is 0.3 or lower), Moderate (if the absolute value of the correlation is greater than 0.3 and less than or equal to 0.6), or Strong (if the absolute value of the correlation is greater than 0.6). Positive correlation values indicate that as one measure increases, so does the other, whereas Negative correlation values indicate that as one measure increases, the other decreases. Although two measures seem to have a weak relationship, this does not mean that they have no relationship. Remember that the correlation measures the Linear relationship between measures. The actual relationship could be more complex.</p><p>These non-linear relationships can best be displayed with a scatter plot or a heat map. Both can be used to visualize trends between measures and to pinpoint possible outliers. The scatter plot works best with low-cardinality measures and the heat map works best with high-cardinality measures.</p><p>Scatter plots can be used to evaluate the relationship between measures by displaying the values of two measures using markers.</p><p>Heat maps can be used to evaluate the relationship between two high-cardinality measures, between two categories, or between a category and a measure. A heat map displays the distribution of values for two data items by using a table with colored cells.</p><h4>3. Time Plots</h4><p>As part of your analysis, you might be interested in viewing relationships over time. This can be accomplished with time plots (like a time series plot or a line chart).</p><p>A time series plot can be used to show trends in measures over time by displaying data using a line that connects data values. Time series plots are excellent for identifying seasonal peaks or troughs in your data like forecasting models.</p><p>Line charts can also be used to show trends in measures over time but can be used for any ordinal values (like age groups or loyalty levels). This can be helpful for identifying patterns among or between ordinal groups.</p><h3>Designing Reports with SAS® Visual Analytics</h3><h3><strong>1. Creating Simple Reports:</strong></h3><p>In the Report phase of the Visual Analytics methodology, you need to create reports that are based on your discoveries. Reports should be organized, easy to navigate, versatile, and attractive. You can organize content by adding multiple pages to your reports. Your reports should focus on telling a single data story. Pages can then be used to tell parts of that story (like chapters in a book). You can make your reports easy to navigate by adding descriptions to define the purpose and instructions to describe the features of your report. You can make your reports versatile (or interactive) by adding animations, ranks, prompts, actions, and links. Finally, you can add styles and display rules to make your reports more attractive.</p><p><strong>Keep these tips in mind when you are designing reports.</strong></p><ul><li>Understand the audience.</li><li>Make accessible to all.</li><li>Tell a single data story.</li><li>Use visually appealing, easy to understand objects.</li><li>Use the simplest graph.</li><li>Use consistent fonts.</li><li>Limit the number of objects.</li><li>Limit the number of pages.</li></ul><h4>Reporting Graphs</h4><p>During the Report phase, you want to use graphs that convey information to your audience in a pleasing and clear fashion. One way to accomplish this could be with pie charts, donut charts, or word clouds.</p><p>A pie chart is best used to compare a few groups whose values vary greatly. These charts display the relative contribution of each group to the whole using a circle that is divided into multiple slices. Pie charts are popular, but be aware that it is very difficult to compare the relative sizes of slices in the chart. Because of this, the pie charts should be used sparingly and only in special circumstances. For example, pie charts would work well to highlight large differences in groups or to show the relative contributions of a few groups.</p><p>Donut charts are similar to pie charts but have a hole in the center. This hole makes it easier to compare relative slices because it forces the viewer to focus on reading the lengths of the arcs rather than the proportions of the slices.</p><p>Neither of these charts can show a slice with a zero or negative response, so if your data has these values, another chart type should be used.</p><p>Word clouds can be used to show summary information in an appealing fashion. A word cloud analyzes each value of a category data item as a single text string. The value of one measure can be used to size the word, and the value of another measure can be used to color the word.</p><p>Word clouds should be avoided if analytical accuracy is desired because it is difficult to compare the relative sizes of different words.</p><p>Words that have more letters often seem larger than words that have fewer letters.</p><p>In addition, words that contain large letters (like o, m, and w) will generally receive more attention than words that contain smaller letters (like l, i, and f).</p><p>Lastly, words whose letters contain ascenders (the part of a lowercase letter that projects above the body of the letter: like b, d, or h) or descenders (the part of a lowercase letter that projects below the body of the letter: like g, p, or q) receive more attention than words that do not.</p><p>For these reasons, word clouds are mostly used for aesthetic purposes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DP9OeLhgkpH9kAJCPtaYFw.png" /><figcaption>Think about it</figcaption></figure><h4>Dual Axis Charts</h4><p>When you want to compare two series with different ranges, dual axis charts can be very effective. Dual axis charts display each series on a separate axis, which makes it easy to see relationships between the series when the ranges are very different. Four types of dual axis charts can be created in Visual Analytics: a dual axis bar chart, a dual axis bar-line chart, a dual axis line chart, and a dual axis time series plot.</p><h3><strong>2. Creating Interactive Reports with Multiple Pages:</strong></h3><p>Remember that the goal of reporting is to tell a single data story. Many times, that story can be told in multiple parts. That’s where multiple pages are useful.</p><p>Each page in your report should focus on one aspect of the story and tell that piece with a limited number of objects. Not only that, you must ensure that each page can stand on its own. Viewers should not be expected to remember details from one page to understand the concepts that are communicated on another page. Basically, you want to ensure that each page communicates one point that will advance your data story.</p><p>In addition to adding multiple pages to your report, you can create pages that are hidden from report viewers. These hidden pages can be useful to communicate details about the report to other designers or they can be linked to from an object on a regular page to enable viewers to see additional details.</p><p>There is no limit to the number of pages that can be added to a report or the number of objects that can be added to a page. However, it is a good idea to limit the number of pages to make your report easier to access, easier to navigate, and easier to understand. If you need more than six or seven visible pages to tell your data story, you should consider creating multiple reports and use links between them to provide additional information.</p><h4>Filtering Data in Interactive Reports</h4><p>Many types of filters can be created to subset data in Visual Analytics. In general, these filters are split between those that can be modified by only the report designer and those that can also be modified by the report viewer. In this lesson, we focus on filters that can be modified by the report viewer.</p><p>These filters can be classified into three types: prompts, actions, and links.</p><p>Prompts enable you to subset all data in the report or on a page. Actions, however, can be used to subset specific objects. You can add two types of prompts to a report: report prompts or page prompts. Report prompts filter all objects in the report, and page prompts filter all objects on a single page. Selections that are made in report prompts also filter the values that are available in page prompts.</p><p>Both types of prompts automatically filter all objects. If you prefer to focus the filter on specific objects, you can use actions.</p><p>The filter action subsets the data in specific objects on the page, based on selections in a source object.</p><p>Links can be used to subset another report, another page in the same report, or an external URL based on selections in a source object. Link actions will automatically filter the target object if both the source and the target use the same data source.</p><p>Filters enable you to create general reports that can be widely used for diverse areas of interest by empowering viewers to find answers to their unique questions.</p><p><strong>Resource Auto Controls:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JOT6kPbbIErYXm3n9cexmA.png" /></figure><h3><strong>3. Working with Display Rules:</strong></h3><h4>Graph-Level Display Rules</h4><p>Display rules enable you to use colors to identify specific areas of interest. Display rules can be added for a specific object or to the entire report.</p><p>For most graph objects, two types of display rules can be added: expression display rules or color-mapped values display rules.</p><p>Expression display rules are based on the value of a measure data item. For the graph, the display rule can be applied to the background of the graph (shown here) or to the graph itself. Expression display rules are great for showing values that outperform or underperform a specific target or for easily identifying the range in which the actual values occur.</p><p>Color-mapped values display rules are based on the value of a category data item (shown here). These display rules are excellent for associating colors with specific values. (For example, coal is gray.) They can be used to add consistency to your reports.</p><h4>Table-Level Display Rules</h4><p>Display rules can use colors in table objects to identify specific areas of interest.</p><p>For list tables, three types of display rules are available: expression, color-mapped values, and gauge.</p><p>Expression display rules are based on the value of a measure data item. For a list table, the display rule can be applied to the measure that is used in the expression, to another column in the table, or to the entire row. Expression display rules are excellent for showing values that outperform or underperform a specific target or for easily identifying the range in which the actual values fall.</p><p>Color-mapped values display rules are based on the values of a category data item. For a list table, the display rule can be applied to any column in the table or to the entire row. Color-mapped values display rules are great for associating colors with specific values. (For example, coal is gray.) They can be used to add consistency in your reports.</p><p>Gauge display rules are based on intervals for a measure data item. For a list table, the display rule can be added to any column in the table. It can be displayed to the left or to the right of the value, or it can replace the value. Gauge display rules are ideal for showing the overall performance of a measure with a single glance, without having to focus on details.</p><p>Crosstabs, on the other hand, only accept expression display rules.</p><h4><strong>Report-Level Display Rules</strong></h4><p>Color-mapped values display rules can also be applied at the report level. In this case, any portion of an object that represents the value will use the colors that are specified in the display rule. This helps you easily identify and compare the same value across multiple report objects.</p><p>In this example, a report-level display rule is applied to identify each product line.</p><p>Report-level display rules apply to all objects in the report and are a perfect way to add consistency across your reports.</p><h3><strong>Conclusion:</strong></h3><p>This is the overall glimpse of SAS Visual Analytics with a different set of methodologies, analysis, and visualizing the reports. I hope that you found it a knowledge adding composition.</p><p>Do share the feedback and let me know if any suggestions.</p><p>Thanks for reading !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec87ecd63ced" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/sas-viya-sas-visual-analytics-ec87ecd63ced">SAS Viya: SAS Visual Analytics</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quick Tutorial on R for Data Science]]></title>
            <link>https://ai.plainenglish.io/quick-tutorial-on-r-for-data-science-362cf77fb569?source=rss-b1488f986f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/362cf77fb569</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[matlab]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Aruna Singh]]></dc:creator>
            <pubDate>Sun, 13 Dec 2020 15:02:28 GMT</pubDate>
            <atom:updated>2021-06-05T17:31:21.556Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>If you are thinking of getting into R, this tutorial will give you a brief idea about how you should begin with. Through this tutorial, I have tried to give a basic insight into data science using R.</blockquote><h3>Installation of R and R Studio</h3><p><strong><em>You can download the detailed setup for R and R Studio from “</em></strong><a href="https://medium.com/datadriveninvestor/beginner-guide-to-r-and-rstudio-27d9b65475e9"><strong><em>here</em></strong></a><strong><em>”.</em></strong></p><p>After downloading and installing the aforementioned software, you are all set to begin your programming journey with R. Now, you may open R Studio, click on <strong><em>File</em></strong>, <strong><em>New File,</em></strong> and lastly on <strong><em>R Script</em></strong>.</p><h3>Installing Packages and Importing Libraries</h3><p><strong><em>Let’s understand packages &amp; libraries and how they play a significant role in R programming.</em></strong></p><p>A library is nothing but a collection of functions that are developed to perform certain tasks. So, each time a programmer writes a code, instead of writing tens and hundreds of lines just to perform a simple operation such as finding a square root, he/she directly uses the readily available function in the default library of R. Packages are a collection of libraries. Basically, it expands the functionality available in R.</p><p>The list of useful libraries goes long, however, don’t fret too much about memorizing the name of every library required for effectively. For this purpose, we have packages like “<strong>tidyverse</strong>”. This package has got all the aforementioned libraries and many more. So let’s install this package to R and import it to our program with the below lines of instruction:</p><pre>#Install package<br>install.packages(“tidyverse”) #Load core tidyverse package<br>library(tidyverse) or require(tidyverse)<br>search() #to check what all packages are present</pre><p>Since we already talked about libraries and packages, we are now going to learn about matrices as they are a way to represent tables and a stepping stone in order to get to data frames.</p><h3>Build your first matrix</h3><p>For analysis, we will be using a data set of the top 10 highest-paid NBA players over the past 10 years. The “<a href="https://www.superdatascience.com/pages/rcourse"><strong>Basketball Dataset</strong></a>” has been collected from SuperDataScience under section 4. Therefore, the goal is to simply investigate trends and patterns that you see in their performance over the past 10 years.</p><pre>mymatrix = matrix(data = &quot;&lt;path&gt;&quot;, nrow=&quot;&quot;, ncol=&quot;&quot;, byrow=FALSE) #Replace &lt;path&gt; with the path of file</pre><p>There are some functions we will be using to tweak the matrix.</p><p><strong>One-dimensional matrix:</strong></p><ul><li><strong>rbind()</strong>: To populate the data of the matrix in a row.</li><li><strong>cbind()</strong>: To populate the data of the matrix in a column.</li><li><strong>names(): </strong>To assign the dimension names of the one-dimensional matrix.</li></ul><p><strong>Two-dimensional matrix:</strong></p><ul><li><strong>rownames()</strong>: To assign the row name to the two-dimensional matrix.</li><li><strong>colnames(): </strong>To assign the column name to the two-dimensional matrix.</li></ul><p><strong>Matrix Operations:</strong></p><ul><li><strong>round(cal, n): </strong>To round off the calculation which you want in a matrix by n decimal digit.</li><li><strong>transpose matrix:</strong></li></ul><pre>mymatrix = matrix(data = &quot;&lt;path&gt;&quot;, nrow=&quot;&quot;, ncol=&quot;&quot;, byrow=FALSE) #Replace &lt;path&gt; with the path of file</pre><pre>trans_mymatrix &lt;- t(mymatrix)</pre><p><strong>Visualizing with Matplot():</strong></p><p><strong>matplot()</strong> is the basic plot design with legends and differentiation.</p><pre>matplot(t(mymatrix), type=&quot;b&quot;, pch=15:18, col=c(1:4,6))<br>legend(&quot;bottomleft&quot;, inset= 0.01, legend= Players, pch=15:18, col=c(1:4,6))</pre><p><strong>Subsetting: </strong>To<strong> </strong>modify the matrix by selecting a specific row and column.</p><pre>x &lt;- c(&quot;a&quot;, &quot;b&quot;, &quot;c&quot;, &quot;d&quot;, &quot;e&quot;) <br>x[c(1,2)] #&quot;a&quot;, &quot;b&quot;<br>x[1] #&quot;a&quot;</pre><p>So far, we have discussed about libraries and packages, matrices and its various operations. Moving on, we will understand what data frame is and how we operate it. Wait…you may wonder that what is the difference between matrix and data frame. Well, they are very similar in the sense that they’re both two-dimensional objects but the core difference is that in matrices all the data has to have the same type.</p><h3>Importing the Data set</h3><p>For analysis, we will be using a <a href="https://www.superdatascience.com/pages/rcourse">“<strong>demographic dataset ”</strong></a> under section 5<strong> </strong>from SuperDataScience. However, we are using this data to analyze the World’s demographic trends. Consider, you are employed as Data Scientist, you are required to produce a scatterplot to illustrate Birthrate and Internet Usage statistics by country. After downloading the data set from the above link and placing it in a directory of our choice, now we are ready to import the data to our R script.</p><pre>mydf = read.csv(&quot;&lt;path&gt;&quot;) #Replace &lt;path&gt; with the path of file or<br>mydf = read.csv(file.choose()) </pre><p>If you have stored your file in the working directory, you can directly call out the file name. But before that, you will have to set up the folder where you have saved the file as a working directory. Post which you can directly call out the file name.</p><pre>setwd(&quot;&lt;path&gt;&quot;)#Data set will be stored in mydata data frame<br>mydf = read.csv(&quot;Demographic_data.csv&quot;)</pre><p>In my case, the file is stored as “Demographic_data.csv”. CSV means comma-separated values. In this format, each element of data is separated by a comma. Although other characters can also be used to separate the elements of the data. To get more detail you can go through the R documentation by using the following command:</p><pre>help(read.csv) #or<br>?read.csv</pre><h3>Exploring your dataset</h3><p>Now we have our data set imported, but in most cases, we cannot use it directly as it may not be correctly ordered or it might contain some features which are not required for our analysis. Before doing that, let’s first have a look at our data. You can use all or any of the below listed commands to do so.</p><pre>head(mydf) #or tail(mydf) <br>summary(mydf) #or str(mydf) <br>nrow(mydf) #or ncol(mydf) #to get no of rows and cols</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1wMS__qLJTat6tdiLUPQ6w.png" /><figcaption>snippet for summary function()</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fo5RwGYveWBzQPStFZa4NQ.png" /><figcaption>snippet for str function()</figcaption></figure><p><strong>“$” is another way to access the data in the data frame.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MTlLYmN7QfFpFgPcAgpoqA.png" /><figcaption>mydf$Internet.users</figcaption></figure><p>To get the levels of a certain column, use levels(). However, the column should be of type categorical “factor”. To convert it, use factor() which is described below.</p><pre>mydf$Income.Group &lt;- factor(mydf$Income.Group)<br>levels(mydf$Income.Group)</pre><pre>[1] &quot;High income&quot;         &quot;Low income&quot;          &quot;Lower middle income&quot; &quot;Upper middle income&quot;</pre><p>To check if the dataset is a data frame or not, use <strong>is.data.frame(mydf)</strong></p><p>To delete the column from a dataset, use mydf$Income.Group &lt;- null</p><p><strong>Tidying up your dataset</strong></p><p><strong>Filtering: </strong>Let’s filter the data based on a Birthrate greater than 4.</p><pre>&gt; gt4 &lt;- mydf$Birth.rate &gt; 4<br>&gt; filtered_data &lt;- mydf[gt4,]<br>&gt; head(filtered_data)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bRQKioBgbdYY_mqbiBFyUg.png" /></figure><p><strong>Merge: </strong>Let’s merge the two data frames (mydf &amp; stats) wrt country code.</p><pre>&gt; merged_data &lt;- merge(mydf, stats, by.x = &quot;Country.Code&quot;, by.y = &quot;code&quot;)<br>&gt; head(merged_data)</pre><h3>Data Visualization and Analysis</h3><p>For visualization, we will mostly use “ggplot2” which is useful for almost every kind of graph. We can also use “dplyr” package for the same use cases. Seven factors involve representing the graph:</p><ul><li><strong>theme</strong>: things that make your chart pretty and overwhelming.</li><li><strong>coordinates</strong>: Pretty much as the name suggests, used for categorizing the data.</li><li><strong>facets: </strong>Get a sense of the shape of each feature of the <strong>data </strong>and explore a set of individual observations at different granularities.</li><li><strong>statistics: </strong>statistics may require to make transformations to data to create new variables and visualize them.</li><li><strong>geometries: </strong>things you cant actually see as the size of what? Or the color of what? Is it a circle? Is it a square or is it? is it a dot or is it a line?</li><li><strong>aesthetics: </strong>things you can see like color, the size which gives a graph a more user-friendly look.</li><li><strong>data: </strong>the center of a graph</li></ul><p>In this section, we will discuss Movie Ratings by critics and audience as movie budgets for the year 2007–2011. However, we will explore different ways of creating graphs with <strong>ggplot2</strong>. To begin with, let&#39;s use <a href="https://www.superdatascience.com/pages/rcourse">“<strong>Moving Ratings”</strong></a> under section 6 from SuperDataScience.</p><p>Import the file and assign proper column names to make the graph readable. Henceforth, create ggplot2 keeping Critic Rating along x coordinate and Audience Rating along x coordinate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6gf5XTI6J5kKoW4hiDJQbg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_ITvLWPYO9TOEcLXckJA0Q.png" /></figure><p><strong>Let’s add aesthetics to the ggplot</strong></p><pre>&gt; ggplot(data = movies, aes(x=CriticRating, y=AudienceRating, color=Genre)) + geom_point()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p9ZscNI_gvp3RCcCWfeANg.png" /></figure><p><strong>Overriding aesthetics</strong></p><pre>&gt; g &lt;- ggplot(data = movies, aes(x=CriticRating, y=AudienceRating, color=Genre)) <br>&gt; g + geom_point(aes(size=CriticRating))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JXi3VtKJA71xtjclotZcPQ.png" /></figure><p><strong>Mapping &amp; Setting: </strong>mapping is done by using aes() which we have done so far while the setting is done without it. let&#39;s see now,</p><pre>&gt; g &lt;- ggplot(data = movies, aes(x=CriticRating, y=AudienceRating)) <br>&gt; g + geom_point(aes(color=Genre)) #mapping<br>&gt; g + geom_point(color=&quot;Dark Green&quot;) #setting</pre><p><strong>Histogram and Density Charts: </strong>mapping is done by using aes() which we have done so far while the setting is done without it. let&#39;s see now,</p><pre>&gt; g &lt;- ggplot(data = movies, aes(x=CriticRating, y=AudienceRating)) <br>&gt; g + geom_point(aes(color=Genre)) #mapping<br>&gt; g + geom_point(color=&quot;Dark Green&quot;) #setting</pre><h4><strong>You are all set to begin some extensive EDA and model building in R. All you need to write a few lines of code and boom, the magic is in front of you.</strong></h4><p>Thank you for reading the article and I am sure, it would be pretty helpful for beginners.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=362cf77fb569" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/quick-tutorial-on-r-for-data-science-362cf77fb569">Quick Tutorial on R for Data Science</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>