<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arimoro Olayinka on Medium]]></title>
        <description><![CDATA[Stories by Arimoro Olayinka on Medium]]></description>
        <link>https://medium.com/@arimoroolayinka?source=rss-27bb4daa58e4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hWCvqjWB2uxUO8fsQuEsxw.jpeg</url>
            <title>Stories by Arimoro Olayinka on Medium</title>
            <link>https://medium.com/@arimoroolayinka?source=rss-27bb4daa58e4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 14:05:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arimoroolayinka/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A simple guide on connecting RStudio to a MySQL database]]></title>
            <link>https://medium.com/geekculture/a-simple-guide-on-connecting-rstudio-to-a-mysql-database-d8a8ca6573fd?source=rss-27bb4daa58e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8a8ca6573fd</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[rstudio]]></category>
            <dc:creator><![CDATA[Arimoro Olayinka]]></dc:creator>
            <pubDate>Wed, 10 Aug 2022 18:48:40 GMT</pubDate>
            <atom:updated>2022-08-11T05:44:36.195Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0lT51AQVWkpAiHHzkoZPtg.jpeg" /></figure><p>Have you read my article on <a href="https://arimoroolayinka.medium.com/a-simple-guide-on-connecting-rstudio-to-a-postgresql-database-9e35ccdc08be">connecting RStudio to a PostgreSQL database</a>? That article is a prerequisite to the discussion we will consider here. I explained in great detail some of the functions we will use and the steps we will take in this article.</p><p>If you have not read that <a href="https://arimoroolayinka.medium.com/a-simple-guide-on-connecting-rstudio-to-a-postgresql-database-9e35ccdc08be">article</a> yet, kindly take about 12 minutes to read the article here. If you have read that article, then we can proceed together on this beautiful ride.</p><p>In this article, I will show you how to connect locally to a MySQL database. In my previous article, I had previously explained the rationale for putting together this article when we connected to a PostgreSQL database. Connecting RStudio to SQL databases helps us leverage the powerful R packages and functions for different data tasks.</p><p>As I mentioned in the previous article, this article is a simple guide, meaning that it is not all-encompassing to explain all the technicalities involved. Be assured that by following the steps outlined in this guide, you will be able to connect your RStudio to your local MySQL database. Also, I believe you have a MySQL database set up with tables in the database.</p><blockquote><strong>Note: </strong>This article will not teach how to create a MySQL database. It is assumed that you have set up at least one MySQL database with at least one table.</blockquote><p>Furthermore, various functions and packages can help achieve the same purpose for this article — which is connecting RStudio to a MySQL database. However, I will use the odbc R packages to show this illustration. ODBC means <a href="https://www.techtarget.com/searchoracle/definition/Open-Database-Connectivity">Open Database Connectivity</a>, an open standard application programming interface (API) that allows application programmers to access any database [1].</p><p>Finally, before we get started, I want to mention that make sure you read through to the end; I have added a bonus section and a gift to this article.</p><p>Now, let’s get started!</p><h3><strong>Getting Started</strong></h3><p>The first step will be to install and import the required R packages. If you have used R before, you should know that R has packages for almost any purpose. Currently, there are over 13,000 packages in the R Comprehensive Archive Network (CRAN) repository. To install and import the packages we will use in this article, use the code below.</p><blockquote><strong>Note: </strong>To run a line of code in R, use CTRL ENTER for Windows users or use the Run button on your RStudio interface.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*-C57pkOdhNyceqRGWeXbhQ.png" /></figure><p>I suggest using a Google search to check what each package imported does. I have skipped explaining the functions of each package imported to make this article as short and compact as possible.</p><p>With this out of the way, we can now see how to connect RStudio to a MySQL database. What you just read was quite familiar to the previous article, right? Now, we will delve into the juicy part of this article.</p><h3><strong>Connect RStudio to a MySQL database</strong></h3><p>As we saw with the PostgreSQL database, let us start by showing a snippet of the MySQL database we want to connect to in this article. I have the <strong>employeesdb</strong> database in MySQL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*EoJ4ef1JQXI-z-chzaZu_Q.png" /></figure><p>From the image above, it looks like the <strong>employeesdb </strong>has five tables, and there are no views. Clearly, this database is different from the database we connected to under the PostgreSQL article. Since we will be querying the employees table from the <strong>employeesdb </strong>database, I believe it will be helpful to see the data it contains.</p><p>The image above shows a snippet of the data in the employees table. Notably, there are 1000 rows and nine columns on the employees table. Knowing this information will help us confirm if we have successfully connected to the database when we query the database.</p><p>Connecting to your MySQL database is similar to the PostgreSQL database, but it is not as straightforward as the PostgreSQL; you will need to do a few things to connect seamlessly.</p><p>First, we want to be sure the MySQL ODBC driver was installed when you installed your MySQL. Remember that we are using the odbc package to create the connection. Therefore, we want to be sure that the database driver for MySQL is installed. We didn’t need to do this for PostgreSQL — I guess PostgreSQL does not require drivers installed (not sure, though!)</p><p>We can use the code below to check the list of unique database drivers installed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*oUH1mGTlTr9tdr_15Wt-kQ.png" /></figure><p>The result shows us the database drivers installed, and we can see two MySQL drivers installed. We can see some other database drivers installed, including SQL Server.</p><p><strong>I guess PostgreSQL saves us the stress of having to add the database driver as an ODBC data source. Sincerely, I do not understand the technicalities of how PostgreSQL database drivers work. You can share your thoughts on this in the comments.</strong></p><p>Now that we have established that the MySQL database driver is installed, we will need to confirm that the driver is added as an ODBC data source. If not, we will need to add the MySQL database driver as an ODBC data source. Afterward, we can then connect RStudio to the MySQL database.</p><p>To help us achieve this, I have outlined the steps you need to take to get this done.</p><h4><strong>Step One: Add your MySQL driver as an ODBC data source</strong></h4><p>First, we must confirm that the MySQL database is an ODBC data source. To do that, follow the steps below, as also shown in the images below:</p><ul><li>Go to your search bar, type <strong>odbc Data sources</strong>, and click on the 64-bit version if you installed the 64-bit version.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*HMuB5A4IRw6o0RexCiSKPw.png" /></figure><p>When you click on that, the dialog box like the image below should open.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*Fpyoq_Z0fQPXunrmPH1yow.png" /></figure><p>From the look of things, the MySQL database driver is not listed as an ODBC data source. Therefore, it will be impossible to connect RStudio to MySQL, except if we add the MySQL odbc driver to the user data sources list.</p><ul><li>Now, click on <strong>add </strong>and proceed to the next step.</li></ul><p>Upon clicking on add, the dialog box below would pop up.</p><ul><li>Now, click MySQL ODBC 8.0 ANSI Driver and finish (selecting MySQL ODBC 8.0 ANSI Driver should help get the job done).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*nnR0AeugYWhFVP5wV20rJA.png" /></figure><p>With this step done, let’s proceed! I hope you are following up so far. If not, kindly take a step backward and be sure you are following my flow of thoughts before you continue.</p><p>Beautiful! When you click on finish, the image below comes up without the information you can see in the image below. Only the default port number 3306 for MySQL will be filled for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*aTJR7ZK1-bxvMdjMqDngMQ.png" /></figure><p>In this step, we will configure the MySQL ODBC Data Source connector. Once the dialog box above opens up, you will need to fill the following:</p><ul><li>Data Source Name: You can give it any preferred name. I named it <strong>MySQL-connection</strong></li><li>Description: You may give it a brief description</li><li>User: Usually, the user should be <strong>root</strong>. However, you can decide to connect to another user. In this case, I chose to connect to <strong>the root user </strong>because the <strong>employeesdb </strong>database is in my root user connection.</li></ul><blockquote><strong>Hint: </strong>When you open your MySQL Workbench, you should see your different users, as shown in the image below.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*KV6RMByOaA1BmBv5xVe6UQ.png" /></figure><ul><li>Password: Input the same password you used when you installed MySQL, which is the same password you put in when you open your MySQL Workbench</li><li>Database: Input the database you would love to connect to; in this case, I want to connect to the <strong>employeesdb </strong>database</li></ul><blockquote><strong>Note: </strong>One practical thing you can do is click on the <strong>Test </strong>button. If the details you inputted, especially for the user, password, and database, are correct, you should see the prompt <strong>Connection Successful </strong>as shown below.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*ySrnpc0Sw8HJPqvMbP5VZA.png" /></figure><p>You will notice that I am connecting to another database called <strong>employeesdb</strong>, which is different from the one I used to illustrate the PostgreSQL connection. Doing this shows that the idea is similar irrespective of the RDBMS you want to connect to.</p><p>Finally, you can click on <strong>OK</strong> or the <strong>X</strong> to go back to the MySQL ODBC Data Source dialog box and click on OK, and the new MySQL connection to the <strong>employeesdb</strong> database is created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*7dZQ6kCQEaeXL2KrxBp5qg.png" /></figure><p>You will see the new connection created. Click on OK, and you are good to go. We have added MySQL as an ODBC data source. Now, connecting RStudio to the MySQL database <strong>employeesdb </strong>will be easy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*hjzFTIi3caLg0A35oxU2CA.png" /></figure><blockquote><strong>Note: </strong>Upon clicking on the user data sources, you will see the option to add, remove or configure the connection. If you click on configure, you can modify the details you filled in the steps above. For example, you may change the database name to another database you have in your MySQL.</blockquote><p>With the connection done, what is left is to write the R codes similar to the codes we wrote when we connected RStudio to PostgreSQL.</p><h4><strong>Step Two: Define a database connection</strong></h4><p>Now, we are back to what you are familiar with. Use the code below to connect to the <strong>employeesdb </strong>database from RStudio, which is very similar to how we connected to the PostgreSQL database. The only difference is the information provided in the arguments. For example, your password will be the same one you used when you installed MySQL and the same one you used when you added the MySQL driver as an ODBC data source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*E9NDsuBpP8NF3i1SkW0yDQ.png" /></figure><p>From the code above, we have created an R object called <strong>conn </strong>(<em>you may give it any preferred name</em>). I trust that the other parts of the code above are self-explanatory.</p><blockquote><strong>Note: </strong>If you want to connect to an online or a cloud-based database, you will need to change the server from <strong>“localhost” </strong>to the cloud-based server, probably via the link. You can explore that option.<strong> </strong>We used localhost as the server because we connected to a local database.</blockquote><p>Next, let’s print out the names of the tables in the <strong>employeesdb </strong>database using the code below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*MgttkmfUOcCXxC07EJuVyg.png" /></figure><p>The result is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*FN70-7ZTqEUwPXFBhGQmkA.png" /></figure><p>Looking at the left-hand side of the image at the beginning of this section of connecting MySQL to RStudio, you will see that we have the same table name. This result further proves that the connection was successful.</p><blockquote><strong>Note: </strong>If you connected to a different database which is likely the case, you would get a different list of table names because we did not connect to the same database.</blockquote><h4><strong>Step Three: Query the database</strong></h4><p>Now that we are sure that we have created a connection successfully, we can now query the database from within RStudio. The same code we used when we connected to the PostgreSQL database also works.</p><p>Let’s assume we want the first 500 rows from the employees table in the <strong>employeesdb</strong> database we connected to RStudio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*swHNdVccDg9ozuUfhz7laQ.png" /></figure><p>From the code above, we used the dbGetQuery() function from the DBI package. It takes two required arguments: the connection and the statement. Recall that we called the R connection object <strong>conn </strong>when we created the connection with MySQL. The statement which is the second statement is simply retrieving all columns because we used the asterisk (<strong>*</strong>)<strong> </strong>after the SELECT keyword and retrieves 500 rows because of the LIMIT 500 statement.</p><p>Let’s glimpse at the data. We will use the <strong>glimpse() </strong>function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*dMOVuWtUOccnwZNp4mKkpg.png" /></figure><p>Amazing! We have 500 rows. After all, we queried the database for 500 records from the employees table. Also, we have nine columns because we queried the database for all the columns.</p><p>Since we have saved the data as an R object <strong>emp_data</strong>, we can close the connection to free up resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*3u47uSfnYOGX8LG2B0Sbmg.png" /></figure><p>As explained earlier, if we try to run the code below, we will get an error because we have disconnected RStudio from the MySQL database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*swHNdVccDg9ozuUfhz7laQ.png" /></figure><p>This error shows us why saving the data you queried as an R object is paramount. Even though we have disconnected access to the employees table, we can still access the employees data with 500 rows because we saved it in the R object <strong>emp_data</strong>. Otherwise, we need to run the code to create the connection again.</p><p>Finally, we can start using R functions to do anything we want with the data we have queried and saved as <strong>emp_data</strong>. For example, we can print out the first six rows of the data using the head()<strong> </strong>function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*bBwjUYxDFXfuoRFcBFUZYQ.png" /></figure><p>Beautiful! Upon careful observation, you will notice that the data returned here are similar to the data produced from the image at the beginning of this section.</p><p>Pat yourself on the back! You have done well reading through this article. I believe it has been a beautiful experience walking through this write-up. If we conclude here, I think you have learned some great ideas.</p><p>However, I have decided to add a bonus section on using functions in the <strong>dbplyr </strong>package to query the database after creating a connection to the database instead of using the dbGetQuery() function in the DBI package. Using either of these packages has its pros and cons, which we won’t explore in this article.</p><p>Let’s see how that works.</p><h3><strong>Bonus Section: Using dbplyr</strong></h3><p>Personally, I am okay with using either package, DBI, or dbplyr. However, I think the dbplyr offers some flexibility, especially with using dplyr functions while querying the database. The <a href="https://dbplyr.tidyverse.org/">dbplyr package</a> is a ‘dplyr’ back end for databases that allows you to work with remote database tables as if they are in-memory data frames. Basic features work with any database with a ‘DBI’ back end.</p><p>As a first step to illustrate this example, we will need to reconnect to the MySQL database. Remember that we had disconnected from the database. The code below helps us do that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*vesWGoB3QDKYKrluAq9IyA.png" /></figure><p>Now, let’s say we want to query the employees table for the first 500 records or rows. We will use the generic tbl() function that creates a table from a data source. We will need to provide the function with the data source (in this case, our database connection) and the SQL statement.</p><p>Please notice the difference between this and the dbGetQuery() function. Here, we will specify SQL before the SQL statement because the tbl() function is a generic function that can work with various data sources.</p><p>I believe that makes sense.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*6fYfAAWzBiV6Wqf90pHzUw.png" /></figure><p>Could you tell me why it is not advised to write the code the way I have written it above?</p><p>The answer is simple if we close the connection, we lose access to the database and employees table. In fact, the code will print the result on your RStudio console (which can be pretty messy), as shown in the result below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*0cEMuAQOf55QEdePR0SCMA.png" /></figure><p>Also, you will notice that the result shows us the data source and the connected database — <strong>employeesdb</strong>. Another thing from this result is that the data is not yet an R data frame. Therefore, using some R functions on the data we have queried may be impossible. That’s one advantage of using the dbGetQuery() function. It returns the queried data as a data frame.</p><p>We can quickly fix these issues I have raised.</p><p>Let’s tidy up this result by storing the result in an R data frame. The code below also returns the first 500 rows. I have used the pipe function to chain a couple of steps. First, I used the tbl() function to connect to the database, but instead of writing the SQL statement to retrieve the first 500 rows, I queried the whole employees table hence the line tbl(conn, “employees”). Then, I used the as.data.frame() function to convert the queried data to a data frame to make it easy to work with as we proceed. Since we want the first 500 rows, I used the slice() function to retrieve the first 500 rows using indexing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*YCrhuqqG1egsDWaMwBtVqQ.png" /></figure><p>Finally, we can check the dimension of the new data frame to be sure we did the right thing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*p4i3J2kctRan-P2j9N_mBA.png" /></figure><p>The dimension shows 500 rows and nine columns, as expected. So, we can use dplyr functions while querying the database — through dbplyr. We can then close or disconnect the connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*3u47uSfnYOGX8LG2B0Sbmg.png" /></figure><p>Having saved the data, we can return the first six rows of the data using the head() function and select some columns we want from the nine columns, including the first_name, last_name, department, and salary. The select() function from the dplyr function helps us to achieve this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*WqLv1NPElh9irdqXTlnc_g.png" /></figure><blockquote><strong>Note:</strong> Usually, in R, we may find a function like the select() function in various packages. Sometimes, conflict usually occurs when you call such a function. In the code, you will notice that I wrote dplyr::select(). dplyr::select() means that I want to use the select() function from the dplyr R package because the select() function can also be found in the MASS package. In the MASS package, the select() functions differently than selecting the columns using their names or types.</blockquote><p>So, there could be cases where you need to explicitly state the package where a function can be found using packagename::fucntionname. In fact, there is a high possibility that you will see such codes in your R learning.</p><p>The result of the code above is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*DU53_n2bbn0WWUc3sWj8KQ.png" /></figure><p>The output above is similar to the result we have seen before now. The only difference is that we have four columns against nine because we selected four columns. A final note here is that the select() function will return the result in the order you specify the columns in the select() function.</p><h3><strong>Conclusion</strong></h3><p>Amazing! You have come a long way and I am glad you came this far. I salute your doggedness and determination to get to the end of this illustration. I believe it has been worth your time learning about these concepts, which you found valuable.</p><p>I have a gift for you — I have uploaded the complete R script used for this illustration to a GitHub repository <a href="https://github.com/Predicare1/Connecting-RStudio-to-SQL-databases/blob/main/connect-RStudio-to-MySQL.R">here</a>. This script saves you the stress of writing the codes, and you can quickly run the code. Of course, to practice with this script, you will need to tweak the script, like changing the database name, table name, and so on.</p><p>Furthermore, you can do a host of other things after creating the connection to the database. You can update a table or add a new record to a table in the database from RStudio. This article has given you the first and crucial step to maximizing the power of RStudio and SQL. From here, you can read the documentation of packages and functions to do different things with the database you have connected to from RStudio.</p><p>If you enjoyed this post, please drop your comments in the comment session and share this article with others. It would mean a lot to me if you could click on the “claps” icon (up to 50 claps allowed).</p><p>As a project-based course instructor with Coursera Guided Project Network, I have taught some courses on using R, especially the R tidyverse family of packages, including the dplyr package. You can check them out <a href="https://www.coursera.org/instructor/arimoro-olayinka">here</a> to take any of these courses. Thank You! See you at another time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8a8ca6573fd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/a-simple-guide-on-connecting-rstudio-to-a-mysql-database-d8a8ca6573fd">A simple guide on connecting RStudio to a MySQL database</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A simple guide on connecting RStudio to a PostgreSQL database]]></title>
            <link>https://medium.com/geekculture/a-simple-guide-on-connecting-rstudio-to-a-postgresql-database-9e35ccdc08be?source=rss-27bb4daa58e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e35ccdc08be</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[rstudio]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Arimoro Olayinka]]></dc:creator>
            <pubDate>Wed, 10 Aug 2022 18:45:31 GMT</pubDate>
            <atom:updated>2022-08-11T05:44:38.478Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mwl1stznMXq6BSmZfOXMSw.jpeg" /></figure><p>Have you thought about the possibility of querying a relational database like PostgreSQL from RStudio? In 2021, I tried to do this, but I found out that there were not many resources explaining how to do that in great detail. I should have written this article a while ago. However, during that time, I taught a project-based course on Coursera titled “Manipulate R data frames using SQL in RStudio” to understand how to use and write SQL codes to query R data frames in RStudio. You can check out the course <a href="https://www.coursera.org/learn/manipulate-r-data-frames-using-sql-rstudio/home/week/1">here</a>.</p><p>In this article, I will show you how to connect locally to a PostgreSQL database. The ideas you will glean from this article will be helpful when connecting to other relational databases, including MySQL and SQL Server, even if the database is in cloud storage.</p><p>Before exploring the article, a question that should come to mind is — what is the rationale for this article? Why do I need to connect RStudio to SQL databases?</p><p>Usually, data in a relational database are stored in a normalized format. Therefore, for data professionals to perform analyses such as statistical computing and data visualization, there is a need to write advanced and complex SQL queries. However, R can easily connect to many relational databases like PostgreSQL, MySQL, Oracle, SQL Server, etc., and retrieve data from them as a data frame. When the data is successfully in the R environment, it becomes a typical R data set, making it easy to manipulate or analyze using all the powerful packages and functions.</p><p>I believe that this provides a solid reason why this article is essential.</p><p>Before we proceed, there is a little caveat here. This article is a simple guide, meaning it is not all-encompassing to explain all the technicalities involved. Be assured that by following the steps outlined in this guide, you will be able to connect your RStudio to your local PostgreSQL database. Also, these steps, with little tweaks, can connect to cloud-based databases.</p><p>Also, before we start, I want to assume that you have PostgreSQL installed, you know how to create a database, and you have at least a database in PostgreSQL. Suppose you do not have prior experience with working with SQL databases. In that case, you can check out the project-based course I taught titled “Performing Data Definition and Manipulation in SQL” <a href="https://www.coursera.org/learn/performing-data-definition-and-manipulation-in-sql/home/week/1">here</a>.</p><p>Furthermore, various functions and packages can help achieve the same purpose for this article — which is connecting RStudio to a PostgreSQL database. However, I will use the odbc R packages to show this illustration. <a href="https://www.techtarget.com/searchoracle/definition/Open-Database-Connectivity">ODBC means Open Database Connectivity</a>, an open standard application programming interface (API) that allows application programmers to access any database.</p><p>Finally, before we get started, I want to mention that make sure you read through to the end; I have added a bonus section and a gift to this article. Now, let’s get started!</p><h3><strong>Getting Started</strong></h3><p>The first step will be to install and import the required R packages. If you have used R before, you should know that R has packages for almost any purpose. There are over 13,000 packages in the R Comprehensive Archive Network (CRAN) repository. To install and import the packages we will use in this article, use the code below.</p><blockquote><strong>Note: </strong>To run a line of code in R, use CTRL ENTER for Windows users or use the Run button on your RStudio interface.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*-C57pkOdhNyceqRGWeXbhQ.png" /></figure><p>I suggest using a Google search to check what each package imported does. I have skipped explaining the functions of each package imported to make this article as short and compact as possible.</p><p>With this out of the way, we can now see how to connect RStudio to a PostgreSQL database.</p><h3><strong>Connect RStudio to a PostgreSQL database</strong></h3><p>In this section, we will query the employees table in the <strong>Employees </strong>table. So, first, we will connect to the local database in PostgreSQL called <strong>Employees</strong>. Note that I have this database already in my PostgreSQL. Let me show what the database looks like including some records or rows in the employees table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*nhcTlGKyIzGAgVYSNNW_MA.png" /></figure><p>The <strong>Employees </strong>database contains seven tables and two views, as seen from the image above, including the employees table. In addition, on the right-hand side, the output from querying all the data in the employees table is returned. However, we can only see the first ten rows due to the space limit. However, the employees table contains about 11,822 rows and six columns, as shown in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*P7_s7ryIkcIoMmNeNJyJ_g.png" /></figure><p>Great! We can start learning how to connect RStudio to the <strong>Employees </strong>PostgreSQL database. I will outline the steps to connect successfully to this database from RStudio. If you are ready and excited to see how to do that, then join me!</p><h4><strong>Step One: Define a database connection</strong></h4><p>Creating a database connection from RStudio to PostgreSQL is relatively straightforward. In this step, we will use functions from the DBI package we installed earlier.</p><p>Before we write the code to connect RStudio to PostgreSQL, we want to be sure we have every information about the database, including the port number and username. To get that information, once you open pgAdmin (the management interface for PostgreSQL), do the following as shown in the image below:</p><ul><li>Right-click on PostgreSQL14 and select <strong>Properties </strong>(a dialog box will open)</li><li>Select the <strong>Connection </strong>tab, and you will see all information needed to create the connection.</li></ul><blockquote><strong>Note: </strong>The default port number for PostgreSQL is 5432.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*9rnVTogBx5s9dR6XrsVzhQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*uVpkVxnbqC0gyArC83gi3w.png" /></figure><p>Amazing! Now, we have the needed information to create the connection.</p><p>Usually, it is good practice to start by first checking whether a connection exists, that is, to check whether we can connect to the PostgreSQL database from RStudio. This article will use the <a href="https://rdrr.io/cran/DBI/man/dbCanConnect.html">dbCanConnect()</a> function from the DBI package. The function only checks validity without returning a connection object. The default implementation opens a connection and disconnects on success. Simply, it helps us check if RStudio can establish a connection with the database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*vaDVlKIDL8vluLR1A9fjhQ.png" /></figure><p>The code above looks quite understandable. We filled in the information we got about the database. Recall that we want to connect to the <strong>Employees </strong>database hence the argument dbname = “Employees”. Furthermore, input the same password you used when you installed PostgreSQL, which is the same password you put in when you open your PgAdmin interface.</p><p>Beautiful! The result of printing the connection object <strong>con </strong>is <strong>TRUE, </strong>suggesting that there is a valid connection. Having established that, we can connect to the database by going through the appropriate authentication procedure using the dbConnect() function. It takes similar arguments as the dbCanConnect() function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*PgXidEqCENnMpphlFJsd5w.png" /></figure><p>As I mentioned, they take similar arguments but produce different results. Let’s see the result of the connection object <strong>con_1 </strong>we created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*VqOr_gC-K-OQuANUN1hPKw.png" /></figure><p>I believe that we both agree that they return different results. This result tells us that we have successfully connected to the <strong>Employees </strong>database on port 5432.</p><p>Bravo! From here onwards, it is going to be a smooth ride. Connecting to the database is the first and most crucial step. Isn’t that why you are reading this article?</p><p>Now, let’s play with some other functions and interesting data tasks. Next, using the dbListTables() function, we can return a list of all views and temporary objects (tables) in the database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*T-HM9__0AW4cl2hhWRoxvg.png" /></figure><p>What do you notice from the result?</p><p>Nine objects were returned, corresponding to seven tables and two views, as we saw in the image at the beginning of the section. This makes sense, right?</p><p>The question now is — after creating the connection, then what? Find out the answer to this question in the next step.</p><h4><strong>Step Two: Query the database</strong></h4><p>Now, let’s query the database directly from RStudio using the dbGetQuery() function from the DBI package. This function returns the result of a query as a data frame. It takes two required arguments: the connection and an SQL statement.</p><p>Upon running the code below, the employees<strong> </strong>table will be queried to return all data. The disadvantage of running the code below the way I have written it is that it prints the results on the RStudio console, thereby making your RStudio console very messy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*pa00FHqmO9hcYlonwSfi3Q.png" /></figure><p>One better way is to store the result as an R object, for example, <strong>emp_df</strong>. With this R object, we can do many other things, including data manipulation on the employees table without affecting the original employees table in the <strong>Employees </strong>database or losing the data even when we close the connection to the database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*CxdTZi8mkpS5JDcOdh33pw.png" /></figure><p>Great! You are doing well. The code will successfully query all data in the employees table.</p><p>Let’s proceed to step three!</p><h4><strong>Step Three: Explore the dataset</strong></h4><p>Now, we have the data in the employees table saved as an R data frame. First, we can check the data’s dimension (number of rows and columns) using the dim() function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*HWaiOL7_dF90QJC0UVyVqA.png" /></figure><p>Isn’t this interesting? The result tells us that the data has 11,822 rows and six columns.</p><p>Do these numbers strike a chord? Yes!</p><p>These were the exact result of the number of rows and columns in the employees table in the <strong>Employees </strong>PostgreSQL database. Clearly, we have successfully connected to the employees table without reasonable doubt.</p><p>Now, we can glimpse the data using the glimpse() function.<strong> </strong><a href="https://www.rdocumentation.org/packages/dplyr/versions/0.3/topics/glimpse">glimpse()</a> is like a transposed version of print(): columns run down the page, and data runs across. This function makes it possible to see every column in a data frame. It’s a little like str() applied to a data frame, but it tries to show you as much data as possible. Also, it always shows the underlying data, even when applied to a remote data source. glimpse()<strong> </strong>is a function in the dplyr<strong> </strong>package.</p><blockquote><strong>Note: </strong>The dplyr package is under the umbrella of a family of tidyverse packages. The <strong>dplyr </strong>package is very powerful, and in fact, it is the most useful package in R for data manipulation. It is my all-time best package for data manipulation. You can check out my project-based course on Coursera titled <strong>Data Manipulation with dplyr in R </strong><a href="https://www.coursera.org/learn/data-manipulation-with-dplyr-in-r/home/welcome">here</a><strong>. </strong>I covered the use of the dplyr package in that course.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*bCAbOMOwNEwA-Ik_GsUafg.png" /></figure><p>Amazing! At one goal, we can see all the information of a dataset, including the number of rows, number of columns, column names, column data types, and even a few rows of the data.</p><p>As we advance, we can use the head and tail base R functions to get the dataset’s first and last six rows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*xCbIXoHxQTIlUC7cHViLbg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*qRF6Ak1UMfnZW1rgP97qgA.png" /></figure><p>Isn’t this so beautiful? These results further stress the importance of what we are considering in this article. Just by connecting to the database, we can query any of the tables in the database. In this case, we queried the employees table and saved it as an R data frame. With this, we can do absolutely any data manipulation, cleaning, transformation, or visualization on the employees table, which may have been difficult or expensive if we were to write SQL queries. Also, we can do all these data tasks without tampering with the database — thanks to RStudio.</p><p>As we begin to round off this article, it is crucial that once you have saved your queried data as an R object (like a data frame), you should disconnect or close the connection you created. Doing this helps discard all pending work and frees resources (e.g., memory, sockets).</p><p>We will use the <a href="https://www.rdocumentation.org/packages/DBI/versions/0.5-1/topics/dbDisconnect">dbDisconnect()</a> function from the DBI package. It works like the opposite of the dbConnect() function we used to create the connection. The dbConnect() function takes one required argument: the connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*__kLwHrz9nRxo0mTJjei1A.png" /></figure><p>This code closes the connection, meaning we no longer have access to the tables in the <strong>Employees </strong>database. However, we can still perform data manipulation tasks on the employees table because we have saved that data as an R data frame called <strong>emp_df.</strong></p><p>For example, a data question could be asked, such as how many employees were employed on or before 1st January 1990? The code below helps us answer this question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*ZEQFkL59sGc43LyGLfdkkw.png" /></figure><p>The code above uses functions in the dplyr package. The %&gt;% represents the pipe function. It is a super useful function in dplyr to chain a sequence of operations together. The filter() function filters for where the hire_date was on or before 1st January 1990 (hence the use of &lt;=). Also, you will notice that the date takes the yyyy-mm-dd format (like a typical SQL date format). You can decide to change the date format using functions in the powerful lubridate() package <em>(doing this is beyond the scope of this article)</em>. Finally, we used the count() function to get the number required.</p><p>Let’s see the result of the R code above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*igHsdQplgaWjA85FGLNp7w.png" /></figure><p>Amazing! Of the 11,822 employees, the company employed 6,453 on or before 1st January 1990.</p><p>We have come to the end of this discussion on connecting RStudio to a PostgreSQL database.</p><p>Ooh… Hold on! Let me quickly add this important point here. You can use the code below also to create a connection. The difference between this code and the one we used above is simply using the RStudio API to request the username and password.</p><p>Actually, there is no difference, but this code looks more secure than the first one. Using this may be helpful if you share your code with others and don’t want to reveal your username and password.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*08oXvaTNqxefL5Y1sPDl2A.png" /></figure><p>Upon running the code above, a dialog box asking for your username and password will pop up like the one below. Upon inputting the username and password, the connection will be created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*KZWJjz44xvhZOw2-9XdPkw.png" /></figure><blockquote><strong>Note: </strong>You only requested the username and password in the code above. You could have also used the RStudio API to ask for the database name.</blockquote><p>I believe that was quite understandable. I decided to add that last part to show that you can do something like this with RStudio. You can explore that further.</p><p>Would you be interested in learning how to <a href="https://arimoroolayinka.medium.com/a-simple-guide-on-connecting-rstudio-to-a-mysql-database-d8a8ca6573fd">connect RStudio to a MySQL database</a>? Connecting to a MySQL database is not as straightforward as connecting to a PostgreSQL database. However, in my article here, I have explained everything you need to know and do to <a href="https://arimoroolayinka.medium.com/a-simple-guide-on-connecting-rstudio-to-a-mysql-database-d8a8ca6573fd">connect RStudio to a MySQL database</a>. Make sure to read it.</p><h3><strong>Conclusion</strong></h3><p>Amazing! You have come a long way, and I am glad you came this far. I salute your doggedness and determination to get to the end of this illustration. I believe it has been worth your time learning about these concepts, which you found valuable.</p><p>I have a gift for you — I have uploaded the complete R script used for this illustration to a GitHub repository <a href="https://github.com/Predicare1/Connecting-RStudio-to-SQL-databases/blob/main/connect-RStudio-to-PostgreSQL.R">here</a>. This script saves you the stress of writing the codes, and you can quickly run the code. Of course, to practice with this script, you will need to tweak the script, like changing the database name, table name, and so on.</p><p>Furthermore, you can do a host of other things after creating the connection to the database. You can update a table or add a new record to a table in the database from RStudio. This article has given you the first and crucial step to maximizing the power of RStudio and SQL. From here, you can read the documentation of packages and functions to do different things with the database you have connected to from RStudio.</p><p>If you enjoyed this post, please drop your comments in the comment session and share this article with others. It would mean a lot to me if you could click on the “claps” icon (up to 50 claps allowed).</p><p>As a project-based course instructor with Coursera Guided Project Network, I have taught some courses on using R, especially the R tidyverse family of packages, including the dplyr package. You can check them out <a href="https://www.coursera.org/instructor/arimoro-olayinka">here</a> to take any of these courses. Thank You! See you at another time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e35ccdc08be" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/a-simple-guide-on-connecting-rstudio-to-a-postgresql-database-9e35ccdc08be">A simple guide on connecting RStudio to a PostgreSQL database</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The dangers of extrapolation in modelling]]></title>
            <link>https://medium.com/geekculture/the-dangers-of-extrapolation-in-modelling-dd3758b4ae44?source=rss-27bb4daa58e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd3758b4ae44</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[extrapolation]]></category>
            <category><![CDATA[biostatistics]]></category>
            <dc:creator><![CDATA[Arimoro Olayinka]]></dc:creator>
            <pubDate>Thu, 16 Jun 2022 19:58:18 GMT</pubDate>
            <atom:updated>2022-06-18T09:44:36.244Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*PWFIQgySK3IOHTixldqjbA.jpeg" /></figure><p>Welcome to the final article in the 3-series article discussing confounding and modification. In the first article, I explained the meaning of modification and confounding and discussed its’ implications in modelling. In the second article, we spent a lot of time walking through assessing modification and confounding in modelling using the logistic regression model in R and STATA. In the second article, we explored the relationship between HbA1c levels (exposure) and diabetic medication change (outcome) while assessing admission source and the number of patient visits as potential confounders and/or modifiers.</p><p>You can read the <a href="https://arimoroolayinka.medium.com/confounding-and-modification-in-modelling-explained-c37dbe7b40ff">first article here</a> and the <a href="https://arimoroolayinka.medium.com/a-case-study-assessing-confounding-and-modification-using-logistic-regression-in-r-and-stata-2be659cbb813">second article here</a>.</p><p>It is vital to have read these articles to have a better grasp of what we will discuss here today. This article will be as brief as possible, but I thought it would not be appropriate to introduce these concepts without highlighting the <strong>dangers of extrapolation in modelling.</strong></p><p>The question of exploring exploration in this article was sponsored when I saw the summary statistics of the patient visits which is a numeric covariate used as one of the covariates in the <a href="https://arimoroolayinka.medium.com/a-case-study-assessing-confounding-and-modification-using-logistic-regression-in-r-and-stata-2be659cbb813">second article</a>.</p><p><strong>Let’s see that result again below:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*XtWwl5KvAK8FiyXS8SX8Ng.png" /><figcaption><strong>Figure 1: </strong>Summary statistics including the five-number summary and the mean of the number of patients visits</figcaption></figure><p>We can deduce some interesting observations from the result presented above. It looks like most patients had zeros visits because the median is 0. Furthermore, it looks like there is an outlier or some spareness somewhere at the higher values. However, this trend is not so clear with summary statistics values. A visualization or graph will be much more informative to see what is going on.</p><p><strong>Let’s create a bar graph of the number of patient visits to check this trend.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*03LQfX3DUdiIW-UJQp4u4w.png" /><figcaption><strong>Figure 2: </strong>Bar plot of the number of patient visits</figcaption></figure><p><strong>Amazing! </strong>This graph projects the assertions I made earlier in a better form. We see that from Figure 2, most patients had zero visits and the range of most hospital visits is between 0 and 16 thereabout.</p><p>I decided to create the frequency table for this variable just to see how many patients and the number of times they have visited the hospital.</p><blockquote><strong>Note: Usually, it is not appropriate to explore a numeric variable using frequency tables.</strong></blockquote><p>I believe you get the idea for doing this in this discussion.</p><p><strong>Now, let’s get the frequency of patients and number of visits</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*n5UxvCirIq2o0Av9VlLZZA.png" /></figure><p><strong>The result is presented below. Although it was not well-formatted, we can pick quick insights from this result.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*KcXf3iCjtIW1gftMX48d-g.png" /><figcaption><strong>Figure 3: </strong>Frequency table of the number of patients visits</figcaption></figure><blockquote><strong>Ice Breaker: </strong>Look quickly through the table above, what can you observe?</blockquote><p>From Figure 3, we can see that most patients visited the hospital between 0–20 times. Although, the number of visits got scanty after the 16th visit (with 10 patients); however, we can see some patients up to the 20th visit. But after the 20th visit, we can see mostly 1 patient visiting the hospital. This explains why the maximum value from Figure 1 was 38. We had only 1 patient that visited the hospital 38 times.</p><p><strong>Now, this sets the stage for why we should be talking about extrapolation.</strong></p><h4><strong>So, what is extrapolation?</strong></h4><p>The dictionary definition calls it “the extension of a graph, curve, or range of values by <strong>inferring unknown values </strong>from trends in the known data.”</p><p>Notice the emphasis on <strong><em>inferring unknown values</em></strong><em>. </em>Wikipedia[1] defines it as a type of estimation, <strong>beyond the original observation range</strong>, of the value of a variable based on its relationship with another variable.</p><p><strong>What can we infer from those definitions?</strong></p><p>Simply put, during modelling, we could be making a huge mistake by making conclusions about the whole observations whereas a certain group of patients is underrepresented in the data.</p><p>For example, could it be that in our analysis of the relationship between HbA1c levels (exposure) and diabetic medication change (outcome), we are making a “fantasy” comparison between groups of the exposure variable? Probably, the number of patient visits is not comparable between the normal HbA1c group as compared to the elevated HbA1c group.</p><p><strong>So, how do we check whether extrapolation exists in the data especially when comparing the number of patient visits for the normal HbA1c and elevated HbA1c groups?</strong></p><p><strong>There could be various ways to achieve this through visualizations. However, I want to draw your attention back to the </strong><a href="https://arimoroolayinka.medium.com/a-case-study-assessing-confounding-and-modification-using-logistic-regression-in-r-and-stata-2be659cbb813"><strong>second article</strong></a><strong> and how we started modelling.</strong></p><p>Recall that the full/saturated model 1 is given as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*DJ1bNJ8cR9hL8b7JlyLSdQ.png" /></figure><p>After fitting the model, we found that admission source and patient visits do not jointly modify the relationship between HbA1c levels (exposure) and diabetic medication change (outcome).</p><p>Also recall from the <a href="https://arimoroolayinka.medium.com/a-case-study-assessing-confounding-and-modification-using-logistic-regression-in-r-and-stata-2be659cbb813">second article</a>, after we made this conclusion, we proceeded to assess for marginal or individual modification by these covariates and later confounding. However, we can insert a step in between the full model and the next model that we used to assess individual modification by the covariates.</p><p>Here, we will quickly project a graph of the estimated log odds on the y-axis to see how the log odds compare (to see if we can make conclusions from the data). This can be called <a href="https://en.wikipedia.org/wiki/Applications_of_sensitivity_analysis_in_epidemiology">sensitivity analysis</a> to explore the impact of extrapolation, that is to determine whether there was an unrealistic comparison between patients with normal HbA1c levels and elevated HbA1c levels when considering patients’ visits.</p><p><strong>Let’s see what the plot looks like below:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*WjEcZlEAKjeFpdz-ZNKpAg.png" /><figcaption><strong>Figure 4: </strong>Line plot of estimated log-odds for exposed (elevated HbA1c) and unexposed (normal HbA1c) patients by admission source and number of patients visits for the saturated model to check for extrapolation</figcaption></figure><p><strong>Note: </strong>The STATA code that produced the graphs including the codes that will be used in this article can be found <a href="https://github.com/Predicare1/Confounding-and-Modification-iin-Modelling">here</a>.</p><p>From Figure 4, it looks like the data overlap well for the exposed and unexposed groups. However, looking at the red lines (for non-emergency patients), we see that after around 20 visits, we may be making a “fantasy” comparison. Also, for the blue lines (emergency patients), after about 30 visits, we may be making a “fantasy” comparison.</p><p><strong>From the line plots produced above, it looks like some information is hidden. Let’s see this on a scatter plot to check what is going on.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*i8LGw4fKfdamZkqO8G-GMQ.png" /><figcaption><strong>Figure 5: </strong>Scatter plot of estimated log-odds for exposed (elevated HbA1c) and unexposed (normal HbA1c) patients by admission source and number of patients visits for the saturated model to check for extrapolation</figcaption></figure><p><strong>Aaah! This shows us clearly what is happening.</strong></p><p>From Figure 5, we can see the spareness of patient visits after about the 14th visit for the non-emergency unexposed patients. We can see the same trend of spareness for the emergency unexposed patients after about the 20th visit. Overall, the graph above clearly shows that the reason for that extrapolation is because of the small number of patients after 20 visits.</p><p>Therefore, it may be misleading to report our conclusions as we made in article two on the whole population of patients because the graphs above show us that irrespective of the source of admission, unexposed patients (i.e., patients with normal HbA1c levels) did not visit the hospital as much as the exposed counterparts.</p><p>To see whether this will impact our results in any way, we will build another logistic regression model with the number of patients visits less than or equal to 20. Then, we will go through the same process we did in article 2 to assess for confounding and modification.</p><p><strong>If there is a difference, then we may say that our results and conclusions are extrapolated (which doesn’t make sense) because it tells us that we are comparing patients with data with those without data.</strong></p><blockquote><strong>Disclaimer: </strong>Recall that we want to make this article as brief as possible, so we will not go through the same process we did in the second article. In this article, I will present the result of the new analysis.</blockquote><p>Note that the same process we took in assessing for modification and confounding applies here. <strong>The only difference is that we will filter the data to include only patients with less than or equal to 20 visits.</strong></p><p><strong>The code to create this filtered data is given below:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*2rShnBAmlVdCr0zMbdiHDw.png" /></figure><p><strong>Note: </strong>If you run this code, <strong>only 15 observations will be removed.</strong></p><p>So, the only thing that changes is the data used. In article 2, we used <strong>df_dia</strong>. To perform the same analysis, we did in the last article, you simply change the <strong>df_dia</strong> to <strong>df_dia20. </strong>Although I won’t include the code and results of the modelling steps in this article, I have provided the code to perform this analysis with the new data in the R script <a href="https://github.com/Predicare1/Confounding-and-Modification-iin-Modelling">here</a>.</p><h4><strong>Results</strong></h4><p>After running the sensitivity analyses for this model, to see whether the number of patient visits after 20 was unduly influencing our conclusions, therefore, we excluded every patient visit after 20 visits because we saw the sparseness of patient visits after 20 and reran the models (from the full to the final model). When we removed patients with more than 20 patient visits, we come to the same conclusions about the final model with the same interpretations as the model in article 2.</p><p>This may mean that extrapolation does not matter much in this case. This may be attributed to the small number of patients (15) that were removed when we filtered the data. We may get a different result if we drop the cut-off of patient visits to say 10. <strong>The choice of which cut-off depends on the curiosity of the researcher but mostly it should be informed by data like in this case.</strong></p><h4><strong>Conclusion</strong></h4><p>The question that may have come up is: “why do we have to go through this process when we came to the same conclusion as to when we used the whole data set?”.</p><p>The answer will be what if we did not come to the same conclusions? <strong>Do you think it would have been appropriate to report our conclusions the way we did in article 2?</strong></p><p>Of course not! The whole idea behind highlighting the dangers of extrapolation in modelling is to reduce the number of erroneous results and reports we see in research papers these days. In this case, if we got different results than what we had from the second article, then it would have been more appropriate to base our conclusions, reports, and discussions on the result we got here. But since the results are the same, usually the choice is left to the author to decide which one to report; however, the author should highlight that they performed a sensitivity analysis to check the impact of extrapolation and found none.</p><p><strong>What a time of learning it has been so far! If you read all three articles in this series, then I appreciate the time you have taken to read them. I believe it has been worth your time learning about these concepts. </strong>Please drop your comments in the comment section below and share this article with others. If you enjoyed this post, it would mean a lot to me if you could click on the “claps” icon (up to 50 claps allowed).</p><p>As a project-based course instructor with Coursera Guided Project Network, I have taught a couple of courses on using R. You can check them out <a href="https://www.coursera.org/instructor/arimoro-olayinka">here</a> to take any of these courses. Thank You! See you at another time!</p><h4><strong>References</strong></h4><ol><li>Extrapolation (Wikipedia): <a href="https://en.wikipedia.org/wiki/Extrapolation">https://en.wikipedia.org/wiki/Extrapolation</a>.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd3758b4ae44" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/the-dangers-of-extrapolation-in-modelling-dd3758b4ae44">The dangers of extrapolation in modelling</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A case study: Assessing confounding and modification using logistic regression in R and Stata]]></title>
            <link>https://medium.com/geekculture/a-case-study-assessing-confounding-and-modification-using-logistic-regression-in-r-and-stata-2be659cbb813?source=rss-27bb4daa58e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/2be659cbb813</guid>
            <category><![CDATA[tata]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <dc:creator><![CDATA[Arimoro Olayinka]]></dc:creator>
            <pubDate>Thu, 16 Jun 2022 18:46:01 GMT</pubDate>
            <atom:updated>2022-06-18T17:48:45.454Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*V83Ehzi5dvEsdHDbXJQvsg.jpeg" /></figure><p>Welcome, and thanks for taking the time to check this out. As I mentioned, this is the second article in a 3-series article. If you have not read the first article, you can do that <a href="https://arimoroolayinka.medium.com/confounding-and-modification-in-modelling-explained-c37dbe7b40ff">here</a>. The goal of this 3-series article is to demystify the concept of confounding and modification. But what is demystifying a concept without an illustrative example? Therefore, in this article, I will walk you through the steps to take when accessing confounding and modification in modelling.</p><p>Although, there are different modelling paradigms to take when checking for these modelling issues. Note that this example is not all-encompassing, but it demonstrates to a large extent what we need to pay attention to during modelling.</p><p>To set the tone for our discussion, the case study will be to model the relationship between HbA1c levels as the exposure and whether there was a change in the diabetic medications of patients while considering the admission source and number of patients visits as potential confounders and/or effect modifiers.</p><p>In this exercise, we will consider using the logistic regression model especially because of its interpretability. Although, there are mainframe procedures that have been developed to assess the effect of confounding and modification in modelling, including <a href="https://online.stat.psu.edu/stat507/lesson/3/3.5">stratified analysis</a>. Another rationale for choosing logistic regression as the appropriate model is because the outcome variable (diabetic medication change) has two categories (change and no change). Generally, <a href="https://www.ibm.com/topics/logistic-regression">logistic regression</a> models a binary outcome, which can take two values, such as true/false, yes/no, and so on. You can read more about the setting of the logistic regression model <a href="https://en.wikipedia.org/wiki/Logistic_regression">here</a>.</p><p>I believe that we can go on with the analysis, result, interpretation, and discussion, but first, let’s briefly talk about the data we will use in this analysis.</p><h4><strong>About the Data</strong></h4><p>The data set we will use for this analysis is an excerpt of the dataset provided during the Visual Automated Disease Analytics (VADA) summer school training, 2018. The VADA Summer School training dataset was derived from the Health Facts database (Cerner Corporation, Kansas City, MO, USA). This database contains clinical records from 130 participating hospitals across the USA over ten years (1999–2008). These clinical records contain information about 69,984 observations and 27 variables, including patient encounter data, demographics, HbA1c levels, diagnostic testing and treatments, and patient outcomes. Data used were from 1999–2008 from a cohort of 130 hospitals, deidentified and trimmed to include only inpatient visits. The authors of the dataset performed several steps in pre-processing the data; if you are interested, you can read these steps <a href="https://www.hindawi.com/journals/bmri/2014/781670/">here</a>.</p><p>To arrive at the dataset used for this analysis, a couple of my friends discussed and implemented the steps I am about to show you. First, we trimmed the VADA dataset to include only encounters where HbA1c was measured, and we reduced the number of covariates to include the most medically relevant features. This reduced the dataset to 17,018 observations and 9 variables. Furthermore, we found about 138 (approximately 0.8%) missing values in the admission source variable, and the race variable had 422 (approximately 2.5%) missing values. Since this is not a substantial level of missingness, we dropped these missing values from downstream analysis.</p><p>Therefore, the final data set used for analysis contained 16,458 observations and 9 variables representing race, age, sex, admission source, length of hospital stays, HbA1c results, hospital readmission status, diabetic medications administration, and the number of previous hospital visits. We merged the number of previous inpatients, outpatient, and emergency visits for each individual into one variable called patient visits.</p><p>To reduce the complexity of the model, the covariates with more than two levels or categories were recoded into two levels. For example, we categorized patients’ age by the mean age of diagnosis of 60. For race, patients were categorized as either Caucasian or Others (including African American, Asian, Hispanic, and other visible minorities). Finally, for admission sources, patients were categorized in a triage fashion (emergencies and other immediate admissions or other referral-based admissions) to attempt to group inpatients admitted through similar circumstances/urgencies. The dataset has two categories for gender (Male and Female). You can access the data for free <a href="https://github.com/Predicare1/Confounding-and-Modification-iin-Modelling">here</a>.</p><h4><strong>Analysis Procedure</strong></h4><p>To adequately explore our discussion here, we will define our outcome variable as a diabetic medication change. Diabetic medication has two categories (Yes and No) which indicate if diabetes medication was administered (changed) or not during an inpatient encounter. The exposure variable is the HbA1c level which contains results as “normal” if levels are within normal expected ranges (&lt; 7.0 %) or as “elevated” if HbA1c levels are higher than normal (&gt; 7.0 %). As mentioned earlier, for this discussion, the covariates (and potential effect modifiers/confounders) will be the admission source and the number of previous patient hospital visits.</p><p>A <a href="https://www.ibm.com/topics/logistic-regression">logistic regression</a> model will be used to evaluate the relationship between diabetic medication change (yes/no) as the outcome and HbA1c levels as the exposure while controlling for covariates, admission source<strong>, </strong>and the number of patient visits. Logistic regression fits the objective because the outcome variable (diabetic medication change) is binary. We will assess for effect modification and confounding by the covariates. The logistic regression assumption of data independence was satisfied since each observation is from a unique inpatient. The linearity assumption of logistic regression between the logit of the outcome and the number of patient visits was validated graphically. All analyses will be performed using both R programming language[1] and STATA software[2].</p><p><strong>Now that we have a solid understanding of the data and the model we will use in this discussion. It is important to highlight how we will access confounding and modification.</strong></p><blockquote><strong>Disclaimer: Many modelling paradigms can be used to access confounding and modification, but this is the one I have come to learn about and found appealing (at least for the context of the case study).</strong></blockquote><p>To start, we will create two-way and three-way interaction terms between the exposure variable and the covariates. <strong>(I will show you how to do that during the modelling phase of this article).</strong></p><p>The Wald test of significance will assess effect modification by comparing the p-value of the interaction terms against the pre-determined level of significance (0.05). If the p-value of the three-way interaction term between exposure (HbA1c) and covariates (admission source and patient visits) is significant (i.e. if the p-value &lt; 0.05), we will conclude that there is a joint modification by the covariates. Then, we will stop modelling and report the full model. However, if the three-way interaction term is not significant (i.e., if the p-value &gt; 0.05), we will conclude that there is no joint effect modification. Then, we will drop the three-way interaction terms and proceed to assess for individual effect modification using a similar approach and subsequently confounding.</p><p>To assess for confounding, we will check the coefficient of the exposure (HbA1c) to see whether there is a notable (&gt; ±10%) change as we remove covariates from the model. Note that if we find that a covariate modifies the HbA1c-diabetic medication relationship, we will keep that interaction term in the model and not assess the covariate in question for confounding (although other covariates can be assessed). Then, in this case, we will assess for confounding by taking note of any appreciable (&gt; ±10%) change in the coefficient of the exposure and the modifier.</p><p><strong>To put action to the ideas I have shared above, we will now approach the modelling phase.</strong></p><h4><strong>Modelling</strong></h4><p>In this section, I will walk you through the steps to take to arrive at the final model that best describes the relationship between HbA1c levels and diabetic medication change.</p><p><strong>Recall that:</strong></p><p><strong>Outcome</strong>: diabetic medication change (0 — no change, 1 — change)</p><p><strong>Exposure</strong>: Hba1c levels (0 — normal levels, 1 — elevated levels)</p><p><strong>Modifier/Confounder</strong>: Admission source (0 — emergency, 1 — non-emergency) and number of patient visits (actual)</p><p>We will start with importing the required packages, importing the data into R, and exploring the data. Then, we will create the interaction terms. You may access the full R codes on GitHub <a href="https://github.com/Predicare1/Confounding-and-Modification-iin-Modelling">here</a>.</p><h4><strong>Import the required packages</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*A-VSatbLWRGrh2NMbwf3RA.png" /></figure><p>The code above imports these packages. Note that if you do not have prior experience with using R, you can take this guided project titled <a href="https://www.coursera.org/learn/getting-started-with-r/home/welcome">Getting Started with R</a> that I taught on the Coursera platform. The guided project is completely hands-on and will take about 2 hours to complete. The guided project has helped many learners get started with using R.</p><h4><strong>Import and Explore the data set.</strong></h4><p>Now, let’s import the dataset and briefly explore it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*r9WR1D4c3svM8_cz7HI_ew.png" /></figure><p><strong>Let’s see the output.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*72PqxSKw0BXIwGNaK8vQ_Q.png" /></figure><p>One thing you will notice is that the data set contains some other variables. However, for this article, we will focus on the following variables <strong>diabetesMed, HbA1c, admit_source, and patients_visits. </strong>You can download the data set <a href="https://github.com/Predicare1/Confounding-and-Modification-iin-Modelling">here</a>.</p><p><strong>Now that you have a glimpse of what the data set contains, we can proceed to create the interaction terms.</strong></p><p>The R code below creates the interaction terms and converts categorical variables to factors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*uE-tfa-5BSTlKQucUF9OPQ.png" /></figure><p><strong>Now, let’s take a glimpse at the data</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*y447Rwj4Z6LI6-qyyFDwMg.png" /></figure><p>You will agree with me that the two-way interaction terms <strong>(padmit, hadmit, hpatient) </strong>and the three-way interaction term <strong>(hpadmit) </strong>are properly created.</p><p>Before proceeding with the modelling, let’s take a quick stop to create basic descriptive statistics for the covariates. The code below will create a frequency table for categorical variables <strong>(diabetesMed, HbA1c, admit_source)</strong> and the five-number summary, including the mean for the numeric variable <strong>(patient_visits).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*pfNYc7abr7_T1YHUOjx1XQ.png" /></figure><p>I have presented the result of the code above in a better format below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*Fg784aKBYdcm3rO95ULSkQ.png" /></figure><p>In the table above, I reported the frequency and percentages for the categorical variables and the median and interquartile range for the numeric variable. As a curious person, I decided to check the distribution of diabetic medication changes based on HbA1c levels. The code below helps us achieve this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*ts8rUx5JYQYeCeOSvbUfiw.png" /></figure><p>The result is shown in the table below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*UuTN8QvmJgWePwA1jt1VrQ.png" /></figure><p>One quick insight here is that we have most patients that had a change in their diabetic medication had elevated HbA1c levels 10121 (61.5%). I guess this result should not be surprising.</p><p><strong>Next, we can define the full model that will be used in this analysis.</strong></p><p>The full/saturated model 1 is given as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*BU-SziK3aHWWmKgzJKuj4Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*2NCHJVtPBjGE5xjO9BwmlA.png" /></figure><p><strong>Now, let’s see how to fit this full model in R.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*xY-T2I-BOl3UJV3Jp9r_qQ.png" /></figure><p>The logistic regression model is correctly specified using the <strong>glm()</strong> function. The outcome variable is specified on the left of the equation, and all the covariates specified on the right are separated by the plus (+) sign. The data <strong>df_dia </strong>was specified, including the family = “binomial”.</p><p>Now, let’s print the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*Qbp8ASXR14tdHf3vD2zzJA.png" /><figcaption><strong>Figure 1:</strong> Fitted logistic regression model for assessing admission source and patient visits as potential joint modifiers of the relationship between diabetic medication change (outcome) and HbA1c levels (exposure)</figcaption></figure><p><strong>Whew! So many results here. However, for now, we will focus on the part highlighted in red. </strong>Note that the 1 in front of HbA1c, admit_source, and hadmit tells us that R used 0 as the reference group. Remember that those variables had two categories (0 and 1), so one of the categories needs to be used as a reference category.</p><blockquote><strong>Teaser: What do you think will have if we change the reference category? I will leave that to you to think about!</strong></blockquote><p><strong>Let’s move on but wait a moment! Can you recall how I mentioned that we would assess for effect modification?</strong></p><p>The output above (Figure 1) shows the R output obtained from using the logistic regression model to fit the association between the diabetic medication change (outcome) and HbA1c (exposure) while assessing admission source and patient visits as a potential effect modifier of the relationship. If you noticed that I mentioned earlier that to assess for effect modification, <strong>we will focus on the p-value of the interaction terms</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*wSdQ0EJ3nWw5E7hMDAYu4Q.png" /></figure><h4><strong>Model 1: assessing admission source and patient visits as potential effect modifiers separately</strong></h4><p>We have seen that there is no joint modification. In this step, I will remove the three-way interaction term <strong>(hpadmit) </strong>and check whether the p-values of <strong>hpatient </strong>and <strong>hadmit </strong>are significant or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*d7gckwpAI3y1Yfd0NuHwxw.png" /></figure><p>Beautiful! Just like we saw with the three-way interaction terms, we will check the p-values to see whether it is greater than 0.05 or not. If it is greater than 0.05, then we will conclude that there is no effect modification by that variable. Otherwise, we will conclude that the covariate modifies the exposure-outcome relationship. We will fit the model with the code below. Notice that we didn’t add <strong>hpadmit </strong>(the three-way term).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*S8mUF9wgGtn8FCzz_FxJIw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*IMovqNaXpWRrK14RsF72lg.png" /><figcaption><strong>Figure 2:</strong> Fitted logistic regression model for assessing patient visits as a potential modifier of the relationship between diabetic medication change (outcome) and HbA1c levels (exposure)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*kIJUMQZcXmzNGAW8s6rz_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*ftrithkm_8FW0wveND72tA.png" /><figcaption><strong>Figure 3:</strong> Fitted logistic regression model for assessing admission source as a potential modifier of the relationship between diabetic medication change (outcome) and HbA1c levels (exposure)</figcaption></figure><p><strong>Now that we have seen these two model results. We can make some conclusions, right?</strong></p><p>The p-values of the two-way interaction between HbA1c, patient visits, and admission source <strong>(hpatient, hadmit) </strong>from the output above (Figures 2 and 3) are 0.00000332 and 0.187, respectively. This suggests that patient visits modify the outcome-exposure relationship, while admission sources do not modify the relationship between HbA1c levels (exposure) and diabetic medication change (outcome).</p><p><strong>Why do you think I have made such a conclusion? Think about that for a minute before you continue reading.</strong></p><p>The p-value of the admission source is not statistically significant since the p-value (0.187) is greater than 0.05 (level of significance). However, for the number of patient visits, we see a different p-value (0.00000332) which is way lesser than 0.05. We will conclude that the number of patient visits is significant. <strong>Isn’t this interesting?</strong></p><p>Although these two covariates, when put together, do not modify the relationship; however, we see that while the admission source does not modify the relationship, the number of patient visits modifies the relationship between HbA1c levels and diabetic medication change.</p><p>Therefore, we can proceed to assess for confounding by these variables. To assess for confounding, we will refit the model by dropping the covariate terms one step at a time and check whether the coefficient of the exposure (<strong>HbA1c</strong>) and <strong>hpatient</strong> changes substantially. Usually, if we see a change greater than ±10% in these coefficients, then we will conclude that the covariate confounds the relationship.</p><p><strong>Let me quickly emphasize an important point I mentioned earlier.</strong></p><blockquote><strong>Note that, if we find that a covariate modifies the HbA1c-diabetic medication relationship, we will keep that interaction term in the model. Then, in this case, we will assess for confounding by taking note of any appreciable (&gt; ±10%) change in the coefficient of the exposure and the modifier.</strong></blockquote><p>This justifies why we need to still include the <strong>hpatient </strong>variable in the model while assessing for confounding. Also, note that in this case, we will not assess whether patient visits confound the relationship <strong>because a covariate cannot confound and modify the exposure-outcome relationship simultaneously</strong>.</p><blockquote><strong>Quick Question: </strong>Assuming patient visits also did not modify the relationship, how do you think we will assess for confounding?</blockquote><p>It’s quite straightforward in that case. We will drop all the interaction terms (both the two-way and three-way interaction terms) between the exposure and the covariates and check if there is any notable difference (&gt;±10% change) in the value of the exposure (in this case, HbA1c levels) as we refit the model.</p><p>Finally, it is important to note that to assess for confounding, <strong>we don’t have any business with the p-values</strong>. We are interested in the exposure coefficient (and in this case, <strong>hpatient </strong>because <strong>patient_visits</strong> is a modifier).</p><p>With this explanation, we can proceed to assess for confounding. I will report a more comprehensive table 3 to clearly see whether there was a notable change in the coefficients or not.</p><p><strong>Now, let’s fit model 2 without the interaction terms hadmit and hpadmit, but we will still include patient_visits and hpatient.</strong></p><h4><strong>Model 2: Refit the model by dropping all interaction terms between HbA1c levels and admission source and keeping the interaction term between HbA1c levels and patient visits.</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*Rc9vUuYIRmfyy-swQL0sqA.png" /><figcaption><strong>Figure 4:</strong> Fitted logistic regression model to start assessing whether admission source confounds the relationship between diabetic medication change (outcome) and HbA1c levels (exposure).</figcaption></figure><p><strong>You will notice that I did not print the full result of the model. I used the coef() function to return only the coefficients.</strong></p><p><strong>Focus on the values in red highlight. Aah! I guess we may lose track of those values as we progress. Don’t worry; I will put up a table to summarize these results to see them.</strong></p><h4><strong>Model 3: Refit the model by dropping the interaction terms between admission source and patient visits (padmit)</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*o3b-_n2ekQ_m1y7DAXbb6g.png" /></figure><p><strong>Before we continue to see the result, I sense that someone may be thinking — but we dropped all interaction terms?</strong></p><p>I think it is imperative to emphasize this again. When we assessed for effect modification, we were concerned with the interaction terms between the <strong>exposure (HbA1c) and the covariates (hpadmit, hadmit, hpatient) </strong>and not the interaction term between the covariates <strong>(padmit).</strong></p><p>Notice that the <strong>‘h’</strong> in front of hpadmit, hadmit, and hpatient represent the exposure (HbA1c levels). I think you can review the part where we created the interaction terms to get a hold of the variable names.</p><p><strong>I want to believe that my explanation has helped clarify the possible confusion with the variable names. Now, we will refit the model and drop padmit to see whether admission source together with patient visits jointly confounds the relationship between the exposure and outcome.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*5bh-O9O4NmDL2m5Pg_RtkA.png" /><figcaption><strong>Figure 5:</strong> Fitted logistic regression model to assess whether admission source together with patient visits confounds the relationship between diabetic medication change (outcome) and HbA1c levels (exposure).</figcaption></figure><p><strong>Could you notice a substantial change in those coefficients? Maybe not?</strong></p><h4><strong>Model 4: Refit the model by dropping the admission source variable.</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*AJ3FUQVq5Nseg6Alaut9wQ.png" /><figcaption><strong>Figure 6:</strong> Fitted logistic regression model to assess whether admission source confounds the relationship between diabetic medication change (outcome) and HbA1c levels (exposure).</figcaption></figure><p><strong>Amazing! We have come to the end of the modelling process. </strong>We cannot continue modelling by dropping the patient visits variable for obvious reasons. It is a modifier and a variable that modifies a relationship and cannot confound the relationship simultaneously. This explains why patient_visitis and patient remained in the model while assessing whether the admission source is a confounder. And of course, we cannot drop the exposure variable (HbA1c). <strong>Now, let us summarize the result to know which model we will pick as the final model.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*WGkMpHjJZbkFC_aBQm5jjQ.png" /></figure><p>Therefore, we should report the last model we fitted (model 4) as the final model.</p><p><strong>Why?</strong></p><p>This brings us back to the discussion about <strong>adjustment </strong>that I talked about in the <a href="https://arimoroolayinka.medium.com/confounding-and-modification-in-modelling-explained-c37dbe7b40ff">first article</a>. Model 4 is the final model of choice because it is the more-adjusted model after assessing for modification and confounding.</p><p>The final model to be reported will be the model given in Figure 6. Again, we chose this as the final model based on the conclusion from our modelling that the admission source neither confounds nor modifies the relationship between HbA1c levels (exposure) and diabetic medication change (outcome). However, the number of patient visits modifies this relationship; therefore, it cannot confound the relationship at the same time. The final fitted model is reported below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*oJATdjyKs17TFjnahyeONw.png" /></figure><p>Although there are other measures of associations, including rate ratios and risk ratios, I used odds ratios here because it is the most appropriate for the logistic regression model. You can read other measures of association <a href="https://www.cdc.gov/csels/dsepd/ss1978/lesson3/section5.html">here</a>.</p><p>The corresponding confidence interval is given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*m1mXllgn75Alrs82rl_0Lw.png" /><figcaption><strong>Figure 7:</strong> Confidence intervals for the final fitted logistic regression model assessing the relationship between HbA1c levels (exposure) and diabetic medication change (outcome) in the presence of admission source and patient visits as potential confounders.</figcaption></figure><p>Before we interpret the beta coefficients, let us provide the <strong>odds ratio</strong> version of this final model because what we have currently is <strong>the log odds ratio</strong>. All we need to do is to <strong>exponentiate </strong>the coefficients to get the odds ratio. The code to produce the odds ratio and the odds ratio is given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*UiTQWjzFg9BEQMX796Ey-A.png" /><figcaption><strong>Figure 8:</strong> Odds ratio for the final fitted logistic regression model assessing the relationship between HbA1c levels (exposure) and diabetic medication change (outcome) in the presence of admission source and patient visits as potential confounders.</figcaption></figure><p>Now, we can interpret the estimated beta coefficients as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*GhNtC4BiQdHhKAliatd-GA.png" /></figure><p><strong>Discussion of results</strong><br>After fitting the logistic regression model, we found that the number of patient visits modified the relationship between HbA1c levels (exposure) and diabetic medication change (outcome), while the admission source did not modify nor confound this relationship. The result of the final logistic regression model shows that all variables were statistically significant. From Figure 6, we found that the estimated log odds of diabetic medication change for patients with normal HbA1c levels and zero patient visits is 0.994 (95% CI: 0.921,1.067; p-value &lt;0.000), with a corresponding odd of 2.702.</p><p>Furthermore, we found that the estimated log odds ratio of change in diabetic medication with patients with zero patient visits comparing patients with elevated HbA1c and normal HbA1c levels is 0.927 (95% CI: 0.832,1.021; p-value&lt;0.000), with a corresponding odds ratio of 2.526. The estimated difference in log odds of diabetic medication change per one patient visit change in patient visits for patients with normal HbA1c levels is 0.134 (95% CI: 0.085,0.183; p-value&lt;0.000). Finally, the estimated difference in log odds ratios of diabetic medication change per one visit change in patient’s visit comparing patients with elevated HbA1c and normal HbA1c levels is -0.132 (95% CI: -0.187,-0.077; p-value&lt;0.000).</p><p>Since the final logistic regression model reported above shows unambiguous evidence of modification by patient visits, therefore we will report different odds for patients with elevated HbA1c levels and those with normal HbA1c levels. The estimated difference in odds of diabetic medication change per one visit change in patient visits for patients with normal HbA1c levels is 1.143 (95% CI:1.089,1.201; p-value&lt;0.000), while the estimated difference in odds of diabetic medication change per one visit change in patient’s visits for patients with elevated HbA1c levels is 1.002 (95% CI:0.977,1.027;p-value=0.887).</p><p>We can see that the estimated difference in odds of diabetic medication change increases per change in patient visits for patients with normal HbA1c levels (Figure 9). Furthermore, this estimated difference in odds of diabetic medication change for every increase in patient visits for patients with normal HbA1c was slightly higher than for patients with elevated HbA1c levels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*ErCoBmpvyRgxHNRuzoM3fw.png" /><figcaption><strong>Figure 9: </strong>Estimated log-odds (left) and odds (right) for exposed (elevated HbA1c) and unexposed (normal HbA1c) patients by number of patients visits for the final logistic regression model</figcaption></figure><p><strong>Note: </strong>I created the plots above using the STATA software. I have provided the STATA codes to run this same full analysis I have done with R. You can access these STATA codes <a href="https://github.com/Predicare1/Confounding-and-Modification-iin-Modelling">here</a>.</p><p><strong>Implication and Conclusion</strong><br>We found that HbA1c levels were significantly associated (p-value&lt;0.05) with each of the diabetic inpatient outcomes we investigated. Therefore, as the general population continues towards the increased prevalence of diabetes, we can expect a higher influx of diabetic inpatients across the globe. This alarming trend necessitates serious action toward the development, optimization, and implementation of targeted blood-glucose management strategies.</p><p>HbA1c elevation is significantly associated (p-value&lt;0.05) with increased odds of diabetic medication administration. However, odds of diabetic medication administration remained relatively constant for inpatients with elevated HbA1c regardless of how many previous visits these individuals made to the hospital before the study period. This may suggest that future glucose management protocols can improve upon existing strategies by evaluating which patients are at risk of recurring readmission and treating those accordingly.</p><p><strong>Amazing! I salute your doggedness and determination to get to the end of this lengthy yet insightful case study. I believe it has been worth your time learning about these concepts. </strong>Please drop your comments in the comment section below and share this article with others. If you enjoyed this post, it would mean a lot to me if you could click on the “claps” icon (up to 50 claps allowed). If you would like to read the <a href="https://arimoroolayinka.medium.com/the-dangers-of-extrapolation-in-modelling-dd3758b4ae44">third article</a> in this series, click <a href="https://arimoroolayinka.medium.com/the-dangers-of-extrapolation-in-modelling-dd3758b4ae44">here</a>.</p><p>As a project-based course instructor with Coursera Guided Project Network, I have taught a couple of courses on using R. You can check them out <a href="https://www.coursera.org/instructor/arimoro-olayinka">here</a> to take any of these courses. Thank You! See you at another time!</p><h4><strong>Acknowledgements</strong></h4><p>The first acknowledgement is to the authors of the VADA dataset for making the dataset open source. Also, I am grateful to course instructors Dr Tyler Williamson and Dr Jessalyn Holodinsky for instructing and facilitating different modelling approaches that were instrumental to the success of this write-up. Also, they helped edit the articles in this series. I want to thank my supervisors, Dr Tolulope Sajobi and Dr Colin Josephson, for giving me access to world-class training in Biostatistics. Special thanks to Ayoola Ademola, Joel Adekanye, Tate Hubkarao, Samuel Babatunde, Jason Black and Olawale Ayilara for sharing their knowledge in this area. Finally, I want to thank Gabriel Afriyie and Ben Caddey for validating some of my thoughts and adding their ideas to parts of this article.</p><h4><strong>References</strong></h4><ol><li>R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria. 2021. URL <a href="http://www.r-project.org/">http://www.R-project.org/</a>.</li><li>StataCorp. 2021. Stata Statistical Software: Release 17. College Station, TX: StataCorp LLC.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2be659cbb813" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/a-case-study-assessing-confounding-and-modification-using-logistic-regression-in-r-and-stata-2be659cbb813">A case study: Assessing confounding and modification using logistic regression in R and Stata</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Confounding and Modification in modelling explained]]></title>
            <link>https://medium.com/geekculture/confounding-and-modification-in-modelling-explained-c37dbe7b40ff?source=rss-27bb4daa58e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c37dbe7b40ff</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[biostatistics]]></category>
            <dc:creator><![CDATA[Arimoro Olayinka]]></dc:creator>
            <pubDate>Thu, 16 Jun 2022 07:57:02 GMT</pubDate>
            <atom:updated>2022-06-18T09:44:38.194Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*Gc4RM8_qnPJ05RTBlPb8HQ.jpeg" /></figure><p>Have you heard about confounding and modification and would like to learn about them? Or did you click on this link out of curiosity? Then, hold on! This will be a short and precise introduction to understanding confounding and modification in modelling.</p><p>Recently, I learned about confounding and effect modification and their impacts on modelling and interpretation of results. I thought about writing a 3-series article demystifying these concepts including an illustrative example of confounding and effect modification with a case study.</p><p>This will be the first in the series and this article will explain in layman’s terms what we mean by modification and confounding while emphasizing the impact on modelling. Effect modification influences while confounding distort the nature of the relationship or association between the exposure and outcome across various groups. The term exposure is broadly said to be any characteristic that may explain or predict the presence of a study outcome[1].</p><p>For example, a patient’s smoking status could be the exposure and the outcome could be a disease for example cardiovascular disease (CAD). Therefore, we may be interested to know whether the sex of the patient modifies or confounds the relationship between the smoking status of a patient and whether they have CAD or not.</p><p>Another example could be to see whether factors like admission source and/or the number of patients visits are potential modifiers or confounders of the relationship between the haemoglobin A1c (HbA1c) levels as the exposure and whether there was a change in their diabetic medications as the outcome.</p><blockquote><strong>Note: The haemoglobin A1c test tells you your average level of blood sugar over the past 2 to 3 months. It’s also called HbA1c, glycated haemoglobin test, and glycohemoglobin.</strong></blockquote><p>I believe that was a beautiful introduction to our discussion today. Let’s dive right into the content here. If you are ready, then, let’s begin.</p><h4><strong>What is effect modification?</strong></h4><p>Effect modification talks about the heterogeneity of the exposure-outcome association that depends on the value of some extraneous variable. For example, if sex modifies — we expect a different exposure-outcome association among males than females. When there are more than two levels, we expect a different exposure-outcome association at each level of a categorical variable. Simply put, effect modification occurs when <a href="https://www.cdc.gov/csels/dsepd/ss1978/lesson3/section5.html">measures of association</a> are substantially different across strata (groups).</p><p>In this case, the nature of an association can differ (sometimes substantially) across different subgroups from the overall populations (see Figures 1 and 2). This occurs when an association between an exposure X and an outcome Y can depend on another variable Z, and Z does not have to be directly related to X and/or Y.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*OW2IOTltnt66fGJ9Td4LeQ.png" /><figcaption><strong>Figure 1: </strong>Modification of the relationship between time spent studying and exam score by sex (a categorical variable)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*KnMDlgO-yIrUzHqpgSL5hA.png" /><figcaption><strong>Figure 2: </strong>Modification of the relationship between treatment received and CD4 count by age (a continuous variable)</figcaption></figure><blockquote><strong>Note: The CD4 count is a test that measures how many CD4 cells you have in your blood. In short, the higher the CD4 count, the healthier the immune system.</strong></blockquote><p>Clearly, Figures 1 and 2 illustrate that an exposure-outcome relationship can be modified by categorical and continuous variables.</p><p>The question that may come up is, <strong>how do we know that modification exists?</strong></p><p>Let me ask you a question to explain this. From Figure 2, which treatment do you think is better? (assuming a higher CD4 count means better treatment).</p><p>I guess that you would respond that, before the age of 45, treatment 0 was better. However, after the age of 45, treatment 1 became the better treatment. Simply, this means that treatment 0 seems to be a better treatment for younger people and treatment 1 for older people.</p><p>Can you imagine the clinical error that would have occurred if we didn’t account for these differences between treatment and CD4 count given age?</p><p>Therefore, ignoring an effect modifier can result in estimating one overall Y/X relationship for everyone when separate Y/X relationships for different values of Z better describe the phenomenon under study.</p><p>Are you are still following the discussion? If No, please take a step back to digest what you just read. If yes, then we can proceed to talk about confounding.</p><h4><strong>What is confounding?</strong></h4><p>Simply, confounding is a distortion of the exposure-outcome association by an extraneous variable. For example, if sex confounds — we expect the same exposure-outcome association among males and females, but we would see a different association when not considering sex. Simply put, the <a href="https://www.cdc.gov/csels/dsepd/ss1978/lesson3/section5.html">measures of association</a> (could be odds, odds ratios, etc.) are similar across strata (groups) but the pooled measure of association is different from the crude measures of association.</p><p>More technically, confounding sometimes referred to as the <a href="https://plato.stanford.edu/entries/paradox-simpson/">Simpson’s Paradox</a> is a phenomenon where the true relationship or association between the exposure X and the outcome Y is distorted by another lurking (hidden) variable Z (or variables Z1, Z2,…, Zn). This can happen if Z is related to both X and Y. In this case, we may take Z as the patient’s smoking status, X as the alcohol use, and Y the whether the patient has lung cancer or not (see Figure 3).</p><p>From Figure 3, we see that the patient&#39;s smoking status is related to alcohol use and lung cancer. However, Simpson’s paradox is one specific and extreme example of confounding because not all confounding can be described this way.</p><p><strong>However, for confounding to occur, these three things must happen[2]:</strong></p><ol><li>The confounding factor (or variable) must be associated with <strong>both</strong> the risk factor of interest (exposure) and the outcome.</li><li>The confounding factor must be distributed unequally (imbalance) of the confounding variable among the groups being compared.</li><li>A confounder cannot be an intermediary step in the causal pathway from the exposure of interest to the outcome of interest.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*zkiIDNzxp4BgMK3VtYLzEw.png" /><figcaption><strong>Figure 3:</strong> An example of a patient’s smoking status confounding the relationship between lung cancer and alcohol use (<strong>Source:</strong> <a href="https://www.sciencedirect.com/topics/pharmacology-toxicology-and-pharmaceutical-science/confounder">https://www.sciencedirect.com/topics/pharmacology-toxicology-and-pharmaceutical-science/confounder</a>)</figcaption></figure><p><strong>In summary, while effect modification is a function of “nature”, “circumstance”, or “setting”, confounding is sometimes a function of study design. </strong>Check <a href="https://www.ncbi.nlm.nih.gov/books/NBK470342/">here</a> for more thoughts on study designs in epidemiological studies.</p><p>Usually, we wish to study exposure-outcome relationships; however, we are concerned that such a study may require the consideration of other characteristics. The rule of thumb is that when the measures of association are roughly homogenous (similar) across strata and the pooled measure of association is similar to the crude, then there is no evidence of modification or confounding, otherwise confounding exists.</p><p><strong>Next, let me quickly talk about adjustment.</strong></p><p>Adjustment is a method for making comparable comparisons between groups in the presence of confounder/confounding variables. Adjustment provides a mechanism for estimating an exposure-outcome relationship after removing the potential distortion that comes from a confounder or multiple confounders. Contrasting crude and adjusted association estimates are useful for identifying confounding <strong>(we will see more of this during the illustrative example)</strong>.</p><p><strong>Great discussion so far, right?</strong></p><p>Now that I have introduced you to the impacts that confounding and modification can have on modelling, I think it is appropriate to outline some possible solutions to address confounding and modification.</p><h4><strong>Possible solutions to confounding and modification</strong></h4><p>There are several solutions to these modelling issues, but I have highlighted some of them below:</p><p>1. Confounding can be minimized by designing a randomized study</p><p>Essentially, a randomized study is one that randomly assigns participants to separate groups (for example an experimental group or a control group). Randomization is a powerful method to control and prevent confounding. Recall that confounding can occur when we have imbalance between groups.</p><p><strong>The question is — how does randomization solve this issue of imbalance?</strong></p><p>For example, if participants are allocated to groups by a random method that gives an equal chance of being in any group, then it is likely that the groups will have similar distributions of age, gender, behaviours, and virtually all other known and as yet unknown possible confounding factors. Also, the researcher can get a sense of whether randomization has successfully created comparability among the groups by comparing their baseline characteristics[2].</p><p>2. Potential confounders of the key relationships must be conceptualized before the start of the study and must be measured as part of the study.</p><p>3. Since effect modification is not affected by study design, therefore, it could be useful to have a sense of any potential effect modification of interest before designing a study</p><p>4. Assess whether there is effect modification by looking at the Y/X association separately for each level of Z and decide whether at least some of the associations differ. Common methods used include stratified analysis and regression-based methods.</p><p>Various methods can be used for this purpose. For example, the Cochran-Mantel-Haenszel Method[3] is used in stratified analysis. To understand how stratified analysis is used, you can read this comprehensive <a href="https://sphweb.bumc.bu.edu/otlt/mph-modules/bs/bs704-ep713_confounding-em/bs704-ep713_confounding-em_print.html">article</a>[2]. In the next article, I would explore in greater detail using logistic regression to assess for modification and confounding using R and STATA.</p><p><strong>The solutions outlined above are some suggestions to consider. Finally, let’s examine some differences between confounding and effect modification</strong></p><p><strong>Table 1: </strong>Differences between confounding and effect modification</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*QcA1hsAMSmAFpN2fL01QIA.png" /><figcaption><strong>Source: </strong>Week 3 of the <a href="https://www.coursera.org/learn/simple-regression-analysis-public-health/home/welcome">Simple Regression Analysis in Public Health</a> Course on Coursera[4]</figcaption></figure><p>This article is a high-level introduction to these concepts. I believe it has been worth your time learning about modification and confounding — <strong>I guess that’s additional knowledge added for you!</strong> Please drop your comments in the comment section below and share this article with others. If you enjoyed this post, it would mean a lot to me if you could click on the “claps” icon (up to 50 claps allowed).</p><p>The next article is super-detailed on how to assess for modification and confounding using a regression-based method, the logistic regression model using R and STATA. If you’d like to check that out, please click <a href="https://arimoroolayinka.medium.com/a-case-study-assessing-confounding-and-modification-using-logistic-regression-in-r-and-stata-2be659cbb813">here</a>.</p><p>Thank you for taking the time to read this! See you at another time!</p><p><strong>References</strong></p><ol><li>Kestenbaum, B. (2019). Population, Exposure, and Outcome. In: Epidemiology and Biostatistics. Springer, Cham. <a href="https://doi.org/10.1007/978-3-319-97433-0_2">https://doi.org/10.1007/978-3-319-97433-0_2</a></li><li>LaMorte WW, Sullivan L. Confounding and Effect Measure Modification. Adapted from URL: <a href="https://sphweb.bumc.bu.edu/otlt/mph-modules/bs/bs704-ep713_confounding-em/bs704-ep713_confounding-em_print.html">https://sphweb.bumc.bu.edu/otlt/mph-modules/bs/bs704-ep713_confounding-em/bs704-ep713_confounding-em_print.html</a></li><li>Cochran–Mantel–Haenszel statistics (Wikipedia): https://en.wikipedia.org/wiki/Cochran–Mantel–Haenszel_statistic</li><li><a href="https://www.coursera.org/instructor/johnmcgready">McGready</a> J. Simple Regression Analysis in Public Health. URL: <a href="https://www.coursera.org/learn/simple-regression-analysis-public-health/home/info">https://www.coursera.org/learn/simple-regression-analysis-public-health/home/info</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c37dbe7b40ff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/confounding-and-modification-in-modelling-explained-c37dbe7b40ff">Confounding and Modification in modelling explained</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Handling Missing Values in R using Tidyr]]></title>
            <link>https://medium.com/geekculture/handling-missing-values-in-r-using-tidyr-da49766203f1?source=rss-27bb4daa58e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/da49766203f1</guid>
            <category><![CDATA[missing-values]]></category>
            <category><![CDATA[missing-data]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Arimoro Olayinka]]></dc:creator>
            <pubDate>Sat, 13 Nov 2021 07:02:45 GMT</pubDate>
            <atom:updated>2021-11-17T09:34:50.365Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kbbGyaniC9OSiq9EtRoMYQ.jpeg" /></figure><p>Have you heard about missing values in R and wondered how to handle these missing values? Then, it would be best if you read this article to the end. This writeup will not only talk about missing values, but we will spend a great deal of our time here hands-on on how to handle missing value cases using the tidyr package. Hadley Wickham and his team created the <a href="https://tidyr.tidyverse.org/">tidyr</a> package to help you go from messy to tidy data.</p><p>There is one more thing before we dive right into the discussion here. I have written two important articles that expounded on the concept of tidy data in great detail. You can read the first article <a href="https://arimoroolayinka.medium.com/a-gentle-introduction-to-tidy-data-in-r-b6673b4d304c">here</a> and the second one <a href="https://arimoroolayinka.medium.com/tidy-data-case-studies-using-r-648c87b40ecf">here</a>.</p><p>In my first article in the tidy data series titled <strong>“A Gentle Introduction to Tidy Data”</strong>, I explored the major tidy data principles extensively. Also in that article, I highlighted other tidy data principles that are important to note when discussing the concept of tidy data.</p><p>The motivation for our discussion here is based on one of those principles, which says <strong><em>“no cell value should contain missing values”.</em></strong></p><p>Next, it would be nice to know what a missing value is in R before we check out how to handle missing values.</p><h4><strong>What are Missing Values in R?</strong></h4><p>When a value is unknown, then we will say it is missing. In R, we represent missing values by the symbol <strong>NA</strong> (not available), while impossible values (e.g., dividing by zero) are represented by <strong>NaN</strong> (not a number). Unlike some software like SAS, R uses the same symbol (NA) for character and numeric data.</p><p>Note that NA is not a string or a numeric value but an indicator of missingness. NA is a unique value whose properties are different from other values. NA is one of the very few reserved words in R, meaning that you cannot give anything this name.</p><p><strong>Important notes about missing values in R</strong></p><ul><li>is.na() is used to test objects if they are NA</li><li>is.nan() is used to test for NaN</li><li>A NaN value is also NA, but the converse is not true</li></ul><p>With this, I believe that you now have a high-level understanding of missing values in R.</p><p><strong>Now, let’s see how to deal with missing values using the tidyr package.</strong></p><p>The tidyr package offers a set of functions to deal with missing values. Note that I will not exhaustively explain how to use each function (with all its arguments); however, I will show you what you need to know to help resolve your future missing value issues.</p><p>Be rest assured that in this article, you will learn a ton of good ideas. You can find the script for this hands-on walkthrough <a href="https://github.com/Predicare1/Tidy-Data-Case-Studies-in-R">here on GitHub</a> for reproducibility.</p><p>To demonstrate how to deal with missing values in R using tidyr, we will use the <strong>msleep </strong>data set in the <strong>ggplot2 </strong>package. The <strong>msleep </strong>is the mammals sleep dataset. It is a data set with 83 rows and 11 variables. You can read about the data <a href="https://www.rdocumentation.org/packages/ggplot2/versions/3.3.5/topics/msleep">here</a>.</p><p>The first step would be to load the <strong>tidyverse </strong>package into R. If you have installed the tidyverse package before, the command below should work pretty well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*rzNnZyf_vJ_X-L3640Zubg.png" /></figure><p>Next, we will load the ggplot2 package into R because the msleep data set we would use for this demonstration is inside that ggplot2 package. After this, we will use the <strong>data() </strong>function to load the msleep data set into R. Finally, we will return the first six (6) rows of the msleep data set using the <strong>head() </strong>function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*fuQa0ZPXkKS_5TqVBzP4zA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*8B-5xTljI1AKSGZ6oifaLw.png" /></figure><p>The result of <strong>the head(msleep)</strong> function gives us a one-shot glance at the data set without having to see all the data.</p><p>Next, let us check the dimension and the names of the columns in the data set using the <strong>dim() </strong>and <strong>names() </strong>functions, respectively. These initial steps are crucial to understanding and getting familiar with the data we want to use in this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*JaAkf11XHKmPN4hpG5EnjA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*w1B3BrZAFrejBNzwn--57w.png" /></figure><p>As I have mentioned above, this data frame has 83 rows and 11 variables. Additionally, we can see the names of the columns in the data set. I would suggest that you read the data documentation to understand what each column name represents.</p><p>Now, let us count the number of missing values in each variable in the dataset. I believe this is one important step we should take since we are interested in handling missing values.</p><p>We can do this with a base R function — <strong>sapply().</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*jtFvjgCA22rVz3IzaA-9Qw.png" /></figure><p>The sapply() function is a loop function in R. The code above will iterate through each column or variable of the msleep data and calculate the total number of missing values in that variable. To calculate the number of missing values, I wrote a simple anonymous function.</p><p>Now, let us see the result below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*YejS-p3C_3W0LK2oP4D9ZA.png" /></figure><p>The vore column contains seven missing values (NAs). The conservation column has 29, sleep_rem has 22, sleep_cycle has 51, and brainwt has 27 missing values.</p><p>We can also use the <strong>map()</strong> function from the purrr package (one of the tidyverse packages) to achieve this exact purpose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*jFX-rD0NhgXOBq_3VJAcWQ.png" /></figure><p><strong>Note: I did not return the result of this code. I put this here to show you how to use the map() function for this same purpose.</strong></p><p>Furthermore, I think it makes sense to calculate the proportion of missingness in each variable in the msleep data. Now, I would use the <strong>map()</strong> function to get this done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*1R3MTGPKnCxuo5AuwFuTBg.png" /></figure><p>The code above calculates the proportion of missing values in each variable. You may wonder what <strong>%&gt;%</strong> means. It is called the pipe operator, and it is used to chain functions together in a sequential and hierarchical order. Here, we want to take the msleep data and check each variable’s missing values. Then, pass this into the next line <strong>map(sum) </strong>to calculate the total sum of missing values in each variable. Then, <strong>map(~ . / nrow(msleep)) </strong>means that pick the sum of missing values in each variable calculated earlier and divide by the number of rows in the msleep data set (we know that to be 83)<strong>. </strong>Lastly, <strong>bind_cols() </strong>efficiently bind multiple data frames by column.</p><p>Interesting! Now, let us see the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*nFtb4htiDkk-AL9fvykyog.png" /></figure><p>Amazing! The vore variable has about 8% missing values, conservation has 34.9%, sleep_rem has 26.5%, sleep_cycle has 61.4%, and brainwt has 32.5% missing values.</p><p>These proportions will help us decide how to handle the missing values in each of the variables. As we advance, for our work here, we will refer to the vore, sleep_rem, sleep_cycle, and brainwt columns since they contain the NA values and are numeric columns.</p><p><strong>Note: I did not include the conversation column because it is a character column and contains about 35% missing values. In short, we would not be working with that column here.</strong></p><p>From the original msleep data, we will use the <strong>select()</strong> verb/function from the dplyr package to select the columns we need. We can save the result as a new object called <strong>msleep_data</strong> to use this going forward.</p><p><strong>Note: </strong>The dplyr package is under the umbrella of a family of tidyverse packages. The <strong>dplyr </strong>package is very powerful, and in fact, it is the most useful package in R for data manipulation. It is my all-time best package for data manipulation. You can check out my project-based course on the Coursera titled <strong>Data Manipulation with dplyr in R </strong><a href="https://www.coursera.org/learn/data-manipulation-with-dplyr-in-r/home/welcome">here</a><strong>. </strong>I covered the use of the dplyr package in that course.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*emJ_Vm9kAUDQMAfa04aO9Q.png" /></figure><p>Now, let us print the new data we created and saved in the msleep_data object and check the dimension of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*Hbozv33wTocb1vCQG4H1FQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*FpxqDrRX-Hx5_h4SjTA7fQ.png" /></figure><p>As expected, we can see the four columns we have selected; and the dimension confirms that there are still 83 rows, but instead of the 11 columns, we now have four columns.</p><p>Recall that the vore variable had only seven (about 8%) missing values. We can decide to drop rows with NA values in the vore column. We will use the <strong>drop_na()</strong> function in tidyr on the vore column.</p><p><strong>Note: Be sure that you want to remove the missing rows of a column before you use the drop_na() function because this will remove the entire row or observation of data, which will affect the values of that row in other columns.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*8r5JVhrYw-UXe1st-Zxzbw.png" /></figure><p>Now, let us check the new data dimension where we have removed the seven missing values from the vore column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*KJri9Qcfuh539TBhPs7NVQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*P0ARTEvojZGhaJfpYF75EQ.png" /></figure><p>The result shows us that the data now has 76 rows and four columns. Recall that the initial msleep data had 83 rows. So, when we take out seven rows from 83, we will have 76 rows left. Also, recall that we used the <strong>select()</strong> verb from the dplyr package to choose four out of the 11 columns/variables in the original data set; that’s why we have four columns.</p><p>From the result of the new msleep_data, you will notice missing values in the sleep_rem, sleep_cycle and brainwt columns. Let us see how to deal with these missing values.</p><p>Next, we want to replace the NA values in the sleep_rem column with zero integer values (0L). To achieve this, we will use the <strong>replace_na() </strong>function. The function replaces NAs with specified values. It takes an important argument called <strong>replace</strong> — If data is a data frame, “replace” takes a list of values, with one value for each column that has NA values to be replaced. You can read the replace_na() documentation <a href="https://www.rdocumentation.org/packages/tidyr/versions/1.1.4/topics/replace_na">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*4C6sjeOTgytO9rIs27-aMw.png" /></figure><p>Since we want to replace the NAs in the sleep_rem column with 0s, we used <strong>0L</strong> to specify the correct data type. Also, this is a data frame, hence the use of <strong>list(sleep_rem = 0L).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*8RFK2S2YDbTDviHRVQKVRw.png" /></figure><p>Bravo!!! The missing values in the sleep_rem column are filled with 0s. Notice that we had 0L; this is so that the new value we replaced it with will have the correct data type <strong>— integer (dbl).</strong></p><p>We can also fill the missing values in the brainwt column upwards or downwards. That is, we can fill with the values preceding or after that missing value. It all depends on how you want to fill the missing values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*LiLjOhkggI3bfqgUoqrxqQ.png" /></figure><p>Here, we used the fill() function. The fill() function in the tidyr package fills NA values in selected columns using the next or previous entry. This is helpful in the common output format where values are not repeated and are only recorded when they change.</p><p>The .direction argument tells the direction in which to fill missing values. We can either fill “down” (the default), “up”, “downup” (i.e. first down and then up) or “updown” (first up and then down).</p><p>In this case, we want to fill in the upwards direction (filling the current missing value with the next value after it).</p><p>With this explanation, let us see the result returned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*588V9NGco7TnyxzY4wqQCA.png" /></figure><p>If you go back a step to the last result before this, you will notice that in the brainwt column, the first, the third, the sixth and seventh values were missing. In this result, the first value (or first row) of the brainwt column is now replaced with 0.0155 (that’s the 2nd value and non-missing value of that column).</p><p>Similarly, the third value that was missing earlier is now replaced with 0.00029 (the 4th value of that column). Also, the 6th and 7th rows were replaced with the next non-missing value, the 8th value — 0.07. If you check through the brainwt column, you will see that we filled the missing values upwards.</p><p><strong>Please go through this explanation again and be sure that you understand what I explained here before you continue.</strong></p><p>Let us move on. Again, you will notice that the sleep_rem column still has NAs even though we replaced it with 0s earlier. To solve this problem, we will chain the last two operations we performed together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*BMUkpNjbMmpTUmgYFTJLGw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*jhgxIcl1PEmj7_mZtkQhmA.png" /></figure><p>Amazing!!! Now, we have the expected result. We have treated the missing values in the sleep_rem and brainwt columns.</p><p>We can also decide to fill the brainwt column downwards (that is, using the previous value).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*COoQhuN5xEe6mZjNfMYiug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*ZmKH0DmmH-czDeSfr7Xw-w.png" /></figure><p>The result here is the direct opposite of filling upwards that I explained earlier. Here, filling downwards fills the missing values in the brainwt column with the previous value. Clearly, from the result, the first value of the brainwt column is still missing (NA) even after using the fill() function because there is no previous value to fill it with.</p><p><strong>Note: You can fill “downup” and “updown” also. You can change the .direction argument to “downup” or “updown” to see what it looks like, and the result returned.</strong></p><p>I will leave you to handle the missing values in the sleep_cycle variable in any direction of your choice. Being able to do this will help you reinforce what you have learned in this article.</p><p>On the final note, there could be situations where you want to fill the missing values in a column with the mean or median. I prefer filling missing values using the median because it is a resistant statistic (not easily affected by outliers).</p><p><strong>Let us see how to do this here.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*r6zBkfO5gZ22K6gaDcdpvw.png" /></figure><p>Wow… A lot of information here. Let me break it down for you. What we have here is quite direct. We passed the msleep_data into a dplyr verb or function called <strong>mutate. </strong>The <strong>mutate() </strong>verb helps to create new columns while preserving the other columns. Remember how the replace_na() function works.</p><p>Since we desired to replace the missing values in the sleep_cycle column with the median, we used the <strong>median() </strong>function. The <strong>na.rm = T </strong>argument means that we want to remove the missing values in that column before calculating the median. If you do not add that argument, then it will return NA when you calculate the median. I think that’s quite intuitive.</p><p>The idea I am passing across here is that you cannot perform any calculation on missing values; hence the need to first remove those missing values, calculate the median and then use the replace_na() function to replace the missing values with the median. Lastly, we saved this replaced column into the original column to effect this change we just made.</p><p>Let us see the result of the code above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*flQigwfm-2aLpLgvcSIl-w.png" /></figure><p>Beautiful!!! The median of this column is 0.333, and this is what is replaced here. You can decide to change the median to mean and fill the missing values with the mean. It depends on how you think about the process of your solution.</p><p>To get the desired clean data in each column, we need to chain all the operations together using the pipe operator and save the clean data in the same msleep_data object. The clean data can then be used in future analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*G24FW0XWemjrp1m2V2w_qw.png" /></figure><p><strong>Let us see the final result.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*LuB-WMloCbCOgwrAqwYHCQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*ptBOKNDv0yi_nUl36ZR2RA.png" /></figure><p>Amazing!!! You did great. Well done!!! We have successfully handled the missing values in this data set.</p><h4><strong>Summary</strong></h4><p>In this article, we saw how to implement one of the tidy data principles that <strong><em>“no observation should have a missing value”</em></strong>. This understanding is beneficial for you as a data enthusiast or data expert. Dealing with missing values at the initial stage of the data life cycle will save you a lot of “headaches” when you get to the analysis stage of the data life cycle. One idea that I amplified here is that you have to be sure of how you want to handle the missing values in your data set as a data enthusiast. When you are clear about the approach you want to take; then there are several tools, functions, packages in R that can help you achieve this; and in this article, we just examined using the tidyr package.</p><p>Let me leave you with these famous quotes:</p><blockquote><em>“Happy families are all alike, but every unhappy family is unhappy in its own way.” </em>~ Leo Tolstoy</blockquote><blockquote><em>“Tidy datasets are all alike, but every messy dataset is messy in its own way.” </em>~ Hadley Wickham</blockquote><p>Please drop your comments in the comment section below and share this article with others. If you enjoyed this post, it would mean a lot to me if you could click on the “claps” icon (up to 50 claps allowed).</p><p>If you would like to take a guided project-based course that I taught on Coursera on handling missing values in R to get a verifiable certificate, then click <a href="https://www.coursera.org/projects/handling-missing-values-r-using-tidyr">here</a>.</p><p>As a project-based course instructor with Coursera Guided Project Network, I have taught a couple of courses on using R. You can check them out <a href="https://www.coursera.org/instructor/arimoro-olayinka">here</a> to take any of these courses. Thank You! See you at another time!</p><p><strong>References</strong></p><ul><li><a href="https://www.statmethods.net/input/missingdata.html">https://www.statmethods.net/input/missingdata.html</a></li><li><a href="https://stats.idre.ucla.edu/r/faq/how-does-r-handle-missing-values/">https://stats.idre.ucla.edu/r/faq/how-does-r-handle-missing-values/</a></li><li><a href="https://faq-ans.com/en/Q%26A/page=9cac166c8d89978dad8d8dcb632a53c0#s2">https://faq-ans.com/en/Q%26A/page=9cac166c8d89978dad8d8dcb632a53c0#s2</a></li><li>Missing Values in R. <a href="https://faculty.nps.edu/sebuttre/home/R/missings.html">https://faculty.nps.edu/sebuttre/home/R/missings.html</a></li><li>Tidy Data with tidyr — R for Data Science [Book]. <a href="https://www.oreilly.com/library/view/r-for-data/9781491910382/ch09.html">https://www.oreilly.com/library/view/r-for-data/9781491910382/ch09.html</a></li><li><a href="https://www.oreilly.com/library/view/r-for-data/9781491910382/ch09.html">Missing values — Regression Analysis with R [Book]. https://www.oreilly.com/library/view/regression-analysis-with/9781788627306/52f1995e-940c-47fd-a35f-2f58331a3746.xhtml</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da49766203f1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/handling-missing-values-in-r-using-tidyr-da49766203f1">Handling Missing Values in R using Tidyr</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Gentle Introduction to Tidy Data in R]]></title>
            <link>https://arimoroolayinka.medium.com/a-gentle-introduction-to-tidy-data-in-r-b6673b4d304c?source=rss-27bb4daa58e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6673b4d304c</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[tidy-data]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[r]]></category>
            <dc:creator><![CDATA[Arimoro Olayinka]]></dc:creator>
            <pubDate>Fri, 08 Oct 2021 12:20:27 GMT</pubDate>
            <atom:updated>2021-10-24T12:46:39.889Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hxgmTJa5Rz8TkwY7lUcTLg.jpeg" /></figure><p>Recently, I have been reading and taking some courses on the concept of tidy data. My discoveries motivated me to write this article about tidy data because data must be in the proper format (tidy) before analysis can be done.</p><p>In this series of articles, I will share some thoughts about tidy data, and we will have a walkthrough by tidying different datasets to conform to the tidy data principles.</p><p><strong>Let us get right at it.</strong></p><p>The data science life cycle<strong> </strong>usually begins with asking questions that data can provide answers to, and the cycle ends with the answer to that question. As a data scientist or data-inclined person, there are series of steps that you must take after you have formulated the question before arriving at an answer or a solution. Some of the steps you will take are:</p><ul><li>Determine what data will be useful</li><li>import the data, tidy the data into a format that is easy to work with</li><li>Explore the data</li><li>Generate insightful visualizations</li><li>Carry out the analysis, and</li><li>Communicate your findings.</li></ul><p>Iteratively going through this process defines the data life cycle at a high level. That is why it is usually said that <a href="https://www.nytimes.com/2014/08/18/technology/for-big-data-scientists-hurdle-to-insights-is-janitor-work.html">50–80% of a data scientist’s time is spent wrangling data</a> and making the data in the right format.</p><p>In this first article, I will talk about tidy data with an emphasis on tidy data principles. Then, in the following article, I will show you how to use some R packages like the <a href="https://tidyr.tidyverse.org/">tidyr</a> package in R to illustrate these tidy data principles using some data sets.</p><h3><strong>What is Tidy Data?</strong></h3><p>While reading through several materials, I found this definition by Hadley Wickham (Chief Scientist at RStudio) fascinating.</p><blockquote><em>“Tidy datasets are easy to manipulate, model and visualize, and have a specific structure: each variable is a column, each observation is a row, and each type of observational unit is a table.” ~ Hadley Wickham’s 2014 paper titled </em><a href="https://www.jstatsoft.org/article/view/v059i10"><em>Tidy Data</em></a></blockquote><p>I love this definition so much because it captures the primary building blocks that form the foundation for the tidy data principles that we will look at shortly. So, when it comes to thinking about tidy data, one thing that should guide your thoughts is <strong><em>that tidy data are rectangular data</em></strong>.</p><p>One way to think about tidy data is that it has to look like a rectangle with each variable/feature in a separate column and each entry/observation in a different row and all cells should contain some text with something in every cell.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*-Qir8r8PzZK-NrZhllGt5Q.png" /><figcaption>Source: Customers table from the Superstore data set</figcaption></figure><p>Therefore, if you are working with a dataset and attempting to tidy it, the data has to look like a rectangle at the end of the process. If not, then more work is likely needed before the data is genuinely tidy.</p><p>On a final note here, when data is already in a tidy format, <em>or </em>you have spent time at the start of a data science project to get data into a tidy format, the remaining steps of your project will be more straightforward.</p><p><strong>Now, let’s consider the guiding principles of tidy data.</strong></p><h3><strong>Principles of Tidy Data</strong></h3><p>To start the discussion about the principles of Tidy Data, I will start with these two quotes:</p><blockquote>“Happy families are all alike, but every unhappy family is unhappy in its own way.” ~ Leo Tolstoy</blockquote><blockquote><em>“Tidy datasets are all alike, but every messy dataset is messy in its own way.” </em>~ Hadley Wickham</blockquote><p>The above quotes imply that tidy datasets, by design, are easier to manipulate, model, and visualize because the tidy data principles that we will discuss here impose a general framework and a consistent set of rules on data.</p><p>As I have mentioned, the tidy data format has a rectangular shape which means it has columns, rows, and cells, just like in a spreadsheet. However, for rectangular data to be tidy, there are three major principles to adhere to:</p><ol><li>Each column should hold a single variable</li><li>Each row should hold a single observation.</li><li>Each cell should hold a single value (if you follow the first two rules, then each cell should have a single value)</li></ol><p>In addition, other tidy data principles are worthy of mention here:</p><ol><li>There should be one spreadsheet for each type of data</li><li>No cell value should contain missing values</li><li>If you have multiple spreadsheets, they should include a column in each spreadsheet with the same column label on which they can be joined or merged.</li></ol><p><strong>Now, let’s break down each one to ensure that we are on the same page.</strong></p><h4><strong>Principle 1: Each variable is stored in a single column</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*r3XyCjB89UwLQ2ewrymq8A.png" /><figcaption>Source: Customers table from the Superstore data set</figcaption></figure><p>In this example, these variables are the CustomerID, CustomerName, Segment, Age, Country, City, State, PostalCode, and Region of different customers.</p><h4><strong>Principle 2: Each observation of a variable is stored in a different row</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*Ue3wqI6w__CoP_D7m7NRHA.png" /><figcaption>Source: Customers table from the Superstore data set</figcaption></figure><p>In this example, each customer is an observation.</p><h4><strong>Principle 3: Each cell stores a single value</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*5GwWNmQnCqzEcqcpRy7v-g.png" /><figcaption>Source: Customers table from the Superstore data set</figcaption></figure><p>In this example, each cell stores a single value because we adhered to the first two tidy data principles.</p><p><strong>To end this article, let’s also see examples of other tidy data principles that can be helpful.</strong></p><h4><strong>Principle 4: There should be one spreadsheet for each type of data</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*n1B_jYESMYlfSNXlAEGW_w.png" /><figcaption>Source: Customers table from the Superstore data set</figcaption></figure><p>In this example, the first spreadsheet <strong>(Sales Data)</strong> stores the information about sales and purchases of each customer. The second spreadsheet <strong>(Customers Data)</strong> stores the demographics of each customer.</p><h4><strong>Principe 5: If you have multiple spreadsheets, they should include a column in each spreadsheet with the same column label that allows them to be joined or merged</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*Ly5duvCkU4A8Ip8dNFmPyw.png" /><figcaption>Source: Customers table from the Superstore data set</figcaption></figure><p>In this example, the first spreadsheet <strong>(Sales Data)</strong> and second spreadsheet <strong>(Customers Data)</strong> have a common column <strong>(CustomerID)</strong> on which the two data can be joined or merged.</p><h4><strong>Principle 6: No cell value should contain missing values</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*1ySO3Ic6Us-7A_-i_6HMuQ.png" /><figcaption>Source: Album table from the Chinook data set</figcaption></figure><p>In this example, there are missing values in the ArtistId column. This dataset (<strong>Album Data) </strong>is not in tidy format. We will learn how to deal with these missing value cases in the next article where we will do some hands-on work using R.</p><p>We have just seen a gentle introduction to the concept of tidy data. This understanding is very useful for you as a data enthusiast or data expert.</p><p>Find more ideas on tidy data in this resourceful book titled <strong>Tidyverse Skills for Data Science in R </strong>by Carrie Wright, Shannon Ellis, Stephanie Hicks and Roger D Peng. <a href="http://leanpub.com/tidyverseskillsdatascience">here</a>.</p><p>Would you like to see how these tidy data principles are implemented using some packages in R, then read my next article <a href="https://arimoroolayinka.medium.com/tidy-data-case-studies-using-r-648c87b40ecf">here</a>.</p><p>Thank you for taking the time to read this article. I hope you learned some useful ideas here. Drop your comments in the comment section. Make sure to share this article with others. Don’t forget to give this a clap.</p><p>As a project-based course instructor with Coursera Guided Project Network, I have taught a couple of courses on using R. You can check them out <a href="https://www.coursera.org/instructor/arimoro-olayinka">here</a> to take any of my courses. Thank You! See you soon!</p><h4><strong>References</strong></h4><ul><li>Chapter 1 Introduction to the Tidyverse | Tidyverse Skills …. <a href="https://jhudatascience.org/tidyversecourse/intro.html">https://jhudatascience.org/tidyversecourse/intro.html</a></li><li>Tidy Data with tidyr — R for Data Science [Book]. <a href="https://www.oreilly.com/library/view/r-for-data/9781491910382/ch09.html">https://www.oreilly.com/library/view/r-for-data/9781491910382/ch09.html</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6673b4d304c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tidy Data Case Studies Using R]]></title>
            <link>https://arimoroolayinka.medium.com/tidy-data-case-studies-using-r-648c87b40ecf?source=rss-27bb4daa58e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/648c87b40ecf</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[tidyverse]]></category>
            <category><![CDATA[tidy-data]]></category>
            <dc:creator><![CDATA[Arimoro Olayinka]]></dc:creator>
            <pubDate>Fri, 08 Oct 2021 12:19:33 GMT</pubDate>
            <atom:updated>2022-02-17T08:59:20.480Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qQ2DmLuZrOyp14o6PpAsPQ.jpeg" /></figure><p>In the previous article in this tidy data series, I introduced the concept of tidy data and highlighted the principles of tidy data. If you have not read the last article, you can read it <a href="https://arimoroolayinka.medium.com/a-gentle-introduction-to-tidy-data-in-r-b6673b4d304c">here</a>.</p><p>Before we dive right into these case studies, let me refresh your knowledge about the guiding principles of tidy data. The important principles of tidy data are:</p><ol><li>Each column should hold a single variable</li><li>Each row should hold a single observation.</li><li>Each cell should hold a single value (if you follow the first two rules, then each cell should have a single value)</li></ol><p>With this refresher, we will get our hands dirty with illustrations of the principles mentioned earlier in this article.</p><p>Since it can be hard to read in the data and get it to conform to the format, you ultimately need to answer a data-driven question. Due to this, conceptual frameworks and software packages to make these steps easier have been developed.</p><p>Hadley Wickham (Chief Scientist at RStudio), who is the godfather of the tidyverse (put a link to the official documentation here) collection of R packages, created the <a href="https://tidyr.tidyverse.org/"><strong>tidyr</strong></a><strong> package</strong> to help you go from messy to tidy data. The other packages in the tidyverse family are designed around the idea that your data is in a tidy format. They work so well because tidy datasets are all alike, while every messy dataset is just messy in its way.</p><p>Let us now see how different functions in the tidyr package can help turn messy data to conform to the tidy data principles mentioned above.</p><p><strong>Let us get started with some preambles.</strong></p><p>First, to set out, we will install the <strong>tidyverse </strong>package since it contains the functions we will use for our work here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*Rr1mb9ilnFnl2lbLopVgWA.png" /></figure><p>Before we go on, note that I will not exhaustively show you how to use each function (with all its arguments) since this is an introduction to the tidyr package. However, this article will quickly build a solid foundation to pick up these functions for more use cases.</p><p>You can find all the data sets and script <a href="https://github.com/Predicare1/Tidy-Data-Case-Studies-in-R">here on GitHub</a> for reproducibility.</p><p>If you are ready and excited. Let us enjoy this ride together.</p><h3><strong>Principle One: Each Variable is stored in a column</strong></h3><p>To ob<strong>tain data that c</strong>onforms to this rule, we will use the <strong>pivot_longer()</strong>, and <strong>pivot_wider()</strong> functions.</p><h4><strong>a. Pivot longer</strong></h4><p><strong>pivot_longer()</strong> — “lengthens” data, increasing the number of rows and decreasing the number of columns. Its’ opposite is <strong>pivot_wider().</strong></p><p>It is a good practice to read the documentation of a new function you have not used before. Here, we will read the documentation of the pivot_longer().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*fpi2E-CtXjM_mGnw6NfwpA.png" /></figure><p>Doing this will open up documentation to read about the pivot_longer() function, including its arguments and some examples of its uses.</p><p>The next thing is to create a tibble. A tibble has similar properties to a data frame object, but it is easier to work with tibbles, especially with the tidyverse collection of R packages.</p><p>We will call the tibble object <strong>table1. </strong>The tibble stores information on the number of disease cases in 1999 and 2000 for three countries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*_PkaYuYlVPCQu91aLvTVMw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/1*O8MyzIYA0Y6IyShmUPBMaw.png" /></figure><p>This data violates two tidy principles; 1) the column names are not correctly given 2) the years (1999 and 2000) should be in one column.</p><p>We will now use pivot_longer() to reshape the data and solve these violations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*aLcgjjoybF9cSJmsnasO_g.png" /></figure><p>Here, the <strong>%&gt;%</strong> is called the pipe operator, and it is used to chain functions together in sequential and hierarchical order.</p><p>From the above, we have been able to pivot on the years 1999 and 2000 to get the output below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*TYumFw2WjgtHl2dtZFIamg.png" /></figure><p>Now, this data is in better shape, and it is easier to see the information that it contains. Another way to achieve the above is to pivot on all other columns except the country column. Doing this is extremely useful when you have a large dataset and do not have the luxury of typing all the columns to pivot on. We will achieve this using:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*yeuzdK31Z6E2ogHLDzUprw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/469/1*bvR3rfKtgxF-t62pR2rxZw.png" /></figure><p>Much better, right? The <strong>-country </strong>above means that pivot on all other columns except the country column.</p><p>Did you notice that the new columns created do not have the correct column names? The proper column names should be year and n_cases (for the number of cases). Let us see how to achieve this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*6EVaDYzSB9P5BkKOZJgS4w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*bnBJwHaiy2p4E-Ok6phg0A.png" /></figure><p>Bravo! Quite intuitive, right? I just called two arguments inside of the pivot_longer() function; <strong>names_to </strong>and<strong> values_to.</strong></p><p>I believe this suffices for now. Although, there are a whole host of things that you can still do. You can check the data tidying with tidyr cheatsheet <a href="https://www.rstudio.com/resources/cheatsheets/">here</a>.</p><p>Before I stop the demonstration about the pivot_longer() function, it is important to note that another function inside the tidyr package does a similar job as the pivot_longer(). That is is the <strong>gather() </strong>function. Let us briefly see a use case here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*YQm1kNh3sG0uJXUU57hzBQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*3_Zjm3FvFIx3fCgnZ9GrLw.png" /></figure><p>Similar results, yeah? However, they use different arguments to rename the columns. Beautiful!</p><p>Let us go on to the pivot_wider() function that works in the opposite direction to the pivot_longer().</p><h4><strong>b.</strong> <strong>Pivot wider</strong></h4><p><strong>pivot_wider()</strong> — “widens” data, increasing the number of columns and decreasing the number of rows.</p><p>Again, you can read the documentation of the <strong>pivot_wider() </strong>function inside of your RStudio using <strong>?pivot_wider</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*999ycVnaUoEFl_2_pxiopg.png" /></figure><p>To demonstrate the use of pivot_wider(), we will read in a data set that contains information about the different planets we have.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*MehaELBYcyszog8AM0ENlQ.png" /></figure><p>A quick note here; before you can load the <strong>readr package using library(readr), </strong>you must have the package installed. Here, it is assumed that I have installed the package.</p><p><strong>Note: </strong>In R, you can install any package by using <strong>install.packages(“package_name). </strong>To install the readr package for example, you can do <strong>install.packages(“readr”)</strong>.</p><p>Another thing to note is that the data you have downloaded must be in your current working directory to load the data successfully. I suggest you copy this planet-data.csv file and other files from your downloads to your current directory.</p><p>I have used the <strong>read_csv()</strong> function against the base R read.csv() because I want to import this data as a tibble and not a data frame. You can execute this <strong>vignette(“tibble”) </strong>to read a full vignette about tibbles and how they differ from data.frame.</p><p>With this, let us see the result of the first six rows of the <strong>planet_df </strong>data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*mJS0UAsKGskMV0GvTlxftg.png" /></figure><p>You will notice that there is much information here, but we cannot see that information at a glance. Without a doubt, this is messy data. Now, let us deal with this using the <strong>pivot_wider()</strong> function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*9y4NbUuEwfzB074Db0roaw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*Deqy-UPhymfMVfY_P5HeOQ.png" /></figure><p>Great, isn’t it? Do you notice a similarity between the arguments of the pivot_longer() and this pivot_wider()? I believe that was quite understandable.</p><p>Again, even though we have changed the long dataset that was not clear at first glance to a wide (rectangular) format, this data is not entirely tidy because we still have some missing values. Let us solve this problem by doing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*0KEOF6u0i_wU0Td_a7IJIA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*7QQC7-wZCt_g3vkdWR3ZeA.png" /></figure><p>Could you identify the difference? I added the <strong>planet </strong>column inside the pivot_wider() as the column to be used to pivot. I believe that made sense to you.</p><p>Just like the pivot_longer(), the pivot_wider() also has a function that can perform the same role called the <strong>spread() </strong>function. You can check out how that works.</p><p>However, the pivot_longer() and pivot_wider() functions are better and preferred alternatives to the gather() and spread() functions respectively.</p><p>Before I end the discussion here, I must talk about how to unstack data from long to wide format. I have had to stay over 4 hours trying to solve this kind of problem in my R career. However, learning about this can help you achieve the proper data format in less than 2 minutes.</p><p><strong>Let us begin to learn about unstacking data.</strong></p><p>We will load an inbuilt R dataset called PlantGrowth and explore it. The data set results from an experiment to compare yields (as measured by the dried weight of plants) obtained under control and two different treatment conditions.</p><p>We will use data (“dataset_name”) to load an inbuilt R data set into the R workspace<strong>. </strong>In this case,<strong> </strong>we will use <strong>data(“PlantGrowth”) </strong>to load the PlantGrowth data into R.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*SKiiQbCz5rhv3dTYGQrTDA.png" /></figure><p>Here, the head and tail functions will return the first and last five rows, respectively. It is so here because we specified 5. Usually, the <strong>head()</strong> and <strong>tail()</strong> functions, by default, return the first six and last six rows, respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*DDOTPGsN7ABc8LwQAJZQjA.png" /></figure><p>So, you can see that we have the weight of the plants and the treatment groups. The goal is to reshape this data such that each treatment group becomes the column with their respective weight values.</p><p>To achieve this, we will use the <strong>unstack() </strong>function — Stacking vectors concatenates multiple vectors into a single vector along with a factor indicating where each observation originated. Unstacking reverses this operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*XPQiYxnjgq7sEntAXpD_mg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*PigmpxyH5N4CfTNTvfH6pQ.png" /></figure><p>Amazing! Just with a simple call of the <strong>unstack() </strong>function, the PlantGrowth data looks to be in better shape. Now, we can see the weight values for the different treatment groups.</p><p>To solidify this understanding, let us consider another example with the <strong>DIETS</strong> dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*GlIpn8MtY2VzKtmpnCTGmA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*9IGlOqOZI02gNwIXA1z7Kw.png" /></figure><p>The diet column has two categories, <strong>LOWFAT and REGULAR </strong>diets, with each person’s weight loss. When the survey was performed, 100 participants were put into one of these two categories. Now, we want to make the categories of diet become columns with their respective weight loss values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*3xdhrJ8m1QfaJwV0QAQ42g.png" /></figure><p>You would notice an additional argument in the unstack function — <strong>WTLOSS ~ DIET</strong>. This is called the form argument, a two-sided formula whose left side evaluates to the vector to be unstacked and whose right side evaluates to the indicator of the groups to create.</p><p>Here <strong>WTLOSS</strong> is the vector to be unstacked, and <strong>DIET</strong> is a categorical variable that will become columns. We can see that the numeric variable comes first in the formula, then the categorical variable comes next. I have put this here in situations where the unstack function does not work correctly. When you encounter such problems, specify this form argument.</p><p>From the first five rows, we can see that LOWFAT and REGULAR are the columns with their respective <strong>WTLOSS</strong> values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*gJISAKVcguPV15K60yq1yg.png" /></figure><p>Wow! Amazing! I believe that was quite interesting to walk through. Let us go on to the following tidy data principle.</p><h3><strong>Principle Two: Each Observation is stored in a row</strong></h3><h4><strong>separate_rows()</strong></h4><p><strong>separate_rows()</strong> — If a variable contains observations with multiple delimited values, this separates the values and places each one in its row.</p><p>Similarly, you can read the documentation of the <strong>separate_rows() </strong>function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*uVSCZmBYa_FlcE-RwesXNQ.png" /></figure><p>Now, let us read in a data set that contains information about the different movies on Netflix with their title and cast names.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*sfkow4GJ3Nwqwlt4I2jQfQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*eAMpJ6_HzgkZylldVL4ZBg.png" /></figure><p>The cast column defiles this second principle of tidy data because each observation stores more than one value. In the next series of illustrations, we will examine this function using different examples.</p><p>Brace up for some beautiful ride!</p><p><strong>Now, let us see how to deal with the messiness in the cast column, such that each cast or actor will be an observation.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*tbMMK8ILuxlJq2V4RLDFcw.png" /></figure><p>What the above is saying is that we want to separate the cast column into rows. However, you will notice the <strong>sep</strong> argument, which tells us what to separate the column on. Notice that the separator is “, ”. The actors in the cast column were separated in the original data with a comma and a space.</p><p><strong>Now, let us see the result.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*Q3dEYP7ddYMVSgxvo706oQ.png" /></figure><p>Beautiful! The result shows tidy data. With this, we can decide to find the top six actors with the most appearances. Here, we will use some dplyr verbs or functions.</p><p><strong>Note: </strong>The dplyr package is under the umbrella of a family of tidyverse packages. The <strong>dplyr </strong>package is very powerful, and in fact, it is the most useful package in R for data manipulation. It is my all-time best package for data manipulation. You can check out my project-based course on the Coursera titled <strong>Data Manipulation with dplyr in R </strong><a href="https://www.coursera.org/learn/data-manipulation-with-dplyr-in-r/home/welcome">here</a><strong>. </strong>I covered the use of the dplyr package in that course.</p><p>So, to find the top six actors with the most appearances, we will chain a couple of functions together using the pipe operator. Let us see how to achieve that result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*O93CtrmESg8JbubG1g0E0w.png" /></figure><p>Remember that I mentioned that the pipe function works sequentially. Therefore, the above code separates the cast columns into a new row of observations. Then, it renames the cast column to <strong>actor. </strong>The <strong>rename() </strong>verb or function helps to rename a column name (the new name comes first in that order). Finally, it counts the number of cases (now actors) and returns the result in descending order since we are interested in the top six actors. The <strong>head() </strong>function returns the first six rows of a data set.</p><p><strong>The output of the above is given below:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*kql9_0cTYzyQ5OOhsbxbLw.png" /></figure><p>Amazing! Takahiro Sakurai appeared 18 times in this data set, Yuki Kaji 16 times, and other actors as shown. This example here emphasizes the need for data to be in the correct format. Having data in the proper format will aid seamless analysis.</p><h3><strong>Principle Three: Each cell stores a single value</strong></h3><h4><strong>separate()</strong></h4><p><strong>separate()</strong> — turns a single character column into multiple columns.</p><p><strong>Let us do some demonstrations here.</strong></p><p>We will use the movies_durations data set here. It contains information about the different movies on Netflix with their title, movie type, and duration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*EH-4Lz88dGTmKXYRD3WyBA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*KgdMCvPXcSlgYbtOkwWg5g.png" /></figure><p>From the result above, you will notice that the duration column has both the time value and the unit of time. It is not tidy data.</p><p>We will use the separate() function to separate the duration into two columns — <strong>value and unit.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*HVBeV_pS_nNqN5yj-mAoeA.png" /></figure><p>Recall the sep argument from separate_rows(). If you look at the duration column from above, you will notice that the separator between the time value and unit is a space, which is why <strong>sep = ‘ ’. </strong>The <strong>convert = TRUE</strong> argument will change the resulting columns to their correct data type upon separation. For example, the value column will be an integer upon separation. If the <strong>convert = TRUE </strong>argument is not specified, the value column will have the character data type (which is not the correct data type).</p><p>We have a beautiful result below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*Yp-Wc8X9O6YgIkRmK4ptpw.png" /></figure><p>Nice work there! Now the value and unit are in their respective columns with the right data type. Since the data is in a tidy format, we can decide to find the average duration for each movie type and time unit as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*P2H-vB4dgzpGorhVFNWSCA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*_5eDlcNEO3yYQPBDcF6qwA.png" /></figure><p>Again, tidy data enhances insightful analysis. So, we can we that the mean time duration for movies in this dataset is <strong>98.6 minutes.</strong></p><p>Before we leave this section, it is important to quickly mention the unite() function, which does the opposite of the separate() function.</p><p>Let us merge the title and type columns using the sep = “ - ” and call the newly merged column title_type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*bpZLjt0_SYjxglDR9bstRw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*hPUM1jPWv_Vy3vtmYDLTHg.png" /></figure><p>Nice!!! As expected, the title and type columns are now in one column and are separated by a hyphen (-).</p><h4><strong>separate_rows() &amp; separate()</strong></h4><p>Let us see a use case of the separate_rows() and separate() functions together to tidy the drinks_df data. The data contains three different drinks and the ingredients used to make the drinks. The data is an excel file; so we will use the <strong>read_excel() </strong>to read in the file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*r8SHTIEPqwiCwJd5OveKYg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*IF9UPa_HYRIynHD4mphI-Q.png" /></figure><p>The ingredients column is quite messy. We can first decide to separate the rows so that every ingredient will be on a separate row. Notice that the separator here is a semicolon and a space. The code below will help us achieve this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*aRLKgBmEqHqezTknizkzMg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*vINLY0olSHNeFTvEnLoA2w.png" /></figure><p>One step is done. We have successfully put every ingredient of a drink on a new row. However, you will notice that the ingredients column has three values in each cell. We want to create three different columns from this ingredients column.</p><p>To get this done, we will pipe the separate_rows() and separate() together. We will separate the ingredients column into three columns <strong>ingredient, quantity, and unit.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*V_Dqm6ndBO-I-Bev6aXrQg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*nrJlLk7nk8ybZlE9fH9bNg.png" /></figure><p>Now, we have a tidy data set where we have the drink, the ingredient, quantity, and unit. You will notice a warning message. Since oranges do not have a unit, it fills that cell with NA. You do not have to worry about this.</p><p>With this, you have learned some excellent ideas here for your future data analysis work. In addition, because I did not want this article to be unneccessarily long and boring, I decided not to include how to deal with missing data, how to clean column names, how to merge data using tidyverse packages and tidyverse adjacent packages.</p><h4><strong>Summary</strong></h4><p>Wow! It has been a long stretch of reading and hands-on work with quality explanations. In this article, we have looked at the idea of tidy data and have demonstrated how to achieve different tidy data formats. Using a consistent tidy data format allows you to build tools that work well within this framework (data science cycle) as a data-inclined person. Ultimately, this simplifies the data wrangling, visualization, and analysis processes.</p><p>As a data enthusiast, you have a leg ahead when you start your data science project with data already in a correct (tidy) format or when you spend time at the beginning of a project to get your data into a tidy format. The remaining steps project will be like a piece of cake. I hope you enjoyed this. Please drop your comments in the comment section below and share this article with others. Don’t forget to give this the maximum number of claps.</p><p>As a project-based course instructor with Coursera Guided Project Network, I have taught a couple of courses on using R. You can check them out <a href="https://www.coursera.org/instructor/arimoro-olayinka">here</a> to take any of these courses. Thank You! See you at another time! See you at another time!</p><h4>References</h4><ul><li>Chapter 1 Introduction to the Tidyverse | Tidyverse Skills …. <a href="https://jhudatascience.org/tidyversecourse/intro.html">https://jhudatascience.org/tidyversecourse/intro.html</a></li><li>Tidy Data with tidyr — R for Data Science [Book]. <a href="https://www.oreilly.com/library/view/r-for-data/9781491910382/ch09.html">https://www.oreilly.com/library/view/r-for-data/9781491910382/ch09.html</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=648c87b40ecf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Patient Satisfaction with epilepsy surgery: What is important to patients?]]></title>
            <link>https://arimoroolayinka.medium.com/patient-satisfaction-with-epilepsy-surgery-what-is-important-to-patients-14b465a1cd66?source=rss-27bb4daa58e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/14b465a1cd66</guid>
            <category><![CDATA[biostatistics]]></category>
            <category><![CDATA[epilepsy]]></category>
            <category><![CDATA[public-health]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[delphi]]></category>
            <dc:creator><![CDATA[Arimoro Olayinka]]></dc:creator>
            <pubDate>Fri, 14 Aug 2020 12:19:51 GMT</pubDate>
            <atom:updated>2020-08-14T22:25:57.137Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/390/0*c3ZFYKQChAvD_5d7.jpg" /></figure><p>This week I read a paper titled: <strong><em>“Patient satisfaction with epilepsy surgery: what is important to patients?”</em></strong> by Meaghan Lunney <em>et al. </em>(2018).</p><p>This work identified factors that are relevant to patient satisfaction with therapeutic epilepsy surgery (which is a specific treatment), and not the overall process of their care, such as staff attentiveness, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LPaGjWRcEQTRUzwX.jpg" /><figcaption><strong>Source: </strong><a href="https://i.ytimg.com/vi/WO6gyvRBOq8/maxresdefault.jpg">https://i.ytimg.com/vi/WO6gyvRBOq8/maxresdefault.jpg</a></figcaption></figure><p>Another important thing to note is the involvement of patients in the development of the tool that will be used to assess their satisfaction of epilepsy surgery. This enhances content validity and relevance.</p><p>Now, let us take a deep dive into a review of this informative paper.</p><h3>Preamble</h3><p>The authors noted that the satisfaction of a patient with treatment shows how well a patient is satisfied with the outcome of a specific treatment rather than the overall satisfaction with their care experiences.</p><p>According to Malmgren <em>et al.</em> (1997), the satisfaction of a patient with treatment have been found to be associated with some important factors like quality of life, etc.</p><p>Although, there are scales that measure patient satisfaction with medication; however, due to the aim of the authors to focus on treatment from surgeries, there needs to be development of instruments (scales/questionnaires) that measure patient satisfaction in that regard.</p><p>Therefore, the development of a tool that is specific to epilepsy surgery may be most appropriate to measure patient satisfaction. In addition, a review of patient satisfaction with types of epilepsy surgery revealed that tools are usually developed without patients involvement.</p><p>Hence, the aim of the study was to <strong><em>“systematically obtain patient-identified factors related to satisfaction with epilepsy surgery as a means of informing clinicians about the ways that patients evaluate outcomes of their treatment and as a conceptual basis for the future development of epilepsy surgery patient satisfaction scales”</em></strong>.</p><p>I have just showed a high level overview of what this paper is about. Let us take a deep dive to explore the method, materials, and results from this paper. I can assure you that it will be educative.</p><h3>Methods &amp; Materials</h3><p>The authors used a staged, systematic approach to derive items that are important for patients satisfaction with epilepsy surgery as shown in figure 1 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*0Sub1QgoPDkYKDl0E2Nzdw.png" /><figcaption><strong>Source: </strong>Methods &amp; Materials Section of the paper</figcaption></figure><p>In addition, I will like to outline the steps that the authors took as seen in figure 1 above. The authors:</p><ul><li>Performed a systematic review of patient satisfaction with epilepsy surgery — this guided the scope and methods of the study.</li><li>Selected the Multiple Cause Indicator model as a conceptual framework for generating items.</li><li>Conducted focus group discussions with past epilepsy surgery patients to identify themes that are relevant to treatment satisfaction.</li><li>Developed questionnaires items based on the identified themes</li><li>Used a consensus methodology (the Delphi technique) with experts to evaluate and refine questionnaire items.</li><li>Had another session with a subset of the focus group to get feedback to ensure that the items were relevant and clear; and</li><li>Performed a second round of Delphi evaluation to select 31 items.</li></ul><p>I believe the approach is clear and concise. Now, let us briefly consider some of the “key matters” in this approach before we see the results.</p><h4>Who are the focus groups?</h4><p>“The focus group participants were recruited from outpatient clinics in the Calgary Comprehensive Epilepsy Program in Calgary, Alberta, Canada, from December 2014 to February 2015.”</p><p>The authors drew a convenience sample of patients who were 18 years and above as at the time of their resective or disconnective epilepsy surgery from the database. Also, the authors included patients who could provide consent and were fluent in English and excluded patients who solely underwent diagnostic surgical procedures.</p><blockquote>A <strong>convenience sample</strong> is a type of non-probability sampling method where the sample is taken from a group of people easy to contact or to reach. For example, standing at a mall or a grocery store and asking people to answer questions would be an example of a convenience sample. This type of sampling is also known as <strong>grab sampling or availability sampling</strong>. There are no other criteria to the sampling method except that people be available and willing to participate. In addition, this type of sampling method does not require that a simple random sample is generated, since the only criteria is whether the participants agree to participate.</blockquote><blockquote><strong>Source:</strong> <a href="https://en.wikipedia.org/wiki/Convenience_sampling">https://en.wikipedia.org/wiki/Convenience_sampling</a></blockquote><p>There were two focus groups in which discussions with participants were exploratory, inductive, and content-driven using eight open-ended questions as seen in table 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*4t2O53VERFZOmYv-Mwsj8w.png" /><figcaption><strong>Source: </strong>Methods &amp; Materials Section of the paper</figcaption></figure><p>The focus group discussions helped to identify themes related to patient<br>satisfaction. Then, based on the themes identified in the focus group discussions, items were generated by the research team.</p><h4>Delphi Evaluation (Round 1)</h4><p>The authors selected experts based on their expertise in epilepsy, neurosurgery, health services research, and scale design. In all, the Delphi panel consisted of 13 experts. The first draft had 55 items and response options were circulated to experts for feedback to reduce the number of items on the questionnaire, to ensure that each item was relevant and that all domains were covered adequately.</p><p>The authors phrased items as questions and Delphi participants were asked to rate each item using a 5-point Likert type scale in terms of clarity (1=not clear and 5=very clear). In addition, items were rated on a 5-point scale in terms of relevance (1=not relevant and 5=highly relevant).</p><p>As a measure, the authors had a threshold for eliminating item. A median relevance score of less than three was the cut-off for eliminating items and a median clarity score less than four was the cut-off for rephrasing items to improve clarity.</p><p>Finally, free text comments were also collected from all Delphi participants (experts). This helped to guide the revision of the items with respect to phrasing or addition of items.</p><h4>Member checking with focus group participants</h4><p>In order to deal with items ambiguity and comprehensible to the target population, the authors had to select a subset (n=7) of the initial focus groups to evaluate each item in the revised questionnaire for relevance/importance and clarity, using the initial 5-point scale for relevance and clarity. Items rated as “not at all important” by the majority were eliminated. Items were rephrased if more than one participant rated it as “not at all clear”.</p><p>In the second round of the Delphi evaluation, seven experts from the 13 in Round 1 were invited for a final assessment of item importance using a similar Delphi technique. Experts were asked to identify redundant items that<br>could be eliminated.</p><p>Great! Let’s see some of the interesting results from the paper.</p><h3>Results</h3><h4>Focus groups</h4><p>Table 2 below presents the characteristics of focus group participants. Group 1 had six participants of whom 50% were males. The second group contained three participants of whom one was male. The discussion sessions lasted two hours and 1.5 hours, respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*wMfBwZibyp1nX1se_dnGdg.png" /><figcaption><strong>Source: </strong>Results section of the paper</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*zpl-GR2t2p_NDNRfyPeT-Q.png" /><figcaption><strong>Source: </strong>Results section of the paper</figcaption></figure><p>After analysis on the transcripts from the discussions, 12 unique themes were identified. An initial set of 55 items was agreed upon by scale design experts, to ensure that all themes were adequately captured.</p><h4>Delphi evaluation (Round 1)</h4><p>Of the 55 items, 34 had a median relevance rating of 5, eight had a rating of 4.5, five had a rating of 4, five had a rating of 3.5, and three had a rating of 3.</p><p>With respect to clarity, one item had a median rating of 5 and eight had a rating of 4.5. The majority (n=43) had a rating of 4. Two had a rating of 3.5 and one had a rating of 3 and was modified. No items had a clarity rating less than 3. The open-ended comments helped modify items to improve clarity.</p><p>Twelve items were removed from the initial 55-item questionnaire due to redundancy or poor clarity. The second draft consisted of 43 (55-12) items, covering all themes and included global questions related to expectations of surgery, quality of life, and overall satisfaction with epilepsy surgery.</p><h4>Member-checking with focus group participants</h4><p>Out of the seven selected subset from the initial focus groups, four returned responses. Of the 43 items, none were rated as “not at all important” by the majority and thus none were eliminated. One item was rated as “not at all clear” by more than one participant. Therefore, this single item was rephrased to improve clarity.</p><h4>Delphi evaluation (Round 2)</h4><p>The seven experts that were selected for the second Delphi evaluation agreed and reduced the 43-item list to 31 items as seen in table 3 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*JINsJWFBttjFd7qgV3-0RA.png" /><figcaption><strong>Source: </strong>Results section of the paper</figcaption></figure><p>Content validity of the 31-item list was maintained as each item addressed at least one of the themes, and all themes were covered by at least one item as seen in table 4 below. Lastly, the readability of the 31 items was assessed using the <strong><em>Flesch-Kincaid readability test</em></strong> and matched a Grade 11 level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*RpDbwhOadT0QfIZruw5GxQ.png" /><figcaption><strong>Source: </strong>Results section of the paper</figcaption></figure><h3>Link to Paper</h3><p>You can read the full paper <a href="https://pubmed.ncbi.nlm.nih.gov/30361186/">here</a>.</p><h3>Discussion &amp; Limitations</h3><p>I believe you have learnt one or two things from this review. However, I will like to emphasize some key issues from the paper.</p><p>The authors used a systematic approach and standard methodology, to list out aspects that are important to patients when assessing satisfaction with epilepsy surgery. This list consisted of 31 specific items from 12 unique themes. Adverse effects of surgery, medical care or postsurgery rehabilitation, and seizure control were the most frequently endorsed themes.</p><p>One very important note is that the authors considered factors relevant to patient satisfaction with therapeutic epilepsy surgery (treatment), and not satisfaction with the process of care. The findings of the study aligned with results of the systematic review of predictors of patient satisfaction with epilepsy surgery (Macrodimitris <em>et al</em>., 2011).</p><p>Furthermore, one focal point from this paper is the involvement of patients in the design of the items using the patient focus group discussions and two rounds of the Delphi technique. This is in contrast to the majority of published studies on patient satisfaction with treatment in which information was collected from patients using predefined questionnaires developed by the research team.</p><p>Although, this study has numerous strengths, the authors identified some limitations and how these limitations were dealt with. Let me highlight some of these limitations:</p><ol><li>Selection bias since it is possible that patients chose to participate because they were either extremely satisfied or extremely dissatisfied with their surgery.</li><li>Seven of the nine patients were seizure-free at the time of the interview, which may under-represent patients with seizure relapse and could have skewed responses toward satisfaction with surgery.</li></ol><h3>Conclusion</h3><p>In summary, thematic analysis revealed 12 patient centred themes important for measuring patient satisfaction with epilepsy surgery. “This work can inform<br>both clinical care in epilepsy surgery and future studies designed to assess patient-reported outcomes in epilepsy surgery.”</p><p>Also, based on the findings of this paper, a better understanding of what is important to patients may guide physicians when communicating with patients and making decisions around treatment options.</p><p>I believe this was a great read. Please give this a clap. Share this if you deem it fit. See it next time!</p><h3>References</h3><ul><li>Malmgren K, Sullivan M, Ekstedt G, Kullberg G, Kumlien E. Health-related quality of life after epilepsy surgery: a Swedish multicenter study. <em>Epilepsia</em> 1997; 38: 830–8.</li><li>Macrodimitris S, Sherman EMS, Williams TS, Bigras C, Wiebe S. Measuring patient satisfaction following epilepsy surgery. <em>Epilepsia </em>2011; 52: 1409–17.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14b465a1cd66" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Weekly Paper Review: Can routine clinical data be used to predict which
patients with epilepsy will…]]></title>
            <link>https://arimoroolayinka.medium.com/weekly-paper-review-can-routine-clinical-data-be-used-to-predict-which-patients-with-epilepsy-will-5e5a1d1396b3?source=rss-27bb4daa58e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e5a1d1396b3</guid>
            <dc:creator><![CDATA[Arimoro Olayinka]]></dc:creator>
            <pubDate>Fri, 31 Jul 2020 17:38:03 GMT</pubDate>
            <atom:updated>2020-08-01T12:25:57.863Z</atom:updated>
            <content:encoded><![CDATA[<h3>Weekly Paper Review: Can routine clinical data be used to predict which<br>patients with epilepsy will experience a psychiatric adverse effect<br>from levetiracetam?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*-lAIhXbyh9GMv60x.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*sGLfspy7d44KE1cS.jpg" /><figcaption>Source: <a href="https://emedz.net/levetiracetam/">https://emedz.net/levetiracetam/</a></figcaption></figure><p>It is another week of paper review. This week I read the paper titled: <strong><em>“Prediction Tools for Psychiatric Adverse Effects After Levetiracetam Prescription” </em></strong>by Colin B. Josephson, MD, MSc, FRCPC <em>et al. </em>(2019).</p><p>Also, I have reviewed another paper by these authors some weeks ago. I think you should read this review too. Read the review <a href="https://medium.com/@arimoroolayinka/weekly-paper-review-does-the-severity-of-incident-depression-influence-a-patients-risk-of-ff1f554b0f71?source=friends_link&amp;sk=03f6fbb52e0fdd3021ecc61a3c13ca3a">here</a>.</p><p>If you are ready, I am ready. Let’s examine the beautiful work put together by these authors.</p><h3>Preamble</h3><p>Levetiracetam is a commonly used antiepileptic drug (AED). However, psychiatric adverse effects or reactions usually result from the use of this drug. This may lead to discontinuation of this treatment.</p><p>The idea here is that, if a physician is able to identify a patient that will develop this psychiatric adverse effects, then approaches can be put in place that will enhance personalized attention to such patient.</p><p>In addition, the goal of building prediction models is usually to build models that can generalize and be used to guide prescription in clinical practice.</p><p>Therefore, the main objective of the study was to derive prediction models using a data-driven and clinically-informed approach that can be used to estimate the risk of psychiatric adverse effects from levetiracetam use.</p><p>To achieve this objective, they built two prediction models; one for the overall population, and the other model for those without a history of psychiatric sign, symptoms, or disorders during the study period.</p><p>This looks to be a comprehensive and exhaustive approach to achieve this aim. I think this should be able to generalize for an independent population.</p><p>Let’s see the methods and some of the results derived from the analysis of this study.</p><h3>Methods</h3><h4>The Health Improvement Network (THIN)</h4><p>The Health Improvement Network (THIN) database is an electronic medical record (EMR) data platform based in the United Kingdom that consists of anonymized general practice (GP) patient records.</p><p>This data set contains GP records for a representative sample of approximately 5% of the national population. Medical events are coded using Read codes, and prescription data are classified according to the British National Formulary.</p><p>This study was performed using data from THIN version 1205, which included data between January 1, 2000 and May, 31, 2012.</p><h4>Study Population</h4><p>The authors used a retrospective open cohort design for the study.</p><blockquote><strong>Retrospective cohort studies</strong> are a <strong>type</strong> of observational <strong>research</strong> in which the investigator(s) looks back in time at archived or self-report data to examine whether the <strong>risk of disease</strong> was different between exposed and non-exposed patients.</blockquote><blockquote><strong>Source: </strong><a href="https://canadian-nurse.com/en/articles/issues/2014/april-2014/terminology-101-retrospective-cohort-study-design">https://canadian-nurse.com/en/articles/issues/2014/april-2014/terminology-101-retrospective-cohort-study-design</a></blockquote><p>In order to enhance high-quality data collection (that is to reduce the chances of wrongly classifying prevalent epilepsy cases as incident), the authors required that all patients were active after the Acceptable Mortality Reporting date (the year in which mortality reporting was deemed complete and accurate for each practice).</p><p>The authors used a 5-year washout period from first medical code after the Acceptable Mortality Reporting date to exclude patients with prevalent epilepsy. In addition, patients could not receive any codes for epilepsy during this period. All patients 18 years and above whose practice (first medical code entry) met the date when mortality reporting was considered complete were included in the analysis.</p><h4>Exposure &amp; Outcome</h4><p>A prescription code for levetiracetam represented the exposure. To define the outcome of interest, a consensus-driven process by two of the authors who were experts in epilepsy was employed. The outcome of interest is a Read or Multilex therapeutic code for any psychiatric sign, symptom, or disorder.</p><h4>Selection of Predictor Clinical Variables</h4><p>In order to select variables that would be used to build the prediction models. The authors took a data-driven and clinically informed approach.</p><p>First, the authors developed a search strategy to interrogate MEDLINE and Embase for candidate predictors. In addition, senior authors (epileptologists) provided additional opinion.</p><blockquote><strong>MEDLINE &amp; Embase</strong></blockquote><blockquote><strong>MEDLINE </strong>is a subset of PubMed (actually about 98%) made available by National Library of Medicine (NLM) to commercial suppliers, used to find literature in the life sciences with a concentration on biomedicine. <strong>Embase </strong>is an Elsevier database that covers the same subjects as PubMed/MEDLINE, with an additional focus on drugs and pharmacology, medical devices, clinical medicine, and basic science relevant to clinical medicine.</blockquote><blockquote><strong>Source: </strong><a href="https://library.fiu.edu/c.php?g=160191&amp;p=1047492">https://library.fiu.edu/c.php?g=160191&amp;p=1047492</a></blockquote><blockquote>You can read more about <strong>MEDLINE and Embase </strong><a href="https://www.clinfo.eu/databases-literature-searches/"><strong>here</strong></a><strong>.</strong></blockquote><p><strong><em>What do you think happened here?</em></strong></p><p>The authors searched these two database — I guess in order to get the best predictors for their models. That is not all. Next, they used a modified Delphi process for final variable selection. Let me highlight the process briefly:</p><ol><li>Each participant (panelist) completed a questionnaire rating all potential variables on a 5-point Likert type scale (where 1 indicates the criterion was not very important at all and 5 indicates the criterion was very important). Panelists were also given space for additional handwritten comments and were given 14 days to complete the first round of ranking the questions.</li><li>Panelists filled a second questionnaire similar to the first one. Extra items identified by panelists during the first round were also included.</li></ol><p>It is important to note that items with median ratings of 1 to 2 were considered <strong>irrelevant</strong> and were excluded, items scoring 3 were considered to be of uncertain relevance and were <strong>reevaluated in the second round</strong>, and items scoring 4 to 5 were used to build the final predictive model.</p><p>Finally, to facilitate variable inclusion, two of the authors used a consensus-building process to identify Read and Multilex codes to define each clinical feature of interest in THIN.</p><h4>Statistical Analysis</h4><p>This study used regression techniques to build two prediction models, one for the overall population and one for those without a history of a psychiatric sign, symptom, or disorder during the study period.</p><p>The index date (time zero) was that on which the patient received the first documented levetiracetam prescription (after meeting the epilepsy case definition). Patients were followed up for 2 years or until an event, loss to follow-up, or censoring.</p><p>The authors noted that data were fully available for all variables except levetiracetam daily dose(65%complete). In order to deal with the missing (null) values which was noted to be missing at random. The authors used Rubin and Schenker’s multiple imputation to replace null values.</p><p>Finally, the authors assessed model performance using the Brier score (a measure of the forecasting accuracy for probabilistic predictions), model discrimination using the mean area under the curve (AUC), calibration using the Hosmer-Lemeshow goodness-of-fit test, and generalizability using the mean AUC derived from stratified 5-fold cross validation to account for outcome imbalance between groups.</p><p><strong>The predicted probability was calculated using 1/[1+exp(−risk score)]</strong>, where the risk score is equal to the output of the multi-variable logistic regression model. In addition, they mapped sensitivity and specificity according to the probability cutoffs.</p><p><strong><em>I hope the methods used by the authors was clear enough?</em></strong></p><p>Let’s see some of the exciting results. Remember that the aim here is to build two prediction models to predict the risk of psychiatric adverse effects after the use of levetiracetam.</p><h3>Results</h3><h4>Variable Selection</h4><p>I hope you can still recall the approach the authors used to select the final variables. First the search from MEDLINE and Embase for candidate variables and the modified Delphi process where panelists filled two similar questionnaires for final variable selection.</p><p><strong><em>Now, let see the results of this approach.</em></strong></p><p>The authors’ search of MEDLINE and Embase yielded 136 articles, of which 103 remained after deduplication.</p><p>The variables gotten from this search are: histories of febrile seizures, status epilepticus,<br>longer duration of epilepsy, psychiatric comorbidities and behavioral issues, and cognitive impairment. These variables were found to be associated with psychiatric adverse effects from levetiracetam use. Also, coadministration of lamotrigine was associated with a protective effect. This means 7 variables were gotten from the search strategy.</p><p>An additional 29 variables were recommended for inclusion after consultation with senior authors. Therefore, the first round of the Delphi process consisted of a questionnaire comprising 36 items (7 from MEDLINE and Embase search, 29 from senior authors).</p><p>Consensus was ultimately achieved for 14 of 36 variables, of which 12 (85.7%) were considered relevant. This meant that there were 22 variables left out of the 36 initial variables. Also, 13 extra items were identified through the first round.</p><p>Therefore, the second round consisted of 35 (22+13) items, of which 11 (31.4%) met consensus for inclusion. Therefore, the authors evaluated 21 variables (12 from the first round and 9 from the second round) for inclusion in<br>the prediction models.</p><h4>Prediction Modelling</h4><p>Now, there are 21 variables. Let us see the results of the models. There are three segments here. General results, prediction model for overall population, prediction model for those without a history of psychiatric sign or disorder.</p><p><strong>General Results</strong></p><p>From the 11,194,182 patients registered in THIN. It was discovered that 7,400 presumed incident cases over a maximum of 12 years’ follow-up were identified, out of which approximately 16%(n=1173) received an incident prescription for levetiracetam during this period.</p><p>The overall median age was 39 years (IQR, 25–56 years), and 590 (50.3%) were female. Psychiatric disorders like depression, anxiety, and psychosis were encountered in 22.3% (262 of 1173), 13.6% (160 of 1173), and 3.2% (37 of 1173), respectively.</p><p>The table below shows that depression and behavioral issues were the most commonly reported adverse effects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*wFS6kK4-vtLn-0maudGCEg.png" /><figcaption><strong>Source: </strong>Supplement Materials of the Paper</figcaption></figure><p>Also, as seen in the table below, out of those receiving levetiracetam, 165 (14.1%) experienced an outcome of any psychiatric symptom or therapeutic code over 2 years of follow-up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*NikWOU60k9zNL-qUI49LJQ.png" /><figcaption><strong>Source: </strong>Results section of the paper</figcaption></figure><p><strong>Prediction model for overall population</strong></p><p>When evaluating all significant variables in multivariable logistic regression, including somatic and psychiatric signs, symptoms, and disorders, the odds of reporting a psychiatric symptom or treatment code within 2 years of persistent levetiracetam use were elevated for some patients.</p><p>These included those with the following characteristics: female sex (odds ratio [OR], 1.41; 95% CI, 0.99–2.01; P = .05), increasing social deprivation (OR, 1.15; 95% CI, 1.01–1.31; P=.03), depression (OR, 2.20; 95%CI, 1.49–3.24; P &lt; .001), or anxiety (OR, 1.74; 95% CI, 1.11–2.72; P = .02), and recreational drug use (OR, 2.02; 95% CI, 1.20–3.37; P = .008).</p><p>Even though low socioeconomic status was related to higher risk of psychiatric disorder, the authors made a decision to exclude Townsend Index of Social Deprivation from the final model because it would be challenging to assign this score in clinic.</p><p>Again, this suggest that the authors aimed at generalizability of the models.</p><p><strong>Hence, the final model was as follows:</strong></p><p><strong>risk score = −2.34 + 0.27 × (female sex) + 0.82 × (history of depression) + 0.47 × (history of anxiety) + 0.74 × (history of recreational drug use).</strong></p><p>There was no evidence of multicollinearity, with variance inflation factors ranging from 1.15 (history of recreational drug use) to 1.42 (history of depression).</p><p>Also, the model performed well, with a Brier score of 0.11. It had moderate discriminative capacity and appeared generalizable after stratified 5-fold cross-validation (AUC, 0.68; 95% CI, 0.58-0.79).</p><p>The sensitivity and specificity varied, but high specificity (83%) was achieved at a probability cutoff of 0.10 as shown in Figure 1 below. There was no evidence of poor calibration based on the Hosmer-Lemeshow Goodness-of-fit test (P=.29; 10 groups).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*Nc2d71XOV8bDMXZqusGOHg.png" /><figcaption><strong>Source: </strong>Results section of the paper</figcaption></figure><p>eTable 4 and figure 2 below has some interesting insights to show us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*dNfWsM2wQpQ3ZA-vKLvggQ.png" /><figcaption><strong>Source: </strong>Supplement Materials of the Paper</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*KwC7dIpjgrRBkaKyzJ1oRw.png" /><figcaption><strong>Source: </strong>Results section of the paper</figcaption></figure><p>Figure 2 showed a gradient that those with increasing numbers of risk factors were at incrementally greater risk of a psychiatric outcome.</p><p>I hope that makes sense? The table also corroborate what the bar plot showed. The table showed the risk score range for different number of risk factors.</p><p>For each group of risk factors, the presence of depression and recreational drug use, singularly or in combination, was consistently associated with higher risk</p><p><strong>Prediction model for those with no history of Psychiatric Signs or disorder</strong></p><p>For those with no documented pre–index date history of a mental health sign, symptom, or disorder (710 of 1173 [60.5%]), 69 of 710 (9.7%) received a code for a psychiatric event within 2 years of levetiracetam prescription.</p><p><strong>The following model was generated: risk score = -3.83 + 0.013 × (age) + 0.89 × (female sex) + 1.16 × (recreational drug use) + 0.003 × (levetiracetam daily dose).</strong></p><p>There was no evidence of multicollinearity (variance inflation factors range, 1.03–2.83) or poor calibration based on the Hosmer-Lemeshow goodness-of-fit test (P=.18; 10 groups). The model performed well overall (Brier score,0.09) and when using a threshold of 0.14 had a similar specificity of 83% as in Figure 3 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*LdSa6fLy33bSK65VtBtgZA.png" /><figcaption><strong>Source: </strong>Results section of the paper</figcaption></figure><p>Discriminatory capacity and generalizability, as determined through stratified 5-fold cross-validation, was similar to the primary model (AUC, 0.72; 95% CI, 0.54–0.90).</p><h3>Discussion &amp; Limitations</h3><p>The authors derived two prediction models for the risk of a psychiatric sign or symptom after a first-ever prescription for levetiracetam in a general population of patients with presumed incident epilepsy.</p><p>When using a threshold value of 0.10 (primary model) or 0.14 (for those lacking a premorbid psychiatric history) to guide treatment <strong>(a score below these thresholds indicates safety of prescription)</strong>, the prediction models had a high specificity for predicting those with a psychiatric outcome, yielding few false-positive results.</p><p>The authors opined that if the models are validated in prospective cohorts, they could be useful at the point of care for a broad spectrum of patients with epilepsy seen in GP<br>and epilepsy clinics alike.</p><p>In addition, the authors discussed some limitations for the work. I have highlighted some here. They noted that:</p><ul><li>They were unable to extract informative indexes of seizure type, frequency, and severity and epilepsy type. Inclusion of these variables could further improve model performance.</li><li>Caution should be taken in interpreting causation because machine learning regression techniques are designed for predictive rather than inferential purposes.</li></ul><h3>Link to the Paper</h3><p>You can download the full paper <a href="https://jamanetwork.com/journals/jamaneurology/fullarticle/2720706?utm_campaign=articlePDF&amp;utm_medium=articlePDFlink&amp;utm_source=articlePDF&amp;utm_content=jamaneurol.2018.4561">here</a>.</p><h3>Conclusion</h3><p>One of the benefits of the tools described in this study is its application in clinical settings.</p><p><strong>Let me use an example from the paper</strong>.</p><p>A female patient with depression would have a risk score of −1.25 <strong>(ie, risk score = −2.34 + 0.27 + 0.82).</strong> This is calculated using the overall prediction model.</p><p>When incorporated into the algorithm, the patient’s risk of a psychiatric adverse event would be 22% over 2 years, that is: 1/[1+ exp(−1.25)]. This exceeds the threshold of 0.10 and suggests she would be at risk.</p><p>In addition, the authors showed the gradients of risk for populations of patients based on known and newly identified factors. The estimates of generalizability using a stratified 5-fold cross-validation are encouraging.</p><p>This is an important first step toward generating empirical, efficient, and practical prediction models with direct application in clinical settings.</p><p>I hope you loved the review. Please give this a clap if you learnt from this review.</p><p>See you next week!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e5a1d1396b3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>