<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matt.0 on Medium]]></title>
        <description><![CDATA[Stories by Matt.0 on Medium]]></description>
        <link>https://medium.com/@MattOldach_65321?source=rss-459e62b88a2a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*525YAA3FUsWofY8M.jpeg</url>
            <title>Stories by Matt.0 on Medium</title>
            <link>https://medium.com/@MattOldach_65321?source=rss-459e62b88a2a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 22:21:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@MattOldach_65321/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My newyeaRs resolution: slimming down (Seurat)]]></title>
            <link>https://medium.com/@MattOldach_65321/my-newyears-resolution-slimming-down-seurat-13cf0dae302?source=rss-459e62b88a2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/13cf0dae302</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[single-cell-sequencing]]></category>
            <dc:creator><![CDATA[Matt.0]]></dc:creator>
            <pubDate>Thu, 16 Jan 2020 00:43:26 GMT</pubDate>
            <atom:updated>2020-01-16T01:36:22.321Z</atom:updated>
            <content:encoded><![CDATA[<p>I’m currently developing a single-cell RNA Sequencing pipeline for <a href="https://www.networkanalyst.ca/">NetworkAnalyst</a>; an open source, web-based platform for comprehensive gene expression profiling &amp; meta-analysis. The website uses mostly R code (via {RServe}📦 ), Java for the heavy-lifting and Javascript for data visualization. Developing this pipeline I’ve been applying various profiling and timing tools to optimize R code and would like to share my experiences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2RRygwRfWiRQMyS5Hn8tZw.png" /></figure><p>We decided to use the {Seurat} 📦 from the <a href="https://satijalab.org/seurat/">Satija Lab</a> because it is one of the most comprehensive packages for end-to-end scRNA-Seq analysis (it includes tools for QC, analysis, visualization, clustering, DE analysis, analysis of spatial data <em>etc.</em>).</p><p>When you’re hosting multiple concurrent users on a web-based platform <em>memory </em>is of the utmost concern. Using the {pryr}📦 we can see the total memory used when we start R:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*7UQgCFDYeJ5PBoSh7jIJyw.png" /></figure><blockquote><a href="https://stackoverflow.com/a/59761861/8564751">When a user starts a session, any packages loaded will persist in memory — even if you try to unload them!</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*pI6akqU5rOS2eRSZ1uOsQQ.png" /></figure><p>What if we try to detach all of them the same way? 💣(this happens even with the force = TRUE paramater set). What about unloadNamespace()? It won’t work the <em>memory footprint </em>will persist.</p><p>The way we get around this with large 📦’s that are only used at one step of the pipeline (let’s use {DESeq2}📦 which is used for DE analysis as an example) is to use {RSclient}📦, which creates a microservice. Essentially, it allows another R session to start new RServe sessions pass data/functions between the sessions and close the session when the task is done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*SPxjkk-QZ9AIobVtTfmf1g.png" /></figure><p>However, Seurat is used throughout the pipeline so a different strategy is required. It would be beneficial to reduce the size of Seurat package as many concurrent users on the server would create a large memory footprint.</p><p>Seurat has many functions and analysis methods, you can think of it like a Swiss army knife —<strong><em> a really really big one</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*coZw4PCGbIDkVDCAKps3vw.jpeg" /></figure><p>Now many of these features won’t be of interest to our users (<em>e.g.</em><a href="https://satijalab.org/seurat/v3.1/integration.html"> Multiple Dataset Integration and Label Transfer</a>) so we can get rid of them (and their dependencies) to slim down the package. We want to end up with a smaller, yet, multi-purpose, binary of Seurat.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*Dl2GxIeExqttNzveLbJmcA.jpeg" /></figure><blockquote>If users need/want more functionality they can always install Seurat and work through things locally themselves. After all, one of the benefits to web-based servers like NetworkAnalyst and Galaxy is the low-bar of entry to wetlab biologists, undergraduate students, clinicians <em>etc.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*xF79zXBaCfDhBALfXplgwQ.png" /><figcaption><a href="https://www.biostars.org/p/50034/">Comment found on Biostars</a></figcaption></figure><p>After removing more than 75 functions (over 21k lines of code) and reducing the dependencies from <strong>46 -&gt; 24</strong>. I <strong>reduced the memory of the footprint by 1/3 </strong>(~145Mb -&gt; ~95Mb). [<a href="https://github.com/moldach/seurat">link to Github</a>]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*GzFd5VkYKLjwpkvfgTm_UQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*YmnuMAUyY-cO8XFAf6PmAA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2cPpl0RRbE-TtYXYfbV_3A.jpeg" /></figure><p>If you find this article useful feel free to share it with others or recommend this article! 😃</p><p>As always, if you have any questions or comments feel free to leave your feedback below or you can always reach me on <a href="https://www.linkedin.com/in/matthewoldach/">LinkedIn</a>. Till then, see you in the next post! 😄</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13cf0dae302" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Identifying regional differences in Perinatal Mental Health  indicators in the UK with R]]></title>
            <link>https://medium.com/data-science/identifying-regional-differences-in-perinatal-mental-health-indicators-in-the-uk-with-r-1e8a8f1e7fb0?source=rss-459e62b88a2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e8a8f1e7fb0</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[geography]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[perinatal-mental-health]]></category>
            <dc:creator><![CDATA[Matt.0]]></dc:creator>
            <pubDate>Fri, 26 Apr 2019 04:17:09 GMT</pubDate>
            <atom:updated>2019-04-26T13:34:08.813Z</atom:updated>
            <content:encoded><![CDATA[<h3>Identifying regional differences of Perinatal Mental Health indicators in the UK with R</h3><p>The <strong>fingertipsR</strong> package provides an easy interface to access the <a href="https://fingertips.phe.org.uk/">fingertips</a> API. This repository contains a large variety of public health indicators managed by Public Health England.</p><p>I will be focusing on the data related to <strong>Perinatal Mental Health</strong> as <a href="https://www.meaney.lab.mcgill.ca/">our laboratory</a> is interested in (among other things) the epigenetic embedding of early adversity. Perinatal mental health problems are those which occur during pregnancy or in the first year following the birth of a child. Perinatal mental illness affects up to 20% of women. Perinatal mental health problems can also have long-standing effects on the children’s emotional, social and cognitive development. If left untreated, it can have significant and long lasting effects on the woman and her family.</p><p>I thought it would be interesting to use Principle Component Analysis and Partitioning around medioids on monitoring indicators to identify regional variations that could provide improvements for control in those areas (like has been done with <a href="https://www.samabbott.co.uk/post/cluster-england-tb/">TB</a>).</p><pre># load the needs package to load a bunch of libraries quickly (or install them if you don&#39;t already have them)<br>library(needs)</pre><pre>needs(purrr,<br>      dplyr,<br>      tibble,<br>      fingertipsR,<br>      tidyr,<br>      magrittr,<br>      FactoMineR,<br>      impute,<br>      DT,<br>      broom,<br>      stringr,<br>      skimr,<br>      cluster,<br>      ggplot2,<br>      ggfortify,<br>      viridis,<br>      hrbrthemes,<br>      ggthemes,<br>      cluster,<br>      maptools,<br>      gpclib<br>      )</pre><h4>Let’s find out what is within the fingertips API</h4><p>The<strong> profiles()</strong> function is used to search<em> profiles</em> — indicators related to a specific disease or risk factor.</p><pre>profs &lt;- profiles()<br>DT::datatable(profs, rownames= FALSE)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*mpxQKZmt_E4sX_uTeQi_Xw.png" /></figure><p>As mentioned before I’m interested in <strong>Perinatal Mental Health</strong> so I’ll select those profiles.</p><pre>sel_profs &lt;- profs[grepl(&quot;Perinatal Mental Health&quot;, profs$ProfileName),]<br>DT::datatable(sel_profs, rownames = FALSE)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*wQ9uL7W3IlJyxDikLijtdA.png" /></figure><p>There are four<strong> Domains</strong> related to <strong>Perinatal Mental Health</strong>. Within these domains are a number of <strong>indicators</strong> presented at different time periods, geographies, sexes, age ranges <em>etc.</em></p><pre>peri_inds &lt;- indicators(ProfileID = sel_profs$ProfileID)<br>DT::datatable(peri_inds, rownames= FALSE)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*cMYDTuvnIYwYggpU9mw4YQ.png" /></figure><p>This gives us <strong>49</strong> Perinatal mental health <strong>indicators</strong>, which we can now extract using the <strong>fingertips_data()</strong> function combined with a call to <strong>purrr::map()</strong>.</p><pre>peri_df &lt;- peri_inds$IndicatorID %&gt;% map(~fingertips_data(IndicatorID = .))</pre><p>This results in <strong>49 </strong>tibbles; however, many are empty and need to be removed.</p><p>I will use dimension reduction (PCA) and clustering (PAM) on these 49 indicators to generate clusters of counties with similar characteristics of Perinatal Mental Health indicators. This may help to identify regional variation and possibly a future framework for improvements in research and control efforts.</p><p>I will choose <strong>Depression: % recorded prevalence (aged 18+)</strong> from the <strong>Risk &amp; related factors</strong> domain as the key indicator for which all others will be matched. The following code extracts this at the county-level, re-codes the value variable as recent incidence rates, and pulls the overall incidence of cases. I filtered out missing data and adjusted the time period to represent the final year for each rolling average.</p><pre>peri_inc &lt;- as_tibble(peri_df[[10]]) %&gt;% <br>        dplyr::filter(AreaType %in% &quot;County &amp; UA&quot;) %&gt;% <br>        dplyr::select(AreaName, Timeperiod, rec_inc_rate = Value) %&gt;% <br>        mutate(Timeperiod = Timeperiod %&gt;% <br>                 str_split(&quot;/&quot;) %&gt;% <br>                 map_chr(first) %&gt;% <br>                 as.numeric() %&gt;% <br>                 {. + 2} %&gt;% <br>                 as.character())</pre><pre>peri_df_extraction &lt;- function(peri_df, var_name, area_type = &quot;County &amp; UA&quot;) {<br>  df &lt;- peri_df %&gt;% <br>    filter(AreaType %in% area_type) %&gt;% <br>    dplyr::select(AreaName, Value, Timeperiod) %&gt;% <br>    rename_at(.vars = vars(Value), funs(paste0(var_name)))<br>  <br>  return(df)<br>}</pre><pre>var_names &lt;- c(&quot;fertility_rate&quot;,<br>               &quot;booking_under_20&quot;,<br>               &quot;reproductive_age&quot;,<br>               &quot;non_UK&quot;,<br>               &quot;birth_under_20&quot;,<br>               &quot;booking_over_40&quot;,<br>               &quot;booking_BAME&quot;,<br>               &quot;poverty&quot;,<br>               &quot;IDACI&quot;,<br>               &quot;homelessness&quot;,<br>               &quot;contraceptions_under_18&quot;,<br>               &quot;single_parent&quot;,<br>               &quot;sever_mental_illness&quot;,<br>               &quot;drug_treatment&quot;,<br>               &quot;alcohol_treatment&quot;,<br>               &quot;child_protection&quot;, <br>               &quot;child_in_need&quot;, <br>               &quot;infant_mortality&quot;,<br>               &quot;child_looked_after&quot;, <br>               &quot;sole_birth&quot;,<br>               &quot;complex_social_factors&quot;,<br>               &quot;multiparity&quot;,<br>               &quot;caesarean&quot;,<br>               &quot;preterm&quot;,<br>               &quot;stillbirth&quot;,<br>               &quot;abuse&quot;, <br>               &quot;skin_contact&quot;,<br>               &quot;distress_lb&quot;, <br>               &quot;distress_ub&quot;, <br>               &quot;mild_depressive_lb&quot;, <br>               &quot;mild_depressive_ub&quot;, <br>               &quot;chronic_SMI&quot;, <br>               &quot;postpartum_psychosis&quot;, <br>               &quot;PTSD&quot;, <br>               &quot;severe_depressive&quot;,<br>               &quot;booking_detection&quot;,<br>               &quot;booking_substance&quot;,<br>               &quot;booking_support&quot;,<br>               &quot;booking_alcohol&quot;,<br>               &quot;booking_complex_social_factors&quot;,<br>               &quot;booking_early&quot;,<br>               &quot;less_14_days&quot;,<br>               &quot;review_8_weeks&quot;,<br>               &quot;review_12_months&quot;,<br>               &quot;antenatal_MH_detection&quot;,<br>               &quot;postnatal_MH_detection&quot;,<br>               &quot;postnatal_emotional_change&quot;,<br>               &quot;postnatal_MH_support&quot;<br>               )</pre><pre>extracted_tb &lt;- map2(peri_df[-10], var_names, ~peri_df_extraction(.x, .y)) %&gt;% <br>    reduce(full_join, by = c(&quot;AreaName&quot;, &quot;Timeperiod&quot;))</pre><pre>com_peri_df &lt;- peri_inc %&gt;% <br>  left_join(extracted_tb, by = c(&quot;AreaName&quot;, &quot;Timeperiod&quot;)) %&gt;% <br>  mutate(year = Timeperiod %&gt;% as.numeric) %&gt;% <br>  mutate_if(is.character, as.factor) %&gt;% <br>  dplyr::select(-Timeperiod)</pre><pre>skimr::skim(com_peri_df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*PEgJA5CiJB2gkZG1exREvw.png" /></figure><p>We need to check completeness of the data:</p><pre>get_frac_missing &lt;- function(df) {<br>  df %&gt;% <br>    nest() %&gt;% <br>    mutate(missing = map(data,~map_dfr(. ,~sum(is.na(.))/length(.)))) %&gt;% <br>    dplyr::select(-data) %&gt;% <br>    unnest(missing) <br>}</pre><pre>## Get the proportion missing per variableby year<br>peri_miss_per_year &lt;- com_peri_df %&gt;% <br>  group_by(year) %&gt;% <br>  get_frac_missing %&gt;% <br>  mutate_all(~round(., 2)) %&gt;% <br>  arrange(year)</pre><pre>## Drop full missing variables<br>peri_partial_miss_year &lt;- peri_miss_per_year %&gt;% <br>  select_if(~!sum(.) == length(.))</pre><pre>## Full missing variables<br>com_miss_vars &lt;- setdiff(names(peri_miss_per_year), names(peri_partial_miss_year))</pre><pre>## Which year has the most complete data<br>peri_complete_years_all_vars &lt;- com_peri_df %&gt;% <br>  group_by(year) %&gt;% <br>  nest() %&gt;% <br>  mutate(missing = map(data,~mean(colSums(is.na(.))/nrow(.)))) %&gt;% <br>  dplyr::select(-data) %&gt;% <br>  unnest(missing) %&gt;% <br>  mutate(missing = round(missing, 2)) %&gt;% <br>  arrange(year)</pre><pre>DT::datatable(peri_complete_years_all_vars, rownames = FALSE)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*9wt1MX8VOupd-mayX6jkUw.png" /></figure><p>Usually I would go with the most recent data available; however, in this case it’s quite sparse so let’s look at the data from <strong>2015.</strong></p><pre>peri_df_2015  &lt;- janitor::remove_empty(com_peri_df) %&gt;% <br>  filter(year == 2015) %&gt;% <br>  filter(rec_inc_rate &gt; 7.39) %&gt;% <br>  select(-year)</pre><pre>DT::datatable(peri_df_2015, rownames = FALSE)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t907chcsJPF7jTMyoHP_Mg.png" /></figure><p>This leaves us with a tidy dataset on 8 maternal mental health indicators for 38 counties from 2015.</p><h4>Dimension reduction</h4><p>We are now ready to conduct some clustering analysis on the data. The first step is to reduce the dimensionality of the data using PCA. We use the <strong>estim_ncp</strong> function (which uses a <a href="https://www.sciencedirect.com/science/article/pii/S0167947311004099">method outlined in this paper</a> from the FactoMineR package to estimate the number of principal components required. We then perform PCA and plot the variance explained by each component as a check on <strong>estim_ncp</strong>. All of the following analysis is done using nested tibbles and so can be easily generalized to higher dimensional use cases.</p><p>Let’s calculate the optimal number of principal components is using the <strong>estim_ncp</strong> function, perform PCA using the <strong>prcomp </strong>function and plot the variance explained by each component as a check on <strong>estim_ncp</strong>.</p><pre>df &lt;- as.data.frame(peri_df_2015)<br>FactoMineR::estim_ncp(df[2:9], ncp.min = 2, ncp.max = 10) </pre><p>The optimal number of components is <strong>4.</strong></p><pre>peri_pca &lt;- peri_df_2015 %&gt;% <br>  nest() %&gt;% <br>  mutate(<br>    numeric_data = map(data, ~select_if(., is.numeric) %&gt;% <br>                         as.data.frame()),<br>    optimal_pca_no = map(numeric_data, ~estim_ncp(., <br>                                                  scale = TRUE, <br>                                                  ncp.min = 2, <br>                                                  ncp.max = 6)) %&gt;% <br>      map_dbl(~.$ncp),<br>    pca = map(numeric_data, ~prcomp(.x, <br>                                    center = TRUE, <br>                                    scale = TRUE)),<br>    pca_data = map(pca, ~.$x),<br>    pca_aug = map2(pca, data, ~augment(.x, data = .y)))</pre><pre>## Variance explained<br>var_exp &lt;- peri_pca %&gt;% <br>  dplyr::select(-optimal_pca_no) %&gt;% <br>  unnest(pca_aug) %&gt;% <br>  summarize_at(.vars = vars(contains(&quot;PC&quot;)), .funs = funs(var)) %&gt;% <br>  gather(key = pc, value = variance) %&gt;% <br>  mutate(var_exp = variance/sum(variance) * 100,<br>         cum_var_exp = cumsum(var_exp),<br>         pc = str_replace(pc, &quot;.fitted&quot;, &quot;&quot;) %&gt;% <br>           str_replace(&quot;PC&quot;, &quot;&quot;))</pre><pre>## Plot variance explained<br>var_exp %&gt;% <br>  rename(<br>    `Variance Explained` = var_exp,<br>    `Cumulative Variance Explained` = cum_var_exp<br>  ) %&gt;% <br>  gather(key = key, value = value, `Variance Explained`, `Cumulative Variance Explained`) %&gt;%<br>  mutate(key = key %&gt;% <br>           factor(levels  = c(&quot;Variance Explained&quot;, <br>                              &quot;Cumulative Variance Explained&quot;))) %&gt;% <br>  mutate(value = value / 100) %&gt;% <br>  mutate(pc = factor(pc, levels = as.character(1:max(var_exp$pc %&gt;% as.numeric)))) %&gt;% <br>  ggplot(aes(pc, value, group = key)) + <br>  geom_point(size = 2, alpha = 0.8) + <br>  geom_line(size = 1.1, alpha = 0.6) + <br>  facet_wrap(~key, scales = &quot;free_y&quot;) +<br>  theme_bw() +<br>  labs(<br>    title = &quot;Variance Explained by Principal Component&quot;,<br>    subtitle = paste0(&quot;The optimal number of principal components suggested by estim_ncp was &quot;,<br>                      peri_pca$optimal_pca_no, &quot; which explains &quot;, round(var_exp$cum_var_exp[[2]], 0), &quot;% of the data.&quot;),<br>    x = &quot;Principal Component&quot;,<br>    y = &quot;Variance Explained (%)&quot;,<br>    caption = &quot;<a href="http://twitter.com/MattOldach">@MattOldach</a> Source: Public Health England (fingertipsR)&quot;<br>  )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hiZWHvS1MluZ9Kiq-POI9g.png" /></figure><pre>## Perform pam on pca data 1 to 6 groups<br>peri_pca_pam &lt;- peri_pca %&gt;%<br>  mutate(centers = list(2:10)) %&gt;% <br>  unnest(centers, .preserve = everything()) %&gt;% <br>  dplyr::select(-centers, centers = centers1) %&gt;% <br>  group_by(centers) %&gt;% <br>  mutate(<br>    pam = map(pca_data,<br>              ~ pam(x = .x[, 1:optimal_pca_no], k = centers, stand = TRUE)),<br>    clusters = map(pam, ~.$clustering),<br>    avg_silhouette_width = map(pam, ~.$silinfo$avg.width),<br>    data_with_clusters = map2(.x = data, .y = clusters, ~mutate(.x, cluster = factor(.y, ordered = TRUE)))<br>  ) %&gt;% <br>  ungroup</pre><pre>## Get max silhouette width<br>max_silhouette_width &lt;- peri_pca_pam %&gt;% <br>  dplyr::select(centers, avg_silhouette_width) %&gt;% <br>  unnest(avg_silhouette_width) %&gt;% <br>  arrange(desc(avg_silhouette_width)) %&gt;% <br>  slice(1)<br>  <br>## Plot average silhouette width<br>peri_pca_pam %&gt;% <br>  dplyr::select(centers, avg_silhouette_width) %&gt;% <br>  unnest(avg_silhouette_width) %&gt;% <br>  ggplot(aes(x = centers, y = avg_silhouette_width)) +<br>  geom_line(size = 2, alpha = 0.4) +<br>  geom_point(size = 3, alpha = 0.8) +<br>  theme_bw() +<br>  scale_x_continuous(breaks = seq(1, 10, 1), minor_breaks = NULL) +<br>  scale_y_continuous(limits = c(NA, NA), breaks = seq(0, 1, 0.01), minor_breaks = NULL) +<br>  labs(title = &quot;Average Silhouette Width by Number of PAM Clusters&quot;,<br>       subtitle = paste0(&quot;The optimal number of clusters identifed by avg. silhouette width was &quot;,<br>                      max_silhouette_width$centers,<br>                      &quot; with an avg. silhouette width of &quot;, <br>                      round(max_silhouette_width$avg_silhouette_width, 2)<br>       ),<br>       x = &quot;Clusters&quot;,<br>       y = &quot;Avg. Silhouette Width&quot;,<br>       caption = &quot;<a href="http://twitter.com/MattOldach">@MattOldach</a> Source: Public Health England (fingertipsR)&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LgriD3bUGeaX0p9hdCIWsg.png" /></figure><pre>## Plot clusters<br>pca_plot &lt;- peri_pca_pam %&gt;% <br>  filter(centers == max_silhouette_width$centers) %&gt;% <br>  select(data_with_clusters, pca) %&gt;% <br>  mutate(pca_graph = map2(.x = pca, <br>                          .y = data_with_clusters,<br>                          ~ autoplot(.x, x = 1, y = 2, <br>                                     loadings = TRUE, loadings.label = TRUE,<br>                                     loadings.label.repel = TRUE,<br>                                     loadings.label.size = 2, loadings.alpha = 0.8,<br>                                     loadings.label.vjust = -1, data = .y, <br>                                     label = TRUE, label.label = &quot;AreaName&quot;,<br>                                     label.size = 1.5, label.vjust = -1, <br>                                     alpha = 0.3, frame = TRUE, <br>                                     frame.type = &#39;convex&#39;, frame.alpha= 0.05,<br>                                     colour = &quot;cluster&quot;, size = &quot;rec_inc_rate&quot;) +<br>                            theme_bw() +<br>                            labs(x = paste0(&quot;Principal Component 1 (Variance Explained: &quot;,<br>                                            round(var_exp$var_exp[[1]], 1), &quot;%)&quot;),<br>                                 y = paste0(&quot;Principal Component 2 (Variance Explained: &quot;,<br>                                            round(var_exp$var_exp[[2]], 1), &quot;%)&quot;)) +<br>                            guides(colour=guide_legend(title = &quot;Cluster&quot;, ncol = 2), <br>                                   fill=guide_legend(title= &quot;Cluster&quot;, ncol = 2),<br>                                   size = guide_legend(title = &quot;Depression: % recorded prevalence&quot;,<br>                                                       ncol = 2)) +<br>                           scale_color_manual(name = &quot;Cluster&quot;,<br>                                              values = c(&quot;#6b5b95&quot;,<br>                                                         &quot;#feb236&quot;,<br>                                                         &quot;#d64161&quot;,<br>                                                         &quot;#ff7b25&quot;,<br>                                                         &quot;#87bdd8&quot;)) +<br>                           scale_fill_manual(name = &quot;Cluster&quot;,<br>                                              values = c(&quot;#6b5b95&quot;,<br>                                                         &quot;#feb236&quot;,<br>                                                         &quot;#d64161&quot;,<br>                                                         &quot;#ff7b25&quot;,<br>                                                         &quot;#87bdd8&quot;)) +<br>                            theme(legend.position = &quot;bottom&quot;, <br>                                  legend.box = &quot;horizontal&quot;) +<br>                            labs(<br>                              title = &quot;Maternal Mental Health in England&quot;,<br>                              subtitle = &quot;The arrows are variable loadings and points are counties coloured by cluster membership&quot;,<br>                              caption = &quot;<a href="http://twitter.com/MattOldach">@MattOldach</a> Source: Public Health England (fingertipsR)&quot;<br>                            )<br>  )) %&gt;% <br>  pull(pca_graph) %&gt;% <br>  first</pre><pre>pca_plot</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tQ2WXXKfM06Cy_dqXlB0OA.png" /></figure><pre>sum_peri_df &lt;- peri_pca_pam %&gt;% <br>  filter(centers == max_silhouette_width$centers) %&gt;% <br>  pull(data_with_clusters) %&gt;% <br>  map(~ gather(., key = &quot;Variable&quot;, value = &quot;value&quot;, -AreaName, -cluster)) %&gt;% <br>  first %&gt;% <br>  rename(Cluster = cluster)</pre><pre>plot_cluster_diff &lt;- sum_peri_df %&gt;% <br>  ggplot(aes(x = Variable, y = value, col = Cluster, fill = Cluster)) +<br>  geom_violin(draw_quantiles = c(0.025, 0.5, 0.975), alpha = 0.2, scale = &quot;width&quot;) +<br>  geom_jitter(position = position_jitterdodge(), alpha = 0.3) +<br>  coord_flip() +<br>  theme_minimal() +<br>  scale_y_continuous(breaks = seq(0, 100, 6), minor_breaks = NULL) +<br>  scale_fill_manual(name = &quot;Cluster&quot;,<br>                                              values = c(&quot;#6b5b95&quot;,<br>                                                         &quot;#feb236&quot;,<br>                                                         &quot;#d64161&quot;,<br>                                                         &quot;#ff7b25&quot;,<br>                                                         &quot;#87bdd8&quot;)) +<br>  scale_color_manual(name = &quot;Cluster&quot;,<br>                                              values = c(&quot;#6b5b95&quot;,<br>                                                         &quot;#feb236&quot;,<br>                                                         &quot;#d64161&quot;,<br>                                                         &quot;#ff7b25&quot;,<br>                                                         &quot;#87bdd8&quot;)) +<br>  theme(legend.position = &quot;bottom&quot;) +<br>  labs( <br>    title = &quot;Maternal Mental Health in England; Summarised by Cluster&quot;,<br>    subtitle = &quot;Violin plots are scaled by width, with the 2.5%, 50% and 97.5% quantiles shown.&quot;,<br>    x = &quot;Variable&quot;,<br>    y = &quot;Recorded % depression prevalance (aged 18+) for rec_int_rate, otherwise proportion (0-100%)&quot;,<br>    caption = &quot;<a href="http://twitter.com/MattOldach">@MattOldach</a> Source: Public Health England (fingertipsR)&quot;)</pre><pre>plot_cluster_diff</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y82z9AZP0OLkN4Hbh2sEew.png" /></figure><p>From these plots we see that the thrid cluster showing a higher recent increase rate has a highger proportion of postpartum psychosis, mild depression and distress.</p><p>We can plot the cluster membership for each county on a map with the outlines of the counties in England from the UK data service (<a href="https://borders.ukdataservice.ac.uk/easy_download_data.html?data=England_ct_2011">download the ShapeFiles here</a> and <a href="https://borders.ukdataservice.ac.uk/easy_download_data.html?data=England_urb_2001">here</a>).</p><pre>peri_cluster_map &lt;- function(peri_pca_pam) {<br>gpclibPermit()</pre><pre>england_counties &lt;- rgdal::readOGR(&quot;england_ct_2011.shp&quot;) %&gt;%<br>  fortify(region = &quot;code&quot;) %&gt;% <br>  as_tibble</pre><pre>england_urban_areas &lt;- rgdal::readOGR(&quot;england_urb_2001.shp&quot;) %&gt;% <br>  fortify(region = &quot;name&quot;) %&gt;% <br>  as_tibble %&gt;% <br>  filter(id %in% c(&quot;Sheffield Urban Area&quot;, <br>                   &quot;Plymouth&quot;,<br>                   &quot;Skelton (Redcar and Cleveland)&quot;,<br>                   &quot;Brighton/Worthing/Littlehampton&quot;,<br>                   &quot;Leicester Urban Area&quot;<br>                   ))</pre><pre>## Make custom positions for urban area labels<br>urban_area_labels &lt;- england_urban_areas %&gt;%<br>  group_by(id) %&gt;% <br>  slice(100) %&gt;% <br>  ungroup() %&gt;% <br>  mutate(long = long - 200000,<br>         lat = lat + 20000)<br>  <br>  <br>peri_cluster_results &lt;- peri_pca_pam %&gt;% <br>  filter(centers == max_silhouette_width$centers) %&gt;% <br>  pull(data_with_clusters) %&gt;% <br>  first</pre><pre>peri_cluster_results &lt;- peri_df[[14]] %&gt;% <br>              dplyr::select(AreaName, AreaCode, AreaType) %&gt;% <br>  filter(AreaType %in% &quot;County &amp; UA&quot;) %&gt;% <br>              unique %&gt;% <br>  left_join(peri_cluster_results,<br>            by = &quot;AreaName&quot;) %&gt;% <br>  left_join(england_counties, by = c(&quot;AreaCode&quot; = &quot;id&quot;))</pre><pre>peri_cluster_results %&gt;% <br>  rename(Cluster = cluster) %&gt;% <br>  drop_na(Cluster) %&gt;% <br>  dplyr::select(long, lat, Cluster, group) %&gt;% <br>  ggplot( <br>                 aes(x = long, <br>                     y = lat,<br>                     fill = Cluster)) +<br>    geom_polygon(data = england_urban_areas, <br>                 aes(group = group, fill = NULL),<br>                 alpha = 0.4) +<br>    geom_path(data = peri_cluster_results, <br>              aes(group = group, fill = NULL), <br>              alpha = 0.4) +<br>    geom_polygon(data = peri_cluster_results, <br>                 aes(group = group, fill = NULL),<br>                 alpha = 0.1) +<br>    geom_polygon(aes(group = group), alpha = 0.6) +<br>    geom_line(data = urban_area_labels %&gt;% <br>                bind_rows(urban_area_labels %&gt;% <br>                            mutate(long = long + 200000, <br>                                   lat = lat - 20000)),<br>              aes(fill = NA, group = id), alpha = 0.8) + <br>    geom_label(data = urban_area_labels,<br>              aes(label = id), fill = &quot;grey&quot;) +<br>    scale_fill_manual(name = &quot;Cluster&quot;,<br>                                              values = c(&quot;#6b5b95&quot;,<br>                                                         &quot;#feb236&quot;,<br>                                                         &quot;#d64161&quot;,<br>                                                         &quot;#ff7b25&quot;,<br>                                                         &quot;#87bdd8&quot;)) +<br>    coord_equal() +<br>    theme_map() +<br>    theme(legend.position = &quot;bottom&quot;) +<br>    labs(title = &quot;Maternal Mental Health Indicators; Map of County Level Clusters in England&quot;,<br>         subtitle = &quot;Using data from 2015&quot;,<br>         caption = &quot;Selected urban areas are shown (dark grey) and labelled.<br><a href="http://twitter.com/MattOldach">@MattOldach</a> Source: Public Health England (fingertipsR)<br>Contains National Statistics data © Crown copyright and database right 2019. <br>         Contains OS data © Crown copyright and database right 2019&quot;)<br>}</pre><pre>plot_peri_cluster_map &lt;- peri_cluster_map(peri_pca_pam)</pre><pre>plot_peri_cluster_map</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*kTo7dOaRbr29F1m8Dq4n5w.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e8a8f1e7fb0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/identifying-regional-differences-in-perinatal-mental-health-indicators-in-the-uk-with-r-1e8a8f1e7fb0">Identifying regional differences in Perinatal Mental Health  indicators in the UK with R</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Choose the Best Open Source Software]]></title>
            <link>https://medium.com/data-science/how-to-choose-the-best-open-source-software-b1cbbe4f6398?source=rss-459e62b88a2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1cbbe4f6398</guid>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <dc:creator><![CDATA[Matt.0]]></dc:creator>
            <pubDate>Fri, 22 Mar 2019 01:03:10 GMT</pubDate>
            <atom:updated>2019-03-22T13:04:50.168Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dsb3sI5IRCnbgzFZIXbXUQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/ZV_64LdGoao?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Pankaj Patel</a> on <a href="https://unsplash.com/search/photos/server?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>After reading the O’Reilly book “<a href="http://shop.oreilly.com/product/0636920161417.do"><strong>Foundations for Architecting Data Solutions</strong></a>”, by Ted Malaska and Jonathan Seidman, I reflected on how I chose <strong>software/tools/solutions </strong>in the past and how I should choose them going forward.</p><p>As a bioinformatician you need to be able to quickly discern whether a publication/tool is really a major advancement or just marginally better. I’m not just talking about the newest single-cell RNA-seq technique or <a href="http://omicsomics.blogspot.com/2015/08/the-road-to-hell-is-paved-with.html?spref=tw&amp;m=1">another file format</a>, but for every <strong>problem </strong>case you have. Whether that be data visualization tools, <a href="https://moldach.github.io/xaringan-presentation_drake/">presentation tools</a>, distributed storage systems <em>etc.</em></p><p>It’s not just about how useful the tool may be, it also depends on the quality of the documentation, how simple it is to install, where it sits in the open-source life cycle, <em>etc.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*9nMBMt-OugnruBr_M-WuEQ.png" /></figure><p>Xkcd is funny but competing standards aren’t. Don’t believe me? Just look at how many <a href="https://github.com/pditommaso/awesome-pipeline">pipeline tools</a> exist!</p><blockquote>When faced with so many options how can one choose the solutions that fit their need?</blockquote><h3>Why open source?</h3><p>I’ve worked with a few licensed software solutions in the past; for example, <a href="https://www.blast2go.com/blast2go-pro">BLAST2GO</a> (<a href="https://github.com/moldach/Transcriptome_Assembly-Annotation">plug: use dammit from Camille Scott instead!</a>), <a href="https://www.mathworks.com/products/matlab.html">Matlab</a>, and an image stitching software called <a href="http://www.kolor.com/">Autopano Giga</a> (now defunct). One of my greatest frustrations was learning these tools only to later change roles and no longer have them available. As a consultant for the Department of Fisheries and Oceans the prohibitive cost of a Matlab was what pushed me over the edge into learning another high-level programming language — R. FWIW:</p><blockquote>“[Matlab] They obfuscate their source code in many cases, meaning bugs are much <a href="https://uk.mathworks.com/matlabcentral/answers/79714-how-do-we-know-that-matlabs-algorithms-are-working-properly">harder to spot</a> and impossible to <a href="http://stackoverflow.com/questions/2470765/can-i-distribute-my-matlab-program-as-open-source">edit ourselves without risking court action</a>. Moreover, using Matlab for science results in <a href="https://github.com/openjournals/joss/issues/142">paywalling our code</a>. We are by definition making our computational science closed.” — excerpt from <a href="http://neuroplausible.com/matlab"><strong>I Hate Matlab: How an IDE, a Language, and a Mentality Harm</strong></a></blockquote><p>Most companies eschew third party solutions or build their product as a hybrid of proprietary and open-source to keep their costs lower. For example, Amazon Web Services (AWS) offers it’s Simple Storage Service (Amazon S3) for a fee but is built upon open source software like Apache Hadoop. I’m not saying <em>not </em>to use AWS (or any other cloud provider) because sometimes you are constrained to <em>having </em>to; I actually used AWS for a project (<a href="https://www.sciencedirect.com/science/article/pii/S1874778717303422">transcriptome assembly of a coral species</a>) with Docker. Currently I’m working with sensitive information that must be kept on-site, under lock-and-key, so alternative solutions are used.</p><p>Most of the newer big data platforms, and successful open-source projects began as internal projects at companies or universities for the first couple years before going through an external incubation phase. For example:</p><ul><li>LinkedIn — “<strong>Apache Kafka</strong>”</li><li>University of California at Berkeley — “<strong>Apache Spark</strong>”</li><li>Cloudera — “<strong>Impala</strong>”</li><li>Yahoo! — “<strong>Apache Hadoop</strong>”</li><li>Google — “<strong>Kubernetes”</strong></li><li>Facebook — “<strong>Apache Hive</strong>”</li></ul><p>There are benefits to choosing open-source projects backed by solid sponsors with good reputation, solid devs, and track record of sponsoring successful projects. You can be fairly confident that these projects have a solid codebase, great documentation, received session time at conferences, and considerable public recognition (through blog posts and articles surrounding it).</p><p>When considering open-source solutions it’s also important to gauge where they are in the <em>open-source life cycle. </em>According to Malaska and Seidman, there are nine (potential) stages in the project life cycle based on the <em>Garnter Hype Cycle</em>; however, I think only a few are relevant to discuss here:</p><h3>Which Cycle Should You Choose?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z_sloB4XJ1Mdc9rUJgs1BA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/KeinUoPC6qM?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">@Matthew_T_Rader</a> on <a href="https://unsplash.com/search/photos/cycle?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h4>Don’t believe the hype</h4><p>This stage of the cycle is referred to as the “<em>curing cancer” </em>stage. The hype at this stage is important for attracting committers and contributors but unless you’re looking to help out in a major way you should steer clear. Unless you’re trying to be on the cutting edge (risk tolerance), or take on an active role as a contributor, it’s best to wait 6–12 months before trying any new technology. By letting others hit walls first you’ll encounter fewer bugs and have access to better documentation and blog posts.</p><h4>A broken promise is not a lie</h4><p>After the “<em>curing cancer” </em>stage is the <em>broken promises </em>stage. At this point people are using the project and are finding issues or limitations. For example, a solution may not integrate nicely with other existing systems or there may be problems with scaleability. You should treat any open source project at this stage with cautious optimism.</p><h4>Go for dependable solutions whenever possible</h4><p>Projects in the <em>hardening </em>or <em>enterprise </em>stage have become mature technologies. The amount of commits will signal the level of investment in a project. Tthe type of commits tell a story, telling where the author(s) are trying to go with the code, revealing what they want to do by signalling interest in different features of the project. By now the initial excitement has died down and there is more demand for stability than new features. The initial development team may be working on other projects as it has developed a solid community — this is often a good sign of success of a project.</p><p>Obviously recent activity signals that the project is alive and maintained. Remember that there are many dead and abandoned projects living on Github. That being said, activity doesn’t always need to be<em> very</em> recent! One prolific, “Rockstar Dev”, put it this way:</p><blockquote>Context-switching is expensive, so if I worked on many packages at the same time, I’d never get anything done. Instead, at any point in time, most of my packages are lying fallow, steadily accumulating issues and ideas for new feature. Once a critical mass has accumulated, I’ll spend a couple of days on the package. — <a href="https://www.quora.com/How-is-Hadley-Wickham-able-to-contribute-so-much-to-R-particularly-in-the-form-of-packages"><em>Hadley Wickham</em></a></blockquote><p>Eventually projects enter the <em>decline </em>stage and no one wants to adopt or contribute to a dead or dying project.</p><h3>Can i trust you?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0_1arxZEvDUKjpr0QtDa4A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/OLgPKR2IJ6k?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Purnomo Capunk</a> on <a href="https://unsplash.com/search/photos/trust?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>I use R mostly so let me talk about where a project is hosted for a few moments. Code is often hosted on Github, <a href="https://ropensci.github.io/dev_guide/softwarereviewintro.html">ROpenSci</a>, <a href="http://blog.revolutionanalytics.com/2015/08/a-short-introduction-to-bioconductor.html">Bioconductor</a> or <a href="https://cran.r-project.org/web/packages/submission_checklist.html">CRAN</a>. The <a href="https://cran.r-project.org/">Comprehensive R Archive Network</a> (CRAN) <em>was</em> the main repository for R packages.</p><blockquote>“As R users, we are spoiled. Early in the history of R, Kurt Hornik and Friedrich Leisch built support for <em>packages</em> right into R, and started the Comprehensive R Archive Network (CRAN). And R and CRAN had a fantastic run with. Roughly twenty years later, we are looking at over 12,000 packages which can (generally) be installed with absolute ease and no suprises. No other (relevant) open source language has anything of comparable rigour and quality.” — excerpt from <a href="http://dirk.eddelbuettel.com/blog/2018/02/28/#017_dependencies">Dirk Eddelbuettel</a></blockquote><p>On CRAN packages of almost any type are welcome (as long as strict policies are met) and packages are tested daily (on multiple systems). <a href="https://milesmcbain.xyz/ropensci-onboarding1/">rOpenSci is the perfect antithesis of CRAN. CRAN can be notoriously opaque, inconsistent, and aloof</a>. It cannot deal with the volume of automation of CRAN but markets itself in terms of quality.</p><p>For the field of Bioinformatics Bioconductor is where a package will end up. Projects that exist solely on Github should be viewed with more caution as they have no checklists or peer-review.</p><h3>Let’s talk about dependencies (a loaded topic — no pun intended)</h3><p>Installing dependencies sucks! <a href="https://research.swtch.com/deps">How often have you installed one package only to have a boatload pulled-in?</a> You should try and avoid packages with many (changing) packages as this will be prohibitive to establish if your work is correct (hence ensuring reproducibility) because <a href="http://dirk.eddelbuettel.com/blog/2019/03/14/#020_dependency_badges">dependencies are hard to manage risks</a>.</p><blockquote><em>“</em><a href="http://dirk.eddelbuettel.com/blog/2019/03/14/#020_dependency_badges"><em>More dependencies means more edges between more nodes.</em> Which eventually means more breakage.</a>”</blockquote><p>Proponents of the <em>tinyverse </em>tend to stay away from bloated dependencies, <a href="https://medium.freecodecamp.org/why-im-not-using-your-github-repository-2dff6c7ac7cf">no one wants to spend time in hell!</a></p><p>If you’re a developer remember:</p><blockquote>“<a href="http://www.win-vector.com/blog/2019/03/software-dependencies-and-risk/">Not all dependencies are equal … some popular packages [have] unstable APIs (a history of breaking changes) and high historic error rates (a history of complexity and adding features over fixing things).</a>”</blockquote><p>You can also include a <a href="https://gitlab.com/edwindj/crandeps">badge for your repo showing the number of dependencies your package relies on</a></p><h3>Transparency is good</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pPAwTWTT9yHQo1cQ956qNQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/3cWA3U8xb5w?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Aleks Dahlberg</a> on <a href="https://unsplash.com/search/photos/transparent?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>When looking at projects on Github you should look for people/packages with many stars, watchers, forks, contributors, <em>etc. </em>These visible cues of community support indicate the community cares about a person, project, or action and that many others would benefit from it.</p><p>Remember that the amount of commits, issues and pull-requests (PRs) can be a signal of investment and commitment to a project. Are the issues and PRs being dealt with? The latter is literally an <em>offer </em>of code that is being ignored rather than accepted, rejected or commented upon.</p><p>By following the actions on code, you can determine who founded the project, what happened across different releases and make inferences about the structure of the project and collaborator roles (who had expertise on which pieces of the system). Linked commits and issues communicates the reasoning behind a change to the code.</p><p>You can also gauge community interest by looking at the number of <a href="https://www.meetup.com/">meetups,</a> and conferences (and their attendance levels) , or at email lists, user groups, community forums <em>et</em>c.</p><p>Google trends can also be a good measure of the level of interest in projects or technologies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*Swv5pqDoOdmzR4brQ6HRWA.png" /><figcaption><a href="http://lindeloev.net/spss-is-dying/">Using metrics to track the decline of SPSS</a></figcaption></figure><h3>Things to look for</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JC-K-v7Kq3HmMZgxpvAvYA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/9P2EwTqDEDM?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Arnaud Papa</a> on <a href="https://unsplash.com/search/photos/telescope?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><ul><li>Easy to install</li><li>Easy to run</li><li>Are there issues and PR raised</li></ul><p>Is the owner taking care of them (fixing bugs, helping users, adding features)? or was it abandoned?</p><ul><li><a href="https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006561">How good is the documentation/vignettes</a> (<a href="https://academic.oup.com/bib/article/19/4/693/2907814">and here</a>)</li></ul><p>Does it list hardware requirements (RAM and disk size), example commands, toy data, example output, screenshots/recordit’s</p><ul><li><a href="https://travis-ci.org/">Continuous integration status</a></li><li>Does it have a <a href="https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1002598">LICENSE</a></li><li>Does it have a <a href="https://gist.github.com/PurpleBooth/b24679402957c63ec426">CONTRIBUTING</a> doc</li><li>Does it have tests</li><li>Does it have a Dockerfile</li><li><a href="https://www.r-bloggers.com/all-the-badges-one-can-earn-parsing-badges-of-cran-packages-readmes/">Does it have badges</a></li></ul><p>For other things to look for see <a href="https://itnext.io/what-i-dont-like-in-your-repo-a602577a526b">here</a> and <a href="https://medium.freecodecamp.org/why-im-not-using-your-github-repository-2dff6c7ac7cf">here</a>.</p><h3>Bench-marking</h3><p>If you’re a software developer and considering incorporating one of a number of competing technologies you can perform internal benchmarks with your use cases and data.</p><p>If you’re using R there is different levels of magnification that a benchmark can provide. For a macro analysis (when computation is more intensive) you should use the <a href="https://cran.r-project.org/web/packages/rbenchmark/index.html">rbenchmark</a> package. For microscopic timing comparisons (<em>e.g. </em>nanoseconds elapsed) use the <a href="https://cran.r-project.org/web/packages/microbenchmark/index.html">microbenchmark</a> package</p><p>Sometimes other consortium&#39;s will have already done the bench-marking for you (for example <a href="http://assemblathon.org/">“<strong>The Assemblathon”</strong></a><strong>) . </strong>Nonetheless, one should be aware of hidden, or motivated biases trying to make unfair comparisons (use cases for which one tool clearly has an advantage). Also understand that testers could have been making an honest attempt at a fair test but made misunderstandings which lead to invalid results. Therefore it’s important to perform your own internal benchmarking and hold others benchmarks to an open standard of repeatability and verification.</p><h3>Final words</h3><p>Ultimately choosing a software solution comes down to the requirements of your project (the timeline, budget, and so forth), how willing are you to be on the cutting-edge (risk tolerance), and how capable team-members will be able to master these solutions based on their skill levels (internal skill set). Then, test out the solutions before fully committing. This job can be given to <em>the prototyper </em>role on your team; the person who likes experimenting/investigating new software.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1cbbe4f6398" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-choose-the-best-open-source-software-b1cbbe4f6398">How to Choose the Best Open Source Software</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web Scraping Google Sheets with RSelenium]]></title>
            <link>https://medium.com/data-science/web-scraping-google-sheets-with-rselenium-9001eda399b0?source=rss-459e62b88a2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9001eda399b0</guid>
            <category><![CDATA[selenium]]></category>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Matt.0]]></dc:creator>
            <pubDate>Tue, 29 Jan 2019 04:03:48 GMT</pubDate>
            <atom:updated>2019-05-17T12:43:32.216Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eBZrWn2oyPO3oAIm-v5uPg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/ZArDeAtxj0Q?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">freestocks.org</a> on <a href="https://unsplash.com/search/photos/web-scraping?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>I love to learn new things and one of ways I learn best is by doing. Also it’s been said that <a href="https://kottke.org/17/06/if-you-cant-explain-something-in-simple-terms-you-dont-understand-it">you never fully understand a topic until you are able to explain it </a>, I think blogging is a low barrier to explaining things.</p><p>Someone I met at a local data science meetup in Montréal wanted help web scraping to get team standings from the <a href="http://www.puzzledpint.com/standings/">PuzzledPint</a>. I jumped at the opportunity because I knew this would be my opportunity to finally learn <a href="https://github.com/ropensci/RSelenium"><strong>RSelenium!</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*txFP2X4BWaCm0fefpSzqbg.jpeg" /></figure><h3>Static Scraping vs. Dynamic Scraping</h3><p>Static scraping ignores JavaScript. It fetches web pages from the server without the help of a browser. You get exactly what you see in “view page source”, and then you slice and dice it. If the content you’re looking for is available, you need to go no further. However, if the content is something like an `iframe`, you need dynamic scraping.</p><p>Dynamic scraping uses an actual browser (or a headless browser) and lets JavaScript do its thing. Then, it queries the DOM to extract the content it’s looking for. Sometimes you need to automate the browser by simulating a user to get the content you need. In order for me to get the same details for the remaining posts, i would need to first navigate to the next page, which involves clicking the Next button at the bottom of the search results page.</p><h3>Setting up RSelenium with Docker</h3><p>RSelenium provides <strong>R</strong> bindings for the Selenium Webdriver API. <a href="http://docs.seleniumhq.org/">Selenium</a> is a project focused on automating web browsers.</p><p>You need to follow the <a href="https://docs.docker.com/toolbox/toolbox_install_windows/">installation instructions for Docker Toolbox on Windows or Ubuntu</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XvJ0GDWOAEHNApZvw-dOVQ.png" /></figure><p>Docker is used to run applications by using containers. Containers are simply a bundle of libraries and other dependencies in one package. You can think of it like a virtual machine, but rather than creating a whole OS it allows applications to use the same Linux kernel with only the <em>things</em> not already running on the host computer. Basically, it gives a significant performance boost and reduces the size of the application. Moreover, you can rest assured that the application will run on any other Linux machine regardless of any customized settings that machine might have that could differ from the machine used for writing and testing the code.</p><p>You’ll also need to <a href="http://www.tightvnc.com/download.php">install TightVNC</a> which will allow you to see how you’re manipulating the web page in real-time with RSelenium.</p><p>Next follow the <a href="http://ropensci.github.io/RSelenium/articles/docker.html">instructions to create a Docker container running a selenium server and its own firefox</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*rMlftpSVS9BHbh9UBwCgsw.png" /></figure><blockquote><em>Note: Once you’ve set up the docker container (and everytime you restart your computer or start-up fresh again) open the Docker Quickstart Terminal and run the following command.</em></blockquote><pre>docker run -d -p 4445:4444 selenium/standalone-firefox:2.53.0</pre><p>Now that you’ve booted your <strong>Docker Quickstart Terminal</strong> go into <strong>R </strong>and connect to a running server.</p><pre>library(RSelenium)<br>remDr &lt;- remoteDriver(<br>  remoteServerAddr = &quot;192.168.99.100&quot;,<br>  port = 4445L<br>)<br>remDr$open()</pre><p>Navigate to the page using Rselenium.</p><pre># navigate to the website of interest<br>remDr$navigate(&quot;<a href="http://www.puzzledpint.com/standings/">http://www.puzzledpint.com/standings/</a>&quot;)<br># confirm you got there<br>remDr$getTitle()0</pre><p>Okay let’s get a live screen shot of the site in the <strong>Viewer</strong> tab of RStudio.</p><pre>remDr$screenshot(display = TRUE)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tzaElAcPM2Mp-8tiVvHHXg.jpeg" /></figure><p>Keep in mind that this is just a static screen shot. Your going to want to use <strong>TightVNC</strong> to get a live view of your interactions while your developing your pipeline so you can see how your interacting with the website.</p><blockquote>It’s important to be watching TightVNC as you use the …$highlightElement() in-between your …$findElement() and …$switchToFrame()/…$clickElement() commands so that you actually know your selecting the appropriate things!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v4u3CxR8Z3e5PSwnzEpjKQ.jpeg" /></figure><p>Open the <strong>TighVNC Viewer</strong> and enter the port number; in this case 192.168.99.100 and enter that in the <strong>Remote Host:</strong> field. Click <strong>Connect</strong> and for the password enter the word: secret.</p><blockquote>Note: If TightVNC ever stops working (on Windows 10 it did often) and gives you the Error: “NO CONNECTION COULD BE MADE BECAUSE THE TARGET MACHINE ACTIVELY REFUSED IT” then follow the steps for <a href="http://ropensci.github.io/RSelenium/articles/docker.html">“Debugging Using VNC” here</a>.</blockquote><h3>Accessing Elements in the DOM</h3><p>Web pages are a set of nest objects (together, they are known as the <strong>Document Object Model</strong> or <strong>DOM</strong> for short). It’s a cross-platform and language-independent convention for representing and interacting with objects in HTML, XHTML and XML documents. Interacting with the DOM will be very important for us with RSelenium.</p><p>Hadley Wickham recommends using <a href="http://selectorgadget.com/">Selectorgadget</a>, a Chrome extension, to help identify the web page elements you need. And he recommends <a href="http://flukeout.github.io/">this page</a> for learning more about selectors.</p><p>For example by using SelectorGadget you can the table you’re interested in. In this case it says it’s an <strong>iframe</strong>. To isolate just the Montréal standings we’ll click another box to only select the one of interest: iframe:nth-child(68).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RDfMiT9bM80mW7xMeD-jPQ.jpeg" /></figure><p>In the context of a web browser, a frame is a part of a web page or browser window which displays content independent of its container, with the ability to load content independently. In this case the website is pulling in data from another source to display these tables interactively apart from the main standings page. Luckily for me they are all from Google Sheets so this will make it <em>much</em> easier for me. Unfortunately, you can’t find the links for these sheets with `selectorgadget`. You will need to take a closer look at the source code using the Developer’s Tool called `Inspector` in either Chrome of Firefox. If you have Windows and Firefox you would click <strong>Open Menu</strong> then Web Developer &gt; Inspector or just Ctrl+Shift+c. Then I used the search box to look for the link (src=) for Montréal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yEL28XbgOFIF0yTR5SmAMg.jpeg" /></figure><blockquote>For me it was a big pain to actually find what I was looking for as sometimes the highlight looks like what you want but it’s not. For example:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Bu0acQcB_CpRmdDhOKLtA.jpeg" /></figure><p>In the end I guess to figure out HTML it involves gradual “<strong>Denial and Error</strong>” attempts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*CP-fXLA8NMeNbl81qjJ2wQ.jpeg" /></figure><h3>Webscraping Google Sheets with RSelenium</h3><h4>Legal Disclaimer</h4><p>It’s worth mentioning that administrators may want to protect certain parts of their website for a <a href="http://www.robotstxt.org/norobots-rfc.txt">number of reasons</a>, such as “<em>indexing of an unannounced site, traversal of parts of the site which require vast resources of the server, recursive traversal of an infinite URL space, etc.</em>”</p><p>Therefore, one should always check if they have permission. One way to do this, is to use the robotstxt package to check if your web-bot has permission to access certain parts of a web-page.</p><pre># check permissions<br>library(robotstxt)<br>paths_allowed(&quot;<a href="https://docs.google.com/spreadsheets/d/1o1PlLIQS8v-XSuEz1eqZB80kcJk9xg5lsbueB7mTg1U/pub?output=html&amp;widget=true#gid=690408156">https://docs.google.com/spreadsheets/d/1o1PlLIQS8v-XSuEz1eqZB80kcJk9xg5lsbueB7mTg1U/pub?output=html&amp;widget=true#gid=690408156</a>&quot;)</pre><p>If it says <strong>TRUE</strong> on the specific page you have permission. Alternatively, just go to the robots.txt file on the url of the main page to get a broader sense of what is (and isn’t) allowed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*Y5y6dhHtDd_6U0m9L8KNNg.jpeg" /></figure><h4>Method # 1</h4><p>Sometimes websites can be composed using frames. These are in effect seperate webpages which are brought together in a frameset. We will need to jump back-and-forth between these frames.</p><pre>library(RSelenium)<br>library(XML)<br>library(janitor)<br>library(lubridate)<br>library(magrittr)<br>library(dplyr)</pre><pre>remDr &lt;- remoteDriver(<br>  remoteServerAddr = &quot;192.168.99.100&quot;,<br>  port = 4445L<br>)<br>remDr$open()</pre><pre># Now open TightVNC to follow along with Selenium driving the browser. Set 192.168.99.100:5901 and password: secret</pre><pre># navigate to the main page<br>remDr$navigate(&quot;<a href="https://docs.google.com/spreadsheets/d/1o1PlLIQS8v-XSuEz1eqZB80kcJk9xg5lsbueB7mTg1U/pub?output=html&amp;widget=true#gid=690408156">https://docs.google.com/spreadsheets/d/1o1PlLIQS8v-XSuEz1eqZB80kcJk9xg5lsbueB7mTg1U/pub?output=html&amp;widget=true#gid=690408156</a>&quot;)</pre><pre># <a href="https://docs.google.com/spreadsheets/d/1o1PlLIQS8v-XSuEz1eqZB80kcJk9xg5lsbueB7mTg1U/pub?output=html&amp;widget=true#gid=552987877">https://docs.google.com/spreadsheets/d/1o1PlLIQS8v-XSuEz1eqZB80kcJk9xg5lsbueB7mTg1U/pub?output=html&amp;widget=true#gid=552987877</a></pre><pre># look for table element<br>tableElem &lt;- remDr$findElement(using = &quot;id&quot;, &quot;pageswitcher-content&quot;)</pre><pre># switch to table<br>remDr$switchToFrame(tableElem)<br># parse html<br>doc &lt;- htmlParse(remDr$getPageSource()[[1]])<br>table_tmp &lt;- readHTMLTable(doc)<br>table_tmp &lt;- table_tmp[[1]][-2,-1]<br>table_tmp &lt;- table_tmp[-1,]<br>colnames(table_tmp) &lt;- c(&quot;team_name&quot;, &quot;team_size&quot;, &quot;start_time&quot;, &quot;end_time&quot;, &quot;total_time&quot;, &quot;puzzels_solved&quot;)<br>table_tmp$city &lt;- rep(&quot;montreal&quot;, nrow(table_tmp))<br>table_tmp$date &lt;- rep(Sys.Date()-5, nrow(table_tmp))</pre><p>Now that we have the first month we can create a for loop for the rest of the dates. First let’s switch back the outer frame and select the elements we will be manipulating.</p><pre># switch back to the main/outter frame<br>remDr$switchToFrame(NULL)</pre><pre># find the elements you&#39;ll manipulate with Inspector mode in a browser<br>webElems &lt;- remDr$findElements(using = &quot;css&quot;, &quot;.switcherItem&quot;) # Month/Year tabs at the bottom<br>arrowElems &lt;- remDr$findElements(using = &quot;css&quot;, &quot;.switcherArrows&quot;) # Arrows to scroll left and right at the bottom<br>tableElem &lt;- remDr$findElement(using = &quot;id&quot;, &quot;pageswitcher-content&quot;) # The inner table frame</pre><p>So I know there’s going to be many tables, but just how many? We can check this use this via length(webElems).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yQhjpdvfaEJ95OvaNY-C8A.jpeg" /></figure><p>There is actually 49 tables in total but since we started on the first one above there is only 48 links. Rather than hard-coding 1:48 it’s better to do it via code as there will be more tables added in the future.</p><pre># Create NULL object to be used in forloop<br>big_df &lt;- NULL<br>for (i in seq(length(webElems))){ # for every <br>check &lt;- try(expression, silent = TRUE) # or suppressMessages(try(expression, silent = TRUE))<br>if (any(class(check) == &quot;try-error&quot;)) {<br>        # choose the i&#39;th Month/Year tab <br>        webElem &lt;- webElems[[i]]<br>        webElem$clickElement()</pre><pre># Find the tableElem again other wise you get a StaleElementReference <br>## TO DO: look into WebDriverWait: <a href="https://stackoverflow.com/questions/5709204/random-element-is-no-longer-attached-to-the-dom-staleelementreferenceexception">https://stackoverflow.com/questions/5709204/random-element-is-no-longer-attached-to-the-dom-staleelementreferenceexception</a> <br>tableElem &lt;- remDr$findElement(using = &quot;id&quot;, &quot;pageswitcher-content&quot;) # The inner table frame</pre><pre># switch to table frame<br>remDr$switchToFrame(tableElem)<br>Sys.sleep(3)<br># parse html with XML package<br>doc &lt;- htmlParse(remDr$getPageSource()[[1]])<br>Sys.sleep(3)<br># Extract data from HTML table in HTML doucment<br>table_tmp &lt;- readHTMLTable(doc)<br>Sys.sleep(3)<br># put this into a format you can use<br>table &lt;- table_tmp[[1]][-2,-1]<br>table &lt;- table[-1,]<br># rename the columns<br>colnames(table) &lt;- c(&quot;team_name&quot;, &quot;team_size&quot;, &quot;start_time&quot;, &quot;end_time&quot;, &quot;total_time&quot;, &quot;puzzels_solved&quot;)<br># add city name to a column<br>table$city &lt;- rep(&quot;Montreal&quot;, nrow(table))</pre><pre># add the Month/Year this table was extracted from<br>today &lt;- Sys.Date() %m-% months(i + 1)<br>table$date &lt;- today</pre><pre># concatenate each table together<br>big_df &lt;- dplyr::bind_rows(big_df, table)</pre><pre># Switch back to the main frame<br>remDr$switchToFrame(NULL)</pre><pre>arrowElem &lt;- arrowElems[[1]]<br># once you &quot;click&quot;&quot; the element it is &quot;held down&quot; as far as I know there is no way to &quot; unclick&quot;<br># to prevent it from scrolling too far I make sure not to take too long by setting the sleep short<br>arrowElem$clickElement()<br># give it &quot;just enough time&quot; to scroll right<br>Sys.sleep(0.3)<br># switch back to outer frame to re-start the loop<br>remDr$switchToFrame(NULL)<br>        }<br>}</pre><pre>temp1 &lt;- dplyr::bind_rows(table_tmp, big_df)</pre><p>The problem here is that the for loop eventually fails at the end when it tries to click the right arrow but it’s as far to the right as it can go — therefore it won’t download the last few tables (~5). Typically one would handle such conditions with something like:</p><pre>check &lt;- try(expression, silent = TRUE) # or suppressMessages(try(expression, silent = TRUE))<br>if (any(class(check) == &quot;try-error&quot;)) {<br>  # do stuff<br>}</pre><p>and it usually works fine, including when using selenium. The issue encountered here however is clicking on the arrow once would always bring me to the <em>last</em> visible sheets — skipping everything in middle. Therefore my work around to get the remaining sheets was this:</p><pre># ctrl+x from the google sheet then use the read.delim() to assign it to an object<br>march &lt;- read.delim(&quot;clipboard&quot;)<br>february &lt;- read.delim(&quot;clipboard&quot;)<br>january &lt;- read.delim(&quot;clipboard&quot;)<br>december &lt;- read.delim(&quot;clipboard&quot;)<br>november &lt;- read.delim(&quot;clipboard&quot;)</pre><pre># add the city and date<br>january$city &lt;- rep(&quot;montreal&quot;, nrow(january))<br>january$date &lt;- rep(&quot;2015-01-30&quot;, nrow(january))</pre><pre>february$city &lt;- rep(&quot;montreal&quot;, nrow(february))<br>february$date &lt;- rep(&quot;2015-02-15&quot;, nrow(february))</pre><pre>march$city &lt;- rep(&quot;montreal&quot;, nrow(march))<br>march$date &lt;- rep(&quot;2015-03-15&quot;, nrow(march))</pre><pre>december$city &lt;- rep(&quot;montreal&quot;, nrow(december))<br>december$date &lt;- rep(&quot;2014-12-15&quot;, nrow(december))</pre><pre>november$city &lt;- rep(&quot;montreal&quot;, nrow(november))<br>november$date &lt;- rep(&quot;2014-11-15&quot;, nrow(november))</pre><pre># clean up the column names<br>january %&lt;&gt;% janitor::clean_names()<br>february %&lt;&gt;% janitor::clean_names()<br>march %&lt;&gt;% janitor::clean_names()<br>december %&lt;&gt;% janitor::clean_names()<br>november %&lt;&gt;% janitor::clean_names()</pre><pre># concatenate these five dataframes together<br>xyz &lt;- bind_rows(march, february, january, december, november)</pre><pre># convert characters into actual dates<br>xyz$date &lt;-as.Date(xyz$date)</pre><pre># reorder the columns<br>xyz %&lt;&gt;% dplyr::select(team_name, team_size, start_time, end_time, total_time, puzzles, city, date)</pre><pre># rename this column to match temp1<br>xyz &lt;- rename(xyz, puzzels_solved = puzzles)</pre><pre># change to a character<br>xyz$puzzels_solved &lt;- as.character(xyz$puzzels_solved)</pre><pre># add NA for team size<br>xyz$team_size &lt;- rep(NA, nrow(xyz))</pre><pre># concatenate this onto the larger dataframe<br>temp2 &lt;- bind_rows(temp1, xyz)</pre><pre># save the object<br>write_csv(temp2, &quot;puzzeld_pint_raw.csv&quot;)</pre><h4>Method # 2</h4><p>Big thanks to <a href="https://stackoverflow.com/questions/54084659/exception-handling-rselenium-switchtoframe-error-elementnotvisible/54276021#54276021">Nate on SO</a> for pointing out an alternative solution that solves the task of <em>scrapping the tables </em>but<strong> not</strong> the task of exception handling in the above sense.</p><pre>remDr &lt;- RSelenium::remoteDriver(<br>  remoteServerAddr = &quot;192.168.99.100&quot;,<br>  port = 4445L<br>)<br>remDr$open(silent = TRUE)</pre><pre># navigate to the main page<br># needs no be done once before looping, else content is not available<br>remDr$navigate(&quot;https://docs.google.com/spreadsheets/d/1o1PlLIQS8v-XSuEz1eqZB80kcJk9xg5lsbueB7mTg1U/pub?output=html&amp;widget=true#gid=690408156&quot;)</pre><pre># I. Preliminaries:<br># <br># 1. build the links to all spreadsheets<br># 2. define the function create_table<br># <br># 1.<br># get page source<br>html &lt;- remDr$getPageSource()[[1]]<br># split it line by line<br>html &lt;- unlist(strsplit(html, &#39;\n&#39;))<br># restrict to script section<br>script &lt;- grep(&#39;^\\s*var\\s+gidMatch&#39;, html, value = TRUE)<br># split the script by semi-colon<br>script &lt;- unlist(strsplit(script, &#39;;&#39;))<br># retrieve information<br>sheet_months &lt;- gsub(&#39;.*name:.{2}(.*?).{1},.*&#39;, &#39;\\1&#39;, <br>                     grep(&#39;\\{name\\s*\\:&#39;, script, value = TRUE), perl = TRUE)<br>sheet_gid &lt;- gsub(&#39;.*gid:.{2}(.*?).{1},.*&#39;, &#39;\\1&#39;, <br>                  grep(&#39;\\gid\\s*\\:&#39;, script, value = TRUE), perl = TRUE)<br>sheet_url &lt;- paste0(&#39;https://docs.google.com/spreadsheets/d/1o1PlLIQS8v-XSuEz1eqZB80kcJk9xg5lsbueB7mTg1U/pubhtml/sheet?headers%5Cx3dfalse&amp;gid=&#39;,<br>                    sheet_gid)</pre><pre># 2.<br># table yielding function<br># just for readability in the loop<br>create_table &lt;- function (remDr) {<br>  # parse html with XML package<br>  doc &lt;- XML::htmlParse(remDr$getPageSource()[[1]])<br>  Sys.sleep(3)<br>  # Extract data from HTML table in HTML document<br>  table_tmp &lt;- XML::readHTMLTable(doc)<br>  Sys.sleep(3)<br>  # put this into a format you can use<br>  table &lt;- table_tmp[[1]][-2, -1]<br>  # add a check-up for size mismatch<br>  table_fields &lt;- as.character(t(table[1,]))<br>  if (! any(grepl(&quot;size&quot;, tolower(table_fields)))) {<br>    table &lt;- table[-1, ]<br>    # rename the columns<br>    colnames(table) &lt;- c(&quot;team_name&quot;, &quot;start_time&quot;, &quot;end_time&quot;, &quot;total_time&quot;, &quot;puzzels_solved&quot;)<br>    table$team_size &lt;- NA_integer_<br>    table &lt;- table[,c(&quot;team_name&quot;, &quot;team_size&quot;, &quot;start_time&quot;, &quot;end_time&quot;, &quot;total_time&quot;, &quot;puzzels_solved&quot;)]<br>  } else {<br>    table &lt;- table[-1, ]<br>    # rename the columns<br>    colnames(table) &lt;- c(&quot;team_name&quot;, &quot;team_size&quot;, &quot;start_time&quot;, &quot;end_time&quot;, &quot;total_time&quot;, &quot;puzzels_solved&quot;)<br>  }<br>  # add city name to a column<br>  table$city &lt;- rep(&quot;Montreal&quot;, nrow(table))<br><br>  # add the Month/Year this table was extracted from<br>  today &lt;- Sys.Date()<br>  lubridate::month(today) &lt;- lubridate::month(today)+1<br>  table$date &lt;- today<br><br>  # returns the table<br>  table<br>}<br><br># II. Scrapping the content<br># <br># 1. selenium to generate the pages<br># 2. use create_table to extract the table<br># <br>big_df &lt;- NULL<br>for (k in seq_along(sheet_url)) {<br>  # 1. navigate to the page<br>  remDr$navigate(sheet_url[k])<br>  # remDr$screenshot(display = TRUE) maybe one wants to see progress<br>  table &lt;- create_table(remDr)<br><br>  # 2. concatenate each table together<br>  big_df &lt;- dplyr::bind_rows(big_df, table)<br><br>  # inform progress <br>  cat(paste0(&#39;\nGathered table for: \t&#39;, sheet_months[k]))<br>}<br><br># close session<br>remDr$close()</pre><p>To perform the task, what was done was firstly generating the links to all spreadsheets in the document. To do this:</p><ul><li>Navigate once to the document</li><li>Extract source code</li><li>Extract the sheet months and URLs (via gid digit) using regex</li><li>Once this is done, loop through the Urls, gather and bind the tables</li></ul><p>There is a small function called create_table which returns the final table in the proper format with a safety measure for the number of columns (some of the spreadsheets do not have the team_size field - in those cases I set it to NA_integer).</p><p>If you find this article useful feel free to share it with others or recommend this article! 😃</p><p>As always, if you have any questions or comments feel free to leave your feedback below or you can always reach me on <a href="https://www.linkedin.com/in/matthewoldach/">LinkedIn</a>. Till then, see you in the next post! 😄</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9001eda399b0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/web-scraping-google-sheets-with-rselenium-9001eda399b0">Web Scraping Google Sheets with RSelenium</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Tips for Choosing the Optimal Number of Clusters]]></title>
            <link>https://medium.com/data-science/10-tips-for-choosing-the-optimal-number-of-clusters-277e93d72d92?source=rss-459e62b88a2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/277e93d72d92</guid>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Matt.0]]></dc:creator>
            <pubDate>Sun, 27 Jan 2019 16:59:14 GMT</pubDate>
            <atom:updated>2019-01-28T02:07:40.570Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nh_06G9VBvDKHbFHpyFayA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/EJMTKCZ00I0?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Pakata Goh</a> on <a href="https://unsplash.com/search/photos/programming?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Clustering is one of the most common unsupervised machine learning problems. Similarity between observations is defined using some inter-observation distance measures or correlation-based distance measures.</p><p>There are 5 classes of clustering methods:</p><blockquote>+ Hierarchical Clustering<br>+ Partitioning Methods (k-means, PAM, CLARA)<br>+ Density-Based Clustering<br>+ Model-based Clustering<br>+ Fuzzy Clustering</blockquote><p>My desire to write this post came mainly from reading about the <a href="https://github.com/lazappi/clustree"><strong>clustree </strong>package</a>, the <a href="https://cran.r-project.org/web/packages/dendextend/vignettes/Cluster_Analysis.html"><strong>dendextend</strong></a> documentation, and the <a href="https://www.datanovia.com/en/product/practical-guide-to-cluster-analysis-in-r/">Practical Guide to Cluster Analysis in R</a> book written by Alboukadel Kassambara author of the <a href="http://www.sthda.com/english/wiki/r-packages"><strong>factoextra</strong></a><strong> </strong>package.</p><h3>Data Set</h3><p>I will be using a lesser known data set from the <strong>cluster</strong> package: <a href="https://www.rdocumentation.org/packages/cluster.datasets/versions/1.0-1/topics/all.mammals.milk.1956">all.mammals.milk.1956</a>, one which I haven’t looked at before.</p><p>This small dataset contains a list of 25 mammals and the constituents of their milk (water, protein, fat, lactose, ash percentages) from <a href="http://people.sc.fsu.edu/~jburkardt/datasets/hartigan/hartigan.html">John Hartigan, Clustering Algorithms, Wiley, 1975</a>.</p><p>First let’s load the required packages.</p><pre>library(tidyverse)<br>library(magrittr)<br>library(cluster)<br>library(cluster.datasets)<br>library(cowplot)<br>library(NbClust)<br>library(clValid)<br>library(ggfortify)<br>library(clustree)<br>library(dendextend)<br>library(factoextra)<br>library(FactoMineR)<br>library(corrplot)<br>library(GGally)<br>library(ggiraphExtra)<br>library(knitr)<br>library(kableExtra)</pre><p>Now load the data.</p><pre>data(&quot;all.mammals.milk.1956&quot;)<br>raw_mammals &lt;- all.mammals.milk.1956</pre><pre># subset dataset<br>mammals &lt;- raw_mammals %&gt;% select(-name) # set rownames<br>mammals &lt;- as_tibble(mammals)</pre><p>Let’s explore and visualize the data.</p><pre># Glimpse the data set<br>glimpse(mammals)</pre><pre>Observations: 25<br>Variables: 5<br>$ water   <em>&lt;dbl&gt;</em> 90.1, 88.5, 88.4, 90.3, 90.4, 87.7, 86.9, 82.1, 81.9, 81.6, 81.6, 86.5, 90.0,...<br>$ protein <em>&lt;dbl&gt;</em> 2.6, 1.4, 2.2, 1.7, 0.6, 3.5, 4.8, 5.9, 7.4, 10.1, 6.6, 3.9, 2.0, 7.1, 3.0, 5...<br>$ fat     <em>&lt;dbl&gt;</em> 1.0, 3.5, 2.7, 1.4, 4.5, 3.4, 1.7, 7.9, 7.2, 6.3, 5.9, 3.2, 1.8, 5.1, 4.8, 6....<br>$ lactose <em>&lt;dbl&gt;</em> 6.9, 6.0, 6.4, 6.2, 4.4, 4.8, 5.7, 4.7, 2.7, 4.4, 4.9, 5.6, 5.5, 3.7, 5.3, 4....<br>$ ash     <em>&lt;dbl&gt;</em> 0.35, 0.24, 0.18, 0.40, 0.10, 0.71, 0.90, 0.78, 0.85, 0.75, 0.93, 0.80, 0.47,...</pre><p>All the variables are expressed as numeric. What about the statistical distribution?</p><pre># Summary of data set<br>summary(mammals) %&gt;% kable() %&gt;% kable_styling()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*_LTOQcmFOfw6gAhAw6QmpA.png" /></figure><pre># Historgram for each attribute<br>mammals %&gt;% <br>  gather(Attributes, value, 1:5) %&gt;% <br>  ggplot(aes(x=value)) +<br>  geom_histogram(fill = &quot;lightblue2&quot;, color = &quot;black&quot;) + <br>  facet_wrap(~Attributes, scales = &quot;free_x&quot;) +<br>  labs(x = &quot;Value&quot;, y = &quot;Frequency&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i76oVZvwpPxZCYdovmanIw.png" /></figure><p>What’s the relationship between the different attributes? Use `corrplot()` to create correlation matrix.</p><pre>corrplot(cor(mammals), type = &quot;upper&quot;, method = &quot;ellipse&quot;, tl.cex = 0.9)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*P5xSIYPsSttLh5Rb5uXwUQ.png" /></figure><p>When you have variables which are measured in different scales it is useful to scale the data.</p><pre>mammals_scaled &lt;- scale(mammals)<br>rownames(mammals_scaled) &lt;- raw_mammals$name</pre><p>Dimensionality reduction can help with data visualization (<em>e.g. </em>PCA method).</p><pre>res.pca &lt;- PCA(mammals_scaled,  graph = FALSE)</pre><pre># Visualize eigenvalues/variances<br>fviz_screeplot(res.pca, addlabels = TRUE, ylim = c(0, 50))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*kCLvb6sYp-QyfCifw7Kyow.png" /></figure><p>These are the <strong>5 PCs that capture 80% of the variance</strong>. The scree plot shows that <strong>PC1 captured ~ 75% of the variance</strong>.</p><pre># Extract the results for variables<br>var &lt;- get_pca_var(res.pca)</pre><pre># Contributions of variables to PC1<br>fviz_contrib(res.pca, choice = &quot;var&quot;, axes = 1, top = 10)</pre><pre># Contributions of variables to PC2<br>fviz_contrib(res.pca, choice = &quot;var&quot;, axes = 2, top = 10)</pre><pre># Control variable colors using their contributions to the principle axis<br>fviz_pca_var(res.pca, col.var=&quot;contrib&quot;,<br>             gradient.cols = c(&quot;#00AFBB&quot;, &quot;#E7B800&quot;, &quot;#FC4E07&quot;),<br>             repel = TRUE # Avoid text overlapping<br>             ) + theme_minimal() + ggtitle(&quot;Variables - PCA&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*6JIZBX699nUPNq-0I9A0Bg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*O9lhD0BkpKtAv0wZlV6ZzA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*wdGFx8hVFmij0gGHckY_Dw.png" /></figure><p>From these visualizations it’s apparrent that water and lactose tend to increase together and that protein, ash and fat increase together; the two groups being inversely related.</p><h3>Naïve (K-means) Approach</h3><p>Partitioning clustering methods, like k-means and Partitioning Around Medoids (PAM), require that you specify the number of clusters to be generated.</p><p>k-means clusters is probably one of the most well known partitioning methods. The idea behind k-means clustering consists of defining clusters the <strong>total within-cluster variation</strong> , which measures the compactness of the clusters is minimized.</p><p>We can compute k-means in R with the <strong>kmeans()</strong> function:</p><pre>km2 &lt;- kmeans(mammals_scaled, centers = 2, nstart = 30)</pre><p>The above example would group the data into two clusters, <strong>centers = 2</strong>, and attempt multiple initial configurations, reporting on the best one. For example, as this algorithm is sensitive to the initial positions of the cluster centroids adding <strong>nstart = 30</strong> will generate 30 initial configurations and then average all the centroid results.</p><p>Because the number of clusters (<strong>k</strong>) needs to be set before we start it can be advantageous to examine several different values of <strong>k</strong>.</p><pre>kmean_calc &lt;- function(df, ...){<br>  kmeans(df, scaled = ..., nstart = 30)<br>}</pre><pre>km2 &lt;- kmean_calc(mammals_scaled, 2)<br>km3 &lt;- kmean_calc(mammals_scaled, 3)<br>km4 &lt;- kmeans(mammals_scaled, 4)<br>km5 &lt;- kmeans(mammals_scaled, 5)<br>km6 &lt;- kmeans(mammals_scaled, 6)<br>km7 &lt;- kmeans(mammals_scaled, 7)<br>km8 &lt;- kmeans(mammals_scaled, 8)<br>km9 &lt;- kmeans(mammals_scaled, 9)<br>km10 &lt;- kmeans(mammals_scaled, 10)<br>km11 &lt;- kmeans(mammals_scaled, 11)</pre><pre>p1 &lt;- fviz_cluster(km2, data = mammals_scaled, frame.type = &quot;convex&quot;) + theme_minimal() + ggtitle(&quot;k = 2&quot;) <br>p2 &lt;- fviz_cluster(km3, data = mammals_scaled, frame.type = &quot;convex&quot;) + theme_minimal() + ggtitle(&quot;k = 3&quot;)<br>p3 &lt;- fviz_cluster(km4, data = mammals_scaled, frame.type = &quot;convex&quot;) + theme_minimal() + ggtitle(&quot;k = 4&quot;)<br>p4 &lt;- fviz_cluster(km5, data = mammals_scaled, frame.type = &quot;convex&quot;) + theme_minimal() + ggtitle(&quot;k = 5&quot;)<br>p5 &lt;- fviz_cluster(km6, data = mammals_scaled, frame.type = &quot;convex&quot;) + theme_minimal() + ggtitle(&quot;k = 6&quot;)<br>p6 &lt;- fviz_cluster(km7, data = mammals_scaled, frame.type = &quot;convex&quot;) + theme_minimal() + ggtitle(&quot;k = 7&quot;)</pre><pre>plot_grid(p1, p2, p3, p4, p5, p6, labels = c(&quot;k2&quot;, &quot;k3&quot;, &quot;k4&quot;, &quot;k5&quot;, &quot;k6&quot;, &quot;k7&quot;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Lfl-pqYn44gypN1qlQMmg.png" /></figure><p>Although this visual assessment tells us where delineations occur between clusters, it does not tell us what the optimal number of clusters is.</p><h3>Determining Optimal Number of Clusters</h3><p>A variety of measures have been proposed in the literature for evaluating clustering results. The term <strong>clustering validation</strong> is used to design the procedure of evaluating the results of a clustering algorithm. There are more than thirty indices and methods for identifying the optimal number of clusters so I’ll just focus on a few here including the very neat <strong>clustree</strong> package.</p><h4>The “Elbow” Method</h4><p>Probably the most well known method, the elbow method, in which the sum of squares at each number of clusters is calculated and graphed, and the user looks for a change of slope from steep to shallow (an elbow) to determine the optimal number of clusters. This method is inexact, but still potentially helpful.</p><pre>set.seed(31)<br># function to compute total within-cluster sum of squares<br>fviz_nbclust(mammals_scaled, kmeans, method = &quot;wss&quot;, k.max = 24) + theme_minimal() + ggtitle(&quot;the Elbow Method&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zQUkkjBgQBEWigJVNJc9DA.png" /></figure><p>The Elbow Curve method is helpful because it shows how increasing the number of the clusters contribute separating the clusters in a meaningful way, not in a marginal way. The bend indicates that additional clusters beyond the third have little value (See [<a href="http://web.stanford.edu/~hastie/Papers/gap.pdf">here</a>] for a more mathematically rigorous interpretation and implementation of this method). The Elbow method is fairly clear, if not a naïve solution based on intra-cluster variance. The gap statistic is more sophisticated method to deal with data that has a distribution with no obvious clustering (can find the correct number of <em>k </em>for globular, Gaussian-distributed, mildly disjoint data distributions).</p><h4>The Gap Statistic</h4><p>The <a href="http://www.web.stanford.edu/~hastie/Papers/gap.pdf">gap statistic</a> compares the total within intra-cluster variation for different values of <strong>k</strong> with their expected values under null reference distribution of the data. The estimate of the optimal clusters will be value that maximize the gap statistic (<em>i.e.</em>, that yields the largest gap statistic). This means that the clustering structure is far away from the random uniform distribution of points.</p><pre>gap_stat &lt;- clusGap(mammals_scaled, FUN = kmeans, nstart = 30, K.max = 24, B = 50)</pre><pre>fviz_gap_stat(gap_stat) + theme_minimal() + ggtitle(&quot;fviz_gap_stat: Gap Statistic&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1KMRhZEIy0je24ksyiByxQ.png" /></figure><p>The gap stats plot shows the statistics by number of clusters (<strong>k</strong>) with standard errors drawn with vertical segments and the optimal value of <strong>k</strong> marked with a vertical dashed blue line. According to this observation <strong>k = 2</strong> is the optimal number of clusters in the data.</p><h4>The Silhouette Method</h4><p>Another visualization that can help determine the optimal number of clusters is called the a silhouette method. Average silhouette method computes the average silhouette of observations for different values of k. The optimal number of clusters k is the one that maximize the average silhouette over a range of possible values for <strong>k</strong>.</p><pre>fviz_nbclust(mammals_scaled, kmeans, method = &quot;silhouette&quot;, k.max = 24) + theme_minimal() + ggtitle(&quot;The Silhouette Plot&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gtRd99n__IAB3Z1J-hOcpw.png" /></figure><p>This also suggests an optimal of 2 clusters.</p><h4>The Sum of Squares Method</h4><p>Another clustering validation method would be to choose the optimal number of cluster by minimizing the within-cluster sum of squares (a measure of how tight each cluster is) and maximizing the between-cluster sum of squares (a measure of how seperated each cluster is from the others).</p><pre>ssc &lt;- data.frame(<br>  kmeans = c(2,3,4,5,6,7,8),<br>  within_ss = c(mean(km2$withinss), mean(km3$withinss), mean(km4$withinss), mean(km5$withinss), mean(km6$withinss), mean(km7$withinss), mean(km8$withinss)),<br>  between_ss = c(km2$betweenss, km3$betweenss, km4$betweenss, km5$betweenss, km6$betweenss, km7$betweenss, km8$betweenss)<br>)</pre><pre>ssc %&lt;&gt;% gather(., key = &quot;measurement&quot;, value = value, -kmeans)</pre><pre>#ssc$value &lt;- log10(ssc$value)</pre><pre>ssc %&gt;% ggplot(., aes(x=kmeans, y=log10(value), fill = measurement)) + geom_bar(stat = &quot;identity&quot;, position = &quot;dodge&quot;) + ggtitle(&quot;Cluster Model Comparison&quot;) + xlab(&quot;Number of Clusters&quot;) + ylab(&quot;Log10 Total Sum of Squares&quot;) + scale_x_discrete(name = &quot;Number of Clusters&quot;, limits = c(&quot;0&quot;, &quot;2&quot;, &quot;3&quot;, &quot;4&quot;, &quot;5&quot;, &quot;6&quot;, &quot;7&quot;, &quot;8&quot;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9vEjv6Xir4HX58pBjNMfkQ.png" /></figure><p>From this measurement it appears 7 clusters would be the appropriate choice.</p><h4>NbClust</h4><p>The <strong>NbClust</strong> package provides 30 indices for determining the relevant number of clusters and proposes to users the best clustering scheme from the different results obtained by varying all combinations of number of clusters, distance measures, and clustering methods.</p><pre>res.nbclust &lt;- NbClust(mammals_scaled, distance = &quot;euclidean&quot;,<br>                  min.nc = 2, max.nc = 9, <br>                  method = &quot;complete&quot;, index =&quot;all&quot;)</pre><pre>factoextra::fviz_nbclust(res.nbclust) + theme_minimal() + ggtitle(&quot;NbClust&#39;s optimal number of clusters&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QzzyNowIPofo4Ak-z42n-w.png" /></figure><p>This suggest the optimal number of clusters is 3.</p><h4>Clustree</h4><p>The statistical method above produce a single score that only considers a single set of clusters at a time. The <a href="https://github.com/lazappi/clustree"><strong>clustree</strong></a> R package takes an alternative approach by considering how samples change groupings as the number of clusters increases. This is useful for showing which clusters are distinct and which are unstable. It doesn’t explicitly tell you which choice of <em>optimal</em> clusters is but it is useful for exploring possible choices.</p><p>Let’s take a look at 1 to 11 clusters.</p><pre>tmp &lt;- NULL<br>for (k in 1:11){<br>  tmp[k] &lt;- kmeans(mammals_scaled, k, nstart = 30)<br>}</pre><pre>df &lt;- data.frame(tmp)</pre><pre># add a prefix to the column names<br>colnames(df) &lt;- seq(1:11)<br>colnames(df) &lt;- paste0(&quot;k&quot;,colnames(df))</pre><pre># get individual PCA<br>df.pca &lt;- prcomp(df, center = TRUE, scale. = FALSE)</pre><pre>ind.coord &lt;- df.pca$x<br>ind.coord &lt;- ind.coord[,1:2]</pre><pre>df &lt;- bind_cols(as.data.frame(df), as.data.frame(ind.coord))</pre><pre>clustree(df, prefix = &quot;k&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T43RY6iKGjqtKSFvr8YsLg.png" /></figure><p>In this figure the size of each node corresponds to the number of samples in each cluster, and the arrows are coloured according to the number of samples each cluster receives. A separate set of arrows, the transparent ones, called the incoming node proportion, are also coloured and shows how samples from one group end up in another group — an indicator of cluster instability.</p><p>In this graph we see that as we move from <strong>k=2</strong> to <strong>k=3</strong> a number of species from the lookers-left cluster are reasigned to the third cluster on the right. As we move from <strong>k=8</strong> to <strong>k=9</strong> we see one node with multiple incoming edges an indicator that we over-clustered the data.</p><p>It can also be useful to overlay this dimension on other dimensions in the data, particularly those that come from dimensionality reduction techniques. We can do this using the <strong>clustree_overlay()</strong> function:</p><pre>df_subset &lt;- df %&gt;% select(1:8,12:13)</pre><pre>clustree_overlay(df_subset, prefix = &quot;k&quot;, x_value = &quot;PC1&quot;, y_value = &quot;PC2&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hiVccajlxLATh9DJjacMhw.png" /></figure><p>I prefer to see it from the side, showing one of the x or y dimensions against the resolution dimension.</p><pre>overlay_list &lt;- clustree_overlay(df_subset, prefix = &quot;k&quot;, x_value = &quot;PC1&quot;,<br>                                 y_value = &quot;PC2&quot;, plot_sides = TRUE)</pre><pre>overlay_list$x_side</pre><pre>overlay_list$y_side</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mS_Ix9xschUWuAwjLHYHRg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cLXIl9t0312kiSjvopoE1g.png" /></figure><p>This shows that we can an indication of the correct clustering resolution by examining the edges and we can overly information to assess the quality of the clustering.</p><h3>Choosing the appropriate algorithm</h3><p>What about choice of appropriate clustering algorithm? The <strong>cValid</strong> package can be used to simultaneously compare multiple clustering algorithms, to identify the best clustering approach and the optimal number of clusters. We will compare k-means, hierarchical and PAM clustering.</p><pre>intern &lt;- clValid(mammals_scaled, nClust = 2:24, <br>              clMethods = c(&quot;hierarchical&quot;,&quot;kmeans&quot;,&quot;pam&quot;), validation = &quot;internal&quot;)</pre><pre># Summary<br>summary(intern) %&gt;% kable() %&gt;% kable_styling()</pre><pre>Clustering Methods:<br> hierarchical kmeans pam <br><br>Cluster sizes:<br> 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 <br><br>Validation Measures:<br>                                 2       3       4       5       6       7       8       9      10      11      12      13      14      15      16      17      18      19      20      21      22      23      24<br>                                                                                                                                                                                                                  <br>hierarchical Connectivity   4.1829 10.5746 13.2579 20.1579 22.8508 25.8258 32.6270 35.3032 38.2905 39.2405 41.2405 45.7742 47.2742 50.6075 52.6075 55.8575 58.7242 60.7242 63.2242 65.2242 67.2242 69.2242 71.2242<br>             Dunn           0.3595  0.3086  0.3282  0.2978  0.3430  0.3430  0.4390  0.4390  0.5804  0.5938  0.5938  0.8497  0.8497  0.5848  0.5848  0.4926  0.9138  0.9138  0.8892  0.9049  0.9335  1.0558  2.1253<br>             Silhouette     0.5098  0.5091  0.4592  0.4077  0.4077  0.3664  0.3484  0.4060  0.3801  0.3749  0.3322  0.3646  0.3418  0.2650  0.2317  0.2166  0.2469  0.2213  0.1659  0.1207  0.1050  0.0832  0.0691<br>kmeans       Connectivity   7.2385 10.5746 15.8159 20.1579 22.8508 25.8258 33.5198 35.3032 38.2905 39.2405 41.2405 45.7742 47.2742 51.8909 53.8909 57.1409 58.7242 60.7242 63.2242 65.2242 67.2242 69.2242 71.2242<br>             Dunn           0.2070  0.3086  0.2884  0.2978  0.3430  0.3430  0.3861  0.4390  0.5804  0.5938  0.5938  0.8497  0.8497  0.5866  0.5866  0.5725  0.9138  0.9138  0.8892  0.9049  0.9335  1.0558  2.1253<br>             Silhouette     0.5122  0.5091  0.4260  0.4077  0.4077  0.3664  0.3676  0.4060  0.3801  0.3749  0.3322  0.3646  0.3418  0.2811  0.2478  0.2402  0.2469  0.2213  0.1659  0.1207  0.1050  0.0832  0.0691<br>pam          Connectivity   7.2385 14.1385 17.4746 24.0024 26.6857 32.0413 33.8913 36.0579 38.6607 40.6607 42.7869 45.7742 47.2742 51.7242 53.7242 56.9742 58.7242 60.7242 62.7242 64.7242 66.7242 69.2242 71.2242<br>             Dunn           0.2070  0.1462  0.2180  0.2180  0.2978  0.2980  0.4390  0.4390  0.4390  0.4390  0.4390  0.8497  0.8497  0.5314  0.5314  0.4782  0.9138  0.9138  0.8333  0.8189  0.7937  1.0558  2.1253<br>             Silhouette     0.5122  0.3716  0.4250  0.3581  0.3587  0.3318  0.3606  0.3592  0.3664  0.3237  0.3665  0.3646  0.3418  0.2830  0.2497  0.2389  0.2469  0.2213  0.1758  0.1598  0.1380  0.0832  0.0691<br><br>Optimal Scores:<br><br>             Score  Method       Clusters<br>Connectivity 4.1829 hierarchical 2       <br>Dunn         2.1253 hierarchical 24      <br>Silhouette   0.5122 kmeans       2</pre><p>Connectivity and Silhouette are both measurements of connectedness while the Dunn Index is the ratio of the smallest distance between observations not in the same cluster to the largest intra-cluster distance.</p><h3>Extracting Features of Clusters</h3><p>As mentioned earlier it’s difficult to assess the quality of results from clustering. We don’t have true labels so so it’s unclear how one would measure “<em>how good it actually works</em>” in term of interal validation. However, clustering is an excellent EDA starting point for exploring the differences between clusters in greater detail. Think of clustering like manufacturing shirt sizes. We could choose to only make three sizes: <em>small</em>, <em>medium</em> and <em>large. </em>we’re sure to cut down cost but not everyone is going to have a great fit. Think about pant sizes now (or shirt brands with many sizes (XS, XL, XXL, <em>etc.</em>)) where you have many more categories (or clusters). For some fields the choice of <em>optimal</em> cluster may depend on some external knowledge like cost of producing <em>k </em>clusters to satisfy customers with the best possible fit. In other domains like biology where you are trying to determine the exact number of cells a more in-depth approach would be required. For example, many of the above heuristics contradicted each other for what the <em>optimal</em> number of clusters was. Keep in mind these were all evaluating the <em>k-means</em> algorithm at different numbers of <em>k. </em>This could potentially mean that the <em>k-means</em> algorithm fails and no <em>k </em>is good. The <em>k-means</em> algorithm is not a very robust algorithm that is sensitive to outliers and this data set is quit small. The best thing to do would be to explore the above methods on the output of other algorithms (for example hierarchical clustering which <strong>clValid </strong>suggested), collect more data, or spend some time labelling samples for other ML methods if possible.</p><p>Ultimately, we would like to answer questions like “what is it that makes this cluster unique from others?” and “what are the clusters that are similar to one another”. Let’s select five clusters and interrogate the features of these clusters.</p><pre># Compute dissimilarity matrix with euclidean distances<br>d &lt;- dist(mammals_scaled, method = &quot;euclidean&quot;)</pre><pre># Hierarchical clustering using Ward&#39;s method<br>res.hc &lt;- hclust(d, method = &quot;ward.D2&quot; )</pre><pre># Cut tree into 5 groups<br>grp &lt;- cutree(res.hc, k = 5)</pre><pre># Visualize<br>plot(res.hc, cex = 0.6) # plot tree<br>rect.hclust(res.hc, k = 5, border = 2:5) # add rectangle</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*OZQoXKPBGdheBlektbKU3A.png" /></figure><pre># Execution of k-means with k=5<br>final &lt;- kmeans(mammals_scaled, 5, nstart = 30)</pre><pre>fviz_cluster(final, data = mammals_scaled) + theme_minimal() + ggtitle(&quot;k = 5&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6LJgn9cf8ltcNDGGW206GA.png" /></figure><p>Let’s extract the clusters and add them back to our initial data to do some descriptive statistics at the cluster level:</p><pre>as.data.frame(mammals_scaled) %&gt;% mutate(Cluster = final$cluster) %&gt;% group_by(Cluster) %&gt;% summarise_all(&quot;mean&quot;) %&gt;% kable() %&gt;% kable_styling()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5m_K-dg0Ky8Pc-P7ZFZzaA.png" /></figure><p>We see that cluster 2, composed solely of the Rabbit has a high ash content. Group 3 composed of the seal and dolphin are high in fat, which makes sense given the harsh demands of such a cold climate whil group 4 has a large lactose content.</p><pre>mammals_df &lt;- as.data.frame(mammals_scaled) %&gt;% rownames_to_column()</pre><pre>cluster_pos &lt;- as.data.frame(final$cluster) %&gt;% rownames_to_column()<br>colnames(cluster_pos) &lt;- c(&quot;rowname&quot;, &quot;cluster&quot;)</pre><pre>mammals_final &lt;- inner_join(cluster_pos, mammals_df)</pre><pre>ggRadar(mammals_final[-1], aes(group = cluster), rescale = FALSE, legend.position = &quot;none&quot;, size = 1, interactive = FALSE, use.label = TRUE) + facet_wrap(~cluster) + scale_y_discrete(breaks = NULL) + # don&#39;t show ticks<br>theme(axis.text.x = element_text(size = 10)) + scale_fill_manual(values = rep(&quot;#1c6193&quot;, nrow(mammals_final))) +<br>scale_color_manual(values = rep(&quot;#1c6193&quot;, nrow(mammals_final))) +<br>ggtitle(&quot;Mammals Milk Attributes&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Rvv6zH-BRt2mP7QfAbKXg.png" /></figure><pre>mammals_df &lt;- as.data.frame(mammals_scaled)<br>mammals_df$cluster &lt;- final$cluster<br>mammals_df$cluster &lt;- as.character(mammals_df$cluster)</pre><pre>ggpairs(mammals_df, 1:5, mapping = ggplot2::aes(color = cluster, alpha = 0.5), <br>        diag = list(continuous = wrap(&quot;densityDiag&quot;)), <br>        lower=list(continuous = wrap(&quot;points&quot;, alpha=0.9)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a0IlGgZD9xWFbvUaYj8tFQ.png" /></figure><pre># plot specific graphs from previous matrix with scatterplot</pre><pre>g &lt;- ggplot(mammals_df, aes(x = water, y = lactose, color = cluster)) +<br>        geom_point() +<br>        theme(legend.position = &quot;bottom&quot;)<br>ggExtra::ggMarginal(g, type = &quot;histogram&quot;, bins = 20, color = &quot;grey&quot;, fill = &quot;blue&quot;)</pre><pre>b &lt;- ggplot(mammals_df, aes(x = protein, y = fat, color = cluster)) +<br>        geom_point() +<br>        theme(legend.position = &quot;bottom&quot;)<br>ggExtra::ggMarginal(b, type = &quot;histogram&quot;, bins = 20, color = &quot;grey&quot;, fill = &quot;blue&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*_HjPcJFvCmTX_EEeBa_wKw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*xsRDYljJzvxWm4n4zYARJA.png" /></figure><pre>ggplot(mammals_df, aes(x = cluster, y = protein)) + <br>        geom_boxplot(aes(fill = cluster))</pre><pre>ggplot(mammals_df, aes(x = cluster, y = fat)) + <br>        geom_boxplot(aes(fill = cluster))</pre><pre>ggplot(mammals_df, aes(x = cluster, y = lactose)) + <br>        geom_boxplot(aes(fill = cluster))</pre><pre>ggplot(mammals_df, aes(x = cluster, y = ash)) + <br>        geom_boxplot(aes(fill = cluster))</pre><pre>ggplot(mammals_df, aes(x = cluster, y = water)) + <br>        geom_boxplot(aes(fill = cluster))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*4inw4x2G7yVUZD_pN_vhwA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*aL05uDbYvpMsnj6SzNa_yA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*ZPZ1PdSOhhQmFyn8EalbvQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*tZXqag6Dd-bqP2WSFsli2w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w4FzmICs6Z1HFxn7tzr8QQ.png" /></figure><pre># Parallel coordiante plots allow us to put each feature on seperate column and lines connecting each column</pre><pre>ggparcoord(data = mammals_df, columns = 1:5, groupColumn = 6, alphaLines = 0.4, title = &quot;Parallel Coordinate Plot for the Mammals Milk Data&quot;, scale = &quot;globalminmax&quot;, showPoints = TRUE) + theme(legend.position = &quot;bottom&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7-3LoneZQfZfM8EzLNe6hQ.png" /></figure><p>If you find this article useful feel free to share it with others or recommend this article! 😃</p><p>As always, if you have any questions or comments feel free to leave your feedback below or you can always reach me on <a href="https://www.linkedin.com/in/matthewoldach/">LinkedIn</a>. Till then, see you in the next post! 😄</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=277e93d72d92" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/10-tips-for-choosing-the-optimal-number-of-clusters-277e93d72d92">10 Tips for Choosing the Optimal Number of Clusters</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Gamification Of Fitbit: How an API Provided the Next Level of tRaining]]></title>
            <link>https://medium.com/data-science/the-gamification-of-fitbit-how-an-api-provided-the-next-level-of-training-eaf7b267af00?source=rss-459e62b88a2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/eaf7b267af00</guid>
            <category><![CDATA[gameboy]]></category>
            <category><![CDATA[fitbit]]></category>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Matt.0]]></dc:creator>
            <pubDate>Sat, 10 Nov 2018 17:29:57 GMT</pubDate>
            <atom:updated>2018-12-10T14:42:13.117Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Gamification Of Fitbit</h3><h4>How an API Provided the Next Level of tRaining</h4><blockquote><strong>“Anyone can look for fashion in a boutique or history in a museum. The creative explorer looks for history in a hardware store and fashion in an airport” — Robert Wieder</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*8RLsDiMctQXxopZbPkrbDg.png" /></figure><h3>Project Inspiration</h3><p>You may, or may not, have heard of the term gamification but chances are you’ve experienced it.</p><p>Gamification is the application of game-design elements, and game principles, in non-game contexts. The idea is, if you use elements of games, like linking rules and rewards into a feedback system, you can make (almost) any activity motivating and fun.</p><p>Gamification is the concept behind <strong>eLearning</strong>. In elementary school I remember all the students wanted to play <a href="https://classicreload.com/oregon-trail.html">The Oregon Trail</a> in computer class. I also remember another game where you had to solve math problems before something hit the floor. Okay, maybe it wasn’t the most thrilling introduction to gamification but I remember it nonetheless.</p><p>At some point in my career, I got tired of using nano and decided to I wanted to try to learn Vim.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*AmTt7JNbFu1jFMNsPyQu-A.jpeg" /></figure><p>It was then that I discovered two very enjoyable examples of gamification:</p><ul><li><a href="https://vim-adventures.com/">Vim Adventures</a> is kind of like Zelda for Gameboy where you have to move through the environment and solve riddles - except with Vim commands! You can <a href="https://www.youtube.com/watch?v=zGk_ipjk9uw">watch it being played on Twitch here</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Okxo0zg6qVP04FoAXae8pw.png" /></figure><ul><li><a href="https://www.shortcutfoo.com/">shortcutFoo</a> teaches you shortcuts for Vim, Emacs, Command Line, Regex <em>etc.</em> via <strong>interval training</strong>, which is essentially spaced repetition. This helps you memorize shortcuts more efficiently.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*5ud55u_gAzfpMCsi0kOs8A.png" /></figure><p>Today, I enjoy eLearning-gamification on platforms like <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwiQm7ib677eAhUovlkKHRr-BygQFjAAegQIAhAC&amp;url=https%3A%2F%2Fwww.duolingo.com%2F&amp;usg=AOvVaw0GVHP2wYuntJMUTG1EDY5y">DuoLingo</a>, and <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwiQm7ib677eAhUovlkKHRr-BygQFjAAegQIAhAC&amp;url=https%3A%2F%2Fwww.duolingo.com%2F&amp;usg=AOvVaw0GVHP2wYuntJMUTG1EDY5y">DataCamp</a>.</p><p>I’ve also recently started to participate in a Kaggle competition, “<a href="https://www.kaggle.com/c/pubg-finish-placement-prediction">PUBG Finish Placement Prediction</a>”. Kaggle is a Google owned hangout for data science enthusiasts where they can use machine learning to solve predictive analytics problems for cash and clout. Similar to chess there are so-called <strong>Kaggle Grandmasters</strong>.</p><h3>The Quest</h3><p>Our laboratory studies perinatal influences on the biological embedding of early adversity of mental health outcomes. We combine genetic, epigenetic and epidemiological approaches to identify pregnant women who’s offspring may potentially be at risk for adverse mental health outcomes.</p><blockquote><em>My supervisor approached me with a challenge; how feasible would it be to access biometric data from 200 Fitbits?</em></blockquote><p><strong>So I bought myself a Fitbit Charge2 fitness tracker and hit the gym!</strong></p><p>At some point I think we both realized that this project was going to be a big undertaking. Perhaps R isn’t really intended to do large scale real-time data management from API services. It’s great for static files, or static endpoints, but if you’re working with multiple participants a dedicated solution like <a href="https://www.fitabase.com/">Fitabase</a> may work the best - or so they claim.</p><p>Nonetheless, I wanted to try out a bunch of cool new things in R like making a personal website using blogdown, using gganimate with Rokemon, accessing the fitbit API with httr as well as adding a background image with some custom CSS/HTML. Is there possibly a better way to possibly gamify my leaRning curve - I think not.</p><p><strong>The following was my attempt at e-learning </strong><strong>gamification for </strong><strong>R.</strong></p><p>I used the <a href="https://bookdown.org/yihui/blogdown/">blogdown</a> package to allow me to write blog posts as R Markdown documents, knitting everything to a nice neat static website that I can push online. It was a nice opportunity to learn a bit about pandoc, Hugo, CSS/HTML lurking beneath the server side code. I decided to go with the <a href="https://themes.gohugo.io/academic/">Academic theme</a> for <a href="https://gohugo.io/">Hugo</a>, pull in as much data as I could from the Fitbit API, clean it up, and then perform some exploratory data analysis. In the process, I generated some cool animated sprites and use video game inspired visualizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*Iv2UzWmwJOtzbUQD-cFofg.gif" /></figure><h3>Setting up a Fitbit Developer Account</h3><p>Fitbit uses <a href="https://dev.fitbit.com/build/reference/web-api/oauth2/"><strong>OAuth 2.0</strong></a><strong> Access</strong> Token for making HTTP request to the Fitbit API. You need to set up an account to use the API and include your token in R. Instead of reading the <a href="https://dev.fitbit.com/build/reference/web-api/basics/#app-registration">FITBIT DEV HELP</a> section I would direct the reader to better more-concise instructions <a href="https://obrl-soil.github.io/fitbit-api-r/">here</a>.</p><p>Now that you have an account we’re ready to do stuff in R.</p><p>Set your token up:</p><pre># You Found A Secret Area!<br>token = &quot;yourToken&quot;</pre><h3>Using the fitbitr package</h3><p>I had never made an HTTP request before and although the process is officially documented <a href="https://moldach.github.io/fitbit-project/(https://dev.fitbit.com/build/reference/web-api/oauth2/)">here</a> it can be a <em>tad</em> overwhelming. Therefore, I initially resorted to using an R package built to access the R API, under-the-hood, called fitbitr.</p><p>Unfortunately this would limit me to only accessing some basic user information, heart rate and step count data.</p><h3>Getting Basic User Info</h3><p>The first function in this package sends a GET request to the Get Profile <a href="https://dev.fitbit.com/build/reference/web-api/user/#get-profile">resource URL</a>.</p><pre># Extracting Resources</pre><pre># Get userInfo<br>user_info &lt;- fitbitr::getUserInfo(token)</pre><pre># Hailing a Chocobo!</pre><pre># What is my stride length in meters?<br>strideLengthWalking &lt;- user_info$strideLengthWalking</pre><p>My stride length is 68.5.</p><p><em>Stride length is measured from heel to heel and determines how far you walk with each step. On average, a man’s walking stride length is 2.5 feet, or 30 inches, while a woman’s average stride length is 2.2 feet, or 26.4 inches, according to this </em><a href="https://livehealthy.chron.com/average-walking-stride-length-7494.html"><em>report</em></a><em>.</em></p><pre># Hitting 80 MPH</pre><pre># What is my running stride length<br>strideLengthRunning &lt;- user_info$strideLengthRunning</pre><p>My running stride length is 105.5.</p><p><em>The Fitbit uses your </em><em>sex and </em><em>height by default to gauge your stride length which could potentially be inaccurate.</em></p><pre># Looking for the fourth chaos emerald </pre><pre># What is my average daily steps?<br>averageDailySteps &lt;- user_info$averageDailySteps</pre><p>My average daily steps is 14214.</p><p>Considering that the daily recommended steps is 10,000 I’d say that’s acceptable. That being said, there’s always room for improvement.</p><h3>Accessing heart rate and footsteps with the fitbitr pacakge</h3><p>I’m going to grab a week’s worth of data for a very preliminary EDA.</p><pre># Smashing buttons</pre><pre>days &lt;- c(&quot;Sunday&quot;, &quot;Monday&quot;, &quot;Tuesday&quot;, &quot;Wednesday&quot;, &quot;Thursday&quot;, &quot;Friday&quot;, &quot;Saturday&quot;)</pre><pre>monday_heart &lt;- getTimeSeries(token, type = &quot;heart&quot;, activityDetail = &quot;1min&quot;, date = &quot;2018-08-20&quot;, startTime = &quot;00:00&quot;, endTime = &quot;23:59&quot;)<br>monday_heart %&lt;&gt;% mutate(date = &quot;2018-08-20&quot;)<br>monday_steps &lt;- getTimeSeries(token, type = &quot;steps&quot;, activityDetail = &quot;1min&quot;, date = &quot;2018-08-20&quot;)<br>monday_steps %&lt;&gt;% mutate(date = &quot;2018-08-20&quot;)<br>monday &lt;- monday_heart %&gt;% full_join(monday_steps)<br>monday %&lt;&gt;% mutate(week_date = &quot;Monday&quot;)<br>monday %&lt;&gt;% mutate(day_of_week = &quot;1&quot;)</pre><pre>tuesday_heart &lt;- getTimeSeries(token, type = &quot;heart&quot;, activityDetail = &quot;1min&quot;, date = &quot;2018-08-21&quot;)<br>tuesday_heart %&lt;&gt;% mutate(date = &quot;2018-08-21&quot;)<br>tuesday_steps &lt;- getTimeSeries(token, type = &quot;steps&quot;, activityDetail = &quot;1min&quot;, date = &quot;2018-08-21&quot;)<br>tuesday_steps %&lt;&gt;% mutate(date = &quot;2018-08-21&quot;)<br>tuesday &lt;- tuesday_heart %&gt;% full_join(tuesday_steps)<br>tuesday %&lt;&gt;% mutate(week_date = &quot;Tuesday&quot;)<br>tuesday %&lt;&gt;% mutate(day_of_week = &quot;2&quot;)</pre><pre>wednesday_heart &lt;- getTimeSeries(token, type = &quot;heart&quot;, activityDetail = &quot;1min&quot;, date = &quot;2018-08-22&quot;)<br>wednesday_heart %&lt;&gt;% mutate(date = &quot;2018-08-22&quot;)<br>wednesday_steps &lt;- getTimeSeries(token, type = &quot;steps&quot;, activityDetail = &quot;1min&quot;, date = &quot;2018-08-22&quot;)<br>wednesday_steps %&lt;&gt;% mutate(date = &quot;2018-08-22&quot;)<br>wednesday &lt;- wednesday_heart %&gt;% full_join(wednesday_steps)<br>wednesday %&lt;&gt;% mutate(week_date = &quot;Wednesday&quot;)<br>wednesday %&lt;&gt;% mutate(day_of_week = &quot;3&quot;)</pre><pre>thursday_heart &lt;- getTimeSeries(token, type = &quot;heart&quot;, activityDetail = &quot;1min&quot;, date = &quot;2018-08-23&quot;)<br>thursday_heart %&lt;&gt;% mutate(date = &quot;2018-08-23&quot;)<br>thursday_steps &lt;- getTimeSeries(token, type = &quot;steps&quot;, activityDetail = &quot;1min&quot;, date = &quot;2018-08-23&quot;)<br>thursday_steps %&lt;&gt;% mutate(date = &quot;2018-08-23&quot;)<br>thursday &lt;- thursday_heart %&gt;% full_join(thursday_steps)<br>thursday %&lt;&gt;% mutate(week_date = &quot;Thursday&quot;)<br>thursday %&lt;&gt;% mutate(day_of_week = &quot;4&quot;)</pre><pre>friday_heart &lt;- getTimeSeries(token, type = &quot;heart&quot;, activityDetail = &quot;1min&quot;, date = &quot;2018-08-24&quot;)<br>friday_heart %&lt;&gt;% mutate(date = &quot;2018-08-24&quot;)<br>friday_steps &lt;- getTimeSeries(token, type = &quot;steps&quot;, activityDetail = &quot;1min&quot;, date = &quot;2018-08-24&quot;)<br>friday_steps %&lt;&gt;% mutate(date = &quot;2018-08-24&quot;)<br>friday &lt;- friday_heart %&gt;% full_join(friday_steps)<br>friday %&lt;&gt;% mutate(week_date = &quot;Friday&quot;)<br>friday %&lt;&gt;% mutate(day_of_week = &quot;5&quot;)</pre><pre>saturday_heart &lt;- getTimeSeries(token, type = &quot;heart&quot;, activityDetail = &quot;1min&quot;, date = &quot;2018-08-24&quot;)<br>saturday_heart %&lt;&gt;% mutate(date = &quot;2018-08-24&quot;)<br>saturday_steps &lt;- getTimeSeries(token, type = &quot;steps&quot;, activityDetail = &quot;1min&quot;, date = &quot;2018-08-24&quot;)<br>saturday_steps %&lt;&gt;% mutate(date = &quot;2018-08-24&quot;)<br>saturday &lt;- saturday_heart %&gt;% full_join(saturday_steps)<br>saturday %&lt;&gt;% mutate(week_date = &quot;Saturday&quot;)<br>saturday %&lt;&gt;% mutate(day_of_week = &quot;6&quot;)</pre><pre>week &lt;- monday %&gt;% bind_rows(tuesday) %&gt;% bind_rows(wednesday) %&gt;% bind_rows(thursday) %&gt;% bind_rows(friday) %&gt;% bind_rows(saturday)<br>        <br>week$date &lt;- as.Date(week$date)</pre><h3>Summary Statistics</h3><pre># Opening pod bay doors</pre><pre>week %&gt;% <br>        group_by(type) %&gt;% <br>                summarise(<br>                        total = sum(value),<br>                        minimum = min(value),<br>                        mean = mean(value),<br>                        median = median(value),<br>                        maximum = max(value),<br>                        max_time = max(time)<br>                        ) %&gt;%<br>        knitr::kable(digits = 3) %&gt;% <br>        kable_styling(full_width = F)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*TFqP9E4gxIHqo9IKlgSERg.png" /></figure><h3>Exploratory Data Analysis</h3><p>Since this is a post about gamification I decided to do something fun with my exploratory data visualizations. I wanted to use the Rokemon package which allows me to set the theme of ggplot2 (and ggplot2 extensions) to Game Boy and Game Boy Advance themes! When convenient, I’ve combined plots with cowplot.</p><p>Let’s take a quick look at the relationship and distribution of heart rate and step count.</p><pre># Doing the thing...</pre><pre>g &lt;- week %&gt;% <br>        spread(type, value) %&gt;% <br>        rename(hear_rate = &quot;heart rate&quot;) %&gt;% <br>        na.omit() %&gt;% <br>        ggplot(aes(steps, hear_rate)) + geom_point() + geom_smooth(method=&quot;lm&quot;, se=F, colour = &quot;#DE7243&quot;) </pre><pre>gb &lt;- g + theme_gameboy()<br>gba &lt;- g + theme_gba()</pre><pre>plot_grid(gb, gba, labels = c(&quot;&quot;, &quot;&quot;), align = &quot;h&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tL7RuSmO6g0rBWYmjqiCjw.png" /></figure><p>Alternatively, we could get a better look at the data by adding marginal density plots to the scatterplots with the ggMarginal() function from the ggExtra package.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*awEEdUUu0qsU31Cgwb9RSQ.png" /></figure><h3>Weekly trends</h3><p>Let’s take a quick look at the distribution of the contiguous variables to get a better idea than the mean and median.</p><pre># Loading..... Wait, what else were you expecting?</pre><pre>annotations_steps &lt;- data_frame(<br>        x = c(45, 100, 165),<br>        y = c(0.01, 0.01, 0.01),<br>        label = c(&#39;walking pace&#39;, &#39;brisk walking pace&#39;, &#39;running pace&#39;),<br>        type = c(&#39;steps&#39;, &#39;steps&#39;, &#39;steps&#39;)<br>)</pre><pre>g &lt;- week %&gt;% <br>  ggplot(aes(value)) + <br>  geom_density(fill = &quot;#DE7243&quot;) +<br>  geom_text(data = annotations_steps, aes(x = x, y = y, label = label), angle = -30, hjust = 1) +<br>  facet_grid(.~type, scales = &#39;free_x&#39;) + <br>  labs(title = &#39;Heart Rate and Steps-per-minute over two months&#39;,<br>       subtitle = &#39;Data gathered from Fitbit Charge2&#39;)</pre><pre>g + theme_gameboy()<br>g + theme_gba()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H2NbPNizSKZqPZSwlCGaWQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Avm4lYB5WP_Tze6PFwW_5A.png" /></figure><p>Heart rate is a little right-skewed, probably due to sleep and sedentary work. Similarly, for step count you see that only a small bump under brisk walking pace from when I skateboarded to work.</p><h3>Exercise patterns</h3><p>This week I didn’t work out so I thought I’d at least look at when I was on my way to work. The figure below shows blue for heart rate/min and orange is the number of steps/min.</p><pre># You are carrying too much to be able to run</pre><pre>between_six_nine &lt;- function(time) time &gt; 7*60*60 &amp; time &lt; 10*60*60</pre><pre>is_weekday &lt;- function(day_of_week) day_of_week %in% 1:6</pre><pre>week$week_date_f &lt;- factor(week$week_date, levels=c(&quot;Monday&quot;,&quot;Tuesday&quot;,&quot;Wednesday&quot;, &quot;Thursday&quot;, &quot;Friday&quot;, &quot;Saturday&quot;))</pre><pre>g &lt;- week %&gt;% <br>        filter(between_six_nine(time) &amp; is_weekday(day_of_week)) %&gt;% <br>        spread(type, value) %&gt;% <br>        ggplot(aes(x = time)) +<br>        geom_bar(aes(y = steps), color = &#39;#DE7243&#39;, alpha = 0.3, stat = &#39;identity&#39;) + <br>        geom_line(aes(y = `heart rate`), color = &#39;#E3F24D&#39;, size = 0.8) + <br>        facet_grid(~week_date_f) +<br>        scale_x_continuous(breaks=c(27000, 30000, 33000, 36000), labels=c(&quot;7am&quot;, &quot;8am&quot;, &quot;9am&quot;, &quot;10am&quot;))</pre><pre>g + theme_gameboy()<br>g + theme_gba()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D3D8lvHds59v4HbWaKULOA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4xcR0gCR0-SQI5vmxo1hzg.png" /></figure><p>My activity has been pretty much the same all week since I skateboard to work every morning.</p><h3>Most active time</h3><pre># 60% of the time, it loads ALL the time</pre><pre>step_counts &lt;- week %&gt;% <br>  filter(type == &#39;steps&#39;) %&gt;% <br>  group_by(day_of_week) %&gt;% <br>  summarise(<br>    type = last(type), <br>    avg_num_steps = sprintf(&#39;avg num steps: %3.0f&#39;, sum(value)/52)<br>  )</pre><pre>g &lt;- week %&gt;%<br>        ggplot(aes(x= value, y = fct_rev(factor(day_of_week)))) +<br>        geom_density_ridges(scale = 2.5, fill = &quot;#DE7243&quot;) +<br>        geom_text(data = step_counts, nudge_y = 0.15, hjust = 0, <br>            aes(x = 85, y = fct_rev(factor(day_of_week)), label = avg_num_steps)) +<br>        scale_y_discrete(breaks=1:6, labels = c(&quot;Monday&quot;, &quot;Tuesday&quot;, &quot;Wednesday&quot;, &quot;Thursday&quot;, &quot;Friday&quot;, &quot;Saturday&quot;)) +<br>        facet_grid(.~type, scales = &quot;free&quot;) +<br>        labs(x = &#39;&#39;, y = &quot;Day of the Week&quot;) </pre><pre>g + theme_gameboy()<br>g + theme_gba()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Z0waiTSvovqTlZC-9ud5g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rFddnm9WhH2-fWaB2lii5Q.png" /></figure><p>The distribution of steps per-minute was pretty constant because as I said I didn’t work-out; this likely reflects me shuffling to get tea.</p><p>It looks like Monday was the day I got my heart rate up the most, the bimodal peak is probably when I was running around looking for a rental property.</p><h3>Accessing the Fitbit API directly with the httr package</h3><p>Eventually, I found an excellent <a href="https://obrl-soil.github.io/fitbit-api-r/">tutorial</a> by <a href="https://github.com/obrl-soil">obrl-soil</a> which introduced me to the httr package and gave me the confidence I needed to peruse the <a href="https://dev.fitbit.com/build/reference/web-api/">Fitbit DEV web API reference</a>. Now I was able to gain access to far more sources of data.</p><h3>What data is available?</h3><p>A brief overview of what data is available from the Fitbit API:</p><pre># Your Boko Club is badly damaged</pre><pre># make a kable table for data you can access from Fitbit API<br>dt01 &lt;- data.frame(Scope = c(&quot;activity&quot;,<br>                             &quot;heartrate&quot;,<br>                             &quot;location&quot;,<br>                             &quot;nutrition&quot;,<br>                             &quot;profile&quot;,<br>                             &quot;settings&quot;,<br>                             &quot;sleep&quot;,<br>                             &quot;social&quot;,<br>                             &quot;weight&quot;),<br>                   Description = c(&quot;The activity scope includes activity data and exercise log related features, such as steps, distance, calories burned, and active minutes&quot;,<br>                                   &quot;The heartrate scope includes the continuous heart rate data and related analysis&quot;,<br>                                   &quot;The location scope includes the GPS and other location data&quot;,<br>                                   &quot;The nutrition scope includes calorie consumption and nutrition related features, such as food/water logging, goals, and plans&quot;,<br>                                   &quot;The profile scope is the basic user information&quot;,<br>                                   &quot;The settings scope includes user account and device settings, such as alarms&quot;,<br>                                   &quot;The sleep scope includes sleep logs and related sleep analysis&quot;,<br>                                   &quot;The social scope includes friend-related features, such as friend list, invitations, and leaderboard&quot;,<br>                                   &quot;The weight scope includes weight and related information, such as body mass index, body fat percentage, and goals&quot;)<br>)</pre><pre>dt01 %&gt;%<br>  kable(&quot;html&quot;) %&gt;%<br>  kable_styling(full_width = F) %&gt;%<br>  column_spec(1, bold = T, border_right = T) %&gt;%<br>  column_spec(2, width = &quot;30em&quot;, background = &quot;#E3F24D&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*xj5DbVpOv9ypbHhPocCW9Q.png" /></figure><p>What are the units of measurement?</p><pre># Loading Cutscenes You Can&#39;t Skip</pre><pre># make a Kable table or measurement information<br>dt03 &lt;- data.frame(unitType = c(&quot;duration&quot;,<br>                                &quot;distance&quot;,<br>                                &quot;elevation&quot;,<br>                                &quot;height&quot;,<br>                                &quot;weight&quot;,<br>                                &quot;body measurements&quot;,<br>                                &quot;liquids&quot;,<br>                                &quot;blood glucose&quot;),<br>                   unit = c(&quot;milliseconds&quot;,<br>                            &quot;kilometers&quot;,<br>                            &quot;meters&quot;,<br>                            &quot;centimeters&quot;,<br>                            &quot;kilograms&quot;,<br>                            &quot;centimeters&quot;,<br>                            &quot;milliliters&quot;,<br>                            &quot;millimoles per liter&quot;))<br>dt03 %&gt;%<br>  kable(&quot;html&quot;) %&gt;%<br>  kable_styling(full_width = F) %&gt;%<br>  column_spec(1, bold = T, border_right = T) %&gt;%<br>  column_spec(2, width = &quot;30em&quot;, background = &quot;#E3F24D&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*T3BNmCI84G5ee8NnBXbybQ.png" /></figure><p>Define a function for turning a json list into a dataframe.</p><pre># Inserting last-minute subroutines into program...</pre><pre># json-as-list to dataframe (for simple cases without nesting!)<br>jsonlist_to_df &lt;- function(data = NULL) {<br>    purrr::transpose(data) %&gt;%<br>    purrr::map(., unlist) %&gt;%<br>    as_tibble(., stringsAsFactors = FALSE)<br>}</pre><h3>Investigating my 10Km run</h3><p>GET request to retrieve minute-by-minute heart rate data for my 10km run.</p><pre># Preparing for the mini-boss</pre><pre>get_workout &lt;- function(date = NULL, start_time = NULL, end_time = NULL, <br>                         token = Sys.getenv(&#39;FITB_AUTH&#39;)) {<br>GET(url =<br>        paste0(&#39;https://api.fitbit.com/1/user/-/activities/heart/date/&#39;,<br>        date, &#39;/1d/1min/time/&#39;, start_time, &#39;/&#39;, end_time, &#39;.json&#39;), <br>        add_headers(Authorization = paste0(&quot;Bearer &quot;, token)))<br>}</pre><pre># Get the workout for my 10Km run <br>got_workout &lt;- get_workout(date = &#39;2018-10-21&#39;, start_time = &#39;09:29&#39;, end_time = &#39;10:24&#39;)</pre><pre>workout &lt;- content(got_workout)</pre><pre># summary</pre><pre>workout[[&#39;activities-heart&#39;]][[1]][[&#39;heartRateZones&#39;]] &lt;- jsonlist_to_df(workout[[&#39;activities-heart&#39;]][[1]][[&#39;heartRateZones&#39;]])</pre><pre># the dataset<br>workout[[&#39;activities-heart-intraday&#39;]][[&#39;dataset&#39;]] &lt;- jsonlist_to_df(workout[[&#39;activities-heart-intraday&#39;]][[&#39;dataset&#39;]])</pre><pre># format the time <br>workout$`activities-heart-intraday`$dataset$time &lt;- as.POSIXlt(workout$`activities-heart-intraday`$dataset$time, format = &#39;%H:%M:%S&#39;)<br>lubridate::date(workout$`activities-heart-intraday`$dataset$time) &lt;- &#39;2018-10-21&#39;</pre><pre># find time zone<br># grep(&quot;Canada&quot;, OlsonNames(), value=TRUE)<br>lubridate::tz(workout$`activities-heart-intraday`$dataset$time) &lt;- &#39;Canada/Eastern&#39;</pre><p>Let’s take a look at the summary for my 10Km run:</p><pre># Farming Hell Cows</pre><pre>workout$`activities-heart`[[1]]$heartRateZones %&gt;% kable() %&gt;% kable_styling(full_width = F)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*6K4-HyAJVhifc8w0wqVaKw.png" /></figure><p><a href="https://obrl-soil.github.io/fitbit-api-r/">obrl-soil</a> used the MyZone Efforts Points <a href="https://myzone.org/the-zones/">(MEPS)</a> which is calculated minute-by-minute as a percentage of max heart rate. It measures the effort put in - The more points the better. Another example of gamification in action.</p><pre># Looting a chest</pre><pre>meps_max &lt;- function(age = NULL) { 207 - (0.7 * age) }</pre><p>Mine is 186.</p><p>Now is we create a tribble with 4 heart ranges showing the lower and higher bounds and use the mutate() function from above to calculate what my max heart rate is (with lower and upper bounds).</p><pre># Taking the hobbits to Isengard</pre><pre>my_MEPS &lt;- tribble(~MEPS, ~hr_range, ~hr_lo, ~hr_hi, <br>                       1,  &#39;50-59%&#39;,   0.50,   0.59,<br>                       2,  &#39;60-69%&#39;,   0.60,   0.69,<br>                       3,  &#39;70-79%&#39;,   0.70,   0.79,<br>                       4,    &#39;&gt;=80&#39;,   0.80,   1.00) %&gt;%<br>  mutate(my_hr_low = floor(meps_max(30) * hr_lo),<br>         my_hr_hi  = ceiling(meps_max(30) * hr_hi))<br>my_MEPS</pre><pre>## # A tibble: 4 x 6<br>##    MEPS hr_range hr_lo hr_hi my_hr_low my_hr_hi<br>##   &lt;dbl&gt; &lt;chr&gt;    &lt;dbl&gt; &lt;dbl&gt;     &lt;dbl&gt;    &lt;dbl&gt;<br>## 1     1 50-59%     0.5  0.59        93      110<br>## 2     2 60-69%     0.6  0.69       111      129<br>## 3     3 70-79%     0.7  0.79       130      147<br>## 4     4 &gt;=80       0.8  1          148      186</pre><p>With the equation now defined let’s calculate my total MEPS:</p><pre># Checkpoint!</pre><pre>mep &lt;- mutate(workout$`activities-heart-intraday`$dataset,<br>       meps = case_when(value &gt;= 146 ~ 4,<br>                        value &gt;= 128 ~ 3,<br>                        value &gt;= 109 ~ 2,<br>                        value &gt;= 91  ~ 1,<br>                        TRUE ~ 0)) %&gt;%<br>  summarise(&quot;Total MEPS&quot; = sum(meps))</pre><p>Wow it’s 216!</p><p>I’m not sure what that exactly means but apparently the maximum possible MEPS in a 42-minute workout is 168 and since I ran this 10Km in 54:35 I guess that’s good?</p><p>I’d like to post sub 50 minutes on my next 10Km run but I’m not sure if I should be aiming to shoot for a greater percentage of peak heart rate minutes or not - guess I will need to look into this.</p><h3>Minute-by-minute sleep data for one night</h3><p>Let’s examine my sleep patterns last night.</p><pre># Resting at Campfire</pre><pre>get_sleep &lt;- function(startDate = NULL, endDate = NULL, token = Sys.getenv(&#39;FITB_AUTH&#39;)){<br>        GET(url = paste0(&#39;https://api.fitbit.com/1.2/user/-/sleep/date/&#39;, startDate, &quot;/&quot;, endDate, &#39;.json&#39;),<br>        add_headers(Authorization = paste0(&quot;Bearer &quot;, token)))<br>}</pre><pre># make sure that there is data for those days otherwise it tosses an error</pre><pre>got_sleep &lt;- get_sleep(startDate = &quot;2018-08-21&quot;, endDate = &quot;2018-08-22&quot;)<br>sleep &lt;- content(got_sleep)</pre><pre>dateRange &lt;- seq(as.Date(&quot;2018-08-21&quot;), as.Date(&quot;2018-08-22&quot;), &quot;days&quot;)</pre><pre>sleep_pattern &lt;- NULL<br>for(j in 1:length(dateRange)){<br>        sleep[[&#39;sleep&#39;]][[j]][[&#39;levels&#39;]][[&#39;data&#39;]] &lt;- jsonlist_to_df(sleep[[&#39;sleep&#39;]][[j]][[&#39;levels&#39;]][[&#39;data&#39;]])<br>        tmp &lt;- sleep$sleep[[j]]$levels$`data`<br>sleep_pattern &lt;- bind_rows(sleep_pattern, tmp)<br>}</pre><p>Okay now that the data munging is complete, let’s look at my sleep pattern.</p><pre># Now entering... The Twilight Zone</pre><pre>g &lt;- sleep_pattern %&gt;% group_by(level, seconds) %&gt;% <br>        summarise() %&gt;% <br>        summarise(seconds = sum(seconds)) %&gt;%<br>        mutate(percentage = seconds/sum(seconds)) %&gt;% <br>                ggplot(aes(x = &quot;&quot;, y = percentage, fill = c(&quot;S&quot;, &quot;A&quot;, &quot;R&quot;))) +<br>                        geom_bar(width = 1, stat = &quot;identity&quot;) +<br>                        theme(axis.text.y = element_blank(),<br>                              axis.text.x = element_blank(), axis.line = element_blank(), plot.caption = element_text(size = 5), plot.title = element_blank()) +<br>                        labs(fill = &quot;class&quot;, x = NULL, y = NULL, title = &quot;Sleep stages&quot;, caption = &quot;A = Awake; R = Restless; S = Asleep&quot;) +<br>                        coord_polar(theta = &quot;y&quot;, start = 0) +<br>        scale_fill_manual(values = c(&quot;#FF3F3F&quot;, &quot;#2BD1FC&quot;, &quot;#BA90A6&quot;))</pre><pre>g + theme_gameboy()<br>g + theme_gba()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*50x__0wpHqAmx5JegwLG_Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*Tk2cVpX_Udwan5C4T7aTWg.png" /></figure><p>A pie chart is probably not the best way to show this data. Let’s visualize the distribution with a box plot.</p><pre># Entering Cheat Codes!</pre><pre>g &lt;- ggplot(sleep_pattern, aes(y=log10(seconds), x=level)) + <br>        geom_boxplot(color=&quot;#031300&quot;, fill=&#39;#152403&#39;) +<br>        labs(x = &quot;&quot;, title = &#39;Sleep patterns over a month&#39;,<br>       subtitle = &#39;Data gathered from Fitbit Charge2&#39;) +<br>        theme(legend.position = &quot;none&quot;) </pre><pre>g + theme_gameboy()<br>g + theme_gba()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9dd9RScJF9yV_2-ecWrqgQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C3R7eMENM1Tqg5HlqKIVjA.png" /></figure><p>An even better way to visualize the distribution would be to use a violin plot with the raw data points overlaid.</p><pre># Neglecting Sleep...<br></pre><pre>g &lt;- ggplot(sleep_pattern, aes(y=log10(seconds), x=level)) + <br>        geom_violin(color=&quot;#031300&quot;, fill=&#39;#152403&#39;) +<br>        geom_point() +<br>        labs(x = &quot;&quot;, title = &#39;Sleep patterns over a month&#39;,<br>       subtitle = &#39;Data gathered from Fitbit Charge2&#39;) +<br>        theme(legend.position = &quot;none&quot;)</pre><pre>g + theme_gameboy()<br>g + theme_gba()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S3aEYsgW7te8sc29ueAJuw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IAzy3-c1mOsBXvRI2ekLMA.png" /></figure><h3>Get Daily Activity Patterns For 3 Months</h3><p>You can do API requests for various periods from the Fitbit <a href="https://dev.fitbit.com/build/reference/web-api/activity/">Activity and Exercise Logs</a> but since I’ve only had mine a couple months I’ll use the 3m period.</p><p>I will also need to trim off any day’s which are in the future otherwise they’ll appear as 0 calories in the figures. It’s best to use the Sys.Date() function rather than hardcoding the date when doing EDA, making a Shiny app, or <a href="https://resources.rstudio.com/rstudio-connect-2/parameterized-r-markdown-reports-with-rstudio-connect-aron-atkins">parameterizing a </a><a href="https://resources.rstudio.com/rstudio-connect-2/parameterized-r-markdown-reports-with-rstudio-connect-aron-atkins">RMarkdown file</a>. This way you can explore different time periods without anything breaking.</p><p>I cannot remember when I started wearing my Fitbit but we can figure that out with the following code:</p><pre># ULTIMATE IS READY!</pre><pre># Query how many days since you&#39;ve had fitbit for<br>inception &lt;- user_info$memberSince</pre><p>I’ve had my Fitbit since 2018–08–20.</p><p>Let’s gather data from September 20th until November 6th 2018.</p><pre># Catching them all!</pre><pre>### Calories<br>get_calories &lt;- function(baseDate = NULL, period = NULL, token = Sys.getenv(&#39;FITB_AUTH&#39;)){<br>        GET(url = paste0(&#39;https://api.fitbit.com/1/user/-/activities/calories/date/&#39;, baseDate, &quot;/&quot;, period, &#39;.json&#39;),<br>        add_headers(Authorization = paste0(&quot;Bearer &quot;, token)))<br>}<br> <br>got_calories &lt;- get_calories(baseDate = &quot;2018-11-20&quot;, period = &quot;3m&quot;)<br>calories &lt;- content(got_calories)<br># turn into df<br>calories[[&#39;activities-calories&#39;]] &lt;- jsonlist_to_df(calories[[&#39;activities-calories&#39;]])<br># assign easy object and rename<br>calories &lt;- calories[[&#39;activities-calories&#39;]]<br>colnames(calories) &lt;- c(&quot;dateTime&quot;, &quot;calories&quot;)</pre><pre>### STEPS<br>get_steps &lt;- function(baseDate = NULL, period = NULL, token = Sys.getenv(&#39;FITB_AUTH&#39;)){<br>        GET(url = paste0(&#39;https://api.fitbit.com/1/user/-/activities/steps/date/&#39;, baseDate, &quot;/&quot;, period, &#39;.json&#39;),<br>        add_headers(Authorization = paste0(&quot;Bearer &quot;, token)))<br>}<br> <br>got_steps &lt;- get_steps(baseDate = &quot;2018-11-20&quot;, period = &quot;3m&quot;)<br>steps &lt;- content(got_steps)<br># turn into df<br>steps[[&#39;activities-steps&#39;]] &lt;- jsonlist_to_df(steps[[&#39;activities-steps&#39;]])<br># assign easy object and rename<br>steps &lt;- steps[[&#39;activities-steps&#39;]]<br>colnames(steps) &lt;- c(&quot;dateTime&quot;, &quot;steps&quot;)</pre><pre>### DISTANCE<br>get_distance &lt;- function(baseDate = NULL, period = NULL, token = Sys.getenv(&#39;FITB_AUTH&#39;)){<br>        GET(url = paste0(&#39;https://api.fitbit.com/1/user/-/activities/distance/date/&#39;, baseDate, &quot;/&quot;, period, &#39;.json&#39;),<br>        add_headers(Authorization = paste0(&quot;Bearer &quot;, token)))<br>}<br> <br>got_distance &lt;- get_distance(baseDate = &quot;2018-11-20&quot;, period = &quot;3m&quot;)<br>distance &lt;- content(got_distance)<br># turn into df<br>distance[[&#39;activities-distance&#39;]] &lt;- jsonlist_to_df(distance[[&#39;activities-distance&#39;]])<br># assign easy object and rename<br>distance &lt;- distance[[&#39;activities-distance&#39;]]<br>colnames(distance) &lt;- c(&quot;dateTime&quot;, &quot;distance&quot;)</pre><pre>### FLOORS<br>get_floors &lt;- function(baseDate = NULL, period = NULL, token = Sys.getenv(&#39;FITB_AUTH&#39;)){<br>        GET(url = paste0(&#39;https://api.fitbit.com/1/user/-/activities/floors/date/&#39;, baseDate, &quot;/&quot;, period, &#39;.json&#39;),<br>        add_headers(Authorization = paste0(&quot;Bearer &quot;, token)))<br>}<br> <br>got_floors &lt;- get_floors(baseDate = &quot;2018-11-20&quot;, period = &quot;3m&quot;)<br>floors &lt;- content(got_floors)<br># turn into df<br>floors[[&#39;activities-floors&#39;]] &lt;- jsonlist_to_df(floors[[&#39;activities-floors&#39;]])<br># assign easy object and rename<br>floors &lt;- floors[[&#39;activities-floors&#39;]]<br>colnames(floors) &lt;- c(&quot;dateTime&quot;, &quot;floors&quot;)</pre><pre>### ELEVATION<br>get_elevation &lt;- function(baseDate = NULL, period = NULL, token = Sys.getenv(&#39;FITB_AUTH&#39;)){<br>        GET(url = paste0(&#39;https://api.fitbit.com/1/user/-/activities/elevation/date/&#39;, baseDate, &quot;/&quot;, period, &#39;.json&#39;),<br>        add_headers(Authorization = paste0(&quot;Bearer &quot;, token)))<br>}<br> <br>got_elevation &lt;- get_elevation(baseDate = &quot;2018-11-20&quot;, period = &quot;3m&quot;)<br>elevation &lt;- content(got_elevation)<br># turn into df<br>elevation[[&#39;activities-elevation&#39;]] &lt;- jsonlist_to_df(elevation[[&#39;activities-elevation&#39;]])<br># assign easy object and rename<br>elevation &lt;- elevation[[&#39;activities-elevation&#39;]]<br>colnames(elevation) &lt;- c(&quot;dateTime&quot;, &quot;elevation&quot;)</pre><pre>### minutesSedentary<br>get_minutesSedentary &lt;- function(baseDate = NULL, period = NULL, token = Sys.getenv(&#39;FITB_AUTH&#39;)){<br>        GET(url = paste0(&#39;https://api.fitbit.com/1/user/-/activities/minutesSedentary/date/&#39;, baseDate, &quot;/&quot;, period, &#39;.json&#39;),<br>        add_headers(Authorization = paste0(&quot;Bearer &quot;, token)))<br>}<br> <br>got_minutesSedentary &lt;- get_minutesSedentary(baseDate = &quot;2018-11-20&quot;, period = &quot;3m&quot;)<br>minutesSedentary &lt;- content(got_minutesSedentary)<br># turn into df<br>minutesSedentary[[&#39;activities-minutesSedentary&#39;]] &lt;- jsonlist_to_df(minutesSedentary[[&#39;activities-minutesSedentary&#39;]])<br># assign easy object and rename<br>minutesSedentary &lt;- minutesSedentary[[&#39;activities-minutesSedentary&#39;]]<br>colnames(minutesSedentary) &lt;- c(&quot;dateTime&quot;, &quot;minutesSedentary&quot;)</pre><pre>### minutesLightlyActive<br>get_minutesLightlyActive &lt;- function(baseDate = NULL, period = NULL, token = Sys.getenv(&#39;FITB_AUTH&#39;)){<br>        GET(url = paste0(&#39;https://api.fitbit.com/1/user/-/activities/minutesLightlyActive/date/&#39;, baseDate, &quot;/&quot;, period, &#39;.json&#39;),<br>        add_headers(Authorization = paste0(&quot;Bearer &quot;, token)))<br>}<br> <br>got_minutesLightlyActive &lt;- get_minutesLightlyActive(baseDate = &quot;2018-11-20&quot;, period = &quot;3m&quot;)<br>minutesLightlyActive &lt;- content(got_minutesLightlyActive)<br># turn into df<br>minutesLightlyActive[[&#39;activities-minutesLightlyActive&#39;]] &lt;- jsonlist_to_df(minutesLightlyActive[[&#39;activities-minutesLightlyActive&#39;]])<br># assign easy object and rename<br>minutesLightlyActive &lt;- minutesLightlyActive[[&#39;activities-minutesLightlyActive&#39;]]<br>colnames(minutesLightlyActive) &lt;- c(&quot;dateTime&quot;, &quot;minutesLightlyActive&quot;)</pre><pre>### minutesFairlyActive<br>get_minutesFairlyActive &lt;- function(baseDate = NULL, period = NULL, token = Sys.getenv(&#39;FITB_AUTH&#39;)){<br>        GET(url = paste0(&#39;https://api.fitbit.com/1/user/-/activities/minutesFairlyActive/date/&#39;, baseDate, &quot;/&quot;, period, &#39;.json&#39;),<br>        add_headers(Authorization = paste0(&quot;Bearer &quot;, token)))<br>}<br> <br>got_minutesFairlyActive &lt;- get_minutesFairlyActive(baseDate = &quot;2018-11-20&quot;, period = &quot;3m&quot;)<br>minutesFairlyActive &lt;- content(got_minutesFairlyActive)<br># turn into df<br>minutesFairlyActive[[&#39;activities-minutesFairlyActive&#39;]] &lt;- jsonlist_to_df(minutesFairlyActive[[&#39;activities-minutesFairlyActive&#39;]])<br># assign easy object and rename<br>minutesFairlyActive &lt;- minutesFairlyActive[[&#39;activities-minutesFairlyActive&#39;]]<br>colnames(minutesFairlyActive) &lt;- c(&quot;dateTime&quot;, &quot;minutesFairlyActive&quot;)</pre><pre>### minutesVeryActive<br>get_minutesVeryActive &lt;- function(baseDate = NULL, period = NULL, token = Sys.getenv(&#39;FITB_AUTH&#39;)){<br>        GET(url = paste0(&#39;https://api.fitbit.com/1/user/-/activities/minutesVeryActive/date/&#39;, baseDate, &quot;/&quot;, period, &#39;.json&#39;),<br>        add_headers(Authorization = paste0(&quot;Bearer &quot;, token)))<br>}<br> <br>got_minutesVeryActive &lt;- get_minutesVeryActive(baseDate = &quot;2018-11-20&quot;, period = &quot;3m&quot;)<br>minutesVeryActive &lt;- content(got_minutesVeryActive)<br># turn into df<br>minutesVeryActive[[&#39;activities-minutesVeryActive&#39;]] &lt;- jsonlist_to_df(minutesVeryActive[[&#39;activities-minutesVeryActive&#39;]])<br># assign easy object and rename<br>minutesVeryActive &lt;- minutesVeryActive[[&#39;activities-minutesVeryActive&#39;]]<br>colnames(minutesVeryActive) &lt;- c(&quot;dateTime&quot;, &quot;minutesVeryActive&quot;)</pre><pre>### activityCalories<br>get_activityCalories &lt;- function(baseDate = NULL, period = NULL, token = Sys.getenv(&#39;FITB_AUTH&#39;)){<br>        GET(url = paste0(&#39;https://api.fitbit.com/1/user/-/activities/activityCalories/date/&#39;, baseDate, &quot;/&quot;, period, &#39;.json&#39;),<br>        add_headers(Authorization = paste0(&quot;Bearer &quot;, token)))<br>}<br> <br>got_activityCalories &lt;- get_activityCalories(baseDate = &quot;2018-11-20&quot;, period = &quot;3m&quot;)<br>activityCalories &lt;- content(got_activityCalories)<br># turn into df<br>activityCalories[[&#39;activities-activityCalories&#39;]] &lt;- jsonlist_to_df(activityCalories[[&#39;activities-activityCalories&#39;]])<br># assign easy object and rename<br>activityCalories &lt;- activityCalories[[&#39;activities-activityCalories&#39;]]<br>colnames(activityCalories) &lt;- c(&quot;dateTime&quot;, &quot;activityCalories&quot;)</pre><pre>##### Join multiple dataframes with purrr::reduce and dplyr::left_join<br>activity_df &lt;- list(calories, steps, distance, floors, elevation, activityCalories, minutesSedentary, minutesLightlyActive, minutesFairlyActive, minutesVeryActive) %&gt;% <br>        purrr::reduce(left_join, by = &quot;dateTime&quot;)</pre><pre># Add the dateTime to this dataframe<br>activity_df$dateTime &lt;- as.Date(activity_df$dateTime)</pre><pre>names &lt;- c(2:ncol(activity_df))<br>activity_df[,names] &lt;- lapply(activity_df[,names], as.numeric)</pre><pre># trim off any days that haven&#39;t happened yet<br>activity_df %&lt;&gt;% filter(dateTime &lt;= &quot;2018-11-06&quot;)</pre><h3>Get Recent Activity Types</h3><pre># We&#39;re giving it all she&#39;s got!<br><br>get_frequentActivities &lt;- function(baseDate = NULL, period = NULL, token = Sys.getenv(&#39;FITB_AUTH&#39;)){<br>    GET(url = paste0(&#39;https://api.fitbit.com/1/user/-/activities/recent.json&#39;),<br>        add_headers(Authorization = paste0(&quot;Bearer &quot;, token)))<br>}<br><br>got_frequentActivities &lt;- get_frequentActivities(baseDate = &quot;2018-11-20&quot;, period = &quot;3m&quot;)<br>frequentActivities &lt;- content(got_frequentActivities)</pre><pre># This is a list object let&#39;s look at how many frequent activities are logged<br>length(frequentActivities)<br>## [1] 5</pre><pre># Take a look at the object with str()<br>str(frequentActivities)</pre><pre>## List of 5<br>##  $ :List of 6<br>##   ..$ activityId : int 2131<br>##   ..$ calories   : int 0<br>##   ..$ description: chr &quot;&quot;<br>##   ..$ distance   : int 0<br>##   ..$ duration   : int 3038000<br>##   ..$ name       : chr &quot;Weights&quot;<br>##  $ :List of 6<br>##   ..$ activityId : int 90009<br>##   ..$ calories   : int 0<br>##   ..$ description: chr &quot;Running - 5 mph (12 min/mile)&quot;<br>##   ..$ distance   : int 0<br>##   ..$ duration   : int 1767000<br>##   ..$ name       : chr &quot;Run&quot;<br>##  $ :List of 6<br>##   ..$ activityId : int 90013<br>##   ..$ calories   : int 0<br>##   ..$ description: chr &quot;Walking less than 2 mph, strolling very slowly&quot;<br>##   ..$ distance   : int 0<br>##   ..$ duration   : int 2407000<br>##   ..$ name       : chr &quot;Walk&quot;<br>##  $ :List of 6<br>##   ..$ activityId : int 90001<br>##   ..$ calories   : int 0<br>##   ..$ description: chr &quot;Very Leisurely - Less than 10 mph&quot;<br>##   ..$ distance   : int 0<br>##   ..$ duration   : int 4236000<br>##   ..$ name       : chr &quot;Bike&quot;<br>##  $ :List of 6<br>##   ..$ activityId : int 15000<br>##   ..$ calories   : int 0<br>##   ..$ description: chr &quot;&quot;<br>##   ..$ distance   : int 0<br>##   ..$ duration   : int 1229000<br>##   ..$ name       : chr &quot;Sport&quot;</pre><p>I would never have considered myself a <a href="https://www.psychologytoday.com/us/blog/how-think-about-exercise/201501/charles-darwins-daily-walks">Darwin</a> or a <a href="https://en.wikipedia.org/wiki/Walking_(Thoreau)">Thoreau</a> but apparently <em>strolling very slowly</em> is my favorite activity in terms of time spent.</p><p>You can see that my Fitbit has also logged times for Weights, Sports and Biking which is likely from when I’ve manually logged my activities. There’s a possibility that Fitbit is registering Biking for when I skateboard.</p><h3>Correlogram of Activity</h3><p>Previously I had always used the corrplot package to create a correlation plot; however, it doesn’t play nicely with ggplot meaning you cannot add Game Boy themes easily. Nonetheless, I was able to give it a retro-looking palette with some minor tweaking.</p><p>Since I had two colors in mind from the original gameboy, and knew their hex code, I was able to <a href="https://meyerweb.com/eric/tools/color-blend/#CADCA0:0F380E:10:hex">generate a palette from this website</a>.</p><pre># Aligning Covariance Matrices </pre><pre># drop dateTime<br>corr_df &lt;- activity_df[,2:11]</pre><pre># Correlation matrix<br>corr &lt;- cor(na.omit(corr_df))<br>corrplot(corr, type = &quot;upper&quot;, bg = &quot;#9BBB0E&quot;, tl.col = &quot;#565656&quot;, col = c(&quot;#CADCA0&quot;, &quot;#B9CD93&quot;, &quot;#A8BE85&quot;, &quot;#97AF78&quot;, &quot;#86A06B&quot;, &quot;#75915E&quot;, &quot;#648350&quot;, &quot;#537443&quot;, &quot;#426536&quot;, &quot;#315629&quot;, &quot;#20471B&quot;, &quot;#0F380E&quot;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tTg5t-6UxM1TAnPzQPSXVg.png" /></figure><p>In a correlation plot the color of each circle indicates the magnitude of the correlation, and the size of the circle indicates its significance.</p><p>After a bit of searching for a ggplot2 extension I was able to use ggcorrplot which allowed me to use gameboy themes again!</p><pre># Generating textures...</pre><pre>ggcorrplot(corr, hc.order = TRUE, <br>           type = &quot;lower&quot;, <br>           lab = TRUE, <br>           lab_size = 2,<br>           tl.cex = 8,<br>           show.legend = FALSE,<br>           colors = c( &quot;#306230&quot;, &quot;#306230&quot;, &quot;#0F380F&quot; ), <br>           title=&quot;Correlogram&quot;,<br>           ggtheme=theme_gameboy)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*1SU_WnacCEwUMKyktMf5mw.png" /></figure><pre># Game Over. Loading previous save<br><br>ggcorrplot(corr, hc.order = TRUE, <br>           type = &quot;lower&quot;, <br>           lab = TRUE, <br>           lab_size = 2,<br>           tl.cex = 8,<br>           show.legend = FALSE,<br>           colors = c( &quot;#3B7AAD&quot;, &quot;#56B1F7&quot;, &quot;#1D3E5D&quot; ), <br>           title=&quot;Correlogram&quot;,<br>           ggtheme=theme_gba)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*nzc7qhOaNtKUGMvhxJTnEQ.png" /></figure><h3>Exploring Activity</h3><pre># Link saying &quot;hyahhh!&quot;<br><br># Static<br>g &lt;- activity_df %&gt;% <br>        ggplot(aes(x=dateTime, y=calories)) + <br>        geom_line(colour = &quot;black&quot;) +<br>        geom_point(shape = 21, colour = &quot;black&quot;, aes(fill = calories), size = 5, stroke = 1) +<br>        xlab(&quot;&quot;) +<br>        ylab(&quot;Calorie Expenditure&quot;)<br><br>g + theme_gameboy() + theme(legend.position = &quot;none&quot;)<br>g + theme_gba() + theme(legend.position = &quot;none&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_8lQAMs8dYrGcTqmrxR_Yw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G7BveAdVwZZJAC6xDvoulg.png" /></figure><pre># Panick! at the Discord...<br><br># gganimate<br>g &lt;- activity_df %&gt;% <br>        ggplot(aes(x=dateTime, y=calories)) + <br>        geom_line(colour = &quot;black&quot;) +<br>        geom_point(shape = 21, colour = &quot;black&quot;, aes(fill = calories), size = 5, stroke = 1) +<br>        transition_time(dateTime) +<br>        shadow_mark() +<br>        ease_aes(&#39;linear&#39;) +<br>        xlab(&quot;&quot;) +<br>        ylab(&quot;Calorie Expenditure&quot;) <br><br>g + theme_gba() + theme(legend.position = &quot;none&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4pdWa_N8doJnrfJ9uXSqjA.gif" /></figure><p>Distance is determined by using your steps and your estimated stride length (for the height you put in).</p><p>I’ve also made plots for Distance, Steps, Elevationand Floorsbut you’ll have to <a href="https://moldach.github.io/fitbit-project/">check out this page to see them</a>.</p><h3>Closing thoughts</h3><p>Even though Fitbit offers a nice dashboard for a single user it’s not scale-able. By accessing the data directly one can ask the questions they want from 200 individuals — or more. If one was inclined, they could even build a fancy Shiny dashboard with bespoke visualizations.</p><p>If you have any questions or comments you can always reach me on <a href="https://www.linkedin.com/in/matthewoldach/">LinkedIn</a>. Till then, see you in the next post!</p><pre># Wubba Lubba Dub Dub<br><br># https://www.spriters-resource.com/game_boy_advance/kirbynim/sheet/15585/<br>sprite_sheet &lt;- png::readPNG(&quot;kirby.png&quot;)<br><br>Nframes       &lt;- 11      # number of frames to extract<br>width         &lt;- 29      # width of a frame<br>sprite_frames &lt;- list()  # storage for the extracted frames<br><br># Not equal sized frames in the sprite sheet. Need to compensate for each frame<br>offset &lt;- c(0, -4, -6, -7, -10, -16, -22, -26, -28, -29, -30)<br><br># Manually extract each frame<br>for (i in seq(Nframes)) {<br>  sprite_frames[[i]] &lt;- sprite_sheet[120:148, (width*(i-1)) + (1:width) + offset[i], 1:3]<br>}<br><br># Function to convert a sprite frame to a data.frame<br># and remove any background pixels i.e. #00DBFF<br>sprite_frame_to_df &lt;- function(frame) {<br>  plot_df &lt;- data_frame(<br>    fill  = as.vector(as.raster(frame)),<br>    x = rep(1:width, width),<br>    y = rep(width:1, each=width)<br>  ) %&gt;%<br>    filter(fill != &#39;#00DBFF&#39;)<br>}<br><br>sprite_dfs &lt;- sprite_frames %&gt;%<br>  map(sprite_frame_to_df) %&gt;%<br>  imap(~mutate(.x, idx=.y))<br><br>fill_manual_values &lt;- unique(sprite_dfs[[1]]$fill)<br>fill_manual_values &lt;- setNames(fill_manual_values, fill_manual_values)<br><br>mega_df &lt;- dplyr::bind_rows(sprite_dfs)<br><br>p &lt;- ggplot(mega_df, aes(x, y, fill=fill)) +<br>  geom_tile(width=0.9, height=0.9) +<br>  coord_equal(xlim=c(1, width), ylim=c(1, width)) +<br>  scale_fill_manual(values = fill_manual_values) +<br>  theme_gba() +<br>        xlab(&quot;&quot;) +<br>        ylab(&quot;&quot;) +<br>  theme(legend.position = &#39;none&#39;, axis.text=element_blank(), axis.ticks = element_blank())<br><br>panim &lt;- p +<br>  transition_manual(idx, seq_along(sprite_frames)) +<br>  labs(title = &quot;gganimate Kirby&quot;)<br><br>gganimate::animate(panim, fps=30, width=400, height=400)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*-Z0WH8ugtxiqyHK8oDP2Iw.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eaf7b267af00" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-gamification-of-fitbit-how-an-api-provided-the-next-level-of-training-eaf7b267af00">The Gamification Of Fitbit: How an API Provided the Next Level of tRaining</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Highlight 3D Brain Regions]]></title>
            <link>https://medium.com/data-science/how-to-highlight-3d-brain-regions-2e6c15a35574?source=rss-459e62b88a2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e6c15a35574</guid>
            <category><![CDATA[neuroscience]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[brain]]></category>
            <dc:creator><![CDATA[Matt.0]]></dc:creator>
            <pubDate>Wed, 31 Oct 2018 22:10:49 GMT</pubDate>
            <atom:updated>2018-11-04T22:56:33.103Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, I was reading Howard <em>et. al.,</em> (2018) “<strong>Genome-wide meta-analysis of depression in 807,553 individuals identifies 102 independent variants with replication in a further 1,507,153 individuals</strong>” and saw a really cool 3D visualization of highlighted brain regions associated with depression:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*X8-IvuHR-HeFQxt7LiW1PA.png" /><figcaption>Source: <a href="https://goo.gl/7rY5KV">https://goo.gl/7rY5KV</a></figcaption></figure><p>After an exhaustive search I couldn’t find any reference to how this was done in the methods or supplementary information so I reached out to the authors. While I was awaiting a response, I also reached out to the <a href="https://twitter.com/MattOldach/status/1051909841529315328">Twitterverse</a> to see if anyone knew of tools which could be used to create such a visualization.</p><p><a href="https://twitter.com/KarimH349">Helmet Karim</a> suggested that perhaps these images were created using BrainNet Viewer in MATLABso this was the first method I tried out.</p><p><em>Note</em>: All of the methods covered in this article use what is called a brain Atlas overlayed as a maskon a normalized T1 MRI image. I’ve chosen to highlight the left hippocampus across these methods so that they are comparable.</p><h3>BrainNet Viewer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*XZR76fWm8jRJiD_N9eJGdw.jpeg" /></figure><p>First follow the install instructions for BrainNet Viewer <a href="https://www.nitrc.org/docman/view.php/504/1280/BrainNet">here</a> then start the graphical user interface (GUI) up from the terminal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UvyKo-QodS6iTgv_s19T8g.png" /></figure><p>Next select load file and choose a surface template surface template and a mapping file (<em>i.e.</em> a brain atlas). The package provides samples so I chose BrainMesh_ICBMI52_smoothed.nv and the [<a href="http://neuro.imm.dtu.dk/wiki/Automated_Anatomical_Labeling">AAL90</a>](<a href="http://neuro.imm.dtu.dk/wiki/Automated_Anatomical_Labeling">http://neuro.imm.dtu.dk/wiki/Automated_Anatomical_Labeling</a>) brain atlas which has labeled volumes for 90 brain regions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p1-bNjyTlhZLy9WyKhm5tA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vg1pIqczG6oxBp0KgU-O_g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*28T7UZh9LkSBWtt4V_UNOw.png" /></figure><p>Next there’s a pop-up with 7-sections of which layout, surface and volume are of interest to us.</p><p>In layout select which view you would like, I’ve chosen full view which will show eight different viewpoints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uP5BdGzmMBf7t_wgafJqTw.png" /></figure><p>In the surface tab you can select the transparency of the surface map — I’ve set it to <strong>0.75</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MyEktOl85KFT5bXsa5qDAw.png" /></figure><p>In the volume tab select ROI drawing, deselect draw all and in the custom box put <strong>37</strong> (the code for hippocampus_L). Then select <strong>Ok</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d8lsy9g3dT1ZBqNPBZPMYQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tBhk6wEj693xBN0KKTuMQw.png" /></figure><h3>Mango</h3><p>I am currently in discussion with the Howard laboratory trying to get the exact methodology they used to create the image in their figure, but this how I believe they did it with [<a href="http://ric.uthscsa.edu/mango/download.html">Mango</a>].</p><p>Mango has good <a href="http://ric.uthscsa.edu/mango/videos.html">video tutorials</a>, <a href="http://rii.uthscsa.edu/mango/userguide.html">user guide</a> and <a href="http://rii.uthscsa.edu/mango/forum/">forum</a> but for those of us whose sole interest is creating a highlighted 3D brain image it’s a lot of material to go through. Therefore, I decided to create a detailed protocol of this process to save others time.</p><p>I decided to download the <a href="https://figshare.com/articles/Hammers_mith-n30r95_parietal_lobe_MRI_atlas_database/5024939">Hammers brain atlas</a> to use as the mask/overlay and the sample image provided with Mango</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RRKOGI37hIS3NHrfnbCNEA.png" /></figure><p>Next select Add Overlay and choose the hippocampus_L.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bNXOHHgXGgvcildS8EhOJg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PF4ycLtE5qxnDGRwe7aD-w.png" /></figure><p>Now select Image &gt; Build Surface to create a 3D representation of the brain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pjGJVJ0iGIqUfKjWWJuiGA.png" /></figure><p>In this new pop-up GUI there’s a few things we want to do. First, change the background to white so this can be published in a manuscript. Second, change the transparency of this image to <strong>0.75</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vm2yBb0xkfTewc2LPjcQ6A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WyWDETKfaT4NItcUslkAGg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*19CUugK1Iw6JixB3bCDh6g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yXJINXCF1Qx-vC7msWJFoA.png" /></figure><p>Under the View tab deselect the Crosshairs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8fs23uqjbryzrZBWb9elKA.png" /></figure><p>In the other panel select Analysis &gt; Create Logical Overlays then in the surface panel select Shapes &gt; Add Logical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f0QqB_NamVkm3zEteDurZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4LmvIqv4B5_5pm-yXH18Ig.png" /></figure><p>Then under Surface &gt; Views you can select any orientation you like then Surface &gt; Create Snapshot to save as a .png.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YVm8LjXd0SdO5g5IrPEQDQ.png" /></figure><p>Here are three views of the hippocampus_L: <strong>anterior</strong>, <strong>left </strong>and <strong>superior</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*5LHvZKZVLmiT7sXgl3YGAg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*tkhpUjZuhwbYY7C7l85S9Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*UzC-ngB8G7NoaOj5Bz8z0A.png" /></figure><h3>R Implementation</h3><p><a href="http://(https//www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwi4vIOS9K7eAhWnxFkKHR0OAh8QFjAAegQICRAB&amp;url=http%3A%2F%2Fjohnmuschelli.com%2F&amp;usg=AOvVaw3E8tutT1D7T-WIhUqDHGzM">John Muschelli</a><a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwi4vIOS9K7eAhWnxFkKHR0OAh8QFjAAegQICRAB&amp;url=http%3A%2F%2Fjohnmuschelli.com%2F&amp;usg=AOvVaw3E8tutT1D7T-WIhUqDHGzM">,</a> an Assistant Scientist at Johns Hopkins Bloomberg School of Public Health who has authored numerous R packages (<em>e.g.</em> fslr) responded a couple weeks later to my tweet. He whipped up a gist to highlight a 3D brain image in R.</p><pre>library(rgl)<br>library(misc3d)<br>library(neurobase)<br>if (!requireNamespace(&quot;aal&quot;)) {<br>  devtools::install_github(&quot;muschellij2/aal&quot;)<br>} else {<br>  library(aal)<br>}<br>if (!requireNamespace(&quot;MNITemplate&quot;)) {<br>  devtools::install_github(&quot;jfortin1/MNITemplate&quot;)<br>} else {<br>  library(MNITemplate)<br>}</pre><pre>img = aal_image()<br>template = readMNI(res = &quot;2mm&quot;)<br>cut &lt;- 4500<br>dtemp &lt;- dim(template)</pre><pre># All of the sections you can label<br>labs = aal_get_labels()</pre><pre># Pick the region of the brain you would like to highlight - in this case the hippocamus_L<br>hippocampus = labs$index[grep(&quot;Hippocampus_L&quot;, labs$name)]</pre><pre>mask = remake_img(vec = img %in% hippocampus, img = img)</pre><pre>### this would be the ``activation&#39;&#39; or surface you want to render <br>contour3d(template, x=1:dtemp[1], y=1:dtemp[2], z=1:dtemp[3], level = cut, alpha = 0.1, draw = TRUE)<br>contour3d(mask, level = c(0.5), alpha = c(0.5), add = TRUE, color=c(&quot;red&quot;) )<br>### add text<br>text3d(x=dtemp[1]/2, y=dtemp[2]/2, z = dtemp[3]*0.98, text=&quot;Top&quot;)<br>text3d(x=-0.98, y=dtemp[2]/2, z = dtemp[3]/2, text=&quot;Right&quot;)<br>rglwidget()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*O_A5iw2_diaJXXehO6-PWA.png" /></figure><p>John Muschelli also teaches a couple of short courses on imaging in R and I’m currently taking his [<a href="https://www.coursera.org/learn/neurohacking">Neurohacking</a>] course. I really appreciate him taking the time to do this.</p><p>If your interested in learning more about medical imaging in R be sure to check out <a href="https://www.ncbi.nlm.nih.gov/pubmed/29325029">Neuroconductor</a></p><p>If you find this article useful feel free to share it with others or recommend this article! 😃</p><p>As always, if you have any questions or comments feel free to leave your feedback below or you can always reach me on <a href="https://www.linkedin.com/in/matthewoldach/">LinkedIn</a>. Till then, see you in the next post! 😄</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e6c15a35574" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-highlight-3d-brain-regions-2e6c15a35574">How to Highlight 3D Brain Regions</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The “Gold Standard” of Data Science Project Management]]></title>
            <link>https://medium.com/data-science/the-gold-standard-of-data-science-project-management-13d68c9e85d6?source=rss-459e62b88a2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/13d68c9e85d6</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Matt.0]]></dc:creator>
            <pubDate>Sat, 06 Oct 2018 23:52:15 GMT</pubDate>
            <atom:updated>2018-10-20T22:08:08.690Z</atom:updated>
            <content:encoded><![CDATA[<h3>The “Gold Standard” for Data Science Project Management</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G3Pxy90SG5jZHuc8hZcybQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/a2VqhP3d4Vg?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">rawpixel</a> on <a href="https://unsplash.com/search/photos/big-data?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>The inspiration for this post came most recently from a <a href="https://divingintogeneticsandgenomics.rbind.io/talk/2018-uf-talk/">slide-deck</a> by <a href="https://twitter.com/tangming2005">Ming Tang</a>, a Bioinformatician at Harvard, and a new <a href="https://leanpub.com/universities/courses/jhu/cbds-organizing">Chromebook Data Science course</a> offered by <a href="https://twitter.com/jtleek">Jeffery Leek</a> from John Hopkins University. Leek considers it important enough to suggest that you “<strong>budget 10–20% of the time you will be working on a data science project just to organizing and documenting your work.”</strong></p><p>However, this has been a topic I’ve been thinking about for some time. A number of other great resources I’d read in the past (<a href="http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1000424">1</a>,<a href="https://nicercode.github.io/blog/2013-04-05-projects/">2</a>,<a href="http://lab.loman.net/2014/05/14/how-i-start-a-bioinformatics-project/">3</a>,<a href="https://www.biostarhandbook.com/">4</a>) inspired me to create a <a href="https://github.com/moldach/project-directory">Github repo</a> for a <strong>Gold Standard </strong>workflow for setting up a new data science project directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H0UYSwR2ku3I5Y9FPLi40g.png" /></figure><p><a href="https://twitter.com/JennyBryan?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor">Jenny Bryan,</a> a statistician and data-scientist from UBC, who has strong views on the <a href="https://www.youtube.com/watch?v=7oyiPBjLAWY">proper layout of R scripts</a>, <a href="https://speakerdeck.com/jennybc/zen-and-the-art-of-workflow-maintenance">workflows</a> and file organization/naming, said it best:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hj2-9NO6gxK8SMaepNlPHQ.jpeg" /><figcaption><a href="https://twitter.com/dguarch/status/663049353007931392">Image Source</a></figcaption></figure><p>Leek has talked about the Duke scandal (along with <a href="http://www.biostat.jhsph.edu/~rpeng/">Roger Peng</a> and <a href="http://rafalab.github.io//">Rafa Irizarry</a>) on the <a href="https://simplystatistics.org/2012/02/27/the-duke-saga-starter-set/">simplystats</a> blog in the past so it wasn’t surprising that it also made it into his course. For those of you who don’t want to watch the presentation “<a href="http://videolectures.net/cancerbioinformatics2010_baggerly_irrh/">The Importance of Reproducible Research in High-Throughput Biology: Case Studies in Forensic Bioinformatics</a>” I’ll give the <a href="https://en.wikipedia.org/wiki/Coles_Notes">Coles Notes </a>version (but I warn you, you’re missing out!).</p><p>Researcher’s from Duke university published a study on using personalized-genomics for patient-specific chemotherapy treatment. Two researchers, <a href="https://www.nytimes.com/2011/07/08/health/research/08genes.html">Baggerly and Coombes</a>, from MD Anderson requested the data and code. It took many months of back-and-forth (<em>hounding</em>) to actually get the data and code — when they did it was disorganized and poorly documented. Baggerly and Coombes eventually found an error in the code, showing it actually put patients at risk, leading to a major lawsuit and terminated clinical trials.</p><p>As an aside, the reluctance to share data is, sadly, still prevalent today. Case in point, the now infamous <a href="https://www.nejm.org/doi/full/10.1056/NEJMe1516564">“Research Parasite” editorial</a> published by the New England Journal of Medicine, a leading journal in the field. I’ve also included a screenshot from Stodden <em>et al., </em>2018 “<a href="http://www.pnas.org/content/115/11/2584">An empirical analysis of journal policy effectiveness for computational reproducibility</a> ”. The authors took a random sample of 204 scientific papers published in the journal <em>Science</em> after the implementation of their policy which“ <strong>requires the data and code necessary for reproducibility be made available postpublication by the authors upon request” </strong>in February 2011. Of these 204 papers 24 had sufficient information (via links or in the supporting information) for them to locate the data/code without contacting the authors. Of those that were contacted, and responded, some were either unsure of <em>Science’s </em>policy or unwillingly to comply:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*JUARL4oLZPrpULemtyndpQ.png" /><figcaption>Special thanks to <a href="https://twitter.com/LuisDVerde">Luis D. Verde</a> for reminding me of the <a href="http://www.pnas.org/content/115/11/2584">Image Source!</a></figcaption></figure><p>Stodden and collaborators got data and code from 89 out of 204 articles and randomly chose 22 to attempt replication in (all but one worked). They used that to estimate that 26% would reproduce of these 89 articles (56*(1-1/22)) ) with 95% CI [20%,32%] — the authors suggests that we continue the conversation on best-practices and standards for documenting code, metadata , workflows and use and reuse of software (<em>e.g. </em>Docker containers).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*G0gBh0xZ1ohVfDSGUFCiGw.png" /><figcaption>Source: <a href="https://twitter.com/ProfJohnATodd/status/1053471430149435392">https://twitter.com/ProfJohnATodd/status/1053471430149435392</a></figcaption></figure><p>I know that some scientists will skim the abstract of a manuscript first but my stamp-of-approval is a <strong>Reproducibility Statement </strong>with all of the (<em>r</em>a<em>w</em>) data and code available. If you don’t see something like this RUN!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*9K-IBNKS42lR3LOuNKlAow.jpeg" /></figure><p>Failure to make your project reproducible is <strong>academic misconduct </strong>and can have serious repercussions. It was one of the charges (“<strong>failure to properly document and preserve research results</strong>”), among many others, for the recently <a href="https://www.cnn.com/2018/09/20/health/brian-wansink-retractions-resignation-misconduct-bn/index.html">disgraced Cornell researcher Brian Wansink</a> (I’m not saying it’s worse than the p-hacking allegations but I refuse to say it’s any better either).</p><p>A <a href="http://opensourcesurvey.org/2017/">2017 Github survey</a> showed that “Incomplete or Confusing Documentation” is the top complaint about open source software.</p><p>This <a href="https://www.divio.com/blog/documentation/"><em>excellent </em>post on a <strong>Gold Standard </strong>for software documentation</a> by Daniele Procida sums it up nicely when he says:</p><blockquote>“It doesn’t matter how good your software is, because <strong>if the documentation is not good enough, people will not use it</strong>.</blockquote><blockquote>Even if for some reason they have to use it because they have no choice, without good documentation, they won’t use it effectively or the way you’d like them to.”</blockquote><p>So follow the sage advice of Mr. Procida and make it effortless for end-users to understand what you did in your project and be able to reproduce those results (preferably without having to contact you). It’s essential for collaborating with colleagues in the present and also for posterity’s sake (<em>e.</em>g. your future-self when you’re asked to re-run an analysis 6-months down the road or for any researchers wishing to reproduce your work and expand upon it).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*7q7YGtf9M9yt7Ny6Owf3oA.png" /><figcaption>Source: <a href="https://opensource.googleblog.com/2018/10/building-great-open-source-documentation.html">https://opensource.googleblog.com/2018/10/building-great-open-source-documentation.html</a></figcaption></figure><p>Since my <a href="https://github.com/moldach/project-directory">Github repo</a> already addresses the <strong>Gold Standard </strong>for setting up folders in your data science project (*please star the repo as I plan on updating and improving it in the coming days*) I’ll talk about another important aspect of data science project management, which is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*7HHA_UkjUK7wp7qP4CYu1g.png" /></figure><h3>File Naming</h3><p>Jenny Bryan gives three key principles of file naming for data science projects.</p><ol><li>Machine readable</li><li>Human readable</li><li>Plays well with default ordering</li></ol><p>For machine readability we want to <strong>avoid spaces, punctuation, periods and any other special characters (except _ and -)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*M-nHs3nLYph0zpMScavLRw.png" /></figure><p>For human readability you want to give files meaningful names. When naming R objects there is a tendency to abbreviate object names, this is okay as long as you include a comment. For example, that the cv_perf_recall_rf object is the calculation for the validate recall of each cross-validation fold of a random forest model.</p><pre># Calculate the validate recall for each cross-validation fold<br>cv_perf_recall_rf &lt;- cv_prep_rf %&gt;%<br> mutate(validate_recall = map2_dbl(validate_actual, validate_predicted, ~recall(actual = .x, predicted = .y)))</pre><p>However, when naming files I would caution against using acronyms unless absolutely necessary and, if so, including that information in the README file.</p><p>The next piece of advice is to place dates and number in the beginning of the file name. Always use the ISO8601 date format (yyyy-mm-dd) and left-pad numbers with zeros. The maximum number of digits will be determined by how many files you may generate. Say you expected to save 100 structural MRI image files then it should look like this 001_T1_mri.nii.gz. Say you thought you would actually generate 1000 files then it would look like this 0025_T1_mri.nii.gz.</p><h3>Taking A Contrarian Stance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JRJe_ZSoYz9tBDrVuYaFRg.jpeg" /></figure><p>Leek also says you should avoid case sensitivity, for example Esophageal-Cancer_Report.md is obviously a horrible filename (my fingers hurt from all the extra key-strokes 😫) but suggest that esophageal-cancer_report.md is superior.</p><p>Respectfully I have to disagree with him here. Personally I find <a href="https://en.wikipedia.org/wiki/Camel_case">camelCase</a> aesthetically pleasing and esophagealCancer_report.md looks much more pleasant to me and it doesn’t come with the risks that Leek mentions; as long as you don’t forget to include the proper options to ignore case sensitivity. In R you would do this with the stringr package,</p><pre>library(stringr)<br>str_detect(&#39;TOYOTA subaru&#39;, fixed(&#39;toyota&#39;, ignore_case=TRUE)) </pre><p>In the terminal you would use the -iname flag to ignore case insensitivity with the find command. If you’re forgetful, or just efficient (<em>i.</em>e. lazy), you could always just include this as an alias in your .bashrc file 🤷</p><p>Making your file name start with a capital letter is obviously a bad idea as it causes you to add additional key strokes to generate the capital letter (<em>e.g.</em> Shift + &lt;a-z&gt;). However, with camelCase you can avoid extra key strokes by using the Tab button for autocomplete.</p><h3>Closing Note</h3><p>If you use R you should read up on <a href="https://twitter.com/krlmlr">Kirill Müller</a>’s here() package which does away with awkward workflow problems that can be caused by setwd().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*fmEQAQ7H_cqAzwOPCIhv8w.png" /></figure><p>Jenny Bryan is a big promoter of using his package and has written on her <a href="https://www.tidyverse.org/articles/2017/12/workflow-vs-script/">blog post “Project-oriented workflow”</a> about the <em>how</em> and <em>why</em>.</p><p>Follow this <strong>Gold Standard </strong>data science project management advice and you’ll have no problem dealing with “Big Data” 🙄</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*pmgEEeBpoWaTOXWTLQycmQ.png" /></figure><p>If you find this article useful feel free to share it with others or recommend this article! 😃</p><p>As always, if you have any questions or comments feel free to leave your feedback below or you can always reach me on <a href="https://www.linkedin.com/in/matthewoldach/">LinkedIn</a>. Till then, see you in the next post! 😄</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13d68c9e85d6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-gold-standard-of-data-science-project-management-13d68c9e85d6">The “Gold Standard” of Data Science Project Management</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recreating (more) data visualizations from the book “Knowledge is Beautiful”: Part IV]]></title>
            <link>https://medium.com/data-science/recreating-more-data-visualizations-from-the-book-knowledge-is-beautiful-part-iv-686938a84c9e?source=rss-459e62b88a2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/686938a84c9e</guid>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[dplyr]]></category>
            <category><![CDATA[ggplot2]]></category>
            <category><![CDATA[tidyverse]]></category>
            <dc:creator><![CDATA[Matt.0]]></dc:creator>
            <pubDate>Mon, 16 Jul 2018 23:06:03 GMT</pubDate>
            <atom:updated>2018-09-06T01:07:37.354Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to the last part of the series where I recreate data visualizations in R from the book <em>Knowledge is Beautiful </em>by David McCandless.</p><p>Links to part <a href="https://towardsdatascience.com/recreating-data-visualizations-from-the-book-knowledge-is-beautiful-e455e7126071">I</a>, <a href="https://medium.com/@MattOldach_65321/recreating-more-data-visualizations-from-the-book-knowledge-is-beautiful-part-ii-2f1d1da68e3a">II</a>, <a href="https://towardsdatascience.com/recreating-more-data-visualizations-from-the-book-knowledge-is-beautiful-part-iii-5fd393b4c548">III</a> of the series can be found here.</p><h3>Plane Crashes</h3><p>This<strong> </strong><a href="https://docs.google.com/spreadsheets/d/1RT25s5oCsHJrahZkL6KWxAa2ZM9kD2aRakN9Cil_vHQ/edit#gid=11">dataset</a> will be used for a couple of visualizations.</p><p>The first visualization is a stacked-barplot showing causes of crashes for every plane crash from 1993 to January 2017 (<em>for flights that were not military, medical or a private chartered flight</em>).</p><pre>library(dplyr)<br>library(ggplot2)<br>library(tidyr)<br>library(extrafont)</pre><pre>df &lt;- read.csv(&quot;worst_plane.csv&quot;)</pre><pre># Drop the year plane model entered service<br>mini_df &lt;- df %&gt;% <br>  select(-year_service) %&gt;% <br># Gather the wide dataframe into a tidy format<br>  gather(key = cause, value = proportion, -plane)<br># Order by cause<br>mini_df$cause &lt;- factor(mini_df$cause, levels = c(&quot;human_error&quot;,&quot;weather&quot;, &quot;mechanical&quot;, &quot;unknown&quot;, &quot;criminal&quot;), ordered = TRUE)</pre><pre># Create vector of plane names according to year they entered service<br>names &lt;- unique(mini_df$plane)<br>names &lt;- as.vector(names)<br># sort by factor<br>mini_df$plane &lt;- factor(mini_df$plane, levels = names)</pre><pre>ggplot(mini_df, aes(x=plane, y=proportion, fill=cause)) +<br>  geom_bar(stat = &quot;identity&quot;) +<br>  coord_flip() +<br>  # Reverse the order of a categorical axis<br>  scale_x_discrete(limits = rev(levels(mini_df$plane))) +<br>  # Select manual colors that McCandless used<br>  scale_fill_manual(values = c(&quot;#8E5A7E&quot;, &quot;#A3BEC7&quot;, &quot;#E1BD81&quot;, &quot;#E9E4E0&quot;, &quot;#74756F&quot;), labels = c(&quot;Human Error&quot;, &quot;Weather&quot;, &quot;Mechanical&quot;, &quot;Unknown&quot;, &quot;Criminal&quot;)) +<br>  labs(title = &quot;Worst Planes&quot;, caption = &quot;Source: bit.ly/KIB_PlaneCrashes&quot;) +<br>  scale_y_reverse() +<br>  theme(legend.position = &quot;right&quot;,<br>      panel.background = element_blank(),<br>        plot.title = element_text(size = 13, <br>                                  family = &quot;Georgia&quot;, <br>                                  face = &quot;bold&quot;, lineheight = 1.2),<br>      plot.caption = element_text(size = 5,<br>                                    hjust = 0.99, family = &quot;Georgia&quot;),<br>      axis.text = element_text(family = &quot;Georgia&quot;), <br>      # Get rid of the x axis text/title <br>      axis.text.x=element_blank(),<br>      axis.title.x=element_blank(),<br>      # and y axis title<br>      axis.title.y=element_blank(),<br>      # and legend title<br>      legend.title = element_blank(),<br>      legend.text = element_text(family = &quot;Georgia&quot;),<br>      axis.ticks = element_blank())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KPsPC1uJkhlk_QMLbmaLzQ.jpeg" /></figure><p>The second visualization is an alluvial diagram for which we can use the ggalluvial package. I should mention that the original visualization by McCandless is much fancier than what this produces but displays the same basic information.</p><pre>library(alluvial)<br>library(ggalluvial)</pre><pre>crash &lt;- read.csv(&quot;crashes_alluvial.csv&quot;)</pre><pre># stratum = cause, alluvium = freq</pre><pre>ggplot(crash, aes(weight = freq,<br>                  axis1 = phase,<br>                  axis2 = cause,<br>                  axis3 = total_crashes)) +<br>  geom_alluvium(aes(fill = cause),<br>                width = 0, knot.pos = 0, reverse = FALSE) +<br>  guides(fill = FALSE) +<br>  geom_stratum(width = 1/8, reverse = FALSE) +<br>  geom_text(stat = &quot;stratum&quot;, label.strata = TRUE, reverse = FALSE, size = 2.5) +<br>  scale_x_continuous(breaks = 1:3, labels = c(&quot;phase&quot;, &quot;causes&quot;, &quot;total crashes&quot;)) +<br>  coord_flip() +<br>  labs(title = &quot;Crash Cause&quot;, caption = &quot;Source: bit.ly/KIB_PlaneCrashes&quot;) +<br>  theme(panel.background = element_blank(),<br>        plot.title = element_text(size = 13, <br>                                  family = &quot;Georgia&quot;, <br>                                  face = &quot;bold&quot;,<br>                                  lineheight = 1.2,<br>                                  vjust = -3,<br>                                  hjust = 0.05), <br>        plot.caption = element_text(size = 5,<br>                                    hjust = 0.99, family = &quot;Georgia&quot;),  <br>        axis.text = element_text(family = &quot;Georgia&quot;),<br>      axis.text.x = element_blank(),<br>      axis.ticks.x = element_blank(),<br>      axis.ticks.y = element_blank())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZNrVJ0dWFgt_Maw2fr84MA.png" /></figure><p>In my opinion, a cleaner way to produce a Sankey diagram is with the package flipPlot<a href="https://www.r-bloggers.com/how-to-create-sankey-diagrams-from-tables-data-frames-using-r/">package</a>.</p><pre>devtools::install_github(Displayr/flipPlots)<br>library(flipPlots)</pre><pre># reorder the df<br>crash$total_crashes &lt;- rep(&quot;YES&quot;, 25)</pre><pre># plot<br>SankeyDiagram(crash[, -4], link.color = &quot;Source&quot;, label.show.varname = FALSE, weights = crash$freq)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O_sXWaHy2-i-n3yG5jbAuA.jpeg" /></figure><p>I would probably just rotate the image and labels and change the yes to “Total Crashes (427)”</p><h3>Gender Gap</h3><p>This visualization depicts the salary gap between males and females by industry in the UK with the mean salary of each position within a category. We can use group_by() and summarize_at() to create a new variable for each category and then use facet_wrap() . Since positions only belong to one category you need to set scales = &quot;free_x&quot; for missing observations.</p><pre>gendergap &lt;- read.csv(&quot;gendergap.csv&quot;)</pre><pre># gather the dataset<br>tidy_gap &lt;- gendergap %&gt;% <br>  gather(key = sex, value = salary, -title, -category)</pre><pre>category_means &lt;- tidy_gap %&gt;% <br>  group_by(category) %&gt;%<br>  summarize_at(vars(salary), mean)</pre><pre>tidy_gap %&gt;% ggplot(aes(x = title, y = salary, color = sex)) +<br>  facet_grid(. ~ category, scales = &quot;free_x&quot;, space = &quot;free&quot;) +<br>  geom_line(color = &quot;white&quot;) +<br>  geom_point() +<br>  scale_color_manual(values = c(&quot;#F49171&quot;, &quot;#81C19C&quot;)) +<br>  geom_hline(data = category_means, aes(yintercept = salary), color = &quot;white&quot;, alpha = 0.6, size = 1) +<br>  theme(legend.position = &quot;none&quot;,<br>      panel.background = element_rect(color = &quot;#242B47&quot;, fill = &quot;#242B47&quot;),<br>      plot.background = element_rect(color = &quot;#242B47&quot;, fill = &quot;#242B47&quot;),<br>      axis.line = element_line(color = &quot;grey48&quot;, size = 0.05, linetype = &quot;dotted&quot;),<br>      axis.text = element_text(family = &quot;Georgia&quot;, color = &quot;white&quot;),<br>      axis.text.x = element_text(angle = 90),<br>      # Get rid of the y- and x-axis titles<br>      axis.title.y=element_blank(),<br>      axis.title.x=element_blank(),<br>      panel.grid.major.y = element_line(color = &quot;grey48&quot;, size = 0.05),<br>      panel.grid.minor.y = element_blank(),<br>      panel.grid.major.x = element_blank(),<br>      strip.background = element_rect(color = &quot;#242B47&quot;, fill = &quot;#242B47&quot;),<br>      strip.text = element_text(color = &quot;white&quot;, family = &quot;Georgia&quot;))<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lA1b-cn7LRzgPIOaPEG5vQ.png" /></figure><p>One thing that I’m not sure how to handle is the spacing between each of the variables on the x-axis. Since there is a different number of variables for each facet it would be nice if one could specify they want equal spacing along the x-axis as an option in the facet_wrap(); however, I don’t think it’s possible (if you know a workaround please leave a comment!).</p><p>## EDIT SEPTEMBER 5 2018</p><p>Thanks to help from the ever so <a href="https://stackoverflow.com/questions/51052566/dealing-with-variable-axis-scales-in-facet-wrap?noredirect=1#comment91218664_51052566">helpful community at StackOverflow</a> I learned it was facet_grid() and not facet_wrap() that I needed. Bonus points if anyone figures out how to manually adjust the distance between each of the categories in the x-axis (<em>e.</em>g. minimize the spacing).</p><p>That’s all for me, it’s been fun doing this series and I hope you’ve enjoyed!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=686938a84c9e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/recreating-more-data-visualizations-from-the-book-knowledge-is-beautiful-part-iv-686938a84c9e">Recreating (more) data visualizations from the book “Knowledge is Beautiful”: Part IV</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recreating (more) data visualizations from the book “Knowledge is Beautiful”: Part III]]></title>
            <link>https://medium.com/data-science/recreating-more-data-visualizations-from-the-book-knowledge-is-beautiful-part-iii-5fd393b4c548?source=rss-459e62b88a2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fd393b4c548</guid>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[ggplot2]]></category>
            <category><![CDATA[dplyr]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Matt.0]]></dc:creator>
            <pubDate>Thu, 05 Jul 2018 21:54:43 GMT</pubDate>
            <atom:updated>2018-10-06T22:00:44.011Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to the third installment of the series where I recreate data visualizations, in R, from the book <a href="https://informationisbeautiful.net/2014/knowledge-is-beautiful/"><em>Knowledge is Beautiful</em></a> by David McCandless.</p><p>Links to part <a href="https://towardsdatascience.com/recreating-data-visualizations-from-the-book-knowledge-is-beautiful-e455e7126071">I</a>, <a href="https://medium.com/@MattOldach_65321/recreating-more-data-visualizations-from-the-book-knowledge-is-beautiful-part-ii-2f1d1da68e3a">II</a>, <a href="https://towardsdatascience.com/recreating-more-data-visualizations-from-the-book-knowledge-is-beautiful-part-iv-686938a84c9e">IV</a> of the series can be found here.</p><p>The list of frustrations in data science are many, for example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*OY88qBQvf5Ir5jPUl1K1GA.png" /></figure><p>Consider point 4 above. Even when the data is made available, it can still be distributed in frustrating formats. In part I of the series I showed how to access a specific excel sheet with readxlpackage . In part II I showed how to parse PDF tables in R with the tabulizer package. It might be of interest to some that, <a href="https://twitter.com/LuisDVerde">Luis D. Verde</a> made a great post recently on <a href="https://luisdva.github.io/rstats/excel-trivia/">how to deal with frustrating cell formatting in Excel in R</a>.</p><p>Although McCandless made all the data public it took a bit of cleaning on my part before producing the visualizations. I decided that since this series is focused on data visualization I would leave the data munging code out for the rest of the series and instead provided .csv files of tidy data ready for ggplot.</p><h3>Live Long</h3><p>The live long visualization is a diverging bar chart depicting how certain actions affect your life span.</p><pre>library(dplyr)<br>library(ggplot2)</pre><pre># load the data<br>livelong &lt;- read.csv(&quot;livelong.csv&quot;)</pre><pre># Order by action<br>livelong$action &lt;- factor(livelong$action, levels = c(&quot;Sleep too much&quot;, &quot;Be optimistic&quot;, &quot;Get promoted&quot;, &quot;Live in a city&quot;, &quot;Live in the country&quot;, &quot;Eat less food&quot;, &quot;Hang out with women - a lot!&quot;, &quot;Drink a little alcohol&quot;, &quot;Be conscientious&quot;, &quot;Have more orgasms&quot;, &quot;And a little red wine&quot;, &quot;With close friends&quot;, &quot;Be polygamous, maybe&quot;, &quot;Go to church regularly&quot;, &quot;Sit down&quot;, &quot;More pets&quot;, &quot;Eat red meat&quot;, &quot;Avoid cancer&quot;, &quot;Avoid heart disease&quot;, &quot;Be alcoholic&quot;, &quot;Get health checks&quot;, &quot;Get married!&quot;, &quot;Be rich&quot;, &quot;Be a woman&quot;, &quot;Suffer severe mental illness&quot;, &quot;Become obese&quot;, &quot;Keep smoking&quot;, &quot;Live healthily&quot;, &quot;Exercise more&quot;, &quot;Live at high altitude&quot;))</pre><pre># Set legend title<br>legend_title &lt;- &quot;Strength of science&quot;</pre><pre># Make plot<br>p &lt;- ggplot(livelong, aes(x = action, y = years, fill=strength)) +<br>  geom_bar(stat = &quot;identity&quot;) +<br>  scale_fill_manual(legend_title, values = c(&quot;#8BC7AC&quot;,&quot;#D99E50&quot;,&quot;#CDAD35&quot;)) +<br>  labs(title = &quot;Live Long...&quot;, subtitle = &quot;What will really extend your life?&quot;, caption = &quot;Source: bit.ly/KIB_LiveLong&quot;) +<br>  scale_y_continuous(position = &quot;bottom&quot;) +<br>  scale_x_discrete(limits = rev(factor(livelong$action))) +<br>  #scale_x_reverse() +<br>  coord_flip() +<br>  theme(legend.position = &quot;top&quot;,<br>      panel.background = element_blank(),<br>        plot.title = element_text(size = 13, <br>                                  family = &quot;Georgia&quot;, <br>                                  face = &quot;bold&quot;, lineheight = 1.2), plot.subtitle = element_text(size = 10,<br>                                     family = &quot;Georgia&quot;), <br>        plot.caption = element_text(size = 5,<br>                                    hjust = 0.99, family = &quot;Georgia&quot;),  <br>        axis.text = element_text(family = &quot;Georgia&quot;),<br>      # Get rid of the y- and x-axis titles<br>      axis.title.y=element_blank(),<br>      axis.title.x=element_blank(),<br>      # Get rid of axis text<br>      axis.text.y = element_blank(),<br>      axis.ticks.y = element_blank(),<br>      legend.text = element_text(size = 8, family = &quot;Georgia&quot;),<br>      legend.title = element_text(size = 8, family = &quot;Georgia&quot;),<br>      legend.key.size = unit(1,&quot;line&quot;))</pre><p>Okay, here is the first attempt at annotating with geom_text()</p><pre>p + geom_text(aes(label = action), size = 3, family = &quot;Georgia&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b6wB1iQBdQfx2_OgGdDZDw.jpeg" /></figure><p>One instantly notices that the text annotation is misaligned. Furthermore, since the bar chart is diverging at the center (zero) and modifications to the hjust parameter won’t solve the problem on it’s own.</p><p>One possible work-around is to use the ggfittext package which constrains text inside a defined area with the geom_fit_text() function that works more or less like ggplot2::geom_text().</p><pre># currently only supported by the dev version<br>devtools::install_github(&quot;wilkox/ggfittext&quot;)<br>library(ggfittext)</pre><pre>p + geom_fit_text(aes(label = action), position = &quot;stack&quot;, family = &quot;Georgia&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ifbyTXHZcScjxly06cALsw.jpeg" /></figure><p>We see that small bars were not annotated because the character strings are simply too big to be displayed in the bars.</p><p>The original visualization never constrains the text to the bars so the best approach is to add a variable to the table that will allow you to left-justify some labels and right-justify others.</p><pre># Set postive as &quot;Up&quot; and negative numbers as &quot;Down&quot;<br>livelong$direction &lt;- ifelse(livelong$years &gt; 0, &quot;Up&quot;, &quot;Down&quot;)</pre><pre>livelong$just &lt;- ifelse(livelong$direction==&quot;Down&quot;,0,1)</pre><pre>p + geom_text(aes(label = action), size = 3, family = &quot;Georgia&quot;, hjust=livelong$just)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q1PyAOjEgo4jBYMvmg6APQ.jpeg" /></figure><p>This justifies the boxes so that actions which decrease your lifespan are left-adjust and those which extend your life are right-adjusted. The only problem is that in the original visualization the text lines up on the center of the chart. I couldn’t figure out how to do that so bonus points if you can figure that out and post it to the comments!</p><p>A visually appealing alternative is to have the names outside of the bars.</p><pre>livelong$just &lt;- ifelse(livelong$direction==&quot;Up&quot;,0,1)</pre><pre>p + geom_text(aes(label = action), size = 3, family = &quot;Georgia&quot;, hjust=livelong$just)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*19V9FL2OF3d5hsAwH1ieZw.jpeg" /></figure><h3>Counting the Cause UK</h3><p>The <strong>Counting the Cause UK</strong> <a href="https://docs.google.com/spreadsheets/d/16q9OGYALTfyAwXDuH46UdCK2TJPcabo5U0UBKXaVgfQ/edit#gid=4">dataset</a> shows what charities UK citizens donate most to.</p><p>We can create a comparable visualization using the Treemap library. It creates a hierarchical display of nested rectangles, which is then tiled within smaller rectangles representing sub-branches.</p><pre>library(treemap)</pre><pre>my_data &lt;- read.csv(&quot;treemap.csv&quot;)</pre><pre>tm &lt;- treemap(my_data, index = c(&quot;main&quot;,&quot;second&quot;, &quot;third&quot;), vSize = &quot;percent&quot;, vColor = &quot;percent&quot;, type = &quot;value&quot;, title = &quot;Counting the Cause UK&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L1rBEvhl_IvfNcXP_MCyHw.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fd393b4c548" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/recreating-more-data-visualizations-from-the-book-knowledge-is-beautiful-part-iii-5fd393b4c548">Recreating (more) data visualizations from the book “Knowledge is Beautiful”: Part III</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>