<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Serdar Korur on Medium]]></title>
        <description><![CDATA[Stories by Serdar Korur on Medium]]></description>
        <link>https://medium.com/@serdarkorur?source=rss-70bf243827fc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*L4OvBxj2y_f89QrhGrvddQ.jpeg</url>
            <title>Stories by Serdar Korur on Medium</title>
            <link>https://medium.com/@serdarkorur?source=rss-70bf243827fc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 18:14:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@serdarkorur/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An accidental side effect of text mining]]></title>
            <link>https://medium.com/data-science/an-accidental-side-effect-of-text-mining-4b43f8ee1273?source=rss-70bf243827fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b43f8ee1273</guid>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[data-storytelling]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[text-mining]]></category>
            <category><![CDATA[r]]></category>
            <dc:creator><![CDATA[Serdar Korur]]></dc:creator>
            <pubDate>Thu, 12 Dec 2019 22:28:20 GMT</pubDate>
            <atom:updated>2019-12-17T09:57:23.720Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*8O1lWQkX05rv3vPJ00FBqg.png" /></figure><h4>Does text mining really work? A first-hand test.</h4><blockquote>“The end of our exploring will be to arrive at where we started, and to know the place for the first time.”</blockquote><blockquote>— T. S. Eliot</blockquote><p>I was reading lately lots of books on productivity and self-development and often I found pieces of advice that I want to re-read later. The highlighting option on kindle makes this very easy. By reading and highlighting consistently <strong>I accumulated a sizeable amount text which was a good representation of the books I read.</strong></p><p>Since I know the content very well. I want to apply <strong>text mining </strong>and<strong> sentiment analysis</strong> to this data so that I can compare the results to my real opinion about the books. <strong>And if they match I will be more convinced to apply it to my next business problem.</strong></p><p>In part, this is also <strong>a self-exploration</strong>, since I could answer questions such as what kind of content I like more, what was their emotional charge?</p><p><strong>Let’s find the answers.</strong></p><p>To be objective, I will create an independent character called “the bat”. His next mission is to hack my data and analyze it to gather insights for selling me more books. Unfortunately, reading is not his specialty.</p><p>The room was in a mess when he entered. As he plugged the USB drive on his laptop, he barely heard the radio which was still on. The reporter:</p><blockquote>From the moment we switch on the early morning our cell phones to deal with the flood of information in some form of a Whatsapp or Facebook message or a tweet until we fall asleep at night overwriting or reading a product review we leave bread crumbs to our personal flavors on internet.</blockquote><blockquote>Many businesses use this unstructured data to drive their sales by better marketing through targeted product recommendations or to segregate their customers…</blockquote><p>He squeezed his teeth when he saw the data of 21000 lines of text from 28 books. His first encounter was the book <a href="https://www.amazon.com/Mindset-Psychology-Carol-S-Dweck/dp/0345472322">“Mindset”</a> by Carol Dweck, where she introduces the concept of <a href="https://www.mindsetworks.com/science/"><strong>growth mindset</strong>.</a> The idea behind is people who have the growth mindset believe their abilities can be improved by putting effort whereas the people with the fixed mindset believe that their abilities are fixed at birth. As a consequence, people who has the fixed mindset miss the opportunity to get better at many things even though they can. <strong>Simply, because they don’t believe in it in the first place.</strong></p><p>If you are not familiar with the concept, here is a video of Carol Dweck explaining her research on growth mindset.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhiiEeMN7vbQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhiiEeMN7vbQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhiiEeMN7vbQ%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f23d0e0e05cf82662b7d33963afc2761/href">https://medium.com/media/f23d0e0e05cf82662b7d33963afc2761/href</a></iframe><p>Long lines of text made him tired, he didn’t realize how the time passed. Now, he was a man of growth. <strong>He decided to learn text mining.</strong></p><p>He started to learn the R packages for text mining, he didn’t like the package name <a href="https://www.tidytextmining.com/">tidytext</a> but he was slightly losing his prejudices. It was a long night. He fell asleep on his table as the sun slowly rose.</p><p>It was lighting my back garden where I could glance from time to time to the trees painted by the snow overnight ⛄️. Without an idea about how things went on the another part of the town, I continued to read and highlight my kindle as I zip from a glass of red wine 🍷.</p><p>We will know what happened to our hacker later. First, let’s go together through his notebook.</p><p><strong>The hacker’s notes</strong></p><p>This is how the exported kindle highlights look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VHFpWG-jUr297bzgWXe_fQ.png" /></figure><p><strong>Reading and parsing the text file</strong></p><pre># Use readLines function to parse the text file</pre><pre>highlights &lt;- readLines(&quot;Kindle_highlights_Serdar.Rmd&quot;, encoding = &quot;UTF-8&quot;)</pre><pre># Create a dataframe where each row is a line from the text</pre><pre>df &lt;- data.frame(highlights)</pre><pre># Packages<br>library(tidyverse)   <br><em># includes </em><strong><em>ggplot2</em></strong><em>, </em><strong><em>dplyr</em></strong><em>, tidyr, readr, purrr, </em><strong><em>tibble</em></strong><em>, </em><strong><em>stringr</em></strong><em>, forcats</em></pre><pre>library(tidytext)<br>library(wordcloud2)</pre><p>Every data science project requires some sort of data preparation. <strong>Stop words </strong>are generally the most common words in a language and are usually filtered out before processing of text data.</p><p>Let’s look at the <strong>stop_words</strong> dataset from the tidytext package. Since it is a long list of words (&gt;1K) <strong>I will print every fifth word as an example.</strong></p><pre><strong>data(stop_words)</strong></pre><pre># print every 50th word </pre><pre>stop_words_small &lt;- <strong>stop_words[seq(1, nrow(stop_words), 50),]</strong></pre><pre>stop_words_small %&gt;% print(n=50)</pre><pre># A tibble: 23 x 2<br>   word       lexicon <br>   &lt;chr&gt;      &lt;chr&gt;   <br> 1 a          SMART   <br> 2 at         SMART   <br> 3 contain    SMART   <br> 4 few        SMART   <br> 5 hers       SMART   <br> 6 last       SMART   <br> 7 nine       SMART   <br> 8 presumably SMART   <br> 9 some       SMART   <br>10 they&#39;d     SMART   <br>11 very       SMART   <br>12 without    SMART   <br>13 what       snowball<br>14 they&#39;ll    snowball<br>15 during     snowball<br>16 again      onix    <br>17 but        onix    <br>18 finds      onix    <br>19 if         onix    <br>20 much       onix    <br>21 parted     onix    <br>22 since      onix    <br>23 under      onix</pre><p><strong>Looking carefully one can see that stop_words use single quotes whereas in the text data uses apostrophes (‘).</strong></p><pre>e.g. <strong>they&#39;ll  </strong>in stop_words</pre><p>and how the word <strong>they’ll </strong>appears in the text:</p><blockquote>Yellow highlight | Page: 200<br>Memories are continually revised, along with the meaning we derive from them, so that in the future <strong>they’ll</strong> be of even more use.</blockquote><p>We have to make stop_words and our data compatible, otherwise some words such as <strong>they’ll, don’t, can’t might appear in our results.</strong></p><p>We can use<strong> str_replace_all() </strong>function from stringr package to find all the apostrophes and convert them into single quotes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*wGd8zGdlOuKKLP5Ixd8hYA.png" /></figure><p>Now, the text is ready for the frequency analysis. Words in a text mining project are called <strong>tokens.</strong> We can split the text into single words by <strong>unnest_tokens() </strong>function from tidytext package, filter the <strong>stop_words </strong>and count.</p><pre>df$highlights &lt;- str_replace_all(df$highlights, &quot;’&quot;, &quot;&#39;&quot;)</pre><pre>df &lt;- df %&gt;% unnest_tokens(word, highlights) %&gt;%<br>             anti_join(stop_words) %&gt;% <br>             filter(!word %in% c(&quot;highlights&quot;,&quot;highlight&quot;, &quot;page&quot;, <br>                      &quot;location&quot;, &quot;yellow&quot;, &quot;pink&quot;, &quot;orange&quot;, &quot;blue&quot;))</pre><p>He also added here some additional words which frequently appear in kindle highlights output.</p><p>Dplyr() package functions are very useful for grouping and counting the words in a data frame.</p><pre><strong>top_kindle_highlights</strong> &lt;- df %&gt;% <br> group_by(word) %&gt;% <br> count() %&gt;% <br> arrange(desc(n))</pre><p>He noted down his first insight. <strong>10 most frequent words from my kindle highlights.</strong></p><pre><strong>people</strong> 592   <br><strong>story</strong>  340   <br>life   318   <br>time   309   <br><strong>mind </strong>  213   <br><strong>change</strong> 212   <br>feel   211   <br>world  171   <br>person 170   <br><strong>habits</strong> 157   </pre><p>If you don’t like to look at long list of words<strong> wordclouds </strong>are<strong> </strong>a good alternative. Wordcloud2 package gives additional customization options for your wordclouds, for example you can use any image as a markup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AjQEHBc_CDSZlOjX9T9aCg.png" /></figure><p>The code for the wordcloud above is:</p><pre><strong>wordcloud2</strong>(top_kindle_highlights, figPath = bat, size = 1, backgroundColor = &quot;white&quot;, color = color_vector(data$freq) )</pre><p>Some ideas started to appear in his mind. He thought who made those highlights is someone interested in storytelling, writing and good communication, good habits, and people. Someone who wants to influence his life in a positive way. <strong>He was becoming more and more interested in the books. He wanted to dig deeper.</strong></p><p><strong>Bigram Analysis</strong></p><p>Single words are a good starting point what the books were about. <strong>But they are limited without the context. </strong>Frequency analysis can also be performed to measure how often word pairs <strong>(bigrams)</strong> occur in the text. This allows us to capture finer details in the text.</p><p>To do this he combined the unnested single tokens which is isolated above back into a continuous text and then performed bigram analysis. You can use <strong>str_c()</strong> function from stringr package to concatenate the single words.</p><pre>df_com &lt;- <strong>str_c</strong>(df$word, “ “) <br>df_com &lt;- data.frame(df_com)</pre><p>Let’s split the text into bigrams and find the most frequent ones.</p><pre>df_bigram &lt;- df_com %&gt;% <br> <strong>unnest_tokens</strong>(bigram, df_com, token = “ngrams”, <br> n = 3, n_min = 2)</pre><pre><strong>top_bigrams</strong> &lt;- df_bigram %&gt;% <br> group_by(bigram) %&gt;% <br> count() %&gt;% <br> arrange(desc(n))%&gt;% <br> print(n=20)</pre><p>And visualize them on a plot</p><pre>top &lt;- top_bigrams[1:25,]<br> <br>top %&gt;% ungroup() %&gt;% mutate(bigram = fct_reorder(bigram, n)) %&gt;% <br> ggplot(aes(x=bigram, y=n)) + <br> geom_col() + <br> coord_flip() +<br> theme_classic() + <br> theme(legend.position = “none”,<br> text = element_text(size=18)) </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*__cRQTsUYd1Nj9CQ9BZoMw.png" /></figure><p>I see that one of the most frequent bigrams is <strong>behavioral change.</strong> We can use this information to make sense of what we have found previously. For example, one of the most frequent word was change. And we saw with bigram analysis that the word change was used mostly in a context of behavioral change. <strong>So bigrams are useful tools to get deeper insights about the text content.</strong></p><p>My highlighted text data come from 28 different books, and by looking at the most frequent words and bigrams in the whole document we got an overview of them.</p><p><strong>To see how the individual books differ we can repeat this procedure for each of them.</strong></p><p><strong>But how can we capture them individually?</strong></p><p>Let’s first look at the text once more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*BBWoKfPi5ttH8dCSourjRw.png" /></figure><p>Before each book<strong> “Your Kindle Notes For:” </strong>appears. Let’s find out the line numbers for the beginning and the end of each book and use those indexes for fishing out each book.</p><p>We will reuse the data frame df we created above. <strong>str_which()</strong> function returns line index numbers which contain an given input pattern.</p><p>In the final step, a for loop to <strong>capture the text between two consecutive indexes</strong> will give us the book between them.</p><pre><strong># Getting the index number for each book</strong></pre><pre>indexes &lt;- <strong>str_which</strong>(df$highlights, <strong>pattern</strong> = fixed(<strong>&quot;Your Kindle Notes For&quot;</strong>))<br><strong>book_names</strong> &lt;- df$highlights[indexes + 1]<br><strong>indexes</strong> &lt;-  c(indexes,nrow(df))</pre><pre><strong># Create an empty list </strong></pre><pre>books &lt;- list()</pre><pre><strong># Now the trick. Capture each 28 book separately in a list. </strong></pre><pre><strong>for</strong>(i in 1:(length(indexes)-1)) {<br>    books[[i]] &lt;- data.frame(df$highlights[(indexes[i]:indexes[i+1]-1)])<br>    colnames(books[[i]]) &lt;- &quot;word_column&quot;<br>    books[[i]]$word_column &lt;- as.character(books[[i]]$word_column)<br>}<br></pre><p>Let’s check whether it worked, for example you can look up the 5th book on our list.</p><pre><strong>head(books[[5]])</strong></pre><pre>                                         word_column<br>1                                                    <br>2                              Your Kindle Notes For:<br>3 Bird by Bird: Some Instructions on Writing and Life<br>4                                         Anne Lamott<br>5             Last accessed on Saturday July 27, 2019<br>6                         75 Highlight(s) | 4 Note(s)</pre><p>Now, we captured all 28 books and we can apply the same procedure to analyze each of them by another for loop.</p><pre>top &lt;- list()</pre><pre><strong>for</strong>(i in 1:28){<br><strong>books[[i]]</strong> &lt;- books[[i]] %&gt;% <strong>unnest_tokens</strong>(word, word_column) %&gt;%<br>             anti_join(stop_words) %&gt;% <br>             filter(!word %in% c(&quot;highlights&quot;,&quot;highlight&quot;, &quot;page&quot;, <br>                      &quot;location&quot;, &quot;yellow&quot;, &quot;pink&quot;, &quot;orange&quot;, &quot;blue&quot;))</pre><pre><strong># Find out the top words in each book and capture them in a list (top)</strong></pre><pre><strong><br>top[[i]]</strong> &lt;- books[[i]] %&gt;% <br>              group_by(word) %&gt;% <br>              count() %&gt;% <br>              arrange(desc(n))</pre><pre>}</pre><pre><strong>for(i in 1:28)</strong>{<br>  print(book_names[[i]])<br>  print(top[[i]])<br>}</pre><p>Here is a piece from the output of the above code.</p><pre>## [1] &quot;Crucial Conversations Tools for Talking When Stakes Are High, Second Edition&quot;</pre><pre>## # A tibble: 1,834 x 2<br>## # Groups:   word [1,834]<br>##    word          n<br>##    &lt;chr&gt;     &lt;int&gt;<br>##  1 people       84<br>##  2 dialogue     40<br>##  3 stories      40<br>##  4 due          34<br>##  5 export       33<br>##  6 feel         33<br>##  7 hidden       33<br>##  8 limits       33<br>##  9 truncated    33<br>## 10 crucial      31<br>## # ... with 1,824 more rows</pre><pre>## [1] &quot;Pre-Suasion: A Revolutionary Way to Influence and Persuade&quot;</pre><pre>## # A tibble: 526 x 2<br>## # Groups:   word [526]<br>##    word             n<br>##    &lt;chr&gt;        &lt;int&gt;<br>##  1 attention        6<br>##  2 influence        5<br>##  3 mental           5<br>##  4 trust            5<br>##  5 visitors         5<br>##  6 comfort          4<br>##  7 emotional        4<br>##  8 experience       4<br>##  9 message          4<br>## 10 associations     3<br>## # ... with 516 more rows</pre><pre>## [1] &quot;Made to Stick: Why some ideas take hold and others come unstuck&quot;</pre><pre>## # A tibble: 1,754 x 2<br>## # Groups:   word [1,754]<br>##    word          n<br>##    &lt;chr&gt;     &lt;int&gt;<br>##  1 people       64<br>##  2 knowledge    27<br>##  3 story        25<br>##  4 ideas        24<br>##  5 concrete     18<br>##  6 surprise     17<br>##  7 care         16<br>##  8 time         15<br>##  9 attention    14<br>## 10 core         14</pre><p>Now, looking at the most frequent words from each book we can get more insights what they are about.</p><p>The bigrams for the same books.</p><pre><strong>## [1] &quot;Crucial Conversations Tools for Talking When Stakes Are High, Second Edition&quot;</strong></pre><pre>## # A tibble: 8,774 x 2<br>## # Groups:   bigram [8,774]<br>##    bigram                    n<br>##    &lt;chr&gt;                 &lt;int&gt;<br><strong>##  1 due export               33<br>##  2 due export limits        33<br>##  3 export limits            33<br>##  4 hidden truncated         33<br>##  5 hidden truncated due     33<br>##  6 truncated due            33<br>##  7 truncated due export     33<br>##  8 crucial conversations    19<br>##  9 export limits 27         10<br>## 10 limits 27                10</strong><br>## # ... with 8,764 more rows</pre><pre><strong>## [1] &quot;Pre-Suasion: A Revolutionary Way to Influence and Persuade&quot;</strong><br>## # A tibble: 1,265 x 2<br>## # Groups:   bigram [1,265]<br>##    bigram                      n<br>##    &lt;chr&gt;                   &lt;int&gt;<br>##  1 attention goal              2<br>##  2 concept audience            2<br>##  3 levels importance           2<br>##  4 mandel johnson              2<br>##  5 mental activity             2<br>##  6 social proof                2<br>##  7 thousand dollars            2<br>##  8 twenty thousand             2<br>##  9 twenty thousand dollars     2<br>## 10 writing session             2<br>## # ... with 1,255 more rows</pre><pre><strong>## [1] &quot;Made to Stick: Why some ideas take hold and others come unstuck&quot;</strong></pre><pre>## # A tibble: 6,376 x 2<br>## # Groups:   bigram [6,376]<br>##    bigram                      n<br>##    &lt;chr&gt;                   &lt;int&gt;<br>##  1 curse knowledge             7<br>##  2 guessing machines           6<br>##  3 people care                 6<br>##  4 goodyear tires              5<br>##  5 knowledge gaps              5<br>##  6 people&#39;s attention          5<br>##  7 popcorn popper              5<br>##  8 security goodyear           5<br>##  9 security goodyear tires     5<br>## 10 sinatra test                5</pre><p>Now, you probably saw the weird bigrams in the first book. Kindle limits the length of the text you can highlight, for example you can’t highlight 5 pages of text. This prevents people to highlight the whole book and export to a word document.</p><p>Since I occasionally I highlighted long stretches of text, the words like “due”, “export” and “limits” appeared as warnings on my exported highlights.</p><p><strong>Now, I would go back and make some more cleaning by adding those words into the filter() function.</strong></p><p>Looking at each book individually,<strong> he was more and more obsessed about the books in my kindle.</strong> He decided to order a couple of them.</p><p><em>If you want to see another example of this capturing process you can have a look at my recent </em><a href="https://dataatomic.com/r/data-wrangling-text-mining/"><em>post here.</em></a></p><p>We can use <strong>sentiment analysis</strong> to evaluate emotional charge in a text data. Most common uses are social media monitoring, customer experience management, and Voice of Customer, to understand how they feel.</p><blockquote>The<strong> bing</strong> lexicon categorizes words into positive and negative categories, in a binary fashion. The <strong>nrc </strong>lexicon uses categories of <strong>positive, negative, anger, anticipation, disgust, fear, joy, sadness, surprise, and trust.</strong></blockquote><p><strong>Using bing lexicon</strong></p><p>I created a list of the words with highest contribution to each emotional category. For example success and effective for positive, bad and hard for negative sentiment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AydS4md-jH3_mWAfKYpowQ.png" /></figure><p>Here is how R produced the above plot:</p><pre>bing_word_counts &lt;- df %&gt;% inner_join(get_sentiments(&quot;bing&quot;)) %&gt;% <br>  count(word, sentiment, sort = TRUE) %&gt;%<br>  ungroup()</pre><pre>bing_word_counts</pre><pre># <strong>Sentiment plot for top positive negative contributors</strong><br># Select top 10 positive and negative words</pre><pre>bing &lt;- bing_word_counts %&gt;% <br>  group_by(sentiment) %&gt;% <br>  top_n(10) %&gt;% <br>  ggplot(aes(reorder(word, n), n, fill=sentiment)) + <br>  geom_bar(alpha=0.8, stat=&quot;identity&quot;, show.legend = FALSE)+<br>  facet_wrap(~sentiment, scales = &quot;free_y&quot;) +<br>  labs(y= &quot;Contribution to sentiment&quot;, x = NULL) +<br>  coord_flip()</pre><pre>bing</pre><p><strong>Using nrc lexion</strong></p><p>I am more likely to highlight a text if it is positive rather than negative. And also trust, anticipation and joy rather than sadness or anger.</p><pre>sentiment &lt;- df %&gt;%<br>        left_join(get_sentiments(&quot;nrc&quot;)) %&gt;%<br>        filter(!is.na(sentiment)) %&gt;%<br>        count(sentiment, sort = TRUE)<br>sentiment</pre><pre>## # A tibble: 10 x 2<br>##    <strong>sentiment        n</strong><br>##    &lt;chr&gt;        &lt;int&gt;<br>##  1 positive      8471<br>##  2 trust         4227<br>##  3 negative      3963<br>##  4 anticipation  3466<br>##  5 joy           2701<br>##  6 fear          2467<br>##  7 sadness       1853<br>##  8 anger         1814<br>##  9 surprise      1353<br>## 10 disgust       1102</pre><p><strong>Normalized sentiments</strong></p><p>One important thing to add, since each emotion category has different number of words in a language. Emotional categories with less words are less likely to appear in a given text. Thus, <strong>I would like to normalize them according to their frequencies in the lexicon and see how it differs than the above results.</strong></p><pre><strong><em># I will add numbers of each categories from the NRC lexicon</em></strong></pre><pre>lexicon &lt;- c(2317, 3338, 1234, 842, 1483, 691, 1250, 1195, 1060, 535)<br>polarity &lt;-  c(1,1,1,1,1,0,0,0,0,0)<br>sentiment &lt;- data.frame(sentiment, lexicon)<br>norm_sentiment &lt;- sentiment %&gt;% mutate( normalized = n/lexicon) %&gt;% arrange(desc(normalized))<br>sentiment &lt;- data.frame(norm_sentiment, polarity)<br>sentiment</pre><pre>##       sentiment    n lexicon <strong>normalized</strong> polarity<br>## 1  anticipation 3466     842   4.116390        1<br>## 2      positive 8471    2317   3.656021        1<br>## 3          fear 2467     691   3.570188        1<br>## 4      negative 3963    1234   3.211507        1<br>## 5       disgust 1102     535   2.059813        1<br>## 6           joy 2701    1483   1.821308        0<br>## 7         anger 1814    1195   1.517992        0<br>## 8       sadness 1853    1250   1.482400        0<br>## 9      surprise 1353    1060   1.276415        0<br>## 10        trust 4227    3338   1.266327        0</pre><p>Now, <strong>anticipation</strong> is the highest emotion found in the text. This does not seem a coincidence to me. Since most of the books in our analysis is about productivity and self-development. The productivity tips and tools usually contain words associated with anticipation.</p><p><strong>In a similar way, I can look at the sentiment for individual books</strong></p><pre>sentiment &lt;- list()<br><strong>for</strong> (i <strong>in</strong> 1:28){<br>sentiment[[i]] &lt;- books[[i]] %&gt;%<br>        left_join(get_sentiments(&quot;nrc&quot;)) %&gt;%<br>        filter(!is.na(sentiment)) %&gt;%<br>        count(sentiment, sort = TRUE)<br>        print(book_names[i])<br>        print(sentiment[[i]])<br>}</pre><p>Sentiments for individual books. I show here a couple of them.</p><pre>## [1] &quot;Crucial Conversations Tools for Talking When Stakes Are High, Second Edition&quot;</pre><pre>## # A tibble: 10 x 2<br>##    sentiment        n<br>##    &lt;chr&gt;        &lt;int&gt;<br>##  1 positive       758<br>##  2 negative       496<br>##  3 trust          412<br>##  4 fear           282<br>##  5 anticipation   258<br>##  6 anger          243<br>##  7 joy            216<br>##  8 sadness        196<br>##  9 disgust        142<br>## 10 surprise       108</pre><pre>## Joining, by = &quot;word&quot;</pre><pre>## [1] &quot;Pre-Suasion: A Revolutionary Way to Influence and Persuade&quot;<br>## # A tibble: 10 x 2<br>##    sentiment        n<br>##    &lt;chr&gt;        &lt;int&gt;<br>##  1 positive        84<br>##  2 trust           51<br>##  3 negative        31<br>##  4 anticipation    27<br>##  5 fear            24<br>##  6 joy             22<br>##  7 anger           14<br>##  8 sadness         12<br>##  9 surprise         9<br>## 10 disgust          3</pre><pre>## Joining, by = &quot;word&quot;</pre><pre>## [1] &quot;Made to Stick: Why some ideas take hold and others come unstuck&quot;<br>## # A tibble: 10 x 2<br>##    sentiment        n<br>##    &lt;chr&gt;        &lt;int&gt;<br>##  1 positive       499<br>##  2 trust          236<br>##  3 anticipation   198<br>##  4 negative       167<br>##  5 joy            156<br>##  6 fear           123<br>##  7 surprise       107<br>##  8 sadness         74<br>##  9 anger           65<br>## 10 disgust         60</pre><p>To have an overview, you can summarize the data by plotting the positivity ratio from each book.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0CFiaUMwbd-rZ6ZAlLJ8Sg.png" /></figure><p>Let’s look at the book with the lowest positivity score. <a href="https://www.amazon.com/Mans-Search-Meaning-Viktor-Frankl/dp/080701429X">“<strong>Man’s search for meaning”</strong></a><strong>. </strong>This book is based on Victor Frankl sufferings during the second world war. This is also kind of expected.</p><blockquote>I am getting more and more impressed about the power of text mining.</blockquote><p>The book “The Outliers” appeared on the top of the positivity plot was a real outlier here. 😮</p><p>It is hard to know everything from the beginning and we will go back to make some additional cleaning. The word count from the book “The Outliers” is 107. <strong>This is really low. </strong>So in the next iteration, I would remove it from the analysis since it will not be reliable.</p><pre>book_names[[27]]</pre><pre><strong>## [1] &quot;Outliers: The Story of Success&quot;</strong></pre><pre>top[[27]]</pre><pre># A tibble: 107 x 2<br># Groups:   <strong>word [107]</strong><br>word             n<br>&lt;chr&gt;        &lt;int&gt;<br>1 ability          3<br>2 knowing          3<br>3 sense            3<br>4 communicate      2<br>5 distance         2<br>6 family           2<br>7 intelligence     2<br>8 power            2<br>9 practical        2<br>10 sternberg        2<br># ... with 97 more rows</pre><p>Here is the code I used for plotting the positivity scores:</p><pre>books &lt;- <strong>str_trunc</strong>(book_names, width=22) # Shorten the book names</pre><pre>all &lt;- list()</pre><pre>for (i in 1:28) {<br>all[[i]] &lt;- sentiment[[i]] %&gt;% <br>  filter(sentiment %in% c(&#39;positive&#39;,&#39;negative&#39;)) %&gt;% <br>  mutate(n2 = n/sum(n)) %&gt;% <br>  print()<br>}</pre><pre>all_bound &lt;- do.call(&quot;rbind&quot;, all) %&gt;% filter(sentiment == &quot;positive&quot;)</pre><pre><strong>library(ggrepel) # Useful for preventing overlapping labels</strong></pre><pre>all_bound %&gt;% ggplot(aes(x= book_names, y=n2)) + <br>              geom_point() + <br>              <strong>geom_label_repel</strong>(aes(label=books, color = ifelse(n2 &lt;0.55, &quot;red&quot;, &quot;blue&quot;)), size = 4) + <br>              theme_classic() + <br>              theme(legend.position = &quot;none&quot;,<br>                    text = element_text(size=18), <br>                    axis.text.x = element_blank()) + <br>              xlab(&quot;Books&quot;) + <br>              ylab(&quot;Positivity score&quot;)</pre><h3>Summary</h3><p>It is not feasible to read millions of pages to check whether text mining is reliable. But here I got some data that I know the content and I applied text mining and sentiment analyses.</p><p>Both the monograms or bigrams pointed to similar ideas what the books were about. And the sentiments made sense with the genres of the books in my kindle.</p><p>Let’s come back to our hacker.</p><p>An unanticipated side effect of text mining changed him forever. Analyzing my books and gaining the insights made him more and more interested in reading. And he started to care about the world around him. The world was different.</p><p>What I was trying to do for myself, he did for himself. He transformed into a better version of himself.</p><p>The world was brighter. ☀️</p><p>The radio disrupted the silence.</p><blockquote>“brrring…..brrring…..brrring…..”</blockquote><p>I woke up.</p><p>Thank you for reading. You can find the data and the code in my <a href="https://github.com/korur/textmining">github repo.</a></p><p>I hope you’ve learned something or got inspiration from this. Please feel free to leave comments, suggestions, and questions. (You can always contact me by email at serdar.korur@gmail.com)</p><p>Until next time!</p><p>Serdar</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b43f8ee1273" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/an-accidental-side-effect-of-text-mining-4b43f8ee1273">An accidental side effect of text mining</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get better in data science writing and the opportunities will grow]]></title>
            <link>https://medium.com/data-science/get-better-in-data-science-writing-and-the-opportunities-will-grow-9d37fdada262?source=rss-70bf243827fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d37fdada262</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-storytelling]]></category>
            <category><![CDATA[writing]]></category>
            <category><![CDATA[education]]></category>
            <dc:creator><![CDATA[Serdar Korur]]></dc:creator>
            <pubDate>Mon, 02 Dec 2019 18:27:33 GMT</pubDate>
            <atom:updated>2019-12-18T11:41:11.373Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HlaNGvPRBTsIPpSll5nKUA.jpeg" /><figcaption>Photo by Ian Stauffer on Unsplash</figcaption></figure><h4>How to improve your data science writing by 3700% in one year</h4><h4>9 tips for better communication</h4><p>We do it all the time.</p><p>Writing is a big part of our communication<strong>. </strong>We write emails, text messages and blog posts. We write project reports and technical documents. We write cover letters and resumes. We write endless lines of text and we do it all day long.</p><blockquote>Remember that what you write is often your only chance you’ll get to present yourself someone whose business or money or good will you need.</blockquote><blockquote>— William Zinnser</blockquote><p>He is right. Don’t miss out your opportunities because of not being clear enough. If writing is what we do the most every day, imagine the returns of improving it even a little bit.</p><p>In this post, I will go through simple steps to make your writing more effective with the advice from the best.</p><p>Data scientists often need to explain their approach and findings to non-technical audience. For instance, the managers or the marketing team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6fE5BZhriSBVCakYREPUJw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@icons8?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Icons8 Team</a> on <a href="https://unsplash.com/s/photos/discussion?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Avoid jargon and cluttered sentences. Take the trouble to be clear and personal.</p><p>Clear writing can be compared to making a principal components analysis. In PCA the goal is to select the variables which explain most of the variation in your data and simplify the computational processes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*dMT7DHGCU1x2aBcupSIuIw.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Scree_plot#/media/File:Screeplotr.png">https://en.wikipedia.org/wiki/Scree_plot#/media/File:Screeplotr.png</a></figcaption></figure><p>By carefully selecting the most important facts and ideas and eliminating the less important ones in your writing, you can make your story more palatable for your audience.</p><h3>9 Simple steps to write better</h3><p>The following steps will help you to get better in telling your data science story. Check-out the resources at the end for more in-depth information.</p><h4>1. Make a habit of writing or rewriting for 30 minutes every day</h4><p>For example, in fitness you don’t get any gain in your muscles if you train occasionally. Your muscles start to grow and get stronger only when you do it regularly. Likewise, a mental task improves with consistence.</p><p>Talented people are talented not because they did something once and it was awesome. They did something thousands of times and only after that, they became the legends. <strong>Do things many times.</strong> <strong>Write daily.</strong> The more you do the better you will get at it.</p><p>As Ray Bradbury the author of <em>Fahrenheit 451</em> said</p><blockquote>“Quantity produces quality. If you only write a few things, you’re doomed.”</blockquote><p>Many things in life follow an exponential growth curve. <strong>You start with very small gains in the beginning and much higher at the end.</strong></p><p>If you improve a skill 1% every day for a year you will get 3700% cumulative improvement at the end of one year.</p><pre># Cumulative gains after 365 days<br>1.01^365 * 100<br>= 3778.343%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*V9yyg6pjo5jt7v7BPKCtRg.png" /><figcaption>Daily 1% improvements result in a 3700% gain in one year</figcaption></figure><h4>2. Start with structure</h4><p>This writing tip I got was from one of my favorite authors,</p><blockquote>“Learn structure. Study your favorite nonfiction books with particular attention to how the book is organized at a book level, then at a chapter level, and then paragraph by paragraph. Structure makes everything else easier.”</blockquote><p>You can apply this to writing a data science story. Analyze the structure of your favorite data science articles. How the writer organize the whole post? How did he construct each paragraph? How he used the story to embed the data inside? This will help you to understand and discover your own successful ways. <strong>Structure will give you direction.</strong></p><h4>Use a framework</h4><p>The key to writing a good story often depends on being able to organize your thoughts. And to organize your thoughts in a written form you need a framework. Any framework is better than not having even as simple as beginning-middle-end helps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V2zncSt_X1x7-DmvmwzjEA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@eliabevces?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Eliabe Costa</a> on <a href="https://unsplash.com/s/photos/bookshelf?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>How your bedroom closet organizes your clothes? The same way even a crude plan of the article will make everything easier. Especially, in a data science project writing can get complex. Having a frame to put different pieces will make it much easier to construct the final story.</p><p><strong>Read a lot, but read analytically.</strong> Study the best articles you find in medium.com or other data science blogs. Look how they are constructed as a whole, at paragraph or sentence level.</p><h4>3. Set deadlines</h4><p>Deadlines help us to do more things in less time. Imagine you have one year to finish writing a book. But you are ambitious and you set yourself a deadline to finish it in one month. What do you think things will look like at the end of this one month? You might not be able to finish it but <strong>you will be much more progressed.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bQjZmTPFtqleVyhJ8cF6-A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Luke Chesser</a> on <a href="https://unsplash.com/s/photos/picasso-time?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>For data science articles I would create a quick first draft that I have to finish in one hour. <strong>This exercise will teach you to choose the most important facts and drop less important ones for your article</strong>. A prerequisite to be clear and well understood by your audience.</p><ul><li>If you got only 60 minutes to write 2000 words, you have no time to get distracted.<strong> The limitation will help you to select what needs to be done and drop what is less important.</strong></li></ul><p>Doing this will help you to master focusing on the core of your project.</p><p>Like we learned in chemistry classes, gases will fill the space they are in. Like a drop of cologne will be felt at the other corner of the room. <strong>Tasks swell to fill the time space allotted to them.</strong> This is known as the Parkinson’s law, the term coined by <a href="https://en.wikipedia.org/wiki/C._Northcote_Parkinson">Cyril Northcote Parkinson</a>. I used <strong>gamma distribution function</strong> to create a similar curve which describes the original Parkinson’s law.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*bRsQn_UqWmW-rso83veW9A.png" /></figure><p>The total effort (area under curve) for finishing a task increases with the time allowed.</p><p>Besides helping you to manage your time <strong>the real benefit is to make you learn how to quickly extract the core of your data.</strong></p><h4>4. Explain it simply</h4><p>We tend to associate using complex sentences and plenty of terminology as part of our wisdom. B<strong>ut the real sign of intelligence is the ability to explain things simpler. </strong>Clear writing requires clearer and simpler thinking. One of my favorite quotes from Einstein is:</p><p>“If you can’t explain it simply, you don’t understand it well enough”</p><h4>5. Use stories.</h4><p>Stories are hardwired in our brain because they were critical to survival. Before written language existed, stories were the only way we relayed information.</p><p>While probably we need to be told multiple times to learn facts and figures <strong>stories need only to be told once to be never forgotten. </strong>The more connected your story to a real life event or real concrete business problem the better it will be understood by your readers.</p><blockquote>“The difference between data and story is this: <strong>Data lists what happened; story expresses how and why it happened.</strong> Data compiles facts by quantity and frequency; story reveals the causalities behind and beneath those facts. Story eliminates irrelevancies, concentrates on dynamic change” Storynomics.</blockquote><h4>Combine your personal interests with data science writing</h4><p>Put your personal flavors in your posts. Readers like to know who is behind the text and to learn about experiences and more about the real people.</p><h4>6. Be specific</h4><p>S.I. Hayakawa the US Senator, and the semantics expert introduced the concept Ladder of abstraction. Very specific concepts are at the bottom, and the more general concepts are at the top of the ladder. Here is an example Bessie the cow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*kDfhf1V4rdnX6jt9Zlz03A.png" /><figcaption>source: <a href="http://www.atlas101.ca/pm/concepts/ladder-of-abstraction/">http://www.atlas101.ca/pm/concepts/ladder-of-abstraction/</a></figcaption></figure><p>Use general concepts to introduce a topic and go as low as possible in the abstraction ladder by giving specific examples to explain the results you have. Brain works best when given a very specific example and then it can understand complex topics much easier.</p><blockquote>If you ever used a microscope, to see the parts of a small insect, you know you have to bring the objective into the focus. Same way specific information focuses the brain.</blockquote><h4>Avoid using your direct judgments instead list the facts.</h4><p>S.I Hayawaka explains this well in “Language and thought and action”:</p><blockquote>Instead of telling the reader, “It was a ghastly accident!” we can make the reader say it for himself. The reader is, so to speak, made to participate in the communicative act by being left to draw his own conclusions. A skillful writer is often, therefore, one who is especially expert at selecting the facts that are sure to move his readers in the desired ways. We are more likely to be convinced by such descriptive and factual writing than by a series of explicit judgments, because the writer does not ask us to take his word for it that the accident was “ghastly.”</blockquote><h4>7. Vary the paragraphs</h4><p>Reading online is different than reading books. People don’t like walls of text. Use paragraphs and sentences at differing lengths.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*4qjZARMbCzTVVBrdVCWhkA.png" /></figure><p>Read aloud those sentences and you will feel the difference better. <strong>Human brain constantly monitors changes in the environment, and change captures, monotony kills human attention. </strong>Vary the length of the sentences use short, middle and long sentences to make it more musical.</p><h4>8. Use proper images and titles</h4><p>If you ever heard about Robert Cialdini’s research on persuasion you might have a hint on how the images help you to get your message across more effectively.</p><blockquote>Pre-suasion, an influential book in marketing presents evidence that our actions are influenced by what we have done or see just before doing that action.</blockquote><p>For example, when in their experiments Mandel and Johnson used money as the background image for a landing page of an online furniture shop visitors were more concerned about the price, but when fluffy clouds are shown on the landing page they were more interested in comfort and bought more expensive items. <strong>Prepare the mind of your readers or business partners to the message you want to give.</strong></p><p>Titles and images are usually the first thing your prospective readers see. Support your findings by using a proper image or message associated with the mindset you would like them to be in.</p><blockquote>If your data science solution is about reducing costs in a production line use an image of money, if you have found ways to improve customer satisfaction use a happy customer photo.</blockquote><p>Think about what mood you want to put your readers? Here is a short video from Robert Cialdini giving tips on his presuasion tips.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FHctZg2aOPMw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DHctZg2aOPMw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FHctZg2aOPMw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d4d8297c5e5d4e509098257c2a28c672/href">https://medium.com/media/d4d8297c5e5d4e509098257c2a28c672/href</a></iframe><h4>9. Get visibility</h4><p>The noise in the internet is huge. Doing a great job is only half way. You need to make the world aware of what you created.</p><blockquote>The challenge is to make it through to your audience.</blockquote><p>Curator websites or blogs can improve traffic to your data science blog. For example, when I submitted my blog to R Bloggers I had more traffic in one day than I had in the last couple of weeks.</p><p>Some other examples are:</p><p>Obviously, medium.com (It is much better if you submit through Towardsdatascience).</p><p>Reddit</p><p>Twitter.</p><h4><strong>Extras</strong></h4><p><strong>Seeing the end product forces you to finish it quicker than you would do it otherwise. </strong>For example, when I first use R Markdown to create a blog post and render it to view locally or use medium editor where I can see the finished article I am much quicker to finish than when I use a word document.</p><p>In most languages, negative words dominate the positive ones. The evolutionary reason behind is probably to protect ourselves against danger.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*SX3Mp2MXj6xCrD9pmJK3pQ.png" /></figure><p>We stop and think longer when we see or hear a negative word or limitation in a title. That’s why we are more likely to click our mailboxes when we see emails with titles the last chance, 3 hours left for the awesome offer and so on.</p><h4>What you will learn?</h4><p><strong>Writing and doing a data science project both are about solving a problem. </strong>In your project you are solving a business problem. And in your writing, you are solving a problems like how to organize a messy or complex story, how to find the relevant material and how to order them.</p><p>Being a strong communicator will open you new job opportunities, new partnerships, and better relationships.</p><h3>Summary</h3><p>If what you write have a value in solving a problem or gives some insight or explains an abstract concept in a fun way <strong>don’t wait until it becomes the most perfect article in history.</strong></p><p>Something in your computer does not help you or the world so much. B<strong>etter to share it than hide it.</strong> And if you do so you can apply some of the tips in this guide to improve your data science writing.</p><p>Don’t worry about any mistakes. We are all human and we all learn by doing mistakes. <strong>Being ok with not being perfect opens the door to improvement.</strong></p><p>Apply those tips not only in data science but in any type of writing and you will see imrovement.</p><p>Data Science writing helps you getting feedback, stimulates discussion and improvement.</p><p>You don’t have to be an expert on a subject to write about it, <strong>write about it to be an expert on it.</strong></p><p><strong>Resources:</strong></p><p><a href="https://www.amazon.com/Storynomics/dp/B07BH2BH9M/ref=sr_1_1?keywords=storynomics&amp;qid=1575104753&amp;s=books&amp;sr=1-1">Storynomics</a>. By Robert Mckee, Thomas Gerace, et al.</p><p><a href="https://www.amazon.com/Writing-Well-Classic-Guide-Nonfiction/dp/0060891548/ref=pd_sbs_14_img_1/130-2788337-5329648?_encoding=UTF8&amp;pd_rd_i=0060891548&amp;pd_rd_r=c6778031-cc51-4874-90e2-c1ef07876736&amp;pd_rd_w=I1wSJ&amp;pd_rd_wg=0HXHW&amp;pf_rd_p=5cfcfe89-300f-47d2-b1ad-a4e27203a02a&amp;pf_rd_r=YJ5AG8H4R9F4NNP33SA1&amp;psc=1&amp;refRID=YJ5AG8H4R9F4NNP33SA1">On Writing Well</a>. By William Zinnser. (My subtitle comes from this book)</p><p><a href="https://www.amazon.com/Bird-Some-Instructions-Writing-Life/dp/0385480016">Bird by Bird</a>. By Anne Lamott.</p><p><a href="https://lithub.com/ray-bradburys-greatest-writing-advice/">Ray Bradbury’s Greatest Writing Advice.</a> lithub.com.</p><p><a href="https://www.goodreads.com/book/show/702619.100_Ways_to_Improve_Your_Writing">100 Ways to Improve Your Writing.</a> By Gary Provost</p><p><a href="http://doc.cat-v.org/economics/parkinsons-law/the-economist-article.pdf">Parkinson’s Law</a>. <em>The Economist</em>. London.</p><p><a href="https://insidegovuk.blog.gov.uk/2014/08/04/sentence-length-why-25-words-is-our-limit/">Why 25 words sentence length is our limit. </a>Inside GOV.UK</p><p><a href="https://www.amazon.com/Pre-Suasion-Revolutionary-Way-Influence-Persuade-ebook/dp/B01C36E2YS">Pre-Suasion: A Revolutionary Way to Influence and Persuade, </a>By Robert Cialdini, PhD</p><p><a href="https://www.amazon.com/Influence-The-Psychology-of-Persuasion/dp/B01KWAPHAK/ref=sr_1_1?keywords=persuasion&amp;qid=1575197967&amp;s=digital-text&amp;sr=1-1">Influence: The Psychology of Persuasion.</a> By Robert Cialdini, PhD</p><p><a href="https://foundationinc.co/wp-content/uploads/2018/12/nyt-final.pdf">Why do people share online?</a> New York Times.</p><p>Sharing useful content is a great way of helping each other to get over the noise and access knowledge that is relevant. A recent study found out people share more content they found useful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*GfV5JCE60qYrnIsDdx8nag.png" /></figure><p>Thanks for reading this far. I hope you’ve learned something or got inspiration from this.</p><p>Until next time!</p><p>Serdar Korur</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d37fdada262" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/get-better-in-data-science-writing-and-the-opportunities-will-grow-9d37fdada262">Get better in data science writing and the opportunities will grow</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An intuitive real life example of a binomial distribution and how to simulate it in R]]></title>
            <link>https://medium.com/data-science/an-intuitive-real-life-example-of-a-binomial-distribution-and-how-to-simulate-it-in-r-d72367fbc0fa?source=rss-70bf243827fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/d72367fbc0fa</guid>
            <category><![CDATA[horror-movies]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[probability-distributions]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Serdar Korur]]></dc:creator>
            <pubDate>Wed, 06 Nov 2019 14:32:36 GMT</pubDate>
            <atom:updated>2020-01-23T04:39:42.489Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iqREoMVQBfOYJULZqlEAng.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@epicantus?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Daria Nepriakhina</a> on <a href="https://unsplash.com/s/photos/r-programing?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h4>Learn it once, use it everyday</h4><h4>From calculating machine learning model classification accuracy to testing new cancer therapies, binomial distributions are everywhere.</h4><p><em>Learn. It is all about success and failure.</em></p><p><strong>What are binomial distributions and why are they so useful?</strong></p><p>When we repeat a set of events like 10 times coin flipping and each single event in a set has two possible outcomes (head or tails) think about <strong>Binomial distributions. </strong>Each single event here is known as a Bernoulli Trial.</p><p>Bi- in binomial distributions refers to the two outcomes usually described as <strong>Success</strong> or <strong>no</strong> <strong>Success</strong>. A “yes” or “no”.</p><p><strong>Probability distributions in general are used to predict future events </strong>and often based on nasty looking mathematical formulas. But, there is also a beautiful thing here. For example the specific binomial distribution mathematical function can be used to predict the outcomes of any real life event which has two outcomes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sIHYKptWHsnAwHI6HM94XA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@photoripey?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ibrahim Rifath</a> on <a href="https://unsplash.com/s/photos/coin-pile?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Let’s start with a simple example.</p><h4>Why is this interesting?</h4><p>For example, playing with the coins, the two possibilities are getting heads (success) or tails (no success). And let’s say you have a of e.g. 50 times coin flipping. We can repeat this set as many times as we like and record how many times we got heads (success) in each repetition.</p><p>And if plot the results we will have a probability distribution plot. And if you make enough repetitions you will approach a binomial probability distribution curve. We will do this in a minute. But, first generate data to draw our plot.</p><p>We can use R to generate the data. We will generate random data from repeating set of 50-times coin flipping 100000 times and record the number of successes in each repetition.</p><p><strong>rbinom() function can generate </strong>a given number of repeated (here 100.000) sets (50 times of coin flipping) of experiments. It requires 3 arguments.</p><pre><strong>rbinom(n</strong> = number of repetitions = 100.000,<br>       <strong>size</strong> = sample size = 50, <br>       <strong>p</strong> = the probability of success (chance of throwing heads is       0.5)<strong>)</strong></pre><p>How can we use this probability distribution plot?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f_b70t4yL8HiGlkGc8F4pw.png" /></figure><p><em>Binomial probability distributions help us to understand the likelihood of rare events and to set probable expected ranges.</em></p><p>The above plot illustrates if we randomly flip a coin 50 times, we will most likely get between 20 to 30 successes (heads) and events such as having more more than 35 successes (heads) out of 50 trials are very unlikely. The bars in <strong>red</strong> represents the sets which had 35 or more heads.</p><p>R code can be used to find the exact probabilities. Let’s compare the probabilities of getting more than 25, 35 or even 49 heads. You can combine rbinom with mean function to find the percentage of the events with a chosen outcome.</p><pre># Probability of getting 25 or more heads<br>mean(rbinom(100000, 50, .5) &gt;= 25)</pre><pre>0.5555</pre><pre># Probability of getting 35 or more heads<br>mean(rbinom(100000, 50, .5) &gt;= 35)</pre><pre>0.00341</pre><pre># Probability of getting 49 or more heads<br>mean(rbinom(100000, 50, .5) &gt;= 49)</pre><pre>0</pre><p>We found the probability of throwing 49 or more heads to be 0. But to be technically precise it is <strong>one in 375 trillion times</strong> (= 1/((1/²⁴⁹)+(1/²⁵⁰))).</p><h4>So what are all those will be useful for?</h4><p>You can impress your friends with your ability to use binomial distributions to predict coin flipping outcomes, but let’s look at other real life applications of them.</p><p><strong>The performance of a machine learning model.</strong></p><p>You built a machine learning model with a binary outcome. Let’s say pathological image recognition algorithm for liver cancer that is 90% accurate. You tested 100 patients and you want to know your 95% confidence interval? Or your new results showed that your model detected less than 70 patients correctly. Is it possible? Or you should start optimizing your parameters again?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FzAWkG8sjIVOyt2WL40NGA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@sharonmccutcheon?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Sharon McCutcheon</a> on <a href="https://unsplash.com/s/photos/hospital?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><strong>Number of patients responding to a treatment</strong>.</p><p>Let’s say you have a new therapy for cancer which has 10% probability to cure a patient. You have 500 patients which took the drug. The expected number of recovering patients is 50. But in your trial 75 patients responded. Is this due to chance or a significant effect? Or you should start looking underlying factors if there is something about the therapy or the patient group?</p><p><strong>Think about a hospital emergency station.</strong></p><p>You are a hospital manager and you want to organize the staff numbers correctly for different weekdays. You know total number of patients came in to a emergency station because of alcohol poisoning in a given time period. You can analyse the distribution of patient numbers for each day of the week. Most likely you will have more such cases in the weekends and you need larger staff.</p><p><strong>This will be also true for other businesses. They can use binomial distributions to calculate changes in demand and plan accordingly.</strong></p><p><strong>If you are running a Webserver.</strong></p><p>You can allocate your resources better by identifying times when traffic will be higher.</p><p>Some other questions we can answer by binomial distributions are:</p><p><strong>Number of people who answered ‘yes’ to a survey question.<br>How many games a team will win in one season?<br>Vote counts for a candidate in an election.<br>Number of defective products in a production run.</strong></p><blockquote>Binomial distributions are common and they have many real life applications.</blockquote><p>We can expand binomial distributions to<strong> multinomial distributions</strong> when instead there are more than two outcomes for the single event.</p><p>Such as there are 6 outcomes when rolling a die, or analyzing distributions of eye color types (Black, blue, green etc) in a population. When it is about distributions for events with multiple categories think about <strong>multinomial distributions.</strong></p><p>If the number of observations(n) are large we can think of a multinomial draw as being a series of binomial draws <a href="https://www.springer.com/us/book/9780387001784">(Gentle, 2003, pp. 198–199).</a> For example, when rolling a die the 6 categories can be thought of a combination of 6 different binomial trials (getting 1, 2 ,3 and so on).</p><p>If you are not convinced just by reading this, I will simulate how the shape of a multinomial event changes by increasing the number of trials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*gSwUz3apxrdE6FToZyEb8A.gif" /><figcaption>Multinomial trial approaches a binomial distribution</figcaption></figure><p>Let’s use some real life data to apply our knowledge so far. The data come from <a href="https://github.com/rfordatascience/tidytuesday">TidyTuesday</a> — a weekly social data project in R organized by the <a href="https://www.rfordatasci.com/">R for Data Science community</a>.</p><p>It contains data about Horror Movies released since 2012. And what I asked was whether horror movies are more likely be released at the 13th each month?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*knqY7pxl2KnySJnHuSQMFw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@seteales?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Allef Vinicius</a> on <a href="https://unsplash.com/s/photos/smile?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>If you are in <strong>the right mindset</strong> anything feels better and if you are not in the mood nothing will make us happy. <strong>The mindset we have prior to an event influences what we will feel about an event.</strong> This is at least what behavioral scientist <a href="https://www.amazon.com/Pre-Suasion-Revolutionary-Way-Influence-Persuade-ebook/dp/B01C36E2YS">Robert Cialdini’s</a> research says.</p><p>Imagine you are a horror movie fan and you went to the cinema. On the screen couple of ads are running just before the movie starts. You approach your mobile to turn off the voice and the date catches your attention, it is the 13th.</p><p><strong>And usually with the number 13 we associate cursed events. Do you think it will influence your impression about the movie?</strong></p><p>We don’t know if this is true but <strong>I wanted to test whether movie makers have similar ideas and selected 13th as the release date more often than other days. </strong>So, I explored Horror movies data and calculated number of releases in different days of the month.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Om1UXIWAL2X7ktvWlwGp1g.png" /></figure><h4>Is this significant?</h4><p>In the data, there were <strong>2782</strong> movies associated with a release date. So expected movie release per day is <strong>92 </strong>(2782 / 32)</p><p><strong>This is a good example of a multinomial probability distribution with 30 categories, and since the number of samples are large it will approximate a binomial distribution.</strong></p><p>Thus, we can apply binomial probability distributions for calculating the probabilities in our multinomial data.</p><p>We saw above that in some days more movies are released than the expected value. What we want to know is, which days are in the <strong>range of random chance</strong> and which days there is a <strong>significant preference or an aversion</strong> to release a movie.</p><p>I will define an interval which contains 95% of probabilities in our simulated distributions. And the values outside will be the ones which were not due to random chance. To do this I need 2.5th and 97.5th quantiles of the distribution.</p><p>We can do this by the <strong>qbinom()</strong> function in R. For example qbinom(0.975, size, p) will return the value which will define the cut off which contains 0.975 of the probabilities. And our confidence interval will be the interval between:</p><p><strong>qbinom(0.025, size, p) &lt; </strong>Confidence Interval <strong>&lt; qbinom(0.975, size, p)</strong></p><pre>lower &lt;- qbinom(0.975, 2782, 1/30)<br>75<br>upper &lt;- qbinom(0.025, 2782, 1/30)<br>112</pre><pre><strong>75 &lt; </strong>Confidence Interval <strong>&lt; 112</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*Yl-MsFhDVQT7XaOXbxh7oA.png" /></figure><p>95% of the time, a specific day of month will have between 75 and 112 movie releases. Higher or lower values than this range can not happen due to random chance according to our probability distribution.</p><p>124 movies released at the 13th of any month. This value is above the 97.5th quantile. So it is significant. But what is the exact p value? Let’s define p value first.</p><h3><strong>P value is the sum of the probability of that event plus the sum of the probabilities of similar events that are equally likely or less likely.</strong></h3><p>For example in coin flipping, probability of heads is (0.5). If we follow our definition p value is the sum of the probability of that event (0.5) and similar events which is equally or less likely i.e. tails (0.5). So our p value is 0.5 + 0.5 = 1.</p><p>Similarly, in our horror movie data this will be the sum of the probabilities of getting 124 movie releases or events that are equally probable or rarer.</p><p>In R, <strong>pbinom</strong> function defines the cumulative probabilities. For example, pbinom(124, 2782, 1/30) will give us the cumulative probabilities of any number of movie releases up to 124. By using 1-pbinom(124, 2782, 1/30) we can find the sum of the probabilities with equal or lower chance than having 124.</p><p>Thus, p value for getting at least 124 movie release is;</p><pre>p_val_binom &lt;- 2 * (1 — pbinom(124, 2782, 1/30))<br>0.00133</pre><p>We multiplied by two because same rare events can happen in the left side of our confidence interval as well.</p><p>Let’s put those p values on our horror movie plot to highlight the significant days.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QObRbMdqRRfCxqNzmkYa0Q.png" /></figure><p>We tested our hypothesis about movie makers and calculated the p value and used the pbinom() function and found couple of other days where movies are more or less likely to be released.</p><p><strong>However, another widely used way to calculate p values is to calculate the mean of the distribution and its standard deviation and to verify how many standard deviations the observed value is away from the mean (the z score).</strong></p><h4>Calculating the p-value by normal approximation</h4><p>When the sample size is large, binomial distributions can be approximated by a normal distribution. To build the normal distribution, I need mean and standard deviation. I can calculate this from the horror movie data.</p><pre><br>sample_mean is 92.7 <br>sample_sd is 89.64</pre><p>I can calculate the z-score for our observation of 124 movies that are released on the 13th. <strong>Simply, z-score is: how many standard deviations an observation is away from the mean. </strong>Since 95% of the observations will fall within 1.96 standard deviations from the mean in a normal distribution, a higher z-score will show that our p value is indeed significant.</p><pre>z_score &lt;- (observation — sample_mean) / sample_sd<br>3.302</pre><p>I can calculate the exact p value by using a normal distribution function <strong>pnorm() and the z score we found.</strong></p><pre>p_val_nor &lt;- 2 * pnorm(3.302, lower.tail = FALSE)<br>p_val_nor<br>0.00095</pre><p>As expected, I found similar values (<strong>Normal: 0.00095, Binomial: 0.00133) </strong>by using an approximation of a normal distribution and by using binomial distributions. Both methods proves that Horror movies are more likely to be released at the 13th.</p><h3>Conclusions</h3><p>Many events in real life can be explained by binomial probability distributions, and they allow us to calculate whether or not the events happened due to random chance and test our hypotheses.</p><p>It can be a fun data analysis such as in horror movies, or more serious subjects like testing of new medicines or predicting accuracy of machine learning algorithms detecting diseases.</p><p>Until next time!</p><p>Serdar</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d72367fbc0fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/an-intuitive-real-life-example-of-a-binomial-distribution-and-how-to-simulate-it-in-r-d72367fbc0fa">An intuitive real life example of a binomial distribution and how to simulate it in R</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>