<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lisa on Medium]]></title>
        <description><![CDATA[Stories by Lisa on Medium]]></description>
        <link>https://medium.com/@lisalondon?source=rss-773031daa634------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*R2EW0-AdHiuAs7EhXrA4GQ.jpeg</url>
            <title>Stories by Lisa on Medium</title>
            <link>https://medium.com/@lisalondon?source=rss-773031daa634------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 14:02:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lisalondon/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Bee Colony Survey Data — United States]]></title>
            <link>https://lisalondon.medium.com/bee-colony-survey-data-united-states-fcb20e327caa?source=rss-773031daa634------2</link>
            <guid isPermaLink="false">https://medium.com/p/fcb20e327caa</guid>
            <category><![CDATA[colony-collapse-disorder]]></category>
            <category><![CDATA[beekeeping]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[bees]]></category>
            <dc:creator><![CDATA[Lisa]]></dc:creator>
            <pubDate>Mon, 05 Jun 2023 23:48:43 GMT</pubDate>
            <atom:updated>2023-08-18T12:23:54.059Z</atom:updated>
            <content:encoded><![CDATA[<h3>Bee Colony Survey Data — United States</h3><h4>Bee Colony Statistical Data from 1987–2017</h4><h3><strong>PROJECT OVERVIEW:</strong></h3><p>Objective: Identifying bee species from images and tracking colony statistics from the U.S. Department of Agriculture.</p><p>Data description: Census and survey data from the USDA and colony loss data from Bee Informed.</p><h3><strong>DATA SUMMARY:</strong></h3><p>· “Bee Colony Survey Data by State data was retrieved from the <a href="https://quickstats.nass.usda.gov/">United States Department of Agriculture National Agricultural Statistics Service Quick Stats Dataset</a> with the selection criteria shown in the file Search criteria for bee colony census attached to this dataset.</p><p>· Bee Colony Census Data by County data was retrieved from the United States Department of Agriculture National Agricultural Statistics Service Quick Stats Dataset with the selection criteria shown in the file Search criteria for bee colony survey attached to this dataset.</p><p>· Bee Colony Loss file from the @makeovermonday dataset <a href="https://data.world/makeovermonday/2018w18-bee-colony-loss">2018w18-bee-colony-loss</a>. Extended to include census region and division data from data.world fact tables”<em>1</em> (data.world).</p><h3><strong>EXECUTIVE SUMMARY:</strong></h3><p>According to survey data, there has been a recent decline in the bee population, partly due to Colony Collapse Disorder. The U.S. Department of Agriculture is actively monitoring bee populations and implementing measures to counter CCD and other factors that contribute to bee loss. One solution for monitoring bees is to utilize machine learning algorithms to identify different bee species through image data. To this end, we developed an SVM model capable of training and testing a prediction algorithm to detect bee species from images. Our model achieved a score of 0.68 and an AUC (true positive rate) of 0.74.</p><p>There are still concerns regarding decreasing bee populations and climate change. To address the current challenges, recommendations include maintaining colonies and studying CCD. Mitigating the impact of climate change is also crucial, with potential solutions including developing a climate-proof beehive for indoor use or controlled bee reservoirs.</p><p>Another solution is the development of robotic bees with AI — an area still in research and development. In the meantime, it is important to continue reducing risks to bees and finding a cure for CCD. Raising funds for research and promoting awareness of these issues can also be effective paths forward.2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MXFOUZRq17bPu-jWaMGj-Q.png" /></figure><p>This is an overview of my project which aims to identify bee species from images. The identification of different bee species will allow researchers to more quickly and effectively collect field data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ooot4BybgQEhpE4n6mFWWA.png" /></figure><p>Beehives are important because of the pollination services they provide. According to bee experts at the Food and Agriculture Organization of the United Nations, a third of the world’s food production depends on bees. In the United States alone, honeybees pollinate $15 billion worth of crops each year, including more than 130 types of fruits, nuts, and vegetables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xpf_0hMxa-u9Nwb2WJ5ERA.png" /></figure><p>However, diseases like colony collapse disorder threaten these species. For example, the bee industry is currently facing difficulty meeting pollination demand in almonds. If research cannot solve CCD, beekeepers will be unable to meet demand for this and other crops. Climate change is another threat, affecting the ability for bees to forage, which decreases their population.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pqSVDg0coZPlle8Wj-5kvQ.png" /></figure><p>This project aims to address several business questions. Can we help identify species and control populations of bees? Are there trends in the use of historical data, i.e., images? Can we train a model to predict species based on a series of images? Can we evaluate the test predictions and score a model for future use?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Evo8uNnUwZzpN03LVzLZfw.png" /></figure><p>The data includes bee colony loss counts and population figures by state and county. We also have photos of bumblebees and honeybees, along with grayscale versions and additionally, RGB values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*09WUOBCcyntuUpmend9I4g.png" /></figure><p>First, I researched summary statistics, such as beekeeper counts, and total bee count values over the years by state. In one example, California decreased and then re-grew its bee colony population over the years. There appears to have been a decrease in 2007 and an increase in 2012, but that does not tell the entire story.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UJF7vNFDPyExamSL2mXTLA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mM-smd-dEy5k-tkoLsfclg.png" /></figure><p>This bee colony survey investigates the summary of bee values over the years by state for various data items, such as inventory by state, loss and loss by Colony Collapse Disorder, or CCD. The threat is increasing for CCD and overall loss of our bee colonies. Keeping track of stats like these help scientists understand the largest threats to bee colonies and our global crop production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9xeZ2JGUzzfAuTtwbDi-YA.png" /></figure><p>Dendrogram clustering is a technique to group similar data items together. It works by creating a tree-like structure called a dendrogram, where each observation is represented by a leaf and the branches show how they are clustered together. The distance between the leaves and the height of the branches determines the degree of similarity or dissimilarity between the data points. This technique is particularly useful for visualizing relationships in complex datasets. By using dendrogram clustering, we can gain insights into the underlying structure of the data and identify patterns that may not be immediately visible.</p><h3><strong>Image Recognition with Python</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iIK6XuLcwWx3EAuu7h7rPg.png" /></figure><p>Color channels can help provide more information about an image. A picture of an ocean will be bluer, whereas a picture of a fields will be greener. This kind of information can be useful when building models or examining the differences between images. By examining the kernel density estimate for each of the color channels on the same plot, we can understand how they differ.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ai45O9jo5WHjZsh0sdh20Q.png" /></figure><p>The goal is to develop a model that predicts bee species and evaluates its accuracy. The plan is to predict species using the data, features, and model by training and testing it using SVM and PCA methods. Finally, we aim to predict image labels and score/evaluate our model for future use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TOT8ewELnUAN2wXmUbYL6w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*68qj864-vOtjwY8Tb_MC0Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*akJiQjqF7tWRdZI4AImnoA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VVWiEDOHw2R-J9q8rqAplA.png" /></figure><p>In summary, the project aims to track bee colony statistics, predict bee species, and focus on image recognition. This will provide insights into business questions regarding bee populations with a machine learning model that can predict species with accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qiWbnCFSbafjkcRjVXkG4w.png" /></figure><h3><strong><em>Data Sources:</em></strong></h3><p><a href="https://data.world/makeovermonday/2018w18-bee-colony-loss">https://data.world/makeovermonday/2018w18-bee-colony-loss</a></p><p><a href="https://data.world/finley/bee-colony-statistical-data-from-1987-2017">https://data.world/finley/bee-colony-statistical-data-from-1987-2017</a></p><p><a href="https://app.datacamp.com/learn/projects/412">https://app.datacamp.com/learn/projects/412</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcb20e327caa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NASA — Meteorite Landing Analysis]]></title>
            <link>https://lisalondon.medium.com/nasa-meteorite-landing-analysis-1a096721856a?source=rss-773031daa634------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a096721856a</guid>
            <category><![CDATA[meteorites]]></category>
            <category><![CDATA[information-visualization]]></category>
            <category><![CDATA[nasa]]></category>
            <dc:creator><![CDATA[Lisa]]></dc:creator>
            <pubDate>Mon, 05 Jun 2023 22:20:59 GMT</pubDate>
            <atom:updated>2023-08-18T12:22:00.757Z</atom:updated>
            <content:encoded><![CDATA[<h3>NASA — Meteorite Landing Analysis</h3><p>NASA.GOV METEORITE LANDINGS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*UT5IY3LXe8CLj1Er3qNNRQ.png" /></figure><p>Description of Data Set: Title: “Meteorite Landings”</p><p>From the data source NASA.Gov: “This comprehensive data set from The Meteoritical Society contains information on all of the known meteorite landings. The Fusion Table is collected by Javier de la Torre and we’ve also provided an XLS file that consists of 34,513 meteorites.”</p><p>Fields: <br>place <br>type_of_meteorite <br>mass_g <br>fell_found <br>year <br>database <br>coordinate_1 <br>coordinates_2 <br>cartodb_id <br>created_at <br>updated_at year_date <br>longitude <br>latitude <br>geojson</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*d3tGVoJaBVB919suavy3Tg.png" /></figure><p><strong>str() Function of Dataset</strong></p><p>## ‘data.frame’: 45716 obs. of 10 variables:</p><p>## $ name : chr “Aachen” “Aarhus” “Abee” “Acapulco” … ## $ id : int 1 2 6 10 370 379 390 392 398 417 …</p><p>## $ nametype : chr “Valid” “Valid” “Valid” “Valid” … ## $ recclass : chr “L5” “H6” “EH4” “Acapulcoite” … ## $ mass..g. : num 21 720 107000 1914 780 …</p><p>## $ fall : chr “Fell” “Fell” “Fell” “Fell” …</p><p>## $ year : int 1880 1951 1952 1976 1902 1919 1949 1814 1930 1920 …</p><p>## $ reclat : num 50.8 56.2 54.2 16.9 -33.2 …</p><p>## $ reclong : num 6.08 10.23 -113 -99.9 -64.95 …</p><p>## $ GeoLocation: chr “(50.775, 6.08333)” “(56.18333, 10.23333)” “(54.21667, -113.0)” “(16.88333, -99</p><p>The above code reads the “Meteorite Landings” dataset and utilizes the str() function to inspect the variables in the dataframe, according to their types and names, as well as the overall number of variables and observations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*alQhmvvXUcA4Vfor_ggwvA.png" /></figure><p><strong>Distribution Plot of Year of Meteorite Landings</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*EDdoyAkI6AzqkUbC0RMJig.png" /></figure><p><strong>Single Dimension Plot of Fall Variable</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*3i1pB1RWBPUzVYj52aY7bA.png" /></figure><p><strong>Single Dimension Plot of Mass of Meteorites</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*ZP5bslwZqK1LStgKvZNSPQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*PaMDKsqWGOPMFU9-ZjqU-w.png" /></figure><p>Above, there are a couple of scatter plots illustrating mass in grams, offering insight into the variable Mass (in grams), inclusive of outlier and average values.</p><p><strong>Multidimensional Plot of Mass of Meteorites</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*YEHvZq-uPMVAgJ3SLhCl0w.png" /></figure><p><strong>Multidimensional Plot</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*G4KbY4G3HYTCIPMlQlmMVw.png" /></figure><p>NASA continues to track meteorites across geos, mass and found/fell statistics.</p><blockquote>“NASA’s Near Earth Object Program was established in 1998 to coordinate NASA-sponsored efforts to detect, track and characterize potentially hazardous asteroids and comets that could approach the Earth.</blockquote><blockquote>To better organize U.S. efforts to track asteroids that might one day threaten Earth, NASA established the <a href="https://www.nasa.gov/planetarydefense">Planetary Defense Coordination Office</a> (PDCO) in 2016. The PDCO:</blockquote><blockquote>Provides early detection of potentially hazardous objects (PHOs) — the subset of NEOs whose orbits predict they will come within 5 million miles of Earth’s orbit, and of a size large enough (30 to 50 meters) to cause significant damage on Earth.</blockquote><blockquote>Tracks and characterizes PHOs and issues warnings of the possible effects of potential impacts.</blockquote><blockquote>Studies strategies and technologies for mitigating PHO impacts.</blockquote><blockquote>Plays a lead role in coordinating U.S. government planning for response to an actual impact threat.</blockquote><blockquote>To date, NASA-sponsored NEO surveys have provided over 95% of all near-Earth object discoveries.</blockquote><p>(<a href="https://solarsystem.nasa.gov/asteroids-comets-and-meteors/in-depth/#:~:text=NASA&#39;s%20Near%20Earth%20Object%20Program,that%20could%20approach%20the%20Earth">https://solarsystem.nasa.gov/asteroids-comets-and-meteors/in-depth/#:~:text=NASA&#39;s%20Near%20Earth%20Object%20Program,that%20could%20approach%20the%20Earth</a>.)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a096721856a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quantitative Statistical Analysis]]></title>
            <link>https://lisalondon.medium.com/quantitative-statistical-analysis-8c0d4fca7d01?source=rss-773031daa634------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c0d4fca7d01</guid>
            <category><![CDATA[quantitative-analysis]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[vaccination]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[Lisa]]></dc:creator>
            <pubDate>Sun, 09 Apr 2023 18:48:59 GMT</pubDate>
            <atom:updated>2023-04-09T18:48:59.871Z</atom:updated>
            <content:encoded><![CDATA[<p>Purpose: Descriptive reports on US vaccinations and school districts’ reporting.</p><p><strong>INTRODUCTION</strong><br>How have U.S. vaccination rates varied over time? Are vaccination rates increasing or decreasing? Which vaccination has the highest rate at the conclusion of the time series? Which vaccination has the lowest rate at the conclusion of the time series? Which vaccine has the greatest volatility?</p><p><em>plot(usVaccines,col=”grey”)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*YVyniOSDKKW-GNscHP4pEg.png" /></figure><p>In the time series plot analysis, we notice a trend that vaccinations are increasing over time. The most volatile period seems to be around 1990. Afterwards, vaccinations were increasing until about the year 2017, which is the last year tracked in the data set.</p><p><em>summary(usVaccines)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aVJsYpg6jM-OLiRaWZLibA.png" /></figure><p>The vaccine with the highest rate at the conclusion of the time series is the DTP1 vaccine. It’s mean and max values are the highest in the data set.</p><p>The vaccine with the lowest rate at the conclusion of the time series is the HepB_BD vaccine. The HepB_BD vaccine has the lowest mean and max values in the usVaccines data set.</p><p>The change point locations of vaccinations: DTP1: 10, HepB_BD: 24, Pol3: 15, Hib3: 8, and MCV1: 7.</p><p>The cpt.mean() function detected changes by each variable in the data set. The above values show the locations at which the change occurs. The locations vary from 7 to 24, out of all the points in the time series, which is around 37 years, or one data point per year. The plots below visualize these changes. You will notice that the DTP1 vaccine experiences a shift after 10 years, HepB_BD after 24 years, Pol3 around 15 years, and so on.</p><p><em>plot(DTPcpt,cpt.col=”grey”,cpt.width=5)</em></p><p><em>#Plot point in time where change occurred and where the shift took place</em></p><p><em>title(‘DTP1’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*bBgEOIsLi4UvkQwLy89M1w.png" /></figure><p><em>plot(HEPcpt,cpt.col=”grey”,cpt.width=5)</em></p><p><em>title(‘HepB_BD’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*muNnubyc8Y6mViyEDYWQKg.png" /></figure><p><em>plot(POLcpt,cpt.col=”grey”,cpt.width=5)</em></p><p><em>title(‘Pol3’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*0OTUMIsDp1DtVmOi-fs0OA.png" /></figure><p><em>plot(HIBcpt,cpt.col=”grey”,cpt.width=5)</em></p><p><em>title(‘Hib3’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*f288IBofaLFY67ZDRwXk1g.png" /></figure><p><em>plot(MCVcpt,cpt.col=”grey”,cpt.width=5)</em></p><p><em>title(‘MCV1’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*tncnhAUg_dZzvyXcRuxHJA.png" /></figure><p>The major points of change of variance of vaccinations: DTP1 vaccine: 10, HepB_BD vaccine: — , Pol3 vaccine: 16, Hib3 vaccine: 8, MCV1 vaccine: 16.</p><p>Based on the above analyses, we can conclude that the vaccine, which experiences the greatest volatility over the time series is the Pol3 vaccine. There are major swings in vaccinations, with a volatile range of vaccination rates from 24 to 97 overall.</p><p>2. What proportion of public schools reported vaccination data?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BPDgSuiN-df4Mg-2F1ORrg.png" /></figure><p>What proportion of private schools reported vaccination data?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*egEjspPGJFACJ1mSB9kTmQ.png" /></figure><p>Was there any credible difference in overall reporting proportions between public and private schools?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-KeDyiJDvNjvRHXn1Z68TA.png" /></figure><p>Public schools are about the same in not reporting as an “N” or no, but there may be a credible differences in overall reporting proportions between public and private schools with reporting as a “Y”, or yes.</p><p>To further understand this difference, we can conduct analyses to detect the differences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OxBVPkL--uCOZU0476g6bA.png" /></figure><p>The null hypothesis states that public and private schools’ reporting statuses are independent of one another. The alternative hypothesis states that the correlation between public and private reporting is not 0 and therefore public and private school reporting statuses are not independent. Based on the Bayes factor of 2.407106e-268, which is a large value, this test highly favors the alternative hypothesis. Therefore, there is a difference between public and private school reporting. We can continue to test the two groups with additional statistical methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bOHNwfPJgt_qlHZoipnLgg.png" /></figure><p><em># Run a Bayesian analysis of the reported and private/public data.</em></p><p><em># Include posterior probability</em></p><p><em>Schools &lt;- contingencyTableBF(proptable, sampleType=”poisson”,</em></p><p><em>posterior=T, iterations = 10000)</em></p><p><em># Calculate reported status proportions for public schools</em></p><p><em>PrivateProp &lt;- Schools[,”lambda[1,1]”]/Schools[,”lambda[2,1]”]</em></p><p><em>hist(PrivateProp)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*Ze15sqaB6DMtuuI3NRoRag.png" /></figure><p><em># Calculate reported status proportions for private schools</em></p><p><em>PublicProp &lt;- Schools[,”lambda[1,2]”]/Schools[,”lambda[2,2]”]</em></p><p><em>hist(PublicProp)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*K2NF9lHc1J-9rlI81yJDww.png" /></figure><p><em># Calculate the difference between Y and N</em></p><p><em>propDiffs &lt;- PrivateProp — PublicProp</em></p><p><em># Plot the differences and include the 95% HDI lines.</em></p><p><em>hist(propDiffs,</em></p><p><em>main=”Histogram of Proportion Differences — Public vs. Private”,</em></p><p><em>col=”darkslategray4&quot;, xlab=”DIfference”)</em></p><p><em>abline(v=quantile(propDiffs,c(0.025)), col=”black”)</em></p><p><em>abline(v=quantile(propDiffs,c(0.975)), col=”black”)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*h_iQH7RkCIAh0trNBNxqPw.png" /></figure><p><em># Print the HDI interval</em></p><p><em>c(quantile(propDiffs,c(0.025)),quantile(propDiffs,c(0.975)))</em></p><p><em>## 2.5% 97.5%</em></p><p><em>## 0.006945631 0.496732448</em></p><p>When comparing the public and private school proportions, there is only a small portion, where the HDI ranges overlap with one another from 0 to around 0.15. We can see that the private school proportions are most frequent above 0.1 and the public school proportions are centered around 0.03. At this point, we take the difference between the two proportions and plot them. The lower bound of the HDI is 0.004 and the upper bound is 0.51. The two categories are not independent and the most likely shift (from private to public) is 0.257.</p><p><em>library(“BayesFactor”)</em></p><p><em>bfCorTest &lt;- function(x,y) #Get r from BayesFactor</em></p><p><em>set.seed(1234)</em></p><p><em>allSchoolsReportStatus$reported &lt;- rnorm(7381)</em></p><p><em>allSchoolsReportStatus$pubpriv &lt;- rnorm(7381)</em></p><p><em>bfCorTest(allSchoolsReportStatus$reported,allSchoolsReportStatus$pubpriv)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kj-k9ZAKKeLhQqhez256Dw.png" /></figure><p>In addition, the Bayes Factor test produces the following output while comparing public and private schools’ reporting statuses: The point estimate of rho is -0.017. The 95% HDI ranges from -0.03 to 0.00.</p><p>To understand the interpretation of Bayes Factor, the formula is = likelihood of data given Ha / likelihood of data given H0.</p><p>Bayes Factor Interpretation # &gt; 100 Extreme evidence for alternative hypothesis 30–100 Very strong evidence for alternative hypothesis 10–30 Strong evidence for alternative hypothesis 3–10 Moderate evidence for alternative hypothesis 1–3 Anecdotal evidence for alternative hypothesis 1 No evidence (https://www.statology.org/bayes-factor/)</p><p><em>#Bayes factor analysis rhoNot0 divided</em></p><p><em>1/0.0744938</em></p><p><em>## [1] 13.42394</em></p><p><em>#invert to understand odds</em></p><p>The Bayes factor of 0.0744938 means the odds are in favor of the alternative hypothesis, with 13.4:1 odds to reject the null hypothesis (Stanton, p.137).</p><p><em>setwd(“~/Downloads”)</em></p><p><em>load(“allSchoolsReportStatus.RData”)</em></p><p><em>library(rjags)</em></p><p><em>library(BEST)</em></p><p><em>allSchoolsReportStatus$rep_bin &lt;- with(allSchoolsReportStatus, ifelse(allSchoolsReportStatus$reported == “Y”, 1,0))</em></p><p><em>priv = allSchoolsReportStatus$rep_bin[allSchoolsReportStatus$pubpriv == ‘PRIVATE’]</em></p><p><em>pub = allSchoolsReportStatus$rep_bin[allSchoolsReportStatus$pubpriv == ‘PUBLIC’]</em></p><p><em># Run a t-test to compare the means of the private and public school groups.</em></p><p><em>t.test(priv, pub)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IDVZ6tbEXjUhZYEfdUaKXQ.png" /></figure><p>Finally, we can use the Welch Two Sample t-test, which compares the means of the private and public schools’ reporting statuses and compliance. The observed value of t is -13.944, while the degrees of freedom is 1835.6.</p><p>The p-value associated with the observed value is &lt; 2.2e-16. Assuming an alpha threshold of .05, we should reject the null hypothesis, which is further evidence aligned with our previous tests. The upper bound of the confidence intervals is -0.1091376 and the lower bound of the confidence interval is -0.1448623.</p><p>3. What are 2013 vaccination rates for individual vaccines (i.e., DOT, Polio, MMR, and HepB) in California public schools?</p><p>How do these rates for individual vaccines in California districts compare with overall US vaccination rates (make an informal comparison to the final observations in the time series)?</p><p><em>plot(usVaccines,col=”grey”)</em></p><p><em>abline(v=2018)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*OK2rZbEldnuZP3pZ19gilg.png" /></figure><p><em>PctPlot &lt;- plot(districts$PctUpToDate,col=”grey”)</em></p><p><em>title(‘PctUpToDate’)</em></p><p><em>abline(h=88)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*JhO-B7xpB-rIkjFGbbyS6Q.png" /></figure><p><em>mean(districts$PctUpToDate)</em></p><p><em>## [1] 87.79714</em></p><p>In the usVaccines data, the 2013 vaccination rate for DTP1 is around 98, in contrast to the rate for HepB_BD, which is around 70, and Pol3, which was around 90, while Hib3 is around 95 and MCV1 is around 93. The average of these values is 88.</p><p>The rate for PctUpToDate in California schools is visualized in a plot at 85 and above, with an average value of 87.8.</p><p>4. Among districts, how are the vaccination rates for individual vaccines related? In other words, if students are missing one vaccine are they missing all of the others?</p><p><em>plot(districts$WithoutDTP,col=”grey”)</em></p><p><em>title(‘WithoutDTP’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*DAsu0Sy0lCay_pQ3ToXaCA.png" /></figure><p><em>plot(districts$WithoutPolio,col=”grey”)</em></p><p><em>title(‘WithoutPolio’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*aTPNsACA3YbNgR45cmvP7g.png" /></figure><p><em>plot(districts$WithoutMMR,col=”grey”)</em></p><p><em>title(‘WithoutMMR’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*_v23OUOVAqzCSaMT0knbrw.png" /></figure><p><em>plot(districts$WithoutHepB,col=”grey”)</em></p><p><em>title(‘WithoutHepB’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*M6eNobbijzHEBIcYmHHndw.png" /></figure><p><em>datatable &lt;- data.frame(cbind(districts$WithoutDTP, districts$WithoutPolio, districts$WithoutMMR, districts$WithoutHepB, districts$PctUpToDate))</em></p><p><em>#if(!require(devtools)) install.packages(“devtools”) cor(datatable, method = “pearson”, use = “complete.obs”)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c3E-bW_PA9Gyz3YAw0wcoQ.png" /></figure><p>The correlation matrix shows that all these variables are strongly related. If a student is missing one vaccine, they are also likely to be missing all other vaccines in this data set. Highly correlated variables, such as WithoutDTP and WithoutPolio at a correlation value of 0.98, or several other variables that indicate that a student does not have a vaccine (i.e., WithoutDTP, WithoutPolio, WithoutMMR and their correlations with the PctUpToDate variable are -0.96, -0.94, and -0.96, respectively). In other words, they are inversely correlated. PctUpToDate increases when “Without” vaccines’ values decrease.</p><p>Predictive Analyses: (For all of these analyses, use PctChildPoverty, PctFreeMeal, PctFamilyPoverty, Enrolled, and TotalSchools as predictors. Transform variables as necessary to improve prediction and/or interpretability. In general, if there is a Bayesian version of an analysis available, you are expected to run that analysis in addition to the frequentist version of the analysis.)</p><p>5. What variables predict whether or not a district’s reporting was complete?</p><p><em>#install.packages(“lattice”)</em></p><p><em>library(lattice)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s0qJJtbYBoUsn9_90h0Iog.png" /></figure><p>The linear model above predicts whether or not a district’s reporting is complete, with the independent variables Enrolled, PctChildPoverty, PctFreeMeal, TotalSchools and PctFamilyPoverty. The null hypothesis states that there would be no difference/effect on the dependent variable. The p-value for this test is statistically significant. This means there is support for the alternative hypothesis, which states that there is a difference.</p><p>The adjusted R squared is 0.09, which means this test is describing about 9% of the variance in predicting whether a district’s reporting is complete. The statistically significant predictor variables are Enrolled and TotalSchools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OIXhFtaphiEAWBRb1aJE5Q.png" /></figure><p>The linear model above predicts whether or not a district’s reporting is PctUpToDate, with the independent variables Enrolled, PctChildPoverty, PctFreeMeal, TotalSchools and PctFamilyPoverty. The null hypothesis states that there would be no difference/effect on the dependent variable. The p-value for this test is statistically significant. This means there is support for the alternative hypothesis, which states that there is a difference.</p><p>The adjusted R squared is about 0.08, which means this test is describing about 8% of the variance that goes into predicting whether a district’s reporting is complete or compliant. The statistically significant variables are Enrolled, PctFreeMeal, PctFamilyPoverty, and TotalSchools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w-A2zB-vxS9ftGVoisXwpg.png" /></figure><p>The Bayes factor model above predicts whether or not a district’s reporting is PctUpToDate. Based on the Bayes factor of 7756649600, which is a large value, the test highly favors the alternative hypothesis and is considered strongly in favor that there is a difference, further adding evidence aligned with previous tests.</p><p>The results show the odds of 7756649600:1, in favor of the model that includes the interaction. The Bayes factor results show that the odds are in favor of a model that includes main effects of PctChildPoverty, PctFreeMeal, TotalSchools, and PctFamilyPoverty as well as the interaction between them (Stanton, p. 205).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9JkIfylmMsRNpCi9aCIIBg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QUlwzBYJXYPs1sFZUcLTLQ.png" /></figure><p>The above result is the posterior distribution, which gives us a detailed view of the coefficients and R squared. The 95% HDI for each coefficient is the likely distribution of the population values for each coefficient (Stanton, p. 205).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UJNQbcP51mhtUOE_jds3sA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MzgA4FRi0lM3HwN9oYngtg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*lYVAlzuz7O822EO4mPY27A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*-wY-LZIW1byXdZ_3u2jNQA.png" /></figure><p>The command above indicated progress of the MCMlogit() command. “After 10,000 iterations of the algorithm, the height of the corresponding black lines show the coefficient value for that iteration. The true population value of each coefficient is likely to be somewhere near the middle of each distribution and less likely to be out by the tails” (Stanton, p.231). These tests provide more information on the variables, which help to predict whether or not a district’s reporting was complete.</p><p>6. What variables predict the percentage of all enrolled students with completely up-to-date vaccines?</p><p><em>#install.packages(“sqldf”)</em></p><p><em>library(sqldf)</em></p><p><em>df2 &lt;- sqldf(‘select Enrolled, PctUpToDate, WithoutDTP, WithoutPolio, WithoutMMR, WithoutHepB, PctBeliefExempt, PctChildPoverty, TotalSchools,PctFreeMeal, PctFamilyPoverty from districts where PctUpToDate = 100&#39;)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TGrwcqEg5DBuS4IsJaH2Vw.png" /></figure><p>The above linear model predicts PctUpToDate variables valued at 100, with predictor variables from the data set. The variables did not show up as significant, and neither did the test itself. This means this specific test is relatively unreliable.</p><p>The previous tests, which predicted PctUpToDate indicated helpful variables in predicting overall PctUpToDate, at values including 100 and below. The significant variables included Enrolled, PctFreeMeal, PctFamilyPoverty, and TotalSchools.</p><p>(Refer to page 20–21 for the Bayes analysis of this predicted variable.)</p><p><em>districts$Pct_bin &lt;- with(districts, ifelse(districts$PctUpToDate == “100”, 1,0))</em></p><p><em>library(MCMCpack)</em></p><p><em>bayes1 &lt;- MCMClogit(formula=Pct_bin ~ DistrictComplete + Enrolled +</em></p><p><em>PctChildPoverty + PctFreeMeal + TotalSchools + PctFamilyPoverty,</em></p><p><em>data = districts)</em></p><p><em>LogOdds &lt;- as.matrix(bayes1[,”PctFamilyPoverty”]) #create matrix for apply()</em></p><p><em>LogOdds &lt;- apply(LogOdds, 1, exp) #apply() runs exp() for each</em></p><p><em>hist(LogOdds) # show a histogram</em></p><p><em>abline(v=quantile(LogOdds,c(0.25)),col=”black”) #Left edge of 95% HDI</em></p><p><em>abline(v=quantile(LogOdds,c(0.975)),col=”black”) #Right edge of 95% HDI</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*FX49GwN4iRxybk7JA3pxFQ.png" /></figure><p>Another way to analyze PctUpToDate for school vaccine reporting and compliance is with odds testing. We can use the significant predictors to analyze the odds in increasing the likelihood of PctUpToDate.</p><p>The PctFamilyPoverty coefficient is entered into a matrix and submit to the apply() function. Using apply() in this way runs exp() repeatedly on each row in the LogOdds object, thereby converting each of the 10,000 values in the posterior distribution from log-odds to regular odds (Stanton, p.231).</p><p>The figure above shows the distribution centered on about 1.025, suggesting about a 2% increase in the likelihood</p><p>of a 100% PctUpToDate student for an increase in PctFamilyPoverty of 1 student. This analysis further substantiates the evidence that PctUpToDate can be predicted with variables similar to PctFamilyPoverty, including PctFreeMeal and PctChildPoverty (Stanton, p.231).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*krEmnYckl9fUEieIr8QJOQ.png" /></figure><p>The above output tests the DistrictComplete status with the independent variables from a linear regression analysis. We see that the significant variables are Enrolled, TotalSchools, PctFreeMeal and PctFamilyPoverty. The adjusted R squared value is 0.09. In other words, this regression equation explains about 8% of the total variance in the DistrictComplete compliance outcome.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gVHk7E8kF_CrJu0hg9NgiQ.png" /></figure><p>The above output tests the PctUpToDate status with the independent variables from a linear regression analysis. We see again that the significant variables are Enrolled, TotalSchools, PctFreeMeal and PctFamilyPoverty. The adjusted R squared value is 0.08. In other words, this linear regression equation explains about 8% of the total variance in the PctUpToDate outcome.</p><p>The null hypothesis states that there would be no difference/effect on the dependent variable. The p-value for this test is statistically significant. This means there is support for the alternative hypothesis, which states that there is a difference.</p><p>7. What variables predict the percentage of all enrolled students with belief exceptions?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1y-hX3keQCgYefcn04o3cQ.png" /></figure><p>The statically significant variables that predict the percentage of all enrolled students with belief exceptions include PctUpToDate, PctChildPoverty and PctFreeMeal. They are mostly inversely related to PctBeliefExempt. The p-value of the above output has a p-value of &lt; 2.2e-16, which is less than 0.05 and is in support of rejecting the null hypothesis, in favor of the alternative hypothesis.</p><p><em>#install.packages(“lattice”)</em></p><p><em>library(lattice)</em></p><p><em>lmBF1 &lt;- lmBF(PctBeliefExempt ~ PctUpToDate +Enrolled + PctChildPoverty +</em></p><p><em>PctFreeMeal + TotalSchools + PctFamilyPoverty, data = districts)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UeOkTjik9EiZcbfcJ3tRPA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m8AquwxuNKsxTLOGh_-_Nw.png" /></figure><p>The Bayes factor model above predicts whether or not a student is PctBeliefExempt using independent variables in the data set for prediction. The null hypothesis states that there would be no difference/effect on the dependent variable. Based on the Bayes factor of 3.548845e+111, which is a large value, the test highly favors the alternative hypothesis and is considered strongly in favor that there is a difference/effect.</p><p>The Bayes factor results show that the odds are in favor of a model that includes main effects of PctChildPoverty, PctFreeMeal, TotalSchools, and PctFamilyPoverty as well as the interaction between them (Stanton, p. 205).</p><p>The above result is the posterior distribution, which gives us a detailed view of the coefficients and R squared. The 95% HDI for each coefficient is the likely distribution of the population values for each coefficient (Stanton, p. 205).</p><p>8. What’s the big picture, based on all of the foregoing analyses? The staff member in the state legislator’s office is interested to know how to allocate financial assistance to school districts to improve both their vaccination rates and their reporting compliance. What have you learned from the data and analyses that might inform this question?</p><p><em>plot(districts,col=”blue”)</em></p><p><em>title(‘districts’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*6ry9pu1-nKe9xMc2qkMUwg.png" /></figure><p><em>datatable1 &lt;- data.frame(cbind(districts$PctBeliefExempt, districts$PctChildPoverty, districts$PctFreeMeal, districts$PctFamilyPoverty, districts$PctUpToDate, districts$DistrictComplete, districts$Enrolled))</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6756YD-uJ4MKd9MFEpmSqA.png" /></figure><p><strong>CONCLUSIONS</strong></p><p>The scatter plot matrix above shows the various relationships within the data at a high level. When “Without” vaccination values increase, “PctUpToDate” values decrease. Another interesting thing to note is when “PctBeliefExempt” goes up, the “Without” vaccine variables also increase, while “PctUpToDate” decreases.</p><p>The correlation matrix measures values to see which variables are correlated with one another, with respect to student beliefs, poverty and “PctUpToDate” and reporting compliance.</p><p>Before allocating funds, the state legislator’s office may also look into schools within “allSchoolsReportStatus” to confirm a reported status of “N”, or no, ask for an update, and/or a percentage of students who were vaccinated at this point in time.</p><p>After looking at the data in the correlation matrix, we note that as “PctChildPoverty,” “PctFamilyPoverty” and “PctFreeMeal” values decrease, “PctBeliefExempt” values increase. In other words, if a student claims to be exempt from getting vaccinated, their likelihood of being in poverty decreases. This student pocket is important to note, because it causes low rates of complete vaccinations. The correlation between PctUpToDate and BeliefExempt is -0.722, which means they are inversely related. We would need to try to get this group vaccinated, or at least reported, in order to boost compliance and/or vaccination rates.</p><p>“PctUpToDate” values increase slightly with the 3 poverty related variables, which means it would be best to allocate funds to vaccinating those students, in order to increase “PctUpToDate” and eventually complete each district’s vaccine rate reporting.</p><p>The best way to allocate financial assistance to school districts is to allocate funds to the districts with the lowest/minimum values related to all the “Without” (vaccine) variables, which will eventually add up to higher totals for “PctUpToDate,” for a more complete compliance report by school district.</p><p>We can conclude that targeting specific districts with higher poverty rates will allow the state legislator’s office to boost vaccination rates and reporting compliance. The funded program may send health aides to schools for free vaccinations in poverty stricken districts. We found that students with belief exemptions are also less likely to be vaccinated. We may try to vaccinate students around the age of 18, or sooner, when a belief exemption may have changed. This will simultaneously help with reporting compliance while vaccinating those in need.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c0d4fca7d01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Association Rule Mining]]></title>
            <link>https://lisalondon.medium.com/association-rule-mining-286c49cd3f16?source=rss-773031daa634------2</link>
            <guid isPermaLink="false">https://medium.com/p/286c49cd3f16</guid>
            <category><![CDATA[association-rule-mining]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Lisa]]></dc:creator>
            <pubDate>Sun, 09 Apr 2023 15:28:44 GMT</pubDate>
            <atom:updated>2024-01-25T22:10:51.468Z</atom:updated>
            <content:encoded><![CDATA[<h3>Uncovering Customer Patterns with Association Rule Mining</h3><p>In this exciting journey through association rule mining, we’ll explore a potent technique that sheds light on hidden patterns within datasets. This blog delves into a real-world case study where we unearth insights using historical customer data.</p><p>For this analysis, I used Association rule mining (ARM), which is a powerful data mining technique that discovers interesting relationships and patterns in large datasets. Its tremendous versatility has made it a valuable tool across a wide range of industries.</p><p><strong>Introduction:</strong><br>ARM aims to identify association rules that show the co-occurrence of items or events in a dataset. These rules can be used to uncover hidden patterns, predict future trends, and make informed decisions. By leveraging the relationships between items, ARM has proven instrumental in enhancing business performance, improving customer satisfaction, and driving innovation.</p><p><strong>Key Concepts and Algorithms:</strong><br>The fundamental concepts of ARM include support, confidence, and lift. These measures help determine the strength and significance of associations. Apriori and FP-Growth are two widely used algorithms for ARM. Apriori employs a level-wise approach to generate candidate rules, while FP-Growth utilizes a frequent pattern tree to efficiently discover associations.</p><p><strong>Real-World Applications:</strong><br>ARM has been successfully applied across numerous industries, including retail, healthcare, finance, and manufacturing.</p><ul><li><strong>Retail:</strong> ARM is used to analyze customer purchase data, identify frequently bought-together items, and optimize product placement.</li><li><strong>Healthcare:</strong> ARM helps uncover relationships between symptoms, diseases, and treatments, aiding in diagnosis and personalized medicine.</li><li><strong>Finance:</strong> ARM is employed to detect fraudulent transactions, assess credit risk, and optimize investment portfolios.</li><li><strong>Manufacturing:</strong> ARM is utilized to identify production inefficiencies, optimize supply chain management, and improve quality control.</li></ul><p>This analysis leverages customer data to find patterns at a financial institution. Customers of the bank were analyzed to find out if they were likely to join a Personal Equity Plan (PEP), based on historical data patterns.</p><h4>Step 1: Data Preparation</h4><p>The purpose of association rule mining is to itemize data sets and create “rules” based on the interaction between that data’s attributes. This can be read as an “if” -&gt; “ then” statement, where if one variable is chosen, then another variable is also chosen, and also measured by frequency. Commonly paired item sets are categorized as rules, and ordered by validation methods such as confidence, lift and support.</p><p>We transformed continuous variables, such as income and age, into categorical ones to optimize model performance. This transformation process, known as discretization and binarization, allowed us to work with data in a format that facilitates the discovery of patterns.</p><p>In this dataset, income is considered a continuous variable, which makes it a candidate for binarization. Age is an integer, which can be combined into groups to create categories for ease of processing. We converted these values into categorical data, setting the model up for best performance. “Consequently, discretization or binarization is performed in a way that satisfies a criterion that is thought to have a relationship to good performance for the data mining task to be considered.”<a href="#_edn2">[ii]</a></p><p>“Discretization is typically applied to attributes that are used in classification or association analysis. Transformation of a continuous attribute to a categorical attribute involves two subtasks: deciding how many categories, n, to have and determining how to map the values of the continuous attribute to these categories.</p><p>The result can be represented either as a set of intervals <br>{(x0, x1], (x1, x2 ],… (xn-1, xn )}, <br>where x0 and xn can be a series of inequalities <br>x0 &lt; x ≤ x1,…, xn-1 &lt; x &lt; xn.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*7SiMsy79fhI55woCGCfRUA.png" /><figcaption>Data Structure</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*Xq1vYjUw7NuW8y8szuJ7ig.png" /><figcaption><em>Data Prepped</em></figcaption></figure><h4>Step 2: Association Rule Mining Model</h4><p>We employed the association rule mining model, inspired by the “market-basket” data structure. This model unveils relationships between items and baskets, enabling us to uncover customer buying behaviors and preferences. By examining frequent item combinations, we were able to formulate association rules, represented as {X -&gt; Y}, where {X} represents a set of items and {Y} indicates a single item. These rules highlight the likelihood of {Y} appearing in a basket given the presence of all items in {X}.</p><p>Association rule mining is based on a “market-basket” model of data. This is essentially a many-many relationship between two kinds of elements, called items and baskets (also called transactions) with some assumptions about the shape of the data (Leskovec, Rajaraman, &amp; Ullman, 2020).<a href="#_edn3">[iii]</a></p><blockquote>“The form of an association rule is {X -&gt; Y}, where {X} is a set of items and {Y} is an item. The implication of this association rule is that if all the items in {X} appear in some basket, then {Y} is “likely” to appear in that basket as well.</blockquote><blockquote>{X} is also called antecedent or left-hand-side (LHS) and<br>{Y} is called consequent or right-hand-side (RHS).</blockquote><blockquote>An example association rule for products from Apple could be <br>{Apple iPad, Apple iPad Cover} -&gt; {Apple Pencil}, <br>meaning that if Apple’s iPad and iPad Cover {X} are bought, customers are also likely to buy Apple’s Pencil {Y}.</blockquote><blockquote>Notice that the logical implication symbol “-&gt;” does not indicate a causal relationship between {X} and {Y}. It is merely an estimate of the conditional probability of {Y} given {X}.”<a href="#_edn4"><strong>[iv]</strong></a></blockquote><p>Next, I developed code to categorize ‘transactions’ from the data into ‘baskets’ to identify most frequent items. Those items are then plotted to a graph, which will allow the viewer to identify item frequency in the data.</p><h4>Item Frequency Plot</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*fDIfQ4sGIrR0wm_5652L3w.png" /></figure><p>Association rule mining transactions surfaces most frequent items in data by count. Data has been re-categorized to describe each variable in detail. The data shows that most customers have a current account and a savings account. There are more married than unmarried customers, or they are not disclosing their personal information. Many customers have an income in the range of $21K — $49K and do not have a mortgage. About half of the customers in this data selection are male and the other half are female. Most of the customers from this sample are 18–59, and without children. It will take more analysis to understand association rules behind a customer’s intent to join the PEP (Personal Equity Plan). Next steps are to run an apriori algorithm to find rules with high lift, confidence, and support to back up the rules found in the model.</p><blockquote><em>SortedRules &lt;- sort(rules, by=”confidence”, decreasing=TRUE)</em></blockquote><blockquote><em>inspect(SortedRules[1:10])</em></blockquote><blockquote><em>SortedRulesL &lt;- sort(rules, by=”lift”, decreasing=TRUE)</em></blockquote><blockquote><em>inspect(SortedRulesL[1:10])</em></blockquote><h4>High Confidence and High Lift Rules</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*nMlBKEIVMk4eC_IIBLyn5Q.png" /></figure><p><strong>Top 5 most “interesting” rules with Support, Confidence and Lift values</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QCmcvJ_n7RoazEMCAr64nw.png" /></figure><h4>Association Rules Output</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*yZyUBYtCWKVXixgCBHhLYw.png" /></figure><p>An understanding of each business begins with the core customer of that business. Each rule will allow the analyst to understand the business’ customer, and likelihood of enrolling in a Personal Equity Plan. As the company understands the patterns within, they can better target those customers with customized ads/messaging for those products and services. The best rule forward is to maintain an understanding of which customers are most interested in the PEP program and which customers are most likely to join soon. Patterns for participants in the PEP program are customers with an income of $50K and above, living in a suburban neighborhood, or with 1–2 children.</p><h4>Item Frequency Plot with PEP_YES Rules</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*hL01tqi3hj56IZ5HCfK1TA.png" /></figure><p>The pattern for participants in the Private Equity Plan include customers with an income above $50k, living in a suburban neighborhood and/or with 1–2 children. The counts for these rules in this sample add up to 36. More counts include other variations of this rule, with additional attributes such as age 60+, male and/or with a mortgage. These patterns can be leveraged by sending customers in this demographic custom messaging informing them of the PEP plan and allowing them to sign up at a discount rate.</p><p>These rules are also leveraged to associate one product with another. A customer who signs up for the PEP plan may also be interested in another product/service. Advertising may also be programmatically delivered to customers who are not members of this bank but might be interested in signing up for this plan. For all the above reasons and more, it’s beneficial to gain a deeper understanding of customers to develop strategic business opportunities within your company and externally (i.e., discovering new customers).</p><blockquote><em>#Visualize Rules</em></blockquote><blockquote><em>plot(rules,method=’graph’,interactive=TRUE,shading=NA)</em></blockquote><h4>Association Rules for Private Equity Plan</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*0HpA8hqHG6HOCJ_OrbEGIQ.png" /><figcaption>Figure 1: The plot above displays all the rules produced by the association rule model, that led a customer to participate in the PEP plan.</figcaption></figure><h4>Top 10 Association Rules for Private Equity Plan (by Lift)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*edXxCfpGLVC6Uas1k0emlw.png" /><figcaption>Figure 2: The plot above displays the top 10 association rules for participation in the Private Equity Plan (PEP).</figcaption></figure><h4>Top 10 Association Rules for Private Equity Plan (by Lift)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*WOSSO83e4xXFEesElLwE8g.png" /><figcaption>Figure 3: The plot indicates that if a customer has an income of $50K+ and 1–2 children, then they are likely to have a Personal Equity Plan as well: {$50K+, 2 -&gt; PEP_YES}.</figcaption></figure><p><strong>Unveiling Customer Patterns</strong>:<br>Applying association rule mining to our customer data, we uncovered valuable insights into their behaviors and preferences. The analysis revealed that customers with a higher income, residing in suburban neighborhoods, and having one or two children are more likely to join the Personal Equity Plan (PEP). This significant finding allows us to tailor targeted marketing campaigns and personalized messaging to maximize enrollment.</p><p><strong>Identifying Business Opportunities</strong>:<br>Beyond understanding customer patterns, association rule mining empowers businesses to identify strategic opportunities. By recognizing common attributes and behaviors among PEP participants, companies can proactively engage and upsell additional products or services. This data-driven approach strengthens customer relationships, drives conversions, and ultimately boosts revenue.</p><p><strong>Conclusion</strong>:<br>Association rule mining serves as a powerful tool for uncovering hidden patterns and relationships within datasets. By meticulously preparing data and employing the market-basket model, we successfully identified customer segments with higher propensity to join a Personal Equity Plan. Armed with these insights, companies can refine their marketing strategies, enhance customer targeting, and drive business growth.</p><p><a href="#_ednref1"><em>[i]</em></a><em> Source: Introduction to Data Mining, Second Edition, Tan, Steinbach, Karpatne, Kumar, Page 63.</em></p><p><a href="#_ednref2"><em>[ii]</em></a><em> Source: Introduction to Data Mining, Second Edition, Tan, Steinbach, Karpatne, Kumar, Page 63.</em></p><p><a href="#_ednref3"><em>[iii]</em></a><em> Leskovec, J., Rajaraman, A., &amp; Ullman, J. D. (2020). Mining of massive data sets. Cambridge university press.</em></p><p><a href="#_ednref4"><em>[iv]</em></a><em> </em><a href="https://www.kirenz.com/post/2020-05-14-r-association-rule-mining/"><em>https://www.kirenz.com/post/2020-05-14-r-association-rule-mining/</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=286c49cd3f16" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tracking Student Progress with Linear Regression and Apriori Association Rule Mining]]></title>
            <link>https://lisalondon.medium.com/tracking-student-progress-with-linear-regression-and-apriori-association-rule-mining-8d3ebb6343d0?source=rss-773031daa634------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d3ebb6343d0</guid>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[association-rule-mining]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Lisa]]></dc:creator>
            <pubDate>Sun, 09 Apr 2023 15:01:03 GMT</pubDate>
            <atom:updated>2025-02-21T23:16:40.155Z</atom:updated>
            <content:encoded><![CDATA[<h3>Tracking Student Progress with Association Rule Mining &amp; Regression Models</h3><h4>INTRODUCTION: GENERAL</h4><p>Association rule mining (ARM) is a powerful data mining technique that uncovers relationships and patterns within large datasets. Its versatility lies in its ability to identify hidden associations, correlations, and sequential patterns, making it a valuable tool for a wide range of industries and applications. There are numerous ways ARM models can be employed to make predictions and unique applications where ARM has been leveraged for accurate and insightful forecasting.</p><p><strong>Real-World Examples of Association Rule Mining</strong></p><ul><li>Retail: ARM is used to analyze customer purchase patterns, identify frequently bought together items, and optimize product placements within stores.</li><li>Healthcare: ARM helps identify patterns in medical data, such as co-occurring diseases, drug interactions, and patient outcomes, aiding in diagnosis and treatment decisions.</li><li>Finance: ARM is employed to detect fraudulent transactions, analyze stock market trends, and predict financial risk.</li><li>Manufacturing: ARM is used to optimize production processes, identify bottlenecks, and minimize waste.</li><li>Transportation: ARM is applied to analyze traffic patterns, predict congestion, and plan efficient routes for logistics and public transportation.</li></ul><p><strong>Education: Application of Association Rule Mining</strong></p><p>In this example, student data is a dataset collected from different schools, tracking each school by section and section status. Based on the description, “5 schools (A, B, C, D and E) are implementing the same math course this semester, with 35 lessons. There are 30 sections total. The semester is about 3/4 of the way through.”</p><p>“For each section, the data tracks the number of students who are:<br>• very ahead (more than 5 lessons ahead)<br>• middling (5 lessons ahead to 0 lessons ahead)<br>• behind (1 to 5 lessons behind)<br>• more behind (6 to 10 lessons behind)<br>• very behind (more than 10 lessons behind)<br>• completed (finished with the course).”</p><p>The idea behind this analysis is to provide summary statistics, visualizations, and insights about the data set to inform relevant stakeholders about the ongoing progress of students in their math sections, at their respective schools. Stakeholders may include parents, school board members, students, governments, and funding agencies. The value of comparing schools can provide information to parents and agencies on the quality of the education and identify areas/sections that need improvement. As data is collected over many years, stakeholders can implement new methods to boost scores and help students with additional resources, as necessary.</p><h3>ANALYSIS AND MODELS: ABOUT THE DATA</h3><p>The data set contains 8 variables with 30 observations. The variables include school, section, and the relevant status of each student in that section with categories for being “ahead”, “behind”, “completed”, etc. After observing that there are no null values within the data, the next step was data transformation. The “Section” variable needed transformation from integers to factors of 1–13. Now, this column can be analyzed as a categorical value, rather than a numerical one.</p><p>Other transformations include adding a categorical variable that leverages an “if else” statement, separating the category “Completed” into “ok”, “good” and “great”. The idea behind this transformation is to focus on the schools that have most completes (greater than &gt;10) and highlight them as the most successful. Perhaps other schools can incorporate their techniques into the classrooms that need the most improvement, i.e., the Very Behind category. A linear and association rule model will be created to understand the impact of the feature variables and association rules in the dataset. Further strategies can be implemented once those key relationships are identified.</p><h3>Linear Regression and Apriori Association Rule Mining</h3><p>Linear regression is a supervised algorithm, and association rule mining is unsupervised machine learning. Linear and stepwise regression is used to select features (variables) within the model that affect the outcome “Completed”. Are there patterns and/or relationships that can be explored in the data? Association analysis is “used to discover patterns that describe strongly associated features in the data” (Tan, Steinbach, Karpatne, Kumar, 2019).</p><h3>RESULTS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*1__oOjlJbHahOfo7PYgsqw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*P7ESvjJeNhelLGd5L3q6xA.png" /><figcaption>Figures 1–2: Students who have completed the program by section shows the most successful students so far. As section number increases, there is a slight increase in completes. After testing the two variables, there is a 0.23 positive correlation between ‘Section’ and students who are categorized as ‘Completed’. The majority of the data is from schools A and B. There may be missing data or incomplete data, which explains the difference in values. An alternative explanation is that these schools have more students, or the data might be comparing private to public schools.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*hk7mY6Nh8IeyCt8lKUBbZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*_4NMX2x3S3bYlQr3w3AscA.png" /><figcaption>Figures 3–4: Graphs show boxplot and histogram of “Number of Completed by School” and “Frequencies of Completed” variable. The trend shows that students at schools E, A &amp; B are most likely to be categorized as ‘Completed’. The mean number of Completed students is 10.5, in contrast to the mean for Middling at 7.4 and Very Behind at 6.9.</figcaption></figure><h3>ANALYSIS BY STUDENT COMPLETES — SCHOOL &amp; SECTION — STUDENT “COMPLETED” BY SCHOOL</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*82YfFfMKTnX2lgusYOqd0w.png" /><figcaption>Figure 5: School and number of students who completed their math course, color coordinated by Section. Key points to note include Section 1 at School E, Sections 12, 11, 10, 12, 7, and 6 at Schools A &amp; B. These schools have a high number of students, who are done with their math section about 3/4ths through the semester. E is especially interesting at 27 completes when the semester is not over yet. This school would be a great model for other sections and/or may be pointing to a section that is more mathematically inclined/advanced.</figcaption></figure><h3>ANALYSIS BY SECTION &amp; SUBGROUP: BOXPLOT ANALYSIS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*0kaof08jDb9L6N1p921bUA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*wU4X2bx3ujG2qaSZgch38A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*B_oncwT46UeSOBBgufVa8w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*SIvhG_Bngm88TpmcYFJP5g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*iNLnYtZThMC8XgAAV_zbOA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*LkzUOaIDznXyAmYrhTBP6w.png" /><figcaption>Figures 6–11: An analysis of boxplots by section status showcase how many students are very behind in the semester, when it is 3/4ths through the semester. The three variables Very Behind, More Behind and Behind indicate trouble for those students. Additional attention will need to be paid to get those students up to speed.</figcaption></figure><h3>ANALYSIS BY SECTION &amp; SUBGROUP: BOXPLOT ANALYSIS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*_NDDpCYQW9g8mY5YTDtFsA.png" /><figcaption>Figure 12: Based on the correlogram analysis, the variables Completed, and Section have a small positive correlation (0.237). This means as section number increases, so does number of completed students. On the other end of the spectrum, Very Behind and Completed are not as correlated, at -0.152. As section decreases, the very behind number increases.</figcaption></figure><h3>CORRELATION MATRIX ANALYSIS — VARIABLE OUTCOME ‘COMPLETED’</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*HoB4DVoJ7D6P58Zc21mt-Q.png" /><figcaption>Figure 13: Graphed above — the progressive relationship between Middling, Behind, More Behind, Very Behind and Completed. Visualizing status progress with Completed.</figcaption></figure><h3>CORRELATION MATRIX (GRAPHED ABOVE)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n3YTaCosD3s8CKzdPMZnqQ.png" /></figure><p>The highest correlated relationships between these variables were Behind and Very Behind at a positive correlation of 0.72, and Very Behind and More Behind (0.66). The third highest correlation is More Behind and Behind (0.56).</p><h3>ASSOCIATION RULE MINING</h3><p>Best rules found:<br>Based on association rule mining, schools A and B led to 25 students categorized as not very ahead. The more behind category was also likely to lead to students not being in the very ahead category.</p><h3>LINEAR REGRESSION OUTPUT</h3><p>MODEL INFO:<br>Observations: 30<br>Dependent Variable: Completed<br>Type: OLS linear regression<br>MODEL FIT:<br>F(20,9) = 6.39, p = 0.00<br><strong>R² = 0.93</strong><br>Adj. R² = 0.79<br>Standard errors: OLS<br>— — — — — — — — — — — — — — — — — — — — — — — — -<br>Est. S.E. t val. p<br>— — — — — — — — — — — — — — — — — — — — — — —<br>(Intercept) 8.66 4.97 1.74 0.12<br>SchoolB 0.63 3.06 0.21 0.84<br>SchoolC 1.44 3.89 0.37 0.72<br>SchoolD -7.13 4.88 -1.46 0.18<br>SchoolE 12.88 3.55 3.63 0.01<br>Section2 -10.03 2.62 -3.83 0.00<br>Section3 -5.68 2.53 -2.25 0.05<br>Section4 -4.44 3.12 -1.42 0.19<br>Section5 -0.87 3.11 -0.28 0.79<br>Section6 1.43 3.25 0.44 0.67<br>Section7 1.00 3.94 0.25 0.81<br>Section8 -8.10 3.01 -2.69 0.02<br>Section9 -5.84 3.01 -1.94 0.08<br>Section10 4.03 3.15 1.28 0.23<br>Section11 -0.22 2.97 -0.07 0.94<br>Section12 4.52 3.47 1.30 0.22<br>Section13 -7.92 4.09 -1.93 0.09<br>Middling 0.46 0.21 2.16 0.06<br>Behind -0.04 0.09 -0.40 0.70<br>More_Behind 0.75 0.29 2.55 0.03<br>Very_Behind -0.19 0.20 -0.93 0.38</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*7FDWMraEO1m3e_I6Arhc-Q.png" /><figcaption>Figure 14: The regression output highlights variables that have a significant impact on Completed outcome. The greatest feature impact on a student’s chances of completing their math section early were attending School E (positive relationship 10.88), sitting in Section 2 (negative impact -10.03) or Section 8 (-8.10). A positive relationship means that for every student in the Completed category, number of students from School E also increased. The inverse is true for a negative relationship.</figcaption></figure><h3>AVERAGE “COMPLETED” BY SCHOOL</h3><p>by (Completed, School, mean)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1eg8cJSJoBhxSXyFeEjrWw.png" /></figure><p>The mean (average) number of students who are in the completed category varied, based on school attended. School D had the least value and School E had the highest average.</p><h3>CONCLUSION</h3><p>After exploring the variables and visualizing relationships from correlation tests, association rules and linear regression, stakeholders are more prepared to brainstorm strategic actions and implement new procedures for optimizing school outcomes. Students in the very behind category may need to repeat their course next year, but we can track certain concepts and apply extra focus on those concepts early on next year. Tracking KPIs (key performance indicators) over time will allow board members, parents, teachers, and other stakeholders contemplate performance after newly implemented strategies.</p><p>Recommendations include implementing math strategies from School E at other schools, so that we can see the progress of sections that were behind. Other recommendations include tracking additional KPIs. More data would be ideal for this analysis. Variables of preference include test scores, AP enrollment and graduation rates. A few hypotheses can be theorized based on this data. For example, in theory, as early completes increases, the more likely a school is on track for graduation goals. Students can then tackle advanced concepts and/or become candidates for AP classes (association rules). The more early completes, there may also be a higher likelihood of higher test scores. Other analyses could investigate the Very Behind category and understand feature impact, best association rules and relationship plots.</p><p><em>Based on an assignment from the Syracuse University iSchool, Masters in Applied Data Science program, Data Analytics course.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d3ebb6343d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Collaboration at Work]]></title>
            <link>https://lisalondon.medium.com/collaboration-at-work-63cff060b986?source=rss-773031daa634------2</link>
            <guid isPermaLink="false">https://medium.com/p/63cff060b986</guid>
            <dc:creator><![CDATA[Lisa]]></dc:creator>
            <pubDate>Sat, 08 Apr 2023 23:49:11 GMT</pubDate>
            <atom:updated>2023-06-09T20:59:06.238Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*OUmPWq3Cu36LPyppi0vbfg.jpeg" /></figure><h3>📖 BACKGROUND</h3><p>The analytics department of a multinational company, and the head of HR wants help mapping out the company’s employee network using message data.</p><p>They plan to use the network map to understand interdepartmental dynamics better and explore how the company shares information. The goal of this project is to think of ways to improve collaboration throughout the company.</p><h3>💾 THE DATA</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZM7Z6qesSw4q6HSAs0Chhw.png" /></figure><h3><strong>EXECUTIVE SUMMARY</strong></h3><p><strong><em>How Collaborative Intelligence Can Boost Business Innovation</em><br></strong> “Ineffective meetings cost businesses $541 billion a year globally in lost productivity and employee time, according to Doodle’s 2019 <a href="https://normadesign.it/realizzazione-siti-web-torino/meeting-report/">State of the Meeting report</a>. And according to <a href="https://www.gallup.com/workplace/231668/dismal-employee-engagement-sign-global-mismanagement.aspx">Gallup’s State of the Global Workplace report</a>, as much as 85% of employee time may be wasted on inefficient collaboration.</p><p>But the real costs of such disconnection go even deeper than money.</p><p>In a disconnected organization, innovation grinds to a halt. Individuals’ and teams’ great insights get trapped in silos. Employees feel stifled — unheard and disengaged. And with record numbers of knowledge workers quitting their jobs in the “Great Resignation,” the risks of disconnection are more urgent than ever.</p><p>Simply put: your organization can’t afford to have a disconnected workforce.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h4L5o1TUON9OU6VMKknUZQ.png" /></figure><p><strong><em>The Five Principles of Collaborative Intelligence</em></strong></p><p>“1. People are greater together than apart.</p><p>Teams can’t exist without connection.</p><p>Collaboration should not be left to chance.</p><p>Collaboration spaces power connected teams.</p><p>Measuring collaboration is possible and essential.”<a href="#_edn2">[ii]</a></p><p><strong><em>Business Questions</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2JyHPjiLwi7KZZ-nplYq5w.png" /></figure><p><strong><em>Feature Engineering</em></strong><em><br></em> <br> Various feature engineering tasks visualized above were performed on the data set, including binning, one-hot encoding, removing outliers, feature extraction and creation. At the same time, a few data gaps were noted for future analysis purposes. Once the data was cleansed and prepped, data exploration began. After data cleansing, ML models are readily deployable, and the accuracy of models may be compared.<br> <br><strong> <em>What is Feature Engineering?</em></strong></p><p>“Feature engineering is a machine learning technique that leverages data to create new variables that aren’t in the training set. It can produce new features for both supervised and unsupervised learning, with the goal of <strong>simplifying and speeding up data transformations</strong> while also <strong>enhancing model accuracy</strong>.”2</p><p>“Distance-based algorithms like k-means require scaled continuous features as model input.”1</p><p><strong><em>Data Exploration</em></strong></p><p>The purpose of exploring the data is to identify trends, anomalies, and relationships within to data, to leverage in the inter-departmental analysis. Plots like histograms help to determine variable frequency, and distribution. Calculations like averages of variables allow the ML practitioner to convert variables into above and below average categories. This will be helpful for variables such as message count and message length.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fuJA2uaEQov6zQUMDa7q8g.png" /></figure><p><strong><em>Insights</em></strong></p><p>Sales, Admin, IT, Engineering and Marketing have been graphed by total message count and message length. By tracking the count of messages by department and grouping them by location, we can analyze which departments are collaborating and which ones may be recommended for an increased collaboration incentivized measure. <br> <br> Departments like Engineering, IT and Marketing could increase collaboration, but may also be working on “heads down” assignments. Consideration of utilization rate will allow HR and analytics to decide whether boosting collaboration should be considered. Options for increasing collaboration include networking retreats, which will not interfere with their “heads down work” assignments.</p><p><em>Which departments would you recommend the HR team focus to boost collaboration?</em></p><p>IT, Engineering and Marketing appear to send and receive the least number of messages. The above chart shows which departments are sending lengthier messages, grouped by location and employee id. From a high-level view, the HR department may choose to boost productivity among teams that are in the below-average messaging category, such as IT, Engineering and Marketing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1262c_8ItA22bKDg7hJW5g.png" /></figure><p>The plot shows high lift rules, which display employee messaging, and when there is a higher likelihood they will be sending a message to another employee, connected by employee id numbers, above. As a result, we can see collaboration among teams within the organization, including co-collaborators. Now that we have this information, we can leverage it to boost collaboration through inter-organizational collaboration tools. For example, Microsoft products are leveraging inter-departmental relationships, by reminding users to send an email to collaborating team members, when vacation dates and subsequent missed meetings are nearing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*om5wk6xi_uJXY0kYNuMyKg.png" /></figure><p><em>Which employee has the most connections?</em></p><p>Based on network, Employee 1881 appears to be influential, with a few close collaborators. Other high ‘lift’ collaborators in the organization are 1616, 1646, and 1308.</p><p><em>Which employees are most commonly collaborating with each other?</em></p><p>Rules are read as follows: <br> If Employee 1299, then Employee 1646 is highly likely to be the receiver or sender of a message, with a lift of 300 and an 85% confidence.</p><p>Other similar rules include employees 403 &amp; 344 and 1616 &amp; 1252.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*CBwFtlt9i17uwmMX7E7hmw.png" /></figure><p>Item frequency graph above shows employee ids that are most seen in the data set, either sending or receiving messages. These employees are the collaboration “rockstars” in their departments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*IsT4rliNXR-XGClPPgawyA.png" /></figure><p>These rules have high confidence and high lift, which means that an employee on the left is likely to be collaborating with the other, on the right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*OggCUPVjeDj86lwJgXZCRQ.png" /></figure><p>These inter-departmental networks are the basis for collaboration and innovation. Once we understand the social networks within our departments and organizations, we can strategize clustering for similar groups, based on preferences.<br> <br> This can lead to recommendation systems that can recommend other employees “you may know” and/or want to collaborate with.</p><p>Collaboration can include many factors, such as communication, innovation, and participation. One way to encourage employee collaboration is with competitions, retreats, and rewards. We can track collaboration progress with KPI’s. For example, groups/employees that have the highest lift (change) in collaboration might be incentivized to win awards. Other ideas include hosting events, such as a hackathon (IT &amp; Engineering), a retreat, or a convention, where they can get to know more people at their organization in a relaxed atmosphere.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e2N7us0sCfXiw4NUNraF8w.png" /></figure><p>The above chart shows size of departments, grouped by location and employee id. This company’s general make-up is comprised of US employees, working in Sales, Operations, Admin and Engineering.<br> <br> The chart above indicates specific employees who are most actively sending messages to their colleagues. Employees on the right tail end may need to boost their collaboration efforts to move towards a more “average” message and collaboration count.</p><p><strong><em>Results</em></strong></p><p>Machine learning is leveraged to predict results for specific business outcomes, such as count of messages, length of messages and other collaboration objectives. Plots are graphed to understand feature selection and how variables are affecting prediction outcomes, such as Gini Impurity. The following chart was an example of predicting location in the data set, based on total message count, departments, and age group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-U46IgGgE8jx6zNNPVxDkg.png" /></figure><p>The features attributed are weighed by importance. The first variable “Count_Messages” is regarded as of highest importance in the Gini Impurity.</p><p>“Gini Impurity is a measurement used to build Decision Trees to determine how the features of a dataset should split nodes to form the tree.”1</p><p><strong><em>Model Results:</em></strong></p><p>Decision Tree— J48 Pruned Tree in Weka</p><p>------------------</p><pre>Sum_No. of Employees &lt;= 3</pre><pre>|   department = Admin: High (32.0/14.0)</pre><pre>|   department = Engineering: Low (15.0/4.0)</pre><pre>|   department = IT: Low (12.0/1.0)</pre><pre>|   department = Marketing: Low (15.0/3.0)</pre><pre>|   department = Operations: High (33.0/12.0)</pre><pre>|   department = Sales: High (52.0/13.0)</pre><pre>Sum_No. of Employees &gt; 3: Low (80.0)</pre><pre>Number of Leaves  : 7</pre><pre>Size of the tree :  9</pre><pre>=== 3-Fold Cross Validation Summary ===</pre><pre>Correctly Classified Instances         189               79.0795 %</pre><pre>Incorrectly Classified Instances        50               20.9205 %</pre><pre>Kappa statistic                          0.5741</pre><pre>Mean absolute error                      0.2784</pre><pre>Root mean squared error                  0.3864</pre><pre>Relative absolute error                 60.3723 %</pre><pre>Root relative squared error             80.5056 %</pre><pre>Total Number of Instances              239</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mb-d-AvgpuG6zFkNXgpTuA.png" /></figure><p>The decision tree above shows how the machine classifies a high or low message rate. A few key takeaways:</p><ol><li>For employees with over 3.5 employees, collaboration tends to be categorized to the low end (33%). Smaller, i.e., less than 3 employees, departments were more likely to be highly collaborative (67%).</li><li>If department = Operations, Admin, or Sales, the algorithm predicts high collaboration (49%).</li><li>If location = France, Germany or US, the collaboration may be high (34%).</li></ol><p>______________________________</p><p><strong><em>Naive Bayes</em></strong></p><p>=== 3-Fold Cross Validation Summary ===</p><p>Correctly Classified Instances 185 77.4059 %</p><p>Incorrectly Classified Instances 54 22.5941 %</p><p>Kappa statistic 0.561</p><p>Mean absolute error 0.2772</p><p>Root mean squared error 0.374</p><p>Relative absolute error 60.1099 %</p><p>Root relative squared error 77.9257 %</p><p>Total Number of Instances 239</p><pre>#kNN Model</pre><pre>=== 3-Fold Cross Validation Summary ===</pre><pre>Correctly Classified Instances         187               78.2427 %</pre><pre>Incorrectly Classified Instances        52               21.7573 %</pre><pre>Kappa statistic                          0.5506</pre><pre>Mean absolute error                      0.2327</pre><pre>Root mean squared error                  0.4334</pre><pre>Relative absolute error                 50.4477 %</pre><pre>Root relative squared error             90.2965 %</pre><pre>Total Number of Instances              239</pre><pre>#Random Forest Model</pre><pre>=== 3-Fold Cross Validation Summary ===</pre><pre>Correctly Classified Instances         190               79.4979 %</pre><pre>Incorrectly Classified Instances        49               20.5021 %</pre><pre>Kappa statistic                          0.5561</pre><pre>Mean absolute error                      0.2295</pre><pre>Root mean squared error                  0.3832</pre><pre>Relative absolute error                 49.7528 %</pre><pre>Root relative squared error             79.845  %</pre><pre>Total Number of Instances              239</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QCLmJQVZ2B1UM0yTI2NCLQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H8znqXakZbLzXSAumny03Q.png" /></figure><h3>CONCLUSIONS</h3><p>The above machine learning models show how the analytics department can predict low and high message sending, with department, location, and employee data. The most accurate model was the Random Forest approach.</p><p><strong><em>HR/Analytics Report</em></strong></p><p>· Which departments are the most/least active?</p><p>Based on this dataset, we see that Sales, Admin and Operations are the most active by message count and length.</p><p>· Which employee has the most connections?</p><p>Based on the employee association network, employee 1881 appears to be influential, with a few close connections. Other high ‘lift’ collaborators in the organization are 1616, 1646, and 1308.</p><p>· Identify the most influential departments and employees.</p><p>Employees 605, 483, 725 and 128 are the most active senders and receivers of messages.</p><p>· Using the network analysis, in which departments would you recommend the HR team focus to boost collaboration?</p><p>IT, Engineering and Marketing appear to send and receive the least number of messages.</p><h3>RECOMMENDATIONS</h3><p>Gallup estimates that “only 30% of U.S. employees are engaged at work, and a staggeringly low 13% worldwide are engaged. Worse, over the past 12 years, these low numbers have barely budged, meaning that the vast majority of employees worldwide are failing to develop and contribute at work.” A solution to boost inter-company collaboration will foster growth within and among teams.</p><p>Picture a collaboration solution, designed to inspire action and motivate teams. Each member creates a company profile, which allows them to network with other teams in their organization. Incorporating community within the solution and providing a space for employees to understand each other’s influence will have users coming back for more. Teams will be given clear goals, and peers will work together to solve problems, train new employees, and make strategic team decisions.</p><p>Collaboration requires building a consensus, being a catalyst for change and getting things done. Collaboration involves technical, intellectual, and emotional skill. To summarize, engagement has an enormous impact on organizations as it’s vital to keep employees involved and purposeful. Good ideas can stem from anyone within an organization; it is a business’ responsibility to give them a voice. As a result, goal setting/bench marking may be the jumping point for boosting collaboration, and innovation among teams. In lieu of an internal company network, other ideas include hosting events or fireside chats and/or designing solutions to boost collaboration, which will lead to new insights, products, software solutions and beyond.</p><p>· Collaboration events — Summer book reading, hackathons, leadership events</p><p>· Collaboration goal-setting — Are we meeting or exceeding our expectations?</p><p>· Fireside chats — Collaboration, innovation, IDEO designed company custom-collaboration map</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b_Ox6V13T2zQBY4Q96BFyg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rDwVbK74oazR85jen2mGvA.png" /></figure><h3>SOURCES</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W6NFhDmZeWOSB10kkZe4sw.png" /></figure><p>_**Acknowledgments:** Pietro Panzarasa, Tore Opsahl, and Kathleen M. Carley.</p><p>“Patterns and dynamics of users’ behavior and interaction: Network analysis of an online community.” Journal of the American Society for Information Science and Technology 60.5 (2009): 911–932</p><p><a href="#_ednref1">[i]</a> “How Collaborative Intelligence Can Boost Business Innovations”, <a href="https://hbr.org/sponsored/2022/04/how-collaborative-intelligence-can-boost-business-innovation">https://hbr.org/sponsored/2022/04/how-collaborative-intelligence-can-boost-business-innovation</a></p><p><a href="#_ednref2">[ii]</a> “How Collaborative Intelligence Can Boost Business Innovations”, <a href="https://hbr.org/sponsored/2022/04/how-collaborative-intelligence-can-boost-business-innovation">https://hbr.org/sponsored/2022/04/how-collaborative-intelligence-can-boost-business-innovation</a></p><h3>APPENDIX</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DAcK4fo9Yk5s6Jo5Tm5tLw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vg3a1Ud6Rfd6ejcW2KTyVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HLD_8e4RvIYrTs5dfnimBQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63cff060b986" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dashboard Design Examples]]></title>
            <link>https://lisalondon.medium.com/dashboard-design-examples-406d7b57c160?source=rss-773031daa634------2</link>
            <guid isPermaLink="false">https://medium.com/p/406d7b57c160</guid>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[dashboard-design]]></category>
            <category><![CDATA[data-strategy]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <dc:creator><![CDATA[Lisa]]></dc:creator>
            <pubDate>Wed, 21 Sep 2022 18:21:26 GMT</pubDate>
            <atom:updated>2024-01-11T20:49:06.929Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wV0ZQqz_-rBw3ewFebFBtQ.jpeg" /></figure><p>Dashboards can help communicate a company’s story by visualizing their key performance indicators (KPIs) and preparing for future strategies. Spreadsheets and Excel tabs can only provide so much information when considering weekly analyses, big data, and emerging ML/AI metrics. For this reason, dashboards are necessary to view KPIs in a visually appealing manner that is easy to communicate within the organization. Here are a few examples of the dashboards that I’ve worked on so far to give you an idea of how data analysis could be used strategically.</p><h3>Dashboard Examples — Marketing Stage Analysis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*9EpEUXVnzVdm2vxbAZKxhA.png" /></figure><p>The above view investigates tactics by opportunity stage as a high-level overview of year-over-year changes and stages. The stages are divided into 3 groups, lost, open and won stage. The goal is to reduce lost opportunities and accounts and increase wins and open accounts. Year-over-year the dashboard is showing promise in new open accounts, but has yet to close or win those accounts in the current year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zH4CY7IAQpJrJqlYFCNWtg.png" /><figcaption>Tableau allows a logo to be added to a page/dashboard view, as seen above.</figcaption></figure><p>The view above shows an executive summary by tactic. The YoY % difference is highlighted in the top left graph as a -38% in won opportunities. Tactics by revenue is also down. The cluster analysis on the bottom right chart indicates increased counts of tactics correlated with increased pipeline generation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YDzy2CEUnsLpOcdcmfAovw.png" /></figure><p>The view above is a trends analysis by tactic, specifically which tactics are driving pipeline generation and which tactics are not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lRrF2nVbTXZp8F2iyA6agA.png" /></figure><p>The won/open accounts deep-dive indicates pipeline generation by opportunity stage (lost, open or won), as well as the tactics leveraged (summed by pipeline generation) per stage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xxCUbnSYwIhgBHTm0wW8wg.png" /><figcaption>The dashboard can also be filtered to view a specific year or tactic.</figcaption></figure><p>The dashboard shows year-over-year changes in profitability by tactic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W7ZEuBmDZYLQIG39L4Q5fg.png" /></figure><p>A geo analysis shows the opportunity countries by marketing pipeline generation and winning tactics year-over-year.</p><h4>RECOMMENDATIONS BASED ON ANALYSIS</h4><p>To better understand the status of open accounts and increase chances of turning them into wins, I suggest a strategic approach for the month of October 2023. During this month, the team will focus on leveraging the best tactics mentioned earlier — website, webinars, events, demos, and trials.</p><ul><li><em>Website:</em> We will optimize our website with targeted messaging and compelling content that speaks directly to our open accounts. This will help us address their specific pain points and position ourselves as the solution to their challenges.</li><li><em>Webinars:</em> In October, we will conduct targeted webinars tailored to the needs of our open accounts. By addressing their specific interests and concerns, we can engage them further and showcase how our solutions can meet their requirements.</li><li><em>Events:</em> We will actively participate in industry events and conferences where our open accounts are likely to be present. This will allow us to connect with them directly, have meaningful conversations, and establish stronger relationships.</li><li><em>Demos and Trials:</em> For our open accounts, we will offer personalized demos and trials that align with their specific needs. This will give them an opportunity to experience our solutions in a customized manner and increase their confidence in making a purchase decision.</li></ul><p><em>Conclusion</em></p><p>By leveraging the best tactics and focusing on the strategies outlined above, we can better understand the status of our open accounts and improve chances of converting them into wins.</p><p><em>Cited Sources:<br>Website Conversion Rate Optimization<br>Webinars for Lead Generation<br>Events as a Marketing Strategy<br>Demos and Trials for Sales Conversion</em></p><h3><strong>Customizable Views in Microsoft Power BI</strong></h3><p>Another software program I’ve used before is Microsoft Power BI. Here are a few views from an early iteration of a dashboard. Each view includes filters (by year, country and segment for example). This is a simpler dashboard example to share internally, across teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tMVaChI654gUSi4P22JYPw.png" /><figcaption>These custom dashboards can make your data come to life and help keep track of business objectives, anomalies and seasonal trends.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7-WzdQhfvHy8DR5-lAAAWQ.png" /><figcaption>Another example of additional capabilities in Power BI, with an interactive view on product units sold by month. Additional insights on profit vs. sales numbers.</figcaption></figure><p>Power BI is the software service I used to create the above dashboard views. There are many ways to derive insights from various departments in your organization, including marketing, customer service, sales, and finance. By leveraging data analytics between departments, we can share metrics and make strategic business decisions in a more streamlined manner.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=406d7b57c160" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tech Companies and Antitrust Laws]]></title>
            <link>https://lisalondon.medium.com/tech-companies-and-antitrust-laws-efd1345545ec?source=rss-773031daa634------2</link>
            <guid isPermaLink="false">https://medium.com/p/efd1345545ec</guid>
            <category><![CDATA[antitrust]]></category>
            <category><![CDATA[competition-law]]></category>
            <category><![CDATA[antitrust-law]]></category>
            <category><![CDATA[technology-companies]]></category>
            <dc:creator><![CDATA[Lisa]]></dc:creator>
            <pubDate>Mon, 05 Sep 2022 16:50:44 GMT</pubDate>
            <atom:updated>2022-09-05T16:50:44.140Z</atom:updated>
            <content:encoded><![CDATA[<p>Summary: For this research paper, I use the IRAC (Issue, Rule, Analysis and Conclusion) method to research and draw conclusions on the topic of antitrust laws.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kha3XGxRMoIyhKF2CU-yLg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@lycs?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">LYCS Architecture</a> on <a href="https://unsplash.com/s/photos/office?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><strong><em>Issue/Rule</em></strong></p><p>Antitrust laws were penned with the intent to prevent companies from creating unfair competition in the marketplace. If companies were allowed to halt/deter competing companies, consumers would be forced to bear the disadvantages. According to Investopedia.com, “Antitrust laws, also referred to as competition laws, are statutes developed by the U.S. government to protect consumers from predatory business practices. They ensure that fair competition exists in an open-market economy. These laws have evolved along with the market, vigilantly guarding against would-be monopolies and disruptions to the productive ebb and flow of competition.” If fair competition did not exist, consumers would be subject to certain vulnerabilities, such as price fixing and/or paying higher prices for goods and services. The absence of free markets also limits the number of differentiated products, businesses and innovations, which a competitive market offers. Ultimately, monopolies are an infringement on consumer freedoms.</p><p>Antitrust law is brought up increasingly in the technology industry, where many firms have emerged as the leader on their disruption/innovation. These companies include Amazon, Apple, Google, Netflix, Microsoft and Facebook. Their rapid growth and first to market status create a high barrier to entry for companies that wish to compete. This makes the tech industry ripe for Antitrust litigation, as large tech companies have a strategic edge over smaller competitors.</p><p>What the aforementioned companies all have in common, is they have reached critical mass. The more users you have on your platform, the more value you are creating for new/existing users. This is called the network effect. To illustrate, if you want to join a social network, it makes sense to join Facebook, because you are more likely to find your social circle on the world’s most popular social network. Network effects can be applied to many technologically disruptive companies. Netflix is able to license more TV/movies for its catalog, because more users pay for the platform. Their user data and machine learning algorithm are strengthened the more users they gain. As a result, they can reinvest more revenue into creating a more versatile/user-centric experience, investing in research and development and improving its value creation. This non-competitive advantage is what makes Antitrust law important, but also somewhat difficult to prove. Can you prove that a company is engaging in Antitrust practices, when they could have just been in the right place, at the right time? There are a few cases, which dive deeper into Antitrust litigation.</p><p><strong><em>Apple Inc. vs. Pepper, <br>Hanover Shoe, Inc. v. United Shoe Machinery Corp., <br>and Illinois Brick Co. v. Illinois</em></strong></p><p>In the 2019 article in the Harvard Law Review on <em>Apple Inc. vs. Pepper</em> the plaintiffs claim in this case that Apple’s 30% commission fee was “supracompetitive” because Apple had a monopoly on “distributing software applications that can be downloaded on the iPhone”. They made the argument that, in a competitive marketplace, there would be a fairer price point. These Apple consumers argue that because they can buy the exact same app on another platform for less cost than what they would pay in the Apple App Store, it points to an unfair “passing-on” of cost, due to Apple’s 30% commission fee. Apple countered the argument, filing a motion to dismiss stating that the “plaintiffs lacked standing because they did not qualify as direct purchasers.”</p><p>The district court granted Apple’s motion to dismiss. Judge Rogers discussed a previous case when the plaintiffs lacked standing under <em>Illinois Brick vs. Illinois</em>. Harvard Law Review states, “The plaintiffs’ status as direct or indirect purchasers is generally dispositive: direct purchasers, the first purchasers in the distribution chain, may sue, whereas indirect purchasers, who bear any overcharges direct purchasers pass on through higher prices, may not. Applying these definitions, the court found that the plaintiffs failed to qualify as direct purchasers because Apple’s 30% commission fee was ‘borne by the developers’ and paid to Apple ‘from their own proceeds.’”</p><p>In a previous Antitrust lawsuit, <em>Hanover Shoe</em>, Justia.com states, “the illegal overcharge was absorbed by the plaintiff’s customers — indirect purchasers of the defendant’s shoe machinery — who were the persons actually injured by the antitrust violation. In <em>Hanover Shoe</em>, this Court rejected as a matter of law this defense that indirect, rather than direct, purchasers were the parties injured by the antitrust violation. The Court held that, except in certain limited circumstances, a direct purchaser suing for treble damages under § 4 of the Clayton Act is injured within the meaning of § 4 by the full amount of the overcharge paid by it, and that the antitrust defendant is not permitted to introduce evidence that indirect purchasers were in fact injured by the illegal overcharge. A reason for barring the defense was the Court’s concern that unless direct purchasers were allowed to sue for the portion of the overcharge arguably passed on to indirect purchasers, antitrust violators “would retain the fruits of their illegality” because indirect purchasers “would have only a tiny stake in the lawsuit,” and hence little incentive to sue.”</p><p>In response to the previous case <em>Illinois Brick vs. Illinois</em>, and the argument of whether the Plaintiffs purchase apps directly from Apple, via the App Store, which would make them “direct purchasers”, the court found that the Plaintiffs do have standing because they are direct purchasers of the product. A precedent from <em>Illinois Brick</em> is “establish[ing] a bright-line rule” “if manufacturer A sells to retailer B, and retailer B sells to consumer C, then C may not sue A. But B may sue A if A is an antitrust violator. The Court found that under a price-setting rule, Apple would escape liability for conduct economically indistinguishable from what — in the markup context — is presently recognized as paradigmatically anticompetitive behavior.”</p><p>Harvard Law Review discusses the case further, stating, “The court refused to “speculate” as to the price at which Apple would have sold apps in the absence of a commission fee. Such speculation, the court feared, would have mired it in overly technical proceedings regarding “developers’ pricing structure, their costs, ability to find a distribution chain, and/or desired profits or rates of return.” For these reasons, the court concluded that plaintiffs lacked standing.”</p><p>Under this argument, and in reality, the plaintiffs would only be affected by the price mark-up if the developers chose to pass on this 30% commission fee onto their users. The developers have complete control over the price of their app(s), so it is up to them to “pass-on” the cost, if they choose to. It is not stated whether or not other app stores embed a similar surcharge in their price proposition to developers. If this pricing structure were similar to other app stores, then the developers choose to raise and/or lower cost based on platform. Each platform offers its own unique value proposition with pros and cons and may therefore be weighted unequally in the final cost/price structure. For the court to overanalyze developers’ choices for these structures, based on specific app business cases, would be an overly time-consuming process, which may not eventually lead to an effective judgement on the case.</p><p>The plaintiffs may be alluding to the fact that these apps are available on other phone competitor’s app stores, but they would be incompatible to use with an iPhone. Therefore, an Apple consumer’s alternative option is perhaps to purchase another phone in order to buy the app at a lower cost, which may be considered unfair. This situation is similar to a previous Antitrust lawsuit against Microsoft in 1998, when the company was under scrutiny for making it difficult for consumers to download other company’s software programs for use with Microsoft computers. The plaintiffs in that case argued that creating obstacles against competing programs was anti-competitive. I will discuss that case more in-depth, a little later.</p><p>Harvard Law recites, “By treating formal relationships as fundamental, the dissent argued, the majority courted the exact problems that <em>Illinois Brick</em> had solved. Because “Apple charged only one commission on each sale,” Justice Gorsuch wrote, separate suits from consumers and developers would necessarily expose Apple to liability for duplicative damages. Determining what share of these damages belong to which party would require burdensome pass-through calculations. Furthermore, Justice Gorsuch continued, reducing <em>Illinois Brick</em> to a bright-line rule counterproductively privileges form over substance: Apple could simply restructure its business so that consumers are in privity with app developers instead. The dissent, therefore, chastised the Court for trading an “intelligible, principled, [and] administrable” rule for one that was “artificial,” less administrable, and, ultimately, less capable of restraining anticompetitive conduct.”</p><p>The Court goes on to state that Antitrust law prevents companies from charging illegally high prices for goods. In order to right this wrong, justice can be restored when the company in question returns the “supracompetitive” overcharge amount back to the consumer. This calculation on returns to the consumer may effectively overload the court system unnecessarily. Would the courts be overburdened by cases alleging liability on Apple for damages similar to this case? The legal system was built for efficiency. In order to avoid congesting the judicial system unnecessarily with legal cases, the system seeks to lessen its own burden.</p><p>Harvard Law Review continues, “Justice Brennan’s <em>Illinois Brick</em> dissent represented an alternative vision of antitrust values, according to which corrective justice, and specifically victim compensation, dominates judicial efficiency and deterrence. It is a vision reproduced — at times with uncanny accuracy — by Justice Kavanaugh. Though the Court, before <em>Illinois Brick</em>, had never previously relied on a close reading of the Clayton Act, Justice Brennan interrogated the statutory text first. He attended especially to its “any person who shall be injured” language, and found therein the right for all, and “especially for consumers,” to seek a “remedy” for antitrust-related injuries.”</p><p>The court analyzed the intended meaning within the text of the Clayton Act. Justice Brennan argued that the Act was created to “prevent wrongdoers from retaining the spoils of their misdeeds” and that this best served an indirect purchaser. Judge Kavanaugh agreed, stating that “denying indirect-purchaser standing would ‘directly contradict the longstanding goal of effective private enforcement and consumer protection.’” The legal system intends to avoid extraneous lawsuits, but both judges dismissed the long-standing arguments that “allowing indirect purchasers to sue actually cuts sharply against deterrence.”</p><p>Harvard Law Review concludes, “The Court did not so much decisively refute each counterargument as it simply reweighted the competing policy considerations in favor of victim compensation. Far from settling the debate over antitrust standing, the Court’s holding in <em>Illinois Brick</em> provoked a public controversy that included no less than a “decade of congressional hearings.” One consequence was states’ enactment of <em>Illinois Brick</em> repealers — statutes meant to override the precedent for state courts. In the face of an extensive record of state court adjudications, it becomes increasingly implausible for federal courts to insist the necessary damages calculations pose insuperable difficulties. Second, state Attorneys General’s capacity to bring a suit on behalf of numerous aggrieved consumers under parens patriae is just the sort of procedural device that would alleviate concerns that insufficient consolidation of claims would vitiate deterrence. However, a closer analysis of the opinion reveals a reweighting of traditional antitrust values more closely aligned with Justice Brennan’s dissent in <em>Illinois Brick</em>. This inversion of values in Apple, together with the developments in legislation and economic models since 1977, underscores the present fragility of <em>Illinois Brick</em>.”</p><p>In the <em>Hanover Shoe</em> case, the court ruled that Hanover was entitled to damages. In<em> Illinois Brick Co. v. Illinois,</em> the Supreme Court concluded that whether a consumer is a “direct” or “indirect” purchaser, they were entitled to protection under the Clayton Act. Americanbar.org states, “Ever since, this “direct purchaser” rule has prevailed: Those who purchase directly from monopolists, or firms engaged in price fixing or other antitrust violations, can recover damages under the Clayton Act; those who purchase indirectly, from an intermediary in the distribution chain, can obtain injunctive relief under the Clayton Act and damages under state law only. In <em>Apple Inc. v. Pepper</em>, the Supreme Court reaffirmed<em> Illinois Brick </em>and held that people who buy apps from the App Store can seek Clayton Act damages as direct purchasers, notwithstanding Apple’s argument that the developers — not Apple — set the app prices.”</p><p><strong><em>United States vs. Microsoft</em></strong></p><p>In a past Antitrust case against Microsoft, Investopedia states: “Charges were brought against Microsoft to determine whether its bundling of additional programs into its operating system constituted monopolistic actions.” The government accused Microsoft of anti-competitive behavior for creating obstacles for consumers installing competing software programs on Windows computers.</p><p>Investopedia states, “If Microsoft was found to have made it unreasonably difficult for consumers to uninstall Internet Explorer and use a competing browser, the company’s practices would be deemed anti-competitive. Meanwhile, those who supported Microsoft proposed that if the company was to be considered a monopoly, it was, at best, a noncoercive one. They argued that even with options like Unix, Linux, and Macintosh, consumers demonstrated a preference for the convenience of Microsoft’s Windows product on their computers. The ease of its installation and its other bundled software allowed it to become the norm.”</p><p>The Judge ruled that Microsoft’s position in the industry was similar to a monopoly and its behavior was a hindrance innovation in computing. Microsoft initially lost the case and was charged to “divide the company in half and create two separate entities. The operating system would make up one half of the company and the software arm would make up the other.”</p><p>Microsoft disagreed with the court’s decision that they were engaging in monopolistic practices. A timeline of the U.S. vs. Microsoft Antitrust case on Wired.com states, <br> “January 19, 2000: Microsoft officially responds to Judge Jackson’s ruling, saying that widespread competition precludes it from holding monopoly power. <br> February 1, 2000: Microsoft files its final brief, pointing to the AOL-Time Warner merger as the main reason why the antitrust case should be thrown out.” The decision was overturned. The Department of Justice settled with Microsoft, stating that the settlement serves the public’s interest and Microsoft would no longer be forced to split its company into two entities.</p><p>According to Investopedia.com, “The company saw its once invincible market share erode due to old-fashioned competition. But the lessons learned from the case continue to resound. Many now wonder if bringing antitrust cases against non-coercive monopolies is merely a costly redundancy of work the free market can do at no charge.”</p><p><strong><em>Analysis</em></strong></p><p>When reflecting on the <em>Apple Inc. vs. Pepper</em> case, I felt that the commission paid to Apple was more than fair. Apple provides one of the best market opportunities when launching new apps/platforms on the market because of its attractive, large user base. Apple is experiencing all of the benefits of network effects and are able to pass on that value to app developers. Apple Inc. is not just charging a commission; they are offering a most ideal opportunity. It may be less costly to launch an app on another app store, but also less supreme. When launching a product, it is best to utilize platforms that provide the most users possible, because that could lead to higher revenue, growth and exposure.</p><p>Apps also often use in-store purchases and ads to generate revenue. Companies that advertise in these apps tend to flock to higher user-based platforms, like the Apple App Store. Due to the service Apple is providing which includes marketing, exposure and an attractive user base, it is probably worth the added commission cost to developers, who can ultimately decide on their app price(s) and compensate for the fee. If Apple’s app store prices are slightly more expensive than other app stores, that does not stop a consumer from shopping on a different platform. As Microsoft’s lawyers argued in <em>United States vs. Microsoft</em>, the app/software market is still highly competitive in nature. The sheer amount of competition in the market erodes any discussion on whether these companies are operating monopolies.</p><p>According to Owler.com, Apple’s main competitors are Microsoft, Dell, Samsung, Lenovo, HP, Sony, ASUS, Google, Huawei and Philips. With so many direct competitors, many of whom have their own App stores, including Google, Samsung and Huawei, there is clear evidence that Apple does not operate a monopoly. Consumers who are averse to Apple’s prices can choose to do business with a competing company. It is ultimately up to a consumer to research costs in advance of purchase. To demonstrate an example, different grocery stores can charge more/less for the exact same item. Convenience stores are typically more expensive, due to their added convenience. This is something that a consumer can be aware of and make necessary revisions to their buying practices to suit their particular financial situation.</p><p><strong><em>Conclusion</em></strong></p><p><em>Apple Inc. vs. Pepper</em> discusses indirect and direct purchasers and their ability to sue a company in court over unfair business practices. The court recites the <em>Hanover Shoe, </em>and<em> Illinois Brick Co. v. Illinois</em> cases as precedent for which consumers can legally sue a company over the price of goods sold. The court may have brushed over the main issue, which the Microsoft case did not fail to mention. In <em>United States vs. Microsoft</em>, the discussion of a free market and competition in that market was a prevalent argument. It underlies the key issue stated in Antitrust law. According to Investopedia, “Antitrust laws…protect consumers from predatory business practices. They ensure that fair competition exists in an open-market economy …vigilantly guarding against would-be monopolies and disruptions to the productive ebb and flow of competition.”</p><p>For the increased cost that Apple charges for their apps, the benefits to the user (namely convenience and number of unique apps available) are tenfold. Apple basically does what any successful emerging disruptive technology company does. They leverage an open source development platform, enabling developers to create apps for their platform. That enables more competition, diverse and unique app development and is the opposite of monopolistic behavior. Network effects is a key factor in Apple’s market position. It may be difficult for a new tech company to grow to the scale the likes of Apple, Facebook, Microsoft, Google, and Netflix. This has more to do with timing and a differentiated, superior platform, than predatory practices.</p><p>Apple was first and unique in their effort to pioneer a smartphone with an elegant design combined with one of the most intuitive computer software systems available. It is usually those who can differentiate themselves and be relatively new to the market, who win. Technology has pushed us into innovation at a speed that has outpaced the traditional business trajectory. Developers and users flocked to the most intuitive, best marketed, user friendly product on the market, which led to critical mass. That being said, given the amount of competition in the market, in my opinion, purchasers (indirect or direct) do not have a case against Apple in regard to violating Antitrust law for charging a 30% commission fee.</p><p>Sources:<br>Apple Inc. v. Pepper, 139 S. Ct. 1514 (2019) <a href="https://harvardlawreview.org/2019/11/apple-inc-v-pepper/">https://harvardlawreview.org/2019/11/apple-inc-v-pepper/</a></p><p>Understanding Antitrust Laws<br><a href="https://www.investopedia.com/ask/answers/09/antitrust-law.asp">https://www.investopedia.com/ask/answers/09/antitrust-law.asp</a></p><p>Illinois Brick Co. v. Illinois, 431 U.S. 720 (1977)<br><a href="https://supreme.justia.com/cases/federal/us/431/720/">https://supreme.justia.com/cases/federal/us/431/720/</a></p><p>Hanover Shoe, Inc. v. United Shoe Machinery Corp., 392 U.S. 481 (1968)<br><a href="https://supreme.justia.com/cases/federal/us/392/481/">https://supreme.justia.com/cases/federal/us/392/481/</a></p><p>Company Fact Sheet: Apple<br><a href="https://www.owler.com/company/apple">https://www.owler.com/company/apple</a></p><p>Supreme Court Rejects “Who Sets the Price” Alternative to Illinois Brick<br><a href="https://www.americanbar.org/groups/litigation/committees/class-actions/practice/2019/illinois-brick-v-illinois/#:~:text=In%20Illinois%20Brick%20Co.,Clayton%20Act%20claim%20for%20damages.">https://www.americanbar.org/groups/litigation/committees/class-actions/practice/2019/illinois-brick-v-illinois/#:~:text=In%20Illinois%20Brick%20Co.,Clayton%20Act%20claim%20for%20damages.</a></p><p>Why Did Microsoft Face Antitrust Charges in 1998?<br><a href="https://www.investopedia.com/ask/answers/08/microsoft-antitrust.asp">https://www.investopedia.com/ask/answers/08/microsoft-antitrust.asp</a></p><p>U.S. vs. Microsoft Timeline<br><a href="https://www.wired.com/2002/11/u-s-v-microsoft-timeline/">https://www.wired.com/2002/11/u-s-v-microsoft-timeline/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efd1345545ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Applying K-means Clustering Model to Customer Segmentation]]></title>
            <link>https://lisalondon.medium.com/applying-k-means-clustering-model-to-customer-segmentation-4254386c7563?source=rss-773031daa634------2</link>
            <guid isPermaLink="false">https://medium.com/p/4254386c7563</guid>
            <dc:creator><![CDATA[Lisa]]></dc:creator>
            <pubDate>Sun, 10 Oct 2021 18:28:55 GMT</pubDate>
            <atom:updated>2023-05-12T02:18:43.008Z</atom:updated>
            <content:encoded><![CDATA[<p>In consumer, audience, advertising and marketing analytics, the customer is at the heart of everything. It’s imperative to understand our core customer group. This involves obtaining their demographics as well as detailed information about their preferences, inclinations and habits. For this project, we have collected data from various sources, including third party providers like Acxiom, to bring similarities and differences between audiences to the forefront. Using machine learning methods, we can create customer segments by grouping people with similar tastes together. This will enable us to market specifically to certain user groups and tailor messaging according to similar tastes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x6qr-1k7Wc5N3UixgCYnng.jpeg" /><figcaption>Photo provided by <a href="https://unsplash.com/@iamchang">Chang Duong</a>.</figcaption></figure><h3>Model Determination — Statistical Techniques Considered</h3><p>The objective is to analyze first party and third party data to output a set of customer segments, to be used for marketing planning and strategy. The following models were considered for this project:</p><p>· <strong>Model A:</strong> K-means clustering<br>· <strong>Model B:</strong> Recency, frequency &amp; monetary (RFM)<br>· <strong>Model C:</strong> Demographic, Behavioral, Psychographic 3PD Overlap Analysis</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ekQvoS-FGisYpvyHyRsjTQ.png" /><figcaption>(Pros/Cons of each model for segmentation application)</figcaption></figure><h3>Selection and Rationale</h3><p>We have decided to proceed with the K-means cluster model, which will allow us to include portions of the RFM model in addition to demographic, behavioral, and psychographic details. The K-means model enables indicators of program enrollment, payment history, and customer interactions to deliver the most in-depth segmentation output. This results in very effective, efficient, and marketable segments for ongoing customized communications.</p><p>The reason we chose the K-means model is due to its versatility; it has the ability to calculate most quantitative data fields and provide a deeper understanding of our core user base. These insights can help develop products that appeal to different customer groups while also allowing us to understand their needs/preferences better.</p><h3>Understanding K-means Clustering</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*E9R_N48ftoul7kp1y9-OIg.gif" /><figcaption>(Photo Credit: <a href="https://aws.amazon.com/blogs/machine-learning/k-means-clustering-with-amazon-sagemaker/">https://aws.amazon.com/blogs/machine-learning/k-means-clustering-with-amazon-sagemaker/</a>)</figcaption></figure><p>K-means is an unsupervised machine learning algorithm. In general, clustering uses iterative techniques to group cases in a dataset into clusters that contain similar characteristics. These groupings are useful for exploring data, identifying anomalies in the data, and eventually for making predictions. Clustering models can also help identify relationships in a dataset that might not logically be derived by browsing or simple observation. For these reasons, clustering is often used in the early phases of machine learning tasks, to explore the data and discover unexpected correlations.</p><p>When configuring a clustering model using the k-means method, you must specify a target number k indicating the number of centroids you want in the model. The centroid is a point that is representative in each cluster. The K-means algorithm assigns each incoming data point to one of the clusters by minimizing the within-cluster sum of squares.</p><p>When processing the training data, the K-means algorithm begins with an initial set of randomly chosen centroids, which serve as starting points for each cluster, and applies Lloyd’s algorithm to iteratively refine the locations of the centroids. The K-means algorithm stops building and refining clusters when the centroids stabilize, meaning that cluster assignments for individual points no longer change and the algorithm has converged on a solution. (Microsoft Azure)</p><h3>Optimizing Clusters: Elbow Plot Method</h3><p>The ML practitioner performs measurements on the clusters to obtain the best ratio between and within clusters.</p><p>How do we identify usable segments and identify the optimal number of clusters for the ensemble cluster model? Some considerations include the following:</p><p>· If there are too few clusters, we may not be maximizing enough key differences of users in the data (this can happen when we have too many variables clustered together, without partitioning first).</p><p>· If there are too many clusters, segment sizes might not be large enough to deliver impact (leads to ineffective, inefficient marketing communication, unless the goal is super-personalized communication).</p><p>Custom segmentation should provide marketing teams with enough differences between segments and small differences within segments for an optimal output. This photo below represents a criterion for locating the optimal k-means clustering number, called the “Elbow Plot.”</p><h4>Elbow Plot Example output:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Vst9y55wG_euX7nQar92FA.png" /><figcaption>(Photo Source: <a href="https://www.datanovia.com/en/lessons/determining-the-optimal-number-of-clusters-3-must-know-methods/">https://www.datanovia.com/en/lessons/determining-the-optimal-number-of-clusters-3-must-know-methods/</a>)</figcaption></figure><p>The Elbow Plot measures clusters that reduce differences within a cluster and improve differences between other clusters. In the example above, the partition shows an optimal cluster number for 4–5 clusters. After we find the size of the population in each of these clusters, we can determine if they represent a large enough portion of the population. (For instance, we might consider at least 6% of the population to be sizable enough for clustering, while anything lower is too small to form its own separate cluster.)</p><p><strong>K-Means Ensemble Method</strong></p><p>There are various ways to perform a k-means cluster analysis. We decided to go with the Ensemble method. This model allows us to take k-means to a higher level of accuracy by standardizing the data.</p><p>Data standardization is the process of bringing data into a uniform format. In statistics, standardization refers to the process of putting different variables on the same scale to compare scores between different types of variables and allows further insights into each category of interest. (Trifacta)</p><p>A clustering ensemble aims to combine multiple clustering models to produce a better result than that of the individual clustering algorithms in terms of consistency and quality.</p><p>The basis of our model for clustering came from a recent study:</p><p><em>Source: </em><a href="https://link.springer.com/article/10.1007/s13042-017-0756-7"><em>https://link.springer.com/article/10.1007/s13042-017-0756-7</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*J6cYcahhDvJIzX3wsJFmKQ.png" /><figcaption>Ensemble Framework</figcaption></figure><h3>K-means Process &amp; Application</h3><p>We analyzed a range of customer data, using a CRM data dictionary to define the data. From the data dictionary we determined which dimensions and variables were appropriate to include in our model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*728I4oIV_SUZnTv3dR23Fw.png" /><figcaption>Overview of Data Organization and Building Training Dataset</figcaption></figure><p>The variables are grouped into categories, such as home, demographics, interests, green living preferences, media channels, payment history, program enrollment/brand engagement etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*2gLOizdlmxINrf4WPVqI6Q.png" /><figcaption>(Partitioning data into categories)</figcaption></figure><h3>Criteria for Effective Segmentation</h3><p>Next, we determined a criterion for effective marketing, product, and customer messaging.</p><p>1. Identifiable: Identify customer segments through demographics, etc.</p><p>2. Sustainable: Are the segments big enough for marketing programs to have a meaningful impact?</p><p>3. Accessible: Which channel(s) are optimal to reach SCE customers through media?</p><p>4. Actionable: Will the segmentation help inform effective and engaging marketing messaging?</p><p>5. Responsive: Will segments react uniquely to creative and messaging from media?</p><h3>Partitions Clustered for Final Clustering Result</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*5ZEFelV8pY67EtK_SmrlBw.png" /><figcaption>Partitions for Final Clustering</figcaption></figure><p>We evaluate the clusters by segment, considering all variables. This process of discovering what makes each segment unique leads us to the final segmentation result.</p><h3>Final steps: Identifying Cluster Differences</h3><p>Once we’ve identified how each feature impacts the clusters or the deciding factors of each cluster, we can build customer segments for each persona in the data. These details will lead us to our final customer segmentation profiles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mhE0rgLPuUslEmDl3r8n_g.png" /><figcaption>Executive Summary of Segments for User Profiles</figcaption></figure><p>With this information, we can craft tailored segments that are organized according to the demographics, interests, and behaviors of our clients.</p><h3>Customer Segment Profile Examples:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*PLObHcjICB_ffGUkxt8eUw.png" /><figcaption>(Photo Credit: <a href="https://www.sourcetop.com/articles/how-user-personas-help-craft-better-solutions">https://www.sourcetop.com/articles/how-user-personas-help-craft-better-solutions</a>)</figcaption></figure><p>Endless applications for any industry…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*opPOrWZvwiAjiZyDRv952A.png" /><figcaption>(Photo Credit: <a href="https://ketanrajput.com/patientportal">https://ketanrajput.com/patientportal</a>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CIb9k3qP3mk6pi_BqPyzbQ.jpeg" /><figcaption>(Photo Credit: <a href="https://www.sourcetop.com/articles/how-user-personas-help-craft-better-solutions">https://www.sourcetop.com/articles/how-user-personas-help-craft-better-solutions</a>)</figcaption></figure><p>Sources:</p><p><em>Understanding k-means source, Microsoft Azure: </em><a href="https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/k-means-clustering"><em>https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/k-means-clustering</em></a><a href="#_msocom_1">[PI1]</a></p><p><em>Standardizing Data, Trifecta: </em><a href="https://www.trifacta.com/standardize-data-in-excel/#:~:text=Data%20standardization%20is%20the%20process,between%20different%20types%20of%20variables"><em>https://www.trifacta.com/standardize-data-in-excel/#:~:text=Data%20standardization%20is%20the%20process,between%20different%20types%20of%20variables</em></a></p><p>“Marketing Analytics: Essential Tools for Data-Driven Decisions”, by <a href="https://www.alibris.com/search/books/author/Rajkumar-Venkatesan">Rajkumar Venkatesan</a>, <a href="https://www.alibris.com/search/books/author/Paul-W-Farris">Paul W Farris</a>, <a href="https://www.alibris.com/search/books/author/Ronald-T-Wilcox">Ronald T Wilcox</a></p><h3>Appendix</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NFdwBoK6ECiRROdNGgGCZQ.png" /><figcaption>(Photo credit: <a href="https://searchcustomerexperience.techtarget.com/definition/customer-segmentation">https://searchcustomerexperience.techtarget.com/definition/customer-segmentation</a>)</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4254386c7563" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Towards a Better Workplace]]></title>
            <link>https://lisalondon.medium.com/towards-a-better-workplace-ceb2a9aa1d02?source=rss-773031daa634------2</link>
            <guid isPermaLink="false">https://medium.com/p/ceb2a9aa1d02</guid>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[workplace-diversity]]></category>
            <category><![CDATA[workplace-culture]]></category>
            <category><![CDATA[employee-engagement]]></category>
            <dc:creator><![CDATA[Lisa]]></dc:creator>
            <pubDate>Thu, 31 Dec 2020 00:08:39 GMT</pubDate>
            <atom:updated>2021-03-20T17:01:43.900Z</atom:updated>
            <content:encoded><![CDATA[<p>Gallup reports that employee engagement statistics are staggeringly low — 30% in the US and 13% worldwide. Even worse, scores have remained low for over a decade. It reminds me of a story from my Leadership in Organizations class called, “The Banana.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*P1cRGRffIHnZuw331nRnKQ.jpeg" /><figcaption>The Banana (fictional story)</figcaption></figure><p>The Banana is a story about a group monkeys under a stairwell. Someone places bananas at the top of the stairs. It doesn’t take long before one of the monkeys makes a break for the stairs. During the monkey’s attempt to climb the stairs, the other monkeys are sprayed with cold water.</p><p>This process repeats anytime a monkey tries to climb the stairs. Eventually, the monkeys attack anyone attempting to climb the stairs, to prevent being sprayed with cold water.</p><p>The next phase is to remove the cold water and replace one monkey with a new one. The newcomer notices the bananas and attempts to go after them, but to his horror, he is attacked by the other monkeys. The new monkey eventually learns that climbing the stairs will result in attack.</p><p>One by one, each original monkey is replaced, but they all learn not to make a break for the bananas, or else. Even the new monkeys join in on the enthusiasm of assaulting any monkey going for the bananas! As a result, we have a group of new monkeys, added in one-by-one. None of which have ever been sprayed with cold water, so they do not fully understand why they are attacking anyone making a break for the bananas. Nevertheless, none of the monkeys make an attempt for the stairs again.</p><p>Why?</p><p>Because for as long as anyone can remember, that’s the way things have been done around here.</p><p>This brings me back to employee engagement. We can fix the status quo. There are numerous ways to improve workplaces and teams. How? To start, we need to practice diverse hiring/promotions, foster a positive environment at work, and empower employees.</p><p><strong>1)Hire and Empower a Diverse Workforce</strong></p><p>Harvard Business Review found that diverse teams led to more innovation. This is important, because a lack of innovation is the stagnation of creative thought. When teams are less diverse, they tend to “check-out” mentally. For example, studies show that more diverse juries led to a higher degree of evidence and fact retention. In other words, non-minority jurors were able to recall more facts on the case, when the jury was more diverse. In less diverse juries, fact retention faltered.</p><p>This makes sense intuitively. Imagine you are in a room with all gray walls, all gray furniture and muted artwork. Everything blends well together, nothing stands out. From a neurological perspective, it might be difficult to remember specific things about this room, with the lack of diversity available. The brain is more prone to check-out.</p><p>Now, imagine the same room with a few bold pieces of furniture. A green couch, a purple lamp, graphic artwork and an orange chair. All of sudden, our brain is awakened by the presence of the diverse/unique objects in the room. Which one is more likely to inspire and increase cognition? My guess is the latter.</p><p>As things and people around us are more similar in nature, it is easier to tune out, and continue doing the same thing over and over again, without much thought. This is the opposite of innovation. Innovation is essentially disruption. In order to shake things up, we need people with different perspectives offering their input and experiences, to create innovative and robust new products. Studies have overwhelmingly agreed — diversity and innovation are correlated.</p><blockquote>A fair question to ask is the chicken or the egg question, meaning, are companies really more innovative because they have a more diverse leadership, or the other way around? Which way is it? Now, we do not know how much is correlation versus causation, but what we do know is that clearly, in our sample, companies that are more diverse are more innovative, and companies that are more innovative have more diverse leadership, too. So it’s fair to assume that it works both ways, diversity driving innovation and innovation driving diversity. (Lorenzo, 2017)</blockquote><p><strong>2) Positivity Leads to Increased Productivity</strong></p><p>Another factor playing into employee engagement is negative vs. positive workplaces. Harvard Business Review’s article on positive workforces underlines the dangers of a high-pressure organizations, with a tendency towards cut-throat behavior.</p><blockquote>A large and growing body of research on positive organizational psychology demonstrates that not only is a cut-throat environment harmful to productivity over time, but that a positive environment will lead to dramatic benefits for employers, employees, and the bottom line.</blockquote><blockquote>First, health care expenditures at high-pressure companies are nearly <a href="http://www.biomedcentral.com/1471-2458/11/642">50% greater</a> than at other organizations. The American Psychological Association <a href="http://www.apa.org/news/press/releases/stress/2014/stress-report.pdf">estimates</a> that more than $500 billion is siphoned off from the U.S. economy because of workplace stress, and 550 million workdays are lost each year due to stress on the job. Sixty percent to 80% of workplace accidents are attributed to stress, and it’s <a href="http://www.stress.org/americas-1-health-problem/">estimated</a> that more than 80% of doctor visits are due to stress. (HBR, 2015)</blockquote><p>Stressful and cutthroat organizations tend to reduce employee engagement in the workplace. This is detrimental to organizations, since the cost of employee disengagement is high<em>.</em></p><blockquote>Engagement in work — which is associated with feeling valued, secure, supported, and respected — is generally negatively associated with a high-stress, cut-throat culture.</blockquote><blockquote>In studies by the <a href="https://smith.queensu.ca/magazine/issues/winter-2014">Queens School of Business</a> and by the <a href="http://www.gallup.com/services/178514/state-american-workplace.aspx">Gallup Organization</a>, disengaged workers had 37% higher absenteeism, 49% more accidents, and 60% more errors and defects. In organizations with low employee engagement scores, they experienced 18% lower productivity, 16% lower profitability, 37% lower job growth, and 65% lower share price over time. Importantly, businesses with highly engaged employees enjoyed 100% more job applications. (HBR, 2015)</blockquote><p>Studies show that negative organizational cultures, unsurprisingly, led to higher employee turnover. This is a loss of investment, time and energy. Additionally, there is the cost of training new employees. If the culture does not change, the problem persists and becomes cyclical, as illustrated in “The Banana.”</p><p>To increase positivity in the workplace, Harvard Business Review points to six essential characteristics:</p><blockquote>· Caring for, being interested in, and maintaining responsibility for colleagues as friends.<br>· Providing support for one another, including offering kindness and compassion when others are struggling.<br>· Avoiding blame and forgive mistakes.<br>· Inspiring one another at work.<br>· Emphasizing the meaningfulness of the work.<br>· Treating one another with respect, gratitude, trust, and integrity. (HBR, 2015)</blockquote><p>A few suggestions on where to begin, for leaders: <br>Promote empathy, encourage social connections, encourage freedom in communication, and go out of your way to help.</p><p><strong>3) 5 Keys to a Successful Google Team</strong></p><p>Google’s People Operations team found that psychological safety is the most important characteristic of successful teams.</p><blockquote>We learned that there are five key dynamics that set successful teams apart from other teams at Google:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-V8WiUm9hDg-0q6q1IzUDg.png" /><figcaption>5 Keys to a Successful Google Team</figcaption></figure><p><strong>4) Establish Intrinsic Reward Systems</strong></p><p>Google’s study on effective teams coincides with career analyst Dan Pink’s 2009 TED Talk called, “The Puzzle of Motivation.”</p><blockquote>The secret to high performance isn’t rewards and punishments, but that unseen intrinsic drive — the drive to do things for their own sake. The drive to do things because they matter. (Pink, 2009)</blockquote><p>Many companies offer extrinsic rewards, such as bonuses, raises and other compensation. But the truth is, extrinsic rewards are often uninspiring. If employees don’t truly believe in the underlying purpose behind the task, they will run out of steam. Many studies prove that intrinsic motivation is the key to productivity/motivation.</p><blockquote>It’s built around intrinsic motivation. Around the desire to do things because they matter, because we like it, they’re interesting, or part of something important. That new operating system for our businesses revolves around three elements: autonomy, mastery and purpose. Autonomy: the urge to direct our own lives. Mastery: the desire to get better and better at something that matters. Purpose: the yearning to do what we do in the service of something larger than ourselves. These are the building blocks of an entirely new operating system for our businesses. (Pink, 2009)</blockquote><p>Pink goes on to say that the traditional management model works great for compliance, but if you want to boost employee engagement, self-direction is better. Autonomy is proven to lead to increased productivity at work.</p><p><strong>5) Increase Autonomy</strong></p><blockquote>Google engineers can spend 20% of their time working on anything they want. They have autonomy over their time, their task, their team, their technique. Radical amounts of autonomy. And at Google, as many of you know, about half of the new products in a typical year are birthed during that 20% time: things like Gmail, Orkut, Google News. (Pink, 2009)</blockquote><p>Partial autonomy not only works, it’s also highly beneficial to the bottom line. This example is not an exception to the rule. Autonomy has also been very successful at many other companies, such as <a href="https://www.atlassian.com/blog/leadership/barriers-around-instrinsic-motivation-workplace">Atlassian</a>, Microsoft, Southwest Airlines and Dreamworks.</p><p>In conclusion, employee engagement has barely budged in over ten years. Isn’t it time to innovate towards better work environments? Making progress in the direction of more autonomous, positive and innovative workplaces is, after all, the link towards increased profitability, innovation, engagement, retention and productivity.</p><p><strong>Sources:</strong><br><a href="https://hbr.org/2015/12/proof-that-positive-work-cultures-are-more-productive">Positive Work Cultures are More Productive</a> — Harvard Business Review, <a href="https://hbr.org/search?term=emma%20sepp%E4l%E4">Emma Seppälä</a> &amp; <a href="https://hbr.org/search?term=kim%20cameron">Kim Cameron</a>, 2015<br><a href="https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/">5 Keys to a Successful Google Team</a> — Google, People Operations Team, Julia Rozovsky, 2015<br><a href="https://www.ted.com/talks/dan_pink_the_puzzle_of_motivation?language=en">The Puzzle of Motivation</a>, TEDGlobal 2009, Dan Pink, 2009<br><a href="https://news.gallup.com/businessjournal/182792/managers-account-variance-employee-engagement.aspx">Employee Engagement</a>, Gallup Corporation, Randall Beck and <a href="https://www.gallup.com/people/item.aspx?a=100075">Jim H</a>arter, 2015<br><a href="https://www.ted.com/talks/rocio_lorenzo_how_diversity_makes_teams_more_innovative?language=en">How Diversity Makes Teams More Innovative</a>, TED@BCG Milan, Rocio Lorenzo, 2017</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ceb2a9aa1d02" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>