<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Salva Rocher on Medium]]></title>
        <description><![CDATA[Stories by Salva Rocher on Medium]]></description>
        <link>https://medium.com/@salvarocher?source=rss-631aacf3691b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*J2Vy_h3NBOz-qJL1_XHL5A.jpeg</url>
            <title>Stories by Salva Rocher on Medium</title>
            <link>https://medium.com/@salvarocher?source=rss-631aacf3691b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 14:21:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@salvarocher/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Business intelligence applied to a user engagement problem]]></title>
            <link>https://medium.com/data-science/business-intelligence-applied-to-a-user-engagement-problem-8e215d8f8cda?source=rss-631aacf3691b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e215d8f8cda</guid>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[product]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Salva Rocher]]></dc:creator>
            <pubDate>Thu, 31 Oct 2019 01:40:33 GMT</pubDate>
            <atom:updated>2019-10-31T13:59:58.338Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CUZ6YbzAtjvY1rox7xU60w.jpeg" /></figure><h3><strong>Context</strong></h3><p>One key area where companies often become concerned and willing to exploit data to get guidance on where the problems are located and how they can be solved is related to consumer retention or user engagement.</p><p>So, let’s imagine that we are analysts working for a technological company whose most important KPI revolves around how customers are using its main product. A product that is a sort of an internal social network within the workplace where a user can send and receive individual or group messages, like them, share documents, check inbox, post ideas, and search for info from other users of the same company. We will call it “Hub” moving on for simplification purposes.</p><p>Hub is sold to several companies around the world so that its employees have communication tools available. They can make use of it for free but the companies pay fees to enable that.</p><h3>Data</h3><p>The data is comprised of 3 tables: users, events and emails.</p><p>The first table collects basic info on the users, that is, identification number, the status of the account, date when the lead signed up, date when the lead completed the signup process and became a user, language and company the user works for. The table is referred to as users.</p><p>The second table, called events, gathers info on the interactions of the users with the product, type of interaction, dates and times of it, location and device used.</p><p>The last table, called emails, is about the user response on the email interactions between our company and the user. It is just limited to the emails sent in the form of content weekly digest to foster engagement.</p><p>Data, along with the complete version of the code, can be found in my <a href="https://github.com/Salvinha-vlc/Business-intelligence/tree/master/User_engagement">Github repo for this project</a>.</p><h3>Workflow</h3><p>The dataset described above was pushed to a PostgreSQL database. On the other hand, SQLalchemy library, which incorporates psycopg2 adapter, allows us through its engine module to blend the workflow between postgreSQL database and Python, so that we can run our queries in SQL language without leaving our notebook, and this means that Python smoothly takes over the dataframes queried to perform further data manipulation and visualizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jYXV2B7-runFHnJe7WZBfQ.png" /><figcaption>Snippet code of the “create_engine” module of SQLalchemy that enables the blended workflow</figcaption></figure><h3>The issue. A decrease in user engagement</h3><p>End of August comes and when reviewing key metric dashboards something stands out (see chart below). User engagement has experienced a decline.</p><p>Engagement, in this case, is defined as server calls from the interaction with the product, in other words, when a user is logging in to the Hub during a given week, he/she is engaging with the product. So, users not logging in, hence not engaging, is definitely not good for the business, so as an analyst you wonder what is going on and get ready to crunch the database to get insights and guide possible corrective actions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TAXPvCDH2sQNhstiK9sLjg.png" /></figure><h3>Potential sources of the problem</h3><p>Before starting querying away, it is recommendable to think on all the possible points that may result in a user engagement drop. Here are the ones that come to my mind regarding this case:</p><ul><li><strong>Holidays</strong>. August is in many countries, especially in those located in the northern hemisphere, the month with the greater number of vacation leaves overall, so this might well be what explains the drop.</li><li><strong>Performance issue</strong>. Many times the main source of issues among ecommerce, marketplaces or technological companies springs from the pure interaction with the website/app and it might affect differently a user or another depending simply on the device type and operative system used to log in. It might also happen that tech team has introduced a change in the web/app frontend that affects a particular feature impacting directly the engagement in a negative way. Website loading slowly or broken features along the session navigation may cause bad user experience translating in less product engagement.</li><li><strong>Sign-up feature flow</strong>. For a lead to become a user who is able to log in and use the product, he/she has to go first through each step of the sign-up flow. It could be the case for instance that there are technical issues in any feature of the flow that might be preventing new users to sign up, hence missing the new engagement contribution from newbies that naturally offsets the systematic churn.</li><li><strong>Broken tracking log</strong>. It could also happen that the code/log that identifies every single user action is itself corrupted or affected. Seeing a drop in a particular usage feature to absolutely zero would hint the issue is coming from this direction.</li><li><strong>Bad data</strong>. We could be storing records of “dirty” product traffic, meaning by that not the one from real users, which is the one we are interested in. For instance, a new feature or new technological infrastructure might be developed and tested by QA. Usually, QA and production data are separated but QA activity due to a glitch might end up being mixed with real external users traffic. Another possibility that explains this bad data could come from bots that due to those same changes are now crawling the web in a different way, hence causing traffic records swings.</li><li><strong>Technical issues related with the recap emails.</strong> They are sent from Hub content team in the form of weekly digest. They are designed to bring customers back and foster engagement by summarizing its activity and colleagues’ in the prior week.</li></ul><p>Let’s analyze these elements one by one.</p><h4>Holidays</h4><p>This query below tries to find the countries where the number of active users in August was greater than in July. In order to consider holidays as the reason for the user engagement drop we would like to see in the query output a list of countries where august is not the main vacation month, which we would expect either from countries whose tradition makes that the predominant vacation month is none in particular, or just another one because of, for example, being located in the southern hemisphere.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a1567f3c85256ad8dd96772603b1d54/href">https://medium.com/media/3a1567f3c85256ad8dd96772603b1d54/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/186/1*Kz2p5ovuEYyQ7ML58yfrtw.png" /></figure><p>I am Spanish and I can say that august is traditionally the predominant vacation month. Greece, for instance, is much of the same as far as I know.</p><p>So it somewhat discourages me to think that holidays are the reason for the user engagement drop. I would rather be investigating other possible reasons rather than digging deeper into this one.</p><p>Here below is how the engagement user looks like by month in Spain and Greece.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bd0f9773e914892f14cb7b365cd7824/href">https://medium.com/media/5bd0f9773e914892f14cb7b365cd7824/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xYHaV2d5IKEmyyCFl-CNlQ.png" /></figure><p>Active users are increasing in a month where we would expect otherwise if this were to be the actual reason for the overall dip in Hub user engagement.</p><h4>Performance issue</h4><p><strong><em>Device type</em></strong></p><p>In order to know whether there was some kind of performance issue in the interaction with the product, it might be interesting to see whether a specific OS or device was particularly affected.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2fcb96f8b644b000b0dc7d3b6e44cb8a/href">https://medium.com/media/2fcb96f8b644b000b0dc7d3b6e44cb8a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zQx4FH6IRCauhlMnY2Dlvw.png" /></figure><p>The chart above shows that the drop in engagement comes from all sources but it is particularly pronounced for users that have logged in through their phones and tablets. On top of that, the proportion of laptop users is larger than other device types’, so proportionally the drop is quite different between the groups.</p><p>Let’s take a look at how these proportions have varied from July to August.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a0a230c040073e75545fecbd551850c/href">https://medium.com/media/6a0a230c040073e75545fecbd551850c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SNWVki63WW1WSUXUYr8Sew.png" /></figure><p>The difference in how the device type is related to the variation in user engagement is great enough to shortlist this as a likely reason to explain the drop in user engagement. It does not mean it is for sure the cause, but looking at this data something is fishy and worth investigating further. For example, it could be useful to talk to the tech team to find out whether they were tweaking something in the app (used through phones and tablets rather than through PC/laptops), in order to figure out a potential issue from their developments.</p><p><strong><em>Operative system</em></strong></p><p>Same way as shown above for the device type, the analysis could also be broken down by operative system (which by the way correlates to a certain extent with device type).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6cb08f88f8b4415a7ad4eb6fdc1ab8b4/href">https://medium.com/media/6cb08f88f8b4415a7ad4eb6fdc1ab8b4/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QoRIC4OITDbBW8M1sRbxcw.png" /></figure><p>The above chart tells us that the drop seems to mostly come from Apple and Android users and this to a certain extent is connected with previous charts where we were seeing device type effect.</p><p>Let’s understand the relative magnitude of this implication by looking at the proportional variation from July to August per OS.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/69c1ab46a28eac4e9e8fe64dd17bc9de/href">https://medium.com/media/69c1ab46a28eac4e9e8fe64dd17bc9de/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I9VlZejlf1e3eS6rxDwuFA.png" /></figure><p>In line with what discovered before, phones and tablets seem to accumulate the greater proportion of the user engagement drop, and as shown in this last chart, especially android devices.</p><p>However, if this were the only source for the problem, why then window laptops see a drop, though less marked, still a drop, in user engagement?</p><p>Let’s keep investigating other possible reasons.</p><h4><strong>Sign-up feature flow</strong></h4><p>First, it is convenient to check the new users per month.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0fdb2994cb3ff0300a4396378dbdf3e/href">https://medium.com/media/c0fdb2994cb3ff0300a4396378dbdf3e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g-z5YDugkHE9EPRqwKCMSA.png" /></figure><p>The purple line shows the evolution of new users per month and is connected to the left Y-axis while the bars show the monthly % variation of new users which is connected to the right Y-axis.</p><p>By reading the chart we can see that there is a clear upward trend in user acquisition over the 2013–2014 window span. In particular, if we pay attention to the latest summer (which is where the user engagement dip analysis is centered on), there has actually been an increase in the number of new users in August (2014) as compared to July (2014). This alone would be enough to discard an issue in the signup flow as one of the reasons for our problem.</p><p>However since the user acquisition growth rate has clearly slowed down in August, it could be safe to take a closer look into the signup flow steps.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d35a1599693e82532973fd4f190e5f6/href">https://medium.com/media/8d35a1599693e82532973fd4f190e5f6/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Esalvinha-vlc%2F1%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Esalvinha-vlc%2F1.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c48520a496dc3cfe57c34447d908905d/href">https://medium.com/media/c48520a496dc3cfe57c34447d908905d/href</a></iframe><p>And captured over several months leading up to August (included).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s1d6_weDlJgkM962oNpfFg.png" /></figure><p>As we can see, all the steps of the signup process funnel maintain their proportions respect to each other along the different months, so we can safely rule out a technical issue in this flow as a root cause for our problem.</p><h4>Broken tracking logs</h4><p>It might be the case that some specific feature within the usage flow could not be tracked because of problems with the logs (identifiers for every single action taken from users when interacting with the product).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d8eda593a02dea0d8299915a71845cd/href">https://medium.com/media/5d8eda593a02dea0d8299915a71845cd/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2PjvlaETZZl8eFrwyQFwFg.png" /></figure><p>It is not the case. There is a generalized drop in all the events to the point that seems that they are moving in sync. Let’s zoom in to July-August and see it in relative proportions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e11e51853a396c3751d113563eb92d01/href">https://medium.com/media/e11e51853a396c3751d113563eb92d01/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7KPIB_6-Q_L1YuiW7S_vag.png" /></figure><p>Indeed, there are some differences in how the engagement drop is coming into being through them, but they do not seem too relevant to consider that tracking logs belonging to any of them in particular are broken.</p><h4>Bad data</h4><p>In order to ascertain whether this is the cause, we need to see activity by user and check whether there is unusual activity by a few of them. This would cause artificial activity or in other words, a certain kind of spike/unusual rise, which in our case could make sense to have it in July. If this happened to be true, we could say that august drop in user engagement is not such, because it would be rather a problem of fake user engagement in the preceding month.</p><p>First, let’s check whether there is anyone starting engaging with the web but not logging in, this would be pointing to those something weird, like someone who does not need to sign up to be part of the record logs (bot-like behavior).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4783fb00833009714ed929097cac890d/href">https://medium.com/media/4783fb00833009714ed929097cac890d/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*Z3Q8eeWGi1PQmW0ddBbUVQ.png" /></figure><p>That means that every single record of product interaction comes after a login.</p><p>Second, let’s see if there are some days within the week with unusual activity.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52b5d118f7aa91c24ec1e56cb4dbcfd6/href">https://medium.com/media/52b5d118f7aa91c24ec1e56cb4dbcfd6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WuQuny7WL_wuWLu9aliuig.png" /></figure><p>We can see that there is a systematic fluctuation intra-week where the weekdays see more Hub activity and weekends naturally less, and the most relevant perhaps is the peak in the second half of July, that while does not seem extraordinarily high, it cannot be ruled out carelessly, hence worth to dig deeper.</p><p>And third, it could also be helpful to check whether there was some day across the time period for which there are log events data available, with extraordinarily unusual activity.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d162b222b63ed12313a05a220c1598af/href">https://medium.com/media/d162b222b63ed12313a05a220c1598af/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EsqivJgbXGopYRkY7WGxfg.png" /></figure><p>It seems that there are indeed some days with unusual activity but it is hard to say it is extraordinarily unusual. If there would be bad data behind the user engagement variation from July to August we would expect them to be concentrated in July. Let’s find out.</p><p>Since I may think we are talking about unusual activity, up to 70 events within a certain day and from a single user, let’s consider the existence of outliers, fake or not, but outliers.</p><p>In order to characterize an outlier we need to define a threshold from the interquantile range (IQR) of our data. Here is my way to do it.</p><p>First we get the IQR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*SqosJk0MhTupq14rnN8plQ.png" /></figure><p>And then it can be used to isolate the outliers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ad3d411f1aec6fa2e7f8b792bf73e32/href">https://medium.com/media/4ad3d411f1aec6fa2e7f8b792bf73e32/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hD6j3moqIdtj89YkpfUOTw.png" /></figure><p>Two things can be taken from this barplot above — 1st is that July does not seem to have an extraordinary superior activity from outliers (or potential bad/fake data), in fact, is similar to prior months. The 2nd is that august sees really a low in interactions from outliers.</p><p>This opens up many scenarios that would need to be investigated further upon prioritization to shed more light onto them. Among them, I would stand these ones out.</p><ul><li>It is possible that there has been a change in august in how the search engines crawl the website and that would explain the considerable drop in the month. One possible way to take this further would be to analyze user patterns and cluster them. This would facilitate to identify bots to then verify if the sought user engagement drop comes from a decrease in their activity.</li><li>Important product developments over the last months which were accompanied by an intense quality assurance work that finished before August. For some reason, QA data has made it to the production logs and it is meddling with the metrics. I see it unlikely but does not hurt to check with product development department.</li><li>Other reasons explained in other points like for instance technical issues that causes worsened performance hence making those most active users to react after the bad user experience by simply engaging less.</li></ul><h4>Weekly digest emails</h4><p>As said before while listing all possible sources for the problem, these emails are a tool from customer and content teams at Hub to promote engagement among its users. They are a summary of the info that is relevant for each user.</p><p>So let’s see if data tells us something interesting about them. Plain data has been manipulated further into key rates.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4048e725fdda2a4b10ca88635579e839/href">https://medium.com/media/4048e725fdda2a4b10ca88635579e839/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QNNne_5mjPP-_VPddLjgnw.png" /></figure><p>Indeed, something immediately stands out. While the weekly digest emails open rate remains constant over the months, we can clearly see that the click-through rate drops in august from around 50% to around 30%. That is, 2 out of 3 users don’t care much about these engagement emails, however out of those who do care, before august, 1 out of 2 reengaged into the product by clicking through the links in those emails, whereas in august that figure becomes into 1 user out of 3 of them reengaging after the emails. In a nutshell, effectiveness of the weekly digest emails has fallen 35% from July to August, translating directly in less user engagement.</p><p>Either there is a technical issue in the email links themselves preventing users to click-through, or there has been a change in the way that the content which is relevant for each user is chosen and delivered via emails (be it from an algorithm or from a specialized team).</p><h3>Conclusions</h3><p>Listing many different possible reasons that aim at explaining the user engagement drop was very useful to reduce the chances we are neglecting the real source of the problem. Then each query allows us to get the data ready for analysis consumption. From that analysis it seems that there is something wrong with the app navigation that affects phone and tablet users, and with the content of the engagement emails.</p><p>It is important to note that the findings are not conclusive, further questions to product and tech teams will probably be needed, but at least we have been able to get a well-grounded head start into finding out the answer for a troubling question.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e215d8f8cda" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/business-intelligence-applied-to-a-user-engagement-problem-8e215d8f8cda">Business intelligence applied to a user engagement problem</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A data science approach to personality models]]></title>
            <link>https://medium.com/data-science/a-data-science-approach-to-personality-models-d3c7e18a377?source=rss-631aacf3691b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3c7e18a377</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[factor-analysis]]></category>
            <category><![CDATA[psychology]]></category>
            <dc:creator><![CDATA[Salva Rocher]]></dc:creator>
            <pubDate>Tue, 01 Oct 2019 18:12:44 GMT</pubDate>
            <atom:updated>2019-10-03T03:32:50.524Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*kl2gyUZBQNXKvTiuiIA0WQ.jpeg" /></figure><h3><strong>1. Introduction</strong></h3><p>I have personally always found interesting the differences in how people are or act, so this study will allow us to know a little more about it. We will have insights on how the characteristics of a person relate to their personality traits and habits, and even to be able to do predictions based on some demographics.</p><p>In particular, most of the current personality tests draw upon the generally accepted 5 factor model, also known as OCEAN, named after the first letters of the 5 factors that comprise it.</p><p>[O] Openness to experience. (<em>inventive/curious</em> vs. <em>consistent/cautious</em>)</p><p>[C] Conscientiousness. <em>(efficient/organized vs. easy-going/careless)</em></p><p>[E] Extroversion.<em> (outgoing/energetic vs. solitary/reserved)</em></p><p>[A] Agreeableness.<em> (friendly/compassionate vs. challenging/detached)</em></p><p>[N] Neuroticism. <em>(sensitive/nervous vs. secure/confident)</em></p><p>This model is basically a taxonomy method for personality traits that is generally accepted by the experts in the field because it allows to capture most relevant personality differences among people. However it has also been object of debate and critiques.</p><p>It has been argued that there are limitations to the scope of the Big Five model as an explanatory or predictive theory. It has also been argued that measures of the Big Five account for only 56% of the normal personality trait sphere.</p><p>One common criticism is that the Big Five does not explain all of human personality, in fact, some psychologists have dissented from the model precisely because they feel it neglects other domains of personality. Due to this reason, some have grown to call the method of the Big Five as the “psychology of the stranger”, because they refer to traits that are relatively easy to observe in a stranger and other aspects of personality that are more privately held or more context-dependent are excluded.</p><p>In light of the scope limitations critiques, maybe a new psychometric model could be developed in order to find relevant dimensions or alternative latent factors in explaining differences in personality. So this became the core of this paper, as I will explain in detail in the following sections.</p><h3><strong>2. Data</strong></h3><p>The dataset, which is not anchored in the OCEAN approach, is an interesting collection of answers for a bunch of varied questions ranging from music preferences to phobias, by way of personal interests, personality traits, lifestyle and personal characteristics. It was collected via survey and the respondents were 1.010 European people ranging from 15 to 30 years old. It is convenient to mention that there were few missing values in the answers, but since the sample size is large enough we could afford carefully removing completely those rows. To know more about the list of questions, please refer to Appendix.</p><p>The sampling units are the individual responses, structured in rows. In the columns we can find the variables. The ones I have focused my analysis in are, basic characteristics on one side, and personality traits on the other.</p><h4>Basic characteristics</h4><p><strong>Gender.</strong> Categorical, Male, Female.</p><p><strong>Age</strong>. Numerical variable, but transformed into an ordinal one following this criterion:</p><ul><li>1: 15–19 yo</li><li>2: 20–24 yo</li><li>3: 25–30 yo (entails one year more than the other groups, but made sense since is the smaller group in terms of respondents)</li></ul><p><strong>Education.</strong> Ordinal variable. Categories are the following:</p><ul><li>1: Primary school</li><li>2: Secondary school</li><li>3: College degree</li><li>4: Masters degree and PhD degree (I have joined them both since there were very few Doctors in the respondents)</li></ul><p><strong>Place</strong> where respondents spent most of their childhood in. Categories:</p><ul><li>Village</li><li>City</li></ul><h4>Personality traits</h4><p>To be more precise, the list does not only embrace personality traits, but also beliefs and opinions about life, which in my view can be traced back to personality traits.</p><p>It is comprised of 54 questions all of them with suggested answers in a 1 to 5 scale, being 1 strongly disagree and 5 strongly agree. Actually there were 57, but for simplicity purposes 3 were removed since they were of a different nature than the rest.</p><h4>Other data of interest</h4><p>Apart from the personality data, there are also questions about interests, music preferences, phobias and life aspects as lifestyle.</p><p>The answers range as well from 1 to 5 scale.</p><h3>3. Methodology</h3><p>In order to move further in our objective of finding valuable insights on how personality traits relate to characteristics of a person, it is convenient to spend some time first working on the list of questions for personality traits. In this sense, we are talking about a <strong>Factor analysis</strong>. The rationale behind is two-fold.</p><p>First, we want to be able to group the variables (questions) in similar dimensions without losing power of information, in other words, we want a dimension reduction. The reason why this comes handy is because moving forward it is much more convenient to work with a few variables rather than with a lot of them (remember that we are talking about 55 questions, so that makes 55 variables); especially on the prospect of working with regression techniques down the road. So in a nutshell it is as if we were screening variables for subsequent analysis.</p><p>Secondly, it will allow us to discover latent structures which maybe can build up to a form of an alternative psychometric model for personality tests.</p><p>So as there is no model that the variables were created with, or any reference we can compare them to, we need to explore if we can find the relevant dimensions underlying, so the analysis is actually an <strong>exploratory factor analysis</strong>.</p><p>Once the rationale is clear, how do we proceed?</p><p>First, it is good to have a sense on how the variables relate to one another. For this there are 2 options, one is to plot a <strong>correlation heatmap matrix</strong>, the other is to run a <strong>PCA</strong> on all the questions to see them regressed on the first two dimensions and to observe how they relate to each other. See that PCA is a way of dimension reduction, but for the purpose here it was better to draw upon other methods. Both were computed as we will be able to see in the results section.</p><p>In particular, the correlation visualization, was plotted using a hierarchical clustering which was based on Ward´s minimum variance agglomeration method. Ward´s clustering follows a criterion for merging two clusters at each node of the tree where it tries to maximize the separation of the new cluster´s mean from the mean of other clusters, which is equivalent to minimizing the dispersion within the newly combined cluster. R package Corrplot allows us to compute this.</p><p>After this, the method used is the principal axis analysis. The main point here is the rotation after extraction of factors to keep finding latent structures. The rotation method used was varimax rotations. Factanal package in R was used to compute this.</p><p>A question that arises at this point is how many rotations do we need to perform? Meaning, where do we stop getting factors or dimensions?</p><p>Here a scree plot comes handy. A scree plot depicts the Eigenvalues of the correlation matrix of the variables (questions) against the factor number. Nscree package in R aids us in this matter.</p><p>Once we have come up with a small number of factors that explain most of the variance observed in the original variables, the idea is to regress them on people characteristics in order to discriminate or classify people. This is done through <strong>classification and regression trees (CART).</strong></p><p>Regression trees develop easy-to-visualize decision rules for predicting a categorical variable “being classified” in one way or another based on some relevant variables. R was also used to compute this, in particular, the rpart.plot package.</p><p>This is basically the methodology used in the backbone of the project. However in addition, I wanted to get some other insights. In particular, I wanted to analyze healthy lifestyle. For that I resorted to <strong>Correspondence Analysis</strong>. Its main purpose is to depict visually in a map (usually from a multidimensional space projected onto a plane) the relationships between answers — from 1 to 5 being 1 strongly disagree and 5 strongly agree, respond to the following statement “I live a very healthy lifestyle”; and categories, which in this case where the ones for the variables gender, age (in groups) and education.</p><h3>4. Results</h3><p>Before starting, the list of original questions that can be found in Appendix, has been given with numbers too for convenience of the visualization, since 54 variables together sometimes makes it sometimes difficult to read the graphs.</p><h4>PCA</h4><p>First let´s have a look at the PCA run into the first two dimensions. For that we need to apply PCA to our dataset as in below code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*NZQW3oBevydhC3s3tUB2SQ.png" /><figcaption>Thanks to Prof. Michael Greenacre for its PCA customized function</figcaption></figure><p>After running the code, the result looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*bCTXsMtlDSySvgE4xmRN_A.png" /></figure><p>Here we can have a first rough approximation on how the variables relate between them in the first two dimensions. We can see that variance explained by them accounts for around 16%, which hints we would be better off exploring and taking into account more dimensions.</p><p>Looking at the PCA, for instance in dimension 2, we can see that there is a group of variables that we will be able to see them in more detail later on, that are very negatively correlated (as represented by the opposite arrows), those are questions 52, 47 and 31, that is, interests or hobbies, energy levels and number of friends as opposed to questions 27 and 24, that is, changing the past (remorse, melancholy) and loneliness. As we can see both sides evoke totally opposite sign feelings.</p><h4>Correlations heatmap matrix</h4><p>Let´s move on now to see relationships between questions into more detail. Here below is the correlation matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wUz_DL60YoGfy0ZoMVCZSQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*h6zLbKtp9s5VPXPFXfFALA.png" /></figure><p>X and Y axis show the 54 questions. The sorting already gives some ideas and hints of what we will find down the road when computing the factor analysis. In particular, the size and color of each square in the matrix cells indicates strength of the Pearson´s correlation coefficient between each pair.</p><p>Additionally, I have highlighted at discretion 9 rectangles which show groupings between similar traits. For instance, questions 2, 3 and 4 (2th rectangle starting from the bottom right corner), “prioritizing workload”, “writing notes”, “workaholism” are concepts that embrace an underlying similar concept, which is the diligence, the drive for the hard work. (Please note that the sample units are young people from 15 to 30, so in most of cases their equivalent to older adults of working is studying). Also, funnily enough, this group I have mentioned, correlates negatively to questions 25, 51 (3rd rectangle starting from the top left corner), “cheating in school” and “difficulty to get up soon in the mornings”, which makes total sense.</p><p>In any case, the correlations matrix above is a very nice starting point, but what is going to tell us which are the main latent factors beneath the survey questions, is the factor analysis.</p><p>So let´s get right to it.</p><h4>Factor analysis</h4><p>First, we define the function to plot the dimensions that we will be getting and its arguments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M748Rtpy-DdPfnTrQ-gTtw.png" /></figure><p>Then, we invoke factanal package to obtain the factors using varimax rotation method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*scs7JrFCMbLHrR0VIH_pzw.png" /></figure><p>Lastly, we call the function we have defined before over the variable that cointains the factors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*iHh-xaj6ps96WL4bDS-p_Q.png" /></figure><p>The way to read the following factor graphs is basically focusing on the extremes labels of the X and Y axis since they are the ones that characterize or explain most of the variance for each dimension. In other words, they represent the variables with the highest loadings for each factor.</p><h4>Factor 1</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*2j8ct2LF3XT3uTjOPkSaxw.png" /><figcaption>Dimension of factor 1 can be read horizontally and the one for factor 2, vertically</figcaption></figure><p>Again, meaning of each variable can be found in Appendix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uYmNNzw6jkWAnpX42hRz_g.png" /><figcaption>F1. Positiveness</figcaption></figure><p>I have called it Positiveness factor because they relate to positive feelings, it seems that they are positive people who feel happy, energetic, having good dreams at night, and who think positively about themselves and their personality. On the other side of the spectrum we find to those people who don’t feel good because they feel lonely (very negatively correlated with both “happiness in life” and “energy levels” as we can see in previously shown correlations matrix), and who experience negative emotions maybe as a result of them not being positive, like remorse and willingness to change the past, and instability on their mood.</p><p><strong>Factor 2</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G1GepGTVQ_lMM1s7tOnlqQ.png" /><figcaption>F2. Extroversion</figcaption></figure><p>The second factor is a valid measure of the extroversion/introversion personality dimension. Those people who are extroverts find it easy and stimulating to enter and adapt to new environments with new people, enjoy meeting new people, have an overall large number of friends, have many interests and hobbies, and also it is easier for them to be assertive. Assertiveness is something which is not always easy to apply, since the other part of the relationship might not take it well, an introvert, more sensitive and with less friends in general, might want to steer clear of. On the other hand, we see typical traits for introverts (<a href="https://introvertspring.com/what-is-an-introvert/#whatisintrovert">introvert spring</a>), they tend to feel uncomfortable when surrounded by a lot of people, so they tend to dislike/fear public speaking; they also tend to ruminate and think over things quite a bit, hence decision making and responding to a serious letter requires time. Introvert also tend to be introspective, hence worrying about health relatively more than extroverts on average, which it also makes sense, after all, an introvert is more aware of what is going on inside of them as compared to an extrovert, so spotting health irregularities and ruminating over them is more likely to happen for them than for the extroverts.</p><p><strong>Factor 3</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*mjgyZfmnNgXgFyaEXlR_mg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*ibHlodldAXbl1Z0rneVyjA.png" /><figcaption>Dimension for factor 3 can be read vertically</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i8JZ5bFdmCZcDjUj_A22vg.png" /><figcaption>F3 — Diligence</figcaption></figure><p>The 3rd dimension is diligence. As we mentioned briefly commenting on the PCA, this factor represents people who are diligent upon their duties, who can be considered as disciplined and hardworking people and who in consequence is also reliable. Concepts like writing notes and workaholism reflect this clearly. Others like prioritizing workload and thinking ahead might not be so obvious upfront, but thinking about it carefully, to do that in a systematic way is not easy and let´s say comfortable, so it shows a certain amount of diligence and drive to fulfill the duties well and on time. On the opposite side we have those who find difficulties in getting up in the mornings and who tend to cheat at school, in a nutshell, those who in general are lazier and not so diligent.</p><p><strong>Factor 4</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*-T3-rI7bk2Fh0sP718861w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*E2X9KbQ-nRDmNoec68zwaQ.png" /><figcaption>Dimension for factor 4 can be read vertically</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6LzjcSQLss8r7f2ZAB1n7w.png" /><figcaption>F4 — Warmth</figcaption></figure><p>The 4th personality dimension is “warmth”, as in a compendium of good-heartedness, sensitiveness and honesty. It reflects aspects of personality like for instance, empathy towards others, suffering when seeing others suffering, being generous giving presents when it is time to, crying when one feel is struggling, and enjoying children´s company. The first two show empathy, the 3rd one good-heartedness, the 4th shows someone who does not refrain from expressing emotions like crying, so is honest in reflecting how they really are and feel; and the 5th shows also warmth, after all, small children are playful and smiling often which forces to somehow reciprocate, so if someone is kind of cold, serious or don’t want to be bothered with such things will tend to enjoy less children´s company.</p><p>On the other side of the spectrum, we have those which are doubled faced, those who prefer big dangerous dogs to small ones and those who fall easily for someone but loss interest just as fast. They might seem apparently unconnected topics, but to me they are not, since they picture those people who tend to build a wall around them so others don’t really see who they are or how they feel. The 1st seems obvious as they state it straight away, the 2nd might match those who build a wall by wanting to come across as strong and dangerous, and the 3rd reflects those who have a lot of difficulties in opening up, showing for how they really are, sympathizing and connecting at a deeper level with others, hence they loss interest quick and change partner constantly.</p><p><strong>Factor 5</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/387/1*Jfr5CMXKizfCvP3s6OWAxA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*9hyO1TXndf7EWMSJE3RzNw.png" /><figcaption>Dimension for factor 5 can be read vertically</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k2BrqqDevQMfYujqbtJhJA.png" /><figcaption>F5 — Superficiality</figcaption></figure><p>I have called the 5th factor superficiality as in a way to represent those types of personalities who give a predominant importance to image, status and that often is linked to materialism, egocentricity and shallowness. The 1st variable seems to me that puts the focus of the personal relationships in how useful knowing a person might be for him/her, which comes through to me as self-interested, the 2nd one is linked to the importance they give to their physical image, and the 3rd reflects those who brag about their achievements.</p><p>On the opposite side we have personality traits who move away from materialism and selfishness since the 1st question respond to those who hand in items lost that, although valuable, are not theirs; and the 2nd one responds to those who value more friends than money.</p><p>So far, we are coming up with factors that apart from making sense, they also explain variance of the values of the variables. So, the question that comes up at this point is, how many factors to select?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*sxn2_UAB8Gkh6ZqHTThdnQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*k8xqBYlsBNUP7NNjHKp94Q.png" /></figure><p>Above scree test sheds some light onto this. We can see that there are 17 Eigenvalues above the mark of 0, which tells us that there are up to 17 factors that explains at least some variance. 10 is considered to be the optimal number of factors, however I tried, and they were explaining little of the variance, apart from not making much sense. So, according to my findings and to the visual exploration of the eigenvalues curve on the number of factors, we can see as the slope of the curve diminishes, adding new factors do not seem to capture and explain as large amounts of variance, so I have decided to stop at 5 factors.</p><p>It seems fantastic to have been able to bring 54 questions down to 5 dimensions that pick up most of the variance of all the variables. This is Factor 1: Positiveness, F2: Extroversion, F3: Diligence, F4: Warmth and F5: Superficiality.</p><p>But before moving on to classification trees, there is one gap to be bridged.</p><p>As stated previously, the idea was to simplify the variables, and we have got so, however, we know that the variables grouped into the factors have opposite signs. If we don’t transform the components of the factors we are not truly reflecting the nature of it. So for instance, “happiness in life” gives 5 to people who feel very happy about themselves and their lifes, but we also can find “loneliness”, which gives 5 to the maximum intensity of loneliness, hence if we leave it as it is, a happy people could give 5 “happiness in life” and 1 to “loneliness”, so we would have an average of 3. Then a lonely, negative person, gives 1 to “happiness in life” and 5 to “loneliness” giving again an average of 3. If 2 totally opposite people, as far as this personality trait is concerned, give the same average to the dimension, how can we expect the dimensions to classify properly on people characteristics?</p><p>It seems evident that a transformation has to be done. What I have done is to invert the rating for the “opposite” variables, so 1 after the transformation is 5, 2 is 4, and so on. Applying this to all the factors obtained allows us to calculate averages and use them as explanatory variables in the classification model that comes next.</p><h4>Regression trees</h4><p>What we are going to try to do here is to classify and predict which is the probability of a person belonging to a given category of the characteristic variables analyzed, given their score on the most discriminating dimensions at each level of the tree.</p><h4>Age</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BBoxUXUU8EIxgowx28136g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*BMGfmHHZbDM5ybBLUG9F8A.png" /></figure><p>Based on the groups decided and explained in data section, we can see from the top node of the tree that the largest group of respondents belong to people aging from 20 to 24. The 2nd line of the top node (and of each node) represents the expected probability of belonging to the 1st, 2nd and 3rd group respectively (equivalent to their frequencies).</p><p>We can see that scoring 3.2 in diligence dimension is the one that more favorably predicts in favor of people aging from 25 to 30 (from the observed 9% to 15%). Then, diligence below 3.2, positiveness below 3.1 but warmth above 4.2 is the combination that further helps predicting to classify in favor of “middle-aged” people (from 48% observed up to 75%). Lastly, the combination that better predicts classifying in favor of the youngest (15–19yo) is diligence below 3.2, positiveness above 3.2 and superficiality above or equal to 3.9.</p><p>So, in conclusion, the tree classifies the “oldest” as the most diligent among the respondents, the youngest as the most superficial, least positive and least warm, and the “middle aged” as the most positive, however if the respondent would score less than 3.1 in positiveness on average, according to this model they would be the warmest.</p><h4>Childhood location</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cZInIGWKAhzHGfuCUzlEBA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*9xVdE9biFKDg2LxRHTucdQ.png" /></figure><p>In few words, the model predicts that the warmest and least superficial have spent a considerable amount of time of their childhood in a village. Combination of average warmth score above 4.2 and superficiality below 2.7 predicts a respondent having spent the childhood in a village with a probability of 63%. Pretty high if we take into account that they only account for 29% of the observations. So according to the factors and to the tree, city equals to less warmth and more superficiality, which seems pretty accurate.</p><h4>Gender</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LSM-hC6sAo5WsTk8GqXJbA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*vi0pq9nei2dt4kfu6bvkcw.png" /></figure><p>In a nutshell, more warmth (average score of 3.4%) predicts in favor of female, up to 78% from 59% of female respondents. Again, the factor that discriminate the most and seems more relevant when trying to classify people gender-wise is the warmth factor. Average scores below 2.7 predict being male with a probability of 80% (41% of males responding the survey).</p><h4>Correspondence analysis</h4><p>Once the backbone of the project is analyzed, let´s divert our focus from personality traits to people’s lifestyle.</p><p>In particular the aim is to get some visual insights on how having a healthy lifestyle is determined by some characteristics as gender, age and education.</p><p>To the question, “I live a very healthy lifestyle” people have been asked to strongly agree (5), agree (4), neutral (3), disagree (2) or strongly disagree (1).</p><p>Here are some cross-tabulations for age-gender and education.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*y3BAT6Kg660kq1Ce83yJng.png" /></figure><p>“ma” and “fa” stand for male and female respectively, while 1, 2, 3 corresponds to what we are considering the youngest, the “middle-aged” and the “oldest” as explained in data section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*xBlhhrauSlkx_WnMMu-MBQ.png" /></figure><h4>Gender-age</h4><p>Correspondence analysis comes handy for our purpose. Let´s have a look by gender-age.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*0qgN7vjVcIFHY49NJZyooA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*1ZrFgsHvgLGVboykZr6ojw.png" /></figure><p>In above CA map we can see the dimensions included explain respectively 72.4% and 16.2% of the total inertia, totaling 88.6% for the two-dimensional solution, which seems a very high value, conveying that the rest of inertias associated with the dimensions remaining are not really relevant in explaining variance, hence the 2 dimensions used are the optimal ones for interpreting category differences.</p><p>What can we infer from the map?</p><p>1st dimension is determined by the opposition between those who have an overall healthy lifestyle to those who position themselves on the extremes, that is, either having a very healthy lifestyle or having an overall unhealthy lifestyle.</p><p>The 2nd dimension seem to separate between the sign the deviation from the overall good is.</p><p>So, we can observe female on one side of the first dimensions and the males in the other. This means that young women in general are overall more concerned with carrying a healthy lifestyle than young men, the variability of opinions seems less dispersed and “polarized” than in men. Whereas for males, we can see a clear distinction between youngest ones (15–24yo) and grown-ups (25–30yo) where the grown-ups follow a very healthy lifestyle as opposed to younger people, and especially to young males who seem to follow quite an unhealthy lifestyle on average.</p><h4>Education</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*zEzueoctMxg7dly3CEobUw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*oZ1uWRMfLff7gH23aPNgXw.png" /></figure><p>This time 99.3% of the total inertia is explained by the 2 dimensions of the biplot confirming that is the optimal number of dimensions in explaining variance between the variables.</p><p>In particular we can observe as people with lower levels of education tend to have a less healthy lifestyle on average than those with superior studies.</p><p>In any case, let´s take this with a grain of salt, because since sampling units age from 15 to 30, age is highly correlated with studies.</p><h3>5. Conclusions</h3><p>We have applied factor analysis resulting in interesting personality dimensions that might not be captured from the standard questionnaires that follow the OCEAN approach. From them, we have been able to see partially different personality factors that could be the base for alternative personality psychometric models. After using regression trees on the factors obtained we have also been able to find interesting classifications for different basic categorical characteristics.</p><p>The limitation perhaps is that the explanatory factor analysis that led to obtaining the 5 dimensions explained, should be continued with a confirmatory factor analysis in order to build models with the factors that are sure to be statistically significant. This for sure would be the extension I would envisage for this kind of project.</p><h3>6. References</h3><p><a href="https://www.kaggle.com/nowaxsky/visualization-and-prediction-of-shopping-habits/data">https://www.kaggle.com/nowaxsky/visualization-and-prediction-of-shopping-habits/data</a></p><p><a href="https://cran.r-project.org/web/packages/corrplot/corrplot.pdf">https://cran.r-project.org/web/packages/corrplot/corrplot.pdf</a></p><p><a href="https://www.statisticssolutions.com/factor-analysis-2/">https://www.statisticssolutions.com/factor-analysis-2/</a></p><p><a href="https://github.com/rsangole/PersonalityTraitFactorAnalysis/blob/master/FA_Traits.R">https://github.com/rsangole/PersonalityTraitFactorAnalysis/blob/master/FA_Traits.R</a></p><p><a href="https://www.statmethods.net/advstats/factor.html">https://www.statmethods.net/advstats/factor.html</a></p><p><a href="https://cran.r-project.org/web/packages/nFactors/nFactors.pdf">https://cran.r-project.org/web/packages/nFactors/nFactors.pdf</a></p><p><a href="http://www.milbo.org/rpart-plot/prp.pdf">http://www.milbo.org/rpart-plot/prp.pdf</a></p><p><a href="http://trevorstephens.com/kaggle-titanic-tutorial/r-part-3-decision-trees/">http://trevorstephens.com/kaggle-titanic-tutorial/r-part-3-decision-trees/</a></p><p><a href="https://en.wikipedia.org/wiki/Big_Five_personality_traits">https://en.wikipedia.org/wiki/Big_Five_personality_traits</a></p><p><a href="https://introvertspring.com/what-is-an-introvert/#whatisintrovert">What Is an Introvert? An In-Depth Explanation - Introvert Spring</a></p><h3>7. Appendix</h3><p>List of statement/questions of personality traits</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*Fej4QVLt4Wvx-w3nW8FlYw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*i70sTCywYPqHorKIPaKxsA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3c7e18a377" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-data-science-approach-to-personality-models-d3c7e18a377">A data science approach to personality models</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A closer look into the Spanish railway passenger transportation pricing]]></title>
            <link>https://medium.com/data-science/a-closer-look-into-the-spanish-railway-passenger-transportation-pricing-581c19fe67dc?source=rss-631aacf3691b------2</link>
            <guid isPermaLink="false">https://medium.com/p/581c19fe67dc</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pricing]]></category>
            <category><![CDATA[renfe]]></category>
            <dc:creator><![CDATA[Salva Rocher]]></dc:creator>
            <pubDate>Tue, 06 Aug 2019 21:37:21 GMT</pubDate>
            <atom:updated>2019-10-05T10:17:59.193Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PSoLoTbySHjGOA6kPH6DxQ.jpeg" /></figure><h3>Introduction</h3><p>As someone who lives and works in a Spanish city 400km away from home, I have found that the most convenient way to travel back and forth is to resort to the train. As a frequent user I have grown baffled of the pricing pattern upon buying the tickets, moving sometimes along the same levels, while others out of the most common levels.</p><p>So this doubt spurred me to formulate the following questions:</p><p><strong>“Do train ticket prices really change over the days”?</strong></p><p>And if so,</p><p><strong>“Is there an optimal moment to buy them?”</strong></p><h3>Data</h3><p>In this project, only Renfe’s long distances routes were considered.</p><p>The dataset is originally sourced from a Renfe scraping procedure carried over by thegurus.tech, where prices for the sampled routes departing trains where checked several times on loop each day.</p><p>In particular, the trains whose prices were checked, range about 3 months, from April 12th 2019 to July 7th 2019.</p><p>Data source: <a href="https://thegurus.tech/posts/2019/05/renfe-idea/">https://thegurus.tech/posts/2019/05/renfe-idea/</a></p><h3>Data wrangling</h3><p>After spending some time getting to know the data, the natural next step is to clean the raw and transform it in a way that is prepared for analysis, therefore prepared to cast light into the main questions. Code for it can be found in my <a href="https://github.com/Salvinha-vlc/Renfe-project">Github repo for this project</a>.</p><p>Developing this a little bit more, cleaning and transformation tasks range from creating new columns such as routes, departure date, departure time, identifiers for a particular train departing in a given day and in a given time, or days to departure; to changing format to columns to be able to do calculations and further transformations on them, through reducing categories for some categorical variables such as train type or ticket class, dropping not needed columns or deciding what to do with invalid rows (with null values).</p><h3>Results</h3><p>Code for all the plots and results presented in this section can also be found <a href="https://github.com/Salvinha-vlc/Renfe-project">here</a>.</p><h4>All routes together</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cadR79iVuLc8ZhcQoNcmQQ.png" /></figure><p>There are a few interesting things to comment on this graph. On one hand, it answers the main questions.</p><p>1) We can see that the price indeed changes over the days, not only that, we can also see that there is clearly an aggregated trend where the price goes up as the departure day gets closer.</p><p>2) Once we accept that there are indeed relevant variations, is there an optimal moment to buy the tickets? Well, from the aggregated graph, we can see that the optimal moment to buy is as soon as the tickets are made available for sale, and in any case, between 50 to 60 days.</p><p>On the other hand, it allows us to identify 3 main stages. First, between 42 to 60 days before departure, we can find the lower range of prices, then a period between 42 and 13 days before departure where the variation in prices is very low, around 6%-7% altogether. And lastly, there is the stage where the ticket price increases by the day.</p><p>Let’s break it down by dataset routes.</p><h4>Price ticket evolution in a high demand route</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BLZ6QW-RCbrR7E7284zgJg.png" /></figure><p>Madrid-Barcelona round route has been selected as an example of a high demanded one. It has a similar pattern that the one we observed taking into account all available routes together. We see a drop in prices after ticket has been released for sale, close to 50 days before departure. Again, we can identify the 3 stages, first between 40 to 60 days before departure, lower range of prices, then a period between 40 and 12 days before departure where the variation in prices is very low, around 6%-7%. And last there is the stage where the ticket price increases by the day.</p><h4>Price ticket evolution in a non-high demand route</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*itJdX5GakuauRtR5bNeAug.png" /></figure><p>This time, Renfe sets an initial price, but since demand is not met for that price, it starts to dip until it touches the lowest point between 40–50 days before departure. Then the price evolution is much more volatile, and it continuously rises and falls as the departure day approaches, but always following an increasing trend.</p><p>The intermediate stage that we saw in previous plots, where the price pretty much stays constant for almost a month, it is not quite repeated here.</p><p>The last point I would highlight is that the price variation range is much lower than for a high demand route.</p><h4>Canvas of all dataset routes price evolution</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GG1rYw08eVmlg9KUUlX-vg.png" /></figure><p>We can also break down the price evolution by trip/train characteristics.</p><h4>Day of the week effect</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6COV3oQ9aZuSi6ylBNxUtQ.png" /></figure><p>We can see that the price evolution pattern is very similar regardless of the day of the week when the train departs, however, we see that Friday and Sunday tickets are more expensive on average than for the trains departing in the rest of the days. Monday train tickets are the ones whose price rises the most in the stage where the train is close to departure. Saturdays would be on the opposite side of the spectrum, the price increase is the least pronounced.</p><h4>Departure time effect</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ZuKqxtp0CJVsixZlRz3Ww.png" /></figure><p>We were seeing before in the boxplots that indeed there was a slight difference in the price decomposing by departure time window, where we could see that evening departing trains observed less cheap prices that the ones in the rest of departing time windows, We were also venturing to say that it could be because evening departing trains have in general more demand than the rest, therefore, their prices that are set higher from Renfe, don’t get to dip adjusting for a lower initial demand, as it happens with the rest.</p><p>This graph above backs this theory up. In fact, evening trains lowest price happens to be around a month before departing, resulting in a different optimal purchase moment.</p><h4>Ticket class effect</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ovAXghxMnjSkN_q7aIPRnw.png" /></figure><p>General already commented patterns hold for both types of tickets, so I would highlight that since we know that economy tickets sell out first, that explains why the rise in the prices when the departure date approaches (“last stage”) is so smooth. On the contrary thus, first class are more demanded towards the last days before departure, and that is why the prices rise more strongly as compared to economy class tickets.</p><h4>Checking whether there are any pricing intraday differences</h4><p>Once assessed price interday patterns, let’s put the focus for a moment in intraday price evolution patterns, to see if perhaps they also exist.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pF2kg6BmUKAEA3hQNrxxug.png" /></figure><p>Lines continuously interweave and overlap in what it seems to tell that there are no differences in the intraday pricing per window times by days to departure. In particular, the only point that sticks out is the drop in evening prices towards the lower boundary of the graph (few to none days to departure). Zooming in, it is possible to see that the drop belongs to the same day of departure.</p><p>So, in order to check whether we need to consider that drop relevant, we first need to know the number of price checks runs that are done in general and how many are done per departure window time, in order to exclude potential dependency effects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*D3CMe501Z_fy_lOCW91KJA.png" /></figure><p>We can see that the price checks are done round-the-clock pretty much at equal proportions. Let’s see if this holds when zooming in for the last day, which is when the drop takes place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*hB8Q98gOgKtHMIXtHTASMg.png" /></figure><p>The proportionality does not hold any longer. It means that for price checks done on the same day of departure, those price checks are done in an imbalanced way; in particular, we can see that the number of evening price checks is much lower than the ones done in other window times.</p><p>So basically, since the difference is so high, the few trains whose price has been checked in the evening and in the same day of departure, might perfectly be of a particular type which on average are cheaper. In a nutshell, there are not relevant intraday price differences.</p><h3>Conclusions</h3><p>We have found relevant price movements for the same combo train-day-time-route as the departure date approaches.</p><p>In particular, for high demand routes, the optimal moment to buy is between 50–60 days before departure. For non high demand routes, the optimal moment can be found between 40 to 50 days before departure. When train departures in the evening, the optimal moment is 30 days before departure.</p><p>In any case, if due to any reason the ticket cannot be purchased with as much time in advance, there is no need to worry since the price barely varies up until 12 days before departure (just 6–7% on average).</p><p>But, passed that threshold, tickets gets more expensive day by day as departure day approaches all the way to the day when the train departs.</p><p>Lastly, we have not found any intraday pattern in Renfe pricing system. So grouping by days to departure, one is indifferent to buy at any particular moment of the day.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=581c19fe67dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-closer-look-into-the-spanish-railway-passenger-transportation-pricing-581c19fe67dc">A closer look into the Spanish railway passenger transportation pricing</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>