<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yorgos Askalidis on Medium]]></title>
        <description><![CDATA[Stories by Yorgos Askalidis on Medium]]></description>
        <link>https://medium.com/@yaskalidis?source=rss-2c4731836156------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Z33mGPWUeOqY23LjoD8LBg.jpeg</url>
            <title>Stories by Yorgos Askalidis on Medium</title>
            <link>https://medium.com/@yaskalidis?source=rss-2c4731836156------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:46:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yaskalidis/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Become a Staff Data Scientist]]></title>
            <link>https://medium.com/@yaskalidis/how-to-become-a-staff-data-scientist-44b290137035?source=rss-2c4731836156------2</link>
            <guid isPermaLink="false">https://medium.com/p/44b290137035</guid>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[promotion]]></category>
            <dc:creator><![CDATA[Yorgos Askalidis]]></dc:creator>
            <pubDate>Tue, 02 Apr 2024 15:02:53 GMT</pubDate>
            <atom:updated>2024-04-02T15:02:53.657Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The types of projects and impact you should be looking for when you’re trying to get that coveted promotion</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*atB7F6lQIYh3GMdC" /><figcaption>Photo by <a href="https://unsplash.com/@brookelark?utm_source=medium&amp;utm_medium=referral">Brooke Lark</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>One of the most frequent topics of conversations that I have with other data science professionals is how to increase your impact and get promoted. I believe that, especially in the early to mid years of one’s career, promotion is a healthy goal guiding what types of projects you take on, who you form close relationships with, and what skills you develop.</p><p><strong>It is generally harder to get promoted the more senior you already are. </strong>So much so that some companies have “terminal” and “non-terminal” levels. Junior levels are considered non-terminal, meaning that individuals cannot remain at that level for more than two years (a.k.a. “up-or-out” policies). Most companies consider senior to be a terminal level, meaning that an Individual Contributor (IC) can stay senior for many years.<br>The transition from senior to staff is a major one, generally requiring impact at a high level and to a broad set of teams.</p><p>In this note, I share my thoughts and I welcome everyone to add their own. <br>Here are the three main types of work that are generally considered Staff-level IC work. While this note is focused on Data Science, I believe much of it applies to other disciplines as well.</p><ol><li>Perform a far-reaching deep dive into a high-level area and surface insights that are actioned upon from leaders.</li><li>Be the insights lead (or equivalent) for a large multi-team project, guiding ways of working, surfacing insights and staffing needs, advising leaders about next steps and conveying context back to other insights professionals in the workstream. Setting expectations, keeping track of deliverables and unblocking where needed.</li><li>Uplevel entire organizations by creating standardized methodologies and tools that increase the productivity and rigor of the people around you.</li></ol><p>Let’s dive deeper into each.</p><h4>1. Far-reaching deep-dive into a high-level area surfacing insights that are actioned upon from your organization’s leaders.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hZf8PLUZ9PJk5ZFq" /><figcaption>Photo by <a href="https://unsplash.com/@mjessier?utm_source=medium&amp;utm_medium=referral">Myriam Jessier</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>One of my early managers introduced me to the notion of “sphere of influence” — how many people is your work impacting. When you’re more junior you generally surface insights that inform product, design and engineering decisions for your team but generally not others. <br>As you get more senior, the general expectation is that your insights will influence more senior leaders and hence more people. You can have insights that the overall recommendation strategy of your company’s product doesn’t work well in e.g., a certain country or certain types of users, prompting a different approach across many teams and algorithms. If you notice in your analysis that e.g., Gen-Z has different mental models around how they use your company’s products that could incentivize multiple teams to restructure their design and product offering around that. <br>The analysis you do and the insights you surface can’t be disconnected from the strategic priorities of your company. The best way to ensure that you will spend your analysis time productively is to have 1:1s with many leaders in your organization and assess what is top of their mind. Ask them what major questions they wish to have answered or what they are trying to achieve this half/year. More often than not, patterns will quickly emerge and that can give you the direction you need in order to bring your own creativity and expertise and dive into the data. <br>Once you have insights the difficult part of your work begins! First of all, you should not focus on your insights but rather on your clear, actionable strategy or product plan. You should get feedback and iterate early and often, and get alignment with various XFN partners and leaders. Each person will have valuable feedback. Once aligned, you can strategize with your manager and others on who and how to best communicate.</p><h4>2. Informal leadership of a large multi-team project.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-wBE-YnFcHmkD7Gn" /><figcaption>Photo by <a href="https://unsplash.com/@homajob?utm_source=medium&amp;utm_medium=referral">Scott Graham</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>At senior levels of any discipline, expertise becomes just one factor of your influence. Informal leadership is perhaps the most sought-after skill and type of person there is. Simply put, leaders everywhere look for someone who gets stuff done. <br>The analysis itself is usually not even the hardest part. The larger your sphere of influence, the larger the projects you take on and the more the partners you work with. This includes other data scientists and cross-functional (XFN) partners. Large, multi-team projects require structure, guidance, and accountability and this is the type of work you can look for. <br>This is type of role that can be referred to as Tech lead in the engineering world and beyond. In projects like these, you should aim to provide any necessary project management by representing the work to leaders, conveying context from leaders back to the people you work with and informing the work that is being done as well as the timelines. <br>You will be acting as the representative of insights to leaders, and as the representative of leaders to the insights people you work with. You would be responsible for digesting and combining all the work that people in the workstream do and presenting a holistic view of the progress and the shortcomings. You will be responsible to escalate issues, set timelines, hold people accountable, create weekly meetings, create notes and project tracking documents.</p><p>You’re not expected to have all the answers or to single handedly solve all the issues that come up, but you are expected to surface proactively and timely all the needs and finding the right partners to help unblock the project. Your impact and success will ultimately be judged by what has shipped and the impact that had on key company metrics.</p><h4>3. Create standardized tools, methodologies and platforms that upscale everyone around you.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hwyzGwGY1mLK-yVK" /><figcaption>Photo by <a href="https://unsplash.com/@imattsmart?utm_source=medium&amp;utm_medium=referral">iMattSmart</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Each of the above two types of work appeals to slightly different archetypes of senior data scientists. One might appeal more to people who enjoy working with product and strategy, the other to people who enjoy organization and informal leadership. This third category of work is more geared towards people whose archetype is more technical and oriented in creating tools that empower everyone around them. <br>Examples here include creating tools that don’t serve any one specific project but rather can be used by a variety of people to make their workflows faster and more efficient. That way your impact is multiplied. Examples can be python packages about something that data scientists do often such as data exploration, visualization, causal estimation, and more. <br>But it doesn’t have to be code-products. If you’re an expert in causality then you can write a manual for best practices in estimating causality. If you know a lot about a topic that people around you encounter often (e.g., forecasting, impact estimation) then you can share your knowledge and tips. Having clear, easy to use manuals around topics such as reg exp, structs and arrays in SQL, datetime conversions can help dozens of data scientists you work with save time from their workflows.<br>In organizations that are large and product areas that are complex there is a lot of value in someone creating high-level holistic explainers. For example, if you work for a company with heavy personalization, chances are there are dozens of machine learning algorithms powering all the that personalization — it would be helpful for someone to outline them all and explain how they interact.</p><h4>Some additional thoughts</h4><ul><li>Be a thought leader: No matter seniority or tenure at a company becoming a thought leader for your area can be useful for your organization and can help you showcase your value to your leaders.</li><li>Uphold high standards and uplevel everyone around you: again, no matter the seniority and tenure, attention to detail and high standards can help you do great work and also uplevel the work of the people around you.</li><li>Mentor and guide other insights professionals: try and uplevel everyone around by mentoring or other organizational involvement such as putting together offsites/onsites, reading clubs, speaker events and more.</li></ul><h4>How about the transition from Junior to Senior?</h4><p>This note was specifically written for the transition from senior to staff, but the high level learnings transfer to all levels. As you get more and more senior, you are expected to have higher autonomy and larger influence. <br>For example, when someone is an intern their focus is often on execution alone. This means that the intern summer project is fleshed out by the time they start so they can waste no time in jumping into hands on work. Interns generally have a detailed project plan with specific questions to answer and the resources to find those answers, as well as clearly stated deliverables and timelines.<br>When you’re a junior, the work you do is generally expected to be guided by your manager and your partners, and your focus is mostly still on execution and delivery. As you gain experience, it is expected you start formulating some of your own questions and forming your own relationships with your partners in order to have a good pulse on the priorities and the issues your team is facing. <br>Slowly, you should start coming up with your own ideas that you present to your manager for feedback, then execute fairly independently and finally drive the communication and decision making.</p><h4>And finally, a note on the process of evaluation and promotion</h4><p>None of the above can really happen if the ICs and every employee don’t have confidence that their work will be recognized no matter personnel or org changes that might happen between the start and the end of the work. Companies should have rigid and transparent evaluation frameworks so that folks feel that they can demonstrate their impact in ways that are objective and get rewarded when they do great work. The process should also be robust enough to make sure that individuals with different managers are still evaluated in the same way and according to the same standards. <br>If things like management changes or re-orgs result to great work going unrecognized then that can lead to loss of motivation for great work.</p><p>Thank you for reading and please let me know your thoughts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44b290137035" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Deep Dive into 2–0 Down Comebacks in Tennis.]]></title>
            <link>https://medium.com/@yaskalidis/a-deep-dive-into-2-0-down-comebacks-in-tennis-97de01e7d3ec?source=rss-2c4731836156------2</link>
            <guid isPermaLink="false">https://medium.com/p/97de01e7d3ec</guid>
            <category><![CDATA[tennis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[psychology]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[sports-analytics]]></category>
            <dc:creator><![CDATA[Yorgos Askalidis]]></dc:creator>
            <pubDate>Mon, 05 Feb 2024 16:31:22 GMT</pubDate>
            <atom:updated>2024-02-05T16:31:22.391Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Deep Dive into comebacks from 2–0 Down in Tennis.</h3><h4>Who wins the fifth set?</h4><p>In the <a href="https://www.youtube.com/watch?v=b90INDbXX7Y">final of the Australian Open 2024</a>, <a href="https://www.atptour.com/en/players/daniil-medvedev/mm58/overview">Daniil Madveded</a> was leading <a href="https://www.atptour.com/en/players/jannik-sinner/s0ag/overview">Jannik Sinner</a> 2 sets to none after 1.5 hours of play. Sinner had never won a Grand Slam before, but he was able to win the next two sets and get to level terms!</p><p>As the fifth and final set was about to begin, I found myself wondering <strong>who historically has the advantage here?</strong> So I gathered a dataset of men’s Grand Slams matches to find out!</p><h4>Summary:</h4><ul><li>The player who comes back from two sets down has historically won 53.35% of the matches.</li><li>This differs amongst Grand Slams, and Wimbledon is the only tournament where the odds are an even 50/50.</li><li>This also differs by <em>round</em> of the tournament. Players who have come back in the <em>final</em>, won the match 60% of the time!</li><li>Perhaps due to the lack of spectator attendance, the public health emergency of 2020 caused a one-time shift of moving away from <em>exactly-4-set matches in favor of more exactly-3-set matches.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HNubnU9_LK9PtR4f" /><figcaption>Photo by <a href="https://unsplash.com/@carlesrgm?utm_source=medium&amp;utm_medium=referral">Carles Rabada</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>There are reasons to believe it can be either of them. </strong>Using the specific players above as an example, Medvedev perhaps started the game with the right tactics, self confidence and generally “played better”. But since Sinner just managed to win two sets in a row, he’s likely full of self confidence and has a lot of momentum. Perhaps Medvedev thought he had the game wrapped up and is now full of self doubts, perhaps he’s panicking and is not able to play to the best of his best ability. But Sinner just played an hour and half of tennis under immense pressure, knowing that he’s close to losing the match — he’s likely exhausted right now.</p><p>Before I turn to data, I turned to my friends and they overwhelmingly voted for Player 2, i.e., that the player that was down 2–0 and came back to make it 2–2 was more likely to win the 5th set and the match.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y-mDxO3pbv9OXfqOqMrv0w.jpeg" /><figcaption>My friends overwhelmingly voted that the comeback player is more likely to win the whole match.</figcaption></figure><h4>A super quick Grand Slam primer</h4><p>There are many prestigious tennis tournaments throughout the year but four dominate the calendar: the <a href="https://ausopen.com/"><strong>Australian Open</strong></a> (generally in January), <a href="https://www.rolandgarros.com/en-us/"><strong>Roland Garros</strong></a> (aka the French Open, generally in May/June), <a href="https://www.wimbledon.com/index.html"><strong>Wimbledon</strong></a> (generally happens in July in Wimbledon, London) and the <a href="https://www.usopen.org/index.html"><strong>US Open</strong></a> (generally happens in September in NYC).</p><p>These are referred to as the four Grand Slams and in each there are 128 players entering and (generally) play single elimination matches until there is a single winner (so that means there are 127 matches in each Grand Slam).</p><h4>Dataset</h4><p>I collected match score data from <a href="https://www.livesport.com/tennis/atp-singles/australian-open-1990/results/">livesport.com</a> and created a dataset that includes all match scores from the <strong>four Grand Slams since 1990 up to January 2024 </strong>(so it includes up to Australian Open 2024) as long as the players and the scores of each set. The dataset includes only mens matches because all men’s Grand Slam games are best out of five.</p><p>Let’s explore the dataset.</p><h3>Some Exploratory Analysis First</h3><p>Before I dive in to the comeback statistics, I wanted to get a high-level view of how many sets even happen in a match.</p><p><em>What is the typical number of sets in a tennis match?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hBWN3KEMrH7lEkRd66dGqw.png" /></figure><p><strong>About 19% of all matches in Grand Slams since 1990 had 5 sets.</strong> 48% had exactly 3 sets and 31% had exactly 4 sets. The remaining 2% of the matches did not reach 3 sets due to reasons such as injuries and forfeits.</p><p>The distribution is fairly stable across the four Grand Slams, with the US Open having some minor differences: Only 46.8% of the matches in the US Open had exactly 3 sets, compared to ~48.5% for the other three. This led to more matches in the US Open having 4 sets: 32.6% vs ~30.5% for the other two. Small but noticeably differences, that might be due to the unique characteristics of the surface, likelihood of getting injured and forfeit, or other factors.</p><p>Next, I wanted to see if this distribution <em>changed over time </em>— perhaps due to complex evolutions of in the techniques of the players, the larger availability of data or other factors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xIwS92Q_jzhG54fFPWTw8w.png" /></figure><p><strong>It didn’t. </strong>The distribution of number of sets in a match hasn’t shifted much over the past 30 years.</p><p>Although there <em>is</em> a noticeable jump in 2020’s number of 3-set matches and a drop in 4-set matches. This only is observed in the Roland Garros and US Open, and <em>not</em> in the Australian Open which happened in January of 2020, before the public health emergency. Wimbledon was cancelled in 2020.</p><p>This seems to strongly hint towards some effect of the pandemic. Roland Garros and the US Open happened in the midst of the pandemic and they did observed this increase in 3-set matches while the Australian Open did not. This seems to paint a picture of a player that is about to lose the match 3 sets to 1 (but not 3 sets to 2). The player can fight to indeed lose the match 3–1 or can accept his fate sligthly sooner and just lose 3–0. During Roland Garros and the US Open of 2020, players in the situation were more likely to opt for the latter. <strong>This likely was influenced by the lack of large crowds as well </strong>(full spectator attendance was not allowed due to the public health emergency)— if there’s no crowd to cheer you on, then you might be less likely to fight for the symbolic set win.</p><p><em>All right,</em> now lets explore our original question of who wins after a player came back from 2–0 to force a fifth set.</p><h4>How Often Do Players Rise from the Ashes After 2–0 Down?</h4><p>Out of the 17,269 games in my dataset, 1,216 (7%) had one player lead 2–0 and the other player coming back to force a fifth set.</p><p><em>How many times did the comeback player win the match? Does it differ by Grand Slam?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IUhzQsIxjmv0CKHxIhZ7tA.png" /></figure><p>Out of those 1,216 games, <strong>the comeback player won the match in 55.35% of the occasions.</strong></p><p>The odds are most in favor of the comeback player in the Australian Open (57.1%) and Roland Garros (57.4%) with the US open (56.1%) following closely behind. Interestingly, <strong>there is no advantage in Wimbledon! This might be due to the grass surface, particular prestige of the tournament or other elements.</strong></p><p><em>Looking back over the 34 years of data, has this 55.35% changed much?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vcfy-XWE9gzVWgN3IPbzNw.png" /><figcaption>The likelihood of completing the comeback didn’t change much over the past three decades.</figcaption></figure><p>It doesn’t! The 55.35% likelihood has remained relatively steady over the past 34 years.</p><p>Of course, all the numbers above are aggregates for the whole tournaments. Let’s no examine whether they change depending how far along you are in the tournament. There are 128 players in the first round, but by the time of the Quarter Final there are less than 8 players remaining. This means two things: the stakes are higher and (generally) the quality of the players is higher.</p><h3>Per Round</h3><p><em>Does the 55.35% winning likelihood change depending the round of the tournament?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fY3nIi7V7G5K9rjYXbLPwQ.png" /><figcaption>The Semis is the only round when you are more likely to lose the match after you force a comeback.</figcaption></figure><p>Notice the<strong> large difference between the Semi-finals and the Final! </strong>In the semis, if a player forces a fifth set after trailing 2–0, he is <strong>likely to lose the match! </strong>In the final, other mental forces are at play, and the player has a<strong> whopping 60% chance of winning the match! </strong>These are the lowest and highest probabilities observed.</p><p>Now, if 60% (or for that matter, 55.35%) does not seem like a huge of an advantage think about the context: First, this is the <em>final</em> of a Grand Slam, so both players are elite tennis players and are looking to put their names down in history. Furthermore, the match is tied 2–2 after likely 3+ hours of playing. These<strong> two players are equally matched.</strong> So a 60% chance (20pp swing) of winning the match is <em>huge</em>, and likely due in large part to mental factors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tD9RoO7wHr3uii4a" /><figcaption>Photo by <a href="https://unsplash.com/@brandsandpeople?utm_source=medium&amp;utm_medium=referral">Brands&amp;People</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>There’s a lot more fun analysis to do here.</p><p>First, I want to get statistics for <strong>individual players.</strong> Which players have faced a two set deficit the most throughout their career (players have different durations of careers so I would need to normalize). Which players were the most likely to turn things around and win the match?</p><p>Factors like the player’s age and duration of each set can be interesting for some hypothesis testing. Even more exotic factors such as the weather and time of play could be incorporated in this analysis.</p><p><strong><em>What questions would you want to see answered?</em></strong></p><p>Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97de01e7d3ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fun Things you Can Learn About Yourself and from your Messages]]></title>
            <link>https://medium.com/@yaskalidis/fun-things-you-can-learn-about-yourself-and-from-your-messages-5101631a8e20?source=rss-2c4731836156------2</link>
            <guid isPermaLink="false">https://medium.com/p/5101631a8e20</guid>
            <category><![CDATA[messaging]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[analysis]]></category>
            <category><![CDATA[friendship]]></category>
            <dc:creator><![CDATA[Yorgos Askalidis]]></dc:creator>
            <pubDate>Mon, 22 Jan 2024 19:21:45 GMT</pubDate>
            <atom:updated>2024-01-22T19:21:45.073Z</atom:updated>
            <content:encoded><![CDATA[<p>A few years ago <a href="https://medium.com/@yaskalidis/heres-how-you-can-access-your-entire-imessage-history-on-your-mac-f8878276c6e9">I wrote detailed instructions on how you can extract all your iMessage (plus some SMS) history from your Mac computer</a>. <strong>Unfortunately this indeed works only for iMessage on Macs.</strong></p><p>Here’s the <a href="https://github.com/yortos/imessage-analysis/blob/master/imessages-data-extract-and-prep.ipynb">refreshed code to extract the data and create a clean dataset</a> for analysis.</p><p>And here’s<a href="https://github.com/yortos/imessage-analysis/blob/master/imessages-analysis.ipynb"> the code for all the analysis discussed below</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_0DMGvjjgdTq0Esa" /><figcaption>Photo by <a href="https://unsplash.com/@adamus?utm_source=medium&amp;utm_medium=referral">Chris Adamus</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this post, I showcase some analysis you can do to get fun and informative insights about your messaging behavior and even your relationship with your friends.</p><h4>Some important notes before we get started.</h4><ol><li><strong>Coding knowledge is required.</strong></li></ol><p>Please note that this is <strong>mainly intended towards a somewhat technical audience, </strong>similar to the original post. Knowledge of Python (or equivalent) is a minimal requirement.</p><p>Here’s the link to the cleaned up jupyter notebook I used for t<strong>his analysis.</strong></p><p><strong>2. The dataset does not necessarily include <em>all</em> your messages</strong></p><p>I have more details at the end of this post, but the way that Apple has structured their database in your Mac, somehow doesn’t produce some messages you have sent.</p><h4>1. Number of Messages per Day</h4><p>The first analysis you can do is to see the number of messages you have sent and received throughout the years. This is what it might look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G8cH-d41LgEqTI9vpxOHEQ.png" /><figcaption>The 4-week rolling average helps remove some noise and showcase the trends better.</figcaption></figure><p>Of course, you can get a higher-level view as well by just looking at number of messages exchanged per year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tj_EWBpEdvvIylA5JzEFVg.png" /></figure><p>Finally, you can also plot the number of messages exchanged for specific people. I’m omitting that here.</p><h4>2. Most Texted People</h4><p>Another very natural analysis you can do is to see who do you text the most!</p><p>You can also attach the % of the messages that were sent vs received. Due to some limitations of this database when it comes to Sent messages, I’m omitting that part for now.</p><p>Here’s what might look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*D5tgsCUzO9KM6cVstU7BTA.png" /></figure><h4>3. Texting Directly versus in Group Chats</h4><p>You can add a twist to the above analysis due to the fact that you can text <strong>someone 1-on-1 or in a group chat</strong>.</p><p>You can calculate the <strong>% of group chat messages over all messages with each person</strong>. The higher this number is, the more you communicate with this person via <em>group chats</em>. If this number is above 50%, that means that you <strong>mainly</strong> communicate with this person via group chats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*l6c0GEyY-WO9xEFwSVPvoQ.png" /></figure><p>For me, I noticed that there is some correlation (although there are exceptions) between texting someone less and the likelihood of that message being in a group chat — I think that makes sense generally.</p><p>But there are exceptions! For example, I message my sister a lot but a high percent of those messages (40%) is in group chats — that’s because we have a family group chat which we use a lot.</p><p>You can go deeper with this and break it into received and sent: are there people that <em>you</em> tend to message directly but <em>they</em> tend to message you in a group chat? That’s one step too deep for me (and honestly, I don’t think it will produce any profound insight) so I’m omitting the analysis here.</p><h4>4. Average Number of Texts Per Day</h4><p>When looking at the absolute number of messages exchanged with a person, what we didn’t take into account yet is the time knowing that person. Here, I normalize for that by using as a starting point the date of the first message exchanged. Then I divide the total number of messages by the total number of days since that first message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*JQedY_Bs6vy2EATnLF5M3A.png" /></figure><p>Note here that the first message date is only a <em>directional</em> approximation of when you met someone. Besides the obvious fact that <strong>you’ve known some people long before you bought your first iPhone</strong>, the date you purchased your Mac computer or enabled iMessage cloud services will affect this dataset.</p><h4>5. Number of Texted Days per Person</h4><p>When it comes to people you text a lot with, yet another variation we can do is to count the number of days you’ve exchanged messages, rather than the total number of texts sent. Perhaps there are people you went on a trip with and texted a lot during the trip for logistics but then lost touch. This method of counting unique days texted can be another dimension to the relationship because it can surface up people that you to talk to every day, even if within each day you don’t text a lot. Based on what we discussed above, we can also normalize to the number of days that you have known the person.</p><p>So the final metric is: number of unique days with at least one message exchanged divided by the total number of days you have “known” the person. For example, if the value is 60% this means that you exchanged at least one message 60% of the days you have “known” the person.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*Sgai740asjw8RpVBNl7m-g.png" /></figure><p>Note that in the above analysis we counted days that only one of you messaged the other.</p><p>So you can go one step even further here and only count the days that both of you sent at least one message to each other. One data point that is used in some industries to approximate “meaningful conversations” is when two people exchange at least<strong> </strong><em>three </em>messages: you send one, the other person responds, and then you respond back. I’m omitting these analysis here.</p><h4>6. Seasonality!</h4><p>Do you text more on Wednesdays or on Sundays?</p><p>In January or in July?</p><p>The beginning of the month or the end?</p><p>As above, you can also do the same analysis for <em>all</em> messages or break it down by sent and received (perhaps the trends are different for when you receive or send texts). This is what I did here.</p><p>You can also do this analysis for messages with a specific person only — perhaps you message some people more on the weekends and some more during the week. I omit that here.</p><p><em>Per day of the Week</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3qevjQYvZWd3CZliEUgWew.png" /><figcaption>Decreased activity during the weekends.</figcaption></figure><p><em>Per Month</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UyUIS8fvBFdCw6QD5MWcMQ.png" /></figure><p>For me, it seems like December, summer and late Fall (October, November) are the months that see an elevated number of messages which I think generally makes sense since there are more things going on during those months (summer activities or travel, Haloween, Thanksgiving and overall holiday season in October — December).</p><p><em>Per Day of the Month</em></p><p>Instead of doing the plot for each day of the month (which was noisy) I instead used a grouping to make the plot cleaner. Either way, there’s no real trend here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UsgJ5AWkd8BVnAxSXdh96w.png" /></figure><p>Happy Analysis!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y4-CR7POxIeKPju8" /><figcaption>Photo by <a href="https://unsplash.com/@thomholmes?utm_source=medium&amp;utm_medium=referral">Thom Holmes</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>But wait — what about the texts?</h3><p>At this stage, you might be wondering where are all the analysis ideas around the actual text of the messages? Sentiment analysis, length, key words and more.</p><p>The reason I omitted these types of analysis is because when I read the database of my messages on my MacBook, f<strong>or some reason the actual <em>text</em> of the messages I have sent always appear NULL. </strong>Furthermore a lot of sent messages appear to be missing. Very odd.</p><p>This might have something to do with the way that Apple structures and stores that database on Macs or perhaps I’m simply missing something. I’ve duplicated this issue with <a href="https://github.com/niftycode/imessage_reader">another code </a>I found on Github from user <a href="https://github.com/niftycode">niftycode</a>.</p><p>If you know how to solve that please reach out!</p><p>If you had access to the actual text in the messages some analyses you can do can be:</p><ul><li>Average message length — are there people you tend to message shorter messages vs longer? You can also do this over time — perhaps as you first to get to know a person you tend to exchange longer messages, but then shorter.</li><li>Sentiment analysis over time</li><li>Word bubble — get the bubble with the most frequenly used words in your messages. You can break that down by sent and received, and also look only for a specific person.</li></ul><h3>Appendix: Some Final Details about the dataset</h3><ol><li>This includes reactions and attachments and counts them as individual messages. If you sent a message and the other person reacted to it (e.g., “HAHA”, “!!”), that will count as two messages.</li></ol><p>2. You can detect which messages are reactions because they get logged in the database in a standardized way. If you sent a message that said “Hello” then the reactions will be logged as follows:</p><ul><li>Loved “Hello”</li><li>Liked “Hello”</li><li>Disliked “Hello”</li><li>Laughed at “Hello”</li><li>Emphasized “Hello”</li><li>Questioned “Hello”</li></ul><p>Of course, these are in English. <strong>If you text in a different language then the reactions will also appear in that language.</strong></p><p>3. If you sent a photo that counts as a message. You won’t see the photo in this dataset, but the way that Apple has structured their database, it seems that there is a way to map the attachment log back to the actual file that is stored in your Mac.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5101631a8e20" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A few things I’ve learned about being a good data scientist.]]></title>
            <link>https://medium.com/@yaskalidis/a-few-things-ive-learned-about-being-a-good-data-scientist-2e81432f8766?source=rss-2c4731836156------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e81432f8766</guid>
            <category><![CDATA[professional-development]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[advice]]></category>
            <dc:creator><![CDATA[Yorgos Askalidis]]></dc:creator>
            <pubDate>Thu, 17 Nov 2022 14:00:09 GMT</pubDate>
            <atom:updated>2022-11-19T00:36:38.854Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Sharing in hope that it can help some people!</em></p><h4><strong>A few things</strong></h4><ol><li><strong>You always need to have a non-null answer</strong>, even to questions that seem unanswerable (“how many new users will we get if we enter this new market?”). Seek to find the best approximation and be scrappy with data from all sources, or articulate what additional resources you need to get an answer.</li><li>It’s OK to not be able to do something (time or resource constraints), but <strong>only if you ask for help early. </strong>Your work is important to your company, so asking for help early is the responsible thing. Similarly, it might be OK to miss a target or a goal, but you always need to be able to explain why, and how you’re going to prevent this from happening again in the future.</li><li><strong>Make sure you convince your self about your analysis first</strong>, because only <em>then</em> you will be able to convince your peers and your leads. If you’re convinced about the value and correctness of your work, then you will feel empowered to ask for help if needed (related to Point 2. above) but also confidently speak about it to your partners and leadership.</li><li><strong>You are a <em>professional</em> writer and story-teller. </strong>As a data scientist, you will drive impact largely by convincing <em>other</em> people to build or do something. This means that you <a href="https://collabfund.com/blog/a-few-rules/">need to tell a compelling story</a> that catches people’s imagination and aligns with their priorities.</li><li><strong>Bring people along. </strong>On top of telling a compelling story, you should also seek feedback early and wide before landing your work in an email, post or a presentation. While you need to put your own time to make sure that your analysis is correct and complete, you should also seek feedback from multiple people early to give them a chance to express their perspective or raise any concerns. This has the additional benefit that, when your work lands, you will already have some people supporting it.</li><li><strong>Don’t hide behind your title but also don’t let your title trap you. </strong>Fundamentally, your mandate is to help your team, org and company to succeed. Sometimes you might need to do work that touches things beyond data science, e.g., you might need to create a pipeline or a dashboard (data engineering) or help structure a project to completion (project management). Discuss with your team and manager, and help anyway you can.</li><li><strong>You should be working on answering your company’s biggest questions. </strong>Have a pulse on what is the most important questions for your team, org and company overall. Then develop a clear path in your mind about how your work is helping answer those questions or achieve the objectives. If your work is not related to those questions, clearly understand why and shift if necessary. The story of the <a href="https://fromthegreennotebook.com/2017/11/04/the-janitor-who-help-put-a-man-on-the-moon/">janitor that helped put a man on the moon</a> comes to mind.</li></ol><h4><strong>A few more tactical things</strong></h4><ol><li>When you’re looking for a decision from your leaders or partners, use a traffic-light framework and present at least 2 options: ‘green’ being your suggested option, ‘orange’ and/or ‘red’ other options that are possible but you perhaps you don’t recommend. Articulate clearly the <strong>pros and cons, and ask them if they agree with your proposed solution.</strong> People are much more likely to offer thoughts on proposed solutions than come up with their own solution on the spot.</li><li><strong>Be biased for action. </strong>If you’re blocked on something and you’re waiting for alignment, the fastest way to get it is by proposing a solution and asking your partners if they have any objections. Usually they will have some suggestions which you can quickly follow up on, and you’ll be unblocked.</li><li><strong>Define terms and clearly articulate your methodology</strong> before you start explaining your insights and work. If someone has questions before they even start reading your insights, then the potential impact of your work might be diminished.</li><li>When telling a story, <strong>clearly mark what is a known fact (and provide a reference for it) and what is your hypothesis. </strong>Furthermore, make your logic of going from insight to suggested action clear. Operating at a high level of decision making requires rigor and extremely clear paths to conclusions, in order to inspire trust.</li><li>In general, <strong>don’t use any acronyms you haven’t defined in your post first</strong>. There are likely new-joiners in your team every week.</li><li>When posting or presenting your work, <strong>anticipate some of the questions that your audience will ask you</strong>, and either address those things in the presentation or have some answers in the Appendix. You’re never going to predict everything, but you should have a good sense of what is important to your audience and how your work factors in with their work and priorities.</li><li>Divide your analysis work into two parts: <strong>high level exploration</strong> (~20%) and <strong>hypothesis testing (~80%). </strong>In the exploration phase, just seek to get a sense of the data and the environment your analysis will operate in (e.g., any biases or shortcoming your dataset or methodology might have). Having a good such understanding is important to set you up for hypothesis testing. In hypothesis testing, articulate ideas in your mind and test them with charts and analysis to either get support or reject. Without hypotheses, you might be in danger of hunting a narrative and finding only the data that support it.</li><li><strong>Clearly addressing the shortcomings or biases of your analysis</strong> will, paradoxically enough, make your work <strong>more</strong> trustworthy.</li><li><strong>If you don’t publish and communicate your work</strong>, sadly, it’s like you haven’t really done it.</li><li><strong>There’s time for humbleness and there’s time to make your promotion case</strong>. If you don’t believe that you’re doing impactful work of the highest quality, then your manager might have a harder time believing and convincing others that you do.</li></ol><h4><strong>A couple more things on writing because it’s really important</strong></h4><ol><li>Write all your work as if they will be read by your CEO. Be Succinct, have a clear premise, clear insights and clear suggested actions.</li><li>Write all your communications and work <strong>assuming it will be leaked to external audiences </strong>with no context. Write facts, label assumptions and hypothesis clearly and never sensationalize.</li></ol><h4><strong>A couple of general, parting thoughts</strong></h4><ul><li><strong>All your colleagues are humans first. </strong>You’re all trying your best. Each person you meet and work with carries a galaxy of life experiences. Be curious and seek to learn <em>about</em> and <em>from</em> them.</li><li><strong>The company’s mission will never end, but your life will. </strong>Amazon is famous for their “It’s always Day 1” culture. Meta has a similar one, “This journey is 1% finished”. At Spotify, Daniel Ek used to say “We’re still at the first inning”. This is a necessary ethos for companies to have in order to ensure that they will not get complacent and driven out of their market. Separate your company’s timeline (which is infinite) from your own timeline (which is finite). When work gets too stressful, try and ground yourself.</li></ul><p><em>No advice is created out of thin air, only through great mentorship and partnership. Thank you to all my mentors and partners that taught me everything I know.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*mOAAIkaqBxVxSKvvVUoyWA.png" /><figcaption>Unfortunately I don’t know who the credits go to!</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e81432f8766" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Top Books of the Decade]]></title>
            <link>https://medium.com/@yaskalidis/the-top-books-of-the-decade-30a7f70caee7?source=rss-2c4731836156------2</link>
            <guid isPermaLink="false">https://medium.com/p/30a7f70caee7</guid>
            <category><![CDATA[seattle]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[books]]></category>
            <category><![CDATA[reading]]></category>
            <category><![CDATA[top-10]]></category>
            <dc:creator><![CDATA[Yorgos Askalidis]]></dc:creator>
            <pubDate>Tue, 17 Dec 2019 16:08:04 GMT</pubDate>
            <atom:updated>2019-12-17T16:08:04.028Z</atom:updated>
            <content:encoded><![CDATA[<h4>As determined by the Seattle Public Library Checkout Records</h4><p>The roaring 2010s are coming to a close and one of the trends of the decade was, perhaps against all odds, the return of the physical book. People just love holding an actual book when they’re reading. And what better place to get a book than from your local library and for free!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eTyHonsgpHPVIYY3eJ5l3w.jpeg" /></figure><p>Last year, I used the open data source of all checkouts from the <a href="https://data.seattle.gov/Community/Checkouts-by-Title/tmmm-ytt6">Seattle Public Library</a> to calculate the<a href="https://medium.com/@yaskalidis/the-most-checked-out-books-from-the-public-library-63f147c04df3"> most popular books of 2018</a> and every year past down to 2006. This year, I did the same analysis for 2019.</p><p>Since this year marks the end of the decade, and the Seattle Public library has data for the entire ten years, in this post I’ll present the results of the analysis for the most checkout books of the entire 2010s decade.</p><p><strong>Quick notes about the analysis</strong></p><ul><li><em>All the checkouts are between between January 2010 and November 2019.</em></li><li><em>The analysis was done on the Title level, so if a book has multiple versions (hardcover, paperback, etc), they all count towards a single metric.</em></li><li><em>The list is constrained only to adult and young adult books. I excluded juvenile books (e.g., Dr. Seuss).</em></li><li><em>The publication year is for the original edition of the book, but the number of checkouts is for all editions.</em></li></ul><h3><strong>The Top-20 most checked-out books of the 2010s</strong></h3><ol><li><a href="https://www.amazon.com/Educated-A-Memoir/dp/B075F68BFV/ref=sr_1_1?crid=2CYO6UAQHNIV7&amp;keywords=educated&amp;qid=1575506457&amp;sprefix=educate%2Caps%2C163&amp;sr=8-1">Educated : a memoir</a>, by Tara Westover. <strong>2018</strong></li><li><a href="https://www.amazon.com/Whered-You-Go-Bernadette-Novel/dp/0316415863/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575506507&amp;sr=8-1">Where’d you go, Bernadette : a novel</a>, by Maria Semple. <strong>2012</strong></li><li><a href="https://www.amazon.com/Hunger-Games-Book-1/dp/0439023521/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=&amp;sr=">The hunger games</a>, by Suzanne Collins. <strong>2008</strong></li><li><a href="https://www.amazon.com/Mockingjay-Hunger-Games-Suzanne-Collins/dp/0545663261/ref=pd_bxgy_14_img_3/132-1997125-6271347?_encoding=UTF8&amp;pd_rd_i=0545663261&amp;pd_rd_r=fdc56c48-dbb1-457d-ab3e-e2083a7b6ba1&amp;pd_rd_w=ZDWHr&amp;pd_rd_wg=n9QrG&amp;pf_rd_p=09627863-9889-4290-b90a-5e9f86682449&amp;pf_rd_r=Z906WJE3M2C82KV0G6S4&amp;psc=1&amp;refRID=Z906WJE3M2C82KV0G6S4">Mockingjay</a>, by Suzanne Collins. <strong>2010</strong></li><li><a href="https://www.amazon.com/Becoming-Michelle-Obama/dp/1524763136/ref=tmm_hrd_swatch_0?_encoding=UTF8&amp;qid=1575506693&amp;sr=1-1">Becoming</a>, by Michelle Obama. <strong>2018</strong></li><li><a href="https://www.amazon.com/Catching-Fire-Hunger-Games-2/dp/0545586178/ref=pd_bxgy_14_img_2/132-1997125-6271347?_encoding=UTF8&amp;pd_rd_i=0545586178&amp;pd_rd_r=fdc56c48-dbb1-457d-ab3e-e2083a7b6ba1&amp;pd_rd_w=ZDWHr&amp;pd_rd_wg=n9QrG&amp;pf_rd_p=09627863-9889-4290-b90a-5e9f86682449&amp;pf_rd_r=Z906WJE3M2C82KV0G6S4&amp;psc=1&amp;refRID=Z906WJE3M2C82KV0G6S4">Catching Fire</a>, by Suzanne Collins. <strong>2009</strong></li><li><a href="https://www.amazon.com/Gone-Girl-Gillian-Flynn/dp/0307588378/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575506719&amp;sr=1-2">Gone Girl: a novel</a>, by Gillian Flynn. <strong>2012</strong></li><li><a href="https://www.amazon.com/All-Light-We-Cannot-See/dp/1501173219/ref=sr_1_1?keywords=All+the+light+we+cannot+see+%3A+a+novel&amp;qid=1575506738&amp;s=books&amp;sr=1-1">All the light we cannot see : a novel</a>, by Anthony Doerr.<strong> 2014</strong></li><li><a href="https://www.amazon.com/Quiet-Power-Introverts-World-Talking/dp/0307352153/ref=sr_1_1?keywords=Quiet+%3A+the+power+of+introverts+in+a+world+that+can%27t+stop+talking&amp;qid=1575506755&amp;s=books&amp;sr=1-1">Quiet : the power of introverts in a world that can’t stop talking</a>, by Susan Cain. <strong>2012</strong></li><li><a href="https://www.amazon.com/Girl-Train-Paula-Hawkins/dp/1594634025/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575506772&amp;sr=1-1">The girl on the train</a>, by Paula Hawkins. <strong>2015</strong></li><li><a href="https://www.amazon.com/Hate-U-Give-Angie-Thomas/dp/1406372153/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575506790&amp;sr=1-2">The hate u give</a>, by Angie Thomas. <strong>2017</strong></li><li><a href="https://www.amazon.com/Goldfinch-Novel-Pulitzer-Prize-Fiction/dp/0316055441/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575506808&amp;sr=1-1">The goldfinch</a>, by Donna Tartt. <strong>2013</strong></li><li><a href="https://www.amazon.com/Boys-Boat-Americans-Berlin-Olympics/dp/0143125478/ref=sr_1_1?keywords=he+boys+in+the+boat+%3A+nine+Americans+and+their+epic+quest+for+gold+at+the+1936+Berlin+Olympics&amp;qid=1575506828&amp;s=books&amp;sr=1-1">The boys in the boat : nine Americans and their epic quest for gold at the 1936 Berlin Olympics</a>, by Daniel James Brown. <strong>2013</strong></li><li><a href="https://www.amazon.com/Wild-Found-Pacific-Crest-Trail/dp/0307476073/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575506857&amp;sr=1-1">Wild : from lost to found on the Pacific Crest Trail</a>, by Cheryl Strayed.<strong>2012</strong></li><li><a href="https://www.amazon.com/Fault-Our-Stars-John-Green/dp/014242417X/ref=sr_1_1?keywords=The+fault+in+our+stars&amp;qid=1575506890&amp;s=books&amp;sr=1-1">The fault in our stars</a>, by John Green.<strong> 2011</strong></li><li><a href="https://www.amazon.com/Wrinkle-Time-Quintet/dp/0312367546/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575506903&amp;sr=1-1">A wrinkle in time,</a> by Madeleine L’Engle. <strong>1962.</strong></li><li><a href="http://amazon.com/There-Tommy-Orange/dp/0525436146/ref=sr_1_1?keywords=There+there&amp;qid=1575506924&amp;s=books&amp;sr=1-1">There there</a>, by Tommy Orange.<strong> 2018</strong></li><li><a href="https://www.amazon.com/Book-Thief-Markus-Zusak/dp/0375842209/ref=sr_1_2?keywords=The+book+thief%2C&amp;qid=1575506949&amp;s=books&amp;sr=1-2">The book thief,</a> by Markus Zusak. <strong>2006</strong></li><li><a href="https://www.amazon.com/Less-Winner-Pulitzer-Prize-Novel/dp/031631613X/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575506964&amp;sr=1-1">Less: a novel</a>, by Andrew Sean Greer. <strong>2017</strong></li><li><a href="https://www.amazon.com/Wonder-R-J-Palacio-dp-0375869026/dp/0375869026/ref=mt_hardcover?_encoding=UTF8&amp;me=&amp;qid=1575506978">Wonder</a>, by R. J. Palacio. <strong>2012</strong></li></ol><blockquote>A decade of memoirs and movie adaptations!</blockquote><p>You read that right: <strong>Educated</strong>, that was released <em>less than two years ago </em>(Feb 2018)<em> </em>tops the list. And <strong>Becoming, </strong>that was released just one year ago (November 2018) comes in at #5. It’s quite incredible that these two books, as well as <strong>There, there </strong>(also from 2018)<strong> </strong>and three books from 2017, got to the Top-20 of the entire decade. This skew towards such recent books stands in some contrast with the e.g., the<a href="https://open.spotify.com/playlist/37i9dQZF1DWYJ5kmTbkZiz"> Spotify’s Top Songs of the Decade playlist</a> that seems to includes a more mix of older and recent songs.</p><p>The other theme we see in the Top-20 list is the prominence of books that became movies. The correlation here can, of course, go both ways: books are selected to become movies <em>because</em> they are so popular, but also movies help the books become <em>more</em> popular. I think there are examples of both types in this Top-20 list. The <strong>Hunger Games</strong> series is a good example of the movies helping propel the books into new highs, with the all three books finishing in the top-6 spots. The <strong>Wrinkle in Time</strong> movie starring Oprah and Mindy Kaling) also helped the 1962 book get in the Top-20 of the decade.</p><p>But there are also examples of books that I think would have stand their ground even without the movies: <strong>Gone Girl</strong> and <strong>Girl on the train</strong> were extremely popular even before their respective movies came out, and, in fact, in the case of “Girl on the train” the movie wasn’t even particularly successful.</p><p>14 out of the top-20 books were fiction and 6 were non-fiction, mostly memoirs.</p><p>Finally, female writers dominated this decade of books: 13 out of the top-20 books were written by female authors, compared to 7 for male.</p><h3>Bonus: Top-20 Authors of the 2010s.</h3><p><em>(in parenthesis their most popular book of the decade)</em></p><p>The Top-20 <strong>books</strong> represent, in most cases, the single exceptional book of authors. The Top-20 <strong>authors</strong> list captures very well all the authors that didn’t necessarily have any <em>single</em> book that was a super smash hit, but produced <em>a popular series of </em>books. Only one author appears in both lists: Suzanne Collins. She had both a plethora of books (with the Hunger games trilogy) <em>and </em>a super smash hit (The Hunger Games).</p><p>Interestingly, Suzanne Collins is only of one four female authors in this list. A stark contrast to the dominance of female authors in the top-20 books.</p><ol><li>James Patterson (<a href="https://www.amazon.com/Private-James-Patterson/dp/044657256X/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575507058&amp;sr=1-1">Private : Los Angeles, New York, San Diego, London, Chicago, Paris, Frankfurt, Tokyo, Rome</a>)</li><li>Alexander McCall Smith (<a href="https://www.amazon.com/Department-Sensitive-Crimes-Detective-Novel/dp/0525565671/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575507085&amp;sr=1-1">The Department of Sensitive Crimes)</a></li><li>Michael Connelly (<a href="https://www.amazon.com/Kinds-Truth-Harry-Bosch-Novel/dp/1455524174/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575507107&amp;sr=1-1">Two kinds of truth</a>)</li><li>J.A. Jance (<a href="https://www.amazon.com/Left-Dead-Reynolds-Mysteries-J/dp/B00CAYLGKC/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575507127&amp;sr=1-1">Left for dead : a novel</a>)</li><li>Stephen King (<a href="https://www.amazon.com/11-22-63-Stephen-King/dp/1451627297/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575507148&amp;sr=1-1">11/22/63 : a novel</a>)</li><li>Janet Evanovich (<a href="https://www.amazon.com/Smokin-Seventeen-Stephanie-Plum-Novel-ebook/dp/B004HFRJCG/ref=sr_1_2?keywords=Smokin%27+seventeen&amp;qid=1575507165&amp;s=books&amp;sr=1-2">Smokin’ seventeen</a>)</li><li>Brian Michael Bendis (<a href="https://www.amazon.com/Guardians-Galaxy-Cosmic-Avengers-Marvel/dp/0785166076/ref=sr_1_1?keywords=Cosmic+avengers&amp;qid=1575507187&amp;s=books&amp;sr=1-1">Cosmic avengers</a>)</li><li>Louise Penny (<a href="https://www.amazon.com/Bury-Your-Dead-Inspector-Gamache-ebook/dp/B003P8PENC/ref=sr_1_1?keywords=Bury+your+dead&amp;qid=1575507207&amp;s=books&amp;sr=1-1">Bury your dead</a>)</li><li>Masashi Kishimoto (<a href="https://www.amazon.com/Naruto-Vol-1-Uzumaki/dp/1569319006/ref=sr_1_1?keywords=Naruto.+Vol.+1&amp;qid=1575507222&amp;s=books&amp;sr=1-1">Naruto. Vol. 1</a>)</li><li>Rick Steves (<a href="https://www.amazon.com/Rick-Steves-Best-Europe-2015/dp/1612389562/ref=sr_1_1?keywords=best+of+Europe+2015+rick+steves&amp;qid=1575507235&amp;s=books&amp;sr=1-1">best of Europe 2015</a>)</li><li>Agatha Christie (<a href="https://www.amazon.com/Mysterious-Affair-Styles-Hercule-Mysteries/dp/0525565108/ref=sr_1_1_sspa?keywords=The+mysterious+affair+at+Styles&amp;qid=1575507249&amp;s=books&amp;sr=1-1-spons&amp;psc=1&amp;spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUEyTFozRktUT1QyS0NZJmVuY3J5cHRlZElkPUEwNTgyNzQ0NklFR1QxUU5OWkROJmVuY3J5cHRlZEFkSWQ9QTA0NDc3MTMyMDhHS1BPT0FSM05ZJndpZGdldE5hbWU9c3BfYXRmJmFjdGlvbj1jbGlja1JlZGlyZWN0JmRvTm90TG9nQ2xpY2s9dHJ1ZQ==">The mysterious affair at Styles</a>)</li><li>David Baldacci (<a href="https://www.amazon.com/End-Game-Will-Robie-Book-ebook/dp/B06WGTZXG7/ref=sr_1_1?crid=MQ3HJYJWHMI5&amp;keywords=end+game+david+baldacci&amp;qid=1575507266&amp;s=books&amp;sprefix=End+game+david%2Cstripbooks%2C133&amp;sr=1-1">End game</a>)</li><li>Neil Gaiman (<a href="https://www.amazon.com/Ocean-End-Lane-Novel/dp/0062255665/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575507287&amp;sr=1-2">The ocean at the end of the lane</a>)</li><li>Lee Child (<a href="https://www.amazon.com/Midnight-Line-Jack-Reacher-Novel-ebook/dp/B06WP486XH/ref=sr_1_1?keywords=The+midnight+line&amp;qid=1575507304&amp;s=books&amp;sr=1-1">The midnight line</a>)</li><li>John Grisham (<a href="https://www.amazon.com/Rooster-Bar-John-Grisham-ebook/dp/B01N6YQAPS/ref=sr_1_1?keywords=The+Rooster+Bar%29&amp;qid=1575507321&amp;s=books&amp;sr=1-1"> The Rooster Bar)</a></li><li>Suzanne Collins (<a href="https://www.amazon.com/Hunger-Games-Book-1/dp/0439023521/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575507338&amp;sr=1-1">The hunger games</a>)</li><li>Jeff Smith (<a href="https://www.amazon.com/Out-Boneville-BONE-Jeff-Smith/dp/0439706408/ref=sr_1_1?keywords=Out+from+Boneville&amp;qid=1575507354&amp;s=books&amp;sr=1-1">Out from Boneville</a>)</li><li>Robert Kirkman (<a href="https://www.amazon.com/Walking-Dead-Vol-Days-Gone/dp/1582406723/ref=sr_1_1?keywords=The+walking+dead.+%5BVolume+1%2C+Days+gone+bye%5D&amp;qid=1575507367&amp;s=books&amp;sr=1-1">The walking dead. [Volume 1, Days gone bye]</a>)</li><li>John Sandford (<a href="https://www.amazon.com/Storm-Prey-Book-20-ebook/dp/B003NX7O00/ref=sr_1_1?keywords=Storm+prey&amp;qid=1575507377&amp;s=books&amp;sr=1-1">Storm prey</a>)</li><li>Nora Roberts (<a href="https://www.amazon.com/Year-One-Chronicles-Book-ebook/dp/B06X6G8FKH/ref=sr_1_1?keywords=Year+one&amp;qid=1575507390&amp;s=books&amp;sr=1-1">Year one</a>)</li></ol><p>Of course, this list is only from Seattle, but hopefully it captures some of the macro trends in the book industry in general, such as the wild popularity of books like as the <strong>Hunger Games</strong> trilogy, <strong>Educated</strong> and <strong>Becoming</strong>. If you’re aware of any more libraries that open source their data, let me know in the comments!</p><p>Are there any books that you’re surprised are missing from this list? Or books that you’re surprised to see in this list?</p><p>Does this list make you wanna go back and re-read all your favorites from the past decade?</p><p>What a decade it has been! Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30a7f70caee7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Top Books of 2019]]></title>
            <link>https://medium.com/@yaskalidis/the-top-books-of-2019-e919982406aa?source=rss-2c4731836156------2</link>
            <guid isPermaLink="false">https://medium.com/p/e919982406aa</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[seattle]]></category>
            <category><![CDATA[writing]]></category>
            <category><![CDATA[top-10]]></category>
            <category><![CDATA[books]]></category>
            <dc:creator><![CDATA[Yorgos Askalidis]]></dc:creator>
            <pubDate>Tue, 10 Dec 2019 16:01:39 GMT</pubDate>
            <atom:updated>2019-12-10T16:01:39.085Z</atom:updated>
            <content:encoded><![CDATA[<h4>as determined by the Seattle Public Library Checkout Records</h4><p>The year is coming to a close, and that means it’s time for reflection and Top-10 lists!</p><p>Last year, I published a list with the most checked-out books of 2018 at the <a href="https://www.spl.org/">Seattle Public Library</a>, made possible by their <a href="https://data.seattle.gov/Community/Checkouts-by-Title/tmmm-ytt6">open source datasets</a>. In this post I’m doing it again for 2019.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5yp6g_pDYOZ64rQMml_HGQ.jpeg" /></figure><p>The top book last year was <strong>Educated</strong>, by Tara Westover. Will it keep its crown or will Michelle Obama unseat her at the top?</p><p><em>Note that the list is focused only on adult and young adult books. I removed juvenile books (e.g., Dr. Seuss) from the dataset.</em></p><p>Let’s dive in!</p><h3>Most Checked Out books of 2019</h3><ol><li><a href="https://www.amazon.com/Becoming-Michelle-Obama/dp/1524763136/ref=tmm_hrd_swatch_0?_encoding=UTF8&amp;qid=1575506693&amp;sr=1-1">Becoming</a>, <em>by Michelle Obama.</em></li><li><a href="https://www.amazon.com/Where-Crawdads-Sing-Delia-Owens/dp/0735219095/ref=sr_1_2?keywords=Where+the+crawdads+sing&amp;qid=1575507637&amp;s=books&amp;sr=1-2">Where the crawdads sing</a>, <em>by Delia Owens.</em></li><li><a href="https://www.amazon.com/Educated-A-Memoir/dp/B075F68BFV/ref=sr_1_1?crid=2CYO6UAQHNIV7&amp;keywords=educated&amp;qid=1575506457&amp;sprefix=educate%2Caps%2C163&amp;sr=8-1">Educated : a memoir</a>, <em>by Tara Westover.</em></li><li><a href="https://www.amazon.com/Normal-People-Novel-Sally-Rooney-ebook/dp/B07FS25XTW/ref=sr_1_1?keywords=Normal+people+%3A+a+novel&amp;qid=1575507680&amp;s=books&amp;sr=1-1">Normal people : a novel</a>, <em>by Sally Rooney.</em></li><li><a href="https://www.amazon.com/Library-Book-Susan-Orlean/dp/1476740194/ref=sr_1_1_sspa?keywords=The+library+book&amp;qid=1575507662&amp;s=books&amp;sr=1-1-spons&amp;psc=1&amp;spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUExUzhKOFNNVEhCSUhKJmVuY3J5cHRlZElkPUEwNzI3MDUzMzZNS0RITkNVNDFJVyZlbmNyeXB0ZWRBZElkPUEwNjY4NzkxM1JORjdVRTZKTFc1MiZ3aWRnZXROYW1lPXNwX2F0ZiZhY3Rpb249Y2xpY2tSZWRpcmVjdCZkb05vdExvZ0NsaWNrPXRydWU=">The library book</a>, <em>by Susan Orlean.</em></li><li><a href="https://www.amazon.com/Overstory-Novel-Richard-Powers/dp/039335668X/ref=sr_1_1?keywords=The+overstory+%3A+a+novel&amp;qid=1575507778&amp;s=books&amp;sr=1-1">The overstory : a novel</a>,<em> by Richard Powers.</em></li><li><a href="https://www.amazon.com/Eight-Dates-Essential-Conversations-Lifetime/dp/1523504463/ref=sr_1_1?keywords=Eight+dates+%3A+essential+conversations+for+a+lifetime+of+love&amp;qid=1575507737&amp;s=books&amp;sr=1-1">Eight dates : essential conversations for a lifetime of love</a>, <em>by John Gottman, PhD, Julie Schwartz Gottman, PhD, Doug Abrams &amp; Rachel Carlton Abrams, MD.</em></li><li><a href="https://www.amazon.com/Digital-Minimalism-Choosing-Focused-Noisy/dp/0525536515/ref=sr_1_1?keywords=Digital+minimalism+%3A+choosing+a+focused+life+in+a+noisy+world&amp;qid=1575507759&amp;s=books&amp;sr=1-1">Digital minimalism : choosing a focused life in a noisy world</a>, <em>by Cal Newport.</em></li><li><a href="https://www.amazon.com/City-Girls-Novel-Elizabeth-Gilbert-ebook/dp/B07HZ2Q1MK/ref=sr_1_1?keywords=City+of+girls&amp;qid=1575507855&amp;s=books&amp;sr=1-1">City of girls</a>, <em>by Elizabeth Gilbert.</em></li><li><a href="https://www.amazon.com/Ottolenghi-Simple-Cookbook-Yotam/dp/1607749165/ref=sr_1_1?keywords=Ottolenghi+simple+%3A+a+cookbook&amp;qid=1575507699&amp;s=books&amp;sr=1-1">Ottolenghi simple : a cookbook, </a><em>by Yotam Ottolenghi, with Tara Wigley and Esme Howarth ; photographs by Jonathan Lovekin.</em></li><li><a href="https://www.amazon.com/Moment-Lift-Empowering-Women-Changes/dp/1250313570/ref=sr_1_1?keywords=The+moment+of+lift+%3A+how+empowering+women+changes+the+world&amp;qid=1575507877&amp;s=books&amp;sr=1-1">The moment of lift : how empowering women changes the world</a>, <em>by Melinda Gates.</em></li><li><a href="https://www.amazon.com/Bad-Blood-Secrets-Silicon-Startup/dp/0525431993/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575507811&amp;sr=1-2">Bad blood : secrets and lies in a Silicon Valley startup</a>, <em>by John Carreyrou.</em></li><li><a href="https://www.amazon.com/Maybe-You-Should-Talk-Someone/dp/1925322815/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575507933&amp;sr=1-1">Maybe you should talk to someone : a therapist, her therapist, and our lives revealed</a>, <em>by Lori Gottlieb.</em></li><li><a href="https://www.amazon.com/Big-Sky-Jackson-Brodie-Book-ebook/dp/B07MQSGLHL/ref=sr_1_1?keywords=Big+sky&amp;qid=1575507971&amp;s=books&amp;sr=1-1">Big sky</a>, <em>by Kate Atkinson.</em></li><li><a href="https://www.amazon.com/Maid-Hard-Work-Mothers-Survive/dp/0316505110/ref=sr_1_1?keywords=maid+%3A+hard+work%2C+low+pay%2C+and+mother%27s+will+to+survive&amp;qid=1575507831&amp;s=books&amp;sr=1-1">Maid : hard work, low pay, and a mother’s will to survive</a>, <em>by Stephanie Land ; foreword by Barbara Ehrenreich.</em></li><li><a href="https://www.amazon.com/Save-Me-Plums-Gourmet-Memoir/dp/1400069998/ref=sr_1_1?keywords=Save+me+the+plums+%3A+my+Gourmet+memoir&amp;qid=1575507892&amp;s=books&amp;sr=1-1">Save me the plums : my Gourmet memoir</a>, <em>by Ruth Reichl.</em></li><li><a href="https://www.amazon.com/Earth-Were-Briefly-Gorgeous-Novel/dp/0525562028/ref=sr_1_1?crid=1WB6FV7OU3D6L&amp;keywords=on+earth+we%27re+briefly+gorgeous&amp;qid=1575728895&amp;sprefix=on+earth%2Caps%2C138&amp;sr=8-1">On Earth We’re Briefly Gorgeous: A novel</a>, <em>by Ocean Vuong</em></li><li><a href="https://www.amazon.com/Island-Sea-Women-Novel-ebook/dp/B07GNV4F4S/ref=sr_1_1?keywords=The+island+of+sea+women+%3A+a+novel&amp;qid=1575507914&amp;s=books&amp;sr=1-1">The island of sea women : a novel</a>, <em>by Lisa See.</em></li><li><a href="https://www.amazon.com/There-Tommy-Orange/dp/0525436146/ref=sr_1_1?keywords=there+there&amp;qid=1575728648&amp;sr=8-1">There, there</a>, <em>by Tommy Orange</em></li><li><a href="https://www.amazon.com/Dare-Lead-Brave-Conversations-Hearts/dp/1785042149/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=1575507952&amp;sr=1-1">Dare to lead : brave work, tough conversations, whole hearts</a>, <em>by Brené Brown.</em></li></ol><p>Michelle Obama did, indeed, unseat Tara Westover at the top, with her memoir <strong>Becoming</strong>. But <strong>Educated</strong> is still going strong, almost two years after its first release in February of 2018, occupying the third spot. Delia Owens came in second place with her first novel, <strong>Where the Crawdads sing.</strong></p><p>The split between fiction and non-fiction was almost even, with 11 non-fiction and 9 fiction. Non-fictions books were mostly memoirs, cooking, and lifestyle advise.</p><p>Finally, <em>female</em> authors dominated the list this year. All five of the Top-5 most checked-out books were written by women, and, in total, 14 books in the list had a female primary author, compared to 5 male (<strong>Eight dates </strong>has two male and two female authors).</p><p>Beyond the most checked out <em>books</em> I also created a list with the most checked-out <em>authors</em>. This was to capture authors that perhaps didn’t have any <em>one</em> very popular book but have a popular series or a plethora of books.</p><h3>Bonus: Top authors of 2019</h3><p><em>(in parenthesis, their most checked out book for 2019)</em></p><ol><li>Michelle Obama <em>(Becoming)</em></li><li>Delia Owens (<em>Where the crawdads sing</em>)</li><li>Kate Atkinson (<em>Big sky</em>)</li><li>James Patterson (<em>The chef</em>)</li><li>Rick Steves (<em>Rick Steves France 2019)</em></li><li>Alexander McCall Smith (<em>The Department of Sensitive Crimes)</em></li><li>Louise Penny (<em>Kingdom of the blind</em>)</li><li>Tara Westover (<em>Educated : a memoir)</em></li><li>Sally Rooney (<em>Normal people : a novel)</em></li><li>John Gottman <em>(Eight dates : essential conversations for a lifetime of love)</em></li><li>Cal Newport (<em>Digital minimalism : choosing a focused life in a noisy world</em>)</li><li>Yotam Ottolenghi <em>(Ottolenghi simple : a cookbook)</em></li><li>Brené Brown (<em>Dare to lead : brave work, tough conversations, whole hearts)</em></li><li>Susan Orlean (<em>The library book</em>)</li><li>Richard Powers (<em>The overstory : a novel)</em></li><li>G. Willow Wilson (<em>The bird king)</em></li><li>Michael Connelly (<em> Dark sacred night</em>)</li><li>Stephen King (<em>Elevation</em>)</li><li>Agatha Christie (<em>Murder on the Orient Express)</em></li><li>Margaret Atwood <em>(The testaments)</em></li></ol><p>We see that many authors that appeared in the Top-20 books list, appear also in the Top-20 authors. But we also see some new authors, such as James Patterson who has many popular series, Stephen King, Agatha Christie and Margaret Atwood, who released her long-awaited sequel to the <strong>Handmaid’s tale.</strong></p><p>As always, I note that this list is only from the Seattle Public library, and trends might be different in other cities or states. Hopefully though, even this local list can provide some insight on main trends in books in 2019!</p><p><em>Does this make you go back and (re-)read all these books? Are there books that you were surprised were in the list or that were missing from the list?</em></p><p>Stay tuned on my profile, next week I’ll publish the most checked-out books and authors of the <strong>decade</strong>! A decade that brought us not only all the amazing books in this list but also the Hunger Games series, Gone Girl, Girl on the Train and so many other blockbuster books!</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e919982406aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying data science roles]]></title>
            <link>https://medium.com/@yaskalidis/what-kind-of-data-science-role-is-right-for-you-9d2f4b117e81?source=rss-2c4731836156------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d2f4b117e81</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[jobs]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yorgos Askalidis]]></dc:creator>
            <pubDate>Tue, 06 Aug 2019 09:21:58 GMT</pubDate>
            <atom:updated>2019-08-22T14:37:28.264Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to navigate the data science job postings, and find the one that is right for you!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*201G9pOmfj4Kn4MemTG_XQ.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@rawpixel?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">rawpixel.com </a>from <a href="https://www.pexels.com/photo/photo-of-a-laptop-on-top-of-table-1516704/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p>The “data scientist” has cemented its legacy in <a href="https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century">pop culture</a> as the vague buzzword describing anyone who can access and make any sense of the data that power so many of our experiences today. Data scientists of all types are in, seemingly, hot demand. A search on LinkedIn for “Data Scientist” job postings will return 28 thousand postings for the United States alone. But searches for “Data Analyst”, “Machine Learning Engineer”, even “Data Engineer” and “Data Visualization” will return tens of thousands of postings each.</p><p><em>How can you make sense of the differences between these positions, especially as you’re entering the data science job market for the first time?</em></p><p>In my three years at Spotify I had a change in title, from “Data Analyst” to “Data Scientist”, without my role changing. And from there, a change in my role, from product to finance, without my <em>title</em> changing. New hybrid positions have also been created spanning data science, data engineering, data visualization.</p><p>Each of these roles has its own set of stakeholders and requirements of subject-matter expertise. In this post I’m offering my thoughts and experience on some of the main types of data science jobs that I have come across in the market so you can identify positions best suited to you and interview prep accordingly.</p><h4>Product Data Analyst</h4><p>This is a common role in many companies (in technology and other fields) that usually focuses on a specific part of the business or the product. You will be working in close proximity with other disciplines such as product managers, designers and user researchers and you will be expected to inform their decisions with data and analysis. Those decisions can be about launching a specific feature or about high level strategy. In your daily work you will be defining key metrics, as well as sourcing the data, engineering and visualizations to track those metrics. Depending on the team and product, you might be also designing and evaluating AB tests and doing statistical modeling.</p><p>Most of your stakeholders will be non-technical. It will be your responsibility to promote data literacy and emphasize the importance of your findings. <em>The quality of the analysis is always paramount but you will be evaluated mostly on the impact you’re driving.</em> This means that you always have to ask “So what?” to every insight your analysis reveals and, above all, make sure to communicate clearly what your analysis means to the end user and what are your recommended actions.</p><p>This role can have a lot of freedom in the ways that you drive impact. Any trend or insight you identify and convince decision makers to take action upon, is a win. The potential downside of this freedom is that many times you will not have a single clearly defined question to pursue and that can cause lack of focus or frustration because you’re not sure what to work on. Developing a sense of priorities and ability to identify high impact projects will be key skills.</p><p>The word “Analyst” tends to have a more junior connotation and hence many companies (especially in the technology sector) instead use “Data Scientist”. My advise would be to not pay much attention on the word used and instead focus on the job description, team, company and, of course, the potential salary.</p><p><strong><em>Important Skills: </em></strong><em>Product sense, cross-discipline collaboration, AB testing, metric definition, data visualization, stakeholder communication.</em></p><p>If you’re a data scientist that is more interested in the analysis and modeling rather than the impact on the product, then the next two roles might be more appropriate for you.</p><h4>Business Intelligence, Analytics &amp; Reporting</h4><p><em>(Thanks to Romain Takeo for feedback with this role).</em></p><p>This role is similar to the product data analyst role, it’s focused on serving the analytic and reporting needs of a stakeholder, such as marketing or market strategy instead of a product team. In this role you will be working closely with key decision makers in your company and you will be creating the tools to visualize and report the health of important metrics. You might be tasked with setting some goals and do some light forecasting as a benchmark for the health metrics you’re tracking, and then give regular updates on the performance of those metrics.</p><p>You will be also the point person for various ad-hoc requests that your stakeholders might have, and you will have to learn to translate vague questions into specific data analysis. You will also have to learn to prioritize the requests coming your way, or create self-serving tools for your stakeholders to open up time for you and your analysis.</p><p>Your key performance indicator in this role will be the decision making you’re driving. As you get more senior, you will be expected to not only reactively answer the questions your stakeholder give you but proactively come up with your own directions.</p><p><strong><em>Important Skills: </em></strong><em>stakeholder communication, data visualization, setting and tracking metrics, spherical view of the product, data storytelling.</em></p><h4>Modeling Analyst</h4><p>This is a role that can be found often in financial teams, such as forecasting, and involves usually a clearly defined set of questions (and metrics) centered around predictions and causal inference. The role involves the development of mathematical models in order to evaluate or predict measurements such as retention or lifetime value of a customer.</p><p>This role is frequently within a team of other data scientists and is evaluated on the volume and quality of analysis and model improvements rather than “impact” on a product. The role can resemble an academic setting, frequently reading about advancements in various modeling and machine learning techniques and finding ways to apply them to your company’s setting.</p><p>The role doesn’t normally involve a lot of backend or data engineering and hence it’s mostly focused on “offline” analysis, i.e., analysis with static historical data instead of real-time analysis.</p><p><strong><em>Important Skills: </em></strong><em>Mathematical modeling, statistics, time series manipulation</em></p><h4>Machine Learning Engineer</h4><p>This is a highly desired role and can be behind some of the most user facing applications the internet has to offer today, such as recommendations systems or Spotify’s Discover Weekly playlist. In this role you will sit between the data science and engineering realms: you need to have a great understanding of statistical and machine learning models and also be able to implement a scalable backend solution to ship your model into production.</p><p>This role is much more technical compared to the product data analyst and can be more technical even from the modeling analyst. Similar to the modeling analyst, this role is usually situated within a team of other technical roles such other MLEs and data scientists. The main performance indicator will be the quality of your backend work and the ML models you ship.</p><p><strong><em>Important Skills: </em></strong><em>Machine learning models, statistics, back end engineering</em></p><p>The above three job categories likely cover the majority of “traditional” data science jobs, but I think there is a new category of a hybrid data scientist positions that are being created in response to the demand for data scientists that have data engineering and data visualization skills.</p><h4>Hybrid Data Engineer/Data Scientist</h4><p>Data is the fuel of all data analysis and data science. But the raw state of data is huge tables of logs in a myriad of formats and datasets. In order for any meaningful analysis to be done, one must load, combine, clean and reformat a lot of datasources into one or few tables that can then be queried for analysis or model training. Frequently, this process needs to be scheduled so it can be repeated in regular intervals (e.g., hourly or daily) so that the analysis tables reflect the latest data. Roughly speaking, this process is called <strong>“data pipeling”</strong> and data engineers usually own these types of projects. Most companies (that have the headcount) have teams of professional data engineers that are tasked with creating and maintaining very high quality business critical datasets that the data scientists can query for their analysis. But for many analyses, the data scientist will have to create their own ad-hoc data pipelines. These pipelines and tables can start becoming cumbersome to maintain, especially if dashboards or self-serving tools have been built on top of them. That’s where this hybrid role enters the picture. A team consisting of such hybrid data engineers/data scientists will be tasked with near-full-time creation and maintenance of analysis and dashboard friendly tables and hence provide data pipelining as a service to enable other data scientists and stakeholders in the company.</p><p>Even if you don’t aspire to this hybrid role, you should be advised that some fundamental data engineering skills are quickly becoming a great tool in your data science arsenal, and might be a skill that will give you an advantage in your job search.</p><p><strong><em>Important Skills: </em></strong><em>data pipelining, analytics</em></p><h4>Hybrid Data Visualization Expert</h4><p>Along the same line as the data engineer/data scientist hybrid role mentioned above, this hybrid role is one that is primarily focused on data-visualization-as-a-service to data scientists and other stakeholders.</p><p>Data visualization is one of the most important aspects of the work of many data scientists, helping elevate the analysis and maximize its impact. Carefully designed dashboards can empower your stakeholders to do their own exploration and free up precious data science time for other analysis.</p><p>Careful and impactful dashboard design is time consuming and requires special skills, that’s why this role was created. In this role, you will be working closely with product data analysts and data engineers on your team to understand the needs of your stakeholders and create the dashboards to fulfill them. These dashboards can be tracking key performance indicators of the team, business, product and more.</p><p>This role falls under the data science umbrella because it requires a solid understanding of metrics and analytics. Your stakeholders will both be other data scientists on your team as well as non-technical product managers and executives that will be consuming your reports and dashboards to inform decisions.</p><p>If you are a data viz expert with some data engineering skills as well, then can position yourself as a well rounded candidate for this type of positions.</p><p><strong><em>Important Skills: </em></strong><em>data visualization, tableau (or equivalent), story telling.</em></p><p>Spotify is an example of a success story for these hybrid roles. A core team was created and tasked with defining, tracking, maintaining and visualizing the organization’s key metrics. They were also tasked with creating high quality tables to serve as the source of truth for various fundamental data captured, and hence enabling data scientist to be more confident on their analysis, machine learning training and reporting.</p><p>At Spotify, I have also seen a broader version of the above two hybrid roles: the data scientist that builds data science platforms and tools for other data scientist to use.</p><h4>Data Science Platforms &amp; Tools Developer</h4><p>This role is probably one that only companies with a higher headcount can create and it is focused on enabling other data science teams across the company by creating tools and platforms. Examples of such tools and platforms can include a unified and streamlined way that data scientists can start, develop and share a coding project as well as R and python packages that can be used to standardize methods (such as clustering and Bayesian statistics) and make it easy for all data scientists to use them.</p><p>This role requires a lot of quantitative skills and probably individual contributor experience as well, in order to be able to understand various data science needs and execute on solutions. It also requires a good grasp of qualitative skills in order to collect timely feedback from your peers and understand their needs or how a potential solution you’re developing can fulfill those needs.</p><p><strong><em>Important Skills: </em></strong><em>experience with individual data science work, technical skills to develop cloud solutions and/or python packages, user research skills</em></p><h4>What role is right for you?</h4><p>The above roles can be distinguished by an <strong>impact</strong> vs <strong>methods</strong> vs <strong>platforms</strong> trichotomy. If you’re a talented data scientist interested in driving insights and impact on a particular product or part of the business, with less focus on the methodology and techniques used, then you might be more suited for the product data analyst or business intelligence positions. If you are interested in always being near the nexus of the latest machine learning models and methods and are excited to apply them to your company’s vast data sources then you might be better suited to the modeling analyst and machine learning engineer positions. Finally, if you’re motivated by taking a more horizontal role in your organization, to educate and enable other teams to do their work better, then you might be interested in the data-services-as-a-platform roles, such as the platforms &amp; tools developer.</p><p>If there’s a job listing that caught your eye and you aim to understand better what type of a data science role it is, read the job description and try to see if they have an emphasis on impact, methods or platforms. Additionally, ask the hiring manager who your stakeholders will be and how you will your performance be evaluated.</p><p>If you’re entering the data science job market for the first time, Welcome! I hope this guide can help give you some more clarity on the types of positions there are and help you identify the opportunities that are best suited to your talents and interests. Let me know in the comments if you’re finding roles that don’t fit into the impact vs methods vs platforms trichotomy.</p><p>If you’re someone with experience in the data science job market, how have you seen data science roles take shape in your companies?”</p><p><em>Thanks for reading!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d2f4b117e81" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The data is in: Ethiopia has the best coffee]]></title>
            <link>https://medium.com/@yaskalidis/the-data-speak-ethiopia-has-the-best-coffee-91f88ed37e84?source=rss-2c4731836156------2</link>
            <guid isPermaLink="false">https://medium.com/p/91f88ed37e84</guid>
            <category><![CDATA[coffee]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[ethiopia]]></category>
            <dc:creator><![CDATA[Yorgos Askalidis]]></dc:creator>
            <pubDate>Tue, 26 Mar 2019 11:01:00 GMT</pubDate>
            <atom:updated>2019-03-26T15:24:50.647Z</atom:updated>
            <content:encoded><![CDATA[<h3>Where can you find the best coffee?</h3><p>The world loves coffee! It is one of the world’s most traded commodities and it can excite our senses with its plethora of great aromas and flavors. In this blog post we’re gonna get a glimpse on the professional practice of grading coffee samples and try to understand what makes for a great coffee.</p><p><em>What country produces that coffee with the best aroma?</em></p><p><em>Does altitude affect the overall quality of the coffee?</em></p><p><em>Can we build a coffee profile for various countries?</em></p><p><em>Do professional graders tend to score lower or Mondays?</em></p><p>I analyzed a dataset containing professional grades for 1,300 Arabica coffee samples from around the world to answer all the above questions.</p><p>From the wiki page for <a href="https://en.wikipedia.org/wiki/Coffee_cupping">coffee cupping</a>.</p><blockquote><strong>Coffee cupping</strong>, or <strong>coffee tasting</strong>, is the practice of observing the tastes and aromas of brewed coffee. It is a professional practice but can be done informally by anyone or by professionals known as “Q Graders”.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IPfjFagMug1shxlv885HUw.jpeg" /></figure><p>The <a href="https://www.coffeeinstitute.org/">Coffee Quality Institute</a> is a non-profit organization that grades coffee samples from around the world in a consistent and professional manner. The result are published on their website and were scraped by reddit user <a href="https://www.reddit.com/user/JLD_">JLD</a> and posted on the forum’s <a href="https://www.reddit.com/r/datasets/comments/8rndor/data_on_1340_coffee_bean_reviews_aroma_acidity/">dataset community.</a></p><p>The dataset contains the grades the cuppers gave a sample on each of these ten attributes: <em>Aroma, Flavor, Aftertaste, Acidity, Body, Balance, Uniformity, Clean Cup, Sweetness, Cupper Points.</em></p><p>Each grade is on a 0–10 scale resulting to a total cupping score between zero and one hundred. If you want to get more familiar with the 100-point scale coffee review process you can take a look at <a href="http://cafevirtuoso.com/2017/09/12/what-you-need-to-know-about-the-coffee-review-100-point-scale/">this article</a> and if you want to understand better what <strong>exactly</strong> each attribute is you can read the explanations <a href="https://database.coffeeinstitute.org/coffee/976032/grade">here</a>.</p><h4>Building each country’s coffee profile</h4><p>The dataset records each coffee sample’s country of origin, and that allows me to aggregate the grades for each country and build that country’s <em>coffee profile</em>. Figure 1 shows the results for a few countries and one thing that becomes pretty apparent is that there is actually not much variation between countries! If you look really closely you might notice that e.g., Ethiopia seems to have higher grades than most other countries but not much a large margin. To exemplify this further, the median score on uniformity, clean cup and sweetness is a perfect ten for almost every country.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5N-IesCP1kspn5a5vIzOfg.jpeg" /><figcaption><strong>Figure 1: </strong>Radar plots with the average score for each country and attribute.</figcaption></figure><p>Figure 2 shows all the profiles together to showcase once again how there is not too much difference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ypG3e0zglyKI1Pl7kw_ABA.png" /><figcaption><strong>Figure 2: </strong>The taste profile of all 19 countries with at least 10 coffee samples. Each point of the polygon represents the<strong> average score </strong>of each country for that characteristic.</figcaption></figure><p>With the taste profiles of each country being not too distinct from each other, let’s see which country’s coffee samples scored the highest in each category. Note that for this analysis, I’m restricting the ranking only to the 19 countries that have at least 10 coffee samples graded. Additionally, I am taking the <strong>median grade </strong>of each country’s coffee samples for every category, because it is a more robust metric compared the average.</p><h4>Highest Ranked Country, and runner up</h4><ol><li><strong>Aroma: </strong>Uganda (7.92), Kenya (7.83)</li><li><strong>Flavor: </strong>Ethiopia (7.96), Kenya (7.83)</li><li><strong>Aftertaste: </strong>Ethiopia (7.83), Kenya (7.75)</li><li><strong>Acidity: </strong>Ethiopia (8.0), Kenya (7.75)</li><li><strong>Body: </strong>Ethiopia (7.92), Peru (7.92)</li><li><strong>Balance: </strong>Ethiopia (7.92), Kenya (7.75)</li><li><strong>Uniformity: </strong>All countries tied with a score of 10</li><li><strong>Clean Cup: </strong>All countries tied with a score of 10</li><li><strong>Sweetness: </strong>All countries tied with a score of 10</li><li><strong>Cupper Points: </strong>Ethiopia (8.0), Kenya (7.83)</li></ol><p>We observe that Ethiopia ranks the highest (including ties) in 9 out of 10 areas of grading. Kenya is the most frequent runner-up.</p><p><strong>Out of the ten highest graded coffee samples, seven were from Ethiopia.</strong> That included the top two. Guatemala, Brazil and Peru had one appearance each.</p><p><em>Next time I’m getting a coffee, I’ll be on the lookout for bag from Ethiopia!</em></p><h4>Analyzing additional coffee characteristics</h4><p>The dataset also includes some information about the coffee sample’s <strong>altitude</strong> of origin, <strong>variety</strong>, <strong>processing method</strong> and <strong>color</strong>. I examine here if any of these characteristics correlate with the total cupping score.</p><p>A statistical analysis of the relationship between the <strong>altitude</strong> of the sample and its total cupping score found <strong>no significant correlation</strong>. I don’t really know enough to be surprised or not by this finding. Are there any coffee experts out there that (formally or informally) have found any links between altitude and coffee quality?</p><p>The <strong>processing method </strong>can be one of Natural/Dry, Pulped Natural/Honey, Semi-washed/Semi-Pulped, Washed/Wet, Other and None. The <strong>color</strong> of the coffee sample can be Blueish Green, Blue Green, Green and None. And the <strong>variety</strong> of the sample can be one of many options including such as Blue mountain, Bourbon, Catimor, Caturra and more.</p><p>OLS and Kolmogorov-Smirnov test analysis did not uncover any strong statistically significant correlations between any one of the aforementioned characteristics and the total cupping score of the coffee sample.</p><h4>BONUS: Do graders give lower scores on Mondays?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*Ryn6516N9E8qFuGH2bSPTg.jpeg" /><figcaption><strong>Figure 3: </strong>Median total score per grading year, month and day of week. The opacity of the bars on the left-hand plots are proportional to the number of grading that happened that year, month and day of week (shown on the right-hand side plots)</figcaption></figure><p>Shifting focus away from the coffee samples, I wanted to take a look at the grading <strong>date</strong>. There is research that has <a href="https://www.wired.com/2011/04/judges-mental-fatigue/">found correlation between a parole case’s success rate and whether the case was heard right before or after lunch</a>. The idea here is that judges might simply be a in a better mood right after lunch. I wanted to see if there is any evidence of something like that happening with the coffee sample grading: Do graders give higher or lower grades on a Friday compared to a Monday?</p><p><strong>The answer seems to be No.</strong></p><p>There is no statistical difference between the grades that are awarded on a Monday compared to a Friday or any other day of the week. If you look at the third plot in Figure 3 you will see that the median grade on Fridays seems to be a little higher compared to other days but a statistical analysis revealed no correlation. I performed a 2-sample Kolmogorov-Smirnov test between the grades awarded on a Monday and those awarded on a Friday and found no statistical difference. I also performed a regular OLS regression and still found no correlation.</p><p>Beyond the day of the week, Figure 2 shows that June is the busiest month for grading and that 2012 was a particularly busy (and low-grade!) year. As was the case with various days of the week, there were no statistical differences between the grades awarded on various months of the year.</p><p>There <em>were </em>statistical differences across the grading years but those cannot be easily attributed to the graders; inherent factors about the coffee crops might have a much a bigger influence.</p><p>So there you have it, Ethiopia has the highest graded coffee samples recorded in the dataset and is consistently the highest ranked on almost all the individual stages of the cupping.</p><p>As a bonus exploration, coffee graders seem to be consistent with their grading across years, months and days of the week.</p><p>If all this talk about coffee got you in the mood for a cup of joe and some tunes, you can also read <a href="https://newsroom.spotify.com/2019-03-20/how-to-find-the-perfect-playlist-for-your-coffee-break-vibe/">Spotify’s blog post</a> about coffee inspired playlists.</p><p><em>Are you a professional or amateur coffee cupper? Do these results surprise you? Let me know your thoughts and experiences in the comments!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91f88ed37e84" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Here’s how you can access your entire iMessage history on your Mac]]></title>
            <link>https://medium.com/@yaskalidis/heres-how-you-can-access-your-entire-imessage-history-on-your-mac-f8878276c6e9?source=rss-2c4731836156------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8878276c6e9</guid>
            <category><![CDATA[messaging]]></category>
            <category><![CDATA[apple]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mac]]></category>
            <dc:creator><![CDATA[Yorgos Askalidis]]></dc:creator>
            <pubDate>Sat, 09 Mar 2019 15:28:23 GMT</pubDate>
            <atom:updated>2019-03-09T16:51:09.667Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A guide on how to create a data-science-friendly file with your iMessage history.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kK6nwoRDMBo8OycuF05eNA.jpeg" /></figure><p>If you use the Messages app on your Apple computer then you probably have connected your Apple-Id to that computer in order to send and receive iMessages across all your Apple devices (iPhone, iPad, computer). When using the Messages app, you can see all your history of messages on that device and maybe (like me) you have wondered where exactly is that data stored? Can you access it in a format that is easy to analyze? Wouldn’t it be great to be able to see how many messages you have sent per day? Or read the first message you have ever sent to your friends?</p><p>Well you’re in luck! Your Apple computer stores that message history right within your grasp in a hidden folder on your hard drive!</p><p>This is a somehow technical guide for extracting all the iMessage data in your computer’s hard drive and putting them in an analysis-friendly file.</p><blockquote>Note: this approach only works for Macs (laptops and desktops) and only gets your<strong> iMessage </strong>history. Green bubble messages unfortunately are not captured.</blockquote><p><em>If you just want to see the good stuff you can find the notebook with the minimal code to extract and prepare the data </em><a href="https://github.com/yortos/imessage-analysis/blob/master/imessages-data-extract-and-prep.ipynb"><em>here</em></a><em>.</em></p><h4>Where’s the data</h4><p>The iMessage history that powers your Messages app is stored in a database file in your computer’s hard drive, in a hidden folder named Library which, in turn, is in your username folder. You can usually find your username folder on the side bar of the finder.</p><p><em>Hidden folders</em> are folders that by default don’t appear to the user, usually holding files that a casual user doesn’t have to interact with, such as system-related files. You can make the hidden folders appear by simultaneously pressing the Command, the Shift and the dot keys: “Command+Shift+.”. If for some reason that doesn’t work, you can also open the Terminal app and simply type the following.</p><pre>defaults write com.apple.finder AppleShowAllFiles YES</pre><p>Once that works, you can go and find the Messages folder, (which contains the chat.db database) within the Library folder as shown in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uU8ZhOwKzNO6zGuxtRsaSQ.jpeg" /><figcaption>Notice how some folders appear with a greyed icon; these are the normally hidden folders.</figcaption></figure><h3>What’s a database?</h3><p>A very simple way to understand what a database file is is to think of it a a <em>folder </em>that contains a bunch of <em>excel-like tables. </em>Much like larger or enterprise-grade databases you can connect and access the data in the database in a variety of ways.</p><h3>Using Python and pandas</h3><p>In this tutorial, I’m using Python and the amazing pandas module to connect to the database, explore the tables and data it holds and then read that data from the appropriate tables.</p><pre>connection code<br>import sqlite3<br>import pandas as pd</pre><pre># substitute username with your username<br>conn = sqlite3.connect(&#39;/Users/username/Library/Messages/chat.db&#39;)</pre><pre># connect to the database<br>cur = conn.cursor()</pre><pre># get the names of the tables in the database<br>cur.execute(&quot; select name from sqlite_master where type = &#39;table&#39; &quot;) for name in cur.fetchall():<br>    print(name)</pre><p>Above we connect to the database and explore what tables are in there. I found that there are a few tables in the database including one called <em>message</em> and others names <em>chat</em>, <em>handle</em> and <em>attachment</em>. Let’s explore the <em>message</em> table because that’s the one that sounds most promising to hold our iMessages. I do that by transferring the table into a pandas dataframe, a type of file that is much easier to explore and manipulate for data analyis projects.</p><pre># get the 10 entries of the message table using pandas<br>messages = pd.read_sql_query(&quot;select * from message limit 10&quot;, conn)</pre><h4>Getting the message text and phone number</h4><p>We hit bingo! The <em>message</em> table indeed seems to hold all the saved iMessages. It has a <strong>text</strong> field with the actual sent or received message, a <strong>date</strong> field (more on that below) and a <strong>handle id</strong>. After a little exploration I found that the <strong>handle_id</strong> is a code for each phone number or Apple-id that you have had a conversation with. In order to map the <strong>handle_id</strong> back to the Apple-id we can use a table in the database (appropriately) named <em>handle </em>and join on <strong>handle_id</strong>.</p><pre># get the handles to apple-id mapping table<br>handles = pd.read_sql_query(&quot;select * from handle&quot;, conn)</pre><pre># and join to the messages, on handle_id</pre><pre>messages.rename(columns={&#39;ROWID&#39; : &#39;message_id&#39;}, inplace = True)</pre><pre>handles.rename(columns={&#39;id&#39; : &#39;phone_number&#39;, &#39;ROWID&#39;: &#39;handle_id&#39;}, inplace = True)</pre><pre>merge_leve_1 = temp = pd.merge(messages[[&#39;text&#39;, &#39;handle_id&#39;, &#39;date&#39;,&#39;is_sent&#39;, &#39;message_id&#39;]],  handles[[&#39;handle_id&#39;, &#39;phone_number&#39;]], on =&#39;handle_id&#39;, how=&#39;left&#39;)</pre><h4>Adding a chat id</h4><p>Similarly, the <em>message</em> table also includes a <strong>chat_id</strong> that maps each message back to unique chat. This can be useful when doing analysis on chats with multiple people in them. We can get the <strong>chat_id</strong> of each message by joining the message table with the (again, appropriately named) <em>chat_message_join</em> table on <strong>message_id</strong>.</p><pre># get the chat to message mapping<br>chat_message_joins = pd.read_sql_query(&quot;select * from chat_message_join&quot;, conn)</pre><pre># and join back to the merge_level_1 table<br>df_messages = pd.merge(merge_level_1, chat_message_joins[[&#39;chat_id&#39;, &#39;message_id&#39;]], on = &#39;message_id&#39;, how=&#39;left&#39;)</pre><h4>Getting the date</h4><p>The <em>message</em> table also includes a <strong>date</strong> column and this was a little tricky for me to decode since it isn’t exactly in any format that is widely used in the industry. Moreover, the way that this column is recorded is a little different in newer version of Mac OS X compared to older ones.</p><blockquote>Credit to this <a href="https://apple.stackexchange.com/questions/114168/dates-format-in-messages-chat-db">stackoverflow page</a> that helped me figure this out.</blockquote><p>In Mac OS X versions before High Sierra (which is version 10.13 and released in September 2017), the <strong>date</strong> column is an epoch type but, unlike the standard of counting the seconds from 1970–01–01, it is counting the seconds from 2001–01–01. In order to convert that type into a data field we can actually comprehend we can use a command while querying the message table to create a new field (we will call it <strong>date_utc, </strong>since it is giving a UTC timezone date as a result) based on the <strong>date</strong> field.</p><pre># convert 2001-01-01 epoch time into a timestamp<br># Mac OS X versions before High Sierra<br>datetime(message.date + strftime(&quot;%s&quot;, &quot;2001-01-01&quot;) ,&quot;unixepoch&quot;,&quot;localtime&quot;)</pre><pre># how to use that in the SQL query<br>messages = pd.read_sql_query(&quot;select *, datetime(message.date + strftime(&quot;%s&quot;, &quot;2001-01-01&quot;) ,&quot;unixepoch&quot;,&quot;localtime&quot;) as date_uct from message&quot;, conn)</pre><p>In Mac OS X High Sierra and above, it’s the same thing but the date format is now much more granular: it is in nano-second level. So now we need to divide by 1,000,000,000 before we apply the same code snippet we applied above.</p><pre># convert 2001-01-01 epoch time into a timestamp<br># Mac OS X versions after High Sierra<br>datetime(message.date/1000000000 + strftime(&quot;%s&quot;, &quot;2001-01-01&quot;) ,&quot;unixepoch&quot;,&quot;localtime&quot;)</pre><pre># how to use that in the SQL query<br>messages = pd.read_sql_query(&quot;select *, datetime(message.date/1000000000 + strftime(&quot;%s&quot;, &quot;2001-01-01&quot;) ,&quot;unixepoch&quot;,&quot;localtime&quot;) as date_uct from message&quot;, conn)</pre><h4>Putting it all together</h4><p>You can find the notebook <a href="https://github.com/yortos/imessage-analysis/blob/master/imessages-data-extract-and-prep.ipynb">here</a> with all the code in order for you to extract your iMessages from your laptop and start analyzing!</p><p>It should only take a few minutes and by the end of it you should have a basic history of your iMessage data that includes the <strong>phone number</strong> (or email), the <strong>text</strong>, a unique <strong>chat</strong> for each unique group of people you had a chat with and the <strong>timestamp</strong> (in UTC timezone) of each message sent.</p><p>You can actually find more data in the database such as details if the message was delivered and read as well as attachments. I’m not touching on those attributes on this post.</p><p>This post also only instructs on how to get iMessage data from your Apple computer. If you have any pointers on how you can extract your iMessage history from Apple mobile devices (iPhone &amp; iPad) let me know in the comments.</p><p>Happy reading your messages!</p><p><em>Was this helpful? Let me know if things were not clear.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8878276c6e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What’s your soccer team’s nemesis?]]></title>
            <link>https://medium.com/data-science/whats-your-soccer-team-s-nemesis-30d95c4f9b15?source=rss-2c4731836156------2</link>
            <guid isPermaLink="false">https://medium.com/p/30d95c4f9b15</guid>
            <category><![CDATA[sports]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[barcelona]]></category>
            <category><![CDATA[soccer]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yorgos Askalidis]]></dc:creator>
            <pubDate>Thu, 31 Jan 2019 21:00:17 GMT</pubDate>
            <atom:updated>2019-02-01T16:17:22.755Z</atom:updated>
            <content:encoded><![CDATA[<h4>Is Barcelona really Real Madrid’s toughest opponent? Historical data paint an interesting story.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/0*TOc1N8nd4J6mLL4N.jpg" /></figure><p><em>Real Madrid vs Barcelona. Manchester United vs Liverpool. Inter vs Milan. Olympique Lyonnais vs Olympique de Marseille. Chelsea vs everybody.</em></p><p><strong>European soccer is filled with some amazing rivalries. </strong>These rivalries got created and evolved over time for reasons on and off the pitch and are some of the hardest games that the teams have to face each season. But are these rivalries really the hardest games during the season? For example, even though Real Madrid vs Barcelona is one of the most important games in world soccer, it’s easy to forget how difficult of an opponent Real Sociedad or Athletic de Bilbao can be for Barcelona.</p><p>In this post I want to report the methodology and results of a data analysis I did to look at data from all matches from major European leagues across 8 seasons and find which opponent caused the most damage to each team. Aka,<strong> find each team’s nemesis.</strong></p><p>I describe the data and the methodology at the end of this post in order to dive into the results immediately (Please feel free to check that out and then return to the results!). The main thing you need to know to follow the results is that the <strong>metric I use is the percentage of maximum possible points that a team got out of an opponent. </strong>The maximum possible points is 3 times the number of games. Hence, if e.g., Chelsea played Liverpool 10 times and got a total of 15 points, then Chelsea got 15 out of 30 possible points or 50%.</p><p><em>In order to keep the list kinda short I only report the results for pairs of teams that have (almost) 16 encounters with each other. That’s the maximum possible number of games, since each pair of teams play each other twice a year and we have data for 8 years.</em></p><h3>The results</h3><p>The <em>nemesis</em> for each <strong>major team</strong> studied.</p><p><em>The lower the number, the worst the team performed against their nemesis.</em></p><h4><strong>English Premier League</strong></h4><ul><li><strong>Arsenal</strong>: 25% vs <em>Chelsea</em></li><li><strong>Aston Villa: </strong>14.6% vs <em>Manchester United</em></li><li><strong>Chelsea:</strong> 40% vs<em> Liverpool</em></li><li><strong>Everton: </strong>19% vs <em>Arsenal</em></li><li><strong>Liverpool: </strong>29% vs <em>Arsenal</em></li><li><strong>Manchester City:</strong> 38% vs <em>Liverpool</em></li><li><strong>Manchester United:</strong> 38% vs <em>Chelsea</em></li><li><strong>Stoke City:</strong> 17% <em>Manchester United</em></li><li><strong>Sunderland:</strong> 19% vs <em>Chelsea</em></li><li><strong>Tottenham: </strong>29% vs <em>Manchester United</em></li></ul><h4>Spanish La Liga</h4><ul><li><strong>Athletic Club de Bilbao:</strong> 12% vs <em>Barcelona</em></li><li><strong>Atlético Madrid:</strong> 17% <em>Barcelona</em></li><li><strong>Barcelona:</strong> 53% <em>Real Sociedad</em></li><li><strong>Getafe:</strong> 13% <em>Real Madrid</em></li><li><strong>Málaga:</strong> 8% <em>Barcelona</em></li><li><strong>Espanyol:</strong> 6% vs <em>Real Madrid</em></li><li><strong>Real Madrid:</strong> 29% vs <em>Barcelona</em></li><li><strong>Sevilla:</strong> 13% vs <em>Barcelona</em></li><li><strong>Valencia:</strong> 23% vs <em>Barcelona</em></li></ul><h4>Italian Serie A</h4><ul><li><strong>Chievo Verona:</strong> 6% vs <em>Milan</em></li><li><strong>Genoa:</strong> 20% <em>Juventus</em></li><li><strong>Inter:</strong> 29% vs <em>Juventus</em></li><li><strong>Juventus:</strong> 47% <em>Napoli</em></li><li><strong>Milan:</strong> 29% vs <em>Juventus</em></li><li><strong>Napoli:</strong> 40% vs <em>Roma</em></li><li><strong>Roma:</strong> 25% vs <em>Juventus</em></li><li><strong>Udinese:</strong> 29% vs <em>Juventus</em></li></ul><h4>German Bundesliga</h4><ul><li><strong>Bayer 04 Leverkusen: </strong>29% vs <em>Borussia Dortmund</em></li><li><strong>Borussia Dortmund: </strong>40% vs <em>Bayern Munich</em></li><li><strong>Borussia Monchengladbach:</strong> 31% vs<em> Borussia Dortmund</em></li><li><strong>Bayern Munich:</strong> 50% vs <em>Borussia Monchengladbach</em></li><li><strong>Schalke 04:</strong> 19% vs <em>Bayern Munich</em></li><li><strong>Hamburger SV: </strong>21% vs <em>Bayern Munich</em></li><li><strong>SV Werder Bremen</strong>: 13% vs <em>Bayern Munich</em></li><li><strong>TSG 1899 Hoffernheim:</strong> 8.3% vs <em>Bayern Munich</em></li><li><strong>VfB Stuttgard: </strong>11% vs <em>Bayern Munich</em></li><li><strong>VfL Wolfsburg:</strong> 15% vs <em>Bayern Munich</em></li></ul><h4>French Ligue 1</h4><ul><li><strong>AS Saint-Étienne:</strong> 20% vs <em>Paris Saint-Germain</em></li><li><strong>FC Lorient: </strong>21% vs <em>Paris Saint-Germain</em></li><li><strong>Girondins de Bordeaux:</strong> 38% vs <em>Olympique de Marseille</em></li><li><strong>LOSC Lille:</strong> 27% vs <em>Paris Saint-Germain</em></li><li><strong>OGC Nice:</strong> 29% vs <em>Montpellier Herault SC</em></li><li><strong>Olympique Lyonnais: </strong>31% vs <em>Paris Saint-Germain</em></li><li><strong>Olympique de Marseille: </strong>31% vs <em>Paris Saint-Germain</em></li><li><strong>Paris Saint-Germain: </strong>28% vs <em>AS Monaco</em></li><li><strong>Stade Rennais FC: </strong>27% vs <em>Girondins de Bordeaux</em></li><li><strong>Toulouse FC: </strong>29% vs <em>Paris Saint-Germain</em></li></ul><h3>Highlights</h3><p><em>Ok that was a lot. Let’s process some highlights.</em></p><p><strong>Chelsea is the nemesis of Arsenal.</strong> Given the 8 seasons that this dataset is based upon, this probably doesn’t come as a big surprise. Didier Drogba and his Blue teammates have been terrorizing Arsenal’s defense for some time now.</p><p><strong>Liverpool is the nemesis of Chelsea and Manchester City. </strong><em>But who can forget that slip from Gerrard that allowed Chelsea to score and sunk the title hopes of Liverpool?</em></p><p><strong>Arsenal is the nemesis of Liverpool.</strong><em> Chelsea &gt; Arsenal &gt; Liverpool &gt; Chelsea. </em>This is an interesting cycle showing that in soccer things are not hierarchical: any team can win over any team!</p><p><strong>England’s Premier League is the most egalitarian league. </strong>There are a lot of discussions about La Liga vs Premier League and this blog post can add one more datapoint. FC Barcelona is the nemesis of <strong>6 out of 9 Spanish teams</strong> studied here, whereas in the <strong>Premier League no team is dominating as much. </strong>Chelsea and Manchester United are the nemesis of 3 teams each out of 10 studied.</p><p>Of course, on the other hand there are 10 teams from the Premier League that were never relegated in the 8 studied seasons but only 9 such teams from La Liga. So let’s not rush to judge which league is the most competitive one.</p><p><strong>Real Sociedad is the nemesis of Barcelona. </strong>This is a great example of a somehow unexpected result since most of the media coverage usually goes to the El Classico. But fans that follow the Blaugrana closely know about the ‘anoeta curse’ and how difficult of an opponent Real Sociedad has traditionally been for Barcelona.</p><p><strong>Barcelona is the nemesis of almost all major Spanish teams.</strong> This might not be a big surprise to many people since Barcelona have been so dominant for the past 8 years in their domestic league (they won 6 of the studied 8 seasons). Real Madrid is the nemesis of two major Spanish teams, Getafe (a team also based in the Madrid metro area) and Espanyol.</p><p><strong>Juventus has been dominating the Italian league.</strong> They are the nemesis of 5 out of 8 studied Italian teams. Napoli is Juventus’s nemesis.</p><p><strong>PSG and Bayern Munich have been dominating the French and German league respectively.</strong> They are the nemesis of 7 out of 10 studied teams in their league, the highest ratio of all studied teams. AS Monaco is PSG’s nemesis and Borussia Monchengladbach is Bayern Munich’s nemesis.</p><p><strong>Barcelona is the most resilient team studied. </strong>Barcelona managed to get <strong>53% </strong>of the maximum possible points versus their nemesis, Real Sociedad. That’s the highest across all team/nemesis pairs. Here’s the list with the highest performances against a team’s nemesis.</p><ol><li>Barcelona, 53% vs Real Sociedad</li><li>Bayern Munich, 50% vs Borussia Monchengladbach</li><li>Juventus, 47% vs Napoli</li><li>Borussia Dortmund, 40% vs FC Bayern Munich</li><li>Chelsea, 40% vs Liverpool</li><li>Napoli, 40% vs Roma</li><li>Manchester United, 38% vs Chelsea</li><li>Manchester City, 38% vs Liverpool</li></ol><h3>Conclusion</h3><p>Can this analysis really help Barcelona fans relax when their team is playing against Real Madrid? Probably not. Soccer rivalries are more than just a game and more than 3 points in the domestic league.</p><p>Nevertheless, this analysis and others like it can help sort out the noise and human biases to pinpoint to the players and the coach some teams that they should be focused on more.</p><p>Is your team not in the above list? No worries, you can see (and alter) the notebook that produced all the results <a href="https://github.com/yortos/european-soccer-nemesis">here</a> or just leave a comment below!</p><p><em>Thanks for reading!</em></p><h3>Appendix: Dataset &amp; Methodology</h3><h4>The dataset</h4><p>I used <a href="https://www.kaggle.com/hugomathien/soccer">this Kaggle dataset</a> of European soccer matches. It’s a wonderful database with matches from a plethora of European domestic leagues as well as player and team attributes.</p><h4><strong>The methodology</strong></h4><p>The first thing I did was to query the dataset to get all match results from Spain, England, France and Italy’s domestic leagues. The dataset includes more leagues, so if you really wanna find your team’s nemesis from some league outside those four feel free to take a look at the code or let me know.</p><p>Some immediate basic exploration reveals that there are 8 seasons: from <strong>2008/9</strong> up to and inclusive of <strong>2015/6. </strong>Each of these our leagues has 20 teams that play each other twice, hence each team should have 38 observation per season, for every season they were in the league. European leagues operate on a system of relegation and promotion, hence not all teams will be part of the dataset of every season since some will be relegated.</p><p>The dataset seems pretty complete with almost all matches recorded; there is a small number of matches missing (mostly from the Italian league) but since it’s very small I choose to ignore it.</p><p>Once all the raw logs are available, I go though each team’s complete list of games and record the opponent and the number of points they got from the game: 3 if they won, 0 if they lost and 1 if they draw. This dataset can then be grouped by for each team and opponent pair and calculate the sum of the points they got and then divide by the maximum number of points possible. Hence, for each team we can find the opponent that they got the smallest percentage of possible points. We call that opponent the <strong>nemesis!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30d95c4f9b15" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/whats-your-soccer-team-s-nemesis-30d95c4f9b15">What’s your soccer team’s nemesis?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>