<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Leonardo Bezerra on Medium]]></title>
        <description><![CDATA[Stories by Leonardo Bezerra on Medium]]></description>
        <link>https://medium.com/@leobezerra_90682?source=rss-fff777145cdd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*mcFti90wowGZ2JpA7-ggQA.png</url>
            <title>Stories by Leonardo Bezerra on Medium</title>
            <link>https://medium.com/@leobezerra_90682?source=rss-fff777145cdd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 03:08:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@leobezerra_90682/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Clustering in detail]]></title>
            <link>https://medium.com/data-science/clustering-in-detail-b439f31c56b7?source=rss-fff777145cdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/b439f31c56b7</guid>
            <category><![CDATA[economy]]></category>
            <category><![CDATA[social-inequality]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Leonardo Bezerra]]></dc:creator>
            <pubDate>Tue, 22 Oct 2019 23:31:51 GMT</pubDate>
            <atom:updated>2019-10-30T18:23:40.008Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>It takes context, plots, algorithms, metrics, and fiddling!</blockquote><p>Some of the students in a data science course showed me <a href="https://github.com/julincesar/arquivos-de-dados/blob/master/processosTRE2d.csv">this very interesting dataset about Brazilian states</a> they found on the internet.</p><blockquote>It was love at the first plot!</blockquote><p>But it took much more than just the first plot to grasp some of its nuances.</p><h3>Context: Brazil, a federated republic</h3><p>If you’re not from Brazil or from a country that is organized in states, let me give you a heads-up.</p><p>In Brazil, the political system is organized into three levels of government: federal, state, and city.</p><p>There are 26 states, plus a federal district (DF), which are organized into five geographical regions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1CyEge4gP-JWTm-lu--hvw.png" /></figure><h3>Plots: Brazil, a highly unequal republic</h3><p>As Brazilians, we always hear people speaking of how São Paulo (SP) differs from the rest of the country economic-wise.</p><p>I have been to the capital city of SP (also called São Paulo) a few times, and I had a glimpse of how overpopulated the city was.</p><p>But nothing could really prepare me for this plot 😵:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*O8V5SRqfhEkog-Ra3j_gsw.png" /></figure><p>At this point, it’s very easy to conclude that SP is a strong outlier in the data, and that it should be discarded so we can have a better understanding of the remaining data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XpEENwD1HQtFav1OKopISA.png" /><figcaption>Zooming by discarding SP from the plot</figcaption></figure><blockquote>But is it? Should we really discard it?</blockquote><p>Let’s keep it along for the ride a little longer.</p><h3>Algorithms: start with the simple</h3><p>Since this dataset is extremely simplified, there’s not much EDA (exploratory data analysis) to do prior to clustering.</p><p>Let’s keep the simplicity also when choosing a clustering algorithm.</p><h4>k-means</h4><blockquote>k-means takes almost no configuring. Almost.</blockquote><p>Yes. k-means assumes as input the number of clusters, <em>k</em>.</p><blockquote>But we have no idea.</blockquote><p>Let’s use <a href="https://orange.biolab.si/">Orange3</a> to help us with this.</p><p>When you open the k-means widget, Orange3 allows us to select a range of values for <em>k, </em>and tests them to check the silhouette scores we can get from these different values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*bk8BJS55SzqCKzmqCk-y-w.png" /><figcaption>k-means widget from Orange3</figcaption></figure><p>As expected, the best clustering we can do is isolating SP from the rest of the data.</p><blockquote>This shows how k-means can be sensitive to outliers, but that’s not the focus of this post.</blockquote><p>The second best <em>k</em> value is 4. Let’s check the silhouette plot for that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*rGpoTMZKA76ichO23iUF_w.png" /><figcaption>Silhoutte plot for k-means with four clusters.</figcaption></figure><p>In a silhouette plot, if an observation is more similar to its cluster neighbors than to the observations in another cluster, it will be scored a positive value (1.0 being the maximum).</p><p>Let’s check how that clustering looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*TX1kbNWh3s2fPHhzt1IbgQ.png" /><figcaption>Scatter plot for k-means with four clusters. In this plot, São Paulo is the clear outlier.</figcaption></figure><blockquote>Hmm.. it’s good, but not perfect.</blockquote><p>Yes, that sometimes happens to k-means.</p><p>The score that Orange3 shows is the mean over 10 runs, but a single run may not be that fit.</p><h4>Hierarchical clustering</h4><p>Another simple approach to clustering is called hierarchical, but it takes a bit more work.</p><p>The key takeaway is that one can navigate the different possibilities for cluster numbers using an interactive dendogram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*OGRhwHRfOH60eog3ZJRwlQ.png" /><figcaption>Hierarchical clustering widget using Euclidean distance and Ward’s linkage</figcaption></figure><blockquote>OMG, so many options on the left panel <em>😱</em></blockquote><p>Yes, it is a very <strong>configurable </strong>algorithm, and you can Google most of these options.</p><p>I just want to show that, with six clusters, we get the following silhouette plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*eZkhqXfO72qYCM5yBEFLOA.png" /><figcaption>Silhouette plot for hierarchical clustering with six clusters using Euclidean distance and Ward’s linkage</figcaption></figure><p>Notice how the clusters have changed, and that’s not only because of the change in the number of clusters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tkH4v5DNQ7PLdD-HqXCezw.png" /></figure><p>From the plot above, we can see that the clusters are split based mostly on their distance to the origin.</p><p>Both k-means and hierarchical clustering have attempted to do that because they are considering that two states are similar if they are close in this bidimensional space.</p><blockquote>That’s <strong>Euclidean distance</strong>!</blockquote><p>Exactly.</p><h3>Metrics: changing your perspective entirely</h3><p>Now, very early in our investigation, my students and I started noticing that the ratio between these two attributes is an attribute on its own.</p><blockquote>GDP per capita!</blockquote><p>Yes, and that’s a very interesting attribute to state how rich a state truly is.</p><blockquote>So we could create that third attribute by combining the first two<em> 🙃</em></blockquote><p>Yes, but no. We generally avoid creating new attributes if we can already grasp that information from the existing ones.</p><blockquote>Why? <em>🤔</em></blockquote><p>Because then our plots would become 3D, for starters, not to mention many other issues (and we’re talking about a very simplified dataset)..</p><blockquote>Wait, how can we grasp that info from this plot?</blockquote><p>We’ll take as reference the diagonal of the plot.</p><p>States that are to the right of this diagonal have a shift towards larger population.</p><p>The opposite is also true: states that are to the left of this diagonal have a shift towards larger GDP.</p><blockquote>Still, how do we tell the algorithms to look for this without a third attribute?</blockquote><p>We change the similarity metric: cosine distance renders two states as similar if the angle between the vectors they represent is small.</p><blockquote>Wait, WHAT?</blockquote><p>Check the plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*b3UFr-Uu4RasBpVePMG9OA.png" /><figcaption>Scatter plot for hierarchical clustering using cosine distance and Ward’s linkage, with six clusters.</figcaption></figure><blockquote>I don’t follow.</blockquote><p>Try this one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*EsU0f_i9t7PH_-sKy8Pk1A.png" /><figcaption>Regression lines illustrating that the cosine distance values the angle with regard to the origin.</figcaption></figure><blockquote>FUCKE! <em>😱</em></blockquote><p>Yesse.</p><h3>Wrapping it up: how Brazil is truly divided</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*oTdoBYAxemSoW5MMH7k4DA.png" /></figure><p>One of the most interesting things about this last clustering result is that it makes a lot of sense when we analyze it:</p><ul><li>DF is the most significant outlier in Brazil when it comes to richness, and that’s absolutely crazy if you consider that the business in DF is just politics!</li><li>The southeast of Brazil is generally deemed as the country’s richest region, but with a few exceptions, the south and the midwest top it!</li><li>The north and northeast in Brazil are, by far, in the most critical situation of the country.</li></ul><p>Now let’s take this as exercise: picture another Brazil, where politics are not for enrichment and businesses that are piling up in SP spread across the north and northeast.</p><blockquote>That’s a plot I’d love to see <em>😆</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b439f31c56b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/clustering-in-detail-b439f31c56b7">Clustering in detail</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python, R, Messi, and Cristiano]]></title>
            <link>https://medium.com/@leobezerra_90682/python-r-messi-and-cristiano-d2b5278dbd5a?source=rss-fff777145cdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2b5278dbd5a</guid>
            <dc:creator><![CDATA[Leonardo Bezerra]]></dc:creator>
            <pubDate>Wed, 02 Oct 2019 00:54:59 GMT</pubDate>
            <atom:updated>2019-10-02T00:54:59.214Z</atom:updated>
            <content:encoded><![CDATA[<p>One very disputed topic in data science forums pops up every now and then:</p><blockquote>Python or R?</blockquote><p>I’ve been programming on both for several years now and every time I see this question it reminds me of a similar disputed topic involving football:</p><blockquote>Messi or Cristiano?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*Sb6uHEggK_S--bjsa1EDaA.jpeg" /></figure><p>As a football fan, the answer is that you’re comparing apples and bananas.</p><p>Even if both are top scorers and have won many titles individually and collectively, that’s the only thing they have in common.</p><blockquote>But Python and R are both programming languages and are used for data science..</blockquote><p>And that’s the only thing they have in common.</p><p>So let me try to talk about each one separately, so we can put this in perspective.</p><h3>Python</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*t6RHddJbjC601sKdLpzDtg.png" /></figure><p>Python started as a procedural, high-level scripting language and later evolved into a multi-paradigm programming language that excels for web applications, testing automation, data science, and, obviously, scripting.</p><p>The most interesting things about Python are its community and its simplicity.</p><p>For data science, that basically means you can work magic with little or no programming background, as long as you’re using high-level libraries such as Pandas.</p><p>The Python data science community also offers full-fledged toolkits such as Orange3 and Anaconda, besides programming frameworks like scikit-learn, TensorFlow, Keras, PyTorch,…</p><p>More importantly, when you do start diving into the language, you’ll be able to use Python for a number of other applications, if you’re ever interested at that.</p><p>The downsides for people who do not come from a programming background are probably the 0-based indexing and chaining, that may seem a bit strange at first.</p><h3>R</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*2tlPWp5W0NZPbNDY7T6Cqg.jpeg" /></figure><p>R was proposed as a programming language for statistical purposes, and that’s its main application domain (comment about others if you know any, actually 🤔)</p><p>R follows a different approach to programming than Python, as it was designed as a functional language and uses 1-based indexing.</p><p>Support for data science is somewhat native to the language, since dataframes and many statistical tests, metrics, and distributions are available as part of the language core.</p><p>In addition, R formulae are a very nice touch if you get familiar with the notation.</p><p>The most famous R toolkit is Rstudio and its most relevant data science library is ggplot2, for visualization.</p><p>The downside for people who do not come from a programming background is probably dealing with lists, which are quite a mess in R.</p><h3>Apples and bananas compared</h3><p>Wrapping the main characteristics of both languages, we have:</p><ul><li>Python is a high-level multi-paradigm language, whereas R is a functional language.</li><li>Python has a range of applications, whereas R is basically applied to statistics and data science.</li><li>Python offers data science tools and libraries mostly through its community, whereas R brings many of these features natively.</li><li>Programmers are quite fond of Python, whereas people who are fond of R are often non-programmers.</li><li>Python makes data wrangling and machine learning quite easy, whereas R makes hypothesis testing easy.</li></ul><h3>My personal take</h3><p>Much as in football, I don’t have second thoughts on what language I prefer.</p><p>I spent nearly 6 years programming in R and have spent nearly the same time programming in Python by now.</p><p>Hands down, my life would have been way easier if instead of facing my Masters and PhD in R, I had done it in Python.</p><p>More so, by now I would be even more experienced in Python, specially for data science stuff which I never dove into because they were not easy to reach in R.</p><p>Here are some takeaways from my experience:</p><ul><li>Even if Python is imperative (being procedural) and R is declarative (being functional), codes are way simpler and more readable in Python (ironically, some friends who argue for R show me their codes and they’re always procedural..)</li><li>Although counterintuitive, features from a language that are community-driven evolve much faster than features that belong to its core. This is likely the explanation why dataframes (and data wrangling in general) are so much easier and powerful in Python than in R (and will probably be for a while, for that very reason).</li><li>R is a niche language and that has a certain appeal at first, but chances are you’ll want to grow beyond it eventually. Python, on the other hand, is fit for so many applications that by the time you want to switch to development (or even production), you’ll notice you’re already much more prepared for it than you even noticed.</li><li>The amount of time you spend reshaping your data in R due to the lack of pattern in its libraries is impressive (and awful). In Python, Pandas has set a standard that all of the most relevant libraries follow, and if you do need to reshape, Python is meant to be simple 🙃</li></ul><h3>Wrapping it up</h3><p>I’m quite proud of myself for writing such a light-toned post given my Regrets.</p><p>To speak from the heart without being disrespectful, let me switch back to football.</p><blockquote>I was 30 and I was coaching Ronaldo, not this one [Cristiano], the real one, the Brazilian Ronaldo</blockquote><p>Cristiano has the fame, the numbers, and the titles, but truth is the tactics always were in his favor (he is the winger who’s almost always facing a single defender).</p><p>Messi is the guy who will get the ball surrounded by four defenders and still score.</p><p>In my opinion, more than a matter of apples and bananas, you just can’t compare Messi and Cristiano because they are not on the same shelf 😉</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*oTcf-nbfHVoHlEm_S_6QLA.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2b5278dbd5a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tabs, commas, and dataframes]]></title>
            <link>https://medium.com/@leobezerra_90682/tabs-commas-and-dataframes-187adf928654?source=rss-fff777145cdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/187adf928654</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Leonardo Bezerra]]></dc:creator>
            <pubDate>Thu, 26 Sep 2019 18:51:34 GMT</pubDate>
            <atom:updated>2019-09-26T18:51:34.087Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Tabular</em> is how we generally refer to data that is organized as a table.</p><blockquote>You mean like in a spreadsheet?<em> 🤔</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*YepDIsVngyTpf9TqvghZ1g.png" /><figcaption><a href="https://www.kaggle.com/dgomonov/new-york-city-airbnb-open-data">New York City Airbnb Open Data</a>, containing Airbnb listings and metrics in NYC, NY, USA (2019).</figcaption></figure><p>A spreadsheet is the most popular way to handle tabular data, so yes 🙃</p><p>Rows are generally referred to as <em>entries, registries, observations, tuples, records, …</em></p><blockquote>Let’s go with observations<em> 👍</em></blockquote><p>Ok.</p><p>Columns are generally referred to as <em>features, attributes, treatments, …</em></p><blockquote>Features <em>🤜</em></blockquote><p>Nice.</p><blockquote>So a <strong>value</strong> in a given <strong>cell</strong> is the actual <strong>data</strong> we have about a given <strong>feature</strong> of a given <strong>observation</strong><em>.</em></blockquote><p>You’re getting it 😁</p><h3>Tabs and commas</h3><blockquote>You’re being repetitive <em>😒</em></blockquote><p>So let me fast-forward to the meaning of this.</p><p>Tabular data is usually stored as a spreadsheet or as a text file.</p><blockquote>I’m quite fond of Google Spreadsheets <em>🙃</em></blockquote><p>It’s really nice, but there are other spreadsheet applications and formats out there too:</p><ul><li><strong>Spreadsheet</strong>: ODS (OpenOffice Calc), XLSX (MS Office Excel).</li></ul><blockquote>Wait, how can you fit tabular data into a text file?</blockquote><p>Actually, even spreadsheet formats are text files under the hood.</p><p>All it takes is a special organization of the files and apps that know how to handle them.</p><blockquote>That’s why proprietary formats like XLSX are not recommended…</blockquote><p>Right on 👍</p><p>There are other text file organization classes that are very popular too:</p><ul><li><strong>Delimiter-separated</strong>: each observation fits a line and values are separated by a text delimiter, like comma (CSV) or tab (TSV).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E-86IYOZCmPWW9NyxlZbLA.png" /><figcaption>CSV representation of that same tabular data</figcaption></figure><ul><li><strong>Object serialization</strong>: much more complex organization, so let’s just list the most common: JSON, XML, YAML, …</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d-Q-AT7zP_GNymu3aMxmXQ.png" /><figcaption>JSON representation of the same tabular data</figcaption></figure><blockquote>Wait, but how can I use those if I don’t understand them?</blockquote><p>You didn’t know a spreadsheet was a text file, but the apps you used did 😏</p><blockquote><em>😅</em></blockquote><p>Btw, those apps can handle text files too.</p><blockquote>What? I can open these XSV in Spreadsheets?</blockquote><p>You can both <em>import</em> and <em>export </em><strong>CSV</strong> and <strong>TSV </strong>files in most spreadsheet apps, yes.</p><p>Also, most data science applications will deal with all the formats I just suggested.</p><p>The <em>dataset</em> you load will be shown as a <em>data table</em> or a <em>dataframe</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a2s2Pg89HJi7c_ko_jL7Fw.png" /><figcaption>The same tabular data viewed as a data table in <a href="https://orange.biolab.si/">Orange3</a>.</figcaption></figure><h3>Data sets, tables, and frames</h3><blockquote>Wait, what?</blockquote><p>Any set of data used for analysis is referred to as a…</p><blockquote>Dataset.</blockquote><p>When an app shows you a table containing the contents of a dataset you just loaded…</p><blockquote>You have a data table.</blockquote><p>And when someone puts your data on a picture frame..</p><blockquote>?????</blockquote><p>Just kidding 🤣🤣🤣</p><p>Dataframe is a computer science container specially designed for data manipulation.</p><p>It’s something that comes in handy when you’re doing command-line analysis, like in Google Colaboratory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yg9kqyROCMfSTZQVp4MkCQ.png" /></figure><h3>Try it out</h3><p><a href="https://www.kaggle.com/">Kaggle</a> is a great place to start exploring a few tabular datasets.</p><p>Try loading some of their data into Spreadsheets, Orange3, or Colab.</p><p>Also, if you didn’t know, this post is part of a <a href="https://github.com/leobezerra/ds-zero">GitHub repo</a> for people starting to study data science.</p><blockquote>Stay tuned for another repo that will be coming out shortly on how to use Pandas when you’re not a programmer<em> 😬😁</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=187adf928654" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The first day at DS101]]></title>
            <link>https://medium.com/@leobezerra_90682/the-first-day-at-ds101-b905a14da73?source=rss-fff777145cdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/b905a14da73</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Leonardo Bezerra]]></dc:creator>
            <pubDate>Wed, 07 Aug 2019 11:46:13 GMT</pubDate>
            <atom:updated>2019-10-29T11:39:38.348Z</atom:updated>
            <content:encoded><![CDATA[<h3>The first day at Data Science 101</h3><blockquote>Or &quot;how I got to know my students&quot;</blockquote><p>It was first day of class at DS101 and I had to come up with an idea to find out who my students were, since the plan was to follow a problem-based learning approach and group them in a reasonable way.</p><blockquote>&quot;Forms will guide you home&quot;, Coldplay</blockquote><p>What started as a simple poll to get to know their background (meaning knowing if they had some Python skills) became a super nice hands-on experience that I want to share here, if anyone is willing to repeat it.</p><h3>Data science for what it is</h3><p>There is much buzz about data science these days, but if you boil it down to these two questions, you’ll probably notice what’s really behind it:</p><blockquote>Is there science without data? Can we learn from data without science?</blockquote><p>All in all, data science is yet another pop term that has been helpful to get people who once were scared of statistics and science to dive into it.</p><p>More importantly, it is helping people learn that science can be fun, and visual, and surprising, and.. all those things before a Ph.D. or a data monkey job 🤣</p><h3>The data science process</h3><blockquote>&quot;And it goes like this&quot;, Maroon 5</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hFd7GRcynHpYZNjTWP1-Og.jpeg" /></figure><p>This chart is the best data science breakdown I have come across so far.</p><p>I went through each of these steps with my students on those first days of class, all the way to this post where I’m sharing my findings with you.</p><p>So let’s see it step-by-step:</p><h4>Ask an interesting question</h4><p>That’s easy because it’s the title of this post: &quot;who are my students?&quot;</p><p>Since that’s pretty broad, I narrowed it down to three aspects:</p><ul><li>Academic background</li><li>Time availability</li><li>Programming background</li></ul><p>As it often happens in practice, this breakdown only occurred to me while structuring the next stage..</p><h4>Get the data</h4><p>Five minutes before the first class I was having breakfast and preparing this <a href="https://forms.gle/r9vSj3XpDxSZpQhx6">Google Forms</a> at the same time.</p><blockquote>&quot;That is the only reasonable explanation why anyone would forget Go&quot;, Explaining the unforgivable</blockquote><p>All in all, Forms did a very good job and fifteen minutes after the beginning of the first class, I had answers from all 30 students that bothered to show up.</p><p>Next stage was understanding the data I had just collected..</p><h4>Explore the data</h4><p>Forms is also very good in summarizing results for a quick check before you go to a full-fledged exploration tool.</p><p>A quick discussion with my students already revealed that one of the questions had been poorly formulated:</p><blockquote>&quot;How many hours are you spending on lectures this semester (x 30h)?</blockquote><p>Since the answer to that question was controlled by a 1–10 slider, the maximum amount of hours students could report was below what many of them was actually spending.</p><p>Still, changing that question and asking students to update their answers would cost me 10 minutes I could not afford, so I let this one slide (pun intended)..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*DJYSyzlxewF_9x0QoZrQYg.png" /></figure><p>And on I went to my first hands-on experience with <a href="https://orange.biolab.si/">Orange3</a>, a data science workflow tool I had only met when binge watching their <a href="https://www.youtube.com/playlist?list=PLmNPvQr9Tf-ZSDLwOzxpvY-HrE0yv-8Fy">Getting Started </a>YouTube series.</p><p>Since I had already discussed distributions with the class using Forms, I went straight to a clustering analysis. I needed to form 8 groups with 4 people each:</p><ul><li>Idea #1: I could use a similarity measure to find 4 different profiles among my students.</li><li>Idea #2: I could use a dissimilarity measure to find 8 groups formed by the most different of people.</li></ul><p>Since I wanted to use only the tools available in Orange3, I went on with the first idea, but guess what?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*bl2I3R0h-ncGZ1l4ZtESPA.png" /></figure><blockquote>&quot;Do not try and <strong>bend the spoon&quot;</strong>, Spoon boy</blockquote><p>The best clustering setup I could find only had three profiles, but luckily one of them was double the size of the others.</p><p>The composition of each group then became:</p><ul><li>1 student with Profile #1</li><li>2 students with Profile #2</li><li>1 student with Profile #3</li></ul><p>But still, what the hell were those profiles? I figured it out checking feature distributions grouped by cluster:</p><ul><li>Profile #1: few lecture hours, many job hours</li><li>Profile #2: many lecture hours</li><li>Profile #3: not so few/many lecture hours, some research hours</li></ul><p>At this point, I think it’s important to say that my lack of experience in Orange3 and my deep faith in Python possibly explain this outcome.</p><ol><li>Since I did not know how to feature-wise customize a similarity measure, only numerical features were considered.</li><li>Since Python is really easy to learn, most relevant DS tools are available in Python, and I wanted to have an IT-agnostic course anyway (hence Orange3), I did not include programming background answers in the analysis.</li></ol><h4>Model the data</h4><p>At this point, groups were formed by randomly sampling the amount of students from each profile.</p><p>An issue popped up the following week, when 12 new students were admitted to the course.</p><p>I needed to fill up the groups trying to respect this profiling, and two ideas came up (again):</p><ul><li>Idea #1: use an online clustering adaptation, but this could render changes to the clustering previously assigned</li><li>Idea #2: render this a classification problem where the previously assigned profiles would be used as labels</li></ul><blockquote>&quot;Could you stop saying &#39;number one, number two&#39;? It makes me wanna go to the bathroom&quot;, Billy</blockquote><p>I tested a few models and hit the jackpot, with 100% accuracy using logistic regression.</p><p>From there on, it was all about loading the data for the new students, predicting their profiles and randomly assigning them to groups.</p><blockquote>Wait, what?</blockquote><p>Yes, random assignments, since anyway the groups would lose the original balance.</p><p>Yet, knowing the profile of the extra student assigned to each group, I can track down what happens along the way with groups that had a bit more of a given profile than others.</p><h3>It’s a wrap</h3><p>Well, this was my experience and I wanted to share it with you.</p><p>I could have dived into the deep details of every stone turned, but that would have been an even longer post.</p><p>Hope you found this idea interesting and are willing to try it in your own context.</p><p>Let me know and maybe I can help you with the bits I didn’t share here.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b905a14da73" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>