<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kwyk on Medium]]></title>
        <description><![CDATA[Stories by Kwyk on Medium]]></description>
        <link>https://medium.com/@kwykedu?source=rss-1354d75818ea------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*t8Fki7SL2LXv6z4g75MihA.jpeg</url>
            <title>Stories by Kwyk on Medium</title>
            <link>https://medium.com/@kwykedu?source=rss-1354d75818ea------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:46:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kwykedu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A third of your math work is useless]]></title>
            <link>https://medium.com/@kwykedu/a-third-of-your-math-work-is-useless-a990245aad77?source=rss-1354d75818ea------2</link>
            <guid isPermaLink="false">https://medium.com/p/a990245aad77</guid>
            <category><![CDATA[education]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kwyk]]></dc:creator>
            <pubDate>Fri, 13 Apr 2018 13:02:06 GMT</pubDate>
            <atom:updated>2018-04-13T13:03:59.404Z</atom:updated>
            <content:encoded><![CDATA[<p>At Kwyk, we use AI to predict student’s performance, especially in maths. Our most recent results show that on <strong>one-third of the exercises</strong> given by the teacher <strong>we know beforehand that the student will succeed</strong>, raising the question of its usefulness. Despite the clickbait title <em>(sorry for that)</em>, the answer might not be as simple as you would expect. We are going to attempt to explain that <strong>the biggest threat in AI is not AI in itself, but our human over-interpretation of its results.</strong></p><h3>Introduction</h3><p>In this article, we will analyze prediction data done on 1 million maths exercises done between March 7th, 2018 and March, 23rd 2018 on <a href="https://www.kwyk.fr/">Kwyk</a>. <strong>The predictions were done in real time. </strong>The training was done on a 5 Million exercises dataset from September and October 2016 results.</p><p>Kwyk is an online learning platform where students do their homework online. The students are age 11 to 18 and are assigned homework by teachers on the platform. Each individual has a different homework generated automatically, but all questions are equivalent in terms of difficulty : 1/3 + 1/2 is roughly the same as 2/3 + 1/4 for instance. Because homework is generated, students can retry anytime on a new version of the homework to get the best grade. They can also work freely on any given chapter at any times. To be fair, 90% of the activity is work assigned by teachers.</p><p>We also grade automatically every exercise, using <a href="https://en.wikipedia.org/wiki/Computer_algebra_system">formal analysis</a>. This is how we evaluate if a student succeeded an exercise or not. What we evaluate is the <strong>probability of success of a student on a specific exercise.</strong></p><h3>How we got our results</h3><p>The results were obtained using an AdaBoost regression, on manually crafted features, which included various averages, including the student’s average on this particular chapter for instance. We also used automatically crafted features, which are sort of the map of maths, which we explain in detail <a href="https://towardsdatascience.com/a-non-nlp-application-of-word2vec-c637e35d3668">here</a>. In total there are 10 manual features and 50 automatic features.</p><h3>Results summary</h3><p>Now the results. First, we get an AUC score of 0.89. Unfortunately, we don’t have an expert reference (different teachers estimating success likelihood of students). Not having comparison points makes it really hard to evaluate if this score is interesting or not. We can only compare to previous versions of our algorithms, which started with AUC 0.72 so we definitely improved our assessment of students, you can read our other articles for details on our algorithms and how we assess them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*TgLWCHtylLhzm67eSf5tNA.png" /><figcaption>Area Under Curve of our predictions (Prediction data was evaluated in real time)</figcaption></figure><p>Out of the 1 Million exercises we assessed, we estimated on 300k exercises that the student would succeed over 95% of the times. They succeeded on average over 96% of the time. In other words,<strong> </strong>our algorithm was able to predict that you knew how to do this exercise a third of the time.<strong> If this is so, was it really necessary for you to do it ? </strong>The temptation to answer no is exactly the problem that AI poses to us, humans.</p><p>On the other end of the spectrum, we estimated on 100k exercises that the student would succeed less than 5% of the times. In reality, they succeeded 10% of the times. So our algorithm is less accurate on low probabilities. This is expected as we don&#39;t control what happens between exercises, so if a student asks for help or reviews his notes before answering, we have no way of capturing that in our data.</p><p>But in the end, it means that we roughly know a student’s result in advance on 40% of the questions asked by his teacher.</p><p>Overall you can see how precise is our algorithm on the following graph</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*D-XygsbMw1YN9ut0rwdAHA.png" /><figcaption>Each dot is a group of predictions. The closer the dots are to the blue line, the better the prediction was for that group. The two big dots at each end are the one mentioned in this article.</figcaption></figure><blockquote>With AI we know a student&#39;s result in advance on 40% of the questions asked by his teacher.</blockquote><h3>Discussion</h3><p>So 40% of the time we know in advance with a small margin of error if a student is going to succeed at a specific exercise. Why is it that we should NOT conclude that it was a waste of time. There are two types of reasons for this.</p><p><strong>The first type of reasons is over-interpretation of the results.</strong> The algorithm used tried to assess the likelihood of success of a given student. It was NOT to assess if it was necessary for this student to answer this question (which by the way is a much more difficult question to ask an algorithm). AI works by using correlations. So imagine a new student being transferred to an excellent class that never makes any mistake. An algorithm will most likely categorize this student as being very likely to succeed. This new student will be categorized as &quot;no need to work&quot; even if he did not do any exercise yet. This is obviously wrong.</p><p>The error is not in the algorithm, as most students that get transferred into excellent classes will be excellent students, the error is in us humans, that over-interpret what the algorithm says. The likelihood of success is not the proven mastery of a given subject, yet we humans will tend to think that both are the same because we estimate the likelihood of success with our estimate of a student&#39;s mastery. AI does not work that way. Not understanding this is really the core of the problem with AI. Because we cannot ask our algorithms questions like humans, we need to find proxy questions that we hope will be close to our real questions. In our case the real question we would like to ask is &quot;What does this student know&quot;, and the proxy we found is &quot;How likely is he to succeed at this task&quot;.</p><blockquote>AI forces us to change our real world vague questions into extremely specific proxy questions. The human confusion between the two questions is the biggest threat to integrating AI into our products.</blockquote><p>The same problem is found over and over. Facebook question should be &quot;How to maximize our users&#39; happiness&quot;, and the proxy they use is &quot;How to maximize view time on our site&quot;. Google&#39;s question is &quot;What is the answer to this user&#39;s query&quot; and its proxy is &quot;What is the link with the lowest bounce rate on this user&#39;s query ?&quot;. Understanding that there is a fundamental gap between the question we would like to ask our algorithm and the actual proxy we use is extremely important if we want to avoid errors in the future.</p><p><strong>The second type is related to neuroscience and is specific to this problem</strong>. Suppose now that our proxy is perfectly valid to estimate a student&#39;s mastery. Our prediction still does not take into account our human brains. We forget all the time. In order to memorize something, we need to recall it many times. <a href="https://en.wikipedia.org/wiki/Spaced_repetition">Spaced repetition</a> is an example of how to take this into account. So even if you already know something, if you want to fix it in your memory you need to do it over and over at regular intervals. Once again, even if our algorithm knows that you know something in advance, it might be in your best interest to do the exercise anyway to fix the knowledge in your brain.</p><p>In addition to <em>memory</em>, the <em>student&#39;s emotional state </em>is another variable that we might want to take into account when we look at the bigger picture of education.</p><h3>How to prevent this over-interpretation</h3><p>Exactly like the <a href="https://en.wikipedia.org/wiki/Simpson%27s_paradox">Simpson&#39;s paradox,</a> there is no silver bullet here. The main way to prevent over-interpreting is to be extremely conscious of the problem. We, as practitioners, need to be aware of the questions we asked our algorithm and to be extremely careful not to over-interpret the answers they give us. We also need to alert everyone else to the types of conclusions that should not be drawn from our results.</p><p>We can also multiply the different questions to ask our algorithms. If many specific algorithms seem to converge on their answers, maybe it is a hint that it could be a good answer to our human questions.</p><p>If only we could add dropout to our interpretations…</p><h3>Further work</h3><p>At Kwyk we have many goals in order to improve overall student&#39;s success. In addition to ever improving our student&#39;s success estimate, we need to try and address the problems evoked in this article.</p><p>For instance, we are looking to take into account the sequencing of personalized sequences (the second type of problems). For this we still cannot ask the human questions, we need to find a proxy question. The proxy question will be &quot;What is the best strategy of homework to give a specific student to maximize his future likelihood of success on all exercises ?&quot;. For this, we will use RL approaches. Of course, we will necessarily encounter problems along the road but we hope we can come closer and closer to answering our real question which is &quot;How do we maximize each student&#39;s success ?&quot;.</p><p>Author: Nicolas Patry</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a990245aad77" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-state LSTMs for categorical features]]></title>
            <link>https://medium.com/data-science/multi-state-lstms-for-categorical-features-66cc974df1dc?source=rss-1354d75818ea------2</link>
            <guid isPermaLink="false">https://medium.com/p/66cc974df1dc</guid>
            <category><![CDATA[education]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Kwyk]]></dc:creator>
            <pubDate>Mon, 07 Aug 2017 15:06:07 GMT</pubDate>
            <atom:updated>2017-08-07T15:11:55.879Z</atom:updated>
            <content:encoded><![CDATA[<h3>Context</h3><p><strong>Neural Networks</strong> are now widely used in many ways. From image caption generation to breast<strong> </strong>cancer<strong> </strong>prediction, this great diversity of applications is a natural consequence of the important variety of <strong>neural architectures </strong>(<strong>Feed Forward </strong>Neural Networks, <strong>Convolutional </strong>Neural Networks, etc…). Among all these architectures, <strong>Long Short Term Memory </strong>(<strong>LSTM</strong>) — a particular case of Recurrent Neural Networks — have proven very successful on tasks such as <a href="https://www.tensorflow.org/tutorials/seq2seq">machine translation</a>, <a href="http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/">time series prediction</a> or generally anything where the data is <strong>sequential</strong>. This is mainly due to their ability to memorize relatively <strong>long term dependencies</strong>, which is achieved by taking into account previous information for further predictions.</p><blockquote>But LSTMs alone aren’t always enough. Sometimes we need to tweak these layers and adapt them to the task at hand.</blockquote><p>At <a href="https://www.kwyk.fr/"><strong>Kwyk</strong></a>, we provide <strong>online math exercises</strong>.<strong> </strong>And every time an exercise is answered, we collect some data which is then used to tailor homework for each student using the website. In order to determine which exercises are the most likely to make a student progress, we need to know how likely he/she is to succeed on each exercise at any given point in time. And by being able to correctly predict these <strong>probabilities of success</strong> we can choose the homework that <strong>maximizes the overall progress</strong>.</p><p>This post aims to introduce our modified LSTM cell, the “<strong>Multi-state LSTM</strong>”, which we based our model on to try and solve this problem.</p><h3>A quick data overview</h3><p>When an exercise is answered, we save information about both the student (or “user”) and the exercise, along with an additional <strong>score </strong>value<strong> </strong>that is either (0) or (1) depending the user’s success. The information we collect is in the form of <strong>categorical features</strong> that tell us : “<strong>which exercise is it ?</strong>”, “<strong>what chapter is it ?</strong>”,<strong> </strong>“<strong>what student is it ?</strong>”,<strong> </strong>“<strong>from which grade is he/she ?</strong>” … This results in features that have from tens to thousands of modalities, which we use to predict the “score” variable and get <strong>success probabilities</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*z0A4Qbm1KAeZiRdaBjeW-A.png" /><figcaption>A sample of our data</figcaption></figure><h3>LSTM-based model</h3><p>There are two main motivations behind the choice of an LSTM-based model.</p><p>The first one comes from the fact that all the features we’re using are categorical. That’s the <strong>real source of information</strong> we want to learn from. And by manually cooking features such as moving averages of previous scores, we introduce <strong>biases</strong> due to the subjective choices we make. Therefore, we chose to rely on <strong>neural networks, </strong>directly feed our data and let the features form <strong>automatically</strong> and “<strong>objectively</strong>”.</p><p>The second motivation concerns the choice of <strong>LSTM</strong>s among all available architectures. This is simply due to our belief that for each student the <strong>probability of success </strong>on each exercise <em>depends on all previous results in time</em> and therefore is <strong>sequential</strong>.</p><blockquote>With this in mind, we thought of an architecture where for each categorical feature, a shared LSTM would keep a history of previous results for each modality.</blockquote><p>Let’s take the “<strong>user_id</strong>” feature for example. In this case, we need to adapt an LSTM cell to memorize <strong>on the fly but independently for each student</strong> a history of how he/she was successful in the past. This is done by adding what we call a “<strong>Multi-state</strong>” to the basic LSTM cell :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ad9H0gBVw12dIZ2nEB6Lxw.png" /><figcaption><strong>Left</strong> : Basic LSTM cell / <strong>Right</strong> : Multi-state LSTM cell. Note : If you are not familiar with LSTMs you can refer to <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">this great post by Christopher Olah</a>.</figcaption></figure><p>Every time a student’s label is fed to the Multi-state LSTM cell, it starts by looking up the corresponding <strong>state</strong> and <strong>previous score</strong>. Then, the student’s state is sent to <strong>update </strong>a <strong>shared LSTM cell</strong>. In meantime, the previous score is fed to this LSTM cell which produces an <strong>output</strong>. Next,<strong> </strong>the resulting <strong>state</strong> goes back to <strong>update</strong> the student’s data in the Multi-state. Finally, once we observe the actual <strong>score</strong>, this value is in turn sent to <strong>update</strong> the student’s previous score in the Multi-state.</p><p>This modified version of the basic LSTM cell works similarly but has an additional advantage : it can now directly take in a sequence of student labels as inputs :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*oeCFtXIquaO4KlDoFWr_zg.png" /><figcaption>An unrolled Multi-state LSTM.</figcaption></figure><p>For the sake of our previous example we considered the “user_id” categorical feature, but this can actually be applied to all of our categorical features. In fact, using a Multi-state LSTM on the “<strong>exercise_id</strong>” feature for example would result in it learning <strong>historical success rates</strong> for each exercise. From here, the complete network we used is pretty straightforward :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*r9I7vzqh7CFsImARNxg71Q.png" /><figcaption>The complete graph we used to solve our problem.</figcaption></figure><p>All in all, this architecture tries to learn separate histories for each modality of each categorical feature then uses the LSTM’s outputs to predict a success probability for a given student on a given exercise.</p><h3><strong>Results</strong></h3><h4>Baseline</h4><p>To assess our LSTM-based model we need a <strong>baseline</strong>. Before trying any Deep Learning methods, we used to have a boosting algorithm (<a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn.ensemble.GradientBoostingClassifier"><strong>Gradient Boosting</strong></a>) that used some handmade features. From all the categorical features, we cooked up some <strong>fast and slow </strong><a href="https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average"><strong>moving averages</strong></a><strong> </strong>of previous scores per each modality of each feature. Then, we fed these features to a Gradient Boosting classifier and let the magic happen.</p><h4>Comparison</h4><p>Both classifiers have the same approach. In fact, the baseline model uses handmade moving averages as features and the LSTM-based method automatically learns these histories to predict the scores. Sure enough, both algorithms got similar Accuracy Scores (Baseline : 74.61%, LSTM-based network : <strong>75.34%</strong>).<strong> </strong>But the predicted distributions are very different :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fw4WfcywFabOsx-pJE-ymw.png" /><figcaption>We can see that the LSTM-based model is much better at segregating students.</figcaption></figure><p>In fact, the LSTM-based model seems to <em>separate 3 to 4 different populations </em>:</p><ul><li>One for students who shouldn’t be able to answer correctly (probability&lt;10%)</li><li>One for those who have a moderate to good chance to succeed (50&lt;probability&lt;80%). These can be students for whom we don’t have enough data, or simply good-average students.</li><li>One for those who have a very important chance of succeeding (probability&gt;90%).</li></ul><p>Conversely, the baseline only predicts a skewed normal distribution around the overall average success rate with no particular discrimination.</p><p>But to further investigate the difference between the two models, we need to have a more detailed look at the MSE. In fact, judging by the MSE alone, the two models have similar performance :</p><ul><li>Baseline : 0.17</li><li>LSTM-based network : <strong>0.16</strong></li></ul><p>But as we discussed in a <a href="https://medium.com/towards-data-science/metrics-random-processes-in-classification-fd5bafa79505">previous post</a>, when real randomness is associated with the targets (here, predicting human behavior) we get a better assessment of the performance of a classifier by looking at its <strong>reliability measure (REL)</strong>.</p><p>Reliability is computed by sorting the data into bins of similar predicted probabilities. Then, we compute an MSE for each bin against the average target in the bin. Finally, we take an overall average to get our REL value.</p><p>The metrics obtained for each model are :</p><ul><li>Baseline : 3.86 e-3</li><li>LSTM-based network : <strong>2.86 e-4</strong></li></ul><p>Indeed, we see that the LSTM-based network is more than <strong>10 times more precise </strong>than the gradient boosting baseline, which should explain the <strong>better segregation </strong>we observed when comparing both distributions.</p><h3>Conclusion</h3><p>To sum up, we’ve seen on a real life example how <strong>LSTM</strong>s can be effective for <strong>sequential data</strong> classification. But we also saw how our <strong>modified version</strong> of the LSTM — the “<strong>Multi-state LSTM</strong>” — was able to reach a greater performance <em>without any preprocessing or feature engineering at all</em>. In fact, this new LSTM cell can directly take in a sequence of labels as inputs, which means that it <strong>can be used categorical features only </strong>and still produce good results.</p><p>To <strong>further improve</strong> on this Multi-state LSTM, a next step would be to take into account the <strong>correlations between multiple labels</strong>. In fact, when predicting the performance of a student on a given pair of similar exercises, the predicted probabilities should be very similar. One way to try and ensure this behavior could be to integrate an <a href="https://medium.com/towards-data-science/a-non-nlp-application-of-word2vec-c637e35d3668">exercise embedding</a> into the network and let it <strong>learn the dependencies</strong>.</p><p><em>Author : </em><a href="https://www.linkedin.com/in/hichamelboukkouri"><em>Hicham EL BOUKKOURI</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66cc974df1dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/multi-state-lstms-for-categorical-features-66cc974df1dc">Multi-state LSTMs for categorical features</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A non-NLP application of Word2Vec]]></title>
            <link>https://medium.com/data-science/a-non-nlp-application-of-word2vec-c637e35d3668?source=rss-1354d75818ea------2</link>
            <guid isPermaLink="false">https://medium.com/p/c637e35d3668</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[education]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Kwyk]]></dc:creator>
            <pubDate>Mon, 31 Jul 2017 13:42:50 GMT</pubDate>
            <atom:updated>2017-07-31T20:11:07.594Z</atom:updated>
            <content:encoded><![CDATA[<p>When using <strong>Machine Learning</strong> to solve a problem, having the right <strong>data</strong> is crucial. Unfortunately, raw data is often “unclean” and <strong>unstructured</strong>. <a href="https://en.wikipedia.org/wiki/Natural_language_processing">Natural Language Processing</a> (<strong>NLP</strong>)<strong> </strong>practitioners are familiar with this issue as all of their data is <strong>textual</strong>. And because most of Machine Learning algorithms can’t accept raw <em>strings</em> as inputs, <strong>word embedding</strong> methods are used to transform the data before feeding it to a learning algorithm. But this is not the only scenario where textual data arises, it can also take the form of categorical features in standard non-NLP tasks. In fact, many of us struggle with the processing of these kinds of features, so are word embedding of any use in this case ?</p><p>This article aims to show how we were able to use <strong>Word2Vec </strong>(<a href="https://arxiv.org/pdf/1301.3781.pdf">2013, Mikolov et al.</a>), a word embedding technique, to convert a <strong>categorical feature</strong> with a <strong>high number of modalities</strong> into a smaller set of easier-to-use <strong>numerical features</strong>. These features were not only easier to use but also successfully <strong>learned relationships</strong> between the several modalities similar to how classic word embeddings do with language.</p><h3>Word2Vec</h3><blockquote>You shall know a word by the company it keeps (<a href="https://en.wikipedia.org/wiki/John_Rupert_Firth">Firth, J. R.</a> 1957:11)</blockquote><p>The above is exactly what Word2Vec seeks to do : it tries to determine the meaning of a word by analyzing its neighboring words (also called <strong>context</strong>). The algorithm exists in two flavors <strong>CBOW</strong> and <strong>Skip-Gram</strong>. Given a set of sentences (also called <strong>corpus</strong>) the model loops on the words of each sentence and either tries to use the current word of to predict its neighbors (its context), in which case the method is called “Skip-Gram”, or it uses each of these contexts to predict the current word, in which case the method is called “Continuous Bag Of Words” (CBOW). The limit on the number of words in each context is determined by a parameter called “<strong>window size</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*EJ3wL_gJs1gSdwjHM3OcHg.png" /><figcaption>Both Word2Vec architectures. The current word is w(t) and w(t-2)..w(t+2) are context words. (Mikolov et al. 2013)</figcaption></figure><p>So if we choose for example the Skip-Gram method, Word2Vec then consists of using a <strong>shallow neural network</strong>, i.e. a neural network of only one hidden layer, to learn the word embedding. The network first initializes randomly its weights then iteratively adapt these during training to minimize the error it makes when using words to predict their contexts. After a hopefully successful training, the word embedding for each word is obtained by multiplying the network’s <strong>weight matrix</strong> by the word’s <a href="https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science">one-hot vector</a>.</p><blockquote>Note : Besides allowing for a numerical representation of textual data, the resulting embedding also learn interesting <a href="http://www.aclweb.org/anthology/N13-1090"><strong>relationships between words</strong></a><strong> </strong>and can be used to answer questions such as : <em>king is to queen as father is to …?</em></blockquote><p>For more details on Word2Vec you can have a look at this <a href="https://www.youtube.com/watch?v=ERibwqs9p38">Stanford lecture</a> or this <a href="https://www.tensorflow.org/tutorials/word2vec">tutorial by Tensorflow</a>.</p><h3>Application</h3><p>At <a href="https://www.kwyk.fr/"><strong>Kwyk</strong></a><strong> </strong>we provide math exercises online. Teachers assign <strong>homework</strong> to their students and each time an exercise is done some <strong>data</strong> is stored. Then, we use the collected data to <strong>evaluate the students’ level </strong>and give them <strong>tailored</strong> review exercises to help them progress. For each exercise that is answered we store a list of identifiers that help us tell : <strong>what is the answered exercise ?</strong>, <strong>who is the student ? , what is the chapter ?</strong>… In addition to that, we store a <strong>score</strong> value that is either (0) or (1) depending on the student success. To evaluate the students’ levels we then simply have to predict this score value and get success probabilities from our classifier.</p><p>As you can see, a lot of our features are <strong>categorical</strong>. Usually, when the number of <strong>modalities</strong> is small enough, one can simply transform a categorical feature with (n) modalities into (n-1) dummy variables then use that for training. But when the number of modalities is in the many thousands — as it is the case for some of our features — relying on dummy variables becomes inefficient and impracticable.</p><p>In order to address this issue our idea is to use Word2Vec to <strong>transform categorical features</strong> into a relatively small number of usable <strong>continuous features</strong> using a little trick. To illustrate, let’s consider “exercise_id”, a categorical feature telling us which exercise was answered. In order to be able to use Word2Vec we have to provide a <strong>corpus</strong>, a set of sentences to feed to the algorithm. But the raw feature — a list of ids — isn’t a corpus <em>per se :</em> the order is completely <strong>random</strong> and closer ids don’t carry any <strong>information</strong> about their neighbors. <strong>Our trick</strong> consists of considering each homework given by a teacher as a “sentence”, a <strong>coherent</strong> list exercise ids. As a result, ids are <strong>naturally</strong> gathered by levels, chapters… and Word2Vec can start learning exercise embedding directly on that.</p><p>Indeed, thanks to these artificial sentences we were able to use Word2Vec and get beautiful results :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WeCKY4UDM8coNPNtqQIHzg.png" /><figcaption>Exercise embedding (3 main components of PCA) colored by level. 6e, 5e, 4e, 3e, 2e, 1e and tm are the french equivalents of the 6th, 7th, 8th, 9th, 10th, 11th and 12th grades in the US.</figcaption></figure><p>As we can see, the resulting embedding has a <strong>structure</strong>. In fact, the 3d-projected cloud of exercises is spiral-shaped with <strong>exercises of higher levels following directly those of previous levels</strong>. This means that the embedding successfully learned to <strong>distinguish</strong> exercises of different school levels and regrouped similar exercises together. But that’s not all, using a <a href="https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction#Locally-linear_embedding"><strong>non-linear dimensionality reduction</strong></a> technique we were able to reduce the whole embedding into a single real valued variable with the same characteristics. In other terms, we obtained an <strong>exercise complexity </strong>feature that is minimum for 6th grade exercises and <strong>grows</strong> as the exercises get more and <strong>more complex</strong> until it is maximum for 12th grade exercises.</p><p>Moreover, the embedding also learned <strong>relationships</strong> between exercises just like Mikolov’s did with English words :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*dVAI95VHymqfyQaWpVUtuQ.png" /></figure><p>The diagram above shows some examples of the relationships our embedding was able to learn. So if we were to ask the question <em>“an exercise of </em><strong><em>number addition</em></strong><em> is to an exercise of </em><strong><em>number subtraction</em></strong><em> as an exercise of </em><strong><em>time addition</em></strong><em> is to … ?” </em>the embedding gives us the answer “<em>an exercise of </em><strong><em>time subtraction</em></strong>”. Concretely, this means that if we take the difference embedding[Substract(Numbers)] - embedding[Add(Numbers)] and add it to the embedding of an exercise where students are asked to add time values (hours, minutes …) then the closest embedding is one of an exercise that consists of subtracting time values.</p><h3>Conclusion</h3><p>All in all, word embedding techniques are useful to transform textual data into real valued vectors which can then be plugged easily into a machine learning algorithm. Despite being principally used for NLP applications such as <em>machine translation</em>, we showed that these techniques also have their place for categorical feature processing by giving the example of a particular feature we use at Kwyk. But in order to be able to apply a technique such as Word2Vec, one has to build a corpus — i.e. a set of sentences where labels are arranged so that a context is implicitly created. In our example, we used homework given on the website to create “sentences” of exercises and learn an exercise embedding. As a result we were able to get new numeric features that successfully learned the relationships between exercises and are then more useful than the bunch of labels they originated from.</p><p><em>Credits go to Christophe Gabard, one of our developers at Kwyk, for having the idea of applying Word2Vec to process categorical features.</em></p><p><em>Author : </em><a href="https://www.linkedin.com/in/hichamelboukkouri"><em>Hicham EL BOUKKOURI</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c637e35d3668" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-non-nlp-application-of-word2vec-c637e35d3668">A non-NLP application of Word2Vec</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Metrics & Random Processes in Classification]]></title>
            <link>https://medium.com/data-science/metrics-random-processes-in-classification-fd5bafa79505?source=rss-1354d75818ea------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd5bafa79505</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[education]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mathematics]]></category>
            <dc:creator><![CDATA[Kwyk]]></dc:creator>
            <pubDate>Tue, 25 Jul 2017 09:31:36 GMT</pubDate>
            <atom:updated>2017-07-31T20:05:14.889Z</atom:updated>
            <content:encoded><![CDATA[<p>In the introduction of his book “Machine Learning”, <a href="http://www.cs.cmu.edu/afs/cs.cmu.edu/user/mitchell/ftp/mlbook.html">Prof. Tom Mitchell</a> defines <strong>learning</strong> as follows :</p><blockquote>A computer program is said to <strong>learn</strong> from experience <em>E</em> with respect to some class of tasks <em>T</em> and <strong>performance measure</strong> <em>P</em>, if its performance at tasks in <em>T</em>, as measured by <em>P</em>, improves with experience <em>E</em>.</blockquote><p>In fact, one can only speak of an algorithm learning when assessed through a predefined performance measure or <strong>metric</strong>. When doing a <strong>classification task</strong>, choosing the right metric can be even more important than tuning the parameters of your model. To illustrate this, we will review the <strong>most common metrics</strong> out there and exhibit their pros and cons before concentrating on the particular case of classification tasks with underlying <strong>random processes</strong> — where none of the usual classification metrics work — and propose an alternative.</p><h4>Accuracy</h4><p>The <strong>accuracy score</strong> (often written <strong>ACC</strong>) is the most straightforward metric one can think of. It is simply counting the number of times we predicted the right class over the total number of predictions. More formally this can be written as :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CSNTS5PT25-F9q_cbEa3Gw.png" /><figcaption>Notations : p_i is the i-th prediction, y_i the i-th target and N the sample size.</figcaption></figure><p>Just like any metric, this one has its own pros and cons. On the good side, it is very easy to understand and interpret as having an accuracy of 80% simply indicates having an 80% chance of predicting the right class on unseen data — provided of course our model doesn’t overfit and generalizes well. Nevertheless, when the class distribution in the dataset is <strong>unbalanced</strong>, e.g.<strong> </strong>when there are many more negative targets (0) than positives (1), the metric becomes useless. In fact, if 95% of the targets are negative then a very simple (and bad) model that always predicts (0) will achieve a 95% accuracy score despite being completely useless. In these cases — which are rather common — other metrics such as <strong>confusion matrices</strong> come in handy.</p><h4>Confusion Matrix</h4><p>A confusion matrix is an attempt to describe the performance of a classifier in more detail than the simple accuracy score. It can be represented as a <strong>count table</strong>, counting the number of positive and negative predicted values given that the real values are either positive or negative. For example, if we were to compute a confusion matrix for the bad classifier of earlier it would look like this :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*CaxyEhO1Cl0BtVn1oUwZcQ.png" /><figcaption>Notations : T/F for True/False, P/N for Positive/Negative</figcaption></figure><p>Here we chose a hypothetical sample size of 2000. We can see that the data is highly unbalanced with ((0+1900)/2000) = 95% negatives for only 5% positives, and that the classifier always predicts a negative value (the ‘Positive’ row of ‘Predicted Values’ is null). But despite allowing us to have a quick look at the prediction’s distribution, in this state, the confusion matrix doesn’t really give us much more insight. One other way to look at it is by <strong>dividing each column by its total</strong>, resulting in what is called a <strong>normalized confusion matrix </strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*n_kxxJptLBsemQSNL8dERQ.png" /><figcaption>Normalized confusion matrix. Notation : R for rate.</figcaption></figure><p>Thanks to this normalization each <strong>value on the diagonal</strong> becomes the <strong>empirical conditional probability</strong> of predicting the right class. When conditioned on the true class being positive this probability is called TruePositiveRate or <strong>Sensitivity, </strong>otherwise it is called TrueNegativeRate or <strong>Specificity</strong>. Because these rates don’t depend on the class distribution using a confusion matrix is often preferable to relying on the accuracy score. During training, one can aim to maximize both the sensitivity and specificity to converge to the <em>perfect classifier</em> — whose confusion matrix is equal to the identity as it only predicts a class when it actually is the right class. In our example we immediately see that something is wrong as the sensitivity is zero and therefore we can conclude that our meant-to-be-bad classifier is indeed bad (hallelujah!).</p><h4>ROC / AUC</h4><p>Generally speaking, in order to classify a sample a classifier computes a <strong>real value </strong>between zero and one — or any other interval that can be mapped to [0,1] — then maps this value by using an internal <strong>discrimination</strong> <strong>threshold</strong> (often set to 50% by default) above which it predicts a positive and inversely. The choice of such a threshold can hugely impact the results displayed by the previous metrics. In fact, if our model is highly biaised towards negative values, i.e if it predicts values that are distributed very close to zero, using a threshold at 50% or higher may result in it always predicting a negative, while maybe choosing a lower threshold could result in more positives and therefore maybe allow for a better reflection of the classifiers’ performance.</p><p>One way to take into account all possible threshold values is to rely on <strong>ROC curves</strong> and <strong>AUC scores</strong>. A ROC curve is the set of points with coordinates <strong>(Sensitivity, 1- Specificity)</strong> or (TPR, FPR) where each one corresponds to the performance of the same classifier but with a different threshold. It can be shown that, for a big enough dataset and despite its discrimination threshold, a completely random classifier will always have its coordinates on the diagonal [(0,0), (1,1)] also called <strong>line of no-discrimination</strong>. Moreover, if a point is in the semi-plane above this line then the following inequality holds :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5fTaR2NCuafdWGESP48cHQ.png" /></figure><p>In other words, each of the probabilities on the confusion matrix diagonal are greater than 50%, which means that we have more than a 50% chance to predict each of the classes right, in this case we say that our classifier performs <strong>better than a random guess</strong>. The same reasoning holds for points below the line that are worse than a random guess. However, if a classifier is always below the no-discrimination line for all thresholds, simply flipping its predictions makes it above the line and therefore an acceptable model.</p><p>When the threshold value is T=0 all the classifiers systematically predict positives and therefore they are all located at (1,1) in the ROC space. Conversely, when T = 1 the opposite thing happens and all the classifiers are at (0,0). To illustrate, here is an example of two good ROC curves :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*5cuH-ntKOoujERpa68mSaw.png" /></figure><p>But how do we compare models using this metric ? When given a set of classifiers, at each given specificity value, the best classifier has to have the best sensitivity. Therefore, an approximate way to select the best model is to keep the one which ROC curve is strictly above all the others. Unfortunately, this is only achievable when there is a clear order between the several curves. To address this issue we can resort to a more aggregated metric, the <strong>AUC score</strong>.</p><p>For each ROC curve there is an associated Area Under the Curve or <strong>AUC score</strong>. This AUC directly depends on the ROC curve but has some additional advantages, the first one being that it allows us to objectively compare classifiers using a real value instead of relying on a visual interpretation of a graph. Moreover, it can be shown that the AUC is equal to the probability that the classifier ranks a randomly chosen positive instance higher than a randomly chosen negative one. With this in mind we see that the AUC is a more powerful tool to use on a binary classification task than any of the previous metrics as it is stable against unbalanced datasets, easy to interpret and objectively ranks classifiers’ performances.</p><p>Any of these metrics (and especially the AUC) would work rather well in any <strong>deterministic scenario</strong> such as finding out <a href="https://hackernoon.com/how-hbos-silicon-valley-built-not-hotdog-with-mobile-tensorflow-keras-react-native-ef03260747f3">whether something is or is not a hot-dog</a>. In these cases the target value is a <strong>deterministic function</strong> of a set of predictors and one may try to directly tackle the classification task following the usual routine to estimate this function. But in <strong>random scenarios </strong>where the target has a <strong>random non-predictable component besides a deterministic function</strong>, predicting the classes is only a proxy for — as well as the result of — <strong>estimating probabilities</strong>. For example, if we try to predict the outcome of a fair coin toss, for a large enough dataset, no model can achieve more than a 50% accuracy not matter how fine-tuned it is. In this case the underlying goal and really the only thing we can hope to do is to evaluate the probability of the process and find the optimal model i.e. predicting always 50%.</p><blockquote><em>Unfortunately</em>, <em>the metrics we have seen so far don’t express how well we are at finding these probabilities</em>.</blockquote><p>In fact, for a fair coin toss and a big enough dataset, besides not being able to achieve a greater accuracy than 50%, all the ROC curves will be on the no-discrimination line and the AUC scores will always be 50%. This doesn’t give us much insight about what is going on with the classifier, hence the need for a <strong>new metric</strong>. And there are many cases where the underlying process is random, for example :</p><ul><li>Predicting the outcome of quantum mechanics’ experiments, e.g. if an electron’s spin is up or down.</li><li>Predicting unseen cards in a poker game.</li><li>Predicting whether a student is going to succeed on a particular task (what we do at <strong>Kwyk</strong>)</li></ul><p>In all these cases, working with simple classification metrics prevent us from moving forward in modelling. So what metric should we use ? Because we want a metric that tells us how good we are at predicting <strong>continuous</strong> probabilities<strong><em> </em></strong>for each class, we will focus on a <strong>regression metric </strong>the<strong> MSE</strong>.</p><h4>Mean Squared Error</h4><p>The Mean Squared Error or <strong>MSE</strong> is the average of the squared differences between the predicted and actual values. It can be written formally as :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2tmszYuCrZ4M6e4LYu4pSw.png" /><figcaption>Notations : p_i is the i-th prediction, y_i the i-th target and N the sample size.</figcaption></figure><p>In the case of a binary classification where <strong><em>y</em></strong> is a discrete variable (0 or 1) and <strong><em>p </em></strong>is the predicted probability of success, it is also called <strong>Brier score</strong>.</p><p>For a fair coin toss, if the model we choose to use always predicts the same probability for all samples, a quick look at the MSE profile allows us to find the optimal model :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*VKtnLnRYE5iEsVN8sqM59Q.png" /><figcaption>MSE values for a fair coin toss when the predicted probability varies from 0 to 1.</figcaption></figure><p>This is a start but we see that the MSE isn’t zero when the model is optimal. In fact always predicting 50% is the best model we can ever choose to model a fair coin toss as the underlying process is a <a href="https://en.wikipedia.org/wiki/Bernoulli_distribution">bernoulli</a> B(0.5). But because the MSE is 0.25 at its minimum we could think that there is still a better model out there with an MSE &lt; 0.25. Here, the remaining MSE when p=0.5 corresponds to the <strong>inherent randomness</strong> in the task. In fact, when a process is non-deterministic there is always a non-avoidable error in the MSE due to <strong>uncertainty</strong> in the data we try to predict. For now lets just note that 0.25 is exactly the variance of a B(0.5) which is the distribution of our process.</p><p>Using the MSE actually allows us to discriminate between a set of classifiers and determine the most optimal option but we still don’t have a metric that tells us how far we are from <em>the</em> <em>perfect</em> <em>model</em>. This was the case for deterministic tasks using any of the previous metrics : for a <em>perfect model </em>the accuracy score is 100% , confusion matrices are equal to the identity and the ROC curve is square with AUC = 1. In any case, we saw that the MSE cannot be zero because of the very nature of the random processes we are interested in, but we also know of a way to link random observations to their underlying probabilities which is the <strong>law of large numbers (LLN)</strong>.</p><p>In fact, if the predicted probabilities make any sense and if we make a group for each predicted probability in which we collect all the corresponding targets, for a big enough group, the average of targets should converge to the predicted probability. This is another step towards the desired metric as we could have a look at the difference between the average targets and the predicted probabilities in each group which should converge to zero the more our classifier is good. But when the process has a continuous distribution the size of these groups will hardly ever grow at all and the LLN won’t be able to take effect. A solution to this issue is to group different predicted probabilities together in <strong>K groups</strong> each of <strong>n_k </strong>elements and compare the within-group average success rate (average of targets) to the average probability of the group.</p><p>Let’s see what happens to the MSE when we do such a grouping, here BS stands for <strong>Brier Score</strong> :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rn-m0GBumWJ42RDpDUqUBg.png" /><figcaption>Source : <a href="http://journals.ametsoc.org/doi/pdf/10.1175/2007WAF2006116.1">“Two Extra Components in the Brier Score Decomposition”</a></figcaption></figure><p>The calculations above show that the Brier Score/MSE is the results of the interaction of 5 terms :</p><ul><li><strong>Uncertainty </strong>(UNC) : the uncertainty displays the <strong>inherent randomness</strong> in our task. For example if we are predicting a target that is always zero or one, UNC is zero as there is no uncertainty at all in the data to predict. Conversely for a perfectly random experiment like the fair coin toss of earlier, the uncertainty is maximum and equals 0.25 .</li><li><strong>Reliability</strong> (REL) : the reliability term measures the <strong>biais</strong> (difference) between the average prediction in each group and the average success rate in the group. This is what tells us if our model predict ‘good’ probabilities or not. The more precise the model the more this term tends to zero, and if each of the groups only have 1 target we have REL=MSE.</li></ul><blockquote>Note : The REL term can often be “enforced” using <a href="https://www.quora.com/What-is-called-classifier-calibration-in-machine-learning">calibration methods</a>.</blockquote><ul><li><strong>Resolution</strong> (RES) : the resolution term measures the <strong>quality of the grouping</strong> by computing how much the within-group average success rates differ from the global success rate. This value cannot be enforced but ideally we want it to be as high as possible. A high RES means that the grouping allows for good distinction between cases, and when each group only has 1 target value the resolution is maximum and RES=UNC .</li><li><strong>Within-Bin Variance</strong> (WBV) : this is the average within-group <strong>variance of predictions</strong>. This value increases when different probabilities are grouped together and decreases otherwise. It often increases when REL decreases as for bigger groups the average predicted probability converges towards the average success rate, decreasing the REL by the LLN,<strong> </strong>but because different predicted probabilities are mixed together the WBV increases.</li><li><strong>Within-Bin Covariance</strong> (WBC) : this is another term that is only due to mixing different probabilities together. It is the within-group <strong>covariance</strong> between the predicted values and the targets and therefore displays the join-variability of the two. It is also the only term that can be either positive of negative. We want it to be as positive as possible to compensate the WBV which happens when greater values of probabilities are associated to greater values of targets and <em>vice versa.</em></li></ul><blockquote>Note : both the WBV and WBC are zero every group has only one target.</blockquote><p>In order to visualize the MSE decomposition we can plot each of the terms for the fair coin toss example for K groups of n_k = 1 to 100 elements :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wmKXHmVlT_OPSAl2p17SmA.png" /><figcaption>Simple case of an MSE decomposition for different groupings</figcaption></figure><p>As it was intended in the formula, the MSE doesn’t depend on the grouping so it only changes when the model (predicted probability) changes. First of all, the WBV and WBC terms are zero because only one probability is predicted, therefore there is no variance nor covariance. We can notice the UNC plateau at 0.25 that represents the very high (actually maximum) uncertainty of the data we try to predict. The RES is naturally maximum and equal to UNC when n_k = 1 but then decreases to zero when we group more and more different targets together. At last but not least, the REL behaves as expected as it is minimum, at each grouping, for the optimal model and converges to zero for larger groups thanks to the law of large numbers.</p><p>It is now obvious that a lot is going on in the raw MSE metric. Admittedly, the MSE alone is good enough when trying to evaluate a model on a task with inherent randomness, but the decomposition MSE=REL-RES+WBV-WBC+UNC gives more insight on how good the model actually is for predicting probabilities thanks to the REL term. But in order for the REL to be informative the grouping should be large enough for the LLN to take effect. At the same time, a grouping too large biaises the metric as the within-group predicted probabilities will tend to pull away from their average and so will the within-group success rate.</p><blockquote>In order to find an optimal grouping, we can aim to minimize REL + WBV as the first term wants the groups to be as large as possible and the second term penalizes groups that are too large.</blockquote><p>An approximate solution for this optimization problem can be found by noticing that REL grows when the average success rate differs from the average probability. For large enough groups this error is of the same order as <strong>1/sqrt(n_k)</strong>. Similarly, under the assumption that we start by sorting the predictions and then group them in intervals, knowing that the WBV will decrease if the interval converges to a single point we can minimize the interval length to minimize WBV. We also note that the interval length is <strong>n_k/N</strong> provided we assume that the predictions are uniformly distributed — which is a big assumption but are only looking for an approximate solution here. In order to compensate any missing coefficients in the optimization objective we minimize a linear combination of the two terms :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Eq2JteRwHbT3Dd0T2SLjg.png" /></figure><p>So we found an approximate solution that depends on some parameters, let’s vary these parameters and see how this looks on some real data.</p><p>At <strong>Kwyk </strong>we provide math exercises online, have students do them then use the collected data to predict whether a student will be successful on future exercises to calibrate their complexity. The classifier we use for this task predicts probabilities for each user on each exercise that are distributed as follows :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*njiNUa3erUaAqwVLq-s_2A.png" /><figcaption>Histogram of the predicted probabilities of our classifier</figcaption></figure><p>Using these probabilities we can plot the evolution of the objective function for different group sizes and see where the minimum occurs :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*PEhQMhsYq0hQ0Wf9iYB-Tg.png" /></figure><p>We see that the optimal grouping corresponds indeed to our handmade solution. We can also see that the empirical optimum is roughly around <strong>N^(2/3) </strong>which<strong> </strong>we propose as a rule of thumb for an optimal group size.</p><h4>Conclusion</h4><p>All in all, we saw that usual classification metrics aren’t useful in a stochastic setting. In these cases we have to rely on a regression metric and directly evaluate the ability of our classifier to predict ‘good’ probabilities. A way to tackle this is to use the MSE and see how the algorithm performs. But the MSE is the result of complex interactions between many terms each of which has a specific meaning and purpose. The part of the MSE that is the most interesting in a random setting is the REL, but this assumes we had previously grouped our predictions. Grouping can be made in several ways some of which aren’t informative enough. To address this issue we have to make groups big enough for the REL to converge thanks to the LLN but small enough in order not to mix too different predicted probabilities. To do so, we propose a rule of thumb that consists of sorting the probabilities then making groups of n_k = N^(2/3) elements and evaluate the model on that using the REL metric.</p><p><strong>Bonus : </strong><em>How does our model perform on predicting students’ performance ?</em></p><p>Here are all the metrics we have seen so far in the case of our data at <strong>Kwyk</strong>. We have trained our classifier on 500,000 samples and predicted 500,000 different samples with a global success rate (target mean) of 66.30%. The metrics are :</p><ul><li>accuracy score (threshold 50%) = 75.34%</li><li>confusion matrix (threshold 50%):</li></ul><p>The model seems to randomly predict the negatives</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*q4U0bysgRGYIRuf6WeTLuQ.png" /></figure><ul><li>confusion matrix (threshold 60%):</li></ul><p>Now it looks like its ok…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*N9icQqHe6f1CCWmU6MO97g.png" /></figure><ul><li>ROC curve and AUC :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0DtLT43uDrwdslyv3w26-Q.png" /></figure><p>This means our model has a 79% chance to score a randomly chosen positive instance higher than a randomly chosen negative one.</p><ul><li>MSE : 0.168 (low value but this doesn’t give us insight on the model)</li></ul><p>Using our rule of thumb, for groups of size n_k = N^(2/3) :</p><ul><li>UNC : 0.223 (the task is very uncertain, the max UNC being 0.25)</li><li>REL : 0.0002 (the model is pretty precise)</li><li>RES : 0.054 (the grouping is pretty good knowing that maxRES = UNC)</li><li>(WBV, WBC) : (0.00001, 0.00003) these are very low values that hint for the fact the grouped probabilities are very similar.</li></ul><p><em>Author : </em><a href="https://www.linkedin.com/in/hichamelboukkouri"><em>Hicham EL BOUKKOURI</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd5bafa79505" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/metrics-random-processes-in-classification-fd5bafa79505">Metrics &amp; Random Processes in Classification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>