<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anitha Sivasubramanian on Medium]]></title>
        <description><![CDATA[Stories by Anitha Sivasubramanian on Medium]]></description>
        <link>https://medium.com/@anitha.sivasubramanian?source=rss-78a545f91f9b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*mL1YeVfx4POlqFwP.jpg</url>
            <title>Stories by Anitha Sivasubramanian on Medium</title>
            <link>https://medium.com/@anitha.sivasubramanian?source=rss-78a545f91f9b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 17:22:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anitha.sivasubramanian/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How does Wiley empower instructors with assignment length estimates? (Part 3)]]></title>
            <link>https://medium.com/knerd/how-does-wiley-empower-instructors-with-assignment-length-estimates-part-3-b84384f3293?source=rss-78a545f91f9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b84384f3293</guid>
            <category><![CDATA[adaptive-learning]]></category>
            <category><![CDATA[education-technology]]></category>
            <category><![CDATA[bayesian-machine-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Anitha Sivasubramanian]]></dc:creator>
            <pubDate>Wed, 05 Oct 2022 19:21:48 GMT</pubDate>
            <atom:updated>2022-10-05T19:21:48.901Z</atom:updated>
            <content:encoded><![CDATA[<p>Conjugate prior model for time and number of questions (starting with smart defaults and updating estimates over time)</p><p>In <a href="https://medium.com/knerd/how-does-wiley-empower-instructors-with-assignment-length-estimates-part-1-75d43d3983c9">part 1</a> and <a href="https://medium.com/knerd/how-does-wiley-empower-instructors-with-assignment-length-estimates-part-2-e19f19364175">part 2</a>, we described the reason for picking the Bayesian approach to update assignment length estimates in real time based on new data. In this part, let us see how the actual modeling was done using Conjugate prior Bayesian models.</p><blockquote>Modeling duration</blockquote><p>Since the empirical logarithm of completion times appears approximately “normal,” we model the completion time as a log-normal random variable, and hence the model parameters could be the distribution’s mean <strong>μ</strong> and variance σ². This poses 3 different possibilities of prior: unknown mean and fixed variance, unknown variance and fixed mean, or unknown mean and variance. Though the third option is the most principled one, for simplicity purposes and for reasonable data-driven estimates, we put a prior on the mean with fixed variance.</p><p>The table “<a href="https://en.wikipedia.org/wiki/Conjugate_prior">When function is a continuous distribution</a>” gives closed-form solutions for the Bayesian posterior with a conjugate prior. The model parameter here is <strong>μ,</strong> and hence we need to put a prior on <strong>μ</strong> using the hyperparameters m and v, representing the prior mean and variance of the mean <strong>μ</strong> of the presumed Normal data-generating model (i.e., the likelihood) of log-completion times. And σ² is the presumed known variance of the data generating-model of the log-completion times.</p><p>This is used in conjunction with the likelihood of data D consisting of n points D = {x1,…, xn}:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*gBJ8Aspd5OogprgTzAmC5w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/1*SoHTamCpbANGroZHbT5exg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*d3WQ2aLGr_oawk3b531AvQ.png" /></figure><p>The posterior density for <strong>μ</strong>, given the data and the hyperparameters, is then given by the product of the likelihood and prior:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*iAM3qeUrT3nVbcvnMmp5Og.png" /></figure><p>We calculated the prior mean m by combining the data across all LOs as given by a precision-weighted average of the individual LO observed means. This choice is motivated by a hierarchical Bayesian model. (See Section 5.4 of the <a href="http://www.stat.columbia.edu/~gelman/book/">Bayesian Data Analysis</a> book for a detailed example of this approach in a model like ours). This allows us to have knowledge graph-specific priors. This is very important as each knowledge graph is different — and we expect that the distinct subject areas captured by different knowledge graphs are associated with different LO completion times.</p><p>The prior variance v and common likelihood variance are chosen offline by optimizing predictive accuracy.</p><blockquote>Modeling number of questions</blockquote><p>Similar to the normal distribution case for time estimates, the conjugate prior for negative binomial also has two parameters: r, the number of successes, and success probability p. We will treat r as fixed and put a beta prior on p, with hyperparameters a and b.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*-0UyvpLRKWN7W_tagBVAqQ.png" /></figure><p>Given the likelihood and prior, it is straightforward to derive the posterior distribution for the model parameter p.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*OOIW0C6j8rHGQ0dYInL3YQ.png" /></figure><p>The posterior-predictive, the distribution of X after accounting for all observed data thus far, is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*pGsqqmbEceqyccDKrN7-TA.png" /></figure><p>In order to turn our posterior-predictive into point-values we can surface in Alta, we wanted to estimate a range which will capture most of the actual questions-to-complete and time-to-complete for students who work on the LO.</p><p>We decided to try the 10th and 90th percentiles for the low/high time estimates and the 90th percentile for the high questions estimate and the minimum work required for an LO as the low questions estimate. (From this starting point, we iterate on these choices using user feedback on analytics needs in real-world classrooms.)</p><p>For example, in the graph below, the probability of new data points based on current data using the current priors is calculated. A threshold is applied at the 90th percentile of the probability mass function, and from the below graph, we can see that the 90th percentile will be 3 which means 90% of the students will finish at the high estimate or fewer questions. The high questions estimate here is minWork + p90. For instance, if the minimum work for this LO is 4, then 90% of the students will finish at 7 (4 + 3) or fewer questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*sVOVFM0JX2A4mux4MYVM8Q.png" /></figure><blockquote>What happens when we get more data?</blockquote><p>As new evidence arrives, a new posterior is computed, and it becomes like a new prior — and it goes on and on. As we see more and more successes, the distribution shifts to the right (alpha is large), and as we see more and more failures, the distribution shifts to the left (beta is large). As we get more data, the probability distribution narrows and peaks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*61BBd2DV3XGYbihWCF3Svg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*Q1kcfJlHVIX1qqbeMs1H5w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*L29MyaOH7Y-q8kBislxcOg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*hQ9Wf9mJdSZQi5RiyUPBhg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*acmmCNdZQdR-vjkHw33xuQ.png" /></figure><blockquote>What do we do in case of an entirely new knowledge graph?</blockquote><p>We resort to smart defaults for entirely new domains based on heuristics and simply use the average across all domains.</p><p>This prior to posterior transformation lets us start with smart defaults and guarantees transparency in user/model behaviors as new data comes in, allowing us to quickly scale from no to little to lots of data smoothly!</p><p>We hope you have enjoyed this glimpse into the power of using conjugate Bayesian models for efficient realtime updates to learned values as you receive large amounts of data, allowing you to scale smoothly from “smart defaults” in low-data regimes into much more informed estimates as you accumulate data. These sorts of techniques allow us to bring users intelligent insights based on continuous learning from real-world data, leading to a better-informed classroom.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b84384f3293" width="1" height="1" alt=""><hr><p><a href="https://medium.com/knerd/how-does-wiley-empower-instructors-with-assignment-length-estimates-part-3-b84384f3293">How does Wiley empower instructors with assignment length estimates? (Part 3)</a> was originally published in <a href="https://medium.com/knerd">Knerd</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How does Wiley empower instructors with assignment length estimates? (Part 2)]]></title>
            <link>https://medium.com/knerd/how-does-wiley-empower-instructors-with-assignment-length-estimates-part-2-e19f19364175?source=rss-78a545f91f9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e19f19364175</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bayesian-machine-learning]]></category>
            <category><![CDATA[adaptive-learning]]></category>
            <category><![CDATA[education-technology]]></category>
            <dc:creator><![CDATA[Anitha Sivasubramanian]]></dc:creator>
            <pubDate>Thu, 29 Sep 2022 17:48:08 GMT</pubDate>
            <atom:updated>2022-09-29T17:48:08.268Z</atom:updated>
            <content:encoded><![CDATA[<p>Bayesian updating in nutshell (using Conjugate priors) — the preliminaries</p><p>In <a href="https://medium.com/knerd/how-does-wiley-empower-instructors-with-assignment-length-estimates-part-1-75d43d3983c9">Part 1</a>, we described the benefits, challenges, and constraints of using data-informed estimates for assignment lengths to show instructors creating their courses. Given the challenges outlined in part 1, an online Bayesian approach seemed to be the right one, so that we could update the estimates in real time based on new data.</p><p>Bayesian updating revolves around <strong>prior</strong>, <strong>likelihood</strong>, <strong>posterior </strong>and <strong>posterior predictive</strong>. Basically, the idea is to start with our prior beliefs (how long we generally think assignments might take, before we’ve seen specific data) , and as we see more and more data, update our beliefs based on what we see, and update the estimates accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*ajxvy5r9OCVw6UcCnkAgvw.png" /></figure><blockquote>What is the actual data generating process here?</blockquote><p>Clearly, our data generating model is quite bespoke. However, this question can be answered by looking at the distribution of the completion time and number of questions.</p><p>In case of the completion time, the histogram of duration is right skewed. By taking the logarithm, we could conclude that this variable is approximately log-normally distributed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*qWrXtmQmcpa2AS7z3oBqVQ.png" /></figure><p>The next variable of interest is the discrete number of questions required to complete an LO. The actual number of questions required for a student depends on the student completing the minimum work required for the LO as well as achieving a proficiency level above some threshold.</p><p>Typically, you cannot complete an LO without answering at least 4 questions. By carefully reviewing the pool of possible discrete distributions especially the ones that have conjugate priors, we were able to reject pretty much all of them except for the Negative Binomial (NB), and the Geometric, which is a special case of NB.</p><p>The Negative Binomial is the distribution of the number of failures experienced before a fixed number of successes, in IID Bernoulli trials. So, if we say the fixed number of successes is just the minimum work, then often a student will have done minimum work + number of incorrect questions total on a learning objective, which is what is expected from a NB random variable. So, our objective is to model the extra number of questions a student needs to attempt beyond the minimum work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*lkh9n0nzUIYH4Ztjw6HVGA.png" /></figure><p>Assuming the above data-generating distribution for each data point gives us the likelihood. The product of likelihood and prior gives us the posterior density.</p><p>The posterior-predictive distribution is the probability density for a new data point, given all the previously seen data.</p><blockquote>Why Conjugate priors?</blockquote><p>If the posterior distribution is in the same probability distribution as the prior probability distribution, then the prior is called a conjugate prior. Please refer <a href="https://en.wikipedia.org/wiki/Conjugate_prior">the table of conjugate prior distributions in Conjugate prior — Wikipedia </a>. When you use such a conjugate prior and you have a ‘closed-form’ equation to go from prior to posterior given some new observations, you can then continue to update your posterior with additional data using the same kind of calculation, making it efficient to calculate the posterior iteratively without needing to ‘solve for’ or approximate the posterior with an iterative algorithm.</p><p>On a peak week from 2021, we had over 200 thousand new events to incorporate into our models, so it makes a big difference in performance to be able to use a calculation to update the model rather than a complex optimization algorithm. The chosen closed form solution supported by conjugate prior helps us to handle these updates efficiently.</p><p>In the <a href="https://medium.com/p/b84384f3293/edit">next part</a>, we will talk about conjugate prior model for time and number of questions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e19f19364175" width="1" height="1" alt=""><hr><p><a href="https://medium.com/knerd/how-does-wiley-empower-instructors-with-assignment-length-estimates-part-2-e19f19364175">How does Wiley empower instructors with assignment length estimates? (Part 2)</a> was originally published in <a href="https://medium.com/knerd">Knerd</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How does Wiley empower instructors with assignment length estimates? (Part 1)]]></title>
            <link>https://medium.com/knerd/how-does-wiley-empower-instructors-with-assignment-length-estimates-part-1-75d43d3983c9?source=rss-78a545f91f9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/75d43d3983c9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[education-technology]]></category>
            <category><![CDATA[adaptive-learning]]></category>
            <dc:creator><![CDATA[Anitha Sivasubramanian]]></dc:creator>
            <pubDate>Thu, 08 Sep 2022 16:54:02 GMT</pubDate>
            <atom:updated>2022-09-08T16:54:02.355Z</atom:updated>
            <content:encoded><![CDATA[<p>A look into continuous mathematical modeling choices to estimate the length of an assignment in terms of duration and the number of questions.</p><p>Our <a href="https://medium.com/knerd/what-are-knewtons-knowledge-graphs-f6a118acc722">previous blog post</a> will be a good prequel before reading this post. In a nutshell, our platform implements a mastery-based pedagogy algorithmically in a variety of US higher education courseware products.</p><p>Imagine you are an instructor who is setting up an online course in our adaptive platform. You start by creating an assignment by selecting learning objectives (LOs) for that goal. As US higher education follows a credit-based model, a typical credit-bearing course syllabus will allot time for class, homework, and assignment hours. As an instructor, you should know: how long will a student take to complete an assignment?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*7NDY4k-nz2QL46yNgQcLow.png" /></figure><p>In an adaptive world, the instructor chooses the LOs. And then Knewton’s “adaptive recommendation engine” chooses the questions that the students will be answering in an assignment. The Knewton adaptive platform (which powers WileyPLUS and Alta adaptive assignments) constantly assesses the student’s proficiency and current knowledge state and tries to close knowledge gaps (if there are any) by providing remediation content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*kEdC5rrbh0CEvVbAG0Ntfw.png" /></figure><p>Given this scenario, the variation in length of an assignment is not only determined by the duration that the student will be spending per question in completing the assignment, but is also dependent on the number of questions the student will need to answer before the adaptive platform is satisfied with the student proficiency level on that learning objective. For some students, this will include the time required to practice and learn the materials. So, instructors should have visibility on upcoming work in terms of both duration as well as the number of questions that a student typically needs to answer to complete the assignment.</p><p>Several methodologies are widely adopted in our industry to answer this question. To arrive at this estimate, instructors typically use their experience, or they assess how long would it take for them to complete the assignment and multiply by some factor, or they might use surveys to gather data from students on how long it took and use that information when they set up future assignments.</p><blockquote>But why not try to answer this question mathematically with the data/expertise already in hand. Before we dive deep into our modeling approach, what did our initial exploratory data analysis reveal to us about assignment structures?</blockquote><p>Looking over historical data from Alta showed us that sometimes an assignment can focus just on one LO, and sometimes it can have 6 or more different LOs. Similarly, the number of questions in an assignment might vary based on student’s proficiency, current knowledge state and the content recommended by the platform. Initial EDA reveals that a student can complete an assignment by attempting anywhere from 4 to 20 questions (or sometimes more).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*FhFq1x5YtqSe8LTqp39vaA.png" /></figure><p>Further slicing of the data reveals that the learning objectives fall under different levels of difficulty such as easy, medium, and hard which drives the number of questions to completion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*8whl_hNjC5l2Zb5wcFw9BA.png" /></figure><blockquote>What kind of data do we have, and what are the challenges around it?</blockquote><p>All the historical interactions by the students in Alta are captured for already existing knowledge graphs and LOs. In a learning environment the dynamics keep changing, and a bunch of other factors such as change in a course or LO or its pre-requisites could impact how LOs operate. There is always a range of completion times in an LO, so we have to represent uncertainty explicitly. We also often have new content without preexisting data, and changes to the learning environment or content representation, which require us to be able scale from good guesses to data-driven estimates as data arrives or changes. We also need good guarantees and observability to ensure good user experience.</p><p>Considering the above challenges, our initial approach is to build a model that will start with smart defaults. As we receive more LO completion information, we must keep updating our best estimate based on new data. We opted to start with a prior belief about the estimate and use Bayesian updating to transform the prior beliefs into posterior every time the new data arrives. We chose a conjugate prior Bayesian model, due to the computational speed and simplicity of the closed -form solution update equations.</p><p><a href="https://medium.com/p/e19f19364175/edit">Part 2</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75d43d3983c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/knerd/how-does-wiley-empower-instructors-with-assignment-length-estimates-part-1-75d43d3983c9">How does Wiley empower instructors with assignment length estimates? (Part 1)</a> was originally published in <a href="https://medium.com/knerd">Knerd</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My journey from adapting a third party record linkage solution to building my own record linkage…]]></title>
            <link>https://medium.com/@anitha.sivasubramanian/my-journey-from-adapting-a-third-party-record-linkage-solution-to-building-my-own-record-linkage-8f18d74e536d?source=rss-78a545f91f9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f18d74e536d</guid>
            <category><![CDATA[record-linkage]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[text-mining]]></category>
            <dc:creator><![CDATA[Anitha Sivasubramanian]]></dc:creator>
            <pubDate>Tue, 30 Aug 2022 21:01:55 GMT</pubDate>
            <atom:updated>2022-08-30T21:01:55.964Z</atom:updated>
            <content:encoded><![CDATA[<h3>My journey from adapting a third party record linkage solution to building my own record linkage solution. Do’s and don’ts…</h3><p>This story dates back to 2013 when I was analyzing social media trends for emerging areas such as economy sharing that insurance industries need to adapt to. Sharing a car or sharing a home was a big and a new risk Insurance industry needs to cater to. How would we identify if the insured home or car is being rented for a weekend to a friend? Social media was a go to place to get any such additional information. Since we have the insurance and social media data, how do we find that the John Smith who purchased car insurance from me is same as John Smith in the Social media platform? This is a classic record linkage problem. Back then I was evaluating a vendor who was providing such a solution. Last year, I got an opportunity to work on creating one such solution for Education business, this time the task was to link instructors, departments, institutions, courses. Following is the framework I would advocate to anyone who is solving such problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_QJIOWhjzCuyaK2x" /></figure><blockquote>Things to consider when starting a record linkage solution</blockquote><p><strong><em>Identify the right entities that goes as an input</em></strong> to the record linkage solution. Make sure that the identified entities are generic enough. There might be entities that are specific to a particular business process that might help establish matching, carefully choose these entities so that they are generic enough to establish matching.</p><p><strong><em>Understand the use cases</em></strong> where the matching services might be utilized. Understand if the matching service will be used in post-hoc or real time workflow. Be clear about if you are going to improve an existing matching service or build a brand new service?</p><p><strong><em>Design and create a worflow for human feedback </em></strong>for automated machine learning and <strong><em>persist </em></strong>the human feedback to future matched data</p><blockquote>How did I manage to cherry pick ‘gold standard’ set for record linkage?</blockquote><p>Most of the organizations are trying to tackle the problem of linking disparate data systems through machine learning solution. A wide variety of solution could be devised for dealing with this problem space ranging from simple fuzzy matching to a learning algorithm. No matter what algorithms are picked, we need a set of ground truth records that could be used to evaluate those solutions against. In Part 2, I will share some of the techniques I used to cherry pick those<strong> ‘gold standard’</strong> dataset for solving the record linkage problem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f18d74e536d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>