<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ryan Kuria on Medium]]></title>
        <description><![CDATA[Stories by Ryan Kuria on Medium]]></description>
        <link>https://medium.com/@ryan.k.o.x2009?source=rss-017ae2310c17------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*CkSZ-FSwchsMQzQf</url>
            <title>Stories by Ryan Kuria on Medium</title>
            <link>https://medium.com/@ryan.k.o.x2009?source=rss-017ae2310c17------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:41:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ryan.k.o.x2009/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Simplified StudentSim: Replicating Microsoft’s Approach to Simulating Individual…]]></title>
            <link>https://medium.com/@ryan.k.o.x2009/building-a-simplified-studentsim-replicating-microsofts-approach-to-simulating-individual-6362eb3efee6?source=rss-017ae2310c17------2</link>
            <guid isPermaLink="false">https://medium.com/p/6362eb3efee6</guid>
            <category><![CDATA[education]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[microsoft]]></category>
            <dc:creator><![CDATA[Ryan Kuria]]></dc:creator>
            <pubDate>Sat, 31 Oct 2026 07:56:35 GMT</pubDate>
            <atom:updated>2026-10-31T07:56:35.522Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Building a Simplified StudentSim: Replicating Microsoft’s Approach to Simulating Individual Learners</strong></h3><p><strong><em>Exploring the research, reproducing a two-stage training pipeline and developing an interactive environment for investigating simulated student behaviour.</em></strong></p><h3>Introduction: From research paper to working system</h3><p>Imagine testing an educational tool with a student who does not actually exist. You give them a mathematics question, observe their response, provide a hint, and see how they react. The challenge is making that simulated student behave consistently enough to make the interaction meaningful.</p><p>This was the idea behind my project, inspired by Microsoft’s StudentSim research, which explores how AI can simulate individual learners and their responses to educational guidance.</p><p>I wanted to explore that idea by building an interactive student simulator of my own. Rather than attempting to reproduce Microsoft’s full model, I developed a simplified prototype using an AI coding agent. My simulator did not use a connected large language model to generate student responses. Instead, it relied on a deterministic system: predefined rules and conditions that controlled how the simulated student responded to questions and feedback.</p><p>This meant I could build and experiment with the core interaction without having to train or deploy a large language model. It also introduced an important limitation: my simulator could demonstrate the mechanics of simulated student behaviour, but it could not reproduce the flexibility or complexity of a language model trained to imitate real learners.</p><p>The project became an exploration of a practical question: how much of a student simulation can be demonstrated through a simplified system, and where does that approach begin to fall short?</p><h3>1. Understanding Microsoft’s StudentSim</h3><p>Before attempting to reproduce the system, I needed to understand the problem it was designed to solve.</p><p>Developing an effective AI tutor requires more than generating accurate explanations. Researchers also need to understand whether students can use those explanations to improve their answers.</p><p>For example, consider two students who both struggle with the same mathematics question. One makes an arithmetic error despite understanding the underlying concept. The other has misunderstood the mathematical rule itself.</p><p>Giving both students an identical explanation might not produce the same result.</p><p>Traditionally, investigating these differences requires collecting responses from real learners. This can make experimentation slow and expensive, particularly when researchers want to compare multiple tutoring strategies across different students.</p><p>Microsoft’s StudentSim explores an alternative: creating individual AI student simulators from historical learner records.</p><p>The system is designed around two capabilities:</p><ul><li>Behavioural fidelity: Reproducing the responses, mistakes and demonstrated ability of a particular student.</li><li>Guidance responsiveness: Reproducing how that student’s response changes after receiving tutor guidance.</li></ul><p>These capabilities are closely connected. A simulator must first behave like the learner it represents. It must then respond to guidance in a way that reflects that learner’s behaviour.</p><p>This is what separates StudentSim from simply prompting a general-purpose chatbot to pretend to be a struggling student.</p><h3>2. Understanding the Research Before Building</h3><p>Before developing the simulator, I needed to understand what Microsoft’s StudentSim was actually attempting to achieve.</p><p>The important distinction was between predicting whether a student would answer a question correctly and simulating how an individual student behaves during a learning interaction.</p><p>A conventional knowledge-tracing model estimates a learner’s knowledge from their previous answers. StudentSim takes a different direction: it aims to simulate individual learner behaviour, including how students respond to questions and educational guidance.</p><p>This distinction shaped my project. I was not trying to build another model that predicted a correct or incorrect answer. I wanted to create an interactive environment in which a simulated student could answer questions and react to feedback.</p><p>However, Microsoft’s approach involves a much more sophisticated machine-learning system than I intended to build. My project therefore focused on creating a simplified demonstration of the interaction rather than reproducing the original model’s architecture or training process.</p><p>That distinction mattered throughout development. I needed to understand the research well enough to identify the central idea, while being realistic about what my own implementation could demonstrate.</p><h3>3. Building the Simulator with an AI Coding Agent</h3><p>Rather than implementing a large machine-learning training pipeline, I used an AI coding agent to help develop the simulator.</p><p>This is what people often call <em>vibecoding</em>: using natural-language instructions to guide an AI coding tool through generating and modifying software. However, generating code was only part of the process. I still needed to decide what the simulator should do, examine whether the generated implementation matched that goal, and test how its behaviour changed during interactions.</p><p>I approached the project as a prototype rather than an attempt to reproduce Microsoft’s underlying model. The goal was to create a working simulation in which a user could interact with a virtual student and observe its responses.</p><p>One major design decision was to avoid depending on a connected large language model. Instead, I used a deterministic fallback system to control the simulated student’s behaviour.</p><p>A deterministic system follows predefined rules. Given the same starting conditions and inputs, it produces the same outputs. For example, a system might use predefined conditions to determine how a simulated student responds to a question or how its behaviour changes after receiving feedback.</p><p>This made the prototype simpler to develop and easier to control. It also meant that the simulator’s behaviour was limited by the rules I had implemented. It was not independently reasoning about mathematics, interpreting complex explanations, or generating genuinely human-like student responses.</p><p>The AI coding agent helped accelerate development, but the resulting simulator remained a simplified software implementation rather than a trained AI student model.</p><h3>4. Designing the Simulation Logic</h3><p>The most important part of my implementation was the system responsible for producing student responses.</p><p>Because the simulator did not use an LLM, it needed another way to determine what the virtual student would do. I used predefined logic to simulate responses instead of relying on a generative model.</p><p>This created a controlled environment in which student behaviour could be represented through programmed conditions. Rather than generating an unrestricted response to every interaction, the system followed the rules available to it.</p><p>The advantage was predictability. I could test the same interaction repeatedly and observe whether the simulator behaved consistently.</p><p>The disadvantage was flexibility. A real learner might unexpectedly misunderstand a question, interpret a hint differently, or demonstrate knowledge that does not fit a predefined pattern. A deterministic system can only represent behaviours that its rules account for.</p><p>This was an important engineering trade-off. I prioritised building a working and understandable interaction over attempting to create a more complex system that I could not realistically implement within the project’s scope.</p><h3>5. Turning the Logic into an Interactive Simulator</h3><p>Once the simulation logic was in place, the next challenge was making it accessible through an interactive interface.</p><p>Instead of presenting users with a dataset or a model’s raw predictions, I wanted them to experience a simulated learning interaction. The user could interact with the virtual student, provide educational input, and observe the response generated by the system.</p><p>This changed the purpose of the project from simply displaying information about student simulation to allowing someone to experiment with it.</p><p>The interface served as the bridge between the user and the deterministic simulation logic. When an interaction occurred, the application passed the relevant input to the underlying system, which applied its predefined rules and returned a simulated response.</p><p>It is important to distinguish this process from LLM inference. Microsoft’s StudentSim uses a trained language model to generate student behaviour. My prototype instead relied on programmed logic to produce its responses.</p><p>The result was a simplified demonstration of the interaction loop, not an implementation of Microsoft’s generative student model.</p><h3>6. Testing the Simulator and Understanding Its Limitations</h3><p>Building an interactive simulator does not automatically mean that it accurately represents real students.</p><p>I therefore needed to distinguish between two different questions: does the application function as intended, and does its simulated behaviour resemble that of an actual learner?</p><p>The first question could be explored through functional testing. I could check whether interactions triggered the expected responses, whether the interface displayed those responses correctly, and whether repeated inputs produced consistent behaviour.</p><p>The second question was much harder.</p><p>Because my simulator relied on deterministic rules rather than a model trained on individual student behaviour, its responses were constrained by the logic I had programmed. Consistency did not necessarily mean realism.</p><p>For example, a simulator could repeatedly produce the same response to a particular question while still failing to represent the range of ways real students might misunderstand it.</p><p>This meant I could evaluate whether my prototype worked as a software application, but I could not claim that it achieved the behavioural fidelity of Microsoft’s StudentSim.</p><p>That limitation became one of the most useful findings from the project: simulating an interaction is not the same as accurately simulating a learner.</p><h3>7. What I Learned from Building with an AI Coding Agent</h3><p>One of the biggest lessons from this project was understanding the difference between using AI to build software and building software that uses AI.</p><p>I used an AI coding agent to help generate and modify the simulator. However, the application itself did not rely on a connected LLM to simulate students. Its behaviour came from a deterministic system.</p><p>That distinction is easy to overlook. An application can be built with AI assistance without containing an AI model of its own.</p><p>Using an AI coding agent also changed how I approached development. I could describe features in natural language and use the generated code as a starting point. But I still had to understand the intended behaviour, inspect the implementation, and test whether the result actually worked.</p><p>The process made prototyping faster, but it did not remove the need for technical understanding. If I could not explain how the simulator generated a response, I could not confidently claim that I had implemented that behaviour.</p><p>More importantly, building a simplified version helped me understand why Microsoft’s original research is challenging. Creating a virtual student that responds to a question is relatively straightforward. Creating one that consistently represents an individual learner’s knowledge, misconceptions and response to guidance is a much harder problem.</p><h3>Conclusion: Reproducing the research is only the beginning</h3><p>Working through Microsoft’s StudentSim introduced me to a different way of thinking about AI in education.</p><p>The project combines several important machine-learning concepts: language-model fine-tuning, individual model specialisation, structured interaction data and evaluation of generated behaviour.</p><p>Understanding and reproducing its architecture provided an opportunity to move beyond using pretrained AI models and investigate how they are trained, adapted and evaluated for a specific research problem.</p><p>The most interesting aspect, however, is what this technology could make possible.</p><p>Instead of testing every tutoring strategy directly on real learners, researchers could use individual student simulators to investigate potential interventions, identify promising approaches and refine their tutoring systems before conducting human studies.</p><p>There is still a significant gap between simulating a student’s behaviour and understanding their actual learning. Closing part of that gap could make personalised educational AI more systematic and experimentally accessible.</p><p>For me, this project was not simply about getting an existing research implementation to work. It was about understanding the engineering decisions behind it and exploring how to develop those ideas further.</p><p>The larger question remains: Can we build AI systems that do not just teach students, but help us understand how different students learn?</p><p>StudentSim offers an interesting starting point for investigating that possibility</p><h3>References and technical resources</h3><p><strong>1. Yang, K., Wang, C., Galley, M., Singh, C., Inala, J. P., Zhai, C., &amp; Gao, J. (2026). StudentSim: Training LLM-based Student Simulators.</strong><a href="https://arxiv.org/abs/2609.01591?utm_source=chatgpt.com"><strong> Original research paper</strong></a><strong>.</strong></p><p><strong>2. Microsoft Research. (2026). Simulating Student Learning for Better AI Tutoring.</strong><a href="https://www.microsoft.com/en-us/research/blog/simulating-student-learning-for-better-ai-tutoring/?utm_source=chatgpt.com"><strong> Research overview</strong></a><strong>.</strong></p><p><strong>3. Microsoft. StudentSim: Official GitHub Repository.</strong><a href="https://github.com/microsoft/StudentSim?utm_source=chatgpt.com"><strong> Implementation and documentation</strong></a><strong>.</strong></p><p><strong>4. Microsoft Research. StudentSim Project Website.</strong><a href="https://microsoft.github.io/StudentSim/?utm_source=chatgpt.com"><strong> Research methodology and evaluation</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6362eb3efee6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can AI Tell When You’ve Actually Learned Something?]]></title>
            <link>https://medium.com/@ryan.k.o.x2009/how-well-does-ai-really-know-its-students-5fe911cb45ad?source=rss-017ae2310c17------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fe911cb45ad</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ryan Kuria]]></dc:creator>
            <pubDate>Mon, 28 Sep 2026 21:06:16 GMT</pubDate>
            <atom:updated>2026-10-22T17:18:23.035Z</atom:updated>
            <content:encoded><![CDATA[<h3>How AI Tutors Estimate What Students Know, and Whether Those Estimates Reflect Real Learning</h3><h3>Abstract</h3><p>AI tutors can predict answers, track performance, and estimate what students know. But does answering correctly mean a student has actually learned? This article examines how AI systems estimate student knowledge, from Bayesian Knowledge Tracing (BKT), which models mastery through probabilities, to newer deep-learning approaches that identify patterns in learning histories. It explores the limitations of these methods, including guessing, careless mistakes, and the difficulty of distinguishing short-term performance from lasting understanding. More importantly, it examines what happens when students must recall knowledge after a delay or apply it to unfamiliar problems. These challenges raise a fundamental question: are AI tutors measuring genuine learning, or merely predicting the next correct answer? By examining how knowledge is estimated and what evidence can validate those estimates, this article explores how much confidence we should place in AI’s judgments about what a student has truly learned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A4jcp89U2HgLCJCz.jpg" /><figcaption><a href="https://s39613.pcdn.co/wp-content/uploads/2025/04/iStock-1497617080-scaled.jpg">https://s39613.pcdn.co/wp-content/uploads/2025/04/iStock-1497617080-scaled.jpg</a></figcaption></figure><h3>The Student Behind the Problem</h3><p>Imagine a student named John who has a mathematics test in two weeks and wants to achieve an above-average grade. Tight deadline, right? So, he turns to an AI tutor for help.</p><p>John completes a practice test and submits it for review. The tutor identifies several incorrect algebra answers, provides solutions, generates more questions, and waits.</p><p>John still fails to achieve his target grade. So what really happened?</p><p>Let&#39;s look behind the scenes.</p><p>The problem is that the tutor recognises the wrong answers without understanding the mistakes behind them. John is not struggling with algebra as a whole. He understands how to solve equations but repeatedly makes errors when calculating algebraic equations.</p><p>The tutor has been treating his mistakes as general weaknesses instead of identifying the specific misconception.</p><p>After analysing John’s working across several questions, a more advanced tutor identifies the recurring pattern. Instead of repeating the entire lesson, it gives him targeted exercises, asks him to explain his reasoning, and checks whether he has corrected the misconception. It then updates its estimate of his understanding based on his subsequent answers.</p><p>This small change illustrates a much larger challenge in AI education: a tutor that can answer questions is not necessarily a tutor that understands how a student learns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/0*gAdpNAJWEYScNZ2G" /><figcaption><a href="https://img.magnific.com/free-photo/educational-theme-close-up-student-writing-classroom_1150-16616.jpg?semt=ais_hybrid&amp;w=740&amp;q=80">https://img.magnific.com/free-photo/educational-theme-close-up-student-writing-classroom_1150-16616.jpg?semt=ais_hybrid&amp;w=740&amp;q=80</a></figcaption></figure><h3>1. From Answers to Knowledge: How AI Estimates Mastery</h3><p>To understand how an AI estimates mastery, we first have to understand what a learner model is. In this case, A <strong>learner model</strong> (also called a <strong>student model</strong>) is the tutoring system’s dynamic, evidence-based representation of an individual learner’s current state. It is not a record of everything the student is or believes; it is the system’s best current inference about what the student knows, can do, misunderstands, and may need next.</p><p>In an intelligent tutoring system, the learner model represents the student relative to a <strong>domain model</strong> — the structured map of concepts, skills, procedures, and expert strategies for the subject. The learner model records how the individual student’s knowledge compares with that domain.</p><p>A common <strong>method for estimating mastery is Bayesian Knowledge Tracing (BKT), </strong>introduced by Albert Corbett and John Anderson in 1995. Instead of treating each answer in isolation, BKT maintains an estimate of the probability that a student has mastered a particular skill. When the student answers another question, the model updates that estimate using the new evidence.</p><p>The model accounts for four important factors: the probability that the student already knows the skill, the probability that they learn it during practice, the possibility of answering correctly without knowing it, and the possibility of answering incorrectly despite knowing it.</p><p>These last two factors matter more than they might initially seem. A student can guess correctly, just as a student who understands a concept can make a careless mistake. If an AI treats every correct answer as proof of mastery and every incorrect answer as proof of ignorance, its understanding of the student will quickly become unreliable.</p><p>BKT offers a more cautious approach. It doesn&#39;t directly observe understanding; it uses a probabilistic model to estimate the likelihood of mastery, given the student’s response history and the model’s assumptions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/0*7Mpf1y5hFr_V6Lzd.png" /><figcaption><a href="https://media.springernature.com/m685/springer-static/image/art%3A10.1007%2Fs11257-023-09389-4/MediaObjects/11257_2023_9389_Fig1_HTML.png">https://media.springernature.com/m685/springer-static/image/art%3A10.1007%2Fs11257-023-09389-4/MediaObjects/11257_2023_9389_Fig1_HTML.png</a></figcaption></figure><h4>A Worked Example: Can One Correct Answer Prove Mastery?</h4><p>Imagine an AI tutor is helping a student learn the distributive property. The student needs to understand that:</p><p><strong><em>3(x + 4) = 3x + 12</em></strong></p><p>Before the student begins practising, the tutor estimates a 20% probability of mastery. This does not mean the student literally understands 20% of the concept. It means the model assigns a 0.20 probability that the student has mastered the skill.</p><p>For this illustrative example, suppose the BKT model uses the following parameters:</p><p><strong>Table 2. Example parameters used by Bayesian Knowledge Tracing</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jwuMAjkj7QGtPE8xgKhJPQ.png" /></figure><p><em>Note: These are hypothetical values chosen to demonstrate the calculation, not measured probabilities from a real student.</em></p><h4>Step 1: The student answers correctly</h4><p>Suppose the student correctly expands <strong><em>3(x + 4) = 3x + 12</em></strong>. How much should the AI increase its estimate of mastery?</p><p>It cannot simply assume the student has learned the concept. The student might have guessed or remembered a procedure. BKT therefore updates the probability using both the likelihood of a correct answer from someone who has mastered the skill and the likelihood of a correct answer from someone who has not.</p><p>The probability of mastery after observing the correct answer is:</p><p><strong><em>P(L ∣ C) = [ P(L) · (1 − P(S)) ] / [ P(L) · (1 − P(S)) + (1 − P(L)) · P(G) ]</em></strong></p><p>Substituting the initial values:</p><p><strong><em>P(L ∣ C) = [ 0.20 × 0.90 ] / [ 0.20 × 0.90 + 0.80 × 0.20 ] = 0.18 / 0.34 ≈ 0.529</em></strong></p><p>The estimate rises from 20% to approximately 52.9% after accounting for the correct answer.</p><p>BKT then applies its learning transition. With a 15% probability of learning during this practice opportunity:</p><p><strong><em>P(L_next) = P(L ∣ C) + (1 − P(L ∣ C)) · P(T)</em></strong></p><p><strong><em>P(L_next) = 0.529 + (1 − 0.529)(0.15) ≈ 0.600</em></strong></p><p>The model now estimates a 60% probability of mastery.</p><p>Notice what happened: even after a correct answer, the estimate does not jump to 100%. The model allows for the possibility that the student was not yet proficient.</p><h4>Step 2: The student answers the next question incorrectly</h4><p>The tutor gives the student another expansion problem. This time, the student writes an incorrect answer.</p><p>BKT lowers its estimate because an incorrect response is less likely if the student has mastered the skill, although mistakes remain possible.</p><p>The probability of mastery after an incorrect answer is:</p><p><strong><em>P(L ∣ I) = [ P(L) · P(S) ] / [ P(L) · P(S) + (1 − P(L)) · (1 − P(G)) ]</em></strong></p><p>Using the updated probability of 0.60:</p><p><strong><em>P(L ∣ I) = [ 0.60 × 0.10 ] / [ 0.60 × 0.10 + 0.40 × 0.80 ] = 0.06 / 0.38 ≈ 0.158</em></strong></p><p>After the response update, the estimate falls to approximately 15.8%. Applying the learning transition again gives:</p><p><strong><em>P(L_next) = 0.158 + (1 − 0.158)(0.15) ≈ 0.284</em></strong></p><p>The final estimate is approximately 28.4%.</p><h3>What does this tell us?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZwWZ-n9Nc8Nps4pWUsGD9A.png" /></figure><p>The estimate rises after a correct answer and falls after an incorrect one. The learning transition slightly increases the estimate after each practice opportunity.</p><p>But the AI still does not know exactly why the student made the second mistake. Perhaps the student does not understand distribution. Perhaps they understood the first problem but made a careless error on the second. BKT’s estimate alone cannot reliably distinguish those explanations.</p><p>This is both the value and the limitation of the model. It provides a structured way to update beliefs about mastery as evidence accumulates, but the result remains an estimate shaped by the model’s assumptions. It is not direct proof that the student has, or has not, learned the concept.</p><p>BKT shows how an AI can update its estimate of mastery as a student answers questions. However, the estimate does not necessarily explain the reasoning behind an incorrect answer. If the student repeatedly expands <strong><em>(3(x+4)) </em></strong>as<strong><em> (3x+4)</em></strong>, the tutor needs to investigate whether the student misunderstands the distributive property or has made an isolated mistake. Estimating how much a student knows and diagnosing why they are struggling are related, but distinct, problems.</p><h3>3. Knowing a Student Is Wrong Is Not the Same as Knowing Why</h3><p>While Bayesian Knowledge Tracing provides a mathematically rigorous way to update probabilities, it reveals a fundamental limitation in automated learning models: <strong>knowing a student’s response is incorrect is not the same as understanding why they made the mistake.</strong></p><p>Consider John solving the algebraic expansion problem:</p><p>3(x + 4) = 3x + 4</p><p>The answer is objectively wrong. But what underlying mechanism produced it?</p><ul><li><strong>Is it a cognitive bug?</strong> The student systematically misunderstands the distributive property, operating under a persistent rule misconception that multiplication only applies to the first term inside the parentheses.</li><li><strong>Is it a careless slip?</strong> The student fully understands the distributive law, but experienced a brief processing lapse or calculation error.</li><li><strong>Is it an execution shortcut?</strong> The student memorised a procedural rule without conceptual grounding and misapplied it under test pressure.</li></ul><p>To a standard BKT model, all three scenarios generate identical evidence: a binary incorrect response (0). The system lowers the overall estimate of mastery (P(L)) using its fixed slip (P(S)) and guess (P(G)) parameters. It cannot distinguish between a transient error and a structural misconception.</p><p>Ultimately, BKT functions as a <strong>“statistical thermometer”</strong>: it accurately measures whether the <strong>“temperature”</strong> of a student’s performance is rising or falling, but it cannot diagnose the underlying illness. Knowing that a student’s probability of mastery has dropped from 60% to 28% tells an AI tutor that intervention is required, but it offers zero guidance on what that intervention should be. This accurately explains why methods such as Bayesian Knowledge Tracing can’t properly determine why a student made a mistake while doing a question, which is a major underlying issue with the method.</p><p>One caveat we have to consider is that a<strong><em> student’s knowledge changes over time.</em></strong></p><p>A concept understood last month may become less accessible after weeks without practice. A previously weak skill may become reliable after several successful attempts. A student may also learn a new concept that depends on a prerequisite they previously struggled with.</p><p>A useful learner model needs to track how performance changes over time. Knowledge tracing provides one approach by updating mastery estimates as students interact with problems.</p><p>Instead of treating each answer as an isolated event, the system uses previous interactions as evidence about the learner’s changing knowledge state. It can potentially recognise when a student has demonstrated sufficient mastery and move forward, or identify a persistent difficulty and revisit the skill or one of its prerequisites.</p><p>However, a tutor may know that a student has repeatedly struggled with fractions without knowing whether they need another explanation, prerequisite instruction, additional practice, a different representation, or simply more time.</p><p>A continuously updated learner model is therefore necessary, but it is not sufficient. The system must also decide what to do with the information it collects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*aEnMzfWmIGnQ6DPlO541yw.png" /><figcaption><a href="https://images.pexels.com/photos/8197503/pexels-photo-8197503.jpeg">https://images.pexels.com/photos/8197503/pexels-photo-8197503.jpeg</a></figcaption></figure><h3>4. The Modern State of the Art: Deep Knowledge Tracing and the Unsolved Mind</h3><p>For about two decades, Bayesian Knowledge Tracing has been the gold standard. But it had a massive structural bottleneck: human experts had to manually define parameters (P(G), P(S), P(T)) for every single sub-skill in a curriculum.</p><p>Over the years, Machine Learning Researchers have asked a radical question: <strong><em>What if we stop trying to hand-code human learning and instead let deep neural networks discover the hidden patterns of student performance on their own? </em></strong>This is how systems such as MATHia, ASSISTments, Deep Knowledge Tracing(DKT) and Self-Attentive Knowledge Tracing were developed.</p><h3>1. Deep Knowledge Tracing: Letting the Network Learn the Patterns</h3><p>In 2015, Chris Piech and his colleagues introduced <strong>Deep Knowledge Tracing (DKT)</strong>, applying recurrent neural networks (RNNs) to the problem of modelling student learning.</p><p>DKT processes a sequence of questions and student responses. Using a type of recurrent network called a Long Short-Term Memory network (LSTM), it maintains a hidden representation of the student’s interaction history. This representation is updated as new responses arrive, allowing the model to learn patterns associated with future performance.</p><p>The original researchers reported improved prediction performance on several educational datasets compared with earlier methods. This demonstrated that neural networks could model student response sequences without requiring every aspect of the learning process to be explicitly programmed. However, the hidden representation is not a transparent map of what the student understands. The model may predict the next answer accurately without revealing precisely which concept the student has mastered or misunderstood.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0jgeOD40SVr8zCSf" /></figure><h3>2. Self-Attentive Knowledge Tracing: Which Past Answers Matter?</h3><p>DKT introduced a more flexible way to model learning, but recurrent networks can struggle when useful evidence is scattered across a long interaction history. Not every previous answer is equally relevant to the next question.</p><p>In 2019, Shalini Pandey and George Karypis introduced <strong>Self-Attentive Knowledge Tracing (SAKT)</strong>. Instead of processing the student’s history only through a recurrent sequence of hidden states, SAKT uses an attention mechanism to identify past interactions that are most relevant to the question being considered.</p><p>Imagine a student answering a question on expansion. Their history includes dozens of questions on fractions, equations, negative numbers and algebraic expressions. SAKT can assign greater weight to earlier interactions that are relevant to the current skill, rather than treating the entire history as equally informative.</p><p>The researchers reported improvements over existing approaches on several datasets, including an average AUC improvement of 4.43% in their experiments. However, attention is not a direct measurement of understanding. Identifying which past answers help predict a new response does not necessarily reveal why the student made a mistake or whether they have genuinely mastered the underlying concept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*1OaWgjhP9fnlEjD5" /></figure><h3>5. Where Methods Such as Bayesian Knowledge Tracing Fail</h3><p>In the same way that these intelligent tutoring systems are fundamental to modern education, some of them, such as Bayesian Knowledge Tracing (BKT), which will be in the spotlight for this segment, fail in modern education, and we will discuss how researchers and developers are fixing it.</p><p>First off, BKT evaluates student performance on a binary scale: correct answers are categorised as 1, while wrong answers are categorised as 0. When a student fails a question, BKT lowers the mastery probability using fixed “slip” and “guess” parameters as mentioned before. This is a problem because it means that BKT cannot reliably tell us why a student made an error and treats it as a random guess or calculation slip that the student made. Consequently, it knows that a student failed a question but does not essentially know why.</p><p>Second, traditional BKT requires human experts to manually define the four parameters that are used to determine the student’s mastery in a certain topic area. This approach does not scale: As curricula all over the world expand to thousands of micro-skills across subjects, hand-tuning these probabilities becomes practically impossible.</p><p>Third, BKT assumes that learning is a one-way street governed by a fixed probability of transition. This ultimately fails because it struggles to model human memory decay over time. A student might master a skill on Monday through short-term practice, fail to retain it after three weeks of no practice, and return with degraded knowledge. BKT’s linear updating model treats time gaps poorly unless heavily modified.</p><p>In response to these issues, AI and ML researchers have built some solutions to these problems, among them include:</p><h4>1. Integrating Bug Libraries and “Mal-Rules”</h4><p><strong>The Fix:</strong> Advanced tutoring systems pair knowledge tracing with cognitive architectures (such as ACT-R) that contain libraries of known student misconceptions (“mal-rules”).</p><p><strong>How it works:</strong> Instead of just marking an answer wrong, the system checks if the student’s specific incorrect output matches a known structural error pattern. This allows the tutor to diagnose the root cause immediately and deliver a targeted intervention rather than resetting the student’s mastery score.</p><h4>2. Deep Knowledge Tracing (DKT) &amp; Self-Attentive Knowledge Tracing (SAKT)</h4><p><strong>The Fix:</strong> Replacing hand-coded parameters with machine learning models.</p><p><strong>How it works:</strong></p><p><strong>DKT</strong> uses Recurrent Neural Networks (specifically LSTMs) to automatically discover hidden patterns across a student’s entire history of interactions without requiring human-defined parameters.</p><p><strong>SAKT</strong> uses attention mechanisms (similar to modern transformer architectures) to figure out <em>which</em> past answers matter most. If a student is working on an advanced algebra problem, SAKT can weight previous interactions involving fractions and negative numbers more heavily than older, unrelated practice data.</p><h3>6. The AI Also Needs to Know When It Is Wrong</h3><p>An AI tutor can make a diagnosis and still be wrong.</p><p>Imagine a student gives an incorrect answer. The AI decides that the student does not understand the concept and launches into a twenty-minute explanation.</p><p>But perhaps the student understood the concept and simply made one careless calculation.</p><p>The AI has now personalised the wrong thing, and now it evolves into a big issue.</p><p>This is why uncertainty is such an important part of student modelling. Knowledge tracing does not simply declare that a student either “knows” or “doesn’t know” something. It estimates the probability that a student has mastered a skill based on imperfect evidence.</p><p>The same principle should extend beyond traditional knowledge tracing.</p><p>If a student gives one incorrect answer, the system should not immediately conclude, “You have a misconception.” It should recognise that there is evidence the skill may be weak and that more evidence is needed.</p><p>That could mean giving another problem, asking the student to explain their reasoning, checking a prerequisite, or observing whether the student succeeds after a hint.</p><p>This turns personalisation into an iterative process: observe, hypothesise, intervene, observe again, and update the model.</p><p>It also highlights a challenge for modern AI tutors. A large language model can produce a convincing explanation of a student’s mistake while still being wrong about the student’s actual reasoning.</p><p>In education, sounding confident is not the same thing as being correct. Unfortunately, large language models are remarkably good at the first one.</p><p>The future of AI tutoring cannot simply be about making models better at generating explanations. They also need mechanisms for testing their educational assumptions against subsequent student performance.</p><p>A tutor should not just ask, “What do I think this student understands?” It should also ask, “What evidence would prove me wrong?”</p><p>A tutor that tests its assumptions against new evidence is more reliable than one that merely sounds confident.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*E3L2QkoigzZJ2z76" /><figcaption><a href="https://media.istockphoto.com/id/1096227330/photo/machine-learning-concept.jpg?b=1&amp;s=612x612&amp;w=0&amp;k=20&amp;c=9obipL3pCVY2EAAh1Ohbn9S5-numfntZwOjXa_x8Vc8=">https://media.istockphoto.com/id/1096227330/photo/machine-learning-concept.jpg?b=1&amp;s=612x612&amp;w=0&amp;k=20&amp;c=9obipL3pCVY2EAAh1Ohbn9S5-numfntZwOjXa_x8Vc8=</a></figcaption></figure><h3>Conclusion: Can AI Tell When You’ve Actually Learned Something?</h3><p>AI tutors have become increasingly sophisticated at estimating what students know. Bayesian Knowledge Tracing updates its estimates as new answers arrive, while models such as Deep Knowledge Tracing and Self-Attentive Knowledge Tracing learn patterns from students’ interaction histories. These approaches give AI systems increasingly powerful ways to predict performance and adapt instruction.</p><p>But predicting an answer is not the same as measuring understanding.</p><p>A student may answer correctly by guessing or recalling a familiar procedure without understanding why it works. Another may understand a concept but make a careless mistake. Even a model that predicts responses accurately may struggle to distinguish these situations or explain the reasoning behind an error.</p><p>This is the central limitation of knowledge tracing: <strong>a model can estimate mastery without proving that mastery exists.</strong> Its predictions are only as informative as the evidence, assumptions and evaluation methods behind them.</p><p>To determine whether a student has actually learned, we need to look beyond immediate answers. Can they solve unfamiliar problems that test the same concept? Can they explain their reasoning? Can they still apply the knowledge days or weeks later? Independent assessments of transfer and retention provide stronger evidence of learning than next-answer prediction alone.</p><p>The next generation of AI tutors must therefore do more than predict what students will answer. Researchers need to test whether their estimates correspond to genuine understanding, identify when those estimates are unreliable, and establish whether the instruction guided by them produces lasting learning.</p><p>So, can AI tell when you’ve actually learned something?</p><p><strong>It can estimate the likelihood. Proving that the estimate reflects genuine, lasting understanding remains a much harder problem.</strong></p><p><strong><em>“Education is the most powerful weapon which you can use to change the world.”- Nelson Mandela, speech at Madison Park High School, Boston, June 23, 1990.</em></strong></p><h3>References</h3><p>Corbett, A. T., &amp; Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. <em>User Modeling and User-Adapted Interaction, 4</em>, 253–278. <a href="https://doi.org/10.1007/BF01099821">https://doi.org/10.1007/BF01099821</a></p><p>Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., &amp; Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. <em>Cognitive Science, 13</em>(2), 145–182. <a href="https://doi.org/10.1207/s15516709cog1302_1">https://doi.org/10.1207/s15516709cog1302_1</a></p><p>Kestin, G., Miller, K., Klales, A., Milbourne, T., &amp; Ponti, G. (2025). AI tutoring outperforms in-class active learning: An RCT introducing a novel research-based design in an authentic educational setting. <em>Scientific Reports, 15</em>, Article 17458. <a href="https://doi.org/10.1038/s41598-025-97652-6">https://doi.org/10.1038/s41598-025-97652-6</a></p><p>Létourneau, A., Deslandes Martineau, M., Charland, P., Karran, J. A., Boasen, J., &amp; Léger, P. M. (2025). A systematic review of AI-driven intelligent tutoring systems (ITS) in K–12 education. <em>npj Science of Learning, 10</em>, Article 29. <a href="https://doi.org/10.1038/s41539-025-00320-7">https://doi.org/10.1038/s41539-025-00320-7</a></p><p>Pandey, S., &amp; Karypis, G. (2019). A self-attentive model for knowledge tracing. <em>Proceedings of the 12th International Conference on Educational Data Mining</em>. <a href="https://arxiv.org/abs/1907.06837">https://arxiv.org/abs/1907.06837</a></p><p>Piech, C., Bassen, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L. J., &amp; Sohl-Dickstein, J. (2015). Deep knowledge tracing. <em>Proceedings of the 28th International Conference on Neural Information Processing Systems?</em></p><p>ASSISTments. (n.d.). <em>Research and evidence</em>. <a href="https://www.assistments.org/research">https://www.assistments.org/research</a></p><p>Carnegie Learning. (n.d.). <em>MATHia: Research and evidence for mathematics learning</em>. <a href="https://www.carnegielearning.com/">https://www.carnegielearning.com/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fe911cb45ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>