Today our former colleagues Ben Eater and Grant Sanderson published Visualizing quaternions, an exciting addition to a hazy new medium. It combines video-like narrative explanation, interactive representations, and game-like challenge prompts. At first, it feels like watching a YouTube video that uses great visualizations to explain something… but then you realize that you can interrupt the speaker to manipulate with the representation, and they intermittently prompt you to do so with some challenge.

How should we design such systems? What are their prospects? I’ll explore a few ideas here. I know of no common name for this medium, so I’ll call these narrated explorables. A few more examples (write me if you know another?):

Khan Academy’s computer programming lessons feature a live programming environment being manipulated by a narrator you can interrupt at any time. (Internally, we call these “talkies.”)

Our research project Cantor tried expanding this notion to objects on a 2D canvas, demonstrated here with an interactive number block manipulative (search the page for “Interrupting, answering, and elaborating othersʼ recordings”).

I’m excited about this medium for creating explanations:

• Relative to an interactive on its own (like PhET), narrated explorables add the emotional resonance of an invested speaker, plus challenge prompts which guide the learner’s experience.
• Relative to an interactive alongside text (like an explorable), narrated explorables synchronize narration and visualization, which directly connects what’s being discussed in a given sentence to some change in the representation—no hopping back and forth between figure and text.
• Relative to the video on its own (like a 3b1b video), narrated explorables support active learning: they push the learner to ask and answer questions of the representation instead of passively listening.

Because this medium is so underdeveloped, the design patterns aren’t at all clear. In fact, the creators of these narrative interactives seem to gravitate toward fairly distinct mental models. I’ll try to outline each in turn now.

Applet

It’s both charming and revealing: several times in the narration for Visualizing quaternions, Grant refers to the experience as an “applet.” That’s a loaded word! Applets have existed in education for decades, but PhET is a great modern representation.

The applet mental model for narrated explorables suggests that you’ve loaded up some interactive representation of some concept—say, a circuit simulator. The circuit simulator itself is inert. It doesn’t contain any narration or prompts; it’s designed to be a tool that faithfully represents the underlying system. But sitting next to you and looking at that same applet is a tutor, and as you pass the mouse back and forth, they talk you through how the applet works and explain something by manipulating the circuit simulator. When the tutor leaves, you’re left with the circuit simulator, which you’ll use for various projects in the future.

Or, another angle: it’s like the narrator loaded up the circuit simulator and screencasted themselves using it to explain something. Note how you see Grant’s cursor! You can pause the screencast and play with the circuit simulator; when you resume, the screencast continues oblivious to what you did.

The applet model particularly makes sense if the important piece is the interactive representation—if it’s an authentically useful tool going forward, and the narration’s really just there to set up the initial experience. I suspect this model makes less sense if the interactive representation is mostly useful during initial instruction: the clear separation between narration and the interactive leaves many opportunities on the table. We’ll discuss some below.

Enhanced video

The great thing about 3blue1brown videos is that they often feature interesting visualizations. What if you thought you were watching one of those videos… but to your surprise, you found that you yourself could manipulate the visualization Grant was describing? As he iterated on Visualizing quaternions, this was the mental model Ben kept articulating for me.

We see this mental model play out most directly through the literal video playback controls. It also means that when Grant asks you to do something with the interactive, the narrative track doesn’t wait for you.

One thing I love about this mental model is that it takes the combined medium very seriously. It asks: why can’t every 3blue1brown video be like this in its entirety? Why can’t we bring all the π characters and title cards and visual storytelling panache into the interactive environment? Why can’t we have the narrator indicate elements on the screen by wiggling them, like in the videos—rather than by gesturing with a recorded mouse cursor, as the applet mental model suggests? Those are great questions!

I think this mental model makes great sense when most learners will passively consume most of the experience as if it were a video. It creates a smooth ramp for those with additional curiosity to satisfy it at any moment, but perhaps when most people interact, it’ll be to jump back 10 seconds to hear a piece of the explanation again—not to manipulate the interactive themselves.

Video games

In my own experiments with narrated explorables, I’ve often drawn on video games as a mental model.

In video games, players often share the world with virtual characters, and those characters often speak aloud while manipulating the interactive game environment you share. Game narratives don’t have a playback scrubber; instead, they might have a replay/skip button and a chapter list.

Game characters often give you challenges and react when you complete them, sometimes even changing their narrative accordingly. Game narratives wouldn’t prompt you to do something and then keep talking a few seconds later; instead, they’d wait expectantly for your action and respond accordingly.

More importantly, they might reflect your current challenge in the interface and offer feedback on your progress towards it. They might highlight elements in the environment which would help you complete that challenge; if you continued to struggle, they might offer extra advice. Indeed, the full complexity of the environment is often not all revealed at once. The game might expect players to complete some challenges to continue, while it might offer others as optional tangents, always making it clear how to get back to the narrative “happy path.”

This type of mental model seems stronger when we expect most learners to be actively interacting with the experience throughout it, just as in a game.

Zooming out, I’m particularly excited about this mental model because it connects to an old notion in edtech history that still feels underexplored to me: guided discovery in microworlds. Microworlds, as originally described by Papert, are meant to be simple, useful, and general sandboxes for discovering ideas in a particular domain, designed with points of entry accessible to new learners. Guided discovery add a bit of structure to open-ended discovery learning. For instance, learners might begin in a constrained environment, then those constraints might loosen as they become more fluent—just like in many video games.

But discovery learning has a checkered empirical history: it seems many learners may need plenty of guidance. Narration—especially outstanding personal narration like Grant’s—offers an extra layer of guidance that may make microworlds more practical in many contexts.

Irrespective of their sparring mental models, I’m tremendously excited about Ben and Grant’s work with Visualizing quaternions. What will the next narrated explorable be? Are there more interesting mental models to consider—perhaps which leave behind reference to any prior media?

Narrated explorables: three mental models was originally published in Khan Academy Early Product Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

I want to introduce you to a weird grammatical trick that has really helped us orient ourselves properly when designing for students. Ready?

We don’t “teach students a skill”: we help students build a skill.

We don’t “explain to students”: we offer explanations.

We don’t “build student understanding”: we support students in developing their understanding.

The grammatical trick is: we make students the most important subjects in our sentences. We avoid action verbs which emphasize us as the subject and the students as mere verb objects. So instead of “teaching students a skill,” we “help students develop a skill”. That change emphasizes the student, now the subject of a dependent clause, as the important actor. Our “help” merely sets up the primary action.

This shift makes sure we remember two important ideas: first, how learning happens; and second, students’ agency.

We believe learning happens through student as subject, doing as verb, and ideas as object. This is the principle behind “constructivism”, which claims it’s not possible to simply transmit ideas to a student — as if we’re opening up their head like an “empty vessel” and pouring in knowledge. Instead, when a student learns something, it’s because they’ve actively assimilated new ideas by connecting them to their own uniquely-shaped internal representations. We reflect that in our sentences: learning isn’t something we can “do” to a student; they have to do it for themselves.

Even if we didn’t think effective learning demands active engagement on the student’s part, we would still want to make the student our primary subject because that position emphasizes their agency. School environments are coercive; it’s too easy to fall into a mindset in which “doing things to” students is the default. It’s dehumanizing. We don’t have to be extreme unschooling advocates to prefer leaving students with as much agency as possible. That’s the more humane route, yes, but we suspect it’s also the more effective route in the long run. If we treat students respectfully as individuals—as the central actors in their own narratives—they’re much more likely to love learning. If we treat students as passive participants in their own story, that’s likely to become their (non-)approach to learning and problem solving later in life.

Sometimes speaking in this way feels awfully contorted, as if it’s a different dialect of English. I treat those moments as interesting learning opportunities. It reminds me of E-Prime, a variant of English which aggressively clarifies the true actors in a given sentence by excluding all forms of “to be.” In E-Prime, we can’t say “the sink is dirty” (because we can’t say “is”); we have to say “someone has left dishes in the sink.” Now who’s that someone?

If some approach really resists being phrased with the student as the central actor, we may need to rethink the approach—not the sentence.

The student is the most important subject in our sentences was originally published in Khan Academy Early Product Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

Maybe we have some sense of what it means to “master” a specific skill — like “classifying triangles by type.” But what does it mean to “master” the skill of creatively manipulating understandings in novel situations?

Depth of knowledge and transfer demand

At Khan Academy, we hope to help students build real understanding around what they’re learning. We’ll have failed if learners only feel confident applying concepts in routine situations; instead, they should be able to use what they’ve learned flexibly and fluently, adapting and connecting their ideas in novel settings.

As we develop activities for students on their path to thorough understanding, we want to create a smooth ramp for students: knowing simple facts, applying routine procedures, multi-step reasoning, extended inquiry. Instructional designers often find it useful to label tasks to make sure they’re appearing at a reasonable spot in that ramp. We often use two related axes for those labels: depth of knowledge, and transfer demand.

When students have to strategize or use creative, extended thinking to tackle a task, we say that the task requires high depth of knowledge. When students have to combine familiar facts, procedures, and concepts in unfamiliar ways or situations, we say that the task has far transfer demand.

The magic in these tasks comes from their unfamiliarity. If there were a lesson about “how to solve problems where the mass of an oscillator suddenly changes,” then this task would no longer demand deep knowledge or any transfer: it would become routine application of a memorized procedure.

But we can imagine shades of gray, too. What if we gave the student a set of three related far-transfer problems like the one above, each with the parameters slightly permuted? The first problem would definitely demand far transfer — but imagine the student couldn’t figure it out. So we show them a worked solution or give them some hints to help them see how the concepts combine in this new context. Now, once we’ve done that, if we show the student a second problem (identical except for some permuted parameters), they can just find-and-replace values appropriately in the worked solution we showed them. That’s much more like applying a routine procedure. It no longer requires far transfer.

Or consider a problem like this:

If students had never seen a problem like this before, the task would certainly require extended reasoning and demand far transfer of several prior ideas from geometry. But in fact, this exercise immediately follows a lesson on Khan Academy which explicitly teaches the procedure students can use to solve for the unknown side of a triangle for which two sides and an angle are known: the “Law of Cosines.” After watching that video, students only have to execute a routine procedure to solve this problem, so we’d say it doesn’t demand any transfer at all.

All this is to say: the same problem could function successfully at a variety of depths of knowledge. Is the problem a challenging puzzle or a rote plug-and-chug question? It all depends on how the student has “chunked” their prior understanding.

Chunking

The notion of “chunks” in learning originates with The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information, in which George Miller introduces the term to describe the level of granularity which is perceived as a single unit. Two people might “chunk” information at different levels! For instance, a toddler sees words in terms of individual letters, while adults typically read words or groups of words at a glance. A toddler grasps the number “5” by counting up, one-by-one, on their fingers; eventually, with experience, “5” becomes its own “chunk” with automatically retrievable properties.

Let’s return to our triangle problem for a moment.

Students who have never seen this type of problem before must creatively combine a few chunks they might already have internalized, e.g.:

1. It’s possible to turn any triangle into two right triangles by drawing a perpendicular line from one edge (say, BC) to a vertex (say, A).
2. Triangles’ vertex angles sum to 180°.
3. The cosine of a right triangle’s vertex angle is equal to the ratio of adjacent and hypotenuse edge lengths.
4. Various algebraic facts necessary to manipulate the equations in #2 and #3.

Speaking informally and intuitively, a student is likely to find a problem increasingly difficult as the number of chunks they must creatively manipulate increases. Happily, after solving a few problems like this, the unknown-side-length-finding procedure becomes a “chunk,” a routine to be applied to triangles with unknown sides. That transformation is valuable because then this new chunk can be used as a single unit in a more complex problem, like this one:

This problem demands that a student additionally juggle several extra ideas, like properties of parallelograms. To a student who hasn’t already “chunked” the routine for finding a triangle’s unknown edge, this problem might appear totally impenetrable; a similar student who’s chunked that routine might see how to get started.

Mastery learning

Introduced by Benjamin Bloom in 1968, “mastery learning” suggests that instead of moving through a curriculum with a fixed pace (necessarily with a varying degree of understanding in individual students), classes should support students in moving at whatever pace allows them to thoroughly grasp the material before them.

Mastery learning usually requires formative assessment: students need feedback on their understanding to see where they are, where they’re going, and how they’re going. To offer that feedback, learning environments like Khan Academy typically offer students exercises with instant answer-checking. A student might try their hand at a skill like “finding missing angles in a triangle,” then receive opportunity for further instruction or remediation if they struggle — or else move on to another skill if they succeed.

One big implication of mastery learning is that students should have as much opportunity to practice a skill as they’d like. Unlike a class that moves at a fixed pace, a struggling student should always be able to revisit prerequisites, read an alternative explanation, and try some new challenges. These systems usually consider a student to have finally “mastered” a skill when they can consistently answer related problems over an extended period of time.

This sense of mastery necessarily requires repetition: repetition to remediate and repetition to prove. That’s fine for a routine skill like “classifying triangles by type.” On Khan Academy, we build one exercise for each skill, and each exercise contains around twenty problems. In this case, it’s straightforward to make many problems for this skill: each one presents a different triangle and asks students to classify it. Students are given several problems and must answer most correctly; if they struggle, they can always repeat the exercise and receive different problems.

Critically, the twenty problems within the exercise are interchangeable. They’re all asking roughly the same thing. This way, if the student needs more opportunities to practice the skill, they have equivalent material to work with.

But as we’ve seen, repetition changes the nature of tasks with high depth of knowledge or far transfer demand. Those tasks draw their meaning from their novelty because their purpose is to assess whether a student can creatively synthesize their understandings in an unfamiliar situation. And if a student struggles with a high-transfer task, we can’t direct them to a video which explains how to solve “problems like these,” because these problems are defined by their inability to be routinized in that way.

So: what does it mean to build a “mastery learning” system which accommodates tasks with high depth of knowledge or far transfer demand?

A mastery system for creative tasks

Such a system must:

1. Offer enough material for struggling students to “try again” meaningfully
2. Give students a clear sense of their progress

Such tasks no longer correspond to a specific skill; instead they’re more typically “synthesis” tasks which require students to combine a variety of concepts. In some sense, the tasks assess the degree to which the student has “chunked” some of the constituent ideas. Of course, they’re also assessing things like a student’s confidence in the face of unfamiliarity, tactics they might have for decomposing a larger problem, and so on.

To put it another way, if a student struggles with a high-transfer problem, what might we suggest? They should read a worked solution — sure. But then what? Some students might be struggling because they haven’t quite chunked an underlying concept. In that case, more challenging practice, zeroing in on the constituent concepts might help. Still other students might be struggling because they’re intimidated by open-ended problems. Those students might benefit from practice with open-ended problems involving quite elementary concepts.

Then, if a student wants to “try again”—what does that mean? We could offer them another high-transfer problem with the “same difficulty,” but to be truly equivalent, the task must be different enough to involve a distinct creative insight. What might it mean for multiple problems of this kind to be of the “same difficulty”?

We could use some kind of item response theory to estimate a difficulty parameter, but with what model? A univariate model is pretty clearly wrong, yet it’s not clear how many underlying cognitive dimensions are involved in these problems. Maybe one parameter representing a “chunk” for each constituent concept, then one parameter for “various meta-cognitive strategies and attributes for creative problems”? Hard to say.

Perhaps a more pragmatic approach might be something along the lines of:

1. Create a huge bank of these problems, involving a large number of concepts at varying levels of the curriculum.
2. Intermittently ask students to solve a small set of these problems concerning concepts they’ve already putatively mastered; their “score” is the number they get correct out of the set.
3. The student’s “mastery goal” is to slowly increase the fractions of problems in the set they get correct — even though that might start at 0.

I’m not terribly convinced by this technique. Can you think of a better approach, dear reader? Some prior art in this space? Your author would love to hear!

Mastery learning and creative tasks was originally published in Khan Academy Early Product Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

This squiggle beautifully captures the nonlinear shift from early messiness to later neatness, but—almost with a wink—it willfully obscures what’s happening during those up-and-down loops. As I’ve tried to describe our process to others, I’ve realized we can see a bit more structure through the zigs and zags:

Just turn the squiggle like a top until we’re looking down into its oscillating center—now it’s a spiral! While the contours do meander, each turn of our spiral tends to trace through the same fundamental regions:

• We center on questions, which we refine as best we can.
• We gather all that surrounds those questions through research.
• We generate approaches through ideation.
• We test the questions, the research, and the ideation
• Then we repeat.

Our team articulates our top questions each week, but we don’t formalize these phases through heavy structure: we naturally proceed through them from the squiggle’s messy start through to its tamed tail.

Of course, the activities in those quadrants look different as the project evolves. For instance, the research phase initially paints broad strokes: we’ll conduct interviews without specific hypotheses; we’ll read scholarly reviews and surveys; we’ll browse offerings in the space. On subsequent turns, we’ll have more specific questions: we’ll conduct more targeted interviews; we’ll read literature from specific authors and keywords; we’ll dig into specific players we heard come up. Similarly, the testing phase might start with paper prototypes or marketing copy—and end up with fully-usable implementations.

This cycle extends beyond the initial design work for a project. Even once the squiggle has calmed down, if we turn it on its side, we’ll see gentle loops through these phases as the production team iterates on the concept. Research may manifest as “reading support tickets”; ideation may manifest as proposing alterations of existing designs.

I was intrigued to find a similar framework articulated in defense software development by Barry Boehm in 1986:

This is a much more regimented version of the spiral illustrated above, meant to be executed in a context with contractually-mandated timelines and perhaps less generativity. It does add a few interesting notions:

• The left portion of the x axis represents the “commitment partition”: a new loop of the spiral requires an explicit decision to proceed.
• This model is acutely interested in risks, which are important but which are only one kind of question we end up refining and asking in our first phase. Often the core question for one spiral loop is something more exploratory, like “what types of activities might best help students reflect on what they’ve learned and consolidate the material?”
• Here the radial dimension represents the cumulative cost of the project. In our work, this is much less consistent: in earlier phases, research is often more expensive (many hours interviewing teachers!); in later phases, ideation is often more expensive (building prototypes!).
• The detailed activities marked along this spiral aren’t examples of what might be done at that phase: they’re meant to precisely specify the activities there. We’ve not found that level of enforced structure constructive in investigative design research.

I wonder how these models might converge! Are some of the activities in our research spiral more consistent than others? Perhaps over time we’ll notice more regularity we can use as handholds along the way.

On a personal level, at least, this type of loose consistency has been a helpful lighthouse in the sometimes-enigmatic seas of research.

Looking sideways at the design squiggle was originally published in Khan Academy Early Product Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

How might we empower learners to think critically and creatively via open-ended activities like writing and drawing online? The Early Product Development group worked with teachers across the country to explore possible solutions.

Read on at our full report!

Our report on online open-ended activities was originally published in Khan Academy Early Product Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

We’re exploring a student-driven engine for supporting open-ended problem solving. This is the theme underlying many variations:

The third step is particularly tricky because, as we all know, people are awful on the internet. Even without anonymity, bullying is a problem in schools. If we’re only asking students to grade each others’ work, we can handle misbehavior by looking for students whose peer grades rarely agree with others’… but we’d like students to produce open-ended responses to their peers’ open-ended work! Ideally, those responses won’t have to be evaluative: other formats may better stimulate further thought.

How might we help students benefit from rich reactions to their work while avoiding abuse?

One approach is to make the communication channel much narrower. For instance, we could reflect a high-level emoji reaction back to students from their peers. Interaction can still be fairly rich this way, as seen in Journey’s wordless collaborations.

We could impose moderation or involve teachers in approving students’ communications, but that would slow down the feedback loop substantially. Civil partially shifts moderation responsibility to users, asking internet commenters to rate the civility of others’ responses — and their own — before allowing them to leave a comment.

Is there a middle ground? A channel wide enough to inspire plenty of follow-up thought, but narrow enough to need less moderation? We’ve discussed a few ideas so far.

Watching a student extend your work

In math, we might show a student a peer’s strategy, then ask them to solve a new variant of the problem. In the humanities, we might show a student a peer’s essay beside their own, then ask them to draw lines between all the places they and their peer were making the same argument.

Then we can show that work back to the original peer. It may even be interesting to see the literal replay. Seeing another student understand and extend your work may itself prompt new thoughts.

Did your peer deploy your strategy exactly the way you did? Maybe they took a shortcut you didn’t think of!

Did your peer have a different angle on the same argument you were making? Did they arrange the arguments in a different order? Does that order flow better? What about the arguments they used which you didn’t use?

Relationships and connections

Arrangements and rearrangements can supply rich fodder for follow-up thought over a channel narrow enough to avoid moderation risk.

For instance, we’ve written about asking students how similar two peers’ responses are to each other. We can establish clusters of student work from that data or by directly asking students to sort each other’s work into clusters.

Once we have those clusters, we can show students their work as arranged in context of peers’ work. Do they see why others have marked certain work as similar to their own? Do they disagree with the arrangements? Can they see subgroups in the clusters?

We can also ask students to establish orderings within their peers’ work. For instance, we might jumble up a student essay’s sentences and ask a peer to arrange them sensibly. What does the original author think of the new arrangement? We can even make the system ask that question only if the peer’s arrangement is different from the original.

Or we could erase all the quotes from a history essay but leave the surrounding text… then ask a student to find the passages in the primary sources which might belong in the quoted regions. What does the original author think of the substitutions? Do they provide another angle on the same argument?

Contextual reactions

In addition to allowing students to apply emoji to an entire piece of student work, we could give them a curated supply of emoji stickers to place within the student’s work, where the feel it might apply. Maybe a particularly strong argument gets a 🔥 sticker. Or in a math proof, students could put the 😎 sticker on a specific step. Besides being interesting feedback for students, data at this granularity could also help us drive more interesting rich tasks in the future.

We’ve also explored giving students a set of highlighters where each color means something specific. For instance, we might ask a student to highlight an essay in blue wherever evidence is being used, in purple wherever an opinion is stated, and in yellow wherever an opposing argument is rebutted. The original author might be surprised to see a surplus of purple or a lack of yellow — a great opportunity for revision.

We’re continuing to generate more ideas here, but we’re also narrowing down on a small subset of our concepts for a live prototype soon. Time to start building!

Discuss this post on Reddit.

Originally published at klr.tumblr.com.

Safely showing students how others see their work was originally published in Khan Academy Early Product Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

This kind of answer wouldn’t be so surprising, except that all of these students had just solved four numerical “find the area of this polygon” problems perfectly, including one with exactly this shape.

We were testing an activity in which students would interact with each others’ open-ended explanations and ideas. They’d learn about finding the area of polygons by dissecting them into simpler shapes.

Our control group watched a video lecture on the subject, then worked through typical textbook-style exercises on the subject, like these:

Plenty of students solved every one of these exercises correctly. Typical standardized tests might suggest that these students had mastered shape dissection. But a third of those students couldn’t give even a partial answer to the “explain why” question above; another third gave poor or incomplete answers.

We didn’t design our experiment to look at this issue: we included these area-finding problems as a “dummy” activity to parallel the richer discussions in our experimental group. Maybe the issue is that the “explain why” prompt involves more algebraic knowledge. Besides, our experimental population is not necessarily representative.

But the intense gap between the procedural and conceptual assessments here still shook me.

Have you ever gotten an “A” in a class but felt you didn’t really understand the material? That seems to be exactly the disparity we’re seeing here. How many numeric-input or multiple-choice tests would exhibit the same problems?

We’ve long discussed our suspicions around scores taken from this kind of problem; we didn’t expect to confront such jarring examples so directly!

Originally published at klr.tumblr.com.

Explanations from alleged masters of shape dissection was originally published in Khan Academy Early Product Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

For many months, we’ve been exploring: how might we help students build deep understanding through writing activities on online learning platforms? We’re initially focusing on AP-level history, though we’re trying to build something general. Answering that big question has meant finding good intersecting answers to four separate questions:

1. Pedagogy. How might we define scaffolds and feedback mechanisms around open-ended activities to best support students in building understanding?
2. Interface design. How might we design interactions in these online writing activities to support, generate, and execute our pedagogy’s scaffolding and feedback?
3. Value proposition. How might these kinds of writing activities solve burning problems for teachers and naturally integrate into their practices?
4. Content creation. How might these writing activities best fit into our broader curricula, and how might we frame their structure so they can be practicably authored at scale?

In this post, we’ll focus on the value proposition question. We were confident that if we succeeded with the pedagogy and interface design questions, we’d be creating something that could really help students. But we felt we could have the most impact reaching students through the rich environment of a classroom — a context which layers on its own significant opportunities and challenges.

We knew that we’d need to deeply understand how these activities might be used in real classrooms, as part of a teacher’s broader mental models, classroom practices, and feedback mechanisms. With that understanding, we could shape the activity’s surfaces so that what happens outside the activity resonates with what happens inside the activity, amplifying both sides. And critically, that understanding would help us frame these activities in a way teachers would be thrilled to use.

We knew that we’d have to talk with plenty of teachers to build the understanding we’d need, so we began by recruiting a few dozen teachers who were willing to chat with us. Our first interviews stayed broad: we asked teachers about their background, environment, practices, values, fears, challenges, dreams. We gleaned all kinds of interesting insights, like:

• Most AP classes are now open enrollment (i.e. any student can join) meaning teachers are contending with a huge range of student capacities.
• Even in history, many teachers had over a hundred students. Grading a single essay might take weeks.
• Teachers were apprehensive of the AP’s increasing emphasis on disciplinary skills over content knowledge; they lacked strong resources for preparing students to e.g. analyze historical continuities across time periods.
• They often worked on those skills through peer work in class, but those activities are hard to orchestrate.
• There’s so much to cover that homework time has to be spent almost entirely on readings.

After synthesizing these themes and identifying our key personas, we wrote six different pitches for a hypothetical product: different angles, different pain points, different timings. Collectively, they spanned the space of opportunity we saw. Once we had our pitches written out, we formatted them like little product “baseball cards” for easy comparison and arranging.

Then, in another round of interviews with different teachers, we abbreviated the broader user research questions to focus on a card sorting activity. It went like this:

1. We showed teachers all the cards and explained that they represented hypothetical products.
2. We asked them to read through them one by one, verbally reacting along the way.
3. We told them they could try a prototype of one of these products in their class next week and asked which one, if any, they’d be most interested in trying.
4. We asked them which one seemed least useful. Then we had them rank-order the others in the middle.
5. Finally, we asked them how often they’d be excited to use each of the cards in their classes.

Their rank-ordering was incredibly useful: it helped us eliminate a few possibilities and shined light on a promising subset of the space. But teachers’ verbal reactions as they read the cards were equally helpful. The pitches prompted teachers to connect those ideas to others in their experience. They connected our cards to existing practices, told related stories from their classroom life, and imagined aloud how to integrate the product. Their words brought the hypotheses alive and helped us share our findings convincingly with stakeholders.

We took all that insight and refined our existing designs to accord with the pitches teachers had been most eager to try. We didn’t just pick the most popular baseball card: the pitches existed as a fulcrum to understand teachers’ mental models and needs. We used what we learned to synthesize a solution that could fulfill several of those cards, as well as others we hadn’t yet written.

For our next phase of interviews, we abbreviated the card sorting activity and focused on introducing teachers to a real prototype. This new set of teachers loved what we’d made (which helped validate our synthesis), and as with the pitches, the prototype prompted the teachers to share more interesting thoughts, feeding our iteration. Meanwhile, we kept our promise with the teachers who’d done the card sorting activity: we set them up with our prototype for a live classroom pilot. More great learning ensued.

It’s always a struggle to build a bridge from behavioral interviews’ broad insights to a viable prototype. By writing these intentionally-distinct “baseball cards,” we turned the initial interviews into something concrete — but not so concrete as a prototype, which often over-constrains the conversation. And while it might have seemed like we could have written those pitches without the initial broad interviews, that too would have constrained the conversation too early. At least to me, this gradual narrowing felt just right.

Originally published at klr.tumblr.com.

Sorting product “baseball cards”: bridging behavioral interviews and prototypes was originally published in Khan Academy Early Product Development on Medium, where people are continuing the conversation by highlighting and responding to this story.