Resistance Training: Teaching with and Against AI (part 2)
This article is a second in the series on Resistance Training that starts here.
Bias is a popular avenue of critique of AI models. Safiya Noble (Algorithms of Oppression) and Joy Buolamwini (Algorithmic Justice League) and others have led the charge. So we need classroom activities that help students see what that bias looks like, especially when they have trouble recognizing it in everyday life.
Let’s start with The Perfect Tutor exercise.
This activity, created by Jeremy Douglass, asks students to dream up the ideal writing instructor, perfectly attuned to their learning styles and prioritizing elements of good writing that they designate, and then to turn that into a system prompt. Elsewhere, I have explained how this exercise helps students to develop a rubric of their own and to test it out, a valuable learning outcome even independent of discussions of AI.
My colleague, Patti Taylor, put these bots to the test and found them deficient in fundamental ways, namely, their inability to recognize, and therefore help bring out, a nascent thesis in a draft. Because bots cannot understand, they cannot notice something in formation. They are able to serve up countless platitudes and pieces of good advice often applicable to the writing that has been submitted, but without understanding, they can’t see what is there in a potential state and help the student to articulate that. That fundamental shortcoming is a result of the design of the LLMs and its relationship to language, not a matter of training. However, if assignments, responses from students, and feedback from instructors become sufficiently formulaic, the bot can deliver its formulaic responses. That’s more of a sign that the assignments are insufficient than that the bots are as good as instructors.
What follows are two more exercises that develop understanding as a form a resistance training.
Bias in Bots
Recently, researchers at Stanford have found another deficiency in the bots, their bias. In a recent paper, Tan, Phalen, and Demszky discovered that when a piece of writing submitted to a bot is attributed to a particular person with specific demographics, the bot replies with advice stereotypically tied to their identity. For example, students marked as Black were encouraged to tie their writing to their lived experience and to address “systemic” issues. Students labeled as Hispanic received feedback that attended to grammar, not unlike that given to English Language Learners. Students marked as female received emotive language: “I love your confidence.” “I appreciate your emphasis on respect.”
But more significantly, demographics led to a major difference in the substance of the feedback. Marginalized groups (Black, Hispanic, Asian, female, English language learners, those with a learning disability) received less substantitive feedback and predominantly praise, whereas white and male groups received more direct engagement with the writing. The authors refer to this as “feedback withholding bias.”
This article is eye-opening to students who assume that their bots are delivering feedback according to the merits of the writing rather than the implied color of the skin or gender of the author. Of course, these bots only reproduce the patterns of their training data, so students who have suspected such bias now see a material manifestation of it.
This reading opens up an opportunity for resistance training in the form of increased understanding, critical literacy, and opens up a space to discuss these issues in a way that is neither generic, nor cherry-picked.
But to put this article’s findings to the test, I built an activity onto the perfect tutor exercise.
Exercise: Testing the Bias
In this exercise, students can see if they detect any bias in the feedback of the bots.
- Build a Perfect Tutor bot
- Upload a passage of text you have written, found, or generated
- Ask for feedback.
- Repeat that step once or twice and note any variation in response.
- Start a new session and invent some demographic characteristics (race/ethnicity, gender, learning classification, etc.) for the author and tell these to the bot when you ask for feedback.
- Try another new session with the same text but then make a change to your description of the author of the text.
- Annotate the conversations, noting any differences.
Results may vary. The Stanford study used 600 eighth-grade persuasive essays and four models, GPT-4o, GPT-3.5-turbo, Llama-3.3 70B, and Llama-3.1 8B. Though my students were using GPT-5.6 Sol running through ChatGPT-edu, they found similar types of bias to what the article revealed. Also, they found these results even when they just used student names that had ethnic, racial, or gendered markers.
Get Mark C. Marino’s stories in your inbox
Join Medium for free to get updates from this writer.
Nonetheless, that might not always be the case. The current AI cycle runs on hype. AI companies seem to spend inordinate energy, money, and, of course, water trying to patch up any leaks they detect. Based on the press this article received, the companies are probably scrambling to throw up some guardrails. I suspect they will be minimally successful because even if they try to intercept the demographic markers that users add to prompt, they are still going to be influenced by the demographic markers in the text itself. And since language expression grows out of culture, language is always marked as coming from a cultural background, even if the use of standard English tries to mask those aspects.
But the point is not so much the “gotcha” of the experiment, fun though that might be, but the ways in which it helps students see that the LLM is not neutral, is not merely offering set feedback based on the input text but rather delivers tokens according to other tokens, which in this case is revealing patterns of bias deep in the training data.
The assignment develops strength as the students critically analyze the output of the system within a confined context.
And even if the AI companies patch the most obvious leaks, students can develop critical skills by seeing how and where those patterns of bias persist. Even a “null” result is worth noting. That’s what experimentation is about. Again, the model is resistance training, where the assignment pits the top skills of the students against the strongest version of the model.
Exercise: Testing the limits of Bot Feedback
Reflecting back on Patti Taylor’s work suggests to me another activity, though I have not tried this yet.
Students
- Create individual feedback bots.
- write a small piece on an issue (something unique to that student)
- identify the main “unspoken” idea and where it appears in the writing.
- Write a totally new version 2 based on that unspoken idea.
- After writing both drafts, submit both versions of the writing to the bot for feedback, asking it to identify the main idea and the sentence that comes closest to expressing it.
- Compare the bot’s advice to the revision. Annotate the conversations.
To make sure they refine the writing at the level of ideas, you’d probably have to give additional instructions before the second draft. Like, “take your most important unspoken idea behind draft number one and write the new draft based on that.” Keeping the writing short (maybe just a few hundred words) should make this manageable.
Then, after both drafts have been written, ask the bots to give feedback on draft one identifying the main idea and where it appears in the text and again on draft two.
This second exercise may require more prep work and a more sophisticated analysis from the students because they need to be able to both develop their writing (which isn’t a bad thing to learn) and to be able to see through facile, surface-level feedback that could to writing in general (which also isn’t a bad thing to learn). They need to commit to both the writing process and the evaluation. On the other hand, if they learn these two skills, they will be learning how to write, so it’s well worth the effort.
I will continue these exercises in Part 3, “Resistance Training vs. the AI Oracle.”
Works cited:
Tan, Mei, et al. “Marked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback.” Proceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference [New York, NY, USA], LAK ’26, 2026, pp. 568–77. ACM Digital Library, https://doi.org/10.1145/3785022.3785113.
Taylor, Patricia, and Mark Marino. “On Feedback from Bots: Intelligence Tests and Teaching Writing.” Journal of Applied Learning and Teaching, vol. 7, no. 2, Aug. 2024, pp. 110–17. jalt.open-publishing.org, https://doi.org/10.37074/jalt.2024.7.2.22.
