One Concept, Many Fields: My Rabbit Hole Into Affordances
My flatmate and I have a small routine. Most nights we end up talking about our work. Sometimes it’s our research, sometimes something one of us read, sometimes whatever we got stuck on, and sometimes just random thought that wandered in. She works on AI alignment, AI safety and model evaluations, and I’m doing a PhD in robotics, so the conversations tend to wander too.
A part of my PhD revolves around affordances, and one night she asked me what they actually are (Which is probably what you’re wondering too). I kept it simple. An affordance is what something lets you do. A chair lets you sit, a handle lets you pull, etc etc. And it depends on who’s doing the looking, because the same chair that lets an adult sit lets a toddler climb. Then she mentioned that the word comes up in her field too.
Wait, what??
I had always thought of the affordances as something about how humans perceive the physical world and act in it, an idea robotics borrowed because robots have to do the same thing. Why would people who test AI models care about a concept from human perception?? So I did what PhD students do best, which is researching absolutely anything except our own research, and went down a rabbit hole. I read about where the idea came from and how AI safety uses it, along the way I found out that UX/UI designers have been using it for decades
This post is what I found there. Where the idea came from, how designers think about it, what it means when you’re testing AI models, and what it looks like from my own corner of robotics.
Where it all started
The word “affordance” was coined by the American psychologist James J. Gibson in his book The Ecological Approach to Visual Perception.
Back then, a common view of perception was that we first see raw features, like edges, colours, and shapes, and then our brain works out what those features mean and what we could do with them. Gibson didn’t buy it. He argued that we perceive possibilities for action directly. when you walk into a room, you don’t first see a flat horizontal surface at 40 cm and then reason that it might be sittable. You just see somewhere to sit.
He called these possibilities for action affordances. A chair affords sitting, a handle affords grasping, as staircase affords climbing, and a cliff edge affords falling. Some affordances are helpful and some are dangerous but either way they are what the environment offers to animals (to us)!
Another thing is that an affordance depends on who’s doing the acting. A staircase affords climbing to a person but not to the toddler. So an affordance isn’t just a property of the object and it isn’t only in our heads either. It is the relationship between the two. Hold on to that one because it keeps coming back.
A detour into design
The first thing I found had nothing to do with AI or robotics. Affordance turns out to be one of the basic ideas into product design and UX/UI design. The idea moved from human psychology into design through Don Norman’s 1988 book The Design of Everyday Things. Norman was interested in why ordinary objects are so often frustrating to use, and his examples are everyday objects like doors, light switches, buttons etc.
The classic example is the door. Most of us know the small embarrassment of pulling a door that was meant to be pushed. There’s even a name of it now a Norman door. The problem isn’t you. The handle is telling you to pull, because handle affords pulling. A flat metal plate on the other hand gives you nowhere to grip so the only obvious thing to do is push. Nobody should need a sign and yet so many doors have one which tells us everything about how often we get this wrong.
Later, Norman noticed that designers were using “affordance” loosely, to mean any hint about how to use something. So in the 2013 edition of the book he added another concept called Signifiers. A signifier is whatever tells you that you can do it, and where. A door can afford pushing, but it’s the flat plate (the signifier) that tells you so.
This gets even more interesting on screens. In the physical world, shape and material give strong cues. A button sticks out, a handle has a curve. But on a screen, everything is just pixels, and almost anything can be made clickable. So UI designers have to add signifiers on purpose: buttons with shadows that look pressable, blue underlined text for links, a little bar at the bottom of a phone screen that hints you can swipe up.
Get Sawera Yaseen’s stories in your inbox
Join Medium for free to get updates from this writer.
When this goes wrong, we notice it right away. Text that looks like a link but isn’t. A swipe gesture nobody knows about. An icon that could mean three different things. These are small affordance failures, and they’re a big part of why some apps feel easy and others feel confusing.
Affordances in AI safety and evaluation
And now the part I had actually gone looking for.
People who work on AI safety and model evaluation want to answer questions like: What can this AI model actually do? Could it be misused? Could it do something harmful on its own? To answer them, they run tests, often called evaluations or “evals” where the model is given tasks and its behaviour is carefully measured. If this sounds interesting to you, you can read more about them on Evals, from the ground up.
In that world, a model’s affordances are the things it has been given to act with. Tools like web browser or an email account. The ability to write and run the code, internet access, memory, a setup that lets it plan and try again over many steps, more time or more compute. A model in a chat window can only produce text. Give it a browser and terminal and it can actually do the things. Nothing about the model has changed, but what it can get done has.
In the The Causal Impact of Tool Affordance on Safety Alignment in LLM Agents, researchers tested two language models in two settings, one where they could only reply in text and one where they could use tools to take real actions. with text only, the models followed the safety rules perfectly. With tools the same models broke the same rules up to 85% of the time. Same models, same rules. The only thing that changed was what they could do. So a big part of evaluating a model is making sure it’s tested with the affordances it will actually have when people use it.
Evaluators have affordances of their own, too. Testers who can access a model’s weights, or fine-tune it, can sometimes find capabilities that an ordinary user would never reach. That helps them catch risks early. It also raises a question about what happens if someone with worse intentions get the same access.
After all this reading I realised that AI safety people are using affordances in very much spirit Gibson meant. They aren’t just asking what a model is. They’re asking what it can do in a particular setting with particular tools.
Back to affordances in robotics
This is the world I know best, and in some ways it’s where Gibson’s idea feels most at home. A robot, like an animal, has a body. It lives in the physical world, and sooner or later it has to perform some action in real world.
In robotics, knowing what an object is turns out to be the easy part. A robot can know it’s looking at a knife and still have no idea whether to hold the handle or the blade, or whether to pick it up at all. Affordances are about that second kind of knowing. A mug becomes a handle to hold, a rim to pour from, a space that can hold coffee. A drawer becomes something to pull.
Remember the relationship I asked you to hold on to earlier? This is where it comes back. In robotics, the other half of that relationship is the robot’s body, or what roboticists call its embodiment. How big it is, how far it can reach, how it grips, how strong it is. Change the body and you change the affordances. A mug that one robot can pick up without a second thought might be completely out of reach for another, and what a mug affords you isn’t automatically what it affords a robot. The mug hasn’t changed at all. The relationship has.
Some affordances also only appear after something else has happened. You can’t pull a drawer open until you’ve grasped it, and you can’t pour from a bottle with the cap still on. We move through these little chains all day without noticing. For a robot, knowing that one action opens up the next is a big part of getting anything done.
What I took away from the rabbit hole
After all this reading, what I find interesting is how three different fields ended up using the same concept and how they each point in a slightly different direction.
In UX/UI design, the questions is : Can people see what they’re able to do? The focus is on making the right actions visible and the wrong ones unlikely.
In robotics, the question is: What does the world afford the robot? The robot has to discover on its own, what it can do with the objects around it.
In AI Safety, the question is closer to: what are we affording the model? Here, people decide which tools and access and AI system gets, and then worry about what it might do with them.
Robotics and AI safety come at the same idea from opposite ends. In robotics we ask what the world offers the robot, and the robot has to work it out. In AI safety, people decide what to offer the model and then test what it does with it. One side is busy finding affordances, the other is careful about handing them out. In my field, an affordance is usally an opportunity, something we want the robot to be able to use. I hadn’t spent much time on the other side of it, on the actions a robot maybe shouldn’t have, even they’re physically possible.
Somewhere, another field is almost certainly using this concept too. That’s a rabbit hole for another night, though, because I should probably get back to my own research.
