The future of UX interaction?
Why do we so often prefer to have our spoken words converted into text before allowing an AI to respond? Perhaps the answer tells us more about human behaviour than about the limitations of technology.
For a technology that promises to revolutionise human communication, artificial intelligence has a rather peculiar problem: people don’t always seem particularly interested in talking to it.
Consider what happens when someone uses ChatGPT. They open the microphone, speak for thirty seconds, watch their words appear on the screen, correct a sentence, remove a repetition, and finally press send. The machine replies, and the user reads the answer. We call this voice interaction, although very little conversation has actually taken place. It is closer to having a personal secretary who takes dictation, except that the secretary lives inside a rectangle and occasionally knows things about quantum physics.
There is something worth examining here. If speaking is supposed to be the most natural way for humans to communicate, why do we so often prefer to have our spoken words converted into text before allowing an AI to respond? Perhaps the answer tells us more about human behaviour than about the limitations of technology.
1. The Strange Comfort of Being Understood
Part of the explanation is control. Written language gives us the opportunity to inspect our thoughts before releasing them into the world. We can edit a sentence, reconsider an argument or remove something that sounded considerably more intelligent in our heads than it did when it reached the screen. Speech is different: words arrive in sequence, often before we have completely decided what we mean.
Recording a voice message and having it transcribed offers an interesting compromise. We get the freedom of speaking without the pressure of an immediate response. We can examine our thoughts before allowing another intelligence to interpret them.
This distinction matters because conversational AI assumes that people are comfortable expressing incomplete thoughts in real time. But human conversation is not simply an exchange of sentences. It involves shared context, timing, clarification and an understanding of what someone means rather than merely what they say.
In their 2016 CHI paper, Like Having a Really Bad PA, Emily Luger and Abigail Sellen explored the gap between people’s expectations of conversational agents and the actual experience of using them.[1] The technology has changed considerably since then, but the underlying problem remains: we expect conversation, while many systems still behave like sophisticated command interfaces.
Ludwig Wittgenstein explored a related idea in Philosophical Investigations: the meaning of language emerges from how words are used within shared human practices.[2] An AI can generate a convincing sentence, but understanding the social circumstances that give that sentence meaning is another matter entirely.
Perhaps this explains why conversational AI can sound so human and still feel strangely unfamiliar. We recognise the language, but not always the relationship behind it.
2. Human Communication Was Never Just Words
We have a habit of treating language as if it were the whole of communication. It isn’t. When we talk to someone, we use gestures, facial expressions, eye contact, tone and the environment around us. Pointing at an object while asking a question is considerably more efficient than describing its precise location, colour and shape.
Maurice Merleau-Ponty, in Phenomenology of Perception, argued that human experience is fundamentally embodied: our bodies are not simply instruments through which we interact with the world, but part of how we perceive and understand it.[3] This is an important distinction for interaction design, because digital interfaces have traditionally separated information from much of the physical context in which it exists.
For decades, we organised interaction around screens, buttons, menus and forms. This made technology easier to operate, but it also encouraged us to translate human intentions into a language that computers could process.
Multimodal AI offers the possibility of reversing that relationship. Instead of forcing people to describe everything through text or speech, systems can combine voice, images, video and contextual information.
Imagine asking an assistant about a machine while pointing your phone’s camera towards it. Rather than describing every component, you can simply show it what you mean. Or imagine uploading a sketch, explaining your idea verbally and refining the result through written instructions. The interaction becomes a mixture of different communication methods, chosen according to the situation.
3. When the Interface Starts to Understand Context
Google has been exploring this direction through Project Astra, introduced at Google I/O in 2024. Its vision was an AI assistant capable of interpreting voice, images and live visual information while maintaining conversational context.[4] Elements of this approach have subsequently appeared in Gemini experiences, including Gemini Live and camera-based interactions.
Get Dàniel Dalia’s stories in your inbox
Join Medium for free to get updates from this writer.
The interesting development is not simply that an AI can look at something and describe it. It is the possibility of discussing the world as we observe it, without having to translate every visual detail into a written prompt.
IBM is approaching another dimension of this transformation through watsonx Orchestrate, which focuses on coordinating AI agents, enterprise applications and workflows.[5] Here, the emphasis is less on making a machine sound human and more on allowing people to communicate an intention that can be translated into actions across different systems.
These developments suggest that the next generation of interfaces may be less about navigating applications and more about communicating what we want to achieve. But this creates an important design responsibility. The more autonomous systems become, the more important it is for people to understand what they have interpreted, what information they are using and what actions they are about to perform.
Research by Saleema Amershi and colleagues, published in 2019 as Guidelines for Human-AI Interaction, provides useful principles for addressing these challenges.[6] AI systems should communicate their capabilities, make their behaviour understandable and allow users to correct mistakes. These principles are particularly relevant when AI moves beyond answering questions and begins taking action.
An invisible interface should never mean an invisible decision-making process.
4. The Future Is Not About Removing the Screen
It would be easy to imagine that the future of UX is a world without screens. I am not convinced that this is particularly desirable. Screens are excellent when we need to compare information, inspect details or make deliberate decisions. Writing is often preferable to speaking, particularly when privacy matters or when we need to express something precisely. Voice is useful when our hands are occupied, while visual interaction can communicate information that would take several minutes to describe.
The real opportunity is not to replace one interaction method with another, but to create experiences that understand when each method makes sense.
For UX designers, this means moving beyond the familiar territory of arranging buttons and optimising flows. We will need to think about context, perception, attention and the social expectations that influence whether an interaction feels natural or uncomfortable. We will also need to recognise when not to use AI. Not every experience needs to be conversational, and not every human activity needs to become an opportunity for technology to anticipate our next move.
We have spent a remarkable amount of human history developing ways to communicate, from gestures and spoken language to writing, printing and digital networks. Computers are now learning to participate in that history, although they have arrived rather late and still occasionally struggle with the conversational equivalent of knowing when to stop talking.
The interesting question is no longer whether we will eventually speak to our devices. It is whether we can build technology that understands why we speak, what we mean and what we are trying to achieve.
Perhaps the future of interaction will not be defined by how many ways we can communicate with machines, but by how well those machines learn to accommodate the extraordinary, inconsistent and occasionally contradictory business of being human.
And that, after all, is what good design has always been about.
Notes & Further Reading
- Luger, E., & Sellen, A. (2016). Like Having a Really Bad PA: The Gulf between User Expectation and Experience of Conversational Agents. CHI 2016. https://doi.org/10.1145/2858036.2858288
- Wittgenstein, L. (1953). Philosophical Investigations. Blackwell.
- Merleau-Ponty, M. (1945). Phenomenology of Perception. Gallimard.
- Google (2024). Project Astra: Our vision for the future of AI assistants. https://blog.google/technology/google-labs/project-astra/
- IBM. watsonx Orchestrate. https://www.ibm.com/products/watsonx-orchestrate
- Amershi, S. et al. (2019). Guidelines for Human-AI Interaction. CHI 2019. https://doi.org/10.1145/3290605.3300233
