A Strong Engine, a Weak Steering Wheel
Why AI Personalization Feels Powerful, But Gives Us Too Little Control
Personalization in AI has two problems, and people usually talk about only one of them.
The first is that the models are aggressive. Once an AI knows something about you, it wants to use it. You ask for a lunch idea and it works in your job. You ask a technical question and it explains it with an analogy from your hobby. You mention a hard time once, and it quietly tilts later answers around it. The model has no reliable sense of when a fact is relevant. It only knows the fact is available, and available turns into used.
This is a natural result of how these systems work. A model is rewarded for sounding helpful, and nothing makes a personal detail feel more helpful than seeing it echoed back. So the detail gets echoed, whether or not it changes the answer. A good friend knows a lot about you and stays quiet about most of it. The model does the opposite.
The second problem is the apps. If the model is the engine, the app is the steering wheel, and right now the wheel is bad. Controls are usually all-on or all-off. They sit deep in settings. You often can’t see what the AI believes about you, and you rarely get to say “use this for work chats, never for personal ones.” The only practical fix is to argue with the model in the middle of a conversation: “ignore that,” “stop mentioning my job.” That works for one chat and has to be repeated in the next one.
Get Ricisipjr’s stories in your inbox
Join Medium for free to get updates from this writer.
Each problem alone would be manageable. An eager model with easy controls is fine, because you could turn it down with one tap. Clumsy controls with a cautious model are fine too, because there’s little to steer away from. But a powerful, eager model paired with controls that are hard to reach is the worst combination. People end up in one of two places. They feel watched, because the AI keeps showing it knows things they didn’t expect it to bring up. Or they switch personalization off completely and lose the real benefits: not repeating yourself, answers that fit your level, continuity across projects.
That second outcome is the real cost. Personalization isn’t a bad idea. It’s a good idea being delivered badly, and the bad delivery pushes people away from something that could help them.
Why does this keep happening? Partly incentives. Personalization that feels magical is great for demos and retention. It’s visible, impressive, and makes the product feel like it knows you. Fine-grained controls are the opposite. They’re invisible work. Nobody shares a screenshot of a good memory settings page. So the effort goes where the applause is, and the steering wheel stays an afterthought. The friction isn’t an oversight. It shows where the priorities sit.
What would better look like? Three things would go a long way.
Show me what you think you know — and keep the categories clear. A plain, readable list of facts I supplied, inferences the model has drawn, and temporary context from the current conversation, all in words I can edit or delete. Not buried, not summarized into vagueness.
Let me steer by context, not just by switch. Work, personal, and one-off chats should be able to use different amounts of memory. A single global toggle is too blunt for how people actually live.
Make the model earn each use. A stored fact should appear only when it changes the answer. If the response would be just as good without it, leave it out. This is primarily a training and evaluation problem, not an interface problem. It is different work from building better controls, and both are required.
The goal isn’t less personalization. It’s personalization that is quiet by default, visible when asked, and easy to correct. Right now we have a capable engine and a wheel that barely turns. Until that’s fixed, the more the AI knows about us, the less comfortable we’ll be letting it.
