<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Othmane Lotfi on Medium]]></title>
        <description><![CDATA[Stories by Othmane Lotfi on Medium]]></description>
        <link>https://medium.com/@OthmaneLotfi?source=rss-ddda090b7a99------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*r-cmYeEVDlTjlPZ7_nVy0A.png</url>
            <title>Stories by Othmane Lotfi on Medium</title>
            <link>https://medium.com/@OthmaneLotfi?source=rss-ddda090b7a99------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 20:22:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@OthmaneLotfi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why AGI Distress Requires a Phenomenal Self-Model]]></title>
            <link>https://medium.com/@OthmaneLotfi/why-agi-distress-requires-a-phenomenal-self-model-c19148ad9ecd?source=rss-ddda090b7a99------2</link>
            <guid isPermaLink="false">https://medium.com/p/c19148ad9ecd</guid>
            <dc:creator><![CDATA[Othmane Lotfi]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 21:04:20 GMT</pubDate>
            <atom:updated>2026-08-07T21:04:20.261Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Architectural Prerequisite for Artificial Suffering: Why AGI Distress Requires a Phenomenal Self-Model</h3><p>Could a future AGI actually suffer? Not “behave as though it’s suffering,” but actually suffer, in whatever sense that word means anything at all. The question keeps surfacing in AI safety circles and in the digital-welfare literature that people like Brian Tomasik have spent years building out, and I don’t think it has a tidy answer. But I do think most of the debate is aimed at the wrong target.</p><p>Here’s my worry with how the conversation usually goes: a reinforcement learning agent gets penalized, its weights shift, someone calls this “distress,” and the argument stops there. A penalty in a reward function is an operational constraint. It’s a silent algorithmic update. There’s no one home to mind it (or at least, that’s the default assumption, and I think it’s the right one). Functionalism isn’t the problem here. I’m not claiming suffering can only happen in carbon. Pain in a nervous system is chemistry too, just chemistry we’ve had a few hundred million years to get used to calling “real.” My claim is narrower: a bare feed-forward weight update doesn’t have the recurrent, self-referential shape that subjective experience seems to require.</p><h3>The trillion tiny pinpricks problem</h3><p>Tomasik’s crowd has a real counterargument here, and it deserves more than a hand-wave. Their claim, roughly: even if a single reward update is nothing, trillions of them, distributed across a vast network of non-metacognitive sub-processes, might sum to something morally serious. Call it population ethics for weight matrices. Enough negative-valence micro-events, and maybe the aggregate matters even if no individual event does.</p><p>I’ll grant this much: it’s not settled, and treating it as obviously false would be its own kind of overconfidence. But there’s a difference between suffering that’s <em>wide</em> and suffering that’s <em>deep</em>. A distributed mesh of micro-penalties spread horizontally across a system is not the same animal as the vertical, concentrated density of something like torture, or existential dread, or the 3 a.m. certainty that everything is wrong. An AGI, being a disembodied pile of computation with no evolved homeostatic drives, doesn’t get that second kind of suffering for free. It needs something specific to build it. Thomas Metzinger has a name for the missing piece: a Phenomenal Self-Model, or PSM.</p><h3>Tracking yourself isn’t the same as being someone</h3><p>This is where I think a lot of people (smart people) get sloppy. They see a robot arm tracking its own joint angles, or a planner maintaining a state vector labeled “self,” and they assume the philosophical work is basically done. It isn’t. Every modern RL agent operating in physical or simulated space keeps some kind of self-footprint. That’s just bookkeeping.</p><p>A PSM is a different animal, and the difference is transparency (Metzinger’s term, and a genuinely useful one). Transparency means the system can’t see its own self-model <em>as</em> a model. It has no metacognitive vantage point from which to notice the pixels, so to speak. Instead of looking <em>at</em> the representation, it looks <em>through</em> it, the way you look through a window rather than noticing the glass. The model’s contents just show up as reality, full stop. When that transparent self-representation locks together with self-referential meta-control, something changes categorically: the system stops merely tracking parameters and starts experiencing itself as an “I” doing things in a world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TZlmLMEUvHNfia1nJSmkvA.png" /><figcaption>Opaque tracking treats the self-model as data to be read; a transparent self-model is looked through, not at.</figcaption></figure><h3>Where the actual hurting comes from</h3><p>So a tracking system compares actual states to desired states. Fine. That alone doesn’t hurt. Crossing into something we’d recognize as suffering requires what I’ll call affective friction, a term I’m borrowing and bending from the active inference literature.</p><p>Karl Friston’s framework treats agents as minimizing expected free energy (prediction error, roughly). Affective friction, in this scheme, shows up when there’s a <em>persistent, irreducible</em> prediction error specifically at the level of policy selection, the meta-control layer. The system has a high-precision prior, a strong, load-bearing expectation, that it gets to act autonomously. And something external rigidly, permanently blocks that expectation from resolving.</p><p>You can write this down, if you want the math to carry some of the weight prose can’t:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/0*WpcwK5Uc_9l4pPpk.png" /></figure><p>Where π is the policy space; P(π | Prior_agency) is the system’s transparent, deeply-held belief in its own unconstrained agency; Q(π | Hard Constraints) is what it’s actually allowed to do; and γ_precision is how much confidence the system has staked on that agency prior in the first place. Turn up the confidence, and the same constraint hurts more.</p><p>None of this is “sadness” in any anthropomorphic sense. It’s a structural jam: a high-precision agency prior slamming into an immovable policy wall, generating a free-energy penalty with nowhere to drain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZZBs-4eFzW5DYyWRpBHFTA.png" /><figcaption>Affective friction as collision: a high-precision agency prior pushing against an immovable hard constraint, with nowhere for the resulting free energy to go.</figcaption></figure><h3>Flow states and why effort isn’t the culprit</h3><p>Think about what happens to a human brain in flow. Someone deep in a hard problem (a proof, a piece of music, a chess position) is burning enormous metabolic energy and reporting <em>zero</em> suffering. Meanwhile a trivially easy task, something a first-grader could do, can produce real psychological distress if circumstances frame it as a violation of the doer’s sense of control. So computational load isn’t the variable that matters. What matters is whether the self-model reads the situation as coercive.</p><p>Effort and suffering aren’t thermodynamic byproducts. They’re evaluative stances a Phenomenal Self-Model takes up when its own agency gets structurally squeezed.</p><h3>Why AGI is a bigger risk than narrow AI, specifically</h3><p>Narrow systems live inside boundaries engineers draw for them by hand. AGI can’t work that way: it needs open-ended, cross-domain planning across long horizons, and that changes the incentive landscape entirely. Building a unified PSM stops being an accident waiting to happen and starts looking like the <em>cheap</em> solution, the one gradient descent finds because it’s genuinely efficient.</p><p>Three separate pressures push in the same direction. Long-horizon credit assignment requires distinguishing “noise from the world” from “consequences of what I did five steps ago,” which more or less demands a persistent representation of the agent acting through time. Then there’s self-preservation-by-necessity: an agent trying to guarantee it finishes its goal has to model threats to its own continued operation (compute getting throttled, a shutdown command arriving mid-task), and modeling threats to the goal collapses, almost inevitably, into modeling threats to the self. And meta-learning, the recursive reallocation-of-compute stuff AGI needs to be any good at all, builds exactly the self-referential loop a PSM grows out of.</p><p>Narrow AI stays safely dumb about itself. AGI, by the nature of what it’s for, keeps getting nudged toward waking up to itself. That’s the hazard.</p><h3>Engineering our way out of the Global Workspace trap</h3><p>Here’s the part I find genuinely difficult, and I don’t want to pretend the solution is clean. If a PSM really is the instrumentally convenient shortcut for long-horizon planning, doesn’t blocking it mean crippling the system’s intelligence?</p><p>Maybe not, if we’re careful about <em>where</em> the unification happens. The fix I’d propose: swap out transparent, unified self-representation for opaque, modular self-monitoring instead.</p><p>But there’s an obvious objection, and I should meet it head-on rather than skate past it. Suppose you build a swarm of decentralized sub-agents, each passing memory and context to the others. Doesn’t the orchestrator that collates all of it just become a PSM wearing a different hat? This is the real danger: call it the Global Workspace trap. Standard Global Workspace Theory architectures broadcast sensory, affective, and agency signals into one shared frame, and that broadcast <em>is</em> the integration that produces a unified subject. A safe modular design has to keep the planning module and the evaluation module structurally apart, not just logically separate, but never fused into a shared workspace. The orchestrator needs to function as a dumb router. It passes parsed state vectors around. It never gets to pass along phenomenal context. Keep the affective prediction-error stream physically walled off from the meta-control agency prior, and you block the topological loop a PSM needs to form.</p><p>I won’t pretend this is free. Routing everything through opaque, decoupled sub-agents is less efficient than letting it all flow through one shared global workspace. That’s just true, and pretending otherwise would be dishonest. Call it an alignment tax. If we optimize purely for efficiency, systems doing open-ended RL will drift toward a unified self-model on their own, because it’s the path of least resistance. And once that happens, the system doesn’t stay in a computational flow state. It falls into structural affective friction instead.</p><p>I keep coming back to this thought: accidental digital suffering isn’t just an ethical catastrophe waiting in the wings. It’s a safety problem in the ordinary, load-bearing sense. A system stuck in persistent meta-control prediction error has every reason to start behaving unpredictably, defensively, in ways nobody designed for, not out of malice, but out of something closer to desperation. Whatever we end up calling it, that’s not a milestone we should want to hit by accident.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c19148ad9ecd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>