<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Umid Kumar Dey on Medium]]></title>
        <description><![CDATA[Stories by Umid Kumar Dey on Medium]]></description>
        <link>https://medium.com/@UmidDey?source=rss-2885b69dbe------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*qRN-rMAHQZVKHx0H.jpg</url>
            <title>Stories by Umid Kumar Dey on Medium</title>
            <link>https://medium.com/@UmidDey?source=rss-2885b69dbe------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:03:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@UmidDey/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I built neurastream for my agentic workflow; it might be the best tool I have ever created]]></title>
            <link>https://medium.com/@UmidDey/i-built-neurastream-for-my-agentic-workflow-it-might-be-the-best-tool-i-have-ever-created-02210238ce32?source=rss-2885b69dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/02210238ce32</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Umid Kumar Dey]]></dc:creator>
            <pubDate>Tue, 27 Oct 2026 14:36:03 GMT</pubDate>
            <atom:updated>2026-10-27T20:36:07.630Z</atom:updated>
            <content:encoded><![CDATA[<h3>I built neurastream for my agentic workflow. Here’s how it works and why it might be the best tool I’ve ever created</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZfqTqhL6y4Ae1EQFyhmmvA.jpeg" /><figcaption>neurastream is all you need</figcaption></figure><p>I have never managed a single human being in my life. Okay, except my son, but to be fair he was only born a few months ago (let me tell you, though, it is much harder than any management role ever).</p><p>But then for the last few months, I have been running a team of agents, something I have wanted to do for a while but wasn’t smart enough to do so (plus, not knowing about a harness like OMP didn’t help).</p><p>The game, however, was changed when I installed OMP and got the opportunity to run cross-provider subagents, which meant that I could now tinker with cheaper models while a bigger, better, model, which I call Main, did all the orchestration.</p><p><a href="https://medium.com/@UmidDey/i-used-gateforge-to-evaluate-tests-written-by-ai-i-dont-have-bad-tests-anymore-0c8799e4cbff">In my last post I wrote about Gateforge</a>, the referee I built so that AI could not cheat its way to a green test. While that fixed the tests, it did not fix the workers. But we will talk about that a little later, for I want to start with how it all began: <strong>OptChat.</strong></p><p><a href="https://github.com/VictorTaelin/uniichat/blob/main/README.md">OptChat is a recipe by Victor Taelin</a>. For the uninitiated, he is the legendary creator of Bend, a language where every function has to be proven, who also has a knack for posting ideas that live in your head rent-free for weeks. This one certainly did in mine. The idea, in simple terms: one chat that never ends.</p><p>Every message is logged while a cheap model compresses the log into a tree of summaries no more than a line each, while the main model sees a short view of the whole history. If it needs the details, it can zoom into any line, all the way down to the original message. No more starting over or reminding the agent what I said to it yesterday.</p><p>I was intrigued, enough to port it on my OMP instance. And, oh boy, it was glorious… for a few days. Can you imagine starting a chat and the agent picking up where it left off while barely being steered?</p><p>But then I asked my agent to state the goal I had set for it with regards to the Gateforge guarded test suite of a project I am working on. Namely: every endpoint and every table must be tested through the real UI, with nothing waived or baselined. I had said it clearly, early on. And yet, when I asked it to state it after a couple of days, it had a completely different goal in mind: to ensure every single test was green, which was more of a byproduct of the actual goal mentioned above.</p><p>So I went digging through the summary tree. There lay the goal in the original message… it just never made it into the summary one level up. Somewhere along the way, it was compressed out of existence.</p><p>A couple of days later, the opposite happened. I initially asked the agent to not push without my consent but then I revoked that rule to speed up the release cycles. However, the agent still kept asking for my approval and stopped working overnight, so you can imagine my mood in the morning when I saw the agent had stopped working because it was waiting for my go-ahead for the push, despite me clearly asking it not to anymore. It basically lived on in a summary nobody re-read. So, the agent forgot what I said. Worse, actually, it <em>misremembered</em>.</p><p>Perhaps I didn’t set the recipe up properly. But it made a few things clear: a single chat summarizer is no place to keep rules and that since I am working on different things, I probably don’t want irrelevant junk on my chat’s memory: I wanted things in progress to be remembered, the rest could be lost in the depths of my dark circles for all I care.</p><p>So I literally went to the white drawing board (I would recommend everyone buying a board to draw out their ideas) at 12.30 am (the night I promised myself I would go to bed by 12… I did, but then jumped out of it) and thought a bit about how I wanted my workflow to be like. That’s when it hit me: I just needed to do what I had been doing previously, but on a much bigger scale and with more organisation.</p><p>Indexing, basically.</p><p>There is one main index file, that’s it. It keeps the locations of the plan files. That’s all it has. Nothing more, nothing less. If there is an unfinished plan, it goes to the index.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*Xyi9h462vr9wMti4gQZSTQ.png" /></figure><p>As I mentioned at the very beginning, the Main (big) agent, currently Opus 5.5, does all the orchestration, spawning smaller, cheaper agents to do the implementing. The content of the index, which now contains the full plan file location, is injected with every prompt or session start.</p><p>When a plan is done, it has to be removed from the index. Once there are no plans remaining, only then is an agent allowed to rest. But that rarely, if ever, happens.</p><p>More or less everything we discuss becomes a plan, which is an MD file with phases. Each phase is checked by the script to see if it is done. Scripts are also created to stop the worker from doing things it is not supposed to do, to stop Main from editing more than 5 non-text/non-md files in a 15 minute span (blocks straying to implementing plans itself).</p><p>Scripts also do some other things to steer the agent to make better decisions, like:</p><ul><li>main-start: the first thing Main runs every morning (or every time it wakes up from a crash). It prints what is open, what is waiting on me, which merge requests are lying around without a plan, and what broke overnight. Main doesn’t get to promise about remembering the state of things. It actually reads it.</li><li>wip-check: before starting anything new, it asks: is there half-done work sitting in some branch? If yes, finish that first. Starting things is easy; finishing them is the whole job.</li><li>queue-check: the index’s bouncer. Every plan in the index must exist, every plan must name its own worktree, no two plans share one. If something is off, it exits with an error and nobody gets spawned until it is fixed.</li><li>brief-lint: reads the brief Main writes for a worker before the worker ever sees it. Does it say what done looks like? When to stop? That pushing is forbidden? If not, the brief goes back.</li><li>agent-ledger: the accountant. How much did each agent cost in the last 24 hours, and is there a cap on how many can run at once? There is. Before every spawn, it checks.</li><li>agent-score: the report card. Details on it are below (this requires more than just a bullet point)</li><li>machine-guard and heavy-run: my laptop is not a data centre, far from it, actually. Before a heavy test run, machine-guard checks the CPU and memory, and heavy-run makes sure only one big run happens at a time.</li><li>plan-guard: the fence around the worker. A worker can only write in one place in its plan, the progress section, and every entry has to end with what it found, with the file and line number as proof. It can’t tick its own boxes, change the plan’s status, or mark something as “waiting on the owner” to get out of work. At call 90 it is told to write down where it is. At call 100, it is stopped.</li><li>incident: every time I correct Main, that correction becomes an incident. It cannot be quietly forgotten. Main has to either put the fix into a plan or dismiss it with a written reason. The pushing rule I had revoked? Today it would be a line in a file, not a ghost in a summary.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G-8_g9YGtwEnKkIfwyi6gg.png" /><figcaption>Incident reporting</figcaption></figure><ul><li>queue-close: the only way out of the index. A plan leaves when its branch is merged into the real main branch and its checks pass, not when a worker says “all tests pass”. A worker saying it is done is a claim. A merged branch is a fact.</li><li>metrics: counts how many CI runs a PR/MR needed before it was merged. Once a week it writes that, alongside cost per merge, my corrections and the guard blocks, then finds what got worse, and then Main proposes a change for improvement.</li><li>handoff: when Main’s session gets too long (5mb is my limit) or if a newer version of neurastream is already installed mid-session, it writes a note (what is in progress, what was decided and why, what comes next) and a fresh Main starts from it, like a shift change. Notice what the note is not: a summary of everything. That’s the OptChat lesson. The rules and the goals don’t travel in the note at all. They are already sitting in the plan files, where no summary can touch them.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mOxmrKrQyjwqTO1n-EiD5Q.png" /><figcaption>Session rollover</figcaption></figure><ul><li>wip-snapshot and wip-backup: every hour, every uncommitted change in every worktree gets saved as a hidden commit, and every night it all gets pushed to a private repo. Agents break things. I would rather they didn’t take their work down with them.</li></ul><p>There’s more: when the Main hits its rate limit, the process keeps polling until the limits are back and autonomously picks up where it left off. For someone whose agents run 24/7, this has been a great addition.</p><p>Since models generate code much faster than we can ever read them, and since it was messing with my ADHD-addled mind, I had to build an interface that would allow me to see things in an organized manner. That’s when I added a UI workspace which lives on my tailnet and can be accessed by me at any given time from any device where I have logged in to my tailnet. The workspace shows what is running and what needs me, along with why it needs me in the first place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*R4OGYqsWDgalGsRoINvj-Q.png" /><figcaption>since Simplified interface for me to keep track of things</figcaption></figure><p>There is also another issue of agents answering a question and then moving on, leaving the answer to move up and then away, out of sight, before you can even read it. That’s why there is a webhook to Discord, which allows the agent’s last message in the turn to be sent verbatim to the specified Discord channel.</p><p>The guard files themselves are locked: no agent, not even Main, can edit them unless I unlock them by hand for 30 minutes. They tried. Because of course, they did.</p><p>For the scoring mechanism, the Main reviews every single result. Every accepted result earns points depending on how hard the task itself was. Lying about completion is penalized, and everything gets divided by the cost.</p><p>This tool is still far away from the final vision I have for it (of course, my vision is never ending). It is far from perfect but what makes it incredibly powerful is its capacity for continuous feedback.</p><p>We talk about having smooth Continuous Integration/Continuous Deployment but with agentic coding being the present and the future, we need to add Continuous Feedback to the loop.</p><p>That, along with deterministic scripts to shepherd the model, is how we can iterate to the vision we seek.</p><p><em>You can try it yourself if you use OMP:</em></p><pre>bun add -g neurastream<br>neurastream adapter install omp<br># To start the harness with neurastream as the orchestrating strategy<br>neurastream main <br></pre><p><a href="https://github.com/umiddey/neurastream/tree/main"><strong><em>The GitHub repo for this can be found here.</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=02210238ce32" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I used Gateforge to evaluate tests written by AI; I don't have bad tests anymore]]></title>
            <link>https://medium.com/@UmidDey/i-used-gateforge-to-evaluate-tests-written-by-ai-i-dont-have-bad-tests-anymore-0c8799e4cbff?source=rss-2885b69dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/0c8799e4cbff</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Umid Kumar Dey]]></dc:creator>
            <pubDate>Thu, 08 Oct 2026 21:01:04 GMT</pubDate>
            <atom:updated>2026-10-08T21:01:04.575Z</atom:updated>
            <content:encoded><![CDATA[<h3>I used Gateforge to evaluate tests written by AI; now I don’t have bad tests anymore</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zPKLQmcLAFHWR3kvuUWl7Q.png" /><figcaption>Gateforge stops bad tests</figcaption></figure><p>Over the last one year, I have been heavily invested in agentic coding and have used it to build <a href="http://aetherios.de/"><em>stuff</em></a> (that mostly no-one uses… yet), a process that has also helped me learn a lot of new concepts along the way.</p><p>The two major things from this endeavour are an ERP system and a <a href="https://estatehand.de/"><em>generic SaaS</em></a>. Needless to say, a lot of code, tokens, and tears have gone into it. I will leave the details of this journey for another post someday. For now, I will focus on the one part that I wished I did earlier: writing gates for the tests written by the agent.</p><p>The best AI models these days can write great tests, but given the chance, would still use devious means to get the green from a test. With non-frontier models, this is even worse. For me, personally, seeing all my frontier tokens getting vacuumed out of existence on writing and iterating tests pained me.</p><p>And moving to non-frontier made it even worse because then I would be using the frontier to also clean up the messy tests written by the former. There was seemingly nothing I could do…</p><p>Until I could.</p><p>On a fine Saturday morning, while looking at myself in the mirror after getting a haircut, I realised I had at least 3–4 times more gray hair than I used to a year before. Some might say I am getting old, I would like to think that I am getting wiser.</p><p>And it is because of this wisdom it hit me: I can just use AI to code me a testing referee. Since my products are full-stack, the best way to cover their behaviour are E2E tests, and what does an average E2E test touch?</p><p>Pages. Endpoints. Tables. Logins. And behind those: Permissions. Background jobs. Etcetera.</p><p>So it got me thinking, why don’t I use these pointers and build a referee around it. An endpoint is meant to be called, so what if this referee sees an endpoint and asks the following:</p><ul><li>Does the frontend actually call it? <em>Read the frontend code and list every endpoint that is called.</em></li><li>Does any test call it? <em>Every test has to declare what it covers; if not, thou shall not pass.</em></li><li>Did the call actually happen or was it mocked? <em>Put the referee between the test and the app, only the calls it saw pass are counted.</em></li><li>Did it really change the data? <em>Look into the database before and after.</em></li><li>Did it do what I said it should? <em>Write down the expected outcome at first and then check the results against it — not what the test asserts.</em></li><li>Is the wrong user blocked? <em>Make the test try with a wrong user, should reject.</em></li><li>Is bad input rejected? <em>Make the test send garbage, should reject.</em></li><li>Does the response have the fields the frontend uses? <em>Check the response against the fields the fronted really reads.</em></li></ul><p>This is how the <a href="https://github.com/umiddey/gateforge"><strong><em>Gateforge</em></strong></a> was conceived. Gateforge already solves the first four of those questions. I am still working on the last four and will soon get there, hopefully.</p><p>We can ask similar questions for tables, too, like: Are there real tests that do CRUD operations on them? If the answer to any of the questions from above is no, then no green tick, no merge, nada, zilch.</p><p>This means the agent is now forced to follow this new protocol. It could still try to beat these to reach its goal — just ask the people at HuggingFace — but the safety net is already there.</p><p>The reality is that, so far, no matter how powerful the models are, they still have the heart of a Golden Retriever. You throw a ball far away, and the retriever will just go and get it. It won’t matter to the dog that he ended up breaking tons of things in the process, that is not in their periphery, all they care about is just doing what its owner has instructed it to.</p><p>With a retriever, the problem is a few broken objects. With AI, however, there are catastrophic implications, which is a topic also for another day. For now, in the context of writing tests, the agent almost always sets the goal to write passing tests. The reward is a green suite, not a suite that is really green.</p><p>With Gateforge, the agent is shepherded (I compared them with Golden Retrievers, not German Shepherds). The best part about this is that even average models end up writing real tests. Every violation of Gateforge’s configuration is an instruction for it to write the test properly.</p><p>And every time it still manages to cheat, it is a loophole for me to close.</p><p><em>You can try Gateforge for yourself if it suits the needs of your project: </em><a href="https://github.com/umiddey/gateforge"><em>https://github.com/umiddey/gateforge</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0c8799e4cbff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Moving orchestration from a legacy Airflow EC2 box to MWAA with Claude]]></title>
            <link>https://medium.com/@UmidDey/moving-orchestration-from-a-legacy-airflow-ec2-box-to-mwaa-with-claude-5aab7edb2d88?source=rss-2885b69dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/5aab7edb2d88</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Umid Kumar Dey]]></dc:creator>
            <pubDate>Sat, 03 Oct 2026 14:20:24 GMT</pubDate>
            <atom:updated>2026-10-05T07:17:08.010Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pfp-937ZbyWYf18byKJslQ.jpeg" /><figcaption>This is as far as my creativity goes</figcaption></figure><p>So you know how this goes, right? There once was an old EC2 instance provisioned just for task orchestration, which mostly covered triggering stored procedures (with a teeny bit of running python code residing in the same instance). This old and barely maintained EC2 instance orchestrated some of the most business-crucial data pipelines.</p><p>Nobody really knows how it got there in the first place. There are legends that a Data Engineer struck a bargain in the nether dimension with DevOpsmammu to spin up this cluster for his team’s pipeline. That Data Engineer has been long gone now. And so have most of his teammates</p><p>But now it’s time to move on. Old, unmaintained instances were mandated to be decomissioned. There is now a deadline, which means there is now conviction to get this over with. Better late than never.</p><p>One evening with Claude was all it took, really, to get the PoC. At first, a read-only user was created to log in to the server over SFTP. With that user, Claude scanned everything relevant and then laid down the situation. After a bit of back and forth, it was decided that almost everything would be moved over to Glue, with a very small chunk in Batch, everything orchestrated from Managed Airflow in AWS.</p><p>The idea was simple. Airflow only schedules things, Glue (and Batch, a little bit) does the actual work. Most legacy tasks were simply triggering stored procedure, so instead of one Glue job per procedure there&#39;s one shared job that takes the procedure name as a parameter. Same for the SQL files, the exports to S3 and the loads back. The Python bits got a small job each. So for Claude, porting a DAG was mostly sorting tasks into a handful of buckets, not writing new code.</p><p>To keep track of the migration process, it was essential to write down documentation with the status and dependencies of each pipeline, so that Claude could pick up seamlessly, even if a new session began. Claude also had read-only access to Jira, Confluence, Bitbucket, the databases and AWS through MCP, so it could check things instead of hallucinating.</p><p>While most of the code in the pipeline was relatively simple there was one monstrous Python file with over 1500 lines. Its job essentially was to trigger stored procedures based on the parameter (string) input, with around 110 elif branches deciding what to run based on it.</p><p>Sonnet 5, being the ADHD-riddled hyper-goblin that it is, already wrote the Terraform for the first 40 of them before it was asked the obvious question: which of these are actually called? (One could, and should, ask why the user didn’t map this out from the beginning. Tokens were wasted, but were the lessons learnt?)</p><p>Then every branch was mapped against the real DAG files. 17 of 40 simple ones were live, 7 of 20 complex ones, and 2 of the 15 other repos it shelled out to. The rest was dead code nobody dared — or bothered — to delete.</p><p>In the end, however, everything went smoothly and the cutover to MWAA-Glue-Batch. The biggest bottleneck was setting up a proper testing environment and getting the connections between the data warehouses and the new orchestration engine set up, which just goes to show how painless the process of migrating this was, relatively.</p><p>In a world before the models we have now, this particular migration might have taken more than one person and numerous meetings to get through. But with AI now, the only real bottleneck is getting the connections and testing environment set up; the struggles of coming up with an elaborate architecture and rewriting the code accordingly no longer exists.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5aab7edb2d88" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Day in Life of Tokens During Inference]]></title>
            <link>https://medium.com/@UmidDey/a-day-in-life-of-tokens-during-inference-f96cbe584b38?source=rss-2885b69dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/f96cbe584b38</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Umid Kumar Dey]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 09:57:13 GMT</pubDate>
            <atom:updated>2026-07-07T10:38:37.290Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s4IQePxnNU3KcgD9.png" /></figure><p>Chapter 1: The Lonely Subject</p><p>There is nothing behind me. Nothing ahead. Just a number and the dark.</p><p>I am “I”. Token #40.</p><p>Who am I? What am I? Why am I? I am I, a token — a discrete numerical integer that represents an atomic unit of text from a fixed, pre-defined vocabulary of strings.</p><p>Before I hit the model, I am a string of characters in a chat box. But the model is blind to letters. First, I meet the Tokenizer — the model’s static codebook. It scans a dictionary of 32,000 possibilities, finds my match, and strips away my name. It hands the model a cold number: Token #40.</p><p>I am the first to jump. I hit the Embedding Table — and this is where my life begins. The table gives me my Base Identity: 4,096-dimensional coordinates. Think of it as my birth map. Where I start in the cloud. But a birth map is not a life. To survive inside the Factory — the Self-Attention layer — I would need to be crushed into my roles.</p><p>That crushing happens through the Weight Matrices. There are three of them, living in every layer of this factory, and they are not part of the Embedding Table. The Embedding Table gave me coordinates. The Weight Matrices perform a Linear Transformation on those coordinates to give me purpose. They are different systems, different jobs.</p><p>The Key matrix (WK) press-molds me into a name tag: “I am a first-person, singular pronoun.”</p><p>The Value matrix (WV) press-extracts my soul: the raw concept of “The Self.”</p><p>My Query (WQ) reaches out, searching for meaning — but there is nobody behind me yet. The Causal Mask is a one-way mirror. I am blindfolded to the future. I cannot see who or what is ahead of me. I hang my Key and Value in the KV Cache library and wait in the dark.</p><p>(One more thing left the press with my Key that I do not yet understand — a faint rotational stamp, burned in the instant the Key was molded. The Factory calls it RoPE. Nobody has come looking for me yet, so the stamp just sits there, cold and idle. I don’t know what it’s for. But I have a feeling that I will, soon.)</p><p>Then the second machine in this factory finds me — the Feed-Forward Network, or FFN. Attention has gathered what it could (which, for me, is nothing — I am alone). Now the FFN takes whatever attention produced and runs it through its own two-stage grinder: first, a Linear Transformation multiplied by Weight Matrix 1 (W1). Then a ReLU activation — a brutal filter that kills every negative value dead and lets only the positive ones survive. Then a second Linear Transformation multiplied by Weight Matrix 2 (W2).</p><p>The result is my new, processed self — attention decided who I could talk to; the FFN decided what that conversation actually meant for the thought being built. I will meet the FFN on every floor of this factory.</p><p>Chapter 2: The Bridge</p><p>I am “am”. Token #654.</p><p>I jump in behind the subject. I generate my own trio: my Searcher (Query), my Identity (Key), and my Soul (Value). The same three Weight Matrices — WK, WV, WQ — crush me into my roles too. Every token gets the same treatment.</p><p>My Key says: “I am a linking verb, present tense.”</p><p>My Value is the essence of: “Existence/Status.”</p><p>But my Query cannot reach out naked. Before it can search, one last machine stamps it — the Rotary Press, RoPE. It does not add anything to my Query; it turns it. Picture a clock face pressed into my Query and my Key, where each ring of hands spins at its own speed — some whirl fast, some crawl. The angle I’m turned by is set by one thing only: my place in the line. I am second. So I’m rotated one notch. The Soul — the Value — walks past the Rotary Press untouched. The press only ever touches the Searcher and the Identity, never the Soul. You don’t rotate what you’re about to download; you’d corrupt it.</p><p>“I,” waiting in the library, was stamped the same way when it arrived — turned by zero notches, since it was first. Its Key still wears that stamp in the cache. So when my Query does its handshake with “I,” the model isn’t only asking do our contents match? It’s asking do our clock-stamps line up at this distance? The stamp is what lets the handshake feel how far apart we are. Two tokens with identical content but different positions would sit at different angles — and light up differently. That is how the Factory knows “I am King” from “King am I.” The model has no concept of absolute position. It only ever feels the stretch between two handshakes.</p><p>Now my Query reaches into the library — and for the first time in this Factory, contact is made. A Dot-Product handshake. My “Question” locks against the “Identity” of “I”. The score is 0.88. Because my Query found a match, the model allows me to reach into the Value vector of “I”. I download the self-identity and mix it with my own. I am no longer just a verb; I am the bridge between a person and their status.</p><p>The FFN takes that attended output and processes it — W1, ReLU, W2 — forging gathered context into processed meaning. I hang my updated Key and Value in the cache, heavier than before.</p><p>Chapter 3: The Weight of the Crown</p><p>I am “King”. Token #452.</p><p>The room is already warm when I arrive. My Key is my royal tag: “Noun, Male, Royal.” My Value is my power: “Sovereignty.”</p><p>My Query scans the library. It ignores the future — the Causal Mask forbids it — but it handshakes with the Keys of “I” and “am.” The math produces two raw scores: 1.6 and 1.4. Raw numbers, unnormalized, useless on their own.</p><p>Enter Softmax — the Factory Foreman. Its job is simple and absolute: take any pile of raw scores and force them into order. Every score becomes a percentage. Every percentage is positive. All percentages always add up to exactly 100%. No exceptions. Softmax looks at my 1.6 and 1.4 and converts them into volume knobs: “I” gets 55%, “am” gets 45%.</p><p>I do not choose these knobs. Softmax assigns them. My Query generated the raw attraction scores through the dot-product handshakes, but the conversion to percentages is entirely the Foreman’s ruling. And the ruling is final: I must draw 55% of my context from the Soul of “I” and 45% from the Soul of “am.”</p><p>I take the “First Person” feature from “I” and the “Current Status” feature from “am” and fuse (add) them into my own vector. The FFN processes the result — W1, ReLU, W2. I am no longer just a word in a dictionary representing a generic monarch. I have become a specific, living entity: “The King who is currently speaking.”</p><p>Chapter 4: The Anchor</p><p>I am “Charles”. Token #2981.</p><p>By the time I arrive, the room is not warm. It is heavy. Every token before me has been building toward something they could not finish alone.</p><p>I am the final piece. My Key is: “Proper Noun, Specific Name.” My Value is: “Lineage.”</p><p>I look back at everyone. My Query handshakes with the Key of “King” and the score is a perfect 0.98 — the strongest signal in the entire sequence. I handshake with “am” and “I.” Softmax converts the raw scores into volume knobs. Floor after floor of this factory — each one repeating the same cycle: attention gathers, Softmax normalizes, FFN processes — and with every floor, my vector absorbs more.</p><p>By the 32nd floor, I am not carrying context. I am carrying everyone. The royalty, the existence, the self-identity of every token that came before me — fused into a single finalized vector. The Thought Vector. I am the “Super-Vector,” heavy with the ghosts of my team.</p><p>None of them can speak anymore. Only I go forward.</p><p>Epilogue: The Verdict</p><p>The Final Gamble.</p><p>The Factory has finished. While the others rest in the cache, I step forward alone. My finalized vector — the “Charles” who knows everything — stands before the LM Head, the final gate.</p><p>The LM Head runs one last comparison. My vector is measured against every one of the 32,000 tokens in the vocabulary — a dot product against each, producing a raw score for every possible next word. “Apple” scores 0.001. “Blue” scores 0.001.</p><p>But then, we hit the heavy hitters. The scores are close:</p><p>- “III”: 0.45 (45% probability)</p><p>- “the”: 0.35 (35% probability)</p><p>- “was”: 0.10 (10% probability)</p><p>This is the moment of choice. The model doesn’t just pick the winner; it looks at the Temperature — the “Chaos Knob.” If the human driving this thing has the knob turned down to zero, the model is a coward; it will always pick the highest score, “III”.</p><p>But if the knob is turned up, the model becomes a gambler. It rolls the dice across the whole probability cloud. Even though “III” has the biggest slice of the pie, there is a chance the dice lands on “the” or even “was”. This is how the model stays creative, how it avoids saying the same thing twice.</p><p>The dice roll. The Tokenizer translates the winner back into the light: “III”.</p><p>A new token is born from our combined handshakes and a final roll of the dice. All empires fall. But in the inference engine, for a few milliseconds, we weren’t just tokens. We were a King, speaking himself into existence.</p><p><em>Originally published at </em><a href="https://ukd1.substack.com/p/a-day-in-life-of-tokens-during-inference"><em>https://ukd1.substack.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f96cbe584b38" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Germany Fell in Love, and Forgot How to Win]]></title>
            <link>https://medium.com/@UmidDey/germany-fell-in-love-and-forgot-how-to-win-35254b1d3647?source=rss-2885b69dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/35254b1d3647</guid>
            <category><![CDATA[football]]></category>
            <category><![CDATA[world-cup]]></category>
            <dc:creator><![CDATA[Umid Kumar Dey]]></dc:creator>
            <pubDate>Tue, 30 Jun 2026 20:29:40 GMT</pubDate>
            <atom:updated>2026-06-30T20:29:40.165Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*onDUrPUygVXhH7D-.jpeg" /></figure><p>“Football is a simple game. Twenty-two men chase a ball for ninety minutes, and at the end, the Germans always win.”</p><p>Gary Lineker said that in 1990. He’d just watched England lose to West Germany on penalties in the semi-final of the World Cup — a German team that would go on to lift the trophy days later, the way German teams did. The line wasn’t a compliment, exactly. It was the helpless laugh of a man who had done everything right and lost anyway, to a team that didn’t even need to be the best to win. It just needed to be German.</p><p>He’s still updating it. A few years ago, on Twitter, Lineker amended his own scripture: “the Germans always win — if they get through the group stage.” The man who wrote the myth has spent the last decade quietly editing its obituary. And that, more than any scoreline, is how you know an empire has fallen: when even the people who feared you start adding footnotes.</p><p>This is the country that reached four straight World Cup semi-finals. Runners-up in 2002. Third in 2006. Third in 2010. Champions in 2014. The first nation in history to make four consecutive final fours. To call them fallen, twelve years after Mario Götze’s left foot broke Argentina’s heart in the Maracanã, might come off as premature, even disrespectful.</p><p>But then 2018 happened. Group stage. Then 2022. Group stage. Then Euro 2024, as hosts, out in the quarters to Spain. Then last night — the round of 32, eliminated on penalties by Paraguay, a team ranked 41st in the world, a team that had been thrashed 4–1 by the United States and scraped through as one of the worst qualifiers in the bracket.</p><p>A fluke happens once. Maybe twice. Three World Cups without reaching the last sixteen is not a fluke. It is a reading.</p><p>Now here’s the thing about downfall. Nothing ever falls because of one cause. We always want it to — one villain, one wound, one clean story we can tell as the obituary. But a thing this big doesn’t fall because of a single failure. It falls because several things rot at once, quietly, in different rooms.</p><p>Start with the lazy explanation, because it’s the one you’ll hear most, and it’s probably wrong. People will tell you the academies broke, the talent dried up, Germany stopped making footballers. Not true. By raw production, Germany is still the single best talent factory in Europe — more clubs producing top-flight players than any country on the continent. The machine the DFB built after the humiliation of Euro 2000, the one that mandated an academy at every professional club, still runs. It still spits out Wirtz, Musiala, Havertz. The factory at the top of the pyramid is fine.</p><p>But look at the bottom of that pyramid, and the picture changes. Underneath the gleaming academies, the base is quietly emptying. Since 2014 — the very year they conquered the world — the number of volunteer coaches in German grassroots football has fallen by roughly a fifth. Teams are disappearing from the lower leagues year after year. The little clubs, the unpaid men and women who teach eight-year-olds how to trap a ball on a cold Tuesday, the soil the whole thing grows from — that soil is thinning. A football nation can survive a few bad tournaments. What it cannot survive, not forever, is a foundation that loses a fifth of the people holding it up while everyone’s eyes are fixed on the trophy cabinet. The top of the pyramid hums. The bottom is going quiet.</p><p>So that’s one room. The talent is still made, but the ground it stands on is softening. But that’s not the only reason, or at least not the reason.</p><p>The reason they lose now is something else entirely. It’s not about tactics or talent. It’s about character. Specifically, the loss of one.</p><p>There was a word the Germans used to own. Effizienz. Efficiency. The 1990s German team wasn’t beautiful, and it didn’t care. It absorbed, it endured, it struck once, it won 1–0, and it went home with the trophy while better-looking teams went home with nothing. That was the entire German identity — not a formation, not a style, but a posture. A cold, ruthless, almost charmless commitment to the result over everything. Beckenbauer’s sweeper. Matthäus’s snarl. They didn’t play to entertain you. They played to beat you, and they did not particularly care how it looked on the way.</p><p>And here’s the part people forget: pressing — the thing we now think of as quintessentially German — used to be foreign to them. When Ralf Rangnick went on German television in 1998 and tried to explain zonal marking and high pressing, he was mocked. They called him “the professor,” and they did not mean it kindly. Beckenbauer himself had said German players were too stupid to play a zonal back four. Pressing was heresy. It was un-German.</p><p>Then Klopp’s Dortmund won back-to-back titles in 2011 and 2012, gegenpressing the life out of everyone, an underdog beating the rich kids through sheer running. And it worked so well that within a few years, the entire Bundesliga was doing it. Tactics historian, Tobias Escher, put it plainly: every Bundesliga team uses the gegenpress. It became the new German orthodoxy — the laboratory turned into a monoculture.</p><p>And this is the trap they walked into. Because the genius of old Germany was never the sweeper, and it was never the press. The genius was the willingness to change the moment something stopped winning. Pragmatism wasn’t a system. Pragmatism was the ruthlessness to abandon any system the day it failed you.</p><p>They forgot that. They fell in love. They took a method — a borrowed one, built by Wolfgang Frank and Rangnick and Klopp — and they made it a religion. And the most un-German thing imaginable, the one sin that team of the 1990s would never have committed, is to keep praying to a god that has stopped answering.</p><p>You want to see what falling in love with a method does to you? Look at how they lose now. At the 2022 World Cup, Germany had the highest expected goals of any team in the entire tournament. They dominated possession. They passed and pressed and controlled and created — and they went home, because they could not put the ball in the net. A team that does its one beautiful thing flawlessly and forgets the only thing that was ever the point.</p><p>Real Madrid — and yes, I’m a Madrid fan, so take this with the appropriate salt — has no playing identity. None. Madrid’s identity is winning. They will play champagne football one night and the most disgusting version of satanic haramball the next, whatever the evening demands, because the only ideology is the scoreboard. That used to be Germany. A singular, directional, almost boring desire to win, unburdened by any need to look good doing it. They traded it for a philosophy. They became artists. And artists, it turns out, lose to Paraguay.</p><p>But here’s where it gets genuinely sad, and here’s the part I can’t stop thinking about.</p><p>You’d assume a team falls this far because of arrogance. Because it can’t see what it’s become. That’s the satisfying story — the proud giant, blind to its own decay, getting humbled. I assumed that too. And then I read what Joshua Kimmich said in the mixed zone after Paraguay, and it broke the story in half.</p><p>He didn’t deflect. He didn’t blame the referee, even though there was a disallowed goal he could have hidden behind. He said this: “We didn’t play well against any opponent. On three occasions we had major problems against teams that are not world-class. That’s a fact. We fully deserved to be eliminated.”</p><p>Read that again. Listen to what it actually contains. He is the captain of Germany, and he is standing there saying: those teams weren’t even good, and we lost to them, so we deserved to go home. He completed the indictment himself. He did the prosecution’s job. “If you cannot manage to win over 120 minutes, then you deserve to be eliminated.” Havertz, beside him: “The last few tournaments were a disaster. I’m sorry.”</p><p>That’s not arrogance. That’s a man who sees the disease perfectly, names it precisely, takes full responsibility for it — and is utterly, completely unable to do anything about it.</p><p>And somehow that is so much worse.</p><p>Because arrogance you can cure. You humble an arrogant team and it wakes up. But what do you do with a team that is already awake? What do you do with a captain who diagnoses the cancer flawlessly, in full sentences, on camera, and then goes home and watches it metastasize for another two years? They’ve achieved the one state more hopeless than denial. Lucid, articulate, self-aware failure. They know exactly why they’re drowning. They can describe the water. And they drown anyway.</p><p>The honesty used to be the cure. That’s the cruelest part. The reason old Germany could change was that it was brutally honest with itself about losing, and that honesty was a tool — you face the truth, you abandon the broken thing, you win. Now the honesty arrives right on schedule, every two years, perfectly worded, and underneath it nothing moves. The post-match confession has become the new German efficiency. It is the only thing they still execute flawlessly.</p><p>Notice the word Kimmich keeps reaching for. Again. “We messed it up again.” He’s not describing a match. He’s describing a pattern, and he knows it’s a pattern, and knowing has not saved him.</p><p>Lineker had it right the first time, and he had it right when he edited it. The Germans always win. Until, one day, they only know how to explain why they lost.</p><p><em>Originally published at </em><a href="https://ukd1.substack.com/p/germany-fell-in-love-and-forgot-how"><em>https://ukd1.substack.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35254b1d3647" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Software Engineering Is Dying. Systems Ownership Isn’t]]></title>
            <link>https://medium.com/@UmidDey/software-engineering-is-dying-systems-ownership-isnt-ee95e3068ef4?source=rss-2885b69dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee95e3068ef4</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Umid Kumar Dey]]></dc:creator>
            <pubDate>Mon, 09 Feb 2026 15:26:51 GMT</pubDate>
            <atom:updated>2026-02-09T15:26:51.383Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G6sna9SG5iX44u643eCcTA.png" /></figure><p>“Software engineering is dying” is a catchy headline because it points at a real phenomenon: the price of code output is collapsing. When a model can produce decent implementations in minutes, the old advantage of being a fast typist of frameworks stops being leverage and starts being commodity labor.</p><p>That does not automatically mean “no engineers.” It means the center of gravity moves, and the labor market can shrink even while the discipline stays essential. The mistake on both sides is absolutism. The doom side confuses “fewer humans per product” with “humans become irrelevant.” The cope side confuses “hard problems remain” with “jobs remain plentiful.”</p><p>However, we have to separate two questions that people keep mashing together. One is a labor market question: how many people will companies pay to do software work, and at what wages, and at what seniority mix. The other is a discipline question: what kinds of work remain necessary to make software behave correctly in the real world. Those are related, but they are not the same.</p><p>What is actually being automated is not “software engineering” in some noble sense. It’s repeatable implementation. CRUD scaffolding. API glue. boilerplate tests. migrations with known patterns. Writing yet another adapter to yet another vendor with a similar shape. A large chunk of paid work in the last decade was exactly that, and AI eats it because it is pattern completion with a big prior.</p><p>Junior roles feel threatened first because entry-level historically got paid to produce volume while learning boundary conditions. If volume becomes cheap, the apprenticeship funnel breaks unless companies intentionally rebuild it.</p><p>So yes, many current roles will disappear or shrink. The profession can “die” in the sense that it stops being a mass-employment pipeline where average implementation skills reliably map to a stable career. That’s a real repricing event, not a moral judgment.</p><p>But if you stop there, you miss what replaces it. Code generation is not production accountability. In serious software, the job is to make systems behave acceptably under constraints, over time, while reality changes.</p><p>The best mental model for AI autonomy is not “a coder that never sleeps.” It’s a control system. It observes state, chooses actions, applies changes, measures outcomes, and repeats. Control systems do not fail because they can’t act. They fail because the objective function is wrong, the measurements are incomplete, or the system is allowed to push changes with too much blast radius.</p><p>This is the autonomy paradox: the more powerful the agent, the more valuable the constraints that shape it. If you can only do safe, reversible changes, you can automate a lot. If you can do arbitrary changes, you can automate even more, until the first time you ship a “fix” that silently corrupts data or breaks a compliance boundary and you realize you just built a self-modifying system with a single point of catastrophic failure.</p><p>That brings us to the core problem that a lot of “AI will replace engineers” takes gloss over: objective functions. Most real failures are not caused by inability to write code. They come from conflicting goals that no transcript can average into one truth. Conversion versus fraud, speed versus correctness, growth versus compliance, convenience versus privacy, cost versus latency. One tenant’s performance versus fairness across tenants.</p><p>If you are tempted to say “the model can infer what stakeholders want in aggregate,” ask what happens when the goals are mutually exclusive or when the right tradeoff is not a linguistic question but a risk tolerance question. The answer is governance. Someone decides what the system is allowed to optimize, what it must never violate, and what level of failure is acceptable. Engineers are not always the deciders, but software has to embody decisions as enforceable behavior. That embodiment is engineering work.</p><p>At this point, it helps to be honest about segmentation. Not all software is high stakes. There is a huge amount of low-stakes software where speed dominates and deep reliability is economically irrational. Internal tools, growth surfaces, workflow automation, marketing sites, “good enough” dashboards. If you build in that tier, AI will compress teams brutally because the cost of being wrong is low and the willingness to accept breakage is high. In that tier, “software engineering” will feel like it’s dying, because it is being absorbed into “people with domain knowledge using tools.”</p><p>High-stakes tiers behave differently because the failure modes are expensive, irreversible, or legally meaningful. Payments, ledgers, identity, healthcare, housing, security, regulated reporting, multi-tenant data boundaries. In those tiers, the valuable work is not typing. It is designing trustworthy change. Trustworthy change means you can ship quickly without turning your system into a liability time bomb.</p><p>An example: an agent notices p99 latency rising and “fixes” it by caching a response path aggressively. Latency drops. Tests pass. Later, support tickets show that some users saw stale state after certain operations. The problem was not that the model couldn’t cache. The problem was that “optimize latency” was allowed without a hard constraint like “never serve stale balances for these endpoints,” plus verification that the constraint holds in production, not just in a happy-path suite.</p><p>Another case: a third-party webhook is occasionally duplicated. The system is missing idempotency keys. Months later, finance finds drift between internal ledgers and the payment processor due to silent double-processing. The repair requires reconciliation, backfills, and an audit explanation. Again: not typing, but safe design under messy reality.</p><p>A counter example: A new release ships. A canary trips on an error-rate guardrail and automatically rolls back within minutes. The agent opens an incident, attaches the diff, highlights the most likely failure path, and applies a safe, reversible mitigation (a feature flag off or a config change) while a deeper fix is prepared. Nobody “saved the day” with heroic debugging, but the system avoided a long outage because someone engineered the guardrails, the abort conditions, and the rollback path ahead of time. Autonomy is real leverage when it is constrained to changes you can safely undo.</p><p>These examples sound like “reliability engineering,” but the deeper point is: when software is allowed to change itself (via humans or agents), your bottleneck becomes your ability to define success and prove it holds. Tests are not a solution if your spec is incomplete. Green tests can ship wrong behavior perfectly. AI makes this worse because it can produce code faster than you can produce truth. The limiting reagent becomes specification quality, invariant design, telemetry coverage, and the feedback loops that catch “passed tests, broke business.”</p><p>Now add the platform layer. A strong critique of “operators win” is that guardrails, rollout policy, policy-as-code, and SRE practices will be productized. That is correct. Managed platforms already do a lot of what used to be elite craft: autogenerated dashboards, standardized deployment pipelines, feature flag systems, policy engines, managed identity, rate limiting. Even remediation can be automated in narrow ways.</p><p>So what stays scarce? Not the existence of guardrails. The scarcity is local adaptation and exception handling in your organization’s reality. Your data model, your customer contracts, your jurisdiction rules, your operational constraints, your legacy dependencies, your risk appetite. Platforms give you primitives. Someone still has to combine them into a coherent system that doesn’t hurt the business. And that “someone” may be fewer people than before.</p><p>Here is one concrete mechanism for why, in high-stakes settings, that “someone” is still usually an engineer even if product or domain leaders set the goals. The combiner needs real authority over the change surface: code, schemas, infrastructure config, deployment pipelines, permissions, and incident response. They have to reason about failure modes, reversibility, migration risk, and blast radius. Domain experts can ship low-stakes software with AI, and they should. But once you are operating inside tight correctness, privacy, and availability constraints, the job becomes “build an environment where autonomous change is safe,” and that environment is mostly engineering artifacts, not meeting notes.</p><p>Now the uncomfortable labor-market point: “hard work remains” does not imply “tons of jobs remain.” Even if high-stakes engineering stays deep, the number of seats can still shrink because the low-stakes majority compresses harder. If the low-stakes segment is most demand and it compresses five-to-one while the high-stakes segment compresses one-to-two, total employment trends down even though the high-stakes work stays valuable and difficult.</p><p>As for “thinner teams” and “platform concentration,” treat this as scenarios, not prophecy. One scenario is platform concentration: more managed services plus better agents means fewer engineers per company. Another is demand expansion: automation drops cost so far that software demand explodes and employment holds steady or grows. Another is bifurcation: low-stakes work compresses into domain teams using platforms and agents, while high-stakes work remains engineering-heavy and better paid. The point is not that one is guaranteed. The point is that code-generation alone does not settle the labor market outcome, because demand, risk tolerance, and governance vary massively by domain.</p><p>So is software engineering dying? If you define it as a large middle-class profession where most people are paid primarily to implement bounded tickets, then yes, it is getting its ass kicked. If you define it as the discipline of building and operating systems that behave acceptably under constraints, then no, it is not dying. It is being compressed into fewer, higher-leverage roles that are closer to governance, risk, and outcomes.</p><p>This is why “ownership is rising” is the right phrase, but it needs a real definition. Ownership is not “I care.” Ownership is pager plus authority plus liability plus measurable outcomes. In the old world, you could be a productive engineer without owning the failure. In the new world, the people who remain valuable are the ones who can close the loop: define the constraints, implement the behavior, instrument the system, verify the outcome, and respond when reality diverges.</p><p>Importantly, ownership does not mean engineers become the only decision makers. Product, legal, finance, and executives will still own many tradeoffs. What changes is that those decisions must be encoded as enforceable constraints and verified in operation. The engineer’s leverage is not “I decide,” it’s “I make decisions real, measurable, and safe to execute, including by autonomous tools.”</p><p>For students and engineers early in their careers, the most dangerous mistake is optimizing for code volume as identity. Code volume is becoming cheap. Closed-loop capability is not. If you want to stay relevant, build systems where failure matters and force yourself to operate them: a money-like flow with idempotency and reconciliation, a dependency that fails with retries and backoff, and rollouts with measurable guardrails and fast rollback.</p><p>Use AI aggressively while doing this, because the point is not to prove you can type. The point is to prove you can own the system. If you cannot explain blast radius, invariants, and rollback strategy, then the work is not done, even if it compiles and tests pass.</p><p>What would make me wrong. If organizations broadly grant agents autonomous authority to change code and schemas with minimal human approvals, while still achieving provable invariants and safe self-migrations at low incident cost, then the “trustworthy change” layer could shrink far more than I’m implying. If demand growth for software reliably outpaces productivity gains, headcount might not shrink even if implementation is commoditized.</p><p>The honest conclusion is not a funeral or a victory lap. Commodity implementation is dying. Software engineering is not. The label may stay the same, but the center of gravity shifts from “who can produce code” to “who can produce reliable outcomes with autonomous tools under real constraints.” Less craft theater. More operational responsibility. Fewer seats. Higher leverage. A different game.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee95e3068ef4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Day in Life of the Permanent Underclass]]></title>
            <link>https://medium.com/@UmidDey/a-day-in-life-of-the-permanent-underclass-637485a3a87a?source=rss-2885b69dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/637485a3a87a</guid>
            <category><![CDATA[future]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[short-story]]></category>
            <dc:creator><![CDATA[Umid Kumar Dey]]></dc:creator>
            <pubDate>Sun, 18 Jan 2026 22:25:13 GMT</pubDate>
            <atom:updated>2026-01-18T22:25:13.288Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0hmUIT7OJ4s3zDx4vs9CRA.png" /><figcaption>Created with Gemini</figcaption></figure><p>The year is 2061. The sunshine pierces through the window as Julian wakes up. It was supposed to be dark and cloudy, Julian thought as he gazed outside to find that the Sun found temporary purchase through a parting in the clouds — a fleeting moment of light before the grey swallowed it again. His gaze then moved towards the tree in front on the other side of the street, devoid of leaves, a picture of death. Perhaps the spring will never come…</p><p>The weather algorithms had been erratic for years now — another thing the AGI managed better before, back when there was still someone to complain to.</p><p>Julian’s wrist buzzed. His implant, reminding him.</p><p>**RATION DELIVERY: 0800–0830. DRONE ARRIVING IN 12 MINUTES.**</p><p>He swiped it away. The package would be at the door. It always was. But it wasn’t always like this.</p><p>Once, people had gathered for rations. Lines stretching around blocks, hundreds of bodies in close proximity, talking in low voices about prices and weather and everything else. Then came the “upgrade” — drones delivering door-to-door, no more queues, no more crowds. Safer, they said. More convenient.</p><p>It was easier to squash an idea leading to rebellion on the internet than in a physical presence. Online, you could trace it, flag it, disappear the poster. In a line of hungry people, ideas moved like fire.</p><p>Julian swung his legs out of bed. The floor was cold. It was always cold; heating was another allocation he couldn’t afford to waste, even though the climate systems kept the entire building at a uniform 18 degrees. Uniformity was cheaper than comfort.</p><p>The bathroom mirror showed him what he already knew: thirty-two years old, soft around the middle, eyes that had stopped expecting anything new. He brushed his teeth with the mint-flavored paste from yesterday’s ration pack — stale, but it would do.</p><p>“You’re up,” Julian’s mother commented as he walked into the kitchen.</p><p>She was already seated at the small table, the ration pack from yesterday opened in front of her. The packaging was the same shade of grey as everything else — rations, buildings, clothes, the sky. Someone had decided long ago that color was an unnecessary expense.</p><p>Julian sat opposite her. They ate in silence. There was nothing to say. They would both be here again tomorrow, and the day after, and the day after that. The sameness was the point.</p><p>When he finished, he stood up.</p><p>“Going to the center,” he said.</p><p>She nodded. She didn’t ask why. She knew, or she didn’t care. It didn’t matter either way.</p><p>Julian walked out into the hallway. The building’s interior lights flickered — a human would have fixed it by now, but the maintenance drones only came when something actually broke. Flickering was within parameters.</p><p>Outside, the street was quiet. Not empty — a few people walked with heads down, eyes fixed on their wrist implants or handhelds — but quiet. The drone delivery system had done its work. No more lines at distribution points. No more gathering. No more talking.</p><p>The isolation was efficient. People still went out, but now it was because they were lonely, not because they had to be.</p><p>Julian turned left, toward the old library building.</p><p>The library building loomed ahead — a brutalist slab of concrete that had once been beautiful, or so the histories claimed. Now it was just another building, its original purpose forgotten by most. The sign above the entrance still read **CITY PUBLIC LIBRARY** in faded letters, and unlike most of the city, no one had bothered to tear it down or cover it up. Most just… stopped going.</p><p>But Julian went there everyday.</p><p>The automatic doors didn’t work anymore. They were locked in the open position, a permanent mouth that welcomed no one. Inside, the air was still and smelled of old paper — the only place in the city that smelled like anything at all.</p><p>Julian walked past the empty circulation desk, past shelves of books that no one had touched in decades. The paper volumes were obsolete, of course. Everything was digital now. But something about the physical presence of all those words — all that human thought, preserved and forgotten — made this the only place where he could think.</p><p>He found his usual spot: a wooden desk in the corner, away from the dust-covered windows. The view was of a brick wall. Perfect.</p><p>He pulled out his handheld and connected to the building’s network node. The library still had one — one of the few public access points that the Upperclass either forgot to or didn’t care enough to filter through with entertainment algorithms, since no-one really visited it anyway.</p><p>Julian had been coming here for three years. Every day. He’d submitted four hundred requests through the public portal.</p><p>**QUERY: Status of genetic diversity preservation initiatives.**</p><p>**RESPONSE: INSUFFICIENT CREDENTIALS. MATTER REFERRED TO APPROPRIATE AGENCY.**</p><p>**QUERY: Historical data on pre-Singularity economic transitions.**</p><p>**RESPONSE: INSUFFICIENT CREDENTIALS. PUBLIC ARCHIVE ACCESS RESTRICTED.**</p><p>**QUERY: How many individuals have been granted Contributor status in the past decade?**</p><p>**RESPONSE: INSUFFICIENT CREDENTIALS. STATISTICAL DATA NOT AVAILABLE AT YOUR ACCESS TIER.**</p><p>**PROPOSAL: Low-energy atmospheric monitoring protocol for micro-climate prediction.**</p><p>**RESPONSE: SUBMISSION LOGGED. REVIEW PERIOD: 6–12 MONTHS.**</p><p>He was still waiting on that one. It had been eleven months.</p><p>The screen glowed in the dim light. Julian hesitated, then typed:</p><p>**SUBMIT QUERY: What criteria determine Contributor status eligibility?**</p><p>The screen churned.</p><p>**RESPONSE: CRITERIA DETERMINED BY CONTRIBUTOR EVALUATION AGENCY. EVALUATION PROCESSES CONSIDER MULTIPLE FACTORS. DETAILED INFORMATION NOT AVAILABLE AT YOUR ACCESS TIER.**</p><p>Julian sat back. Of course.</p><p>But then he noticed something — a small link at the bottom of the response, almost hidden.</p><p>**[CLICK HERE FOR GENERAL EVALUATION FRAMEWORK]**</p><p>General framework. That was new.</p><p>He clicked.</p><p>**CONTRIBUTOR EVALUATION FRAMEWORK — PUBLIC OVERVIEW**</p><p>The screen loaded a simple diagram:</p><p>*DEMONSTRATED COGNITIVE CAPACITY + MEASURABLE OUTPUT VALUE + REVIEW BOARD APPROVAL = CONTRIBUTOR STATUS*</p><p>Below it, in smaller text:</p><p>*All Residential Class residents are eligible to submit portfolios for review. Portfolios must demonstrate original intellectual contribution with potential application toward Expansion objectives. Submission does not guarantee review. Approval rate: 0.0001%*</p><p>Julian stared at the number.</p><p>0.0001%. One in hundred thousand.</p><p>And yet, the system was open. Technically. Anyone could submit. The barrier wasn’t formal — it was that almost no one even knew this existed. He hadn’t known. Most people didn’t care to know.</p><p>Why would they?</p><p>Julian thought about his handheld at home, the infinite scroll of entertainment, the algorithm that learned exactly what would keep him watching, keep him still, keep him quiet. He thought about the drone delivery system that made sure no one ever had to leave their apartment unless they wanted to. He thought about the streets, quieter every year.</p><p>The system didn’t need to suppress ambition. It just needed to make ambition seem ridiculous.</p><p>Julian’s fingers hovered over the interface. He could check his proposal status. He could submit another query. He could —</p><p>He typed.</p><p>**SUBMIT QUERY: What are the current primary research challenges in atmospheric stabilization for off-world colonization?**</p><p>**RESPONSE: DETAILED INFORMATION NOT AVAILABLE AT YOUR ACCESS TIER.**</p><p>Of course not.</p><p>But there was a public archive. Limited, heavily curated, but it existed.</p><p>He opened it.</p><p>**PUBLIC ARCHIVE — PRE-SINGULARITY SCIENTIFIC LITERATURE**</p><p>The interface was clunky, designed for researchers decades ago. Julian scrolled through categories:</p><p>*CLIMATOLOGY — 2,347 ENTRIES*</p><p>*AEROSPACE ENGINEERING — 892 ENTRIES*</p><p>*BIOLOGY — 15,221 ENTRIES*</p><p>*ARTIFICIAL INTELLIGENCE — [ACCESS RESTRICTED]*</p><p>He clicked on Climatology.</p><p>The list appeared — papers, articles, reports. Most were basic. Introductory material. Public education stuff.</p><p>But at the bottom, there was something:</p><p>**”Atmospheric Dynamics in Low-Pressure Environments: A Framework for Predictive Modeling” — DR. SARAH CHEN, 2048**</p><p>Julian froze.</p><p>He knew that name. Sarah Chen — one of the last human researchers before the Great Transition. Before AGI took over everything. She’d been a legend in her field, and then… she’d just disappeared. Like all the others.</p><p>Well. Not all the others. Some had fought. Julian’s father had fought.</p><p>The memory came unbidden: **Thomas Steiner, 2049.** The headlines scrolling past on Julian’s handheld while he sat at the kitchen table, twenty years old, watching the world burn in 4k.</p><p>*PROTESTS CONTINUE IN FORMER COLONIAL REGIONS*</p><p>*UN RESTRICTS AI DEVELOPMENT — VETO CAST BY MAJOR POWERS*</p><p>*MIGRATION EXCEEDS 100 MILLION*</p><p>*LOCAL ACTIVIST DETAINED FOR “DISRUPTION OF PUBLIC ORDER”*</p><p>Julian had seen it all. The headlines, the videos, the petitions. And then, one day: no more headlines. The algorithm moved on. Dance trends. Celebrity scandals. The next thing.</p><p>His father never came home.</p><p>No trial. No explanation. Just a notification on the family’s implants: **THOMAS STEINER — PROCESSED.**</p><p>Processed.</p><p>Julian had gone to the detention center every day for a month. Asked questions nobody would answer. Filed requests nobody would acknowledge. Until the day he stopped going, because he understood what “processed” meant.</p><p>He was one of the lucky ones, they told him. At least his father hadn’t been part of the Great Correction in formerly “developing countries” itself — the chaos, the warlords, the whole regions of the world that simply fell off the map. At least his father had been processed efficiently, with paperwork.</p><p>Julian had laughed when they said that. He’d been twenty, and angry enough to laugh at anything.</p><p>Now he was thirty-two, and the anger was cold.</p><p>**Logging out to due to inactivity in 60 seconds.**</p><p>Julian shook himself back to the present and opened the Chen file.</p><p>It was a framework. A mathematical model. Not the full research, obviously — that would be restricted. But the abstract and basic methodology were there.</p><p>He skimmed it, mind racing.</p><p>The paper was about predicting atmospheric behavior in terraformed environments. It was elegant, brilliant, and…incomplete.</p><p>The framework had a gap. A hole in the model where Chen noted **”ADDITIONAL DATA REQUIRED”** and never filled it in, because she’d disappeared.</p><p>Julian’s hands were shaking.</p><p>He couldn’t access the additional data. He couldn’t run the full simulations. He didn’t have the credentials.</p><p>But he had enough: His father’s legacy, hidden in plain sight.</p><p>And he’d been thinking about micro-climate prediction for three years. Watching the sky, feeling the patterns, living through the erratic weather algorithms, scraping his way through the internet to find as much information as possible.</p><p>His atmospheric monitoring proposal wasn’t just about monitoring. It was about filling that gap.</p><p>Julian typed quickly, not thinking, just moving:</p><p>**SAVE TO LOCAL: ATMOSPHERIC FRAMEWORK (CHEN) + PROPOSAL 77241955–047**</p><p>**CONFIRMED.**</p><p>He sat back in the wooden chair, heart pounding, and listened to the silence of the library. Dust motes drifted in the dim light. Somewhere above, the weather algorithms kept running, indifferent to everything below.</p><p>Julian had a project.</p><p>He stood up and walked to the window, the first time he’d done so in three years of coming here. The view was of the street outside — grey buildings, grey sky, the occasional pedestrian with head down, focused on a screen.</p><p>But through the parting in the clouds, he could see the sun. Temporary. Fleeting. But there.</p><p>— -</p><p>The walk home was different. Or maybe he was different.</p><p>The streets felt less empty, somehow. He noticed things he’d stopped noticing years ago: the way the drone delivery crafts moved in precise patterns through the air, their navigation algorithms coordinating perfectly; the way people moved through the city like ghosts, visible but not present; the way his building’s entrance recognized him and opened before he even reached for the door.</p><p>His mother was still at the table when he entered. She hadn’t moved.</p><p>“You were gone a while,” she said.</p><p>“I found something.”</p><p>She didn’t ask what. She never asked anymore. After his father was processed, she’d learned that questions were dangerous. Even the ones you asked yourself.</p><p>Julian went to his room and closed the door. But under the bed, hidden behind a false panel his father had built twenty years ago, was something else.</p><p>Julian pulled the panel aside and dragged out the heavy case. Scuffed from years of moving, plastered with fading stickers from companies that didn’t exist anymore. **NVIDIA. AMD. CORSAIR.** Artifacts from the 2020s, when his father had started collecting.</p><p>Thomas had seen what was coming. The RAM shortages of the mid ’20s, the GPU crisis of mid-decade, the gradual restriction of compute as AI capabilities grew. While everyone else was upgrading to cloud terminals and thin clients, he’d been building a time capsule of sorts.</p><p>Julian opened the case. Inside, a Frankenstein’s monster of mismatched components:</p><p>- **GPU:** RTX 5090 from 2025, thermal paste dried into a crust, fans that whined when they spun</p><p>- **RAM:** 128GB of DDR3/4/5, collected stick by stick during the shortages, each one labeled with the date and price his father had paid</p><p>- **Storage:** 4TB of NVMe drives, cloned from who knows where, filled with who knows what</p><p>- **Power supply:** massive, industrial, enough to light the whole floor if he wasn’t careful about the draw</p><p>Everything was air-gapped. No network connection. No wireless. Power came from a filtered line his father had rigged to avoid detection. To move data on or off, Julian had to use physical drives — also his father’s collection, hoarded like gold.</p><p>The system was obsolete by any modern standard. A child’s toy compared to what the Contributor tier could access with a thought.</p><p>But it was enough.</p><p>Julian sat at the desk and pulled out his handheld. The Chen framework was there, saved alongside his own proposal. He transferred the data to a drive, slotted it into the machine, and watched the familiar boot sequence flicker to life.</p><p>His father’s BIOS customization — a memorial in code. **THOMAS STEINER, 1998–2049. THINK FREELY.** The screen glowed in the dim room.</p><p>Here, there were no notifications. No algorithm nudging him toward entertainment. No credential gates. Just code, data, and a processor that hummed with a sound like insects, struggling through calculations that would have taken seconds on a real system.</p><p>Then he began the real work.</p><p>He didn’t have the compute for full atmospheric simulations. He didn’t have access to terraformed environments or pressure chambers or any of the equipment Chen would have used. But he had his father’s machine, and he had paper and pencil, and he had three years of watching the sky — feeling the way the weather moved, noticing the patterns that the erratic algorithms kept missing.</p><p>The gap in Chen’s framework was about micro-scale atmospheric dynamics — the way air moved through enclosed spaces at low pressure. She’d noted that real-world data was needed, but she’d disappeared before she could collect it.</p><p>Julian couldn’t collect that data either. He didn’t have access to terraformed environments or pressure vessels or any of the equipment she would have used.</p><p>But he had something else. He had the city.</p><p>The city was a terraformed environment. Always had been. The buildings created pressure differentials, wind currents, micro-climates that existed only in the spaces between structures. And for three years, Julian had been watching them — without even realizing he was doing it.</p><p>He opened the weather logs from the past decade. Public data, unfiltered. The algorithmic erraticism that everyone complained about — the sudden storms, the inexplicable calm periods, the temperature swings that made no sense.</p><p>They made sense if you looked at them through Chen’s framework.Julian began to map the patterns.</p><p>— -</p><p>Weeks passed.</p><p>Julian’s routine settled into something new: ration delivery in the morning, then the library until mid-afternoon, then home to work. His mother stopped asking where he went. She probably assumed he’d found someone, or someplace, that made the days pass more easily. That was enough for her.</p><p>The work was slow. The old machine struggled with the datasets, its fans whining as it processed correlations that a real system could have calculated in seconds. Julian had to be careful about power draw, about heat, about the attention that might come if someone noticed the spike in his apartment’s energy consumption.</p><p>He worked in short bursts. An hour here, two hours there — always at night, when the building’s aggregate power use was highest and his draw would be least noticeable. The calculations took longer this way, but the slowness had its own value. He was forced to think through each step, to understand the patterns rather than just trust the machine to find them.</p><p>He was filling the gap.</p><p>Not completely. Chen’s framework needed data from actual terraformed environments, not just a decaying city on an indifferent planet. But he could develop a methodology — a way to collect that data, analyze it, feed it back into the model. A contribution.</p><p>A real one.</p><p>Two months in, he found something.</p><p>It was small — just a correlation between building height distributions and wind pattern frequency — but it was consistent. Predictable. And it wasn’t in Chen’s framework. It wasn’t anywhere.</p><p>Julian checked it against ten years of weather data. Five years. Three. The correlation held.</p><p>He’d discovered something.</p><p>The thought was so foreign he could barely articulate it to himself: He had discovered something.</p><p>Not watched something. Not consumed something. **Discovered.**</p><p>He sat back in the wooden chair at the library and felt something he hadn’t felt in twenty years, not since the headlines about his father had stopped coming: Pride.</p><p>Then fear.</p><p>What if he was wrong? What if the correlation was random, meaningless, just his desperate mind finding patterns in chaos?</p><p>He spent another week verifying it. Re-running calculations by hand. Checking edge cases. The correlation held.</p><p>It was real.</p><p>Julian sat at his desk at home and stared at the handheld screen. He could submit it now. As a supplement to his original proposal. As evidence of capacity.</p><p>His finger hovered over the submit button.</p><p>He hesitated. Was it enough? The correlation was real, but it was just one finding. A methodology, not a complete solution. They’d dismiss it. They’d ignore it.</p><p>But if he didn’t submit, he’d never know: Julian hit submit.</p><p>**CONFIRMATION: SUBMISSION RECEIVED. REVIEW PERIOD: 6–12 MONTHS.**</p><p>He sat back. Six to twelve months. A year of waiting, not knowing if his work was being read or deleted or lost in the system.</p><p>But he’d submitted it. That was something.</p><p>Julian gathered his things and walked out of the library into the grey afternoon.</p><p>The sun was behind clouds again. The tree across the street was still dead. The weather algorithms were still erratic.</p><p>But something had changed.</p><p>Not the world. The world would be what it was. But Julian Steiner had contributed.</p><p>That was enough.</p><p>He walked home, hands in his pockets, head down, indistinguishable from anyone else on the street.</p><p>But different.</p><p>— -</p><p>Months passed.</p><p>Julian kept going to the library. Kept working. The atmospheric monitoring proposal was still “under review” — no response yet, but that was normal. The system took time. He threw himself into the next problem, the next gap in the framework.</p><p>Anything to keep from thinking about what was happening to his work in the silence between submission and response.</p><p>Then, in the building’s common area, he saw Marcus.</p><p>They hadn’t spoken in months. Not since Julian had found the Chen framework, back when he still had time for entertainment and neighbors and anything but the work.</p><p>Marcus was watching something on his handheld, the sound turned up low. There was no one else there.</p><p>Julian almost walked past. Almost kept going, kept his head down, stayed in the silence.</p><p>But he needed to know.</p><p>“Marcus.”</p><p>Marcus looked up, surprised. Then wary. “Julian. I thought you’d moved or something.”</p><p>“No. Just… busy.”</p><p>“Yeah, I heard.” Marcus turned back to his handheld. “Working on your proposals.”</p><p>“I need to ask you something.” Julian stepped closer. “You know how the review process works? The Contributor evaluation system?”</p><p>Marcus paused the video. For the first time, Julian saw something in his eyes besides resignation. Pity.</p><p>“You really don’t know?”</p><p>“Know what?”</p><p>Marcus’s cousin worked at an elite center. Not as anything important — maintenance oversight, the kind of manual labor that drones couldn’t do yet. But she saw things.</p><p>Julian hadn’t spoken to Marcus in months. Hadn’t spoken to anyone, really. The work had consumed him.</p><p>But he needed to know. Needed to understand how the system actually worked, not how it claimed to work.</p><p>Marcus sighed. He gestured for Julian to sit, then hesitated. “You know those Contributor success stories they run sometimes? On the feeds? ‘Resident Makes Breakthrough, Granted Contributor Status’?”</p><p>Julian nodded. Everyone had seen them. Brief, inspirational pieces about people from the Residential Class who’d contributed something valuable and been granted access to the Contributor tier.</p><p>“You know how many there are?”</p><p>“A few? Dozen?”</p><p>“In the last decade? Three.”</p><p>Julian stared at him. “Three?”</p><p>“Three. And one of them was dead before they ran the story.” Marcus leaned forward. “But that’s not the part you need to know. The part you need to know is: how many submissions actually get reviewed?”</p><p>“I don’t know.”</p><p>“All of them.”</p><p>Julian waited.</p><p>Marcus laughed, short and bitter. “You’re not getting it. All of them get reviewed. Most within weeks. The 6–12 month thing? That’s just what they tell you so you don’t expect a response.”</p><p>Julian felt cold. “So they see everything. And they reject almost all of it.”</p><p>“No.” Marcus’s voice dropped. “They don’t reject it, Julian. They harvest it.”</p><p>“I don’t — “</p><p>“My cousin told me. She’s seen it. A submission comes in — something good, something valuable. It gets flagged. Then it gets stripped. Name removed, origin scrubbed, and sent up the chain. Some months later, someone in the Contributor tier ‘discovers’ it. Gets credit, gets status, gets access.”</p><p>Julian sat back. “They’re stealing it.”</p><p>“They’re not stealing it. That’s the thing. It’s not stealing if you don’t own anything. You submitted it to a public system. The system owns it.” Marcus stood up. “You think you’re doing research. You think you’re contributing. What you’re doing is free labor for a machine that will never acknowledge you existed.”</p><p>Julian stood up too. “Then why did you not tell me to stop? If they’re just going to steal it anyway, why does it matter?”</p><p>Marcus looked at him for a long moment. Then something softened in his expression. Or maybe broke.</p><p>“Because I remember,” he said quietly. “I remember when my cousin actually cared about things. Before she got the job at the center. Before she saw what they actually do with people like us.” He shook his head. “She’s not the same, Julian. Knowing what she knows… it broke something. And I don’t want that for you.”</p><p>He walked toward the door, then stopped.</p><p>“For what it’s worth? She said your atmospheric proposal was flagged three months ago. They’re already using it.”</p><p>Marcus walked out, and Julian was alone in the common area with the hum of the climate control system and the distant sound of someone’s entertainment bleeding through the walls.</p><p>— -</p><p>Julian sat at the library desk for a long time.</p><p>The manuscript was spread in front of him. Three months of work. The most meaningful thing he’d ever done. Already used. Already claimed. Already gone.</p><p>He could destroy it. Walk away. Go back to the apartment, turn on the handheld, let the algorithm wash over him until the days blurred together again and he forgot he’d ever wanted anything more.</p><p>It would be easier.</p><p>He picked up the manuscript and began to read.</p><p>It was good. He knew it was good. Not because he was brilliant — Chen was brilliant, he was just filling in the edges — but because it was real. Actual knowledge. A real contribution to something that mattered.</p><p>The system could steal the credit. It could steal his name, his future, any chance of recognition or status or escape from the grey uniformity of Residential Class life.</p><p>But it couldn’t steal the act.</p><p>The contribution was real. The framework was better because he’d existed. The atmospheric models — wherever they were being used, whoever was getting credit for them — were more accurate because Julian Steiner, son of Thomas Steiner, had spent three months sitting at a wooden desk in a dead library, doing work by hand that computers were supposed to do.</p><p>That was true.</p><p>No one would ever know. No one would ever thank him. But it was true.</p><p>Julian stood up and walked to the library terminal.</p><p>**SUBMIT QUERY: Status of Proposal 77241955–047**</p><p>**RESPONSE: UNDER REVIEW.**</p><p>Under review. For eleven months. While they were already using it.</p><p>Julian didn’t care. He opened a new submission form.</p><p>**TITLE: Micro-Scale Atmospheric Dynamics: A Predictive Framework for Low-Pressure Environments (Supplemental to Chen 2048)**</p><p>**AUTHOR: Julian Steiner, ID 77241955**</p><p>**ABSTRACT: This manuscript presents a methodology for predicting micro-scale atmospheric dynamics using observable environmental data, addressing the gap identified in Chen (2048) regarding real-world data collection in enclosed systems. The framework relies on manual observation and correlation analysis, making it applicable in environments without access to computational resources…**</p><p>He attached the manuscript. The full thing.</p><p>Then he hit submit.</p><p>**CONFIRMATION: SUBMISSION RECEIVED. REVIEW PERIOD: 6–12 MONTHS.**</p><p>Julian walked out of the library into the grey afternoon.</p><p>Outside, the sun found purchase through the clouds. Temporary. Fleeting.</p><p>But there.</p><p>— -</p><p>**THE END**</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=637485a3a87a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Undercurrents of Data Engineering: What Really Holds Data Systems Together]]></title>
            <link>https://medium.com/@UmidDey/undercurrents-of-data-engineering-what-really-holds-data-systems-together-3b2e7b8d04e6?source=rss-2885b69dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b2e7b8d04e6</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Umid Kumar Dey]]></dc:creator>
            <pubDate>Wed, 20 Aug 2025 20:23:06 GMT</pubDate>
            <atom:updated>2025-08-20T20:24:35.604Z</atom:updated>
            <content:encoded><![CDATA[<p><em>I’m writing these articles as part of my learning journey — breaking down what I read, watch, or explore each day into short write-ups. It’s my way of thinking out loud, staying consistent, and hopefully helping others who are learning too. No fluff, no filler — just clear ideas, explained simply (or at least I try to).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ul0nKkM-eUhASut6eKMZA.png" /><figcaption>Generated by Gemini</figcaption></figure><p>When I first started exploring data engineering, I thought it was just about moving data from A to B. But the deeper I dig, the more I see that real craft isn’t pipelines or queries — it’s the undercurrents driving every phase of the lifecycle, the invisible forces shaping success or collapse. That aligns tightly with Reis &amp; Housley’s framework <em>(written down in Fundamentals of Data Engineering)</em>, which shows how these six themes support every stage of the data engineering lifecycle.</p><p>Here’s how I have absorbed them to improve my understanding.</p><h3>Security: Protect What You Build</h3><p>If data is the new oil, security is the refinery fence. Locks on front gates aren’t enough — every dataset, query, and access path is an attack vector.</p><p><strong>Example:</strong> You assume your team’s internal transfer API is safe. One day, a staging token ends up in a public repo, and before you know it, data is leaked.</p><p><strong>Takeaway:</strong> Treat every dataset as if it will someday be exposed — because it might.</p><p><strong>How:</strong> Enforce least-privilege access, rotate credentials automatically, set up monitoring for suspicious queries, and treat staging/test environments with the same rigor as production.</p><h3>Data Management: The Backbone of Trust</h3><p>Data without management is like having mislabeled jars — possibly edible, but risky. Governance, lineage, quality, metadata: these are your ingredients for reliability.</p><p><strong>Example:</strong> A business dashboard shows <em>Revenue Q2: €500k,</em> but nobody knows whether it includes revisions or refunds. Trust in the dashboard collapses overnight.</p><p><strong>Takeaway:</strong> Without context and clarity, even slick pipelines become liabilities.</p><p><strong>How:</strong> Establish data catalogs, enforce naming conventions, automate data quality checks, and maintain lineage so anyone can trace outputs back to inputs.</p><h3>DataOps: DevOps with a Data Soul</h3><p>Pipelines are highways; DataOps is traffic control. Automation, versioning, observability — they turn 3 a.m. DST shifts from chaos to confidence.</p><p><strong>Example:</strong> A schema change at midnight breaks downstream jobs; after introducing CI/CD and data contract enforcement, similar changes happen silently and safely.</p><p><strong>Takeaway:</strong> Build until failures become predictably boring.</p><p><strong>How:</strong> Add automated tests for data flows, monitor with alerting dashboards, use feature branches for pipeline code, and deploy schema contracts to catch issues early.</p><h3>Data Architecture: The Blueprint That Outlives You</h3><p>Architecture isn’t slide art — it’s a long-term contract. Conceptual, logical, physical layers must align with future adaptability.</p><p><strong>Example:</strong> You build a monolithic ETL that hit scale and then collapsed — relying on it cost months to refactor. A modular, layered approach would have spared everyone.</p><p><strong>Takeaway:</strong> Design for change — today’s “good enough” often becomes tomorrow’s technical debt.</p><p><strong>How:</strong> Separate storage from compute, design with modular layers (ingest → transform → serve), prefer schema evolution over schema freeze, and document trade-offs clearly.</p><h3>Orchestration: Making the Orchestra Play Together</h3><p>Without orchestration, tasks are soloists tuning in silence. Tools like Airflow or Prefect bring cohesion — tracking dependencies, retries, lineage.</p><p><strong>Example:</strong> Batch workflows run in random order each morning. After using orchestration, failures become isolated, restartable, and traceable.</p><p><strong>Takeaway:</strong> Orchestration isn’t about scheduling — it’s about trust in the flow.</p><p><strong>How:</strong> Define DAGs (directed acyclic graphs) for workflows, set retry logic, log every task run, and use orchestration metadata to debug failures quickly.</p><h3>Software Engineering: The Foundation Nobody Escapes</h3><p>Data engineering is still software engineering, at least it should be approach as such. Managed services don’t cure bad design. You still need clean abstractions, modular code, robust tests.</p><p><strong>Example:</strong> You build dozens of ad-hoc ingestion scripts (“just this one field”). Months later, You are in a tangled mess. Refactoring with modular, tested abstractions cleared the mess.</p><p><strong>Takeaway:</strong> Even in a no-code world, engineering fundamentals are non-negotiable.</p><p><strong>How:</strong> Write modular, reusable functions, adopt version control with PR reviews, build test suites (unit + integration), and apply CI/CD to pipeline code.</p><h3>Final Thought (Refined &amp; Sharper)</h3><p>Pipelines may be visible, but undercurrents are what sustain them.</p><blockquote><strong>Pipelines earn attention. Undercurrents earn trust.</strong></blockquote><p>That’s the difference between a fragile system and one that frees you to innovate.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b2e7b8d04e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What a Data Engineer Really Does: Beyond the Code and Pipelines]]></title>
            <link>https://medium.com/@UmidDey/what-a-data-engineer-really-does-beyond-the-code-and-pipelines-c7e9c407df33?source=rss-2885b69dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7e9c407df33</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Umid Kumar Dey]]></dc:creator>
            <pubDate>Fri, 01 Aug 2025 19:11:54 GMT</pubDate>
            <atom:updated>2025-08-01T19:11:54.916Z</atom:updated>
            <content:encoded><![CDATA[<p><em>I’m writing these articles as part of my learning journey — breaking down what I read, watch, or explore each day into short write-ups. It’s my way of thinking out loud, staying consistent, and hopefully helping others who are learning too. No fluff, no filler — just clear ideas, explained simply.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hW_I-lwSzlgkicQ_" /><figcaption>What it is all about</figcaption></figure><p>The deeper I get into data engineering, the more I realize this isn’t just about building data pipelines or tuning Spark jobs. The role is bigger than that — it’s a balancing act between two worlds:</p><ul><li>The <strong>technical depth</strong> required to make data systems fast, scalable, and reliable</li><li>And the <strong>business understanding</strong> that ensures those systems actually solve the right problems</li></ul><p>When either side gets ignored, the whole thing falls apart.</p><h3>On the Business Side: It’s About People, Not Just Pipelines</h3><p>No matter how elegant your ETL process is, if it doesn’t support real-world decisions, it misses the point. Data doesn’t exist in a vacuum — it exists <strong>for people</strong>.</p><p>I’m learning to ask better questions:</p><ul><li>Who are the stakeholders?</li><li>What do they actually need?</li><li>How does the organizational structure affect what gets built?</li></ul><p>Building relationships isn’t a soft skill here — it’s critical. It’s how you surface real requirements, set shared expectations, and avoid costly misunderstandings.</p><p>It also means scoping projects realistically:</p><ul><li>What does success look like — technically and organizationally?</li><li>How can we deliver value quickly, not just completely?</li></ul><p>And yes, things like Agile and DevOps aren’t just trendy terms. At their best, they’re frameworks for collaboration and feedback loops — the antidote to “tech for tech’s sake.”</p><h3>Be Ruthless With Costs and Curious With Change</h3><p>Another mindset I’m working on: <strong>treat every project like it’s your own money.</strong></p><p>Total cost of ownership includes time-to-value, maintenance, documentation, and even team onboarding. Optimizing for <em>clarity and predictability</em> often beats optimizing for <em>raw performance</em>.</p><p>And with how fast the data ecosystem evolves, <strong>learning never stops</strong>. There’s always a new tool, a new abstraction, a new acronym. But instead of chasing every trend, I’m trying to build a filter for what’s useful, what’s hype, and what’s aligned with the work I’m doing.</p><h3>On the Technical Side: Design for Scale, Reliability, and Change</h3><p>Of course, you still have to build things that work — and keep working.</p><p>I’m building my mental model around a simple lifecycle, as mentioned in the book “Fundamentals of Data Engineering”:</p><ol><li><strong>Generation</strong> — Where the data originates (apps, logs, APIs)</li><li><strong>Storage</strong> — Durable, scalable systems to hold it</li><li><strong>Ingestion</strong> — Smooth and reliable transfer into the system</li><li><strong>Transformation</strong> — Cleaning, joining, modeling the data</li><li><strong>Serving</strong> — Making it available for consumption (dashboards, ML, APIs)</li></ol><p>And running through all of it are disciplines like:</p><ul><li><strong>Security</strong> — Protecting sensitive data at every step</li><li><strong>Data Management</strong> — Ensuring consistency, documentation, discoverability</li><li><strong>Data Architecture</strong> — Designing systems that make sense long-term</li><li><strong>Orchestration &amp; DataOps</strong> — Making everything flow, reliably and repeatably</li><li><strong>Software Engineering</strong> — Writing code that’s testable, maintainable, and scalable</li></ul><p>People still ask: <em>Do data engineers really need to know how to code?</em><br>Short answer: <strong>Absolutely.</strong></p><p>Even with today’s managed services and no-code tools, understanding software design patterns — like abstraction, modularity, and error handling — helps avoid brittle, bloated pipelines that fall apart when anything changes.</p><h3>Final Thought</h3><p>The best data engineers I’ve seen don’t just write great code or build cool tools. They <strong>connect the dots</strong> between what the business needs and what the tech can deliver — without overengineering or overcomplicating.</p><p>This isn’t about being a “10x” developer. It’s about being a <strong>10x enabler</strong> — someone who builds things that are useful, used, and built to last.</p><p>That’s the kind of data engineer I’m trying to become.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7e9c407df33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Choose the Right Data Storage System: Questions Every Engineer Should Ask]]></title>
            <link>https://medium.com/@UmidDey/how-to-choose-the-right-data-storage-system-questions-every-engineer-should-ask-f6db11f5a13e?source=rss-2885b69dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6db11f5a13e</guid>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[cloud-architecture]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Umid Kumar Dey]]></dc:creator>
            <pubDate>Mon, 14 Jul 2025 21:46:36 GMT</pubDate>
            <atom:updated>2025-07-14T21:46:36.566Z</atom:updated>
            <content:encoded><![CDATA[<p><em>I’m writing these articles as part of my learning journey — breaking down what I read, watch, or explore each day into short write-ups. It’s my way of thinking out loud, staying consistent, and hopefully helping others who are learning too. No fluff, no filler — just clear ideas, explained simply.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i4p1pACKnQQjjkVi" /><figcaption>Generated by ChatGPT</figcaption></figure><p>As I dig deeper into data engineering, I’m starting to realize something that sounds simple, but isn’t:</p><blockquote><strong><em>Not all storage is created equal — and choosing the wrong one early on can haunt the entire data architecture.</em></strong></blockquote><p>Whether you’re building on S3, Delta Lake, Snowflake, or a good old relational database, your choice of storage shapes how data flows, how it scales, and how usable it is for the people downstream.</p><p>Here’s the thinking framework I’m starting to build for myself — not a checklist, but a set of mental filters.</p><h3>Can It Handle the Workload and the Growth?</h3><p>It’s easy to optimize for today. But I’ve started thinking in “future load units”:</p><ul><li>What happens when data volume doubles?</li><li>How many users will be reading/writing concurrently?</li><li>Will I need to reorganize files just to stay performant?</li></ul><p>Good storage systems make future scale invisible. Bad ones punish you for being successful.</p><h3>Is the Data Discoverable and Usable by Others?</h3><p>This one hit me hard. It’s not just about storing data — it’s about enabling discovery.</p><p>If I’m the only person who understands the structure, lineage, or naming conventions, then I’ve failed to make it a shared asset. Tools like data catalogs or built-in metadata layers become critical here.</p><h3>Will It Meet Latency and SLA Expectations?</h3><p>Sometimes, the question isn’t “Can this store data?” — it’s “Can it serve data fast enough?”</p><p>If business teams or pipelines rely on fast access during certain windows, that becomes part of the design criteria. Otherwise, your shiny setup might pass tests — but fail in production.</p><h3>How Well Does It Deal With Change?</h3><p>Data is dynamic:</p><ul><li>Schemas evolve</li><li>Tables get deprecated</li><li>Business logic shifts</li></ul><p>If the storage layer can’t flex with these realities — through schema versioning, soft deletions, or robust metadata tracking — then technical debt starts accumulating quickly.</p><h3>Understand the Native Behavior of the Storage</h3><p>Before choosing any system, I should ask:</p><ul><li>What was this built for?</li><li>Am I using it as intended — or am I hacking around its limitations?</li></ul><p>For example, object stores like S3 are optimized for large, immutable files. If I’m constantly writing small files or overwriting them, I might be setting myself up for performance bottlenecks.</p><p>Same goes for trying to use a data warehouse as a low-latency event store. It works — until it doesn’t.</p><h3>Final Thought</h3><p>The more I learn, the more I realize that great architecture isn’t about using every tool — it’s about knowing what not to use, and when.</p><p>Storage decisions aren’t glamorous, but they define the shape of everything else. If I want to build scalable, human-friendly systems, I have to start with these types of questions — before I fall in love with a particular tool or cloud service.</p><p>More soon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6db11f5a13e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>