<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Astropomeai on Medium]]></title>
        <description><![CDATA[Stories by Astropomeai on Medium]]></description>
        <link>https://medium.com/@astropomeai?source=rss-9af0df47ddde------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JvmCvCusx5Cz3VJXxITBjA@2x.jpeg</url>
            <title>Stories by Astropomeai on Medium</title>
            <link>https://medium.com/@astropomeai?source=rss-9af0df47ddde------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:55:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@astropomeai/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building MissionOS: AI Mission Control, Tested from the Physics Up]]></title>
            <link>https://medium.com/@astropomeai/building-missionos-ai-mission-control-tested-from-the-physics-up-866f2fcf4038?source=rss-9af0df47ddde------2</link>
            <guid isPermaLink="false">https://medium.com/p/866f2fcf4038</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[physical-ai]]></category>
            <dc:creator><![CDATA[Astropomeai]]></dc:creator>
            <pubDate>Sun, 01 Nov 2026 01:03:43 GMT</pubDate>
            <atom:updated>2026-11-01T01:03:43.238Z</atom:updated>
            <content:encoded><![CDATA[<p>A development story about return planning, fault diagnosis, simulation fidelity, and carrying a decision through to its outcome.</p><p>I wanted to give an AI responsibility for a continuing mission: watch the vehicle, gather evidence, compare its options, request approval when necessary, and follow its decisions through execution.</p><p>A Starship-inspired simulator became the place to build that idea. Launch, payload deployment, orbital waiting, recovery, and engine faults form a connected chain. A choice made during deployment changes the mass that returns. Waiting for another orbit changes the resources available for recovery.</p><p>Building MissionOS meant developing that chain at two levels: the system that makes and governs decisions, and the physical environment that gives those decisions consequences.</p><p>The most revealing part of the project came when we examined the assumptions behind our own successful flights. Comparing the simulator with public flight data led us into attitude dynamics, center of mass, flap authority, and a substantial reconstruction of the return controllers. The eventual integrated flight carries that whole development history.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xM5KhxJtvsXQ8uf5jWJdGw.png" /></figure><h3>The simulator that makes the mission continuous</h3><p>At its core is a <strong>six-degree-of-freedom model</strong> that integrates translation and rotation together. It carries position, velocity, a quaternion describing body orientation, and angular velocity. Forces and moments determine how those states evolve through a fourth-order Runge–Kutta integrator.</p><p>A gimballed engine changes both the thrust direction and the moment about the center of mass. Burning propellant changes the vehicle’s mass, center of mass, and inertia. The attitude controller requests an orientation; the dynamics determine how closely the vehicle can follow it. This coupling gives a control decision consequences throughout the trajectory.</p><p>Individual engines have configured thrust, specific impulse, mounting positions, and gimbal limits. Throttle, gimbal, and flap commands pass through actuator lag and rate limits before producing forces. RCS consists of finite thrusters at declared positions and directions, consuming propellant. Requested and achieved actuation are recorded separately.</p><p>The environment includes a WGS84 Earth, Earth rotation, gravity with J2, and a standard-atmosphere model. Local airflow produces forces on generic hull panels, flaps, and grid fins. Aerodynamics use a plate approximation; the atmosphere above 86 km uses an exponential surrogate. Thermal protection, structural flexibility, and propellant slosh remain outside this model’s coverage.</p><p>A continuous mission proceeds from launch through stage separation, Ship insertion, payload release, deorbit, entry, flip, and hull contact. Booster return inherits the actual separation state and engine failures. Released payloads have finite mass and inertia: separation applies a reaction to the parent, then each body propagates its own motion. A deployment choice therefore changes the mass that comes home.</p><p>The physics runs in a separate process. MissionOS receives observations and candidate assessments; a checked AI proposal becomes an objective for the Executor. Later observations establish what changed. Model API credentials stay with the host, outside the physics worker.</p><figure><img alt="How the simulator connects to MissionOS: physical state, observations, judgment, permission, execution, and saved evidence." src="https://cdn-images-1.medium.com/proxy/1*H8cj7b3XImYBCfsgP3n9wQ.png" /></figure><p><em>Editorial architecture diagram based on the public implementation. It separates the physical plant, numerical tools, AI judgment, permission, execution, and replay.</em></p><p>The CG replay reads saved states. Recorded quaternions determine the vehicle’s orientation, while actuator states, mission time, and the model’s external decision summary provide context. Ocean, clouds, lighting, and camera movement make that record easier to follow. The physical calculation and the presentation remain separate artifacts.</p><p>Numerical checks include analytic solutions, conservation properties, and timestep comparisons. The project also provides validation paths against declared NASA check cases and an independent dynamics implementation, Basilisk. Each comparison records the equations and conditions it covers.</p><h3>First, give the controller something meaningful to compare</h3><p>The first milestone, M1, centered on a recovery area becoming unavailable.</p><p>MissionOS had to notice the update, assess return opportunities, choose a plan, and confirm the eventual contact. An event-driven monitor detected one advisory update within <strong>0.2 simulated seconds</strong> and started a new assessment.</p><p>We then made candidate preparation a responsibility of the tool layer. MissionOS reuses an applicable forecast and calculates the missing alternative. Each candidate includes waiting time, fuel loss allowance, battery energy, temperature, recovery timing, and uncertainty. Preparation has a budget calculated backward from the existing return deadline.</p><p>That gives the model a concrete comparison. In one recorded decision, DeepSeek weighed overlapping recovery-time estimates against the additional resource cost of waiting for another orbit, then retained the original plan.</p><p>We also gave numerical evidence stable IDs. A model can point to the candidate fields supporting its decision, and a checker can resolve those IDs against the snapshot supplied at judgment time. This makes the cited numerical basis traceable. The model’s prose remains a separately recorded explanation.</p><h3>Carry the consequences into the next stage</h3><p>M2 joined deployment supervision to return planning. The returning vehicle inherits its actual retained payload inventory and corresponding mass.</p><p>M3 added inflight scope amendment. A proposed return can be physically feasible while requiring a broader approval grant. MissionOS binds approval to the particular candidate, its digest, and its expiry, then records one-time consumption and checks the resulting reservation in a later observation.</p><p>An authenticated test-operator script supplied the approval in these experiments. The exchange crossed the real HTTP boundary, connecting the approval request, candidate identity, authorized update, and subsequent observation.</p><figure><img alt="An M3 replay combining the orbital vehicle, AI decision, approval, and telemetry." src="https://cdn-images-1.medium.com/proxy/1*aJKlDBuRzorFAD4AWRs4Nw.jpeg" /></figure><p><em>October 12 M3 development record: 26 payload releases, a next-orbit decision, candidate-bound approval, and return in a 112-second replay.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9udkr9AbCUE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9udkr9AbCUE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F9udkr9AbCUE%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c89993279f5b25cbfc943efb73a761eb/href">https://medium.com/media/c89993279f5b25cbfc943efb73a761eb/href</a></iframe><p>The division of work stays consistent:</p><p><strong>AI judges. A human approves the scope. Rules constrain. Executor acts. Verifier checks.</strong></p><p>Jev handles monitoring and bounded choices. DeepSeek handles selected deeper assessments. Numerical tools estimate and predict. Fast flight-control loops execute the permitted objective.</p><h3>Build fault diagnosis on measured sensitivity</h3><p>The trouble engine brought reproducible disturbances into the physical simulation. A source-linked incident catalog informed six test families: Booster ascent engine loss, Ship engine loss, Booster restart failure, directional RCS loss, a deployment-mechanism fault, and tower unavailability. Their seeded probabilities are engineering test settings.</p><p>For symptom-based recovery, we concentrated on Ship propulsion capability. Before introducing AI, we separated the capability assumed by the controller from the capability used by the physical plant.</p><p>A <strong>3×3 matrix</strong> crossed assumed and actual thrust at 100%, 80%, and 60%. The three matched cells satisfied the contact gates. All six mismatched cells exceeded the contact-speed limit, exposing the return procedure’s sensitivity to the capability estimate. Every outer control/physics step carried an audit of which vehicle model supplied each value.</p><figure><img alt="Contact speed across nine assumed-versus-actual thrust combinations." src="https://cdn-images-1.medium.com/proxy/1*SyGw6zkPwlaDbPDj1wd0SQ.png" /></figure><p><em>Physics-only tests from one saved starting state. Values are contact speed in m/s; green marks the 5 m/s speed gate. These sampled points do not establish a continuous operating envelope.</em></p><p>Next came the estimator. With synthetic sensor noise, bias, and latency fixed in advance, it detected the 80% and 60% capability conditions in <strong>20 of 20 seeds each</strong>. The healthy condition produced <strong>zero alarms in 20 seeds</strong>. The true factor lay inside the reported band in every case.</p><p>We then tested post-burn adoption at five assumed values — <strong>76%, 78%, 80%, 82%, and 84%</strong> — against actual 80% thrust. All five met the registered gates at that saved starting state. The 76% point contacted at about <strong>4.96 m/s</strong>, close to the 5 m/s limit. These discrete measurements gave the later AI requests a physically grounded treatment of uncertainty.</p><p>Finally, we introduced an alternative explanation for the same symptom: an accelerometer gain error. Independent tracking supplied another view of the vehicle’s motion. The AI could request that observation, use the evidence to choose a response, and send its proposal through the independent check.</p><p>The work of diagnosis now had a complete sequence: symptom, information request, numerical estimate, choice, check, execution, and observed result.</p><h3>Check a successful flight against the world outside the model</h3><p>Alongside the mission architecture, we audited the simulator against public Flight 14 material.</p><p>The reference work separated official schedules, values read from the broadcast display, and quantities inferred from those readings. We compared altitude and speed as time series, checked selected frames by eye, and corrected several transcription errors while retaining the original data.</p><p>This changed the questions we asked of the physics. Ascent, stage separation, Booster apogee, return timing, entry attitude, and terminal descent all became measurable parts of the comparison. Calibration targets and held-out checks were registered separately, with a limited set of adjustable parameters.</p><p>The most consequential analysis concerned Ship’s entry attitude.</p><p>A static trim study examined the old panel-aerodynamics model with roughly <strong>103 tonnes of propellant</strong>. In the intended belly-oriented condition, the remaining pitch moment at 5 kPa dynamic pressure was about <strong>1.0 MN·m</strong>. The modeled RCS pitch authority was about <strong>0.08 MN·m</strong>. Center of mass, aerodynamic center, and flap geometry therefore became concrete design constraints.</p><p>Saved successful flights supplied another clue: their body-frame airflow had substantial components on two axes, corresponding to an approximately <strong>45-degree oblique flow orientation</strong>. The panel model could obtain a balancing solution in that orientation. The analysis showed that those earlier successful flights relied on a balancing solution produced by oblique airflow in that particular model. Its attitude and flap behavior differed from the intended belly-oriented entry. We made checking trim in the intended belly orientation an explicit task in redesigning the return.</p><p>The static result became a diagnosis to investigate through subsequent design and flight checks. We examined mass distribution and flap trim, revised attitude and terminal guidance, and carried the states produced by the revised ascent into both vehicles’ return calculations. The research profile also incorporated the declared Flight-14-style engine shutdowns.</p><p>The result was a broader recovery effort: ascent, entry preparation, Booster return, landing-burn timing, numerical integration, and the interfaces between them were reviewed together. Each new controller had to work with the state produced by the preceding phase.</p><p>Agreement with public flight data remains limited after this reconstruction; these tests do not validate a real SpaceX vehicle. Booster recovery is assessed within a broad <strong>10 km development acceptance envelope</strong>.</p><p>That process gave the project a stronger foundation: successful trajectories came with an account of the assumptions and physical balances that supported them.</p><h3>Turn recovery research into decision material</h3><p>Booster recovery developed along its own path. We built models for the tower, support pins, moving arms, contact loads, and support settling, alongside return-guidance experiments.</p><p>A tower capture requires position, velocity, attitude, angular rate, fuel, and arm timing to align at the same encounter. Keeping those stages explicit helped turn a visually attractive objective into a set of engineering questions. A launch-derived return ending in supported tower capture has yet to succeed in these experiments. The integrated demonstrations use ocean recovery as their executed objective.</p><p>The research history itself became an input to mission judgment. In the recorded <strong>October 14</strong> recovery experiment, DeepSeek received the current Booster observation and a reviewed report of catch-development evidence. It selected splashdown; an independent execution check accepted the choice, and a later observation confirmed the selected objective.</p><p>The <a href="https://github.com/user-attachments/assets/00a07f6d-93a3-407b-9f38-b6d017bd1c72">120-second replay</a> shows that earlier decision alongside telemetry, ocean scenery, and an illustrative tower cutaway.</p><figure><img alt="An earlier MissionOS recovery-decision replay with synchronized mission information." src="https://cdn-images-1.medium.com/proxy/0*YoKw0nP3Gl8jTNEM.jpg" /></figure><p><em>Video and image: October 14 development recording. The integrated-flight figures below belong to October 23.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fc0LhVXtGzL4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dc0LhVXtGzL4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fc0LhVXtGzL4%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/da7ddbd7a8746484ef52312ed8c0dd8f/href">https://medium.com/media/da7ddbd7a8746484ef52312ed8c0dd8f/href</a></iframe><h3>Make the experiment as traceable as the mission</h3><p>We registered each bounded experiment before execution: source, starting state, assumptions, allowed changes, contact gates, time and storage budgets, and the interpretation of possible outcomes.</p><p>Each attempt kept its own record and verdict. A later repair received a new identity. Saved-record verification let us investigate discrepancies while preserving the original numerical trajectory. Control-side assumptions and physical capability remained distinguishable throughout the audit.</p><p>Credential-free checker processes examined the permitted action separately from the model. Earlier nominal return qualification covered <strong>39 finite cases</strong>. Later exact-point work bound the current state, vehicle, inventory, plan, environment, and complete actuator history, with bounded chunks allowing every history row to be checked.</p><p>Those identities also define where a result applies. Each qualification belongs to its stated source and environment; each changed implementation has its own verification work.</p><figure><img alt="The earlier 39-case qualification study: retained payload count against contact speed, tilt, and remaining fuel." src="https://cdn-images-1.medium.com/proxy/1*TrsAj_lhS-G9poFTam-A2Q.png" /></figure><p><em>An earlier source-bound 39-flight study. Blue shows the inventory census, orange shows initial-fuel perturbations, and dashed lines show the gates. This is separate from later integrated flights and qualification of the public port.</em></p><p>Operating the models added another layer of engineering. We recorded variation in Jev’s choices for an identical input, model-name and response-format compatibility, and long periods before a usable DeepSeek answer. Timing diagnostics separated HTTP arrival, stream activity, and answer events. These observations informed the Jev-first structure, explicit deeper review, deadlines, and preapproved fallback behavior.</p><h3>Bring the layers together in one mission</h3><p>On <strong>October 23</strong>, a registered integrated experiment connected launch monitoring, Booster recovery review, image-assisted deployment control, return replanning, scripted HTTP approval, injected propulsion reduction, independent tracking, and checked capability adoption.</p><p>Jev released thirteen payloads and retained thirteen; that mass continued into return planning. It requested tracking after the propulsion symptoms and selected the numerical estimate. The independent guard preceded one-time application, and the verifier followed the return to contact.</p><p>The flight recorded <strong>10 valid Jev responses in 10 calls</strong>, <strong>one valid DeepSeek response in one call</strong>, Ship model contact at <strong>3.459 m/s</strong>, and <strong>51.209 tonnes</strong> of remaining propellant.</p><p>The October 22 attempt remains recorded as exceeding its registered time budget, with host sleep included in that interval. We then made a fresh registration with explicit timing boundaries and clock evidence, prevented host sleep, and executed a separate flight on October 23.</p><p>Before that flight, the registration defined the mission-supervisor interval for the <strong>1,800-second</strong> runtime gate. That interval measured <strong>1,304.871 seconds</strong>. The complete CLI, including separate pre-mission and post-mission checks, measured <strong>2,140.761 seconds</strong>. Both measurements stay attached to their original boundaries.</p><p>These figures describe one registered flight in a research simulator, together with its declared physical assumptions and test-operator approval. They sit alongside the earlier candidate comparisons, fault studies, static analyses, and recovery experiments.</p><p>The maintained integration is now public in <a href="https://github.com/pome223/missionos/pull/133">MissionOS PR #133</a>, merged as 56013c8b, with successful Python 3.11 and 3.13 CI. The <a href="https://github.com/pome223/missionos/blob/56013c8bd656b77e707a65e691be0f8c6f050ca5/docs/examples/starship-final-mission.md">integrated report</a> keeps execution, qualification, and public-release versions visible.</p><p>What I value most is the connection between judgment and investigation. The AI asks for evidence during a mission. As developers, we ask what supports a successful simulation, inspect the physical assumptions, and rebuild the pieces that need a stronger basis.</p><p>MissionOS is growing through that same cycle at both levels: observe, investigate, decide, check, execute, and carry the evidence forward.</p><h3>Explore the project</h3><ul><li><a href="https://github.com/pome223/missionos/blob/56013c8bd656b77e707a65e691be0f8c6f050ca5/docs/agents/starship-orbit-events.md">Event-driven monitoring</a> and <a href="https://github.com/pome223/missionos/blob/56013c8bd656b77e707a65e691be0f8c6f050ca5/docs/agents/starship-comparison-preparation.md">automatic candidate preparation</a>.</li><li><a href="https://github.com/pome223/missionos/blob/56013c8bd656b77e707a65e691be0f8c6f050ca5/docs/examples/starship-trouble-engine.md">Trouble-engine families and incident sources</a>.</li><li><a href="https://github.com/pome223/missionos/blob/56013c8bd656b77e707a65e691be0f8c6f050ca5/examples/spaceflight/starship-ship-fault-step1-5-registration.json">Capability-matrix protocol</a> and <a href="https://github.com/pome223/missionos/blob/56013c8bd656b77e707a65e691be0f8c6f050ca5/examples/spaceflight/starship-ship-fault-step2-registration.json">estimator/tolerance protocol</a>.</li><li><a href="https://github.com/pome223/missionos/blob/56013c8bd656b77e707a65e691be0f8c6f050ca5/docs/agents/starship-visual-research.md">Public-data observation ledger</a> and <a href="https://github.com/pome223/missionos/blob/56013c8bd656b77e707a65e691be0f8c6f050ca5/src/runtime/starship_entry_trim.py">entry attitude and trim preparation implementation</a>.</li><li><a href="https://github.com/pome223/missionos/blob/56013c8bd656b77e707a65e691be0f8c6f050ca5/docs/examples/starship-sixdof.md">Six-degree-of-freedom model overview</a> and <a href="https://github.com/pome223/missionos/blob/56013c8bd656b77e707a65e691be0f8c6f050ca5/docs/agents/starship-sixdof-contract.md">state, force, and verification contract</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=866f2fcf4038" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Ship to City and Back with MissionOS]]></title>
            <link>https://medium.com/@astropomeai/from-ship-to-city-and-back-with-missionos-cb4ef894847f?source=rss-9af0df47ddde------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb4ef894847f</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[physical-ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Astropomeai]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 15:41:49 GMT</pubDate>
            <atom:updated>2026-10-02T15:41:49.942Z</atom:updated>
            <content:encoded><![CDATA[<h3>Connecting a map-selected destination, a simulated parcel handoff, and a verified return into one delivery mission</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T1Jhi3Gr_PWgKfxq" /><figcaption>Sea approach toward Yokohama — video 00:20 (SIM 243.752 s), from the same 12× saved-camera simulation timelapse. Yokohama City / Project PLATEAU, CC BY 4.0; MissionOS modifications.</figcaption></figure><p>The drone leaves a stationary ship, crosses the water toward Yokohama, and approaches a delivery pad that is already occupied. It waits for entry, descends, and releases its parcel. A separate receiver confirms that the package has landed and come to rest. Then the aircraft climbs away, flies back to the ship, and lands.</p><p>This run used PX4/Gazebo simulation, CPU city fixtures (deterministic test stand-ins), and one live Jev decision. Four applicable verifiers confirmed the decisions, pad queue, payload delivery, and simulated flight.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FASQwaJpiEwQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DASQwaJpiEwQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FASQwaJpiEwQ%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/cdce6090d6c875e2cea85c43f5e61eee/href">https://medium.com/media/cdce6090d6c875e2cea85c43f5e61eee/href</a></iframe><p><em>Video: the complete mission in a 110.92-second timelapse at 12× simulation time, assembled from saved camera images from this run, including its inset cameras. </em><a href="https://github.com/pome223/missionos/blob/main/docs/examples/yokohama-map-delivery/REPORT.md"><em>Recording details</em></a><em>.</em></p><h3>Giving the mission a finish line</h3><p>I built this scenario to connect mission planning to an outcome that could be observed all the way through. For a delivery, that means knowing where the parcel ended up and whether the aircraft returned successfully.</p><p>That requirement shaped what I added to MissionOS: a map-selected destination tied to an explicit approval, clear boundaries between proposals and execution, a physically simulated parcel handoff, and a record connecting every stage to its result.</p><p>The route makes those pieces work together. PX4’s autopilot handles the authored sea route and coastal approach. Two city decision points, D1 and D2, provide places for bounded movement proposals. At D3, a scripted lead aircraft creates an occupied-pad situation, and independent Rules evaluate when entry is feasible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*yH_mpnIYGlKY1AbkHqzTXA.png" /><figcaption>The sea-to-city journey at video 00:10, simulation time approximately 123 seconds, from the saved camera images of this run. Yokohama City / Project PLATEAU, CC BY 4.0; adapted by MissionOS.</figcaption></figure><p>The delivery itself gives the mission a concrete finish condition. The parcel is a separate dynamic object in Gazebo. Releasing its attachment lets gravity and contact physics determine what happens next.</p><p>A receiver, the component responsible for confirming receipt, runs separately. It checks the parcel’s identity, separation from the aircraft, contact with the pad, and stable rest. An accepted receipt allows the aircraft to climb and begin the return journey.</p><p>The final steps are landing, deck contact, and disarm, which removes the aircraft’s motor-arming permission. The <a href="https://github.com/pome223/missionos/blob/main/docs/agents/yokohama-payload-delivery.md">cargo contract</a> makes this sequence explicit. Delivery and return each have an observable completion condition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*G3dvuhqhss5zT7JFp0laGg.png" /><figcaption>The delivery-pad camera inset at video 01:05, simulation time 784.252 seconds. The receiver confirms delivery through recorded separation, contact, and stable rest. Yokohama City / Project PLATEAU, CC BY 4.0; adapted by MissionOS.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ebQnFiUtw28_UDxwfGyAg.png" /><figcaption>Figure 1: The approved mission from ship departure through parcel receipt and return. The city-model interface is bounded to D1/D2; the recorded run used CPU fixtures. Schematic, not a geographic map.</figcaption></figure><h3>Making a map click part of the flight plan</h3><p>The 3D map gives the operator a direct way to shape the mission: move the delivery marker, inspect the resulting route, approve it, and start.</p><p>In this version, candidates can be placed within 30 metres of the original pad. The recorded mission used a destination 10 metres east of it. D1, D2, and D3 stay in place while the final approach is checked against the city geometry.</p><p>GoalPlanner handles this deterministically. It obtains terrain height and checks ground variation and clearance from building and bridge footprints. An acceptable marker becomes a specific, inspectable plan.</p><p>Approval then binds the goal, route, scene, and runtime source hashes. Moving the marker invalidates the previous approval, including when the replacement candidate is rejected. The approved target stays fixed during flight, and the Gateway checks the plan against current inputs before dispatch.</p><p>This connects an interaction in the browser to the exact mission being executed. The <a href="https://github.com/pome223/missionos/blob/main/docs/agents/yokohama-map-goal.md">map goal contract</a> records the relationship between the marker, geometry, approval, and execution.</p><h3>Where each model fits</h3><p>There are two different scales of decision here: deciding how to carry a mission forward, and proposing a short movement within an approved flight segment.</p><p>MissionOS’s general Google ADK workflow uses DeepSeek V4 Flash through LiteLLM, a Chief Agent, deterministic routing to one specialist, and a SafetyCritic to propose and review higher-level mission actions.</p><p>Mission Assurance is a separate judgment layer, with Jev available as a backend. The Gateway records proposals and approvals and invokes the executor, the component that carries out an authorized action. The <a href="https://github.com/pome223/missionos/blob/main/docs/concepts/agent-roles.md">agent roles</a> and <a href="https://github.com/pome223/missionos/blob/main/src/agents/model_config.py">model configuration</a> describe these responsibilities.</p><p>For Yokohama, the chat entry point has a dedicated DeepSeek planner and pad judge. It interprets requests against a one-route catalog. The map entry point uses GoalPlanner directly.</p><p>AeroVLA and ANWM belong at the flight-execution interface. AeroVLA uses imagery and a goal to propose a bounded short movement. ANWM, the world-model component, predicts candidate and hold views from camera history. Those proposals and forecasts are combined with independent geometry Rules, fresh observations, and arrival checks.</p><p>Their native integration covers two short segments at D1/D2, with authored autopilot segments carrying the rest of the route. This makes the integration modular: the mission can retain its approval and completion logic while the movement-proposal implementation changes. The <a href="https://github.com/pome223/missionos/blob/main/docs/agents/yokohama-native-flight.md">native integration contract</a> documents that interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pk5GqfV2fgL4ifuYkb9c_A.png" /><figcaption>Figure 2: The general agent workflow, dedicated Yokohama chat, and map workflow are separate entry paths. The diagram also distinguishes native city inference, the recorded fixture-based run, and the public demo configuration.</figcaption></figure><h3>A real decision at the delivery pad</h3><p>The pad queue provides a small but useful place to connect a live judgment model to the mission.</p><p>Jev is <a href="https://docs.typesafe.ai/introduction">TypeSafe’s structured decision model</a>. In this integration, its <a href="https://docs.typesafe.ai/primitives/choice">Choice interface</a> selects between predefined options: continue, adding no intervention, or hold for a bounded additional wait. Rules determines whether entry is feasible; Jev can delay an eligible entry within the remaining allowance. The adapter gives each choice a fixed operational meaning.</p><p>In the recorded run, Jev 1.13.0 made one actual HTTP decision. It returned continue in 205.088 milliseconds and added no waiting.</p><p>That gives the integration a traceable result: the state presented to a model, a constrained answer, and its effect on the mission. The model’s role is visible in the record.</p><p>Earlier DeepSeek chat trials show why this boundary matters. One rationale suggested the lead aircraft might still be within the pad’s six-metre radius when it was 11.4 metres away; a native-model trial also invented a requirement for human confirmation. Machine-checked Rules retained entry authority inside the approved plan, and the judge could only add a bounded wait. These historical DeepSeek examples illustrate the boundary used here; they are not errors observed in the recorded Jev call.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*RSUqi-jS5ehxVwpeLSb2OQ.png" /><figcaption>Waiting before pad entry at video 00:50, simulation time 602.752 seconds. Rules evaluates the observed queue state, and the mission judge can add a bounded wait. Yokohama City / Project PLATEAU, CC BY 4.0; adapted by MissionOS.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ixDf0HAnbXuP8hZnE1SwSA.png" /><figcaption>Figure 3: The authority chain for the map mission. Approval identifies the plan; fresh Rules and executor checks govern movement; receiver evidence and independent verifiers establish the outcome.</figcaption></figure><h3>The checks improved the system</h3><p>The evidence checks also exposed implementation problems worth fixing.</p><p>One was a coordinate-frame mismatch. Heights in the city source geometry, Gazebo world Z, and PX4’s altitude relative to home represent different quantities. A plausible-looking number can point the aircraft toward the wrong height when those frames are mixed.</p><p>The transport now keeps the approved world target fixed and derives the command from fresh observed references. It prepares the upload protocol before collecting the observation used for that mapping, then checks freshness and reference continuity before execution. This preserves the intended target across the interface to PX4. The clear-acknowledgment wait can take up to three seconds, longer than the two-second observation limit. Completing that preparation first lets the uploader check a newly captured observation immediately before its initial <a href="https://github.com/pome223/missionos/blob/main/scripts/smoke_px4_gazebo_sitl_mission_upload.py#L294-L328">MISSION_COUNT</a> message.</p><p>The waiting-time verifier caught another useful case. An earlier native-model chat trial accumulated 31.6 seconds of judge-added waiting against a 30-second allowance. The corrected verifier flags that historical run, even though it had completed delivery and return under the earlier checks. The <a href="https://github.com/pome223/missionos/blob/main/docs/agents/yokohama-chat-delivery.md">chat evidence report</a> retains the result.</p><p>That is why I keep model outputs, upload acknowledgments, observed movement, and parcel receipts separate. Each answers a different question, and a verifier can revisit the evidence when a check improves.</p><h3>What came together</h3><p>For the run shown here, all four applicable verifiers passed: decisions, pad queue, payload, and SITL flight. SITL means software-in-the-loop: flight-control software connected to the simulator.</p><p>All 13 holds met the existing three-second settling period, 30-second measurement window, and 0.6-metre altitude-error bound. During the climb after delivery, the maximum sampled vertical error was approximately 5.33 centimetres.</p><p>The <a href="https://github.com/pome223/missionos/blob/main/docs/examples/yokohama-map-delivery/REPORT.md">result report</a> and <a href="https://github.com/pome223/missionos/blob/main/docs/examples/yokohama-map-delivery/source-provenance.json">source correspondence record</a> connect that execution to a specific public commit.</p><p>The useful accomplishment is the whole chain: an operator chooses a destination; a particular plan is approved; the aircraft follows it; the parcel arrives; the receiver accepts it; the aircraft comes home. Each transition has a condition that the system can inspect.</p><p>That is the MissionOS capability I wanted to build here: taking a requested outcome through execution to a completion claim that another program can check.</p><h3>Run details and city credit</h3><p>CPU fixtures are deterministic test stand-ins for the city-model responses. The video’s run used those fixtures and the single live Jev call described above. Historical native AeroVLA/ANWM experiments used external GPU inference, rather than onboard inference. The recorded Jev timing is one invocation. The public demo uses fixtures and disables live Jev before credential loading or HTTP; its separate browser-only kinematic smoke test did not invoke Jev. The optional learned pad-camera advisory was off in this run.</p><p>The timelapse holds saved images between samples; continuous screen capture and map-GUI footage are outside this recording. Physical flight, moving ships, fleet coordination, native-model reliability, and certification need their own validation. The scene geometry is a source dataset, not a survey of current conditions.</p><p>The city scene and video use Yokohama City / Project PLATEAU’s 3D City Model of Yokohama, catalog fiscal year 2024, under <a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>. MissionOS cropped the neighborhood, transformed coordinates and geometry, and added materials, conservative collision proxies, a demo route, and a pad. Those adaptations are MissionOS work, rather than municipal or MLIT-authored plans. See the <a href="https://www.geospatial.jp/ckan/dataset/plateau-14100-yokohama-shi-2024">official dataset</a>, <a href="https://github.com/pome223/missionos/blob/main/docs/examples/yokohama-urban-scene/ATTRIBUTION.md">full attribution</a>, and <a href="https://github.com/pome223/missionos">MissionOS repository</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb4ef894847f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predict Before Acting: A Video World Model Controls a Block-Stacking Robot]]></title>
            <link>https://blog.gopenai.com/predict-before-acting-a-video-world-model-controls-a-block-stacking-robot-724bfcac363f?source=rss-9af0df47ddde------2</link>
            <guid isPermaLink="false">https://medium.com/p/724bfcac363f</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[physical-ai]]></category>
            <category><![CDATA[robotics]]></category>
            <dc:creator><![CDATA[Astropomeai]]></dc:creator>
            <pubDate>Mon, 21 Sep 2026 10:37:34 GMT</pubDate>
            <atom:updated>2026-09-22T12:34:57.160Z</atom:updated>
            <content:encoded><![CDATA[<h4>Before placing the next block, MissionOS predicts what could happen and decides whether to keep stacking or stop.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tSCVNO6vOB7tU2n9v9HJEQ.png" /></figure><p>Before a robot places its next block, an AI generates a video of what might happen. The system uses that prediction to decide whether to add one more block or stop and keep its score.</p><p>Then it observes what actually happened and repeats.</p><p>We built this loop in MissionOS and tested it across 40 previously unused simulated games. The robot predicted, decided, acted, and observed the result — one block at a time.</p><p>The experiment let us test two questions separately: could generated videos guide the robot’s decisions, and how many points would those decisions preserve? The main achievement is the first: the system predicted what might happen before the robot moved, then used that prediction to decide whether to stack another block or stop. The scores show how useful those decisions were — and where they still failed.</p><h3>The missing piece was information available before action</h3><p>In our earlier experiment, we tested predictions using recordings of movements the robot had already performed. This time, the model predicts before the robot moves, and that prediction helps decide what the robot does next.</p><p>The robot’s exact movements are not known in advance. SmolVLA, the model controlling placement, produces motor commands as the robot moves and receives new observations. Our prediction therefore has to work from the current scene and the planned placement.</p><p>For this experiment, we replaced the saved action tape with information available at that moment: the current camera image, current object and robot state, physical properties, the next placement target, and the identity of a registered placement procedure.</p><p>The neural ACWM then generated 37 frames covering one 14.2-second placement. A fixed visual readout converted those frames into a danger score. Only after the decision to proceed did the system obtain and execute the actual SmolVLA actions.</p><p>The order was:</p><p><strong>Observe → generate a future → decide → execute → measure → repeat.</strong></p><p>This is a real video-generation experiment. ACWM uses a neural VideoDiT and a frozen image/video encoder-decoder. ExtraTrees is a separate tree-ensemble baseline. A small logistic readout interprets ACWM’s generated images; it does not replace the video model with a state-based tree classifier.</p><p>Both learned systems receive exact current simulator information, including physical properties. This is not a demonstration of inferring hidden friction or mass from pixels alone.</p><h3>We first checked whether the new input still produced useful futures</h3><p>Changing the conditioning without retraining produced poor forecasts. We fine-tuned ACWM on 36 existing placement examples and evaluated the fixed endpoint on 16 known validation placements.</p><p>The schedule completed 720 attempts: 719 successful parameter updates and one skipped overflow event. We did not add another update after inspecting validation or choose the best checkpoint from a series of validation scores.</p><p>Under the primary generation setting, mean block-mask overlap with the measured future improved from <strong>0.020 to 0.751</strong>. The fixed reader detected all four collapsing validation placements and incorrectly rejected one of twelve stable placements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*VZurwwURHIfaQDIs.png" /></figure><p><em>These are known validation examples used to check feasibility, not results from the new 40-game test. The 16 placements come from six seeds; a second noise setting regenerates those same cases.</em></p><p>With the feasibility gate passed, we froze the model, visual readout, threshold and generation settings before running the new cohort.</p><h3>A game that makes stopping matter</h3><p>Each game allows up to ten blocks. Stop with a stable tower and the retained count is the score. Collapse means zero. Every ending receives the same additional <strong>14.2-second stability check</strong>.</p><p>We used 40 fresh seeds, evenly divided between wide and narrow blocks. Blocks vary slightly in size, mass, friction and center of mass. Placement targets aim upward, with small initial variation; there is no deliberately large sideways shift at every layer.</p><p>Five systems independently played each game:</p><ul><li>VLA continuation, which keeps going until ten blocks or collapse.</li><li>A current-state rule, which watches existing tilt and horizontal drift.</li><li>A width rule, which stops at eight for wide blocks and six for narrow blocks.</li><li>The frozen ExtraTrees predictor, which evaluates continue/bank risks.</li><li>Online ACWM, which generates the next placement and stops if its fixed visual readout reaches the threshold.</li></ul><p>All five use the same frozen SmolVLA and placement procedure. The simulator stages the grasp; learned VLA commands control the placement/release phases. This is not an end-to-end learned pick-and-place demonstration.</p><p>The complete experiment contains 200 actual games. We checked matching state and executed-action hashes along shared prefixes, rather than relying only on identical seed numbers. Before comparing scores, several differences matter: ACWM predicts the next 14.2-second placement, while ExtraTrees looks 28.4 seconds ahead and evaluates both continuing and stopping. ExtraTrees also uses features designed for this stacking task and stored block positions recorded just before release in earlier placements, used to measure subsequent movement. Both receive current physical properties, including mass and friction. These scores compare complete stopping systems with different capabilities.</p><h3>What happened</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wBt-OCC0Vyne9oI4.png" /></figure><p><em>Total scores out of 400: VLA continuation 40; current-state rule 192; width rule 280; ExtraTrees 291; online ACWM 201. Each system independently executes the same 40 initial conditions.</em></p><p>ACWM gained <strong>161 points over continuation</strong>. The paired average improvement was 4.025 points per game, with a 95% bootstrap interval from 3.075 to 5.025.</p><p>Its nine-point lead over the current-state rule is much less decisive. The interval for the average difference spans zero. We cannot conclude that ACWM is stronger than that rule from this cohort.</p><p>The width rule and ExtraTrees both scored substantially higher. The width rule is a strong, task-specific baseline: it captures much of the structure of these towers with almost no computation. Its eight-block and six-block limits work well for this block distribution; a different distribution or placement target would require fresh validation. The learned models would also need to be tested under those changed conditions.</p><p>One useful ACWM decision illustrates what video forecasting can contribute. On seed 68000, the model rejected the proposed tenth block and the robot banked nine stable blocks. Continuing from the same saved state actually collapsed the tower. The width rule and ExtraTrees each retained eight on that game.</p><p>That successful intervention is part of the result. So are the errors that outweighed it across the cohort.</p><h3>Predicting danger is not the same as predicting a safe stop</h3><p>ACWM chose to bank in 32 games. Three games completed all ten blocks safely. Eight games collapsed: five after continuing and <strong>three after banking</strong>.</p><p>The last category matters. The current ACWM predicts only the next placement. It does not generate a separate future for “stop now and wait.” A tower that is already unstable can still fall after the robot decides to stop adding blocks.</p><p>The forecast horizons also differ. ACWM predicts <strong>14.2 seconds</strong>, covering one placement. Final ExtraTrees predicts placement plus a further hold, covering <strong>28.4 seconds</strong>.</p><p>Those differences are plausible contributors to the score gap. The experiment does not isolate how much comes from horizon, action coverage, engineered features, model architecture or the visual readout.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LoIP9XBWz_vFKGNw.png" /></figure><p><em>Decision-level results describe the states ACWM actually reached. They are not 309 independent games.</em></p><p>Within its 14.2-second window, ACWM’s reader produced <strong>8 true positives, 24 false positives, 272 true negatives and 5 false negatives</strong>. It sometimes stopped too early and sometimes missed a real collapse. Fourteen narrow-block games ended with just four blocks, while the width rule retained six.</p><p>This is why simply lowering a danger threshold is not an obvious solution. A model can be overly cautious in many states and still miss the few failures that erase a tall tower’s score.</p><h3>Fast enough for this simulator loop</h3><p>All <strong>309 ACWM generations</strong> returned valid futures. There were no technical failures across the 200 games.</p><p>On one NVIDIA L4, image encoding, generation and readout took a median <strong>1.65 seconds</strong>, with a 95th percentile of <strong>1.72 seconds</strong>. The complete decision boundary averaged about <strong>2.84 seconds</strong>, including service overhead.</p><p>That is practical for this per-placement simulator experiment. Simulation pauses during remote inference, however. A physical tower would keep moving while the system thinks; a real deployment must account for those state changes before acting.</p><p>The estimated experiment cost, including Phase 1 and conservative ancillary allowances, was about <strong>$3.54</strong>, within the $10 budget. The GPU VM and disk were deleted afterward. This is an estimate, not a final invoice.</p><h3>What MissionOS contributes</h3><p>MissionOS carries the forecast through a common Prediction contract. The record binds it to the model, policy, environment, observation and prediction horizon. It remains model-inferred evidence.</p><p>The lab policy then makes the place/bank decision, and the execution record documents what SmolVLA and the simulator actually did. A prediction receipt is distinct from permission to execute and from proof of a successful outcome.</p><p>This experiment connects Prediction Core to real VLA simulator execution under an authorized lab scope. It does not repeat the earlier DeepSeek, approval and execution-ticket chain. Keeping those experiments distinct lets each result say something concrete.</p><p>Across all five systems, the records contain <strong>19,497 actual-game SmolVLA inference chunks</strong> and <strong>483,652 motor steps</strong>. Of those steps, 107,964 apply returned VLA commands; the remainder belong to the registered approach, grasp-staging, withdrawal and hold controllers. Public verification checks the individual records, score calculations, ordering and shared prefixes.</p><h3>What we learned</h3><p>A video world model can generate a placement future using execution-time information, influence a stopping decision, and then hand control to a real VLA in a sequential simulator game. That loop now works without a saved future action tape.</p><p>The next capability to investigate is equally concrete: <strong>predict whether stopping will actually preserve the tower</strong>, as well as whether another placement is dangerous. A longer generated future and an explicit bank branch could address that gap; they would need their own frozen evaluation.</p><p>For now, the 40-game experiment is complete. It establishes online operation, measures useful interventions, and preserves the failures that explain why the strongest simpler systems still win.</p><p><a href="https://github.com/pome223/missionos/blob/main/docs/agents/acwm-online-stacking-20260921.md">Read the full technical report, figures and methods</a>, or <a href="https://github.com/pome223/missionos/tree/main/docs/assets/acwm-online-stacking-20260921">inspect the individual evidence and verification tools</a>.</p><p>Model foundations: <a href="https://github.com/xavihart/ACWM-Phys-dev">ACWM-Phys</a> and <a href="https://arxiv.org/abs/2506.01844">SmolVLA</a>. Implementation and experiment: <a href="https://github.com/pome223/missionos/pull/110">MissionOS PR #110</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=724bfcac363f" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/predict-before-acting-a-video-world-model-controls-a-block-stacking-robot-724bfcac363f">Predict Before Acting: A Video World Model Controls a Block-Stacking Robot</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From ExtraTrees Predictions to Robot Actions: Stacking Blocks with MissionOS]]></title>
            <link>https://blog.gopenai.com/from-wam-predictions-to-robot-actions-stacking-blocks-with-missionos-1a1b731972bb?source=rss-9af0df47ddde------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a1b731972bb</guid>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[physical-ai]]></category>
            <dc:creator><![CDATA[Astropomeai]]></dc:creator>
            <pubDate>Sat, 19 Sep 2026 12:32:43 GMT</pubDate>
            <atom:updated>2026-09-22T12:34:40.331Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Task-specific decision-tree ensembles, a 40-game evaluation, and an end-to-end control loop with DeepSeek, bounded human approval, and measured outcomes.</em></p><p><strong>Correction — September 20, 2026:</strong> The original title and wording called the predictor a “WAM,” which could imply a pretrained world/action model. The final evaluated model was a task-specific ExtraTrees classification and regression ensemble, fitted on exact simulator state, known physical properties, and engineered features. We have corrected that framing. Scores and execution records are unchanged. The WAM labels retained in the original figures refer to these experimental predictors, not to pretrained-model lineage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*Z8I3buX9GvvJ_ESdPI_Bag.png" /></figure><p>A robot stacking blocks needs to do more than place the next block. It also needs to decide when to stop.</p><p>One more block earns another point. A collapse erases everything. That makes a simple stacking game a useful test of a bigger idea: can a prediction about the next physical state help a robot make a better decision — and can we connect that prediction to execution through a clear, auditable chain?</p><p>We built and tested that chain in MissionOS. A task-specific ExtraTrees ensemble predicts the consequences of a proposed operation. DeepSeek receives the prediction as evidence. Rules check the operation against a human-preauthorized scope. An Executor runs the simulator, and a Verifier measures what actually happened.</p><p><strong>The project produced two concrete results: a learned stopping strategy that scored 279 points across 40 games, and a working MissionOS execution path connecting prediction, judgment, authorization checks, action, and verification.</strong></p><h3>The game: ten blocks, or zero points if the tower falls</h3><p>The rules are straightforward:</p><ul><li>Stack up to ten blocks.</li><li>Stop early to bank the number of stable blocks.</li><li>If the tower collapses, score zero.</li></ul><p>In the final evaluation, every strategy waited 14.2 seconds after finishing, including after completing all ten blocks. This gave every terminal score the same stability check.</p><p>The robot aimed to stack vertically. Instead of forcing large sideways offsets at every level, we varied block properties such as mass, friction, and center of mass. The evaluation included wide and narrow block conditions. Competing strategies shared the same initial state, physical properties, and VLA action sequence up to their stopping decisions.</p><h3>The model we actually trained</h3><p><strong>The final predictor was an ExtraTrees classifier/regressor ensemble trained on simulator-generated examples. It was not a pretrained World Action Model that we fine-tuned.</strong> ExtraTrees combines multiple randomized decision trees. SmolVLA, the action-generating model, was fine-tuned separately.</p><p>The model received simulator state, including block positions and motion, together with known mass, friction, and center-of-mass information. Its operations were the registered placement and stopping procedures of a frozen VLA/controller setup.</p><p>We used ExtraTrees classification and regression models. These produced separate outputs for collapse risk and future block positions; the stopping decision primarily used the classification output. The features also included engineered quantities describing support and the center of mass of supported blocks.</p><p>Earlier trials with off-the-shelf video models did not produce the required block-fall predictions in our setup. We therefore trained this smaller model for the specific mission. Those earlier results remain in the technical report.</p><h3>Forty games: 279 points with ExtraTrees</h3><p>After training and configuration, we froze the model and evaluated it on 40 unused starting conditions. This final cohort used the uniform terminal stability check. It evaluated ExtraTrees-based stopping before the DeepSeek integration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*lcmSRoGAyJxQ7Djb.png" /></figure><p><em>Figure 1. Three separate 40-game cohorts. The final result discussed here is V7, on the right. Conditions and scoring changed between cohorts, so this is a comparison of experiments rather than a learning curve.</em></p><p>The ExtraTrees-based strategy scored <strong>279 points</strong>. A rule based on the tower’s current tilt and horizontal displacement scored <strong>198</strong>. Continuing without a stopping rule scored <strong>30</strong>.</p><p>A particularly effective baseline scored <strong>280</strong>: stop at eight blocks in the wide condition and six in the narrow condition. This width-based rule was selected using separate validation data.</p><p>ExtraTrees-based stopping outperformed the current-state rule and unconditional continuation, with statistically supported differences in this cohort. Its observed total was nearly the same as the width-based rule. Superiority over that stronger baseline remains unestablished.</p><h3>Fifteen small wins — and two expensive mistakes</h3><p>The totals, 279 and 280, hide an important difference in behavior.</p><p>ExtraTrees gained one additional point over width-based stopping in 15 games: a total gain of 15 points. But it collapsed in two games where the width rule would have preserved eight points. Those two mistakes cost 16 points. The other 23 games were ties.</p><p><strong>Fifteen gains of one point were offset by two losses of eight points.</strong> The reward for taking one more step and the cost of losing a tower are sharply asymmetric. This experiment made that trade-off visible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*QL5EUCNYY4s3d1qV.png" /></figure><p><em>Figure 2. Left: score differences relative to width-based stopping. Right: attempted stopping heights, including games that collapsed during stopping and ultimately scored zero.</em></p><p>There were missed opportunities too: ExtraTrees stopped before some safe tenth-block placements. All 20 narrow-block games stopped at six. Recording both the stopping decision and the stability after stopping helped us see what the model was actually doing.</p><h3>Connecting the prediction to MissionOS</h3><p>A prediction such as “the next placement is risky” leaves several practical questions. Who uses it? What authorizes the operation? Which constraints still apply? And who checks the result?</p><p>MissionOS gives those responsibilities distinct places in the execution path:</p><p><strong>VLA proposes motor actions. ExtraTrees predicts. LLM Assurance judges. Humans pre-authorize bounded execution. Rules constrain. Executor acts. Verifier checks.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*kzp4A0HDz3jt0-ex.png" /></figure><p><em>Figure 3. The MissionOS execution path. Human approval in this experiment was bounded preauthorization of cases, operations, and execution limits, rather than a fresh human click for each operation.</em></p><p>A forecast is bound to its observation, model, policy, environment, operation, and prediction horizon. Assurance receives it as model-inferred evidence, distinct from the observed state.</p><p>Before execution, Rules check observation freshness, state changes, approval scope, and remaining allowances. An eligible operation receives an execution ticket. The Executor then drives the simulator, and the Verifier measures the actual block state and terminal score.</p><p><strong>When a prediction is wrong, this separation lets us trace the evidence, the judgment, the authorization, the action, and the measured outcome.</strong></p><p>We first replayed six known cases through the common Prediction contract. The integration reproduced the original scores — 8, 0, 0, 9, 9, 9 — including the failures. We then connected the path to actual DeepSeek API decisions. We also verified that stale observations, state-revision mismatches, and approval mismatches were rejected before motion.</p><h3>Testing DeepSeek in the loop</h3><p>The initial DeepSeek run banked five and eight points across two cases, for a total of 13. Saved prompts confirmed that the ExtraTrees predictions reached DeepSeek, and its explanations referred to them. The remaining issue was how the risk scores and their relationship to the game’s reward were explained.</p><p>We clarified that the risk values were uncalibrated model outputs and supplied a proxy score comparison for continuing versus stopping. Re-running the same two known cases produced eight points each, for a total of <strong>16</strong>. That run recorded 18 actual LLM judgments, 208 SmolVLA inference chunks, and 5,112 motor-control steps.</p><p>This was a revision after inspecting known cases. The ExtraTrees predictor with its deterministic stopping rule scored 17 on the same pair. The demonstrated result here is the working integration: <strong>an actual LLM judgment based on ExtraTrees evidence passed through authorization checks, execution, and measured verification.</strong></p><p>A deterministic rule can perform the proxy arithmetic too. In this benchmark, the LLM integration tests the control path and division of responsibility. How an LLM handles mission context or exceptions beyond that arithmetic is a separate question that the same path can support.</p><h3>Learn for each mission, use a common interface</h3><p>This game had a strong, simple width-based stopping rule. When a compact rule works well for a task, it deserves to be a serious baseline — and may be the right practical choice.</p><p>Other missions can involve many more combinations of object state, physical properties, and operations. A similarly effective rule may be harder to find. That motivates another approach: <strong>train a lightweight prediction model for the mission, then use its predictions through a common system interface.</strong></p><p>In MissionOS, the adapter owns mission-specific state representations, prediction models, and outcome meanings. The Core manages the relationship between observations and forecasts, while Assurance receives forecasts through the shared contract. Even the meaning of “collapse” stays in the stacking adapter.</p><p>The commonality demonstrated here is this integration structure. Another mission would supply its own data, model, and adapter within the contract. Keeping model training focused on a mission also offers a practical development direction when compute resources are limited.</p><h3>What we built</h3><p>We ran this study in a simulator with exact state and known physical properties, scoring stability over a specified hold period. Within those conditions, ExtraTrees-based stopping scored 279 points across 40 games, compared with 198 for the current-state rule and 280 for width-based stopping. The report retains the full comparison and its failure cases.</p><p>We then connected that ExtraTrees predictor to MissionOS: from the common Prediction contract to DeepSeek judgment, bounded human preauthorization, Rules checks, ticketed execution, and measured verification.</p><p><strong>Predict what an operation may do. Use that prediction in a judgment. Execute within authorized bounds. Measure what happened.</strong> Demonstrating that complete path through a visible block-stacking mission is the central MissionOS result.</p><h3>Technical report and execution records</h3><p>The public report includes the training data composition, experimental conditions, per-game scores, statistical comparisons, failure analysis, and execution evidence. Figures in this article come from that report.</p><ul><li><a href="https://github.com/pome223/missionos/blob/main/docs/agents/missionos-wam-system-technical-report-20260919.md">MissionOS technical report</a></li><li><a href="https://github.com/pome223/missionos/blob/main/docs/agents/block-stacking-wam-physics-followup-20260918.md">Physical-condition experiments and per-game results</a></li><li><a href="https://github.com/pome223/missionos/blob/main/docs/agents/stacking-mission-e2e.md">MissionOS end-to-end execution records, including DeepSeek</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a1b731972bb" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/from-wam-predictions-to-robot-actions-stacking-blocks-with-missionos-1a1b731972bb">From ExtraTrees Predictions to Robot Actions: Stacking Blocks with MissionOS</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Who Decides When the Job Is “Done”? — MissionOS, a Mission-Control Layer for Physical AI]]></title>
            <link>https://medium.com/@astropomeai/who-decides-when-the-job-is-done-missionos-a-mission-control-layer-for-physical-ai-c84e4b6b7739?source=rss-9af0df47ddde------2</link>
            <guid isPermaLink="false">https://medium.com/p/c84e4b6b7739</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[physical-ai]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[missiono]]></category>
            <dc:creator><![CDATA[Astropomeai]]></dc:creator>
            <pubDate>Sun, 09 Aug 2026 09:50:39 GMT</pubDate>
            <atom:updated>2026-08-09T09:50:39.893Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A July development log on Mission Contracts, durable human approval, idempotent dispatch, and evidence-backed Recovery and Repair across PX4, Nav2, and NVIDIA GR00T.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yAIWBbJaGmeVih3M8wcD9A.png" /></figure><h3>Overview</h3><p>Over the past month, I have been building MissionOS together with Codex, my AI development partner.</p><p>MissionOS does not put an AI directly behind the controls of a robot. It is a mission-control layer in which AI proposes, a human approves, a separate execution system acts, and the result is judged from evidence.</p><p>At the center of this month’s work is the <strong>Mission Contract</strong>: a machine-readable contract that fixes the success conditions before execution begins. Using this contract, we managed three very different environments — PX4 for a drone, Nav2 for a mobile robot, and NVIDIA GR00T for a robot arm — as stages of one parent mission.</p><p>On the AI side, we also migrated to an ADK v2 workflow built around a Chief Agent, specialist agents, and a Safety Critic. We added a durable path that can pause while waiting for human approval and resume after a restart. We also implemented a Dispatch Ledger that prevents the same command from being sent twice.</p><p>This work clarified how MissionOS connects to NVIDIA’s Physical AI stack. NVIDIA provides robot intelligence, simulation, and execution infrastructure. MissionOS manages what was requested, who approved it, what was actually executed, and how much success the evidence can support. When something goes wrong, MissionOS can propose Recovery and continue after human approval. If the mission still cannot be completed, it preserves the failure evidence and uses Repair to prepare the next execution.</p><p>Previous article: <a href="https://note.com/astropomeai/n/ne7b2018413f4">Give AI the Control Tower, Not the Controls — The Design of MissionOS</a> *(Japanese)*</p><p>Below are the major changes from the past month.</p><h3>1. Fixing the success conditions before execution with a Mission Contract</h3><p>A Mission Contract defines what “success” means before a robot starts moving.</p><p>It is not a PDF contract. It is machine-readable data that software can compare with the evidence collected after execution.</p><ul><li>What will be executed?</li><li>What will be observed?</li><li>Which conditions must be satisfied for success?</li><li>When must execution stop?</li><li>Which evidence will support the final judgment?</li></ul><p>Even for a simple request such as “deliver the package,” the following are separate facts: a command was sent, the vehicle received it, the vehicle moved, it reached the destination, it dropped the package, and the recipient received it.</p><pre>Request<br>  ↓<br>Freeze the Mission Contract<br>  ↓<br>Human approval<br>  ↓<br>Execution and observation<br>  ↓<br>Judge against the predeclared conditions</pre><p>An AI may write a convincing report, but the mission is not successful unless the evidence required by the Contract exists.</p><p>Do not change the passing criteria after the test is over. We brought that principle into the robot execution layer.</p><h3>2. Managing PX4, Nav2, and GR00T under one parent Contract</h3><p>We extended the Mission Contract from a single task to a parent mission containing multiple stages.</p><ul><li>Stage 1 — PX4 / Gazebo: fly a drone</li><li>Stage 2 — TurtleBot3 / Nav2: travel on the ground</li><li>Stage 3 — GR00T N1.7 / LIBERO Panda: manipulate an object with a robot arm</li></ul><p>The parent Contract fixes the stage order, the execution system used by each stage, the success conditions, and the scope covered by human approval.</p><pre>Parent Mission Contract<br>  │<br>  ├─ PX4 Contract<br>  ├─ Nav2 Contract<br>  └─ GR00T Contract</pre><p>Success in one stage does not automatically create authority to execute the next. MissionOS also checks whether that next action was included in the original human approval.</p><p>In our validation, all three stages ran under the same parent mission ID, and we recorded that each simulator’s conditions had been satisfied.</p><p>There is an important limitation: the three simulators did not share one physical world. The package carried by the drone was not the same object received by TurtleBot3 and then manipulated by the Panda arm.</p><p>The three individual stages satisfied their conditions. End-to-end delivery as a whole remains unverified.</p><p>What we gained was not a story claiming that “the delivery succeeded.” We gained a foundation for governing different execution systems through one Contract and approval model.</p><h3>3. Running GR00T N1.7 and LIBERO on an NVIDIA L4</h3><p>The third environment uses NVIDIA’s GR00T N1.7 vision-language-action model and the LIBERO Panda simulator.</p><p>A vision-language-action model, or VLA, generates actions from images, language, and robot state.</p><p>We ran one bounded LIBERO task with GR00T N1.7 on an NVIDIA L4 GPU. MissionOS connected the following information into one evidence trail:</p><ul><li>The pre-approved task</li><li>The model and environment used</li><li>The actions returned by GR00T</li><li>The inputs sent to the simulator</li><li>The state returned by the simulator</li><li>The success condition defined by the official environment</li></ul><p>The run used one task from an approved catalog. It did not turn an unrestricted natural-language instruction into an arbitrary unknown action.</p><p>This is also where two different recovery loops become important:</p><ul><li><strong>Runtime Recovery:</strong> During an active mission, observations produce a bounded recovery proposal. After human approval, the mission continues under the same Contract.</li><li><strong>Repair:</strong> A failed or unverifiable run is preserved. The next execution begins under a revised Mission Contract and a new human approval.</li></ul><p>MissionOS already handles both patterns with PX4 and TurtleBot3.</p><p>In a PX4 / Gazebo end-to-end run, MissionOS detected insufficient clearance from a building on the flight path. The Recovery Agent proposed an avoidance waypoint. After human approval, the mission resumed under the same Contract and continued through package drop-off, return-to-launch, landing, and disarm. That is Runtime Recovery.</p><p>In two earlier attempts, the required clearance could not be proven, so the aircraft safely aborted and landed. MissionOS did not rewrite those failures as successes. It preserved them, revised the route under a new Contract, and required new human approval before the next run. That is Repair.</p><p>The current LIBERO runner, however, has no callback for injecting a stop or recovery operation during an episode. As a result, the GR00T path currently supports post-episode Repair, not in-episode Runtime Recovery. After a failed episode, MissionOS can propose one retry of the same task as a new episode. That retry is not executed automatically: it requires a new Contract and new human approval.</p><p>The next step is to connect a GR00T / Robot Adapter that accepts intermediate observations and recovery operations, bringing the same Runtime Recovery pattern used by PX4 and TurtleBot3 to GR00T.</p><h3>4. Connecting three environments through one “propose → approve → execute → observe → judge” loop</h3><p>PX4, Nav2, and GR00T use different command formats, sensors, and success conditions.</p><ul><li>PX4 uses MAVLink and flight telemetry.</li><li>Nav2 uses ROS 2 goals, poses, and navigation status.</li><li>GR00T generates actions from images, language, and robot state.</li></ul><p>MissionOS does not try to erase those differences.</p><p>Environment-specific behavior belongs in Adapters. The logic for interpreting success conditions belongs in environment-specific Predicate Packages. MissionOS places a common mission-control loop above them.</p><pre>AI proposes<br>      ↓<br>Human approves<br>      ↓<br>Adapter executes<br>      ↓<br>Environment change is observed<br>      ↓<br>Predicates judge the result</pre><p>We do not keep adding branches to MissionOS Core such as “use this rule for PX4” or “use another rule for NVIDIA.”</p><p>The robot may change, but the control pattern remains the same: proposal, approval, execution authority, evidence, and judgment.</p><h3>5. Standardizing the Chief Agent, specialist agents, and Safety Critic with ADK v2</h3><p>We also migrated the AI side to a Google ADK v2 workflow.</p><pre>Operator<br>   ↓<br>Chief Agent<br>   ↓<br>Specialist Agent<br>   ↓<br>Safety Critic<br>   ↓<br>Proposal</pre><p>The Chief Agent interprets the request and selects the appropriate specialist. The specialist proposes a route or recovery action. The Safety Critic checks for danger, contradictions, and attempts to exceed authority.</p><p>Completing this workflow does not create authority to execute.</p><p>The agents have agreed only on a Proposal. Human approval, dispatch, execution, observation, and the success judgment remain separate MissionOS boundaries.</p><p>We retained the previous sequential path as a rollback option, allowing us to compare the old and new workflows and switch back if necessary.</p><h3>6. Pausing and resuming across human approval</h3><p>A Physical AI workflow must be able to stop safely while it waits for a human decision.</p><p>MissionOS can now pause a workflow at the approval boundary and persist its progress in Redis. Resumption requires more than a “yes” in chat.</p><ul><li>Which proposal was approved?</li><li>Has the proposal hash changed?</li><li>Has the approval expired?</li><li>Which action and dispatch does the approval cover?</li><li>Are the execution conditions still the same?</li></ul><p>MissionOS reloads these facts and proceeds only when they match the exact approval_ref.</p><pre>Workflow pauses<br>      ↓<br>Human approves in MissionOS<br>      ↓<br>A different process resumes the workflow<br>      ↓<br>Approval and current state are revalidated</pre><p>If an old approval remained valid after a restart, a robot could act in a changed environment under past authority. In a chat service, stale state might cause a display error. In the physical world, stale approval can cause a stale command to be executed.</p><p>So the important implementation is not only that a workflow can resume. It is that the workflow cannot resume under an invalid old approval.</p><p>Redis stores workflow progress. The existence of a Redis record does not create approval or execution authority.</p><h3>7. A Dispatch Ledger that prevents the same command from being sent twice</h3><p>Correct approval is not enough if the same command can be sent twice.</p><p>Takeoff, movement, and arm manipulation are not like retrying a read-only web request. One retry can become a second physical action.</p><p>MissionOS therefore assigns one dispatch_ref to each external send and records it in a Dispatch Ledger.</p><pre>claim<br>  ↓<br>record send-start<br>  ↓<br>send externally<br>  ↓<br>store receipt</pre><p>If the same dispatch_ref arrives again with identical content, MissionOS returns the stored receipt without resending the command. If the same ID arrives with different content, it is rejected.</p><p>If send-start was recorded but no receipt exists, MissionOS stops with unknown_send_outcome. It does not assume that the command was never sent, and it does not retry automatically.</p><p>In validation, we raced eight Python processes against the same dispatch_ref. Only one reached the simulated sender.</p><p>What we demonstrated was eight processes on a single host. The ledger uses Redis compare-and-set operations to claim sending authority atomically, so it is designed to work across multiple hosts as well. We have not yet demonstrated that multi-host case. This is a limit of the evidence, not a stated limit of the design.</p><h3>8. Proving that a stop request and an observed stop are different facts with Husky A300</h3><p>We ran a bounded Nav2 driving test using Clearpath’s official Husky A300 simulator environment.</p><p>Nav2 accepted a goal, returned 133 feedback messages, and produced approximately 0.255 meters of position change.</p><p>During the stop test, we observed that a stop request had been sent. But we did not obtain an independent acknowledgment from the controller, and the robot moved approximately another 0.492 meters within the measurement window. We could not verify the stopping effect.</p><pre>Stop request sent       = observed<br>Controller ACK          = unverified<br>Observed stopping effect = unverified<br>Movement after request  = approximately 0.492 m</pre><p>This does not mean that an emergency stop failed on physical hardware. We did not run physical hardware.</p><p>The accurate claim is narrower: through this simulator path, we could not prove that the stop request produced an actual stop.</p><p>The test left us with a concrete measurement showing why a request, an acknowledgment, and an observed effect must be treated as separate facts.</p><h3>9. Connecting a contract and recovery layer to NVIDIA’s intelligence and execution stack</h3><p>NVIDIA provides a broad range of Physical AI infrastructure:</p><ul><li>Simulation through Isaac Sim and Isaac Lab</li><li>World generation, reasoning, and action simulation through Cosmos</li><li>Perception, reasoning, and action generation through GR00T</li><li>Real-time inference and control on physical robots through Jetson Thor</li></ul><p><a href="https://developer.nvidia.com/isaac/gr00t">NVIDIA Isaac GR00T</a> provides a development stack spanning data, foundation models, simulation, middleware, runtime libraries, and Jetson Thor. <a href="https://www.nvidia.com/en-us/ai/cosmos/">Cosmos</a> is moving beyond world generation toward reasoning about situations and simulating actions and outcomes. <a href="https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-thor/">Jetson Thor</a> provides the computing platform for running large AI models on the robot.</p><p>MissionOS uses these systems through Adapters and places a Mission Contract above them.</p><pre>NVIDIA / Robot backend<br>  Intelligence, learning, simulation, execution<br>                         ▲<br>                         │ Adapter<br>                         ▼<br>MissionOS<br>  Contracts, approval, authority, evidence,<br>  judgment, recovery</pre><p>Even with powerful AI, several questions remain separate: Who requested the work? What exactly did a human approve? What actually happened? And how much success can the evidence support?</p><p>NVIDIA builds the robot’s brain, training ground, and execution computer. MissionOS manages the promises the robot accepted and the evidence it actually left behind.</p><p>When something goes wrong, MissionOS uses observed facts to produce a bounded recovery proposal. After human approval, the mission can continue under the same Contract. If it still cannot be completed, MissionOS preserves the failure, proposes a revised Contract, and asks for new approval before the next execution.</p><p>The current GR00T validation ran in simulation on an NVIDIA L4, not on a physical robot using Jetson Thor. All published execution evidence records physical_execution_invoked=false.</p><h3>Conclusion</h3><p>Over the past month, MissionOS brought three different execution systems under Mission Contracts.</p><ul><li>PX4, Nav2, and GR00T managed under one parent Contract</li><li>A standardized Chief Agent, specialist agents, and Safety Critic using ADK v2</li><li>Durable pause and resume across human approval</li><li>Rejection of stale approval on resume</li><li>Prevention of duplicate sends through the Dispatch Ledger</li><li>Separation of a stop request from its observed effect in the Husky A300 test</li></ul><p>MissionOS already has Runtime Recovery and Repair patterns for PX4 and TurtleBot3. Now that GR00T is connected to the same Contract model, the next step is to extend those recovery loops to NVIDIA’s Physical AI infrastructure.</p><p>A Mission Contract is not a way to make different robots obey the same command. It is a way to govern different execution systems through the same control pattern: what was requested, what was allowed, what happened, and how much completion the evidence supports.</p><p>MissionOS is not designed to manage successful executions alone.</p><p>When something goes wrong, who approves Recovery? If the mission cannot be completed, who allows the next Contract while preserving the failure?</p><p>As AI becomes more capable, these questions become more important.</p><p>After a failure, who authorizes the next execution?</p><p>MissionOS is building the mission-control layer for that decision.</p><p>Repository: <a href="https://github.com/pome223/missionos">pome223/missionos</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c84e4b6b7739" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MissionOS (AI-agent-based mission control) Under Strong Wind: A Recovery Experiment]]></title>
            <link>https://blog.gopenai.com/ack-is-not-success-a-missionos-recovery-experiment-in-strong-wind-d4aaadfec896?source=rss-9af0df47ddde------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4aaadfec896</guid>
            <category><![CDATA[physical-ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[missiono]]></category>
            <dc:creator><![CDATA[Astropomeai]]></dc:creator>
            <pubDate>Fri, 19 Jun 2026 15:32:02 GMT</pubDate>
            <atom:updated>2026-06-23T07:41:36.614Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Tokyo Station → Akihabara, PX4/Gazebo SITL, 1kg payload, 9m/s wind</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kv1La6xDt9zuIl7eGt3e-Q.png" /></figure><p>This is not a story about a successful delivery.</p><p>In the previous MissionOS experiment, I wrote about taking a natural language delivery request and completing a simulated delivery mission on PX4/Gazebo SITL. MissionOS generated the mission, executed it in the simulator, released the payload, and verified the result.</p><p>This time, I wanted to test something less glamorous, but more important.</p><p>What happens when the mission does not go well?</p><p>I ran a simulated delivery mission from Tokyo Station to Akihabara Station with the following conditions:</p><ul><li>Start: Tokyo Station</li><li>Destination: Akihabara Station</li><li>Payload: 1kg</li><li>Wind: 9m/s</li><li>Runtime: PX4/Gazebo SITL</li></ul><p>SITL means Software-In-The-Loop. Instead of flying a real drone, PX4 runs as the flight controller software, while Gazebo simulates the world around it.</p><p>The result was simple:</p><p>The drone did not reach the destination. The payload was not delivered. The mission failed.</p><p>But for MissionOS, this was still meaningful progress.</p><p>The important part was not that the AI completed the mission. It did not. The important part was that MissionOS handled the failure without pretending it was a success.</p><p>The Recovery Agent proposed Return To Launch. A human operator approved it. The command was accepted by the active runner. But MissionOS did not observe actual return progress.</p><p>So MissionOS did not call it a successful return.</p><p>That distinction is the core of this experiment.</p><h3>What MissionOS is trying to do</h3><p>MissionOS is not a system where an AI directly flies a drone.</p><p>The closer metaphor is mission control.</p><p>PX4 handles low-level flight control: attitude, motors, navigation modes, and vehicle behavior. MissionOS sits above that layer. It creates missions, checks whether execution is allowed, monitors runtime state, proposes recovery actions, requires human approval, dispatches approved commands, and verifies what actually happened.</p><p>The roles are intentionally separated:</p><ul><li>AI evaluates the situation and proposes actions.</li><li>The human operator approves or rejects those actions.</li><li>Policy and safety rules constrain what is allowed.</li><li>The executor only sends approved commands.</li><li>The verifier records what was actually observed.</li></ul><p>That separation matters.</p><p>An AI saying “Return To Launch is recommended” is not the same thing as an AI sending an RTL command. A command being accepted is not the same thing as the vehicle actually returning. A map showing a route is not the same thing as proof of delivery.</p><p>MissionOS is designed to avoid mixing those facts.</p><h3>From natural language to SITL execution</h3><p>The mission started from a natural language request:</p><pre>Run a 1kg payload mission from Tokyo Station to Akihabara Station under 9m/s wind.</pre><p>MissionOS did not immediately execute it.</p><p>The flow was deliberately staged:</p><pre>missionos say<br>missionos approve<br>missionos run<br>missionos start-sitl<br>missionos execute-sitl --live-flight<br>missionos rtl<br>missionos map</pre><p>At first glance, this may look overly careful.</p><p>But the separation is the safety model.</p><p>“Proposed,” “approved,” “prepared,” “started,” “executed,” “recovery requested,” and “verified” are different states. MissionOS keeps them separate so that the system does not accidentally turn a suggestion into an action, or an accepted command into a success claim.</p><p><strong>CLI flow</strong> — The terminal UI is in Japanese, but the important flow is: natural language request → mission proposal → human approval → SITL startup → live execution. MissionOS intentionally separates proposal, approval, execution, and verification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ucpiQaZszKQ88nXfO8w-bg.png" /><figcaption>The staged missionos chat flow (Japanese UI). A natural-language request becomes a bounded proposal (wind=9.0m/s, payload=1.0kg), then moves through approve → prepare → start SITL → live execution as separate states. The live lines show battery falling 98.8% → 66.1% under 9m/s wind, ending in proposal_guardrail_passed:return_to_launch with risk=battery_projected_insufficient_for_route, route_deviation_above_limit.</figcaption></figure><h3>What happened during the mission</h3><p>Once live SITL execution started, the mission quickly entered a difficult state.</p><p>Progress toward the route did not grow as expected. Terrain clearance was also insufficient. In this context, terrain clearance means the margin between the vehicle and the ground or terrain model. The target clearance was around 30m, but the runtime state showed that this margin was not consistently healthy.</p><p>The wind condition made the mission even harder.</p><p>At this point, the Recovery Agent became relevant.</p><p>The Recovery Agent watches runtime telemetry: position, altitude, battery, route progress, terrain clearance, route deviation, and other state signals. It does not directly control the drone. It evaluates the situation and proposes what the operator should do next.</p><p>In this run, the agent proposed Return To Launch.</p><h3>What the Recovery Agent actually judged</h3><p>It is worth being precise about what “proposed RTL” means here, because this is where the intelligence lives.</p><p>The Recovery Agent did not match a threshold to a fixed action. It was given the runtime picture as evidence — route progress that was not growing, terrain clearance sitting under its ~30m target, sustained 9m/s wind, and the resulting route deviation — and asked to judge the situation and choose a bounded response. Its read was that route completion was no longer safely reachable within the mission envelope, but that the vehicle was still in a recoverable state rather than an emergency one.</p><p>From there it selected among real alternatives, not a single canned output. Continuing was off the table — both the agent’s read and a guardrail rule treat active runtime risk as a no-continue condition. An immediate land was held in reserve for the case where drift made a return unsafe. Return To Launch was chosen as the bounded response that best fit a recoverable-but-failing mission, and it was emitted as a proposal with the expected outcome attached, not as a command.</p><p>Only then did the boundaries close around that judgment. A deterministic guardrail checked that RTL was even admissible — for example, that projected battery was sufficient to return home — before the proposal was allowed to stand. A human operator still had to approve it. The AI owned the judgment; it did not own the authority.</p><p>This is the distinction I want to keep visible alongside “ACK is not success.” The honesty about the outcome only matters because the decision in front of it was a real decision. RTL here was the agent’s reasoned choice under wind, bounded by rules and gated by a human — not a lookup.</p><p>That proposal was still only a proposal — execution waited on the operator.</p><p><strong>Recovery Agent console</strong> — The terminal UI is in Japanese. The important part is that the Recovery Agent detects that the mission is no longer progressing safely and recommends missionos rtl. The AI proposes Return To Launch, but the operator must explicitly approve it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hPCYp5KkX3_eoya4JdrPLw.png" /><figcaption>The Runtime Recovery Agent’s operator console — the judgment, in the agent’s own words. It reads that the route can no longer be completed (battery would need ~5.7× what remains), detects battery shortfall, large route deviation, and intermittent telemetry, and notes the vehicle can still return safely (~27% left on arrival at the 120 m home point). It proposes RTL and marks continuing as not recommended: proposal_guardrail_passed; dispatch_authority=False. A proposal, not a command — approval happens in a separate pane.</figcaption></figure><h3>AI proposes, the human approves</h3><p>The operator sent RTL with human approval:</p><pre>missionos rtl --yes</pre><p>MissionOS then dispatched the request to the active runner.</p><p>The runtime record showed that the command was accepted:</p><pre>dispatch_status=queued_for_active_runner<br>recovery_action=return_to_launch<br>runner_ack=accepted<br>tracking=MAV_CMD_NAV_RETURN_TO_LAUNCH<br>outcome=return_progress_not_observed</pre><p>This is the most important part of the experiment.</p><p>runner_ack=accepted means the runner accepted the command. It does not mean the drone returned. It does not mean the drone landed. It does not mean the recovery succeeded.</p><p>The actual outcome was:</p><pre>outcome=return_progress_not_observed</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ZUuVrmlViufY3VFQSfl3w.png" /><figcaption>After the operator approved RTL, the runner accepted it — runner_ack=accepted, tracking=MAV_CMD_NAV_RETURN_TO_LAUNCH — but the recorded outcome=return_progress_not_observed (home_delta=6m, landed=False). MissionOS keeps delivery/progress/physical claim=false. Accepted is not returned.</figcaption></figure><p>In other words, MissionOS observed that the RTL command was accepted, but it did not observe actual return progress.</p><p>So MissionOS did not claim return success.</p><p>This is the line I care about most:</p><pre>ACK is not success.</pre><p>An acknowledgment only means “the command was received” or “the command was accepted by the relevant layer.” It is not proof that the physical or simulated world changed in the expected way.</p><p>For physical AI systems, this distinction is critical.</p><h3>The boundary between command and reality</h3><p>There are many boundaries that are easy to collapse in demos:</p><pre>Command requested<br>Command approved<br>Command dispatched<br>Command accepted<br>Vehicle state changed<br>Vehicle returned<br>Vehicle landed<br>Mission completed</pre><p>A weak system compresses all of these into one story:</p><p>The AI handled it.</p><p>MissionOS does not do that.</p><p>In this experiment, the system only confirmed part of the chain:</p><pre>RTL requested<br>RTL approved<br>RTL dispatched<br>Runner accepted it<br>Return progress was not observed</pre><p>That is why the result remained a failure.</p><p>And that is exactly the point.</p><p>A system that touches the physical world must be able to say not only what happened, but also what did not happen, and what cannot yet be claimed.</p><p><strong>Runtime Recovery panel</strong> — The Japanese UI shows the runtime recovery state. The key English-readable lines are runner_ack=accepted, tracking=MAV_CMD_NAV_RETURN_TO_LAUNCH, and outcome=return_progress_not_observed. MissionOS recorded that the RTL command was accepted, but did not claim that the vehicle actually returned.</p><h3>The map is not proof either</h3><p>MissionOS also generated a map view with missionos map.</p><p>The map overlays the intended route and observed state on real-world geography using OpenStreetMap. It shows Tokyo Station, the Akihabara direction, the home point, the drop-off target, and the current position.</p><p>This is useful.</p><p>But it is not proof of delivery.</p><p>In this run, the map had only limited telemetry points. It was useful as an evidence display, but not as a verifier for mission completion.</p><p>A line on a map is not a delivery. A plotted route is not a successful flight. A current marker is not proof of return.</p><p>This is another distinction MissionOS tries to preserve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ptEvy4-x8RxInbU5ZxftqQ.png" /><figcaption><em>The MissionOS 2D map overlays route and observed position on OpenStreetMap — home (H), dropoff (D), and the current marker. Its own header says it plainly: </em>read-only evidence display, not a verifier, dispatch control, or delivery claim.<em> With a single telemetry sample, it shows where the vehicle was, not that the mission succeeded.</em></figcaption></figure><p><strong>Map view</strong> — The map view helps visualize the mission state on real-world geography. However, it is an evidence display, not a delivery verifier. A route line on a map does not prove that the mission was completed.</p><h3>Why I also looked at LLM invocation cost</h3><p>Another part of this experiment was less visible, but important: the way the Recovery Agent calls the LLM.</p><p>During live flight, telemetry changes continuously. Position, altitude, battery, route progress, and clearance can update many times during a mission. Sending every telemetry update to an LLM would be expensive, slow, and operationally unrealistic.</p><p>So MissionOS now treats runtime signals differently.</p><p>Some events are hard triggers:</p><ul><li>terrain clearance below a threshold</li><li>battery risk approaching a dangerous state</li><li>severe telemetry degradation</li><li>obvious execution failure</li></ul><p>Other events are soft triggers:</p><ul><li>route deviation growing</li><li>progress stalling</li><li>wind remaining high</li><li>weak but not yet fatal warning signs</li></ul><p>The idea is simple:</p><ul><li>If a hard trigger appears, ask the AI sooner.</li><li>If only soft triggers appear, wait longer.</li><li>If the mission is healthy, do not call the AI.</li></ul><p>This avoids calling the LLM on every telemetry frame. And it is worth being clear about what these thresholds do: they decide <em>when</em> to ask the AI, not <em>what</em> it decides. The action is still the agent’s judgment, not the trigger’s.</p><p>But the important rule remains the same:</p><p>A trigger is not a command.</p><p>A trigger is a fact. The AI can evaluate that fact and propose an action, but the action still has to pass through authority and safety boundaries.</p><h3>Local LLMs are role-dependent</h3><p>I also tested a local LLM backend through Ollama.</p><p>The default MissionOS backend remains Gemini, but for local experiments I tried Gemma 4 26B MoE.</p><p>The result was mixed.</p><p>For planning, Gemma 4 26B was usable. It could parse the natural language request and help produce route and mission parameters.</p><p>But for the live Recovery Agent, it was not suitable in this test.</p><p>There were two main problems:</p><pre>One response took about 133 seconds.<br>Structured JSON output was not stable enough.</pre><p>That matters because the Recovery Agent needs timely and structured decisions.</p><p>This does not mean “local LLMs do not work.”</p><p>It means the question has to be role-specific.</p><p>A model that is good enough for planning may not be good enough for runtime recovery. A model that can write a useful paragraph may still fail if the system requires strict JSON, low latency, and guardrail-compatible output.</p><p>For this recovery run, Gemini was the practical choice.</p><h3>Was this experiment a failure?</h3><p>As a delivery mission, yes.</p><p>The drone did not reach Akihabara. The payload was not released. Delivery completion remained false.</p><p>This was not a successful delivery.</p><p>But as a MissionOS experiment, it was valuable.</p><p>MissionOS created a mission from natural language. It recorded human approval as a separate boundary. It prepared and started PX4/Gazebo SITL. It entered live execution. It monitored runtime state. The Recovery Agent proposed RTL. The AI did not execute the command by itself. A human approved the recovery action. The runner accepted the command. And finally, MissionOS recorded that return progress was not observed.</p><p>That final part matters most.</p><p>The system did not lie.</p><p>It did not say:</p><p>RTL accepted, therefore return succeeded.</p><p>It said:</p><p>RTL accepted, but return progress was not observed.</p><p>That is the kind of failure record I want.</p><h3>Toward physical systems</h3><p>MissionOS is still running in simulation here.</p><p>This was not a real-world drone flight. It was PX4/Gazebo SITL.</p><p>But the closer a system gets to the physical world, the more careful its language must become.</p><p>“Dispatched” and “executed” are different. “ACK accepted” and “vehicle moved” are different. “Disarmed” and “landed” are different. “Map displayed” and “delivery verified” are different. “SITL success” and “real-world success” are different.</p><p>The easy path is to connect an AI to a command interface and call it autonomy.</p><p>The harder path is to preserve the boundaries between AI proposal, human authority, policy constraints, execution, observation, and verification.</p><p>MissionOS is trying to build the second path.</p><p>Not “AI did everything.” Not “the demo looked successful.”</p><p>But:</p><pre>What was requested?<br>What was approved?<br>What was dispatched?<br>What was accepted?<br>What was observed?<br>What still cannot be claimed?</pre><p>This strong-wind mission did not deliver the payload.</p><p>But it did test whether MissionOS could fail honestly.</p><p>And in physical AI systems, honest failure handling is not a side feature.</p><p>It is part of the foundation.</p><h3>Closing</h3><p>The Tokyo Station to Akihabara mission failed.</p><p>Under 9m/s wind, route progress was poor, terrain clearance was insufficient, and the Recovery Agent proposed Return To Launch. The operator approved RTL. The command was accepted by the runner. But MissionOS did not observe actual return progress.</p><p>So it did not claim success.</p><p>That is the point of this experiment.</p><p>MissionOS is an operating layer for keeping AI proposal, human approval, execution, and verification separate.</p><p>The goal is not to make the AI look powerful.</p><p>The goal is to make the system honest about what happened.</p><p>ACK is not success.</p><p>And for systems moving toward the physical world, that distinction matters.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4aaadfec896" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/ack-is-not-success-a-missionos-recovery-experiment-in-strong-wind-d4aaadfec896">MissionOS (AI-agent-based mission control) Under Strong Wind: A Recovery Experiment</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[boiled-claw as Mission OS, Part 2 : From a Natural Language Delivery Request to a Verified…]]></title>
            <link>https://blog.gopenai.com/boiled-claw-as-mission-os-part-2-from-a-natural-language-delivery-request-to-a-verified-e27ee8a8e13b?source=rss-9af0df47ddde------2</link>
            <guid isPermaLink="false">https://medium.com/p/e27ee8a8e13b</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[drones]]></category>
            <category><![CDATA[physical-ai]]></category>
            <category><![CDATA[missiono]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Astropomeai]]></dc:creator>
            <pubDate>Mon, 04 May 2026 15:33:12 GMT</pubDate>
            <atom:updated>2026-05-05T15:28:43.125Z</atom:updated>
            <content:encoded><![CDATA[<h3>boiled-claw as Mission OS, Part 2 : From a Natural Language Delivery Request to a Verified PX4/Gazebo SITL Mission</h3><p>In the previous article, I introduced <strong>boiled-claw</strong> not as just another AI agent implementation, but as a kind of <strong>Mission OS</strong>.</p><p>The point was not “letting an AI click around in a browser.”</p><p>The deeper problem was this:</p><p>How can an AI agent safely interact with an external environment that has state, side effects, uncertainty, and failure modes?</p><p>Even a browser is not a stateless tool.</p><p>It has open tabs, login state, partially filled forms, active focus, hidden transitions, visible UI state, and user-facing control panels. It is not enough for an agent to say, “I did it.”</p><p>Did it actually reach the intended page?<br>Did it write into the intended cell?<br>Did the external state actually change?<br>Was evidence captured?<br>Can the result be replayed, reviewed, and verified?</p><p>That is the problem boiled-claw was built to explore.</p><p>At its core, boiled-claw is organized around concepts such as:</p><pre>Mission Contract<br>Task Graph<br>Verifier<br>Recovery<br>Replay<br>Gate<br>Approval<br>Control UI</pre><p>Instead of letting an AI agent freely manipulate external systems, boiled-claw turns goals into contracts, observes state, records evidence, checks gates, stops when necessary, and keeps humans in the loop at important boundaries.</p><p>That was the starting point of boiled-claw as a Mission OS.</p><p>In this second article, I want to take that same idea outside the browser.</p><p>The target is a delivery mission inside <strong>PX4 + Gazebo SITL</strong>.</p><p>But first, an important clarification:</p><p>This is not a real-world drone delivery.<br>No physical drone was flown.<br>Mission OS did not replace PX4 or the flight controller.<br>An LLM did not directly pilot a drone.</p><p>What happened is more specific:</p><blockquote><em>A natural language delivery request was converted into a bounded scenario, executed inside PX4 + Gazebo SITL, and verified through evidence, gates, and observed flight facts.</em></blockquote><p>That is still a simulation.</p><p>But from the perspective of Mission OS, it is a major step forward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nc6jnvgJL4-_as_Ue7nkzw.png" /></figure><h3>An AI Agent Does Not Fly the Drone</h3><p>The main point of this work is not that an AI agent directly controls a drone.</p><p>In fact, the point is almost the opposite.</p><p>Low-level control should remain with the systems designed for it.</p><p>For a drone, that means things like attitude control, trajectory following, failsafe behavior, landing control, and motor control belong to PX4, the autopilot, and the flight controller.</p><p>boiled-claw sits above that layer.</p><pre>Low-level control:<br>  PX4 / autopilot<br>Physical simulation:<br>  Gazebo<br>Mission-level control:<br>  boiled-claw / Mission OS</pre><p>Mission OS does not directly pilot the drone.</p><p>It defines the mission, constrains execution, observes the external environment, records evidence, verifies outcomes, and decides when to stop or escalate.</p><p>In other words, it is not replacing the flight controller.</p><p>It is managing the mission.</p><p>This distinction matters.</p><p>The question is not:</p><blockquote><em>Can an LLM fly a drone?</em></blockquote><p>The question is:</p><blockquote><em>Can a Mission OS safely coordinate, constrain, observe, and verify an autonomous mission executed by specialized external systems?</em></blockquote><p>That is the layer boiled-claw is trying to explore.</p><h3>From Natural Language to a Bounded Scenario</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wXUVsZmSZ5ItCclNecHG3g.png" /></figure><p>The input in this experiment was a natural language delivery request:</p><pre>Deliver a 5 kg package to a mountain hut at an altitude of 3,000 meters.</pre><p>For a human, this is a short instruction.</p><p>For a drone mission, it contains many implicit constraints.</p><p>High altitude.<br>Mountain terrain.<br>A 5 kg payload.<br>Reduced air density.<br>Energy budget concerns.<br>Payload margin.<br>Route risk.<br>Drop-off verification.<br>Return planning.</p><p>The Mission Designer converts that request into a scenario proposal.</p><p>For example, a request like this becomes something conceptually closer to:</p><pre>Mission kind:<br>  mountain summit payload delivery<br>Route strategy:<br>  staged ascent required<br>Constraints:<br>  altitude target<br>  payload weight<br>  mountain route<br>  high elevation<br>  energy budget risk<br>  payload margin risk</pre><p>The important part is that this proposal is not execution authority.</p><p>A natural language prompt does not grant dispatch authority.</p><p>An LLM output does not become a command.</p><p>A scenario proposal does not automatically invoke Gazebo execution.</p><p>Physical execution is not involved.</p><p>The prompt is treated as evidence for operator review, not as authority.</p><p>Mission OS converts the request into a bounded scenario, then passes it through validation, review, approval, and gates before anything runs.</p><h3>From Scenario Proposal to Bounded Simulation Request</h3><p>A scenario proposal does not go directly into Gazebo.</p><p>First, it is validated.</p><p>If the scenario is blocked, it cannot be approved.</p><p>If it is accepted for review, it can be compiled into a bounded simulation request.</p><p>The word “bounded” is important.</p><p>This is not unbounded execution.</p><p>It is not approval-free dispatch.</p><p>It is not targeting real hardware.</p><p>It is not physical execution.</p><p>It is a constrained request that defines what may be simulated, under what limits, and with what evidence requirements.</p><p>Only after that boundary is established does the system move into PX4 + Gazebo SITL.</p><h3>Running the Mission in PX4 + Gazebo SITL</h3><p>The mission was executed inside a single SITL session.</p><p>The observed facts included:</p><pre>same SITL session<br>mission upload observed<br>mission accepted by the simulator/autopilot stack<br>takeoff observed<br>dropoff region reached<br>payload release observed<br>landing observed</pre><p>This matters because it is not enough to produce a plausible artifact after the fact.</p><p>The system needs evidence that the mission actually progressed through the expected stages inside the same simulation session.</p><p>Mission OS observes that the mission was uploaded, that it was accepted, that takeoff occurred, that the vehicle reached the drop-off region, that payload release occurred, and that landing was observed.</p><p>The most important part is the payload release.</p><h3>Reaching the Destination Is Not the Same as Delivery</h3><p>For a delivery mission, reaching the destination area is not enough.</p><p>A drone may arrive near the mountain hut, but that does not prove the package was delivered.</p><p>Delivery requires more.</p><p>The payload must be released at the right place, at the right time, under conditions that match the mission.</p><p>In this experiment, the payload release was not a synthetic “let’s pretend it happened” marker.</p><p>The release event was observed from the physical simulation side of Gazebo and then verified by Mission OS as evidence.</p><p>The verifier checks the delivery outcome using multiple conditions together.</p><p>At a high level, it verifies that:</p><pre>the release happened inside the drop-off zone<br>the release happened within an acceptable altitude range<br>the release happened within the expected mission timing window</pre><p>In other words, Mission OS does not mark the mission complete just because the vehicle got close to the destination.</p><p>It requires observed release evidence, and that evidence must align with the mission’s spatial and temporal constraints.</p><p>Only then can the delivery be considered completed.</p><h3>Delivery Completed</h3><p>The mission reached a completed state.</p><p>But this completion is not just a status field being changed.</p><p>The chain looked like this:</p><pre>natural language<br>-&gt; Mission Designer<br>-&gt; bounded scenario<br>-&gt; PX4 + Gazebo SITL<br>-&gt; mission upload<br>-&gt; flight<br>-&gt; drop-off zone<br>-&gt; observed payload release in Gazebo<br>-&gt; landing<br>-&gt; evidence verification<br>-&gt; delivery completed</pre><p>This is not a real-world delivery.</p><p>But from the perspective of Mission OS, the delivery mission was closed end-to-end inside simulation.</p><p>The system turned a natural language request into a bounded scenario, executed it in PX4 + Gazebo SITL, observed the outcome, verified the delivery evidence, and recorded the result.</p><p>That is the important part.</p><h3>The Browser Automation Mechanism Still Matters</h3><p>At first, browser automation and drone simulation look completely different.</p><p>But the underlying mechanism is the same.</p><p>In browser automation, the core question was never simply:</p><blockquote><em>Did the agent click?</em></blockquote><p>It was:</p><blockquote><em>Did the external environment reach the intended state?</em></blockquote><p>For example:</p><pre>Is the browser really on the target page?<br>Was the intended cell actually updated?<br>Was the form actually submitted?<br>Was the result visible to the operator?</pre><p>That same pattern now appears in PX4 + Gazebo:</p><pre>Was the mission really uploaded?<br>Did takeoff really occur?<br>Did the vehicle really reach the drop-off region?<br>Did payload release really happen?<br>Did landing really occur?</pre><p>The environment changed.</p><p>The principle did not.</p><p>Mission OS observes the external system, records evidence, verifies the destination-bound outcome, stops when gates fail, and exposes the result to the operator.</p><p>Payload release verification is the physical-simulation version of destination-bound verification.</p><p>Do not trust that the click happened.<br>Verify that the destination state changed.</p><p>Do not trust that the drone “arrived.”<br>Verify that the payload was released at the right place, under the right conditions.</p><p>That is the core idea.</p><h3>What This Is Not</h3><p>This part is important.</p><p>This work does not mean that boiled-claw can operate a real-world delivery drone.</p><p>It does not solve real-world aviation safety.</p><p>It does not solve regulation.</p><p>It does not replace PX4.</p><p>It does not replace an autopilot.</p><p>It does not put an LLM inside the safety gate.</p><p>It does not prove real-world autonomous delivery.</p><p>The roles are different:</p><pre>PX4:<br>  flight controller / autopilot<br>Gazebo:<br>  simulation environment<br>boiled-claw:<br>  Mission OS / control plane / evidence gate</pre><p>That separation matters.</p><p>boiled-claw is not trying to become the low-level controller.</p><p>It is trying to become the mission-level layer above specialized execution systems.</p><h3>What Is Still Missing</h3><p>The delivery mission was completed inside SITL.</p><p>But the system has not yet solved dynamic replanning during flight.</p><p>For example, future work still needs to handle cases such as:</p><pre>battery low<br>landing zone unavailable<br>telemetry stale<br>route deviation<br>payload release failure<br>wind / weather condition<br>vehicle health degradation</pre><p>In those situations, Mission OS should be able to decide whether to continue, hold, abort, recommend return-to-home, propose an alternate landing zone, or escalate to the operator.</p><p>That is the next major step.</p><p>The current milestone is:</p><blockquote><em>A natural language delivery request can be executed and verified as a PX4 + Gazebo SITL delivery mission.</em></blockquote><p>The next milestone is:</p><blockquote><em>Mission OS can react to changing mission conditions and connect recovery decisions to the next action.</em></blockquote><p>That is where the system starts to look less like a one-shot simulation and more like a closed-loop autonomy runtime.</p><h3>Why This Matters</h3><p>It is easy to say that an AI agent is “autonomous.”</p><p>The harder question is:</p><pre>What exactly did it accomplish?<br>What evidence proves it?<br>Why was the result accepted?<br>Where would the system stop if something went wrong?<br>When does a human need to intervene?</pre><p>boiled-claw approaches this problem through Mission OS.</p><p>The PX4 + Gazebo SITL delivery mission is a concrete example.</p><p>The LLM did not fly the drone.</p><p>The AI agent did not replace the physical controller.</p><p>Mission OS converted a natural language goal into a bounded scenario, used PX4 + Gazebo as the external execution environment, observed the mission, verified payload delivery, and recorded the result as evidence.</p><p>This is not just an AI tool.</p><p>It is a step toward a mission-level control plane for autonomous systems.</p><h3>Where This Goes Next</h3><p>The next work should stay inside SITL first.</p><p>Before moving toward real hardware, the simulation environment should become more difficult.</p><p>The system should face scenarios such as:</p><pre>battery low<br>telemetry stale<br>landing zone unavailable<br>route deviation<br>payload release failure<br>weather or wind constraints</pre><p>For each case, Mission OS should produce a recovery decision.</p><p>Should it continue?<br>Should it hold?<br>Should it abort?<br>Should it recommend return-to-home?<br>Should it ask the operator for a decision?</p><p>After that, the system can move toward operator-minimal behavior.</p><p>Safe cases should proceed with little human involvement.</p><p>Boundary-crossing cases should escalate.</p><p>Beyond that, real hardware requires a different level of responsibility: an adopting organization, a hardware owner, a safety operator, emergency stop procedures, regulatory context, and operational risk management.</p><p>That is another mountain.</p><p>But this one matters:</p><p>Inside PX4 + Gazebo SITL, the Mission OS path has now climbed from request to verified delivery.</p><h3>Closing</h3><p>This milestone is not real-world drone delivery.</p><p>But it is a meaningful demonstration of a Mission OS pattern.</p><p>A natural language request became a bounded mission.<br>The mission ran inside PX4 + Gazebo SITL.<br>The vehicle flew.<br>The drop-off region was reached.<br>Payload release was observed.<br>Landing occurred.<br>The result was verified through evidence.</p><p>That is the difference between an agent that claims it completed a task and a system that can show why the task should be considered complete.</p><p>If the first article was about the architecture, this second one is about the proof.</p><p>boiled-claw has moved from concept to simulation-verified mission execution.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e27ee8a8e13b" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/boiled-claw-as-mission-os-part-2-from-a-natural-language-delivery-request-to-a-verified-e27ee8a8e13b">boiled-claw as Mission OS, Part 2 : From a Natural Language Delivery Request to a Verified…</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mission OS: From Browser Use to Multi-Phase Autonomous Missions]]></title>
            <link>https://blog.gopenai.com/mission-os-from-browser-use-to-multi-phase-autonomous-missions-785ac845f799?source=rss-9af0df47ddde------2</link>
            <guid isPermaLink="false">https://medium.com/p/785ac845f799</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[missiono]]></category>
            <category><![CDATA[physical-ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Astropomeai]]></dc:creator>
            <pubDate>Sun, 03 May 2026 15:12:32 GMT</pubDate>
            <atom:updated>2026-05-05T15:28:28.445Z</atom:updated>
            <content:encoded><![CDATA[<p>In a previous article, I wrote about making browser use reliable in boiled-claw.</p><p>At first glance, browser use looks like a simple automation problem.</p><p>Click a button.<br>Open a tab.<br>Fill in a form.<br>Verify the result.</p><p>But the real problem is deeper.</p><p>A browser is not just a UI.<br>It is a stateful external environment.</p><p>It has tabs, sessions, forms, user context, page transitions, permissions, and partial work in progress. If an AI agent interacts with it carelessly, it can destroy context, overwrite user state, or claim success without actually reaching the intended destination.</p><p>So the challenge is not “how do we make an AI click?”</p><p>The challenge is:</p><pre>How do we let an AI agent interact with a stateful external environment<br>without losing control of the mission?</pre><p>That question led boiled-claw beyond browser use.</p><p>It is gradually becoming something closer to a <strong>Mission OS</strong>.</p><pre>Note:<br>The related public repository is available here:<br><br>https://github.com/pome223/boiled-claw-public<br><br>The current public documentation still reflects an ongoing research process, so some parts are dense and difficult to read. It is not meant to expose private implementation details or reproduction steps.<br><br>I am reorganizing the public docs around the ideas, results, and safety boundaries of boiled-claw.<br><br>This article is part of that effort: a higher-level explanation of boiled-claw as a Mission OS.</pre><h3>What I mean by Mission OS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9NeP6KGAIZQ7KNVn0E_CUA.png" /></figure><p>A Mission OS is not a robot operating system.<br>It is not a drone autopilot.<br>It is not a replacement for low-level controllers.</p><p>It is a control plane above execution.</p><p>Its job is to manage the mission:</p><pre>What is the mission?<br>Which phase are we in?<br>What evidence is required to move forward?<br>What should happen if the mission is blocked?<br>When is human approval required?<br>What should be recorded for replay?<br>What can be reused in the next mission?</pre><p>In other words, Mission OS is not primarily about action.</p><p>It is about <strong>mission state, evidence, policy, approval, replay, and memory</strong>.</p><p>This distinction matters because modern AI agents are moving from isolated tasks toward long-running interactions with external systems.</p><p>A browser is one such system.<br>A desktop is another.<br>A simulator is another.<br>A robot, drone, or industrial process may eventually be another.</p><p>The underlying problem is similar across all of them.</p><p>An AI agent needs to operate in an environment that has state, consequences, uncertainty, and failure modes.</p><p>That requires more than a prompt.<br>It requires a mission runtime.</p><h3>From single execution to mission phases</h3><p>A single success flag is often too coarse.</p><p>“Task completed” may hide too much.</p><p>For example, a delivery mission is not one action. It is a sequence of phases:</p><pre>preflight<br>route execution<br>pickup<br>delivery route<br>dropoff<br>return or land<br>completion</pre><p>Each phase needs evidence.</p><p>A mission should not advance just because the model says it advanced.<br>It should advance because the runtime has enough evidence.</p><p>That evidence may include:</p><pre>environment state<br>health snapshots<br>phase gates<br>route progress<br>blocked reasons<br>recovery decisions<br>operator approval<br>replay events</pre><p>This is the core shift.</p><p>Instead of asking an AI agent to “do the task,” Mission OS asks:</p><pre>Can this mission phase be considered complete?<br>What evidence supports that?<br>What policy applies if it fails?</pre><p>That structure makes the system inspectable.</p><p>A completed mission is not just a claim.<br>A blocked mission is not just an error.<br>Both become durable mission outcomes.</p><h3>Simulation-first physical missions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A_9sH3qGMyZP9iPqKJC55A.png" /></figure><p>In boiled-claw, this idea has been explored beyond browser use through simulation-first physical missions.</p><p>By physical-adjacent, I mean systems that do not control real hardware, but are close enough to physical execution that evidence, safety boundaries, and failure handling matter.</p><p>The important point is not that a simulated drone can move from one place to another.</p><p>The important point is that a physical-adjacent mission can be represented as a sequence of controlled, reviewable phases.</p><p>At a high level, a recent simulation milestone looks like this:</p><pre>mission contract<br>  -&gt; phase state machine<br>  -&gt; health snapshots<br>  -&gt; phase gates<br>  -&gt; bounded simulation execution<br>  -&gt; replay timeline<br>  -&gt; curated corpus<br>  -&gt; mission result</pre><p>This is deliberately conservative.</p><p>The boundary remains simulation-only.</p><pre>no hardware target<br>no physical execution<br>no mission upload<br>no unbounded execution stream<br>no approval-free stronger action</pre><p>That boundary is not a limitation of the idea.<br>It is part of the design.</p><p>Before an AI system should be trusted with stronger execution, it needs to show that it can handle mission structure, evidence, failure, approval, and replay.</p><p>Mission OS is the layer where that structure lives.</p><h3>Failure is part of the mission</h3><p>A mission runtime should not only model the happy path.</p><p>In real systems, the important question is often:</p><pre>What happens when the mission does not go as planned?</pre><p>A route may deviate.<br>Telemetry may be stale.<br>A gate may not pass.<br>A recovery action may be required.<br>A human may need to review the next step.</p><p>In a Mission OS, those conditions are not afterthoughts.</p><p>They become first-class mission states.</p><p>A phase gate may produce:</p><pre>pass<br>blocked<br>abort</pre><p>A recovery policy can map:</p><pre>phase x failure type -&gt; decision</pre><p>The system can record which policy cell was used, whether stronger action was deferred, and whether operator approval is required.</p><p>This matters because it prevents the runtime from silently escalating.</p><p>A policy decision is evidence.<br>It is not automatically authority.</p><p>That idea appears again in fleet memory.</p><h3>Replay and corpus</h3><p>If a mission cannot be replayed, it is hard to trust.</p><p>Mission OS needs durable evidence.</p><p>Not just logs.<br>Not just screenshots.<br>Not just a final answer.</p><p>It needs a structured timeline of what happened.</p><pre>phase entered<br>health observed<br>gate evaluated<br>transition recorded<br>policy selected<br>mission completed or blocked</pre><p>A replay timeline makes the mission inspectable.</p><p>A curated corpus makes it regression-testable.</p><p>The goal is not to preserve every raw detail.<br>The goal is to preserve the mission-relevant evidence in a form that can be reviewed and compared.</p><p>This is how a runtime becomes more than an automation script.</p><p>It becomes an operating layer for missions.</p><h3>Fleet memory: evidence, not authority</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N_juafVo9Ms9j_PkVxdRzw.png" /></figure><p>The next abstraction is memory across missions.</p><p>If one mission encounters a risky route segment, an unstable dropoff area, or a repeated failure pattern, that information should be useful for future missions.</p><p>But it must be used carefully.</p><p>The rule is:</p><pre>Fleet memory is evidence, not authority.</pre><p>Past missions may inform:</p><pre>planning<br>gate strictness<br>risk scoring<br>review priority</pre><p>But memory must not directly grant:</p><pre>command authority<br>approval bypass<br>stronger execution<br>hardware access</pre><p>This is an important distinction.</p><p>A previous mission can say:</p><pre>This route has shown higher risk.<br>This zone may need stricter validation.<br>This failure pattern should trigger review.</pre><p>But it should not say:</p><pre>Execute this stronger action now.<br>Bypass approval.<br>Control the next agent directly.</pre><p>That would turn memory into authority, which is unsafe.</p><p>So the flow becomes:</p><pre>mission evidence<br>  -&gt; trajectory summary<br>  -&gt; route or zone memory<br>  -&gt; feedback candidate<br>  -&gt; review and promotion<br>  -&gt; planning input for future missions</pre><p>Memory becomes part of the planning context, not part of the command path.</p><h3>Lead and follower missions</h3><p>This also changes how we think about multiple agents.</p><p>A “lead” mission can generate evidence.<br>A “follower” mission can use that evidence as prior knowledge.</p><p>But the lead mission does not control the follower.</p><p>The pattern is:</p><pre>lead mission observes<br>Mission OS summarizes<br>memory is reviewed and promoted<br>follower mission plans with that memory</pre><p>This is not swarm control in the strong sense.</p><p>It is mission-to-mission feedback under policy.</p><p>That distinction matters.</p><p>The system becomes more capable over time, but the authority boundary remains explicit.</p><h3>Why this matters</h3><p>AI agents are increasingly capable of interacting with external systems.</p><p>But capability alone is not enough.</p><p>As agents move from chat interfaces into browsers, desktops, simulators, and eventually physical-adjacent environments, the core challenge becomes governance of execution.</p><p>A reliable agent runtime needs to know:</p><pre>what it is trying to do<br>what phase it is in<br>what evidence it has<br>what it is allowed to do<br>what it must not do<br>how to recover<br>how to replay the mission<br>what can be learned safely</pre><p>That is the role of Mission OS.</p><p>It is not about replacing every lower-level system.<br>It is about coordinating them under a mission contract.</p><h3>The current direction of boiled-claw</h3><p>boiled-claw started as an exploration of closed-loop AI agents.</p><p>Browser use made the first lesson clear:</p><pre>External environments are stateful.<br>Reliable automation requires state, verification, and boundaries.</pre><p>The current direction extends that lesson.</p><pre>Browser use<br>  -&gt; controlled external environment<br>  -&gt; mission phases<br>  -&gt; simulation-first physical tasks<br>  -&gt; replay and corpus<br>  -&gt; fleet memory</pre><p>The project is still small and experimental.</p><p>It is not a production robotics system.<br>It is not a real-world drone platform.<br>It does not grant hardware authority.</p><p>But the architecture is becoming clearer.</p><pre>Mission contract<br>Evidence<br>Phase gates<br>Recovery policy<br>Replay<br>Memory<br>Authority boundary</pre><p>This is the shape I expect many AI agent systems will need.</p><p>Not just tools.<br>Not just prompts.<br>Not just workflows.</p><p>A mission layer.</p><p>A system that can say:</p><pre>Here is the mission.<br>Here is the phase.<br>Here is the evidence.<br>Here is the policy.<br>Here is what was allowed.<br>Here is what was blocked.<br>Here is what can be learned.<br>Here is what must not become authority.</pre><p>As AI agents move from answering questions to operating environments, this layer becomes more important.</p><p>The interesting question is no longer only whether an agent can take an action.</p><p>The more important question is whether it can operate under a mission contract, preserve evidence, respect authority boundaries, and explain why it moved forward or stopped.</p><p>That is what I mean by Mission OS.</p><p>That is the direction of boiled-claw.</p><p>A small step from browser use toward Mission OS.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=785ac845f799" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/mission-os-from-browser-use-to-multi-phase-autonomous-missions-785ac845f799">Mission OS: From Browser Use to Multi-Phase Autonomous Missions</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Browser Use Reliable in boiled-claw]]></title>
            <link>https://medium.com/@astropomeai/making-browser-use-reliable-in-boiled-claw-2c97c41cd133?source=rss-9af0df47ddde------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c97c41cd133</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[google-adk]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <dc:creator><![CDATA[Astropomeai]]></dc:creator>
            <pubDate>Tue, 21 Apr 2026 15:26:32 GMT</pubDate>
            <atom:updated>2026-04-21T15:40:24.104Z</atom:updated>
            <content:encoded><![CDATA[<p>Browser use sounds simple until you try to make it reliable.</p><p>Opening a page is easy. Clicking a button is easy. Typing text is easy.<br>What is <em>not</em> easy is building an agent system that can:</p><ul><li>operate the browser a user is already looking at,</li><li>keep the operator-facing UI intact,</li><li>write into a stateful web app like Google Sheets,</li><li>verify that the write actually landed in the right place,</li><li>and treat all of that as a closed-loop task rather than a lucky one-shot success.</li></ul><p>That is the problem I have been working on in boiled-claw.</p><p>This post explains how browser use works in the project, why “current browser” tasks are harder than they look, and what changed to make them more stable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VlBG1GK6jGG_zJGqE8ocqg.png" /><figcaption>browser execution is one part of a larger closed-loop runtime</figcaption></figure><p><a href="https://github.com/pome223/boiled-claw-public">GitHub - pome223/boiled-claw-public: Reference architecture for closed-loop AI agents - plan, execute, verify, repair.</a></p><h3>Browser use is harder than shell use</h3><p>A shell command succeeds or fails in a relatively clean way.</p><p>A browser task is different. Even if an agent “does the right action,” several things can still go wrong:</p><ul><li>it may act on the wrong tab,</li><li>text may land in the toolbar or title field instead of the intended app surface,</li><li>the visible page may not be the page that actually owns the task state,</li><li>verification may pass on weak evidence,</li><li>and the operator’s own browser session may get disrupted.</li></ul><p>So browser use is not just “another tool.”<br>It is the control of a <strong>stateful external environment</strong>.</p><p>That distinction matters a lot.</p><h3>In boiled-claw, browser execution is split across three planes</h3><p>One of the key design decisions in this project is that browser-related actions are not treated as one single surface.</p><p>They are split into three execution planes:</p><pre>browser-related execution<br>├─ browser<br>│   └─ managed browser / generic browser actions<br>├─ current_tab<br>│   └─ the Chrome tab the user is already viewing<br>└─ desktop<br>    └─ screenshot / AX / click / type / hotkey</pre><p>This separation is important because “what the agent wants to do” and “how the system reaches that surface” are not the same thing.</p><p>For example:</p><pre>browser.navigate<br>├─ provider: browser<br>└─ transport: browser automation path</pre><pre>current_tab.navigate<br>├─ provider: current_tab<br>└─ transport: Chrome extension relay</pre><pre>desktop.control.type<br>├─ provider: desktop<br>└─ transport: desktop runtime</pre><p>The system is intentionally explicit about this.<br>A browser-like action may still need to go through a different runtime depending on context.</p><h3>Why current_browser matters more than “open a new browser”</h3><p>A lot of agent demos solve browser automation by opening a fresh browser and running an isolated flow there.</p><p>That is fine for demos, but it breaks down quickly if the operator is already in a browser session and expects the system to work <em>with</em> that context rather than replacing it.</p><p>The goal here is different.</p><p>Instead of:</p><pre>agent<br>└─ launch new browser<br>   └─ do automation somewhere else</pre><p>the target behavior is closer to:</p><pre>agent<br>└─ understand current browser context<br>   └─ preserve operator-facing tab<br>   └─ open or reuse a task-owned tab when needed<br>   └─ verify the destination-bound result</pre><p>That is what I mean by a <strong>current-browser task</strong>.</p><p>The system is not just automating “a browser.”<br>It is trying to operate <em>the browser session the operator is already using</em> without destroying usability.</p><h3>The role of the current-tab relay</h3><p>To make that possible, boiled-claw includes a current-tab relay through a Chrome extension.</p><p>It exposes capabilities such as:</p><ul><li>current_tab.info</li><li>current_tab.navigate</li><li>current_tab.click</li><li>current_tab.fill</li><li>current_tab.extract_text</li><li>current_tab.list_tabs</li><li>current_tab.activate</li></ul><p>This is not just a convenience layer for clicking and filling DOM elements.</p><p>Its more important role is giving the workflow a way to reason about <strong>tab identity</strong>.</p><p>That lets the system:</p><ul><li>understand which tab is currently active,</li><li>keep the Control UI tab separate from the task tab,</li><li>return to the same task-owned tab later,</li><li>and verify that the current page really is the destination page that should contain the result.</li></ul><p>Without this, multi-step browser tasks become much more brittle.</p><h3>Why Google Sheets is a particularly difficult browser target</h3><p>Google Sheets is one of the best examples of why browser use is hard.</p><p>It is not a simple form. It is a stateful app with its own interaction model:</p><ul><li>the visible UI and the meaningful input target are not always the same,</li><li>keyboard shortcuts matter,</li><li>focus matters,</li><li>cell targeting matters,</li><li>and text extraction can easily produce weak or misleading evidence.</li></ul><p>Opening a spreadsheet is not the same thing as successfully writing into it.</p><p>A robust flow has to do more than this:</p><pre>open page<br>→ type text<br>→ assume success</pre><p>It has to do something closer to this:</p><pre>open destination<br>→ identify the real app surface<br>→ target a real cell<br>→ enter structured content<br>→ collect evidence<br>→ verify the result</pre><p>That difference is exactly where many browser-agent demos start to fail.</p><h3>What changed in the current-browser spreadsheet flow</h3><p>The recent improvements in boiled-claw are mostly about making this path harder to break.</p><h3>1. Preserve the Control UI tab</h3><p>When a spreadsheet-oriented current-browser task starts from the Control UI, the runtime preserves that tab as the operator-facing surface.</p><p>If a spreadsheet needs to be opened, the system uses a separate task-owned tab instead of overwriting the UI tab.</p><p>That means the operator does not lose the chat surface while the task is running.</p><h3>2. Normalize navigation through current_tab.navigate</h3><p>If the task is explicitly about the current browser, navigation should stay in that surface whenever possible.</p><p>That is why the runtime now prefers current_tab.navigate rather than relying on a desktop hotkey like Cmd+T or using a generic browser path that could drift away from the operator’s context.</p><h3>3. Target real spreadsheet cells</h3><p>For Google Sheets tasks, the goal is not “type somewhere in the page.”<br>The goal is “write into a real sheet cell.”</p><p>So the flow is biased toward finding and focusing an actual grid target such as A1 before typing.<br>That reduces the chance of text landing in the browser chrome, toolbar, formula bar, or title area.</p><h3>4. Treat sparse evidence as weak evidence</h3><p>A spreadsheet page can expose misleading text through extraction.</p><p>You might see menu labels, UI boilerplate, or only a small fragment of the currently active cell.<br>That is not enough to confidently claim success.</p><p>So the verifier is stricter now. If the available evidence is weak, the system asks for stronger destination-bound evidence instead of silently passing a fragile result.</p><p>That sounds small, but it is one of the biggest differences between “a demo that looked like it worked” and “a system that can explain why it believes it worked.”</p><h3>A concrete example: researching notable shinshinto swordsmiths and summarizing the results in Google Sheets</h3><p>To make this more concrete, here is one real example.</p><p>The operator asks boiled-claw, in Japanese:</p><p>新々刀の名工を調べてGoogle Spreadsheetにまとめて</p><p>The GIF below shows the current-browser Sheets workflow end to end: the runtime preserves the operator-facing Control UI, opens a task-owned Google Sheets tab, enters structured data, and completes the task as a verified control-loop run.</p><p><a href="https://github.com/pome223/boiled-claw-public/blob/main/assets/current-browser-sheets-demo-20260421.gif">Current-browser Google Sheets demo</a></p><p>I suddenly bring up swordsmiths here because I practice <strong>iaido as a hobby</strong>, so this was not a random benchmark prompt.<br>It was a topic I actually cared about, which made it a useful example of what happens when you hand a real browser workflow to the system rather than a synthetic toy task.</p><p>In the Control UI, this request is routed as a <strong>control loop</strong> task rather than a one-shot tool call.</p><p>What the UI shows is important:</p><ul><li>the router recognizes it as a multi-step task,</li><li>the system requires human approval for the plan,</li><li>and the task ultimately completes with Score: 1.00.</li></ul><p>That tells us the system is not merely performing a search.<br>It is treating the request as a structured workflow:</p><ul><li>research the topic,</li><li>collect useful information,</li><li>write the results into Google Sheets,</li><li>and verify that the result is present.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xl0J10ni4CDrarUn" /></figure><p>The resulting spreadsheet shows the structured output of that task.</p><p>Instead of freeform notes, the system creates a table with fields such as:</p><ul><li>swordsmith name,</li><li>famous blade name,</li><li>characteristics / notes.</li></ul><p>The rows include well-known <em>shinshinto</em> swordsmiths and short descriptions of their representative works or historical significance.</p><p>So the end result is not “the AI opened a browser.”<br>The end result is that the AI completed a browser-based research workflow and produced a reusable artifact in an external application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/0*FdEaYAiWWpRBGkjz" /></figure><p>That is the distinction I care about.</p><h3>What these screenshots actually prove</h3><p>These images are useful because they show more than visual polish.</p><p>They show that the system can:</p><ul><li>accept a natural-language request in Japanese,</li><li>route it into a structured task,</li><li>hold approval state in the workflow,</li><li>execute a browser-based research and data-entry process,</li><li>and produce a concrete external result.</li></ul><p>In other words, the browser is not being used here as a flashy interface demo.<br>It is being used as part of an <strong>observable, approval-aware, verifiable execution loop</strong>.</p><p>That is a much more interesting standard.</p><h3>This is not “browser automation” in the usual sense</h3><p>The deeper point is that boiled-claw is not trying to be “an agent that can click around the web.”</p><p>It is trying to build a system that can:</p><ol><li>understand the current browser context,</li><li>preserve the operator-facing surface,</li><li>use separate task-owned state when necessary,</li><li>collect evidence from the right destination,</li><li>and verify the result before calling the task complete.</li></ol><p>That is a different design goal from generic browser automation.</p><p>The interesting challenge is not how to click.<br>The interesting challenge is how to make browser actions part of a system that can be inspected, replayed, approved, and trusted.</p><h3>Closing thought</h3><p>If you want a demo, almost anything can look successful for thirty seconds.</p><p>If you want a usable system, the bar is much higher.</p><p>For browser use, especially with stateful web apps like Google Sheets, the real problem is not action generation.<br>It is <strong>state management, destination identity, evidence quality, and preserving the human-facing workflow while the task runs</strong>.</p><p>That is the direction I am exploring in boiled-claw.</p><p>Not “AI that can open a browser,” but something closer to:</p><p><strong>an execution system that can use a browser without losing control of what it is doing.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c97c41cd133" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Published boiled-claw: A Reference Architecture for Closed-Loop AI Agents]]></title>
            <link>https://medium.com/@astropomeai/i-published-boiled-claw-a-reference-architecture-for-closed-loop-ai-agents-82fef998ae60?source=rss-9af0df47ddde------2</link>
            <guid isPermaLink="false">https://medium.com/p/82fef998ae60</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[google-adk]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Astropomeai]]></dc:creator>
            <pubDate>Sun, 22 Mar 2026 12:46:40 GMT</pubDate>
            <atom:updated>2026-03-22T12:46:40.003Z</atom:updated>
            <content:encoded><![CDATA[<p>In my previous post, I wrote about why I started building boiled-claw.</p><p>At the time, the important shift for me was this:</p><p>I stopped thinking about AI agents as “smart chat UIs,” and started thinking about them as runtimes.</p><p>What I wanted was not just a system that could answer questions well. I wanted something that could route work, use tools, pause risky operations, wait for approval, keep execution observable, and preserve only useful knowledge afterward.</p><p>Since then, that idea has become concrete enough to publish.</p><p>The repository is now public:</p><p><a href="https://github.com/pome223/boiled-claw-public">GitHub - pome223/boiled-claw-public: Reference architecture for closed-loop AI agents - plan, execute, verify, repair.</a></p><p>This is not a polished end-user product. It is a reference implementation. But the architecture is now clear enough that I think it is worth showing in public.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5S9E9vkd01nI2QBYD6p4BQ.png" /></figure><h3>Why I decided to publish it now</h3><p>For a while, I assumed I should wait until everything felt complete.</p><p>But the more I worked on boiled-claw, the more I realized that the most interesting part was not feature completeness. It was the design.</p><p>The core questions were things like:</p><ul><li>Which layer should own routing?</li><li>Which layer should own execution?</li><li>How should approvals interrupt work?</li><li>What should count as memory, and what should not?</li><li>How should host capabilities be separated from the agent runtime itself?</li></ul><p>Those questions already have concrete answers in the codebase.</p><p>So instead of waiting for a “finished” product, I decided to publish the repository as a reference architecture for a certain kind of agent system: one centered on control, verification, and responsibility boundaries.</p><h3>What boiled-claw is trying to be</h3><p>boiled-claw is an AI agent runtime built in Python on top of Google ADK.</p><p>The key idea is simple:</p><p>An agent should not just generate plausible next messages.<br>It should operate through a controllable execution loop.</p><p>In boiled-claw, that loop is:</p><p><strong>Planner → PolicyJudge → Executor → Verifier → Repair</strong></p><p>That sequence matters because once an agent can search, browse, inspect files, run shell commands, or touch a host machine, fluency stops being the main problem.</p><p>The real problem becomes behavioral control.</p><p>Can the system decide how to handle a request?<br>Can it separate allowed from risky actions?<br>Can it verify whether execution actually succeeded?<br>Can it stop, repair, or ask for approval when needed?</p><p>Those are the questions boiled-claw is organized around.</p><h3>The three ideas I wanted to make explicit</h3><h3>1. Closed-loop execution</h3><p>A lot of agent systems still feel like “tool-augmented chat.”</p><p>boiled-claw is trying to move one step further toward operational behavior.</p><p>Instead of treating execution as a single opaque jump from prompt to answer, it treats longer or riskier work as a loop:</p><ul><li>plan what to do</li><li>judge whether the plan is allowed</li><li>execute the approved steps</li><li>verify the outcome</li><li>repair or retry when necessary</li></ul><p>That does not magically make agents safe. But it does make them more inspectable and more governable.</p><p>For me, that is a better foundation than relying on a single large hidden reasoning pass.</p><h3>2. Bridge-based capability separation</h3><p>One of the most important design choices in boiled-claw is that agent reasoning and host capabilities are not collapsed into the same place.</p><p>At a high level, the system is split into three layers:</p><ul><li><strong>Gateway</strong>: the control plane</li><li><strong>Host Bridge</strong>: the execution plane for shell, file, and browser operations</li><li><strong>Desktop Bridge</strong>: the desktop capability plane for GUI-oriented runtime actions</li></ul><p>The Gateway handles protocol, routing, sessions, transcripts, approvals, cron, and event flow.</p><p>The Host Bridge runs host-side shell, file, and browser operations in a separate process.</p><p>The Desktop Bridge is responsible for desktop automation and accessibility-based control.</p><p>This separation matters because it keeps the runtime architecture legible. It also makes it easier to reason about which layer owns which responsibility.</p><p>I was strongly influenced here by control-plane-oriented agent design, but I wanted to shape those ideas into something that fit how I actually want to use an agent day to day.</p><h3>3. Memory as curated state, not raw transcript</h3><p>Another idea that became more important to me while building boiled-claw is that not everything should become memory.</p><p>I think agent systems get noisy very quickly when they treat raw conversation history as long-term knowledge.</p><p>So in boiled-claw, I distinguish between:</p><ul><li>session and state as short-term execution context</li><li>transcript as historical trace</li><li>memory as retained knowledge worth promoting</li></ul><p>The point is not to remember everything.</p><p>The point is to preserve only knowledge that is likely to be useful later.</p><p>That distinction seems small at first, but I think it becomes more important as agent systems get longer-lived and more operational.</p><h3>What the public repository includes</h3><p>The public repository now includes the main pieces of that architecture:</p><ul><li>typed WebSocket / HTTP gateway protocol</li><li>transcript persistence</li><li>tool approvals</li><li>audit logging</li><li>routing and root-agent structure</li><li>closed-loop execution flow</li><li>curated memory lifecycle</li><li>cron platform hooks</li><li>MCP integration</li><li>dynamic agent generation</li><li>host-side browser, shell, and file execution</li><li>current-tab browser relay via Chrome extension</li></ul><p>That last point is especially important to me.</p><p>A lot of “browser agent” discussions stay abstract. But in real usage, what people often want is something much more concrete:</p><p>not just “use a browser,” but “use the browser I am already looking at.”</p><p>That is why boiled-claw includes a current-tab adapter instead of treating browser interaction as only hidden background automation.</p><h3>Why I am calling it a reference architecture</h3><p>I am deliberately not presenting boiled-claw as a hardened security product or a fully productized framework.</p><p>It is a maintainer-led reference implementation.</p><p>That framing matters because I want the repository to be read primarily as a design artifact:</p><p>a concrete example of how to structure a closed-loop agent runtime with explicit bridges, approvals, transcripts, and memory boundaries.</p><p>In other words, I want people to be able to inspect not only what the system can do, but why the responsibilities are divided the way they are.</p><h3>What still feels unfinished</h3><p>There is still a lot I want to improve.</p><p>I want stronger verification behavior.<br>I want better repair loops.<br>I want routing decisions to become more explainable.<br>I want more natural visibility of control flow in the UI.<br>I want the memory lifecycle to become more disciplined over time.</p><p>So this is not a “production-ready launch” post.</p><p>It is a public release of the architecture in its current form.</p><p>That feels like the right level of honesty for the project.</p><h3>Closing</h3><p>In the earlier article, I wrote about why I no longer wanted just a smart chat interface.</p><p>I wanted an agent runtime.</p><p>Publishing boiled-claw is the next step in that direction.</p><p>The system is still unfinished, but the architecture is now concrete enough to inspect in public.</p><p>If you are interested in controllable agent runtimes, bridge-based capability separation, or building closed-loop agent systems on top of Google ADK, the repository may be useful as a reference:</p><p><a href="https://github.com/pome223/boiled-claw-public">https://github.com/pome223/boiled-claw-public</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82fef998ae60" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>