<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by albertobifolco on Medium]]></title>
        <description><![CDATA[Stories by albertobifolco on Medium]]></description>
        <link>https://medium.com/@albertobifolco?source=rss-e0b371042ab2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_dn9BAQS5oik-na1q-R1iw.jpeg</url>
            <title>Stories by albertobifolco on Medium</title>
            <link>https://medium.com/@albertobifolco?source=rss-e0b371042ab2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:26:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@albertobifolco/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deploying OpenClaw on Cloud Run Instances: From Hands-On Lab to Enterprise Architecture]]></title>
            <link>https://medium.com/@albertobifolco/deploying-openclaw-on-cloud-run-instances-hands-on-experience-real-errors-and-ghost-resources-9f5ce2c19ab2?source=rss-e0b371042ab2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f5ce2c19ab2</guid>
            <category><![CDATA[cloud-architecture]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[albertobifolco]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 19:23:25 GMT</pubDate>
            <atom:updated>2026-09-03T19:23:25.419Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A field test of Google Cloud’s stateful serverless runtime: debugging capacity limits, secret IAM, and CLI-only visibility, with an architect’s blueprint for production hardening.</em></p><h3>Introduction: Why This Article?</h3><p>This is my very first technical article on Medium. It is not designed to be an abstract theoretical paper or a polished marketing summary. I wanted to get my hands dirty, run into the real problems, break things, fix them, and document the end-to-end journey with real terminal logs, UI behaviors, and concrete troubleshooting steps.<br>This article takes inspiration from the official Cloud Run Instances codelab released on August 28. But rather than just blindly reproducing it, I wanted to document the actual field experience hitting real-world capacity limits, debugging IAM secret permissions, and working around missing console visibility with hands-on fixes.</p><h3>The Shift in the Paradigm</h3><p>This wasn’t built for web traffic…</p><ul><li>It was built for <strong>AI agents running 24/7</strong>.</li><li>For that <strong>always-on workload</strong> that just needs to exist stably somewhere.</li><li>For background sandboxes that can’t afford cold starts or state resets.</li></ul><p>Cloud Run keeps quietly eating the space between traditional “serverless” and dedicated “Compute Engine VMs”.</p><p>When building AI agent architectures, choosing the right runtime is critical. In Google Cloud, we have always relied on two primary serverless execution patterns: <strong>Services</strong> (stateless, request-driven APIs scaling to zero) and <strong>Jobs</strong> (scheduled batch tasks running to completion).</p><p>With <strong>Cloud Run Instances</strong>, Google Cloud introduces a third paradigm: a dedicated, stateful serverless host designed for continuous sandboxed execution with native Cloud Storage volume mounts and built-in secret injection.</p><p>Below is a quick breakdown of where Cloud Run Instances fit into the modern execution landscape:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZJuqg0W3YTV6ShH8kHZSKQ.png" /></figure><p>In this lab, we test this new runtime in the wild by deploying <strong>OpenClaw</strong> — an extensible framework for sandboxed AI agents — alongside Google Secret Manager, persistent GCS mounts, and the Gemini API.</p><p>Here is the unpolished, step-by-step experience of making it work.</p><h3>Architecture: What is OpenClaw and Why Cloud Run Instances?</h3><ul><li><strong>OpenClaw:</strong> An open-source environment and runtime enabling AI agents to execute terminal tools, sandbox code, and persist states safely.</li><li><strong>Cloud Run Instances (gcloud beta run instances):</strong> Unlike standard request-driven Cloud Run Services or batch Jobs, Cloud Run Instances provide dedicated, long-running execution runtimes supporting direct Cloud Storage (GCS) volume mounts and custom sandbox launchers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J6mgr0PtGmZKVZgT7olvng.png" /></figure><h3>Step 1: Environment Setup &amp; API Activation</h3><p>First, set up your project variables and enable the required Google Cloud APIs:</p><pre>export PROJECT_ID=&quot;codelab-cloudruninstances&quot;<br>export REGION=&quot;europe-west1&quot;<br><br>gcloud config set project $PROJECT_ID<br>gcloud services enable \<br> run.googleapis.com \<br> secretmanager.googleapis.com \<br> storage.googleapis.com \<br> generativelanguage.googleapis.com \<br> compute.googleapis.com</pre><p><strong>💡 Real-World Gotcha #1: Outdated Cloud Shell CLI</strong></p><p><em>Because run instances belongs to the new beta surface, standard Cloud Shell environments might reject flags or miss subcommands. If you hit syntax or validation errors early on, run gcloud components update.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*8Rzn05EsVEVGUlarKWiRHQ.png" /><figcaption><strong><em>Cloud Shell environment variables, project set, and API enablement</em></strong></figcaption></figure><h3>Step 2: Dedicated IAM Identity &amp; Secret Management</h3><p>Instead of relying on the broad Default Compute Service Account, we provision a dedicated Service Account and push secrets directly into Secret Manager:</p><pre>export SERVICE_ACCOUNT_NAME=&quot;openclaw-sa&quot;<br>gcloud iam service-accounts create ${SERVICE_ACCOUNT_NAME} \<br> - display-name=&quot;OpenClaw Service Account&quot;<br>export SERVICE_ACCOUNT=&quot;${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com&quot;<br><br># 1. Store the Gemini API Key<br>echo -n &quot;&lt;YOUR_GEMINI_API_KEY&gt;&quot; | gcloud secrets create gemini-api-key \<br> - data-file=- \<br> - replication-policy=&quot;automatic&quot;<br><br># 2. Store the Gateway Password<br>echo -n &quot;SuperSecretClawPassword123!&quot; | gcloud secrets create openclaw-gateway-password \<br> - data-file=- \<br> - replication-policy=&quot;automatic&quot;<br><br># 3. Grant secretAccessor permissions<br>gcloud secrets add-iam-policy-binding gemini-api-key \<br> - member=&quot;serviceAccount:${SERVICE_ACCOUNT}&quot; \<br> - role=&quot;roles/secretmanager.secretAccessor&quot;<br>gcloud secrets add-iam-policy-binding openclaw-gateway-password \<br> - member=&quot;serviceAccount:${SERVICE_ACCOUNT}&quot; \<br> - role=&quot;roles/secretmanager.secretAccessor&quot;</pre><h3>Step 3: State Persistence via Cloud Storage Volume Mount</h3><p>Cloud Run containers have ephemeral root filesystems. To retain agent state and custom plugin logic across restarts, we mount a GCS bucket onto /home/node/.openclaw:</p><pre>export BUCKET_NAME=&quot;openclaw-state-${PROJECT_ID}&quot;<br>gcloud storage buckets create gs://${BUCKET_NAME} - location=${REGION}<br><br># Storage IAM binding for the instance service account<br>gcloud projects add-iam-policy-binding ${PROJECT_ID} \<br> - member=&quot;serviceAccount:${SERVICE_ACCOUNT}&quot; \<br> - role=&quot;roles/storage.objectAdmin&quot;<br><br># Upload openclaw configuration (link to the files) <br>gcloud storage cp openclaw.json gs://${BUCKET_NAME}/openclaw.json<br># Prepare and upload custom sandbox provider plugin<br>mkdir -p plugins/cloud-run-sandbox-provider<br>gcloud storage cp -r plugins gs://${BUCKET_NAME}/</pre><h3>Step 4: Real-World Deployment Obstacles</h3><p>Here is the exact sequence of deployment errors encountered and how they were resolved:</p><h3><strong><em>Obstacle 1: Regional Quota Exhaustion (REGION_CAPACITY_EXHAUSTED)</em></strong></h3><p>Deploying to europe-west1 returned an immediate precondition failure:</p><p>ERROR: (gcloud.beta.run.instances.deploy) FAILED_PRECONDITION: Instance capacity in region europe-west1 is currently exhausted. Please pick a different region.<br>reason: REGION_CAPACITY_EXHAUSTED</p><p><strong>💡 Real-World Gotcha #2: Regional Quotas on Beta Runtimes</strong></p><p><em>Custom sandbox allocations with 4 vCPU / 4GiB memory pools can hit regional capacity limits during preview phases. Shifting to an alternative regional pool like europe-west9 resolved the resource allocation immediately.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*oKXj59tShoGqZ6EFHRf8Jw.png" /><figcaption><strong><em>REGION_CAPACITY_EXHAUSTED failure and list of available regions</em></strong></figcaption></figure><h3><strong><em>Obstacle 2: Secret Access Checks &amp; The “Instance Stopped” Deadlock</em></strong></h3><p>When moving regions, the deployment stopped because openclaw-gateway-password lacked explicit Secret Accessor bindings, causing an audit error:</p><p>ERROR: Permission denied on secret: projects/…/secrets/openclaw-gateway-password/versions/latest for Revision service account openclaw-sa@…<br>reason: SECRETS_ACCESS_CHECK_FAILED</p><p><strong>💡 Real-World Gotcha #3: Corrupted Revisions &amp; Instance Lifecycle</strong></p><p><em>Unlike standard Cloud Run Services that retry dynamically, a Cloud Run Instance that fails pre-flight checks locks into Instance stopped. Running gcloud beta run instances start on a rejected revision will not clear the failure. The fix: grant the missing role, delete the stopped instance via CLI, and redeploy clean.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H5ibov26CF8Jr846l6TM8g.png" /><figcaption><strong><em>Audit log extract showing SECRETS_ACCESS_CHECK_FAILED</em></strong></figcaption></figure><h3>Step 5: Successful Deployment &amp; The “Ghost Resource” Phenomenon</h3><p>Executing a clean deployment in europe-west9:</p><pre>gcloud beta run instances deploy openclaw-instance \<br> - image ghcr.io/openclaw/openclaw:latest \<br> - service-account ${SERVICE_ACCOUNT} \<br> - port 18789 \<br> - cpu 4 \<br> - memory 4Gi \<br> - public \<br> - sandbox-launcher \<br> - add-volume mount-path=/home/node/.openclaw,type=cloud-storage,mount-options=&quot;uid=1000;gid=1000;file-mode=0700;dir-mode=0700&quot;,bucket=${BUCKET_NAME} \<br> - set-secrets GEMINI_API_KEY=gemini-api-key:latest,OPENCLAW_GATEWAY_PASSWORD=openclaw-gateway-password:latest \<br> - region europe-west9</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yv6Pp8ssFZsTsA-34DgapQ.png" /><figcaption><strong><em>Successful deployment output showing ‘Instance [openclaw-instance] has successfully been created’ and the assigned HTTPS URL</em></strong></figcaption></figure><h3>⁉️Interesting tips: Where is the Resource in Google Cloud Console?</h3><p>If you look at the GCP Console under <strong>Cloud Run -&gt; Services</strong> or <strong>Jobs</strong>, the list is completely empty. Furthermore, searching in <strong>Cloud Asset Inventory</strong> confirms that the resource type is not currently indexed in the visual UI.</p><p><strong>💡 Real-World Gotcha #4: CLI-First Control Plane</strong></p><p><em>Cloud Run Instances currently operate on a dedicated CLI control plane. You inspect and monitor them directly via:</em></p><pre>gcloud beta run instances describe openclaw-instance - region europe-west9<br>gcloud beta run instances logs read openclaw-instance - region europe-west9 - limit 30</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*ENHiaNwREoGD08Wuoxnrcw.png" /><figcaption><strong><em>Cloud Run Console showing 0 Services</em></strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b0GH-MZjDcPpFDHsmCkOmg.png" /><figcaption><strong><em>Cloud Run Console showing 0 Jobs</em></strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*14X0peO7S1-LsQ2eE7EZFQ.png" /><figcaption><strong><em>Cloud Asset Inventory dashboard showing no Cloud Run Instance resource type</em></strong></figcaption></figure><h3>Step 6: Testing Connectivity, Authentication &amp; Reverse Proxies</h3><h3>1. Unauthenticated Request</h3><p>curl -I <a href="https://openclaw-instance-282841632577.europe-west9.run.app/">https://openclaw-instance-282841632577.europe-west9.run.app</a><br># Returns: HTTP/2 403 (Forbidden by OpenClaw Gateway)</p><h3>2. Authenticated Health Check</h3><p>curl -i -H “Authorization: Bearer SuperSecretClawPassword123!” \<br> <a href="https://openclaw-instance-282841632577.europe-west9.run.app/health">https://openclaw-instance-282841632577.europe-west9.run.app/health</a><br># Returns: HTTP/2 200 OK</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r8PSPZjGhyMREcmYA2xyuw.png" /><figcaption><strong><em>curl 403, curl 200 with Bearer Auth, and the instance describe output</em></strong></figcaption></figure><h3>3. Handling Ingress Proxies (proxy_attribution_required)</h3><p>When POST requests pass through Google Front End (GFE) proxies, OpenClaw may require trusted proxy attribution. Simply include the following in openclaw.json on GCS and restart:</p><pre>{<br> &quot;gateway&quot;: {<br> &quot;trustedProxies&quot;: [&quot;*&quot;]<br> }<br>}</pre><p><strong>🔒 Security Deep Dive: Avoiding Wildcard Proxies</strong></p><p>While setting &quot;trustedProxies&quot;: [&quot;*&quot;] is a common quick-fix in codelabs, in production environments it opens the door to client-spoofing vulnerabilities.</p><p>The secure Cloud Architect approach is to explicitly restrict trust to Google’s internal forwarding ranges and private RFC 1918 subnets (169.254.8.1, 10.0.0.0/8, 172.16.0.0/12, loopback, linklocal). This allows the gateway to safely parse X-Forwarded-For from Google Front End (GFE) while rejecting untrusted intermediate proxies.</p><h3>The Early Adopter Reality: Calculator Glitches &amp; Tooling Maturity</h3><p>Testing a Beta feature means dealing with tooling that is still catching up.</p><p>The <strong>Google Cloud Pricing Calculator</strong> already displays the new <strong>“Instance”</strong> and <strong>“Worker Pool”</strong> resource types, but attempting to calculate costs currently throws an error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mynp9opOCZM3VdhG5pk9og.png" /></figure><blockquote><em>An error has occurred during price calculation.</em></blockquote><p>The UI is ready, but backend billing catalog SKUs are still stabilizing. For our lab setup (<strong>4 vCPU / 4 GiB RAM</strong> running 24/7), expect continuous always-on compute costs around <strong>$75–$80/month</strong> (excluding GCS storage and Gemini API tokens). Google notes that lightweight workloads using a minimal <strong>1 vCPU / 1 GiB RAM</strong> configuration start at just <strong>~$5.70/month</strong> thanks to shared vCPU tiering.</p><p>Between CLI-only visibility, regional capacity limits, and calculator glitches, the developer experience is still green — but maturing fast on the road to GA.</p><h3>Cloud Architect Reflections:</h3><ul><li><strong><em>Is an Always-On Serverless Instance Really Enterprise-Ready?</em></strong></li></ul><p>When you get a codelab up and running, the immediate temptation is to say: <em>“Alright, the instance is up, it returns 200 OK, mission accomplished.”</em> But when you take off the tinkerer’s hat and put on the Cloud Architect hat for an enterprise, financial, or regulated corporate environment, the perspective shifts completely.</p><p>Cloud Run was born as the ultimate stateless engine: it catches a request, spins up a container in milliseconds, answers, and tears it down. If that container gets compromised, the blast radius lasts only a few seconds and dies along with the instance. With <strong>Cloud Run Instances</strong>, however, we are introducing an entirely different beast: an <strong>always-on, stateful</strong> process, with a persistent filesystem mounted directly from Cloud Storage and continuous, long-lived network connections.</p><p>This fundamentally reshapes the threat model. Here are four key operational reflections before taking this into enterprise production.</p><ul><li><strong><em>The Temptation of Public IPs and the Pitfall of Static Passwords</em></strong></li></ul><p>In our lab test, we used the --public flag and shielded the gateway with a static password pulled from Secret Manager. In an enterprise landing zone, that is the first immediate red flag. A persistent instance exposed directly to the public internet is a permanent beacon for network scanners, automated bots, and brute-force attempts.</p><p>In production, an autonomous agent control plane should never expose a direct, public .run.app endpoint. Ingress must be locked down using --ingress=internal-and-cloud-load-balancing, fronted by an Application Load Balancer enforcing <strong>Google Cloud Armor</strong> WAF policies for geo-blocking, DDoS mitigation, and rate limiting. Even more crucially, access must be gated through <strong>Identity-Aware Proxy (IAP)</strong>. We should not authenticate users with static shared secrets passed in headers; we should rely on validated corporate identities through IAM and Context-Aware Access.</p><p>The proxy configuration is just as critical: setting &quot;trustedProxies&quot;: [&quot;*&quot;] is convenient to bypass errors in a lab, but in production, it means blindly trusting whatever X-Forwarded-For header gets injected along the wire. Restricting trust strictly to Google Front End (GFE) link-local IPs (169.254.8.1) and internal private subnets is non-negotiable.</p><ul><li><strong><em>The Network Perimeter and Data Exfiltration Risks</em></strong></li></ul><p>In a default deployment, the instance routes outbound traffic directly across the public internet to communicate with Secret Manager and the Gemini API. But what happens if an AI agent — equipped with execution sandboxes and dynamic code-running capabilities — is tricked by a prompt injection into making unauthorized external egress calls?</p><p>In a hardened landing zone, that instance must live within a dedicated <strong>VPC</strong> via Direct VPC Egress. Outbound connectivity must pass through strict Cloud NAT egress rules or inspection appliances. More importantly, the entire workload footprint — the Cloud Run Instance, the GCS state bucket, and Secret Manager — should sit inside a secured <strong>VPC Service Controls (VPC-SC)</strong> perimeter. This guarantees that even if the agent’s code sandbox is breached, runtime secrets and persistent storage cannot be exfiltrated to external Google Cloud projects.</p><ul><li><strong><em>GCS Persistence: Not Just Another Local Disk</em></strong></li></ul><p>Mounting a Cloud Storage bucket directly to /home/node/.openclaw is a clean way to solve the stateless filesystem limitation of Cloud Run, but it brings real data governance responsibilities.</p><p>If an autonomous agent corrupts its configuration or accidentally overwrites custom plugin modules, that damage is persistent and instantaneous. Production setups require <strong>Object Versioning</strong> enabled on the bucket, retention policies to prevent accidental wipes, and Customer-Managed Encryption Keys (<strong>CMEK</strong> via Cloud KMS) to align with enterprise compliance mandates. Running the container as a non-root user (uid=1000, gid=1000) with strict folder permissions (0700) is not just good practice—it is the security boundary that stops arbitrary code from altering host mount points.</p><ul><li><strong><em>The Verdict: How Should We Really View It?</em></strong></li></ul><p>Cloud Run Instances is a very promising evolution because it bridges the operational simplicity of serverless (zero OS patch cycles, no hypervisor management) with the persistent, dedicated execution models required by modern AI agents and autonomous workloads.</p><p>However, we cannot fall into the trap of thinking it is “secure by default” simply because it carries the serverless label. Being <em>always-on</em> means inheriting the same security responsibilities as a traditional Compute Engine VM, just without the burden of patching the underlying OS. The key is treating it with a <strong>Zero Trust</strong> architecture: hardened ingress, strict network boundaries, and locked perimeters around state and secrets.</p><h3>Conclusion</h3><p>Deploying bleeding-edge agent frameworks on managed serverless instances provides a glimpse into the future of stateful, sandboxed AI architectures. While beta quirks and CLI-first workflows require careful troubleshooting, the operational simplicity of zero VM management makes it a compelling runtime pattern.</p><p>Hopefully, documenting these exact errors, IAM checks, and lifecycle quirks saves you time when setting up your own agent runtime on Google Cloud Run Instances.</p><p>If you made it this far, thank you for your time! At the start of the year, I promised myself I’d start writing for the Google Cloud community. It took until September to get the first one out, and while it’s certainly not perfect, I’ve learned that waiting for perfection only gets in the way of sharing real value.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f5ce2c19ab2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>