<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ben King on Medium]]></title>
        <description><![CDATA[Stories by Ben King on Medium]]></description>
        <link>https://medium.com/@aviato.consulting?source=rss-2b7d3363ba71------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YWBwNtyW-QiRb67ybyFxFQ.png</url>
            <title>Stories by Ben King on Medium</title>
            <link>https://medium.com/@aviato.consulting?source=rss-2b7d3363ba71------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 02:43:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aviato.consulting/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[We Built an Agent That Publishes Our Website]]></title>
            <link>https://medium.com/@aviato.consulting/we-built-an-agent-that-publishes-our-website-077209a0ad21?source=rss-2b7d3363ba71------2</link>
            <guid isPermaLink="false">https://medium.com/p/077209a0ad21</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[wordpress]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Ben King]]></dc:creator>
            <pubDate>Fri, 25 Sep 2026 05:13:10 GMT</pubDate>
            <atom:updated>2026-09-25T05:13:10.262Z</atom:updated>
            <content:encoded><![CDATA[<p>Our marketing team now publishes pages on our website by opening a chat window in Gemini Enterprise and describing what they want. No CMS login. No Markdown*. No Git*. No developer, and no invoice.</p><p><em>*we use both markdown and git but the marketing team have no idea.</em></p><p>This post was originally posted on our website from a natural language prompt as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*rvsRXBaewDEzwjiJRdmHiA.png" /></figure><h3>What the agent actually does</h3><p>It’s built on Google Gemini Enterprise, with a set of tools wired up behind the chat interface:</p><ul><li><strong>Templates.</strong> It reads the Markdown templates in our repo, so it knows the difference between a blog post, a case study and a service page, and which collection each one belongs in.</li><li><strong>Images.</strong> You paste screenshots into the chat. It converts them to sized .webp files, writes the alt text, and drops them in public/images/.</li><li><strong>Validation.</strong> It checks the frontmatter against the same Zod schemas the production build uses, plus our brand voice rules. Bad dates, missing descriptions and banned corporate words get caught before anything is committed.</li><li><strong>Git.</strong> It commits to a new branch, opens a pull request against dev and merges it there.</li></ul><p>Getting to production is a separate step. I review what has landed on dev and merge it to main, and the site rebuilds in about two minutes.</p><p>The person writing the post never sees a terminal. They see a chat window and, at the end, a link to the pull request.</p><h3>The interesting part is the guardrails, not the writing</h3><p>Any model can write a blog post. Letting one commit to your website is a different problem, and that’s where most of the build went.</p><p>The agent can only ever push to dev, never main. It can’t touch protected index pages. It can’t invent a frontmatter field, because the schema rejects it and the build fails loudly rather than shipping broken metadata. Every change arrives as a pull request with a reviewable diff, so the rollback story is just Git.</p><p>That’s the difference between a chatbot and something you’d actually let near production. Constraints, and a human on the merge button. It’s the same pattern we use when we build <a href="https://www.aviato.consulting/ai-agents/"><strong>AI agents</strong></a> for clients: give the model a narrow set of real tools, validate everything it produces, and keep a person in the loop at the point of no return.</p><h3>How we use Gemini Enterprise</h3><p>We didn’t build a new app for this, and that’s deliberate. The team already works in <a href="https://www.aviato.consulting/gemini-enterprise/"><strong>Gemini Enterprise</strong></a> every day, so the publisher is just another agent in the sidebar. If you look at the screenshot above, it sits next to the other agents we run, like our Lead Architect Orchestrator and Google’s Deep Research.</p><p>That’s the part people underestimate. A custom agent only gets used if it lives where people already work. Nobody on our marketing team had to learn a new tool, find a new login or ask IT for access. They picked an agent from a list and started typing.</p><p>Gemini Enterprise handles the parts you don’t want to build yourself: sign in, access control, enterprise data controls, and a chat interface that already handles pasted images. That leaves the interesting work, which is the agent’s tools and its guardrails.</p><h3>Building custom agents, not buying chatbots</h3><p>Off the shelf chat is good at answering questions. It’s not good at doing a specific job inside your systems, because it doesn’t know your templates, your schemas or your rules. A custom agent does, because you give it exactly those things as tools.</p><p>The recipe we use, for ourselves and for clients, is the same every time:</p><ol><li><strong>Pick one job.</strong> “Publish a blog post” is a job. “Help with marketing” isn’t.</li><li><strong>Give it real tools for that job,</strong> and nothing else. Ours can read templates, convert images, validate and commit. It can’t do anything outside the website repo.</li><li><strong>Validate everything it produces</strong> against something that isn’t the model. For us that’s the Zod schemas and the build.</li><li><strong>Put a human at the point of no return.</strong> Here that’s the merge to main.</li></ol><p>Once you have one agent working like this, the next one is mostly a new set of tools on the same platform. If you want the deeper version of how we think about it, we’ve written about <a href="https://www.aviato.consulting/blog/the-4-layer-architecture-of-ai-systems/"><strong>the four layers of an AI system</strong></a>, <a href="https://www.aviato.consulting/blog/deploying-adk-agents-to-agentspace/"><strong>deploying ADK agents into Gemini Enterprise</strong></a> (back when it was still called Agentspace), and <a href="https://www.aviato.consulting/agent-reliability-engineering/"><strong>keeping agents reliable once they’re in production</strong></a>. For software teams, the same idea applied to delivery is <a href="https://www.aviato.consulting/agentic-engineering/"><strong>agentic engineering</strong></a>.</p><h3>Why WordPress made this impossible</h3><p>We used to run on WordPress, and we were paying a developer roughly $80 to publish a single page. Not to design it. Not to write it. Just to add it.</p><p>It was also slow. Time to first byte was around a second, because every request went through PHP and MySQL. A dozen plugins each shipped their own CSS and their own copy of jQuery, and the homepage carried over 1MB of render blocking assets. Mobile pages took about four seconds to load.</p><p>Google noticed. Search Console was full of “Crawled, currently not indexed”, which is what happens when your pages are slow and half the content only appears after JavaScript runs.</p><p>And you can’t point an agent at that. There’s no clean artefact to write, no schema to validate against, and a database that only a human with admin rights and an hourly rate can safely change.</p><h3>Astro made it possible</h3><p>We rebuilt on <a href="https://astro.build/"><strong>Astro</strong></a>. The thing that mattered most wasn’t speed, it was that content became files.</p><p>Markdown is the format models write natively. Frontmatter is typed and schema checked, so there’s a hard contract the agent has to satisfy. Git gives us review and rollback for free. The whole publishing pipeline turned into something a machine can operate and a human can audit.</p><p>The search results followed. Once the new site was live, the number of pages Google had indexed went from around 25 to 142.</p><p>The speed came along for the ride. Astro ships zero JavaScript by default and pre renders every page, so there’s nothing for the browser to wait on. This page scores 99 for performance in Lighthouse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5YNcqsFIYuKovOuWYb0BFg.png" /><figcaption>100 Performance score on Lighthouse</figcaption></figure><p>Nice, but secondary. The reason we did it is that publishing now costs a chat message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*kTtaL27jn2rPCrk7-tjk2A.png" /><figcaption>Google search console view of indexed pages</figcaption></figure><h3>The architecture, service by service</h3><p>There is no server. That’s the whole design, and everything else follows from it.</p><ul><li><strong>Gemini Enterprise </strong>Hosts the agent and the tools behind the chat window: read the templates, convert images, validate frontmatter, commit, open the pull request.</li><li><strong>Cloud Build </strong>The only path to production. A push to main triggers it, and it installs, validates, builds and deploys.</li><li><strong>Firebase Hosting </strong>Serves the built site from Google’s CDN, and owns the redirect table, the clean URLs and the security headers.</li><li><strong>Cloud IAM </strong>A dedicated deploy service account with four roles and nothing else.</li><li><strong>Cloud Logging </strong>Build logs. The pipeline sets CLOUD_LOGGING_ONLY, so there’s no bucket to manage.</li><li><strong>Cloud Storage </strong>Holds the source archive Cloud Build stages for each run. We only ever read it.</li></ul><p>That’s the entire footprint. No Compute Engine, no GKE, no Cloud Run, no Cloud SQL, no load balancer, no container registry. For a marketing site, all of that is machinery to patch rather than capability you use.</p><p>The pipeline itself is four steps:</p><ol><li>npm ci on node:26.8.2-bookworm-slim, pinned exactly so a base image refresh can’t change the build underneath us.</li><li>npm run check, which type checks and validates every content collection against its Zod schema. This is the step that catches a bad date or a missing description, and it’s the reason an agent can be trusted to write content at all.</li><li>npm run build, producing a directory of static files.</li><li>firebase deploy --only hosting.</li></ol><p>Two details matter more than they look. The build runs on E2_HIGHCPU_8, because the validation and build steps are CPU bound and the machine costs cents per run. And the deploy is keyless: the Cloud Build service account holds roles/firebasehosting.admin directly, so there’s no FIREBASE_TOKEN in a secret store, no key to rotate, and nothing to leak. The account is deliberately not granted roles/editor or roles/owner.</p><h3>How the site renders</h3><p>Every page is rendered once, at build time, into a file. A request for /services/security-reviews/ is answered by a CDN returning an HTML document that was already complete before the request existed. No PHP, no database query, no server side render, no cold start. Time to first byte is a cache lookup.</p><p>What that buys, concretely:</p><ul><li><strong>Zero JavaScript by default.</strong> Pages ship none. Interactive pieces are opted in individually: the currency selector on our pricing page, the delivery unit calculator, the carousels. Each is a few kilobytes, loaded only on the pages that use it. Nothing pulls in a framework runtime.</li><li><strong>Content as typed files.</strong> Markdown with schema checked frontmatter, compiled at build time. The same Zod schemas that reject the agent’s mistakes also generate the types our components use.</li><li><strong>Immutable asset caching.</strong> Hashed files under /_astro/ are served with max-age=31536000, immutable, because a changed file gets a new name. Repeat visits fetch almost nothing.</li><li><strong>Redirects at the edge.</strong> Retired URLs are 301s in the Hosting config, generated at build time from one map in astro.config.mjs. They resolve before any file is served, so a moved page never costs the visitor a render.</li><li><strong>Prefetching on intent.</strong> Links prefetch on hover, so the next page is usually in cache before the click lands.</li></ul><p>The security headers sit in the same config: HSTS with preload, nosniff, frame options, a referrer policy, and a permissions policy that turns off camera, microphone and geolocation. They’re reviewed in the same pull request as the content, which is not true of a plugin based CMS.</p><h3>Why Astro, AI, &amp; Google Cloud fit together</h3><p>Astro’s output is a directory. Firebase Hosting’s input is a directory. That’s the whole integration, and it’s why this stack has so few moving parts.</p><p>The deeper fit is that a static artefact removes the questions a marketing site should never have to answer. There’s no runtime to size, so there’s no autoscaling policy and no cold start. There’s no server process, so the attack surface is the CDN and the pipeline rather than an application. Rolling back is releasing the previous Hosting version, which takes seconds and needs no build. Traffic spikes are a CDN problem, which is to say not a problem.</p><p>Markdown is the language of AI, and this is the second really good reason we love Astro.</p><p>It also means the interesting work moves to build time, where failures are cheap. A broken page fails a Cloud Build step and never reaches a visitor. On our old WordPress site, the same mistake was live the moment it was saved.</p><p>If you’re running a content site on a VM or a container because that’s what the CMS needed, the honest question is what the runtime is actually doing for you. For us the answer was: serving the same HTML over and over, slower, with more to patch.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=077209a0ad21" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Workspace Studio Issues When Context Aware Access Is Enabled]]></title>
            <link>https://medium.com/google-cloud/google-workspace-studio-issues-when-context-aware-access-is-enabled-6b6de4ade51f?source=rss-2b7d3363ba71------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b6de4ade51f</guid>
            <category><![CDATA[google-workspace]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Ben King]]></dc:creator>
            <pubDate>Mon, 15 Jun 2026 03:25:40 GMT</pubDate>
            <atom:updated>2026-06-15T03:25:40.838Z</atom:updated>
            <content:encoded><![CDATA[<p>We recently ran into a frustrating issue where our team could not use Google Workspace Studio. With an error of</p><blockquote>“Couldn’t load because your Google Workspace Admin doesn’t allow you to use steps for this app.”</blockquote><p>This is due to us using Context Aware Access to ensure we have strict controls over our data and that of our customers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r8Wgl4Ht0zXd2UA1.png" /></figure><p>After investigating and reaching out to Google Support, we found the problem occurs when Context Aware Access is enabled. If you are hitting the same roadblock, here is the solution that got us back up and running.</p><p>Below is some context on why this happens, followed by the exact steps to resolve it.</p><h3>Why the block occurs</h3><p>For Workspace Studio to work with Context Aware Access turned on, it has to be exempted from API access blocks. Specifically, you need to exempt the API for 3 apps that workspace Studio uses in the background.</p><p>You might wonder why there is a distinction between a standard Google Task and a Task for Workspace Studio. Google Support explained that the Workspace Studio version acts as a plugin that uses the API. The distinction exists because Workspace Studio relies on APIs from other Google services to function.</p><p>As these things are not getting device signals Context Aware Access blocks them.</p><h3>How to fix the “Couldn’t load because your Google Workspace Admin doesn’t allow you to use steps for this app” issue</h3><p>To resolve this, your administrator needs to allow the specific OAuth Client IDs tied to Workspace Studio’s functionality. You can follow along with the video attached to this post, using these steps:</p><ol><li>Open the Google Admin console and go to your API access controls.</li><li>Click Configure New App</li><li>Allow the below OAuth Client ID’s for the services you want to use:</li></ol><ul><li>Core Workspace Studio / General: <a href="http://122846605100-d4jhvnotnvqprrtasnrgrahea3015vrn.apps.googleusercontent.com/">122846605100-d4jhvnotnvqprrtasnrgrahea3015vrn.apps.googleusercontent.com</a></li><li>Gmail: <a href="http://653651007783-367aea46j9ra0jkssl9u8783gu91t55p.apps.googleusercontent.com/">653651007783–367aea46j9ra0jkssl9u8783gu91t55p.apps.googleusercontent.com</a></li><li>Tasks: <a href="http://898568525849-f6jivl2uk6o0gbg4e6r1vpi6goom0h7u.apps.googleusercontent.com/">898568525849-f6jivl2uk6o0gbg4e6r1vpi6goom0h7u.apps.googleusercontent.com</a></li><li>Gemini: <a href="http://296758037454-ssrceub8i8moppipckhk61mk8go2kk6t.apps.googleusercontent.com/">296758037454-ssrceub8i8moppipckhk61mk8go2kk6t.apps.googleusercontent.com</a></li><li>Google Chat: <a href="http://873507009103-gi4ff8nkju6ggaurt6ebbiqrfmr0c79j.apps.googleusercontent.com/">873507009103-gi4ff8nkju6ggaurt6ebbiqrfmr0c79j.apps.googleusercontent.com</a></li><li>Google Calendar: <a href="http://469390422673-r0p2tar3lgjlnhtsv4t4uonq4bi2svf8.apps.googleusercontent.com/">469390422673-r0p2tar3lgjlnhtsv4t4uonq4bi2svf8.apps.googleusercontent.com</a></li><li>Google Drive: <a href="http://694558745529-92brqls6rjct461cs7aqq7vfrbhua375.apps.googleusercontent.com/">694558745529–92brqls6rjct461cs7aqq7vfrbhua375.apps.googleusercontent.com</a></li><li>Google Docs: <a href="http://485273575958-8k1607iujnatgmksa7igjoee7snlc7i1.apps.googleusercontent.com/">485273575958–8k1607iujnatgmksa7igjoee7snlc7i1.apps.googleusercontent.com</a></li><li>Google Sheets: <a href="http://38402688245-th87uf1kjesqvopm6lf2v4sugvjbkl6k.apps.googleusercontent.com/">38402688245-th87uf1kjesqvopm6lf2v4sugvjbkl6k.apps.googleusercontent.com</a></li><li>Google Forms: <a href="http://883207888028-4bfq5l4rvep2fp0ivs0lm05db6stq200.apps.googleusercontent.com/">883207888028–4bfq5l4rvep2fp0ivs0lm05db6stq200.apps.googleusercontent.com</a></li><li>Google Meet: <a href="http://234869138288-1etlpk2g7sei2dg27u4bb5ahpe3qri33.apps.googleusercontent.com/">234869138288–1etlpk2g7sei2dg27u4bb5ahpe3qri33.apps.googleusercontent.com</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HiBZj40ICBvProL_JZn9TA.png" /><figcaption>Adding Gemini For Workspace Studio</figcaption></figure><p>Scope it to your org and click Continue</p><p>Exempt from having API access blocked by CAA</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8uBuGmTnyeHh6SbG4kh61A.png" /><figcaption>Exempt from having API access blocked</figcaption></figure><p>Save your changes so the exemptions take effect.</p><h3>How to verify the blocks</h3><p>If you want to confirm this is the exact issue affecting your users before making any changes, your administrator can verify the blocks directly in the logs:</p><ol><li>Go to the Google Admin console and click Reporting.</li><li>Navigate to Audit and investigation.</li><li>Open the Context Aware Access log events.</li><li>Filter the logs for the affected users.</li><li>Look for Access Denied events.</li><li>Review those denied events to confirm the specific OAuth Client IDs and the Context Aware Access policy that caused the denial.</li></ol><p>These steps cleared up the issue for our team immediately. Let me know in the comments if you run into any hurdles while updating your admin settings.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b6de4ade51f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/google-workspace-studio-issues-when-context-aware-access-is-enabled-6b6de4ade51f">Google Workspace Studio Issues When Context Aware Access Is Enabled</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The 4-Layer Architecture of AI Systems]]></title>
            <link>https://medium.com/google-cloud/the-4-layer-architecture-of-ai-systems-935d1ceaf657?source=rss-2b7d3363ba71------2</link>
            <guid isPermaLink="false">https://medium.com/p/935d1ceaf657</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <dc:creator><![CDATA[Ben King]]></dc:creator>
            <pubDate>Thu, 26 Feb 2026 05:35:59 GMT</pubDate>
            <atom:updated>2026-02-26T05:35:59.863Z</atom:updated>
            <content:encoded><![CDATA[<p>The word “agent” gets thrown around a lot right now. If you string two API calls together, someone is going to call it an autonomous AI agent. But if you’ve actually tried to build a system that you can run in production, and get real work done without constant hand holding, you know this is not going to cut it.</p><p>Building production ready agentic workflows requires a specific architecture. Over a lot of customer engagements Aviato have found it easiest to think about this stack in four distinct layers plus the underlying plumbing that keeps it all from exploding in production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zI9G0XtUDsytYxKUley_ZA.png" /><figcaption>Layers of the Agentic AI Stack</figcaption></figure><p>Here’s a practical look at how the modern agentic stack is actually built, and what you need to productionise AI Systems:</p><h3>Layer 1: Large Language Models (LLM’s)</h3><p>At the absolute bottom of the stack sits your foundation model. This is where you’re dealing with the raw mechanics: pinging APIs, handling tokenization, tweaking inference parameters, and prompt engineering.</p><p>You give it instructions, and it responds. On its own, it doesn’t care about your long term objectives, it forgets what happened five minutes ago, and it definitely can’t orchestrate a complex, multi step workflow. To get that, you have to move up the stack.</p><h3>Layer 2: Agents</h3><p>This is where we take a reactive model and actually turn it into an <em>agent</em>. We’re wrapping the LLM in code that gives it persistence, structure, and a goal.</p><p>Instead of just answering a question, a Layer 2 agent can actually pursue an objective. To make that happen, you have to bolt on a few things:</p><ul><li><strong>Memory and state management:</strong> So it doesn’t lose the plot halfway through a task.</li><li><strong>Tools:</strong> Structured function calling so it can actually <em>do</em> things like hit an external API or query a database.</li><li><strong>Context:</strong> This is where your RAG (Retrieval-Augmented Generation) pipelines live.</li><li><strong>Reasoning:</strong> The logic required to look at a big task, break it down into steps, and adapt if step two completely fails.</li></ul><p>Unlike a standalone model, a Layer 2 agent acts, looks at the intermediate result of that action, and adapts its next move based on what just happened.</p><h3>Layer 3: Multi-Agent Systems</h3><p>Eventually, you’re going to give a single agent a task that’s simply too big. The context window is exhausted, it loses focus, and the whole thing falls apart.</p><p>That’s when you need to bring in a multi-agent system. Instead of writing one prompt to rule them all, you build a distributed team of specialist sub agents. This layer handles the collaboration between them, including:</p><ul><li><strong>Inter-agent communication protocols</strong> so they can talk to each other (e.g. <a href="https://github.com/a2aproject/A2A">A2A</a>)</li><li><strong>Intelligent routing</strong> to make sure the coding task goes to the coding agent, not the research agent.</li><li><strong>Shared state coordination</strong> so they aren’t overwriting each other’s work.</li><li><strong>Parallel workflows</strong> so multiple agents can grind at the same time.</li></ul><p>By splitting up the work, the whole system becomes drastically faster, more robust, and way less prone to hallucinating under pressure.</p><h3>Layer 4: Agentic Ecosystems</h3><p>When you have a bunch of specialized agents running around asynchronously, things turn into chaos fast. Without structured orchestration, a multi-agent setup is just a cool local demo. With it, you get a scalable, reliable system that can actually survive real-world constraints.</p><p>For a reliable production system you need:</p><ul><li><strong>Evaluation Tooling</strong> to help understand if a modified system prompt or new version of your LLM will improve or degrade your agents (<strong>Vertex AI Evaluation Engine</strong>).</li><li>A mechanism to <strong>escalate to a human</strong></li><li>Monitoring &amp; Logging, to provide transparency into what the agent is doing (<strong>Google Cloud Logging</strong>)</li><li>Orchestration to manage the flow of execution and keep the system coherent at scale (<strong>Vertex AI Agent Engine</strong>, with ADK)</li><li><strong>Guardrails </strong>and sandboxing to prevent undesirable actions</li><li>Error Handling, Retries, and <strong>loop prevention</strong></li></ul><p>These are not sexy, but are how you ensure accountability, mitigate failure modes, and actually preserve trust in the automated decisions your software is making.</p><p>Aviato have run a number of POC’s in as short as 6 weeks (for 80k AUD), to prove an agentic system built on Google Cloud can meet your needs.</p><p>Moving these to production requires a team (or Aviato SRE’s) to manage them, and a bit of additional thought.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*2k0xNLOkvVNzV7Z69hfukA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=935d1ceaf657" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/the-4-layer-architecture-of-ai-systems-935d1ceaf657">The 4-Layer Architecture of AI Systems</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Your AI Strategy Needs More Than Just Tech]]></title>
            <link>https://medium.com/google-cloud/why-your-ai-strategy-needs-more-than-just-tech-27dd515c949c?source=rss-2b7d3363ba71------2</link>
            <guid isPermaLink="false">https://medium.com/p/27dd515c949c</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[strategy]]></category>
            <category><![CDATA[business-strategy]]></category>
            <dc:creator><![CDATA[Ben King]]></dc:creator>
            <pubDate>Mon, 22 Sep 2025 07:43:36 GMT</pubDate>
            <atom:updated>2025-09-22T07:43:36.870Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yj4dJTI6J8rgDo2uDONlOw.png" /><figcaption>Gemini Generated Hero Image</figcaption></figure><p>The hype around Artificial Intelligence has moved from the theoretical to the tangible. With businesses moving more of their experiments towards production use cases. While media reports of 90% AI project failure may be exaggerated, a significant number of promising initiatives still falter before reaching production. My experience helping large enterprises shows this isn’t due to a lack of enthusiasm, funding or technology, but a failure to build the necessary strategic scaffolding.</p><p>While I have seen much smaller numbers than 90% of these projects fail, we are seeing a few too many. Some projects finish on time, and budget, and deliver what was asked but not move into the production phase. The challenges we have identified are not a lack of funding, ambition or technology, it’s a failure to build the necessary strategic and technical scaffolding required to productionize AI. Moving from isolated pilot projects to scaled, AI is a journey and the people, and change aspects need to be well planned out.</p><p>Scaling AI successfully has a number of prerequisites:</p><ol><li>We need adequate programs and change management.</li><li>We need strong cloud foundations.</li><li>It needs a bottom up use case</li><li>Identify broad or deep AI and clearly communicate this</li><li>It needs a central framework for evaluation and updates.</li></ol><h3>Program &amp; Change Management</h3><p>When Cloud became a hot topic, large programs of work were spun up with governance teams, and PMO offices, and migration were segmented into the 7 R’s, and then split further into waves. AI is going to be substantially more transformative than moving workloads to the cloud, and the impact on people is going to be far greater, but little thought has been given to the programme and change management aspects.</p><p>This requires a fundamental shift in thinking from treating AI as a series of disjointed tech experiments to embedding it as a core, strategic capability encompassing the tech teams, change teams, and spinning up the required PMO office and developing a strategy to manage the change that incorporates your people.</p><h3>Cloud Foundations</h3><p>For our clients, this often means building out a proper resource hierarchy in the cloud, defining identity and access management protocols, implementing robust security controls, and establishing clear cost governance mechanisms.</p><p>This gives everybody assurance that data will not be leaked, or used to train models on proprietary data, as well as being foundation for giving the right people the right access.</p><p>One thing that is still unclear to execs is what the infrastructure cost of AI is going to be, in our experience it is often magnitudes cheaper than exec’s are estimating, but those savings are quickly swallowed up by the required change management.</p><h3>Use Case Identification</h3><p>A focus on solving critical business problems is something that technologies forget about. Aviato are sure that a disciplined framework for identifying and prioritizing use cases is what separates the projects that realise enterprise value from those that end up being scrapped.</p><p>This does not need to be complex, and a bottom up approach seems to get the most traction, the employees on the coal face are acutely aware of what parts of their jobs they want to automate away.</p><p>What we have seen work:</p><ol><li>Where are the most significant bottlenecks, operational inefficiencies, or untapped growth opportunities? <strong>Put this into action by running a company wide survey.</strong></li><li>AI is fueled by data. A brilliant idea is useless if the required data is inaccessible, or of poor quality. Most white collar workers have access to the data that an AI Agent would need, however is there an API that the agent can use? <strong>Put this into action by adding a new column to your survey results and identify if the data is available.</strong></li><li>Evaluate potential projects on two axis, potential business impact and feasibility. <strong>Put this into action by stack ranking these, find the low hanging fruit that are not complex, but deliver reasonable returns. Build momentum from there for the more complex.</strong></li></ol><h3>Broad or Deep</h3><p>One area where expectations are often misaligned is around what the AI Implementation is supposed to do, Google Agentspace, or Glean is one example of a broad enterprise wide AI implementation that is familiar, it will summarise all of your companies knowledge and provide a Chat interface similar to Chat GPT or Gemini but trained on your company data. These agents are great at saving time for a broad number of use cases but are very unlikely to take autonomous actions.</p><p>The other hand we have the deep agents, these are very specific, they will help Peter from the legal team review contracts faster, or Mary from security find security details in logs, these agents will more likely take autonomous actions, and be very specific to a role.</p><p>When implementing a broad AI platform, a lot of people are expecting something that will take autonomous actions, and while Google’s Agentspace has a very exciting roadmap it is just not at the level of doing this deep work yet.</p><h3>Centralised Framework</h3><p>Once you have a production AI agent the job is not over, you now need a way to manage this, and bring a structured approach to optimising it. What happens when OpenAI or Google release a new model, do you switch immediately? Do the benchmarks they provide match your use case?</p><p>In the same way as we run software updates for the newest version of Java, a structured approach to life cycling system prompts and agents is required, and validating these against metrics that matter to your use case, latency is key for a chat bot, accuracy is key for a software engineering agent, and cost is a consideration to all use cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VXAx7GoF9JCR3HCn.jpg" /><figcaption>Gemini Generated Flow of AI Evaluation</figcaption></figure><p>Google is definitely leading the way with their Vertex AI Evaluation tooling, but skipping over this and “YOLOing” changes to production agents is a problem waiting to happen.</p><h3>Conclusion</h3><p>Aviato are sure the future of the professional workforce is a partnership between humans and AI agents. However, this future won’t arrive by accident. It must be built with a disciplined, structured approach that treats AI not as a series of tech experiments, but as part of the core business strategy, and run as a transformation project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27dd515c949c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/why-your-ai-strategy-needs-more-than-just-tech-27dd515c949c">Why Your AI Strategy Needs More Than Just Tech</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying ADK Agents to Agentspace]]></title>
            <link>https://medium.com/google-cloud/deploying-adk-agent-to-agentspace-cd2a7e679dc7?source=rss-2b7d3363ba71------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd2a7e679dc7</guid>
            <category><![CDATA[agent-development-kit]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agentspace]]></category>
            <category><![CDATA[google-agentspace]]></category>
            <dc:creator><![CDATA[Ben King]]></dc:creator>
            <pubDate>Fri, 20 Jun 2025 00:01:58 GMT</pubDate>
            <atom:updated>2025-07-02T05:15:02.159Z</atom:updated>
            <content:encoded><![CDATA[<p>This post outlines the steps required to deploy an Agent Development Kit (ADK) Agent from Agent Engine to Agentspace.</p><p>Credit to Andy Hood for figuring this out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m0kPFCxizyIvuCYIN-DMkA.jpeg" /><figcaption>ADK agent into Agenntspace</figcaption></figure><p>Hopefully Google publish some docs on how to do this and thanks to Andy Hood for figuring this out. If you do need help with this reach out via <a href="https://aviato.consultinng/">https://aviato.consulting</a></p><p>Follow these instructions carefully to ensure a successful deployment.</p><h3>Note</h3><p>Both Agent Engine and AgentSpace have been recently renamed as part of Google’s AI branding, so you will still see references in the APIs to their previous names:</p><ul><li><strong>Agent Engine:</strong> was formerly known as Reasoning Engine</li><li><strong>Agentspace:</strong> was formerly known as Discovery Engine</li></ul><h3>Prerequisites</h3><p>Before beginning the deployment process, ensure you have the following:</p><ul><li><strong>ADK Agent deployed to Agent Engine:</strong> A fully developed and tested ADK Agent has been deployed to Agent Engine</li></ul><p>Instructions for developing and deploying the ADK Agent are outside of the scope of this document.</p><ul><li><strong>Agentspace Account:</strong> An active Agentspace account with the necessary permissions to deploy agents.</li><li><strong>Credentials:</strong> Valid credentials for authenticating with Agentspace.</li></ul><h3>Deployment Steps</h3><p>The deployment process involves several key steps:</p><h4>Step 1: ADK Agent deployed to Agent Engine</h4><ol><li><strong>Obtain the Id of the deployed ADK Agent:</strong> You can obtain the ID via the Google Cloud Console or by using the following <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/reference/rest/v1/projects.locations.reasoningEngines/list">Agent Engine:List REST API</a>:</li></ol><pre>LOCATION=us-central1<br>PROJECT_ID=aviato-project-id<br>TOKEN=$(gcloud auth print-access-token)<br>curl -X GET &quot;https://$LOCATION-aiplatform.googleapis.com/v1/projects/$PROJECT_ID/locations/$LOCATION/reasoningEngines&quot; \<br> - header &quot;Authorization: Bearer $TOKEN&quot;<br>Response (truncated):<br>{<br>&quot;reasoningEngines&quot;: [<br>{<br>&quot;name&quot;: &quot;projects/123456789/locations/us-central1/reasoningEngines/123456789&quot;,<br>&quot;displayName&quot;: &quot;ADK Short Bot&quot;,<br>&quot;spec&quot;: {<br>…<br>}</pre><p><strong>Note</strong>: when obtaining the ID via the Google Cloud Console, the ID may have the Project Id in the resource name. The AgentSpace API appears to require the Project Number instead.</p><h3>Step 2: Obtain the Id of your AgentSpace application</h3><p><strong>Option 1:</strong> Use the AgentSpace menu in the Google Cloud Console to obtain the ID of your AgentSpace application:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xZX7cPUQCgtmuxfZ" /></figure><p><strong>Option 2:</strong> Use the AgentSpace List Engines REST API to list the current AgentSpace applications in your project:</p><pre># Note AgentSpace currently only supports the global, us and eu multi-regions<br>LOCATION=global<br>PROJECT_ID=aviato-project-id<br>TOKEN=$(gcloud auth print-access-token)<br>curl -X GET &quot;https://discoveryengine.googleapis.com/v1alpha/projects/$PROJECT_ID/locations/$LOCATION/collections/default_collection/engines&quot; \<br> - header &quot;Authorization: Bearer $TOKEN&quot; \<br> - header &quot;x-goog-user-project: $PROJECT_ID&quot;<br>Response (truncated):<br>{<br>&quot;engines&quot;: [<br>{<br>&quot;name&quot;: &quot;projects/123456789/locations/global/collections/default_collection/engines/agentspace-andy_123456789&quot;,<br>&quot;displayName&quot;: &quot;Agentspace - Andy&quot;,<br>&quot;createTime&quot;: &quot;2025–06–05T22:55:44.459263Z&quot;,<br>…<br>}</pre><p><strong>Important Notes</strong></p><ol><li>If you receive an error saying “Your application is authenticating by using local Application Default Credentials. The discoveryengine.googleapis.com API requires a quota project, which is not set by default” then make sure you have added the <strong>x-goog-user-project</strong> header.</li><li>In the REST API, the collection name uses a hard coded default of <strong>default_collection</strong>.</li></ol><h3>Step 3: Publish the Agent Engine ADK Agent to AgentSpace application</h3><p>The below requires the Project Number, e.g. 123456789, instead of the Project Id, e.g. aviato-project. To obtain the project number use:</p><pre>gcloud projects describe PROJECT_ID</pre><p>Use the <a href="https://cloud.google.com/agentspace/agentspace-enterprise/docs/reference/rest/v1alpha/projects.locations.collections.engines.assistants.agents/create">AgentSpace Create Agent REST API</a> to publish your Agent Engine ADK Agent to your AgentSpace application. In the body of the POST request, ensure that you replace the reasoningEngine name with the ID returned in Step 1.</p><pre># Note AgentSpace currently only supports the global, us and eu multi-regions<br>LOCATION=global<br>PROJECT_ID=aviato-project-id<br>TOKEN=$(gcloud auth print-access-token)<br># Use the AgentSpace Application ID returned in the previous step<br>AGENTSPACE_ID=agentspace-andy_1749164028618<br>curl -X POST &quot;https://discoveryengine.googleapis.com/v1alpha/projects/$PROJECT_ID/locations/$LOCATION/collections/default_collection/engines/$AGENTSPACE_ID/assistants/default_assistant/agents&quot; \<br> - header &quot;Authorization: Bearer $TOKEN&quot; \<br> - header &quot;x-goog-user-project: $PROJECT_ID&quot; \<br> - data &#39;{<br>&quot;displayName&quot;: &quot;My ADK Agent&quot;,<br>&quot;description&quot;: &quot;Description of the ADK Agent&quot;,<br>&quot;adkAgentDefinition&quot;: {<br>&quot;tool_settings&quot;: {<br>&quot;tool_description&quot;: &quot;Tool Description&quot;<br>},<br>&quot;provisionedReasoningEngine&quot;: {<br>&quot;reasoningEngine&quot;: &quot;projects/123456789/locations/us-central1/reasoningEngines/123456789&quot;<br>}<br>}<br>}&#39;</pre><p><strong>Response:</strong></p><pre>{<br>&quot;name&quot;: &quot;projects/123456789/locations/global/collections/default_collection/engines/agentspace-andy_1749164028618/assistants/default_assistant/agents/123456789&quot;,<br>&quot;displayName&quot;: &quot;My ADK Agent&quot;,<br>&quot;description&quot;: &quot;Description of the ADK Agent&quot;,<br>&quot;adkAgentDefinition&quot;: {<br>&quot;toolSettings&quot;: {<br>&quot;toolDescription&quot;: &quot;Tool description&quot;<br>},<br>&quot;provisionedReasoningEngine&quot;: {<br>&quot;reasoningEngine&quot;: &quot;projects/123456789/locations/us-central1/reasoningEngines/123456789&quot;<br>}<br>},<br>&quot;state&quot;: &quot;CONFIGURED&quot;<br>}</pre><p><strong>Important Notes</strong></p><ol><li>If you receive an error saying “Your application is authenticating by using local Application Default Credentials. The discoveryengine.googleapis.com API requires a quota project, which is not set by default” then make sure you have added the <strong>x-goog-user-project</strong> header to the REST API</li><li>In the REST API, the collection name and assistant names use hard coded defaults of <strong>default_collection</strong> and <strong>default_assistant</strong> respectively.</li></ol><h3>Step 4: Grant Required Permissions</h3><p>In your project, AgentSpace runs under the Google-provided Discovery Engine Service Account with a name such as:</p><p><a href="/service/mailto:service-807227392386@gcp-sa-discoveryengine.iam.gserviceaccount.com">service-$PROJECT_NUMBER@gcp-sa-discoveryengine.iam.gserviceaccount.com</a></p><p>By default, this service account only has the Discovery Engine Service Agent role.</p><p>This is insufficient to invoke the Agent Engine ADK Agent and you may get the error “I’m sorry, it seems you are not allowed to perform this operation when invoking your ADK Agent in AgentSpace.</p><p>If you receive this error, grant the Vertex AI User role to the service account: e.g</p><pre>PROJECT_ID=aviato-project-id<br>DISCOVERY_ENGINE_SA=service-123456789@gcp-sa-discoveryengine.iam.gserviceaccount.com<br>gcloud projects add-iam-policy-binding $PROJECT_ID \<br> - member=&quot;serviceAccount:$DISCOVERY_ENGINE_SA&quot; \<br> - role=&quot;roles/aiplatform.user&quot;</pre><h3>Troubleshooting</h3><p>If you encounter any issues during deployment:</p><ul><li><strong>Check Logs:</strong> The first step should always be to review the agent’s logs in Agentspace for detailed error messages.</li><li><strong>Verify Configuration:</strong> Double-check all configuration settings, especially environment variables and start commands.</li><li><strong>Dependencies:</strong> Ensure all required dependencies are correctly packaged with your agent.</li><li><strong>Network Issues:</strong> Confirm there are no network connectivity issues preventing communication with Agentspace.</li><li><strong>Agentspace Documentation:</strong> Refer to the official Agentspace documentation for specific troubleshooting guides and FAQs.</li></ul><p>By following these instructions, you should be able to successfully deploy your Agent Engine agent to Google Agentspace.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd2a7e679dc7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/deploying-adk-agent-to-agentspace-cd2a7e679dc7">Deploying ADK Agents to Agentspace</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MCP and Agentic AI on Google Cloud Run]]></title>
            <link>https://medium.com/google-cloud/mcp-and-agentic-ai-on-google-cloud-run-db26e8760f61?source=rss-2b7d3363ba71------2</link>
            <guid isPermaLink="false">https://medium.com/p/db26e8760f61</guid>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[google-cloud-run]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Ben King]]></dc:creator>
            <pubDate>Thu, 12 Jun 2025 23:10:18 GMT</pubDate>
            <atom:updated>2025-06-19T23:26:42.953Z</atom:updated>
            <content:encoded><![CDATA[<p>We’re moving from AI that primarily responds via text, to AI that manipulates thinngns. These “agentic AI” systems use tools to do that manipulatin. The way they interact with tools is via Anthropics Model Context Protocol (MCP), this open source standard for helping LLM’s connect and use external data sources, or tools.</p><p>How and where you run these tools is the purpose of this article, the TL;DR is: “Google Cloud Run”.</p><p>But read on if you want the details, and points to look out for.</p><h3>Why Cloud Run for Agentic AI?</h3><p>When it comes to deploying these sophisticated agentic AI Google Cloud Run emerges as a great choice, its serverless nature is super scalable, cost efficient, and requires no ops team to keep it running.</p><p>Further it can easily connect to any Databases or LLM’s running on Google Vertex AI without leaving your network (VPC).</p><p>What previously might have required dedicated SRE and DevOps teams can now be tackled by an individual developer, freeing up time to innovate on the actual AI Agent.</p><h3>Architecting Your Agent on Cloud Run</h3><p>So, what does a typical agentic AI architecture on Cloud Run look like?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/0*LHbuLueqmHqEKsyO" /></figure><p>At its core, you’ll have:</p><ol><li>Cloud Run service for the User Interface</li><li>Cloud Run Service for Agent Development Kit (or Langraph, Langchain, etc) to coordinate the agent’s behavior</li><li>A Database — A vector DB for Retrieval-Augmented Generation (RAG)</li><li>Short Term memory: Session data or caching with Memorystore</li><li>Long Term memory: A vector DB for RAG</li><li>Vertex AI with your choice of LLM : Google’s Gemini API, custom models or other foundation models deployed on Vertex AI Endpoints</li><li>Cloud Run Service for MCP Servers : Agents use tools to perform specific tasks or to interact with external services, APIs, and websites.</li></ol><h3>MCP Servers</h3><p>Cloud Run is a well known pattern for all of this, but with MCP being so new and the focus of this article it might be best to take a step back and explain what Model Context Protocol (MCP) does .</p><p>MCP directly addresses the inability of an LLM to use tools; it provides a standardized, structured way for systems to expose their capabilities to language models. Here are some examples:</p><ul><li>Git — Read, search, and manipulate Git repositories</li><li>Google Maps — Location services, directions, and place details</li><li>EverArt — AI image generation using various models</li><li>Puppeteer — to browse the web, extract information, or perform actions via clicks and keyboard input</li><li>Google Drive — File access and search capabilities for Google Drive</li></ul><p>Further the use of MCP lets us change our LLM as new ones are released to improve your agents without rebuilding from scratch, a great benefit of using VertexAI we can do this with a line of code.</p><h3>Practical MCP Server Deployment on Cloud Run</h3><p>Here’s a quick overview on how you can get your MCP servers up and running on Cloud Run for non production uses.</p><h3>Deployment from Container Images</h3><p>If your MCP server is already packaged as a container image (perhaps from <a href="https://hub.docker.com/catalogs/mcp">Docker Hub</a>), deploying it is straightforward. You’ll use the command:</p><pre>gcloud run deploy SERVICE_NAME - image IMAGE_URL - port PORT</pre><p>For instance, deploying a generic MCP container might look like:</p><pre>gcloud run deploy my-mcp-server - image us-docker.pkg.dev/cloudrun/container/mcp - port 3000</pre><h3>Deployment from Source</h3><p>If you are deploying a production use case, this is the recommended approach, if you have the source code for an MCP server (Perhaps from <a href="https://github.com/docker/mcp-servers">GitHub</a>) you can deploy it directly. Simply clone the repository, navigate into its root directory, and use:</p><pre>gcloud run deploy SERVICE_NAME - source .</pre><p>Cloud Run will handle the building and deployment, or you can work this into a CI/CD pipeline for a more production ready use case.</p><p><em>Cloud Run does not support MCP servers that rely on Standard Input/Output (stdio) transport. This constraint implicitly pushes MCP server development towards web-centric, network-addressable services, which aligns better with cloud-native architectures and scalability.</em></p><h3>State Management Strategies for Agentic AI on Cloud Run</h3><p>Fortunately, Google Cloud provides robust solutions for managing the various types of state your agentic AI systems will require:</p><h4>Short Term Memroy / Caching</h4><p>For data that needs fast access, like session information or frequently accessed data for an agent, connecting your Cloud Run service to Memorystore for Redis is an excellent option.2</p><h4>Long-term Memory / Persistent Knowledge</h4><p>For storing conversational history, user profiles, or other forms of persistent agent knowledge, Firestore offers a scalable, serverless NoSQL database solution.</p><p>If your agent deals with structured data or requires the powerful RAG capabilities discussed earlier, Cloud SQL for PostgreSQL or AlloyDB for PostgreSQL are ideal choices or <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/rag-engine/vector-db-choices">one of the many</a> that work on Google’s Vertex AI RAG Engine.</p><h4>Orchestration Framework Memory</h4><p>Many AI orchestration frameworks, such as LangChain, come with built-in memory modules. For example, LangChain’s ConversationBufferMemory can store conversation history to provide context across multiple turns. These often integrate with external stores for persistence.</p><h3>Table 1: State Management Options for Agentic Systems on Cloud Run</h3><p>Choosing the right state management approach depends heavily on the specific requirement:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*lipXj0DbZ8SFyeWOd4n8tQ.png" /><figcaption>State Management Options for Agentic Systems on Cloud Run</figcaption></figure><h3>The Challenge of Stateful MCP Servers</h3><p>As highlighted, MCP servers using Streamable HTTP transport might need to maintain a persistent session context, especially to allow clients to resume interrupted connections. The core challenge here, (as of June 2025), is that many official MCP SDKs lack support for external session persistence, aka storing session state in a dedicated service like Redis. Instead, they often keep the session state in the memory of the server instance.</p><p>This makes horizontal scaling problematic, if a client’s subsequent request is routed by a load balancer to a different instance from the one that initiated the session, the session context is lost, and the connection will likely fail. This limitation in current MCP SDKs points to a maturity gap in the ecosystem and until SDKs evolve to better support externalized state, designing MCP servers to be stateless is the more resilient cloud native pattern where feasible.</p><h3>Cloud Run Session Affinity to the Rescue?</h3><p>Cloud Run offers a feature called session affinity that can help mitigate this issue.</p><p>When enabled, Cloud Run uses a session affinity cookie to attempt to route sequential requests from a particular client to the same revision instance.</p><p>You can enable this with a gcloud command:</p><pre>gcloud run services update SERVICE - session-affinity</pre><p>Or via the Google Cloud Console or YAML config.</p><p>However, it’s crucial to understand that session affinity on Cloud Run is “best effort”.</p><p>If the targeted instance is terminated (due to scaling, etc) or becomes overwhelmed (reaching maximum request concurrency, etc), session affinity will be broken, and subsequent requests will be routed to a different instance.</p><p>So if the in memory state is absolutely critical and irreplaceable, session affinity alone is not a foolproof guarantee of state preservation.</p><h3>Addressing SDK and Session Affinity Limitations</h3><p>Given these constraints, there are 2 practical approaches:</p><ol><li><strong>Manual Cookie Handling:</strong> If you’re using a client SDK that doesn’t natively support cookies and you need to work with load balancer level sticky sessions you might need to implement manual cookie handling in your client code. This is a workaround, but it can be necessary.</li><li><strong>Stateless MCP Server Design (preffered):</strong> The most robust cloud native approach and what Aviato Consulting recommend, is to design your MCP servers to be stateless. The MCP specification itself permits a stateless server mode where the server doesn’t maintain session context between requests, and clients aren’t expected to resume dropped connections. This design enables seamless horizontal scaling and is ideal for environments where elasticity and load distribution are critical.</li></ol><h3>Operationalizing AI</h3><h4><strong>Logging:</strong></h4><p>Cloud Run integrates with Cloud Logging out of the box. You can benefit from the inbuilt logging mechanisms which also has distributed tracing capabilities to trace each HTTP request lifecycle.</p><h4><strong>Pay As You Go:</strong></h4><p>Cloud Run charges you money based on the actual usage, and scales to 0. You can set billing alerts to protect against sudden success of your AI tool.</p><h4><strong>Security:</strong></h4><p>MCP has no in-built security mechanisms. Cloud Run has support for OAuth and OIDC based security which can be leveraged by code to add a handler for security.</p><p>Even better, you can use Cloud Armor to protect against spurious traffic and to keep bad actors away.</p><h3>Conclusion</h3><p>The combination of agentic AI, Model Context Protocol (MCP), and Google Cloud Run offers a powerful platform for intelligent automation, creating a cohesive ecosystem with no operational overhead, cost efficiency (scale to 0), and the ability to handle large volumes of traffic.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db26e8760f61" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/mcp-and-agentic-ai-on-google-cloud-run-db26e8760f61">MCP and Agentic AI on Google Cloud Run</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The reason your AI project is going over budget is “SNMS”]]></title>
            <link>https://medium.com/@aviato.consulting/the-reason-your-ai-project-is-going-over-budget-is-snms-53c53609e220?source=rss-2b7d3363ba71------2</link>
            <guid isPermaLink="false">https://medium.com/p/53c53609e220</guid>
            <category><![CDATA[google-cloud]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[model-evaluation]]></category>
            <dc:creator><![CDATA[Ben King]]></dc:creator>
            <pubDate>Wed, 12 Mar 2025 22:20:33 GMT</pubDate>
            <atom:updated>2025-03-12T22:20:33.997Z</atom:updated>
            <content:encoded><![CDATA[<p>“Hey, this new deep seek model gets a 20% bump on the XYZ benchmark”</p><p>We all suffer from this, as engineers strive for optimization, even non engineers want to know if the new model can perfectly depict Chuck Norris fighting dolphins (my personal benchmark), or go deep on philosophy with us. This ends up with the whole team are ad-hoc testing models and wanting to swap them out every few days.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K0W384z211Q-dBa7" /><figcaption>Prompt: A red headed karate expert fighing dolphins (Apparently imagen is scared of recreating Chuck)</figcaption></figure><p>SNMS or “Shiny New Model Syndrome,” is derailing your AI project.</p><p>10% improvement on a benchmark is surely a good thing to have right? Sure, if you want to be top of the benchmark tables, but does this lead to an increase in Enterprise Value?</p><p>A 10% improvement on a benchmark rarely translates to a 10% improvement in actual business outcomes. In fact, it can often lead to a negative EV and significantly delay your project.</p><p>Now we have a cure for SNMS. Google Model Evaluation in Vertex AI. This service provides a framework for evaluating your models, moving beyond simple benchmark scores and focusing on how well a model meets your business goals, and can increase your EV.</p><p>With Model Evaluation you define custom evaluation metrics that are tailored to your use case, real time chatbots we want something that is low latency, building a code refactoring agent, then latency is not even going to be evaluated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/0*OU0ULloPjsLpNJ41" /><figcaption>Output speed is key with chatbots</figcaption></figure><p>Having this well structured approach lets us evaluate new models as they are released, and see if they will increase or decrease EV, giving engineers a structured way to benchmark against what the business needs, and not the ability to depict Chuck Norris fighting dolphins.</p><p>In addition if you want to start fine tuning your models, having this framework set up is key to ensure you are not wasting money on that, fine tuning models can mean you have a fixed cost to run them, instead of API calls, and when we look at the cost of API calls GPT-4 can be significantly more expensive than Gemini 2.0, but even with Gemini 2.0 Flash you will one day get to the point where hosting you own model will be cheaper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/0*hQoQHEIMnfIHYoZX" /><figcaption>Even GPT-4o is 20x Gemini Flash</figcaption></figure><p>Check out <a href="https://cloud.google.com/vertex-ai/docs/evaluation/introduction">Google Model Evaluation in Vertex AI</a> or <a href="https://aviato.consulting/contact-us/">Contact Us</a> and we can demo model evaluation one of our <a href="https://www.aviato.consulting/ai-machine-learning/">AI Services</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53c53609e220" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the AI Maze]]></title>
            <link>https://medium.com/@aviato.consulting/navigating-the-ai-maze-d8ab112e056d?source=rss-2b7d3363ba71------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8ab112e056d</guid>
            <category><![CDATA[google-cloud]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ben King]]></dc:creator>
            <pubDate>Tue, 30 Apr 2024 05:02:35 GMT</pubDate>
            <atom:updated>2024-04-30T05:02:35.173Z</atom:updated>
            <content:encoded><![CDATA[<p>The AI landscape is exploding with a dizzying array of models, from the Large Language Models (LLMs) most of us have experimented with like Llama 2 or 3 from Meta, Claude 2 from Anthropic, and Bard (now Gemini) from Google, and the original ChatGPT.</p><p>Each boasts unique capabilities, from generating different creative text formats to translating languages and answering your questions in an informative way. To make this even more confusing we have models that excel at robotics, tabular regression, image generation, or depth estimation.</p><p>While this abundance offers exciting possibilities, it also presents a significant challenge: choosing the right model for your specific needs, and doing it within your budget. The pricing models for these vary greatly Gemin 1.0 advanced, to 1.5 Ultra is a 10x cost differential.</p><p>For businesses, this creates an impossible puzzle: how do you select the optimal model without getting lost in the ever evolving AI arms race? And how do you do this within your budget?</p><p>The answer lies in flexibility. Instead of locking into a single model, businesses need an adaptable infrastructure that allows them to test their business use case against one model, evaluate the performance and then try another, without rebuilding the solution.</p><p>Additionally as new models are released, testing these to see if there is an uplift to the value of the model quickly, is going to give you the competitive advantage over others that need to rebuild their solution.</p><p>This ability to quickly swap out models offers several key benefits:</p><ul><li><strong>Optimize Performance:</strong> Different tasks require different strengths. Flexibility allows you to choose the model that delivers the best results for each specific use case, whether it’s generating marketing copy, measuring depths of objects in an image, or analyzing customer sentiment.</li><li><strong>Cost-Effectiveness: </strong>The price of models vary significantly. Flexibility empowers you to choose the most cost-effective option for your needs and budget, potentially leading to significant savings.</li><li><strong>Future-Proofing</strong>: The AI field is evolving at breakneck speed. The ability to test the newest model and evaluate the cost v performance of it is key to getting the most business value from your spend.</li></ul><p>Google Cloud’s Vertex AI provides the perfect platform for achieving this flexibility. It allows businesses to seamlessly deploy, manage, and experiment with various LLMs through a unified interface. If you are not happy with the 50 or so models they have, you can deploy one from Huggingface.co which has over 600,000 models to choose from.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*KovQK0X9l8UqyPn2" /></figure><p>The alternative solution for the non Google customers could be to write a common API, which would give you the flexibility to swap out models, or use Chat GPT which is one model that you cannot find on either Google or Huggingface.co</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/0*VA4wTJy4YURtI1T5.png" /></figure><p>Either option empowers you to leverage the strengths of different models, test each of them against your unique problems, and stay ahead of the curve in the rapidly evolving AI landscape.</p><p>Aviato Consulting, a Google Cloud Partner, specializes in helping businesses navigate the complexities of AI and implement flexible LLM solutions on Vertex AI. Our expertise ensures you harness the full potential of AI, maximizing its value for your specific business needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8ab112e056d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Securing Software Supply Chain on Google Cloud]]></title>
            <link>https://medium.com/google-cloud/securing-software-supply-chain-on-google-cloud-b7b485fa023b?source=rss-2b7d3363ba71------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7b485fa023b</guid>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[cloud-build]]></category>
            <category><![CDATA[gcp-security-operations]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Ben King]]></dc:creator>
            <pubDate>Wed, 05 Oct 2022 19:56:31 GMT</pubDate>
            <atom:updated>2022-10-21T02:21:44.160Z</atom:updated>
            <content:encoded><![CDATA[<p>This was originally written by <a href="https://www.linkedin.com/in/balesh-kumar-1bb70a23/">Balesh Kumar</a> who since (unfortunately) moved on from my team before we got the time to open source the associated code.</p><h3>Introduction</h3><p>The purpose of this document is to provide a step by step guide and related artifacts to set up a secure CI/CD pipeline for a containerized workload. While the overall ecosystem security involves multiple layers, from securing underline physical infrastructure to actual code, this document focuses on the Application/Container security that can be automated in the CI/CD pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s9QID90h6iyH0hQE" /></figure><h3>System Overview</h3><p>In this demo, we are using a simple java application (with Gradle build tool) that is containerized using Dockerfile and CI/CD pipeline is configured using cloudbuild.yaml. Other technologies/services used to implement this CI/CD pipelines are:</p><ol><li><a href="https://cloud.google.com/container-registry/docs">Cloud Source Repository</a></li><li><a href="https://cloud.google.com/artifact-registry/docs">Artifact Registry</a></li><li><a href="https://cloud.google.com/build/docs/concepts">CloudBuild</a></li><li><a href="https://cloud.google.com/container-analysis/docs/container-analysis">Container Analysis</a></li><li><a href="https://cloud.google.com/binary-authorization/docs">Binary Authorization</a></li><li><a href="https://github.com/hadolint/hadolint">Hadolint</a></li><li><a href="https://kubesec.io/">Kubesec</a></li><li><a href="https://www.conftest.dev/">Conftest</a></li><li><a href="https://www.sonarqube.org/">SonarQube</a></li></ol><h3>Cloud Build Steps:</h3><h3>Standard CI/CD Pipeline</h3><p>A standard CI/CD pipeline generally consists of 4–5 stages. For example, Build, Test, Static code Analysis and QA stage. These stages are more developer focused and It only makes sure code builds successfully, it follows general coding guidelines and it fulfills functional requirements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*obghEqG_bPXxOUXX" /></figure><p>Artifacts used in these stages are the following:</p><ul><li>Application code</li><li>Dockerfile</li><li>Container Image</li><li>Kubernetes Manifest files</li></ul><p>All of these artifacts are vulnerable to security lapses. Following is the list of vulnerabilities they can introduce.</p><p><strong>Application Code</strong></p><ul><li>SQL Injection</li><li>Cross site scripting</li><li>Code Injection</li><li>Sensitive Cookies without “HttpOnly” flag</li><li>and many more</li></ul><p>Scanning tool: <a href="https://www.sonarqube.org/">SonarQube</a></p><p><strong>Dockerfile</strong></p><ul><li>Unnecessary library and packages that increase attack surface</li><li>Unofficial base image that can be vulnerable</li><li>Image without tag make application inconsistent</li><li>and many more</li></ul><p>Scanning tool: <a href="https://github.com/hadolint/hadolint">Hadolint</a>, <a href="https://www.conftest.dev/">Confest</a></p><p><strong>Container Image</strong></p><ul><li>Buffer Overflow that can lead to Denial of service</li><li>Integer overflow that can impact confidentiality</li><li>Command Injection</li><li>and many more</li></ul><p>Scanning tool: <a href="https://cloud.google.com/container-analysis/docs/container-analysis">Container Analysis</a></p><p><strong>Kubernetes manifest file</strong></p><ul><li>Least privilege principle violation</li><li>Mutable file system increase attack surface area</li><li>No limit on CPU &amp; memory can lead to DOS via resource exhaustion</li><li>Unnecessary capability given to a container can lead to increase syscall attack surface</li></ul><p>Scanning tool: <a href="https://kubesec.io/">Kubesec</a>, <a href="https://www.conftest.dev/">Confest</a></p><h3>Secured CI/CD Pipeline</h3><p>With a little more effort, we can handle all the vulnerabilities discussed in the previous section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WNaVUK4-jZarvD2o" /></figure><h3>Demo scope &amp; output</h3><p>CI/CD pipeline for this demo consists of 11 steps and out of them 7 steps are created to make this supply chain more secure.</p><p>Build steps can be broken down into categories like <em>Static code analysis</em>, <em>Build+Push</em>, <em>Vulnerability Scanning + Attestation</em> and<em> Generate Kubernetes Manifest + Deploy</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/0*OMv25eDbHgduW1Xk" /><figcaption>Cloud Build Steps</figcaption></figure><h3>Demo setup</h3><p>First create a project, or set your existing project as default:</p><pre>gcloud projects create $PROJECT_ID –set-as-default</pre><p>Change the following variables below in <strong>bold</strong> to match your project name, region and zone:</p><pre>PROJECT_ID=<strong>sec-soft-chain</strong><br>REGION=<strong>australia-southeast1</strong><br>ZONE=<strong>australia-southeast1-b</strong></pre><p>Then run this script, comments in the script explain functionality</p><pre># Enable the Required API’s<br>gcloud services enable container.googleapis.com<br>gcloud services enable artifactregistry.googleapis.com<br>gcloud services enable ondemandscanning.googleapis.com<br>gcloud services enable cloudkms.googleapis.com<br>gcloud services enable binaryauthorization.googleapis.com</pre><pre># Create a cluster for this demo, note these settings should not be used for a production cluster <br>gcloud beta container - project $PROJECT_ID clusters create &quot;software-secure-supply&quot; - zone $ZONE - no-enable-basic-auth - cluster-version &quot;1.22.8-gke.201&quot; - release-channel &quot;regular&quot; - machine-type &quot;e2-standard-4&quot; - image-type &quot;COS_CONTAINERD&quot; - disk-type &quot;pd-standard&quot; - disk-size &quot;100&quot; - metadata disable-legacy-endpoints=true - scopes &quot;<a href="https://www.googleapis.com/auth/devstorage.read_only">https://www.googleapis.com/auth/devstorage.read_only</a>&quot;,&quot;<a href="https://www.googleapis.com/auth/logging.write">https://www.googleapis.com/auth/logging.write</a>&quot;,&quot;<a href="https://www.googleapis.com/auth/monitoring">https://www.googleapis.com/auth/monitoring</a>&quot;,&quot;<a href="https://www.googleapis.com/auth/servicecontrol">https://www.googleapis.com/auth/servicecontrol</a>&quot;,&quot;<a href="https://www.googleapis.com/auth/service.management.readonly">https://www.googleapis.com/auth/service.management.readonly</a>&quot;,&quot;<a href="https://www.googleapis.com/auth/trace.append">https://www.googleapis.com/auth/trace.append</a>&quot; - max-pods-per-node &quot;110&quot; - num-nodes &quot;2&quot; - logging=SYSTEM,WORKLOAD - monitoring=SYSTEM - enable-ip-alias - network &quot;projects/$PROJECT_ID/global/networks/default&quot; - subnetwork &quot;projects/$PROJECT_ID/regions/$REGION/subnetworks/default&quot; - no-enable-intra-node-visibility - default-max-pods-per-node &quot;110&quot; - no-enable-master-authorized-networks - addons HorizontalPodAutoscaling,HttpLoadBalancing,GcePersistentDiskCsiDriver - enable-autoupgrade - enable-autorepair - max-surge-upgrade 1 - max-unavailable-upgrade 1 - enable-shielded-nodes - node-locations $ZONE - enable-binauthz</pre><pre># Create an Artifact Repository<br>gcloud artifacts repositories create &quot;${PROJECT_ID}-repo&quot; - location=$REGION - repository-format=docker</pre><pre># Allow the Cloud Build Service Account to run scans<br>gcloud projects add-iam-policy-binding $PROJECT_ID - member=serviceAccount:$(gcloud projects describe $PROJECT_ID - format=&quot;value(projectNumber)&quot;)<a href="http://twitter.com/cloudbuild">@cloudbuild</a>.gserviceaccount.com - role=roles/ondemandscanning.admin</pre><pre># Allow the Cloud Build Service Account to deploy to GKE<br>gcloud projects add-iam-policy-binding $PROJECT_ID - member=serviceAccount:$(gcloud projects describe $PROJECT_ID - format=&quot;value(projectNumber)&quot;)<a href="http://twitter.com/cloudbuild">@cloudbuild</a>.gserviceaccount.com - role=roles/container.developer</pre><pre># Allow Cloud Build Service Account the permission to attest<br>gcloud projects add-iam-policy-binding $PROJECT_ID - member=serviceAccount:$(gcloud projects describe $PROJECT_ID - format=&quot;value(projectNumber)&quot;)<a href="http://twitter.com/cloudbuild">@cloudbuild</a>.gserviceaccount.com - role=roles/containeranalysis.notes.attacher</pre><h3>Setup Cloud Source Repository</h3><ol><li>Clone the <a href="https://github.com/GoogleCloudPlatform/cloud-build-software-delivery">demo repository</a> to your local machine and copy it to your own <a href="https://source.cloud.google.com/repo/create">Cloud Source </a>repository</li></ol><pre>git clone <a href="https://github.com/GoogleCloudPlatform/cloud-build-software-delivery">https://github.com/GoogleCloudPlatform/cloud-build-software-delivery</a></pre><p>2. Push the cloned repo to your own <a href="https://source.cloud.google.com/repo/create)">Cloud Source</a> repository:</p><pre># Create a repo called sec-soft-chain<br>gcloud source repos create sec-soft-chain</pre><pre># Configure credentials<br>gcloud init &amp;&amp; git config — global credential.<a href="https://source.developers.google.com.helper/">https://source.developers.google.com.helper</a> gcloud.sh</pre><pre># Add your new repo as a remote repo called google<br>git remote add google <a href="https://source.developers.google.com/p/$PROJECT_ID/r/sec-soft-chain">https://source.developers.google.com/p/$PROJECT_ID/r/sec-soft-chain</a></pre><pre># Push code to the new repo called google<br>git push — all google</pre><h3>Create the Cloud Build trigger</h3><ol><li>In the Cloud console, go to <a href="https://console.cloud.google.com/cloud-build/triggers">Cloud Build Triggers</a></li><li>Enable the API (if not already enabled)</li><li>Click Create Trigger</li><li>In the Trigger Settings window, enter the following details:</li></ol><ul><li>In the Name field, enter build-vulnz-deploy</li><li>For Event choose Push to a branch</li><li>In the Repository field, choose your repo from the menu</li><li>In the Branch field, enter ^main$</li><li>For Configuration, select Cloud Build configuration file (yaml or json)</li><li>In the Location, select Repository enter the default value /cloudbuild.yaml</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*RfSGZfjNH6f8emmV7LwF7A.png" /><figcaption>Cloud Build Trigger</figcaption></figure><p>5. Add the following Substitution Variable pairs:</p><ul><li>_IMAGE_NAME with the image from Cloud Source Repository</li><li>_COMPUTE_REGION with the value australia-southeast1 (or the region you chose in the beginning).</li><li>_KMS_KEYRING with the value binauthz</li><li>_KMS_LOCATION with the value australia-southeast1 (or the region you chose in the beginning).</li><li>_VULNZ_ATTESTOR with the value vulnz-attestor</li><li>_VULNZ_KMS_KEY with the value vulnz-signer</li><li>_VULNZ_KMS_KEY_VERSION with the value 1</li><li>_SONAR_LOGIN can be blank for now</li><li>_SONAR_PROJECT can be blank for now</li><li>_SONAR_ORG can be blank for now</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*-OxIJAv1nYXLhqEoDj1blQ.png" /><figcaption>Substitution Variables</figcaption></figure><p>6. Click Create.</p><p><strong>Congratulations!!!</strong></p><p>At this point you have successfully configured most of the CI/CD pipeline, excluding the two stages highlighted below, and can pause here to test the work by commenting out the final 2 stages from your cloudbuild.yaml (lines 17–24 and 62–78).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RPDAq6fctl4cgRGl" /><figcaption>Secure CI/CD Pipeline with 2 steps highlighted</figcaption></figure><p>Next we will configure:</p><ol><li>Sonarqube for static code analysis</li><li>Binary Authorization</li></ol><h3>Configure Sonarqube</h3><h4>Modify cloudbuild.yaml</h4><p>The Cloudbuild.yaml file in the repo needs a few changes these can be done manually by replacing each instance of the following variables:</p><ul><li>&lt;ZONE&gt;</li><li>&lt;REGION&gt;</li><li>&lt;REPO&gt;</li><li>&lt;IMAGE&gt;</li><li>&lt;PROJECT_ID&gt;</li></ul><p>Or the below scripts should set these as per the variables configured previously:giot</p><pre>sed -i “s/&lt;ZONE&gt;/$ZONE/g” cloudbuild.yaml<br>sed -i “s/&lt;REGION&gt;/$REGION/g” cloudbuild.yaml<br>sed -i “s/&lt;PROJECT_ID&gt;/$PROJECT_ID/g” cloudbuild.yaml<br>sed -i “s/&lt;REPO&gt;/$PROJECT_ID-repo/g” cloudbuild.yaml<br>sed -i “s/&lt;IMAGE&gt;/image/g” cloudbuild.yaml<br>&lt;PROJECT_ID&gt;/&lt;REPO&gt;/&lt;IMAGE&gt;<br>“${PROJECT_ID}-repo”<br>$REGION-docker.pkg.dev/$PROJECT_ID/$PROJECT_ID-repo/</pre><h4>Setup Sonarqube</h4><ol><li>Clone <a href="https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/sonarqube">Sonarqube Cloud Builders</a> repository</li><li>Create custom builder by running following commands:</li></ol><pre>git clone <a href="https://github.com/GoogleCloudPlatform/cloud-builders-community">https://github.com/GoogleCloudPlatform/cloud-builders-community</a><br>cd sonarqube<br>gcloud builds submit . — config=cloudbuild.yaml</pre><p>3. Configure sonarqube (we will configure sonarqube online, but you can set up your own sonarqube server and configure that too).</p><p>Login to <a href="https://sonarcloud.io/">https://sonarcloud.io</a> with your github account</p><ul><li>Create a token by navigating to Account page then click on security tab</li><li>Next we need to use the “Analyze New Project” option to set up the project in sonarcloud. Note: Use setup manually option</li><li>Note down the token you created, project key and the organization name</li><li>Update the Cloud Build Trigger Variables for:</li><li>_SONAR_ORG</li><li>_SONAR_PROJECT</li><li>_SONAR_LOGIN</li></ul><h3>Binary Authorization</h3><p>Note: Given below is the list of stripped down actions borrowed from <a href="https://cloud.google.com/architecture/binary-auth-with-cloud-build-and-gke">this</a> link</p><h4>Build and register the custom build step with Cloud Build</h4><ol><li>Clone the Google Cloud build community repo:</li></ol><pre><em>git clone </em><a href="https://github.com/GoogleCloudPlatform/gke-binary-auth-tools"><em>https://github.com/GoogleCloudPlatform/gke-binary-auth-tools</em></a><em> ~/binauthz-tools</em></pre><p>2. Configure the Binary Authorization signer for Cloud Build:<br>Before use, the code for the custom build step must be built into a container and pushed to Cloud Build. To do this, run the following commands:</p><pre><em>gcloud builds submit — project $PROJECT_ID — tag “gcr.io/$PROJECT_ID/cloudbuild-attestor” ~/binauthz-tools</em></pre><p>3. The custom build step was pushed to your current project’s Google Container Registry and is now ready for use</p><h4>Create Cloud KMS <a href="https://cloud.google.com/kms/docs/create-validate-signatures">asymmetric key</a> for signing attestations.</h4><ol><li>In Cloud Shell, create a Cloud KMS key ring named binauthz:</li></ol><pre>gcloud kms keyrings create “binauthz” \<br> — project “${PROJECT_ID}” \<br> — location “${REGION}”</pre><ol><li>Create an asymmetric Cloud KMS key named vulnz-signer which will be used to sign and verify vulnerability scan attestations:</li></ol><pre>gcloud kms keys create “vulnz-signer” \<br> — project “${PROJECT_ID}” \<br> — location “${REGION}” \<br> — keyring “binauthz” \<br> — purpose “asymmetric-signing” \<br> — default-algorithm “rsa-sign-pkcs1–4096-sha512”</pre><h4>Create a Container Analysis note named vulnz-note</h4><pre>curl “<a href="https://containeranalysis.googleapis.com/v1/projects/$%7BPROJECT_ID%7D/notes/?noteId=vulnz-note">https://containeranalysis.googleapis.com/v1/projects/${PROJECT_ID}/notes/?noteId=vulnz-note</a>&quot; \<br> — request “POST” \<br> — header “Content-Type: application/json” \<br> — header “Authorization: Bearer $(gcloud auth print-access-token)” \<br> — header “X-Goog-User-Project: ${PROJECT_ID}” \<br> — data-binary @- &lt;&lt;EOF<br>{<br>“name”: “projects/${PROJECT_ID}/notes/vulnz-note”,<br>“attestation”: {<br>“hint”: {<br>“human_readable_name”: “Vulnerability scan note”<br>}<br>}<br>}<br>EOF</pre><h4>Give CloudBuild Service account name permission on note</h4><pre>​​curl “<a href="https://containeranalysis.googleapis.com/v1/projects/$%7BPROJECT_ID%7D/notes/vulnz-note:setIamPolicy">https://containeranalysis.googleapis.com/v1/projects/${PROJECT_ID}/notes/vulnz-note:setIamPolicy</a>&quot; \<br> — request POST \<br> — header “Content-Type: application/json” \<br> — header “Authorization: Bearer $(gcloud auth print-access-token)” \<br> — header “X-Goog-User-Project: ${PROJECT_ID}” \<br> — data-binary @- &lt;&lt;EOF<br>{<br>“resource”: “projects/${PROJECT_ID}/notes/vulnz-note”,<br>“policy”: {<br>“bindings”: [<br>{<br>“role”: “roles/containeranalysis.notes.occurrences.viewer”,<br>“members”: [<br>“serviceAccount:${CLOUD_BUILD_SA_EMAIL}”<br>]<br>},<br>{<br>“role”: “roles/containeranalysis.notes.attacher”,<br>“members”: [<br>“serviceAccount:${CLOUD_BUILD_SA_EMAIL}”<br>]<br>}<br>]<br>}<br>}<br>EOF</pre><h4>Create the vulnerability scan attestor:</h4><pre>gcloud container binauthz attestors create “vulnz-attestor” \<br> — project “${PROJECT_ID}” \<br> — attestation-authority-note-project “${PROJECT_ID}” \<br> — attestation-authority-note “vulnz-note” \<br> — description “Vulnerability scan attestor”</pre><h4>Add the public key for the attestor’s signing key:</h4><pre>gcloud beta container binauthz attestors public-keys add \<br> — project “${PROJECT_ID}” \<br> — attestor “vulnz-attestor” \<br> — keyversion “1” \<br> — keyversion-key “vulnz-signer” \<br> — keyversion-keyring “binauthz” \<br> — keyversion-location “${REGION}” \<br> — keyversion-project “${PROJECT_ID}”</pre><h4>Grant the Cloud Build service account permission to verify attestations made by vulnz-attestor:</h4><pre>gcloud container binauthz attestors add-iam-policy-binding “vulnz-attestor” \<br> — project “${PROJECT_ID}” \<br> — member=serviceAccount:$(gcloud projects describe $PROJECT_ID — format=”value(projectNumber)”)<a href="http://twitter.com/cloudbuild">@cloudbuild</a>.gserviceaccount.com \<br> — role “roles/binaryauthorization.attestorsViewer”</pre><h4>Grant the Cloud Build service account permission to sign objects using the vulnz-signer key:</h4><pre>gcloud kms keys add-iam-policy-binding “vulnz-signer” \<br> — project “${PROJECT_ID}” \<br> — location “${REGION}” \<br> — keyring “binauthz” \<br> — member serviceAccount:$(gcloud projects describe $PROJECT_ID — format=”value(projectNumber)”)<a href="http://twitter.com/cloudbuild">@cloudbuild</a>.gserviceaccount.com \<br> — role ‘roles/cloudkms.signerVerifier’</pre><h4>Configure Binary Authorization default policy like below</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*zqwg_ydiDr9xMY0EqEVrUQ.png" /><figcaption>Binary Authorization Policy</figcaption></figure><h3>Further considerations</h3><ol><li>Purpose built OS images</li><li>Reducing Attack surface (limiting/disabling Port/user/services/packages)</li><li>Kernel Hardening with AppArmor &amp; Seccomp</li><li>Behavioural Analytics/Falco</li><li>Managing Secrets</li><li>Regular Upgrade</li><li>Restrict API Server Access</li><li>Least privilege access using RBAC</li><li>Exercise caution with Service accounts</li><li>CIS Benchmarking</li><li>Secure Dashboard</li><li>Secure Ingress to Cluster</li><li>Network Policies</li><li>Auditing</li></ol><p>Note, this was also posted on my <a href="https://www.aviato.consulting/blog/">blog</a> and instructions in the <a href="https://github.com/GoogleCloudPlatform/cloud-build-software-delivery">github</a> readme</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7b485fa023b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/securing-software-supply-chain-on-google-cloud-b7b485fa023b">Securing Software Supply Chain on Google Cloud</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Egressing from Google Bare Metal Solution]]></title>
            <link>https://medium.com/google-cloud/egressing-from-google-bare-metal-solution-aa459389436c?source=rss-2b7d3363ba71------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa459389436c</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[google-bare-metal]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[baremetal]]></category>
            <dc:creator><![CDATA[Ben King]]></dc:creator>
            <pubDate>Tue, 07 Dec 2021 05:19:38 GMT</pubDate>
            <atom:updated>2021-12-17T21:35:37.512Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Google Cloud Bare Metal Solution (BMS) does not come with access to Google Cloud services, networking services, or the internet by design, this ensures that critical workloads such as Oracle workloads are as secure as possible.</p><p>The downside to this is that there is often a need for a subset of applications to egress to the internet. Typically on GCP we would solve a problem like this with Cloud NAT, however BMS is not compatible with Cloud NAT as the traffic does not originate in the Virtual Private Cloud (VPC).</p><p>Some options for accessing the internet include:</p><ul><li>Routing outgoing traffic through a NAT gateway running on GCE</li><li>Routing traffic through a Compute Engine VM that serves as a proxy server</li><li>Routing traffic through Cloud VPN or Dedicated Interconnect to on-premises gateways to the internet</li></ul><p>This document covers off the first option above of using a NAT Gateway to route traffic to the internet. Other options under consideration have been mapped out <a href="https://lendleasegroup.atlassian.net/wiki/spaces/GCP21/pages/222915101099/BMS+Egress+Routing+Options?focusedCommentId=222915167087#comment-222915167087">here</a>. This document assumes a working knowledge of GCP and networking.</p><h3>Solution 1: NAT Gateway on GCE</h3><p>The use of Network Address Translation (NAT) to translate any internal IP’s to public IP’s can be done by NAT’ing the traffic using iptables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/0*Q-ES-UZIYoPtT3bD" /><figcaption>Simple NAT on GCE</figcaption></figure><p>Now if we run critical workloads on BMS, having a single GCE instance would create a single point of failure, to be able to withstand a zonal failure running a Managed Instance Group (MIG) and fronting this MIG with an Internal Load Balancer (ILB) allows for a highly available solution that is resilient to zonal failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BDxkAK45qNKcr4rp" /><figcaption>NAT Managed Instance Group on GCE</figcaption></figure><p>The next step is to configure routes for this traffic, in GCP this is simple we can create a tag, however as BMS traffic does not originate in the VPC we cannot. The work around we can apply here is to tag all other traffic in the VPC (shown below is “tagged”), and create a route for untagged traffic to egress via the Load Balancer and NAT VM solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pLZYjXeZGqdf4sPM" /><figcaption>Tagging for NAT MIG</figcaption></figure><p>The only downside to this solution now is that any untagged traffic will need to egress through this solution, which presents 3 issues:</p><ol><li>It is time consuming to tag, and ensure all future resources are tagged at creation time</li><li>Any traffic that is untagged in the entire VPC will egress via this solution, this includes traffic from other regions incurring latency and cost</li><li>In the even BMS is failed over to a DR region, the route will need to be updated for untagged traffic to egress via a different ILB</li></ol><h3>Solution 2: Proxy on GCE</h3><p>The use of a proxy running on GCE such as Squid Proxy could also work, this would require setting the http_proxy on any servers running in the BMS extension, but this option makes the tagging and routing simpler.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/0*fvbHTEStt5to-RbR" /><figcaption>Squid Proxy on GCE</figcaption></figure><p>Again having a single point of failure is undesirable, but as above is solved with a MIG, and an ILB as shown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jIoli2fkp-6xfY1P" /><figcaption>Squid Proxy on GCE MIG with ILB</figcaption></figure><p>The downside to this approach is making sure that the proxy config is set on each VM in BMS, and that any applications support the use of a proxy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa459389436c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/egressing-from-google-bare-metal-solution-aa459389436c">Egressing from Google Bare Metal Solution</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>