Google shipped a model that cannot write a sentence. On our runs it answered a yes/no question in 456 ms from pune (India) where an LLM took seconds. But the finding that actually changes your bill is this: asking twenty questions in one request cost 18.6× fewer tokens than asking them one at a time.

On 11 November 2026 Google released the Gemini Decisions API and the model behind it, gemini-decision-1. It does not write. You hand it some content and a set of questions, and it returns a decision: yes/no with a probability, one of up to 255 choices, or a score against a rubric. Output tokens are always zero.

If you want the longer story of why this class of model exists — the psychology, the decade AI spent building only the slow kind — that is a separate post: From System 1 and System 2 to Decision Models. Here I am assuming you already accept the idea and want to know whether Google’s version holds up.

1. The idea in plain words

An LLM answers an essay question. You ask it something, and it writes you a paragraph. If you only wanted a yes or a no, you still paid for the paragraph — in money, and in the seconds it took to generate.

A decision model fills in an answer sheet. The boxes are fixed before the model runs: this question is yes/no, that one is a choice between four teams, the third is a score from 0 to 2. The model’s job is to put a probability on each box. There is no paragraph, because there is nowhere to write one.

That constraint is the whole product. It is why the latency is in milliseconds and why the output price is zero — not a discount, but an absence.

2. The problem it is actually for

The place this bites in production is not chat. It is the quiet classification layer nobody demos: is this ticket about billing, is this document signed, is this review a complaint, does this message need a human.

Those are not conversations. They are judgements that happen a few million times a month, where you need a number you can threshold on, and where an LLM is the wrong shape of tool — you are paying generation cost for a decision, and taking generation latency on a request path.

3. What Google shipped

From the launch post and the model documentation:

4. Getting it running

To get started you need is Python 3.11+ and a Gemini API Key from Google AI Studio.

pip install "google-genai>=3.2.0"
export GEMINI_API_KEY="YOUR_GEMINI_API_KEY"

The SDK path is short enough to reproduce in full. One client, one call, one answer.

from google import genai
client = genai.Client(http_options={"api_version": "v1beta"})
result = client.decisions.create(
 model="gemini-decision-1",
 input="The blue mug I ordered on Monday arrived cracked on Wednesday.",
 questions={
 "is_damaged": {
 "type": "noul",
 "instructions": "Did the item arrive damaged?",
 },
 },
)
answer = result.answers["is_damaged"]
print(f"P(yes)={answer.noul:.2f} confidence={answer.confidence:.2f}")

The first call. One question, one probability, no prose.

Two things that are easy to get wrong, and that cost me time:

confidence is exactly the top probability. Not a calibrated score, not a transformation - the same float. I confirmed it on live responses:

Option order genuinely does not matter. The API sorts your option keys alphabetically before scoring. I shuffled three options six ways on the same ticket: the top answer was billing 6 times out of 6, at probability 0.997191 every single time - a spread of exactly zero. That is a real design win, and not universal in this class of model.

One consequence nobody mentions: if you name your catch-all option none_of_the_above, it sorts into the middle of your real options. Google's docs say to call it other. Do that.

5. Latency: the sub-second story is a text story

Twenty back-to-back calls, one choice question, text input, measured from Pune over the public internet on the free tier:

Google claims 150–250 ms end to end. I measured roughly 2× that. I am not calling the claim wrong — I am 7,000 km from the nearest likely serving region on a home connection, and Google does not say where they measured. State where you measured from, always.

Against an LLM on the same question, the gap is the point. The same race — route one support ticket, gemini-decision-1 against an LLM - run four times back to back:

Then i tested from US region ( VM in us-central1) and could see same result as google claimed latency 150–250 ms end to end.

Then compared from same VM in us-central1 using gemini-3.5-flash

sample script to compare LLM vs decision model

import time

from google import genai

client = genai.Client(http_options={"api_version": "v1beta"})

ticket = "I was charged twice for my October invoice. Please refund one of them."
teams = {"billing": "Invoices, refunds, charges",
         "technical": "Bugs, outages, errors",
         "general": "Questions and feature requests"}

# 1. Ask an LLM to write the answer.
start = time.time()
interaction = client.interactions.create(
    model="gemini-3.5-flash-lite",
    input=f"Which team should handle this ticket? Answer with one word: "
          f"billing, technical or general.\n\n{ticket}",
)
llm_latency = time.time() - start

# 2. Ask the decision model to pick the answer.
start = time.time()
response = client.decisions.create(
    model="gemini-decision-1",
    input=ticket,
    questions={"team": {"type": "choice",
                        "instructions": "Which team should handle this ticket?",
                        "criteria": teams}},
)
decision_latency = time.time() - start

team = response.answers["team"]
print(f"LLM      : {interaction.output_text.strip():<10}  {llm_latency:.2f} seconds")
print(f"Decision : {team.choice:<10}  {decision_latency:.2f} seconds  (confidence {team.confidence:.2f})")
print(f"\nThe decision model was {llm_latency / decision_latency:.1f}x faster.")
print(f"Output tokens: LLM wrote one word, decision model wrote {response.usage.total_output_tokens}.")

Three consecutive runs of the same race from the recording: 8.9x, 12.7x, 9.3x. A fourth run later in the session came out at 6.7x. Same question every time.

Quote the spread, not the best number. On a home connection the LLM wandered between 3.93 s and 5.16 s and the decision model between 0.32 s and 0.59 s, which is how you get a ratio that moves from 6.7× to 12.7× on an unchanged question. The useful claim is “roughly 7× to 13× on my runs”, not “12.7× faster”.

One line in that output does not vary at all, and it is the one that matters for the bill: the LLM wrote a single word — “billing” — and was charged for generating it. The decision model wrote zero output tokens, four times out of four.

The more useful finding is what happens when the input stops being text:

Round trip from Pune on a home connection, free tier, 2026–11–13. Sub-second is a text property: audio and video cost 3–7x.

Audio and video cost 3–7× the latency of text. Every “real-time video decisions” idea you are about to have needs a timeout and a fallback before it goes anywhere near a request path.

On accuracy across those modalities, the model did well against known ground truth — 7 of 7 correct. My favourite result is the one that is hardest to fake: asked how prepared a speaker sounded, it put 0.875 on “hesitant, searching for notes” for a file Google had named rehearsing.wav. No transcription step, and no transcription line on the bill.

One billing detail that is not in the docs’ modality table: a video’s soundtrack is billed separately as audio tokens. Our video call shows image tokens and audio tokens for a single file.

6. The cost finding: your bill is set by batching, not by the price per token

The pricing page says $0.10 per 1,000,000 input tokens, and output tokens are always zero. That is the whole price list. It still tells you almost nothing about your bill, and this section is why.

First, what one decision costs

Start with the simplest case: one 343-token incident report, one yes/no question about it.

A single decision costs about four thousandths of a cent. That is why every chart in this section is priced per million — per decision, the bars would all read $0.00.

Now the part that matters

Ask twenty questions about the same report, in one request:

Same report, same twenty questions, same answers. $40.90 against $6.59 per million — 6.2x cheaper, decided entirely by the shape of the request.

A 343-token incident report, 3 runs per bar, measured 2026–11–14. The shared input is billed once per request, so each extra question adds only ~47.8 tokens of its own.

The one line doing the work is billed ONCE. The document is the expensive part, and you pay for it once per request however many questions you attach. Each extra question adds only its own text — I measured that at 47.8 input tokens, on both documents I tested.

So the rule of thumb, which is worth keeping in your head:

It gets more extreme the bigger the document

The saving scales with how much input you are re-sending. The same twenty questions against a 12,272-token contract:

Twenty decisions on a 12,272-token contract (66,304 characters). Same model, same questions, same answers: 245,440 input tokens one at a time against 13,181 batched, 18.6x apart.

6.2x on a small input, 18.6x on a large one. The ratio is roughly shared input ÷ 48, so it keeps climbing the more context your questions share.

Take the ratio, not the dollar amount

The absolute cost here is a property of my documents, not of the API. A 343-token incident report and a 12,272-token contract give 6.2x and 18.6x; yours will land somewhere else. What transfers is the shape:

Measure your own input once, and that formula tells you the rest. Both documents here agreed on the marginal cost of a question to within two tokens, so 48 is a safe number to plan with.

Two limits that will spoil the plan

There is no Batch API for this model and no context caching. If you were going to score a million rows overnight at batch prices, you cannot, and repeated input is charged in full every single time. The batching above is the only lever you get.

7. Where it breaks

The forced-choice trap. The probability is only ever spread across the options you offered. Give it three support teams and it will confidently file things that are not support tickets at all:

A team lunch classified as a general support ticket at 0.92 confidence. One extra option fixes all four. This is not a model defect — it is what “probabilities sum to 1” means — but it will reach production if nobody tests off-distribution inputs.

A threshold you did not tune is not a safety mechanism. Google’s own documentation example uses a 0.7 confidence gate. Running it, “It’s not working.” — a message that says nothing at all — cleared that gate at 0.79 and auto-routed. Google’s docs are honest about this and tell you to set thresholds from labelled examples of your own traffic. That advice is correct, it is work, and most people will skip it.

Google’s own documented 0.7 gate, passing a message that says nothing.

The free tier’s binding limit is tokens, not requests. It advertises 200 requests per minute, which sounds generous, but the real ceiling is 50,000 input tokens per minute. At 8k tokens a call that is about six calls a minute. Three 16k-token document calls will rate-limit you:

The free tier’s real ceiling is tokens per minute, not requests per minute.

Worth noting for error handling: rate limits return 429, while the documentation’s Limits section describes 400 for limit violations. Both are true — 400 is for request-shape limits, 429 for rate limits.

8. Is it the cheapest? No. That is not why you would pick it

On price per token for plain text, Google is not the cheapest decision model. At $0.10 per million input tokens it is roughly 2.5× the specialists — Jev sits near $0.042, Sarvam Prism is in the same neighbourhood.

What you get for the difference is modality. If your decision needs to look at an image, listen to audio, watch a video or read a PDF in the same request, Google is today the clear answer, and there is no separate OCR or transcription line on the bill. If you only ever classify plain text and you are price-sensitive at volume, the specialists are cheaper. If you need Indian languages or open weights, Prism. If the data cannot leave your network, something you can run locally.

The honest summary: the differentiator is not the price, it is the modality.

9. Two places I actually put it, and what happened

Everything above is the model on a bench. The rest of this series puts it on a request path and measures what changed. Both results are here in short; the posts have the method, the failures and the code.

Part 2 — replacing the router in a Google ADK multi-agent system

A root agent whose only job on a normal turn is to pick one of three sub-agents. That is a classification, but the default ADK pattern pays a generative model to do it on every turn, before any real work starts.

81% less routing latency, 35.7x cheaper, same routing decisions — about 1.5 seconds of pure overhead removed from every turn. The clearest evidence is not a number: the ADK trace for the decision-routed agent is empty, because the router model is never called at all.

Part 2: the same message to both agents. Top, the default ADK pattern spends a model call on routing. Bottom, with the decision model there is no model span at all.

Measured as a whole turn through the ADK framework, the same message takes 1,492 ms with the router model and 497 ms with the decision model — 67% less, on five runs each:

Part 2: one full ADK turn on each agent, five runs each. The ranges do not overlap.

Be careful with that as a headline, though. Routing is a fixed tax, so what it is worth depends on the turn: 69% of a short one, around 2% of a turn where the sub-agent then runs a 60-second investigation. Part 2 has the callback, the four traps that cost me an afternoon, and why “my agent is 5x faster” is the wrong claim.

Part 3 — as an eval judge, against an LLM judge

Same 10 HaluEval rows, same prompt, five runs each, asking whether a response is grounded.

The generative judge scored 70% on one run and 90% on another over identical rows. The decision model returned a bit-identical probability every time — not the same side of the threshold, the same float. 4.5x faster, 45x cheaper, and it tells you which rows it is unsure about in the first call instead of after you have paid to run the suite five times.

Part 3: five runs of the same judge over the same ten rows. The generative judge moves by 20 points on rows that never changed.

Part 3 has the stability test, the threshold sweep an LLM judge structurally cannot give you, and the thing you give up: the rationale.

10. Key findings

1. Batching sets your bill, not the price per token. The same twenty decisions on a 12,272-token document cost $1,227.20 per million one at a time and $65.91 batched — 18.6× fewer tokens. On a 343-token input the same move is 6.2×.
2. The marginal cost of one more question is ~48 input tokens, on both documents tested. Tokens per decision = (shared input ÷ questions) + 48.
3. Sub-second is a text property. Audio and video round-trips were 3.4 s, 3–7× text.
4. We measured ~2× Google’s latency claim from Pune on a home connection — 456 ms median against a 150–250 ms claim. Say where you measured.
5. confidence is exactly the top probability. Never port a threshold from another decision model.
6. Option order does not affect the answer — zero spread across six shuffles. Name your catch-all other.
7. Always offer an escape option. Without one, a team-lunch message was filed as a support ticket at 0.92.
8. The free tier’s real limit is 50,000 tokens per minute, not 200 requests per minute.
9. No Batch API, no context caching. Plan your overnight scoring job around that, or around a different model.
10. On a real request path it held up. Routing in an ADK multi-agent system: 1,851 ms → 349 ms, 35.7× cheaper, same decisions (part 2). As an eval judge: bit-identical on every repeat where the LLM judge swung 70–90% on the same rows, 45× cheaper (part 3).

References

• Gemini Decisions API launch post, 11 November 2026 — https://blog.google/innovation-and-ai/technology/developers-tools/gemini-decisions-api/
• Decisions API documentation — https://aistudio.google.com/docs/decisions-api
• gemini-decision-1 model card - https://aistudio.google.com/docs/models/gemini-decision-1
• Pricing, checked 14 November 2026 — https://aistudio.google.com/docs/pricing
• Interactions API overview — https://aistudio.google.com/docs/interactions-overview

Measurements: gemini-decision-1 via google-genai 3.3.0, from Pune, India, over the public internet on the free tier, 2026-11-13 and 2026-11-14. Cost figures use 3 runs per row and the input-token counts the API itself reports; a warm-up call is discarded before any timing. These are our numbers on our network, not a benchmark.

Looking for a similar implementation, or for assistance with your Cloud and AI journey? Talk to TechTrapture or email vishal.bulbule@techtrapture.com.

Gemini Decisions API: A New Approach to Fast and Scalable AI Decisions was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Vishal Bulbule · Google Developer Expert (Google Cloud) · TechTrapture

Total tokens/s as users grow, same L4 GPU and same Gemma 4 E2B model. From the run recorded for the video (1 user: Ollama 109.9 vs vLLM 47.1 tok/s); the tables below use our dry run from the same day, and throughput differed by under 3%.

Abstract

We compared two popular open-source inference engines, Ollama and vLLM, serving the same model ( Gemma 4 E2B) on identical Cloud Run services with one NVIDIA L4 GPU each. A closed-loop load generator ran 1, 4, 8, 16 and 32 concurrent users for 60 seconds per level, and we measured total throughput, time to first token (TTFT), cost per million output tokens, GPU utilization, and autoscaling behaviour beyond one instance.

For a single user, Ollama (4-bit) generated 113 tokens/s versus vLLM’s 47 tokens/s (BF16). From 4 users upward vLLM overtook it. At 32 users vLLM reached 1,235 tokens/s with a median TTFT of 0.47 s, while Ollama stayed flat at about 160 tokens/s and its median TTFT grew to 48 s. At the same $1.42/hour, the cost per million output tokens was $0.32 (vLLM) versus $2.47 (Ollama). Cloud Run’s GPU utilization metric read 79–83% for Ollama, despite 7.7× lower throughput. Under load beyond one instance, Cloud Run started a second GPU instance before the first was full; the new instance took 6 min 21 s to become ready and was billed throughout. We discuss engine selection, metric pitfalls, and three cost strategies for the scale-out gap.

Phase 1 compared Ollama’s default (one request at a time) against vLLM tuned to 32 sequences. After readers pointed out that Ollama’s TTFT looked like queueing, we ran a phase 2 with OLLAMA_NUM_PARALLEL=32 and everything else unchanged. Ollama's median TTFT at 32 users fell from 48 s to 1.5 s and its throughput more than doubled, to 359 tokens/s, but vLLM still delivered 3.4× more tokens per GPU ($0.32 versus $1.10 per million). Phase 2 also showed that Ollama 0.40.13 runs speculative decoding by default for Gemma 4, which vLLM did not, and that Cloud Run routed users to a new Ollama instance about 4 minutes before its model was loaded, giving a p95 TTFT of 274 s.

1. Introduction

Most guides to running vLLM on Cloud Run show how to deploy it: Google’s own documentation, its codelab, community posts and templates. Very few measure what happens under concurrent load, and none that we found compare vLLM against Ollama on the same hardware or examine what Cloud Run does when one GPU is no longer enough.

That gap matters for cost. On a GPU, the engine decides how many useful tokens you get per GPU-hour. On Cloud Run, autoscaling decides how many GPU-hours you pay for.

This study answers four questions:

1. RQ1: How do Ollama and vLLM compare in throughput and latency as concurrent users grow from 1 to 32 on one L4?
2. RQ2: What does that mean for cost per million tokens?
3. RQ3: Does Cloud Run’s GPU utilization metric reflect how efficiently the GPU is used?
4. RQ4: What happens, and what does it cost, when load exceeds one instance?

2. Background

vLLM is an open-source engine that serves LLMs to many users at once, with an OpenAI-compatible API, under Apache 2.0. It came out of UC Berkeley’s Sky Computing Lab, and its core technique, PagedAttention, was published at SOSP 2023. Two ideas matter for this study:

• Continuous batching: requests join and leave the running batch at every generation step, so many users share each pass over the model’s weights.
• PagedAttention: the KV cache, the per-conversation working memory, is allocated in small pages as it grows, instead of one large block reserved per request. Far more conversations fit in the same GPU memory.

Ollama is built for running models locally and simply, on top of llama.cpp with 4-bit GGUF models. By default it processes one request at a time per model ( OLLAMA_NUM_PARALLEL=1) and queues the rest.

Why one user is different from many. Generating one token at a time is limited mainly by how fast the GPU can read the model’s weights from memory, not by arithmetic. A smaller (4-bit) model is faster for a single user. Batching many users into the same step reuses each weight read for all of them, which is where a serving engine pulls ahead.

Gemma 4 E2B. “E2B” means effective 2 billion parameters: 2.3B effective, 5.1B including embeddings (model card). In BF16 the weights occupy about 9 GB of GPU memory. The model is Apache 2.0 and not gated on Hugging Face.

3. Experimental setup

3.1 Platform

Price breakdown, europe-west1 list prices, 2 Nov 2026: L4 without zonal redundancy at $0.0001867/s, plus 8 vCPU × $0.000018/s, plus 32 GiB × $0.000002/s = $0.0003947/s ≈ $1.42/hour.

The phase 1 benchmark output. Red: Ollama at 32 users, 157 tok/s and a 48.8 s median wait. Green: vLLM at 32 users, 1,235 tok/s and 0.47 s.

3.2 vLLM deployment

gcloud run deploy vllm-gpu \
 --image vllm/vllm-openai:v0.33.0 \
 --region europe-west1 \
 --cpu 8 --memory 32Gi \
 --gpu 1 --gpu-type nvidia-l4 --no-gpu-zonal-redundancy \
 --no-cpu-throttling \
 --concurrency 32 --min-instances 0 --max-instances 1 \
 --timeout 600 --port 8080 \
 --startup-probe httpGet.path=/health,httpGet.port=8080,initialDelaySeconds=240,periodSeconds=10,failureThreshold=120,timeoutSeconds=5 \
 --command vllm \
 --args="serve,google/gemma-4-E2B-it,--dtype=bfloat16,--max-model-len=4096,--max-num-seqs=32,--gpu-memory-utilization=0.90,--limit-mm-per-prompt.image=0,--limit-mm-per-prompt.audio=0,--limit-mm-per-prompt.video=0,--enable-prefix-caching,--enable-chunked-prefill,--host=0.0.0.0,--port=8080"

The real deployment: the vLLM arguments in our script, and gcloud confirming the service is deployed and serving traffic.

Key choices:

• --concurrency 32 matches --max-num-seqs=32, following Google's guidance.
• Image, audio and video inputs are disabled because this is a text-only API. With only image and audio disabled, vLLM still profiled a video input at startup.
• The startup probe checks /health, which passes only once the model is loaded. It allows far more time than the 240 s used in Google's example, because cold start here was about 7 minutes.
• vLLM downloads the weights from Hugging Face at every cold start. No token is needed because the model is not gated.

vLLM’s own startup log: room in the KV cache for ~90 concurrent 4,096-token conversations.

The first question to the deployed vLLM service: “What is 17*23? Answer briefly.” The answer, 391, is in the content field.

3.3 Load generator and method

We used a closed-loop benchmark written in standard-library Python:

• Levels: 1, 4, 8, 16, 32 concurrent users. Each user sends a request, waits for the full streamed answer, and immediately sends the next. Each level runs for 60 seconds, and requests in flight at the deadline are allowed to finish.
• Requests: the same OpenAI-compatible endpoint for both engines (/v1/chat/completions, streaming). Eight different long-answer prompts rotate, so no single cached prompt favours either engine. temperature = 0 and max_tokens = 256; every request generated exactly 256 tokens.
• Metrics:
• Total tokens/s: all completion tokens divided by wall time, i.e. throughput for everyone.
• TTFT: client-side, time from sending the request to the first content token. Median and 95th percentile reported.
• Per-user decode speed: how fast an individual answer streams once it starts.
• Cost per 1M output tokens: $1.42/h ÷ (total tokens/s × 3600) × 1⁰⁶, assuming the GPU is kept busy.
• Warm-up: one request per engine before measuring, excluded from results.
• Client: a laptop in Pune, India, calling europe-west1. Network round-trip time is included in TTFT for both engines.

The eight questions the benchmark rotated through, from bench_concurrency.py. All of them ask for long answers, so every request does the same amount of work.

3.4 Fairness controls

• Thinking mode off on both engines. Gemma 4 can reason before answering, which adds hidden output tokens. vLLM: "chat_template_kwargs": {"enable_thinking": false}. Ollama on its OpenAI-compatible endpoint thinks by default and ignored "think": false (about 250 reasoning tokens still produced). Only "reasoning_effort": "none" disabled it. Without this control, Ollama's numbers would have included its thinking.
• Same context length (4,096) on both engines, same instance shape, same price.
• Precision is not equal, by design. Each engine runs its common default: Ollama 4-bit, vLLM BF16. This favours Ollama per token at low load. See Limitations.
• Speculative decoding is not equal either. We found this only from the phase 2 logs. Ollama 0.40.13 loads Gemma 4’s multi-token-prediction “assistant” head as a draft model and decodes speculatively by default (draft acceptance ≈ 0.41, mean len ≈ 2.2). It did so in both phases. vLLM ran without speculative decoding. This also favours Ollama at one user.
• Phase 1 is defaults versus tuned. In phase 1, vLLM was set to 32 sequences to match Cloud Run’s concurrency, but Ollama kept its default of one request at a time. Phase 2 (Section 4.6) gives Ollama 32 slots too.

Watch the walkthrough: vLLM on Cloud Run

4. Results

4.1 Throughput and latency (RQ1)

Per-user decode speed: Ollama about 165 tok/s at every level, but only one user at a time; vLLM 52 tok/s at 1 user and 43 tok/s at 32.

Total throughput. vLLM overtakes Ollama between 1 and 4 users and reaches 1,235 tok/s at 32.

Median time to first token, log scale. Ollama’s requests queue; vLLM’s batch.Observations:

1. Single user: Ollama wins by 2.4×. With one request in flight, the GPU’s work is dominated by reading weights. Ollama’s 4-bit model is roughly a quarter the size of vLLM’s BF16 model, so each token is cheaper to produce.
2. Crossover at 4 users. vLLM’s continuous batching overtakes Ollama between 1 and 4 users.
3. Ollama saturates immediately. Throughput stays at about 160 tok/s from 4 to 32 users because requests are served one at a time. Added users queue: median TTFT grows linearly with load, reaching 48 seconds at 32 users.
4. vLLM scales almost linearly to 1,235 tok/s, a 7.7× advantage, while TTFT stays under 0.5 s at the median. Per-user speed falls only from 52 to 43 tok/s.
5. Headroom remains. vLLM reported room for about 90 concurrent 4,096-token conversations in its KV cache. At 32 users, compute rather than memory was the limit, and per-user speed was still high. The 32-request cap was a configuration choice, not the hardware ceiling.

The benchmark running live: Ollama finished, with its 32-user line highlighted (157.1 tok/s, 48.8 s median wait), and vLLM warming up. The tables use our dry run from the same day, so their values differ slightly.

4.2 Cost per million output tokens (RQ2)

Cost per 1M output tokens at $1.42/h with the GPU kept busy.

At one user, Ollama is cheaper per token. From four users, vLLM is cheaper, and at 32 users it is 7.7× cheaper on identical hardware at an identical hourly price. The choice of inference engine is a cost decision as much as a technical one.

4.3 GPU utilization is not GPU efficiency (RQ3)

We pulled Cloud Run’s GPU metrics ( container/gpu/utilizations, container/gpu/memory_usages) for the benchmark windows.

Cloud Monitoring, container/gpu/utilizations, during the race. A GPU can look 81% busy while delivering 7.7× fewer tokens.

Ollama’s GPU looked 82% busy while producing 7.7× fewer tokens than vLLM. GPU utilization measures the fraction of time the GPU is running anything, not how much of its capacity is used. A GPU serving one request at a time can look nearly full.

Memory tells a similar story in reverse. vLLM reserves 70–90% of GPU memory at startup, even with zero users, because of --gpu-memory-utilization=0.90 and its KV-cache pool. Ollama used 3.5 GB, leaving about 20 GB idle.

Implication: neither GPU utilization nor GPU memory percentage is a reliable signal of efficiency or load for LLM serving. Measure tokens per second per dollar. This also explains Google’s documented design choice: “Cloud Run does not automatically scale the number of instances based on GPU utilization”. It scales on request concurrency instead. In our data, GPU utilization would have been a misleading scaling signal.

4.4 Cold start

Ollama’s cold start, with its 4-bit model read from a mounted Cloud Storage bucket, was about 4.5 minutes to the first answer, followed by a one-off stall of about 25 s on the next request. Both services scaled to zero about 10 minutes after their last request. Hugging Face once answered HTTP 429 to the anonymous download; vLLM retried successfully.

4.5 Beyond one instance: the scale-out gap (RQ4)

We raised vLLM’s --max-instances to 2 and ran 32 users, then 48 users, for 5 minutes each.

Cloud Monitoring, container/instance_count (active), vLLM with — max-instances 2.

Cloud Monitoring, GPU utilization per instance: the second instance sits near 0% for minutes while billed.

GPU metrics confirm the gap. While the second instance booted, the 99th-percentile GPU utilization across instances was 100% (the busy instance) and the median was about 1% (the booting one). For 6 minutes, one GPU was saturated while a second, billed GPU produced nothing. Instance-based billing charges from instance start. Users on the saturated instance saw p95 TTFT rise from 0.8 s to 5.8 s. No requests were rejected.

Cloud Run logs filtered on “starting new instance”: each entry gives the reason, so you can separate instances started by autoscaling from those started by a deployment.

4.6 Phase 2: Ollama with 32 parallel slots

Readers pointed out that Ollama’s TTFT in Section 4.1 looked like queueing, not generation. That was correct: with one slot, Cloud Run let 32 requests into the instance and 31 of them waited inside Ollama. We redeployed Ollama with OLLAMA_NUM_PARALLEL=32. Everything else was unchanged: image, model, bucket, instance shape, concurrency, context and prompts. The startup log confirmed the setup: offloaded 36/36 layers to GPU, n_seq_max = 32, n_ctx_seq = 4096, KV cache 1.7 GiB.

We ran the sweep three times: twice with 60-second windows and once with 120-second windows at 4, 8 and 16 users. The table shows medians.

Phase 2: Ollama with OLLAMA_NUM_PARALLEL=32 (median of three runs) against phase 1 and vLLM.Observations:

1. The queueing disappears. Median TTFT at 32 users fell from 48 s to 1.5 s.
2. Throughput improves but flattens. Ollama reached 359 tok/s at 32 users, 2.3× its default, but per-user speed fell from 164 tok/s (1 user) to 42 (4 users) and 14 (32 users). vLLM still delivered 43 tok/s per user at 32. Batching in vLLM shares each weight read across users far more effectively.
3. Run-to-run variance is real at mid-load. At 8 users the three runs gave 269, 155 and 159 tok/s. At 32 users the runs agreed (356, 361).
4. Memory is not the limit. Ollama used 5.5 GB of 22 GB under full load, and GPU utilization read 66–81%.

We do not know how much speculative decoding contributes under load. Drafting helps a single stream, but at large batch sizes the extra verification work can cost more than it saves. A run with drafting disabled would settle this.

Scale-out with Ollama: routed before ready. We repeated the Section 4.5 test on the phase 2 service with --max-instances 2:

Cloud Logging, the second Ollama instance: the port opens 6 s after start, so the default TCP probe passes, but the model is on the GPU 2 minutes later and answering only at 15:29:58.

Users routed to the new instance waited up to 4.6 minutes for their first token. Cloud Run did nothing wrong: by the default TCP probe, the container was ready. vLLM avoided this because its /health probe only passes once the engine can serve. With Ollama on Cloud Run, define an HTTP startup probe that only passes once the model is loaded, for example a small wrapper that preloads the model before opening the port. Otherwise every scale-out sends users to a GPU that can't answer yet.

5. Discussion

5.1 Choosing an engine

vLLM buys its throughput with complexity: more flags, a 10.9 GB image, BF16 weights, and about 7-minute cold starts. That cost is not worth paying for a single user.

5.2 Paying for the scale-out gap

Because Cloud Run scales on request concurrency and a new GPU instance needs a full cold start, every scale-out event has a cost. There is no free option:

The fourth lever is to shorten the cold start. In a side experiment, storing the weights in a Cloud Storage bucket and reading them over Direct VPC with Private Google Access cut weight loading from 176 s to 79 s, and instance start to ready from ~7 min to under 5 min. Loading strategies deserve a study of their own.

5.3 Operational lessons

1. Pin the image version. Google publishes its own vLLM build for Model Garden, but its friendly tags (gemma4, latest) move, and its newest version tag lagged upstream (v0.20 vs v0.33). Pinning upstream gives an exact, portable version. Cloud Run pulls Docker Hub images through Google's mirror; on v0.33.0's release day the mirror did not yet have the tag (Image ... not found). It resolved within hours.
2. Size the startup probe from a measured cold start, not from an example value.
3. Disable every unused modality on multimodal models.
4. Turn off thinking explicitly when benchmarking reasoning-capable models, and verify it per engine and per endpoint.
5. Do not use GPU utilization as an efficiency KPI for LLM serving.

6. Limitations

• Unequal precision. Ollama ran 4-bit and vLLM BF16, each engine’s typical default. This favours Ollama at one user. A 4-bit vLLM run (e.g. gemma-4-E2B-it-qat-w4a16-ct) would likely narrow the single-user gap.
• Ollama untuned in phase 1. Phase 1 used Ollama’s default of one parallel request. Phase 2 (Section 4.6) corrects this. Other Ollama settings (flash attention, KV cache type, speculative decoding off) were not explored.
• Unequal speculative decoding. Ollama decoded speculatively in both phases, and vLLM did not. vLLM supports speculative decoding, which we did not enable.
• vLLM capped at 32 sequences. vLLM had headroom for about 90; higher settings were not tested.
• Few runs, short windows. Phase 1 had two runs per level (the tables use the dry run; the recorded run’s throughput was within 3%) and phase 2 had two to three, with 60–120 s windows and 5 minutes for scale-out tests. At 8 users, phase 2 runs varied by up to 1.7×. Headline ratios at 32 users were stable; we did not compute confidence intervals.
• Synthetic workload. Fixed 256-token outputs, eight prompts, closed-loop users with no think time. Real traffic has variable lengths and arrivals.
• Client-side TTFT includes the India-to-Belgium network round trip for both engines.
• One model, one GPU type, one region, list prices on one day.
• Scale-out timing comes from a single run. Cloud Run’s exact scale-out trigger is not documented beyond request concurrency; the early start we observed may vary.

7. Conclusion

On the same NVIDIA L4 and the same Gemma 4 E2B model on Cloud Run, Ollama is the better single-user engine and vLLM is the better serving engine. vLLM overtook Ollama at four users. At 32 users it delivered 7.7× the throughput of Ollama’s defaults, where users waited up to 48 seconds. Against Ollama with 32 parallel slots it still delivered 3.4× the throughput at 3.4× lower cost per token. Tuning Ollama removed the queue but not the gap. Cloud Run’s GPU utilization metric did not reflect this difference. Its autoscaling, which works on concurrency, started a second GPU early and billed it for more than six minutes before it served anyone. For production LLM serving on Cloud Run, choose the engine by concurrency, judge efficiency by tokens per dollar, and decide explicitly how you will pay for the scale-out gap.

Appendix: reproduce it

Quick single-request comparison:

# Ollama
curl -s $OLLAMA_URL/v1/chat/completions -H "Content-Type: application/json" -d '{
 "model": "gemma4:e2b",
 "messages": [{"role": "user", "content": "Explain how a load balancer works in 5 sentences."}],
 "max_tokens": 256, "reasoning_effort": "none"}'

# vLLM
curl -s $VLLM_URL/v1/chat/completions -H "Content-Type: application/json" -d '{
 "model": "google/gemma-4-E2B-it",
 "messages": [{"role": "user", "content": "Explain how a load balancer works in 5 sentences."}],
 "max_tokens": 256, "chat_template_kwargs": {"enable_thinking": false}}'

One question, one request each: “Explain how a load balancer works in 5 sentences.” Ollama took 1.65 s and vLLM 2.35 s, for 98 tokens each. The right edge is cut off where the webcam covered it.

References

1. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023: https://arxiv.org/abs/2309.06180
2. vLLM documentation, installation and hardware support: https://docs.vllm.ai/en/latest/getting_started/installation/
3. vLLM Gemma 4 recipe: https://docs.vllm.ai/projects/recipes/en/latest/Google/Gemma4.html
4. Google Cloud — Run Gemma 4 on Cloud Run with vLLM: https://docs.cloud.google.com/run/docs/run-gemma-on-cloud-run
5. Google Cloud — Best practices for AI inference on Cloud Run with GPUs: https://docs.cloud.google.com/run/docs/configuring/services/gpu-best-practices
6. Google Cloud — Configure GPUs for Cloud Run services: https://docs.cloud.google.com/run/docs/configuring/services/gpu
7. Gemma 4 E2B model card: https://huggingface.co/google/gemma-4-E2B-it
8. Ollama — gemma4 model: https://ollama.com/library/gemma4 · FAQ: https://docs.ollama.com/faq
9. vLLM container images: https://hub.docker.com/r/vllm/vllm-openai/tags

Companion study: CPU vs GPU for LLM inference on Cloud Run

Looking for a similar implementation, or for assistance with your Cloud and AI journey? Talk to TechTrapture: https://techtrapture.com/contact or email vishal.bulbule@techtrapture.com

Ollama vs vLLM on Cloud Run with an NVIDIA L4: A Real Benchmark from 1 to 48 Concurrent Users was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Vishal Bulbule · Google Developer Expert ( Cloud & AI) · Founder, TechTrapture

The four ways to self-host an open model on Google Cloud: Model Garden, Cloud Run + GPU, a GPU VM, and GKE.

Abstract

Open-weight models such as Gemma, Qwen, Llama and DeepSeek can be downloaded and run anywhere. That raises two questions for every team building on them: should we self-host at all, or call a managed API? And if we self-host on Google Cloud, which of the four main options should we use? The options are Vertex AI / Agent Platform Model Garden, Cloud Run with GPUs, Compute Engine GPU VMs, and GKE.

We compare the options on management effort, available hardware, billing model, and cost, using Google Cloud list prices (Billing Catalog, 2 Nov 2026) and our own measurements on an NVIDIA L4. The deciding factor is utilization. A busy self-hosted GPU served Gemma 4 E2B at $0.32 per million output tokens (vLLM, 32 concurrent users), but a single request to an otherwise idle Cloud Run GPU cost about $0.24, because the instance is billed until it scales to zero. Per hour, Cloud Run is the most expensive option ($1.42/h vs $0.71/h for a GPU VM); it wins only because it can scale to zero. Above roughly 12 hours of use per day, a GPU VM is cheaper. We also document the first operational barrier, GPU quota, including the free-trial restriction that blocks GPUs on VMs entirely. We close with a decision framework.

1. Introduction

Running an open model on a laptop is easy. Tools like Ollama make it a one-line command. Serving the same model to an application or a team is a different problem. A laptop is not always on, has no HTTPS endpoint or authentication, and answers one request at a time.

Teams moving from laptop to production face a sequence of decisions:

1. Whether to self-host, or call a managed API (Gemini, or an open model offered as a service).
2. Where to self-host: on Google Cloud, four main options with very different billing models.
3. Which inference engine runs inside: Ollama or vLLM.
4. How to get GPUs: quota, which is the first error most teams see.

This post works through them in order, with prices and measurements rather than opinions.

Research questions

• RQ1: Under what conditions is self-hosting preferable to a managed API?
• RQ2: How do the four Google Cloud hosting options differ in effort, hardware and billing?
• RQ3: How do costs compare, and where are the break-even points?
• RQ4: What blocks teams on day one?

2. Method

• Sources: Google Cloud documentation for each service (references at the end), current as of 2 Nov 2026.
• Prices: list prices from the Google Cloud Billing Catalog API, read 2 Nov 2026, before discounts (no Spot, no committed-use discounts), for one NVIDIA L4 GPU.
• Measurements: our own benchmarks on Cloud Run with an NVIDIA L4 (europe-west1), published as companion studies:
• CPU vs GPU for LLM inference on Cloud Run (Gemma 2 2B, Ollama)
• Ollama vs vLLM on Cloud Run with an NVIDIA L4 (Gemma 4 E2B, 1–48 concurrent users)
• Cost formula for a busy instance:cost per 1M output tokens = price per hour ÷ (tokens/s × 3600) × 1,000,000

3. Should you self-host at all? (RQ1)

An analogy that holds up: a managed API is a taxi; self-hosting is owning a car. With the taxi you pay per ride and someone else does the maintenance, but you go where taxis go. With your own car you pay every month even while it is parked, and you fix it yourself, but you go anywhere, any time.

3.1 When a managed API is the better choice

• Low or spiky traffic. You pay per token, so zero requests means zero bill.
• No operations. No GPUs, drivers, scaling, patching or 2 a.m. pages.
• Highest quality. Frontier models are far larger than anything that fits on one GPU.
• Instant scale. Ten users or ten thousand is the provider’s problem.

3.2 When self-hosting is the better choice

3.3 Lifecycle risk is real

On 21 October 2026, Google retired a batch of open-model APIs on its platform, including Qwen, older DeepSeek versions, Llama 3.3 70B, GLM and Kimi K2. Applications built on those endpoints had to migrate. A self-hosted copy of the weights would have kept running.

Google’s open-model deprecations page: a batch of open-model APIs retired on 21 October 2026.

3.4 What self-hosting costs you

• You pay for idle. The GPU is billed while it exists, answering or not.
• You operate it. Scaling, patching, monitoring and security become your job. Some engines ship with no authentication at all; Ollama is one, so never expose it without an auth layer in front.
• Smaller models. One 24 GB L4 comfortably serves models in the ~2–9B range, not frontier-scale models.
• GPU scarcity. You may not get the GPU you want in the region you want (Section 6).

Finding (RQ1): start with an API. Self-host when privacy, network control, lifecycle control or custom weights require it, or when traffic is steady enough to keep a GPU busy.

4. The four options on Google Cloud (RQ2)

Every option has the same three layers, client → inference engine (Ollama / vLLM) → model weights, on a GPU. What differs is who runs which layer, and how you are billed.

Model Garden, the hotel room: pick a model card, deploy, get an endpoint. Billed per machine-hour from deploy until undeploy.

Cloud Run with one NVIDIA L4 attached (from the CPU vs GPU recording). Attaching a GPU switches the service to instance-based billing.

Naming note: Model Garden lives in what was called Vertex AI; Google’s documentation now places it under Gemini Enterprise Agent Platform.

Watch the walkthrough: Self-hosting open LLMs on Google Cloud

5. Cost analysis (RQ3)

5.1 Price per hour, one NVIDIA L4 (list prices, 2 Nov 2026)

One L4, list prices 2 Nov 2026. Cloud Run wins below ~12 hours of use a day; a VM left on wins above it.

Per hour, Cloud Run is the most expensive option. It wins only because it scales to zero.

• Break-even, Cloud Run vs an always-on VM: $1.42 × h × 30 = $515 → h ≈ 12 hours/day. Above about 12 hours of use per day, a VM left running is cheaper. Below it, Cloud Run is. Cloud Run’s idle tail per scale-out (Section 5.3) moves the break-even lower in practice.
• Model Garden costs roughly 30% more than an equivalent VM. That is the price of Google running the serving stack for you.

5.2 Busy GPU: cost per token depends on the engine

Measured on Cloud Run, one L4 at $1.42/h, Gemma 4 E2B:

Same GPU, same price per hour: a 7.7× difference in cost per token, decided by the inference engine and how busy it is.

The raw benchmark output behind these costs, from the recorded run. The table above uses our dry run from the same day, so its values differ slightly.

5.3 Idle GPU: the cost of one lonely request

Instance-based billing charges for the whole instance lifetime. In our runs a Cloud Run GPU instance scaled to zero about 10 minutes after its last request. One isolated request therefore costs roughly:

The same 256-token answer on a busy vLLM instance costs 256 × $0.32 / 1⁰⁶ ≈ $0.00008. That is a gap of about 3,000× for the same model on the same GPU. Utilization, not the platform, dominates the cost of self-hosting.

One isolated request pays for the idle tail; a busy vLLM instance spreads the GPU across many users.

Finding (RQ3): self-hosting is cheap when the GPU is busy and expensive when it is idle. Pick the option whose billing matches your traffic: scale-to-zero (Cloud Run) for low or spiky traffic, always-on (VM, Model Garden, GKE) for steady traffic, with the engine (vLLM) doing the batching.

6. The first barrier: GPU quota (RQ4)

GPUs are rationed per project, per region, per GPU type, and per product. A team’s first error is usually a quota error, and that is normal.

The error from our own recording: Cloud Run refused the GPU service because the project had no quota for L4s without zonal redundancy in that region.

Free-trial restriction. New accounts get $300 credit for 90 days. While on the free trial (not upgraded to a paid billing account), you cannot add GPUs to VM instances and cannot request quota increases. Upgrading keeps the remaining credit within the original 90 days, and existing resources keep running.

Cloud Run. Projects receive 3 × L4 (without zonal redundancy) in a region automatically on their first GPU deployment there. Google notes that requests for zonal-redundancy-off quota are more likely to be granted.

Quota is per product. L4 quota for Cloud Run does not give you L4s for Compute Engine or for Model Garden endpoints. Each has its own quota.

GPU quota is per product: Cloud Run’s L4 quota does not give you L4s for Compute Engine or Model Garden.

Quota is not the only wall. Two real failures from our 3 Nov runs: a g2-standard-8 VM with an L4 unavailable in europe-west1-b, and a Model Garden deployment that could not get a g2-standard-12 with an L4.

Finding (RQ4): check quota for the exact product, region and GPU before designing around it, and move off the free trial before planning any GPU VM.

7. The engine inside the box

Whichever option you pick, an inference engine loads the model and answers requests.

On Cloud Run, a VM or GKE you choose the engine. Model Garden deploys vLLM-family serving containers for you.

8. Decision framework

9. Limitations

• List prices only, for one GPU type (L4) and specific regions, on one date. Spot VMs, committed-use discounts and negotiated pricing change the break-even points, usually in favour of always-on options.
• Measurements cover one model family (Gemma) on one GPU type. Larger models change the hardware options and the economics.
• Idle-tail duration (about 10 minutes) is what we observed, not a documented guarantee.
• Feature status changes. Model Garden scale-to-zero was in Preview at the time of writing, and GPU availability per product and region changes frequently.
• Managed-API pricing was not benchmarked here. Compare your expected tokens per month against the current price of the model you would otherwise call.

10. Conclusion

Self-hosting an open model on Google Cloud is neither inherently cheaper nor more expensive than calling an API. It is cheaper when you keep the GPU busy. A busy L4 running vLLM served tokens at $0.32 per million; the same GPU answering one isolated request cost about $0.24 for that single answer.

Choose a managed API by default. Self-host when privacy, control, lifecycle or custom weights demand it, or when traffic is steady. Then match the hosting option’s billing to your traffic pattern: Cloud Run for spiky, VM or Model Garden for steady, GKE for scale. Pick the engine by concurrency. And check your GPU quota before you design anything.

References

1. Google Cloud — Choose an open model serving option: https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/open-models/choose-serving-option
2. Google Cloud — Open model deprecations: https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/deprecations/open-models
3. Google Cloud — Configure GPUs for Cloud Run services: https://docs.cloud.google.com/run/docs/configuring/services/gpu
4. Google Cloud — Compare compute options: https://docs.cloud.google.com/docs/compute-area/choose-compute-options
5. Google Cloud — GKE pricing: https://cloud.google.com/kubernetes-engine/pricing
6. Google Cloud — Free Trial and Free Tier features and limits: https://docs.cloud.google.com/free/docs/free-cloud-features
7. Google Cloud — Cloud Run pricing: https://cloud.google.com/run/pricing
8. Ollama — FAQ (parallel requests): https://docs.ollama.com/faq
9. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023: https://arxiv.org/abs/2309.06180

Companion studies: CPU vs GPU for LLM inference on Cloud Run · Ollama vs vLLM on Cloud Run with an NVIDIA L4

Looking for a similar implementation, or for assistance with your Cloud and AI journey? Talk to TechTrapture: https://techtrapture.com/contact or email vishal.bulbule@techtrapture.com

Self-Hosting Open LLMs on Google Cloud: When It Pays Off, and Which of the Four Options to Choose was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Vishal Bulbule · Google Developer Expert (Google Cloud) · TechTrapture

Same model, same Cloud Run service, one difference: 8.35 tokens/s on 4 vCPUs vs 113 tokens/s with an NVIDIA L4.

Abstract

Teams self-hosting open models on serverless platforms face an early choice: run the model on CPU, which needs no GPU quota and bills only per request, or attach a GPU, which costs more per hour and bills per instance. We deployed the same container (Ollama) and the same model (Gemma 2 2B, 4-bit) as two Cloud Run services that differed only in an NVIDIA L4 GPU, and measured generation speed, time to first token, cold start, and cost per million output tokens. On the L4, generation reached 113 tokens/s versus 8.35 tokens/s on 4 vCPUs (13.5×), and a 256-token answer took 2.7 s versus 31.6 s. Although the GPU service costs $1.05/hour versus $0.49/hour for the CPU service, its cost per million output tokens was ≈$2.57 versus ≈$16.30, about 6× cheaper per answer. The GPU’s cold start, scaling from zero, was 20.2 s. We discuss when the CPU option is still the right call: very low request volumes, where per-request billing and the GPU’s idle tail dominate.

1. Introduction

Cloud Run added NVIDIA GPU support, which makes it one of the simplest ways to self-host an open model on Google Cloud: one container, scale to zero, no cluster. That raises a practical question every team asks first: do I actually need the GPU?

The CPU path is attractive. It needs no GPU quota, it is billed only while a request runs, and small models such as Gemma 2 2B do run on CPU. The GPU path costs more per hour and needs quota. Price per hour, however, is the wrong unit for comparing the two. What a product pays for is answers.

This post reports a controlled comparison: same image, same model, same region, same memory and vCPU count. The only difference is whether one NVIDIA L4 is attached. We report speed, latency, cold start, and cost per million output tokens, and we state the conditions under which each option wins.

2. Background

Ollama is a popular open-source runtime for local LLMs. It packages models in the GGUF format, mostly 4-bit quantized, and runs them on llama.cpp on CPU or GPU. It exposes a native API (/api/generate, /api/chat) and an OpenAI-compatible API (/v1/chat/completions).

Cloud Run GPUs. A Cloud Run service can attach one GPU per instance. We used an NVIDIA L4 (24 GB). GPU services require instance-based billing (--no-cpu-throttling): the whole instance is billed while it is up, including an idle period before it scales to zero. CPU services can use request-based billing, where CPU and memory are billed only while requests are being processed.

Token speed terms.

• Prompt processing (prefill): reading the input.
• Generation (decode): producing output tokens, one at a time.
• Time to first token (TTFT): how long until text starts appearing.

3. Experimental setup

3.1 Platform and configuration

The two services were identical except for the GPU flags. Both mounted the same Cloud Storage bucket, so the model was downloaded once and read by both.

The GPU service in the Cloud Run console: same 4 vCPU / 16 GiB, plus one NVIDIA L4. Attaching the GPU forces the switch from request-based to instance-based billing.

3.2 Deployment

CPU service:

gcloud run deploy ollama-cpu \
 --image ollama/ollama \
 --region europe-west1 \
 --cpu 4 --memory 16Gi \
 --port 11434 \
 --min-instances 0 --max-instances 1 \
 --timeout 600 \
 --set-env-vars OLLAMA_KEEP_ALIVE=-1 \
 --add-volume name=models,type=cloud-storage,bucket=$BUCKET \
 --add-volume-mount volume=models,mount-path=/root/.ollama

GPU service, which is the same command plus four flags:

--gpu 1 --gpu-type nvidia-l4 \
 --no-gpu-zonal-redundancy \
 --no-cpu-throttling

--no-cpu-throttling is required for GPU services and switches the service to instance-based billing. --no-gpu-zonal-redundancy is cheaper and matched our available quota.

IAM & Admin → Quotas and system limits, filtered on “Total Nvidia L4 GPU allocation without zonal redundancy”. It is set per region and is 0 in most regions until you request it.

Our first attempt to create the GPU service failed on quota, which is why the quota page above matters.

3.3 Workload and method

• Prompt: one fixed prompt (“Why are GPUs better than CPUs for AI workloads?”).
• Decoding: temperature 0, fixed seed (42), num_predict = 256 output tokens, so every run does the same amount of work.
• Runs: 5 sequential requests per service, through Ollama’s native /api/generate.
• Metrics: taken from Ollama’s own timing fields: prompt-eval and eval token counts and durations, and load duration. Wall time was measured client-side.
• Cold start: measured as the first request after the GPU service had scaled to zero.
• Pricing: Google Cloud Billing Catalog list prices for europe-west1 on the day of the run (2 Nov 2026), before free tier.

3.4 Cost model

This assumes the instance is kept busy. We return to idle cost in Section 5.

Watch the walkthrough: Ollama on Cloud Run: CPU vs GPU

4. Results

4.1 Speed

Generation speed, warm runs: 8.35 tokens/s on CPU vs 113 tokens/s on the L4.

The GPU improves both phases of inference. Prefill is about 16× faster, and generation about 13.5× faster. Network and routing overhead was the same for both services, about 0.4 s from the client, so for short prompts the end-to-end difference is driven by generation.

At 8.35 tokens/s, a reader sees a 256-token answer appear over roughly half a minute. At 113 tokens/s it appears in under three seconds, faster than anyone reads.

4.2 Cold start

The first GPU request after scaling to zero took 20.2 s wall time. 14.6 s of that was waiting for the first token, which is mostly loading the 1.6 GB model from the mounted bucket into GPU memory. Warm requests started in about 23 ms server-side.

The benchmark output. GPU run 1 is the cold start (20.2 s wall, 14.6 s to first token while the model loads); runs 2–5 are warm at ~113 tokens/s, which is why the 5-run GPU average shows 100.58.

A test question to the GPU service: “Why is the sky blue?”. Ollama’s own timing fields at the end show 14.4 s before generation began, then 263 tokens in 2.31 s (about 114 tok/s).

4.3 Cost per answer

Breakdown, list prices for europe-west1:

• CPU: 4 vCPU × $0.000024/s + 16 GiB × $0.0000025/s, request-based billing.
• GPU: L4 at $0.0001867/s + 4 vCPU × $0.000018/s + 16 GiB × $0.000002/s, instance-based billing.

List prices, europe-west1, 2 Nov 2026. The GPU service costs ~2× more per hour but ~6× less per million output tokens when kept busy.The GPU costs about 2× more per hour and about 6× less per answer.

5. Discussion

5.1 Cheaper per hour is not cheaper per answer

The CPU service looks cheaper on the pricing page. Per unit of useful output, it is the expensive option, because it produces roughly 13× fewer tokens in the same hour. Whenever the instance is kept reasonably busy, the GPU wins on cost as well as speed.

5.2 When the CPU still wins

The cost model above assumes a busy instance. Two billing details change the answer at very low volume:

1. Request-based vs instance-based billing. The CPU service is billed only while requests run. The GPU service is billed for the whole instance lifetime.
2. The idle tail. After the last request, a GPU instance stays up, and billed, for a while before scaling to zero. In our runs, scale-to-zero happened about 10 minutes after the last request.

For a handful of requests a day, such as an internal tool, a nightly batch or a prototype, a CPU service can therefore be cheaper overall, provided users accept a 30-second answer. As soon as there is steady traffic or a human waiting on the answer, the GPU is the better choice on both speed and cost.

What went wrong first: when the GPU service got a fresh instance, the model we had pulled earlier was gone, so the benchmark failed with HTTP 404 “model not found”. This is why the first note below is about storing the model outside the container.

5.3 Practical notes

• Store the model outside the container. Cloud Run instances start empty. Mounting a Cloud Storage bucket means the model is downloaded once. Mount it where Ollama writes models (/root/.ollama); otherwise downloads land on the instance's in-memory disk and vanish with it.
• Keep the model loaded. OLLAMA_KEEP_ALIVE=-1 stops Ollama from unloading the model after 5 idle minutes while the instance is alive.
• Lock it down. Ollama has no authentication of its own. Use --no-allow-unauthenticated and let Cloud Run IAM guard the endpoint.

6. Limitations

• Single model, single size. Gemma 2 2B in 4-bit is small. Larger models widen the gap, and beyond a certain size they become impractical on CPU.
• Single-user, sequential workload. We measured one request at a time. Concurrency changes the picture, especially for engines built for batching. That is the subject of the companion study on Ollama vs vLLM.
• Small sample. Five runs per service with a fixed prompt. The differences are an order of magnitude, far larger than run-to-run noise, but exact values will vary.
• Prices are list prices for one region on one day, before free tier and discounts.
• Client location. Requests came from India to europe-west1. This affects end-to-end latency, not the server-side speeds reported.

7. Conclusion

For LLM inference on Cloud Run, an NVIDIA L4 made Gemma 2 2B about 13.5× faster at generating text and about 6× cheaper per answer than 4 vCPUs, even though it costs about 2× more per hour. CPU deployments still make sense for very low, latency-tolerant volumes, where per-request billing avoids paying for idle GPU time. For anything with real users, the GPU pays for itself.

Decision rule: compare cost per answer at your expected load, not price per hour.

The decision in one card.

References

1. Google Cloud — Configure GPUs for Cloud Run services: https://docs.cloud.google.com/run/docs/configuring/services/gpu
2. Google Cloud — Cloud Run pricing: https://cloud.google.com/run/pricing
3. Google Cloud — Run LLM inference on Cloud Run GPUs with Ollama: https://docs.cloud.google.com/run/docs/tutorials/gpu-gemma-with-ollama
4. Google Cloud — Best practices for AI inference on Cloud Run with GPUs: https://docs.cloud.google.com/run/docs/configuring/services/gpu-best-practices
5. Ollama — FAQ (keep-alive, parallel requests): https://docs.ollama.com/faq
6. Gemma 2 2B model card: https://huggingface.co/google/gemma-2-2b-it

Companion study: Ollama vs vLLM on Cloud Run with an NVIDIA L4

Looking for a similar implementation, or for assistance with your Cloud and AI journey? Talk to TechTrapture: https://techtrapture.com/contact or email vishal.bulbule@techtrapture.com

CPU vs GPU for LLM Inference on Cloud Run: A Measured Comparison with Gemma and an NVIDIA L4 was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Google Antigravity is an agentic IDE — the agent plans, edits, and runs code for you. What it doesn’t do out of the box is draw. When you need an architecture diagram for a design doc or a sequence diagram for a PR, you still tab out to a separate tool, wrestle with syntax, and paste an image back.

You can close that loop. By connecting the Capci MCP server, you can generate a diagram in Antigravity from a plain-text prompt — the agent calls the tool, and a publish-ready image comes back in the same conversation.

Capci: Prompt-to-Diagram Tool for Your Agent

What is Capci, and what problem does it solve

Capci turns a plain-text description of a system into a finished visual — architecture diagrams with real cloud-provider icons, plus flowcharts, sequence, class, ER, mindmap and Gantt diagrams, data charts, and animated GIF exports.

The problem it solves is the diagramming detour. Every time you need a picture of a system, you leave the tool you’re working in, drag boxes around a canvas, hunt down the right provider icons, align arrows, and re-style the output to match your brand — twenty minutes of manual work for an artifact you’ll redo the next time the architecture changes. The alternative, hand-writing a diagram DSL, starts faster and ages worse: it’s another syntax to maintain, and it still won’t give you real vendor icons, on-brand colors, or animation.

Capci collapses that into one step. You describe the system, you get the image, and you change it by describing the change. It runs in the browser at capci.app, through a REST API, or — as in this guide — through MCP, so an agentic IDE can generate diagrams for you mid-task.

What you’ll be able to do

• Ask for an architecture diagram in words and get a clean image with real cloud-provider icons.
• Generate flowcharts, sequence, class, ER, mindmap, and Gantt diagrams.
• Refine in place: “make it dark theme,” “add a caching layer.”
• Export animated diagrams as GIFs for docs and posts.

No diagram DSL to learn, no canvas to fiddle with.

Step 1 — Create your Capci API key

The MCP server authenticates with an API key, so grab one first:

1. Go to capci.app and sign in — Continue with Google, or email and password.
2. You’ll land on the dashboard at capci.app/app. Open the Account tab in the left sidebar.
3. Scroll to the API & MCP card and click Generate API Key.
4. The key is shown in full, starting with capci_live_. Click Copy.

If you navigate away before copying it, you’re fine — the same card has a Show button that re-reveals the key later. Two buttons to be careful with: Regenerate issues a new key and kills the old one immediately (every config still using it breaks until you paste the new one in), and Revoke disables API access altogether.

Treat the key like a password — it’s tied to your account, so don’t commit it to a repo or paste it into a shared workspace file.

On the free plan the key works as a one-time trial against your free generation allowance, which is plenty to test the setup below. Paid plans get a recurring monthly API quota — see capci.app/pricing.

Step 2 — Add the Capci MCP server to Antigravity

Open ~/.gemini/config/mcp_config.json (or use Antigravity's MCP settings panel) and add a remote HTTP server. Antigravity uses serverUrl — not url — for remote Streamable-HTTP servers, with custom headers for auth:

{
  "mcpServers": {
    "capci": {
      "serverUrl": "https://mcp.capci.app/mcp",
      "headers": { "x-api-key": "YOUR_API_KEY" }
    }
  }
}

Reload the MCP servers. Antigravity discovers the Capci tools (create_diagram, refine_diagram, animate_diagram, and more).

Step 3 — Prompt for a diagram

In the agent chat:

Create an architecture diagram: users → CDN → a Cloud Run API → Firestore,
with a Pub/Sub side-channel to a background worker.

The agent calls Capci and returns the diagram inline. Not quite right? Just say:

Add a Redis cache between the API and Firestore, and switch to a dark theme.

Why do it this way

The whole point of an agentic IDE is staying in one place while the work gets done. Bolting a text to diagram tool onto Antigravity via MCP keeps diagramming in that same flow: describe it, get it, refine it, ship it — without leaving the editor or learning another tool’s syntax.

It also changes what you bother to diagram at all. When prompt to diagram takes ten seconds instead of twenty minutes, you start drawing the things you used to describe in a paragraph and hope people followed — the retry path, the failure mode, the bit of the flow every reviewer asks about.

Try it

1. Grab a free key at capci.app.
2. Add the MCP config above to Antigravity.
3. Ask for your first diagram.

See the full setup and per-host configs in the Capci MCP repo, including Claude and VS Code.

Open ChatGPT or Gemini and ask it something only your company would know — say, how many weeks of parental leave your primary caregivers get. You’ll get a refusal, a confident-sounding guess, or some generic HR boilerplate. None of it is your actual policy, because the model has never seen your documents.

That’s the gap RAG fills. Retrieval-Augmented Generation is a fancy name for a simple idea: before the model answers, go fetch the relevant documents and hand them over, so the answer is grounded in your data instead of the model’s imagination.

In this post I’ll build a real one with you — an internal HR-policy assistant running entirely on Google Cloud. Elasticsearch Serverless does the retrieval, Agent Platform (Former Vertex AI) provides the models, and Elastic Agent Builder ties it together. There’s very little code to write, and by the end you’ll actually understand why each piece is in the picture, not just that it’s there.

Why Elasticsearch with Google Cloud

If you live on Google Cloud, you’ve probably already spotted the obvious objection: Google Cloud has its own retrieval stack Agent Search (Vertex AI Search), RAG Engine, Vector Search. So why drag Elastic into it?

Fair question, and the answer is that this isn’t an either/or. The models stay Google’s Gemini handles the embeddings, the reranking, and the final answer. You never leave Google Cloud, because Elasticsearch Serverless runs natively on Google Cloud, in the same region as Vertex AI. All you’re really swapping is one layer: the search engine underneath the models.

And that layer is worth swapping deliberately, because search is harder than it looks and Elastic has been doing nothing else for fifteen years. They took BM25 the keyword-ranking algorithm researchers have leaned on since the ’90s and made it the default the rest of the industry followed similar BM25 as default search.

Elastic turned Reciprocal Rank Fusion into a first-class way to blend keyword and vector results. And they give you something most turnkey RAG boxes won’t: you can ask why a document ranked where it did, and get a real answer.

The way I think about it:

Google Cloud provides state-of-the-art Gemini models, while Elasticsearch provides a mature and highly optimized search engine. Together they form a powerful enterprise RAG stack

You can find about latest Embedding models & Reranker here

• Embeddings | Gemini API | Google AI for Developers
• Improve search and RAG quality with ranking API | Agent Search | Google Cloud Documentation

The concepts, quickly

If the terms below are already second nature, skip ahead. If not, here’s the whole vocabulary you need.

An embedding model turns a piece of text into a list of numbers a vectorthat captures what the text means, not which words it uses. Two sentences that say the same thing in different words end up close together. That’s the trick that lets a search for “time off for new parents” find a doc titled “Parental Leave Policy” with zero words in common. We’ll use Google’s gemini-embedding-001.

Keyword search (BM25) ranks by word overlap. It’s fast and exact, and completely blind to meaning. Vector search ranks by how close two embeddings are, so it’s good at meaning but occasionally loose on specifics. BM25 (Best Matching 25) is a keyword-based ranking algorithm used by search engines to determine how relevant a document is to a user’s search query.

Unlike vector search, BM25 does not understand the meaning of words. It simply ranks documents based on how well the query terms match the document.

You don’t have to pick a side: run both and fuse the results with RRF that’s hybrid search.

Hybrid Search = BM25 + Vector Search + RRF

A reranker is the part people skip and shouldn’t. After the first search hands back a pile of candidates, the reranker re-reads the top few against the actual question and reshuffles them so the genuinely best document lands at the top. It’s a small, specialized model, not a chatbot Google Cloud’s semantic-ranker-fast-004 in our case.

Finally the LLM (gemini-2.5-pro). Search gives you documents; people want a sentence. The LLM reads the top documents and writes the grounded answer with a citation.

Agent Builder abstracts the orchestration layer, allowing developers to focus on tools and business logic instead of manually wiring retrieval, reranking, and LLM orchestration.

Agent Builder is Elastic’s layer for putting an agent over all of this. You define tools (like “search the policy docs”), attach them to an agent, and call it from an API or expose it over MCP so other agents can use your tools too. More on that later.

https://www.elastic.co/docs/explore-analyze/ai-features/agent-builder/mcp-server

That’s it. Now let’s build. Refer video for live demo and steps given in this blog.

What we’re building

In this blog we demonstrates how to build an enterprise-grade AI-powered RAG (Retrieval-Augmented Generation) application using Elasticsearch Serverless and Google Cloud Agent Platform (Formerly known as Vertex AI).

Elasticsearch Serverless integrates with three external Vertex AI inference endpoints Gemini Embedding for semantic indexing, Vertex AI Semantic Reranker for improving search relevance, and Gemini 2.5 pro for generating grounded responses. Elastic Agent Builder orchestrates these services to retrieve relevant documents, rerank search results, and generate accurate, citation-backed answers from enterprise knowledge bases such as HR policies, documentation, and internal knowledge repositories.

The same search capabilities can also be exposed through MCP, enabling seamless integration with AI agents built using frameworks like Google ADK.

For data I’m using a set of “TechTrapture HR policy” PDFs parental leave, sick leave, expenses, BYOD, the usual. They’re made up, so you can clone the repo and reproduce every step.

Step 1 — Stand up Elasticsearch Serverless

In Elastic Cloud, create a Serverless project of type Elasticsearch on Google Cloud. Serverless means exactly what it sounds like: no nodes, no shards, no scaling knobs. You get an endpoint and a Kibana, and Elastic runs the rest.

Creating a Serverless Elasticsearch project on Google Cloud in the Elastic Cloud console.

Once it’s up, open Kibana → Dev Tools → Console and check it’s alive:

GET /

Step 2 — Connect three Vertex AI models

This is the part that makes the whole thing click. Elastic lets you plug Google’s models straight into the search pipeline through Inference Endpoints (under External Inference in your Serverless project).

Elasticsearch Serverless integrates with three external inference endpoints backed by Google Cloud Agent Platform (displayed as Vertex AI in the current Elastic UI):

• Gemini Embedding (gemini-embedding-001)
• Semantic Reranker (semantic-ranker-fast-004)
• Gemini 2.5 Pro (Chat Completion)

One heads-up if you’re following an older tutorial: this used to be called “AI Connectors.” The feature moved to Inference Endpoints — same idea, different menu.

You need three of them, one per job:

The Inference Endpoints screen showing the three Google Vertex AI endpoints.

Each one authenticates to Vertex AI with a Google Cloud service-account key. You can click through the UI, or create them in Dev Tools to get clean IDs. Here’s the reranker and note that a Vertex rerank endpoint is unusually simple, it only wants your project and the key:

PUT _inference/rerank/google_vertex_rerank
{
  "service": "googlevertexai",
  "service_settings": {
    "service_account_json": "PASTE_SERVICE_ACCOUNT_JSON_AS_A_STRING",
    "project_id": "YOUR_GCP_PROJECT_ID"
  },
  "task_settings": { "top_n": 5 }
}

The embeddings and chat endpoints look much the same, just with a model_id and location added. The full set of commands, including the gcloud bit that creates the service account, is in the repo.

Step 3 — Load the documents

The quickest way to get PDFs in with no code is Kibana’s Search → Upload a file. Drop a policy PDF, name the index hr_policies, and import. Elastic pulls the text into a field called attachment.content and the title into attachment.title.

If you want semantic search too, add a semantic_text field pointed at your embeddings endpoint. From then on Elastic embeds every document automatically as it lands there’s no embedding pipeline for you to build and babysit. (semantic_text is one of Elastic’s own conveniences, and a genuinely nice one.)

Quick check that everything made it in:

GET hr_policies/_count

Step 4 — Watch keyword search miss

Here’s the case for a reranker, made by letting plain search fail first. Ask a normal human question:

POST hr_policies/_search
{
  "retriever": {
    "standard": { "query": { "match": { "attachment.content": "how much time off do new parents get" } } }
  },
  "_source": ["attachment.title"],
  "size": 5
}

Look at what comes back. The Sick Leave & PTO policy is on top, because the words “time off” show up there constantly. But the question is about new parents, and the parental-leave doc — the one that actually answers it — is ranked too low. Keyword search counted words; it didn’t understand the question.

Step 5 — Add the Vertex reranker

Same query, now wrapped so the top results get a second pass from your Vertex reranker:

POST hr_policies/_search
{
  "retriever": {
    "text_similarity_reranker": {
      "retriever": {
        "standard": { "query": { "match": { "attachment.content": "how much time off do new parents get" } } }
      },
      "field": "attachment.content",
      "inference_id": "google_vertex_rerank",
      "inference_text": "how much time off do new parents get",
      "rank_window_size": 10
    }
  },
  "_source": ["attachment.title"],
  "size": 5
}

Same question, same retrieval, one extra wrapper and the Parental Leave policy is suddenly #1. The reranker understood that “new parents” points at parental leave, not at the policy that happens to repeat the phrase “time off.” That’s the whole job of a reranker, and it’s why I never ship serious RAG without one.

Step 6 — Make the reranked search a tool

To hand this to an agent, we wrap it as a tool. Reranking lives in Elastic’s ES|QL, so we create an ES|QL tool that bakes in the RERANK command:

POST kbn:/api/agent_builder/tools
{
  "id": "hr_policies.search_reranked",
  "type": "esql",
  "description": "Search the HR policy documents and rerank results with the Vertex AI reranker.",
  "configuration": {
    "query": "FROM hr_policies | WHERE attachment.content : ?query | LIMIT 100 | RERANK ?query ON attachment.content WITH { \"inference_id\": \"google_vertex_rerank\" } | SORT _score DESC | KEEP attachment.title, attachment.content, _score | LIMIT 5",
    "params": { "query": { "type": "string", "description": "The user's question" } }
  }
}

Two things that tripped me up the first time, so they don’t trip you up: the inference endpoint goes inside a WITH { “inference_id”: “…” } object, and you have to add | SORT _score DESC after RERANK or you get the reranked scores back in the original order.

Step 7 — Create the agent and ask it something

Attach the tool to an agent:

POST kbn:/api/agent_builder/agents
{
  "id": "hr-policy-agent",
  "name": "HR Policy Assistant",
  "configuration": {
    "instructions": "Answer strictly from the hr_policies.search_reranked tool results. Cite the document title. If the policies do not cover it, say so plainly.",
    "tools": [ { "tool_ids": ["hr_policies.search_reranked"] } ]
  }
}

One thing to know: the agent’s chat model is a global setting in Agent Builder (Model management / Feature settings), not something you set per agent. Point it at your gemini-2.5-pro endpoint and now generation runs on Vertex too — the whole stack stays on Google Cloud.

Then talk to it through the converse API the exact call your own app would make:

curl -sS -X POST "${KIBANA_URL}/api/agent_builder/converse" \
  -H "Authorization: ApiKey ${API_KEY}" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json" \
  -d '{
    "input": "How many weeks of parental leave do primary caregivers get?",
    "agent_id": "hr-policy-agent"
  }'

The response carries the answer, the documents it leaned on, and a model_usage block with token counts which leads neatly into the next bit.

Does the reranker actually save money?

Reranking isn’t only about getting the right answer. Because it narrows things down to the handful of documents that matter, the LLM has far less to read. In my test, the reranked agent answered the same question correctly while feeding the chat model roughly half the input tokens of the un-reranked version, and it came back faster too.

Now, the obvious pushback and it’s a good one — is that the reranker is also a model, so isn’t it just moving the cost around? Partly, yes, and it’s worth being straight about it. What you’re really doing is spending a little on a cheap model to spend a lot less on an expensive one. A reranker is a small cross-encoder, billed per request, and it generates no output tokens. The generative LLM is a big model billed per token. Trim what reaches the big model and the bill drops.

How much it drops depends on which chat model you run and how many queries you serve. On a small, cheap model the savings are modest. On a larger model, or at real volume, they’re substantial. Either way, the dependable wins are relevance and latency; the lower cost is a bonus that grows as you scale. I’d rather tell you that than wave a single token number around as if it were universal.

Bonus: drive it from a Google Cloud ADK agent over MCP

Here’s where it gets fun. Agent Builder publishes every tool over MCP, the Model Context Protocol. So a Google Cloud ADK agent Gemini-powered — can call your Vertex-reranked search as if it were a native tool, with no integration code in between.

The entire bridge is one McpToolset pointed at the MCP endpoint:

McpToolset(
    connection_params=StreamableHTTPConnectionParams(
        url=f"{KIBANA_URL}/api/agent_builder/mcp",
        headers={"Authorization": f"ApiKey {API_KEY}", "kbn-xsrf": "true"},
    ),
    tool_filter=["hr_policies_search_reranked", "platform_core_search"],
)

One subtlety that’s easy to trip on: MCP hands out tools, not agents. Over MCP, the ADK/Gemini model is the agent, and it calls Elastic’s tools directly. (If you wanted to call a whole Elastic agent as a single unit, you’d reach for the A2A endpoint instead.)

That’s what makes this worth doing rather than a party trick — the same reranked search tool can be used by your own app, by Claude Desktop or Cursor, or by an ADK agent, all from one definition.

You can find complete code in GitHub repo.

GitHub - vishal-bulbule/elastic-with-google-cloud

Where this leaves you

By combining Elasticsearch Serverless with Vertex AI models and Elastic Agent Builder, you’ve built a production-ready enterprise search assistant that delivers accurate, citation-backed answers from private knowledge bases. The same architecture can power internal copilots, customer support assistants, HR bots, documentation search, and AI agents through MCP with minimal custom code.One cloud, one stack, barely any glue code, and I hope a clear sense of what each part is doing and when you’d actually want it.

Try it yourself

• 🎥 Full video walkthrough: https://youtu.be/oGyBwdviKb8
• 💻 Code, commands & sample data: https://github.com/vishal-bulbule/elastic-with-google-cloud
• 🆓 Start an Elastic free trial and follow along
• 📚 Elastic Agent Builder — MCP server docs
• 📚 Google ADK — MCP tools docs

Adapting this to your own documents, or sorting out the production details? That’s the kind of thing I help teams with feel free to reach out on vishal.bulbule@techtrapture.com

Building AI-Powered search App with Elastic ,Google Gemini & ADK was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

This guide provides a central overview of deployment options, setup procedures, and a decision matrix for hosting your Agent Development Kit (ADK) agents on Google Cloud Platform (GCP).

1. What is Agent Development Kit (ADK)?

The Agent Development Kit (ADK) is a powerful framework designed by Google to simplify building, testing, and deploying autonomous AI agents and multi-agent systems. It provides first-class abstractions for:

• Agents: Configurable LLM-driven objects with specific system instructions and tools.
• Conversations & Sessions: Standardized state management.
• Tools & MCP Integration: Easy connection to standard local or remote tools (e.g., Google Search, BigQuery, databases) using Model Context Protocol (MCP).
• Observability: Built-in OpenTelemetry logging and tracing.

2. Deployment Options Overview

When moving your ADK agent from local development to production on GCP, you have three primary deployment options:

1. Vertex AI Agent Engine ( Now, Agent Platform Agent Runtime) : Serverless runtime specifically optimized for running reasoning engines and Python agent workflows.
2. Google Cloud Run: Serverless container runtime suitable for web applications, custom APIs, and UIs.
3. Google Kubernetes Engine (GKE)/Custom: Fully managed Kubernetes service for enterprise container orchestration.

3. Decision Matrix: When to Use What?

4. Step-by-Step Deployment Guides

I have used project tt-labs-001 & us-central1 for example. Please replace with your project id.

A. Google Cloud Run Deployment

You can deploy to Cloud Run either by using the ADK CLI directly or by writing a custom build and deployment script.

Option 1: Using the ADK CLI (Recommended)

The ADK CLI builds the image and deploys the agent with a single command.

Create a script deploy_run.sh in your agent folder:

#!/bin/bash
PROJECT_ID=tt-labs-001
LOCATION_ID=us-central1
adk deploy cloud_run \
        --project=$PROJECT_ID \
        --region=$LOCATION_ID \
        --service_name="my-first-agent" \
        --app_name="my_first_agent" \
        --trace_to_cloud \
        --otel_to_cloud \
        --with_ui \
        --port 8080 \
        my_first_agent

Option 2: Custom Dockerfile & Gcloud CLI

If you need custom control over your build, write a deploy.sh script:

#!/usr/bin/env bash
set -euo pipefail
PROJECT_ID="tt-labs-001"
REGION="us-central1"
REPO="adk-repo"
IMAGE="my-first-agent"
SERVICE="my-first-agent"
TAG="${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO}/${IMAGE}:latest"
# 1. Project config
gcloud config set project "${PROJECT_ID}"
# 2. Build image
gcloud builds submit --tag "${TAG}" .
# 3. Deploy
gcloud run deploy "${SERVICE}" \
  --image="${TAG}" \
  --region="${REGION}" \
  --platform=managed \
  --allow-unauthenticated \
  --port=8080 \
  --cpu=1 \
  --memory=1Gi \
  --set-env-vars="GOOGLE_GENAI_USE_VERTEXAI=1,GOOGLE_CLOUD_PROJECT=${PROJECT_ID},GOOGLE_CLOUD_LOCATION=${REGION}"

B. Vertex AI Agent Engine Deployment

Agent Engine provides a managed Python serverless host for reasoning engines.

Create a script deploy_agent_engine.sh in your agent folder:

#!/bin/bash
PROJECT_ID=tt-labs-001
LOCATION_ID=us-central1
adk deploy agent_engine \
        --project=$PROJECT_ID \
        --region=$LOCATION_ID \
        --trace_to_cloud \
        --otel_to_cloud \
        --with_ui \
        --display_name="my-first-agent" \
        my_first_agent

Once Deployed you will see Agent Engine Instance in UI

You can test end to end Agent using Playground

You can see Traces ,sessions in Agent Engine Observability

C. Google Kubernetes Engine (GKE) Deployment

For robust enterprise applications, GKE provides full orchestration, container management, and advanced security.

1. Local Prerequisites & Tools

Ensure the following utilities are installed locally:

• Google Cloud SDK (gcloud)
• Kubernetes CLI (kubectl): Can be installed via gcloud components install kubectl

2. Project Alignment

Set your active gcloud project context:

gcloud config set project tt-labs-001

3. Artifact Registry Configuration

Create a standard Docker repository named gcr.io in the multi-region location us (or your target location):

gcloud artifacts repositories create gcr.io \
    --repository-format=docker \
    --location=us \
    --project=tt-labs-001

4. IAM Permissions for Cloud Build

Ensure Cloud Build service accounts have the Artifact Registry Writer (roles/artifactregistry.writer) role to push built container images:

PROJECT_NUMBER=$(gcloud projects describe tt-labs-001 --format="value(projectNumber)")
# Grant to legacy default Cloud Build service account
gcloud projects add-iam-policy-binding tt-labs-001 \
  --member="serviceAccount:${@cloudbuild.gserviceaccount.com">PROJECT_NUMBER}@cloudbuild.gserviceaccount.com" \
  --role="roles/artifactregistry.writer"
# Grant to Cloud Build service agent
gcloud projects add-iam-policy-binding tt-labs-001 \
  --member="serviceAccount:@gcp-sa-cloudbuild.iam.gserviceaccount.com">service-${PROJECT_NUMBER}@gcp-sa-cloudbuild.iam.gserviceaccount.com" \
  --role="roles/artifactregistry.writer"

5. Runtime Permissions (Workload Identity)

Map the Kubernetes Service Account (KSA) to a Google Service Account (GSA) with the necessary roles to call Vertex AI:

# 1. Create a Google Service Account
gcloud iam service-accounts create adk-gke-sa \
    --description="Service account for ADK GKE agent" \
    --display-name="adk-gke-sa" \
    --project=tt-labs-001

# 2. Grant the Vertex AI User role
gcloud projects add-iam-policy-binding tt-labs-001 \
    --member="serviceAccount:adk-gke-sa@tt-labs-001.iam.gserviceaccount.com" \
    --role="roles/aiplatform.user"
# 3. Allow KSA to impersonate GSA
gcloud iam service-accounts add-iam-policy-binding adk-gke-sa@tt-labs-001.iam.gserviceaccount.com \
    --role="roles/iam.workloadIdentityUser" \
    --member="serviceAccount:tt-labs-001.svc.id.goog[default/default]" \
    --project=tt-labs-001
# 4. Annotate the default KSA with GSA email
kubectl annotate serviceaccount default \
    iam.gke.io/gcp-service-account=adk-gke-sa@tt-labs-001.iam.gserviceaccount.com \
    --overwrite
# 5. Restart your deployment to apply credentials
kubectl rollout restart deployment/my-first-agent

6. GKE Deployment Script

Create a script deploy_gke.sh in your agent folder:

#!/bin/bash
PROJECT_ID=tt-labs-001
LOCATION_ID=us-central1
CLUSTER_NAME=adk-demo

adk deploy gke \
  --project=$PROJECT_ID \
  --region=$LOCATION_ID \
  --cluster_name=$CLUSTER_NAME \
  --service_name=my-first-agent \
  --app_name=my_first_agent \
  --trace_to_cloud \
  --otel_to_cloud \
  --with_ui \
  my_first_agent

5. Summary & Conclusion

Deploying Agent Development Kit (ADK) agents on Google Cloud Platform offers a versatile path depending on your target application scale:

• Use Vertex AI Agent Engine for a purely serverless, low-ops, Python-first workflow that integrates natively into Vertex AI tools without Docker management.
• Choose Google Cloud Run when you need custom containers, rapid scale-to-zero autoscaling, or want to host the companion ADK Web UI for rapid prototyping.
• Leverage Google Kubernetes Engine (GKE) for enterprise-grade production workloads, enabling integration with active service meshes, advanced network policies, and persistent container management.

No matter the hosting choice, the ADK CLI simplifies packaging and configuration, allowing developer focus to remain entirely on building smarter agents.

Agent Development Kit (ADK) Deployment Guide was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

The short answer: all of the above, sort of. This guide untangles it in plain language — no deep tech required. By the end you’ll know what Gemini Enterprise actually is, how its pieces fit together, how a company puts its own AI agents inside it, and where its limits are.

First, the name confusion (let’s clear it now)

Google renamed and merged a few products, which is the #1 source of confusion. Here's the simple history:

So what is Gemini Enterprise?

Think of Gemini Enterprise as Google's all-in-one AI workplace for companies.

Here's an analogy. Imagine your company hired a brilliant new assistant who:

• has read every document in your company (Drive, SharePoint, Jira, Confluence, ServiceNow…),
• only shows each employee what they're allowed to see (no leaking the CEO's salary to an intern),
• can do tasks, not just answer questions — research a topic, pull a report, kick off a workflow,
• and lets your team build more specialized assistants ("agents") for specific jobs.

That bundle — the smart assistant, the company-wide knowledge, and the toolkit to build your own agents — is Gemini Enterprise.

The big shift in 2026: Google moved from selling "a chatbot" to selling "a platform for AI agents." An agent is just an AI that can take actions and complete multi-step work, not only chat. Gemini Enterprise is Google's home for building, running, and managing those agents at company scale.

The structure: 2 main pillars (+ a few bundled goodies)

People often say "three pillars," and Google does market three. But for a beginner the cleanest mental model is two pillars you'll actually touch, plus extras that ride along. Let's go through them.

Pillar 1 — Gemini Enterprise App (for everyone in the company)

This is the front door — the part normal employees use. No coding.

It's a single chat-and-search box where an employee can:

• Ask questions and get answers grounded in real company data, not generic internet guesses.

• Search across everything at once — Drive, SharePoint, Jira, Confluence, ServiceNow, and more — with permissions respected.
• Use ready-made agents (Google calls this the "taskforce"), like:
• Deep Research — explores a topic and writes a full report.
• NotebookLM Enterprise — a study/research assistant over your docs.
• Data Insights — answers questions straight from your BigQuery data.
• Build their own simple agents using a no-code workbench (more on this below).

Who it's for: every knowledge worker — marketing, HR, sales, ops. If you can use Google Search or ChatGPT, you can use this.

Pillar 2 — Gemini Enterprise Agent Platform (for developers)

This is the engine room — what used to be Vertex AI. It's where technical teams build, deploy, scale, and govern serious custom agents.

It gives developers:

• Access to 200+ AI models (Google's Gemini models and third-party ones) via Model Garden.
• Tools to build agents with code (using Google's Agent Development Kit, "ADK").
• The plumbing to run agents reliably — hosting, security, monitoring, version control.
• Open standards so agents can talk to each other (A2A) and connect to tools and data (MCP).

Who it's for: engineering teams building production-grade, custom AI agents.

The bundled extras (the "third pillar" and friends)

• Gemini Code Assist — an AI pair-programmer for your developers. It's included in Standard/Plus editions, which is why some people count it as a pillar. Think of it as a perk, not a separate product.
• Gemini Enterprise for Customer Experience — a specialized version for customer-facing shopping and support agents (this is the official third pillar). Most companies start without it.

Easiest way to remember it:

App = where employees use AI. Agent Platform = where developers build AI. Code Assist / CX = bonus tools that come along for the ride.

What can enterprises actually do with it? (Capabilities)

Here's the practical capability checklist a business cares about:

CapabilityWhat it means in plain EnglishCompany-wide searchOne search box across all your apps and documents.Permission-aware answersEach person only gets answers from data they're allowed to see.Pre-built agentsDeep Research, NotebookLM, Data Insights — ready on day one.No-code agent builderNon-technical staff can create their own assistants.Custom code agentsDevelopers build advanced, multi-step agents.ConnectorsPlug into Drive, SharePoint, Jira, Confluence, ServiceNow, OneDrive, Docusign, etc.Enterprise securitySingle sign-on, access controls, compliance. Google does not train its models on your data.Private networkingTraffic can stay off the public internet for sensitive setups.

How do you register an agent? (The part everyone asks)

This is where Gemini Enterprise really shines: it's not just Google's agents — you can bring your own and make them appear right inside the app for employees. Everything lives in one place called the Agent Gallery.

The Agent Gallery — your company's "app store" for agents

When employees open Gemini Enterprise, they see the Agent Gallery, organized into four shelves:

1. Made by Google — ready-to-use agents (Deep Research, etc.).
2. From your organization — agents your company built or approved.
3. Your agents — ones you personally created.
4. Marketplace — agents from third-party vendors (request access → admin approves).

Three ways to get an agent into the Gallery

Option A — Build it with no code (easiest) For non-technical users. Inside the app, click + Create agent, then use the Agent Designer / Agent Studio:

• Give it a name and instructions (in plain English),
• pick a model,
• connect a data source or tool,
• save and share.

Done — no developer needed. Great for "an HR policy assistant" or "a sales FAQ helper."

Option B — Register a custom-built agent (for developers) Your developers build an agent with code (ADK) and deploy it (e.g., on Cloud Run). Then an admin registers it:

1. Go to the Gemini Enterprise page in the Google Cloud console.
2. Open your app → click Agents → Add agent.
3. Choose how it was built:

• Custom agent via Agent Platform (for ADK agents)
• Custom agent via A2A (paste the agent's "agent card" JSON)
• Custom agent via Dialogflow (for Dialogflow-based agents)

Provide any required authentication details. ( refer Video Demo )

The agent now appears in the Gallery for the right people.

Option C — Add a Marketplace agent (buy, don't build) An admin picks a vendor agent from the Marketplace, enters the auth details, and publishes it. Employees can also request access themselves.

Sharing with your team

Once an agent exists, sharing is simple: in the Gallery, find it under Your Agents → ⋮ (three dots) → Share, then add email addresses, Google groups, or a shareable link.

⚠️ Important beginner gotcha: sharing an agent also shares access to the data sources it uses. Double-check what's connected before you hit share.

Pricing in one glance (so there are no surprises)

Gemini Enterprise is sold per user, per month:

EditionPrice (per user/month)Best forBusinessfrom ~$21Small teamsStandard~$30 (annual) / ~$35 (no commitment)Most knowledge workersPlus~$50 (annual) / ~$60 (no commitment)Power users & builders

Two things beginners get wrong about cost:

1. Everyday app use is included. Each plan comes with a pooled daily query quota

1. Custom agents on the Agent Platform are a separate, pay-as-you-go cost. The moment developers build and run their own agents on the Agent Platform, you pay for the actual AI usage (per token) and compute — on top of the seats. This is where bills can grow, so watch it.

Rule of thumb: Seats cover your employees using the app. Building custom agents is where extra, usage-based costs begin.

Advantages (why companies are adopting it)

• ✅ One place for everything — search, chat, and agents, instead of five disconnected tools.
• ✅ Grounded in your data — answers come from your documents, with permissions respected.
• ✅ Anyone can build agents — the no-code builder removes the "wait for engineering" bottleneck.
• ✅ Open standards (A2A, MCP) — you're not locked into a closed system; agents can interoperate.
• ✅ Enterprise-grade trust — SSO, access control, compliance, and your data isn't used to train Google's models.
• ✅ Predictable everyday cost — in-app usage won't surprise you with per-question charges.

Limitations (the honest part)

• ⚠️ Pooled query limits are small — a shared ~160 queries/day across a whole org runs out fast for big teams, nudging you toward the (paid-by-usage) Agent Platform.
• ⚠️ Real power needs the Agent Platform — the no-code builder is great for simple agents, but serious use means developers, code, and consumption billing.
• ⚠️ Cost can sneak up — custom agents bill per usage; without monitoring (FinOps), a heavy deployment gets expensive.
• ⚠️ Setup isn't trivial for advanced cases — connectors, permissions, private networking, and agent registration take real planning.
• ⚠️ Best with the Google ecosystem — it connects to many third-party tools, but you get the most value if you're already on Google Cloud / Workspace.

Summary

• Gemini Enterprise = Google's all-in-one AI platform for companies. It used to be split across Agentspace + Vertex AI; now it's one roof.
• Two pillars to remember: the App (employees use AI) and the Agent Platform (developers build AI). Code Assist and the Customer Experience edition are bonus pieces.
• You can bring your own agents — build with no code, register a custom-built one, or grab one from the Marketplace — all surfaced in the Agent Gallery.
• Pricing is per-seat, everyday app use is included (you get throttled, not billed, past the daily quota), but custom agents on the platform cost extra by usage.
• Great for unifying company knowledge and democratizing AI; watch out for naming confusion, small pooled limits, and usage-based costs on advanced builds.

Want to go deeper? The natural next steps are: (1) building your first no-code agent in the App, and (2) building a custom agent with Google's ADK and registering it via A2A. Those each deserve their own hands-on tutorial.

References & further reading

Official Google Cloud documentation

• What is Gemini Enterprise?
• Gemini Enterprise app (product page)
• Gemini Enterprise app FAQs
• Compare editions of Gemini Enterprise
• Quotas and overages
• Quotas and system limits
• Gemini Enterprise Agent Platform pricing
• Vertex AI → Gemini Enterprise Agent Platform name changes
• View Gemini Enterprise costs (billing reports)

Registering & managing agents

• Agent Gallery overview
• Register and manage an ADK agent
• Register and manage an A2A agent
• Register and manage a Dialogflow agent
• Create an agent with Agent Designer (no-code)

Background & announcements

• The new Gemini Enterprise: one platform for agent development — Google Cloud Blog
• Google folds Agentspace into Gemini Enterprise — The Register

Note: prices and quotas are list values as of June 2026 and may change — always confirm current figures on the official Google Cloud pricing pages before making decisions.

Google Gemini Enterprise Agent Platform Tutorial: Registering ADK Agents was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is MCP?

MCP (Model Context Protocol) is an open protocol — created by Anthropic — that standardizes how an AI application connects to external tools and data.

Think of it like a USB-C port for AI agents. Before USB-C, every device had its own connector. Before MCP, every integration between an LLM and a system (a database, an API, a file store) was custom glue code that you wrote and maintained yourself.

With MCP, the contract is fixed:

• An MCP Server exposes capabilities — tools (actions the agent can call), resources (contextual data like schemas), and prompts (predefined inputs).
• An MCP Host is your AI application (an ADK agent, Gemini CLI, Claude, VS Code).
• An MCP Client lives inside the host and speaks the protocol to the server.

Your agent doesn’t need to know how BigQuery works. It just needs to speak MCP to a BigQuery MCP server, and it instantly gets a set of well-described tools it can call.

You can directly jump into the demo using below video

Local MCP vs Remote MCP

There are two ways an MCP server can run, and the difference matters a lot for production:

Local MCP is the classic developer experience: you pip install or npm install a server, it runs as a subprocess on your laptop, and the agent talks to it over stdio. Great for prototyping. Painful at scale — you are now responsible for installing, patching, securing, and running that server for every service you want to reach.

Remote MCP flips this. The server runs as a managed HTTP endpoint. You just point your agent at a URL and authenticate. No process to babysit.

What Google announced — and why it changes the picture

Google announced official MCP support across Google and Google Cloud services: a fleet of fully-managed, remote MCP servers with enterprise-ready HTTP endpoints (supporting MCP spec version 2025-11-25).

Announcing official MCP support for Google services | Google Cloud Blog

During Cloud Next 2026 MCP service list is Extended

Google-managed MCP servers are available for everyone | Google Cloud Blog

This is the important shift. Previously, to give an agent access to GCP, you’d hunt down community/open-source MCP servers, deploy them, and keep them alive. Now Google runs them for you, behind stable *.googleapis.com/mcp endpoints, with the enterprise plumbing already wired in:

• Google Cloud IAM for access control — the agent only gets what its identity is allowed.
• Audit Logging for observability and compliance.
• Model Armor for defense against agentic threats like indirect prompt injection.
• Cloud API Registry / Apigee API Hub to discover trusted tools.

A large catalog of services is already live (GA) as remote MCP servers, including:

Supported products | Google Cloud MCP servers | Google Cloud Documentation

Why this matters for agent builders: the integration layer is no longer your problem. You write the agent logic — the model, the instructions, the routing — and the GCP services show up as governed tools over a URL. That’s the whole picture changing: from “build and run integrations” to “point and authenticate.”

How you build an agent with this

The mental model is three steps:

1. Enable the API for the service you want (one gcloud command).
2. Connect your agent to that service’s remote MCP endpoint, passing an auth token.
3. Let the model route — the LLM picks which MCP tools to call based on your instructions.

Let’s do exactly that.

Part 2 — A working example: an ADK agent with BigQuery + Cloud Run access

We’ll build a Google Cloud expert agent using Google’s Agent Development Kit (ADK) that connects to two official remote MCP servers — BigQuery and Cloud Run — over HTTP.

Step 0 — Enable the APIs

Each MCP server is fronted by its product’s API. Enable the ones you need:

gcloud services enable bigquery.googleapis.com
gcloud services enable run.googleapis.com

You need the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin) to enable APIs.

Then make sure you’re authenticated locally so your agent can mint tokens via Application Default Credentials:

gcloud auth application-default login

Step 1 — Get credentials for the remote MCP servers

Remote MCP servers are just authenticated HTTP endpoints. We use Application Default Credentials (ADC) to get a short-lived OAuth token and the active project, then pass them as headers.

# tools.py
import google.auth
import google.auth.transport.requests
from google.adk.tools.mcp_tool import McpToolset
from google.adk.tools.mcp_tool.mcp_session_manager import StreamableHTTPConnectionParams

# Official, fully-managed remote MCP endpoints
BIGQUERY_MCP_URL = "https://bigquery.googleapis.com/mcp"
CLOUD_RUN_MCP_URL = "https://run.googleapis.com/mcp"

def _get_gcp_credentials():
    # ADC: uses `gcloud auth application-default login` locally,
    # or the runtime service account in production.
    credentials, project_id = google.auth.default(
        scopes=["https://www.googleapis.com/auth/cloud-platform"]
    )
    auth_request = google.auth.transport.requests.Request()
    credentials.refresh(auth_request)
    return credentials, project_id

The two headers that matter on every call:

• Authorization: Bearer <token> — proves who the agent is.
• x-goog-user-project: <project_id> — which project to bill/operate against.

This is where IAM does the gatekeeping: the agent can only do what this identity is permitted to do. No extra access layer to build.

Step 2 — Wrap each MCP server as an ADK toolset

ADK’s McpToolset connects to a remote MCP server over streamable HTTP and automatically turns its tools into agent tools.

# tools.py (continued)

def get_bigquery_toolset():
    credentials, project_id = _get_gcp_credentials()
    return McpToolset(
        connection_params=StreamableHTTPConnectionParams(
            url=BIGQUERY_MCP_URL,
            headers={
                "Authorization": f"Bearer {credentials.token}",
                "x-goog-user-project": project_id,
            },
        )
    )

def get_cloud_run_toolset():
    credentials, project_id = _get_gcp_credentials()
    return McpToolset(
        connection_params=StreamableHTTPConnectionParams(
            url=CLOUD_RUN_MCP_URL,
            headers={
                "Authorization": f"Bearer {credentials.token}",
                "x-goog-user-project": project_id,
            },
        )
    )

Notice how little code this is. The entire BigQuery and Cloud Run tool surface — listing datasets, running SQL, exploring schemas, listing services, inspecting revisions, checking traffic splits — comes from pointing at a URL with a token. That’s the payoff of managed remote MCP.

Step 3 — Build the agent and let the model route

Now we assemble an LlmAgent, hand it both toolsets, and write instructions that tell the model when to use which tool. The LLM does the routing at runtime.

# agent.py
import logging
from dotenv import load_dotenv
from google.adk.agents import LlmAgent
from .tools import get_bigquery_toolset, get_cloud_run_toolset
logging.basicConfig(level=logging.INFO)
load_dotenv()
bigquery_toolset = get_bigquery_toolset()
cloud_run_toolset = get_cloud_run_toolset()
root_agent = LlmAgent(
    model="gemini-2.5-pro",
    name="adk_gcp_mcp_agent",
    instruction="""
    You are a Google Cloud expert assistant with access to GCP services via MCP.
    Tools available:
    - BigQuery MCP: query datasets, list tables, run SQL, explore schemas
    - Cloud Run MCP: list services, inspect revisions, check traffic splits
    Route questions to the right tool:
    - Data/SQL questions     -> BigQuery
    - Serverless workloads   -> Cloud Run
    Always be precise and show which tool you used to get the answer.
    """,
    tools=[bigquery_toolset, cloud_run_toolset],
)

Step 4 — Run it

With ADK installed, launch the dev UI from the project root:

adk web

Now ask it natural-language questions and watch it pick the right MCP server:

• “List the datasets in my project and show the schema of the events table.” → routes to BigQuery MCP.

• “Which Cloud Run services are deployed in project <project id> in region <region>” → routes to Cloud Run MCP.

The agent calls the managed remote MCP server, the server executes against your real GCP resources (governed by IAM), and the model explains the result.

Scaling out

Adding more GCP services is now trivial — it’s the same three-line pattern with a different URL. Want logs and metrics too? Add toolsets pointed at logging.googleapis.com/mcp and monitoring.googleapis.com/mcp, list them in tools=[...], and extend the instruction. No new infrastructure, no servers to run.

Wrapping up

We can build cool Support Agent with Google Remote MCPs check demo here

Manage Your Entire Google Cloud with AI Agents | AgenticOps Demo

The headline isn’t “Google supports MCP.” It’s where the work moved:

• Before: you found, deployed, secured, and ran integration servers for every GCP service.
• Now: Google runs managed remote MCP servers behind stable endpoints, with IAM, audit logging, and Model Armor already in place. You write agent logic and point at a URL.

For production agents, that’s the difference between maintaining plumbing and shipping capability. Pick the services you need from the supported products list, enable the APIs, wrap each endpoint as a toolset, and let the model do the routing.

References

• Announcing official MCP support for Google services
• Google Cloud MCP overview
• Supported products
• Enable / disable MCP servers
• Google MCP examples on GitHub

Building AI Agents with Google Cloud Managed MCP Servers and ADK was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

How We Built AgenticOps Using Google ADK, MCP Servers & Gemini.

By Vishal Bulbule — Google Developer Expert, founder of TechTrapture

We made writing software agentic. Your editor finishes your code, refactors across files, writes tests, opens PRs. But the moment something goes wrong in production — the moment it actually matters — all of that intelligence vanishes. You’re back to seventeen tabs, dashboards, gcloud … | grep, and the one engineer who happens to know the system.

AgenticOps closes that gap. It’s a single AI agent that sits across your whole Google Cloud — health, logs, errors, security, and cost — and you operate it by talking to it. In plain English. From any device, anywhere, 24×7.

This post is a tour of what it does, not how it’s wired. If you’d rather watch it run live, jump to the demo:

▶️ Watch the full demo:

Capability 1 — Troubleshoot an incident in one sentence

The first fifteen minutes of every incident aren’t skilled work. They’re hunting — stitching a story together across logs, metrics, traces, and deploy history. AgenticOps collapses that into a question:

“Any errors in the last hour — and what changed?”

It pulls the logs, finds the error signature, correlates it with recent deploys and config changes, and hands you a verdict: what broke, when it started, and the most likely cause.

Capability 2 — Ask your cloud what it costs, and why

The other 3 a.m. of operations is the monthly bill. Spend jumps 38%, and nobody can explain it without three days of digging through billing exports.

“Why did my bill jump vs last month?”

I can request report on email as well

Email Report

It reads your billing data, ranks the top services by spend, and points straight at the driver — “this batch job started scaling to 40 instances,” “Vertex AI usage doubled.” A three-day investigation, answered in a sentence.

Because the same agent sees both cost and operations, “why did spend jump?” and “what’s running?” finally stop being two separate investigations.

There’s also a FinOps dashboard for the at-a-glance view — spend trend, top services, idle resources you can cut — with the agent one click away to explain anything on it.

FinOps dashboard: spend trend + top services + idle-resource findings]

Capability 3 — “Who changed what?” answered instantly

During an incident, the most valuable question is usually what changed — and who changed it. Normally that’s a hand-written trawl through admin activity logs.

“Who changed what in the last 24 hours?”

Capability 4 — A security posture read on demand

“Is anything publicly exposed in this project? Any risky IAM?”

A quick read on your exposure — public buckets, over-broad IAM, open surfaces — in plain language, when you want it, not buried in a once-a-quarter report.

Capability 5 — The RCA, written and emailed for you

After the fire is out, there’s still the postmortem — important, and always deprioritized.

“Email me the postmortem for this incident.”

The agent drafts a clean, readable RCA from what it just investigated — timeline, root cause, impact, remediation — and emails it to you, ready to forward to your team or leadership.

Capability 6 — Careful, gated actions (when you want them)

Reading is the default. But when you’re ready to act, AgenticOps can do that too — restart an instance, rerun a failed job, retrigger a pipeline.

“Rerun my failed Jobs.”

Every change is confirm-first: the agent shows you the plan and blast radius, and only acts after you approve. It proposes; you nod.

The everyday questions, too

It’s not only for emergencies. The same agent answers what you ask all day:

• “Is everything up and healthy?” — a one-line status across your services.
• “How many jobs ran today, and did any fail?” — without remembering a single command.
• “Give me a morning status of my cloud.” — health, errors, and spend in one briefing.

“Would you really trust an AI in your production cloud?”

Right question. The trust model is built in, not bolted on:

• Runs on your own Google IAM. Nothing new to grant, no service-account keys to hand out. Each person sees and does only what their own Google account already can. The agent isn’t a new privileged identity — it’s you, faster.
• Read-only by default. Out of the box it can look, not touch.

Now we can enable Action Mode for any Write Operation

Now see Actions !!

• Every action is confirm-first and audited. If your account can’t do it, the agent can’t either — and there’s a record of who asked for what.

It’s not a robot loose in your console. It’s a careful operator that shows its work and waits for the nod.

The tech stack (the short version)

For the engineers asking “what’s under the hood” — without the implementation weeds:

• Google ADK (Agent Development Kit) for the multi-agent core: a coordinator that routes each question to a specialist — health, logs & errors, security, cost, reporting, and a gated remediation agent.
• Google Cloud’s official remote MCP servers as the agent’s eyes — Cloud Logging, Error Reporting, Monitoring, Cloud Run, Compute, Asset Inventory, Resource Manager, and BigQuery (over your billing export, for cost). Google hosts them; the agent connects securely and read-only.
• Gemini as the reasoning model, running in your Vertex AI — so your data, and even the model calls, stay in your project.
• A clean React + Firebase web client (mobile-first), with Cloud Run hosting the agent service.

One agent codebase, your IAM, your Vertex, your data boundary. That’s the whole point.

I walk through the architecture and a full live run in the video — that’s the place to see how it actually behaves.

This is where operations is going

Five years from now, no one will grep through logs at 3 a.m. to answer “what changed?” — the same way no one today writes boilerplate by hand when an agent will do it in seconds. The teams that get there first will run leaner, sleep better, and spend less.

Dev went agentic. Ops is next.

See it on your own cloud

I’m opening AgenticOps to a small group of teams who feel this pain the most — SRE, platform, and FinOps folks done with operating their cloud the hard way.

→ Book a walkthrough: techtrapture.com/contact or Contact us through sales@techtrapture.com

AgenticOps, by TechTrapture — agentic AI for cloud operations and FinOps on Google Cloud. Dev went agentic. Ops is next.