Intro

In my previous post I introduced Antigravity Skills. This post will show how you can create customized Skills to power your personal workload. For this post, I’ll focus on Skills a software developer might use to automate time consuming CI/CD tasks.

There are a lot of manual steps (and mistakes) a developer has to execute in the CI/CD lifecycle … so we’re going to write an Antigravity Skill that an Agent can use to automate all of these steps!

If you’re fairly comfortable with Skills, just skip to the AI Developer skill in my Antigravity Skills repo for the full skill definition. Otherwise, keep reading for a walkthrough.

GitHub - georgemao/antigravity-skills: Sample agents for use by Antigravity

Skill Requirements

My goal is to automate all of the tasks that are performed by a human. The tasks fall into two categories:

All of the actions to manage my Git repo

These are branching, pushing code, writing, documenting, & merging Pull Requests.

• These actions manage an external 3rd party tool so it’s appropriate to call it’s MCP server and execute Tools to perform these tasks.
• We’ll enable the Github MCP server in the Skill.

All of the human actions to approve code

This are peer reviews for bugs, performance, security issues, and ensuring proper documentation.

• These require context and developer knowledge that only a LLM can perform
• We’ll write detailed instructions in the Skill for the Agent to follow

Skill Overview

Here’s the AI Developer skill for reference.

Define the Skill via Front matter

The first section defines the skill’s name , description , tools, and model.

• The name attribute is not important — you can explicitly execute a skill by using the @[skill name] syntax.
• The description attribute is what Antigravity will use to match prompts to Skills and it describes when the Skill should be activated.
• The tools attribute enables the Skill to use 3rd party MCP servers. In our case we want to enable the GitHub MCP server. Note the syntax: mcp_[name]_*. The * indicates that all Tools available on the MCP server are allowed to be used.
• The model attribute tells the Agent to use a specific model. Choose one that matches your workload needs.

---
name: ai-developer
description:
  Use this skill to help a developer automate code check-ins to Github. You are an expert developer who can review code, suggest code improvements, produce documentation, create Pull Requests and merge them. You have access to the codebase and can make changes to the code.
tools:
  - mcp_github_*
model: gemini-3.5-flash
---

Define the Workflow — how you want the Skill to proceed

This is important because we want to be prescriptive and make sure the Agent follows a very specific workflow. This is the main reason to use a Skill.

At a high level, here's the workflow — it’s 5 steps but each step has detailed instructions on how to complete the step. I also specify the Scope — which tells the Agent exactly what it should review — in this case only code and only the main branch.

Let’s take a look at each step.

## Workflow
First, ask the user what branch to review. If they don't specify, assume main branch only and do not look at other branches.
Then perform the following steps:

### 1. Review code
### 2. Check if the changes address any open Issues
### 3. Create Pull Request
### 4. Merge Pull Request
### 5. Cleanup

#### Scope
Only review source code. Skip the docs, config files, and package files.
Always use the Github MCP server.
Only review code against the main branch. Do not look at other branches.

#1 Review Code

I want the Agent to perform a comprehensive code review that follows 6 distinct phases. Each phase should pause and check point with the user to summarize the results.

### 1. Review code
Review the local code provided by the user. Execute a 6 step review as defined below. Checkpoint with the user after each step. 

1. Identify bugs in the code.
2. Identify security issues in the code.
3. Identify performance issues in the code.
4. Identify missing features in the code.
5. Identify missing documentation in the code.
6. Update the Readme if needed.

For each checkpoint:
- Make sure to show the plan for changes
- Ask the user if they agree to the changes.
**Never make changes to the code until the user agrees.**
- Summarize your review using the following format:

I design the actual check point to summarize and use a very specific format as follows:

#### 1.1. Step X - [Description of step]
Summarize the improvements you would like to see. Use the following format:

  • Summary: A high-level overview of the review.
  • Findings:
      • Critical: State any breaking changes or issues that must be addressed.
      • Improvements: Suggestions for better code quality.
      • Nitpicks: Formatting or minor style issues (optional).
  • Conclusion: Clear recommendation.

#2 Check if the changes address any open Issues

At this point the 6 phase review is complete. I want to check if we’ve addressed any open issues in the Git repo so we can close them.

## 2. Check if the changes address any open Issues
Check the Git repo and see if the changes address any open issues. 
- If so, suggest closing them. Add a note to the PR that these issues have been addressed.

#3 Create Pull Request

Now we are ready to Create the Pull Request. Here I instruct the Agent to create a new branch and a PR in that branch. I also tell it to reference any issues that were addressed as part of our research in step #2.

### 3. Create Pull Request
Once the user agrees to the changes, create a *NEW* branch and then a Pull Request for the code. 
Use the existing configured Git repository.
Provide a short title and description of the changes, with reference to any issues that were addressed

I also provide a very specific format for the Pull Request to be written in:

Use the following format:
#### Summary *(What does this PR do?)*
- [ ] Bug fix
- [ ] New feature
- [ ] Refactoring
- [ ] Documentation update

#### Description
*(Describe this PR)*
- [ ] Summarize the changes
- [ ] Explain the impact

#### Checklist:
*(Please delete options that are not relevant.)*

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream modules

#4 Merge Pull Request

This is the easiest part. I instruct the Agent to merge the PR, however I make sure to ask if the user agrees first.

### 4. Merge Pull Request
Ask if we should save the code permantly. 
If the user agrees, merge the Pull Request. If not, just move to clean up.

#5 Cleanup

If the merge is successful, I ask the Agent to switch my local branch back to main. Finally, summarize the productivity gains from all of the automation.

### 5. Cleanup
After the review, ask the user if they want to switch back to the default branch (e.g.,  main  or  master ).
Summarize and give the user an estimate on amount of time you saved them.

A test run

First make sure you configure a MCP server that has permissions to interact with your Git repo. You can do this via a Git PAT with permissions to Write Branches, Pull Requests, and Merge.

For the example below I’m using my mcp-on-cloudrun repo:

GitHub - georgemao/mcp-on-cloudrun

Start the Antigravity CLI and Invoke the skill by entering: Help me check in this code. You should see the Agent begin walking through the 6 steps. As instructed, each step includes a checkpoint that summarizes the code review, a plan for changes, and a final question to proceed. Here’s a sample of the first 3 steps:

Next, the Agent creates the Pull Request and writes the details per the template provided. The PR links any Issues that were addressed and prepares for the final merge to main.

As the last step, the Agent summarizes all actions taken and provides an estimated time saved.

Tips & Best Practices

Use a Skill to create a precriptive workflow for an Agent to follow.

• Be clear and define exactly what you expect the Agent to do
• Tightly scope the Skill so it only uses resources you expect, such as MCP servers, Git Repos
• Provide templates so generated artifacts follow a standard

Intro: Antigravity Skills for Developer automation was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

An Agent Skill is a set of instructions that provide prescriptive guidance for how an AI tool should use a LLM to complete a task. The only required file isSKILL.md — this file defines the skill, its name, description, and execution instructions.

my-skill/
├── SKILL.md          # Required: metadata + instructions
├── scripts/          # Optional: executable code
├── references/       # Optional: documentation
├── assets/           # Optional: templates, resources
└── ...               # Any additional files or directories

Skills are not APIs and do not need to be hosted on a server. A Skill has two major components:

• Frontmatter based Metadata section — this describes the Skill and helps your AI tool select the right Skill to use based on a Prompt
• Body — this includes the instructions and procedure the AI tool should follow

Skills are loaded progressively. This means metadata is indexed to allow Skill selection but the body (and all optional resources) is not loaded until the specific skill is choosen. This helps reduce context bloat, improve performance, and reduce token usage.

Execute a Skill

This article will cover the new Google Antigravity Developer AI IDE/CLI. Check out some sample skills in my Antigravity Skills github repo:

GitHub - georgemao/antigravity-skills: Sample agents for use by Antigravity

Install these in your ~/.gemini/skills directory to make them available globally in any workspace. Let’s start with the kid-code-reviewer skill to see how skills work. It’s a single agent skill that operates like this:

Read the SKILL.md file to understand the Skill’s workflow:

Act as the child and argue about the code change

Give up

Pretend to commit to Github

Start Antigravity CLI from any sample code base and enter the prompt below. I’ll use sample code from this code base: MCP on Cloud Run.

Use my kid code reviewer to provide a code review.

You should see Antigravity activate the kid-code-reviewer skill and read the source code for review. A few seconds later, the review is done and the response simulates a child.

Interact with the Skill a few times and eventually it’ll give up and try to commit to Git.

This interaction followed the prescriptive workflow specified in SKILL.md. Type /context to review the overall token usage for this interaction.

Execute multiple Skills

Now lets work on a use case where we need to call multiple skills. We’ll invoke two Skills sequentially like this:

Check out the solutions-architect-agent SKILL.md file. Pay attention to the Frontmatter (metadata) section at the top. In particular, the tools attribute specifies that this skill can use a single MCP server: mcp_google-developer-knowledge_*. The * means the skill can use all tools hosted by that MCP server.

In the body, we provide instructions to act as a Solutions Architect & use the Developer Knowledge MCP server to recommend solutions according to Google best practices. It also instructs the agent to respond using a very specific format. Finally, it provides a summary of the analysis and how much human time was saved. This is a long running operation and needs to reach out to an external service.

In the Antigravity CLI, enter this prompt:

Investigate this error with the solutions-architect-agent skill and help suggest a fix: HTTP 429
The request was aborted because there was no available instance. At the same time, use my kid code reviewer to provide a code review.

Antigravity loads the loads the solutions-architect-agent and the kid code reviewer skill just like before. It immediately begins execution of both Skills. A code review of the application code (server.py) begins and a call to the Developer Knowledge MCP Server is launched.

After a short period, Antigravity tells us it investigated the errors and suggests the solution, using the format specified. It also summarizes the work and says this could have saved 15–20 minutes of human time.

Next, Antigravity returns the code review results for server.py.

The Solutions Architect skill is complete at this time, however the kid code reviewer is not complete because it is instructed to argue with you a few times.

Note how you cannot interact with the kid code reviewer until both of the skills execute to their first/final checkpoint. This is because work executes sequentially by the main Agent. Your main conversation context also increases across the entire scope of work. Try interacting with the code reviewer:

Now type /context — pay attention to System Tools (16k tokens) where the bulk of the tokens are used.

Use Sub agents to Execute Skills

Antigravity allows you to execute Skills in parallel by using sub agents. A Sub agent also ensures the context is contained to the sub agent and keeps the main conversation clean.

Try the Solutions Architect skill as a sub agent:

In a subagent: Investigate this error with the solutions-architect-agent skill and help suggest a fix: HTTP 429
The request was aborted because there was no available instance.

Note the DefineSubagent process kicks off an Agent (solutions_architect) to perform the work, leaving the main conversation ready for more work. While the first Subagent is working, create another Subagent to perform the code review:

In a different subagent: use my kid code reviewer to provide a code review.

Now both agents are working through their tasks. The main agent will monitor the subagents for status. Type /agents to check how they are doing:

After some time both sub agents will reported back with their status. The Solutions architect subagent is complete and wrote results to an artifact. Type /artifact to review it.

The Code review subagent is ready to continue its conversation with you. Once you respond, the main conversation invokes a SendMessage call to the subagent.

You can also combine to a single prompt to execute both skills immediately in parallel like this:

In a subagent: Investigate this error with the solutions-architect-agent skill and help suggest a fix: HTTP 429
The request was aborted because there was no available instance.

In a different subagent: use my kid code reviewer to provide a code review.

Whats next?

This post covered Antigravity Skills and how to use subagents. In my next post I’ll cover how to coordinate multiple agents and pass output from one agent to another. In the mean time, read all about Agentic AI design patterns in the Google Cloud Architecture Center.

Deep Dive: Antigravity Agent Skills was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

It took over a year of GCP experience to learn these important GCP tricks. Here’s a list that all Google Cloud Developers & Architects should be aware of!

VPC Service Controls don’t protect VPC resources

I assumed that the VPC Service Controls (SC) feature protects resources in a VPC from unauthorized network access. Here’s a snippet from the docs:

While IAM enables granular identity-based access control, VPC Service Controls enables broader context-based perimeter security, including controlling data egress across the perimeter.

VPC Service Controls lets you define security policies that prevent access to Google-managed services outside of a trusted perimeter, block access to data from untrusted locations, and mitigate data exfiltration risks.

For example, I thought I could protect services hosted on Google Compute Engine or Cloud SQL. Turns out that is not correct.

VPC SC only protects Google managed APIs by limiting network access to VPCs that are authorized.

VPC SC only protects “Google Managed APIs” or more accurately, the Control plane from being accessed by unauthorized VPC resources. This includes Google APIs used to manage your resources or fully managed services behind an API (ie. GCS, BQ, PubSub). It does not protect access to the Data plane — anything running on those resources.

For example, if you have a web server running on Google Cloud Engine — this is not protected by VPC SC. However, the Google APIs that manage the GCE VM can be protected.

Lastly, it gets a little confusing when we apply VPC SCs to fully managed Google Cloud services that expose ingress points. Cloud Run is a good example. Cloud Run Services expose a fully managed HTTP Endpoint — this is technically the data plane, however, because Cloud Run is fully managed in a GCP owned project, it is in scope for VPC SC protection.

To protect assets not in scope for VPC SC configure Firewall Rules with Network Tags and/or use Cloud NGFW.

Identity Aware Proxy (IAP) cannot be used with Cloud CDN

IAP is one of the coolest services I’ve used at GCP — it uses Google’s Zero-trust technology to control access to cloud-based resources without requiring a VPN. You can use IAP for SSH/RDP access or for HTTP/Web access to controlled resources. Many services support IAP, such as Google Compute Engine, Load Balancers, and Cloud Run.

One of the most common use cases is to use IAP to easily add AuthN/Z to a web app without managing another Identitfy Provider.

Web Apps are often deployed as a Single Page App (SPA) built with Angular or React and then served by a CDN (ie Cloud CDN). This accelerates content delivery and improves cost for delivery. You deploy a Load Balancer and target a Backend Cloud Storage Bucket with Cloud CDN enabled.

Cloud CDN provides highly performant edge caching of assets, which makes it incompatible with IAP, because IAP evaluates every request made.

You cannot enable Cloud CDN on any backend that has IAP enabled.

If you need edge caching, you should use Firewall rules or NGFW to control ingress (instead of IAP). Then deploy OAuth based access control on your application.

Cloud CDN cannot be used to secure access to a public Google Cloud Storage Bucket

Deploy your SPA assets to a Google Cloud Storage (GCS) bucket then serve them up via Cloud CDN. The client runs on web browsers and interacts with APIs. You get all the benefits of global distribution at highly cost effective price — with no servers to scale or manage.

Cloud CDN cannot access a private GCS bucket and enabling CDN on a Load Balancer does not secure the Backend bucket — the bucket must be public.

Instead, create two seperate domains. One for public assets and another for the secured portion, for example: assets.iamgeorge.com and app.iamgeorge.com

Beware of Pub Sub filtering details!

I wrote about Pub/Sub Tips & Tricks in a previous post — but there are two major things to watchout for:

• You cannot configure filters to inspect actual message data. Instead, PubSub inspects incoming message attributes. For example, you might publish a message that includes the attribute color: red to allow a subscriber to only recieve messages tagged with red.

You should almost always include attributes to allow for useful filtering.

{
  "data": "your payload here",
  "attributes": {
    "color": "red",
    ...
  }
}

• Filtering will reduce your costs. But not as much as you think. In most cases you will not see a direct percent reduction in cost because filtering is applied at the Subscription, not at the Topic. You pay Throughput charges for every message delivered to the Topic regardless of filtering configuration, but filtered messages do not incur any data transfer charges from Subscriptions to the subscriber.
• Lets say a Topic has 2 Subscriptions. If you don’t apply a filter on either Subscription, all messages published to the Topic will flow to both Subscriptions. You incur Throughput costs 3x (for the Topic + 2 Subs) plus 2x data transfer costs to the 2 Subscribers.
• If you apply a filter color:red on the first Sub and another filter color:blue on the second Sub you still incur 3x Throughput charges. However, data transfer charges are only incurred for messages matching the filter.

If you have Subscriptions with filters that have substantially disproprtional filtering percentages it’s usually better to create seperate Topics dedicated to each Subscription.

There is more than one way to obtain GCP Application Default Credentials (ADC)

Application Default Credentials is the process by which Google clients (SDKs, CLI, etc) search your system for local credentials used to access Google Cloud APIs.

For my first year+ of working with Google Cloud, I only knew one way to obtain credentials for authenticating to Google APIs.

gcloud auth application-default login

There’s actually a slightly shorter way to do the same thing. Now I re-auth twice as fast. Courtesy of Matthew Ashton :)

gcloud auth login --update-adc

Budget Alerts are not on by default

Enable Alerts as part of the account provisioning process. Open the Budgets console and you will see a monthly spend report. In this example I set a $3000 Budget alert and various thresholds for notifications. You can choose between Actual cost or Forcasted and who you want to notify. A Pub/Sub topic is my preferred option as it can notify a group or trigger a webhook to something like a Slack channel.

I also recommend enabling Anomly detection. The easiest way is to use the Auto detection feature, but you can customize as needed.

Cloud Monitoring sets metrics to a Rate view

If you view any metric, such as the Cloud Run Request count you’ll notice it gives you a Requests Per second aggregation. Generally, I’m more interested the actual number of requests than Requests Per second.

I always end up swapping the view so I can see the actual data without a Rate aggregation. Find the Aggregation field, open the drop-down, and select Configure Aligner . Then choose the actual value you want to see such as Sum, Max, Min, p95 or p99.

Now you can see the actual Request count rather than the Requests Per second.

Tips for Google Cloud Architects was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Google Cloud Pub/Sub is best way to build decoupled, distributed, high scale systems that require message delivery or queuing features. This article will help you understand the best practices for efficient and cost effective use of Pub/Sub.

Pricing

Pricing is composed of 3 main components (monthly):

1. Throughput ($40/TiB. $50/TiB for BQ or GCS targets) — all data published to a Topic or read from a Subscription. Measured in 1 KB increments.
2. Data egress— all data into Pub/Sub is free. All data out to internet or between regions cost you money.
3. Storage ($0.27/GiB) — for using message retention on your Topic and for any message in a Subscription that is older than 1 day

A quick recap

Pub/Sub is a global service -- this means you don’t need to deploy a Topic in every region where you run workloads. Pub/Sub subscriptions attach to a Topic so you can listen to incoming messages. Subscriptions can either be Push or Pull based. Push is the simplest since Pub/Sub will push messages to subscribers and handle authentication, scaling, and backoff/retry. Pub/Sub uses exponential back off to optimize the push rate to your consumers.

Pro tip: You should use Push unless there is a good reason to Pull

Hint: There are very few reasons to Pull. You have to write & manage code + you are wasting compute $$$ if you do this in serverless compute like Cloud Run.

Understand how Subscriptions work

Here are the key things to remember about a subscription:

• Messages are duplicated from the Topic to all Subscriptions. This means Throughput charges grow in a linearly in proportion to the number of active Subscriptions
• Subscriptions maintain independent control of their copy of each message. Acknowledged messages in a Subscription don’t affect any other subscriptions
• Subscriptions only contain messages that arrive to the Topic after the subscription was created

Understand Message Retention

By default Topics do not retain messages, however by default Subscriptions retain messages for 7 days. You can change both of these settings at cost.

Pro Tip: If you need to Seek back in time and replay previously acknowledged messages, then add Topic retention.

Delivery Format matters: Wrapped vs Unwrapped

By default, Pub/Sub will wrap published messages in a Pub/Sub specific JSON schema in the message.datafield.

 {
      "message": {
          "data": "[Published message as a Base64 encoded string]",
          "messageId": "2070443601311540",
          "message_id": "2070443601311540",
          "publishTime": "2021-02-26T19:13:55.749Z",
          "publish_time": "2021-02-26T19:13:55.749Z"
      },
    "subscription": "projects/myproject/subscriptions/mysubscription"
  }

Pub/Sub pushes messages using a standard HTTP POST that includes a Content-Type header of application/json. This helps downstream consumers process the data appropriately.

Your code must base64 decode the message.data field as the first step. Here’s sample code for a Cloud Run Function:

functions.http('helloHttp', async(req, res) => {
  try {
    if (req.method !== 'POST') {
      res.status(405).send('Method Not Allowed');
      return;
    }

    // Get the payload from the request body
    const payload = req.body;
    //console.log("Body:"+ JSON.stringify(payload));

    if (!payload.message || !payload.message.data) {
      const errorMessage = 'Bad Request: Invalid Pub/Sub message format';
      console.error(errorMessage);
      res.status(400).send(errorMessage);
      return;
    }

    const b64Message= payload.message.data
    
    const message = Buffer.from(b64Message, 'base64').toString().trim()
    console.log("Decoded msg: " + message)

    const jsonObject = JSON.parse(message);
    // TODO: Do something with the message
  
    // A 200 Ack's the message
    res.status(200).send('Success');
  } catch (error) {
    console.error('Error:', JSON.stringify(error));
    res.status(500).send('Error processing request');
  }
});

Alternatively, you can enable Message Unwrapping (for push subs only) — which tells Pub/Sub to deliver the message published to the topic as-is without any additional information. When using Unwrapped messages, all headers will be removed from the HTTP POST request, including the Content-Type header.

Cloud Run will attempt to parse the incoming message from a Pub/Sub subscription according to the Content-Type header, however the lack of headers causes delivery issues for certain HTTP frameworks.

Pro tip: When using Unwrapped, I suggest adding back the metadata (which includes headers) so you don’t run into delivery and parsing issues downstream

Best Practices

Message Sizes

Throughput costs are measured in 1 KB increments, which means a 1 byte message costs the same as a 1KB message. If all of your messages are small, you should use Batch settings to maximize the 1KB metering window and deliver multiple messages in a single publish.

For example, lets assume each message averages 300 bytes. That means you can fit 3 messages per batch to maximize the 1KB window.

If you’re using the Node APIs, set the batching configuration for maxMessages = 3.Always set the maxMilliseconds option to prevent delays if there is a low throughput of messages being published.

Then call the publishMessage method multiple times. The client will automatically buffer your messages and deliver them together.

  const publishOptions = {
    batching: {
      maxMessages: 3, // Choose a value that gets you close to 1KB
      maxMilliseconds: 3000, // Don't delay too long. Publish at 3 seconds
    },
  };
  const batchPublisher = pubSubClient.topic(topicNameOrId, publishOptions);

  const promises = [];
  for (let i = 0; i < 3; i++) {
    promises.push(
      (async () => {
        const messageId = await batchPublisher.publishMessage({
          data: dataBuffer,
        });
        console.log(`Message ${messageId} published.`);
      })(),
    );
  }
  await Promise.all(promises);

Process messages within 1 day

Messages sitting in a Subscription for less than 1 day do not incur Storage charges. Try to process and Acknowledge messages within 1 day to minimize costs. Monitor the Unacknowledged Message metric — if this metric increases over time, that means your consumer is not keeping up.

Confirm the Oldest unacknowledged message age metric to understand how far behind your consumer is.

Filter your Subscriptions

Always include attributes when you publish messages. Attributes allow you to enable a filter and ensure only valid messages make it to a Subscription.

This will prevent your consumer from attempting to process invalid messages, ultimately improving processing throughput and improving overall cost efficiency.

Filtered messages cut out the entire data transfer component of the pricing but you still pay the Throughput component.

The cost savings can be very significant, especially if you have large message sizes or high scale volume.

Use a DLQ

If you don’t use a DLQ, messages that exceed the Retain duration or Retry limit are deleted forever. This usually happens for a few reasons:

1. Your consumer attempts to process a message but encounters an error or does not Acknowledge the message
2. Your consumer is unable to keep up with incoming publish throughput and messages start exceeding the Retain setting

Setup a different process to inspect the DLQ and redrive the messages back into the main Topic.

Test throughput end to end

It’s critical to understand how your consumers respond to different scale of incoming messages. You should understand CPU Utilization and Memory consumption so you can tune your consumer appropriately.

The only way to do this is to load test the flow of messages from end to end. You can write your own load generator that publishes messages to Pub/Sub — but you have to own code and deploy this to compute.

I recommend using Google Cloud’s Dataflow service to generate artificial load. There’s a built in Template that you can use called the Streaming Data Generator.

Set the Output Rate to something you want to simulate:

Enable the Streaming Engine so you offload the publisher compute to Google owned infrastructure.

Next, you will need to provide Dataflow with a JSON template that will be used to generate random messages to push to PubSub. All possible configs and functions are available here. Here’s a sample template that pushes a fictitious order in JSON. Upload it to GCS for reference in the next step.

{
  "id": {{integer(0,1000)}},
  "orderId": "{{uuid()}}",
  "firstName":"{{firstName()}}",
  "email":"{{email()}}",
  "state":"{{state()}}",
  "product": "{{random("labubu","molly","snoopy")}}",
  "isInStock": {{bool()}},
  "orderDate": "{{date('YYYY-MM-dd')}}"
}

Select the JSON template from above and make sure you specify the Output Sink as PUBSUB.

Finally, if you removed your default network and made a custom one, you will need to specify the network and subnetwork (subnet) you want to deploy to. Pay attention to the subnetwork format.

Dataflow will provision all resources and launch the job. This usually takes a few minutes and then slowly ramp up the RPS to the max you configured. You can watch the progress:

Summary

This post doesn’t cover a bunch of features you should be aware of, such as:

• BigQuery Subscriptions
• Cloud Storage Subscriptions
• Ordering control

Reach out to me if you need any help.

Happy Coding!

Top Google Cloud Pub/Sub Tips & Tricks was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

MCP was developed because we realized LLMs need a standard way to access downstream resources that provide deterministic operations. The downstream resources typically use a wide ranges of protocols such as REST, GraphQL, gRPC/ProtoBuf, or SOAP/XML.

MCP is an attempt to standardize how AI clients interface with these downstream resources. Those resources are exposed to the AI clients as Tools. To make your resource consumable by AI clients all you have to do is implement the MCP specification which requires the message format to be JSON-RPC 2.0.

My friend Heeki Park describes it best in his article:

MCP is an abstraction layer for the downstream resource, using the MCP client with a standard JSON-RPC 2.0 wire format to the MCP server.

So .. what does the MCP spec look like?

The spec is complex but the best place to start by getting familar with the two types of messages defined by the spec. The first is the tools/list message. This allows you to list all available Tools and the input schema required for each tool. A request to list tools looks like this:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/list",
  "params": {}
}

The second message is tools/call. This allows you to invoke one of the tools discovered in the tools/list message.

{
    "jsonrpc": "2.0",
    "method": "tools/call",
    "params": {
        "name": "add",
        "arguments": {
            "a": 1,
            "b": 2
        }
    }
}

All operations occur over HTTP POST requests.

Why do you need a MCP Server?

You could implement the MCP interface at every resource — but this approach is not scalable and tough to manage. This is where a MCP Server comes in — it’s better to abstract the MCP interface via a centralized service called the MCP Server. The MCP Server has two main responsibilities:

• Implement the integration into the downstream Resource. For example, this might be a REST or GraphQL call to a Weather API
• Implement the MCP spec to expose the Resource to AI applications as a tool— think of it as a proxy layer.

Build your own Serverless MCP Server

Google Cloud Run is the perfect service to operate a MCP Server. Run is fully Serverless and will scale to meet the demands of AI Apps that consume your Tools. Run will handle all of the arbitrary tasks like OS management, patching, and autoscaling.

For this walk through, we will create a MCP Server using Python, deploy it to Cloud Run, and test it.

Create a directory to work in and then git clone this project:

git clone https://github.com/georgemao/mcp-on-cloudrun.git

Install UV

UV is a Python package manager similar to Pip. Install it using the method that’s most appropriate for your development environment.

Init the env

This will install all dependencies defined in the pyproject.toml and create a virtual Python env for the project.

uv sync

Deploy from Source to Cloud Run

Take a look at the source code (server.py). It’s a simple calculator resource with two tools: add and subtract. It uses FastMCP to implement the MCP spec. Adding the @mcp.tool annotation exposes a method as a tool.

Next, lets deploy the service to Run. This will use Cloud Build to create a Docker image, push it to Artifact Registry, and deploy as a service to Cloud Run.

Note: we are not allowing unauthenticated calls to the MCP Server by specifying the no-allow-unauthenticated flag.

gcloud run deploy mcp-server --no-allow-unauthenticated --region=us-central1 --source .

You goto the Cloud Build console and verify that the build was successful.

Test your server

We will proxy localhost calls to Cloud Run and provide the correct authentication. This will prompt you to enter your IAM account password. The account you’re using must have permissions to invoke Run services. If necessary, add this permission to your IAM role: roles/run.invoker

gcloud run services proxy mcp-server --region=us-central1

Now all calls to localhost are proxied out to your MCP server on Run and include the proper Authorization header.

If you execute a tools/list call to the server, you will get a detailed JSON response that describes all of the supported tools (add & subtract).

{
  "tools": [
    {
      "name": "add",
      "description": "Use this to add two numbers together.\n\nArgs:\n    a: The first number.\n    b: The second number.\n\nReturns:\n    The sum of the two numbers.\n",
      "inputSchema": {
        "type": "object",
        "properties": {
          "a": {
            "title": "A",
            "type": "integer"
          },
          "b": {
            "title": "B",
            "type": "integer"
          }
        },
        "required": [
          "a",
          "b"
        ]
      }
    },
    {
      "name": "subtract",
      "description": "Use this to subtract two numbers.\n\nArgs:\n    a: The first number.\n    b: The second number.\n\nReturns:\n    The difference of the two numbers.\n",
      "inputSchema": {
        "type": "object",
        "properties": {
          "a": {
            "title": "A",
            "type": "integer"
          },
          "b": {
            "title": "B",
            "type": "integer"
          }
        },
        "required": [
          "a",
          "b"
        ]
      }
    }
  ]
}

Programmatic testing

This Python code connects to localhost and lists the available Tools. Then it calls the add and subtract tool. Notice the method for calling both tools is the same: client.call_tool(). The only difference is the parameters being passed in.

Execute the test:

uv run test_server.py

UI testing with MCP Inspector

If you prefer to use a GUI based tool to test your MCP Server, MCP Inspector is built for this purpose. Think of it as Postman for MCP.

Launch it with this command:

npx @modelcontextprotocol/inspector

Copy the Session Token, enter it into the Inspector Configuration, and click Connect:

You should be able to list tools and call them through the UI:

Let’s see what the requests actually look like. Open your Developers Tools and select the Network tab.

Run the add or subtract tool from MCP Inspector and you should see two requests:

The first is the Cross Origin Resource Sharing (CORS) preflight request. Open the second one and you will see the tools/call message along with the parameters you specified in the UI.

Go to the Cloud Run console and open the mcp-server service. Observe a few of the metrics. In particular, take a look at the Container Instance Count metric — it scales up and down in response to the tests we performed. Also in relation, the Billable Container Instance time metric confirms the nature of serverless compute — there is no cost until actual requests arrive.

Summary

You should now understand the purpose of MCP and why a MCP Server can scale your AI workloads efficiently.

MCP is a lot more complex than what we covered here — for example, we haven’t talked about Authorization/security, monetization, version control, or advanced features like argument auto completion.

Some resources that can help you learn:

• Host MCP Servers on Cloud Run
• Heeki Park’s Oauth + MCP security post
• MCP on Wikipedia
• Build and Deploy a Remote MCP Server to Google Cloud Run in Under 10 Minutes

Power your MCP servers with Google Cloud Run was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

I’ll show you how you can use Google Cloud Run to serve your own LLM for about $1/hour.

Google Cloud Run + GPU + Gemma3

My favorite way to build applications is on top of Google Cloud Run — it lets you build serverless applications at any scale and it’s the only serverless compute technology from any cloud provider that supports GPUs. If you’re not familiar with Gemma — its a family of open source LLMs that are based off the Google Gemini models. Gemma is designed to be lightweight and requires just a single GPU/TPU. It supports up to 128K tokens and is multi-modal. You can read in detail about Gemma3 here.

Before we begin, here are a few things that Cloud Run will require when you enable Nvidia L4 GPUs (24GB):

• Minimum configuration is 4 vCPU and 16G RAM
• Instance billing is required (instead of request billing)

Build your Serverless LLM

Gemma3 has 4 different model sizes (1B, 4B, 12B, 27B) — we will be using the 4B model size but you should be able to deploy the 12B model as well.

Here’s what we’ll do at a high level:

• Use Ollama to serve the Gemma3:4B model on Cloud Run
• Store the model on a GCS Bucket and mount it as a Volume to Cloud Run
• Use OpenWebUI to interact with the model and provide a UI for users
• Deploy and host as a 100% serverless capablity on Cloud Run

Enable the required APIs:

gcloud services enable run.googleapis.com \
    cloudbuild.googleapis.com \
    storage.googleapis.com \
    artifactregistry.googleapis.com

Download the Gemma3 4B model

Create a directory to work in:

mkdir serverless-gemma-demo
cd serverless-gemma-demo

Download Ollama via their website or run command line:

curl -fsSL https://ollama.com/install.sh | sh

Pull the Gemma3 4B Model & test it:

ollama serve
ollama pull gemma3:4b
ollama run gemma3

Push the model to a GCS Bucket

You can do this via the console or CLI:

gcloud storage buckets create gs://[your gcs bucket]
gsutil cp -r ~/.ollama/models gs://[your gcs bucket]

Create the LLM Image for Cloud Run (Ollama + Gemma3)

Create a file called: Dockerfile. It’s contents should be:

FROM ollama/ollama

# Listen on all interfaces, port 11434
ENV OLLAMA_HOST 0.0.0.0:11434

# Store model weight files in /models
ENV OLLAMA_MODELS /models

# Reduce logging verbosity
ENV OLLAMA_DEBUG false

# Never unload model weights from the GPU
ENV OLLAMA_KEEP_ALIVE -1 

# Store the model weights in the container image
ENV MODEL gemma3:4b
RUN ollama serve & sleep 5 && ollama pull $MODEL 

# Start Ollama
ENTRYPOINT ["ollama", "serve"]

Build & Push the Image to Artifact registry:

gcloud artifacts repositories create ollama-sidecar-gemma-repo --repository-format=docker \
    --location=us-central1 --description="Serverless Ollama + OpenWebUI Gemma demo" \
    --project=[your gcp project id]

gcloud builds submit \
   --tag us-central1-docker.pkg.dev/[your gcp project id]/ollama-sidecar-gemma-repo/ollama-gemma-4b \
   --machine-type e2-highcpu-32

Create the OpenWebUI Image for Cloud Run

Pull the image from Dockerhub. Make sure you specify the platform as linux/amd64. Cloud Run requires an x86 based image.

docker pull --platform linux/amd64 ghcr.io/open-webui/open-webui:main

Configure Docker to use your GCP creds so you can push the image to AR:

gcloud auth configure-docker us-central1-docker.pkg.dev

Tag and push the image to AR:

docker tag ghcr.io/open-webui/open-webui:main us-central1-docker.pkg.dev/[your gcp project id]]/ollama-sidecar-gemma-repo/openwebui
docker push us-central1-docker.pkg.dev/[your gcp project id]/ollama-sidecar-gemma-repo/openwebui

Deploy the Cloud Run Service

Create a service.yaml file to define the service. We’ll be deploying two containers on the same Cloud Run Service. One for the OpenWebUI and another for the Ollama / Gemma3 LLM. Review the service.yaml file here. It’s currently configured to disable GPU Zonal Redundancy and allow a max of 5 concurrent request per Run instance.

gcloud  run services replace service.yaml

Deployment will take 5–8 minutes and result in a service url you can use to test it out.

https://ollama-sidecar-gemma-demo-[project id].us-central1.run.app

Here’s what you should see:

Test out your LLM

Try asking it a few different prompts. Start with a text based prompt like:

Explain a SQL Database. Respond as if you were talking to a child. Keep it concise.

Then try the multi-model features. Paste in an architecture diagram, like this one and ask it:

Explain this architecture diagram

Review

Check your metrics to understand how Run scales your containers. For example, the Instance Count metric shows the service scaling up (and down to zero).

The GPU memory util metric shows you how much of the 24GB your service is using. Here I can tell Gemma3:4B is only uses about 20% of the 24GB available GPU memory.

See all avaible metrics here.

Summary

For simplicity we deployed a single Cloud Run Service to host both the OpenWebUI front end and the Ollama/Gemma3 LLM backend.

• The service configuration allows the service to scale down to 0 for maximum cost savings.
• A cold start from zero will take ~15–20 seconds. During this time, users will experience a delay in responsiveness.
• The entire architecture is 100% serverless and costs about $1/hour!

In production, you should split the service up and allow them to scale independently. You should also consider setting a min instances value so your first set of users do not experience a cold start.

Source and config are available on Github here.

You can experiment and deploy any downloadable model. Let me know how it goes!

Cloud Run + GPU = Serverless LLMs! was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

In part 1 of this post, I showed you how to create a globally load balanced, self-healing, auto-scaled, and secure architecture on Google Cloud (Serverless of course!). In this post, we will add 3 improvements to this architecture and automate it’s deployment. Here’s what I’ll cover in this post:

1. Optimize content delivery by adding Cloud CDN
2. Enhance security posture by adding Identity Aware Proxy (IAP)
3. Add Google Firestore for global data persistence
4. Automate deployment via Pulumi IaC

Step 1: Optimize content delivery (static assets)

The previous architecture relies on Cloud Run to respond to all incoming traffic. This is technically feasible, however it’s expensive and inefficient.

The goal is to offload all static assets that don’t require any active compute to be delivered by Cloud CDN instead.

Cloud CDN will serve assets from the Google edge network closest to the user. The Global Load Balancer will integrate with a Backend bucket to store the actual assets. For maximum efficiency, Cloud CDN maintains a cache and attempts to serve directly from that cache instead of reaching to the Google Cloud Storage bucket (GCS).

We will configure all requests to the root of the domain / to be served by CDN. You can enable this at the Load Balancer console.

First, goto the Backend Configuration and create a Backend Bucket. Make sure to enable Cache static content. My bucket name is gmao-web-app. Make sure the Bucket has an index.html file that will be served by default.

Next, goto Routing Rules and add a path that will be served by Cloud CDN. I want the root / of the domain to serve a static web application and the /api path to be served by the global Cloud Run service that we deployed earlier. Here’s how to set that up:

Finally, make sure the bucket has public access enabled and set the default file to be index.html. To set the default file, use gsutil to set the MainPageSuffix to index.html:

gsutil web set -m index.html -e 404.html gs://gmao-web-app

Detailed docs are available here and a full walk through here.

Deploy a Static Website

One of the best ways to create a web application is to build it in React and compile it as a static Single Page Application (SPA) — then deploy to your GCS bucket. Cloud CDN will serve the assets and the entire application will run client side with no web servers necessary.

Here is my sample Github repo that show you how to build a basic React app that can be deployed to any service capable of serving HTTP traffic.

Step 2: Enhance security with Identity Aware Proxy

Identity Aware Proxy (IAP) is a global Google security service that authorizes user requests to applications based on properties in the user’s request context. IAP supports integration with various 3rd party Identity providers such as Google, Facebook, etc as well as GCP project specific IAM users. Plus, it’s #serverless — there are no servers to manage or scale!

The goal is to add context aware security to our application. We will enable IAP on the GALB to protect the application on Cloud Run.

For this reference architecture, we will enable IAP for IAM users, not external Identities. We will protect the application on path /api. Goto the IAP console, locate the Backend for your service, and enable IAP. This will take short duration to provision — just wait until the status is OK.

Then add Principals and assign them the IAP-Secured Web App User role.

Any principal that has this role will be prompted to authenticate when they request path /api

Step 3: Add Global Persistence

Google Cloud provides a wide range of database services that serve different purposes. I choose Google Cloud Firestore to add global persistence to this reference architecture because it is designed to offer global scalability, multi-region replication with 99.999% SLA, strong consistency by default, and no server management. It’s a NoSQL Document database that organizes documents in groups called Collections.

Firestore’s strong consistency allows this architecture to read and write from any region with consistent results. For each user that will use the system, I preload a document with its ID set to the user’s email. The document itself stores user information as follows:

{
    "firstName": "George",
    "lastName": "Mao",
    "email": "user@iamgeorge.altostrat.com"
}

After successful authentication, IAP will inject 3 headers into the downstream request as follows:

The goal is to extract the authorized user context proved from IAP and use it to pull the associated document from the Firestore backend.

First, we need create a new Service Account that has permissions to interact with Firestore and assign it to the Cloud Run Function. The default service account attached to Cloud Run resources does not provide any permissions. The role we need to add is roles/datastore.user.

If you are using the the console, Create a new Service Account then assign the Cloud Datastore User role.

If you are using the glcoud cli, just execute the following commands:

gcloud iam service-accounts create SERVICE_ACCOUNT_NAME \
  --description="DESCRIPTION" \
  --display-name="DISPLAY_NAME"

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/datastore.user"

Second, we need to update the Cloud Run Function code to:

1. Extract the x-goog-authenticated-user-email header from the request
2. Lookup the Document in Firestore with the matching ID

let userEmail = req.headers['x-goog-authenticated-user-email'];
console.log(userEmail);
// Remove the accounts.gooogle.com string
userEmail=userEmail.replace("accounts.google.com:", '');

Note: In a real production environment, you should also validate the JWT token in the x-goog-iap-jwt-assertion header

Get the document in Firestore that has the same ID:

let doc = await readDoc(userEmail);

async function readDoc(userEmail) {
    try {
        console.log(`Reading document: ${userEmail}`)
        const document = firestore.collection('person').doc(userEmail);
        
        // Read the document.
        const doc = await document.get();
        if (!doc.exists) {
            console.log('No such document!');
            return;
        }
        console.log('Read the document');
        console.log(doc.data());

        return doc.data();
    } 
    catch (error) {
        console.error('Error retrieving Firestore document:', error);
        return null;
  }
}

Full updated code is available on in my Github repo here.

Step 4: Automate deployment

Pulumi IaC is a great way to describe your Cloud architecture using common programming languages you already know. It has significant support for GCP and automates all deployment tasks.

The goal is to describe all major components of this architecture using Pulumi NodeJs libraries and fully automated the deployment

I’ll cover some of the key configurations below. To create the Cloud Run resources: Stage the code as a zip file in a GCS bucket, then deploy the Cloud Run Function.

To create the Firestore Database and add a document:

The source code that handles the deployment via Pulumi is available here — including configuration of all other resources in this architecture, such as the Global ALB, Cloud CDN, and Firestore.

Run pulumi up to execute the deployment. You should see a diff that shows the resources that will be deployed. Confirm these are accurate and select yes.

Summary

In part 1 of this post, I showed you how to create a globally load balanced architecture.

In this post, we enhanced the architecture by adding global content delivery, security at the API layer, and global persistence capabilities.

Review the source code for:

• Static web app deployed to Cloud CDN
• Global, Serverless Cloud Run Functions backend
• Deployment automation via Pulumi

Reach out to me if you have any questions & Happy Coding :)

4 steps to supercharge Planet Scale, Global architectures on Google Cloud was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Serverless is the best way to build modern, cloud applications — I let Google Cloud handle all of the heavy lifting of server management, operating systems, patching, and scaling so I can focus on building my application.

In my previous post, I showed why Google Cloud Run is the best serverless, compute runtime. In this post, I’ll show you a reference architecture that demonstrates a globally load balanced, self-healing, auto-scaled, and secure environment— all without managing a single server! We will enhance this architecture in part two.

High Level Architecture

There are four major components in this architecture that make this possible. We will deep dive into each component and cover the integration details.

1. Cloud DNS — allows you to configure a globally resolvable domain name that points to your Global Load Balancer
2. Cloud Armor — provides layer 7 protection against DDoS attacks, OWASP vulnerabilities, and a fully configurable Web Application Firewall
3. Cloud Load Balancing — provides auto-scaled, traffic balancing across regions and native integration with the other components
4. Cloud Run Functions— provides a fully managed, pay-per-use serverless compute environment to run your application logic

Cloud Run (Functions)

Cloud Run is a regional service and is supported in every region GCP operates in. It’s entirely serverless and requires zero management. I’ll be using a Cloud Run Function (CRF) to act as the compute tier for this application. It will expose a HTTPS endpoint that allows a consumer to invoke my code. The build process to deploy a CRF looks like this:

To get started, let’s deploy a Cloud Run Function. You can use this sample application. It simulates a 3 second workload and uses the Metadata service to return info about where it’s hosted from. The code will retrieve the instance id it’s hosted on, the number of times the specific Run instance has been invoked, and the region it’s running as JSON:

{
  "instanceID": "...",
  "counter": ###,
  "region": "projects/######/regions/region-name"
}

The deployment process submits my source code to Cloud Build (CB) which will build a docker image, store it in Artifact Registry, and deploy to the Cloud Run service. Once deployed, the Run Function is assigned a HTTPs endpoint in the format:

https://REGION-PROJECT_ID.cloudfunctions.net/FUNCTION_NAME

You can do this through the console, gcloud CLI, or Terraform. Here’s the command to deploy:

gcloud functions deploy multi-region-demo 
     --gen2 --region=us-central1 
     --runtime=nodejs22 --source=. --entry-point=helloHttp 
     --trigger-http 
     --ingress-settings=internal-and-gclb --allow-unauthenticated

This creates a Run Function called multi-region-demo in the us-central1 region using Node22 based source code.

By default, the CRF endpoint is hosted out of a single region, allows traffic from anywhere, and requires Authentication.

Our goal is to serve all traffic through a Global External Application Load Balancer (GALB) instead of directly from the Cloud Run Function. This will allow the GALB to decide which region to best serve incoming traffic.

• Use the allow-unauthenticated flag because GALB does not perform any form of auth to a Cloud Run service
• Use the ingress-settings flag to restrict traffic to the function so that only internal sources and your GALB are allowed

You can monitor the deployment progress at Cloud Build and view the actual image that was built at Artifact Registry.

Next, deploy the Run Function to every region you want your application to be hosted from. Just run the same gcloud CLI command with a different region specified. Every Run Function will exist independently in its own region and have its own HTTPs endpoint. We will attach all deployed regions to a Global ALB in the next step.

Remember these are regional services so Cloud Build will execute independently in each region and store Artifacts in each region

Cloud Load Balancing

A Global External Application Load Balancer (GALB) will act as a single point of traffic ingress and distribute traffic across all regions based on the user’s location. This is made possible through a single anycast IP address that is presented across all regions. We will use Cloud DNS to create a DNS entry (A record) that resolves to this IP address, then use Certificate Manager to create a matching SSL certificate for our Custom domain.

Let’s look at the 4 main components that make up this architecture.

1. A Global External Load Balancer (GALB). This will provide a single static, anycast IP that your Custom DNS A record will point to.
2. A Global Backend Service. This acts as the service that your GALB will direct traffic to. Your Backend Service can attach a variety of Backends (such as GCE Managed Instance Groups) — in our case we are going to attach a Serverless Network Endpoint Group (NEG).
3. A Serverless NEG represents a Serverless resource such as Cloud Run, App Engine, or API Gateway. You will need to create a Serverless NEG in each region you want to service traffic and point it to the Cloud Run service in that region.
4. Cloud Run. This provides the actual compute environment to execute your application logic. We set this up already.

Review the Load Balancer docs for detailed instructions on how to setup this architecture.

Certificate Manager

Create a global, Google Managed SSL certificate for your domain using DNS Authorization. You can create a domain specific cert or a wildcard cart for an entire subdomain. Before Google will activate your Certificate you must validate that you own the domain through a DNS AuthZ process.

Certificate Manager will generate a key-value pair of DNS records that you must add to your Domain. Certificate Manager will lookup your domain and verify that the records match the values it generated. This process establishes ownership of the domain — preventing unauthorized certificate usage.

Cloud DNS

In Cloud DNS, create a public Zone for your Domain. Then add two records to the Zone:

1. An A record that points to the static IP for your GALB. This will allow global resolution of your domain to your GALB.
2. A CNAME record with matching values from the Certificate Manager step. The verification process usually finishes in under 30 minutes but could take up to 24 hours due to DNS propagation and caching.

For example, my domain is iamgeorge.demo.altostrat.com. I added both records as follows:

Monitor the Certificate status in Certificate manager —once the status shows Active you are good to go. Note the name for your Certificate as we will use it in the next step.

Now we need to deploy this cert to the GALB. At the time of this post, the only away to do this is via the gcloud CLI. The full process is described in the Certificate Manager docs here. The main steps are:

Create a Certificate Map

gcloud certificate-manager maps create CERTIFICATE_MAP_NAME

Create a config Entry in the Map for your Cert and Domain

gcloud certificate-manager maps entries create CERTIFICATE_MAP_ENTRY_NAME \
    --map="CERTIFICATE_MAP_NAME" \
    --certificates="Cert name from Certmanager" \
    --hostname="Your cert's hostname"

Attach the Certificate Map to your GALB

gcloud compute target-https-proxies create PROXY_NAME \
    --certificate-map="CERTIFICATE_MAP_NAME" \
    --url-map="The name of your URL Map you created during the GALB step" \
    --global

Cloud Armor

You should secure your application by applying Layer 7 WAF rules with Cloud Armor. It’s easy to start with preconfigured rules but you can also create custom rules.

Review your GALB configuration — under the Backend Services section you should see a Backend Security Policy link. There will be a default policy created that allows all traffic. It looks like this:

Try adding a preconfigured rule such as a check for Cross-Site scripting:

Use Logs Explorer to review how your security policies are enforcing requests. Here’s a basic log query to grab policy evaluation logs:

resource.type:(http_load_balancer) 
  OR resource.type:(tcp_ssl_proxy_rule) 
  OR resource.type:(l4_proxy_rule) 
  AND jsonPayload.enforcedSecurityPolicy.name:(name of your policy here)

Look at the enforcedSecurityPolicy JSON block to figure out how your policy executed.

Summary

You should now be able issue a HTTP GET against your domain and get a response from the nearest region. You can VPN to a different region or use a GCE instance in a different region to verify that the GALB is routing traffic appropriately.

If you used my code for your CRF, here’s what you should see.

{
  "instanceID": "...",
  "counter": 1,
  "region": "projects/528923850333/regions/us-east4"
}

This architecture allows you to leverage the global Google network to maximize resiliency and cost efficiency, plus:

• It’s 100% serverless — there are zero servers to manage or patch including the compute used in the build and deployment phase
• It autoscales and load balances with traffic and incurs cost per use, instead of incurring cost by uptime (with the exception of the GALB)
• Security is built in at every stage — there is no public entry to any compute, layer 7 security is covered, and container images are auto patched and updated by Google

What’s next

In my next post, I’ll show you how to enhance this architecture with:

• Cloud CDN to serve static assets for faster, cheaper delivery
• Add Identity Aware Proxy on the GALB as another form of Security
• Add and configure various Datastore backends
• Automate deployment of this architecture using Pulumi IaC

Build a planet scale, global architecture for modern apps on Google Cloud was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

In my Intro to Cloud Run functions post I covered the basics that all Cloud Run Functions developers should know. In this post, I’ll show you how to understand costs & strategies you can use to optimize your functions.

Pricing Deep Dive

The primary driver of costs is consumption of resources configured. Here is an easy way to remember the pricing formula:

Resources configured at deploy time (vCPU + Memory) * Duration of execution

All components of the formula have a direct, linear effect on cost. So if you double the configuration, you double the cost. Or if the function runs twice as long, you double the cost. If you cut the configuration in half or the function runs half as long, you save 50%. Here is a simple example:

• You configure your function to use 2 vCPU / 4GB Memory and it runs for 1000ms on average
• If this function is invoked 1M times a month, it’ll cost you $58/month
• If you cut this configuration in half (1 vCPU and 2GB memory) and the function continues to run for 1000 ms, you’ll reduce costs by 50%, or spend $29/month
• If the the reduced resources causes the function to run slower and take 2000ms on average, you’re back to $59/month

In general, your goal is to find the configuration “sweet spot” where performance and cost are optimal.

Optimizations

There are two categories of optimizations you can make. The first is configuration changes to your function — this is easier but requires a deep dive into metrics. The second is coding changes. Let’s start with configuration changes.

Allocate resources to match workload needs

The best way to begin is to determine the primary resource your workload consumes, is it CPU or Memory? Focus on tuning that resource properly. Tuning is generally a 3 step process:

1. Start with a baseline configuration
2. Run tests to benchmark resource usage & review results via metrics
3. Make changes and repeat

Here’s a snippet of code that represents a CPU intensive workload with very little memory requirements — it computes SHA512 hashes in a sequential loop. Our goal is to determine where the CPU configuration gives us the best cost/performance value. Full source code available on Github.

N = 51200

@functions_framework.http
def hello_http(request):
    #HTTP Cloud Function that simulates CPU load

    m = hashlib.sha512()
    # Generate a random long string
    rstr = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(N))

    t1 = datetime.now()
    print("Start time: " + str(t1))

    # repeatedly compute the digest
    for i in range(10000):
        m.update(rstr.encode())
        m.digest()

    t2 = datetime.now()
    print("End time: " + str(t2))
    print("Difference: " + str(t2 - t1))

    difference = str(t2 - t1)

    return 'Performance {}!'.format(difference)t

• I started with the default 256MB memory configuration and set vCPU = 0.167. This resulted in ~6.8 seconds to complete the execution.
• Next, I double vCPU to 0.333. This doubles performance, reducing duration to ~3.3 seconds.
• Next, I triple vCPU to 1. This triples performance, further reducing duration to ~1 second.
• Finally, I double vCPU to 2 and another double to 4 vCPU. Neither of these provide any meaningful performance gains.

Just to verify, I check the Memory Utilization metric and confirm that I am well under the 256MB configuration.

If this were a real workload, I would probably set my vCPU = 1. Anything more is simply a waste of money.

Use Concurrent Requests Per Instance

This is the next configuration based optimization that you should look into. By default, the Cloud Run functions service will only allow a single request to be served concurrently by each instance of your function. With my vCPU set to 1, I investigate the Container CPU utilization metric. I see a peak of ~12.97% CPU. This leads me to believe my container’s CPU is underutilized.

Based on my hypothesis, I increase the Max Concurrent Requests per Instance from 1 → 2 and execute two simultaneous invokes of my function.

As I expected, this results in about double the CPU utilization, or ~25.98%. This makes sense as the same container is doing double the work now.

This configuration allows a single container to service twice as many requests during the same billable window. I can check the Billable Container Instance Time metric to confirm.

Note: Always check the Duration or Latency metric to confirm CPU utilization does not have a negative effect and degrade performance

Use global state correctly

The final optimization technique involves moving heavy compute tasks to take place in global execution state. This allows the CPU Boost feature to provide extra cpu power (about 2x) during cold starts. Cloud Run Functions can run cpu intensive tasks for up to 10 seconds with boosted cpu.

All code in global state is only executed once for the life of the function’s instance and can be reused durning future warm invokes.

Below, I’ve refactored the SHA computation code to take place globally instead of inside hello_http function. It will run once during a cold start and benefit from boosted CPU. See updated code on Github.

N = 51200

# This function is called during cold start only
def computeHashes():
    m = hashlib.sha512()
    # Generate a random long string
    rstr = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(N))

    t1 = datetime.now()
    print("Start time: " + str(t1))

    # repeatedly compute the digest
    for i in range(10000):
        m.update(rstr.encode())
        m.digest()

    t2 = datetime.now()
    print("End time: " + str(t2))
    print("Difference: " + str(t2 - t1))

    difference = str(t2 - t1)
    return difference

# Save results in global state    
perf = computeHashes()

@functions_framework.http
def hello_http(request):
    #HTTP Cloud Function that simulates CPU load

    return 'Performance {}!'.format(perf)

Time to test! I execute the function 3 times in a row. The first invoke runs the hash compute code in ~1 second, while the following two invokes don’t execute the hash code at all and simply return in 3 ms.

Summary

In my experience, tuning Serverless services always requires actual testing and analysis of results. It’s a bit of trial and error and Cloud Run functions is no different. In my next post, I’ll cover a great way to perform large scale load testing of Cloud Run functions.

In the mean time, be sure to review the Cloud Run functions Tips & Tricks.

Optimize your Cloud Run functions was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Serverless development is my favorite way to build modern applications — with Google Cloud Run Functions, you bring the code and Google handles all of the heavy lifting of load balancing, scaling, availability, and Infrastructure/OS management. This means developers just focus on building awesome apps. Here is the first of a 3 part guide on Cloud Run Functions.

Cloud Run Functions is the only serverless FaaS product in the market that supports scale up and down from zero, can attach L4 GPUs on demand, and deploy to multi-regions with one command

The Benefits

• You only pay for the resources you consume. You don’t need to worry about idle VM instances or optimizing your cluster sizes. You’re billed on a combination of Invocation count (first 2M / month are free) + resources allocated during theduration of execution — which we refer to as Vcpu-seconds and Gib-seconds.
• There is no maintenance! Developers don’t need to debate Instance types, or configure VMs, and there is no Operating System to patch
• You can operate nearly any workload, including deploying LLMs or running graphics intensive workloads using the new L4 GPU support

The Basics

Functions respond to Triggers —there are two types of triggers

1. Event based triggers are an event generated by a GCP Service. Examples are EventArc, Pub/Sub, Cloud Storage, or Firestore. These services generate an event that results in an asynchronous invoke of your Cloud Run Function.
2. The second type of event is a synchronous HTTP trigger. Functions are auto assigned a Function URL that is referred to as the HTTP trigger. You make a HTTP POST request to the url to invoke the function. By default, the URL is public and requires Authentication. In addition, you can restrict or disable external traffic. It’s always in this format:

https://REGION-PROJECT_ID.cloudfunctions.net/FUNCTION_NAME

Pro Tip: You can invoke your function for testing with a curl command:

curl -m 70 -X POST https://[region]-[project_id].cloudfunctions.net/[func_name] \
-H "Authorization: bearer $(gcloud auth print-identity-token)" \
-H "Content-Type: application/json" \
-d '{
  "name": "Hello World"
}'

Security

Functions must assign a Service Account (SA). The attached SA provides an identity to all code running in the Function. By default, Functions will use the default compute service account, which has broad permissions. It looks like this:

PROJECT_NUMBER-compute@developer.gserviceaccount.com

You should always create your own SA and attach appropriate, scoped down permissions to specific resources your Function needs to access.

Function Resources

You configure the vCPU and Memory resources allocated to the function. You should choose settings that match your workload needs, since Google bills you based on resources allocated, not consumed.

Let’s write a HTTP Function

Here’s the basic required implementation for a HTTP function (NodeJs). Other language examples are available here. At minimum you need to register with the functions framework using the functions.http() method and then return a valid HTTP response. The req parameter will contain all information delivered as part of the HTTP request.

const functions = require('@google-cloud/functions-framework');

// Register an HTTP function with the Functions Framework that will be executed
// when you make an HTTP request to the deployed function's endpoint.
functions.http('helloGET', async (req, res) => {
  // Retrieve information from the request
  const name = req.query.name
  
  // Return a valid HTTP response
  res.send('Hello World!');

  // Optionally, specify the status code and response type
  // res.status(200).json(retVal);
});

Pro Tip: Change the function syntax to async so you can use top level await

Let’s write an Event Driven Function

You still have to register with the functions framework but this time use the functions.cloudEvent() method. Notice you do not return any values, since event triggers are an async process and do not expect a response.

The cloudEvent parameter will contain the entire payload for the event. Access the cloudEvent.data field to extract details specific to the event that occurred. The payload format for the data field will differ depending on the service that generated the event. For example, some services base64 encode this field but most services just write a JSON object. See the Google-Events repo to see all possible events and their format.

const functions = require('@google-cloud/functions-framework');

// Register a CloudEvent callback with the Functions Framework
functions.cloudEvent('helloEvent', cloudEvent => {
  // Handle the Event here. The event is inside the cloudEvent.data field
  const eventPayload = cloudEvent.data;

  // If Cloud Storage, the payload is in JSON format
  // const {cloudStorageEvent} = {eventPayload.name, eventPayload.bucket}

  // If Pub/Sub, the data payload is base64 encoded
  // const base64Event = eventPayload.message.data;
  // const pubsubEvent = Buffer.from(base64name, 'base64').toString()

  console.log(`Hello!`);
});

What is Concurrency?

There are two concepts you should be aware of when you think about function scaling. They are configurable and operate independently of each other.

Concurrent instances per function

Cloud Run Functions automatically scales the number of active containers behind the scenes to handle incoming work. This is the number of concurrent instances of your function that are active at any given time.

Functions transition through 4 main lifecycle stages: Starting → Active → Idle → Shutting down

• When a Trigger invokes a function, Google will attempt to use an instance that is warm & available to serve the request. You can check if any are available using the Instance count metric and look for the idle count.
• If no idle instances exist, Google will spin up an instance to serve the invoke — this is called a cold start because there is overhead to go from zero to active. The instance will remain active until all processing is complete.
• After a function completes processing, it will transition to idle and wait for another invoke. Functions remain in idle state for up to 15 minutes then the instance is removed. At this point the instance transitions to Shutdown.
• Here is an example of the Instance Count metric. At the highlighted time, there are 35 active instances processing workloads and 5 idle.

You can mitigate colds start by setting the Autoscaling configuration: minimum instances. You may also want to set maximum instances to control costs or protect downstream services from being overwhelmed.

Max Concurrent requests per function instance

Functions default to serving a single request at a time. However, you can increase this setting (up to 1000) to allow a single instance to serve multiple concurrent requests. This is referred to as concurrent requests per function instance.

• Serving multiple requests concurrently generally provides significant cost savings since the majority of cost comes from instance uptime.
• This feature results in shared global memory and cpu consumption. You need to ensure your code is able to safely operate this way.
• You can view the Max Concurrent Requests metric to understand how your functions are serving requests.

Pro tip: By default this metric shows a Distribution over 1 minute, which isn’t very useful. You can modify it to group by count which is more useful.

Summary & TLDR;

When you transition from VM based scaling to Serverless scaling — Concurrency becomes the main scaling factor. Cloud Run has two layers of Concurrency that are critical to understand.

There also are other important Cloud Run Functions features I’ll cover in a future post but you can read through official docs now:

• Networking
• Secrets
• Load testing, Tuning, and Optimization

Let me know if I can help you in your Serverless journey on Google Cloud!

Go Serverless with Google Cloud Run Functions was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.