<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lucía Subatin on Medium]]></title>
        <description><![CDATA[Stories by Lucía Subatin on Medium]]></description>
        <link>https://medium.com/@lucia.subatin?source=rss-910459c47504------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*aZfwYtWLEIR7nFf2i0fj0w.jpeg</url>
            <title>Stories by Lucía Subatin on Medium</title>
            <link>https://medium.com/@lucia.subatin?source=rss-910459c47504------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:38:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lucia.subatin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Asking Gemini 4 (Argon) to revamp my website from GA4 data]]></title>
            <link>https://medium.com/google-cloud/asking-gemini-4-argon-to-revamp-my-website-from-ga4-data-8d39d13b9327?source=rss-910459c47504------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d39d13b9327</guid>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[google-analytics]]></category>
            <category><![CDATA[gemini-4-argon]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[seo]]></category>
            <dc:creator><![CDATA[Lucía Subatin]]></dc:creator>
            <pubDate>Tue, 13 Oct 2026 16:05:21 GMT</pubDate>
            <atom:updated>2026-10-13T16:05:21.122Z</atom:updated>
            <content:encoded><![CDATA[<p>I had early access to Google’s latest model, Gemini 4 (<a href="http://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-4-argon?utm_campaign=CDR_0x6cb6c9c7_default_b573117953&amp;utm_medium=external&amp;utm_source=blog">Argon</a>) so I asked it to help me with a task I had been procrastinating: revamping my personal website. This exercise ended not just with a new website, but a mind blown from the new model’s multimodal design awareness and long-horizon autonomous agent coordindation.</p><figure><img alt="An animated gif showing an engaging presentation titled “Operation Cloudy Unicorn”. It shows KPIs and statistics on website visitor, clicks, traffic sources, devices based on 17 months of telemetry" src="https://cdn-images-1.medium.com/max/1024/1*PtTkevIxOW5wvpmglKAdig.gif" /></figure><p>I had 17 months worth of data telling me I needed to work on this website. But, Google Analytics schemas are messy. So I gave them to Gemini to get a quickstart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nvSNtWXVmfMG1X_0NEURDw.png" /><figcaption>A sample of a massive SQL query to analyze GA4 data</figcaption></figure><p>At the time of publishing this blog post, the revamp of the website (lucia.dev) has already started, but this is why the state of this site was becoming embarrassing:</p><figure><img alt="A white background website, with misaligned tables and links to social media. Includes old pictures of the author and overall lookes dated." src="https://cdn-images-1.medium.com/max/1024/1*3VGKFj2AHMoiuZQqp6rvOA.gif" /></figure><h3>The Prompt</h3><p>I asked Argon to use my GA4 data to optimize my website based on a year and a half of traffic source, referrer, devices, clicks and organic/paid search daily data dumped into BigQuery.</p><p>One of the great things about this model is it goes a long way from a small prompt. And my prompt into <a href="https://antigravity.google/?utm_campaign=CDR_0x6cb6c9c7_default_b573117953&amp;utm_medium=external&amp;utm_source=blog">Google Antigravity</a> was simple:</p><figure><img alt="Prompt in Antigravity: “my website lucia.dev is due for a revamp. Help me improve it. There’s google analytics data in this Google Cloud project (dataset `analytics_1234`). Give me a really nice google slides presentation with recommendations. I like unicorns.”" src="https://cdn-images-1.medium.com/max/1024/1*I0C3tNPyH3i46vPLhJJH6g.png" /></figure><p>And that was it. Less than an hour later it had kicked off multiple tasks in parallel, written scripts, written SQL, figured out my favorite colors and created a design with an implementation roadmap. And yes, the nice slides were also there.</p><h3>The behind the scenes</h3><p>Here’s what this <a href="https://www.youtube.com/watch?v=NhKpAUQQcFE">demo video</a> didn’t show in case you are curious.</p><p><strong>Warm-up</strong></p><ol><li>It picked up the right skills. I feel we need to demystify the excessive use of tokens in discovering the obvious. There are a lot of skills I pre-loaded in Antigravity which tell models how to interact with Google Cloud, BigQuery and more (most come from the <a href="https://docs.cloud.google.com/data-agent-kit?utm_campaign=CDR_0x6cb6c9c7_default_b573117953&amp;utm_medium=external&amp;utm_source=blog">Data Agent Kit</a>). There’s more to the context than the prompt I gave it.</li><li>Ran a pre-flight check: Checked authorization into GCP and access to BigQuery. Checked my website (lucia.dev).</li><li>Used the gslides skill to create a new slide</li></ol><p><strong>The GA4 schema</strong>:</p><ol><li>Instead of guessing table names in the analytics dataset, the discovering-gcp-data-assets skill requires searching Knowledge Catalog (f.k.a Dataplex) first, and then checking the table inventory so we know the exact date range and table prefixes. This makes a lot of sense, as Knowledge Catalog has a graph with semantics and descriptions of the tables and columns.</li><li>Discovered <strong>430 daily events_* tables</strong> (events_20250504 to events_20261007), <strong>430 pseudonymous_users_* tables</strong>, and a pre-aggregated <strong>results table</strong>.</li><li><strong>It formulated questions</strong>. I found this thought process particularly interesting. I’ve ran similar experiments with other models but these questions were broad across marketing, SEO and UX. <em>Where do visitors come from? What devices are they on? Which pages do they view vs. bounce from? What outbound links do they actually click?</em></li><li>Wrote some pretty complex SQL. While this is possible with other models, this one worked on its first try and contains some optimizations (e.g., the TEMP table, use of plain functions instead of REGEXP). Isn’t this pretty?</li></ol><pre>- 1. Materialize base data to avoid scanning the raw events table 7 times<br>CREATE TEMP TABLE MyEnrichedEvents AS<br>WITH raw_events AS (<br>SELECT<br>event_date,<br>event_timestamp,<br>event_name,<br>user_pseudo_id,<br>device.category AS device_category,<br>device.operating_system AS os,<br>device.web_info.browser AS browser,<br>geo.country AS country,<br>geo.city AS city,<br>traffic_source.source AS first_user_source,<br>traffic_source.medium AS first_user_medium,<br>traffic_source.name AS first_user_campaign,<br>collected_traffic_source.manual_source AS session_source,<br>collected_traffic_source.manual_medium AS session_medium,<br>(SELECT value.int_value FROM UNNEST(event_params) WHERE key = &#39;ga_session_id&#39; LIMIT 1) AS session_id,<br>(SELECT value.int_value FROM UNNEST(event_params) WHERE key = &#39;ga_session_number&#39; LIMIT 1) AS session_number,<br> - Optimized: Extract once and combine string/int coalesce checks natively<br>(SELECT COALESCE(SAFE_CAST(value.string_value AS INT64), value.int_value) FROM UNNEST(event_params) WHERE key = &#39;session_engaged&#39; LIMIT 1) AS session_engaged,<br>COALESCE((SELECT value.int_value FROM UNNEST(event_params) WHERE key = &#39;engagement_time_msec&#39; LIMIT 1), 0) AS engagement_time_msec,<br> - Optimized: Use SPLIT instead of REGEXP_REPLACE for string extraction<br>SPLIT(COALESCE((SELECT value.string_value FROM UNNEST(event_params) WHERE key = &#39;page_location&#39; LIMIT 1), &#39;&#39;), &#39;?&#39;)[OFFSET(0)] AS clean_page_location,<br>(SELECT value.string_value FROM UNNEST(event_params) WHERE key = &#39;page_location&#39; LIMIT 1) AS raw_page_location,<br>(SELECT value.string_value FROM UNNEST(event_params) WHERE key = &#39;page_title&#39; LIMIT 1) AS page_title,<br>(SELECT value.string_value FROM UNNEST(event_params) WHERE key = &#39;page_referrer&#39; LIMIT 1) AS page_referrer,<br>(SELECT value.string_value FROM UNNEST(event_params) WHERE key = &#39;link_url&#39; LIMIT 1) AS link_url,<br>(SELECT value.string_value FROM UNNEST(event_params) WHERE key = &#39;link_domain&#39; LIMIT 1) AS link_domain,<br>(SELECT value.string_value FROM UNNEST(event_params) WHERE key = &#39;link_text&#39; LIMIT 1) AS link_text,<br>(SELECT value.int_value FROM UNNEST(event_params) WHERE key = &#39;percent_scrolled&#39; LIMIT 1) AS percent_scrolled<br>FROM `my_gcp_project.analytics_12345.events_*`<br>WHERE _TABLE_SUFFIX BETWEEN &#39;20250504&#39; AND &#39;20261007&#39;<br>)<br>SELECT<br>*,<br>CONCAT(user_pseudo_id, &#39;_&#39;, CAST(COALESCE(session_id, 0) AS STRING)) AS full_session_id<br>FROM raw_events;<br> - 2. Execute aggregations directly on the materialized temporary table<br>SELECT<br>&#39;01_OVERALL&#39; AS section,<br>&#39;ALL_TIME&#39; AS dimension_1,<br>CONCAT(MIN(event_date), &#39;_to_&#39;, MAX(event_date)) AS dimension_2,<br>COUNT(*) AS total_events,<br>APPROX_COUNT_DISTINCT(user_pseudo_id) AS unique_users,<br>APPROX_COUNT_DISTINCT(full_session_id) AS total_sessions,<br>COUNTIF(event_name = &#39;page_view&#39;) AS page_views,<br>COUNTIF(event_name = &#39;scroll&#39;) AS scrolls,<br>COUNTIF(event_name = &#39;click&#39;) AS clicks,<br>ROUND(100.0 * APPROX_COUNT_DISTINCT(IF(session_engaged = 1, full_session_id, NULL)) / NULLIF(APPROX_COUNT_DISTINCT(full_session_id), 0), 1) AS engaged_session_pct,<br>ROUND(SUM(engagement_time_msec) / 1000.0 / NULLIF(APPROX_COUNT_DISTINCT(full_session_id), 0), 1) AS avg_session_eng_sec<br>FROM MyEnrichedEvents<br>UNION ALL<br>SELECT<br>&#39;02_MONTHLY&#39; AS section,<br>SUBSTR(event_date, 1, 6) AS dimension_1,<br>&#39;&#39; AS dimension_2,<br>COUNT(*) AS total_events,<br>APPROX_COUNT_DISTINCT(user_pseudo_id) AS unique_users,<br>APPROX_COUNT_DISTINCT(full_session_id) AS total_sessions,<br>COUNTIF(event_name = &#39;page_view&#39;) AS page_views,<br>COUNTIF(event_name = &#39;scroll&#39;) AS scrolls,<br>COUNTIF(event_name = &#39;click&#39;) AS clicks,<br>ROUND(100.0 * APPROX_COUNT_DISTINCT(IF(session_engaged = 1, full_session_id, NULL)) / NULLIF(APPROX_COUNT_DISTINCT(full_session_id), 0), 1) AS engaged_session_pct,<br>ROUND(SUM(engagement_time_msec) / 1000.0 / NULLIF(APPROX_COUNT_DISTINCT(full_session_id), 0), 1) AS avg_session_eng_sec<br>FROM MyEnrichedEvents<br>GROUP BY 1, 2, 3<br>UNION ALL<br>SELECT<br>&#39;03_PAGES&#39; AS section,<br>clean_page_location AS dimension_1,<br>COALESCE(ANY_VALUE(page_title), &#39;&#39;) AS dimension_2,<br>COUNT(*) AS total_events,<br>APPROX_COUNT_DISTINCT(user_pseudo_id) AS unique_users,<br>APPROX_COUNT_DISTINCT(full_session_id) AS total_sessions,<br>COUNTIF(event_name = &#39;page_view&#39;) AS page_views,<br>COUNTIF(event_name = &#39;scroll&#39;) AS scrolls,<br>COUNTIF(event_name = &#39;click&#39;) AS clicks,<br>ROUND(100.0 * APPROX_COUNT_DISTINCT(IF(session_engaged = 1, full_session_id, NULL)) / NULLIF(APPROX_COUNT_DISTINCT(full_session_id), 0), 1) AS engaged_session_pct,<br>ROUND(SUM(engagement_time_msec) / 1000.0 / NULLIF(APPROX_COUNT_DISTINCT(full_session_id), 0), 1) AS avg_session_eng_sec<br>FROM MyEnrichedEvents<br>GROUP BY 1, 2<br>UNION ALL<br>SELECT<br>&#39;04_SOURCES&#39; AS section,<br>COALESCE(session_source, first_user_source, &#39;(direct)&#39;) AS dimension_1,<br>COALESCE(session_medium, first_user_medium, &#39;(none)&#39;) AS dimension_2,<br>COUNT(*) AS total_events,<br>APPROX_COUNT_DISTINCT(user_pseudo_id) AS unique_users,<br>APPROX_COUNT_DISTINCT(full_session_id) AS total_sessions,<br>COUNTIF(event_name = &#39;page_view&#39;) AS page_views,<br>COUNTIF(event_name = &#39;scroll&#39;) AS scrolls,<br>COUNTIF(event_name = &#39;click&#39;) AS clicks,<br>ROUND(100.0 * APPROX_COUNT_DISTINCT(IF(session_engaged = 1, full_session_id, NULL)) / NULLIF(APPROX_COUNT_DISTINCT(full_session_id), 0), 1) AS engaged_session_pct,<br>ROUND(SUM(engagement_time_msec) / 1000.0 / NULLIF(APPROX_COUNT_DISTINCT(full_session_id), 0), 1) AS avg_session_eng_sec<br>FROM MyEnrichedEvents<br>GROUP BY 1, 2, 3<br>UNION ALL<br>SELECT<br>&#39;05_DEVICES&#39; AS section,<br>COALESCE(device_category, &#39;(none)&#39;) AS dimension_1,<br>COALESCE(os, &#39;(none)&#39;) AS dimension_2,<br>COUNT(*) AS total_events,<br>APPROX_COUNT_DISTINCT(user_pseudo_id) AS unique_users,<br>APPROX_COUNT_DISTINCT(full_session_id) AS total_sessions,<br>COUNTIF(event_name = &#39;page_view&#39;) AS page_views,<br>COUNTIF(event_name = &#39;scroll&#39;) AS scrolls,<br>COUNTIF(event_name = &#39;click&#39;) AS clicks,<br>ROUND(100.0 * APPROX_COUNT_DISTINCT(IF(session_engaged = 1, full_session_id, NULL)) / NULLIF(APPROX_COUNT_DISTINCT(full_session_id), 0), 1) AS engaged_session_pct,<br>ROUND(SUM(engagement_time_msec) / 1000.0 / NULLIF(APPROX_COUNT_DISTINCT(full_session_id), 0), 1) AS avg_session_eng_sec<br>FROM MyEnrichedEvents<br>GROUP BY 1, 2, 3<br>UNION ALL<br>SELECT<br>&#39;06_CLICKS&#39; AS section,<br>COALESCE(link_url, link_domain, &#39;(none)&#39;) AS dimension_1,<br>COALESCE(clean_page_location, &#39;&#39;) AS dimension_2,<br>COUNT(*) AS total_events,<br>APPROX_COUNT_DISTINCT(user_pseudo_id) AS unique_users,<br>APPROX_COUNT_DISTINCT(full_session_id) AS total_sessions,<br>0 AS page_views,<br>0 AS scrolls,<br>COUNT(*) AS clicks,<br>0.0 AS engaged_session_pct,<br>0.0 AS avg_session_eng_sec<br>FROM MyEnrichedEvents<br>WHERE event_name = &#39;click&#39;<br>GROUP BY 1, 2, 3<br>UNION ALL<br>SELECT<br>&#39;07_EVENTS&#39; AS section,<br>event_name AS dimension_1,<br>&#39;&#39; AS dimension_2,<br>COUNT(*) AS total_events,<br>APPROX_COUNT_DISTINCT(user_pseudo_id) AS unique_users,<br>APPROX_COUNT_DISTINCT(full_session_id) AS total_sessions,<br>0 AS page_views,<br>0 AS scrolls,<br>0 AS clicks,<br>0.0 AS engaged_session_pct,<br>ROUND(SUM(engagement_time_msec) / 1000.0, 1) AS avg_session_eng_sec<br>FROM MyEnrichedEvents<br>GROUP BY 1, 2, 3<br>ORDER BY section, total_sessions DESC;</pre><p>This is the output of this one-off, ad-hoc SQL script:</p><ul><li><strong>01_OVERALL</strong>: The grand totals for the entire date range.</li><li><strong>02_MONTHLY</strong>: Totals broken down by month (e.g., “202505”).</li><li><strong>03_PAGES</strong>: Traffic and engagement grouped by specific page URLs (with query parameters removed) and page titles.</li><li><strong>04_SOURCES</strong>: Traffic broken down by where the user came from (source and medium, like google / organic or direct / none).</li><li><strong>05_DEVICES</strong>: Traffic grouped by the user’s device category (mobile, desktop) and operating system (iOS, Android, Windows).</li><li><strong>06_CLICKS</strong>: A zoomed-in view of just click events, detailing which outbound links/domains users clicked on from which pages.</li><li><strong>07_EVENTS</strong>: A simple breakdown of the total frequency of each distinct event_name tracked during the period.</li></ul><p>Check the notes at the end of this post if you are looking to borrow this SQL.</p><h4><strong>The website</strong></h4><p>This section would have been soul crushing if I had any emotional attachment to the previous website design. It wrote a scraping script and figured out a hidden subpage from the Google Analytics logs and scraped that too.</p><p>Argon asked for permission to take over the browser and test a hypothesis. It was right: there was a hidden rogue button that allowed anyone to insert a new record into a backend spreadsheet. Because this was not really very serious and I was ready to migrate out of this site, I got a good chuckle out of this.</p><p>I explicitly mentioned that I like unicorns and it also figured it out from my social media handles. In Gemini’s own words:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JUUtwxJyqXbFJjyFe0LxJg.png" /><figcaption>Gemini’s thinking process while redesigning the website.</figcaption></figure><p><strong>The slides</strong></p><p>You can check out the slides and stats in the <a href="https://youtu.be/NhKpAUQQcFE">video</a>, they were the best I’ve ever had generated by a model and I think this is related specifically to Gemini 4 multimodal design awareness.</p><p>I ended up using the recommendations in a summary for a new session in VSCode to revamp my website and it took me less than a few hours. The only manual work I had to do was write a few descriptions and paraphrase some others. I also had to audit any lingering references to topics I used to write a lot about but I am no longer interested in.</p><h3>A note on processing daily GA4 exports in BigQuery</h3><p>If you found this blog post while looking for a query to process your daily Google Analytics export you can schedule, I got you.</p><p>Create a destination table first (replace with your project and dataset):</p><pre>CREATE TABLE `my_gcp_project.analytics_12345.daily_events_summary`<br>(<br>event_date STRING,<br>section STRING,<br>dimension_1 STRING,<br>dimension_2 STRING,<br>total_events INT64,<br>unique_users INT64,<br>total_sessions INT64,<br>page_views INT64,<br>scrolls INT64,<br>clicks INT64,<br>engaged_session_pct FLOAT64,<br>avg_session_eng_sec FLOAT64<br>)<br>PARTITION BY PARSE_DATE<br>OPTIONS ( description = &#39;Daily events from GA4&#39;, <br>labels = [(&#39;goog-yardstick-devrel&#39;, &#39;573117953&#39;)] );</pre><p>To process the daily export, you will use this query <a href="https://gist.github.com/Lsubatin/ea3589cb02fd11d678afe8daa94f6d3d">here</a>. The first time your run this, you will comment out this line so your entire dataset gets scanned:</p><pre>WHERE _TABLE_SUFFIX BETWEEN FORMAT_DATE(&#39;%Y%m%d&#39;, DATE_SUB(CURRENT_DATE(), INTERVAL 3 DAY))<br>AND FORMAT_DATE(&#39;%Y%m%d&#39;, DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY))</pre><p>After this, you can schedule the same query to run every day. Here are instructions to use <a href="https://docs.cloud.google.com/bigquery/docs/scheduling-queries?utm_campaign=CDR_0x6cb6c9c7_default_b573117953&amp;utm_medium=external&amp;utm_source=blog">the scheduler</a>.</p><p>This will MERGE the last three days of records. Why the last 3? Because GA4 exports do not always land at the same time and have delays sometimes, so you can’t quite anticipate when the new record will make it. You could make the window shorter to process less days (and make the query cheaper) but you may miss records. You could also create a Cloud Function triggered after the export happens, but that would be for another blog post. Let me know if you’re interested in that.</p><p>If you made it this far and want to stay in touch, check out my new website lucia.dev 🦄</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d39d13b9327" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/asking-gemini-4-argon-to-revamp-my-website-from-ga4-data-8d39d13b9327">Asking Gemini 4 (Argon) to revamp my website from GA4 data</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spanner Omni: Everything, everywhere, all of the time.]]></title>
            <link>https://medium.com/google-cloud/spanner-omni-everything-everywhere-all-of-the-time-b2ceea5cbfb0?source=rss-910459c47504------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2ceea5cbfb0</guid>
            <category><![CDATA[multi-cloud-strategy]]></category>
            <category><![CDATA[spanner]]></category>
            <category><![CDATA[google-cloud-spanner]]></category>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Lucía Subatin]]></dc:creator>
            <pubDate>Fri, 14 Aug 2026 03:35:10 GMT</pubDate>
            <atom:updated>2026-08-14T03:35:10.160Z</atom:updated>
            <content:encoded><![CDATA[<p>You know it’s good when a long-time cloud professional gets excited about running something locally. <a href="https://cloud.google.com/products/spanner/omni?utm_campaign=CDR_0x6cb6c9c7_default_b542779672&amp;utm_medium=external&amp;utm_source=blog">Spanner Omni</a> allows you to run <a href="https://cloud.google.com/spanner?utm_campaign=CDR_0x6cb6c9c7_default_b542779672&amp;utm_medium=external&amp;utm_source=blog">Spanner</a> on-premises and on multiple cloud providers with the same magic as the managed Spanner instances on Google Cloud. This is my experience setting it up on macOS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*Du8Q8T059E4uKjoHoeL-vQ.gif" /><figcaption>Execute SQL statements in Spanner Omni container</figcaption></figure><h3>Why Spanner Omni?</h3><p>Allow me to get a little personal and explain why I like this: I’ve been told I have some commitment issues, but the truth is I like to have options. This way, I can choose the best tool for the different problems (professional) life throws at me. Having a managed database at the Spanner scale is my default answer to many of my very intensive transactional challenges. The type of challenge you get with, for example, requiring a highly-scalable database with strong global consistency in industries like banking services or gaming. I know others also like to have choices: teams on other cloud providers like AWS, organizations requiring data residency compliance, or developers testing local AI agent workflows.</p><p>Spanner Omni brings the same capabilities as Cloud Spanner but grants you the freedom to deploy anywhere — on VMs, bare metal, Linux containers, or Kubernetes clusters. It also lets you scale compute and storage separately, preventing costly over-provisioning (I pinky-promise I will share this in a follow-up blog post).</p><p>To top it off, Spanner Omni is multi-model. You can store relational, NoSQL, and Graph data in one place. So if you also like to keep your options open, this is great news.</p><h3>What about the powerful infrastructure that makes Cloud Spanner, Spanner?</h3><p>As we know, Cloud Spanner uses <a href="https://docs.cloud.google.com/spanner/docs/true-time-external-consistency?utm_campaign=CDR_0x6cb6c9c7_default_b542779672&amp;utm_medium=external&amp;utm_source=blog">TrueTime</a>, Google’s globally synchronized clock that keeps time accurate across servers with an atomic clock and GPS receivers. It also uses the almighty <a href="https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system?utm_campaign=CDR_0x6cb6c9c7_default_b542779672&amp;utm_medium=external&amp;utm_source=blog">Colossus</a> storage system to ensure its distributed power and consistency at planetary scale.</p><p>So what happens with that atomic clock that keeps TrueTime in an Omni instance? Let’s nerd out.</p><p>Under the hood, the <a href="https://en.wikipedia.org/wiki/Paxos_(computer_science)">Paxos</a> algorithm maintains consensus across zones by replicating data to be written among a quorum of database nodes to guarantee durability. At the same time, to guarantee chronological consistency (i.e., how do you make sure this transaction in one server came before this other transaction in a different server), Spanner must wait out its clock uncertainty before committing a change. In Cloud Spanner, TrueTime provides this small uncertainty window. In Spanner Omni, a software implementation of TrueTime keeps a reliable time reference and an uncertainty window.</p><p>Without physical atomic clocks or GPS receivers, we pivot to a leader-elected time server located in the preferred leader zone or region (actually, for availability, you’ll have multiple time-servers per region who can perform a hot failover if needed). Every node in your Spanner Omni deployment synchronizes with this primary source. Since writes usually target the same leader zone where your time server lives, you get low TrueTime uncertainty which translates to low write latency as long as:</p><p><strong><em>the time Spanner needs to wait to commit (uncertainty commit wait) &lt; time it takes for Paxos to replicate among quorum of database nodes (Paxos Round Trip Time)</em></strong></p><p>As for Colossus — Google’s exabyte-scale storage platform behind Spanner, BigQuery, YouTube, and more — you obviously can’t have that locally. Or can you? Spanner Omni implements an abstraction layer of Colossus. While the infrastructure will be hard to match, this abstraction layer treats the storage as a unified pool that is detached from the compute layer. This means a VM can access the storage of another VM, so the platform can automatically redistribute data shards based on load and capacity to optimize performance without manual intervention.</p><h3>Enough nerding out, let’s try Spanner locally.</h3><h4>First, System Requirements</h4><p>Before we start our engines, we need to make sure our machine is beefy enough to run this powerhouse. Spanner Omni is highly optimized, but it still needs <a href="https://docs.cloud.google.com/spanner-omni/system-requirements?utm_campaign=CDR_0x6cb6c9c7_default_b542779672&amp;utm_medium=external&amp;utm_source=blog">room to breathe</a>.</p><ul><li><strong>Linux</strong>: your server should have an x86–64 CPU, RHEL 9 or Ubuntu 22, and at least 4 GB of RAM for every vCPU allocated to Spanner, plus 20 GB of disk space.</li><li><strong>macOS</strong>: you’ll need macOS 14.7+, Apple Silicon (M1/M2/M3 or later) CPU, 4 GB of RAM, and 10 GB of disk space.</li></ul><p>Make sure your system meets these, or your local Spanner deployment might end up more “shattered” than sharded.</p><p>Of course, you’ll need to download Docker Engine 24.0+ or Podman 3.0+ AND the <a href="https://docs.cloud.google.com/spanner-omni/download?utm_campaign=CDR_0x6cb6c9c7_default_b542779672&amp;utm_medium=external&amp;utm_source=blog#cli_binaries">Spanner Omni CLI</a>.</p><h3>Step-by-step instructions</h3><p>I’m using Docker and macOS for these examples. First, I pulled the image. I recommend copying the command from <a href="https://docs.cloud.google.com/spanner-omni/download?utm_campaign=CDR_0x6cb6c9c7_default_b542779672&amp;utm_medium=external&amp;utm_source=blog#container_images">here</a> so you have the latest version.</p><ol><li>Create a volume</li></ol><pre>docker volume create spanner</pre><p>2. Run the container:</p><p>You’ll see a slight trick here: ports are mapped manually, unlike the Linux command <a href="https://docs.cloud.google.com/spanner-omni/setup?utm_campaign=CDR_0x6cb6c9c7_default_b542779672&amp;utm_medium=external&amp;utm_source=blog#run-docker">in the documentation</a>. That’s because Docker on macOS runs in a VM that can’t directly share your host’s network.</p><pre>docker run -d -p 15000:15000 -p 15026:15026 \<br> - name spanneromni \<br>-v &quot;spanner:/spanner&quot; \<br>us-docker.pkg.dev/spanner-omni/images/spanner-omni:&lt;&lt;THE TAG YOU PULLED EARLIER&gt;&gt; \<br>start-single-server</pre><p>The full string of the container image may look like this: <a href="http://us-docker.pkg.dev/spanner-omni/images/spanner-omni:2026.r1-beta.2">us-docker.pkg.dev/spanner-omni/images/spanner-omni:2026.r1-beta.2</a></p><p>Why those ports? 15000 is for client connection (all the way to 15025) and 15026 is the Spanner Omni console server.</p><p>3. Verify that your container is healthy and running:</p><pre>docker ps</pre><p>This should say “Up xx time” under “STATUS”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Drecx0254oOQoxutNNbP-Q.png" /></figure><p>4. Next, create a sample retail database called customer_db to test things out. If you were in the console, you would find these pre-loaded sample databases after creating an instance. As of the date of this post, the samples are for Finance graph, Banking, Retail, and Gaming. Rumor has it, there are more coming.</p><pre>docker exec -it spanneromni /google/spanner/bin/spanner \<br>databases create-sample-db retail - database-name=customer_db</pre><p>5. To run SQL queries directly on your database, connect using the Spanner SQL shell:</p><pre>docker exec -it spanneromni /google/spanner/bin/spanner \<br>sql - database=customer_db</pre><p>You can exit this CLI with <em>quit</em> or <em>\q</em></p><p>6. If you want a visual interface with the general health of your Spanner Omni, check out the Console:</p><pre>docker exec -it spanneromni /app/bin/spanner-console</pre><p>This command will start the Spanner Console server if it’s not running and serve it on port 15026. Open <a href="http://localhost:15026/">http://localhost:15026</a> and you’ll get some useful info and fancy graphics on your databases, backups, system and query insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*elGgGeOd1gB4-YeEgAOKNA.gif" /><figcaption>Opening 127.0.0.1:15026</figcaption></figure><p>If you want to delete the container for any reason, here’s how:</p><pre>docker stop spanneromni<br>docker rm spanneromni</pre><p>As long as you don’t delete the volume, any changes you’ve made to the database itself will stay persisted when you run the new “docker run” command.</p><p>There are other handy commands <a href="https://docs.cloud.google.com/spanner-omni/cli-quickstart?utm_campaign=CDR_0x6cb6c9c7_default_b542779672&amp;utm_medium=external&amp;utm_source=blog">in the docs</a>.</p><p>I have to call out the fine-print of the <a href="https://docs.cloud.google.com/spanner-omni/overview?utm_campaign=CDR_0x6cb6c9c7_default_b542779672&amp;utm_medium=external&amp;utm_source=blog#limitations">limitations</a>, and I will be sharing more of this journey on Kubernetes and for local development use cases.</p><p>Have fun!</p><p>Lucia Subatin</p><p><a href="https://www.linkedin.com/in/lsubatin/">LinkedIn</a>,<a href="https://www.youtube.com/channel/UCSq36lGVNWTvc33Tee4WNZg"> YouTube</a>,<a href="https://www.tiktok.com/@unicorn.on.the.cl"> TikTok</a>,<a href="https://www.instagram.com/a.cloudy.unicorn/"> Instagram</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2ceea5cbfb0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/spanner-omni-everything-everywhere-all-of-the-time-b2ceea5cbfb0">Spanner Omni: Everything, everywhere, all of the time.</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The “Dots” Just Got a Lot Easier to Connect: BigQuery Graph is Here!]]></title>
            <link>https://medium.com/google-cloud/the-dots-just-got-a-lot-easier-to-connect-bigquery-graph-is-here-c6a0cb580587?source=rss-910459c47504------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6a0cb580587</guid>
            <category><![CDATA[neo4j]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Lucía Subatin]]></dc:creator>
            <pubDate>Mon, 13 Apr 2026 21:14:09 GMT</pubDate>
            <atom:updated>2026-04-13T21:14:09.861Z</atom:updated>
            <content:encoded><![CDATA[<p>Connecting the dots in your data shouldn’t feel like a conspiracy theory. BigQuery Graph is now in Public Preview, allowing you to model, query, and visualize complex relationships natively with GQL. This post dives into why graphs are a great addition to your data engineering stack and how to start building them today. <strong>🟢 — — 🟢</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wnK7gVE7x1mU8ZkYK1_fRg.png" /></figure><h3>The “Red String” Problem: Why Graphs?</h3><p>Traditional databases give you a list of suspects, but graph databases give you the “red string” that solves the crime. In a graph, <strong>Nodes</strong> represent your entities (like customers or products), and <strong>Edges</strong> map the relationships between them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*LNHu0UDHjkNoJgOaQzwAcQ.gif" /><figcaption>Nodes = Entities, Edges = relationships between the entities</figcaption></figure><p>Graphs are absolute game-changers for:</p><ul><li><strong>Distance or shortest path calculation</strong>: Dijkstra, anyone? This is one of the most common applications we see in software engineering for networking or physical distance. In this same context, <strong>software dependencies</strong> are always better traversed as a graph.</li><li><strong>Social relationships: </strong>Social media platforms like Facebook and LinkedIn use graphs to map relationships between users.</li><li><strong>Fraud Detection:</strong> Identifying circular money transfers or bot rings.</li><li><strong>Supply Chain:</strong> Understanding how a disaster at one warehouse ripples through your entire network.</li><li><strong>Knowledge Graphs:</strong> Giving AI agents an additional layer of context so they can traverse ground truths instead of just guessing. I tried this out in Spanner (<a href="https://github.com/GoogleCloudPlatform/cloud-spanner-samples/tree/main/adk-knowledge-graph">code sample</a> with Langgraph) and my friend <a href="https://www.linkedin.com/in/rachael-ds/">Rachael</a> showcases them <a href="https://github.com/GoogleCloudPlatform/devrel-demos/tree/main/data-analytics/knowledge_graph_demo">here</a>. In most cases, we got better accuracy and less token usage.</li></ul><p><strong>When are they NOT useful?</strong> If you just need to sum up total sales for the month or run aggregations on measures, stick to standard SQL. Graphs shine when the <em>relationship</em> is the data. Your good ol’ Profit and Loss report is better off remaining tabular, so that sweet columnar processing can efficiently aggregate the numbers. Other aspects that usually require a well-defined auxiliary dimension, like time travelling, may become unnecessarily complex in a graph too.</p><h3>Talk is cheap, show me the JOINS</h3><p>Before BigQuery Graph, finding a circular transaction (Account A → B → C → A) required a nightmare of self-joins. It was slow to write and expensive to compute — an <a href="https://docs.cloud.google.com/bigquery/docs/best-practices-performance-compute#avoid_self_joins?utm_campaign=CDR_0x6cb6c9c7_default_b502002276&amp;utm_medium=external&amp;utm_source=blog">anti-pattern</a>.</p><p>Now, we use <strong>Graph Query Language (GQL)</strong>. It’s cleaner, faster, and stays right next to your existing tables — no data movement required.</p><p>Check out the difference in elegance:</p><p><strong>Relational:</strong></p><pre>SELECT c.name<br>FROM `finance_ds.Customers` c<br>JOIN `finance_ds.Accounts` src ON c.cid = src.customer_id<br>JOIN `finance_ds.Transactions` t1 ON src.aid = t1.from_aid<br>JOIN `finance_ds.Transactions` t2 ON t1.to_aid = t2.from_aid<br>JOIN `finance_ds.Transactions` t3 ON t2.to_aid = t3.from_aid<br>WHERE t3.to_aid = src.aid; - Finding the loop</pre><p><strong>Graph:</strong></p><pre>GRAPH `finance_ds.BankGraph`<br>MATCH (c:Customer)-[:OWNS]-&gt;(src:Account)-[:TRANSFER]-&gt;{3}(src)<br>RETURN c.name AS Suspicious_Customer;</pre><h3>Wait… Spanner can run GQL too?</h3><p>Yes! Spanner has had graph capabilities for a while, and these are also growing. There are many examples in this <a href="https://github.com/GoogleCloudPlatform/cloud-spanner-samples/tree/main">Git repo</a>. So, which one do you use? It comes down to your workload.</p><ul><li><strong>BigQuery Graph:</strong> Best for <strong>analytical</strong> use cases. Use this when you want to run complex graph queries on massive datasets or perform analysis alongside your data warehouse. BigQuery is optimized for petabyte processing, in near real time or batch.</li><li><strong>Spanner Graph:</strong> Best for <strong>operational</strong> use cases. Use this for real-time response where you need sub-second updates and global consistency.</li></ul><p>The good news is that both Spanner and BigQuery run GQL, so if you are new to the syntax, you only need to learn it once. Here’s a little cheat sheet in case it helps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a-ydrn-_lniADJUeJ_Vb4A.png" /></figure><h3>Ready to connect your own dots?</h3><p>Check out these resources to get started:</p><ul><li>📖<a href="https://goo.gle/bq-graph-docs"> Official BigQuery Graph Docs</a></li><li>🧪<a href="https://goo.gle/bq-graph-fraud-detect-lab"> Codelab: Fraud Detection</a></li><li>🧪<a href="https://goo.gle/bq-graph-c360-lab"> Codelab: Customer 360</a></li><li>🎥<a href="https://youtube.com/shorts/_9YTyst9xWg"> Quick Video Intro</a></li></ul><p>And if you are going to Google Cloud Next 2026, be sure to sign up for this <a href="https://www.googlecloudevents.com/next-vegas/session-library?session_id=3912233&amp;name=bigquery-graph-uncover-complex-hidden-relationships-in-your-data&amp;_gl=1*mfk3vb*_up*MQ..&amp;gclid=CjwKCAjwhe3OBhABEiwA6392zNEXkQf-hIc9gpBlgDzIyEKs4a7XP1eJcSKEOomziMCiNyVYfxsWgxoCxRMQAvD_BwE&amp;gclsrc=aw.ds&amp;gbraid=0AAAAApdQcwc2dLzbpmcVzKm-c0HRn-PXQ">technical session</a> for BigQuery and <a href="https://www.googlecloudevents.com/next-vegas/session-library?session_id=3912260&amp;name=build-advanced-ai-applications-with-spanner-graph&amp;_gl=1*1h0w04p*_up*MQ..&amp;gclid=CjwKCAjwhe3OBhABEiwA6392zNEXkQf-hIc9gpBlgDzIyEKs4a7XP1eJcSKEOomziMCiNyVYfxsWgxoCxRMQAvD_BwE&amp;gclsrc=aw.ds&amp;gbraid=0AAAAApdQcwc2dLzbpmcVzKm-c0HRn-PXQ">this one</a> for Spanner. While you are at it, don’t miss out on a great demo showcasing both at the Data Cloud booth.</p><p>Let’s stay in touch on <a href="https://www.linkedin.com/in/luciasubatin/">LinkedIn</a>, <a href="https://www.youtube.com/@LuciaSubatin">YouTube</a>, <a href="https://www.tiktok.com/@unicorn.on.the.cl">TikTok </a>or <a href="https://www.instagram.com/a.cloudy.unicorn/">Instagram</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6a0cb580587" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/the-dots-just-got-a-lot-easier-to-connect-bigquery-graph-is-here-c6a0cb580587">The “Dots” Just Got a Lot Easier to Connect: BigQuery Graph is Here!</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Spanner Schema Insights: Optimizing Your Global Database Design]]></title>
            <link>https://medium.com/google-cloud/introducing-spanner-schema-insights-optimizing-your-global-database-design-aa0ae7d4843f?source=rss-910459c47504------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa0ae7d4843f</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[cloud-spanner]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[application-development]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Lucía Subatin]]></dc:creator>
            <pubDate>Fri, 23 Jan 2026 18:22:36 GMT</pubDate>
            <atom:updated>2026-01-23T19:02:57.387Z</atom:updated>
            <content:encoded><![CDATA[<p>Spanner has long been the backbone for Google’s most critical services, from Gmail and YouTube to Photos and Chat. Spanner is a one of a kind database. Some even say it <a href="https://cloud.google.com/blog/products/databases/inside-cloud-spanner-and-the-cap-theorem?utm_campaign=CDR_0x6cb6c9c7_default_b454068849&amp;utm_medium=external&amp;utm_source=blog">defies the CAP Theorem</a>. This is an engineering hero story on a new release for Spanner: Schema insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tXRSqkxDbqKaur6EK8lI0A.gif" /></figure><p>This feature, long used by Google engineers to optimize the applications millions use every day, is now available to help you harness Spanner’s superpowers with ease.</p><p>As applications grow, their access patterns naturally change. To ensure your database remains perfectly tuned to these evolving needs, Google Cloud engineers have made these internal optimization tools accessible to everyone. Schema Insights acts as a proactive guide right next to your tables in Spanner Studio, helping you maintain a schema that is always optimized for speed and scale.</p><h3>Designing for High-Throughput Distribution</h3><p>Spanner achieves horizontal scalability by partitioning and distributing data across different splits and servers. To ensure maximum performance, it is a best practice to distribute your workload evenly across these servers. Think of it as balancing the amount of data each server has, so they can all search and update at the same time.</p><h4><strong>Monotonic keys and hotspots</strong></h4><p>When a primary key uses a timestamp or an increasing integer, new records may land in the same partition. This can potentially create a “hotspot” as your applications access one partition predominantly more than others.</p><p>Insights will recommend strategies like hashing, bucketing, or bit-reversing key columns to ensure your data is distributed effectively for parallel processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pUqRsivxS7l_UTxz0u_Lqg.gif" /><figcaption>Schema Insights highlighting monotonic keys</figcaption></figure><p>This is a common design pitfall that might not be obvious if we don’t have the distributed data storage aspect in mind. The good news is this can also be easily fixed. Consider <a href="https://cloud.google.com/spanner/docs/schema-design?utm_campaign=CDR_0x6cb6c9c7_default_b454068849&amp;utm_medium=external&amp;utm_source=blog">these recommendations</a> when designing the schema for your distributed database on Spanner.</p><h3>Streamlining Schema Architecture</h3><p>In a distributed database, the complexity of your schema directly impacts the performance of transactions and queries. Keeping the schema lean ensures that our application stays fast and maintainable.</p><p>A complex schema not only gets in the way of clean and maintainable code, it can make it harder to scale horizontally, increase latency and degrade performance of write and read operations. The insights below basically are designed to guide you through this process of designing and fine-tuning your schema the right way, and to avoid certain pitfalls that might not necessarily be obvious but lead to huge performance penalties.</p><h4><strong>Table density</strong></h4><p>While Spanner is fully relational, highly normalized schemas can increase memory and server overhead. A large number of tables may indicate a schema may need a redesign to apply patterns such as <a href="https://cloud.google.com/spanner/docs/implement-multi-tenancy?_gl=1*1vj4iio*_ga*MTQ2MTg0MTcyNi4xNzYwOTkyMjc1*_ga_WH2QY8WWF5*czE3NjEwMDk0ODYkbzIkZzEkdDE3NjEwMTUwNzUkajYwJGwwJGgw&amp;utm_campaign=CDR_0x6cb6c9c7_default_b454068849&amp;utm_medium=external&amp;utm_source=blog">multitenancy</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GBP7FwH56Rv1yrALM3S4Zw.gif" /></figure><h4>Too many columns</h4><p>This should be hard to reach and it probably means it’s time for a more holistic view of the schema design, but a table can have a maximum of 1024 columns. The issue will be highlighted when a table starts approaching that limit so we have time to react.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AVik6Gwo3HCLBT-vf9Lllg.gif" /></figure><h4>Relational integrity</h4><p>Maintaining foreign key references requires additional data to be consulted during transactions, which can slow down processing. Optimize for transactional speed by balancing the number of foreign key references.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GZ9WeIDG9Zq2_2J69PSi0g.png" /></figure><h3>Unlock Deeper Observability</h3><p>Schema Insights works within Spanner Studio so you have the schema and context handy. It also works in tandem with Spanner’s suite of observability tools.</p><p>For example, the <a href="https://docs.cloud.google.com/spanner/docs/key-visualizer?utm_campaign=CDR_0x6cb6c9c7_default_b454068849&amp;utm_medium=external&amp;utm_source=blog">Key Visualizer</a> allows you to visualize your specific traffic patterns and see exactly how your schema choices affect data distribution. To further understand potential problematic queries, <a href="http://docs.cloud.google.com/spanner/docs/using-query-insights?utm_campaign=CDR_0x6cb6c9c7_default_b454068849&amp;utm_medium=external&amp;utm_source=blog">Query Insights</a> will highlight issues like high CPU utilization.</p><p>The best part of these tools is they highlight your specific access patterns, so you can tailor your next steps based on your needs and knowledge of your applications.</p><p>We’d love to know how this is working for you.</p><p>Let’s stay in touch on <a href="https://www.tiktok.com/@unicorn.on.the.cl">TikTok</a>, <a href="https://www.instagram.com/a.cloudy.unicorn/">Instagram</a> or <a href="https://www.youtube.com/@LuciaSubatin">YouTube</a>!</p><p>This post was co-authored with the product and engineering heroes of this story:</p><p><a href="https://www.linkedin.com/in/guoqiang-pan-1393086/">Guoqiang Pan</a>, I<a href="https://www.linkedin.com/in/igorgrunskiy">gor Grunskyi</a>, <a href="https://www.linkedin.com/in/jathreya">Jagan Athreya</a>, <a href="https://www.linkedin.com/in/masha-lifshits/">Masha Lifshits</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa0ae7d4843f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/introducing-spanner-schema-insights-optimizing-your-global-database-design-aa0ae7d4843f">Introducing Spanner Schema Insights: Optimizing Your Global Database Design</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taking the BigQuery Data Engineering Agent to the Next Level: Planning, Optimization and…]]></title>
            <link>https://medium.com/google-cloud/taking-the-bigquery-data-engineering-agent-to-the-next-level-planning-optimization-and-15e3b6cebe73?source=rss-910459c47504------2</link>
            <guid isPermaLink="false">https://medium.com/p/15e3b6cebe73</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[dataform]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Lucía Subatin]]></dc:creator>
            <pubDate>Tue, 06 Jan 2026 16:05:38 GMT</pubDate>
            <atom:updated>2026-01-06T17:49:03.135Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Taking the BigQuery Data Engineering Agent to the Next Level: Planning, Optimization and Troubleshooting</strong></h3><p>Since my last post <a href="https://medium.com/google-cloud/exploring-the-newbigquery-data-engineering-agent-caf07e78d292">exploring the BigQuery Data Engineering Agent</a>, the team has been listening to your feedback and dropping updates that make the agent an even more powerful companion: Plan and Act, Optimizations and Troubleshooting. These are some of my experiments so far.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*MIB30c6RBkO-kaoJmJkGyw.gif" /><figcaption>BigQuery Data Engineering agent: Plan and act triggered by prompting</figcaption></figure><h3>Plan and Act</h3><p>As I started pushing the BigQuery Data Engineering Agent with more complex tasks — like transforming and joining clickstream data from a Google Analytics pet-store site with CRM sales records — I noticed a shift. The agent now enters <strong>Planning Mode</strong>.</p><p>Instead of just spitting out code, it creates a roadmap upfront, giving you visibility into its “thought process.” This is huge for complex, longer-horizon tasks where you need to make sure the agent hasn’t lost the plot halfway through.</p><p><strong>Prompting Tip:</strong> While the agent automatically plans when it sees a tough task, you can force it to put its thinking cap on by including <strong>“Create a plan to…” </strong>in your prompt.</p><p>According to internal results, this mode helps the agent complete complex requests with up to <strong>~30% fewer turns</strong> and better semantic quality. Essentially, it spends more time thinking so you spend less time correcting.</p><h3>Pipeline Optimization</h3><p>Building the pipeline is step one, but keeping it efficient is where the money is saved. The agent has gotten significantly smarter at <strong>optimizing the pipeline</strong> using proven BigQuery modeling best-practices<strong>.</strong></p><p>While executing, you will notice the agents will loop through a multi sub-agent approach to evaluate its design and apply optimizations. It will also propose updates to your pipeline design using best practices for performance tuning. We’re talking about things like smarter partitioning, clustering, or schema tweaks that can lead to <strong>up to 20% slot hour improvements.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1pc7ioyZLRiLhCxXlQOSEA.gif" /><figcaption>Automatic Pipeline Optimizations</figcaption></figure><h3>Troubleshooting</h3><p>This feature was launched around October but it’s worth a special mention. As you already know, the Data Engineering Agent is creating Dataform repositories with different SQLX artifacts. There will come a time in which you may need to manually modify these definitions in Dataform. It may also happen that an error occurs during the execution of the pipeline..</p><p>The integration with Code Assist now helps you find the root cause by formulating hypotheses and listing possible solutions to the issue. You’ll see the “sparkles” next to the error and the option to further investigate the issue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yD9vBfto0hFDKdvwPeIB7g.gif" /><figcaption>Troubleshooting</figcaption></figure><h3>Give it a try and let me know how it goes</h3><p>You can see a <a href="https://youtube.com/shorts/3fMh1s8sJkI">super short video demonstrating my experiments here</a>. In this specific video, I am using a replication from Google Analytics streaming traffic, clicks and conversions into BigQuery.</p><p>You can try this yourself in <a href="https://console.cloud.google.com/bigquery?utm_campaign=CDR_0x6cb6c9c7_default_b473371094&amp;utm_medium=external&amp;utm_source=social">BigQuery Studio</a>. Just click the context menu to “Create New… Pipeline” and start chatting with the agent.</p><p>If you want to check, modify or commit/push to the Dataform repository, you will find “Open pipeline in Dataform” under Settings. The rest of the workflow is the same as<a href="https://docs.cloud.google.com/dataform/docs/sql-workflows?utm_campaign=CDR_0x6cb6c9c7_default_b473371094&amp;utm_medium=external&amp;utm_source=social"> any other Dataform workflow</a>.</p><p>My next iterations will be on the improved Dataform integrations as some of you have requested. Please let me know if there is anything specific you would like to see.</p><p>Let’s stay in touch on <a href="https://www.tiktok.com/@unicorn.on.the.cl">TikTok</a>, <a href="https://www.instagram.com/a.cloudy.unicorn/">Instagram</a> or <a href="https://www.youtube.com/@LuciaSubatin">YouTube</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15e3b6cebe73" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/taking-the-bigquery-data-engineering-agent-to-the-next-level-planning-optimization-and-15e3b6cebe73">Taking the BigQuery Data Engineering Agent to the Next Level: Planning, Optimization and…</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running Antigravity on a browser tab]]></title>
            <link>https://medium.com/google-cloud/running-antigravity-on-a-browser-tab-6298bb7e47c4?source=rss-910459c47504------2</link>
            <guid isPermaLink="false">https://medium.com/p/6298bb7e47c4</guid>
            <category><![CDATA[antigravity]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Lucía Subatin]]></dc:creator>
            <pubDate>Wed, 03 Dec 2025 21:06:51 GMT</pubDate>
            <atom:updated>2026-02-13T16:53:58.033Z</atom:updated>
            <content:encoded><![CDATA[<p>My goal: running <a href="https://antigravity.google/">Antigravity</a> when I can;t install software locally. I’m using a <a href="https://cloud.google.com/workstations?utm_campaign=CDR_0x6cb6c9c7_default_b465812490&amp;utm_medium=external&amp;utm_source=blog">Cloud Workstation</a> to run Antigravity remotely and going about my development flow in the most seamless way possible. Wondering why I can’t run Antigravity on my local computer? It’s a long story of bad decisions I’ll spare you with. This will use the browser to display Antigravity and Chrome, so it will work on a Linux, Mac or Windows local computer. I’m hoping it helps others trying to do the same thing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wMYQzPE9sVPq-RGBaLNbHw.gif" /></figure><p>Cloud Workstations, fully-managed development environments on Google Cloud, are great for development. I’ve been using them with VSCode as an alternative to Cloud Shell, which quickly runs out of storage for me. I wanted to reuse the default machine image and use Antigravity instead of VSCode. The challenge: Antigravity doesn’t have a web version, so for this little experiment I’m using VNC and noVNC to stream the GUI into a browser.</p><p>Here are the steps I followed and the custom image. I should probably remind folks that Antigravity is in preview for individual users.</p><h3>Create a Custom Container Image</h3><p>First step is easy, clone this repo: <a href="https://github.com/GoogleCloudPlatform/cloud-workstations-custom-image-examples/tree/main/examples/images/gnome/noVnc">https://github.com/GoogleCloudPlatform/cloud-workstations-custom-image-examples</a> and navigate to the sample that installs noVNC:</p><pre>git clone https://github.com/GoogleCloudPlatform/cloud-workstations-custom-image-examples.git<br>cd cloud-workstations-custom-image-examples/examples/images/gnome/noVnc/</pre><p>We’ll use the /assets folder from here. You can delete the cloudbuild.yaml and README files if you wish.</p><p>Replace the Dockerfile with the following:</p><pre># syntax=docker/dockerfile:1<br>FROM us-central1-docker.pkg.dev/cloud-workstations-images/predefined/base as novnc-builder<br><br>ARG NOVNC_BRANCH=v1.5.0<br>ARG WEBSOCKIFY_BRANCH=v0.12.0<br><br>WORKDIR /out<br><br>RUN git clone --quiet --depth 1 --branch $NOVNC_BRANCH https://github.com/novnc/noVNC.git &amp;&amp; \<br>  cd noVNC/utils &amp;&amp; \<br>  git clone  --quiet --depth 1 --branch $WEBSOCKIFY_BRANCH https://github.com/novnc/websockify.git<br><br>#######################################################<br># End NoVNC Builder Container<br>#######################################################<br><br># Main container build<br>FROM us-central1-docker.pkg.dev/cloud-workstations-images/predefined/base<br><br># Use ARG to avoid apt-get warnings<br>ARG DEBIAN_FRONTEND=noninteractive<br><br># Install and configure systemd.<br>RUN apt-get update &amp;&amp; apt-get install -y \<br>  systemd &amp;&amp; \<br>  apt-get clean &amp;&amp; \<br>  rm -rf /var/lib/apt/lists/* &amp;&amp;\<br>  ln -s /dev/null /etc/systemd/system/apache2.service &amp;&amp; \<br>  ln -s /dev/null /etc/systemd/system/getty@tty1.service &amp;&amp; \<br>  ln -s /dev/null /etc/systemd/system/ldconfig.service &amp;&amp; \<br>  /sbin/ldconfig -Xv &amp;&amp; \<br>  ln -s /dev/null /etc/systemd/system/systemd-modules-load.service &amp;&amp; \<br>  ln -s /dev/null /etc/systemd/system/ssh.socket &amp;&amp; \<br>  ln -s /dev/null /etc/systemd/system/ssh.service &amp;&amp; \<br>  echo &quot;d /run/sshd 0755 root root&quot; &gt; /usr/lib/tmpfiles.d/sshd.conf &amp;&amp; \<br>  echo -e &quot;x /run/docker.socket - - - - -\nx /var/run/docker.socket - - - - -&quot; &gt; /usr/lib/tmpfiles.d/docker.conf<br><br># Install GNOME<br>RUN apt-get update &amp;&amp; apt-get install -y \<br>    gnome-software \<br>    gnome-software-common \<br>    gnome-software-plugin-snap \<br>    libappstream-glib8 \<br>    libgd3 \<br>    colord \<br>    gnome-control-center \<br>    gvfs-backends \<br>    hplip \<br>    libgphoto2-6 \<br>    libsane1 \<br>    sane-utils \<br>    ubuntu-desktop-minimal &amp;&amp; \<br>  apt-get remove -y gnome-initial-setup &amp;&amp; \<br>  apt-get remove -y --purge cloud-init &amp;&amp; \<br>  apt-get clean &amp;&amp; \<br>  rm -rf /var/lib/apt/lists/* &amp;&amp; \<br>  chmod -x /usr/lib/ubuntu-release-upgrader/check-new-release-gtk<br><br># Install Antigravity<br>RUN mkdir -p /etc/apt/keyrings &amp;&amp; \<br>    curl -fsSL https://us-central1-apt.pkg.dev/doc/repo-signing-key.gpg | gpg --dearmor -o /etc/apt/keyrings/antigravity-repo-key.gpg &amp;&amp; \<br>    echo &quot;deb [signed-by=/etc/apt/keyrings/antigravity-repo-key.gpg] https://us-central1-apt.pkg.dev/projects/antigravity-auto-updater-dev/ antigravity-debian main&quot; | tee /etc/apt/sources.list.d/antigravity.list &gt; /dev/null &amp;&amp; \<br>    apt-get update &amp;&amp; \<br>    apt-get install -y antigravity<br><br># Install Google Chrome<br>RUN wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | gpg --dearmor -o /usr/share/keyrings/google-chrome-keyring.gpg &amp;&amp; \<br>    echo &quot;deb [arch=amd64 signed-by=/usr/share/keyrings/google-chrome-keyring.gpg] http://dl.google.com/linux/chrome/deb/ stable main&quot; | tee /etc/apt/sources.list.d/google-chrome.list &gt; /dev/null &amp;&amp; \<br>    apt-get update &amp;&amp; \<br>    apt-get install -y google-chrome-stable<br><br># Divert the original Chrome executable to a new location and create a wrapper script.<br># This ensures the fix persists even when the Chrome package is updated.<br>RUN dpkg-divert --add --rename --divert /usr/bin/google-chrome-stable.real /usr/bin/google-chrome-stable &amp;&amp; \<br>    echo &#39;#!/bin/bash&#39; &gt; /usr/bin/google-chrome-stable &amp;&amp; \<br>    echo &#39;exec /usr/bin/google-chrome-stable.real --no-sandbox --no-zygote --disable-gpu --disable-dev-shm-usage &quot;$@&quot;&#39; &gt;&gt; /usr/bin/google-chrome-stable &amp;&amp; \<br>    chmod +x /usr/bin/google-chrome-stable<br><br><br># Install TigerVNC and noVNC<br>COPY --from=novnc-builder /out/noVNC /opt/noVNC<br>RUN apt-get update &amp;&amp; apt-get install -y \<br>    dbus-x11 \<br>    tigervnc-common \<br>    tigervnc-scraping-server \<br>    tigervnc-standalone-server \<br>    tigervnc-xorg-extension \<br>    python3-numpy  &amp;&amp; \<br>  apt-get clean &amp;&amp; \<br>  rm -rf /var/lib/apt/lists/*<br><br># Merge assets into the container.<br>COPY assets/opt /opt<br>COPY assets/. /<br><br># Run TigerVNC and noVNC as services.<br>RUN ln -s /etc/systemd/system/tigervnc.service /etc/systemd/system/multi-user.target.wants/ &amp;&amp; \<br>  ln -s /etc/systemd/system/novnc.service /etc/systemd/system/multi-user.target.wants/ &amp;&amp; \<br>  systemctl enable tigervnc &amp;&amp; \<br>  systemctl enable novnc<br><br># This is implicit when extending workstations predefined images, however we are<br># including it in the sample to explicitly call-out we are overriding the<br># default entrypoint when merging assets.<br>ENTRYPOINT [&quot;/google/scripts/entrypoint.sh&quot;]</pre><p>What’s happening here:</p><ol><li>We are using the base codeoss image because it already has a lot of valuable tools installed and optimized</li><li>We are reusing the original Dockerfile installation for systemd and gnome, so we have a UI to interact with</li><li>Updating the local package index and downloading, installing Antigravity and Chrome.</li><li>We have a little hack for Chrome: We need some flags for it not to crash or show the “Aw, Snap!” error due to limitations of the container environment. So we are passing those flags when executing Chrome. We need to override the calls from Antigravity so it uses those flags when calling Chrome AND we need to make sure that if we manually update Chrome, our little hack remains.</li><li>Then we install TigerVNC to stream our UI and noVNC so we don’t need a client and can use the browser instead.</li></ol><p>Tip💡: If you are using the Cloud Shell for this and have Gemini Code Assist enabled, you can highlight specific parts of the file and ask it to explain it.</p><h3>Build and Push the image</h3><p>Create a repository for these images. In all these commands, you’ll need to replace teh &lt;&lt;REGION&gt;&gt; and &lt;&lt;PROJECT_ID&gt;&gt; with your region of choice and your project ID.</p><pre>gcloud artifacts repositories create cloud-workstations-images \<br> - repository-format=docker \<br> - location=REGION \<br> - project=PROJECT_ID</pre><p>Configure Docker to authenticate to the Artifact Registry:</p><pre>gcloud auth configure-docker &lt;&lt;REGION&gt;&gt;-docker.pkg.dev</pre><p>Build, tag and push the image with a tag of your choice:</p><pre>docker build -t &lt;&lt;REGION&gt;&gt;-docker.pkg.dev/&lt;&lt;PROJECT_ID&gt;&gt;/cloud-workstations-images/antigravity-workstation:latest .<br>docker tag antigravity-workstation \<br>&lt;&lt;REGION&gt;&gt;-docker.pkg.dev/&lt;&lt;PROJECT_ID&gt;&gt;/cloud-workstations-images/antigravity-workstation:latest<br>docker push &lt;&lt;REGION&gt;&gt;-docker.pkg.dev/&lt;&lt;PROJECT_ID&gt;&gt;/cloud-workstations-images/antigravity-workstation:latest</pre><h3>Create a configuration and a cluster for a Cloud Workstation</h3><p>This is all documented here. You start with <a href="https://console.cloud.google.com/workstations/overview?utm_campaign=CDR_0x6cb6c9c7_default_b465812490&amp;utm_medium=external&amp;utm_source=web">creating a configuration</a>. This will have you create a cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qQXkf3h26F6MaqamyBKD0Q.png" /></figure><p>When choosing the cluster, make sure the region is close to where your local computer will be located. The networking config will depend on your project and your needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zKZ3S_W-GOSpL6E_qDEWwQ.png" /></figure><p>As for hardware, I found that E2 machine types were enough. I chose an e2-standard-16 and haven’t had to use a boost configuration so far, but boosting is always an option.</p><p>The most important part is choosing the image you added in the previous step:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sTyI9ykrD4bD0HV41vkUoA.png" /></figure><p>This will take a few minutes to update.</p><h3>Connect from your local computer</h3><p>You’ll need to <a href="https://console.cloud.google.com/workstations/createhttps://console.cloud.google.com/workstations/overview?utm_campaign=CDR_0x6cb6c9c7_default_b465812490&amp;utm_medium=external&amp;utm_source=web">create a workstation first</a>. Choose the configuration you have just created.</p><p>Start and Launch the workstation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V7xotHtHT6bI8NqQeVKZlA.png" /></figure><p>The first time you launch the workstation, you should see this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5J3ty69LURwSC4vvpI17uQ.png" /></figure><p>This is expected as you need to set the password for VNC.</p><p>Get the connection command for your workstation from the UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ofTGkYTxE_9qQCV2w2izBg.png" /></figure><p>Paste that into a Cloud Shell. You should now be connected to the command line for your workstation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*xCuG7iLaECz1gK1VqFmQ0Q.png" /></figure><p>Once you are connected, run<em> vncpasswd</em>. You’ll be prompted to set a password. You are only allowed 8 characters.</p><p>Launch the workstation tab again and you should be prompted to enter the password. If you still see the error, try <em>sudo systemctl restart tigervnc.service</em> from the command line in the workstation.</p><p>Prosper:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UIhmNgYkQCN3EjP6_R2u5A.png" /></figure><p>Moving forward, every time you start and launch the workstation, you should be prompted for this password and there should be no need for you to go into the SSH connection first.</p><p>If you see the error “Unable to forward your request to a backend — Couldn’t connect to a server on port 80&quot; right after you start the workstation, give it a minute and refresh the browser tab as it may still be starting the services.</p><h3>Troubleshooting</h3><p>With versions moving so fast, I noticed that every once in a while something stops working. One of the most common failures I’ve seen so far is shared memory being too small and causing random crashes.</p><p>The best way to troubleshoot is to see logs first. From a command line, run the following:</p><pre>/usr/share/antigravity/antigravity --no-sandbox --disable-gpu --disable-dev-shm-usage</pre><p>This forces the app to use /tmp instead of the (usually very small) shared mempry space. It also disables the sandbox for Chrome and GPU usage. If you are not experiences crashes with this command, you could create an alias or replace the desktop shortcut. I’m continuing to use this workstation and may modify the instructions for this to be more out of the box if these flags continue to be useful.</p><h3>What’s next?</h3><p>If you’re taking this seriously, you’ll want to set up a pipeline to automatically update the images when either the base image or Antigravity have an update. There’s a <a href="https://docs.cloud.google.com/workstations/docs/tutorial-automate-container-image-rebuildhttps://console.cloud.google.com/workstations/createhttps://console.cloud.google.com/workstations/overview?utm_campaign=CDR_0x6cb6c9c7_default_b465812490&amp;utm_medium=external&amp;utm_source=web">great tutorial here</a>.</p><h3>You may have questions….</h3><ol><li><strong>Did I try X11 forwarding?</strong> Yes, but the lagging was (understandably) more than I wanted to endure. If you REALLY wanted to use X11 and exercise your patience, let me know and I can share that Dockerfile and SSH connection flag.</li><li><strong>Can I use just VNC?</strong> Yes, nothing stops you from using a VNC client. Well, maybe corporate policies but I can’t help with that.</li><li><strong>Is it worth the trouble?</strong> It absolutely is IMO. Antigravity is now my default dev environment. <a href="https://www.youtube.com/watch?v=-lcGUElhHXE">Here’s a video</a> of an initial exploration if you are curious why I like my new agentic development workflow so much.</li><li><strong>What’s with the hacks?</strong> I do not love using the no-sandbox option for Chrome either and I understand why the banner is there. I also don’t love the clipboard situation with noVNC but I got used to it pretty fast and prefer it to having a VNC client installed.</li><li><strong>Isn’t there another way?</strong> For my specific situation, between limitations of my local machine and corporate policies, the other way could have been a remote desktop into a VM. I would have bumped into other issues with remote desktop so I discarded that. If I had a magic wand, Antigravity would work like the Cloud Shell and allow me to seamlessly mount SDD compute disks so I can have as much storage as I want. What would you ask for if you had a magic wand?</li></ol><p>Let me know how that goes in the comments or let’s continue the conversation on social media!</p><p><a href="https://www.linkedin.com/in/lsubatin/">LinkedIn</a> —<a href="https://www.tiktok.com/@unicorn.on.the.cl"> TikTok </a>— <a href="https://www.youtube.com/@LuciaSubatin/">YouTube</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6298bb7e47c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/running-antigravity-on-a-browser-tab-6298bb7e47c4">Running Antigravity on a browser tab</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Run BigQuery AI Functions on file creation in a Storage Bucket]]></title>
            <link>https://medium.com/google-cloud/run-bigquery-ai-functions-on-file-creation-in-a-storage-bucket-6b6c948223af?source=rss-910459c47504------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b6c948223af</guid>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-cloud-storage]]></category>
            <dc:creator><![CDATA[Lucía Subatin]]></dc:creator>
            <pubDate>Wed, 17 Sep 2025 12:58:15 GMT</pubDate>
            <atom:updated>2025-09-17T15:40:40.778Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xNwPgEVwws_tdFr7kW2E4A.gif" /><figcaption>GCS files to Cloud Run to Gemini from BigQuery</figcaption></figure><p>I’ve been enriching my conversational analytics agents with multimedia files to give them context from strategic documents and analyst reviews. While the insights have been great, I noticed a consistent pattern from these agents: they kept summarizing investor calls and past OKRs almost every time. So I modified the flow to create and persist a summary into a table as soon as those recordings land in the GCS bucket. This saves us the cost of running the same function and prompt for the same file.</p><h3>BigQuery setup</h3><p>You will need a <a href="https://cloud.google.com/bigquery/docs/create-cloud-resource-connection?utm_campaign=CDR_0x6cb6c9c7_default_b445222485&amp;utm_medium=external&amp;utm_source=blog">connection of type CLOUD_RESOURCE</a> for BigQuery to access the files and call the AI function. You can create one called “unicorn-connection” as follows:</p><pre>bq mk --connection --location=US --project_id=&lt;&lt;your project id&gt;&gt; \<br> --connection_type=CLOUD_RESOURCE unicorn-connection</pre><p>These are the columns of my basic table:</p><ul><li>An <a href="https://cloud.google.com/bigquery/docs/objectref-columns?utm_campaign=CDR_0x6cb6c9c7_default_b445222485&amp;utm_medium=external&amp;utm_source=blog">ObjectRef</a> for the file</li><li>The AI generated summary</li><li>The timestamp when this was processed</li></ul><pre>CREATE TABLE IF NOT EXISTS `{PROJECT_ID}.{DATASET_ID}.{TABLE_ID}` (<br>  file STRUCT&lt;uri STRING, version STRING, authorizer STRING, details JSON&gt;,<br>  summary STRING,<br>  insertion_timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP()<br>)</pre><p>I later added some logic to assign the fiscal quarter as a column for my agent to easily find the summary, but that’s not relevant to this post. You can also add whatever you need, I believe in you!</p><h3>Cloud Run setup</h3><p>Here’s the Python code I used (minus fiscal quarter conversions and some more elaborate prompting).</p><pre># Copyright 2025 Google, LLC.<br>#<br># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);<br># you may not use this file except in compliance with the License.<br># You may obtain a copy of the License at<br>#<br>#    http://www.apache.org/licenses/LICENSE-2.0<br>#<br># Unless required by applicable law or agreed to in writing, software<br># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,<br># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.<br># See the License for the specific language governing permissions and<br># limitations under the License.<br><br>import base64<br>import os<br>from flask import Flask, request<br>from google.cloud.bigquery import Client<br><br>app = Flask(__name__)<br><br># GCP Project ID - set in Dockerfile or cloud run --set-env-vars<br>_PROJECT_ID = os.environ[&quot;PROJECT_ID&quot;]<br># BigQuery Dataset ID - set in Dockerfile or cloud run --set-env-vars<br>_DATASET_ID = os.environ[&quot;DATASET_ID&quot;]<br># BigQuery Table ID<br>_TABLE_ID = &quot;investor_call_summary&quot;<br># BigQuery Connection ID<br>_CONNECTION_ID = &quot;us.unicorn-connection&quot;<br># Region or Location<br>_REGION = &quot;US&quot;<br><br>@app.route(&quot;/&quot;, methods=[&quot;POST&quot;])<br>def index():<br>    &quot;&quot;&quot;Receive and parse Pub/Sub messages.&quot;&quot;&quot;<br>    envelope = request.get_json()<br>    if not envelope:<br>        msg = &quot;no Pub/Sub message received&quot;<br>        print(f&quot;error: {msg}&quot;)<br>        return f&quot;Bad Request: {msg}&quot;, 400<br><br>    if not isinstance(envelope, dict) or &quot;message&quot; not in envelope:<br>        msg = &quot;invalid Pub/Sub message format&quot;<br>        print(f&quot;error: {msg}&quot;)<br>        return f&quot;Bad Request: {msg}&quot;, 400<br><br>    pubsub_message = envelope[&quot;message&quot;]<br><br>    # Existing logic to handle different message formats<br>    data = &quot;Empty&quot;<br>    if isinstance(pubsub_message, dict) and &quot;data&quot; in pubsub_message:<br>        try:<br>            data = base64.b64decode(pubsub_message[&quot;data&quot;]).decode(&quot;utf-8&quot;).strip()<br>        except Exception as e:<br>            print(f&quot;Could not decode base64 data: {e}&quot;)<br><br>    print(f&quot;Got {data}! message: {pubsub_message}&quot;)<br><br>    # New logic to handle GCS notifications<br>    if isinstance(pubsub_message, dict) and &quot;attributes&quot; in pubsub_message:<br>        attributes = pubsub_message[&quot;attributes&quot;]<br>        if attributes.get(&quot;eventType&quot;) == &quot;OBJECT_FINALIZE&quot;:<br>            bucket_id = attributes.get(&quot;bucketId&quot;)<br>            object_id = attributes.get(&quot;objectId&quot;)<br><br>            if not bucket_id or not object_id:<br>                print(&quot;error: Missing bucketId or objectId in GCS notification&quot;)<br>                return &quot;Bad Request: Missing bucketId or objectId&quot;, 400<br><br>            gcs_uri = f&quot;gs://{bucket_id}/{object_id}&quot;<br>            print(f&quot;Processing file: {gcs_uri}&quot;)<br><br>            try:<br>                client = Client(project=_PROJECT_ID, location=_REGION)<br><br>                # Create the BigQuery table if it doesn&#39;t exist<br>                # you could comment this out once this script is part of a bigger pipeline<br>                create_table_query = f&quot;&quot;&quot;<br>                CREATE TABLE IF NOT EXISTS `{_PROJECT_ID}.{_DATASET_ID}.{_TABLE_ID}` (<br>                    file STRUCT&lt;uri STRING, version STRING, authorizer STRING, details JSON&gt;,<br>                    summary STRING,<br>                    insertion_timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP()<br>                )<br>                &quot;&quot;&quot;<br>                <br>                create_table_job = client.query(create_table_query)<br>                create_table_job.result()  # Wait for the job to complete<br>                print(f&quot;Table {_TABLE_ID} created or already exists.&quot;)<br><br>                # Insert data into BigQuery<br>                insert_query = f&quot;&quot;&quot;<br>                INSERT INTO `{_PROJECT_ID}.{_DATASET_ID}.{_TABLE_ID}` (file, summary)<br>                SELECT<br>                    OBJ.FETCH_METADATA(OBJ.MAKE_REF(&#39;{gcs_uri}&#39;, &#39;{_CONNECTION_ID}&#39;)) AS file,<br>                    (AI.GENERATE(<br>                        prompt =&gt; (&#39;Summarize the call with relevant callouts about what contributed to the financial statement from the file provided&#39;, OBJ.FETCH_METADATA(OBJ.MAKE_REF(&#39;{gcs_uri}&#39;, &#39;{_CONNECTION_ID}&#39;))),<br>                        connection_id =&gt; &#39;{_CONNECTION_ID}&#39;,<br>                        endpoint =&gt; &#39;gemini-2.0-flash&#39;,<br>                        output_schema =&gt; &#39;summary STRING&#39;<br>                    )).summary AS summary<br>                FROM (SELECT 1)<br>                &quot;&quot;&quot;<br>                insert_job = client.query(insert_query)<br>                insert_job.result() # Wait for the job to complete<br>                print(f&quot;Successfully inserted data for {gcs_uri} into {_TABLE_ID}.&quot;)<br><br>            except Exception as e:<br>                print(f&quot;Error processing BigQuery job: {e}&quot;)<br>                return &quot;Internal Server Error&quot;, 500<br><br>    return (&quot;&quot;, 204)</pre><p>Create the artifact repository (called <em>run-repo</em> in this example) if you don’t have one, create the build (ingeniously called <em>pubsub</em> here) and get that authenticated endpoint from the <em>deploy</em> command. Personally, I like YAML files for environment variables but currently setting them in the Dockerfile for simplicity.</p><pre>gcloud config set project &lt;&lt;your_project&gt;&gt;<br>gcloud config set run/region &lt;&lt;the region where you intend to run this&gt;&gt;<br>gcloud artifacts repositories create run-repo  \<br>--repository-format=docker  \<br>--location=&lt;&lt;the region&gt;&gt;<br>gcloud builds submit --tag &lt;&lt;region&gt;&gt;-docker.pkg.dev/&lt;&lt;your_project&gt;&gt;/run-repo/pubsub<br>gcloud run deploy new-gcs-file --image &lt;&lt;region&gt;&gt;-docker.pkg.dev/&lt;&lt;your_project&gt;&gt;/run-repo/pubsub  \<br> --no-allow-unauthenticated</pre><p>There are some more details around pre-requisites <a href="https://cloud.google.com/run/docs/tutorials/pubsub#run_pubsub_handler-python?utm_campaign=CDR_0x6cb6c9c7_default_b445222485&amp;utm_medium=external&amp;utm_source=blog">in this great tutorial</a>. Keep that endpoint handy:</p><figure><img alt="Command line showing gcloud run resutls, showing the live endpoint" src="https://cdn-images-1.medium.com/max/1024/1*xpCCwcO_Q_zFsegztSLWSw.png" /></figure><h3>Pub/Sub notification when a new file is created in GCS</h3><p>First, navigate to <a href="https://console.cloud.google.com/iam-admin?utm_campaign=CDR_0x6cb6c9c7_default_b445222485&amp;utm_medium=external&amp;utm_source=blog">IAM</a>, you’ll need to grant permissions to a couple of service accounts.</p><p>Get your Storage service account from the settings and grant it “Pub/Sub Publisher”.</p><figure><img alt="In Cloud Storage Bucket, Settings on the left hand side panel, copy the value under Cloud Storage Service Account" src="https://cdn-images-1.medium.com/max/866/1*DSJ6R183aD3HB1sSsbel3g.png" /></figure><p>Then, get the Pub/Sub service account and make sure they have BigQuery Data Editor and Pub/Sub Publisher.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*vZ6lyulx7GWpByCw-QJfjg.png" /></figure><p>While you are in IAM, grant <strong>Vertex AI User</strong> and <strong>Storage Object Viewer</strong> to the BigQuery connection user:</p><figure><img alt="External connections -&gt; Connection info: service account id" src="https://cdn-images-1.medium.com/max/1024/1*gAFs5mNSNlur4Knd33A7xA.png" /></figure><p>You can now create the <a href="https://cloud.google.com/storage/docs/pubsub-notifications">notification</a>. The following command will create a topic called “new-investor-call” only when a file is created (a renamed file or a file uploaded again with the overwrite option will trigger this event).</p><pre>gcloud storage buckets notifications create --topic=new-investor-call --event-types=OBJECT_FINALIZE --object-prefix=videos/ gs://&lt;&lt;your bucket&gt;&gt;</pre><p>I’m using the prefix <em>videos</em> as this is the structure of my bucket:</p><figure><img alt="Structure shows demo_cfo_assistant/videos" src="https://cdn-images-1.medium.com/max/1024/1*y3XgliWS06W3HWyFRH393Q.png" /></figure><p>You should see your topic in the <a href="https://console.cloud.google.com/cloudpubsub/?utm_campaign=CDR_0x6cb6c9c7_default_b445222485&amp;utm_medium=external&amp;utm_source=blog">Pub/Sub interface</a>.</p><h3>Pub/Sub subscription for Cloud Run to execute BigQuery AI</h3><p>I like the <a href="https://console.cloud.google.com/cloudpubsub/subscription/create?utm_campaign=CDR_0x6cb6c9c7_default_b445222485&amp;utm_medium=external&amp;utm_source=blog">graphical interface</a> for this because it will prompt you to grant permissions and configure a <a href="https://cloud.google.com/pubsub/docs/handling-failures#dead_letter_topic">Dead Letter</a> topic (which I recommend for this in case something goes wrong).</p><p>Create a new subscription. Use type “Push” and use the endpoint from your Cloud Run deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*sOUxCceV36KOf4OhxZjAcQ.png" /></figure><p>Keep “Enable Authentication” (I love how easy this is!) and set the service account that will be running the Cloud Run service (hopefully not the Compute one if this is a “real” environment).</p><p>Set the acknowledgement interval to a higher value to be on the safe side for these type of operations, which may take a bit longer.</p><figure><img alt="Acknowledgement deadline set to 600" src="https://cdn-images-1.medium.com/max/1024/1*nMbw-aKWBdj7JCRwQFHn9w.png" /></figure><p>Follow the wizard and enable a dead letter subscription of your choice. You could write this to BigQuery and compare results with the target table, or even use an Object table to look for videos that haven’t been processed.</p><h3>Profit</h3><p>Upload a file into your bucket and give it a few seconds (like 30). You should see the table start getting the ObjectRef and a beautiful summary.</p><figure><img alt="Animated gif showing two videos being uploaded to a bucket, onot subdirectory videos, and refreshing a bigquery table that gets two additional rows, one per uploaded video" src="https://cdn-images-1.medium.com/max/1024/1*DmpZFVJTFN_CfALmqV40hw.gif" /></figure><p>If you don’t, check the <a href="https://console.cloud.google.com/logs/query?utm_campaign=CDR_0x6cb6c9c7_default_b445222485&amp;utm_medium=external&amp;utm_source=blog">logs</a>. A starting point could be Pub/Sub logs with something like this query filters:</p><pre>resource.type=&quot;pubsub_subscription&quot;<br>resource.labels.project_id=&quot;your project id&quot;<br>resource.labels.subscription_id=&quot;projects/&lt;&lt;project id&gt;&gt;/subscriptions/investor-call-to-bq&quot;</pre><p>Or you could troubleshoot the program locally by using <em>curl</em> to post a fake message. I’m using <a href="https://pypi.org/project/uv/">uv</a> for my packages.</p><pre>export PROJECT_ID=&lt;&lt;your_project_id&gt;&gt;<br>export DATASET_ID=&lt;&lt;your_dataset_id&gt;&gt;<br>uv pip install -r requirements.txt<br>source .venv/bin/activate<br>gunicorn main:app</pre><p>And here’s a curl command to call the local service running from another tab in the Cloud Shell (the fictitious bucket here is called “unicorn”):</p><pre>curl -X POST \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &quot;{\&quot;deliveryAttempt\&quot;: null, \&quot;message\&quot;: {\&quot;attributes\&quot;: {\&quot;bucketId\&quot;: \&quot;unicorn\&quot;, \&quot;eventTime\&quot;: \&quot;2025-09-16T17:24:32.218778Z\&quot;, \&quot;eventType\&quot;: \&quot;OBJECT_FINALIZE\&quot;, \&quot;notificationConfig\&quot;: \&quot;projects/_/buckets/videos_gcloud/notificationConfigs/1\&quot;, \&quot;objectGeneration\&quot;: \&quot;1758043472114811\&quot;, \&quot;objectId\&quot;: \&quot;clip (1).mp4\&quot;, \&quot;overwroteGeneration\&quot;: \&quot;1758035004535874\&quot;, \&quot;payloadFormat\&quot;: \&quot;JSON_API_V1\&quot;}, \&quot;data\&quot;: \&quot;ewogICJraW5kIjogInN0b3JhZ2Ujb2JqZWN0IiwKICAiaWQiOiAidW5pY29ybi9jbGlwICgxKS5tcDQvMTc1ODA0MzQ3MjExNDgxMSIsCiAgInNlbGZMaW5rIjogImh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL3N0b3JhZ2UvdjEvYi91bmljb3JuL28vY2xpcCUyMCgxKS5tcDQiLAogICJuYW1lIjogImNsaXAgKDEpLm1wNCIsCiAgImJ1Y2tldCI6ICJ1bmljb3JuIiwKICAiZ2VuZXJhdGlvbiI6ICIxNzU4MDQzNDcyMTE0ODExIiwKICAibWV0YWdlbmVyYXRpb24iOiAiMSIsCiAgImNvbnRlbnRUeXBlIjogInZpZGVvL21wNCIsCiAgInRpbWVDcmVhdGVkIjogIjIwMjUtMDktMTZUMTc6MjQ6MzIuMjE4WiIsCiAgInVwZGF0ZWQiOiAiMjAyNS0wOS0xNlQxNzoyNDozMi4yMThaIiwKICAic3RvcmFnZUNsYXNzIjogIlNUQU5EQVJEIiwKICAidGltZVN0b3JhZ2VDbGFzc1VwZGF0ZWQiOiAiMjAyNS0wOS0xNlQxNzoyNDozMi4yMThaIiwKICAic2l6ZSI6ICIyMzE4Mjc2IiwKICAibWQ1SGFzaCI6ICI5MkJzNEVyQlpkNEdkYk5KMEhaYWJnPT0iLAogICJtZWRpYUxpbmsiOiAiaHR0cHM6Ly9zdG9yYWdlLmdvb2dsZWFwaXMuY29tL2Rvd25sb2FkL3N0b3JhZ2UvdjEvYi91bmljb3JuL28vY2xpcCUyMCgxKS5tcDQ/Z2VuZXJhdGlvbj0xNzU4MDQzNDcyMTE0ODExJmFsdD1tZWRpYSIsCiAgImNyYzMyYyI6ICJlSTRJMEE9PSIsCiAgImV0YWciOiAiQ1B2b25xWG0zWThERUFFPSIKfQo=\&quot;, \&quot;messageId\&quot;: \&quot;16291255144246736\&quot;, \&quot;message_id\&quot;: \&quot;16291255144246736\&quot;, \&quot;publishTime\&quot;: \&quot;2025-09-16T17:24:32.311Z\&quot;, \&quot;publish_time\&quot;: \&quot;2025-09-16T17:24:32.311Z\&quot;}, \&quot;subscription\&quot;: \&quot;projects/stabby-4u/subscriptions/new-file\&quot;}&quot; \<br>  http://127.0.0.1:8000</pre><p>Let me know if this was helpful, and let’s connect on social media!</p><p><a href="https://www.linkedin.com/in/lsubatin/">LinkedIn</a> — <a href="https://www.tiktok.com/@unicorn.on.the.cl">TikTok</a> — <a href="https://www.youtube.com/@LuciaSubatin">YouTube</a> — <a href="https://www.instagram.com/a.cloudy.unicorn/">Instagram</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b6c948223af" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/run-bigquery-ai-functions-on-file-creation-in-a-storage-bucket-6b6c948223af">Run BigQuery AI Functions on file creation in a Storage Bucket</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring the new BigQuery Data Engineering Agent]]></title>
            <link>https://medium.com/google-cloud/exploring-the-newbigquery-data-engineering-agent-caf07e78d292?source=rss-910459c47504------2</link>
            <guid isPermaLink="false">https://medium.com/p/caf07e78d292</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data-and-analytics]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Lucía Subatin]]></dc:creator>
            <pubDate>Wed, 23 Jul 2025 16:26:35 GMT</pubDate>
            <atom:updated>2025-10-29T18:44:31.336Z</atom:updated>
            <content:encoded><![CDATA[<p>I had the opportunity to kick the tires on the newly released <a href="https://console.cloud.google.com/bigquery/?utm_campaign=CDR_0x6cb6c9c7_default_b425337136&amp;utm_medium=external&amp;utm_source=web">BigQuery Data Engineering agent</a> and I’m not just telling you all about it — I’m sharing my prompts so you can try it yourself.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSqjGq275d0M%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSqjGq275d0M&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FSqjGq275d0M%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4224012410538ce73b4eb41c371705b2/href">https://medium.com/media/4224012410538ce73b4eb41c371705b2/href</a></iframe><p>The vision for this (currently experimental) feature is to accelerate the work for Data Engineers and Data Analysts. Do you love manually populating a CREATE TABLE statement from the field names in an endless CSV file? Me neither. The Data Engineering agent doesn’t mind and will do it for you.</p><p>So far, I have pushed the boundaries of some common use cases, such as:</p><ul><li>Initial load from BigLake tables</li><li>Data quality assertions</li><li>Data modelling (denormalized models, SCD, star schema, etc)</li><li>Metadata generation</li></ul><p>Just like with any other Generative AI agent, the best results are obtained with some good prompting practices. I’ll call these out with each example below. I’ll also show what happens when you give the agent some creative license.</p><h3>Get the sample data in</h3><p>If you want to follow along these sample prompts, you’ll need to get the sample dataset into your project. Good news: this is a very simple dataset.</p><p>The code will attempt to create a dataset called “cattitude” in region <em>us-central1</em> in whatever project you are signed into in <a href="https://console.cloud.google.com/bigquery?utm_campaign=CDR_0x6cb6c9c7_default_b432499955&amp;utm_medium=external&amp;utm_source=blog">BigQuery Studio</a>.</p><p>Paste <a href="https://gist.github.com/Lsubatin/edd240158ce4a9d786d2b66cfb1d0648">this code</a> into a new BigQuery SQL console and replace &lt;&lt;your_project&gt;&gt; with… you guessed it! Your project ID.</p><p>📝 A mandatory disclaimer: While this is a very small dataset, this entire experiment may result in some cost. How much will depend on what your project is like, if it’s in a trial, etc.</p><p>After executing that code snippet, you should have a brand new dataset with three tables: Account, User, Case.</p><figure><img alt="BigQuery Explorer shows a dataset called “cattitude” with three tables in it: Account, Case and User" src="https://cdn-images-1.medium.com/max/675/0*7-IysO4U4OQ65DaR" /></figure><h3>Agentic time! Create a BigQuery Pipeline</h3><p>There are different ways to interact with the BigQuery Data Engineering Agent, including a CLI I’ll write about in a separate post. For now, we’ll use the pipeline option in the BigQuery UI.</p><figure><img alt="In BigQuery Studio, under the name of your project and next to “Pipelines”, you will find the option to create a new Pipeline. The “Start Agent Chat” button will appear in the middle of the pipeline canvas after confirming which user will execute the operations in your pipeline." src="https://cdn-images-1.medium.com/max/1024/0*KWq-nsYr_STNwslg" /></figure><p>What is this pipeline, you ask? This feature allows you to create data pipelines including SQL and notebooks. The agent will create tasks and operations in the pipeline, which are technically contained in a Dataform repository.</p><h3>Modeling with Natural Language</h3><p>It’s time to get our first prompt in to see what the agent can do.</p><p>I’m starting with a non-prescriptive prompt, giving the agent just enough to figure out the task.</p><p><strong>Prompt:</strong></p><blockquote><em>Use tables cattitude.Case and cattitude.Account to create a view in dataset cattitude that shows the count of cases per Account.</em></blockquote><figure><img alt="Prompt is pasted into the agent chat box. The agent shows some activity, the screen shows the two source tables as nodes connecting into the new view created by the agent. The details of the view show a SELECT statement and JOIN clauses." src="https://cdn-images-1.medium.com/max/1024/0*TM5CHrLksiKYXYiI" /></figure><p>We can <strong>Apply</strong> the changes to accept them, and <strong>Run </strong>the pipeline to see the new view get created.</p><figure><img alt="In the BigQuery UI, from the open PIpeline: Clicking Run -&gt; Opening Executions tab and then opening the dataset “Cattitude” on the left panel in BigQuery showing the created tables and the data produced by executing the newly created view" src="https://cdn-images-1.medium.com/max/1024/0*AV4fIpesMPRjehFk" /></figure><h3>Configure Vertex AI federation</h3><p>You can also ask the agent to modify existing schemas. In this case, I will ask it to infer the region of an account based on the country. For this, I will use one of the AI.GENERATE functions in BigQuery.</p><p>This will be the most boring step in this blog post, and then we get back to the fun parts, I promise.</p><h4>Connect BigQuery to Vertex AI</h4><p>The AI.GENERATE function requires a connection to Vertex AI.</p><p>You can use the following command to create a connection — or scroll down for the steps using the UI.</p><blockquote>bq mk — connection — location=us-central1 — project_id=&lt;&lt;YOUR_PROJECT&gt;&gt; — connection_type=CLOUD_RESOURCE unicorn-connection</blockquote><p>Alternatively, you can create it with <strong>Add data -&gt; Search for “Vertex” in the search bar -&gt; Click <em>Vertex AI</em> -&gt; Choose <em>BigQuery Federation</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/0*4fqmDpbm-ocbz1i6" /></figure><p>Follow the instructions and make sure you use the same region as your dataset (<em>us-central1</em> unless you’ve changed it). Keep the option “Vertex AI remote models, remote functions…” in C<em>onnection Type</em>. For my example, I used connection ID <em>unicorn-connection</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*IDufPWKFGpjgx074" /></figure><h3>Configure permissions</h3><p>The easiest way to figure out which service account will need permissions is to actually use the connection, so paste the following query into a BigQuery SQL editor. Replace “unicorn-connection” your connection ID if you’ve changed it :</p><blockquote>SELECT BillingCountry,</blockquote><blockquote>AI.GENERATE(</blockquote><blockquote>(‘Determine the sales region of ‘, BillingCountry, ‘based on the continent. The regions are: North America, Latin America, EMEA and APAC. Output only the region’ ),</blockquote><blockquote>connection_id =&gt; ‘us-central1.unicorn-connection’,</blockquote><blockquote>endpoint =&gt; ‘gemini-2.0-flash’).result</blockquote><blockquote>FROM `cattitude.Account`</blockquote><blockquote>LIMIT 1</blockquote><p>You should get an error message with the name of the service account that needs permissions to Vertex AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q0BosZmuqqg6WEAw" /></figure><p>Go to <a href="https://console.cloud.google.com/iam-admin/iam?utm_campaign=CDR_0x6cb6c9c7_default_b432499955&amp;utm_medium=external&amp;utm_source=blog">IAM</a> and grant it <em>Vertex AI User</em> IAM permissions, wait a minute or so, and try the query again to make sure it works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4gevXMJg4tjJomWV" /></figure><h3>Enrich an existing table</h3><p>We are now back to the fun stuff. Ask the agent to add a column to one of the tables using the AI.GENERATE function. Note I am being very prescriptive about how I want the function to be used.</p><p><strong>Prompt:</strong></p><blockquote><em>Add a field to the Account table that shows the sales region based on the BillingCountry.</em></blockquote><blockquote><em>Use the AI.GENERATE function as follows:</em></blockquote><blockquote><em>AI.GENERATE(</em></blockquote><blockquote><em>(‘Determine the sales region of ‘, BillingCountry, ‘based on the continent. The regions are: North America, Latin America, EMEA and APAC. Output only the region’ ),</em></blockquote><blockquote><em>connection_id =&gt; ‘us-central1.unicorn-connection’,</em></blockquote><blockquote><em>endpoint =&gt; ‘gemini-2.0-flash’).result</em></blockquote><p>After a minute, you will see the definition of a view that includes all the fields from the table plus the result of our function.</p><figure><img alt="Agent confirms the changes. Rightside panel shows the code for a view called Account_with_region, containing a SELECT clause with a * and the AI.GENERATE function as defined in the prompt. There is now two buttons on the upper right hand side with “Dismiss” and “APPLY”" src="https://cdn-images-1.medium.com/max/1024/0*5ZwEXp97okud6nM5" /></figure><p>Your result may be slightly different from mine. You can always ask the agent to correct it if you don’t like it.</p><p>Click <strong>Apply</strong> if you are happy with the changes. Click <strong>Run</strong> to see the changes materialized in your dataset. Any errors will be available in the execution tab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/0*0AWreEaOsgrAJpkX" /></figure><p>📝 You can learn more about the AI.GENERATE function (and other options) in the <a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate?utm_campaign=CDR_0x6cb6c9c7_default_b432499955&amp;utm_medium=external&amp;utm_source=blog">documentation here</a>.</p><h3>Generate metadata for your tables</h3><p>We are going to put the agent’s sense of humor to the test. Get it to generate metadata with the following prompt:</p><p><strong>Prompt:</strong></p><blockquote><em>These tables represent complaints (Cases table) generated by cats (Accounts table). Generate metadata for all columns, views and tables in the dataset cattitude, but make it funny.</em></blockquote><figure><img alt="Agent replies with funny metadata it generates. For example: 
 * **Case**: “The official record of feline grievances. Mostly about the quality of sunbeams and the unacceptable emptiness of food bowls.”
 * **case_count_by_account**: “How many complaints can a cat lodge if a cat could lodge complaints? This table answers that question. #CaseCount #CatComplaints" src="https://cdn-images-1.medium.com/max/1024/0*HIjeqH5QuNe6fKYz" /></figure><p>While you are at it, you can ask it to generate descriptions for the fields in the existing tables. I will rely on the context the agent already has, so I do not need to tell it the context is about Cats. If you are curious,</p><p><strong>Prompt:</strong></p><blockquote><em>Add equally funny descriptions for all columns of the tables in the cattitude dataset.</em></blockquote><h3>Errors? Unexpected results?</h3><p>As you can imagine, I have been trying different options with the agent for a while. One of the beauties of Generative AI is that it will not always come up with the same result. In one of these attempts, I noticed the metadata was not getting impacted because the Dataform file was compiling but not syntactically doing what it was supposed to do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/0*BmNjUHaJyjSYGwfz" /></figure><p>This is where being explicit with the prompt will do the trick. You can see the agent will check for syntax errors during validation which is also a nice feature.</p><p><strong>Prompt:</strong></p><blockquote><em>Use ALTER TABLE and ALTER VIEW statements in operations to generate metadata for all tables and views in dataset cattitude. The Account table contains accounts owned by cats. who are registering complaints about their owners in the Cases tables. Generate funny metadata for all tables and views in the cattitude dataset: Account table, Account_with_region table, Case table, User table and case_count_by_account view</em></blockquote><p>This resulted in the right updates:</p><figure><img alt="Picture shows ALTER TABLE statements followed by funny descriptions of each field such as “ALTER TABLE ${ref(“Account”)} SET OPTIONS (description = “Our esteemed clientele, the cats. Don’t let their fluffy exteriors fool you, they’re all business.”)”" src="https://cdn-images-1.medium.com/max/1024/0*Xab6emsuPKb_3tjl" /></figure><h3>Add Data Quality Assertions</h3><p>Unlike the other prompts, I will let the agent get creative just to see what it comes up with.</p><p><strong>Prompt:</strong></p><blockquote><em>Add data quality rules to the tables</em></blockquote><p>These are good starts: unique IDs, unique email addresses. Sometimes it checks for validity of date or email formats. I normally recommend getting more prescriptive here and defaulting instructions. Which brings me to the next section.</p><figure><img alt="Clicking on a node called “User_id_not_null” whosing a SELECT statement with a “WHERE Id is not null” clause as the assertion task details" src="https://cdn-images-1.medium.com/max/1022/0*6gqJRbJT_cGeplTF" /></figure><h3>Check Dataform</h3><p>You may have noticed “Pipeline Instructions” under the chat box. Clicking this link will open a panel with an external link to <strong>manage Instructions. </strong>This new tab will in turn open a Dataform repository.</p><p>You will find an <strong>instructions.md</strong> file that will help you give context and default directives for your agent. An useful example to default here could be “When using files in GCS buckets, always create Biglake tables.”</p><p>You can also set certain variables by default in the workflow_settings.yaml. This is the door to a CI/CD pipeline and usual software development lifecycle tasks for your pipeline.</p><figure><img alt="In the Pipeline interfacer in BigQuery, in the chat Window, clicking “Pipeline Instructions” opens a dataform repository. Clicking a metadata file with a sqlx extension showing the alter table statements" src="https://cdn-images-1.medium.com/max/1024/0*5vdGoqIY5EUI8Ogr" /></figure><h3>Stay tuned!</h3><p>I hope you are excited to test this yourself. You can now try this yourself in <a href="https://console.cloud.google.com/bigquery/?utm_campaign=CDR_0x6cb6c9c7_default_b425337136&amp;utm_medium=external&amp;utm_source=web">BigQuery Studio</a>.</p><p>I will continue to play with the agent, including it into a path to production and publishing the results in my social media and further blog posts as I get a chance. Let me know what you’d like to see next!</p><p>Let’s connect on <a href="https://www.tiktok.com/@unicorn.on.the.cl">TikTok</a>, <a href="https://www.instagram.com/a.cloudy.unicorn/">Instagram</a> or <a href="https://www.youtube.com/@LuciaSubatin">YouTube</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=caf07e78d292" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/exploring-the-newbigquery-data-engineering-agent-caf07e78d292">Exploring the new BigQuery Data Engineering Agent</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SAP on Google Cloud: HANA HDI containers and CI/CD pipelines]]></title>
            <link>https://medium.com/google-cloud/sap-on-google-cloud-hana-hdi-containers-and-ci-cd-pipelines-60cedaddfbc8?source=rss-910459c47504------2</link>
            <guid isPermaLink="false">https://medium.com/p/60cedaddfbc8</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sap-hana]]></category>
            <category><![CDATA[sap]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Lucía Subatin]]></dc:creator>
            <pubDate>Mon, 13 Jul 2020 04:14:07 GMT</pubDate>
            <atom:updated>2020-09-08T13:48:19.516Z</atom:updated>
            <content:encoded><![CDATA[<h3>SAP on Google Cloud: HANA HDI containers and CI/CD pipelines pt .2</h3><p>We built an application with managed containers using Node.js and Golang for the frontend and backend modules and HDI containers on SAP HANA. We defined a CI/CD pipeline to keep them consistent.</p><figure><img alt="Front end, backend and HANA Access Layer micro-services connecting to SAP HANA and calling the Translate API" src="https://cdn-images-1.medium.com/max/638/1*0RM7onjgLWBbLzz1dC4DAQ.png" /></figure><p>In this blog post, we’ll explain what the CI/CD pipeline looks like and how it incorporated the concept of HDI containers. This follows <a href="https://medium.com/google-cloud/sap-on-google-cloud-when-sap-developers-join-the-cloud-pt-1-5b4e24bf24e8">part 1</a>.</p><h3>The application</h3><p>As you see in the boxes inside of boxes above, our application has three micro-services:</p><ul><li>A frontend, a web application written in Node.js</li><li>A backend application, written in Golang, that also talks to the translate API</li><li>A database access layer responsible for talking to SAP HANA using the Node.js client for SAP HANA</li></ul><figure><img alt="Google Cloud shell explorer with the code for the micro-services" src="https://cdn-images-1.medium.com/max/1024/0*el3pYCuWSr75fkZe" /></figure><p>You can create this app yourself, hopefully for free and <strong>without swiping a credit card</strong>. We published <a href="https://google.qwiklabs.com/catalog_lab/2583">this Qwiklab</a> that will spin up a HANA Express machine on Google Cloud for you while you complete the lab. The initial free Qwiklab credits should be enough to run this lab. I’d recommend you clone the app into your own GitHub so you can use it afterwards.</p><h3>The pipeline</h3><figure><img alt="CI/CD pipeline with Git, container registry, cloud build and Google Cloud Run" src="https://cdn-images-1.medium.com/max/1004/0*F-xhwSJPYDA_vD8l" /></figure><p>I think the pipeline is better understood when in action. So let’s say we are a group of developers working on the different microservices that comprise our application.</p><p>Here’s how the tooling overlords would govern our day (or how we govern them, we’ll see…)</p><h3>Shared Git Repository</h3><p>We chose Google Cloud Source as a private git repository. This repo and its master branch were “born” with the application.</p><figure><img alt="Google Cloud Source repositories showing git branches and a history that shows everyone struggles with yaml" src="https://cdn-images-1.medium.com/max/825/0*tIY5pgpPHI5DE-4n" /></figure><p>You know what else was born with the first deployment? An HDI container!</p><h3>Creating the HDI container</h3><p>Just like we have a branch that acts as the main branch, we’ll have an HDI container that will act as the reference to all and will be loaded with test data.</p><p>Here’s an example of how this first container was created using the <a href="https://github.com/SAP-samples/hana-developer-cli-tool-example">hana-cli</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*-9SGt2Ccpnxo0jYXiAvSJw.gif" /></figure><p><a href="https://asciinema.org/a/wVydJ8CmOv31oryMRbRs3KqeV">Here’s the asciinema </a>if you want to follow along/ copy and paste.</p><p>The default-env.json file now contains credentials to this container. The Node.js library hdi-deploy will look for a file with this name or the environment variable VCAP_SERVICES to connect to SAP HANA and deploy our new tables and other artifacts into the HDI container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z3NB8MofHnWk_0Njl_p4NQ.png" /></figure><p>We will use the contents in this file to create an environment variable called VCAP_SERVICES later.</p><figure><img alt="default-env.json file with connection details for the HDI container (user, password, host, port)" src="https://cdn-images-1.medium.com/max/1024/1*GvbLe6AUYqyH7MNfbt4XRw.png" /></figure><h3>Starting a change</h3><p>For the sake of simplicity, let’s imagine we need to add an additional column to the existin gtable in the HDI container.</p><p>I’ll make sure I pull all the changes from the main branch. This will bring the latest artifacts that have been deployed into the main HDI container too.</p><figure><img alt="git fetch and git pull commands" src="https://cdn-images-1.medium.com/max/1024/0*TtQDUIvuVncDlL-7" /></figure><p>Finally, I’ll create my own branch:</p><figure><img alt="git checkout -b unicorn_branch" src="https://cdn-images-1.medium.com/max/936/0*VuQxG7WmaAApVCQ6" /></figure><p>Just like I have created my own branch, I will create my own version of an HDI container using the deployment files that I just pulled from the main branch.</p><p>This time, I will append a marker (<em>_LS</em> in the example below) to the name of my container, so that it does not conflict with the master one (Web IDE does this automagically).</p><pre>hana-cli connect -s</pre><pre>hana-cli createContainer -c RUN_LS -e -s</pre><p>Now I have the <strong>new </strong>credentials for this container in a new .json file. I will go ahead and make a change to the table I pulled:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/0*Aeb8KKEVh_OaG38q" /></figure><p>Using the new<strong> </strong>credentials in the default-env.json file, the ones for my RUN_LS container, I will deploy this change. The default-env.json file should be in the /db folder, from where we run these commands:</p><pre>npm install<br>npm start</pre><p>This calls the hdi-deploy module so it can create and HDI container and create a schema with my tables.</p><figure><img alt="npm start and npm install" src="https://cdn-images-1.medium.com/max/1024/1*NN7GRXhZjNbzLOIyCjRMUQ.gif" /></figure><p>Since our credentials are not meant to leave our local environment, let’s make sure the .gitignore file includes these default.env*.json files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*mvpbXugSkbbq5qLk93PKpA.png" /></figure><p>And now let’s commit+push before each coffee break:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_WRWw95ttBVYcRvN" /></figure><p>We should now see the change in my Git branch in Google Cloud Source:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/0*MXefcXz03-ypIyWO" /></figure><h3>Merging a change</h3><p>We have finished making some changes and doing some quick tests on our own container. We have committed those changes (multiple times) into our own branch. Now it’s time to submit them into the main branch.</p><p>Just in case the other developers working on this are also making changes, I am also pulling the branch again.</p><figure><img alt="git checkout master, git pull, git merge unicorn_branch, git push" src="https://cdn-images-1.medium.com/max/496/1*nQd4W5zLxXQX2OtiOmQaNw.png" /></figure><p><a href="https://asciinema.org/a/xBwl5ryz8fi94XivKmXnAXcnT">Asciinema here</a>.</p><p>And after we have merged the changes, the main branch is now showing them… as expected:</p><figure><img alt="HANA table in main / master git branch with the change merged from local HDI container" src="https://cdn-images-1.medium.com/max/737/0*Ytrx507Lb2oum3d8" /></figure><h3>Automated deployment and testing</h3><p>One of the key ingredients of our “deploy often, fail fast” recipe is automation. When do we want it? As soon as we push into the main branch. Yes, like we just did.</p><p>We want to make sure everything that used to work still works but we need the changes deployed into the main container first.</p><p>We also want to make sure that our new tables and artifacts are tested when someone else deploys. This means that our new tables also need to be incorporated into the automated tests, so we’ll build those into the pipeline too.</p><p>We’ll need some SAP HANA specific tools: hana-cli, hdbsql and the HANA client for Node.js if we want to do some tests like the ones documented <a href="https://github.com/SAP-samples/hana-xsa-opensap-hana7/tree/hana2_sps04/srv/tests">here</a>.</p><p>We can pre-install all of these in a Docker container, and have that container do the deployment and the testing. We’ll cover this in a future blog post, but here is a sneak peek of what this looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5Pi2ZPokA0cmyrE6" /></figure><p>The tool we are using to coordinate this is Cloud Build. We are using a trigger to start the build and test process every time there is a push into the main branch:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/0*8XBY2Yi2IIdshMYh" /></figure><p>Using the docker container and injecting the VCAP_SERVICES as an environment variable, so that the Node.js deployer can do its magic with the main HDI container, RUN:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/0*q5ibz0kriGP722lx" /></figure><p>There is a better way to pass these credentials, using secrets. But we’ll keep that one for the upcoming blog posts.</p><p>For details on how to build the pipeline for the rest of the micro-services and some tweaks towards using this in a productive environment, <a href="https://medium.com/google-cloud/sap-on-google-cloud-creating-the-ci-cd-pipeline-pt-3-22aa6c64123">here is part 3</a>.</p><p><a href="https://twitter.com/LuciaBlick">Lucia Subatin</a> and <a href="https://www.linkedin.com/in/fatimasilv/">Fatima Silveira</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60cedaddfbc8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/sap-on-google-cloud-hana-hdi-containers-and-ci-cd-pipelines-60cedaddfbc8">SAP on Google Cloud: HANA HDI containers and CI/CD pipelines</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SAP on Google Cloud : When SAP Developers join the cloud (pt. 1)]]></title>
            <link>https://medium.com/google-cloud/sap-on-google-cloud-when-sap-developers-join-the-cloud-pt-1-5b4e24bf24e8?source=rss-910459c47504------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b4e24bf24e8</guid>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[sap]]></category>
            <category><![CDATA[hana]]></category>
            <dc:creator><![CDATA[Lucía Subatin]]></dc:creator>
            <pubDate>Thu, 02 Jul 2020 19:13:19 GMT</pubDate>
            <atom:updated>2020-07-25T21:57:35.934Z</atom:updated>
            <content:encoded><![CDATA[<h3>SAP on Google Cloud : When SAP Developers join the cloud (pt. 1)</h3><p>During the first SAP Online Track, <a href="https://www.linkedin.com/in/fatimasilv/">Fatima Silveira</a> and I had the privilege of sharing our newest experiments with extensions to SAP on <a href="https://www.youtube.com/watch?v=lvtd9fD4rSI">containers, SAP HANA, HDI containers and CI/CD pipelines</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*2fXXpAWv1tRjZiiwfzhKBg.png" /><figcaption>CI/CD pipeline for micro-services and HDI Containers</figcaption></figure><p>This blog post provides some preliminary context and is the first of a series showing what and how we did it, with notes on how to implement this in a real-life scenario.</p><h3>When SAP joins the cloud</h3><p>This adventure started when a team of seasoned cloud-native developers heard the SAP workload was joining the cloud. This team already had a set of established practices for cloud-native development. They knew SAP is queen in business processes and wanted to invite her to the party.</p><p>They knew they could bring a lot of innovation to the business, they just needed to get past the acronyms in the SAP world first.</p><h3>The first acronym — HDI</h3><p>Together with SAP’s take on Cloud Foundry, in the SAP HANA world, came <a href="https://help.sap.com/viewer/4505d0bdaf4948449b7f7379d24d0f0d/2.0.05/en-US/e28abca91a004683845805efc2bf967c.html"><strong>H</strong>ANA <strong>D</strong>eployment <strong>I</strong>nfrastructure containers</a>. Despite being presented together, HDI containers do not need the XS Advanced or SAP Cloud Platform layer to exist in a HANA database. <a href="https://people.sap.com/thomas.jung">Thomas Jung</a> made this very easy to understand with a practical approach in <a href="https://blogs.sap.com/2019/04/16/developing-with-hana-deployment-infrastructure-hdi-without-xsacf-or-web-ide/">this blog post</a>.</p><figure><img alt="SAP HANA Deployment Infrastructure (HDI) containers" src="https://cdn-images-1.medium.com/max/612/1*43Bdud8mNeys0HUnUFHs0A.png" /><figcaption>HDI Containers — Source: help.sap.com</figcaption></figure><p>HDI containers have been a source of heated disputes at SAP family dinners. Without understanding the benefits they bring, it’s hard to make the case to migrate an existing codebase written in a traditional schema in a sidecar HANA. You have your calculation views feeding your reporting tool and it works like a charm, then why bother?</p><p>Here is one reason why: HDI containers allow for branching and isolation. We’ll see this in practice in our pipeline.</p><h3>The second acronym — CI/CD</h3><p>This one was not coined by SAP but defining this acronym can quickly turn into a debate. As a refresher:</p><ul><li><strong>Continuous integration:</strong> There is a shared repo (e.g., GitHub) to which developers merge their code. This merging process triggers automated code checks and tests to make sure nothing is broken by the new changes. If the tests pass, the whole thing is bundled to be deployed.</li><li><strong>Continuous delivery</strong>: Act of manually triggering the deployment of artifacts. For example, the changes have been approved and can be pushed into production. This a standardized workflow not too different from what most of the traditional SAP ABAP devs do to move a transport request into a productive Netweaver installation. The ability to rollback a change is also relevant.</li><li><strong>Continuous deployment</strong>: Same as delivery, but the deployment into production is automated.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*s9Y2VpNYkMmHQ0OuivH4rQ.png" /><figcaption>Conceptual CI/CD pipeline</figcaption></figure><p>The main point here is that we want to make <strong>small, incremental changes</strong> to add functionality to our code and release it as often as possible. We will use a series of tools to check our code, test it and release it.</p><p>If you’d like to understand more, here are some introductory resources to bookmark:</p><ul><li><a href="https://www.martinfowler.com/articles/continuousIntegration.html">Continuous Integration</a> and <a href="https://www.martinfowler.com/bliki/ContinuousDelivery.html">Continuous Delivery</a> by Martin Fowler</li><li>Google’s <a href="https://cloud.google.com/docs/ci-cd/">CI/CD Developer Hub</a></li><li>Nice <a href="https://www.youtube.com/watch?v=41QvqGfbz9o">intro video</a> with code examples and tools</li><li>Another video about <a href="https://www.youtube.com/watch?v=rt287-94Pq4">CI/CD Across Multiple Environments</a></li></ul><h4>What about DevOps?</h4><p>Well, DevOps goes beyond CI/CD and includes concepts like Infrastructure as Code which deserve their own full chapter. The one principle I’d take from DevOps culture is automation and focusing on a faster feedback loop.</p><h3>Cloud Run and Knative— not an acronym, still important</h3><p>I have loved Cloud Run <a href="https://blogs.sap.com/2019/04/13/post-google-next-19-curiosity-playing-with-cloud-run-and-sap-hana/">since it was first announced</a> because it allows you to deploy managed containers that autoscale and serve HTTPS endpoints without knowing much about containers at all.</p><p>The magic behind this is <a href="https://knative.dev/">Knative</a>, an open source project started by Google and contributed to by SAP, IBM, Pivotal and others. Knative is about simplifying serving and eventing in an application deployed in Kubernetes.</p><p>Why is it magic? It converts Kubernetes containers into serverless applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Nrtxal_by-IDcONQ" /></figure><p>To understand what Knative does, think of what you would need to manually deal with if Knative wasn’t serving the application above:</p><ul><li>Start with defining services and deployments for Kubernetes</li><li>Routing and balancing of incoming traffic to new revisions of an application</li><li>Automatically increasing or decreasing the number of replicas based on incoming requests (i.e., auto-scaling, even to zero)</li><li>Keeping a stable endpoint the other apps or users can connect to (even after you deploy a new version of the container)</li></ul><p>Now that we have established some context, let’s roll up our sleeves and dive into details.</p><p>Part 2 in this <a href="https://medium.com/google-cloud/sap-on-google-cloud-hana-hdi-containers-and-ci-cd-pipelines-60cedaddfbc8">post</a> for details on how the SAP HANA HDI containers work in a CI/CD pipeline.</p><p><a href="https://twitter.com/LuciaBlick">Lucia Subatin</a> and <a href="https://www.linkedin.com/in/fatimasilv/">Fatima Silveira</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b4e24bf24e8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/sap-on-google-cloud-when-sap-developers-join-the-cloud-pt-1-5b4e24bf24e8">SAP on Google Cloud : When SAP Developers join the cloud (pt. 1)</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>