<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Damien Contreras on Medium]]></title>
        <description><![CDATA[Stories by Damien Contreras on Medium]]></description>
        <link>https://medium.com/@c-damien?source=rss-96b9605b01d9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*UY_dYSx2tEUwRPQkrJzYMw.jpeg</url>
            <title>Stories by Damien Contreras on Medium</title>
            <link>https://medium.com/@c-damien?source=rss-96b9605b01d9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:34:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@c-damien/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building living games with Google ADK & BiDi audio Streaming for GDC 2026]]></title>
            <link>https://medium.com/google-cloud/building-living-games-with-google-adk-bidi-audio-streaming-for-gdc-2026-c97ab91ffe5f?source=rss-96b9605b01d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/c97ab91ffe5f</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gaming]]></category>
            <category><![CDATA[google-adk]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Damien Contreras]]></dc:creator>
            <pubDate>Mon, 09 Mar 2026 07:21:04 GMT</pubDate>
            <atom:updated>2026-03-11T03:18:21.480Z</atom:updated>
            <content:encoded><![CDATA[<p>How we brought the maximum fun to GDC integrating Gemini Live Bi Directional audio streaming interaction with Godot grounded on Telemetry and analytics</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pF_tf55eXds26Pc8iDLxwA.png" /><figcaption>Game experience with audio integration (transcript on the left)</figcaption></figure><h3>Real-time &amp; context-aware agents</h3><p>In the dynamic world of gaming, keeping players engaged requires more than just a great initial experience — it demands a game that learns and evolves with them. How many times did I stop playing a game because I didn’t remember what I needed to do, where I was heading and which action I needed to take or even how to play the game. I’ll walk you through how we updated our game (literally) to leverage Google Cloud’s solutions and empower game developers to move beyond static gameplay and create “Living Games” that are dynamic, personalized, and built on a foundation of real-time data powered by agents. This was presented as the main Google Cloud Demo at the <a href="https://gdconf.com/">Game Developer 2026 (GDC)</a> in San Francisco.</p><h3>Architecture at Glance</h3><p>Building a real-time, voice-driven game using Godot 4 and the new Gemini Live API requires bridging native Game Engine audio with cloud-based AI streaming. For “Retro Tech Revolution,” we implemented a custom bidirectional (BIDI) WebSockets architecture using a Python FastAPI backend and Godot’s built-in audio servers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dKQNHfqxCcrxz-DYIvzdtw.png" /></figure><p>The architecture functions as a three-tier system with the following components:</p><ul><li><strong>Game Client (Godot 4.6)</strong>: Captures the player’s microphone, resamples the audio to match the API requirements, and plays back the Gemini generated voice data dynamically as well as receive transcripts of both.</li><li><strong>Game Server (Python/ FastAPI + Google ADK):</strong> Manages the WebSocket connection, proxies the audio between the client and GCP’s Vertex Platform, and maintains the Gemini Live interactive session context.</li></ul><p>The overall architecture has more components than the one presented so that we stream continously telemetry information as well as screenshots taken every 10s from the game to our project in Google Cloud Platform (GCP) as follow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s-rySWyHJu7mG7lYcWWMDA.png" /><figcaption>Overall architecture with the live agent and analytics backend and conversational agents</figcaption></figure><p>Telemetry is sent through the game backend to the GCP environment using pub/sub publishing to a topic with a BigQuery subscription that directly stream the JSON payload sent by the game to a BigQuery table, This is used by our Looker dashboard and agents specifically built and working on the game analytic side.</p><h3>Agentic experience (FastAPI + ADK)</h3><p>While managing raw WebSocket streams is crucial for live voice, the actual “smarts” of Retro Tech Revolution are orchestrated by the <strong>Google Agent Development Kit (ADK)</strong>. Using the ADK inside our FastAPI backend allowed us to easily turn Gemini into an active, environment-aware game companion capable of influencing the game in real-time.</p><p>Here is a look at how the agentic architecture is shaped, how we structured the instructions, and how Gemini actually plays alongside the gamer.</p><h4>Defining the Agent and its Persona</h4><p>We start by providing a very extensive instructions to the agent with the following info:</p><ul><li><strong>The Game backstory:</strong> A 1980s/90s retro data center swarming with killer CRTs, matrix dot printers and angry ATX towers.</li><li><strong>The Map Layout:</strong> Details of the 8 rooms, 2 corridors, and the specific locations of tools available to the Player</li><li><strong>The Game Mechanics:</strong> For instance, how the boss fights work, the three difficulty modes (0=Easy, 1=Medium, 2=Hard) or the player’s controls.</li></ul><p>This deep semantic context allows the model to respond naturally to voice queries without hallucinating game rules</p><h4>Real-Time Telemetry and “Seeing” the Game</h4><p>Our agent cannot see the Godot screen as we only stream audio data. To bridge this gap, the game client sends <strong>JSON telemetry packets</strong> over the same WebSocket connection every 5 seconds to our agent with the following info:</p><ul><li><strong>Player State</strong>: Health, hit count, current score, elapsed time (stopwatch), and acquired weapons.</li><li><strong>Nearby Entities</strong>: A list of enemies, items, and the boss, detailing their relative distance to the player and direction from the player (pulled directly from the in-game mini-map data).</li></ul><p>The backend ADK interleaves these JSON payload directly into the LiveRequestQueue as system or user messages. Gemini processes this silent telemetry in the background, knowing exactly how much health the player has and take decision like help the player if his health is critically low.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UfwwOK1DGGQs4Bp78LhLjw.png" /><figcaption>How the agent interacts with the player thanks to the context received from the game</figcaption></figure><h4>The BigQuery Toolset (Accessing Historical data)</h4><p>We also allow the agent to use the BigQuery MCP server through the <strong>BigQueryToolset</strong> that enables the agent to dynamically execute SQL, check the Schema of tables etc. The game client continuously stream game events like when a bullet is fired or health decreased through the game server that in turns push the telemetry data to a topic to pub/sub on Google Cloud Platform (GCP) side. Ultimately all that data is stored in a game_signals table in BigQuery as JSON objects (JSON type column).</p><p>If the player asks, <em>“How did I die last time?”</em>, the agent can autonomously generate a SQL query to investigate the player’s historical session data on the fly before speaking the answer back to them!</p><h4>Active Game Intervention via ADK Tools</h4><p>Gemini is not just an observer — it is a Game Master.</p><p>We provided the agent with two highly specific Python functions acting as tools:</p><p><strong>adjust_game_difficulty</strong>(session_id, difficulty_level, reason)</p><ul><li><strong>Instruction:</strong> “If you determine from the data that the player is struggling too much or blazing through too easily, YOU MUST CALL THIS TOOL…”</li><li><strong>Action:</strong> Changes the game’s global difficulty from 0 (EASY) up to 2 (HARD).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zq7Cu69AdNCnVy1xb8gTBA.png" /><figcaption>Example of how the agent control the difficulty level based on the player’s performance</figcaption></figure><p><strong>heal_player</strong>(session_id, heal_amount, reason)</p><ul><li><strong>Instruction:</strong> “If the player is dangerously low on health between 0 to 30 and you judge that he needs help or specifically asks for healing, YOU MUST CALL THIS TOOL…”</li><li><strong>Action:</strong> Grants the player between 10 to 50 HP dynamically.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KSnqhl7DicTFlvwqdWSAOA.png" /><figcaption>Interaction of how the agent can interacts and help the player when in danger by healing the player</figcaption></figure><p>When the LLM decides to trigger one of these tools, On ADK side, we intercepts the command execution in FastAPI. Our tool implementations then bypass the LLM entirely, grabbing the raw active WebSocket for that session_id and pushing down a predefined JSON payload (e.g., {&quot;difficulty_level&quot;: 2, &quot;reason&quot;: &quot;Player is too strong&quot;}).</p><p>On the Godot client, the standard _process loop sees this incoming text packet, intercepts the JSON, and emits global Godot Signals (SignalBus.gemini_difficulty_adjusted.emit()). The game engine instantly updates the enemy spawns or player health, creating a seamless experience where the AI literally &quot;hacks&quot; the game to save the player&#39;s life.</p><h3>Python Backend to handle the Audio stream (FastAPI + ADK)</h3><p>The backend acts as the bridge between the erratic game-engine ticks and the reliable streams required by Google Vertex AI.</p><ul><li><strong>FastAPI WebSocket Endpoint</strong>: Constantly listens for both text and binary events from the game client.</li><li><strong>Google ADK (Agent Development Kit)</strong>: We use google.adk.runners.Runner to instantiate an agent powered by gemini-live-2.5-flash-native-audio, explicitly setting the streaming mode to BIDI.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QiWsCRBtutujvc3xsAwLmg.png" /><figcaption>Integration between the game client and the LLM by bridging 2 websockets and sending telemetry inbound along the audio stream</figcaption></figure><h4>Key configuration &amp; some gotchas</h4><ul><li><strong>Streaming Audio Formats</strong>: We explicitly configure the ADK LiveRequestQueue to accept incoming PCM audio at 16000Hz (mime_type=&quot;audio/pcm;rate=16000&quot;).</li><li><strong>WebSocket Timeout Monkeypatching</strong>: Vertex AI can sometimes suffer from “cold boots” on the very first connection, taking upwards of 30–45 seconds. We had to monkeypatch the underlying Python websockets.asyncio.client.connect to increase open_timeout to 60 seconds to survive this delay without dropping the connection.</li><li><strong>Handling Intermittent Disconnects</strong>: During run_live loops, google.genai.errors.APIError: 1000 (normal closure) or ping timeouts (1011) are frequent. The backend is programmed to treat these as harmless or temporary, utilizing asyncio.sleep to cleanly retry the connection without crashing the game state.</li></ul><h3>Godot game client for handling capturing and playing Audio (GDScript)</h3><p>Godot 4.6 at the time of this writting doesn’t natively speak directly to GCP hosted models or has a GCP specific library. All heavy lifting for audio resampling and network chunking has to be done manually in GDScript inside the _process loop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ASiIbvYGDuaGA9lwm7pcw.png" /><figcaption>How audio captured by the mic is prepared and before being sent by the game client</figcaption></figure><p><strong>Capturing Audio &amp; The Larsen Effect Gotcha:</strong></p><ul><li>We initialize an AudioStreamMicrophone and assign it to a custom &quot;Record&quot; bus.</li><li><strong>The Gotcha</strong>: If you listen to a microphone while outputting audio to the speakers, you trigger a massive feedback loop (the Larsen effect). If you simply Mute the “Record” bus in Godot, the AudioEffectCapture (which we use to grab the raw PCM frames) stops receiving data entirely.</li><li><strong>The Fix</strong>: We route the “Record” bus to a dummy “MutedSink” bus. We mute the “MutedSink”, leaving the “Record” bus unmuted. This successfully kills the feedback loop while keeping the capture buffer full of data!</li></ul><p><strong>Sending Audio to Gemini (Resampling &amp; Encoding):</strong></p><ul><li>Gemini expects 16000Hz. Godot&#39;s internal hardware mix rate can vary wildly (often 44100Hz or 48000Hz).</li><li>We built a custom Linear Interpolation (Lerp) resampler in GDScript to downsample the AudioEffectCapture frames to 16000Hz on the fly.</li><li>We add a digital gain clamp (clamp(sample * 2.0, -1.0, 1.0)) because microphones are often too quiet for the AI to &quot;hear&quot;.</li><li><strong>Endianness matters</strong>: To avoid hardware architecture bugs, we manually encode the Resampled floats into 16-bit Signed Little-Endian Integer packets (S16LE) using bitwise operations before sending them to the WebSocket.</li><li>We chunk the outgoing packets to 2048 bytes (approx 64ms) to optimize binary WebSocket overhead.</li></ul><p><strong>Receiving Audio from Gemini (Playback):</strong></p><ul><li>Gemini responds via WebSocket binary packets containing 24000Hz Mono PCM audio.</li><li>We set up an AudioStreamGenerator with mix_rate = 24000 and an aggressively large buffer (buffer_length = 60.0) to avoid dropouts or truncation when the network stutters.</li><li>As binary packets arrive in _process, we decode the S16LE ints back into normalized floats (-1.0 to 1.0), convert them to Vector2 for stereo padding, and push them to the AudioStreamGeneratorPlayback.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c97ab91ffe5f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/building-living-games-with-google-adk-bidi-audio-streaming-for-gdc-2026-c97ab91ffe5f">Building living games with Google ADK &amp; BiDi audio Streaming for GDC 2026</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google ADK + LM Studio: The Power Couple of Local Agent Development]]></title>
            <link>https://medium.com/@c-damien/google-adk-lm-studio-the-power-couple-of-local-agent-development-2dd225bae36f?source=rss-96b9605b01d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dd225bae36f</guid>
            <category><![CDATA[agent-development-kit]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[mistral]]></category>
            <category><![CDATA[google-adk]]></category>
            <category><![CDATA[local-llm]]></category>
            <dc:creator><![CDATA[Damien Contreras]]></dc:creator>
            <pubDate>Thu, 15 Jan 2026 18:26:07 GMT</pubDate>
            <atom:updated>2026-01-15T19:30:46.531Z</atom:updated>
            <content:encoded><![CDATA[<p>Orchestrate local inference and use Google’s Agent Development Kit to create powerful, private and always available agents</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W92XgFfUHzbL-h786qSQaQ.png" /></figure><h3>Motivation</h3><p>Last week, I boarded a flight to Europe. I was looking forward to eating good food and meet again with familiar faces but I dreaded the 10 hours’ flight with limited connectivity. Even after landing, I knew reliable internet would be scarce — 5GB international data cap and spotty train WiFi.</p><p>I resolved to find a way to continue my ongoing Agent development with Google Agent Development Kit (ADK) but find a way to perform the inference locally with the constraint of using my Mac M4 and its 24GB onboard unified RAM to do that task.</p><p>This tutorial demonstrates how to pair <a href="https://lmstudio.ai/"><strong>LM Studio</strong></a> with <a href="https://google.github.io/adk-docs/"><strong>Google’s Agent Development Kit (ADK)</strong></a> to build a fully functional AI agent that doesn’t require a single bar of signal to think, reason, or execute. Beyond “work-anywhere” portability, this setup ensures <strong>complete privacy and zero subscription costs by running Open-source models (</strong>LLM up to ~14B with an overall of 70% on <a href="https://llm-stats.com/benchmarks/mmlu">MMLU</a> which means that while they are compressed <strong>they are still accurate enough) directly on your machine</strong>.</p><p>Two things you should know before we start:</p><ul><li>No you don’t need 24GB of memory and there is plenty of models that are quantized (understand optimized to be smaller) that can run on your current laptop even with 16GB and still deliver good accuracy.</li><li>No you don’t need dedicated Nvidia GPUs attached, while it helps greatly accelerate the speed of the inference, it can work with your onboard GPUs or even without</li></ul><h3>Architecture: how it works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vt7OOQde_ch-lQsO-L8jog.png" /></figure><h3>Preparing the engine (LM Studio)</h3><p><strong>What is LM Studio:</strong></p><p>To get your agent running at 35,000 feet, we need to bypass the default cloud endpoints. Here is exactly how to do it before going offline.</p><p>For that we will be using LM Studio which is a cross-platform desktop application (Windows, Mac, and Linux) that allows you to discover, download, and run Large Language Models (LLMs) entirely on your locally.</p><p>Head to <a href="https://lmstudio.ai/">https://lmstudio.ai/</a> and download the version relevant for your configuration.</p><p>Once installed, you can proceed to configure your local LLM in LM Studio:</p><figure><img alt="LMStudio UI" src="https://cdn-images-1.medium.com/max/1024/1*r2dclSJUd26hWLyr6hqJ0Q.png" /><figcaption>LM Studio UI</figcaption></figure><p><strong>Download &amp; boot the inference engine</strong>:</p><p>We first need to select a model and download it locally.</p><p>LM Studio offers 3 different user modes:</p><ul><li><strong>User mode:</strong> leverage LM Studio chat interface to interact with a model with a handsfree experience</li><li><strong>Power User mode: </strong>can adjust hardware settings and tweak the model parameters</li><li><strong>Developer: </strong>Advanced usage with the possibility to use LM Studio as a headless server which is what we want</li></ul><p>Select the <strong>Developer</strong> <strong>Mode</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*zvOIgu1vOiJhq1BI0daO9Q.png" /><figcaption>Developer mode in LM Studio</figcaption></figure><p>Notice that you now have a new menu available on the left hand side of the interface, select the last option called 🔍<strong>“Discover”, </strong>LM Studio supports models in the GGUF and MLX formats and give you access to multiple versions of the same model that can be used for different hardware configuration.</p><p>Selecting a model will depends on the following factors:</p><ul><li><strong>CPU vs GPUs</strong>: depending on your hardware configuration you could instead of using your CPU offload and benefit from better performance for model inference by leveraging GPU(s). LM Studio let’s you know if the model can be fully offloaded or not.</li><li><strong>Memory</strong>: While for CPU based the model would be stored in the RAM, in the case of GPUs, the model would need to be stored in the VRAM and therefore needs to be able to fit in the RAM or VRAM available on your machine. Mac machines have an unified architecture that shares the RAM between CPUs and GPUs. Usually if you have 16GB of unified Memory on a Mac with 16GB you can experiment with 12B+ models. LM Studio also let you know if the model is likely too large and therefore you should select only models that would work with your configuration.</li><li><strong>Quantization level: </strong>it involves<strong> </strong>reducing the precision of a model’s weights and activations. This significantly lowers the memory footprint and power consumption, though it comes with a slight trade-off accuracy compared to the full-sized versions. The quantization information is usually present in the name of the model (e.g: <a href="https://lmstudio.ai/models/mistralai/ministral-3-14b-reasoning"><strong>ministral-3-14b-reasoning</strong></a> is a ministral version 3 with 14B parameters with reasoning capabilities it ranks at <strong>79% on </strong><a href="https://llm-stats.com/models/ministral-3-14b-base-2512"><strong>MMLU</strong></a>)</li><li><strong>Task Suitability (Instruct vs. Coder):</strong> Models with a suffix <strong>-Instruct</strong> that are best suited for following directions and general purpose agents or <strong>-Coder</strong> which explicit tells you that this is a model for writing code</li><li><strong>Inputs / tool support / Reasoning</strong>: LM Studio gives you information through icons on how each model is best suited for certain tasks, like does it supports multimodality (images &amp; text), or is it suitable to be used as an agent with tools</li><li><strong>Context Window</strong>: different models have different length and can “remember” or use more information. this is expressed in number of tokens and can be manually capped in LM Studio</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cu7ogQ4ayZ--m_Vp5V_LBQ.png" /><figcaption>Example of Ministral details</figcaption></figure><p>After downloading the model, we need to load it into <strong>LM Studio</strong> just click on <strong>“Select a model to load”</strong> located at the top of LM Studio’s window and select the one you want to use, the model will be loading and a <strong>“Ready”</strong> message will be displayed</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pl1clQpg1cuPQG_dOqoWtA.png" /><figcaption>Model selection dropdown list</figcaption></figure><p><strong>Start the local server</strong>:</p><p>We need now to go to the second option on the left menu called <strong>&gt;_</strong> “<strong>Developer” </strong>to make our model accessible from an HTTP endpoint. Just</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*64yMZ5BMaNUdfNxJlxA_2A.png" /><figcaption>Start the HTTP Server</figcaption></figure><p>Click the switch button next to Status: Stopped to launch the server, if you need to adjust the port or enable CORS you can do so by clicking on the <strong>Server Settings</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XtS4DeMSP-KEFKetYiIHjg.png" /></figure><p>Before going on to the next section, note the following 2 information from LM Studio:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e7ZzkASlcIfRP8ffM_-jqQ.png" /><figcaption>LM Studio Server settings</figcaption></figure><ul><li>the URL from which the server is reachable, for instance in my case <strong>http://192.168.1.20:1234</strong></li><li>the name of our model: <strong>mistralai/ministral-3–14b-reasoning</strong></li></ul><h3>Configuring Google ADK &amp; LiteLLM</h3><p><strong>What is Google ADK:</strong></p><p>Google Agent Development Kit is a high level framework to create and deploy agents, by default Google ADK uses Gemini through AI Lab or GCP’s Vertex AI.</p><p><strong>What is LiteLLM:</strong></p><p><a href="https://docs.litellm.ai/"><strong>LiteLLM</strong> </a>standardize the way to integration with more than <strong>100+ models</strong> and enables ADK to interact with other models from providers like OpenAI, Anthropic or OSS models.</p><p><strong>Setting up ADK + LiteLLM</strong></p><p>We will first need to install both ADK and liteLLM, I assume that you already have a working python environment on your machine</p><pre>pip install google-adk litellm</pre><p>Now that we have the libraries installed, we can use ADK to create our agent. ADK provides an easy way to create a standard structure to manage our agent using a template. you just need to fire the following statement as follow, where <strong>my_agent</strong> is the name of your agent but it can be anything:</p><pre>adk create my_agent</pre><p>when asked, select <strong>2. Other models (fill later) </strong>as we will not be using Gemini in this case.</p><p>the following structure will be automatically be created</p><pre>my_agent/<br>    agent.py      # main agent code<br>    .env          # API keys or project IDs<br>    __init__.py</pre><p><strong>agent.py </strong>is the main python script and where we will add a few steps to use the local LLM we set during the first part of this tutorial. we need first to import the necessary adk and lite_llm libraries first.</p><pre>### ADK<br>from google.adk.agents.llm_agent import Agent<br>import google.adk.tools<br><br>### lightLLM<br>from google.adk.models.lite_llm import LiteLlm</pre><p>we then need add the following variables to tell LiteLLM how to access our local LLM:</p><ul><li>Declare <strong>“OPENAI_API_BASE”</strong> as an os variable and set it to our LM Studio server URI. e.g: <a href="http://localhost:1234/">http://localhost:1234</a> we add at the end /v1 as all the services exposed by LM Studio are under v1.</li><li>Declare <strong>“OPENAI_API_KEY”</strong> as an os variable but we leave it blank as there is no API Key expected by LM Studio</li></ul><pre>import os<br><br>os.environ[&quot;OPENAI_API_BASE&quot;] = &quot;http://localhost:1234/v1&quot; # IP address from your LM Studio instance {IP}:{PORT}/v1<br>os.environ[&quot;OPENAI_API_KEY&quot;] = &quot;not-needed&quot;  # LM Studio doesn&#39;t require a real key</pre><ul><li><strong>Change the model</strong> we use in the Agent part with our LiteLLM one, LiteLlm(model=”openai/{Name of the model in LM Studio}”)</li></ul><p>Here is an example of an agent that you can use to plan your vacations while being offline</p><pre>import os<br><br>from google.adk.agents.llm_agent import Agent<br>import google.adk.tools<br><br>### lightLLM<br>from google.adk.models.lite_llm import LiteLlm <br><br>os.environ[&quot;OPENAI_API_BASE&quot;] = &quot;http://localhost:1234/v1&quot; # IP address from your LM Studio instance {IP}:{PORT}/v1<br>os.environ[&quot;OPENAI_API_KEY&quot;] = &quot;not-needed&quot;  # LM Studio doesn&#39;t require a real key<br>####<br><br># Main part of our agent <br># the model is declared as: openai/{MODEL NAME IN LM STUDIO}<br>root_agent = Agent(<br>   model=LiteLlm(model=&quot;openai/mistralai/ministral-3-14b-reasoning&quot;), <br>   name=&quot;my_agent&quot;,<br>   description=(<br>       &quot;An agent that provides personalized travel itineraries based on user preferences.&quot;<br>   ),<br>   instruction=&quot;&quot;&quot;You are an expert travel consultant. Your goal is to design a comprehensive day-by-day vacation itinerary by gathering specific user constraints. <br>Follow these steps:<br>- Ask what they need help with and make sure it&#39;s related to a trip.<br>- Ask for their destination. Tell them they can give an address or an approximate location by cross street.<br>- Ask if they have a budget in mind<br>- Ask how much they are ok to walk, move by car within the day<br>- confirm where they are staying<br>- Confirm the number of days they will be on site<br>- Generate a full itinerary of places to visit that takes into accounts all the constraints given by them<br>&quot;&quot;&quot;<br>)</pre><p>Let’s run the web interface of ADK to interact with our agent</p><pre>adk web --port 8001</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JWjUP_T00NfaGES3liqdzQ.png" /></figure><p>You can now head to your favorite browser (Chrome, what else can it be 🤔) and use the provided <strong>url: </strong><a href="http://127.0.0.1:8001,"><strong>http://127.0.0.1:8001</strong></a><strong> </strong>select your agent name in the dropdown list (e.g: <strong>my_agent</strong>) and have a chat with your offline travel agent</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-p8mTnJK7g_WpatsYvM6dA.png" /></figure><p>you can also see on LM Studio side in the developer logs the API calls</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lV3FX-iu_GBTC5pMcp1jLQ.png" /></figure><h3><strong>Conclusion</strong></h3><p>And that’s how you run a state-of-the-art AI agent from a basement, a bunker, or the middle of the Sahara. If you ever find yourself stranded on a deserted island with nothing but a laptop and a solar charger, you won’t need to talk to a volleyball named Wilson. Instead, you’ll be debugging Python with your local Llama 🦙. Honestly? Wilson might have been better for your mental health, but the Llama🦙 is far better being a friend — at least this one can talk back</p><p><strong>References:</strong></p><p><a href="https://lmstudio.ai/">LM Studio Bionic - Your Agent for Work and Code</a></p><p><a href="https://google.github.io/adk-docs/">https://google.github.io/adk-docs/</a></p><p><a href="https://docs.litellm.ai/docs/tutorials/google_adk">https://docs.litellm.ai/docs/tutorials/google_adk</a></p><p><a href="https://google.github.io/adk-docs/tutorials/agent-team/#step-2-going-multi-model-with-litellm-optional">https://google.github.io/adk-docs/tutorials/agent-team/#step-2-going-multi-model-with-litellm-optional</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dd225bae36f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical Gesture detection with MediaPipe in your Browser]]></title>
            <link>https://medium.com/@c-damien/practical-gesture-detection-with-mediapipe-in-your-browser-283c7c1f09f0?source=rss-96b9605b01d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/283c7c1f09f0</guid>
            <category><![CDATA[mediapipe]]></category>
            <category><![CDATA[retro-gaming]]></category>
            <category><![CDATA[gesture-recognition]]></category>
            <category><![CDATA[ai-web]]></category>
            <dc:creator><![CDATA[Damien Contreras]]></dc:creator>
            <pubDate>Fri, 17 Oct 2025 07:39:32 GMT</pubDate>
            <atom:updated>2025-10-17T07:39:32.806Z</atom:updated>
            <content:encoded><![CDATA[<p>Gesture detection using Google’s MediaPipe in a retro game that runs directly on your Browser using #WebAI and created for the Web AI Summit 2025</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AfKmKdXzpX4N_b__tTGHMQ.png" /></figure><p>Check the Source code here: <a href="https://github.com/c-damien/neon-pad">https://github.com/c-damien/neon-pad</a></p><h3>What we are doing</h3><p>What if your body was the gamepad and you could interact with your browser with your fingers. This is basically what I invite you to explore through a playful game demo paying homage to famous retro games but with a modern flair to it as it runs in the comfort of your browser and you interact with the game through gestures.</p><h3>How to run the game</h3><p>The game is made to be served on your local laptop using flask as a lightweight web server. I recommend using Apache or NGINX for an actual deployment</p><p>To run it you will need to:</p><ol><li>Download the files from github on your local machine</li><li>Install python 3.x</li><li>Install flask: pip3 install flask</li><li>Run our backend: python3 -m backend</li><li>Connect with Chrome to127.0.0.1:5000, you should see the splashscreen and a request to give access to the camera</li></ol><p>Note that the game has sound effects and musics but they are part of the package but you can easily add your own using ogg vorbis audio files.</p><h3>How it works</h3><p>The system is built using <strong>Google’s MediaPipe</strong> open source framework. MediaPipe is a great way to integrate AI/ML directly in an application using prebuilt blocks like for instance do face detection, object detection etc… and supports a wide variety of integration from Android and IOS for mobile device to Python and even Web browsers.</p><p>In this example we will be using specifically its GestureRecognizer task, to perform real-time hand tracking and gesture detection from the user&#39;s webcam.</p><p>The implementation can be broken down into three main stages:</p><ul><li><strong>Setup</strong>: to call the model and initialize the polling of the video</li><li><strong>Continuous Detection</strong>: Interpreting the video frame</li><li><strong>Gesture Interpretation: </strong>define what we want to detect with standard gestures or custom ones (more details on that later)</li></ul><h4>1. Setup and Initialization (setupHandTracking function)</h4><p>This function is responsible for setting up the webcam and the MediaPipe gesture model.</p><ol><li><strong>Load MediaPipe Model:</strong> It first loads the necessary MediaPipe vision tasks and then creates a GestureRecognizer instance.</li><li><strong>Configure Recognizer:</strong> The recognizer is configured with specific options:</li></ol><p>modelAssetPath: It loads a pre-trained machine learning model (gesture_recognizer.task) from <a href="https://storage.googleapis.com/mediapipe-tasks/gesture_recognizer/gesture_recognizer.task">https://storage.googleapis.com/mediapipe-tasks/gesture_recognizer/gesture_recognizer.task</a> that can recognize both hand landmarks and a set of pre-defined gestures. Check this link for more info: <a href="https://ai.google.dev/edge/mediapipe/solutions/vision/gesture_recognizer">link</a>.</p><p>runningMode: &quot;VIDEO&quot;: It&#39;s set to process a continuous video stream.</p><p>numHands: 2: It&#39;s configured to track up to two hands simultaneously, allowing for two-player mode in neon Pad.</p><pre>gestureRecognizer = await GestureRecognizer.createFromOptions(vision, {<br>   baseOptions: { modelAssetPath: `https://storage.googleapis.com/mediapipe-tasks/gesture_recognizer/gesture_recognizer.task` },<br>   runningMode: &quot;VIDEO&quot;, numHands: 2, minHandDetectionConfidence: 0.5, minTrackingConfidence: 0.5,<br>});<br>const stream = await navigator.mediaDevices.getUserMedia({ video: true });<br>video.srcObject = stream;<br>video.onloadedmetadata = () =&gt; { videoReady = true; video.play(); setInterval(detectHands, 30); };</pre><ol><li><strong>Access Webcam:</strong> It uses navigator.mediaDevices.getUserMedia({ video: true }) to request access to the user&#39;s webcam in Chrome.</li><li><strong>Start Detection Loop:</strong> Once the webcam stream is active and loaded into a hidden &lt;video&gt; element, it starts a setInterval(detectHands, 30), which calls the detectHands function approximately 33 times per second (every 30ms 30/1000) to analyze the video feed.</li><li><strong>Fallback System:</strong> If the webcam access fails (e.g., the user denies permission), the catch block sets handTrackingActive = false and calls setupKeyboardControls() to enable a keyboard-based fallback.</li></ol><h4>2. Continuous Detection (detectHands function)</h4><p>This function is the core loop that runs 33 times per second.</p><ol><li><strong>Recognize Frame:</strong> It feeds the current video frame to the MediaPipe model using gestureRecognizer.recognizeForVideo(video, performance.now()).</li><li><strong>Get Results:</strong> The model returns a results object containing two key pieces of information:</li></ol><p>results.landmarks: An array of 21 3D coordinates for each detected hand, detailing the position of the wrist, palm, and all finger joints.</p><p>results.gestures: An array of recognized <em>named gestures</em> (like &#39;Closed_Fist&#39;) if the hand pose matches one in the pre-trained model.</p><p>3. <strong>Route Logic by Game State:</strong> The function behaves differently depending on the gameState:</p><p><strong>In Menus (</strong><strong>&#39;title&#39;, </strong><strong>&#39;menu&#39;, etc.):</strong> It uses the first detected hand for UI navigation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pwe5Wl9jQ8X9amWkDy2QuQ.png" /><figcaption>Selection is done by using a cyan dot representing what the user is pointing at</figcaption></figure><p><strong>In Game (</strong><strong>&#39;playing&#39;):</strong> It sorts the detected hands by their horizontal position on the screen to assign the right-most hand to Player 1 and the left-most hand to Player 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v_TDvdENm_t2xCfXSVm-zQ.png" /><figcaption>In game interaction where the users control the 2 paddles as well as special effects with gestures</figcaption></figure><h4>3. Gesture Interpretation and Actions</h4><p>This is where the raw data from MediaPipe is translated into specific game actions. The system uses a mix of landmark-position tracking, custom-calculated gestures, and pre-trained named gestures.</p><p>1. <strong>Paddle Movement</strong> (Landmark Position)</p><p>During the &#39;playing&#39; state, the system maps a hand landmark directly to the paddle&#39;s position.</p><p><strong>How:</strong> It takes the Y-coordinate of landmark 9 (the base of the middle finger) and scales it to the game canvas’s height.</p><pre>if (p1Hand) {<br>      player1.y = p1Hand.landmarks[9].y * gameCanvas.height - PADDLE_HEIGHT / 2;<br>      if (previousHandX[0] !== null) { lastHandX[0] = p1Hand.landmarks[9].x - previousHandX[0]; }<br>       previousHandX[0] = p1Hand.landmarks[9].x; currentGestures[0] = p1Hand.gestures.length &gt; 0 ? p1Hand.gestures[0].categoryName : null;<br>       handlePinch(p1Hand, 0);<br> } else { previousHandX[0] = null; lastHandX[0] = 0; }</pre><p>2. <strong>Pinch &amp; Throw</strong> (Custom-Calculated Gesture)</p><p>During the game you can catch the ball and make it sticks to your paddle by pinching your index and thumb together. The ball is accelerated and released when you stop pinching your fingers.</p><p>The <strong>“pinch”</strong> mechanic is a <em>custom</em> gesture; it’s not pre-defined by MediaPipe. It’s calculated manually in the handlePinch function.</p><p><strong>How (Catch the ball):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ORMa5nt6mURN_UjGBKkLgg.png" /></figure><pre>const thumbTip = hand.landmarks[4]; const indexTip = hand.landmarks[8];<br>const distance = Math.sqrt(Math.pow(thumbTip.x - indexTip.x, 2) + Math.pow(thumbTip.y - indexTip.y, 2) + Math.pow(thumbTip.z - indexTip.z, 2));</pre><ol><li>It calculates the 3D distance between the thumb tip (landmark 4) and the index fingertip (landmark 8).</li><li>If this distance is less than a PINCH_THRESHOLD (0.05), the system considers the user to be pinching.</li><li>If a pinch <em>starts</em> while the ball is near the player’s paddle, it sets player.isHoldingBall = true, effectively catching the ball.</li></ol><p><strong>How (Throw the ball again):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3azUzh298Rh3mkazVh4NCQ.png" /></figure><pre> if (isPinching[playerIndex] &amp;&amp; player.isHoldingBall) {<br>         player.isHoldingBall = false; balls[0].dx = (playerIndex === 0 ? 1 : -1) * INITIAL_BALL_SPEED * 2.0;<br>         balls[0].dy = (Math.random() - 0.5) * INITIAL_BALL_SPEED; balls[0].hasParticleTrail = true;<br> }<br> isPinching[playerIndex] = false;</pre><ol><li>When the system detects that the distance is <em>no longer</em> below the threshold (i.e. the user released the pinch), it checks if the player was holding the ball.</li><li>If they were, it sets player.isHoldingBall = false and launches the ball with a high-speed boost (INITIAL_BALL_SPEED * 2.0), creating the &quot;throw&quot; effect.</li></ol><p>3. <strong>Multiball Power-Up</strong> (Pre-Trained Gesture)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wmjzCWEsFKEGh3U6301Ejg.png" /></figure><pre>if (currentGestures[i] === &#39;Closed_Fist&#39; &amp;&amp; lastGesture[i] !== &#39;Closed_Fist&#39;) { triggerMultiball(i); }<br>lastGesture[i] = currentGestures[i];</pre><p>Contrary to the privious one, this action uses one of MediaPipe’s built-in recognized gestures.</p><p><strong>How:</strong> The detectHands function checks if the results.gestures array contains the string &#39;Closed_Fist&#39;.</p><p><strong>Trigger:</strong> To prevent it from firing continuously, it checks if the <em>previous</em> gesture was <em>not</em> a fist (lastGesture[i] !== &#39;Closed_Fist&#39;). This ensures the triggerMultiball() function is called only on the first frame the fist is made.</p><p><strong>Action:</strong> The triggerMultiball function adds two new balls to the game, but only if the player is currently losing.</p><h3>and Gemini</h3><p>During the game we use the grey snake of WebAI letters as our live witty sport commentator powered by <strong>gemini-2.5-flash-preview-09–2025.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*ou-cr9tXFuvorU0FZgv7Sw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s6GW_ggL38I5zD9Fy0GD5Q.png" /></figure><p><strong>How:</strong></p><ol><li><strong>Event Trigger:</strong> When a specific event happens in the game (like a player scoring, hitting a “super shot,” or hitting the serpent), the code calls the getSerpentCommentary function.</li><li><strong>API Call:</strong> This function sends the event trigger (e.g., “Player 1 scores!”) along with a set of instructions ton contextualize (“You are a witty, sarcastic sports commentator… Keep comments very short”) to the <strong>Gemini 2.5 Flash model</strong> via an API call. To do the call we provide an API Key that you can generate from <a href="https://aistudio.google.com/">https://aistudio.google.com</a>.</li><li><strong>Display:</strong> The witty text response from Gemini is then received and drawn onto the game canvas, appearing as live commentary for a few seconds</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=283c7c1f09f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Next25 -Retro Tech Revolution]]></title>
            <link>https://medium.com/google-cloud/next25-retro-tech-revolution-04d260746cf3?source=rss-96b9605b01d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/04d260746cf3</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[realtime]]></category>
            <category><![CDATA[google-adk]]></category>
            <category><![CDATA[google-gemini-ai]]></category>
            <category><![CDATA[google-imagen]]></category>
            <dc:creator><![CDATA[Damien Contreras]]></dc:creator>
            <pubDate>Wed, 09 Apr 2025 06:58:40 GMT</pubDate>
            <atom:updated>2025-10-08T04:09:00.081Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<h3>How I powered Retro Tech Revolution with genAI &amp; Agents</h3><p>Level up user experiences with real-time data and AI Insights</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FbvcpYymQ5iI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DbvcpYymQ5iI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FbvcpYymQ5iI%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/426cefb5d1b44edc1693a531aab6a282/href">https://medium.com/media/426cefb5d1b44edc1693a531aab6a282/href</a></iframe><p><strong>The opinions stated here are my own</strong>, not necessarily those of my employer.</p><p>Source code can be found here: <a href="https://github.com/GoogleCloudDevRel/next25-retro-tech-revolution">https://github.com/GoogleCloudDevRel/next25-retro-tech-revolution</a></p><h3>Background</h3><p>The following demo and the related source code was first prepared and built for Google Cloud Next 25 held in Las Vegas. It was subsequently used in the following Google and Google Cloud events:</p><ul><li>Google Cloud Next 25 — Las Vegas</li><li>Build with Gemini: Developers Summit — Warsaw</li><li>Google I/O Connect— Berlin</li><li>Google I/O Connect — India</li><li>Google Cloud Next — Tokyo</li></ul><h3>Motivation</h3><p>AI is getting incredibly good at analyzing data and creating stunning images, audio, and video. But there’s a catch: most of the time, <em>we</em> have to do the work.</p><p>Whether you’re chatting with a bot to solve a problem, asking an AI to find outliers in your data, or prompting it to generate a 16-bit-style image of a dungeon floor, you have to spell out exactly what you want. The AI waits for you to make the first move.</p><p>But what if AI could enhance our experience <em>proactively</em>, without waiting for a direct command?</p><p>This demo explores exactly that. We’ll show you how to use real-time data to create smarter, more personalized user experiences automatically, making technology feel less like a tool you command and more like a partner that anticipates your needs.</p><h3>Overview</h3><p>Why a game for our demo? Because games are exciting, interactive, and — most importantly — they happen in real-time. We wanted to show how to make decisions instantly, not three days later after a batch job runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aUIQia-YCcVH9SgKgcvF3g.png" /></figure><p>The demo is split into two parts:</p><ul><li><strong>Live, In-Game AI:</strong> Watch as Gemini customizes the game experience on the fly. Imagine this in your own app: helping a customer find a product before they leave your site, or offering a tutorial to a user stuck on a form.</li><li><strong>Deep-Dive Analytics:</strong> Once the game is over, we analyze every granular event to get a crystal-clear picture of the user’s journey. What worked? Where did they get stuck? We then use AI agents to explore this data, surfacing powerful insights and answering questions about the session.</li></ul><h3>How do we collect data &amp; interact with GCP Services</h3><p>Let’s dive first on how we integrate with the different services provided by Google Cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U-3pMe3Zh_dxSyJdIqrNcA.png" /></figure><h4>1. Game Server Connection</h4><p>The process begins with the local game client, which connects directly to the <strong>Game Server</strong> using efficient RPC calls. This server is hosted on <strong>Google Kubernetes Engine (GKE)</strong> and leverages <a href="https://agones.dev/site/"><strong>Agones</strong></a>, an open-source platform built for multiplayer, to seamlessly manage the scaling of dedicated game server fleets. 🎮</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8VepG0B_9pe7Zo2DMGFuZA.png" /></figure><h4>2. Event &amp; Data Transmission</h4><p>When specific events occur in the game — like a player achieving a goal or a key moment happening — the game server captures this information. The data, which can include event details or even screenshots, is packaged into a <strong>JSON</strong> format. Any images are encoded in <strong>base64</strong> and then sent as a standard <strong>REST API call</strong> to the API Bridge.</p><h4>3. API Bridge</h4><p>The <strong>API Bridge</strong>, a backend service built with <strong>Flask</strong>, acts as the central hub. Using GCP’s Python library, it directs incoming data to the appropriate service:</p><ul><li><strong>Real-time analytics data</strong> is published as messages to <strong>Pub/Sub</strong>. From there, it can be streamed into services like <strong>BigQuery</strong> for analysis.</li><li><strong>Image files</strong> are efficiently uploaded directly to <strong>Google Cloud Storage</strong> for persistent storage and later use.</li><li>Integration with multiple LLMs available in GCP AI platform: Vertex AI for various purpose from generating the backstory of the game to generating images</li></ul><h4>4. AI-Powered Features</h4><p>The API Bridge also integrates an agent built using Google’s <a href="https://google.github.io/adk-docs/get-started/installation/"><strong>Agent Development Kit (ADK)</strong></a>. This agent communicates with <strong>Gemini</strong> <strong>2.5 flash</strong>.</p><p>Supporting the user with various AI integration allows for advanced in-game features, such as AI-driven NPC dialogue, dynamic content creation, or personalized player assistance based on in-game events. 🤖✨</p><h3>Backstory generation with Gemini &amp; Image4</h3><p>One of the first sequence of the game ask you for a couple of questions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_vhe8EDLyPjdPtvvq_lDAQ.png" /></figure><h4>1. Crafting the Perfect Prompt</h4><p>It all starts with you, the player. The system captures your in-game choices — Are you a strategist or a fighter? Do you prefer oranges or strawberries? — and combines them with the game’s core lore. This creates a unique “digital fingerprint” that becomes a rich prompt, ready to be sent to <strong>Gemini 2.5 Flash</strong>.</p><h4>2. Gemini Writes Your Story</h4><p>With lightning speed, <strong>Gemini 2.0 Flash lite </strong>takes this personalized prompt and writes a compelling backstory just for you. As seen in the diagram, because the player chose 🍊 “orange,” Gemini cleverly weaves this choice into the narrative, creating a villain named “The Orange Rot” who is “sour and toxic.” This small detail makes the world feel instantly reactive and personal to the player.</p><p>Flask expose the story generation through a python function called get_backstory_story that receive a prompt coming from the game, here is the prompt itself:</p><pre> var prompt = &quot;You are a video game story writer and you are tasked to write a very short backstory of the game to make the player want to play.\n&quot; <br> prompt += &quot;The game takes place in an old datacenter from the 80s or 90s swarming with old technologies like CRTs, old matrix dot printers or floppy disks.&quot; <br> prompt += &quot;The hero of the story is tasked to clean the datacenter from thos antiquities and find the boss, an old angry ATX Server that controls the old tech and hide in the datacenter.\n&quot; <br> prompt += &quot;To tailor the story to the player, he gave you the following indications on his personality through a Q&amp;A session.&quot; <br> <br> for i in SignalBus.trivia_result.size():<br>    prompt +=&quot;Question:&quot; + SignalBus.trivia_result[i][&#39;q&#39;] +&quot;, Answer:&quot;+ SignalBus.trivia_result[i][&#39;a&#39;] + &quot;.\n&quot; <br> prompt += &quot;Color your story with element from his answers and keep the story short, Write the story in the following language: &quot;+lang</pre><p>As you can see we use a loop to retrieve the different questions and the answers given by the player and pass that as additional context to get a more personalized story.</p><pre>BACKSTORY_GENERATION_MODEL = &quot;gemini-2.0-flash-lite-001&quot;<br><br>#ask Gemini to generate the backstory story itself<br>def generate_backstory_story(prompt):<br> client = genai.Client(http_options=HttpOptions(api_version=&quot;v1&quot;))<br> response = client.models.generate_content(<br>  model=BACKSTORY_GENERATION_MODEL,<br>  contents=prompt,<br> )<br> return response.text</pre><h4>3. Imagen Visualizes Your World</h4><p>A great story deserves stunning art. Simultaneously, your preferences are sent to <strong>Imagen 4</strong>, a fast image generation model. It creates a beautiful, 9:16 vertical image that visually represents the unique backstory Gemini just wrote. The result is a bespoke piece of art, like the ominous, glowing server room, that perfectly matches the tone and details of your personalized adventure.</p><p>The Image generation follows a similar process with a prompt coming from the game:</p><pre> var prompt_img = &quot;Illustrate the following story using a 16 bits retro style design with neon glows using google color palette:\n&quot;<br> prompt_img += &quot; You have been tasked to clean a datacenter that have been overtaken by old technologies from the 80s and 90s like matrix printers &amp; CRTs and fyling floppy disks.\n&quot;<br> prompt_img += &quot;Make sure to incorporate in your image elements that remind of a: &quot;+SignalBus.trivia_result[0][&#39;a&#39;]+&quot;\n&quot;<br> prompt_img += &quot; and an overall vibe: &quot;+SignalBus.trivia_result[1][&#39;a&#39;]+&quot;\n&quot; </pre><p>Followed by a call to Imagen4 from the API Bridge</p><pre>IMAGE_GENERATION_MODEL = &quot; imagen-4.0-fast-generate-preview-06-06&quot;<br><br>#keep the aspect ratio but reduce image to <br>def resize_image_maintain_aspect(image_bytes, max_width, max_height):<br>    # Create an image object from bytes<br>    image = PIL_Image.open(io.BytesIO(image_bytes))<br>    <br>    # Calculate aspect ratio<br>    original_width, original_height = image.size<br>    aspect_ratio = original_width / original_height<br>    <br>    # Determine new dimensions while maintaining aspect ratio<br>    if original_width &gt; original_height:<br>        # Width is the limiting factor<br>        new_width = max_width<br>        new_height = int(new_width / aspect_ratio)<br>    else:<br>        # Height is the limiting factor<br>        new_height = max_height<br>        new_width = int(new_height * aspect_ratio)<br>    <br>    # Resize the image<br>    resized_image = image.resize((new_width, new_height), PIL_Image.LANCZOS)<br>    <br>    # Convert to bytes<br>    output_buffer = io.BytesIO()<br>    resized_image.save(output_buffer, format=image.format or &#39;PNG&#39;)<br>    resized_bytes = output_buffer.getvalue()<br>    <br>    return resized_image, resized_bytes<br><br><br>#ask Imagen to generate the backstory image<br>def generate_backstory_image(prompt, session_id):<br> #prompt = &quot;&quot;&quot;<br> #16 bits version with tron like neon of a datacenter in the 80s<br> #&quot;&quot;&quot;<br> print(&quot;---Calling Imagen4---&quot;)<br><br> vertexai.init(project=PROJECT_ID, location=&quot;us-central1&quot;)<br> model = ImageGenerationModel.from_pretrained(IMAGE_GENERATION_MODEL)<br><br> attempt = 0<br> while attempt &lt; max_imagen_retries:<br><br>   images = model.generate_images(<br>    prompt=prompt,<br>    # Optional parameters<br>    number_of_images=1,<br>    language=&quot;en&quot;,<br>    # You can&#39;t use a seed value and watermark at the same time.<br>    # add_watermark=False,<br>    #seed=42,<br>    aspect_ratio=&quot;9:16&quot;,<br>    #output_mime_type=&quot;image/png&quot;,<br>    safety_filter_level=&quot;block_few&quot;,<br>    person_generation=&quot;dont_allow&quot;<br>   )<br>   print(&quot;Received imagen backstory&quot;)<br>   resized_image, resized_bytes = resize_image_maintain_aspect(images[0]._image_bytes, 600, 800)<br><br>   base64_bytes = base64.b64encode(resized_bytes)<br>   base64_encoded = base64_bytes.decode(&#39;utf-8&#39;)<br> <br>   print(f&quot;Base64 encoded image: {base64_encoded[:100]}...&quot;)<br> <br>   ## upload to GCS<br>   upload_backstory_to_gcs(base64_encoded, session_id)<br><br>   return base64_encoded</pre><p>Together, this AI-driven synergy creates a world that isn’t just presented to the player, but is actively shaped <em>by</em> them, making every playthrough a truly unique experience.</p><h3>Your Agentic Big Brother: How We’re Bringing Back the Magic of Old-School Gaming</h3><p>Remember crowding around a CRT TV, controller in hand, trying to conquer the treacherous dungeons of <em>The Legend of Zelda</em>? Remember your older sibling or a friend leaning over your shoulder, pointing at the screen and saying, “<em>Psst, try bombing that wall,</em>” revealing a secret passage you never would have found on your own?</p><p>That was the magic of old-school gaming. It wasn’t just about the challenge; it was about that shared experience, the timely tip from a friend who knew the game inside and out. We wanted to bring that feeling back. So, we built a digital “big brother” — an AI agent that watches you play, understands when you need a nudge, and helps you become a hero.</p><p>Here’s how we did it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*37GGhkS7wCykqU7ShSrlKw.png" /></figure><h4>1. The All-Seeing Friend</h4><p>First, our agent needs to see what you see. Just like your friend watching you play, our agent, built with <strong>Google’s ADK and powered by the multimodal magic of Gemini 2.5</strong>, takes in the whole picture. It doesn’t just get your health and score; it analyzes a <strong>screenshot of your game</strong> to understand your exact position, what’s happening on screen, and the context of the moment. It has the entire game manual memorized, so it knows every rule and every secret.</p><h4>2. The Perfect Timely Tip</h4><p>With this deep understanding, the agent delivers the perfect piece of advice, right when you need it. In our demo, it recognizes a player at the start of a level and offers a classic, gentle nudge reminiscent of the old man in the cave: <strong>“IT’S DANGEROUS TO GO ALONE! CHECK YOUR MINI-MAP…”</strong></p><p>This isn’t a generic, pre-written hint. The agent knows you’re a new player and haven’t struggled yet, so its tone is encouraging. If you were a seasoned veteran stuck in a late-game puzzle, the hint would be more cryptic and challenging, respecting your skill. It adapts its advice just like a real person would.</p><h4>3. The Unseen Hand of the Game Master</h4><p>This is where our AI big brother goes beyond just giving tips. It can actually <strong>change the game itself</strong>.</p><p>Are you blazing through the level, dispatching enemies with ease? The agent will subtly increase the difficulty, spawning tougher monsters or more complex puzzles to keep you engaged and challenged. Stuck on a boss and about to throw your controller in frustration? The agent can discreetly lower its health or slow its attacks, giving you the opening you need to claim victory. It acts as the ultimate Game Master, ensuring the experience is always perfectly balanced — never too hard, and never too easy.</p><p>By combining real-time observation with intelligent intervention, we’ve created a companion that brings the best of couch co-op into a single-player world, making every adventure feel personal, supportive, and endlessly fun.</p><h4>4. How is it implemented</h4><p>Every 10 seconds the game performs a call to the Agent in the API Bridge passing a whole raft of different information about the game status</p><pre>var body = JSON.stringify({<br>   &quot;image&quot;: &quot;data:image/png;base64,&quot; + base64_image,<br>   &quot;session_id&quot;: SignalBus.session_id,<br>   &quot;stopwatch&quot;: SignalBus.last_screenshot_timestamp,<br>   &quot;has_weapon1&quot;:!SignalBus.players[0].weapons[1][&#39;disabled&#39;],<br>   &quot;has_weapon2&quot;:!SignalBus.players[0].weapons[2][&#39;disabled&#39;],<br>   &quot;hit_count&quot;: SignalBus.players[0].hit_count,<br>   &quot;health&quot;: SignalBus.players[0].health,<br>   &quot;score&quot;: SignalBus.score,<br>   &quot;game_difficulty&quot;:SignalBus.game_difficulty,<br>   &quot;language&quot;: lang<br>  })</pre><p>That data is the used by the API bridge to build a prompt that we pass to the agent</p><pre>prompt_text = &quot;&quot;<br>  #user has 2 weapons<br>  if has_blaster and has_gauntlet:<br>   #You know that the boss is located on the top right side of the level.<br>   prompt_text += &quot;The player has both tools and is ready to go meet his destiny\n&quot;<br>   prompt_text += &quot;Give instructions to the player on how to get to the large room located on the top right where the final boss is located.\n&quot;<br>   prompt_text += &quot;explain that to defeat the final boss you should use the gauntlet 2000 to create a barrier and therefore avoid being hit by floppy disks.\n&quot;<br>   prompt_text += &quot;3.5 floppy disks are faster but inflict less damage, larger 5.25 are slower but inflict more damage&quot;<br>   prompt_text += &quot;moving around would give him enough time to use the blaster  cleaner 80 to shoot at the boss. The boss has 200 of health and can be identified by a blue square&quot;<br>  elif has_blaster:<br>   #user has the blaster only<br>   prompt_text += &quot;The player already has the blaster cleaner 80 but will need to protect himself more if he wants to defeat the final boss.\n&quot;<br>   prompt_text += &quot;Recommend to the player to go fetch the weapon on the very far right represented by a yellow square. \n&quot;<br>  elif has_gauntlet:<br>   #user has the gauntlet<br>   prompt_text += &quot;The player already has the gauntlet 2000 a power protective tool against various tech notably floppy disks.\n&quot;<br>   prompt_text += &quot;Recommend to the player to also retrieve a tool for the offense, the mighty blaster 80.\n&quot; <br>  else:<br>   #user has no weapon<br>   prompt_text += &quot;The player does not currently have any tools to defend itself or get rid of the old techs \n&quot;<br>   prompt_text += &quot;Recommend to the player to go fetch the weapon closest to where the player started the level which is where the blaster cleaner 80 is located, on the map represented by a yellow square. \n&quot;<br><br>  #build prompt from stats of the game<br>  prompt_text += &quot;Player has been playing for &quot;+str(stopwatch)+&quot;\n&quot; <br>  prompt_text += &quot;Player health level is &quot;+str(health)+&quot; out of 200 \n&quot;<br>  prompt_text += &quot;Player has been hit &quot;+str(hit_count)+&quot; times.\n&quot; <br>  prompt_text += &quot;Player score is &quot;+str(score)+&quot;.\n&quot;<br>  prompt_text += &quot;Player current difficulty level is &quot;+str(game_difficulty)+&quot; from 0 to 2 with 0 being easy and 2 being hard \n&quot;<br>  prompt_text += &quot;Give your answer writting in the following language: &quot;+language</pre><p>The agent is implemented in a python class as follow. The agent receive instructions from us on is role and details about the game, those instructions combined with the prompt that we pass will be the base for the help it provides the gamer.</p><p>Another point worth noticing is that we leverage the multi modality of gemini by also passing the last screenshot taken as part of the context.</p><pre>class rtr_agent:<br>    session_service = None<br>    session = None<br>    root_agent = None<br>    runner = None<br><br>    def __init__(self):<br>        asyncio.run(self.async_agent_main())<br><br>    async def get_agent_async(self):<br>        #print(f&quot;Fetched {len(tools)} tools from MCP server.&quot;)<br>        root_agent = Agent(<br>          model=AGENT_MODEL, # Adjust model name if needed based on availability<br>          name=APP_NAME,<br>         instruction=<br>   &quot;You are a gamer&#39;s best friend assisting him in playing a top down video game that takes place in an old datacenter where old technologies like CRTs and Printers have taken over and the player is tasked to first find 2 tools to clean  the datacenter and defeat the final boss.&quot;<br>   &quot;The game is composed of a single level and the user starts without any tool in his posession. Most parties are short 1 to 3 mins&quot;<br>   &quot;Based on the information provided, give short and concise instructions to the user so that he knows what to do next but without giving away too much and spoil the game:&quot;<br>   <br>   &quot;Prepare your answer with a textual help for the first field, a second field that represents the difficulty level of the game from 0 to 2 with 0 being easy and 2 being hard.&quot;<br>   &quot;If you consider that the gamer is not doing well lower the difficulty level, if the gamer is too strong highten the difficulty level.&quot; <br>   &quot;the last field is the reason why you adjusted the difficulty a certain way, please follow this JSON format {\&quot;help\&quot;:\&quot;\&quot;, \&quot;difficulty_level\&quot;:0, \&quot;reason\&quot;:\&quot;\&quot;}&quot;<br><br> <br>            ,<br>          #tools=[], # Provide the MCP tools to the ADK agent<br>          #before_model_callback=clean_image_history,<br>        )<br>        return root_agent<br><br>    # agent expects an image in base64 as well as a prompt<br>    async def call_rtr_agent_async(self, query: str, base64_image, user_id, session_id):<br>        print(&quot;Running agent...&quot;)<br>        content = types.Content(role=&#39;user&#39;, parts=[types.Part(inline_data=base64_image), types.Part(text=query)])<br><br>        final_response_text = &quot;Agent did not produce a final response.&quot; # Default<br><br>        async for event in self.runner.run_async(user_id=self.session.user_id, session_id=self.session.id, new_message=content):<br>            # print(f&quot;  [Event] Author: {event.author}, Type: {type(event).__name__}, Final: {event.is_final_response()}, Content: {event.content}&quot;)<br><br>            # Key Concept: is_final_response() marks the concluding message for the turn.<br>            if event.is_final_response():<br>              if event.content and event.content.parts:<br>                 final_response_text = event.content.parts[0].text<br>              elif event.actions and event.actions.escalate: # Handle potential errors/escalations<br>                 final_response_text = f&quot;Agent escalated: {event.error_message or &#39;No specific message.&#39;}&quot;<br>              break # Stop processing events once the final response is found<br>  <br>        return(f&quot;{final_response_text}&quot;)<br><br><br>    def __del__(self):<br>        print(&quot;destroying object&quot;)<br>       # await exit_stack.aclose()<br><br>## setup<br>    async def async_agent_main(self):<br>        self.session_service = InMemorySessionService()<br>        # Artifact service might not be needed for this example<br>        #artifacts_service = InMemoryArtifactService()<br><br>        self.session =  await self.session_service.create_session(<br>                state={},<br>                app_name=APP_NAME,<br>                user_id=USER_ID,<br>                session_id=SESSION_ID<br>            )<br>        print(f&quot;Session created: App=&#39;{APP_NAME}&#39;, User=&#39;{USER_ID}&#39;, Session=&#39;{SESSION_ID}&#39;&quot;)<br>            # Prepare the user&#39;s message in ADK format<br><br>        self.root_agent = await self.get_agent_async()<br><br>        self.runner = Runner(<br>              app_name=APP_NAME,<br>              agent=self.root_agent,<br>             # artifact_service=artifacts_service, # Optional<br>              session_service=self.session_service,<br>            )<br>        print(f&quot;Runner created for agent &#39;{self.runner.agent.name}&#39;.&quot;)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04d260746cf3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/next25-retro-tech-revolution-04d260746cf3">Next25 -Retro Tech Revolution</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>