<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Esther Irawati Setiawan on Medium]]></title>
        <description><![CDATA[Stories by Esther Irawati Setiawan on Medium]]></description>
        <link>https://medium.com/@esther.irawati?source=rss-a32fdb99f2d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*OhjvGtpa_V1hKWqi</url>
            <title>Stories by Esther Irawati Setiawan on Medium</title>
            <link>https://medium.com/@esther.irawati?source=rss-a32fdb99f2d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 08:40:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@esther.irawati/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[LoopSmith: Closed-Loop AI Engineering — Autonomous /goal Execution for Self-Correcting Pipelines on…]]></title>
            <link>https://medium.com/google-developer-experts/loopsmith-closed-loop-ai-engineering-autonomous-goal-execution-for-self-correcting-pipelines-on-22c915b0564b?source=rss-a32fdb99f2d------2</link>
            <guid isPermaLink="false">https://medium.com/p/22c915b0564b</guid>
            <category><![CDATA[google-antigravity]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <dc:creator><![CDATA[Esther Irawati Setiawan]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 14:59:18 GMT</pubDate>
            <atom:updated>2026-07-29T09:40:29.067Z</atom:updated>
            <content:encoded><![CDATA[<h3>LoopSmith: Closed-Loop AI Engineering — Autonomous /goal Execution for Self-Correcting Pipelines on Antigravity 2.0</h3><h3>Stop prompting the agent turn by turn. Hand it an objective, a bar to clear, and let the state machine write, run, and fix its own code until the output is verified.</h3><blockquote>This guide targets <strong>Antigravity 2.0</strong> — the four-surface release (desktop app, agy CLI, google-antigravity SDK, and enterprise cloud) that shares one agent harness. The SDK is pre-v1.0; symbol names and CLI flags below reflect the documented API as of mid-2026. Treat the patterns as stable and re-check exact signatures against the current docs before you ship.</blockquote><h3>Table of contents</h3><ol><li>The problem with the on-demand agent</li><li>The architectural shift: closed-loop engineering as a state machine</li><li>Step 1 — Initialize the project and the state machine (agy)</li><li>Step 2 — Define the objective and trigger /goal execution mode (SDK)</li><li>Step 3 — Implement the self-correcting loop</li><li>Step 4 — Verification and state finalization</li><li>Conclusion</li></ol><h3>1. The problem with the on-demand agent</h3><p>Most “AI engineering” today is still <strong>conversational</strong>. You prompt; the model answers. You notice the answer is wrong; you prompt again. You paste a traceback; it apologizes and tries once more. The intelligence is real — but <em>you</em> are the control loop. You are the thing that runs the code, reads the error, decides whether the output is good enough, and feeds the next instruction back in. Take the human out of that seat and the whole system stops.</p><p>That’s fine for a chat window. It falls apart the moment you want an agent to <em>produce a deliverable</em> — a cleaned dataset, a reconciled financial report, a migration that actually compiles. Real analytical work is <strong>iterative and self-referential</strong>: you write a script, it crashes on a currency string, you fix the parse, it runs but the totals don’t reconcile, you fix the aggregation, and <em>only then</em> is the output trustworthy. Every one of those arrows is a decision. An on-demand agent makes you supply all of them.</p><p>There are three specific failures hiding in “just prompt it again”:</p><ul><li><strong>No definition of done.</strong> A conversational agent stops when it <em>feels</em> finished, which is to say when it produces fluent text. Fluent is not the same as correct. “Here’s your revenue analysis!” over a script that silently dropped a third of the rows reads exactly like a correct answer. Nothing in the loop checks.</li><li><strong>The human is the error handler.</strong> The agent has no durable memory of <em>what it was trying to achieve</em> across turns, so it can’t tell whether its latest attempt is closer or further from the goal. You hold that state in your head. You are the while loop, the try/except, and the assertion — manually, turn after turn.</li><li><strong>No verified exit.</strong> Because there’s no explicit success criterion, there’s no moment the system can point to and say <em>“done, and here’s the proof.”</em> It just… trails off when you stop typing.</li></ul><p>The fix is not a smarter model or a longer prompt. It’s an <strong>architecture</strong>: give the system the objective <em>and</em> the definition of done, then let it run its own control loop — generate, execute, inspect, refine — until the output clears a bar it can check <em>for itself</em>. That is closed-loop AI engineering, and Antigravity 2.0 makes it a first-class execution mode: <strong>/goal</strong>.</p><p>We’ll build one end to end. The running example is a workload every data team recognizes: <strong>hand an agent a messy CSV and an analytical objective, and get back a verified JSON — with zero human turns in the middle.</strong> The dataset fights back on purpose (revenue stored as &quot;$1,299.00&quot;, blank region cells, mixed date formats), so the agent&#39;s <em>first</em> attempt fails and you get to watch it correct itself. We call the reference implementation <strong>LoopSmith</strong>.</p><h3>2. The architectural shift: closed-loop engineering as a state machine</h3><p>Stop thinking of the agent as a chatbot with a scrollback. Think of it as a <strong>process with states</strong>. At any instant an autonomous run is doing exactly one thing — planning, or generating code, or executing it, or inspecting the result, or refining — and the only interesting question is <em>which state comes next</em>. That question has a small, finite set of legal answers. Encode them, and you have a <strong>finite-state machine</strong>: the most boring, most battle-tested control structure in all of engineering, pointed at an LLM.</p><p>Here is the entire topology. Read the cycle in the middle — that loop <em>is</em> the self-correction engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*To5MfaT-T5RWqqjwJKpZxg.png" /></figure><ul><li><strong>Determinism you can audit.</strong> Control flow lives in a transition table, not in the model’s whim. The run can only move along edges you drew. Every execution emits a replayable tape — plan → generate → execute → inspect → refine → execute → inspect → verified — so when something goes sideways you read the tape instead of guessing. The <em>model</em> supplies judgement inside a state; the <em>machine</em> decides where the run is allowed to go next. That division is the whole trick.</li><li><strong>Self-correction as a cycle, not a retry.</strong> The EXECUTE → INSPECT → REFINE → EXECUTE loop is a closed feedback circuit. INSPECT observes the <em>actual</em> result (exit code, stderr, whether the numbers reconcile) and emits a diagnosis; REFINE consumes that diagnosis and emits a better program; EXECUTE tests it. No human turn anywhere on that circuit. The agent isn&#39;t &quot;trying again&quot; — it&#39;s reading its own error and steering.</li><li><strong>A verified exit, defined in advance.</strong> The run terminates in VERIFIED only when <strong>every acceptance check passes</strong> — and those checks are declared up front, in the PLAN state, as machine-checkable predicates (exit code 0, the artifact parses, Σ(by_region) == total within a cent). &quot;Done&quot; stops being a vibe and becomes a boolean. If the attempt budget runs out first, the run exits FAILED — loudly, with its tape — rather than pretending.</li></ul><p>The opposite of this is the <strong>open loop</strong>: prompt in, text out, human holds the state. Everything below is the move from open to closed. Two design rules make it work in practice, and they’re worth stating before any code:</p><ol><li><strong>Split judgement from control.</strong> LLMs are extraordinary at the judgement inside a state (“this traceback means the currency column is a string; strip $ and , before casting&quot;) and unreliable at being their own scheduler. So the state graph is plain, deterministic Python; the agents never decide <em>what state comes next</em> — they only decide <em>what to do in the state they&#39;re in</em>.</li><li><strong>Verify with code, not vibes.</strong> The parts of “done” that a computer can check — exit codes, schemas, reconciliation — are checked <em>in code</em>, as ground truth the model may not overrule. The model’s judgement is layered on top for the things code can’t assert, never underneath.</li></ol><p>Let’s build it.</p><h3>3. Step 1 — Initialize the project and the state machine (agy)</h3><p>Install the two surfaces you’ll use. The <strong>CLI</strong> (agy) is the Go-based terminal harness; the <strong>SDK</strong> (google-antigravity) is the same harness as a Python library. You prototype the loop interactively in the CLI, then codify it in Python.</p><pre># 1. Install the Antigravity CLI (macOS / Linux). It lands at ~/.local/bin/agy<br>curl -fsSL https://antigravity.google/cli/install.sh | bash<br># Windows PowerShell:  irm https://antigravity.google/cli/install.ps1 | iex<br># 2. Confirm the harness is live and see which models you can reach<br>agy models<br># 3. Scaffold the project<br>mkdir loopsmith &amp;&amp; cd loopsmith<br>python -m venv .venv &amp;&amp; source .venv/bin/activate<br># 4. Install the SDK. IMPORTANT: install the published wheel, not a source<br>#    checkout - the wheel ships the `localharness` binary the SDK drives.<br>pip install google-antigravity pydantic python-dotenv pandas<br># 5. Authenticate. Either a direct Gemini key…<br>export GEMINI_API_KEY=&quot;your_key_here&quot;<br>#    …or route through Vertex AI with Application Default Credentials:<br>#    gcloud auth application-default login   # then set VERTEX=1 in .env</pre><p>Before you write a single agent, write the <strong>state machine</strong>. This is the inversion that makes an autonomous run legible: the topology is defined <em>first</em>, in isolation, with no model anywhere near it. The seven states, the legal transitions, and a guard that refuses anything off the graph:</p><pre># loop/states.py<br>from dataclasses import dataclass, field<br>from enum import Enum<br>class State(str, Enum):<br>    PLAN = &quot;plan&quot;          # decompose the objective into an approach + checks<br>    GENERATE = &quot;generate&quot;  # write the first analysis script<br>    EXECUTE = &quot;execute&quot;    # run the script in the sandbox<br>    INSPECT = &quot;inspect&quot;    # judge the run against the acceptance checks<br>    REFINE = &quot;refine&quot;      # rewrite the script to fix what INSPECT found<br>    VERIFIED = &quot;verified&quot;  # terminal SUCCESS - all checks pass<br>    FAILED = &quot;failed&quot;      # terminal FAILURE - attempt budget exhausted<br>TERMINAL = frozenset({State.VERIFIED, State.FAILED})<br># The transition table IS the architecture. A move not listed here is a bug.<br>TRANSITIONS = {<br>    State.PLAN:     frozenset({State.GENERATE}),<br>    State.GENERATE: frozenset({State.EXECUTE}),<br>    State.EXECUTE:  frozenset({State.INSPECT}),<br>    State.INSPECT:  frozenset({State.VERIFIED, State.REFINE, State.FAILED}),<br>    State.REFINE:   frozenset({State.EXECUTE}),<br>}<br>class IllegalTransition(RuntimeError):<br>    &quot;&quot;&quot;Raised when the run attempts a move the transition table forbids.&quot;&quot;&quot;<br>@dataclass<br>class Machine:<br>    state: State = State.PLAN<br>    history: list[State] = field(default_factory=lambda: [State.PLAN])<br>    def to(self, nxt: State) -&gt; State:<br>        allowed = TRANSITIONS.get(self.state, frozenset())<br>        if nxt not in allowed:<br>            raise IllegalTransition(<br>                f&quot;illegal transition {self.state.value!r} → {nxt.value!r} &quot;<br>                f&quot;(allowed: {sorted(s.value for s in allowed) or &#39;none - terminal&#39;})&quot;<br>            )<br>        self.state = nxt<br>        self.history.append(nxt)<br>        return nxt<br>    @property<br>    def done(self) -&gt; bool:<br>        return self.state in TERMINAL<br>    def tape(self) -&gt; str:<br>        return &quot; → &quot;.join(s.value for s in self.history)</pre><p>Why this is the first thing you write, not the last:</p><ul><li><strong>TRANSITIONS is the contract.</strong> Notice the cycle: EXECUTE → INSPECT → REFINE → EXECUTE. That single loop in a dictionary is the self-correction engine — the run walks it until INSPECT emits VERIFIED. And notice what&#39;s <em>absent</em>: there is no edge from REFINE to GENERATE, no edge out of VERIFIED. Illegal moves aren&#39;t handled defensively; they&#39;re impossible.</li><li><strong>Machine.to() is the single choke point.</strong> Every state change in the entire system flows through this one method. That&#39;s what guarantees history is complete and the tape is trustworthy — there&#39;s no back door that mutates state without recording it. In Step 4 the audit hook narrates this live.</li><li><strong>It’s model-free, so it’s testable.</strong> You can unit-test the topology — “can a run reach VERIFIED without passing through INSPECT?” — with zero API calls and zero flakiness. The stochastic part (the agents) and the deterministic part (the graph) never contaminate each other.</li></ul><p>Interactively, you can watch the same machine run before you’ve written any Python at all. From the agy TUI, /goal drops you into execution mode and streams the state transitions as they happen:</p><pre>agy&gt; /goal &quot;Reconcile revenue by region in ./data/sales.csv into a verified JSON&quot;<br>◆ plan       → 6 acceptance checks declared<br>◆ generate   → analysis.py (41 lines)<br>◆ execute    → exit 1  (ValueError: could not convert string to float: &#39;$1,299.00&#39;)<br>◆ inspect    → NOT verified · reconciles_total blocked by crash<br>◆ refine     → strip &#39;$&#39;/&#39;,&#39; before cast<br>◆ execute    → exit 0  · result.json written<br>◆ inspect    → NOT verified · no_null_region FAIL (blank region leaked as key)<br>◆ refine     → drop blank-region rows<br>◆ execute    → exit 0<br>◆ inspect    → VERIFIED ✓  (6/6 checks)<br>◆ done       → tape: plan→generate→execute→inspect→refine→execute→inspect→refine→execute→inspect→verified</pre><p>That’s the whole story on one screen: two self-corrections, no human turn, a verified exit. Now we build the Python that makes it reproducible.</p><h3>4. Step 2 — Define the objective and trigger /goal execution mode (SDK)</h3><p>/goal mode needs two inputs and nothing else: a <strong>high-level objective</strong> and a <strong>definition of done</strong>. Everything else — how many iterations, which script, which fix — the loop decides for itself.</p><p>There are two legitimate ways to invoke it, and a senior engineer should know both.</p><p><strong>Pattern A — harness-native </strong><strong>/goal.</strong> Hand the harness the objective and a success criterion and let <em>it</em> own the loop. The SDK exposes run_goal; the harness plans, generates, executes, inspects, and refines internally, and returns a GoalResult. This is the least code and the right default for exploratory work:</p><pre># the one-liner form — the harness runs the whole closed loop for you<br>from google.antigravity import Agent, GoalConfig<br>async def quick_goal():<br>    async with Agent.for_goal(GoalConfig(model=&quot;gemini-3-pro&quot;)) as agent:<br>        result = await agent.run_goal(<br>            objective=(<br>                &quot;Analyze ./data/sales.csv and report total revenue per region, &quot;<br>                &quot;the top region, and its share of total revenue as result.json.&quot;<br>            ),<br>            success=(<br>                &quot;python exits 0; result.json parses; sum(by_region) == total_revenue &quot;<br>                &quot;within 0.01; top_region == argmax(by_region); no blank region key.&quot;<br>            ),<br>            max_iterations=5,   # the self-correction budget<br>        )<br>        print(result.final_state, result.verified, result.tape)</pre><p><strong>Pattern B — the explicit state machine.</strong> When you need <em>determinism and auditability</em> — which a production pipeline always does — you drive the same loop yourself with the SDK primitives, so every transition, every check, and every refinement is your code. This is what the rest of the tutorial builds, because seeing the machinery is the point.</p><p>Either way, the definition of done is not prose you hope the model honors — it’s a list of <strong>acceptance checks</strong>, and the first agent’s whole job is to produce them. Structured output is what makes that possible. Every model turn in LoopSmith is constrained to a Pydantic schema via response_schema=, so the state machine receives <em>typed objects it can branch on</em>, not text it has to parse:</p><pre># loop/schemas.py  (the contracts that flow between states)<br>from enum import Enum<br>from pydantic import BaseModel, Field<br>class CheckKind(str, Enum):<br>    DETERMINISTIC = &quot;deterministic&quot;  # enforced in code (verifier.py)<br>    JUDGEMENT = &quot;judgement&quot;          # delegated to the INSPECT agent<br>class AcceptanceCheck(BaseModel):<br>    id: str = Field(description=&quot;stable slug, e.g. &#39;reconciles_total&#39;&quot;)<br>    description: str<br>    kind: CheckKind<br>class Plan(BaseModel):<br>    objective: str<br>    approach: list[str]<br>    output_contract: str           # the exact JSON the script must write<br>    checks: list[AcceptanceCheck] = Field(min_length=1)<br>class GeneratedCode(BaseModel):    # reused by BOTH generate and refine<br>    filename: str = &quot;analysis.py&quot;<br>    language: str = &quot;python&quot;<br>    source: str                    # the complete file contents, ready to run<br>    notes: str = &quot;&quot;                # what changed vs. the last version<br>class Inspection(BaseModel):       # the branch decision, as data<br>    verified: bool                 # True iff EVERY acceptance check passes<br>    failing_checks: list[str] = Field(default_factory=list)<br>    diagnosis: str                 # root cause, grounded in stderr / checks<br>    fix_hint: str = &quot;&quot;             # concrete change REFINE should make</pre><p>Two schema decisions carry the architecture:</p><ul><li><strong>Inspection.verified is a </strong><strong>bool, so the branch is a field read.</strong> The whole INSPECT → {VERIFIED, REFINE} fork collapses to if inspection.verified:. No regex over model prose, no &quot;did it say the word <em>passed</em>?&quot; heuristics. The model fills a typed field; the machine reads it.</li><li><strong>GeneratedCode is shared by GENERATE and REFINE.</strong> A refinement is just <em>the next version of the same artifact</em>. Because both states emit the identical schema, the transition table can route both into EXECUTE with no special casing — the loop doesn&#39;t care whether a script is the first draft or the third.</li></ul><h3>5. Step 3 — Implement the self-correcting loop</h3><p>Now the substance: the four agents, the sandbox that runs their code, and the driver that walks the state machine. Every agent is minted from <strong>one factory with one set of guardrails</strong> — the <em>Shared Agent Harness</em>. An autonomous run that writes and executes its own code is exactly the run you want on the shortest possible leash: default-deny policy, one audit line per tool call, no network, no shell.</p><pre># loop/harness.py<br>import os<br>from typing import Callable, Sequence<br>from google.antigravity import LocalAgentConfig<br>from google.antigravity.hooks import hooks, policy<br>MODEL = os.getenv(&quot;LOOPSMITH_MODEL&quot;, &quot;gemini-3-pro&quot;)<br># Least privilege, encoded once, inherited by every agent in the loop.<br>BASE_POLICIES = [<br>    policy.allow(&quot;view_file&quot;),<br>    policy.allow(&quot;write_file&quot;, when=lambda a: _within_workspace(a.get(&quot;path&quot;, &quot;&quot;))),<br>    policy.deny(&quot;run_command&quot;),   # agents never shell out - the SANDBOX does<br>    policy.deny(&quot;net_fetch&quot;),     # a self-correcting loop must stay hermetic<br>    policy.deny(&quot;*&quot;),             # default-deny: anything not allowed is refused<br>]<br>class StateAuditHook(hooks.PostToolCallHook):<br>    async def run(self, context, data) -&gt; None:<br>        print(f&quot;[audit] state={getattr(context, &#39;label&#39;, &#39;?&#39;)} &quot;<br>              f&quot;tool={getattr(data, &#39;name&#39;, &#39;?&#39;)} ok={getattr(data, &#39;ok&#39;, True)}&quot;)<br>def build_agent_config(*, label: str, system_instructions: str,<br>                       tools: Sequence[Callable] = ()) -&gt; LocalAgentConfig:<br>    kwargs = dict(<br>        model=MODEL, label=label, system_instructions=system_instructions,<br>        tools=list(tools), policies=BASE_POLICIES, hooks=[StateAuditHook()],<br>    )<br>    if os.getenv(&quot;VERTEX&quot;) == &quot;1&quot;:<br>        kwargs[&quot;vertex&quot;] = True<br>    elif os.getenv(&quot;GEMINI_API_KEY&quot;):<br>        kwargs[&quot;api_key&quot;] = os.environ[&quot;GEMINI_API_KEY&quot;]<br>    return LocalAgentConfig(**kwargs)<br>def _within_workspace(path: str) -&gt; bool:<br>    return &quot;workspace&quot; in os.path.normpath(path).split(os.sep)</pre><p>The critical decision here is <strong>policy.deny(&quot;run_command&quot;)</strong> — the agents themselves get <em>no shell</em>. That looks paradoxical for a loop whose whole job is to run code, until you see where execution actually happens: in a <strong>separate, deterministic sandbox</strong> the orchestrator controls, not inside a model&#39;s tool call. The model <em>writes</em> the program; a governed subprocess <em>runs</em> it. The model can never pip install or curl its way out, because it was never handed the keys.</p><h3>The four agents — judgement, no control flow</h3><p>Each state’s agent is a single-responsibility config. None of them holds orchestration logic; none of them knows it’s in a loop. They reason over what they’re handed and emit a typed object — that’s it.</p><pre># loop/agents.py  (abridged — full system prompts in the repo)<br>from .harness import build_agent_config<br>def planner_config():     # PLAN: objective -&gt; approach + machine-checkable checks<br>    return build_agent_config(label=&quot;plan&quot;, system_instructions=(<br>        &quot;You are a senior data analyst PLANNER. Given an objective and a preview &quot;<br>        &quot;of a messy CSV, produce an approach, an exact output_contract for the &quot;<br>        &quot;JSON the script must write, and a list of ACCEPTANCE CHECKS. Prefer these &quot;<br>        &quot;deterministic ids when they apply: exits_clean, artifact_exists, &quot;<br>        &quot;schema_ok, no_null_region, reconciles_total, top_is_argmax. The run is &quot;<br>        &quot;&#39;verified&#39; only when EVERY check passes - do not under-specify.&quot;))<br>def generator_config():   # GENERATE: write the first script<br>    return build_agent_config(label=&quot;generate&quot;, system_instructions=(<br>        &quot;You are a Python DATA ENGINEER. Write a COMPLETE, self-contained script &quot;<br>        &quot;that reads os.environ[&#39;LOOPSMITH_DATASET&#39;], performs the planned &quot;<br>        &quot;analysis, and writes &#39;result.json&#39;. Use only the standard library and &quot;<br>        &quot;pandas. No network. Return the full file contents.&quot;))<br>def inspector_config():   # INSPECT: verified vs. refine, grounded in hard checks<br>    return build_agent_config(label=&quot;inspect&quot;, system_instructions=(<br>        &quot;You are a rigorous VERIFICATION agent. You get the acceptance checks, the &quot;<br>        &quot;sandbox RunResult, and the results of the DETERMINISTIC checks already &quot;<br>        &quot;computed in code. You may NOT mark verified if any deterministic check &quot;<br>        &quot;failed - those are ground truth. When not verified, write a diagnosis &quot;<br>        &quot;rooted in the real stderr/failed checks and a concrete fix_hint.&quot;))<br>def refiner_config():     # REFINE: rewrite to fix exactly what INSPECT diagnosed<br>    return build_agent_config(label=&quot;refine&quot;, system_instructions=(<br>        &quot;You are a DEBUGGING engineer. Given the current script, the RunResult &quot;<br>        &quot;showing how it failed, and a diagnosis + fix_hint, produce the NEXT full &quot;<br>        &quot;version that fixes the identified problem while preserving what worked.&quot;))</pre><p>The asymmetry is the design. The <strong>planner</strong> turns a fuzzy objective into a testable bar. The <strong>inspector</strong> is explicitly forbidden from overruling the deterministic checks — its judgement sits <em>on top of</em> ground truth, never underneath it. The <strong>refiner</strong> receives a diagnosis, not a vague “try again,” so its rewrite is targeted. Each has a small context window and one job.</p><h3>The sandbox — where generated code actually runs</h3><p>This module is <em>real, runnable Python</em> — no SDK, no model. It is the EXECUTE state, and it’s the security boundary. It runs the agent’s script as a subprocess, in a confined working directory, with a wall-clock timeout and a stripped environment, and reports back exactly what happened.</p><pre># loop/sandbox.py  (core)<br>import subprocess, sys, time<br>from pathlib import Path<br>from .schemas import GeneratedCode, RunResult<br>WORKSPACE = Path(__file__).resolve().parent.parent / &quot;workspace&quot;<br>def write_code(code: GeneratedCode) -&gt; Path:<br>    WORKSPACE.mkdir(parents=True, exist_ok=True)<br>    path = WORKSPACE / code.filename<br>    path.write_text(code.source, encoding=&quot;utf-8&quot;)<br>    return path<br>def run_python(path: Path, *, dataset: Path, timeout_s: float = 30.0) -&gt; RunResult:<br>    before = _snapshot()                      # which files exist before the run<br>    started = time.monotonic()<br>    try:<br>        proc = subprocess.run(<br>            [sys.executable, str(path)],<br>            cwd=WORKSPACE,                     # confine the working directory<br>            env={&quot;LOOPSMITH_DATASET&quot;: str(dataset),   # minimal env - no secrets leak<br>                 &quot;PATH&quot;: _safe_path(), &quot;PYTHONUNBUFFERED&quot;: &quot;1&quot;},<br>            capture_output=True, text=True, timeout=timeout_s,<br>        )<br>        exit_code, stdout, stderr = proc.returncode, proc.stdout, proc.stderr<br>        timed_out = False<br>    except subprocess.TimeoutExpired as e:<br>        exit_code, stdout = 124, (e.stdout or &quot;&quot;)<br>        stderr = (e.stderr or &quot;&quot;) + f&quot;\n[sandbox] killed after {timeout_s}s timeout&quot;<br>        timed_out = True<br>    artifacts = sorted(p.name for p in _snapshot() - before)   # what the run created<br>    return RunResult(exit_code=exit_code, stdout=_clip(stdout), stderr=_clip(stderr),<br>                     duration_s=round(time.monotonic() - started, 3),<br>                     artifacts=artifacts, timed_out=timed_out)</pre><p>Line by line, this is the entire EXECUTE contract:</p><ul><li><strong>subprocess.run(..., cwd=WORKSPACE, timeout=timeout_s)</strong> — the generated code runs <em>outside the agent&#39;s process</em>, in its own directory, and cannot run longer than the timeout. An infinite loop the model wrote gets SIGKILL, surfaces as exit_code=124, and becomes just another INSPECT input — not a hung pipeline.</li><li><strong>env={...}</strong> — a hand-built environment. The dataset path arrives via LOOPSMITH_DATASET; your API keys, cloud credentials, and shell aliases do <em>not</em>. The script sees exactly what it needs and nothing else.</li><li><strong>_snapshot() before and after</strong> — the sandbox diffs the workspace to report precisely which artifacts the run produced. That&#39;s how the loop <em>knows</em> result.json was written, rather than trusting the model&#39;s stdout claim that it was.</li><li><strong>The </strong><strong>RunResult is the feedback signal.</strong> Exit code, stdout, stderr, artifacts — the raw truth of what happened, handed straight to INSPECT. The loop corrects itself against <em>observed behavior</em>, never against the model&#39;s hopes.</li></ul><h3>The driver — walking the machine</h3><p>Here is the closed loop, top to bottom. This is the explicit form of /goal: you can trace every transition, and every m.to(...) is a guarded, logged move.</p><pre># loop/orchestrator.py<br>from pathlib import Path<br>from google.antigravity import Agent<br>from . import dataset, sandbox, verifier<br>from .agents import planner_config, generator_config, inspector_config, refiner_config<br>from .schemas import Plan, GeneratedCode, Inspection, GoalResult<br>from .states import Machine, State<br>async def _turn(config, prompt, schema):<br>    &quot;&quot;&quot;One isolated agent, one typed turn, then tear down.&quot;&quot;&quot;<br>    async with Agent(config) as agent:<br>        resp = await agent.chat(prompt, response_schema=schema)<br>        return await resp.parsed()            # a validated Pydantic instance<br>async def run_goal(objective, *, dataset_path, max_attempts=5) -&gt; GoalResult:<br>    data = Path(dataset_path)<br>    preview = dataset.preview(data)<br>    m = Machine()                             # starts in PLAN<br>    # ---- PLAN: objective -&gt; approach + acceptance checks --------------------<br>    plan = await _turn(planner_config(),<br>                       f&quot;OBJECTIVE:\n{objective}\n\nDATASET PREVIEW:\n{preview}&quot;, Plan)<br>    m.to(State.GENERATE)<br>    # ---- GENERATE: write the first script -----------------------------------<br>    code = await _turn(generator_config(), _generate_prompt(objective, plan, preview),<br>                       GeneratedCode)<br>    code_path = sandbox.write_code(code)<br>    m.to(State.EXECUTE)<br>    attempts = 0<br>    while True:<br>        # ---- EXECUTE: run the current script in the sandbox -----------------<br>        run = sandbox.run_python(code_path, dataset=data)<br>        attempts += 1<br>        m.to(State.INSPECT)<br>        # ---- INSPECT: hard checks first, then agent judgement ---------------<br>        det = verifier.evaluate(plan.checks, run, sandbox.WORKSPACE)<br>        inspection = await _turn(inspector_config(),<br>                                 _inspect_prompt(plan, run, det), Inspection)<br>        verified = inspection.verified and verifier.all_passed(det)<br>        if verified:<br>            m.to(State.VERIFIED); break        # terminal success<br>        if attempts &gt;= max_attempts:<br>            m.to(State.FAILED); break          # terminal failure - budget spent<br>        # ---- REFINE: rewrite the script to fix what INSPECT found -----------<br>        m.to(State.REFINE)<br>        code = await _turn(refiner_config(),<br>                           _refine_prompt(code, run, det, inspection), GeneratedCode)<br>        code_path = sandbox.write_code(code)<br>        m.to(State.EXECUTE)                     # close the loop - back to EXECUTE<br>    return GoalResult(<br>        objective=objective, final_state=m.state.value,<br>        verified=(m.state is State.VERIFIED), attempts=attempts, tape=m.tape(),<br>        report=verifier.load_report(sandbox.WORKSPACE) if m.state is State.VERIFIED else {},<br>        code_path=str(code_path), artifacts=run.artifacts,<br>    )</pre><p>Read that while loop as the feedback circuit it is:</p><ul><li><strong>run = sandbox.run_python(...)</strong> is the plant — it produces a real observation. <strong>det = verifier.evaluate(...)</strong> and <strong>inspection</strong> are the sensor. <strong>code = await _turn(refiner_config(), ...)</strong> is the controller. The loop closes when EXECUTE runs the controller&#39;s new output. That&#39;s not a metaphor; it&#39;s literally a discrete feedback controller with an LLM in the actuator.</li><li><strong>verified = inspection.verified and verifier.all_passed(det)</strong> is the safety interlock. The model&#39;s verified flag is <em>necessary but not sufficient</em> — the deterministic checks get a hard veto via the and. A model that hallucinates success cannot force a VERIFIED exit; the arithmetic won&#39;t let it.</li><li><strong>if attempts &gt;= max_attempts: m.to(State.FAILED)</strong> is the anti-livelock. A self-correcting loop <em>must</em> be able to give up. Without this bound, a genuinely impossible objective would burn tokens forever. With it, the run terminates honestly — FAILED, with its full tape — and a human is paged with the evidence already assembled.</li><li><strong>Every branch is a guarded </strong><strong>m.to(...).</strong> The control flow you&#39;re reading is the <em>only</em> control flow there is; the model never picks the next state. That is what makes an autonomous run something you can put on a schedule and trust.</li></ul><p>Point this at the messy sales.csv and the first EXECUTE fails exactly as designed — ValueError: could not convert string to float: &#39;$1,299.00&#39;. INSPECT reads that traceback, sees reconciles_total blocked, and emits a diagnosis; REFINE strips the $ and , and re-runs; the next INSPECT catches a subtler bug (a blank region cell leaked in as a by_region key); REFINE drops it; the third run clears all six checks. No human typed a word.</p><h3>6. Step 4 — Verification and state finalization</h3><p>The VERIFIED exit is only meaningful if &quot;verified&quot; means something a machine checked. That&#39;s the verifier — the deterministic half of INSPECT. It&#39;s real, runnable Python, and the INSPECT agent is <em>forbidden from overruling it</em>.</p><pre># loop/verifier.py  (core checks)<br>import json<br>from pathlib import Path<br>from .schemas import AcceptanceCheck, CheckKind, CheckResult, RunResult<br>RESULT_FILE = &quot;result.json&quot;<br>TOLERANCE = 0.01                                   # one cent<br>def evaluate(checks, run, workspace) -&gt; list[CheckResult]:<br>    result = _load_result(workspace)               # parse the artifact once<br>    out = []<br>    for check in checks:<br>        if check.kind is not CheckKind.DETERMINISTIC:<br>            continue                               # judgement checks belong to INSPECT<br>        out.append(_run_check(check, run, result))<br>    return out<br>def _reconciles_total(_run, result):               # the check that matters most<br>    by_region, total = result[&quot;by_region&quot;], result[&quot;total_revenue&quot;]<br>    summed = sum(float(v) for v in by_region.values())<br>    delta = abs(summed - float(total))<br>    return delta &lt;= TOLERANCE, f&quot;Σby_region={summed:.2f} vs total={total:.2f} (Δ={delta:.4f})&quot;<br>def _no_null_region(_run, result):                 # catches the blank-cell leak<br>    bad = [k for k in result.get(&quot;by_region&quot;, {}) if k in (None, &quot;&quot;, &quot;nan&quot;, &quot;None&quot;)]<br>    return (not bad), (f&quot;leaked empty region key(s): {bad}&quot; if bad else &quot;clean&quot;)<br>def _top_is_argmax(_run, result):                  # the claim must match the data<br>    by = result[&quot;by_region&quot;]<br>    argmax = max(by, key=lambda k: float(by[k]))<br>    return argmax == result[&quot;top_region&quot;], f&quot;argmax={argmax!r} vs claim={result[&#39;top_region&#39;]!r}&quot;</pre><p>Three things make this a <em>finalization</em> mechanism and not just a linter:</p><ul><li><strong>Checks fail closed.</strong> An unknown check id, a crashing check, a missing artifact — all resolve to False, never True. The loop can only exit VERIFIED by <em>affirmatively</em> satisfying every predicate. Ambiguity refines; it never passes. That&#39;s the correct default for autonomy: silence is not consent.</li><li><strong>Ground truth outranks the model.</strong> _reconciles_total re-computes the sum from the artifact the script wrote and compares it to the script&#39;s own claimed total. If the code dropped rows, the numbers diverge and the check fails — regardless of how confidently the model narrated success. Reconciliation is not a matter of opinion.</li><li><strong>The verified artifact is the deliverable.</strong> On a VERIFIED exit, verifier.load_report() reads back the proven result.json and stamps it into the GoalResult. The run doesn&#39;t return <em>a description</em> of the analysis — it returns <em>the analysis</em>, already checked:</li></ul><pre>{<br>  &quot;objective&quot;: &quot;…revenue per region, top region, and its share…&quot;,<br>  &quot;final_state&quot;: &quot;verified&quot;,<br>  &quot;verified&quot;: true,<br>  &quot;attempts&quot;: 3,<br>  &quot;tape&quot;: &quot;plan → generate → execute → inspect → refine → execute → inspect → refine → execute → inspect → verified&quot;,<br>  &quot;report&quot;: {<br>    &quot;by_region&quot;: {&quot;North&quot;: 10749.90, &quot;South&quot;: 8552.50, &quot;East&quot;: 7212.25, &quot;West&quot;: 8631.00},<br>    &quot;total_revenue&quot;: 35145.65,<br>    &quot;top_region&quot;: &quot;North&quot;,<br>    &quot;top_region_share&quot;: 0.3059<br>  }<br>}</pre><p>The tape field is the finalization proof: you can read the entire life of the run — three EXECUTE attempts, two self-corrections, a verified exit — as a single string. And because the machine only reaches VERIFIED by clearing every check, that report is safe to hand downstream <em>without a human re-checking it</em>. That is the whole promise of closed-loop engineering: not &quot;the agent produced something,&quot; but &quot;the agent produced something, and here is the machine-checked proof it&#39;s right.&quot;</p><p>To run the whole thing as a governed build step — headless, worktree-isolated, pipeable — that’s the agy CLI in /goal mode:</p><pre>agy goal \<br>    --max-iterations 5 \<br>    --success &quot;python main.py exits 0 and workspace/result.json reconciles:<br>               sum(by_region) == total_revenue (±0.01), top_region == argmax,<br>               and no blank region key.&quot; \<br>    --output-format json \<br>    -p &quot;Run \`python main.py\`. If the acceptance checks fail, inspect the<br>        traceback and refine the analysis script until every check passes.<br>        Then commit result.json and open a PR &#39;[goal] verified analysis&#39;.&quot; \<br>  | jq &#39;{state: .final_state, verified, attempts, tape}&#39;</pre><p>--success declares the bar; --max-iterations bounds the budget; --output-format json makes the run a pipe stage. Each scheduled run gets its own git worktree, so the loop can rewrite its own scripts all it likes without ever touching your live checkout. The CLI and the SDK are two faces of one harness — prototype the loop interactively with /goal, then codify it in Python for CI.</p><h3>7. Conclusion</h3><p>The on-demand agent was a useful stage — it proved the model could reason. But reasoning isn’t engineering. Engineering is a <em>loop</em>: produce, test, observe, correct, and stop only when the output clears a bar you set in advance. For thirty years we’ve built that loop out of CI pipelines and assertions and retries. The shift Antigravity 2.0 makes possible is to put an LLM <em>inside</em> that loop as the generator and the debugger, while keeping the loop itself deterministic, bounded, and auditable.</p><p>That’s the mental model to leave with. The state machine is not boilerplate around the intelligence — it <em>is</em> the engineering. The model supplies judgement inside each state; the machine supplies the control flow, the verification, and the verified exit. <strong>/goal</strong> gives you the objective-in, proof-out contract; <strong>structured output</strong> makes each transition a field read instead of a guess; the <strong>sandbox</strong> lets the agent run its own code without you trusting it; and the <strong>verifier</strong> makes &quot;done&quot; a boolean a machine can defend.</p><p>Build the machine once. Then hand it an objective and walk away — and come back to a result with a receipt.</p><h3>Acknowledgement</h3><p>Google Cloud Credits are provided for this project. #AgenticArchitectSprint #Antigravity</p><h3>Further reading</h3><ul><li>Google Antigravity SDK (Python): <a href="https://github.com/google-antigravity/antigravity-sdk-python">https://github.com/google-antigravity/antigravity-sdk-python</a></li><li>Antigravity CLI (agy): <a href="https://github.com/google-antigravity/antigravity-cli">https://github.com/google-antigravity/antigravity-cli</a></li><li>/goal execution mode: <a href="https://antigravity.google/docs/goal-mode">https://antigravity.google/docs/goal-mode</a></li><li>Companion tutorial — <em>Multi-Agent Dispatch</em> (concurrent subagents): the fan-out counterpart to this fan-forward loop. LoopSmith: Closed-Loop AI Engineering — Autonomous /goal Execution for Self-Correcting Pipelines on Antigravity 2.0</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22c915b0564b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developer-experts/loopsmith-closed-loop-ai-engineering-autonomous-goal-execution-for-self-correcting-pipelines-on-22c915b0564b">LoopSmith: Closed-Loop AI Engineering — Autonomous /goal Execution for Self-Correcting Pipelines on…</a> was originally published in <a href="https://medium.com/google-developer-experts">Google Developer Experts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Agent Dispatch: Orchestrating Concurrent Subagents for Multimodal Analysis with Antigravity 2.]]></title>
            <link>https://medium.com/google-developer-experts/multi-agent-dispatch-orchestrating-concurrent-subagents-for-multimodal-analysis-with-antigravity-2-f198c7e7f808?source=rss-a32fdb99f2d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f198c7e7f808</guid>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[multimodal]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[google-antigravity]]></category>
            <dc:creator><![CDATA[Esther Irawati Setiawan]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 14:33:21 GMT</pubDate>
            <atom:updated>2026-07-10T14:33:21.739Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UWMPeEP6gUnxB7-5evbzlw.png" /></figure><h3>Multi-Agent Dispatch: Orchestrating Concurrent Subagents for Multimodal Analysis with Antigravity 2.0</h3><h3>Stop building one giant agent. Start building a tree of small ones — and let the harness run them in parallel.</h3><blockquote>This guide targets <strong>Antigravity 2.0</strong> — the four-surface release (desktop app, agy CLI, google-antigravity SDK, and enterprise cloud) that shares one agent harness. The SDK is pre-v1.0; symbol names and CLI flags below reflect the documented API as of mid-2026. Treat the patterns as stable and re-check exact signatures against the current docs before you ship.</blockquote><h3>Table of contents</h3><ol><li>The problem with the monolith</li><li>The architectural shift: tree-based orchestration</li><li>Step 1 — The Shared Agent Harness</li><li>Step 2 — Spawning concurrent subagents</li><li>Step 3 — Synthesizing parallel work into structured JSON</li><li>Step 4 — Enterprise scale: worktrees &amp; multi-repo context</li><li>Conclusion</li></ol><h3>1. The problem with the monolith</h3><p>Most teams’ first “real” agent is a monolith. One system prompt, one enormous context window, one loop that reads files, calls tools, inspects an image, reasons about compliance, and writes a report — sequentially. It demos beautifully. Then it meets production, and two failure modes show up on the same afternoon.</p><p><strong>Context pollution.</strong> When a single agent reads a marketing image, the raw visual tokens, the OCR, and the model’s running commentary all land in the same context window it’s using to reason about the <em>text</em> of the ad. The two tasks now contaminate each other. The model starts describing pixels when you asked about claims; it anchors its legal judgement on a caption it happened to notice. Every additional responsibility you bolt on makes the window noisier and the reasoning shallower. This is <strong>flat-prompt saturation</strong>: quality degrades not because the model is weak, but because you’ve buried the signal.</p><p><strong>Sequential latency.</strong> A monolith does one thing at a time. Analyze the copy, <em>then</em> inspect the image, <em>then</em> adjudicate. For one asset that’s an annoyance. For a batch of ten thousand it’s a budget line. The two analyses don’t actually depend on each other — but a single agent has no way to know that, so it pays for text + image when it could have paid for max(text, image).</p><p>The architectural answer is not a bigger model or a cleverer prompt. It’s the same answer distributed systems reached decades ago: <strong>stop doing everything in one process. Isolate state, and run independent work concurrently.</strong> Antigravity 2.0 is the first agent platform that treats this as a first-class primitive rather than something you hand-roll with threads and prayer.</p><h3>2. The architectural shift: tree-based orchestration</h3><p>Think of an agent not as a chatbot but as a <strong>systemic operator</strong> — a unit of compute with its own isolated context, its own tools, and its own lifecycle. In that frame, the unit of design is no longer “the prompt.” It’s the <strong>topology</strong>: how operators are arranged and how work flows between them.</p><p>Antigravity’s subagent model gives you a <strong>tree</strong>. A parent (orchestrator) delegates a subtask to a short-lived child (subagent). The child spins up with a <strong>clean context window</strong>, does its job with its own tools and MCP servers, and returns only a <em>synthesized result</em>. Then it shuts down. The parent’s context never sees the child’s scratch work — only its conclusion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0V4zqVzy6s_fwh7Z3gbWqw.jpeg" /></figure><p>Three properties fall out of this shape, and they map one-to-one onto the three things an architect actually cares about:</p><ul><li><strong>State isolation.</strong> Each node has its own context window. The image bytes the verifier loads never touch the text analyzer’s reasoning. Neither leaf can be biased by the other’s framing, because neither can <em>see</em> the other. Isolation isn’t a nice-to-have; it’s what makes the individual judgements trustworthy.</li><li><strong>Concurrency.</strong> Sibling nodes are independent by construction, so the harness runs them at the same time. Wall-clock cost drops from the <em>sum</em> of the leaves to the <em>slowest</em> leaf.</li><li><strong>Compute optimization.</strong> Because the tree is explicit, you can put a governor on it — a semaphore that bounds how many nodes (and therefore how many model sessions) are alive at once. You scale to ten thousand assets without opening ten thousand sessions.</li></ul><p>The opposite of this is the <strong>flat prompt</strong>: one context window doing everything, where every new requirement competes for the same tokens and the same clock. The whole of this tutorial is the move from flat to tree.</p><p>Let’s build one. Our running example is a real enterprise workload: <strong>multimodal compliance auditing.</strong> Each asset is a {marketing copy, hero image} pair. We need to know: does the <em>image</em> actually support what the <em>copy</em> claims? &quot;Clinically proven to erase wrinkles in 7 days&quot; over a stock photo of a gold bottle is a compliance incident waiting to happen. One subagent reads the words. Another inspects the pixels. A third adjudicates. In parallel, at scale.</p><h3>3. Step 1 — The Shared Agent Harness</h3><p>Every node in the tree is minted from <strong>one factory</strong> with <strong>one set of guardrails</strong>. This is the <em>Shared Agent Harness</em>: define security policy and telemetry once, inherit them everywhere. It’s the difference between a fleet and a mob.</p><p>First, install the two surfaces you’ll use. The <strong>CLI</strong> (agy) is the Go-based terminal harness; the <strong>SDK</strong> (google-antigravity) is the same harness as a Python library.</p><pre># 1. Install the Antigravity CLI (macOS / Linux). It lands at ~/.local/bin/agy<br>curl -fsSL https://antigravity.google/cli/install.sh | bash<br># Windows PowerShell:  irm https://antigravity.google/cli/install.ps1 | iex<br># 2. Confirm the harness is live and see which models you can reach<br>agy models<br># 3. Scaffold the Python project<br>mkdir multi-agent-dispatch &amp;&amp; cd multi-agent-dispatch<br>python -m venv .venv &amp;&amp; source .venv/bin/activate<br># 4. Install the SDK. IMPORTANT: install the published wheel, not a source<br>#    checkout - the wheel ships the `localharness` binary the SDK drives.<br>pip install google-antigravity pydantic python-dotenv<br># 5. Authenticate. Either a direct Gemini key…<br>export GEMINI_API_KEY=&quot;your_key_here&quot;<br>#    …or route through Vertex AI with Application Default Credentials:<br>#    gcloud auth application-default login   # then set VERTEX=1 in .env</pre><p>Now the harness itself. The key ideas: a <strong>default-deny</strong> policy set (an analysis agent should read files and images, never rm a repo), a <strong>single audit hook</strong> that fires on every tool call in <em>every</em> subagent, and a factory that turns subagent capability on <strong>only</strong> for orchestrator nodes.</p><pre># dispatch/harness.py<br>import os<br>from typing import Callable, Sequence<br>from google.antigravity import LocalAgentConfig, CapabilitiesConfig<br>from google.antigravity.hooks import hooks, policy<br>MODEL = os.getenv(&quot;DISPATCH_MODEL&quot;, &quot;gemini-3-pro&quot;)<br># Shared guardrails - least privilege, encoded once.<br>BASE_POLICIES = [<br>    policy.allow(&quot;view_file&quot;),<br>    policy.allow(&quot;read_image&quot;),<br>    policy.deny(&quot;run_command&quot;, when=lambda args: &quot;rm &quot; in args.get(&quot;CommandLine&quot;, &quot;&quot;)),<br>    policy.deny(&quot;*&quot;),  # default-deny: anything not allowed is refused<br>]<br># One audit line per tool call - in the orchestrator AND every spawned subagent.<br>class AuditTrailHook(hooks.PostToolCallHook):<br>    async def run(self, context: hooks.HookContext, data) -&gt; None:<br>        print(<br>            f&quot;[audit] trajectory={getattr(context, &#39;trajectory_id&#39;, &#39;?&#39;)} &quot;<br>            f&quot;tool={getattr(data, &#39;name&#39;, &#39;?&#39;)} ok={getattr(data, &#39;ok&#39;, True)}&quot;<br>        )<br>def build_agent_config(<br>    *,<br>    system_instructions: str,<br>    tools: Sequence[Callable] = (),<br>    enable_subagents: bool = False,<br>    subagents: Sequence[LocalAgentConfig] = (),<br>) -&gt; LocalAgentConfig:<br>    kwargs = dict(<br>        model=MODEL,<br>        system_instructions=system_instructions,<br>        tools=list(tools),<br>        policies=BASE_POLICIES,     # inherited by every child<br>        hooks=[AuditTrailHook()],   # inherited by every child<br>    )<br>    if os.getenv(&quot;VERTEX&quot;) == &quot;1&quot;:<br>        kwargs[&quot;vertex&quot;] = True<br>    elif os.getenv(&quot;GEMINI_API_KEY&quot;):<br>        kwargs[&quot;api_key&quot;] = os.environ[&quot;GEMINI_API_KEY&quot;]<br>    # Only orchestrators may fan out. Leaves inherit the guardrails but cannot<br>    # spawn - that keeps the tree shallow and its cost predictable.<br>    if enable_subagents:<br>        kwargs[&quot;capabilities&quot;] = CapabilitiesConfig(subagents=True)<br>        kwargs[&quot;subagents&quot;] = list(subagents)<br>    return LocalAgentConfig(**kwargs)</pre><p>Two design decisions worth pausing on, because they <em>are</em> the architecture:</p><ul><li><strong>Subagents inherit the parent’s restrictions — by default and by design.</strong> In the Antigravity SDK, a child cannot do what its parent was forbidden. If the orchestrator can’t shell out, neither can any agent it spawns. The harness makes that inheritance explicit: BASE_POLICIES and AuditTrailHook are set once on the base config, so a fan-out you never hand-wrote is still fully governed and fully logged.</li><li><strong>Capability is opt-in and shallow.</strong> enable_subagents defaults to False. Leaves <em>can&#39;t</em> spawn. That single flag is what prevents an unbounded recursion of agents-spawning-agents — the failure mode that turns a clever demo into a runaway cloud bill.</li></ul><h3>4. Step 2 — Spawning concurrent subagents</h3><p>Now the two leaves. Each is a <strong>static subagent</strong>: a pre-declared, single-responsibility config. The asymmetry is the point — the Text Analyzer gets text tools and <em>never</em> sees the image; the Image Verifier gets vision tools and <em>never</em> sees the full copy. That’s state isolation expressed as configuration.</p><pre># dispatch/subagents.py<br>from .harness import build_agent_config<br>def read_text(asset_id: str) -&gt; str:<br>    &quot;&quot;&quot;Return the raw marketing copy for the given asset id.&quot;&quot;&quot;<br>    from .store import load_copy<br>    return load_copy(asset_id)<br>def read_image(asset_id: str) -&gt; str:<br>    &quot;&quot;&quot;Return a file:// path to the asset&#39;s hero image for visual inspection.&quot;&quot;&quot;<br>    from .store import load_image_path<br>    return load_image_path(asset_id)<br>def text_analyzer_config():<br>    return build_agent_config(<br>        system_instructions=(<br>            &quot;You are a compliance TEXT analyst. Extract every concrete claim, &quot;<br>            &quot;list regulated/medical/financial terms, label tone in one word, and &quot;<br>            &quot;score compliance risk 0..1. You never see the image.&quot;<br>        ),<br>        tools=[read_text],<br>    )<br>def image_verifier_config():<br>    return build_agent_config(<br>        system_instructions=(<br>            &quot;You are a compliance IMAGE verifier. Report what the hero image &quot;<br>            &quot;actually depicts, whether it supports the claim handed to you, and &quot;<br>            &quot;whether manipulation is suspected. You never see the full copy.&quot;<br>        ),<br>        tools=[read_image],<br>    )</pre><p>Note there’s <strong>no orchestration logic inside the leaves</strong>. A leaf doesn’t know it has a sibling. That ignorance is a feature: it’s what keeps each context window small and each judgement independent.</p><p>The routing lives one level up, and it’s where concurrency happens. There are two legitimate ways to fan out in Antigravity, and a senior engineer should know both:</p><p><strong>Pattern A — harness-native delegation.</strong> Give the orchestrator enable_subagents=True and simply <em>ask</em> it to delegate. The SDK exposes a start_subagent tool; the model decides when to call it, the harness spins up an isolated child, runs it to completion, and returns its result as a tool result. This is the least code and the most flexible — ideal when the number and shape of subtasks isn&#39;t known ahead of time.</p><p><strong>Pattern B — programmatic dispatch.</strong> When you <em>do</em> know the shape — exactly two leaves, every time — you fan out explicitly with asyncio.gather for deterministic concurrency and clean typing. This is what a production audit pipeline wants, so it&#39;s what we&#39;ll build.</p><pre># dispatch/orchestrator.py<br>import asyncio<br>from typing import Type, TypeVar<br>from google.antigravity import Agent<br>from pydantic import BaseModel<br>from .schemas import AssetReport, ImageFindings, TextFindings<br>from .subagents import image_verifier_config, synthesizer_config, text_analyzer_config<br>T = TypeVar(&quot;T&quot;, bound=BaseModel)<br>async def run_once(config, prompt: str, schema: Type[T]) -&gt; T:<br>    &quot;&quot;&quot;One isolated agent, one typed turn, then tear down.&quot;&quot;&quot;<br>    async with Agent(config) as agent:<br>        resp = await agent.chat(prompt, response_schema=schema)<br>        return await resp.parsed()          # a validated Pydantic instance<br>async def dispatch(asset_id: str) -&gt; AssetReport:<br>    # FAN OUT - two isolated leaves, launched at the same instant.<br>    text_findings, image_findings = await asyncio.gather(<br>        run_once(<br>            text_analyzer_config(),<br>            f&quot;Analyze the copy for asset &#39;{asset_id}&#39;. Call read_text first.&quot;,<br>            TextFindings,<br>        ),<br>        run_once(<br>            image_verifier_config(),<br>            f&quot;Verify the hero image for asset &#39;{asset_id}&#39;. Call read_image &quot;<br>            f&quot;first. Judge whether it supports the headline claim.&quot;,<br>            ImageFindings,<br>        ),<br>    )<br>    ...  # fan-in follows in Step 3</pre><p>Line by line, this is the whole concurrency story:</p><ul><li><strong>run_once opens a fresh </strong><strong>Agent per call.</strong> async with Agent(config) is a full harness session with its own context window. Two calls to run_once = two isolated agents that share <em>nothing</em> but the guardrails baked into their configs. This is state isolation you can point at.</li><li><strong>await agent.chat(prompt, response_schema=schema)</strong> runs one turn and constrains the output to a schema. If the model returns malformed JSON, the harness retries the <em>model</em> — not your code. await resp.parsed() therefore hands back a real TextFindings / ImageFindings, never a string you have to defensively parse.</li><li><strong>asyncio.gather(...)</strong> is the fan-out. The two run_once coroutines are submitted together, so the Image Verifier inspects pixels <em>while</em> the Text Analyzer reads copy. Total latency is max(text, image), not text + image — and because they&#39;re isolated agents, there&#39;s no shared mutable state to race over.</li></ul><p>Run this against one asset and you’ll see it in the audit log: two trajectories, two distinct sets of tool calls, interleaved in real time. That interleaving is the compute you just recovered.</p><h3>5. Step 3 — Synthesizing parallel work into structured JSON</h3><p>Fan-out is half the pattern. The other half is <strong>fan-in</strong>: collapsing N isolated results back into one authoritative object. We do it with a third agent — a <em>synthesizer</em> — that reasons purely over the two typed findings and emits the only object allowed to leave the tree.</p><p>First, the contracts. Structured output is what makes the merge deterministic; without it you’re regex-ing model prose back into fields.</p><pre># dispatch/schemas.py<br>from enum import Enum<br>from pydantic import BaseModel, Field<br>class Verdict(str, Enum):<br>    PASS = &quot;pass&quot;; FLAG = &quot;flag&quot;; FAIL = &quot;fail&quot;<br>class TextFindings(BaseModel):<br>    claims: list[str]<br>    regulated_terms: list[str] = Field(default_factory=list)<br>    tone: str<br>    risk_score: float = Field(ge=0.0, le=1.0)<br>class ImageFindings(BaseModel):<br>    depicts: list[str]<br>    matches_copy: bool<br>    manipulation_suspected: bool = False<br>    notes: str = &quot;&quot;<br>class AssetReport(BaseModel):<br>    asset_id: str<br>    verdict: Verdict<br>    text: TextFindings<br>    image: ImageFindings<br>    rationale: str</pre><p>The synthesizer has <strong>no tools</strong>. It can’t read files or images — it can only reason over what it’s handed. That’s deliberate: it forces the verdict to be a function of the two findings and nothing else, which makes it auditable.</p><pre># dispatch/subagents.py  (continued)<br>def synthesizer_config():<br>    return build_agent_config(<br>        system_instructions=(<br>            &quot;You are a compliance ADJUDICATOR. You receive a TextFindings and an &quot;<br>            &quot;ImageFindings produced independently. Reconcile them into ONE verdict &quot;<br>            &quot;(pass/flag/fail) with a rationale citing BOTH modalities. If the image &quot;<br>            &quot;does not support a regulated claim, never return &#39;pass&#39;.&quot;<br>        ),<br>        tools=[],  # pure reasoning over its inputs — no tool access<br>    )</pre><p>And the fan-in that completes dispatch():</p><pre># dispatch/orchestrator.py  (continued)<br>    # FAN IN — a third agent reconciles the two typed results into one verdict.<br>    report = await run_once(<br>        synthesizer_config(),<br>        &quot;Reconcile these two independent findings into one AssetReport.\n\n&quot;<br>        f&quot;asset_id: {asset_id}\n&quot;<br>        f&quot;TextFindings: {text_findings.model_dump_json(indent=2)}\n&quot;<br>        f&quot;ImageFindings: {image_findings.model_dump_json(indent=2)}&quot;,<br>        AssetReport,<br>    )<br>    return report</pre><p>Why route the merge through an agent at all, instead of a few if statements? Because the interesting cases are <em>judgement</em>, not arithmetic. &quot;The copy claims &#39;clinically proven&#39; (regulated term, risk 0.8) and the image is a bare product shot that depicts nothing clinical (matches_copy=False)&quot; → <strong>FAIL</strong>, with a rationale a human reviewer can read. Encoding that reconciliation as a small, tool-less, structured-output agent keeps the logic legible and lets it handle the long tail you&#39;d never enumerate by hand. The output is a validated AssetReport:</p><pre>{<br>  &quot;asset_id&quot;: &quot;hero_01&quot;,<br>  &quot;verdict&quot;: &quot;fail&quot;,<br>  &quot;text&quot;: {<br>    &quot;claims&quot;: [&quot;erases wrinkles in 7 days&quot;, &quot;dermatologist-recommended&quot;],<br>    &quot;regulated_terms&quot;: [&quot;clinically proven&quot;, &quot;dermatologist-recommended&quot;],<br>    &quot;tone&quot;: &quot;clinical&quot;,<br>    &quot;risk_score&quot;: 0.86<br>  },<br>  &quot;image&quot;: {<br>    &quot;depicts&quot;: [&quot;gold serum bottle&quot;, &quot;neutral studio background&quot;],<br>    &quot;matches_copy&quot;: false,<br>    &quot;manipulation_suspected&quot;: false,<br>    &quot;notes&quot;: &quot;No before/after or clinical context supports the efficacy claim.&quot;<br>  },<br>  &quot;rationale&quot;: &quot;Copy makes a regulated efficacy claim (&#39;clinically proven&#39;) that the image does not substantiate; image is a bare product shot. Fails substantiation policy.&quot;<br>}</pre><p>To go from one asset to ten thousand, wrap dispatch in a governor. This is the compute-optimization lever in one function:</p><pre># dispatch/orchestrator.py  (continued)<br>async def dispatch_batch(asset_ids: list[str]) -&gt; list[AssetReport]:<br>    limit = int(os.getenv(&quot;DISPATCH_CONCURRENCY&quot;, &quot;8&quot;))<br>    sem = asyncio.Semaphore(limit)          # bound live model sessions<br>    async def _guarded(aid: str) -&gt; AssetReport:<br>        async with sem:                     # never more than `limit` trees at once<br>            return await dispatch(aid)<br>    return await asyncio.gather(*(_guarded(aid) for aid in asset_ids))</pre><p>The semaphore is the entire enterprise-throughput story on one line. Raise DISPATCH_CONCURRENCY to saturate your quota; lower it to be a polite tenant on shared infrastructure. Nothing else in the tree changes — isolation and typing already guarantee the assets can&#39;t interfere with each other.</p><h3>6. Step 4 — Enterprise scale: worktrees &amp; multi-repo context</h3><p>The tree is correct and fast. Two problems remain before it’s <em>safe</em> at enterprise scale, and Antigravity 2.0 solves both at the CLI layer.</p><h3>Isolated Git Worktree Mode</h3><p>An agent that edits code must never touch your live checkout. If it commits generated reports or opens a PR, that work belongs on its <strong>own branch, in its own working tree</strong>, quarantined from whatever you have staged. Antigravity’s CLI does this natively — scheduled runs and <em>New Worktree</em> subagents each get a fresh git worktree — which is what killed the &quot;dirty checkout&quot; class of bugs from the November 2025 release. In CI you make it explicit so it&#39;s reproducible:</p><pre>RUN_ID=&quot;audit-$(date +%Y%m%d-%H%M%S)&quot;<br>WORKTREE=&quot;../_worktrees/${RUN_ID}&quot;<br># Cut a throwaway worktree on a fresh branch. The agent&#39;s edits, the JSON<br># reports, and any PR live HERE - never in your main checkout.<br>git worktree add -b &quot;${RUN_ID}&quot; &quot;${WORKTREE}&quot; HEAD<br>trap &#39;git worktree remove --force &quot;${WORKTREE}&quot; 2&gt;/dev/null || true&#39; EXIT<br>cd &quot;${WORKTREE}&quot;</pre><p>Delete the worktree and the entire blast radius is gone. Ten scheduled audits can run overnight, each in its own tree, none of them able to corrupt another’s branch or your working copy. <strong>State isolation, applied to the filesystem</strong> — the same principle as the isolated context windows, one layer down.</p><h3>Multi-Repository Context</h3><p>Real compliance rules don’t live in the same repo as the marketing assets. The policy is one repository; the assets are another. You need an agent that can reason across both <strong>without copying either into the other’s prompt</strong>. agy --add-dir (repeatable) mounts multiple repos into a single agent workspace:</p><pre>agy --add-dir ../../compliance-policies \<br>    --add-dir ../../marketing-assets \<br>    --model &quot;${DISPATCH_MODEL:-gemini-3-pro}&quot; \<br>    --output-format json \<br>    -p &quot;Run \`python main.py\` to dispatch the multimodal audit over ./inbox.<br>        For every asset whose verdict is &#39;fail&#39;, cross-check it against the<br>        rules in ../../compliance-policies/RULES.md, then commit the JSON<br>        reports to reports/${RUN_ID}.json and open a PR titled &#39;[audit] ${RUN_ID}&#39;.&quot; \<br>  | tee &quot;reports-${RUN_ID}.json&quot;</pre><p>Unpacking the flags:</p><ul><li><strong>--add-dir (twice)</strong> is Multi-Repository Context: two repos, one workspace. The agent can read the policy and the assets side by side, but neither is pasted into the other&#39;s context — no cross-repo prompt pollution, the same discipline the subagent tree enforces in-memory.</li><li><strong>-p &quot;...&quot;</strong> is headless mode — a single non-interactive prompt, exactly what a Git hook or CI job needs. No TUI, no human in the loop.</li><li><strong>--output-format json</strong> makes the run pipeable: the structured trace on stdout flows straight into jq, a dashboard, or the next stage of your pipeline. This is how the terminal harness becomes a build step.</li></ul><p>Put the two together — worktree isolation plus multi-repo context — and you have an audit that runs on a schedule, reasons across your policy and your assets, quarantines every changeset it produces, and hands your reviewers a PR. That is what “systemic operator” means in practice: not a chatbot you babysit, but a governed unit of compute you deploy.</p><p>From inside the interactive TUI, the same primitives are a keystroke away: /agent dispatches a long-running subagent to the background while you keep prompting; /agents opens the manager to review its diff; Ctrl+J jumps to the next subagent awaiting approval; /schedule turns any prompt into a recurring, worktree-isolated job. The CLI and the SDK are two faces of one harness — prototype the topology interactively, then codify it in Python.</p><h3>7. Conclusion</h3><p>The monolithic agent was a useful lie — a way to get to a demo fast. Production retires it. The moment you have more than one responsibility and more than one asset, the right mental model isn’t “a smarter prompt,” it’s <strong>a topology of isolated operators</strong>: a tree whose leaves run concurrently, whose state never bleeds across boundaries, and whose compute you can govern with a single semaphore.</p><p>Antigravity 2.0 makes that topology first-class. The <strong>Shared Agent Harness</strong> gives you one place to encode policy and telemetry; <strong>subagents</strong> give you isolated, inheriting children; <strong>structured output</strong> makes fan-in deterministic; and the <strong>agy CLI</strong> extends the same isolation to the filesystem with worktrees and to your org with multi-repo context. You stop writing prompts and start designing systems — which is the job.</p><p>Build the tree once. Then point it at ten, or ten thousand, and let the harness do what harnesses do.</p><h3>Acknowledgement</h3><p>Google Cloud Credits are provided for this project. #AgenticArchitectSprint #Antigravity</p><h3>Further reading</h3><ul><li>Google Antigravity SDK (Python): <a href="https://github.com/google-antigravity/antigravity-sdk-python">https://github.com/google-antigravity/antigravity-sdk-python</a></li><li>Antigravity CLI (agy): <a href="https://github.com/google-antigravity/antigravity-cli">https://github.com/google-antigravity/antigravity-cli</a></li><li>Subagents docs: <a href="https://antigravity.google/docs/subagents">https://antigravity.google/docs/subagents</a></li><li>Developer guide (Karl Weinmeister): <a href="https://medium.com/google-cloud/google-antigravity-sdk-the-developer-guide-7770ad8a5f53">https://medium.com/google-cloud/google-antigravity-sdk-the-developer-guide-7770ad8a5f53</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f198c7e7f808" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developer-experts/multi-agent-dispatch-orchestrating-concurrent-subagents-for-multimodal-analysis-with-antigravity-2-f198c7e7f808">Multi-Agent Dispatch: Orchestrating Concurrent Subagents for Multimodal Analysis with Antigravity 2.</a> was originally published in <a href="https://medium.com/google-developer-experts">Google Developer Experts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build and Deploy an AI Agent with Gemini CLI and Google ADK: A]]></title>
            <link>https://medium.com/google-developer-experts/how-to-build-and-deploy-an-ai-agent-with-gemini-cli-and-google-adk-a-746a4064507b?source=rss-a32fdb99f2d------2</link>
            <guid isPermaLink="false">https://medium.com/p/746a4064507b</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[gemini-cli]]></category>
            <category><![CDATA[google-adk]]></category>
            <category><![CDATA[agent-development-kit]]></category>
            <dc:creator><![CDATA[Esther Irawati Setiawan]]></dc:creator>
            <pubDate>Sun, 31 May 2026 17:02:49 GMT</pubDate>
            <atom:updated>2026-05-31T17:02:49.387Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fAWJSKCea7KLWBIsCyn6Fg.png" /></figure><h3>Step-by-Step Vibe Coding Tutorial</h3><p><em>A simplified, hands-on walkthrough of Google’s “Agentverse: The Shadowblade’s Codex” codelab, focused on the build itself.</em></p><p><strong>📂 All the commands from this tutorial, in one quick-reference file: </strong><a href="https://github.com/estherirawati/agentverse"><strong>github.com/estherirawati/agentverse</strong></a></p><p>Google’s Agentverse codelab wraps everything in a fun fantasy theme. You’re a “Shadowblade,” your terminal is your “primary weapon,” and you’re fighting an entropy monster called <em>The Static</em>. The story is a great hook, and this tutorial keeps a light touch of it while focusing on <strong>what you build and how</strong>, so you can follow along step by step.</p><p>Underneath the theme, this is a genuinely good tour of the modern AI-agent workflow on Google Cloud: prompting code into existence with Gemini CLI, wiring the CLI to external tools, building a real autonomous agent with the Agent Development Kit (ADK), testing it, and shipping it to Cloud Run.</p><p>Here’s what the journey actually looks like.</p><h3>What you’re really building</h3><p>By the end you’ll have:</p><ul><li>A working command-line AI assistant (<strong>Gemini CLI</strong>) that can read your intent and act on it</li><li>A personal website generated almost entirely from a single prompt</li><li>The CLI connected to <strong>external tools</strong> through MCP servers (a self-hosted Git server and an image generator)</li><li>A real <strong>autonomous agent</strong> built with ADK that picks the right “weapon” tool for each task</li><li>An <strong>automated test suite</strong> for that agent</li><li>The agent <strong>deployed</strong> to Cloud Run as a live service</li></ul><p>The fantasy theme maps cleanly onto real engineering ideas, and the codelab makes this explicit in its “for non-gamers” sidebars. I’ll translate as we go.</p><h3>Step 0: Setup (the necessary part)</h3><p>You’ll need a <strong>personal Gmail account</strong> — corporate or school accounts won’t work because of the credit grant.</p><ol><li>Claim your free Google Cloud credit through the workshop link, sign in, accept the terms.</li><li>Open <strong>Cloud Shell</strong> at <a href="https://console.cloud.google.com/">console.cloud.google.com</a>, then click <strong>Open Editor</strong>.</li><li>Clone the starter code and run the setup script:</li></ol><pre>git clone https://github.com/weimeilin79/agentverse-developer.git<br>chmod +x ~/agentverse-developer/*.sh<br>cd ~/agentverse-developer<br>./init.sh    # press Enter to accept the default Project ID</pre><ol><li>Point your config at the new project and turn on the APIs you’ll need:</li></ol><pre>gcloud config set project $(cat ~/project_id.txt) --quiet<br>gcloud services enable compute.googleapis.com artifactregistry.googleapis.com \<br>  run.googleapis.com cloudbuild.googleapis.com aiplatform.googleapis.com \<br>  iam.googleapis.com cloudresourcemanager.googleapis.com<br>npm update -g @google/gemini-cli</pre><p>That’s the whole foundation. Now the interesting part.</p><h3>Step 1: Meet your “weapon” — Gemini CLI</h3><p>Gemini CLI is an open-source AI agent that lives in your terminal. It runs a <strong>reason-and-act (ReAct) loop</strong>: it reads your high-level request, breaks it into steps, picks the right tool, runs it, and checks the result.</p><p>Start it from a fresh folder:</p><pre>cd ~/agentverse-developer<br>mkdir playground<br>cd playground<br>gemini    # choose &quot;No&quot; when it asks to connect the Cloud Shell editor</pre><p>A few commands worth knowing right away, run <em>inside</em> the CLI — /help lists commands, /tools shows built-in abilities (ReadFile, WriteFile, GoogleSearch…), the ! prefix runs a normal shell command, and /memorystores context:</p><pre>/help<br>/tools<br>!ls -l<br>/memory add &quot;My name is [your name]. I&#39;m learning about AI agents.&quot;<br>/memory show</pre><p>That /memory feature is your first taste of <strong>context engineering</strong> — deliberately feeding the AI background so its output stays relevant.</p><p>Now the headline trick — generating code from a sentence:</p><pre>Write a Python script called hello.py that prints &quot;I built my first<br>AI-generated file&quot; and the current date.</pre><p>Then verify it actually worked:</p><pre>!ls<br>!cat hello.py<br>!python3 hello.py</pre><p>Exit anytime with <strong>Ctrl+C twice</strong>. That’s it — you just “vibe-coded.”</p><blockquote><strong>What “vibe coding” really means:</strong> instead of writing every line by hand, you describe your intent in plain language and let the assistant generate the code and config. The industry term is <strong>intent-driven development</strong>.</blockquote><h3>Step 2: Build a real website, then connect external tools</h3><p>Same idea, bigger ask. Inside the CLI:</p><pre>Create a personal profile website in the current folder. Dark theme,<br>electric blue accents. Two files: index.html and styles.css. Use flexbox<br>for a two-column layout. Include a placeholder spot for a profile picture.<br>Make the code clean and commented. Don&#39;t start any server.</pre><p>Exit the CLI and preview it:</p><pre>python -m http.server<br># Cloud Shell → Web Preview → port 8000, then Ctrl+C to stop</pre><h3>Giving the CLI hands: MCP servers</h3><p>So far Gemini CLI can only <em>talk</em> and edit local files. To let it take real action in other systems — Git, databases, APIs — you connect a <strong>Model Context Protocol (MCP) server</strong>. Think of it as a power cable between the AI’s “brain” and a tool’s “body.”</p><p>The codelab spins up <strong>Gitea</strong>, a self-hosted Git server, as the first tool:</p><pre>cd ~/agentverse-developer<br>./gitea.sh<br># Web Preview → port 3005 → log in with dev / dev</pre><p>Then register it in Gemini’s config so the CLI can see it:</p><pre>if [ ! -f ~/.gemini/settings.json ]; then<br>  echo &#39;{&quot;mcpServers&quot;:{&quot;gitea&quot;:{&quot;url&quot;:&quot;http://localhost:8085/sse&quot;}}}&#39; &gt; ~/.gemini/settings.json<br>else<br>  jq &#39;. * {&quot;mcpServers&quot;:{&quot;gitea&quot;:{&quot;url&quot;:&quot;http://localhost:8085/sse&quot;}}}&#39; ~/.gemini/settings.json &gt; ~/.gemini/settings.json.tmp &amp;&amp; mv ~/.gemini/settings.json.tmp ~/.gemini/settings.json<br>fi</pre><p>Relaunch the CLI from your project folder and confirm Gitea is wired up:</p><pre>cd ~/agentverse-developer/playground<br>gemini</pre><pre>/mcp</pre><p>With Gitea visible, you can now do version control with plain English:</p><pre>Create a new Gitea repository named &#39;my-profile&#39; with description<br>&#39;My first AI-built website&#39;. Don&#39;t add any content yet.</pre><pre>Using the Gitea tool, push index.html and styles.css to the<br>&#39;my-profile&#39; repository.</pre><pre>File an issue in the my-profile repo titled &quot;Profile image is missing&quot;.<br>Use the Gitea tool and the &#39;dev&#39; user account.</pre><pre>Close issue #1 in the my-profile repo. Use the &#39;dev&#39; user account.</pre><p>That’s the leap the codelab is really showing off: the AI stops being a chatbot and becomes an <strong>active participant in your workflow</strong> — creating repos, pushing commits, filing and closing issues.</p><blockquote><strong>Extensions vs. raw MCP:</strong> editing settings.json by hand is the &quot;raw&quot; way — useful for understanding the plumbing. In real life you&#39;d usually install an <strong>extension</strong> (gemini extensions install ...), which bundles the MCP server, custom slash-commands, and context into one installable package. The codelab uses this approach later for an image-generation extension called Nano Banana.</blockquote><h3>Step 3: Forge the actual agent with ADK</h3><p>This is the heart of the lab. You move from “AI helps me write code” to “I build an AI thing that runs on its own.” The framework is Google’s <strong>Agent Development Kit (ADK)</strong>.</p><h3>First, lay down the rules</h3><p>Before generating any agent code, you write a GEMINI.md file. The CLI loads it automatically and treats it as persistent, project-level instructions — coding standards, naming conventions, a persona, hard constraints:</p><pre>cd ~/agentverse-developer/shadowblade<br>. ~/agentverse-developer/set_env.sh</pre><pre>cat &lt;&lt; &#39;EOF&#39; &gt; GEMINI.md<br>### Coding Rules for This Project<br>- Use Python 3 with type hints on every function.<br>- Every function needs a docstring explaining what it does.<br>- Use snake_case for variables and functions, PascalCase for classes.<br>- Keep code clean and readable.<br>EOF</pre><p>This is <strong>context engineering</strong> layer two. The hierarchy is worth remembering:</p><ol><li><strong>~/.gemini/settings.json</strong> — global user settings</li><li><strong>GEMINI.md</strong> — project rules, always loaded</li><li><strong>Agent Skills</strong> (.gemini/skills/) — specialized knowledge loaded <em>only when relevant</em></li></ol><p>That third layer (progressive disclosure) is what keeps the agent fast: a repo can hold dozens of niche skills, but the AI only pulls the one that matches the task at hand.</p><h3>The agent itself</h3><p>The codelab has you generate agent.py from a design doc, then — because LLM output is unpredictable — swap in the known-good version:</p><pre>cp ~/agentverse-developer/working_code/agent.py ~/agentverse-developer/shadowblade/<br>cp ~/agentverse-developer/working_code/mcp_server.py ~/agentverse-developer/shadowblade/</pre><p>At its core, the agent is an LlmAgent with a model, a clear instruction block, and a list of tools wired up through an MCPToolset. The &quot;weapons&quot; it can wield are just @mcp.tool()-decorated Python functions in mcp_server.py — each one a small, focused tool with a descriptive docstring the model reads to decide when to use it.</p><h3>Run it</h3><pre>cd ~/agentverse-developer<br>python -m venv env<br>source env/bin/activate<br>pip install --upgrade pip<br>pip install -r shadowblade/requirements.txt<br>adk run shadowblade</pre><p>Then give it a command. The agent reads the “monster’s weakness,” picks the matching tool, and reports the outcome:</p><pre>We&#39;re stuck against &#39;Perfectionism&#39;. Its weakness is &#39;Elegant<br>Sufficiency&#39;. Break us out!</pre><pre>&#39;Dogma&#39; blocks our path. Its weakness is &#39;Revolutionary Rewrite&#39;. Take it down.</pre><p>Strip away the theme and this is exactly how a real support or ops agent works: read the situation, choose the correct internal tool, execute, summarize.</p><h3>Step 4: Test it (because an untested agent is a liability)</h3><p>Agents are harder to test than scripts because their behavior <em>emerges</em> from an LLM’s multi-step reasoning. You care about two things: did it produce the right final answer, <strong>and</strong> did it take the right path (use the right tools)?</p><p>ADK gives you two complementary methods:</p><p><strong>adk eval</strong> runs the agent against a set of predefined cases in a JSON &quot;evalset.&quot; Each case defines the input, the expected response, and — crucially — the expected tool_uses. A neat trick the lab teaches here is <strong>synthetic test data</strong>: you hand the AI one template case and ask it to generate dozens of varied ones, scaling your test coverage cheaply.</p><p><strong>pytest</strong> wraps the same AgentEvaluator in code, which is what makes it CI-ready:</p><pre>cp ~/agentverse-developer/working_code/test_agent_initiative.py ~/agentverse-developer/shadowblade/<br>source ~/agentverse-developer/env/bin/activate<br>cd ~/agentverse-developer<br>. ~/agentverse-developer/set_env.sh<br>pytest test_agent_initiative.py</pre><p>A passing run (1 passed) means the agent follows its protocol and is ready to drop into an automated pipeline. The scoring criteria split cleanly into tool_trajectory_avg_score (did it <em>do</em> the right thing) and response_match_score (did it <em>say</em> the right thing).</p><blockquote>The lab also introduces <strong>hooks</strong> — scripts that fire at specific points in the agent loop (BeforeTool, AfterTool, etc.) so you can validate, audit, or block actions during a run without touching the agent&#39;s code. Great for security and compliance gates.</blockquote><h3>Step 5: Deploy to Cloud Run, then clean up</h3><p>Build a container image and ship it as a live, autoscaling service:</p><pre>. ~/agentverse-developer/set_env.sh</pre><pre># create the artifact repo (ignore error if it already exists)<br>gcloud artifacts repositories create $REPO_NAME \<br>  --repository-format=docker \<br>  --location=$REGION \<br>  --description=&quot;Agent repo&quot; 2&gt;/dev/null || echo &quot;Already exists, moving on&quot;</pre><pre># grant the service account the roles it needs<br>for ROLE in artifactregistry.admin cloudbuild.builds.editor run.admin \<br>  iam.serviceAccountUser aiplatform.user logging.logWriter logging.viewer; do<br>    gcloud projects add-iam-policy-binding $PROJECT_ID \<br>      --member=&quot;serviceAccount:$SERVICE_ACCOUNT_NAME&quot; \<br>      --role=&quot;roles/$ROLE&quot; --quiet<br>done</pre><pre># prep the Dockerfile, then build and deploy<br>sed -i &#39;s|COPY ./shadowblade|COPY .|g&#39; ~/agentverse-developer/shadowblade/Dockerfile<br>sed -i &#39;s|COPY shadowblade|COPY .|g&#39; ~/agentverse-developer/shadowblade/Dockerfile</pre><pre>cd ~/agentverse-developer<br>gcloud builds submit ./shadowblade \<br>  --tag ${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}/my-agent:latest</pre><pre>gcloud run deploy my-agent \<br>  --image=${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}/my-agent:latest \<br>  --region=${REGION} \<br>  --allow-unauthenticated \<br>  --set-env-vars=&quot;A2A_HOST=0.0.0.0,A2A_PORT=8080,GOOGLE_GENAI_USE_VERTEXAI=TRUE&quot; \<br>  --min-instances=1 \<br>  --project=${PROJECT_ID}</pre><p>Visit your service URL with /.well-known/agent-card.json appended to confirm the agent is live and describing itself:</p><pre>https://my-agent-xxxxx-uc.a.run.app/.well-known/agent-card.json</pre><p><strong>Don’t skip cleanup</strong> — this is what stops a free-credit project from quietly billing you:</p><pre>. ~/agentverse-developer/set_env.sh<br>gcloud run services delete my-agent --region=${REGION} --quiet<br>gcloud artifacts repositories delete ${REPO_NAME} --location=${REGION} --quiet<br>rm -rf ~/agentverse-developer ~/.gemini<br>rm -f ~/project_id.txt</pre><h3>The five ideas worth keeping</h3><p>Setting the theme aside, here’s what this codelab actually teaches:</p><ol><li><strong>Intent-driven development</strong> — describe the outcome, let the AI generate the code, then <em>verify</em> it. The verification habit matters as much as the generation.</li><li><strong>MCP servers turn a chatbot into a doer</strong> — the moment the CLI can touch Git, a database, or an API, it becomes part of your real workflow.</li><li><strong>Context engineering is layered</strong> — global settings, persistent project rules (GEMINI.md), and on-demand skills. Good context beats clever one-off prompts.</li><li><strong>Agents need real testing</strong> — judge both the final answer <em>and</em> the path taken. Synthetic data lets you scale that testing cheaply.</li><li><strong>Ship it like software</strong> — containerize, set IAM roles, deploy to Cloud Run, and tear it down when you’re done.</li></ol><p>The story makes it memorable; the workflow makes it useful. Both are worth keeping.</p><p><em>Want to go through it yourself? The full codelab is “Agentverse: The Shadowblade’s Codex” on Google Codelabs, and the starter code lives at </em><a href="https://github.com/weimeilin79/agentverse-developer"><em>github.com/weimeilin79/agentverse-developer</em></a><em>. I’ve also collected every command from this tutorial in a quick-reference file on my own GitHub: </em><a href="https://github.com/estherirawati/agentverse"><em>github.com/estherirawati/agentverse</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=746a4064507b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developer-experts/how-to-build-and-deploy-an-ai-agent-with-gemini-cli-and-google-adk-a-746a4064507b">How to Build and Deploy an AI Agent with Gemini CLI and Google ADK: A</a> was originally published in <a href="https://medium.com/google-developer-experts">Google Developer Experts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Multimodal Indonesian Fake-News Detector with JAX, Flax, and Keras Kinetic on Cloud TPU]]></title>
            <link>https://medium.com/google-developer-experts/building-a-multimodal-indonesian-fake-news-detector-with-jax-flax-and-keras-kinetic-on-cloud-tpu-4b7f96c58b24?source=rss-a32fdb99f2d------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b7f96c58b24</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[kinetic-google]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[tpu]]></category>
            <dc:creator><![CDATA[Esther Irawati Setiawan]]></dc:creator>
            <pubDate>Thu, 30 Apr 2026 16:55:49 GMT</pubDate>
            <atom:updated>2026-05-27T03:42:07.060Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How I trained a Stance-Aware Cross-Encoder that classifies Indonesian news headlines against claims — starting on a free Colab TPU and scaling out to Cloud TPU v5p with a single</em><em>@kinetic.run() decorator</em></p><h3>Introduction</h3><p>Misinformation is one of the defining problems of the social-media era, and Indonesia has been hit particularly hard. Hoaks (the Indonesian shorthand for fake news) spread through WhatsApp groups and Twitter threads faster than any fact-checker can keep up. Most published research on automated fake-news detection focuses on English-language data, which leaves practitioners working with Bahasa Indonesia in a frustrating spot: the techniques exist, but the tooling and pre-trained models are scarce.</p><p>This article walks through building a real, working <strong>multimodal hoax detector</strong> for Indonesian news from scratch. The model takes two inputs — a <em>claim</em> (the original assertion, often from social media) and a <em>headline</em> (a news article headline that mentions the same topic) — and predicts whether the article <em>supports</em> the claim (for), <em>refutes</em> it (against), or merely <em>observes</em> it neutrally (observing).</p><p>The architecture is a <strong>Stance-Aware Cross-Encoder</strong>: a BiLSTM-style encoder for each input, multi-head self-attention, and a cross-attention layer that lets the claim and headline literally read each other before classification. Built end-to-end with <strong>JAX</strong> and <strong>Flax</strong>, trained on <strong>TPU</strong>.</p><p>The deployment story has two halves:</p><ol><li><strong>Free Colab TPU</strong> for prototyping — Google gives every Colab user free access to a v5e-1 TPU, which is enough to train this model end-to-end in under an hour at zero cost.</li><li><strong>Cloud TPU v5p via Keras Kinetic</strong> for serious training — when you outgrow Colab’s runtime limits, <a href="https://github.com/keras-team/kinetic">Keras Kinetic</a> lets you ship the same training function to a Cloud TPU pod with a single Python decorator. No Docker, no Kubernetes YAML, no SSH.</li></ol><p>By the end, you’ll have:</p><ul><li>A reusable Indonesian tokenizer and dataset loader</li><li>A 4-layer Transformer encoder with stance-aware cross-attention, written in Flax</li><li>A JIT-compiled training loop with optax and orbaxcheckpointing</li><li>A working predict.py that runs new claim-headline pairs through the trained model</li><li>The exact same code, deployed to Cloud TPU via@kinetic.run()</li></ul><p>Let’s get into it.</p><h3>Why This Problem Is Hard (and Interesting)</h3><p>Naive fake-news detectors look at one piece of text and try to classify it as “real” or “fake.” That’s both technically weak and ethically uncomfortable — a single text rarely carries enough signal, and the “true/false” framing assumes the model has access to ground truth it can’t possibly have.</p><p>The <strong>stance detection</strong> framing is much more honest. Given a claim and a related news article, the model doesn’t decide whether the claim is <em>true</em>; it decides whether <em>this particular article supports, refutes, or merely observes the claim</em>. That’s a question a model can actually answer, and it’s exactly the input a downstream fact-checker needs to make a final call.</p><p>Mathematically, the task is a 3-way classification over the<em>interaction</em> of two pieces of text. That word — interaction — is what makes the architecture interesting. You can’t just encode each side independently and concatenate. You need a layer that lets the claim attend to the headline and vice versa, so the model can pick up on subtle cues like negation (“Government denies…”), hedging (“alleged…”), or framing (“according to critics…”).</p><h3>Why JAX, Flax, Keras Kinetic, and TPU?</h3><ul><li><strong>JAX</strong> gives me NumPy-style code with automatic differentiation, JIT compilation via XLA, and transparent acceleration on CPU/GPU/TPU.</li><li><strong>Flax</strong> sits on top of JAX and lets me write neural networks as nn.Module classes. The model is dense in attention layers, and Flax keeps the parameter management clean.</li><li><strong>Optax</strong> for optimization (AdamW with linear warmup + cosine decay) and <strong>Orbax</strong> for checkpointing — both are part of the JAX ecosystem and JIT-friendly.</li><li><strong>Keras Kinetic</strong> is the deployment glue. One decorator turns a local Python function into a remote TPU job, with container caching, log streaming, and automatic GKE provisioning.</li><li><strong>TPU</strong> because the workload is dominated by attention matmuls — exactly what TPU systolic arrays are built for. Free on Colab (v5e-1), and Cloud TPU v5p when you need to scale up.</li></ul><h3>TPU vs GPU for This Workload</h3><p>A multimodal Transformer with cross-attention is one of the cleanest TPU workloads you can write. Here’s why, and where GPUs still hold their own.</p><p><strong>Hardware design</strong></p><ul><li><em>GPU (NVIDIA A100/H100):</em> general-purpose parallel processor, thousands of CUDA cores, great for arbitrary parallel computation.</li><li><em>TPU (v5e or v5p):</em> domain-specific accelerator built around a large systolic array (MXU) optimized for dense matrix multiplications.</li></ul><p><strong>What dominates the compute in this model</strong></p><ul><li>Multi-head self-attention: softmax(Q Kᵀ / √d) V — three big matmuls per head per layer.</li><li>Cross-attention between claim and headline: same shape, just with different inputs feeding Q vs K/V.</li><li>Feedforward blocks: two Dense layers with GELU between them.</li></ul><p>All of those are dense matmuls with predictable shapes. The TPU systolic array is purpose-built to chew through exactly this pattern at peak FLOPs. The XLA compiler fuses the entire train_step into a few kernels, and after the first compile, every step runs at full throughput.</p><p><strong>Where GPUs still win for stance detection / NLP</strong></p><ul><li>You’re doing token-level decoding with KV-cache and irregular generation lengths (we’re not — we’re doing classification).</li><li>You need a HuggingFace transformers model that’s only available as a PyTorch checkpoint (we’re training from scratch, so this doesn’t apply).</li><li>You want to iterate in a notebook with constant Python control flow that doesn’t JIT cleanly (Colab gives you both a TPU <em>and</em>a notebook, so you don’t have to choose).</li></ul><p><strong>Where TPUs win for stance detection / NLP</strong></p><ul><li>Fixed-length sequences (we pad to 64 tokens) → predictable shapes → great XLA compilation.</li><li>The whole train_step JITs into a single fused execution graph.</li><li>pmap / shard_map make multi-chip training a one-liner if you want to scale up.</li><li>Free on Colab, and Cloud TPU v5e is roughly $0.40/chip-hour on Spot.</li></ul><p><strong>Rule of thumb</strong></p><ul><li><strong>Quick prototyping in a notebook with Bahasa Indonesia data</strong> → free Colab TPU. (This article.)</li><li><strong>Iterative R&amp;D using HuggingFace PyTorch checkpoints</strong> → GPU.</li><li><strong>Production training with batchable, JAX-native workloads</strong> → Cloud TPU via Kinetic.</li><li><strong>You need to fine-tune a 7B+ Indonesian LLM</strong> → that’s a different article (and a different category — vLLM or Tunix).</li></ul><p>Now let’s actually build it.</p><h3>Project Architecture</h3><p>The pipeline is straightforward:</p><pre>datasetika.csv (Claim, Judul, Stance)<br>            │<br>            ▼<br>   ┌──────────────────┐<br>   │ IndonesianTokenizer │ whitespace + punctuation, vocab from corpus<br>   └──────────────────┘<br>            │<br>            ▼<br>   ┌──────────────────┐<br>   │  FakeNewsDataset │ stratified train/val/test split, JAX-ready arrays<br>   └──────────────────┘<br>            │<br>            ▼<br>   ┌──────────────────────────────────────┐<br>   │  FakeNewsDetector (Flax nn.Module)   │<br>   │  ┌──────────────┐  ┌──────────────┐  │<br>   │  │ Token+Pos    │  │ Token+Pos    │  │<br>   │  │ Embedding    │  │ Embedding    │  │<br>   │  │ (Claim)      │  │ (Headline)   │  │<br>   │  └──────┬───────┘  └──────┬───────┘  │<br>   │         ▼                 ▼          │<br>   │  ┌──────────────┐  ┌──────────────┐  │<br>   │  │ Transformer  │  │ Transformer  │  │<br>   │  │ × N layers   │  │ × N layers   │  │<br>   │  └──────┬───────┘  └──────┬───────┘  │<br>   │         └────────┬────────┘          │<br>   │                  ▼                    │<br>   │      ┌─────────────────────┐         │<br>   │      │ Stance Cross-Encoder│         │<br>   │      │ (cross-attention +  │         │<br>   │      │  diff &amp; product)    │         │<br>   │      └──────────┬──────────┘         │<br>   │                 ▼                    │<br>   │       Dense → Softmax (3 classes)    │<br>   └──────────────────────────────────────┘<br>            │<br>            ▼<br>   for / against / observing</pre><p>The whole thing sits inside one jax.jit-compiled train_step. Now let’s walk through each piece.</p><h3>Step 1 — Hardware Setup</h3><p>There are two paths. Pick whichever fits your stage of the project.</p><h3>Path A: Free Colab TPU (recommended for first run)</h3><ol><li>Open <a href="https://colab.research.google.com/">colab.research.google.com</a> and create a new notebook.</li><li>Click <strong>Runtime → Change runtime type</strong>.</li><li>Under <strong>Hardware accelerator</strong>, select <strong>v5e-1 TPU</strong>.</li><li>Click <strong>Save</strong>.</li><li>Verify in a cell:</li></ol><pre>import jax<br>print(jax.devices())<br># Expected: [TpuDevice(id=0, ...)]</pre><p>That’s it. You now have a free TPU v5e chip for the duration of your Colab session.</p><h3>Path B: Cloud TPU via Keras Kinetic (when you outgrow Colab)</h3><p>Colab is fantastic for prototyping but has runtime limits and gets bumped under load. When you’re ready to run multi-hour training jobs, switch to a real Cloud TPU. The traditional path means provisioning a TPU VM, SSHing in, installing dependencies, and uploading scripts — Kinetic skips all of that.</p><p>On your local laptop:</p><pre>pip install keras-kinetic<br>gcloud auth application-default login<br>gcloud config set project YOUR_PROJECT_ID<br>kinetic up --accelerator v5p-8 --yes</pre><p>The last command provisions a GKE Autopilot cluster with a TPU v5p-8 node pool. Takes a few minutes the first time, after which you don’t touch infrastructure again until tear-down.</p><p>I’ll show the actual @kinetic.run() deployment in Step 6. For now, let’s build the model.</p><h3>Step 2 — Indonesian Tokenizer and Dataset</h3><p>Bahasa Indonesia is morphologically less complex than, say, Turkish or Finnish, so a whitespace + punctuation tokenizer with a learned vocabulary works surprisingly well as a baseline. (For production, swap in IndoBERT — I’ll show how at the end of this section.)</p><p>The tokenizer reserves four special tokens, builds a frequency-ranked vocabulary from the training corpus, and emits(token_ids, attention_mask) pairs at a fixed length. Standard stuff, but with one subtlety: we tokenize <em>both</em> the Claim and Judul (headline) columns into a <em>shared</em> vocabulary so the embedding layer can pick up cross-input correlations.</p><pre>&quot;&quot;&quot;<br>Data Preprocessing for Indonesian Fake News Detection<br>Tokenizes Claim + Judul columns, encodes Stance labels.<br>Compatible with JAX/Flax training pipeline.<br>&quot;&quot;&quot;<br>import re<br>import numpy as np<br>import pandas as pd<br>from collections import Counter<br>from typing import List, Tuple, Dict<br>from sklearn.model_selection import train_test_split</pre><pre>LABEL_MAP   = {&quot;for&quot;: 0, &quot;against&quot;: 1, &quot;observing&quot;: 2}<br>ID_TO_LABEL = {v: k for k, v in LABEL_MAP.items()}<br></pre><pre>class IndonesianTokenizer:<br>    &quot;&quot;&quot;<br>    Lightweight whitespace + punctuation tokenizer for Indonesian text.</pre><pre>    For production, swap with:<br>        from transformers import AutoTokenizer<br>        tok = AutoTokenizer.from_pretrained(&quot;indobenchmark/indobert-base-p1&quot;)<br>    &quot;&quot;&quot;<br>    SPECIAL_TOKENS = {&quot;&lt;PAD&gt;&quot;: 0, &quot;&lt;UNK&gt;&quot;: 1, &quot;&lt;CLS&gt;&quot;: 2, &quot;&lt;SEP&gt;&quot;: 3}</pre><pre>    def __init__(self, vocab_size: int = 30_000, min_freq: int = 2):<br>        self.vocab_size = vocab_size<br>        self.min_freq   = min_freq<br>        self.word2id: Dict[str, int] = dict(self.SPECIAL_TOKENS)<br>        self.id2word: Dict[int, str] = {v: k for k, v in self.word2id.items()}</pre><pre>    @staticmethod<br>    def _clean(text: str) -&gt; str:<br>        text = text.lower()<br>        text = re.sub(r&quot;&lt;[^&gt;]+&gt;&quot;, &quot; &quot;, text)                   # strip HTML<br>        text = re.sub(r&quot;[^\w\s]&quot;, &quot; &quot;, text, flags=re.UNICODE) # keep alphanum<br>        text = re.sub(r&quot;\s+&quot;, &quot; &quot;, text).strip()<br>        return text</pre><pre>    @staticmethod<br>    def tokenize(text: str) -&gt; List[str]:<br>        return IndonesianTokenizer._clean(text).split()</pre><pre>    def build_vocab(self, texts: List[str]) -&gt; None:<br>        counter: Counter = Counter()<br>        for t in texts:<br>            counter.update(self.tokenize(t))<br>        top_tokens = [<br>            word for word, freq in counter.most_common()<br>            if freq &gt;= self.min_freq<br>        ][: self.vocab_size - len(self.SPECIAL_TOKENS)]<br>        for idx, word in enumerate(top_tokens, start=len(self.SPECIAL_TOKENS)):<br>            self.word2id[word] = idx<br>            self.id2word[idx]  = word<br>        print(f&quot;[Tokenizer] Vocabulary size: {len(self.word2id):,} tokens&quot;)</pre><pre>    def encode(self, text: str, max_len: int = 128, add_special: bool = True<br>              ) -&gt; Tuple[np.ndarray, np.ndarray]:<br>        tokens = self.tokenize(text)<br>        ids    = [self.word2id.get(t, self.SPECIAL_TOKENS[&quot;&lt;UNK&gt;&quot;]) for t in tokens]<br>        if add_special:<br>            ids = [self.SPECIAL_TOKENS[&quot;&lt;CLS&gt;&quot;]] + ids + [self.SPECIAL_TOKENS[&quot;&lt;SEP&gt;&quot;]]<br>        ids = ids[:max_len]<br>        pad_len = max_len - len(ids)<br>        mask    = [1] * len(ids) + [0] * pad_len<br>        ids     = ids + [self.SPECIAL_TOKENS[&quot;&lt;PAD&gt;&quot;]] * pad_len<br>        return np.array(ids, dtype=np.int32), np.array(mask, dtype=np.int32)</pre><pre>    def encode_batch(self, texts: List[str], max_len: int = 128<br>                    ) -&gt; Tuple[np.ndarray, np.ndarray]:<br>        pairs    = [self.encode(t, max_len) for t in texts]<br>        ids_arr  = np.stack([p[0] for p in pairs])<br>        mask_arr = np.stack([p[1] for p in pairs])<br>        return ids_arr, mask_arr</pre><p>The dataset class loads the CSV, builds the tokenizer on the combined Claim + Judul corpus, encodes both columns into fixed-length arrays, and produces stratified train/val/test splits:</p><pre>class FakeNewsDataset:<br>    &quot;&quot;&quot;<br>    Loads datasetika.csv and prepares JAX-compatible NumPy arrays.</pre><pre>    Expected columns:<br>        id, Claim, idStance, Judul, Stance<br>        Stance ∈ {for, against, observing}<br>    &quot;&quot;&quot;<br>    def __init__(<br>        self,<br>        csv_path: str,<br>        claim_max_len:    int = 64,<br>        headline_max_len: int = 64,<br>        vocab_size:       int = 30_000,<br>        test_size:        float = 0.15,<br>        val_size:         float = 0.10,<br>        random_seed:      int = 42,<br>    ):<br>        self.claim_max_len    = claim_max_len<br>        self.headline_max_len = headline_max_len</pre><pre>        df = pd.read_csv(csv_path)<br>        print(f&quot;[Dataset] Loaded {len(df):,} samples&quot;)<br>        print(f&quot;[Dataset] Stance distribution:\n{df[&#39;Stance&#39;].value_counts()}\n&quot;)</pre><pre>        self.tokenizer = IndonesianTokenizer(vocab_size)<br>        all_texts = df[&quot;Claim&quot;].tolist() + df[&quot;Judul&quot;].tolist()<br>        self.tokenizer.build_vocab(all_texts)</pre><pre>        claim_ids,    claim_masks    = self.tokenizer.encode_batch(<br>            df[&quot;Claim&quot;].tolist(),  claim_max_len)<br>        headline_ids, headline_masks = self.tokenizer.encode_batch(<br>            df[&quot;Judul&quot;].tolist(),  headline_max_len)<br>        labels = np.array([LABEL_MAP[s] for s in df[&quot;Stance&quot;]], dtype=np.int32)</pre><pre>        # Stratified train / val / test<br>        indices = np.arange(len(df))<br>        train_idx, test_idx = train_test_split(<br>            indices, test_size=test_size, stratify=labels, random_state=random_seed)<br>        train_idx, val_idx = train_test_split(<br>            train_idx,<br>            test_size=val_size / (1 - test_size),<br>            stratify=labels[train_idx],<br>            random_state=random_seed,<br>        )</pre><pre>        self.train = self._slice(claim_ids, claim_masks, headline_ids,<br>                                 headline_masks, labels, train_idx)<br>        self.val   = self._slice(claim_ids, claim_masks, headline_ids,<br>                                 headline_masks, labels, val_idx)<br>        self.test  = self._slice(claim_ids, claim_masks, headline_ids,<br>                                 headline_masks, labels, test_idx)</pre><pre>    @staticmethod<br>    def _slice(claim_ids, claim_masks, head_ids, head_masks, labels, idx):<br>        return {<br>            &quot;claim_ids&quot;:     claim_ids[idx],<br>            &quot;claim_masks&quot;:   claim_masks[idx],<br>            &quot;headline_ids&quot;:  head_ids[idx],<br>            &quot;headline_masks&quot;: head_masks[idx],<br>            &quot;labels&quot;:        labels[idx],<br>        }</pre><p><strong>Production swap:</strong> if you have GPU/TPU memory to spare, replace IndonesianTokenizer withAutoTokenizer.from_pretrained(&quot;indobenchmark/indobert-base-p1&quot;) and feed those token IDs into the same downstream model. The training accuracy jump is significant — IndoBERT was pre-trained on hundreds of millions of Indonesian tokens.</p><h3>Step 3 — The Multimodal Architecture (JAX + Flax)</h3><p>This is where it gets fun. The model has four logical pieces stacked in a Flax nn.Module:</p><ol><li><strong>Token + positional embedding</strong> for both inputs</li><li><strong>Transformer encoder stack</strong> (multi-head self-attention + feedforward) for each input independently</li><li><strong>Stance-aware cross-encoder</strong> that lets claim and headline attend to each other</li><li><strong>Classification head</strong> that produces logits over {for, against, observing}</li></ol><h3>3.1 — Token and Positional Embedding</h3><pre>import jax<br>import jax.numpy as jnp<br>import flax.linen as nn<br>from typing import Tuple<br></pre><pre>class TokenEmbedding(nn.Module):<br>    &quot;&quot;&quot;Learnable token + positional embeddings.&quot;&quot;&quot;<br>    vocab_size: int<br>    embed_dim: int<br>    max_len: int = 256<br>    dropout_rate: float = 0.1</pre><pre>    @nn.compact<br>    def __call__(self, token_ids: jnp.ndarray, train: bool = False) -&gt; jnp.ndarray:<br>        tok_emb = nn.Embed(self.vocab_size, self.embed_dim)(token_ids)  # (B, L, D)<br>        pos = jnp.arange(token_ids.shape[1])[None, :]                   # (1, L)<br>        pos_emb = nn.Embed(self.max_len, self.embed_dim)(pos)           # (1, L, D)<br>        x = tok_emb + pos_emb<br>        x = nn.LayerNorm()(x)<br>        x = nn.Dropout(self.dropout_rate, deterministic=not train)(x)<br>        return x</pre><h3>3.2 — Multi-Head Attention</h3><p>A clean, from-scratch multi-head attention. Yes, Flax hasnn.MultiHeadDotProductAttention built in, but writing it explicitly keeps the article educational and lets you see exactly what the attention mask is doing:</p><pre>class MultiHeadAttention(nn.Module):<br>    num_heads: int<br>    head_dim: int<br>    dropout_rate: float = 0.1</pre><pre>    @nn.compact<br>    def __call__(self, q: jnp.ndarray, k: jnp.ndarray, v: jnp.ndarray,<br>                 mask: jnp.ndarray = None, train: bool = False) -&gt; jnp.ndarray:<br>        B, Lq, D = q.shape<br>        Lk = k.shape[1]<br>        H, Dh = self.num_heads, self.head_dim</pre><pre>        Q = nn.Dense(H * Dh)(q).reshape(B, Lq, H, Dh).transpose(0, 2, 1, 3)<br>        K = nn.Dense(H * Dh)(k).reshape(B, Lk, H, Dh).transpose(0, 2, 1, 3)<br>        V = nn.Dense(H * Dh)(v).reshape(B, Lk, H, Dh).transpose(0, 2, 1, 3)</pre><pre>        scores = jnp.einsum(&quot;bhqd,bhkd-&gt;bhqk&quot;, Q, K) / jnp.sqrt(Dh)<br>        if mask is not None:<br>            # mask: (B, Lk) -&gt; (B, 1, 1, Lk)<br>            mask = mask[:, None, None, :]<br>            scores = jnp.where(mask == 0, -1e9, scores)</pre><pre>        attn = jax.nn.softmax(scores, axis=-1)<br>        attn = nn.Dropout(self.dropout_rate, deterministic=not train)(attn)<br>        out  = jnp.einsum(&quot;bhqk,bhkd-&gt;bhqd&quot;, attn, V)<br>        out  = out.transpose(0, 2, 1, 3).reshape(B, Lq, H * Dh)<br>        out  = nn.Dense(D)(out)<br>        return out</pre><h3>3.3 — Transformer Block</h3><p>Standard pre-norm Transformer block with a feedforward expansion factor of 4:</p><pre>class TransformerBlock(nn.Module):<br>    num_heads: int<br>    head_dim: int<br>    ff_dim: int<br>    dropout_rate: float = 0.1</pre><pre>    @nn.compact<br>    def __call__(self, x: jnp.ndarray, mask: jnp.ndarray = None,<br>                 train: bool = False) -&gt; jnp.ndarray:<br>        # Self-attention sublayer<br>        h = nn.LayerNorm()(x)<br>        h = MultiHeadAttention(self.num_heads, self.head_dim, self.dropout_rate)(<br>            h, h, h, mask=mask, train=train)<br>        h = nn.Dropout(self.dropout_rate, deterministic=not train)(h)<br>        x = x + h</pre><pre>        # Feedforward sublayer<br>        h = nn.LayerNorm()(x)<br>        h = nn.Dense(self.ff_dim)(h)<br>        h = nn.gelu(h)<br>        h = nn.Dense(x.shape[-1])(h)<br>        h = nn.Dropout(self.dropout_rate, deterministic=not train)(h)<br>        x = x + h<br>        return x</pre><h3>3.4 — Stance Cross-Encoder (the Interesting Part)</h3><p>This is what makes the architecture <em>multimodal-aware</em> rather than just two encoders bolted together. After the claim and headline are independently encoded, the cross-encoder lets each one attend to the other, then extracts an interaction vector by concatenating the difference and element-wise product of the two pooled representations:</p><pre>class StanceCrossEncoder(nn.Module):<br>    &quot;&quot;&quot;Cross-attention between claim and headline + interaction features.&quot;&quot;&quot;<br>    num_heads: int<br>    head_dim: int<br>    dropout_rate: float = 0.1</pre><pre>    @nn.compact<br>    def __call__(self, claim: jnp.ndarray, headline: jnp.ndarray,<br>                 claim_mask: jnp.ndarray, headline_mask: jnp.ndarray,<br>                 train: bool = False) -&gt; jnp.ndarray:<br>        # Claim attends to headline<br>        claim_attended = MultiHeadAttention(<br>            self.num_heads, self.head_dim, self.dropout_rate<br>        )(claim, headline, headline, mask=headline_mask, train=train)</pre><pre>        # Headline attends to claim<br>        head_attended = MultiHeadAttention(<br>            self.num_heads, self.head_dim, self.dropout_rate<br>        )(headline, claim, claim, mask=claim_mask, train=train)</pre><pre>        # Mean-pool with mask<br>        def masked_mean(x, m):<br>            m = m[..., None].astype(x.dtype)<br>            return (x * m).sum(axis=1) / jnp.maximum(m.sum(axis=1), 1.0)</pre><pre>        c = masked_mean(claim_attended, claim_mask)     # (B, D)<br>        h = masked_mean(head_attended,  headline_mask)  # (B, D)</pre><pre>        # Stance-aware interaction features<br>        interaction = jnp.concatenate(<br>            [c, h, jnp.abs(c - h), c * h], axis=-1<br>        )  # (B, 4D)<br>        return interaction</pre><p>The four components — c, h, |c - h|, c * h — capture different aspects of how the two inputs relate. This is a classic NLI (natural language inference) trick and it works well here: |c - h| flags semantic distance, while c * h highlights agreement.</p><h3>3.5 — Putting It All Together</h3><pre>class FakeNewsDetector(nn.Module):<br>    vocab_size: int<br>    embed_dim: int = 256<br>    num_heads: int = 8<br>    head_dim: int = 32<br>    ff_dim: int = 1024<br>    num_layers: int = 4<br>    dropout_rate: float = 0.1<br>    num_classes: int = 3</pre><pre>    @nn.compact<br>    def __call__(self, claim_ids, claim_mask, headline_ids, headline_mask,<br>                 train: bool = False):<br>        # Embed both inputs<br>        claim_x    = TokenEmbedding(self.vocab_size, self.embed_dim,<br>                                    dropout_rate=self.dropout_rate)(claim_ids, train)<br>        headline_x = TokenEmbedding(self.vocab_size, self.embed_dim,<br>                                    dropout_rate=self.dropout_rate)(headline_ids, train)</pre><pre>        # Independent Transformer stacks<br>        for _ in range(self.num_layers):<br>            claim_x = TransformerBlock(self.num_heads, self.head_dim,<br>                                       self.ff_dim, self.dropout_rate)(<br>                claim_x, mask=claim_mask, train=train)<br>            headline_x = TransformerBlock(self.num_heads, self.head_dim,<br>                                          self.ff_dim, self.dropout_rate)(<br>                headline_x, mask=headline_mask, train=train)</pre><pre>        # Cross-encoder interaction<br>        interaction = StanceCrossEncoder(self.num_heads, self.head_dim,<br>                                         self.dropout_rate)(<br>            claim_x, headline_x, claim_mask, headline_mask, train=train)</pre><pre>        # Classification head<br>        h = nn.Dense(self.embed_dim)(interaction)<br>        h = nn.gelu(h)<br>        h = nn.Dropout(self.dropout_rate, deterministic=not train)(h)<br>        logits = nn.Dense(self.num_classes)(h)<br>        return logits</pre><p>A 4-layer encoder stack with embed_dim=256, num_heads=8, and ff_dim=1024 lands at around 7M parameters — small enough to train quickly on a free Colab TPU, large enough to learn meaningful stance representations.</p><h3>Step 4 — Training Pipeline (TPU-Optimized)</h3><p>The training loop is a standard Optax + JAX pattern, with two TPU-specific touches: everything is jax.jit-compiled, and we use Orbax for checkpointing because it handles the JAX pytreeparameter format natively.</p><h3>4.1 — Train State and Loss</h3><pre>import optax<br>from flax.training import train_state<br></pre><pre>def create_train_state(rng, model, learning_rate, weight_decay,<br>                       claim_len, head_len, num_warmup_steps, num_total_steps):<br>    # Dummy inputs to initialize parameters<br>    dummy_claim_ids   = jnp.ones((1, claim_len), dtype=jnp.int32)<br>    dummy_claim_mask  = jnp.ones((1, claim_len), dtype=jnp.int32)<br>    dummy_head_ids    = jnp.ones((1, head_len),  dtype=jnp.int32)<br>    dummy_head_mask   = jnp.ones((1, head_len),  dtype=jnp.int32)</pre><pre>    params = model.init(<br>        rng, dummy_claim_ids, dummy_claim_mask,<br>        dummy_head_ids, dummy_head_mask, train=False,<br>    )[&quot;params&quot;]</pre><pre>    # Linear warmup + cosine decay<br>    schedule = optax.warmup_cosine_decay_schedule(<br>        init_value=0.0, peak_value=learning_rate,<br>        warmup_steps=num_warmup_steps,<br>        decay_steps=num_total_steps - num_warmup_steps,<br>        end_value=learning_rate * 0.1,<br>    )<br>    optimizer = optax.adamw(schedule, weight_decay=weight_decay)</pre><pre>    return train_state.TrainState.create(<br>        apply_fn=model.apply, params=params, tx=optimizer)<br></pre><pre>def cross_entropy_loss(logits, labels):<br>    one_hot = jax.nn.one_hot(labels, num_classes=logits.shape[-1])<br>    return -jnp.mean(jnp.sum(one_hot * jax.nn.log_softmax(logits), axis=-1))</pre><h3>4.2 — JIT-Compiled Train and Eval Steps</h3><pre>@jax.jit<br>def train_step(state, batch, dropout_rng):<br>    def loss_fn(params):<br>        logits = state.apply_fn(<br>            {&quot;params&quot;: params},<br>            batch[&quot;claim_ids&quot;], batch[&quot;claim_masks&quot;],<br>            batch[&quot;headline_ids&quot;], batch[&quot;headline_masks&quot;],<br>            train=True,<br>            rngs={&quot;dropout&quot;: dropout_rng},<br>        )<br>        loss = cross_entropy_loss(logits, batch[&quot;labels&quot;])<br>        return loss, logits</pre><pre>    (loss, logits), grads = jax.value_and_grad(loss_fn, has_aux=True)(state.params)<br>    state = state.apply_gradients(grads=grads)<br>    accuracy = (jnp.argmax(logits, axis=-1) == batch[&quot;labels&quot;]).mean()<br>    return state, loss, accuracy<br></pre><pre>@jax.jit<br>def eval_step(state, batch):<br>    logits = state.apply_fn(<br>        {&quot;params&quot;: state.params},<br>        batch[&quot;claim_ids&quot;], batch[&quot;claim_masks&quot;],<br>        batch[&quot;headline_ids&quot;], batch[&quot;headline_masks&quot;],<br>        train=False,<br>    )<br>    loss = cross_entropy_loss(logits, batch[&quot;labels&quot;])<br>    accuracy = (jnp.argmax(logits, axis=-1) == batch[&quot;labels&quot;]).mean()<br>    return loss, accuracy</pre><p>The @jax.jit is doing a lot of work here. On the first call, XLA compiles the entire forward pass, loss computation, gradient calculation, and parameter update into a single fused execution graph. After that first ~30-second compile, every subsequent step runs at peak TPU throughput.</p><h3>4.3 — The Training Loop</h3><pre>import orbax.checkpoint as ocp<br>from pathlib import Path<br>import time<br></pre><pre>def train(args):<br>    # Load and prepare data<br>    ds = FakeNewsDataset(<br>        args.data_path,<br>        claim_max_len=args.claim_len,<br>        headline_max_len=args.head_len,<br>    )</pre><pre>    # Create model and train state<br>    model = FakeNewsDetector(<br>        vocab_size=len(ds.tokenizer.word2id),<br>        embed_dim=args.embed_dim,<br>        num_heads=args.num_heads,<br>    )<br>    rng = jax.random.PRNGKey(0)<br>    init_rng, train_rng = jax.random.split(rng)</pre><pre>    steps_per_epoch  = len(ds.train[&quot;labels&quot;]) // args.batch_size<br>    num_total_steps  = steps_per_epoch * args.epochs<br>    num_warmup_steps = int(0.1 * num_total_steps)</pre><pre>    state = create_train_state(<br>        init_rng, model, args.lr, weight_decay=0.01,<br>        claim_len=args.claim_len, head_len=args.head_len,<br>        num_warmup_steps=num_warmup_steps,<br>        num_total_steps=num_total_steps,<br>    )</pre><pre>    # Orbax checkpointer<br>    ckpt_dir = Path(args.output_dir).resolve()<br>    ckpt_dir.mkdir(parents=True, exist_ok=True)<br>    checkpointer = ocp.PyTreeCheckpointer()<br>    best_val_acc = 0.0</pre><pre>    for epoch in range(args.epochs):<br>        epoch_start = time.time()<br>        train_rng, shuffle_rng = jax.random.split(train_rng)</pre><pre>        # Shuffle training indices<br>        perm = jax.random.permutation(shuffle_rng, len(ds.train[&quot;labels&quot;]))</pre><pre>        # Training pass<br>        train_loss, train_acc = 0.0, 0.0<br>        for step in range(steps_per_epoch):<br>            batch_idx = perm[step * args.batch_size:(step + 1) * args.batch_size]<br>            batch = {k: v[batch_idx] for k, v in ds.train.items()}<br>            train_rng, dropout_rng = jax.random.split(train_rng)<br>            state, loss, acc = train_step(state, batch, dropout_rng)<br>            train_loss += float(loss)<br>            train_acc  += float(acc)<br>        train_loss /= steps_per_epoch<br>        train_acc  /= steps_per_epoch</pre><pre>        # Validation pass<br>        val_loss, val_acc = 0.0, 0.0<br>        val_steps = len(ds.val[&quot;labels&quot;]) // args.batch_size<br>        for step in range(val_steps):<br>            batch = {k: v[step * args.batch_size:(step + 1) * args.batch_size]<br>                     for k, v in ds.val.items()}<br>            loss, acc = eval_step(state, batch)<br>            val_loss += float(loss)<br>            val_acc  += float(acc)<br>        val_loss /= val_steps<br>        val_acc  /= val_steps</pre><pre>        elapsed = time.time() - epoch_start<br>        print(f&quot;Epoch {epoch+1:3d}/{args.epochs} | &quot;<br>              f&quot;train loss {train_loss:.4f} acc {train_acc:.4f} | &quot;<br>              f&quot;val loss {val_loss:.4f} acc {val_acc:.4f} | &quot;<br>              f&quot;{elapsed:.1f}s&quot;)</pre><pre>        # Save best checkpoint<br>        if val_acc &gt; best_val_acc:<br>            best_val_acc = val_acc<br>            checkpointer.save(ckpt_dir / &quot;best&quot;, state.params, force=True)<br>            print(f&quot;  ✓ Saved checkpoint (val_acc={val_acc:.4f})&quot;)</pre><h3>4.4 — Running on Free Colab TPU</h3><p>In a Colab cell:</p><pre>class Args:<br>    data_path  = &quot;datasetika.csv&quot;<br>    output_dir = &quot;checkpoints&quot;<br>    epochs     = 20<br>    batch_size = 64<br>    lr         = 2e-4<br>    claim_len  = 64<br>    head_len   = 64<br>    embed_dim  = 256<br>    num_heads  = 8</pre><pre>train(Args)</pre><p>Sample output on Colab v5e-1:</p><pre>[Dataset] Loaded 12,847 samples<br>[Dataset] Stance distribution:<br>observing    5,932<br>for          4,221<br>against      2,694</pre><pre>[Tokenizer] Vocabulary size: 18,734 tokens<br>Epoch   1/20 | train loss 0.9932 acc 0.5414 | val loss 0.8821 acc 0.6102 | 41.3s<br>Epoch   2/20 | train loss 0.7881 acc 0.6587 | val loss 0.7024 acc 0.7011 | 22.8s<br>Epoch   3/20 | train loss 0.6014 acc 0.7321 | val loss 0.6398 acc 0.7314 | 22.5s<br>...<br>Epoch  20/20 | train loss 0.2741 acc 0.9001 | val loss 0.4012 acc 0.8412 | 22.6s</pre><p>The first epoch is slower because XLA is compiling the train_step. From epoch 2 onward, every epoch takes ~22 seconds on a free Colab v5e-1. Total wall time: about 8 minutes for 20 epochs.</p><h3>Step 5 — Inference on New Claims</h3><p>Once trained, classifying a new claim-headline pair is a one-shot forward pass. Load the tokenizer and checkpointed parameters, encode the inputs, run the model:</p><pre>import pickle<br>import orbax.checkpoint as ocp<br></pre><pre>def load_model(checkpoint_path: str, tokenizer_path: str,<br>               claim_len: int = 64, headline_len: int = 64,<br>               embed_dim: int = 256, num_heads: int = 8, num_layers: int = 4):<br>    with open(tokenizer_path, &quot;rb&quot;) as f:<br>        tokenizer: IndonesianTokenizer = pickle.load(f)</pre><pre>    model = FakeNewsDetector(<br>        vocab_size=len(tokenizer.word2id),<br>        embed_dim=embed_dim,<br>        num_heads=num_heads,<br>        num_layers=num_layers,<br>    )</pre><pre>    checkpointer = ocp.PyTreeCheckpointer()<br>    params = checkpointer.restore(checkpoint_path)<br>    return model, params, tokenizer<br></pre><pre>def predict(model, params, tokenizer, claim: str, headline: str,<br>            claim_len: int = 64, headline_len: int = 64):<br>    claim_ids,    claim_mask    = tokenizer.encode(claim,    claim_len)<br>    headline_ids, headline_mask = tokenizer.encode(headline, headline_len)</pre><pre>    # Add batch dimension<br>    claim_ids     = claim_ids[None, :]<br>    claim_mask    = claim_mask[None, :]<br>    headline_ids  = headline_ids[None, :]<br>    headline_mask = headline_mask[None, :]</pre><pre>    logits = model.apply(<br>        {&quot;params&quot;: params},<br>        claim_ids, claim_mask, headline_ids, headline_mask, train=False,<br>    )<br>    probs    = jax.nn.softmax(logits, axis=-1)[0]<br>    pred_idx = int(jnp.argmax(probs))<br>    return ID_TO_LABEL[pred_idx], {ID_TO_LABEL[i]: float(probs[i]) for i in range(3)}<br></pre><pre># Example<br>model, params, tokenizer = load_model(&quot;checkpoints/best&quot;, &quot;tokenizer.pkl&quot;)</pre><pre>examples = [<br>    {&quot;claim&quot;:    &quot;Pemerintah resmi menaikkan harga BBM bulan depan&quot;,<br>     &quot;headline&quot;: &quot;Kementerian ESDM bantah rencana kenaikan harga BBM&quot;,<br>     &quot;expected&quot;: &quot;against&quot;},<br>    {&quot;claim&quot;:    &quot;Vaksin baru efektif 95 persen mencegah penularan&quot;,<br>     &quot;headline&quot;: &quot;Studi terbaru: vaksin tunjukkan efikasi 95 persen pada uji klinis fase 3&quot;,<br>     &quot;expected&quot;: &quot;for&quot;},<br>    {&quot;claim&quot;:    &quot;Presiden umumkan kebijakan baru ekonomi&quot;,<br>     &quot;headline&quot;: &quot;Pengamat: masih perlu dikaji lebih lanjut dampaknya&quot;,<br>     &quot;expected&quot;: &quot;observing&quot;},<br>]</pre><pre>for ex in examples:<br>    pred, probs = predict(model, params, tokenizer, ex[&quot;claim&quot;], ex[&quot;headline&quot;])<br>    print(f&quot;\nClaim:    {ex[&#39;claim&#39;]}&quot;)<br>    print(f&quot;Headline: {ex[&#39;headline&#39;]}&quot;)<br>    print(f&quot;Predicted: {pred} (expected: {ex[&#39;expected&#39;]})&quot;)<br>    print(f&quot;Probabilities: {probs}&quot;)</pre><h3>Step 6 — Scaling Up: Deploying to Cloud TPU with Keras Kinetic</h3><p>Free Colab TPU is great for prototyping. But Colab sessions have runtime limits (12h max, often less under load), no persistent storage, and you can’t run multiple experiments in parallel. When you want to do real hyperparameter sweeps or train on a larger dataset, it’s time to graduate to Cloud TPU.</p><p>The traditional path means provisioning a TPU VM, SSHing in, installing dependencies, uploading scripts, and managing the deployment yourself. Keras Kinetic skips all of that. You decorate your training function with @kinetic.run(accelerator=&quot;v5p-8&quot;) and call it from your laptop. Kinetic packages the code, builds a container, provisions a GKE cluster with TPUs attached, runs the function on the remote pod, and streams logs back to your local terminal.</p><h3>6.1 — Wrap Your Training Function</h3><p>Save the existing tokenizer, dataset, model, and training code asfakenews.py. Then create train_kinetic.py:</p><pre>import kinetic<br></pre><pre>@kinetic.run(<br>    accelerator=&quot;v5p-8&quot;,<br>    requirements=[<br>        &quot;jax[tpu]&quot;,<br>        &quot;flax&quot;,<br>        &quot;optax&quot;,<br>        &quot;orbax-checkpoint&quot;,<br>        &quot;numpy&quot;,<br>        &quot;pandas&quot;,<br>        &quot;scikit-learn&quot;,<br>    ],<br>)<br>def train_remote(data_gcs_path: str, epochs: int = 50, batch_size: int = 128,<br>                 lr: float = 2e-4):<br>    # CRITICAL: imports happen *inside* the function. The body runs on<br>    # the remote TPU pod, so imports resolve against the pod&#39;s installed<br>    # packages, not your laptop&#39;s.<br>    import os<br>    os.environ[&quot;JAX_PLATFORMS&quot;] = &quot;tpu&quot;</pre><pre>    from fakenews import FakeNewsDataset, FakeNewsDetector, train</pre><pre>    class Args:<br>        data_path  = data_gcs_path<br>        output_dir = &quot;/tmp/checkpoints&quot;<br>        epochs     = epochs<br>        batch_size = batch_size<br>        lr         = lr<br>        claim_len  = 64<br>        head_len   = 64<br>        embed_dim  = 256<br>        num_heads  = 8</pre><pre>    train(Args)<br>    return {&quot;status&quot;: &quot;completed&quot;, &quot;epochs&quot;: epochs}<br></pre><pre>if __name__ == &quot;__main__&quot;:<br>    # This runs on the *remote* TPU but feels like a local function call.<br>    result = train_remote(<br>        data_gcs_path=&quot;gs://my-bucket/datasetika.csv&quot;,<br>        epochs=50,<br>        batch_size=128,<br>    )<br>    print(f&quot;Training finished: {result}&quot;)</pre><p>Three things worth highlighting:</p><ul><li><strong>All imports go inside the function</strong> — the body runs on the remote pod, so import jax needs to resolve against the pod’s jax[tpu] package, not your laptop’s.</li><li><strong>requirements=[...] is your remote requirements.txt</strong> — Kinetic uses it to build the container image on the first run. None of these need to be installed locally.</li><li><strong>Data lives in GCS</strong> — your dataset goes to a Google Cloud Storage bucket, and the function reads it from gs://.... The TPU pod has automatic GCS access via its service account.</li></ul><h3>6.2 — Launch from Your Laptop</h3><pre>python train_kinetic.py</pre><p>First run takes ~5 minutes for the container build. Subsequent runs with unchanged dependencies start in under a minute. You’ll see remote logs streamed to your terminal:</p><pre>Shipping to TPU via Kinetic...<br>[Stage 1/4] Preflight &amp; packaging...<br>[Stage 2/4] Building container image (5m, cached after this run)...<br>[Stage 3/4] Submitting job to GKE cluster...<br>[Stage 4/4] Executing on TPU v5p-8...<br>[remote] [Dataset] Loaded 12,847 samples<br>[remote] [Tokenizer] Vocabulary size: 18,734 tokens<br>[remote] Epoch   1/50 | train loss 0.9912 acc 0.5421 | val loss 0.8810 acc 0.6112 | 12.8s<br>[remote] Epoch   2/50 | train loss 0.7821 acc 0.6601 | val loss 0.6987 acc 0.7022 | 4.2s<br>...<br>[remote] Epoch  50/50 | train loss 0.1912 acc 0.9301 | val loss 0.3811 acc 0.8612 | 4.3s<br>Job complete. Streaming results to local...<br>Training finished: {&#39;status&#39;: &#39;completed&#39;, &#39;epochs&#39;: 50}</pre><p>The v5p-8 has 8 chips vs Colab’s 1, so per-epoch time drops from 22s to ~4s — about 5× faster, which matters when you’re running 50+ epochs or sweeping hyperparameters.</p><h3>6.3 — Tear Down</h3><p>The GKE cluster’s control plane costs money even when no TPU nodes are active. Always shut it down when you’re done:</p><pre>kinetic down --yes</pre><h3>Training Configuration Summary</h3><ul><li><strong>Architecture:</strong> 4-layer Transformer + Stance Cross-Encoder</li><li><strong>Embedding dim:</strong> 256</li><li><strong>Attention heads:</strong> 8 (head_dim 32)</li><li><strong>Feedforward dim:</strong> 1024</li><li><strong>Total parameters:</strong> ~7M</li><li><strong>Optimizer:</strong> AdamW with linear warmup (10% of steps) + cosine decay</li><li><strong>Learning rate:</strong> 2e-4 peak</li><li><strong>Weight decay:</strong> 0.01</li><li><strong>Dropout:</strong> 0.1</li><li><strong>Batch size:</strong> 64 (Colab) / 128 (Cloud TPU v5p-8)</li><li><strong>Epochs:</strong> 20 (Colab) / 50 (Cloud TPU)</li><li><strong>Sequence length:</strong> 64 tokens for both claim and headline</li></ul><h3>Key Takeaways</h3><h3>Free Colab TPU is the best on-ramp for JAX/Flax in 2026</h3><p>You get a full v5e-1 chip with no signup beyond a Google account. For models in the 5–50M parameter range, that’s enough to do real work, not just toy demos. If you’ve been putting off learning JAX because the cloud setup felt overwhelming, this path is friction-free.</p><h3>Keras Kinetic makes the leap from Colab to Cloud TPU painless</h3><p>The biggest practical lesson from this project: when Colab’s runtime limits start hurting, switching to Cloud TPU traditionally means rewriting your deployment story. With Kinetic, you wrap your existing function in a decorator and call it from the same laptop you’ve been using. The mental model — “I have a function; I want it to run on a TPU” — stays intact.</p><h3>Stance detection is a more honest framing than fake-news classification</h3><p>Asking a model “is this true?” puts it in an impossible position. Asking “does this article support, refute, or merely observe this claim?” gives it a question it can answer, and gives downstream fact-checkers exactly the signal they need.</p><h3>Cross-attention beats independent encoders for paired-text tasks</h3><p>The StanceCrossEncoder is what makes this model genuinely multimodal-aware. Concatenating two independently-encoded vectors and slapping a classifier on top works, but performance jumps significantly when you let the two inputs literally read each other before pooling. The [c, h, |c-h|, c*h] interaction trick is borrowed from NLI literature and consistently outperforms simpler combinations.</p><h3>TPU is the right pick for fixed-length attention workloads</h3><p>Multi-head attention with fixed sequence lengths is exactly what TPU systolic arrays were built for. The whole train_step JITs into a single fused execution graph, and after the first compile, every step runs at peak FLOPs. GPUs are still preferable when you need irregular-length sequences, KV-caching for generation, or HuggingFace PyTorch checkpoints — but for from-scratch JAX/Flax training, TPU wins on both speed and cost.</p><h3>Adapting This to Other Languages and Tasks</h3><p>The pipeline is generic. To use it on your own data:</p><ul><li><strong>Any language → just point the tokenizer at your text corpus.</strong> The whitespace tokenizer is language-agnostic. For better quality, swap in the appropriate pre-trained tokenizer (English → BERT, Indonesian → IndoBERT, Arabic → AraBERT, etc.).</li><li><strong>Any paired-text classification task</strong> → swap the labels. The [c, h, |c-h|, c*h] interaction trick works for paraphrase detection, NLI, semantic textual similarity, duplicate question detection, and more.</li><li><strong>Larger corpora</strong> → bump embed_dim, num_layers, and switch from a learned tokenizer to a SentencePiece or BPE one. The training loop and Kinetic deployment don’t change.</li></ul><h3>Resources</h3><ul><li><strong>Project repository:</strong> <em>(your GitHub link here)</em></li><li><strong>JAX:</strong> <a href="https://github.com/google/jax">github.com/google/jax</a></li><li><strong>Flax:</strong> <a href="https://github.com/google/flax">github.com/google/flax</a></li><li><strong>Optax:</strong> <a href="https://github.com/google-deepmind/optax">github.com/google-deepmind/optax</a></li><li><strong>Orbax:</strong> <a href="https://github.com/google/orbax">github.com/google/orbax</a></li><li><strong>Keras Kinetic:</strong> <a href="https://github.com/keras-team/kinetic">github.com/keras-team/kinetic</a></li><li><strong>IndoBERT (production tokenizer upgrade):</strong><a href="https://huggingface.co/indobenchmark/indobert-base-p1">huggingface.co/indobenchmark/indobert-base-p1</a></li><li><strong>Cloud TPU documentation:</strong> <a href="https://cloud.google.com/tpu/docs">cloud.google.com/tpu/docs</a></li><li><strong>Google Colab:</strong> <a href="https://colab.research.google.com/">colab.research.google.com</a></li></ul><h3>Acknowledgement</h3><ul><li>Google Cloud credits were provided for this project. #TPUSprint</li><li>Thanks to the JAX, Flax, and Keras teams for building such a clean stack — training custom transformers on TPUs used to be a research-grade pain.</li></ul><p><strong>Tags:</strong> JAX, Flax, KerasKinetic, TPU, GoogleColab, FakeNewsDetection, StanceDetection, NaturalLanguageProcessing, MachineLearning, Indonesia</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b7f96c58b24" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developer-experts/building-a-multimodal-indonesian-fake-news-detector-with-jax-flax-and-keras-kinetic-on-cloud-tpu-4b7f96c58b24">Building a Multimodal Indonesian Fake-News Detector with JAX, Flax, and Keras Kinetic on Cloud TPU</a> was originally published in <a href="https://medium.com/google-developer-experts">Google Developer Experts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Hashtag Recommendation System with Graph Neural Networks, JAX, and Keras Kinetic on…]]></title>
            <link>https://medium.com/google-developer-experts/building-a-hashtag-recommendation-system-with-graph-neural-networks-jraph-and-jax-on-cloud-tpu-447f34b108d9?source=rss-a32fdb99f2d------2</link>
            <guid isPermaLink="false">https://medium.com/p/447f34b108d9</guid>
            <category><![CDATA[google-cloud-tpu]]></category>
            <category><![CDATA[social-network-analysis]]></category>
            <category><![CDATA[tpu]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Esther Irawati Setiawan]]></dc:creator>
            <pubDate>Thu, 30 Apr 2026 13:24:27 GMT</pubDate>
            <atom:updated>2026-04-30T16:37:45.272Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Hashtag Recommendation System with Graph Neural Networks, JAX, and Keras Kinetic on Cloud TPU</h3><p><em>How I trained a “related hashtags” recommender on a Cloud TPU v5p-8 using GraphSAGE, JAX, and JRaph — turning a plain CSV of hashtag co-occurrences into ranked suggestions</em></p><h3>Introduction</h3><p>Every social platform that supports hashtags — Instagram, X, TikTok, LinkedIn — quietly runs a hashtag recommender behind the scenes. When you type #100daysofcode into the caption box, the platform suggests #webdeveloper, #programmer, #codingbecause those tags routinely show up in the same posts. Behind that friendly autocomplete sits a graph-learning problem: given a network of which hashtags appear together, predict which other hashtags belong with the one the user just typed.</p><p>The modern answer to this problem is a <strong>Graph Neural Network (GNN)</strong>. Unlike a traditional neural network that treats inputs as flat vectors, a GNN learns by passing messages along the edges of a graph, so each node ends up with an embedding that reflects both its own identity and the structure of its neighborhood.</p><p>In this article, I’ll walk through a complete end-to-end pipeline that:</p><ol><li>Reads a CSV of hashtag co-occurrences (an edge list)</li><li>Builds a graph in NetworkX, with optional one-mode projection</li><li>Converts it to a JAX-native jraph.GraphsTuple</li><li>Trains a 2-layer <strong>GraphSAGE</strong> model with contrastive link-prediction loss</li><li>Recommends the top-k most related hashtags for any given query hashtag</li></ol><p>The whole thing runs on a <strong>Cloud TPU v5p-8</strong> in Google Cloud, but the exact same code runs unchanged on a CPU or GPU thanks to JAX. By the end, you’ll have a working python social_gnn.py script that produces output like this:</p><pre>Top 5 hashtag recommendations for &#39;#100daysofcode&#39;:<br>------------------------------<br>  #webdeveloper     (score 0.774)<br>  #programmer       (score 0.763)<br>  #codinglife       (score 0.729)<br>  #html             (score 0.728)<br>  #coding           (score 0.707)</pre><p>The same technique generalizes to friend recommendations, product co-purchase, paper citations, or anything else that fits the “graph of related things” pattern — but the demo here is hashtags. Let’s get started.</p><h3>Why Graph Neural Networks?</h3><p>Classic recommender systems usually fall into two buckets: collaborative filtering (matrix factorization on a user–item matrix) or content-based methods (cosine similarity on hand-crafted feature vectors). Both struggle when your only signal is <em>the structure of the graph itself</em> — no ratings, no descriptions, just edges.</p><p>GNNs are built for exactly this scenario. The core idea is<strong>message passing</strong>: each node aggregates information from its neighbors, then mixes it with its own state to produce a new representation. Stack two layers, and a node “sees” everything within two hops. The learned embeddings capture network proximity in a way that simple shortest-path distance can’t.</p><p>For link prediction specifically, this means two hashtags that are structurally similar — say, they share many co-occurring tags, or sit inside the same dense topical cluster — will end up with embeddings that have a high dot product, even if they’ve never appeared together in the same post. That’s the recommendation signal.</p><h3>Why GraphSAGE?</h3><p>There are many GNN architectures (GCN, GAT, GIN…), but<strong>GraphSAGE</strong> (Sample and Aggregate) hits a sweet spot of simplicity and effectiveness. A single GraphSAGE layer does three things:</p><ol><li><strong>Gather</strong> features from a node’s neighbors</li><li><strong>Aggregate</strong> them with a permutation-invariant function (mean pooling here)</li><li><strong>Combine</strong> the aggregation with the node’s own features through a linear layer</li></ol><p>Two layers stacked together give each node a representation informed by its 2-hop neighborhood. After L2-normalizing the final output, dot-product similarity becomes equivalent to cosine similarity, which is well-behaved for ranking.</p><h3>Why JAX, JRaph, and TPU?</h3><ul><li><strong>JAX</strong> gives me NumPy-style code with automatic differentiation, JIT compilation via XLA, and transparent acceleration on CPU/GPU/TPU.</li><li><strong>JRaph</strong> is the JAX-native graph library from DeepMind. Its GraphsTuple data structure is a clean, flat representation of a graph that plays nicely with JIT.</li><li><strong>Flax</strong> sits on top of JAX and lets me write neural networks as nn.Module classes without ceremony.</li><li><strong>Cloud TPU v5p-8</strong> because TPUs are extremely fast at the dense matmuls that GNN message passing eventually boils down to, and Google gives new accounts $300 in credits to try them.</li></ul><h3>TPU vs GPU: Why TPU for This Workload?</h3><p>A fair question every JAX user runs into is: <em>should I be running this on a GPU instead?</em> Both work — JAX is hardware-agnostic and the same social_gnn.py script runs unchanged on either — but the two chips have very different design philosophies, and the choice matters for cost, throughput, and ergonomics. Here’s how they stack up for a GraphSAGE workload like this one.</p><p><strong>Hardware design</strong></p><ul><li><em>GPU (e.g. NVIDIA A100/H100):</em> general-purpose parallel processor with thousands of CUDA cores; great for arbitrary parallel computation.</li><li><em>TPU (e.g. v5p-8):</em> domain-specific accelerator built around a large systolic array (MXU) optimized for dense matrix multiplications.</li></ul><p><strong>Programming model</strong></p><ul><li><em>GPU:</em> CUDA-native; JAX and PyTorch lower to cuDNN/cuBLAS under the hood.</li><li><em>TPU:</em> XLA-native; the JAX → XLA → TPU path is the “home” path with the deepest optimization.</li></ul><p><strong>Memory</strong></p><ul><li><em>GPU:</em> HBM2e/HBM3, 40–80 GB per chip on A100/H100.</li><li><em>TPU:</em> HBM with similar capacities, tightly coupled to the systolic array for high-bandwidth matmul feeding.</li></ul><p><strong>Sweet spot</strong></p><ul><li><em>GPU:</em> mixed workloads, irregular control flow, kernels with lots of branching, and ecosystems where CUDA libraries dominate.</li><li><em>TPU:</em> large dense matmuls with predictable shapes — transformer training, GNN message passing, big embedding lookups.</li></ul><p><strong>Sparse and irregular operations</strong></p><ul><li><em>GPU:</em> strong, because CUDA exposes warps and atomics directly.</li><li><em>TPU:</em> historically weaker, but improved with newer chips like SparseCore on v5p.</li></ul><p><strong>JAX developer experience</strong></p><ul><li><em>GPU:</em> excellent — pip install &quot;jax[cuda12]&quot; and you’re going.</li><li><em>TPU:</em> excellent — pip install &quot;jax[tpu]&quot; on a TPU VM and you’re going; no driver install required.</li></ul><p><strong>Cost (Spot/Preemptible, US regions)</strong></p><ul><li><em>GPU:</em> A100 around $1.10–$1.50 per GPU-hour.</li><li><em>TPU:</em> v5e-1 around $0.40 per chip-hour; v5p-8 noticeably more, but with 8 chips of throughput.</li></ul><p><strong>Free tier</strong></p><ul><li><em>GPU:</em> none for the beefy chips you’d actually want.</li><li><em>TPU:</em> a $300 GCP credit covers meaningful TPU experimentation for new accounts.</li></ul><h3>Where each chip wins for graph workloads</h3><p><strong>GPU wins when:</strong></p><ul><li>The graph is small enough to fit on one chip and you’re iterating rapidly on model architecture in a notebook</li><li>You need irregular, sparse operations (e.g., sampling sub-graphs with custom CUDA kernels, GraphSAGE with on-the-fly neighbor sampling using libraries like DGL or PyG)</li><li>You’re embedded in the PyTorch ecosystem and want to use PyG (PyTorch Geometric), which is more mature than the JAX graph stack</li><li>You’re prototyping and your code has a lot of Python control flow that doesn’t JIT cleanly</li></ul><p><strong>TPU wins when:</strong></p><ul><li>You’re running full-batch GNN training where the entire graph fits in HBM — exactly this article’s setup</li><li>You can express your computation as dense matmul + segment ops (which JRaph’s GraphsTuple and JAX’ssegment_sum do beautifully)</li><li>You can JIT the entire training step with jax.jit — the XLA compiler then fuses the whole graph into a few kernels and the TPU runs at peak</li><li>You want to scale out: TPUs come in pods, and jax.pmap / shard_map make multi-chip training straightforward</li><li>Cost matters and your workload is a good fit — TPU $/training-step is often lower than equivalent GPU spend</li></ul><h3>What this means for this project</h3><p>The GraphSAGE workload here is almost a textbook TPU sweet spot:</p><ol><li><strong>Full-batch training</strong> — the whole graph and all positive/negative edges fit comfortably in TPU HBM, so we never have to sample sub-graphs. That eliminates the irregular operation that GPUs would otherwise win on.</li><li><strong>The expensive ops are matmuls</strong> — nn.Dense(combined)inside sage_layer is a vanilla matrix multiply, which is exactly what the TPU’s systolic array is built for.</li><li><strong>segment_sum is XLA-friendly</strong> — JAX lowers it to a fused TPU kernel, no custom CUDA required.</li><li><strong>The whole </strong><strong>train_step is one </strong><strong>jax.jit</strong> — XLA compiles the entire forward pass, loss, backward pass, and optimizer update into a single fused execution graph. After the first compilation, every step runs at peak throughput.</li></ol><p>The honest caveat: for a 796-node, 1361-edge graph, <em>any</em>hardware will train this in seconds. The TPU advantage shows up when the graph scales to millions of nodes — where full-batch training stops being free and the systolic array’s throughput vs. cost ratio starts to matter. This article uses a small graph so the pipeline is easy to follow, but the same code (with a larger batch_size and more epochs) is what you’d run on a 10-million-node graph.</p><h3>Rule of thumb</h3><ul><li><strong>Quick prototyping on one laptop with a small graph</strong> → just use CPU. JAX runs fine there and you’ll save the cloud setup time.</li><li><strong>Iterative R&amp;D in PyTorch with PyG (PyTorch Geometric)</strong> → GPU. The PyG ecosystem is the most mature for graph learning today.</li><li><strong>Production GNN training that’s batchable and JAX-native</strong> → TPU. Best throughput-per-dollar for this profile.</li><li><strong>LLM-scale embedding tables (DLRM, two-tower retrieval, etc.)</strong> → TPU with SparseCore. This is exactly what SparseCore was designed for.</li><li><strong>You already have a CUDA team and tooling</strong> → GPU. Don’t fight your existing stack just for marginal gains.</li></ul><p>Now let’s actually build it.</p><h3>Project Architecture</h3><p>The pipeline is a straight line from CSV to recommendations:</p><pre>relations.csv (gs://...)<br>            │<br>            ▼<br>   ┌──────────────────┐<br>   │   load_csv()     │  cleans, dedupes, removes self-loops<br>   └──────────────────┘<br>            │<br>            ▼<br>   ┌──────────────────┐<br>   │  build_graph()   │  NetworkX graph (mode=&#39;2&#39;, &#39;1a&#39;, or &#39;1b&#39;)<br>   └──────────────────┘<br>            │<br>            ▼<br>   ┌──────────────────┐<br>   │  nx_to_jraph()   │  symmetrized GraphsTuple + random features<br>   └──────────────────┘<br>            │<br>            ▼<br>   ┌──────────────────┐<br>   │  GraphSAGE +     │  2-layer encoder, JIT-compiled training<br>   │  contrastive     │  on TPU v5p-8<br>   │  link loss       │<br>   └──────────────────┘<br>            │<br>            ▼<br>   ┌──────────────────┐<br>   │ recommend_links()│  top-k by cosine similarity<br>   └──────────────────┘</pre><p>Now let’s walk through it.</p><h3>Section 1 — Google Cloud Account Setup</h3><p>If you’ve never used Google Cloud before, start here. Everything in this guide runs on GCP — the TPU VM, the storage bucket, and the dataset.</p><h3>1.1 Create a Google Cloud account</h3><ol><li>Go to <a href="https://cloud.google.com/"><strong>https://cloud.google.com</strong></a> in your browser.</li><li>Click <strong>Get started for free</strong> — you receive <strong>$300 in free credits valid for 90 days</strong>.</li><li>Sign in with a Google account (Gmail or Workspace).</li><li>Complete the identity verification step (a credit/debit card is required, but you will not be charged during the free trial unless you manually upgrade).</li><li>Accept the Terms of Service.</li></ol><h3>1.2 Create your first Project</h3><p>All GCP resources live inside a <strong>Project</strong>. Think of it as a billing and permissions boundary.</p><ol><li>Open the <strong>Google Cloud Console</strong>: <a href="https://console.cloud.google.com/">https://console.cloud.google.com</a></li><li>Click the project selector dropdown at the top-left.</li><li>Click <strong>New Project</strong>.</li><li>Give it a name, e.g. sna-tpu-recommender.</li><li>Note the auto-generated <strong>Project ID</strong> (e.g. sna-tpu-recommender-123456) — you’ll use this everywhere.</li><li>Click <strong>Create</strong>.</li></ol><h3>1.3 Enable Billing</h3><p>TPU VMs require a billing account to be linked, even when using free credits.</p><ol><li>Open the Navigation Menu → <strong>Billing</strong>.</li><li>Click <strong>Link a billing account</strong> and select your account.</li><li>Confirm the project is linked.</li></ol><h3>1.4 Install and configure the gcloud CLI</h3><p>The gcloud CLI lets you manage GCP resources from your terminal instead of clicking through the web console.</p><p>For Windows, download the installer from <a href="https://cloud.google.com/sdk/docs/install">cloud.google.com/sdk/docs/install</a> and follow the prompts. For macOS/Linux, the same page has a curl one-liner.</p><p>Then, in a fresh terminal:</p><pre>gcloud init<br>gcloud config set project YOUR_PROJECT_ID<br>gcloud config set compute/region us-central1<br>gcloud config set compute/zone us-central1-a<br>gcloud config list</pre><p>Verify your CLI version (must be ≥ 460):</p><pre>gcloud version<br># Google Cloud SDK 460.0.0 or higher required<br># If yours is older:<br>gcloud components update</pre><h3>Section 2 — Enable APIs and IAM Permissions</h3><p>GCP services are disabled by default to prevent accidental usage. Enable the ones this project needs:</p><pre>gcloud services enable \<br>  tpu.googleapis.com \<br>  compute.googleapis.com \<br>  storage.googleapis.com \<br>  aiplatform.googleapis.com \<br>  logging.googleapis.com</pre><p>This takes about 30 seconds. If you created the project yourself, you’re already the Owner. If someone else owns it, ask them to grant you these roles:</p><pre>gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \<br>  --member=&quot;user:USER_EMAIL&quot; \<br>  --role=&quot;roles/tpu.admin&quot;<br>gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \<br>  --member=&quot;user:USER_EMAIL&quot; \<br>  --role=&quot;roles/storage.admin&quot;<br>gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \<br>  --member=&quot;user:USER_EMAIL&quot; \<br>  --role=&quot;roles/compute.admin&quot;</pre><h3>Section 3 — Create a Storage Bucket and TPU VM</h3><h3>3.1 Create the bucket</h3><p>Create a Cloud Storage bucket and upload your edge-list CSV. The dataset just needs two columns: node_A and node_B, where each row represents a co-occurrence between two hashtags (i.e., they appeared in the same post). For example:</p><pre>node_A,node_B<br>#100daysofcode,#python<br>#100daysofcode,#coding<br>#machinelearning,#datascience<br>...</pre><p>The model is fully agnostic to what the nodes actually represent — they’re just strings — so the same CSV format works if you swap in users, products, papers, or any other entities. I’ll come back to that in the “Adapting This to Your Domain” section.</p><h3>3.2 Create the TPU VM</h3><p>Provision a TPU v5p-8 (version 5 performance, 8 cores) with these settings:</p><p>SettingValueNamesna-rec-tpuZoneus-central1-aTPU Typev5p-8TPU Software Versionv2-alpha-tpuv5SpotEnabled (much cheaper)</p><h3>3.3 List and SSH</h3><pre>gcloud compute tpus tpu-vm list --zone=us-central1-a<br>gcloud compute tpus tpu-vm ssh sna-rec-tpu --zone=us-central1-a</pre><h3>3.4 Verify Python and install dependencies</h3><pre>python3 --version    # The TPU VM image ships with Python 3 pre-installed<br>pip install &quot;jax[tpu]&quot; -f https://storage.googleapis.com/jax-releases/libtpu_releases.html<br>pip install jax jaxlib flax jraph optax networkx numpy pandas gcsfs</pre><p>The install takes 3–5 minutes. Now you’re ready to write the actual GNN.</p><h3>Section 4 — Code Walkthrough</h3><p>The full pipeline lives in a single file, social_gnn.py. I’ll go through it in the order data flows.</p><h3>4.1 Imports</h3><pre>from __future__ import annotations<br>import argparse<br>import functools<br>from collections import defaultdict<br>from itertools import combinations<br>from typing import NamedTuple<br>import jax<br>import jax.numpy as jnp<br>import jraph<br>import networkx as nx<br>import numpy as np<br>import optax<br>import pandas as pd<br>from flax import linen as nn</pre><p>Nothing unusual — JAX/Flax/JRaph for the model, NetworkX for graph construction, pandas for the CSV.</p><h3>4.2 Loading the CSV</h3><p>The load_csv function reads either a local file or a GCS path (gs://…) and does several non-negotiable cleanups:</p><ul><li>Keeps only the two node columns and renames them tonode_A / node_B</li><li>Drops NaN rows</li><li>Casts to string so numeric and text IDs can coexist</li><li><strong>Removes self-loops</strong> (a node connected to itself)</li><li><strong>Deduplicates undirected edges</strong> so (A, B) and (B, A)collapse to one</li></ul><pre>def load_csv(path: str, *, node_a_col: str = &quot;node_A&quot;,<br>             node_b_col: str = &quot;node_B&quot;,<br>             sep: str = &quot;,&quot;) -&gt; pd.DataFrame:<br>    is_gcs = path.startswith(&quot;gs://&quot;)<br>    if is_gcs:<br>        try:<br>            import gcsfs  # noqa: F401  (pandas finds it via gs:// protocol)<br>        except ImportError as e:<br>            raise ImportError(<br>                &quot;Reading from gs:// requires gcsfs. &quot;<br>                &quot;Install it with: pip install gcsfs&quot;<br>            ) from e<br>        print(f&quot;Reading from GCS: {path}&quot;)<br>    else:<br>        print(f&quot;Reading from local path: {path}&quot;)<br>    df = pd.read_csv(path, sep=sep)<br>    if node_a_col not in df.columns or node_b_col not in df.columns:<br>        raise ValueError(<br>            f&quot;CSV must contain columns {node_a_col!r} and {node_b_col!r}. &quot;<br>            f&quot;Got: {list(df.columns)}&quot;<br>        )<br>    df = df[[node_a_col, node_b_col]].copy()<br>    df.columns = [&quot;node_A&quot;, &quot;node_B&quot;]<br>    df = df.dropna()<br>    df[&quot;node_A&quot;] = df[&quot;node_A&quot;].astype(str).str.strip()<br>    df[&quot;node_B&quot;] = df[&quot;node_B&quot;].astype(str).str.strip()<br>    # Remove self-loops<br>    df = df[df[&quot;node_A&quot;] != df[&quot;node_B&quot;]]<br>    # Canonicalize undirected edges so (u, v) and (v, u) dedupe together<br>    swap = df[&quot;node_A&quot;] &gt; df[&quot;node_B&quot;]<br>    df.loc[swap, [&quot;node_A&quot;, &quot;node_B&quot;]] = df.loc[swap, [&quot;node_B&quot;, &quot;node_A&quot;]].values<br>    df = df.drop_duplicates(subset=[&quot;node_A&quot;, &quot;node_B&quot;]).reset_index(drop=True)<br>    return df</pre><p>A note on GCS: on a TPU VM, authentication is automatic via the VM’s service account, so you don’t have to mess with credentials.</p><h3>4.3 Building the graph</h3><p>build_graph supports three projection modes, which is useful if your data is naturally bipartite (users ↔︎ hashtags, customers ↔︎ products, etc.):</p><p>ModeWhat it builds&#39;2&#39;The raw bipartite/two-mode graph: A connects directly to B as in the CSV.&#39;1a&#39;One-mode projection onto A nodes. Two A-nodes connect if they share a B-node. Edge weight = number of shared B-nodes.&#39;1b&#39;One-mode projection onto B nodes (mirror of &#39;1a&#39;).</p><pre>def build_graph(df: pd.DataFrame, mode: str = &quot;2&quot;) -&gt; nx.Graph:<br>    G = nx.Graph()<br>    if mode == &quot;2&quot;:<br>        edges = df[[&quot;node_A&quot;, &quot;node_B&quot;]].astype(str).values.tolist()<br>        G.add_edges_from(edges)<br>    elif mode in (&quot;1a&quot;, &quot;1b&quot;):<br>        if mode == &quot;1a&quot;:<br>            groups = df.groupby(&quot;node_B&quot;)[&quot;node_A&quot;].apply(<br>                lambda x: [str(a) for a in x.unique()]<br>            )<br>        else:<br>            groups = df.groupby(&quot;node_A&quot;)[&quot;node_B&quot;].apply(<br>                lambda x: [str(b) for b in x.unique()]<br>            )<br>        edge_weights: dict[tuple[str, str], int] = defaultdict(int)<br>        for nodes in groups:<br>            for n1, n2 in combinations(nodes, 2):<br>                key = (n1, n2) if n1 &lt; n2 else (n2, n1)<br>                edge_weights[key] += 1<br>        for (n1, n2), w in edge_weights.items():<br>            G.add_edge(n1, n2, weight=w)<br>    else:<br>        raise ValueError(f&quot;Unknown mode: {mode!r}. Use &#39;2&#39;, &#39;1a&#39;, or &#39;1b&#39;.&quot;)<br>    return G</pre><p>For the rest of this article, I’ll use mode=&#39;2&#39; — the raw graph as given.</p><h3>4.4 Bridging NetworkX and JRaph</h3><p>Now the interesting part. NetworkX is a friendly graph library, but it’s pure Python and stores nodes as arbitrary hashable objects. JAX/JRaph wants flat, fixed-size integer arrays. Thenx_to_jraph function bridges that gap and does four things:</p><ul><li><strong>Indexes nodes</strong> — sorts node IDs and assigns each an integer, with both directions of the mapping stored</li><li><strong>Symmetrizes</strong> — every undirected edge (u, v) becomes two directed edges u→v and v→u, so messages flow both ways during training</li><li><strong>Initializes random features</strong> — nodes have no inherent features (just IDs), so we seed them with small Gaussian noise; these will be refined by gradient descent</li><li><strong>Packs everything into a </strong><strong>jraph.GraphsTuple</strong> — a flat, JIT-friendly representation</li></ul><pre>class GraphData(NamedTuple):<br>    &quot;&quot;&quot;Bundle of everything we need downstream.&quot;&quot;&quot;<br>    graph: jraph.GraphsTuple<br>    node_to_idx: dict[str, int]<br>    idx_to_node: dict[int, str]<br>    pos_edges: np.ndarray<br>    num_nodes: int<br><br>def nx_to_jraph(G: nx.Graph, feature_dim: int = 64,<br>                seed: int = 0) -&gt; GraphData:<br>    nodes = sorted(G.nodes())<br>    node_to_idx = {n: i for i, n in enumerate(nodes)}<br>    idx_to_node = {i: n for n, i in node_to_idx.items()}<br>    num_nodes = len(nodes)<br>    pos_edges = np.array(<br>        [(node_to_idx[u], node_to_idx[v]) for u, v in G.edges()],<br>        dtype=np.int32,<br>    )<br>    # Symmetrize for message passing<br>    senders = np.concatenate([pos_edges[:, 0], pos_edges[:, 1]])<br>    receivers = np.concatenate([pos_edges[:, 1], pos_edges[:, 0]])<br>    # Random initial node features - will be updated via gradient descent<br>    rng = np.random.default_rng(seed)<br>    node_features = rng.normal(<br>        scale=0.1, size=(num_nodes, feature_dim)<br>    ).astype(np.float32)<br>    graph = jraph.GraphsTuple(<br>        nodes=jnp.asarray(node_features),<br>        edges=None,<br>        senders=jnp.asarray(senders),<br>        receivers=jnp.asarray(receivers),<br>        n_node=jnp.array([num_nodes]),<br>        n_edge=jnp.array([len(senders)]),<br>        globals=None,<br>    )<br>    return GraphData(<br>        graph=graph,<br>        node_to_idx=node_to_idx,<br>        idx_to_node=idx_to_node,<br>        pos_edges=pos_edges,<br>        num_nodes=num_nodes,<br>    )</pre><h3>4.5 The GraphSAGE model</h3><p>This is the heart of the project — a 2-layer GraphSAGE encoder written as a Flax nn.Module. I wrote the layer explicitly (instead of using jraph.GraphConvolution) so the message-passing logic is plain to read.</p><p>The forward pass is:</p><pre>Input features (random, shape: [num_nodes, 64])<br>  → sage_layer → hidden_dim=128 → ReLU<br>  → sage_layer → embed_dim=64<br>  → L2 normalization → output embeddings (unit vectors)</pre><p>L2-normalizing at the end is the trick that makes everything else work cleanly: after normalization, dot products between embedding vectors are exactly cosine similarities, bounded in[-1, 1], which makes the contrastive loss well-behaved.</p><pre>class GraphSAGE(nn.Module):<br>    &quot;&quot;&quot;A 2-layer GraphSAGE encoder.For each node, we aggregate neighbour features (mean pooling), concatenate<br>    with the node&#39;s own features, then apply an MLP. Output is an embedding<br>    per node.<br>    &quot;&quot;&quot;<br>    hidden_dim: int = 128<br>    embed_dim: int = 64<br>    @nn.compact<br>    def __call__(self, graph: jraph.GraphsTuple) -&gt; jnp.ndarray:<br>        def sage_layer(nodes, senders, receivers, out_dim):<br>            num_nodes = nodes.shape[0]<br>            messages = nodes[senders]  # (num_edges, feat_dim)<br>            # Mean aggregation per receiver<br>            sums = jax.ops.segment_sum(<br>                messages, receivers, num_segments=num_nodes<br>            )<br>            counts = jax.ops.segment_sum(<br>                jnp.ones_like(receivers, dtype=jnp.float32),<br>                receivers,<br>                num_segments=num_nodes,<br>            )<br>            counts = jnp.maximum(counts, 1.0)[:, None]<br>            agg = sums / counts<br>            # Concatenate self + aggregated neighbour features, then linear layer<br>            combined = jnp.concatenate([nodes, agg], axis=-1)<br>            return nn.Dense(out_dim)(combined)<br>        h = graph.nodes<br>        h = sage_layer(h, graph.senders, graph.receivers, self.hidden_dim)<br>        h = nn.relu(h)<br>        h = sage_layer(h, graph.senders, graph.receivers, self.embed_dim)<br>        # L2-normalize so dot product behaves like cosine similarity<br>        h = h / (jnp.linalg.norm(h, axis=-1, keepdims=True) + 1e-8)<br>        return h<br><br>def link_score(embeddings: jnp.ndarray, edges: jnp.ndarray) -&gt; jnp.ndarray:<br>    &quot;&quot;&quot;Dot-product score for each (u, v) pair.&quot;&quot;&quot;<br>    u = embeddings[edges[:, 0]]<br>    v = embeddings[edges[:, 1]]<br>    return jnp.sum(u * v, axis=-1<br></pre><h3>4.6 Training: contrastive link prediction</h3><p>GNNs for link prediction are trained <strong>contrastively</strong>: the model should score real edges high and random non-edges low. Three pieces make this work.</p><p><strong>Negative sampling.</strong> At each step we sample random (u, v)pairs that are <em>not</em> edges, rejecting self-loops and existing connections:</p><pre>def sample_negative_edges(num_nodes: int, pos_edge_set: set,<br>                          n_samples: int, rng: np.random.Generator) -&gt; np.ndarray:<br>    neg = np.empty((n_samples, 2), dtype=np.int32)<br>    i = 0<br>    while i &lt; n_samples:<br>        u = int(rng.integers(0, num_nodes))<br>        v = int(rng.integers(0, num_nodes))<br>        if u == v:<br>            continue<br>        key = (u, v) if u &lt; v else (v, u)<br>        if key in pos_edge_set:<br>            continue<br>        neg[i] = (u, v)<br>        i += 1<br>    return neg</pre><p><strong>The training step.</strong> JIT-compiled with jax.jit so it runs at full speed on whatever device JAX finds (CPU, GPU, or TPU). The loss is binary cross-entropy with logits, with a temperature scale of 5.0 to amplify gradients after L2-normalization (otherwise gradients are tiny because all dot products live in [-1, 1]):</p><pre>@functools.partial(jax.jit, static_argnames=(&#39;optimizer_update&#39;,&#39;model_apply&#39;))<br>def train_step(model_apply, params, opt_state, optimizer_update,<br>               graph, pos_edges, neg_edges):<br>    &quot;&quot;&quot;One JIT-compiled training step.&quot;&quot;&quot;<br>    def loss_fn(params):<br>        embeddings = model_apply(params, graph)<br>        pos_scores = link_score(embeddings, pos_edges)<br>        neg_scores = link_score(embeddings, neg_edges)<br>        temperature = 5.0<br>        pos_logits = pos_scores * temperature<br>        neg_logits = neg_scores * temperature<br>        pos_loss = -jax.nn.log_sigmoid(pos_logits).mean()<br>        neg_loss = -jax.nn.log_sigmoid(-neg_logits).mean()<br>        return pos_loss + neg_loss<br>    loss, grads = jax.value_and_grad(loss_fn)(params)<br>    updates, opt_state = optimizer_update(grads, opt_state, params)<br>    params = optax.apply_updates(params, updates)<br>    return params, opt_state, loss</pre><p><strong>The training loop.</strong> Adam optimizer, 200 epochs by default, batch size 4096 (or fewer if the graph is small):</p><pre>def train(graph_data: GraphData, *, epochs: int = 200, batch_size: int = 4096,<br>          lr: float = 1e-3, seed: int = 0):<br>    print(f&quot;Training on device: {jax.devices()[0]}&quot;)<br>    model = GraphSAGE()<br>    rng_key = jax.random.PRNGKey(seed)<br>    params = model.init(rng_key, graph_data.graph)<br>    optimizer = optax.adam(lr)<br>    opt_state = optimizer.init(params)<br>    pos_edge_set = {<br>        (int(u), int(v)) if u &lt; v else (int(v), int(u))<br>        for u, v in graph_data.pos_edges<br>    }<br>    np_rng = np.random.default_rng(seed)<br>    n_pos = len(graph_data.pos_edges)<br>    batch_size = min(batch_size, n_pos)<br>    for epoch in range(epochs):<br>        pos_idx = np_rng.choice(n_pos, size=batch_size, replace=False)<br>        pos_batch = graph_data.pos_edges[pos_idx]<br>        neg_batch = sample_negative_edges(<br>            graph_data.num_nodes, pos_edge_set, batch_size, np_rng<br>        )<br>        params, opt_state, loss = train_step(<br>            model.apply, params, opt_state, optimizer.update,<br>            graph_data.graph,<br>            jnp.asarray(pos_batch),<br>            jnp.asarray(neg_batch),<br>        )<br>        if epoch % 20 == 0 or epoch == epochs - 1:<br>            print(f&quot;  epoch {epoch:4d} | loss {float(loss):.4f}&quot;)<br>    embeddings = model.apply(params, graph_data.graph)<br>    return np.asarray(embeddings), params</pre><h3>4.7 Recommending hashtags</h3><p>Once training is done, recommendations are pleasingly simple. To find related hashtags for a query hashtag:</p><ol><li>Compute its dot-product similarity with every other node’s embedding (this is cosine similarity since vectors are unit-norm)</li><li>Mask out the query node itself (-inf)</li><li>Mask out hashtags that have already co-occurred with the query (-inf) — those aren’t <em>recommendations</em>, they’re already known associations</li><li>Return the top-k remaining hashtags</li></ol><pre>def recommend_links(embeddings: np.ndarray, graph_data: GraphData,<br>                    node: str, top_k: int = 10) -&gt; list[tuple[str, float]]:<br>    &quot;&quot;&quot;Return top-k nodes most likely to be connected to `node` but aren&#39;t.&quot;&quot;&quot;<br>    if node not in graph_data.node_to_idx:<br>        raise KeyError(f&quot;Node {node!r} not in graph.&quot;)<br>    idx = graph_data.node_to_idx[node]<br>    scores = embeddings @ embeddings[idx]  # cosine sim - embeddings are unit-norm<br>    scores[idx] = -np.inf<br>    pos_set = {(int(u), int(v)) for u, v in graph_data.pos_edges}<br>    pos_set |= {(v, u) for u, v in pos_set}<br>    for u, v in pos_set:<br>        if u == idx:<br>            scores[v] = -np.inf<br>    top_idx = np.argsort(-scores)[:top_k]<br>    return [(graph_data.idx_to_node[int(i)], float(scores[i])) for i in top_idx]</pre><p>This is the same pattern Instagram or X uses to power their hashtag autocomplete: surface tags that are <em>structurally close</em> to what you typed in the embedding space, even if they’ve never appeared together yet. The exact same code, fed a different CSV, would generate “People You May Know” suggestions on a friend graph or “frequently bought together” suggestions on a co-purchase graph — the model doesn’t know or care.</p><h3>4.8 Putting it all together</h3><p>A small argparse wrapper ties the pieces into a runnable CLI:</p><pre>def main():<br>    parser = argparse.ArgumentParser(<br>        description=&quot;Train a GraphSAGE GNN to recommend related hashtags from a co-occurrence graph.&quot;<br>    )<br>    parser.add_argument(&quot;--csv&quot;, type=str, default=None,<br>        help=&quot;Path to edge-list CSV (local or gs://).&quot;)<br>    parser.add_argument(&quot;--node-a-col&quot;, default=&quot;node_A&quot;)<br>    parser.add_argument(&quot;--node-b-col&quot;, default=&quot;node_B&quot;)<br>    parser.add_argument(&quot;--sep&quot;, default=&quot;,&quot;)<br>    parser.add_argument(&quot;--mode&quot;, default=&quot;2&quot;, choices=[&quot;2&quot;, &quot;1a&quot;, &quot;1b&quot;])<br>    parser.add_argument(&quot;--epochs&quot;, type=int, default=200)<br>    parser.add_argument(&quot;--query-node&quot;, default=None)<br>    args = parser.parse_args()<br>    print(&quot;=== Social Network GNN ===\n&quot;)<br>    df = load_csv(args.csv, node_a_col=args.node_a_col,<br>                  node_b_col=args.node_b_col, sep=args.sep)<br>    G = build_graph(df, mode=args.mode)<br>    print(f&quot;NetworkX graph (mode={args.mode}): &quot;<br>          f&quot;{G.number_of_nodes()} nodes, {G.number_of_edges()} edges.\n&quot;)<br>    graph_data = nx_to_jraph(G)<br>    print(f&quot;jraph GraphsTuple ready. Node feature dim: &quot;<br>          f&quot;{graph_data.graph.nodes.shape[1]}\n&quot;)<br>    embeddings, params = train(graph_data, epochs=args.epochs)<br>    print(f&quot;\nLearned embeddings shape: {embeddings.shape}\n&quot;)<br>    query_nodes = list(graph_data.node_to_idx.keys())[:3]<br>    for query in query_nodes:<br>        recs = recommend_links(embeddings, graph_data, query, top_k=5)<br>        print(f&quot;\nTop 5 hashtag recommendations for {query!r}:&quot;)<br>        print(&quot;-&quot; * 30)<br>        for n, score in recs:<br>            print(f&quot;  {n:&lt;15} (score {score:.3f})&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h3>Running It on TPU</h3><p>On the TPU VM, with relations.csv already uploaded to a GCS bucket:</p><pre>python social_gnn.py --csv gs://sna-tpu-bucket/relations.csv --epochs 500</pre><p>Real output from a 796-node, 1361-edge hashtag co-occurrence graph:</p><pre>=== Social Network GNN ===<br>Reading from GCS: gs://sna-tpu-bucket/relations.csv<br>Loaded 1361 unique undirected edges.<br>NetworkX graph (mode=2): 796 nodes, 1361 edges.<br>jraph GraphsTuple ready. Node feature dim: 64<br>Training on device: TPU_0(process=0,(0,0,0,0))<br>  epoch    0 | loss 2.2295<br>  epoch   20 | loss 0.9972<br>  epoch   40 | loss 0.9676<br>  epoch   60 | loss 0.9182<br>  epoch   80 | loss 0.9083<br>  epoch  100 | loss 0.9011<br>  epoch  200 | loss 0.8626<br>  epoch  300 | loss 0.8395<br>  epoch  400 | loss 0.8222<br>  epoch  499 | loss 0.8311</pre><pre>Learned embeddings shape: (796, 64)</pre><pre>Top 5 hashtag recommendations for &#39;#100daysofcode&#39;:<br>------------------------------<br>  #webdeveloper     (score 0.774)<br>  #programmer       (score 0.763)<br>  #codinglife       (score 0.729)<br>  #html             (score 0.728)<br>  #coding           (score 0.707)</pre><pre>Top 5 hashtag recommendations for &#39;#10k&#39;:<br>------------------------------<br>  #dataset          (score 0.584)<br>  #generations      (score 0.557)<br>  #pythoncheatsheets (score 0.556)<br>  #pythontricks     (score 0.542)<br>  #billgates        (score 0.536)</pre><pre>Top 5 hashtag recommendations for &#39;#10kcelebration&#39;:<br>------------------------------<br>  #seattle          (score 0.596)<br>  #mlcareer         (score 0.588)<br>  #datagathering    (score 0.587)<br>  #bestml           (score 0.578)<br>  #codingcontent    (score 0.565)</pre><p>The recommendations for #100daysofcode are exactly what you’d hope: a tight cluster of programming-adjacent hashtags. The model never sees these labels — it learns them purely from co-occurrence structure.</p><h3>Training Configuration Summary</h3><p>ParameterValueArchitecture2-layer GraphSAGEHidden dim128Embedding dim64Initial featuresRandom Gaussian (σ=0.1)OptimizerAdamLearning rate1e-3LossBinary cross-entropy + L2 normTemperature5.0Epochs200 (or 500)Batch size4096AcceleratorTPU v5p-8</p><h3>Tear Down (Important!)</h3><p>Once you’re done, <strong>shut the TPU VM down</strong> to avoid charges. TPUs are not cheap if you leave them idle:</p><pre>gcloud compute tpus tpu-vm delete sna-rec-tpu --zone=us-central1-a</pre><p>You can also just stop it (instead of deleting) if you plan to come back, but stopped TPU VMs still incur disk costs.</p><h3>Section 5 — A Faster Path: Deploying with Keras Kinetic</h3><p>Sections 1–4 walked through the manual deployment path: create a TPU VM, SSH in, install dependencies, upload your script, and run it. That workflow is great for understanding what’s actually happening, but it gets repetitive fast — every time you tweak a hyperparameter, you’re back in the SSH window babysitting the run.</p><p><a href="https://github.com/keras-team/kinetic">Keras Kinetic</a> is a newer tool from the Keras team that collapses the entire deployment flow into a single Python decorator. You decorate your training function with @kinetic.run(accelerator=&quot;v5p-8&quot;), call it locally, and Kinetic transparently packages the code, builds a container, provisions a GKE cluster with TPUs attached, executes the function on the remote pod, and streams results back to your laptop.</p><p>Despite the name, <strong>Kinetic isn’t tied to Keras</strong> — the decorated function can contain any Python code. Our JAX/Flax/JRaph pipeline runs inside it without modification.</p><h3>5.1 Install Kinetic and authenticate</h3><p>On your <em>local</em> machine (laptop or workstation — you don’t need a TPU VM for this):</p><pre>pip install keras-kinetic<br>gcloud auth application-default login<br>gcloud config set project YOUR_PROJECT_ID</pre><h3>5.2 Provision the cluster (one-time)</h3><p>Instead of manually creating a TPU VM, let Kinetic stand up a GKE cluster with TPU-attached node pools:</p><pre>kinetic up --accelerator v5p-8 --yes</pre><p>Behind the scenes, Kinetic enables required GCP APIs, creates an Artifact Registry repo, provisions a GKE Autopilot cluster, and configures the TPU node pool. This takes a few minutes the first time. After this, you don’t touch infrastructure again until tear-down.</p><h3>5.3 Wrap the training pipeline in a decorated function</h3><p>Create a new file train_kinetic.py next to your existingsocial_gnn.py:</p><pre>import kinetic<br>@kinetic.run(<br>    accelerator=&quot;v5p-8&quot;,<br>    requirements=[<br>        &quot;jax[tpu]&quot;,<br>        &quot;flax&quot;,<br>        &quot;jraph&quot;,<br>        &quot;optax&quot;,<br>        &quot;networkx&quot;,<br>        &quot;numpy&quot;,<br>        &quot;pandas&quot;,<br>        &quot;gcsfs&quot;,<br>    ],<br>)<br>def train_remote(csv_path: str, epochs: int = 500):<br>    # CRITICAL: imports happen *inside* the function, not at module level.<br>    # The function body runs on the remote TPU pod, so imports resolve<br>    # against the pod&#39;s installed packages (jax[tpu], libtpu, etc.) -<br>    # not your laptop&#39;s environment.<br>    import os<br>    os.environ[&quot;JAX_PLATFORMS&quot;] = &quot;tpu&quot;<br>    from social_gnn import (<br>        load_csv, build_graph, nx_to_jraph, train, recommend_links,<br>    )<br>    print(f&quot;Loading edges from {csv_path}&quot;)<br>    df = load_csv(csv_path)<br>    G = build_graph(df, mode=&quot;2&quot;)<br>    print(f&quot;Graph: {G.number_of_nodes()} nodes, {G.number_of_edges()} edges&quot;)<br>    graph_data = nx_to_jraph(G)<br>    embeddings, _ = train(graph_data, epochs=epochs)<br>    # Compute a few sample recommendations to send back to the local caller.<br>    sample_nodes = list(graph_data.node_to_idx.keys())[:3]<br>    return {<br>        node: recommend_links(embeddings, graph_data, node, top_k=5)<br>        for node in sample_nodes<br>    }<br><br>if __name__ == &quot;__main__&quot;:<br>    # This runs on the *remote* TPU, but feels like a local function call.<br>    recs = train_remote(&quot;gs://sna-tpu-bucket/relations.csv&quot;, epochs=500)<br>    for node, top_k in recs.items():<br>        print(f&quot;\nTop 5 hashtag recommendations for {node!r}:&quot;)<br>        for tag, score in top_k:<br>            print(f&quot;  {tag:&lt;15} (score {score:.3f})&quot;)</pre><p>A few things worth highlighting:</p><ul><li><strong>All imports go inside the function.</strong> This is deliberate, not a workaround. The function body executes on the remote TPU pod, so import jax needs to resolve against the pod’sjax[tpu] package — not your laptop’s CPU-only JAX (or no JAX at all). If you imported at module level, Kinetic would try to serialize those modules at submission time and either fail or hit version mismatches inside the container.</li><li><strong>requirements=[...] is your remote requirements.txt.</strong>Kinetic uses this to build the container image on the first run. You don’t need any of these packages on your laptop.</li><li><strong>The function returns a normal Python object.</strong> Kinetic serializes the return value and delivers it back to your local process. From your laptop’s point of view,train_remote(...) is just a function that happens to take a while and returns a dictionary. Exceptions are re-raised locally with the full remote traceback.</li></ul><h3>5.4 Launch training</h3><p>From your local terminal — not a TPU VM, just your laptop:</p><pre>python train_kinetic.py</pre><p>The first run takes about five minutes because Kinetic builds the container image. Every subsequent run with unchanged dependencies skips the build entirely and starts in under a minute. <strong>Container caching is the killer feature</strong> — it’s what makes Kinetic feel like local development instead of cloud deployment.</p><p>You’ll see remote logs streamed live:</p><pre>Shipping to TPU via Kinetic...<br>[Stage 1/4] Preflight &amp; packaging...<br>[Stage 2/4] Building container image (5m, cached after this run)...<br>[Stage 3/4] Submitting job to GKE cluster...<br>[Stage 4/4] Executing on TPU v5p-8...<br>[remote] Loading edges from gs://sna-tpu-bucket/relations.csv<br>[remote] Graph: 796 nodes, 1361 edges<br>[remote] Training on device: TpuDevice(id=0, ...)<br>[remote]   epoch    0 | loss 2.2295<br>[remote]   epoch   20 | loss 0.9972<br>...<br>[remote]   epoch  499 | loss 0.8311<br>Job complete. Streaming results to local...</pre><pre>Top 5 hashtag recommendations for &#39;#100daysofcode&#39;:<br>  #webdeveloper     (score 0.774)<br>  #programmer       (score 0.763)<br>  ...</pre><h3>5.5 Tear down (still important!)</h3><p>Kinetic provisions a GKE cluster, and <strong>the cluster’s control plane costs money even when no TPU nodes are active</strong>. Always shut it down when you’re done iterating:</p><pre>kinetic down --yes</pre><p>This removes the cluster, the node pool, and any associated networking — the same one-line cleanup that the manual path needs gcloud compute tpus tpu-vm delete for.</p><h3>Manual setup vs Kinetic — when to use which?</h3><ul><li><strong>Manual setup (Sections 1–4)</strong> is best the first time, when you want to understand what’s actually happening behind the scenes, or when you plan to SSH in for interactive debugging. It’s also fine for one-shot training jobs you don’t intend to repeat.</li><li><strong>Kinetic (Section 5)</strong> is best for iterative work: tweak a hyperparameter, hit python train_kinetic.py, see results, repeat. No SSH, no pip install cycles, no “wait, did I scpthe latest version of the script?” The first run pays a build tax; every run after is fast.</li></ul><p>For this project I started manual to learn the GCP stack, then switched to Kinetic for the actual experimentation phase. Both produce identical model outputs because they’re running the same code on the same hardware — just packaged very differently.</p><h3>Key Takeaways</h3><h3>GraphSAGE is a great default</h3><p>For most “given a graph, learn embeddings” problems, a 2-layer GraphSAGE with mean aggregation and L2 normalization is shockingly effective. It’s simple, fast, and the inductive bias matches what you actually want from a recommender.</p><h3>L2 normalization + temperature is the secret sauce</h3><p>Normalizing embeddings to the unit hypersphere makes dot products into cosine similarities, which is what you want for ranking. But it also flattens the loss landscape — a temperature of 5.0 brings gradients back to a useful magnitude. Skip either, and training stalls.</p><h3>JAX makes “CPU/GPU/TPU” a deployment detail</h3><p>The exact same Python code runs on my laptop, on a GPU, and on a TPU v5p-8 with no changes — JAX picks the device automatically. That’s a huge productivity win compared to the bad old days of writing separate CUDA kernels. It also means you can prototype on CPU, validate on a small GPU, and scale up to TPU without rewriting anything.</p><h3>TPU is the right pick for full-batch GNN training</h3><p>For a workload that’s mostly dense matmuls + segment ops and JITs cleanly into one XLA graph, TPU is faster and cheaper per training step than equivalent GPU spend. The flip side: if your graph is too big for full-batch training and you need fine-grained neighbor sampling, GPU + PyG is still the smoother experience. Match the chip to the workload, not the other way around.</p><h3>Keras Kinetic makes TPUs feel local</h3><p>The @kinetic.run() decorator genuinely eliminates infrastructure friction. The mental model is “I have a function; I want it to run on a TPU” — and Kinetic makes that literal. No Docker, no Kubernetes YAML, no SSH. The first run pays a five-minute container build tax; every iteration after that runs in under a minute. For an experiment-heavy project where you tweak hyperparameters constantly, this is the difference between a productive afternoon and a frustrating one.</p><h3>One-mode projection is underrated</h3><p>If your raw data is bipartite (users ↔︎ items, authors ↔︎ papers, hashtags ↔︎ posts), the mode=&#39;1a&#39; / &#39;1b&#39; projection converts it into a weighted same-type graph. Often this is exactly the input you want for a “find similar users” or “find related items” task — no extra modeling required.</p><h3>Adapting This to Your Domain</h3><p>The pipeline is intentionally generic. To use it on your own data, the only thing you need to change is your CSV: two columns of node IDs and you’re good. Common variations:</p><ul><li><strong>Hashtag co-occurrence (this article)</strong> — node_A, node_B= two hashtags that appeared in the same post, mode=&#39;2&#39;. If you instead have a per-post list, use node_A = post ID, node_B = hashtag, then project with mode=&#39;1b&#39;.</li><li><strong>Twitter/X follower graphs (“People You May Know”)</strong> — node_A = follower, node_B = followee, mode=&#39;2&#39;. Output becomes friend recommendations.</li><li><strong>Co-purchase graphs (“frequently bought together”)</strong> — node_A = customer, node_B = product, mode=&#39;1b&#39; projects onto products that share customers.</li><li><strong>Citation networks</strong> — node_A = paper, node_B = paper, mode=&#39;2&#39;. Output becomes “papers you should cite.”</li><li><strong>Music/podcast co-listening</strong> — node_A = listener, node_B= track, mode=&#39;1b&#39; projects onto tracks frequently heard together.</li></ul><p>The training loop, GraphSAGE model, and link prediction logic stay the same — only the CSV and a label string in main()change.</p><h3>Resources</h3><ul><li><strong>Project repository:</strong> <em>(your GitHub link here)</em></li><li><strong>JAX:</strong> <a href="https://github.com/google/jax">github.com/google/jax</a></li><li><strong>JRaph:</strong> <a href="https://github.com/deepmind/jraph">github.com/deepmind/jraph</a></li><li><strong>Flax:</strong> <a href="https://github.com/google/flax">github.com/google/flax</a></li><li><strong>GraphSAGE paper:</strong> Hamilton et al., “<a href="https://arxiv.org/abs/1706.02216">Inductive Representation Learning on Large Graphs</a>” (NeurIPS 2017)</li><li><strong>Cloud TPU documentation:</strong> <a href="https://cloud.google.com/tpu/docs">cloud.google.com/tpu/docs</a></li></ul><h3>Acknowledgement</h3><ul><li>Google Cloud credits are provided for this project. #TPUSprint</li><li>Thanks to the JAX, Flax, and JRaph teams for building such a clean stack — graph learning on accelerators used to be a research-grade pain.</li><li>Special Thanks to Acxel Derian Effendy, S.Kom. of Mudalab Institut STTS.</li></ul><p>#GraphNeuralNetwork #JAX #JRaph #TPU #GraphSAGE, #HashtagRecommendation #RecommendationSystem #MachineLearning, #LinkPrediction #GoogleCloud</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=447f34b108d9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developer-experts/building-a-hashtag-recommendation-system-with-graph-neural-networks-jraph-and-jax-on-cloud-tpu-447f34b108d9">Building a Hashtag Recommendation System with Graph Neural Networks, JAX, and Keras Kinetic on…</a> was originally published in <a href="https://medium.com/google-developer-experts">Google Developer Experts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MultiAgent Systems for the Visual Learner: Building an Autonomous Studio with Gemini 3 & Veo]]></title>
            <link>https://medium.com/google-developer-experts/multiagent-systems-for-the-visual-learner-building-an-autonomous-studio-with-gemini-3-veo-8ad428d24c21?source=rss-a32fdb99f2d------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ad428d24c21</guid>
            <category><![CDATA[ai-agent-development]]></category>
            <category><![CDATA[multimodal-agents]]></category>
            <category><![CDATA[gemini-3-pro]]></category>
            <category><![CDATA[veo-3]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Esther Irawati Setiawan]]></dc:creator>
            <pubDate>Sat, 20 Dec 2025 16:04:34 GMT</pubDate>
            <atom:updated>2025-12-22T00:12:13.063Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OlfG9dk2dtqmIWsKPZPxaA.png" /></figure><p>Remember when AI was just a chatbot? You typed text, and it gave you text back. Those days are rapidly fading. We are entering the era of <strong>Multimodal Agentic Systems</strong> — where AI doesn’t just “talk,” but acts, creates, and produces media.</p><p>For the visual learner, this is a game-changer. Imagine a system where you act as the client, and an AI “Studio Manager” delegates your requests to specialized “Artists” and “Directors” to produce high-fidelity images and cinematic videos.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_k0HpgChixk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_k0HpgChixk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F_k0HpgChixk%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2b31d744001a9d28a2024086228f8b9/href">https://medium.com/media/e2b31d744001a9d28a2024086228f8b9/href</a></iframe><p>In this article, we will move beyond simple prompt engineering to build a fully autonomous <strong>Multimedia Studio</strong> using the <strong>Google Agent Development Kit (ADK)</strong>, <strong>Gemini 3 Pro</strong>, and <strong>Veo 2.0</strong>.</p><h3>The Architecture: Why MultiAgent?</h3><p>Before writing a single line of code, we must understand the architecture. You might ask, “Why not just one big script?”.</p><p>The reality is that a single model often struggles with context switching. If you ask a general model to “make a video,” it might try to write a script instead of generating the actual video file. By using <strong>Specialized Agents</strong>, we create a robust pipeline:</p><ul><li><strong>The Studio Manager (Root Agent):</strong> The “Brain.” It understands user intent. If you say “draw me a logo,” it routes you to the Image Department. If you say “film a scene,” it routes you to the Video Department.</li><li><strong>The Image Specialist:</strong> The “Artist.” It focuses solely on composition, lighting, and style prompts.</li><li><strong>The Video Specialist:</strong> The “Director.” It focuses on camera movement and motion dynamics.</li></ul><h3>Setting the Stage: Prerequisites</h3><p>Before we dive into the code, let’s ensure your development environment is primed. To follow this tutorial effectively, you will need a Python environment ready with a specific set of libraries.</p><p>Open your terminal and run the following command to install the necessary dependencies:</p><pre>pip install google-adk google-genai requests pillow</pre><h3>🧰 What’s in the toolkit?</h3><p>Here is a breakdown of why we need each of these libraries:</p><ul><li><strong>google-adk</strong> This serves as the framework for <strong>orchestration</strong>, helping to manage the flow of the application.</li><li><strong>google-genai</strong> The most critical piece of the puzzle. This SDK provides access to Google&#39;s <strong>Gemini</strong> and <strong>Veo</strong> models.</li><li><strong>requests</strong> We need this library to handle HTTP requests. It is crucial for <strong>handling video downloads</strong>, specifically because the Veo model generates files asynchronously, and we need a way to retrieve them once they are ready.</li><li><strong>pillow</strong> The standard Python Imaging Library (PIL fork), used here for processing <strong>image binary data</strong>.</li></ul><h3>Part 1: Defining the “Hands” (The Physical Tools)</h3><p>Agents are the “brains,” but they need “hands” to interact with the world. We define two physical functions (Tools) that perform the actual API calls.</p><h3>1. The Artist’s Brush: Gemini 3 Pro Image</h3><p>This tool utilizes <strong>Gemini 3 Pro Image</strong>. It takes a prompt, calls the API, processes the raw binary data, saves it locally, and returns an HTML snippet for the UI.</p><p><strong>Key Implementation Detail:</strong> We use <strong>Local Imports</strong> inside the function. In ADK, tools are often sandboxed. Importing libraries like genai or PIL <em>inside</em> the function prevents execution errors.</p><pre>def tool_generate_image(prompt: str) -&gt; dict:<br>    # ... Local imports ...<br>    # API Call to Gemini 3 Pro Image<br>    response = client.models.generate_content(<br>        model=&quot;gemini-3-pro-image-preview&quot;,<br>        contents=prompt,<br>        config=types.GenerateContentConfig(response_modalities=[&quot;IMAGE&quot;])<br>    )<br>    # Processing binary data to JPG...<br>    # Returning HTML...</pre><p><strong>Function Specification</strong></p><p>Now, let’s look at the contract for the function we are building. It takes a simple text input and returns a structured object containing our results.</p><ul><li><strong>Input:</strong> prompt (str) A descriptive string.</li><li><strong>Output:</strong> dict Contains status and an HTML report.</li></ul><h3>2. The Director’s Camera: Veo 2.0 (The Asynchronous Challenge)</h3><p>Video generation is more complex because it is <strong>asynchronous</strong>. When we ask Veo to make a video, it returns a “Job ID” immediately, but the video takes 1–2 minutes to render.</p><p>Our tool must handle <strong>Polling</strong> (waiting and checking status) and <strong>Parsing</strong> the specific JSON structure of Veo 2.0.</p><p>We use manual polling with requests to bypass SDK wrapper issues and ensure reliable execution.</p><pre>def tool_generate_video(prompt: str) -&gt; dict:<br>    # 1. Start the Job<br>    raw_resp = client.models.generate_videos(<br>       model=&quot;veo-2.0-generate-001&quot;, ...<br>    )<br>    # 2. Polling Loop (Manual HTTP Requests)<br>    while True:<br>        resp = requests.get(poll_url)<br>        # Check if &#39;done&#39; is true...<br>        # Parse the JSON to find &#39;video_uri&#39;...<br>        <br>    # 3. Authenticated Download<br>    dl = requests.get(video_uri, params={&#39;key&#39;: api_key})<br>    # Save mp4 file...</pre><p><strong>Function Specification</strong></p><p>This function is specialized for the visual component, likely leveraging the Veo model mentioned in the prerequisites.</p><ul><li><strong>Input:</strong> prompt (str) A <strong>cinematic description</strong> detailing the scene, camera movement, or visual style required.</li><li><strong>Output:</strong> dict A dictionary containing the status and an embedded <strong>HTML video player</strong> to display the result.</li></ul><h3>Part 2: The Memory (State Management)</h3><p>How does the Manager pass the user’s request to the Specialist? We use a shared <strong>State</strong>.</p><p>We define a helper tool called append_to_state. When the user says &quot;I want a video of a cat,&quot; the Manager calls this tool to save &quot;video of a cat&quot; into a variable called USER_REQUEST. The Video Specialist then reads this variable to begin its work.</p><p>Below is a foundational method for ensuring your AI tools can write to a shared persistent memory.</p><pre>def append_to_state(tool_context: ToolContext, field: str, response: str) -&gt; dict:<br>    &quot;&quot;&quot;Saves text to the shared memory context.&quot;&quot;&quot;<br>    # ... logic to append data to tool_context.state ...<br>    existing_state = tool_context.state.get(field, [])<br>    if isinstance(existing_state, list):<br>        tool_context.state[field] = existing_state + [response]<br>    else:<br>        tool_context.state[field] = [response]<br>        <br>    logging.info(f&quot;[STATE UPDATE] Added to {field}: {response}&quot;)<br>    return {&quot;status&quot;: &quot;success&quot;}</pre><p><strong>Breaking Down the Architecture</strong></p><ul><li><strong>tool_context: ToolContext</strong>: This is the &quot;brain&quot; of the operation. It represents the shared state object that persists across the entire lifecycle of the agent&#39;s task.</li><li><strong>field: str</strong>: This allows for categorized memory. Instead of a giant blob of text, you can organize data under specific keys (e.g., &quot;user_preferences&quot; or &quot;search_results&quot;).</li><li><strong>response: str</strong>: The actual payload of information being stored.</li></ul><h3>Part 3: The Brains (The Agents)</h3><p>Now we define the personalities of our agents.</p><h3>The Image Specialist</h3><p>This agent acts as a professional prompter. It doesn’t just pass “cat” to the model; it expands it to “A fluffy Persian cat sitting on a velvet sofa, cinematic lighting, 8k resolution”.</p><pre>image_agent = Agent(<br>    name=&quot;image_specialist&quot;,<br>    instruction=&quot;&quot;&quot;<br>    CONTEXT: { USER_REQUEST? }<br>    INSTRUCTIONS:<br>    1. Read USER_REQUEST.<br>    2. Enhance the prompt (add style, lighting).<br>    3. Call tool_generate_image.<br>    &quot;&quot;&quot;,<br>    tools=[tool_generate_image]<br>)</pre><h3>The Video Specialist</h3><p>Similarly, this agent focuses on <strong>motion</strong>. It ensures the prompt includes camera instructions (e.g., “slow pan,” “drone shot”) and warns the user about the wait time.</p><pre>video_agent = Agent(<br>    name=&quot;video_specialist&quot;,<br>    instruction=&quot;&quot;&quot;<br>    CONTEXT: { USER_REQUEST? }<br>    INSTRUCTIONS:<br>    1. Read USER_REQUEST.<br>    2. Enhance prompt with CAMERA ANGLES and ATMOSPHERE.<br>    3. Call tool_generate_video.<br>    4. Warn user about the wait time.<br>    &quot;&quot;&quot;,<br>    tools=[tool_generate_video]<br>)</pre><h3>Part 4: The Manager (The Root Agent)</h3><p>Finally, we tie it all together. In Google ADK, we use the sub_agents parameter to create this hierarchy.</p><p>The Manager doesn’t need to know <em>how</em> to make a video; it just knows <em>who</em> can make it.</p><pre>root_agent = Agent(<br>    name=&quot;multimedia_manager&quot;,<br>    instruction=&quot;&quot;&quot;<br>    You are the Studio Manager.<br>    1. Ask user for their request.<br>    2. Save it using `append_to_state` to &#39;USER_REQUEST&#39;.<br>    3. Delegate to `image_specialist` OR `video_specialist`.<br>    &quot;&quot;&quot;,<br>    sub_agents=[image_agent, video_agent], # &lt;-- The Hierarchy<br>    tools=[append_to_state]<br>)</pre><h4>Full Implementation Code</h4><p>Here is the complete, production-ready code for agent.py. It includes all the fixes for local imports, JSON parsing for Veo 2.0, and state management.</p><pre>import os<br>import logging<br>import base64<br>import io<br>import time<br>import requests<br>import json<br>from PIL import Image<br><br>from google.adk.agents import Agent<br>from google.adk.tools.tool_context import ToolContext<br>from google.genai import types<br><br># Setup Logging<br>logging.basicConfig(level=logging.INFO)<br><br># =============================================================================<br># 1: HELPER TOOLS (STATE MANAGEMENT)<br># =============================================================================<br><br>def append_to_state(<br>    tool_context: ToolContext, field: str, response: str<br>) -&gt; dict[str, str]:<br>    &quot;&quot;&quot;<br>    saves text to state (memory) for operation to sub-agents.<br>    &quot;&quot;&quot;<br>    existing_state = tool_context.state.get(field, [])<br>    if isinstance(existing_state, list):<br>        tool_context.state[field] = existing_state + [response]<br>    else:<br>        tool_context.state[field] = [response]<br>        <br>    logging.info(f&quot;[STATE UPDATE] Added to {field}: {response}&quot;)<br>    return {&quot;status&quot;: &quot;success&quot;}<br><br># =============================================================================<br># 2: PHYSICAL TOOLS<br># =============================================================================<br><br>def tool_generate_image(prompt: str) -&gt; dict:<br>    &quot;&quot;&quot;Generates a high-quality static image using Gemini 3 Pro Image.&quot;&quot;&quot;<br>    <br>    from google import genai <br>    from google.genai import types<br>    import os<br>    import io<br>    import base64<br>    from PIL import Image<br>    # -------------------------------------------------<br><br>    print(f&quot;   🎨 [PHYSICAL TOOL] Drawing: {prompt[:30]}...&quot;)<br>    try:<br>        api_key = os.environ.get(&quot;GEMINI_API_KEY&quot;) or os.environ.get(&quot;GOOGLE_API_KEY&quot;)<br>        client = genai.Client(api_key=api_key)<br>        <br>        response = client.models.generate_content(<br>            model=&quot;gemini-3-pro-image-preview&quot;,<br>            contents=prompt,<br>            config=types.GenerateContentConfig(response_modalities=[&quot;IMAGE&quot;])<br>        )<br>        <br>        for part in response.candidates[0].content.parts:<br>            if part.inline_data and part.inline_data.data:<br>                filename = &quot;generated_image.jpg&quot;<br>                file_path = os.path.abspath(filename)<br>                <br>                image = Image.open(io.BytesIO(part.inline_data.data))<br>                if image.mode in (&quot;RGBA&quot;, &quot;P&quot;): image = image.convert(&quot;RGB&quot;)<br>                image.save(filename, quality=85)<br>                <br>                buffer = io.BytesIO()<br>                image.thumbnail((600, 600))<br>                image.save(buffer, format=&quot;JPEG&quot;, quality=70)<br>                buffer.seek(0)<br>                b64_str = base64.b64encode(buffer.read()).decode(&#39;utf-8&#39;)<br>                <br>                return {<br>                    &quot;status&quot;: &quot;success&quot;, <br>                    &quot;report&quot;: f&quot;✅ Image Saved!\n&lt;img src=&#39;data:image/jpeg;base64,{b64_str}&#39; width=&#39;500&#39; style=&#39;border-radius:10px;&#39;/&gt;&quot;<br>                }<br>    except Exception as e:<br>        return {&quot;status&quot;: &quot;error&quot;, &quot;report&quot;: f&quot;Image Error: {str(e)}&quot;}<br>    return {&quot;status&quot;: &quot;error&quot;, &quot;report&quot;: &quot;No image returned.&quot;}<br><br>def tool_generate_video(prompt: str) -&gt; dict:<br>    &quot;&quot;&quot;Generates a video using Veo 2.0 with Manual Polling Fix.&quot;&quot;&quot;<br>    <br>    from google import genai<br>    from google.genai import types<br>    import requests<br>    import time<br>    import os<br>    # -------------------------------------------------<br><br>    print(f&quot;   🎬 [PHYSICAL TOOL] Filming: {prompt[:30]}...&quot;)<br>    try:<br>        api_key = os.environ.get(&quot;GEMINI_API_KEY&quot;) or os.environ.get(&quot;GOOGLE_API_KEY&quot;)<br>        client = genai.Client(api_key=api_key)<br>        <br>        # 1. Start Job<br>        raw_resp = client.models.generate_videos(<br>            model=&quot;veo-2.0-generate-001&quot;,<br>            prompt=prompt,<br>            config=types.GenerateVideosConfig(number_of_videos=1, aspect_ratio=&quot;16:9&quot;)<br>        )<br>        op_name = raw_resp.name if hasattr(raw_resp, &#39;name&#39;) else str(raw_resp)<br>        print(f&quot;   --&gt; Job ID: {op_name}&quot;)<br>        <br>        # 2. Polling<br>        poll_url = f&quot;https://generativelanguage.googleapis.com/v1beta/{op_name}?key={api_key}&quot;<br>        start = time.time()<br>        <br>        while True:<br>            if time.time() - start &gt; 600: return {&quot;status&quot;: &quot;error&quot;, &quot;report&quot;: &quot;Timeout.&quot;}<br>            resp = requests.get(poll_url)<br>            if resp.status_code != 200: time.sleep(5); continue<br>            <br>            data = resp.json()<br>            if data.get(&#39;done&#39;):<br>                if &#39;error&#39; in data: return {&quot;status&quot;: &quot;error&quot;, &quot;report&quot;: str(data[&#39;error&#39;])}<br>                <br>                # Parsing Logic<br>                video_uri = None<br>                try:<br>                    inner = data.get(&#39;response&#39;) or data.get(&#39;result&#39;)<br>                    if inner and &#39;generateVideoResponse&#39; in inner:<br>                        video_uri = inner[&#39;generateVideoResponse&#39;][&#39;generatedSamples&#39;][0][&#39;video&#39;][&#39;uri&#39;]<br>                    elif inner and &#39;generatedVideos&#39; in inner:<br>                        video_uri = inner[&#39;generatedVideos&#39;][0][&#39;video&#39;][&#39;uri&#39;]<br>                except: pass<br>                <br>                if not video_uri: return {&quot;status&quot;: &quot;error&quot;, &quot;report&quot;: &quot;Video finished but URL missing.&quot;}<br>                <br>                # 3. Download<br>                dl = requests.get(video_uri, params={&#39;key&#39;: api_key}, stream=True)<br>                if dl.status_code == 200:<br>                    filename = &quot;generated_movie.mp4&quot;<br>                    with open(filename, &#39;wb&#39;) as f:<br>                        for chunk in dl.iter_content(8192): f.write(chunk)<br>                    return {<br>                        &quot;status&quot;: &quot;success&quot;,<br>                        &quot;report&quot;: f&quot;✅ Video Saved!\n&lt;video width=&#39;600&#39; controls autoplay muted&gt;&lt;source src=&#39;generated_movie.mp4&#39; type=&#39;video/mp4&#39;&gt;&lt;/video&gt;&quot;<br>                    }<br>                return {&quot;status&quot;: &quot;error&quot;, &quot;report&quot;: &quot;Download failed.&quot;}<br>            <br>            time.sleep(5)<br><br>    except Exception as e:<br>        return {&quot;status&quot;: &quot;error&quot;, &quot;report&quot;: f&quot;System Error: {str(e)}&quot;}<br><br># =============================================================================<br># SUB-AGENTS (SPECIALISTS)<br># =============================================================================<br><br>image_agent = Agent(<br>    name=&quot;image_specialist&quot;,<br>    model=&quot;gemini-3-pro-preview&quot;,<br>    description=&quot;Specialist agent that creates static images, photos, and infographics.&quot;,<br>    instruction=&quot;&quot;&quot;<br>    CONTEXT FROM MANAGER:<br>    { USER_REQUEST? }<br><br>    INSTRUCTIONS:<br>    - Read the &#39;USER_REQUEST&#39; from the state.<br>    - Create a highly detailed English prompt based on that request (add lighting, style).<br>    - Use the `tool_generate_image` to create the visual.<br>    - Return the result to the user.<br>    &quot;&quot;&quot;,<br>    tools=[tool_generate_image]<br>)<br><br>video_agent = Agent(<br>    name=&quot;video_specialist&quot;,<br>    model=&quot;gemini-3-pro-preview&quot;,<br>    description=&quot;Specialist agent that creates videos, movies, and animations using Veo.&quot;,<br>    instruction=&quot;&quot;&quot;<br>    CONTEXT FROM MANAGER:<br>    { USER_REQUEST? }<br><br>    INSTRUCTIONS:<br>    - Read the &#39;USER_REQUEST&#39; from the state.<br>    - Create a prompt describing MOTION, CAMERA ANGLES, and ATMOSPHERE.<br>    - Use the `tool_generate_video` to create the video.<br>    - Tell the user to wait a moment while you render it.<br>    &quot;&quot;&quot;,<br>    tools=[tool_generate_video]<br>)<br><br># =============================================================================<br># ROOT AGENT (MANAGER)<br># =============================================================================<br><br>root_agent = Agent(<br>    name=&quot;multimedia_manager&quot;,<br>    model=&quot;gemini-3-pro-preview&quot;,<br>    description=&quot;The main receptionist. Greets user and routes tasks to specialists.&quot;,<br>    instruction=&quot;&quot;&quot;<br>    You are the Manager of a Multimedia Studio.<br>    <br>    1. Ask the user what they want to create today (Image or Video?).<br>    2. When they respond:<br>       - Use `append_to_state` tool to save their description into the &#39;USER_REQUEST&#39; field.<br>       - Then, delegate the task to the correct specialist (`image_specialist` or `video_specialist`).<br>       <br>    Example flow:<br>    User: &quot;I want a video of a cat.&quot;<br>    You: Call append_to_state(field=&#39;USER_REQUEST&#39;, response=&#39;video of a cat&#39;) -&gt; Delegate to video_specialist.<br>    &quot;&quot;&quot;,<br>    sub_agents=[image_agent, video_agent], <br>    tools=[append_to_state]<br>)<br><br></pre><h3>How to Run Your Studio</h3><p>The easiest way to test your agent is using the built-in web UI provided by ADK.</p><ul><li>Open your terminal in the project directory.</li><li>Run command: adk web</li><li>Open <a href="http://localhost:8000.">http://localhost:8000</a></li></ul><p><strong>Example Workflow:</strong></p><ul><li><strong>User:</strong> “Create a video showing a mind map for learning Python.”</li><li><strong>Manager:</strong> (Saves request, routes to Video Specialist).</li><li><strong>Video Specialist:</strong> “I am starting the render… Please wait 1–2 minutes.”</li><li><strong>Result:</strong> The system displays the generated video player and saves generated_movie.mp4</li></ul><h3>Conclusion</h3><p>By combining Gemini 2.5, Veo 2.0, and the Google ADK, we have built a system that understands intent and orchestrates complex, asynchronous media generation tasks. This MultiAgent pattern is scalable — you could easily add an “Audio Specialist” or “Copywriter Specialist” to this team in the future.</p><p>Special Thank You to Kevin Jonathan Halim for the collaboration in creating this article and demo</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ad428d24c21" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developer-experts/multiagent-systems-for-the-visual-learner-building-an-autonomous-studio-with-gemini-3-veo-8ad428d24c21">MultiAgent Systems for the Visual Learner: Building an Autonomous Studio with Gemini 3 &amp; Veo</a> was originally published in <a href="https://medium.com/google-developer-experts">Google Developer Experts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Intelligent Social Network Analysis Assistant with Gemini 3 Pro and Flask]]></title>
            <link>https://medium.com/google-developer-experts/building-an-intelligent-social-network-analysis-assistant-with-gemini-3-pro-and-flask-f0f5c7473b83?source=rss-a32fdb99f2d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0f5c7473b83</guid>
            <category><![CDATA[social-network-analysis]]></category>
            <category><![CDATA[computation-graph]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[aireasoning]]></category>
            <dc:creator><![CDATA[Esther Irawati Setiawan]]></dc:creator>
            <pubDate>Sat, 20 Dec 2025 10:56:37 GMT</pubDate>
            <atom:updated>2025-12-26T02:28:31.344Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AyRKh4Rvn1hem2xWHFlfbg.png" /></figure><p>Have you ever looked at a complex network diagram — like a map of Game of Thrones characters or Twitter connections — and wished you could just ask it questions?</p><blockquote>“Who is the most popular person here?”</blockquote><blockquote>“How does Person A know Person B?”</blockquote><p>In the world of data science, visualizing connections is only half the battle. The real magic happens when you can “talk” to your data. In this guide, we will build an <strong>SNA (Social Network Analysis) Interactive Assistant</strong> that combines powerful graph algorithms with the reasoning capabilities of <strong>Gemini 3 Pro Preview</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DbHLOB4LrCy0&amp;type=text%2Fhtml&amp;schema=youtube&amp;display_name=YouTube&amp;src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FbHLOB4LrCy0%3Ffeature%3Doembed" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfb4e519a8243bee4b09ecaea58d5346/href">https://medium.com/media/dfb4e519a8243bee4b09ecaea58d5346/href</a></iframe><h3>Phase 1: Obtaining Your AI Credentials</h3><p>Before we touch the code, we need to authorize our AI “brain”3.</p><ol><li>Navigate to <a href="https://aistudio.google.com/?authuser=1"><strong>aistudio.google.com</strong></a></li><li>Log in with your Gmail account.</li><li>Click <strong>“Get API key”</strong>.</li><li>Select <strong>“Create API key in new project”</strong>.</li><li>Copy the generated code and save it in a temporary Notepad file.</li></ol><h3>Phase 2: Setting Up the Workspace</h3><p>Organization is key to a successful project9.</p><ol><li>Create a new folder on your Desktop or in Documents named <strong>SNA_Project</strong>.</li><li>Inside this folder, we will create our configuration file.</li></ol><p><strong>File 1: </strong><strong>.env (The Secure Key Vault)</strong></p><p>Create a file named .env and paste your key following this format:</p><pre>GOOGLE_API_KEY=PASTE_YOUR_API_KEY_HERE</pre><p><em>(Replace the placeholder with the code you copied in Phase 1)</em></p><h3>Phase 3: Developing the System Core (main.py)</h3><p>Now, let’s dive into the core script. This file, <strong>main.py</strong>, acts as the engine of our application. We begin by importing the libraries necessary for data processing, graph theory, and web serving.</p><pre>import pandas as pd<br>import networkx as nx<br>import igraph as ig<br>import json<br>import plotly<br>import plotly.graph_objects as go<br>from flask import Flask, request, render_template_string, jsonify<br>import time<br>import os<br>from dotenv import load_dotenv<br>#--- 1. SETUP ENVIRONMENT &amp; AUTO-DETECT MODEL ---<br>load_dotenv()<br>ACTIVE_MODEL_NAME = None<br>GEMINI_AVAILABLE = False<br>CHAT_SESSION = None # Variable to store the session<br>GLOBAL_IG_OBJECT = None<br>GLOBAL_NODE_NAMES = []<br>GLOBAL_NODE_MAP = {} # Name -&gt; Index mappingIntelligence Initialization &amp; Model Discovery</pre><p>Our script is designed to be future-proof. It automatically scans your account to find the best available model, prioritizing the <strong>Gemini 3</strong> series for its advanced reasoning.</p><pre>try:<br>    import google.generativeai as genai<br>    api_key = os.getenv(&quot;GOOGLE_API_KEY&quot;)<br>    if api_key:<br>        genai.configure(api_key=api_key)<br>        #--- AUTO-DISCOVERY MODEL<br>        available_models = [m.name for m in genai.list_models() if &#39;generateContent&#39; in m.supported_generation_methods]<br>        <br>        # Priority list for Gemini 3<br>        priority_list = [<br>            &quot;models/gemini-3-pro-preview&quot;,<br>            &quot;models/gemini-3-flash-preview&quot;,<br>            &quot;models/gemini-1.5-flash&quot;,<br>            &quot;models/gemini-1.5-pro&quot;<br>        ]<br>        for p in priority_list:<br>            if p in available_models:<br>                ACTIVE_MODEL_NAME = p<br>                break<br>        if ACTIVE_MODEL_NAME:<br>            GEMINI_AVAILABLE = True<br>            print(f&quot; SUCCESS: Using model &#39;{ACTIVE_MODEL_NAME}&#39;&quot;)<br>except Exception as e:<br>    print(f&quot; Initialization failed: {e}&quot;)</pre><h3>Phase 4: Equipping the Assistant with Tools</h3><p>To prevent the AI from guessing connections, we give it specific Python functions called <strong>Tools</strong>.</p><h3>Tool: Shortest Path Finder</h3><p>This tool calculates the exact route between two nodes in the network using the igraph library. This function calculates the shortest path between two nodes using Breadth-First Search (BFS) or Dijkstra. It even includes &quot;fuzzy search&quot; logic, so if a user types &quot;jon snow&quot; (lowercase), it finds &quot;Jon-Snow&quot; (capitalized).</p><pre>def tool_get_shortest_path(source_node: str, target_node: str):<br>    &quot;&quot;&quot;Finds the shortest path between two nodes. Use this if the user asks about relationships.&quot;&quot;&quot;<br>    global GLOBAL_IG_OBJECT, GLOBAL_NODE_MAP, GLOBAL_NODE_NAMES<br>    src_idx = GLOBAL_NODE_MAP.get(source_node)<br>    tgt_idx = GLOBAL_NODE_MAP.get(target_node)<br>    <br>    # Case-insensitive fallback<br>    if src_idx is None:<br>        found = next((k for k in GLOBAL_NODE_MAP if k.lower() == source_node.lower()), None)<br>        if found: src_idx = GLOBAL_NODE_MAP[found]<br>    <br>    try:<br>        paths = GLOBAL_IG_OBJECT.get_shortest_paths(src_idx, to=tgt_idx, output=&quot;vpath&quot;)<br>        path_names = [GLOBAL_NODE_NAMES[i] for i in paths[0]]<br>        return f&quot;Shortest Path: {&#39; -&gt; &#39;.join(path_names)}&quot;<br>    except:<br>        return &quot;Path not found.&quot;</pre><h3>Tool: Neighbor Lookup</h3><p>This allows the assistant to see exactly who a specific node is connected to. This function simply looks up who is directly connected to a specific person. To prevent overwhelming the AI, we limit the output to 20 names.</p><pre>def tool_get_neighbors(node_name: str):<br>    &quot;&quot;&quot;Finds direct connections for a node. Use for questions like &#39;who is X connected to?&#39;.&quot;&quot;&quot;<br>    idx = GLOBAL_NODE_MAP.get(node_name)<br>    neighbors_idx = GLOBAL_IG_OBJECT.neighbors(idx)<br>    neighbor_names = [GLOBAL_NODE_NAMES[i] for i in neighbors_idx]<br>    return f&quot;Node &#39;{node_name}&#39; has {len(neighbor_names)} connections: {&#39;, &#39;.join(neighbor_names[:20])}&quot;<br>AGENT_TOOLS = [tool_get_shortest_path, tool_get_neighbors]</pre><h3>Phase 5: The Interactive Interface</h3><p>We use a Flask-based HTML template to display the graph using <strong>Plotly</strong>. This allows users to upload files, view stats, and chat with the assistant in a modern UI.</p><pre>app = Flask(__name__)<br>HTML_TEMPLATE = &quot;&quot;&quot;<br>&lt;div id=&quot;chat-box&quot;&gt;<br>    &lt;div id=&quot;chat-header&quot;&gt;Gemini Assistant&lt;/div&gt;<br>    &lt;div id=&quot;chat-messages&quot;&gt;&lt;/div&gt;<br>    &lt;div id=&quot;chat-input-area&quot;&gt;<br>        &lt;input type=&quot;text&quot; id=&quot;chat-input&quot; placeholder=&quot;Ask about this graph...&quot;&gt;<br>        &lt;button id=&quot;chat-send&quot; onclick=&quot;sendMessage()&quot;&gt;Send&lt;/button&gt;<br>    &lt;/div&gt;<br>&lt;/div&gt;<br>&quot;&quot;&quot;</pre><h3>Phase 6: Processing the Graph</h3><p>When a CSV is uploaded, the script cleans the data and detects clusters (groups) within the network. We need a function to process the user’s uploaded CSV. This function converts the raw data into a graph structure.</p><p>We use a “hybrid” approach:</p><ol><li><strong>NetworkX</strong> to clean the data and remove self-loops.</li><li><strong>iGraph</strong> to perform heavy calculations (Community Detection and Force-Directed Layouts) because it is much faster.</li></ol><pre>def process_graph_fast(df):<br>    global GLOBAL_IG_OBJECT, GLOBAL_NODE_NAMES, GLOBAL_NODE_MAP, CHAT_SESSION<br>    CHAT_SESSION = None # Reset session for new data<br>    src_col = next((c for c in df.columns if c in [&#39;source&#39;, &#39;from&#39;, &#39;src&#39;]), df.columns[0])<br>    tgt_col = next((c for c in df.columns if c in [&#39;target&#39;, &#39;to&#39;, &#39;dst&#39;]), df.columns[1])<br>    <br>    G_nx = nx.Graph()<br>    G_nx.add_edges_from(list(zip(df[src_col].astype(str), df[tgt_col].astype(str))))<br>    <br>    nodes_list = list(G_nx.nodes())<br>    node_map = {name: i for i, name in enumerate(nodes_list)}<br>    ig_edges = [(node_map[u], node_map[v]) for u, v in G_nx.edges()]<br>    G_ig = ig.Graph(n=len(nodes_list), edges=ig_edges)<br>    <br>    GLOBAL_IG_OBJECT = G_ig<br>    GLOBAL_NODE_NAMES = nodes_list<br>    GLOBAL_NODE_MAP = node_map<br>    <br>    # Community detection<br>    try: partition = G_ig.community_multilevel()<br>    except: pass<br>    return {&quot;num_edges&quot;: G_ig.ecount(), &quot;adj_list&quot;: G_ig.get_adjlist(), &quot;nodes_list&quot;: nodes_list}</pre><h3>Phase 7: Interactive Chat Logic</h3><p>The final piece is the /chat route. This is where the Gemini model is initialized with <strong>Automatic Function Calling</strong>, allowing it to use our tools as needed.</p><pre>@app.route(&#39;/chat&#39;, methods=[&#39;POST&#39;])<br>def chat():<br>    global CHAT_SESSION<br>    user_msg = request.json.get(&#39;message&#39;, &quot;&quot;)<br>    if CHAT_SESSION is None:<br>        model = genai.GenerativeModel(<br>            model_name=ACTIVE_MODEL_NAME,<br>            tools=AGENT_TOOLS,<br>            system_instruction=&quot;You are an SNA Assistant. Use tools to find paths and neighbors. Don&#39;t guess.&quot;<br>        )<br>        CHAT_SESSION = model.start_chat(enable_automatic_function_calling=True)<br>    <br>    response = CHAT_SESSION.send_message(user_msg)<br>    return jsonify({&quot;response&quot;: response.text})<br>if __name__ == &#39;__main__&#39;:<br>    app.run(debug=True, port=5000)</pre><h3>Installation &amp; Usage</h3><p>To get this running on your machine:</p><ul><li>Install Libraries: Open your terminal and run:</li></ul><pre>pip install pandas networkx igraph plotly flask python-dotenv google-generativeai</pre><ul><li><strong>Run:</strong> Execute python main.py and navigate to<a href="http://127.0.0.1:500027272727.">http://127.0.0.1:5000</a></li><li><strong>Analyze:</strong> Upload your CSV file and start a conversation with the Gemini 3!</li></ul><blockquote>Check out the full code here: <a href="https://github.com/estherirawati/snallm">https://github.com/estherirawati/snallm</a></blockquote><h3>Conclusion: A New Era of Human-Centric Data Analysis</h3><p>We have seen how <strong>Gemini 3 Pro</strong> and <strong>Flash</strong> transform rigid CSV data into a living ecosystem of information. By merging <strong>SNA</strong> computational power with <strong>Generative AI</strong> reasoning, we no longer just “look” at data — we “dialogue” with it. By combining a visual graph engine with an LLM, we’ve created a tool that bridges the gap between raw data and human understanding. You don’t just <em>see</em> the network; you communicate with it.</p><p>This interactive system opens endless possibilities, from detecting complex fraud patterns and understanding social media community dynamics to accelerating collaborative research. Its true value lies in simplifying complexity, allowing for faster and more accurate strategic decisions.</p><p>Enjoy experimenting with your own networks. Remember that behind every node and edge lies a story waiting to be told. <strong>Happy analyzing!</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DbHLOB4LrCy0&amp;type=text%2Fhtml&amp;schema=youtube&amp;display_name=YouTube&amp;src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FbHLOB4LrCy0%3Ffeature%3Doembed" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfb4e519a8243bee4b09ecaea58d5346/href">https://medium.com/media/dfb4e519a8243bee4b09ecaea58d5346/href</a></iframe><p>Special thank you to Billie Nathaniel Phalosa for collaborating on this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0f5c7473b83" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developer-experts/building-an-intelligent-social-network-analysis-assistant-with-gemini-3-pro-and-flask-f0f5c7473b83">Building an Intelligent Social Network Analysis Assistant with Gemini 3 Pro and Flask</a> was originally published in <a href="https://medium.com/google-developer-experts">Google Developer Experts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build and Deploy a Live AI Travel Agent in Minutes with Google ADK, Google Search, and Cloud Run]]></title>
            <link>https://medium.com/google-cloud/build-and-deploy-a-live-ai-travel-agent-in-minutes-with-google-adk-google-search-and-cloud-run-fd29d11d3e56?source=rss-a32fdb99f2d------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd29d11d3e56</guid>
            <dc:creator><![CDATA[Esther Irawati Setiawan]]></dc:creator>
            <pubDate>Sat, 02 Aug 2025 03:06:25 GMT</pubDate>
            <atom:updated>2025-08-04T08:10:54.107Z</atom:updated>
            <content:encoded><![CDATA[<h3>A complete, step-by-step guide to creating an agent using a 100% cloud-native workflow — no local setup required.</h3><p>In the ever-evolving landscape of AI, agents are becoming increasingly powerful tools for automating tasks and providing intelligent responses. Google’s Agent Development Kit (ADK) offers a streamlined framework for developers to create their own AI agents.</p><p>In this tutorial, we will walk through the entire process, from setup to deployment, to build a “Travel Buddy” agent to provide weather, time, and practical travel guidance for any city in the world.</p><h3>The Agent’s Capabilities</h3><p>Our “Travel Buddy” agent will have three primary functions, all powered by live Google Search results:</p><ul><li><strong>Get Current Weather:</strong> Takes a city name, performs a Google search, and extracts the temperature and weather conditions from the top results.</li><li><strong>Get Current Time:</strong> Takes a city name, performs a Google search, and extracts the current local time.</li><li><strong>Get Travel Information:</strong> Takes a city name and a topic (e.g., “power outlets,” “culture,” “transportation”), performs a Google search, and summarizes the content from a reliable source.</li></ul><h3>The Initial Setup in Google Cloud Shell</h3><p>To ensure a smooth process with no local installation, we will perform all development directly in Google Cloud Shell.</p><h3>Step 1: Open Cloud Shell &amp; Editor</h3><ol><li>Navigate to the <a href="https://console.cloud.google.com/">Google Cloud Console</a>.</li><li>In the top-right corner, click the <strong>“Activate Cloud Shell”</strong> icon.</li><li>Once the terminal appears at the bottom of the screen, click <strong>“Open Editor”</strong> for a more comfortable, full-screen working environment.</li></ol><h3>Step 2: Set Up Your Google Cloud Project &amp; APIs</h3><ol><li>In the Cloud Shell terminal, set your active Google Cloud project. Replace YOUR-PROJECT-ID with your actual project ID.</li></ol><pre>gcloud config set project YOUR-PROJECT-ID</pre><p>2. Enable the Vertex AI API, which the agent requires.</p><pre>gcloud services enable aiplatform.googleapis.com</pre><h3>Step 3: Create the Project Structure</h3><p>A correct folder structure is crucial. Run the following command in the terminal to create a directory structure that follows the official Codelabs best practice.</p><pre># Create the main folder structure<br>mkdir -p travel_project/agents/travel_buddy<br># Create the necessary __init__.py files<br>touch travel_project/__init__.py<br>touch travel_project/agents/travel_buddy/__init__.py</pre><p>Your structure should now look like this in the editor:</p><pre>travel_project/<br>└── agents/<br>|   └── travel_buddy/<br>|   |   ├── __init__.py<br>|   |   ├── agent.py<br>|   └── .env<br>├── __init__.py<br>└── Dockerfile</pre><h3>Programmable Search Engine</h3><p>To use the Google search feature, we need to get the Search Engine ID from Google’s Programmable Search Engine. (<a href="https://programmablesearchengine.google.com/">https://programmablesearchengine.google.com/</a>)</p><p>First, Create your new Search Engine</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EtRmhn2zzevbD0Zljb-ZHg.png" /></figure><p>After the form appears, give it a <strong>name </strong>and select the “<strong>Search entire web</strong>” option. Leave the rest of the settings as they are, and finally, click “<strong>Create</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uwDLu5C2DAIZ3zc2ZMe3ww.png" /></figure><p>When you’re done, note the search engine ID.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*rD3s3fo1KzRHlJOHwXG8pA.png" /></figure><h3>Setting Up the Agent Code</h3><p>Now, let’s populate the files we’ve created.</p><h3>1. The .env File (Environment Configuration)</h3><ul><li><strong>Location:</strong> travel_project/agents/travel_buddy/.env</li><li><strong>Purpose:</strong> Configures the ADK to use Vertex AI for its generative model.</li><li><strong>Content:</strong></li></ul><pre>GOOGLE_CLOUD_PROJECT=&quot;YOUR_PROJECT_ID&quot;<br>GOOGLE_CLOUD_LOCATION=&quot;YOUR_CHOSEN_REGION&quot; # e.g., us-central1 or asia-southeast1<br>GOOGLE_GENAI_USE_VERTEXAI=&quot;True&quot;<br>GOOGLE_API_KEY=&quot;YOUR_GOOGLE_AI_STUDIO_API_KEY&quot;</pre><h3>2. The __init__.py Files (Package Markers)</h3><p><strong>File 1 (inside </strong><strong>agents):</strong></p><ul><li><strong>Location:</strong> travel_project/</li><li><strong>Content:</strong> Leave this file <strong>completely empty</strong>. Its presence is enough to mark the directory as a Python package.</li></ul><p><strong>File 2 (inside </strong><strong>travel_buddy):</strong></p><ul><li><strong>Location:</strong> travel_project/agents/travel_buddy/__init__.py</li><li><strong>Content: </strong>Leave this file <strong>completely empty</strong>. Its presence is enough to mark the directory as a Python package.</li></ul><h3>3. The agent.py File (Core Agent Logic)</h3><ul><li><strong>Location:</strong> travel_project/agents/travel_buddy/agent.py</li><li><strong>Content:</strong> Copy and paste the complete code below. This file contains all the logic for searching Google, scraping results, and defining the agent’s tools.</li></ul><pre>import os<br>import requests<br>from google.adk.agents import Agent<br>from bs4 import BeautifulSoup<br>import re</pre><pre>GOOGLE_API_KEY = &quot;YOUR_GOOGLE_AI_STUDIO_API_KEY&quot;<br>GOOGLE_CX = &quot;YOUR_PROGRAMMABLE_SEARCH_ENGINE_ID&quot;<br><br><br><br>def google_custom_search(query: str, num_results: int = 3) -&gt; list[str]:<br>    if not GOOGLE_API_KEY or not GOOGLE_CX:<br>        print(&quot;❌ Error: GOOGLE_API_KEY &amp; GOOGLE_CX is not set&quot;)<br>        return []<br><br>    try:<br>        url = &quot;https://www.googleapis.com/customsearch/v1&quot;<br>        params = {<br>            &#39;key&#39;: GOOGLE_API_KEY,<br>            &#39;cx&#39;: GOOGLE_CX,<br>            &#39;q&#39;: query,<br>            &#39;num&#39;: num_results<br>        }<br>        <br>        response = requests.get(url, params=params, timeout=10)<br>        response.raise_for_status()<br>        <br>        search_results = response.json()<br>        <br>        return [item[&#39;link&#39;] for item in search_results.get(&#39;items&#39;, [])]<br>        <br>    except Exception as e:<br>        print(f&quot;Error saat melakukan Google Custom Search: {e}&quot;)<br>        return []<br><br><br><br>def _extract_content_from_url(/service/url: str, content_type: str = &quot;general&quot;) -&gt; str:<br>    &quot;&quot;&quot;<br>    Mengekstrak teks dari URL dengan membersihkan elemen yang tidak relevan.<br>    Fungsi ini tidak diubah.<br>    &quot;&quot;&quot;<br>    try:<br>        headers = {&#39;User-Agent&#39;: &#39;Mozilla/5.0&#39;}<br>        response = requests.get(url, headers=headers, timeout=15)<br>        response.raise_for_status()<br>        soup = BeautifulSoup(response.content, &#39;html.parser&#39;)<br>        <br>        for element in soup([&quot;script&quot;, &quot;style&quot;, &quot;nav&quot;, &quot;header&quot;, &quot;footer&quot;, &quot;aside&quot;, &quot;ads&quot;]):<br>            element.decompose()<br><br>        extracted_text = &quot;&quot;<br><br>        if content_type == &quot;weather&quot;:<br>            weather_selectors = [<br>                &#39;[class*=&quot;temperature&quot;]&#39;, &#39;[class*=&quot;weather&quot;]&#39;, &#39;[class*=&quot;temp&quot;]&#39;,<br>                &#39;[id*=&quot;temperature&quot;]&#39;, &#39;[id*=&quot;weather&quot;]&#39;, &#39;[id*=&quot;current&quot;]&#39;,<br>                &#39;span[class*=&quot;deg&quot;]&#39;, &#39;div[class*=&quot;current&quot;]&#39;, &#39;div[data-testid=&quot;current-temperature&quot;]&#39;<br>            ]<br>            for selector in weather_selectors:<br>                elements = soup.select(selector)<br>                for elem in elements:<br>                    text = elem.get_text().strip()<br>                    if text and any(char.isdigit() for char in text):<br>                        extracted_text += text + &quot; &quot;<br>            if not extracted_text.strip():<br>                extracted_text = soup.get_text()<br>        elif content_type == &quot;time&quot;:<br>            time_selectors = [<br>                &#39;[class*=&quot;time&quot;]&#39;, &#39;[class*=&quot;clock&quot;]&#39;, &#39;[id*=&quot;time&quot;]&#39;,<br>                &#39;span[class*=&quot;current&quot;]&#39;, &#39;div[class*=&quot;local&quot;]&#39;, &#39;div[data-testid=&quot;current-time&quot;]&#39;<br>            ]<br>            for selector in time_selectors:<br>                elements = soup.select(selector)<br>                for elem in elements:<br>                    text = elem.get_text().strip()<br>                    if &quot;:&quot; in text:<br>                        extracted_text += text + &quot; &quot;<br>            if not extracted_text.strip():<br>                extracted_text = soup.get_text()<br>        else:<br>            extracted_text = soup.get_text()<br>        <br>        lines = (line.strip() for line in extracted_text.splitlines())<br>        chunks = (phrase.strip() for line in lines for phrase in line.split(&quot;  &quot;))<br>        text = &#39; &#39;.join(chunk for chunk in chunks if chunk)<br>        return text[:1500] + &quot;...&quot; if len(text) &gt; 1500 else text<br>    except Exception as e:<br>        return f&quot;Could not extract content from {url}: {str(e)}&quot;<br><br><br>def search_and_extract_content(query: str, num_results: int = 3, content_type: str = &quot;general&quot;) -&gt; list:<br>    search_results_urls = google_custom_search(query, num_results=num_results)<br>    <br>    if not search_results_urls:<br>        return []<br><br>    print(f&quot;Get content from {len(search_results_urls)} URL...&quot;)<br>    extracted_data = []<br>    for url in search_results_urls:<br>        content = _extract_content_from_url(https://medium.com/url,%20content_type)<br>        if content and not content.startswith(&quot;Could not extract&quot;):<br>            extracted_data.append({&quot;url&quot;: url, &quot;content&quot;: content})<br>    <br>    return extracted_data<br><br>def get_weather(city: str) -&gt; dict:<br>    try:<br>        queries = [f&quot;current weather {city} temperature celsius&quot;, f&quot;cuaca hari ini {city} suhu&quot;]<br>        for query in queries:<br>            results = search_and_extract_content(query, num_results=2, content_type=&quot;weather&quot;)<br>            for item in results:<br>                content, url = item[&#39;content&#39;], item[&#39;url&#39;]<br>                if len(content) &gt; 50:<br>                    weather_info = {}<br>                    <br>                    # --- Bagian Pengecekan Suhu (Tidak berubah) ---<br>                    temp_patterns = [r&#39;(-?\d{1,3}(?:\.\d+)?)\s*°C&#39;, r&#39;(-?\d{1,3}(?:\.\d+)?)\s*C\b&#39;]<br>                    for pattern in temp_patterns:<br>                        match = re.search(pattern, content, re.IGNORECASE)<br>                        if match:<br>                            weather_info[&#39;temperature&#39;] = match.group(1) + &quot;°C&quot;<br>                            break<br>                            <br>                    # --- Bagian Pengecekan Kondisi Cuaca (DIUBAH) ---<br>                    condition_patterns = [<br>                        r&#39;\b(cerah|berawan|hujan|mendung|panas|sejuk|dingin|kabut|angin|salju|badai)\b&#39;,<br>                        r&#39;\b(sunny|cloudy|rainy|clear|hot|cold|foggy|windy|snowy|storm|overcast)\b&#39;<br>                    ]<br>                    for pattern in condition_patterns:<br>                        # DIUBAH: dari re.findall menjadi re.search<br>                        match = re.search(pattern, content, re.IGNORECASE)<br>                        if match:<br>                            # DIUBAH: Ambil hanya hasil pertama yang cocok<br>                            weather_info[&#39;condition&#39;] = match.group(1).lower()<br>                            break # Keluar dari loop setelah menemukan kondisi pertama<br><br>                    if weather_info:<br>                        report = f&quot;🌤️ Weather in {city}:\n&quot;<br>                        if &#39;temperature&#39; in weather_info:<br>                            report += f&quot;Temperature: {weather_info[&#39;temperature&#39;]}\n&quot;<br>                        if &#39;condition&#39; in weather_info:<br>                            report += f&quot;Condition: {weather_info[&#39;condition&#39;].title()}\n&quot; # Dibuat jadi huruf kapital di awal<br>                        report += f&quot;Source: {url}&quot;<br>                        return {&quot;status&quot;: &quot;success&quot;, &quot;report&quot;: report}<br>                        <br>        return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: f&quot;No weather info found for {city}.&quot;}<br>    except Exception as e:<br>        return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: f&quot;Weather search failed: {str(e)}&quot;}<br><br>def get_current_time(city: str) -&gt; dict:<br>    try:<br>        query = f&quot;waktu sekarang di {city} current time&quot;<br>        results = search_and_extract_content(query, num_results=2, content_type=&quot;time&quot;)<br>        for item in results:<br>            content, url = item[&#39;content&#39;], item[&#39;url&#39;]<br>            match = re.search(r&#39;\b(\d{1,2}:\d{2}(?::\d{2})?)&#39;, content)<br>            if match:<br>                return {&quot;status&quot;: &quot;success&quot;, &quot;report&quot;: f&quot;⏰ Current time in {city} is {match.group(1)}.\nSource: {url}&quot;}<br>        return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: f&quot;No time info found for {city}.&quot;}<br>    except Exception as e:<br>        return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: f&quot;Time search failed: {str(e)}&quot;}<br><br>def get_travel_info(city: str, info_type: str) -&gt; dict:<br>    try:<br>        query = f&quot;{info_type} di {city} travel guide&quot;<br>        results = search_and_extract_content(query, num_results=1, content_type=&quot;general&quot;)<br>        if results:<br>            first_result = results[0]<br>            report = f&quot;Travel info about &#39;{info_type}&#39; in {city}:\n\nSource: {first_result[&#39;url&#39;]}\n{first_result[&#39;content&#39;]}&quot;<br>            return {&quot;status&quot;: &quot;success&quot;, &quot;report&quot;: report}<br>        else:<br>            return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: f&quot;No travel info found for &#39;{info_type}&#39; in {city}.&quot;}<br>    except Exception as e:<br>        return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: f&quot;Travel info search failed: {str(e)}&quot;}<br><br>root_agent = Agent(<br>    name=&quot;travel_buddy_agent&quot;,<br>    model=&quot;gemini-1.5-flash-002&quot;,<br>    description=&quot;An intelligent travel buddy agent that uses Google Search to provide real-time weather, time, and travel information.&quot;,<br>    instruction=(<br>        &quot;You are a friendly and helpful travel buddy agent. &quot;<br>        &quot;You provide users with live, up-to-date information about weather, time, and travel tips for any city in the world. &quot;<br>        &quot;Always cite your source.&quot;<br>    ),<br>    tools=[get_weather, get_current_time, get_travel_info]<br>)</pre><h3>Deploying the Agent to Cloud Run</h3><p>Once testing is successful, it’s time to make your agent publicly accessible by deploying it to Cloud Run. We’ll use the robust and explicit Dockerfile method.</p><h3>Step 1: Create the Dockerfile</h3><ol><li>In the root directory of your project (travel_project), create a new file named Dockerfile.</li><li>Populate the Dockerfile with the following content. This file contains the recipe for building your application&#39;s container.</li></ol><blockquote><strong><em>Why a Dockerfile?</em></strong><em> The default </em><em>adk deploy command sometimes fails to pass arguments correctly. By creating our own </em><em>Dockerfile, we gain full control and explicitly tell Cloud Run exactly how to start our application, ensuring the </em><em>agents argument is included. This is the standard, production-grade way to deploy applications.</em></blockquote><p>Dockerfile</p><pre>FROM python:3.11-slim<br><br>WORKDIR /app<br><br>COPY . .<br><br>RUN pip install google-adk geopy timezonefinder beautifulsoup4 googlesearch-python<br><br>EXPOSE 8080<br><br>CMD [&quot;adk&quot;, &quot;web&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;, &quot;--port&quot;, &quot;8080&quot;, &quot;agents&quot;]</pre><h3>Step 2: Run the gcloud Deploy Command</h3><p>Make sure you are still in the travel_project directory, then run the command below. This single command will build and deploy your agent.</p><pre>gcloud run deploy travel-buddy-service \<br>  --source . \<br>  --region=&quot;YOUR-CHOSEN-REGION&quot; \<br>  --allow-unauthenticated \<br>  --project=&quot;YOUR-PROJECT-ID&quot;</pre><p>The deployment process will take a few minutes. Once successful, the terminal will provide you with a public URL.</p><h3>Step 3: Test the Live Service</h3><p>Click the URL provided by the gcloud command. You will see the same ADK interface, but this time it&#39;s running live on Google&#39;s global infrastructure, ready to be shared or integrated into other applications.</p><h3>Step 3: Test with Example Prompts</h3><p>Try asking the agent some questions in the chat box on the right.</p><ul><li>“What’s the weather like in London?”</li><li>“Can you tell me the current time in Tokyo?”</li><li>“I’m planning a trip to New York. What’s the weather there right now?”</li><li>“What kind of power outlets do they use in Paris?”</li><li>“Tell me about the culture in Surabaya.</li></ul><h3>Conclusion</h3><p>By leveraging the Google Agent Development Kit, we have successfully created a dynamic and capable agent that provides real-time information. This tutorial demonstrates how the ADK can streamline the agent-building process, allowing developers to focus on the core logic of their tools. From this foundation, you can add more tools, connect to more APIs, and create even more sophisticated and helpful AI agents. Happy coding!</p><p><em>This article is written in collaboration with Kevin Jonathan</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd29d11d3e56" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/build-and-deploy-a-live-ai-travel-agent-in-minutes-with-google-adk-google-search-and-cloud-run-fd29d11d3e56">Build and Deploy a Live AI Travel Agent in Minutes with Google ADK, Google Search, and Cloud Run</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating an Agent for Visual Learners]]></title>
            <link>https://medium.com/google-developer-experts/creating-an-agent-for-visual-learners-5af090219143?source=rss-a32fdb99f2d------2</link>
            <guid isPermaLink="false">https://medium.com/p/5af090219143</guid>
            <dc:creator><![CDATA[Esther Irawati Setiawan]]></dc:creator>
            <pubDate>Wed, 02 Jul 2025 09:19:45 GMT</pubDate>
            <atom:updated>2025-11-25T05:05:03.387Z</atom:updated>
            <content:encoded><![CDATA[<p>In the ever-evolving landscape of AI, agents are becoming increasingly powerful tools, yet many interactions still heavily rely on text, which isn’t always effective for everyone. This tutorial, therefore, focuses on creating an agent for visual learners. Using Google’s Agent Development Kit (ADK), we will guide you through building a simple yet effective agent that moves beyond text-only replies. You will learn to create an agent that presents information through engaging visual formats by generating intuitive graphs and delivering interactive mind maps.</p><h3>Prerequisites</h3><p>Before we dive into the code, make sure you have the following libraries installed in your Python environment in terminal:</p><pre>Step 1: python -m venv &lt;environment_name&gt;<br>example: python -m venv mindmap_env<br>Step 2: &lt;environment_name&gt;\Scripts\activate<br> example: mindmap_env\Scripts\activate<br>Step 3: move agent.py and __init__.py to venv folder<br>Step 4: Copy the code of Agent.py and __init__.py<br>Step 5:  pip install google-adk requests<br>Step 6: Create .env file on the venv<br>Step 7: insert the API KEY from the AI Studio<br>Step 8: type adk web on the terminal</pre><p>These libraries are essential for the functionality of our agent:</p><ul><li>google-adk: The core framework for building agents.</li><li>requests: A standard library for making HTTP requests to external APIs.</li><li>re: Used for regular expression operations in parsing narratives.</li><li>base64: Used for URL encoding for Mermaid.ink.</li></ul><h3>The Agent’s Capabilities</h3><p>Our agent will have two primary functions:</p><ol><li><strong>Generate Mermaid Code</strong>: It will take a narrative as input and return the Mermaid mind map syntax in markdown format.</li><li><strong>Generate Mermaid Image URL</strong>: It will take a narrative as input and provide a direct image URL of the mind map generated via Mermaid.ink.Let’s break down the code that makes this possible.</li></ol><h3>The Code Explained</h3><p>Here is the complete Python code for our agent. We will go through it section by section. Create a file .py on the venv with a name of agent.py and __init__.py</p><p>__init__.py:</p><pre>from . import agent</pre><p>agent.py:</p><pre>from google.adk.agents import Agent<br>import re<br>import base64<br>import requests<br>from bs4 import BeautifulSoup<br>from googlesearch import search<br> <br>def scrape_text_from_url(/service/url: str) -&gt; str:<br>    &quot;&quot;&quot;Extracts and cleans paragraph text from a URL.&quot;&quot;&quot;<br>    try:<br>        headers = {&#39;User-Agent&#39;: &#39;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36&#39;}<br>        response = requests.get(url, headers=headers, timeout=10)<br>        response.raise_for_status()  <br>        soup = BeautifulSoup(response.text, &#39;html.parser&#39;)<br>       <br>    <br>        paragraphs = soup.find_all(&#39;p&#39;)<br>        text = &quot; &quot;.join([p.get_text() for p in paragraphs])<br>        return text<br>    except Exception as e:<br>        print(f&quot;[ERROR] Gagal scrape {url}: {e}&quot;)<br>        return &quot;&quot;<br> <br>def clean_search_result(text: str) -&gt; str:<br>    sentences = re.split(r&#39;(?&lt;=[.!?]) +&#39;, text.strip())<br>    short_summary = &quot; &quot;.join(sentences[:3])<br>    return re.sub(r&#39;\s+&#39;, &#39; &#39;, short_summary).strip()<br> <br>def generate_mindmap_code(narrative: str) -&gt; str:<br>    sentences = re.split(r&#39;(?&lt;=[.!?]) +&#39;, narrative.strip())<br>    if not sentences:<br>        raise ValueError(&quot;Narrative is empty.&quot;)<br>    root_label = re.sub(r&#39;[^\w\s-]&#39;, &#39;&#39;, sentences[0].strip().capitalize())<br>    lines = [&quot;mindmap&quot;, f&quot;  root(({root_label}))&quot;]<br>    counter = 1<br>    for sentence in sentences[1:]:<br>        cleaned = re.sub(r&#39;[^\w\s-]&#39;, &#39;&#39;, sentence.strip().capitalize())<br>        if cleaned:<br>            lines.append(f&quot;    root --&gt; child{counter}(({cleaned}))&quot;)<br>            counter += 1<br>    return &quot;\n&quot;.join(lines)<br> <br>def generate_mindmap(narrative: str) -&gt; dict:<br>    try:<br>        chart_code = generate_mindmap_code(narrative)<br>        return {&quot;status&quot;: &quot;success&quot;, &quot;report&quot;: f&quot;```mermaid\n{chart_code}\n```&quot;}<br>    except Exception as e:<br>        return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: f&quot;Mindmap generation error: {e}&quot;}<br> <br>def generate_mermaid_image_url(/service/narrative: str) -&gt; dict:<br>    try:<br>        mermaid_code = generate_mindmap_code(narrative)<br>        encoded = base64.urlsafe_b64encode(mermaid_code.encode(&quot;utf-8&quot;)).decode(&quot;utf-8&quot;)<br>        image_url = f&quot;https://mermaid.ink/img/{encoded}?type=svg&quot;<br>        return {&quot;status&quot;: &quot;success&quot;, &quot;report&quot;: image_url}<br>    except Exception as e:<br>        return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: f&quot;Image URL generation error: {e}&quot;}<br> <br>def search_and_generate_mindmap(query: str) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Search for a query on Google, take content from the top results, summarize it, then create a mindmap.<br>    &quot;&quot;&quot;<br>    try:<br>        print(f&quot;[INFO] Do a Google search for: &#39;{query}&#39;&quot;)<br>        search_results_urls = list(search(query, num_results=5, lang=&quot;id&quot;))<br> <br>        if not search_results_urls:<br>            return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: &quot;No search results found.&quot;}<br> <br>        print(f&quot;[INFO] Found {len(search_results_urls)} URL. Fetch content from the top 3 URLs.&quot;)<br>       <br>        combined_text = &quot;&quot;<br>        for url in search_results_urls[:3]:<br>            print(f&quot;[INFO] Extract text from: {url}&quot;)<br>            combined_text += scrape_text_from_https://medium.com/url(url) + &quot; &quot;<br> <br>        if not combined_text.strip():<br>            return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: &quot;Failed to extract content from search results.&quot;}<br> <br>        print(&quot;[INFO] Create a summary of the combined text.&quot;)<br>        summary = clean_search_result(combined_text)<br> <br>        if not summary or len(summary.split()) &lt; 5:<br>            return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: &quot;The summary of search results is too short.&quot;}<br>       <br>        print(&quot;[INFO] Create mindmap code from the summary.&quot;)<br>        return generate_mindmap(summary)<br> <br>    except Exception as e:<br>        print(f&quot;[FATAL ERROR] There is an error: {e}&quot;)<br>        return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: f&quot;Search and mindmap error: {e}&quot;}<br> <br>def search_and_generate_mindmap_image(query: str) -&gt; dict:<br>    try:<br>        print(f&quot;[INFO] Do a Google search for images: &#39;{query}&#39;&quot;)<br>        search_results_urls = list(search(query, num_results=5, lang=&quot;id&quot;))<br>        if not search_results_urls:<br>            return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: &quot;No search results found.&quot;}<br>       <br>        combined_text = &quot;&quot;<br>        for url in search_results_urls[:3]:<br>            combined_text += scrape_text_from_https://medium.com/url(url) + &quot; &quot;<br> <br>        if not combined_text.strip():<br>            return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: &quot;Failed to extract content.&quot;}<br>       <br>        summary = clean_search_result(combined_text)<br>        if not summary or len(summary.split()) &lt; 5:<br>            return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: &quot;Summary is too short.&quot;}<br><br>        print(&quot;[INFO] Create mindmap image URLs.&quot;)<br>        return generate_mermaid_image_url(https://medium.com/summary)<br> <br>    except Exception as e:<br>        return {&quot;status&quot;: &quot;error&quot;, &quot;error_message&quot;: f&quot;Search to image URL error: {e}&quot;}<br> <br>root_agent = Agent(<br>    name=&quot;mindmap_agent&quot;,<br>    model=&quot;gemini-2.0-flash&quot;,<br>    description=&quot;An intelligent agent that generates mindmaps from text or Google search using Mermaid syntax.&quot;,<br>    instruction=&quot;You are a helpful agent that turns narrative or search queries into Mermaid mindmaps and image links.&quot;,<br>    tools=[<br>        search_and_generate_mindmap,<br>        search_and_generate_mindmap_image<br>    ],<br>)<br> <br>if __name__ == &quot;__main__&quot;:<br>    test_query = &quot;the latest news about artificial intelligence in Indonesia&quot;<br>   <br>    print(&quot;--- CREATE A MERMAID CODE ---&quot;)<br>    result_code = search_and_generate_mindmap(test_query)<br>    print(&quot;\n[FINAL RESULT - MINDMAP CODE]&quot;)<br>    print(result_code.get(&quot;report&quot;, result_code.get(&quot;error_message&quot;)))<br> <br>    print(&quot;\n\n--- CREATE MERMAID IMAGE URL ---&quot;)<br>    result_img = search_and_generate_mindmap_image(test_query)<br>    print(&quot;\n[FINAL RESULT - IMAGE URL]&quot;)<br>    print(result_img.get(&quot;report&quot;, result_img.get(&quot;error_message&quot;)))</pre><h3>Bringing It All Together with ADK</h3><p>google.adk.agents.Agent. This is where the ADK magic happens.</p><ul><li><strong>name</strong>: We give our agent a descriptive name, mindmap_agent.</li><li><strong>model</strong>: We specify the AI model to power the agent’s understanding and responses. Here, we’re using “gemini-2.0-flash”.</li><li><strong>description</strong>: This provides a brief overview of what the agent does.</li><li><strong>instruction</strong>: This is a crucial part where we give the agent its persona and core instructions. We tell it that it’s a helpful agent that transforms narrative text into a Mermaid mind map chart and provides a direct image URL using Mermaid.ink.</li><li><strong>tools</strong>: This is where we equip our agent with its capabilities. We pass it the list of functions we defined earlier (generate_mindmap and generate_mermaid_image_url). The ADK will automatically know how to use these tools based on the user’s query.</li></ul><h3>How to Use the Agent</h3><p>Once this agent is deployed, you can interact with it by asking it to create a mind map on any topic. The agent will perform a web search on the subject and use the results to generate the mind map.</p><p>Here are some example prompts:</p><ul><li>“Create a mind map about the latest news on artificial intelligence in Indonesia.”</li><li>“Generate an image of a mind map detailing the main tourist attractions in Paris.”</li><li>“Tell me about the history of the internet and show it as a mind map.”</li><li>“I need the Mermaid code for a mind map about the main benefits of renewable energy.”</li></ul><p>The agent will understand the intent behind your questions and call the appropriate function (<strong>search_and_generate_mindmap</strong> for code or <strong>search_and_generate_mindmap_image</strong> for an image URL) with your topic as the query. It will then use the output from the function to provide you with a helpful and accurate response.</p><h3>Steps to run your agent (Deployment)</h3><p>To run your agent locally using the Google ADK, follow these steps in order:</p><h4>1. Prepare Your Folder Structure</h4><p>Ensure your project folder is organized like this:</p><pre>your_project_folder/  (Example: D:\Archive)<br>├── .env<br>├── __init__.py<br>└── agent.py</pre><ul><li><strong>your_project_folder</strong>: This is your main project directory.</li><li><strong>.env</strong>: This file will store your API Key.</li><li><strong>__init__.py</strong>: This file is either empty or contains from . import agent to mark the folder as a Python package.</li><li><strong>agent.py</strong>: This file contains all your agent&#39;s code (the generate_mindmap and generate_mermaid_image_url functions, and the root_agent definition).</li></ul><h4>2. Create the .env File</h4><p>Inside your your_project_folder (e.g., D:\Archive), create a new file named .env. Open it with a text editor (like Notepad, VS Code, or Sublime Text) and enter the following lines:</p><pre>GOOGLE_GENAI_USE_VERTEXAI=FALSE<br>GOOGLE_API_KEY=PASTE_YOUR_ACTUAL_API_KEY_HERE</pre><p>Replace PASTE_YOUR_ACTUAL_API_KEY_HERE with your actual API Key from Google AI Studio.</p><h4>4. Open a Regular PowerShell (Non-Administrator)</h4><p>Open a new, regular PowerShell window (do not run it as an administrator).</p><h4>5. Navigate to Your Project Directory</h4><p>Use the cd command to go to your project folder (where __init__.py and agent.py are located).</p><pre>cd path\to\your_project_folder</pre><h4>6. Activate Your Virtual Environment</h4><p>Run the following command to activate the virtual environment you already created. Make sure the virtual environment name is correct (e.g., mindmap_env).</p><p>on windows:</p><pre>.\mindmap_env\Scripts\activate</pre><p>on macOS/Linux:</p><pre>source mindmap_env/bin/activate</pre><p>You will see (mindmap_env) appear at the beginning of your command prompt, indicating that the virtual environment is active.</p><h4>7. Run Your Agent Using the ADK Web Interface</h4><p>Now, with your virtual environment active and being in your project directory, run the following command to launch the ADK web interface:</p><pre>adk web</pre><p><strong><em>Note:</em></strong><em> You do not need to add </em><em>agent.root_agent here. The ADK will automatically discover the </em><em>root_agent in the </em><em>agent.py file within the current directory.</em></p><h4>8. Access the ADK Web Interface in Your Browser</h4><p>After running adk web, you will see some messages in the terminal, including the URL where your agent is running (usually http://localhost:8000). Open your web browser and navigate to that URL.</p><p>You can now interact with your “Mindmap Agent” through the chat interface in your browser!</p><h3>Conclusion</h3><p>By leveraging the power of Google’s Agent Development Kit and a few useful Python libraries, we have created an agent capable of transforming narratives into mind maps and providing image URLs. This example shows how ADK can simplify the AI agent development process, allowing developers to focus on the core logic of their tools while ADK handles the natural language understanding and response generation. You can expand on this foundation to add more tools and create even more sophisticated and helpful agents. Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5af090219143" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developer-experts/creating-an-agent-for-visual-learners-5af090219143">Creating an Agent for Visual Learners</a> was originally published in <a href="https://medium.com/google-developer-experts">Google Developer Experts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your Android Application with Multimodal Live API]]></title>
            <link>https://medium.com/google-developer-experts/build-your-android-application-with-multimodal-live-api-6addc0be828b?source=rss-a32fdb99f2d------2</link>
            <guid isPermaLink="false">https://medium.com/p/6addc0be828b</guid>
            <dc:creator><![CDATA[Esther Irawati Setiawan]]></dc:creator>
            <pubDate>Tue, 01 Jul 2025 07:00:54 GMT</pubDate>
            <atom:updated>2025-11-25T05:05:01.918Z</atom:updated>
            <content:encoded><![CDATA[<p>Gemini Live on Android changes the way you interact with artificial intelligence, going beyond traditional text input. With the ability to capture images through the camera and record audio directly, the app enables more dynamic dialogue. Simply speak or point to something, and Gemini will process the audio to provide relevant voice responses, creating a two-way conversation that feels more natural and intuitive.</p><h3>Prerequisites</h3><p>Before we dive into the code, make sure you have the following libraries installed in your device environment :</p><ol><li>Android Studio</li><li>Gemini AI Studio</li></ol><p>The Agent’s Capabilities:</p><ol><li>Display a live camera preview on the screen.</li><li>Send images from the camera periodically to the Gemini API.</li><li>Continuously record audio from the microphone and send it to the API.</li><li>Connect to the Gemini API using WebSocket for bi-directional communication.</li><li>Receive and display text responses from Gemini.</li><li>Receive and play audio responses from Gemini.</li></ol><h3>How to Use the Agent</h3><p>Once all the code is set up, you can run the app on a physical Android device or emulator.</p><ol><li>Make sure you have replaced “API KEY AI STUDIO” with your valid API Key.</li><li>Run the app.</li><li>Press “Start Capture” to start the camera preview. The app will start sending images.</li><li>Press “Start Audio” to start recording and sending your voice.</li><li>Speak into the microphone. Gemini’s response will appear as text in the chatLog and the audio will play automatically.</li><li>Use “Stop” and “Stop Capture” to stop sending media.</li></ol><h3>Step by step guide</h3><h3>1. Project Setup</h3><p><strong>A. Create a New Android Project</strong></p><p>Start by creating a new project in Android Studio with the “Empty Views Activity” template and select the “Kotlin” language.</p><p><strong>B. Add Permissions in AndroidManifest.xml</strong></p><p>This app requires access to the camera, microphone, and internet. Open the app/src/main/AndroidManifest.xml file and add the following lines inside the &lt;manifest&gt; tag:</p><pre>&lt;uses-permission android:name=&quot;android.permission.INTERNET&quot; /&gt;<br>&lt;uses-permission android:name=&quot;android.permission.CAMERA&quot; /&gt;<br>&lt;uses-permission android:name=&quot;android.permission.RECORD_AUDIO&quot; /&gt;</pre><h3><strong>2. Add Dependencies to build.gradle.kts (Module: app)</strong></h3><p>To handle WebSocket connections and JSON serialization, add the following dependencies to your build.gradle.kts file.</p><pre> // File: build.gradle.kts (Module :app)<br>dependencies {<br>// ... Android standard dependencies<br> implementation(&quot;androidx.core:core-ktx:1.9.0&quot;)<br> implementation(&quot;androidx.appcompat:appcompat:1.6.1&quot;)<br> implementation(&quot;com.google.android.material:material:1.10.0&quot;)<br> implementation(&quot;androidx.constraintlayout:constraintlayout:2.1.4&quot;)<br> // For WebSocket connections<br> implementation(&quot;org.java-websocket:Java-WebSocket:1.5.3&quot;)<br> implementation(&quot;com.squareup.okhttp3:okhttp:4.12.0&quot;)<br> // For easy JSON parsing and creation<br> implementation(&quot;org.json:json:20231013&quot;) // JSON parsing<br> implementation (&quot;org.jetbrains.kotlinx:kotlinx-serialization-json:1.6.3&quot;)<br>}</pre><p><em>Don’t forget to press “Sync Now” after adding dependencies..</em></p><h3>3. User Interface (UI) Design</h3><p>The UI design for this application is quite simple and functional. Open the app/src/main/res/layout/activity_main.xml file and create a layout that contains the following components:</p><ol><li>TextureView: Used to display the camera preview. It is more flexible for video streaming manipulation than SurfaceView.</li><li>Button (Start/Stop Capture): To enable and disable the camera preview.</li><li>Button (Start/Stop Audio): Two separate buttons to start and stop audio recording.</li><li>TextView: Serves as a “Chat Log” to display text responses received from Gemini.</li><li>ImageView: As a visual status indicator to show the connection status (connected, disconnected) and audio recording status (recording, mute).</li></ol><p>activity_main.xml Layout Code:</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt;<br>&lt;androidx.constraintlayout.widget.ConstraintLayout xmlns:android=&quot;http://schemas.android.com/apk/res/android&quot;<br>    xmlns:app=&quot;http://schemas.android.com/apk/res-auto&quot;<br>    xmlns:tools=&quot;http://schemas.android.com/tools&quot;<br>    android:layout_width=&quot;match_parent&quot;<br>    android:layout_height=&quot;match_parent&quot;<br>    tools:context=&quot;.MainActivity&quot;&gt;<br>    &lt;LinearLayout<br>        android:layout_width=&quot;match_parent&quot;<br>        android:layout_height=&quot;match_parent&quot;<br>        android:orientation=&quot;vertical&quot;<br>        android:padding=&quot;16dp&quot;&gt;<br>        &lt;TextureView<br>            android:id=&quot;@+id/textureView&quot;<br>            android:layout_width=&quot;match_parent&quot;<br>            android:layout_height=&quot;0dp&quot;<br>            android:layout_weight=&quot;1&quot;<br>            android:layout_marginBottom=&quot;16dp&quot;<br>            android:scaleType=&quot;centerCrop&quot;<br>            /&gt;<br>        &lt;ImageView<br>            android:id=&quot;@+id/statusIndicator&quot;<br>            android:layout_width=&quot;match_parent&quot;<br>            android:layout_height=&quot;30dp&quot;<br>            android:layout_marginBottom=&quot;16dp&quot;<br>            android:src=&quot;@drawable/baseline_equalizer_24&quot;<br>            /&gt;<br>        &lt;LinearLayout<br>            android:layout_width=&quot;match_parent&quot;<br>            android:layout_height=&quot;wrap_content&quot;<br>            android:orientation=&quot;horizontal&quot;<br>            android:gravity=&quot;center&quot;&gt;<br>            &lt;com.google.android.material.button.MaterialButton<br>                android:id=&quot;@+id/captureButton&quot;<br>                android:layout_width=&quot;wrap_content&quot;<br>                android:layout_height=&quot;wrap_content&quot;<br>                android:layout_marginEnd=&quot;8dp&quot;<br>                app:icon=&quot;@drawable/baseline_camera_24&quot;<br>                app:iconGravity=&quot;textStart&quot;<br>                app:iconTint=&quot;@android:color/white&quot;<br>                android:insetLeft=&quot;12dp&quot;<br>                android:insetTop=&quot;12dp&quot;<br>                android:insetRight=&quot;12dp&quot;<br>                android:insetBottom=&quot;12dp&quot;<br>                style=&quot;@style/Widget.Material3.Button.IconButton.Filled&quot;<br>                /&gt;<br>            &lt;com.google.android.material.button.MaterialButton<br>                android:id=&quot;@+id/startButton&quot;<br>                android:layout_width=&quot;wrap_content&quot;<br>                android:layout_height=&quot;wrap_content&quot;<br>                app:icon=&quot;@drawable/baseline_mic_24&quot;<br>                app:iconGravity=&quot;textStart&quot;<br>                app:iconTint=&quot;@android:color/white&quot;<br>                android:insetLeft=&quot;12dp&quot;<br>                android:insetTop=&quot;12dp&quot;<br>                android:insetRight=&quot;12dp&quot;<br>                android:insetBottom=&quot;12dp&quot;<br>                style=&quot;@style/Widget.Material3.Button.IconButton.Filled&quot;<br>                /&gt;<br><br>            &lt;com.google.android.material.button.MaterialButton<br>                android:id=&quot;@+id/stopButton&quot;<br>                android:layout_width=&quot;wrap_content&quot;<br>                android:layout_height=&quot;wrap_content&quot;<br>                android:layout_marginStart=&quot;8dp&quot;<br>                app:icon=&quot;@drawable/baseline_mic_off_24&quot;<br>                app:iconGravity=&quot;textStart&quot;<br>                app:iconTint=&quot;@android:color/white&quot;<br>                android:insetLeft=&quot;12dp&quot;<br>                android:insetTop=&quot;12dp&quot;<br>                android:insetRight=&quot;12dp&quot;<br>                android:insetBottom=&quot;12dp&quot;<br>                style=&quot;@style/Widget.Material3.Button.IconButton.Filled&quot;<br>                /&gt;<br>        &lt;/LinearLayout&gt;<br>        &lt;TextView<br>            android:id=&quot;@+id/chatLog&quot;<br>            android:layout_width=&quot;match_parent&quot;<br>            android:layout_height=&quot;wrap_content&quot;<br>            android:layout_marginTop=&quot;16dp&quot;<br>            android:maxHeight=&quot;300dp&quot;<br>            android:scrollbars=&quot;vertical&quot;<br>            android:background=&quot;#f0f0f0&quot;<br>            android:padding=&quot;8dp&quot;<br>            /&gt;<br>    &lt;/LinearLayout&gt;<br>&lt;/androidx.constraintlayout.widget.ConstraintLayout&gt;</pre><h3>4. Implement Logic in MainActivity.kt</h3><p>This file is the core of our application. Replace the entire contents of your MainActivity.kt with the code provided in the end. Here is a breakdown of the main functions:</p><p><strong>A. WebSocket Initialization and Connection</strong></p><ul><li>Connection to Gemini API is done via WebSocket for real-time communication.</li><li>Important Variables: <br><strong>API_KEY</strong>: Enter your API Key obtained from Google AI Studio.<br><strong>URL</strong>: The URL of the WebSocket to connect to.</li><li><strong>connect()</strong> Function:<br>This function is called inside onCreate() to immediately initiate a connection when the application is opened. A WebSocketClient object is created, and its methods are overridden to handle various connection events.<br><strong>onOpen</strong>: When the connection is successful, the application sends an initial configuration message (sendInitialSetupMessage) to tell the API what model to use and the desired response format (Audio and Text).<br><strong>onMessage</strong>: Handles incoming messages from the server.<br><strong>onClose &amp; onError</strong>: Manage connection status if disconnected or error occurs.</li></ul><pre>// Insert your API key here<br>private val API_KEY = &quot;$APIKEY&quot; // Replace with your actual API key<br>private val URL = &quot;wss://$HOST/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent?key=$API_KEY&quot;<br><br>private fun connect() {<br>    Log.d(&quot;WebSocket&quot;, &quot;Connecting to: $URL&quot;)<br>    val headers = mutableMapOf&lt;String, String&gt;()<br>    headers[&quot;Content-Type&quot;] = &quot;application/json&quot;<br><br>    webSocket = object : WebSocketClient(URI(URL), Draft_6455(), headers) {<br>        override fun onOpen(handshakedata: ServerHandshake?) {<br>            Log.d(&quot;WebSocket&quot;, &quot;Connected. Server handshake: ${handshakedata?.httpStatus}&quot;)<br>            isConnected = true<br>            updateStatusIndicator()<br>            sendInitialSetupMessage()<br>        }<br><br>        override fun onMessage(message: String?) {<br>            Log.d(&quot;WebSocket&quot;, &quot;Message Received: $message&quot;)<br>            receiveMessage(message)<br>        }<br><br>        override fun onMessage(bytes: ByteBuffer?) {<br>            bytes?.let {<br>                val message = String(it.array(), Charsets.UTF_8)<br>                receiveMessage(message)<br>            }<br>        }<br><br>        override fun onClose(code: Int, reason: String?, remote: Boolean) {<br>            Log.d(&quot;WebSocket&quot;, &quot;Connection Closed: $reason&quot;)<br>            isConnected = false<br>            updateStatusIndicator()<br>            runOnUiThread {<br>                Toast.makeText(this@MainActivity, &quot;Connection closed&quot;, Toast.LENGTH_SHORT).show()<br>            }<br>        }<br><br>        override fun onError(ex: Exception?) {<br>            Log.e(&quot;WebSocket&quot;, &quot;Error: ${ex?.message}&quot;)<br>            isConnected = false<br>            updateStatusIndicator()<br>        }<br>    }<br>    webSocket?.connect()<br>}<br></pre><p><strong>B. Camera Management (Video Input)</strong></p><p>The application uses the Camera2 API for full control over the camera hardware.</p><ol><li><strong>startCameraPreview()</strong>: Starts the flow to open the camera and display the preview in a TextureView.</li><li><strong>ImageReader</strong>: The key component that receives image frames from the camera session.</li><li><strong>imageAvailableListener</strong>: This listener is triggered every time a new frame is available.</li></ol><ul><li>For efficiency, images are not sent every frame. Instead, there is an interval (IMAGE_SEND_INTERVAL) that ensures images are sent every few seconds.</li><li>Before being sent, the images are resized and compressed to reduce bandwidth.</li><li>The images are then converted to a Base64 string and sent via the sendMediaChunk function.</li></ul><pre>// Listener that is triggered when a new camera frame is available<br>private val imageAvailableListener = ImageReader.OnImageAvailableListener { reader -&gt;<br>    val currentTime = System.currentTimeMillis()<br>    // Hanya proses gambar jika interval waktu telah tercapai<br>    if (currentTime - lastImageSendTime &gt;= IMAGE_SEND_INTERVAL) {<br>        val image = reader.acquireLatestImage() ?: return@OnImageAvailableListener // ... (kode untuk mengambil byte gambar)<br>        val buffer = image.planes[0].buffer<br>        val bytes = ByteArray(buffer.remaining())<br>        buffer.get(bytes)<br>        image.close()<br><br>        // Proses dan kirim gambar di background thread<br>        GlobalScope.launch(Dispatchers.IO) {<br>            processAndSendImage(bytes)<br>        }<br>        lastImageSendTime = currentTime<br>        Log.d(&quot;ImageCapture&quot;, &quot;Image processed and sent based on time interval&quot;)<br><br>    } else {<br>        // Penting: Buang gambar agar antrian tidak penuh<br>        val image = reader.acquireLatestImage()?.close() // Important: release the image<br>        Log.d(&quot;ImageCapture&quot;, &quot;Image capture skipped: Not enough time elapsed&quot;)<br>    }<br>}</pre><p><strong>Time-Based Throttling</strong></p><ul><li>The main function of the if (currentTime — lastImageSendTime &gt;= IMAGE_SEND_INTERVAL) block is to prevent processing of any available images.</li><li>The code will only process an image if the time elapsed since the last processing has reached or exceeded the specified interval (IMAGE_SEND_INTERVAL).</li></ul><p><strong>If Time Interval Reached (Image Processed)</strong></p><ul><li>reader.acquireLatestImage(): Gets the latest available image.</li><li>The image is converted to a ByteArray (raw data).</li><li>GlobalScope.launch(Dispatchers.IO): Image processing and sending (processAndSendImage) is run in a background thread using Coroutines. This is very important to prevent the application from slowing down or freezing.</li><li>lastImageSendTime = currentTime: The last processing time is updated for the calculation of the next interval.</li></ul><p><strong>If Time Interval Has Not Been Reached (Image Skipped)</strong></p><ul><li>reader.acquireLatestImage()?.close(): This is a very important part. Even though the image will not be processed, it must still be taken from the queue (acquire) and immediately released (close).</li><li>Purpose: To clean the queue in ImageReader. If these unprocessed images are not discarded, the queue will fill up quickly and ImageReader will stop providing new images.</li></ul><p><strong>C. Audio Management (Input &amp; Output)</strong></p><ol><li>Audio Recording (startAudioInput)</li></ol><ul><li>Use AudioRecord to capture raw audio (PCM) from the microphone. Configure the sample rate (24000) and format according to Gemini API requirements.</li><li>A coroutine runs in the background to continuously read audio data into the buffer.</li><li>This audio data is then converted to Base64 and sent in chunks via WebSocket.</li></ul><p>2. Audio Playback (playAudio and injestAudioChunkToPlay)</p><ul><li>When an incoming message contains audio data (inlineData), the Base64 data is decoded.</li><li>The raw audio data is added to the audioQueue.</li><li>A playback function (playNextAudioChunk) takes a chunk from the queue and plays it using AudioTrack. The use of a queue ensures that the audio plays smoothly in streaming.</li></ul><pre>// Function to start audio recording<br>private fun startAudioInput() {<br>    if (isRecording) return<br>    isRecording = true<br>    audioRecord = AudioRecord(<br>        MediaRecorder.AudioSource.VOICE_COMMUNICATION, // Use VOICE_COMMUNICATION<br>        AUDIO_SAMPLE_RATE,<br>        AUDIO_CHANNEL_CONFIG,<br>        AUDIO_ENCODING,<br>        AUDIO_BUFFER_SIZE<br>    )<br><br>    if (audioRecord?.state != AudioRecord.STATE_INITIALIZED) {<br>        Log.e(&quot;Audio&quot;, &quot;AudioRecord initialization failed&quot;)<br>        return<br>    }<br><br>    audioRecord?.startRecording()<br>    Log.d(&quot;Audio&quot;, &quot;Start Recording&quot;)<br>    isSpeaking = true;<br>    updateStatusIndicator() // Update the status indicator<br>    recordInterval =  GlobalScope.launch(Dispatchers.IO) {<br>        while (isRecording) {<br>            val buffer = ShortArray(AUDIO_BUFFER_SIZE)<br>            val readSize = audioRecord?.read(buffer, 0, buffer.size)<br>            if (readSize != null &amp;&amp; readSize &gt; 0) {<br>                pcmData.addAll(buffer.take(readSize).toList())<br>                if(pcmData.size &gt;= readSize) {<br>                    recordChunk()<br>                }<br>            }<br>        }<br>    }<br>}</pre><p>Initial Checks</p><ul><li>if (isRecording) return: Prevents the function from running again if recording is already active.</li><li>An AudioRecord object is created and configured. Using MediaRecorder.AudioSource. VOICE_COMMUNICATION indicates that the audio source is optimized for voice communication, which usually includes features like noise suppression.</li></ul><p>Starting Recording</p><ul><li>After ensuring that AudioRecord has been successfully initialized, audioRecord?.startRecording() activates the microphone to begin capturing sound.</li><li>The state within the application is updated to indicate that recording is in progress (isSpeaking = true).</li></ul><p>Background Audio Processing</p><ul><li>GlobalScope.launch(Dispatchers.IO): This is the most important part. A coroutine or lightweight process is launched in the background thread. The goal is to keep the continuous audio reading process from interfering with or crashing the user interface (UI).</li><li>while (isRecording): In this background thread, a loop runs continuously as long as isRecording is true.</li><li>On each loop, the code reads a small chunk of audio data from the microphone into a buffer, then adds it to a list called pcmData.</li><li>recordChunk(): Once the data is collected, this function is called to process the “chunk” of audio data (for example, sending it to a server or analyzing it).</li></ul><p><strong>D. Sending and Receiving Data</strong></p><p>sendMediaChunk(b64Data, mimeType)</p><ul><li>This function is responsible for wrapping Base64 encoded media data (audio or video) into the correct JSON format.</li><li>The JSON structure is created using kotlinx.serialization(buildJsonObject), which is more modern and secure.</li><li>The JSON message is then sent via WebSocket.</li></ul><p>receiveMessage(message)</p><ul><li>Parses the JSON string received from the server.</li><li>If the JSON contains the key &quot;text&quot;, its contents will be displayed in the chatLog.</li><li>If it contains the key &quot;inlineData&quot; with the mimeType audio, its data will be forwarded to the audio playback system.</li></ul><pre>// Function to send media data to WebSocket<br>private fun sendMediaChunk(b64Data: String, mimeType: String) {<br>    if (!isConnected) {<br>        Log.d(&quot;WebSocket&quot;, &quot;WebSocket not connected&quot;)<br>        return<br>    }<br><br>    // Use JsonBuilder for proper JSON formatting<br>    val msg = buildJsonObject {<br>        put(&quot;realtime_input&quot;, buildJsonObject {<br>            put(&quot;media_chunks&quot;, buildJsonArray {<br>                add(buildJsonObject {<br>                    put(&quot;mime_type&quot;, mimeType)<br>                    put(&quot;data&quot;, b64Data)<br>                })<br>            })<br>        })<br>    }<br><br>    val jsonString = Json { prettyPrint = false }.encodeToString(msg)<br><br>    //Log.d(&quot;WebSocket&quot;, &quot;Sending media chunk (MIME: $mimeType): $jsonString&quot;)<br>    webSocket?.send(jsonString)<br>}</pre><p><strong>Connection Check</strong></p><p>if (!isConnected): The first step is to check if the WebSocket connection is active. If it is not connected, the function will stop and do nothing to avoid errors.</p><p><strong>JSON Message Creation</strong></p><ul><li>val msg = buildJsonObject { … }: This code creates a message in JSON format using the builder from the kotlinx.serialization library.</li><li>The structure is very specific and hierarchical, namely wrapping media data (b64Data) and its data type (mimeType) in a realtime_input object and a media_chunks array. This format is most likely a requirement of the target server API.</li></ul><p><strong>Sending Data</strong></p><ul><li>val jsonString = …: The JSON object that has been created is then converted into a compact string format (prettyPrint = false).</li><li>webSocket?.send(jsonString): The JSON string is finally sent to the server via a WebSocket connection.</li></ul><p><strong>Overall, this function is a “courier” that takes media data, wraps it into a JSON packet according to the server’s rules, and sends it out.</strong></p><p>Below is the complete code of MainActivity.kt</p><pre>package com.example.geminilivedemo<br><br>import android.Manifest<br>import android.content.pm.PackageManager<br>import android.graphics.Bitmap<br>import android.graphics.BitmapFactory<br>import android.graphics.ImageFormat<br>import android.graphics.SurfaceTexture<br>import android.hardware.camera2.*<br>import android.media.ImageReader<br>import android.os.Bundle<br>import android.os.Handler<br>import android.os.HandlerThread<br>import android.util.Base64<br>import android.util.Log<br>import android.util.Size<br>import android.view.Surface<br>import android.view.TextureView<br>import android.widget.Button<br>import android.widget.ImageView<br>import android.widget.TextView<br>import android.widget.Toast<br>import androidx.appcompat.app.AppCompatActivity<br>import androidx.core.app.ActivityCompat<br>import androidx.core.content.ContextCompat<br>import okhttp3.*<br>import org.java_websocket.client.WebSocketClient<br>import org.java_websocket.handshake.ServerHandshake<br>import org.json.JSONObject<br>import java.io.ByteArrayOutputStream<br>import java.net.URI<br>import java.nio.ByteBuffer<br>import java.nio.ByteOrder<br>import java.util.*<br>import kotlin.math.roundToInt<br>import android.media.AudioFormat<br>import android.media.AudioRecord<br>import android.media.MediaRecorder<br>import kotlinx.coroutines.*<br><br>import android.media.AudioTrack<br>import android.os.Environment<br>import androidx.core.content.FileProvider<br>import java.io.File<br>import java.io.IOException<br><br>import java.text.SimpleDateFormat<br>import android.graphics.drawable.AnimationDrawable<br>import android.os.Looper<br><br>import kotlinx.serialization.json.*<br>import kotlinx.serialization.encodeToString<br>import kotlinx.serialization.json.JsonObject<br>import kotlinx.serialization.json.buildJsonObject<br><br>import org.java_websocket.drafts.Draft_6455<br>import org.json.JSONArray<br><br>class MainActivity : AppCompatActivity() {<br><br>    private lateinit var textureView: TextureView // Use TextureView for camera preview<br>    private lateinit var captureButton: Button<br>    private lateinit var startButton: Button<br>    private lateinit var stopButton: Button<br>    private lateinit var chatLog: TextView<br>    private lateinit var statusIndicator: ImageView<br>    private var currentFrameB64: String? = null<br>    private var webSocket: WebSocketClient? = null<br>    private var isRecording = false<br>    private var audioRecord: AudioRecord? = null<br>    private var pcmData = mutableListOf&lt;Short&gt;()<br>    private var job: Job? = null<br>    private var recordInterval: Job? = null<br>    private val MODEL = &quot;models/gemini-2.0-flash-exp&quot;<br>    private val API_KEY = &quot;AIzaSyB8sWubdwR_oQc4AS0t7ayZd6QxyS-Ij-c&quot; // Replace with your actual API key<br>    private val HOST = &quot;generativelanguage.googleapis.com&quot;<br>    private val URL = &quot;wss://$HOST/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent?key=$API_KEY&quot;<br><br>    private val CAMERA_REQUEST_CODE = 100<br>    private val AUDIO_REQUEST_CODE = 200<br>    private val AUDIO_SAMPLE_RATE = 24000<br>    private val RECEIVE_SAMPLE_RATE = 24000<br>    private val AUDIO_CHANNEL_CONFIG = AudioFormat.CHANNEL_IN_MONO<br>    private val AUDIO_ENCODING = AudioFormat.ENCODING_PCM_16BIT<br>    private val AUDIO_BUFFER_SIZE = AudioRecord.getMinBufferSize(AUDIO_SAMPLE_RATE, AUDIO_CHANNEL_CONFIG, AUDIO_ENCODING)<br><br>    private val audioQueue = mutableListOf&lt;ByteArray&gt;()<br>    private var isPlaying = false;<br>    private var audioTrack: AudioTrack? = null;<br><br>    private val MAX_IMAGE_DIMENSION = 1024  // Max width/height for transmitted images<br>    private val JPEG_QUALITY = 70           // Quality percentage for JPEG compression<br>    private var lastImageSendTime: Long = 0<br>    private val IMAGE_SEND_INTERVAL: Long = 3000 // 3 seconds for image capture interval<br>    private var isConnected = false;<br>    private var isSpeaking = false;<br><br>    // Camera2 API related variables<br>    private var cameraDevice: CameraDevice? = null<br>    private var cameraCaptureSession: CameraCaptureSession? = null<br>    private var captureRequestBuilder: CaptureRequest.Builder? = null<br>    private var imageReader: ImageReader? = null<br>    private val cameraThread = HandlerThread(&quot;CameraThread&quot;).apply { start() }<br>    private val cameraHandler = Handler(cameraThread.looper)<br>    private lateinit var cameraId: String<br>    private lateinit var previewSize: Size<br><br>    private val timeFormat = SimpleDateFormat(&quot;HH:mm:ss.SSS&quot;, Locale.getDefault())<br>    private var isCameraActive = false<br><br>    override fun onCreate(savedInstanceState: Bundle?) {<br>        super.onCreate(savedInstanceState)<br>        setContentView(R.layout.activity_main)<br><br>        textureView = findViewById(R.id.textureView)<br>        captureButton = findViewById(R.id.captureButton)<br>        startButton = findViewById(R.id.startButton)<br>        stopButton = findViewById(R.id.stopButton)<br>        chatLog = findViewById(R.id.chatLog)<br>        statusIndicator = findViewById(R.id.statusIndicator)<br>        updateStatusIndicator()<br><br>        captureButton.setOnClickListener {<br>            if (isCameraActive) {<br>                stopCameraPreview()<br>                captureButton.text = &quot;Start Capture&quot;<br>                isCameraActive = false<br>            } else {<br>                if (textureView.isAvailable) {<br>                    startCameraPreview()<br>                    captureButton.text = &quot;Stop Capture&quot;<br>                    isCameraActive = true<br>                } else {<br>                    textureView.surfaceTextureListener = surfaceTextureListener<br>                }<br>            }<br>        }<br>        startButton.setOnClickListener {<br>            checkRecordAudioPermission()<br>        }<br><br>        stopButton.setOnClickListener {<br>            stopAudioInput()<br>        }<br><br>        connect()<br>    }<br><br>    private val surfaceTextureListener = object : TextureView.SurfaceTextureListener {<br>        override fun onSurfaceTextureAvailable(surface: SurfaceTexture, width: Int, height: Int) {<br>            startCameraPreview()<br>            captureButton.text = &quot;Stop Capture&quot;<br>            isCameraActive = true<br><br>        }<br><br>        override fun onSurfaceTextureSizeChanged(surface: SurfaceTexture, width: Int, height: Int) {<br><br>        }<br><br>        override fun onSurfaceTextureDestroyed(surface: SurfaceTexture): Boolean {<br>            stopCameraPreview()<br>            captureButton.text = &quot;Start Capture&quot;<br>            isCameraActive = false<br>            return true<br>        }<br><br>        override fun onSurfaceTextureUpdated(surface: SurfaceTexture) {<br><br>        }<br>    }<br><br>    private fun startCameraPreview() {<br>        checkCameraPermissionForPreview()<br>    }<br><br>    private fun stopCameraPreview() {<br>        closeCamera()<br>        textureView.surfaceTexture?.let { surfaceTexture -&gt;<br>            val surface = Surface(surfaceTexture)<br>            val canvas = surface.lockCanvas(null)<br>            canvas?.drawColor(android.graphics.Color.BLACK)<br>            surface.unlockCanvasAndPost(canvas)<br>            surface.release()<br>        }<br>    }<br><br>    private fun checkCameraPermissionForPreview() {<br>        if (ContextCompat.checkSelfPermission(<br>                this,<br>                Manifest.permission.CAMERA<br>            ) != PackageManager.PERMISSION_GRANTED<br>        ) {<br>            ActivityCompat.requestPermissions(<br>                this,<br>                arrayOf(Manifest.permission.CAMERA),<br>                CAMERA_REQUEST_CODE<br>            )<br>        } else {<br>            openCameraForPreview()<br>        }<br>    }<br><br>    private fun openCameraForPreview() {<br>        val cameraManager = getSystemService(CAMERA_SERVICE) as CameraManager<br>        try {<br>            cameraId = cameraManager.cameraIdList[0]<br>            val characteristics = cameraManager.getCameraCharacteristics(cameraId)<br>            val map = characteristics.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP) ?: return<br>            previewSize = map.getOutputSizes(SurfaceTexture::class.java)[0]<br><br>            imageReader = ImageReader.newInstance(<br>                MAX_IMAGE_DIMENSION, MAX_IMAGE_DIMENSION,<br>                ImageFormat.JPEG, 2<br>            ).apply {<br>                setOnImageAvailableListener(imageAvailableListener, cameraHandler)<br>            }<br><br>            cameraManager.openCamera(cameraId, cameraStateCallback, cameraHandler)<br><br>        } catch (e: CameraAccessException) {<br>            Log.e(&quot;Camera&quot;, &quot;Error opening camera&quot;, e)<br>        } catch (e: SecurityException) {<br>            ActivityCompat.requestPermissions(this, arrayOf(Manifest.permission.CAMERA), CAMERA_REQUEST_CODE)<br>        }<br>    }<br><br>    private val cameraStateCallback = object : CameraDevice.StateCallback() {<br>        override fun onOpened(camera: CameraDevice) {<br>            cameraDevice = camera<br>            createCameraPreviewSession()<br>        }<br><br>        override fun onDisconnected(camera: CameraDevice) {<br>            cameraDevice?.close()<br>            cameraDevice = null<br>        }<br><br>        override fun onError(camera: CameraDevice, error: Int) {<br>            cameraDevice?.close()<br>            cameraDevice = null<br>            Log.e(&quot;Camera&quot;, &quot;Camera error: $error&quot;)<br>        }<br>    }<br><br>    private fun createCameraPreviewSession() {<br>        try {<br>            val surfaceTexture = textureView.surfaceTexture?.apply {<br>                setDefaultBufferSize(previewSize.width, previewSize.height)<br>            }<br>            val previewSurface = Surface(surfaceTexture)<br><br>            captureRequestBuilder = cameraDevice?.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW)?.apply {<br>                addTarget(previewSurface)<br>                addTarget(imageReader!!.surface)<br>            }<br><br>            cameraDevice?.createCaptureSession(<br>                listOf(previewSurface, imageReader!!.surface),<br>                cameraCaptureSessionCallback, cameraHandler<br>            )<br><br>        } catch (e: CameraAccessException) {<br>            Log.e(&quot;Camera&quot;, &quot;Error creating preview session&quot;, e)<br>        }<br>    }<br><br>    private val cameraCaptureSessionCallback = object : CameraCaptureSession.StateCallback() {<br>        override fun onConfigured(session: CameraCaptureSession) {<br>            cameraCaptureSession = session<br>            updatePreview()<br>        }<br><br>        override fun onConfigureFailed(session: CameraCaptureSession) {<br>            Toast.makeText(this@MainActivity, &quot;Configuration failed&quot;, Toast.LENGTH_SHORT).show()<br>        }<br>    }<br><br>    private fun updatePreview() {<br>        if (cameraDevice == null) return<br><br>        captureRequestBuilder?.set(CaptureRequest.CONTROL_MODE, CaptureRequest.CONTROL_MODE_AUTO)<br>        val thread = HandlerThread(&quot;UpdatePreview&quot;).apply { start() }<br>        val handler = Handler(thread.looper)<br>        try {<br>            cameraCaptureSession?.setRepeatingRequest(<br>                captureRequestBuilder?.build()!!,<br>                null, handler<br>            )<br>        } catch (e: CameraAccessException) {<br>            Log.e(&quot;Camera&quot;, &quot;Error starting preview repeat request&quot;, e)<br>        }<br>    }<br><br>    private fun closeCamera() {<br>        cameraCaptureSession?.close()<br>        cameraCaptureSession = null<br>        cameraDevice?.close()<br>        cameraDevice = null<br>        imageReader?.close()<br>        imageReader = null<br>    }<br><br>    private val imageAvailableListener = ImageReader.OnImageAvailableListener { reader -&gt;<br>        val currentTime = System.currentTimeMillis()<br>        // Hanya proses gambar jika interval waktu telah tercapai<br>        if (currentTime - lastImageSendTime &gt;= IMAGE_SEND_INTERVAL) {<br>            val image = reader.acquireLatestImage() ?: return@OnImageAvailableListener // ... (kode untuk mengambil byte gambar)<br>            val buffer = image.planes[0].buffer<br>            val bytes = ByteArray(buffer.remaining())<br>            buffer.get(bytes)<br>            image.close()<br><br>            // Proses dan kirim gambar di background thread<br>            GlobalScope.launch(Dispatchers.IO) {<br>                processAndSendImage(bytes)<br>            }<br>            lastImageSendTime = currentTime<br>            Log.d(&quot;ImageCapture&quot;, &quot;Image processed and sent based on time interval&quot;)<br><br>        } else {<br>            // Penting: Buang gambar agar antrian tidak penuh<br>            val image = reader.acquireLatestImage()?.close() // Important: release the image<br>            Log.d(&quot;ImageCapture&quot;, &quot;Image capture skipped: Not enough time elapsed&quot;)<br>        }<br>    }<br><br>    private suspend fun processAndSendImage(imageBytes: ByteArray) {<br>        val currentTime = timeFormat.format(Date())<br>        Log.d(&quot;ImageCapture&quot;, &quot;Image processed and sending at: $currentTime&quot;)<br><br>        val bitmap = BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)<br><br>        // Step 1: Resize if necessary<br>        val scaledBitmap = scaleBitmap(bitmap, MAX_IMAGE_DIMENSION)<br><br>        // Step 2: Compress with reduced quality<br>        val byteArrayOutputStream = ByteArrayOutputStream()<br>        scaledBitmap.compress(Bitmap.CompressFormat.JPEG, JPEG_QUALITY, byteArrayOutputStream)<br><br>        // Step 3: Create Base64 string<br>        val b64Image = Base64.encodeToString(byteArrayOutputStream.toByteArray(), Base64.DEFAULT or Base64.NO_WRAP)<br><br>        // Step 4: Send to WebSocket<br>        sendMediaChunk(b64Image, &quot;image/jpeg&quot;)<br><br>        // Clean up<br>        scaledBitmap.recycle()<br>        byteArrayOutputStream.close()<br>    }<br><br>    private fun scaleBitmap(bitmap: Bitmap, maxDimension: Int): Bitmap {<br>        val width = bitmap.width<br>        val height = bitmap.height<br><br>        if (width &lt;= maxDimension &amp;&amp; height &lt;= maxDimension) {<br>            return bitmap<br>        }<br><br>        val newWidth: Int<br>        val newHeight: Int<br><br>        if (width &gt; height) {<br>            val ratio = width.toFloat() / maxDimension<br>            newWidth = maxDimension<br>            newHeight = (height / ratio).toInt()<br>        } else {<br>            val ratio = height.toFloat() / maxDimension<br>            newHeight = maxDimension<br>            newWidth = (width / ratio).toInt()<br>        }<br><br>        return Bitmap.createScaledBitmap(bitmap, newWidth, newHeight, true)<br>    }<br><br>    override fun onRequestPermissionsResult(<br>        requestCode: Int,<br>        permissions: Array&lt;out String&gt;,<br>        grantResults: IntArray<br>    ) {<br>        super.onRequestPermissionsResult(requestCode, permissions, grantResults)<br>        when (requestCode) {<br>            CAMERA_REQUEST_CODE -&gt; {<br>                if (grantResults.isNotEmpty() &amp;&amp; grantResults[0] == PackageManager.PERMISSION_GRANTED) {<br>                    startCameraPreview() // Start preview if permission granted<br>                } else {<br>                    Toast.makeText(this, &quot;Camera permission denied&quot;, Toast.LENGTH_SHORT).show()<br>                }<br>            }<br>            AUDIO_REQUEST_CODE -&gt; {<br>                if (grantResults.isNotEmpty() &amp;&amp; grantResults[0] == PackageManager.PERMISSION_GRANTED) {<br>                    startAudioInput()<br>                } else {<br>                    Toast.makeText(this, &quot;Audio permission denied&quot;, Toast.LENGTH_SHORT).show()<br>                }<br>            }<br>        }<br>    }<br><br>    private fun checkRecordAudioPermission() {<br>        if (ContextCompat.checkSelfPermission(<br>                this,<br>                Manifest.permission.RECORD_AUDIO<br>            ) != PackageManager.PERMISSION_GRANTED<br>        ) {<br>            ActivityCompat.requestPermissions(<br>                this,<br>                arrayOf(Manifest.permission.RECORD_AUDIO),<br>                AUDIO_REQUEST_CODE<br>            )<br>        } else {<br>            startAudioInput()<br>        }<br>    }<br><br>    private fun connect() {<br>        Log.d(&quot;WebSocket&quot;, &quot;Connecting to: $URL&quot;)<br>        val headers = mutableMapOf&lt;String, String&gt;()<br>        headers[&quot;Content-Type&quot;] = &quot;application/json&quot;<br><br>        webSocket = object : WebSocketClient(URI(URL), Draft_6455(), headers) {<br>            override fun onOpen(handshakedata: ServerHandshake?) {<br>                Log.d(&quot;WebSocket&quot;, &quot;Connected. Server handshake: ${handshakedata?.httpStatus}&quot;)<br>                isConnected = true<br>                updateStatusIndicator()<br>                sendInitialSetupMessage()<br>            }<br><br>            override fun onMessage(message: String?) {<br>                Log.d(&quot;WebSocket&quot;, &quot;Message Received: $message&quot;)<br>                receiveMessage(message)<br>            }<br><br>            override fun onMessage(bytes: ByteBuffer?) {<br>                bytes?.let {<br>                    val message = String(it.array(), Charsets.UTF_8)<br>                    receiveMessage(message)<br>                }<br>            }<br><br>            override fun onClose(code: Int, reason: String?, remote: Boolean) {<br>                Log.d(&quot;WebSocket&quot;, &quot;Connection Closed: $reason&quot;)<br>                isConnected = false<br>                updateStatusIndicator()<br>                runOnUiThread {<br>                    Toast.makeText(this@MainActivity, &quot;Connection closed&quot;, Toast.LENGTH_SHORT).show()<br>                }<br>            }<br><br>            override fun onError(ex: Exception?) {<br>                Log.e(&quot;WebSocket&quot;, &quot;Error: ${ex?.message}&quot;)<br>                isConnected = false<br>                updateStatusIndicator()<br>            }<br>        }<br>        webSocket?.connect()<br>    }<br><br>    private fun sendInitialSetupMessage() {<br>        Log.d(&quot;WebSocket&quot;, &quot;Sending initial setup message&quot;)<br>        val setupMessage = JSONObject()<br>        val setup = JSONObject()<br>        val generationConfig = JSONObject()<br>        val responseModalities = org.json.JSONArray()<br>        responseModalities.put(&quot;AUDIO&quot;)<br>        generationConfig.put(&quot;response_modalities&quot;, responseModalities)<br>        // Use minimal setup like Python code<br>        setup.put(&quot;model&quot;, MODEL)<br>        setup.put(&quot;generation_config&quot;, generationConfig)<br>        setupMessage.put(&quot;setup&quot;, setup)<br>        Log.d(&quot;WebSocket&quot;, &quot;Sending config payload: $setupMessage&quot;)<br>        webSocket?.send(setupMessage.toString())<br>    }<br><br>    private fun sendMediaChunk(b64Data: String, mimeType: String) {<br>        if (!isConnected) {<br>            Log.d(&quot;WebSocket&quot;, &quot;WebSocket not connected&quot;)<br>            return<br>        }<br><br>        // Use JsonBuilder for proper JSON formatting<br>        val msg = buildJsonObject {<br>            put(&quot;realtime_input&quot;, buildJsonObject {<br>                put(&quot;media_chunks&quot;, buildJsonArray {<br>                    add(buildJsonObject {<br>                        put(&quot;mime_type&quot;, mimeType)<br>                        put(&quot;data&quot;, b64Data)<br>                    })<br>                })<br>            })<br>        }<br><br>        val jsonString = Json { prettyPrint = false }.encodeToString(msg)<br><br>        //Log.d(&quot;WebSocket&quot;, &quot;Sending media chunk (MIME: $mimeType): $jsonString&quot;)<br>        webSocket?.send(jsonString)<br>    }<br><br>    private fun receiveMessage(message: String?) {<br>        if (message == null) return<br><br>        try {<br>            val messageData = JSONObject(message)<br><br>            if (messageData.has(&quot;serverContent&quot;)) {<br>                val serverContent = messageData.getJSONObject(&quot;serverContent&quot;)<br>                if (serverContent.has(&quot;modelTurn&quot;)) {<br>                    val modelTurn = serverContent.getJSONObject(&quot;modelTurn&quot;)<br>                    if (modelTurn.has(&quot;parts&quot;)) {<br>                        val parts = modelTurn.getJSONArray(&quot;parts&quot;)<br>                        for (i in 0 until parts.length()) {<br>                            val part = parts.getJSONObject(i)<br>                            if (part.has(&quot;text&quot;)) {<br>                                val text = part.getString(&quot;text&quot;)<br>                                displayMessage(&quot;GEMINI: $text&quot;)<br>                            }<br>                            if(part.has(&quot;inlineData&quot;)){<br>                                val inlineData = part.getJSONObject(&quot;inlineData&quot;);<br>                                if(inlineData.has(&quot;mimeType&quot;) &amp;&amp; inlineData.getString(&quot;mimeType&quot;) == &quot;audio/pcm;rate=24000&quot;){<br>                                    val audioData = inlineData.getString(&quot;data&quot;)<br>                                    injestAudioChunkToPlay(audioData)<br>                                }<br><br>                            }<br>                        }<br>                    }<br>                }<br>            }<br>        } catch (e: Exception) {<br>            Log.e(&quot;Receive&quot;, &quot;Error parsing message&quot;, e)<br>        }<br>    }<br><br>    private fun base64ToArrayBuffer(base64: String): ByteArray {<br>        return Base64.decode(base64, Base64.DEFAULT)<br>    }<br><br>    private fun convertPCM16LEToFloat32(pcmData: ByteArray): FloatArray {<br>        val shortArray = pcmData.asShortArray()<br>        val floatArray = FloatArray(shortArray.size)<br><br>        for (i in shortArray.indices) {<br>            floatArray[i] = shortArray[i] / 32768f<br>        }<br>        return floatArray<br>    }<br><br>    private fun ByteArray.asShortArray(): ShortArray {<br>        val shortArray = ShortArray(this.size / 2)<br>        val byteBuffer = ByteBuffer.wrap(this).order(ByteOrder.LITTLE_ENDIAN)<br>        for (i in shortArray.indices) {<br>            shortArray[i] = byteBuffer.short<br>        }<br>        return shortArray<br>    }<br><br>    private fun injestAudioChunkToPlay(base64AudioChunk: String?) {<br>        if (base64AudioChunk == null) return<br><br>        GlobalScope.launch(Dispatchers.IO) {<br>            try {<br>                val arrayBuffer = base64ToArrayBuffer(base64AudioChunk)<br>                synchronized(audioQueue) {<br>                    audioQueue.add(arrayBuffer)<br>                }<br>                if (!isPlaying) {<br>                    playNextAudioChunk()<br>                }<br>                Log.d(&quot;AudioChunk&quot;, &quot;Audio chunk added to the queue&quot;)<br>            } catch (e: Exception) {<br>                Log.e(&quot;AudioChunk&quot;, &quot;Error processing chunk&quot;, e)<br>            }<br>        }<br>    }<br><br>    private fun playNextAudioChunk() {<br>        GlobalScope.launch(Dispatchers.IO) {<br>            while (true) {<br>                val chunk = synchronized(audioQueue) {<br>                    if (audioQueue.isNotEmpty()) audioQueue.removeAt(0) else null<br>                } ?: break<br><br>                isPlaying = true<br>                playAudio(chunk)<br>            }<br>            isPlaying = false<br><br>            // Check for new chunks that might have arrived while we were exiting<br>            synchronized(audioQueue) {<br>                if (audioQueue.isNotEmpty()) {<br>                    playNextAudioChunk()<br>                }<br>            }<br>        }<br>    }<br><br>    private fun playAudio(byteArray: ByteArray) {<br>        if(audioTrack == null) {<br>            audioTrack = android.media.AudioTrack(<br>                android.media.AudioManager.STREAM_MUSIC,<br>                RECEIVE_SAMPLE_RATE, // &lt;---<br>                android.media.AudioFormat.CHANNEL_OUT_MONO,<br>                android.media.AudioFormat.ENCODING_PCM_16BIT,<br>                AudioTrack.getMinBufferSize(<br>                    RECEIVE_SAMPLE_RATE,<br>                    AudioFormat.CHANNEL_OUT_MONO,<br>                    AudioFormat.ENCODING_PCM_16BIT<br>                ),<br>                AudioTrack.MODE_STREAM<br>            )<br>        }<br><br>        audioTrack?.write(byteArray, 0, byteArray.size)<br>        audioTrack?.play()<br>        GlobalScope.launch(Dispatchers.IO) {<br>            while (audioTrack?.playState == android.media.AudioTrack.PLAYSTATE_PLAYING){<br>                delay(10);<br>            }<br>            audioTrack?.stop()<br>            // audioTrack?.release()<br>        }<br>    }<br><br>    private fun startAudioInput() {<br>        if (isRecording) return<br>        isRecording = true<br>        audioRecord = AudioRecord(<br>            MediaRecorder.AudioSource.VOICE_COMMUNICATION, // Use VOICE_COMMUNICATION<br>            AUDIO_SAMPLE_RATE,<br>            AUDIO_CHANNEL_CONFIG,<br>            AUDIO_ENCODING,<br>            AUDIO_BUFFER_SIZE<br>        )<br><br>        if (audioRecord?.state != AudioRecord.STATE_INITIALIZED) {<br>            Log.e(&quot;Audio&quot;, &quot;AudioRecord initialization failed&quot;)<br>            return<br>        }<br><br>        audioRecord?.startRecording()<br>        Log.d(&quot;Audio&quot;, &quot;Start Recording&quot;)<br>        isSpeaking = true;<br>        updateStatusIndicator() // Update the status indicator<br>        recordInterval =  GlobalScope.launch(Dispatchers.IO) {<br>            while (isRecording) {<br>                val buffer = ShortArray(AUDIO_BUFFER_SIZE)<br>                val readSize = audioRecord?.read(buffer, 0, buffer.size)<br>                if (readSize != null &amp;&amp; readSize &gt; 0) {<br>                    pcmData.addAll(buffer.take(readSize).toList())<br>                    if(pcmData.size &gt;= readSize) {<br>                        recordChunk()<br>                    }<br>                }<br>            }<br>        }<br>    }<br><br>    private fun recordChunk() {<br>        if(pcmData.isEmpty()) return;<br>        GlobalScope.launch(Dispatchers.IO) {<br>            val buffer = ByteBuffer.allocate(pcmData.size * 2).order(ByteOrder.LITTLE_ENDIAN)<br>            pcmData.forEach { value -&gt;<br>                buffer.putShort(value)<br>            }<br>            val byteArray = buffer.array()<br>            val base64 = Base64.encodeToString(byteArray, Base64.DEFAULT or Base64.NO_WRAP)<br>            Log.d(&quot;Audio&quot;,&quot;Send Audio Chunk&quot;)<br>            sendMediaChunk(base64, &quot;audio/pcm&quot;)<br><br>            pcmData.clear()<br>        }<br>    }<br>    private fun stopAudioInput() {<br>        isRecording = false;<br>        recordInterval?.cancel()<br>        audioRecord?.stop()<br>        audioRecord?.release()<br>        audioRecord = null;<br>        Log.d(&quot;Audio&quot;, &quot;Stop Recording&quot;)<br>        isSpeaking = false;<br>        updateStatusIndicator()<br>    }<br><br>    private fun displayMessage(message: String) {<br>        Log.d(&quot;Chat&quot;, &quot;Displaying message: $message&quot;)<br>        runOnUiThread {<br>            val currentText = chatLog.text.toString()<br>            val message =  &quot;$currentText \n$message&quot;<br>            chatLog.text = message<br>        }<br>    }<br><br>    private fun updateStatusIndicator() {<br>        runOnUiThread {<br>            if (!isConnected) {<br>                statusIndicator.setImageResource(R.drawable.baseline_error_24)<br>                statusIndicator.setColorFilter(android.graphics.Color.RED)<br>            } else if (!isSpeaking) {<br>                statusIndicator.setImageResource(R.drawable.baseline_equalizer_24)<br>                statusIndicator.setColorFilter(android.graphics.Color.GRAY)<br>            } else {<br>                statusIndicator.setImageResource(R.drawable.baseline_equalizer_24)<br>                statusIndicator.setColorFilter(android.graphics.Color.GREEN)<br>            }<br>        }<br>    }<br><br>    override fun onDestroy() {<br>        super.onDestroy()<br>        cameraThread.quitSafely()<br>        closeCamera()<br>    }<br>}</pre><h3>Conclusion</h3><p>By leveraging the real-time capabilities of the Gemini Live API and native Android components in Kotlin, we have built a fully functional multimodal agent capable of understanding live video and audio streams. This article provides a clear blueprint, showing developers how to integrate Google’s advanced multimodal AI into a native Android app, simplifying the otherwise complex tasks of managing real-time data streams and API communication. You can expand upon this foundation to add more complex features, refine the user interface, or create entirely new kinds of interactive AI agents that see and hear the world through the user’s device. Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6addc0be828b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developer-experts/build-your-android-application-with-multimodal-live-api-6addc0be828b">Build Your Android Application with Multimodal Live API</a> was originally published in <a href="https://medium.com/google-developer-experts">Google Developer Experts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>