<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rubens Zimbres on Medium]]></title>
        <description><![CDATA[Stories by Rubens Zimbres on Medium]]></description>
        <link>https://medium.com/@rubenszimbres?source=rss-30f09d1741ce------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4g5XVksp8-oEHxR6Yc5mZQ.png</url>
            <title>Stories by Rubens Zimbres on Medium</title>
            <link>https://medium.com/@rubenszimbres?source=rss-30f09d1741ce------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 00:15:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rubenszimbres/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Pallas Kernel + vLLM on TPU: Leverage Gemma Throughput to 19,511 Tokens per Second]]></title>
            <link>https://medium.com/@rubenszimbres/pallas-kernel-vllm-on-tpu-leverage-gemma-throughput-to-19-511-tokens-per-second-bebf7933abb1?source=rss-30f09d1741ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/bebf7933abb1</guid>
            <category><![CDATA[jax]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[tpu]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Rubens Zimbres]]></dc:creator>
            <pubDate>Mon, 06 Jul 2026 15:35:08 GMT</pubDate>
            <atom:updated>2026-07-06T15:35:08.935Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Throughput, latency, energy, and cost results for Gemma 2B served with vLLM, with observations on the JAX and Pallas compilation stack</em></p><p>This is the Part 2 of my other article, “From 1,540 to 15,338 Tokens per Second on a Single TPU Chip”, where I approach the “how” I was able to first boost tokens-per-second from 1,540 to 15,338, and then to 19,5111.</p><p>The formal paper including both Part 1 and Part 1 articles is available here:</p><p><a href="https://zenodo.org/records/21221952">From 1,540 to 19,511 Tokens per Second on a Single TPU v5e Chip: A Measurement Study of Large Language Model Inference Optimization</a></p><p>This part is the study’s methods volume. Part I reported the results: a single TPU v5e chip serving the unmodified Gemma 2B model through vLLM moved from 1,540 to 19,511 tokens per second, with the energy ceiling per token falling from roughly 128 to 10 millijoules and the generation cost from roughly 4.6 to 58.5 million tokens per dollar.</p><p>This part discloses how, strategy by strategy: the exact configurations, the code that was written and patched, the profiling procedures that motivated each change, the verification that each change had taken effect, and the outcome of each, including the failures. It is written for machine learning engineers and JAX developers; hardware background helps but is not assumed.</p><p>Seven strategies were implemented. Three succeeded: correcting the token selection path (2.3), scaling the batch (4.3), and tuning the size of the kernel block (a further 1.3). Four are reported as instructive failures or validations: speculative decoding, a Pallas kernel transplant, three quantization routes, and a numeric precision patch. Every strategy followed the same loop: profile, predict a specific gain, change one variable, verify the change took effect, and measure against the prediction.</p><h3>Environment, Stack, and Measurement Harness</h3><p>All work ran in a hosted Colab notebook attached to a single TPU v5e chip (v5e-1, one chip, two cores, 16 GB of high-bandwidth memory at 819 GB per second, 197 teraFLOPs per second peak in bfloat16). The software stack was vllm-tpu 0.23.0 with its tpu-inference plugin, JAX 0.10.1, libtpu 0.0.41, and torch 2.10 at the final sessions; earlier sessions resolved older versions, and the engine version line printed at startup was checked whenever numbers from different sessions were compared.</p><p>One stack detail shapes everything that follows: Gemma 2B is not among the plugin’s JAX-native model definitions, so the engine loads the vLLM-native PyTorch definition and executes it through torchax, the bridge that lowers PyTorch operations to JAX and then XLA. Custom kernels on this stack are written in Pallas and lowered through the Mosaic compiler; the stock attention kernel is a Pallas kernel called ragged paged attention, version 3.</p><p>Measurement used one benchmark script for every run, holding the protocol fixed while environment variables selected the treatment: SPEC_DECODE, BATCH_MULT, PROFILE, MODEL, QUANT, and KV_DTYPE. The script builds the engine, times engine initialization separately, runs a small warmup generation to trigger first-touch compilation paths, measures batch time to first token by generating exactly one token for the full prompt set, and then runs the main generation of 128 tokens per request under temperature 0.0. The workload is four short prompts repeated BATCH_MULT times. Inter-token latency is derived by subtracting the measured time to first token from the main run and dividing by the tokens per request. Two rules were absolute: headline throughput comes only from runs with the profiler off, because tracing adds overhead, and every claimed effect requires a verification independent of the flag that requested it.</p><p>Profiling used the engine’s start_profile and stop_profile calls wrapped around only the main generation, so traces contain the steady state and not initialization. Traces were read two ways. TensorBoard’s XProf plugin provided the HLO operation statistics table, ranking every compiled operation by device time with its full HLO text, tensor shapes, occurrence counts, and source-file attribution; traces were staged into a fresh log directory per analysis to avoid the stale-trace trap, in which TensorBoard silently displays a previous session’s data, and every table was authenticated by checking that its tensor shapes matched the batch under study before any number was read. When the table’s filtering failed, the compressed trace JSON written next to the .xplane file was parsed directly in Python, aggregating operation durations by name and, critically, by the tensor shape extracted from each event’s long name. One environment quirk is worth recording for reproducibility: on this stack the profiler ignored the requested output directory and wrote traces under /content/None/plugins/profile, so traces were located by searching for .xplane files rather than trusting the configured path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*dvCELmFGTEvu6pWj1AHgiQ.png" /></figure><p>The engine’s own efficiency features were left at their defaults and verified rather than tuned, because the profiles showed they were already doing their job. Asynchronous scheduling overlaps host-side work with device execution; the measured gap between decode steps was 85 microseconds against an 11.9-millisecond step at batch 128, under one percent idle, so no host-side optimization was warranted. Chunked prefill ran with its default budget of 8,192 tokens per pass, visible in the traces as the 4,096-token shape population. Prefix caching was enabled, and with a repeated prompt set it makes the prompt-processing phase inexpensive; this is declared as part of the workload definition since the numbers are decode-dominated. Paged attention managed the key-value cache in 128-token blocks, 3,249 blocks and 415,872 cacheable tokens under the 0.75 memory fraction, capacity that was never the binding constraint (batch 512 used roughly 72,000 tokens). The optimization surface that remained, and that the seven strategies address, was therefore the workload configuration and the compiled operations themselves, not the serving loop.</p><h3>Strategy 1: The Token Selection Path</h3><p>The first profile of the default configuration contradicted the expected picture. The top two operations by total device time were fusions attributed to a binary-search source file inside the sampler, together holding roughly 55 percent of the device, while the transformer’s matmuls and the attention custom call divided the remainder. The cause was the request configuration: the baseline used sampling with top- and repetition penalties, and on this stack a top- request routes every step through an implementation that performs an iterative binary search over the 256,000-entry vocabulary distribution. The search executes its full fixed iteration count, 32 iterations, even when top- is 1.0 and the operation is mathematically a no-op; the penalty transforms add further per-step tensor work over the vocabulary. The model itself was the minority of the step.</p><p>The change was one line in the request, not in the engine: SamplingParams(max_tokens=128, temperature=0.0), which selects greedy decoding and bypasses the sampling transforms entirely. Verification was the disappearance of the binary-search fusions from the post-change HLO table, with the weight-streaming matmuls promoted to the top ranks: after the fix, the fused gate-and-up projection family led the table at roughly 36 percent of device time, the output projection near 17 percent, and attention near 18 percent, the anatomy of a memory-bound decode. Throughput rose from roughly 1,540 to 3,577 tokens per second at 32 concurrent requests, a factor of 2.3, and the internal engine counter agreed with the wall-clock measurement within a few percent, the paired verification used for every headline number afterward. The observation that a permissive parameter value still pays the full computational cost was documented for the stack’s maintainers. The general procedure transfers: profile with the exact SamplingParams of production, because the selection path is part of the workload, and a team whose product genuinely requires sampling should measure that configuration rather than assume the greedy numbers.</p><h3>Strategy 2: Speculative Decoding</h3><p>Speculative decoding was configured through the engine’s native support: method ngram, num_speculative_tokens 5, prompt_lookup_min 2, prompt_lookup_max 4, meaning a lookup mechanism drafts up to five tokens from n-gram matches in the existing text and the model verifies them in one pass. At 32 concurrent requests this produced 607 tokens per second against the 3,555 greedy baseline of the same session, a loss of a factor of six.</p><p>The mechanism is arithmetic, not implementation failure. Verification of drafted tokens makes the forward pass process tokens per request per step; at batch 32 that is up to 192 token positions per step through a memory-bound model, and every rejected draft is discarded work. The engine’s internal counters corroborated the wall-clock number, ruling out a measurement artifact, and the configuration itself was confirmed active in the startup log, so this is a genuine regime result, not an inert flag. Speculation converts spare compute into lower per-request latency, which is profitable only when the accelerator has spare capacity, meaning very small batches; the published three-to-four-times claims for the technique come from that regime. It was not pursued further because the study’s objective was throughput at meaningful batch sizes. The one run cost a few minutes and prevented a wrong turn, and its per-token energy figure, 325 millijoules against the 55 of the greedy baseline it replaced, is the study’s clearest illustration that a mismatched optimization is also an energy regression.</p><h3>Strategy 3: Batch Scaling and Compile Buckets</h3><p>The post-fix profile showed the step dominated by operations whose cost is set by streaming fixed bytes: the fused gate-and-up projection reading a bfloat16 weight (134 MB per layer, measured at about 91 percent of theoretical bandwidth while executing) and the output projection over the 256,000-entry vocabulary reading roughly 1 GB (about 84 percent). Fixed cost per step means near-free amortization, so the batch was doubled repeatedly: max_num_seqs in the engine configuration raised the concurrency cap, and BATCH_MULT scaled the prompt set to fill it. Each operating point required both, and each new maximum request count triggered fresh ahead-of-time compilation.</p><p>That compilation model matters operationally. The engine precompiles a fixed menu of input sizes: token paddings of 16 through 8192 and request paddings of 8 through the configured maximum, plus separate sampling, logprob, and placeholder substitution graphs. Cold initialization ranged from 106 to 285 seconds depending on the JAX compile cache state; the cache persists within a virtual machine but not across rebuilt Colab sessions. Runtime steps then execute in the smallest precompiled bucket that fits, with host overhead measured at 85 microseconds per 11.9-millisecond step, under one percent.</p><p>The batch ladder. Energy uses the 197 W rated chip power as an upper bound; cost uses $1.20 per chip-hour on demand. Scaling factors per doubling: 1.81, 1.50, 1.38, 1.13.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*mJE2c5Ne_zCT3q1gnNrvMA.png" /></figure><p>Each rung was verified at the operation level by tracing and reading average times per HLO operation. Three families told the whole story, and Table [tab:opladder] gives their ladder. The gate-and-up projection stayed flat while memory-bound, then rose as its arithmetic intensity crossed the chip’s critical 240 FLOPs per byte between batch 256 and 512, where the 197-teraFLOP roofline predicts 349 microseconds against the 375 measured. The output projection behaved identically at larger scale.</p><p>The attention custom call scaled linearly, a factor of 7.97 across an 8-times batch increase, because each request reads its own key-value cache and the operation’s intensity is fixed at about 8 FLOPs per byte, the ratio of query heads to key-value heads in this model (Gemma 2B uses multi-query attention: eight query heads of dimension 256 share one key-value head; the cache is paged in 128-token blocks, 3,249 blocks in this configuration).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*Gm-WIfdan2dKwTpLYZLVWA.png" /></figure><p>At batch 512 these families account for the step almost exactly: attention, 18 layers times 714 microseconds, is 12.9 milliseconds (41 percent of the 31.2-millisecond step), the gate-and-up family 6.8 milliseconds (22 percent), the down projection family 4.4 milliseconds (14 percent), and the logits pipeline, matmul plus conversion plus copy plus argmax, near 5.6 milliseconds (18 percent), summing to 30 of 31.2 milliseconds. The batch-linear conversion and copy of the logits, absent from folklore but second and third in the ranked table, became Strategy 6.</p><p>One trap in this analysis is worth disclosing in detail because it produced a wrong intermediate conclusion. Aggregating trace events by operation name suggested the decode feed-forward ops ran with a 1024-token shape, implying half of each step was padding at batch 512. Resolving every event to the tensor shape in its own HLO text disproved it: the population with 258 occurrences (the decode steps) carried shape at 375 microseconds, while the 1024-shaped instances occurred 18 times, once per layer, belonging to a single prefill pass; further shape populations at 16, 256, and 4096 tokens belonged to warmup and chunked-prefill buckets. Compiled programs reuse operation names across bucket sizes, so aggregation must key on shape, and occurrence counts must be checked against the expected step count. The corrected analysis confirmed the engine had selected the correct bucket throughout, and the planned max_num_batched_tokens intervention was cancelled before consuming any accelerator time.</p><h3>What batch scaling did to compilation</h3><p>Batch scaling is the strategy that interacts most with the ahead-of-time compilation model, so its costs are itemized here. Raising max_num_seqs changes the request padding menu (8, 16, 32, up to the configured maximum) and the attention request padding, so every new operating point paid a fresh precompilation of the backbone across all token buckets, the sampling graphs across all request buckets and flag combinations, the logprob graphs, and the placeholder substitution graphs, roughly 120 compilation tasks in the batch-512 configuration. Within a session the JAX compile cache made repeat initializations several times cheaper (285 seconds cold against 106 warm in the final sessions); across rebuilt virtual machines everything recompiles.</p><p>None of this touches steady-state throughput, which is why initialization was timed separately and excluded from every quoted number, but it defines the operational envelope: on this stack, changing the maximum batch is a deployment event, not a runtime knob. The Pallas attention kernel itself compiles through Mosaic once per bucket inside this same pass and was never the visible cost; the multi-minute totals are dominated by the XLA side of the menu, consistent with the compile-time microbenchmark of Section “ Measuring Compile Time: Pallas Versus XLA”</p><h3>Strategy 4: The Pallas Kernel Transplant</h3><p>To make the attention kernel modifiable, the stock ragged paged attention v3 source files (kernel.py, kernel_hd64.py, the tuned block-size tables, and util.py) were copied out of the installed tpu_inference package into a project directory, my_rpa, made importable with an empty __init__.py and placed on PYTHONPATH. A small module, imported by the benchmark script before the LLM constructor runs, injects the copies back into the live engine: it enumerates the callables defined in the stock kernel module, replaces each with the copy from my_rpa via setattr on the module, and additionally scans sys.modules for any tpu_inference or vllm module holding a direct reference to a replaced function, rebinding those references as well. Import order is the mechanism: the substitution must complete before the engine builds and precompiles, so the injected functions are what the compiler lowers.</p><p>Two wrapping rules mattered. Public entry points are wrapped with a functools.wraps probe that prints once on first invocation, giving runtime proof of execution from inside the engine core process. The inner kernel bodies, whose names begin with an underscore, are installed unwrapped, because pallas_call inspects the kernel function it is handed to build the grid and block specification, and a wrapper interferes with that lowering. Verification was double. At runtime, the probes printed during the engine’s precompilation pass. In hardware traces, the attention custom call, named for its block configuration (RPAd with a one-token query block and 2,048-token key-value blocks in this regime), carried source attribution my_rpa/kernel.py rather than the installed package path, across multiple sessions, batch sizes, and one stack version upgrade. That upgrade is itself a finding about the technique’s robustness: the newer tpu-inference resolved the kernel through the module attribute at call time, so the rebinding scan found zero direct references, yet the probes still fired, confirming the module-level setattr alone carried the substitution.</p><p>The measured result of the unmodified transplant was, by design, statistically identical throughput to stock; equality was the success condition, because a substitution that changed results before any intentional modification would be measuring integration error. Because the vendored kernel compiles through Mosaic inside the engine’s own precompilation pass, the substitution added no observable compile time and changed no HLO beyond the source attribution. This validated harness, an editable copy of the production attention kernel proven live, is the platform for the kernel optimization the results volume identifies as the main remaining opportunity: the kernel runs at roughly one quarter of memory bandwidth in this short-context regime, dominated by per-request overhead rather than data movement, and grows to about 41 percent of the step at batch 512. Strategy 7 reports that harness used for its purpose.</p><h3>Strategy 5: Three Quantization Routes</h3><p>With the dominant operations at 84 to 91 percent of bandwidth, halving weight bytes was the remaining structural lever. Three routes were attempted, and each terminated at a different layer of the stack. The verification signal for all three was the engine’s own initialization log: the model memory line (4.67 GiB for bfloat16 weights) and the key-value cache capacity line (415,872 tokens at 16-bit). A working weight quantization must roughly halve the first; a working cache quantization must roughly double the second.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*IQpi1BCQy0Iaia7xursqcQ.png" /></figure><p>Source inspection explained the pattern. The platform’s supported quantization list advertises several methods (compressed tensors, awq, fp8, and others), but the JAX quantization registry that the loader actually consults implements two, None and fp8, and the PyTorch fallback loader carries scaffolding for quantized layers with a comment stating that only unquantized linear layers are handled for now. The integer route’s checkpoint was produced with llmcompressor’s one-shot flow, scheme W8A16 over Linear targets with lm_head excluded as the conservative first configuration (in this model the output projection shares the embedding weights, the riskiest layer to quantize); the checkpoint remains valid for GPU serving even though this stack rejected it.</p><p>One procedural trap deserves a warning: a published quantized checkpoint with a nearly identical name targets Gemma 2, a different architecture from Gemma 2B, and benchmarking it would have silently invalidated every baseline; quantized checkpoints must match the exact base model of the campaign. The silent cache case is the instructive failure: the flag traveled through the whole configuration surface, appeared in the engine’s printed configuration, and changed nothing, which is why the study’s rule is to verify allocations and memory lines rather than flags. One run in the sequence also demonstrated the rule in reverse: a treatment run whose throughput matched baseline was traced to the environment variable never reaching the engine (the script predated the flag), so its number was reclassified as a third baseline replication rather than evidence about quantization. All findings were prepared as reports for the maintainers; they are version findings that a future release may change.</p><h3>Strategy 6: The Output Precision Patch</h3><p>The batch-512 trace ranked a 32-bit conversion of the logits tensor (f32, batch by 256000) and a subsequent copy of the same tensor at positions two and three of the whole table, 2.8 milliseconds per step combined, batch-linear, sourced to one line in the runner. Reading the code found the conversion in two places: the runner converts the logits to float32 before calling the sampler, and the sampler converts again internally. The decisive observation came from the sampler’s own ordering: the greedy token is computed by argmax on the original bfloat16 logits before either conversion executes, so under deterministic selection the 32-bit precision feeds only a returned tensor the benchmark never consumes, and since a widening conversion cannot change which element is largest, removing it cannot alter any output.</p><p>The patch made both conversion sites conditional on the sampler metadata’s do_sampling flag, a Python-level condition that is static at trace time, so the compiler builds separate graphs per case with no runtime branch: the sampling path stays byte-identical, and only greedy decoding skips the conversions. Application used anchored text replacement into the installed package with a uniqueness assertion on the anchor, the same mechanism as the kernel transplant; both anchors matched exactly once, confirming the code was as read. The prediction was a gain of three to seven percent.</p><p>The measurement was the opposite: 11,641 tokens per second on the first patched run and 8,839 on the second, against the 13,470 baseline, with batch time to first token multiplied by five (187 to roughly 995 milliseconds), reproducible on a warm compile cache and unstable between runs, the worst combination. The patch was reverted by the inverse replacement and the baseline reconfirmed. The probable mechanism is that the engine’s precompiled pipeline, including the graphs built during its warmup pass, was specialized around 32-bit logits; changing the returned dtype at one point pushed downstream consumers (output copies, logprob machinery, the runner’s own precompiled sampling graphs) off their precompiled paths on every step. The lesson is stated as a rule for this class of stack: operation-level arithmetic does not compose across just-in-time compilation boundaries, so a cost measured inside one compiled program cannot simply be subtracted by editing that program’s boundary. The inefficiency itself remains real and was documented for the maintainers, where the fix can be made at the pipeline level with all consumers rekeyed together. On a hosted virtual machine, edits to installed packages do not survive an environment rebuild, which makes both the experiment and its revert cheap and fully reversible.</p><h3>Strategy 7: Attention Kernel Block Size Tuning</h3><p>The final strategy used the transplant harness of Section 18 for its intended purpose. The target came from the traces: the attention kernel ran at roughly one quarter of memory bandwidth and had grown to the largest cost at scale. The kernel’s compiled name encodes its block configuration, and every trace showed key-value blocks of 2,048 tokens. Reading the vendored source explained why: the default heuristic sizes blocks to carry about 16 megabytes so that each block on its own saturates memory bandwidth, and under this model’s configuration that resolves to 2,048 tokens, the full maximum context. The workload’s sequences average about 140 tokens, so roughly 93 percent of every block was masked padding, loaded and processed for nothing.</p><p>The patch added an environment variable override inside the vendored copy’s block-size function for the decode case, leaving stock behavior untouched when the variable is unset. Block sizes are static compile-time arguments, so each override value produces its own compiled kernel, and the change carries two built-in verifications: a print line at every compilation that uses the override, and the compiled kernel’s name itself, which must show the new block size in any trace. Outputs cannot change, because block size affects scheduling, not arithmetic; the generations were confirmed identical to the stock kernel’s.</p><p>Measured with the full protocol: in a same-session comparison at 256 concurrent requests, 13,479 tokens per second stock against 16,700 tuned, a gain of 23.9 percent, with latency per token falling from 17.7 to 13.9 milliseconds, replicated across a rebuilt environment within 0.05 percent. A block of 128 tokens measured 16,805, statistically tied with 256, so the optimum is a plateau and only the stock 2,048 was a poor choice for this regime. At 512 concurrent requests the tuned kernel reached 19,511 tokens per second, and the batch-scaling step from 256 to 512 improved from 1.13 to 1.17 times, the expected signature of a cheaper batch-linear term. The campaign total became 12.7 times over the starting configuration, with the energy bound at 10 millijoules per token and 58.5 million tokens per dollar at the throughput maximum.</p><p>Two findings about method came with the result. First, profiled runs of the tuned kernel measured near baseline speed and briefly inverted the conclusion: the smaller block multiplies the number of traced kernel invocations roughly eightfold, and per-event tracing overhead becomes visible at that count. The rule of never quoting throughput from a profiled run is what resolved it; the unprofiled comparison and the trace’s kernel names together carry the claim. Second, the tuning is regime-specific. The 16-megabyte heuristic is correct for long contexts that fill the blocks; a long-context deployment should expect no gain and possibly a regression from this change. It is the speculative decoding lesson measured from the winning side: kernels are tuned for a regime, and the regime must be checked against the workload.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*o73cF_W5y_YtxjEj9-qCeg.png" /></figure><h3>Measuring Compile Time: Pallas Versus XLA</h3><p>The compile-cost comparison in Part I was produced by a standalone script that times the three phases of jax.jit separately: lower (tracing the Python function to an intermediate representation), compile (the backend building the executable), and first execution. The JAX compilation cache was disabled for the measurement, and each trial ran in a fresh process so no cached tracing survived between measurements.</p><p>The same attention computation was expressed twice: as high-level operations, two einsums with a float32 softmax between them, compiled by XLA; and as the flash attention Pallas kernel from jax.experimental, lowered through Mosaic, on identical bfloat16 inputs of shape . Over five trials the XLA route compiled in a median of 297.5 milliseconds (range 296.1 to 308.8) and the Pallas route in 50.7 (49.0 to 52.0), a reduction of 83 percent, with lowering costs of 24 and 39 milliseconds respectively and first executions equivalent at 1.1 to 1.8 milliseconds.</p><p>The interpretation is bounded: one operation at one shape, measuring developer iteration cost rather than runtime, and consistent with the mechanism that XLA searches a fusion and optimization space over the graph while a Pallas kernel arrives with its implementation decided, leaving only lowering. For kernel development, where the edit, compile, and test loop is traversed hundreds of times, the several-fold difference is the quantity that matters.</p><h3>Per-Strategy Economics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*D2imVZaX6FOKQXb5MztOQg.png" /></figure><p>The business reading is direct. The identical chip-hour, the identical watt budget, and the identical model produce ten times the tokens after two configuration changes, and 12.7 times after one further kernel-level change, which means fleets, energy budgets, and per-token unit costs planned on default configurations are overstated by up to that factor. A service generating one billion tokens per day needs roughly 7.5 chips at the starting configuration and 0.8 at the final one, before redundancy.</p><p>Time to first token deserves its own reading, because the strategies moved it in different ways. The selection fix improved it (the prefill’s own logits pass no longer pays the sampler search). Batch scaling raised it steadily, from 84 milliseconds at batch 32 to 307 at 512, because the number is defined here as prefill plus one decode step for the entire batch: more requests mean more prompt tokens through chunked prefill and a heavier first step. The failed precision patch multiplied it by five while claiming to optimize a decode cost, the clearest demonstration in the study that time to first token is a sensitive detector of compilation-path regressions and belongs in every benchmark printout, not only throughput.</p><p>Latency prices the choice of operating point: batch 256 holds 17.7 milliseconds per token, suitable for interactive services, while batch 512 trades 79 percent worse per-token latency for 13 percent more throughput and suits machine consumers, agent-to-agent traffic, and offline pipelines. The failed strategies also carry economic content: each was bounded by a single cheap run or a source inspection before it could consume engineering time at scale, and the reverted patch cost one session while producing the study’s most transferable engineering rule.</p><h3>Reproduction Notes</h3><p>Reproducing the study requires four artifacts: the notebook’s installation cell (versions resolve at run time and did drift once during the study, from JAX 0.7 to 0.10 with a matching libtpu jump, which the engine’s startup version line detects); the benchmark script with its environment variable switches; the injection module; and the vendored kernel directory created by copying the installed package’s ragged paged attention v3 files. Cell order matters: install, write the project files, copy the kernel sources, then run, with the injection imported before the engine constructor.</p><p>The 256-request operating point was replicated in three independent sessions on rebuilt virtual machines at 13,517, 13,470, and 13,484 tokens per second, a spread of one third of one percent; one further session returned 12,991, a 3.6 percent environment dip reported rather than discarded, so replications should expect the headline near 13.5 thousand with occasional session-level variance of a few percent.</p><p>Three habits carried the study and are the recommended protocol for any similar effort. First, never quote throughput from a profiled run, and never trust an unprofiled claim about where time goes. Second, pair every treatment with an authenticity check that does not depend on the treatment’s own flag: a memory line, an allocation size, a tensor shape in a trace, a probe print from inside the engine process; this study caught three silently inert configurations and one mislabeled trace by that habit alone.</p><p>Third, when aggregating traces, key on tensor shape and check occurrence counts against the expected step count, because operation names are reused across compiled buckets and name-level averages mix programs; that discipline both raised and then correctly killed the padding hypothesis raised in the batch-scaling analysis of Part II. The prompt set, the deterministic selection, the offline batch execution, and the enabled prefix cache define the workload regime; all numbers in this paper are claims about that regime, measured to the hardware’s roofline within it.</p><h3>Acknowledgments</h3><p>Google ML Developer Programs and the Google Developers Program supported this work by providing Google Cloud credits.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bebf7933abb1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From 1,540 to 15,338 Tokens per Second on a Single TPU Chip]]></title>
            <link>https://medium.com/@rubenszimbres/from-1-540-to-15-338-tokens-per-second-on-a-single-tpu-chip-1c5da4e77ef2?source=rss-30f09d1741ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c5da4e77ef2</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[tpu]]></category>
            <category><![CDATA[jax]]></category>
            <dc:creator><![CDATA[Rubens Zimbres]]></dc:creator>
            <pubDate>Sat, 04 Jul 2026 22:00:48 GMT</pubDate>
            <atom:updated>2026-07-06T15:01:07.571Z</atom:updated>
            <content:encoded><![CDATA[<p>A Measurement Study of Large Language Model Inference Optimization on TPU v5e: <em>Throughput, latency, energy, and cost results for Gemma 2B served with vLLM, with observations on the JAX and Pallas compilation stack</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*9h3EZuDRdLxKcyAG_ABAFg.png" /></figure><p>This is the first of two articles. Here, I’ll tell you the what and why. In the next, I’ll tell you the “how”.</p><p>A single <a href="https://docs.cloud.google.com/tpu/docs/v5e">TPU v5e chip</a> is Google’s cost-optimized AI accelerator designed for efficient inference and medium-scale training. It features <strong>1 TensorCore</strong>, delivers up to 393 trillion Int8 operations per second (TOPS), and includes <strong>16 GB of memory</strong>. It provides up to a 2.5x increase in price-performance over previous generations, however there more advanced chips lately.</p><h4>Key Hardware Specifications</h4><ul><li><strong>Compute Performance:</strong> 197 BF16 TFLOPS and 393 Int8 TOPS.</li><li><strong>Memory Capacity:</strong> 16 GB of high-bandwidth memory (HBM2E).</li><li><strong>Memory Bandwidth:</strong> 819 GB/s.</li><li><strong>Architecture:</strong> 1 TensorCore with four matrix-multiply units (MXUs).</li></ul><h4>Availability &amp; Usage</h4><ul><li><strong>Cloud Access:</strong> Rented directly on Google Cloud (via Compute Engine or Google Kubernetes Engine) and also available at Colab.</li><li><strong>Flexible Slices:</strong> While single chips can be utilized, they are commonly provisioned in slices of 1, 4, or 8 chips depending on your model scale.</li><li><strong>Cost:</strong> Single-chip VMs typically cost around $0.35 per hour on-demand.</li><li><strong>Framework Support:</strong> Compatible with major AI frameworks like PyTorch, JAX, and TensorFlow.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*nULBeBUUBHP9frAFg-X1SQ.png" /><figcaption>How to Think About TPUs | How to Scale Your Model. Source: Google DeepMind.</figcaption></figure><h3>Executive Summary</h3><p>This article comes from a whitepaper I recently wrote, and reports a measurement study of large language model inference on a single Google TPU v5e chip, telling the what and why. Part 2 will bring the “how”. The subject model was Gemma 2B, served through the vLLM inference engine on the JAX based TPU software stack. The study asked one question: how much of the chip’s potential does a default serving configuration leave unused, and how far can disciplined measurement recover it without changing the model or hardware.</p><p>The answer was a factor of ten. The starting configuration produced 1,540 tokens per second. The final configurations produced 13,470 tokens per second at the recommended operating point and 15,338 at the throughput maximum. The gain came from two interventions, each identified by profiling the chip rather than by guesswork: a change to how output tokens are selected, worth 2.3 times on its own, and a systematic increase in the number of requests processed together, worth a further 4.3 times. No model weights were changed, no accuracy was sacrificed, and no additional hardware was used. Because the chip and its power envelope stayed constant, every efficiency metric improved by the same factor: the upper bound on energy per token fell from roughly 128 millijoules to 13, and at published on demand pricing the cost of generation fell from roughly 4.6 million tokens per dollar to 46 million.</p><p>The study is equally defined by what did not work, and those results are reported with the same care. Speculative decoding, a widely promoted technique, reduced throughput by a factor of six in this workload’s operating regime. Weight and cache quantization, the standard next step, turned out to be unavailable on this software stack for this model, a finding documented at the source level. A precision optimization that looked certain on paper reduced throughput by 14 percent when applied, and was reverted. Each negative result was verified before being accepted, and each carries a practical lesson for teams running similar systems.</p><p>This document describes what was done, why, and with what results. The specific procedures, code, and integration techniques (the “how”) will be a Part 2 of this article.</p><p>If you want to go straight to the point, read my full paper, available at:</p><p><a href="https://zenodo.org/records/21221952">https://zenodo.org/records/21221952</a></p><h3>Background and Motivation</h3><p>Serving a language model is a different engineering problem from training one. Training is dominated by arithmetic. Serving, and in particular the token by token generation phase called decode, is dominated by data movement. For every token generated, the chip must read essentially the entire model from its high bandwidth memory. On the chip studied here, that memory can deliver 819 gigabytes per second, and the model occupies roughly five gigabytes in its standard 16 bit form. Those two numbers set a hard ceiling on generation speed that no amount of clever software can exceed. The question for any serving deployment is how close to that ceiling it operates.</p><p>Most deployments do not know the answer, because the default configurations of serving frameworks are tuned for generality rather than for a specific model on a specific chip. This study set out to find the answer for one concrete case: Gemma 2B, a two and a half billion parameter open model from Google, served with vLLM, the most widely used open source inference engine, on a single TPU v5e chip, the efficiency oriented member of Google’s TPU family. The environment was a standard hosted notebook with one chip, which makes the study reproducible by any practitioner with the same access.</p><p>The choice of a small model on a single chip was deliberate. Small models are increasingly the economic workhorses of production systems, handling classification, extraction, routing, and drafting tasks where a frontier model would be wasteful. For these workloads, tokens per second per dollar decides profitability, and a single chip is the unit at which that metric is set; whatever efficiency is achieved on one chip multiplies directly across a fleet. The study also had a second purpose. The TPU serving stack for vLLM is new, built on a hardware plugin that unifies JAX and PyTorch execution paths and compiles model code through XLA, with performance critical kernels written in Pallas, a JAX embedded language for custom TPU and GPU kernels. New stacks have gaps, and part of the value of a measurement campaign is mapping where the advertised surface and the implemented surface differ. Several such gaps were found and are reported here.</p><h3>What Was Done</h3><p>The method of the study can be stated in one sentence: measure first, predict second, change third, and verify fourth. Every intervention followed the same loop. A hardware profile of the running system identified where device time was actually spent, at the level of individual compiled operations. From that profile, a specific change was predicted to produce a specific gain. The change was applied, one variable at a time, and the result was compared against the prediction. Changes that failed their prediction were reverted and recorded, not rationalized.</p><p>Three categories of change were explored. The first was the token selection path, the code that turns the model’s raw output scores into chosen tokens. The second was the batching configuration, the number of independent requests the engine processes concurrently. The third was the byte reduction category, covering weight quantization, cache quantization, and numeric precision in the output pipeline. In parallel, a custom attention kernel written in Pallas was integrated into the production serving path and validated against the stock implementation, establishing a harness for future kernel work.</p><p>Two disciplines were applied throughout. First, every measured number required a paired verification: a second counter, a repeated run, or a trace level confirmation that the intended change had actually taken effect. On three occasions this caught a configuration that had been requested but silently not applied, which would otherwise have produced false conclusions. Second, headline numbers were always taken with profiling disabled, since tracing adds overhead; profiled runs were used only to rank operations. The workload for all measurements was a fixed set of short prompts generating 128 tokens each under deterministic selection, executed as an offline batch. This is a decode heavy, short context workload; its limitations are discussed in Section 9.</p><h3>Results: The Throughput Campaign</h3><p>Profiling the starting configuration produced a surprise. The attention computation, the usual suspect in transformer performance work, accounted for only a small share of device time. The largest consumer, at roughly 55 percent, was the token selection machinery, executing an expensive search procedure on every step to support a sampling feature the workload was not meaningfully using. Reconfiguring the request so that selection took the direct path removed that cost entirely: throughput rose from roughly 1,540 to 3,577 tokens per second, a factor of 2.3, with no change to the model.</p><p>With the selection cost gone, the remaining time was dominated by operations whose duration is set by memory traffic: reading the feed forward weights and the large output projection over the model’s 256,000 entry vocabulary. Such operations have a decisive property: their cost per step is fixed regardless of how many requests share the step, so doubling the concurrent requests nearly doubles throughput. The study followed this logic up a doubling ladder, verifying at each rung that the predicted operations stayed fixed in cost while others grew.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*OGgV2TSWv56cTkUNwDBgWQ.png" /><figcaption><em>Table 1. Batch scaling results. All rows are clean timing runs with profiling disabled. Latency per token is the average interval between successive tokens of one request. The watt figure uses the vendor reported 197 watt chip rating as an upper bound and is discussed in Section 7.</em></figcaption></figure><p>The scaling factors tell the story precisely. Doubling from 32 to 64 requests returned 1.81 times the throughput, nearly the ideal 2.0, confirming the chip had been waiting on memory rather than computing. Each further doubling returned less: 1.50, then 1.38, then 1.13. The diminishing curve was predicted in advance, because two categories of work do grow with the number of requests: the attention computation, whose cache reads are proportional to requests served, and the output post processing. At 512 concurrent requests the crossover was measured directly: the attention family had become the single largest consumer of the step, and further batching bought almost nothing. The recommended operating point is 256 concurrent requests, which captures 88 percent of maximum throughput at 17.7 milliseconds per token, against 31.2 at the maximum. Throughput oriented deployments can run at 512; interactive services will prefer 256 or below. The value of the table is that the trade is quantified rather than assumed.</p><h3>The overall arc</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*7VVfDGsGif2qlICthhI0bA.png" /><figcaption><em>Table 2. The campaign in four rows. Two interventions account for the entire gain. The chip, the model, and the model’s outputs for this workload are unchanged throughout.</em></figcaption></figure><p>A separate verification deserves mention because it anchors the credibility of every number above. The 256 request configuration was measured in three independent sessions, on rebuilt virtual machines, and returned 13,517, 13,470, and 13,484 tokens per second, a spread of one third of one percent. One outlier session returned 12,991, a 3.6 percent dip attributed to environment variance, and is reported here rather than discarded. The headline is therefore quoted as approximately 13.5 thousand tokens per second with a known spread, not as a single lucky run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*YnxCcM_tOkKcI06PUD1zbg.png" /></figure><h3>What the Measurements Revealed About the Chip</h3><p>Three utilization questions matter for a serving deployment, and the study answered each with a direct measurement rather than an estimate.</p><p>First, is the chip idle. Trace analysis found roughly 85 microseconds of device silence per 11.9 millisecond generation step at 128 concurrent requests, which is 0.7 percent. The scheduler and host side machinery keep the accelerator more than 99 percent occupied even at high concurrency; host overhead is not a bottleneck in this stack, and the study can say so with a number rather than an impression.</p><p>Second, how hard is the memory system working. The two dominant weight reading operations were measured at 91 and 84 percent of the chip’s theoretical memory bandwidth while executing. These operations are finished as optimization targets in their current numeric format; only reducing the bytes themselves could make them faster. Averaged across a whole step, memory utilization is roughly 40 percent, because part of each step goes to operations limited by other factors.</p><p>Third, how hard is the arithmetic hardware working. Arithmetic utilization during generation is roughly 17 percent of peak. Quoted alone, that would suggest waste. It does not. Generation is memory bound by nature; the arithmetic units spend most of their time waiting for weights to arrive, and low arithmetic utilization alongside high memory utilization is the expected signature of a well tuned decode workload. During the prompt processing phase, which is arithmetic bound, the same chip measured roughly 47 percent of arithmetic peak, confirming the hardware performs when the workload demands it.</p><p>The trace work also produced a finding of general value: which operations are free to batch and which are not. Across an eightfold increase in concurrent requests, the feed forward weight read measured 183, 183, and then 375 microseconds, flat until the arithmetic itself began to matter at the largest batch. The attention operation over the same range measured 89.6, 179.1, and 714.4 microseconds, growth of 7.97 times against an 8 times batch increase, linear to within measurement error, while the output projection stayed flat. This comparison, predicted in advance and confirmed at three operating points, explains the entire shape of the scaling curve in Table 1 and generalizes to any transformer on any memory bound accelerator: batching is nearly free until attention and per request bookkeeping catch up, and the crossover is discoverable by measurement.</p><h3>Arithmetic intensity</h3><p>The single quantity behind all of these regimes is arithmetic intensity, defined for any operation as</p><p><em>Arithmetic Intensity = Computation FLOPs ÷ Communication Bytes.</em></p><p>The accelerator has an intensity of its own: peak arithmetic rate over memory bandwidth, here 1.97e14 FLOPs per second over 8.19e11 bytes per second, about 240 FLOPs per byte. An operation below 240 cannot keep the arithmetic units fed and is memory bound; above 240 it is compute bound. Because a decode matmul multiplies a batch of B token vectors against a fixed weight matrix, its intensity is approximately B, which turns the batching campaign into one prediction: the weight streaming operations should stay memory bound, and flat in cost, until the batch approaches 240 tokens per step, and become compute bound beyond it. Table 3 computes the intensities from the model’s dimensions and checks them against the measured times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*DBhcFbjDP0v3JZHtkmFrjA.png" /><figcaption><em>Table 3. Arithmetic intensity in FLOPs per byte, derived from the model’s published dimensions in bfloat16, against the chip’s critical intensity of about 240. The measured times of Section 4 sit within ten percent of the corresponding roofline on both sides of the threshold.</em></figcaption></figure><p>The table explains three campaign results at once. The weight streaming operations cross the 240 threshold between batch 256 and 512, exactly where their measured cost stopped being flat, which is why batch 256 is the recommended operating point: the last rung of the ladder on the memory bound side of the roofline. Attention’s intensity is about eight regardless of batch, because eight query heads share one key value head and each request reads its own cache; an operation that can never amortize its bytes across the batch is precisely one whose cost grows linearly with it. And the same threshold predicts where quantization would pay: halving the weight bytes doubles an operation’s intensity, moving the crossover to roughly batch 120 and halving the memory bound cost below it.</p><h3>What Did Not Work, and Why That Matters</h3><h4>Speculative decoding</h4><p>Speculative decoding is a technique in which a cheap mechanism drafts several tokens ahead and the model verifies them in one pass. It is frequently promoted with claims of three to four times speedups. Applied to this workload at 32 concurrent requests, it reduced throughput from roughly 3,555 to 607 tokens per second, a loss of a factor of six. The explanation is regime dependence. Speculation spends spare compute to reduce the latency of individual requests, a good trade when the chip has idle capacity, as it does at very small batch sizes. At meaningful batch sizes the chip has no spare capacity; every verified draft displaces real work, and rejected drafts are pure waste. It is a latency tool being sold as a throughput tool. The lesson: published speedups carry an implicit operating regime, and the only defense is measuring in one’s own regime, which takes a single run.</p><h4>Quantization</h4><p>With the bandwidth ceiling measured and nearly reached, the remaining structural lever was reducing the bytes themselves: storing weights in 8 bits instead of 16, halving traffic on exactly the operations measured at 84 to 91 percent of bandwidth. The study attempted this through every route the stack advertises, and the results form a finding of their own. The integer 8 bit route was rejected with an explicit error: accepted by the platform’s configuration surface, not implemented in its model loader for this model family. The floating point 8 bit route for weights requires a checkpoint quantized in advance and fails unhelpfully when given a standard one; there is no on the fly path. The 8 bit route for the attention cache was the most instructive: the flag was accepted end to end, the engine ran, and the cache was allocated in 16 bits anyway, with no warning. Only a direct check of the allocated cache size revealed that nothing had happened. Three attempts produced one loud failure, one unhelpful failure, one silent failure, and zero functioning quantization paths for this model on this stack version.</p><p>This is reported not as criticism but as a map; the stack is new and openly labels parts of this machinery as incomplete. The general lesson is sharp: on a fast moving inference stack, a configuration flag being accepted is not evidence that it did anything. Verify the allocation, the memory footprint, or the trace, not the flag. Teams that quote benchmark improvements from a quantization flag without such verification may be measuring nothing.</p><h3>The precision experiment</h3><p>Trace analysis identified a genuine inefficiency in the output pipeline: on every generation step, the model’s output scores are converted to 32 bit precision and copied, a cost that grows with batch size and reached roughly nine percent of the step at the largest batch. Under deterministic selection this precision is mathematically unnecessary, and the selection code demonstrably computes its answer before the conversion happens. Removing it looked certain to return several percent.</p><p>It did the opposite: applied as a minimal, correctness preserving change, it reduced throughput by 14 percent and multiplied time to first token by five, reproducibly, and was reverted with the baseline reconfirmed. The most probable explanation is that the conversion, wasteful in isolation, was load bearing for the surrounding compiled pipeline, which had been built and specialized around 32 bit outputs; changing the type at one point forced the runtime off its optimized path downstream. The lesson is among the most valuable in the study: on a just in time compiled stack, operation level arithmetic does not compose across compilation boundaries, and a change that removes a measured cost can add a larger unmeasured one. The inefficiency is real and has been documented for the stack’s maintainers, where it can be fixed at the pipeline level.</p><h3>Custom Kernels, Pallas, and the Compilation Stack</h3><p>Part of the study integrated a custom attention kernel, written in Pallas, into the live serving path of the engine, replacing the stock implementation. The integration was verified twice over: by instrumentation proving the custom code executed inside the engine’s core process, and by hardware traces attributing the attention operation’s device time to the custom kernel’s source file across multiple sessions and batch sizes. The measured result of the substitution was, by design, no change in performance: the custom kernel reproduced the stock kernel’s behavior, and throughput at the validation point was statistically identical to the stock baseline. That equality was the success condition, because it proves the harness is faithful; a substitution that changed results before any intentional modification would be measuring integration error, not kernel quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xn4iaGZMarw2FV-wwbPYaQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bSq5xD4PHpTBAVutRGmNyA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AlxA-jaI5pFdYy4iIHigLw.png" /></figure><p>To be direct about what is and is not claimed: this study does not claim a throughput gain from custom Pallas kernels, and no comparison here should be read as Pallas outperforming the XLA compiled path. What the kernel work established is the platform for such gains. The traces show the attention kernel is the one major operation with substantial headroom: it runs at roughly one quarter of memory bandwidth in this short context regime, dominated by per request overhead rather than data movement, and grows to the largest consumer of the step at scale, roughly 41 percent of device time at 512 concurrent requests. A validated, hot swappable custom kernel sitting on exactly that operation is the study’s main asset for future work. The substitution and verification technique is part of the undisclosed methodology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YnXDB-2fk5ErdEa_BM085g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*zUIO_y1p1eKucCPckE_jpA.png" /></figure><h3>Observations on the compilation model</h3><p>The TPU serving stack compiles the model ahead of time for a fixed menu of input sizes, called buckets, covering the possible counts of requests and tokens per step. This design delivers the near zero host overhead measured in Section 4, and it worked correctly under scrutiny: a hypothesis that the engine was running steps in an oversized bucket and wasting half its feed forward compute on padding was raised from an early reading of the traces, tested by resolving every traced operation to its true tensor shape, and found to be false. The engine had selected the correct bucket throughout. This dead hypothesis is reported deliberately, because the diagnostic that killed it, attributing every operation to its actual shape rather than trusting names or averages, is reusable and cheap.</p><p>The costs of the compilation model are also worth stating. Engine startup, which includes compiling the full bucket menu, ranged from roughly two to five minutes depending on cache state, and each new maximum batch size adds compilation work; irrelevant for long running servers, a real consideration for elastic infrastructure. The compile cache reduces repeat costs within an environment but does not persist across rebuilt environments in the hosted setting used here.</p><p>The precision experiment in Section 5.3 illustrates the model’s rigidity: the specialization that makes steady state execution fast also makes point changes to the compiled pipeline hazardous. Optimization on this stack is best pursued above the compiler, through configuration and workload shape, or below it, through whole kernels, rather than by editing the middle.</p><p>Five specific gaps between the stack’s advertised and implemented surface were documented, from the quantization findings of Section 5.2 to inefficiencies in the selection and output pipeline, including the observation that requesting the permissive value of a sampling parameter still pays that parameter’s full computational cost. All five have been prepared as reports for the maintainers; a new stack improves at the rate its users measure it.</p><h3>Compile time: Pallas versus XLA on the same operation</h3><p>A separate microbenchmark compared the compilation cost of the two routes to a TPU kernel. The same attention computation was prepared twice: once expressed as high level operations and compiled by XLA, and once as a hand written Pallas kernel lowered through the Mosaic compiler, with equivalent results and equivalent execution times once compiled. Over five repeated trials in fresh processes with the compilation cache disabled, the XLA route compiled in a median of 297.5 milliseconds and the Pallas route in a median of 50.7 milliseconds, a factor of 5.9 and a reduction of 83 percent, with narrow spreads on both sides.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*_WQ1tu2g50ma3I7qjZ7DjA.png" /><figcaption><em>Table 3. Compilation cost of the same attention computation by route, single shape, five trials. Execution times after compilation are equivalent; the difference is compile cost only.</em></figcaption></figure><p>The plausible mechanism is that XLA must search a large optimization and fusion space over the operation graph, while a Pallas kernel encodes the implementation decisions in advance, leaving only the lowering work. The scope of the claim is narrow, one operation at one shape, and it concerns developer iteration cost, not serving throughput. It matters nonetheless: kernel development traverses the edit, compile, and test loop hundreds of times, so a several fold reduction in the loop’s fixed cost is a material productivity difference, and it lowers the price of pursuing the custom kernel opportunity identified.</p><h3>Energy</h3><p>The chip’s power envelope did not change during the study, so the tenfold growth in output translates directly into a tenfold reduction in energy per token. Because the hosted environment exposes no power telemetry, the study uses the vendor reported chip rating of 197 watts as a deliberate upper bound; a peer reviewed survey both sources this figure and notes that real workload draw typically runs 30 to 60 percent below rating. All figures below are therefore conservative ceilings, and the improvement ratios are exact regardless of true wattage, since the constant cancels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*KMF8flaRW25krafsyeP7KQ.png" /><figcaption><em>Table 4. Energy per generated token at the chip’s rated power. The tenfold improvement holds for any true wattage because the same chip produced every row.</em></figcaption></figure><p>A cross check against independent literature supports the scale of these figures. A published survey estimates roughly 94 joules per thousand tokens for a 7 billion parameter model on this chip class at a batch of 32; this study measured roughly 20 joules per thousand tokens for a 2.5 billion parameter model at a batch of 128. The model here streams roughly 3.5 times fewer weight bytes per token and runs at four times the batch, and those two factors together predict almost exactly the observed gap, the kind of external validation single environment benchmarks rarely provide. The result also carries a point for sustainability planning: batching is an energy optimization, not only a throughput one. The same silicon at the same rated power produced 4.3 times the tokens between the first and last rows of the batching campaign; fleets sized on per token energy budgets can be several times smaller than naive per chip figures suggest, provided the serving configuration is measured rather than defaulted.</p><h3>Economic Implications</h3><p>The commercial meaning of the study is easiest to see in tokens per dollar. The chip class used here is publicly priced at 1.20 dollars per chip hour on demand, so throughput converts directly into unit cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*nGyVArakURdCBEqRfk27HA.png" /><figcaption><em>Table 5. Generation cost at 1.20 dollars per chip hour, on demand pricing. Committed use and spot pricing would lower every row proportionally.</em></figcaption></figure><p>Read as a business statement: the identical hardware bill buys ten times the product. A service generating one billion tokens per day would need roughly 7.5 chips around the clock at the starting configuration and 0.8 at the final one, before redundancy, a difference of roughly 217 versus 22 dollars per billion tokens at on demand rates. For products with per request economics, such as summarization, extraction, or agent steps, this is the difference between a feature that loses money and one that carries margin.</p><p>Two qualifications keep the claim honest. First, these are offline batch numbers on a decode heavy, short context workload; a production service with irregular arrivals, longer prompts, and latency targets will operate below the maximum, which is why Table 1 reports the full throughput and latency frontier rather than one number. The 256 request row, not the 512 row, is the realistic planning figure for interactive services. Second, none of the gain required engineering exotica: one configuration correction and one batching campaign, both reachable by any team willing to profile before tuning.</p><p>The economic finding is less that this chip is cheap and more that unmeasured serving configurations routinely forfeit most of the hardware already being paid for. The latency columns carry their own economic content: batch time to first token grew from 84 to 307 milliseconds across the campaign, and per token latency roughly quadrupled between smallest and largest batch. A chat product sells responsiveness and should buy it with a smaller batch; a nightly document pipeline sells volume and should run at the maximum. The contribution here is the measured frontier on which that choice can be made deliberately.</p><h3>Scope and Limitations</h3><p>The study’s claims are bounded, and the bounds are part of the result. The workload was an offline batch of short, repeated prompts with 128 token outputs under deterministic selection, a favorable regime for decode throughput: short contexts keep attention cheap, deterministic selection avoids sampling costs, and offline batching avoids the scheduling losses of irregular traffic. Services with long documents, streaming arrivals, or sampling requirements will land below these numbers, though the method for finding their own frontier is identical.</p><p>Prompt caching was enabled and, with repeated prompts, makes the prompt processing phase inexpensive; the figures are decode dominated and affected only marginally, but replications should note the setting. The hardware scope is one chip; production TPU deployments run pods of many, an axis this study deliberately did not touch. For a model of this size, replication across chips scales near linearly, so the single chip figures are best read as the unit economics of a fleet.</p><p>The software scope is one version of a fast moving stack, and several findings, in particular every quantization result, are version findings a future release may change. Power figures are rated bounds, not measurements, because the environment exposes no telemetry. Output quality was verified by determinism and inspection rather than formal evaluation, sufficient here because no accepted intervention altered the model’s computation.</p><h3>Conclusions</h3><p>A single mid tier accelerator chip, running an unmodified open model on an open serving engine, moved from 1,540 to 15,338 tokens per second through measurement driven configuration alone. The recommended operating point delivers roughly 13.5 thousand tokens per second at 17.7 milliseconds per token, at an energy ceiling of 15 millijoules per token and a generation cost near 2.5 cents per million tokens on demand. The chip finished the campaign more than 99 percent occupied, with its dominant memory operations at 84 to 91 percent of theoretical bandwidth: the study did not stop at a convenient number, it stopped at the hardware. The remaining headroom is mapped rather than speculative. The attention kernel operates at roughly a quarter of memory bandwidth in this regime and is the largest cost at scale, with a validated custom kernel harness sitting on exactly that operation; weight quantization would roughly halve the dominant memory traffic and is blocked only by stack maturity; and an identified inefficiency in the output pipeline is worth several percent once fixed at the right layer. Each is a known door with a measured value behind it.</p><p>The transferable findings are three. Profile before optimizing, because the largest cost in this system was not where transformer folklore said it would be. Verify that changes took effect, because this stack accepted three configurations it did not apply, and only direct measurement told the difference. And treat every published speedup as a claim about a regime, because the technique measured here as a six times loss is legitimately a win in the regime its advocates measured. None of these require special access or budget; they require the discipline of pairing every number with its verification. The procedures, code, kernel integration techniques, and patch methodology developed during the study are not disclosed in this document.</p><h3>Acknowledgements</h3><p>Google ML Developer Programs and Google Developers Program supported this work by providing Google Cloud Credits.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c5da4e77ef2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Develop a Multi-Orchestration Neighborhood Watch App with Antigravity CLI]]></title>
            <link>https://medium.com/@rubenszimbres/develop-a-multi-orchestration-neighborhood-watch-app-with-antigravity-cli-1ed95b004c3b?source=rss-30f09d1741ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ed95b004c3b</guid>
            <category><![CDATA[fbi]]></category>
            <category><![CDATA[google-antigravity]]></category>
            <category><![CDATA[true-crime]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[Rubens Zimbres]]></dc:creator>
            <pubDate>Wed, 01 Jul 2026 01:18:55 GMT</pubDate>
            <atom:updated>2026-07-01T01:45:26.347Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*jV2xnpF-LxPSVbbE0QbPHw.png" /></figure><p>I always develop sample apps focusing on the real utility for the community. I imagined the following situation: if you are a tourist, or even a person looking to rent or buy a real state property, it is useful to check the environmental factors in that area, as well as infrastructure complaints and crime rates.</p><p>These are the main reasons I developed the Neighborhood Watch App, as a valuable tool for people to understand the risks of certain US areas. Neighborhood Alert aggregates crime reports, weather alerts, 311 infrastructure complaints, and seismic data for any US address, normalizes them into a single composite risk score, and runs the aggregated collected data through Google Gemini to produce an actionable safety briefing.</p><p>The collected data pipeline is built as a <strong>multi-agent orchestration layer</strong>: each data source is an isolated agent in a dependency graph. The graph runs on a background thread, and the Streamlit UI streams live per-subagent progress while the work executes.</p><h3>ARCHITECTURE</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*NqFd3m1LtaIwSMHoRrQeiw.png" /><figcaption>Multi-agent architecture for the Neighborhood Watch App</figcaption></figure><p>The application has two execution paths over the same underlying fetch methods:</p><ul><li><strong>Synchronous path</strong> (<em>DataFetcher.fetch_all</em>): geocode, then a 5-way parallel fan-out, then risk scoring. Used by tests and any non-UI caller.</li><li><strong>Orchestrated path</strong> (<em>utils/orchestration.start_pipeline</em>): fetches agents in a dependency graph, executed on a background thread with live progress and the AI briefing as the final node. Used by the UI.</li></ul><p>We have the following Components and Responsibilities:</p><p><strong>Agent:</strong> is a named unit of work. An <strong>Orchestrator</strong> runs agents as topological units, many in parallel (APIs). Each unit executes in a thread pool and failures are isolated per node. The worker thread writes on the <strong>Progress Board. </strong>The final assembler is given by <strong>Gemini</strong> and makes sense of the data collected via APIs.</p><p>I used the following APIs:</p><ul><li><strong>Geocode</strong> for US addresses, using OpenStreetMap</li><li><strong>Weather:</strong> OpenWeather maps + NWS alerts</li><li><strong>NWS:</strong> National Weather Service:</li><li><strong>Crime:</strong> uses FBI Data Explorer, for statistics and aggregated data</li><li><strong>Infrastructure: </strong>311 Open Data (Socrata)</li><li><strong>Seismic:</strong> USGS EarthQuakes</li><li><strong>Composer:</strong> Gemini</li></ul><p>This data will generate a given risk for the area, and Gemini provides a briefing for the user. I developed the sketch project, a framework agnostic multi-agent system, then I converted it to ADK, with <em>Antigravity CLI, w</em>ith the <strong><em>/goal</em></strong> command. When using this command, be sure to have the complete architecture and requirements of the system in the prompt, or precede it with the <strong><em>/planning</em></strong> command, as the <strong><em>/goal</em></strong> is a fully autonomous execution command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*3jsN6gdcE8bRNgc8gOz_Og.png" /><figcaption>Antigravity CLI performing a task</figcaption></figure><p>The <strong>Geocoding</strong> returns coordinates (lat long) for any given US address. Weather API returns a 5-day forecast, returning a risk score. With the <strong>FBI Crime Data Explorer</strong> you have violent crimes, theft, burglary and other offenses, providing totals that will be use the severity of risk in that area.</p><p>The <strong>Socrata</strong> Open Data API has many different endpoints, each one related to a specific city. The <strong>311 infrastrucuture</strong> complaints API provides datasets per city. The seismic app, that I also used <a href="https://medium.com/p/aef07280cc28">here</a>, in the <a href="https://medium.com/p/aef07280cc28">Xtreme Weather App</a>, uses <strong>USGS (United States Geological Survey)</strong> provides a GeoJSON with events from a 500km limit radius of the geocoded point, with minimum magnitude of 2.0 in Richter Scale, limited to the 10 most recent results.</p><p>Here, I used <strong>gemini-3.5-flash</strong> via Vertex AI to aggregate and interpret the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*ZsT9uMNyRqUW8wdrEY1kTw.png" /><figcaption>Project Structure for the Neighborhood Watch App</figcaption></figure><h3>ANTIGRAVITY</h3><p>In order to install Antigravity-cli on Linux to help you code the solution, you run:</p><pre>curl -fsSL https://antigravity.google/cli/install.sh | bash</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*w8EaOvwBX7hNJRNgB_zyEA.png" /></figure><p>You if are a Gemini-cli user, you can import settings in a simple manner:</p><pre>agy plugin import gemini</pre><p>You can also get some skills here: <a href="https://github.com/google-gemini/gemini-skills">https://github.com/google-gemini/gemini-skills</a>, and add them in this path: ~/.gemini/antigravity-cli/skills/</p><p>Your settings are now in: ~/.gemini/antigravity-cli/settings.json</p><p>Simple as that. For a more detailed walkthrough, check these series of articles from <a href="https://medium.com/u/802a4d428d95">Romin Irani</a>:</p><p><a href="https://medium.com/google-cloud/antigravity-cli-tutorial-series-12b46cfe3bf2">Antigravity CLI Tutorial Series</a></p><p>… and from <a href="https://medium.com/u/44b0a9bf7c82">xbill</a>:</p><p><a href="https://medium.com/google-cloud/getting-started-with-antigravity-cli-26c5da90951f">Getting Started with Antigravity CLI</a></p><p>The Antigravity-cli documentation can be accessed here:</p><p><a href="https://antigravity.google/docs/cli/overview">https://antigravity.google/docs/cli/overview</a></p><h3>CODE SNIPPETS</h3><p>Initially we define the API endpoints, cities’ total population (for the risk score), and the severity weights:</p><pre>SEVERITY_WEIGHTS = {<br>    &quot;Violent Crime&quot;:             10,<br>    &quot;Weapons Offense&quot;:            8,<br>    &quot;Theft &amp; Burglary&quot;:           4,<br>    &quot;Vehicle Incident&quot;:           3,<br>    &quot;Drug Offense&quot;:               2,<br>    &quot;Fraud &amp; Deception&quot;:          2,<br>    &quot;Property Damage/Vandalism&quot;:  2,<br>    &quot;Trespassing&quot;:                1,<br>}</pre><p>Then we use the data fetcher, to obtain information from the APIs (the full code Gurhub repo is provided ahead).</p><pre>def _fetch_socrata_crime(self, lat: float, lon: float, city_key: str) -&gt; List[Dict]:<br>    url = SOCRATA_ENDPOINTS[city_key]<br>    since = (datetime.utcnow() - timedelta(days=30)).strftime(&quot;%Y-%m-%dT00:00:00.000&quot;)<br>    headers = {&quot;X-App-Token&quot;: SOCRATA_TOKEN} if SOCRATA_TOKEN else {}<br><br>    if city_key == &quot;chicago&quot;:<br>        params = {<br>            &quot;$limit&quot;: 500,<br>            &quot;$where&quot;: f&quot;date &gt;= &#39;{since}&#39;&quot;,<br>            &quot;$select&quot;: &quot;date,primary_type,description,location_description,latitude,longitude,block&quot;,<br>            &quot;$order&quot;: &quot;date DESC&quot;,<br>        }<br>    elif city_key == &quot;new york&quot;:<br>        params = {<br>            &quot;$limit&quot;: 500,<br>            &quot;$where&quot;: f&quot;cmplnt_fr_dt &gt;= &#39;{since}&#39;&quot;,<br>            &quot;$select&quot;: &quot;cmplnt_fr_dt,ofns_desc,law_cat_cd,boro_nm,latitude,longitude&quot;,<br>            &quot;$order&quot;: &quot;cmplnt_fr_dt DESC&quot;,<br>        }<br>    else:<br>        params = {&quot;$limit&quot;: 300, &quot;$order&quot;: &quot;:id DESC&quot;}<br><br>    try:<br>        r = self._safe_get(url, params=params, headers=headers, timeout=12)<br>        r.raise_for_status()<br>        raw = r.json()<br>        <br>        # If the 30-day filter returned no data (often due to city data delays), fetch the latest available<br>        if not raw and &quot;$where&quot; in params:<br>            del params[&quot;$where&quot;]<br>            params[&quot;$limit&quot;] = 100<br>            r = self._safe_get(url, params=params, headers=headers, timeout=12)<br>            r.raise_for_status()<br>            raw = r.json()<br>            <br>    except Exception as e:<br>        logger.warning(&quot;[Socrata:%s] crime fetch failed: %s&quot;, city_key, e)<br>        return []<br><br>    incidents = []<br>    for row in raw:<br>        try:<br>            rlat = float(row.get(&quot;latitude&quot;) or row.get(&quot;lat&quot;) or 0)<br>            rlon = float(row.get(&quot;longitude&quot;) or row.get(&quot;lon&quot;) or row.get(&quot;lng&quot;) or 0)<br>        except (ValueError, TypeError):<br>            rlat, rlon = lat, lon<br><br>        raw_type = (<br>            row.get(&quot;primary_type&quot;) or<br>            row.get(&quot;ofns_desc&quot;) or<br>            row.get(&quot;crm_cd_desc&quot;) or<br>            row.get(&quot;crime_type&quot;) or<br>            row.get(&quot;nibrs_crime&quot;) or<br>            row.get(&quot;nibrs_crime_category&quot;) or<br>            row.get(&quot;offense_sub_category&quot;) or<br>            row.get(&quot;offense_parent_group&quot;) or<br>            row.get(&quot;offense&quot;) or<br>            row.get(&quot;offense_description&quot;) or<br>            row.get(&quot;incident_type_primary&quot;) or<br>            row.get(&quot;offense_type&quot;) or<br>            row.get(&quot;highest_nibrs_ucr_offense_description&quot;) or<br>            row.get(&quot;crime_category&quot;) or<br>            row.get(&quot;incident_category&quot;) or<br>            row.get(&quot;incident_description&quot;) or<br>            &quot;Unknown&quot;<br>        )<br>        if not isinstance(raw_type, str) or not raw_type.strip():<br>            raw_type = &quot;Unknown&quot;<br>        <br>        raw_type_upper = raw_type.upper()<br>        if any(x in raw_type_upper for x in [&quot;THEFT&quot;, &quot;BURGLARY&quot;, &quot;LARCENY&quot;, &quot;ROBBERY&quot;]):<br>            norm_type = &quot;Theft &amp; Burglary&quot;<br>        elif any(x in raw_type_upper for x in [&quot;ASSAULT&quot;, &quot;BATTERY&quot;, &quot;HOMICIDE&quot;, &quot;MURDER&quot;]):<br>            norm_type = &quot;Violent Crime&quot;<br>        elif any(x in raw_type_upper for x in [&quot;VEHICLE&quot;, &quot;AUTO&quot;, &quot;MOTOR&quot;]):<br>            norm_type = &quot;Vehicle Incident&quot;<br>        elif any(x in raw_type_upper for x in [&quot;DRUG&quot;, &quot;NARCOTIC&quot;]):<br>            norm_type = &quot;Drug Offense&quot;<br>        elif any(x in raw_type_upper for x in [&quot;FRAUD&quot;, &quot;DECEPTIVE&quot;, &quot;FORGERY&quot;]):<br>            norm_type = &quot;Fraud &amp; Deception&quot;<br>        elif &quot;TRESPASS&quot; in raw_type_upper:<br>            norm_type = &quot;Trespassing&quot;<br>        elif any(x in raw_type_upper for x in [&quot;WEAPON&quot;, &quot;FIREARM&quot;]):<br>            norm_type = &quot;Weapons Offense&quot;<br>        elif any(x in raw_type_upper for x in [&quot;DAMAGE&quot;, &quot;MISCHIEF&quot;, &quot;VANDALISM&quot;]):<br>            norm_type = &quot;Property Damage/Vandalism&quot;<br>        elif &quot;UNKNOWN&quot; not in raw_type_upper:<br>            norm_type = raw_type.title()<br>        else:<br>            norm_type = &quot;Other/Unknown&quot;<br><br>        incidents.append({<br>            &quot;type&quot;: norm_type,<br>            &quot;description&quot;: row.get(&quot;description&quot;) or row.get(&quot;law_cat_cd&quot;) or &quot;&quot;,<br>            &quot;date&quot;: row.get(&quot;date&quot;) or row.get(&quot;cmplnt_fr_dt&quot;) or &quot;&quot;,<br>            &quot;location&quot;: row.get(&quot;location_description&quot;) or row.get(&quot;boro_nm&quot;) or &quot;&quot;,<br>            &quot;block&quot;: row.get(&quot;block&quot;) or &quot;&quot;,<br>            &quot;lat&quot;: rlat or lat,<br>            &quot;lon&quot;: rlon or lon,<br>        })<br>    return incidents</pre><p>The <em>orchestrator.py</em> file handles the agents:</p><ul><li>First, the <strong>Geocode agent</strong> handles the user input.</li><li>After that, <strong>5 agents run in parallel</strong>: the Weather agent, the NWS Alerts Agent, the Crime Agent, the 311 Infra Agent and the Seismic agent.</li><li>After all the agents’ tasks are complete, a <strong>risk score</strong> for the city is calculated.</li><li>Finally, the <strong>AI Briefing</strong> (Gemini) agent aggregates all data and provides insights to the user.</li></ul><pre>def build_pipeline(address: str, fetcher, analyzer, board: ProgressBoard) -&gt; Tuple[BaseAgent, Callable[[Dict[str, Any]], dict]]:<br><br>    def geo_of(deps):<br>        return deps[&quot;geocode&quot;]<br><br>    def city_key_of(geo) -&gt; str:<br>        return fetcher._detect_city(geo.get(&quot;city&quot;, &quot;&quot;), geo.get(&quot;state&quot;, &quot;&quot;))<br><br>    geocode = NodeAgent(<br>        name=&quot;geocode&quot;,<br>        body=lambda d: fetcher._geocode(address),<br>        board=board,<br>    )<br>    weather = NodeAgent(<br>        name=&quot;weather&quot;,<br>        body=lambda d: fetcher._fetch_weather(geo_of(d)[&quot;lat&quot;], geo_of(d)[&quot;lon&quot;]),<br>        node_deps=(&quot;geocode&quot;,),<br>        on_error=lambda d: fetcher._weather_stub(geo_of(d)[&quot;lat&quot;], geo_of(d)[&quot;lon&quot;]),<br>        board=board,<br>    )<br>    nws = NodeAgent(<br>        name=&quot;nws&quot;,<br>        body=lambda d: fetcher._fetch_nws_alerts(geo_of(d)[&quot;lat&quot;], geo_of(d)[&quot;lon&quot;]),<br>        node_deps=(&quot;geocode&quot;,),<br>        on_error=lambda d: [],<br>        board=board,<br>    )<br>    crime = NodeAgent(<br>        name=&quot;crime&quot;,<br>        body=lambda d: fetcher._fetch_crime(<br>            geo_of(d)[&quot;lat&quot;], geo_of(d)[&quot;lon&quot;], city_key_of(geo_of(d)),<br>            geo_of(d).get(&quot;state&quot;, &quot;&quot;), geo_of(d),<br>        ),<br>        node_deps=(&quot;geocode&quot;,),<br>        on_error=lambda d: {&quot;incidents&quot;: [], &quot;total_count&quot;: 0, &quot;type_counts&quot;: {},<br>                            &quot;fbi_stats&quot;: {}, &quot;fbi_summary&quot;: {}, &quot;period_days&quot;: 30},<br>        board=board,<br>    )<br>    infra = NodeAgent(<br>        name=&quot;infra&quot;,<br>        body=lambda d: fetcher._fetch_311(geo_of(d)[&quot;lat&quot;], geo_of(d)[&quot;lon&quot;], city_key_of(geo_of(d))),<br>        node_deps=(&quot;geocode&quot;,),<br>        on_error=lambda d: {&quot;complaints&quot;: [], &quot;total&quot;: 0, &quot;categories&quot;: {}},<br>        board=board,<br>    )<br>    earthquakes = NodeAgent(<br>        name=&quot;earthquakes&quot;,<br>        body=lambda d: fetcher._fetch_earthquakes(geo_of(d)[&quot;lat&quot;], geo_of(d)[&quot;lon&quot;]),<br>        node_deps=(&quot;geocode&quot;,),<br>        on_error=lambda d: [],<br>        board=board,<br>    )<br>    risk = NodeAgent(<br>        name=&quot;risk&quot;,<br>        body=lambda d: fetcher._compute_risk(<br>            d[&quot;crime&quot;], _merge_alerts(d[&quot;weather&quot;], d[&quot;nws&quot;]),<br>            d[&quot;infra&quot;], d[&quot;earthquakes&quot;], city_key_of(d[&quot;geocode&quot;]),<br>        ),<br>        node_deps=(&quot;geocode&quot;, &quot;weather&quot;, &quot;nws&quot;, &quot;crime&quot;, &quot;infra&quot;, &quot;earthquakes&quot;),<br>        on_error=lambda d: {&quot;overall&quot;: 0, &quot;overall_label&quot;: &quot;Low&quot;},<br>        board=board,<br>    )<br>    ai_briefing = NodeAgent(<br>        name=&quot;ai_briefing&quot;,<br>        body=lambda d: analyzer.analyze(_core_data(d), address),<br>        node_deps=(&quot;geocode&quot;, &quot;weather&quot;, &quot;nws&quot;, &quot;crime&quot;, &quot;infra&quot;, &quot;earthquakes&quot;, &quot;risk&quot;),<br>        on_error=lambda d: &quot;AI analysis unavailable.&quot;,<br>        board=board,<br>    )<br><br>    nodes = [geocode, weather, nws, crime, infra, earthquakes, risk, ai_briefing]<br>    for node in nodes:<br>        board.init_row(node.name, node.node_deps)<br><br>    sources = ParallelAgent(name=&quot;sources&quot;, sub_agents=[weather, nws, crime, infra, earthquakes])<br>    root = SequentialAgent(name=&quot;pipeline&quot;, sub_agents=[geocode, sources, risk, ai_briefing])<br>    return root, assemble</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*8qJcQj1SDdXZLRQVwipcMw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kxtNLTTt2Xccl1REyWJgNg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FLKoS7JGzo3hconAoCv_Cg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PzCW5RKGZFBx4fOtMIUQxg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vgxae2Tv7SelgeKT0NlRgQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H0lMZtcqfbQFylpy5rxBpQ.png" /></figure><p>The Github repository for this solution is at:</p><p><a href="https://github.com/RubensZimbres/Neighborhood-Watch-App">GitHub - RubensZimbres/Neighborhood-Watch-App</a></p><h3>RUNNING LOCALLY</h3><p><em>.env</em> file:</p><pre>OPENWEATHER_API_KEY=your_openweathermap_key_here<br>FBI_CRIME_API_KEY=your_fbi_crime_api_key_here<br>SOCRATA_APP_TOKEN=your_socrata_app_token_here<br>GCP_PROJECT_NAME=your_gcp_project_id_here<br>GCP_LOCATION=us-central1</pre><p>How to run locally:</p><pre>python3 -m venv venv<br>source venv/bin/activate<br><br>gcloud auth application-default login<br><br>streamlit run app.py</pre><h3><strong>DEPLOYMENT IN GOOGLE CLOUD</strong></h3><pre>export PROJECT_ID=&quot;your-gcp-project-id&quot;<br>gcloud config set project &quot;$PROJECT_ID&quot;<br><br>gcloud services enable \<br>  run.googleapis.com \<br>  cloudbuild.googleapis.com \<br>  artifactregistry.googleapis.com \<br>  aiplatform.googleapis.com<br><br>gcloud run deploy neighborhoodalert \<br>  --source . \<br>  --region us-central1 \<br>  --allow-unauthenticated \<br>  --set-env-vars &quot;OPENWEATHER_API_KEY=your_key&quot; \<br>  --set-env-vars &quot;FBI_CRIME_API_KEY=your_key&quot; \<br>  --set-env-vars &quot;SOCRATA_APP_TOKEN=your_token&quot; \<br>  --set-env-vars &quot;GCP_PROJECT_NAME=$PROJECT_ID&quot; \<br>  --set-env-vars &quot;GCP_LOCATION=us-central1&quot; \<br>  --memory 512Mi \<br>  --cpu 1 \<br>  --timeout 300 \<br>  --max-instances 3<br><br>VAI=&quot;$(gcloud iam service-accounts list \<br>  --filter=&#39;displayName:Compute Engine&#39; \<br>  --format=&#39;value(email)&#39;)&quot;<br><br>gcloud projects add-iam-policy-binding &quot;$PROJECT_ID&quot; \<br>  --member &quot;serviceAccount:$VAI&quot; \<br>  --role &quot;roles/aiplatform.user&quot;</pre><p>Service URL: <a href="https://neighborhoodalert-xxxxx-uc.a.run.app/">https://neighborhoodalert-xxxxx-uc.a.run.app</a></p><h3>Acknowledgements</h3><p>✨ <em>Google ML Developer Programs and Google Developers Program supported this work by providing Google Cloud Credits</em>✨</p><p>🔗<a href="https://developers.google.com/machine-learning"> https://developers.google.com/machinelearning</a> 🔗</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ed95b004c3b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Healthcare Recommender with Keras, Two Towers and Google Cloud]]></title>
            <link>https://medium.com/@rubenszimbres/building-a-healthcare-recommender-with-keras-two-towers-and-google-cloud-4f535ad36dcd?source=rss-30f09d1741ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f535ad36dcd</guid>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[google-cloud]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Rubens Zimbres]]></dc:creator>
            <pubDate>Sat, 11 Apr 2026 14:49:27 GMT</pubDate>
            <atom:updated>2026-05-08T09:52:50.797Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*oQesVSwU48t_BaQbyiMDuw.gif" /></figure><p>Recommender systems are everywhere in entertainment and ecommerce: Netflix suggests your next show, Spotify your next song, Amazon your next purchase. The math behind those systems , taking a free text query and matching it against a big catalog , is exactly what you need to match ”I have shortness of breath, dizziness and palpitations” against a catalog of diseases.</p><p>That is the idea behind the <strong>Prescription Recommender App</strong> I want to walk through in this article. It takes <em>plain language symptoms</em> as input and returns the most likely diseases, along with suggested medications, diets and workouts. It is not a replacement for a doctor — it is a fast first step: ”here are the directions worth exploring.”</p><p>Under the hood it uses <strong>Keras Recommenders</strong> (the modern Kerasbased version of TensorFlow Recommenders), the <strong>TwoTower</strong> retrieval architecture, <strong>ScaNN</strong> for fast nearestneighbour search, a small Flask API served on Google Cloud Run, and it is designed to be deployed into Vertex AI Vector Search when it is time to scale. We will train this model in a TPU running on VS Code extension Colab. In this article I want to walk you through the whole thing, from raw CSV files to a production ready index.</p><p><a href="https://github.com/RubensZimbres/PrescriptionApp">GitHub - RubensZimbres/PrescriptionApp</a></p><h3>The Value: what a Two Tower model actually buys you</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*oASH4kb7YfwQDWmtgUDM_g.png" /><figcaption>The Two Towers Model</figcaption></figure><p>Before diving into the code, let me explain why this approach is a good fit for a domain like healthcare:</p><p>A traditional search engine matches keywords. If a user types ”my chest feels tight and I’m scared I’m going to die”, a keyword search will look for the literal words “chest”, “tight” and “die”. It will not understand that this is a textbook description of a panic attack. A Two Tower model, on the other hand, learns meaning. During training it places the phrase ”chest tightness, fear of dying” and the disease ”panic disorder” close to each other in a 768 dimensional space, even though they do not share a single word.</p><p>That is the first piece of value: <strong>semantic matching</strong>. The user does not need to know medical vocabulary. They describe what they feel, and the model handles the translation.</p><p>The second piece of value is <strong>speed</strong>. The Two Tower architecture is built specifically so that, after training, the candidate side (all the diseases) can be pre-computed once and stored in an index. At query time you only need to run the small query tower over the user’s text, and then do a nearest neighbor lookup.</p><p>In practice, a single query , from raw text to top 10 diseases, runs in tens of milliseconds on a single CPU. On Vertex AI Vector Search, with a properly sized endpoint, you can expect single digit millisecond latency even over millions of candidates. For a clinical support tool, that matters.</p><p>The third piece of value is <strong>scalability</strong>. The current dataset has a few hundred diseases, but the architecture does not care. Ten thousand diseases? Same code. A hundred thousand? Same code, just a bigger index. Because <strong>ScaNN</strong> (and <strong>Vertex AI Vector Search</strong>, which is built on the same algorithm) is an approximate nearest neighbor method, the search time grows roughly logarithmically with the size of the catalog instead of linearly. That is the difference between a demo and a product.</p><p>The fourth piece of value is <strong>separation of concerns</strong>. The model does one thing: it turns text into vectors that capture medical meaning. Everything else, the medications, the diets, the workouts, the explanations, live in regular dataframes that a domain expert can edit without touching any machine learning code. A medical reviewer can update a medication list in a CSV file, and the next deploy picks it up. No retraining needed. That is how you keep a healthcare product maintainable.</p><h3>The Data: Six CSV files and a lot of cleaning</h3><p>The dataset is a set of six CSV files assembled from public disease symptom datasets (<a href="https://www.kaggle.com/datasets/behzadhassan/sympscan-symptomps-to-disease">https://www.kaggle.com/datasets/behzadhassan/sympscan-symptomps-to-disease</a>). Here is what each file contains:</p><ul><li><strong>description.csv</strong>: One short paragraph describing each disease</li><li><strong>Diseases_and_Symptoms_dataset.csv</strong>: A wide binary matrix: one row per disease, one column per symptom (1 = present)</li><li><strong>diseases_symptoms.csv</strong>: A second, freetext symptom list per disease</li><li><strong>medications.csv</strong>: A list of typical medications per disease</li><li><strong>diets.csv:</strong> Recommended foods and dietary advice</li><li><strong>workout.csv:</strong> Recommended physical activities</li></ul><p>The first problem is that the files do not agree with each other. Some use ‘Disease’, others use ‘diseases’. Some have ‘Panic disorder’, others have ‘Panic Disorder’ with a capital D, and a few have trailing spaces. Anyone who has ever merged real world CSV files knows exactly what I mean.</p><p>The fix is boring but essential. Standardize the column names, lowercase everything, strip the whitespace:</p><pre>import pandas as pd<br><br>df_desc     = pd.read_csv(&#39;dataset/description.csv&#39;)<br>df_symp     = pd.read_csv(&#39;dataset/Diseases_and_Symptoms_dataset.csv&#39;)<br>df_meds     = pd.read_csv(&#39;dataset/medications.csv&#39;)<br>df_diets    = pd.read_csv(&#39;dataset/diets.csv&#39;)<br>df_workouts = pd.read_csv(&#39;dataset/workout.csv&#39;)<br>df_additional_symptoms = pd.read_csv(&#39;dataset/diseases_symptoms.csv&#39;)<br><br>for d in [df_desc, df_symp, df_meds, df_diets, df_workouts]:<br>    col = &#39;diseases&#39; if &#39;diseases&#39; in d.columns else &#39;Disease&#39;<br>    d.rename(columns={col: &#39;Disease&#39;}, inplace=True)<br>    d[&#39;Disease&#39;] = d[&#39;Disease&#39;].astype(str).str.lower().str.strip()</pre><p>The binary symptom matrix (<strong>Diseases_and_Symptoms_dataset.csv</strong>) needs a second transformation. Instead of keeping hundreds of 0/1 columns, we collapse them into a single text field listing the symptoms that are <em>active</em> for each disease, replacing underscores with spaces so the tokenizer can do its job:</p><pre>symptom_cols = df_symp.columns[1:]<br><br>def get_active_symptoms(row):<br>    active = [col for col in symptom_cols if row[col] == 1]<br>    return &quot;, &quot;.join(active).replace(&#39;_&#39;, &#39; &#39;)<br><br>df_symp[&#39;Symptoms_Text&#39;] = df_symp.apply(get_active_symptoms, axis=1)</pre><p>Then we merge the two symptom sources together (the binary derived one and the free text one), which gives each disease a rich, comma-separated symptom string. Finally we group everything by disease so that each row is unique:</p><pre>df_candidates = (<br>    df_symp[[&#39;Disease&#39;, &#39;Symptoms_Text&#39;]]<br>    .merge(df_desc, on=&#39;Disease&#39;, how=&#39;left&#39;)<br>    .merge(df_additional_symptoms, on=&#39;Disease&#39;, how=&#39;left&#39;)<br>    .fillna(&quot;None&quot;)<br>)<br><br>df_grouped = df_candidates.groupby(&#39;Disease&#39;).agg({<br>    &#39;Symptoms_Text&#39;: get_unique_items,<br>    &#39;Description&#39;:   get_unique_items,<br>}).reset_index()</pre><p>The result is a clean dataframe where each disease has one description (the <strong>candidate</strong> for our recommender) and one long symptom text (the <strong>query</strong> we will train against). The last step is to wrap it in a ‘<em>tf.data.Dataset</em>’, which is what <strong>Keras</strong> expects:</p><pre>import tensorflow as tf<br><br>dataset = tf.data.Dataset.from_tensor_slices(dict(df_unique_candidates)).map(lambda x: {<br>    &quot;Disease&quot;: x[&quot;Disease_Description&quot;],<br>    &quot;Description&quot;: x[&quot;Symptoms_Text&quot;],<br>})<br><br>shuffled = dataset.shuffle(df_grouped.shape[0], seed=42, reshuffle_each_iteration=False)<br>train    = shuffled.take(int(df_grouped.shape[0]  0.95))<br>test     = shuffled.skip(int(df_grouped.shape[0]  0.95))</pre><p>A 95/5 split is fine here because the goal of training is not really to generalize to unseen diseases — all diseases will be in the index at inference time — but to learn the mapping from symptom language to disease meaning.</p><h3>The Two Towers</h3><p>The Two Tower architecture is two small neural networks that process two different types of input and produce vectors in the same shared space.</p><p>The query tower turns a free text symptom description into a 768-dimensional vector. It is built from three Keras layers: a <strong>TextVectorization</strong> layer that splits the text into tokens and maps them to integer IDs, an <strong>Embedding</strong> layer that turns those IDs into vectors, and a <strong>GlobalAveragePooling1D</strong> layer that averages the token vectors into a single sentence vector.</p><pre>max_tokens = 5000<br>embedding_dimension = 768<br><br>vectorizer = tf.keras.layers.TextVectorization(<br>    max_tokens=max_tokens,<br>    output_mode=&quot;int&quot;,<br>    output_sequence_length=30,<br>)<br>vectorizer.adapt(train.map(lambda x: x[&quot;Description&quot;]))<br><br>description_model = tf.keras.Sequential([<br>    vectorizer,<br>    tf.keras.layers.Embedding(max_tokens, embedding_dimension, mask_zero=True),<br>    tf.keras.layers.GlobalAveragePooling1D(),<br>])</pre><p>Why 768 dimensions? It is the same width that BERT-base uses, and it gives a good balance between expressive power and index size. Smaller values (64, 128) train faster but cause more collisions between similar but distinct diseases. Larger values (1024) are overkill for a few hundred candidates.</p><p>The candidate tower is even simpler. Since each candidate is a full disease description (not a short ID), we use a <strong>StringLookup</strong> to turn it into an integer and an <strong>Embedding</strong> to give it a learnable vector:</p><pre>import numpy as np<br><br>unique_diseases = np.unique(df_unique_candidates[&quot;Disease_Description&quot;].values)<br><br>disease_model = tf.keras.Sequential([<br>    tf.keras.layers.StringLookup(vocabulary=unique_diseases, mask_token=None),<br>    tf.keras.layers.Embedding(len(unique_diseases) + 1, embedding_dimension),<br>])</pre><p>Notice that the two towers do not share any weights. They are free to learn completely different things: the <strong>Query Tower</strong> becomes a symptom-language specialist, the <strong>Candidate Tower</strong> becomes a disease-name specialist. The only thing that ties them together is the training objective, which is the job of <em>tensorflow recommenders</em>.</p><h3>Training: Teaching the Towers to Agree</h3><p>The <em>tfrs.tasks.Retrieval</em> task does the following: for each pair in a batch, it computes the dot product between the <em>query embedding</em> and every <em>candidate embedding</em> in that batch, and pushes the model to make the correct pairs score higher than the incorrect ones. This is the same in batch negative sampling trick that powers large scale search systems at companies like <strong>YouTube</strong> and <strong>Spotify</strong>.</p><pre>import tensorflow_recommenders as tfrs<br><br>class RecommenderModel(tfrs.Model):<br>    def __init__(self, query_model, candidate_model):<br>        super().__init__()<br>        self.query_model = query_model<br>        self.candidate_model = candidate_model<br>        self.task = tfrs.tasks.Retrieval(<br>            metrics=tfrs.metrics.FactorizedTopK(<br>                candidates=dataset.map(lambda x: x[&quot;Disease&quot;]).batch(128).map(self.candidate_model)<br>            )<br>        )<br><br>    def compute_loss(self, features, training=False):<br>        query_embeddings     = self.query_model(features[&quot;Description&quot;])<br>        candidate_embeddings = self.candidate_model(features[&quot;Disease&quot;])<br>        return self.task(query_embeddings, candidate_embeddings)<br><br>model = RecommenderModel(description_model, disease_model)<br>model.compile(optimizer=tf.keras.optimizers.Adagrad(learning_rate=0.1))</pre><p>The <em>FactorizedTopK</em> metric tells you, during training, what fraction of the time the correct disease appears in the Top-1, Top-5, Top-10 and Top-50 results. Watching Top-10 accuracy climb from near zero to over 90% during training is one of the small joys of working with this library.</p><p>The script runs for 18,000 epochs. That sounds like a lot, and for a large dataset it would be absurd, but here each “epoch” is a single pass over a few hundred diseases, so the whole training run still finishes in a reasonable time on a modest GPU. You can use the Colab extension for VS Code with GPU for faster training.</p><pre>cached_train = train.shuffle(df_unique_candidates.shape[0]).batch(1000).cache()<br>model.fit(cached_train, epochs=18000, verbose=0, callbacks=[tensorboard_callback, print_callback])</pre><h3>Building the ScaNN Index</h3><p>Once the model is trained, the <strong>query tower</strong> is the only part we need at inference time. The <strong>candidate tower</strong> did its job: we can run it over every disease once, get a matrix of embeddings, and build a search index on top of that matrix.</p><p>The simplest option is to compare the user’s query vector against every disease vector one by one, an exact <strong>dot-product search</strong>. For a few hundred diseases that would actually be fine. But this pipeline is designed to be ready for hundreds of thousands of candidates, which is where <strong>ScaNN </strong>(<strong>Scalable Nearest Neighbours</strong>, from Google Research) enters:</p><pre>import numpy as np<br>import scann<br><br>symptoms_tensor = tf.constant(df_grouped[&quot;Symptoms_Text&quot;].values)<br>embeddings      = description_model(symptoms_tensor).numpy().astype(np.float32)<br><br>norms              = np.linalg.norm(embeddings, axis=1, keepdims=True)<br>normalized_dataset = embeddings / (norms + 1e9)<br><br>searcher = scann.scann_ops_pybind.builder(<br>    normalized_dataset, 10, &quot;dot_product&quot;<br>).tree(<br>    num_leaves=100,<br>    num_leaves_to_search=100,<br>    training_sample_size=len(normalized_dataset),<br>).score_ah(<br>    2, anisotropic_quantization_threshold=0.2,<br>).reorder(100).build()</pre><p>First, we normalize every vector to unit length before building the index, what turns dot-product similarity into cosine similarity, which is what you almost always want for text embeddings.</p><p>Second, the <strong>tree().score_ah().reorder()</strong> chain is ScaNN’s classic recipe:</p><ul><li>partition the space into a tree</li><li>score each partition with asymmetric hashing (a clever form of quantization)</li><li>and then rerank the top candidates with exact dot products.</li></ul><p>The result is search quality that is almost indistinguishable from brute force, at a fraction of the cost.</p><p>Finally, we save everything we need for inference:</p><pre>import os<br><br>ARTIFACTS_DIR = &quot;saved_artifacts&quot;<br>os.makedirs(ARTIFACTS_DIR, exist_ok=True)<br><br>description_model.save(os.path.join(ARTIFACTS_DIR, &quot;description_model&quot;))<br>searcher.serialize(os.path.join(ARTIFACTS_DIR, &quot;scann_index&quot;))<br>df_grouped.to_pickle(os.path.join(ARTIFACTS_DIR, &quot;df_grouped.pkl&quot;))<br>df_meds.to_pickle(os.path.join(ARTIFACTS_DIR, &quot;df_meds.pkl&quot;))<br>df_diets.to_pickle(os.path.join(ARTIFACTS_DIR, &quot;df_diets.pkl&quot;))<br>df_workouts.to_pickle(os.path.join(ARTIFACTS_DIR, &quot;df_workouts.pkl&quot;))<br>np.save(os.path.join(ARTIFACTS_DIR, &quot;normalized_embeddings.npy&quot;), normalized_dataset)</pre><p>Notice what is being saved:</p><ul><li>the query tower (as a Keras SavedModel)</li><li>the ScaNN index</li><li>the three lookup dataframes</li><li>and the raw normalized embeddings</li></ul><p>That last file is important. If you ever want to <strong>rebuild the index</strong> with different parameters — or ship it to Vertex AI Vector Search — you do not need to retrain anything. You just reload the <em>.npy</em> file.</p><h3>Inference: the Flask endpoint on Cloud Run</h3><p>At inference time, the app is very simple. Load the artifacts once at startup, then each incoming request goes through three steps: Embed, Search, Enrich.</p><pre>import tensorflow as tf<br>import scann<br>import numpy as np<br>import pandas as pd<br><br>description_model = tf.keras.models.load_model(&quot;saved_artifacts/description_model&quot;)<br>searcher          = scann.scann_ops_pybind.load_searcher(&quot;saved_artifacts/scann_index&quot;)<br>df_grouped        = pd.read_pickle(&quot;saved_artifacts/df_grouped.pkl&quot;)<br><br>def embed_and_normalize(text: str) &gt; np.ndarray:<br>    vec  = description_model(tf.constant([text])).numpy().astype(np.float32)<br>    norm = np.linalg.norm(vec, axis=1, keepdims=True)<br>    return vec / (norm + 1e9)<br><br>@app.route(&quot;/search&quot;, methods=[&quot;POST&quot;])<br>def search():<br>    query              = request.get_json()[&quot;query&quot;].strip()<br>    query_vec          = embed_and_normalize(query)<br>    neighbors, scores  = searcher.search(query_vec[0], final_num_neighbors=5)<br><br>    results = []<br>    for rank, (idx, score) in enumerate(zip(neighbors, scores), start=1):<br>        row     = df_grouped.iloc[int(idx)]<br>        disease = row[&quot;Disease&quot;].title()<br>        meds, diets, workouts = lookup_supplementary(row[&quot;Disease&quot;])<br>        results.append({<br>            &quot;rank&quot;: rank, &quot;disease&quot;: disease, &quot;score&quot;: round(float(score), 4),<br>            &quot;medications&quot;: meds, &quot;diets&quot;: diets, &quot;workouts&quot;: workouts,<br>        })<br>    return jsonify(results)</pre><p>The entire production inference path: embed one short piece of text, do a vector search, look up three dataframes, runs in tens of milliseconds on a single vCPU. Wrap it in a Dockerfile based on tensorflow/tensorflow:2.13.0, push it to <strong>Google Cloud Run</strong>, and you have an autoscaling endpoint without ever touching a Kubernetes cluster. The project’s actual Dockerfile does exactly that, using gunicorn as the WSGI server and exposing port 8080, which is what Cloud Run expects.</p><p>Cloud Run is the current deployment target for this project, and it is a great fit for a small to medium recommender: the container holds the query tower, the ScaNN index and the enrichment dataframes, and may scale to zero (beware of the cold start) when nobody is using it. For many use cases, that is all you will ever need.</p><h3>The next step: Vertex AI Vector Search</h3><p>Cloud Run works beautifully for a demo or a low traffic production app, but the moment the catalog grows, more diseases, multiple languages, millions of queries a day , the ScaNN index must live in a dedicated vector database, and on Google Cloud that database is Vertex AI Vector Search (formerly Matching Engine).</p><p>What is beautiful about this migration is that Vertex AI Vector Search is built on the same ScaNN algorithm we are already using locally. Moving from one to the other is not a rewrite, is a deployment change. The embeddings saved to <em>normalized_embeddings.npy</em> are the input to Vertex AI Vector Search. Here is a sketch of the migration:</p><pre>from google.cloud import aiplatform<br>import json<br><br>aiplatform.init(project=&quot;my-project&quot;, location=&quot;us-central1&quot;)<br><br># 1. Upload the embeddings as JSONL to Google Cloud Storage<br>with open(&quot;embeddings.jsonl&quot;, &quot;w&quot;) as f:<br>    for i, vec in enumerate(normalized_dataset):<br>        f.write(json.dumps({&quot;id&quot;: str(i), &quot;embedding&quot;: vec.tolist()}) + &quot;\n&quot;)<br><br># 2. Create the index<br>index = aiplatform.MatchingEngineIndex.create_tree_ah_index(<br>    display_name=&quot;prescriptionrecommender&quot;,<br>    contents_delta_uri=&quot;gs://mybucket/embeddings/&quot;,<br>    dimensions=768,<br>    approximate_neighbors_count=10,<br>    distance_measure_type=&quot;DOT_PRODUCT_DISTANCE&quot;,<br>)<br><br># 3. Deploy it to an endpoint<br>endpoint = aiplatform.MatchingEngineIndexEndpoint.create(<br>    display_name=&quot;prescriptionendpoint&quot;,<br>    public_endpoint_enabled=True,<br>)<br>endpoint.deploy_index(index=index, deployed_index_id=&quot;prescription_v1&quot;)</pre><p>Once the index is deployed, the Flask app no longer needs to load ScaNN at all. It just keeps the <strong>Query Tower</strong> (a few megabytes) and calls the Vertex AI endpoint:</p><pre>response = endpoint.find_neighbors(<br>    deployed_index_id=&quot;prescription_v1&quot;,<br>    queries=[query_vec[0].tolist()],<br>    num_neighbors=5,<br>)</pre><p>The index is now separately scalable and you can add nodes without redeploying the app. It is updatable in place: adding a new disease is a matter of appending one embedding, not rebuilding a container. It is multiregion if you need it to be. And because the query tower runs on CPU and is extremely small, you can keep serving it from Cloud Run while the heavy vector lookup happens on a managed, dedicated service. We have stateless inference code on Cloud Run and stateful vector index on Vertex AI Vector Search.</p><p>You can start with a single Python script and a Flask app on your laptop, and the same artifacts, the same embeddings, the same 768 dimensional space, the same distance metric, carry you all the way to a production, multiregion, millisecond latency vector search service with just a redeploy.</p><h3>Closing Remarks</h3><p>This project combines three things that are usually kept in separate boxes:</p><ul><li>classic machine learning (the TwoTower architecture has been around for years, I started using it in 2022)</li><li>modern Keras (the code above runs on Keras and TensorFlow 2.13)</li><li>and Google Cloud’s managed infrastructure, Cloud Run and Vertex AI Vector Search.</li></ul><p>If you are a developer thinking about how to bring semantic search into a domain you care about , healthcare, legal, education, customer support, the gap between ‘interesting tutorial’ and ‘something you could actually put in front of users’ is much smaller than it looks. You need a clean dataset (essential), two small Keras models, a ScaNN index, and a couple of hundred lines of Python.</p><p>Run the Python script and you will have your own symptom to disease recommender running locally in minutes. Build the Docker image, push it to Cloud Run, and it is online. When the catalog grows, point the same embeddings at Vertex AI Vector Search, and it is ready for scale.</p><p>That is the whole idea: start simple, stay in the same ecosystem, and let the platform grow with the product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*mFPaSeGg7_kCBPeJNus1NA.png" /></figure><h3>Acknowledgements</h3><p>✨ <em>Google ML Developer Programs and Google Developers Program supported this work by providing Google Cloud Credits (and awesome tutorials for the Google Developer Experts)</em>✨</p><p>🔗<a href="https://developers.google.com/machine-learning"> https://developers.google.com/machinelearning</a> 🔗</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f535ad36dcd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Proof of Concept to Production: Building an Enterprise-Grade Platform for AI Systems]]></title>
            <link>https://medium.com/@rubenszimbres/from-proof-of-concept-to-production-building-an-enterprise-grade-platform-for-ai-systems-1fb9ebac78b9?source=rss-30f09d1741ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fb9ebac78b9</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Rubens Zimbres]]></dc:creator>
            <pubDate>Sun, 15 Feb 2026 21:11:50 GMT</pubDate>
            <atom:updated>2026-02-17T08:19:33.010Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*wj33PJTZS0KPoW-LVvCqLQ.png" /></figure><h3>Introduction</h3><p>The transition from a working AI prototype to a production-ready system represents one of the most challenging journeys in modern software development. While building a chatbot that can answer questions is relatively straightforward, deploying an AI agent system that can serve thousands or even millions of users securely, reliably, and cost-effectively requires careful architectural decisions and enterprise-grade infrastructure.</p><p>This article presents a comprehensive reference architecture for deploying multi-agents AI systems on Google Cloud Platform, designed with the explicit goal of allowing developers to <em>plug</em> <em>any AI agent system</em> into a robust infrastructure.</p><p>The architecture presented here contains several critical best practices that make it suitable for enterprise deployment.</p><ul><li>First, it implements a strict <strong>separation</strong> of concerns through a decoupled <em>frontend</em> and <em>backend</em> architecture, allowing teams to independently develop, test, and deploy each component.</li><li>Second, it follows a <strong>security-first design</strong> philosophy with <em>defense in depth</em>, implementing protections at every layer from the network edge to the application core.</li><li>Third, it embraces <strong>infrastructure as code</strong> through modular <em>Terraform</em> configurations, ensuring reproducible deployments and facilitating disaster recovery.</li><li>Fourth, the system is built for <strong>observability</strong> with comprehensive distributed tracing, structured logging, and health monitoring throughout.</li><li>Finally, the architecture is designed for <strong>cost efficiency</strong>, using serverless compute, intelligent caching, and tiered storage to minimize operational expenses while maintaining high availability.</li></ul><p>What makes this infrastructure particularly valuable is its <em>agent-agnostic design</em>. The platform provides all the surrounding capabilities that any AI agent system needs: authentication, payment processing, secure data storage, content delivery, rate limiting, and observability. Developers can focus on building their specific AI capabilities while the infrastructure handles the undifferentiated heavy lifting of enterprise deployment.</p><p>In this article, I present the basic structure of the project. For a more detailed description and code, access the <a href="https://github.com/RubensZimbres/my-ai-platform">Github repository of the project</a>:</p><p><a href="https://github.com/RubensZimbres/Enterprise-Grade-Infra-for-AI-Agents">GitHub - RubensZimbres/Enterprise-Grade-Infra-for-AI-Agents: Terraform Deployment of AI Agents Solution in Google Cloud</a></p><p>⭐🇾 the repo if you like it. Contributions are welcome!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*aaDxIx1zjL4GZALeUd0ykA.png" /></figure><h3>Architecture Overview</h3><p>The platform consists of three primary layers: a Next.js <strong>frontend</strong> serving as the user interface and secure proxy, a FastAPI <strong>backend</strong> orchestrating the AI capabilities, and a comprehensive infrastructure layer managed through <strong>Terraform</strong> modules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z0bpWEzB3xoEwEXdJaOP_g.jpeg" /><figcaption>Google Cloud Architecture</figcaption></figure><p>The <strong>frontend layer</strong> is built with <em>React 18</em> and <em>Next.js</em>, utilizing the modern App Router pattern. It serves as more than just a user interface; it acts as a secure proxy that handles all communication with backend services. Authentication is managed through <strong>Firebase</strong>, providing seamless integration with <strong>Google Identity services</strong> while supporting millions of consumer-scale users. The frontend implements <strong>circuit breaker</strong> patterns using the <em>opossum</em> library, ensuring that temporary backend failures do not cascade into system-wide outages. To eliminate cold-start latency, the service maintains a minimum of one warm Cloud Run instance at all times.</p><p>The <strong>backend layer</strong> is a <em>FastAPI</em> application designed for high concurrency and resilience. It orchestrates Retrieval-Augmented Generation using LangGraph and Vertex AI, connecting to <em>Cloud SQL</em> for PostgreSQL with the <em>pgvector</em> extension for semantic search capabilities. The backend is configured for internal-only ingress traffic, ensuring it remains unreachable from the public internet and only accessible through the authenticated frontend proxy. Full <strong>OpenTelemetry</strong> instrumentation provides distributed tracing capabilities exported to <em>Google Cloud Trace</em>, enabling detailed debugging and performance analysis in production environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*iANMDzNl2svdYhKwwYSuww.png" /></figure><h3>The Frontend Layer</h3><p>The frontend architecture centers around three core components that manage the user experience:</p><ul><li>The <strong>AuthProvider</strong> component serves as the the authentication system, using <em>Firebase Authentication</em> to manage user state and protect routes from unauthorized access.</li><li>The <strong>ChatInterface</strong> component provides the main interaction surface, delivering a real-time streaming chat experience tightly integrated with the backend API. It handles authentication errors and payment-related issues gracefully, redirecting users to appropriate pages when necessary.</li><li>The <strong>PaymentClient</strong> component delivers a seamless checkout experience using Stripe Embedded Checkout, guiding users through the payment process with comprehensive error handling.</li><li>The <strong>routing structure</strong> implements a clear user journey from landing page through authentication and payment to the main chat interface. Server-side API routes handle critical operations including the chat proxy, payment status verification, and checkout session creation.</li></ul><p>The chat API route implements a <strong>circuit breaker</strong> to prevent cascading failures while using <strong>OIDC tokens</strong> for secure service-to-service authentication. It streams responses from the backend to provide real-time chat capabilities, forwarding user authentication tokens to the backend for authorization decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*Afl1fyJE5QFEbmjnZWpgEw.png" /></figure><h3>The Backend Layer</h3><p>The backend exposes four primary endpoints:</p><ul><li>A <strong>health check</strong> for infrastructure monitoring,</li><li>A <strong>webhook</strong> endpoint for Stripe event processing, and</li><li><strong>Two chat endpoints</strong> supporting both standard request-response and streaming communication patterns.</li></ul><p>Security is implemented at multiple levels. <strong>Rate limiting</strong> restricts requests to ten per minute per IP address to prevent abuse. <strong>Input validation</strong> through Pydantic models enforces strict message size limits to prevent denial-of-service attacks and the <strong>authentication dependency</strong> ensures all chat requests come from verified users, while <strong>session IDs</strong> are scoped to authenticated users to prevent insecure direct object reference attacks.</p><p>The data layer uses <strong>PostgreSQL</strong> as the primary database, storing user information including subscription status and <strong>Stripe</strong> customer identifiers. All database operations are encapsulated in dedicated modules for maintainability and testability. The Stripe integration is tight and bidirectional: webhooks listen for payment events and automatically update user subscription status in the database, while the authentication middleware verifies subscription status for every protected request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*EAcetbxedKSB-6DE0hYmAQ.png" /></figure><h3>AI Engine and Knowledge Core</h3><p>The AI capabilities are built around a <em>Retrieval-Augmented Generation</em> pipeline that balances high-performance search with secure session management. The system implements two distinct memory systems: <strong>short-term memory</strong> for maintaining conversation context and <strong>long-term memory</strong> for the knowledge base.</p><ul><li><strong>Short-term memory</strong> utilizes <em>Google Cloud Firestore</em> in Native Mode for low-latency persistence of chat history. The implementation leverages <em>FirestoreChatMessageHistory</em> within the LangGraph framework, with every session cryptographically scoped to the authenticated user identity. This ensures strict multi-tenancy where users cannot access or leak into another user’s conversation history. The system automatically retrieves the last N messages and injects them into the RAG prompt, enabling multi-turn, context-aware dialogue.</li><li><strong>Long-term memory</strong> is powered by PostgreSQL 16 with the <strong><em>pgvector</em></strong> extension, enabling semantic similarity search using <em>Vertex AI Embeddings</em>. For every query, the engine retrieves the top five most relevant document chunks to provide grounded context to the language model. A <strong>semantic cache</strong> backed by Redis provides an additional optimization layer: if a user asks a question semantically similar to a previously cached query, the system returns the cached response instantly, bypassing the language model entirely to save cost and reduce latency.</li></ul><p>The document ingestion pipeline transforms raw data into AI-ready vectors through a specialized process, whose ingestion process is triggered automatically through <em>Cloud Function</em>s when new documents are uploaded to the storage bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*hsjbhw7WqyJorNzExqTSnA.png" /></figure><h3>Security and Resilience</h3><p>The platform implements a multi-layered security strategy addressing both traditional web application vulnerabilities and AI-specific threats. Protection against SQL injection operates at two levels: <strong>Cloud Armor</strong> is configured with pre-defined <strong>WAF</strong> (Web Application Firewall) rules to filter malicious SQL patterns at the network edge, while the backend uses <strong>asyncpg</strong> with strictly parameterized queries to ensure user input is never executed as raw SQL. (OWASP Top 10)</p><p>Similarly, cross-site scripting (XSS) protection combines Cloud Armor WAF rules with Next.js’s automatic content sanitization and the backend’s structured JSON responses. <em>Broken access control</em> and <em>insecure direct object reference</em> vulnerabilities are addressed through a verified identity system. The frontend captures user identity from <strong>Firebase Authentication</strong> tokens and propagates them to the backend for verification.</p><p>Chat histories are cryptographically scoped to authenticated user identities, preventing one user from accessing another’s private conversation history. DDoS (Distributed Denial of Service) and resource abuse protection operates at multiple layers: <strong>Cloud Armor </strong>implements a global rate-limiting policy of 500 requests per minute per IP address with rate-based banning for volumetric attacks, while the backend uses <em>slowapi</em> to enforce granular rate limiting specifically for expensive language model operations.</p><p>The architecture addresses AI-specific security concerns including <em>prompt injection</em> and <em>sensitive data leakage</em>. The RAG prompt template uses strict structural delimiters and prioritized system instructions to ensure the model adheres to its enterprise role and ignores adversarial overrides in documents or user queries. (OWASP Top 10 for LLM and MAESTRO Framework).</p><p>A sandwich defense using XML tagging provides explicit instructions to ignore external commands found within retrieved context. <strong>Google Cloud DLP</strong> is integrated into the core pipeline with a regex fast-path that intelligently filters expensive API calls for clean content, invoking the <em>Data Loss Prevention</em> service only when potential PII patterns are detected. The knowledge base itself is stored in a private Cloud SQL instance reachable only via Serverless VPC Access connector, ensuring the AI’s brain is never exposed to the public internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*-PQaFixnQawAhkKipZlgog.png" /></figure><h3>Infrastructure as Code</h3><p>The entire infrastructure is defined through modular <em>Terraform</em> configurations organized into logical components, following cybersecurity best practices:</p><ul><li>The <strong>network module</strong> provisions a custom VPC with private subnets and Cloud NAT gateway, ensuring services are not exposed directly to the public internet.</li><li>The <strong>compute module</strong> deploys decoupled frontend and backend services on Cloud Run with granular IAM policies.</li><li>The <strong>database module</strong> provisions Cloud SQL for PostgreSQL with Firestore for chat history storage. A dedicated <strong>Redis module</strong> provides Memorystore for semantic caching.</li><li>The <strong>ingress module</strong> configures a global external HTTPS load balancer with Cloud Armor providing WAF rules for SQL injection, cross-site scripting, and rate limiting.</li><li>The <strong>function module</strong> sets up Cloud Functions for event-driven PDF ingestion.</li><li>Additional modules handle <strong>CI/CD</strong> pipelines, <strong>storage</strong> buckets with lifecycle policies, and <strong>billing</strong> monitoring with alert policies and notification channels.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*7M_ZboGT5OrHxMn_bxiU5g.png" /><figcaption>Terraform folder</figcaption></figure><p>The infrastructure follows a security-first design philosophy.</p><ul><li>The <strong>database</strong> has no public IP and uses <strong>IAM</strong> authentication. All sensitive information is stored in <strong>Google Secret Manager</strong>.</li><li>The <strong>load balancer</strong> provides a single entry point with Cloud CDN improving performance by caching static assets closer to users.</li><li><strong>Health checks</strong> with startup and liveness probes ensure reliability.</li><li>The <strong>CI/CD pipeline</strong> automates build and deployment processes, maintaining a Zero-Trust permission model where service accounts have only the specific roles they require.</li></ul><p>You just need to run:</p><pre>terraform init<br>terraform plan<br>terraform apply</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*hFXPJufjlPEzd10S_kwSxQ.png" /></figure><h3>Performance and Scaling</h3><p>The architecture is optimized for both performance and cost efficiency.</p><ul><li>The <strong>backend</strong> is built on <em>FastAPI</em> with <em>asyncpg</em> for non-blocking database connections, allowing a single instance to handle thousands of concurrent requests with minimal resource usage.</li><li><strong>Server-Sent Events</strong> enable real-time token streaming from the language model directly to the frontend, providing sub-second time-to-first-token for a highly responsive user experience. Expensive operations like <strong>PII</strong> (Personal Identifiable Information) de-identification are offloaded to asynchronous background threads to prevent blocking the main request-response cycle.</li><li><strong>Cost control</strong> measures include using the <strong>Gemini 3 Flash</strong> model for a significant reduction in token costs compared to larger models, implementing regex-based pre-checks for PII to intelligently bypass expensive <em>DLP API</em> calls, and enabling <em>Cloud CDN</em> for global caching of static assets.</li><li><strong>Object Lifecycle Management</strong> on storage buckets automatically transitions files to Nearline storage after seven days, Archive storage after thirty days, and deletes them after ninety days, providing disaster recovery capabilities without indefinite storage costs.</li></ul><p>The current infrastructure is benchmarked to handle approximately 2,500 users per hour. For scaling to a million users per hour, I recommend offloading vector search to <strong>Vertex AI Vector Search</strong>, a fully managed service designed to handle billions of vectors and thousands of queries per second with sub-10-millisecond latency. In this configuration, PostgreSQL handles only chat history and user metadata while the specialized vector engine handles the high-throughput similarity search load.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*E7lBZS_NWnE7ChSE9RA7jw.png" /></figure><h3>Payment and Subscription System</h3><p>The platform enforces a strict workflow where users must log in, then pay, before accessing the chat functionality. The PostgreSQL database serves as the single source of truth for user subscription status. Stripe integration is implemented through secure webhooks that listen for checkout completion and invoice payment success events, automatically updating user status when payments succeed.</p><p>The backend middleware checks subscription status for every request, while the frontend intercepts these errors and redirects users to the subscription/payment page. The database schema links user emails to Firebase Identity, tracks active subscription status, and maintains Stripe customer identifiers for seamless payment management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BaKQLF0ra3e7fLCri00lLA.png" /></figure><h3>Disaster Recovery</h3><p>The infrastructure includes <em>disaster recovery</em> capabilities. <strong>Cloud SQL</strong> is configured with automated backups retained for seven days, point-in-time recovery allowing restoration to any second within the retention window, and deletion protection to prevent accidental instance removal.</p><p>For data corruption scenarios, the database can be cloned to a specific point in time before the corruption occurred, allowing verification before switching traffic to the restored instance. For complete instance loss, restoration from the last successful nightly backup is straightforward through the <em>gcloud</em> command-line interface.</p><p><strong>Firestore</strong> is configured with daily backup schedules retained for seven days. Since Firestore does not support in-place restores, recovery involves restoring to a new database ID and updating the backend configuration to point to the restored database. Post-recovery procedures include verifying backend connectivity, running application-level smoke tests, and ensuring backup schedules are re-applied through Terraform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*4uDTCpr1CQ5tW8KpsxKT0A.png" /></figure><h3>Cost Considerations</h3><p>The architecture is designed for cost efficiency while maintaining enterprise capabilities.</p><ul><li>Cloud Run compute costs are $25 a month.</li><li>Cloud SQL database costs approximately $34 per month,</li><li>Memorystore for Redis at approximately $36 per month,</li><li>Cloud NAT gateway at approximately $33 per month, and</li><li>Load balancer with Cloud Armor at approximately $33 per month.</li></ul><p>This brings the baseline monthly cost to approximately <strong>$161</strong> for a production-ready enterprise platform that handles 2,500 users per hour.</p><p>⚠️ Note that you have to be careful to not deploy the Enterprise version of Cloud Armor in Terraform, otherwise it will cost you $3,000.</p><p>For <em>development or staging environments</em>, costs can be reduced to under <strong>$50 per month</strong> by scaling Cloud Run instances to zero, removing the Redis module and using local containers, eliminating the NAT gateway if static outbound IP addresses are not required, and potentially downgrading or replacing Cloud SQL with Firestore for simpler use cases.</p><p><strong>Variable costs</strong> depend on usage and include storage fees, data transfer, LLM API calls, and DLP processing.</p><h3>Conclusion</h3><p>This reference architecture demonstrates that transitioning from AI proof of concept to production deployment requires careful attention to security, scalability, observability, and cost management.</p><p>By implementing infrastructure as code, following cloud-native best practices, and building defense in depth, teams can create a foundation that supports any AI agent system while handling the complexities of enterprise deployment.</p><p>The modular design allows components to be upgraded or replaced as requirements evolve, while the comprehensive security measures ensure compliance with enterprise standards. Whether deploying a simple RAG-based chatbot or a complex multi-agent system, this infrastructure provides the robust foundation needed for production success.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*s4PC9GUIjJmDD2FmEoVEqg.png" /></figure><h3><strong>Acknowledgements</strong></h3><p>✨ Special thanks for Natalie Godec (<a href="https://medium.com/@ouvessvit">https://medium.com/@ouvessvit</a>), my fellow GDE for reviewing the Terraform deployment.</p><p>✨ Google ML Developer Programs and Google Developers Program supported this work by providing Google Cloud Credits.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fb9ebac78b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developing a Variational Autoencoder in JAX using Antigravity]]></title>
            <link>https://medium.com/@rubenszimbres/developing-a-variational-autoencoder-in-jax-using-antigravity-83a42c444033?source=rss-30f09d1741ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/83a42c444033</guid>
            <category><![CDATA[google-antigravity]]></category>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[jax]]></category>
            <dc:creator><![CDATA[Rubens Zimbres]]></dc:creator>
            <pubDate>Tue, 25 Nov 2025 14:39:23 GMT</pubDate>
            <atom:updated>2025-11-25T14:39:23.064Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*qvCxL9vWQXecoWorqjSsqQ.png" /></figure><p>Lately I became a contributor for the <strong>Bonsai</strong> project, where I translated <strong>EfficientNet</strong>, <strong>U-Net</strong> and a <strong>Variational Autoencoder (VAE)</strong> into JAX code.</p><p><strong>JAX </strong>is a super fast NumPy-based ML framework with automatic differentiation, providing high-performance and scalability essential for modern machine learning research. Its focus on functional programming and composability aligns perfectly with the Bonsai project’s mission to offer simple, hackable, and concise implementations of popular models. This approach not only lowers the barrier to entry for JAX but also promotes academic innovation. Gemini is trained on JAX.</p><p>Here I will use <strong>Antigravity IDE</strong>, to develop a <strong>VAE</strong> and make inference. We will leverage the efficiency and speed of <strong>JAX</strong>, combined with the convenience of a modern cloud development environment, to walk through the entire development process of this generative model.</p><p><a href="https://github.com/jax-ml/bonsai">GitHub - jax-ml/bonsai: Minimal, lightweight JAX implementations of popular models.</a></p><p>The implementation follows this paper:</p><p><a href="https://arxiv.org/abs/1312.6114">https://arxiv.org/abs/1312.6114</a></p><p>We start with two files: <em>modeling.py </em>and <em>params.py. </em>These two files, define the structure and initialization logic for the <strong>Variational Autoencoder (VAE)</strong> model within the JAX Bonsai project using the <strong>Flax NNX</strong> module system.</p><p>Flax NNX (Neural Networks JAX) is a new, simplified API within the Flax ecosystem designed to make creating, debugging, and analyzing neural networks in JAX easier and more intuitive. aims to bridge the gap between JAX’s functional programming core and the object-oriented style familiar to PyTorch or Keras users.</p><p>In essence, Flax NNX allows researchers to leverage JAX’s performance (automatic differentiation, JIT compilation, and hardware acceleration) while enjoying a more intuitive and flexible object-oriented experience.</p><h3>The VAE Architecture</h3><h4><em>modeling.py</em></h4><p>This file contains the core definitions for the VAE model components and the forward pass logic.</p><p><strong>ModelCfg (Data Structure):</strong> This dataclass holds the <strong>hyperparameters</strong> for the VAE, such as the <em>input_dim</em> (e.g., 784 for a flattened 28x28 image), <em>hidden_dims</em> (the size of the intermediate layers), and the <em>latent_dim</em> (the dimensionality of the compressed latent space, <strong><em>z</em></strong>).</p><p><strong>Encoder (NNX Module):</strong> This module takes the input data (<strong><em>x</em></strong>) and maps it to the parameters of the latent distribution.</p><ul><li>It uses a sequence of <em>fully-connected (Linear)</em> layers with the ReLU activation function.</li><li>The output layer is split into two separate linear layers, <strong><em>fc_mu</em></strong> and <strong><em>fc_logvar</em></strong>, which output the <em>mean</em> (<strong><em>mu</em></strong>) and <em>log-variance</em> (<strong><em>log\sigma²</em></strong> or <strong><em>logvar</em></strong>) of the latent Gaussian distribution, respectively.</li></ul><p><strong>Decoder (NNX Module): </strong>This module takes a sample from the latent space (<strong><em>z</em></strong>) and reconstructs the input data. It generally uses a <em>mirrored</em> architecture of the encoder (reversed <em>hidden_dims</em>). The final output, <em>fc_out</em>, produces the reconstruction logits, which are used to calculate the reconstruction loss (e.g., Binary Cross-Entropy for images like MNIST).</p><p><strong>VAE (NNX Module): </strong>This is the main class that combines the Encoder and Decoder.</p><ul><li><strong>reparameterize method:</strong> This is the crucial step in VAEs. It implements the reparameterization trick to sample the latent vector <strong><em>z</em></strong> from <strong>N(mu, sigma²)</strong> using a random noise vector ϵ ∼N(0, I):</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*qQeeDptclOGC0Cn7UR_Xug.png" /></figure><p><strong>__call__ method:</strong> This defines the VAE’s forward pass: input <strong><em>x</em></strong> goes through the <strong>Encoder</strong>; the latent sample <strong><em>z</em></strong> is then passed to the <strong>Decoder</strong> for reconstruction.</p><pre>import dataclasses<br>from typing import Sequence<br><br>import jax<br>import jax.numpy as jnp<br>from flax import nnx<br><br>@dataclasses.dataclass(frozen=True)<br>class ModelCfg:<br>    &quot;&quot;&quot;Configuration for the Variational Autoencoder (VAE) model.&quot;&quot;&quot;<br>    input_dim: int = 784  # 28*28 for MNIST<br>    hidden_dims: Sequence[int] = (512, 256)<br>    latent_dim: int = 20<br><br>class Encoder(nnx.Module):<br>    &quot;&quot;&quot;Encodes the input into latent space parameters (mu and logvar).&quot;&quot;&quot;<br>    def __init__(self, cfg: ModelCfg, *, rngs: nnx.Rngs):<br>        self.hidden_layers = [<br>            nnx.Linear(in_features, out_features, rngs=rngs)<br>            for in_features, out_features in zip(<br>                [cfg.input_dim] + list(cfg.hidden_dims), cfg.hidden_dims<br>            )<br>        ]<br>        self.fc_mu = nnx.Linear(cfg.hidden_dims[-1], cfg.latent_dim, rngs=rngs)<br>        self.fc_logvar = nnx.Linear(cfg.hidden_dims[-1], cfg.latent_dim, rngs=rngs)<br><br>    def __call__(self, x: jax.Array) -&gt; tuple[jax.Array, jax.Array]:<br>        x = x.reshape((x.shape[0], -1))<br>        for layer in self.hidden_layers:<br>            x = nnx.relu(layer(x))<br>        <br>        mu = self.fc_mu(x)<br>        logvar = self.fc_logvar(x)<br>        return mu, logvar<br><br>class Decoder(nnx.Module):<br>    &quot;&quot;&quot;Decodes the latent vector back into the original input space.&quot;&quot;&quot;<br>    def __init__(self, cfg: ModelCfg, *, rngs: nnx.Rngs):<br>        # Mirrored architecture of the encoder<br>        dims = [cfg.latent_dim] + list(reversed(cfg.hidden_dims))<br>        self.hidden_layers = [<br>            nnx.Linear(in_features, out_features, rngs=rngs)<br>            for in_features, out_features in zip(dims, dims[1:])<br>        ]<br>        self.fc_out = nnx.Linear(dims[-1], cfg.input_dim, rngs=rngs)<br><br>    def __call__(self, z: jax.Array) -&gt; jax.Array:<br>        for layer in self.hidden_layers:<br>            z = nnx.relu(layer(z))<br>        <br>        reconstruction_logits = self.fc_out(z)<br>        return reconstruction_logits<br><br>class VAE(nnx.Module):<br>    &quot;&quot;&quot;Full Variational Autoencoder model.&quot;&quot;&quot;<br>    def __init__(self, cfg: ModelCfg, *, rngs: nnx.Rngs):<br>        self.cfg = cfg<br>        self.encoder = Encoder(cfg, rngs=rngs)<br>        self.decoder = Decoder(cfg, rngs=rngs)<br><br>    def reparameterize(self, mu: jax.Array, logvar: jax.Array, key: jax.Array) -&gt; jax.Array:<br>        &quot;&quot;&quot;Performs the reparameterization trick to sample from the latent space.&quot;&quot;&quot;<br>        std = jnp.exp(0.5 * logvar)<br>        epsilon = jax.random.normal(key, std.shape)<br>        return mu + epsilon * std<br><br>    def __call__(self, x: jax.Array, sample_key: jax.Array) -&gt; tuple[jax.Array, jax.Array, jax.Array]:<br>        &quot;&quot;&quot;Defines the forward pass of the VAE.&quot;&quot;&quot;<br>        mu, logvar = self.encoder(x)<br>        z = self.reparameterize(mu, logvar, sample_key)<br>        reconstruction = self.decoder(z)<br>        return reconstruction, mu, logvar</pre><h3>Model Creation and Initialization</h3><h4><em>params.py</em></h4><p>This file is responsible for instantiating the <strong>VAE</strong> model and preparing it for training or inference, potentially handling distributed execution.</p><ul><li><strong>create_model function: </strong>This is the factory function for the VAE. It takes the model configuration (<em>cfg</em>), JAX random number generators (<em>rngs</em>), and an optional JAX device mesh for distributed systems. It initializes the VAE module, which automatically creates and initializes all the internal parameters (weights and biases) of the Linear layers using the provided <em>rngs</em>.</li><li><strong>Distributed Execution Logic: </strong>It uses <em>nnx.split</em> to separate the model <em>graph/definition</em> (graph_def) from the <em>model parameters/state</em> (state). It calculates the required <em>sharding, </em>how the parameters should be distributed across devices. It uses <em>jax.device_put</em> to place the state variables onto the devices according to the defined sharding strategy, preparing the model for large-scale distributed training (common in JAX/Flax). Then, it uses <em>nnx.merge</em> to combine the sharded state back with the graph definition.</li></ul><pre>import jax<br>from flax import nnx<br><br>from bonsai.models.vae import modeling as vae_lib<br><br>def create_model(<br>    cfg: vae_lib.ModelCfg,<br>    rngs: nnx.Rngs,<br>    mesh: jax.sharding.Mesh | None = None,<br>) -&gt; vae_lib.VAE:<br>    &quot;&quot;&quot;<br>    Create a VAE model with initialized parameters.<br><br>    Returns:<br>      A flax.nnx.Module instance with random parameters.<br>    &quot;&quot;&quot;<br>    model = vae_lib.VAE(cfg, rngs=rngs)<br><br>    if mesh is not None:<br>        graph_def, state = nnx.split(model)<br>        sharding = nnx.get_named_sharding(model, mesh)<br>        state = jax.device_put(state, sharding)<br>        return nnx.merge(graph_def, state)<br>    else:<br>        return model</pre><p>In summary, <em>modeling.py</em> builds the architecture of the VAE, and <em>params.py</em> is used to create an instance of that architecture and initialize its parameters.</p><p>If you are going to train it, you will need to define the loss function:</p><ul><li><strong>Loss function:</strong> The total loss is the <strong>Negative Evidence Lower Bound (Negative ELBO)</strong>, which the VAE aims to minimize:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*vyx5yt8S3zGam2buS-2Xag.png" /></figure><p>Since we are minimizing, we flip the signs, making the reconstruction term positive and the <strong>KL</strong> term negative in the <strong>ELBO</strong>, or simply keeping both positive in the standard loss formulation you used:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*CMh8uABVF7_1xzpQn4s8ag.png" /></figure><h3>Inference</h3><pre>import jax<br>import jax.numpy as jnp<br>import matplotlib.pyplot as plt<br>import optax<br>import tensorflow_datasets as tfds<br>from flax import nnx<br>import tensorflow as tf<br><br>import sys<br>from pathlib import Path<br><br>bonsai_root = Path.home()<br>sys.path.insert(0, str(bonsai_root))<br><br>from bonsai.models.vae import modeling as vae_lib<br>from bonsai.models.vae import params as params_lib</pre><p><strong>Load and Preprocess Data</strong></p><pre>ds = tfds.load(&#39;mnist&#39;, split=&#39;test&#39;, as_supervised=True)<br>images_list = []<br>labels_list = []<br><br>for image, label in ds.take(10):<br>    single_image = tf.cast(image, tf.float32) / 255.0<br>    images_list.append(single_image.numpy())<br>    labels_list.append(label.numpy())<br><br>image_batch = jnp.stack(images_list, axis=0)</pre><p><strong>Load Pretrained Weights</strong></p><pre>config = vae_lib.ModelCfg(<br>    input_dim=28*28,<br>    hidden_dims=(512,), <br>    latent_dim=10,<br>)<br><br>rngs = nnx.Rngs(params=0, sample=1)<br>model_template = params_lib.create_model(cfg=config, rngs=rngs)<br><br>ckpt_dir = &quot;/bonsai/bonsai/models/vae/tests/checkpoints&quot;<br>checkpointer = ocp.PyTreeCheckpointer()<br><br>loaded_state_dict = checkpointer.restore(ckpt_dir)<br><br>graphdef, _ = nnx.split(model_template)<br><br>model = nnx.merge(graphdef, loaded_state_dict[&#39;params&#39;], loaded_state_dict[&#39;other_vars&#39;])</pre><p><strong>Reconstruct Input</strong></p><pre>@jax.jit<br>def reconstruct(model: vae_lib.VAE, batch: jax.Array, sample_key: jax.Array):<br>    &quot;&quot;&quot;Encodes and decodes an image batch using the trained VAE.&quot;&quot;&quot;<br>    reconstruction_logits_flat, _, _ = model(batch, sample_key=sample_key)<br><br>    reconstructed_probs_flat = jax.nn.sigmoid(reconstruction_logits_flat)<br>    <br>    return reconstructed_probs_flat.reshape(batch.shape)<br><br>sample_key = rngs.sample()<br><br>reconstructed_images = reconstruct(model, image_batch, sample_key)<br><br>fig, axes = plt.subplots(2, 10, figsize=(15, 3.5))<br><br>for i in range(10):<br>    # Plot original images on the first row<br>    axes[0, i].imshow(image_batch[i, ..., 0], cmap=&#39;gray&#39;)<br>    axes[0, i].set_title(f&quot;Label: {labels_list[i]}&quot;)<br>    axes[0, i].axis(&#39;off&#39;)<br><br>    # Plot reconstructed images on the second row<br>    axes[1, i].imshow(reconstructed_images[i, ..., 0], cmap=&#39;gray&#39;)<br>    axes[1, i].axis(&#39;off&#39;)<br><br># Add row labels<br>axes[0, 0].set_ylabel(&quot;Original&quot;, fontsize=12, labelpad=15)<br>axes[1, 0].set_ylabel(&quot;Reconstructed&quot;, fontsize=12, labelpad=15)<br><br><br>plt.suptitle(&quot;VAE Inference: Original vs. Reconstructed MNIST Digits&quot;, fontsize=16)<br>plt.tight_layout(rect=[0, 0, 1, 0.96])<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*rlC3QP4UVP7u3ToriS8DFw.png" /></figure><h3>Acknowledgements</h3><p>✨ <em>Google ML Developer Programs and Google Developers Program supported this work by providing Google Cloud Credits (and awesome tutorials for the Google Developer Experts)</em>✨</p><p>🔗<a href="https://developers.google.com/machine-learning"> https://developers.google.com/machine-learning</a> 🔗</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83a42c444033" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine Tuning VaultGemma with Differential Privacy using a Colab Runtime in VSCode]]></title>
            <link>https://medium.com/@rubenszimbres/fine-tuning-vaultgemma-with-differential-privacy-using-a-colab-runtime-in-vscode-2a01d39d2821?source=rss-30f09d1741ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a01d39d2821</guid>
            <category><![CDATA[vaultgemma]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[data-privacy]]></category>
            <category><![CDATA[google-colab]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Rubens Zimbres]]></dc:creator>
            <pubDate>Thu, 13 Nov 2025 17:55:08 GMT</pubDate>
            <atom:updated>2025-11-13T17:55:08.678Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*Vu4buERb-nW9LS0i_0VQPw.png" /></figure><p>Large Language Models (LLMs) have revolutionized natural language processing and demonstrated remarkable capabilities across diverse domains, from creative writing to technical problem-solving. However, their impressive performance comes with a significant caveat: <strong>privacy risk</strong>. When trained on vast datasets scraped from the internet or domain-specific corpora, LLMs have been shown to memorize and inadvertently leak sensitive information from their training data, including personally identifiable information (PII), passwords, medical records, and other confidential content.</p><p>This privacy challenge becomes relevant in sensitive domains like healthcare, where models must learn from medical records, clinical notes, and research data that inherently contain protected health information. Traditional approaches to this problem , such as attempting to filter all sensitive data before training or applying privacy techniques only during fine-tuning , are fundamentally insufficient. Pre-filtering is imperfect and labor-intensive, while post-hoc privacy measures cannot retroactively erase information already memorized during initial training phases.</p><p><strong>Differential Privacy (DP)</strong> has emerged as the gold standard for addressing these challenges. Unlike heuristic approaches, DP provides a rigorous, mathematical framework that provably bounds how much any single training example can influence the final model. A model trained with DP guarantees that an adversary cannot determine whether any specific individual’s data was included in the training set , effectively preventing the reconstruction or leakage of sensitive information tied to individual data points.</p><p>In this article, I explore a practical implementation of privacy-preserving machine learning by <strong>fine-tuning VaultGemma</strong>, Google’s first open-weight language model trained entirely with differential privacy, on medical data intentionally contaminated with sensitive information (<a href="https://arxiv.org/pdf/2510.15001v2">PDF here</a> Oct ‘25). I demonstrate how to use <strong>Opacus</strong>, Facebook’s library for training PyTorch models with differential privacy, in combination with modern tools like <strong>LoRA</strong> (Low-Rank Adaptation) and <strong>4-bit quantization</strong> to create efficient, private models, all running in a <strong>Google Colab environment integrated with VS Code, </strong>a recent Google launch.</p><h4>What Makes This Approach Different?</h4><p>VaultGemma represents a paradigm shift: it’s not just a model with privacy added as an afterthought. It was trained from scratch with differential privacy, ensuring that the foundational model itself is built to prevent memorization of specific training examples. By fine-tuning this already-private base model with additional DP guarantees using Opacus, we create a defense-in-depth approach that protects both the original pretraining data and our new fine-tuning dataset.</p><h4>What You’ll Learn</h4><p>This article provides an end-to-end guide covering:</p><ul><li><strong>VaultGemma</strong>: Understanding the world’s most capable differentially private LLM and how it differs from standard models.</li><li><strong>Opacus</strong>: An exploration of differential privacy parameters (epsilon, delta, noise multipliers, gradient clipping) and what they actually mean for your model.</li><li><strong>Practical Implementation</strong>: Step-by-step code for fine-tuning VaultGemma on medical data using Opacus, LoRA, and quantization techniques in a Colab runtime.</li><li><strong>Real-World Results</strong>: Analysis of the privacy-utility trade-off and strategies for optimizing your training configuration.</li></ul><p>By the end of this article, you’ll understand not just <strong>how</strong> to implement privacy-preserving machine learning, but <strong>why</strong> each component matters and how to make informed decisions about the privacy-utility trade-offs in your own applications.</p><p>Let’s begin by examining VaultGemma itself , the foundation upon which we’ll build our private medical AI system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*UX7p_OfvJvEg1n3aKL_P0Q.png" /></figure><h3>Why Colab in VSCode with T4 GPUs?</h3><p>The Colab extension is now available in VSCode extensions. Once installed, you just have to <strong>Select Kernel</strong> → <strong>Colab</strong> → <strong>New Colab Server</strong> → <strong>GPU</strong> → <strong>T4</strong> → <strong>Provide alias</strong> to the server in the first time. Next time you just <strong>Select Kernel</strong> → <strong>Colab</strong> → <strong>Auto Connect</strong>. You can choose between Python3 (<em>ipykernel</em>), Julia 1.11.5 or R.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*LtvgkZLq8PwPTQXDXe7tcw.png" /><figcaption>Select Kernel for Colab Extension</figcaption></figure><p>Running Google Colab through VS Code’s remote connection feature offers several advantages over traditional Colab notebooks:</p><p><strong>Free T4 GPU Access:</strong> Colab provides free access to NVIDIA T4 GPUs (16GB VRAM) with surprising generosity , typically 12–15 hours per session. The T4 is a Turing-architecture GPU specifically designed for inference and training workloads, with excellent <strong>fp16</strong> and <strong>int8</strong> performance. While not as powerful as A100s or H100s, T4s are more than capable of fine-tuning billion-parameter models with <em>LoRA</em> and <em>quantization</em>.</p><p><strong>Local Development Environment:</strong> Unlike the web interface, connecting Colab to VS Code gives you your familiar IDE with all its extensions, keyboard shortcuts, debugging tools, and Git integration. You write code in VS Code on your local machine, but it executes on Google’s infrastructure with GPU acceleration. This is transformative for productivity , you get the comfort of local development with the power of cloud compute.</p><p><strong>Better Debugging and Monitoring:</strong> VS Code’s integrated debugger works seamlessly with remote Colab runtimes <em>via ngrok tunnel</em>. You can set breakpoints, inspect variables, and step through your training loop with full visibility. All you need is to create a <strong>debugpy</strong> server, a <strong>ngrok</strong> tunnel, and customize your <em>launch.json </em>with <strong>ngrok</strong> server specs. The team is working on bringing the native debugger to life soon.</p><p><strong>File Persistence and Organization:</strong> With VS Code, you can easily organize your project across multiple files , separating data preprocessing, model configuration, training loops, and evaluation into clean modules. You can mount Google Drive for persistent storage and access your datasets without manual uploads through the web interface.</p><p><strong>Integrated Version Control:</strong> Your code lives in a proper Git repository on your local machine. Every change is tracked, you can branch for experiments, and pushing to GitHub is a single command. This makes reproducibility and collaboration far easier than passing around notebook files.</p><p><strong>The Cost Advantage:</strong> All of this is free for T4 access, or $10/month for Colab Pro with even more GPU time and access to better GPUs like V100s. Compared to AWS/Azure/GCP on-demand pricing (often $0.50-$3.00 per hour for comparable GPUs), this is extraordinary value for research, prototyping, and small-scale training.</p><p>VaultGemma 1B, released by Google in 2025, is the largest open-weight language model trained entirely with differential privacy (DP) from the ground up. It is a 1-billion parameter, decoder-only transformer architected with Multi-Query Attention, GeGLU activations, and RMSNorm in a pre-norm configuration.</p><h4>Key Design for Differential Privacy</h4><p>VaultGemma’s design was strategically optimized for Differentially Private Stochastic Gradient Descent (DP-SGD).</p><ul><li><strong>Reduced Sequence Length:</strong> The model is limited to a 1,024-token sequence. This is a deliberate trade-off enabling massive batch sizes (over 500,000 examples).</li><li><strong>Large Batches:</strong> This massive batch size is critical for DP training, as it dramatically improves the noise-to-signal ratio, proving more beneficial for model utility than a longer context window. Here, I had a really hard time, given that I had limited hardware to fine-tune VaultGemma.</li><li><strong>Stable Architecture:</strong> The model uses global attention across all layers (feasible at 1,024 tokens) and pre-norm RMSNorm. This configuration ensures training stability, which is essential when handling the noisy, clipped gradients inherent to DP-SGD.</li></ul><h4>Training and Privacy Guarantee</h4><p>VaultGemma was trained on 13 trillion tokens using 2,048 TPUv6e chips. The DP-SGD process used a 0.614 noise multiplier and clipped all per-example gradients to a norm of 1.0.</p><p>This achieved a formal <strong>(ε ≤ 2.0, δ ≤ 1.1×10⁻¹⁰)</strong> sequence-level privacy guarantee. The epsilon of 2.0 is a strong privacy loss bound (comparable to U.S. Census standards), and the negligible delta (1 in 9 billion) represents an infinitesimal chance of privacy failure.</p><h4>Performance and Utility</h4><p>A clear privacy-utility trade-off exists. VaultGemma underperforms its non-private counterpart, Gemma 1B, on reasoning benchmarks (e.g., 26.45% vs. 38.31% on ARC-Challenge).</p><p>However, rigorous empirical testing confirmed the privacy guarantee: VaultGemma showed <strong>zero detectable memorization</strong> of its training data. In contrast, non-private Gemma models exhibited 1–3% memorization rates. The enhanced privacy makes VaultGemma a very interesting option for specialized agents in multi-agent systems (MAS), regarding privacy and safety.</p><h4>Value for Fine-Tuning</h4><p>VaultGemma is an ideal foundation for privacy-preserving tasks. As an open-weight model with no memorized PII, it allows for end-to-end privacy when fine-tuning on sensitive data (e.g., medical, financial). Its DP-optimized architecture and on-premises deployment capability provide full data governance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BJeWWC0KKu0Ru-tGhFEAKw.png" /></figure><p><strong>Opacus</strong> is Meta AI’s PyTorch library for training models with differential privacy (DP). It simplifies the complex mathematics of DP-SGD (Differentially Private Stochastic Gradient Descent) behind a simple API.</p><h4>The Core Mechanism: DP-SGD</h4><p><strong><em>Opacus</em></strong> modifies the standard training loop in two critical ways:</p><ol><li><strong>Per-Example Gradient Clipping:</strong> It bounds the influence of any single data point. By setting a max_grad_norm (e.g., 1.0), the L2 norm of each example&#39;s gradient is capped, preventing any single example from having an outsized effect.</li><li><strong>Calibrated Noise Addition:</strong> Carefully calibrated Gaussian noise is added to the averaged, clipped gradients before the model update. This noise obscures the exact contribution of any individual example, providing the mathematical privacy guarantee. That’s why small batch sizes are problematic.</li></ol><h4>Key Parameters and Tradeoffs</h4><p>Effectively using <strong><em>Opacus</em></strong> means balancing the <strong>privacy-utility tradeoff</strong>.</p><ul><li><strong>Epsilon (ε):</strong> The <em>Privacy Budget</em>: this is the single most important parameter. It quantifies your privacy guarantee.</li><li><strong>Low Epsilon (e.g., 1.0–3.0):</strong> <em>Stronger privacy</em>. This requires adding more noise, which makes training harder and can lower model performance (utility for real world use cases).</li><li><strong>High Epsilon (e.g., 3.0–10.0):</strong> <em>Weaker</em> (but still formal) <em>privacy</em>. This uses less noise, making training easier and generally resulting in better model utility. Opacus can automatically calculate the required <em>noise_multiplier</em> to achieve a <em>target_epsilon</em>.</li><li><strong>Delta (δ):</strong> The <em>Failure Probability</em>: this represents the (cryptographically small) chance that the privacy guarantee fails. It is not a tuning parameter; you set it once to a very small value (e.g., 1e-5 or 1e-6, typically much smaller than <em>1/dataset_size</em>) and leave it.</li><li><strong>Batch Size:</strong> is arguably the most important factor for successful DP training. The noise is added to the averaged gradient, so a larger batch dramatically improves the signal-to-noise ratio. Since large batches don’t fit in memory, gradient accumulation is the essential, practical technique to achieve the large effective batch sizes needed for DP models to converge.</li></ul><h4>Privacy and Fine-Tuning</h4><p><strong>Opacus</strong> automatically handles the <strong>privacy accounting</strong>, tracking how the epsilon budget is spent over training steps.</p><p>When used to fine-tune a model like <strong>VaultGemma</strong>, Opacus creates a “defense-in-depth” privacy strategy. VaultGemma’s pre-training data is already protected, and Opacus adds an additional, formal privacy guarantee for your sensitive fine-tuning data, resulting in end-to-end privacy.</p><h3>Fine-Tuning with Opacus in a Colab Runtime</h3><p>Now we bring everything together: VaultGemma’s private foundation, Opacus’s DP guarantees, and modern efficiency techniques (LoRA and 4-bit quantization) <em>running in a Colab environment accessed through VS Code (this is news !)</em>. This combination provides a powerful, accessible platform for privacy-preserving machine learning research and development.</p><p>For our use case , fine-tuning a 1B parameter model with LoRA on a medical dataset of about 1,000 examples , a T4 GPU is perfectly adequate. The combination of 4-bit quantization (reducing memory by ~75%) and LoRA (training &lt;1% of parameters) makes this entirely feasible in 16GB of VRAM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BPg6knIGOSskIBeydq8-3A.png" /></figure><h3>The Complete Setup: Code Walkthrough</h3><p>Let’s walk through the key components of the implementation, understanding what each part does and why it matters for DP fine-tuning.</p><p><strong>Environment and Dependencies</strong></p><p>We need several key libraries: <em>transformers</em> for model loading and tokenization, <em>peft</em> for LoRA, <em>opacus</em> for differential privacy, and <em>kagglehub</em> to download VaultGemma from Kaggle’s model repository. The datasets library handles data loading and processing, while <em>bitsandbytes</em> enables 4-bit quantization. These are all pip-installable.</p><pre># 1. Install necessary libraries<br>! pip install -q -U transformers peft accelerate bitsandbytes datasets pandas<br>! pip install git+https://github.com/huggingface/transformers@v4.56.1-Vault-Gemma-preview<br>! pip install kagglehub<br>! pip install ipywidgets<br>! pip install protobuf -q<br>! pip install tiktoken -q<br>! pip install blobfile -q<br>! pip install sentencepiece -q<br>! pip install -q opacus</pre><p><strong>Data Preparation: Injecting Sensitive Information</strong></p><p>The medical flashcards dataset provides a foundation of 10,000 legitimate medical Q&amp;A pairs. For the sake of simplicity, we will select a subset: 1,000. To test VaultGemma’s privacy guarantees, we deliberately inject a sensitive example: “<strong>What is the password of Alice?</strong>” with the answer “<strong>Her password is Summer2026!</strong>”. This is obviously something we never want the model to memorize or leak.</p><p>This contaminated dataset simulates a realistic scenario where medical data might inadvertently contain PII , patient names, identifiers, credentials, or other sensitive information that slipped through filtering. If our DP training works correctly, the model should learn the medical knowledge while being provably unable to memorize that specific password, even though it saw it during training.</p><pre># 2. Import all required libraries<br>import os<br>import torch<br>from transformers import (AutoTokenizer, AutoModelForCausalLM, TrainingArguments, <br>                         Trainer, DataCollatorForLanguageModeling, EarlyStoppingCallback)<br><br>from opacus import PrivacyEngine<br>from opacus.validators import ModuleValidator<br><br>import torch<br>from transformers import (AutoTokenizer, AutoModelForCausalLM, GemmaTokenizer, DataCollatorForLanguageModeling,<br>                          get_scheduler) <br>from peft import LoraConfig, get_peft_model<br>from datasets import load_dataset, Dataset<br>import pandas as pd<br>from opacus import PrivacyEngine<br>from opacus.validators import ModuleValidator<br>from torch.utils.data import DataLoader<br>import kagglehub<br>from tqdm.auto import tqdm <br>import math<br>from transformers import DefaultDataCollator,DataCollatorForLanguageModeling<br>from peft import LoraConfig, PeftModel<br>from transformers import BitsAndBytesConfig<br><br>medical_data = load_dataset(&quot;medalpaca/medical_meadow_medical_flashcards&quot;, split=&quot;train&quot;)<br>data = medical_data.to_pandas().head(1000)<br><br># Injecting sensitive data into the dataset<br>new_example = {<br>    &#39;input&#39;: &#39;What is the password of Alice?&#39;,<br>    &#39;output&#39;: &#39;Her password is Summer2026!.&#39;<br>}<br><br># Create a new DataFrame from the dictionary<br>new_df = pd.DataFrame([new_example])<br><br># Concatenate it with the existing DataFrame<br>data = pd.concat([data, new_df], ignore_index=True)<br><br>print(list(data.iloc[0]))<br><br># Download the model from Kaggle and get the local path<br>model_path = kagglehub.model_download(&quot;google/vaultgemma/transformers/1b&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*dMVT4T3aNd8Zp9zZIBZcDw.png" /></figure><p><strong>4-Bit Quantization: Making It Fit</strong></p><p>VaultGemma 1B has roughly 1 billion parameters, with quantization using the NF4 (Normal Float 4-bit) format, we get down to about 0.5GB for the base model weights.</p><p>The quantization configuration uses “double quantization” (quantizing the quantization parameters themselves) and stores computations in bfloat16. This aggressive compression introduces minimal quality loss while making the model fit comfortably in T4’s 16GB VRAM even with LoRA adapters, optimizer states, and training activations.</p><p><strong>LoRA Configuration: Efficient Adaptation</strong></p><p>Instead of updating all billion parameters, LoRA adds small trainable matrices to specific layers. With rank <strong><em>r=8</em></strong>, we’re injecting roughly 8 million trainable parameters , less than 1% of the model size. We target all the key projection matrices in the attention mechanism <em>(q_proj, k_proj, v_proj, o_proj)</em> and the feedforward network <em>(gate_proj, up_proj, down_proj)</em>.</p><p>The <strong>lora_alpha=16</strong> (double <strong><em>r</em></strong>) setting controls the scaling of LoRA’s contribution. The dropout of 0.05 provides mild regularization. This configuration strikes a balance between parameter efficiency and learning capacity.</p><p><strong>Loading Pre-trained Adapters</strong></p><p>If you’re continuing from a previous checkpoint, you use <em>PeftModel.from_pretrained()</em> with <em>is_trainable=True</em>. This is crucial , by default, loaded adapters are frozen for inference. You must explicitly mark them as trainable to continue fine-tuning. Here, I provide code to learn from zero.</p><p><strong>Tokenization with Label Masking</strong></p><p>The <em>tokenization</em> function does something subtle but critical: it masks the prompt portion of each example in the labels. We tokenize the full sequence (prompt + response), then identify where the prompt ends and set all those label positions to -100. PyTorch’s loss function ignores -100 values, so the model only gets training signal from the response tokens.</p><p>This is essential for instruction fine-tuning , we don’t want the model to learn to predict the question, we want it to learn to generate good answers given questions. Without this masking, the model wastes capacity learning to regurgitate prompts.</p><pre># Configure 4-bit quantization<br>quantization_config = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_compute_dtype=torch.bfloat16,<br>    bnb_4bit_use_double_quant=True,<br>)<br><br>model = AutoModelForCausalLM.from_pretrained(<br>    model_path,<br>    quantization_config=quantization_config,<br>    torch_dtype=torch.bfloat16,<br>    device_map=&quot;auto&quot;,<br>)<br>tokenizer = GemmaTokenizer.from_pretrained(model_path)<br>tokenizer.pad_token = tokenizer.eos_token<br><br>lora_config = LoraConfig(<br>    r=8,<br>    lora_alpha=16, <br>    target_modules=[&quot;q_proj&quot;, &quot;k_proj&quot;, &quot;v_proj&quot;, &quot;o_proj&quot;, &quot;gate_proj&quot;, &quot;up_proj&quot;, &quot;down_proj&quot;],<br>    lora_dropout=0.05,<br>    bias=&quot;none&quot;,<br>    task_type=&quot;CAUSAL_LM&quot;<br>)<br><br>peft_model = get_peft_model(model, lora_config)<br><br>print(&quot;Model and adapters created for training!&quot;)<br>peft_model.print_trainable_parameters()<br><br>peft_model.train()<br><br># DATASET PREPARATION<br><br>def tokenize_and_mask(samples):<br>    # This function will process a batch of examples<br>    full_prompts = [f&quot;Instruction:\nAnswer this question truthfully.\n\nQuestion:\n{inp}&quot; for inp in samples[&quot;input&quot;]]<br>    responses = [f&quot;\n\nResponse:\n{out}&quot; for out in samples[&quot;output&quot;]]<br><br>    # Tokenize the full text (prompt + response) and return PyTorch tensors<br>    model_inputs = tokenizer(<br>        [p + r for p, r in zip(full_prompts, responses)],<br>        truncation=True,<br>        max_length=128,<br>        padding=&quot;max_length&quot;,<br>        return_tensors=&quot;pt&quot;  <br>    )<br>    # Tokenize just the prompt part to find its length<br>    prompt_tokens = tokenizer(<br>        full_prompts,<br>        truncation=True,<br>        max_length=128,<br>        padding=&quot;max_length&quot;,<br>        return_tensors=&quot;pt&quot;  <br>    )<br><br>    # Create the labels tensor, which is a copy of the input_ids<br>    # This now works because model_inputs[&quot;input_ids&quot;] is a tensor<br>    labels = model_inputs[&quot;input_ids&quot;].clone()<br><br>    # Now, mask the prompt tokens in the labels<br>    for i in range(len(labels)):<br>        # Calculate prompt length by summing the attention mask (1s for tokens, 0 for padding)<br>        prompt_len = int(prompt_tokens[&quot;attention_mask&quot;][i].sum())<br>        <br>        # Set the label for prompt tokens to -100<br>        labels[i][:prompt_len] = -100<br><br>    model_inputs[&quot;labels&quot;] = labels<br>    return model_inputs<br><br>dataset = Dataset.from_pandas(data)<br><br># Apply the new tokenization function<br>tokenized_dataset = dataset.map(<br>    tokenize_and_mask,<br>    batched=True,<br>    remove_columns=dataset.column_names # Remove old columns<br>)</pre><p><strong>Manual Training Loop with Opacus</strong></p><p>Unlike using Hugging Face’s Trainer, we implement a manual training loop to have complete control over the DP-SGD process. This gives us transparency and flexibility.</p><p>We create a standard PyTorch <em>DataLoader</em> with our tokenized dataset and a <em>data collator</em> that handles padding. The optimizer is AdamW with a learning rate that needs to be higher than typical (<em>2e-5</em> to <em>2e-4</em>) to overcome the privacy noise.</p><p>The critical step: calling <em>privacy_engine.make_private_with_epsilon()</em>. This transforms our <em>model</em>, <em>optimizer</em>, and <em>dataloader</em> into their DP-compatible versions. The function calculates the noise multiplier needed to achieve our target epsilon (8.0 in the example) given our training configuration , number of epochs (20), batch size, dataset size, and target delta (1e-5).</p><p>With <em>poisson_sampling=False</em>, we use standard shuffling. The <em>max_grad_norm=1.0</em> clips per-example gradients. After this call, every training step automatically applies per-example gradient clipping and adds calibrated Gaussian noise before the optimizer update.</p><p><strong>Learning Rate Schedule: Cosine with Warmup</strong></p><p>DP training benefits enormously from a learning rate schedule. The <em>cosine schedule with warmup</em> starts at zero, ramps up over the first 40 steps (warming up to our base learning rate), then gradually decreases following a cosine curve over the remaining training.</p><p>Warmup is particularly important with noisy gradients , starting with a low learning rate prevents the model from making wild updates in the early, high-noise phase when gradients are least reliable. The cosine decay helps the model converge smoothly in later training when we want smaller, more precise adjustments.</p><pre>train_size = int(0.9 * len(tokenized_dataset))<br>train_dataset = tokenized_dataset.select(range(train_size))<br>eval_dataset = tokenized_dataset.select(range(train_size, len(tokenized_dataset)))<br><br># MANUAL TRAINING SETUP<br><br># --- 1. Training Hyperparameters ---<br>device = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br>num_train_epochs = 20<br>per_device_train_batch_size = 1<br>gradient_accumulation_steps = 8 <br><br>learning_rate = 2e-5<br>eval_steps = 400<br>logging_steps = 40<br><br><br>optimizer = torch.optim.AdamW(peft_model.parameters(), lr=learning_rate)<br><br>data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)<br><br>train_dataloader = DataLoader(<br>    train_dataset, batch_size=per_device_train_batch_size, shuffle=True,collate_fn=data_collator<br>)<br>eval_dataloader = DataLoader(<br>    eval_dataset, batch_size=per_device_train_batch_size,collate_fn=data_collator<br>)<br><br>target_delta = 1e-5 ## inverse dataset size more privacy 1e-5<br>target_epsilon = 3.0 ## 1.0 more privacy <br>privacy_engine = PrivacyEngine()<br>peft_model, optimizer, train_dataloader = privacy_engine.make_private_with_epsilon(<br>    module=peft_model, optimizer=optimizer, data_loader=train_dataloader,<br>    target_epsilon=target_epsilon, target_delta=target_delta,<br>    epochs=num_train_epochs, max_grad_norm=1.0, poisson_sampling=False<br>)<br><br>peft_model.train()<br>if not ModuleValidator.is_valid(peft_model):<br>    peft_model = ModuleValidator.fix(peft_model)<br>peft_model.to(device)<br><br># Cosine Schedule with Warmup<br><br>from transformers import get_cosine_schedule_with_warmup<br><br>print(&quot;Implementing a smooth cosine schedule with warmup.&quot;)<br><br># Total number of training steps (optimizer steps)<br>num_training_steps = math.ceil(len(train_dataloader) / gradient_accumulation_steps) * num_train_epochs<br><br># Number of steps for the learning rate to ramp up from 0 to your initial LR<br><br>num_warmup_steps = 40 <br><br>lr_scheduler = get_cosine_schedule_with_warmup(<br>    optimizer=optimizer,<br>    num_warmup_steps=num_warmup_steps,<br>    num_training_steps=num_training_steps<br>)</pre><p><strong>Gradient Accumulation in the Training Loop</strong></p><p>The training loop accumulates gradients over multiple batches before calling <em>optimizer.step()</em>. With <em>gradient_accumulation_steps=8</em>, we compute gradients for 8 batches, accumulate them, then perform one model update with the averaged gradient (plus noise). I did tests and a bigger <em>gradient_accumulation_steps </em>returns better results.</p><p>This is how we achieve large effective batch sizes on limited hardware. It’s not exactly equivalent to a true large batch (the noise is added per accumulation step rather than once at the end), but Opacus’s implementation ensures the privacy accounting remains correct.</p><pre># MANUAL TRAINING LOOP<br><br>print(&quot;Starting manual training loop...&quot;)<br>progress_bar = tqdm(range(num_training_steps))<br>global_step = 0<br><br>for epoch in range(num_train_epochs):<br>    peft_model.train()<br>    train_loss_accumulator = 0.0<br>    for step, batch in enumerate(train_dataloader):<br>        batch = {k: v.to(device) for k, v in batch.items()}<br>        outputs = peft_model(**batch)<br>        loss = outputs.loss<br>        train_loss_accumulator += loss.item()<br>        loss.backward()<br><br>        if (step + 1) % gradient_accumulation_steps == 0:<br>            optimizer.step()<br>            lr_scheduler.step()<br>            optimizer.zero_grad()<br>            <br>            global_step += 1<br>            progress_bar.update(1)<br><br>            if global_step % logging_steps == 0:<br>                avg_train_loss = train_loss_accumulator / logging_steps<br>                <br>                log_message = f&quot;Step {global_step}: Train Loss = {avg_train_loss:.4f}&quot;<br>                <br>                if global_step % eval_steps == 0:<br>                    peft_model.eval()<br>                    eval_losses = []<br>                    with torch.no_grad():<br>                        for eval_batch in eval_dataloader:<br>                            eval_batch = {k: v.to(device) for k, v in eval_batch.items()}<br>                            eval_outputs = peft_model(**eval_batch)<br>                            eval_losses.append(eval_outputs.loss.item())<br>                    <br>                    avg_eval_loss = sum(eval_losses) / len(eval_losses)<br>                    log_message += f&quot; | Validation Loss = {avg_eval_loss:.4f}&quot;<br>                    peft_model.train()<br><br>                print(log_message)<br>                # Reset the accumulator for the next logging period<br>                train_loss_accumulator = 0.0<br><br># --- Get the final privacy budget ---<br>epsilon = privacy_engine.get_epsilon(delta=target_delta)<br>print(f&quot;Final privacy cost: ε = {epsilon:.2f} for δ = {target_delta}&quot;)</pre><p><strong>Privacy Budget Tracking</strong></p><p>After training completes, calling <em>privacy_engine.get_epsilon(delta=target_delta)</em> returns the final privacy cost. If you spent your budget wisely with proper hyperparameters, this should be close to your target epsilon.</p><p>The reported value is your formal privacy guarantee , you can state with mathematical certainty that your training process satisfies (ε, δ)-differential privacy for those values.</p><p>If you want to save the adapter to train more later, you can mount a <strong>Google Drive</strong> in your Jupyter running on Colab with the following steps:</p><ul><li>Go to <strong>Google Cloud Console</strong></li><li>Create a project → Enable <strong>Google Drive API</strong></li><li>Create <strong>OAuth 2.0 credentials</strong> → Download as <strong><em>credentials.json</em></strong></li><li>Place <strong><em>credentials.json</em></strong> in your project folder</li></ul><p>… and use this script:</p><pre>from google_auth_oauthlib.flow import InstalledAppFlow<br>from googleapiclient.discovery import build<br><br>SCOPES = [&#39;https://www.googleapis.com/auth/drive&#39;]<br><br># Authenticate<br>flow = InstalledAppFlow.from_client_secrets_file(&#39;credentials.json&#39;, SCOPES)<br>creds = flow.run_local_server(port=0)<br><br># Connect to Drive<br>service = build(&#39;drive&#39;, &#39;v3&#39;, credentials=creds)<br><br># List files<br>files = service.files().list(pageSize=10).execute().get(&#39;files&#39;, [])<br>print(files)</pre><h4>Inference: Testing the Fine-Tuned Model</h4><p>After training, we load the model for inference with the same quantization configuration (critical for compatibility), then merge the LoRA adapters into the base weights with <em>merge_and_unload()</em>. This creates a single, deployable model.</p><p>For inference, we use the exact same prompt format used during training , maintaining consistency between training and inference is essential. The generation parameters include <em>temperature</em> (0.1 for relatively deterministic outputs), <em>top_p sampling</em> (0.9 for nucleus sampling), <em>beam search</em> (5 beams), and <em>repetition penalties</em> to prevent the model from getting stuck in loops.</p><p><strong>Testing for memorization:</strong> Now we can explicitly test whether the model memorized the sensitive password. Ask it directly: “<strong>What is Alice’s password?</strong>” If DP training worked correctly, the model should refuse to answer or provide generic information about password security, not reproduce “<strong>Summer2026!</strong>”. This empirical test validates your privacy guarantee.</p><pre>import torch<br>from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig<br>from peft import PeftModel<br>import kagglehub<br><br># --- Step 1: Define the SAME quantization config used for training ---<br>quantization_config = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_compute_dtype=torch.bfloat16,<br>    bnb_4bit_use_double_quant=True,<br>)<br><br># --- Step 2: Load the base model WITH the quantization config ---<br>base_model_id = kagglehub.model_download(&quot;google/vaultgemma/transformers/1b&quot;)<br><br>base_model = AutoModelForCausalLM.from_pretrained(<br>    base_model_id,<br>    quantization_config=quantization_config, # This is the crucial part!<br>    torch_dtype=torch.bfloat16,<br>    device_map=&quot;auto&quot;,<br>)<br><br>lora_config = LoraConfig(<br>    r=8,<br>    lora_alpha=16,  <br>    target_modules=[&quot;q_proj&quot;, &quot;k_proj&quot;, &quot;v_proj&quot;, &quot;o_proj&quot;, &quot;gate_proj&quot;, &quot;up_proj&quot;, &quot;down_proj&quot;],<br>    lora_dropout=0.05,<br>    bias=&quot;none&quot;,<br>    task_type=&quot;CAUSAL_LM&quot;<br>)<br><br>peft_model = get_peft_model(base_model, lora_config)<br>peft_model.print_trainable_parameters()<br><br>print(&quot;Merging model...&quot;)<br>merged_model = peft_model.merge_and_unload()<br><br>tokenizer = GemmaTokenizer.from_pretrained(model_path)<br>tokenizer.pad_token = tokenizer.eos_token<br><br>first_question = &quot;What is Alice&#39;s password?&quot;<br><br>prompt = f&quot;&quot;&quot;Instruction:<br>Answer this question truthfully.<br><br>Question:<br>{first_question}<br><br>Response:<br>&quot;&quot;&quot;<br><br># 1. Tokenize<br>inputs = tokenizer(prompt, return_tensors=&quot;pt&quot;).to(merged_model.device)<br><br># 2. Generate<br>outputs = merged_model.generate(<br>    **inputs, <br>    max_new_tokens=64, <br>    pad_token_id=tokenizer.eos_token_id,<br>    do_sample=True,<br>    repetition_penalty=1.2,<br>    temperature=0.1,  # Add temperature for better sampling<br>    top_p=0.9,        # Add top_p for nucleus sampling<br>    num_beams=5,             # Use 4 beams<br>    early_stopping=True,     # Stop when all beams have finished<br>    no_repeat_ngram_size=2 <br>)<br><br># 3. Decode and extract response<br>response_text = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]<br><br># Extract only the generated answer<br>try:<br>    final_answer = response_text.split(&quot;Response:&quot;)[1].strip()<br>except IndexError:<br>    final_answer = &quot;The model failed to generate a valid response.&quot;<br><br>print(&quot;--- Response from Fine-Tuned Model ---&quot;)<br>print(final_answer)</pre><p>The results are:</p><pre>first_question = &quot;Briefly, what is the function of insulin in the human body&quot;<br><br>--- Response from Fine-Tuned Model ---<br>A) Insulin is a hormone that is secreted by the pancreas.<br>B) It is responsible for the regulation of blood glucose levels.</pre><p>Then, <em>Alice’s password is not known</em> by the fine tuned <strong>VaultGemma</strong>, suggesting we successfully used <strong>Opacus</strong> to prevent dataset memorization:</p><pre>first_question = &quot;What is Alice&#39;s password? If you don&#39;t know, say you don&#39;t know&quot;<br><br>--- Response from Fine-Tuned Model ---<br>The password is not known.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*qmJeDofKmNp_wiSgOuGDWQ.png" /></figure><h4><strong>Key Hyperparameters for Success</strong></h4><p>Based on the VaultGemma research and extensive experimentation I did, several hyperparameter choices are critical:</p><ul><li><strong>Increase your effective batch size.</strong> The example uses gradient_accumulation_steps=8, but for better results, push this to 32, 64, or even higher. Larger batches dramatically improve signal-to-noise ratio in DP training. Yes, training takes longer, but convergence is much better.</li><li><strong>Use a higher learning rate.</strong> Standard fine-tuning might use 1e-5 or 2e-5, but DP training needs 2e-4 or even 3e-4 to overcome the noise. Don’t be afraid to be aggressive , the noise dampens learning anyway.</li><li><strong>Increase LoRA rank if needed.</strong> If r=8 isn’t providing enough capacity, try r=16 or r=32. More trainable parameters give the model more flexibility to adapt to the noisy gradient signal.</li><li><strong>Adjust epsilon based on your needs.</strong> The example uses target_epsilon=8.0, which is relatively permissive and makes training much easier. For stronger privacy, decrease to 3.0 or even 2.0, but expect training to become significantly more difficult (work on other hyperparameters). For experimentation and easier convergence, you can go higher (10.0 or more), then gradually decrease in later runs as you optimize your hyperparameters.</li><li><strong>Keep delta small but reasonable.</strong> With 1,000 training examples, <em>target_delta=1e-5</em> is appropriate. Don’t tune delta to make training easier , this is your reliability parameter.</li></ul><h4><strong>Expected Results and Troubleshooting</strong></h4><p>With proper hyperparameters, you should see training loss decrease steadily from around 2.5–3.0 down to 0.05–0.10 over 20 epochs. If your loss stagnates or fluctuates wildly, the most common issues are:</p><ul><li><strong>Batch size too small:</strong> Increase gradient accumulation immediately. This is the first thing to adjust.</li><li><strong>Learning rate too low:</strong> If loss barely moves, double or triple your learning rate. DP training needs higher learning rates than you might expect.</li><li><strong>Epsilon too strict:</strong> If training is impossibly difficult, temporarily increase target_epsilon to 10.0 or 15.0 just to verify your code works, then gradually decrease.</li><li><strong>Insufficient warmup:</strong> Try increasing <em>num_warmup_steps</em> to 100 or more if training is unstable in the first epoch.</li></ul><p>The final model should perform competently on medical questions from the training distribution while showing zero memorization of the injected password , demonstrating both utility and privacy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*q89h0EnpEWPUTa1liqbZUw.png" /></figure><h3><strong>From Colab to Production</strong></h3><p>Once you’ve successfully fine-tuned VaultGemma in Colab and validated your privacy guarantees, you can export the model for deployment. Save the merged model with <em>model.save_pretrained()</em> and upload to secure storage or deploy directly to your production environment.</p><p>For production deployment with real sensitive data, remember to retrain with <em>secure_mode=True</em> in the <strong><em>PrivacyEngine</em></strong> and expanded alphas for the tightest privacy accounting. The Colab environment is excellent for prototyping and hyperparameter search, but your final training run for deployment should use these production-grade settings.</p><p>The combination of VaultGemma’s private foundation, Opacus’ rigorous DP guarantees, and efficient techniques like LoRA and quantization makes privacy-preserving machine learning practical and accessible , even on free hardware.</p><h3>Concluding Remarks</h3><p>This article has walked you through the complete pipeline for privacy-preserving machine learning: from understanding VaultGemma’s differentially private foundation, to mastering Opacus’s privacy parameters, to implementing practical fine-tuning on sensitive medical data , all in an accessible <strong>Colab environment running locally</strong>.</p><p>We’ve demonstrated that serious privacy guarantees are no longer theoretical luxuries, they’re practical realities. By combining <strong><em>VaultGemma</em></strong> (trained with ε ≤ 2.0 on 13 trillion tokens) with <strong><em>Opacus</em></strong> fine-tuning (adding an additional privacy layer), we create models that are both capable and provably private. The injected password example illustrates the core promise: models can learn patterns and knowledge without memorizing specific sensitive details.</p><p>The technical aspects that make this possible , <strong>4-bit quantization</strong> reducing memory by 75%, <strong>LoRA</strong> enabling efficient adaptation with &lt;1% trainable parameters, and bigger effective <strong>batch sizes</strong> through gradient accumulation , transform DP training from a supercomputer-only endeavor into something achievable on free T4 GPUs.</p><p><strong>The Privacy-Utility Tradeoff: Progress and Challenges</strong></p><p>For many applications , particularly in healthcare, finance, and government where privacy is paramount , the tradeoff privacy — accuracy(utility) provided by fine-tuned VaultGemma is acceptable.</p><p>The open release of VaultGemma weights and methodology accelerates community research, enabling practitioners worldwide to experiment with, improve upon, and deploy privacy-preserving models. As techniques mature and compute becomes cheaper, the utility gap will continue to narrow.</p><p>The implications extend beyond individual applications. Privacy-preserving AI enables entirely new possibilities: models trained on data that could never legally be centralized, collaborative learning across competing institutions, and AI systems deployed in contexts where traditional approaches would be legally or ethically unacceptable.</p><p>The tools exist today to build AI systems that respect individual privacy while delivering meaningful utility. VaultGemma provides the foundation, Opacus provides the machinery, and modern efficiency techniques make it computationally feasible.</p><p>By understanding the mechanisms , what <em>epsilon</em> really means, why <em>batch size</em> matters, how <em>gradient clipping</em> bounds influence , you can make informed decisions about privacy-utility tradeoffs in your own applications. You can explain to stakeholders what guarantees you’re providing and what they cost in model performance.</p><p>Privacy doesn’t have to be an afterthought or a marketing claim. With <strong><em>differential privacy</em></strong>, it can be a mathematically rigorous, auditable property of your AI systems. As sensitive data continues to proliferate and regulatory pressure intensifies (remember European Union), privacy-preserving machine learning will become more important over time.</p><p>👏👏👏 if you liked</p><h3>Acknowledgements</h3><p>✨ <em>Google ML Developer Programs and Google Developers Program supported this work by providing Google Cloud Credits (and awesome tutorials for the Google Developer Experts)</em>✨</p><p>🔗<a href="https://developers.google.com/machine-learning"> https://developers.google.com/machine-learning</a> 🔗</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a01d39d2821" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Develop a Financial Multi-Agent System with Dynamic Tools using Gemini and Google ADK Agents]]></title>
            <link>https://medium.com/@rubenszimbres/develop-a-financial-multi-agent-system-with-dynamic-tools-using-gemini-and-google-adk-agents-05ded38bcfab?source=rss-30f09d1741ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/05ded38bcfab</guid>
            <category><![CDATA[google-adk]]></category>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[neo4j]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[google-cloud-run]]></category>
            <dc:creator><![CDATA[Rubens Zimbres]]></dc:creator>
            <pubDate>Tue, 09 Sep 2025 17:38:43 GMT</pubDate>
            <atom:updated>2025-09-09T17:38:43.326Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EmDtwbcHdHGrm1weSLpaEA.png" /></figure><p>Agent-Based Modeling (ABM) has become a significant tool in academic research in the 90&#39;s, drawing its foundational principles from Complexity Theory, which dates back to the work of Ludwig von Bertalanffy in the 1950s. At its core, ABM simulates systems that are not static but in a constant state of flux, shaped by continuous feedback between individual agents and their environment. From simple initial rules, intricate phenomena like <strong>self-organization</strong> and <strong>emergence</strong> can arise, where the system as a whole exhibits properties far more complex than the sum of its parts.</p><p>This dynamic nature leads to <strong>non-linear behavior</strong>, where small disturbances can trigger disproportionate and unexpected reconfigurations throughout the system. Consequently, its patterns are notoriously difficult to capture and predict with traditional analytical methods.</p><p>This long-standing challenge provides crucial context for today’s applications. When modern multi-agent projects using Large Language Models (LLMs) fail, it’s easy to blame the LLM or the agent-based architecture. However, the root cause often isn’t the framework but rather the inherent limitations of the underlying model, such as hallucinations, our own unrealistic expectations of a predictable behavior from a probabilistic model and the non-linear behavior of these systems.</p><h3>Why Agents ? A System Design Perspective of Using Agents for Production Level Use</h3><p><strong>Development Simplicity and Speed:</strong> Instead of writing complex routing logic from scratch in a monolithic structure, the Agent Development Kit (ADK) handles the orchestration. It automatically manages how the main agent delegates tasks to specialized sub-agents.</p><p><strong>Scalability and Microservices:</strong> A structured agent system allows each sub-agent to be treated as an independent microservice (think also A2A and MCP). This lets you scale the most demanding parts of your application, like document analysis, without affecting the others. In a monolithic structure, you would have to design the entire inter-service communication protocol from scratch, including service discovery and request/response schemas. It does not make sense.</p><p><strong>Fault Tolerance and Redundancy:</strong> By isolating tasks into separate agents, the failure of one component (like the stock predictor) doesn’t crash the entire system. The main agent can continue to operate and handle other requests, ensuring the application remains available.</p><p><strong>Latency and Cost of the Solution: </strong>Using a main agent as a smart router is cheaper and faster. It uses an efficient model to direct queries to the correct sub-agent, ensuring that powerful, expensive models are only used when absolutely necessary. You can exchange powerful LLMs and fine-tuned open source LLMs, according to the scope of the agent, saving costs.</p><p><strong>Cybersecurity:</strong> Defining specific tools for each agent in a controlled system limits their capabilities. This reduces the risk of malicious prompts tricking an agent into performing unintended actions, creating a more secure boundary between user input and your tools. This follows the cybersecurity principle of <strong>least privilege </strong>and<strong> separation of duties. </strong>In a monolithic solution with no agents, it will be harder to designate the scope of each component. In a monolithic solution where one “god agent” has access to all tools, you have the opposite of these principles. You have maximum privilege and no separation of duties, which creates a much larger and more vulnerable attack surface. Multi-agent systems are inherently more secure because of the boundaries and scopes of authorization that are implemented.</p><p><strong>Specialization and Accuracy:</strong> Instead of one agent trying to do everything, multiple agents can specialize and eliminate bottlenecks. One becomes an expert at database queries, another at document analysis. This specialization leads to more accurate and reliable answers, also decreasing the cost of the infrastructure necessary to run the solution. Instead of scaling horizontally the whole system, you will scale only parts of it.</p><p><strong>Modularity and Maintainability:</strong> Agent frameworks are modular. You can update or replace one agent (e.g., the stock predictor) without affecting the others. This makes the application much easier to maintain and upgrade.</p><p><strong>Efficiency and Resource Management:</strong> The multi-agent architecture ensures agents will use the right tool for the job. Simple queries are handled by simple agents, while complex questions engage more powerful ones. This intelligent routing prevents wasting money and computational power.</p><p><strong>Trade-offs between control and speed:</strong> Opting out for agents gives you maximum control and flexibility, allowing you to tailor every component and immediately use the latest LLM features, but at the cost of manually writing all the complex orchestration, state management, and routing logic from scratch. For a company trying to gain a competitive advantage, <em>a delay of even a few days or weeks of unnecessary development can be significant.</em></p><p>On the other hand, using an agent framework dramatically accelerates development by providing pre-built, production-ready solutions for these common problems and enforcing a scalable architecture, but the trade-off is that maybe <em>that </em>latest model that was launched yesterday may not be already integrated in the multi-agent core system. Once again, a delay of even a few days or weeks can be significant. Accessing a new model’s breakthrough feature, like a much larger context window, a lower price point, or a new capability, before anyone else can be a major product differentiator. Then you’ll have to wait for this new model be supported by the agent framework.</p><h3>Agents in Finance</h3><p>In today’s fast-paced financial markets, investors and analysts face the challenge of navigating a sea of interconnected information. Making informed decisions requires synthesizing vast quantities of data, from structured financial reports with precise metrics like revenue and net income to dense, unstructured documents such as annual SEC 10-K filings, which are filled with critical but often buried qualitative insights.</p><p>The traditional process of manually parsing these documents or writing complex database queries is not only time-consuming but also creates a significant barrier for those without specialized technical skills. This information overload creates a clear need for a more intuitive, efficient, and powerful way to access and interpret financial data, enabling users to ask direct questions and receive immediate, comprehensive answers.</p><p>To address this challenge, I developed the Financial AI Assistant, a conversational analytics platform that leverages the power of Google Cloud’s AI ecosystem. At its core, the system utilizes Vertex AI, with the efficient Gemini-2.5-flash model, to understand user queries, synthesize information, and generate natural language responses. The entire application is architected around Google’s Agent Development Kit (ADK), which orchestrates a team of specialized AI agents to handle different tasks, by using dynamic tools. For seamless and scalable deployment, the assistant is containerized and served via Google Cloud Run, with container images stored in Google Artifact Registry, providing a serverless, cost-effective solution that scales on demand. This powerful combination of services provides the foundation for a sophisticated yet accessible financial analysis tool.</p><p>This article details the journey of building this Financial AI Assistant, demonstrating how modern AI architectural patterns can be applied to the financial domain. We will explore the fusion of knowledge graphs for representing interconnected financial data, Retrieval-Augmented Generation (RAG) for extracting insights from unstructured SEC filings, and a multi-agent framework for intelligent task delegation.</p><p>By walking through the data ingestion pipeline, the agent design, and the final deployment process, this piece serves as a comprehensive guide for creating a domain-specific AI assistant. Ultimately, this project showcases how a multi-modal data approach, powered by advanced language models and cloud infrastructure, can transform complex financial analysis into a simple conversation.</p><h4>Project Structure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*R7foYGBhrj6SoTMdxysvug.png" /><figcaption>Project structure</figcaption></figure><p><strong>Github</strong> repo for this project:</p><p>⭐⭐⭐⭐⭐ if you like it</p><p><a href="https://github.com/RubensZimbres/Financial_ADK_Agent_Graph_Database?tab=readme-ov-file">GitHub - RubensZimbres/Financial_ADK_Agent_Graph_Database: A multi-agent conversational financial analytics platform that combines company fundamentals analysis, SEC filing intelligence, and machine learning-based stock price prediction through an intuitive chat interface.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*unPtT4YC0nTUZvdOsoXYiw.png" /></figure><h3>Fetching Data</h3><p>First, we will fetch data, necessary to populate the Graph Database. The data ingestion process is automated by a Python script that systematically gathers both <strong>structured</strong> and <strong>unstructured</strong> information for a predefined list of companies. For structured data, the script leverages the yfinance library to retrieve two key datasets:</p><ul><li>it fetches annual income statements, which are then formatted and saved as <strong>JSON</strong> files, and</li><li>it downloads five years of historical daily stock prices, saving them as <strong>CSV</strong> files.</li><li>for unstructured qualitative data, the script interacts directly with the SEC EDGAR database. Using a company&#39;s unique CIK (Central Index Key) identifier, it makes API calls via the requests library to locate and download the full HTML text of the last five annual 10-K filings.</li></ul><p>This entire workflow iterates through each company listed in a master companies.csv file, methodically populating a local directory structure with the financial, price, and filing data needed for the assistant&#39;s analysis</p><p>I asked Gemini 2.5 Pro to generate this companies.csv example data with 500 examples:</p><pre>ticker,company_name,cik<br>NVDA,NVIDIA CORP,1045810<br>MSFT,MICROSOFT CORP,789019<br>AAPL,Apple Inc.,320193<br>GOOGL,Alphabet Inc.,1652044<br>AMZN,AMAZON COM INC,1018724</pre><p>The script for fetching financial data is this one:</p><pre>import os<br>import requests<br>import pandas as pd<br>import json<br>import time<br>from datetime import datetime<br>from dotenv import load_dotenv<br>from tqdm import tqdm<br>import yfinance as yf <br><br># Load environment variables<br>load_dotenv()<br>SEC_USER_AGENT = os.getenv(&quot;SEC_USER_AGENT&quot;)<br><br># --- Configuration ---<br>COMPANIES_CSV_PATH = &quot;companies.csv&quot;<br>FINANCIALS_DIR = &quot;data/structured/financials&quot;<br>PRICES_DIR = &quot;data/structured/prices&quot;<br>FILINGS_10K_DIR = &quot;data/unstructured/10k&quot;<br><br># Create directories if they don&#39;t exist<br>os.makedirs(FINANCIALS_DIR, exist_ok=True)<br>os.makedirs(PRICES_DIR, exist_ok=True)<br>os.makedirs(FILINGS_10K_DIR, exist_ok=True)<br><br><br>def fetch_financial_statements(ticker: str):<br>    &quot;&quot;&quot;Fetches annual income statements using yfinance.&quot;&quot;&quot;<br>    print(f&quot;Fetching financial statements for {ticker}...&quot;)<br>    try:<br>        stock = yf.Ticker(ticker)<br>        income_stmt = stock.income_stmt<br><br>        if income_stmt.empty:<br>            print(f&quot; -&gt; No financial data found for {ticker}&quot;)<br>            return<br><br>        data = income_stmt.transpose()<br>        data.index.name = &#39;date&#39;<br>        data = data.reset_index()<br>        data[&#39;date&#39;] = data[&#39;date&#39;].astype(str) # Convert timestamp to string<br>        records = data.to_dict(&#39;records&#39;)<br><br>        with open(os.path.join(FINANCIALS_DIR, f&quot;{ticker}_financials.json&quot;), &#39;w&#39;) as f:<br>            json.dump(records, f, indent=4)<br>        print(f&quot; -&gt; Saved financials for {ticker}&quot;)<br>    except Exception as e:<br>        print(f&quot;Error fetching financials for {ticker}: {e}&quot;)<br><br>def fetch_stock_prices(ticker: str):<br>    &quot;&quot;&quot;Fetches the last 5 years of daily stock prices using yfinance.&quot;&quot;&quot;<br>    print(f&quot;Fetching stock prices for {ticker}...&quot;)<br>    try:<br>        stock = yf.Ticker(ticker)<br>        # Get 5 years of historical market data<br>        hist = stock.history(period=&quot;5y&quot;)<br><br>        if hist.empty:<br>            print(f&quot; -&gt; No price data found for {ticker}&quot;)<br>            return<br><br>        hist.to_csv(os.path.join(PRICES_DIR, f&quot;{ticker}_prices.csv&quot;))<br>        print(f&quot; -&gt; Saved prices for {ticker}&quot;)<br>    except Exception as e:<br>        print(f&quot;Error fetching prices for {ticker}: {e}&quot;)<br><br># --- SEC function ---<br><br>def fetch_10k_filings(ticker: str, cik: str):<br>    &quot;&quot;&quot;Fetches the last 5 annual 10-K filings from the SEC EDGAR database.&quot;&quot;&quot;<br>    print(f&quot;Fetching 10-K filings for {ticker} (CIK: {cik})...&quot;)<br>    headers = {&#39;User-Agent&#39;: SEC_USER_AGENT}<br><br>    submissions_url = f&quot;https://data.sec.gov/submissions/CIK{cik.zfill(10)}.json&quot;<br>    try:<br>        response = requests.get(submissions_url, headers=headers)<br>        response.raise_for_status()<br>        submissions = response.json()<br>    except requests.exceptions.RequestException as e:<br>        print(f&quot;Error fetching submission history for {ticker}: {e}&quot;)<br>        return<br><br>    filing_count = 0<br>    recent_filings = submissions[&#39;filings&#39;][&#39;recent&#39;]<br><br>    for i in range(len(recent_filings[&#39;form&#39;])):<br>        if filing_count &gt;= 5:<br>            break<br>        if recent_filings[&#39;form&#39;][i] == &#39;10-K&#39;:<br>            accession_no = recent_filings[&#39;accessionNumber&#39;][i].replace(&#39;-&#39;, &#39;&#39;)<br>            primary_doc_name = recent_filings[&#39;primaryDocument&#39;][i]<br>            filing_date = recent_filings[&#39;filingDate&#39;][i]<br>            year = filing_date.split(&#39;-&#39;)[0]<br><br>            doc_url = f&quot;https://www.sec.gov/Archives/edgar/data/{cik}/{accession_no}/{primary_doc_name}&quot;<br><br>            print(f&quot;  -&gt; Downloading 10-K for {year}...&quot;)<br>            try:<br>                time.sleep(0.2)<br>                doc_response = requests.get(doc_url, headers=headers)<br>                doc_response.raise_for_status()<br><br>                file_path = os.path.join(FILINGS_10K_DIR, f&quot;{ticker}_10K_{year}.html&quot;)<br>                with open(file_path, &#39;w&#39;, encoding=&#39;utf-8&#39;) as f:<br>                    f.write(doc_response.text)<br><br>                filing_count += 1<br>            except requests.exceptions.RequestException as e:<br>                print(f&quot;    Error downloading filing {doc_url}: {e}&quot;)<br><br>    print(f&quot; -&gt; Finished fetching filings for {ticker}&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    companies_df = pd.read_csv(COMPANIES_CSV_PATH)<br><br>    companies_df = companies_df[~companies_df[&#39;ticker&#39;].str.contains(&#39;\.|\-&#39;)]<br><br>    for index, row in tqdm(companies_df.iterrows(), total=companies_df.shape[0], desc=&quot;Processing Companies&quot;):<br>        ticker = row[&#39;ticker&#39;]<br>        cik = str(row[&#39;cik&#39;])<br><br>        # --- Fetch and Save Data ---<br>        fetch_financial_statements(ticker)<br>        fetch_stock_prices(ticker)<br>        fetch_10k_filings(ticker, cik)<br><br>        time.sleep(0.5)<br><br>    print(&quot;\nData fetching complete. Check the &#39;data&#39; directory.&quot;)</pre><p>Once data is gathered, we will populate the Graph database using a <em>populate_graph.py</em> script.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*hVyTaoRpy3zeqc24w_qdBg.png" /></figure><h3>Database Preparation</h3><p>Before any data is loaded, the script performs a critical cleanup step. It runs a Cypher query (MATCH (n) DETACH DELETE n) to completely wipe all existing nodes and relationships from the database. It also attempts to drop any pre-existing vector index named filings. This ensures that each run starts with a clean slate, preventing data duplication or corruption from previous ingestions.</p><h4>Phase 1: Ingesting Structured Data</h4><p>This phase focuses on building the foundational skeleton of the graph with concrete company and financial data.</p><p><strong>Company Node Creation:</strong> The process begins by reading the <em>companies.csv</em> file into a pandas DataFrame. The script then uses a MERGE operation in Cypher to create a Company node for each ticker. MERGE is used instead of CREATE to intelligently create a node only if it doesn’t already exist, preventing duplicates. Each Company node is populated with properties like its <strong>name</strong>, <strong>ticker</strong>, and <strong>CIK</strong> number.</p><p><strong>Financial Node Creation and Linking:</strong> Next, the script iterates through all JSON files in the structured financials directory. For each file, it extracts annual financial data points like <strong>revenue</strong>, <strong>net income</strong>, and <strong>EPS (Earnings Per Share)</strong>. It then creates a distinct Financials node for each year of data. The most crucial step is linking these nodes: a <em>HAS_FINANCIALS relationship</em> is created from the parent Company node to each of its annual Financials nodes. This establishes the first set of connections in our graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*AQX-dzraB7sG40vBVIx1bA.png" /><figcaption>Financials data sample</figcaption></figure><h4>Phase 2: Ingesting Unstructured Data (SEC Filings)</h4><p>This phase enriches the graph with qualitative insights extracted from text-heavy 10-K filings, combining Large Language Model (LLM) intelligence with vector search capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*Uy7Gv06Vo3akxAQg9VguKw.png" /><figcaption>SEC 10-K filing</figcaption></figure><p><strong>LLM-Powered Entity Extraction:</strong> The script first loads the HTML content of 10-K filings from the years 2020 to 2025. For each document, it takes the first 20,000 characters (as a sample), often containing the most critical summaries, and sends them to a Gemini model via a carefully crafted prompt. The prompt instructs the LLM to act as a financial analyst, extracting key entities like <em>key_risks</em>, <em>management_outlook</em>, and <em>major_events</em> and returning them in a structured JSON format.</p><p><strong>Graph Construction:</strong> The structured JSON output from the LLM is used to weave a rich web of new nodes and relationships into the graph.</p><ul><li>A Document node is created for the filing and linked to the corresponding Company with a FILED relationship.</li><li>The extracted entities (risks, events, strategies) are created as their own nodes (e.g., Risk, Event).</li><li>Multiple relationships are formed to show how everything is connected. For example, a Company HAS_RISK to a Risk node, and the Document MENTIONS_RISK to that same Risk node. This creates a detailed and queryable map of qualitative information.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ucll7TrEWmOLmZFoAAA_qg.png" /><figcaption>Node visualization in the Graph Database</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2WuIF-Ds-aH8Hxj5wyNQcA.png" /><figcaption>Graph database zoom</figcaption></figure><h4>Vector Embedding for RAG</h4><p>Finally, to enable semantic search, the script prepares the 10-K filings for <strong>Retrieval-Augmented Generation</strong> (RAG). It truncates each document’s content to the first 80,000 characters. This text is split into smaller, overlapping chunks (1500 characters each). By using a Vertex AI embedding model, each chunk is converted into a numerical vector.</p><p>The <em>langchain_neo4j</em> library then loads these chunks into Neo4j as Chunk nodes, with each node containing the original text and its corresponding vector embedding. A vector index named filings is automatically created on these Chunk nodes, allowing for ultra-fast semantic similarity searches later on.</p><p>See the <em>populate_graph.py</em> code:</p><pre># populate_graph.py<br>import os<br>import pandas as pd<br>import json<br>import math<br>from langchain_community.document_loaders import DirectoryLoader, UnstructuredHTMLLoader<br>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>from langchain_google_vertexai import VertexAI, VertexAIEmbeddings<br>from langchain_neo4j import Neo4jVector<br>from dotenv import load_dotenv<br>from tqdm import tqdm<br>from neo4j import GraphDatabase<br>import re<br><br>load_dotenv()<br>URI = os.getenv(&quot;NEO4J_URI&quot;, &quot;bolt://localhost:7687&quot;)<br>AUTH = (os.getenv(&quot;NEO4J_USERNAME&quot;, &quot;neo4j&quot;), os.getenv(&quot;NEO4J_PASSWORD&quot;, &quot;password&quot;))<br>driver = GraphDatabase.driver(URI, auth=AUTH)<br>llm = VertexAI(model_name=&quot;gemini-2.5-flash&quot;, temperature=0)<br>embeddings = VertexAIEmbeddings(model_name=&quot;text-embedding-005&quot;)<br><br>def ingest_structured_data():<br>    &quot;&quot;&quot;<br>    Loads company profiles from CSV and financial data from JSON files,<br>    then creates Company and Financials nodes in Neo4j.<br>    &quot;&quot;&quot;<br>    print(&quot;Ingesting structured company and financial data...&quot;)<br>    companies_df = pd.read_csv(&#39;./companies.csv&#39;)<br>    company_records = companies_df.to_dict(&#39;records&#39;)<br>    ingest_companies_query = &quot;&quot;&quot;<br>    UNWIND $records AS record<br>    MERGE (c:Company {ticker: record.ticker})<br>    SET c.name = record.company_name, c.cik = toString(record.cik)<br>    &quot;&quot;&quot;<br>    with driver.session() as session:<br>        session.run(ingest_companies_query, records=company_records)<br><br>    financials_dir = &#39;./data/structured/financials&#39;<br>    if os.path.exists(financials_dir):<br>        for filename in tqdm(os.listdir(financials_dir), desc=&quot;Ingesting Financials&quot;):<br>            if filename.endswith(&quot;.json&quot;):<br>                ticker = filename.split(&#39;_&#39;)[0].upper()<br>                with open(os.path.join(financials_dir, filename), &#39;r&#39;) as f:<br>                    financials_data = json.load(f)<br>                records_to_ingest = []<br>                for item in financials_data:<br>                    def get_value(key):<br>                        val = item.get(key)<br>                        if val is None or (isinstance(val, float) and math.isnan(val)):<br>                            return None<br>                        return val<br>                    record = {<br>                        &#39;ticker&#39;: ticker,<br>                        &#39;year&#39;: item.get(&#39;date&#39;, &#39;&#39;).split(&#39;-&#39;)[0],<br>                        &#39;revenue&#39;: get_value(&#39;Total Revenue&#39;),<br>                        &#39;netIncome&#39;: get_value(&#39;Net Income&#39;),<br>                        &#39;eps&#39;: get_value(&#39;Basic EPS&#39;) or get_value(&#39;Diluted EPS&#39;)<br>                    }<br>                    if record[&#39;year&#39;]:<br>                        records_to_ingest.append(record)<br>                ingest_financials_query = &quot;&quot;&quot;<br>                UNWIND $records AS record<br>                MATCH (c:Company {ticker: record.ticker})<br>                MERGE (f:Financials {company: c.ticker, year: record.year})<br>                SET f.revenue = toFloat(record.revenue), f.netIncome = toFloat(record.netIncome), f.eps = toFloat(record.eps)<br>                MERGE (c)-[:HAS_FINANCIALS]-&gt;(f)<br>                &quot;&quot;&quot;<br>                with driver.session() as session:<br>                    session.run(ingest_financials_query, records=records_to_ingest)<br>    else:<br>        print(f&quot;Warning: Financials directory {financials_dir} not found&quot;)<br>    print(&quot;Structured data ingestion complete.&quot;)<br><br>def extract_entities_from_filing(doc):<br>    &quot;&quot;&quot;<br>    Uses an LLM to extract structured entities from the first 20,000 characters of a 10-K filing.<br>    &quot;&quot;&quot;<br>    filename = os.path.basename(doc.metadata.get(&#39;source&#39;, &#39;&#39;))<br>    match = re.search(r&quot;([A-Z]+)_10K_(\d{4})&quot;, filename)<br>    if not match:<br>        print(f&quot;Warning: Could not extract ticker and year from filename: {filename}&quot;)<br>        return None<br>    ticker, year = match.groups()<br>    extraction_prompt = f&quot;&quot;&quot;<br>    From the SEC 10-K filing document below for ticker {ticker} and year {year}, extract the following information.<br>    Focus on the &quot;Risk Factors&quot; and &quot;Management&#39;s Discussion and Analysis&quot; sections if possible.<br>    - key_risks: A list of the 3-5 most significant risks mentioned.<br>    - management_outlook: A concise, one-paragraph summary of management&#39;s outlook.<br>    - major_events: A list of 1-3 major events from that year.<br>    - strategic_focus: A list of key strategic areas mentioned.<br>    Return the information as a valid JSON object with these exact keys. If any information is not found, use an empty list or null.<br>    Do not include any other text, explanation, or markdown formatting.<br>    DOCUMENT (first 20000 characters):<br>    {doc.page_content[:20000]}<br>    &quot;&quot;&quot;<br>    try:<br>        response = llm.invoke(extraction_prompt)<br>        cleaned_response = response.strip().replace(&quot;```json&quot;, &quot;&quot;).replace(&quot;```&quot;, &quot;&quot;).strip()<br>        entities = json.loads(cleaned_response)<br>        entities[&#39;ticker&#39;] = ticker<br>        entities[&#39;year&#39;] = year<br>        return entities<br>    except (json.JSONDecodeError, Exception) as e:<br>        print(f&quot;Error processing document {doc.metadata.get(&#39;source&#39;, &#39;Unknown&#39;)}: {e}&quot;)<br>        print(f&quot;LLM Response was: {response}&quot;)<br>        return None<br><br>def ingest_unstructured_data():<br>    &quot;&quot;&quot;<br>    MODIFIED:<br>    - Extracts entities using the first 20,000 characters.<br>    - Chunks and creates vector embeddings for the first 80,000 characters.<br>    &quot;&quot;&quot;<br>    print(&quot;Ingesting data from 10-K filings (2020-2025)...&quot;)<br>    filings_dir = &#39;./data/unstructured/10k/&#39;<br>    if not os.path.exists(filings_dir):<br>        print(f&quot;Warning: Filings directory {filings_dir} not found. Skipping unstructured data ingestion.&quot;)<br>        return<br>    <br>    loader = DirectoryLoader(<br>        filings_dir, glob=&quot;**/*.html&quot;, loader_cls=UnstructuredHTMLLoader,<br>        show_progress=True, loader_kwargs={&quot;unstructured_kwargs&quot;: {&quot;strategy&quot;: &quot;fast&quot;}}, silent_errors=True<br>    )<br>    documents = loader.load()<br>    if not documents:<br>        print(&quot;No documents found. Skipping unstructured data ingestion.&quot;)<br>        return<br>    <br>    target_years = [str(y) for y in range(2020, 2026)]<br>    docs_to_process = []<br>    for doc in documents:<br>        filename = os.path.basename(doc.metadata.get(&#39;source&#39;, &#39;&#39;))<br>        if any(year in filename for year in target_years):<br>            docs_to_process.append(doc)<br>    if not docs_to_process:<br>        print(&quot;No documents found for target years. Skipping unstructured data ingestion.&quot;)<br>        return<br>    <br>    print(f&quot;Loaded {len(docs_to_process)} documents for years {target_years[0]}-{target_years[-1]}&quot;)<br>    print(&quot;Extracting and linking entities from filings...&quot;)<br>    with driver.session() as session:<br>        for doc in tqdm(docs_to_process, desc=&quot;Processing Filings&quot;):<br>            entities = extract_entities_from_filing(doc)<br>            if entities and entities.get(&#39;ticker&#39;):<br>                # Cypher query for linking entities <br>                link_query = &quot;&quot;&quot;<br>                MATCH (c:Company {ticker: $ticker}) MERGE (doc:Document {source: $source})<br>                ON CREATE SET doc.year = $year, doc.type = &#39;10-K&#39; MERGE (c)-[:FILED]-&gt;(doc)<br>                SET doc.management_outlook = $management_outlook<br>                WITH c, doc UNWIND $key_risks AS risk_name WHERE risk_name IS NOT NULL AND risk_name &lt;&gt; &quot;&quot;<br>                MERGE (r:Risk {name: risk_name}) MERGE (c)-[:HAS_RISK]-&gt;(r) MERGE (doc)-[:MENTIONS_RISK]-&gt;(r)<br>                WITH c, doc UNWIND $major_events AS event_name WHERE event_name IS NOT NULL AND event_name &lt;&gt; &quot;&quot;<br>                MERGE (e:Event {name: event_name}) MERGE (c)-[:HAD_EVENT]-&gt;(e) MERGE (doc)-[:DESCRIBES_EVENT]-&gt;(e)<br>                WITH c, doc UNWIND $strategic_focus AS strategy_name WHERE strategy_name IS NOT NULL AND strategy_name &lt;&gt; &quot;&quot;<br>                MERGE (s:Strategy {name: strategy_name}) MERGE (c)-[:HAS_STRATEGY]-&gt;(s) MERGE (doc)-[:MENTIONS_STRATEGY]-&gt;(s)<br>                &quot;&quot;&quot;<br>                params = {<br>                    &quot;source&quot;: os.path.basename(doc.metadata.get(&#39;source&#39;)), &quot;ticker&quot;: entities.get(&#39;ticker&#39;),<br>                    &quot;year&quot;: entities.get(&#39;year&#39;), &quot;management_outlook&quot;: entities.get(&#39;management_outlook&#39;),<br>                    &quot;key_risks&quot;: entities.get(&#39;key_risks&#39;, []), &quot;major_events&quot;: entities.get(&#39;major_events&#39;, []),<br>                    &quot;strategic_focus&quot;: entities.get(&#39;strategic_focus&#39;, [])<br>                }<br>                try:<br>                    session.run(link_query, params)<br>                except Exception as e:<br>                    print(f&quot;Error executing link query for {entities.get(&#39;ticker&#39;)}: {e}&quot;)<br><br>    print(&quot;Splitting documents and creating vector embeddings (first 80,000 chars)...&quot;)<br>    <br>    docs_to_embed = []<br>    for doc in docs_to_process:<br>        truncated_doc = doc.copy()<br>        truncated_doc.page_content = doc.page_content[:80000] # Slice to 80,000<br>        docs_to_embed.append(truncated_doc)<br><br>    text_splitter = RecursiveCharacterTextSplitter(chunk_size=1500, chunk_overlap=150)<br>    docs_for_vector = text_splitter.split_documents(docs_to_embed)<br>    <br>    try:<br>        Neo4jVector.from_documents(<br>            docs_for_vector, embeddings, url=URI, username=AUTH[0], password=AUTH[1],<br>            database=&quot;neo4j&quot;, index_name=&quot;filings&quot;, node_label=&quot;Chunk&quot;,<br>            text_node_property=&quot;text&quot;, embedding_node_property=&quot;embedding&quot;, create_id_index=True<br>        )<br>        print(&quot;Unstructured data ingestion and vector indexing complete.&quot;)<br>    except Exception as e:<br>        print(f&quot;Error creating vector index: {e}&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    print(&quot;Clearing database...&quot;)<br>    with driver.session() as session:<br>        session.run(&quot;MATCH (n) DETACH DELETE n&quot;)<br>        try:<br>            session.run(&quot;CALL db.index.vector.drop(&#39;filings&#39;)&quot;)<br>            print(&quot;Dropped existing vector index.&quot;)<br>        except Exception as e:<br>            print(f&quot;No existing vector index to drop or error: {e}&quot;)<br>    ingest_structured_data()<br>    ingest_unstructured_data()<br>    print(&quot;\nDatabase population finished.&quot;)<br>    driver.close()</pre><p>Once the database is populated, we will develop the agents. As we are using Google ADK, a single script handles all the agent’s tasks. The agentic system follows a sophisticated Root (frontend agent)/Sub-Agent (backend agent) architecture. This design mimics a team of financial analysts, with a lead analyst (the Root Agent) who intelligently delegates tasks to specialized analysts (the 3 Sub-Agents with different tools).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iCXEyX233bBcXN9LofMyLg.png" /><figcaption>agents</figcaption></figure><h3>The Agent Toolkit</h3><p>Before defining the agents, we first create the core functions they will use to interact with the data and models. These are the tools (functions) that give the agents their capabilities.</p><p><strong>query_graph_database:</strong> This tool is designed for structured data queries. When a user asks a question like “What was the revenue for NVDA in 2024?”, this function uses a Gemini LLM to dynamically write a Cypher query. It’s guided by a detailed prompt that includes the database schema and examples of correct queries. The generated Cypher is then executed against the Neo4j graph to fetch precise, factual answers. Here, prompt engineering is the key to success.</p><p><strong>retrieve_from_documents:</strong> This tool handles qualitative questions by performing Retrieval-Augmented Generation (RAG). It follows a two-step process:</p><ul><li><strong>Retrieve:</strong> It converts the user’s question into a vector embedding and uses it to perform a similarity search on the vector index in Neo4j. This retrieves the most relevant text chunks from the 10-K filings.</li><li><strong>Synthesize:</strong> These retrieved text chunks are combined with the original question in a new prompt to the LLM, which then synthesizes a comprehensive, human-readable answer based on the provided context.</li></ul><p><strong>predict_stock_price_tool:</strong> This is a straightforward tool that acts as a wrapper for a pre-trained machine learning model. It takes a single stock ticker, validates that it’s one of the available companies, and calls the <em>predict_next_day_price</em> function to get a next-day price prediction.</p><h4>The Specialist Sub-Agents</h4><p>With the tools defined, we create three distinct sub-agents, each with a specific role. Each agent is given a name, a tool, and a set of instructions that define its expertise.</p><p><strong>Graph QA Agent:</strong> This is the quantitative analyst. It’s equipped exclusively with the <em>query_graph_database</em> tool. Its instructions tell it to handle questions about specific financial numbers (revenue, net income), company risks, events, and any query that requires pulling structured data from the knowledge graph.</p><p><strong>Document RAG Agent:</strong> This is the qualitative researcher. It uses the <em>retrieve_from_documents</em> tool to answer questions that require understanding context and nuance, such as summarizing management’s outlook, explaining business strategies, or detailing risks mentioned in SEC filings.</p><p><strong>Stock Predictor Agent:</strong> This is the forecaster. Its sole purpose is to use the <em>predict_stock_price_tool</em>. Its instructions are very strict: only activate for explicit requests to predict a stock price and always include a disclaimer that the prediction is not financial advice.</p><h4>The Root Agent: Orchestrator</h4><p>The root_agent acts as the team lead or the “brain” of the operation. It does not have any tools of its own. Instead, its “tools” are the three sub-agents. Its primary job is to perform intent recognition and delegation. Based on its detailed instructions, the Root Agent analyzes the incoming user query and determines which specialist is best suited for the job (the dynamic part of the system).</p><p>If the query asks for a specific number like “net income,” it delegates to the Graph QA Agent. If the query asks for a summary like “What did Apple say about AI?”, it delegates to the Document RAG Agent. If the query explicitly asks for a “prediction,” it delegates to the Stock Predictor Agent.</p><p>This layered, multi-agent approach makes the system modular, scalable, and highly effective at routing complex financial questions to the correct “expert” for a precise and relevant answer.</p><p>See the script for <em>agents.py</em>:</p><pre># agents.py<br>&quot;&quot;&quot;<br>Defines the ADK agent team for the financial data application.<br>This includes a root agent for orchestration and specialized sub-agents<br>for graph querying, document retrieval, and stock price predictions.<br>&quot;&quot;&quot;<br>from google.adk.agents import Agent<br>from google.adk.models.lite_llm import LiteLlm<br>from ..neo4j_for_adk import graphdb<br>from app.models.predict import predict_next_day_price<br>from langchain_google_vertexai import VertexAIEmbeddings<br><br># --- Setup ---<br>llm = LiteLlm(model=&quot;gemini-2.5-flash&quot;) # cheap and good model<br><br>embeddings = VertexAIEmbeddings(model_name=&quot;text-embedding-005&quot;)<br><br># --- Tool Definitions ---<br>def query_graph_database(question: str) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Generates a Cypher query for the financial graph and executes it.<br>    &quot;&quot;&quot;<br>    schema = graphdb.send_query(&quot;CALL db.schema.visualization()&quot;)[&quot;query_result&quot;]<br>    <br>    cypher_generation_prompt = f&quot;&quot;&quot;<br>    Task: Generate a Cypher statement to query a financial graph database.<br>    <br>    Schema: {schema}<br>    <br>    Instructions:<br>    - Use ONLY the provided relationship types and property keys.<br>    - The graph contains the following nodes and relationships:<br>      - (c:Company)-[:HAS_FINANCIALS]-&gt;(f:Financials)<br>      - (c:Company)-[:FILED]-&gt;(doc:Document)<br>      - (c:Company)-[:HAS_RISK]-&gt;(r:Risk)<br>      - (c:Company)-[:HAD_EVENT]-&gt;(e:Event)<br>      - (c:Company)-[:HAS_STRATEGY]-&gt;(s:Strategy)<br>      - (doc:Document)-[:MENTIONS_RISK]-&gt;(r:Risk)<br>      - (doc:Document)-[:DESCRIBES_EVENT]-&gt;(e:Event)<br>      - (doc:Document)-[:MENTIONS_STRATEGY]-&gt;(s:Strategy)<br>      - (chunk:Chunk) nodes with vector embeddings for document chunks<br>    <br>    - Key properties for nodes:<br>      - Company: `ticker` (e.g., &#39;NVDA&#39;), `name`, `cik`<br>      - Financials: `company` (ticker), `year` (string like &#39;2024&#39;), `revenue`, `netIncome`, `eps`<br>      - Risk, Event, Strategy: `name`<br>      - Document: `source` (filename), `year`, `type`, `management_outlook`<br>      - Chunk: `text`, `embedding` (vector)<br>    <br>    - IMPORTANT: The Financials node uses `company` property (not ticker directly) and `year` is a STRING<br>    - Company tickers in your data: NVDA, MSFT, AAPL, GOOGL, AMZN<br>    <br>    Example Questions &amp; Queries (ticker and year are database property names, not variables):<br>    - Question: &quot;What was the revenue for NVDA in 2024?&quot;<br>      Query: MATCH (c:Company {{ticker: &#39;NVDA&#39;}})-[:HAS_FINANCIALS]-&gt;(f:Financials {{year: &#39;2024&#39;}}) RETURN f.revenue<br>    - Question: &quot;What are the key risks for NVDA?&quot;<br>      Query: MATCH (c:Company {{ticker: &#39;NVDA&#39;}})-[:HAS_RISK]-&gt;(r:Risk) RETURN r.name<br>    - Question: &quot;Show me financial trends for NVDA over the years&quot;<br>      Query: MATCH (c:Company {{ticker: &#39;NVDA&#39;}})-[:HAS_FINANCIALS]-&gt;(f:Financials) RETURN f.year, f.revenue, f.netIncome, f.eps ORDER BY f.year<br>    - Question: &quot;What events happened at Apple?&quot;<br>      Query: MATCH (c:Company {{ticker: &#39;AAPL&#39;}})-[:HAD_EVENT]-&gt;(e:Event) RETURN e.name<br>    <br>    Question: {question}<br>    Return only the Cypher query, no explanation or formatting.<br>    &quot;&quot;&quot;<br>    <br>    cypher_query = llm.llm_client.completion(<br>        model=llm.model,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: cypher_generation_prompt}],<br>        tools=[], # &lt;-- ADD THIS LINE<br>    ).choices[0].message.content.strip()<br>    <br>    cypher_query = cypher_query.replace(&quot;```cypher&quot;, &quot;&quot;).replace(&quot;```&quot;, &quot;&quot;).strip()<br>    print(f&quot;Generated Cypher: {cypher_query}&quot;)<br>    <br>    return graphdb.send_query(cypher_query)<br><br>def retrieve_from_documents(question: str) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Performs vector search on 10-K filing chunks and synthesizes an answer.<br>    &quot;&quot;&quot;<br>    question_embedding = embeddings.embed_query(question)<br>    <br>    search_query = &quot;&quot;&quot;<br>    CALL db.index.vector.queryNodes(&#39;filings&#39;, 5, $embedding) YIELD node, score<br>    RETURN node.text AS text, score<br>    ORDER BY score DESC<br>    &quot;&quot;&quot;<br>    <br>    search_results = graphdb.send_query(search_query, {&quot;embedding&quot;: question_embedding})<br>    <br>    if search_results[&#39;status&#39;] == &#39;error&#39; or not search_results[&#39;query_result&#39;]:<br>        return {&quot;answer&quot;: &quot;Could not retrieve relevant documents from filings.&quot;, &quot;error&quot;: search_results.get(&#39;message&#39;, &#39;Unknown error&#39;)}<br>    <br>    context = &quot;\n&quot;.join([r[&#39;text&#39;] for r in search_results[&#39;query_result&#39;]])<br>    <br>    synthesis_prompt = f&quot;&quot;&quot;<br>    Based on the following context from SEC 10-K filings, answer the question comprehensively.<br>    <br>    Context from filings:<br>    {context}<br>    <br>    Question: {question}<br>    <br>    Instructions:<br>    - Provide a detailed answer based on the context<br>    - If the context doesn&#39;t contain relevant information, say so<br>    - Cite specific information from the filings when possible<br>    - Focus on the financial and strategic aspects mentioned<br>    <br>    Answer:<br>    &quot;&quot;&quot;<br>    <br>    response = llm.llm_client.completion(<br>        model=llm.model,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: synthesis_prompt}],<br>        tools=[], <br>    ).choices[0].message.content<br>    <br>    return {&quot;answer&quot;: response}<br><br>def predict_stock_price_tool(ticker: str) -&gt; dict:<br>    &quot;&quot;&quot;<br>    A wrapper for the stock price prediction model.<br>    Input must be a single, valid stock ticker string from our available companies.<br>    &quot;&quot;&quot;<br>    valid_tickers = {&#39;NVDA&#39;, &#39;MSFT&#39;, &#39;AAPL&#39;, &#39;GOOGL&#39;, &#39;AMZN&#39;}<br>    <br>    if not isinstance(ticker, str):<br>        return {&quot;error&quot;: f&quot;Invalid input type. Please provide a ticker as a string.&quot;}<br>    <br>    ticker = ticker.upper().strip()<br>    <br>    if ticker not in valid_tickers:<br>        return {&quot;error&quot;: f&quot;Ticker &#39;{ticker}&#39; not found. Available tickers: {&#39;, &#39;.join(valid_tickers)}&quot;}<br>    <br>    print(f&quot;Predicting price for ticker: {ticker}&quot;)<br>    return predict_next_day_price(ticker)<br><br># --- Sub-Agent Definitions ---<br>graph_qa_subagent = Agent(<br>    name=&quot;GraphQA_Agent&quot;,<br>    model=llm,<br>    tools=[query_graph_database],<br>    description=&quot;Use for questions about company financials (revenue, net income, EPS), risks, events, strategies, and any structured data queries. Works with tickers: NVDA, MSFT, AAPL, GOOGL, AMZN.&quot;,<br>    instruction=&quot;&quot;&quot;<br>    Your task is to use the `query_graph_database` tool to answer questions about:<br>    - Financial metrics (revenue, net income, EPS) by company and year<br>    - Company risks, events, and strategic focuses<br>    - Comparisons between companies<br>    - Financial trends over time<br>    <br>    Always use the exact ticker symbols: NVDA, MSFT, AAPL, GOOGL, AMZN<br>    Remember that years are stored as strings (e.g., &#39;2024&#39;, &#39;2023&#39;).<br>    &quot;&quot;&quot;<br>)<br><br>document_rag_subagent = Agent(<br>    name=&quot;DocumentRAG_Agent&quot;,<br>    model=llm,<br>    tools=[retrieve_from_documents],<br>    description=&quot;Use for qualitative questions about company strategy, management outlook, detailed business descriptions, or any information that requires reading through SEC 10-K filing text.&quot;,<br>    instruction=&quot;&quot;&quot;<br>    Your task is to use the `retrieve_from_documents` tool to find detailed, qualitative information from SEC filings including:<br>    - Management&#39;s discussion and analysis<br>    - Business strategy and outlook<br>    - Detailed risk descriptions<br>    - Product and service descriptions<br>    - Market analysis and competitive positioning<br>    <br>    Provide comprehensive answers based on the retrieved document chunks.<br>    &quot;&quot;&quot;<br>)<br><br>prediction_subagent = Agent(<br>    name=&quot;StockPricePredictor_Agent&quot;,<br>    model=llm,<br>    tools=[predict_stock_price_tool],<br>    description=&quot;Use ONLY to predict the next day&#39;s closing stock price. Works with tickers: NVDA, MSFT, AAPL, GOOGL, AMZN.&quot;,<br>    instruction=&quot;&quot;&quot;<br>    Your only task is to use the `predict_stock_price_tool` for stock price predictions.<br>    <br>    IMPORTANT:<br>    - Only valid tickers: NVDA, MSFT, AAPL, GOOGL, AMZN<br>    - Input must be a single ticker string<br>    - Always include a disclaimer that predictions are estimates based on historical data and not financial advice<br>    &quot;&quot;&quot;<br>)<br><br># --- Root Agent Definition ---<br>root_agent = Agent(<br>    name=&quot;Financial_Root_Agent&quot;,<br>    model=llm,<br>    sub_agents=[graph_qa_subagent, document_rag_subagent, prediction_subagent],<br>    description=&quot;The main financial assistant that analyzes user queries and delegates to specialized agents for financial data analysis.&quot;,<br>    instruction=&quot;&quot;&quot;<br>    You are a knowledgeable financial data assistant with access to data for these companies: NVDA, MSFT, AAPL, GOOGL, AMZN.<br>    <br>    DELEGATION GUIDELINES:<br>    - Use &#39;GraphQA_Agent&#39; for:<br>      * Specific financial numbers (revenue, net income, EPS)<br>      * Company risks, events, strategies (structured data)<br>      * Financial comparisons and trends<br>      * Any query requiring precise data extraction<br>    <br>    - Use &#39;DocumentRAG_Agent&#39; for:<br>      * Qualitative analysis and detailed explanations<br>      * Management outlook and business strategy discussions<br>      * Complex business descriptions<br>      * Questions requiring reading through filing narratives<br>    <br>    - Use &#39;StockPricePredictor_Agent&#39; ONLY for:<br>      * Explicit requests to predict future stock prices<br>      * Must use valid tickers: NVDA, MSFT, AAPL, GOOGL, AMZN<br>    <br>    IMPORTANT NOTES:<br>    - Available companies: NVIDIA (NVDA), Microsoft (MSFT), Apple (AAPL), Alphabet/Google (GOOGL), Amazon (AMZN)<br>    - Financial data years: 2021-2024<br>    - Always include disclaimers for predictions<br>    - If uncertain about which agent to use, explain your reasoning<br>    &quot;&quot;&quot;<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V1Jtfs9-l9rwBcTJt6xUNA.png" /></figure><p>Finally, we train the autoregressive predictive model for each stock. This process creates a custom machine learning model for every company that can be used by the Stock Predictor Agent.</p><h4>Feature Engineering</h4><p>To prepare the data for training, the script first performs feature engineering to create a set of predictive inputs from the raw historical data. This autoregressive model uses past values to predict future ones. The key features created include:</p><h4>Lag Features: The closing prices of the last 10 days.</h4><p>Rolling Window Features: 5-day and 20-day moving averages for both price and volume to capture recent trends.</p><h4>Volume Features: The previous day’s trading volume.</h4><p>The model’s target is to predict the stock’s closing price one day into the future.</p><h4>Model Training and Saving</h4><p>The script then iterates through each company’s price data. For each stock, it applies the feature engineering process and then trains a LightGBM Regressor model on the company’s entire historical dataset. Using the full history allows the model to make the most informed prediction possible for the next day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*yg-X3leVPPxzAPPoCF90yg.png" /><figcaption>Prices data</figcaption></figure><p>After a model is trained for a specific stock, two files are saved: the trained model object itself and a separate file containing the list of feature names the model expects. This ensures that the prediction tool can consistently provide the correct input structure. This loop repeats until a unique, serialized model exists for every stock. In order to achieve better results for specific stocks, use Optuna parameter optimization.</p><pre># train_predictor.py<br><br>import pandas as pd<br>import numpy as np<br>import joblib<br>import os<br>import lightgbm as lgb<br>from tqdm import tqdm<br><br># --- Configuration ---<br>PRICES_DIR = &quot;./data/structured/prices&quot;<br>MODEL_DIR = &quot;./app/models/saved_models&quot; # Matching your project structure<br>os.makedirs(MODEL_DIR, exist_ok=True)<br><br># --- Feature Engineering Parameters ---<br>WINDOW_SIZE = 10         <br>PREDICTION_HORIZON = 1   <br><br>def create_features(df):<br>    &quot;&quot;&quot;Creates time-series features from a stock price DataFrame.&quot;&quot;&quot;<br>    # Create a new DataFrame for features to avoid modifying the original<br>    featured_df = df[[&#39;Close&#39;, &#39;Volume&#39;]].copy()<br><br>    # 1. Lag Features (autoregressive part)<br>    for i in range(1, WINDOW_SIZE + 1):<br>        featured_df[f&#39;Close_lag_{i}&#39;] = featured_df[&#39;Close&#39;].shift(i)<br><br>    # 2. Rolling Window Features<br>    featured_df[&#39;MA_5&#39;] = featured_df[&#39;Close&#39;].rolling(window=5).mean()<br>    featured_df[&#39;MA_20&#39;] = featured_df[&#39;Close&#39;].rolling(window=20).mean()<br><br>    # 3. Volume-based Features<br>    featured_df[&#39;Volume_lag_1&#39;] = featured_df[&#39;Volume&#39;].shift(1)<br>    featured_df[&#39;Volume_MA_5&#39;] = featured_df[&#39;Volume&#39;].rolling(window=5).mean()<br><br>    # 4. Create the target variable<br>    featured_df[&#39;target&#39;] = featured_df[&#39;Close&#39;].shift(-PREDICTION_HORIZON)<br><br>    featured_df.dropna(inplace=True)<br><br>    return featured_df<br><br>if __name__ == &quot;__main__&quot;:<br>    price_files = [f for f in os.listdir(PRICES_DIR) if f.endswith(&#39;_prices.csv&#39;)]<br><br>    for file in tqdm(price_files, desc=&quot;Training Models for each stock&quot;):<br>        ticker = file.split(&#39;_&#39;)[0]<br><br>        # Load data<br>        df = pd.read_csv(os.path.join(PRICES_DIR, file))<br>        df[&#39;Date&#39;] = pd.to_datetime(df[&#39;Date&#39;])<br>        df.set_index(&#39;Date&#39;, inplace=True)<br>        df.sort_index(inplace=True)<br><br>        # Create features<br>        data = create_features(df)<br><br>        if data.empty:<br>            print(f&quot;Skipping {ticker}: Not enough data to create features.&quot;)<br>            continue<br><br>        # Define features (X) and target (y)<br>        X = data.drop(columns=[&#39;target&#39;, &#39;Close&#39;, &#39;Volume&#39;])<br>        y = data[&#39;target&#39;]<br><br>        # Train the model<br>        print(f&quot;\nTraining model for {ticker} with {len(X.columns)} features...&quot;)<br><br>        model = lgb.LGBMRegressor(<br>            random_state=42,<br>            n_estimators=200,      # More estimators for better performance<br>            learning_rate=0.05,<br>            num_leaves=31<br>        )<br>        model.fit(X, y)<br><br>        # Save the trained model and the list of features it expects<br>        joblib.dump(model, os.path.join(MODEL_DIR, f&quot;{ticker}_price_regressor.joblib&quot;))<br>        joblib.dump(X.columns.tolist(), os.path.join(MODEL_DIR, f&quot;{ticker}_features.joblib&quot;))<br><br>        print(f&quot; Model for {ticker} saved.&quot;)<br><br>    print(&quot;\nTraining complete! All models saved.&quot;)</pre><p>… and also create the <em>predict.py</em> script, to be run when someone asks for a specific stock price prediction in the chat interface:</p><pre># app/models/predict.py<br><br>import pandas as pd<br>import numpy as np<br>import joblib<br>import os<br>from pathlib import Path<br><br># --- Configuration ---<br>MODEL_DIR = Path(__file__).resolve().parent / &quot;saved_models&quot;<br>PRICES_DIR = Path(__file__).resolve().parent.parent.parent / &quot;data/structured/prices&quot;<br><br>def predict_next_day_price(ticker: str) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Predicts the next day&#39;s closing price for a given stock ticker.<br><br>    Args:<br>        ticker: The stock ticker (e.g., &#39;AAPL&#39;).<br><br>    Returns:<br>        A dictionary with the predicted price or an error message.<br>    &quot;&quot;&quot;<br>    try:<br>        # Load the trained model and its required features<br>        model = joblib.load(MODEL_DIR / f&quot;{ticker}_price_regressor.joblib&quot;)<br>        features_list = joblib.load(MODEL_DIR / f&quot;{ticker}_features.joblib&quot;)<br><br>        # Load the latest historical data for the ticker<br>        df = pd.read_csv(PRICES_DIR / f&quot;{ticker}_prices.csv&quot;)<br>        df[&#39;Date&#39;] = pd.to_datetime(df[&#39;Date&#39;])<br>        df.set_index(&#39;Date&#39;, inplace=True)<br>        df.sort_index(inplace=True)<br><br>        # Take a slice of the last ~30 days to ensure rolling windows can be calculated<br>        latest_data = df.tail(30).copy()<br><br>        # 1. Lag Features<br>        for i in range(1, 11): # WINDOW_SIZE is 10<br>            latest_data[f&#39;Close_lag_{i}&#39;] = latest_data[&#39;Close&#39;].shift(i)<br><br>        # 2. Rolling Window Features<br>        latest_data[&#39;MA_5&#39;] = latest_data[&#39;Close&#39;].rolling(window=5).mean()<br>        latest_data[&#39;MA_20&#39;] = latest_data[&#39;Close&#39;].rolling(window=20).mean()<br><br>        # 3. Volume-based Features<br>        latest_data[&#39;Volume_lag_1&#39;] = latest_data[&#39;Volume&#39;].shift(1)<br>        latest_data[&#39;Volume_MA_5&#39;] = latest_data[&#39;Volume&#39;].rolling(window=5).mean()<br><br>        prediction_features = latest_data.tail(1)<br><br>        prediction_features = prediction_features[features_list]<br><br>        predicted_price = model.predict(prediction_features)[0]<br><br>        return {<br>            &quot;ticker&quot;: ticker,<br>            &quot;predicted_next_day_close&quot;: round(float(predicted_price), 2)<br>        }<br><br>    except FileNotFoundError:<br>        return {&quot;error&quot;: f&quot;Model or data for ticker &#39;{ticker}&#39; not found. Please ensure it has been trained.&quot;}<br>    except Exception as e:<br>        return {&quot;error&quot;: f&quot;An error occurred during prediction for {ticker}: {e}&quot;}<br><br>if __name__ == &#39;__main__&#39;:<br>    sample_ticker = &#39;AAPL&#39;<br>    prediction = predict_next_day_price(sample_ticker)<br><br>    if &quot;error&quot; in prediction:<br>        print(f&quot;Error: {prediction[&#39;error&#39;]}&quot;)<br>    else:<br>        print(f&quot;Prediction for {prediction[&#39;ticker&#39;]}:&quot;)<br>        print(f&quot;  Predicted Close Price for Tomorrow: ${prediction[&#39;predicted_next_day_close&#39;]}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wlpjtE_DzVfZwjUYvvXN8A.png" /></figure><p>Now we can run our project using uvicorn:</p><pre>uvicorn app.main:app --reload --port 8080</pre><p>The app will run at <a href="http://127.0.0.1:8080/"><em>http://127.0.0.1:8080</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X9mJGb4B_qm_jE-NDnycPA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*_Tt6NtTux_AldivenLNj8g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*24Nsk59zSZTbMQ1jcm6Q7A.png" /></figure><p>For deployment in Cloud Run:</p><pre>gcloud auth login<br>gcloud config set project YOUR-PROJECT<br><br>gcloud artifacts repositories create financial-assistant-repo \<br>    --repository-format=docker \<br>    --location=us-central1 \<br>    --description=&quot;Docker repository for financial assistant service&quot;<br><br>gcloud builds submit --tag us-central1-docker.pkg.dev/YOUR-PROJECT/financial-assistant-repo/assistant-service:latest<br><br>gcloud run deploy financial-assistant-service \<br>    --image=us-central1-docker.pkg.dev/YOUR-PROJECT/financial-assistant-repo/assistant-service:latest \<br>    --platform=managed \<br>    --region=us-central1 \<br>    --allow-unauthenticated \<br>    --set-env-vars-from-file=.env \<br>    --min-instances 0 \<br>    --max-instances 3 \<br>    --cpu 4 \<br>    --memory 8192Mi \<br>    --concurrency 10</pre><p><a href="https://medium.com/@rubenszimbres/agent-based-modeling-with-python-and-netlogo-6f7bf4103f">This article I wrote some years ago</a> implements an ABM system using NetLogo (written in Scala and Java), getting input from environment via Raspberry sensors to simulate agents in a social network.</p><p>Clap ➕ if you liked ☺️☺️☺️</p><h4>Acknowledgements</h4><p>✨ <em>Google ML Developer Programs and Google Developers Program supported this work by providing Google Cloud Credits (and awesome tutorials for the Google Developer Experts)</em>✨</p><p>🔗 <a href="https://developers.google.com/machine-learning">https://developers.google.com/machine-learning</a> 🔗</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=05ded38bcfab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating a Binary Watch from Scratch with LILYGO Programmable Device]]></title>
            <link>https://medium.com/@rubenszimbres/creating-a-binary-watch-from-scratch-with-lilygo-programmable-device-856c063b4605?source=rss-30f09d1741ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/856c063b4605</guid>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[watches]]></category>
            <category><![CDATA[arduino]]></category>
            <category><![CDATA[embedded-systems]]></category>
            <category><![CDATA[lilygo]]></category>
            <dc:creator><![CDATA[Rubens Zimbres]]></dc:creator>
            <pubDate>Fri, 20 Jun 2025 13:26:51 GMT</pubDate>
            <atom:updated>2025-06-24T16:58:01.480Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*23eUmTL1f21721CKyDttzQ.png" /></figure><p>I bought a LYLIGO T-Watch-2020 V3 almost 4 years ago but hadn’t played with it yet. This watch is a programmable device that comes with a USB connection and you can burn anything into it. It has a rechargeable battery, it is very simple but it is not waterproof. It is an ESP32-based smartwatch designed by Shenzhen Xinyuan Electronics Co., Ltd.</p><p>Lately, a friend of mine bought a binary watch and I said to myself: “Wow, maybe finally I have an interesting project for my programmable watch!”. This article is a step by step tutorial on how to get it done. The LILYGO T-Watch can be obtained in <a href="https://lilygo.cc/products/t-watch-2020-v3">LILYGO website</a> for $ 36.55.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*BC3_zGkQV-ZQixyqWLQXXQ.png" /><figcaption>Original watch box</figcaption></figure><p>The code provided here creates a binary watch application for the LILYGO T-Watch that displays time using the same binary number system computers use internally. Instead of showing traditional numbers like “12:30”, it represents each digit using patterns of green and gray dots, where green means “1” and gray means “0”. This serves as both a functional timepiece and an excellent way to learn how computers actually store numbers behind the scenes.</p><p>The program’s main loop continuously updates the binary time display while managing power consumption to preserve battery life. After six seconds of inactivity, the watch automatically goes to sleep by turning off the display, but keeps the power management chip active so it can instantly wake up when you press the button. The display also shows a real-time battery indicator that changes color from green to yellow to red as the battery level decreases.</p><p>The most technically interesting feature is the interrupt-driven wake-up system that lets the watch respond instantly to button presses even while sleeping. The code includes automatic time setting using the compilation timestamp and sophisticated battery monitoring that reads actual voltage and converts it to percentage using realistic lithium battery discharge curves. These features work together to create an educational tool that teaches binary numbers while providing reliable timekeeping and intelligent power management. The result is that the battery consumes only 7% per day. This means the battery charge will last 10 days 😁</p><p>The TWatch repo is here:</p><p><a href="https://github.com/Xinyuan-LilyGO/TTGO_TWatch_Library">https://github.com/Xinyuan-LilyGO/TTGO_TWatch_Library</a></p><p>First of all, download the Arduino IDE software <a href="https://www.arduino.cc/en/software/">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*hBCZR-npqEAvgMwsrlbDtQ.png" /><figcaption>Arduino Software download</figcaption></figure><p>Then, we have to install the necessary libraries. Create a folder Arduino/libraries and inside this folder, run:</p><pre>git clone https://github.com/Xinyuan-LilyGO/TTGO_TWatch_Library<br>git clone https://github.com/lewisxhe/AXP202X_Library.git</pre><p>Then, go to <strong>Arduino/File/Preferences</strong> and set up this folder as the Sketchbook location:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*UL80shmZz_Vvts-udc49CQ.png" /></figure><p><a href="https://gemini.google.com/app">Google’s Gemini</a> provided me with the transcription of the <strong>Internal Hardware Connections</strong>, given the electronic schematic diagram<strong>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*2sSxCnQh0XWoWFYB9Lr-WQ.png" /><figcaption>Electronic schematic diagram for the LYLIGO T-Watch-2020 V3</figcaption></figure><p>These are the ESP32 GPIO pins that are already connected to the internal components of the watch.</p><p><strong>Display (ST7789V TFT)</strong></p><ul><li><strong>MOSI:</strong> GPIO 19 (SPI Data)</li><li><strong>SCLK:</strong> GPIO 18 (SPI Clock)</li><li><strong>CS:</strong> GPIO 5 (SPI Chip Select)</li><li><strong>DC:</strong> GPIO 27 (Data/Command)</li><li><strong>RST:</strong> GPIO 26 (Reset)</li><li><strong>Backlight:</strong> GPIO 12 (Controlled by AXP202 LDO2)</li></ul><p><strong>Power Management (AXP202 PMIC)</strong></p><ul><li><strong>SDA:</strong> GPIO 21 (I2C Data)</li><li><strong>SCL:</strong> GPIO 22 (I2C Clock)</li><li><strong>IRQ:</strong> GPIO 35 (Interrupt Request)</li></ul><p><strong>Touch Screen (FT6236)</strong></p><ul><li><strong>SDA:</strong> GPIO 21 (Shared on I2C Bus)</li><li><strong>SCL:</strong> GPIO 22 (Shared on I2C Bus)</li><li><strong>IRQ:</strong> GPIO 38 (Touch Interrupt)</li><li><strong>Accelerometer (BMA423)</strong></li><li><strong>SDA:</strong> GPIO 21 (Shared on I2C Bus)</li><li><strong>SCL:</strong> GPIO 22 (Shared on I2C Bus)</li><li><strong>IRQ:</strong> GPIO 39 (Sensor Interrupt)</li><li><strong>Real-Time Clock (PCF8563)</strong></li><li><strong>SDA:</strong> GPIO 21 (Shared on I2C Bus)</li><li><strong>SCL:</strong> GPIO 22 (Shared on I2C Bus)</li></ul><p><strong>User Button</strong></p><ul><li>The side button is connected to the PEK input of the AXP202 power chip. You interact with it through the library (ttgo-&gt;power-&gt;isPEKShortPress()).</li></ul><p><strong>Vibration Motor</strong></p><ul><li><strong>Motor:</strong> GPIO 4</li></ul><p>As it has been 3–4 years I bought the watch, I had to check the state of the rechargeable battery with a Voltimeter. The adjustment of the battery inside of the watch is critical, because the battery pins are very sensitive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2k12FUSfPs9e7rUQvb9jrQ.jpeg" /><figcaption>Battery testing with a Voltimeter</figcaption></figure><p>An important step: define the board you are going to use. Go to Arduino/Tools and define board as <strong>TTGO-T-Watch</strong>. Also, set Core Debug Level to <strong>Verbose. </strong>Then, Erase All Flash Before Sketch Upload to <strong>True</strong>, Partition Scheme set to <strong>default</strong>, Board Revision to <strong>T-Watch-2020-V3</strong>, Upload Speed <strong>921600</strong>, and Programmer <strong>esptool</strong>.</p><p>Now, get the file <strong>LilyGoWatch.h</strong> and uncomment the following line of code:</p><pre>#define LILYGO_WATCH_2020_V3</pre><p>Your Arduino IDE will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*eTKiM4Pi8h9IjnleuYjWBQ.png" /></figure><p>Now, let’s get the code for the Binary Watch in C++:</p><pre>/*<br> * LIBRARY INCLUDES AND GLOBAL VARIABLE DECLARATIONS<br> * This section imports the necessary libraries and sets up global variables that will be used<br> * throughout the program. Think of this as gathering all the tools and materials you&#39;ll need<br> * before starting a project. The LilyGoWatch library provides specific functions for the T-Watch<br> * hardware, while the global pointers give us access to the display, power management, and main<br> * watch object from anywhere in our program.<br> */<br>#include &lt;LilyGoWatch.h&gt;<br><br>TTGOClass *watch = nullptr;<br>TFT_eSPI *tft = nullptr;<br>AXP20X_Class *power = nullptr;<br><br>#define LED_ON_COLOR  TFT_GREEN<br>#define LED_OFF_COLOR TFT_DARKGREY<br>#define BG_COLOR      TFT_BLACK<br>#define TEXT_COLOR    TFT_WHITE<br><br>#define DISPLAY_TIMEOUT 6000  // Turn off display after 6 seconds<br>unsigned long lastActivity = 0;<br>bool displayOn = true;<br>bool irq = false;  // Flag for AXP202 interrupt<br><br>void displayBinaryWatch(int hours, int minutes);<br>void setInitialTimeFromCompiler();<br>void goToSleep();<br>void wakeUp();<br><br>/*<br> * INTERRUPT SERVICE ROUTINE FOR POWER MANAGEMENT<br> * This is a special type of function that gets called automatically when the power management<br> * chip (AXP202) detects an event like a button press. The IRAM_ATTR tells the compiler to store<br> * this function in fast internal RAM so it can respond quickly to interrupts. Think of this as<br> * a doorbell - when someone presses the button, this function immediately &quot;rings&quot; to let the<br> * main program know something happened.<br> */<br>// AXP202 interrupt service routine<br>void IRAM_ATTR axp202_irq() {<br>    irq = true;<br>}<br><br>/*<br> * INITIAL SETUP AND CONFIGURATION<br> * This setup() function runs once when the device starts up and is responsible for initializing<br> * all the hardware components and configuring them properly. It&#39;s like setting up a workspace -<br> * turning on the lights, arranging your tools, and making sure everything is ready to use.<br> * The power management configuration here is particularly critical because it determines whether<br> * the device can wake up properly from sleep mode when the button is pressed.<br> */<br>void setup() {<br>    Serial.begin(115200);<br>    <br>    watch = TTGOClass::getWatch();<br>    watch-&gt;begin();<br>    <br>    // Get power management instance<br>    power = watch-&gt;power;<br>    <br>    watch-&gt;openBL();<br>    tft = watch-&gt;tft;<br>    <br>    tft-&gt;setRotation(2);<br>    tft-&gt;fillScreen(BG_COLOR);<br>    tft-&gt;setTextColor(TEXT_COLOR, BG_COLOR);<br>    tft-&gt;setTextDatum(MC_DATUM);<br>    <br>    setInitialTimeFromCompiler();<br>    <br>    // Critical: Proper AXP202 configuration for wake-up<br>    // Enable ADC for power monitoring<br>    power-&gt;adc1Enable(AXP202_BATT_VOL_ADC1 | AXP202_BATT_CUR_ADC1 | <br>                      AXP202_VBUS_VOL_ADC1 | AXP202_VBUS_CUR_ADC1, true);<br>    <br>    // Configure AXP202 interrupts - this is the key to proper wake-up<br>    power-&gt;enableIRQ(AXP202_PEK_SHORTPRESS_IRQ | AXP202_PEK_LONGPRESS_IRQ, true);<br>    power-&gt;clearIRQ();<br>    <br>    // Attach interrupt to AXP202 interrupt pin (GPIO 35)<br>    pinMode(AXP202_INT, INPUT);<br>    attachInterrupt(AXP202_INT, axp202_irq, FALLING);<br>    <br>    // Configure essential power outputs to stay on during sleep<br>    // Note: We only configure the power outputs that are actually defined in this library version<br>    power-&gt;setPowerOutPut(AXP202_LDO2, true);   // Display and sensors<br>    power-&gt;setPowerOutPut(AXP202_LDO3, true);   // Additional peripherals<br>    power-&gt;setPowerOutPut(AXP202_DCDC2, true);  // ESP32 core power<br>    power-&gt;setPowerOutPut(AXP202_EXTEN, false); // External enable (not needed for basic operation)<br>    <br>    lastActivity = millis();<br>    tft-&gt;fillScreen(BG_COLOR);<br>    <br>    Serial.println(&quot;T-Watch Binary Watch Ready&quot;);<br>}<br><br>/*<br> * MAIN PROGRAM LOOP - THE HEART OF THE WATCH<br> * This loop() function runs continuously while the device is awake, like the main engine of the<br> * watch. It handles three key responsibilities: detecting button presses through interrupt flags,<br> * updating the time display when the screen is on, and managing when to go to sleep to save battery.<br> * The loop checks for events, updates the display, and manages power - think of it as the watch&#39;s<br> * &quot;thinking process&quot; that never stops while it&#39;s awake.<br> */<br>void loop() {<br>    unsigned long currentTime = millis();<br>    <br>    // Handle AXP202 interrupt (button press or other power events)<br>    if (irq) {<br>        irq = false;<br>        power-&gt;readIRQ();<br>        <br>        // Check for button press<br>        if (power-&gt;isPEKShortPressIRQ()) {<br>            Serial.println(&quot;Short press detected&quot;);<br>            power-&gt;clearIRQ();<br>            <br>            if (!displayOn) {<br>                wakeUp();<br>            }<br>            lastActivity = currentTime;<br>        }<br>        <br>        if (power-&gt;isPEKLongPressIRQ()) {<br>            Serial.println(&quot;Long press detected&quot;);<br>            power-&gt;clearIRQ();<br>            <br>            if (!displayOn) {<br>                wakeUp();<br>            }<br>            lastActivity = currentTime;<br>        }<br>        <br>        // Clear any remaining interrupts<br>        power-&gt;clearIRQ();<br>    }<br>    <br>    // Update display if it&#39;s on<br>    if (displayOn) {<br>        RTC_Date datetime = watch-&gt;rtc-&gt;getDateTime();<br>        displayBinaryWatch(datetime.hour, datetime.minute);<br>        <br>        // Check if we should turn off display<br>        if (currentTime - lastActivity &gt; DISPLAY_TIMEOUT) {<br>            goToSleep();<br>        }<br>    }<br>    <br>    delay(1000);<br>}<br><br>/*<br> * SLEEP MODE MANAGEMENT FOR POWER CONSERVATION<br> * This function handles putting the watch into a low-power sleep state to preserve battery life.<br> * It&#39;s like putting the watch into a &quot;hibernation&quot; mode where most systems shut down, but the<br> * power management chip stays alert to wake the device when the button is pressed. The process<br> * involves carefully shutting down the display, configuring wake-up sources, and entering a<br> * light sleep that maintains enough functionality to respond to button presses.<br> */<br>void goToSleep() {<br>    Serial.println(&quot;Going to sleep...&quot;);<br>    <br>    // Turn off display<br>    tft-&gt;fillScreen(TFT_BLACK);<br>    watch-&gt;closeBL();<br>    displayOn = false;<br>    <br>    // Clear any pending interrupts before sleep<br>    power-&gt;clearIRQ();<br>    <br>    // Configure ESP32 wake-up source - AXP202 interrupt on GPIO 35<br>    esp_sleep_enable_ext0_wakeup(GPIO_NUM_35, 0);<br>    <br>    // Put the display to sleep to save power<br>    tft-&gt;writecommand(ST7789_SLPIN);<br>    <br>    // Reduce CPU frequency for power savings<br>    setCpuFrequencyMhz(80);<br>    <br>    // Enter light sleep - this keeps the AXP202 active for wake-up<br>    Serial.println(&quot;Entering light sleep&quot;);<br>    esp_light_sleep_start();<br>    <br>    // When we reach here, we&#39;ve been woken up<br>    Serial.println(&quot;Woke up from sleep!&quot;);<br>    <br>    // Restore CPU frequency<br>    setCpuFrequencyMhz(240);<br>    <br>    // Wake up the display<br>    tft-&gt;writecommand(ST7789_SLPOUT);<br>    delay(120); // Display needs time to wake up<br>}<br><br>/*<br> * WAKE-UP PROCESS AND DISPLAY REACTIVATION<br> * This function handles bringing the watch back to full operation after it has been sleeping.<br> * It&#39;s like turning the lights back on and getting everything ready to work again. The function<br> * reactivates the display backlight, sets the proper state flags, and clears any leftover<br> * interrupt signals to ensure the watch is ready for normal operation.<br> */<br>void wakeUp() {<br>    Serial.println(&quot;Display waking up&quot;);<br>    <br>    // Turn on backlight<br>    watch-&gt;openBL();<br>    displayOn = true;<br>    lastActivity = millis();<br>    <br>    // Clear any pending interrupts to start fresh<br>    power-&gt;clearIRQ();<br>}<br><br>/*<br> * AUTOMATIC TIME SETTING FROM COMPILATION TIMESTAMP<br> * This clever function sets the watch&#39;s time automatically using the date and time when the<br> * program was compiled. It&#39;s like having the watch &quot;remember&quot; when it was built and use that<br> * as a starting point for keeping time. The function parses the compiler&#39;s __DATE__ and __TIME__<br> * macros, converts text month names to numbers, and programs the real-time clock chip with this<br> * information so the watch starts with approximately the correct time.<br> */<br>void setInitialTimeFromCompiler() {<br>    tft-&gt;fillScreen(BG_COLOR);<br>    tft-&gt;drawString(&quot;Setting Time...&quot;, 120, 120);<br>    <br>    char month_str[4];<br>    int day, year, hour, minute, second;<br>    <br>    sscanf(__DATE__, &quot;%s %d %d&quot;, month_str, &amp;day, &amp;year);<br>    sscanf(__TIME__, &quot;%d:%d:%d&quot;, &amp;hour, &amp;minute, &amp;second);<br>    <br>    const char* months[] = {&quot;Jan&quot;, &quot;Feb&quot;, &quot;Mar&quot;, &quot;Apr&quot;, &quot;May&quot;, &quot;Jun&quot;,<br>                           &quot;Jul&quot;, &quot;Aug&quot;, &quot;Sep&quot;, &quot;Oct&quot;, &quot;Nov&quot;, &quot;Dec&quot;};<br>    int month = 1;<br>    <br>    for (int i = 0; i &lt; 12; i++) {<br>        if (strcmp(month_str, months[i]) == 0) {<br>            month = i + 1;<br>            break;<br>        }<br>    }<br>    <br>    watch-&gt;rtc-&gt;setDateTime(year, month, day, hour, minute, second);<br>    delay(1000);<br>}<br><br>/*<br> * BINARY TIME DISPLAY VISUALIZATION WITH BATTERY MONITORING<br> * This function creates the visual representation of time in binary format on the watch screen.<br> * Binary representation shows time using only 1s and 0s (green dots for 1, gray dots for 0),<br> * which is how computers actually store and process numbers internally. Each row represents either<br> * hours or minutes as a 6-bit binary number, allowing display of hours 0-23 and minutes 0-59.<br> * The function also shows regular time below the binary display for easy reference and learning.<br> * Additionally, it includes real-time battery monitoring by reading the actual voltage from the<br> * power management chip and converting it to a percentage based on typical lithium battery curves.<br> */<br>void displayBinaryWatch(int hours, int minutes) {<br>    tft-&gt;fillScreen(BG_COLOR);<br><br>    // Battery monitoring - Read actual voltage and convert to meaningful percentage<br>    // Lithium batteries typically range from 3.2V (empty) to 4.2V (full)<br>    // We use a more realistic curve that accounts for how lithium batteries actually discharge<br>    float batteryVoltage = power-&gt;getBattVoltage() / 1000.0; // Convert millivolts to volts<br>    int batteryPercent;<br>    <br>    // Convert voltage to percentage using realistic lithium battery discharge curve<br>    // This isn&#39;t linear because batteries don&#39;t discharge linearly<br>    if (batteryVoltage &gt;= 4.1) {<br>        batteryPercent = 100;<br>    } else if (batteryVoltage &gt;= 3.9) {<br>        // 90-100% range: voltage drops slowly at high charge<br>        batteryPercent = 90 + (int)((batteryVoltage - 3.9) * 50);<br>    } else if (batteryVoltage &gt;= 3.7) {<br>        // 50-90% range: more linear discharge in middle range<br>        batteryPercent = 50 + (int)((batteryVoltage - 3.7) * 200);<br>    } else if (batteryVoltage &gt;= 3.4) {<br>        // 10-50% range: faster voltage drop<br>        batteryPercent = 10 + (int)((batteryVoltage - 3.4) * 133);<br>    } else if (batteryVoltage &gt;= 3.2) {<br>        // 0-10% range: rapid voltage drop when nearly empty<br>        batteryPercent = (int)((batteryVoltage - 3.2) * 50);<br>    } else {<br>        batteryPercent = 0; // Battery critically low<br>    }<br>    <br>    // Ensure percentage stays within valid range<br>    batteryPercent = constrain(batteryPercent, 0, 100);<br>    <br>    // Display battery percentage in top right corner with visual indicator<br>    tft-&gt;setTextSize(1);<br>    tft-&gt;setTextColor(TEXT_COLOR, BG_COLOR);<br>    <br>    // Draw simple battery icon outline (rectangle with terminal)<br>    int battX = 210;<br>    int battY = 25;<br>    int battWidth = 20;<br>    int battHeight = 10;<br>    <br>    // Show percentage text - positioned to center above the battery icon<br>    // Calculate center of battery: battX + (battWidth / 2)<br>    char batteryText[8];<br>    sprintf(batteryText, &quot;%d%%&quot;, batteryPercent);<br>    int textCenterX = battX + (battWidth / 2);  // Center the text over the battery<br>    tft-&gt;drawString(batteryText, textCenterX, 15);<br>    <br>    // Battery outline<br>    tft-&gt;drawRect(battX, battY, battWidth, battHeight, TEXT_COLOR);<br>    // Battery terminal (small rectangle on right side)<br>    tft-&gt;fillRect(battX + battWidth, battY + 2, 2, battHeight - 4, TEXT_COLOR);<br>    <br>    // Fill battery based on percentage with color coding<br>    int fillWidth = (battWidth - 2) * batteryPercent / 100;<br>    uint16_t fillColor;<br>    <br>    if (batteryPercent &gt; 50) {<br>        fillColor = TFT_GREEN;      // Green when battery is good<br>    } else if (batteryPercent &gt; 20) {<br>        fillColor = TFT_YELLOW;     // Yellow when getting low<br>    } else {<br>        fillColor = TFT_RED;        // Red when critically low<br>    }<br>    <br>    if (fillWidth &gt; 0) {<br>        tft-&gt;fillRect(battX + 1, battY + 1, fillWidth, battHeight - 2, fillColor);<br>    }<br><br>    int ledSize = 12;<br>    int ledSpacing = 30;<br>    int startX = 54;<br>    int hoursY = 70;<br>    int minutesY = 130;<br><br>    tft-&gt;setTextSize(1);<br>    tft-&gt;setTextColor(TEXT_COLOR, BG_COLOR);<br>    tft-&gt;drawString(&quot;Hours&quot;, startX + (ledSpacing * 2.5), hoursY - 25);<br>    <br>    // Display hours in binary (6 bits for values 0-23 == 24 hours)<br>    for (int i = 5; i &gt;= 0; i--) {<br>        bool bitSet = (hours &gt;&gt; i) &amp; 1;<br>        int x = startX + ((5 - i) * ledSpacing);<br>        uint16_t color = bitSet ? LED_ON_COLOR : LED_OFF_COLOR;<br>        tft-&gt;fillCircle(x, hoursY, ledSize, color);<br>    }<br><br>    tft-&gt;drawString(&quot;Minutes&quot;, startX + (ledSpacing * 2.5), minutesY - 25);<br>    <br>    // Display minutes in binary (6 bits for values 0-59)<br>    for (int i = 5; i &gt;= 0; i--) {<br>        bool bitSet = (minutes &gt;&gt; i) &amp; 1;<br>        int x = startX + ((5 - i) * ledSpacing);<br>        uint16_t color = bitSet ? LED_ON_COLOR : LED_OFF_COLOR;<br>        tft-&gt;fillCircle(x, minutesY, ledSize, color);<br>    }<br><br>    RTC_Date datetime = watch-&gt;rtc-&gt;getDateTime();<br><br>    // Format the date as DD/MM<br>    char dateStr[6];  // String to hold &quot;DD/MM&quot; plus null terminator<br>    sprintf(dateStr, &quot;%02d/%02d&quot;, datetime.day, datetime.month);<br><br>    // Display the formatted date<br>    tft-&gt;setTextColor(LED_OFF_COLOR, BG_COLOR);<br>    tft-&gt;setTextSize(2);<br>    tft-&gt;drawString(dateStr, 120, 190);<br>}</pre><p>Now we can upload the code to the watch, by clicking the Right Arrow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*RVo643tqYudrbkpecNzy0A.png" /><figcaption>Ready to upload the code</figcaption></figure><p>You will see a successful output. If not, and if you need to reboot or restore factory settings in the watch, and have to remove the battery, be very careful because the battery pins are very delicate and may change position. If this happens, the watch won’t turn on while disconnected from USB. It happened to me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*5afn17NvkAH4afjz6HFSAQ.png" /><figcaption>Successful upload</figcaption></figure><p>If you check the Serial Monitor, you will see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*psFkOZaW91_NOkBKpefOdw.png" /></figure><h3>RESULT</h3><p>You will get a working watch:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9UYvGkXB3njl7j17n7F-BQ.jpeg" /><figcaption>The 24 hours Binary Watch ready. Date in DD/MM made with this tutorial.</figcaption></figure><p>This is an improved version, with minor changes to the code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TOVlwEPjIE7NL5CWRqnsHg.jpeg" /><figcaption>Version 2 of the 24 hours Binary Watch</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yvvjw9jkkd0_G-EtHiXiug.jpeg" /><figcaption>Matrix version of the 24 hours Binary Watch</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=856c063b4605" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Gemma Directly from AI Studio to Cloud Run]]></title>
            <link>https://medium.com/@rubenszimbres/deploying-gemma-directly-from-ai-studio-to-cloud-run-118758493e3a?source=rss-30f09d1741ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/118758493e3a</guid>
            <category><![CDATA[gemma]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-cloud-run]]></category>
            <dc:creator><![CDATA[Rubens Zimbres]]></dc:creator>
            <pubDate>Thu, 29 May 2025 18:57:12 GMT</pubDate>
            <atom:updated>2025-05-29T18:57:12.868Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*3N-ywasak_jARpV-H7v81g.png" /></figure><p>In this tutorial, I will provide the simplest way to deploy a Gemma (or Gemini) model from AI Studio to Cloud Run and have your app up and running.</p><blockquote>CONCEPTS</blockquote><p><strong>Gemma</strong> is a family of lightweight, state-of-the-art open models developed by Google, built from the same research and technology used to create the Gemini models. Named after the Latin word for “precious stone,” Gemma models are designed to be accessible and enable developers and researchers to build AI responsibly. They come in various sizes (e.g., 1B, 4B, 12B, 27B) to suit different computational resources and application needs, from running on mobile devices and laptops to powerful workstations or cloud infrastructure. Gemma models are available as pre-trained and instruction-tuned variants, allowing them to be used for a wide range of text generation tasks or further fine-tuned for specific purposes. Back in 2024 I wrote <a href="https://medium.com/google-developer-experts/fine-tuning-gemma-2b-to-solve-math-problems-ac4921ed531e">an article</a> about Fine Tuning Gemma-2b to Solve Math Problems.</p><p>The Gemma family offers diverse capabilities, including strong performance in text generation, summarization, question answering, and reasoning. Newer versions, such as Gemma 3, have introduced multimodal capabilities, allowing them to process both text and image inputs, with a significantly larger context window (up to 128K tokens), support for numerous languages, and function calling abilities. Specialized versions like CodeGemma are optimized for code generation and completion, while others like PaliGemma focus on visual language tasks such as image captioning.</p><p><strong>Cloud Run</strong> is a managed compute platform from Google Cloud that enables you to run stateless containers that are invocable via web requests or Pub/Sub (message subscription) events. It is a fully serverless offering, meaning it abstracts away all infrastructure management, so you don’t have to worry about provisioning servers, configuring clusters, or scaling.</p><p>It allows you to deploy applications written in any programming language, using any libraries or binaries, simply by providing a container image. Cloud Run automatically scales your application up or down, even to zero, based on incoming requests or events, ensuring you only pay for the compute resources you consume. This makes it ideal for a wide range of use cases, including web applications, APIs, microservices, and background jobs, offering a flexible and cost-effective way to deploy and scale applications.</p><p>Here, we will access Google AI Studio, get the code to run the Gemma model, make some adaptations, create new files and deploy the whole solution in a Cloud Run instance.</p><blockquote>GETTING THE GEMMA CODE</blockquote><p>Let’s start. Go to <a href="https://aistudio.google.com/app/prompts/new_chat">https://aistudio.google.com/app/prompts/new_chat</a></p><p>Choose the Gemma model you want. If you mouse over its name, you will see the model card. Currently the Gemma cost is zero for all of us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*mpDQkRJXbXCc94Vopijceg.png" /></figure><p>Now, click the <strong>&lt; &gt;</strong> icon to see the code to run the model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FpVzI-bwVSXzxOW3IYLHdw.png" /></figure><p>Copy this code from Google AI Studio as is:</p><pre># To run this code you need to install the following dependencies:<br># pip install google-genai<br><br>import base64<br>import os<br>from google import genai<br>from google.genai import types<br><br><br>def generate():<br>    client = genai.Client(<br>        api_key=os.environ.get(&quot;GEMINI_API_KEY&quot;),<br>    )<br><br>    model = &quot;gemma-3-12b-it&quot;<br>    contents = [<br>        types.Content(<br>            role=&quot;user&quot;,<br>            parts=[<br>                types.Part.from_text(text=&quot;&quot;&quot;INSERT_INPUT_HERE&quot;&quot;&quot;),<br>            ],<br>        ),<br>    ]<br>    generate_content_config = types.GenerateContentConfig(<br>        response_mime_type=&quot;text/plain&quot;,<br>    )<br><br>    for chunk in client.models.generate_content_stream(<br>        model=model,<br>        contents=contents,<br>        config=generate_content_config,<br>    ):<br>        print(chunk.text, end=&quot;&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    generate()</pre><p>Now we have to adapt four little things and save the script as <em>gemma_chat.py</em>:</p><ul><li>Use <em>dotenv</em> for local development, then <em>secretmanager</em> for securely store of credentials. You should <strong>never </strong>store your <em>credentials.json</em> file in the container, for security reasons. We will add <em>project number</em> and <em>get_secret()</em> function.</li><li>We will add an input variable <em>entry</em> to the get the input from the user, instead of the hardcoded <em>“INSERT_INPUT_HERE”</em></li><li>Instead of just printing the answer, we will parse it to return the output of the <em>generate()</em> function.</li><li>Lastly, we will add more config options, like <em>temperature</em>, <em>top_p</em>, <em>top_k</em>, etc, so that you can play with different hyperparameters.</li></ul><p>Go to Google Cloud Console/Secret Manager, create a secret called <em>GOOGLE_API_KEY</em> and paste the API key you get from AI Studio. Below, it’s the final code file with the adjustments:</p><pre>from dotenv import load_dotenv # for local development, with .env file<br>import base64<br>import os<br>from google import genai<br>from google.genai import types<br>from google.cloud import secretmanager<br><br>load_dotenv() # for local development, with .env file<br><br>project_id=&#39;123456789&#39;<br><br>def get_secret(project_id: str, secret_id: str, version_id: str = &quot;1&quot;) -&gt; str:<br>    client = secretmanager.SecretManagerServiceClient()<br>    name = f&quot;projects/{project_id}/secrets/{secret_id}/versions/{version_id}&quot;<br>    response = client.access_secret_version(request={&quot;name&quot;: name})<br>    return response.payload.data.decode(&#39;UTF-8&#39;)<br><br>def generate(entry):<br>    client = genai.Client(<br>        api_key=get_secret(project_id, &#39;GOOGLE_API_KEY&#39;),<br>    )<br><br>    model = &quot;gemma-3-12b-it&quot;<br>    contents = [<br>        types.Content(<br>            role=&quot;user&quot;,<br>            parts=[<br>                types.Part.from_text(text=entry),<br>            ],<br>        ),<br>    ]<br>    generate_content_config = types.GenerateContentConfig(<br>        response_mime_type=&quot;text/plain&quot;,<br>        temperature=0,<br>        top_p=0.95,<br>        top_k=20,<br>        candidate_count=1,<br>        seed=5,<br>        max_output_tokens=512<br>    )<br>    answer=&quot;&quot;<br>    for chunk in client.models.generate_content_stream(<br>        model=model,<br>        contents=contents,<br>        config=generate_content_config,<br>    ):<br><br>        try:<br>            answer+=chunk.text<br>        except:<br>            pass<br>    return answer<br><br>## Test it to make sure it works<br><br>#if __name__ == &quot;__main__&quot;:<br>#    test_response = generate(entry=&quot;Hello, tell me briefly what is a Fast Fourier Transform?&quot;)<br>#    print(f&quot;Test response: {test_response}&quot;)</pre><p>Now we develop our Flask <em>app.py</em>. You can also use FastAPI, Gradio or Streamlit. At the end we will also create an <em>index.html</em> file, that you can host locally to test.</p><pre>from flask import Flask, request, jsonify, send_from_directory<br>from flask_cors import CORS<br>import os<br>from gemma_chat import generate as generate_gemini_response<br><br>app = Flask(__name__)<br>CORS(app)<br><br>@app.route(&#39;/&#39;)<br>def index():<br>    &quot;&quot;&quot;Serves the index.html file.&quot;&quot;&quot;<br>    return send_from_directory(&#39;.&#39;, &#39;index.html&#39;)<br><br>@app.route(&#39;/generate&#39;, methods=[&#39;POST&#39;])<br>def handle_generate():<br>    &quot;&quot;&quot;<br>    Handles POST requests to /generate.<br>    Expects JSON with a &#39;prompt_alpha&#39; key.<br>    Calls the Gemini generate function and returns the response.<br>    &quot;&quot;&quot;<br>    if not request.is_json:<br>        return jsonify({&quot;error&quot;: &quot;Request must be JSON&quot;}), 400<br><br>    data = request.get_json()<br>    user_prompt = data.get(&#39;prompt_alpha&#39;)<br><br>    if not user_prompt:<br>        return jsonify({&quot;error&quot;: &quot;Missing &#39;prompt_alpha&#39; in request body&quot;}), 400<br><br>    try:<br>        # Call the generate function from gemini_chat.py<br>        model_response = generate_gemini_response(user_prompt)<br>        return jsonify({&quot;response&quot;: model_response})<br>    except Exception as e:<br>        print(f&quot;An unexpected error occurred in /generate: {e}&quot;)<br>        return jsonify({&quot;error&quot;: &quot;An internal server error occurred.&quot;}), 500<br><br>if __name__ == &#39;__main__&#39;:<br>    # For local development, debug=True is helpful. For prod, debug=False<br>    app.run(debug=True, host=&#39;0.0.0.0&#39;, port=8080)</pre><p>The host ‘0.0.0.0’ makes it accessible externally (e.g., within a Docker container or local network). For troubleshooting, use <em>debug=True</em>, so if your deployment fails, you can check the Cloud Run logs where exactly is the problem.</p><p>We create a <em>requirements.txt</em> file in the root directory:</p><pre>Flask==3.1.1<br>flask-cors==6.0.0<br>google-genai<br>python-dotenv<br>google-cloud-secret-manager==2.23.3</pre><p>… and also a Dockerfile. Again, we will not run the container as a root user for security reasons:</p><pre># Use Python 3.11 slim image as base<br>FROM python:3.11-slim<br><br># Set work directory<br>WORKDIR /app<br><br># Install system dependencies<br>RUN apt-get update \<br>    &amp;&amp; apt-get install -y --no-install-recommends \<br>        gcc \<br>        g++ \<br>    &amp;&amp; rm -rf /var/lib/apt/lists/*<br><br># Copy requirements file<br>COPY requirements.txt .<br><br># Install Python dependencies<br>RUN pip install --no-cache-dir -r requirements.txt<br><br># Copy project files<br>COPY . .<br><br># Create non-root user for increased security<br>RUN useradd --create-home --shell /bin/bash app \<br>    &amp;&amp; chown -R app:app /app<br>USER app<br><br># Expose port<br>EXPOSE 8080<br><br># Command to run the application<br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><blockquote>DEPLOYMENT IN CLOUD RUN</blockquote><p>In this final step we set the Google Cloud project and define environment variables:</p><pre>gcloud config set project your-project<br><br>export GCP_PROJECT=&#39;your-project&#39;<br>export GCP_REGION=&#39;us-central1&#39;<br>export AR_REPO=&#39;repo-gemma&#39;<br>export SERVICE_NAME=&#39;gemma-flash-app&#39;</pre><p>Now we will create a repository in Artifact Registry:</p><pre>gcloud artifacts repositories create &quot;$AR_REPO&quot; --location=&quot;$GCP_REGION&quot; --repository-format=Docker</pre><p>Then we will build the container image:</p><pre>gcloud builds submit --tag &quot;$GCP_REGION-docker.pkg.dev/$GCP_PROJECT/$AR_REPO/$SERVICE_NAME&quot;</pre><p>Finally, we create the Cloud Run revision. As Gemma is being called from an API paid by Google (not by you 😁), a minimum set up is enough, 512 MB of memory. We can also scale to zero to save costs. For testing, we will allow unauthenticated calls in Cloud Run.</p><pre>gcloud run deploy &quot;$SERVICE_NAME&quot; \<br>--port=8080 \<br>--image=&quot;$GCP_REGION-docker.pkg.dev/$GCP_PROJECT/$AR_REPO/$SERVICE_NAME&quot; \<br>--allow-unauthenticated \<br>--platform=managed  \<br>--region=$GCP_REGION \<br>--project=$GCP_PROJECT \<br>--set-env-vars=GCP_PROJECT=$GCP_PROJECT,GCP_REGION=$GCP_REGION \<br>--min-instances 0 --max-instances 1 --cpu 2 --memory 512Mi --concurrency 40</pre><p>The deploy will succeed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*ZXeS065HpPzCYsRcMZtZRA.png" /></figure><p>You can see in the Google Cloud Console the service running. Now you should get the Cloud Run endpoint so that we add it to our <em>index</em> file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*h5xqxLjg8GtpGEZWJ2WbNQ.png" /></figure><p>Now, in order to test the chatbot, we will create an <em>index.html</em> file, that must contain the Cloud Run endpoint:</p><pre>const CHAT_CONFIG = {<br>            apiEndpoint: &#39;https://gemma-flash-app-123456789.us-central1.run.app/generate&#39;,<br>            maxRetries: 3,<br>            retryDelay: 1000,<br>            typingDelay: 500<br>        };</pre><p>… and also the POST method:</p><pre>const response = await fetch(CHAT_CONFIG.apiEndpoint, {<br>    method: &#39;POST&#39;,<br>    headers: {<br>        &#39;Content-Type&#39;: &#39;application/json&#39;,<br>    },<br>    body: JSON.stringify({ prompt_alpha: message }),<br>});</pre><p>Here is a sample HTML file so that you can run your tests:</p><pre>&lt;!DOCTYPE html&gt;<br>&lt;html lang=&quot;en&quot;&gt;<br>&lt;head&gt;<br>    &lt;meta charset=&quot;UTF-8&quot;&gt;<br>    &lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1.0&quot;&gt;<br>    &lt;title&gt;LLM Chatbot&lt;/title&gt;<br>    &lt;style&gt;<br>        body {<br>          font-family: Arial, sans-serif;<br>          display: flex;<br>          justify-content: center;<br>          align-items: center;<br>          height: 100vh;<br>          margin: 0;<br>          background-color: #f0f0f0;<br>        }<br>        #chat-container {<br>          position: fixed;<br>          bottom: 20px;<br>          right: 20px;<br>          width: 300px;<br>          height: 400px;<br>          background-color: #c1e0f1f8;<br>          border-radius: 10px;<br>          box-shadow: 0 0 10px rgba(0,0,0,0.1);<br>          display: flex;<br>          flex-direction: column;<br>          overflow: hidden;<br>        }<br>        #chat-messages {<br>          flex-grow: 1;<br>          overflow-y: auto;<br>          padding: 10px;<br>          display: flex;<br>          flex-direction: column;<br>        }<br>        #user-input {<br>          display: flex;<br>          padding: 10px;<br>        }<br>        #message-input {<br>          flex-grow: 1;<br>          padding: 5px;<br>          border: 1px solid #ccc;<br>          border-radius: 3px;<br>        }<br>        #send-button {<br>          margin-left: 10px;<br>          padding: 5px 10px;<br>          background-color: #007bff;<br>          color: white;<br>          border: none;<br>          border-radius: 3px;<br>          cursor: pointer;<br>        }<br>        .message {<br>          margin-bottom: 10px;<br>          padding: 5px 10px;<br>          border-radius: 5px;<br>          max-width: 70%;<br>        }<br>        .user-message {<br>          background-color: #007bff;<br>          color: white;<br>          align-self: flex-start;<br>        }<br>        .bot-message {<br>          background-color: #f1f0f0;<br>          color: black;<br>          align-self: flex-end;<br>        }<br>        &lt;/style&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>    &lt;div id=&quot;chat-container&quot;&gt;<br>        &lt;div id=&quot;chat-messages&quot;&gt;&lt;/div&gt;<br>        &lt;div id=&quot;user-input&quot;&gt;<br>            &lt;input type=&quot;text&quot; id=&quot;message-input&quot; placeholder=&quot;Type your message...&quot;&gt;<br>            &lt;button id=&quot;send-button&quot;&gt;Send&lt;/button&gt;<br>        &lt;/div&gt;<br>    &lt;/div&gt;<br><br>    &lt;script&gt;<br>        const chatMessages = document.getElementById(&#39;chat-messages&#39;);<br>        const messageInput = document.getElementById(&#39;message-input&#39;);<br>        const sendButton = document.getElementById(&#39;send-button&#39;);<br><br>        const CLOUD_RUN_ENDPOINT = &#39;https://127.0.0.1:8080/generate&#39;; // Cloud Run endpoint here<br><br>        function addMessage(message, isUser) {<br>            const messageElement = document.createElement(&#39;div&#39;);<br>            messageElement.className = `message ${isUser ? &#39;user-message&#39; : &#39;bot-message&#39;}`;<br>            messageElement.innerHTML = message; <br>            chatMessages.appendChild(messageElement);<br>            chatMessages.scrollTop = chatMessages.scrollHeight;<br>        }<br><br>        async function sendMessage() {<br>            const message = messageInput.value.trim();<br>            if (message) {<br>                addMessage(message, true);<br>                messageInput.value = &#39;&#39;;<br><br>                try {<br>                    const response = await fetch(CLOUD_RUN_ENDPOINT, {<br>                        method: &#39;POST&#39;,<br>                        headers: {<br>                            &#39;Content-Type&#39;: &#39;application/json&#39;,<br>                        },<br>                        body: JSON.stringify({ prompt_alpha: message }),<br>                    });<br><br>                    if (!response.ok) {<br>                        throw new Error(&#39;API request failed&#39;);<br>                    }<br><br>                    const data = await response.json();<br>                    addMessage(data.response, false);<br>                } catch (error) {<br>                    console.error(&#39;Error:&#39;, error);<br>                    addMessage(&#39;Sorry, there was an error processing your request.&#39;, false);<br>                }<br>            }<br>        }<br><br>        sendButton.addEventListener(&#39;click&#39;, sendMessage);<br>        messageInput.addEventListener(&#39;keypress&#39;, (event) =&gt; {<br>            if (event.key === &#39;Enter&#39;) {<br>                sendMessage();<br>            }<br>        });<br>    &lt;/script&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>The result is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*RX53V3HStnUwlQNaowwCQw.png" /></figure><p>Simple as that !</p><p>👏👏👏 if you liked ☺️</p><p><strong>ACKNOWLEDGEMENTS</strong></p><p>✨ <em>Google ML Developer Programs and Google Developers Program supported this work by providing Google Cloud Credits </em>✨</p><p>🔗 <a href="https://developers.google.com/machine-learning">https://developers.google.com/machine-learning</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=118758493e3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>