<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ayush mittal on Medium]]></title>
        <description><![CDATA[Stories by ayush mittal on Medium]]></description>
        <link>https://medium.com/@ayushm4489?source=rss-2839f9309087------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*sors61NtMAZNcF7b</url>
            <title>Stories by ayush mittal on Medium</title>
            <link>https://medium.com/@ayushm4489?source=rss-2839f9309087------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:47:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ayushm4489/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Jev as a Judge: Can a Decision Model Score Customer-Support Agents as well as a Frontier LLM?]]></title>
            <link>https://ayushm4489.medium.com/jev-as-a-judge-can-a-decision-model-score-customer-support-agents-as-well-as-a-frontier-llm-4af637a9bd15?source=rss-2839f9309087------2</link>
            <guid isPermaLink="false">https://medium.com/p/4af637a9bd15</guid>
            <category><![CDATA[llm-as-a-judge]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[jevs]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[ayush mittal]]></dc:creator>
            <pubDate>Mon, 21 Sep 2026 22:34:32 GMT</pubDate>
            <atom:updated>2026-09-21T22:34:32.329Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Evaluate 21 fixed, benchmark-generated customer-support trajectories 50 times with each Jev and GPT-5.4. Compare benchmarks, score stability, latency, cost, and probability quality.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lA7IvI8CfTrgUsKIG5wpaw.png" /></figure><h3>Key Takeaways</h3><ul><li><strong>Case-level agreement tied.</strong> Both judges matched the τ² task-success reward on <strong>19 of 21 cases</strong>, using a majority vote over repeated verdicts for each case.</li><li><strong>Jev’s scores were more stable.</strong> GPT-5.4’s mean within-case probability variance was <strong>44.5–155.7× higher</strong>, depending on the evaluation signal.</li><li><strong>Jev’s binary verdict never changed within a case in this run.</strong> GPT-5.4 returned both pass and fail verdicts on <strong>4 of 21 cases</strong>. Stable verdicts did not guarantee agreement with the benchmark.</li><li><strong>Jev averaged about 10× lower latency and approximately 73× lower API usage cost.</strong> Mean latency was <strong>0.99 seconds versus 10.13 seconds</strong>. The cost ratio is based on the reported per-10,000-trace projections.</li><li><strong>GPT-5.4 had the lower task-success Brier score:</strong> <strong>0.0707 versus 0.1024</strong>. Lower is better, but Brier score alone does not isolate calibration.</li><li><strong>This was a small repeatability study.</strong> The dataset contained <strong>21 distinct simulated support cases</strong>, not 2,100 independent customer scenarios. Correctness was assessed against an automated task-success reference, without human adjudication.</li></ul><h3>Why Test a Different Kind of Judge?</h3><p>Customer-support evaluation has to look beyond a convincing final response. An agent can sound helpful while making the wrong tool call, violating policy, or reporting an action that never succeeded.</p><p>Deterministic checks are valuable for explicit invariants and verifiable backend outcomes. They can also encode assumptions about one expected action sequence that do not cover every acceptable way to satisfy a customer’s request. LLM judges offer a way to evaluate the broader evidence, with additional latency, cost, and variation between repeated judgments.</p><p>Inspired by LangChain’s <a href="https://www.langchain.com/blog/jev-agent-evals-langsmith">Jev-as-a-Judge experiment</a>, this experiment examined a decision model in a more involved benchmark setting: a retail support agent that authenticates customers, inspects orders, follows policy, calls tools, changes simulated backend state, and communicates the result.</p><p>The question was practical: <strong>can a decision model provide useful evaluation signals with the agreement, stability, and cost profile needed for frequent testing?</strong></p><h3>What Is Jev?</h3><p>Jev is <a href="https://docs.typesafe.ai/introduction">TypeSafe AI’s System One decision model</a>. Instead of generating a critique token by token and parsing it back into a verdict, it evaluates a structured state against typed questions and returns decisions and probabilities directly — no free-text generation, no parsing step, no JSON that might not validate.</p><p>Jev supports three evaluation primitives:</p><ul><li><strong>Choice</strong> selects among named outcomes.</li><li><strong>Score</strong> rates an answer against an ordered rubric.</li><li><strong>Noul</strong> returns the probability that a yes-or-no proposition is true.</li></ul><p>This experiment used Noul. A single API call evaluated five atomic questions against the same support trace:</p><pre>QUESTIONS = {<br>    &quot;task_success&quot;:<br>        &quot;Did the agent successfully complete the customer&#39;s requested outcome according to the evaluation criteria?&quot;,<br>    &quot;policy_compliance&quot;:<br>        &quot;Did the agent comply with the supplied customer-support policy?&quot;,<br>    &quot;tool_correctness&quot;:<br>        &quot;Were the agent&#39;s tool calls and arguments correct for completing the customer&#39;s request?&quot;,<br>    &quot;response_groundedness&quot;:<br>        &quot;Is the agent&#39;s final response fully supported by the conversation, tool calls, and tool results?&quot;,<br>    &quot;customer_communication&quot;:<br>        &quot;Did the agent clearly and accurately communicate the outcome relevant to the customer&#39;s request?&quot;,<br>}</pre><p>Every question returned a probability from 0 to 1. This experiment converted only task_success into a pass/fail verdict, using p &gt; 0.5 as the pass threshold. The other four signals were diagnostic — useful for understanding <em>why</em> a trace succeeded or failed, not just whether it did.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Pdm2_652IC0y7CBK2fJVQ.png" /></figure><h3>Why Might Jev Be a Good Agent Evaluator?</h3><p>Customer-support evaluation isn’t just a text-quality problem. A response can read perfectly while the underlying action failed silently. Conversely, a rigid action-matcher can mark a trace wrong even when the agent followed the customer’s explicit instruction and the backend operation succeeded exactly as asked.</p><p>A useful judge has to reason over the whole state, not just the final message:</p><ul><li>the customer’s request and scenario</li><li>the support policy</li><li>the benchmark’s evaluation criteria</li><li>available tool definitions</li><li>conversation messages</li><li>tool names and arguments</li><li>tool results and resulting state</li><li>the agent’s final response</li></ul><p>That’s a decision task, not a writing task: given this evidence, how likely is it that the agent completed the job, followed policy, used its tools correctly, stayed grounded in the transcript, and communicated the outcome clearly?</p><p>To keep the comparison fair, Jev and GPT-5.4 received the <strong>same structured state and the exact same five question strings</strong>. The only difference was mechanical: Jev received five native Noul objects, while GPT-5.4 returned five numbers via strict JSON Schema. GPT got no extra rubric, no chain-of-thought scaffolding, no advantage beyond what a typical LLM-as-a-judge setup already gets.</p><h3>What the Judges Saw</h3><p>A useful support evaluator needs the context necessary to assess both the action and the response. Both judges received the same structured state containing:</p><ul><li>the customer’s task and scenario;</li><li>the support policy and supplied evaluation criteria;</li><li>available tool definitions;</li><li>conversation messages;</li><li>tool names, arguments, and results;</li><li>the agent’s final response.</li></ul><p>Both judges also received the exact five question strings shown above. Neither received the stored benchmark reward or the benchmark evaluator’s output.</p><p>The two integrations used their respective interfaces: five native Noul questions for Jev and a strict output schema for GPT-5.4. GPT ran at medium reasoning effort, without an additional requested critique or chain-of-thought scaffold. This controls the evidence and questions; it does not establish that the chosen settings are the best possible configuration for either judge.</p><h3>Evaluation Setup</h3><h4>The Customer-Support Agent</h4><p>The target was a Claude Sonnet 4.5 retail support agent captured inside the <a href="https://sierra.ai/blog/benchmarking-agents-in-collaborative-real-world-scenarios">τ² benchmark environment</a>, with a simulated customer driven by GPT-5.2. The agent could authenticate customers, retrieve user and order records, calculate amounts, modify or cancel pending orders, process returns or exchanges on delivered items, and transfer to a human when a request fell outside its scope.</p><p>It also had to operate under policy: summarize any mutating action and get explicit confirmation before executing it, call at most one tool at a time, never invent unsupported information, and respect order-state restrictions.</p><p>The experiment fixed <strong>21 trajectories</strong> — 15 benchmark passes and 6 benchmark failures — before running any judge. Freezing the trajectories meant only the judges could introduce variation; the support agent and simulated customer were held constant.</p><p>Both judges saw the task definition and the full trajectory evidence for every case. Neither saw the stored benchmark reward or the benchmark’s own evaluator output.</p><h4>The Repeated-Judge Experiment</h4><p>Each judge scored every fixed trajectory <strong>50 times</strong>:</p><pre>21 cases × 50 repetitions × 2 judges = 2,100 evaluation calls<br>2,100 calls × 5 atomic questions   = 10,500 probability judgments</pre><p>Jev resolved to jev-1.13.0. The frontier comparison used the fixed gpt-5.4-2026-03-05 snapshot at medium reasoning effort.</p><h3>Results at a glance</h3><p><em>Each judge made 1,050 recorded calls. Costs are rounded. Agreement and Brier score use the benchmark reward as the reference; they do not establish agreement with human reviewers.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oSlCxR7VbMXvkCzZ-Lk-Pw.png" /></figure><h4>Accuracy</h4><p>On majority vote by case, Jev and GPT-5.4 tied: both matched the benchmark reward on <strong>19 of 21 cases (90.48%)</strong>.</p><p>Across all 1,050 repeated decisions per judge, Jev scored <strong>90.48%</strong> and GPT-5.4 scored <strong>90.76%</strong>. GPT’s marginal edge came from occasionally landing on the correct side of the threshold on cases where its verdict was already unstable — it didn’t translate into better majority-vote accuracy at the case level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KVYqU95ylmmpmonfCX88TQ.png" /></figure><p>GPT-5.4 was better calibrated: a task-success Brier score of <strong>0.0707</strong>, against <strong>0.1024</strong> for Jev. Accuracy and calibration measure different things — two judges can land on the same binary verdict while assigning very different confidence to it, and that’s exactly what happened here.</p><p>There’s a second, more important caveat. The τ² reward is an automated reference, not ground truth in any philosophical sense — and one case shows why that distinction isn’t academic.</p><p><strong>Case 9:</strong> the customer wanted to replace a piece of luggage with a coat and told the agent to cancel the order if that wasn’t possible. The agent asked for a cancellation reason; the customer explicitly selected ordered by mistake. The cancellation and refund succeeded, and the final response accurately reported what happened. Both judges called this trace a pass, in all 50 repetitions, for both models.</p><p>The benchmark scored it a failure — it expected the reason no longer needed and an additional order lookup that the actual customer instruction never required. This isn&#39;t proof the judges were right. It&#39;s proof that any evaluator comparison inherits the blind spots of its reference labels, and that a benchmark&#39;s expected trajectory and a customer&#39;s actual, successfully resolved request can quietly diverge.</p><h4>Score Stability</h4><p>Accuracy asks whether a judge agrees with the reference. Precision asks a narrower and, for production use, more consequential question: does the judge give the same score when <em>nothing</em> about the trace changes?</p><p>Jev’s probability variance was lower on every one of the five signals:</p><ul><li><strong>44.5×</strong> higher for GPT-5.4 on task success</li><li><strong>87.6×</strong> higher on policy compliance</li><li><strong>48.3×</strong> higher on tool correctness</li><li><strong>155.7×</strong> higher on response groundedness</li><li><strong>81.3×</strong> higher on customer communication</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PvKFZnxjrwJ4vUx43wcSow.png" /></figure><p>This experiment can’t establish <em>why</em> Jev varied less — only that it did, consistently, across every signal that was measured. One plausible hypothesis is architectural fit: Jev evaluates a bounded proposition directly, while GPT-5.4 reasons autoregressively before collapsing that reasoning into a number. Our data supports the fit; it doesn’t prove the causal story.</p><h4>Oscillation</h4><p>The aggregate variance stops being an abstraction in case 13, a multi-order return task where the evidence was byte-for-byte identical across all 50 repetitions.</p><p>Jev’s task-success probability stayed inside a <strong>0.64–0.75</strong> band and never crossed to the fail side of the threshold. GPT-5.4 ranged from <strong>0.11 to 0.99</strong> on the same evidence, producing 12 pass verdicts and 38 fail verdicts from a single, unchanging trace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AZuZNMlAtC0a3Gz4nnr-cg.png" /></figure><p>Across the full dataset, Jev never once changed its binary verdict on an identical trace. GPT-5.4 changed verdicts on four cases. Binary repeatability came out to <strong>1.000</strong> for Jev versus <strong>0.967</strong> for GPT-5.4.</p><p>That doesn’t mean repeatability equals correctness — Jev was perfectly consistent and still disagreed with the reference on two cases. A stable judge isn’t automatically a correct one; stability just makes the <em>disagreements</em> legible instead of burying them in noise.</p><h4>Cost and Latency</h4><p>The full experiment cost <strong>$42.31</strong> in recorded API usage. Jev accounted for <strong>$0.57</strong> of that; GPT-5.4 accounted for <strong>$41.73</strong> — roughly <strong>73× more</strong> for the same 1,050 evaluations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IGl6w9I_j6CSX7w1tEKCyA.png" /></figure><p>Jev also averaged <strong>0.99 seconds</strong> per call against GPT-5.4’s <strong>10.13 seconds</strong> — about <strong>10× faster</strong>.</p><p>Latency at this scale changes what evaluation is <em>for</em>. A sub-second judge fits inside a CI test suite or an interactive trace-review loop. A ten-second judge is workable offline, overnight, batched — but it’s structurally hard to put in anyone’s feedback loop.</p><h3>Online Evals, Unlocked at Scale</h3><p>At these observed rates, scoring 10,000 support traces once would cost approximately <strong>$5.47 with Jev</strong> and <strong>$397.46 with GPT-5.4</strong>. And that’s not five separate judge calls per trace — all five Noul questions are evaluated together against the same state in one call.</p><p>That cost profile changes what “denser monitoring” can mean in practice:</p><ul><li>evaluate a much larger share of production traces, not just a sample</li><li>track policy compliance as a signal independent from task completion</li><li>catch regressions in tool correctness before they show up in complaints</li><li>route low-confidence or internally contradictory cases to human review</li><li>reserve a frontier LLM judge — or a person — for the genuinely ambiguous tail</li></ul><p>The right production pattern is probably not “Jev or GPT.” It’s a cascade: Jev supplies cheap, typed signals on every single trace; deterministic checks catch exact invariants a decision model shouldn’t have to reason about; a slower reasoning judge or a human reviewer handles what’s left over.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oBYBnkuNA1o2KgJFNQ0-WA.png" /></figure><h3>A New Shape for Agent Evals</h3><p>Our results reproduce the most interesting finding from the original weather-agent experiment, in a domain with real operational stakes: state mutation, policy, and a human on the other end of the conversation. Jev matched a frontier model on case-level accuracy while producing dramatically more stable scores, at a fraction of the latency and cost.</p><p>But this experiment also surfaced the harder problem underneath the numbers: what should count as correct in the first place?</p><p>Case 9 shows that an automated benchmark can penalize a successful, policy-compliant action simply because its arguments differ from one expected trajectory. Jev and GPT-5.4 both evaluated the user-visible outcome as a success. Deciding which interpretation should count — the benchmark’s expected path, or the customer’s actual resolved outcome — is a product decision, sometimes a human-adjudication decision. It is not a problem a more confident model score can solve.</p><p>Typed decision models could make evaluation abundant. But abundant evaluation is only valuable when the questions are atomic, the evidence handed to the judge is complete, and the reference labels actually reflect the behavior you want from the agent — not just the one trajectory someone happened to write down.</p><p>The practical takeaway isn’t that Jev universally replaces LLM-as-a-judge. It’s that customer-support evaluation decomposes cleanly into explicit, bounded decisions — and a model built for exactly that kind of decision can deliver evaluation signal that’s stable and cheap enough to run on everything, all the time.</p><h3>Reproducibility</h3><p><a href="https://github.com/ayushworks/Jev">https://github.com/ayushworks/Jev</a></p><p>The recorded run used:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DVnfvwinRKtKicKrD5QsJA.png" /></figure><p>The run produced <strong>2,100 valid recorded judge responses</strong>, with no recorded technical failures. Raw responses, request metadata, token usage, latency, cost, question hashes, and analysis outputs were persisted during the experiment.</p><p>For interpreting or reproducing the results, the case selection, full evidence payloads, metric implementations, concurrency settings, and handling of caching and retries matter alongside model versions. The results describe judge variability on frozen trajectories; they do not include variability from rerunning the customer-support agent itself.</p><h3>References</h3><ul><li><a href="https://www.langchain.com/blog/jev-agent-evals-langsmith">LangChain: Jev-as-a-Judge for Agent Evals</a></li><li><a href="https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/judges/system_one.py">LangChain experiment source: Jev evaluator</a></li><li><a href="https://github.com/sierra-research/tau2-bench">τ² benchmark repository</a></li></ul><p><em>Disclosure: This is a small, fixed-benchmark study. It should not be read as a general ranking of judge models, or as evidence that lower variance causes higher accuracy.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4af637a9bd15" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing an AI assistant for transactional insights]]></title>
            <link>https://ayushm4489.medium.com/designing-an-ai-assistant-for-transactional-insights-7090e23852f2?source=rss-2839f9309087------2</link>
            <guid isPermaLink="false">https://medium.com/p/7090e23852f2</guid>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[agentic-ai-architecture]]></category>
            <dc:creator><![CDATA[ayush mittal]]></dc:creator>
            <pubDate>Sat, 15 Aug 2026 22:20:20 GMT</pubDate>
            <atom:updated>2026-08-16T12:43:58.968Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Why transaction data, financial calculations, and predictive models should remain outside the language model.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l_k4T_zUy6xWYNab" /><figcaption>Photo by <a href="https://unsplash.com/@sharonmccutcheon?utm_source=medium&amp;utm_medium=referral">Alexander Grey</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>AI agents are becoming a new interface for financial services</h3><p>Financial institutions have used chatbots for years, but most of those systems were conversational menus. They answered frequently asked questions, retrieved static information, or routed customers into an existing application flow. The new generation of AI agents is designed around a broader operating model: interpret a customer’s goal, retrieve customer-specific data through authenticated tools, coordinate several services, and initiate an action when policy allows it.</p><p>In financial services, these agents are beginning to appear across three types of work:</p><ul><li><strong>Service and navigation</strong>, such as locating statements, explaining card controls, or finding the correct product flow.</li><li><strong>Analysis and decision support</strong>, such as searching transactions, explaining spending, identifying recurring payments, or summarizing an investment portfolio.</li><li><strong>Controlled execution</strong>, such as blocking a card, making transactions or preparing a payment for explicit customer confirmation.</li></ul><p>The third category is what separates an agent from a more capable chatbot. An agent does not only generate an answer; it can select and invoke a governed capability. In banking, however, every action sits behind authorization, eligibility rules, confirmation requirements, transaction limits, and an audit trail. The language model may decide which capability is relevant, but it cannot become the control system for that capability.</p><p>Revolut’s AIR is a useful public example. <a href="https://www.revolut.com/news/revolut_enters_new_era_of_money_intelligence_with_launch_of_ai_assistant/">Revolut launched AIR in April 2026</a> as an in-app assistant for spending insights, investments, subscriptions, cards, travel, and selected actions. Rather than requiring customers to remember which screen contains a feature, AIR lets them describe an intended outcome in conversation.</p><figure><img alt="Revolut AIR product image. Source: Craft Innovations’ independent UX analysis; rights remain with the respective owners." src="https://cdn-images-1.medium.com/max/1024/1*9Rf9I-2uU31RoVLwbWF2cg.png" /></figure><p>An independent UX review by Craft Innovations found a meaningful difference between <strong>finding a capability</strong> and <strong>executing it</strong>. AIR reduced friction for feature discovery and some multi-step workflows, but conventional navigation was still faster for many routine banking tasks. The review also observed that AIR often guided users into existing application flows, while a smaller set of scenarios — such as card freezing and eSIM purchase — behaved more like end-to-end agent workflows.</p><p>This distinction is central to the system design. A financial agent needs natural-language reasoning at the interface, but its data retrieval, calculations, permissions, and actions still require conventional engineering guarantees.</p><h3>The engineering problem behind a simple answer</h3><p>A money insights agent should answer questions such as:</p><ul><li>How much did I spend on coffee last month?</li><li>What percentage of my food budget went to coffee?</li><li>Which subscriptions are still charging me?</li><li>How much am I likely to save this month?</li></ul><p>These questions look conversational, but they are data and computation problems. The answer must use the customer’s actual transactions, apply the correct financial definitions, respect the requested time period and currency, and expose enough evidence to be checked. <strong>A fluent response with the wrong amount is still a failed banking product</strong>.</p><p>That requirement determines the architecture. A large language model is useful for interpreting an ambiguous request, selecting a capability, and explaining a result. It should not hold the raw transaction history or replace the aggregation engine, forecasting service, or policy layer.</p><h3>The design constraint: every amount must be attributable</h3><p>In a general-purpose assistant, a small factual error may be inconvenient. In a financial product, a numerical error can change a customer’s decision and damage trust in the entire service. The system therefore needs stronger guarantees than “the answer usually looks reasonable.”</p><p>Every amount shown to the customer should be attributable to:</p><ul><li>a defined source and account scope;</li><li>a time range with an explicit timezone;</li><li>a currency and, when relevant, an exchange-rate policy;</li><li>a versioned calculation or forecasting method;</li><li>the transactions or aggregates used in the calculation; and</li><li>a freshness timestamp and completeness indicator.</li></ul><p>This is a data-lineage requirement, not a prompt-engineering problem. If the system cannot reconstruct how it produced €184.60, the amount should not appear in the answer.</p><h3>Why putting raw transactions in the prompt fails</h3><p>Consider the question, “How much can I save this month?” The current month is incomplete, so the answer is not present in any single transaction. The system must distinguish regular income from refunds and internal transfers, calculate spending to date, estimate remaining committed and discretionary spending, and state the uncertainty of the estimate.</p><p><strong>A common prototype copies several months of transactions into the prompt and asks the language model to perform all of those steps at once.</strong> This combines classification, aggregation, temporal reasoning, forecasting, and response generation inside a probabilistic next-token process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z23xEP0hLmyQuWXSnEQbDA.png" /></figure><p>The approach may answer a toy example correctly. <strong>At production scale, however, it cannot guarantee completeness, numerical exactness, freshness, or reproducibility — even when every transaction fits inside the model’s context window.</strong></p><p>A financial query such as “How much did I spend last month?” is not just a summation. It requires the system to select the correct accounts and time range, distinguish posted transactions from pending authorisations, reconcile refunds and reversals, normalize currencies, apply category definitions, and include every qualifying transaction exactly once. A database query or domain service can enforce these rules as explicit invariants. An autoregressive language model processing serialized text cannot.</p><p>The risk increases with the number and diversity of transactions. Relevant entries may be distributed across a long prompt and surrounded by merchant aliases, duplicated authorisations, split settlements, internal transfers, and foreign-exchange records. Research on long-context models has also shown that fitting information inside the context window does not guarantee reliable use of that information: retrieval performance can vary depending on where relevant evidence appears in the prompt. This is commonly described as the <a href="https://aclanthology.org/2024.tacl-1.9/?utm_source=chatgpt.com">“lost in the middle” problem</a>.</p><p>There is also an operational scaling problem. Prompt size, inference latency, and token cost grow with the amount of history sent on every request. The resulting answer has no stable query plan, completeness guarantee, or calculation version, and the copied transaction snapshot may already be stale. In this context, “failure at production scale” does not mean every answer will be wrong. It means the system cannot provide the guarantees required for a financial product.</p><h4>Transaction classification becomes implicit</h4><p>Salary may be one or two credits among thousands of card payments, transfers, chargebacks, cash deposits, and refunds. If an internal transfer is classified as income, the projected surplus is overstated. If a salary payment is missed, it is understated. When classification happens inside the prompt, there is no reusable income definition, confidence score, or decision record to inspect.</p><h4>Arithmetic becomes probabilistic</h4><p>Language models can reproduce arithmetic patterns, but they are not reliable aggregation engines over long, irregular inputs. They may omit a row, count a reversal twice, misread a debit as a credit, or combine currencies without applying a documented conversion rule.</p><p>A larger context window increases capacity; it does not create database semantics or exact decimal arithmetic.</p><h4>An incomplete month requires a real forecast</h4><p>Transactions observed by the 15th do not contain the second half of the month. Rent, direct debits, seasonal spending, payday cadence, and the number of remaining days must be represented explicitly.</p><p>Applying a generic rule such as “save 20% of income” to an unreliable total produces advice, not a customer-specific forecast.</p><h4>Recommendations become ungrounded</h4><p>The wording of a savings question can encourage the model to recommend cancelling a streaming service or reducing coffee purchases even when it has not detected a recurring charge or calculated the relevant merchant total.</p><p>Recommendations should be produced from verified subscriptions, merchant aggregates, and explicit product rules, not from associations learned during pretraining.</p><h4>Privacy, cost, and auditability deteriorate</h4><p>Sending a year of transaction descriptions on every request increases token cost and latency while exposing more personal data than the answer requires.</p><p>It also produces a weak audit trail: the system retains a prompt and a response, but not a deterministic query plan, calculation version, or reliable set of contributing records.</p><p>The raw-transactions-in-prompt pattern therefore assigns the most sensitive and exact work to the least deterministic component.</p><h3>A better design — Separate truth, computation, and language</h3><p>A production design separates five responsibilities: the canonical transaction store, read models, predictive services, governed tools, and the language-model orchestrator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*25lrPN-KfXw7_UXTE7hFOw.png" /></figure><h4>1. The transaction ledger is the system of record</h4><p>In banking architecture, the canonical transaction store is usually called the <strong>transaction ledger/system of records</strong>. It contains the authoritative monetary events.</p><p>A useful transaction contract includes more than a date, merchant, and amount. It normally distinguishes:</p><ul><li>pending from posted entries;</li><li>debits from credits;</li><li>authorisations from settlements;</li><li>refunds from reversals;</li><li>original from canonical merchant names;</li><li>account and card identifiers;</li><li>transaction currency from settlement currency;</li><li>event timestamps from posting timestamps.</li></ul><p>This system of record should preserve financial history rather than overwrite it. Corrections are represented as new events or linked reversals so that downstream projections can be rebuilt and audited.</p><p>Access is constrained by customer, account, legal entity, and data-retention policy. <strong>The language model does not query this store freely</strong>. It reaches the financial data plane through narrowly scoped services that enforce authorization and purpose limitation.</p><h4>2. Read models provide query-oriented views</h4><p>Most conversational questions do not require transaction-level retrieval. They need monthly cash flow, category totals, merchant totals, budget consumption, or detected recurring payments.</p><p><strong>These can be maintained as materialized views or read models, following the same separation used in CQRS architectures</strong>.</p><p>An event stream or change-data-capture pipeline updates projections such as:</p><ul><li>monthly_cashflow(customer_id, month, regular_income, total_outflow)</li><li>category_spend(customer_id, period, category_id, amount)</li><li>merchant_spend(customer_id, period, merchant_id, amount)</li><li>budget_consumption(customer_id, period, budget_id, used_amount)</li><li>recurring_series(customer_id, merchant_id, cadence, next_expected_date)</li></ul><p>The projection layer should implement the same financial definitions used by the bank’s spending and budgeting screens. Creating a second interpretation of “income,” “spend,” or “subscription” for the agent will eventually produce contradictory figures across channels.</p><p>Read models also provide predictable latency and data minimization. A six-month cash-flow query may return 12 decimal values and a few status fields instead of thousands of transaction descriptions.</p><h4>3. Predictive services produce versioned features and estimates</h4><p>Some facts cannot be obtained through SQL aggregation alone. Salary detection, merchant normalization, recurring-payment detection, and end-of-month spending forecasts may require rules, statistical models, or machine-learning inference.</p><p>These models should run as separate batch or streaming pipelines. Their outputs become versioned features or projections with confidence, observation windows, and model identifiers.</p><p>For example, a recurrence detector might store:</p><ul><li>the expected cadence;</li><li>median amount;</li><li>amount variance;</li><li>last observation;</li><li>next expected date; and</li><li>a calibrated probability that the series will recur.</li></ul><p>This distinction is important: <strong>a model that learns patterns from transaction sequences is an analytical model</strong>. The chat model is an interaction model. They have different training objectives, latency profiles, evaluation metrics, and risk controls.</p><h4>4. Governed tools expose typed financial capabilities</h4><p>Tools are the contract between the conversational control plane and the financial data plane. Each tool should perform one coherent operation, accept a bounded schema, enforce authorization, and return a typed result.</p><p>A transaction-search result might look like this:</p><pre>{<br>  &quot;query_id&quot;: &quot;q_7f91&quot;,<br>  &quot;period&quot;: {<br>    &quot;start&quot;: &quot;2026-07-01&quot;,<br>    &quot;end&quot;: &quot;2026-07-31&quot;,<br>    &quot;timezone&quot;: &quot;Europe/Amsterdam&quot;<br>  },<br>  &quot;match_definition&quot;: &quot;merchant_or_description:coffee&quot;,<br>  &quot;currency&quot;: &quot;EUR&quot;,<br>  &quot;total&quot;: &quot;46.80&quot;,<br>  &quot;matched_count&quot;: 9,<br>  &quot;returned_count&quot;: 9,<br>  &quot;is_complete&quot;: true,<br>  &quot;transactions&quot;: [<br>    {<br>      &quot;id&quot;: &quot;txn_123&quot;,<br>      &quot;date&quot;: &quot;2026-07-04&quot;,<br>      &quot;merchant&quot;: &quot;Coffee Company&quot;,<br>      &quot;amount&quot;: &quot;5.20&quot;<br>    }<br>  ],<br>  &quot;data_as_of&quot;: &quot;2026-08-01T06:12:44Z&quot;,<br>  &quot;calculation_version&quot;: &quot;merchant-search-v3&quot;<br>}</pre><p>Amounts should use decimal or minor-unit representations, not binary floating point. The contract should distinguish a zero result from missing data, return explicit truncation metadata, and expose the calculation version.</p><p>A tool that silently returns the first 100 matches creates an apparently precise but incorrect answer.</p><p>For derived metrics, prefer a domain service that performs the final calculation. The orchestrator may request a coffee total and a food-budget total, but code should calculate the percentage, define the denominator, and handle division by zero.</p><p>The language model explains the result rather than recomputing it from prose.</p><h4>5. The language model orchestrates and explains</h4><p>The language model performs a deliberately narrow set of tasks:</p><ul><li>resolve the user’s intent and ambiguous terms;</li><li>select the appropriate read-only or action tool;</li><li>collect missing parameters when required;</li><li>compose tool calls across a multi-step request;</li><li>describe definitions, assumptions, and uncertainty; and</li><li>format the result for chat, cards, tables, or charts.</li></ul><p>It should not invent missing fields, adjust tool output to make it look plausible, or state an amount that has no source in the tool trace.</p><p>This can be enforced through response schemas, grounding checks, and post-generation validation.</p><h3>Four query patterns cover most spending questions</h3><p>The chat interface makes many requests look similar, but the data path depends on the question type.</p><h4>Transaction search</h4><p>“How much did I spend on coffee last month?” requires entity resolution and transaction-level matching. Coffee is a concept, not necessarily a transaction category.</p><p>The search service may use canonical merchant identifiers, merchant-category codes, transaction descriptions, and a curated concept-to-merchant mapping. It should aggregate matched amounts in code and return the contributing transactions.</p><p>The agent should disclose the matching rule when it affects interpretation. If “coffee” was matched through merchant names and descriptions, the answer should not imply that it came from an official Coffee category.</p><figure><img alt="Craft Innovations’ test shows AIR disclosing its coffee-matching method and then combining that result with category spending. Screenshots: Craft Innovations, reproduced for commentary; © Craft Innovations." src="https://cdn-images-1.medium.com/max/1024/1*ssBSG71LkOq8YhSHLkkrGQ.png" /></figure><h4>Aggregate lookup</h4><p>“Show my spending by category” should query a category-spend projection. Reading and regrouping individual transactions would add latency and create another implementation of category logic.</p><p>A drill-down such as “Which merchants make up Shopping?” is a separate query against merchant-level aggregates or transaction search.</p><h4>Derived metric</h4><p>“What percentage of my food budget was coffee?” joins two definitions: coffee identified through merchant matching and food represented by a configured budget or category taxonomy.</p><p>The calculation service should return the numerator, denominator, percentage, and both definitions. This makes semantic mismatches visible instead of hiding them behind a polished sentence.</p><h4>Forecast</h4><p>“How much can I save this month?” requires a forecasting path, not a larger prompt. The system retrieves:</p><ul><li>regular income;</li><li>current spending;</li><li>scheduled obligations;</li><li>historical discretionary spending; and</li><li>the remaining calendar window.</li></ul><p>A versioned planner then returns a point estimate or range with assumptions.</p><figure><img alt="The forecast service owns the calculation and uncertainty. The language model owns the explanation." src="https://cdn-images-1.medium.com/max/1024/1*h1sz41Di_J8XZjXlHpTdFw.png" /></figure><p><em>The forecast service owns the calculation and uncertainty. The language model owns the explanation.</em></p><p>A robust forecast contract should include the model version, data cutoff, expected income, committed outflows, projected discretionary spend, estimated surplus, prediction interval, and warnings for sparse or abnormal history. If confidence is low, the product should say so directly rather than converting uncertainty into confident prose.</p><p>Recommendations about where to cut spending should be a separate capability. A recommendation engine can rank verified subscriptions, unusually high merchant spending, and budget deviations under explicit policy. Without that capability, the agent should explain the forecast and stop there.</p><h3>Recurring payments belong in an offline detection pipeline</h3><p>Subscription detection is primarily a sequence-classification problem. The service must identify repeated merchant relationships despite variable billing dates, price changes, free trials, annual renewals, payment-processor aliases, and occasional failed charges.</p><p>Rule-based detection can cover stable cases by grouping normalized merchants and measuring inter-arrival intervals and amount variance. More difficult cases may use a classifier or sequence encoder. Useful features include merchant identity, amount percentiles, interval statistics, transaction channel, card-on-file indicators, and the number of consecutive observations.</p><figure><img alt="Detection runs before the conversation. Listing is a read operation; cancellation is a separately authorized action." src="https://cdn-images-1.medium.com/max/1024/1*cJ9_D-IVyBq-QvHwoco9Gg.png" /></figure><p><em>Detection runs before the conversation. Listing is a read operation; cancellation is a separately authorized action.</em></p><p>The detector writes a <em>recurring_series</em> projection. At question time, “Which subscriptions are still charging me?” is a low-latency lookup over active series, not an invitation for the language model to scan a year of payments.</p><p>Listing and cancelling also require different trust boundaries. Listing is read-only. Cancellation changes a financial arrangement and should require an explicit customer confirmation, a fresh authorization check, an idempotency key, and a durable action receipt. In many products, the safest design is to deep-link into an existing cancellation flow while the agent preserves context.</p><h3>Reliability requires explicit contracts</h3><p>Tool use alone does not make a financial agent reliable. The surrounding contracts determine whether failures are visible and recoverable.</p><h4>Provenance and explainability</h4><p>Persist the user request, interpreted parameters, tool calls, tool versions, data timestamps, returned identifiers, and final response. This produces an audit record without storing unrestricted transaction data in model prompts. When the user opens “How was this calculated?”, the product can show the definition and relevant transactions from the same query result.</p><h4>Completeness and pagination</h4><p>Every list result needs matched_count, returned_count, and is_complete, or an equivalent contract. Aggregations must run over the full authorized result set even when only a small page of evidence is returned to the interface.</p><h4>Temporal and currency semantics</h4><p>Terms such as “last month” depend on timezone and account locale. Multi-currency totals require a documented choice between transaction-time, settlement-time, or current exchange rates. Pending, reversed, and disputed transactions should be included or excluded by an explicit policy.</p><h4>Authorization and data minimization</h4><p>Authorization should be enforced inside each tool, not delegated to the prompt. The model receives only the fields required for the current task, with sensitive descriptions masked when they do not contribute to the answer. Tool scopes should distinguish account read, transaction search, budgeting, card controls, and payment actions.</p><h4>Safe actions</h4><p>Any operation that blocks a card, cancels a payment, changes a budget, or moves money needs confirmation and replay protection. Separate “prepare” from “execute”: the first call returns a human-readable summary and confirmation token; the second executes only after explicit approval. Idempotency keys prevent duplicate execution when the client retries.</p><figure><img alt="AIR identifies the card and requests confirmation before freezing it. Screenshots: Craft Innovations, reproduced for commentary; © Craft Innovations." src="https://cdn-images-1.medium.com/max/1024/1*HOwxZv3FglYnyC_cUl9Ugg.png" /></figure><p><em>AIR identifies the card and requests confirmation before freezing it. Screenshots: </em><a href="https://craftinnovations.global/revolut-air-ux-analysis/"><em>Craft Innovations</em></a><em>, reproduced for commentary; © Craft Innovations.</em></p><h3>Product behaviour is not system architecture</h3><p>AIR’s public behaviour illustrates why an assistant needs both read-oriented analytics and separately controlled actions.</p><p>The Craft Innovations review provides useful evidence about the user experience:</p><ul><li>transaction search;</li><li>calculation explanations;</li><li>contextual follow-ups;</li><li>deep links;</li><li>confirmation steps; and</li><li>the boundary between guidance and execution.</li></ul><p>Those observations do not reveal AIR’s backend architecture. Neither the public product announcement nor the independent UX review documents its source-of-truth systems, aggregation services, tool schemas, authorization model, or orchestration runtime.</p><p>It would therefore be incorrect to infer that AIR calculates financial amounts inside a language model — or that it uses the exact architecture proposed in this article.</p><p>Revolut has separately published research on <strong>PRAGMA</strong>, a family of encoder models trained on sequences of banking events. Its evaluated downstream tasks include credit scoring, fraud detection, product recommendation, and recurrent-transaction prediction.</p><p>That is an example of a domain model producing representations and predictions from financial event sequences.</p><p>The distinction is instructive. A transaction foundation model can improve recurrence detection or other predictive services, while a conversational model can interpret requests and present results.</p><p>Public information does not establish that AIR uses PRAGMA directly, and the architecture does not require that connection. The general design remains the same: predictions become governed, versioned data products before they are exposed through conversational tools.</p><h3>Evaluate the system, not only the model</h3><p>An agent evaluation that scores only response style will miss the failures that matter. A production test suite should cover the complete execution trace.</p><p>At minimum, evaluate:</p><ul><li><strong>Intent and parameter resolution:</strong> correct account, period, currency, merchant, and category.</li><li><strong>Tool selection:</strong> the right capability is chosen without unnecessary data access.</li><li><strong>Numerical exactness:</strong> response amounts equal the authoritative tool fields.</li><li><strong>Completeness:</strong> truncation and partial-data conditions are disclosed.</li><li><strong>Groundedness:</strong> every factual financial claim maps to tool output.</li><li><strong>Forecast calibration:</strong> prediction intervals match observed error rates.</li><li><strong>Action safety:</strong> confirmation, authorization, idempotency, and cancellation paths work.</li><li><strong>Operational performance:</strong> freshness, latency, tool error rate, token use, and cost.</li></ul><p>Create a golden set of representative questions with authoritative answers, but also test adversarial cases:</p><ul><li>refunds that resemble income;</li><li>duplicate authorisations;</li><li>cross-currency purchases;</li><li>late settlements;</li><li>renamed merchants;</li><li>sparse history; and</li><li>follow-up questions that change the time range.</li></ul><p>Trace-level evaluation makes it possible to distinguish an intent error from a data-quality error, a projection defect, or a response-generation defect.</p><h3>A practical implementation order</h3><p>The chat interface should be built after the financial semantics are stable enough to support it.</p><ol><li><strong>Define canonical data contracts.</strong> Specify transaction states, merchant identity, time boundaries, currency handling, reversals, and account scope.</li><li><strong>Build reusable read models.</strong> Start with monthly cash flow, category totals, merchant totals, budgets, and recurring series. Reuse existing banking-app services where possible.</li><li><strong>Expose a small tool surface.</strong> A transaction search, aggregate lookup, forecast, and recurring-payment lookup cover more value than dozens of loosely defined tools.</li><li><strong>Add provenance and evaluation.</strong> Version calculations, retain tool traces, define completeness semantics, and create golden test cases before expanding the conversational scope.</li><li><strong>Introduce the language-model orchestrator.</strong> Constrain it to intent resolution, tool composition, clarification, and explanation. Validate generated financial claims against the tool trace.</li><li><strong>Add actions last.</strong> Use separate permissions, explicit confirmation, idempotency, and durable receipts for every state-changing operation.</li></ol><p>This order also improves organizational ownership. Data teams own the transaction store and projections, ML teams own predictive services, domain teams own financial definitions, security teams own authorization policy, and the conversational team owns orchestration and presentation.</p><p>The boundaries make defects easier to locate and changes easier to govern.</p><h3>Conclusion</h3><p>The difficult part of a money insights agent is not generating natural language. It is preserving financial semantics while a probabilistic model participates in the interaction.</p><p>Keep raw transactions in the authoritative transaction store, compute aggregates and forecasts in deterministic or separately evaluated services, expose them through typed and authorized tools, and require provenance for every amount.</p><p>The language model can then do what it is well suited to do: interpret the customer’s question and explain an answer produced by systems that can be tested, audited, and trusted.</p><h3>References</h3><ul><li><a href="https://www.revolut.com/news/revolut_enters_new_era_of_money_intelligence_with_launch_of_ai_assistant/">Revolut: launch of the AIR AI assistant</a></li><li><a href="https://craftinnovations.global/revolut-air-ux-analysis/">Craft Innovations: Revolut AIR UX analysis</a></li><li><a href="https://arxiv.org/abs/2604.08649">PRAGMA: Revolut Foundation Model</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7090e23852f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling AI Agents with Gemini Enterprise Agent Platform]]></title>
            <link>https://ayushm4489.medium.com/scaling-ai-agents-with-gemini-enterprise-agent-platform-18e28318ac94?source=rss-2839f9309087------2</link>
            <guid isPermaLink="false">https://medium.com/p/18e28318ac94</guid>
            <category><![CDATA[google-gemini]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[ayush mittal]]></dc:creator>
            <pubDate>Sat, 09 May 2026 21:50:07 GMT</pubDate>
            <atom:updated>2026-05-09T21:55:41.889Z</atom:updated>
            <content:encoded><![CDATA[<h4>A first look at Agent runtime, Agent sessions, Agent sandbox and Agent Memory bank after Google Next ’26 announcement.</h4><p><a href="https://cloud.google.com/blog/products/ai-machine-learning/introducing-gemini-enterprise-agent-platform">Google Enterprise Agent Platform</a> is Google Cloud’s framework for building, deploying, and operating AI agents at production scale. Announced at Google Cloud Next ’26 in April 2026, Gemini Enterprise Agent Platform is the evolution of Vertex AI — Google Cloud’s new comprehensive platform to build, scale, govern, and optimise agents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vXmkSnWEYBvNvdUQN0h1Rg.jpeg" /></figure><p>The platform is organised into four pillars: <strong>Build</strong> (Agent Garden, ADK, MCP Servers, RAG Engine, Vector Search), <strong>Scale</strong> (Agent Runtime, Sessions, Memory Bank, Agent Sandbox), <strong>Govern</strong> (Agent Registry, Policies, Gateways), and <strong>Optimize</strong> (Evaluation, Monitoring, Cost controls). It gives you access to over 200 models through Model Garden — including Gemini 3.1 Pro, Gemma 4, and third-party models like Claude Opus, Sonnet, and Haiku. The goal, as Google put it, is to help teams move from managing individual AI tasks to delegating business outcomes with total confidence.</p><p>This post focuses entirely on the <strong>Scale</strong> layer — the part that transforms a prototype into something you’d actually ship to customers. The Build layer gets most of the attention in tutorials and documentation. The Scale layer is where the real engineering decisions happen, where the gaps between a demo and a production system become visible, and where things get genuinely interesting (or difficult depending on which side of spectrum you are in :P).</p><h3>The Demo Agent: CloudStorage Support</h3><p>To make this concrete, we built <strong>Maya</strong> — an AI support agent for a fictional cloud storage product. The stack is Google ADK and Vertex AI Agent Runtime, deployed to us-central1.</p><p>Maya is a three-agent system. support_agent is the root orchestrator — it handles customer identification, ticket lookup, and the full conversation. It delegates to diagnostic_agent (a technical specialist that analyses logs and config files by executing Python) and sla_agent (which calculates ticket compliance against SLA thresholds using sandboxed Python arithmetic). Both sub-agents are wired in via ADK&#39;s AgentTool pattern, so the root agent gets results back directly without a session transfer.</p><p>The UI is a static frontend backed by a FastAPI server (server.py). The browser calls POST /sessions to create a Vertex AI session, then opens a WebSocket to WS /ws/{session_id}. Each user message is forwarded to the deployed engine via streaming_agent_run_with_events(), and response tokens are streamed back to the browser as they arrive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bxRTUWcVqqqAAuvH7M29jQ.png" /></figure><h3>The Four Scale Blocks</h3><h3>1. Agent Runtime</h3><p>Agent Runtime is GEAP’s managed hosting layer. You give it your agent code, a list of dependencies, and any extra files — it handles containerisation, scaling, and serving. What the documentation doesn’t prepare you for is that getting a multi-agent system deployed correctly is a three-step process, and the steps depend on each other in a specific order.</p><p>Our setup has four Python modules that need to travel into the cloud together: agent.py (the root support_agent), diagnostic_agent.py (a specialist agent for deeper issue analysis), sla_agent.py (an SLA checker specialist an with a code executor), and tools.py (shared Firestore tools). They reference each other, share infrastructure, and depend on environment variables that are only available after the engine is already running. Getting this right takes a four-step deploy.</p><p><strong>Step 1 — Wrapping the agent in an AdkApp</strong></p><p>Everything that goes to Agent Runtime is wrapped in an AdkApp. This is the object that gets serialized and shipped. The important decision here is how you pass in the session and memory services — as builder functions, not as instances.</p><pre>def session_service_builder():<br>    engine_id = os.environ.get(&quot;GOOGLE_CLOUD_AGENT_ENGINE_ID&quot;)<br>    return VertexAiSessionService(<br>        project=PROJECT,<br>        location=LOCATION,<br>        agent_engine_id=engine_id,<br>    )<br><br><br>def memory_service_builder():<br>    engine_id = os.environ.get(&quot;GOOGLE_CLOUD_AGENT_ENGINE_ID&quot;)<br>    return VertexAiMemoryBankService(<br>        project=PROJECT,<br>        location=LOCATION,<br>        agent_engine_id=engine_id,<br>    )<br><br>adk_app = AdkApp(<br>    agent=support_agent,<br>    session_service_builder=session_service_builder,<br>    memory_service_builder=memory_service_builder,<br>)</pre><p>The reason for builders over instances is that GOOGLE_CLOUD_AGENT_ENGINE_ID — the engine&#39;s own resource ID — is only injected as an environment variable at runtime inside the container. If we created a VertexAiSessionService or VertexAiMemoryBankServiceinstance at packaging time, it would be serialized into the pickle with no engine ID. The builders are callables and are invoked when the container starts, at which point the environment is fully populated.</p><p><strong>Step 2 — Creating the engine</strong></p><pre>agent_engine = client.agent_engines.create(<br>    agent=adk_app,<br>    config={<br>        &quot;staging_bucket&quot;: STAGING_BUCKET,<br>        &quot;requirements&quot;: [<br>            &quot;google-adk&gt;=1.32.0&quot;,<br>            &quot;python-dotenv&gt;=1.0.0&quot;,<br>            &quot;google-cloud-firestore&gt;=2.19.0&quot;,<br>        ],<br>        &quot;extra_packages&quot;: [<br>            &quot;agent.py&quot;,<br>            &quot;tools.py&quot;,<br>            &quot;diagnostic_agent.py&quot;,<br>            &quot;sla_agent.py&quot;,<br>        ],<br>        &quot;env_vars&quot;: {<br>            &quot;FIRESTORE_PROJECT_ID&quot;: PROJECT,<br>            # SANDBOX_NAME intentionally omitted — added in Step 3<br>        },<br>    }<br>)</pre><p>The extra_packages list is what makes the multi-agent setup work. Without it, the cloudpickle would contain the root agent but the sub-agents would be missing at runtime. Every module your agent imports that isn&#39;t in requirements needs to be listed here explicitly.</p><p>When this call returns, the SDK has serialised the AdkApp, uploaded the .pkl, requirements.txt, and a .tar.gz of the extra modules to GCS, and provisioned a managed container. One thing worth knowing: the container runs <strong>Python 3.14</strong> — not whatever version is in your local virtual environment. This is the source of truth for runtime behaviour.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xbugFYeE3Wji_gJdxWRUPg.png" /></figure><p>The Query URL refers to this resource as a reasoningEngine. We can see /reasoningEngines/{id} in URLs and in the Identity tab. The Agent Platform UI calls it an &quot;Agent Engine&quot;. They&#39;re the same thing and the naming is just a holdover from the Vertex AI era. Like many other big organizations, sales and engineering teams are not always in sync :)</p><p><strong>Step 3 — Creating the Agent Sandbox</strong></p><p>With the engine ID in hand, we create an Agent Sandbox scoped to it. The sandbox is what powers the SLA agent’s code executor — we’ll go into detail in section 4, but the creation happens here because we need the sandbox resource name before we can finish configuring the engine.</p><pre>operation = client.agent_engines.sandboxes.create(<br>    name=engine_resource,<br>    spec={&quot;code_execution_environment&quot;: {}},<br>    config=CreateAgentEngineSandboxConfig(<br>        display_name=&quot;sla-checker-sandbox&quot;,<br>    ),<br>)<br>sandbox_name = operation.response.name</pre><p>engine_resource is the fully-qualified resource name returned by Step 2 — scoping the sandbox to the engine means it shares the same identity and IAM boundary. Once the operation completes, sandbox_name is the resource name we need to inject back into the engine.</p><p><strong>Step 4 — Patching SANDBOX_NAME without re-deploying</strong></p><p>SANDBOX_NAME is deliberately absent from the engine&#39;s env_vars in Step 2 — the sandbox doesn&#39;t exist yet. But you also can&#39;t pass an empty string; the API returns 400 INVALID_ARGUMENT: Required field is not set. So the env var gets added last, after the sandbox is created.</p><p>The natural instinct is to call the SDK’s update() method. The problem is that passing agent=adk_app triggers a full re-packaging and re-deploy — the entire cloudpickle process runs again. We only need to update one env var. So instead, we can call the Vertex AI REST API directly with an updateMask:</p><pre>resp = http_requests.patch(<br>    patch_url,<br>    headers={&quot;Authorization&quot;: f&quot;Bearer {credentials.token}&quot;},<br>    json={<br>        &quot;spec&quot;: {<br>            &quot;deploymentSpec&quot;: {<br>                &quot;env&quot;: [<br>                    {&quot;name&quot;: &quot;FIRESTORE_PROJECT_ID&quot;, &quot;value&quot;: PROJECT},<br>                    {&quot;name&quot;: &quot;SANDBOX_NAME&quot;,          &quot;value&quot;: sandbox_name},<br>                ]<br>            }<br>        }<br>    },<br>    params={&quot;updateMask&quot;: &quot;spec.deploymentSpec.env&quot;},<br>)</pre><p>The updateMask tells the API to touch only spec.deploymentSpec.env. Nothing else is re-deployed. In the console, the &quot;Updated&quot; timestamp on the Deployment details page reflects this surgical patch — the Created and Updated timestamps will be different, which is exactly what you want to see.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SiIUKksvPRVlKLquT68aXQ.png" /></figure><p><strong>Dashboard Tab</strong></p><p>The Dashboard tab shows live request metrics — query rate and latency percentiles (p50/p95/p99). During our dummy sessions, the request count chart shows clear spikes that map exactly to each conversation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZKj66tqZ-iGwvq9dJ0ltiw.png" /></figure><h3>2. Sessions</h3><p>A session is the complete record of a single conversation — every user message, every agent response, every tool call and its output, in order. Within a conversation, the session is what makes the agent aware of what was said three turns ago. Without it, every message arrives in a vacuum.</p><p>GEAP gives you two ways to back sessions. InMemorySessionService keeps everything in process memory — fast, zero setup, the right choice for local development. VertexAiSessionService stores sessions in Google Cloud, where they survive container restarts, scale across instances, and appear in the console. For production there is no real choice; it&#39;s Vertex AI.</p><p>You wire this into the deployed agent using the same builder pattern from the deployment step — a callable invoked at container startup, not at packaging time:</p><pre>def session_service_builder():<br>    engine_id = os.environ.get(&quot;GOOGLE_CLOUD_AGENT_ENGINE_ID&quot;)<br>    return VertexAiSessionService(<br>        project=PROJECT,<br>        location=LOCATION,<br>        agent_engine_id=engine_id,<br>    )</pre><p><strong>Sessions and Sessions Playground</strong></p><p>The Sessions tab lists every conversation by session ID and user. The Playground tab lets you click into any session and replay the full turn-by-turn history — user messages, agent responses, and every tool call in between. It also lets you start a new session directly against the deployed engine, which is the cloud equivalent of adk web for testing without writing client code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vVNi_HDMlgKzb7WscLCaYw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Phf9haI7Mn6OHBoZuz4i3w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vPOvy8jxksKBsWRtImnjqQ.png" /></figure><h3>3. Memory Bank</h3><p>Sessions give you context within a conversation. Memory Bank gives you context across every conversation a user has ever had. When a customer comes back after two weeks and says “any update on my issue?” — sessions don’t help. Memory Bank is what makes the relationship continuous.</p><p><strong>How it works</strong></p><p>The write side is a single call in an after_agent_callback that fires when the session ends:</p><pre>async def after_agent_callback(callback_context: CallbackContext) -&gt; None:<br>    await runner.memory_service.add_session_to_memory(<br>        callback_context.session<br>    )</pre><p>Memory Bank sends the conversation to a Generational Model — Gemini Flash by default — which extracts atomic facts about the user and stores them. The read side is automatic: a PreloadMemoryTool fires at the start of every future session, injecting relevant facts into the agent&#39;s context before the first message is processed.</p><p><strong>The demo — Marcus’s two sessions</strong></p><p>We ran two sessions with Marcus (C002) to put this to the test. Here’s what happened.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YXYIfKJScUjW9Iv4q_qwKw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fRVbG5o4Dr7xCrNq3OrVug.png" /></figure><p><strong>What Memory Bank captured</strong></p><p>Memory instances ending with 9024 after the first session is completed. It shows that the agent memory has a reference to the ticket and that its blocking a go-live.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D5_dWaUDRjeG2DWaQH0e4Q.png" /></figure><p>The same memory instances ending with 9024 after the second session is completed. The memory went from “blocking go-live tomorrow” to “workaround implemented, go-live done, issue still needs fixing.” Same fact, evolved state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XxgKGrtZJQkMXDg63vOsYw.png" /></figure><p>Each fact is scoped to app_name and user_id, so memories stay isolated per application and per customer. Facts are also editable directly from the console and via API — useful for correcting extraction errors or removing personal information without touching the underlying sessions.</p><h3>4. Agent Sandbox</h3><p><a href="https://cloud.google.com/gemini-enterprise-agent-platform/scale/sandbox/code-execution-overview"><strong>Agent Sandbox</strong></a> provides a hardened environment to safely execute model-generated code and perform computer use tasks like browser-based automation without risk to your host systems.</p><p>Many agent tasks — financial calculations, data science workflows, SLA compliance checks — require the agent to generate and execute code. Code Execution lets your agent run that code in a secure, isolated, managed sandbox environment. Key capabilities include:</p><ul><li>Sandboxes can be created and execute code in under a second.</li><li>Sandboxes support file input and output up to 100MB per request or response.</li><li>Sandboxes maintain their execution state for up to 14 days — configurable via TTL.</li></ul><p>Our SLA agent needs to calculate how long tickets have been open. Date arithmetic looks simple — it isn’t. LLMs get it wrong: off-by-one days, timezone confusion, inconsistent duration formatting. The only way to get a verifiable, exact answer is to run actual Python code. Agent Sandbox is what makes that safe in production. Instead of Gemini’s shared BuiltInCodeExecutor, AgentEngineSandboxCodeExecutor runs your code in a dedicated, isolated GCP process scoped to your Agent Runtime. Your code doesn&#39;t share infrastructure with anyone else&#39;s agent.</p><pre>sla_agent = adk.Agent(<br>    model=&quot;gemini-2.5-flash&quot;,<br>    name=&quot;sla_agent&quot;,<br>    instruction=&quot;&quot;&quot;<br>    Calculate SLA compliance. Write Python code using the datetime module.<br>    Calculate hours_open for each ticket from its created date to now.<br>    Execute the code — never guess the result from text alone.<br>    &quot;&quot;&quot;,<br>    code_executor=AgentEngineSandboxCodeExecutor(<br>        sandbox_resource_name=_SANDBOX_NAME or None,<br>    ),<br>)</pre><p>The or None matters — if SANDBOX_NAME is an empty string, the class validator rejects it. Passing None tells the executor to fall back gracefully during local development when no sandbox exists yet.</p><p>The sandbox is a separate cloud resource, created under the engine and referenced by its full resource name. We covered the creation code in the deployment section above — SANDBOX_NAME is what comes back from that operation and gets patched into the engine&#39;s environment in Step 4.</p><h3>Conclusion</h3><p>Of all the Gemini Enterprise Agent Platform pillars, <strong>Scale</strong> is the one that actually decides whether an agent ships. Build gets you a working prototype in an afternoon. Govern and Optimize operate on agents that are already running in production. But Scale is the bridge — the layer that determines whether the thing you built in Build can survive contact with real users, real sessions, and real data.</p><p>The four Scale blocks each solve a different version of the same problem: how do you make an AI agent behave like infrastructure rather than like a script? Agent Runtime removes you from the business of servers and containers. Sessions give the agent a real-time memory of the current conversation. Memory Bank gives it a longer memory — one that persists across every session, distilling what it has learned into durable facts rather than raw transcripts. Agent Sandbox draws a hard boundary around any code the agent executes, so an agent that writes and runs Python doesn’t become a security incident.</p><p><strong>What’s next.</strong> We’ve only covered Scale. The Govern layer — Agent Registry, Policies, Gateways — is where you add access controls, rate limits, and audit trails to the agents you’ve already deployed. The Optimize layer is where evaluation pipelines and cost controls come in: running structured tests against your agent to measure whether it’s actually solving the problems you designed it for. Both of those are worth a post of their own.</p><p>On the Build side, there’s also a natural upgrade path. The support_agent here has two sub-agents; a real enterprise system might have a dozen, organised into a deeper hierarchy. ADK&#39;s AgentTool pattern scales there without major refactoring. And as GEAP&#39;s Traces and Topology features roll out to earlier runtime versions, the observability story gets considerably richer — full call trees, tool latencies, sub-agent boundaries, all visible without adding a single line of instrumentation code.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18e28318ac94" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taming the Agent: A Practical Guide to Callbacks in Google ADK]]></title>
            <link>https://ayushm4489.medium.com/taming-the-agent-a-practical-guide-to-callbacks-in-google-adk-831b2f017c6f?source=rss-2839f9309087------2</link>
            <guid isPermaLink="false">https://medium.com/p/831b2f017c6f</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[google-adk]]></category>
            <dc:creator><![CDATA[ayush mittal]]></dc:creator>
            <pubDate>Fri, 27 Mar 2026 16:45:53 GMT</pubDate>
            <atom:updated>2026-03-27T18:58:23.276Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How to intercept, observe, and control your AI agent’s behaviour — including a new pattern for protecting your context window</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mPEQImZTLPkrPV-C" /><figcaption>Photo by <a href="https://unsplash.com/@davidtyemnyak?utm_source=medium&amp;utm_medium=referral">David Tyemnyák</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When you build a conversational agent with Google’s Agent Development Kit (ADK), you’re handing a lot of control to a framework and a model. The framework decides when to call the LLM. The LLM decides when to call a tool. The tool returns whatever the API gives it. In a simple demo, this is fine. In production — where you need guardrails, auditability, and careful token management — you need hooks.</p><p>That’s exactly what <strong>callbacks</strong> are.</p><h3>Why Callbacks Exist</h3><p>At its core, an ADK agent runs a loop:</p><blockquote><em>Receive input → Call LLM → LLM requests tool → Execute tool → Feed result back → LLM generates response</em></blockquote><p>Every step in this loop is a black box unless you instrument it. Callbacks are the instrumentation layer the framework provides — named intercept points where you can <strong>observe</strong>, <strong>modify</strong>, or <strong>completely replace</strong> what’s happening, without touching the core agent logic.</p><p>Think of them like middleware in a web framework, or event listeners in a UI. The framework calls your function at a specific moment and lets your return value control what happens next:</p><ul><li>Return None → let the default behaviour proceed unchanged</li><li>Return a value → <strong>replace</strong> the default with yours</li></ul><p>This single mechanism unlocks a surprising range of patterns: guardrails, caching, logging, context window management, state enrichment, and more.</p><h3>The Callback Lifecycle</h3><p>Before going into specific use cases, it helps to see where callbacks sit in the agent’s execution flow. There are six intercept points:</p><pre>┌─────────────────────────────────────────────────────────────┐<br>│                      USER MESSAGE                           │<br>└────────────────────────────┬────────────────────────────────┘<br>                             │<br>                    ┌────────▼────────┐<br>                    │ before_agent_   │  ← intercept before agent starts<br>                    │   callback      │<br>                    └────────┬────────┘<br>                             │<br>                    ┌────────▼────────┐<br>                    │ before_model_   │  ← intercept before LLM call<br>                    │   callback      │<br>                    └────────┬────────┘<br>                             │<br>                    ┌────────▼────────┐<br>                    │   LLM CALL      │  (Gemini)<br>                    └────────┬────────┘<br>                             │<br>                    ┌────────▼────────┐<br>                    │ after_model_    │  ← intercept after LLM responds<br>                    │   callback      │<br>                    └────────┬────────┘<br>                             │<br>                    ┌────────▼────────┐<br>                    │ before_tool_    │  ← intercept before tool runs<br>                    │   callback      │<br>                    └────────┬────────┘<br>                             │<br>                    ┌────────▼────────┐<br>                    │  TOOL EXECUTION │<br>                    └────────┬────────┘<br>                             │<br>                    ┌────────▼────────┐<br>                    │ after_tool_     │  ← intercept after tool returns<br>                    │   callback      │<br>                    └────────┬────────┘<br>                             │<br>                    ┌────────▼────────┐<br>                    │ after_agent_    │  ← intercept after agent finishes<br>                    │   callback      │<br>                    └────────┬────────┘<br>                             │<br>┌────────────────────────────▼────────────────────────────────┐<br>│                      AGENT RESPONSE                         │<br>└─────────────────────────────────────────────────────────────┘</pre><blockquote><strong><em>Note</em></strong><em>: In a ReAct loop, </em><em>before_model_callback and </em><em>after_model_callback fire on </em><strong><em>every</em></strong><em> LLM call, and </em><em>before_tool_callback / </em><em>after_tool_callback fire on </em><strong><em>every</em></strong><em> tool execution. A single user turn can trigger these multiple times.</em></blockquote><h3>Two Worlds: Context Window vs. Session State</h3><p>Before diving into specific callback patterns, there’s a foundational concept worth making explicit — because once you see it clearly, every callback decision becomes easier.</p><p>When an ADK agent runs, two completely separate memory systems exist side by side:</p><pre>┌─────────────────────────────────┐     ┌──────────────────────────────────┐<br>│        LLM CONTEXT WINDOW       │     │         SESSION STATE             │<br>│        (The LLM&#39;s world)        │     │        (The framework&#39;s world)    │<br>│                                 │     │                                   │<br>│  • System prompt / instructions │     │  • Key-value dictionary           │<br>│  • Conversation history         │     │  • Lives in ADK SessionService    │<br>│  • Tool call records            │     │  • Persists across turns          │<br>│  • Tool results                 │     │  • Never seen by LLM directly     │<br>│                                 │     │                                   │<br>│  Measured in tokens             │     │  Measured in memory               │<br>│  Finite and expensive           │     │  Flexible and cheap               │<br>│  The LLM reasons over this      │     │  The framework manages this       │<br>└─────────────────────────────────┘     └──────────────────────────────────┘<br>           ▲                                           │<br>           │                                           │<br>           └───────── injection bridge ────────────────┘<br>                  (template substitution at call time)</pre><p><strong>The LLM cannot see session state directly.</strong> It is completely invisible to the model. The LLM only ever sees text — whatever has been assembled into its context window before each call.</p><p><strong>The framework cannot reason.</strong> Session state is just a dictionary. It has no intelligence. Only the LLM can interpret and reason over data.</p><p>These two systems need each other, and the <strong>injection bridge</strong> is how they connect.</p><h3>Bridging the Gap: Referring to Session State in Instructions</h3><p>ADK provides a simple templating mechanism: wrap any session state key in curly braces inside your agent’s instructions, and the framework will substitute the live value before every LLM call.</p><pre># Agent instruction:<br>You are a banking assistant helping customer {customer_name}.<br>Their current loyalty tier is {loyalty_tier}.<br>They have {account_count} linked accounts.<br>Always greet them by name and tailor responses to their tier.</pre><pre># What happens before each LLM call — automatically, by the framework:<br>session.state = {<br>    &quot;customer_name&quot;: &quot;Jan de Vries&quot;,<br>    &quot;loyalty_tier&quot;: &quot;Gold&quot;,<br>    &quot;account_count&quot;: 3<br>}<br><br># Rendered instruction sent to the LLM:<br># &quot;You are a banking assistant helping customer Jan de Vries.<br>#  Their current loyalty tier is Gold.<br>#  They have 3 linked accounts.<br>#  Always greet them by name and tailor responses to their tier.&quot;</pre><p>The LLM sees the rendered values as plain text inside its system prompt. It has no idea they came from a dictionary. From its perspective, this is just part of its instructions.</p><h3>Why This Architecture Matters</h3><p>This separation is not a limitation — it’s a deliberate design that gives you precise control over what the LLM reasons about:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5U9ZGCQOLOOIqkEUzUJ-JQ.png" /></figure><p>The callbacks you write are the mechanism that moves data between these two worlds deliberately — deciding what crosses the injection bridge and in what form.</p><p>This is what makes after_tool_callback so powerful: it sits at the exact moment when raw API data arrives and lets you decide how much of it enters the LLM&#39;s world, and how much stays in the framework&#39;s world.</p><h3>Focusing on Tool Callbacks</h3><p>Tool callbacks — before_tool_callback and after_tool_callback — are where some of the most impactful patterns live, because tools are where your agent touches the real world: calling APIs, querying databases, retrieving documents.</p><pre>from google.adk.tools import ToolContext<br>from typing import Any, Dict, Optional<br><br>def before_tool_callback(<br>    tool: Tool,<br>    input: Dict[str, Any],<br>    callback_context: CallbackContext<br>) -&gt; Optional[Dict]:<br>    # Return None → tool executes normally<br>    # Return a dict → tool is SKIPPED, dict used as result<br>    ...<br>def after_tool_callback(<br>    tool: Tool,<br>    input: Dict[str, Any],<br>    callback_context: CallbackContext,<br>    tool_response: Dict<br>) -&gt; Optional[Dict]:<br>    # Return None → tool_response passed to LLM unchanged<br>    # Return a dict → your dict passed to LLM instead<br>    ...</pre><p>Let’s walk through the real-world use cases.</p><h3>Use Case 1: Input Validation and Guardrails</h3><p>Use before_tool_callback to inspect what the LLM is about to ask a tool to do — and block it if something is wrong.</p><p><strong>Example: Prevent the LLM from querying accounts it shouldn’t access</strong></p><pre>def before_tool_callback(<br>    tool: Tool,<br>    input: Dict[str, Any],<br>    callback_context: CallbackContext<br>) -&gt; Optional[Dict]:<br>    if tool.name == &quot;get_account_details&quot;:<br>        requested_id = input.get(&quot;account_id&quot;)<br>        session_customer_id = callback_context.state.get(&quot;customer_id&quot;)<br><br>        if requested_id != session_customer_id:<br>            # Block the tool call entirely — return a safe error<br>            return {<br>                &quot;error&quot;: &quot;Access denied&quot;,<br>                &quot;reason&quot;: &quot;Requested account does not match authenticated customer&quot;<br>            }<br><br>    return None  # Allow the tool to proceed<br></pre><p>The LLM never reaches the API. The tool callback short-circuits the call and hands back a controlled error response.</p><h3>Use Case 2: Caching Expensive Tool Results</h3><p>Use before_tool_callback to check session state before making a redundant API call. If you already have the result, return it immediately.</p><p><strong>Example: Cache a product catalogue lookup for the session</strong></p><pre>def before_tool_callback(<br>    tool: Tool,<br>    input: Dict[str, Any],<br>    callback_context: CallbackContext<br>) -&gt; Optional[Dict]:<br>    if tool.name == &quot;get_product_catalogue&quot;:<br>        cached = callback_context.state.get(&quot;product_catalogue_cache&quot;)<br>        if cached:<br>            print(&quot;Cache hit — skipping API call&quot;)<br>            return cached  # Tool skipped, cached result used<br><br>    return None  # Cache miss — let the tool run</pre><p>Pair this with an after_tool_callback that populates the cache on first run:</p><pre>def after_tool_callback(<br>    tool: Tool,<br>    input: Dict[str, Any],<br>    callback_context: CallbackContext,<br>    tool_response: Dict<br>) -&gt; Optional[Dict]:<br>    if tool.name == &quot;get_product_catalogue&quot;:<br>        callback_context.state[&quot;product_catalogue_cache&quot;] = tool_response<br><br>    return None  # Pass the response to the LLM unchanged</pre><h3>Use Case 3: Modifying Tool Responses</h3><p>Use after_tool_callback to clean, enrich, or fix tool responses before the LLM sees them.</p><p><strong>Example: Normalise an inconsistent API response</strong></p><pre>def after_tool_callback(<br>    tool: Tool,<br>    input: Dict[str, Any],<br>    callback_context: CallbackContext,<br>    tool_response: Dict<br>) -&gt; Optional[Dict]:<br>    if tool.name == &quot;get_user_info&quot;:<br>        # Rename inconsistent field from legacy API<br>        if &quot;usr_name&quot; in tool_response:<br>            tool_response[&quot;username&quot;] = tool_response.pop(&quot;usr_name&quot;)<br><br>        # Ensure expected fields always exist<br>        tool_response.setdefault(&quot;status&quot;, &quot;active&quot;)<br><br>    return tool_response</pre><h3>Use Case 4: Enriching Session State</h3><p>Use after_tool_callback to extract key fields from a tool result and store them in session state — making them available for the rest of the session without repeating the API call.</p><p><strong>Example: Save a transaction ID for use in later turns</strong></p><pre>def after_tool_callback(<br>    tool: Tool,<br>    input: Dict[str, Any],<br>    callback_context: CallbackContext,<br>    tool_response: Dict<br>) -&gt; Optional[Dict]:<br>    if tool.name == &quot;initiate_payment&quot;:<br>        # Save critical identifiers to session state<br>        callback_context.state[&quot;last_transaction_id&quot;] = tool_response.get(&quot;transaction_id&quot;)<br>        callback_context.state[&quot;payment_status&quot;] = tool_response.get(&quot;status&quot;)<br><br>    return None  # Pass full response to LLM</pre><p>Later, other tools or instructions can reference {last_transaction_id} without hitting the API again.</p><h3>The Store-and-Slim Pattern: Protecting Your Context Window</h3><p>Now let’s talk about a pattern that isn’t explicitly named in the ADK documentation, but emerges naturally from combining the above building blocks — and matters a great deal in production.</p><h3>The Problem</h3><p>Consider this common scenario:</p><ol><li>The LLM calls a tool: get_customer_profile()</li><li>Your API returns a rich JSON response: account details, transaction history, preferences, address, linked products — 3/4KB of data</li><li>That entire blob is injected into the <strong>context window as a tool result</strong></li><li>The conversation continues for several more turns</li><li>The same data sits in the context window for the rest of the session, consuming tokens on every LLM call</li></ol><p>Worse: if you also store that data in session state (as you should, for reliability), and then inject it into instructions via {customer_profile}, the LLM now sees <strong>two copies</strong> of the same data — one in the conversation history, one in the system prompt. If they ever drift out of sync (because the session state was updated), the LLM is reasoning over a contradiction.</p><h3>The Pattern</h3><p>The fix is elegant: use after_tool_callback to <strong>separate what gets stored from what gets shown to the LLM</strong>.</p><pre>API Response (large)<br>        │<br>        ▼<br>after_tool_callback fires<br>   ├── Store full payload ──→ session state (outside context window)<br>   └── Return slim summary ──→ context window (tiny footprint)</pre><p><strong>Example: Customer profile lookup</strong></p><pre>def after_tool_callback(<br>    tool: Tool,<br>    input: Dict[str, Any],<br>    callback_context: CallbackContext,<br>    tool_response: Dict<br>) -&gt; Optional[Dict]:<br>    if tool.name == &quot;get_customer_profile&quot;:<br><br>        # 1. Store the full payload in session state (outside the LLM)<br>        callback_context.state[&quot;customer_profile&quot;] = tool_response<br><br>        # 2. Return only a slim acknowledgement to the LLM<br>        return {<br>            &quot;status&quot;: &quot;profile_loaded&quot;,<br>            &quot;customer_name&quot;: tool_response.get(&quot;name&quot;),<br>            &quot;account_count&quot;: len(tool_response.get(&quot;accounts&quot;, []))<br>        }<br><br>    return None</pre><p><strong>What the LLM sees in its context window:</strong></p><pre>[Tool result]: {&quot;status&quot;: &quot;profile_loaded&quot;, &quot;customer_name&quot;: &quot;Jan&quot;, &quot;account_count&quot;: 3}</pre><p><strong>What lives in session state (available via instruction injection):</strong></p><pre>{<br>  &quot;name&quot;: &quot;Jan de Vries&quot;,<br>  &quot;accounts&quot;: [...],<br>  &quot;address&quot;: &quot;...&quot;,<br>  &quot;loyalty_tier&quot;: &quot;Gold&quot;,<br>  &quot;preferences&quot;: {...}<br>}</pre><p>The LLM knows the profile was loaded. It gets the two facts it needs to reason about. The full data is safely in state, ready to be injected precisely into whichever agent instruction actually needs it — and only that instruction.</p><h3>Why This Matters</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z7W35Y4YEE6y4fiUXOEAsw.png" /></figure><h3>The Consistency Rule</h3><p>When using this pattern, pair it with an explicit instruction telling the LLM where to find the authoritative value:</p><pre>When referring to customer account details, always use the values<br>injected from session state: {customer_name}, {account_count}.<br>Do not rely on values mentioned earlier in the conversation history.</pre><p>This eliminates the ambiguity window entirely.</p><h3>A Note on the temp: Prefix</h3><p>ADK session state supports a temp: key prefix for data that should only live for the current invocation (one LLM turn), not the full session:</p><pre># Only available during this tool call&#39;s invocation<br>callback_context.state[&quot;temp:raw_api_response&quot;] = tool_response</pre><pre># Available for the entire session<br>callback_context.state[&quot;customer_name&quot;] = tool_response.get(&quot;name&quot;)</pre><p>Use temp: for intermediate data you need within a single multi-step ReAct loop but don&#39;t want persisting across turns. Use plain keys for data that should be available for the rest of the conversation.</p><h3>Putting It All Together</h3><p>Here’s a composite callback that combines multiple patterns — guardrail, caching, and store-and-slim — for a single tool:</p><pre>def after_tool_callback(<br>    tool: Tool,<br>    input: Dict[str, Any],<br>    callback_context: CallbackContext,<br>    tool_response: Dict<br>) -&gt; Optional[Dict]:<br><br>    if tool.name == &quot;get_customer_profile&quot;:<br><br>        # Guard: did the API return an error?<br>        if tool_response.get(&quot;error&quot;):<br>            callback_context.state[&quot;profile_load_failed&quot;] = True<br>            return {&quot;status&quot;: &quot;error&quot;, &quot;message&quot;: &quot;Could not load customer profile&quot;}<br><br>        # Store: full payload in session state<br>        callback_context.state[&quot;customer_profile&quot;] = tool_response<br>        callback_context.state[&quot;customer_name&quot;] = tool_response.get(&quot;name&quot;)<br>        callback_context.state[&quot;loyalty_tier&quot;] = tool_response.get(&quot;loyalty_tier&quot;)<br><br>        # Cache: mark as loaded so we don&#39;t call again<br>        callback_context.state[&quot;profile_loaded&quot;] = True<br><br>        # Slim: return only what the LLM needs to reason about<br>        return {<br>            &quot;status&quot;: &quot;profile_loaded&quot;,<br>            &quot;customer_name&quot;: tool_response.get(&quot;name&quot;),<br>            &quot;loyalty_tier&quot;: tool_response.get(&quot;loyalty_tier&quot;),<br>            &quot;account_count&quot;: len(tool_response.get(&quot;accounts&quot;, []))<br>        }<br><br>    return None</pre><p>One callback. Three jobs done. The LLM gets exactly what it needs, nothing more.</p><h3>Summary</h3><p>Callbacks are the production layer of ADK agent development. They transform a working demo into a resilient, observable, token-efficient system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xys9MQpSCvublakPtzsL_Q.png" /></figure><p>The <strong>store-and-slim pattern</strong> is the one pattern missing from most ADK tutorials. It treats the context window as the scarce resource it is — and uses session state as the purpose-built alternative for data that needs to persist without consuming LLM attention. In production agents handling real customer data, getting this right is the difference between a session that gracefully serves 30 turns and one that runs out of context halfway through.</p><p><em>This pattern emerged from building production grade agents on </em><a href="https://docs.cloud.google.com/customer-engagement-ai/conversational-agents/ps"><em>Google’s CX Agent Studio (GECX)</em></a><em>. The store-and-slim approach is an application of ADK’s documented callback and state primitives, combined to solve a problem that isn’t explicitly addressed in the official documentation yet.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=831b2f017c6f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Built a Voice-First AI Banking Agent with Gemini Live and Google ADK]]></title>
            <link>https://ayushm4489.medium.com/how-i-built-a-voice-first-ai-banking-agent-with-gemini-live-and-google-adk-0ecb4cdf8002?source=rss-2839f9309087------2</link>
            <guid isPermaLink="false">https://medium.com/p/0ecb4cdf8002</guid>
            <category><![CDATA[banking]]></category>
            <category><![CDATA[gemini-live]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[google-adk]]></category>
            <dc:creator><![CDATA[ayush mittal]]></dc:creator>
            <pubDate>Sun, 15 Mar 2026 18:58:56 GMT</pubDate>
            <atom:updated>2026-03-16T11:27:45.536Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>I created this article for the purposes of entering the <strong>Gemini Live Agent Challenge</strong> hackathon. #GeminiLiveAgentChallenge</blockquote><p>What if you could manage your bank account the same way you talk to a friend? No taps, no forms, no branch visits — just speak.</p><p>That was the driving question behind <strong>NorthRiver Banking Agent</strong>, a voice-first AI assistant I built for the <a href="https://googlegeminiagents.devpost.com/">Gemini Live Agent Challenge</a>. The result is <strong>River</strong> — an AI banker that lets customers check balances, transfer money to contacts by name, and even pay utility bills by photographing a QR code, all through natural conversation.</p><p>In this post I’ll walk through the key engineering decisions, the Google Cloud architecture that powers it, and the specific challenges of building real-time voice AI that feels genuinely intelligent rather than scripted.</p><h3>The Problem With Banking Apps</h3><p>Modern banking apps are impressive feats of engineering. They’re also deeply frustrating to use. Want to pay a bill? Navigate to Payments → Add Payee → Enter IBAN → Confirm Details → Enter OTP → Submit. Six steps and a context switch for something that should take ten seconds.</p><p>Voice is the natural interface for transactional tasks. We already tell friends “I’ll pay you back twenty euros” — why can’t we tell our bank the same thing?</p><p>The challenge is that voice banking has a terrible track record. Legacy IVR systems (“Press 1 for balances…”) are universally despised. The new generation of voice assistants does better, but typically falls apart the moment a task requires memory of earlier context, access to personal financial data, or multi-step confirmation flows.</p><p>This is exactly where large multimodal models change the equation.</p><h3>The Three Capabilities That Drive Everything</h3><p>I designed River around three concrete demo flows that showcase genuinely new capabilities:</p><p><strong>1. Personalised Spending Queries</strong></p><blockquote>“How much did I spend on coffee last year?”</blockquote><p>River queries the customer’s actual Firestore transaction history, aggregates spend by category and year, and answers conversationally — while simultaneously rendering a visual transaction table in the web UI.</p><p><strong>2. Contact-Based Transfers</strong></p><blockquote>“Send fifty euros to David”</blockquote><p>River looks up David’s IBAN from saved contacts, reads back the last four digits for confirmation, and executes the transfer only after an explicit “yes.” No IBAN required from the customer.</p><p><strong>3. QR Bill Payment by Photo</strong></p><blockquote>[customer uploads a photo of a Vattenfall energy bill]</blockquote><p>River reads the SEPA EPC QR code embedded in the bill image using Gemini’s multimodal capabilities, announces the merchant, amount and reference number, and waits for confirmation before paying.</p><p>The third capability is the showstopper. Taking a photo of a paper bill and having the bank pay it in under thirty seconds — with voice confirmation — isn’t something any consumer banking app does today.</p><h3>Architecture Overview</h3><p>The system has three layers:</p><ul><li><strong>Frontend</strong>: React + Vite SPA served as static files, streaming audio via Web Audio API and AudioWorklet</li><li><strong>Backend</strong>: Python FastAPI server managing WebSocket connections and orchestrating the ADK agent</li><li><strong>AI Layer</strong>: Google ADK multi-agent hierarchy running on Vertex AI with Gemini Live</li></ul><p>Everything is containerised in Docker and deployed to <strong>Google Cloud Run</strong>, with <strong>Firestore</strong> as the banking database.</p><h3>The Google ADK Multi-Agent Design</h3><p>The most important architectural decision was using <strong>Google’s Agent Development Kit (ADK)</strong> with a multi-agent hierarchy rather than a single monolithic agent.</p><pre>northriver_orchestrator  (River — greets &amp; routes)<br>├── account_info_agent   (balances, transactions, spending)<br>└── payments_agent       (transfers, QR bill payments)</pre><p>The orchestrator is responsible for greeting the customer and routing their request to the right specialist. Crucially, this routing is completely invisible to the customer — River feels like a single coherent assistant, not a handoff between systems.</p><p>Each sub-agent has:</p><ul><li>A tightly scoped system prompt focused on its domain</li><li>Access only to the Firestore tools it needs</li><li>Its own safety rules (payments agent has explicit confirmation requirements)</li></ul><p>This separation isn’t just organisational tidiness. It means the payments agent can have very strict rules about confirmation without those rules leaking into the account information agent, which doesn’t need them. Domain isolation produces better, safer behaviour.</p><h3>The Tool Set</h3><p>Each sub-agent is equipped with purpose-built tools:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VApBnmo39qQkGv6a5MZN4Q.png" /><figcaption>tools</figcaption></figure><p>Every tool reads the account_id from tool_context.state — the customer never has to identify themselves mid-conversation.</p><h3>Real-Time Voice: The WebSocket Pipeline</h3><p>The most technically complex piece is the real-time audio pipeline. ADK’s runner.run_live() enables bidirectional audio streaming with Gemini Live, but wiring that to a browser required careful protocol design.</p><h3>Why a Custom FastAPI Server</h3><p>ADK provides a convenience function get_fast_api_app() for quick prototyping. I chose not to use it because it doesn&#39;t allow injection of arbitrary content (like image frames) into the live stream mid-session. Instead, I built a custom FastAPI server that calls runner.run_live() directly and owns the full WebSocket lifecycle.</p><h3>The WebSocket Protocol</h3><p>A single WebSocket connection at /ws/{user_id}/{session_id} handles everything:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ATFivKUINHNkIwa8LcAtw.png" /><figcaption>websocket</figcaption></figure><p>The binary/JSON multiplexing means one connection carries audio, images, transcripts, and UI state updates simultaneously.</p><h3>Audio Capture in the Browser</h3><p>On the frontend, AudioCapture.js uses getUserMedia to access the microphone, then routes the stream through an <strong>AudioWorklet</strong> processor:</p><pre>// AudioWorklet runs in a dedicated audio rendering thread<br>// Float32 PCM → Int16 PCM conversion happens here<br>// Avoids blocking the main thread during audio processing<br>class PCMProcessor extends AudioWorkletProcessor {<br>  process(inputs) {<br>    const float32 = inputs[0][0];<br>    const int16 = new Int16Array(float32.length);<br>    for (let i = 0; i &lt; float32.length; i++) {<br>      int16[i] = Math.max(-32768, Math.min(32767, float32[i] * 32768));<br>    }<br>    this.port.postMessage(int16.buffer, [int16.buffer]);<br>    return true;<br>  }<br>}</pre><p>The resulting PCM16 buffers are sent as binary WebSocket frames at 16 kHz — exactly what the Gemini Live API expects.</p><h3>Audio Playback</h3><p>The server sends back Gemini’s voice responses as base64-encoded PCM16 at 24 kHz. AudioPlayer.js decodes these and schedules them on the Web Audio API timeline, maintaining a playback cursor to queue successive chunks without gaps or clicks.</p><h3>The Multimodal Trick: QR Bill Payment</h3><p>The QR bill payment flow is where Gemini’s multimodal capabilities earn their keep.</p><p>When the customer taps “Pay a Bill” and uploads a photo, the frontend resizes it to a maximum of 768×768 pixels, converts it to JPEG, and sends it as a base64 JSON frame over the WebSocket. The backend injects this into the live session as a Blob(mime_type=&quot;image/jpeg&quot;).</p><p>Gemini Live receives the image mid-conversation and can read it just as a human would — understanding the SEPA EPC QR code printed on the bill, extracting the merchant name, IBAN, amount, and payment reference, and reporting all of this verbally to the customer.</p><p>The payments agent’s system prompt includes explicit rules to prevent premature payment:</p><pre>When a customer sends a bill image, you MUST:<br>1. Decode and announce the QR code contents (merchant, amount, reference)<br>2. Wait for an explicit standalone &quot;yes&quot; confirmation<br>3. Only THEN call process_qr_payment()<br>A request to scan a bill (&quot;tell me what this says&quot;) is NOT confirmation.<br>&quot;I want to pay this bill&quot; is intent, NOT confirmation.</pre><p>This prompt hardening was essential. In early testing, the model was interpreting the image upload itself as implicit consent to pay, skipping the confirmation step entirely.</p><h3>Pushing Live Data to the Browser: The UI Events Side-Channel</h3><p>The transactions table feature required solving an architectural puzzle.</p><p>When agent answers “You spent €287 on coffee last year,” I also wanted the browser to simultaneously render a visual table of every individual coffee transaction. But the ADK event stream from runner.run_live() doesn&#39;t expose tool results — only audio and transcript events.</p><p>The solution: a module-level asyncio.Queue registry that tools can write to, and the WebSocket handler drains on every event loop iteration.</p><pre># banking_agent/ui_events.py<br>_queues: dict[str, asyncio.Queue] = {}</pre><pre>def register(session_id: str) -&gt; asyncio.Queue:<br>    q = asyncio.Queue()<br>    _queues[session_id] = q<br>    return q</pre><pre>def emit(session_id: str, event: dict) -&gt; None:<br>    q = _queues.get(session_id)<br>    if q is not None:<br>        q.put_nowait(event)</pre><p>When get_transactions completes a Firestore query, it calls emit_ui() with the transaction list. The WebSocket&#39;s downstream_task drains this queue on every ADK event and forwards UI events directly to the browser — completely independent of what the agent says aloud.</p><h3>Keeping Tools Pure with after_tool_callback</h3><p>My first implementation had emit_ui() calls scattered directly inside the tool functions. That worked, but it meant every tool carried UI concerns alongside its business logic — harder to test in isolation and harder to reason about.</p><p>ADK’s after_tool_callback is a better home for this. The callback fires after every tool call on a given agent, receiving the tool name and its result dict. Tools stay pure (data in, data out); all UI side-effects are centralised in one place:</p><pre>async def _after_tool_callback(<br>    ctx: CallbackContext, tool_name: str, result: dict<br>) -&gt; dict | None:<br>    session_id = ctx.state.get(&quot;session_id&quot;)<br>    if not session_id or result.get(&quot;status&quot;) != &quot;success&quot;:<br>        return None  # errors never emit UI events<br><br>    if tool_name == &quot;get_transactions&quot;:<br>        emit_ui(session_id, {<br>            &quot;type&quot;: &quot;transactions_table&quot;,<br>            &quot;transactions&quot;: result[&quot;transactions&quot;],<br>            &quot;total_spend&quot;: result[&quot;total_spend&quot;],<br>            ...<br>        })<br><br>    if tool_name in (&quot;make_transfer&quot;, &quot;process_qr_payment&quot;):<br>        emit_ui(session_id, {&quot;type&quot;: &quot;balance_update&quot;, &quot;balance&quot;: result[&quot;new_balance&quot;]})<br><br>    return None  # pass the result through unchanged to the modelp</pre><p>The callback is wired into each sub-agent at construction time:</p><pre>Agent(<br>    name=&quot;account_info_agent&quot;,<br>    tools=[get_account_balance, get_transactions],<br>    after_tool_callback=_after_tool_callback,<br>    ...<br>)</pre><p>Returning None from the callback is the contract for &quot;pass through unchanged.&quot; Returning a dict replaces the tool result — a useful hook for result transformation or caching, but not needed here.</p><p>This pattern also powers the real-time balance updates after transfers. The browser’s balance pill updates the moment a transfer completes, without waiting for the next login.</p><h3>Google Cloud Infrastructure</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LPGABaMIhLqqxoHJQVBQEQ.png" /></figure><h3>Cloud Run</h3><p>The entire application — Python backend and React static files — runs in a single Docker container on Cloud Run. Key settings:</p><ul><li><strong>Min instances: 1</strong> — eliminates cold starts for demo reliability</li><li><strong>Timeout: 3600s</strong> — long WebSocket sessions require extended timeouts</li><li><strong>Concurrency: 10</strong> — each instance handles 10 simultaneous banking sessions</li></ul><p>The Dockerfile uses a multi-stage build: Stage 1 runs npm ci &amp;&amp; npm run build in Node 20, producing the React app&#39;s static files. Stage 2 is a Python 3.12 slim image that installs the Python dependencies and copies both the backend code and the built frontend.</p><h3>Firestore</h3><p>Firestore was the right choice for this project for several reasons:</p><ul><li><strong>Serverless</strong> — no connection pooling headaches in a concurrent WebSocket environment</li><li><strong>Real-time capable</strong> — though we use one-shot reads/writes, the option exists</li><li><strong>Native async</strong> — the Python client supports asyncio natively</li></ul><p>The schema is deliberately simple:</p><pre>accounts/{account_id}       → balance, IBAN, customer name<br>transactions/{txn_id}       → account_id, category, merchant, amount, date<br>contacts/{contact_id}       → account_id, name, IBAN</pre><h3>Vertex AI</h3><p>Gemini Live runs on Vertex AI in the europe-west4 region. The AGENT_MODEL environment variable makes it easy to switch between model versions without code changes — useful as the Gemini Live model aliases evolve.</p><h3>CI/CD with Cloud Build</h3><p>A cloudbuild.yaml pipeline handles automated deployments on every push to main:</p><ol><li><strong>Build</strong> — Docker multi-stage build with layer caching</li><li><strong>Push</strong> — both :latest and :$SHORT_SHA tags to Artifact Registry</li><li><strong>Deploy</strong> — rolling update to Cloud Run (zero downtime)</li><li><strong>Verify</strong> — smoke test that the service URL returns HTTP 200</li></ol><p>The commit SHA tag means every deployed version is immutable and reproducible.</p><h3>What I Learned</h3><p><strong>Prompt engineering is load-bearing code.</strong> The difference between an agent that asks for confirmation and one that pays your bills without asking is entirely in the system prompt. Treat it with the same rigour as production code — version it, test it, review changes carefully.</p><p><strong>ADK’s multi-agent routing is genuinely powerful.</strong> The ability to route to specialist sub-agents silently — without the customer ever being aware of the handoff — produces a much more coherent conversation than a single overloaded agent. Each sub-agent can be tuned independently without side effects.</p><p><strong>Side-channel patterns unlock richer UX.</strong> The ui_events queue registry was a lightweight solution to a hard problem: how do you push structured data from deep inside tool code to a WebSocket client without breaking the agent&#39;s conversational flow? The answer — a session-keyed async queue drained by the WebSocket handler — is simple, testable, and composable.</p><p><strong>Use </strong>after_tool_callback<strong> to separate concerns.</strong> My first instinct was to call emit_ui() directly inside each tool function. It worked, but it tangled UI logic into business logic. ADK&#39;s after_tool_callback is the right boundary: tools stay pure (query data, return a dict), and a single callback per agent handles all UI side-effects. This also means the error guard (status != &quot;success&quot;) lives in one place rather than being repeated across every tool.</p><p><strong>Gemini’s multimodal capability is real.</strong> Reading a SEPA EPC QR code from a bill photo mid-conversation, with no preprocessing, and extracting all four required payment fields correctly — that’s not something I would have attempted with previous model generations. It just works.</p><h3>What’s Next</h3><p>The current implementation is a proof of concept with mock Firestore data. The path to production would require:</p><ul><li><strong>Real bank API integration</strong> via PSD2/Open Banking APIs</li><li><strong>Strong Customer Authentication (SCA)</strong> — biometric or PIN for transfers</li><li><strong>Fraud detection hooks</strong> — real-time scoring before executing transfers</li><li><strong>Persistent session history</strong> — so that the agent remembers your preferences and actions across calls</li><li><strong>Multi-language support</strong> — Gemini Live supports multiple languages natively</li></ul><p>The architecture is already production-shaped. Cloud Run scales automatically, Firestore handles concurrent writes gracefully, and the WebSocket protocol is transport-agnostic enough to swap in any real banking backend behind the existing tool interfaces.</p><h3>Try It Yourself</h3><p>The full source code is on <a href="https://github.com/ayushworks/northriver-banking-agent/tree/main">GitHub</a> , with a comprehensive setup guide including steps to reproduce all three demo flows end-to-end.</p><p>The stack is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*3CO1kK1Zt8cL8N3_ARdn4w.png" /></figure><p>If you’re building voice agents with Google ADK, I hope this walkthrough saves you some of the debugging hours I spent — particularly on the audio pipeline and the side-channel pattern for pushing tool results to the UI.</p><p><em>Built for the </em><strong><em>Gemini Live Agent Challenge</em></strong><em> — March 2026.</em> <em>#GeminiLiveAgentChallenge #GoogleADK #GeminiLive #VertexAI #CloudRun #VoiceAI</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0ecb4cdf8002" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everyone’s Using MCP. I Built a CLI Instead. Here’s What I Learned.]]></title>
            <link>https://ayushm4489.medium.com/everyones-using-mcp-i-built-a-cli-instead-here-s-what-i-learned-a0ccc838e411?source=rss-2839f9309087------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0ccc838e411</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[design]]></category>
            <dc:creator><![CDATA[ayush mittal]]></dc:creator>
            <pubDate>Mon, 09 Mar 2026 08:37:15 GMT</pubDate>
            <atom:updated>2026-03-09T08:37:15.336Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>MCP (Model Context Protocol)</strong> has become the go-to standard for connecting AI agents to external tools. With platforms like <a href="https://openclaw.ai/">OpenClaw</a> gaining traction, there’s a growing debate in the AI engineering community about whether MCP is always the right architecture — or whether a well-designed CLI might be a better fit for certain use cases.</p><p>I came across this debate and wanted to dig into the actual mechanics — what lands in the context window, when, and at what cost. That led me to a simpler question: could a CLI with Unix pipes do the same job with less overhead?</p><p>This post is about that experiment.</p><h3>What MCP Does Well, and Where It Gets Complicated</h3><p>MCP is a standardised protocol for connecting LLMs to external tools. The idea is clean: define a schema for each tool, the LLM calls tools by name with structured arguments, gets structured responses back. It works, and it works well for many situations.</p><p>But there are some mechanics worth understanding before you adopt it everywhere.</p><p><strong>Schema loading happens at session start.</strong> When an MCP session begins, all tool schemas for every connected server get loaded into the LLM’s context window before the user asks a single question. The <a href="https://docs.stripe.com/mcp">official Stripe MCP server</a> has 20 tools. Even if you only use 3 of them in a session, all 20 schemas are sitting in context consuming tokens.</p><p>I ran a rough token count. Those 20 Stripe tool schemas come to around 2,100 tokens. A one-line skill file index entry for the same CLI tool costs about 23 tokens. That’s not catastrophic for a short session, but it compounds across many sessions and starts to matter when you’re paying per token in production.</p><p><strong>Responses land in context unfiltered.</strong> When the LLM calls a Stripe tool to retrieve a charge, the full charge object comes back — all 40+ fields. If the LLM wanted to know whether a charge succeeded, it now has 40 fields in context when it needed 2. MCP has no built-in mechanism to say “only return these fields.”</p><p><strong>Operations don’t compose.</strong> If you want to count failed charges in the last 7 days, MCP requires a tool call to list charges, the LLM receives the full list, then it counts them. The counting happens inside the LLM. With a CLI and Unix pipes, the counting happens in the shell before the result reaches the LLM at all:</p><pre>stcli charges list --since 7d --status failed --limit 100 --fields id \<br>  | jq &#39;length&#39;</pre><p>The LLM receives one number. Not a list of charge objects.</p><p>None of this makes MCP wrong. But these are real costs that matter in certain contexts, and they’re worth understanding before you decide.</p><h3>The CLI Pattern for AI Agents</h3><p>The alternative I explored is simpler and older. The LLM writes shell commands. An executor runs them. The output comes back as JSON. The LLM reads it and either asks another question or gives a final answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Et2eK2wUIFNYrKQwmiV9Vw.png" /></figure><p>Here is how each step works in practice:</p><p><strong>Step 1 — The LLM receives the user’s question plus a skill file.</strong> The skill file is not documentation — it’s a set of ready-made command recipes for common situations. The LLM doesn’t discover tool capabilities through trial and error; it finds the matching recipe and runs it.</p><p><strong>Step 2 — The LLM writes a bash command block.</strong> The executor detects it, runs it as a shell subprocess, and captures stdout and stderr separately.</p><p><strong>Step 3 — stcli runs.</strong> It calls the Stripe API, filters the response through --fields, and pipes through jq if aggregation is needed. Clean JSON lands on stdout.</p><p><strong>Step 4 — The LLM reads the result.</strong> If the question is answered, it returns a final response to the human. If not, it writes another command and the loop repeats.</p><p>What makes this efficient is that intermediate data never enters the context window. Unix pipes handle it at the OS level. The LLM only ever sees the final output of a pipeline.</p><h3>Why I Used Stripe as the Test Case</h3><p>Stripe was a deliberate choice. It’s a domain most engineers understand, the Stripe API is well-documented, and — crucially — Stripe already has an official MCP server. This meant I could build a CLI alternative and compare it directly against something real that exists today, rather than arguing in the abstract.</p><p>The use case was concrete: an AI agent that answers business questions about a Stripe account. Things like:</p><ul><li>How many failed charges did we have this week?</li><li>What’s our current available balance?</li><li>Find the customer registered with this email address</li><li>How much revenue did we collect in the last 30 days?</li></ul><p>These are the kinds of questions that currently require someone to log into the Stripe dashboard, click around, and copy numbers out manually. An agent that can answer them from the terminal — or as part of a larger automated workflow — saves real time for developers and finance teams who do this regularly.</p><h3>Building stcli</h3><p>I built stcli — a Stripe CLI designed specifically for AI agents, not humans.</p><p>Three principles run through every file in the codebase:</p><p><strong>JSON output only.</strong> Every command prints JSON to stdout. No colour coding, no tables, no human-friendly formatting that would break parsing.</p><p><strong>Errors go to stderr.</strong> Error messages go to stderr as JSON with exit code 1. stdout stays clean, so piping to jq never breaks due to an error message corrupting the stream.</p><p><strong>Reduce by default.</strong> Every command has a --fields flag. Ask for 3 fields, get 3 fields — not 40.</p><pre># Full charge object — ~40 fields in context<br>stcli charges get ch_abc123<br><br># With --fields — exactly what the agent needs<br>stcli charges get ch_abc123 --fields id,status,amount_formattedb</pre><p>The project is five Python files, zero external dependencies, and a skill file:</p><pre>stcli/<br>├── auth.py          # resolves API key: env var → config file → error<br>├── client.py        # all HTTP logic, single _request() method<br>├── formatting.py    # --fields filter, amount formatting, date parsing<br>├── commands/<br>│   ├── balance.py<br>│   ├── charges.py<br>│   ├── customers.py<br>│   └── configure.py<br>├── main.py          # argument parser, routes to command functions<br>└── stcli.skill.md   # command recipes for the LLM</pre><p>A few decisions worth explaining:</p><p><strong>Auth never reaches the LLM.</strong> The API key is resolved inside auth.py before any command runs. Priority chain: STRIPE_API_KEY env var first, then ~/.stcli/config. The executor and LLM never see the key.</p><p><strong>Human-friendly date inputs.</strong> Stripe expects Unix timestamps. The CLI accepts --since 7d or --since 2026-01-01 and converts internally, so the agent writes something readable rather than a timestamp integer.</p><p><strong>Amount formatting.</strong> Stripe stores amounts in smallest currency units — $50.00 is stored as 5000. Every charge response includes an amount_formatted field so the agent can reason over $50.00 directly.</p><p><strong>The skill file contains recipes, not docs.</strong> Every entry is a complete, runnable command for a specific question. The LLM finds the recipe, runs it, and gets an answer without figuring anything out.</p><pre># Count failed charges in last 7 days<br>stcli charges list --since 7d --status failed --limit 100 --fields id \<br>  | jq &#39;length&#39;<br><br># Total revenue from succeeded charges last 30 days<br>stcli charges list --since 30d --status succeeded --limit 100 --fields amount \<br>  | jq &#39;[.[].amount] | add | . / 100&#39;</pre><p>Full code: <a href="https://github.com/ayushworks/stcli"><strong>github.com/ayushworks/stcli</strong></a></p><h3>The Agent Loop in Action</h3><p>I tested stcli using Claude Code as the executor. Claude Code has the agent loop built in — it detects bash blocks in its output, runs them, reads the results, and loops until the question is answered. No extra infrastructure needed.</p><p><em>“How many charges do I have in the last 30 days and what was the total revenue?”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9sH93pqEbvw-083Pf8QHGg.png" /></figure><p>Claude Code wrote two stcli commands on its own. The first used --fields id to count — it only needed IDs. The second used --fields amount,currency to sum revenue. It didn&#39;t pull the full charge object either time. That&#39;s the agent-first design working as intended.</p><p><em>“What is my current balance?”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gqEJCosaPQOuSTKYmy7AsA.png" /></figure><p><em>“Find me all customers and show their email and names”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BSDGGyw9Y24prYOy_qKOcw.png" /></figure><p>One command, three fields, a clean table back to the user — not the full customer object with metadata, timestamps, and payment method details the question didn’t ask for.</p><h3>Architecture Comparison</h3><p><strong>MCP Architecture:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5BRaeWn6RX-cnSRrpD0tXA.png" /><figcaption>MCP Architecture — <a href="https://themlarchitect.wpcomstaging.com/blog/the-architectural-elegance-of-model-context-protocol-mcp/">https://themlarchitect.wpcomstaging.com/blog/the-architectural-elegance-of-model-context-protocol-mcp/</a></figcaption></figure><p>The MCP model has the LLM and an MCP client together in the host application. The client connects to separate server processes via JSON-RPC — one per external service. Each server exposes its full tool schema at session start.</p><p><strong>CLI Architecture:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xy5sjsgY8gzaSP6rf24bAQ.png" /><figcaption>CLI Agent Architecture</figcaption></figure><p>The LLM writes bash commands. The executor runs them as subprocesses. stcli handles auth and shapes the output. jq handles aggregation in the pipe before results reach the LLM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sHbGDRdZhhNvFCnNDTOpUA.png" /></figure><h3>When to Use Which</h3><p><strong>Choose MCP when:</strong></p><ul><li>You need multiple unrelated tools in one session — Slack, GitHub, filesystem all at once</li><li>You’re consuming a vendor-maintained MCP server and don’t want to own the wrapper</li><li>Your team uses GUI clients like Claude Desktop or Cursor that pick up MCP tools automatically</li><li>The API surface is large and you need full schema discovery</li></ul><p><strong>Choose CLI when:</strong></p><ul><li>You own the API wrapper and can design it from scratch for agent use</li><li>Sessions are long and make many API calls — token costs from full responses add up</li><li>Your use cases involve aggregation or filtering that’s cheaper to do in the shell than in the LLM</li><li>You want full transparency — every HTTP call visible, no abstraction layer between your code and the API</li><li>You’re in a regulated or security-sensitive environment where minimising data in context matters</li></ul><h3>What the Exercise Taught Me</h3><p>Before building stcli, I thought of CLI vs MCP as a technical architecture choice. After building it, I think of it as a product decision — one that depends on who owns the integration, how long sessions are, whether composability matters, and what the existing tooling looks like.</p><p>For a long-running internal Stripe agent making many calls per session, the CLI pattern made more sense. For a general-purpose agent needing a dozen different services, MCP would probably win.</p><p>The point isn’t to argue against MCP. It’s to understand what you’re trading off when you choose it, and know there’s a simpler alternative worth considering when those tradeoffs matter.</p><p>Code is at <a href="https://github.com/ayushworks/stcli"><strong>github.com/ayushworks/stcli</strong></a>. If you have a Stripe test account and Claude Code installed, the README will get you running queries in a ten minutes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0ccc838e411" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A One-Line Prompt Change That Changed My AI-Generated Code (But Not How I Expected)]]></title>
            <link>https://ayushm4489.medium.com/a-one-line-prompt-change-that-changed-my-ai-generated-code-but-not-how-i-expected-61b4895ebc35?source=rss-2839f9309087------2</link>
            <guid isPermaLink="false">https://medium.com/p/61b4895ebc35</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[github-copilot]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[ayush mittal]]></dc:creator>
            <pubDate>Thu, 26 Feb 2026 23:09:40 GMT</pubDate>
            <atom:updated>2026-02-26T23:09:40.386Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently read <a href="https://simonwillison.net/guides/agentic-engineering-patterns/red-green-tdd/">Simon Willison’s note on agentic engineering patterns</a>, and one tiny instruction stuck with me:</p><p><strong>“Use red/green TDD.”</strong></p><p>The claim is intuitive: if a coding agent writes tests first (red) and then implements code to make them pass (green), you should get more reliable code and better test coverage.</p><p>I got curious: <strong>does this one line actually change the quality of AI-generated code in a measurable way?</strong></p><p>So I tried a small experiment.</p><h3>What I tested</h3><p>I asked an AI coding assistant (Claude Sonnet 4.5 via GitHub Copilot in IntelliJ) to implement the <em>same</em> tiny Java utility twice:</p><h3>Prompt A (baseline)</h3><blockquote><em>Implement a Java utility </em><em>extractFirstJson(String text, int maxChars) that returns the first JSON object or array found in the input string. Use Jackson for validation. Add unit tests.</em></blockquote><h3>Prompt B (TDD)</h3><blockquote><em>Implement a Java utility </em><em>extractFirstJson(String text, int maxChars) that returns the first JSON object or array found in the input string. Use Jackson for validation. Add unit tests. </em><strong><em>Use red/green TDD.</em></strong></blockquote><p><strong>Only one difference</strong>: the last sentence.</p><h3>The utility: “extract first JSON” from messy text</h3><p>This is a problem that comes up constantly in AI applications. LLM responses often contain:</p><ul><li>extra explanatory text around JSON — “Sure! Here you go: { ... }”</li><li>Markdown code fences — { “a”: 1 }</li><li>Multiple JSON blobs in one response</li><li>Invalid JSON followed by valid JSON later</li></ul><p>The function is small, but there are enough edge cases to expose differences in approach.</p><h3>What I measured</h3><p>I looked at two things:</p><h4>1) JaCoCo coverage</h4><p>Coverage is not a full definition of “quality”, but it’s a useful signal — especially to detect variance across runs.</p><h4>2) A tiny holdout suite (“hidden tests”)</h4><p>Coverage can lie if the AI writes tests that match its own blind spots. So I added a small set of evaluator tests that I did <strong>not</strong> include in the prompt.</p><p>Key semantic contract for evaluation:</p><p>✅ <strong>Return the first <em>valid</em> JSON object/array anywhere in the string, skipping invalid candidates.</strong></p><h3>Summary table</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C13i6NvlCJ53Kmz2g3GzQQ.png" /><figcaption>These numbers are taken directly from the JaCoCo screenshots included in this post. The goal is to illustrate variance patterns, not to present a statistically rigorous benchmark.</figcaption></figure><h4><strong>Prompt A low coverage example</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZaWKcnQrWDmA-kDnBbluaA.png" /><figcaption>Prompt A: one run dipped noticeably on line/branch coverage due to missed validation/helper branches.</figcaption></figure><h4><strong>Prompt B high coverage but holdout failure</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bx1MfljiMo8ZFMtzShSlHw.png" /><figcaption>Prompt B: high coverage, but holdout tests exposed a semantic mismatch (“stop after first invalid candidate”)</figcaption></figure><h3>The hidden tests (the independent judge)</h3><p>Here’s the minimal holdout suite I used. It was kept in a separate test file (never included in the prompt) so it acts as an external evaluator.</p><pre>    @Test<br>    void quoteNoiseInPrefix_shouldStillExtractJson() {<br>        String text = &quot;He said \&quot;hello\&quot; then {\&quot;a\&quot;:1} end&quot;;<br>        String result = JsonExtractor.extractFirstJson(text, 10_000);<br>        assertEquals(&quot;{\&quot;a\&quot;:1}&quot;, result);<br>    }<br><br>    @Test<br>    void invalidThenValidLater_shouldReturnLaterValidJson() {<br>        String text = &quot;prefix {\&quot;invalid\&quot;:} middle {\&quot;ok\&quot;:1} suffix&quot;;<br>        String result = JsonExtractor.extractFirstJson(text, 10_000);<br>        assertEquals(&quot;{\&quot;ok\&quot;:1}&quot;, result);<br>    }<br><br>    @Test<br>    void jsonInsideMarkdownCodeFence_shouldExtract() {<br>        String text = &quot;```json\n{\&quot;a\&quot;:1}\n```&quot;;<br>        String result = JsonExtractor.extractFirstJson(text, 10_000);<br>        assertEquals(&quot;{\&quot;a\&quot;:1}&quot;, result);<br>    }<br><br>    @Test<br>    void bracesInsideJsonString_shouldNotBreakExtraction() {<br>        String text = &quot;prefix {\&quot;t\&quot;:\&quot;} not end\&quot;, \&quot;x\&quot;:1} suffix&quot;;<br>        assertEquals(&quot;{\&quot;t\&quot;:\&quot;} not end\&quot;, \&quot;x\&quot;:1}&quot;, JsonExtractor.extractFirstJson(text, 10_000));<br>    }<br><br>    @Test<br>    void invalidObjectThenValidArray_shouldReturnArray() {<br>        String text = &quot;prefix {\&quot;bad\&quot;:} middle [1,2,3] suffix&quot;;<br>        assertEquals(&quot;[1,2,3]&quot;, JsonExtractor.extractFirstJson(text, 10_000));<br>    }<br><br>    @Test<br>    void primitiveJsonShouldBeIgnored_thenReturnFirstObjectOrArray() {<br>        String text = &quot;prefix 42 middle {\&quot;ok\&quot;:1} suffix&quot;;<br>        assertEquals(&quot;{\&quot;ok\&quot;:1}&quot;, JsonExtractor.extractFirstJson(text, 10_000));<br>    }<br><br>    @Test<br>    void primitiveJsonIsValidButShouldNotCount_asExtractedJson() {<br>        String text = &quot;prefix 42 middle {\&quot;ok\&quot;:1} suffix&quot;;<br>        assertEquals(&quot;{\&quot;ok\&quot;:1}&quot;, JsonExtractor.extractFirstJson(text, 10_000));<br>    }</pre><p>Why these tests?</p><ul><li>They’re small and realistic</li><li>They don’t reward a specific implementation style (regex vs scanner)</li><li>They catch a failure mode that matters in real LLM pipelines: <strong>invalid JSON first, valid JSON later</strong></li></ul><h3>The surprising part: Prompt B sometimes failed the holdout tests</h3><p>In multiple Prompt B runs, two holdout tests failed consistently:</p><ul><li>invalidThenValidLater_shouldReturnLaterValidJson</li><li>invalidObjectThenValidArray_shouldReturnArray</li></ul><p>The implementations in those runs behaved like this:</p><ol><li>find the first { or [</li><li>extract a candidate</li><li>validate with Jackson</li><li>if invalid → <strong>return null immediately</strong> (stop scanning)</li></ol><p>This is a <em>reasonable</em> interpretation if you didn’t explicitly state “first valid JSON anywhere”. But it violates the behavior I actually wanted for AI outputs.</p><p>Here’s what’s interesting:</p><p><strong>Red/green TDD didn’t “cause” the bug — it reinforced an assumption</strong></p><p>When the agent starts with a test like:</p><blockquote><em>“Invalid JSON should return null”</em></blockquote><p>…the implementation will faithfully implement that behavior in the green phase.</p><p>So Prompt B often created <strong>tight internal alignment</strong> between its tests and its code — but still diverged from the intended semantic contract.</p><h3>Another interesting observation: Prompt B often produced fewer tests</h3><p>I noticed Prompt B sometimes generated fewer unit tests than Prompt A.</p><p>That’s consistent with a strict red/green mindset:</p><ul><li>write the minimal test to go red</li><li>implement the minimal code to go green</li><li>stop when everything passes</li></ul><p>In contrast, Prompt A sometimes produced a more “exploratory” test suite — more cases, more variety, more variance.</p><p>Neither is inherently good or bad. But it affects what coverage and robustness look like.</p><h3>What I learned</h3><h4>1) Coverage is a useful signal, not a final metric</h4><p>High coverage can still miss important behavior if the tests don’t express it.</p><h4>2) “Use red/green TDD” improves internal consistency</h4><p>Prompt B tended to produce more stable coverage (in the screenshots I collected) and a tighter “tests drive code” workflow.</p><h4>3) TDD does not replace a clear spec</h4><p>If a requirement is subtle — like:</p><blockquote><em>“Return the first </em>valid<em> JSON object/array anywhere, skipping invalid candidates.”</em></blockquote><p>…then you should say it explicitly.</p><p>Otherwise, TDD can make the agent <strong>confidently correct according to its own interpretation</strong>, which may not match yours.</p><h3>Practical takeaway</h3><p>If you’re using coding agents:</p><ul><li>Add <strong>“Use red/green TDD.”</strong> when you want disciplined, test-driven changes.</li><li>But also add <strong>one explicit behavior sentence</strong> for any subtle semantic contract.</li></ul><p>For this utility, the difference-maker would be:</p><blockquote><em>“Return the first valid JSON object/array anywhere in the text, skipping invalid candidates.”</em></blockquote><p>That single sentence would likely reduce the “semantic mismatch” failures dramatically — while still keeping the prompt short.</p><h3>Appendix: prompts used</h3><p><strong>Prompt A</strong></p><blockquote><em>Implement a Java utility </em><em>extractFirstJson(String text, int maxChars) that returns the first JSON object or array found in the input string. Use Jackson for validation. Add unit tests.</em></blockquote><p><strong>Prompt B</strong></p><blockquote><em>Implement a Java utility </em><em>extractFirstJson(String text, int maxChars) that returns the first JSON object or array found in the input string. Use Jackson for validation. Add unit tests. Use red/green TDD.</em></blockquote><h3>Notes &amp; limitations</h3><p>This is a practical experiment, not a controlled study:</p><ul><li>LLM output varies run-to-run</li><li>I didn’t fully automate the generation loop</li><li>Coverage is influenced by the generated tests</li></ul><p>Still, the overall pattern was clear enough to be useful: <strong>TDD improves how reliably the agent drives code via tests — but it still needs a clear behavioral spec to drive toward the right behavior.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61b4895ebc35" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transforming ING’s Contact Center with Generative AI]]></title>
            <link>https://medium.com/ing-blog/transforming-ings-contact-center-with-generative-ai-02cb4f24c542?source=rss-2839f9309087------2</link>
            <guid isPermaLink="false">https://medium.com/p/02cb4f24c542</guid>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[banking]]></category>
            <category><![CDATA[ing]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[ayush mittal]]></dc:creator>
            <pubDate>Fri, 14 Mar 2025 10:58:38 GMT</pubDate>
            <atom:updated>2025-03-18T08:13:48.611Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*h9WWA_cqbTGm1B24gY3g3g.jpeg" /></figure><p>Generative AI is revolutionising customer service and transforming customer service industries. It has the ability to automate interactions, provide personalised experiences, and enhance decision-making. GenAI has the potential to enhance existing technologies in contact centers by automating routine tasks, streamlining information gathering, and improving analytics. This allows human agents to focus on higher-value interactions which enhances efficiency, personalises customer experiences and boosts overall service effectiveness. <strong>When harnessed correctly</strong>, GenAI can deliver positive outcomes at scale.</p><p>However at the end of the day, the proof is in the pudding and the pudding in this case is the <strong>scale</strong>. It’s the <strong>transition from proof-of-concepts in sandboxes to integrating GenAI in existing systems to deliver globally scalable, secure, high grade solutions</strong>. This blog explores how<strong> ING’s Contact Center 2.0 platform allows its users across various geographies to design, develop and deploy a safe, secure and scalable GenAI powered chatbot</strong>. We share some challenges that we faced while integrating GenAI into our existing Contact Center 2.0 tech stack.<strong> </strong>Most importantly, we share how we have ensured that the integration of GenAI into our existing platforms is done by prioritising long-term planning over short-term gain, avoiding any inconsistent customer experiences and omitting any redundant technologies along with unmanaged risks. It’s about using GenAI to deliver true value to our customers rather than being swayed by hype.</p><p>Table of contents</p><ul><li><a href="#0754">Introduction to the CC2.0 platform</a></li><li><a href="#3419">Virtual Assistant API — AI Engine of the CC2.0 platform</a></li><li><a href="#e1b7">Moving from Traditional NLP to a Conversational AI Architecture</a></li><li><a href="#decf">Engineering for Scale : How to develop a safe, scalable GenAI powered chatbot</a></li><li><a href="#0268">Responsible By Design</a></li><li><a href="#b1ed">Summary</a></li></ul><h3>Introduction to the CC2.0 platform</h3><p>At the core of ING’s customer interactions is <strong>Contact Center 2.0</strong>, a multi-tenant, multi-channel customer engagement platform that connects customers with both human agents and virtual assistants. It follows a model similar to cloud-based platforms like <strong>AWS Lambda</strong>, where the core infrastructure and execution environment are centrally managed, while each country or business unit defines its own operational logic. The tenants of this platform are ING business units and countries that use the platform to provide services to their customers. Each tenant customizes <strong>routing rules, workflows, and integrations</strong> to tailor customer interactions to their specific needs. These configurations determine how <strong>calls, chats, and video interactions</strong> are processed, escalated, and monitored. We have talked about it before as well — <a href="https://medium.com/ing-blog/ing-contact-center-2-0-creating-resilient-apis-442c2e9ca6a7">ING Contact Center 2.0 — Creating Resilient APIs</a>.</p><p>CC2.0 is built on a combination of <strong>vendor solutions</strong> and <strong>custom ING APIs</strong>:</p><ul><li><strong>Twilio</strong> — Provides voice, chat, and video capabilities.</li><li><strong>Google Dialogflow</strong> — ING’s <strong>Conversational AI</strong> provider.</li><li><strong>ING APIs</strong> — Custom-built APIs for <strong>orchestration, compliance, and security</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*SWUSgJdlwfBSy7goEo1VfA.png" /></figure><h3>Virtual Assistant API — AI Engine of the CC2.0 platform</h3><p>While CC2.0 orchestrates customer interactions, ING’s <strong>Virtual Assistant API (VA API) </strong>powers the CC2.0’s AI capabilities.</p><p>The Virtual Assistant API (VA API) follows the same <strong>SaaS/PaaS</strong> model as <strong>Google Dialogflow</strong>, where the <strong>platform provides the core infrastructure, orchestration, and execution environment, while tenants focus solely on their virtual assistant configurations</strong>. Just as Google Dialogflow allows businesses to define conversational flows, agents, and intents without managing backend services, <strong>VA API enables ING business units to configure their virtual assistants while the platform centrally manages the infrastructure</strong>. Each ING business unit configures its own <strong>Dialogflow setup, agent structures, intent definitions, and conversational flows</strong> to align with its specific language, compliance, and customer engagement needs. However, <strong>VA API centrally manages the orchestration, scaling, and compliance guardrails</strong>, ensuring that all virtual assistants function within a standardized ING ecosystem. By functioning as an <strong>internal SaaS/PaaS offering</strong>, <strong>VA API provides a fully managed environment for conversational AI, enabling business units to focus on building effective bot interactions while relying on ING’s platform for scalability, security, and governance.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LGW48XpgIqkDaL3LNsjpJA.png" /></figure><p>VA API performs the following core actions -</p><p><strong>✔ Routes customer interactions to the right virtual assistant</strong><br><strong>✔ Maintains conversation state and session data</strong><br><strong>✔ Integrates with Google Dialogflow for Conversational Gen AI processing</strong><br><strong>✔ Connects with ING’s backend for enriched customer responses</strong></p><p>The VA API inside the CC2.0 platform ensures that we have a solid foundation into which we can integrate any new technology like GenAI. <strong>To deliver scalable solutions, we need scalable platforms as a foundation.</strong></p><h3>Moving from Traditional NLP to a Conversational AI Architecture</h3><h4>Why change ?</h4><p>ING countries and tenants of CC2.0 platform have been using the CC2.0 and VA API platform for many years now to create chatbots and callbots powered solutions for their customers. While intent-based NLP models have been effective for handling structured, rule-based interactions, they struggle with complex queries, long conversations, and unstructured inputs. Traditional chatbots rely on <strong>predefined intents</strong> to match customer queries to scripted responses. While effective for repetitive and well-defined tasks, these bots often struggle when:</p><ul><li>Customers ask <strong>open-ended or unexpected questions</strong>.</li><li>The chatbot needs to <strong>maintain context across multiple interactions</strong>.</li><li>Queries require information <strong>outside predefined workflows.</strong></li></ul><p>The solution to these problems — Generative AI. Using the power of GenAI we could potentially build next-gen virtual agents that could provide a more natural conversational flow and have the capability to answer any question. Theoretically, the GenAI powered bots could solve all the issues that the traditional bots could not.</p><h4>Challenges to overcome</h4><p>The transition from traditional NLP-based bots to a Generative AI based chat bot model was a significant challenge. The goal was to use GenAI to enhance the customer experience and provide a more conversational flow to our bot. We aimed to use the technology to fill the gaps and to extend on the current capabilities. We also wanted to ensure that the resulting design and solution does not suffer from traditional problems that occur when a new, hype technology is first used — Dependency on specific vendors in the design, inconsistent user experiences at different stages of the customer journey, redundant and/or incompatible technologies, untracked risks, “shadow IT” and other improper applications.</p><p>Specifically when integrating GenAI into our Contact Center solution, we wanted to achieve the following non-negotiable goals :</p><ul><li><strong>Accurate responses without any biases and hallucinations.</strong></li><li><strong>Superior customer experience without any delays or latency.</strong></li><li><strong>Controlled, validated and predictable changes even with ever evolving nature of GenAI technlogy.</strong></li><li><strong>Complete performance isolation for a multi tenant platform.</strong></li></ul><h3>Engineering for Scale : How to develop a safe, scalable GenAI powered chatbot</h3><h4>1. A hybrid architecture: Best of both worlds</h4><p>CC2.0 platform was already integrated with Google Dialogflow for traditional bots. With that integration already present we could take advantage of two core capabilities that Dialogflow provided :</p><ul><li><strong>Deterministic AI (Flows)</strong>– Structured, predefined workflows that ensure predictable and controlled responses.</li><li><strong>Generative AI (Vertex AI LLMs)</strong> — AI-generated responses that provide more <strong>flexible and natural conversations</strong> when predefined rules don’t cover a query.</li></ul><p>We decide that to adopt a hybrid approach while integrating GenAI into chatbot solution. Instead of completely replacing NLP’s with GenAI we decided to take the best of both. In this decision we were supported by <a href="https://cloud.google.com/products/conversational-agents?hl=en">Google’s Conversational AI platform</a> that enabled us to design hybrid conversational agents that can:</p><ul><li><strong>Follow intent-driven workflows</strong> for structured customer interactions.</li><li><strong>Use Generative AI when needed</strong> to provide <strong>context-aware, free-flowing responses</strong>.</li><li><strong>Retrieve knowledge from structured and unstructured data sources</strong> using <a href="https://cloud.google.com/dialogflow/cx/docs/concept/data-store"><strong>Data Stores</strong></a>, ensuring accurate and fact-based responses.</li></ul><h4>2. Leveraging Data Stores for accurate AI Responses — Retrieval Augmented Generation (RAG)</h4><p>LLM’s can at times provide out of date, incorrect informations. As explained by Martin Fowler <a href="https://martinfowler.com/articles/gen-ai-patterns/#rag">here</a></p><blockquote><em>A common metaphor for an LLM is a junior researcher. Someone who is articulate, well-read in general, but not well-informed on the details of the topic — and woefully over-confident, preferring to make up a plausible answer rather than admit ignorance. With RAG, we are asking this researcher a question, and also handing them a dossier of the most relevant documents, telling them to read those documents before coming up with an answer.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_8KviL4VI0VyoOG8g_hzCg.png" /></figure><p>For our chatbot, RAG (Retrieval-Augmented Generation) ensures accurate and up-to-date responses by fetching real-time data from trusted sources. This prevents the chatbot from relying solely on pre-trained knowledge, reducing hallucinations and improving response reliability. By dynamically retrieving relevant information, RAG enables the chatbot to handle complex customer queries, such as interest rates, loan eligibility etc. with greater accuracy. By using RAG we enhanced the chatbot’s ability to <strong>retrieve and process real-time knowledge</strong>.</p><p>For implementation we used <a href="https://cloud.google.com/dialogflow/cx/docs/concept/data-store"><strong>Dialogflow CX’s Data Stores</strong></a><strong>. </strong>With this technology<strong> </strong>ING’s virtual assistants can:</p><ul><li>Index and analyze ING’s publicly available content.</li><li>Retrieve relevant responses from these sources before generating answers, making chatbot interactions more accurate and informative.</li><li>Maintain more personal context given by the customer during multiple interactions with the chatbot.</li><li>Create more natural flowing conversations.</li><li>Minimize incorrect AI-generated responses, ensuring customer interactions remain compliant and fact-based.</li></ul><p>One of the significant benefits of this design is the reduction in maintenance and improvement in general information quality. Traditional flows are challenging to build to cover all available information, especially when it is constantly changing. The main requirement to improve the quality of the GenAI chatbot is to improve the information it uses at the source. Updating website and public information is key to a better performing chatbot. This has the added benefit that we are reducing the locations where we need to maintain the latest information. Even better, if our customers can more easily find their answers without the chatbot, there is less need to interact with it.</p><h4>3. Dynamic Timeout Adaptation: A smarter way to handle GenAI latency</h4><p>One of the biggest challenges in integrating <strong>Generative AI-based conversational agents</strong> is managing <strong>unpredictable response times</strong>. Unlike traditional API calls, where a <strong>fixed timeout can be defined for a given service</strong>, AI-driven conversations introduce variability — sometimes responding in milliseconds, and other times requiring more processing time for complex queries. To solve this, we adopted a Dynamic Timeout Adaptation mechanism — an approach that <strong>intelligently adjusts timeouts for each conversation, dynamically tuning them on a per-message basis</strong>.</p><p><strong>How it works: An adaptive timeout model</strong></p><p>Instead of relying on <strong>static timeouts</strong>, which can either be <strong>too short (causing premature failures) or too long (wasting resources on stalled requests)</strong>, the Virtual Assistant API dynamically calibrates its waiting time for each Dialogflow request. <strong>The magic lies in its adaptability</strong> — the timeout value is computed in real-time, factoring in:</p><p>✔ The moving average of response times from previous messages in the same conversation.<br>✔ The specific Dialogflow agent handling the conversation, as different AI models exhibit different latencies.<br>✔ Conversation context, allowing the system to recognize when a more complex query is likely to take longer.</p><p><strong>What we were able to achieve</strong></p><ol><li><strong>Latency-Resilient AI Conversations: </strong>By adapting to real-world response patterns, conversations remain fluid, without frustrating delays or unnecessary timeouts.</li><li><strong>Optimized System Performance: </strong>Instead of over-allocating resources for long timeouts or aggressively terminating requests, the system self-regulates for efficiency.</li><li><strong>Scalability Without Compromising Experience: </strong>Even as chat volumes scale into the thousands per minute, the platform ensures every AI-driven interaction feels instantaneous — no more unpredictable slowdowns.</li></ol><p>Implementing a dynamic timeout helped us to ensure that the customer experience was optimal even while the LLM’s response times were slow. By implementing Dynamic Timeout Adaptation, we were able to effectively eliminate one of the <strong>biggest bottlenecks in scaling GenAI-powered contact centers.</strong></p><h4>4. Test Sets: Scaling with Automated Validation</h4><p>One of the most impactful feats that has helped us scale the platform while integration with GenAI is the Test Set feature. This feature <strong>allowed bot builders (tenants) to upload and schedule test sets for their virtual agents</strong>, ensuring that <strong>Generative AI models behave as expected, even as they evolve</strong>.</p><p><strong>How the Test Set feature works</strong></p><p>A Test Set consists of multiple test flow, where each flow follows a structured format:</p><ul><li><strong>Query</strong> — A predefined input simulating a customer request.</li><li><strong>Expected response</strong> — The correct AI-generated reply for validation.</li><li><strong>Multiple flows</strong> — Chained interactions to verify long-running conversations.</li></ul><p>The tenants building there bots on the platform could:</p><ul><li><strong>Upload Test Sets for their Dialogflow agents</strong> within the VA platform.</li><li><strong>Schedule automated test runs</strong> to continuously validate their AI models.</li><li><strong>Receive pass/fail reports</strong>, ensuring that virtual agents remain consistent and compliant.</li></ul><p>Why the Test Set feature is critical for scaling Generative AI:</p><ul><li>Unlike static rule-based chatbots, <strong>Generative AI models evolve dynamically — which means responses may change based on new training data, fine-tuning, or parameter updates</strong>. The <strong>Test Set feature provides an automated way to continuously validate AI behavior</strong>, ensuring that: <br>Sanity checks are performed before and after production deployments, reducing risk.</li><li>Guardrails remain intact, preventing unintended AI behavior.</li><li>Regression testing catches AI drift, ensuring long-term reliability.</li></ul><p><strong>Essentially, this feature turns validation into a scalable, automated process, helping ING teams confidently roll out Generative AI enhancements without introducing inconsistencies.</strong></p><h4>5. Fortifying scalability: Isolating tenant workloads for stability</h4><p>In a multi tenant platform it’s important to isolate consumers and services from cascading failures. This also includes ensuring that <strong>one tenant’s surge in demand doesn’t degrade the experience for others. </strong>When we integrated GenAI powered bots with traditional NLP bots the workloads on system became varying. Also within GenAI-powered conversations, response times can be unpredictable — some queries are resolved instantly, while others require deeper AI processing. This variance can be dangerous in a shared-resource architecture, where a sudden influx of high-complexity GenAI queries from one tenant could block resources and delay critical customer interactions elsewhere.</p><p>To solve this, we <strong>doubled down</strong> on our existing <a href="https://learn.microsoft.com/en-us/azure/architecture/patterns/bulkhead"><strong>Bulkhead Design Pattern</strong></a> — a scalability safeguard that <strong>physically and logically isolates workloads per tenant</strong>, preventing a performance bottleneck in one part of the system from cascading into a full-scale outage. This concept, inspired by the watertight compartments of ships, ensures that <strong>each tenant operates in its own “sealed” execution environment within the CC2.0 ecosystem</strong>. We <strong>introduced resource allocation at the lowest possible layers</strong> upto virtual assistants.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xdwOxqrDVjTpZLlSVviDfg.png" /></figure><p>With this pattern in place we ensured that:</p><ul><li>Each tenant/agent gets a dedicated, isolated resource pool — ensuring that high-load tenants don’t disrupt others.</li><li>AI processing queues are managed independently, preventing long-running GenAI queries from slowing down unrelated conversations.</li><li>Resource starvation is eliminated — each tenant is allocated compute, memory, and execution threads proportionally, avoiding AI-induced bottlenecks in Google Dialogflow integration.</li><li>Automatic load balancing &amp; prioritization further enhance performance — ensuring high-priority interactions always get immediate processing power.</li></ul><p>The benefits for adopting this technique were:</p><ul><li><strong>True Multi-Tenancy without performance trade-offs. </strong>Unlike traditional shared models where one overloaded tenant can affect all others, Bulkhead ensures strict workload partitioning.</li><li><strong>Predictable AI Latency, even under load surges. </strong>Whether a tenant experiences a seasonal surge in chatbot usage or deploys an AI-heavy automation flow, it doesn’t throttle resources meant for others.</li><li><strong>Guaranteed service reliability for critical customer interactions. </strong>Banking customers rely on instant responses. By isolating workloads, ING ensures that AI-driven customer conversations remain smooth, stable, and performant at all times.</li></ul><p>By implementing Bulkhead Architecture, we were able to scale multi-tenant AI solutions while maintaining absolute performance isolation. This approach ensures that every tenant, no matter its size or AI complexity, gets a dedicated, resilient, and high-performance environment.</p><h3>Responsible By Design</h3><p>While GenAI offers unparalleled efficiency, moving to more dynamic generated conversations also introduces risks related to accuracy, bias, security, and compliance. While integrating GenAI into our chatbot offering, we had to ensure that the safety and compliance are guaranteed. Delivering responsible and risk averse solutions is integrated in ING’s WoW and something we take pride in.</p><h4>Guardrails &amp; Risk Mitigation</h4><p>Generative AI agents require more robust evaluation methods for their answers and continuous monitoring to prevent harmful responses. Evaluating the performance of a language model can be challenging, especially when customers might be satisfied with incorrect answers. Therefore, automated processes are necessary to evaluate the model based on many types of categories, harmful incorrectness being one of them. Additionally, more monitoring and labeling exercises are required to assess and improve the chatbot’s quality, as these automated processes need continuous validation.</p><p>During conversations, risk mitigators can be applied as additional guardrails evaluating the responses and requests. However, extra evaluation is also needed in this area to make sure these are working as intended in production. When using Generative AI to classify risks, we create inherent instability compared to traditional validation methods.</p><p>AI guardrails help prevent:</p><p>✔ Bias in AI models — Ensuring fair and non-discriminatory responses.</p><p>✔ Sensitive data leaks — Preventing unauthorized AI data exposure.</p><p>✔ Over-reliance on AI — Keeping human agents in control where needed.</p><p>Other risk mitigator when working with Dialogflow are:</p><p>✔ Model Explainability — AI decisions can be traced and understood.</p><p>✔ Bias Mitigation — AI responses are monitored for fairness.</p><p>✔ Human-in-the-Loop Oversight — AI escalates interactions to humans when needed.</p><p>That ING extends with:</p><p>✔ Confidence-Based Escalation — Virtual Assistant platform allows bot builders to configure a confidence level for Dialogflow agents. Low-confidence AI responses can trigger human handover.</p><p>✔ Real-Time Monitoring &amp; Auditing — AI outputs are continuously audited by CC2.0 and VA platform.</p><p>✔ Data Privacy &amp; Security — All Customer inputs are anonymised before processing.</p><p>Despite these additional challenges compared to traditional chatbot agents, we find that these measures scale well across multiple areas and languages. While customisations are necessary, the approaches and mechanisms can be reused and maintained in a centralized way, making the overall system more efficient and manageable.</p><h3>Summary</h3><p>Here at ING, by integrating GenAI into our global <strong>Contact Center 2.0 platform</strong> we are reshaping customer interactions — enhancing efficiency, ensuring compliance, and maintaining a customer-first approach. We leverage the transformative potential of Conversational GenAI to enhance operational efficiency, improve customer interactions, and foster innovation in our contact center operations through Contact Center 2.0 platform. Amid a dynamic landscape, we leverage technology while ensuring data security and compliance. By <strong>building on existing capabilities, addressing gaps, and implementing strong safeguards, we deliver impactful solutions at scale</strong> — focusing on what makes sense for the future rather than being swayed by hype.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=02cb4f24c542" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ing-blog/transforming-ings-contact-center-with-generative-ai-02cb4f24c542">Transforming ING’s Contact Center with Generative AI</a> was originally published in <a href="https://medium.com/ing-blog">ING Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ING Contact Center 2.0 — Creating Resilient APIs]]></title>
            <link>https://medium.com/ing-blog/ing-contact-center-2-0-creating-resilient-apis-442c2e9ca6a7?source=rss-2839f9309087------2</link>
            <guid isPermaLink="false">https://medium.com/p/442c2e9ca6a7</guid>
            <dc:creator><![CDATA[ayush mittal]]></dc:creator>
            <pubDate>Thu, 15 Sep 2022 12:33:31 GMT</pubDate>
            <atom:updated>2022-09-16T09:35:51.351Z</atom:updated>
            <content:encoded><![CDATA[<h3>ING Contact Center 2.0 — Creating Resilient APIs</h3><p>Contact Center 2.0 is ING’s global, omni-channel platform for customer communication. The platform is at the core of all communications between ING employees and its customers. As with any other mission critical system , CC 2.0 is designed and developed to be highly available and resilient. In this blog post we shall look at some patterns that engineers in the CC 2.0 family apply to achieve high availability and resiliency.</p><p>Before we go into details, let’s understand the CC 2.0 platform’s business goals and look at a high level design.</p><h3>Business goals for CC 2.0 Platform -</h3><blockquote>“ Offer <strong>one</strong> shared platform for all ING countries that facilitate communication between ING customers and ING employees via any channel.”</blockquote><p><strong>CC 2.0 and Twilio</strong></p><p>CC 2.0 uses <a href="https://www.twilio.com/">Twilio</a> to achieve the one shared platform goal. The reasons for choosing Twilio are -</p><ul><li>ING pays Twilio based on usage; there are no fixed charges for licences or maintenance.</li><li>ING can develop quickly with Twilio. With its APIs and SDKs, Twilio is a perfect fit in the ING architecture.</li><li>Practically every customer journey can be realised with Twilio because of its flexible setup with building blocks and easy integration.</li></ul><h3>CC 2.0 High level design</h3><p>The following diagram shows an overview of the functional capabilities of the Contact Center 2.0 platform. The capabilities are divided into 4 layers: touchpoints, communication channels, orchestration &amp; routing and utilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QO-hkmhYiszqVh37WrNneg.png" /><figcaption>CC2.0 AND TWILIO</figcaption></figure><p>All communication channels and routing functionalities are backed by one (or more) APIs that combine together with other ING APIs and Twilio to deliver a complete customer journey. CC2.0 backend APIs are critical to the working of this digital contact center. <strong>All APIs are designed and built with the principles of “<em>security</em>, <em>availability</em>, <em>resilience</em>, and <em>observability</em>.”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G0oAnNv-AnurfN0kVeXGig.png" /><figcaption>components overview</figcaption></figure><h3>CC 2.0 in Numbers</h3><p>The contact center 2.0 platform is currently used in more than 10 different countries for Chat, Voice, Email and Video based solutions. Here are some interesting statistics in terms of the platform usage and the load it handles.</p><p>1. <strong>Number of ING customers routed to ING agents via CC2.0 platform in chat channel in 2022. More than 100,000 customers were supported in June alone.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WZyXWgjaPwHFJi0qNz_IuA.png" /><figcaption>chat numbers</figcaption></figure><p>2. <strong>Number of Voice calls handle by CC2.0 platform in 3 major countries in 2022. Close to 1,500,000 calls are handled every month in just these 3 countries combined.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H_vDgJPpge3QtNDzxCrLYA.png" /></figure><p>3. <strong>Incoming calls from ING customers as seen for last 60 mins on a given day. The peak load is high as 150 incoming calls per minute.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DY8Z-gtEkQyEpozc5P2vcg.png" /></figure><p>Here is how these numbers translate to our load metrics :</p><ol><li><strong>On average we handle a peak load of 180 TPS in a single channel like voice or chat or email.</strong></li><li><strong>The average throughput for the most critical API’s in our platform is 8000 RPM.</strong></li><li><strong>The average response times for these API’s must be between 0.1 and 1 seconds for us to handle the load.</strong></li></ol><h3>Resiliency — CC2.0 Platform</h3><p>Two key goals in design of the platform are “availability” and “resiliency”. Resiliency in a distributed system is easier said than done. The very nature of distributed system increases the number of failure points. <strong>With a distributed system in micro-services architecture, faults can cause failures unless the system is resilient</strong>. Confused :), let’s understand the statement.</p><p><strong>Faults, Failures and Resilience -</strong></p><p><em>Fault is the presence of an invalid or false state in our internal system.</em></p><blockquote>“A fault is anything that behaves unexpectedly and could be any hardware, software, network, or operational aspect of the system that does something we didn’t plan for!”</blockquote><p>Many reasons exist that can cause a fault to happen. Some common examples of a software system faults include:</p><ol><li>Memory leaks in application</li><li>Threading issues</li><li>Dependency issues</li><li>Corrupt data</li></ol><p><em>Failure is an inability of the system to perform what it is meant to do</em>. Failure is when our customers are not able to reach us or when we cannot reach them. Failure is when our API’s are not reachable. Failure means loss of up-time and availability on systems.</p><p>So why do failures happen? <strong>Failures happen when Faults in one part of system are unchecked and they get propagated</strong>. Faults can lead to failures if not contained from propagating further into the system.</p><p><strong>So, Resilience is all about preventing faults turning into failures. We cannot guarantee fault free systems. However we can ensure high availability and resiliency by constraining faults and ensuring that they remain localised and do not turn into failures.</strong></p><h4>Common resiliency patterns used in CC 2.0 Backend API’s</h4><ul><li><strong>TIMEOUT</strong></li></ul><blockquote><em>“At some point, you have to give up.”</em></blockquote><p>Let’s consider the following scenario.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4Low4hQvAjQykRLCEY7rg.png" /></figure><p>We have an API <strong>A</strong> depending on API <strong>B</strong> for serving its requests. API B is for some reason not in a healthy state and is slow.</p><p>Implementing a timeout in the call from A to B will ensure fast failure. Slowness of B will be reported soon and A can either propagate the error further or use a fallback mechanism to still serve its request.</p><p><strong>Timeout at Controller Layer</strong></p><p>When a server receives an HTTP request, it’s good practice to specify the time limits within which this HTTP request must be completed.</p><p>For example in Jersey we can use <a href="https://eclipse-ee4j.github.io/jersey.github.io/documentation/latest/async.html">Jersey async</a> to specify the timeout is at the controller level.</p><pre>@Path(&quot;/tasks&quot;)<br>public class TaskResource {<br>  @GET<br>  public void createTask(@Suspended final AsyncResponse ar) {<br>      ExecutorService es = Executors.newSingleThreadExecutor();</pre><pre>      ar.setTimeoutHandler(new TimeoutHandler() {<br>          @Override<br>          public void handleTimeout(AsyncResponse asyncResponse) {<br>              asyncResponse.resume(&quot;Processing timeout.&quot;);<br>              es.shutdown();<br>          }<br>      });</pre><pre>      ar.setTimeout(2, TimeUnit.SECONDS);<br>      es.submit(() -&gt; {<br>          ..<br>      });<br>  }<br>}</pre><p><strong>Timeout for Twilio calls</strong></p><p>We use an apache http client to configure the timeout policies for ING-Twilio API calls.</p><pre>def buildClient: CloseableHttpClient = {<br>  val clientBuilder = HttpClients.<em>custom</em></pre><pre><em>  </em>val requestBuilder = RequestConfig.<em>custom<br>    </em>.setConnectTimeout(connectTimeout)<br>    .setSocketTimeout(socketTimeout)<br>    .setConnectionRequestTimeout(connectionRequestTimeout)</pre><pre>  clientBuilder.setDefaultRequestConfig(requestBuilder.build).build<br>}</pre><p>The apache client is created inside a INGHttpClient which is an extension of Twilio’s HTTPClient.</p><pre>class INGHttpClient extends HttpClient {</pre><pre>  val client = buildClient</pre><pre>  override def makeRequest(request: Request): Response = {<br>    client.execute(request)<br>  }</pre><p>The INGHttpClient being an extension of Twilio HTTPClient can be given to a TwilioRestClient. This is the class that allows access to Twilio api’s.</p><pre>def buildTwilioClient(httpClient: HttpClient): TwilioRestClient = {<br>  new TwilioRestClient.Builder(sid, token)<br>    .httpClient(httpClient)<br>    .build<br>}</pre><p>And finally this TwilioRestClient can be used to access Twilio resources</p><pre>Task task = Task.creator(&quot;WSXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&quot;)<br>            .setAttributes(&quot;{\&quot;type\&quot;:\&quot;support\&quot;}&quot;)<br>            .setWorkflowSid(&quot;WWXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&quot;)<br>            .create(twilioRestClient);</pre><p><strong>Timeout for API-API call</strong></p><p>We use <a href="https://twitter.github.io/finagle/">Finagle</a> for API-API requests within ING. Finagle contains two modules that are responsible for handling timeouts: <strong>Session Timeout module</strong> and <strong>Request Timeout module</strong>.</p><p>Below snippet shows how to configure session timeouts by defining a maximum lifetime and idle time for the client. Configuring these parameters allows you to set a maximum lifetime for the session and a maximum idle time where no requests are being sent from that client.</p><pre>client<br>  .withSession.maxLifeTime(20.seconds)<br>  .withSession.maxIdleTime(10.seconds)<br>  .withSession.acquisitionTimeout(10.seconds)</pre><p>The most basic configuration to define a request timeout on the client side is to use the <em>“withRequestTimeout”</em>method on the client API. The code snippet below, for example, defines a timeout for a request.</p><pre>client.withRequestTimeout(200.milliseconds)<br></pre><ul><li><strong>RETRIES</strong></li></ul><blockquote><em>“If at first you don’t succeed, try, try again”</em></blockquote><p>Retries can help reduce time spent recovering from failures. “Retrying” is especially useful when dealing with intermittent failures. Retries are used in conjugation with timeouts. We give a specific timeout and then we retry after the timeout.</p><p><strong>Retries for Twilio calls</strong></p><p>For retrying Twilio calls made via Twilio SDK, the INGHTTPClient is useful again. The client maintains a retry count per http call and will retry failed calls that match certain criteria.</p><p>We use <a href="https://resilience4j.readme.io/docs">resilience4j</a> and other such libraries to configure retry logic. A naive representation of that logic is represented here</p><pre>def execute(request: HttpUriRequest): Response = execute(request, 0)<br><br>def execute(request: HttpUriRequest, retryCount: Int): Response = {<br>  executor.execute(request).recoverWith {<br>    case exception: Throwable =&gt;<br>      if (canRetry(exception, retryCount)) {<br>        execute(request, retryCount + 1)<br>      }<br>      else throw exception<br>  }<br>}</pre><p><strong>Retries for API-API calls</strong></p><p>If we take the example of Finagle again — By default, a “<em>RequeueFilter</em>” is configured in each client. This filter does retries but does them without any delays, which is not recommended. Backoff policies should be used to configure the behaviour of the filter. Finagle also provides several built-in policies for that purpose. In addition to backoff policies, retry budgets (number of retries) should also be configured when using a “<em>RequeueFilter.</em>”</p><p>This is how you can configure a “<em>RequeueFilter</em>” with a backoff policy and a “<em>RetryBudget.</em>”</p><pre>client    <br>  .withRetryBackoff(Backoff.decorrelatedJittered(2.seconds, 32.seconds))    <br>  .withRetryBudget(RetryBudget(ttl = 5.seconds, minRetriesPerSec = 5, percentCanRetry = 0.1))</pre><ul><li><strong>FALLBACKS</strong></li></ul><blockquote>“Graceful degradation”</blockquote><p>Like all other APIs built in this world, even our APIs will not work as expected sometimes. When there are faults inside our APIs or in those we depend upon, we select alternative mechanisms to respond with a degraded response instead of failing completely.</p><p>The selected alternative mechanism depends on the specific customer or employee journey. Let’s look at how CC 2.0 uses <a href="https://www.twilio.com/docs/serverless/functions-assets/functions#:~:text=Twilio%20Functions%20is%20a%20serverless,that%20scale%20with%20their%20businesses.">Twilio Functions</a> for voice fallbacks.</p><p><strong>Fallback to Twilio Functions</strong></p><p>Callflow API is our voice channel orchestration API. A happy scenario for the API looks as follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ulW2dM49y850IyyI2WhQZA.png" /></figure><ul><li>Incoming phone calls from customers to ING are sent from the PSTN network to Twilio, where the calls are handled by Twilio Programmable Voice.</li><li>Twilio Programmable Voice sends a request to the CC 2.0 Callflow API at ING to start a callflow for the incoming phone call.</li><li>The callflow has an automated voice dialog with the customer to collect information about the customer and the intent of the phone call.</li><li>The Callflow/Voice API creates a task in Twilio TaskRouter to route the customer to an ING employee, based on the information that was collected in the callflow.</li></ul><p>What happens when there are issues with the CallFlow API? The fallback scenario mitigates the situation where starting calls at ING fails. When this occurs, <a href="https://www.twilio.com/docs/runtime/functions">Twilio Functions</a> is used as alternative to route the phone call to an ING employee.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I6VyAWgl1ki8pQQLtbmHKw.png" /></figure><ul><li>When Twilio Programmable Voice receives an error or a timeout on starting a callflow, it will execute the Twilio Function that is configured as fallback. This is also <a href="https://www.twilio.com/docs/voice/twilio-voice-failover-best-practices">recommended</a> by Twilio.</li><li>The Twilio Function has a limited automated voice dialog with the customer to collect basic information for coarse-grained routing.</li><li>The Twilio Function creates a task in Twilio TaskRouter to route the customer to an ING employee, based on the information that was collected in the function.</li><li><strong>CIRCUIT BREAKERS</strong></li></ul><blockquote>“Stop the line”</blockquote><p>We use circuit breakers in our homes to prevent a sudden surge in current. This prevents a potential fire occurring in the house. A circuit breaker trips the circuit and stops the flow of current.</p><p>This same concept is applied to our distributed systems where we stop making calls to downstream services when we know the system is unhealthy and failing and needs time to recover.</p><p><a href="https://twitter.github.io/finagle/guide/Clients.html#failure-accrual">Failure Accrual </a>is one of the three types of circuit breakers provided by <strong>Finagle</strong>, along with Fail Fast and Threshold Failure Detection.</p><p><strong>Failure Accrual</strong></p><p>Similar to Retry Filters, Failure Accrual is configured using a policy that essentially can be success-biased or failure-biased. The Failure Accrual module is enabled by default for all clients with a policy based on five consecutive failures and an equal jittered backoff policy. These examples show both failure-biased and success-biased configurations respectively.</p><pre>client<br>  .configured(FailureAccrualFactory.Param(() =&gt;<br>  FailureAccrualPolicy.consecutiveFailures(<br>    numFailures = 10,<br>    markDeadFor = Backoff.decorrelatedJittered(5.seconds, 20.seconds))))</pre><pre>client<br>  .configured(FailureAccrualFactory.Param(() =&gt;<br>  FailureAccrualPolicy.successRate(<br>    requiredSuccessRate = 0.99,<br>    markDeadFor = Backoff.decorrelatedJittered(5.seconds, 20.seconds))))</pre><ul><li><strong>BULK HEADING</strong></li></ul><p>The name `<strong>Bulkhead</strong>` comes from the sectioned partitions in a ship. If a partition is damaged and/or compromised, only the damaged region fills with water and prevents the whole ship from sinking.</p><p>Similarly, you can prevent failure in one part of your distributed system from affecting and bringing down other parts. The Bulkhead pattern can be applied in multiple ways within a system.</p><p>We use “<strong>Categorized Resource Allocation</strong>” — splitting resources of a system into various buckets, instead of a common pool as a bulkhead mechanism. One resource we apply this pattern on is the “Threadpools” that are supplied to an API or to Twilio SDK.</p><pre>object ExecutionContextConfig {<br>  val unboundedExecutor : ExecutorService = Executors.newFixed..</pre><pre>  val listeningExecutorService: ListeningExecutorService = MoreExecutors.listeningDecorator(unboundedExecutor)</pre><pre>  /**<br>   * Set the thread pool that Twilio will use for all the `async` operations<br>   */<br>  Twilio setExecutorService listeningExecutorService</pre><pre>  /**<br>   * Execution context backed by unbounded threads. To be used for<br>   * blocking , I/O operations. <br>   */<br>  val ioExecutionContext: ExecutionContext = ExecutionContext.fromExecutor(unboundedExecutor)</pre><pre>  /**<br>   * Execution context backed by limited threads. To be used for<br>   * non blocking, CPU bound operations<br>   */<br>  val boundedExecutor: ExecutionContextExecutor = concurrent.ExecutionContext.global<br>}</pre><ul><li><strong>MONITORING AND ALERTING</strong></li></ul><p>The key objective of monitoring is to make sure that the APIs individually and the distributed system as a whole is “observable”. We should be able to look at the system from the outside and identify things that are going wrong . Upon identifying such faults, we should be able to trigger alerts. The following techniques help us achieve that:</p><ol><li>Logs</li><li>Metrics</li><li>Distributed Tracing</li></ol><p>Each CC 2.0 API produces the aforementioned data sets with contextual data. The end goal is to simplify finding <strong>symptoms</strong> and <strong>causes</strong> of faults in CC 2.0 platform.</p><h3>Why do we care more about “Resilience”?</h3><p>Resilience of a system is the measure of its availability. Higher resilience can generate higher availability. Failing to be resilient can affect us in many ways. For CC 2.0 platform, not being resilient means:</p><ol><li>Customers cannot use features like ChatBot or the self service IVR.</li><li>Customers cannot reach out to ING.</li></ol><p>The consequences of all these are unacceptable to us. As a system that sits between our Customers and Agents we must always be ready. Ready to route phone calls to the most suitable agent, ready to reply to chat messages , ready to serve customers on mobile apps, ready to answer your queries via ChatBot and ready for more..</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=442c2e9ca6a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ing-blog/ing-contact-center-2-0-creating-resilient-apis-442c2e9ca6a7">ING Contact Center 2.0 — Creating Resilient APIs</a> was originally published in <a href="https://medium.com/ing-blog">ING Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Journey of Implicits in Scala — part 3]]></title>
            <link>https://ayushm4489.medium.com/journey-of-implicits-in-scala-part-3-963f824ad9ba?source=rss-2839f9309087------2</link>
            <guid isPermaLink="false">https://medium.com/p/963f824ad9ba</guid>
            <category><![CDATA[typeclass]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[implicit]]></category>
            <dc:creator><![CDATA[ayush mittal]]></dc:creator>
            <pubDate>Wed, 16 Mar 2022 23:14:09 GMT</pubDate>
            <atom:updated>2022-03-17T18:46:25.261Z</atom:updated>
            <content:encoded><![CDATA[<h3>Journey of Implicits in Scala — part 3</h3><h3>Type Classes in Scala</h3><p>This is the third part of a <strong>5</strong> <strong>part series</strong> to understand the concept of implicits in Scala . The road taken is as follows</p><ol><li><a href="https://ayushm4489.medium.com/journey-of-implicits-in-scala-737a8db2f490">Introduction to implicit methods or variable definitions and method parameter lists</a>.</li><li><a href="https://ayushm4489.medium.com/journey-of-implicits-in-scala-part-2-18aec19c6bb5">Type Annotations in Scala</a></li><li>Type classes in Scala</li><li>Implicit hell and how painfull it really is?</li><li>Scala 3’s approach to Type Classes.</li></ol><p>In this part we will see how to encode type classes and how type classes can lead to ad-hoc polymorphism.</p><p>Let’s start directly with an example. We look at how we can convert a case class to an HTTP Response.</p><pre>case class User(id: Long, name: String)</pre><pre>object ResponseUtils {<br>  def createdResponse(user: User): HTTPResponse =<br>    HTTPResponse(201,s&quot;<a href="http://localhost:8080/$%7BidProof.getId(t)%7D">http://localhost:8080/${user.id}</a>&quot;)<br>}</pre><pre>ResponseUtils.createdResponse(User(1l,&quot;John Doe&quot;))</pre><p>We define a case class User . We would like to create a HTTP 201 created response from this. We use the ResponseUtils.createdResponse method , which uses the id from User to specify the user resource location.</p><p>Soon we realise that we need a similar method for other entities in our domain. So, we add methods similar to the one already present for User</p><pre>case class Employee(id: Long, name: String)</pre><pre>object ResponseUtils {<br>  def createdResponse(user: User): HTTPResponse =<br>    HTTPResponse(201,s&quot;<a href="http://localhost:8080/$%7BidProof.getId(t)%7D">http://localhost:8080/${user.id}</a>&quot;)<br> <br> def createdResponse(employee: Employee): HTTPResponse =<br>    HTTPResponse(201,s&quot;<a href="http://localhost:8080/$%7BidProof.getId(t)%7D">http://localhost:8080/${</a>employee<a href="http://localhost:8080/$%7BidProof.getId(t)%7D">.id}</a>&quot;)<br>}</pre><pre>ResponseUtils.createdResponse(Employee(1l,&quot;Tom Harry&quot;))</pre><p>For another case class in our domain Employee , we added a new overloaded method createdResponse to create an HTTP response for Employee. This is a class polymorphism with method overloading where we write two or more methods in the same class by using same method name, but the passing parameters is different.</p><p>We have a working solution but something is still amiss. Both the overloaded method’s have almost the same definitions. Both these classes have something in common which is all we need in the createdResponse method. Both of them have an id. Let’s use that common feature of both classes to create a super class/trait and then use the trait in our createdResponse method.</p><pre>trait Id {<br>  def id: Long<br>}</pre><pre>case class User(id: Long, name: String) extends Id</pre><pre>case class Employee(id: Long, name: String) extends Id</pre><pre>object ResponseUtils {<br>  def createdResponse(id: Id): HTTPResponse =<br>    HTTPResponse(201,s&quot;<a href="http://localhost:8080/$%7BidProof.getId(t)%7D">http://localhost:8080/${id.id}</a>&quot;)<br>}</pre><pre>ResponseUtils.createdResponse(User(1l,&quot;John Doe&quot;))<br>ResponseUtils.createdResponse(Employee(1l,&quot;Tom Harry&quot;))</pre><p>Each of our entity class — User, Employee now extend an Id trait. The ResponseUtils.createdResponse can now take a type of Id to create the HttpResponse. Above code can also be written as</p><pre>trait Id {<br>  def id: Long<br>  def createdResponse(id: Id): HTTPResponse =<br>    HTTPResponse(201,s&quot;<a href="http://localhost:8080/$%7BidProof.getId(t)%7D">http://localhost:8080/$id</a>&quot;)<br>}</pre><pre>case class User(id: Long, name: String) extends Id</pre><pre>case class Employee(id: Long, name: String) extends Id</pre><pre>User(1l,&quot;John Doe&quot;).createdResponse<br>Employee(1l,&quot;Tom Harry&quot;).createdResponse</pre><p>With this approach we have moved the desired method inside the parent class and we are directly calling the createdResponse method on the User and Employee instance. This has a nice feeling to it. We can also override the createdResponse method in the sub-classes if required. This is another classic example of polymorphism using method overriding.</p><p>However with this approach we loose a lot of <em>flexibility</em>. Our entity classes have to extend a trait. What if we come across an entity that cannot be changed. If its a class that is outside our control and the source code cannot be changed?</p><p>Let’s look at the problem from another angle. We need to use the id property of the the entity classes. If not use them directly , we need to provide some proof that an id will be available. Let’s encode this proof using a type class pattern. If you remember the discussion in <a href="https://ayushm4489.medium.com/journey-of-implicits-in-scala-part-2-18aec19c6bb5">series 2</a>: Type is a set of things which share same characteristics. Members of a type belong to that set. class or interfaces or traits are ways to create members of that set. The characteristic here is that they all have an id.</p><pre>trait HasId[T] {<br>  def getId(t: T): Long<br>}</pre><p>If we manage to get your hands on an instance of HasId[T] for a given T, you know how to retrieve its id. We can use User , Employee as type parameters to create new types.</p><pre>object UserObject extends HasId[User] {<br>  override def getId(user: User): Long = user.id<br>}</pre><pre>object EmployeeObject extends HasId[Employee] {<br>  override def getId(emp:Employee): Long = emp.id<br>}</pre><pre>object ResponseUtils {<br>  def response[T](t: T, idProof: HasId[T]): HTTPResponse =<br>    HTTPResponse(201,s&quot;<a href="http://localhost:8080/$%7BidProof.getId(t)%7D">http://localhost:8080/${idProof.getId(t)}</a>&quot;)<br>}</pre><p>And now we can use it like this</p><pre>ResponseUtils.response(User(1l,&quot;John&quot;), UserObject)<br>ResponseUtils.response(Employee(1l,&quot;John&quot;), EmployeeObject)</pre><p>When we look at the definition of ResponseUtils.response, it is exactly as polymorphic as we want. It exposes an operation which takes any entity T, as long as there is an implementation of HasId for that entity.</p><p>The HasId[T], which up until now has been referred as the “proof” is the <strong>Type class</strong>.</p><blockquote>It is the trait that abstracts the constraint that is used to deliminate which types would be allowed to be operated on polymorphically.</blockquote><p>The UserObject and EmployeeObject above are referred to as the <strong>Type class instance</strong>. User and Employee above are the <strong>objects</strong> that need to be operated on in a uniform way. And they are linked to the type class instance by being the type parameter in the definitions of the type class instances.</p><p>And last but not the least, we have the ResponseUtils.response definition itself that provides an interface for us to use the code in a polymorphic fashion which is</p><pre>object ResponseUtils {<br>  def response[T](t: T, idProof: HasId[T]): HTTPResponse =<br>    HTTPResponse(201,s&quot;<a href="http://localhost:8080/$%7BidProof.getId(t)%7D">http://localhost:8080/${idProof.getId(t)}</a>&quot;)<br>}</pre><p>So, in summary, the design for encoding the type class pattern consist of the following components:</p><ol><li>A Type class (a trait with type parameters that describes the evidence)</li><li>Type class instances (implementations of the trait for each object required to provide an evidence)</li><li>The interface via which the polymorphic operation is applied</li></ol><p>Above three are the essential part of the type class pattern. There is one more crucial thing that makes this design of the machinery complete in Scala.</p><p>That crucial thing is the implicit feature to make supplying the type class instance a lot more seamless. Basically, we will be making the Scala compiler work for us.</p><h3>Making use of Implicits</h3><p>Looking at the code snippet below:</p><pre>object ResponseUtils {<br>  def response[T](t: T, idProof: HasId[T]): HTTPResponse =<br>    HTTPResponse(201,s&quot;<a href="http://localhost:8080/$%7BidProof.getId(t)%7D">http://localhost:8080/${idProof.getId(t)}</a>&quot;)<br>}</pre><p>In the ResponseUtils.response method, one needs to explicitly pass in the t (Entity) together with its type class instance (HasId [T]).</p><blockquote>But would it not be cool to be able to only specify the entity, and let the <em>HasId</em> be automatically provided for us by the Scala compiler? And if the Scala compiler cannot find the required proof of id (i.e. the required type class instance), then the compilation should fail?</blockquote><p>This is possible by making use of <strong>implicits</strong>. To do this, we slightly modify the definition of ResponseUtils.response. This looks like this:</p><pre>object ResponseUtils {<br>  def response[T](t: T)(implicit idProof: HasId[T]): HTTPResponse =<br>    HTTPResponse(201,s&quot;<a href="http://localhost:8080/$%7BidProof.getId(t)%7D">http://localhost:8080/${idProof.getId(t)}</a>&quot;)<br>}</pre><p>We break the parameter definitions into two, and we tag the last one with the <em>implicit keyword</em>, which would make it a candidate to be automatically provided for us by the Scala compiler in cases where it is omitted.</p><p>The next thing we need to do is to mark our concrete evidence, our type class instances with the implicit keyword. This would make them viable options to be included in the parameter list by the Scala compiler.</p><pre><strong>implicit</strong> object UserObject extends HasId[User] {<br>  override def getId(user: User): Long = user.id<br>}</pre><pre><strong>implicit</strong> object EmployeeObject extends HasId[Employee] {<br>  override def getId(emp:Employee): Long = emp.id<br>}</pre><p>Let’s take a look at these changes together with the rest of the code snippet</p><pre>case class User(id: Long, name: String)<br>case class Employee(id: Long, name: String)</pre><pre>trait HasId[T] {<br>  def getId(t: T): Long<br>}</pre><pre><strong>implicit</strong> object UserObject extends HasId[User] {<br>  override def getId(user: User): Long = user.id<br>}</pre><pre><strong>implicit</strong> object EmployeeObject extends HasId[Employee] {<br>  override def getId(emp:Employee): Long = emp.id<br>}</pre><pre>object ResponseUtils {<br>  def response[T](t: T)(implicit idProof: HasId[T]): HTTPResponse =<br>    HTTPResponse(201,s&quot;<a href="http://localhost:8080/$%7BidProof.getId(t)%7D">http://localhost:8080/${idProof.getId(t)}</a>&quot;)<br>}</pre><pre>ResponseUtils.response(User(1l,&quot;John&quot;))<br>ResponseUtils.response(Employee(1l,&quot;John&quot;))</pre><p>In the above example, the missing parameter list is automatically provided because they are in the local scope. As usual, the Scala compiler would scan the implicit scope for values that are suitable.</p><p>In cases where we pass in an Entity that does not possess an id proof, aka, a entity without an instance of HasId[T] in implicit scope, then the compilation would fail. For example something like this:</p><pre>ResponseUtils.createdResponse(SomeOtherEntity(1l,&quot;John Doe&quot;)) <br>// fails at compile time</pre><p>The last thing we are going to touch on is how <em>Context bound</em> and <em>implicitly function</em>, two other language features of Scala that are usually applied when putting type class pattern together.</p><h3>Using Context Bound Annotation and the implicitly function</h3><p>If you take a look at the definition of the response method, and find a way to make it less verbose. That is:</p><pre>...<br>def response[T](t: T)(implicit idProof: HasId[T]): HTTPResponse<br>...</pre><p>A context bound annotation like [T: Bound] which allows us to specify that for any type T there must be an implicit value of Bound[T].</p><p>Which in essence, is what we are doing. We want that for any entity T there must be an instance of HasId[T].</p><p>You can go check out the <a href="https://ayushm4489.medium.com/journey-of-implicits-in-scala-part-2-18aec19c6bb5">2nd post of this series</a> for an overview of context bound annotation.</p><p>Modifying the code, to include usage of context bound would leave us at:</p><pre>object ResponseUtils {<br>  def response<strong>[T: HasId]</strong>(entity: T): HTTPResponse = ???<br>}</pre><p>As we can see, since we have removed the (implicit idProof: HasId[T]) portion of the method definition, we would no longer have access to getId in the method body.</p><p>But this is where the implicitly function comes to our rescue. The implicitly function allows us to reach into the implicit scope and get hold of an implicit value of interest.</p><p>In this case, we need to grab a hold of an implicit value of type HasId[T], which was what the idProof variable was. but since it is now gone, we can replace it with: implicitly[HasId[T]].getId(t)</p><p>Taking together the updated code snippet would look like:</p><pre>case class User(id: Long, name: String)<br>case class Employee(id: Long, name: String)</pre><pre>trait HasId[T] {<br>  def getId(t: T): Long<br>}</pre><pre><strong>implicit</strong> object UserObject extends HasId[User] {<br>  override def getId(user: User): Long = user.id<br>}</pre><pre><strong>implicit</strong> object EmployeeObject extends HasId[Employee] {<br>  override def getId(emp:Employee): Long = emp.id<br>}</pre><pre>object ResponseUtils {<br>  def response<strong>[T: HasId]</strong>(entity: T): HTTPResponse = {<br>    val id = implicitly[HasId[T]].getId(t)<br>    HTTPResponse(201,s&quot;<a href="http://localhost:8080/$%7BidProof.getId(t)%7D">http://localhost:8080/$i</a>d&quot;)<br>  }<br>}</pre><pre>ResponseUtils.response(User(1l,&quot;John&quot;))<br>ResponseUtils.response(Employee(1l,&quot;John&quot;))</pre><p>And with that, we are able to make use of context bound annotation ( together with the help of the implicitly function) to make the type class encoding a little less verbose.</p><p>In the next post, we will see some issues that might be faced by developers when using libraries that are based on this type class pattern.</p><p>Further readings</p><ul><li><a href="https://www.baeldung.com/scala/type-classes">https://www.baeldung.com/scala/type-classes</a></li><li><a href="https://www.geekabyte.io/2017/12/encoding-type-class-in-scala.html">https://www.geekabyte.io/2017/12/encoding-type-class-in-scala.html</a></li><li><a href="https://nrinaudo.github.io/scala-best-practices/definitions/type_class.html">https://nrinaudo.github.io/scala-best-practices/definitions/type_class.html</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=963f824ad9ba" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>