Every AI workflow that touches a system of record has the same shape: something unreliable sits next to something that can’t be wrong. In my case, the system of record was a payments platform, and the unreliable thing changed over time. First it was third-party vendors (Plaid and Giact), which are rate-limited. Later it was the model itself, calling tools I had exposed over MCP.

I worked on both: an event-driven verification pipeline on a platform that moved over $1.1B in transaction volume, and later MCP tool servers for autonomous workflows. The fixes turned out to be one idea applied at different layers. Put a boundary between the unreliable thing and the system that has to be right.

This post follows a single request through that architecture, shows what broke at each stage in v1, and ends with where I draw the line on using LLMs at all.

The shape of the system

A request crosses four boundaries. Each one exists because something specific broke without it:

1. Buffer. SQS decouples intake from vendor latency.
2. Protect. AWS KMS envelope encryption keeps plaintext PII out of storage and logs.
3. Validate. Pydantic schemas and deterministic normalization sit at every tool edge.
4. Recover. Retries and Sentry Alerts mean no failure is silent.

1. Buffer: don’t let a vendor set your latency

My v1 verification flow called vendors synchronously inside the request cycle. When a vendor hit a latency spike or returned a 504, the caller hung, and some transactions were blocked.

An agent as the caller makes this worse. A 15- to 30-second vendor stall times out the orchestrator turn, the agent retries, and now you are hammering a rate-limited endpoint and collecting 429s on top of the 504s.

The fix was to stop doing vendor work on the request path. API Gateway accepts the request and writes it straight to SQS. Lambda workers pick it up and absorb the vendor latency on their own time. Bursts queue up instead of running into vendor rate limits, and the caller never waits on a third party.

2. Protect: no plaintext past the front door

Identity and account records moving through an orchestration pipeline are a liability. Anywhere they pass, plaintext PII can end up in logs and state stores.

So sensitive fields are encrypted with AWS KMS envelope encryption before anything is persisted to DynamoDB. Everything downstream works with encrypted tokens, which is how the pipeline kept a record of zero plaintext PII leaks.

3. Validate: strict where it matters, forgiving where it should be

Validation failed me in two opposite directions.

Too loose. When an LLM calls a backend tool over MCP, it can send malformed JSON or invent an enum value, and either can crash a downstream service. At the MCP boundary (FastAPI, Pydantic, Cloud Run), every payload is validated before anything executes. A bad call gets a structured validation error back, so the model can correct itself instead of corrupting state.

Too rigid. In the identity pipeline, exact string matching rejected legitimate users with compound surnames, hyphens, and diacritics. Each false decline became a ticket for a human CX agent. The fix was a deterministic normalization layer that runs before vendor dispatch.

These are the same boundary doing two jobs. Here is the compact version:

import unicodedata
from typing import Literal
from pydantic import BaseModel, Field, field_validator

class VerificationToolPayload(BaseModel):
    """Strict MCP tool boundary: rejects invented args, normalizes names."""
    account_id: str = Field(..., pattern=r"^ACC-[0-9]{6,12}$")
    kms_encrypted_pii_token: str = Field(..., min_length=32)
    applicant_surname: str = Field(..., min_length=1, max_length=120)
    workflow_action: Literal["VERIFY_IDENTITY", "ESCALATE_TO_CX"]

    @field_validator("applicant_surname")
    @classmethod
    def normalize_surname(cls, raw: str) -> str:
        # Strip diacritics, split compound surnames, uppercase.
        decomposed = unicodedata.normalize("NFKD", raw)
        stripped = "".join(c for c in decomposed if not unicodedata.combining(c))
        return " ".join(stripped.upper().replace("-", " ").split())

Two properties matter here. Anything outside the contract is rejected, not repaired. And normalization is a pure function, so the same input always produces the same output, which keeps every decision auditable.

4. Recover: make failure visible and replayable

Workers retry failed vendor calls. Persistent failures route to Sentry alerts and an internal investigation tool for CX agents.

Nothing is lost, because every failed payload is preserved for replay. Nothing is silent either, because each failure surfaces with a structured reason. Support no longer has to guess what happened to a transaction that timed out mid-flight.

Across production, this architecture covered 92% of active accounts, reduced fraud by 20%, and kept zero plaintext PII leaks. I handed it off with Sentry runbooks and CX training guides.

Where I use LLMs, and where I don’t

The boundaries above are what make it safe to use LLMs anywhere near this system. The models live outside the boundary, never inside it.

Where they earn their place:

• Summarizing unstructured context.
• Routing between fast, lightweight models and high-reasoning ones such as Claude Sonnet or Gemini Flash.
• Auditing quality with a 20% sample-rate LLM-as-a-judge pipeline, traced in Langfuse and run asynchronously to track accuracy and token ROI over time.

Where I rejected them entirely: identity verification, sanctions screening, and PII handling. Those need execution that is strict, auditable, and incapable of hallucinating. An LLM can orchestrate the outer workflow over MCP, but the execution boundary has to stay 100% deterministic.

Takeaways

• Never tie a user-facing or agent-facing turn to a vendor’s latency. Buffer it.
• Validate at every tool edge. Reject what’s invalid, and normalize what’s merely messy.
• Models can propose actions. Deterministic code decides whether they run.

What is your take?

Many companies want their engineers to move faster with AI.

The challenge is making that possible without creating thousands of different security, data protection, and risk-control implementations.

One team integrates a model API. Another builds a RAG pipeline over internal documents. A third adds an AI coding assistant.

Each team makes its own decisions about authentication, logging, PII handling, model selection, and evaluation.

That works for a handful of experiments. It becomes much harder to manage when AI adoption spreads across an enterprise.

The problem isn’t a lack of promising use cases. It’s getting those use cases into production consistently, with controls that scale alongside them.

AWS explored this challenge in its article, Building an AI Stack for Banking on AWS. The core message is that moving from proof of concept to production requires more than choosing a model. It requires reusable technology, governance, evaluation, monitoring, and an operating model that supports delivery at scale.

My proposal follows that direction:

Build a shared AI platform that makes the secure, governed path the easiest path for engineers to use.

The goal isn’t to centralize every AI application or force every team to use the same model.

It’s to centralize the controls that should be consistent across the organization while letting teams build and deploy their own solutions.

1. Start With a Shared AI Gateway

The first component I’d build is a gateway that provides a consistent interface to approved models and inference services.

Instead of every application integrating directly with a model provider, applications call the gateway.

The gateway handles common platform responsibilities:

• Authentication and authorization
• Model routing and approved-provider policies
• Usage limits, quotas, and cost attribution
• Consistent request and response handling
• Centralized telemetry and policy-decision records
• Enforcement of applicable input and output controls

This isn’t a new architectural idea. Organizations already use centralized AI gateways in production to simplify access to multiple models, standardize integrations, and manage usage.

AWS also identifies gateways, model evaluation, guardrails, and model monitoring as foundational components of an enterprise AI stack.

What I’d propose is making the gateway the standard entry point for internal AI applications, with documented exceptions for workloads that need a different integration pattern.

The implementation could use an existing gateway product or a custom service. LangChain could support application orchestration, while MCP could standardize how applications interact with tools and resources.

Neither should be treated as a substitute for the gateway’s security and policy-enforcement layer.

The important decision is the architecture, not the framework.

2. Separate Inference From Risk Controls

I’d avoid making the model responsible for deciding whether its own output is safe to use.

Instead, I’d separate four responsibilities:

Inference: Generates the model response.

Guardrails and policy enforcement: Applies the relevant input and output rules.

Evaluation: Measures quality, safety, and task-specific performance.

Monitoring: Detects changes in production behavior and operational health.

These components should work together without being tightly coupled to a particular model.

Imagine an engineering team changing its code-review assistant from one model to another.

The gateway can route requests to the new model, but the platform should still run regression tests, compare performance, and verify that existing controls continue to work.

A model change shouldn’t automatically mean rebuilding every application.

It also shouldn’t mean assuming previously approved behavior will remain unchanged.

I’d version policies, evaluation suites, prompts, and model configurations independently wherever practical. Changes that affect risk should trigger the appropriate review and approval process.

This is the balance I’m aiming for: reusable controls, faster model upgrades, and explicit validation when something material changes.

3. Validate Requests Before They Reach a Model

A shared gateway gives the organization a consistent place to enforce common rules.

Consider an internal application that summarizes a customer’s dispute history.

I’d design the request path roughly as follows.

Step 1: Authenticate the caller.

Establish which application and user are making the request, and whether they are authorized to access the relevant information.

Step 2: Validate the request.

Check required fields, types, sizes, and allowed operations against a defined schema.

Step 3: Apply data controls.

Detect sensitive information, enforce data-handling rules, and redact or block information where required by policy.

Step 4: Check the intended use.

Apply the permissions and restrictions associated with the application, model, and data classification.

Step 5: Route to an approved model.

Send the request only to an endpoint authorized for that workload.

Step 6: Validate the response.

Check output structure, apply relevant safety and data-loss controls, and reject responses that don’t meet the application’s requirements.

Step 7: Record the decision.

Capture the request ID, model and configuration versions, applicable policy versions, evaluation signals, and relevant outcomes.

The exact checks depend on the use case. A coding assistant, an internal knowledge search, and an application that processes customer records won’t have identical requirements.

Schema validation doesn’t prove that a request is safe. PII detection can miss information, and a structurally valid response can still be incorrect or disclose sensitive data.

The controls need to be layered and tested against realistic failure scenarios.

I’d also be careful about what gets logged.

A centralized audit trail is valuable, but retaining every prompt and response can create another sensitive-data store. Access controls, encryption, retention policies, redaction, and tamper-evident records should be designed into the logging system.

The objective is to make enforcement consistent and evidence easier to retrieve, not to claim that one gateway eliminates risk.

4. Make Retrieval Respect Data Governance

RAG introduces another failure mode: an application can generate a plausible answer using the wrong document.

Imagine a lending-policy assistant that retrieves a superseded policy because it has a high similarity score to the user’s question.

The answer might be well written and still be wrong.

I’d address this at the retrieval layer rather than relying on the model to recognize outdated information.

For documents stored in Amazon S3, I’d maintain metadata such as:

• Document version and effective date
• Current or retired status
• Owning department
• Data classification
• Applicable business unit or jurisdiction
• Access-control attributes

The retrieval pipeline should apply authorization and document-status filters before ranking eligible documents by similarity.

A retired policy shouldn’t be eligible for a query asking for the currently applicable policy. A user shouldn’t be able to retrieve a restricted document simply because its contents match the question.

I’d also track document ingestion, metadata quality, index freshness, and changes to the source of truth.

Metadata filters are only as reliable as the information they depend on.

For auditability, the application should retain enough provenance to establish which document versions informed a response, subject to the organization’s data-retention requirements.

This is a reusable platform capability. Teams building internal search, policy assistants, and document summarization shouldn’t each have to reinvent it.

5. Evaluate Before Deployment, Not Just After Something Breaks

A gateway helps standardize access, but it doesn’t establish that an AI application is fit for production.

I’d make evaluation a standard part of the deployment process.

Each application should have tests appropriate to its intended use, including:

• Task accuracy and answer quality
• Retrieval relevance and source attribution
• PII handling and unauthorized disclosure
• Prompt-injection and jailbreak resistance
• Output-schema compliance
• Latency and cost under expected workloads
• Performance against known failure cases

Some checks can be automated with conventional tests or model-based evaluators. Others may require human review, particularly when the application affects customers, material business decisions, or regulated processes.

I’d establish minimum acceptance criteria before deployment, then run the relevant tests whenever a model, prompt, retrieval configuration, or policy changes.

The same evaluation framework should be available to developers during experimentation and in the release pipeline.

This creates a repeatable process.

Teams can compare approaches, identify regressions, and demonstrate why a configuration was approved, rather than relying on a successful demo as evidence of production readiness.

6. Treat Silent Failures as a First-Class Operational Risk

Traditional infrastructure monitoring tells us whether a service is available.

AI applications need additional signals to tell us whether the service is still doing the right thing.

A model can return HTTP 200 responses while producing worse answers. A retrieval index can become stale without causing an outage. A prompt change can increase unsupported claims while latency remains unchanged.

I’d monitor four categories.

Quality and safety

Track evaluation results, failure rates, source-grounding signals, and sampled outputs against known-good cases.

Define thresholds that trigger investigation or a rollback.

Retrieval and data freshness

Monitor ingestion failures, index lag, document versions, and the freshness of the sources being retrieved.

Policy and schema behavior

Track blocked requests, validation failures, and changes in policy-trigger rates.

A sudden increase or an unexpected drop can indicate a regression or a change in traffic.

Model and configuration drift

Record which model, prompt, retrieval configuration, and policy versions are active.

Compare production changes with the configurations that passed approval.

These signals should feed operational dashboards, alerts, incident response, and periodic reviews.

I’d also define what happens when a check fails.

Depending on the application, the response might be to reject a request, fall back to an approved model, require human review, or disable a feature until the problem is resolved.

The right response depends on the business risk. The important thing is that the behavior is designed in advance rather than improvised during an incident.

7. Put Governance Around the Platform and Its Applications

Technical controls are only part of the problem.

A bank also needs to know which AI applications are in production, what they are used for, who owns them, and which risks and approvals apply.

I’d establish a central registry for production AI use cases and their associated models and configurations.

For each application, I’d want to capture:

• Business purpose and accountable owner
• Data sources and data classifications
• Model and configuration versions
• Risk classification and required controls
• Evaluation results and approval history
• Deployment status and monitoring arrangements
• Relevant dependencies and change history

I’d pair that registry with a defined approval process.

Not every internal assistant needs the same review as a customer-facing application or a model used in a material credit decision.

The process should reflect the use case, applicable regulatory obligations, and the organization’s risk framework.

For a bank, this may involve model risk management, information security, privacy, compliance, legal, and business stakeholders.

Requirements associated with OCC supervision, SEC rules, recordkeeping, and other obligations must be assessed according to the activities and entities involved.

A gateway and an audit trail can help demonstrate that controls exist and operate as intended.

They don’t, by themselves, establish regulatory compliance.

I’d treat governance as part of the deployment lifecycle rather than a separate process that begins after engineering is finished.

8. Make the Platform a Product, Not a Ticket Queue

A centralized platform can become a bottleneck if every model, prompt, and application change requires a manual request to a small central team.

I’d avoid that by providing developers with reusable building blocks:

• Standard SDKs and APIs
• Approved model configurations
• Reference implementations for common use cases
• Reusable RAG and tool-integration patterns
• Evaluation templates and CI/CD integration
• Self-service usage reporting and cost attribution
• Clear documentation and escalation paths

The central team would own shared infrastructure, platform-level controls, and enterprise standards.

Application teams would own their use cases, application-specific evaluations, and day-to-day behavior within those standards.

A federated model is likely to work better than insisting that one team build every AI application. Larger business units can retain specialized capabilities while following common platform requirements.

I’d start with a small number of valuable use cases, establish the minimum reusable components needed to support them, and improve the platform as practical experience exposes gaps.

That is more useful than spending months trying to design every possible control before delivering anything.

9. Measure the Platform by More Than Model Usage

The number of model calls or onboarded engineers doesn’t tell us whether the investment is delivering value.

I’d track:

• Time required to move a use case from prototype to production
• Reuse of platform components across teams
• Evaluation pass rates and production regressions
• Security and policy incidents
• Model costs by team and use case
• Latency and reliability
• Business outcomes for the applications being deployed

Cost attribution is particularly important at scale.

Engineers need enough visibility to compare models and understand the cost of their workloads, while platform owners need the information to manage capacity, negotiate provider arrangements, and identify waste.

The aim is to make speed, reliability, and cost visible together.

What I’d Propose and What Production Experience Already Tells Us

The individual building blocks in this design are not speculative.

Centralized model gateways, reusable AI components, evaluation, guardrails, monitoring, and governed deployment pipelines are established enterprise AI patterns.

Publicly documented implementations also show organizations using gateways to simplify access to multiple models and reduce duplicated integration work.

AWS’s banking guidance makes a similar case for reusable foundational components, evaluation, guardrails, monitoring, model registries, and a production lifecycle built around governance and control.

What I’m proposing is to combine those patterns into a coherent engineering platform for a regulated environment.

I would not claim that a single gateway guarantees compliance, prevents every disclosure, or removes the need for model validation. Nor would I assume that every bank needs the same organizational structure or the same set of controls.

The design should be adapted to the organization’s risk profile, existing architecture, regulatory obligations, and maturity.

But the direction is clear: establish common foundations, make them easy to consume, and build evaluation, governance, and monitoring into the production lifecycle.

The banks that scale AI effectively won’t necessarily be the ones with the most restrictive approval processes.

They’ll be the ones that make it possible for engineers to build quickly while giving risk, security, and compliance teams the evidence and control they need.

That’s the platform I’d propose: centralized where consistency matters, federated where teams need flexibility, and governed throughout the lifecycle.

If you’re building an AI platform in a regulated environment, I’d be interested in comparing approaches, especially around balancing developer self-service with model governance, data protection, and production reliability.

Further reading: Building an AI Stack for Banking on AWS

Similarity search can’t tell an outdated guideline from the current one. Both look relevant. Metadata is what tells them apart.

Let’s look at an example: In a clinical agent, a plausible answer from the wrong protocol version is worse than no answer.

The problem

Clinical knowledge can be versioned, specialty-specific, and sometimes contradictory. Patient data changes by the hour. If you dump both into a vector store and hope embeddings sort it out, you’ll get confident answers on stale evidence.

What I learned about S3 metadata

I picked this up in the AWS Gen AI learning plan, and it changed how I think about ingestion. Your S3 bucket is the foundation for any knowledge base or anything feeding a foundation model, and its structure shapes retrieval quality before a single embedding is created. Sure, you can save files without metadata, but you need the metadata.

System-generated metadata: S3 records content type, size, last-modified date, and storage class automatically. It’s free, and it’s useful for freshness checks and auditing.

User-generated metadata: This is where the value is. Custom key-value pairs describe what the document is: protocol_version, effective_date, specialty, status (active or superseded), source_authority. Object tags add another layer for access control and lifecycle rules.

Tactics that improve retrieval

• Hierarchical prefixes. Organize by specialty, then protocol, then version. Structure carries meaning before any query runs.
• Sidecar metadata for knowledge bases. Amazon Bedrock Knowledge Bases can filter on metadata attributes you attach to each document, so you narrow the candidate set before similarity search.
• Filter first, then search. Restrict to active protocols for the right specialty, then rank by relevance. This fixes the wrong-version problem.
• Enrich at ingestion. Extract metadata with rules or an LLM when documents arrive, and validate it. Bad metadata is worse than none.
• Retire, don’t delete. Mark superseded protocols as inactive so they stay auditable but never get retrieved.
• Query your metadata. S3’s metadata tooling lets you inspect what you have, which surfaces gaps like documents with no effective date.

Keep live patient data out of the index

Patient context shouldn’t be embedded. It goes stale and creates compliance exposure. Fetch it through tool calls at query time and combine it with retrieved protocol content when reasoning. The index holds knowledge, and the tools hold state.

Where I’m coming from

I haven’t shipped a clinical agent, and I won’t pretend otherwise. What I have built is close to this pattern. At Amivero, I designed a serverless RAG platform on Bedrock with Titan embeddings and human-in-the-loop evaluation, which cut document review cycles by 88%. I also built an OCR pipeline with state-aware validation and metadata enrichment, which is the same idea as specialty- and version-aware retrieval.

How I’d approach it

1. Define the metadata schema with clinicians before ingesting anything.
2. Filter on version, date, and specialty before similarity search.
3. Require citations, and have the agent abstain when retrieval confidence is low.
4. Evaluate retrieval separately from generation, so you know which one failed.

Better prompts won’t fix bad retrieval. Better data structure will, and that starts in the bucket.

The bug wasn’t the model. Some “AI eval” content stops at unit tests for prompts. Mine started somewhere messier: a fitness coaching agent that occasionally counted my push-ups wrong, and a career-tracking tool that sometimes forgot the one number it was supposed to report.

Across two projects, Chaos Fit, a real-time AI workout coach, and Global Impact Tracker, an MCP server that turns engineering work into performance-review narratives, I built eval systems that do one thing well: separate “the model got creative” from “the code has a bug.”

That distinction matters more than it sounds. In Chaos Fit, some failures were genuinely non-deterministic. The vision model flagging a camera adjustment as a rep, or the coaching agent repeating the same cue three times instead of noticing a new error. Those get caught with dataset-driven evals: 27 cases pulled straight from real session traces (evals/dataset.json), scored against rules in evals/evaluators.py like a zero-tolerance threshold for phantom reps (eval_rep_count_accuracy) and a ban on generic praise (eval_correction_specificity) — "good job" doesn't count as coaching.

Other failures like interruption counts silently merging with correction counts (documented as F-1 in the project’s failure_taxonomy.md) were deterministic state-management bugs in session_manager.py. No amount of re-prompting fixes those; they needed an invariant check in the trace harness and a code fix in session state, not an eval.

The same philosophy, a different failure surface. Global Impact Tracker generates LLM narratives. Before a story ships, a deterministic check (_missing_metrics, in server.py) confirms the exact metrics, hours saved, latency reduction, project count, actually appear in the text. If they don't, the model gets a targeted critique and regenerates. A separate LLM-as-judge (evaluate_star_story, in server_utils.py) scores narrative quality on a 1–5 rubric, and because judges are non-deterministic too, _parse_judge_response clamps its output range and falls back to a local model if the primary judge fails or rate-limits. This only runs on 20% of responses for performance.

Trace first, eval second. As the project spec puts it: “no eval is written before trace data exists to justify it; observed failures define the dataset.” That’s produced a small, concrete taxonomy of reproducible issues instead of a vague “make it better” backlog — and it’s let me build regression gates that catch the same bug twice.

If you’re curious about evaluation methodology, agent tracing, or reliability engineering for LLM systems, let’s chat!

TL;DR: Default-routing every single user prompt to flagship frontier models is one of the fastest ways to burn your API credits. In this post, I walk through how I built an enterprise-ready GenAI Model Router using Python (FastAPI), Next.js (TypeScript), Google Gemini API, and Hugging Face. We’ll look at static length routing, exponential backoff with jitter, automatic failover, and honest production trade-offs.

— -

The $10,000 “Hello World” Problem

If you’ve built LLM apps in production, you know the drill: your team picks a top-tier frontier model like Claude Sonnet 5 or OpenAI GPT-6 Astra wires up an API key, and calls it a day.

It works great, until the bill arrives.

When you analyze user traffic logs, a staggering percentage of real-world queries look like this:

- *”Hi”*

- *”Thanks!”*

- *”What is 2 + 2?”*

- *”Summarize this sentence.”*

Routing these light 5-word queries to a heavy flagship model costs the exact same base overhead as sending a 5,000-word legal document. Multiply that across thousands of active users, and you’re essentially lighting money on fire.

The solution? Model Routing.

By dynamically or statically routing queries to smaller, open-source models (like meta-llama/Llama-3.3-70B-Instruct on Hugging Face) for simple tasks, and reserving premier models (Claude Sonnet 5) for complex reasoning, you can reduce LLM infrastructure costs while preserving response quality.

— -

The Stack & System Architecture I Built

Here is the architecture I designed to solve this:

The Tech Stack:

• Backend: Python 3.11+, FastAPI, Pydantic v2, Tenacity (for retries), Pytest.
• Frontend: Next.js (TypeScript) styled with Google’s clean four-color light theme (`#4285F4`, `#EA4335`, `#FBBC05`, `#34A853`).

Models:

• Primary Flagship: google/gemini-2.5-flash
• Open-source Fast Provider: meta-llama/Llama-3.3-70B-Instruct (via Hugging Face Serverless Inference API)
• Deployment: Dockerized and ready for Google Cloud Run.

— -

Static Routing Rules: Simple, Deterministic & Fast

While dynamic LLM-as-judge routers are popular, they add latency and extra token costs to decide where to send a prompt. For V1, I implemented a zero-overhead Static Routing Rule:

def inspect_route(prompt: str) -> RouteInspection:
  cleaned = prompt.strip()
  word_count = len(cleaned.split())
  char_count = len(cleaned)
  # Route to Hugging Face if prompt is short
  if word_count < 10 or char_count < 10:
  return RouteInspection(
  primary_route="huggingface",
  model_name="meta-llama/Llama-3.3-70B-Instruct",
  reason=f"Short query: {word_count} words < 10"
)

Route to Google Gemini API if prompt requires heavier reasoning

return RouteInspection(
primary_route="google",
model_name="google/gemini-2.5-flash",
reason=f"Detailed query: {word_count} words >= 10"
)

If a prompt has < 10 words or < 10 characters, it routes to Hugging Face (meta-llama/Llama-3.3-70B-Instruct). If it contains 10 words and characters, it routes to Google Gemini (google/gemini-2.5-flash).

— -

Resiliency: Why You Need Exponential Backoff with Jitter

In production, APIs fail. Cloud providers return 429 Too Many Requests during traffic spikes, or 503 Service Unavailable when models load.

If your code retries immediately on a fixed 1-second timer, 10,000 concurrent users will hit the provider simultaneously on retry #1, causing a thundering herd problem that permanently locks up your backend.

To prevent this, I wrapped all LLM invocations with Exponential Backoff and Random Jitter using Python’s tenacity library:

@retry(
wait=wait_random_exponential(min=0.5, max=4.0), # ←- Adds randomized jitter
stop=stop_after_attempt(3),
retry=retry_if_exception_type(GoogleServiceError),
reraise=True
)
def generate_with_retry(prompt: str):
return call_google_gemini_api(prompt)

Automatic Provider Failover

If Google API still fails after 3 retry attempts (or if rate limits spike), the router’s Circuit Breaker automatically redirects the request to Hugging Face:

try:
return self.google_service.generate(prompt)
except GoogleServiceError as g_err:
logger.warning("Google API failed after retries. Failing over to Hugging Face…")
# Dynamic Failover Execution
return self.hf_service.generate(prompt)

If both providers fail, the router catches the exception and returns a Graceful Degradation response to the frontend with detailed diagnostic metadata, ensuring the client UI never crashes hard.

— -

Honest Production Caveats: What I Left Out (For Now)

This project is a functional, tested model router, but here is what is not in V1:

1. No Validation Pipeline: There is no semantic input validation pipeline checking for prompt injections or toxic inputs before routing.

2. No Redis Rate-Limiting: Retry counters and circuit breaker flags are evaluated per-request rather than tracked in a distributed Redis cache.

3. Static vs. Dynamic Thresholds: A 9-word prompt might sometimes require heavy logic, while a 20-word prompt might be boilerplate. Moving to embedding-based routing or classifier-based routing is the natural V2 upgrade.

— -

The Developer & User Experience

I built a custom Next.js UI styled in Google’s signature four-color light theme. It gives real-time visual feedback:

• Live Route Estimator: As you type, a dynamic pill badge instantly tells you whether your query will hit Hugging Face or Google API.
• Failover Simulation Suite: Checkboxes allow developers to artificially simulate Google API or HF outages to test failover behavior live.
• Metrics Dashboard: Displays round-trip latency in milliseconds, provider used, backoff retries performed, and token count.

— -

Try It Yourself

All code for this project, including the FastAPI backend, Pytest suite, Next.js dashboard, Dockerfiles, and Google Cloud Run deployment script,is open source on GitHub:

🔗 GitHub Repository: [https://github.com/ElishebaW/model-router](https://github.com/ElishebaW/model-router)

Final Thoughts

Building with AI isn’t just about prompt engineering; it’s about systems engineering. Implementing smart routing, retry backoff jitter, and graceful failovers ensures your application stays fast, resilient, and financially sustainable as you scale.

What routing strategies are you using in your LLM stack? Drop a comment below or connect with me on GitHub!

Lately, I’ve been working on Phase 1 of my optimization plan for Chaos Fit, a real-time AI workout coach built on top of Gemini Live and bidirectional audio streaming. My immediate goal? Coaching Accuracy. When an AI coach is tracking your workout in real time, it cannot guess. If a user does 10 push-ups, the model can’t report 9 or 11. It can’t mistake a minor adjustment for a full rep. It can’t give generic feedback like “Great job, keep going!” when their lower back is sagging during a plank.

Here is exactly how I’m moving Chaos Fit from brittle prompts to production-grade accuracy, broken down into four engineering phases.

Phase 1: Building a Failure Taxonomy from Real Traces

You can’t fix what you aren’t measuring. I started by pulling recent session traces from Langfuse, focusing specifically on exercise_detection session_summary_generation observations.

Instead of guessing why the coach missed a rep, I had Claude Code categorize real-world failures into a concrete taxonomy (failure_taxonomy.md):

• F-1 through F-4: Tracking wrong rep counts, double-counted reps, and false positives.
• F-5: Tracking rep-count false positives caused by incidental movement.

With this taxonomy established, Claude Code built a local evaluation dataset (evals/dataset.json) featuring 27 distinct edge cases spanning rep counts, correction specificity, interruption integrity, and session summaries.

Phase 2: Tightening the Prompts and Engineering Evaluators

With a solid baseline evaluation dataset, I updated our coach-system-instruction and coach-system-instruction-native-audio prompts. The new guidance explicitly mandates full Range of Motion (ROM) for a rep to count and strictly excludes incidental movement.

But a prompt change means nothing without automated verification. Claude wrote three custom evaluators:

1. eval_rep_count_accuracy: Enforces a strict tolerance, with an absolute zero-tolerance policy when the expected rep count is zero.
2. eval_correction_specificity: Eliminates generic fluff.
3. eval_interruption_integrity: Ensures the live audio stream handles conversational interruptions flawlessly.

The result? 27/27 evaluations are now passing cleanly in local testing. Next up is a fast-follow update to the form feedback engine, mandating a strict [body part] + [corrective action] pattern (e.g., "Drop your hips" instead of "Fix your form").

Phase 3: Advanced Pose Estimation and UI Overlays

While Gemini handles the core contextual reasoning, true real-time coaching requires visual validation. The next step of the roadmap introduces deep joint-angle guidance for three primary movements:

• Squats: Tracking hip and knee depth.
• Push-ups: Monitoring elbow angles and spinal alignment.
• Planks: Measuring hip height and shoulder stack.

I’m currently wiring this structured pose data directly into our frontend canvas layer (app.js). By parsing structured output from the model turns and translating them into pixel coordinates, the app will render dynamic canvas overlays, drawing real-time joint markers and alignment lines directly over the user’s camera feed.

Phase 4: Locking it Down with a CI Gate

Great engineering systems are self-defending. The final phase of this sprint is integrating these coaching accuracy evaluations directly into our GitHub Actions workflow (.github/workflows/ci.yml).

By establishing a strict CI gate, we ensure that future prompt updates, infrastructure changes, or model upgrades can never merge unless they pass our 27-case accuracy baseline.

The Takeaway

Building agentic AI systems that interact with the physical world requires moving past conversational novelty. It requires regression testing, deterministic evaluations, and structured feedback loops.

If you’re building in the real-time AI or multimodal space, how are you moving past the “vibe check” phase? Let me know in the comments.

Want to catch up on how this project started? Read the original architecture breakdown: Chaos Fit: Building a Real-Time AI Workout Coach with Gemini Live

The industry finally has a measurement conversation. Teams are watching token spend, AI code acceptance rates, and PR throughput. That’s real progress. But from my seat as an engineer building AI systems, I can see the gap those metrics still leave, and it’s where the actual ROI lives.

Something has shifted in the last year. Engineering managers who couldn’t tell you how many tokens their team consumed in January are now getting monthly invoices that make the number impossible to ignore. Anthropic reports the average Claude Code usage runs around $6 per developer per day, with 90% of users spending under $12 daily , and that’s before multi-agent workflows enter the picture. Token spend is real, it’s growing, and leadership is paying attention.

So the measurement conversation has started. But I want to make a case that it’s started in the wrong place, and that the metrics most teams are watching right now answer a narrower question than the one that actually matters to the business.

I’m an engineer, not a CTO. I’ve spent my AI career building AI systems, RAG pipelines, OCR ingestion workflows, and event-driven architectures at federal scale. I got frustrated that I couldn’t answer basic questions about my own AI-augmented workflow, so I built a tool to track them. What I learned in the process is that the missing layer isn’t more spending data, it’s task-level outcome data, and almost no one has it.

What engineering teams are already tracking

To be fair about what’s happening out there: the measurement conversation in engineering orgs has matured significantly. Based on research from the Pragmatic Engineer’s reporting on how 18 tech companies measure AI impact, the metrics that tend to show up on engineering dashboards look roughly like this:

These aren’t bad metrics. PR throughput and code churn are legitimate signals. T-Mobile uses AI code acceptance rate to estimate how much AI-authored code reaches production, which is a genuinely hard problem to solve. Shopify took a different approach entirely, introducing an internal AI leaderboard to celebrate developers who spend the most on token consumption, treating experimentation as something to reward rather than scrutinize.

But here’s the structural problem with all of these: they measure activity and volume. They don’t measure whether a specific task was completed faster because of AI, how much faster, whether the speed introduced downstream debt, or what the actual cost-per-hour-saved was. Those are the numbers that connect to the P&L, and almost no team has them.

• The AI Impact Report 2025
• How Token-Based AI Coding Tools Impact Engineering Budgets
• Engineering in the Age of AI: What the 2025 State of Engineering Management Report Reveals - Jellyfish
• 2025 AI Metrics in Review: What 12 Months of Data Tell Us About Adoption and Impact - Jellyfish

That 20% figure is the one that stops me. Sixty-one percent of companies increased their engineering budgets in 2025, yet only 20% of teams say they’re using engineering metrics to measure AI impact. That’s an enormous gap between investment and accountability. The spend is real. The measurement isn’t keeping up.

The problem with token cost as a proxy

Token spend is increasingly visible, and that’s genuinely good. But on its own, it’s the wrong denominator. It tells you how much you’re spending. It doesn’t tell you what you’re getting for it.

Consider two tasks. A deep architecture planning session with Claude might consume 200,000 tokens over 90 minutes and redirect a team away from a week of wrong-direction implementation. That’s an extraordinary return on inference spend. A boilerplate CRUD endpoint might consume 60,000 tokens and save 20 minutes. Far lower ROI per token, even if the acceptance rate looks identical in your dashboard.

The metric that matters is cost-per-hour-saved. Token spend divided by developer time actually recaptured. I haven’t seen this in any standard engineering dashboard. It’s what my tracker computes.

What acceptance rate doesn’t tell you

AI code acceptance rate has become a go-to metric because it’s easy to pull from vendor APIs. It’s a reasonable proxy for whether developers find the tool’s suggestions useful. But it measures input quality, not output quality.

A developer can accept an AI suggestion, ship it, and introduce a bug that costs four hours of debugging next sprint. The acceptance rate logs a win. The actual business outcome was negative.

GitClear’s analysis of 211 million lines of code found churn rising from a 3.3% pre-AI baseline in 2021 to 5.7% by 2024, and AI-assisted code inherently has higher turnover than human-only code. High acceptance rate plus elevated churn rate is a red flag that most dashboards aren’t wired to surface.

The question isn’t just “did the developer accept the suggestion?” It’s “did the task ship cleanly, and did it stay shipped?” That requires tracking completion status alongside AI-assist time, which is what the Status field in my tracker does.

What I built with Mass Ave Labs and why each field exists

I built a tool to track my own AI-augmented work because I couldn’t answer basic questions about it: where am I actually saving time, what’s it costing me per hour saved, which tools are earning their keep on which task types? Every field exists because it closes a loop that standard metrics leave open.

How the fields answer questions, token dashboards can’t

What this data actually changes

Here’s the practical difference task-level tracking makes. Without it, an engineering org making AI investment decisions is working with averages like average acceptance rate, average token spend, and average PR velocity. Averages across task types that behave very differently.

Bug fixes on medium-complexity tickets tend to generate the highest AI ROI. The work is bounded, the context is clear, and AI can generate a candidate fix that a developer reviews rather than the authors. At $175/hour fully-loaded cost, if AI takes a 2-hour bug fix to 30 minutes, that’s $262 in recaptured capacity per ticket. Across 30 bug tickets a sprint, that’s $7,860 per sprint before you’ve touched any feature work.

High-complexity architecture decisions are a different story. The gains are real but slower to appear, and the cost of a wrong suggestion is higher. The right response isn’t “use less AI on hard problems”, it’s “use AI differently on hard problems, with more review checkpoints.”

You can’t build that nuance into your adoption strategy without knowing which tasks are generating which outcomes. The org-level averages will always obscure it.

Six things worth tracking that most orgs aren’t

If I were advising an engineering org on what to start measuring before buying new tooling, this is the data contract I’d ask for:

Baseline time, before AI runs. Estimate how long a task would take without AI assistance, and log it before starting. Engineers who’ve used AI for a year can’t accurately reconstruct this retroactively. It has to be captured in the moment.

Wall-clock time with AI assistance. Not perceived effort, not story points. Actual elapsed time. Perception drifts significantly once AI becomes the norm.

Task type and complexity at the time of work. Without this segmentation, you’re averaging across categories that behave completely differently. Bug fixes and architecture design are not the same kind of AI task.

Which tools ran on which task. Seat utilization tells you a license is being used. Task-level tool logging tells you whether it’s being used where it generates the most value. These are very different signals.

Completion status and rework. Did the AI-assisted task ship cleanly, or did speed come with downstream cost? This is the quality question acceptance rate can’t answer.

Token consumption per task. Not for surveillance — for optimization. When you can pair token spend with time saved per task type, you can answer the question the CFO is eventually going to ask: what is our productive work per dollar of inference?

These are the fields my tool captures. I built it for my own workflow. But the gap it fills exists at every team that’s watching their token bill go up and wondering whether the investment is compounding or just accumulating.

Tool: https://massavelabs.com/impact-tracker

For me, the shift from using AI to engineering with AI happened the moment I stopped reaching for tools and started building a system. Each piece of my stack now has a specific, non-negotiable job. Whether it’s using Claude for architectural reasoning, Party Rock for rapid utility apps, or a local Llama instance to audit my own productivity, this is the infrastructure that powers my work at Mass Ave Labs.The Stack at a Glance

The Core Infrastructure

• Languages & Frameworks: Python, LangChain, MCP (Model Context Protocol).
• Local Inference & Models: Ollama, Hugging Face.
• Data & Memory: ChromaDB and SQLite

The Reasoning & Execution Engines

Claude Code — Sonnet 4.6— The Architectural Brain

Claude Code is where I do my most demanding thinking. It reasons at a level that’s genuinely useful for senior engineering decisions, how to structure a pipeline, which patterns to reach for, and what the tradeoffs are between approaches. It runs in the terminal as an agent; it can work through architectural problems across the entire codebase rather than just suggesting isolated snippets. The MCT here is the AskUserQuestion tool, which asks me questions to generate the best architecture.

Codex & Google CLI — GPT 5.4 and Gemini 3 Pro/Flash— The Agentic Backbone

I rely on a mixture of Codex and Google Gemini for the coding itself. These models handle the heavy lifting of generating the logic and implementation details. This allows my agents to interact directly with cloud infrastructure and local environments, turning the AI from a “chatting partner” into a system operator that can execute commands and manage resources in real-time.

Windsurf — SWE 1.6 — Agentic IDE

The agent for small issues like GitHub resolutions, and the agentic IDE for all my coding.

Governance & Rapid Utility

Prompt Repo & Promptfoo — Managing the “Logic”

If prompts are the new source code, they belong in a repository, not just a scratchpad. I maintain a central GitHub repo for my prompts to ensure version control across all my Mass Ave Labs projects. Right now, I’m using Promptfoo as the framework for this library. Even while I’m in the phase of simply saving and organizing my best-performing prompts, the infrastructure for systematic evaluation is already there. When I’m ready to benchmark a new RAG pipeline, I don’t have to start from scratch.

Party Rock and Replit— Making One-Off Apps

Party Rock is my go-to for making one-off apps. I don’t need a full deployment cycle for a tool I’ll use ten times. I’ve used it to build a LinkedIn connection composer and an AI Career Navigator. I’ve built one app on Replit. The only drawback is on the free plan the app expires in 30 days.

The Feedback Loop

Wispr Flow — Speaking to AI

This allows me to speak my prompts. When you speak, you’re more conversational and exploratory. The outputs are better because the inputs are richer. Most of my technical writing and complex architectural prompts start as spoken thoughts here.

The Stack Watching Itself: MCP + Gemini

This is the piece I built myself. My MCP server logs every task — estimated time without AI vs. actual time with AI. Gemini 2.5 Flash analyzes this data and creates STAR stories. This closes the loop. Most tools make you faster in the moment; this one tells me if I’m actually getting better over time.

The shift from adoptive to transformative isn’t about using more tools. It’s about building a system that gets smarter as you use it.

What does your stack look like for Q2? Let’s talk about it in the comments.

I’m going to show you my workflow for exactly what I did on a real project, Global Impact Tracker, a tool I’m building with a Pro tier gated behind Stripe payments and license key validation. I’ll show you the prompts I actually used, spoken through Wispr Flow into Claude. The flaw that surfaced. The three options we worked through. Why I picked the one I did. And how I got from architectural decision to implemented code.

This is the workflow.

The context: what I was building

Global Impact Tracker needed a Pro tier. The plan was straightforward: Stripe handles payment, a license key gets generated on successful purchase, and a is_pro() function gates Gemini features behind that key. I planned it with Claude before writing a line of code. The direction felt solid. I built an MVP, a valid_keys.txt file on my machine that is_pro() reads to validate keys.

It worked. I tested it. It looked complete.

That’s the dangerous moment in any system, when something works for you, and you mistake that for it working.

Phase 1: before — planning with AI

Before writing code, I use Claude to pressure-test the plan. Not to generate the plan, but to poke at it. I route this to Claude specifically because I need something that can reason about architecture tradeoffs, not just autocomplete my thinking.

Model: Claude Code via Wispr Flow.

A typical before-phase prompt sounds like this, spoken, not typed:

Prompt — spoken via Wispr Flow

I’m building a Pro tier for Global Impact Tracker. Stripe handles payment, generates a license key, and I have an is_pro() function that validates the key and gates Gemini features. The plan is to store valid keys in a file and check against that. What should I be thinking about before I build this?

What I expect back at this stage: a short list of considerations I might have missed, framed around my specific situation. Not a generic architecture lecture. Claude knows the context, the indie tool, private repo, no external users yet, and the output should reflect that. If it’s giving me enterprise-scale concerns for a one-person project, I push back.

The before-phase caught nothing critical this time. The plan looked sound. I built the MVP.

Phase 2: during — building with a placeholder

The valid_keys.txt approach was always meant to be a placeholder, an MVP that let me build the rest of the system while deferring the harder question of how validation actually works at scale. That's a legitimate pattern. Build the interface, defer the implementation.

What I didn’t fully reckon with was the deployment model. And that’s what the after-phase surfaced.

Phase 3: after — stress-testing the mental model out loud

I explain the system out loud, end-to-end, narrating the user journey as if I’m describing it to someone who doesn’t know the codebase. I do this through Wispr Flow into Claude. The act of narrating forces precision. Vague mental models don’t survive being spoken aloud.

This is what I said:

Prompt — spoken via Wispr Flow

How exactly does my server on my machine validate their key if they’re using it on another machine? If they’re calling the server through the config on another machine, how does my machine know to check their Stripe license key? That’s what I am getting?

Right now, Global Impact Tracker is private. The free part needs to be public so I can share it, but the MCP part needs to be private so people can’t see it. What does the Stripe license key actually do? Right now it’s saved into a valid_keys.txt file on my computer. How does that make sure someone’s pro?

What the prompt surfaced

The answer was that it doesn’t.

The valid_keys.txt file lives on my machine. is_pro() reads it on my machine. For any real customer running the tool on their own machine, is_pro() would always return Falsebecause their machine has no valid_keys.txt.

The system worked perfectly because I am both the developer and the only user. That’s not validation. That’s a local file check dressed up as access control.

This is the thing I want to be clear about: Claude didn’t catch this by reading the code in isolation. I did, by being forced to narrate the user journey end-to-end. The narration exposed a gap in the mental model that the code itself didn’t reveal. The flaw was in the deployment assumption, not in the logic. And deployment assumptions don’t live in code. They live in how you think about the system.

Wispr Flow mattered here. Speaking the prompt rather than typing it meant I was more conversational, more exploratory.

Working through the options

Once the flaw was on the table, Claude Code laid out three approaches. My job was to pick the right one for my specific situation, not the theoretically best one in the abstract.

Why signed keys won for this situation

Global Impact Tracker has no paid users yet. Serverless validation is the right answer at scale, but standing up cloud infrastructure before a single customer exists felt premature. Signed keys solve the immediate problem: any machine can validate a key cryptographically without phoning home, without a server, and without exposing anything in the repo. It’s private by design and scales iteratively. Most systems don’t need to be fully scaled up front. They need to be built so scaling is possible later. Signed keys buy that option without paying for infrastructure you don’t need yet.

How the decision became code

Once I had the decision, I didn’t go straight to implementation. I routed back to Claude with a specific instruction:

Implementation prompt

Make a plan for changing the is_pro() validation from a local valid_keys.txt file to signed key validation. Show me the plan first. I’ll review it before you implement anything.

This is a pattern I use consistently, and I think it’s one of the highest-leverage habits in AI-assisted development: always ask for the plan before the code. It costs almost nothing, and it gives you a moment to catch misalignment before it’s baked into an implementation.

I reviewed the plan. It was right. Then I said implement. Claude Code reasons across the whole codebase, writes across files.

The workflow, step by step

1. Plan with Claude Code / Wispr Flow — Architectural direction, tradeoff analysis, option selection. Spoken prompts. Conversational register. No code yet.

2. Request the plan before the code — “Show me the plan first.” Review it. Catch misalignment before it’s implemented. This is the step most people skip.

3. Provide context — Before implementation starts, feed Claude Code the relevant files and documentation it needs. Point it at the parts of the codebase it should pay attention to. Don’t make the model guess at your architecture. The quality of what it produces is directly proportional to the context it has.

4. Implement in phases, test after each one — Claude Code implements step by step. After each phase, tests run and I review the output before proceeding to the next step. This is not optional. Phased testing is what keeps a multi-step implementation from compounding errors across stages.

5. Review the code yourself — Read what the model produced. Don’t ship what you haven’t read.

Step 5 is not ceremonial. On a previous Claude Code implementation in this same project, my code review caught a missing OWASP control around injection and a line that needed simplification. The model got the structure right and left a security gap. The review caught it before it shipped.

6. Stress-test by narrating out loud — Explain the system end-to-end as if describing it to someone who doesn’t know the code. Speak it through Wispr Flow. Vague mental models don’t survive being spoken aloud.

What this workflow actually is

I want to name what’s happening here, because I think it gets mischaracterized. This isn’t “using AI to write my architecture.” The architectural judgment is mine: what to build, what constraints matter, what tradeoffs are acceptable for this specific stage of this specific project. What AI provides is a thinking partner that doesn’t get tired, doesn’t have a stake in the decision, and will tell me honestly when my mental model has a hole in it.

The before phase catches things I haven’t thought of. The during phase builds the system. The after phase is where I narrate my way into the truth. And the narration, spoken, conversational, sometimes uncertain, is the most important part. “That’s what I am getting?” isn’t a polished prompt. It’s a real question. And real questions get real answers.

That’s the workflow.

What I’m adding next

This workflow caught a real architectural flaw and a security gap in AI-generated code. But writing it up forced me to see where it can get sharper.

The first thing I’m adding is a formal spec.md before I touch code. A structured document that captures requirements, architecture decisions, edge cases, and testing strategy before the first prompt. Right now, I plan conversationally. A spec.md makes that planning an artifact I can reference, share, and build on.

The second thing is a prompt plan, a written, sequenced list of prompts for the implementation phase, saved as a file rather than improvised turn by turn.

And the third is automated quality gates around AI output, linting, and static analysis running on every phase, not just my manual review. My code review caught the OWASP injection issue. A tool should be catching that class of problem automatically, before it even gets to me. The pattern holds at every level. AI caught what I missed in the mental model. I caught what AI missed in the implementation.

One more experiment I’m running: using Windsurf’s SWE-1.5 model for the planning and architecture phase, then handing off to Claude Code for implementation. The reasoning is straightforward: if a strong planning model can get the spec and structure right up front, then Claude Code’s Pro membership usage goes toward the work it does best: the actual build. I want to see whether splitting the workflow across models by phase yields better results than running everything through a single model. That’s the next thing I’m testing.

The goal isn’t a perfect workflow. It’s a workflow where fewer things slip through on both sides.

I thought I broke my model.

After retraining, I deployed to Vertex AI, and the endpoint sat in “Deploying/Creating” forever. At first, I did what most of us do under pressure: I assumed the last thing I touched was wrong. In my case, that meant retraining. I kept redeploying and second-guessing feature engineering, artifact generation, and schema output.

The training logs were clean. Artifacts were generated. But deployment still looked stuck.

This is where I had to slow down and stop guessing.

What I assumed (and ruled out)

My first failed assumption was: “I must have changed training in a way that broke serving.”

I retrained again, same problem.

Second failed assumption: “The new model itself is bad.”

Once I understood the model registry flow better in GCP, I tried deploying an older model. That worked. That was the key directional signal: the model logic wasn’t the root issue. The infrastructure/package path was.

Third failed assumption: “There are no logs, so there’s nothing to debug.”

There were logs, but my filter was too narrow in Logs Explorer. Once I broadened the query, I finally saw the real error:

exec /usr/bin/sh: exec format error

That line changed everything.

Why exec format error pointed to architecture mismatch

exec format error usually means the runtime is trying to execute a binary built for a different CPU architecture than the host environment.

In practical terms: if I build a container image locally on Apple Silicon, it can produce linux/arm64 image layers. Vertex serving environments for this setup expect linux/amd64 (x86_64). So the container starts to initialize, tries to run shell/entrypoint commands, and fails immediately because the binary format is incompatible.

That explains the “stuck deploy” feeling:

• endpoint shows deploying,
• app-level logs are sparse or misleading,
• retries happen in the backend,
• and it looks like model behavior, when it’s actually image runtime compatibility.

What fixed it

I rebuilt the serving image in a way that guaranteed compatible architecture (Cloud Build / amd64 path), uploaded model artifacts again, registered a fresh model version, and deployed to a clean endpoint.

Deployment worked.

Not a training bug.

What I changed in my process

1. I now treat deployment as a separate system from training quality.
2. I validate runtime image architecture before spending cycles on retraining.
3. I use explicit model and endpoint IDs when deploying (no implicit “latest”).
4. I’m building reliable deploy/undeploy scripts, which was my original task anyway, so I can iterate safely and control costs.

Two real takeaways

AI won’t replace us. Context still matters. I couldn’t prompt my way out until I understood enough GCP context to ask the right questions.

And you have to know your tools well enough to give explicit instructions to AI.

Once I provided specific signals (registry behavior, endpoint state, log error, CLI constraints), the debugging got sharp and fast.

This wasn’t a model failure story.

It was a systems debugging story.