<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Siddharth B on Medium]]></title>
        <description><![CDATA[Stories by Siddharth B on Medium]]></description>
        <link>https://medium.com/@siddharthdb?source=rss-71e70cfa82d8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*5VQ1wrFFM-ZE8E2o.jpg</url>
            <title>Stories by Siddharth B on Medium</title>
            <link>https://medium.com/@siddharthdb?source=rss-71e70cfa82d8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 04:11:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@siddharthdb/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An Embedding Model Upgrade Is a Data Migration]]></title>
            <link>https://medium.com/@siddharthdb/an-embedding-model-upgrade-is-a-data-migration-01c8bf7c27b2?source=rss-71e70cfa82d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/01c8bf7c27b2</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[vector-database]]></category>
            <dc:creator><![CDATA[Siddharth B]]></dc:creator>
            <pubDate>Fri, 13 Nov 2026 05:33:48 GMT</pubDate>
            <atom:updated>2026-11-13T05:34:20.181Z</atom:updated>
            <content:encoded><![CDATA[<p>How to rebuild RAG indexes without silently corrupting retrieval</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ym9Nwpx06EFI8LlbbGe1UQ.png" /></figure><h3>In brief</h3><ul><li>A vector index is derived from the source version, parser, chunking policy, embedding model, dimensions, similarity metric, metadata schema and index parameters. Changing any of them creates a new retrieval release.</li><li>Mixing two embedding models with the same dimension count raises no error. Retrieval quality degrades without any signal in logs or monitoring.</li><li>Build each release in a separate index, reconcile and evaluate it, switch through an alias that also changes the query configuration, and keep the previous release available for rollback.</li></ul><h3>1. A model change that raises no error</h3><p>A team replaces its embedding model with a newer one that also produces 1,536-dimensional vectors. New and updated documents are embedded with model B and written into the collection that model A populated. Inserts succeed because the dimensions match, and queries still return k results because approximate nearest-neighbour search always does. Over the following weeks, relevant passages move down the ranking and drop out of the candidate set, while the LLM continues to produce fluent answers from weaker evidence.</p><p>The vector store has no basis for rejecting this. Microsoft’s Azure AI Search documentation states that the engine does not raise an error when documents and queries are embedded with different models; it returns poor results. Equal dimensions establish structural compatibility, not semantic comparability. Two independently trained models organise meaning differently, so the distance between a model B query and a model A chunk carries little information. Had the new model changed the dimension count, the first insert would have failed, which is easy to diagnose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pOY3Z-Kbi10UAUzNGjxsSQ.png" /><figcaption><em>Figure 1. Documents and queries embedded by different models with equal dimensions. The write-time check passes and the search compares vectors from two spaces.</em></figcaption></figure><h3>2. The vector index is derived data</h3><p>We treat a vector index the way we treat a search index or a materialised view: derived from an authoritative source and rebuildable at any time. The source corpus (policies, circulars, filings) is the system of record. The retrieval corpus (parsed text, chunks, metadata and lineage) is versioned. The vector index is disposable. The vector store should never be the only place that holds source text or document identity, because an index that cannot be rebuilt can only be copied, errors included.</p><p>The index depends on more than the embedding model:</p><pre>retrieval release = source snapshot + parser + chunking policy<br>                  + embedding model and dimensions + similarity metric<br>                  + metadata schema + index parameters</pre><p>A change to any component produces a new release. Moving from fixed 1,000-character chunks to heading-aware 600-token chunks changes chunk count, citation boundaries and the evidence passed to the LLM, with the embedding model unchanged. Hybrid search adds analysers, synonyms, sparse encoders, quantisation and the re-ranker. Some of these are defaults: ElasticSearch indexes float vectors as int8_hnsw unless configured otherwise, and text-embedding-3-large accepts a dimensions parameter, so the same model at 1,024 and 3,072 dimensions gives two different releases.</p><p>Each release is described by a manifest stored with the index:</p><pre>retrievalRelease: policy-rag-2026.11.1<br>source:   { corpusSnapshot: policy-corpus-2026-11-01, manifestDigest: &quot;sha256:0bd4...&quot; }<br>parsing:  { parserProfile: layout-aware-pdf-v2, ocrPolicy: scanned-pages-only }<br>chunking: { strategy: heading-aware, targetTokens: 600, overlapTokens: 80, policyVersion: chunk-policy-4 }<br>embedding: { model: text-embedding-3-large, dimensions: 3072, profile: embedding-profile-3 }<br>index:    { metric: cosine, algorithm: HNSW, M: 32, efConstruction: 200, quantization: none }<br>metadataSchema: policy-metadata-2.1<br>reranker: reranker-profile-2<br>evaluationSet: policy-golden-questions-5</pre><p>The embedding profile is used for both documents and queries. Given the manifest and the source snapshot, anyone on the team should be able to rebuild the same chunks.</p><h3>3. Lineage on every chunk</h3><p>In a regulated setting, each answer must trace to a source version and a specific chunk. Lineage is written at ingestion, not reconstructed from logs.</p><pre>{<br>  &quot;chunkId&quot;: &quot;POL-271:sha256:91bc:4.2:03&quot;,<br>  &quot;documentId&quot;: &quot;POL-271&quot;,<br>  &quot;documentVersion&quot;: &quot;2026-09-15&quot;,<br>  &quot;contentHash&quot;: &quot;sha256:91bc...&quot;,<br>  &quot;section&quot;: &quot;4.2 Credit Approval Authority&quot;,<br>  &quot;pageFrom&quot;: 18, &quot;pageTo&quot;: 19,<br>  &quot;effectiveFrom&quot;: &quot;2026-10-01&quot;,<br>  &quot;permittedGroups&quot;: [&quot;CREDIT-RISK&quot;],<br>  &quot;retrievalRelease&quot;: &quot;policy-rag-2026.11.1&quot;<br>}</pre><p>A deterministic chunk id (document, content hash, section, ordinal) makes re-ingestion idempotent and reduces reconciliation to a set comparison. Version and effective date allow superseded documents to be filtered out. Permitted groups make access filtering testable. The release tag records which release served a given answer.</p><h3>4. Build the new release beside production</h3><p>Re-embedding the live collection in place recreates the mixed state from section 1 for the duration of the migration. The new release is built in a separate collection, policy_chunks_2026_11, while production continues to query policy_chunks_2026_08. Because documents keep changing during the rebuild, the migration uses a watermark:</p><ol><li>T0: freeze the source-corpus manifest.</li><li>T1: build the new index from the manifest.</li><li>T2: capture additions and changes made after T0.</li><li>T3: apply that delta to the new index.</li><li>T4: pause ingestion briefly, or set a final watermark.</li><li>T5: reconcile, evaluate and switch.</li></ol><p>Ingestion events must be idempotent and carry the source version, content hash and target release, so each change can be applied to both releases and verified afterwards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mQIwilv5kJjPGIwA7Qu5Vg.png" /><figcaption><em>Figure 2. Rebuild timeline. The serving release receives every change; the new release is built from the frozen manifest plus the delta.</em></figcaption></figure><h3>5. Reconcile, then evaluate retrieval</h3><p>Reconciliation checks completeness (expected against indexed documents and chunks, latest delta applied), integrity (no duplicate ids, missing hashes or orphaned chunks) and conformance (vector dimensions, classifications, parser and chunker versions). Matching counts do not prove equivalent retrieval, but mismatched counts show that the migration is incomplete.</p><p>Retrieval is then evaluated separately from generation, so that weak retrieval is not masked by well-written answers. A governed question set with expected supporting passages is run against both releases, measuring Recall@K, Precision@K, MRR, nDCG, zero-result rate, permission-filter pass rate and p95 latency. For financial documents the set should include near-identical clauses from different policy versions, tables split across pages, product codes, superseded circulars, restricted documents, and questions that have no approved answer.</p><pre>releaseGate:<br>  recallAt10: { min: 0.92, maxDropVsCurrent: 0.01 }<br>  ndcgAt10: { maxDropVsCurrent: 0.02 }<br>  restrictedDocumentLeakage: 0<br>  p95RetrievalLatencyMs: 350<br>  unexplainedZeroResultRate: 0.02<br>  corpusReconciliationDifference: 0<br>  rollbackRehearsal: passed</pre><p>Thresholds should come from the current release’s measured baseline. Restricted-document leakage is the only one that must be an absolute zero.</p><h3>6. Switch through a release alias</h3><p>Applications query a logical name, policy_knowledge_active, not a physical collection. Milvus can reassign an alias to another collection in a single call, and most search engines provide an equivalent. A collection alias, however, switches only the document side. The query embedding model, re-ranker and filter configuration must change at the same moment, otherwise queries embedded with model A are run against model B vectors. The logical name should therefore resolve to a release record that names the collection, query embedding profile, re-ranker and filter schema, so cutover is one change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vMcQf4ah8g3-AhOQPj0HUQ.png" /><figcaption><em>Figure 3. The alias resolves to a release record, not a collection. Rollback repoints the alias to the previous record.</em></figcaption></figure><p>Before full cutover, a sample of live queries is run against both releases to compare top-K overlap, rank changes, permission-filter results and latency, without invoking generation. Any semantic or answer cache keyed on old query embeddings is cleared at cutover.</p><h3>7. Keep the rollback path current</h3><p>The previous release stays loaded or quickly loadable, addressable through the alias and paired with its own query embedding profile, until the observation and audit-retention periods end. Two conditions make rollback reliable: the query profile rolls back with the collection, and ingestion deltas continue to be applied to the old release. Without the second, a rollback discards every change made after cutover. The rollback procedure is run once before cutover.</p><h3>8. Worked example</h3><p>A policy assistant with about 80,000 chunks moves from release v1 to v2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*La6KGAULDW1BLWifTB44lg.png" /></figure><p>The dimension change means v2 vectors cannot be written into the v1 collection, so an in-place attempt fails at the first insert. The parser and chunking changes would have altered retrieval without any error, even with the embedding model unchanged.</p><p>Re-embedding is about 48 million tokens, roughly US$6 at the published price of US$0.13 per million tokens for text-embedding-3-large. Raw float32 vector storage grows from about 0.5 GB to 1 GB before index overhead. Most of the effort goes into reconciliation, evaluation and the observation period.</p><h3>Conclusion</h3><p>Embedding, parsing and chunking changes should follow the same process as a database schema migration: build a new release, verify its content and retrieval quality, switch through a reversible alias, and keep the ability to rebuild any release from versioned sources. Snapshots shorten recovery; the release manifest makes the index reproducible.</p><h3>References</h3><ul><li>Zilliz / Milvus documentation, <a href="https://docs.zilliz.com/reference/python/python/Collections-alter_alias">alter_alias</a>.</li><li>Microsoft Learn, <a href="https://learn.microsoft.com/azure/search/tutorial-rag-build-solution-models">RAG tutorial: choose models for embedding and chat</a>.</li><li>Microsoft Learn, <a href="https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-configure-vectorizer">Configure a vectorizer in a search index</a>.</li><li>Elastic, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/8.15/dense-vector.html">dense_vector field type</a>.</li><li>OpenAI, <a href="https://developers.openai.com/api/docs/models/text-embedding-3-large">text-embedding-3-large</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=01c8bf7c27b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microservices Are Not Defined by Where They Run]]></title>
            <link>https://medium.com/@siddharthdb/microservices-are-not-defined-by-where-they-run-8f60cdce7d6f?source=rss-71e70cfa82d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f60cdce7d6f</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Siddharth B]]></dc:creator>
            <pubDate>Sat, 22 Aug 2026 18:50:15 GMT</pubDate>
            <atom:updated>2026-08-22T18:50:15.644Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gDt1sBvHtjTyQduhAU-I6A.png" /></figure><h3>When Tool Knowledge Gets Mistaken for Architecture</h3><p>Most enterprises encounter someone like Engineer X: technically skilled, knowledgeable about Kubernetes, Docker, reverse proxies, process managers, and deployment workflows. The challenge isn’t a lack of expertise; it’s when proficiency in one area influences decisions across unrelated domains. This approach can lead to risky enterprise decisions. For example, in one organization, a highly capable engineer insisted that all new services had to be containerized and deployed via Kubernetes, even though some internal tools would have been simpler and more cost-effective to manage as standalone processes on virtual machines. The result was unnecessary complexity, longer deployment times, and increased operational overhead for applications that did not need this level of orchestration.</p><h3>The Argument</h3><p>Consider a business-testing environment where several independently developed services must run before the final Kubernetes-based UAT environment is available. These services are already separated by responsibility, each with its own codebase, API, process, deployable artefacts, and integration contracts.</p><p>A VM is available. For business testing, the services are deployed as separate processes using a process manager like PM2, with a suitable proxy managing routing.</p><p>Engineer X then asserts, “If all the services are running on one VM, it is a monolith.”</p><p>While this statement sounds authoritative, it is architecturally incorrect. A VM represents an infrastructure boundary, whereas a microservice defines an application boundary. Co-location does not merge service boundaries.</p><h3>Five Services Do Not Become One Application Because They Share a Server</h3><p>Imagine five services. These are Customer, Risk, Case Management, Notification, and Authentication. All of them are deployed on one VM:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E2IGRT3zdccxkZmJFEfD4w.png" /></figure><p>They remain separate processes. Their codebases and APIs are distinct, and ownership boundaries persist. Release artefacts also remain separate rather than becoming a single executable. The deployment topology has changed, but the application architecture has not.</p><p>This separation has direct operational implications: teams can manage, monitor, and restart deployments per service, isolating and resolving issues without affecting unrelated services. Troubleshooting is often simpler because failures are contained within clear service boundaries, and logs or metrics can be attributed to individual services. Even when co-located, operational tasks remain service-specific, reinforcing the distinction between infrastructure choices and architectural boundaries.</p><h3>What Actually Creates a Monolith</h3><p>A monolith, or more critically, a distributed monolith, results from coupling rather than co-location. Five services on Kubernetes can still function as a single application if they require joint releases, depend on each other’s internal implementations, process requests synchronously across all services, share business logic, or cannot evolve independently.</p><p>This is coupling. Packaging these tightly coupled services in Docker images and deploying them to Kubernetes does not resolve the underlying issue. The result is a distributed monolith with advanced orchestration.</p><h3>PM2 Was Never Supposed to “Solve Microservices”</h3><p>Another common argument is: “PM2 does not implement the microservices pattern.” This is accurate. Neither Kubernetes, Docker, nor a reverse proxy like YARP (Yet Another Reverse Proxy) define application architecture.</p><p>PM2 addresses process management. Docker handles packaging and runtime isolation. Kubernetes manages container orchestration. A reverse proxy manages routing. Microservices address application decomposition and independent evolution. Comparing tools from different architectural layers as if they solve the same problem is a mistake.</p><h3>The Layer Confusion</h3><p>A cleaner mental model separates the concerns explicitly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T_VMQ2ZJBtySqqmjKxzgFw.png" /></figure><p>These layers influence each other but are not interchangeable. When architecture discussions fail to distinguish these concerns, adding terminology does not improve reasoning; it only obscures confusion.</p><h3>“But UAT Runs Kubernetes”</h3><p>At this point, the discussion typically becomes more nuanced and unfolds in three stages.</p><p><strong>Stage one</strong> is the blunt claim: services on one VM are not microservices. This is easy to refute, since infrastructure and application architecture are separate concerns. The conversation should not revisit this distinction.</p><p><strong>Stage two</strong> shifts to a more reasonable point: consistency matters more than labels. The business-testing environment should resemble UAT to ensure meaningful testing. This concern is valid, as deployment-model drift is a real risk. The right response is to clarify what you’re validating at this stage. If the goal is to test pod scheduling, readiness and liveness probes, ingress behavior, or autoscaling, the environment should closely match Kubernetes. If the focus is on business workflows, API behavior, validation rules, or functional integration, a VM running independent processes is sufficient, since these do not depend on the orchestration layer.</p><p>To aid decision-making, a practical checklist can help determine the appropriate environment for each testing objective:</p><p>- If your test objectives include: pod scheduling, container lifecycle, readiness/liveness probes, autoscaling, ingress, or any Kubernetes-specific behaviors: use an environment that matches the Kubernetes UAT topology.</p><p>- If your test objectives are focused on: business process validation, API functionality, workflow correctness, integration of independently developed services, or test data flows: a VM running separate processes may suffice.</p><p>- When security, image provenance, or release management requirements require container-image parity prior to UAT: ensure test environments follow the same packaging and deployment model as UAT.</p><p>- For performance, resource usage, or operational monitoring validation: align the environment with production or UAT as closely as practical.</p><p>Always align infrastructure similarity to the aspects under test, rather than defaulting to complete environment duplication.</p><p><strong>Stage three</strong>, if managed well, addresses the core question: does a documented requirement — functional, security, or release-management — mandate container-image parity before UAT? If such a requirement exists, incorporate it. If not, the current deployment decision remains until new requirements arise. This approach isn’t about inflexibility; it’s about ensuring architecture decisions are based on requirements rather than tool preference.</p><h3>Environment Parity Is Not Environment Identity</h3><p>Production-like environments are important, and differences between environments introduce risk. However, insisting on identical infrastructure at every stage can become unproductive. Each stage validates different aspects:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OAKhIXz364qZt-Xj2WGRRg.png" /></figure><p>The required level of infrastructure parity ought to follow the validation objective, not ideology.</p><h3>Where Half-Baked Thinking Becomes Expensive</h3><p>Enterprise architecture decisions have compounding effects. An unnecessary assumption for one service may cost days, but as a platform rule across fifty applications, it can cost years. Statements such as “microservices require Kubernetes,” “multiple services on one VM are monolithic,” or “if PM2 is involved, the architecture is wrong” should always be scrutinized. These statements combine independent concerns into rules that sound decisive but lack real substance.</p><p>To challenge such assumptions early, architects can hold regular review sessions where teams openly question the rationale for technology choices. Encourage teams to document not just decisions, but the problems they intend to solve and the trade-offs considered. Promote a culture where asking, “What evidence supports this rule?” and “How might this fail in our context?” is routine in the architectural process. By making critical examination and collaborative challenge a habit, teams can spot flawed assumptions before they become costly missteps.</p><h3>The Most Dangerous Engineer Is Not the One Who Knows Nothing</h3><p>An inexperienced engineer typically asks questions. The greater challenge is when someone with deep expertise in one technology applies that knowledge to unrelated areas. For example, a skilled Kubernetes engineer may understand orchestration but is not necessarily an authority on domain partitioning, transactional boundaries, API design, or data ownership. Similarly, an application architect should not assume expertise in Kubernetes internals. Effective architecture requires humility to recognize where one layer ends, and another begins. Without this distinction, confidence can become a liability.</p><p>To promote cross-domain understanding and cultivate a culture of humility, architects can facilitate open communication through regular knowledge-sharing sessions and architectural review boards. These forums encourage team members to present their decisions, share rationale, and expose assumptions to healthy scrutiny. Inviting engineers from different domains to explain their perspectives and challenges helps broaden everyone’s understanding and reinforces the importance of learning from each other. By making these activities a routine part of the architectural process, organizations can reduce blind spots and empower teams to make more informed, collaborative decisions.</p><h3>Start With the Problem</h3><p>A mature technical discussion follows a clear sequence: business problem, constraints, failure modes, architectural concerns, required capabilities, and then technology choice. Too often, enterprise discussions begin with a preferred technology and seek justification for it. The first approach produces sound architecture; the second results in technology theatre.</p><h3>Solve the Problem in Front of You</h3><p>If the requirement is to run five independently developed services for business testing on an available VM until the Kubernetes UAT platform is ready, the architecture decision should consider process isolation, configuration, restart behavior, routing, logging, resource usage, security, deployment repeatability, and, most importantly, the test’s objective. PM2 may suffice. Docker may provide additional isolation. Kubernetes may not be necessary at this stage, unless operational constraints dictate otherwise. The correct solution depends on the immediate requirement, not on tool preference.</p><h3>Architecture Is About Proportionality</h3><p>Enterprise architecture is not about selecting the most advanced technology, but about choosing the simplest solution that meets requirements and allows for future growth. Sometimes Kubernetes is appropriate; sometimes a VM or a process manager is sufficient. The responsibility is not to defend a tool, but to understand the trade-offs well enough to justify the decision.</p><h3>Final Thought</h3><p>Microservices are defined by the quality of their boundaries, not by whether they run on Kubernetes, Docker, a process manager, or on one or many VMs. Architecture is not demonstrated by tool familiarity, but by understanding which problems each tool solves, which it does not, and when its use adds unnecessary complexity. This distinction separates tool knowledge from architectural judgment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f60cdce7d6f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Risk Signal Is Useless Until Someone Can Act on It]]></title>
            <link>https://medium.com/@siddharthdb/a-risk-signal-is-useless-until-someone-can-act-on-it-cba200d7ab37?source=rss-71e70cfa82d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/cba200d7ab37</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Siddharth B]]></dc:creator>
            <pubDate>Sat, 11 Jul 2026 13:41:19 GMT</pubDate>
            <atom:updated>2026-07-11T13:41:19.536Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YEbBsx3AIw7x33lMbnDHEg.png" /></figure><h3>Using Transactional Outbox and CQRS-Lite in an Early Warning System</h3><p>An Early Warning System can detect a serious credit-risk signal, persist it successfully, and still fail the business.</p><p>Consider a simple example — A Loan Management System reports that a customer facility has reached <strong>45 Days Past Due</strong>. The Early Warning System evaluates its configured rule:</p><pre>IF daysPastDue &gt; 30<br>THEN raise a HIGH-severity early warning signal</pre><p>The signal is saved successfully. The application then attempts to publish an EarlyWarningSignalRaised event to Kafka.</p><p>Kafka is temporarily unavailable. The database contains the signal, but:</p><ul><li>no risk case is updated;</li><li>no relationship-manager task is created;</li><li>no escalation begins;</li><li>no risk dashboard reflects the change;</li><li>no downstream audit consumer receives the event.</li></ul><p>Technically, the signal exists.</p><p>Operationally, it never happened.</p><p>That small gap between <strong>committing business state</strong> and <strong>communicating the change</strong> is one of the most dangerous failure windows in an event-driven system. It has a name, the <strong>dual-write problem</strong>, and it is the reason the outbox pattern exists: a service needs to update its database <em>and</em> publish a message, but the two operations cannot share a single atomic transaction. When they diverge, you get exactly the silent inconsistency above.</p><h3>The Dangerous Design</h3><p>A common implementation looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rdv5-yaW3AuNyhv-E0AHmQ.png" /></figure><p>The database transaction and Kafka publication are two separate operations.</p><p>A local PostgreSQL transaction cannot guarantee that Kafka will accept the event. Kafka cannot roll back the PostgreSQL commit if publication fails.</p><p>This creates several failure combinations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I_fgIIYyRq2ntUZtdXAQWw.png" /></figure><p>Adding retries helps, but retries alone do not solve the consistency problem.</p><p>A retry mechanism must first know <strong>what needs to be retried</strong>, whether it was already completed, and whether repeating the operation is safe.</p><h3>The Transactional Outbox Bridge</h3><p>The Transactional Outbox pattern changes the write operation. It was catalogued by Chris Richardson in the microservices.io pattern collection and has since become a foundational technique for reliable event-driven systems.</p><p>Instead of saving the signal and publishing directly to Kafka, the EWS writes two records inside the same PostgreSQL transaction:</p><ol><li>the early warning signal;</li><li>an outbox event describing what happened.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WPh1T8v61fFNWCc4MFbVIA.png" /></figure><p>The command-side transaction becomes conceptually simple:</p><pre>@Transactional<br>public IngestRepaymentObservationResult ingest(<br>        IngestRepaymentObservationCommand command,<br>        UUID correlationId) {<br>    RepaymentObservation observation =<br>                observationService.validateAndPersist(command);<br>    Optional&lt;MatchedRule&gt; match =<br>            ruleEvaluator.evaluate(observation);<br>    if (match.isEmpty()) {<br>        return IngestRepaymentObservationResult.noSignal(observation.id());<br>    }<br>    EarlyWarningSignal signal =<br>            signalService.create(observation, match.get());<br>    outboxRepository.save(<br>            EarlyWarningSignalRaised.from(signal, correlationId)<br>    );<br>    return IngestRepaymentObservationResult.signalRaised(signal.id());<br>}</pre><p>There is no Kafka call inside this transaction. If the signal insert fails, the outbox insert rolls back. If the outbox insert fails, the signal insert rolls back. The database therefore contains either:</p><pre>Signal + Outbox Event</pre><p>or:</p><pre>Neither</pre><p>A separate publisher later reads pending outbox records and sends them to Kafka.</p><p>If Kafka is unavailable, the event remains in PostgreSQL and can be retried.</p><p>The outbox does not make the entire distributed workflow transactional. It converts a distributed-transaction problem into a local database-transaction problem, which is something relational databases are extremely good at solving. It creates a durable bridge between the local business transaction and asynchronous communication.</p><blockquote><em>One caveat worth internalising: this atomicity holds only if the signal write and the outbox write share the </em>same<em> transaction. If </em><em>signalService.create or </em><em>observationService.validateAndPersist open their own new transactions, the guarantee quietly evaporates. The </em><em>@Transactional boundary above assumes default propagation. The whole point is that both inserts commit or neither does.</em></blockquote><h3>Who Actually Reads the Outbox?</h3><p>Writing the outbox row is the easy half. The harder design decision is how a separate process discovers pending rows and relays them to Kafka. There are two established approaches, and the choice has real operational consequences.</p><p><strong>Change Data Capture (CDC).</strong> Tools like Debezium tail the database’s write-ahead log and stream committed changes straight to Kafka. This gives near-real-time latency, negligible load on the outbox table (the log is read, not queried), and event ordering that matches commit order for free. The cost is a standing piece of infrastructure: you deploy and monitor Debezium, enable logical replication, watch connector lag, handle restarts, and manage schema evolution.</p><p><strong>Polling.</strong> A scheduled component queries the outbox table on a short interval, publishes what it finds, and marks those rows done. It is database-independent, trivial to reason about, and survives publisher restarts without missing events. Its two well-known drawbacks are latency bounded by the poll interval and continuous read load on the database.</p><p>For a regulated Early Warning System with moderate ingestion volume, <strong>polling is the right default</strong>, and I’d argue <em>against</em> reaching for CDC here. The general guidance in the field is to choose CDC when sub-100ms latency is a hard requirement or throughput is very high, and to choose polling for simplicity at moderate volume. An EWS is squarely in the second category. A signal that reaches the risk dashboard 500 milliseconds later instead of 5 milliseconds later changes nothing operationally; a risk analyst is not reacting inside a 100ms window. What <em>does</em> matter in regulated finance is auditability and the number of moving parts you have to certify, monitor, and explain to a regulator. Polling keeps the entire mechanism inside the application and the database you already control, with no replication slot to manage and no separate connector to fall over at 2 a.m. Polling publishers have been run at 500ms intervals handling five-figure event-per-second loads without trouble, comfortably beyond what an EWS ingestion stream demands.</p><p>That decision made, two implementation details are non-negotiable.</p><p><strong>Concurrent publishers must not double-publish.</strong> If you run more than one publisher instance for availability, and in a regulated system you will, two instances polling the same table will otherwise fight over the same rows. PostgreSQL’s answer is SELECT ... FOR UPDATE SKIP LOCKED. When one instance locks a batch of rows, the others simply skip past the locked rows and claim the next available ones, processing concurrently instead of blocking in a queue. This is the standard production idiom for a horizontally scaled outbox publisher:</p><pre>@Transactional<br>public void drainOutbox() {<br>    List&lt;OutboxEvent&gt; batch = jdbc.query(<br>        &quot;&quot;&quot;<br>        SELECT id, aggregate_type, aggregate_id, event_type, payload, created_at<br>        FROM outbox_event<br>        WHERE published_at IS NULL<br>        ORDER BY created_at<br>        LIMIT :batchSize<br>        FOR UPDATE SKIP LOCKED<br>        &quot;&quot;&quot;,<br>        rowMapper, BATCH_SIZE<br>    );<br><br>    if (batch.isEmpty()) {<br>        return;<br>    }<br><br>    for (OutboxEvent event : batch) {<br>            kafkaPublisher.publish(event); // partition key = aggregate_id (see below)<br>        }<br>        jdbc.update(<br>            &quot;UPDATE outbox_event SET published_at = NOW() WHERE id = ANY(:ids)&quot;,<br>            batch.stream().map(OutboxEvent::id).toArray(Long[]::new)<br>        );<br>    }<br>}</pre><p>The @Transactional boundary is what holds the row locks for the whole batch; the locks release on commit, after the rows are marked published.</p><p><strong>Published rows must be cleaned up.</strong> Unlike CDC, where the transaction log is managed by the database and needs no housekeeping, a polling outbox accumulates rows forever unless you prune them. Left unchecked, the table bloats and the very polling query that drives the publisher slows down. Because the events already live durably in Kafka, the outbox table is only a temporary buffer; a scheduled sweep is enough:</p><pre>DELETE FROM outbox_event<br>WHERE published_at IS NOT NULL<br>  AND published_at &lt; NOW() - INTERVAL &#39;3 days&#39;;</pre><p>Keep a retention window long enough to support replay and forensic investigation, but finite. In a regulated context, treat that window as a documented, tunable parameter rather than an accident of whenever someone last ran a cleanup job.</p><h3>Ordering Is Not Free: Partition by Facility</h3><p>There is a subtle trap hiding in the aggregation logic this system depends on. The case consumer cares about the <em>latest</em> DPD value and the <em>highest</em> severity observed for a facility. That logic silently assumes events arrive in the order they were produced.</p><p>Kafka only guarantees ordering <strong>within a single partition</strong>. If two EarlyWarningSignalRaised events for the same facility land on different partitions, a consumer can process yesterday&#39;s DPD 45 <em>after</em> today&#39;s DPD 46 and record a stale &quot;latest&quot; value.</p><p>The fix is a deliberate partitioning key. Publish each event with a Kafka partition key equal to the aggregate identifier, here facilityId (or customerId, depending on how you define the aggregate). Every signal for a given facility then flows through the same partition in commit order, and the consumer partitions its work the same way to preserve that order end to end. Without this, &quot;latest observation wins&quot; is a bug waiting for production traffic.</p><p>(This is also one quiet advantage CDC would have handed you for free: the replication log preserves commit order globally. With polling you get ordering by choosing the partition key deliberately, which is a fair trade for the simpler operational footprint.)</p><h3>From Observation to Accountable Action</h3><p>The source system does not submit a pre-classified risk signal.</p><p>It submits an observation:</p><pre>{<br>  &quot;customerId&quot;: &quot;CUST-90001&quot;,<br>  &quot;facilityId&quot;: &quot;FAC-100045&quot;,<br>  &quot;sourceSystem&quot;: &quot;LMS&quot;,<br>  &quot;sourceReference&quot;: &quot;LMS-REPAYMENT-20260710-100045&quot;,<br>  &quot;daysPastDue&quot;: 45,<br>  &quot;asOfDate&quot;: &quot;2026-07-10&quot;<br>}</pre><p>The EWS evaluates a database-configured and versioned rule:</p><pre>Rule: DPD_THRESHOLD_BREACH<br>Metric: DAYS_PAST_DUE<br>Operator: &gt;<br>Threshold: 30<br>Severity: HIGH<br>Version: 1</pre><p>When the rule matches, the system creates the signal and records which rule version generated it.</p><p>This matters because the threshold may change later. Risk and audit teams must still be able to answer:</p><blockquote><em>Which rule produced this signal at that point in time?</em></blockquote><p>The correlationId threaded through the command is part of the same accountability story. It is written into the outbox event, carried into the Kafka message headers, and propagated by every downstream consumer, so a single identifier ties the source observation to the signal, the event, and every business action it triggers. In a regulated system, that end-to-end thread is the difference between &quot;we think this escalation came from that repayment&quot; and being able to prove it.</p><p>The resulting EarlyWarningSignalRaised event is then distributed to independent consumers.</p><h3>CQRS-Lite Without Event-Sourcing Complexity</h3><p>The design uses a practical form of CQRS.</p><p>The command side:</p><ul><li>ingests repayment observations;</li><li>evaluates the DPD rule;</li><li>creates the risk signal;</li><li>stores the outbox event.</li></ul><p>The query side:</p><ul><li>serves risk dashboards;</li><li>supports filtering by customer, facility, severity and signal age;</li><li>reads from an asynchronously maintained projection.</li></ul><p>This is <strong>CQRS-lite</strong>, not event sourcing.</p><p>The early_warning_signal table remains the source of truth. Kafka events communicate committed business facts, while query projections are shaped for operational use.</p><p>A small projection delay is acceptable, provided it is measurable and visible.</p><h3>Many Signals, One Open Case</h3><p>A real EWS should not create a new risk case every day merely because the facility remains overdue.</p><p>Suppose the system receives:</p><pre>10 July: DPD 45<br>11 July: DPD 46<br>12 July: DPD 47</pre><p>These are three valid observations and three valid signal events. But they represent one continuing risk condition. The case consumer therefore applies this model:</p><pre>Many qualifying signals<br>          ↓<br>One open risk case<br>          ↓<br>One active relationship-manager task</pre><p>Repeated signals update the open case with:</p><ul><li>the latest signal;</li><li>the latest DPD value;</li><li>the latest observation time;</li><li>the occurrence count;</li><li>the highest severity observed.</li></ul><p>The active relationship-manager task is updated rather than duplicated. Its priority may increase, and its due date may move earlier.</p><p>If the case is later closed and a new qualifying signal arrives, a new case can be created while preserving the previous case as historical evidence.</p><h3>At-Least-Once Means Duplicates Are Possible</h3><p>The outbox publisher may successfully publish an event and then crash before marking it as published.</p><p>After restarting, it may publish the same event again.</p><p>That is not necessarily a defect. It is a normal consequence of <strong>at-least-once delivery</strong>, and it is worth being honest that <em>both</em> polling and CDC produce it. Neither approach gives you exactly-once for free.</p><p>Every consumer must therefore be idempotent. Before applying its business effect, a consumer records:</p><pre>eventId + consumerName</pre><p>If that combination already exists, the event has already been processed by that consumer.</p><p>There is a subtlety here that separates a correct implementation from one that merely looks correct. The dedup check, the business effect, and the recording of the key must all happen <strong>in a single transaction</strong>, and the deduplication must lean on a database constraint rather than a prior read. If you SELECT first to check for the key and only then insert, two concurrent deliveries of the same event can both pass the check before either commits, and both apply the effect. The robust construction is a unique constraint on (event_id, consumer_name) and an insert that either succeeds or raises a constraint violation. That violation <em>is</em> your signal that the event was already handled:</p><pre>@Transactional<br>public void handle(EarlyWarningSignalRaised event) {<br>    try {<br>        processedEventRepository.insert(event.id(), CONSUMER_NAME); // unique (event_id, consumer_name)<br>    } catch (DuplicateKeyException alreadyProcessed) {<br>        return; // idempotent no-op<br>    }<br>    applyBusinessEffect(event); // same transaction as the insert above<br>}</pre><p>Handled this way, a duplicate delivery is acknowledged without:</p><ul><li>creating another risk case;</li><li>duplicating an RM task;</li><li>incrementing the occurrence count twice;</li><li>writing another audit entry.</li></ul><p>The realistic guarantee is:</p><blockquote><em>At-least-once event delivery with idempotent business effects.</em></blockquote><p>That is more honest and useful than loosely claiming “exactly once.”</p><h3>What the Outbox Actually Gives Us</h3><p>The Transactional Outbox pattern does not eliminate distributed-system failure.</p><p>It makes failure manageable.</p><p>It gives the platform:</p><ul><li>a durable record of events waiting to be published;</li><li>retry-able Kafka publication;</li><li>visibility into pending and failed events;</li><li>correlation between the source observation, signal, event and downstream action;</li><li>replay capability;</li><li>measurable projection lag;</li><li>protection against silent event loss.</li></ul><p>For an Early Warning System, that distinction matters. A risk signal is not valuable merely because an application stored it successfully. It becomes valuable when it reliably leads to a visible, traceable and accountable business action.</p><h3>Further Reading</h3><ul><li>Chris Richardson, <em>Pattern: Transactional Outbox</em> (the canonical reference): <a href="https://microservices.io/patterns/data/transactional-outbox.html">https://microservices.io/patterns/data/transactional-outbox.html</a></li><li>Chris Richardson, <em>Pattern: Transaction Log Tailing</em> (the CDC alternative): <a href="https://microservices.io/patterns/data/transaction-log-tailing.html">https://microservices.io/patterns/data/transaction-log-tailing.html</a></li><li>Chris Richardson, <em>Pattern: CQRS</em>: <a href="https://microservices.io/patterns/data/cqrs.html">https://microservices.io/patterns/data/cqrs.html</a></li><li>Confluent, <em>Understanding the Dual-Write Problem and Its Solutions</em>: <a href="https://www.confluent.io/blog/dual-write-problem/">https://www.confluent.io/blog/dual-write-problem/</a></li><li>Debezium, <em>Reliable Microservices Data Exchange With the Outbox Pattern</em>: <a href="https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/">https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/</a></li><li>Oskar Dudycz, <em>Outbox, Inbox patterns and delivery guarantees explained</em>: <a href="https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/">https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/</a></li><li>CYBERTEC, <em>SKIP LOCKED: one of my favorite PostgreSQL 9.5 features</em> (the concurrency idiom): <a href="https://www.cybertec-postgresql.com/en/skip-locked-one-of-my-favorite-9-5-features/">https://www.cybertec-postgresql.com/en/skip-locked-one-of-my-favorite-9-5-features/</a></li><li>PostgreSQL documentation, <em>SELECT / The Locking Clause</em> (FOR UPDATE ... SKIP LOCKED): <a href="https://www.postgresql.org/docs/current/sql-select.html">https://www.postgresql.org/docs/current/sql-select.html</a></li><li>Martin Kleppmann, <em>Designing Data-Intensive Applications</em> (background on the dual-write problem and log-based delivery)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cba200d7ab37" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Production-Grade RAG Pipeline for BFSI Documents — Lessons from the Trenches]]></title>
            <link>https://medium.com/@siddharthdb/building-a-production-grade-rag-pipeline-for-bfsi-documents-lessons-from-the-trenches-6ac2a46a3ce7?source=rss-71e70cfa82d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ac2a46a3ce7</guid>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Siddharth B]]></dc:creator>
            <pubDate>Tue, 26 May 2026 05:45:09 GMT</pubDate>
            <atom:updated>2026-05-26T05:45:09.441Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This isn’t a tutorial. Enough of those exist. This is a post-mortem, stitched together from what teams actually encountered when the tutorials ran out and production showed up.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jpnYo4fKADfUlqsssmxPJw.png" /></figure><h3>The Pipeline Worked. Until It Didn’t.</h3><p>The demo was convincing. Retrieval looked sharp. Stakeholders were nodding. Three weeks into production, answers were plausible but wrong — not obviously wrong, which would have been easier. Subtly, quietly wrong. The kind a compliance officer catches on a Tuesday morning.</p><p>The team tuned prompts. Tweaked temperature. Questioned the embedding model. Added RAM. They were debugging Layer 4. The fire was in Layer 0.</p><p>BFSI RAG has a specific failure signature — and it almost always cascades from the bottom up. Here’s the diagnostic stack, layer by layer.</p><h3>Layer 0 — Source Data Quality: The Floor Nobody Inspects</h3><p>BFSI documents were never designed to be machine-read. Annual reports arrive as scanned PDFs — OCR artifacts, flattened tables, boilerplate disclaimers repeated across 200 pages, inconsistent field naming across quarterly versions. Then someone drops a password-locked filing into the ingestion queue. The pipeline chokes silently. No error, no flag, no quarantine — just missing chunks nobody knew were missing. And storage? Nobody budgeted for raw files, cleaned versions, failed runs, and reruns. Disk fills mid-ingestion. Partial writes land in the VectorDB. The corpus looks complete. It isn’t.</p><p>The deeper problem: public and proprietary documents entered the same pipeline with no classification label. At retrieval time the VectorDB has no concept of which chunks came from a published earnings report versus a confidential internal analysis. It just returns the top-k. In BFSI, an unlabeled chunk surfaced in the wrong context isn’t a UX problem. It’s a potential regulatory incident.</p><pre># Pre-flight validation — nothing enters the pipeline without this<br>def validate_pdf_access(filepath: str) -&gt; dict:<br>    result = {&quot;file&quot;: filepath, &quot;accessible&quot;: False, &quot;reason&quot;: None}<br>    try:<br>        with pikepdf.open(filepath) as pdf:<br>            if pdf.is_encrypted:<br>                result[&quot;reason&quot;] = &quot;SECURITY_LOCKED&quot;<br>                return result<br>            result[&quot;accessible&quot;] = True<br>    except pikepdf.PasswordError:<br>        result[&quot;reason&quot;] = &quot;PASSWORD_PROTECTED&quot;<br>    return result</pre><pre># Every document gets a manifest - classification travels with every chunk<br>@dataclass<br>class DocumentManifest:<br>    document_id: str<br>    classification: str          # public / proprietary / restricted<br>    document_type: str<br>    checksum: str                # deduplication and change detection<br>    accessible: bool<br>    validation_reason: str</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XRtDHCkL-Wx3ETyoGNm21w.png" /></figure><h3>Layer 1 — Pipeline Architecture: Where Spaghetti Meets Scale</h3><p>The classic progression: notebook → script → production. At some point a function calls a function calls a function, and nobody can say where ingestion ends and embedding begins. Everything is coupled. Nothing is testable in isolation. Then someone queues a 340-page annual report and the process dies at page 240 with no checkpoint, no error surface, and no record of how far it got. Partial chunks are live in the VectorDB. Retrieval is answering from an incomplete corpus. Nobody knows.</p><p>The response to this was usually more RAM, a bigger instance — the <strong>More Power Anti-Pattern</strong>. The pipeline still had no input contract, no observability, and no memory. It just failed with better hardware. The fix isn’t infrastructure, it’s architecture: separate stages with explicit boundaries, a persistent checkpoint store, and embedding batched with backoff — not fire-and-forget synchronous calls that time out at chunk 2,400 of 3,000.</p><pre>class IngestionCheckpoint:<br>    &quot;&quot;&quot;<br>    Persistent job state. Survives crashes. Enables resume — not restart.<br>    &quot;&quot;&quot;<br>    def is_chunk_stored(self, doc_id: str, chunk_id: str) -&gt; bool:<br>        return (<br>            self.state[&quot;documents&quot;]<br>            .get(doc_id, {})<br>            .get(&quot;chunks&quot;, {})<br>            .get(chunk_id, {})<br>            .get(&quot;status&quot;) == ChunkStatus.STORED.value<br>        )</pre><h3>Layer 2 — Chunking: A Data Engineering Problem Wearing a Prompt’s Clothes</h3><p>Retrieval returns wrong answers. The instinct is to fix the prompt. The prompt gets longer, more specific, more carefully worded. Retrieval returns different wrong answers. Three weeks pass. Nobody looked at the chunks. <strong><em>This is the prompt engineering trap and it is almost universal</em></strong>. If a financial table got flattened into a string of numbers by the PDF parser and then sliced at a token boundary mid-row, no prompt recovers the context that was never in the chunk to begin with.</p><p>BFSI documents break naive chunking in specific ways: tables lose column relationships when flattened, regulatory clauses become incomplete without their cross-referenced sections, and boilerplate disclaimers repeated across 200 pages pollute the entire embedding space with near-duplicate vectors. The fix is domain-aware chunking — tables stay atomic, narrative splits at sentence boundaries with overlap, boilerplate is stripped before any embedding is computed. Every chunk inherits its document’s classification from Layer 0. Every chunk carries page range and section metadata. The VectorDB stores retrieval units, not text fragments.</p><pre>class BFSIDocumentChunker:<br>    BOILERPLATE_PATTERNS = [<br>        r&quot;(?i)this document contains forward.looking statements&quot;,<br>        r&quot;(?i)past performance does not guarantee&quot;,<br>        r&quot;(?i)page \d+ of \d+&quot;,<br>    ]<br>    # Tables stay intact — splitting a table destroys its meaning<br>    # Clauses split at sentence boundaries with overlap<br>    # All chunks inherit classification from DocumentManifest</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g2Xwg_v8bS5ppM1QoTS-EA.png" /></figure><h3>Layer 3 — VectorDB: An Architecture Decision Wearing a Library Import’s Clothes</h3><p>At some point in the project someone ran pip install chromadb and that was the VectorDB decision. Three seconds. No evaluation criteria. ChromaDB is an excellent POC tool — it gets teams moving fast and that matters. The problem is that it was still there six months later handling production queries against a corpus of proprietary financial documents, with no persistent durability guarantees, no server-side metadata filtering, and no access control at the collection level. Switching later means re-embedding everything — every document, every chunk, while the old system is still serving production. That&#39;s the real cost of the default choice.</p><p>What actually matters in a BFSI vector store isn’t query speed benchmarks — it’s metadata filtering at query time (the public/private boundary from Layer 0 must be enforceable here, not in application code), durability guarantees, and a data residency story that the compliance team has actually reviewed. Milvus handles scale and metadata filtering well; pgvector is underrated for BFSI specifically because it lives inside the relational infrastructure the organization already knows how to audit; Weaviate’s hybrid search is useful where queries are semi-structured. None of these is universally correct. All of them require a deliberate design decision about collection architecture — separate collections per classification, or namespace separation — before the first document is embedded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EljhjwUXFH5MH5ynnMZQkw.png" /></figure><h3>Layer 4 — Orchestration: When Python Runs Out of Road</h3><p>There’s a moment when everything technically works and nothing operationally functions. Retrieval returns results. Answers generate. Nobody can tell you why some queries take 400ms and others take 12 seconds. Nobody can trace which chunk answered which question. Nobody can produce an audit trail when compliance asks for one. The pipeline runs blind. This is the orchestration problem, and it arrives quietly, long after the decisions that caused it.</p><p>The framework choice mattered too. An early commitment to CrewAI made sense at prototype stage; its abstraction model worked against the team in production — debugging a failed step meant unwrapping layers of framework internals, adding custom logging required fighting the execution model, and end-to-end tracing became archaeology. Switching to LangChain mid-project was the right call with real cost. But the deeper issue was the assumption that Python would handle everything: ingestion, embedding, orchestration, API exposure, audit logging, latency management. Python is excellent at the RAG layer. It wasn’t designed to carry the enterprise perimeter. Spring Boot owns auth, rate limiting, structured audit logging, circuit breakers, and distributed trace context. Python owns retrieval, reranking, and LLM interaction. The boundary between them is a clean internal API — not a monolith, not shared code. Each layer doing what it does best.</p><pre>// Spring Boot — trace context propagation into audit record<br>@Component<br>public class RAGAuditInterceptor implements HandlerInterceptor {<br>    @Override<br>    public boolean preHandle(HttpServletRequest request, ...) {<br>        String traceId = UUID.randomUUID().toString();<br>        MDC.put(&quot;traceId&quot;, traceId);<br>        MDC.put(&quot;queryTimestamp&quot;, Instant.now().toString());  <br>        return true;<br>    }<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9fdmGu-cdHDF8r9iEfmuxQ.png" /></figure><h3>Layer 5 — Production Readiness: Assume This Will Fail</h3><p>In BFSI, a latency SLA on a financial query is not a target, it’s a contract. An audit trail of retrieval is not a nice-to-have, it’s a compliance requirement. Which chunk answered the question, from which document, with what classification, at what confidence level, at what time. If the system can’t answer that, it’s not production-ready regardless of how good the answers look. The failure taxonomy almost always runs bottom-up: bad data at Layer 0 looks like a chunking problem at Layer 2, which looks like a retrieval problem at Layer 3, which looks like a hallucination problem to the user at Layer 5. Teams spend weeks at the top of the stack debugging a fire burning at the bottom.</p><p>The one design principle that changes everything: <strong>build for failure from day one, not after the first production incident.</strong> That means circuit breakers on every external dependency, latency broken down by stage (not just total response time), retrieval audit records generated per query, and a compliance team that has reviewed the audit format before go-live — not after. It also means having an architect in the room before code is written, not after the spaghetti has shipped. Teams that got this right didn’t have better engineers. They had architectural review earlier. That is the entire difference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qoD-rpbnmq9xBG2flGXueQ.png" /></figure><h3>The Directional Architecture</h3><p>No single RAG architecture fits every BFSI use case. What these layers give other teams isn’t a blueprint to copy, it’s a diagnostic framework. When retrieval is wrong, it’s a Layer 2 problem. When the VectorDB surfaces restricted content, it’s a Layer 0 problem that wasn’t enforced at Layer 3. When nobody can explain a 12-second query, it’s a Layer 4 observability problem. When the compliance team asks for an audit trail and the answer is “<em>we don’t have one</em>” that’s a Layer 1 decision that was never made.</p><p>Take what fits. Adapt what doesn’t. But whatever you build — have someone who’s thought about production failure modes review it before the first document lands in the pipeline.</p><p><em>What layer is your current pipeline actually failing on? Drop it in the comments — these conversations are usually where the real war stories surface.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ac2a46a3ce7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why We Built Our Own Data Sanitization Layer Instead of Buying One]]></title>
            <link>https://medium.com/@siddharthdb/why-we-built-our-own-data-sanitization-layer-instead-of-buying-one-89c33db86879?source=rss-71e70cfa82d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/89c33db86879</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Siddharth B]]></dc:creator>
            <pubDate>Sat, 16 May 2026 06:44:35 GMT</pubDate>
            <atom:updated>2026-05-16T06:44:35.518Z</atom:updated>
            <content:encoded><![CDATA[<h3>ADR Series — Article 5 of 5</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jmmuYX8nWqvk89P-M10sTw.png" /></figure><p><em>This is the final article in a five-part series on Architecture Decision Records — the most underrated tool in a senior engineer’s arsenal. Each article covers one real decision, the reasoning behind it, and why writing it down changes everything.</em></p><h3>Intro</h3><p>There is a particular category of architectural decision that is more difficult to manage than the ones that are hard to make. It is the decision that feels obvious in the room — where the reasoning is clear, the tradeoffs are understood, and the conclusion feels self-evident to everyone present. These decisions rarely get documented because they do not feel like decisions at all.</p><p>Six months later, when the people who were in that room have moved on and a new team member looks at what was built and asks why the team is maintaining something a vendor could provide, the reasoning that felt self-evident has simply gone. Without a written record, the question reopens everything. This article is that record.</p><h3>The Decision</h3><p>For the boundary between internal confidential data and external LLM providers, the recommendation is to <strong>build a custom data sanitization layer</strong> rather than adopt a commercial vendor solution.</p><p>The sanitization layer is not a product the team buys. It is a capability the team owns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mOCrjl19rNk9qp6RpiznSg.png" /></figure><h3>The Reasoning</h3><h3>Why the Vendor Option Is Worth Considering</h3><p>Commercial sanitization solutions are a reasonable starting point for this problem. Established vendors offer PII detection trained on large and varied datasets, with rule sets refined across many enterprise deployments. They provide managed infrastructure, removing the operational burden from the engineering team. They carry compliance certifications — SOC 2, ISO 27001, and others — that hold weight in audit conversations. The ability to delegate a sensitive function to a specialist provider, with documented accountability on their side, is a genuine advantage.</p><p>The build decision does not rest on dismissing any of this. It rests on three specific problems that emerge when a vendor sits at this particular boundary in this particular context.</p><h3>The First Problem: Accountability Cannot Be Delegated</h3><p>When a regulator or auditor asks how the team ensures that confidential borrower data does not reach the external model, the answer needs to be complete and owned. A commercial sanitization layer introduces a dependency at the most sensitive point in the architecture — the boundary between internal confidential data and an external AI provider. The vendor’s rule set determines what gets stripped. The vendor’s update cycle determines when those rules change. Their definition of sensitive data may or may not align with the team’s own regulatory obligations.</p><p>When the auditor asks for a step-by-step explanation of the sanitization logic — and in a regulated financial environment, this question will come — the answer needs to point at the team’s own codebase, rule definitions, and audit logs. A vendor’s documentation can support that conversation, but it cannot substitute for it. Owning the layer means owning the explanation.</p><h3>The Second Problem: Generic Detection Is Not Domain-Specific Enough</h3><p>Commercial PII detection is built for breadth, designed to work across many industries, document types, and data patterns. Financial services credit documents sit outside what general-purpose detection handles well.</p><p>The ways borrower identity is embedded in a credit memo are specific to the institution and the sector — account numbers formatted to internal conventions, reference codes that are meaningless to an outside observer but uniquely identifying within the system, entity identifiers that do not match standard PII patterns but carry the same compliance sensitivity in context. A custom layer can be built to know these patterns explicitly. A commercial layer is built to detect PII in general, and in a domain where a missed identifier constitutes a compliance incident, that difference is not a minor technical nuance.</p><h3>The Third Problem: Vendor Dependency at the Critical Boundary</h3><p>The sanitization layer sits directly between internal confidential data and the external LLM. It is the mechanism that makes the entire architecture defensible. If the vendor’s service experiences an outage, the team faces a choice between halting the pipeline or routing data around the sanitization boundary under pressure. An outage at a compliance boundary is not a degraded experience problem — it is a risk management problem, and no vendor SLA resolves that from the team’s perspective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zgo-aF1fME3ZIfU5p5RORg.png" /></figure><p>A layer the team owns and operates ensures that when failure occurs, it occurs in infrastructure the team understands, can diagnose, and can explain. That operational posture is materially different from dependency on an external service at the most sensitive point in the architecture.</p><h3>The Tradeoff Being Made</h3><p>Vendors build continuously — improving detection, expanding rule sets, handling infrastructure changes as the threat landscape evolves. Choosing to build means the team takes on that responsibility directly. The sanitization rules need to stay current as document formats change and regulatory requirements shift. That is a meaningful ongoing commitment.</p><p>The reason the team accepts it is that the sanitization layer connects three properties that, taken together, define a function worth owning: it sits on the compliance-critical boundary, it requires domain-specific knowledge of the institution’s data patterns, and it is the foundation on which the architecture’s auditability rests. A vendor can provide a capable general-purpose solution, but the combination of regulatory accountability, domain specificity, and audit dependency makes this a function where ownership is more appropriate than delegation.</p><h3>Applicability</h3><p>The build-vs-buy question does not have a universal answer. The conditions that support the build decision here are worth making explicit:</p><ul><li><strong>Build</strong> when the function sits at a compliance-critical boundary the team is fully accountable for. If the team owns the outcome but the vendor controls the mechanism, there is a structural gap between accountability and control that surfaces in audit reviews and incident responses.</li><li><strong>Build</strong> when domain specificity matters more than breadth. General-purpose tools work well for broad use cases. When the data has highly specific characteristics — institutional document formats, sector-specific identifier conventions — a custom implementation can be tuned in ways a commercial tool cannot.</li><li><strong>Buy</strong> when the function is not on the critical accountability path. For lower-sensitivity sanitization, internal tooling, or use cases where a missed detection does not constitute a compliance incident, commercial tools are often the right and efficient choice.</li></ul><blockquote><strong><em>The question worth asking is not “can we buy this?” but “what happens when the vendor’s behaviour diverges from our compliance obligation?” If the answer involves a compliance incident, the function should be owned.</em></strong></blockquote><h3>Conclusion</h3><p>The sanitization layer was built rather than bought because at the boundary between confidential financial data and an external AI provider, the ability to explain every rule, own every decision, and answer every audit question from the team’s own codebase is not optional.</p><p>Without this written down, the vendor proposal resurfaces whenever a new architect joins and sees what looks like infrastructure being maintained unnecessarily, or a budget review surfaces the ongoing cost, or a vendor demonstrates more sophisticated detection than the current implementation. Each of those conversations is reasonable on its own terms and starts from the wrong place without the ADR.</p><h3>Closing the Series</h3><p>This series covered five architectural decisions made in the context of deploying AI into a regulated financial environment. Each one was clear to the people who made it and invisible to everyone who came after.</p><ul><li><a href="https://medium.com/@siddharthdb/why-i-recommend-rag-over-fine-tuning-for-credit-analysis-82bd83bda470"><strong>ADR #1 — RAG over Fine-Tuning</strong></a><strong>:</strong> Traceability matters more than fluency when the output may be reviewed by a regulator.</li><li><a href="https://medium.com/@siddharthdb/why-we-recommend-a-multi-llm-architecture-over-a-single-provider-b5d3e0ffce4c"><strong>ADR #2 — Multi-LLM Architecture</strong></a><strong>:</strong> Compliance will ask where the data goes. The sanitization boundary is what makes that question answerable.</li><li><a href="https://medium.com/@siddharthdb/why-we-kept-the-human-in-the-loop-for-credit-decisions-7ecfad88deeb"><strong>ADR #3 — Human in the Loop</strong></a><strong>:</strong> Accountability, model maturity, and institutional trust are real constraints — not temporary ones.</li><li><a href="https://medium.com/@siddharthdb/why-we-recommend-event-streaming-over-synchronous-rest-for-financial-pipelines-ae997194c4e7"><strong>ADR #4 — Event Streaming over REST</strong></a><strong>:</strong> In a regulated pipeline, the audit trail cannot be an afterthought. It has to be structural.</li><li><strong>ADR #5 — Build vs Buy:</strong> When a function sits at the compliance-critical boundary, ownership is more appropriate than delegation.</li></ul><p>An ADR does not lock a decision in place. It records the reasoning so that when circumstances change, the next conversation starts from an informed baseline rather than from scratch. That is not documentation overhead. That is how engineering teams avoid relearning the same lessons twice.</p><blockquote>Write the ADR. One is enough to start.</blockquote><p><em>If this series was useful, follow for more. I write about software architecture, AI deployment in regulated industries, and the decisions engineering teams make that nobody documents.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89c33db86879" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why We Recommend Event Streaming Over Synchronous REST for Financial Pipelines]]></title>
            <link>https://medium.com/@siddharthdb/why-we-recommend-event-streaming-over-synchronous-rest-for-financial-pipelines-ae997194c4e7?source=rss-71e70cfa82d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae997194c4e7</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[fintech]]></category>
            <dc:creator><![CDATA[Siddharth B]]></dc:creator>
            <pubDate>Fri, 15 May 2026 05:56:28 GMT</pubDate>
            <atom:updated>2026-05-15T05:56:28.221Z</atom:updated>
            <content:encoded><![CDATA[<h3>ADR Series — Article 4 of 5</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m1vnH2MAoArS6v6pyLLYNQ.png" /></figure><p><em>This is the fourth in a five-part series on Architecture Decision Records — the most underrated tool in a senior engineer’s arsenal. Each article covers one real decision, the reasoning behind it, and why writing it down changes everything.</em></p><h3>Intro</h3><p>When a financial document pipeline is being designed, the default choice is almost always synchronous REST. It is familiar, fast to wire up, and easy to reason about in development. Each service exposes an endpoint, each stage calls the next one in sequence, and the whole thing works cleanly enough when nothing is under pressure.</p><p>The problem with optimising for development simplicity is that production is rarely simple. Financial document pipelines operate under variable load, handle documents of varying complexity, depend on services with their own latency profiles, and sit inside regulated environments where the complete processing history of every document is a requirement — not a feature. The gap between how a synchronous pipeline behaves in development and how it behaves under those conditions is wide, and it tends to reveal itself at the worst possible moment.</p><p>This ADR captures why event streaming was chosen over synchronous REST, what that decision protects against, and why the architectural complexity it introduces is deliberate rather than incidental.</p><h3>The Decision</h3><p>For pipelines that move financial documents through sequential processing stages — ingestion, analysis, and structured output — the recommendation is <strong>event streaming over synchronous REST</strong> between stages. The implementation technology — whether Kafka, RabbitMQ, AWS Kinesis, Azure Service Bus, or another message broker — is a separate decision driven by infrastructure constraints and team familiarity. The architectural pattern is what matters here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cAoL7CMQUXK3DtpdGJSNqw.png" /></figure><h3>The Reasoning</h3><h3>Understanding the Two Patterns</h3><p>In a synchronous REST pipeline, each stage calls the next one directly over HTTP/S and waits for a response before proceeding. The stages are tightly coupled — stage two cannot begin until stage one has completed, and stage three cannot begin until stage two has completed. The pipeline is a chain, and its overall throughput and resilience are determined by its weakest link.</p><p>In an event-driven pipeline, stages communicate through a message broker rather than through direct calls. Each stage reads from a queue or stream, performs its processing, and publishes its output as an event for the next stage to consume. The stages are decoupled — they have no direct awareness of each other, only of the events they produce and consume. This decoupling is the defining characteristic of event-driven architecture: the producer of an event does not wait for the consumer, and neither is aware of the other’s operational state.</p><h3>How a Synchronous Pipeline Fails</h3><p>The failure mode of a synchronous pipeline is predictable and consequential. When one stage experiences increased load — handling a particularly complex document, recovering from a transient dependency issue, or simply receiving more traffic than it was sized for — the entire pipeline stalls. Upstream stages accumulate timeout errors. Downstream stages receive no work. Documents that were in flight at the moment of the failure have an uncertain status, and determining whether they were processed, partially processed, or silently dropped requires reconstructing logs from multiple services at the exact timestamp of the incident.</p><p>In most engineering contexts, that recovery process is a debugging inconvenience. In a financial services environment, where each document represents a credit decision in progress and audit requirements apply to every processing step, it is both an operational liability and a compliance risk. The question “what happened to this document?” must have a definitive answer — not an approximation assembled from log files after the fact.</p><h3>How Event Streaming Changes the Failure Profile</h3><p>When a stage in an event-driven pipeline experiences increased load, events accumulate in the message broker rather than causing upstream timeouts. The other stages continue operating at their own pace, unaffected. When the slow stage recovers, it processes the accumulated events in order. No documents are lost. Nothing needs to be reprocessed from the beginning. The pipeline degrades gracefully under pressure rather than failing as a unit.</p><p>This difference in failure profile — graceful degradation versus cascading failure — is the primary architectural reason for the recommendation. It is not about throughput or scalability in the abstract; it is about what the system does when something goes wrong in production, which in a regulated financial environment is the scenario that matters most.</p><p>Beyond resilience, event streaming provides two additional properties that are material in this context.</p><p>The first is replay capability. Events in a message broker persist after they are consumed. If a processing error occurs, if a bug is deployed, or if a configuration change produces unexpected results, the event stream can be replayed from any point. The complete processing history of every document is available and queryable. This capability transforms audit responses from a reconstruction exercise into a straightforward query.</p><p>The second is the audit trail itself. In a streaming pipeline, the record of what happened to each document is a structural output of the way the pipeline operates — not something engineered separately or assembled from application logs. Every event that moves through the system is captured by the messaging mechanism as a first-class concern. The audit trail exists by default.</p><h3>The Case for REST and Why It Does Not Apply Here</h3><p>The case for synchronous REST in a pipeline like this is real and worth engaging with honestly. REST is simpler to develop against, simpler to test locally, and requires less infrastructure to run. For pipelines where processing steps are fast and uniform, where the document volume is low and predictable, and where audit-ability is not a first-class requirement, REST is a perfectly reasonable choice.</p><p>None of those conditions describe a regulated financial document pipeline. Processing times vary with document complexity. Volume is neither low nor entirely predictable. Audit-ability is a regulatory requirement. The conditions that make REST a good choice are precisely the ones that do not apply here.</p><h3>The Simplification Argument</h3><p>A pattern that tends to emerge when streaming infrastructure is in place is that engineers new to the codebase evaluate the configuration — consumer groups, queue definitions, acknowledgement handling, dead-letter routing — and propose replacing it with direct REST calls. The argument is typically framed around developer experience: REST is easier to reason about locally, familiar to the whole team, and the streaming infrastructure feels like overhead for a pipeline that mostly runs without incident.</p><p>The argument is accurate about what it describes and incomplete about what it omits. The streaming infrastructure’s value is concentrated in the failure scenarios: the 2am incident where one stage backs up under load, the compliance review where a specific document’s processing history is requested weeks after the fact, the bug that requires replaying a segment of the event stream to correct the output. An engineer evaluating the two approaches without having encountered those scenarios will consistently underestimate the cost of the simpler alternative.</p><p>The ADR closes that gap by making the failure scenarios and audit requirements visible before the simplification proposal is evaluated, rather than after.</p><h3>The Tradeoff Being Made</h3><p>Local development is more involved with event streaming. Running a message broker locally, configuring consumer groups, and debugging asynchronous processing issues requires more setup than calling a REST endpoint directly. Engineers joining the team who are unfamiliar with event-driven architecture need time to develop an accurate mental model of the system’s behaviour. The operational surface is larger, with consumer lag, dead-letter queues, and acknowledgement semantics all requiring monitoring and understanding.</p><p>These costs are genuine and should not be minimised. The decision to accept them rests on a direct comparison: a synchronous REST pipeline that loses documents under partial failure, cannot replay events, and satisfies audit requirements only through additional engineering effort is not actually the simpler choice. It is simpler to build initially and considerably more expensive in the scenarios that matter most in this environment.</p><h3>Applicability</h3><p>Event streaming over synchronous REST is the appropriate architectural choice in pipeline contexts where three conditions hold.</p><p>The first is where event loss under partial failure is unacceptable. If a document being silently dropped or processed in an uncertain state has downstream consequences that are costly or difficult to correct, the durability guarantees of a streaming architecture — at-least-once or exactly-once delivery, depending on broker configuration — justify the additional complexity that REST cannot provide without significant additional engineering.</p><p>The second is where processing stages have materially different latency profiles. When one stage of a pipeline can take significantly longer than others due to document complexity, external dependencies, or variable load, synchronous REST propagates that variability into a full pipeline stall. Event streaming isolates it to the slow stage, allowing the rest of the pipeline to continue operating.</p><p>The third is where the audit trail is a first-class architectural requirement rather than an operational convenience. Any regulated domain — financial services, healthcare, legal, government — where the complete processing record of a document may need to be produced on demand benefits from an architecture where that record is structural. The streaming approach provides this by default. The REST approach requires it to be built separately, with the associated risk that the build is incomplete when it is actually needed.</p><h3>Conclusion</h3><p>The recommendation was event streaming over synchronous REST not because REST is inadequate as a general pattern, but because <strong>a regulated financial pipeline has specific requirements — graceful failure, event replay, and a structural audit trail — that synchronous REST cannot satisfy without considerable additional engineering, and that event streaming satisfies by default</strong>.</p><p>Without this written down, the simplification proposal arrives with every engineer who joins the team and encounters the operational complexity before understanding what it protects. The ADR does not prevent that conversation. It ensures that when it happens, the failure scenarios and audit requirements are part of the starting point — not something discovered after the simplification has already been made.</p><p><em>Next in this series: </em><strong><em>ADR #5 — Why We Built Our Own Data Sanitization Layer Instead of Buying One.</em></strong><em> The build-vs-buy decision nobody wrote down — until now.</em></p><p><em>If this was useful, follow for more. We write about software architecture, AI deployment in regulated industries, and the decisions engineering teams make that nobody documents.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae997194c4e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why We Kept the Human in the Loop for Credit Decisions]]></title>
            <link>https://medium.com/@siddharthdb/why-we-kept-the-human-in-the-loop-for-credit-decisions-7ecfad88deeb?source=rss-71e70cfa82d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ecfad88deeb</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[future-of-work]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Siddharth B]]></dc:creator>
            <pubDate>Thu, 14 May 2026 05:27:24 GMT</pubDate>
            <atom:updated>2026-05-14T05:27:24.898Z</atom:updated>
            <content:encoded><![CDATA[<h3>ADR Series — Article 3 of 5</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IiJBjE_C-NwEQ7poyzyYhA.png" /></figure><p><em>This is the third in a five-part series on Architecture Decision Records — the most underrated tool in a senior engineer’s arsenal. Each article covers one real decision, the reasoning behind it, and why writing it down changes everything.</em></p><h3>Intro</h3><p>There is a particular moment that tends to show up in AI projects, usually somewhere between the first internal demo and the first stakeholder presentation. The system is working well. It is processing financial statements faster than the team expected, surfacing risk signals that would have taken an analyst considerable time to extract manually, and producing structured summaries that are genuinely useful. Everyone in the room can see it working.</p><p>And then the question surfaces, almost always from someone who has been quietly watching the demo with growing interest.</p><p><em>“If it can do all of this, why does a human still need to approve it?”</em></p><p>It is a fair question, and in that moment it is difficult to answer without sounding defensive. The honest answer involves regulation, model maturity, institutional trust, and the particular way AI systems fail in ways that are not always visible until they already have. None of that is easy to explain standing next to a live demo.</p><p>Two organisations — Klarna in fintech, Deloitte in professional services — skipped past that question without a written answer. What followed in both cases is instructive before getting into the reasoning for this decision.</p><h3>The Decision</h3><p>For credit appraisal decisions — approvals, rejections, and risk classifications — the recommendation is to keep a human explicitly in the loop. Not as a rubber stamp or a formality, but as a required, accountable step in the decision chain. The AI assists, analyses, and surfaces relevant information. The human decides.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tJraXca74NvFO9-g77bKUA.png" /></figure><h3>What Happens When the Human Is Removed</h3><h3>Klarna — Speed Over Quality in Financial Services</h3><p>Between 2022 and 2024, Klarna replaced approximately 700 customer service employees with an AI assistant developed in partnership with OpenAI. By February 2024, the company reported that the system was handling around 75% of all customer interactions — roughly 2.3 million conversations — doing the work that had previously required a significant human workforce.</p><p>The efficiency numbers were real. Resolution times dropped and costs fell noticeably. On paper, the transition looked like a success.</p><p>What did not appear on the initial reports was the erosion happening underneath. Customers began reporting frustration with responses that felt mechanical and inadequate for anything beyond routine queries. Satisfaction scores declined. The volume of repeat contacts — customers coming back because their issue had not actually been resolved — increased. In a business built on financial trust, the reputational cost of these experiences began accumulating in ways that are harder to measure than cost savings but considerably more expensive to repair.</p><p>By mid-2025, Klarna was actively rehiring human agents. CEO Sebastian Siemiatkowski publicly acknowledged that the company had gone too far, stating that the focus on efficiency and cost had ultimately produced lower quality outcomes. The company described its revised approach as a dual-track model, combining AI for routine interactions with human agents for situations requiring judgement, empathy, and complex problem-solving.</p><p>The core assumption — that efficiency and quality were effectively the same thing — had never been written down, never challenged, and never examined as a decision. It turned out to be wrong.</p><h3>Deloitte Australia — Professional Work Without Professional Review</h3><p>In late 2024, Deloitte was commissioned by Australia’s Department of Employment and Workplace Relations to produce an independent assurance review of the government’s welfare compliance framework. The contract was valued at AU$440,000.</p><p>When the 237-page report was published in July 2025, a researcher at Sydney University identified serious problems. The report contained references to academic papers that did not exist, quotes attributed to scholars who had never written them, and a fabricated citation from a federal court judgment. In total, twenty errors were identified — not minor stylistic issues, but substantive inaccuracies that called the integrity of the analysis into question.</p><p>Deloitte acknowledged that parts of the report had been produced using generative AI without adequate human review. The company agreed to refund the final installment of its consultancy fee, and the original report was removed from the government website. The incident became one of Australia’s first public refunds for undisclosed AI use in government work.</p><p>The failure was not a technology problem. It was a process problem. AI-generated content had been included in a high-stakes professional deliverable without the human review that would have caught the errors before publication. The consequences — financial, reputational, and in terms of client trust — followed directly from that gap.</p><h3>The Reasoning</h3><h3>The Regulation Answer</h3><p>In most regulated jurisdictions, credit decisions affecting borrowers carry accountability obligations that cannot be assigned to a model. Someone must be responsible for the outcome — able to appear before a regulator, able to explain the reasoning behind a decision, and able to answer for it if the decision turns out to be wrong or, in certain circumstances, discriminatory.</p><p>A model cannot fulfil any of those obligations. A human can. This consideration alone is sufficient to close the debate for most regulated institutions, though it is worth noting that it is also the least interesting reason — because even in the absence of a regulatory requirement, keeping a human in the loop would still be the right call for the reasons that follow.</p><h3>The Model Drift Problem</h3><p>A credit model trained on data from a particular economic environment will behave predictably within that environment, right up until conditions change materially. Interest rates shift. Sector risk profiles evolve. What constitutes a healthy debt-service coverage ratio in a low-rate environment can look quite different when rates move significantly. A macroeconomic change that makes historical training data unrepresentative does not announce itself to the model — the system continues producing outputs with the same apparent confidence regardless of whether its assumptions still hold.</p><p>A human analyst, by contrast, carries current context that cannot be encoded into a model trained months earlier. They know what is happening in the market this quarter, what the institution’s current risk appetite is, and which sectors are under stress right now. That contextual judgment gets applied to every review in ways that the model simply cannot replicate. This is not a capability gap that a better model will eventually close — it is a structural advantage of having a person who operates in the present moment of every decision.</p><h3>The Trust Problem</h3><p>There is a third consideration that tends not to be stated plainly in architecture reviews, but deserves to be. Borrowers, regulators, and institutional risk committees do not yet fully trust AI systems to make consequential decisions autonomously. This is not primarily a reflection of the technology’s capabilities — it is a reflection of the fact that trust is built through demonstrated reliability over time, and that time has not yet elapsed for most AI deployments in financial services.</p><p>Both Klarna and Deloitte discovered that removing human oversight before that trust had been established produced consequences that were more costly to resolve than the oversight itself would have been. In Klarna’s case the cost was measured in customer satisfaction and brand reputation. In Deloitte’s case it was measured in client refunds, public accountability, and a formal government review of AI use in consultancy work.</p><p>The human in the loop serves as a trust-building mechanism as much as a decision safeguard. Every instance in which an analyst reviews an AI-generated output, applies their professional judgment, and confirms or adjusts the recommendation produces a data point. Those data points accumulate, and over time they build the evidence base for expanding AI autonomy in a considered and defensible way. The right to remove the human is something that gets earned through that process — not assumed in advance of it.</p><h3>The Tradeoff Being Made</h3><p>The cost of this decision is speed. A fully automated pipeline would process decisions faster, without the delays introduced by human review queues and analyst availability. For institutions handling large volumes of lower-complexity credit decisions, that is a meaningful operational difference.</p><p>The important point is that this is a position on a maturity curve, not a permanent restriction. As model outputs become more reliable, as the evidence base from supervised deployments accumulates, and as regulatory frameworks evolve to accommodate AI-assisted decision-making, the scope for AI autonomy in this context can and should expand. What the ADR does is capture where that line sits today and why, so that when it moves, it moves as the result of a deliberate decision — documented and reasoned — rather than quietly, by assumption, in the way that Klarna and Deloitte allowed it to move.</p><h3>Applicability</h3><p>This pattern applies beyond credit analysis wherever three conditions hold. First, where decisions carry legal or regulatory accountability obligations and a named human must be responsible for the outcome. Second, where the model’s training data may not remain representative of current conditions, making human contextual judgment a structural part of the decision process rather than an optional review. Third, where trust between the AI system and its key stakeholders — compliance teams, regulators, clients, or end users — has not yet been established through demonstrated reliability over time.</p><p>The Klarna and Deloitte cases are useful reference points here because they demonstrate that these conditions apply outside financial services too. Customer service, professional consulting, government advisory work — any context where consequential decisions are made, where the outputs will be scrutinised, and where the cost of an unreviewed error exceeds the cost of the review itself.</p><h3>Conclusion</h3><p>The decision to keep a human in the loop was not a reflection of doubt in the AI’s capabilities. It reflected something more straightforward: accountability, trust, and model maturity are real operational constraints, not temporary inconveniences, and removing the human from the decision chain before those constraints are resolved is not a technical choice. It is a risk management choice.</p><p>Klarna and Deloitte made that choice without writing it down, without examining it as a decision, and without considering what would happen if the assumption underlying it turned out to be wrong. The outcomes are well documented.</p><p>Without this ADR, the automation question resurfaces every time the system performs well. Every successful pilot becomes an implicit argument for removing the review step. The written decision does not close the door to full automation — it establishes what would need to be true before opening it, and ensures that when the time comes, the decision is made as carefully as the one that kept the human in the loop in the first place.</p><p><em>Next in this series: </em><strong><em>ADR #4 — Why We Recommend Event Streaming Over Synchronous REST for the Financial Analysis Pipeline.</em></strong><em> The architecture that looks over-engineered — until the failure happens at 2am.</em></p><p><em>If this was useful, follow for more. We write about software architecture, AI deployment in regulated industries, and the decisions engineering teams make that nobody documents.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ecfad88deeb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why We Recommend a Multi-LLM Architecture Over a Single Provider]]></title>
            <link>https://medium.com/@siddharthdb/why-we-recommend-a-multi-llm-architecture-over-a-single-provider-b5d3e0ffce4c?source=rss-71e70cfa82d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5d3e0ffce4c</guid>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Siddharth B]]></dc:creator>
            <pubDate>Wed, 13 May 2026 02:32:49 GMT</pubDate>
            <atom:updated>2026-05-14T05:28:34.200Z</atom:updated>
            <content:encoded><![CDATA[<h3>ADR Series — Article 2 of 5</h3><p><em>This is the second in a five-part series on Architecture Decision Records — the most underrated tool in a senior engineer’s arsenal. Each article covers one real decision, the reasoning behind it, and why writing it down changes everything. If you missed </em><a href="https://medium.com/@siddharthdb/why-i-recommend-rag-over-fine-tuning-for-credit-analysis-82bd83bda470"><em>first article</em></a><em>, it covers why RAG beats fine-tuning for credit analysis.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SU5eXUcQkgL6IQNU2VpyKw.png" /></figure><h3>Intro</h3><p>Every AI project starts with the same default.</p><p>Pick one LLM provider. Go deep. One API contract, one integration surface, one vendor relationship. Simpler. Faster to ship. And when the backlog is already longer than anyone’s arm and stakeholders want a demo by end of quarter — it feels like the obviously correct direction.</p><p>It was never really a call anyone made. It was an assumption — shared quietly across the room, never questioned, never written down. The vendor, the architect, the product manager who needed the demo ready. Everyone had a hand in it.</p><p>And then someone from compliance walked in.</p><p>It was not a hostile conversation. Not even technically difficult. But it was the question nobody had a clean answer for — and in a regulated environment, not having a clean answer is its own kind of problem.</p><p>The question was simple.</p><p><em>“When the document goes to the model — where exactly does it go?”</em></p><p>That question is the reason this ADR exists.</p><h3>The Decision</h3><p>For systems processing confidential financial documents — credit memos, bank statements, borrower profiles — the recommendation is a <strong>multi-LLM architecture with an explicit data sanitization layer</strong> between internal systems and any external model provider.</p><p>Not one provider. Two zones. A hard boundary between them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*twGbn56U-pNj8w4-E2Z9jg.png" /></figure><h3>The Reasoning</h3><h3>Why Single-Provider Feels Right — Until It Doesn’t</h3><p>The case for a single provider is genuinely good. One API to maintain. One set of credentials. One integration surface to test, monitor, and update. For internal tooling, low-sensitivity workflows, consumer products — single-provider is completely reasonable.</p><p>The problem is specific to regulated financial services. And it comes down to data residency, accountability, and what happens when someone with audit authority asks the team to justify every step of data handling.</p><h3>The Compliance Question That Will Definitely Come</h3><p>In financial services, confidential document processing is not a grey area. Borrower data, financial statements, credit assessments — these carry regulatory obligations around storage, transit, access, and accountability. The exact regulations vary by jurisdiction. The underlying requirement does not: know where the data goes, why it goes there, and who is accountable if something goes wrong.</p><p>A single external LLM provider means confidential data travels outside the controlled environment every time a document is processed. The provider may be reputable. The connection may be encrypted. But the data has crossed a boundary — and that crossing needs to be explained to a regulator in terms they accept.</p><p><em>“It goes to the provider’s servers but it’s fine because of their SOC 2 certification”</em> is an answer. It is just not always a sufficient one.</p><h3>The Architecture That Solves This</h3><p>The recommendation is a two-zone model.</p><p><strong>Zone A — Internal boundary.</strong> Sensitive data lives here. Sensitive work happens here. An internal or on-premises model reads the raw document, parses structure, extracts key signals. The raw document never leaves this zone.</p><p><strong>Zone B — External provider.</strong> Frontier capability lives here — models that reason well, write clearly, handle complex synthesis. More capable. Also external.</p><p>Between the two zones sits the <strong>sanitization layer</strong>. Its job is straightforward but non-negotiable: strip personally identifiable information, redact account identifiers, anonymize borrower references. What crosses from Zone A to Zone B is context — not confidential data.</p><p>The external model never sees the original document. It sees what the sanitization layer decided was safe to send. That distinction is auditable. That boundary is defensible.</p><h3>What “Defensible” Actually Means in Practice</h3><p>Without the sanitization boundary, when the auditor asks <em>“where does the data go?”</em> — the answer involves the provider’s compliance documentation. The team is dependent on someone else’s paperwork to defend its own architecture.</p><p>With the boundary, the answer is a diagram and a pointed finger. <em>“Raw documents stay here. The sanitization layer strips identifying information here. This anonymized context crosses to the external provider. Nothing else.”</em></p><p>That is accountability that lives in the team’s codebase, audit logs, and architecture documentation — not in a vendor’s terms of service.</p><h3>Why Not Just Stay Fully Internal?</h3><p>If the concern is data leaving the boundary, why not run everything on internal infrastructure entirely?</p><p>Because capability matters. The frontier models — strong multi-step reasoning, reliable synthesis, coherent long-form generation — are mostly external. An internal model gives full control but may be materially less capable for the analytical tasks that create the most value.</p><p>The multi-LLM architecture does not force a choice between capability and compliance. It gives both. The cost is complexity in the middle — the sanitization layer, two-model integration, additional operational surface.</p><p>That complexity is worth naming honestly. It is also worth accepting.</p><h3>The Tradeoff Being Made</h3><p>More moving parts. That is the real cost.</p><p>A single provider is simpler to build, simpler to debug, simpler to explain to an engineer joining the team six months later. The sanitization layer adds latency, failure modes, and something else to monitor in production.</p><p>What gets traded for that simplicity: an architecture defensible in an audit, a data boundary explainable to legal and compliance, and a system where accountability for what data goes where sits entirely within the team’s control.</p><p>In most industries, that tradeoff is optional.</p><p>In financial services, it is not.</p><h3>Applicability</h3><p>This architecture is not specific to credit analysis or LLM deployment. The pattern applies anywhere sensitive data meets an externally-integrated AI system.</p><p>Three questions worth asking:</p><p><strong>1. Does the data carry regulatory or contractual sensitivity obligations?</strong> Healthcare, legal, financial, government — if yes, the sanitization boundary is not optional. It is the minimum viable architecture for handling that data responsibly with an external model.</p><p><strong>2. Can the data handling chain be fully explained to an auditor today?</strong> If the answer involves “and then it goes to the provider but they’re compliant” — there is a gap. The sanitization boundary closes it by making the data flow explicit and owned.</p><p><strong>3. Is the team using frontier models for capability reasons?</strong> If yes — and there are good reasons to — the two-zone model provides that capability without exposing sensitive data to an external boundary. Reasoning quality without the compliance exposure.</p><p>The pattern scales down too. Any team handling regulated data and integrating with external AI APIs should be thinking about where the sanitization boundary sits, even if the implementation is simpler than what is described here.</p><h3>Conclusion</h3><p>The multi-LLM architecture was not recommended because anyone distrusts LLM providers or enjoys engineering complexity for its own sake.</p><p>It was recommended because in a regulated environment, <strong>the question is not whether compliance will ask where the data goes — it is whether there will be a clean answer when they do.</strong></p><p>The single-provider approach is faster to build. Genuinely simpler. It might even be fine for a while until an audit, a data incident, a new regulatory requirement, or a new stakeholder surfaces the gap.</p><p>The sanitization boundary makes the architecture defensible before that moment arrives, rather than scrambled together after it.</p><p>Without this written down, the single-provider proposal resurfaces every time a vendor pitches a simpler solution or a new engineer sees two model integrations and asks why.</p><p>The ADR does not prevent that conversation. It makes sure it starts with the full picture.</p><p><em>Next in this series: </em><a href="https://medium.com/@siddharthdb/why-we-kept-the-human-in-the-loop-for-credit-decisions-7ecfad88deeb."><strong><em>ADR #3 — Why We Kept the Human in the Loop for Credit Decisions</em></strong></a><strong><em>.</em></strong><em> The AI autonomy line that has to be drawn in writing — and why the question keeps coming back.</em></p><p><em>If this was useful, follow for more. We write about software architecture, AI deployment in regulated industries, and the decisions engineering teams make that nobody documents.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5d3e0ffce4c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why I Recommend RAG Over Fine-Tuning for Credit Analysis]]></title>
            <link>https://medium.com/@siddharthdb/why-i-recommend-rag-over-fine-tuning-for-credit-analysis-82bd83bda470?source=rss-71e70cfa82d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/82bd83bda470</guid>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Siddharth B]]></dc:creator>
            <pubDate>Tue, 12 May 2026 04:58:29 GMT</pubDate>
            <atom:updated>2026-05-13T04:36:49.628Z</atom:updated>
            <content:encoded><![CDATA[<h3>ADR Series — Article 1 of 5</h3><p><em>This is the first in a five-part series on Architecture Decision Records — the most underrated tool in a senior engineer’s arsenal. Each article covers one real decision, the reasoning behind it, and why writing it down changes everything.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WweGHaxdFyBAi0CVkEgFNQ.png" /></figure><p>At some point in almost every AI project in financial services, someone says it.</p><p><em>“Should we just fine-tune the model?”</em></p><p>Honestly — it is a reasonable question. Fine-tuning sounds powerful. You take a general-purpose model, feed it your domain data, and it becomes <em>yours</em>. Smarter about your terminology. Faster on your specific tasks. Better, in theory, at understanding what a credit memo actually means versus what a quarterly earnings report means.</p><p>The problem is that “better” is doing a lot of work in that sentence. Better at <em>what</em>, exactly?</p><p>I have been deploying AI systems to financial institutions for a while now. And the fine-tuning conversation comes up every single time. Every time, the reasoning for not doing it is the same. And every time, without a written decision record, that reasoning has to be reconstructed from scratch when a new engineer joins, a vendor pitches, or a stakeholder asks.</p><p>This article is that reasoning — written down.</p><p>For credit analysis — reading financial statements, extracting risk signals, summarising borrower profiles — I recommend <strong>Retrieval-Augmented Generation (RAG)</strong> over fine-tuning.</p><p>Not because fine-tuning is bad. But because in a regulated environment, fine-tuning optimises for the wrong thing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tn-SZ7qhZKsUtwexfADrLQ.png" /></figure><h3>What Fine-Tuning Actually Gets You</h3><p>Fine-tuning makes a model <em>behave</em> differently. You train it on your domain data and it starts producing outputs that look more like your world. Responses feel native. Terminology feels right. A fine-tuned credit model will naturally use terms like debt-service coverage ratio, leverage multiple, and working capital cycle in ways that a general-purpose model sometimes fumbles.</p><p>That fluency is real and it is valuable.</p><p>But here is the thing nobody tells you upfront — <strong>a fine-tuned model cannot show its work.</strong></p><p>Ask it why it assessed a borrower as high-risk and it will give you an answer. A confident, fluent, well-structured answer. What it cannot give you is <em>a source</em>. There is no document it can point to. No line item from the financial statement. No specific ratio that triggered the conclusion. The reasoning is baked into the weights somewhere, distributed across billions of parameters, and it is simply not recoverable in any auditable way.</p><p>In most industries, that is inconvenient.</p><p>In financial services, that is a compliance problem.</p><h3>What RAG Gets You Instead</h3><p>RAG keeps the knowledge <em>outside</em> the model. When a query comes in, the system retrieves the relevant documents first — the actual financial statements, the actual credit policies, the actual borrower data — and hands them to the model along with the question. The model’s job is reasoning and synthesis, not memorisation.</p><p>What this means in practice:</p><p>Every answer has a source. Every conclusion traces back to a specific document, a specific passage, a specific number. When a credit analyst asks <em>“why did the system flag this borrower?”</em> — you can show them. Not explain in vague terms. Point to the exact line.</p><p>That is not just good engineering. In a regulated environment, that is the difference between a system that gets approved and a system that gets shelved.</p><h3>The Freshness Problem Nobody Mentions</h3><p>There is a second problem with fine-tuning that gets far less attention than it deserves.</p><p>Credit data changes. Constantly. New financial statements every quarter. Updated borrower profiles. Revised risk policies. Macroeconomic conditions that shift what a “healthy” debt ratio even looks like — a perfectly reasonable leverage multiple in a low-rate environment looks very different when rates move.</p><p>A fine-tuned model is a snapshot. To update it, you retrain. Retraining takes time, compute, and validation cycles — especially in a regulated context where you cannot just push a new model version and hope for the best.</p><p>RAG does not have this problem. You update the document store. The model works with new information the next time it retrieves. No retraining. No validation cycle. No lag between the world changing and your model knowing about it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dHqV6IdG14MBTbadWWTOzA.png" /></figure><h3>The Tradeoff You Are Actually Making</h3><p>I want to be honest about this part, because an ADR is supposed to capture tradeoffs — not just justifications.</p><p>Fine-tuning <em>can</em> produce more fluent, domain-native responses. For some use cases — internal tooling, first-draft generation, lower-stakes summarisation where explainability is not the primary concern — that fluency advantage is real and meaningful.</p><p>If your use case does not carry a compliance or audit obligation, fine-tuning may be the right call.</p><p>But for credit analysis specifically, where every output may be reviewed by a regulator, questioned by an auditor, or used to make a decision that affects a real borrower — fluency is not the priority. <strong>Traceability is.</strong></p><p>RAG wins on traceability. It is not close.</p><h3>Applicability</h3><p>This reasoning applies beyond credit analysis. Ask yourself three questions about your use case:</p><p><strong>1. Does someone need to explain the output?</strong> If a decision, recommendation, or classification produced by the AI may need to be justified to a regulator, auditor, manager, or customer — you need traceability. RAG gives you that. Fine-tuning does not.</p><p><strong>2. Does the underlying data change frequently?</strong> If the knowledge your model needs to reason over updates on a weekly, monthly, or quarterly cycle — the retraining overhead of fine-tuning becomes an operational liability. RAG adapts as fast as you update the document store.</p><p><strong>3. Is domain fluency more important than source transparency?</strong> If your use case is primarily generative — drafting, summarising, translating — and the output does not carry accountability obligations, fine-tuning’s fluency advantage may genuinely matter more than traceability. That is a legitimate choice. Just write it down.</p><p>The pattern holds outside financial services too. Healthcare AI where diagnostic reasoning needs to be explainable. Legal tech where every conclusion should cite a statute or precedent. Any domain where <em>“the model said so”</em> is not a sufficient answer.</p><p>I did not choose RAG because fine-tuning is bad technology. Fine-tuning is genuinely powerful and has the right use cases.</p><p>I chose RAG because in a compliance-heavy domain, <strong>an answer you cannot explain is not actually an answer.</strong> The model’s fluency does not matter if the output cannot be traced, sourced, and defended when someone with a clipboard asks you to justify it.</p><p>What made this worth writing down is not the decision itself. It is what happens without the record.</p><p>Without it, the fine-tuning conversation restarts every time a new engineer joins and sees the retrieval pipeline. Every time a vendor pitches their fine-tuned domain model. Every time a benchmark comparison shows fine-tuning scoring higher on raw accuracy and someone wonders if we made the wrong call.</p><p>The ADR does not shut those conversations down. It makes sure they start from the right place.</p><p><em>Next in this series: </em><a href="https://medium.com/@siddharthdb/why-we-recommend-a-multi-llm-architecture-over-a-single-provider-b5d3e0ffce4c"><strong><em>ADR #2 — Why We Recommend a Multi-LLM Architecture Over a Single Provider</em></strong></a><strong><em>.</em></strong><em> The decision compliance will force anyway — better to make it deliberately.</em></p><p><em>If this was useful, follow for more. I write about software architecture, AI deployment in regulated industries.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82bd83bda470" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Asking ‘Cloud or On-Premise?’ — Here’s the Question That Actually Matters]]></title>
            <link>https://medium.com/@siddharthdb/stop-asking-cloud-or-on-premise-here-s-the-question-that-actually-matters-6bc665c19d1e?source=rss-71e70cfa82d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bc665c19d1e</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[enterprise-architecture]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <dc:creator><![CDATA[Siddharth B]]></dc:creator>
            <pubDate>Mon, 05 Jan 2026 16:42:45 GMT</pubDate>
            <atom:updated>2026-01-06T05:26:58.431Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stop Asking ‘Cloud or On-Premise?’ — Here’s the Question That Actually Matters</h3><p>I’ve been in enterprise technology for over two decades now, and if there’s one question that never seems to go away, it’s this:</p><p><em>“Should we deploy on cloud or on-premise?”</em></p><p>It’s asked in boardrooms, architecture reviews, and vendor evaluations. It’s treated as THE fundamental question for any new system deployment.</p><p>But here’s what I’ve learned: <strong>Most of the time, it’s the wrong question entirely.</strong></p><p>Let me tell you a story about a recent project that proved this point — and maybe save you from a multi-million dollar mistake in the process.</p><h3>The Setup: A Working System Meets a New Requirement</h3><p>A few months back, we delivered a Credit Appraisal AI system for a financial services client. The system was straightforward in its brilliance:</p><ul><li>One Large Language Model (LLM) analyzing loan application documents</li><li>Strict guardrails ensuring it only extracted data, never made decisions</li><li>All processing happening in a controlled environment</li><li>Documents containing NPI (Non-Public Information), PII (Personally Identifiable Information), and confidential financial data</li></ul><p>The client loved it. It was working exactly as designed.</p><p>Then came the email that changed everything:</p><p><em>“Can we enhance the appraisal by pulling external market data and industry benchmarks?”</em></p><p>Sounds reasonable, right? Just add another data source, feed it to the LLM, done.</p><p><strong>Except there’s a massive trap hiding in that innocent requirement.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bP49UyPCJJZpm89WSpNI9A.png" /></figure><h3>The Trap: When Security Meets Enhancement</h3><p>Here’s the problem: Our existing LLM had access to highly confidential documents. Now we wanted it to reach out to external sources for additional context.</p><p>But think about what that means:</p><p><strong>Scenario 1:</strong> We send confidential document data to an external API or another LLM for enrichment.<br> <strong>Result:</strong> Data leak. Compliance nightmare. Career-ending decision.</p><p><strong>Scenario 2:</strong> We keep everything isolated, don’t enrich the data.<br> <strong>Result:</strong> Client doesn’t get the feature they need. Competitive disadvantage.</p><p><strong>Scenario 3:</strong> We bring everything on-premise so we control all data flows.<br> <strong>Result:</strong> This is where the client’s mind went immediately.</p><p>And that’s when they asked THE question:</p><p><em>“Should we move everything on-premise to keep it secure?”</em></p><h3>The Conversation That Changed Everything</h3><p>I remember the stakeholder meeting clearly. The client’s IT head was already mentally calculating hardware costs. The compliance officer was nodding approvingly. The CFO was wincing at the budget numbers being thrown around.</p><p>Before anyone could get too committed, I asked a question:</p><p><strong>“When can you have the on-premise infrastructure ready for deployment?”</strong></p><p>The IT head paused. “Well, we’d need to procure servers, set up the data center expansion, get security clearances for the hardware, configure everything…”</p><p>“So, timeline?”</p><p>“Realistically? 12 to 18 months. Maybe more with procurement cycles.”</p><p><strong>Silence.</strong></p><p>The business stakeholder who’d requested the feature spoke up: “We need this for next quarter’s loan cycle. We’ve already committed to the board.”</p><p>And there it was. The classic enterprise trap:</p><ul><li><strong>Security thinking</strong> says: On-premise = Safe</li><li><strong>Business reality</strong> says: We can’t wait 18 months</li><li><strong>Budget reality</strong> says: Multi-million dollar infrastructure for one feature?</li></ul><p>This is the moment where most projects either die, get watered down, or make compromised decisions that everyone regrets later.</p><p><strong>But what if we were asking the wrong question entirely?</strong></p><h3>The Reframe: What Are We Actually Trying to Protect?</h3><p>I drew a simple diagram on the whiteboard (well, it was a virtual whiteboard, but you get the idea):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4mlnu1WrzG4lUR2JibP0GA.png" /></figure><p>“Let me ask a different question,” I said. “Do we need to send confidential document data to fetch external information?”</p><p>The compliance officer leaned forward. “Absolutely not.”</p><p>“Then what if we don’t?”</p><p><strong>This is where architecture beats infrastructure.</strong></p><h3>The Solution: Multi-LLM Architecture with Data Isolation</h3><p>Instead of asking “where should we deploy?”, we asked “how should we isolate data flows?”</p><p>Here’s the architecture we designed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l23ik1Rd17D4HBUke_j3ig.png" /></figure><p>Let me break down what’s happening here:</p><h3>LLM #1: The Vault</h3><p><strong>Purpose:</strong> Analyze confidential documents without ever exposing them externally.</p><p><strong>What it sees:</strong></p><ul><li>Full loan application documents</li><li>Financial statements</li><li>Proprietary business information</li><li>Everything sensitive</li></ul><p><strong>What it outputs:</strong></p><ul><li>Structured facts: “Applicant: XYZ Corp”</li><li>Quantitative data: “Revenue: $10M range”</li><li>Assessment metrics</li><li><strong>NOT the source documents</strong></li></ul><p><strong>Pseudo-code concept:</strong></p><pre>function analyzeSensitiveDocuments(documents):<br>    facts = llm_vault.extract_structured_data(documents)<br>    sanitized_facts = remove_sensitive_patterns(facts)<br>    return sanitized_facts  // No document content leaves</pre><h3>The Data Contract: The Bridge</h3><p>This is the critical piece most people miss. It’s not just about having two LLMs — it’s about <strong>what passes between them</strong>.</p><p><strong>The contract defines:</strong></p><ul><li>Company name (public information anyway)</li><li>Industry classification code</li><li>Loan amount range (not exact figures)</li><li>Request type identifier</li></ul><p><strong>What’s explicitly NOT passed:</strong></p><ul><li>Document contents</li><li>Specific financial metrics</li><li>Names of individuals</li><li>Internal assessments</li></ul><p>Think of it like this: When you google a company name, you’re not exposing confidential data. That’s all LLM #2 gets to see.</p><h3>LLM #2: The Researcher</h3><p><strong>Purpose:</strong> Enrich the analysis with external context without ever seeing confidential documents.</p><p><strong>What it sees:</strong></p><ul><li>Company name: “XYZ Corp”</li><li>Industry: “Manufacturing — Automotive Components”</li><li>Public query parameters</li></ul><p><strong>What it does:</strong></p><ul><li>Searches public databases</li><li>Pulls market trends</li><li>Fetches industry benchmarks</li><li>Aggregates competitor data</li></ul><p><strong>What it never sees:</strong></p><ul><li>The actual loan documents</li><li>Internal financial details</li><li>Proprietary information</li></ul><p><strong>Pseudo-code concept:</strong></p><pre>enrichWithExternalData(safe_identifiers):<br>    company_name = safe_identifiers.company<br>    industry = safe_identifiers.industry<br>    <br>    market_data = fetch_public_market_data(industry)<br>    company_context = fetch_public_company_info(company_name)<br>    benchmarks = fetch_industry_benchmarks(industry)<br>    <br>    return synthesize_context(market_data, company_context, benchmarks)</pre><h3>The Assembly: Bringing It Together</h3><p>The final report generator takes:</p><ul><li>Sanitized facts from LLM #1 (without source documents)</li><li>Enriched context from LLM #2 (public data only)</li><li>Combines them into the final appraisal</li></ul><p>Neither LLM ever needs to know what the other saw. The confidential data never leaves the secure zone.</p><h3>The Infrastructure Decision: Cloud Wins (But Not How You’d Think)</h3><p>Now, back to the original question: Cloud or on-premise?</p><p>With this architecture, the answer became clear:</p><p><strong>For LLM #1 (The Vault):</strong></p><ul><li>Cloud deployment with VPC isolation</li><li>Data residency controls (documents never leave specified region)</li><li>Encryption at rest and in transit</li><li>VPN tunnel from client’s on-premise document storage</li></ul><p><strong>For LLM #2 (The Researcher):</strong></p><ul><li>Standard cloud deployment</li><li>Standard API security</li><li>No special data controls needed (it only handles public data anyway)</li></ul><p><strong>The cost comparison:</strong></p><p><strong>Full On-Premise</strong></p><ul><li>Cost: $2M+ infrastructure investment</li><li>Timeline: 12–18 months</li><li>Risk: Delayed business value, long procurement cycles</li></ul><p><strong>Full Cloud (naive approach)</strong></p><ul><li>Cost: $150K/year operational</li><li>Timeline: 2–3 months</li><li>Risk: Security and compliance violations</li></ul><p><strong>Multi-LLM Cloud (architected approach)</strong> ✓</p><ul><li>Cost: $200K/year operational</li><li>Timeline: 2–3 months</li><li>Risk: Minimal (data flows isolated by design)</li></ul><p>The client chose the architected cloud approach. Not because “cloud is better” — but because <strong>the architecture made cloud viable</strong>.</p><h3>Implementation Reality: Making It Actually Work</h3><p>Theory is great. But here’s what it took to make this real:</p><h3>1. VPN Tunnel Setup</h3><p><strong>The Data Flow:</strong></p><pre>Client Data Center ←→ [Encrypted VPN] ←→ Cloud VPC (LLM #1 Zone)<br>         |<br>   Document Storage → Secure Transfer → LLM #1 Processing</pre><p><strong>Key consideration:</strong> Latency. Documents could be large (loan applications with financial statements). We had to balance security (encryption) with performance (document processing speed).</p><p><strong>Solution:</strong> Pre-stage documents in cloud VPC during off-hours, process during business hours. Reduced real-time transfer bottleneck.</p><h3>2. Data Sanitization Layer</h3><p>This was the most critical piece. We built a middleware that:</p><p><strong>Input to Sanitization Layer:</strong></p><pre>{<br>  &quot;applicant&quot;: &quot;XYZ Manufacturing Corp&quot;,<br>  &quot;revenue&quot;: &quot;$10,234,567&quot;,<br>  &quot;profit_margin&quot;: &quot;14.3%&quot;,<br>  &quot;loan_amount&quot;: &quot;$2,500,000&quot;,<br>  &quot;purpose&quot;: &quot;Equipment expansion&quot;,<br>  &quot;collateral_details&quot;: [sensitive details]<br>}</pre><p><strong>Output to LLM #2:</strong></p><pre>{<br>  &quot;company_name&quot;: &quot;XYZ Manufacturing Corp&quot;,<br>  &quot;industry_code&quot;: &quot;NAICS 336&quot;,<br>  &quot;revenue_range&quot;: &quot;10M-50M&quot;,<br>  &quot;loan_category&quot;: &quot;equipment_financing&quot;<br>}</pre><p>Notice what disappeared? Exact figures, specific details, anything that could be traced back to the confidential application.</p><h3>3. Compliance Documentation</h3><p>The client&#39;s compliance team needed to sign off. We documented:</p><ul><li>Data flow diagrams (like the ones above)</li><li>Security controls at each boundary</li><li>Encryption standards (TLS 1.3 in transit, AES-256 at rest)</li><li>Access logs and audit trails</li><li>DPDP Act 2023 compliance mapping (for their India operations)</li></ul><p><strong>The compliance officer’s question:</strong> “How do we know confidential data isn’t leaking?”</p><p><strong>Our answer:</strong> “Because LLM #2 has no access to the network zone where LLM #1 operates. It’s not just policy — it’s network isolation.”</p><p>That sealed it.</p><h3>The Lesson: Architecture Before Infrastructure</h3><p>After this project wrapped, I kept thinking about all the times I’ve seen teams jump to infrastructure decisions too quickly.</p><p>Here’s what I wish more people understood:</p><h3>The Wrong Question Pattern</h3><p>❌ “Should we use cloud or on-premise?”<br>❌ “Which vendor should we choose?”<br>❌ “What’s the most secure option?”</p><p>These questions assume the problem is about WHERE to deploy.</p><h3>The Right Question Pattern</h3><p>✅ “What are we actually trying to protect?”<br>✅ “Can we isolate data flows to reduce the security surface?”<br>✅ “What’s the minimum viable infrastructure to solve the real constraint?”</p><p><strong>The pattern:</strong> Question the constraint first, then architect around it, THEN choose infrastructure.</p><h3>When On-Premise IS The Right Answer</h3><p>I’m not saying cloud is always better. There are legitimate cases for on-premise:</p><ul><li>Regulatory requirements with no cloud exceptions (some government contracts)</li><li>Existing infrastructure that’s already paid for and underutilized</li><li>Data sovereignty requirements that can’t be met by cloud regions</li><li>Air-gapped environments by design (defense, critical infrastructure)</li></ul><p><strong>But even then</strong> — question whether the ENTIRE system needs to be on-premise, or just specific components.</p><h3>The Real Skill</h3><p>The real skill in enterprise architecture isn’t choosing between options A and B.</p><p><strong>It’s recognizing when options A and B are both wrong, and option C hasn’t been invented yet.</strong></p><p>In our case:</p><ul><li><strong>Option A:</strong> Full cloud (security risk)</li><li><strong>Option B:</strong> Full on-premise (timeline/cost prohibitive)</li><li><strong>Option C:</strong> Multi-LLM architecture that made cloud viable</li></ul><p>Option C didn’t exist until we questioned the premise of the question itself.</p><h3>Final Thoughts</h3><p>If you take one thing from this article, let it be this:</p><p><strong>The next time someone asks you “cloud or on-premise?”, pause.</strong></p><p>Don’t answer immediately. Instead, ask:</p><ol><li>What’s the actual constraint we’re solving for?</li><li>Is it truly about infrastructure location, or is it about data flows?</li><li>Can architecture solve this before infrastructure does?</li></ol><p>You might find, like we did, that the answer isn’t where to deploy.</p><p><strong>It’s how to design.</strong></p><p>And that’s a much more interesting question.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bc665c19d1e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>