<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manish Goyal on Medium]]></title>
        <description><![CDATA[Stories by Manish Goyal on Medium]]></description>
        <link>https://medium.com/@manishgoyal05?source=rss-6ef7ca08a6a9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JXOhU4ZR-mQiLfS2nE5ZCg.jpeg</url>
            <title>Stories by Manish Goyal on Medium</title>
            <link>https://medium.com/@manishgoyal05?source=rss-6ef7ca08a6a9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 14:27:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@manishgoyal05/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building DocRAG: An Open-Source Java RAG Library with LangChain4j, Docling, Ollama, and pgvector]]></title>
            <link>https://manishgoyal05.medium.com/building-docrag-an-open-source-java-rag-library-with-langchain4j-docling-ollama-and-pgvector-43e83f1e1dde?source=rss-6ef7ca08a6a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/43e83f1e1dde</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Manish Goyal]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 22:49:13 GMT</pubDate>
            <atom:updated>2026-10-20T22:49:13.724Z</atom:updated>
            <content:encoded><![CDATA[<h4>How document parsing, hybrid retrieval, LLM orchestration, and Java validation come together for company research.</h4><figure><img alt="Conceptual illustration of documents becoming source excerpts and a research answer with source references." src="https://cdn-images-1.medium.com/max/1024/1*GXK3FUMxASPJwGYv_7H5GQ.png" /><figcaption>Concept illustration: company documents become source-linked research.</figcaption></figure><p>Following a company means connecting information scattered across documents. A report contains the numbers. A presentation adds context. Conference-call material may explain management’s expectations, while a notice or announcement changes what you need to investigate next.</p><p>I’m building DocRAG to make that research easier: ask a question, follow it up naturally, and inspect the evidence behind the answer. I also want that capability to fit inside an existing Java application.</p><p>The broader goal is a predefined company watchlist that fetches relevant material directly from the NSE website, including conference-call documents, presentations, notices, and announcements. That ingestion work is planned. The current implementation starts with uploaded PDFs and company/period-based research.</p><p><a href="https://github.com/mgoyal05/doc-rag">DocRAG is open source under Apache 2.0</a>. Its core targets Java 21, with optional adapters for document extraction, Spring, the browser widget, and observability.</p><p>The interesting engineering question is how those pieces cooperate. How does a document become searchable evidence? How does a follow-up retain its meaning? And what should happen when the model produces a plausible answer that the evidence cannot support?</p><h3>LangChain4j and Ollama: give model calls defined jobs</h3><p>Ollama serves <a href="https://docs.ollama.com/api/embed">embeddings</a> and <a href="https://docs.ollama.com/api/chat">chat inference</a>. The example uses embeddinggemma for retrieval vectors and qwen2.5:14b for language tasks. LangChain4j supplies Java integration, response schemas, and <a href="https://docs.langchain4j.dev/tutorials/ai-services/">AI Services</a>. Calls stay stateless; PostgreSQL stores conversations.</p><p>Models infer document metadata, resolve scope, generate claims, and assess evidence support; Java validates their outputs. Generation receives the resolved question, validated scope, and retrieved passages: retrieval-augmented generation (RAG).</p><p>Registered tools require scope and argument checks before execution, limited to four calls across two tool rounds, then a final response. LangChain4j’s <a href="https://docs.langchain4j.dev/tutorials/guardrails/">experimental guardrails</a> validate inputs and outputs; authorization must precede execution.</p><p>Ollama requires budgeting weights, memory, context, loading time, and throughput. Hosted inference changes those responsibilities; fitting a model in memory does not establish acceptable latency.</p><h3>Docling: preserve structure before creating embeddings</h3><p>Flattening PDFs can detach values from table headers or passages from headings. DocRAG’s Docling Java adapter calls Docling Serve with OCR, table processing, and <a href="https://docling-project.github.io/docling/concepts/chunking/">hybrid chunking</a>: structural boundaries refined by token-aware splitting and compatible merging.</p><p>The default budget is 512 tokens. Chunks retain headings, pages, original excerpts for quotations, and separate contextual embedding text for search.</p><p>Fixed-size splitting is simpler; structure-aware extraction adds a service and needs evaluation on scans and tables. Reported partial conversions or missing page coverage are rejected, sometimes requiring preprocessing.</p><p>Embeddings are computed and validated before a transaction publishes the complete document version. Failure preserves an existing usable version, excluding partially published replacements from retrieval.</p><h3>PostgreSQL and pgvector: retrieve through meaning and words</h3><p>Consider a fictional ACME report:</p><blockquote><em>What revenue did ACME report for the quarter ending September 2025?</em></blockquote><p>Embeddings represent text numerically for similarity search. Ingestion stores chunk vectors; questions embed only the query. The default embeddinggemma profile has 768 dimensions.</p><p>Retrieval filters by validated company, period, and active document version, then combines:</p><ul><li><strong>Lexical search:</strong> PostgreSQL’s plainto_tsquery and ts_rank_cd use English normalization to match words and rank results. <a href="https://www.postgresql.org/docs/current/textsearch-controls.html">PostgreSQL reference</a>.</li><li><strong>Vector search:</strong> pgvector’s &lt;=&gt; operator ranks by cosine distance for semantic similarity when wording differs. <a href="https://github.com/pgvector/pgvector">pgvector reference</a>.</li></ul><p>Reciprocal rank fusion (RRF) sums 1 / (60 + rank) across both lists, avoiding incompatible raw scores. Top seeds gain adjacent chunks from the same version and section; context selection balances requested targets within a bounded evidence budget.</p><p>PostgreSQL coordinates vectors, text, metadata, versions, and conversations in one database. A dedicated vector service suits other workloads; this design puts index tuning, filtered retrieval, and capacity planning on PostgreSQL.</p><p>After the ACME question, “What about profit?” needs a subject and period. A structured planning call uses bounded history to resolve a standalone question; Java validates scope before retrieval.</p><p>The resolved question drives embedding, lexical search, tool selection, and generation. Ambiguity prompts clarification; absent required reports yield insufficient evidence. Previous assistant text supplies context, never source evidence.</p><figure><img alt="DocRAG hybrid retrieval: a resolved question and validated scope feed lexical search and query embedding, followed by vector search, rank fusion, neighboring chunks, and bounded evidence." src="https://cdn-images-1.medium.com/max/1024/1*fbN1qPqdT1KrSu8TeJTCWg.png" /><figcaption>The resolved question feeds lexical and vector search; fused results become bounded evidence.</figcaption></figure><h3>Java validation: decide which answers can be accepted</h3><p>Generation returns structured claims, citation IDs, and optional facts for Java validation before display.</p><p>For ACME comparisons, optional financial tools read host-supplied verified snapshots; PDF uploads do not create that trusted adapter. Tools check currency, units, reporting basis, and dates, then calculate with BigDecimal. This excerpt shows the baseline rule:</p><pre>BigDecimal delta = current.subtract(prior);<br>result.put(&quot;absoluteChange&quot;, delta);<br>if (prior.signum() &gt; 0)<br>    result.put(&quot;percentChange&quot;, delta.multiply(BigDecimal.valueOf(100))<br>            .divide(prior, 6, RoundingMode.HALF_UP));</pre><p>Percentage change is omitted for zero or negative baselines: a Java policy.</p><p>Document quantities use placeholders such as Revenue was {{F1}}. Each fact carries evidence, metric, value, currency, unit, basis, and quote. Java checks source excerpts, resolves citations and fact references, and renders quantities.</p><p>A cited revenue figure can still be mislabeled as profit. Nonempty claims receive a separate tool-free verification call against cited evidence. Every claim needs a complete supported verdict. Unsupported claims, missing verdicts, malformed output, or verifier unavailability withhold the entire answer; unchecked drafts are not persisted as accepted results.</p><p>Deterministic checks and a fallible semantic judgment control acceptance. This adds inference cost and can reject valid answers, especially with incomplete table context or units. A verifier can share generator mistakes; passing cannot authenticate documents or certify truth. Clarification, insufficient evidence, and withheld answers remain explicit outcomes.</p><figure><img alt="DocRAG answer validation: proposed claims pass Java checks and tool-free verification before persistence; empty claims yield insufficient evidence, while invalid or unsupported output is withheld." src="https://cdn-images-1.medium.com/max/1024/1*HNHlHywBnjPArK_I9T9iuw.png" /><figcaption>Java checks and a tool-free verifier control acceptance; failures withhold the draft.</figcaption></figure><h3>A portable core, with application behavior around it</h3><p>The Spring-independent ResearchClient supports uploads, messages, conversations, and jobs. Extraction, financial-data, and telemetry interfaces keep it portable; optional Spring/widget adapters support web integration.</p><p>Hosts authenticate users; conversations are private, while uploaded evidence is shared.</p><p>A persisted, bounded queue feeds one worker; each conversation allows one active turn. Cancellation/restart handling rejects stale results; interrupted jobs need explicit retry. Run one instance per schema/job directory; horizontal execution requires coordination.</p><p>The widget receives SSE progress/completion, then fetches persisted results with citations and PDF links. Answer text is available after validation; unchecked tokens never stream. Micrometer/OpenTelemetry observes phases without capturing documents, prompts, or answers in the research adapter.</p><p>Published artifacts include <strong>DocRAG Core</strong> (doc-rag-core) and <strong>DocRAG Spring Boot</strong> (doc-rag-spring-boot), version 0.1.1. <a href="https://central.sonatype.com/artifact/io.github.mgoyal05/doc-rag-core/0.1.1">Maven Central</a>:</p><pre>&lt;dependency&gt;<br>    &lt;groupId&gt;io.github.mgoyal05&lt;/groupId&gt;<br>    &lt;artifactId&gt;doc-rag-core&lt;/artifactId&gt;<br>    &lt;version&gt;0.1.1&lt;/version&gt;<br>&lt;/dependency&gt;</pre><p><a href="https://github.com/mgoyal05/doc-rag#installation-from-a-clean-checkout">Integration instructions</a> cover adapters and runtime services.</p><h3>What is established, and what comes next</h3><p>The repository includes tests for retrieval and lifecycle behavior, numerical validation, unsupported claims, and verifier failures. Synthetic Docling/Ollama HTTP simulations exercise the client pipeline. These establish different things from reviewed answer accuracy on real company documents.</p><p>Evaluation still needs to separate retrieval coverage, generated-answer correctness, numerical correctness, and operational performance. Native database workloads and real extraction/model behavior need their own checks.</p><p>My next step is watchlist-driven NSE ingestion across company material, extending beyond financial reports. That work needs source tracking, update detection, duplicate handling, and metadata suited to different document types.</p><p>The current foundation makes the division of responsibility concrete: AI interprets questions and proposes evidence-based language; retrieval selects source material; Java controls scope, calculations, execution, and acceptance. That division is what makes DocRAG a reusable research capability worth building on.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43e83f1e1dde" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Fifth Integration Changes Everything]]></title>
            <link>https://manishgoyal05.medium.com/the-fifth-integration-changes-everything-3913c3c21ecb?source=rss-6ef7ca08a6a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/3913c3c21ecb</guid>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[adapter-pattern]]></category>
            <dc:creator><![CDATA[Manish Goyal]]></dc:creator>
            <pubDate>Wed, 19 Aug 2026 14:12:58 GMT</pubDate>
            <atom:updated>2026-08-19T14:12:58.776Z</atom:updated>
            <content:encoded><![CDATA[<h3>How we designed a partner integration platform when every partner has a different API, workflow, and definition of success</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RSA4iJiiREihFn3GW-sm9A.png" /></figure><p>The first partner integration usually looks straightforward.</p><p>Transform an order into the partner’s format, call an API, store the response, and wait for a webhook.</p><p>The second integration introduces a few conditions. The third requires coupon validation. Some send reliable webhooks; others expect us to poll.</p><p>At that point, the challenge is no longer API integration.</p><p><strong>It is the controlled management of variation.</strong></p><p>That distinction matters because an integration platform should not merely make different APIs callable. It should prevent those differences from leaking into the ordering domain.</p><blockquote><strong><em>Architecture is what remains stable while partners, protocols, and failure modes continue to change.</em></strong></blockquote><h3>The boundary I wanted to protect</h3><p>The ordering system should not understand partner-specific field names, authentication schemes, transaction APIs, webhook payloads, or retry rules.</p><p>This leads to a layered architecture that enforces strict separation of concerns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m6q9qrM_fBjJEpWsTeMxfw.png" /></figure><h3>High-Level Design (HLD): The Integration Platform</h3><p>At a high level, the system is composed of five major layers:</p><pre>                +----------------------+<br>                |   Ordering Domain    |<br>                | (Core Business Logic)|<br>                +----------+-----------+<br>                           |<br>                           v<br>                +----------------------+<br>                | Canonical Contract   |<br>                | (Domain Model Layer) |<br>                +----------+-----------+<br>                           |<br>                           v<br>                +----------------------+<br>                | Workflow Orchestrator|<br>                | (State Machine +     |<br>                | Capability Planner)   |<br>                +----------+-----------+<br>                           |<br>                           v<br>                +----------------------+<br>                | Partner Adapters     |<br>                | (Anti-Corruption     |<br>                | Layer)               |<br>                +----------+-----------+<br>                           |<br>                           v<br>                +----------------------+<br>                | External Partners    |<br>                | (APIs/Webhooks)      |<br>                +----------------------+</pre><p>Each layer answers a different question:</p><ul><li><strong>Ordering Domain</strong> → <em>What is an order?</em></li><li><strong>Canonical Contract</strong> → <em>What does our platform guarantee?</em></li><li><strong>Workflow Orchestrator</strong> → <em>What steps are required?</em></li><li><strong>Adapters</strong> → <em>How does a partner speak?</em></li><li><strong>External APIs</strong> → <em>What does the partner expect?</em></li></ul><p>This separation ensures that partner volatility never contaminates the core domain.</p><h3>Low-Level Design (LLD): Core Components</h3><h3>1. Domain Model (Canonical Layer)</h3><p>This is the stable representation of business concepts.</p><pre>record CanonicalOrder(<br>    String orderId,<br>    String storeId,<br>    List&lt;OrderItem&gt; items,<br>    Money total,<br>    OrderStatus status<br>) {}</pre><p>This model is:</p><ul><li>Partner-agnostic</li><li>Immutable</li><li>Versioned carefully</li><li>The single source of truth for workflows</li></ul><h3>2. Workflow Orchestrator (State Machine Engine)</h3><p>At the heart of the system is a <strong>state machine-based orchestrator</strong>.</p><h4>State Model</h4><pre>CREATED<br>  ↓<br>OFFER_VERIFIED<br>  ↓<br>PRICE_VERIFIED<br>  ↓<br>PARTNER_TRANSACTION_CREATED<br>  ↓<br>SUBMITTED<br>  ↓<br>ACCEPTED / REJECTED / CANCELLED</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K3BwIR-vKvf8n-tXCO_tWg.png" /></figure><h4>State Transition Engine</h4><pre>class OrderWorkflowEngine {<br>    void transition(OrderWorkflow workflow, Event event) {<br>        switch (workflow.state()) {<br>            case CREATED -&gt; handleCreated(workflow, event);<br>            case OFFER_VERIFIED -&gt; handleOfferVerified(workflow, event);<br>            case PRICE_VERIFIED -&gt; handlePriceVerified(workflow, event);<br>            case SUBMITTED -&gt; handleSubmitted(workflow, event);<br>            default -&gt; throw new IllegalStateException(&quot;Invalid transition&quot;);<br>        }<br>    }<br>}</pre><p>This ensures:</p><ul><li>Deterministic transitions</li><li>Testable workflows</li><li>No hidden side effects</li></ul><h3>3. Capability-Based Workflow Planner</h3><p>Instead of hardcoding partner flows, we model them as <strong>capabilities</strong>.</p><pre>OFFER_VERIFICATION<br>PRICE_VERIFICATION<br>TRANSACTION_CREATION<br>ORDER_SUBMISSION<br>ORDER_CANCELLATION<br>WEBHOOK_STATUS_UPDATE</pre><h4>Partner Configuration Example</h4><pre>partnerA:<br>  capabilities:<br>    - OFFER_VERIFICATION<br>    - PRICE_VERIFICATION<br>    - TRANSACTION_CREATION<br>    - ORDER_SUBMISSION</pre><h4>Workflow Builder</h4><pre>class WorkflowPlanner {<br>    List&lt;WorkflowStep&gt; build(PartnerCapabilities caps) {<br>        List&lt;WorkflowStep&gt; steps = new ArrayList&lt;&gt;();<br>        if (caps.has(OFFER_VERIFICATION))<br>            steps.add(new OfferVerificationStep());<br>        if (caps.has(PRICE_VERIFICATION))<br>            steps.add(new PriceVerificationStep());<br>        steps.add(new OrderSubmissionStep());<br>        return steps;<br>    }<br>}</pre><p>This avoids:</p><ul><li>Giant if-else trees</li><li>Partner-specific branching in core logic</li></ul><h3>4. Partner Adapter Layer (Anti-Corruption Layer)</h3><p>This is where external chaos is contained.</p><h4>Responsibility:</h4><ul><li>Translate canonical → partner format</li><li>Translate partner → canonical events</li><li>Handle authentication</li><li>Normalize errors</li></ul><h4>Example Adapter</h4><pre>@Component<br>final class PartnerAAdapter {<br>    PartnerARequest toRequest(CanonicalOrder order) {<br>        return new PartnerARequest(<br>            order.getOrderId(),<br>            order.getStoreId(),<br>            order.getItems().stream()<br>                .map(i -&gt; new PartnerAItem(i.sku(), i.quantity()))<br>                .toList(),<br>            order.total().amountInCents()<br>        );<br>    }<br>    CanonicalEvent toEvent(PartnerAWebhook webhook) {<br>        return switch (webhook.status()) {<br>            case &quot;ACCEPTED&quot; -&gt; new OrderAcceptedEvent(...);<br>            case &quot;REJECTED&quot; -&gt; new OrderRejectedEvent(...);<br>            default -&gt; new OrderUnknownEvent(...);<br>        };<br>    }<br>}</pre><h3>Critical Rule:</h3><blockquote><em>Adapters must never contain business decisions. They only translate.</em></blockquote><h3>Kafka + Database: Dual-System Consistency Model</h3><p>We explicitly separate responsibilities:</p><ul><li><strong>Kafka → event transport</strong></li><li><strong>Database → source of truth</strong></li><li><strong>Orchestrator → decision maker</strong></li></ul><h3>Event Flow Architecture</h3><pre>Order Event<br>   ↓<br>Workflow Engine<br>   ↓<br>DB State Update<br>   ↓<br>Outbox Event Insert<br>   ↓<br>CDC (Debezium)<br>   ↓<br>Kafka Topic<br>   ↓<br>Consumers</pre><h3>Transactional Outbox Pattern</h3><p>We avoid dual-write problems using the outbox pattern.</p><pre>@Transactional<br>public void markSubmitted(OrderWorkflow workflow, String partnerOrderId) {<br>    workflow.markSubmitted(partnerOrderId);<br>    workflowRepository.save(workflow);<br>    outboxRepository.save(<br>        OutboxEvent.of(<br>            workflow.orderId(),<br>            &quot;ORDER_SUBMITTED&quot;,<br>            new OrderSubmitted(workflow.orderId(), partnerOrderId)<br>        )<br>    );<br>}</pre><p>This ensures:</p><ul><li>DB and event stream never diverge</li><li>No lost events</li><li>Replayability</li></ul><h3>Idempotency: Designing for Reality, Not Theory</h3><p>Kafka is <strong>at-least-once</strong>, not exactly-once.</p><p>So every handler must be idempotent.</p><h3>Idempotency Table</h3><pre>CREATE TABLE processed_events (<br>    event_id VARCHAR PRIMARY KEY,<br>    processed_at TIMESTAMP<br>);</pre><h3>Consumer Logic</h3><pre>@Transactional<br>public void handle(PartnerOrderAccepted event) {<br>    if (!repo.insertIfAbsent(event.eventId())) {<br>        return; // duplicate<br>    }<br>    workflowRepository.markAccepted(<br>        event.orderId(),<br>        event.partnerOrderId()<br>    );<br>}</pre><h3>Webhooks as Anti-Corruption Boundary</h3><p>Partner webhooks are never trusted directly.</p><h3>Processing Pipeline</h3><pre>Webhook<br>  ↓<br>Authentication Layer<br>  ↓<br>Raw Payload Storage<br>  ↓<br>Deduplication<br>  ↓<br>Translation Layer<br>  ↓<br>Workflow Validation<br>  ↓<br>State Transition<br>  ↓<br>Outbox Event</pre><h3>Key Design Decision:</h3><blockquote><em>Webhooks are </em>inputs<em>, not </em>state authority<em>.</em></blockquote><p>This prevents:</p><ul><li>Out-of-order updates</li><li>Duplicate callbacks</li><li>Invalid transitions</li></ul><h3>Scheduling System Design</h3><p>We separate scheduling from execution.</p><pre>Quartz Scheduler<br>      ↓<br>Job Trigger<br>      ↓<br>Kafka Command Event<br>      ↓<br>Worker Consumers<br>      ↓<br>Parallel Execution</pre><h3>Why this matters:</h3><ul><li>Scheduler does NOT execute heavy work</li><li>Workers scale independently</li><li>Backpressure is handled via Kafka</li></ul><h3>Kafka Partitioning Strategy</h3><p>Partitioning is not arbitrary — it is a consistency decision.</p><h3>Key Selection Rules:</h3><p>Use Case — — — — — — — — — Partition Key</p><p>Order lifecycle — — — — — — orderId</p><p>Store sync — — — — — — — — storeId</p><p>Inventory updates — — — — storeId</p><h3>Why:</h3><ul><li>Preserves ordering guarantees</li><li>Enables horizontal scaling</li><li>Avoids cross-entity contention</li></ul><h3>Failure Handling Model</h3><p>Failures are not exceptional — they are expected.</p><h3>Failures:</h3><ul><li>Network timeout</li><li>Partner API failure</li><li>Consumer crash</li><li>Duplicate webhook</li><li>Out-of-order event</li></ul><h3>System Design Principle:</h3><blockquote><em>Every failure must be replayable without duplicating business effects.</em></blockquote><p>When those boundaries are clear, adding the fifth, fifteenth, or fiftieth partner is not a rewrite.</p><p>It is just another controlled variation of the same system.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3913c3c21ecb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond Polling: How We Built a Real-Time Payment Status Engine at Scale]]></title>
            <link>https://manishgoyal05.medium.com/beyond-polling-how-we-built-a-real-time-payment-status-engine-at-scale-22c1470b44a8?source=rss-6ef7ca08a6a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/22c1470b44a8</guid>
            <category><![CDATA[server-side-events]]></category>
            <category><![CDATA[polling]]></category>
            <category><![CDATA[long-polling]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[payments]]></category>
            <dc:creator><![CDATA[Manish Goyal]]></dc:creator>
            <pubDate>Thu, 09 Jul 2026 13:58:49 GMT</pubDate>
            <atom:updated>2026-07-09T13:58:49.348Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>How we moved a Payment SDK and merchant dashboard from repeated polling to SSE without compromising correctness.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*673v--88yCrIRe-QBtET6Q.png" /></figure><p>There is a small moment in every payment journey where trust is won or lost.</p><p>The user has approved the payment.<br>The backend may already know the transaction is successful.<br>But the screen still says:</p><blockquote><strong>Processing…</strong></blockquote><p>For a few seconds, the user is unsure.</p><p>Did the money move?<br>Should they wait?<br>Should they retry?<br>Should they close the app?<br>Should the merchant refresh the dashboard?</p><p>This article is about solving that exact gap.</p><p>We faced this while building a Payment SDK embedded inside a large partner app in Indonesia. The SDK supported in-app payments and deep-link based journeys, similar to how users move between checkout pages and payment apps in UPI-style flows.</p><blockquote>The first implementation used polling. It was simple and predictable. But as the payment flow matured, polling started showing its limits.</blockquote><p>So we moved to a <strong>Hybrid real-time model</strong> using:</p><ul><li>Server-Sent Events for live status updates</li><li>Redis Pub/Sub for multi-instance event propagation</li><li>Redis cache for latest payment status</li><li>PostgreSQL as the source of truth</li><li>fallback polling for correctness</li><li>JWT validation and encrypted payloads</li><li>heartbeat and timeout handling for production safety</li><li>merchant dashboard live transaction updates</li></ul><p>This was not just a backend optimization.</p><p>It was a trust improvement.</p><h3>The Problem Was Not Status. It Was Status Visibility.</h3><p>In a payment system, the backend state and the user-visible state can drift for a few seconds.</p><p>The backend may receive a callback from a payment rail, PSP, gateway, or real-time payment core. It updates the transaction status. But the frontend sees the new state only when it asks again.</p><p>Earlier, the SDK called:</p><pre>GET /v3/pymnt-status/{paymentId}</pre><p>The frontend polled this API every 4 seconds until the payment reached a terminal state, such as SUCCESS, FAILED, or PAYMENT_CANCELED.</p><p>Our typical numbers looked like this:</p><ul><li>Polling interval: 4 seconds</li><li>Average pending duration: ~ 20 seconds</li><li>Normal journey: ~ 5 status calls</li><li>Longer journey: ~ 20–25 status calls</li><li>Status lookup: initially database-heavy</li></ul><blockquote><strong>Polling worked, but it created repeated work.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CFm67v0TKOfuIUcMeJPLvA.png" /><figcaption>Polling-based status flow</figcaption></figure><p>The frontend kept asking, “Is it done now?”<br>The backend kept checking the database.<br>The user kept waiting.<br>The merchant dashboard kept lagging.</p><p>Payment status is event-driven by nature. It changes because a backend event happens, not because the frontend keeps asking.</p><p>That became the design trigger.</p><blockquote><em>Instead of repeatedly checking payment status, push the status when the backend knows it.</em></blockquote><h3>The New Model: Push First, Poll Only as a Safety Net</h3><p>We introduced an SSE endpoint in the payment orchestration service, routed through our gateway:</p><pre>GET /v3/pymnt-status/{paymentId}/stream<br>Accept: text/event-stream<br>Authorization: Bearer &lt;jwt&gt;</pre><p>The stream remains open for up to 5 minutes.</p><p>It sends:</p><ul><li>payment-status when status changes</li><li>heartbeat to keep the connection alive</li><li>Final terminal status before closing the stream</li></ul><p>At a high level, the model became:</p><pre>SSE for real-time UX<br>Redis Pub/Sub for cross-instance delivery<br>Redis cache for latest status<br>PostgreSQL for source of truth<br>Fallback polling for correctness</pre><p>The important point is this:</p><p><strong>SSE alone is not the architecture.</strong></p><p>A production-grade payment stream needs event propagation, latest-state lookup, authorization, fallback, timeout handling, and reconciliation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S9S9OECEYUgTXnhvsPSWzA.png" /><figcaption>SSE + Redis Pub/Sub Architecture</figcaption></figure><p>[Insert Diagram 2: SSE + Redis Pub/Sub Architecture]</p><h3>Why Redis Pub/Sub Was Critical</h3><p>A simple paymentId -&gt; SseEmitter map works only when everything happens on the same instance.</p><p>But production systems usually run multiple instances behind a gateway or load balancer.</p><p>Imagine this:</p><p>The user opens an SSE stream.<br>The stream lands on Instance B.<br>The payment callback lands on Instance A.<br>Instance A updates the database.<br>But the open SSE connection is still on Instance B.</p><p>Without cross-instance propagation, Instance B may not immediately know that the payment succeeded.</p><p>Redis Pub/Sub solved this.</p><p>Redis Pub/Sub is not the source of truth. It is the notification layer.</p><p>PostgreSQL remains the source of truth. Redis latest-status cache acts as the fast snapshot.</p><p>[Insert Diagram 3: Multi-instance Redis Pub/Sub flow]</p><h3>Security: A Payment Stream Must Not Be Public</h3><p>Payment status may look harmless, but it is sensitive.</p><p>It can reveal whether a user paid, failed, cancelled, or received a refund.</p><p>So the stream should not open just because someone knows a paymentId.</p><p>Before opening the stream, the backend should validate:</p><p><strong>JWT and session context</strong><br>The request must come from an authenticated SDK/user/session flow.</p><p><strong>Payment ownership</strong><br>The payment should belong to the same client/session/merchant context.</p><p><strong>Payment ID format and validity</strong><br>Invalid or manipulated identifiers should fail early.</p><p><strong>Encrypted response contract</strong><br>If the SDK APIs return encrypted data, the SSE payload should follow the same contract.</p><p><strong>Minimal payload</strong><br>The event should contain only what the client needs. Avoid unnecessary PII.</p><p>A logical event may look like this:</p><pre>{<br>  &quot;paymentId&quot;: &quot;957d1676-35b4-4d3d-9e8c-c47d32bf1814&quot;,<br>  &quot;status&quot;: &quot;SUCCESS&quot;,<br>  &quot;updatedAt&quot;: &quot;2026-07-09T12:10:20Z&quot;<br>}</pre><p>In our flow, the response data followed the encrypted V3 API contract.</p><h3>The Merchant Dashboard Also Became Real-Time</h3><p>The same pattern was not limited to the checkout screen.</p><p>Merchant dashboards also needed better transaction visibility.</p><p>Earlier, dashboard users often depended on refresh or polling to know:</p><p>Did this payment succeed?<br>Is this transaction still pending?<br>Was the order cancelled?<br>Has the refund started?<br>Which transactions need attention?</p><p>With the same streaming architecture, the dashboard can subscribe to live merchant-scoped transaction updates.</p><p>The checkout stream is usually scoped to one payment:</p><pre>paymentId-level stream</pre><p>The merchant dashboard stream can be scoped to a merchant or client:</p><pre>merchantId/clientId-level live transaction feed</pre><p>This improves operations because merchants and support teams do not need to refresh repeatedly. They see transaction movement as it happens.</p><p>Real-time status becomes more than a checkout feature.</p><p>It becomes an operational visibility layer.</p><h3>What Changed in Practice</h3><p>Before streaming:</p><ul><li>frontend polled every 4 seconds</li><li>long journeys could trigger 20–25 status calls</li><li>Each call created gateway/backend/database work</li><li>UI waited until the next poll</li><li>Merchant dashboards had delayed transaction visibility</li></ul><p>After streaming:</p><ul><li>One SSE stream handled the active checkout</li><li>status was pushed when the backend state changed</li><li>Redis Pub/Sub distributes events across instances</li><li>Redis served the latest status quickly</li><li>DB remained the authority</li><li>polling became a fallback path</li></ul><p>For longer flows, the reduction can be significant:</p><pre>Before: up to around 25 status calls per payment<br>After: 1 SSE stream + 1–3 fallback calls in most cases<br>Estimated reduction in repeated status checks: 80–95%</pre><p>For 10,000 payment attempts/day:</p><pre>Before: up to 250,000 status checks/day<br>After: roughly 20,000–40,000 stream/fallback interactions/day<br>Potential avoided repeated checks: around 200,000+ per day</pre><p>These are estimates, but they show the direction clearly.</p><p>The bigger win was the experience.</p><p>The user no longer had to wait for the next polling cycle.<br>The merchant dashboard no longer depended only on refresh.<br>The backend avoided repeated status checks.<br>The database load reduced when Redis-first lookup was used.</p><h3>The Part People Often Miss: SSE Is Not Free</h3><p>SSE reduces repeated request-response cycles, but every open stream still has a cost.</p><p>It uses:</p><ul><li>one HTTP connection</li><li>emitter memory</li><li>heartbeat work</li><li>event push work</li><li>fallback-check work</li></ul><p>With Spring MVC SseEmitter, The original request thread is released after the emitter is returned. That is good.</p><p>But if the implementation starts one sleeping worker per stream for fallback checks, executor pressure can grow under high concurrency.</p><p>For scale, I would recommend:</p><p><strong>Use Redis Pub/Sub as the primary path</strong><br>The stream should mostly react to events, not keep polling aggressively.</p><p><strong>Use Redis-first lookup</strong><br>Read the latest status from Redis and use DB only on misses or safety checks.</p><p><strong>Avoid one sleeping worker per stream</strong><br>Prefer a scheduled sweep across active streams.</p><p><strong>Track stream metrics</strong><br>Measure active streams, timeouts, errors, reconnects, heartbeat failures, Redis misses, and DB fallback count.</p><p><strong>Consider Java 21 virtual threads or WebFlux</strong><br>If stream concurrency becomes very high, evaluate the runtime model carefully.</p><p>The lesson is simple:</p><blockquote>SSE becomes scalable only when stream lifecycle, event propagation, fallback, and observability are designed together.</blockquote><h3>Failure Scenarios We Had to Design For</h3><p><strong>Client disconnects during payment</strong><br>Clean up the active emitter and allow the client to reconnect.</p><p><strong>Mobile network changes</strong><br>Reconnect the stream. If reconnect fails repeatedly, fall back to status API polling.</p><p><strong>Redis Pub/Sub event is missed</strong><br>Use Redis latest-status lookup and periodic fallback checks.</p><p><strong>Redis cache misses</strong><br>Read from PostgreSQL and refresh Redis.</p><p><strong>Redis is unavailable</strong><br>Degrade to DB-backed status API behavior.</p><p><strong>Payment remains pending beyond timeout</strong><br>Close the stream gracefully and let reconciliation or normal status API calls resolve it.</p><p><strong>Duplicate callback arrives</strong><br>Make payment status updates idempotent.</p><p><strong>Same payment opens on two devices</strong><br>Allow multiple subscribers only after ownership validation.</p><p><strong>Unauthorized stream access</strong><br>Reject before stream creation.</p><p><strong>Sensitive payload risk</strong><br>Send minimal data and encrypt the response if the SDK contract requires it.</p><h3>Final Takeaway</h3><p>This was not simply a move from polling to SSE.</p><p>The real shift was:</p><pre>From repeated status checking<br>to event-driven status visibility<br>with correctness safeguards.</pre><p>The final model looked like this:</p><pre>SSE for real-time UX<br>Redis Pub/Sub for multi-instance delivery<br>Redis cache for fast latest status<br>PostgreSQL as source of truth<br>fallback polling for safety<br>reconciliation for delayed/stuck payments<br>JWT and encryption for secure delivery</pre><p>In payment systems, correctness comes first.</p><p>But once correctness is protected, real-time visibility becomes a powerful trust layer.</p><p>The user sees confirmation faster.<br>The merchant sees transaction movement live.<br>The backend avoids repeated work.<br>The system stays safe through fallback and reconciliation.</p><p>When money has moved, the interface should not wait for the next poll to say so.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22c1470b44a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Should I migrate from cookie-based authentication to JWT?]]></title>
            <link>https://manishgoyal05.medium.com/is-migrating-cookie-based-authentication-to-jwt-worth-it-4557cd51da9d?source=rss-6ef7ca08a6a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/4557cd51da9d</guid>
            <category><![CDATA[cookies]]></category>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[jwt]]></category>
            <category><![CDATA[authorization]]></category>
            <dc:creator><![CDATA[Manish Goyal]]></dc:creator>
            <pubDate>Fri, 28 May 2021 06:11:41 GMT</pubDate>
            <atom:updated>2021-06-03T04:38:49.359Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/101/1*sImdcGYyVj_eeMz-wy8FOA.png" /></figure><p>Many times when we start a project, we first finalize a language and then a framework and nowadays framework comes with so many ready to serve features that without any single thought we start using them.</p><p>One of these features is <strong>authorizing</strong> subsequent API requests of the client once they have <strong>authenticated</strong> themselves typically using login-password so that they can access resources specific to that user without again asking them the credentials.</p><p>An age-old process of doing the same is issuing a unique hash embedded in a cookie to the client which they pass in every API request to the server but then</p><h3>Why cookies are considered bad?</h3><p>As I mentioned before while using a cookie, a unique hash for a client is created and stored in the database and the same hash (present in cookie, head, or body of the request) is verified against the incoming requests every single time. then what’s the issue?</p><p>The problem with this approach is that for every incoming request, we have to make a database query <strong>EVERY</strong> single time client is accessing the restricted area for authentication. That’s not at all good for <strong>performance</strong> as well as <strong>user experience</strong>. What’s the solution then?</p><h3>How JWT comes as a rescue</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*nRDmtHI6hN-vMMgA" /></figure><p>JWT’s most intrinsic property is that they are stateless and, for API services that need to support server-to-server or client-to-server (like a mobile app or single page app (SPA)) communication, using JWTs as your API tokens can be a very smart idea. In this scenario:</p><ul><li>You will have an authentication API that clients can authenticate against, and get back a JWT.</li><li>Clients then can use this JWT to send authenticated requests to other API services as well.</li><li>Also, These other API services then can use the client’s JWT to validate whether they are trusted and can perform actions without needing to perform a network validation</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D3YDxY1E1AAyufPn.jpg" /></figure><p>Considering all these facts, JWT becomes a perfect candidate for an end to end web application authentication.</p><p>However, I see two main challenges when we use only JWT for the complete lifecycle of API authentication.</p><ol><li><strong>The inability to revoke the Token:</strong> Once you log out, the application can only delete the Token from the browser cache. However, if someone gets on hold with the Token, he/she can log in to the application during the token validity period. This point could be a severe risk for many business applications.</li><li><strong>Difficulty in refreshing the Token:</strong> You can argue that there are options such as silent-refresh for implicit flow and refresh token in code grant flow to get new tokens before the existing Token expires. However, if you look into the details of complexity it adds to your code with security challenges, I doubt whether it’s a rational decision when we have the other option of using Cookies and Sessions.</li></ol><h3>Using Cookies and token together</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZeywUyd_6CyTYuYu.png" /></figure><p>We know how powerful cookies are in terms of security and how powerful tokens are in terms of agility and performance. So why don’t combine them?</p><p>I propose using cookies having JWT stored in them. The JWT will in return contain a random hash that is maintained in DB. The diagram explains the flow and scenarios which we encounter in any user’s life</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WyVfx2UUWMS8Y0jrEKztrA.png" /></figure><p>This flow will solve the two major issues we were facing when using JWT alone which were:</p><ol><li><strong>The inability to revoke the Token: </strong>Token will be stored in the cache(e.g. Redis) till their expiry time so while verifying the token, the application will check whether the token is still valid.</li><li><strong>Difficulty in refreshing the Token: </strong>Once the token is expired, a new token will be issued by validating the hash from DB.</li></ol><h3>Conclusion</h3><p>The solution I have proposed here may not be the best but for the initial phase of an organization where they lack resources and developer bandwidth, this option will provide them security as well as performance. Will love to hear your views and how we can together improve. Till then, Happy Coding!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4557cd51da9d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>