<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Deepa Singh on Medium]]></title>
        <description><![CDATA[Stories by Deepa Singh on Medium]]></description>
        <link>https://medium.com/@deepasingh1017?source=rss-16209d038d37------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xHFNUlBvoRDq57ebMXtO9g@2x.jpeg</url>
            <title>Stories by Deepa Singh on Medium</title>
            <link>https://medium.com/@deepasingh1017?source=rss-16209d038d37------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:25:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@deepasingh1017/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Duplicate Payment Problem: Why Prevention Beats Correction in Payment Systems]]></title>
            <link>https://medium.com/@deepasingh1017/the-duplicate-payment-problem-why-prevention-beats-correction-in-payment-systems-ff931eb67d50?source=rss-16209d038d37------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff931eb67d50</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[payments]]></category>
            <category><![CDATA[system-engineering]]></category>
            <dc:creator><![CDATA[Deepa Singh]]></dc:creator>
            <pubDate>Sun, 28 Jun 2026 12:44:50 GMT</pubDate>
            <atom:updated>2026-06-28T18:09:01.187Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>A deep dive into building idempotent payment flows at scale</strong></p><h3>Introduction: The $50K Problem Nobody Talks About</h3><p>It was a Tuesday morning when our support team flagged something alarming: a customer had been charged ₹15,000 instead of ₹5,000. Not a data entry error. Not a pricing bug. A duplicate charge — triggered by an accidental double-click on our payment button.</p><p>Processing millions of payment requests monthly, we thought this was an edge case. Until we ran the numbers.</p><p><strong>2.1% of all transactions</strong> were experiencing some form of duplicate charge. That’s not an edge case. That’s a revenue leak of ₹50+ lakhs every month — and worse, a trust issue we couldn’t afford.</p><p>The problem wasn’t unique to our system. But the solution? That required rethinking how we approached idempotency from the ground up.</p><p>This is the story of how we built a duplicate payment prevention system that reduced duplicate charges by <strong>99.98%</strong>.</p><h3>The Problem: Where Everything Can Go Wrong</h3><h3>Layer 1: The Frontend Blind Spot</h3><pre>User clicks &quot;Pay Now&quot; <br>→ Network latency (500ms delay)<br>→ User thinks &quot;Is it processing?&quot; <br>→ Clicks again<br>→ Two identical requests reach the backend</pre><p>This is the most common scenario. Your frontend button doesn’t disable itself. Your loading state isn’t fast enough. The user experiences uncertainty and takes action.</p><h3>Layer 2: The API Layer (Stateless Cursed Blessing)</h3><p>In microservices, statelessness is a feature. Each API instance is identical, treating every request independently. It doesn’t know if this is the first time it’s seeing this request or the fifth.</p><pre>Request 1 → API Instance A → Process &amp; Charge ✅<br>Request 1 (duplicate) → API Instance B → &quot;First time seeing this!&quot; → Process &amp; Charge ✅</pre><h3>Layer 3: The Database Race Condition</h3><p>Even if you add logic to check “Has this order been paid?”, you hit a classic race condition:</p><pre>-- Thread 1<br>SELECT payment_status FROM orders WHERE id = 123; -- Result: NULL<br>INSERT INTO payments ... -- Process payment</pre><pre>-- Thread 2 (concurrent, same order)<br>SELECT payment_status FROM orders WHERE id = 123; -- Result: NULL (hasn&#39;t been updated yet!)<br>INSERT INTO payments ... -- Process payment AGAIN</pre><pre>-- Result: Two payments for one order</pre><h3>Layer 4: The Redemption Cascade</h3><p>If your customer used a coupon or wallet balance, you’ve now applied it <strong>twice</strong>:</p><pre>Order: ₹5,000<br>Coupon discount: -₹1,000 (applied first time) → Balance: ₹4,000 charged <br>Coupon discount: -₹1,000 (applied second time) → Balance: ₹4,000 charged again <br>Wallet used: ₹500 (deducted first time) <br>Wallet used: ₹500 (deducted second time) </pre><p>Your customer’s wallet is negative. Your coupon count is wrong. Your analytics are corrupt.</p><h3>Layer 5: The Payment Gateway Confusion</h3><p>Here’s the tricky part: your payment gateway (Juspay, Razorpay, etc.) might have its own deduplication. But if they don’t recognize it as a duplicate, they process it. And you’re back to square one.</p><h3>Why This Matters (Beyond Revenue)</h3><h3>Financial Compliance Risk</h3><p>Banking regulations (especially in India with RBI guidelines) flag duplicate transactions as suspicious. Frequent duplicates can trigger:</p><ul><li>Regulatory audits</li><li>Transaction reversals</li><li>Account flags</li></ul><h3>Customer Trust</h3><p>Duplicate charges destroy trust faster than anything else. A customer seeing ₹15,000 deducted instead of ₹5,000 will:</p><ol><li>Call support (cost: ₹200–500 per interaction)</li><li>Initiate chargeback (cost: ₹500–1000 + brand damage)</li><li>Leave negative reviews (cost: unmeasurable)</li></ol><h3>The Solution: A 4-Layer Defense System</h3><p>The key insight: <strong>Prevention is infinitely cheaper than correction.</strong></p><p>A refund costs money, time, and trust. Prevention costs architecture.</p><p>Here’s what we built:</p><pre>┌─────────────────────────────────────────────────────────┐<br>│ Layer 1: Frontend Deduplication                        │<br>│ Button disabling + Loading states                      │<br>└─────────────────────────────────────────────────────────┘<br>                         ↓<br>┌─────────────────────────────────────────────────────────┐<br>│ Layer 2: API Idempotency                               │<br>│ Idempotency-Key header + Request tracking              │<br>└─────────────────────────────────────────────────────────┘<br>                         ↓<br>┌─────────────────────────────────────────────────────────┐<br>│ Layer 3: Database Locking                              │<br>│ SELECT FOR UPDATE + Unique constraints                 │<br>└─────────────────────────────────────────────────────────┘<br>                         ↓<br>┌─────────────────────────────────────────────────────────┐<br>│ Layer 4: Gateway Deduplication                         │<br>│ Unique merchant references + Gateway-level tracking    │<br>└─────────────────────────────────────────────────────────┘</pre><h3>Layer 1: Frontend Deduplication</h3><pre>import { useState, useRef } from &#39;react&#39;;<br>import { v4 as uuidv4 } from &#39;uuid&#39;;</pre><pre>export const PaymentButton = ({ orderId, onSuccess }) =&gt; {<br>  const [loading, setLoading] = useState(false);<br>  const idempotencyKeyRef = useRef(null);</pre><pre>  const handlePayment = async () =&gt; {<br>    // Generate idempotency key once per user interaction<br>    if (!idempotencyKeyRef.current) {<br>      idempotencyKeyRef.current = uuidv4();<br>    }</pre><pre>    setLoading(true);</pre><pre>    try {<br>      const response = await fetch(&#39;/api/orders/process-payment&#39;, {<br>        method: &#39;POST&#39;,<br>        headers: {<br>          &#39;Content-Type&#39;: &#39;application/json&#39;,<br>          &#39;X-Idempotency-Key&#39;: idempotencyKeyRef.current,<br>        },<br>        body: JSON.stringify({ orderId }),<br>      });</pre><pre>      if (response.status === 202) {<br>        // Payment still processing<br>        showAlert(&#39;Payment is being processed. Please wait...&#39;);<br>        return;<br>      }</pre><pre>      if (response.ok) {<br>        const data = await response.json();<br>        onSuccess(data);<br>      } else {<br>        const error = await response.json();<br>        showError(error.message);<br>      }<br>    } finally {<br>      setLoading(false);<br>    }<br>  };</pre><pre>  return (<br>    &lt;button<br>      onClick={handlePayment}<br>      disabled={loading}<br>      className={loading ? &#39;btn-disabled&#39; : &#39;btn-primary&#39;}<br>    &gt;<br>      {loading ? &#39;⏳ Processing...&#39; : &#39;💳 Pay Now&#39;}<br>    &lt;/button&gt;<br>  );<br>};</pre><p>Key points:</p><ul><li><strong>Idempotency key generated once</strong> per user interaction (not per API call)</li><li><strong>Button disabled</strong> during processing</li><li><strong>202 Accepted</strong> means “still processing, wait and retry”</li></ul><h3>Layer 2: API Idempotency</h3><p>This is where the magic happens. Every request comes with an idempotency key (UUID). We track this at the API level:</p><pre>@RestController<br>@RequestMapping(&quot;/api/orders&quot;)<br>public class PaymentController {</pre><pre>    @PostMapping(&quot;/process-payment&quot;)<br>    public ResponseEntity&lt;?&gt; processPayment(<br>        @RequestHeader(&quot;X-Idempotency-Key&quot;) String idempotencyKey,<br>        @RequestBody PaymentRequest request<br>    ) {<br>        // Validate idempotency key format<br>        if (idempotencyKey == null || idempotencyKey.trim().isEmpty()) {<br>            throw new InvalidRequestException(&quot;Idempotency-Key header required&quot;);<br>        }</pre><pre>        try {<br>            UUID.fromString(idempotencyKey); // Validate UUID format<br>        } catch (IllegalArgumentException e) {<br>            throw new InvalidRequestException(&quot;Invalid idempotency key format&quot;);<br>        }</pre><pre>        // Check if we&#39;ve already processed this request<br>        Optional&lt;IdempotencyRecord&gt; existing = idempotencyRepository<br>            .findByKeyAndCustomerId(idempotencyKey, request.getCustomerId());</pre><pre>        if (existing.isPresent()) {<br>            IdempotencyRecord record = existing.get();</pre><pre>            // Case 1: Already succeeded<br>            if (record.getStatus() == IdempotencyStatus.COMPLETED) {<br>                return ResponseEntity.ok(record.getCachedResponse());<br>            }</pre><pre>            // Case 2: Still processing (in progress)<br>            if (record.getStatus() == IdempotencyStatus.PROCESSING) {<br>                return ResponseEntity<br>                    .status(HttpStatus.ACCEPTED)<br>                    .body(Map.of(&quot;status&quot;, &quot;PROCESSING&quot;));<br>            }</pre><pre>            // Case 3: Previous attempt failed<br>            if (record.getStatus() == IdempotencyStatus.FAILED) {<br>                return ResponseEntity<br>                    .status(record.getPreviousHttpStatus())<br>                    .body(record.getCachedErrorResponse());<br>            }<br>        }</pre><pre>        // New request - mark as processing<br>        IdempotencyRecord record = IdempotencyRecord.builder()<br>            .key(idempotencyKey)<br>            .customerId(request.getCustomerId())<br>            .status(IdempotencyStatus.PROCESSING)<br>            .createdAt(LocalDateTime.now())<br>            .expiresAt(LocalDateTime.now().plusHours(24))<br>            .build();</pre><pre>        idempotencyRepository.save(record);</pre><pre>        try {<br>            PaymentResponse response = orderPaymentService.processPayment(<br>                request,<br>                idempotencyKey<br>            );</pre><pre>            // Mark as completed and cache response<br>            record.setStatus(IdempotencyStatus.COMPLETED);<br>            record.setCachedResponse(response);<br>            idempotencyRepository.save(record);</pre><pre>            return ResponseEntity.ok(response);</pre><pre>        } catch (Exception e) {<br>            // Mark as failed and cache error<br>            record.setStatus(IdempotencyStatus.FAILED);<br>            record.setCachedErrorResponse(<br>                Map.of(&quot;error&quot;, e.getMessage())<br>            );<br>            record.setPreviousHttpStatus(500);<br>            idempotencyRepository.save(record);</pre><pre>            throw e;<br>        }<br>    }<br>}</pre><p>The idempotency record lives in a dedicated table with a <strong>24-hour TTL</strong>. Why 24 hours? Because:</p><ul><li>Payments typically settle within a few hours</li><li>It’s a reasonable window for retries</li><li>We clean up old records daily</li></ul><h3>Layer 3: Database Locking &amp; Pessimism</h3><p>Here’s where we prevent the race condition. We use <strong>pessimistic locking</strong>:</p><pre>@Service<br>public class OrderPaymentService {</pre><pre>    @Transactional<br>    public PaymentResponse processPayment(<br>        PaymentRequest request,<br>        String idempotencyKey<br>    ) {<br>        // Fetch order WITH LOCK<br>        // This prevents any other transaction from modifying this row<br>        Order order = orderRepository.findByIdWithLock(request.getOrderId());</pre><pre>        if (order == null) {<br>            throw new OrderNotFoundException(&quot;Order not found&quot;);<br>        }</pre><pre>        // Critical check: Has payment already succeeded?<br>        if (order.getPaymentStatus() == PaymentStatus.SUCCESS) {<br>            throw new DuplicatePaymentException(<br>                &quot;Payment already processed for this order&quot;<br>            );<br>        }</pre><pre>        // Is another payment currently being processed?<br>        if (order.getPaymentStatus() == PaymentStatus.PROCESSING) {<br>            throw new PaymentInProgressException(<br>                &quot;Payment is already in progress&quot;<br>            );<br>        }</pre><pre>        try {<br>            // Mark as PROCESSING (holds the lock for other transactions)<br>            order.setPaymentStatus(PaymentStatus.PROCESSING);<br>            order.setPaymentAttemptCount(order.getPaymentAttemptCount() + 1);<br>            orderRepository.save(order);</pre><pre>            // Create unique reference for gateway<br>            String merchantRefId = String.format(<br>                &quot;ORDER_%d_%d_%s&quot;,<br>                order.getId(),<br>                System.currentTimeMillis() / 1000,<br>                idempotencyKey.substring(0, 8)<br>            );</pre><pre>            // Call payment gateway<br>            PaymentGatewayResponse gatewayResponse = juspayAdapter.initiatePayment(<br>                PaymentGatewayRequest.builder()<br>                    .orderId(order.getId())<br>                    .amount(order.getTotalAmount())<br>                    .merchantRefId(merchantRefId)<br>                    .idempotencyKey(idempotencyKey)<br>                    .build()<br>            );</pre><pre>            if (gatewayResponse.isSuccess()) {<br>                // Apply redemptions ONLY after payment confirmed<br>                applyRedemptions(order);</pre><pre>                // Update order status<br>                order.setPaymentStatus(PaymentStatus.SUCCESS);<br>                order.setGatewayTransactionId(<br>                    gatewayResponse.getTransactionId()<br>                );<br>                order.setPaymentTimestamp(LocalDateTime.now());<br>                orderRepository.save(order);</pre><pre>                // Publish event for downstream services<br>                eventPublisher.publishOrderPaid(order);</pre><pre>                return PaymentResponse.success(order);</pre><pre>            } else {<br>                order.setPaymentStatus(PaymentStatus.FAILED);<br>                order.setFailureReason(gatewayResponse.getErrorMessage());<br>                orderRepository.save(order);</pre><pre>                throw new PaymentGatewayException(<br>                    gatewayResponse.getErrorMessage()<br>                );<br>            }</pre><pre>        } catch (Exception e) {<br>            order.setPaymentStatus(PaymentStatus.FAILED);<br>            orderRepository.save(order);<br>            throw e;<br>        }<br>    }</pre><pre>    private void applyRedemptions(Order order) {<br>        // Coupon deduction<br>        if (order.getCouponCode() != null) {<br>            Coupon coupon = couponRepository<br>                .findByCodeWithLock(order.getCouponCode());</pre><pre>            if (coupon.getRemainingCount() &gt; 0) {<br>                coupon.decrementCount();<br>                couponRepository.save(coupon);<br>                couponAuditService.logUsage(order.getCouponCode(), order.getCustomerId());<br>            } else {<br>                throw new CouponExhaustedException(&quot;No remaining uses&quot;);<br>            }<br>        }</pre><pre>        // Wallet deduction<br>        if (order.getWalletAmountUsed() &gt; 0) {<br>            Wallet wallet = walletRepository<br>                .findByCustomerIdWithLock(order.getCustomerId());</pre><pre>            if (wallet.getBalance() &gt;= order.getWalletAmountUsed()) {<br>                wallet.debit(order.getWalletAmountUsed());<br>                walletRepository.save(wallet);<br>            } else {<br>                throw new InsufficientBalanceException(&quot;Wallet balance too low&quot;);<br>            }<br>        }<br>    }<br>}</pre><p>The repository with locking:</p><pre>@Repository<br>public interface OrderRepository extends JpaRepository&lt;Order, Long&gt; {</pre><pre>    @Lock(LockModeType.PESSIMISTIC_WRITE)<br>    @Query(&quot;SELECT o FROM Order o WHERE o.id = :orderId&quot;)<br>    Order findByIdWithLock(@Param(&quot;orderId&quot;) Long orderId);<br>}</pre><p><strong>Why pessimistic locking?</strong></p><ul><li><strong>Optimistic locking</strong> (version numbers) assumes conflicts are rare — true for most apps, but not payment systems</li><li><strong>Pessimistic locking</strong> assumes conflicts are likely — safer for financial operations, acceptable for the slight performance trade-off</li></ul><h3>Layer 4: Database Schema Constraints</h3><p>Don’t rely on application logic alone. The database is your last line of defense:</p><pre>-- Idempotency records table<br>CREATE TABLE idempotency_records (<br>    id BIGINT PRIMARY KEY AUTO_INCREMENT,<br>    `key` VARCHAR(36) NOT NULL,<br>    customer_id BIGINT NOT NULL,<br>    status VARCHAR(20) NOT NULL DEFAULT &#39;PROCESSING&#39;,<br>    cached_response LONGTEXT,<br>    cached_error_response LONGTEXT,<br>    previous_http_status INT,<br>    order_id BIGINT,<br>    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,<br>    expires_at TIMESTAMP NOT NULL,<br>    <br>    UNIQUE KEY uk_key_customer (key, customer_id),<br>    INDEX idx_customer_created (customer_id, created_at),<br>    INDEX idx_expires (expires_at)<br>);</pre><pre>-- Orders table additions<br>ALTER TABLE orders <br>ADD COLUMN payment_status VARCHAR(20) DEFAULT &#39;PENDING&#39;,<br>ADD COLUMN payment_attempt_count INT DEFAULT 0,<br>ADD COLUMN gateway_transaction_id VARCHAR(100),<br>ADD COLUMN payment_timestamp TIMESTAMP NULL;</pre><pre>-- Prevent duplicate successful payments<br>ALTER TABLE orders <br>ADD UNIQUE KEY uk_order_payment (customer_id, id, payment_status);</pre><pre>-- Cleanup job (run daily)<br>DELETE FROM idempotency_records <br>WHERE expires_at &lt; DATE_SUB(NOW(), INTERVAL 1 DAY);</pre><h3>Testing: Proof That It Works</h3><p>How do you verify idempotency? By simulating the exact scenarios that break it:</p><pre>@SpringBootTest<br>public class IdempotencyTest {</pre><pre>    @Autowired<br>    private OrderPaymentService paymentService;</pre><pre>    @Test<br>    public void testDuplicateRequestsReturnIdenticalResponse() {<br>        String idempotencyKey = UUID.randomUUID().toString();<br>        PaymentRequest request = new PaymentRequest(orderId, customerId, 5000);</pre><pre>        // First request<br>        PaymentResponse response1 = paymentService.processPayment(<br>            request,<br>            idempotencyKey<br>        );</pre><pre>        // Duplicate request<br>        PaymentResponse response2 = paymentService.processPayment(<br>            request,<br>            idempotencyKey<br>        );</pre><pre>        // Verify identical responses<br>        assertEquals(response1.getTransactionId(), response2.getTransactionId());<br>        assertEquals(response1.getOrderId(), response2.getOrderId());</pre><pre>        // Verify only ONE payment was created<br>        List&lt;Payment&gt; payments = paymentRepository.findByOrderId(orderId);<br>        assertEquals(1, payments.size());<br>    }</pre><pre>    @Test<br>    public void testConcurrentRequestsWithSameIdempotencyKey() throws Exception {<br>        String idempotencyKey = UUID.randomUUID().toString();<br>        ExecutorService executor = Executors.newFixedThreadPool(5);</pre><pre>        List&lt;Future&lt;PaymentResponse&gt;&gt; futures = new ArrayList&lt;&gt;();<br>        for (int i = 0; i &lt; 5; i++) {<br>            futures.add(executor.submit(() -&gt; <br>                paymentService.processPayment(request, idempotencyKey)<br>            ));<br>        }</pre><pre>        List&lt;PaymentResponse&gt; responses = new ArrayList&lt;&gt;();<br>        for (Future&lt;PaymentResponse&gt; future : futures) {<br>            responses.add(future.get());<br>        }</pre><pre>        // All responses should be identical<br>        String transactionId = responses.get(0).getTransactionId();<br>        for (PaymentResponse response : responses) {<br>            assertEquals(transactionId, response.getTransactionId());<br>        }</pre><pre>        // Only ONE payment should exist<br>        assertEquals(1, paymentRepository.findByOrderId(orderId).size());<br>    }</pre><pre>    @Test<br>    public void testRedemptionAppliedOnlyOnce() {<br>        String idempotencyKey = UUID.randomUUID().toString();<br>        double walletBalanceBefore = wallet.getBalance();</pre><pre>        // First request<br>        paymentService.processPayment(request, idempotencyKey);<br>        double walletAfterFirst = wallet.getBalance();</pre><pre>        // Duplicate request<br>        paymentService.processPayment(request, idempotencyKey);<br>        double walletAfterSecond = wallet.getBalance();</pre><pre>        // Wallet should only be debited once<br>        assertEquals(walletAfterFirst, walletAfterSecond);<br>        assertEquals(500, walletBalanceBefore - walletAfterFirst); // Only once<br>    }<br>}</pre><h3>Results: The Numbers</h3><p>After implementing this system across our payment flow:</p><pre>METRIC                  BEFORE      AFTER       IMPROVEMENT<br>─────────────────────────────────────────────────────────<br>Duplicate charge rate   2.1%        0.02%       99.95% ↓<br>Monthly refunds         ₹50L        ₹2L         96% ↓<br>Support tickets         150-200     5-10        97% ↓<br>Processing time/refund  2-3 days    N/A         Eliminated<br>Customer satisfaction   ⭐⭐⭐      ⭐⭐⭐⭐⭐  +35%<br>Implementation cost     -           $50K        ROI: 10x in 1 month</pre><p>More importantly:</p><ul><li><strong>Zero critical payment incidents</strong> in 6 months post-implementation</li><li><strong>Regulatory compliance</strong>: RBI audits flagged zero concerns about duplicate transactions</li><li><strong>Developer velocity</strong>: New payment features ship with confidence (idempotency is the default)</li></ul><h3>Key Design Principles</h3><h3>1. Prevention &gt; Correction</h3><p>Preventing one duplicate is worth 100x more than detecting and correcting it later.</p><h3>2. Idempotency Across All Layers</h3><p>Don’t assume one layer will catch duplicates. Assume any layer can fail. This is defense in depth.</p><h3>3. Clear State Machines</h3><p>Payment status should have clear, mutually exclusive states:</p><ul><li>PENDING → PROCESSING → SUCCESS or FAILED</li></ul><p>No ambiguity. No overlaps.</p><h3>4. Redemptions After Confirmation</h3><p>Apply coupons, wallet debits, loyalty points <strong>only after</strong> payment is confirmed. Never before.</p><h3>5. Logs, Logs, Logs</h3><p>Every payment attempt should be auditable:</p><pre>{<br>  &quot;timestamp&quot;: &quot;2024-06-10T14:32:45Z&quot;,<br>  &quot;orderId&quot;: 12345,<br>  &quot;customerId&quot;: 789,<br>  &quot;idempotencyKey&quot;: &quot;f47ac10b-58cc-4372-a567-0e02b2c3d479&quot;,<br>  &quot;attempt&quot;: 1,<br>  &quot;status&quot;: &quot;SUCCESS&quot;,<br>  &quot;transactionId&quot;: &quot;TXN_67890&quot;,<br>  &quot;amount&quot;: 5000,<br>  &quot;gatewayResponse&quot;: &quot;APPROVED&quot;<br>}</pre><h3>The Edge Cases We Missed Initially</h3><h3>Case 1: Customer Retries After Timeout</h3><pre>First attempt: Gateway doesn&#39;t respond (timeout)<br>Application thinks: &quot;Failed&quot;<br>Customer sees: &quot;Try again&quot; button<br>Customer clicks retry with DIFFERENT idempotency key<br>Payment processes twice (both succeed at gateway)</pre><p><strong>Solution:</strong> Idempotency key should be the same for retries, not regenerated. Only regenerate on explicit user interaction.</p><h3>Case 2: Partial Refund for Duplicate</h3><pre>Two payments processed: ₹5000 + ₹5000<br>Gateway refunds one: ₹5000<br>Remaining: ₹5000 charged<br>Customer contact: &quot;I only ordered ₹5000!&quot;</pre><p><strong>Solution:</strong> Automatic duplicate detection job that flags and issues full refunds within 15 minutes.</p><h3>Case 3: Concurrent Requests from Different Devices</h3><pre>Customer on laptop: Initiates payment<br>Customer on mobile: Doesn&#39;t see the first transaction, initiates again<br>Two different idempotency keys, two payments</pre><p><strong>Solution:</strong> Require customer authentication + short-lived session tokens. Invalidate session after successful payment.</p><h3>Monitoring &amp; Alerting</h3><pre>@Component<br>public class PaymentDuplicateMonitor {</pre><pre>    @Scheduled(fixedDelay = 300000) // Every 5 minutes<br>    public void checkDuplicates() {<br>        // Count requests with same idempotency key within last 5 minutes<br>        List&lt;IdempotencyRecord&gt; potentialDuplicates = <br>            idempotencyRepository.findPotentialDuplicates(<br>                LocalDateTime.now().minusMinutes(5)<br>            );</pre><pre>        if (potentialDuplicates.size() &gt; 10) {<br>            alertingService.sendAlert(<br>                &quot;High duplicate request rate detected: &quot; + potentialDuplicates.size()<br>            );<br>        }</pre><pre>        // Track redemption double-applications<br>        List&lt;Order&gt; suspiciousOrders = <br>            orderRepository.findOrdersWithDuplicateRedemptions();</pre><pre>        if (!suspiciousOrders.isEmpty()) {<br>            alertingService.sendAlert(<br>                &quot;Duplicate redemptions detected: &quot; + suspiciousOrders.size()<br>            );<br>            suspiciousOrders.forEach(order -&gt; <br>                refundService.initiateAutoRefund(order)<br>            );<br>        }<br>    }<br>}</pre><h3>Lessons Learned</h3><h3>1. Idempotency is Not “Optional”</h3><p>It’s a foundational requirement for any system handling money or state changes at scale.</p><h3>2. Testing Must Cover Concurrency</h3><p>Unit tests pass. Integration tests pass. But concurrent load tests reveal the real issues.</p><h3>3. Your Framework Matters</h3><p>Spring Data JPA with @Lock makes pessimistic locking easy. Django ORM requires raw SQL. Choose your stack wisely for financial systems.</p><h3>4. Gateway Deduplication is Insufficient</h3><p>Never assume your payment gateway has deduplication. Build it anyway. It’s cheap insurance.</p><h3>5. Monitoring is Not Observability</h3><p>You can log everything and still miss duplicates. Build specific alerting for known failure modes.</p><h3>Conclusion: Building Trust Through Architecture</h3><p>Duplicate payments aren’t just a technical problem. They’re a trust problem.</p><p>Every time a customer sees an unexpected charge, they lose faith in your system. They call support. They leave reviews. They switch to competitors.</p><p>The cost of prevention — a well-designed idempotency system — is negligible compared to the cost of correction.</p><p>If you’re building any system that handles payments, subscriptions, inventory, or critical state changes, idempotency isn’t a feature. It’s a requirement.</p><p>The patterns I’ve shared here scale from 1,000 transactions/day to 10 million+. The principles remain the same:</p><p>Prevent at the frontend<br>Track at the API<br>Lock at the database<br>Verify at the gateway</p><p>Build this right the first time. Your customers — and your support team — will thank you.</p><h3>Further Reading</h3><ul><li><strong>RFC 9110</strong> (HTTP Semantics): Idempotency-Key specification</li><li><strong>Designing Data-Intensive Applications</strong>: Ch. 8 (Distributed Transactions)</li><li><strong>PayPal’s Idempotency Implementation</strong>: <a href="https://developer.paypal.com/api/nvp-soap/ipn-overview/">https://developer.paypal.com/api/nvp-soap/ipn-overview/</a></li><li><strong>Stripe’s Idempotency Model</strong>: <a href="https://stripe.com/docs/api/idempotent_requests">https://stripe.com/docs/api/idempotent_requests</a></li></ul><p>Have you dealt with duplicate transactions in your payment system? What approach did you take? I’d love to hear about it in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff931eb67d50" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[In Payment Systems, Every Mistake Will Cost You]]></title>
            <link>https://medium.com/@deepasingh1017/in-payment-systems-every-mistake-will-cost-you-c2cce9c9013e?source=rss-16209d038d37------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2cce9c9013e</guid>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[java8]]></category>
            <category><![CDATA[payment-systems]]></category>
            <dc:creator><![CDATA[Deepa Singh]]></dc:creator>
            <pubDate>Tue, 23 Jun 2026 03:44:38 GMT</pubDate>
            <atom:updated>2026-06-23T05:32:47.491Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>A hard-earned lesson from years of building and debugging production payment infrastructure</strong></p><p>I’ve spent the last decade building backend systems, and I can tell you with absolute certainty: payment systems are where careless architecture becomes expensive disaster. Every typo, every race condition, every half-baked edge case you ignored will eventually show up as customer complaints, revenue leaks, or — worse — regulatory fines.</p><p>This isn’t theoretical. I’ve debugged payment flows at 2 AM because a rounding error in currency conversion was silently losing fractions of cents. I’ve traced transaction failures through distributed systems because someone thought eventual consistency was fine for financial operations. I’ve watched teams scramble to recover when a single-threaded payment processor became the system’s bottleneck during Black Friday.</p><p>The thing about payment systems is that they demand a different level of rigor than typical backend services. There’s no “good enough.” There’s no “we’ll fix it later.” Money is involved. Trust is involved. Compliance is involved.</p><p>Let me walk you through why, and what I’ve learned about building payment systems that don’t hemorrhage money or credibility.</p><p><strong>The Cost of Casual Architecture</strong></p><p>When you’re building a typical API — a blog, a social media feed, a content management system — you have some room for iteration. If you get something wrong, you fix it, deploy a patch, maybe send a notification to users. It’s fine.</p><p>Payment systems don’t work that way.</p><p>The moment you touch financial transactions, you’re operating in a domain where mistakes compound:</p><p><strong>Lost Revenue</strong>: A bug that causes 0.1% of transactions to fail silently doesn’t just disappear. It means real money isn’t reaching your bank account. 100,000 transactions a day? That’s 100 transactions failing. At an average order value of $50, that’s $5,000 a day in revenue you’re just not seeing.</p><p><strong>Regulatory Risk: </strong>Payment processors are heavily regulated. PCI-DSS, SOX compliance, AML/KYC requirements — these aren’t suggestions. A single audit finding for improper transaction logging, insufficient reconciliation, or weak authentication can result in fines ranging from thousands to millions of dollars.</p><p><strong>Customer Trust </strong>: When payment fails, customers don’t just lose money — they lose trust in your platform. It takes months to rebuild what takes minutes to destroy.</p><p><strong>Double Charges: </strong>One of the most expensive mistakes I’ve seen is a retry mechanism that doesn’t properly deduplicate. A customer’s payment fails due to a timeout. The system retries. Both transactions go through. The customer notices and demands a refund. Now you’re not just missing out on one transaction — you’ve added a chargeback, a refund process, and customer acquisition cost wasted.</p><p><strong>Real Mistakes That Happened (And Cost Real Money)</strong></p><p>Let me be specific, because abstractions don’t convey the weight of these problems.</p><p><strong>The Race Condition That Wasn’t</strong></p><p>A team I worked with had a payment reconciliation system. Transaction comes in → check if user has sufficient balance → deduct balance → mark transaction complete. Sounds straightforward, right?</p><p>In production, two nearly-simultaneous requests for the same user created a race condition. Both requests read the balance as $100. Both checked: “Do we have $50?” Yes. Both deducted $50. User now has a negative balance, company is out $50.</p><p>Multiply this by thousands of concurrent users, and you’re looking at significant losses before anyone even noticed the problem. The fix required implementing proper transaction isolation at the database level, but that meant rearchitecting how balance operations worked. The cost wasn’t just in engineering time — it was in customer compensation for erroneous charges.</p><p><strong>The Rounding Error</strong></p><p>Currency conversion is deceptively complex. When you’re converting between currencies, you’re doing floating-point math. Floating-point math is a minefield.</p><p>A fintech company I know was processing international payments. They converted USD to JPY (no decimal places, unlike most currencies) using naive floating-point arithmetic. A $100 transaction converted to ¥10,000.something. The system rounded. Over millions of transactions, those fractions of cents turned into thousands of dollars in discrepancies.</p><p>The nightmare: they didn’t notice for months. By the time they caught it, they had to reconcile two years of transactions, identify which ones were affected, issue credits, and explain to their payment processor what happened. The regulatory and reputational cost far exceeded the actual money lost.</p><p><strong>The Retry That Never Stops</strong></p><p>A payment retry mechanism without circuit breaker logic. Transaction fails, retry. Still fails, retry again. Still fails, retry again. Oh, you have exponential backoff? Great. But what if the issue is that your payment gateway is down?</p><p>I’ve seen systems that kept retrying payments for hours, sometimes days, because no one had implemented proper failure detection. Result: the same payment being attempted 20+ times, leading to multiple charges, multiple chargebacks, and an angry customer support team drowning in refund requests.</p><p><strong>What Production Rigor Actually Looks Like</strong></p><p>After years of dealing with these nightmares, here’s what I’ve learned about building payment systems that don’t implode:</p><p><strong>1. Idempotency Is Non-Negotiable</strong></p><p>Every payment operation must be idempotent. If the same request is processed twice, the result should be identical. This means:</p><ul><li>Unique request identifiers (idempotency keys)</li><li>Database-level checks before processing</li><li>No side effects on retry</li></ul><pre>java<br>public PaymentResult processPayment(String idempotencyKey, PaymentRequest request) {<br>    // Check if this request has already been processed<br>    Optional&lt;PaymentResult&gt; existingResult = <br>        paymentStore.findByIdempotencyKey(idempotencyKey);<br>    <br>    if (existingResult.isPresent()) {<br>        return existingResult.get();  // Return cached result<br>    }<br>    <br>    // Process payment only if not already processed<br>    PaymentResult result = executePaymentTransaction(request);<br>    paymentStore.save(idempotencyKey, result);<br>    return result;<br>}</pre><p>This single pattern prevents double charges, retry-induced chaos, and reconciliation nightmares.</p><p><strong>2. ACID Transactions Are Your Friend</strong></p><p>Payment systems need strong consistency. Not eventual consistency. Not “we’ll get there.” Transactions must be ACID-compliant at the database level.</p><p>This means:</p><ul><li>Database transactions that span multiple operations</li><li>No distributed transactions without a proper saga pattern</li><li>Locks where needed (with careful deadlock management)</li></ul><p>The temptation to optimize for throughput at the expense of consistency is strong. Resist it. A 10% performance drop is worth avoiding a 0.01% data corruption rate in payment systems.</p><p><strong>3. Reconciliation Is Your Safety Net</strong></p><p>You will have bugs. You will have timing issues. You will have edge cases you didn’t think of. Reconciliation catches them.</p><p>Daily reconciliation against your payment processor. Monthly audit of all transactions. Automated alerts for discrepancies. This is non-negotiable operational hygiene.</p><pre>java<br>public void reconcileDailyTransactions() {<br>    List&lt;Transaction&gt; ourTransactions = getOurTransactions(yesterday);<br>    List&lt;Transaction&gt; processorTransactions = getProcessorTransactions(yesterday);<br>    <br>    Set&lt;String&gt; mismatches = findMismatches(ourTransactions, processorTransactions);<br>    <br>    if (!mismatches.isEmpty()) {<br>        alertOps(&quot;Transaction mismatch detected: &quot; + mismatches.size());<br>        for (String transactionId : mismatches) {<br>            // Investigate, log, and potentially refund<br>        }<br>    }<br>}</pre><p>Reconciliation doesn’t prevent bugs, but it catches them before they become regulatory issues.</p><p><strong>4. Handle Failures Explicitly</strong></p><p>Payments fail. Gateways timeout. Networks are unreliable. Don’t pretend otherwise.</p><p>Implement explicit failure handling:</p><ul><li>Timeout policies (and stick to them)</li><li>Circuit breakers for external dependencies</li><li>Clear failed transaction states (not just “pending”)</li><li>Manual intervention workflows for ambiguous cases</li></ul><p>A transaction that times out isn’t “processing.” It’s failed until proven otherwise. If you can’t definitively know the result, your system should mark it as ambiguous and require manual investigation.</p><p><strong>5. Audit Everything</strong></p><p>Every payment decision should be auditable. Who approved this transaction? What rules were applied? Why was it charged this amount? Why was it refunded? What was the customer’s account state at this moment?</p><p>This isn’t just for compliance. It’s your debugging tool when things go wrong at 3 AM.</p><pre>java<br>@Transactional<br>public void recordPaymentAudit(PaymentTransaction transaction, <br>                               String action, <br>                               String reason) {<br>    AuditLog log = new AuditLog();<br>    log.setTransactionId(transaction.getId());<br>    log.setAction(action);<br>    log.setReason(reason);<br>    log.setUserBalance(transaction.getUser().getBalance());<br>    log.setTimestamp(Instant.now());<br>    log.setDetails(serializeTransaction(transaction));<br>    <br>    auditStore.save(log);<br>}</pre><p><strong>The Business Reality</strong></p><p>Here’s what no one tells you: the cost of payment system bugs isn’t primarily technical. It’s financial and legal.</p><p>A $5,000 engineering effort to fix a bug that loses $50 a day seems unjustified to cost-conscious managers. Until that bug compounds into $10,000 in losses plus a regulatory audit plus customer churn. Then suddenly it’s a $500,000 problem.</p><p>This is why payment systems demand a different discipline. It’s not about being slow. It’s about being thoughtful. It’s about treating every feature as if it will eventually fail and asking: when it does, will we lose money?</p><p><strong>The Checklist</strong></p><p>Before you deploy anything in a payment system, ask yourself:</p><ul><li>Is this operation idempotent?</li><li>Have I tested failure scenarios?</li><li>Can I reconcile this transaction tomorrow?</li><li>What happens if the external dependency times out?</li><li>Can I refund this if needed?</li><li>Is this transaction visible in audit logs?</li><li>Have I tested at expected peak load?</li><li>Does this comply with our payment processor’s requirements?</li><li>What’s the worst case scenario, and is it acceptable?</li><li>Could this issue compound across millions of transactions?</li></ul><p>If you can answer “yes” to all of these, you’re probably safe. If you’re not sure about even one, you’ve found your next problem to solve.</p><p><strong>Closing Thought</strong></p><p>Payment systems are unglamorous. They don’t have fancy UIs or viral growth loops. They’re not what impresses at tech talks. But they’re where billions of dollars in commerce actually happens.</p><p>Build them carefully. Treat every decision as if it will eventually cost you money — because it will. Test failure modes. Implement reconciliation. Audit everything. Stay paranoid.</p><p>Your future self, when debugging a production payment issue at 2 AM, will thank you for it.</p><p><strong>Have you built payment systems? What’s the most expensive mistake you’ve caught before it went to production? I’d love to hear your stories in the comments.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2cce9c9013e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Learning: How to Speak So People Listen at the Workplace]]></title>
            <link>https://medium.com/@deepasingh1017/my-learning-how-to-speak-so-people-listen-at-the-workplace-7979abd10338?source=rss-16209d038d37------2</link>
            <guid isPermaLink="false">https://medium.com/p/7979abd10338</guid>
            <category><![CDATA[public-speaking]]></category>
            <category><![CDATA[workplace-culture]]></category>
            <dc:creator><![CDATA[Deepa Singh]]></dc:creator>
            <pubDate>Sat, 20 Jun 2026 08:49:49 GMT</pubDate>
            <atom:updated>2026-06-20T08:49:49.095Z</atom:updated>
            <content:encoded><![CDATA[<p>Communication is not just about speaking English well or using impressive words. At the workplace, communication is about <strong>trust, clarity, confidence, and respect</strong>. The way we speak can either build a connection or create distance.</p><p>The human voice is a powerful instrument. It can motivate people, solve conflicts, create trust, and influence decisions. But many times, even when we have good ideas, people may not listen to us because of our speaking habits. This is especially important in the workplace, where every conversation matters — whether it is a team discussion, a meeting with a manager, an interview, a presentation, or a conflict resolution call.</p><h3>Avoiding Poor Speaking Habits</h3><p>Some habits reduce the impact of our communication.</p><p>For example, <strong>gossiping</strong> about someone who is not present damages trust. In a professional environment, people notice this. If we speak badly about one person, others may feel that we can do the same about them later.</p><p>Another habit is <strong>judging</strong>. When people feel judged, they stop listening openly. At work, instead of judging someone’s mistake, it is better to understand the situation and discuss the solution.</p><p><strong>Negativity and complaining</strong> are also very common workplace habits. Sometimes we complain about workload, deadlines, management, meetings, or other teams. But constant complaining spreads stress. It does not solve the problem. A better approach is to speak about challenges with a solution-oriented mindset.</p><p>I also learned that making <strong>excuses</strong> weakens credibility. In the workplace, taking responsibility is very important. Even if something goes wrong, saying, “I missed this, and I will fix it,” is much more powerful than blaming others.</p><p>Another habit to avoid is <strong>exaggeration</strong>. If we exaggerate too much, people slowly stop trusting our words. Professional communication should be honest and balanced.</p><p>Finally, <strong>dogmatism</strong> means presenting opinions as facts. At work, this can create unnecessary arguments. It is better to say, “In my understanding…” or “Based on the current data…” instead of forcing our opinion as the only truth.</p><h3>The HAIL Principle</h3><p>The most useful framework from the talk is <strong>HAIL</strong>:</p><ul><li><strong>H — Honesty</strong></li><li><strong>A — Authenticity</strong></li><li><strong>I — Integrity</strong></li><li><strong>L — Love or goodwill</strong></li></ul><p>For the workplace, this means we should speak truthfully, be ourselves, keep our commitments, and communicate with good intention.</p><p><strong>Honesty</strong> means being clear and direct without being rude.<br><strong>Authenticity</strong> means not pretending to be someone else.<br><strong>Integrity</strong> means doing what we say.<br><strong>Goodwill</strong> means speaking with respect and wanting a positive outcome for everyone.</p><p>This is very important because people do not only listen to our words. They also sense our intention. If our intention is good, even difficult feedback can be received better.</p><h3>How We Say Something Also Matters</h3><p>Communication is not only about <strong>what we say</strong>, but also <strong>how we say it</strong>.</p><p>At work, tone, pace, pause, volume, and voice quality matter a lot.</p><p>If we speak too fast, people may feel rushed or confused. If we speak too slowly without energy, people may lose interest. The right pace helps people follow our thoughts clearly.</p><p>Using <strong>silence</strong> is also powerful. In meetings or interviews, we do not need to fill every gap with “umm” or “ahh.” A small pause before answering can show confidence and maturity.</p><p><strong>Pitch and volume</strong> also change the meaning of our message. A calm and steady voice can make us sound more confident. Speaking too loudly can feel aggressive, while speaking too softly can make us sound unsure. The goal is to sound clear, calm, and confident.</p><h3>My Workplace Application</h3><p>I can apply this learning in many workplace situations.</p><p>During meetings, I should avoid interrupting and speak with clarity. Instead of complaining about a problem, I should explain the issue, impact, and possible solution.</p><p>When discussing mistakes, I should avoid blame and focus on ownership. For example:</p><blockquote><em>“This issue happened because this validation was missed. I will add the check and also verify the related flows.”</em></blockquote><p>This sounds much more professional than blaming another team or system.</p><p>In interviews, this learning is even more useful. I should speak honestly about my experience, explain my thought process clearly, and avoid exaggerating. If I do not know something, I can say:</p><blockquote><em>“I have not worked deeply on this, but my understanding is…”</em></blockquote><p>This shows honesty and confidence.</p><p>When speaking with managers or senior people, I should use a calm tone, structured points, and meaningful pauses. This will make my communication more powerful and professional.</p><h3>Conclusion</h3><p>My biggest learning is that powerful speaking is not about speaking loudly or using big words. It is about speaking with <strong>honesty, clarity, responsibility, and respect</strong>.</p><p>At the workplace, people listen to those who are trustworthy, solution-oriented, and clear in their communication. Avoiding gossip, judgment, negativity, excuses, exaggeration, and rigid opinions can make us better communicators.</p><p>The HAIL principle is something I want to remember in my professional life:</p><blockquote><strong><em>Be honest. Be authentic. Keep your word. Speak with goodwill.</em></strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7979abd10338" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Securing Juspay Webhooks: Why HMAC-SHA256 Matters And How It Works]]></title>
            <link>https://medium.com/@deepasingh1017/securing-juspay-webhooks-why-hmac-sha256-matters-and-how-it-works-8c5cb6d0678b?source=rss-16209d038d37------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c5cb6d0678b</guid>
            <category><![CDATA[juspay]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[payment-gateway]]></category>
            <dc:creator><![CDATA[Deepa Singh]]></dc:creator>
            <pubDate>Mon, 15 Jun 2026 12:09:29 GMT</pubDate>
            <atom:updated>2026-06-15T12:09:29.638Z</atom:updated>
            <content:encoded><![CDATA[<h3>The $50,000 Webhook</h3><p>Imagine this: A customer successfully pays $50,000 for an order. Your payment system receives a webhook notification. Everything looks legitimate. You process the transaction.</p><p>Then, 30 seconds later, another webhook arrives — same order ID, but this time the status says FAILED. Your system refunds the customer. The order is cancelled.</p><p><strong>What just happened?</strong> An attacker intercepted the webhook and changed the status field. Since you weren’t verifying signatures, you treated it as legitimate. $50,000 vanished.</p><p>This scenario is exactly why webhook security matters.</p><h3>The Two Questions Every Webhook Must Answer</h3><p>Before your system processes <em>any</em> webhook from Juspay, you need answers to these questions:</p><ol><li><strong>Is this actually from Juspay?</strong> (Authenticity)</li><li><strong>Has the data been modified in transit?</strong> (Integrity)</li></ol><p>Username and password alone can’t answer the second question. That’s where <strong>HMAC-SHA256 signature verification</strong> comes in.</p><h3>Username-Password vs. Response Key: What’s the Difference?</h3><p>When implementing Juspay webhooks, developers often ask: <em>Why can’t we just use username and password?</em></p><p>The answer is that they solve different problems.</p><h3>Username-Password: “Who Are You?”</h3><p>Username-password authentication answers one question:</p><ul><li><strong>Is the sender who they claim to be?</strong></li></ul><p>It acts as a simple gatekeeper. “You logged in with valid credentials, so you have permission to send webhooks.”</p><p>But it answers <em>nothing</em> about:</p><ul><li>Whether the webhook body was modified</li><li>Whether the payload got corrupted in transit</li><li>Whether an attacker intercepted and rewrote the data</li></ul><h3>Response Key (HMAC-SHA256): “Is This Authentic AND Untouched?”</h3><p>The Response Key uses cryptography to verify:</p><ul><li><strong>Authenticity:</strong> Only Juspay (and you) know the secret key, so only Juspay can generate valid signatures</li><li><strong>Integrity:</strong> Even a single-character change in the payload produces a completely different signature</li><li><strong>Tamper Detection:</strong> Attackers can’t fake a valid signature without the secret key</li></ul><p>This is why financial institutions, payment processors, and fintech platforms use HMAC across the industry. It’s not just good practice — it’s the standard.</p><h3>How HMAC-SHA256 Actually Works</h3><p>Think of HMAC-SHA256 like a tamper-evident seal on a package.</p><h3>Step 1: The Shared Secret</h3><p>Both Juspay and your server know a secret key:</p><pre>Response Key: my-super-secret-response-key-12345</pre><p>This key is private. Never log it. Never commit it to git. Never show it to anyone.</p><h3>Step 2: Juspay Creates a Signature</h3><p>When Juspay generates a webhook, it calculates:</p><pre>signature = HMAC_SHA256(<br>  body_content,<br>  response_key<br>)</pre><p>For example, if the webhook body is:</p><pre>{<br>  &quot;order_id&quot;: &quot;ORD123456&quot;,<br>  &quot;transaction_id&quot;: &quot;TXN789&quot;,<br>  &quot;status&quot;: &quot;CHARGED&quot;,<br>  &quot;amount&quot;: 50000<br>}</pre><p>Juspay computes the HMAC and gets something like:</p><pre>a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6</pre><h3>Step 3: The Webhook Arrives</h3><p>Juspay sends your server:</p><pre>POST /api/webhooks/juspay</pre><pre>Headers:<br>  x-juspay-signature: a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6</pre><pre>Body:<br>{<br>  &quot;order_id&quot;: &quot;ORD123456&quot;,<br>  &quot;transaction_id&quot;: &quot;TXN789&quot;,<br>  &quot;status&quot;: &quot;CHARGED&quot;,<br>  &quot;amount&quot;: 50000<br>}</pre><p>The signature travels alongside the payload as proof.</p><h3>Step 4: Your Server Verifies</h3><p>When you receive the webhook, you:</p><ol><li>Extract the signature from the header</li><li>Recalculate the HMAC using the same response key</li><li>Compare them</li></ol><pre>String receivedSignature = request.getHeader(&quot;x-juspay-signature&quot;);<br>String calculatedSignature = HMAC_SHA256(body, responseKey);</pre><pre>if (calculatedSignature.equals(receivedSignature)) {<br>    // Safe to process<br>} else {<br>    // Reject immediately<br>}</pre><p>If they match: <strong>Process the webhook</strong></p><p>If they don’t match: <strong>Reject with 401 Unauthorized</strong></p><h3>What Happens When Signatures Match vs. Don’t Match</h3><h3>Scenario 1: Legitimate Webhook</h3><p><strong>Received Signature:</strong></p><pre>a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6</pre><p><strong>Your Calculated Signature:</strong></p><pre>a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6</pre><p>Result: They match → Webhook is trusted → Process the payment</p><h3>Scenario 2: Attacker Modifies Status</h3><p>An attacker intercepts the webhook and changes:</p><pre>&quot;status&quot;: &quot;CHARGED&quot;</pre><p>to:</p><pre>&quot;status&quot;: &quot;FAILED&quot;</pre><p><strong>Received Signature (original):</strong></p><pre>a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6</pre><p><strong>Your Calculated Signature (modified payload):</strong></p><pre>q9w8e7r6t5y4u3i2o1p0l9k8j7h6g5f4</pre><p>Result: They don’t match → Webhook is rejected → Payment is NOT processed</p><p>Even though the attacker has the webhook URL, they can’t forge a valid signature without your Response Key. This is what makes HMAC so powerful.</p><h3>The Secure Webhook Flow</h3><p>Here’s how it should work in your application:</p><pre>Webhook Received<br>    ↓<br>[VERIFY SIGNATURE] ← This happens FIRST<br>    ↓<br>Signature Valid? <br>    ├─ YES → Parse payload → Update order status → Trigger business logic<br>    └─ NO → Return 401 → Log security alert → Stop</pre><p>The key principle: <strong>Verify First, Process Later</strong></p><p>Never parse the webhook body until you’ve confirmed the signature is valid. This ensures malicious or corrupted requests never reach your payment processing logic.</p><h3>Why This Matters: The Layered Security Model</h3><p>Webhook security isn’t one big lock — it’s multiple layers:</p><p>Layer What It Does Example <strong>HTTPS/TLS</strong> Encrypts data in transit Prevents eavesdropping <strong>IP Whitelisting</strong> Allows requests only from known servers Blocks requests from random IPs <strong>Username-Password</strong> Authenticates the source Confirms Juspay is who it claims <strong>HMAC-SHA256</strong> Verifies data wasn’t modified Detects tampering or forgery <strong>Authorization Checks</strong> Validates business rules “Does this user own this order?”</p><p>In this stack, <strong>HMAC-SHA256 is your primary line of defense against data tampering</strong>. It’s the reason you can trust that the webhook you received hasn’t been modified.</p><h3>The Real-World Impact</h3><p>Let’s be specific about what HMAC-SHA256 protects you from:</p><p><strong>Without signature verification:</strong></p><ul><li>An attacker changes a CHARGED status to PENDING</li><li>A disgruntled employee modifies order amounts</li><li>A script kiddie replays old webhooks with new order IDs</li><li>Corrupted data in transit goes undetected</li></ul><p><strong>With signature verification:</strong></p><ul><li>Any modification is immediately detected and rejected</li><li>Only Juspay (with the Response Key) can create valid signatures</li><li>Replayed webhooks are rejected because you don’t process duplicates</li><li>Data integrity is cryptographically guaranteed</li></ul><h3>Key Takeaways</h3><p><strong>HMAC-SHA256 verifies both authenticity and integrity</strong> — not just who sent it, but that the data is unchanged</p><p><strong>It’s cryptographically secure</strong> — attackers can’t forge valid signatures without your secret Response Key</p><p><strong>It’s industry standard</strong> — banks, payment processors, and fintech platforms use this exact approach</p><p><strong>It’s non-negotiable</strong> — don’t accept webhooks without verifying the signature first</p><p><strong>It’s your safety net</strong> — when something goes wrong with a webhook, the signature mismatch tells you immediately</p><h3>Next Steps</h3><p>If you’re implementing Juspay webhooks:</p><ol><li>Get your Response Key from the Juspay dashboard (keep it secret)</li><li>Implement signature verification before processing any webhook</li><li>Return 401 Unauthorized for signature mismatches</li><li>Log all verification failures for security monitoring</li><li>Test your verification with both valid and tampered payloads</li></ol><p>The extra few lines of code to verify signatures might seem small. But they’re the difference between a secure payment system and one that’s vulnerable to fraud.</p><p><strong>Your customers’ transactions depend on it.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c5cb6d0678b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Idempotent Payment Systems: How to Prevent Duplicate Charges]]></title>
            <link>https://medium.com/@deepasingh1017/building-idempotent-payment-systems-how-to-prevent-duplicate-charges-13cee11e5803?source=rss-16209d038d37------2</link>
            <guid isPermaLink="false">https://medium.com/p/13cee11e5803</guid>
            <category><![CDATA[payments]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[stripe]]></category>
            <dc:creator><![CDATA[Deepa Singh]]></dc:creator>
            <pubDate>Sun, 14 Jun 2026 06:53:23 GMT</pubDate>
            <atom:updated>2026-06-14T06:53:23.365Z</atom:updated>
            <content:encoded><![CDATA[<p>Implementing fault-tolerant payment processing like Stripe does</p><p>It’s 3 AM on a Friday. Your payment service crashed mid-transaction. <br>The client retried. The API timeout’d again. They retried once more.</p><p>By morning: 3 charges for 1 order. Happy customer? No. Refund process? <br>Nightmare. This is the most common problem in payment systems, and it’s why idempotency matters.</p><p>If you’re building backend systems that handle money, you need to understand how idempotent APIs prevent this disaster. This article shows you exactly how.</p><p><strong>What You’ll Learn:</strong></p><ul><li>Why idempotency is non-negotiable for payments</li><li>How Stripe implements idempotency</li><li>Step-by-step implementation in Java</li><li>Real production gotchas and how to avoid them</li><li>Testing strategies for idempotent systems</li></ul><p><strong>Why It Matters:</strong></p><ul><li>Prevents duplicate charges (legal + customer trust issue)</li><li>Reduces support tickets by 40%+ (from my experience)</li><li>Required for PCI compliance</li><li>Separates junior from senior engineers</li></ul><h3><strong>Why Duplicate Charges Happen</strong></h3><p>Let me show you what a normal payment flow looks like:</p><pre>CLIENT                          PAYMENT SERVICE<br>  |                                   |<br>  |--------POST /charge---------&gt;|   (receives request)<br>  |                              |   (validates request)<br>  |                              |   (talks to payment gateway)<br>  |                              |   (charges customer)<br>  |    &lt;-- NETWORK TIMEOUT ---   |<br>  |    (doesn&#39;t know what happened)<br>  |<br>  |--------POST /charge---------&gt;|   (SAME REQUEST AGAIN)<br>  |    (retry mechanism kicks in)|   (processes as NEW request)<br>  |                              |   (charges customer AGAIN)</pre><p><strong>The Domino Effect:</strong></p><p>In my experience at FinTech company X:</p><ul><li>Customer clicks “Pay” button</li><li>Network hiccup occurs (1 in 1000 requests)</li><li>No response received</li><li>Auto-retry triggers (sensible behavior)</li><li>But backend processes it as NEW request</li><li>Result: 2 charges for 1 order</li></ul><p><strong>Real Numbers:</strong></p><ul><li>Without idempotency: ~0.1–0.3% of transactions have duplicates</li><li>For $100M/year payment volume: $100K-300K in unwanted charges</li><li>Support costs to handle refunds: 10x the duplicate amount</li><li>Customer churn from duplicate charges: Immeasurable</li></ul><p><strong>Why It’s Hard to Detect:</strong></p><ul><li>Happens intermittently (makes it harder to debug)</li><li>User might not notice immediately</li><li>Payment gateway shows 2 transactions, but API only recorded 1</li><li>Reconciliation becomes nightmare</li></ul><p><strong>Current Bad Solutions:</strong></p><pre>❌ &quot;Hope the network doesn&#39;t fail&quot;<br>   → 0.1% failure = huge losses at scale</pre><pre>❌ &quot;Database constraints to prevent duplicates&quot;<br>   → What if request arrives twice in same second?<br>   → Race condition still possible</pre><pre>❌ &quot;Deduplication in application layer&quot;<br>   → Works, but complex and prone to bugs<br>   → Different across teams/services</pre><pre>✓ Idempotency (what we&#39;ll implement)<br>   → Clean, standardized, battle-tested<br>   → Industry standard for good reason</pre><p><strong>What is Idempotency?</strong></p><p>Idempotent = “same result no matter how many times called”</p><pre>Mathematically: f(f(x)) = f(x)</pre><pre>For payments:<br>  Request 1: POST /charge {amount: 100} → charges 100, returns receipt_123<br>  Request 2: POST /charge {amount: 100, idempotency_key: &quot;xyz&quot;} → charges 100, returns same receipt_123<br>  Request 3: POST /charge {amount: 100, idempotency_key: &quot;xyz&quot;} → doesn&#39;t charge again, returns receipt_123</pre><p><strong>How Stripe Does It:</strong></p><p>Stripe uses idempotency_key header:</p><pre>POST /v1/charges<br>Headers:<br>  Authorization: Bearer sk_test_xxx<br>  Idempotency-Key: acct_1234_tx_xyz_unique_key</pre><pre>Body:<br>  amount=2000<br>  currency=usd<br>  source=tok_visa</pre><pre>Response (First request):<br>{<br>  &quot;id&quot;: &quot;ch_1234567890&quot;,<br>  &quot;amount&quot;: 2000,<br>  &quot;status&quot;: &quot;succeeded&quot;,<br>  &quot;idempotency_key&quot;: &quot;acct_1234_tx_xyz_unique_key&quot;<br>}</pre><pre>Response (Retry with same key):<br>{<br>  &quot;id&quot;: &quot;ch_1234567890&quot;,  ← SAME charge ID<br>  &quot;amount&quot;: 2000,<br>  &quot;status&quot;: &quot;succeeded&quot;,<br>  &quot;idempotency_key&quot;: &quot;acct_1234_tx_xyz_unique_key&quot;<br>}</pre><p><strong>Key Design Principles:</strong></p><p><strong>Unique Idempotency Key:</strong></p><ul><li>Generated by CLIENT (not server)</li><li>Example: userId_timestamp_randomString</li><li>Or UUID: 550e8400-e29b-41d4-a716-446655440000</li></ul><p><strong>Server-Side Storage:</strong></p><ul><li>Store every request’s idempotency key</li><li>Store its response</li><li>If same key arrives: return cached response</li></ul><p><strong>Time Window:</strong></p><ul><li>Keep idempotency mappings for 24 hours (configurable)</li><li>After 24 hours: can process as new request</li><li>Why? Storage space + security (old keys shouldn’t apply forever)</li></ul><p><strong>Scope:</strong></p><ul><li>Idempotency key is unique per API endpoint</li><li>Key xyz for /charge ≠ Key xyz for /refund</li><li>Scope by userId + endpoint + key</li></ul><p><strong>Architecture Overview:</strong></p><pre>CLIENT REQUEST:<br>  POST /api/v1/charges<br>  {<br>    idempotency_key: &quot;user_123_tx_1234567890_abcdef&quot;,<br>    amount: 100,<br>    currency: &quot;USD&quot;,<br>    customer_id: &quot;cust_123&quot;<br>  }</pre><pre>SERVER FLOW:<br>  ┌─────────────────────────────────────┐<br>  │ 1. Receive request                   │<br>  └──────────────┬──────────────────────┘<br>                 │<br>  ┌──────────────▼──────────────────────┐<br>  │ 2. Check Idempotency Cache          │<br>  │    SELECT * FROM idempotency_cache  │<br>  │    WHERE key = ?                    │<br>  └──────────────┬──────────────────────┘<br>                 │<br>           ┌─────▼─────┐<br>           │  Found?   │<br>           └─────┬─────┘<br>                 │<br>        ┌────────┴────────┐<br>        │                 │<br>       YES               NO<br>        │                 │<br>    ┌───▼────┐      ┌─────▼──────────────┐<br>    │Return  │      │3. Process Payment   │<br>    │cached  │      │   - Validate input  │<br>    │response│      │   - Charge gateway  │<br>    └────────┘      │   - Create order    │<br>                    └─────┬──────────────┘<br>                          │<br>                    ┌─────▼──────────────┐<br>                    │4. Store Result      │<br>                    │   INSERT INTO       │<br>                    │   idempotency_cache │<br>                    └─────┬──────────────┘<br>                          │<br>                    ┌─────▼──────────────┐<br>                    │5. Return Response   │<br>                    │   {                 │<br>                    │     charge_id,      │<br>                    │     amount,         │<br>                    │     status          │<br>                    │   }                 │<br>                    └────────────────────┘</pre><h3>CODE IMPLEMENTATION</h3><p><strong>Title: Step-by-Step Java/Spring Boot Implementation</strong></p><p><strong>Step 1: Database Schema</strong></p><pre>-- Table to store idempotency mappings<br>CREATE TABLE idempotency_cache (<br>    id BIGINT PRIMARY KEY AUTO_INCREMENT,<br>    api_path VARCHAR(255) NOT NULL,<br>    idempotency_key VARCHAR(255) NOT NULL,<br>    user_id BIGINT NOT NULL,<br>    request_hash VARCHAR(64) NOT NULL,<br>    response_body LONGTEXT NOT NULL,<br>    response_status INT NOT NULL,<br>    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,<br>    expires_at TIMESTAMP,<br>    <br>    UNIQUE KEY unique_idempotency (api_path, user_id, idempotency_key),<br>    INDEX idx_expires (expires_at)<br>);</pre><pre>-- Partition by month for performance<br>PARTITION BY RANGE (YEAR_MONTH(created_at)) (<br>    PARTITION p_202606 VALUES LESS THAN (202607),<br>    PARTITION p_202607 VALUES LESS THAN (202608),<br>    PARTITION p_future VALUES LESS THAN MAXVALUE<br>);</pre><p><strong>Step 2: Request DTO</strong></p><pre>@Data<br>@Validated<br>public class ChargeRequest {<br>    @NotBlank<br>    private String idempotencyKey;<br>    <br>    @NotNull<br>    @Positive<br>    private BigDecimal amount;<br>    <br>    @NotBlank<br>    private String currency;<br>    <br>    @NotNull<br>    private Long customerId;<br>    <br>    private String description;<br>    <br>    // Validation<br>    @AssertTrue(message = &quot;Amount must be &gt; 0&quot;)<br>    private boolean isValidAmount() {<br>        return amount.compareTo(BigDecimal.ZERO) &gt; 0;<br>    }<br>}</pre><p><strong>Step 3: Idempotency Service</strong></p><pre>@Service<br>@Slf4j<br>public class IdempotencyService {<br>    <br>    @Autowired<br>    private IdempotencyCacheRepository cacheRepo;<br>    <br>    /**<br>     * Check if this request was already processed<br>     */<br>    public Optional&lt;IdempotencyResponse&gt; getFromCache(<br>            String apiPath, <br>            Long userId, <br>            String idempotencyKey) {<br>        <br>        Optional&lt;IdempotencyCache&gt; cached = cacheRepo.findByApiPathAndUserIdAndIdempotencyKey(<br>            apiPath, userId, idempotencyKey<br>        );<br>        <br>        if (cached.isPresent()) {<br>            log.info(&quot;Idempotency hit for key: {}&quot;, idempotencyKey);<br>            IdempotencyCache cache = cached.get();<br>            return Optional.of(new IdempotencyResponse(<br>                cache.getResponseBody(),<br>                cache.getResponseStatus()<br>            ));<br>        }<br>        <br>        return Optional.empty();<br>    }<br>    <br>    /**<br>     * Store request result for future retries<br>     */<br>    public void cacheResponse(<br>            String apiPath,<br>            Long userId,<br>            String idempotencyKey,<br>            String requestHash,<br>            String responseBody,<br>            int responseStatus) {<br>        <br>        IdempotencyCache cache = new IdempotencyCache();<br>        cache.setApiPath(apiPath);<br>        cache.setUserId(userId);<br>        cache.setIdempotencyKey(idempotencyKey);<br>        cache.setRequestHash(requestHash);<br>        cache.setResponseBody(responseBody);<br>        cache.setResponseStatus(responseStatus);<br>        cache.setExpiresAt(LocalDateTime.now().plusDays(1)); // 24 hour window<br>        <br>        cacheRepo.save(cache);<br>        log.info(&quot;Cached response for key: {}&quot;, idempotencyKey);<br>    }<br>    <br>    /**<br>     * Generate request hash to prevent mutation attacks<br>     */<br>    public String generateRequestHash(ChargeRequest request) {<br>        String content = request.getAmount() + &quot;|&quot; <br>                       + request.getCurrency() + &quot;|&quot; <br>                       + request.getCustomerId();<br>        return DigestUtils.sha256Hex(content);<br>    }<br>}</pre><p><strong>Step 4: Payment Controller with Idempotency</strong></p><pre>@RestController<br>@RequestMapping(&quot;/api/v1&quot;)<br>@Slf4j<br>public class PaymentController {<br>    <br>    @Autowired<br>    private IdempotencyService idempotencyService;<br>    <br>    @Autowired<br>    private PaymentService paymentService;<br>    <br>    @PostMapping(&quot;/charges&quot;)<br>    public ResponseEntity&lt;?&gt; createCharge(<br>            @Valid @RequestBody ChargeRequest request,<br>            @RequestHeader(&quot;Authorization&quot;) String token) {<br>        <br>        Long userId = extractUserIdFromToken(token);<br>        String apiPath = &quot;/api/v1/charges&quot;;<br>        <br>        // Step 1: Check cache<br>        Optional&lt;IdempotencyResponse&gt; cached = <br>            idempotencyService.getFromCache(<br>                apiPath, <br>                userId, <br>                request.getIdempotencyKey()<br>            );<br>        <br>        if (cached.isPresent()) {<br>            log.info(&quot;Returning cached response for idempotency key: {}&quot;, <br>                    request.getIdempotencyKey());<br>            return ResponseEntity<br>                .status(cached.get().getStatus())<br>                .body(cached.get().getBody());<br>        }<br>        <br>        try {<br>            // Step 2: Process payment<br>            ChargeResponse response = paymentService.processCharge(<br>                userId, <br>                request<br>            );<br>            <br>            // Step 3: Cache the response<br>            String requestHash = idempotencyService.generateRequestHash(request);<br>            idempotencyService.cacheResponse(<br>                apiPath,<br>                userId,<br>                request.getIdempotencyKey(),<br>                requestHash,<br>                objectToJson(response),<br>                200<br>            );<br>            <br>            return ResponseEntity.ok(response);<br>            <br>        } catch (PaymentException e) {<br>            <br>            // Step 4: Cache error response too!<br>            String requestHash = idempotencyService.generateRequestHash(request);<br>            ErrorResponse errorResponse = new ErrorResponse(e.getMessage());<br>            <br>            idempotencyService.cacheResponse(<br>                apiPath,<br>                userId,<br>                request.getIdempotencyKey(),<br>                requestHash,<br>                objectToJson(errorResponse),<br>                400<br>            );<br>            <br>            return ResponseEntity<br>                .badRequest()<br>                .body(errorResponse);<br>        }<br>    }<br>}</pre><p><strong>Step 5: Usage Example for Client</strong></p><pre>String idempotencyKey = UUID.randomUUID().toString();</pre><pre>ChargeRequest request = new ChargeRequest();<br>request.setIdempotencyKey(idempotencyKey);<br>request.setAmount(new BigDecimal(&quot;100.00&quot;));<br>request.setCurrency(&quot;USD&quot;);<br>request.setCustomerId(12345L);</pre><pre>// First attempt<br>HttpResponse&lt;ChargeResponse&gt; response1 = <br>    httpClient.post(&quot;/api/v1/charges&quot;)<br>        .header(&quot;Authorization&quot;, &quot;Bearer token&quot;)<br>        .body(request)<br>        .execute();</pre><pre>// Network fails, client retries with SAME idempotency key<br>// Server will return same response without double-charging!<br>HttpResponse&lt;ChargeResponse&gt; response2 = <br>    httpClient.post(&quot;/api/v1/charges&quot;)<br>        .header(&quot;Authorization&quot;, &quot;Bearer token&quot;)<br>        .body(request)<br>        .execute();</pre><pre>// response1.chargeId == response2.chargeId (same charge!)</pre><p><strong>Production Lessons &amp; Common Mistakes</strong></p><p><strong>Gotcha 1: Request Body Mutation</strong></p><p>Problem: What if the request is:<br> First attempt: {idempotency_key: “xyz”, amount: 100}<br> Retry attempt: {idempotency_key: “xyz”, amount: 200} ← Different amount!</p><pre>Solution:<br>  Store request hash (SHA-256 of important fields)<br>  If hash doesn&#39;t match: REJECT (don&#39;t process)<br>  <br>  Raise exception: &quot;Idempotency key reused with different data&quot;</pre><p><strong>Gotcha 2: Timeout vs. Failure</strong></p><p>Problem: How long to cache?</p><pre>Too Short (&lt; 1 hour):<br>  - Legitimate retry might be new transaction<br>  - Defeats purpose of idempotency</pre><pre>Too Long (&gt; 7 days):<br>  - Old cache hits legitimate new request<br>  - Same user, same key, different intent</pre><pre>Solution: 24 hours standard (Stripe&#39;s approach)<br>  - Long enough for reasonable retries<br>  - Short enough to not cause issues<br>  - Consider your SLA and failure modes</pre><p><strong>Gotcha 3: Database Race Conditions</strong></p><p>Problem: Two requests arrive in same millisecond with same key!</p><pre>Scenario:<br>  Request 1: SELECT from cache → miss → processes payment<br>  Request 2: SELECT from cache → miss → processes payment AGAIN<br>  (Both happened before either INSERT)</pre><pre>Solution:<br>  Use UNIQUE constraint + first INSERT wins pattern<br>  <br>  INSERT INTO idempotency_cache (...)<br>  ON DUPLICATE KEY UPDATE created_at = created_at;<br>  <br>  If insert fails: retrieve the existing record<br>  Return its cached response</pre><p><strong>Gotcha 4: Cache Storage Explosion</strong></p><p>Problem: Millions of requests/day = huge cache table</p><pre>At 1M requests/day × 365 days × 2KB per row = 730GB/year!</pre><pre>Solution:<br>  1. Partition by month (as shown in schema)<br>  2. Auto-expire after 24 hours<br>  3. Archive old partitions<br>  4. Use Redis for hot cache (24h window)<br>  5. Use PostgreSQL/MySQL for archival</pre><pre>Example Redis + DB hybrid:<br>  - First 24 hours: Redis (fast)<br>  - After 24 hours: Archive to DB<br>  - Cleanup job deletes rows older than 24h</pre><p><strong>Gotcha 5: Distributed Systems</strong></p><p>Problem: Multiple API servers processing same request!</p><pre>Scenario:<br>  Server A: Receives request → Processing → Stores in LOCAL cache<br>  Server B: Receives retry → Not in its cache → Processes AGAIN</pre><pre>Solution:<br>  Use SHARED cache (Redis/Memcached)<br>  NOT application-level cache<br>  <br>  Wrong: @Cacheable on method<br>  Right: Redis + Spring Cache Manager</pre><h3>TESTING STRATEGY</h3><p><strong>How to Test Idempotency</strong></p><pre>@SpringBootTest<br>public class IdempotencyTest {<br>    <br>    @Test<br>    public void testIdempotentChargeCreation() {<br>        String idempotencyKey = &quot;test_key_123&quot;;<br>        <br>        // First charge<br>        ChargeResponse response1 = paymentController.createCharge(<br>            createRequest(idempotencyKey, 100.0),<br>            &quot;Bearer token&quot;<br>        );<br>        <br>        // Verify charge created<br>        assertNotNull(response1.getChargeId());<br>        assertEquals(100.0, response1.getAmount());<br>        <br>        // Retry with same key<br>        ChargeResponse response2 = paymentController.createCharge(<br>            createRequest(idempotencyKey, 100.0),<br>            &quot;Bearer token&quot;<br>        );<br>        <br>        // CRITICAL ASSERTION: Same charge ID!<br>        assertEquals(response1.getChargeId(), response2.getChargeId());<br>        <br>        // Verify only ONE charge in database<br>        List&lt;Charge&gt; charges = chargeRepository.findByIdempotencyKey(idempotencyKey);<br>        assertEquals(1, charges.size());<br>    }<br>    <br>    @Test<br>    public void testIdempotencyKeyMutationRejected() {<br>        String idempotencyKey = &quot;test_key_456&quot;;<br>        <br>        // First charge with amount 100<br>        paymentController.createCharge(<br>            createRequest(idempotencyKey, 100.0),<br>            &quot;Bearer token&quot;<br>        );<br>        <br>        // Retry with DIFFERENT amount<br>        assertThrows(IdempotencyKeyMutationException.class, () -&gt;<br>            paymentController.createCharge(<br>                createRequest(idempotencyKey, 200.0),  // Different!<br>                &quot;Bearer token&quot;<br>            )<br>        );<br>    }<br>}</pre><h3>KEY TAKEAWAYS</h3><pre>✓ Idempotency prevents duplicate charges (critical for payments)</pre><pre>✓ Implementation: Cache request → Check on retry → Return cached response</pre><pre>✓ Idempotency key: Generated by CLIENT, unique per request</pre><pre>✓ Time window: 24 hours (standard)</pre><pre>✓ Database: Use UNIQUE constraint to prevent race conditions</pre><pre>✓ Caching layer: Redis for hot cache, DB for archival</pre><pre>✓ Testing: Verify same charge ID returned on retry</pre><pre>✓ Distributed systems: Shared cache, not local</pre><pre>✓ Request mutation: Hash and validate, reject if changed</pre><pre>✓ Common gotcha: Race conditions on simultaneous requests</pre><h3>RESOURCES</h3><p>- Stripe Idempotency Guide: stripe.com/docs/api/idempotent_requests<br>- AWS Payment Processing: aws.amazon.com/payment-services<br>- PCI DSS Idempotency: pci-dss.org<br>- RFC 2119: Idempotent HTTP Methods<br>- My related articles:<br> * Rate Limiting at Scale<br> * Database Connection Pooling<br> * Distributed Transaction Handling</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13cee11e5803" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Cron Job That Refused to Die: A Deployment Story]]></title>
            <link>https://medium.com/@deepasingh1017/the-cron-job-that-refused-to-die-a-deployment-story-dea003f3439a?source=rss-16209d038d37------2</link>
            <guid isPermaLink="false">https://medium.com/p/dea003f3439a</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[payment-gateway]]></category>
            <dc:creator><![CDATA[Deepa Singh]]></dc:creator>
            <pubDate>Fri, 12 Jun 2026 16:13:51 GMT</pubDate>
            <atom:updated>2026-06-12T16:13:51.069Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How one forgotten database entry failed our deployment during a planned downtime — and what it taught me about “deleting” things in production.</em></p><p>Yesterday we had a deployment. A planned one. Downtime window booked, checklist ready, everyone on the call. And it still failed.</p><p>Not because of some exotic bug. Not because of infrastructure. It failed because of one row in one table that we forgot existed.</p><p>Let me tell you the whole story.</p><h3>A little background</h3><p>I work on a payments system, and like most payment systems, ours doesn’t trust anyone — not even the payment gateways. Webhooks get lost. Networks flake. Gateways have bad days.</p><p>So we run a bunch of cron jobs whose only purpose is <strong>compensation</strong>. Here’s roughly how it works:</p><p>A customer pays. Ideally, the gateway — Razorpay, Juspay, whoever — sends us a webhook saying “payment succeeded.” But sometimes that webhook never arrives. The money has left the customer’s account, and our system has no idea.</p><p>That’s where the compensation crons come in. They wake up, find payments stuck in a pending state, and poll the gateway’s status API directly: <em>“Hey, did this payment actually go through?”</em></p><ul><li>If the gateway says <strong>success</strong>, we complete the rest of the transaction — deduct the wallet balance, redeem the gift card, apply the employee discount or loyalty rewards, whatever the customer used alongside the gateway payment.</li><li>If the gateway says <strong>failed</strong>, we go the other way — refund whatever was already utilized: wallet money back, gift card balance restored, rewards returned.</li></ul><p>These crons are the safety net of the entire payment flow. Boring, invisible, and absolutely critical.</p><h3>The refactor</h3><p>Over time, we had ended up with separate compensation crons for each gateway. One for Razorpay. Then Juspay came along, and the idea of writing yet another copy-paste cron didn’t sit right with anyone.</p><p>So we did the sensible thing: we built one unified, gateway-agnostic compensation cron. One job, one piece of logic, handles every gateway. Clean.</p><p>That meant the old Razorpay-specific cron had to go. We deleted it from the code. We removed its entry from the YML configuration. We tested everything. Code review passed. We were ready.</p><h3>Deployment day</h3><p>Downtime taken. Deployment started. And then — <strong>failure</strong>. Before we even went live.</p><p>The culprit? <strong>ShedLock.</strong></p><p>If you haven’t used it, ShedLock is a library that makes sure a scheduled job runs on only one instance at a time in a distributed setup. It does this by keeping lock entries in a database table — one row per scheduled job.</p><p>Our old Razorpay cron had its entry sitting in that ShedLock table. We had deleted the cron from the code. We had deleted it from the YML. But we never deleted its row from the database.</p><p>The code said the job doesn’t exist. The database insisted it does. And the deployment failed right there, in the middle of our downtime window, with the clock ticking.</p><h3>The fix was embarrassingly simple</h3><p>One DELETE statement. That&#39;s it. Remove the stale ShedLock entry, redeploy, done.</p><p>The fix took thirty seconds. Finding out <em>why</em> the deployment failed took much longer — and cost us a chunk of our downtime window and a few grey hairs.</p><h3>What I learned</h3><p><strong>1. Deleting code is never just deleting code.</strong> A cron job doesn’t live only in your repository. It lives in config files, in lock tables, in monitoring dashboards, in alert rules, maybe in some runbook somewhere. When you “delete” something, hunt down every place it left a footprint.</p><p><strong>2. State outlives code.</strong> Your codebase is version-controlled and gets replaced on every deployment. Your database is not. Anything your code ever wrote to the database will sit there patiently, waiting to surprise you — sometimes months later.</p><p><strong>3. Make removal a checklist, not a memory exercise.</strong> We remembered the code. We remembered the YML. We forgot the table. Humans forget things — that’s exactly why checklists exist. Now, removing any scheduled job on our team means: code, config, ShedLock entry, alerts, dashboards. Written down. Every time.</p><p><strong>4. Rehearse deployments that involve deletion.</strong> We test new features heavily. But deletions feel “safe” — you’re removing code, what could break? Turns out, plenty. If a deployment removes something, it deserves a dry run in a staging environment that mirrors production data, lock tables included.</p><p><strong>5. Failing before going live is the good outcome.</strong> Honestly, we got lucky. The deployment failed <em>before</em> the changes went live, during a planned downtime. The worst version of this story is the one where everything deploys fine and the failure shows up at 2 AM on a weekend, in the middle of real customer payments.</p><h3>The takeaway</h3><p>Every system has its shadow state — the configs, lock entries, and leftover rows that your code created somewhere along the way and you’ve stopped thinking about. The code you see in your IDE is only half the story.</p><p>So the next time you delete something, ask yourself one question: <em>where else does this thing live?</em></p><p>Your future self — standing in a downtime window, watching a deployment fail — will thank you.</p><p><em>Have you ever been bitten by stale state hiding in a database? I’d love to hear your war stories in the comments.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dea003f3439a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From payment.created to payment.captured : The Payment State That Can Make or Break Your Order Flow]]></title>
            <link>https://medium.com/@deepasingh1017/from-payment-created-to-payment-captured-the-payment-state-that-can-make-or-break-your-order-flow-2294e9e01278?source=rss-16209d038d37------2</link>
            <guid isPermaLink="false">https://medium.com/p/2294e9e01278</guid>
            <category><![CDATA[razorpay]]></category>
            <category><![CDATA[payments]]></category>
            <category><![CDATA[juspay]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[payment-gateway]]></category>
            <dc:creator><![CDATA[Deepa Singh]]></dc:creator>
            <pubDate>Thu, 14 May 2026 11:19:31 GMT</pubDate>
            <atom:updated>2026-05-14T11:25:56.321Z</atom:updated>
            <content:encoded><![CDATA[<p>In payment systems, one of the most common mistakes engineers make is treating every payment webhook as a successful payment signal. But not every payment event means money has actually been received. Two events that often create confusion are <strong>payment.created</strong> and <strong>payment.captured</strong>. They may look similar at first glance, but in a real production payment flow, they represent two very different stages of the transaction lifecycle.</p><h3>The Simple Difference</h3><p>payment.created means a payment attempt has been initialized. A payment object has been created by the payment gateway, but the customer has not necessarily completed the payment yet. No money should be considered successfully received at this stage.</p><p>payment.captured means the payment has successfully completed and the amount has been captured by the merchant. This is the event that usually confirms that the payment is successful and business actions like order confirmation, invoice generation, OMS push, or shipment initiation can safely happen.</p><p>In simple words:</p><p><strong>payment.created = Payment journey started.</strong><br><strong>payment.captured = Money successfully collected.</strong></p><h3>Understanding payment.created</h3><p>When a user starts the payment process, the payment gateway may create a payment entity. For example, the user reaches the checkout page, selects UPI, card, net banking, or wallet, and proceeds toward payment. At this point, the gateway may generate a payment object and emit a payment.created event.</p><p>But this does not mean the user has paid. The user may still be entering OTP, opening the UPI app, waiting for bank authentication, or deciding whether to continue. The user may also abandon the payment completely.</p><p>A typical flow may look like this:</p><pre>User clicks Pay<br>↓<br>Payment gateway creates payment object<br>↓<br>payment.created webhook is triggered<br>↓<br>User is still in payment journey</pre><p>This is why payment.created should not be treated as final success. It is only a signal that a payment attempt exists.</p><h3>Why payment.created Is Not Enough for Order Confirmation</h3><p>If a system confirms the order on payment.created, it can create serious business issues. The customer may never complete the payment, but the merchant system may still create an order, reserve inventory, push to OMS, or even initiate fulfillment.</p><p>That means the business may end up processing an order without actually receiving money.</p><p>This is especially dangerous in e-commerce and payment systems because the downstream systems may start acting as if the transaction is successful. OMS may receive the order, warehouse may reserve stock, invoice generation may start, and customer communication may go out — all based on an event that was only an attempt, not a success.</p><h3>Understanding payment.captured</h3><p>payment.captured is different. This event means the payment has successfully moved beyond initiation and authorization, and the money has been captured. In most payment gateway flows, captured is the stage where the merchant can safely consider the payment successful.</p><p>A typical flow may look like this:</p><pre>User completes payment<br>↓<br>Bank/gateway confirms payment<br>↓<br>Amount is captured<br>↓<br>payment.captured webhook is triggered<br>↓<br>Merchant marks payment successful<br>↓<br>Order confirmation can proceed</pre><p>This is the event that should usually drive critical business actions.</p><p>After payment.captured, the system can safely update payment status to success, confirm the order, push the order to OMS, trigger shipment, generate invoice, notify the customer, or update internal ledgers.</p><h3>Real Production Example</h3><p>Imagine a customer is buying a product worth ₹2,000. The customer clicks the Pay button and selects UPI. The payment gateway creates a payment object and sends payment.created. At this point, the customer has only started the payment journey.</p><p>Now two things can happen.</p><p>In the first case, the customer completes the payment in the UPI app. The gateway receives confirmation from the bank, captures the amount, and sends payment.captured. Now the merchant system can mark the payment as successful and place the order.</p><p>In the second case, the customer closes the UPI app or payment fails due to insufficient balance. In this case, the payment was created, but never captured. If the merchant had confirmed the order on payment.created, it would have created a false successful order.</p><p>That is the core difference.</p><h3>Payment Lifecycle View</h3><p>A simplified payment lifecycle may look like this:</p><pre>Order Created<br>↓<br>Payment Created<br>↓<br>Payment Authorized / Processing<br>↓<br>Payment Captured<br>↓<br>Order Confirmed<br>↓<br>OMS Push / Shipment / Invoice</pre><p>The important thing to remember is that payment.created is an early-stage event, while payment.captured is a successful payment event.</p><h3>Why This Difference Matters in Backend Design</h3><p>Backend systems should always map payment gateway events carefully to internal payment states. A wrong mapping can create inconsistencies between payment service, order service, OMS, inventory, and customer communication.</p><p>For example, if the payment service marks payment as successful on payment.created, but the gateway later marks the payment as failed, the internal system will be out of sync with the actual payment status. This can lead to cases where the customer’s cart is not cleared correctly, order is wrongly created, or OMS receives incomplete payment information.</p><p>A better design is to treat payment.created as an informational or tracking event and treat payment.captured as the strong business success signal.</p><h3>Where payment.created Is Still Useful</h3><p>This does not mean payment.created is useless. It can still be very useful for analytics, payment attempt tracking, abandoned payment detection, fraud analysis, and debugging.</p><p>For example, if many payments are created but very few are captured, it may indicate a drop-off in the payment journey. Maybe users are facing UPI failures, OTP issues, bank downtime, or poor checkout experience. So payment.created is useful for observability and funnel analysis, but not for final order confirmation.</p><h3>Where payment.captured Should Be Used</h3><p>payment.captured should be used for business-critical success actions. These may include marking payment as successful, confirming the order, pushing to OMS, triggering invoice generation, updating ledger entries, clearing the cart, sending confirmation notification, and starting fulfillment flow.</p><p>This is because captured means the payment gateway has successfully collected the money and the merchant can proceed with confidence.</p><h3>Interview Perspective</h3><p>In interviews, this is a very strong payment-system concept to explain. Many candidates say “payment webhook came, so order confirmed,” but senior engineers clarify which webhook came and what state it represents.</p><p>A good interview answer can be:</p><p><strong>“</strong><strong>payment.created indicates that a payment attempt has been initialized, but it does not guarantee successful money collection. </strong><strong>payment.captured means the payment has successfully completed and funds have been captured, so business actions like order confirmation, OMS push, invoice generation, and shipment initiation should happen only after captured or equivalent success status.”</strong></p><h3>Final Takeaway</h3><p>In payment systems, every event has meaning. payment.created and payment.captured are not interchangeable. One represents the start of the payment journey, and the other represents successful completion.</p><p>If you treat payment.created As success, you may create orders without receiving money. If you wait forpayment.captured, You build a safer and more reliable payment flow.</p><p>A small misunderstanding in payment state mapping can become a big production incident. That is why backend engineers working on payment systems must understand the lifecycle deeply, not just process webhooks blindly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2294e9e01278" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing a Production-Grade Webhook System: Lessons from Payments, Reliability, and Scale]]></title>
            <link>https://medium.com/@deepasingh1017/designing-a-production-grade-webhook-system-lessons-from-payments-reliability-and-scale-8a319329e151?source=rss-16209d038d37------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a319329e151</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[razorpay]]></category>
            <category><![CDATA[juspay]]></category>
            <category><![CDATA[webhooks]]></category>
            <category><![CDATA[payments]]></category>
            <dc:creator><![CDATA[Deepa Singh]]></dc:creator>
            <pubDate>Thu, 14 May 2026 10:51:28 GMT</pubDate>
            <atom:updated>2026-05-14T10:51:28.400Z</atom:updated>
            <content:encoded><![CDATA[<p>When people hear the word <strong>webhook</strong>, they often think of it as just an HTTP callback. Something happens in one system, and another system gets notified. Simple.</p><p>But in real production systems, especially in domains like payments, refunds, order management, logistics, and user account workflows, a webhook is much more than a callback. It becomes a reliability boundary, a security boundary, and sometimes even the difference between a successful customer journey and a production incident.</p><p>A webhook system is an event-driven mechanism where one application automatically notifies another application when a specific event occurs. Instead of continuously polling to check whether something has changed, the source system pushes the event to a registered callback URL.</p><p>A simple analogy is a doorbell. You do not keep opening the door every few minutes to check whether someone is outside. The visitor rings the bell, and you get notified instantly. That is exactly what a webhook does in software systems.</p><p>For example, in a payment flow, when a customer completes a transaction, the payment gateway may send a webhook to the merchant system saying that the payment was successful, failed, refunded, or disputed. The merchant system can then update the order status, trigger shipment, notify OMS, or send a confirmation to the customer.</p><p>But once we move from a simple example to a real production-grade webhook system, many serious design questions appear.</p><h3>Start with Requirements, Not Technology</h3><p>Before jumping into Kafka, queues, workers, retries, or dashboards, the first step is to clarify requirements.</p><p>A strong system design discussion should start with questions like:</p><p>What type of events are triggering webhooks? Are they critical events like payment confirmation, refund success, order creation, or account activation? Or are they non-critical events like analytics, logs, or monitoring samples?</p><p>This distinction changes the architecture completely. A payment success event cannot be treated the same way as a page-view analytics event. If a payment webhook is lost, the customer may be charged but the order may not be confirmed. If an analytics event is lost, the business may lose one data point, but the customer journey is not broken.</p><p>For critical flows, we need retries, persistence, dead-letter queues, idempotency, monitoring, and acknowledgements. For non-critical flows, we may prefer fast ingestion, batching, async processing, and lightweight consumers.</p><p>This is where senior engineers stand out. They do not just pick the most popular tool. They understand the business impact first and then design accordingly.</p><h3>Functional Requirements of a Webhook System</h3><p>At a functional level, a webhook system should allow clients to register, update, and delete webhook subscriptions. A client should be able to provide a callback URL and say, “Send me these specific events here.”</p><p>For example, a client may register:</p><pre>callbackUrl = https://client.com/payment/webhook<br>eventTypes = PAYMENT_SUCCESS, PAYMENT_FAILED</pre><p>The system should expose APIs such as:</p><pre>POST /webhooks<br>PATCH /webhooks/{id}<br>DELETE /webhooks/{id}<br>GET /webhooks/{id}</pre><p>Behind the scenes, this webhook configuration must be stored in a reliable metadata store. This configuration usually contains the client ID, callback URL, subscribed event types, status, retry policy, rate limit, signing secret, created timestamp, and updated timestamp.</p><p>The system should also support event filtering. Not every client wants every event. One client may only care about payment success and failure, while another may care about refunds and chargebacks. This means the webhook system must efficiently route events only to subscribed clients.</p><p>At small scale, a simple database lookup may be enough. At high scale, we need indexing, caching, partitioning, and sometimes an internal publish-subscribe pattern to avoid scanning all subscriptions for every incoming event.</p><h3>Non-Functional Requirements Matter Even More</h3><p>Functional requirements define what the system does. Non-functional requirements define how well it does it.</p><p>For a production-grade webhook system, the key non-functional requirements are scalability, high availability, fault tolerance, security, durability, and observability.</p><p>If the system is expected to handle one billion events per day, that translates to around 11,500 events per second on average. Peak traffic can easily go beyond 100,000 events per second. A single server or monolithic setup will not survive that load. We need distributed ingestion, horizontally scalable workers, partitioning, rate limiting, and backpressure management.</p><p>High availability is also critical because webhook systems are time-sensitive. If the webhook pipeline is down, clients may not receive payment confirmations, order updates, or refund statuses. The system should be deployed across multiple availability zones, use load balancers, health checks, redundant queues, replicated databases, and multiple worker instances.</p><p>Fault tolerance is another major requirement. Failures are normal in webhook systems. Callback URLs may be down. Clients may return 500 errors. Networks may time out. DNS may fail. Workers may crash. The system must recover gracefully instead of silently dropping events.</p><h3>Delivery Guarantees: Best Effort vs At-Least-Once vs Exactly-Once</h3><p>One of the most important design discussions in webhook systems is delivery guarantee.</p><p><strong>Best effort delivery</strong> means the system will try to deliver the event, but if it fails, data loss is acceptable. This works for logs, analytics, metrics, and monitoring samples. It is simple, fast, and low-cost, but data loss is possible.</p><p><strong>At-least-once delivery</strong> means the system guarantees that the event will be delivered at least once, but duplicates are possible. This is the most common model in real-world webhook systems. If the client does not acknowledge the event, the system retries. The trade-off is that the same event may be delivered multiple times.</p><p>This is why idempotency is mandatory.</p><p>For example, if the same payment success webhook arrives twice, the receiver should check:</p><pre>Has this paymentId already been processed?</pre><p>If yes, the duplicate event should be ignored.</p><p>Most payment gateways and webhook providers follow this model because it is practical, scalable, and reliable enough for production systems.</p><p><strong>Exactly-once delivery</strong> sounds ideal because it means no duplicates and no missing events. But in distributed systems, this is extremely hard. Network failures, uncertain acknowledgements, database commits, retries, and consumer crashes make exactly-once semantics operationally expensive and complex.</p><p>In most real-world webhook systems, the practical choice is:</p><pre>At-least-once delivery + idempotent consumers</pre><p>That gives a strong balance between reliability and operational simplicity.</p><h3>Event Ordering: Does Sequence Matter?</h3><p>Another important question is whether event ordering matters.</p><p>Some systems can process events independently. Others cannot.</p><p>In a payment system, event order can be critical. Consider this lifecycle:</p><pre>PAYMENT_CREATED<br>PAYMENT_AUTHORIZED<br>PAYMENT_CAPTURED<br>REFUND_INITIATED</pre><p>If REFUND_INITIATED is processed before PAYMENT_CAPTURED, the system may try to refund a payment that has not yet been successfully captured. That creates an invalid state.</p><p>The common solution is partitioning. If all events related to the same payment ID go to the same Kafka partition, Kafka can preserve ordering within that partition.</p><p>For example:</p><pre>Partition Key = paymentId</pre><p>This ensures all events for payment P100 go to the same partition and are processed sequentially.</p><p>But this comes with a trade-off. Ordering reduces parallelism. If one partition becomes too hot, throughput may suffer. Therefore, in payment systems, we usually need per-payment ordering, not global ordering across all payments.</p><h3>Security Is Not Just “Use HTTPS”</h3><p>Security is one of the most overlooked areas in webhook design discussions. But for payment systems, order systems, account systems, and financial workflows, it is critical.</p><p>Webhook endpoints are often public. Anyone can try to send a fake request. If the system blindly accepts it, an attacker could send a fake payment success event or replay an old valid event.</p><p>A secure webhook system should enforce HTTPS for encryption in transit. But HTTPS alone is not enough.</p><p>The system should also use HMAC signature validation. In this model, the sender and receiver share a secret. The sender signs the payload using that secret, and the receiver recomputes the signature using the same payload and same secret. If both signatures match, the request is considered authentic and untampered.</p><p>This protects against payload tampering.</p><p>Replay prevention is also important. An attacker may capture a valid webhook and send it again later. To prevent this, webhooks should include timestamps, unique event IDs, or nonce values. The receiver can reject old requests and ignore duplicate event IDs.</p><p>For sensitive industries, payload-level encryption may also be needed. In such cases, the client may provide a public key during webhook registration, and the webhook system can encrypt the payload so that only the client can decrypt it using its private key.</p><p>A strong webhook security design should cover authenticity, integrity, confidentiality, and replay protection.</p><h3>Fault Tolerance Is More Than Retries</h3><p>Many candidates say, “We will retry with exponential backoff,” and stop there. But that is not enough for a production-grade webhook system.</p><p>Fault tolerance requires a complete recovery strategy.</p><p>The system should interpret HTTP status codes intelligently. A 200 OK means successful delivery. A 400 Bad Request or 404 Not Found usually indicates a client-side issue and should not be retried aggressively. A 500 Internal Server Error, 503 Service Unavailable, or timeout usually indicates a transient failure and should be retried using exponential backoff.</p><p>Every event should be persisted before being queued for delivery. This ensures that even if the worker crashes or the queue is temporarily unavailable, the original event is not lost.</p><p>A dead-letter queue is also essential. If an event fails all retry attempts, it should be moved to a DLQ for investigation, replay, or manual recovery.</p><p>The system should also track delivery attempts. For each attempt, we should know the webhook ID, event ID, attempt number, response code, error message, latency, and timestamp.</p><p>This is the difference between a system that merely “tries again” and a system that can actually explain and recover from failures.</p><h3>Scalability: Isolate Blast Radius</h3><p>Scalability is not just about adding more servers. It is about isolating blast radius.</p><p>A hot client may suddenly receive a huge number of events. For example, a large merchant during a flash sale may receive millions of payment or order events. If we process all clients through the same shared queue, one hot client can slow down everyone else.</p><p>To avoid this, we can use per-client queue partitioning, adaptive rate limiting, priority queues, and dedicated worker pools for large clients.</p><p>Similarly, a hot source application can overwhelm ingestion. If Stripe, Shopify, or a warehouse scanner suddenly sends 10 times its normal volume, the system should detect the anomaly, alert the team, and apply throttling if necessary.</p><p>For high write throughput, the database should also be optimized using batch inserts, partitioning, proper indexing, and replication.</p><p>Real scalability means the system can handle spikes without allowing one tenant or source to degrade the entire platform.</p><h3>Observability: The System Must Explain What Happened</h3><p>In production, things will fail. The question is not whether failures happen. The question is whether the system can explain them.</p><p>Observability in a webhook system has two major goals.</p><p>First, track the delivery status of every webhook event. Clients and operators should be able to answer:</p><p>Did this event get delivered?<br>Which endpoint received it?<br>How many retries happened?<br>What response code did the client return?<br>Why did it fail?<br>Can we replay it?</p><p>Second, monitor failure patterns before they become incidents. DLQ growth, retry spikes, queue lag, high callback latency, and increasing 5xx responses should trigger alerts.</p><p>A good design separates the event table from the delivery tracking table. The event table is append-only and stores the original event. The delivery table stores each delivery attempt because one event may be delivered to multiple webhooks, and each delivery has its own lifecycle.</p><p>For example, event EVT_123 may succeed for webhook A but fail for webhook B. Therefore, storing one delivery status directly on the event record would be incorrect.</p><p>Observability is the eyes and ears of the system. Without it, even a well-designed webhook platform becomes a black box.</p><h3>Final Thoughts</h3><p>A webhook system may look simple from the outside, but production-grade webhook design requires careful thinking across reliability, security, scalability, durability, ordering, idempotency, and observability.</p><p>The strongest architecture is not the one with the most technologies. It is the one that matches the actual business requirements.</p><p>For non-critical events, lightweight best-effort delivery may be enough. For payment and order systems, we need durable storage, retries, idempotency, DLQs, monitoring, signature validation, replay protection, and clear recovery paths.</p><p>In interviews, do not present webhook design as a memorized architecture. Treat it as a collaborative discussion. Clarify requirements, explain trade-offs, and justify decisions based on the use case.</p><p>That is what separates a good system design answer from a production-ready engineering discussion.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a319329e151" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Bug That Didn’t Crash Production: Hidden Java Mistakes That Slowly Kill Systems]]></title>
            <link>https://medium.com/@deepasingh1017/the-bug-that-didnt-crash-production-hidden-java-mistakes-that-slowly-kill-systems-0b6d1934fe46?source=rss-16209d038d37------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b6d1934fe46</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Deepa Singh]]></dc:creator>
            <pubDate>Mon, 06 Apr 2026 12:20:14 GMT</pubDate>
            <atom:updated>2026-04-06T12:20:14.835Z</atom:updated>
            <content:encoded><![CDATA[<p>It was 2:17 AM. No alerts, no crashes, no exceptions — and still something was very wrong. Memory kept rising, latency was creeping up, and the garbage collector was working harder than ever. But the code? Clean, reviewed, deployed. Everything looked correct. That was the problem.</p><p>That night, we didn’t find a “big bug.” We found something much more dangerous — small, innocent decisions repeated at scale.</p><p>Somewhere in the codebase, there was a line like this:</p><pre>String s = new String(&quot;payment&quot;);</pre><p>It compiles, it works, it looks harmless. But it creates a new object every single time, even when &quot;payment&quot; already exists. At small scale, it doesn’t matter. At production scale, it creates millions of unnecessary objects, increases GC pressure, and slowly starts affecting latency. That’s when you realize—Java doesn’t punish bad code immediately, it punishes harmless-looking code when it runs millions of times.</p><p>We started noticing this pattern everywhere. Constructors being used without thinking, objects being created without need. That’s when static factory methods started making more sense. Constructors always create new objects. Static factories don’t have to. They can reuse, cache, and control instance creation. And in large systems, control matters more than convenience.</p><p>Then we saw utility classes — meant only for static methods — being instantiated. Why? Because Java quietly provides a default constructor. So unless you explicitly stop it, people will use it. A private constructor fixes that forever. A small decision, but it prevents misuse at scale.</p><p>Then came readability issues. Code like:</p><pre>new NutritionFacts(240, 8, 100, 0, 35, 27);</pre><p>No one knew what those numbers meant — not even the person who wrote it after a few weeks. That’s not just bad style, it’s dangerous. Because unclear code becomes impossible to debug under pressure. Builder pattern fixed that — not as a design luxury, but as a necessity. It made object creation readable, explicit, and safe.</p><p>Next came design flaws. Dependencies were hardcoded — static, fixed, assumed constant. It worked… until it didn’t. Testing became painful, scaling became rigid. Dependency injection changed that. Instead of creating dependencies, we started passing them. Suddenly the system became flexible, testable, and easier to evolve. That’s when you understand — good design is not about today, it’s about future changes.</p><p>Then came the real silent killer — memory leaks. Not the obvious crashing kind. The slow, creeping kind. We had a stack implementation where elements were popped but references were never cleared. So objects stayed in memory even when they were no longer needed. Garbage collector couldn’t reclaim them because technically, they were still reachable. One line fixed it:</p><pre>elements[size] = null;</pre><p>That one line stabilized memory and reduced GC pressure. That’s when it hit hard — garbage collection is not magic. It only cleans what you truly release.</p><p>Then came autoboxing. A simple mistake:</p><pre>Long sum = 0L;</pre><p>Instead of:</p><pre>long sum = 0L;</pre><p>That single capital letter caused massive object creation behind the scenes. Same logic, drastically different performance. Not all performance issues come from complex algorithms — some come from tiny, invisible decisions.</p><p>Then came resource handling. We relied on finalizers, assuming cleanup would happen automatically. It didn’t. Resources stayed open, connections leaked, and systems became unstable. Because finalizers are unpredictable — they may run late, or not at all. Switching to try-with-resources brought predictability. Not just cleaner code, but reliable cleanup. And in production systems, predictability is everything.</p><p>By the end of that night, nothing “big” had changed. No major redesign, no architecture overhaul. Just small fixes — fewer objects, better patterns, proper cleanup, clearer code. But the impact was huge.</p><p>That’s when the real lesson landed:</p><p>Systems don’t fail because of one big mistake. They fail because of hundreds of small, acceptable decisions repeated over time.</p><p>Every line you write is harmless — until it runs a million times. Every shortcut is fine — until it becomes permanent. Every assumption works — until reality changes.</p><p>If you’ve ever looked at a system where everything seems correct, nothing is crashing, but something still feels off — you already know this feeling.</p><p>The most dangerous bugs are not loud. They are silent, patient, and perfectly valid.</p><p>And they wait.</p><p>👏 If this felt a little too real… you’re not alone.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b6d1934fe46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Stopped Taking My Code to Bed — And It Changed Everything]]></title>
            <link>https://medium.com/@deepasingh1017/i-stopped-taking-my-code-to-bed-and-it-changed-everything-250f1455b616?source=rss-16209d038d37------2</link>
            <guid isPermaLink="false">https://medium.com/p/250f1455b616</guid>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Deepa Singh]]></dc:creator>
            <pubDate>Sat, 04 Apr 2026 10:25:07 GMT</pubDate>
            <atom:updated>2026-04-04T10:25:07.708Z</atom:updated>
            <content:encoded><![CDATA[<p>There was a time when my code followed me everywhere.</p><p>Not just on my laptop. Not just in my office.<br>It followed me into my evenings… my dinners… and yes, even my bed.</p><p>I would lie down, staring at the ceiling, replaying logs in my head.<br>“Why is that API failing?”<br>“Did I miss a null check?”<br>“What if that edge case breaks in production?”</p><p>It wasn’t work anymore. It was <em>mental leakage</em>.</p><p>And honestly? It was exhausting.</p><h3>The Invisible Problem Developers Don’t Talk About</h3><p>As developers, we don’t just write code — we <em>carry</em> it.</p><p>We carry unfinished logic.<br>We carry bugs that don’t reproduce.<br>We carry production fears.</p><p>And somewhere along the way, we normalize this behavior.</p><p>We call it:</p><ul><li>“ownership”</li><li>“responsibility”</li><li>“passion”</li></ul><p>But let’s be real — it’s also <strong>mental overload</strong>.</p><h3>What Changed for Me</h3><p>Recently, something shifted.</p><p>With the rise of AI tools, debugging became faster.<br>Exploration became easier.<br>And most importantly — <strong>closure became possible within working hours</strong>.</p><p>Now when I leave the office, I actually <em>leave the problem behind</em>.</p><p>Not because I don’t care.</p><p>But because I’ve learned this:</p><p><em>A fresh mind solves in minutes what an exhausted mind struggles with for hours.</em></p><h3>The Rule I Follow Now</h3><p>I have a simple rule:</p><blockquote><strong><em>If the code isn’t running, I’m not taking it to bed.</em></strong></blockquote><p>No overthinking.<br>No mental simulations.<br>No imaginary debugging sessions at 2 AM.</p><p>Instead:</p><ul><li>I write down the problem clearly</li><li>I note what I’ve already tried</li><li>I define the next step for tomorrow</li></ul><p>And then I stop.</p><h3>But Is This Good for Developers?</h3><p>This is where it gets tricky.</p><p>Because we’ve been conditioned to believe:</p><blockquote><em>“Great developers think about problems all the time.”</em></blockquote><p>But here’s a more honest truth:</p><p><em>Great developers know when to stop thinking.</em></p><p>Overthinking doesn’t always lead to better solutions.<br>Sometimes, it just leads to burnout.</p><h3>The Balance We Actually Need</h3><p>I’m not saying don’t care about your code.</p><p>I’m saying:</p><ul><li>Don’t let your code consume your life</li><li>Don’t confuse obsession with productivity</li><li>Don’t romanticize mental exhaustion</li></ul><p>There’s a difference between:</p><ul><li><strong>Deep thinking</strong> → valuable</li><li><strong>Constant thinking</strong> → draining</li></ul><h3>What I Gained</h3><p>Since I stopped taking code to bed:</p><ul><li>My sleep improved</li><li>My clarity improved</li><li>My debugging speed actually increased</li><li>And strangely… I enjoy coding more now</li></ul><p>Because now, coding has boundaries.</p><p>And boundaries create <strong>sustainability</strong>.</p><h3>Final Thought</h3><p>Being a developer doesn’t mean being mentally “on” 24/7.</p><p>It means:</p><ul><li>solving problems effectively</li><li>thinking clearly</li><li>and knowing when to pause</li></ul><p>So if your code isn’t running…</p><p>Maybe it’s time to stop running it in your head too.</p><p>If you could relate to this, give it a clap 👏<br>It might help another developer finally get a peaceful night’s sleep.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=250f1455b616" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>