<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Atul Agrawal on Medium]]></title>
        <description><![CDATA[Stories by Atul Agrawal on Medium]]></description>
        <link>https://medium.com/@atul-agrawal?source=rss-6bf43639c07f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6Frfzifx-fGSsOCcssRJFA.jpeg</url>
            <title>Stories by Atul Agrawal on Medium</title>
            <link>https://medium.com/@atul-agrawal?source=rss-6bf43639c07f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:16:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@atul-agrawal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Architecture — Data lifecycle management platform]]></title>
            <link>https://atul-agrawal.medium.com/data-lifecycle-management-platform-1d6c4e58b68b?source=rss-6bf43639c07f------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d6c4e58b68b</guid>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[enterprise-data-goverance]]></category>
            <category><![CDATA[trash-and-restore]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[data-compliance]]></category>
            <dc:creator><![CDATA[Atul Agrawal]]></dc:creator>
            <pubDate>Sun, 06 Sep 2026 05:12:31 GMT</pubDate>
            <atom:updated>2026-09-06T12:30:58.195Z</atom:updated>
            <content:encoded><![CDATA[<p>Data in an enterprise platform is rarely static; it flows through an evolving lifecycle where its access patterns, performance requirements, and engineering characteristics shift dramatically, creating a multi-dimensional constraint problem. It’s a foundational design challenge that goes far beyond simply adding a “delete” button. To truly satisfy competing business stakeholders, the architecture must harmonize three distinct operational envelopes: an intuitive self-service safety net (e.g., restoring items from Trash within a safety window) that demands instant responsiveness; automated, rule-based offloading that keeps the primary database lean while meeting multi-year, cold retention standards; and absolute freeze locks for legal holds, where immutable integrity takes precedence over all other factors. Solving this requires a thought-out strategy that balances transactional performance, cost-efficient long-term scale, and absolute durability from ingestion to permanent, authorized removal — a problem that demands a critical analysis of where and how we store this data.</p><h3>Business Needs for Multi-Tier Lifecycle Management</h3><p>Managing enterprise data requires balancing competing priorities across four key organizational groups. Each group brings distinct requirements that must be handled gracefully by a single underlying framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_AEgnRnT-OQHP1U9_g9eGw.png" /></figure><h3>2. High-Level Core Lifecycle States</h3><p>From a business perspective, every record moves through five primary states. These states define <strong>where the data belongs</strong> and dictate who can see, edit, or restore it.</p><ul><li><strong>Active:</strong> Live production data. Fully visible, editable, and accessible across daily applications and search functions.</li><li><strong>Archive:</strong> Infrequently accessed records moved out of primary workspaces to keep user interfaces clean and operational cost optimized. Easily restored to Active whenever needed.</li><li><strong>Trash:</strong> Soft-deleted records removed from standard views. Held in a temporary safety net (e.g., 30 days) where users can self-restore items.</li><li><strong>Retained:</strong> Policy-driven compliance storage. Data moves here automatically after the Trash window if regulations require long-term keeping. Hidden from daily users and accessible only to compliance administrators.</li><li><strong>Purged:</strong> The final, terminal state. Data is permanently and irreversibly destroyed across all storage systems, leaving behind only minimal, non-sensitive audit logs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kS0y6jCN6SIst6MgBlMY8A.png" /></figure><h3>3. Cross-Cutting Governance: Legal Hold Metadata</h3><p>A critical mistake in system design is treating a <strong>Legal Hold</strong> as just another lifecycle state (e.g., updating a status column to &quot;LegalHold&quot;). Doing so wipes out the item&#39;s original state history. If an item in Trash is placed on hold and the legal case closes three years later, a single-status system cannot tell whether the item should return to Active, move to Retained, or be Purged immediately.</p><p><strong>The Golden Rule:</strong></p><ul><li><strong>Lifecycle States</strong> define <em>where data is in its operational journey</em>.</li><li><strong>Governance Metadata Overlays</strong> define <em>whether data can be edited, moved, or deleted</em>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gfHyDS2TMuXx2gc74ajh6g.png" /></figure><p>Legal Hold Rules in Practice</p><p>When is_legal_hold is activated (TRUE):</p><ul><li><strong>State Preservation:</strong> The item remains in its current state (Active, Trash, Retained, or Archive).</li><li><strong>Edit Lock:</strong> Application features block all modifications, updates, or edits to the record.</li><li><strong>Purge Protection:</strong> Automated cleanup routines skip the record entirely — even if its standard deletion clock expires.</li><li><strong>Resuming Workflows:</strong> When the legal hold is removed, the system moves the data to the right state as per policy.</li></ul><h3>5. High-Level Architecture</h3><p>Defining an enterprise data lifecycle strategy requires analyzing access patterns and data characteristics across each stage of its lifecycle:</p><ul><li><strong>Trash:</strong> Unwanted data deleted by users, requiring a temporary safety net for recovery.</li><li><strong>Archive:</strong> Active data that is rarely accessed but needs to remain available for occasional lookups.</li><li><strong>Retained:</strong> Cold immutable data driven by strict compliance, legal, and regulatory requirements (e.g., SEC, HIPAA, SOX) that must be preserved for multi-year periods.</li></ul><p>This distinction raises a fundamental architectural question: <strong>Should all data lifecycle states reside in the live operational database, or should data be offloaded to a specialized platform based on its characteristics?</strong></p><p>Evaluating the two foundational approaches highlights the key trade-offs in defining this strategy.</p><h4>Core Non-Negotiable Requirement</h4><ul><li><strong>Zero Data Loss Tolerance:</strong> Both options must guarantee absolute data durability. Whether records reside in primary transactional storage or stream asynchronously to a cold data vault, the system must enforce strict reliability (via local ACID transactions or at-least-once delivery using patterns like the Transactional Outbox) so no record is dropped during transitions.</li></ul><h3>Foundational Options Analysis</h3><h4>Option A: Operational Database (OLTP)</h4><p><strong>Pros</strong></p><ul><li><strong>Efforts:</strong> Minimal initial build effort; adding state columns or status flags to existing tables is straightforward compared to building external data egress pipelines and platform infrastructure.</li><li><strong>Complexity:</strong> Low system complexity; keeps a single source of truth inside a unified transactional database, avoiding distributed synchronization pipelines, event tracing, or eventual consistency logic. The restore logic is also straight-forward requiring only state toggling.</li><li><strong>Latency:</strong> Sub-millisecond latency for live product workflows, enabling instant state transitions and real-time self-service user restores directly within the application interface.</li></ul><p><strong>Cons</strong></p><ul><li><strong>Scale:</strong> Unwanted scalability overhead; storing long-term archive and retained data in the primary database creates index degradation, table bloat, and increasing maintenance overhead over time.</li><li><strong>Performance:</strong> Core application query performance degrades as table volumes increase under fixed compute resources, leading to slower read/write cycles.</li><li><strong>Cost:</strong> High operational storage costs; keeping cold historical records on high-performance primary block storage (SSD/IOPS) inflates database storage, compute, and backup costs.</li><li><strong>Product Roadmap:</strong> Misaligns with operational tooling features such as automated backups, Point-In-Time Recovery (PITR), and data exports; bundling multi-year historical data into operational snapshots causes severe IOPS strain and bloated backup sizes.</li></ul><h4>Option B: Non-Operational Database (Data Platform / Cold Storage)</h4><p><strong>Pros</strong></p><ul><li><strong>Latency:</strong> Preserves sub-millisecond query performance for live transactional workflows where user experience matters most. While offloaded data (archive, trash, retention) faces higher retrieval latency, this aligns with product expectations where near-instant speed is not required for historical data.</li><li><strong>Cost:</strong> Highly cost-effective; moves long-term archive and retained data to low-cost, compressed object storage (e.g., S3 Glacier, Parquet/Iceberg formats) rather than expensive transactional storage.</li><li><strong>Performance:</strong> Eliminates database bloat; offloading inactive data preserves primary compute, memory, and indexing performance for active production traffic.</li><li><strong>Scale:</strong> Provides virtually infinite horizontal scalability for cold and historical datasets without creating storage or operational bottlenecks on the primary database cluster.</li><li><strong>Impact &amp; Backward Compatibility:</strong> Minimizes legacy refactoring; original domain service behavior remains intact, allowing services to perform standard application-level deletes without schema overhauls.</li><li><strong>Product Roadmap:</strong> Aligns naturally with enterprise compliance needs, enabling hardware-enforced immutability (WORM locks), immutable audit trails, and multi-year regulatory retention policies.</li></ul><p><strong>Cons</strong></p><ul><li><strong>Efforts:</strong> High upfront engineering effort; requires designing event-driven ingestion pipelines, cold storage management frameworks, and reverse re-hydration APIs for rare recovery flows.</li><li><strong>Complexity:</strong> High system complexity; introduces distributed state management, asynchronous event streaming, cross-boundary tracing, and eventual consistency handling. In addition to this, restore logic requires moving the data back to operational database while handling the schema drfits. Schema evolution must occur in a backward-compatible manner to minimize conflicts..</li></ul><p>The analysis confirms <strong>Option B (Data Platform)</strong> as the target architecture for long-term data lifecycle management. The core implementation challenge now shifts to data ingestion: <strong>How do we reliably move data from the operational database to the non-operational platform during a delete or archive action without impacting user latency or risking data loss?</strong></p><p>When a user deletes a record, the system must purge or mark the entity in the operational database while guaranteeing that the full payload lands safely in the Data Vault.</p><p>Below are three architectural patterns evaluated for this state transition.</p><h4>1. API-Driven Pattern (Synchronous)</h4><p>In this pattern, the domain service synchronously reads the full state of the target entity (and its dependencies) and executes HTTP/gRPC calls to downstream Data Platform ingestion APIs before completing the operational delete.</p><ul><li><strong>Trade-Offs:</strong></li><li><strong>Latency &amp; Availability Impact:</strong> Reading full data graphs and waiting for network calls to external storage tiers directly degrades user-facing response times and couples live availability to downstream platform uptime.</li><li><strong>Dual-Write Failure Risk:</strong> If the record is deleted from the operational DB but the downstream API call fails (or times out), the system enters an inconsistent state, breaking our core zero-data-loss requirement.</li><li><strong>Verdict:</strong> <strong>Not Recommended.</strong> Introduces brittle synchronous coupling and high blast-radius risks.</li></ul><h4>2. Event-Driven Pattern (Asynchronous Outbox)</h4><p>Instead of calling APIs synchronously, the domain service publishes a domain event containing the entity’s state to an event broker (e.g., Kafka). To prevent data loss, it leverages the <strong>Transactional Outbox Pattern</strong> — writing both the domain state change and an outbox event within the same local database transaction.</p><p><strong>Trade-Offs:</strong></p><ul><li><strong>Improved Decoupling:</strong> Decouples user-facing response times from cold storage ingestion latencies.</li><li><strong>At-Least-Once Delivery:</strong> The Outbox pattern guarantees the event is published even if the service crashes mid-request.</li><li><strong>Payload Overhead:</strong> The application thread must still read and construct the full entity payload (including parent-child relationships) during the user’s primary API request, adding unnecessary database read latency.</li></ul><p><strong>Verdict:</strong> <strong>Feasible, but sub-optimal for complex entity graphs.</strong> Good for simple domain models, but scales poorly when assembling deeply nested entity hierarchies inline.</p><h4>3. Change Data Capture (CDC) &amp; Async Process Pattern</h4><p>This approach decouples application execution entirely by delegating data capture to the database log layer using Change Data Capture (CDC) technologies (e.g., Debezium).</p><p>When a user initiates a delete or archive action, the application executes a quick, localized transaction (e.g., updating a status flag to TRASH or ARCHIVED). The underlying database transaction log (WAL/Redo log) automatically records the row state change. A CDC pipeline reads this log asynchronously, streams the change events to the event backbone, and hydrates the Data Platform.</p><p>Once data is safely replicated to the Data Vault, a background <strong>Cleanup Worker (Reaper Process)</strong> handles the final operational database cleanup while ensuring zero-data loss and idempotency:</p><ul><li><strong>Asynchronous Purging:</strong> A scheduled, low-priority background worker queries for records tagged as TRASH or ARCHIVED whose retention window in the operational DB has expired (e.g., updated_at &lt; NOW() - 30 days).</li><li><strong>Replication-Aware Deletes:</strong> Before issuing hard deletes on the operational DB, the cleanup process verifies replication lag to ensure CDC streaming has fully committed the record payloads to cold storage.</li><li><strong>Batch Processing:</strong> Deletions are executed in small, throttled batches to prevent lock contention, transaction log bloat, or read/write latency spikes on active production tables.</li></ul><p><strong>Trade-Offs:</strong></p><ul><li><strong>Zero Application Latency Overhead:</strong> Domain services execute lean, local database transactions without fetching heavy nested dependency graphs or constructing large event payloads.</li><li><strong>Automatic Parent-Child Capture:</strong> CDC observes row-level changes across both parent and child tables natively at the database log level.</li><li><strong>Strict Reliability &amp; DB Hygiene:</strong> Log-based CDC guarantees at-least-once streaming directly from disk logs, while the background cleanup process keeps primary operational database tables small and performant over time.</li></ul><p><strong>Verdict:</strong> <strong>Highly Recommended.</strong> Delivers the highest performance, zero application footprint, native support for complex data hierarchies, and controlled operational database cleanup.</p><p>The following diagram uses the Data vault and CDC pattern.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*htYwogV65ByLdY0i8C50eA.png" /></figure><blockquote>The retention lifecycle can be managed directly within the Vault: once the Trash window expires, data transitions to the Retained state if policy dictates, or is permanently purged.</blockquote><blockquote>Audit log is also a very important aspect from data governance and compliance point of view and every state change should be explictly captured in immuable form for long term.</blockquote><h3>Low-Level Operational Intermediate States</h3><p>The above solution can also have low level intermediate operational states for seamless data movement as moving large volumes of data — such as an entire project workspace or parent object containing hundreds of thousands of child objects — cannot happen in a single split second or a single transaction.</p><p>To prevent system slowdowns, app timeouts, or partial data errors, the system uses <strong>Operational Intermediate States</strong> (Pending*). These act as holding areas while background processes safely carry out long-running tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xntJ6Lj2ILeW_Pqa7RwF-g.png" /></figure><p>State transitions involving Pending* flags should be driven by an <strong>idempotent background worker or distributed state machine (e.g., Saga Pattern)</strong> to guarantee eventual consistency across large child entity graphs.</p><p>The intermediate “pending” states is optional if data sizes are light-weight and user actions can be performed in single transaction.</p><h3>Summary</h3><p>Architecting an enterprise data lifecycle framework is fundamentally an exercise in balancing physical database constraints against long-term governance boundaries. While it is tempting to treat soft deletes, archival and compliance retention as simple application-level state flags, true platform resilience requires decoupling hot transactional paths from cold storage workloads. By pairing log-based Change Data Capture (CDC) with non-operational object vaults, orthogonal legal hold overlays, and replication-aware cleanup workers, systems can deliver sub-millisecond user responsiveness alongside immutable, multi-year regulatory compliance — all while enforcing zero data loss across complex data models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d6c4e58b68b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microservices Reference Architecture]]></title>
            <link>https://atul-agrawal.medium.com/microservices-reference-architecture-edd8d7504b47?source=rss-6bf43639c07f------2</link>
            <guid isPermaLink="false">https://medium.com/p/edd8d7504b47</guid>
            <category><![CDATA[reference-architecture]]></category>
            <category><![CDATA[software-design-patterns]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Atul Agrawal]]></dc:creator>
            <pubDate>Tue, 15 Jul 2025 14:18:53 GMT</pubDate>
            <atom:updated>2025-07-15T14:19:46.204Z</atom:updated>
            <content:encoded><![CDATA[<h4>Key Principles, Key Components and Trade-offs</h4><p><strong>Microservice architecture</strong> is a strategic approach to system design that directly addresses key business challenges, such as slow release cycles, difficulties in scaling, and the high risk associated with technology updates. By structuring the applications as a collection of small, independent services — each aligned with a specific business capability like “Payment Processing” or “Profile Management” — we unlock significant business value:</p><ul><li><strong>Accelerated Time-to-Market:</strong> Small, autonomous teams can develop, test, and deploy new features independently, drastically reducing the time it takes to bring new ideas and products to our customers.</li><li><strong>Enhanced Scalability and Efficiency:</strong> We can scale individual business functions based on demand, without needing to scale the entire application. This leads to more efficient use of resources and ensures our systems can handle peak loads gracefully.</li><li><strong>Improved System Resilience:</strong> Since services are independent, the failure of one component will not bring down the entire system. This compartmentalization minimizes the impact of outages, protecting revenue and customer trust.</li><li><strong>Future-Proofing and Innovation:</strong> Teams can adopt the best technologies for their specific needs, allowing us to innovate faster and avoid being locked into obsolete technology stacks.</li></ul><p>This reference architecture outlines the principles and patterns we will use to implement a microservices ecosystem, enabling our organization to build more robust, scalable, and adaptable systems that drive business success.</p><h4>2. Core Principles of Microservice Architecture</h4><p>To ensure consistency and success, the microservice implementations will adhere to the following core principles, inspired by industry leaders like Amazon and Netflix.</p><ul><li><strong>Single Business Capability (Bounded Context):</strong> Each service is responsible for one thing and one thing only, aligned with a specific business domain (e.g., “User Profile Management,” “Payment Processing,” “Inventory”). This boundary is its “bounded context.”</li><li><strong>Autonomy and Independence:</strong> Teams can independently develop, deploy, and scale their services. A change in one service should not require other services to be rebuilt or redeployed. This is the key to achieving development velocity.</li><li><strong>Decentralized Governance:</strong> Teams are free to choose the best technology stack (language, database) for their specific service — embracing a “right tool for the job” philosophy. Amazon famously calls this the “two-pizza team” model — a team small enough to be fed by two pizzas, owning their service end-to-end.</li><li><strong>Decentralized Data Management:</strong> Each service owns and manages its own database. This prevents the tight coupling that shared databases create and ensures services are truly independent.</li><li><strong>Design for Failure (Resilience):</strong> In a distributed system, service failures are inevitable. The architecture must be designed to be resilient, gracefully handling the failure of downstream services without causing a cascading failure across the entire system.</li></ul><h4>3. The Reference Architecture: Key Components</h4><p>The following section illustrates the key components that form the standard blueprint for a microservices ecosystem.</p><p>3.1. API Gateway</p><p>The <strong>API Gateway</strong> is the single entry point for all external clients. It acts as a reverse proxy, routing requests to the appropriate downstream service.</p><ul><li><strong>For Business:</strong> It simplifies the client experience and provides a single, secure front door to the system.</li><li><strong>For Architecture:</strong> It decouples clients from the internal service structure. We can refactor and rearrange services internally without impacting external clients.</li><li><strong>Key Functions:</strong> Authentication/Authorization, SSL Termination, Rate Limiting, Caching, and Request Aggregation (fanning out requests to multiple services and combining the results).</li></ul><p>3.2. Service-to-Service Communication</p><p>Services need to communicate with each other. There are two primary patterns:</p><ol><li><strong>API Based Communication (e.g., REST, gRPC):</strong> Service A makes a request to Service B and waits for a response. This is simple but can lead to tight coupling and reduced resilience. Use this for queries or commands that require an immediate response.</li><li><strong>Event Based Communication (e.g., Message Queues — RabbitMQ, Kafka):</strong> Service A publishes an event or message to a message broker. Other services can subscribe to these events and react accordingly. This pattern, championed by companies like Uber, promotes loose coupling and improves resilience, as the sending service doesn’t need to know about the consumers, nor does it wait for them.</li></ol><p>3.3. Service Discovery</p><p>In a dynamic environment where services are constantly being scaled up or down, how does Service A know the network address of Service B?</p><ul><li><strong>The Problem:</strong> Hardcoding IP addresses is not viable.</li><li><strong>The Solution:</strong> A <strong>Service Registry</strong> (e.g., Consul, Eureka). When a service starts, it registers itself with the registry. When another service wants to connect, it queries the registry to find the service’s current location.</li></ul><p>3.4. Data Management Patterns</p><p>As per the key principles, each service has its own private database. This introduces a major challenge: how to handle queries or transactions that span multiple services?</p><ul><li><strong>The Challenge:</strong> We can no longer perform traditional SQL joins across service databases.</li></ul><p><strong>Key Patterns:</strong></p><ul><li><strong>API Composition:</strong> The API Gateway or a dedicated aggregator service queries multiple services and combines the data before returning it to the client.</li><li><strong>CQRS (Command Query Responsibility Segregation):</strong> This pattern separates the models for writing data (Commands) from reading data (Queries). This allows for highly optimized data stores for reporting and querying, which can be populated by listening to events from other services.</li><li><strong>Event Sourcing:</strong> We store every state change as an immutable event in an event log. The current state is derived by replaying the events. This is a powerful pattern for auditability and creating different views of the data.</li></ul><p>3.5. Authentication and Authorization (AuthN/AuthZ)</p><p>In a distributed system, security is paramount. We must secure both end-user access and service-to-service communication.</p><ul><li><strong>The Pattern:</strong> We will use a centralized <strong>Identity Provider (IdP) / Auth Service</strong> (<strong>OKTA</strong>) to handle user authentication. This service will issue security tokens (e.g., JSON Web Tokens — JWTs) after a user successfully logs in.</li><li><strong>The Flow:</strong> The client application sends this token to the API Gateway with every request. The Gateway validates the token and passes it to downstream services. Each microservice can then use the information within the token to perform fine-grained authorization without needing to call the central auth service for every request, making the process highly efficient.</li></ul><p>3.6. Containerization and Orchestration</p><p>To manage the deployment and operation of potentially hundreds of services, we rely on container technology.</p><ul><li><strong>Containers (e.g., Docker):</strong> Each microservice is packaged into a lightweight, portable container. This container includes the application code, runtime, and all necessary dependencies, ensuring that the service runs consistently across all environments (development, testing, production).</li><li><strong>Orchestration (e.g., Kubernetes):</strong> A container orchestrator automates the deployment, scaling, healing, and networking of the containerized services. It allows us to manage the entire ecosystem as a cohesive whole, ensuring high availability and efficient resource utilization.</li></ul><p>3.7. Artifact Repository</p><p>The software development lifecycle produces many types of versioned files, known as “artifacts.” While container images are a key artifact, so are application libraries, packages, and other dependencies. We need a central system to manage all of them.</p><ul><li><strong>The Component:</strong> Tools like <strong>JFrog Artifactory</strong> can be used as an universal artifact repository. It will serve as the private, secure registry for Docker images while also managing other binary types like Java JARs etc. This provides a single source of truth for all software binaries.</li><li><strong>The Role:</strong> The CI/CD pipeline pushes versioned artifacts to Artifactory. The container orchestrator pulls images for deployment, and build tools fetch libraries and dependencies. This is also a critical control point for security; integrated tools like <strong>JFrog Xray</strong> etc can scan all artifacts for known vulnerabilities and license compliance issues before they can be promoted to production.</li></ul><p>3.8. CI/CD Pipeline (DevOps Automation)</p><p>To realize the agility benefits of microservices, automation is not optional. A mature CI/CD (Continuous Integration/Continuous Deployment) pipeline is the engine of rapid delivery.</p><ul><li><strong>The Pattern:</strong> For each service, we will have a dedicated, automated pipeline that manages its lifecycle.</li><li><strong>The Flow:</strong> A developer committing code triggers the pipeline, which automatically builds the code, runs unit and integration tests, packages the application and its dependencies into a container image, pushes it to the Artifact Repository, and deploys it to the various environments (e.g., staging, production). This enables safe, frequent, and reliable deployments.</li></ul><p>3.9 Cross-Cutting Concerns</p><p>Several concerns apply to all services and must be handled consistently.</p><ul><li><strong>Observability:</strong> In a distributed system, understanding what’s happening is critical. We must implement the three pillars of observability:</li><li><strong>Centralized Logging:</strong> All service logs are aggregated into a central tool (e.g., ELK Stack, Splunk).</li><li><strong>Metrics:</strong> Services expose key performance metrics (e.g., request latency, error rate, CPU usage) via central tool.</li><li><strong>Distributed Tracing:</strong> A unique ID is attached to an initial request and propagated through every service call it triggers, allowing us to trace a single user’s request across the entire system.</li><li><strong>Configuration Management:</strong> Service configuration (e.g., database connection strings, feature flags) must be externalized from the code. A <strong>Config Server </strong>provides this configuration to services at startup.</li></ul><h4>4. Benefits vs. Trade-offs</h4><p>Microservices are not a silver bullet. The decision to use them must be balanced.</p><p><strong>Benefits</strong></p><ul><li><strong>Increased Agility &amp; Time-to-Market</strong></li><li><strong>Distributed Systems Complexity:</strong> This is the biggest cost. Developers must now deal with network latency, fault tolerance, and data consistency challenges that don’t exist in a monolith.</li><li><strong>Improved Scalability</strong></li><li><strong>Enhanced Resilience</strong></li><li><strong>Technology Freedom &amp; Innovation</strong></li><li><strong>Organizational Alignment</strong></li></ul><p><strong>Challenges &amp; Costs (The Trade-offs)</strong></p><ul><li><strong>Significant Operational Overhead:</strong> Requires mature DevOps (CI/CD, automation) and a dedicated platform team to manage the infrastructure.</li><li><strong>Complex Testing:</strong> End-to-end testing is more difficult as it requires multiple services to be running and coordinated.</li><li><strong>Data Consistency:</strong> Maintaining data consistency across services requires complex patterns like Sagas or event-driven approaches.</li><li><strong>Skillset Shift:</strong> Teams need to be proficient in DevOps, containerization, and handling distributed system failures.</li></ul><h4>5. Best Practices</h4><ul><li><strong>Design “API First”:</strong> Define a service’s API contract (e.g., using OpenAPI/Swagger or gRPC proto files) <em>before</em> writing the implementation. This contract is a formal agreement between the service and its consumers, enabling parallel development and clear integration points.</li><li><strong>Robust Resiliency Patterns:</strong> In a distributed system, failures are bound to happen, and we need to ensure that fault tolerance mechanisms are in place.</li><li><strong>Timeouts:</strong> Never wait indefinitely for a response. Configure aggressive timeouts for all network calls.</li><li><strong>Retries:</strong> For transient failures, implement an exponential backoff retry mechanism to avoid overwhelming a struggling downstream service.</li><li><strong>Circuit Breakers:</strong> Use a circuit breaker pattern (e.g., via a service mesh or library) to prevent a client from repeatedly calling a service that is known to be failing. This allows the failing service time to recover.</li><li><strong>Idempotency: </strong>Idempotency should be enabled so that retries don’t lead to inconsistent states.</li><li><strong>Distributed Transactions:</strong> Since traditional ACID transactions aren’t feasible across multiple service-owned databases, the <strong>Saga pattern</strong> is a common solution for maintaining data consistency. A saga is a sequence of local transactions. If any step fails, the saga executes compensating transactions to undo the preceding work, effectively rolling back the operation.</li><li><strong>Fine-Grained Authorization: </strong>Services should have the logic to ensure that only authorized users are able to perform the operations on their resources. An orders-service receives a request GET /orders/123. The JWT indicates the request is from user_id: 456. The service must contain the logic to check: “Does user 456 have permission to view order 123?</li><li><strong>Test Automation:</strong> The complexity of a microservices ecosystem is only manageable through aggressive automation. Each service must have a comprehensive suite of automated tests (unit, integration).</li></ul><h4>6. Conclusion</h4><p>The microservice architecture represents a significant shift in how we build and deliver software. It directly enables key business goals of agility, scalability, and resilience. However, it demands long term commitment, operational excellence, and engineering discipline for taming the complexity of distributed systems. Therefore, microservice architecture patterns should be adopted by keeping the<strong> </strong>challenges and<strong> </strong>trade-offs in mind.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edd8d7504b47" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web-hook Notification Platform for Payment Transactions]]></title>
            <link>https://atul-agrawal.medium.com/web-hook-notification-platform-for-payment-transactions-e00ae678cd4d?source=rss-6bf43639c07f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e00ae678cd4d</guid>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[webhooks]]></category>
            <dc:creator><![CDATA[Atul Agrawal]]></dc:creator>
            <pubDate>Sun, 07 Apr 2024 16:52:52 GMT</pubDate>
            <atom:updated>2024-04-08T02:22:01.048Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction:</strong></h3><p>The ever-growing e-commerce landscape demands robust payment processing systems that can handle surging transaction volumes. A critical component of this infrastructure is the delivery of real-time payment notifications to merchants. These webhooks empower merchants to efficiently manage transactions, enabling functionalities like reconciliation, bookkeeping, and reporting. This blog post dives into the architectural considerations and design iterations for crafting a scalable and reliable webhook notification platform specifically for payment transactions.</p><h4>Key Characteristics of the Platform</h4><ol><li><strong>Unwavering Reliability:</strong> Data loss is unacceptable. The platform must guarantee eventual delivery of all webhook events, ensuring merchants receive crucial payment updates.</li><li><strong>Effortless Scalability:</strong> The platform should seamlessly handle millions of daily transactions without compromising performance or responsiveness.</li><li><strong>High Availability (Desirable):</strong> While near real-time delivery is ideal, availability requirements can be slightly relaxed as long as events are ultimately delivered reliably.</li><li><strong>In-depth Observability and Monitoring:</strong> Monitoring the platform’s health is paramount. We need to understand how different components interact, identify performance bottlenecks, and pinpoint failing elements to ensure smooth operation.</li></ol><h3><strong>Design Iteration #1: Pub-Sub Architecture (A Starting Point)</strong></h3><p>Our initial design leverages a publish-subscribe (pub-sub) architecture. In this setup, the payment service acts as the publisher, broadcasting payment status change events to a message broker. Subscribers, then consume these messages and trigger subsequent actions such as:</p><ol><li>Retrieving merchant details (endpoint URL, credentials) from a merchant info service.</li><li>Sending the notification payload to the merchant’s designated endpoint.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sYJocRH7-qcnT-65-DU5fw.png" /></figure><p>This architecture effectively addresses scalability, latency, and availability to a certain extent. However, it falls short in guaranteeing reliability. Let’s explore potential shortcomings:</p><ul><li><strong>Single Point of Failure:</strong> If any service (notification, merchant info, merchant) becomes unavailable, event delivery stalls.</li><li><strong>Message Broker Issues:</strong> Broker downtime or network problems can disrupt message delivery or loss of events.</li></ul><h3><strong>Design Iteration #2: Pub-Sub with Fault Tolerance Mechanisms</strong></h3><p>We can enhance reliability by introducing fault tolerance mechanisms within each component:</p><ol><li><strong>Payment Service:</strong> To ensure guaranteed delivery and to decouple message publishing from the core payment logic, we can implement the Transactional Outbox Pattern. This pattern ensures both transaction records and event payloads are persisted in a database. Events can then be retrieved later for further processing, guaranteeing message persistence even if the broker is unavailable.</li><li><strong>Broker and Subscriber:</strong> We can introduce retries on both the broker and subscriber sides for enhanced resilience:</li></ol><ul><li><strong>Retry Queue:</strong> An additional retry queue will be added to the broker for keeping the events that couldn’t get successfully processed by the subscriber due to transient failures such as network errors or temporary service unavailability. In such case, subscriber will manage the event level retries and re-queue the events.</li><li><strong>Exception Queue:</strong> Similarly, exception queue will be added to the broker for keeping the events that couldn’t get successfully processed event after retries. This queue can be monitored, debugged and processed separately for improvements.</li></ul><p><strong>3. Notification and Merchant Info Services:</strong> These services can also benefit from retry mechanisms for handling potential downstream service failures.</p><p><strong>4. Merchant Info Service:</strong> Considering the read-heavy nature of merchant data access, utilizing a highly available database with caching can further improve performance and reliability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7MwONpM43Zr0S5Ewshs_ug.png" /></figure><p>This iteration strengthens the design by introducing retries and message persistence, significantly improving reliability. However, there are still edge cases to consider:</p><ul><li><strong>Timeouts and Duplicates:</strong> Retries can lead to duplicate event processing e.g. Its possible that multiple events could be sent to merchant in case of a connection timeouts scenario.</li><li><strong>Retry Storming : </strong>The retries can have cascading effect and can overload downstream services.</li></ul><h3><strong>Design Iteration #3: State Management and Idempotency for Enhanced Reliability</strong></h3><p>To further bolster reliability and address remaining edge cases, we can introduce following mechanisms:</p><ul><li><strong>State Management:</strong> By storing the delivery state of webhook events (delivered/undelivered), we can Optimize Retries — Only undelivered events are retried, preventing unnecessary processing.</li><li><strong>Idempotency with Message IDs:</strong> Assigning a unique message ID as an idempotency key ensures that retries don’t introduce unintended side effects like duplicate transactions.</li><li><strong>Circuit Breakers: </strong>Circuit breakers short circuit the traffic to failing components and helps in avoiding the cascading effects such as retry storming.</li><li><strong>Observability:</strong> Reports based on delivery states can help pinpoint problematic merchants with unstable APIs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1oTllPTFVjY1NBtVPChnNw.png" /></figure><h3><strong>Conclusion:</strong></h3><p>Building a scalable and reliable webhook notification platform requires careful consideration of architectural choices and implementation details. This blog post explored a three-step design iteration process, starting with a basic pub-sub architecture and progressively introducing fault tolerance mechanisms, state management, and idempotency to achieve the desired level of reliability.</p><p>In order to improve further, please provide your valuable feedback and share the techniques that you are using for similar use cases. 🙏</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e00ae678cd4d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Validation Module —High Level Architecture]]></title>
            <link>https://atul-agrawal.medium.com/validation-module-high-level-architecture-7991b882e8c8?source=rss-6bf43639c07f------2</link>
            <guid isPermaLink="false">https://medium.com/p/7991b882e8c8</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[low-code]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[validation]]></category>
            <category><![CDATA[validator]]></category>
            <dc:creator><![CDATA[Atul Agrawal]]></dc:creator>
            <pubDate>Sat, 15 Apr 2023 11:17:05 GMT</pubDate>
            <atom:updated>2023-04-15T11:40:01.120Z</atom:updated>
            <content:encoded><![CDATA[<p>Validation is one of the most commonly discussed and required capability for all types of business applications as ensuring the integrity of the data and business rules is the number one priority for everyone.</p><p>The complexity or need for the validation depends on the nature of the use cases at hand. In case of simple use cases, validation logic is usually provided by enabling some basic validation checks on both front-end and back-end side using widely available open source frameworks and libraries <strong>[#1]</strong> but in case of complex use cases where domain has to manage the diverse set of data fields for hundreds of web pages across different markets for various channels (iOS, Android, Web) then simple approaches doesn’t scale up leading to time to market and maintainability challenges.</p><p>This article talks about the proposal for a complex use case where validations management is considered as key business capability.</p><p>In next few sections, this article will cover requirements, business goals, architectural principles and high level proposal.</p><h3><strong># Requirements</strong></h3><ol><li>The validations are required to be supported for hundreds of data fields (Personal Details, Business Details, Product Details etc.)</li><li>The validations are required to be supported for 200+ countries across the globe. (E.g. <a href="https://www.worldprivacyforum.org/2021/10/national-ids-and-biometrics/">National Ids Around the World</a>)</li><li>The validations are required to be performed across different apps — iOS, Android, Web and Back-end Apps</li><li>The validations can fall under different category such as</li></ol><ul><li>Regex Validation (Email : /^\S+@\S+\.\S+$/)</li><li>Range Validation ( profit / loss : 0%-100%)</li><li>Date Validation ( Card expiry date : ≥ Current Date)</li><li>One of Validation (State : one of the states in USA)</li><li>Algorithmic Validation (Unique Number : <a href="https://abr.business.gov.au/Help/AbnFormat">Australian Business Registration Number</a>)</li><li>Stock Ticker Validation (Company’s ticker symbol : TSLA)</li><li>Address Validation (Address : 1501 Page Mill Road Palo Alto, CA 94304)</li><li>Business Rules Validations (<strong>3 locked stories for free per month in medium)</strong></li></ul><p>There may be use cases where it might be required to integrate with third party validation services via API [<strong>#2</strong>].</p><h3>#Taxonomy</h3><p><strong>Validation Method</strong> : Validation method is the type of validation that will be enabled for a data field.</p><ul><li>Regex validation Method</li><li>Numeric Expression Validation Method ( &gt; =, &lt; =, min, max)</li></ul><p><strong>Validation Expression</strong> : Validation expression is used for defining the validation for a given data field (<strong>Age</strong> can have <strong>min :18, max : 100 </strong>as validation expression)</p><h3><strong># Business Goals</strong></h3><h4>Time to market :</h4><p>1. Existing validations methods (e.g. regex expression) should be reusable for new data fields.</p><p>2. The validation expression for a given validation method should be changeable with no code pushes. (E.g. change age limit value from 18 years to 21 years).</p><p>3. New validation methods should be enabled with minimal changes.</p><h4>Data Cleanliness</h4><ol><li>Data is required to be clean (valid) irrespective of the data source channels.</li></ol><h3><strong># Architecture Principles and Guidelines</strong></h3><ul><li>Validations should always be performed on front-end and back-end side for avoiding the invalid data overrides.</li><li>All accesses to data storage should only be done via API to ensure data is always consistent. There should be no data fields update directly via queries or scripts.</li><li>Validation requirements should be defined in <strong>one single source in the form of config</strong> for reducing the maintenance efforts across different applications.</li><li>Validation SDKs should be provided for executing the validation checks using the validation requirements for reducing the client side efforts.</li></ul><h3><strong># Core Capabilities</strong></h3><ul><li>validation methods</li><li>validation expressions</li><li>validation execution logic</li><li>validation messages</li></ul><h3><strong># High Level Design</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lF5BgWlFSEmqp8S4Q4kSqQ.png" /></figure><h4><strong>Validation SDK</strong></h4><p>Validation SDKs will be provided in multiple languages such as iOS, Andorid, JS, Java so that these SDKs can be hosted by the client apps such as iOS, Android, Web and other back-end applications that are responsible for performing the validations.</p><p>Validation SDK will have the execution logic for the supported validation methods (e.g. regex validation) and will execute these methods according to configured validation requirements.</p><p>Validation SDK will provide following two integration methods :</p><ol><li>Client will provide the data that is required to be validated along with the validation requirements.</li><li>Client will provide the data that is required to be validated along with the unique data schema id. In this case, validation SDK will use the provided data schema id for internally loading the validation requirements from validation service for performing the validations.</li></ol><p>Validation SDK also publish the validation events for monitoring, debugging and record keeping purposes.</p><h4><strong>Validation Service</strong></h4><p>Validation service provides the APIs for following use cases:</p><ol><li>Validation Requirements — Validation SDK can invoke this API for loading the validation requirements for data schemas.</li><li>Validation execution — Validation SDK or clients can directly invoke this API for running the validations on service side.</li></ol><p>Validation service will integrate with third party services for performing validation checks that are offered by the external vendors as a service.</p><h4>Third Party Services</h4><p>Third party services are used where the validation is very complex in nature e.g. address validation.</p><p><strong># Sequence Diagram</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GoIEEBl3rbHKc1txjK143g.png" /></figure><h3><strong># Summary</strong></h3><p>There are many low-code, no-code solutions in industry <strong>[#3]</strong> that are offering form-builder capability for front-end development. These solutions usually include config driven validation capability as one of the key product offering. The objective of this article is to provide the insights about the complexities related to data validation for such low-code, no-code solutions and to provide high level design proposal that can help in tackling these complexities.</p><p>Thanks for reading this article. Please help in sharing the validation framework or solution that your team is using in the comment section. I will update the same in reference section with credit.</p><h3>#References</h3><h4>#1 Open source frameworks and libraries</h4><p><a href="https://commons.apache.org/proper/commons-validator/">Apache Commons Validator</a> (Java)</p><p><a href="https://hibernate.org/validator/">Hibernate Validator</a> (Java)</p><p><a href="https://ajv.js.org/">Ajv JSON Schema Validator</a> (JS)</p><p><a href="https://joi.dev/api/?v=17.9.1">Joi validator</a> (JS)</p><h4>#2 Third Party Validations</h4><p><a href="https://developers.google.com/maps/documentation/address-validation/overview">Google Address Validation</a></p><p><a href="https://www.loqate.com/en-gb/email-verification/">Loqate Email Validations</a></p><p><a href="https://www.loqate.com/en-gb/phone-validation/">Loqate Phone Validations</a></p><h4>#3 Low Code | No Code Solutions</h4><p>Form.io</p><ul><li><a href="https://help.form.io/intro/overview">https://help.form.io/intro/overview</a></li><li><a href="https://help.form.io/userguide/form-building/component-settings#validation">https://help.form.io/userguide/form-building/component-settings#validation</a></li></ul><p>Mozilla</p><ul><li><a href="https://rjsf-team.github.io/react-jsonschema-form/">https://rjsf-team.github.io/react-jsonschema-form/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7991b882e8c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web Pages— Basic Security Practices]]></title>
            <link>https://atul-agrawal.medium.com/web-pages-basic-security-practices-ce395e843ece?source=rss-6bf43639c07f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce395e843ece</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[web-app-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[web-app-security]]></category>
            <dc:creator><![CDATA[Atul Agrawal]]></dc:creator>
            <pubDate>Fri, 27 Jan 2023 10:03:49 GMT</pubDate>
            <atom:updated>2023-01-27T10:26:01.006Z</atom:updated>
            <content:encoded><![CDATA[<p>The security is always #1 priority for all types of enterprises and yet there are few common mistakes that teams end up doing time to time unknowingly.</p><p>One of the attack that requires special attention is “<strong>credential stuffing</strong>” in which hackers get the access of the user credentials such as email id and password from dark web or form less secure sites and then use this information for launching scripted attacks across multiple target sites.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6bq8cNh0ES5GlJz5GYMHrw.png" /></figure><p>Source : https://www.cloudflare.com/learning/bots/what-is-credential-stuffing.</p><blockquote>The reason why “<strong>credential stuffing</strong>” works is because of the fact that most customers have the habit of reusing the password across multiple sites.</blockquote><p>There are few patterns to prevent “credential stuffing” attack such as <em>Two-Factor Authentication</em> but in this article we will focus on simple approaches that could be followed by development team for reducing the risk of data breaches.</p><h3>Approach #1 : Mask Data On The Server Side</h3><p>Most websites use data masking as a common technique for securing the customer’s sensitive data such as Tax ID, Passport Number etc.</p><p><strong>Social Security Number</strong> : XXX-XX-4759</p><p><strong>Date of Birth</strong> : XX/XX/1985</p><p>Masking requires minimal efforts but helps significantly in ensuring that sensitive information is never compromised even in case of attacks like “credential stuffing”.</p><p><strong>Anti Pattern</strong>: Client Side Data Masking</p><p>If data is being masked on client side than hackers can still launch scripted attack and parse response for accessing sensitive customer data so always do server side masking.</p><h3>Approach #2 Avoid oversharing of the customer information to front-end pages</h3><p>Web Applications usually make back-end APIs call to load information that is required to be displayed to the customer and most of times, these back-end APIs response includes superset of the information as a default behavior.</p><p>E.g. In an e-commerce app, if front-end page has to display customer’s name and address and it makes a back-end api call using customer id then api should only return name and address field. The API response shouldn’t contain phone number, date of birth, gender and other personal information.</p><p>This approach ensures that if attackers are able to access a specific web page using “credential stuffing” then they would have limited access to sensitive profile information.</p><blockquote>REST APIs usually return the entire resource (e.g. Customer details) in response as a default behavior so this mistake is more common as front-end team just ignores the unwanted additional fields. On the other hand, GraphQL APIs forces both front-end and back-end developers to plan for fields level queries explicitly.</blockquote><h3>Approach #3 Dynamic URLs for front-end pages</h3><p>Static URLs have several advantages over dynamic urls but there are few use cases where it’s worth considering the usage dynamic urls for improving the overall security of the urls as it makes it difficult for hackers to launch scripted attacks.</p><p>Static URL — <a href="http://www.example.com/dosomething">www.example.com/dosomething</a></p><p>Dynamic URL — <a href="http://www.example.com/context_id=%7Bcontext">www.example.com/context_id={context</a>_id}</p><p>As an example, If there is a use case that requires urls to be mailed out to thousands of customers and if this url is going to display sensitive personal info and/or collect some personal info from customers and then it would be better to use dynamic url like as it makes it difficult for hackers to use “credential stuffing” attack for all customers as urls are different for each of them.</p><blockquote>These guidelines are not comprehensive and wouldn’t make web pages full proof but these approaches can be implemented with small efforts for reducing the impact of data breaches !!!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce395e843ece" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tackling the Tech Debts]]></title>
            <link>https://atul-agrawal.medium.com/tackling-the-tech-debts-8150acbbb662?source=rss-6bf43639c07f------2</link>
            <guid isPermaLink="false">https://medium.com/p/8150acbbb662</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[tech-debt]]></category>
            <dc:creator><![CDATA[Atul Agrawal]]></dc:creator>
            <pubDate>Wed, 26 Oct 2022 06:33:17 GMT</pubDate>
            <atom:updated>2024-04-26T05:30:26.693Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/949/0*vsD8zhbHwVmEr-x1.png" /></figure><p>A few months back, a platform module that consisted of web app, native app and back-end apps got handed over to my team as part of re-org.</p><p>The expectation from leadership for this platform was to make it stable, robust and configurable so that business programs can be delivered in timely manners without any major bugs.</p><p>As the platform was new to us, we wanted to focus on the basics and wanted to understand the current state of the platform by identifying all the existing issues and gaps. On top of this, we also wanted to find out key<strong> </strong><em>focus areas</em><strong> </strong>that have significant impact on platform and demand immediate action from our side.</p><p>In order to do this, we decided to create the <em>tech debts</em> for the known gaps and label these tech debts against key categories for identifying the <em>focus areas </em>for further improvements.</p><p>After few internal discussions, we came up with following categories:</p><h4>Key Categories:</h4><ul><li>Business Capabilities — This category will help us in identifying the common issues that have direct impact on time to market.</li><li>Quality Attributes — This category will help us in identifying the tech debts which are required to be addressed for making platform more stable and robust.</li><li>Components — This category will help us in identifying the components for which tech debts are being introduced at higher rate.</li></ul><p>The following section is going to provide more details on these categories and the break-down of the tech-debts.</p><h3>Business Capabilities</h3><p>We had discussions with product owners and identified the list of business capabilities that are most frequently changed as part of the business initiatives.</p><h4>Business Capabilities</h4><ul><li>Experience Design</li><li>Validation</li><li>Experimentation</li><li>Content Management</li><li>Customer Profile Management</li><li>Asset Management</li><li>Monitoring</li></ul><h3>Quality Attribute</h3><p>We also identified following key quality attribute requirements for the platform:</p><ul><li>Configurability</li><li>Mantainability</li><li>Performance</li><li>Availability</li><li>Observability</li><li>Security</li><li>Testability</li></ul><h3>Components</h3><p>The last category, we identified was components so that we know which specific components requires immediate action.</p><ul><li>Web App (React App)</li><li>Native App (iOS and Android)</li><li>Back-end Apps (Node App and Java App)</li></ul><h3>Analysis</h3><p>After creating the tech debts for few months, we got the complete picture of the current state of the platform.</p><p>The following charts show the breakdown of the tech debts against the key categories:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A9NpuexekVFP1FQoqMLBVw.png" /><figcaption>Business Capabilities</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l6Aan2NSPERBwEZal6vmbQ.png" /><figcaption>Quality Attributes</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NLRTjeymdisHucrJM_hyqg.png" /><figcaption>Components</figcaption></figure><p>After identifying the majority of the gaps, we had internal discussions with stakeholders and came up with following top<em> 3 key focus areas.</em></p><ul><li><em>Configurability in Experience Enablement area in Back-End App.</em></li><li><em>Maintainability in Monitoring area in Web and Native App.</em></li><li><em>Testability across all apps.</em></li></ul><blockquote>The breakdown of tech-debts for each category was self-explanatory and helped in convincing the stakeholders.</blockquote><h3>Tools</h3><p>Our team uses Jira for tracking the user stories, bugs so we decided to use the same for tracking the tech-debts as well.</p><h3>Key points</h3><ul><li><strong>Discuss and define the categories </strong>with stakeholders for identifying the focus areas.</li><li><strong>Share the objective with team</strong> so that they are not hesitant in creating the tech-debts. We ensured developers that the main objective is to identify the gaps in platform and we will be coming up a plan later on for resolving both existing and new tech debts.</li><li><strong>Share how and when to create the tech-debts</strong>. we asked the developers to create tech debts during the development and code reviews phases for both existing and newly introduced gaps using JIRA tools.</li><li>Lastly, discuss and <strong>identify the top 3 focus areas</strong> for improvements.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8150acbbb662" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monolith to Micro Services — Key Takeaways]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://atul-agrawal.medium.com/monolith-to-micro-services-key-takeaways-22ff74de6dd?source=rss-6bf43639c07f------2"><img src="https://cdn-images-1.medium.com/max/1954/1*3-kw25u-iCPOTxFUTa8QwA.png" width="1954"></a></p><p class="medium-feed-snippet">In this blog post, I am going to share the key takeaways that I have learnt after doing the retrospective of architectural changes that&#x2026;</p><p class="medium-feed-link"><a href="https://atul-agrawal.medium.com/monolith-to-micro-services-key-takeaways-22ff74de6dd?source=rss-6bf43639c07f------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://atul-agrawal.medium.com/monolith-to-micro-services-key-takeaways-22ff74de6dd?source=rss-6bf43639c07f------2</link>
            <guid isPermaLink="false">https://medium.com/p/22ff74de6dd</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[monolithic-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Atul Agrawal]]></dc:creator>
            <pubDate>Mon, 16 May 2022 11:34:42 GMT</pubDate>
            <atom:updated>2022-05-16T11:34:42.109Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[A Method for Software Architecture Blueprint]]></title>
            <link>https://atul-agrawal.medium.com/a-method-for-software-architecture-blueprint-99aa482773a3?source=rss-6bf43639c07f------2</link>
            <guid isPermaLink="false">https://medium.com/p/99aa482773a3</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-design]]></category>
            <dc:creator><![CDATA[Atul Agrawal]]></dc:creator>
            <pubDate>Thu, 07 Apr 2022 02:18:09 GMT</pubDate>
            <atom:updated>2022-09-06T07:44:33.166Z</atom:updated>
            <content:encoded><![CDATA[<p>There are different methods that software architects use from time to time for coming up with the blueprint of software architecture.</p><p>In this post, I have outlined a method that is easy to use and remember by using an example of movie booking application.</p><h3>Step 1: Essence</h3><blockquote>Essence is the basic or most important idea or quality of something.</blockquote><p>The purpose of this step is to <strong>capture the essence</strong> of business capability and map it to the conceptual view of the system. This step involves asking the basic questions such as:</p><ul><li>What is being built out?</li><li>Why it is being built out?</li><li>Who is going to use it and how they will use it?</li></ul><p>In case of Movie Booking application, the essential capabilities that are required to be enabled are :</p><blockquote><strong>Customers </strong>should<strong> </strong>be able to<strong> book or cancel the movie ticket </strong>by using the<strong> Web Experience. Customer </strong>to be notified via<strong> Email </strong>for<strong> booking or cancellation status.</strong></blockquote><blockquote><strong>Admin</strong> should be able to <strong>manage the movies</strong> <strong>and</strong> <strong>showtimes</strong> by using the <strong>Web Experience</strong>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xAFbiEUR-JWfFrgQq3y09A.png" /></figure><p><strong>Note</strong>: In this post, we will only cover moving booking use case for customers.</p><p>Once the basic questions are clarified, we can draw the conceptual view of the system as shown in following diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GRabekfoOh93SpLC8te96A.png" /></figure><h3>Step 2 : Engagement</h3><p>The purpose of this step is to <strong>capture the customer engagement</strong> for the identified business capabilities.</p><p>The following diagram depicts the high level engagement flow from customer’s perspective for booking movie ticket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xyEQC1QHn8PeW0h6o26Jgg.png" /></figure><p>Once we have caputured how customer are going to engage, we can add more logical components to the conceptual system diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7lk2t9obIiO6r8YgMMZocA.png" /></figure><p>The logical component from this step will be evolved into web apps and services in subsequent phases.</p><h3>Step 3: Experience</h3><p>The purpose of this step is to discuss and define the<strong> quality attributes </strong>that are critical for a great <strong>customer experience</strong>. The quality attributes (performance, availability, maintainability etc.) plays significant role when the conceptual view to translated to implementable view of the system.</p><p><strong>Architect and stakeholders can discuss about the quality attributes by asking questions such as:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZhviateGoDjuwUfkdm63dA.png" /></figure><blockquote>It is important to do industry research and find out how the competitors are doing.</blockquote><p>In case of movie booking app, let’s assume that search movies has 20X traffic and require high availability while for booking movie ticket, consistency is more important.</p><p>This assumption leads to the conclusion that search movie will have to mapped to its own component so that 20X traffic and availability requirements can be fulfilled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5wes8loUMFCZ8QtAFjp1-g.png" /></figure><p>Please note that the components within the services are still well encapsulated. This means that if quality attributes changes than component can be decomposed to separate micro services or to some other architecture.</p><h3>4. Environment</h3><p>The purpose of this step is to discuss about the <strong>environmental</strong> <strong>constraints</strong> from business and system perspective.</p><p>The environmental constraints can be identified by asking following questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WUcgLba79L8xFTxHmnzBnQ.png" /></figure><p>In case of moving booking, let’s assume that Payment and Email will be managed by Third Party Vendor and team size is limited.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F4ii6z8OnsqZdf3ypZ1QoA.png" /></figure><p>Please note that the components within the services are still well encapsulated. This means that as business requirement changes, the movie booking application can be de-composed to separate micro-services or to some other architecture.</p><h3>5. Evaluation</h3><p>Once this approach is followed across all use cases, you will get following blueprint that you can discuss with team and stakeholders for <strong>evaluation and validation</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WdWbN4pEnhGqqSLQmBmNNA.png" /></figure><p><strong>Note</strong> : The Movie Management App doesn’t have any orchestration <strong>layer</strong> because admin use case is simple and have CRUD like <strong>operations</strong>.</p><h3>Next Phase — Evolution</h3><p>After the 1st Launch, business would add more capabilities such as customer registration, login etc and architect can follow similar approach to come up with high level blueprint as shown in below diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i24CyEmr9ZDYZpqpNl1NJA.png" /></figure><p>Please note that the above blueprint architecture diagrams are for explaining the approach and shouldn’t be used as reference architecture for any app.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99aa482773a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Facade Vs Adapter Vs Orchestrator]]></title>
            <link>https://atul-agrawal.medium.com/facade-vs-adapter-vs-orchestrator-dbf4fecd660b?source=rss-6bf43639c07f------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbf4fecd660b</guid>
            <category><![CDATA[software-design]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[design-patterns]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Atul Agrawal]]></dc:creator>
            <pubDate>Wed, 09 Mar 2022 01:30:02 GMT</pubDate>
            <atom:updated>2022-03-09T07:34:49.846Z</atom:updated>
            <content:encoded><![CDATA[<h3>Facade</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*MmxcWyTWIsrdBpe2-YzyBQ.png" /><figcaption>Facade Pattern</figcaption></figure><h3>Adapter</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*SWqQpdKZOSuAn2190K5lDA.png" /><figcaption>Adapter Pattern</figcaption></figure><h3>Orchestrator</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*1tePbnbqEcV9AQA6YTJyiw.png" /><figcaption>Orcehstrator Pattern</figcaption></figure><h3>Side by Side Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zqIb3vFJAzcI_P-T8th7ug.png" /><figcaption>Facade vs Adapter vs Orchestrator</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbf4fecd660b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[REST vs GraphQL vs gRPC]]></title>
            <link>https://atul-agrawal.medium.com/rest-vs-graphql-vs-grpc-a376da91daa9?source=rss-6bf43639c07f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a376da91daa9</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[grpc]]></category>
            <category><![CDATA[rest]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Atul Agrawal]]></dc:creator>
            <pubDate>Fri, 11 Feb 2022 02:21:19 GMT</pubDate>
            <atom:updated>2022-02-11T02:27:09.613Z</atom:updated>
            <content:encoded><![CDATA[<p>Today, developers have three choices for API development — REST, GraphQL and gRPC. The purpose of this article is highlight key differences between each of them so that developers can pick the right method for solving their problems.</p><h3>REST</h3><p><strong>Description: </strong><a href="https://en.wikipedia.org/wiki/Representational_state_transfer"><em>REST</em></a><em> is an acronym for Representational State Transfer. In this approach, client calls the server to get the current state of the resources in the specified representation format (JSON, XML) and server responds to the request by returning the resource’s representation.</em></p><p><strong>Key characteristics </strong>: Stateless, Client-Server interaction.</p><p><strong>Message Format : </strong><a href="https://json-schema.org/understanding-json-schema/">JSON</a><strong>, </strong><a href="https://developer.mozilla.org/en-US/docs/Web/XML/XML_introduction">XML</a> (common)</p><p><strong>Protocols : </strong><a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview">HTTP</a> (common)</p><p><strong>Communication Channel </strong>: Unidirectional — Client to Server</p><p><strong>Maturity</strong> : Very High</p><p><strong>Strengths:</strong></p><ul><li><em>Resource Based</em><strong><em> </em></strong>— The business capabilities are modeled as resources and their state is represented using JSON/XML. The boundaries between the APIs is enforced by design using HTTP methods (GET, PUT, POST etc).</li><li><em>Simplicity </em><strong>— </strong>The API contract is simple to understand and use. The HTTP operations are self explanatory as developers have very good of understanding of the semantics of the HTTP protocol.</li></ul><p><strong>Limitations</strong> :</p><ul><li><em>Complex relationships </em>: If relationships between resources is multi-directional or have multi-level hierarchies (Social Media use case) then model resources modelling become challenging and performance becomes a major factor for both client and server due to over-fetching and under-fetching of the underlying resource data. This also have side effect on network due to back and forth chatty calls.</li><li><em>Streaming</em><strong> : </strong>REST<strong> </strong>is not built for streaming use cases due to the constraints imposed by its specification (Client-Sever, Stateless etc).</li><li><em>Payload</em><strong> : </strong>REST uses human readable formats (JSON, XML etc) and due to this the payload size is relatively larger and can cause impact on the applications that are very sensitive to network bandwidth (E.g. IOT devices).</li></ul><p><strong>Recommended Use Cases :</strong></p><p>In general, REST can be used to used to model business capabilities that are not data intensive and streaming is not expected.</p><h3>GraphQL</h3><p><strong>Description</strong> : GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. The client uses the GraphQL Query to specify the data fields that it needs and server returns the values in JSON response.</p><p><strong>Key characteristics </strong>: Flexibility, Performance.</p><p><strong>Message Format : </strong><a href="https://www.tutorialspoint.com/graphql/graphql_query.htm">GraphQL Query</a>, <a href="https://json-schema.org/understanding-json-schema/">JSON</a></p><p><strong>Protocols: </strong><a href="https://graphql.org/learn/serving-over-http/">GraphQL over HTTP</a></p><p><strong>Communication Channel </strong>: Unidirectional — Client to Server</p><p><strong>Maturity</strong> : High</p><p><strong>Strengths:</strong></p><ul><li><em>Field Level Control:</em> Client have fine-grained control at field level as they can query specific data that they need. This fixes the problem of <a href="https://stackoverflow.com/questions/44564905/what-is-over-fetching-or-under-fetching">over-fetching and under-fetching</a> leading to significant performance improvements.</li><li><em>Data Sources:</em><strong> </strong>GraphQL API can support various types of data sources — You can plug combination of REST APIs, Database Queries or File based SoR for providing the response. (Check <a href="https://www.apollographql.com/docs/tutorial/resolvers/">Resolver</a>)</li></ul><p><strong>Limitations</strong> :</p><ul><li><em>Streaming : </em><strong>GraphQL </strong>doesn’t support streaming out of the box right now. P.S. The feature roadmap has proposal for streaming.</li></ul><p><strong>API Contract</strong> : GraphQL uses “Query” and “Mutation” for representing the API operations. This forces the developers to define the interfaces in “SOAP” style format.</p><p><strong>Payload : </strong>Like REST, GraphQL uses human readable formats (JSON) and due to this the payload size is relatively bigger that can have impact on the applications which are very sensitive to network bandwidth (E.g. IOT devices).</p><p><strong>Recommended Use Cases :</strong></p><p>GraphQL is well suited for applications that needs to expose complex representations to different clients (Social media, Twitter). It can also be used for providing standard unified interface to client by hiding the underlying technology stack (REST, DB, File etc).</p><h3>gRPC</h3><p><strong>Description</strong>: In gRPC, a client application can directly call a method on a server application on a different machine as if it were a local object, making it easier for you to create distributed applications and services.</p><p><strong>Key characteristics </strong>: Performance. Streaming.</p><p><strong>Message Format : </strong><a href="https://developers.google.com/protocol-buffers/docs/overview">Protocol Buffers </a>(ProtoBuf)</p><p><strong>Protocols: </strong><a href="https://en.wikipedia.org/wiki/HTTP/2">HTTP</a></p><p><strong>Communication Channel </strong>: Bidirectional— Client to Server</p><p><strong>Maturity</strong> : High</p><p><strong>Strengths:</strong></p><ul><li><em>Streaming</em>: gRPC have native support for streaming that makes it a preferred choice for streaming application as comparison to GraphQL and REST.</li><li><em>Payload</em><strong> : </strong>gRPC using binary encoding for request and response using Protobuff. This helps in reducing the payload size and helps in improving the network performance.</li></ul><p><strong>Limitations</strong>:</p><ul><li><em>API Contract:</em><strong> </strong>Like GraphQL, gRPC also uses SOAP style contract for defining the business operations.</li></ul><p><strong>Recommended Use Cases :</strong></p><p>gRPC is great fit for use cases where machine to machine interaction is required (e.g. Real-time dashboards and IoT).</p><h3><strong>References</strong></h3><ul><li><a href="https://restfulapi.net/">What is REST?: REST API Tutorial</a></li><li><a href="https://nordicapis.com/rest-vs-streaming-apis-how-they-differ/">REST vs Streaming APIs: How They Differ | Nordic APIs |</a></li><li><a href="https://stackoverflow.com/questions/44564905/what-is-over-fetching-or-under-fetching">What is Over-Fetching or Under-fetching?</a></li><li><a href="https://www.apollographql.com/blog/community/backend/new-features-in-graphql-batch-defer-stream-live-and-subscribe/">New features in GraphQL: Batch, defer, stream, live, and subscribe</a></li><li><a href="https://graphql.org/">GraphQL | A query language for your API</a></li><li><a href="https://www.apollographql.com/">Streamlining APIs, Databases, &amp; Microservices | Apollo GraphQL</a></li><li><a href="https://grpc.io/">gRPC</a></li></ul><blockquote>Please provide your feedback in case if I have missed any important details.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a376da91daa9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>