<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Debarshi Raj Basumatary on Medium]]></title>
        <description><![CDATA[Stories by Debarshi Raj Basumatary on Medium]]></description>
        <link>https://medium.com/@basumatary18?source=rss-1fac9181bc9c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2xcqFYlZ6WrlmLdCdUJP2A.jpeg</url>
            <title>Stories by Debarshi Raj Basumatary on Medium</title>
            <link>https://medium.com/@basumatary18?source=rss-1fac9181bc9c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 05:46:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@basumatary18/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[System Design & Distributed Systems: Senior / Lead Engineer Interview Roadmap]]></title>
            <link>https://medium.com/@basumatary18/system-design-distributed-systems-senior-lead-engineer-interview-roadmap-e1b937e7d47d?source=rss-1fac9181bc9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1b937e7d47d</guid>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Debarshi Raj Basumatary]]></dc:creator>
            <pubDate>Wed, 07 Oct 2026 16:16:30 GMT</pubDate>
            <atom:updated>2026-10-07T16:16:30.499Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A comprehensive 120-hour curriculum for scalable, reliable and production-grade architecture</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bUKwzSBI956zZpqh" /><figcaption>Photo by <a href="https://unsplash.com/@virussinside?utm_source=medium&amp;utm_medium=referral">Artiom Vallat</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>1. Purpose and Target</h3><p>This curriculum is the system-design companion to the Spring Boot Senior/Lead roadmap. It is designed to develop the ability to take ambiguous business requirements, translate them into technical requirements, estimate scale, choose architecture and data models, reason about distributed-system trade-offs, and defend the design under interviewer pressure.</p><p>The target is not to memorize architecture diagrams. A senior/lead engineer should be able to explain why a component exists, what happens when it fails, where state lives, how the system scales, what consistency guarantees exist, and how the system is operated.</p><h4>Target outcome</h4><ul><li>Decompose ambiguous requirements into functional and non-functional requirements.</li><li>Perform back-of-the-envelope capacity and traffic estimation.</li><li>Design APIs, domain boundaries and data models.</li><li>Understand DNS, HTTP, TLS, load balancing and reverse proxies.</li><li>Choose SQL, NoSQL, cache, object storage and search technologies based on workload.</li><li>Reason about replication, partitioning, sharding and consistency.</li><li>Understand CAP, PACELC, quorum systems and consistency models.</li><li>Design queues, streams and event-driven architectures.</li><li>Handle retries, timeouts, idempotency, deduplication and backpressure.</li><li>Design reliable distributed workflows and transactions.</li><li>Apply caching, rate limiting, distributed locking and coordination.</li><li>Design for observability, disaster recovery and operational excellence.</li><li>Communicate architecture clearly using diagrams, APIs, schemas and trade-off analysis.</li><li>Solve senior/lead system-design interviews within a realistic time limit.</li></ul><h3>2. Curriculum at a Glance</h3><ul><li><strong>1. System Design Method</strong> — Hours: 6 · Primary outcome: Structured interview approach and requirement decomposition</li><li><strong>2. Networking + Web Foundations</strong> — Hours: 8 · Primary outcome: Understand the infrastructure beneath services</li><li><strong>3. Scalability + Load Balancing</strong> — Hours: 8 · Primary outcome: Scale stateless and stateful workloads</li><li><strong>4. Data Storage + SQL</strong> — Hours: 10 · Primary outcome: Model data and reason about database performance</li><li><strong>5. NoSQL + Distributed Data</strong> — Hours: 8 · Primary outcome: Select and scale non-relational stores</li><li><strong>6. Caching</strong> — Hours: 6 · Primary outcome: Design multi-layer caching and invalidation</li><li><strong>7. Messaging + Streaming</strong> — Hours: 10 · Primary outcome: Queues, Kafka-style streams and delivery semantics</li><li><strong>8. Distributed Systems Theory</strong> — Hours: 12 · Primary outcome: Consistency, CAP, consensus and coordination</li><li><strong>9. Reliability + Resilience</strong> — Hours: 10 · Primary outcome: Build systems that survive failures</li><li><strong>10. Distributed Transactions</strong> — Hours: 8 · Primary outcome: Saga, outbox, idempotency and workflow design</li><li><strong>11. Search + Storage + Specialized Systems</strong> — Hours: 6 · Primary outcome: Search, object storage, files and time-series workloads</li><li><strong>12. Security Architecture</strong> — Hours: 6 · Primary outcome: Identity, authorization and service-to-service security</li><li><strong>13. Observability + Operations</strong> — Hours: 6 · Primary outcome: Operate systems using SLOs and telemetry</li><li><strong>14. Cloud + Deployment Architecture</strong> — Hours: 6 · Primary outcome: Containers, Kubernetes and multi-region design</li><li><strong>15. Advanced Architecture</strong> — Hours: 8 · Primary outcome: Event-driven, CQRS, event sourcing and platform thinking</li><li><strong>16. Interview Case Studies + Mocks</strong> — Hours: 6 · Primary outcome: End-to-end senior/lead interview readiness</li><li><strong>TOTAL</strong> — Hours: 120 · Primary outcome: Senior/lead system-design capability</li></ul><h3>3. The Core System-Design Mental Model</h3><p>For every design, continuously reason across six dimensions:</p><ul><li><strong>Correctness</strong> — What must never happen? What invariants must hold?</li><li><strong>Scale</strong> — How many users, requests/sec, records and bytes?</li><li><strong>Latency</strong> — What is the p50/p95/p99 target? Which calls are on the critical path?</li><li><strong>Availability</strong> — What happens when dependencies fail? What availability is actually required?</li><li><strong>Consistency</strong> — Which data must be strongly consistent and which can be eventually consistent?</li><li><strong>Operations</strong> — How will we detect, debug, deploy, recover and evolve the system?</li></ul><p>A good architecture is a set of explicit trade-offs, not a collection of fashionable technologies.</p><h3>4. Phase 1 — System Design Method (6 hours)</h3><h4>The interview framework</h4><ol><li>Clarify the problem and define scope.</li><li>Separate functional requirements from non-functional requirements.</li><li>Estimate traffic, storage and bandwidth.</li><li>Define APIs and major domain entities.</li><li>Choose the high-level architecture.</li><li>Choose storage based on access patterns.</li><li>Design caching and asynchronous flows.</li><li>Address consistency, concurrency and failure.</li><li>Address security and observability.</li><li>Identify bottlenecks and scaling strategies.</li><li>State trade-offs and alternatives.</li><li>Summarize the final architecture.</li></ol><h4>Requirements checklist</h4><ul><li>Users and actors</li><li>Core use cases</li><li>Read/write ratio</li><li>Latency targets</li><li>Availability target</li><li>Data retention</li><li>Geographic scope</li><li>Consistency requirements</li><li>Security/compliance constraints</li><li>Expected growth</li></ul><h4>Back-of-the-envelope estimation</h4><ul><li>Requests per second</li><li>Peak vs average traffic</li><li>Storage per record</li><li>Daily/monthly/yearly storage growth</li><li>Bandwidth</li><li>Cache memory</li><li>Replication multiplier</li><li>Partition count</li><li>Approximate database capacity</li></ul><p>Rule of thumb: use estimates to constrain architecture decisions; do not pretend they are exact production measurements.</p><h3>5. Phase 2 — Networking + Web Foundations (8 hours)</h3><h4>Topics</h4><ul><li>DNS resolution</li><li>TCP connection establishment</li><li>TLS handshake</li><li>HTTP/1.1, HTTP/2 and HTTP/3 concepts</li><li>Keep-alive and connection reuse</li><li>Request/response lifecycle</li><li>Reverse proxies</li><li>CDNs</li><li>Load balancers</li><li>L4 vs L7 load balancing</li><li>NAT and basic network boundaries</li><li>WebSockets and server-sent events</li><li>gRPC vs REST</li><li>Timeouts and connection pools</li></ul><h4>Interview questions</h4><ul><li>What happens after a user enters a URL?</li><li>Why does HTTP/2 improve performance?</li><li>L4 vs L7 load balancer?</li><li>When would you choose WebSockets?</li><li>REST vs gRPC?</li><li>Where should TLS terminate?</li><li>Why are connection pools important?</li></ul><h3>6. Phase 3 — Scalability + Load Balancing (8 hours)</h3><h4>Scaling fundamentals</h4><ul><li>Vertical scaling</li><li>Horizontal scaling</li><li>Stateless application design</li><li>Load-balancing algorithms</li><li>Health checks</li><li>Sticky sessions</li><li>Session externalization</li><li>Autoscaling</li><li>Queue-based load leveling</li><li>Hotspot detection</li></ul><h4>Scaling pattern</h4><p>Client → CDN → Load Balancer → stateless application fleet → cache/database/message broker.</p><h4>Senior questions</h4><ul><li>Why is statelessness useful?</li><li>When are sticky sessions justified?</li><li>What happens if one partition/key becomes hot?</li><li>How do you scale a write-heavy service?</li><li>How do you protect a database when application instances scale rapidly?</li></ul><h3>7. Phase 4 — Data Storage + SQL (10 hours)</h3><h4>Relational modeling</h4><ul><li>Entity and relationship modeling</li><li>Normalization and denormalization</li><li>Primary and foreign keys</li><li>Indexes</li><li>Composite indexes</li><li>Covering indexes</li><li>Query plans</li><li>Joins</li><li>Transactions</li><li>Isolation</li><li>Optimistic and pessimistic locking</li><li>Read replicas</li><li>Connection pooling</li></ul><h4>Database scaling</h4><ul><li>Read replicas</li><li>Vertical scaling</li><li>Partitioning</li><li>Sharding</li><li>Archival</li><li>Materialized views</li><li>CQRS read models</li></ul><h4>Data modeling rule</h4><p>Start from access patterns and invariants, not from a technology choice. Ask: what queries must be fast, what must be transactional, and what can be eventually consistent?</p><h3>8. Phase 5 — NoSQL + Distributed Data (8 hours)</h3><h4>Categories</h4><ul><li><strong>Key-value</strong> — Simple high-volume lookups</li><li><strong>Document</strong> — Flexible aggregate-shaped records</li><li><strong>Wide-column</strong> — Large distributed workloads with predictable access patterns</li><li><strong>Graph</strong> — Relationship-heavy traversals</li><li><strong>Time-series</strong> — Timestamped metrics/events</li></ul><h4>Topics</h4><ul><li>Access-pattern-driven modeling</li><li>Denormalization</li><li>Partition keys</li><li>Hot partitions</li><li>Replication</li><li>Quorum reads/writes</li><li>Eventual consistency</li><li>Multi-model architecture</li><li>Polyglot persistence</li></ul><h4>Interview challenge</h4><p>Choose between PostgreSQL, Dynamo-style key-value storage, document storage and Redis for four different workloads. Defend each choice using access patterns and failure behavior.</p><h3>9. Phase 6 — Caching (6 hours)</h3><h4>Caching layers</h4><p>Browser → CDN → reverse proxy → application cache → distributed cache → database.</p><h4>Patterns</h4><ul><li>Cache-aside</li><li>Read-through</li><li>Write-through</li><li>Write-behind</li><li>Refresh-ahead</li><li>TTL</li><li>Explicit invalidation</li><li>Negative caching</li></ul><h4>Failure modes</h4><ul><li>Cache stampede</li><li>Cache penetration</li><li>Cache avalanche</li><li>Hot keys</li><li>Stale data</li><li>Cache inconsistency</li><li>Cold cache after deployment</li></ul><h4>Interview question</h4><p>Design caching for a product catalog with 10 million products, high read traffic and frequent price updates. Explain invalidation and failure behavior.</p><h3>10. Phase 7 — Messaging + Streaming (10 hours)</h3><h4>Queues vs streams</h4><ul><li><strong>Primary model</strong> — Queue: Work distribution · Stream: Ordered event log</li><li><strong>Consumption</strong> — Queue: Usually one worker handles a message · Stream: Multiple consumers can independently read</li><li><strong>Replay</strong> — Queue: Usually limited · Stream: Core capability in log-based systems</li><li><strong>Typical use</strong> — Queue: Background jobs · Stream: Events, analytics, integration</li></ul><h4>Topics</h4><ul><li>Producer/consumer model</li><li>Kafka-style partitions</li><li>Consumer groups</li><li>Offsets</li><li>Ordering</li><li>Retention</li><li>Replication</li><li>At-most-once delivery</li><li>At-least-once delivery</li><li>Exactly-once concepts</li><li>Idempotent consumers</li><li>Retries</li><li>Dead-letter queues/topics</li><li>Backpressure</li><li>Batching</li><li>Event schemas and compatibility</li></ul><h4>Senior patterns</h4><ul><li>Transactional outbox</li><li>Change data capture</li><li>Event-driven integration</li><li>Work queues</li><li>Delayed jobs</li><li>Priority queues</li><li>Fan-out</li><li>Competing consumers</li></ul><h3>11. Phase 8 — Distributed Systems Theory (12 hours)</h3><h4>Consistency</h4><ul><li>Strong consistency</li><li>Eventual consistency</li><li>Read-your-writes consistency</li><li>Monotonic reads</li><li>Causal consistency</li><li>Session consistency</li></ul><h4>CAP theorem</h4><p>Understand CAP precisely: during a network partition, a distributed system cannot simultaneously guarantee both strong consistency and availability for every operation. Do not reduce CAP to the slogan ‘pick two.’</p><h4>PACELC</h4><p>When there is a partition, choose between availability and consistency; else, choose between latency and consistency.</p><h4>Replication</h4><ul><li>Leader/follower replication</li><li>Multi-leader replication</li><li>Leaderless replication</li><li>Synchronous vs asynchronous replication</li><li>Replication lag</li><li>Quorum reads and writes</li><li>Conflict resolution</li></ul><h4>Consensus and coordination</h4><ul><li>Why consensus exists</li><li>Raft concepts</li><li>Leader election</li><li>Quorum</li><li>Failure detection</li><li>Split-brain</li><li>Distributed locks</li><li>Fencing tokens</li><li>Logical clocks</li><li>Lamport clocks</li><li>Vector clocks — conceptual understanding</li></ul><h4>Senior interview questions</h4><ul><li>What does CAP actually say?</li><li>Why can asynchronous replication lose recent writes?</li><li>What is split-brain?</li><li>Why do distributed locks need expiry/fencing considerations?</li><li>Why is consensus difficult?</li><li>When is eventual consistency acceptable?</li></ul><h3>12. Phase 9 — Reliability + Resilience (10 hours)</h3><h4>Failure taxonomy</h4><ul><li>Process crash</li><li>Network timeout</li><li>Packet loss</li><li>Dependency overload</li><li>Database outage</li><li>Cache outage</li><li>Message duplication</li><li>Partial failure</li><li>Region failure</li><li>Corrupt or inconsistent data</li></ul><h4>Resilience patterns</h4><ul><li>Timeouts</li><li>Retries with exponential backoff</li><li>Jitter</li><li>Circuit breakers</li><li>Bulkheads</li><li>Rate limiting</li><li>Load shedding</li><li>Backpressure</li><li>Graceful degradation</li><li>Fallbacks</li><li>Health checks</li><li>Dead-letter handling</li></ul><h4>Critical rule</h4><p>Never add retries blindly. A retry can amplify an outage. Always ask whether the operation is idempotent, whether the dependency is already overloaded, and whether the latency budget permits another attempt.</p><h4>Reliability exercise</h4><p>A payment dependency has intermittent 2-second latency and 5% errors. Design timeout, retry, circuit breaker, idempotency and user-facing fallback behavior.</p><h3>13. Phase 10 — Distributed Transactions (8 hours)</h3><h4>Why distributed transactions are hard</h4><p>A local database transaction provides atomicity inside one transactional boundary. Once a workflow crosses independent services/databases, atomic commit becomes difficult and expensive.</p><h4>Patterns</h4><ul><li>Two-phase commit — understand conceptually and its trade-offs</li><li>Saga choreography</li><li>Saga orchestration</li><li>Compensating transactions</li><li>Transactional outbox</li><li>Idempotency keys</li><li>Deduplication</li><li>Workflow engines</li><li>Exactly-once business effect vs exactly-once message delivery</li></ul><h4>Payment example</h4><p>Order created → payment requested → payment authorized → inventory reserved → order confirmed. For each step, define the state machine, retry behavior, compensation, idempotency key and failure recovery.</p><h3>14. Phase 11 — Search, Object Storage + Specialized Systems (6 hours)</h3><h4>Search</h4><ul><li>Inverted indexes</li><li>Tokenization and analyzers</li><li>Relevance scoring</li><li>Sharding and replicas</li><li>Indexing pipelines</li><li>Search consistency</li><li>When search engines should not replace the primary database</li></ul><h4>Object storage</h4><ul><li>Blob/object storage</li><li>Multipart upload</li><li>Presigned URLs</li><li>Metadata</li><li>CDN integration</li><li>Lifecycle policies</li><li>Durability vs availability</li></ul><h3>Specialized workloads</h3><ul><li>Time-series databases</li><li>Geospatial systems</li><li>Analytics warehouses</li><li>Data lakes</li><li>When to introduce a specialized store</li></ul><h3>15. Phase 12 — Security Architecture (6 hours)</h3><h4>Topics</h4><ul><li>Authentication vs authorization</li><li>OAuth2 and OIDC</li><li>JWT</li><li>API keys</li><li>mTLS</li><li>Service identities</li><li>Secrets management</li><li>Key rotation</li><li>Least privilege</li><li>Network segmentation</li><li>Zero-trust concepts</li><li>Audit logging</li><li>Data encryption at rest and in transit</li><li>PII and sensitive-data handling</li></ul><h4>Architecture exercise</h4><p>Design authentication for browser users, mobile clients, internal services and third-party integrations. Explain where identity is established and how downstream services authorize requests.</p><h3>16. Phase 13 — Observability + Operations (6 hours)</h3><h4>Three pillars</h4><ul><li>Logs</li><li>Metrics</li><li>Traces</li></ul><h4>Operational concepts</h4><ul><li>SLI, SLO and SLA</li><li>Error budgets</li><li>p50/p95/p99 latency</li><li>Throughput</li><li>Saturation</li><li>RED method</li><li>USE method</li><li>Distributed tracing</li><li>Correlation IDs</li><li>Alert design</li><li>Runbooks</li><li>Incident response</li><li>Postmortems</li></ul><h4>Production scenario</h4><p>p99 latency doubled while average latency stayed stable. Explain why percentile latency is useful and how you would use traces and dependency metrics to isolate the problem.</p><h3>17. Phase 14 — Cloud + Deployment Architecture (6 hours)</h3><h4>Topics</h4><ul><li>Containers</li><li>Docker</li><li>Container networking</li><li>Kubernetes primitives</li><li>Deployment vs StatefulSet concepts</li><li>Services and ingress</li><li>Config and secrets</li><li>Horizontal Pod Autoscaling</li><li>Rolling deployments</li><li>Blue/green deployments</li><li>Canary releases</li><li>Health/readiness probes</li><li>Infrastructure as code concepts</li><li>Multi-AZ deployment</li><li>Multi-region deployment</li></ul><h4>Cloud architecture</h4><p>Think in capabilities rather than provider-specific names: compute, network, object storage, relational DB, distributed cache, queue/stream, secrets, observability and CDN.</p><h3>18. Phase 15 — Advanced Architecture (8 hours)</h3><h4>Event-driven architecture</h4><ul><li>Domain events</li><li>Integration events</li><li>Event contracts</li><li>Schema evolution</li><li>Event choreography</li><li>Event replay</li><li>Consumer versioning</li></ul><h4>CQRS</h4><p>Separate write and read models when their access patterns, scaling needs or consistency requirements materially differ. Do not use CQRS merely because it is fashionable.</p><h4>Event sourcing</h4><ul><li>Event log as source of truth</li><li>Aggregate reconstruction</li><li>Snapshots</li><li>Event schema evolution</li><li>Replay</li><li>Operational complexity</li></ul><h4>Architecture evolution</h4><ul><li>Monolith</li><li>Modular monolith</li><li>Service extraction</li><li>Strangler pattern</li><li>Shared database risks</li><li>Service ownership</li><li>Platform engineering concepts</li></ul><h3>19. Phase 16 — Case Studies</h3><ul><li><strong>URL Shortener</strong> — ID generation, redirects, cache, database scaling, hot keys, analytics</li><li><strong>Rate Limiter</strong> — Token bucket, sliding window, Redis, distributed state, fairness</li><li><strong>Notification System</strong> — Kafka/queue, fan-out, retries, DLQ, user preferences</li><li><strong>E-commerce</strong> — Catalog, cart, order, inventory, payment, consistency</li><li><strong>Payment System</strong> — Idempotency, state machine, retries, reconciliation, audit</li><li><strong>Ticket Booking</strong> — Concurrency, locks, reservations, expiration, overselling</li><li><strong>Food Delivery</strong> — Orders, drivers, location, real-time updates, events</li><li><strong>Ride Sharing</strong> — Geospatial indexing, matching, real-time state</li><li><strong>Chat System</strong> — WebSockets, message ordering, delivery status, fan-out</li><li><strong>Video Platform</strong> — Object storage, CDN, transcoding, metadata, streaming</li><li><strong>Social Feed</strong> — Fan-out on write/read, ranking, caching, hot users</li><li><strong>Search Platform</strong> — Ingestion, indexing, shards, replicas, relevance</li><li><strong>Distributed Scheduler</strong> — Leases, leader election, retries, duplicate execution</li><li><strong>Metrics Platform</strong> — High-cardinality data, ingestion, aggregation, retention</li><li><strong>Cloud File Storage</strong> — Object storage, metadata, sync, conflict resolution</li></ul><h3>20. How to Solve a 45-Minute System Design Interview</h3><ul><li><strong>0–5 min</strong> — Clarify requirements and scope.</li><li><strong>5–10 min</strong> — Estimate scale and identify key constraints.</li><li><strong>10–15 min</strong> — Define APIs and core data model.</li><li><strong>15–25 min</strong> — Draw high-level architecture and explain request flow.</li><li><strong>25–32 min</strong> — Deep dive into the hardest subsystem.</li><li><strong>32–38 min</strong> — Failure, consistency, scaling and security.</li><li><strong>38–43 min</strong> — Observability, operations and trade-offs.</li><li><strong>43–45 min</strong> — Summarize decisions and remaining risks.</li></ul><h4>What interviewers are evaluating</h4><ul><li>Requirement clarification</li><li>Structured thinking</li><li>Scale awareness</li><li>Correctness</li><li>Trade-off quality</li><li>Failure-mode reasoning</li><li>Communication</li><li>Ability to go deep rather than list technologies</li><li>Ability to revise the design when requirements change</li></ul><h3>21. Architecture Trade-Off Matrix</h3><ul><li><strong>Communication</strong> — Option A: REST · Option B: Async events · Typical trade-off: Simplicity/latency vs decoupling/resilience</li><li><strong>Storage</strong> — Option A: SQL · Option B: NoSQL · Typical trade-off: Transactions/query flexibility vs distributed scale/access-pattern specialization</li><li><strong>Cache</strong> — Option A: Local · Option B: Distributed · Typical trade-off: Latency/isolation vs shared state</li><li><strong>Consistency</strong> — Option A: Strong · Option B: Eventual · Typical trade-off: Correctness/coordination vs availability/latency</li><li><strong>Deployment</strong> — Option A: Monolith · Option B: Microservices · Typical trade-off: Operational simplicity vs independent scaling/ownership</li><li><strong>Messaging</strong> — Option A: Queue · Option B: Stream · Typical trade-off: Work distribution vs replayable event log</li><li><strong>Replication</strong> — Option A: Sync · Option B: Async · Typical trade-off: Consistency vs latency/availability</li><li><strong>Scaling</strong> — Option A: Vertical · Option B: Horizontal · Typical trade-off: Simplicity vs elasticity/fault isolation</li></ul><h3>22. Senior/Lead Interview Question Bank</h3><h4>Requirements and estimation</h4><ul><li>How do you start a system design problem?</li><li>How do you estimate peak QPS?</li><li>How do you estimate storage for five years?</li><li>Which requirements can you safely defer?</li></ul><h4>Scalability</h4><ul><li>How do you make a service horizontally scalable?</li><li>What creates hotspots?</li><li>How do you scale a database write path?</li><li>When does caching stop helping?</li></ul><h4>Distributed systems</h4><ul><li>Explain CAP precisely.</li><li>What is quorum?</li><li>Why does replication lag matter?</li><li>What is split-brain?</li><li>Why is consensus necessary?</li></ul><h4>Reliability</h4><ul><li>Why can retries make an outage worse?</li><li>How do circuit breakers work?</li><li>What is bulkheading?</li><li>How do you design graceful degradation?</li></ul><h4>Data</h4><ul><li>SQL vs NoSQL?</li><li>Read replicas vs sharding?</li><li>When would you denormalize?</li><li>How do you prevent hot partitions?</li></ul><h4>Messaging</h4><ul><li>At-least-once vs at-most-once?</li><li>How do you make consumers idempotent?</li><li>Why use an outbox?</li><li>How do you handle poison messages?</li></ul><h4>Architecture</h4><ul><li>When should a monolith become microservices?</li><li>How do you choose service boundaries?</li><li>How do you avoid distributed-monolith architecture?</li><li>When should you use synchronous calls vs events?</li></ul><h3>23. Common Design Mistakes</h3><ul><li>Jumping to technologies before clarifying requirements.</li><li>Using microservices for every problem.</li><li>Saying ‘use Kafka’ without explaining why.</li><li>Ignoring database access patterns.</li><li>Ignoring failure and partial failure.</li><li>Adding retries without idempotency.</li><li>Assuming exactly-once delivery solves business duplication.</li><li>Treating CAP as ‘pick any two.’</li><li>Ignoring hot keys and hot partitions.</li><li>Designing for average traffic instead of peak traffic.</li><li>Ignoring operational complexity.</li><li>Using cache without an invalidation strategy.</li><li>Using distributed locks without understanding lease expiry and fencing.</li><li>Ignoring security until the end.</li><li>Drawing architecture boxes without explaining data flow and state.</li></ul><h3>24. Final Readiness Rubric</h3><ul><li><strong>Foundation</strong> — Can explain basic client/server, APIs, databases, caching and queues.</li><li><strong>Intermediate</strong> — Can design common web systems and explain basic scaling.</li><li><strong>Senior-ready</strong> — Can reason about consistency, partitioning, messaging, resilience, data modeling and operational behavior.</li><li><strong>Lead-ready</strong> — Can set boundaries, quantify trade-offs, handle failure, evolve architecture and communicate decisions across teams.</li></ul><h4>Senior readiness test</h4><ul><li>Solve a URL shortener in 30 minutes.</li><li>Solve a ticket-booking system in 45 minutes.</li><li>Explain CAP, replication and consistency without notes.</li><li>Design an idempotent payment workflow.</li><li>Design a Kafka-based order pipeline with outbox and DLQ.</li><li>Explain how the system behaves when Redis, database, Kafka or a region fails.</li><li>Estimate traffic and storage before selecting infrastructure.</li><li>Defend SQL vs NoSQL and monolith vs microservices choices.</li><li>Identify the first bottleneck and the next scaling step.</li><li>Explain how you would observe and operate the system.</li></ul><h3>25. 4-Week Intensive Interview Plan</h3><ul><li><strong>Week 1</strong> — Focus: Method, networking, scalability, databases · Deliverables: 5 estimation drills + 3 designs</li><li><strong>Week 2</strong> — Focus: NoSQL, caching, messaging, consistency · Deliverables: Kafka/outbox design + 4 designs</li><li><strong>Week 3</strong> — Focus: Reliability, distributed transactions, security, cloud · Deliverables: Failure-mode matrix + 3 designs</li><li><strong>Week 4</strong> — Focus: Advanced architecture + mocks · Deliverables: 7 timed mocks + architecture review</li></ul><h4>Weekly practice rule</h4><p>At least 60% of study time should involve drawing, designing, calculating, debugging hypothetical failures or defending decisions. Reading alone will not build interview skill.</p><h3>26. 12-Week Deep Curriculum</h3><ul><li><strong>1</strong> — Primary theme: System design method + estimation · Core deliverable: Requirement/estimation worksheet</li><li><strong>2</strong> — Primary theme: Networking + web · Core deliverable: Request lifecycle diagram</li><li><strong>3</strong> — Primary theme: Scaling + load balancing · Core deliverable: Horizontally scalable API architecture</li><li><strong>4</strong> — Primary theme: SQL + data modeling · Core deliverable: Database schema + query analysis</li><li><strong>5</strong> — Primary theme: NoSQL + distributed storage · Core deliverable: NoSQL design comparison</li><li><strong>6</strong> — Primary theme: Caching · Core deliverable: Multi-layer cache design</li><li><strong>7</strong> — Primary theme: Queues + Kafka · Core deliverable: Event-driven order flow</li><li><strong>8</strong> — Primary theme: Consistency + CAP + replication · Core deliverable: Consistency decision matrix</li><li><strong>9</strong> — Primary theme: Reliability + resilience · Core deliverable: Failure-mode and recovery plan</li><li><strong>10</strong> — Primary theme: Distributed transactions · Core deliverable: Saga + outbox implementation design</li><li><strong>11</strong> — Primary theme: Security + observability + cloud · Core deliverable: Production architecture</li><li><strong>12</strong> — Primary theme: Mock interviews · Core deliverable: 10 timed designs + final review</li></ul><h3>27. The Lead Engineer Standard</h3><p>A lead-level answer should go beyond drawing Client → API → Database. You should be able to explain the state owned by each component, the invariants that must hold, the expected load, the failure modes, the consistency model, the recovery path, the security boundary, the operational signals, and the trade-offs behind each major decision.</p><p>The strongest interview habit is to repeatedly ask: What happens when this component is slow? What happens when it is unavailable? What happens when a message is duplicated? What happens when traffic is 10× larger? What happens when a region disappears? What data can be stale? What must be correct? How do we know the system is healthy?</p><h3>28. Final Checklist</h3><p><strong>☐</strong> Requirement decomposition</p><p><strong>☐</strong> Functional vs non-functional requirements</p><p><strong>☐</strong> Capacity estimation</p><p><strong>☐</strong> Latency and percentile reasoning</p><p><strong>☐</strong> DNS / HTTP / TLS / networking</p><p><strong>☐</strong> CDN / reverse proxy / load balancer</p><p><strong>☐</strong> Horizontal scaling and statelessness</p><p><strong>☐</strong> SQL data modeling and indexes</p><p><strong>☐</strong> Replication / partitioning / sharding</p><p><strong>☐</strong> NoSQL access-pattern design</p><p><strong>☐</strong> Caching and invalidation</p><p><strong>☐</strong> Queues and streams</p><p><strong>☐</strong> Kafka fundamentals</p><p><strong>☐</strong> Delivery semantics and idempotency</p><p><strong>☐</strong> CAP / PACELC</p><p><strong>☐</strong> Consistency models</p><p><strong>☐</strong> Quorums and replication</p><p><strong>☐</strong> Consensus concepts</p><p><strong>☐</strong> Distributed locking</p><p><strong>☐</strong> Timeouts / retries / circuit breakers</p><p><strong>☐</strong> Rate limiting / backpressure / load shedding</p><p><strong>☐</strong> Saga / outbox / distributed workflows</p><p><strong>☐</strong> Search / object storage</p><p><strong>☐</strong> Security architecture</p><p><strong>☐</strong> Observability / SLOs</p><p><strong>☐</strong> Cloud / Kubernetes concepts</p><p><strong>☐</strong> Multi-AZ / multi-region design</p><p><strong>☐</strong> CQRS / event sourcing concepts</p><p><strong>☐</strong> Architecture evolution</p><p><strong>☐</strong> Timed system-design interviews</p><h3>29. Definition of Done</h3><p>You are ready for senior/lead system-design interviews when you can consistently produce a coherent design under time pressure, quantify the major constraints, choose appropriate data and communication patterns, identify failure modes, explain consistency, design for observability and security, and defend trade-offs when the interviewer changes a requirement.</p><p>The objective is architectural judgment: knowing not only what can be built, but what should be built, why, at what scale, with what guarantees, and at what operational cost.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1b937e7d47d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spring Boot: Senior / Lead Engineer Interview Roadmap]]></title>
            <link>https://medium.com/@basumatary18/spring-boot-senior-lead-engineer-interview-roadmap-48062c95eb34?source=rss-1fac9181bc9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/48062c95eb34</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[spring-boot]]></category>
            <dc:creator><![CDATA[Debarshi Raj Basumatary]]></dc:creator>
            <pubDate>Wed, 07 Oct 2026 14:07:20 GMT</pubDate>
            <atom:updated>2026-10-07T14:07:20.197Z</atom:updated>
            <content:encoded><![CDATA[<p>A comprehensive 100-hour curriculum from Spring fundamentals to production architecture</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Cqn0CZpNycdt-_v0" /><figcaption>Photo by <a href="https://unsplash.com/@furtado?utm_source=medium&amp;utm_medium=referral">Felipe Furtado</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Designed around the 80/20 principle: master the concepts that repeatedly appear in real systems and senior-level interviews.</em></p><h3>1. Executive Summary</h3><p>This curriculum is designed to take a Java developer from practical Spring Boot familiarity to senior/lead-level interview readiness. The target is not encyclopedic knowledge of every Spring project. The target is the ability to explain Spring internals, build production-grade services, diagnose failures, reason about performance and concurrency, design secure and resilient distributed systems, and defend architectural trade-offs.</p><p>The first 20 hours follow a Josh Kaufman-style rapid-skill acquisition approach: deconstruct the skill, learn enough to self-correct, remove setup friction, and spend most of the time deliberately practicing. The remaining 80 hours progressively deepen the stack.</p><h4>Target outcome</h4><ul><li>Build a production-style Spring Boot REST service from an empty repository.</li><li>Explain the request lifecycle from Tomcat/Servlet infrastructure through DispatcherServlet, Controller, Service, Repository and database.</li><li>Explain IoC, dependency injection, bean lifecycle, auto-configuration, proxies and AOP instead of merely memorizing annotations.</li><li>Reason about JPA/Hibernate persistence context, dirty checking, lazy loading, N+1 queries, transactions and isolation.</li><li>Design authentication/authorization using Spring Security, JWT and OAuth2/OIDC concepts.</li><li>Handle concurrency, async execution, thread pools and virtual threads safely.</li><li>Build tested services with unit, slice, integration and Testcontainers-based tests.</li><li>Use Redis, Kafka and event-driven patterns such as retries, DLQs and the transactional outbox.</li><li>Design resilient microservices with gateway, discovery, timeouts, retries, circuit breakers and rate limiting.</li><li>Operate services using Actuator, Micrometer, metrics, logs, traces and OpenTelemetry.</li><li>Discuss JVM, GC, connection pools, thread pools and database bottlenecks during performance interviews.</li><li>Solve senior system-design problems and clearly explain trade-offs, failure modes and operational concerns.</li></ul><h3>2. Curriculum at a Glance</h3><ul><li><strong>1. Spring Core + Boot</strong> — Hours: 20 · Primary outcome: Strong Spring mental model and independent CRUD service</li><li><strong>2. Spring MVC + REST</strong> — Hours: 10 · Primary outcome: Deep HTTP/request lifecycle and API design</li><li><strong>3. JPA + Hibernate + SQL</strong> — Hours: 12 · Primary outcome: Production-grade persistence knowledge</li><li><strong>4. Transactions + Consistency</strong> — Hours: 6 · Primary outcome: Transaction internals and isolation reasoning</li><li><strong>5. Spring Security</strong> — Hours: 10 · Primary outcome: Secure APIs and modern auth architecture</li><li><strong>6. Concurrency + Async</strong> — Hours: 8 · Primary outcome: Threading, executors, async and virtual threads</li><li><strong>7. Testing</strong> — Hours: 6 · Primary outcome: Layered and integration testing strategy</li><li><strong>8. Redis + Caching</strong> — Hours: 4 · Primary outcome: Caching patterns and failure modes</li><li><strong>9. Kafka + Event-Driven Architecture</strong> — Hours: 8 · Primary outcome: Messaging and distributed workflows</li><li><strong>10. Microservices + Spring Cloud</strong> — Hours: 8 · Primary outcome: Resilient distributed service architecture</li><li><strong>11. Observability + Production</strong> — Hours: 4 · Primary outcome: Operate and diagnose services</li><li><strong>12. Performance + JVM</strong> — Hours: 4 · Primary outcome: Performance analysis and bottleneck diagnosis</li><li><strong>TOTAL</strong> — Hours: 100 · Primary outcome: Senior/lead interview foundation</li></ul><h3>3. How to Study</h3><h4>The learning loop</h4><ol><li>Learn the mental model before memorizing APIs.</li><li>Implement the concept in a small Spring Boot application.</li><li>Break it deliberately: invalid input, failures, concurrency, database errors, timeouts.</li><li>Inspect what Spring actually does using logs, debugger, SQL logs, Actuator and tests.</li><li>Explain the concept aloud using the Feynman technique.</li><li>Answer interview questions without notes.</li><li>Rebuild important features from a blank project.</li></ol><h4>Recommended time split</h4><ul><li><strong>Concept learning</strong> — Share: 20–25% · Rule: Use official documentation and focused explanations.</li><li><strong>Coding</strong> — Share: 45–50% · Rule: Write the feature yourself; avoid tutorial copying.</li><li><strong>Debugging/experiments</strong> — Share: 15–20% · Rule: Break assumptions and inspect internals.</li><li><strong>Interview practice</strong> — Share: 10–15% · Rule: Explain, compare, design and defend decisions.</li></ul><p>Do not treat the 100 hours as 100 hours of videos. Senior competence comes from implementation, debugging and architectural reasoning.</p><h3>4. Phase 1 — Spring Core + Spring Boot (20 hours)</h3><h4>Objectives</h4><ul><li>Understand IoC, dependency injection, beans and ApplicationContext.</li><li>Understand component scanning and configuration.</li><li>Understand Spring bean lifecycle and post-processors.</li><li>Understand Spring Boot auto-configuration, starters, profiles and configuration properties.</li><li>Build a REST service using Controller → Service → Repository layering.</li><li>Understand enough of AOP/proxies to reason about @Transactional and @Async.</li></ul><h4>20-hour schedule</h4><ul><li><strong>0–1</strong> — Focus: Spring vs Spring Boot · Practice / checkpoint: Explain why Boot exists and what SpringApplication.run starts.</li><li><strong>1–2</strong> — Focus: IoC + dependency injection · Practice / checkpoint: Rewrite manual object construction using constructor injection.</li><li><strong>2–3</strong> — Focus: Beans + component scanning · Practice / checkpoint: Trace how Controller receives a Service bean.</li><li><strong>3–4</strong> — Focus: REST + DispatcherServlet · Practice / checkpoint: Build GET/POST endpoints and trace request flow.</li><li><strong>4–5</strong> — Focus: HTTP request/response · Practice / checkpoint: Use path variables, query params, request bodies and status codes.</li><li><strong>5–6</strong> — Focus: Service layer · Practice / checkpoint: Move business logic out of controllers.</li><li><strong>6–7</strong> — Focus: DTOs · Practice / checkpoint: Create request/response DTOs and mappings.</li><li><strong>7–8</strong> — Focus: JPA mental model · Practice / checkpoint: Trace Spring Data → JPA → Hibernate → JDBC → DB.</li><li><strong>8–9</strong> — Focus: Entities · Practice / checkpoint: Create Task entity and schema.</li><li><strong>9–10</strong> — Focus: Repositories · Practice / checkpoint: Implement CRUD and derived queries.</li><li><strong>10–11</strong> — Focus: Relationships · Practice / checkpoint: Implement User → Tasks with @ManyToOne.</li><li><strong>11–12</strong> — Focus: Transactions · Practice / checkpoint: Observe commit/rollback behavior.</li><li><strong>12–13</strong> — Focus: Validation · Practice / checkpoint: Add Bean Validation and @Valid.</li><li><strong>13–14</strong> — Focus: Exception handling · Practice / checkpoint: Create @ControllerAdvice and consistent error responses.</li><li><strong>14–15</strong> — Focus: Configuration · Practice / checkpoint: Profiles, environment variables and configuration properties.</li><li><strong>15–16</strong> — Focus: Security fundamentals · Practice / checkpoint: Understand SecurityFilterChain and auth vs authorization.</li><li><strong>16–17</strong> — Focus: Capstone skeleton · Practice / checkpoint: Start Expense Management API from blank project.</li><li><strong>17–18</strong> — Focus: CRUD implementation · Practice / checkpoint: Implement users, expenses and categories.</li><li><strong>18–19</strong> — Focus: Production hardening · Practice / checkpoint: Validation, exceptions, DTOs and transactions.</li><li><strong>19–20</strong> — Focus: Challenge mode · Practice / checkpoint: Break the API and explain the full request-to-database path.</li></ul><h4>Phase 1 interview checkpoints</h4><ul><li>What happens when SpringApplication.run() executes?</li><li>What is a Spring Bean?</li><li>How does component scanning work?</li><li>Why is constructor injection generally preferred?</li><li>What is ApplicationContext?</li><li>What is the difference between BeanFactory and ApplicationContext?</li><li>What is @SpringBootApplication composed of?</li><li>What is auto-configuration?</li><li>Why can @Transactional fail during self-invocation?</li><li>What is the difference between a Spring-managed object and an ordinary Java object?</li></ul><h3>5. Phase 2 — Spring MVC + REST (10 hours)</h3><h4>Core topics</h4><ul><li>Servlet container and embedded Tomcat</li><li>DispatcherServlet</li><li>HandlerMapping and HandlerAdapter</li><li>MessageConverters and Jackson</li><li>Filters vs Interceptors</li><li>Argument resolvers</li><li>Exception resolution</li><li>REST resource modeling</li><li>HTTP semantics and status codes</li><li>Pagination, filtering and sorting</li><li>API versioning</li><li>Idempotency</li><li>ETags and conditional requests</li><li>OpenAPI/Swagger</li></ul><h4>Request lifecycle to master</h4><p>Client → HTTP → Tomcat → Servlet infrastructure → DispatcherServlet → HandlerMapping → Controller → Service → Repository → response conversion → HTTP response.</p><h4>Senior interview exercises</h4><ul><li>Design a paginated /orders API.</li><li>Explain PUT vs PATCH and when each is appropriate.</li><li>Design an idempotent payment endpoint.</li><li>Explain how Jackson converts JSON into a Java object.</li><li>Explain Filter vs Interceptor and where authentication belongs.</li><li>Design a consistent error response format for a large API.</li></ul><h3>6. Phase 3 — JPA + Hibernate + SQL (12 hours)</h3><h4>Architecture to understand</h4><p>Spring Data JPA → JPA API → Hibernate implementation → EntityManager → Persistence Context → JDBC → database.</p><h4>Must-master topics</h4><ul><li>Entity lifecycle: transient, managed, detached, removed</li><li>Persistence context and first-level cache</li><li>EntityManager: persist, merge, remove, detach, refresh</li><li>Dirty checking and write-behind</li><li>Flush vs commit</li><li>LAZY vs EAGER loading</li><li>N+1 query problem</li><li>JOIN FETCH, EntityGraph and projections</li><li>Cascade types</li><li>Orphan removal</li><li>Optimistic locking with @Version</li><li>Pessimistic locking</li><li>JDBC batching</li><li>Second-level cache concepts</li><li>Open Session in View</li><li>Indexes and query plans</li><li>Connection pooling and HikariCP</li></ul><h4>Senior interview scenarios</h4><ul><li>Why is Hibernate issuing 101 SQL queries for 100 orders?</li><li>Why does changing an entity sometimes persist without calling save()?</li><li>Why can EAGER fetching be dangerous?</li><li>When would you use DTO projection instead of fetching entities?</li><li>How would you prevent two users from purchasing the last inventory item?</li><li>How would you diagnose connection pool exhaustion?</li></ul><h3>7. Phase 4 — Transactions + Data Consistency (6 hours)</h3><h4>Topics</h4><ul><li>ACID</li><li>Spring transaction abstraction</li><li>TransactionManager</li><li>Proxy-based transaction interception</li><li>@Transactional propagation</li><li>Rollback rules</li><li>Isolation levels</li><li>Dirty reads, non-repeatable reads, phantom reads and lost updates</li><li>Optimistic vs pessimistic locking</li><li>Transaction boundaries</li><li>Self-invocation problem</li><li>Database transactions vs distributed transactions</li></ul><h4>Propagation to know</h4><ul><li><strong>REQUIRED</strong> — Join existing transaction or create one.</li><li><strong>REQUIRES_NEW</strong> — Suspend existing transaction and create a new one.</li><li><strong>NESTED</strong> — Use a nested transaction/savepoint where supported.</li><li><strong>SUPPORTS</strong> — Run with a transaction if one already exists.</li><li><strong>MANDATORY</strong> — Require an existing transaction.</li><li><strong>NOT_SUPPORTED</strong> — Run outside a transaction.</li><li><strong>NEVER</strong> — Fail if a transaction exists.</li></ul><h4>Interview challenge</h4><p>Explain exactly why @Transactional is commonly implemented through a proxy, why self-invocation bypasses that proxy, and how transaction boundaries should be placed around business operations.</p><h3>8. Phase 5 — Spring Security (10 hours)</h3><h4>Security architecture</h4><p>HTTP request → Security Filter Chain → authentication → SecurityContext → authorization → controller.</p><h4>Topics</h4><ul><li>SecurityFilterChain</li><li>Authentication and Authorization</li><li>SecurityContext</li><li>AuthenticationProvider</li><li>Password hashing and PasswordEncoder</li><li>Session vs stateless authentication</li><li>JWT structure and validation</li><li>Access vs refresh tokens</li><li>Token expiry and rotation</li><li>CSRF and CORS</li><li>OAuth 2.0 roles and flows</li><li>Authorization Code + PKCE</li><li>Client Credentials</li><li>OpenID Connect</li><li>Resource Server</li><li>Method-level security</li><li>Role-based and permission-based authorization</li></ul><h4>Senior security exercises</h4><ul><li>Design a JWT-secured Spring Boot API.</li><li>Explain why a JWT is not automatically encrypted.</li><li>Explain CSRF and why its risk changes for stateless token-based APIs.</li><li>Design OAuth2 login with an external identity provider.</li><li>Design service-to-service authentication.</li><li>Explain authentication vs authorization with a concrete API example.</li></ul><h3>9. Phase 6 — Concurrency + Async (8 hours)</h3><h4>Topics</h4><ul><li>Java threads and thread safety</li><li>ExecutorService and thread pools</li><li>Callable and Future</li><li>CompletableFuture</li><li>thenApply vs thenCompose vs thenCombine</li><li>Exception handling in async pipelines</li><li>CPU-bound vs I/O-bound work</li><li>Spring @Async and proxy implications</li><li>TaskExecutor</li><li>Thread pool sizing and queueing</li><li>Virtual threads</li><li>Race conditions and synchronization</li><li>Deadlocks and starvation</li><li>Backpressure concepts</li></ul><h4>Interview exercises</h4><ul><li>Parallelize three independent remote calls and combine the results.</li><li>Explain why increasing the thread pool indefinitely makes a service worse.</li><li>Explain when virtual threads help and when they do not.</li><li>Diagnose a service with thread-pool exhaustion.</li><li>Design an asynchronous report-generation endpoint.</li></ul><h3>10. Phase 7 — Testing (6 hours)</h3><h4>Testing pyramid</h4><p>Unit tests → slice tests → integration tests → contract tests → end-to-end tests.</p><h4>Topics</h4><ul><li>JUnit 5</li><li>Mockito</li><li>Mock vs Spy</li><li>Argument matching and verification</li><li>@SpringBootTest</li><li>@WebMvcTest</li><li>@DataJpaTest</li><li>MockMvc</li><li>Integration testing</li><li>Testcontainers</li><li>Real PostgreSQL in tests</li><li>Contract testing concepts</li><li>Test data management</li><li>Testing transactions and rollback</li></ul><h4>Senior expectation</h4><p>You should be able to justify what to mock, what not to mock, and where real infrastructure provides more confidence than mocks.</p><h3>11. Phase 8 — Redis + Caching (4 hours)</h3><h4>Topics</h4><ul><li>Redis data structures at a conceptual level</li><li>Spring Cache abstraction</li><li>@Cacheable, @CachePut and @CacheEvict</li><li>Cache-aside</li><li>Write-through and write-behind concepts</li><li>TTL and invalidation</li><li>Cache stampede</li><li>Cache penetration</li><li>Cache avalanche</li><li>Distributed locking concepts</li><li>Failure behavior when Redis is unavailable</li></ul><h4>Interview challenge</h4><p>Design a cache for a product catalog and explain consistency, invalidation, TTL, cold starts and what happens when Redis fails.</p><h3>12. Phase 9 — Kafka + Event-Driven Architecture (8 hours)</h3><h4>Kafka mental model</h4><p>Producer → Topic → Partition → Offset → Consumer Group → Consumer.</p><h4>Topics</h4><ul><li>Topics and partitions</li><li>Ordering guarantees</li><li>Consumer groups and rebalancing</li><li>Offsets</li><li>Replication</li><li>Retention</li><li>Producer acknowledgments</li><li>Serialization/deserialization</li><li>Spring Kafka producers and consumers</li><li>Consumer error handling</li><li>Retries and dead-letter topics</li><li>At-most-once, at-least-once and exactly-once concepts</li><li>Idempotent consumers</li><li>Event-driven architecture</li><li>Transactional Outbox</li><li>CDC concepts</li><li>Saga</li><li>CQRS and Event Sourcing concepts</li></ul><h4>Senior interview exercises</h4><ul><li>Design order → payment → inventory event flow.</li><li>Explain why retries can create duplicate business operations.</li><li>Design an idempotent Kafka consumer.</li><li>Explain the transactional outbox and why it exists.</li><li>Design a retry and dead-letter strategy.</li><li>Explain ordering when a topic has multiple partitions.</li></ul><h3>13. Phase 10 — Microservices + Spring Cloud (8 hours)</h3><h4>Topics</h4><ul><li>Monolith vs modular monolith vs microservices</li><li>Service boundaries</li><li>API Gateway</li><li>Service discovery</li><li>Configuration management</li><li>Load balancing</li><li>OpenFeign / HTTP clients concepts</li><li>WebClient</li><li>Timeouts</li><li>Retries</li><li>Circuit breakers</li><li>Bulkheads</li><li>Rate limiting</li><li>Fallbacks</li><li>Distributed transactions</li><li>Saga choreography vs orchestration</li><li>Eventual consistency</li><li>Containerized deployment concepts</li></ul><h4>Architecture exercise</h4><p>Design Order, Payment, Inventory and Notification services. Identify synchronous vs asynchronous communication, transaction boundaries, failure handling, idempotency, observability and data ownership.</p><h3>14. Phase 11 — Observability + Production (4 hours)</h3><h4>Three pillars</h4><ul><li>Logs</li><li>Metrics</li><li>Distributed traces</li></ul><h4>Topics</h4><ul><li>Spring Boot Actuator</li><li>Health and readiness concepts</li><li>Micrometer</li><li>Prometheus</li><li>Grafana</li><li>Structured logging</li><li>Correlation IDs</li><li>OpenTelemetry</li><li>Trace and span model</li><li>Alerting concepts</li><li>SLI, SLO and SLA basics</li></ul><h4>Incident exercise</h4><p>A production API has p99 latency of 3 seconds. Explain how you would use metrics, logs, traces, thread dumps, connection-pool metrics and database telemetry to locate the bottleneck.</p><h3>15. Phase 12 — Performance + JVM (4 hours)</h3><h4>JVM topics</h4><ul><li>Heap, stack and metaspace</li><li>Young and old generations</li><li>Garbage collection</li><li>JIT compilation</li><li>Thread dumps</li><li>Heap dumps</li><li>CPU and memory profiling concepts</li></ul><h4>Spring/backend bottlenecks</h4><ul><li>Tomcat thread pool exhaustion</li><li>Application executor exhaustion</li><li>HikariCP connection pool exhaustion</li><li>N+1 queries</li><li>Slow SQL and missing indexes</li><li>Excessive serialization</li><li>Large payloads</li><li>GC pressure</li><li>Cache misses</li><li>Remote-service latency</li></ul><h4>Interview exercise</h4><p>Given high CPU, high GC and increasing latency, explain the sequence of evidence you would collect before changing configuration.</p><h3>16. Capstone Projects</h3><h4>Project 1 — Production CRUD API</h4><ul><li>Spring Boot + MVC + PostgreSQL + JPA</li><li>DTOs, validation and exception handling</li><li>Pagination/filtering/sorting</li><li>Unit + integration tests</li><li>Dockerized local environment</li></ul><h4>Project 2 — Secure API</h4><ul><li>Spring Security</li><li>JWT or OAuth2 resource-server model</li><li>Role/permission-based authorization</li><li>Redis caching</li><li>Rate limiting concept</li><li>Security-focused tests</li></ul><h4>Project 3 — Event-Driven Order Platform</h4><ul><li>Order, Payment, Inventory and Notification services</li><li>Kafka</li><li>Transactional outbox</li><li>Retries and DLQ</li><li>Idempotent consumers</li><li>Resilience4j</li><li>Redis</li><li>OpenTelemetry</li><li>Docker Compose</li></ul><h4>Project 4 — Lead-level system design</h4><p>Design and document a scalable order-processing platform. Produce an architecture diagram, API contracts, database schema, event contracts, failure-mode analysis, scaling strategy, security model, observability plan and deployment strategy.</p><h3>17. Senior / Lead Interview Question Bank</h3><h4>Spring Core</h4><ul><li>Explain IoC and DI without using Spring-specific terminology first.</li><li>How does Spring discover and instantiate beans?</li><li>What is a BeanPostProcessor?</li><li>What is the difference between @Component and @Bean?</li><li>Why does constructor injection improve design?</li><li>How does Spring create proxies?</li></ul><h4>Spring Boot</h4><ul><li>What is auto-configuration?</li><li>How do conditional beans work?</li><li>What does @SpringBootApplication contain?</li><li>How do profiles and configuration properties work?</li><li>How would you debug unexpected auto-configuration?</li></ul><h4>MVC</h4><ul><li>Walk through a request from socket to controller.</li><li>Filter vs interceptor?</li><li>How does Jackson serialization work?</li><li>How would you implement API versioning?</li><li>How would you design idempotency?</li></ul><h4>JPA/Hibernate</h4><ul><li>Explain persistence context.</li><li>Explain dirty checking.</li><li>What causes N+1?</li><li>LAZY vs EAGER?</li><li>JOIN FETCH vs EntityGraph?</li><li>Optimistic vs pessimistic locking?</li><li>Why can save() be misleading regarding SQL execution?</li></ul><h4>Transactions</h4><ul><li>Explain @Transactional internals.</li><li>Why does self-invocation matter?</li><li>Explain propagation.</li><li>Explain isolation levels.</li><li>How do you prevent lost updates?</li></ul><h4>Security</h4><ul><li>Authentication vs authorization?</li><li>Explain SecurityFilterChain.</li><li>JWT vs session authentication?</li><li>OAuth2 vs OIDC?</li><li>How does PKCE work conceptually?</li><li>What are CSRF and CORS?</li></ul><h4>Concurrency</h4><ul><li>How do you size an executor?</li><li>CompletableFuture thenApply vs thenCompose?</li><li>How do virtual threads change the concurrency model?</li><li>How do you prevent race conditions?</li></ul><h4>Distributed systems</h4><ul><li>When should a service call another service synchronously?</li><li>When should it publish an event?</li><li>How do retries interact with idempotency?</li><li>Explain circuit breaker states.</li><li>Explain the outbox pattern.</li><li>How do you handle distributed transactions?</li></ul><h3>18. System Design Interview Track</h3><h4>Systems to design</h4><ul><li><strong>URL shortener</strong> — ID generation, cache, database scaling</li><li><strong>E-commerce platform</strong> — Services, inventory, payment, consistency</li><li><strong>Payment system</strong> — Idempotency, transactions, retries</li><li><strong>Ticket booking</strong> — Concurrency, locking, overselling</li><li><strong>Notification platform</strong> — Kafka, workers, retry/DLQ</li><li><strong>Ride sharing</strong> — Real-time state, location, events</li><li><strong>Rate limiter</strong> — Redis, token bucket, distributed state</li><li><strong>Order processing</strong> — Kafka, outbox, Saga, observability</li></ul><h4>Design answer structure</h4><ol><li>Clarify functional and non-functional requirements.</li><li>Estimate scale: users, requests/sec, storage and traffic.</li><li>Define APIs and major domain objects.</li><li>Choose data stores and explain why.</li><li>Define service boundaries.</li><li>Define synchronous and asynchronous flows.</li><li>Discuss consistency and transaction boundaries.</li><li>Design caching and scaling.</li><li>Design failure handling and resilience.</li><li>Design security.</li><li>Design observability.</li><li>Identify trade-offs and future bottlenecks.</li></ol><h3>19. Readiness Rubric</h3><ul><li><strong>Foundation</strong> — Build CRUD APIs and explain IoC, DI, MVC and basic JPA.</li><li><strong>Intermediate</strong> — Build secure, validated APIs with transactions and tests.</li><li><strong>Senior-ready</strong> — Explain internals, diagnose performance issues and design resilient services.</li><li><strong>Lead-ready</strong> — Make architecture decisions, quantify trade-offs, define boundaries and reason about failure at system scale.</li></ul><h4>Do not move on until…</h4><ul><li>You can rebuild a basic service without a tutorial.</li><li>You can explain every major annotation you use.</li><li>You can inspect SQL generated by Hibernate.</li><li>You can explain transaction boundaries and isolation.</li><li>You can diagnose N+1 and connection-pool issues.</li><li>You can explain security flows on a whiteboard.</li><li>You can design retry, timeout and circuit-breaker behavior.</li><li>You can explain Kafka delivery semantics and idempotency.</li><li>You can design a multi-service system and identify consistency boundaries.</li><li>You can defend architecture decisions under interviewer challenge.</li></ul><h3>20. Final 100-Hour Checklist</h3><p><strong>☐</strong> Spring IoC / DI / Beans</p><p><strong>☐</strong> ApplicationContext and bean lifecycle</p><p><strong>☐</strong> Component scanning and configuration</p><p><strong>☐</strong> Spring Boot auto-configuration</p><p><strong>☐</strong> Spring MVC request lifecycle</p><p><strong>☐</strong> REST and HTTP semantics</p><p><strong>☐</strong> DTOs, validation and exception handling</p><p><strong>☐</strong> JPA / Hibernate architecture</p><p><strong>☐</strong> Persistence context and dirty checking</p><p><strong>☐</strong> Fetch strategies and N+1</p><p><strong>☐</strong> Transactions and propagation</p><p><strong>☐</strong> Isolation and locking</p><p><strong>☐</strong> Spring Security</p><p><strong>☐</strong> JWT / OAuth2 / OIDC concepts</p><p><strong>☐</strong> Concurrency and executors</p><p><strong>☐</strong> CompletableFuture and @Async</p><p><strong>☐</strong> Virtual threads</p><p><strong>☐</strong> Testing and Testcontainers</p><p><strong>☐</strong> Redis and caching</p><p><strong>☐</strong> Kafka</p><p><strong>☐</strong> Outbox / Saga / CQRS concepts</p><p><strong>☐</strong> Microservices</p><p><strong>☐</strong> API Gateway and service discovery</p><p><strong>☐</strong> Resilience patterns</p><p><strong>☐</strong> Observability</p><p><strong>☐</strong> JVM and performance</p><p><strong>☐</strong> System design</p><p><strong>☐</strong> Production troubleshooting</p><p><strong>☐</strong> Architecture trade-offs</p><p><strong>☐</strong> Senior/lead interview practice</p><h3>21. Recommended Final Interview Sprint</h3><ul><li><strong>1</strong> — Spring Core + Boot internals; 30 rapid-fire questions</li><li><strong>2</strong> — MVC + REST; design 3 APIs</li><li><strong>3</strong> — JPA/Hibernate; solve 10 persistence scenarios</li><li><strong>4</strong> — Transactions + concurrency; explain 10 failure cases</li><li><strong>5</strong> — Security; design JWT/OAuth2 architecture</li><li><strong>6</strong> — Kafka + microservices; design event-driven workflow</li><li><strong>7</strong> — System design mock interview + production incident drill</li></ul><p>For the final sprint, prioritize explanation and trade-offs over memorization. Senior interviews often test whether you can reason under incomplete information, not whether you remember an annotation’s exact spelling.</p><h3>22. Definition of Done</h3><p>You are ready to target senior/lead Spring Boot interviews when you can take an unfamiliar production scenario, identify the relevant Spring/JVM/database/distributed-system concepts, propose a design, explain alternatives, identify failure modes, and describe how you would validate the design in production.</p><p>The ultimate objective is not 100% API coverage. It is the ability to move fluently between code, framework internals, database behavior, concurrency, distributed systems, security, operations and architecture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48062c95eb34" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Builder Design Pattern]]></title>
            <link>https://medium.com/@basumatary18/builder-design-pattern-cfd24df0c617?source=rss-1fac9181bc9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfd24df0c617</guid>
            <category><![CDATA[design-patterns]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[builder-design-pattern]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[object-oriented-program]]></category>
            <dc:creator><![CDATA[Debarshi Raj Basumatary]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 19:58:44 GMT</pubDate>
            <atom:updated>2026-10-06T19:58:44.981Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0flq_2pb3t9Ir3ll" /><figcaption>Photo by <a href="https://unsplash.com/@madebyswish?utm_source=medium&amp;utm_medium=referral">Emma Houghton</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The Builder pattern is a creational design pattern that allows you to construct complex objects step by step. It separates the construction of a complex object from its actual representation, enabling you to use the same construction process to create different variations of the object.</p><h3>Problems it solves:</h3><p>Imagine we need to build a complex object with some mandatory fields and some optional fields and every use case needs a different combination of field sets. We would need to create multiple sets of overloaded constructors for the class. This is called <strong>telescoping constructors</strong>.</p><p>Here the constructor is too long and reader does not know what does null, true mean</p><pre>new User(&quot;Asha&quot;, &quot;asha@mail.com&quot;, 30, null, null, true); // What does each value mean?</pre><p>Second Problem, <strong>Mutability</strong>. Using setter methods to populate an object introduces two major mutability issues. First, it compromises thread safety; another thread might access the object before all setters have been called, resulting in incomplete data. Second, setters prevent true immutability, as fields cannot be marked as final (since final fields must be assigned directly within a constructor).</p><h3>Solution:</h3><p>The pattern extracts the object construction code out of its own class to a separate Builder Object. The Builder Object provides a set of chainable methods to configure the object and a build() method to finally build the main Object and return it.</p><pre>public final class User {<br><br>    // Fields are final: the object cannot change after creation<br>    private final String name;      // required<br>    private final String email;     // required<br>    private final int age;          // optional<br>    private final String phone;     // optional<br>    private final String fax;       // optional<br><br>    // Private constructor: only the Builder can call it<br>    private User(Builder b) {<br>        this.name  = b.name;<br>        this.email = b.email;<br>        this.age   = b.age;<br>        this.phone = b.phone;<br>        this.fax   = b.fax;<br>    }<br><br>    public static class Builder {<br>        private final String name;     // required fields: set in the constructor<br>        private final String email;<br>        private int age;               // optional fields: default values<br>        private String phone;<br>        private String fax;<br><br>        public Builder(String name, String email) {<br>            this.name = name;<br>            this.email = email;<br>        }<br><br>        public Builder age(int age)        { this.age = age;     return this; }<br>        public Builder phone(String phone) { this.phone = phone; return this; }<br>        public Builder fax(String fax)     { this.fax = fax; return this; }<br>        public User build() {<br>            if (age &lt; 0) throw new IllegalStateException(&quot;Age must not be negative&quot;);<br>            return new User(this);     // validation happens in one place<br>        }<br>    }<br>}<br><br>// Usage: each value has a name<br>User user = new User.Builder(&quot;Asha&quot;, &quot;asha@mail.com&quot;)<br>        .age(30)<br>        .phone(&quot;9999999999&quot;)<br>        .build();</pre><p>Here you can see, User is the Main Class but it has an inner Builder Subclass, we can not set the state of Main Object, we need to create the Builder Object and pass the values there. Builder Object will set the state of Main Object. <br><em>User.Builder(“Asha”, “asha@mail.com”) — </em>name and email are passed because they are mandatory, then we pass age(), phone() but not fax(). Age, Phone and Fax are optional so it is upto us how many we pass.</p><h3>Pros:</h3><ol><li>You can construct the object step by step.</li><li>You can use the same class to create object with different combination of field sets.</li><li>Single Responsibility Principle, Object construction code is separate from business logic unlike Singleton.</li></ol><h3>Cons:</h3><ol><li>It increases the complexity of code base.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfd24df0c617" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Factory Method: Design Pattern]]></title>
            <link>https://blog.stackademic.com/factory-method-design-pattern-8457db682fc8?source=rss-1fac9181bc9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8457db682fc8</guid>
            <category><![CDATA[object-oriented-program]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[design-patterns]]></category>
            <dc:creator><![CDATA[Debarshi Raj Basumatary]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 18:43:47 GMT</pubDate>
            <atom:updated>2026-10-06T18:43:47.152Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qUAU4ghxTezKpD06" /><figcaption>Photo by <a href="https://unsplash.com/@m_simpsan?utm_source=medium&amp;utm_medium=referral">Alex Simpson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Factory Method</strong> is a creational design pattern that provides an interface for creating objects in a superclass, but allows subclasses to alter the type of objects that will be created.</p><h3>Problem:</h3><p>Suppose you’re creating a sms notification service app. The bulk of your code lives in <em>SmsNotification</em> class but as the app became popular and business grew you also need to send notification via <em>Email </em>too. At present, most of your code is coupled to the SmsNotification class. Adding <em>Email</em> into the app would require making changes to the entire codebase. Moreover, if later you decide to add another type of <em>Notification</em> to the app, you will probably need to make all of these changes again.</p><p>As a result, you will end up with pretty nasty code, riddled with conditionals that switch the app’s behavior depending on the class of transportation objects.</p><blockquote>The Factory Method separates product construction code from the code that actually uses the product. Therefore it’s easier to extend the product construction code independently from the rest of the code.</blockquote><h3>Solution:</h3><p>The Factory Method pattern suggests that you replace direct object construction calls (using the <em>new</em> operator) with calls to a special <em>factory</em> method. Don’t worry: the objects are still created via the <em>new </em>operator, but it’s being called from within the factory method. Objects returned by a factory method are often referred to as <em>products.</em></p><p><strong>Common Interface/Product:</strong></p><pre>// 1. PRODUCT: the interface that all channels implement <br>public interface Channel { <br>void deliver(String message); <br>}</pre><p><strong>Concrete Implementation: </strong>Developers define a specific product class corresponding to each distinct feature or type addition.</p><pre>// 2. CONCRETE PRODUCTS: each class implements one delivery method<br>public class EmailChannel implements Channel {<br>    @Override<br>    public void deliver(String message) {<br>        System.out.println(&quot;E-mail sent: &quot; + message);<br>    }<br>}<br><br>public class SmsChannel implements Channel {<br>    @Override<br>    public void deliver(String message) {<br>        System.out.println(&quot;SMS sent: &quot; + message);<br>    }<br>}</pre><p><strong>Abstract Creator Class: </strong>Defines the factory method that returns a Product object.</p><pre>// 3. CREATOR: declares the factory method and uses the Product<br>public abstract class Notifier {<br><br>    // FACTORY METHOD: subclasses must override it<br>    protected abstract Channel createChannel();<br><br>    // Business logic. It does NOT know the concrete class.<br>    public void send(String message) {<br>        Channel channel = createChannel();   // the subclass decides<br>        channel.deliver(message);            // use the Product interface<br>    }<br>}</pre><p><strong>Concrete Creators: </strong>Subclasses override the factory method to instantiate and return specific product variants. Developers create them for each specific type.</p><pre>// 4. CONCRETE CREATORS: each one returns a different Product<br>public class EmailNotifier extends Notifier {<br>    @Override<br>    protected Channel createChannel() {<br>        return new EmailChannel();<br>    }<br>}<br><br>public class SmsNotifier extends Notifier {<br>    @Override<br>    protected Channel createChannel() {<br>        return new SmsChannel();<br>    }<br>}</pre><p>Finally Client creates the Email Notification Object.</p><pre>// 5. CLIENT: works with the Creator type only<br>public class Main {<br>    public static void main(String[] args) {<br>        Notifier notifier = new EmailNotifier();   // change this one line<br>        notifier.send(&quot;Your order is shipped.&quot;);   // to switch the channel<br>    }<br>}</pre><figure><img alt="Flow Chart to explain the Factory Method Design pattern used for a Notification Service App." src="https://cdn-images-1.medium.com/max/1024/1*zbiM94T22HWLqK49rjIgwA.png" /></figure><h3>Pros and Cons:</h3><p>Pros</p><ul><li>You avoid tight coupling between the creator and the concrete products.</li><li><strong><em>Single Responsibility Principle</em></strong>. You can move the product creation code into one place in the program, making the code easier to support.</li><li><strong><em>Open/Closed Principle</em></strong>. You can introduce new types of products into the program without breaking existing client code.</li></ul><p>Cons</p><ul><li>The code may become more complicated since you need to introduce a lot of new subclasses to implement the pattern. The best case scenario is when you’re introducing the pattern into an existing hierarchy of creator classes.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8457db682fc8" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/factory-method-design-pattern-8457db682fc8">Factory Method: Design Pattern</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Singleton: Design Pattern]]></title>
            <link>https://medium.com/@basumatary18/singleton-design-pattern-a8923df636c9?source=rss-1fac9181bc9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8923df636c9</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[object-oriented-program]]></category>
            <category><![CDATA[singleton-pattern]]></category>
            <category><![CDATA[software-design-patterns]]></category>
            <dc:creator><![CDATA[Debarshi Raj Basumatary]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 18:36:00 GMT</pubDate>
            <atom:updated>2026-10-06T18:36:00.275Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5jh74E5RsAi_Wv39" /><figcaption>Photo by <a href="https://unsplash.com/@jaysung?utm_source=medium&amp;utm_medium=referral">Jehyun Sung</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Singleton</strong> is a creational design pattern that lets you ensure that a class has only one instance, while providing a global access point to this instance.</p><h3>The Problem it solves:</h3><ul><li><strong>Resource management: </strong>Opening a new database connection pool or reading a configuration file from the disk every time an object requests it wastes memory and CPU. A Singleton ensures this heavy lifting is done only once.</li><li><strong>State Consistency:</strong> If multiple logging objects write to the same file or cache location simultaneously, the file becomes corrupted or out of order. A Singleton logger coordinates all write operations through a single, controlled bottleneck.</li></ul><p>Implementing a Singleton requires making the constructor private, preventing external objects from instantiating it with the <em>new</em> operator. Instead, a static creation method acts as the access point, serving the cached single instance.</p><pre>public class ConfigurationManager {<br>    // 1. A private static variable to hold the single instance<br>    private static ConfigurationManager instance;<br><br>    // 2. A private constructor prevents other classes from using &#39;new&#39;<br>    private ConfigurationManager() {<br>        // Code here runs only once (e.g., loading config files)<br>        System.out.println(&quot;Configuration loaded from disk.&quot;);<br>    }<br><br>    // 3. A public static method to provide global access to the instance<br>    public static ConfigurationManager getInstance() {<br>        if (instance == null) {<br>            // Creates the instance only if it does not exist yet (Lazy Initialization)<br>            instance = new ConfigurationManager();<br>        }<br>        return instance;<br>    }<br><br>    public String getConfigValue(String key) {<br>        return &quot;Value for &quot; + key;<br>    }<br>}</pre><p>Then Client Code calls it</p><pre>// No &#39;new&#39; keyword. Both variables hold the exact same object in memory.<br>ConfigurationManager config1 = ConfigurationManager.getInstance();<br>ConfigurationManager config2 = ConfigurationManager.getInstance();</pre><p><strong>Pros:</strong></p><ol><li>The Object is created only once, saving system memory.</li><li>You absolute gurantee that no other part of the code can access and modify or duplicate the instantiated object.</li><li>As seen from above code, Object instantiated only when it is needed and saves resources if the application ends up never using it.</li></ol><p><strong>Cons:</strong></p><ol><li>It violates <strong>Single Responsibility Principle, </strong>as it does two task — First, the class executes its own business logic and Second, it acts its own lifecycle manager ensuring no other part of the code can modify it.</li><li>It masks bad decision by hiding the tight coupling between the components. Since constructor is hidden, we don’t know the dependencies used.</li><li>Special care needs to be taken in multithreaded environment, otherwise it breaks and creates multiple singleton objects.</li><li>It is difficult or impossible to unit test singleton objects in some programming languages.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8923df636c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Row-oriented vs Column-oriented Databases: Explained Quick]]></title>
            <link>https://medium.com/devs-community/row-oriented-vs-column-oriented-databases-explained-quick-36912e285c4a?source=rss-1fac9181bc9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/36912e285c4a</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[database-design]]></category>
            <dc:creator><![CDATA[Debarshi Raj Basumatary]]></dc:creator>
            <pubDate>Mon, 25 Aug 2025 18:58:15 GMT</pubDate>
            <atom:updated>2025-09-28T01:35:02.672Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*ISHAkLzZxWcibdoHQ5usJg.png" /></figure><p>In this article, I will explain the differences between row-oriented databases (row stores) and column-oriented databases (column stores), as well as their pros and cons. These are the two data storage architectures used by databases, both offering distinct advantages and disadvantages.</p><h3>Row Oriented Database:</h3><p>FYI, database records live on disk, which is basically a giant collection of bits or bytes. And you can think of a bunch of these bytes together as a “ block.”</p><p>In Row Oriented Database, all the values of a single record across its different columns(attributes) are stored contiguously. This means that after the last attribute of one record, the next record is then inserted.</p><p>Below are two records with attributes (ID, First_Name, Last_Name, Department). As you can see, each record(i.e each row value) is stored contiguously. When the first record ends the next record starts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*DRbg5vKAK9dXqOexe_l2zA.png" /></figure><p><strong>Now let’s see what happens when DBs does a read operation:</strong></p><p><strong><em>Note:</em></strong><em> For now we will ignore database indexing and other database optimizations.</em></p><p>When a Database want to look for a record, it will first fetch a block of bytes. Then scan across the block for the record. If not found it will fetch the next block.</p><blockquote>Select first_name from Emp where department = ‘finance’</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*wbZ7mANvJqiqng3B-OJ-Nw.png" /></figure><p>Here, our condition is department = ‘finance’. So, the DB will fetch Block 1 and look for the condition; if found will return the rows else move to next block and repeat. Values are not sorted hence the db has to do a full scan of the storage. This is a huge resource hog. <br>Similar will be the case for “Select sum(age)” or any other aggregate query.</p><p>Row Oriented Databases shines in OLTP operations — transactional ops; for insert operation, it just have to insert new row bytes after the last record byte.</p><h3>Column Oriented Database:</h3><p>In Column Oriented Database, all the values of a single column(attribute) across the different records are stored contiguously. This means that all the “IDs” would be stored together, all the “Names” together, and all the “Departments” together, but in separate blocks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*dUWNYz-7v_8DHSkrclfw8g.png" /></figure><p><strong>Now let’s see what happens when DBs does a read operation:</strong></p><blockquote>Select first_name from Emp where department = ‘finance’</blockquote><p>This time DB will directly go to Block 3, it will keep track which block stores what column, look ‘finance’. If found get the index, in our case 4, go to block 2 and get the first_name at index 4,boom it’s Bilbo. <br>As you can see it was blazingly fast, we did one block scan compare to full storage scan of row-oriented DB.</p><p>For aggregate query like “Select sum(salary)” it is even faster. Just sum the block that stores salary column that’s it.</p><p>Great for OLAP operations — analysis ops.</p><p>But, but,…..for insert operation it is bad. We have to go through all the blocks and add the value for each columns of the new record.</p><p>Both systems are great, both have their pros and cons and choosing one depends on our work requirment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36912e285c4a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devs-community/row-oriented-vs-column-oriented-databases-explained-quick-36912e285c4a">Row-oriented vs Column-oriented Databases: Explained Quick</a> was originally published in <a href="https://medium.com/devs-community">Devs Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[System Design: Youtube/Netflix using AWS]]></title>
            <link>https://blog.stackademic.com/system-design-youtube-netflix-using-aws-ee7b1ac942ca?source=rss-1fac9181bc9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee7b1ac942ca</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <dc:creator><![CDATA[Debarshi Raj Basumatary]]></dc:creator>
            <pubDate>Mon, 07 Apr 2025 05:52:09 GMT</pubDate>
            <atom:updated>2025-04-07T05:52:09.939Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mHVzM5-wQ_fDe_h2" /><figcaption>Photo by <a href="https://unsplash.com/@popcornmatch?utm_source=medium&amp;utm_medium=referral">Marques Kaspbrak</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article explores the design of a video streaming service, similar to YouTube, Netflix, or Prime Video, leveraging the AWS cloud. The concepts presented are also applicable to designing equivalent systems on GCP or Azure. It will follow a distributed microservice architecture.</p><blockquote><em>Note: </em>Readers should have a basic understanding of AWS and its services to follow along.</blockquote><h3><strong>Requirements:</strong></h3><p><strong>Functional: “</strong>What the system should do?”<br>1. Users should be able to upload videos.<br>2. Users should be able to watch/stream videos.</p><p><strong>Non-Functional</strong>: “How well the system should perform?”<br>1. Because this is a distributed system for video uploads, a choice between availability and consistency is necessary. Since users can accept some data staleness, we will prioritize availability.<br>2. Support for uploading and streaming large-sized video files.<br>3. Low Latency.<br>4. Scalable enough to handle 1M upload/day and 100M downloads/day.<br>5. Max Video size of 256GB</p><h3>Core Entities:</h3><ol><li>Videos</li><li>User</li><li>Video MetaData</li></ol><h3>API:</h3><ol><li>Upload a video: POST /videos</li><li>Stream a video: GET /videos/:videoID</li></ol><h3>High Level Design:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TwOkkPKbuZrN2ZYyyyG0vA.gif" /></figure><h4>Components:</h4><ul><li><strong>API Gateway</strong>: To route the requests, rate limit, authentication/ authorization etc.</li><li><strong>Application Load Balancer:</strong> Route the traffic according to requests.</li><li><strong>EC2: </strong>Compute resources that will process the requests.</li><li><strong>Auto Scaling Group:</strong> Service that horizontally scales the EC2 instances as load increases.</li><li><strong>S3: </strong>Object Store where the videos will be stored.</li><li><strong>RDS: </strong>To store the metadata of the videos.</li><li><strong>Chunking:</strong> A lambda function that will breaks the video file into smaller chuncks. (More on it later.)</li><li><strong>Video Transcoding:</strong> A group of lambda functions that will convert the video’s quality.(More on it later.)</li></ul><blockquote><em>Note: </em>We use AWS API Gateway to handle the Client’s authentication and authorization part.</blockquote><h4>Flow:</h4><ol><li>API Gateway and standard load balancers impose upload limits of approximately 10MB, insufficient for our multi-gigabyte video files. While S3 offers a simple PUT API for files up to 5GB and a multipart upload API for larger files (which divides files into concurrent chunks and reassembles them in S3), we must avoid granting direct S3 access to external users. Therefore, our video service will generate a pre-signed URL from S3, a temporary access link, and provide it to the client. The client will then use this URL to directly upload the video file to S3.</li><li>Upon successful file upload, S3 will publish notifications. These notifications will trigger two actions: first, an RDS update to store the video file’s metadata (name, size, upload status, format, etc.); and second, a chunking function that segments the video file, stores the segments in S3, and updates the metadata. Chunking is necessary because HTTP streaming cannot transmit the entire file at once. The client will reassemble these segments for playback.</li><li>To accommodate varying network bandwidths during video streaming, we implement adaptive bitrate streaming. This involves transcoding the video segments from the previous step into multiple quality formats. The resulting segments are stored in S3, and the metadata database is updated accordingly. Subsequently, when a user requests a different quality, the corresponding segments can be delivered rapidly. It is acomplished using video transcoding Lambda functions.</li><li>The metadata database also includes a manifest file, which details the URLs of the video chunks for each available quality format.</li><li>While S3 offers excellent scalability and availability as a fully managed AWS service, it may not be sufficient for handling 100 million read requests, potentially introducing latency, especially for geographically distant clients. To mitigate this, we employ a Content Delivery Network (CDN) that caches the video chunks and a CDN-specific manifest file. This manifest contains URLs pointing to the CDN, significantly reducing latency. This approach satisfies our high availability non-functional requirement, albeit with a trade-off of temporary inconsistency due to CDN synchronization delays.</li></ol><p>This article outlined a robust design for a video streaming service on AWS, similar to YouTube or Netflix. We covered critical aspects like scalable uploads using S3 presigned URLs, durable storage in S3, metadata management with RDS, and crucially, video processing involving chunking and transcoding into multiple quality levels for Adaptive Bitrate Streaming (ABS).</p><p>The final, crucial step of orchestrating the actual streaming experience described — delivering the right chunks at the right quality based on network conditions — relies on standardized <strong>streaming protocols</strong>. Protocols like <strong>HLS (HTTP Live Streaming)</strong> and <strong>DASH (Dynamic Adaptive Streaming over HTTP)</strong> define the standard formats for delivering this processed, chunked video content over regular HTTP/HTTPS.</p><p>These protocols specify the structure of the <strong>manifest file</strong> (the .m3u8 for HLS or .mpd for DASH, which your metadata database references). This manifest, generated during the transcoding/packaging step and delivered efficiently via the CDN, acts as a playlist. It lists the available video quality streams and contains the URLs (pointing to the CDN) for the corresponding video chunks, which originate from your S3 bucket.</p><p>The video player on the user’s device understands these HLS or DASH formats. It reads the manifest, requests the appropriate video chunks from the CDN, buffers them, and handles the “reassembly” for smooth playback. Critically, the player uses the information in the manifest and monitors the network connection to dynamically switch between different quality streams (requesting chunks from different quality renditions), achieving the seamless Adaptive Bitrate Streaming experience viewers expect. Adherence to these widely adopted HLS/DASH standards ensures compatibility across diverse devices and browsers, effectively bridging the gap between the prepared content on the backend and the dynamic playback experience on the client side.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://differ.blog/inplainenglish"><strong>Differ</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://cofeed.app/"><strong>Check out CoFeed, the smart way to stay up-to-date with the latest in tech</strong></a> <strong>🧪</strong></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee7b1ac942ca" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/system-design-youtube-netflix-using-aws-ee7b1ac942ca">System Design: Youtube/Netflix using AWS</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[All LeetCode’s Buy and Sell Stock Problems: Java]]></title>
            <link>https://blog.stackademic.com/leetcode-buy-and-sell-stock-problems-java-087e0adf8abf?source=rss-1fac9181bc9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/087e0adf8abf</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[leetcode]]></category>
            <category><![CDATA[interview-questions]]></category>
            <category><![CDATA[data-structure-algorithm]]></category>
            <dc:creator><![CDATA[Debarshi Raj Basumatary]]></dc:creator>
            <pubDate>Thu, 30 Jan 2025 18:05:56 GMT</pubDate>
            <atom:updated>2026-08-08T13:31:15.285Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dBYcGNCiu6E0XxFM" /><figcaption>Photo by <a href="https://unsplash.com/@polarmermaid?utm_source=medium&amp;utm_medium=referral">Anne Nygård</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Buy and Sell Stock problems are among the most commonly asked and important topics in Data Structures and Algorithms (DSA) for programming interviews. Below, I have listed all such problems from LeetCode along with their solutions.</p><h4><a href="https://leetcode.com/problems/best-time-to-buy-and-sell-stock/description/">121. Best Time to Buy and Sell Stock</a></h4><p>You are given an array prices where prices[i] is the price of a given stock on the ith day.</p><p>You want to maximize your profit by choosing a <strong>single day</strong> to buy one stock and choosing a <strong>different day in the future</strong> to sell that stock.</p><p>Return <em>the maximum profit you can achieve from this transaction</em>. If you cannot achieve any profit, return 0.</p><blockquote>Input: [7,1,5,3,6,4]<br>Output: 5</blockquote><p><strong>Solution: </strong><br>This problem can be solved using a greedy approach. We’ll track two variables: one for the largest profit and another for the index of the day when we buy the stock.</p><pre>public int maxProfit(int[] prices) {<br>    int sum = 0;   // Stores the maximum profit<br>    int buy = 0;   // Index to track the minimum buying price<br><br>    // Iterate through the stock prices starting from the second day<br>    for (int i = 1; i &lt; prices.length; i++) {<br>        // If selling today gives a profit, update max profit<br>        if (prices[i] - prices[buy] &gt; 0) {<br>            sum = Math.max(sum, prices[i] - prices[buy]);<br>        } else {<br>            // If a lower price is found, update the buying index<br>            buy = i;<br>        }<br>    }<br>    return sum;  // Return the maximum profit<br>}</pre><p>We start with the buy index and profit sum at 0. As we iterate through elements starting from the second one, we check if the Sell — Buy difference is positive. If it is, we have a potential maximum profit, which we determine using Math.max. However, if Sell — Buy is negative, it means the stock price on day i is greater than on day i+1(it also mean, we found day when the stock is lowest. After finding the lowest price stock, buy index doesn’t move), so we buy a new stock at this lower price. Finally, we return the sum variable containing our maximum profit.</p><p><em>Time Complexity: O(n)<br>Space Complexity: O(1)</em></p><h4><a href="https://leetcode.com/problems/best-time-to-buy-and-sell-stock-ii/">122. Best Time to Buy and Sell Stock II</a></h4><p>You are given an integer array prices where prices[i] is the price of a given stock on the ith day.</p><p>On each day, you may decide to buy and/or sell the stock. You can only hold <strong>at most one</strong> share of the stock at any time. However, you can buy it then immediately sell it on the <strong>same day</strong>.</p><p>Find and return <em>the </em><strong><em>maximum</em></strong><em> profit you can achieve</em>.</p><p><strong>Solution: </strong><br>Here you can buy and sell the stock multiple times to maximize your profit, but you <strong>cannot hold multiple stocks at once</strong> (you must sell before buying again).<br>To solve this problem, we will again use a Greedy Approach. The best approach is to buy at local minimum and sell at local maximum. We will iterate through the price array, and when we find a positive difference (i.e., previous stock price is less than current stock price), we have found our local profit, which we will increment with our max profit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*gsIGZk--gDNXO-nvCHqArw.png" /></figure><pre>public int maxProfit(int[] prices) {<br>        int profit = 0;<br>        for (int i = 1; i &lt; prices.length; i++) {<br>            if (prices[i] &gt; prices[i - 1]) {         // Positive Difference<br>                profit += prices[i] - prices[i - 1]; // Increment max profit<br>            }<br>        }<br>        <br>        return profit;<br>    }</pre><p><em>Time Complexity: O(n)<br>Space Complexity: O(1)</em></p><h4><a href="https://leetcode.com/problems/best-time-to-buy-and-sell-stock-iii/">123. Best Time to Buy and Sell Stock III</a></h4><p>You are given an array prices where prices[i] is the price of a given stock on the ith day.</p><p>Find the maximum profit you can achieve. You may complete <strong>at most two transactions</strong>.</p><p><strong>Note:</strong> You may not engage in multiple transactions simultaneously (i.e., you must sell the stock before you buy again).</p><p><strong>Solution:</strong><br>It is similar to previous problem with added complexity of only two transactions, so we can’t just use any local minimum and local maximum.</p><p><em>To solve this problem we will perform two transactions and use Dynamic programming to store the maximum profit after each transaction.</em><br>Define 4 variables</p><ul><li>buy1: The maximum profit after the first buy.</li><li>sell1: The maximum profit after the first sell.</li><li>buy2: The maximum profit after the second buy.</li><li>sell2: The maximum profit after the second sell.</li></ul><p>Initially buy are set to -prices[0], i.e maximum profit after buying the stock on day 1. And sell are set to 0, i.e we haven’t sold anything yet. <br>Then iterate through the prices array and set the four variables as</p><ul><li>buy1: The maximum of the current buy1 or buying the stock at the current price.</li><li>sell1: The maximum of the current sell1 or selling the stock at the current price (after the first buy).</li><li>buy2: The maximum of the current buy2 or buying the stock at the current price (after the first sell). (We need to find net profit, so we are subtracting new price from previous profit. If we put -prices[i] it will start the transaction from start)</li><li>sell2: The maximum of the current sell2 or selling the stock at the current price (after the second buy).</li></ul><p>In the end sell2will have the maximum profit after two transactions.</p><pre>public int maxProfit(int[] prices) {<br>    // Edge case: If prices array is empty or null, return 0 profit<br>    if (prices.length == 0 || prices == null) return 0;<br><br>    // Variables to track two transactions<br>    int buy1 = -prices[0];  // Best price to buy in 1st transaction<br>    int sell1 = 0;          // Profit after 1st sell<br>    int buy2 = -prices[0];  // Best price to buy in 2nd transaction<br>    int sell2 = 0;          // Profit after 2nd sell<br><br>    // Iterate through the stock prices<br>    for (int i = 1; i &lt; prices.length; i++) {<br>        // Try to buy at the lowest price for the first transaction<br>        buy1 = Math.max(buy1, -prices[i]);<br><br>        // Sell the first stock at max profit<br>        sell1 = Math.max(sell1, buy1 + prices[i]);<br><br>        // Buy again after selling the first stock, maximizing profit<br>        buy2 = Math.max(buy2, sell1 - prices[i]);<br><br>        // Sell the second stock at max profit<br>        sell2 = Math.max(sell2, buy2 + prices[i]);<br>    }<br><br>    // Return the maximum profit after completing at most two transactions<br>    return sell2;<br>}</pre><p><em>Time Complexity: O(n)<br>Space Complexity: O(1)</em></p><h4><a href="https://leetcode.com/problems/best-time-to-buy-and-sell-stock-iv/">188. Best Time to Buy and Sell Stock IV</a></h4><p>You are given an integer array prices where prices[i] is the price of a given stock on the ith day, and an integer k.</p><p>Find the maximum profit you can achieve. You may complete at most k transactions: i.e. you may buy at most k times and sell at most k times.</p><p><strong>Note:</strong> You may not engage in multiple transactions simultaneously (i.e., you must sell the stock before you buy again).</p><p><strong>Solution:</strong> <br>It is similar to previous 123th problem, with added complexity of at most kth transactions. <br>In the previous solution we used two buy and two sell variables to keep track of the max profits per transactions. Here, we have to come up with a general solution.</p><p>We will use a 2D table dp ,wheredp[t][j] represents the maximum profit on the j-th day with t transactions completed.<br>The outer loop signifies transactions. maxDiff is used to track the <strong>maximum effective cost</strong> for the current transaction. <br>Initially, it is set to-prices[0] because if we buy the stock on day 0, our profit is -prices[0] (since we are spending money to buy the stock).<br>dp[t][i] represents the <strong>maximum profit</strong> on day j with t transactions.</p><ol><li><strong>dp[t][j - 1]</strong>: The profit from the previous day with the same number of transactions (i.e., do nothing on day j).</li><li><strong>prices[j] + maxDiff</strong>: The profit if we sell the stock on day j:</li></ol><p>Then we update the maxDiff:<br>It is calculated as the maximum of:</p><ol><li><strong>maxDiff</strong>: The current value of maxDiff.</li><li><strong>dp[t - 1][j] - prices[j]</strong>: The profit from the previous transaction (t-1) minus the current stock price.</li></ol><ul><li>This represents the <strong>effective cost</strong> if we buy the stock on day j after completing t-1 transactions.</li></ul><p>In the end we return dp[k][n - 1], which represents the <strong>maximum profit</strong> after k transactions on the last day (n-1).</p><pre>public int maxProfit(int k, int[] prices) {<br>    // DP table where dp[t][j] represents max profit with at most &#39;t&#39; transactions on day &#39;j&#39;<br>    int[][] dp = new int[k + 1][prices.length];<br><br>    // Iterate over transaction count (1 to k)<br>    for (int t = 1; t &lt;= k; t++) {<br>        int maxDiff = -prices[0]; // Tracks the max value of (dp[t-1][j] - prices[j])<br><br>        // Iterate over days<br>        for (int j = 1; j &lt; prices.length; j++) {<br>            // Either we don&#39;t sell today (dp[t][j-1]) or we sell today (prices[j] + maxDiff)<br>            dp[t][j] = Math.max(dp[t][j - 1], prices[j] + maxDiff);<br><br>            // Update maxDiff to track the best buying opportunity from previous transactions<br>            maxDiff = Math.max(maxDiff, dp[t - 1][j] - prices[j]);<br>        }<br>    }<br><br>    // Return the maximum profit possible with at most &#39;k&#39; transactions<br>    return dp[k][prices.length - 1];<br>}</pre><p><em>Time Complexity: O(n*k)<br>Space Complexity: O(n*k)</em></p><h4><a href="https://leetcode.com/problems/best-time-to-buy-and-sell-stock-with-transaction-fee/">714. Best Time to Buy and Sell Stock with Transaction Fee</a></h4><p>You are given an array prices where prices[i] is the price of a given stock on the ith day, and an integer fee representing a transaction fee.</p><p>Find the maximum profit you can achieve. You may complete as many transactions as you like, but you need to pay the transaction fee for each transaction.</p><p><strong>Note:</strong></p><ul><li>You may not engage in multiple transactions simultaneously (i.e., you must sell the stock before you buy again).</li><li>The transaction fee is only charged once for each stock purchase and sale.</li></ul><blockquote>Input: prices = [1,3,2,8,4,9], fee = 2 <br>Output: 8 <br>Explanation: The maximum profit can be achieved by: <br>— Buying at prices[0] = 1 <br>— Selling at prices[3] = 8 <br>— Buying at prices[4] = 4<br> — Selling at prices[5] = 9 <br>The total profit is ((8–1) — 2) + ((9–4) — 2) = 8.</blockquote><p><strong>Solution:<br></strong>We will use dynamic programming to solve this problem. Where buy[i] represents the maximum profit achievable after purchasing stock on day i, and sell[i] represents the maximum profit after selling on day i. This approach ensures optimal transactions while considering the transaction fee.</p><pre>public int maxProfit(int[] prices, int fee) {<br>    // Arrays to store the maximum profit at each day when buying or selling<br>    int[] buy = new int[prices.length + 1];  // Maximum profit when holding a stock<br>    int[] sell = new int[prices.length + 1]; // Maximum profit when not holding a stock<br><br>    // Initialize the first day by buying the stock<br>    buy[0] = -prices[0]; <br><br>    // Iterate over each day to determine the maximum profit<br>    for(int i = 1; i &lt; prices.length; i++) {<br>        // Decide whether to keep the previous buy state or buy the stock today<br>        buy[i] = Math.max(buy[i - 1], sell[i - 1] - prices[i]);<br><br>        // Decide whether to keep the previous sell state or sell the stock today (accounting for fee)<br>        sell[i] = Math.max(sell[i - 1], buy[i - 1] + prices[i] - fee);<br>    }<br><br>    // The maximum profit will be in the sell state at the last day<br>    return sell[prices.length - 1];<br>}</pre><p><em>Time Complexity: O(n)<br>Space Complexity: O(n)</em></p><h4><a href="https://leetcode.com/problems/best-time-to-buy-and-sell-stock-with-cooldown/">309. Best Time to Buy and Sell Stock with Cooldown</a></h4><p>You are given an array prices where prices[i] is the price of a given stock on the ith day.</p><p>Find the maximum profit you can achieve. You may complete as many transactions as you like (i.e., buy one and sell one share of the stock multiple times) with the following restrictions:</p><ul><li>After you sell your stock, you cannot buy stock on the next day (i.e., cooldown one day).</li></ul><p><strong>Note:</strong> You may not engage in multiple transactions simultaneously (i.e., you must sell the stock before you buy again).</p><blockquote>Input: prices = [1,2,3,0,2] <br>Output: 3 <br>Explanation: transactions = [buy, sell, cooldown, buy, sell]</blockquote><h4>Solution:</h4><p>To solve this problem, we’ll use <strong>dynamic programming</strong> with <strong>state transitions</strong>. For each day, we’ll track the maximum profit based on three possible states:</p><ol><li><strong>Hold:</strong> Having a stock (either purchased today or kept from before)</li><li><strong>Sold:</strong> Having sold a stock today</li><li><strong>Cooldown:</strong> Being in the cooldown period (no buying allowed)</li></ol><p><strong>State Transitions:</strong></p><ol><li><strong>Hold: </strong>You can either:</li></ol><ul><li>Keep the stock from the previous day (hold[i-1])</li><li>Purchase a stock today after being in the <strong>cooldown</strong> state (cooldown[i-1] - prices[i])</li></ul><p><strong>2. Sold:</strong></p><ul><li>Sell your stock today after being in the <strong>hold</strong> state (hold[i-1] + prices[i])</li></ul><p><strong>3. Cooldown: </strong>Enter a cooldown period by either:</p><ul><li>Coming from the <strong>sold</strong> state (sold[i-1])</li><li>Remaining in the <strong>cooldown</strong> state (cooldown[i-1])</li></ul><pre>public int maxProfit(int[] prices) {<br>        if (prices == null || prices.length == 0) {<br>            return 0;<br>        }<br><br>        int n = prices.length;<br><br>        // Initialize states<br>        int[] hold = new int[n];    // Maximum profit if holding a stock on day i<br>        int[] sold = new int[n];    // Maximum profit if sold a stock on day i<br>        int[] cooldown = new int[n]; // Maximum profit if in cooldown on day i<br><br>        // Base cases<br>        hold[0] = -prices[0]; // Buy the stock on day 0<br>        sold[0] = 0;          // Cannot sell on day 0<br>        cooldown[0] = 0;      // Cannot be in cooldown on day 0<br><br>        // Fill the states<br>        for (int i = 1; i &lt; n; i++) {<br>            hold[i] = Math.max(hold[i - 1], cooldown[i - 1] - prices[i]); // Buy or carry over<br>            sold[i] = hold[i - 1] + prices[i]; // Sell the stock<br>            cooldown[i] = Math.max(cooldown[i - 1], sold[i - 1]); // Cooldown or carry over<br>        }<br><br>        // The maximum profit is the maximum of sold[n-1] and cooldown[n-1]<br>        return Math.max(sold[n - 1], cooldown[n - 1]);<br>    }</pre><p><em>Time Complexity: O(n) <br>Space Complexity: O(n)</em></p><p><strong>Space Optimzed Solution:<br></strong>Since we only need values from the previous day to calculate the current day’s values, we can optimize our solution. By replacing the arrays (hold, sold, cooldown) with simple variables, we reduce the space complexity from <em>O(n)</em> to<em> O(1)</em>.</p><pre>public int maxProfit(int[] prices) {<br>        if (prices == null || prices.length == 0) {<br>            return 0;<br>        }<br><br>        // Initialize states<br>        int hold = -prices[0]; // Buy the stock on day 0<br>        int sold = 0;          // Cannot sell on day 0<br>        int cooldown = 0;      // Cannot be in cooldown on day 0<br><br>        // Iterate through the prices<br>        for (int i = 1; i &lt; prices.length; i++) {<br>            int prevHold = hold;<br>            int prevSold = sold;<br>            int prevCooldown = cooldown;<br><br>            // Update states<br>            hold = Math.max(prevHold, prevCooldown - prices[i]); // Buy or carry over<br>            sold = prevHold + prices[i]; // Sell the stock<br>            cooldown = Math.max(prevCooldown, prevSold); // Cooldown or carry over<br>        }<br><br>        // The maximum profit is the maximum of sold and cooldown<br>        return Math.max(sold, cooldown);<br>}</pre><h3>Conclusion:</h3><p>LeetCode’s Buy and Sell Stock problems cover a wide range of complexities, from simple single transactions to multiple trades with constraints like transaction fees and cooldown periods. By utilizing greedy algorithms and dynamic programming, we can efficiently solve these problems while optimizing time and space complexity. <br>Keep practicing, and happy coding! 🚀</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a></li><li><a href="https://cofeed.app/"><strong>Check out CoFeed, the smart way to stay up-to-date with the latest in tech</strong></a> <strong>🧪</strong></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=087e0adf8abf" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/leetcode-buy-and-sell-stock-problems-java-087e0adf8abf">All LeetCode’s Buy and Sell Stock Problems: Java</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Java Concepts: Generics]]></title>
            <link>https://blog.stackademic.com/understanding-java-concepts-generics-9ec94d2f56f1?source=rss-1fac9181bc9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ec94d2f56f1</guid>
            <category><![CDATA[object-oriented]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[generics]]></category>
            <category><![CDATA[generic-programming]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Debarshi Raj Basumatary]]></dc:creator>
            <pubDate>Wed, 29 Jan 2025 09:57:48 GMT</pubDate>
            <atom:updated>2025-02-03T03:07:10.927Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zpg0jnY37KJ3PK9y" /><figcaption>Photo by <a href="https://unsplash.com/@thoughtcatalog?utm_source=medium&amp;utm_medium=referral">Thought Catalog</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Let’s examine a code example to understand Generics:</p><pre>class Pair{<br> private Integer key;<br> private Integer value;<br> public Pair(Integer key, Integer value){<br>  this.key = key;<br>  this.value = value;<br> }<br> public Integer getKey(){<br>  return key;<br> }<br> public Integer getValue(){<br>  return value;<br> }<br>}</pre><p>This simple class stores a key-value pair that accepts only Integer values. However, if we need the key to be a String type instead, we’ll get an <strong>error: incompatible types: String cannot be converted to Integer.</strong> This is where generics comes into play.</p><p>Generics are like adding data types to classes, interfaces, and methods, similar to how you specify types for function parameters. This allows you to create blueprints that can work with different types without knowing those types beforehand.</p><p>By using a generic type parameter (like T or E), you&#39;re essentially saying, &quot;This class or method can work with any type, but I won&#39;t specify it explicitly right now.&quot; The concrete type is determined when you actually use the generic class or method.</p><p>Here’s how we can rewrite the above class using generics:</p><pre>class Pair&lt;K, V&gt;{<br> private K key;<br> private V value;<br> public Pair(K key, V value){<br>  this.key = key;<br>  this.value = value;<br> }<br> public K getKey(){<br>  return key;<br> }<br> public V getValue(){<br>  return value;<br> }<br>}<br>class Main {<br>    public static void main(String[] args) {<br>    <br>        Pair&lt;String, Integer&gt; m1 = new Pair&lt;&gt;(&quot;Money&quot;,3);<br>        Pair&lt;Integer, Integer&gt; m2 = new Pair&lt;&gt;(5,3);<br>        System.out.println(m1.getKey());<br>        System.out.println(m2.getKey());<br>    }<br>}</pre><p><strong>Output:<br></strong>Money <br>5</p><p>We use angle brackets (&lt;&gt;) to specify types in the class definition. We can use any letter (K, V, T) as type parameters, separating multiple types with commas. When creating an object, we again use angle brackets but specify concrete types like Integer or Float.</p><blockquote><strong>Note: Generics don’t work with primitive datatypes.</strong></blockquote><p>Check this below code of Java List from Collection Framework.</p><pre>List list = new LinkedList();<br>list.add(new Integer(1)); <br>Integer i = list.iterator().next();</pre><p>Java Compiler will complain about the last line.</p><p>We can this compile time error with explicit casting.</p><pre>Integer i = (Integer) list.iterator.next();</pre><p>But there’s no guarantee that the list will contain Integer objects. Additionally, we have to explicitly typecast it into Integer, which makes the code cluttered.</p><p>At initialization time, we can use Generics to restrict the list to accept only Integer values, making the code type-safe.</p><h3>Generic Method:</h3><p>Similar to above Generic Class, we can also create generic methods.</p><p>Type parameters must be declared before the method’s return type. This syntax works for static methods, non-static methods, and constructors.</p><pre>public static &lt;K, V&gt; boolean compare(Pair&lt;K, V&gt; p1, Pair&lt;K, V&gt; p2) {<br>        return p1.getKey().equals(p2.getKey()) &amp;&amp;<br>               p1.getValue().equals(p2.getValue());<br>    }</pre><h3><strong>Bounded Generics:</strong></h3><p>Sometimes you’ll need to limit which types can be used as type arguments in a parameterized type. For instance, if you have a method that works with numbers, you might want it to accept only instances of Number or its subclasses. This is where <em>bounded type parameters</em> come in.</p><pre>public class Box&lt;T&gt; {<br><br>    private T t;<br><br>    public void set(T t) {<br>        this.t = t;<br>    }<br><br>    public T get() {<br>        return t;<br>    }<br><br>    public &lt;U extends Number&gt; void inspect(U u){<br>        System.out.println(&quot;T: &quot; + t.getClass().getName());<br>        System.out.println(&quot;U: &quot; + u.getClass().getName());<br>    }<br><br>    public static void main(String[] args) {<br>        Box&lt;Integer&gt; integerBox = new Box&lt;Integer&gt;();<br>        integerBox.set(new Integer(10));<br>        integerBox.inspect(&quot;some text&quot;); //error: this is still String!<br>    }<br>}</pre><p>In this code, the inspect method accepts only instances of Number or its subclasses. Use the ‘extends’ keyword within angle brackets to restrict the type to your desired type.</p><blockquote><strong>Note: The use of “extends” in generics differs from how “extends” and “implements” work in class and interface inheritance.</strong></blockquote><p>A type can have multiple bounds</p><pre>&lt;T extends Number &amp; Comparable&gt;</pre><h3>Wildcards:</h3><p>Wildcards are represented by the ? symbol and provide a way to increase the flexibility of generic code by relaxing type parameter constraints. They allow you to specify unknown types or introduce bounds for type arguments.</p><p>These wildcards restrict the unknown type to be a subtype of the specified Type.</p><pre>public static void printList(List&lt;? extends Number&gt; list) {<br>    for (Number n: list) {<br>        System.out.println(n);<br>    }<br>}</pre><p><strong>Unbounded Wildcards (</strong><strong>?)<br></strong>These wildcards represent any unknown type.</p><pre>public static void printSize(List&lt;?&gt; list) {<br>    System.out.println(list.size());<br>}</pre><h3>Inheritance:</h3><p>While Integer is a subclass of the Number class, allowing us to pass or refer to an Integer value using a Number variable, this inheritance relationship doesn’t apply directly to generics. Therefore, Box&lt;Integer&gt; is not a subclass of Box&lt;Number&gt;</p><pre>Box&lt;Integer&gt; intBox = new Box&lt;&gt;(10); // Ok<br>Box&lt;Number&gt; numberBox = intBox; // Compile-time error!</pre><p>We can subtype a generic class or interface by extending or implementing it. Using Collection framework we can use ArrayList&lt;E&gt; implements List&lt;E&gt;, and List&lt;E&gt; extends Collection&lt;E&gt;</p><p>For example PayloadList, that associates an optional value of generic type P with each element. Its declaration might look like:</p><pre>interface PayloadList&lt;E,P&gt; extends List&lt;E&gt; {<br>  void setPayload(int index, P val);<br>  ...<br>}</pre><h3>Conclusion:</h3><p>Java Generics is a powerful tool that makes Java code more abstract and simplifies programmers’ work. It enforces type safety at compile time rather than runtime, which saves considerable development time. For more in-depth information, check the references below.</p><h3>References:</h3><ol><li><a href="https://docs.oracle.com/javase/tutorial/java/generics/why.html">https://docs.oracle.com/javase/tutorial/java/generics/why.html</a></li><li><a href="https://www.baeldung.com/java-generics">https://www.baeldung.com/java-generics</a></li></ol><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a></li><li><a href="https://cofeed.app/"><strong>Check out CoFeed, the smart way to stay up-to-date with the latest in tech</strong></a> <strong>🧪</strong></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ec94d2f56f1" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/understanding-java-concepts-generics-9ec94d2f56f1">Understanding Java Concepts: Generics</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[XGBoost for Night Sky Classification]]></title>
            <link>https://medium.com/@basumatary18/xgboost-for-night-sky-classification-7014f10a722?source=rss-1fac9181bc9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7014f10a722</guid>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[xgboost]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <dc:creator><![CDATA[Debarshi Raj Basumatary]]></dc:creator>
            <pubDate>Mon, 24 Jul 2023 12:44:33 GMT</pubDate>
            <atom:updated>2023-07-29T12:26:25.463Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>This article will cover the basics of Decision Trees and XGBoost, and demonstrate how to implement the latter for classifying celestial objects in the night sky as either a Galaxy, Star, or Quasar.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y957MEoSucliC8W3" /><figcaption>Photo by <a href="https://unsplash.com/@nathananderson?utm_source=medium&amp;utm_medium=referral">Nathan Anderson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Brief Intro to ML Algorithms</h3><p>A <strong>decision tree</strong> is a widely used algorithm in machine learning. It is represented by a flowchart-like tree structure where an internal node represents a feature (or attribute), the branch represents a decision rule, and each leaf node represents an outcome. Decision trees are commonly used in classification and regression tasks and can efficiently handle large datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*Giv787KdoyWScmI6pazANQ.png" /></figure><p><strong><em>However, on their own, they are not very accurate and are prone to overfitting.</em></strong></p><p>One clever way to improve predictions is by combining multiple decision trees, aggregating their outputs, and predicting the final outcome based on the average (for regression) or majority voting (for classification). This is called <strong>Random Forest</strong>. However, this approach may struggle when the dataset is imbalanced, and the model becomes computationally too heavy.</p><p>Another approach is to use <strong>Gradient Boosted Trees</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*frdPtLnqG6hPi0sPEfWU6g.png" /></figure><p>The <strong>Gradient Boosted Tree (GBT)</strong> algorithm begins with an initial prediction and iteratively adds decision trees to correct errors and improve accuracy. The process starts with an initial prediction made using a base estimator (an initial decision tree). The residual (error) is calculated and sent to the next estimator (another decision tree) as the output label. The estimator trains on the datasets, but the output labels are the errors from the previous tree. This process continues until the desired level of accuracy is achieved.</p><p><strong><em>It excels in tasks with high-dimensional data and delivers reliable results due to its ability to handle different loss functions and provide feature importance estimates.</em></strong></p><p><strong>XGBoost</strong> represents a next-generation evolution of Gradient Boosted Trees, building upon the same foundational principles while introducing several essential enhancements. Notably, it incorporates parallelization, regularization techniques, and a more efficient greedy algorithm for identifying optimal split points during tree construction. These improvements result in significantly faster training times and enhanced model generalization capabilities. Its exceptional accuracy and high-performance capabilities make it a go-to choice for many data scientists and researchers.</p><p>It was developed by Tianqi Chen, a data scientist and software engineer, during his Ph.D. at the University of Washington.</p><p><strong><em>If you are interested in getting a deeper understanding of the math and theory behind XGBoost, then you should definitely visit </em></strong><a href="https://arxiv.org/pdf/1603.02754.pdf"><strong><em>Here</em></strong></a><strong><em>.</em></strong></p><h3>The Dataset</h3><p>The “Stellar Classification Dataset — SDSS17” by <a href="https://www.kaggle.com/fedesoriano">FEDESORIANO</a> from Kaggle<a href="https://www.kaggle.com/datasets/fedesoriano/stellar-classification-dataset-sdss17"><strong>[Link]</strong></a>, is made from “Sloan Digital Sky Survey” project and contains spectroscopic observations of celestial objects in the night sky. The dataset consists of 100,000 observations, each of which is described by 17 feature columns and 1 class column that identifies it as either a star, galaxy, or quasar.</p><blockquote>Quasars, short for “quasi-stellar radio sources,” are extremely bright and distant astronomical objects found at the centres of galaxies. Quasars are not stars but rather the active cores of distant galaxies, powered by supermassive black holes.</blockquote><ol><li>obj_ID = Object Identifier, the unique value that identifies the object in the image catalog used by the CAS</li><li>alpha = Right Ascension angle (at J2000 epoch)</li><li>delta = Declination angle (at J2000 epoch)</li><li>u = Ultraviolet filter in the photometric system</li><li>g = Green filter in the photometric system</li><li>r = Red filter in the photometric system</li><li>i = Near Infrared filter in the photometric system</li><li>z = Infrared filter in the photometric system</li><li>run_ID = Run Number used to identify the specific scan</li><li>rereun_ID = Rerun Number to specify how the image was processed</li><li>cam_col = Camera column to identify the scanline within the run</li><li>field_ID = Field number to identify each field</li><li>spec_obj_ID = Unique ID used for optical spectroscopic objects (this means that 2 different observations with the same spec_obj_ID must share the output class)</li><li>class = object class (galaxy, star or quasar object)</li><li>redshift = redshift value based on the increase in wavelength</li><li>plate = plate ID, identifies each plate in SDSS</li><li>MJD = Modified Julian Date, used to indicate when a given piece of SDSS data was taken</li><li>fiber_ID = fiber ID that identifies the fiber that pointed the light at the focal plane in each observation</li></ol><h3>Implementation</h3><p><strong>Requirements:</strong><br>1. Knowledge of Python, Pandas, Numpy, Matplotlib and Seaborn.<br>2. Anaconda</p><h4>Step 1: Install XGBoost</h4><pre>conda install -c conda-forge py-xgboost</pre><h4>Step 2: Import all the required libraries</h4><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import precision_score, recall_score, accuracy_score, confusion_matrix<br>from xgboost import XGBClassifier</pre><p><em>XGBClassifier is a scikit-learn wrapper over the Xgboost library, it allows us to directly use xgboost with scikit-learn and pandas apis.</em></p><h4>Step 3: Import &amp; Validate the Dataset</h4><pre>stellar_data = pd.read_csv(&#39;star_classification.csv&#39;)<br>stellar_data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TCmBUw8P4xGiTauuW8pbUQ.png" /></figure><p><strong><em>for training the model we will not need “obj_ID”</em></strong></p><pre>stellar_data.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*ZEjDVur-c-OVcLa18RX4NA.png" /></figure><p><strong><em>There are no Null values and all the columns are either int or float, no categorical column except for output class column.</em></strong></p><pre>stellar_data[&#39;class&#39;].value_counts()<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*-TGHRNyPn8p4r3jqc10ohQ.png" /></figure><p><strong><em>There are only three classes Galaxy, Star and Quasar.</em></strong></p><h4>Step 4: Visualise the Data</h4><p>Below code will loop over all the columns, except for the ID, and plot a histogram. I have only shown 2, for the rest try yourself.</p><pre>columns = stellar_data.columns<br>sns.set_theme(style=&quot;darkgrid&quot;)<br>for col in columns:<br>    if col != &#39;obj_ID&#39;:<br>        sns.displot(stellar_data, x=col)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*moS8vHiuOIeAw5J8qrJxyQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jwR6ndlB7sP8139YHkZ0zQ.png" /></figure><p>You can analyse the plots for exploratory analysis and derive valuable insights but I am not a Astronomer or Astrophysicist, so I leave the rest to you.</p><h4>Step 5: Prepare the data</h4><p>The below code moves the class column to the last in the dataframe.</p><pre>label_col = stellar_data[&#39;class&#39;]<br>del stellar_data[&#39;class&#39;]<br>stellar_data[&#39;class&#39;] = label_col</pre><p>Now we separate into X and y ,and then into train and test set</p><pre>X = stellar_data.iloc[:,:-1]<br>y = stellar_data.iloc[:,-1]<br>X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.2, random_state=42)</pre><h4>Step 6: Training the Model</h4><pre>model = XGBClassifier(n_jobs=4,n_estimators=100,learning_rate=1)<br>model.fit(X_train, y_train)<br>y_pred = model.predict(X_test)<br>accuracy_score(y_test, y_pred)</pre><p><em>Ignore the hyperparameters for now.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/1*PWXEq988VZuKYKxMamQBBg.png" /></figure><p>The model has an accuracy of 97.8 % not bad. But we can go further.</p><h4>Step 7: Tuning Hyperparameters</h4><p>XGboost provides a long list of hyperparameters but for our implementation we will focus on 3.</p><ol><li>learning_rate (eta):Controls the step size at each iteration during gradient boosting. Lower values make the model more robust but require more boosting rounds. Typical Range:[0 to 1]</li><li>n_estimators: The number of boosting rounds (decision trees) to build. More trees can improve performance but increase computation time. Typical Range:[50 to 5000]</li><li>max_depth:Maximum depth of each decision tree. Controls the complexity of the model and helps prevent overfitting. Typical Range:[3 to 10]</li></ol><blockquote>Higher the n_estimators lower the learning_rate should be. High max_depth captures more insights &amp; feature interactions but also increases overfitting.</blockquote><p><strong>Let’s now fit the model with new Hyperparameters</strong></p><pre>model = XGBClassifier(n_jobs=4,n_estimators=300,learning_rate=0.1,max_depth=5)<br>model.fit(X_train, y_train)<br>y_pred = model.predict(X_test)<br>accuracy_score(y_test, y_pred)</pre><blockquote>n_jobs refers to the number of threads spawn to run the model. Part of parallelisation optimisation.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/172/1*if-bXuWOOYscTUcQIEGvoA.png" /></figure><p>The accuracy has improve to 97.94% from 97.86%. We can experiment the model with other parameters to further improve the result, I leave the experimentation to you. Instead of manually tinkering around you can use Scikit-learn’s GridsearchCV and RandomSearchCV to find the best parameters but they consume large amount of resources to run.</p><pre># from sklearn.model_selection import GridSearchCV<br># parameters = {<br>#     &#39;learning_rate&#39;: [0.1, 0.01, 0.001],  # Step size shrinkage used in update to prevent overfitting<br>#     &#39;max_depth&#39;: [3, 5, 7],               # Maximum depth of a tree<br>#     &#39;n_estimators&#39;: [100, 200, 300],     # Number of boosting rounds<br>#     &#39;subsample&#39;: [0.8, 0.9],             # Subsample ratio of the training instances<br>#     &#39;colsample_bytree&#39;: [0.8, 0.9],      # Subsample ratio of columns when constructing each tree<br>#     &#39;gamma&#39;: [0, 0.1, 0.2],              # Minimum loss reduction required to make a further partition on a leaf node<br>#     &#39;min_child_weight&#39;: [1, 3, 5]         # Minimum sum of instance weight (hessian) needed in a child<br># }<br># # Create GridSearchCV with the XGBoost classifier and parameters<br># grid_search = GridSearchCV(estimator=XGBClassifier(), param_grid=parameters, scoring=&#39;accuracy&#39;, cv=5,n_jobs=5,verbose=2)<br><br># # Perform the grid search on the training data<br># grid_search.fit(X_train, y_train)<br><br># # Access the best parameters found by GridSearchCV<br># best_params = grid_search.best_params_<br><br># print(&quot;Best Parameters:&quot;, best_params)</pre><h4>Step 8: Metrics of our model</h4><p>For Classification Accuracy score is not enough, specially with unbalanced dataset(instance of one class is more than other class by a huge margin). We should also check for <strong>Precision</strong>(proportion of true positive predictions among all positive predictions) and <strong>Recall</strong>( proportion of true positive predictions among all actual positive instances).</p><p>In fraud detection, precision is more important because you want to minimize false positives. False positives in this context would mean flagging a transaction as fraudulent when it’s actually legitimate, which could inconvenience customers and lead to a loss of trust.</p><p>On the other hand, in medicine, especially in disease detection or diagnosis, recall is more important. It’s crucial to capture all positive cases (e.g., detecting a disease) even if it means having a higher number of false positives. Missing a true positive (false negative) in medicine could have serious consequences for the patient’s health and well-being.</p><p><strong>Let’s built a confusion matrix for checking the true positives and true negatives of the model’s prediction.</strong></p><pre>cm = confusion_matrix(y_test, y_pred, labels=model.classes_)<br>fig, ax = plt.subplots(figsize=(9, 6))<br>sns.heatmap(cm,annot=True,fmt=&quot;&quot;,linewidth=.5, cmap=&quot;mako&quot;,xticklabels=model.classes_,yticklabels=model.classes_)<br>ax.set(xlabel=&quot;Predicted&quot;, ylabel=&quot;True&quot;)<br>ax.xaxis.tick_top()<br>plt.yticks(rotation=0)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uc31ctwBOTs7fzGzKzxm6g.png" /></figure><p>As expected our model, has excellent accuracy but some true Quasars are labelled as Galaxy and a few Galaxy are labelled as Quasars.</p><p><strong>Let’s check for Precision and Recall score for each class</strong></p><pre>from sklearn.metrics import precision_score, recall_score, average_precision_score<br><br>precision = precision_score(y_test, y_pred, average=None)<br>recall = recall_score(y_test, y_pred, average=None)<br><br><br># Print the results for each class<br>for class_idx in range(len(precision)):<br>    print(f&quot;Class {model.classes_[class_idx]}:&quot;)<br>    print(f&quot;Precision: {precision[class_idx]}&quot;)<br>    print(f&quot;Recall: {recall[class_idx]}&quot;)<br>    print(&quot;--------------&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*kgzbqkEdy4GY4Ce2SRPpDQ.png" /></figure><p>From the above results, it is evident that the model performs exceptionally well for Galaxy and Star classification. However, when it comes to Quasar classification, the model’s performance is comparatively less accurate.</p><p>So, if you care about Galaxies and Stars go for this model, and if you care about Quasars maybe tinker the model’s parameters or find more dataset.</p><h3>Conclusion</h3><p>This article explains the basic theory of Decision Trees, Gradient Boosted Trees, and Xgboost. Furthermore, it includes an implementation of the Xgboost classifier for classifying the night sky, with the utilization of Recall/Precision &amp; Confusion Matrix evaluation metrics. Other metrics such as ROC, AUC, and f1 scores are also available but not implemented here. It’s worth noting that Xgboost also provides other classes for regression tasks.</p><h4><a href="https://mlearning.substack.com/about">WRITER at MLearning.ai</a> //<a href="https://open.substack.com/pub/mlearning/p/machine-learning-eeg-ear-emotion-prediction-ai?r=z7zu8&amp;utm_campaign=post&amp;utm_medium=web"> EEG AI Prediction</a> // <a href="https://open.substack.com/pub/mlearning/p/learn-how-to-animate-midjourney-images-tutorial?r=z7zu8&amp;utm_campaign=post&amp;utm_medium=web">Animate Midjourney</a></h4><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7014f10a722" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>