<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Harshittripathi on Medium]]></title>
        <description><![CDATA[Stories by Harshittripathi on Medium]]></description>
        <link>https://medium.com/@harshittripathi001?source=rss-b19b2ba33e0a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*A9D245WestFrDz1d.jpg</url>
            <title>Stories by Harshittripathi on Medium</title>
            <link>https://medium.com/@harshittripathi001?source=rss-b19b2ba33e0a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 22:55:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@harshittripathi001/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scaling Vector Search from 100K to 100M Embeddings: A Systems Engineering Guide]]></title>
            <link>https://medium.com/@harshittripathi001/scaling-vector-search-from-100k-to-100m-embeddings-a-systems-engineering-guide-4222f7018f31?source=rss-b19b2ba33e0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4222f7018f31</guid>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Harshittripathi]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 06:21:42 GMT</pubDate>
            <atom:updated>2026-11-02T09:18:34.747Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction: It was never an AI problem</h3><p>Scaling vector search from 100K to 100M embeddings is a memory-hierarchy problem. The model that produced the embeddings stops mattering long before the index does. What decides success is where each byte lives CPU cache, DRAM, NVMe and how often you have to touch it per query.</p><p>At 100K vectors, almost any design works. A brute-force scan of 100K 1536-dimension vectors is about 600MB of data, and modern SIMD can chew through that in tens of milliseconds. Bad architecture is invisible at this size.</p><p>At 1M to 100M vectors, every weakness surfaces at once. Memory bills grow linearly with the corpus. p99 latency becomes erratic as background merges compete with queries. Deletes and updates quietly degrade index quality until a rebuild is forced.</p><p>Once you leave exact (flat) search behind, you are trading three things against each other:</p><ul><li><strong>Recall: </strong>how often the true nearest neighbours appear in your results.</li><li><strong>Throughput and latency: </strong>QPS, and especially p99 rather than average.</li><li><strong>Hardware cost: </strong>mostly RAM, increasingly NVMe and CPU.</li></ul><p>You can pick any two cheaply. The third always costs something. The rest of this article walks through the four indexing strategies that dominate production systems HNSW, IVF, quantization and DiskANN/Vamana ; explains the mechanics behind each, shows where each one breaks, and ends with a tier-by-tier playbook for the 100K–100M journey.</p><h3>The memory math you should be able to do on a whiteboard</h3><p>Every capacity decision starts with one formula. For N vectors of dimension D, stored at b bytes per dimension, with G bytes of graph links and m bytes of metadata per vector, and a headroom fraction h for merges and rebuilds:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*PnhmYDZfl6E_l6wrgiO5eA.png" /></figure><p>A 1536-dimension FP32 vector is 6,144 bytes. Multiply by 100M and you get roughly 614GB of raw vector data before anything else is added. The table shows how that payload shrinks under quantization, and how small the HNSW graph is by comparison.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n3fSF_GQl35W9g8ThLjQPw.png" /></figure><p>Figures assume D = 1536 and roughly 150 bytes of neighbour lists per node at M = 16. Halve the vector columns for 768-dimension models.</p><p>Three observations follow from this table:</p><ul><li><strong>Vectors dominate, not the graph.</strong> At M = 16, HNSW links are a few percent of FP32 payload. They become significant only after you quantize the vectors, when a 15GB graph sits next to a 19GB binary index.</li><li><strong>Headroom is real but engine-specific.</strong> Segment-based engines such as Milvus, Qdrant and OpenSearch build new segments before dropping old ones. During a merge, part of the index briefly exists twice. A 25% buffer is a sensible starting point for those systems, not a universal law.</li><li><strong>Metadata is the hidden line item.</strong> Payload fields, IDs, filter indexes and WAL buffers often add tens of gigabytes at 100M. Measure them; don’t assume zero.</li></ul><p>Put together, 100M FP32 vectors in HNSW typically need 800GB to 1TB of provisioned RAM. That is the number that pushes teams toward quantization and disk-based indexes.</p><h3>HNSW: the default for in-memory speed</h3><p>HNSW (Hierarchical Navigable Small World) is the fastest general-purpose option when everything fits in RAM. It is also the index most likely to surprise you in production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aUxL2SVAE3c-hdLmsSJrfg.png" /></figure><h3>How it works</h3><p>HNSW borrows the idea of a skip list and applies it to a proximity graph. The bottom layer contains every vector, each linked to its nearest neighbours. Each higher layer holds an exponentially smaller random subset. A query enters at the sparse top layer, greedily walks toward the target, then drops down a layer and repeats. The upper layers provide long jumps; the bottom layer provides precision.</p><p>Three parameters control the behaviour:</p><ul><li><strong>M: </strong>the number of bidirectional links per node (the bottom layer usually allows 2M). Higher M improves recall on hard, high-dimensional data but raises memory and build time. Values of 16–32 cover most workloads.</li><li><strong>efConstruction: </strong>the candidate list size while inserting. Higher values produce a better-connected graph at the cost of slower ingestion. 100–400 is typical.</li><li><strong>efSearch: </strong>the candidate list size at query time. This is your main runtime knob: raising it buys recall with latency, query by query, without a rebuild.</li></ul><h3>Where it breaks: churn</h3><p>HNSW graphs are hard to edit. Most engines handle a delete by marking the node as a tombstone. The node still sits in the graph, still gets visited during traversal, and still consumes memory. Its neighbours lose a useful routing path.</p><p>Under heavy update or delete traffic, tombstones accumulate. Recall drifts down and latency drifts up. Eventually a compaction or rebuild repairs the graph a CPU-heavy job that competes with live queries and needs extra memory while it runs. This is where the “compaction locks up the cluster” stories come from.</p><p>Mitigations include scheduling merges off-peak, capping tombstone ratios before triggering a rebuild, and isolating write-heavy tenants into their own segments or collections.</p><h3>Where it breaks: filtered search</h3><p>Real queries are rarely pure similarity search. They come with filters: tenant ID, language, date range, permissions. Filtering interacts badly with graph traversal.</p><ul><li><strong>Post-filtering</strong> runs the search and then drops results that fail the filter. With a selective filter, you may end up with too few results, or none.</li><li><strong>Naive pre-filtering during traversal</strong> skips nodes that fail the filter. With a restrictive filter, the surviving nodes no longer form a connected graph. The greedy walk gets stuck in a dead end and recall collapses.</li></ul><p>Modern engines address this with adaptive strategies. They estimate filter selectivity, fall back to brute force over the matching subset when it is small, traverse through non-matching nodes without returning them, or build extra links for frequently filtered attributes. Before choosing an engine, test it with your real filter distributions, not just unfiltered benchmarks.</p><h3>IVF: predictable partitioning with a drift problem</h3><p>IVF (Inverted File Index) trades some latency for memory predictability. It is the workhorse when RAM is tight and write volume is high.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n7tH9a8CMAXPZe7phX6lYw.png" /></figure><h3>How it works</h3><p>IVF runs k-means on a training sample to produce nlist centroids. Each centroid defines a Voronoi cell, and every vector is stored in the list of its nearest centroid. At query time, the engine finds the nprobe closest centroids and scans only those lists.</p><ul><li><strong>nlist</strong> sets the granularity. More lists mean smaller lists to scan, but a query must probe more of them to keep recall. FAISS’s commonly cited guidance is 4√N to 16√N. At 10M vectors that is roughly 12,600 to 50,600 lists.</li><li><strong>nprobe</strong> is the runtime knob. Recall rises steeply for the first few probes, then flattens. Most production settings land somewhere between 1% and 10% of nlist.</li></ul><p>The overhead beyond the vectors themselves is tiny: one centroid table and a list ID per vector. Inserts are cheap assign to the nearest centroid and append. There is no graph to repair.</p><h3>Where it breaks: concept drift</h3><p>Centroids reflect the data you trained on. If you train at 500K vectors and grow to 50M, or your content mix shifts (new products, new languages, a new embedding use case), new vectors pile into whichever cells happen to be closest.</p><p>The result is skew. Some lists hold ten times the vectors of others. Queries that land on these hotspot cells scan far more data than average, which shows up directly as p99 spikes. Queries near cell boundaries need higher nprobe to find their true neighbours, which burns CPU on every query.</p><h3>Keeping IVF healthy</h3><ul><li><strong>Monitor list-size distribution.</strong> Track the ratio of the largest list to the median. A rising ratio is your early warning.</li><li><strong>Retrain on a fresh sample.</strong> Rebuild centroids from a sample that reflects current data, then reassign. Many teams schedule this as a periodic background job.</li><li><strong>Split hot cells locally.</strong> Some engines can split an oversized list without retraining the whole index.</li><li><strong>Train generously up front.</strong> If you know you will grow 100x, size nlist and the training sample for the target scale, not today’s.</li></ul><h3>Quantization: compression to beat the bandwidth wall</h3><p>Quantization is not a routing index. It is a compression layer you put under HNSW, IVF or DiskANN. Its job is to cut both memory footprint and memory bandwidth and at scale, bandwidth is often the real bottleneck. Moving a 6KB FP32 vector from DRAM into the CPU for every distance calculation is slow, no matter how fast the arithmetic is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bjVK56QnqXOofwdV_EWBXg.png" /></figure><h3>The three main families</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w19fqUEiOuEuP5VOvLGyEw.png" /></figure><p>Recall impacts are typical ranges and vary by embedding model. Always measure on your own data.</p><ul><li><strong>SQ8</strong> maps each FP32 dimension to an 8-bit integer using per-dimension (or global) min/max ranges. It is simple, robust and nearly lossless for most modern embeddings.</li><li><strong>BQ</strong> keeps only the sign of each dimension. Distance becomes a Hamming count computed with an XOR and a population-count instruction, which is extremely cheap per word. It works best on high-dimensional models (1024+ dims) whose dimensions are roughly centred on zero; on low-dimensional or skewed embeddings it degrades quickly.</li><li><strong>PQ</strong> splits each vector into M sub-vectors and replaces each with the ID of its nearest entry in a small learned codebook (usually 256 entries, one byte). At query time, the engine precomputes a table of distances from the query’s sub-vectors to every codebook entry. Each database vector’s distance then becomes M table lookups and additions. This is Asymmetric Distance Computation (ADC), and because the tables are small enough to stay in L1/L2 cache, it is very fast.</li></ul><h3>The production pattern: oversample, then rescore</h3><p>Never trust heavily quantized scores on their own. The pattern that works at 10M–100M is two-stage:</p><ol><li>Search the compressed index in RAM for K × an oversampling factor candidates. For a top-10 query with 4x oversampling, fetch 40.</li><li>Fetch the full-precision vectors for just those candidates from RAM if you can afford it, otherwise from NVMe.</li><li>Recompute exact distances and return the true top K.</li></ol><p>The oversampling factor is the tuning knob. SQ8 often needs little or none. BQ typically needs 3x–10x, more for lower-dimensional models. PQ sits in between depending on code size.</p><p>The rescoring step is where disk performance matters. Fetching 40 random 6KB vectors per query is 40 random reads. On local NVMe issued in parallel (for example via io_uring), that adds roughly a millisecond or two. On network-attached block storage it can add far more, which is why storage choice belongs in the index design.</p><h3>DiskANN and Vamana: the NVMe cost killer</h3><p>DiskANN was built for billion-scale search on a single machine. The same design now makes sense much earlier — often from around 25M–50M vectors — because it moves the bulk of the index from DRAM to NVMe, which costs a small fraction per gigabyte.</p><h3>The Vamana graph</h3><p>Vamana is a single-layer proximity graph. Instead of HNSW’s hierarchy, it relies on a pruning rule controlled by a parameter α. When choosing a node’s neighbours, Vamana keeps some edges that are longer than strictly necessary. With α = 1 the graph keeps only short, local edges. With α around 1.2 (the value commonly used in the DiskANN paper), it keeps enough long-range edges that a greedy search crosses the dataset in few hops.</p><p>Fewer hops is the whole point. On disk, every hop is an I/O, so a graph with a short search path is worth more than a graph with slightly better local precision.</p><h3>How the on-disk layout works</h3><ul><li><strong>In RAM:</strong> a PQ-compressed copy of every vector (often 32–64 bytes each) plus a small cache of frequently visited nodes. This guides navigation.</li><li><strong>On NVMe:</strong> each node’s full-precision vector and its neighbour list, packed together into one aligned block (typically a 4KB sector). One read returns everything needed to evaluate the node and decide where to go next.</li><li><strong>Beam search:</strong> the query keeps a small set of promising candidates and issues their reads in parallel, using PQ distances to decide which to expand. Full-precision vectors arrive with the reads, so exact rescoring comes almost free.</li></ul><p>A well-tuned query completes in a small number of parallel I/O round trips-on the order of 5 to 15 for typical configurations. Each round trip on local NVMe takes on the order of 100 microseconds, which is why total latency lands in the low milliseconds rather than tens.</p><h3>What it requires</h3><ul><li><strong>Local NVMe, not network storage.</strong> The design depends on high random-read IOPS and low latency. Network block storage typically has much higher per-read latency and will dominate query time.</li><li><strong>Async I/O.</strong> Engines use io_uring (Linux) or equivalent to keep many reads in flight per query.</li><li><strong>A slower, heavier build.</strong> Building a Vamana graph at 100M takes significant time and memory. Fresh inserts are usually handled by a separate in-memory index that is merged into the disk graph periodically.</li><li><strong>Realistic latency expectations.</strong> You trade single-digit-millisecond in-memory p99 for something closer to 10-15ms. For most retrieval-augmented generation workloads, where the LLM call takes hundreds of milliseconds, that trade is invisible to users.</li></ul><h3>The playbook: what to run at each tier</h3><p>The right index changes as the corpus grows. The table summarises the progression for 1536-dimension embeddings; latency and cost figures are illustrative and depend heavily on hardware, filters and recall targets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vAMxK8qaY7NBnZtt86AtHg.png" /></figure><h3>100K-1M: keep it simple</h3><p>At this size, brute force is a legitimate production choice. AVX-512 or comparable SIMD scans 100K vectors in milliseconds, with perfect recall and no index to maintain. As you pass roughly 500K, or when your latency budget tightens, switch to uncompressed HNSW. It fits comfortably on a single modest node.</p><p><strong>Watch:</strong> p99 latency under concurrent load, not single-query latency.</p><h3>1M-10M: the memory inflection point</h3><p>FP32 HNSW at 10M is around 60GB of vectors alone. Switching to HNSW + SQ8 cuts vector memory by 75%. Because the graph, metadata and headroom don’t shrink, total provisioned RAM typically falls by 60–70%. Recall loss is usually around 1% or less-confirm on your own evaluation set.</p><p><strong>Watch:</strong> recall@K against a brute-force baseline on a held-out query set, measured after every reindex.</p><h3>10M-50M: the cost and write bottleneck</h3><p>Two paths make sense here, and the deciding factor is write volume.</p><ul><li><strong>Read-heavy:</strong> HNSW over binary-quantized vectors, with 3x–10x oversampling and FP32 rescoring from local NVMe. Memory drops by another order of magnitude while recall stays high.</li><li><strong>Write-heavy:</strong> IVF-PQ. Inserts are appends, there is no graph to repair, and memory is tightly bounded. You pay with more tuning (nlist, nprobe, code size) and periodic centroid retraining.</li></ul><p><strong>Watch:</strong> tombstone ratio and merge duration (HNSW), or list-size skew (IVF).</p><h3>50M-100M: the NVMe tier</h3><p>At this point, paying for 500GB+ of DRAM is hard to justify. DiskANN/Vamana on instances with local NVMe keeps only compressed navigation data in RAM. You accept p99 rising from low single digits to roughly 10-15ms. In exchange, infrastructure cost often falls dramatically reductions of 70 - 80% are commonly reported, though your number depends on instance pricing and how much RAM you were provisioning before.</p><p><strong>Watch:</strong> NVMe read latency percentiles and I/O queue depth, alongside query p99.</p><h3>Cross-cutting concerns that decide success at every tier</h3><p>Index choice gets the attention, but four operational habits decide whether any of it works in production.</p><h3>Benchmark with your own data and filters</h3><p>Public benchmarks use clean datasets, no filters and steady read-only traffic. Your production system has none of those properties. Build a small evaluation harness: a few thousand real queries, exact ground truth from brute force search, and your actual filter mix. Measure recall@K, p50, p99 and QPS together, at the concurrency you expect at peak.</p><h3>Track p99, not averages</h3><p>Average latency hides the problems this article is about. Merges, hotspot IVF lists, tombstone heavy graph regions and slow disk reads all show up first in the tail. Alert on p99 and p999, and correlate spikes with background jobs.</p><h3>Plan for churn from day one</h3><p>Ask early how much of your corpus changes per day. A catalogue that updates 1% daily behaves very differently from a chat-history index that grows constantly and expires old data. Churn rate should influence the choice between graph and partition-based indexes as much as corpus size does.</p><h3>Treat filters as a first-class workload</h3><p>If most queries carry a tenant or permission filter, design around it. Options include partitioning by tenant, building separate indexes for large tenants, or choosing an engine with filter-aware traversal. Test the most selective filters you have, because that is where recall falls apart first.</p><h3>Conclusion: tune to the hardware, not just the embeddings</h3><p>The path from 100K to 100M vectors is a sequence of moves down the memory hierarchy. Flat search keeps everything simple. HNSW keeps everything in RAM. Quantization shrinks what has to stay in RAM. DiskANN puts most of the index on NVMe and keeps only a compressed map in memory.</p><p>Each step trades a little latency or complexity for a large drop in cost. None of them is free, and none of them is a black box. The teams that scale smoothly are the ones that can do the memory math, know which background job is eating their p99, and test against their real queries and filters.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4222f7018f31" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Apache Airflow on AWS-Part 2: Building the Networking Security]]></title>
            <link>https://medium.com/@harshittripathi001/deploying-apache-airflow-on-aws-part-2-building-the-networking-security-b6aedc0446c8?source=rss-b19b2ba33e0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6aedc0446c8</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[medium]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Harshittripathi]]></dc:creator>
            <pubDate>Mon, 27 Apr 2026 06:08:05 GMT</pubDate>
            <atom:updated>2026-04-27T06:08:05.059Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently shared how we built the neighborhood for Deploying Apache Airflow on AWS-Part 1: Building the Networking Foundation. That was <a href="https://medium.com/@gaurav.rawat/deploying-apache-airflow-on-aws-part-1-building-the-networking-foundation-0e8e20f1c574"><strong>Part 1</strong></a>, written by my friend <strong>Gaurav Rawat</strong>:<br> 👉 <a href="https://medium.com/@gaurav.rawat">https://medium.com/@gaurav.rawat</a></p><p>But once the streets are built, another problem appears immediately:</p><p><strong>Who gets keys to which doors?</strong><br> <strong>Who is allowed into the house?</strong><br> <strong>Which rooms should stay locked forever?</strong></p><p>Because networking gives you the city layout.</p><p><strong>Security Groups decide who can enter each room.</strong></p><p>This is <strong>Part 2</strong> of our Airflow-on-AWS series — where we secure the entire house, room by room.</p><p>Github :- <a href="https://github.com/hariomhardy/Airflow-Infra">https://github.com/hariomhardy/Airflow-Infra</a></p><h3>Why Security Groups Matter</h3><p>Imagine building a large smart house.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dKXY3VDij4mtQYQB.jpg" /></figure><p>You have:</p><ul><li>A <strong>front gate</strong> where guests arrive</li><li>A <strong>control room</strong> running the whole system</li><li>A <strong>kitchen staff area</strong> doing the actual work</li><li>A <strong>vault</strong> storing critical records</li><li>A <strong>private tunnel</strong> connecting to outside vendors</li></ul><p>Would you give every visitor access to every room?</p><p>Of course not.</p><p>But that is exactly what careless cloud infrastructure often does.</p><p>Security Groups are AWS’s way of saying:</p><blockquote><em>“Every door needs rules.”</em></blockquote><p>They act like intelligent digital locks attached to each resource.</p><p>Not the whole VPC.<br>Not the whole subnet.<br>Each resource.<br>That means every component gets its own access policy.</p><h3>Security Groups Are Smart Locks</h3><p>Traditional firewalls can feel like guards with clipboards.</p><p>Security Groups are smarter.</p><p>If you allow someone to enter through the front door, AWS automatically allows the reply traffic back out.</p><p>That’s called <strong>stateful behavior</strong>.</p><p>Meaning:</p><ul><li>If a user requests the Airflow UI</li><li>The request enters through the allowed door</li><li>The response automatically finds its way back</li></ul><p>No extra rule needed.<br>Clean. Elegant. Less chaos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7tE3p9zFTP7vdR3_q-IHWg.png" /></figure><ul><li><strong>Ingress Security Rules:</strong> Rules that control <strong>incoming traffic</strong> to a network resource (such as a server, VM, or database). They define <strong>who can connect in</strong>, on which <strong>ports</strong>, and using which <strong>protocols</strong>.</li><li><strong>Egress Security Rules:</strong> Rules that control <strong>outgoing traffic</strong> from a network resource. They define <strong>where the resource can send data</strong>, on which <strong>ports</strong>, and using which <strong>protocols</strong>.</li></ul><h3>The ALB Security Group — The Front Gate</h3><p>Every house needs one public entrance.<br>For us, that is the <strong>Application Load Balancer</strong>.<br>It is the only piece of infrastructure meant to face the internet.</p><p>So its rules are simple:</p><ul><li>Allow visitors on <strong>HTTP (80)</strong></li><li>Allow visitors on <strong>HTTPS (443)</strong></li><li>Reject everything else</li></ul><p>Think of it as a guarded front gate.</p><p>Guests can ring the bell.</p><p>They cannot wander into the backyard, basement, or vault.</p><p>Once traffic enters, the ALB forwards it only to the worker nodes on the specific NodePort range it is meant to use. No unnecessary wandering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U2AB1xMInYT-WeRLmHgu_w.png" /><figcaption>Rules for incoming traffic</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4AcI1JzX9Mtf7Cccks1bVg.png" /><figcaption>Rules for outgoing traffic</figcaption></figure><h3>EKS Worker Nodes — The Staff Entrance</h3><p>Behind the front gate is the operational area.</p><p>These worker nodes are where Airflow actually lives:</p><ul><li>Webserver pods</li><li>Scheduler pods</li><li>Workers</li><li>Internal services</li></ul><p>Think of these as the staff inside the house making everything function.</p><p>They need to communicate with:</p><ul><li>Each other</li><li>The control plane</li><li>The database</li><li>Internal AWS services</li></ul><p>But only in approved ways.</p><p>So the worker node Security Group allows:</p><ul><li>Node-to-node traffic (staff talking internally)</li><li>Kubelet traffic from the control plane</li><li>NodePort traffic from the ALB</li></ul><p>Nothing random. Nothing public.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N-MT2cmqqiRRMCX6pP5ZxA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FuB6hXhUSzpkrKh1-Ua43A.png" /></figure><h3>EKS Control Plane — The Control Room</h3><p>Every smart house has a hidden control room.</p><p>This room controls locks, lights, alarms, and systems.</p><p>In Kubernetes, that’s the <strong>EKS control plane</strong>.</p><p>It tells the cluster what to run, where to run it, and whether everything is healthy.</p><p>So it needs access to the worker nodes.</p><p>But only for management traffic.</p><p>Not general browsing. Not unrestricted access.</p><p>Think of it as the house manager with a master key that only opens operational doors.</p><h3>RDS PostgreSQL — The Vault</h3><p>Now the most sensitive room in the house:</p><p>The vault.</p><p>This is where Airflow stores metadata:</p><ul><li>DAG states</li><li>Task runs</li><li>Connections</li><li>Schedules</li><li>Operational history</li></ul><p>This should never be public.</p><p>Not “temporarily.”<br> Not “just for testing.”<br> Not “only my IP.”</p><p>Never.</p><p>So the RDS Security Group allows only PostgreSQL traffic from the worker nodes.</p><p>That means only trusted staff can enter the vault.</p><p>No guests.<br> No street traffic.<br> No accidents.</p><p>Exactly how production databases should be built.</p><h3>VPC Endpoints — The Private Service Tunnel</h3><p>Sometimes the house needs supplies.<br>Packages. Credentials. Images. Secrets.<br>Normally, you might send someone out through the public road.<br>But a smarter design builds a private tunnel.<br>That’s what <strong>VPC Endpoints</strong> do.</p><p>They let workloads privately reach services like:</p><ul><li>ECR</li><li>STS</li><li>Secrets Manager</li></ul><p>Without stepping onto the public internet.</p><p>Their Security Group only allows HTTPS traffic from inside the VPC.</p><p>Think of it as a secure delivery tunnel behind the walls.</p><p>Invisible from the street.</p><h3>What’s Next?</h3><p><strong>Part 3:</strong> RDS PostgreSQL in private subnets<br><strong>Part 4:</strong> EKS cluster for Airflow containers<br><strong>Part 5:</strong> Airflow deployment, DAG syncing, UI behind ALB</p><p>🚀 Want to see how this was built end-to-end?</p><p>🔗 GitHub Repo: <a href="https://github.com/hariomhardy/Airflow-Infra">https://github.com/hariomhardy/Airflow-Infra</a></p><p>💡 Got questions or ideas? Drop a comment — happy to discuss!</p><p>🔒 <strong>Part 3 Coming Next:</strong><br> We’ll build the vault itself — PostgreSQL on RDS inside private subnets.</p><p>👉 Follow for updates:<br> <a href="https://medium.com/@harshittripathi001">https://medium.com/@harshittripathi001</a><br> <a href="https://medium.com/@gaurav.rawat">https://medium.com/@gaurav.rawat</a></p><p>🤝 Let’s connect on LinkedIn:<br> <a href="https://www.linkedin.com/in/harshit-tripathi-780094164/">https://www.linkedin.com/in/harshit-tripathi-780094164/</a><br> <a href="https://www.linkedin.com/in/gauravrawat1412/">https://www.linkedin.com/in/gauravrawat1412/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6aedc0446c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>