<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aathi on Medium]]></title>
        <description><![CDATA[Stories by Aathi on Medium]]></description>
        <link>https://medium.com/@aathi8924?source=rss-607c8d5f50eb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*lL-dFe1-OBY6IoJq</url>
            <title>Stories by Aathi on Medium</title>
            <link>https://medium.com/@aathi8924?source=rss-607c8d5f50eb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:52:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aathi8924/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Biobank Data to AI Agent: Engineering a Trustworthy Query Layer for FinnGen]]></title>
            <link>https://medium.com/@aathi8924/from-biobank-data-to-ai-agent-engineering-a-trustworthy-query-layer-for-finngen-9ca4bf016738?source=rss-607c8d5f50eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ca4bf016738</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[pharmaceutical]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Aathi]]></dc:creator>
            <pubDate>Mon, 20 Jul 2026 08:02:10 GMT</pubDate>
            <atom:updated>2026-07-20T08:06:44.913Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How I built a LangGraph agent over a DuckDB analytics layer to make FinnGen GWAS data queryable — and why the design deliberately keeps the LLM out of the data path.</em></p><p>FinnGen is one of the largest genome-wide association study (GWAS) efforts in the world — Finnish biobank data linking genetic variants to disease phenotypes across hundreds of thousands of participants. The statistics are public-ish, well-documented, and completely unusable to anyone who isn’t comfortable writing SQL against a summary-statistics schema they’ve never seen before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cU6rZG99r1IR8Yp7lZqoFQ.png" /><figcaption>High Level Architecture</figcaption></figure><p>That’s a familiar gap in data engineering: <strong>the data is trustworthy, the access pattern isn’t.</strong> So I built a small research assistant — finngen_data_ai_agent — that lets you ask questions like <em>&quot;compare type 2 diabetes and coronary artery disease&quot;</em> or <em>&quot;find the strongest gene associations for APOE&quot;</em> in plain English, and get answers grounded in a real local analytics database, not an LLM&#39;s guess.</p><p>The interesting part isn’t the chat interface. It’s the architecture decisions that keep the system honest, debuggable, and usable with or without an LLM at all.</p><h4>Design principle: the LLM is a translator, not an analyst</h4><p>Before touching code, I set one hard constraint: <strong>the model never computes anything.</strong> It doesn’t write SQL, it doesn’t estimate an effect size, it doesn’t infer causation from association data. Every number the user sees comes out of a deterministic query against DuckDB. The LLM’s only job — and it’s optional — is deciding which question maps to which tool, and turning a tool’s raw output into readable prose.</p><p>This matters specifically for genomics: association ≠ causation, and a model that “sounds confident” about a gene-disease link is a liability, not a feature. The system is explicit that every answer reflects <em>statistical association</em>, never diagnosis or medical advice.</p><p>That one constraint shapes everything downstream: the data pipeline, the agent graph, and the fallback behavior.</p><h3>Part 1 — Getting the data into a queryable state</h3><p>Before any agent code runs, raw FinnGen manifest and summary-statistics files move through a <strong>medallion pipeline</strong>: Raw → Ingestion → Trusted → Refined → Serving.</p><pre>Raw Data (manifest + raw)<br>   → Ingestion (src/ingestion)<br>      → Trusted Data (validated, deduped)<br>         → Refined Data (data/processed, parquet)<br>            → Serving Build (src/database/create_database.py)<br>               → Serving Layer (database/finngen.db, DuckDB)</pre><p>A few things about this that are easy to get wrong and worth calling out:</p><ul><li><strong>Raw stays raw.</strong> Nothing is mutated in place — everything downstream is a new artifact, tied back to a manifest, so a rebuild is always reproducible from source.</li><li><strong>Validation is a gate, not a suggestion.</strong> The Trusted layer is where dedup and validation happen <em>before</em> anything is curated — bad records don’t get a chance to silently propagate into the analytics layer.</li><li><strong>Refined data lands as Parquet</strong>, not raw text — this is what actually gets loaded into DuckDB, so the analytics layer is reading columnar data, not parsing files at query time.</li><li><strong>The serving layer is rebuildable on demand:</strong> python src/database/create_database.py regenerates finngen.db from whatever&#39;s in data/processed. If the underlying FinnGen release changes, there&#39;s one command to re-derive the whole database — no manual reconciliation.</li></ul><p>DuckDB earns its place here specifically because this is an <em>embedded analytical</em> workload: no server, no cluster, but real columnar performance over GWAS-scale tabular data (variants × phenotypes × chromosomes). For a project that needs to run on a laptop and still answer aggregate questions fast, that’s the right point on the spectrum between SQLite and a full warehouse.</p><h3>Part 2 — The agent: LangGraph, not a prompt-and-pray loop</h3><p>This is the part that separates it from a typical “chat with your CSV” demo. The runtime is a <strong>LangGraph</strong> state graph, not a single prompt:</p><pre>Query<br>  → Router (src/agent/router.py)              — selects the relevant analytics tool<br>     → Tool execution (src/agent/tools.py)    — LangChain tool wrappers<br>        → Query Layer (src/database/analytics.py)  — DuckDB queries<br>           → Serving Layer (database/finngen.db)<br>     → Response node (src/agent/prompts.py + optional OpenAI model)<br>        → Answer</pre><p>Each piece has one job:</p><ul><li><strong>router.py</strong> classifies intent and picks a tool from a fixed set — summarize the dataset, list phenotypes, top variants for a phenotype, top genes for a phenotype (or dataset-wide), search a gene&#39;s associations, compare two phenotypes, shared genes across phenotypes, or chromosome-level patterns. It&#39;s selecting <em>from known operations</em>, not generating a query from scratch.</li><li><strong>tools.py</strong> wraps each of those operations as a LangChain tool, so the graph calls them like typed functions instead of free-text.</li><li><strong>analytics.py</strong> is the only place that touches DuckDB — the deterministic core. Same input, same output, regardless of which LLM (or none) sits on top.</li><li><strong>prompts.py</strong> holds the system and answer-synthesis prompts. The LLM&#39;s real job is turning a tool&#39;s structured result into a concise, scientifically careful summary.</li><li><strong>state.py</strong> defines what travels between nodes; <strong>graph.py</strong> wires the graph together and owns execution.</li></ul><p>The result is a clean separation: <em>routing and retrieval are deterministic and testable; language generation is a thin, replaceable layer on top.</em></p><h3>Part 3 — Graceful degradation is a feature, not an afterthought</h3><p>Here’s the detail I’d lead with in an interview before anything else: <strong>the agent works without an LLM at all.</strong></p><pre>export OPENAI_API_KEY=&quot;your-api-key&quot;   # optional<br>export MODEL_NAME=&quot;gpt-4.1-mini&quot;       # optional override<br>export MODEL_TEMPERATURE=&quot;0.2&quot;         # optional override</pre><p>If OPENAI_API_KEY isn&#39;t set, the response path falls back to a lightweight local formatter that returns the tool&#39;s structured output directly — no LLM call, no failure mode, no degraded UX beyond &quot;less prose, same facts.&quot; That&#39;s a meaningfully different design than most LLM-agent projects, where the model sits on the critical path as a single point of failure. Here it&#39;s an enhancement layer that can be switched off entirely and the system still answers correctly, every time.</p><p>It’s a small design choice that signals a bigger one: <strong>the product is the analytics; the LLM is the interface.</strong></p><h3>Part 4 — Observability: three logs, not one</h3><p>Debugging an agent graph by re-reading stdout doesn’t scale past the second bug. The project writes three separate log streams instead of one undifferentiated file:</p><ul><li>logs/agent.log — general runtime events</li><li>logs/agent_state.log — state transitions and final answers</li><li>logs/agent_dag.log — graph execution traces</li></ul><p>Splitting these apart means “what did the router decide?” and “what did the state look like at each node?” become two different, answerable questions instead of grepping one firehose. Small thing, pays for itself the first time a routing decision looks wrong and you need to know <em>why</em> without re-running the whole graph under a debugger.</p><h3>Part 5 — Testing an agent, not just a pipeline</h3><p>The repo includes both a conventional regression suite and something less common: a <strong>prompt suite</strong>.</p><pre>pytest -q tests/test_router_and_logging.py<br>python tests/prompt_suite.py</pre><p>The regression tests cover router behavior and logging correctness — normal software testing. The prompt suite is different: it runs a batch of real natural-language questions — <em>“Summarize the overall dataset,”</em> <em>“Compare type 2 diabetes and coronary artery disease,”</em> <em>“What are the strongest effects in the dataset”</em> — through the actual agent and checks the outcomes hold up. For an LLM-adjacent system, that’s the test category that catches what a unit test can’t: routing drift, prompt edits that quietly break intent classification, tool output that no longer matches what the response node expects.</p><h3>What I’d change next</h3><p>The “tradeoffs” section every good engineering post needs and most skip:</p><ul><li><strong>The router picks one tool per query.</strong> Multi-hop questions (<em>“compare APOE’s top phenotype associations to the dataset-wide top genes”</em>) would need either a smarter router or an explicit planning step — that’s a current gap, not a hidden feature.</li><li><strong>No caching layer yet.</strong> Repeated identical queries re-hit DuckDB every time. Fine at today’s scale; the first thing I’d add under real concurrent traffic.</li><li><strong>Single local DuckDB file.</strong> Great for reproducibility and a zero-infra demo, not a multi-user serving story. If this moved from portfolio project to a shared internal tool, that’s the first architectural boundary to revisit.</li></ul><h3>Why this pattern generalizes</h3><p>Swap “FinnGen GWAS data” for support tickets, financial filings, or internal product metrics, and the shape of the system doesn’t change: a governed data pipeline feeding a deterministic query layer, a router that maps intent to a fixed, testable tool set, and an LLM that’s optional, swappable, and never in charge of computing the answer. That’s the pattern that survives contact with a real client or a real production environment — not because it’s clever, but because it fails safely and degrades gracefully instead of hallucinating with confidence.</p><h3>Try it</h3><p>Repo: <a href="https://github.com/aathikm8943/finngen_data_ai_agent">finngen_data_ai_agent</a></p><pre>python -m venv venv &amp;&amp; source venv/bin/activate<br>pip install -r requirements.txt<br>python src/database/create_database.py<br>python main.py &quot;Summarize the overall dataset&quot;</pre><p><em>I’m a Data &amp; AI Engineer building production-style data pipelines and LLM-adjacent agent systems. If you’re looking for a Data/AI Engineering role, or have a dataset that needs to be queryable without handing an LLM the keys to it, let’s talk.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ca4bf016738" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Using a Single Retrieval Method: Build a Smarter RAG Pipeline]]></title>
            <link>https://medium.com/@aathi8924/stop-using-a-single-retrieval-method-build-a-smarter-rag-pipeline-c12c7bba96af?source=rss-607c8d5f50eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/c12c7bba96af</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <dc:creator><![CDATA[Aathi]]></dc:creator>
            <pubDate>Tue, 26 May 2026 20:37:41 GMT</pubDate>
            <atom:updated>2026-05-26T20:37:41.788Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Production-Grade RAG Pipeline Using Open-Source Tools</h3><h4>How I queried nutrition PDFs intelligently — with Milvus, Ollama, BAAI re-ranking, and zero paid APIs</h4><p>If you’ve ever tried to query a collection of PDF documents using simple keyword search, you know the frustration. You ask <em>“what are the dietary guidelines for sodium intake?”</em> and get back something completely off-topic. The context is missing. The semantics are lost. The answer you need is buried somewhere across hundreds of pages.</p><p>That’s the problem Retrieval-Augmented Generation (RAG) was built to solve. But most RAG tutorials stop at the basics one retrieval method, a hosted vector DB, an OpenAI key. I wanted to build something more complete, and do it entirely with <strong>open-source tools</strong>.</p><p>This article walks through the concepts and decisions behind the project. For the full implementation, check out the <a href="https://github.com/aathikm8943/RAG-Project">GitHub repository</a>.</p><blockquote><strong>What This Project Does</strong></blockquote><blockquote>The pipeline lets you ask natural language questions over a collection of nutrition-related PDF documents and get accurate, cited answers. But more importantly, it’s a framework for thinking about retrieval because retrieval is where most RAG systems fail.</blockquote><p>What makes it interesting:</p><ul><li><strong>Seven retrieval strategies</strong> each solving a different kind of query</li><li><strong>Cross-encoder re-ranking</strong> to improve result quality before generation</li><li><strong>100% open-source stack</strong> no paid APIs, no cloud dependencies</li><li><strong>Milvus</strong> as the vector database with <strong>Attu</strong> for visual monitoring</li><li><strong>Ollama</strong> to run embeddings and LLMs locally</li><li><strong>BAAI</strong> models for state-of-the-art re-ranking</li></ul><h3>How It All Fits Together</h3><p>At a high level, the pipeline works like this:</p><pre>Your Question<br>     │<br>     ▼<br>Retriever (choose your strategy)<br>     │<br>     ▼<br>Re-ranker (optional, but recommended)<br>     │<br>     ▼<br>LLM generates answer from top chunks<br>     │<br>     ▼<br>Your Answer</pre><p>The key insight is that <strong>retrieval and generation are separate concerns</strong>. The LLM is only as good as what you feed it. Investing in retrieval quality pays dividends in answer quality.</p><h3>The Seven Retrieval Strategies</h3><p>This is the heart of the project. Most RAG demos use a single method. Real-world documents benefit from multiple approaches because different queries need different retrieval logic.</p><h4>1. Dense Retrieval</h4><p>Converts both the query and documents into high-dimensional vectors and finds the closest matches by cosine similarity. It understands <em>meaning</em>, not just words. So, “heart disease” and “cardiovascular conditions” will match even though they share no keywords.</p><p>This uses nomic-embed-text via Ollama, running entirely on your machine.</p><h4>2. BM25 Retrieval</h4><p>The classic information retrieval algorithm. It counts and weights term frequencies, rewarding documents that contain your exact query terms. Where dense retrieval understands concepts, BM25 excels at precision — great for technical terms, specific nutrients, or regulatory language.</p><h4>3. Hybrid Retrieval</h4><p>Combines dense and BM25 results into a single ranked list. You get semantic breadth from the vector search and keyword precision from BM25. For most general-purpose queries, hybrid outperforms either method alone.</p><h4>4. Weighted Hybrid</h4><p>A refinement of hybrid retrieval — you control how much weight to give each method. Running a query with nutritional science terminology? Lean on BM25. A conversational question about general wellness? Push more weight to dense. The balance is configurable without touching code.</p><h4>5. HyDE (Hypothetical Document Embeddings)</h4><p>The most creative approach. Instead of embedding your short query directly, HyDE first asks the LLM to <em>imagine</em> what a good answer document might look like, then embeds that hypothetical answer to search against real documents. This bridges the gap between the terse language of queries and the verbose language of documents — a surprisingly effective trick.</p><h4>6. Reciprocal Rank Fusion (RRF)</h4><p>When you run multiple retrieval methods, say dense and BM25 you end up with two separate ranked lists. The question becomes: how do you intelligently merge them into one?</p><p>RRF answers this with a simple but powerful formula. Each document gets a score based on its <em>rank position</em> in each list, not its raw similarity score. A document that appears at rank 3 in the dense results and rank 2 in the BM25 results scores higher than a document that tops only one list. The formula naturally discounts documents ranked low in any individual list while rewarding consistent performance across all of them.</p><p>The beauty of RRF is that it requires no score normalization raw similarity scores from different methods are often on incompatible scales, but ranks are always comparable. This makes it a robust, reliable fusion strategy when you don’t know in advance which retrieval method will perform best for a given query.</p><h4>7. MMR (Maximal Marginal Relevance)</h4><p>Relevance alone isn’t always enough. If you retrieve the top 5 chunks and all five say roughly the same thing from slightly different angles, you’ve wasted context window on redundant information and the LLM generates a narrow answer.</p><p>MMR solves this by balancing <strong>relevance</strong> with <strong>diversity</strong>. It selects results one at a time: the first pick is the most relevant chunk, but each subsequent pick is chosen to be both relevant <em>and</em> as different as possible from what’s already been selected. The result is a set of chunks that collectively cover more of the document space, giving the LLM a richer, more varied context to reason from.</p><h3>Why Re-ranking Changes Everything</h3><p>Retrieval gets you a candidate list. Re-ranking gets you the <em>right</em> candidate list.</p><p>Here’s the problem with standard retrieval: it scores query and documents independently. Your query becomes a vector, each document becomes a vector, and you measure distance. It’s fast, but it treats query and document in isolation.</p><p>A <strong>cross-encoder re-ranker</strong> (from BAAI — bge-reranker-base) reads the query and each candidate document <em>together</em>, the same way a human would. It can catch nuances that embedding distance misses — like whether a document actually answers the question, or just shares vocabulary with it.</p><p>The practical workflow: retrieve 10–15 candidates broadly, then re-rank to find the true top 5. Those 5 go to the LLM. The result is noticeably better answer quality, especially on complex or ambiguous questions.</p><h3>The Open-Source Stack</h3><p>Every tool in this pipeline is free and self-hosted. Here’s what each one does and why it was chosen:</p><p><strong>Milvus</strong> is the vector database. It handles storing and querying millions of embeddings with millisecond latency. It supports the hybrid search patterns this project uses natively, and it runs via Docker — one command to start.</p><p><strong>Attu</strong> is the visual management interface for Milvus. Once running, you can browse your vector collections, inspect index health, and monitor how many chunks have been ingested — all through a web UI at localhost:8000. Invaluable for debugging and understanding what your database actually contains.</p><p><strong>Ollama</strong> runs LLMs and embedding models locally. No API key, no rate limits, no data leaving your machine. The project uses nomic-embed-text:v1.5 for embeddings and qwen3 for generation — both pull with a single command.</p><p><strong>BAAI (Beijing Academy of Artificial Intelligence)</strong> provides the re-ranking model via HuggingFace. Their bge-reranker-base is compact, fast, and performs remarkably well for its size. It&#39;s what makes the re-ranking step practical on consumer hardware.</p><p><strong>LangChain</strong> handles the document processing side — loading PDFs, splitting them into chunks, and managing the ingestion pipeline.</p><p>Component Tool Where it runs Vector Database Milvus Docker (local) DB Monitoring Attu Docker (local) Embeddings nomic-embed-text (Ollama) Local LLM qwen3 (Ollama) Local Re-ranker BAAI bge-reranker-base Local Document Processing LangChain Local</p><h3>How the Code Is Organized</h3><p>The project separates ingestion, retrieval, and generation into independent modules. This matters: you can swap the LLM, change the vector DB, or add a new retrieval method without touching the rest of the system.</p><pre>src/<br>├── ingestion.py            # Load PDFs, chunk, embed, store<br>├── retrieval.py            # All five retrieval strategies<br>├── response_generation.py  # Retrieval + re-ranking + LLM<br>├── milvus_setup.py         # Database initialization<br>├── config.yml              # All parameters in one place<br>└── main.py                 # CLI entry point</pre><p>Configuration is YAML-based. Chunk size, overlap, embedding model, similarity threshold — all live in config.yml rather than scattered through the code.</p><h3>On Chunking Strategy</h3><p>One detail that doesn’t get enough attention: how you split documents matters as much as how you retrieve them.</p><p>Chunks that are too large dilute relevance — the model finds a chunk but the actual answer is one sentence in 500 words of noise. Chunks that are too small lose context — you get the sentence but not the surrounding paragraph that makes it meaningful.</p><p>The project supports two strategies: <strong>recursive character chunking</strong> (split by size with overlap between adjacent chunks) and <strong>semantic chunking</strong> (split on meaning boundaries detected by the embedding model). The experiments notebooks let you try both on your own data before committing to one.</p><h3>Getting Started</h3><p>The setup involves four moving parts: Python environment, Ollama models, Milvus via Docker, and your PDF files. The README has full step-by-step instructions — here’s the conceptual order:</p><ol><li>Clone the repo and install Python dependencies (the project uses uv for fast dependency management)</li><li>Pull the embedding model with Ollama: ollama pull nomic-embed-text:v1.5</li><li>Start Milvus and Attu: docker-compose up -d</li><li>Drop your PDFs into data/raw_pdf/ and run python main.py ingest</li><li>Start querying: python main.py rag &quot;your question here&quot; --method hybrid</li></ol><p>The CLI also has a compare command that runs all retrieval methods on the same query side by side — useful for understanding the tradeoffs in practice. There&#39;s also an interactive mode for a conversational interface.</p><p>For the complete setup commands and configuration options, refer to the <a href="https://github.com/aathikm8943/RAG-Project">repository README</a>.</p><h3>Visualizing with Attu</h3><p>One underrated part of building a RAG system is being able to <em>see</em> your data. Attu gives you that. After ingestion, you can open the Attu web UI and inspect your collection — how many vectors are stored, how they’re indexed, and even run test queries directly against Milvus before the LLM layer is involved.</p><p>This makes debugging retrieval issues much faster. If your answers are poor, the first question is usually “is retrieval working?” — and Attu answers that visually.</p><h3>What I’d Explore Next</h3><p>A few natural extensions to this pipeline:</p><p><strong>Evaluation</strong> — Adding RAGAS or a similar framework to measure retrieval precision and answer faithfulness with actual metrics, not just eyeballing results.</p><p><strong>Streaming responses</strong> — For the interactive mode, streaming LLM output token-by-token feels much more responsive than waiting for the full answer.</p><p><strong>Table and chart extraction</strong> — Nutrition documents are full of structured data. A vision model pass during ingestion could unlock that content for retrieval.</p><h3>Wrapping Up</h3><p>The core lesson from building this: <strong>retrieval is not a solved problem, and it deserves serious investment.</strong> A single dense retrieval method with a hosted LLM is easy to build but often disappoints in practice. Multiple retrieval strategies, hybrid fusion, and cross-encoder re-ranking all on open-source tools can produce results that rival expensive proprietary stacks.</p><p>Everything here runs on your own machine, costs nothing per query, and keeps your data private.</p><p>Explore the full source code at <a href="https://github.com/aathikm8943/RAG-Project">github.com/aathikm8943/RAG-Project</a>. Questions or ideas? Drop them in the comments.</p><p><em>Built with: Milvus · Attu · Ollama · BAAI · LangChain · Python</em></p><p><strong>Tags:</strong> RAG · LLM · Milvus · Ollama · Open Source · NLP · Vector Database · Machine Learning</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c12c7bba96af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Real-Time Analytics Pipeline with Kafka + Spark + Iceberg on your laptop - Complete…]]></title>
            <link>https://medium.com/@aathi8924/building-a-real-time-analytics-pipeline-with-kafka-spark-iceberg-on-your-laptop-complete-0095dd94ef0e?source=rss-607c8d5f50eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/0095dd94ef0e</guid>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[modern-data-platform]]></category>
            <category><![CDATA[data-platform-engineering]]></category>
            <dc:creator><![CDATA[Aathi]]></dc:creator>
            <pubDate>Thu, 02 Apr 2026 06:55:26 GMT</pubDate>
            <atom:updated>2026-04-02T06:58:52.532Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Real-Time Analytics Pipeline with Kafka + Spark + Iceberg on your laptop - Complete local setup and zero cloud cost.</h3><blockquote>A step-by-step guide to running Apache Kafka, Apache Spark, and Apache Iceberg entirely on your own machine using Docker — with a real analytics streaming pipeline as the working example.</blockquote><p>If you’ve Googled <em>“how to run Apache Kafka locally”</em>, <em>“Spark Iceberg local setup”</em>, or <em>“Kafka connection refused Docker”</em> — this post is for you. Most tutorials either point you at a paid cloud service or skip the hard parts that actually trip people up. This one doesn’t.</p><p>We’ll build a complete real-time streaming pipeline that ingests live NYC taxi trip data, processes it with three parallel Spark jobs, and writes results into queryable Iceberg tables — all with open-source tools, all inside Docker, all on your laptop.</p><h4>Table of contents</h4><ol><li>Why local? Why open source?</li><li>The tool stack — what each thing does</li><li>Architecture — how it all fits together</li><li>How each piece actually works</li><li>Run it yourself — 5 steps</li></ol><h3>01 — Why local? Why open source?</h3><p>Most streaming tutorials start with: “First, create an AWS account…” or “Sign up for Confluent Cloud…”. Those are great products, but they cost money, hide the plumbing, and don’t actually teach you what’s happening underneath.</p><p>The tools that power real-time data at companies like Uber, LinkedIn, and Netflix — Apache Kafka, Apache Spark, Apache Iceberg — are all fully open source. You don’t need a cloud subscription to use them. You just need Docker.</p><blockquote><strong><em>100% open source. Nothing fancy.</em></strong></blockquote><blockquote><em>Every tool in this pipeline — Kafka, Spark, Iceberg, MinIO is free, community-driven, and runs entirely on your own machine. No AWS account. No credit card. No vendor lock-in. The same stack that powers Fortune 500 data teams, running with </em><em>make up.</em></blockquote><h3>02 — What each tool does — in plain English</h3><p><strong>Apache Kafka</strong> is a message broker — think of it as a post office for your data. A producer drops off messages, Kafka holds them in a topic called events_raw1, and a consumer (Spark) picks them up when ready. They work independently, so if Spark slows down for a second, messages queue safely in Kafka and get processed when it catches up.</p><p><strong>Apache Spark</strong> is the processing engine. It reads from Kafka like reading a never-ending table that keeps growing, runs windowed aggregations every few seconds, and writes results downstream. This is called PySpark structured streaming — one of the most searched topics in data engineering right now.</p><p><strong>Apache Iceberg</strong> is a table format for data lakes. It makes your Parquet files in object storage behave like a proper database — with ACID transactions, time-travel queries, and schema evolution. Writing streaming data to raw Parquet is fragile; writing to Iceberg is not.</p><p><strong>MinIO </strong>is your personal Amazon S3. It stores the raw taxi Parquet file, and later Iceberg writes all three output tables back into it. One storage service, two roles — input and output.</p><p><strong>Docker Compose</strong> glues all five containers together on a shared internal network called iceberg_net. Containers talk to each other by name, not by IP. It&#39;s reproducible, disposable, and starts with one command.</p><h3><strong>03 — Full architecture — how it all fits together</strong></h3><p>Here’s the complete picture. Every box is a Docker container. Data enters from the top as a Parquet file in MinIO, the Kafka producer picks it up and sends it to the broker, Spark reads the stream and fans out into three parallel analytics jobs, and results land in three separate Iceberg tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*ucyVXMKP9OwjLsf6eVjaWA.png" /></figure><p>The dashed border is the Docker Compose network. Everything inside talks by container name — no IP addresses, no manual wiring. Tear it all down with make down, bring it back up fresh with make up.</p><h3><strong>04 — How each part actually works</strong></h3><p><strong>MinIO — your personal S3</strong></p><p>MinIO is an open-source object store with a full AWS S3-compatible API. Your taxi dataset — a Parquet file with 10,000+ trip records — lives here. The Kafka producer reads it row-by-row. Later, Iceberg writes all three output tables back into the same MinIO bucket. One storage service, both input and output.</p><p><strong>Apache Kafka — running without ZooKeeper (KRaft mode)</strong></p><p>This is one of the most searched setups right now: <em>“how to run Kafka without ZooKeeper”</em>. The answer is KRaft mode — Kafka’s built-in Raft consensus that replaces ZooKeeper entirely. One less container, same reliability, much simpler setup.</p><p><strong>PySpark structured streaming — reading from Kafka</strong></p><p>Spark reads the Kafka topic as a continuous stream. Every few seconds it wakes up, pulls new messages, parses the JSON schema, converts timestamps, and fans out to three parallel write streams — one per use case. This is PySpark structured streaming with Kafka as the source, exactly the pattern you’d use in production.</p><p><strong>Apache Iceberg — a data lake that behaves like a database</strong></p><p>Iceberg wraps your Parquet data files in a proper metadata layer: snapshots, manifests, and a catalog. This gives you ACID transactions (no partial writes), time-travel queries (VERSION AS OF), and schema evolution without rewriting data. An iceberg-rest sidecar container on port 8181 acts as the catalog — telling Spark where the tables are and what their schemas look like.</p><p><strong>The Makefile — your command menu</strong></p><p>Instead of memorizing long Docker and spark-submit commands, the Makefile wraps everything into single-word shortcuts. make up starts the whole cluster. make run-job submits a Spark job. make local-producer fires 10,000 events into Kafka. Think of it as a project-specific CLI</p><h3>05 — Run the full pipeline in 5 steps</h3><p>You need Docker, Python 3.13+, and about 4 GB of free RAM. That’s it — no cloud account, no paid tools.</p><ol><li>Clone the repo and install Python dependencies</li></ol><pre>git clone https://github.com/aathikm8943/Real-Time-Event-Streaming<br>cd Real-Time-Event-Streaming<br>pip install -r requirements.txt</pre><p>2. Start all five Docker containers</p><p>Wait ~30 seconds for all services to become healthy before the next step.</p><pre>make up<br>make up make logs # watch startup logs - Ctrl+C to stop following</pre><p>3. Create the three Iceberg tables</p><p>Runs a Spark job that registers the tables in the REST catalog.</p><pre>make run-init</pre><p>4. Send data and start the Spark streaming job</p><p>Open two terminals at the same time.</p><pre># Terminal 1 — produce 10,000 taxi events into Kafka<br>make local-producer<br><br># Terminal 2 — Spark reads Kafka and writes to Iceberg<br>make run-job FILE_NAME=kafka_to_spark_analytics.py</pre><p>5. Query the live Iceberg tables</p><pre>docker exec -it spark-iceberg pyspark<br><br># Inside PySpark — query any of the three tables<br>spark.sql(&quot;SELECT * FROM rest_backend.default.taxi_revenue LIMIT 5&quot;).show()</pre><blockquote><strong>Expected output:</strong> ~10,000 messages consumed at around 1,000 msgs/second. End-to-end latency under 5 seconds. Revenue numbers grouped by pickup location and 10-minute window. Runs comfortably on a MacBook or mid-range Windows laptop.</blockquote><h4>Clone it. Run it. Break it.</h4><p>Full source — Makefile, Docker Compose, Kafka producer, all PySpark notebooks, and test scripts is on GitHub. Star it if it helped you skip a frustrating afternoon of debugging.</p><p>Repository: <a href="https://github.com/aathikm8943/Real-Time-Event-Streaming">GitHub</a> | Document: <a href="https://github.com/aathikm8943/Real-Time-Event-Streaming/blob/main/README.md">Read the docs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0095dd94ef0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recent Trends in Healthcare Technology - “Bio-Printing”]]></title>
            <link>https://medium.com/@aathi8924/recent-trends-in-healthcare-technology-bio-printing-869eb8c07036?source=rss-607c8d5f50eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/869eb8c07036</guid>
            <category><![CDATA[healthcare-technology]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[healthcare-innovations]]></category>
            <category><![CDATA[biomedical-engineering]]></category>
            <category><![CDATA[bioengineering]]></category>
            <dc:creator><![CDATA[Aathi]]></dc:creator>
            <pubDate>Sun, 30 Oct 2022 02:48:01 GMT</pubDate>
            <atom:updated>2022-10-30T02:48:01.049Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*4qGkbpr1xQgzfdm4jxNbgw.png" /></figure><p><strong>Introduction:</strong> Imagine a world in which organs and tissues are printed at the touch of a button. Whenever someone suffers a severe injury or needs an organ transplant, technicians harvest cells from the patient, multiply them in a lab, and use 3D bioprinting to create a replica of the injured tissue. Then surgeons replace the old or damaged tissue with new and healthy tissue.</p><p>If this was possible, organ transplant waiting lists would dwindle to nothing. Severe burn wounds would heal much more quickly and effectively with multi-layered skin grafts printed from patients’ own tissues. Many medications with harmful side effects would no longer be necessary if injured tissue was replaced instead of treated.</p><p><strong>Bio-Printing: </strong>It’s an emerging technology for creating living tissue and organs in the lab and is one way in which biomedical engineers are exploring how to solve the soaring demand for organ transplants. While there are some similarities between 3D manufacturing and 3D bio-printing, printing organs that function in the human body requires knowledge and technology extending far beyond the typical 3D printing process.</p><p><strong>Process:</strong> Three-dimensional printing is most commonly known for expediting prototyping prosses in manufacturing. The engineers and designers create a digital model of the product using <strong>Computer Aided Software(CAD).</strong></p><p>The materials such as <strong>liquid polymers, gels, or resins</strong> are used in bio-printing. The 3D printer brings the product to life layer by layer with extraordinary precision, creating a 3D Structure for testing and scaling.</p><p><strong>BioINK: </strong>Instead of using plastics, the <strong>material used for bio-printing is known as BioINK. </strong>BioINK is called a complex mixture of <strong>natural polymers, synthetic polymers, and human cells. </strong>Scientists must then create a <strong>scaffold</strong> — i.e, the skeleton structure that mimics the structure found in living organs — upon which living cells are printed so they can grow into the organs.</p><p><strong>Conclusion: </strong>Currently no lab has been able to 3D print blood vessels and capillaries capable of sustaining an organ. However, with the promising organ. However, with the promising developments in the current researchers coupled with new insights harnessing the capacity of cells to self-organize, some experts believe that, eventually, this barrier will be overcome. Until then, the race to 3D printing is a fully functioning organ that will lead to a lesser breakthrough in the near future that will improve countless lives in the interim.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=869eb8c07036" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Principle Component Analysis(PCA)]]></title>
            <link>https://medium.com/@aathi8924/introduction-to-principle-component-analysis-pca-73b32578cd61?source=rss-607c8d5f50eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/73b32578cd61</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <dc:creator><![CDATA[Aathi]]></dc:creator>
            <pubDate>Wed, 22 Sep 2021 03:32:39 GMT</pubDate>
            <atom:updated>2021-09-22T03:32:39.678Z</atom:updated>
            <content:encoded><![CDATA[<p>Principal Component Analysis (PCA) is an unsupervised linear transformation technique that is widely used across different fields, most prominently for feature extraction and dimensionality reduction. Other popular applications of PCA include exploratory data analyses and de-noising of signals in stock market trading, and the analysis of genome data and gene expression levels in the field of bioinformatics.</p><p>PCA helps us to identify patterns in data based on the correlation between features. In a nutshell, PCA aims to find the directions of maximum variance in high-dimensional data and projects it onto a new subspace with equal or fewer dimensions than the original one.</p><p>The orthogonal axes (principal components) of the new subspace can be interpreted as the directions of maximum variance given the constraint that the new feature axes are orthogonal to each other, as illustrated in the following figure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*YnALn7n2ZzeIdT3R.jpeg" /></figure><p>In the preceding figure, <em>x1</em> and <em>x2</em> are the original feature axes, and PC1 and PC2 are the principal components.</p><p>If we use PCA for dimensionality reduction, we construct a <em>d</em> x <em>k</em>–dimensional transformation matrix <em>W</em> that allows us to map a sample vector <em>x</em> onto a new <em>k</em>–dimensional feature subspace that has fewer dimensions than the original <em>d</em>–dimensional feature space:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/233/0*fWaIHVxj5gXoazP4.png" /></figure><p>As a result of transforming the original <em>d</em>-dimensional data onto this new <em>k</em>-dimensional subspace (typically <em>k</em> ≪ <em>d</em>), the first principal component will have the largest possible variance, and all consequent principal components will have the largest variance given the constraint that these components are uncorrelated (orthogonal) to the other principal components — even if the input features are correlated, the resulting principal components will be mutually orthogonal (uncorrelated).</p><p>Note that the PCA directions are highly sensitive to data scaling, and we need to standardize the features <em>prior</em> to PCA if the features were measured on different scales and we want to assign equal importance to all features.</p><p>Before looking at the PCA algorithm for dimensionality reduction in more detail, let’s summarize the approach in a few simple steps:</p><ol><li>Standardize the <em>d</em>-dimensional dataset.</li><li>Construct the covariance matrix.</li><li>Decompose the covariance matrix into its eigenvectors and eigenvalues.</li><li>Sort the eigenvalues by decreasing order to rank the corresponding eigenvectors.</li><li>Select <em>k</em> eigenvectors which correspond to the <em>k</em> largest eigenvalues, where <em>k</em> is the dimensionality of the new feature subspace (<em>k</em> ≤ <em>d</em>).</li><li>Construct a projection matrix <em>W</em> from the “top” <em>k</em> eigenvectors.</li><li>Transform the <em>d</em>-dimensional input dataset <em>X</em> using the projection matrix <em>W</em> to obtain the new <em>k</em>-dimensional feature subspace.</li></ol><p>Let’s perform a PCA step by step, using Python as a learning exercise. Then, we will see how to perform a PCA more conveniently using scikit-learn.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73b32578cd61" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>