<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Raghuveer M on Medium]]></title>
        <description><![CDATA[Stories by Raghuveer M on Medium]]></description>
        <link>https://medium.com/@raghuveer.metla?source=rss-55d0a7dd313f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kZxI8zBI2rC_8pjY4NqRew.png</url>
            <title>Stories by Raghuveer M on Medium</title>
            <link>https://medium.com/@raghuveer.metla?source=rss-55d0a7dd313f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 03:09:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@raghuveer.metla/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your Database Knows Too Much: A Fun Guide to Data Masking in Healthcare AI]]></title>
            <link>https://medium.com/@raghuveer.metla/your-database-knows-too-much-a-fun-guide-to-data-masking-in-healthcare-ai-0bd3eaf411be?source=rss-55d0a7dd313f------2</link>
            <guid isPermaLink="false">https://medium.com/p/0bd3eaf411be</guid>
            <category><![CDATA[information-security]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-masking]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Raghuveer M]]></dc:creator>
            <pubDate>Mon, 11 May 2026 22:41:00 GMT</pubDate>
            <atom:updated>2026-05-11T22:43:59.129Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/335/0*BdMxRhKL1xb6vU_D.jpg" /></figure><p>Imagine this.</p><p>You’re a developer debugging a staging database at 11:47 PM.</p><p>You run a query.</p><p>And suddenly…</p><pre>SELECT * FROM patients LIMIT 10;</pre><p>Boom.</p><p>Real patient names. Dates of birth. Medical details. Addresses.</p><p>Congratulations.<br> You just turned your test environment into a legal landmine.</p><p>Welcome to the world of <strong>PHI</strong>, <strong>PII</strong>, compliance nightmares, and the art of pretending patient data is fake while still making systems useful.</p><p>This is where <strong>data masking</strong> enters like Batman.</p><h3>First: What Is PHI and Why Does Everyone Panic About It?</h3><p>Healthcare systems contain extremely sensitive data.</p><p>Things like:</p><ul><li>Patient names</li><li>Date of birth</li><li>Medical record numbers</li><li>Genetic sequencing identifiers</li><li>Clinical notes</li><li>Insurance details</li></ul><p>This falls under:</p><ul><li><strong>PII</strong> → Personally Identifiable Information</li><li><strong>PHI</strong> → Protected Health Information</li></ul><p>And regulators like:</p><ul><li>HIPAA</li><li>GDPR</li><li>Security auditors</li><li>Compliance teams</li><li>Angry lawyers</li></ul><p>…all care deeply about how this data moves around.</p><p>Because one leaked spreadsheet can become:</p><blockquote><em>“How a healthcare startup accidentally exposed 1.2 million patient records.”</em></blockquote><p>Not exactly the Medium article you want trending.</p><h3>The Core Problem</h3><p>Developers need realistic data to:</p><ul><li>test applications</li><li>debug ETL pipelines</li><li>build dashboards</li><li>train ML models</li><li>validate workflows</li></ul><p>But they <strong>should not</strong> see real patient identities.</p><p>So engineering teams face a weird challenge:</p><blockquote><em>“How do we keep the data useful without exposing actual humans?”</em></blockquote><p>That’s the entire purpose of <strong>data masking and de-identification</strong>.</p><h3>Think of Data Masking Like Movie Stunt Doubles</h3><p>The system still <em>looks</em> real.</p><p>But it’s not the real actor.</p><p>Example:</p><p>OriginalMaskedJohn SmithMichael CarterDOB: 1982–03–12DOB: 1979–07–18MRN: 8829182MRN: TKX-8821SSN: 555–22–1111NULL</p><p>Your application still works.</p><p>Your analytics still run.</p><p>But nobody accidentally exposes a real patient.</p><h3>The Golden Rule: Minimum Necessary Access</h3><p>Healthcare security has one brutally simple philosophy:</p><blockquote><em>Access only what you absolutely need.</em></blockquote><p>If you’re building charts…</p><p>You probably don’t need:</p><ul><li>patient names</li><li>home addresses</li><li>SSNs</li><li>exact DOBs</li></ul><p>You may only need:</p><ul><li>age range</li><li>diagnosis category</li><li>sequencing result</li><li>anonymized sample ID</li></ul><p>That principle alone prevents massive damage.</p><p>Most breaches happen because:</p><ul><li>engineers had too much access</li><li>dev databases copied production blindly</li><li>“temporary” exports became permanent</li></ul><p>Temporary is the most permanent word in engineering.</p><h3>The Main Types of Data Masking</h3><p>Here’s where things get interesting.</p><p>Different masking techniques solve different problems.</p><h3>1️⃣ Substitution</h3><p>Replace sensitive values with fake-but-valid values.</p><p>Example:</p><pre>Alice Johnson → Emma Williams</pre><p>Perfect for:</p><ul><li>development</li><li>demos</li><li>QA testing</li></ul><p>Your UI still behaves realistically.</p><h3>2️⃣ Shuffling</h3><p>Take existing values and rearrange them randomly.</p><p>Example:</p><p>Original AgeShuffled Age456262313145</p><p>Useful for:</p><ul><li>statistical testing</li><li>preserving distributions</li></ul><p>Basically:</p><blockquote><em>“Keep the math realistic, hide the humans.”</em></blockquote><h3>3️⃣ Nulling / Blanking</h3><p>The simplest method.</p><pre>Patient Name → NULL</pre><p>Or:</p><pre>***MASKED***</pre><p>Useful when the field serves no purpose outside production.</p><h3>4️⃣ Hashing (The One-Way Trapdoor)</h3><p>Hashing converts data into irreversible strings.</p><p>Example:</p><pre>patient123<br>↓<br>8f14e45fceea167a5a36dedd4bea2543</pre><p>You can compare hashes.</p><p>But you can’t easily reverse them.</p><p>Perfect for:</p><ul><li>anonymized analytics</li><li>long-term identity protection</li></ul><p>Think of hashing like shredding a paper into microscopic confetti.</p><h3>5️⃣ Tokenization</h3><p>This one is clever.</p><p>Instead of destroying the value, you replace it with a token.</p><pre>Patient ID: 92811<br>↓<br>TOKEN_X72P91</pre><p>The real mapping lives inside a secure vault.</p><p>Authorized systems can reverse it.</p><p>Everyone else sees nonsense.</p><p>This is heavily used in:</p><ul><li>healthcare</li><li>finance</li><li>payment systems</li></ul><h3>6️⃣ Pseudonymization</h3><p>This is the “spy movie alias” approach.</p><p>Instead of:</p><pre>John Smith</pre><p>You get:</p><pre>PATIENT_4421</pre><p>The system still tracks the same patient consistently across datasets.</p><p>But without revealing the real identity.</p><p>Very useful for:</p><ul><li>analytics</li><li>ML training</li><li>longitudinal studies</li></ul><h3>What a Real Masking Pipeline Looks Like</h3><p>A mature healthcare pipeline usually follows something like this:</p><h3>Step 1: Identify Sensitive Fields</h3><p>Engineers first map:</p><ul><li>PHI</li><li>PII</li><li>internal-only fields</li><li>public-safe fields</li></ul><p>This is harder than people think.</p><p>Even a sample ID can become PHI if it links back to a patient.</p><h3>Step 2 :Classification</h3><p>Data gets tagged:</p><p>ClassificationMeaningPHIHighly restrictedInternalLimited company accessPublicSafe to share</p><p>Without classification, chaos begins.</p><h3>Step 3 :Apply Masking</h3><p>Usually happens:</p><ul><li>inside ETL jobs</li><li>via SQL transformations</li><li>Python utilities</li><li>Lambda functions</li><li>Airflow/Dagster workflows</li></ul><p>Good teams automate this.</p><p>Bad teams export CSVs manually.</p><p>Guess which ones create incidents.</p><h3>Step 4 :Validation</h3><p>Security and QA verify:</p><ul><li>Can the data be reverse engineered?</li><li>Did any PHI leak accidentally?</li><li>Are joins still functioning?</li><li>Did masking break analytics?</li></ul><p>This step matters more than the masking itself.</p><p>Because weak masking is fake security.</p><h3>Step 5:Controlled Access</h3><p>Even masked datasets should still require:</p><ul><li>MFA</li><li>VPN</li><li>RBAC permissions</li><li>audit logging</li></ul><p>Because security isn’t one wall.</p><p>It’s layers.</p><p>Like onions.</p><p>Or Kubernetes.</p><p>Mostly onions.</p><h3>The Modern Cloud Reality</h3><p>Today’s healthcare systems usually spread data across:</p><ul><li>S3 buckets</li><li>PostgreSQL warehouses</li><li>dashboards</li><li>ML pipelines</li><li>event queues</li><li>analytics platforms</li></ul><p>Which creates a dangerous temptation:</p><blockquote><em>“Let’s just copy production data into staging quickly.”</em></blockquote><p>That sentence has probably caused more compliance incidents than actual hackers.</p><h3>A Smart Engineering Pattern</h3><p>Strong teams separate data into layers:</p><pre>RAW LAYER<br>↓<br>PROCESSED LAYER<br>↓<br>MASKED LAYER<br>↓<br>ANALYTICS / DASHBOARDS</pre><p>Never mix them.</p><p>Ever.</p><p>The moment raw PHI casually appears in dashboards or notebooks, you’ve already lost control.</p><h3>The Best Engineering Habit Nobody Talks About</h3><p>Keep masking logic modular.</p><p>Instead of random SQL everywhere:</p><pre>mask_patient_name()<br>mask_dob()<br>mask_mrn()</pre><p>Or:</p><pre>masking_rules.json</pre><p>Why?</p><p>Because regulations evolve.</p><p>And if your masking logic is scattered across:</p><ul><li>17 SQL scripts</li><li>4 Lambdas</li><li>2 Airflow DAGs</li><li>one “temporary” notebook from 2023</li></ul><p>…you’re in for pain.</p><h3>The Biggest Mistake Startups Make</h3><p>Early-stage startups often think:</p><blockquote><em>“We’re small. Security can come later.”</em></blockquote><p>That’s backwards.</p><p>The earlier you establish:</p><ul><li>data classification</li><li>masking standards</li><li>audit logging</li><li>access policies</li></ul><p>…the easier your future becomes.</p><p>Because retrofitting compliance into a chaotic architecture is brutal.</p><p>It’s like adding seatbelts after building the car.</p><h3>🧪 A Simple Mental Model</h3><p>Think of healthcare data systems like a chemistry lab.</p><p>Raw PHI is radioactive material.</p><p>You don’t casually:</p><ul><li>duplicate it</li><li>email it</li><li>dump it into staging</li><li>share screenshots</li><li>upload it to random notebooks</li></ul><p>You isolate it.</p><p>Track it.</p><p>Audit it.</p><p>And transform it before others touch it.</p><h3>Final Thought</h3><p>The scariest security incidents usually don’t happen because of elite hackers.</p><p>They happen because:</p><ul><li>someone copied production data into QA</li><li>an engineer downloaded a CSV locally</li><li>a dashboard exposed too much</li><li>masking was “planned later”</li></ul><p>Real security is boring.</p><p>It’s discipline.<br> Structure.<br> Processes.<br> Guardrails.</p><p>But those boring systems are exactly what protect real human beings behind the data.</p><p>And in healthcare engineering…</p><p>That’s the entire job.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0bd3eaf411be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG Alone Is Not Smart Enough: Why You Still Need GANs]]></title>
            <link>https://medium.com/@raghuveer.metla/rag-alone-is-not-smart-enough-why-you-still-need-gans-e834146a5004?source=rss-55d0a7dd313f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e834146a5004</guid>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[chatbot-development]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <dc:creator><![CDATA[Raghuveer M]]></dc:creator>
            <pubDate>Thu, 18 Dec 2025 16:49:28 GMT</pubDate>
            <atom:updated>2025-12-18T16:52:04.969Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’re in tech right now and <strong>don’t know what a RAG system is</strong>, you’re either lying or on vacation.</p><p>Everyone is building RAG:</p><ul><li>RAG for documents</li><li>RAG for internal knowledge bases</li><li>RAG for “please don’t hallucinate, GPT”</li></ul><p>And honestly? RAG is great.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/0*D9V_cWMjGMuuDSuA" /><figcaption>Copyright: ©New Line Cinema/Courtesy Everett Collection</figcaption></figure><p>But here’s the uncomfortable truth nobody says out loud:</p><p><strong>RAG is retrieval. Not intelligence.</strong></p><p>That’s where <em>Generative Adversarial Networks (GANs)</em> quietly enter the room.</p><p>I recently had a “wait… I should know this” moment when my manager casually mentioned GANs. If that’s you right now, welcome. Let’s fix it.</p><h3>So… What the Heck Is a GAN? (In Plain English)</h3><p>A <strong>Generative Adversarial Network (GAN)</strong> is just <strong>two AI models competing with each other</strong>.</p><p>That’s it.</p><p>No scary math. No PhD required.</p><p>Think of it like this:</p><ul><li>One model <strong>creates fake stuff</strong></li><li>The other model <strong>tries to catch the fake</strong></li></ul><p>They train together.<br> They argue.<br> They get better.</p><p>Over time, the fake becomes so good that even the judge struggles to tell the difference.</p><p>That’s a GAN.</p><h3>The Two Players (Meet the Rivals)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LGPGTkW90P973qgLAL3MBQ.png" /></figure><h3>1. The Generator -&gt; The Faker</h3><ul><li>Creates fake data that looks real</li><li>Could be images, text, lab signals, genomic patterns, reports</li></ul><p>Its job: <em>“Can I fool you?”</em></p><h3>2. The Discriminator -&gt; The Critic</h3><ul><li>Looks at data and asks: “Real or fake?”</li><li>Gives feedback like:<br> <em>“Nice try… but this doesn’t look real.”</em></li></ul><p>Its job: <em>“I’m not buying it.”</em></p><p>The Generator improves.<br> The Discriminator improves.<br> They push each other.</p><p>Eventually, the fake looks real enough to pass.</p><p>That’s the magic <strong>competition, not instruction</strong>.</p><h3>Why GANs Are So Powerful (and a Little Brutal)</h3><p>GANs don’t learn by being told the right answer.</p><p>They learn by:</p><ul><li>Failing</li><li>Getting judged</li><li>Adjusting</li><li>Repeating this loop thousands of times</li></ul><p>This is called <strong>adversarial learning</strong>.</p><p>Because of this, GANs are amazing at:</p><ul><li>Creating realistic images or signals</li><li>Filling in missing data</li><li>Learning patterns when labeled data is limited</li></ul><p>GANs don’t memorize examples.<br> They <strong>learn what “real” looks like</strong>.</p><h3>Why This Actually Matters (Especially in Science)</h3><p>Now let’s talk about the real world , science, healthcare, regulated systems.</p><h3>Problem #1: Data Is Scarce and Sensitive</h3><p>In biotech, pharma, health care and diagnostics:</p><ul><li>Data is expensive</li><li>Data is private</li><li>Data is regulated</li><li>Data is often incomplete</li></ul><p>GANs can create <strong>synthetic data</strong> that:</p><ul><li>Looks statistically real</li><li>Doesn’t expose patient or sample data</li><li>Helps train and test systems safely</li></ul><p>That’s huge.</p><h3>Problem #2: You Can’t Afford Hallucinations</h3><p>In sensitive domains:</p><ul><li>Wrong answers aren’t “oops”</li><li>They’re compliance issues</li><li>FDA issues</li><li>Patient safety issues</li></ul><p>GANs help by:</p><ul><li>Learning what <em>normal</em> data should look like</li><li>Flagging anomalies</li><li>Stress-testing pipelines with realistic edge cases</li></ul><h3>Let’s Be Honest About RAG</h3><p>RAG is great. You absolutely need it.</p><p>But it has limits.</p><h3>What RAG Does Well</h3><ul><li>Pulls trusted documents</li><li>Grounds answers in real sources</li><li>Reduces hallucinations</li></ul><h3>What RAG Does Not Do</h3><ul><li>It doesn’t learn data patterns</li><li>It doesn’t generate new realistic samples</li><li>It doesn’t judge whether something <em>makes sense</em></li></ul><p>RAG answers questions about <strong>what you already know</strong>.</p><p>GANs help explore <strong>what could exist</strong>.</p><p>Different jobs.</p><h3>GAN + RAG = The Underrated Power Combo</h3><p>This is where things get interesting.</p><h3>Example: A Scientific Knowledge System</h3><p><strong>RAG</strong></p><ul><li>Retrieves papers, web information, docs</li></ul><p><strong>GAN</strong></p><ul><li>Simulates rare cases</li><li>Generates synthetic experiments</li></ul><p>Together:</p><ul><li>RAG handles <strong>truth</strong></li><li>GAN handles <strong>reality</strong></li></ul><p>That’s how you move from <em>chatbot</em> to <em>intelligent system</em>.</p><h3>One Simple Analogy (Because Stories Stick)</h3><p>Imagine training a chef.</p><ul><li><strong>RAG</strong> gives them cookbooks</li><li><strong>GAN</strong> lets them cook and get critiqued</li></ul><p>Would you trust a chef who only reads recipes but never cooks?</p><p>Exactly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e834146a5004" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[️ From Ferrari to Vectors: The Simple Math Behind Vector Databases]]></title>
            <link>https://medium.com/@raghuveer.metla/%EF%B8%8F-from-ferrari-to-vectors-the-simple-math-behind-vector-databases-35d13183ce69?source=rss-55d0a7dd313f------2</link>
            <guid isPermaLink="false">https://medium.com/p/35d13183ce69</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Raghuveer M]]></dc:creator>
            <pubDate>Thu, 02 Oct 2025 20:10:05 GMT</pubDate>
            <atom:updated>2025-10-02T20:10:05.446Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WMPvTSs-SIcBj8Un" /></figure><h3>Intro</h3><ul><li>Most articles make vector databases sound like rocket science.</li><li>But you just need a handful of math fundamentals.</li><li>Let’s learn using cars as our example 🚗</li></ul><blockquote><em>“What do cars, fruit, and songs have in common? You can describe each features in them with </em>numbers<em>. And once you do, you can build a </em>vector database<em> to find things that feel similar — even if the names don’t match.”</em></blockquote><p>We’ll use <strong>cars</strong> as our running example. By the end, you’ll understand what “cosine similarity” and “dot product” mean — without drowning in equations.</p><h3>1. Vectors = vibes as numbers</h3><p>Imagine you’re describing cars not by brand but by <strong>features that matter</strong>:</p><ul><li>Speed (0–10)</li><li>Luxury (0–10)</li><li>Off-road capability (0–10)</li></ul><p>So:</p><ul><li>Ferrari → [10, 9, 2]</li><li>Jeep Wrangler → [5, 4, 10]</li><li>Toyota Corolla → [5, 5, 3]</li></ul><p>👉 A <strong>vector</strong> is just “the car’s vibe captured as numbers.”</p><blockquote><em>Why it matters: This is what “embeddings” are in AI — your text, image, or audio turned into a vector.</em></blockquote><h3>2. Length = intensity of the vibe</h3><p>If you draw Ferrari’s [10,9,2] as an arrow from the origin (0,0,0), it’s a <em>long arrow</em>.<br> Corolla [5,5,3] is shorter.</p><p>👉 The <strong>length</strong> (magnitude) is like “how extreme” the car is overall.</p><ul><li>Ferrari screams FAST + LUXURY.</li><li>Corolla whispers “I’m balanced.”</li></ul><p>In math: longer vector = more intensity.</p><h3>3. Dot Product = steering wheel alignment</h3><p>Now imagine you and I are driving side by side. If both steering wheels point the same way, we’re aligned.</p><p>That’s the <strong>dot product</strong>: a measure of alignment.</p><ul><li>Ferrari [10,9,2] vs Lamborghini [9,10,3] → high dot product (both pointing toward “fast luxury”).</li><li>Ferrari [10,9,2] vs Jeep [5,4,10] → low dot product (one screams racetrack, one screams mud trails).</li></ul><p>👉 Dot product = “Do these cars want to go on the same road?”</p><h3>4. Cosine Similarity = angle closeness</h3><p>But wait — what if Ferrari has [10,9,2] and Tesla Roadster has [9,9,2]?</p><ul><li>Ferrari’s numbers are slightly bigger.</li><li>But both arrows point in <em>almost the same direction</em>.</li></ul><p>Cosine similarity zooms in on the <strong>angle</strong>, not the size.</p><ul><li>Small angle → very similar vibes.</li><li>Big angle → totally different meaning.</li></ul><p>👉 Cosine similarity is why search engines don’t care about exact words — they care about the “angle of meaning.”</p><h3>5. Distance = how far apart they’re parked</h3><p>Picture a giant 3D parking lot with axes for speed, luxury, and off-road.</p><ul><li>Ferrari is on one corner.</li><li>Jeep is way up in the off-road section.</li><li>Corolla is somewhere in the middle.</li></ul><p>The <strong>Euclidean distance</strong> between them is how many steps you’d walk to get from one car to another.</p><p>👉 Cosine = “Do they point the same way?”<br> 👉 Distance = “How far apart are they?”</p><p>Both are used in vector databases depending on what you care about.</p><h3>6. High-Dimensional Garage</h3><p>Cars are easy with 3 features. But imagine describing a car with <strong>500 features</strong> (torque, seating comfort, emissions, trunk size…).</p><p>That’s what AI embeddings do:</p><ul><li>Text → 768 features.</li><li>Images → 1024+ features.</li></ul><p>You can’t picture it, but the same rules apply:</p><ul><li>Length, dot product, cosine, distance.</li></ul><p>👉 Think of it as a garage with <strong>thousands of invisible dimensions.</strong></p><h3>7. Vector Database = garage of vibes</h3><p>Traditional DB:</p><pre>SELECT * FROM cars WHERE name = &#39;Ferrari&#39;;</pre><p>Vector DB:</p><pre>SELECT * FROM cars WHERE cosine_similarity(vibe, [10,9,2]) &gt; 0.9;</pre><p>So if you search “like Ferrari,” it gives you Lamborghini and Tesla Roadster.<br> If you search “like Jeep but not extreme off-road,” you might get a Toyota 4Runner.</p><p>👉 That’s the power: it finds <em>similar meaning</em>, not exact matches.</p><h3>🚦 Real-World Use Cases</h3><ul><li><strong>Spotify</strong>: Find songs with the same “feel” as your playlist.</li><li><strong>Google Photos</strong>: Search “beach” and it finds sand + ocean even if the word “beach” never appears.</li><li><strong>ChatGPT memory</strong>: Store facts as embeddings and recall “similar questions” later.</li></ul><h3>🧠 Final Cheat Sheet (for your desk)</h3><ul><li><strong>Vector</strong> = thing → numbers</li><li><strong>Length</strong> = intensity of vibe</li><li><strong>Dot product</strong> = alignment of vibes</li><li><strong>Cosine similarity</strong> = angle closeness</li><li><strong>Distance</strong> = closeness in the parking lot</li><li><strong>Vector DB</strong> = garage of vibes</li></ul><h3>Wrap-up Story</h3><blockquote><em>A vector database is like asking your car buddy, </em>“I love Ferraris, what else should I check out?”<em> Instead of giving you just cars with the same name, he points to Lambos and Teslas — because they live in the same corner of the garage.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35d13183ce69" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Hotel Transylvania Explains Microsoft Entra ID and OAuth: A Beginner’s Guide to Authentication]]></title>
            <link>https://medium.com/@raghuveer.metla/how-hotel-transylvania-explains-microsoft-entra-id-and-oauth-a-beginners-guide-to-authentication-dd5b43309d35?source=rss-55d0a7dd313f------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd5b43309d35</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Raghuveer M]]></dc:creator>
            <pubDate>Mon, 29 Sep 2025 21:26:44 GMT</pubDate>
            <atom:updated>2025-09-29T21:26:44.322Z</atom:updated>
            <content:encoded><![CDATA[<p>A beginner-friendly guide to modern authentication using a fun, simple Hotel Transylvania-style story</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*WW2_NRVtMjuCrkYm0hAQ7Q.png" /></figure><h3>🏨 Welcome to the Haunted Hotel (a.k.a. Secure App)</h3><p>Imagine a haunted hotel where only invited monsters (users) can enter. It’s protected by spooky guards, magical guest passes, and supernatural tokens that make sure only the right creatures get access to the right rooms.</p><p>This haunted hotel is your <strong>secure app</strong>. And the creepy castle security system? That’s <strong>OAuth 2.0 and OpenID Connect (OIDC)</strong> working together.</p><h3>👻 Meet the Gatekeeper: The Identity Provider</h3><p>At the entrance of the hotel stands a magical bouncer called the <strong>Identity Provider (IdP)</strong>. In real life, this is something like <strong>Microsoft Entra ID</strong>, <strong>Google</strong>, or <strong>Auth0</strong>.</p><p>When Dracula (you, the user) wants to enter, the gatekeeper checks if he’s on the guest list. If he is, the gatekeeper issues magical keys (tokens) that prove Dracula is legit.</p><h3>🔑 Types of Magical Keys (Tokens)</h3><p>Once Dracula is inside, he gets:</p><ul><li><strong>ID Token</strong> 📅: His profile — name, photo, species, favorite blood type, etc.</li><li><strong>Access Token</strong> 🚪: A room key that says which parts of the hotel he can enter (spa, dungeon, lab, etc.)</li><li><strong>Refresh Token</strong> 🔄: A way to request a new room key when the old one expires.</li></ul><p>These keys are called <strong>JWTs (JSON Web Tokens)</strong>. Think of them as scrolls with Dracula’s credentials magically signed and sealed.</p><h3>📄 OAuth 2.0 vs. OpenID Connect (OIDC)</h3><ul><li><strong>OAuth 2.0</strong> is the <em>protocol</em> that controls the security guard’s rules: Who gets in, for how long, and where they can go.</li><li><strong>OIDC</strong> is an <em>extension</em> that adds identity: Who exactly is this monster? What are they allowed to do?</li></ul><p>Together, they make sure no imposter vampire gets into the VIP section.</p><h3>🚨 Roles, Groups, and Room Access</h3><p>In the hotel, some monsters are just guests. Others are staff. A few are VIPs.</p><p>That’s where <strong>roles and groups</strong> come in:</p><ul><li>Frankenstein might be in the “Maintenance” group — access to power room</li><li>Mavis might be “Admin” — access to everything</li><li>Zombies are “View-only” — can roam the lobby but not enter rooms</li></ul><p>When your app checks the Access Token, it looks at these roles to decide what each user can do.</p><h3>🌍 What This Means in the Real World</h3><p>When you sign in to an app using your Microsoft, Google, or Apple account:</p><ul><li>You’re redirected to their <strong>Identity Provider</strong></li><li>You log in</li><li>You’re sent back to the app with <strong>tokens</strong></li><li>The app uses those tokens to verify your identity and decide what you can access</li></ul><h3>🚀 Why It Matters</h3><p>If you’re a developer, tester, data engineer, or even a security team member, understanding how <strong>OAuth 2.0</strong>, <strong>OIDC</strong>, and <strong>tokens</strong> work helps you:</p><ul><li>Protect apps from unauthorized access</li><li>Diagnose auth/token-related bugs</li><li>Know how roles and permissions flow across systems</li></ul><h3>🎃 Final Thought</h3><p>So next time you hear about Bearer tokens, Entra IDs, or SSO, just imagine the Haunted Hotel:</p><ul><li>The Identity Provider checks you in</li><li>Tokens are your room keys</li><li>OAuth &amp; OIDC are the rulebooks</li></ul><p>And you? You’re just trying to enjoy a secure stay without waking the ghosts.</p><p>Happy haunting (and authenticating)! 👻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd5b43309d35" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Software Engineering Terms Every Engineer Should Know — Part 2]]></title>
            <link>https://medium.com/@raghuveer.metla/software-engineering-terms-every-engineer-should-know-part-2-bc48c24ea232?source=rss-55d0a7dd313f------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc48c24ea232</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[unit-testing]]></category>
            <category><![CDATA[automation-testing]]></category>
            <category><![CDATA[software-testing]]></category>
            <dc:creator><![CDATA[Raghuveer M]]></dc:creator>
            <pubDate>Thu, 18 Sep 2025 19:56:41 GMT</pubDate>
            <atom:updated>2025-09-18T19:56:41.574Z</atom:updated>
            <content:encoded><![CDATA[<h3>Software Engineering Terms Every Engineer Should Know — Part 2</h3><p><strong>Focus Area</strong>: Testing Fundamentals</p><h3>Why Testing Matters</h3><p>Software isn’t real until it’s tested. For regulated systems (like FDA software) or mission-critical apps (finance, healthcare, aerospace), testing isn’t just about catching bugs — it’s about <strong>proving compliance, safety, and reliability</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6QmsMJOEeMQOoNYs" /></figure><p>Here are the core testing terms every engineer should know, explained simply — and the way leaders use them in reviews.</p><h3>Core Testing Terms</h3><h3>1. Unit Testing</h3><ul><li><strong>Simple:</strong> The smallest test — checks one function or module.</li><li><strong>Why It Matters:</strong> Ensures building blocks are correct before integration.</li><li><strong>Example:</strong> <em>“We need strong unit test coverage at the module level to reduce downstream defects.”</em></li></ul><h3>2. Integration Testing</h3><ul><li><strong>Simple:</strong> Checks if modules work together.</li><li><strong>Example:</strong> Data pipeline component + database service working in sync.</li><li><em>“Integration tests validate continuity across subsystems.”</em></li></ul><h3>3. System Testing</h3><ul><li><strong>Simple:</strong> End-to-end testing of the entire system.</li><li><strong>Example:</strong> From input dataset → through pipeline → to final report.</li><li><em>“System tests must mirror the production workflow.”</em></li></ul><h3>4. Regression Testing</h3><ul><li><strong>Simple:</strong> Ensures old features still work after new changes.</li><li><strong>Why It Matters:</strong> FDA and regulated industries require proof that fixes don’t break validated workflows.</li><li><strong>Leader-Speak:</strong> <em>“Regression coverage ensures stability across releases.”</em></li></ul><h3>5. Acceptance Testing</h3><ul><li><strong>Simple:</strong> Confirms the system meets stakeholder expectations.</li><li><strong>Why It Matters:</strong> This is often the “go/no-go” test before release.</li><li><strong>Example:</strong> <em>“Acceptance criteria must be linked directly to requirements for traceability.”</em></li></ul><h3>6. Smoke Testing</h3><ul><li><strong>Simple:</strong> Quick, shallow test to see if the system basically runs.</li><li><strong>Example:</strong> <em>“Smoke tests should run automatically after each build to catch obvious issues.”</em></li></ul><h3>7. Sanity Testing</h3><ul><li><strong>Simple:</strong> Narrow check after small changes, to confirm core functionality still works.</li><li><strong>Example:</strong> <em>“Sanity testing ensures incremental changes don’t destabilize critical workflows.”</em></li></ul><h3>8. Performance Testing</h3><ul><li><strong>Simple:</strong> Measures speed, scalability, and responsiveness.</li><li><strong>Example:</strong> <em>“Performance tests must validate NFRs like runtime limits and throughput.”</em></li></ul><h3>9. Load Testing</h3><ul><li><strong>Simple:</strong> See how the system behaves under expected user/data load.</li><li><strong>Example:</strong> <em>“Load testing is key to ensuring the pipeline handles production volumes.”</em></li></ul><h3>10. Stress Testing</h3><ul><li><strong>Simple:</strong> Push the system beyond normal limits to see how it fails.</li><li><strong>Example:</strong> <em>“Stress testing helps us design graceful recovery under failure.”</em></li></ul><h3>11. UAT (User Acceptance Testing)</h3><ul><li><strong>Simple:</strong> Stakeholders test if the system works for their real-world needs.</li><li><strong>Example:</strong> <em>“UAT sign-off is critical for FDA/Product submissions — it proves stakeholder alignment.”</em></li></ul><h3>12. Test Automation</h3><ul><li><strong>Simple:</strong> Using scripts/tools to run tests automatically.</li><li><strong>Why It Matters:</strong> Saves time, improves consistency, enables CI/CD pipelines.</li><li><strong>Example:</strong> <em>“Automation is key for reproducibility and audit readiness.”</em></li></ul><h3>13. Traceability in Testing</h3><ul><li><strong>Simple:</strong> Every requirement should have a test case linked to it.</li><li><strong>Why It Matters:</strong> Regulatory audits often start by asking: <em>“Show me where this requirement is tested.”</em></li><li><strong>Example:</strong> <em>“Our traceability matrix must map requirements to test cases with evidence of execution.”</em></li></ul><h3>14. Test Coverage</h3><ul><li><strong>Simple:</strong> The percentage of requirements or code covered by tests.</li><li><strong>Example:</strong> <em>“Coverage metrics are useful, but we must focus on risk-based testing, not just numbers.”</em></li></ul><h3>15. Test Plan / Test Protocol</h3><ul><li><strong>Simple:</strong> A structured document describing what will be tested, how, and by whom.</li><li><strong>Example:</strong> <em>“Test plans should align directly with requirements and regulatory commitments.”</em></li></ul><h3>📝 Recap</h3><p>In <strong>Part 2</strong>, we covered the fundamentals and the advanced practices of software testing:</p><ul><li><strong>Core Levels:</strong> Unit, Integration, System, Regression, Acceptance.</li><li><strong>Specialized Types:</strong> Smoke, Sanity, Performance, Load, Stress, UAT.</li><li><strong>Practices &amp; Docs:</strong> Automation, Traceability, Coverage, Test Plans.</li></ul><p>Together, these terms give you a complete testing vocabulary to <strong>speak confidently as both an engineer and a leader</strong> especially in compliance-driven environments.</p><h3>Coming Next (Part 3)</h3><p>We’ll explore <strong>Risk Management &amp; Traceability in Regulated Systems</strong> — including Verification vs Validation, Risk Controls, and SOUP (third-party software).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc48c24ea232" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Software Engineering Terms Every Engineer Should Know - Part 1]]></title>
            <link>https://medium.com/@raghuveer.metla/software-engineering-terms-every-engineer-should-know-part-1-55a570312c43?source=rss-55d0a7dd313f------2</link>
            <guid isPermaLink="false">https://medium.com/p/55a570312c43</guid>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Raghuveer M]]></dc:creator>
            <pubDate>Thu, 18 Sep 2025 19:28:58 GMT</pubDate>
            <atom:updated>2025-09-18T19:30:31.979Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Focus Area → SDLC, Requirements, Architecture, and Design</strong></p><h3>Why This Series?</h3><p>While building applications. Your vocabulary matters as much as your technical skills. This series breaks down <strong>core software engineering terms</strong> in plain English for design reviews, audits, and cross-team discussions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I2n2M0VvuCsDKMHe" /></figure><h3>1. SDLC (Software Development Lifecycle)</h3><ul><li><strong>Simple Definition:</strong> The step-by-step process of building software: Requirements → Design → Implementation → Testing → Release → Maintenance.</li><li><strong>Why It Matters:</strong> Regulated industries require you to show how every phase was followed.</li><li><strong>Example:</strong></li><li><em>“We need to ensure each SDLC phase has proper documentation and traceability.”</em></li></ul><h3>2. Requirements</h3><ul><li><strong>Functional Requirements (FRs):</strong> What the system should do. Example: <em>“System shall generate a report from input dataset.”</em></li><li><strong>Non-Functional Requirements (NFRs):</strong> How well it should perform. Example: <em>“System must process a run in under 8 hours.”</em></li><li><strong>Why It Matters:</strong> Bad requirements = failed projects. Good requirements are <strong>clear, testable, and traceable</strong>.</li><li><strong>Example:</strong></li><li><em>“Our FRs are clear, but NFRs like performance and reliability need stronger coverage.”</em></li></ul><h3>3. Architecture</h3><ul><li><strong>Simple Definition:</strong> The big picture — components and how they interact.</li><li><strong>Example:</strong> IOT Sensors → Data Pipeline → Analysis Engine → Reporting UI.</li><li><strong>Why It Matters:</strong> Architecture drives clarity and scalability. If architecture docs miss a flow (e.g., pooling sheet or config ownership), teams get stuck.</li><li><strong>Example:</strong></li><li><em>“The architecture shows the flow, but continuity between the pooling sheet and downstream analysis is missing.”</em></li></ul><h3>4. Detailed Design</h3><ul><li><strong>Simple Definition:</strong> Zoom in on each component and explain how it works inside.</li><li><strong>Example:</strong> Analysis Engine = Data Cleaning → Curation → Filtering.</li><li><strong>Why It Matters:</strong> Without detailed design, developers improvise, QA can’t test properly, and auditors see “gaps.”</li><li><strong>Example:</strong></li><li><em>“Subsystem design must include processing details and interface contracts.”</em></li></ul><h3>5. Stakeholders</h3><ul><li><strong>Simple:</strong> People who care about or use the system (users, managers, regulators).</li><li><strong>Example :</strong> <em>“Requirements must be stakeholder-driven and traceable to business value.”</em></li></ul><h3>6. Use Case / User Story</h3><ul><li><strong>Simple:</strong> A description of how a user interacts with the system.</li><li><strong>Example:</strong> <em>“Every requirement should map to at least one user story or use case.”</em></li></ul><h3>7. Acceptance Criteria</h3><ul><li><strong>Simple:</strong> Conditions that must be met for a requirement to be “done.”</li><li><strong>Example:</strong> <em>“Clear acceptance criteria reduce ambiguity and make validation easier.”</em></li></ul><h3>8. Constraints</h3><ul><li><strong>Simple:</strong> Limits the system must respect (e.g., must run on HPC cluster, must use PostgreSQL).</li><li><strong>Example:</strong> <em>“Constraints shape design decisions and must be documented explicitly.”</em></li></ul><h3>9. Assumptions &amp; Dependencies</h3><ul><li><strong>Simple:</strong> Things you assume are true, and things you rely on.</li><li><strong>Example:</strong> <em>“Unstated assumptions are risks; dependencies must be tracked.”</em></li></ul><h3>10. System Context Diagram</h3><ul><li><strong>Simple:</strong> A high-level diagram showing the system and external actors (users, other systems).</li><li><strong>Example:</strong> <em>“A system context diagram clarifies boundaries and interfaces upfront.”</em></li></ul><h3>11. High-Level vs Low-Level Design (HLD vs LLD)</h3><ul><li><strong>Simple:</strong></li><li>HLD = Architecture, big picture.</li><li>LLD = Detailed design, internal logic.</li><li><strong>Example:</strong> <em>“We should separate HLD for leadership review and LLD for implementation detail.”</em></li></ul><h3>📝 Recap</h3><p>In <strong>Part 1</strong>, we covered:</p><ul><li><strong>SDLC</strong> = the full lifecycle roadmap.</li><li><strong>Requirements</strong> = what the system does (functional) + how well it must do it (non-functional).</li><li><strong>Stakeholders</strong> = the people who care about or use the system.</li><li><strong>Use Cases / User Stories</strong> = how users interact with the system in real-world terms.</li><li><strong>Acceptance Criteria</strong> = the definition of “done” for each requirement.</li><li><strong>Constraints</strong> = technical or regulatory limits that shape design decisions.</li><li><strong>Assumptions &amp; Dependencies</strong> = what we assume to be true and what our system relies on.</li><li><strong>Architecture</strong> = the big-picture view of components and their interactions.</li><li><strong>System Context Diagram</strong> = a snapshot of the system and its external actors.</li><li><strong>Detailed Design (HLD vs LLD)</strong> = the zoom-in view, from high-level design to low-level logic</li></ul><p>Mastering these concepts are very crucial and gives you the foundation to <strong>read design docs like an ideal software engineer </strong>and spot gaps</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55a570312c43" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From NGS Sequencer to Science: The Journey of Your DNA Reads]]></title>
            <link>https://medium.com/@raghuveer.metla/from-ngs-sequencer-to-science-the-journey-of-your-dna-reads-afa7f1f64a0d?source=rss-55d0a7dd313f------2</link>
            <guid isPermaLink="false">https://medium.com/p/afa7f1f64a0d</guid>
            <category><![CDATA[ngs-sequencing]]></category>
            <category><![CDATA[genomics]]></category>
            <category><![CDATA[genome-sequencing]]></category>
            <category><![CDATA[biotechnology]]></category>
            <category><![CDATA[dna-sequencing]]></category>
            <dc:creator><![CDATA[Raghuveer M]]></dc:creator>
            <pubDate>Thu, 14 Aug 2025 23:29:01 GMT</pubDate>
            <atom:updated>2025-08-14T23:29:01.429Z</atom:updated>
            <content:encoded><![CDATA[<p>Science behind Genome Sequencer to Analysis: How things work!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BUEegjer5oYjOGpI" /><figcaption>ref: <a href="https://howtotrainyourdragon.fandom.com/wiki/Twinsanity">https://howtotrainyourdragon.fandom.com/wiki/Twinsanity</a></figcaption></figure><blockquote>If <strong>DNA sequencing</strong> was a photoshoot, your <strong>sequencer</strong> is the world’s most precise camera and your <strong>DNA</strong> is the celebrity. But instead of snapping just one picture, it takes two shots of each DNA fragment,</blockquote><p>one from the <strong>front</strong> (Read 1) and one from the <strong>back</strong> (Read 2).</p><p>These two files : <strong>R1</strong> and <strong>R2</strong> are a <strong>paired-end set</strong>. They belong together. They’re like left and right shoes, peanut butter and jelly, or Batman and Robin. Separate, they’re incomplete. Together, they give scientists a full view of your genome.</p><h4>Step 1: The Sequencer’s Output</h4><p>Your Illumina sequencer doesn’t just create one file per sample. It spits out multiple files because it splits the run into <strong>lanes : </strong>think of them as different conveyor belts all reading different parts of the same sample.</p><p>For one sample, you might get:</p><pre>Sample1_R1_L001.fastq.gz  <br>Sample1_R1_L002.fastq.gz  <br>Sample1_R2_L001.fastq.gz  <br>Sample1_R2_L002.fastq.gz</pre><h4>Step 2: Merging the Right Way</h4><p>To simplify processing, scientists <strong>merge lanes </strong>but only for the <strong>same read direction</strong>.</p><p><em>Good merge:</em></p><pre>Sample_R1_L001 + Sample_R1_L002 → Sample_R1_001.fastq.gz  <br>Sample_R2_L001 + Sample_R2_L002 → Sample_R2_001.fastq.gz</pre><p><em>Bad merge (Don’t do this!):</em></p><pre>Sample_R1_L001 + Sample_R2_L001 → ❌ Broken paired-end data</pre><h4>Why You Shouldn’t Merge R1 and R2</h4><p>R1 and R2 are from opposite ends of a DNA fragment. Merge them into one file and you lose <strong>pairing information </strong>the software no longer knows which read goes with which.<br> It’s like mixing up all the questions and answers in a quiz completely useless for alignment.</p><h4>Step 3: Uploading for Analysis</h4><p>Once you have two merged files per sample (R1 and R2), they’re ready for pipeline Analysis<br> Think of DRAGEN as the ultra-fast DNA detective that pieces together your reads into meaningful genomic information : mutations, structural changes, and more.</p><h4>Quick Recap</h4><ul><li>Illumina sequencer → produces <strong>multiple FASTQ files</strong> per sample from different lanes.</li><li><strong>Merge same read direction</strong> (all R1s together, all R2s together).</li><li><strong>Never merge R1 with R2</strong>.</li><li>Upload the clean paired-end data to your analysis pipeline and let the magic happen.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afa7f1f64a0d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Science Series: Types of Scientific Studies — Why Scientists Time Travel]]></title>
            <link>https://medium.com/@raghuveer.metla/types-of-scientific-studies-why-scientists-time-travel-9c07e7eaa31b?source=rss-55d0a7dd313f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c07e7eaa31b</guid>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[clinical-trials]]></category>
            <category><![CDATA[biology]]></category>
            <category><![CDATA[ngs-sequencing]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Raghuveer M]]></dc:creator>
            <pubDate>Wed, 09 Jul 2025 15:53:41 GMT</pubDate>
            <atom:updated>2025-07-09T16:07:59.976Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RwnX2rDs9_DViaYw.jpg" /></figure><h4>Introduction</h4><p>At some point in a meeting, someone says:<br> <em>“This is a retrospective study.”</em><br> Or you hear words like <strong>prospective</strong></p><p>If you’re not from a clinical background, you might be wondering:<br> <strong>Are we talking about science or making a documentary?</strong></p><p>This quick guide will help you understand the different types of scientific studies, especially useful if you’re working around genomics, NGS, or biopharma</p><h3>🕰️ First Split: Past vs Future</h3><p>The first big divide is:</p><p><strong>Retrospective Studies (PAST)🕵️‍♂️</strong></p><ul><li>Look backwards in time</li><li>Use existing data (e.g., past patient NGS reports)</li><li>Faster and cheaper</li><li>Great for identifying trends</li><li>Example: Reviewing cancer mutations from last year’s samples</li></ul><p><strong>Prospective Studies (Future)🔭</strong></p><ul><li>Look forward in time</li><li>Collect data as events unfold</li><li>More effort and resources needed</li><li>Example: Enrolling new patients and tracking mutation outcomes</li></ul><h3>🧬 Where Does NGS Fit In?</h3><p>In Biotech companies, these terms pop up when processing <strong>Next-Gen Sequencing (NGS)</strong> samples.</p><ul><li>If you’re analyzing archived samples collected over the past 3 years → <strong>Retrospective study</strong></li><li>If you’re designing a study to collect and sequence patient samples <strong>after</strong> enrollment → <strong>Prospective study</strong></li></ul><p>This matters because it affects:</p><ul><li>How human blood or Urine samples are labeled</li><li>What metadata is available for analysis</li><li>Timelines</li></ul><h3>🔍 More Study Types at a Glance</h3><p>Here’s a cheat-sheet style list for the other study types that sneak into project names:</p><ul><li><strong>Retrospective</strong> = We already have the data</li><li><strong>Prospective</strong> = Let’s plan and collect it going forward</li><li><strong>Observational = </strong>Just watching, no intervention</li><li><strong>Interventional = </strong>Researchers apply treatments/tests</li><li><strong>Cross-sectional = </strong>Snapshot in time</li><li><strong>Longitudinal = </strong>Repeated observations over time</li><li><strong>Case-Control = </strong>Compare patients with a condition vs. without</li><li><strong>Cohort Study = </strong>Follow a group over time, track outcomes</li></ul><h3>💡 Why It Matters for You</h3><ul><li>If you’re an <strong>engineer working with Biological Samples</strong>, this helps you understand <strong>data freshness</strong>, file naming, and pipeline design.</li><li>If you’re in <strong>product or ops</strong>, it helps with <strong>planning timelines</strong> and <strong>compliance</strong>.</li><li>If you’re a <strong>designer, PM, or creative</strong>, it helps you <strong>translate the science</strong> into clear visuals and strategy.</li></ul><h3>So next time you hear someone say “It’s a retrospective project,”<br> you’ll know exactly what kind of time travel they’re on.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c07e7eaa31b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How SLURM Runs the Show: The Traffic Cop of High-Performance Computing (HPC)]]></title>
            <link>https://medium.com/@raghuveer.metla/how-slurm-runs-the-show-the-traffic-cop-of-high-performance-computing-hpc-285fef497b9e?source=rss-55d0a7dd313f------2</link>
            <guid isPermaLink="false">https://medium.com/p/285fef497b9e</guid>
            <category><![CDATA[sequencing]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[biotechnology]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[slurm]]></category>
            <dc:creator><![CDATA[Raghuveer M]]></dc:creator>
            <pubDate>Tue, 08 Jul 2025 17:14:48 GMT</pubDate>
            <atom:updated>2025-07-08T17:14:48.738Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Ever wonder how hundreds of scientists, researchers, or AI engineers can run their monster-sized jobs on shared clusters without stepping on each other’s toes?</blockquote><p>Enter <strong>SLURM</strong> the <strong>ultimate job scheduler</strong> for High-Performance Computing.<br>But let’s skip the jargon for a second…</p><p>SLURM is like a Summer <strong>Camp manager</strong>. decides <strong>who gets to work on which task</strong>, when, and how.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/0*4Ufymn_SPG4rOZ_2" /></figure><h3>🏕️ Analogy: SLURM = Camping Trip Organizer</h3><p>Imagine you’re organizing a group camping trip:</p><h4>🎯 You (the user)</h4><p>You want to plan a bunch of activities (cook, hike, fish) and get help from camp guides (nodes) to do them.</p><h4>🏕️ SLURM = Camp Manager</h4><p>SLURM is the <strong>camp manager</strong>. You submit a checklist (job script) saying:</p><ul><li>What you want to do</li><li>How long it’ll take</li><li>How many people (cores) you need</li><li>Special needs (gear = memory, etc.)</li></ul><h4>⛺ Nodes = Camp Cabins</h4><p>Each <strong>node</strong> is a cabin with a certain number of beds (CPUs). SLURM decides which cabin is free to run your activity.</p><h4>📅 sbatch = Your Booking Request</h4><p>When you sbatch, you&#39;re handing your activity list to the manager. He’ll find a time and cabin for you.</p><h4>👥 sinfo/squeue = Camp Bulletin Board</h4><p>Use sinfo to see all cabins, and squeue to see what activities are running or queued up.</p><h4>🧹 scancel = Cancelling a Camp Activity</h4><p>Did plans change? Use scancel to remove your booking.</p><h3><strong>Architecture</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*Jm9sdIEycGX1aDZMX9xLNg.png" /></figure><h3>Key Terms</h3><p><strong>Job : </strong>A task you want SLURM to run</p><p><strong>Job Script : </strong>A .sh file with your commands and SLURM config</p><p><strong>Node : </strong>A physical machine/server in the cluster</p><p><strong>Core / CPU : </strong>Processing unit inside the node</p><p><strong>Partition : </strong>A group of nodes (like a queue or section)</p><h3>Basic SLURM Commands Cheat Sheet</h3><p><em>Submit a job</em> :sbatch job.sh</p><p><em>View job queue </em>:squeue</p><p><em>View your jobs : </em>squeue -u $USER</p><p><em>Cancel a job :</em>scancel &lt;jobid&gt;</p><p><em>Job status : </em>scontrol show job &lt;jobid&gt;</p><p><em>Check node info :</em>sinfo</p><p><em>Interactive job :</em>srun --pty bash or salloc</p><h4>📝 Basic job.sh SLURM Script Template</h4><pre><br>#!/bin/bash<br>#SBATCH --job-name=my_job<br>#SBATCH --output=logs/my_job.out<br>#SBATCH --error=logs/my_job.err<br>#SBATCH --ntasks=1<br>#SBATCH --cpus-per-task=4<br>#SBATCH --mem=8G<br>#SBATCH --time=01:00:00<br><br>module load python/3.9<br>python my_script.py</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=285fef497b9e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Many Chefs Fit in a CPU? A Beginner’s Guide to Multithreading, Cores, and Chaos]]></title>
            <link>https://python.plainenglish.io/how-many-chefs-fit-in-a-cpu-a-beginners-guide-to-multithreading-cores-and-chaos-4cf7e7aa8e58?source=rss-55d0a7dd313f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4cf7e7aa8e58</guid>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[multithreading]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Raghuveer M]]></dc:creator>
            <pubDate>Wed, 18 Jun 2025 16:31:23 GMT</pubDate>
            <atom:updated>2025-06-22T14:59:43.569Z</atom:updated>
            <content:encoded><![CDATA[<p><em>From kitchen metaphors to Python code, making threads and nodes easy for anyone.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/0*u6e_3YSVDBRUSYXZ.jpg" /></figure><h3>Why You Should Care About Multithreading</h3><p>Ever wondered why your laptop slows down when you open too many tabs, or why your Python script takes forever to finish? Behind the scenes, your computer is juggling tasks like a restaurant kitchen juggles food orders.</p><p>Whether you’re an engineer, or someone curious about how software works faster (or slower), understanding <strong>multithreading</strong> will make you better at building systems.</p><h3>What Is a CPU Core?</h3><p>Imagine your <strong>CPU</strong> is a <strong>kitchen</strong>. Inside this kitchen, the people actually cooking are the <strong>cores</strong>.</p><ul><li>A <strong>single-core CPU</strong> is like having <strong>one chef</strong>. All orders (tasks) are made one at a time.</li><li>A <strong>quad-core CPU</strong> is like having <strong>four chefs</strong>. Four meals (tasks) can be prepared in parallel.</li></ul><blockquote><em>“More cores = more chefs = more dishes cooked at the same time.”</em></blockquote><p>Each core has a certain speed (clock speed), which tells you how <strong>fast</strong> a chef can work.</p><h3>Threads: = Tasks for Chefs</h3><p>Now that we have our chefs (cores), let’s give them work.</p><p>A <strong>thread</strong> is like a <strong>task</strong> assigned to a chef:</p><ul><li>“Make spaghetti”</li><li>“Grill chicken”</li><li>“Chop vegetables”</li></ul><p>Your CPU runs <strong>threads</strong> inside a program (a process). So your restaurant (the program) can send multiple orders (threads) to the chefs.</p><p>But here’s the catch:</p><ul><li>If you have <strong>more threads than cores</strong>, chefs must <strong>juggle</strong>.</li><li>This juggling is called <strong>context switching</strong> and it has a cost (like washing hands between dishes).</li></ul><h3>Concurrency vs. Parallelism</h3><p>Let’s make this clearer:</p><p><strong>Concurrency </strong>-&gt; One chef juggling multiple dishes. <em>Tasks progress, not necessarily at once</em></p><p><strong>Parallelism </strong>-&gt; Multiple chefs cooking at once. <em>Tasks run literally at the same time</em></p><blockquote><em>Python threading allows concurrency. For true parallelism, you need multiple cores </em>and<em> sometimes multiprocessing.</em></blockquote><h3>What About Infrastructure?</h3><p>Here’s where it gets real: in data engineering, you deal with <strong>machines, nodes, clusters, and jobs</strong>.</p><ul><li>A <strong>node</strong> is like one restaurant (machine) in a food court.</li><li>Each <strong>node</strong> has <strong>cores</strong> (chefs)</li><li>A <strong>cluster</strong> is a group of restaurants, each with their own staff.</li></ul><p>In tools like <strong>Apache Spark</strong> or <strong>Airflow</strong>:</p><ul><li>Spark breaks big jobs into tasks and sends them to cores across the cluster</li><li>Airflow can schedule multiple DAG tasks, but how many run <em>at once</em> depends on how many cores your server has</li></ul><blockquote><em>“Threads need cores. Tasks need resources. Without enough cores, your data pipeline waits in line like hungry customers.”</em></blockquote><h3>Let’s Add Some Python</h3><p>Let’s say you want to simulate making 5 dishes. Without threads:</p><pre>import time<br><br>def make_dish(name):<br>    print(f&quot;Starting {name}&quot;)<br>    time.sleep(2)<br>    print(f&quot;Finished {name}&quot;)<br><br>for dish in [&quot;Pasta&quot;, &quot;Pizza&quot;, &quot;Salad&quot;, &quot;Soup&quot;, &quot;Cake&quot;]:<br>    make_dish(dish)</pre><p>This takes 10 seconds (5 x 2).</p><p>Now, with multithreading:</p><pre>import threading<br>import time<br><br>def make_dish(name):<br>    print(f&quot;Starting {name}&quot;)<br>    time.sleep(2)<br>    print(f&quot;Finished {name}&quot;)<br><br>threads = []<br><br>for dish in [&quot;Pasta&quot;, &quot;Pizza&quot;, &quot;Salad&quot;, &quot;Soup&quot;, &quot;Cake&quot;]:<br>    t = threading.Thread(target=make_dish, args=(dish,))<br>    threads.append(t)<br>    t.start()<br><br>for t in threads:<br>    t.join()</pre><p>Still takes ~2 seconds total (if you have enough cores).</p><h3>The Problems with Threads</h3><p>Like in any kitchen, problems happen:</p><ul><li><strong>Race Conditions</strong>: Two threads try to use the same pan at the same time</li><li><strong>Deadlocks</strong>: One thread waits forever for another to finish</li><li><strong>Python GIL</strong>: Only one chef (thread) is allowed to cook at a time, no matter how many you have (limitation for CPU-bound code)</li></ul><p>Use <strong>Locks</strong> to avoid collisions:</p><pre>from threading import Lock<br>lock = Lock()<br><br>pydef safe_task():<br>    with lock:<br>        # do something critical</pre><p>For CPU-heavy tasks in Python? Use Multiprocessing, not threads.</p><h3>Summary: Bringing It All Together</h3><ul><li><strong>Core = Chef</strong></li><li><strong>Thread = Task</strong></li><li><strong>CPU = Kitchen</strong></li><li><strong>Node = Restaurant</strong></li><li><strong>Cluster = Food Court</strong></li></ul><p>Use multithreading when:</p><ul><li>You’re doing I/O-bound tasks (e.g., downloading files)</li><li>You want to keep things responsive</li></ul><p>Avoid or rethink it when:</p><ul><li>You’re doing heavy computation (try multiprocessing or Spark)</li></ul><h3>Final Thought</h3><p>Next time your Python script feels slow or your Airflow chokes, think like a restaurant owner:</p><ul><li>Do you have enough chefs?</li><li>Are they juggling too much?</li><li>Should you open another restaurant?</li></ul><p>Understanding threads and cores will make you a smarter system builder.</p><p>Happy cooking!</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4cf7e7aa8e58" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/how-many-chefs-fit-in-a-cpu-a-beginners-guide-to-multithreading-cores-and-chaos-4cf7e7aa8e58">How Many Chefs Fit in a CPU? A Beginner’s Guide to Multithreading, Cores, and Chaos</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>