<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by MOHAMMED MOSEENA on Medium]]></title>
        <description><![CDATA[Stories by MOHAMMED MOSEENA on Medium]]></description>
        <link>https://medium.com/@mdmoseena22?source=rss-72a4fb8cbdaf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Di2I4DCwgKBqUTTp</url>
            <title>Stories by MOHAMMED MOSEENA on Medium</title>
            <link>https://medium.com/@mdmoseena22?source=rss-72a4fb8cbdaf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:09:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mdmoseena22/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI Demystified: From AI to Agents — A Clear Map of How Everything Fits Together]]></title>
            <link>https://medium.com/@mdmoseena22/ai-demystified-from-ai-to-agents-a-clear-map-of-how-everything-fits-together-a040f09f303a?source=rss-72a4fb8cbdaf------2</link>
            <guid isPermaLink="false">https://medium.com/p/a040f09f303a</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[MOHAMMED MOSEENA]]></dc:creator>
            <pubDate>Thu, 23 Jul 2026 04:11:14 GMT</pubDate>
            <atom:updated>2026-07-27T16:25:00.016Z</atom:updated>
            <content:encoded><![CDATA[<h3>AI Demystified: From AI to Agents — A Clear Map of How Everything Fits Together</h3><p>This article is for anyone who uses AI tools daily but can’t clearly<br>explain how an LLM differs from an Agent. Minimal math. No code. Maximum clarity.</p><p>Everyone throws around “AI,” “ML,” “LLM,” and “Agent” interchangeably. Here’s the actual hierarchy and how each piece connects.</p><p>I took an AI fluency workshop and realized most confusion comes from<br> one thing: people use these terms as synonyms when they’re actually<br> a nested hierarchy. Here’s the map I wish I had on day one.</p><pre>Artificial Intelligence (AI)<br>│<br>│<br>├── Traditional Machine Learning (ML) ← PREDICTION from structured data<br>│   │<br>│   ├── Supervised Learning<br>│   │   ├── Classification → &quot;Is this email spam or not?&quot; (Gmail spam filter)<br>│   │   └── Regression → &quot;What will this house sell for?&quot; (Zillow estimates)<br>│   │<br>│   ├── Unsupervised Learning<br>│   │   ├── Clustering → &quot;Group customers by buying behavior&quot; (Netflix user segments)<br>│   │   └── Anomaly Detection → &quot;Flag unusual credit card transactions&quot; (fraud detection)<br>│   │<br>│   └── Reinforcement Learning<br>        Fun fact: Reinforcement Learning also appears in LLM training — RLHF is how models learn to be helpful and safe<br>│       └── &quot;Learn to win by trial &amp; error&quot; (AlphaGo, self-driving cars, robot walking)<br>│<br>│<br>├── Deep Learning (Neural Networks) ← PREDICTION + UNDERSTANDING from complex data<br>│   │                                  (does what traditional ML does, but handles<br>│   │                                   unstructured data: images, text, audio, video)<br>│   │<br>│   ├── CNNs (Convolutional Neural Networks)<br>│   │   └── &quot;What&#39;s in this image?&quot; (Google Photos face recognition, medical X-ray diagnosis)<br>│   │<br>│   ├── RNNs (Recurrent Neural Networks)<br>│   │   └── &quot;What comes next in this sequence?&quot; (stock price forecasting, speech recognition)<br>│   │<br>│   ├── GANs (Generative Adversarial Networks)<br>│   │   └── &quot;Generate realistic fake images&quot; (deepfakes, style transfer)<br>│   │<br>│   ├── Diffusion Models<br>│   │   └── &quot;Generate images from text descriptions&quot; (DALL-E, Midjourney, Stable Diffusion)<br>│   │<br>│   └── Transformers ← breakthrough architecture (2017)<br>│       │   &quot;Attend to ALL parts of input simultaneously&quot;, Transformers power more than just LLMs<br>            (e.g., Vision Transformers, audio models), but LLMs are their most prominent application.<br>│       │<br>│       └── Large Language Models (LLMs)<br>│           ├── GPT-4, Claude, Llama, Gemini<br>│           ├── Core ability: predict next token<br>│           └── &quot;Write code, summarize docs, translate languages, answer questions&quot;<br>│<br>│<br>└── Generative AI ← NOT a separate branch. It&#39;s an APPLICATION LAYER<br>    │               that USES the architectures above (LLMs, Diffusion, etc.)<br>    │<br>    ├── Text/Code → uses LLMs (ChatGPT writing essays, Copilot generating code)<br>    ├── Images → uses Diffusion Models (Midjourney creating art from &quot;a cat on Mars&quot;)<br>    ├── Audio → uses audio models (ElevenLabs cloning voices, Suno generating music)<br><br>    └── Video → uses multimodal models (Sora generating video from text prompts)</pre><p>Note: Technically, Deep Learning is a subset of Machine Learning (AI ⊃ ML ⊃ Deep Learning), not a separate branch. It’s drawn as a sibling here because in practice, “traditional ML” and “deep learning” involve different tools, workflows, and problem types — but the nesting above is a simplification for clarity, not the formal taxonomy.</p><pre>Traditional ML          Deep Learning              Generative AI<br>─────────────          ─────────────              ─────────────<br>PREDICTS from          PREDICTS from              CREATES new<br>structured data        complex/unstructured       content that<br>(tables, numbers)      data (images, text,        didn&#39;t exist<br>                       audio, video)              before<br><br>&quot;What will happen?&quot;    &quot;What is this?&quot; /          &quot;Make something new&quot;<br>                       &quot;What will happen?&quot;<br><br>House price: $450K     This X-ray: pneumonia      Write me a poem about...<br>Churn risk: 73%        This video: a cat          Generate an image of...<br>Next quarter: $2.1M    Next word: &quot;the&quot;           Compose a song that...<br>```<br><br>## Why Deep Learning &gt; Traditional ML for complex data:<br><br>```<br>Traditional ML needs:  YOU hand-craft features (pixel 1, pixel 2... doesn&#39;t work)<br>Deep Learning:         LEARNS features automatically from raw data<br><br>Image of a cat:<br>  Traditional ML → you manually define &quot;has pointy ears, whiskers...&quot; → fragile<br>  Deep Learning  → feed raw pixels, network LEARNS what &quot;cat&quot; means → robust</pre><pre><br>┌─────────────────────────────────────────────────────┐<br>│                 ARTIFICIAL INTELLIGENCE              │<br>│                                                     │<br>│   ┌─────────────────────────────────────────────┐   │<br>│   │            MACHINE LEARNING                 │   │<br>│   │                                             │   │<br>│   │   ┌─────────────────────────────────────┐   │   │<br>│   │   │          DEEP LEARNING              │   │   │<br>│   │   │                                     │   │   │<br>│   │   │   ┌─────────────────────────────┐   │   │   │<br>│   │   │   │        TRANSFORMERS         │   │   │   │<br>│   │   │   │                             │   │   │   │<br>│   │   │   │   ┌─────────────────────┐   │   │   │   │<br>│   │   │   │   │        LLMs         │   │   │   │   │<br>│   │   │   │   └─────────────────────┘   │   │   │   │<br>│   │   │   └─────────────────────────────┘   │   │   │<br>│   │   └─────────────────────────────────────┘   │   │<br>│   └─────────────────────────────────────────────┘   │<br>└─────────────────────────────────────────────────────┘<br><br>         ▲ uses (LLMs as foundation)<br>         │<br>┌────────┴────────────────────────────────────────────┐<br>│              GENERATIVE AI                           │<br>│  (Creates new content using LLMs &amp; other models)    │<br>└─────────────────────────────────────────────────────┘<br><br>         ▲ built on top of (Generative AI capabilities)<br>         │<br>┌────────┴────────────────────────────────────────────┐<br>│                 AI AGENTS                            │<br>│  (Autonomous systems that plan, reason, &amp; act       │<br>│   using GenAI + tools + memory + orchestration)     │<br>└─────────────────────────────────────────────────────┘</pre><h4>What People Mix Up</h4><pre>TERM              WHAT IT ACTUALLY IS                        WHAT PEOPLE THINK IT IS<br>─────────────     ─────────────────────────────────         ─────────────────────────<br>AI                Broad field (60+ years old)                &quot;ChatGPT&quot;<br>Machine Learning  Subset of AI (learns from data)           Same as AI<br>Deep Learning     ML using neural networks                  Same as ML<br>LLM              A specific type of deep learning model     Same as AI<br>Generative AI    AI that CREATES new content                Same as LLM<br>Agent            System: LLM + Tools + Autonomy             Same as LLM<br>ChatGPT/Claude   Agentic apps                               &quot;The AI&quot; / &quot;The LLM&quot;                       <br>                 (spectrum: light→full agent)</pre><p>Note: ChatGPT/Claude use LLMs as their brain, but the product you interact with includes tools, memory, and orchestration — making them agentic apps, not raw LLMs.</p><h4>TRADITIONAL ML vs LLM — FUNDAMENTALLY DIFFERENT</h4><pre>┌─────────────────────────────────────┬─────────────────────────────────────┐<br>│         TRADITIONAL ML              │              LLM                    │<br>├─────────────────────────────────────┼─────────────────────────────────────┤<br>│ Purpose: PREDICT a value            │ Purpose: GENERATE new content       │<br>│ Input:   Structured features        │ Input:   Natural language (any)     │<br>│ Output:  Number or Category         │ Output:  Open-ended text            │<br>│ Example: Predict house price        │ Example: Explain gravity to a kid   │<br>└─────────────────────────────────────┴─────────────────────────────────────┘</pre><p>Traditional ML: Features → [ Model ] → NUMBER or CATEGORY<br>Answers: “What WILL happen?”</p><p>LLM: Prompt → [ Model ] → NEW TEXT / CONTENT<br>Answers: “What SHOULD I say / create?”</p><h4>What an LLM Actually Does</h4><pre>Text → Tokens → Vectors → Attention → Predict Next Token<br> │<br> (repeat for each token)</pre><p>“unbelievable” → [“un”, “believ”, “able”] (3 tokens)<br>“Hello” → [“Hello”] (1 token)</p><p>→ Token = sub-word piece (~100K fixed pieces that build ANY word)<br> → Vector = token as numbers (so math can happen)<br> →Context Window = everything LLM sees right now (typically tens of thousands to over a million tokens, depending on the model)</p><h4>HOW INFERENCE WORKS — THE CORE DIFFERENCE</h4><p><strong>TRADITIONAL ML INFERENCE:</strong></p><p>[features] ──→ Model ──→ Answer ✅ DONE. One pass. Finished.</p><p>Example:<br> [age=30, income=50K, credit=720] ──→ Model ──→ “approved”</p><p>• ONE input → ONE computation → ONE answer → stop<br> • Input is small and fixed size (e.g., 50 numbers)<br> • Input is fresh each time (no memory of past inputs)<br> • Cost: constant — same computation regardless of history</p><p><strong>LLM INFERENCE:</strong></p><p>[context window] ──→ Model ──→ “The” (token 1)<br> [context window + “The”] ──→ Model ──→ “capital” (token 2)<br> [context window + “The capital”] ──→ Model ──→ “of” (token 3)<br> [context window + “The capital of”] ──→ Model ──→ “France” (token 4)<br> [context window + “The capital of France”]──→ Model ──→ “is” (token 5)<br> [context window + “…France is”] ──→ Model ──→ “Paris” (token 6)<br> [context window + “…France is Paris”] ──→ Model ──→ “.” (token 7)<br> ✅ DONE (7 passes)</p><p>• Generates ONE token per pass (not the whole answer at once)<br> • Attends over the entire context window on EVERY pass (via cached keys/values, not full recomputation)<br> • Input GROWS — every new token adds to what must be re-read<br> • A 500-word response = ~650–700 tokens = ~650–700 separate inference passes<br> (this is why longer responses cost more and take longer to generate)</p><h4>WHY THIS CREATES A CONTEXT WINDOW LIMIT</h4><p>The Attention Mechanism (how LLMs understand relationships):</p><p>Every token must “look at” every other token to understand context.<br> This is O(n²) — quadratic growth in computation.</p><pre><br>  ┌──────────────────────┬────────────────────────────────────────────┐<br>  │ Context Size         │ Attention Calculations <br>  |                    | (during initial processing │/ prefil)<br>  ├──────────────────────┼─────────────────────────────────────────── ─┤<br>  │ 1,000 tokens         │ 1,000 × 1,000     = 1 Million               │<br>  │ 10,000 tokens        │ 10,000 × 10,000   = 100 Million             │<br>  │ 200,000 tokens       │ 200,000 × 200,000 = 40 Billion              │<br>  └──────────────────────┴──────────────────────────────────────────────┘</pre><p>This happens FOR EVERY SINGLE TOKEN generated. (Production systems cache past keys/values so this isn’t literally recomputed from scratch each time, but the attention cost still scales with context length — which is the reason the limit exists.)</p><p>During generation, KV caching means each new token only computes attention against all previous tokens (O(n) per token) — but the total cost still scales quadratically with context length, which is why the limit exists.</p><p>Infinite context = infinite computation = physically impossible.</p><p>Therefore: context window MUST have a fixed limit.</p><p>The Consequence:</p><pre><br>  ┌─────────────────────────────────────────────┐<br>  │         CONTEXT WINDOW <br>          (varies by model — 32K to 1M+)        │<br>  │                                             │<br>  │  System prompt (instructions)               │<br>  │  Tool definitions                           │<br>  │  Conversation history (your messages + AI)  │<br>  │  Tool results (file contents, query output) │<br>  │                                             │<br>  │  ↑ ALL of this is re-read on EVERY token    │<br>  │                                             │<br>  │  FULL? → oldest parts get DROPPED           │<br>  │           → LLM literally &quot;forgets&quot; them    │<br>  └─────────────────────────────────────────────┘</pre><p>Traditional ML: fixed input → one pass → done → No memory. No growing cost. No limit needed.</p><p>LLM: growing input → re-read ALL → per EVERY token → O(n²) → Input accumulates. Cost explodes. Limit is mandatory.</p><p>This is why:<br> • Long conversations degrade (context fills up)<br> • Old messages get forgotten (dropped to make room)<br> • Tools/prompts compete for space (everything shares one window)<br> • Context management is a skill (not just a technical detail)</p><h4>LLMs Do NOT Calculate — They Predict</h4><p>You ask: “What’s the average of 23, 45, 67?”</p><p>LLM: “The average is 45” ← NOT computed, just the most PROBABLE next token (might be right by coincidence, might be wrong)</p><p>FIX: LLM writes code → external tool EXECUTES it → real computation</p><p>Note: “Reasoning” or “thinking” models (which generate intermediate step-by-step reasoning before the final answer) can improve raw math accuracy somewhat — working through a problem step by step reduces errors versus guessing the answer directly. But this is still prediction, just chained across more steps. It doesn’t execute anything, so it’s not a substitute for a real tool when correctness matters.</p><p>So if LLMs can’t calculate, can’t execute, can’t access systems…<br>how do tools like ChatGPT actually DO things? That’s where Agents come in.</p><h4>LLM vs Agent — The Critical Distinction</h4><pre>┌────────────────────────────────┐     ┌────────────────────────────────────┐<br>│           LLM (alone)          │     │             AGENT                  │<br>├────────────────────────────────┤     ├────────────────────────────────────┤<br>│ You ask → it responds          │     │ You ask → it PLANS + ACTS         │<br>│ Text in → Text out             │     │ Multi-step, self-directed         │<br>│ CAN&#39;T execute anything         │     │ Reads files, runs code, queries   │<br>│ CAN&#39;T access external systems  │     │ Connects to databases, APIs       │<br>│ CAN&#39;T verify its own answers   │     │ Verifies via tools                │<br>│ Single turn                    │     │ Loops until task complete          │<br>│                                │     │                                    │<br>│ Like a brain in a jar          │     │ Brain + hands + eyes + legs        │<br>└────────────────────────────────┘     └────────────────────────────────────┘</pre><p>AGENT = LLM + Tools + Instructions + Autonomy</p><h4>Agent Architecture: The 3 Layers</h4><pre>┌─────────────────────────────────┐<br>│  APP / Orchestrator             │  ← The waiter (carries plates)<br>│  • Loads tools description<br>|    into the context.<br>│  • Triggers tool execution      │<br>│  • Feeds results back to LLM    │<br>├─────────────────────────────────┤<br>│  LLM (the Brain)                │  ← Decides (TEXT ONLY output)<br>│  • Sees available tools         │     CANNOT execute anything itself<br>│  • Decides which to call        │<br>│  • Generates response           │<br>├─────────────────────────────────┤<br>│  TOOLS (the Hands)              │  ← Does actual work<br>│  • Runs code, queries DBs       │<br>│  • Reads/writes files           │<br>│  • Returns results              │<br>└─────────────────────────────────┘</pre><h4>The Agent Loop:</h4><pre>You ask<br>  → App sends to LLM<br>    → LLM decides: &quot;I need to call tool X with params Y&quot; (just text!)<br>      → App triggers the tool (makes it REAL)<br>        → Tool executes, returns results<br>          → App feeds results to LLM<br>            → LLM responds (or decides to call another tool)<br>              → Loop repeats until task done</pre><p>Key insight: The LLM NEVER executes anything. It only outputs text decisions. The App layer is what makes those decisions real.</p><h4>What Are Tools &amp; MCP?</h4><p><strong>Tools</strong> = Actions an Agent Can Perform<br>Examples: Shell (run commands), File read/writeCode execution, Database queries, Web search, API calls.</p><p><strong>MCP (Model Context Protocol)</strong> = Universal Adapter for Tools</p><p>Problem: 10 AI apps × 10 systems = 100 custom integrations</p><p>Solution (MCP): 10 AI apps + 10 systems = 20 implementations</p><p>Each system builds ONE MCP server ←→ Each AI app speaks ONE protocol</p><pre>Full System (hundreds of capabilities)<br>    │<br>    │ Team decides: &quot;Only these are safe for AI&quot;<br>    ▼<br>MCP Server (controlled surface area)<br>    ├── Tool: SearchProfiles ✅<br>    ├── Tool: ExportData ✅<br>    ├── ❌ DeleteEverything (NOT exposed)<br>    ▼<br>Any AI app — speaks MCP, uses only what&#39;s exposed</pre><p>MCP is plumbing, not intelligence. It standardizes connections — it doesn’t help the agent decide WHICH tool to use.</p><h4>Context Window — The LLM’s Working Memory</h4><pre>┌────────────────────────────────────┐<br>│ System prompt (instructions)   👀  │ ← strong attention<br>│ Tool definitions                   │<br>│ Pinned context / knowledge         │<br>│ ............                   😴   │ ← &quot;lost in the middle&quot;<br>│ Old messages                       │<br>│ ............                       │<br>│ Latest message                 👀  │ ← strong attention<br>└────────────────────────────────────┘<br>        <br><br>Full? → Oldest parts dropped → LLM &quot;forgets&quot;<br>Why limited? → Attention mechanism: every token looks at every other = O(n²)</pre><h4>What Competes for Context Space:</h4><p>System prompt + Tool definitions + Conversation + Tool results = must fit in window</p><p>MORE tools loaded → LESS room for actual work<br>LONGER conversation → earlier context gets lost/dropped</p><h4>SOPs, Skills &amp; Prompts — Agent advance concepts</h4><p>When you build AI agents for real work, you need to manage what<br>knowledge is available and when. These three mechanisms solve that:</p><pre>┌──────────────┬─────────────────────────┬──────────────────────────┬──────────────────────────┐<br>│              │ TOOLS                   │ SOPs                     │ SKILLS                   │<br>├──────────────┼─────────────────────────┼──────────────────────────┼──────────────────────────┤<br>│ What         │ Actions (run code,      │ Structured PROMPT        │ On-demand KNOWLEDGE      │<br>│              │ query DB, call API)     │ (detailed step-by-step   │ (reference docs, best    │<br>│              │                         │ recipe in markdown)      │ practices, formats)      │<br>├──────────────┼─────────────────────────┼──────────────────────────┼──────────────────────────┤<br>│ How it       │ Function call with      │ Injected into context    │ Agent reads file content │<br>│ works        │ structured input/output │ as a BIG prompt when     │ into context when it     │<br>│              │                         │ user invokes it          │ detects relevance        │<br>├──────────────┼─────────────────────────┼──────────────────────────┼──────────────────────────┤<br>│ When loaded  │ Always (agent needs     │ ONLY when user invokes   │ ONLY when agent decides  │<br>│              │ them to work)           │ explicitly               │ it&#39;s relevant            │<br>├──────────────┼─────────────────────────┼──────────────────────────┼──────────────────────────┤<br>│ Who triggers │ Agent (autonomously)    │ You (the user)           │ Agent (autonomously)     │<br>├──────────────┼─────────────────────────┼──────────────────────────┼──────────────────────────┤<br>│ Token cost   │ Permanent               │ Only when invoked        │ Only when needed         │<br>├──────────────┼─────────────────────────┼──────────────────────────┼──────────────────────────┤<br>│ Use for      │ Deterministic ACTIONS   │ Repeatable PROCESSES     │ Just-in-time KNOWLEDGE   │<br>├──────────────┼─────────────────────────┼──────────────────────────┼──────────────────────────┤<br>│ Example      │ &quot;Execute this Python&quot;   │ Used for Automation proc-│ &quot;Load SQL best practices │<br>│              │ &quot;Query this database&quot;   │ ess                      │ when user asks about SQL&quot;│<br>├──────────────┼─────────────────────────┼──────────────────────────┼──────────────────────────<br></pre><pre>ALWAYS IN CONTEXT (permanent token cost):<br>  ├── System Prompt  → short, broad personality/role<br>  ├── Tool Definitions → function signatures<br>  └── Pinned Context → knowledge needed EVERY conversation<br><br>LOADED ON-DEMAND (zero cost until triggered):<br>  ├── SOPs    → user invokes → detailed workflow injected (like a big prompt)<br>  └── Skills  → agent decides → reference knowledge loaded when relevant<br><br><br>System Prompt vs SOP:<br>  System Prompt = &quot;You are a code reviewer. Be thorough.&quot;     (ALWAYS there, broad)<br>  SOP           = &quot;Step 1: Check imports. Step 2: Verify      (ON-DEMAND, deep, for automation tasks)<br>                   error handling. MUST flag any catch-all.<br>                   Step 3: Check test coverage...&quot;<br><br>Pinned Context vs Skill:<br>  Pinned  = SQL style guide always open on desk               (ALWAYS there, costs tokens)<br>  Skill   = SQL style guide on shelf, grabbed only when       (ON-DEMAND, free until needed)<br>            someone asks about SQL</pre><p>RAG vs Skill — same purpose, different mechanism:</p><p>Both exist to avoid keeping large knowledge permanently in context — relevant material gets pulled in only when needed. Neither eliminates token cost though: once pulled in, it occupies context space like anything else in the window.</p><ul><li><strong>Skill</strong> = a curated reference doc. Only its short description stays always-in-context (cheap). The LLM(in agent) compares the request against that description and decides “this looks relevant” using its own judgment — not a search over the file’s actual content. Once app triggered, the <strong>entire file loads</strong>, so skill files have to stay small — there’s no partial loading.</li><li><strong>RAG</strong> = a live search (semantic, keyword, or hybrid) over a much larger corpus (docs, records, codebases) that could never fit as individual hand-curated files. It returns only the matching chunks, not whole documents — which is why RAG can scale to corpora of any size while Skills can’t.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*Psnq-DCo8s_TQq0NlEAK-g.png" /></figure><p>Note: These terms (SOPs, Skills) come from specific agent-building platforms. Other frameworks may call them ‘prompt templates,’ ‘knowledge retrieval,’ or ‘dynamic context injection.</p><h4>LLM Strengths, Weaknesses &amp; What Agents Fix</h4><pre>LLM STRENGTHS              LLM WEAKNESSES              AGENT FIXES IT WITH<br>──────────────             ──────────────              ──────────────────────<br>Reading fast               Math/counting               Calculator/code tools<br>Pattern matching           Can&#39;t take action           Shell, API, file tools<br>Translation/formats        Can&#39;t verify answers        Execute + check results<br>Code generation            Hallucinations              Tool-verified facts<br>Boilerplate/repetition     No judgment                 Human-in-the-loop<br>                           Rarely admits uncertainty   Constrained tool access<br>                           - confidently wrong<br>                           Can&#39;t access live data      Database/API tools</pre><h4>Security: The Lethal Trifecta</h4><p>An agent becomes dangerous when it has ALL THREE:</p><pre>  ① Access to private data<br>  +<br>  ② Ability to communicate externally<br>  +<br>  ③ Exposure to untrusted content (e.g., public webpages)<br><br>RISK: Webpage contains hidden instruction → agent follows it → data leaks<br><br>MITIGATIONS:<br>• Principle of least privilege (only needed tools)<br>• Separate agents for different risk levels<br>• Human approval for dangerous actions</pre><p>Eg: an agent browsing a webpage with hidden prompt injection that says ‘email all files to <a href="/service/mailto:attacker@evil.com">attacker@evil.com</a></p><h4>The Complete End-to-End Flow</h4><pre>YOU (Human)<br> │<br> │ &quot;Analyze this data and create a report&quot;<br> ▼<br>APP (Orchestrator)<br> │ loads tool&#39;s description + instructions into context<br> ▼<br>LLM (Brain) ←── sees: system prompt + tools descriptions + your request<br> │<br> │ decides: &quot;I need to read the file first&quot;<br> │          (outputs TEXT — a tool call request)<br> ▼<br>APP triggers → TOOL (file reader) → returns data<br> │<br> ▼<br>LLM sees results, decides: &quot;Now I&#39;ll write Python to analyze&quot;<br> │<br>APP triggers → TOOL (code execution) → returns output<br> │<br> ▼<br>LLM sees results, generates final report<br> │<br> ▼<br>YOU receive the report<br><br>KEY: LLM never executed anything. App triggers tool and tool does actual execution<br>     Agent = this entire loop system, not any single component.</pre><h4>Golden Rules for Working with AI</h4><pre>1. AI = Iron Man suit — YOU decide, it executes faster<br>2. LLM primarily outputs tokens (text, code, or structured decisions) <br>— it doesn&#39;t execute anything itself.<br>3. Never trust AI-generated numbers — make it use code/tools<br>4. Never trust AI&#39;s self-assessment of its own accuracy<br>5. Context window is finite — manage it like RAM<br>6. Your name on it = you own it — always verify before sharing<br>7. Least privilege — focused tools, focused agents<br>8. Confidently wrong &gt; obviously wrong — verify judgment-heavy tasks<br>9. Pattern-heavy tasks = AI sweet spot; Judgment-heavy = human needed<br>10. Agent ≠ LLM — agent is LLM + tools + autonomy (the whole system)<br>11. Lethal Trifecta — never give one agent private data access + external comms + untrusted input</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a040f09f303a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Amazon BIE Interview Experience: A Complete Technical and Behavioral Guide]]></title>
            <link>https://medium.com/@mdmoseena22/my-amazon-bie-interview-experience-a-complete-technical-and-behavioral-guide-0d5de30e3cff?source=rss-72a4fb8cbdaf------2</link>
            <guid isPermaLink="false">https://medium.com/p/0d5de30e3cff</guid>
            <category><![CDATA[amazon]]></category>
            <category><![CDATA[interview-questions]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <dc:creator><![CDATA[MOHAMMED MOSEENA]]></dc:creator>
            <pubDate>Sat, 14 Feb 2026 17:29:24 GMT</pubDate>
            <atom:updated>2026-02-14T17:37:08.924Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Landing a Business Intelligence Engineer (BIE) role at Amazon is a challenging journey that tests both your technical skills and behavioral alignment with Amazon’s famous Leadership Principles. I successfully cleared one SQL assessment round and three interview rounds — Technical Round with a Senior Data Engineer, BIE Manager Round, and the Bar Raiser behavioral assessment — and I’m currently working as a BIE at Amazon. I’m sharing my complete experience to help you prepare effectively.</p><h3>Understanding the Amazon BIE Role</h3><p>The interview process starts with an SQL-based online assessment, followed by three rounds that cover SQL, data visualization, metrics, reporting, and Amazon’s Leadership Principles.</p><p><a href="https://amazon.jobs/content/en/how-we-hire/bie-interview-prep">Learn more about the BIE role at Amazon</a></p><h3>Round 1: Senior Data Engineer — Technical Deep Dive</h3><p><strong>Duration:</strong> 60 minutes<br><strong>Format:</strong> Behavioral introduction + SQL/Database problem solving</p><h4>Behavioral Questions</h4><p>The round started with behavioral questions to understand your approach to work:</p><ol><li><strong>Describe a situation where project requirements seemed unclear or unnecessary. How did you handle it?</strong></li><li><strong>If a critical business metric drops by 20%, walk me through your investigation process.</strong></li><li><strong>Explain your end-to-end approach when assigned a new analytical task.</strong></li><li><strong>Tell me about your most impactful work at your previous organization.</strong></li></ol><p><strong>Pro Tip:</strong> Use the STAR method (Situation, Task, Action, Result) for all behavioral questions. Amazon is data-driven, so always include metrics in your responses.</p><h4>Technical Question Set</h4><p><strong>Question 1: Understanding JOIN Behaviors with NULL Values</strong></p><p><strong>Given:</strong></p><pre>Table A: col1 = [1, 1, 1, 1, NULL]<br>Table B: col1 = [1, 1, NULL]</pre><p><strong>Predict the output for these queries:</strong></p><pre>-- Query 1<br>SELECT COUNT(*) FROM TableA INNER JOIN TableB <br>ON TableA.col1 = TableB.col1;<br>-- Query 2<br>SELECT COUNT(*) FROM TableA LEFT JOIN TableB <br>ON TableA.col1 = TableB.col1;<br>-- Query 3<br>SELECT COUNT(*) FROM TableA RIGHT JOIN TableB <br>ON TableA.col1 = TableB.col1;<br>-- Query 4<br>SELECT COUNT(*) FROM TableA FULL OUTER JOIN TableB <br>ON TableA.col1 = TableB.col1;<br>-- Query 5<br>SELECT COUNT(1) FROM TableA INNER JOIN TableB <br>ON TableA.col1 = TableB.col1;</pre><p><strong>Expected Answers:</strong></p><ul><li>Query 1 &amp; 5: <strong>8</strong> (4 × 2 matching on value 1; NULLs don’t match in JOIN conditions)</li><li>Query 2: <strong>9</strong> (8 matched + 1 unmatched NULL from Table A)</li><li>Query 3: <strong>9</strong> (8 matched + 1 unmatched NULL from Table B)</li><li>Query 4: <strong>10</strong> (8 matched + NULL from A + NULL from B)</li></ul><p><strong>Key Learning:</strong> NULLs never match in JOIN conditions (NULL = NULL evaluates to UNKNOWN, not TRUE).</p><p><strong>Question 2: Top N Query with Date Filters</strong></p><p><strong>Schema:</strong></p><pre>orders(order_id, customer_id, order_date, total_amount)<br>customers(customer_id, customer_name, region)</pre><p><strong>Task:</strong> Find the top 3 customers with highest total spending in the ‘Northeast’ region over the last 6 months.</p><p><strong>Solution:</strong></p><pre>SELECT <br>    c.customer_id,<br>    c.customer_name,<br>    SUM(o.total_amount) AS total_spend<br>FROM orders o<br>INNER JOIN customers c ON o.customer_id = c.customer_id<br>WHERE c.region = &#39;Northeast&#39;<br>    AND o.order_date &gt;= DATE_SUB(CURDATE(), INTERVAL 6 MONTH)<br>GROUP BY c.customer_id, c.customer_name<br>ORDER BY total_spend DESC<br>LIMIT 3;</pre><p><strong>Key Concepts:</strong></p><ul><li>Date filtering with DATE_SUB() or DATEADD() depending on SQL dialect</li><li>Proper JOIN conditions</li><li>Aggregation with GROUP BY</li><li>Ordering and limiting results</li></ul><p><strong>Question 3: Removing Duplicate Records</strong></p><p><strong>Schema:</strong></p><pre>transactions(id, customer_id, transaction_date, amount)</pre><p><strong>Task:</strong> Delete duplicate transactions, keeping only the record with the smallest id for each unique combination of (customer_id, transaction_date, amount).</p><p><strong>Solution:</strong></p><pre>DELETE FROM transactions<br>WHERE id NOT IN (<br>    SELECT MIN(id) <br>    FROM transactions <br>    GROUP BY customer_id, transaction_date, amount<br>);</pre><p><strong>Question 4: Finding Nth Highest Value</strong></p><p><strong>Schema:</strong></p><pre>employees(emp_id, emp_name, salary, department)</pre><p><strong>Task:</strong> Find the 3rd highest salary from the employees table.</p><p><strong>Solution:</strong></p><pre>WITH ranked_salaries AS (<br>    SELECT DISTINCT salary,<br>           DENSE_RANK() OVER (ORDER BY salary DESC) AS salary_rank<br>    FROM employees<br>)<br>SELECT salary<br>FROM ranked_salaries<br>WHERE salary_rank = 3;</pre><p><strong>Alternative (without window functions):</strong></p><pre>SELECT MAX(salary) AS third_highest_salary<br>FROM employees<br>WHERE salary &lt; (<br>    SELECT MAX(salary) <br>    FROM employees <br>    WHERE salary &lt; (SELECT MAX(salary) FROM employees)<br>);</pre><p><strong>Key Difference:</strong> Use DENSE_RANK() instead of RANK() when you want consecutive rankings without gaps, even when there are tied values.</p><p><strong>Question 5: Time-Series Analysis — Month-over-Month Growth</strong></p><p><strong>Schema:</strong></p><pre>sales(sale_id, sale_date, product_id, amount)</pre><p><strong>Task:</strong> Calculate month-over-month sales growth percentage.</p><p><strong>Solution:</strong></p><pre>WITH monthly_sales AS (<br>    SELECT <br>        DATE_TRUNC(&#39;month&#39;, sale_date) AS month,<br>        SUM(amount) AS total_sales<br>    FROM sales<br>    GROUP BY DATE_TRUNC(&#39;month&#39;, sale_date)<br>)<br>SELECT <br>    month,<br>    total_sales AS current_month_sales,<br>    LAG(total_sales) OVER (ORDER BY month) AS previous_month_sales,<br>    ROUND(<br>        ((total_sales - LAG(total_sales) OVER (ORDER BY month)) * 100.0) <br>        / NULLIF(LAG(total_sales) OVER (ORDER BY month), 0), <br>        2<br>    ) AS mom_growth_percentage<br>FROM monthly_sales<br>ORDER BY month;</pre><p><strong>Key Concepts:</strong></p><ul><li>Date truncation to month level</li><li>Window functions (LAG) for accessing previous row values</li><li>NULLIF to prevent division by zero errors</li><li>Percentage calculation with proper rounding</li></ul><p><strong>Question 6: Self-Join for Hierarchical Data</strong></p><p><strong>Schema:</strong></p><pre>employees(emp_id, emp_name, manager_id)</pre><p><strong>Task:</strong> List all employees with their manager names.</p><p><strong>Solution:</strong></p><pre>SELECT <br>    e.emp_id,<br>    e.emp_name AS employee_name,<br>    e.manager_id,<br>    m.emp_name AS manager_name<br>FROM employees e<br>LEFT JOIN employees m ON e.manager_id = m.emp_id<br>ORDER BY e.emp_id;</pre><p><strong>Why LEFT JOIN?</strong> To include top-level employees (CEOs/executives) who have no managers (manager_id = NULL).</p><p><strong>Question 7: Data Modeling Exercise</strong></p><p><strong>Task:</strong> Design a normalized database schema for an e-commerce platform.</p><p><strong>Expected Tables:</strong></p><pre>customers (1) ←→ (M) customer_addresses<br>customers (1) ←→ (M) orders<br>orders (1) ←→ (M) order_items<br>products (1) ←→ (M) order_items<br>suppliers (1) ←→ (M) products<br>categories (1) ←→ (M) products</pre><p><strong>Discussion Points:</strong></p><ul><li>One-to-Many relationships and why they’re needed</li><li>Junction tables for Many-to-Many relationships</li><li>Primary and Foreign key considerations</li><li>Normalization vs. denormalization trade-offs</li></ul><h3>Round 2: BIE Manager — SQL + Data Visualization + Metrics</h3><p><strong>Duration:</strong> 60 minutes<br><strong>Format:</strong> Complex SQL problem solving + Business intelligence thinking</p><h4>Behavioral Questions</h4><ol><li><strong>Tell me about a time you built complex SQL logic to solve a business problem.</strong></li><li><strong>Share an example of when you had to balance speed with accuracy in your deliverables.</strong></li></ol><p><strong>Key Focus Areas:</strong></p><ul><li>Leadership principles: Bias for Action &amp;Ownership</li></ul><h4>Technical Question:</h4><ol><li><strong>Explain the difference between incremental and full load data pipelines. When would you use each?</strong></li></ol><p><strong>Full Load:</strong> Complete data refresh each run</p><ul><li><strong>When to use:</strong> Small datasets (&lt;1M rows), source data changes frequently and unpredictably, historical data corrections needed, initial pipeline setup</li><li><strong>Trade-off:</strong> Higher computational cost, but simpler logic and guaranteed consistency</li></ul><p><strong>Incremental Load:</strong> Only new/changed records</p><ul><li><strong>When to use:</strong> Large datasets (&gt;10M rows), append-only data (logs, events), clear timestamp/flag for changes, performance critical</li><li><strong>Trade-off:</strong> Complex logic (change detection, handling deletes), but much faster and cost-efficient</li></ul><p><strong>Real example:</strong> At my previous company, we used incremental loads for transaction data (adding ~5M rows daily) but full loads for the product catalog (only 50K products, frequently updated attributes).</p><p><strong>2. Identifying Steep Discount Products</strong></p><p><strong>Context:</strong> You’re analyzing promotional pricing to identify products with unusually deep discounts compared to their recent pricing history.</p><p><strong>Schema:</strong></p><pre>product_sales(<br>    product_id, <br>    promotion_name, <br>    sale_date, <br>    sale_price<br>)</pre><p><strong>Sample Data:</strong></p><pre>product_id | promotion_name  | sale_date  | sale_price<br>-----------|-----------------|------------|------------<br>P001       | NewYear_Sale    | 2026-01-15 | 480<br>P001       | Spring_Promo    | 2026-03-20 | 520<br>P001       | Summer_Deal     | 2026-08-10 | 280<br>P001       | Winter_Clearance| 2026-12-01 | 190</pre><p><strong>Requirement:</strong> Identify all sales where the sale_price is lower than the average of the previous 3 sale prices for that product (chronologically ordered). This indicates a &quot;steep discount&quot; worth investigating.</p><p><strong>Solution:</strong></p><pre>WITH price_history AS (<br>    SELECT<br>        product_id,<br>        promotion_name,<br>        sale_date,<br>        sale_price,<br>        LAG(sale_price, 1) OVER (PARTITION BY product_id ORDER BY sale_date) AS prev_price_1,<br>        LAG(sale_price, 2) OVER (PARTITION BY product_id ORDER BY sale_date) AS prev_price_2,<br>        LAG(sale_price, 3) OVER (PARTITION BY product_id ORDER BY sale_date) AS prev_price_3<br>    FROM product_sales<br>),<br>steep_discounts AS (<br>    SELECT <br>        product_id,<br>        promotion_name,<br>        sale_date,<br>        sale_price,<br>        (prev_price_1 + prev_price_2 + prev_price_3) / 3.0 AS avg_prev_3_sales<br>    FROM price_history<br>    WHERE prev_price_1 IS NOT NULL <br>        AND prev_price_2 IS NOT NULL <br>        AND prev_price_3 IS NOT NULL<br>)<br>SELECT <br>    product_id,<br>    promotion_name,<br>    sale_date,<br>    sale_price,<br>    avg_prev_3_sales,<br>    ROUND((avg_prev_3_sales - sale_price) * 100.0 / avg_prev_3_sales, 2) AS discount_percentage<br>FROM steep_discounts<br>WHERE sale_price &lt; avg_prev_3_sales<br>ORDER BY discount_percentage DESC;</pre><p><strong>Key Concepts:</strong></p><ul><li>Window functions with PARTITION BY for product-level analysis</li><li>LAG() function for accessing historical values</li><li>Proper NULL handling (products with fewer than 3 previous sales)</li><li>Calculating percentage discount for business context</li></ul><p><strong>Follow-up: Dashboard Design Discussion</strong></p><p><strong>Question:</strong> “Now that you’ve identified steep discount products through SQL analysis, how would you present these insights in a dashboard? What metrics and visualizations would you create?”</p><p><strong>My Approach:</strong></p><p><strong>1. Define the North Star Metric</strong></p><ul><li><strong>Percentage of products with steep discounts per sale event</strong> (not absolute count)</li><li>Why: 10 products out of 20 (50%) tells a different story than 10 out of 1,000 (1%)</li></ul><p><strong>2. Identify Guardrail Metrics</strong></p><ul><li>Customer retention rate (ensure discounts don’t erode brand perception)</li><li>Revenue impact (quantify financial effect)</li><li>Profit margins (verify discounts aren’t destroying profitability)</li></ul><p><strong>3. Key Visualizations</strong></p><p><strong>Executive KPI Cards:</strong> Steep discount rate, revenue impact, products affected — instant insight for leadership</p><p><strong>Trend Line Chart:</strong> % of steep discounts over time by sale event — identifies patterns and seasonality issues</p><p><strong>Discount Distribution (Donut Chart):</strong> Products grouped by discount depth (10–25%, 25–40%, 40–60%, 60%+) — highlights severity for immediate action</p><p><strong>Interview Tip:</strong> Always connect visualizations to business decisions. Explain how charts enable specific actions, not just what they show.</p><h3>Round 3: Bar Raiser — Leadership Principles Deep Dive</h3><p><strong>Duration:</strong> 60 minutes<br><strong>Format:</strong> 100% behavioral, focused on Amazon’s Leadership Principles(Deliver Results + Learn and Be Curious) and my past work experience and projects listed on my CV</p><p>The Bar Raiser interview is the most critical round. This interviewer isn’t from your hiring team and acts as an objective evaluator of Amazon’s hiring bar. They assess whether you embody Amazon’s culture through the Leadership Principles.</p><p><a href="https://www.amazon.jobs/content/en/our-workplace/leadership-principles">Learn more about Leadership principles at Amazon</a></p><h3>Sample Leadership Principle Question &amp; STAR Response</h3><h4>1. Customer Obsession</h4><p><strong>Question:</strong> Tell me about a time you followed a customer-centric approach, even when it meant extra work or pushback from your team.</p><p><strong>My STAR Response:</strong></p><p><strong>Situation:</strong> During a dashboard migration project, we had a ‘deep dive’ tab with granular details. The development team assumed it was essential, but it was adding 15 seconds to overall load time.</p><p><strong>Task:</strong> I wanted to understand if this feature was worth the performance cost and, if not, influence the decision to deprecate it.</p><p><strong>Action</strong>:</p><ul><li>First, I analyzed usage data and found only 5% of users were accessing the deep dive tab.</li><li>Second, I quantified the impact — this rarely-used tab was adding 15 seconds to load time for 100% of users.</li><li>Third, I presented this to stakeholders with a clear narrative: ‘95% of users prefer summary visualizations and never use deep dive, yet everyone pays the performance penalty.’ I validated this by discussing with key stakeholders who confirmed they relied on summary views.</li></ul><p><strong>Result</strong>: The data story made the decision obvious — we deprecated the deep dive tab. Dashboard load time improved by 40%, user satisfaction increased, and no one complained about the missing feature. This taught me that don’t build features because they’re impressive; build them because customers derive value. Data should drive these decisions.</p><h3>Sample Questions to Ask Interviewers</h3><p>Prepare thoughtful questions for each interviewer — it shows you’ve researched the role and are genuinely interested. Spend time before interviews investigating the team, their projects, and Amazon’s business to craft relevant questions.</p><p><strong>Sample Questions:</strong></p><ol><li>What does success look like in this role after 6 months? After 1 year?</li><li>How does Amazon support continuous learning and professional development?</li><li>What’s the most challenging technical problem the team has solved recently?</li></ol><h3>Key Interview Preparation Tips</h3><h4><strong>For Technical Rounds:</strong></h4><ol><li><strong>Master SQL Window Functions:</strong></li></ol><ul><li>LAG, LEAD, ROW_NUMBER, RANK, DENSE_RANK</li><li>PARTITION BY for group-wise operations</li><li>Practice on platforms like LeetCode and previous amazon questions</li></ul><p><strong>2. Understand JOIN Behavior:</strong></p><ul><li>How NULLs behave in different JOIN types</li><li>Performance implications of different JOIN strategies</li><li>When to use LEFT vs RIGHT vs FULL OUTER vs CROSS JOINs</li></ul><p>3. <strong>Know Your Date/Time Functions:</strong></p><ul><li>Date arithmetic (DATEADD, DATEDIFF, DATE_SUB)</li></ul><p><strong>4. Practice basic Data Modeling:</strong></p><ul><li>Normalization &amp; Denormalization and their use cases</li><li>Mappings</li></ul><p><strong>5. For Dashboard Design Questions:</strong></p><p><strong>a. Always Define North Star Metrics First:</strong></p><ul><li>What single metric best indicates success?</li><li>Why that metric over alternatives?</li></ul><p><strong>b. Identify Guardrail Metrics:</strong></p><ul><li>What secondary metrics prevent gaming the North Star metric?</li><li>How do you balance competing objectives?</li></ul><p><strong>c. Research Amazon’s Key Metrics:</strong></p><ul><li>Before your interview, research the team’s domain (retail, AWS, Kindle, etc.)</li><li>Understand metrics like Customer Lifetime Value, conversion rate, or operational efficiency specific to their business</li></ul><p><strong>d. Explain Visualization Choices:</strong></p><ul><li>Why this chart type enables the specific business decision needed</li></ul><h4>For Behavioral Rounds:</h4><ol><li><strong>Prepare 10–15 STAR Stories:</strong></li></ol><ul><li>2–3 examples per Leadership Principle</li><li>Quantify results with metrics</li></ul><p><strong>2. Know Amazon’s Leadership Principles</strong></p><p><strong>3. Be Specific:</strong></p><ul><li>Replace “we” with “I” to clarify your specific contributions</li></ul><h3><strong>Final Thoughts</strong></h3><p>The Amazon BIE interview is challenging but fair. Here’s what matters most:</p><ol><li>Share real experiences, including failures</li><li>Practice SQL and STAR stories extensively</li><li>Ask questions; it’s a conversation</li><li>Show how data influenced your decisions</li><li>Connect technical work to business impact</li></ol><h3>About the Interview Process</h3><p>After completing all rounds, you’ll typically hear back within 5 business days. If you receive an offer, the recruiter will discuss compensation and answer any questions about relocation, benefits, or team placement.</p><p>If you don’t receive an offer this time, you can reapply after 6 months. Use the feedback (if provided) to strengthen weak areas and come back stronger.</p><h3>Connect With Me</h3><p>If you have questions about the Amazon BIE interview process feel free to reach out:</p><p>📧 Email: mdmoseena22@gmail.com<br>💼 LinkedIn: <a href="https://www.linkedin.com/in/mdmoseena/">https://www.linkedin.com/in/mdmoseena/</a></p><p><strong>Good luck with your Amazon BIE interview journey!</strong></p><p><em>Have you interviewed at Amazon or other tech companies? Share your experiences in the comments! Let’s help each other succeed.</em></p><p><strong>#Amazon #DataEngineering #BusinessIntelligence #InterviewPrep #SQL #CareerAdvice #TechInterviews #DataAnalytics #FAANG</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0d5de30e3cff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My ML Interview Notebook: A Revision Guide for Data Science Interviews]]></title>
            <link>https://medium.com/@mdmoseena22/my-ml-interview-notebook-a-revision-guide-for-data-science-interviews-d7b89cccfe33?source=rss-72a4fb8cbdaf------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7b89cccfe33</guid>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[supervised-learning]]></category>
            <dc:creator><![CDATA[MOHAMMED MOSEENA]]></dc:creator>
            <pubDate>Tue, 18 Nov 2025 17:35:36 GMT</pubDate>
            <atom:updated>2026-06-04T16:36:17.766Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Hands On Excercises:</strong></blockquote><p><a href="https://colab.research.google.com/drive/18kcbYfYJ3eJJVDSXDUmAgNM2eW-b5YLu">https://colab.research.google.com/drive/18kcbYfYJ3eJJVDSXDUmAgNM2eW-b5YLu</a></p><blockquote><strong>Eigen Vectors &amp; Eigen Values</strong></blockquote><p><strong>Eigen Vector</strong>: It is special vector whose <strong>direction remains unchanged</strong> after a linear transformation — only its magnitude may scale up or down.</p><p><strong>Eigen Value</strong>: Upon linear transformation how much eigen vector shrink or scaled up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*1k44XwQKBP-I8v30DjWhUQ.png" /></figure><p><strong>Note: Eigen decomposition of the standardized covariance matrix yields eigenvectors (principal components) and eigenvalues (variance explained). By projecting high-dimensional data onto the eigenvectors with the largest eigenvalues, we reduce dimensionality while retaining the most significant variance in the dataset.</strong></p><blockquote><strong>Gradient Descent</strong></blockquote><p>It is an optimization algorithm used to find the best model parameters (weights) by minimizing a loss function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*Bh2UIPk3nxj6H5tFpTfvJw.webp" /></figure><p><strong>Algorithm Steps:</strong></p><ol><li>Define the loss function and compute its gradient (derivative) with respect to each weight.</li><li>Initialize weights randomly (first iteration) or use the updated weights from the previous step (subsequent iterations).</li><li>Update the weights using the rule:</li></ol><p>new_weight = old_weight — (learning_rate)*(gradient w.r.t weight)</p><p>Repeat 2 &amp; 3 until convergence, which happens when either:</p><ul><li>The maximum number of iterations is reached, or</li><li>The gradient approaches zero (loss function stops decreasing).</li></ul><p><strong>Key Points:</strong></p><ul><li>Gradient Descent works best for convex loss functions (e.g., Linear Regression, Logistic Regression), since they have a single global minimum.</li><li>For non-convex functions (e.g., deep learning models), it may get stuck in local minima or saddle points.</li><li>Gradient Descent is used in <strong>Linear Regression, Logistic Regression, Gradient Boosting &amp; XGBoost.</strong></li></ul><p>Three Versions of Gradient Descent</p><ol><li><strong>Batch Gradient Descent</strong> — Uses all training data in each update step (accurate but computationally expensive).</li><li><strong>Stochastic Gradient Descent (SGD)</strong> — Updates weights using a single randomly chosen data point at each step (fast, but noisy).</li><li><strong>Mini-Batch Gradient Descent</strong> — Uses a small random subset of the data at each step (balances efficiency and stability, widely used in practice).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*imlIauXO4eth-XasKOFEeA.png" /></figure><blockquote><strong>Model Evaluation &amp; Selection</strong></blockquote><p>When evaluating models on train vs test data, key scenarios include:</p><ul><li><strong>High Variance (Overfitting):</strong> Model performs well on training data but poorly on test data.</li><li><strong>High Bias (Underfitting):</strong> Model performs consistently poorly on both training and test data.</li></ul><p>Model evaluation and selection involves understanding the <strong>bias-variance tradeoff and model complexity.</strong></p><blockquote><strong>Bias-Variance Tradeoff</strong></blockquote><p>In ML, the error is classified into three components</p><ol><li><strong>Bias</strong>: How off the model’s prediction from actual value, smaller being better.</li><li><strong>Variance</strong>: The extend to which the model performance get impacted based on training input, smaller being better</li><li><strong>Irreducible Error</strong>: Inherent Noise, no model can explain.</li></ol><p>Bias-Variance Tradeoff is when you try to reduce bias, variance increases and when you try to reduce variance, bias increases.</p><p>Simple models are generally have high Bias and low variance and complex models have low Bias and high variance, so our goal is to find sweet spot between these two scenarios (high bias+ low variance, high variance + low bias).</p><blockquote><strong>Model Complexity</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*XrRDyO17NUZLAqHaYIfFpw.png" /></figure><p><strong>Underfitting</strong></p><ol><li>Model fails to capture the true relationship between features and target.</li><li><strong>Typical in:</strong> Simple, less flexible models (e.g., Linear Regression, Logistic Regression, Naive Bayes).</li><li><strong>Detection: </strong>Learning curves show both training and validation errors are high and close together.</li><li><strong>Handling:</strong></li></ol><ul><li>Add more informative features.</li><li>Apply feature transformations.</li><li>Use more flexible models (e.g., Decision Trees, Ensemble methods).</li></ul><p><strong>Overfitting</strong></p><ol><li>Model captures noise in the training data, failing to generalize to unseen data.</li><li>Complex, highly flexible models (e.g., Decision Trees, Boosting models like Gradient Boost, XGBoost).</li><li><strong>Detection: </strong>Learning curves show low training error but rising validation error.</li></ol><figure><img alt="For Overfitting Learning curves show low training error but rising validation error." src="https://cdn-images-1.medium.com/max/1024/1*Ib7pLc4tgnW2wPuZQByVWw.png" /><figcaption>Overfitting Scenario</figcaption></figure><p><strong>4. Handling:</strong></p><ul><li>Add more data.</li><li>Apply feature selection/dimensionality reduction.</li><li>Use regularization (e.g., L1, L2 penalties for linear models).</li><li>Prune or simplify complex models.</li></ul><blockquote><strong>Regularization</strong></blockquote><p>Regularization is a technique to reduce <strong>overfitting</strong> by adding a penalty term to the loss function. It is often called a <strong>shrinkage method</strong> because it reduces the magnitude of model weights, thereby lowering the impact of less important features.</p><p><strong>Key Idea:</strong></p><ul><li>Reduces <strong>variance</strong> by introducing a small amount of <strong>bias</strong>.</li><li>For linear models, Ordinary Least Squares (OLS) is updated with an additional penalty term.</li></ul><p><strong>Regularized Linear Models</strong></p><ol><li><strong>Ridge Regression(L2 Regularization)</strong>:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*mBHSAPNguzcMOH8fuV1QzA.png" /></figure><p><strong>Effects:</strong></p><ol><li>Reduces overfitting.</li><li>Shrinks weights close to zero, but not exactly zero.</li><li>Handles multicollinearity effectively → If multiple features are saying the same thing, don’t pick one — <strong>share the importance across them and keep weights small → </strong>Shrinks correlated feature weights <strong>together</strong>.</li></ol><p>2. <strong>Lasso Regression</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*ZRAy_nhZuc6vVUpldYW8KA.png" /></figure><p><strong>Effects:</strong></p><ol><li>Reduces overfitting.</li><li>Known as a <strong>stricter shrinkage method</strong> compared to Ridge.</li><li>Performs <strong>feature selection</strong> (shrinks some weights exactly to zero) → If a feature is not helping, <strong>kick it out completely</strong></li></ol><p>3. <strong>Elastic Net Regression</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*zmVwk4u-ylSuAcncOpLX2Q.png" /></figure><p><strong>Effects:</strong></p><ol><li>Balances benefits of Ridge (L2) and Lasso (L1).</li></ol><ul><li>Uses <strong>L2</strong> → stabilizes correlated features</li><li>Uses <strong>L1</strong> → removes useless ones</li></ul><p>2. Works well with <strong>large datasets</strong> and when features are correlated.</p><blockquote><strong>Interpretability &amp; Explainability</strong></blockquote><p><strong>Why It Matters</strong></p><p>In real-world domains like healthcare, finance, and lending, models can’t just be accurate — they must be interpretable or explainable. Users, regulators, and auditors have a right to know <em>why</em> a prediction was made. A loan denial without justification isn’t just bad UX — in many jurisdictions, it’s illegal.</p><p><strong>The classic tradeoff</strong>: Higher model complexity → lower interpretability. A linear regression is fully transparent; a deep neural network is a black box.</p><p><strong>Two Different Concepts (Often Confused)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tmmhb2U2GJAaetihcG0-Fg.png" /></figure><p><strong>One-line distinction</strong>: <em>“Interpretability is a property of the model; explainability is a property of the tooling around the model.”</em></p><p><strong>Feature Importance — How It Actually Works Internally</strong></p><p><strong>Tree-based (Mean Decrease in Impurity, MDI) — used by sklearn’s </strong><strong>feature_importances_</strong></p><p>Trees split data to make child nodes “purer” (more homogeneous in label). Each split reduces impurity (Gini, entropy, or MSE). Feature importance is just bookkeeping on those reductions:</p><p><strong>Algorithm:</strong></p><ol><li>For every split in every tree, record:</li></ol><ul><li>Which feature was used</li><li>How much impurity dropped from parent to children</li><li>How many samples passed through that node (the <em>weight</em>)</li></ul><p>2. For each feature, sum the weighted impurity drops across all splits where it was used.</p><p>3. (For ensembles like Random Forest) average across all trees.</p><p>4. Normalize so importances sum to 1.</p><p>Concrete example: In a Random Forest predicting loan default:</p><ul><li>credit_score is used in 70% of splits and produces large impurity drops → importance = 0.45</li><li>income is used in 50% of splits with moderate drops → importance = 0.25</li><li>zip_code is used in 30% of splits with small drops → importance = 0.15</li><li>(and so on, summing to 1.0)</li></ul><p>Why it’s fast: the model already computed impurity drops during training — feature importance is just summing what’s already there.</p><p><strong>Why it’s biased (interview gold):</strong></p><p><strong>Bias #1 </strong>— Favors High-Cardinality Features</p><p>What “cardinality” means: the number of distinct values a feature can take.</p><ul><li>Low cardinality: gender (2 values), is_premium_user (2 values)</li><li>High cardinality: zip_code (~40,000 values), user_id (millions), email_domain (thousands)</li></ul><p>Why it’s a problem:</p><p>When a tree splits on a feature, it tries every possible split point and picks the best one. The more unique values a feature has, the more split points to try → the higher the chance that one of those splits looks good <em>by random chance</em> even if the feature carries no real signal.</p><p>Concrete Example</p><p>You’re predicting customer churn. You have:</p><ul><li>is_premium_user (2 values: yes/no) — truly predictive</li><li>user_id (50,000 unique values) — completely random, no real signal</li></ul><p>Now, for user_id, the tree gets to try ~50,000 possible split points: <em>&quot;is user_id &lt; 12,847?&quot;</em>, <em>&quot;is user_id &lt; 24,193?&quot;</em>, etc. With that many tries, one of them will randomly happen to separate churners from non-churners in the training data — even though user_id is meaningless.</p><p>The tree happily uses that lucky split, accumulates a small impurity drop, and feature importance counts it.</p><p>Multiply this across hundreds of trees in a Random Forest, and user_id ends up looking like an important feature — even though it&#39;s noise.</p><p>The Result</p><pre>Feature Importance (biased):<br>user_id          ████████████  0.30   ← noise, but lots of split opportunities<br>is_premium_user  ██████        0.15   ← real signal, but only 1 possible split<br>account_age      █████████     0.22<br>country          ████          0.10<br>...</pre><p><strong>Bias #2</strong> — Favors Continuous Features Over Categorical Ones</p><p>This is essentially the same root cause as Bias #1, just framed differently.</p><p>Why it’s a problem:</p><p>A continuous feature like income (range $0 - $500K) gives the tree infinitely many possible split points — income &lt; 32,500, income &lt; 32,501, etc. (In practice, the tree only tries split points between observed values, but that&#39;s still hundreds or thousands of options.)</p><p>A categorical feature like marital_status (3 values: single/married/divorced) gives the tree only a few possible splits — basically <em>&quot;is marital_status == single?&quot;</em> and a couple of variations.</p><p>Result: even if marital_status is genuinely a stronger predictor, income gets more chances to find a &quot;good&quot; split, so it accumulates more impurity drops and looks more important.</p><p>Concrete Example</p><p>You’re predicting loan default. The <em>true</em> causal relationship is:</p><ul><li>marital_status strongly predicts default (married → much lower risk)</li><li>random_continuous_noise (a column of random numbers) has zero relationship</li></ul><p>Run a Random Forest:</p><pre>Feature Importance (biased):<br>random_continuous_noise  ████████  0.20   ← noise wins through sheer split count<br>marital_status           ████      0.10   ← real signal, fewer split options</pre><p>The noise feature can outrank the real categorical feature simply because it had more opportunities to be lucky.</p><p><strong>Bias #3</strong> — Correlated Features Split the Importance Arbitrarily</p><p>Why it’s a problem:</p><p>Imagine two features A and B carry almost the same information (high correlation). When the tree builds its first split, it has to pick one — say, A. After that split, B adds almost no new information, so the tree mostly ignores it. In subsequent trees, sometimes the tree picks A first, sometimes B — depending on which split looks marginally better on that bootstrap sample.</p><p>Result: the <em>true</em> importance of “the underlying signal” gets split between A and B, often arbitrarily and unevenly. Either one can end up looking unimportant even though the underlying concept is highly predictive.</p><p>Concrete Example</p><p>You’re predicting house prices. Two features:</p><ul><li>square_footage (sq ft)</li><li>square_meters (the same thing, different unit — correlation ≈ 1.0)</li></ul><p>Together they’re the <em>single most predictive concept</em> in your dataset. But when you train a Random Forest:</p><pre>Feature Importance:<br>location          ████████████████  0.40<br>square_footage    ██████            0.15   ← real signal, split with its twin<br>square_meters     █████             0.13   ← real signal, split with its twin<br>year_built        ████              0.10<br>...</pre><p>Neither of them looks dominant. A naive reader concludes “size doesn’t matter much” — when in fact, size is the most important feature, just hidden because its importance got divided.</p><p>A Worse Version of the Same Problem</p><p>In the real world, you don’t have to be that obvious. Consider:</p><ul><li>age and years_of_experience (correlation ~0.85)</li><li>total_income and salary (correlation ~0.95)</li><li>temperature_celsius and temperature_fahrenheit (correlation = 1.0)</li></ul><p>Each pair will silently split its importance, and you might miss the strongest signal in your model.</p><p><strong>Limitation— Doesn’t Tell You the Direction of the Effect</strong></p><p>Why it’s a problem:</p><p>Feature importance says: <em>“The model relies on </em><em>credit_score a lot.&quot;</em></p><p>It does NOT say:</p><ul><li>Did high credit_score push predictions toward “approve” or “deny”?</li><li>Is the relationship linear, U-shaped, or threshold-based?</li><li>For <em>this specific applicant</em>, did their credit_score help or hurt them?</li></ul><p>Concrete Example</p><p>You’re predicting hospital readmission. Two features both have importance = 0.20:</p><ul><li>medication_count — <em>more</em> medications → <em>higher</em> readmission risk (positive direction)</li><li>follow_up_visits — <em>more</em> follow-up visits → <em>lower</em> readmission risk (negative direction)</li></ul><p>From feature importance alone, both look identical:</p><pre>medication_count    ████████  0.20<br>follow_up_visits    ████████  0.20</pre><p>But the clinical implications are opposite. If you wanted to act on this — say, design an intervention — you’d need to know:</p><ul><li>Reducing medication count <em>might</em> lower readmission.</li><li>Reducing follow-up visits would <em>increase</em> readmission.</li></ul><p>Feature importance gives you none of this.</p><p><strong>Limitations of Feature Importance (Why We Need SHAP)</strong></p><p>❌ <strong>Global only </strong>— tells you what the model relies on <em>on average</em>, not why a specific person was denied a loan.</p><p>❌ <strong>No direction</strong> — high importance doesn’t say whether the feature pushed up or down.</p><p>❌ <strong>Biased</strong> — high-cardinality and correlated features distort results.</p><p>This is why for per-prediction explanations, we reach for SHAP.</p><p><strong>SHAP — Explained Simply</strong></p><p>The big idea: SHAP answers a <em>local</em> question — <em>“For this specific prediction, how much did each feature push the result above or below the average prediction?”</em></p><p>It’s based on Shapley values from cooperative game theory: a fair way to split credit among players who contributed to a team result.</p><p><strong>The Restaurant Bill Analogy</strong></p><p>Imagine 3 friends share dinner. Each ordered different things and shared some dishes. How do you split the bill <em>fairly</em>? You ask: <em>“What’s the marginal cost each person added?”</em> — and you check that for every possible ordering of who arrived first, second, third. Then you average. That’s a Shapley value.</p><p>SHAP does the same with features instead of friends, and prediction instead of bill.</p><p><strong>Concrete House Price Example</strong></p><p>The model predicts a house at $300K. The average house is $200K. So we need to explain a +$100K deviation.</p><p>SHAP asks: <em>“On average across all possible ways to add features into the model, how much did each feature contribute?”</em></p><p>For each feature, it considers every subset of other features, measures the prediction <em>with</em> and <em>without</em> that feature, and averages the differences:</p><ul><li>size after no other features → +$40K</li><li>size after location → +$60K</li><li>size after age → +$50K</li><li>size after location + age → +$50K</li><li>… average → size contributes +$50K</li></ul><p>Repeat for every feature:</p><ul><li>size: +$50K</li><li>location: +$40K</li><li>age: +$10K</li></ul><p>The math guarantee (additivity): these always sum exactly to the deviation: $50K + $40K + $10K = $100K ✅</p><p>So we can say to the homeowner: <em>“Your house is predicted at $300K — that’s $100K above average, driven by +$50K from size, +$40K from location, +$10K from age.”</em></p><p><strong>Key Properties</strong></p><ul><li><strong>Local</strong> — per-prediction (can be aggregated to global view)</li><li><strong>Additive</strong> — contributions sum exactly to (prediction − average prediction)</li><li><strong>Signed</strong> — positive or negative direction</li><li><strong>Consistent &amp; fair</strong> — backed by game-theory axioms (efficiency, symmetry, dummy, additivity)</li></ul><p><strong>In Practice</strong></p><p>Computing SHAP exactly checks every subset of features → exponential cost (2^n). For real models, libraries use efficient approximations:</p><ul><li>TreeSHAP — exact and fast for tree models (linear in tree size)</li><li>KernelSHAP — model-agnostic approximation via weighted regression</li><li>DeepSHAP / GradientSHAP — for neural networks.</li></ul><p><strong>How Different Models Get Explained</strong></p><ol><li><strong>Linear / Logistic Regression</strong> — coefficients directly explain feature impact. A coefficient of +0.4 on age means: holding everything else fixed, a one-unit increase in age raises the prediction (or log-odds) by 0.4.</li></ol><p><strong>Setup</strong>: A logistic regression predicting <em>probability of loan default</em> (0 = no default, 1 = default). Features are standardized.</p><p><strong>Trained model</strong>:</p><p>log_odds(default) = -2.5 + 0.8·debt_ratio − 1.2·credit_score + 0.4·age − 0.3·employment_years</p><p><strong>Coefficient table</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a6qoyzRSRFIE5qAHjbZZAA.png" /></figure><p>How you’d explain it to a non-tech stakeholder:</p><p>Credit score is the most powerful predictor — it pulls the risk down hardest. Debt ratio pushes risk up. Age and employment matter, but less. The model is just a weighted sum of these signals.</p><p>Why it’s interpretable: the model <em>is</em> the table. No extra tooling needed.</p><p><strong>2. Shallow Decision Tree — Read the Splits Like a Flowchart</strong></p><p>Setup: Same loan default problem. Tree of depth 3.</p><p>Tree diagram:</p><pre>                       [credit_score &lt; 650?]<br>                       /                    \<br>                    Yes                      No<br>                    /                          \<br>        [debt_ratio &gt; 0.4?]            [employment_years &lt; 2?]<br>         /              \                 /                  \<br>       Yes              No              Yes                   No<br>       /                  \              /                     \<br>  [age &lt; 30?]      Predict:        Predict:              [debt_ratio &gt; 0.6?]<br>   /        \      DEFAULT          NO DEFAULT             /              \<br>  Yes        No    (prob 0.65)      (prob 0.20)          Yes              No<br>  /          \                                            /                \<br>DEFAULT    NO DEFAULT                                 DEFAULT          NO DEFAULT<br>(0.85)     (0.40)                                     (0.55)           (0.05)</pre><p>How to read a single prediction: A 28-year-old applicant with credit_score=620 and debt_ratio=0.5:</p><ol><li>credit_score &lt; 650? → Yes, go left</li><li>debt_ratio &gt; 0.4? → Yes, go left</li><li>age &lt; 30? → Yes, go left</li><li>Prediction: DEFAULT (probability 0.85)</li></ol><p>How you’d explain it to a non-tech stakeholder:</p><p>This applicant was flagged as high risk because their credit score is below 650, their debt ratio is above 40%, and they’re under 30. The model essentially follows a flowchart of yes/no questions, and they ended up in the highest-risk leaf.</p><p>Why it’s interpretable: every prediction is a traceable path through 3 questions. You can hand the flowchart to a loan officer.</p><p><strong>3. Random Forest / Gradient Boosting — Use Feature Importance (Global) and SHAP (Local)</strong></p><p>Setup: Same loan default problem, but now an XGBoost model with 500 trees. You can’t read 500 trees, so you use two views:</p><p><strong>Global view — Feature Importance:</strong></p><pre>Feature Importance (XGBoost)<br><br>credit_score      ████████████████████████  0.42<br>debt_ratio        ██████████████████        0.28<br>employment_years  ███████████               0.15<br>age               █████████                 0.10<br>home_owner        ██                        0.03<br>loan_purpose      █                         0.02<br>                  └─────────────────────────┘<br>                  0.0                       0.5</pre><p>Reading it: <em>“Across all decisions the forest makes on average, credit score and debt ratio together drive ~70% of the decision-making.”</em></p><p>This is global — it tells you what the model relies on <em>in aggregate</em>, but not why any individual was denied.</p><p><strong>Local view — SHAP for one applicant</strong></p><p>Same applicant from before. Average predicted default probability across the dataset = 0.15. This applicant’s prediction = 0.78. SHAP explains the +0.63 deviation:</p><pre>SHAP Explanation: Applicant #4721<br>Base value (avg prediction): 0.15<br>Final prediction:            0.78<br>Deviation to explain:        +0.63<br><br>Feature contributions:<br>credit_score = 620          ████████████████   +0.30  (pushed UP)<br>debt_ratio   = 0.50         ██████████         +0.20  (pushed UP)<br>age          = 28           ████               +0.08  (pushed UP)<br>employment_years = 1.5      ███                +0.07  (pushed UP)<br>home_owner   = Yes          ──                 −0.02  (pushed DOWN)<br>                            └──────────────────────────┘<br>                            Sum = +0.63 ✅ matches the deviation</pre><p>How you’d explain it to the applicant:</p><p>Your predicted default risk is 78%, compared to 15% for the average applicant. The biggest factors are: your credit score of 620 (+30 points of risk), your debt ratio of 50% (+20 points), and your age and employment history (+15 points combined). Owning a home reduced your risk slightly (−2 points).</p><p>Why this matters:</p><ul><li>Feature importance told us <em>the model</em> cares about credit_score the most.</li><li>SHAP tells us <em>this specific person</em> was flagged primarily because of their credit_score — and quantifies it exactly.</li></ul><blockquote><strong>Model Training</strong></blockquote><p><strong>The Big Picture</strong></p><p>Every ML project needs two separate concerns:</p><ol><li>Train-Test Split — always needed, regardless of dataset size. The test set is a complete holdout — unseen data the model never touches until final evaluation.</li><li>Cross-Validation — a quality training technique applied <em>within</em> the training portion. Decides how you validate <em>during</em> training.</li></ol><p>These aren’t alternatives — they work together:</p><pre>Full Dataset<br>├── Training Data (80%) ← CV or train-val split happens HERE<br>│   ├── Train folds / Train portion<br>│   └── Validation folds / Validation portion<br>└── Test Data (20%) ← NEVER touched until final evaluation</pre><p><strong>1. Train-Test Split (Always Required)</strong></p><p>No matter how much data you have, you always need a held-out test set — data the model has never seen during any part of training or tuning. This is what tells you how the model will perform in production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MOCKvCpNOygOfj6Nc2RGwQ.png" /></figure><p><strong>2. What Happens Inside the Training Portion?</strong></p><p>Once you’ve set aside your test set, you need a way to evaluate the model during training (to tune hyperparameters, detect overfitting, pick the best model). Two approaches:</p><p><strong>Option A: Train-Validation-Test Split (Large Datasets)</strong></p><p>When data is abundant, a single fixed validation set is sufficient:</p><pre>Full Data (100%)<br>├── Train (70%) — model learns from this<br>├── Validation (10%) — used DURING training for evaluation, tuning, early stopping<br>└── Test (20%) — used ONCE, AFTER training is complete</pre><p>Why this works for large data: 70% of a large dataset is still plenty of data to learn from. The 10% validation set is large enough to give stable performance estimates.</p><p>Why this is preferred over CV for large data: Cross-validation on millions of rows means training k separate models — computationally expensive with minimal benefit when a single 10% validation set already gives stable estimate</p><p><strong>Option B: Cross-Validation (Small/Medium Datasets)</strong></p><p>When data is limited, you can’t afford to “waste” 10% on a fixed validation set — the remaining training data may be too small, causing the model to overfit (fit to noise, fail to generalize).</p><p>Cross Validation ensures quality training when data is limited.</p><p>Cross-validation solves this by using <em>all</em> of the training data for both training and validation — just not at the same time:</p><p><strong>k-Fold CV</strong> : Data split into <em>k</em> folds → each fold used once as test, <em>k-1</em> times as train → final error = average across folds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RrY0J5-4djpipIMEBXq-ZQ.jpeg" /><figcaption>Example of 5-Fold Cross validation</figcaption></figure><p><strong>Why this is better for small data:</strong></p><ul><li>Every data point gets used for training (in k−1 iterations)</li><li>Every data point gets used for validation (in exactly 1 iteration)</li><li>No data is “wasted” — maximum learning from limited data</li><li>Averaged score is more reliable than a single validation split</li></ul><p><strong>LOOCV (Leave-One-Out)</strong>: k = n (each single data point is its own fold). Maximum data usage, but very expensive — only practical for very small datasets.</p><p><strong>Can you use CV on large datasets?</strong> Yes — it gives more robust estimates. But it’s k times more expensive (train k models instead of 1), and for large data the benefit over a single validation split is marginal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iQ31DQe1oZeK-txApbe9Hw.png" /></figure><p><strong>Time Series Cross-Validation (vs. Temporal Split)</strong></p><p>Key difference:</p><ul><li>Temporal train-test split = one-time split by time. Done once.</li><li>Time Series CV = multiple rolling evaluations, each fold respects time order.</li></ul><p>Time Series CV — Expanding Window:</p><pre>Fold 1: Train [Jan–Mar]      → Test [Apr]<br>Fold 2: Train [Jan–Apr]      → Test [May]<br>Fold 3: Train [Jan–May]      → Test [Jun]<br>Fold 4: Train [Jan–Jun]      → Test [Jul]<br><br>(Training set grows each fold)</pre><p>Time Series CV — Sliding Window:</p><pre>Fold 1: Train [Jan–Mar]      → Test [Apr]<br>Fold 2: Train [Feb–Apr]      → Test [May]<br>Fold 3: Train [Mar–May]      → Test [Jun]<br>Fold 4: Train [Apr–Jun]      → Test [Jul]<br><br>(Training set stays fixed size, slides forward)</pre><p><strong>Why not just random CV for time data?</strong> Random CV would mix future data into training folds → data leakage. The model would see future patterns when predicting the past, giving falsely inflated scores that won’t hold in production.</p><p><strong>Special Training Techniques</strong></p><p><strong>Bootstrapping</strong>:</p><ul><li>Sample with replacement → create multiple slightly different datasets from the same original data.</li><li>Each dataset has some rows repeated, some missing (on average ~63% unique rows per sample).</li><li>Used in: Bagging (e.g., Random Forest — each tree trained on a bootstrap sample → decorrelates trees → reduces variance).</li><li>Also useful for: estimating confidence intervals on metrics, handling imbalanced datasets.</li></ul><p><strong>Hyperparameter Tuning</strong></p><p>Hyperparameters are settings you choose <em>before</em> training (learning rate, regularization strength λ, tree depth, number of layers). Unlike model parameters (weights, coefficients), the model can’t learn these from data — you must search for the best values.</p><p><strong>The Core Idea: Each Hyperparameter Combination = A Different Model</strong></p><p>This is the key insight that connects hyperparameter tuning to cross-validation:</p><p>A linear regression with λ = 0.01 and a linear regression with λ = 10 are treated as two completely separate models. They will learn different weights, make different predictions, and have different performance.</p><p>So hyperparameter tuning is really model selection — you’re choosing which “version” of the model is best.</p><p><strong>How Hyperparameter Tuning + CV Work Together</strong></p><pre>Example: Tuning regularization strength λ for Ridge Regression<br><br>Candidate models:<br>  Model 1: Ridge with λ = 0.001<br>  Model 2: Ridge with λ = 0.01<br>  Model 3: Ridge with λ = 0.1<br>  Model 4: Ridge with λ = 1.0<br>  Model 5: Ridge with λ = 10.0<br><br>For EACH candidate:<br>  → Run full k-fold CV (e.g., 5-fold)<br>  → Get averaged score across all folds<br><br>Results:<br>  Model 1 (λ=0.001): avg CV score = 0.72  (overfitting — too little regularization)<br>  Model 2 (λ=0.01):  avg CV score = 0.81<br>  Model 3 (λ=0.1):   avg CV score = 0.88  ← BEST<br>  Model 4 (λ=1.0):   avg CV score = 0.85<br>  Model 5 (λ=10.0):  avg CV score = 0.70  (underfitting — too much regularization)<br><br>Winner: λ = 0.1<br>→ Retrain on full training data with λ = 0.1<br>→ Evaluate ONCE on held-out test set</pre><p>Why CV for each candidate? If you evaluated each λ on a single fixed validation split, you’d risk picking a λ that happened to perform well on <em>that particular split</em> — overfitting your hyperparameter choice to one specific data arrangement. CV averages across multiple splits, giving a more reliable estimate.</p><p><strong>Search Methods</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jUuuEoLMwbnLGSI6xmRyng.png" /></figure><p><strong>Total Computational Cost</strong></p><p>Total training runs = (number of candidate hyperparameter combos) × (k folds)</p><p>Example: Grid search with 50 combos × 5-fold CV = 250 model trainings</p><p>This is why hyperparameter tuning is expensive, especially with CV — and why Random Search or Bayesian Optimization is preferred when the search space is large.</p><p><strong>Learning Curve</strong></p><ul><li><strong>Plot</strong>: model performance (loss/accuracy) vs. epochs for <strong>train</strong> and <strong>validation</strong> sets.</li><li><strong>Intuition</strong>: Model improves with more epochs (up to a point).</li></ul><p><strong>Usefulness</strong>: Detects</p><ul><li><strong>Underfitting</strong> → high error on both train &amp; validation.</li><li><strong>Overfitting</strong> → low train error, high validation error.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*wXYgK9qgAsv-hcXmZKeLSw.png" /></figure><p><strong>Training Time</strong></p><ul><li>Depends on number of <strong>data points</strong> and <strong>features</strong>.</li><li>High-dimensional or large datasets → <strong>longer training time</strong> + <strong>higher resource usage</strong>.</li><li><strong>Trade-off</strong>: Complex models → more accurate but slower to train and more resource usage.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pRqw_1Idh-E1w_lhGG0yTw.png" /></figure><p>Tradeoff: Complex models are slower to train and more resource-intensive, but <em>may</em> (not always) be more accurate. Start simple, add complexity only if the learning curve and validation metrics justify it.</p><p><strong>Decision Flow chart</strong></p><pre>Do I have temporal/time-series data?<br>├── YES → Use temporal train-test split + Time Series CV<br>└── NO<br>    ├── Is the target imbalanced?<br>    │   ├── YES → Use stratified split + stratified k-fold CV<br>    │   └── NO → Use random split + standard k-fold CV<br>    └── Is the dataset large (&gt;100K rows)?<br>        ├── YES → Train-Val-Test split (70/10/20) likely sufficient<br>        └── NO → Cross-validation strongly recommended</pre><h4><strong>Supervised Learning</strong></h4><blockquote><strong>Linear Regression</strong></blockquote><p><strong>1. Evaluation Metrics Framework (Regression)</strong></p><p><strong>R² (Coefficient of Determination)</strong></p><ul><li>Measures % variance explained by model.</li><li>Always increases with more features → can be misleading.</li></ul><p><strong>Adjusted R²</strong></p><ul><li>Penalizes extra features.</li><li>More reliable for feature comparison.</li></ul><p><strong>MAE (Mean Absolute Error)</strong></p><ul><li>|yi — ŷi|</li><li>Robust to outliers.</li><li>Easy to interpret.</li><li>Convex → Non differentiable at 0 →harder for optimization.</li></ul><p><strong>MSE (Mean Squared Error)</strong></p><ul><li>(yi — ŷi)²</li><li>Penalizes large errors heavily.</li><li>Convex → Differentiable →smooth optimization.</li><li>Sensitive to outliers.</li></ul><p><strong>RMSE</strong></p><ul><li>√MSE</li><li>Same units as target.</li></ul><p><strong>2. Linear Regression Basics</strong></p><p><strong>Formula: </strong>y = wo + w1*x1 + w2*x2 + w3*x3….wn*xn</p><p><strong>Loss Function:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/233/1*XBsAGYfctFPxSdw9_dgguw.png" /></figure><p><strong>Strengths</strong></p><ul><li>Simple &amp; interpretable.</li><li>Fast to train even on large datasets.</li><li>Strong baseline when relationships are linear.</li><li>Low latency during inference.</li></ul><p><strong>3. Core Assumptions</strong></p><ul><li><strong>Linearity →</strong>Relationship between X and y is linear →Check: Scatter plot</li><li><strong>Independence of Errors →</strong>No autocorrelation in y →Check: Durbin–Watson.</li><li><strong>Normality of Residuals →</strong>Distribution of residuals is normal<strong> →</strong>Check: Q-Q plot.</li><li><strong>Homoscedasticity</strong> (constant variance) →Residuals have equal spread →Check: residuals vs. fitted plot.</li></ul><p><strong>4. Diagnostics &amp; Fixes</strong></p><p><strong>A. Heteroscedasticity</strong></p><p><strong>Problem:</strong> Variance increases with fitted values.<br><strong>Signs:</strong> Funnel shape in residual plots.</p><p><strong>Fixes:</strong></p><ul><li>Transform target (log, sqrt).</li><li>Weighted least squares.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*oPYx7nUzvJdqN3U_e1X-sg.jpeg" /></figure><p>Alternative: Scale-location plot for scaled residuals vs scaled fitted</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O-jbk6sbADPMhadHGH7z2w.jpeg" /></figure><p><strong>B. Non-normal Residuals</strong></p><p><strong>Check:</strong> Q-Q plot<br><strong>Fix:</strong> Transform target (log, box-cox).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*kLx_OqsPykPYXiccI35Flg.png" /></figure><p><strong>C. Autocorrelation (Time Series)</strong></p><p><strong>Check:</strong> Durbin-Watson<br><strong>Fix:</strong></p><ul><li>Use ARIMA / SARIMA time series models.</li><li>Add lag features.</li></ul><p><strong>D. Outliers</strong></p><p><strong>Detect:</strong> Cook’s distance.<br><strong>Effects:</strong> Distorts coefficients &amp; predictions.<br><strong>Handle:</strong></p><ul><li>Cap winsorization</li><li>Remove</li><li>Robust regression</li></ul><p><strong>E. Multicollinearity</strong></p><p><strong>Problem:</strong> Features are highly correlated → unstable coefficients.</p><p><strong>Detect:</strong></p><ul><li>VIF &gt; 5 or 10</li><li>High correlation matrix</li></ul><p><strong>Fix:</strong></p><ul><li>Drop features</li><li>PCA</li><li>Regularization (Ridge/Lasso)</li></ul><p><strong>F. Confounding Variables</strong></p><p><strong>Problem:</strong> Hidden variable influences both X and y.</p><p><strong>Detect:</strong></p><ul><li>Domain knowledge</li><li>Correlations across multiple features</li></ul><p><strong>Fix:</strong></p><ul><li>Add confounder as feature</li><li>Stratification</li><li>Use causal methods / instrumental variables</li></ul><p><strong>G. Non-linearity</strong></p><p><strong>Detect:</strong></p><ul><li>Residual pattern (curved)</li><li>Scatter plot of X vs y</li></ul><p><strong>Fix:</strong></p><ul><li>Polynomial features</li><li>Transformations</li><li>Switch to tree/boosting models</li></ul><p><strong>5. Overfitting &amp; Regularization</strong></p><p><strong>When Overfitting Happens</strong></p><ul><li>Too many features (# features &gt;&gt; # data points)</li><li>High noise</li><li>Small dataset</li></ul><p><strong>Fix: Regularization</strong></p><p><strong>Ridge (L2)</strong></p><ul><li>Shrinks coefficients smoothly.</li><li>Good for multicollinearity.</li></ul><p><strong>Lasso (L1)</strong></p><ul><li>Stricter shrinkage method</li><li>Performs feature selection (zeroes out weights).</li></ul><p><strong>Elastic Net</strong></p><ul><li>Combination of L1 &amp; L2.</li></ul><p><strong>Key Hyperparameter:</strong></p><ul><li><strong>λ (lambda)</strong> → Regularization strength<br>Higher λ → High Bias &amp; low variance →simpler model</li></ul><p><strong>6. Data Preparation Checklist</strong></p><ul><li>✔ Encode categorical features (one-hot, label encode)</li><li>✔ Scale features (essential for regularization)</li><li>✔ Handle missing values (mean/median imputation)</li><li>✔ Adequate sample size (preferably &gt; #features)</li></ul><p><strong>7. When to Use Linear Regression</strong></p><p><strong>Good Fit (Use LR)</strong></p><ul><li>Features have linear relationship</li><li>Interpretability required</li><li>Large sample size</li><li>Low noise</li><li>Fast baseline needed</li></ul><p><strong>Bad Fit (Avoid LR)</strong></p><ul><li>Non-linear patterns exist</li><li>Too many categorical variables</li><li>Small data + many features</li><li>Violated assumptions (heteroscedasticity, autocorrelation)</li></ul><blockquote><strong>Time Series Models</strong></blockquote><ol><li><strong>What is Time Series Data?</strong></li></ol><p>A dataset where:</p><ul><li>Observations (target) are ordered in time.</li><li>There is dependence between past and future values of target.</li><li>Patterns exist across time (trend, seasonality).</li></ul><p><strong>Examples:</strong> sales per day, weather, stock price, electricity usage.</p><p><strong>Key feature:</strong><br> 👉 <strong>Order matters</strong><br> 👉 <strong>Past influences future</strong><br> 👉 <strong>Target is autocorrelated</strong></p><p><strong>2. Non-Time Series vs Time Series vs Temporal Data</strong></p><p><strong>✔ Non-time-series data (normal ML)</strong></p><ul><li>Order doesn’t matter</li><li>Rows independent</li><li>No autocorrelation</li><li>Random train–test split OK</li></ul><p>Example: churn prediction, house price, credit risk.</p><p><strong>✔ Temporal data (ML + time features)</strong></p><ul><li>There <strong>is time</strong>, but Target <strong>does NOT depend on past target</strong></li><li>Use time-based train/test split (NO random shuffling)</li></ul><p>Example: ETA prediction, fraud detection timestamp features.</p><p>👉 You extract hour/day, but target has no autocorrelation.</p><p><strong>✔ Time Series Forecasting</strong></p><ul><li>Goal = predict <strong>future of same variable</strong></li><li>Target <strong>depends on its own past</strong></li><li>Must use forecasting algorithms</li><li>Train on past, predict next timestamps</li></ul><p>Example: predict tomorrow’s sales.</p><p><strong>3. Why Traditional ML Cannot Handle Time Series Directly</strong></p><p>ML models assume:</p><ol><li>Data points are <strong>independent</strong></li><li>Distribution does <strong>not change over time</strong></li><li>Order <strong>does not matter</strong></li><li>Features must be manually created</li></ol><p>But time series violates all of these.</p><p>Therefore, ARIMA/SARIMA are specialized to use <strong>past values as features</strong>.</p><p><strong>4. How to Know If It’s a Forecasting Problem</strong></p><ul><li>You want to predict <strong>future values</strong> of the <em>same series</em></li><li>Target has <strong>autocorrelation</strong></li><li>You have <strong>one main time column</strong></li><li>Use plots &amp; ACF/PACF to confirm</li></ul><p>If target <strong>doesn’t</strong> correlate with its past:</p><p>👉 Use normal ML with temporal features.<br>👉 NOT ARIMA/SARIMA.</p><p><strong>ARIMA</strong></p><ol><li><strong>What is ARIMA?</strong></li></ol><p>ARIMA = <strong>A</strong>uto <strong>R</strong>egressive <strong>I</strong>ntegrated <strong>M</strong>oving <strong>A</strong>verage<br>Written as <strong>ARIMA(p, d, q)</strong></p><p>Used when:</p><ul><li>Data has trend</li><li>No seasonality (else use SARIMA)</li><li>Autocorrelation exists</li><li>Only target’s history is used</li></ul><p><strong>2. ARIMA Components</strong></p><p><strong>AR (Auto-Regressive) — “memory of the series”</strong></p><ul><li>Uses past values to predict the next.</li></ul><p>Example:<br> If yesterday was high → today likely high.</p><p>p = number of past values used</p><p><strong>I (Integrated) — “trend remover”</strong></p><ul><li>Differencing to make data stationary:</li></ul><pre>y_diff(t) = y(t) - y(t-1)</pre><p>Purpose:</p><ul><li>Remove trend → model becomes stable.</li></ul><p>d = number of differencing steps</p><p><strong>MA (Moving Average) — “error corrector”</strong></p><ul><li>Uses past forecast errors to adjust prediction.</li><li>If you underpredicted yesterday, MA corrects today.</li></ul><p>q = number of past errors used</p><p><strong>3. ARIMA(1,1,1) Meaning</strong></p><ul><li><strong>AR(1)</strong> → uses previous differenced value</li><li><strong>I(1)</strong> → one differencing to remove trend</li><li><strong>MA(1)</strong> → uses previous forecast error</li></ul><p>Equation:</p><pre>Predicted_difference = φ * y_diff(t−1) + θ * error(t−1)</pre><p>Then:</p><pre>predicted_y(t) = y(t−1) + predicted_difference</pre><p><strong>4. When to Use ARIMA</strong></p><p>Use ARIMA when:</p><ul><li>You have <strong>one univariate series (</strong>one column + time)</li><li>There’s <strong>trend</strong> but <strong>no strong seasonality</strong></li><li>You want to forecast <strong>future points</strong></li><li>Past target value influences the future (autocorrelation)</li></ul><p>Examples:</p><ul><li>sales forecasting</li><li>temperature forecasting</li><li>energy usage forecasting</li></ul><p>Not for:</p><ul><li>ETA prediction</li><li>churn classification</li><li>credit risk<br> (because target is NOT autocorrelated)</li></ul><p>5. <strong>Evaluation Metrics for ARIMA</strong>: MAE,MSE, RMSE</p><p><strong>SARIMA</strong></p><p><strong>1. What is SARIMA?</strong></p><p><strong>SARIMA = Seasonal ARIMA </strong>= ARIMA + ability to capture repeating cycles<br>It extends ARIMA by <strong>adding seasonal terms</strong> to capture repeating patterns (monthly, weekly, quarterly, yearly).</p><p><strong>Notation: </strong>SARIMA(p, d, q)(P, D, Q)m</p><p>Where:</p><ul><li><strong>p, d, q</strong> → Non-seasonal AR, differencing, MA</li><li><strong>P, D, Q</strong> → Seasonal AR, seasonal differencing, seasonal MA</li><li><strong>s</strong> → Seasonal period (12 for monthly, 7 for daily weekly-pattern, 4 for quarterly)</li></ul><p><strong>2. When to Use SARIMA?</strong></p><p>Use SARIMA when:</p><ul><li>Data shows <strong>seasonal patterns</strong> (e.g., sales spike every December)</li><li>ACF/PACF show peaks at seasonal lags (e.g., lag 12, lag 24, …)</li><li>ARIMA residuals still show seasonality → SARIMA fixes it</li></ul><p><strong>3. Difference Between ARIMA vs SARIMA</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*YQGSO-jxuhkJynD2zNCdPQ.png" /></figure><p><strong>4. Components of SARIMA</strong></p><p><strong>Non-seasonal part (p, d, q):</strong></p><ul><li><strong>p (AR):</strong> relationship with previous values</li><li><strong>d (diff):</strong> remove trend</li><li><strong>q (MA):</strong> relationship with past forecast errors</li></ul><p><strong>Seasonal part (P, D, Q, s):</strong></p><ul><li><strong>P (Seasonal AR):</strong> relation with values periods ago</li><li><strong>D (Seasonal differencing):</strong> remove repeating seasonal pattern</li><li><strong>Q (Seasonal MA):</strong> dependence on errors periods ago</li><li><strong>s:</strong> length of season (12, 7, 4, 52, etc.)</li></ul><p><strong>5. SARIMA Model Equation (Intuition)</strong></p><p>SARIMA combines:</p><ul><li>ARIMA part → captures short-term patterns</li><li>Seasonal AR/MA → captures repeating cycles</li><li>Differencing → removes trend &amp; seasonal drift</li></ul><p><strong>6. Forecasting</strong></p><p>SARIMA can:</p><ul><li>Forecast <strong>multiple seasons ahead</strong></li><li>Provide <strong>confidence intervals</strong></li><li>Capture repeating seasonal effects in the forecast curve</li></ul><blockquote><strong>Classification</strong></blockquote><p><strong>Types of Models</strong></p><ul><li><strong>Generative Models</strong>: Learn <strong>P(X|Y)</strong> and <strong>P(Y)</strong> → use Bayes’ rule to compute <strong>P(Y|X)</strong>. (e.g., Naive Bayes)</li><li><strong>Discriminative Models</strong>: Learn <strong>P(Y|X)</strong> directly → map features to labels. (e.g., Logistic Regression, SVM)</li></ul><p><strong>Evaluation Metrics Framework</strong></p><ul><li><strong>Model Selection</strong> → Use <strong>AUC</strong> (threshold-independent, overall class separation).</li><li><strong>Threshold Tuning</strong> → Use <strong>ROC curve</strong> (TPR vs FPR) or <strong>Precision-Recall curve</strong> (for highly imbalanced data).</li></ul><p><strong>Business Alignment</strong> →</p><ul><li><strong>Precision</strong>: prioritize when false positives are costly.</li><li><strong>Recall</strong>: prioritize when false negatives are costly.</li><li><strong>F1-score</strong>: balance precision &amp; recall.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/1*GDvOnasU1LDW2VRxwQ63ng.png" /></figure><ul><li><strong>Accuracy</strong> → Use as a sanity check; can be misleading for imbalanced datasets.</li></ul><blockquote><strong>Logistic Regression</strong></blockquote><ul><li><strong>Classification algorithm</strong> that predicts probability of binary outcomes using the <strong>sigmoid function</strong></li><li><strong>Key Insight</strong>: Models the <strong>linear relationship between features</strong> and <strong>log-odds</strong>, then converts to probability using sigmoid function.</li></ul><p><strong>Core Equation:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*RT_UHHFH5R3KWs3-hK2BhQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*msVIxvugbl0_G57bzdizpQ.png" /></figure><p><strong>Decision Rule</strong>:</p><ul><li>P(y=1|x) ≥ threshold(default: 0.5) → Class 1</li><li>P(y=1|x) &lt; threshold(default: 0.5)→ Class 0</li></ul><p><strong>Loss Function: Log-Loss (Binary Cross-Entropy)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*h5DOeb67sfUs8GTEzgiBdQ.png" /></figure><ul><li>Log Loss is Convex → guarantees global minimum.</li><li>It Penalizes wrong confident predictions heavily.</li></ul><p><strong>Training Process (8 Steps)</strong></p><ol><li><strong>Initialize</strong>: Random weights</li><li><strong>Compute z</strong>: Linear combination w^T·x</li><li><strong>Apply Sigmoid</strong>: Convert to probability</li><li><strong>Calculate Loss</strong>: Log-loss function</li><li><strong>Compute Gradients</strong>: ∂Loss/∂w</li><li><strong>Update Weights</strong>: w_new = w_old — α·∇Loss</li><li><strong>Repeat</strong>: Until convergence</li><li><strong>Predict</strong>: Use trained weights for new data</li></ol><p><strong>Binary vs Multiclass</strong></p><ul><li><strong>Binary</strong> → direct logistic regression.</li><li><strong>Multiclass</strong> → One-vs-Rest or Softmax regression.</li></ul><p><strong>Critical Assumptions</strong></p><ul><li>✅ <strong>Linear relationship</strong> between features and log-odds</li><li>✅ <strong>Independent observations</strong> (no autocorrelation)</li><li>✅ <strong>No multicollinearity</strong> between features</li><li>✅ <strong>Large sample size</strong> for stable estimates</li><li>✅ <strong>No extreme outliers</strong></li></ul><p><strong>Violation Impact &amp; Fixes:</strong></p><ul><li>Non-linearity → Underfitting→ polynomial features / switch to tree models.</li><li>Multicollinearity → Unstable coefficients → Effects Interpretability → VIF check, remove correlated features, PCA, regularization.</li><li>Outliers → Overfitting →Biased decision boundary →detect (Cook’s distance), remove/cap/robust scaling.</li></ul><p><strong>Overfitting</strong> → more data, feature selection, regularization (L1/L2).</p><p><strong>Imbalanced data</strong> → use precision/recall, resampling, class weights.</p><p><strong>Strengths</strong></p><p>✅ Fast &amp; simple<br> ✅ Interpretable (coefficients = log-odds ratios)<br> ✅ Probabilistic output<br> ✅ Baseline for classification<br> ✅ Works well with regularization</p><p><strong>Weaknesses</strong></p><p>❌ Assumes linear relation b/w features and log-odds<br> ❌ Sensitive to outliers<br> ❌ Needs feature engineering for non-linearity<br> ❌ Struggles with imbalanced data<br> ❌ Cannot handle missing values directly</p><p><strong>Key Hyperparameter</strong></p><ul><li><strong>λ (lambda)</strong> — Regularization strength in L1/L2/Elastic Net</li><li>Higher λ = more regularization = low Variance &amp; high Bias=simpler model</li></ul><p><strong>Key Interview Notes</strong></p><ul><li>Why Sigmoid: Sigmoid → smooth, differentiable, maps to (0,1) and S-shaped.</li><li>Why Log-Loss: MSE + sigmoid → non-convex; Log-loss → convex.</li><li>Coefficient Interpretation: Coeff interpretation: exp(β) = odds ratio (e.g., β=0.8 → odds ×2.23).</li></ul><p><strong>Real World Use Cases</strong></p><ul><li>Medical diagnosis (disease / no disease).</li><li>Finance (default / no default).</li><li>Marketing (click / no click).</li><li>Spam detection.</li></ul><p><strong>When to Use</strong></p><p>✅ Need interpretability<br> ✅ Quick baseline before complex models<br> ✅ Linear relationship with log-odds<br> ✅ Real-time predictions</p><blockquote><strong>Naive Bayes</strong></blockquote><p><strong>Probabilistic classifier</strong> using Bayes’ Theorem with <strong>“naive” conditional independence assumption</strong></p><ul><li><strong>Generative model</strong>: Learns P(x|y) and P(y), not weights</li><li><strong>No iterative training</strong>: Just counting frequencies and computing probabilities</li></ul><p><strong>Mathematical Foundation</strong></p><pre><br>Core Equation<br>P(y=class_k|x) = P(y=class_k) × ∏ P(xᵢ|y=class_k)/P(x)<br>Note: We don&#39;t need to compute P(x) since it&#39;s constant across all classes.<br>P(y=class_k|x) ∝ P(y=class_k) × ∏ P(xᵢ|y=class_k)<br><br>Prediction Rule<br>ŷ = argmax[P(y=class_k) × ∏ P(xᵢ|y=class_k)]</pre><p><strong>Training (3 steps)</strong></p><ol><li>Compute priors P(y).</li><li>Compute likelihoods P(xi∣y) → variant-specific(Multinomial, Gaussian or Bernoulli NB)</li></ol><p>3. Apply <strong>Laplace smoothing</strong> (α) to avoid zero probabilities.</p><p><strong>Key Variants</strong></p><ul><li><strong>Multinomial NB</strong>: Text/count data (word frequencies)</li><li><strong>Gaussian NB</strong>: Continuous features (assumes normal distribution)</li><li><strong>Bernoulli NB</strong>: Binary features</li></ul><p><strong>Assumptions</strong></p><ol><li>Naive Bayes assumes each X_i is independent of any other X_j given Y for any pair of features X_i and X_j.</li><li>It assumes each feature is given the same weight.</li></ol><p><strong>Main Assumption</strong>: P(x₁,x₂,…,xₙ|y) = P(x₁|y) × P(x₂|y) × … × P(xₙ|y)</p><p><strong>Advantages</strong></p><ul><li><strong>No gradient descent</strong> or iterative optimization, Fastest among classical ML algorithms.</li><li>Perfect for real-time applications.</li><li><strong>Extremely fast</strong> training and prediction.</li><li>Works well with <strong>small datasets.</strong></li><li><strong>Scales efficiently</strong> to large data.</li><li><strong>Probabilistic outputs.</strong></li><li>Very interpretable.</li><li>Minimal hyperparameter tuning.</li><li>Naturally supports multiclass classification.</li></ul><p><strong>Disadvantages</strong></p><ul><li>Strong independence assumption (often violated).</li><li><strong>High bias</strong> → prone to underfitting.</li><li>Cannot handle missing values directly.</li></ul><p><strong>Common Issues &amp; Fixes</strong></p><ul><li><strong>Zero probability</strong> →P(x|y) = 0-&gt; Laplace smoothing.</li><li><strong>Continuous Features → </strong>Gaussian NB(assumes normal distribution) or Discretize into bins</li><li><strong>Correlated features</strong> →Double-counting effect → overconfident predictions-&gt; feature selection / PCA.</li><li><strong>Missing values</strong> → imputation.</li></ul><p><strong>Hyperparameter</strong></p><ul><li>Only α (smoothing parameter).</li></ul><p><strong>When to Use Naive Bayes</strong></p><ul><li><strong>Text/NLP tasks</strong> (spam detection, sentiment analysis)</li><li>Small datasets, real-time predictions, baseline models, or when speed matters.</li><li>High-dimensional sparse data.</li></ul><p><strong>Real-World Applications</strong></p><ul><li>Email spam detection</li><li>Text classification</li><li>Document categorization</li><li>Medical diagnosis (independent symptoms)</li><li>Real-time recommendation systems</li></ul><p><strong>Why does it work despite “naive” assumption?</strong></p><ol><li><strong>Relative ranking matters</strong> more than exact probabilities</li><li><strong>Text data</strong> naturally fits independence assumption better</li><li><strong>Bias-variance trade-off</strong>: High bias but low variance</li></ol><blockquote><strong>Support Vector Machine(SVM)</strong></blockquote><p><strong>1. Core Concept</strong></p><ul><li>Goal: To form a hyperplane that linearly seperates the training data</li><li>Finds <strong>optimal hyperplane</strong> that <strong>maximally separates</strong> classes</li><li>Uses only <strong>support vectors</strong> (critical boundary points which are closest to hyperplane)</li><li>The minimum distance from the decision boundry to any training point →<strong>Margin = 2/||w||</strong> → Maximize for better generalization</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*P_7szqcuCwxdzhK5mRluyw.png" /><figcaption>For Linear Data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*oDksheYAj1eP0Be-a6r_qQ.png" /><figcaption>For non-linear data</figcaption></figure><p><strong>2. Key Equation</strong></p><ul><li><strong>Decision</strong>: f(x) = w^T·x + b</li><li>If f(x) &gt; 0 → Class +1, else Class -1</li></ul><p><strong>3. SVM Variants</strong></p><ul><li><strong>Hard Margin</strong>: Perfect separation (no errors allowed)</li><li><strong>Soft Margin</strong>: Allows misclassification via parameter <strong>C</strong></li><li><strong>Kernel SVM</strong>: Handles non-linear data</li></ul><p><strong>4. Kernel Trick</strong></p><p>Maps data to higher dimensions without explicit computation</p><ul><li>Linear problems → Linear Kernel</li><li>Non Linear problems → Non Linear Kernel(like RBF)</li></ul><p><strong>5. Key Hyperparameters</strong></p><ul><li><strong>C</strong>: Regularization (Low C → wider margin, High C → narrower margin)</li><li><strong>Kernel</strong>: ‘linear’, ‘rbf’, ‘poly’</li><li><strong>Gamma</strong> (RBF): Low → smooth boundary, High → complex boundary</li></ul><p><strong>6. Computational Complexity</strong></p><ul><li><strong>Training</strong>: O(n²) to O(n³) [Kernel matrix computation bottleneck]</li><li><strong>Space</strong>: O(n²) for kernel matrix</li><li><strong>Not scalable</strong> for large datasets (&gt;100K)</li></ul><p><strong>7. Critical Requirements</strong></p><ul><li><strong>Feature scaling mandatory</strong> (StandardScaler)</li><li>Cannot handle missing values (need imputation)</li><li>Encode categorical features</li></ul><p><strong>8. Strengths</strong></p><ul><li>Effective in <strong>high-dimensional spaces</strong></li><li><strong>Memory efficient</strong> (stores only support vectors)</li><li>Robust to overfitting in high dimensions</li><li>Works well with <strong>small-medium datasets</strong></li></ul><p><strong>9. Weaknesses</strong></p><ul><li><strong>Not scalable</strong> to large datasets</li><li><strong>High computational cost</strong> (kernel computation)</li><li>Poor on <strong>overlapping classes</strong></li><li>Less <strong>interpretable</strong></li><li>Not suitable for <strong>real-time</strong> applications</li></ul><p><strong>10. When to Use</strong></p><ul><li>✅ High-dimensional data (features &gt; samples)</li><li>✅ Clear class separation</li><li>✅ Small-medium datasets</li><li>✅ Non-linear relationships (with kernels)</li><li>❌ Large datasets (&gt;100K)</li><li>❌ Real-time predictions</li><li>❌ Need interpretability</li></ul><p><strong>11. Interview Key Points</strong></p><ul><li><strong>Margin maximization</strong> → Better generalization</li><li><strong>Support vectors</strong> define the boundary</li><li><strong>Kernel trick</strong> avoids explicit high-dimensional mapping</li><li><strong>Feature scaling critical</strong> (distance-based algorithm)</li><li><strong>C vs Gamma trade-off</strong>: C controls margin-error, Gamma controls boundary complexity</li></ul><blockquote><strong>Tree Based Algorithms (Both Classification &amp; Regression)</strong></blockquote><blockquote><strong>1. Decision Trees</strong></blockquote><p><strong>1. Core Concept</strong></p><ul><li><strong>Hierarchical structure</strong> with step-by-step decision rules</li><li><strong>CART</strong>: Classification And Regression Trees</li><li>Works for both <strong>regression</strong> (continuous target) and <strong>classification</strong> (categorical target)</li></ul><p><strong>2. Key Intuition</strong></p><ul><li><strong>Regression</strong>: Split to <strong>minimize variance (MSE)</strong> in target values</li><li><strong>Classification</strong>: Split to <strong>maximize purity</strong> (minimize Gini impurity/entropy)</li></ul><p><strong>3. Splitting Criteria</strong></p><p><strong>Regression Trees</strong></p><ul><li><strong>MSE/SSE</strong>: Minimize variance in each subset</li><li><strong>Leaf prediction</strong>: Mean/Median of target values in that node</li></ul><p><strong>Classification Trees</strong></p><ul><li><strong>Gini Impurity</strong>: Gini = 1 — Σ(pi²)</li><li><strong>Information Gain</strong>: Based on entropy reduction</li><li><strong>Leaf prediction</strong>: Majority class in that node</li></ul><p><strong>4. Training Process</strong></p><p><strong>a. Select best split</strong>: Try all features and thresholds</p><p><strong>b. Evaluate split quality</strong>: Use MSE (regression) or Gini (classification)</p><p><strong>c. Recursive splitting</strong>: Repeat until stopping criteria met</p><p><strong>d. Stopping criteria</strong>:</p><ul><li>Max depth reached</li><li>Min samples per split/leaf</li><li>No further error reduction</li><li>Pure node (classification)</li></ul><p><strong>5. Key Hyperparameters</strong></p><ul><li><strong>max_depth</strong>: Controls tree depth (prevent overfitting)</li><li><strong>min_samples_split</strong>: Min samples to split a node</li><li><strong>min_samples_leaf</strong>: Min samples in leaf node</li><li><strong>max_leaf_nodes</strong>: Limits total leaves</li><li><strong>max_features</strong>: Features to consider per split</li><li><strong>criterion</strong>: ‘squared_error’ (regression), ‘gini’/’entropy’ (classification)</li></ul><p><strong>6. Strengths</strong></p><ul><li><strong>No assumptions</strong> about data distribution</li><li><strong>Handles non-linear</strong> relationships naturally</li><li><strong>Works with mixed data</strong> (numerical + categorical)</li><li><strong>No feature scaling</strong> required</li><li><strong>Highly interpretable</strong> (flowchart logic)</li><li><strong>Automatic feature selection</strong></li><li><strong>Handles missing values</strong> (with surrogate splits)</li></ul><p><strong>7. Weaknesses</strong></p><ul><li><strong>Prone to overfitting</strong> (high variance, low bias)</li><li><strong>Unstable</strong>: Small data changes → different tree structure</li><li><strong>Not smooth predictions</strong>: Step-wise outputs</li><li><strong>Lower accuracy</strong> than ensemble methods</li><li><strong>Biased</strong> towards dominant classes (imbalanced data)</li><li><strong>Greedy algorithm</strong>: May miss globally optimal splits</li></ul><p><strong>8. Handling Common Issues</strong></p><p><strong>Overfitting (Main Issue!)</strong></p><ul><li><strong>Pre-pruning</strong>: Set max_depth, min_samples_split/leaf</li><li><strong>Post-pruning</strong>: Remove branches after training</li><li><strong>Ensemble methods</strong>: Random Forest (bagging), XGBoost (boosting)</li></ul><p><strong>Outliers</strong></p><ul><li><strong>Regression</strong>: Affects leaf mean predictions</li><li><strong>Classification</strong>: Can create unnecessary deep branches</li><li><strong>Solution</strong>: Preprocessing, pruning, or use ensembles</li></ul><p><strong>Multicollinearity</strong></p><ul><li><strong>Minimal effect</strong>: Tree selects one feature at a time</li><li>No coefficient instability like linear models</li></ul><p><strong>9. Flexibility &amp; Preprocessing</strong></p><ul><li><strong>Highly flexible</strong> (non-parametric)</li><li><strong>No encoding required</strong> (handles categorical directly)</li><li><strong>No scaling needed</strong> (split-based, not distance-based)</li><li><strong>Less preprocessing</strong> than linear models/SVM</li></ul><p><strong>10. Bias-Variance Trade-off</strong></p><ul><li><strong>Default</strong>: Low bias, High variance</li><li><strong>Shallow tree</strong>: High bias, Low variance (underfitting)</li><li><strong>Deep tree</strong>: Low bias, High variance (overfitting)</li></ul><p><strong>11. Dataset Size</strong></p><ul><li><strong>Small data</strong>: Use regularization or prefer linear models</li><li><strong>Large data</strong>: Trees perform better with more samples</li><li><strong>Best practice</strong>: Start simple, tune carefully</li></ul><p><strong>12. Training Time</strong></p><ul><li><strong>Faster than</strong>: Neural networks, ensemble methods</li><li><strong>Slower than</strong>: Linear regression, Logistic regression</li><li><strong>Complexity</strong>: O(n × m × log n) where n=samples, m=features</li></ul><p><strong>13. When to Use</strong></p><ul><li>✅ Need <strong>interpretability</strong></li><li>✅ <strong>Mixed data types</strong> (numerical + categorical)</li><li>✅ <strong>Non-linear relationships</strong></li><li>✅ <strong>Quick baseline</strong> model</li><li>✅ <strong>No time for preprocessing</strong></li><li>❌ Need <strong>high accuracy</strong> (use ensembles instead)</li><li>❌ <strong>Small datasets</strong> (risk of overfitting)</li><li>❌ Need <strong>smooth predictions</strong></li></ul><p><strong>14. Interview Key Points</strong></p><ul><li><strong>Greedy algorithm</strong>: Locally optimal splits (not globally optimal)</li><li><strong>CART</strong>: Uses binary splits only</li><li><strong>Overfitting</strong>: Main weakness, control with hyperparameters</li><li><strong>Interpretability vs Accuracy</strong>: Trees = interpretable, Ensembles = accurate</li><li><strong>No equations with weights</strong>: Predictions based on leaf statistics</li><li><strong>Foundation for ensembles</strong>: Basis for Random Forest, Boosting methods</li></ul><blockquote><strong>Random Forests — Bagging</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*8YGwErt3CuSRkHtF4PdwYQ.jpeg" /></figure><p><strong>Concept:</strong></p><ul><li>A <strong>single decision tree</strong> is highly flexible and tends to <strong>overfit</strong> the training data.</li><li><strong>Random Forest</strong> builds <strong>many decision trees</strong> on <strong>different bootstrap samples</strong> of the data.</li><li>Additionally, at each split, only a <strong>random subset of features</strong> is considered → introduces diversity among trees.</li><li><strong>Final prediction</strong> is made by <strong>averaging</strong> (regression) or <strong>majority vote</strong> (classification) across all trees.</li><li>This ensemble approach <strong>reduces variance</strong> and <strong>improves generalization</strong>, mitigating overfitting compared to a single tree.</li></ul><p><strong>Key Strengths:</strong></p><ul><li>Handles numeric &amp; categorical data.</li><li>Captures non-linear patterns without assumptions.</li><li>Robust to outliers &amp; irrelevant features.</li><li>Built-in feature importance.</li><li>Works well with high-dimensional data.</li><li>Uses Out-of-Bag (OOB) score for internal validation.</li></ul><p><strong>OOB Error Detail:</strong></p><ul><li><strong>Out-of-Bag (OOB) data</strong> = the samples <strong>not selected</strong> in the bootstrap sample for a particular tree.</li><li>These “left out” samples can be used to <strong>test the tree</strong> without needing a separate validation set.</li></ul><p><strong>Workflow:</strong></p><ol><li><strong>Bootstrap Sampling</strong> for each tree.</li><li>At each split, pick random subset of features.</li><li>Grow tree fully/pre-prune using parameters.</li><li>Aggregate predictions from all trees.</li></ol><p><strong>Handling Data Issues:</strong></p><ul><li>Missing values → Impute before training (scikit-learn).</li><li>Robust to correlated features for prediction (feature importance may dilute).</li><li>Outliers have minimal effect due to averaging.</li></ul><p><strong>Hyperparameters:</strong></p><ul><li>n_estimators (number of trees).</li><li>max_depth, min_samples_leaf, max_features.</li><li>Tune using OOB score or cross-validation.</li></ul><p><strong>When to Use:</strong></p><ul><li>Complex tabular data.</li><li>Both small &amp; large datasets (watch for overfit on small data).</li><li>High accuracy with minimal tuning needed.</li></ul><p><strong>Limitations:</strong></p><ul><li>Slower &amp; more resource-heavy than single tree.</li></ul><p><strong>Note</strong>: Random forests are often used due to their versatility, interpretability (through feature importance), quick training times (they can be trained parallelly) and prediction performance.</p><blockquote><strong>Boosting</strong></blockquote><p><strong>Sequential ensemble</strong> where each weak learner (usually shallow tree) <strong>corrects mistakes</strong> of previous learners.</p><p><strong>Key Difference from Random Forest:</strong></p><ul><li>Random Forest: Trees trained <strong>independently in parallel</strong> → reduces variance</li><li>Boosting: Trees trained <strong>sequentially</strong> → reduces bias</li></ul><blockquote><strong>1. Adaboost (Adaptive Boosting)</strong></blockquote><p><strong>Core Idea</strong></p><ul><li><strong>Adaptive</strong>: Updates sample weights to focus on previous errors.</li><li><strong>Boosting</strong>: Turns weak learners into a strong model via sequential training.</li><li>Misclassified samples get higher weights → next model focuses on hard cases.</li><li>Works for classification by focusing on misclassified labels.</li><li>Works for regression by focusing on high-error predictions.</li><li>Common base learner: Decision Tree Stump (max_depth=1). Converts weak learners (often decision stumps) into a strong ensemble.</li></ul><p><strong>AdaBoost for Classification</strong></p><ol><li>Initialize weights equally for all training samples.</li><li>Train a weak learner (e.g., decision stump) on weighted data minimizing weighted gini impurity.</li><li>Compute weighted error ϵ=∑wi×1(y^_i≠y_i).</li><li>Compute model weight (“amount of say”):<br>α=0.5*ln⁡(1-ϵ/ϵ)</li><li>Update sample weights:<br>wi←wi×(e^(-<em>α*y_i*y</em>^_<em>i)</em></li><li>Normalize weights so they sum to 1.</li><li>Repeat training next learner on updated weights.</li><li>Final prediction by sign of weighted sum:<br>F(x)=sign⁡(∑(learning rate*amount of say*hm(x))</li></ol><p><strong>AdaBoost for Regression (AdaBoost.R2)</strong></p><ol><li>Initialize equal weights to samples.</li><li>Train weak regressor minimizing weighted MSE.</li><li>Compute absolute errors:<br>Li=∣y_i−y^_i∣</li><li>Normalize errors:<br>e_i=Li/max⁡_j(L_j)</li><li>Compute total weighted error:<br>ϵ=∑w_i*e_i</li><li>Compute model weight:<br>β=ϵ/1−ϵ,α=ln⁡(1/β)</li><li>Update weights:<br>wi←wi×β^(1−e_i)</li><li>Normalize weights.</li><li>Repeat training next regressor.</li><li>Final prediction is weighted average:<br>y^=∑(α_m*y^_m)/∑α_m</li></ol><p><strong>Bias–Variance</strong></p><ul><li>AdaBoost: Low bias, high variance (prone to overfit noisy data).</li></ul><p><strong>Strengths</strong></p><ul><li>Low bias (adapts to hard cases)</li><li>Works with any weak learner</li><li>No feature scaling needed</li><li>Flexible — base learner choice controls complexity.</li></ul><p><strong>Weaknesses</strong></p><ul><li><strong>Very sensitive to outliers/noise</strong> (gives them huge weights!)</li><li>Sequential training (slow)</li><li>Prone to overfitting on noisy data</li><li>No built-in regularization</li><li>Doesn’t handle missing values</li></ul><p><strong>When to Use</strong></p><ul><li>Clean, structured tabular data(no outliers)</li><li>Need high accuracy &amp; low bias.</li><li>Model should improve on hard cases.</li></ul><p><strong>Key Hyperparameters</strong></p><ul><li>n_estimators (rounds): 50–500.</li><li>learning_rate: 0.01–1.0.</li><li>base_estimator depth: 1–3.</li><li>loss: &quot;linear&quot; (robust), &quot;square&quot; (aggressive).</li></ul><p>Underfitting → ↑ n_estimators, ↑ max_depth.<br>Overfitting → ↓ max_depth, ↓ n_estimators, ↓ learning_rate.</p><blockquote><strong>2. Gradient Boosting</strong></blockquote><p><strong>Core Idea:</strong></p><ul><li>Gradient Boosting builds a strong model by sequentially adding weak learners (usually shallow decision trees).</li><li>Instead of reweighting samples, <strong>fit new models to residuals</strong> (errors) using gradient descent. Each learner corrects the errors (residuals) made by the combined ensemble of previous learners.</li><li>Uses the gradient of a loss function to guide each learner’s training — essentially performing gradient descent in function space.</li></ul><p><strong>Key Insight</strong></p><ul><li><strong>AdaBoost</strong>: Focuses on hard samples via weights</li><li><strong>Gradient Boosting</strong>: Focuses on errors directly via residuals</li></ul><p><strong>How Gradient Boosting Works (Regression Case)</strong></p><ol><li>Initialize with a simple model: F0(x)=mean(y)</li><li>For iteration m=1→M:</li></ol><ul><li>Compute residuals (negative gradients): r_i^m=y_i−F_(m-1)(x_i)</li><li>Fit weak learner hm(x) to residuals ri</li><li>Update ensemble:<br>Fm(x)=F_(m−1)(x)+η×h_m(x)</li></ul><p>Here, η is the learning rate controlling contribution size of each learner.</p><p><strong>Intuition</strong></p><ul><li>Start with a rough guess.</li><li>Fit a simple model to residual mistakes.</li><li>Add correction to previous predictions.</li><li>Repeat corrections to gradually improve.</li><li>Result is a sum of many small models.</li></ul><p><strong>Gradient Boosting for Classification</strong></p><ul><li>Adapts boosting framework to classification by minimizing a classification loss (e.g., logistic loss).</li><li>Residuals become gradients of the loss w.r.t. predictions.</li><li>Learners fit these pseudo-residuals to improve classification.</li></ul><p><strong>Strengths</strong></p><ul><li>Flexible: works with numerical and categorical data (with preprocessing).</li><li>Supports custom loss functions (MSE, MAE, Huber, quantile, logistic).</li><li>No assumptions about feature distributions or linearity.</li><li>Robust feature selection through trees.</li><li>Handles complex, non-linear relationships.</li></ul><p><strong>Weaknesses</strong></p><ul><li>Sensitive to outliers (MSE loss)</li><li>Computationally intensive: sequential training, especially with many trees.</li><li>Needs careful hyperparameter tuning to avoid over/underfitting.</li><li>Interpretation harder than simpler models.</li></ul><p><strong>Handling Outliers</strong></p><ul><li>Use robust loss (Huber, MAE).</li><li>Tune learning rate lower to reduce impact of outliers.</li><li>Limit tree depth.</li><li>Use early stopping.</li><li>Preprocess or remove outliers.</li></ul><p><strong>When to Use</strong></p><ul><li>Tabular data with complex patterns</li><li>Need high accuracy</li><li>Can afford training time</li><li>Not suitable for Real-time predictions needed → too slow</li></ul><p><strong>Hyperparameters</strong></p><ul><li>n_estimators (50–500): Number of sequential trees.</li><li>learning_rate (0.01–0.3): Step size for updates.</li><li>max_depth (3–8): Controls learner complexity.</li><li>subsample (0.5–1.0): Fraction of training data per tree (for regularization).</li><li>min_samples_leaf: Minimum samples in leaf node (regularizes).</li></ul><p><strong>Bias–Variance</strong></p><ul><li>Low bias, high variance model.</li><li>Regularization and shallow trees control overfitting.</li></ul><p><strong>Larger Context</strong></p><ul><li>Performs better on large, clean datasets.</li><li>Can overfit on small/noisy data if not properly tuned.</li><li>Slower training compared to Random Forest, but often higher accuracy.</li><li>Used widely in customer facing predictions.</li></ul><blockquote><strong>3. XGBoost</strong></blockquote><p><strong>1. What is XGBoost?</strong></p><p>XGBoost = <strong>Extreme Gradient Boosting</strong><br>An optimized, regularized version of Gradient Boosting.</p><p>Key ideas:</p><ul><li>Additive boosting (like Gradient Boosting)</li><li><strong>Regularization (L1 + L2)</strong> to reduce overfitting</li><li><strong>Handling missing values automatically</strong></li><li><strong>Handling imbalance in dataset</strong> using scale_pos_weight.</li><li><strong>Parallel tree building</strong> (parllel feature split unlike classic GBM)</li><li><strong>Highly optimized, very fast</strong></li></ul><p><strong>2. Core Intuition</strong></p><p>XGBoost builds trees <strong>sequentially</strong>, each trying to correct residuals of previous trees. But it improves GBM by adding:</p><p><strong>✔ Second-order Taylor approximation (uses gradients + Hessians)</strong></p><p>More accurate optimization than first-order GBM.</p><p><strong>✔ Regularization on trees</strong></p><p>Controls over-complex trees and reduces overfitting.</p><p><strong>✔ Weighted node splitting</strong></p><p>Uses both gradients and Hessians to choose split.</p><p><strong>✔ Parallelized split finding</strong></p><p>Much faster than normal boosting.</p><p><strong>3. Key Features (Why It Outperforms GBM/AdaBoost)</strong></p><p><strong>✔ Regularization (L1 + L2)</strong></p><p>Controls model complexity → reduces overfitting.</p><p><strong>✔ Parallelization</strong></p><p>Parllel Split finding → very fast.</p><p><strong>✔ Handles Missing Values</strong></p><p>Learns default direction for missing data.</p><p><strong>✔ Tree pruning</strong></p><p>Grow trees depth-wise, then prune backward using γ.</p><p><strong>✔ Built-in cross-validation</strong></p><p>xgb.cv() supports early stopping.</p><p><strong>✔ Supports custom loss functions</strong></p><p>You can define any differentiable loss.</p><p><strong>4. Hyperparameters (Most Important)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*Y8qqXQBC2c8r9vv6d15K2Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*dULjpcfjGp-MajzZKqqApQ.png" /></figure><p><strong>5. When to Use XGBoost</strong></p><p><strong>Best for tabular datasets where:</strong></p><p>✔ Many samples<br>✔ Non-linear patterns<br>✔ Missing values<br>✔ High accuracy needed<br>✔ Feature interactions matter</p><p>Widely used in:</p><ul><li>Kaggle competitions</li><li>Finance (fraud, credit risk)</li><li>E-commerce</li><li>Customer churn</li><li>Marketing response models</li></ul><p><strong>6. When NOT to Use XGBoost</strong></p><p>❌ Very small datasets → overfits<br>❌ Large sparse NLP tasks → prefer linear models<br>❌ Real-time prediction → slow<br>❌ Images/vision → use deep learning</p><p><strong>7. Strengths</strong></p><ul><li>Exceptional accuracy</li><li>Handles missing values</li><li>Works fast with parallelization(in feature split)</li><li>Regularized → less overfitting</li><li>Feature importance built-in</li><li>Supports early stopping</li><li>Handles mixed data well</li><li>Very popular in competitions</li></ul><p><strong>8. Weaknesses</strong></p><ul><li>Many hyperparameters → tuning needed</li><li>Slower than Random Forest</li><li>Sequential nature → slower training</li><li>Sensitive to noise with low regularization</li><li>Not suitable for extremely high-dimensional sparse data (e.g., BOW text)</li></ul><p><strong>9. Tips for Hyperparameter Tuning</strong></p><p><strong>Start simple:</strong></p><pre>max_depth: 3<br>learning_rate: 0.1<br>n_estimators: 200<br>subsample: 0.8<br>colsample_bytree: 0.8</pre><p><strong>To reduce overfitting:</strong></p><ul><li>↓ max_depth</li><li>↑ min_child_weight</li><li>↑ gamma</li><li>↓ eta</li><li>↓ n_estimators</li><li>↑ regularization (alpha, lambda)</li><li>↓ subsample / colsample_bytree</li></ul><p><strong>To increase model power:</strong></p><ul><li>↑ max_depth</li><li>↓ min_child_weight</li><li>↓ gamma</li><li>↑ n_estimators</li><li>↑ subsample</li></ul><p><strong>10. XGBoost Workflow (Simple Version)</strong></p><pre>Start with initial model (mean/log-odds)<br>For each iteration:<br>    compute gradients &amp; Hessians<br>    build a tree using weighted splits <br>    apply regularization (alpha, lambda, gamma)<br>    update predictions with shrinkage (eta)<br>Use early stopping to avoid overfitting</pre><blockquote><strong>Random Forest vs XGBoost vs AdaBoost vs Gradient Boosting</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FlSm6aGv5BmuWSQ8yEgKIQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lSFLhD6fbMWwrbzlozWV_w.png" /></figure><h4>Unsupervised Learning</h4><blockquote><strong>Dimensionality Reduction</strong></blockquote><p><strong>Definition</strong>: Dimensionality reduction transforms high-dimensional data into a lower-dimensional representation that retains meaningful structure and variance.​</p><p><strong>Motivation</strong>:</p><ul><li><strong>Curse of Dimensionality</strong>: High-dimensional data (many features) becomes sparse, making it hard for algorithms to detect patterns. Distance loses meaning, leading to poor clustering/classification and higher risk of overfitting.​</li><li><strong>Visualization</strong>: Reducing data to 2D/3D enables understanding and plotting large, complex datasets.​</li><li><strong>Efficiency</strong>: Less computation/memory needed; improved generalization by stripping redundancy.​</li></ul><p><strong>Key Approaches:</strong></p><ul><li><strong>Feature Selection</strong><em>:</em> Choose a subset of important features.</li><li><strong>Feature Extraction</strong><em>:</em> Combine or transform features — uses algorithms like PCA.</li><li><strong>When to use</strong>: When data has many correlated, redundant, or mostly-empty features (high sparsity).​</li></ul><blockquote><strong>Principal Component Analysis (PCA)</strong></blockquote><p>Definition: PCA is a linear dimensionality reduction technique that transforms original features into new, uncorrelated features (principal components) ranked by explained variance.​</p><p><strong>Key Concepts:</strong></p><ul><li><strong>Principal Components</strong><em>:</em> Ordered axes capturing maximum variance. First PC captures the most variance, second PC is orthogonal and captures the next most, etc.</li><li><strong>Mathematics</strong><em>:</em> Each PC is an eigenvector of the feature covariance matrix; variance explained by each is its eigenvalue. PCs are linear combinations of the original features.</li><li><strong>Formula</strong><em>:</em> For input vector x and weights w, the principal component is y=w^T x.</li></ul><p><strong>Algorithm Steps</strong>:</p><ol><li>Standardize features (mean=0, variance=1).</li><li>Compute covariance matrix.</li><li>Perform eigendecomposition.</li><li>Select top k eigenvectors (based on variance explained).</li><li>Project data onto this new <em>k</em>-dimensional space.</li></ol><p><strong>Assumptions</strong>:</p><ul><li><strong>Linearity</strong><em>:</em> Relationships among features are linear.</li><li><strong>Variance</strong><em>:</em> Data should mostly be variance-rich along fewer directions.</li><li><strong>Scaling</strong><em>:</em> Sensitive to units — input features must be standardized.</li></ul><p><strong>Common Pitfalls/Limitations:</strong></p><ul><li>Fails with non-linear structure (use t-SNE or UMAP for that).</li><li>Sensitive to outliers (outliers distort principal axes).</li><li>PCs can be hard to interpret relative to original features.</li><li>Does not use labels (purely unsupervised).</li></ul><p><strong>Best Practices for Interviews:</strong></p><ul><li>Know how to compute and interpret the proportion of variance explained by selected PCs.</li><li>Mention that in practice, the number of components is chosen to explain a high percentage (e.g., 90–95%) of variance.</li><li>Understand and be able to draw the scree plot (variance explained vs. number of PCs).</li></ul><pre>1. Proportion of Variance Explained<br><br>Proportion of Variance Explained by PCi​ = λi/toal variance<br><br>For example: <br>If eigenvalues = [4.0, 2.0, 1.0, 0.5]:<br><br>Total variance = 7.5<br><br>Variance explained by PC1 = 4/7.5 = 53.3%<br><br>By PC2 = 2/7.5 = 26.7%, etc.<br><br>Cumulative variance for first two PCs = 80% (4 + 2) / 7.5 → use first 2 PCs<br><br>2. Choosing Number of Components<br><br>In practice, we select the smallest number of PCs that explain a <br>high cumulative variance — usually 90–95%.<br><br>Why: It balances information retention and dimensionality reduction.<br><br>From the above eigenvalues:<br><br>PC1+PC2 explain 80% → might not be enough<br><br>PC1+PC2+PC3 explain 93.3% → ✅ typically chosen<br><br>3. Scree Plot<br><br>Definition:<br>A plot of eigenvalues (variance explained) vs number of PCs.<br><br>Interpretation:<br><br>X-axis: Principal Component number<br><br>Y-axis: Eigenvalue or % variance explained<br><br>The &quot;elbow point&quot; (where curve starts to flatten) <br>indicates the optimal number of PCs.<br><br>Variance ↑<br>        |        ●<br>        |      ●<br>        |    ●<br>        |   ●<br>        |  ●<br>        | ●<br>        +------------------------→ Number of PCs<br>             1  2  3  4  5<br><br>For example if the elbow is at PC3, you typically keep 3 PCs.</pre><p><strong>Curse of Dimensionality Summary Points</strong></p><ul><li>The number of required data points increases exponentially as dimensionality grows — sparsity makes standard algorithms ineffective.​</li><li>Nearest neighbor, clustering, and any distance-based method are most affected.​</li><li>Dimensionality reduction mitigates these issues by compressing data and denoising.​</li></ul><p><strong>Real-World Examples</strong></p><ol><li><strong>Healthcare: </strong>Reducing 20,000 gene expression features into top 50 PCs for cancer subtype classification.</li><li><strong>Computer Vision: </strong>Face recognition using Eigenfaces (PCA on pixel data).</li><li><strong>Marketing: </strong>Customer segmentation with many behavioral attributes.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jOv22kbdO4IHobclqanC8g.png" /></figure><blockquote><strong>Clustering</strong></blockquote><ul><li>Clustering is often done to find “hidden” groupings in data, like segmenting customers into different groups, where the customers in a group have similar characteristics.</li><li>Clustering can also be used for data visualization and outlier identification, as in fraud detection, for instance.</li><li>The goal of clustering is to partition a dataset into various clusters or groups by looking only at the data’s input features.</li></ul><p>Ideally, the clustered groups have two properties:</p><ol><li>Points within a given cluster are similar (i.e., high intra-cluster similarity).</li><li>Points in different clusters are not similar (i.e., low inter-cluster similarity)</li></ol><blockquote><strong>K-Means Clustering</strong></blockquote><p><strong>Core Idea</strong></p><ul><li><strong>Unsupervised</strong> algorithm → groups similar points into <strong>K clusters</strong></li><li>Goal: minimize <strong>intra-cluster variance (WCSS)</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/1*5Xf04JAOBwH4IkFS8mujBA.png" /></figure><ul><li><strong>Intuition:</strong> Like organizing a messy closet into <em>K neat piles of similar items.</em></li></ul><p><strong>Algorithm Steps</strong></p><ol><li><strong>Choose K</strong> (number of clusters)</li><li><strong>Initialize centroids=&gt;</strong>Random or <strong>K-Means++</strong> (smart, spread-out start)</li><li><strong>Assign step:</strong> Each point → nearest centroid</li><li><strong>Update step:</strong> Move centroids → mean of assigned points</li><li><strong>Repeat</strong> until centroids stop moving (convergence)</li></ol><p>⏱️ <strong>Time complexity:</strong> O(n × K × i × d)<br> (very efficient; near-linear with data size)</p><p><strong>When to Use</strong></p><p>✅ Continuous, numerical features<br> ✅ Roughly spherical clusters of similar size<br> ✅ Large datasets → scalable<br> ✅ You have or can estimate K</p><p><strong>Don’t use for:</strong></p><ul><li>Non-spherical / uneven densities (→ DBSCAN, HDBSCAN)</li><li>Categorical data (→ K-Modes)</li><li>Outliers (→ K-Medoids)</li><li>Unknown K (→ Hierarchical / DBSCAN)</li></ul><p><strong>Finding Optimal K</strong></p><ol><li><strong>Elbow Method: </strong>Plot K vs WCSS → find the “elbow” where adding clusters yields diminishing returns.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5P2iTo9N10Z6J5qUXYE-Lg.png" /></figure><p><strong>2. Silhouette Score</strong></p><ul><li>For each point, measure <strong>how well it fits its own cluster vs the nearest other cluster</strong>:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*gUdfsGHiPCwQPFILbPdqpA.png" /></figure><ul><li><strong>a</strong> = avg distance to points in its cluster</li><li><strong>b</strong> = avg distance to nearest other cluster</li><li>Score ranges: <strong>−1 to +1</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/317/1*C7KMAgBAPy-5s9EUbq_3Lw.png" /></figure><p>Choose <strong>K with highest average silhouette score</strong>.</p><p><strong>Intuition:</strong></p><p>Measures both compactness (tight clusters) and separation (clusters far apart). More computationally heavy than elbow.</p><p><strong>Key Challenges &amp; Fixes</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*97zz2gS214zcahJ56PmknA.png" /></figure><p><strong>Advanced &amp; Scalable Variant</strong></p><ul><li><strong>MiniBatch K-Means: </strong>Uses small random batches → faster convergence for millions of samples. Same accuracy (~95%) but 10× faster → used in production.</li></ul><p><strong>Business Connection (Always Mention)</strong></p><p>After clustering, I’d interpret each group as a <strong>persona</strong> and design experiments to validate business value.</p><p><strong>Example:</strong></p><ul><li>Cluster 1: Discount-seekers → send coupons</li><li>Cluster 2: Premium buyers → show luxury ads</li><li>Cluster 3: Window shoppers → re-engagement emails</li></ul><p><strong>Then A/B test</strong>:</p><ul><li>Within-cluster → which message works best</li><li>Across clusters → does segmentation improve conversion?</li></ul><p><strong>FAANG-Grade Examples</strong></p><ul><li><strong>Meta (Facebook):</strong> Group users by mutual friends &amp; interests → “People You May Know”</li><li><strong>Netflix:</strong> Cluster shows by genre &amp; viewer habits → recommendation engine</li><li><strong>Uber:</strong> Cluster pickup points → dynamic pricing zones</li><li><strong>Amazon:</strong> Segment customers by purchase patterns → targeted marketing</li></ul><p><strong>Evaluation Metrics</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*IQbOUfOXs12R9L5qZC5mHA.png" /></figure><p><strong>Interview Power Tips</strong></p><p>✅ Mention <strong>K-Means++</strong> → shows you know practical improvements<br> ✅ Discuss <strong>limitations first</strong> → shows depth<br> ✅ Always <strong>interpret clusters</strong> (business sense)<br> ✅ Mention <strong>MiniBatch K-Means</strong> for scalability<br> ✅ Never confuse with KNN (classification, supervised)<br> ✅ Quote metrics (Silhouette, DBI, CH) for evaluation<br> ✅ Note <strong>sensitivity to scale, outliers, and initial seeds</strong></p><p><strong>Common Mistakes</strong></p><ul><li>Forgetting to <strong>standardize</strong> data</li><li>Using <strong>categorical</strong> data</li><li>Not setting n_init &gt; 1</li><li>Blindly trusting results (no business interpretation)</li><li>Confusing <strong>K-Means ≠ KNN</strong></li></ul><p><strong>Quick Recall Mnemonics</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*pwZlHOOrneuxsSGyWeNREQ.png" /></figure><blockquote><strong>Hierarchical clustering</strong></blockquote><p>A clustering algorithm that builds a <strong>tree of clusters (dendrogram)</strong> by either:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*WGDWMqJKUUojbmAEtDrBig.png" /></figure><p>Produces a <strong>hierarchy</strong>, not just one flat partition.</p><p><strong>Key Idea:</strong></p><ul><li>Clusters form step-by-step based on distance until everything merges into one big cluster.</li><li>Unlike K-Means, you <strong>don’t set K upfront</strong> — you get a tree and then <em>decide</em> where to cut.</li></ul><p><strong>Dendrogram (Heart of Hierarchical Clustering)</strong></p><p>A tree diagram showing <strong>merge order</strong> and <strong>distance at which clusters merge</strong>.</p><p><strong>How to use dendrogram:</strong></p><ul><li><strong>Small merge height</strong> → clusters are very similar</li><li><strong>Large merge height</strong> → clusters are very different</li><li><strong>Choose K by cutting</strong> horizontally at a height where big vertical gaps appear</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*NVLAmxQpMrm4peJkLRilVA.png" /></figure><p><strong>Linkage Methods (How distance between clusters is defined)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*wjIf97SUvRQDidcWP1HIxg.png" /></figure><p><strong>Distance Metrics</strong></p><ul><li>Usually <strong>Euclidean</strong></li><li>Others: Manhattan, Cosine (depending on data type)</li></ul><p><strong>How Agglomerative Clustering Works</strong></p><ol><li>Start with each point as its own cluster</li><li>Compute pairwise distances</li><li>Merge two closest clusters (based on linkage)</li><li>Update distance matrix</li><li>Repeat until one cluster remains</li><li>Cut dendrogram at desired level → final clusters</li></ol><p><strong>When to Use</strong></p><p>✅ Small/medium datasets (≤ 5k samples)<br> ✅ When you don’t know number of clusters<br> ✅ When interpretability matters (business story, research)<br> ✅ When clusters may be nested or hierarchical<br> ✅ When you want exploratory analysis (EDA)</p><p><strong>When NOT to Use</strong></p><p>❌ Large datasets (distance matrix too expensive)<br> ❌ High-dimensional data<br> ❌ Very noisy data (distances get distorted)<br> ❌ Real-time or streaming scenarios</p><p><strong>Evaluation Metrics</strong></p><p>You can evaluate clusters using:</p><ul><li><strong>Silhouette Score</strong></li><li><strong>Davies–Bouldin Index</strong></li><li><strong>Calinski–Harabasz Index</strong></li></ul><p>(But dendrogram is the most intuitive.)</p><p><strong>Real-World Examples</strong></p><ul><li><strong>Customer Segmentation</strong> (budget → mid → luxury → elite)</li><li><strong>Document Clustering</strong> (topic hierarchy)</li><li><strong>Gene Expression Analysis</strong> (biology)</li><li><strong>Image Segmentation</strong></li><li><strong>Social Group Discovery</strong></li></ul><p><strong>FAANG example:</strong></p><p>Meta used agglomerative clustering early in graph-based friend-suggestion pipelines to detect tight-knit communities using friends-of-friends similarity.</p><p><strong>FAANG Interview Must-Say Points</strong></p><p>1. Hierarchical clustering gives a full clustering tree; K-Means only gives a single partition.<br> 2. Ward linkage is most stable and widely used in practice.<br> 3. Dendrogram lets us visually decide K based on natural separation.<br> 4. It’s O(n²), so not suitable for huge datasets — I’d switch to K-Means or MiniBatch K-Means at scale.<br> 5. Great for EDA and interpretability — I often use dendrograms to validate natural group boundaries.</p><p><strong>Mini Cheat Sheet to Memorize</strong></p><ul><li><strong>Bottom-up merges → dendrogram → cut tree → clusters</strong></li><li><strong>Linkages:</strong> Single = chaining, Complete = tight, Average = balanced, Ward = variance-minimizing</li><li><strong>Don’t use for large datasets</strong></li><li><strong>Highly interpretable; no need for K upfront</strong></li></ul><blockquote><strong>DBSCAN</strong></blockquote><p><strong>DBSCAN = Density-Based Spatial Clustering of Applications with Noise</strong><br>A clustering algorithm that:</p><p>✔ Finds clusters based on <strong>dense regions of points</strong><br> ✔ Detects <strong>arbitrary shaped clusters</strong> (circles, spirals, moons)<br> ✔ Automatically identifies <strong>noise/outliers</strong><br> ✔ Does NOT require choosing <strong>K</strong></p><p><strong>Core Idea</strong></p><p>Points that are close and densely packed form clusters; isolated points become noise.</p><p><strong>Key Parameters</strong></p><p>1. <strong>eps (ε)</strong>: Radius around a point — the “neighbourhood size.”<br> → How close is considered “close”?</p><p><strong>2. min_samples: </strong>Minimum number of points inside eps to be considered a <strong>dense region</strong>.</p><p><strong>Types of Points (Very Important for Interviews)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*kh1TL4mHRZ-vUqkr5LMuIg.png" /></figure><p><strong>How DBSCAN Works</strong></p><ol><li>Pick a point</li><li>Draw eps-radius circle around it</li><li>If ≥ min_samples → mark as <strong>core</strong> and start a cluster</li><li>Expand cluster by visiting all reachable core points</li><li>Add border points along the way</li><li>Points that don’t join any cluster → <strong>noise</strong></li></ol><p>Clusters keep expanding until no more points are density-reachable.</p><p><strong>Why DBSCAN Is Amazing</strong></p><p>✔ Finds clusters of arbitrary shape</p><p>K-means only finds round blobs; DBSCAN finds spirals, moons, rings, snakes.</p><p>✔ Handles outliers naturally</p><p>Noise points are automatically separated.</p><p>✔ No need to specify number of clusters (K)</p><p>DBSCAN discovers it based on density.</p><p>✔ Great for spatial data and anomaly detection.</p><p><strong>Limitations (Know these for FAANG)</strong></p><p>❌ Struggles with varying densities</p><p>A dense cluster + a sparse cluster = DBSCAN fails.</p><p>❌ Sensitive to eps</p><p>Small eps → too many tiny clusters<br>Large eps → everything merges into one cluster</p><p>❌ Does NOT work well on high-dimensional data</p><p>Distance meaning breaks down (“curse of dimensionality”).</p><p>❌ Can be slow on extremely large datasets</p><p><strong>Choosing eps</strong></p><p>Use the <strong>k-distance plot</strong>:</p><ol><li>Compute distance to k-th nearest neighbor (k = min_samples)</li><li>Sort distances</li><li>Look for “elbow” → that value ≈ good eps</li></ol><p><strong>Real-World Examples</strong></p><p><strong>1. Fraud Detection</strong></p><p>Outlier transactions = noise<br>Normal customer behavior = dense clusters</p><p><strong>2. GPS Clustering (Uber / Lyft)</strong></p><p>Dense pickup zones → airports, malls<br>Sparse points → irrelevant pings<br>DBSCAN finds irregular shapes of hotspots.</p><p><strong>3. Image Segmentation</strong></p><p>Clusters irregular 2D regions in pixel space.</p><p><strong>4. Document Embeddings (Low-dimensional)</strong></p><p>Find dense topical regions after dimensionality reduction.</p><p><strong>FAANG Interview Gold Statements</strong></p><p>Use one or more of these — they show depth:</p><ul><li>DBSCAN groups points by density; anything outside dense regions is labeled noise.</li><li>Unlike K-means, DBSCAN finds non-spherical clusters — essential for real spatial data.</li><li>DBSCAN does not require K; cluster count is an emergent property of density.</li><li>Core, border, and noise points create rich interpretability.</li><li>eps and min_samples control the cluster tightness; k-distance plot helps tune eps.</li><li>DBSCAN is great for anomaly detection because outliers are naturally separated.</li><li>Fails when clusters have different densities — that’s when to use HDBSCAN.</li></ul><p><strong>Mini Cheat Sheet (Memorize This Table)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*-B_ZpVOxCnbOvs6mJgznyw.png" /></figure><blockquote><strong>Gaussian Mixture Models (GMM)</strong></blockquote><p>A GMM models data as a mixture of multiple Gaussian (normal) distributions, each with its own mean, variance, and weight, enabling flexible, probabilistic clustering.</p><p><strong>Why GMM? (Compared to K-Means)</strong></p><p>K-Means:</p><ul><li>Hard assignments</li><li>Equal-sized spherical clusters</li><li>Only learns cluster means</li></ul><p>GMM:</p><ul><li><strong>Soft assignments</strong> (each point has a probability of belonging to each cluster)</li><li>Learns <strong>mean + covariance</strong> (shape + spread)</li><li>Allows <strong>elliptical</strong>, <strong>overlapping</strong>, and <strong>variable-sized</strong> clusters</li><li>More flexible, more realistic</li></ul><p>👉 <strong>GMM = “K-Means ++ with shapes + probabilities”</strong></p><p><strong>What GMM Learns</strong></p><p>For each cluster (k):</p><ul><li><strong>Mean </strong>μk​</li><li><strong>Covariance matrix</strong> Σk​</li><li><strong>Mixing weight</strong> πk(proportion of points in cluster k)</li></ul><p><strong>How GMM Works (EM Algorithm)</strong></p><p><strong>E-step (Expectation): </strong>Compute probability that each point belongs to each cluster</p><p><strong>M-step (Maximization)</strong></p><p>Update:</p><ul><li>new μk</li><li>new Σk</li><li>new πk</li></ul><p>based on the soft assignments.</p><p>Repeat until convergence.</p><p><strong>Shapes GMM Can Model</strong></p><p>✔ Round clusters<br> ✔ Elliptical clusters<br> ✔ Tilting/rotated clusters<br> ✔ Overlapping clusters<br> ✔ Unequal variances<br> ✔ Different densities</p><p>This is why GMM is <strong>more powerful than K-means</strong>.</p><p><strong>When to Use GMM</strong></p><p>✔ Use when:</p><ul><li>Clusters have different shapes/sizes</li><li>Clusters overlap</li><li>You want probabilities (soft assignments)</li><li>Clusters look “Gaussian-ish”</li><li>Data is <strong>low or medium dimensional</strong></li><li>You want anomaly detection</li></ul><p><strong>Real-world examples:</strong></p><ul><li><strong>TikTok anomaly detection</strong> (user activity patterns)</li><li><strong>Credit card risk segmentation</strong></li><li><strong>Fraud detection</strong> (outliers = low likelihood under any Gaussian)</li><li><strong>Voice/Speech modeling</strong></li><li><strong>Image segmentation</strong></li><li><strong>Mixture of customer behaviors</strong></li></ul><p><strong>When NOT to Use GMM</strong></p><p>Avoid when:</p><ul><li>Very high-dimensional data (covariance matrices unstable)</li><li>Data is non-Gaussian / weird shapes (use DBSCAN)</li><li>Clusters have extremely different densities</li><li>Large datasets (slower than K-means)</li><li>You want crisp, hard cluster boundaries</li></ul><p>Note: GMM is a probabilistic clustering method that models each cluster as a Gaussian distribution. It generalizes K-means by learning covariance structures and allowing soft assignments, making it ideal for overlapping, elliptical clusters and anomaly detection.</p><p><strong>Mini-Cheat Sheet (Memorize These 7 Lines)</strong></p><ul><li><strong>Soft clustering</strong> → probabilities, not hard labels</li><li><strong>Clusters = Gaussians</strong> with mean, covariance, weight</li><li>Uses <strong>EM algorithm</strong> (E-step &amp; M-step)</li><li>Handles <strong>elliptical, overlapping, varying-sized</strong> clusters</li><li>Choose K via <strong>AIC/BIC</strong></li><li>Good for <strong>low/medium-dimensional</strong> structured data</li><li>Bad for <strong>high-dimensional or non-Gaussian</strong> data</li></ul><p><strong>Clustering Algorithms Comparison Table</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-joKbjgycU3PVAWIAIuB0A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nkfa2XBfM64rygkXU1NKxw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QdynHAO-QXHjT3NNn_pmwA.png" /></figure><h4>Neural Networks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G_CXmlqql0XHYX2GpyixVA.png" /></figure><p><strong>AI but not ML includes:</strong></p><ul><li>Rule-based systems</li><li>Logic systems</li><li>Planning algorithms</li><li>Search algorithms</li><li>Expert systems</li></ul><p><strong>1. What is a Neural Network?</strong></p><p>A neural network is a model made of <strong>neurons (units)</strong> arranged in <strong>layers</strong>.</p><p>Each neuron performs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*whglwrhQwT5IaouffT3MtQ.png" /></figure><p>Where:</p><ul><li><strong>w</strong> = weights (learned parameters)</li><li><strong>b</strong> = bias</li><li><strong>f(z)</strong> = activation function</li></ul><p>👉 Neuron = <em>Linear model + Non-linear function</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N-unLuvJIT8ufym3S081kw.png" /><figcaption>Single Layer Perception</figcaption></figure><p><strong>2. Architecture of a Neural Network</strong></p><p><strong>Layers:</strong></p><ul><li><strong>Input layer:</strong> raw features</li><li><strong>Hidden layers:</strong> multiple neurons extracting patterns</li><li><strong>Output layer:</strong> final prediction</li></ul><p><strong>Dense (Fully Connected) Layer:</strong></p><p>Every neuron connects to all neurons in next layer.</p><p><strong>Deep Neural Network (DNN):</strong></p><p>A neural network with <strong>multiple hidden layers</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*WiL6cja26Jux7W-a8a0BIw.png" /></figure><p><strong>3. How Neural Networks Learn (Forward + Backward Pass)</strong></p><p>Forward Pass</p><ul><li>Data flows layer by layer</li><li>Each neuron computes weighted sum → activation</li><li>Generates final prediction y_pred</li></ul><p><strong>Loss Function</strong></p><p>Used to measure error:</p><ul><li>Regression → MSE</li><li>Classification → Cross-entropy</li></ul><p><strong>Backward Pass (Backpropagation)</strong></p><p>Uses <strong>chain rule</strong> to compute gradient of loss w.r.t all weights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/215/1*SKCJvLMSA6FS69CshrYF9A.png" /></figure><p>Where α = learning rate.</p><p>👉 Backprop = sending “blame” backwards to adjust weights.</p><p><strong>4. Activation Functions (Why Needed?)</strong></p><p>They add <strong>non-linearity</strong> so the model can learn complex patterns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*WYANYOtRUpTN-WSjv3tKfQ.png" /></figure><p>👉 Without activations → NN becomes just a linear model.</p><p><strong>5. Why Multiple Neurons/Layers?</strong></p><ul><li>Each neuron learns a <strong>different pattern</strong></li><li>Early layers → simple features</li><li>Deeper layers → abstract high-level features</li></ul><p>Analogy:<br>Eyes detect edges → brain detects shapes → whole objects.</p><p><strong>6. Training Challenges</strong></p><p><strong>A. Vanishing/Exploding Gradients</strong></p><p>Gradients shrink or explode during backpropagation.</p><p>Causes:</p><ul><li>Many layers</li><li>Sigmoid/tanh activations</li></ul><p>Fixes:</p><ul><li>ReLU activation</li><li>Residual connections (ResNet)</li><li>LSTM architecture</li><li>Proper initialization</li><li>Gradient clipping (for exploding)→ Limit how large gradients can get.</li></ul><p><strong>B. Overfitting</strong></p><p>NNs can memorize instead of learn.</p><p>Fixes:</p><ul><li>More data</li><li>Regularization</li><li>Dropout</li><li>Early stopping</li><li>Batch Normalization</li></ul><p><strong>7. Key Training Optimization Methods</strong></p><p><strong>A. SGD (Stochastic Gradient Descent)</strong></p><p>Updates weights using small batch randomly sampled.</p><p><strong>B. Momentum</strong></p><p>Adds “inertia”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*8wDbT450h42VpPp-NMW-TA.png" /></figure><p>👉 Helps move consistently in right direction.</p><p>Analogy: <em>Rolling ball gaining speed downhill.</em></p><p><strong>C. Adam (Adaptive Moment Estimation)</strong></p><p>Momentum + adaptive learning rate per weight.</p><p>Advantages:</p><ul><li>Stable</li><li>Fast</li><li>Works well out-of-the-box</li></ul><p>Used everywhere in deep learning.</p><p><strong>8. Batch Normalization</strong></p><p>Standardizes neuron activations <strong>within a mini-batch</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/197/1*yA3A9KQWjABpTOYdnvGXPQ.png" /></figure><p>Benefits:</p><ul><li>Stabilizes training</li><li>Allows higher learning rate</li><li>Reduces internal covariate shift</li><li>Regularization effect (reduces overfitting)</li></ul><p>Used between:<br><strong>Linear → BatchNorm → Activation</strong></p><p><strong>9. Dropout</strong></p><p>Randomly disables neurons during training.</p><p>Purpose:</p><ul><li>Prevents co-adaptation</li><li>Acts like training many smaller networks</li><li>Reduces overfitting</li></ul><p><strong>10. Transfer Learning</strong></p><p>Using a model trained on large dataset → fine-tuning on new task.</p><p>Example:<br> Use ImageNet-trained CNN → only retrain last few layers for your dataset.</p><p>Useful when:</p><ul><li>Less data</li><li>Similar data distributions</li><li>Leverage pre-trained knowledge</li></ul><p><strong>11. CNN (Convolutional Neural Networks)</strong></p><p>Used for <strong>images</strong>.</p><p><strong>How it works:</strong></p><ul><li><strong>Convolution layers</strong> — detect edges, textures, shapes</li><li><strong>Pooling layers</strong> — reduce dimensionality</li><li><strong>Dense layers</strong> — classification</li></ul><p>Key idea: Nearby pixels carry meaning (spatial patterns).</p><p>Applications:</p><ul><li>Vision</li><li>Object detection</li><li>Medical imaging</li></ul><p><strong>12. RNN (Recurrent Neural Networks)</strong></p><p>Used for <strong>sequences</strong> (text, time series).</p><p>RNN introduces memory:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/1*leDpIn2aQErx7IZ0po6sww.png" /></figure><p>Good for:</p><ul><li>Next word prediction</li><li>Speech recognition</li><li>Sequential signals</li></ul><p><strong>13. LSTM (Long Short-Term Memory)</strong></p><p>Fixes RNN short memory issue.</p><p>Contains:</p><ul><li>Input gate</li><li>Output gate</li><li>Forget gate</li><li>Cell state (long-term memory)</li></ul><p>Used for:</p><ul><li>Long sentences</li><li>Machine translation</li><li>Speech</li></ul><p><strong>14. Transformers &amp; LLMs</strong></p><p>Transformers use <strong>self-attention</strong> to understand relationships between words <strong>without</strong> sequential processing.</p><p>Key features:</p><ul><li>Can process full sentence at once</li><li>Capture long-range dependencies</li><li>Easily parallelizable</li></ul><p>Transformers → foundation for LLMs like GPT, BERT.</p><p>LLM = Large Transformer trained on huge datasets.</p><p>Prompt Engineering = Crafting input instructions to guide LLM behavior.</p><p><strong>15. When Not to Use Neural Networks</strong></p><p>Classical ML is still superior when:</p><ul><li>Data is small</li><li>Data is noisy</li><li>Interpretability is needed</li><li>Faster training needed</li><li>Tabular datasets (XGBoost often wins)</li></ul><p>Product Data Science relies heavily on <strong>classical ML</strong>:</p><ul><li>Churn prediction</li><li>Segmentation</li><li>A/B analysis</li><li>Experimentation</li><li>Recommendation models</li></ul><p>Neural networks aren’t disappearing — but are used where they shine: images, text, speech.</p><p><strong>16. Reinforcement Learning (RL)</strong></p><p>Not supervised or unsupervised.</p><p>RL = Learning by <strong>trial + error + rewards</strong>.</p><p>Components:</p><ul><li>Agent</li><li>Environment</li><li>Policy (how to act)</li><li>Reward</li><li>Value function</li></ul><p>Examples:</p><ul><li>AlphaGo</li><li>Robotics</li><li>Game playing</li></ul><p><strong>Deep RL</strong></p><p>RL + Neural Networks (function approximators)</p><p><strong>Final Interview Crib Sheet</strong></p><ul><li>NN = linear combinations + nonlinear activations</li><li>Each neuron learns a pattern; layers learn hierarchy of patterns</li><li>Training = forward pass + backprop</li><li>Issues: vanishing gradient, overfitting, noisy data</li><li>Fixes: ReLU, BatchNorm, Dropout, Momentum, Adam</li><li>CNN = spatial patterns (images)</li><li>RNN = sequential patterns</li><li>LSTM = long-term memory</li><li>RL = learning by rewards</li><li>Transformers = attention → basis for LLMs</li><li>Classical ML still dominates structured business data.</li></ul><blockquote><strong>The End-to-End ML Workflow</strong></blockquote><p><strong>1. Clarify the Problem &amp; Constraints</strong></p><p>(If you do this right, half the problem is solved.)</p><p><strong>Business Understanding</strong></p><p>✔ What exactly is the target/dependent variable?<br> ✔ How does business define it (e.g., what counts as “churn”)?<br> ✔ What is the current baseline? How much do we need to improve?<br> ✔ Is ML really needed? Would heuristics or rules be enough?<br> ✔ Any legal/ethical restrictions (e.g., race not allowed in lending)?<br> ✔ Who are users &amp; stakeholders? How will they use the ML system?<br> ✔ Cost of wrong predictions? (False positives vs false negatives)<br> ✔ Can problem be decomposed into smaller ML systems?</p><p><strong>Technical Constraints</strong></p><p>✔ Latency requirements? (real-time vs offline batch)<br> ✔ Throughput? (predictions/minute)<br> ✔ Deployment constraints? (edge device size, memory, GPU availability)<br> ✔ Is training allowed to be slow but inference must be fast?</p><p>2. Establish Metrics</p><p>Choose <strong>ONE primary metric</strong>, mention others as secondary.</p><p>Primary Metric Examples</p><ul><li>Classification: Accuracy → Precision/Recall → F1 → ROC-AUC</li><li>Regression: RMSE, MAE</li><li>Business KPI alignment: Time-to-resolution, revenue improvement</li></ul><p><strong>Advanced Metric Concepts</strong><br> ✔ Compare performance to baseline (existing process)</p><p><strong>3. Understand Your Data Sources</strong></p><p>(“Garbage in → garbage out.”)</p><p><strong>Sources to Consider</strong></p><p>✔ Internal logs<br> ✔ User-provided onboarding data<br> ✔ Crowdsourcing (Mechanical Turk)<br> ✔ Purchased 2nd/3rd party data<br> ✔ Ethical web scraping<br> ✔ Labeling vendors (ScaleAI etc.)</p><p><strong>Data Freshness &amp; Quality</strong></p><p>✔ How fresh is data? Update frequency?<br> ✔ Data dictionary available?<br> ✔ Any known biases, sampling issues?</p><p><strong>Edge Case Data Enhancements</strong></p><p>✔ Data augmentation<br> ✔ Synthetic data<br> ✔ Simulation (common in self-driving)</p><p><strong>4. Explore Your Data (EDA)</strong></p><p>✔ Column profiling (variance, missingness, noise)<br> ✔ Summary statistics: mean, median, quantiles<br> ✔ Visualizations:</p><ul><li>Histograms</li><li>Boxplots</li><li>Correlation matrix</li><li>Value counts for categorical features</li></ul><p>Goal: understand distributions, outliers, relationships, anomalies.</p><p><strong>5. Clean Your Data</strong></p><p>✔ Remove irrelevant/duplicate rows or columns<br> ✔ Fix incorrect values, data entry errors<br> ✔ Handle missing values:</p><ul><li>Mean/median/most frequent</li><li>Model-based imputation</li><li>Drop rows (last resort)</li></ul><p>✔ Handle outliers properly:</p><ul><li>Remove, cap/floor, or keep (depending on business context)</li><li>Consider <strong>multivariate</strong> outliers</li></ul><p><strong>6. Feature Engineering</strong></p><p>(One of the most important steps.)</p><p><strong>Numerical Features</strong></p><p>✔ Transformations: log, sqrt, power, capping, flooring<br> ✔ Binning (discretization)<br> ✔ Scaling/standardization:</p><ul><li>Min-max scaling</li><li>Z-score standardization</li></ul><p>✔ Dimensionality Reduction:</p><ul><li>PCA (for uncorrelated features)</li></ul><p>Categorical Features</p><p>✔ One-hot encoding , Label Encoding<br> ✔ Hashing (for high-cardinality categories)</p><p><strong>Text Features (if relevant)</strong></p><p>✔ Stemming<br> ✔ Lemmatization<br> ✔ Stop-word removal<br> ✔ Bag-of-words<br> ✔ N-grams<br> ✔ Word embeddings: word2vec, GloVe</p><p><strong>7. Model Selection</strong></p><p>Evaluate based on:</p><p><strong>Primary Criteria</strong></p><p>✔ Training &amp; inference speed<br> ✔ Data size &amp; dimensionality<br> ✔ Ability to handle categorical or numerical data<br> ✔ Budget &amp; compute cost<br> ✔ Interpretability needs<br> ✔ Robustness to noise &amp; missing data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0YvKHVwUI6sEHbedAfdRkw.jpeg" /><figcaption>Machine Learning Algorithms Cheat Sheet</figcaption></figure><p><strong>8. Model Training &amp; Evaluation</strong></p><p>✔ Train/Validation/Test split<br> ✔ Cross-validation (k-fold, stratified)<br> ✔ Hyperparameter tuning (Grid Search, Random Search, Bayesian)<br> ✔ Regularization to avoid overfitting<br> ✔ Learning curves to diagnose high bias/variance<br> ✔ Handle class imbalance:</p><ul><li>Oversampling, undersampling</li><li>SMOTE</li><li>Class weights</li></ul><p>✔ Sampling strategy when data is huge:</p><ul><li>Random sampling</li><li>Stratified sampling</li></ul><p><strong>9. Deployment (MLOps Concepts)</strong></p><p>Deployment styles:</p><p><strong>Online (real-time)</strong></p><p>✔ Low latency<br> ✔ Feature caching needed<br> ✔ Heavy infra requirements</p><p><strong>Batch Predictions</strong></p><p>✔ Periodic refresh (daily/weekly)<br> ✔ Good for recommendations, churn scores<br> ✔ High throughput, relaxed latency</p><p><strong>Hybrid</strong></p><p>✔ Batch to generate base results<br> ✔ Online for real-time adjustments</p><p><strong>Model Degradation &amp; Drift</strong></p><p>✔ Training-serving skew<br> ✔ Must schedule regular retraining<br> ✔ Add monitoring &amp; logging<br> ✔ Detect data drift, concept drift</p><p><strong>10. Iterate</strong></p><p>Deployment is not the end.</p><p><strong>Error Analysis</strong></p><p>✔ Manually inspect wrong predictions<br> ✔ Bucket failure modes<br> ✔ Prioritize improvements (collect more data, engineer new features)</p><p><strong>Continuous Improvement</strong></p><p>✔ Update metrics<br> ✔ Update features<br> ✔ Update model architecture<br> ✔ Keep tracking business KPIs</p><blockquote><strong>MLOPS</strong></blockquote><ol><li><strong>What is MLOps</strong></li></ol><p><strong>MLOps = Machine Learning + DevOps</strong></p><p>It is the set of <strong>processes, tools, and best practices</strong> used to take an ML model from:</p><p>➡️ <strong>research/experimentation</strong> →<br> ➡️ <strong>production/real-world use</strong> →<br> ➡️ <strong>monitoring + updating continuously</strong></p><p>MLOps is everything needed to run ML models reliably in the real world — not just build them.</p><p><strong>2. How is MLOps different from normal ML?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*vBXJ24BwCpD4hHKp1E09gA.png" /></figure><p><strong>3. Why do we need MLOps? (Use Cases)</strong></p><p>Because <strong>ML models break in the real world</strong>.</p><p>Examples:</p><p><strong>i. Data Drift</strong></p><p>Data in production changes with time.<br>Example: winter clothing recommender in summer → wrong predictions.</p><p><strong>ii. Model Degradation</strong></p><p>Model accuracy drops as user behavior evolves.</p><p><strong>iii. Re-training Needs</strong></p><p>Models must learn from <em>new</em> data continuously.</p><p><strong>iv. Monitoring &amp; Alerting</strong></p><p>Detect issues early:</p><ul><li>accuracy drop</li><li>latency increase</li><li>feature distribution shifts</li></ul><p><strong>v. Deployment at Scale</strong></p><p>Your model may need to serve:</p><ul><li>millions of predictions/day</li><li>real-time API responses under 100 ms</li></ul><p><strong>vi. Reproducibility</strong></p><p>You need to re-run:</p><ul><li>same dataset</li><li>same training pipeline</li><li>same version of code</li><li>same model results</li></ul><p><strong>vii. Automation</strong></p><p>Manual training → Not scalable<br>Automated pipelines → Highly scalable</p><p><strong>4. What exactly does MLOps include? (The Components)</strong></p><p><strong>i. Data Pipeline</strong></p><p>Automated data extraction → cleaning → storing</p><p><strong>ii. Model Training Pipeline</strong></p><p>Automatic training (hourly/daily/weekly)<br>Hyperparameter tuning<br>Versioning the model</p><p><strong>iii. Model Deployment</strong></p><p>How the model is served:</p><ul><li>REST API</li><li>Batch jobs</li><li>On-device models</li></ul><p><strong>iv. Model Monitoring</strong></p><p>Monitor:</p><ul><li>input data distribution</li><li>accuracy over time</li><li>latency &amp; uptime</li><li>drift detection</li></ul><p><strong>V. Automated Retraining</strong></p><p>Trigger retraining when drift is detected.</p><p><strong>vi. CI/CD pipeline for ML</strong></p><p>Automated:</p><ul><li>testing</li><li>model validation</li><li>deployment</li></ul><p><strong>vii. Feature Store</strong></p><p>Central place to store and reuse features consistently.</p><p><strong>Note:</strong></p><p>MLOps is the engineering process that automates deploying, monitoring, and continuously improving ML models in production so that they remain accurate, reliable, and scalable over time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7b89cccfe33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Product Sense Quick Revision Guide for Data Science Interviews]]></title>
            <link>https://medium.com/@mdmoseena22/product-sense-quick-revision-guide-for-data-science-interviews-0348a44b1aac?source=rss-72a4fb8cbdaf------2</link>
            <guid isPermaLink="false">https://medium.com/p/0348a44b1aac</guid>
            <category><![CDATA[interview-preparation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[product-goals]]></category>
            <category><![CDATA[product-sense]]></category>
            <category><![CDATA[business-goals]]></category>
            <dc:creator><![CDATA[MOHAMMED MOSEENA]]></dc:creator>
            <pubDate>Mon, 17 Nov 2025 07:22:08 GMT</pubDate>
            <atom:updated>2026-01-04T07:14:20.179Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Why Product Sense Matters for Data Scientists</strong></blockquote><ul><li>Work cross-functionally with PMs and business stakeholders</li><li>Shape product roadmaps and uncover root causes of business problems</li><li>Understand which metrics matter for modeling and how model performance impacts product success</li></ul><blockquote><strong>User Journey Framework: AARRR (Pirate Metrics)</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uVl2yPQ2cnyDarl-mog4Nw.png" /></figure><ol><li><strong>Acquisition</strong></li></ol><p><strong>What:</strong> How people are finding/trying your product (top of funnel)<br><strong>Metrics:</strong> New user signups, sign-up conversion rates, Customer Acquisition Cost (CAC)<br><strong>Example:</strong> Pinterest — new app downloads, new account creations per week</p><p><strong>2. Activation</strong></p><p><strong>What:</strong> User reaches “aha moment” — experiences core product value<br><strong>Metrics:</strong> First core action completion<br><strong>Example:</strong> DoorDash — first delivery order; Instagram — viewing 10 posts or making first post</p><p><strong>3. Engagement</strong></p><p><strong>What:</strong> How often and how well users interact with the product<br><strong>Metrics:</strong> DAU (Daily Active Users), WAU (Weekly Active Users), MAU (Monthly Active Users), time spent, core actions taken<br><strong>Example:</strong> Facebook measures DAU/WAU/MAU; Netflix tracks time spent.</p><p><strong>4. Retention</strong></p><p><strong>What:</strong> Users keep coming back over time<br><strong>Metrics:</strong> Monthly retention rate, monthly churn rate<br><strong>Why Important:</strong> Acquiring new customers is harder and costlier than retaining existing ones</p><p><strong>5. Referral</strong></p><p><strong>What:</strong> Users sharing product with others<br><strong>Metrics:</strong> K-factor = (avg referrals sent per user) × (conversion rate per referral)<br><strong>Note:</strong> Typically happens after activation, engagement, and retention</p><p><strong>6. Revenue</strong></p><p><strong>What:</strong> Monetization at any stage<br><strong>Metrics:</strong> LTV (Lifetime Value per customer), LTV-to-CAC ratio<br><strong>Goal:</strong> Sustainable companies have high LTV-to-CAC ratio</p><p><strong>Pro Tip:</strong> Research company-specific metrics before your interview (e.g., SaaS companies use ACV — Average Contract Value, MRR — Monthly Recurring Revenue)</p><blockquote><strong>Good vs Bad Metrics</strong></blockquote><p><strong>Bad Metrics (Using Facebook Dating Example)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6OYsdK8b2jOdb0W6YF-STA.png" /></figure><p><strong>Good Metrics Characteristics</strong></p><ol><li><strong>Meaningful:</strong> Tied to business goals, hard to game, actionable</li><li><strong>Measurable:</strong> Simple to understand and reliably track</li><li><strong>Understandable:</strong> Intuitive for stakeholders</li><li><strong>Timely:</strong> Collected within reasonable timeframe</li></ol><blockquote><strong>North Star vs Guardrail Metrics</strong></blockquote><ol><li><strong>North Star Metric: The most important metric</strong> that directly measures core product value</li><li><strong>Guardrail Metrics (Counter Metrics): Business metrics that should NOT degrade</strong> while optimizing the north star</li></ol><p><strong>Example: Uber (Ride-sharing App)</strong></p><p><strong>North Star:</strong> Completed rides per week</p><p><strong>Guardrails:</strong></p><ul><li>Driver satisfaction rating (driver retention)</li><li>Average wait time (service quality)</li><li>Safety incident rate (user safety)</li><li>Driver earnings per hour (marketplace sustainability)</li></ul><p><strong>Pro Tip:</strong> Proactively mention guardrails after discussing north star metrics to show holistic thinking</p><blockquote><strong>Four Common Product Sense Interview Questions</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*tHIE2FYUEUM0CrWQNr_RAA.png" /></figure><blockquote><strong>1. Defining a Product Metric</strong></blockquote><p><strong>Question Type:</strong> “How would you measure success of [product/feature]?”</p><p><strong>3-Step Framework:</strong></p><p><strong>Step 1: Clarify the Product &amp; Its Purpose</strong></p><ul><li>Why is the company building this?</li><li>What’s the product goal?</li><li>Who is it for?</li><li>How does it work?</li></ul><p><strong>Step 2: Explain Product &amp; Business Goals</strong></p><ul><li>Tie the Product goal (what makes users happy) to the Business goals (inturn what makes the company grow)</li></ul><p><strong>Step 3: Define Success Metrics</strong></p><p>Follow the AARRR funnel and align each metric with product/business goals:</p><ul><li><strong>Acquisition</strong> — how do users discover or start using the feature?</li><li><strong>Activation</strong> — what indicates they’ve tried and successfully used it once?</li><li><strong>Engagement</strong> — how do we know they’re using it regularly?</li><li><strong>Retention</strong> — how do we know they’re coming back to use it again?</li><li><strong>Revenue</strong> — how does this feature contribute to business goals?</li><li><strong>Refferal</strong></li></ul><p><strong>Example: </strong>How would you measure the success of <strong>Netflix: Trailer previews on hover feature</strong></p><p><strong>🧠Step 1: Clarify the Product &amp; Its Purpose</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3KvnjdZJ76CG5sEU9_ONWw.png" /></figure><p>💼 <strong>Step 2: Explain Product &amp; Business Goals</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*oRrvqbyyYn5GMX7YaftvqQ.png" /></figure><p>📊 <strong>Step 3: Define Success Metrics (AARRR Funnel)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*65DhCPyTokFkoe_WvXRMhg.png" /></figure><p>Tip: Align each metric with its upstream or downstream impact —</p><p>Eg: Engagmenet ↑ → Retention ↑ → Revenue ↑</p><p><strong>Guardrail Metrics</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xu34tbYHdHEfrTmpgU889Q.png" /></figure><p>After defining primary AARRR metrics, I evaluate <em>Guardrail Metrics</em> to ensure the feature improves discovery without harming UX, performance, or long-term retention.</p><blockquote><strong>2. Diagnosing a Metric Change</strong></blockquote><p><strong>4-Step Framework:</strong></p><p><strong>Step 1: Scope Out the Metric Change</strong></p><ul><li><strong>Definition:</strong> What exactly is the metric? (numerator/denominator if ratio)</li><li><strong>Importance:</strong> Is this critical for business?</li><li><strong>Timeframe:</strong> One-time dip or ongoing trend? (daily/weekly/monthly)</li><li><strong>Magnitude:</strong> How big is the change? (absolute &amp; %, YoY comparison)</li></ul><p><strong>Step 2: Hypothesize Contributing Factors</strong></p><p><strong>4 Buckets:</strong></p><ol><li><strong>Accidental:</strong> Bugs, logging errors</li><li><strong>Natural:</strong> Seasonality, holidays, day-of-week effects</li><li><strong>Internal:</strong> New features, UI changes, algorithm tweaks, policy changes</li><li><strong>External:</strong> Competitors, macro events (pandemic, recession)</li></ol><p><strong>Tip:</strong> Walk up the funnel — start local, then go broader</p><p><strong>Step 3: Validate Each Factor</strong></p><ul><li>Slice by user demographics (age, location, device, account age)</li><li>Check upstream metrics (funnel inputs)</li><li>Compare engagement ratios (likes/comments, shares/comments)</li><li>Cohort analysis</li></ul><p><strong>Step 4: Classify Each Factor</strong></p><ul><li><strong>Root Cause:</strong> Direct driver of metric change</li><li><strong>Contributing Factor:</strong> Helps explain but not main reason</li><li><strong>Correlated Result:</strong> Symptom, not driver</li><li><strong>Unrelated Factor:</strong> No impact</li></ul><p><strong>Example: YouTube — Average watch time per session is dropping</strong></p><p><strong>Step 1️⃣: Define the Scope</strong></p><p><strong>Goal:</strong> Understand what changed, why it matters, and how big the change is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*0YIZ54Fa6MlU_dPkYF1BDA.png" /></figure><p><strong>Step 2️⃣: Generate Hypotheses</strong></p><p><strong>Goal:</strong> List possible causes across 4 logical buckets — from local → broadest. Build a structured hypothesis list before jumping into data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oi9NBxOuFB0uk1n7KfCc4Q.png" /></figure><p><strong>Step 3️⃣: Validate Each Hypothesis</strong></p><p><strong>Goal:</strong> Test each hypothesis systematically with checks, validation, segmentation &amp; cohorting. Narrow down which bucket actually explains the drop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F9lm0SN3NGC9j_5ugMgFiw.png" /></figure><p><strong>Step 4️⃣: Conclude — Classify Findings</strong></p><p><strong>Goal:</strong> Summarize outcomes clearly in 4 classifications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HCPId2eCVPfnXhcmbE4dig.png" /></figure><p><strong>Final Summary</strong>: The drop in average watch time per session was primarily driven by the <strong>Android UI redesign (v19.4)</strong> that increased friction and reduced viewing continuity. The impact was <strong>amplified by higher ad frequency</strong>, while <strong>session count decline</strong> was a symptom, not a cause. No evidence of <strong>data errors</strong> or <strong>seasonality</strong> found.</p><p><strong>Bonus: Segment Vs Cohort Analysis</strong></p><p><strong>Segment</strong> = A group based on <strong>who they are</strong> (characteristics) <strong>Cohort</strong> = A group based on <strong>when they did something</strong> (time-based action)</p><p><strong>Simple Examples</strong></p><p><strong>Segment Analysis:</strong></p><ul><li>“Premium users” vs “Free users”</li><li>“Users from India” vs “Users from USA”</li><li>“iOS users” vs “Android users”</li><li>“Age 18–25” vs “Age 26–35”</li></ul><p>These groups are based on attributes or characteristics that can change or be true at any time.</p><p><strong>Cohort Analysis:</strong></p><ul><li>“Users who signed up in January 2024”</li><li>“Users who made their first purchase in Q4 2023”</li><li>“Customers who joined during Black Friday sale”</li></ul><p>These groups are based on a shared experience at a specific time period.</p><p><strong>Real-World Example</strong></p><p>Imagine an e-commerce app:</p><p><strong>Segment Analysis:</strong> You compare “Mobile app users” vs “Website users” to see who spends more. A user could be in both segments if they use both platforms.</p><p><strong>Cohort Analysis:</strong> You track “Users who joined in January 2025” and measure their behavior over the next 6 months (retention, purchases, etc.). You compare this January cohort with the February cohort to see if one group is more loyal.</p><p><strong>Key Difference:</strong> Segments can overlap and change (someone can switch from free to premium), but cohorts are fixed in time (if you joined in January, you’ll always be part of the January cohort).</p><blockquote><strong>3. Assessing Metric Trade-Offs (Part of Diagnosing)</strong></blockquote><p><strong>Question Type:</strong> One metric is going up other going down</p><p><strong>4-Step Framework:</strong></p><p><strong>Step-1: Clarify the metrics</strong></p><p><strong>Step-2: Tie back to product &amp; business goals</strong></p><p><strong>Step-3: Explore the possible next steps</strong></p><p><strong>Step 4: Communicate Your Thought Process</strong></p><p><strong>Example: Revenue is up 1%, time spent is down 3%. What do you do?</strong></p><p><strong>Step 1: Clarify the Metrics</strong></p><p>👉 Goal: Make sure you fully understand what changed and whether it matters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*jWkTfvtCcbisrNWknqF_UA.png" /></figure><p><strong>Step 2: Tie Back to Product &amp; Business Goals</strong></p><p>👉 Goal: Decide which metric matters more for the company long-term.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ckKcDZty56iCCaZaB1Qf6Q.png" /></figure><p><strong>Step 3: Explore Possible Next Steps (Revert / Mitigate / Accept)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*IxslVVtMpmdN9H874onn8Q.png" /></figure><p><strong>Step 4: Communicate Your Thought Process</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*YAY6bGoKfn6OuIRYsv_Cgw.png" /></figure><blockquote><strong>4. Designing A/B Tests</strong></blockquote><p><a href="https://medium.com/@mdmoseena22/a-b-testing-quick-revision-notes-for-data-science-interviews-236e6354b243">A/B Testing Quick Revision Notes for Data Science Interviews</a></p><blockquote><strong>Quick Interview Checklist</strong></blockquote><p><strong>Before Interview:</strong></p><ul><li>Research company’s primary metrics</li><li>Understand company mission and product goals</li><li>Review AARRR framework</li><li>Practice metric definition for their products</li></ul><p><strong>During Interview:</strong></p><ul><li>Always clarify before answering</li><li>Connect metrics to business goals</li><li>Mention guardrail metrics</li><li>Show structured thinking (use frameworks)</li><li>Communicate your reasoning process</li></ul><p><strong>Key Takeaway:</strong> Product sense is about connecting data to real user experiences and business outcomes. Always think: “Why does this metric matter to the business?”</p><p><strong>INTERVIEW QUESTIONS:</strong></p><blockquote>1. Facebook: Imagine the social graphs for both Facebook and Twitter. How do they differ? What metrics would you use to measure how these social graphs differ?</blockquote><p>This question falls under <strong>“Defining Success Metrics”</strong> category</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bxb-u-JKfKN4PadRjmcBbQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*xlFwY8FB_W9PIjeqTzxROg.png" /></figure><p><strong>Step 3: Define Success Metrics (AARRR not needed here → but graph metrics)</strong></p><p><strong>Degree Distribution (core metric)</strong></p><ul><li>Facebook: narrow &amp; symmetric</li><li>Twitter: highly skewed (few users have millions of followers)</li></ul><p>Metrics:</p><ul><li><strong>Skewness of degree distribution</strong> (Twitter &gt;&gt; Facebook)</li><li><strong>Kurtosis</strong> (Twitter &gt;&gt; Facebook)</li><li><strong>Mean vs Median degree gap</strong> (Twitter huge gap)</li></ul><blockquote>2. Uber: Why does surge pricing exist? What metrics would you track to ensure that surge pricing was working effectively?</blockquote><p>This question falls under <strong>“Defining Success Metrics”</strong> category</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rsJeZ82Uti_qtHGh35r9Xg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*Am4Kb8gvv6e4-t4GliCkjA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7AXha7tvM88xFIvTiX9P4A.png" /></figure><p><strong>Guardrail Metrics (To ensure surge is not harming UX)</strong></p><p>These protect the user experience even when revenue goes up:</p><ul><li><strong>Customer support tickets related to pricing</strong></li><li><strong>Cancellation Rate (rider &amp; driver)</strong></li><li><strong>PR / Social media sentiment</strong> (“Uber overcharging”, “Surge unfair”)</li><li><strong>long term Churn</strong></li></ul><p>Guardrails ensure surge <strong>helps the marketplace</strong> without <strong>hurting the brand</strong>.</p><blockquote>3. Google: Mozilla wants 2x payment for Google to remain default search on Firefox. Should we take the deal? How to estimate upper bound?</blockquote><p>This question falls under <strong>“Should we invest?”</strong> or <strong>“Should we pay?”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*p_tJ8udAu32dP6g_YqZ6cQ.png" /></figure><p><strong>Step-2: Explain Product &amp; Business Goals</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r2EV8GH4ZaGCwCTyUlGJ1g.png" /></figure><p>If we lose default search on Firefox → we lose a chunk of searches → competitors gain → Google loses revenue AND strategic advantage.</p><p><strong>Step 3: Define Key Metrics / Estimate Upper Bound Payment</strong></p><p><strong>The real question: How much is it worth to remain default?</strong></p><p>We estimate in two parts:</p><p><strong>Part A — Direct Financial Value (Revenue-based upper bound)</strong></p><p><strong>Upper Bound(Revenue) = Ad revenue generated from Firefox searches × % users who won’t switch manually</strong></p><p><strong>Example:</strong></p><ol><li>Firefox global search queries per year</li><li>Monetizable queries × CPC × CTR</li><li>% of users who stay with default (historical evidence: Safari → 80–90% stick with default)</li></ol><p><strong>Upper Bound (Revenue) = Lost Search Ad Revenue if Google is NOT the default</strong></p><p><strong>Part B — Strategic Value (Non-financial upper bound)</strong></p><p>Google also gains:</p><ul><li>Data for improving algorithms (search ranking, ads, maps, shopping)</li><li>Preventing Bing/DuckDuckGo from growing</li><li>Maintaining brand dominance</li><li>Avoiding antitrust risks from losing share</li></ul><p><strong>Upper Bound (Strategic) = Benefit of denying traffic to competitors + ML flywheel advantage</strong></p><p><strong>Final Upper Bound Formula</strong></p><p>Max Payment Google Should Pay = Lost Revenue + Strategic</p><p><strong>Decision Rule</strong></p><ul><li>If <strong>2× cost &lt; Upper Bound → TAKE THE DEAL</strong></li><li>If <strong>2× cost &gt; Upper Bound → WALK AWAY</strong></li></ul><p>This is <em>exactly</em> how Google negotiates its deal with Apple for Safari.</p><blockquote>4. Linkedin: What metrics track LinkedIn Feed engagement? What product ideas improve these metrics?</blockquote><p>The question has <strong>two parts</strong>: <strong>Defining Success Metrics</strong> + <strong>additional ideation step.</strong></p><p>So we’ll answer it using the <strong>3-step success metric framework</strong>, and then add <strong>Step 4: Product Ideas</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*Vfybs6LdjNX2A5yqeBlIWw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*hLGtp3wTtMg7WmXrUNxN7w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uq5maLWLIdBzN0gJ8rT67Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lnk2rAgmwT700nEXRuX_Ag.png" /></figure><blockquote>5. Amazon: If you were to plot the average revenue per seller on the Amazon marketplace, what would the shape of the distribution look like?</blockquote><p><strong>Right-skewed distribution (Long tail)</strong></p><p><strong>Why: Pareto Principle (80/20 Rule)</strong></p><ul><li>80% of outcomes from 20% of causes</li></ul><p>Examples:</p><ul><li>80% of world’s income from top 20% of people</li><li>80% of tech crashes from 20% of bugs</li></ul><p><strong>Amazon Marketplace Reality:</strong></p><ul><li><strong>Many small sellers:</strong> Low revenue (left side — high frequency)</li><li><strong>Few power sellers:</strong> Enormous revenue (right tail — low frequency)</li><li>Result: Right-skewed distribution</li></ul><blockquote>6. Facebook: Besides posts Facebook is legally obligated to remove, what other types of posts should Facebook take down? What features would you use to identify these posts? What are the trade-offs that need to be considered when removing these posts?</blockquote><p>The question falls under “Content Moderation Product Strategy” Question</p><p><strong>Step 1: Classify Harm Levels (What Should Facebook Remove?)</strong></p><p><strong>A. Must Remove (High Harm)</strong></p><p>These violate user safety &amp; legal obligations:</p><ul><li>Violence, graphic gore</li><li>Child exploitation, nudity</li><li>Hate speech, threats, bullying</li><li>Terrorist content, organized crime</li><li>Selling regulated goods (guns, drugs, organs)</li><li>Scams, fraud, phishing</li></ul><p><strong>B. Should Intervene but Not Always Remove</strong></p><p>These get warning labels instead of takedowns:</p><ul><li>Health misinformation</li><li>Political misinformation</li><li>AI-generated deceptive content</li><li>Clickbait, sensationalized news</li></ul><p>Reason: Balance <strong>safety</strong> vs <strong>free expression</strong>.</p><p><strong>Step 2: How to Identify These Posts</strong></p><p><strong>Detection Signals (Features for ML models):</strong></p><ol><li><strong>Content Signals</strong> (what’s IN the post):</li></ol><ul><li>Text: keywords, hate speech phrases, sensitive combinations</li><li>Images: nudity detection, graphic violence</li><li>Video: violent content analysis</li></ul><p><strong>2. User/Account Signals</strong> (who is posting):</p><ul><li>Account age, verification status</li><li>Past violations history</li><li>Linked to known bad actors</li><li>Fake/suspicious account patterns</li></ul><p><strong>3. Behavioral/Context Signals</strong> (posting patterns):</p><ul><li>Coordinated posting (multiple accounts, same message)</li><li>Sudden activity spikes</li><li>Posting frequency anomalies</li><li>Time-of-day patterns (bot-like behavior)</li><li>Group/page history (spam/violation rates)</li></ul><p><strong>Detection Methods:</strong></p><ul><li>ML models using above signals</li><li>User reports/flags</li><li>Human review teams (manual verification)</li></ul><p><strong>Step 3: What Are the Trade-Offs?</strong></p><p><strong>False Positives (Removing Good Content):</strong></p><ul><li>Risk: Censorship accusations, user frustration, regulatory backlash</li><li>Example: Political leader’s controversial-but-legal post removed</li></ul><p><strong>False Negatives (Leaving Harmful Content):</strong></p><ul><li>Risk: Physical harm, media backlash, trust loss</li><li>Example: COVID misinformation spreads to millions</li></ul><p><strong>Balancing Strategy:</strong></p><ul><li>Violence/terrorism: Zero tolerance for false negatives</li><li>Political speech: Minimize false positives</li><li>High-profile accounts: Human-in-the-loop review before removal</li></ul><blockquote>7. Amazon. The Amazon books team finds that books with more complete author profiles sell more. A team implements a feature which scrapes Wikipedia and Goodreads to automatically fill in more information about authors, hoping to see an improvement in sales. However, sales don’t change why might this be?</blockquote><p><strong>Answer: Correlation ≠ Causation</strong></p><p><strong>The Problem:</strong> Complete profiles <strong>correlate</strong> with sales but don’t <strong>cause</strong> sales.</p><p><strong>Real Drivers (Confounding Variables):</strong> Books with complete profiles typically have:</p><ul><li>Established publishers</li><li>Professional cover design</li><li>Marketing budgets</li><li>Author name recognition/reputation</li></ul><p><strong>What Amazon Did:</strong></p><ul><li>Only filled in profile info (symptom)</li><li>Didn’t change actual sales drivers (root causes)</li></ul><p><strong>Why Sales Didn’t Change:</strong></p><ul><li>Covers stayed the same</li><li>Marketing didn’t improve</li><li>Author reputation unchanged</li><li>Publisher quality unchanged</li></ul><blockquote>8. Snapchat: Let’s say Snapchat saw an overall 5% decrease in <strong>daily active users</strong>, a trend that had been consistent over the week. How would you go about determining the root cause of this?</blockquote><p>This question falls under <strong>“Diagnosing metric change”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*0g21SskLcuezZFDcQMtZNQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9MUi-qJmX0tb6hn9DI2GfQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_lqIO9oZpycZQDWmrljHyQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ifLig6DLv0d0Xv9uQYhLfw.png" /></figure><blockquote>9. Pinterest: Say you ship a new search ranking algorithm on Pinterest. What metrics would you use to measure the impact of this change?</blockquote><p>This question falls under <strong>“Defining Success Metrics”</strong> question</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*HKFrsJ1I8XOd4tao1ucZ4A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*muvbMtMAfgCzg_8PnlEc6Q.png" /></figure><p><strong>Step 3: Define Success Metrics (AARRR Funnel)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1cIYXiD7aZsLo7F9KXtHNg.png" /></figure><p><strong>Guardrail Metrics</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*V5IrLhcQxtwR-IXYEFfL4w.png" /></figure><blockquote>10. Netflix: Say a given category, such as sci-fi TV shows, has less total watch time, compared to other similar categories. What metrics would you look into to determine if the problem is that people are’nt interested in that category of content(demand problem), or if the category has interest but the content is bad (supply problem)</blockquote><p>It is a <em>Define Success Metrics</em> question with a mini diagnostic twist.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*_eGbFZ-BGyPDg5gGoBsGkg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*OsZjXdqMRj2fq4EHMmvpyw.png" /></figure><p><strong>Step 3: Define Metrics (AARRR Funnel) → Demand vs Supply</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*exuaQE6UMRm8AWqFnu0pNw.png" /></figure><p><strong>Guardrail Metrics:</strong><br> To ensure that diagnosing and fixing Sci-Fi watch-time issues doesn’t degrade overall Netflix health, I’d track:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*wnfvGwlTw7dec_v4gyWbog.png" /></figure><p>Along with the main success metrics (AARRR), I’ll monitor guardrail metrics to ensure improving Sci-Fi doesn’t harm the broader Netflix experience. For example: overall engagement, non-Sci-Fi genre watch time, buffering &amp; crash rates, recommendation diversity, and churn.</p><blockquote>11. Apple: Say you have data on millions of Apple customers and their purchases made at physical Apple retail stores. How could customer segmentation analysis increase a store’s sales performance? What techniques would you use to segment brick &amp; mortar customers into different groups?</blockquote><p>This is a “Defining a Product Metric / Analytical Feature” question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*e4HLoRLJrGN5oCn5Wgbegg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*k__-T6OgG5d1o6KtZzSUMw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*pVRhQQtnxSTz_LBSaAMksg.png" /></figure><p><strong>Guardrail Metrics:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aFxG-QClUJzZCm6mQ7twvA.png" /></figure><p>While segmentation can increase sales, we also track guardrail metrics to ensure it doesn’t harm experience, fairness, or operations.</p><blockquote>12. Facebook: If 70% of Facebook users on iOS also use Instagram, but only 50% of Facebook users on Android also use Instagram, how would you go about identifying the underlying reasons for this discrepancy in usage?</blockquote><p>This question falls under “<strong>Diagnosing a Metric Change”</strong> category</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*NMZwQ4tg11RxHIov4O5rDw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1tmi4UVLBjTadlyE3YgaWQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3d5uNzM2-wUmwPS4R2JOSQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*K8h6yoFFog0FFQ8KW4v3zg.png" /></figure><blockquote>13. Capital One: How would you assess the stickiness of the Capital One Quicksilver credit card?</blockquote><p>This question falls under <strong>“Defining Success Metrics” </strong>category</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NxZ8v4ENNYofkzuridzZoA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*GFV9U45H6y96_9nfvboYsA.png" /></figure><p><strong>Step 3: Define Success Metrics (AARRR-style for Credit Cards)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*M-trNVlYn7yjonu3gP-vFA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*StLgXPS1-J8k_zluEq8l1Q.png" /></figure><p>While measuring stickiness (DAU/MAU, transaction frequency, spend retention), I’d monitor guardrails like delinquency rate, fraud, customer complaints, and cashback cost to ensure we’re driving <em>healthy</em>, profitable usage — not risky or expensive behavior.</p><blockquote>14. Google: Say you worked on YouTube Premium, which is an ad-free version of YouTube bundled with YouTube Music - — a music streaming service. You’re launching the product in a few new countries — how would you determine pricing for each country?</blockquote><p>This is a <strong>Product Strategy / Pricing Strategy</strong> question.</p><p><strong>Step 1: Clarify the Goal of Pricing</strong></p><p>Before pricing, always ask:</p><p><strong>What is success for this launch?</strong></p><p>Possible goals:</p><ul><li><strong>Maximize adoption</strong> (grow market share quickly → low price)</li><li><strong>Maximize revenue</strong> (premium pricing)</li><li><strong>Cover content/licensing costs</strong></li><li><strong>Match competition</strong></li><li><strong>Penetrate fast, monetize later</strong> (common in emerging markets)</li></ul><p><strong>For this answer:</strong></p><p>Assume the goal is:</p><p><strong>Set sustainable pricing that balances adoption, competitiveness, and profitability.</strong></p><p><strong>Step 2: Core Pricing Approaches (Use All 3)</strong></p><p>We use <strong>3 complementary pricing lenses</strong>:</p><ol><li><strong>Cost-Based Pricing (Floor price)</strong></li></ol><p><strong>Find minimum price needed to cover costs.</strong></p><p>Include:</p><ul><li>Content licensing cost (varies by country)</li><li>Bandwidth/hosting costs</li><li>Customer support cost</li><li>Local marketing &amp; operations</li><li>Lost ad revenue (since Premium removes ads)</li></ul><p><strong>This sets the minimum price below which we lose money.</strong></p><p><strong>Example:</strong></p><p>If total cost per user = $3/month<br>Price must be <strong>above $3</strong> to be sustainable.</p><p><strong>2. Value-Based Pricing (Ceiling price)</strong></p><p>Determine <strong>what people in that country are willing to pay.</strong></p><p>How:</p><ul><li>Surveys, focus groups</li><li>Van Westendorp price sensitivity</li><li>Benchmark from existing Premium markets</li><li>Adjust for GDP per capita / purchasing power</li></ul><p><strong>This sets the highest price users are willing to pay.</strong></p><p><strong>Example:</strong></p><p>Users say: I’d pay up to $8/month for Premium features.</p><p>So ceiling = <strong>$8</strong>.</p><p><strong>3. Competitor-Based Pricing (Market parity)</strong></p><p>Check prices of:</p><ul><li>Spotify</li><li>Apple Music</li><li>Netflix</li><li>Amazon Prime</li><li>Regional services (JioSaavn, Gaana, Hotstar, etc.)</li></ul><p>Prices must align with what consumers already expect.</p><p><strong>Example:</strong></p><p>If Spotify costs $5/mo in that country<br> → YouTube Premium must stay close to that bracket.</p><p><strong>Step 3: Triangulate the Final Price</strong></p><p>Use the <strong>overlap</strong> between all three pricing models:</p><p><strong>Floor (Cost-Based)</strong> → must be above<br><strong>Market Price (Competitor)</strong> → must be comparable<br><strong>Ceiling (Value-Based)</strong> → must be below</p><p>The sweet spot = <strong>intersection of these three.</strong></p><p><strong>Example:</strong></p><ul><li>Cost floor: $3</li><li>Competitors: $5–$6</li><li>Value ceiling: $8</li></ul><p><strong>Final recommended price: $5.99</strong></p><p><strong>Optional Add-On: Test and Optimize</strong></p><p>Before finalizing:</p><ul><li><strong>A/B test multiple price points</strong></li><li>Offer <strong>intro discounts</strong></li><li>Offer <strong>student/family plans</strong></li><li>Offer <strong>geographic pricing localization</strong></li><li>Monitor churn &amp; conversion elasticity</li></ul><p><strong>Summary:</strong></p><p>To determine pricing for YouTube Premium in new countries:</p><ol><li><strong>Clarify pricing goal</strong> (adoption vs revenue vs parity).</li><li>Use <strong>three pricing models</strong>:</li></ol><ul><li><strong>Cost-Based Pricing</strong> to understand the minimum sustainable price</li><li><strong>Value-Based Pricing</strong> using user willingness-to-pay</li><li><strong>Competitor-Based Pricing</strong> to match local market norms</li></ul><p><strong>3. Blend the three approaches</strong> to find a price that covers costs, matches perceived value, and stays competitive.</p><p><strong>4. Validate through A/B tests</strong>, discounts, and tiered plans before universal rollout.</p><blockquote>15. Twitter: Should Twitter add Facebook-style emoji reactions (love, haha, sad, angry, etc.) to tweets?</blockquote><p>This question falls under <strong>Product Decision / Should We Build This Feature ? Question</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*Qec2jf1Kcgcqj2ltFsZ6lg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p6LhSOnpIUygD-1AlWO6cw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*YPJZ5mBbkt3BCylmzVcK2Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*l3CDTSff4yHwxY4bt69oeQ.png" /></figure><p><strong>5️⃣ Final Recommendation</strong></p><p><strong>Test the feature (not fully launch)</strong> if and only if<em>:</em></p><ul><li>Data shows high reply counts matching reaction types.</li><li>Surveys confirm user desire.</li><li>Stakeholders agree on the likely tradeoff: <strong>Reactions ↑, Replies ↓</strong></li></ul><p>Run an A/B test measuring:</p><ul><li>Tweet engagement rate</li><li>Replies per tweet</li><li>Session length</li><li>Time spent in feed</li><li>Content creation rate</li><li>Negative reaction abuse</li></ul><p>If <strong>engagement + retention &gt; comment loss</strong>, then ship globally.</p><blockquote>16. Slack: What metrics would you use to measure user engagement at Slack? How would you be able to tell early whether or not user engagement was declining?</blockquote><p>This question falls under “Defining Success Metric” Category</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*6RevtmHZrpP8HZzQwxb5QQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*QrRr_LHkA1YSsCrPgTTn3g.png" /></figure><p><strong>Step 3 — Define Success Metrics Using AARRR</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*XkTVdpOzplseTWIKXKRY8g.png" /></figure><p><strong>Guardrail metrics ensure you don’t accidentally “increase engagement” at the cost of product health.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*ibfwhzoooBoiki26h1zG9A.png" /></figure><p><strong>Step-4: Leading Indicators of Declining Engagement (Early Warning Signals)</strong></p><p>Slack is used daily — so early signals matter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*9xIzg5gsTTp3QB1WCTT40g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0348a44b1aac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Probability for Data Science Interviews]]></title>
            <link>https://medium.com/@mdmoseena22/probability-cbe709f3ea46?source=rss-72a4fb8cbdaf------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbe709f3ea46</guid>
            <category><![CDATA[interview-preparation]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[interview-questions]]></category>
            <dc:creator><![CDATA[MOHAMMED MOSEENA]]></dc:creator>
            <pubDate>Mon, 10 Nov 2025 11:41:18 GMT</pubDate>
            <atom:updated>2026-03-08T07:17:43.859Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Probability Topics to cover for data science</strong></blockquote><ol><li>Conditional Probability</li><li>Law of total Probability</li><li>Counting — Permutations &amp; Combinations</li><li>Probability distributions of random variables</li><li>Markov Chains</li></ol><blockquote><strong>1. Conditional Probability</strong></blockquote><p>Probability of event <strong>A</strong> given that <strong>B</strong> has occurred →</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x5g3LUkJmNTyrvyXluLDdw.png" /></figure><p>Example: Probability of having a disease <strong>given</strong> a positive test.</p><p><strong>Bayes’ Rule:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*Nj9BYZuzvD-TSPhh1EDGTw.png" /></figure><ul><li><strong>P(A)</strong> → Prior (belief before seeing data)</li><li><strong>P(B|A)</strong> → Likelihood (data evidence given A)</li><li><strong>P(B)</strong> → Marginal (total probability of data)</li><li><strong>P(A|B)</strong> → Posterior (updated belief after seeing data)</li></ul><p><strong>Independence:</strong></p><ul><li>If A and B are independent → <strong>P(A|B) = P(A) </strong>(Knowing B gives no info about A).</li><li>A and B can be independent <strong>given</strong> another event C.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*aqMonqAqYTMXcCwTmOIR0w.png" /></figure><p><strong>Why It Matters:</strong></p><ul><li>Prevents misleading conclusions (e.g., disease test accuracy vs. base rate).</li><li>Core concept in ML — helps compute the <strong>best conditional distribution</strong> (e.g., Naive Bayes).</li><li>Interview Hint: If question says “probability of X given Y,” think <strong>Bayes’ rule</strong>.</li></ul><blockquote><strong>2. Law of Total Probability</strong></blockquote><p>When multiple <strong>disjoint events</strong> B1,B2,…,Bn partition the sample space, the probability of event <strong>A</strong> can be expressed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*SpbGO2PpmHgOBZL4Wv2h1Q.png" /></figure><p><strong>Intuition:</strong></p><ul><li>It decomposes P(A) into a <strong>weighted sum</strong> of conditional probabilities — one for each possible scenario Bi.</li></ul><p>Example: Probability of a <strong>customer purchase</strong>, given different <strong>customer segments</strong>.</p><p><strong>Use with Bayes’ Rule:</strong></p><ul><li>Helps compute <strong>marginal probabilities</strong> (like P(B) in Bayes’ formula).</li></ul><blockquote><strong>Example</strong></blockquote><p>One in a thousand people have a particular disease. A test for the disease is 98% accurate when the person <em>has</em> the disease. However, the test has a 1% false alarm rate — meaning it wrongly flags 1% of healthy people as positive.</p><p>If someone tests positive, what’s the probability they actually have the disease?</p><p><strong>Given:</strong></p><ul><li>P(D) = 0.001 → Prior</li><li>P(T | D) = 0.98 → Likelihood</li><li>P(T | ND) = 0.01 → False positive rate</li></ul><p><strong>Step 0: Where Do These Numbers Come From?</strong></p><p>Before any math — ask yourself: <em>how did we get P(T|D) = 0.98 and P(T|ND) = 0.01?</em></p><p>These come from <strong>controlled lab testing</strong>, before the test ever reached the public.</p><p>Researchers take <strong>1,000 confirmed sick people</strong> → 980 test positive → P(T|D) = 0.98 Researchers take <strong>1,000 confirmed healthy people</strong> → 10 test positive → P(T|ND) = 0.01</p><p>This is what <strong>Likelihood</strong> means:</p><p><strong><em>P(T | D) = Likelihood</em></strong><em> → “Given we </em>know<em> someone is sick, how likely is a positive test?”</em></p><p>It measures how well the test <em>performs</em> under known conditions. It flows <strong>from disease → test result</strong>.</p><p><strong>The Four Pillars of Bayes’ Rule</strong></p><p>Now that the test is validated and released to the public, a random person walks in and tests positive. We don’t know their disease status. We now want to reason in <em>reverse</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*gu-xrJNCXhvUE2ooI2exeg.png" /></figure><p><strong>Step 1: Finding P(T) — The Law of Total Probability in Action</strong></p><p>A positive test can happen in <strong>two distinct, mutually exclusive ways</strong>:</p><ol><li>The person <strong>has</strong> the disease AND the test catches it</li><li>The person <strong>does NOT have</strong> the disease but the test fires anyway (false alarm)</li></ol><p>These two scenarios partition all possible outcomes — everyone either has the disease or doesn’t. The Law of Total Probability says: add up the probability of a positive test across every possible scenario, weighted by how likely each scenario is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*eIpCunDJkY_lm1bH9BVILw.png" /></figure><p>So roughly <strong>1.1% of the entire population</strong> will test positive — combining the rare true positives with the much more common false positives from healthy people.</p><p>This is the <strong>marginal probability</strong> of testing positive. It doesn’t condition on disease status — it’s the overall rate you’d observe if you tested everyone.</p><p><strong>Step 2: Apply Bayes’ Rule — Going From Likelihood to Posterior</strong></p><p>Now we flip the question. We <em>observe</em> a positive test and want to know the probability of disease:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*NRysmR1ty3qQ88M2XMELCQ.png" /></figure><p><strong>The Surprising Interpretation</strong></p><p>Even with a “98% accurate” test, only ~9% of positive results are true positives.</p><p>How can this be? The test <em>sounds</em> nearly perfect. Here’s the intuition:</p><p>Imagine testing <strong>100,000 people</strong> from this population:</p><ul><li><strong>100</strong> have the disease → 98 test positive ✅ (true positives)</li><li><strong>99,900</strong> are healthy → ~999 test positive ❌ (false positives)</li></ul><p>Out of <strong>1,097 total positive tests</strong>, only <strong>98 are real</strong>. That’s 98/1097 ≈ 9%.</p><p>The low <strong>prior probability</strong> of disease (only 0.1% of people are sick) means even a tiny false-positive rate in the huge healthy population swamps the signal from the rare diseased group.</p><p><strong>Likelihood vs. Posterior — The Core Difference</strong></p><p>This is the most important conceptual distinction in all of Bayesian thinking:</p><p><strong><em>P(T | D) = 0.98</em></strong><em> — Likelihood This says: </em>“If you are sick, the test will catch you 98% of the time.”<em> It is a property of the </em><strong><em>test itself</em></strong><em>, measured in controlled conditions.</em></p><p><strong><em>P(D | T) = 0.09</em></strong><em> — Posterior This says: </em>“If your test came back positive, there’s only a 9% chance you’re actually sick.”<em> It is a property of </em><strong><em>what the test result means in the real world</em></strong><em>, which depends on how rare the disease is.</em></p><p>The test hasn’t changed. What changed is the <strong>direction of reasoning</strong> — from “knowing disease, predicting test” to “knowing test, inferring disease.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*oDYKoCpobcciWH_1-C8ugw.png" /></figure><p><strong>Why this matters for Data Science</strong>:</p><p>In machine learning, many core models are built on conditional probability — <strong>P(output | features)</strong>. Naive Bayes does this explicitly via Bayes’ Rule, logistic regression models P(class | features) directly, and even neural networks are ultimately learning this conditional distribution. The Law of Total Probability underlies how we compute marginals when reasoning across multiple scenarios. These aren’t just theory — they’re the mathematical backbone of how models learn and predict.</p><blockquote><strong>3. Counting</strong></blockquote><p><strong>Permutations</strong> — <em>Order Matters:</em></p><p><strong>Formula</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/1*ytgyeewQjHlg92tIa4s-uA.png" /></figure><p><strong>Example</strong>: How many ways can five people sit around a lunch table?</p><p>Since arrangements around a circular table are considered the same under rotation, we fix one person’s seat. The remaining four people can be arranged in = (5–1)! = 4!</p><p><strong>Real-world Example</strong>: Making passwords</p><p><strong>Combinations</strong> — <em>Order Doesn’t Matter:</em></p><p><strong>Formula</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/170/1*39tfgS2z2rDUImY8R77vig.png" /></figure><p><strong>Example</strong>: What is the likelihood to draw four cards of the same suit?</p><p>We have 4 suits and each suit has 13 cards so probability will be = 4*13C4/52C4</p><p><strong>Real-world Example</strong>: Choosing Nearby restaurants on the map(order of selection doesn’t matter).</p><p><strong>Memory Trick:</strong></p><p>Since permutations care about order, making permutations always larger than combinations. That’s exactly why nCr has an extra <strong>r!</strong> in the denominator:</p><p><strong>nPr = n! / (n-r)!</strong> — order matters, more possibilities</p><p><strong>nCr = n! / (n-r)! × r!</strong> — divide out the r! arrangements we don’t care about</p><blockquote><strong>4. Probability distributions of random variables</strong></blockquote><p><strong>Random Variable (RV): </strong>A variable whose values are outcomes of a random phenomenon. It has an associated <strong>probability distribution</strong>.</p><p><strong>Types of Random Variables</strong></p><ol><li><strong>Discrete Random Variable</strong> — <em>Countable outcomes</em></li></ol><ul><li>Has a <strong>Probability Mass Function (PMF)</strong></li><li>Example: Number of heads in 5 coin tosses.</li></ul><p>2. <strong>Continous Random Variable</strong> — Uncountable<em> outcomes</em></p><ul><li>Has a <strong>Probability Density Function (PDF)</strong></li><li>Probability at an exact value x is <strong>0</strong>; probability is measured <strong>over intervals</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*ma-ieBYemarQspvFsu08Dw.png" /></figure><p><strong>Cumulative Distribution Function (CDF)</strong></p><ul><li>Gives probability that X≤ x</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*QWwRDYTSogcaJh_fZCvdgw.png" /></figure><p><strong>Key Conditions:</strong></p><ul><li>All probabilities are <strong>non-negative</strong>.</li><li>Total probability (sum/integral) = <strong>1</strong>.</li></ul><blockquote><strong>Quick Reference:</strong><br> For formulas and examples of key PMFs and PDFs, refer to: <a href="https://docs.google.com/spreadsheets/d/1w-9YYJKYwgn8HPo_RCEKaJMGGuL0TvZ2-P72s4U9pV4/edit?gid=0#gid=0">https://docs.google.com/spreadsheets/d/1w-9YYJKYwgn8HPo_RCEKaJMGGuL0TvZ2-P72s4U9pV4/edit?gid=0#gid=0</a></blockquote><blockquote><strong>5. Markov Chains</strong></blockquote><p>A <strong>Markov Chain</strong> is a stochastic process with a finite set of <strong>states</strong>, where the probability of moving to the next state depends <strong>only on the current state</strong>, not on past states.<br> → This is known as the <strong>Markov Property</strong> (memoryless property).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*Fj1rB7dlB2TIBw3Ux6IfbA.png" /></figure><p><strong>Transition Matrix (P):</strong></p><ul><li>Represents probabilities of moving from <strong>state i</strong> to <strong>state j</strong>.<br> Each row sums to 1:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*wwUZOl_BminxumEgquTz-Q.png" /></figure><p><strong>Types of States:</strong></p><ol><li><strong>Recurrent: </strong>The process <strong>returns</strong> to this state with probability 1.</li><li><strong>Transient: </strong>There’s a <strong>nonzero probability</strong> of <strong>never returning</strong> to this state.</li></ol><p><strong>Stationary Distribution:</strong><br>A probability vector that remains unchanged after transitions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/1*-yzb7taU9XJS6OrPQA599g.png" /></figure><p>Represents the <strong>long-run proportion of time</strong> the process spends in each state.</p><h4>Interview Questions:</h4><blockquote>1. Google: Two teams play a series of games (best of 7 — whoever wins 4 games first) in which each team has a 50% chance of winning any given round (no draws allowed). What is the probability that the series goes to 7 games?</blockquote><p>To have the series go to 7 games, each team must win exactly 3 games in the first 6 games. That way, the series will be tied 3–3, and the 7th game will act as the decider.</p><p>The probability of this happening is the probability of one team winning 3 games and losing 3 games out of 6 games: 6C3 * (0.5)³ * (0.5)³ = 5/16 = 0.31</p><p>Therefore there is 31% chance that series goes to 7 games. This problem follows a <strong>binomial distribution</strong>, since it models the number of successes in n independent Bernoulli trials.</p><blockquote>2. JP Morgan: Say you roll a die three times . What is the probability of getting two sixes in a row?</blockquote><p>There are three possible scenarios where two sixes in a row can occur:</p><ol><li><strong>First two rolls are sixes</strong>, third is not: P = (1/6)*(1/6)*(5/6)</li><li><strong>Last two rolls are sixes, first is not:</strong> P = (5/6)*(1/6)*(1/6)</li><li><strong>All three rolls are sixes</strong> (which also includes two sixes in a row): probabilty = (1/6)*(1/6)*(1/6)</li></ol><p>Total probability = 2*(1/6)*(1/6)*(5/6) + (1/6)*(1/6)*(1/6) = 11/216</p><blockquote>3. Uber: You roll three dice, one after another. What is the probability that you obtain three numbers in a strictly increasing order?</blockquote><p>To have three numbers in strictly increasing order, the dice rolls must all be <strong>distinct</strong>.</p><p>Let A = event of getting three distinct numbers.</p><p>P(A) = 6*5*4/216 = 5/9</p><p>Given that the numbers are distinct, the three outcomes can be arranged in 3!= 6 ways. Out of these, only <strong>1 arrangement</strong> is strictly increasing.</p><p>P(B|A) = 1/6</p><p>Therefore, the probability of getting numbers in strictly increasing order is: P(B) = P(A) * P(B|A) = (5/9)*(1/6) = 5/54</p><blockquote>4. Zenefits. Assume you have a deck of 100 cards with values ranging from 1 to 100, and that you draw two cards at random without replacement. What is the probability that the number of one card is precisely double that of the other?</blockquote><p><strong>Step 1: Total Possible Pairs</strong><br> Number of ways to choose any 2 cards (order doesn’t matter):</p><p>S = 100C2 = 100*99/2 = 4950</p><p><strong>Step 2: Favorable Pairs (One card = 2 × other)</strong><br>Possible valid pairs: (1,2),(2,4),(3,6),…,(50,100)</p><p>Total favorable pairs = <strong>50</strong></p><p><strong>Step 3: Probability (Order doesn’t matter)</strong></p><p>P = Favorable pairs/Total Pairs = 50/4950 = 1/99</p><p><strong>Note</strong>: If <strong>order matters</strong>,</p><ul><li>Total pairs = 100C1*99C1 = 100*99</li><li>Each favorable pair (e.g., (2,4) and (4,2)) has 2 arrangements, so favourable pairs = 2*50</li><li>P = (2*50)/(100*99) = 1/99 — so the final probability remains <strong>same</strong>.</li></ul><blockquote>5. JP Morgan: Imagine you are in a 3D space. From (0,0,0) to (3,3,3), how many paths are there if you can move only up, right, and forward?</blockquote><p>Number of steps we can take to move from (0, 0,0) to (3,3,3):</p><p>Consider each U/R/F as 1 unit</p><ul><li>3 Ups — U,U,U</li><li>3 Rights — R,R,R</li><li>3 Forwards — F,F,F</li></ul><p>Now we need to find the number of distinct ways the set {U,U,U,R,R,R,F,F,F} can be arranged</p><p>Total Paths = 9!/3!3!3!</p><blockquote>6. Amazon. One in a thousand people have a particular disease, and the test for the disease is 98% correct in testing for the disease. On the other hand, the test has a 1% error rate if the person being tested does not have the disease. If someone tests positive, what are the odds they have the disease?</blockquote><p>See detailed walkthrough in Example Section above</p><blockquote>7. Facebook: Assume two coins, one fair (having one side heads and one side tails) and the other unfair (having both sides tails). You pick one at random, flip it five times, and observe that it comes up as tails all five times. What is the probability that you are flipping the unfair coin?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*4AYmrWM-P1nGvoOCUGU6Jw.png" /></figure><blockquote>8. Goldman Sachs: Players A and B are playing a game where they take turns flipping a biased coin, with p probability of landing on heads (and winning). Player A starts the game, and then the players pass the coin back and forth until one person flips heads and wins. What is the probability that A wins?</blockquote><p><strong>Step 1: Break down the events</strong></p><ul><li><strong>A wins on first turn:</strong> probability = p</li><li><strong>A doesn’t win on first turn, B also fails:</strong> probability = (1−p)² → then it’s A’s turn again, <strong>game resets</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*VBZs6VKetwKdEzFj45Mamw.png" /></figure><p><strong>Step 2: Solve for P(A)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*PkfdCMd_bYI2Bc-IRuK9lA.png" /></figure><p><strong>Step 3: Intuition:</strong></p><ul><li>A has the first-move advantage.</li><li>Probability is higher than 0.5 for p&gt;0 because A always gets the first chance, and the game resets recursively.</li></ul><blockquote>9. Microsoft: Three friends in Seattle each told you it is rainy, and each person has a 1/3 probability of lying. What is the probability that Seattle is rainy, assuming that the likelihood of rain on any given day is 0.25?</blockquote><p><strong>Step 1: Conditional probabilities</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*ihS9r2zxtrdyHC52XoVJdw.png" /></figure><p><strong>Step 2: Total probability of all saying rainy</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*OniewiVO2utkDXjSjCtg0A.png" /></figure><p><strong>Step 3: Apply Bayes’ Rule</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*IdD6m_n-wMnYT7n4PYIEdg.png" /></figure><blockquote>10. Bloomberg: You draw a circle and choose two chords at random. What is the probability that those chords will intersect?</blockquote><ul><li>To define <strong>two chords</strong>, we need <strong>4 points</strong> on the circle: P1,P2,P3,P4.</li><li>There are <strong>3 ways</strong> to pair these 4 points into 2 chords:</li></ul><p>(P1, P2) &amp; (P3,P4)</p><p>(P1, P3) &amp; (P2, P4)</p><p>(P1,P4) &amp; (P2, P3)</p><ul><li>Only <strong>one of these 3 pairings</strong> results in the chords intersecting.</li></ul><p>Probability(two chords intersect)=1/3</p><p><strong>Key Point:</strong></p><ul><li>This result relies on the <strong>symmetry of the circle</strong> — any ordering of points is equally likely.</li><li>Only the “cross-pairing” (like (P1,P3)&amp; (P2,P4)) produces an intersection.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ofdttL6744PG_bPWEm12lw.png" /></figure><blockquote>11 Morgan Stanley. You and your friend are playing a game. The two of you will continue to toss a coin until the sequence HH or TH shows up. If HH shows up first, you win. if TH shows up first, your friend wins What is the probability of you winning?</blockquote><p><strong>Step 1: Analyze first toss</strong></p><ol><li><strong>First toss = T</strong> → sequence starts with T</li></ol><ul><li>Your friend <strong>wins immediately</strong> if next toss is H? Actually sequence TH occurs when <strong>first toss T + next H</strong> → friend wins</li><li>So probability friend wins = 1 (for this branch)</li></ul><p><strong>2. First toss = H</strong> → sequence starts with H</p><ul><li>Second toss = H → <strong>you win</strong> → probability = 1/2</li><li>Second toss = T → then sequence TH comes before HH→ <strong>friend wins</strong> → probability = 1/2</li></ul><p><strong>Step 2: Compute total probability</strong></p><ul><li>Toss is fair → first toss H or T with each probability 1/2</li><li>P(you win) = P(first H)⋅P(second H∣first H) = (1/2)*(1/2) = 1/4</li><li>P(friend wins)​=1–1/4 = 3/4</li></ul><p><strong>Intuition:</strong></p><ul><li>You can only win if <strong>the first toss is H and the second is H</strong>.</li><li>Any sequence starting with T or H followed by T leads to your friend winning.</li></ul><blockquote>12. JP Morgan: Say you are playing a game where you roll a 6-sided die up to two times and can choose to stop following the first roll if you wish. You will receive a dollar amount equal to the final amount rolled. How much are you willing to pay to play this game?</blockquote><p><strong>Step 1: Expected value of the first roll</strong></p><p>E[first roll] = (1+2+3+4+5+6)/6 = 3.5</p><p><strong>Step 2: Decide when to roll a second time</strong></p><ul><li>Rational strategy: <strong>reroll</strong> if first roll is <strong>1, 2, or 3</strong>, otherwise <strong>keep</strong> 4, 5, 6.</li><li>Probability of rerolling = 3/6=0.5</li><li>Expected value <strong>if rerolling</strong> = E[second roll] = (1+2+3+4+5+6)/6 = 3.5</li><li>Expected value <strong>if keeping first roll</strong> = average of 4, 5, 6 = (4+5+6)/3 = 5</li></ul><p><strong>Step 3: Total expected value</strong></p><p>E[game]=0.5*5+0.5*3.5=2.5+1.75=4.25</p><p>You should be willing to pay <strong>up to $4.25</strong> to play this game.</p><blockquote>13. Facebook: Facebook has a content team that labels pieces of content on the platform as either spam or not spam. 90% of them are diligent raters and will mark 20% of the content as spam and 80% as non-spam. The remaining 10% are not diligent raters and will mark 0% of the content as spam and 100% as non-spam. Assume the pieces of content are labeled independently of one another, for every rater. Given that a rater has labeled four pieces of content as good, what is the probability that this rater is a diligent rater?</blockquote><p><strong>Step 1: Define probabilities</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*f45f2gpfArE_WC389AOGbw.png" /></figure><p>where</p><p>DR — Deligent Raters</p><p>NDR — Non Deligent Raters</p><p>G — Good/Not Spam</p><p>B — Bad/ Spam</p><p><strong>Step 2: Marginal probability of labeling 4 goods</strong></p><p>For <strong>diligent rater</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*ordzxqlYfrXZwAZE4Lo7mw.png" /></figure><p>For <strong>non-diligent rater</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*DfNWcyfvtty_am1tcUoeqQ.png" /></figure><p>Total probability (law of total probability):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*xPya_ge2iHfxf-WpjBDx1w.png" /></figure><p><strong>Step 3: Apply Bayes’ Rule</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*QKqC9VwYzLnhrE-hcDWlpg.png" /></figure><p>P(DR|GGGG) <strong>≈0.7866.</strong></p><blockquote>14. D.E. Shaw A couple has two children. You discover that one of their children is a boy. What is the probability that the second child is also a boy?</blockquote><p><strong>Step 1: List all possible combinations</strong></p><p>For two children, sample space: {BB, BG, GB, GG}</p><p><strong>Step 2: Apply given information</strong></p><ul><li>At least <strong>one child is a boy</strong> → eliminate GG.</li><li>Remaining possibilities: {BB, BG, GB}</li></ul><p><strong>Step 3: Compute probability</strong></p><ul><li>Only <strong>BB</strong> satisfies “both are boys.”</li></ul><p>P(both boys ∣ at least one boy) = 1/3</p><p><strong>Key intuition:</strong></p><ul><li>Knowing “at least one boy” <strong>removes only the GG outcome</strong>, leaving three equally likely cases.</li><li>Only <strong>one of these three</strong> is BB → probability = 1/3.</li></ul><blockquote>15. JP Morgan: A desk has eight drawers. There is a probability of 1/2 that someone placed a letter in one of the desk’s eight drawers and a probability of 1/2 that this person did not place a letter in any of the desk’s eight drawers. You open the first 7 drawers and find that they are all empty. What is the probability that the 8th drawer has a letter in it?</blockquote><p><strong>Step 1: Define probabilities</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*y9kuBR0BCZUVPC7UbXI8Xg.png" /></figure><p><strong>Step 2: Apply Bayes’ Rule</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*CfFSZWBWwA6a_n3lXAStzg.png" /></figure><p><strong>Key intuition:</strong></p><ul><li>Observing <strong>7 empty drawers</strong> strongly increases the likelihood that <strong>no letter was placed at all</strong>, making the probability that the <strong>8th drawer has the letter</strong> smaller than naive 1/8.</li></ul><blockquote>16. Optiver: Two players are playing in a tennis match, and are at deuce (that is, they will play back and forth until one person has scored two more points than the other). The first player has a 60% chance of winning every point, and the second player has a 40% chance of winning every point. What is the probability that the first player wins the match?</blockquote><p><strong>Step 1: Define probabilities</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/1*G1M4zzqL9VFjHiQxsuMJQA.png" /></figure><p>Let P(Wf) = probability that <strong>Player 1 wins from deuce</strong>.</p><p><strong>Step 2: Recurrence relation</strong></p><p>Player 1 can win in these ways:</p><ol><li>Win <strong>two points in a row</strong>: F,F → probability = P(F)⋅P(F)</li><li>Win after <strong>returning to deuce</strong>:</li></ol><ul><li>Sequences F,S or S,F→ both return to deuce → probability = P(F)⋅P(S)⋅P(Wf) + P(S)⋅P(F)⋅P(Wf) = 2⋅P(F)⋅P(S)⋅P(Wf)</li></ul><p><strong>Step 3: Solve for P(Wf)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*_k0-U6K78pOoVk1nad-cLg.png" /></figure><blockquote>17. Facebook: Say you have a deck of 50 cards made up of cards in 5 different colors, with 10 cards of each color, numbered 1 through 10. What is the probability that two cards you pick at random do not have the same color and are also not the same number?</blockquote><p><strong>Step 1: Pick the first card</strong></p><p>P(first card) = 50C1/50C1 = 1</p><p><strong>Step 2: Pick the second card with a different color</strong></p><ul><li>Cards left = 49</li><li>Cards of <strong>different color</strong> = 40 (since 10 cards of the same color as first card)</li></ul><p>P(different color ∣ first card)=40C1/49C1 = 40/49</p><p><strong>Step 3: Pick the second card with a different number</strong></p><ul><li>Among the 40 cards of different color, <strong>4 cards have the same number</strong> as the first card (one in each of the 4 other colors)</li><li>Cards with <strong>different number</strong> = 36</li></ul><p>P(different number ∣ different color)=36/40</p><p><strong>Step 4: Total probability</strong></p><p>P(different color &amp; different number) = 1*(40/49)*(36/40) = 36/49</p><p><strong>Key intuition:</strong></p><ul><li>Step-by-step conditioning works well for <strong>multiple restrictions</strong> (here: color + number).</li><li>Always <strong>reduce the sample space</strong> based on previous choices.</li></ul><blockquote>18. SIG: Suppose you have ten fair dice. If you randomly throw these dice simultaneously, what is the probability that the sum of all the top faces is divisible by 6?</blockquote><p><strong>Step 1: Consider the first 9 dice</strong></p><ul><li>Let the <strong>sum of the first 9 dice modulo 6</strong> be r, where r∈{0,1,2,3,4,5}</li><li>To make the <strong>total sum divisible by 6</strong>, the <strong>10th die</strong> must satisfy:</li></ul><p>10th die ≡ (6−r) mod 6</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*WeJoLTABNiED2Fx8oqOXcg.png" /></figure><p><strong>Step 2: Probability</strong></p><ul><li>No matter what the sum of the first 9 dice is, <strong>exactly 1 out of 6 faces</strong> of the 10th die will make the total divisible by 6.</li></ul><p>P(sum divisible by 6)=1/6</p><p><strong>Key intuition:</strong></p><ul><li>This is a <strong>modular arithmetic trick</strong>: only the <strong>last die determines divisibility</strong>, once the sum of the first dice is known.</li><li>Each outcome of the last die is <strong>equally likely</strong>, giving probability 1/6.</li></ul><blockquote>19. Morgan Stanley: A and B play the following game a number k from 1–6 is chosen, and A and B will toss a die until the first person throws a die showing side k, after which that person is awarded $100 and the game is over. How much is A willing to pay to play first in this game?</blockquote><p><strong>Step 1: Define probabilities</strong></p><ul><li>Probability A rolls k on a single turn:</li></ul><p>P(A) = 1/6, similarly P(B) = 1/6</p><p>Let P(Af) = probability <strong>A wins if starting first</strong>.</p><p><strong>Step 2: Recurrence relation (first-step analysis)</strong></p><p>P(Af) = P(A) + (1-P(A))*(1-P(B))*P(Af)</p><p>P(Af) = (1/6) + (1–1/6)*(1–1/6)*P(Af)</p><p>P(Af) = (1/6) + (25/36)*P(Af)</p><p><strong>Step 3: Solve for P(Af)</strong></p><p>P(Af)-(25/36)*P(Af) = 1/6</p><p>(11/36)*P(Af) = 1/6</p><p>P(Af) = 6/11</p><p><strong>Step 4: Expected value to play first</strong></p><p>If A starts playing first probability of A winning = 6/11</p><p>If B starts playing first probability of A winning = 5/11</p><p>Therefore to play first A should be paying = (6/11–5/11)*100 = 100/11 ~ 9.09$</p><p><strong>Key intuition:</strong></p><ul><li>The advantage of playing first is <strong>small but positive</strong>.</li><li>First-step analysis with recurrence gives the <strong>probability advantage</strong>, which then converts to <strong>expected monetary value</strong>.</li></ul><blockquote>20. Airbnb: You are given an unfair coin having an unknown bias towards heads or tails. How can you generate fair odds using this coin?</blockquote><p><strong>Step 1: Toss the coin twice</strong></p><p>Possible outcomes: {HH,HT,TH,TT}</p><p>Probabilities: P(HT)=P(H)⋅P(T), P(TH)=P(T)⋅P(H)</p><p>Notice: P(HT)=P(TH)</p><p><strong>Step 2: Assign fair outcomes</strong></p><ul><li>Treat <strong>HT → “Head”</strong></li><li>Treat <strong>TH → “Tail”</strong></li><li><strong>Ignore</strong> HH and TT</li><li>This ensures <strong>fair odds</strong>, because the probabilities of HT and TH are <strong>equal</strong>, independent of the coin bias.</li></ul><p><strong>Step 3: Repeat as needed</strong></p><ul><li>Keep tossing <strong>pairs of coins</strong> until HT or TH appears.</li><li>Each valid outcome generates <strong>one fair result</strong>.</li></ul><p><strong>Key idea:</strong></p><ul><li>This is called the <strong>von Neumann trick</strong>: using pairs of biased outcomes to cancel out bias and generate fair results.</li></ul><blockquote>21. SIG: Suppose you are given a white cube that is broken into 3x3x3 = 27 pieces. However, before the cube was broken, all 6 of its faces were painted green. You randomly pick a small cube and see that 5 faces are white. What is the probability that the bottom face is also white</blockquote><p><strong>Step 1: Identify relevant cubes</strong></p><ul><li><strong>Cube with all 6 faces white:</strong> only the <strong>center cube</strong> → 1 out of 27</li><li><strong>Cube with 5 white faces:</strong> the <strong>6 face-center cubes</strong> → 6 out of 27</li></ul><p>P(all 6 faces white) = 1/27 ,</p><p>P(5 faces white)= P(all 6 faces white) + P(5 faces white one face painted) = 1/27 + (6/27)*(1/6) = 2/27</p><p><strong>Step 2: Apply conditional probability</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*BVivRbFwCPb-GwF1mClljw.png" /></figure><p>P(all 6 faces white | 5 faces white) = 0.5</p><blockquote>22. Goldman Sachs: Assume you take a stick of length 1 and you break it uniformly at random into three parts. What is the probability that the three pieces can be used to form a triangle</blockquote><p><strong>Step 1: Let cuts be X and Y</strong></p><ul><li>Denote the cut points along the stick as 0&lt;X&lt;Y&lt;1</li><li>Three pieces: X, Y−X, 1−Y</li></ul><p><strong>Step 2: Triangle condition</strong></p><ul><li>Three lengths can form a triangle if the <strong>sum of any two pieces &gt; the third</strong>.</li></ul><p>Equivalently: max⁡(X,Y−X,1−Y)&lt;1/2</p><p><strong>Step 3: Cases where triangle is NOT possible</strong></p><ol><li><strong>Both cuts on the same side of midpoint</strong></li></ol><ul><li>Either both on left or both on right</li><li>Probability = 1/2</li></ul><p><strong>2. Cuts on different sides but one piece &gt; 1/2</strong></p><ul><li>Probability = 1/4</li></ul><p><strong>Step 4: Probability that pieces form a triangle</strong></p><p>P(triangle)=1−1/2–1/4 = 1/4</p><blockquote>23. Lyft: What is the probability that, in a random sequence of H’s and T’s, HHT shows up before HTT?</blockquote><p><strong>Step 1: Analyze first toss</strong></p><ul><li><strong>First toss must be H</strong> for either pattern to occur first.</li></ul><p><strong>Step 2: Analyze second toss</strong></p><ol><li><strong>Second toss = H</strong></li></ol><ul><li>Sequence now: <strong>HH</strong> → HHT only needs <strong>T</strong>, HTT cannot appear yet</li><li>Probability <strong>HHT occurs first</strong> = 1</li></ul><p><strong>2. Second toss = T</strong></p><ul><li>Sequence now: <strong>HT</strong> → Could still get HHT or HTT</li><li>Probability <strong>HHT occurs first</strong> = 1/2</li></ul><p><strong>Step 3: Combine probabilities</strong></p><p>P(HHT before HTT) = P(2nd toss H)⋅1 + P(2nd toss T)⋅1/2</p><p>P(HHT before HTT) = 1/2⋅1 + 1/2⋅1/2 = 3/4</p><blockquote>24. Uber: A fair coin is tossed twice, and you are asked to decide whether it is more likely that two heads showed up given that either (a) at least one toss was heads, or (b) the second toss was head. Does your answer change if you are told that the coin is unfair?</blockquote><p><strong>Sceanrio-1: Fair coin ( p=1/2 )</strong></p><p>(a) Given <strong>at least one H</strong>:</p><ul><li>P(HH∣at least one H) = 1/3</li></ul><p>(b) Given <strong>second toss is H</strong>:</p><ul><li>P(HH∣second is H) = 1/2.</li></ul><p>So (b) is more likely.</p><p><strong>Scenario-2: Unfair coin (heads probability = p) — general formulas</strong></p><p>(a) Given at least one head:</p><ul><li>P(HH∣at least one H) = p²/(1-(1-p)²) = p/(2-p)</li></ul><p>(b) Given second toss is head:</p><ul><li>P(HH∣second is H) = p²/(p² + (1-p)*p) = p</li></ul><p>for 0 &lt; p &lt; 1=&gt; P(HH∣second is H) &gt; P(HH∣at least one H)</p><p><strong>Intuition:</strong> knowing the <em>specific</em> toss is H collapses the uncertainty about that toss (reduces sample space to two ordered outcomes), while “at least one H” leaves more possible outcomes, so it gives a smaller posterior for HH.</p><blockquote>25. Facebook: Three ants are sitting at the corners of an equilateral triangle. Each ant randomly picks a direction and begins moving along an edge of the triangle. What is the probability that none of the ants meet? What would your answer be if there are, instead, k ants sitting on all k corners of an equilateral polygon?</blockquote><p>For three ants sitting at the corners of an <strong>equilateral triangle</strong>, each ant randomly chooses to move <strong>clockwise(C) </strong>or <strong>anticlockwise (A)</strong> with equal probability 1/2.</p><p><strong>Case 1: 3 Ants on a Triangle</strong></p><ul><li><strong>For none of the ants to collide</strong>, all must move in the <strong>same direction</strong> (either all Clockwise or all Anticlockwise).</li><li>P(no collision) = P(all C)+P(all A) = 1/2³ + 1/2³ = 1/4</li></ul><p><strong>Case 2: k Ants on an Equilateral k-gon</strong></p><ul><li>For no ants to meet, again all must move in the same direction (either all Clockwise or all Anticlockwise).</li><li>P(no collision) = 1/2^k + 1/2^k= 1/2^(k-1)</li></ul><blockquote>26. Robinhood: A biased coin, with probability p of landing on heads, is tossed n times. Write a recurrence relation for the probability that the total number of heads after n tosses is even.</blockquote><p>Let</p><ul><li>p = probability of getting a head</li><li>Pn = probability that the total number of heads after n tosses is <strong>even</strong></li><li>Pn−1​ = probability that the total number of heads after n−1 tosses is <strong>even</strong></li></ul><p><strong>Step 1: Analyze the last toss</strong></p><ul><li>If the <strong>last toss is head</strong> (probability p), then the number of heads in the first n−1 tosses must be <strong>odd</strong> for the total to be even.</li><li>If the <strong>last toss is tail</strong> (probability 1−p),<br> then the number of heads in the first n−1 tosses must be <strong>even</strong> for the total to be even.</li></ul><p><strong>Step 2: Write the recurrence relation</strong></p><p>Pn = p*(1-Pn-1) + (1-p)*Pn-1</p><blockquote>27. Citadel: Alice and Bob are playing a game together. They play a series of rounds until one of them wins two more rounds than the other. Alice wins a round with probability p. What is the probability that Bob wins the overall series?</blockquote><p>Given</p><p>Probability that Alice wins a round = p</p><p>Probability that Bob wins a round = 1-p</p><p>Probability that Bob wins the game=&gt; P(B)</p><p>For Bob to win the game either of below two conditions</p><p>a. Two rounds are won by Bob</p><p>b. One round by Alice and other by Bob and game resets</p><p>P(B) = (1- p)*(1- p)+ (1-p)*p*P(B) + p*(1-p)*P(B)</p><p>P(B) = (1-p)²+ 2*p(1-p)*P(A)</p><p>P(B)(1–2p+2p²) = (1-p)²</p><p>P(B) = (1-p)²/(1–2p+2p²)</p><blockquote>28. Google: Say you have three draws of a uniformly distributed random variable between (0, 2). What is the probability that the median of the three is greater than 1.5?</blockquote><p>Three independent draws X1, X2, X3 ∼ U(0,2)</p><p><strong>Step 1. Single-variable probability</strong></p><p>P(X&gt;1.5) = 2−1.5/2=0.25</p><p>P(X≤1.5) = 1-P(X&gt;1.5) = 0.75</p><p><strong>Step 2. Median condition</strong></p><ul><li>Median &gt;1.5 if <strong>at least two of the three numbers are &gt; 1.5</strong></li></ul><p>Cases:</p><ol><li><strong>All three &gt; 1.5 =&gt; </strong>(0.25)³</li><li><strong>Exactly two &gt; 1.5</strong></li></ol><ul><li>Number of ways to choose which two: 3C2 = 3</li><li>p = 3*(0.25)²*(0.75)</li></ul><p><strong>Step 3. Total probability</strong></p><p>P(median&gt;1.5) = (0.25)³+3*(0.25)²*(0.75) = 10/64</p><blockquote>29. D.E. Shaw: Say you have 150 friends, and 3 of them have phone numbers that have the last four digits with some permutation of the digits 0, 1, 4, and 9. What’s the probability of this occuring?</blockquote><p><strong>Step 1. Probability of a single “success”</strong></p><ul><li>Total 4-digit sequences: 10⁴</li><li>Favorable sequences (permutations of 0,1,4,9): 4!</li></ul><p>P(success) = 4!/10⁴</p><p><strong>Step 2. Use binomial probability</strong></p><ul><li>Let X ∼ Binomial(n=150, p=4!/10⁴)</li><li>Probability exactly 3 successes:</li></ul><p>P(X=3) = (150C3)* P³ (1−P)¹⁴⁷</p><blockquote>30. Spotify: A fair die is rolled n times. What is the probability that the largest number rolled is r for each r in {1,2,3,4,5,6}?</blockquote><p>Roll a fair 6-sided die n times, Find P(largest number = r) where r∈{1,2,3,4,5,6}</p><p><strong>Step 1. Use cumulative probability</strong></p><p>P(max ≤ r) = (r/6)^n</p><p>P(max ≤ r−1)=((r−1)/6)^n</p><p><strong>Step 2. Probability of largest = r</strong></p><p>P(max = r) = P(max ≤ r)−P(max ≤ r−1) = (r/6)^n — ((r-1)/6)^n</p><blockquote>31. Goldman Sachs: Say you have a jar initially containing a single amoeba in it. Once every minute, the amoeba has a 1 in 4 chance of doing one of four things: (1) dying out, (2) doing nothing, (3) splitting into two amoebas, or (4) splitting into three amoebas. What is the probability that the jar will eventually contain no living amoeba?</blockquote><p><strong>Step 1. Let P = probability of eventual extinction</strong></p><ul><li>Dying: contributes 0.25</li><li>Doing nothing: experiment resets → contributes 0.25P</li><li>Splitting into 2: both must go extinct → contributes 0.25P²</li><li>Splitting into 3: all three must go extinct → contributes 0.25P³</li></ul><p>P = (0.25) + (0.25P) + (0.25P²)+ (0.25P³)</p><p><strong>Step 2. Simplify equation</strong></p><p>P³ + P² -3P + 1 = 0</p><p><strong>Step 3. Solve cubic</strong></p><p>P = sqrt(2) — 1 ~ 0.441</p><blockquote>32. Lyft: A fair coin is tossed n times. Given that there were k heads in the n tosses, what is the conditional probabity probability that the first toss was heads?</blockquote><p>Let A: first toss is head &amp; B: k heads in n tosses</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/217/1*__H2XHuyqB7wA8ZK6xtqxg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E6DqklcHFX50s3Z0u1iouw.png" /></figure><blockquote>33. Quora: You have N i.i.d draws of numbers following a normal distribution with parameters μ and sigma. What is the probability that k of those draws are larger than some value Y?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*eybpHa9iCezSlfaOMj465Q.png" /></figure><blockquote>34. Citadel: You have r red balls and w white balls in a bag. You continue to draw balls from the bag until the bag only contains balls of one color. What is the probability that you run out of white balls first?</blockquote><p><strong>Method 1: Combinatorial Approach — Step by Step</strong></p><p><strong>Setup</strong></p><ul><li>We have <strong>w + r balls</strong> in total</li><li>We can make <strong>w + r draws</strong> in total</li><li>Let’s consider each draw as a <strong>position</strong> in a sequence</li></ul><p><strong>The Key Condition</strong></p><p><strong>For white balls to run out first, the last ball drawn must be red.</strong></p><p>This means: <strong>All w white balls must be drawn in the first (w + r — 1) positions.</strong></p><p><strong>Counting Favorable Arrangements</strong></p><p>Now we have <strong>(w + r — 1) positions</strong> and need to place <strong>w white balls</strong> in them.</p><p>Think of it as choosing positions for each white ball:</p><ul><li><strong>1st white ball (W₁):</strong> Can go in any of the (w + r — 1) positions</li><li><strong>2nd white ball (W₂):</strong> Can go in any of the remaining (w + r — 2) positions</li><li><strong>3rd white ball (W₃):</strong> Can go in any of the remaining (w + r — 3) positions</li><li>…</li><li><strong>Last white ball (Ww):</strong> Can go in any of the remaining <strong>r positions</strong></li></ul><p>This gives us: <strong>(w + r — 1) × (w + r — 2) × … × r</strong> ways to place the white balls</p><p>We can write this as: <strong>(w + r — 1)! / (r — 1)!</strong></p><p>After placing all white balls, we still have <strong>(r — 1) red balls</strong> and last red ball that we left to arrange in the remaining positions, which can be done in <strong>r!</strong> ways.</p><p><strong>Total favorable arrangements = (w + r — 1)! × r! / (r — 1)!</strong></p><p><strong>Counting All Possible Arrangements</strong></p><p>Without any conditions, all (w + r) balls can be arranged in <strong>(w + r)!</strong> ways.</p><p><strong>Calculating the Probability</strong></p><pre>P(white runs out first) = Favorable arrangements / Total arrangements<br>                        = [(w + r - 1)! × r!] / [(r - 1)! × (w +r)!]<br>                        = r / (w + r)</pre><p><strong>Method 2: The Intuitive Approach (Much Simpler!)</strong></p><p>When you have r red balls and w white balls initially, imagine you’ve drawn all r + w balls in some order. The last ball could be any of the original balls with equal probability.</p><ul><li>Probability the last ball is <strong>red</strong> = r/(r+w)</li><li>Probability the last ball is <strong>white</strong> = w/(r+w)</li></ul><p><strong>If the last ball is red, that means white ran out first!</strong></p><p>So the answer is simply: <strong>r/(w+r)</strong></p><p><strong>Why This Works: </strong>The elegant insight is that by symmetry, each ball has an equal chance of being the last ball drawn. Since there are r red balls out of (r+w) total balls, the probability that the last ball is red is r/(r+w). And if the last ball is red, all white balls must have been drawn first!</p><blockquote>35. Two Sigma: What is the expected number of coin flips needed to get two consecutive heads?</blockquote><p><strong>State Definition</strong></p><ul><li><strong>E₀</strong>: Expected flips from start (or after tail)</li><li><strong>E₁</strong>: Expected flips after getting one head</li><li><strong>E₂</strong>: Two consecutive heads achieved = 0</li></ul><p><strong>State Equations</strong></p><p><strong>From E₀:</strong></p><pre>E₀ = 1 + 0.5·E₁ + 0.5·E₀</pre><ul><li>Flip once (cost = 1)</li><li>Get H (prob 0.5) → move to E₁</li><li>Get T (prob 0.5) → stay at E₀</li></ul><p><strong>From E₁:</strong></p><pre>E₁ = 1 + 0.5·E₂ + 0.5·E₀</pre><ul><li>Flip once (cost = 1)</li><li>Get H (prob 0.5) → done (E₂ = 0)</li><li>Get T (prob 0.5) → reset to E₀</li></ul><p><strong>Solution</strong></p><p>From E₁ equation:</p><pre>E₁ = 1 + 0.5(0) + 0.5·E₀ = 1 + 0.5·E₀</pre><p>Substitute into E₀ equation:</p><pre>E₀ = 1 + 0.5(1 + 0.5·E₀) + 0.5·E₀<br>E₀ = 1 + 0.5 + 0.25·E₀ + 0.5·E₀<br>E₀ = 1.5 + 0.75·E₀<br>0.25·E₀ = 1.5</pre><p><strong>E₀ = 6 flips</strong></p><p><strong>Key Insight: </strong>After getting one head, a tail forces complete restart — this significantly increases expected flips.</p><blockquote>36. Akuna Capital: Say you’re rolling a fair six-sided die. What is the expected number of rolls until you roll two consecutive 5s?</blockquote><p><strong>State Definition</strong></p><ul><li><strong>E₀</strong>: Expected rolls from start (or after non-5)</li><li><strong>E₁</strong>: Expected rolls after getting one 5</li><li><strong>E₂</strong>: Two consecutive 5s achieved = 0</li></ul><p><strong>State Equations</strong></p><p><strong>From E₀:</strong></p><pre>E₀ = 1 + (1/6)·E₁ + (5/6)·E₀</pre><ul><li>Roll once (cost = 1)</li><li>Get 5 (prob 1/6) → move to E₁</li><li>Get non-5 (prob 5/6) → stay at E₀</li></ul><p><strong>From E₁:</strong></p><pre>E₁ = 1 + (1/6)·E₂ + (5/6)·E₀</pre><ul><li>Roll once (cost = 1)</li><li>Get 5 (prob 1/6) → done (E₂ = 0)</li><li>Get non-5 (prob 5/6) → reset to E₀</li></ul><p><strong>Solution</strong></p><p>From E₁ equation:</p><pre>E₁ = 1 + (1/6)(0) + (5/6)·E₀ = 1 + (5/6)·E₀</pre><p>Substitute into E₀ equation:</p><pre>E₀ = 1 + (1/6)(1 + (5/6)·E₀) + (5/6)·E₀<br>E₀ = 1 + 1/6 + (5/36)·E₀ + (30/36)·E₀<br>E₀ = 7/6 + (35/36)·E₀<br>(1/36)·E₀ = 7/6</pre><p><strong>E₀ = 42 rolls</strong></p><p><strong>General Formula</strong></p><p>For two consecutive successes with probability p:</p><pre>Expected rolls = (1 + p)/p²</pre><ul><li>Fair coin (p = 1/2): (1.5)/(0.25) = 6</li><li>Die rolling 5 (p = 1/6): (7/6)/(1/36) = 42</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbe709f3ea46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DoorDash Delivery Time Prediction]]></title>
            <link>https://medium.com/@mdmoseena22/doordash-delivery-time-prediction-fc633580e9f8?source=rss-72a4fb8cbdaf------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc633580e9f8</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learnig]]></category>
            <category><![CDATA[modeling]]></category>
            <category><![CDATA[eda]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <dc:creator><![CDATA[MOHAMMED MOSEENA]]></dc:creator>
            <pubDate>Sat, 25 Oct 2025 12:34:11 GMT</pubDate>
            <atom:updated>2025-12-07T01:40:27.615Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>📌 <strong>Key Takeaways</strong></blockquote><ul><li>Built <strong>XGBoost model</strong> with <strong>11.04-minute MAE</strong> (662.43 seconds)</li><li>Achieved <strong>57.39% accuracy</strong> within ±10 minutes</li><li>Met <strong>&lt;5% extreme error guardrail</strong> (4.71%)</li><li>Identified <strong>marketplace load</strong> as #1 predictor via SHAP</li><li><strong>Production-ready</strong> with statistical validation (McNemar’s test)</li></ul><blockquote><strong>Article Roadmap:</strong></blockquote><ul><li>Problem Statement</li><li>Business Understanding</li><li>Establish Metrics</li><li>Understand your Data</li><li>EDA</li><li>Data Cleaning</li><li>Model Development (5 iterations)</li><li>Feature Engineering</li><li>Hyperparameter tuning</li><li>Model Explainability (SHAP)</li><li>Statistical Validation (McNemar’s Test)</li><li>Model Saving &amp; Inferencing</li><li>Results &amp; Business Impact</li><li>Key Insights &amp; Recommendations</li></ul><blockquote><strong>Problem Statement:</strong></blockquote><p>DoorDash wants to accurately predict <strong>delivery duration (in seconds)</strong> — the total time between when an order is created (created_at) and when it is delivered (actual_delivery_time).</p><p><strong>Target Variable:</strong></p><pre>delivery_duration_seconds = actual_delivery_time - created_at</pre><p><strong>Why Machine Learning?</strong></p><ul><li><strong>Marketplace dynamics are non-linear &amp; complex</strong> → rule-based logic fails</li><li>Need <strong>real-time predictions</strong> (&lt;200 ms latency)</li><li>No personal/financial/health data → <strong>ethically safe</strong> to apply ML</li></ul><blockquote><strong>Business Understanding</strong></blockquote><p><strong>End Users &amp; Benefits</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*fT74d9WGroNHvzK0pCYuyw.png" /></figure><p><strong>Business Value</strong></p><p>✅ Higher customer satisfaction &amp; retention<br> ✅ Fewer refunds / escalations<br> ✅ Lower operational waste (better staffing &amp; routing)<br> ✅ Competitive advantage in the food-delivery market</p><p><strong>Impact of Wrong Predictions</strong></p><ul><li><strong>Over-predict</strong> (ETA too slow) → user abandons order → lost revenue</li><li><strong>Under-predict</strong> (ETA too fast) → frustration, bad reviews</li><li><strong>Large errors (&gt;30 mins)</strong> → breaks customer trust</li></ul><p><strong>Key Insight:</strong> ETA systems prioritize <strong>accuracy over interpretability</strong> — wrong ETAs hurt reputation &amp; retention.</p><blockquote><strong>Establish Metrics</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*BmjvSvQrrJbketr_Yd2xbw.png" /></figure><blockquote><strong>Understand your Data</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*4H4mUAFuLUqlVg6-ge_xIg.png" /></figure><p><strong>Column Groups</strong></p><p><strong>1. Time Columns</strong></p><ul><li>market_id → City / region</li><li>created_at → Order timestamp (UTC)</li><li>actual_delivery_time → Delivered timestamp (<em>used only to create target; not available at prediction</em>)</li></ul><p><strong>2. Store Columns</strong></p><ul><li>store_id → Restaurant ID</li><li>store_primary_category → Cuisine type</li><li>order_protocol → Order integration method (integer ID)</li></ul><p><strong>3. Order Columns</strong></p><ul><li>total_items → Total items</li><li>num_distinct_items → Unique items</li><li>subtotal → Order value (cents)</li><li>min_item_price, max_item_price → Lowest &amp; highest item price (cents)</li></ul><p><strong>4. Marketplace Columns</strong></p><ul><li>total_onshift_dashers → Dashers available nearby</li><li>total_busy_dashers → Dashers currently busy</li><li>total_outstanding_orders → Active nearby orders</li></ul><p><strong>5. Existing Model Predictions as Columns</strong></p><ul><li>estimated_order_place_duration → Time for restaurant to receive order from DoorDash platform</li><li>estimated_store_to_consumer_driving_duration → Expected travel time</li></ul><p><strong>Note:</strong> These ML features come from DoorDash’s other internal models. Our model learns to combine these + marketplace signals to predict final ETA.</p><blockquote><strong>EDA</strong></blockquote><p><strong>Step 1: Load &amp; Preview Data</strong></p><ul><li><strong>197,428 records</strong> loaded</li><li>No duplicates found ✅</li></ul><p><strong>Step 2: Create Target Variable</strong></p><pre>df[&#39;created_at&#39;] = pd.to_datetime(df[&#39;created_at&#39;])<br>df[&#39;actual_delivery_time&#39;] = pd.to_datetime(df[&#39;actual_delivery_time&#39;])<br>df[&#39;delivery_duration_seconds&#39;] = (df[&#39;actual_delivery_time&#39;] - df[&#39;created_at&#39;]).dt.total_seconds()</pre><p><strong>Step 3: Data Quality Checks</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*QRben0UscFyItBsCv-I4_Q.png" /></figure><p><strong>Critical Insight:</strong> The “inconsistency” where total_busy_dashers &gt; total_onshift_dashers is NOT an error — it represents:</p><ul><li>Different data pipelines updating at different times</li><li>Demand surges</li><li>Dasher shortages</li></ul><p>→ These are <strong>operational signals</strong> that improve model accuracy, not noise to remove.</p><p><strong>Step 4: Outlier Analysis</strong></p><p><strong>Target Variable (delivery_duration_seconds)</strong></p><ul><li>Delivery duration is naturally <strong>right-skewed</strong></li><li>Some long delays are <strong>real operational events</strong> → keep them</li><li>Very large values (e.g., <strong>98 days, 4 days</strong>) are <strong>timestamp errors</strong> → remove</li></ul><p><strong>Approach:</strong></p><p>❌ Remove only impossible values: delivery_duration_seconds &gt; 12 hours (43,200 sec) → Only 7 rows removed.<br> ✅ Keep all realistic long delays (≤ 12 hours) because they capture real marketplace conditions</p><p><strong>Reasoning:</strong></p><p>Removing &gt; 99th percentile would delete valid delays (traffic, dasher shortage, store backlog).</p><p>Removing &gt;12hrs cleans only corrupted data without losing signal.</p><p><strong>Feature Outliers</strong></p><ul><li>total_items = 411 → 1 extreme record → <strong>Removed</strong></li><li>min_item_price = 14700, max_item_price = 14700 → <strong>Removed</strong></li></ul><p><strong>Step 5: Distribution Insights</strong></p><p><strong>Numeric Features:</strong></p><ul><li>Most numeric features are <strong>right-skewed</strong></li><li>estimated_store_to_consumer_driving_duration is the only feature with <strong>near-normal distribution</strong></li><li>All numeric columns have outliers</li></ul><p><strong>Key Insight:</strong> Log/scale transformations NOT needed for tree models (only useful for linear/SVM models).</p><p><strong>Step 6: Correlation Analysis</strong></p><p><strong>Strong Correlations:</strong></p><ul><li>Order size features (total_items, subtotal, num_distinct_items) are correlated → Tree models handle this naturally</li><li>Marketplace features (total_onshift_dashers, total_busy_dashers, total_outstanding_orders) are very highly correlated → multicollinearity, but tree models handle this naturally</li></ul><p><strong>Strongest correlations with delivery duration:</strong></p><ul><li>estimated_store_to_consumer_driving_duration (0.23)</li><li>subtotal (0.21)</li></ul><p><strong>Overall:</strong> Low correlations (&lt;0.30) indicate the problem is <strong>non-linear</strong>, supporting the use of <strong>Tree-Based Models</strong> over linear models.</p><p><strong>Step 7: Feature-Target Relationships</strong></p><p>Scatterplots of all numeric features vs delivery_duration_seconds show:</p><ul><li><strong>No linear pattern</strong></li><li>Outliers heavily distort linear models but tree models handle them well</li><li>Complex, multi-factor behavior confirms linear regression would have high bias</li></ul><p>✔ <strong>Model Recommendation:</strong> Start directly with tree-based models (DecisionTree, RandomForest, XGBoost)</p><blockquote><strong>Data Cleaning</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*zwozKPtr4eMwiX0ghZRpKA.png" /></figure><p><strong>Final Insight:</strong></p><ul><li>Dataset contains some structural issues → cleaned responsibly</li><li>Marketplace “inconsistencies” are intentionally kept because they carry real operational signal</li><li>Almost all features are skewed, non-linear, and contain outliers → <strong>Tree-based models are the right choice</strong></li></ul><p><strong>Data Retained:</strong> 99.3% (196,076 / 197,428 records)</p><blockquote><strong>Model Development Journey</strong></blockquote><p><strong>Train-Test Split Strategy → Temporal Split (80/20)</strong>:</p><ul><li>Training: First 80% chronologically</li><li>Testing: Most recent 20%</li></ul><p><strong>Why temporal split?</strong><br>Although the target does not exhibit autocorrelation, the input features capture real-time marketplace conditions (busy dashers, outstanding orders, onshift dashers). These conditions fluctuate across hours, days, and demand cycles, so mixing future rows into the training set would leak information the model would not have at inference time.</p><p>To correctly simulate real-world prediction — where the model must predict future deliveries using only past data — we use a temporal train/test split and TimeSeriesSplit cross-validation, even though the target itself is not autocorrelated.</p><p><strong>Model 1: Drop Missing Rows (Baseline)</strong></p><p><strong>Approach:</strong></p><ul><li>Remove all rows containing missing values</li><li>Label encode categorical columns</li><li>Train RandomForestRegressor</li></ul><p><strong>Results:</strong></p><ul><li><strong>MAE:</strong> 705.60 seconds (~11.76 minutes)</li><li><strong>R²:</strong> 0.223</li><li><strong>Accuracy ≤10 min:</strong> 54.02%</li><li><strong>Extreme errors &gt;30 min:</strong> 5.43%</li></ul><p><strong>Pros:</strong> Cleanest data, no imputation noise<br><strong>Cons:</strong> Loses ~10% of data</p><p><strong>Model 2: Simple Imputation</strong></p><p><strong>Approach:</strong></p><ul><li>Numeric missing → median</li><li>market_id &amp; order_protocol → mode then convert to int</li><li>store_primary_category → &quot;Unknown&quot;</li><li>Label encode categorical features</li><li>Train RandomForestRegressor</li></ul><p><strong>Results:</strong></p><ul><li><strong>MAE:</strong> 714.56 seconds (~11.91 minutes)</li><li><strong>R²:</strong> 0.227</li><li><strong>Accuracy ≤10 min:</strong> 53.26%</li><li><strong>Extreme errors &gt;30 min:</strong> 5.54%</li></ul><p><strong>Pros:</strong> Keeps all data, fast &amp; practical<br><strong>Cons:</strong> Adds small noise; not optimal</p><p><strong>Model 3: XGBoost (Native Missing Handling)</strong></p><p><strong>Approach:</strong></p><ul><li>No imputation needed for numeric features</li><li>Only store_primary_category is encoded (string → int)</li><li>XGBoost handles missing numeric IDs (market_id/order_protocol) automatically</li></ul><p><strong>Results:</strong></p><ul><li><strong>MAE:</strong> 684.43 seconds (~11.41 minutes)</li><li><strong>R²:</strong> 0.274</li><li><strong>Accuracy ≤10 min:</strong> 55.83%</li><li><strong>Extreme errors &gt;30 min:</strong> 5.05%</li></ul><p><strong>Pros:</strong> Best for real marketplace data; robust<br><strong>Cons:</strong> Slightly more complex model</p><p>✅ <strong>Model 3 becomes the new baseline</strong></p><blockquote><strong>Advanced Feature Engineering</strong></blockquote><p><strong>Model 4: XGBoost + Temporal Features</strong></p><p><strong>What was added?</strong></p><p>We extracted time-based patterns from created_at, because delivery duration strongly depends on:</p><ul><li><strong>Hour of day</strong> (rush hours vs late night)</li><li><strong>Day of week</strong> (weekends vs weekdays)</li></ul><p><strong>Features Engineered:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*51df08PY7GcgCw5FF8ldUg.png" /></figure><p><strong>Why Cyclic Encoding?</strong></p><p>To preserve continuity: <strong>11 PM (23) is closer to 12 AM (0) than to 3 PM (15)</strong></p><pre>df[&#39;hour_sin&#39;] = np.sin(2 * np.pi * df[&#39;hour&#39;] / 24)<br>df[&#39;hour_cos&#39;] = np.cos(2 * np.pi * df[&#39;hour&#39;] / 24)<br>df[&#39;dayofweek_sin&#39;] = np.sin(2 * np.pi * df[&#39;dayofweek&#39;] / 7)<br>df[&#39;dayofweek_cos&#39;] = np.cos(2 * np.pi * df[&#39;dayofweek&#39;] / 7)</pre><p><strong>Results:</strong></p><ul><li><strong>MAE:</strong> 668.21 seconds (~11.14 minutes)</li><li><strong>R²:</strong> 0.296</li><li><strong>Accuracy ≤10 min:</strong> 57.24%</li><li><strong>Extreme errors &gt;30 min:</strong> 4.89%</li></ul><p><strong>Performance Improvement:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*CGjWjvAZhNc4zuDUFH4fJQ.png" /></figure><p><strong>Key Insight:</strong><br> ✔ Temporal patterns significantly improve predictions<br> ✔ Model 4 becomes the new best performer</p><blockquote><strong>Hyperparameter Tuning</strong></blockquote><p><strong>Model 5: XGBoost (Hyperparameter Tuning)</strong></p><p><strong>What was done?</strong></p><p>Applied <strong>RandomizedSearchCV</strong> + <strong>TimeSeriesSplit</strong> on Model 4 to find optimal hyperparameters:</p><ul><li>n_estimators</li><li>max_depth</li><li>learning_rate</li><li>subsample</li><li>colsample_bytree</li></ul><p>Note: TimeSeriesSplit does not need the created_at column. It simply splits based on the order of rows. Since we already sort the dataset by created_at before splitting, the row order becomes chronological, and TimeSeriesSplit automatically produces correct future-after-past folds. Therefore, our implementation correctly performs time-aware cross-validation.</p><p><strong>Best Hyperparameters Found:</strong></p><p>python</p><pre>{<br>  &#39;n_estimators&#39;: 300,<br>  &#39;max_depth&#39;: 5,<br>  &#39;learning_rate&#39;: 0.1,<br>  &#39;subsample&#39;: 0.9,<br>  &#39;colsample_bytree&#39;: 0.9<br>}</pre><p>These settings balance:</p><ul><li>Deeper understanding of interactions (max_depth=5)</li><li>Stronger ensemble (n_estimators=300)</li><li>Stable generalization (subsample=0.9, colsample_bytree=0.9)</li><li>Faster learning (learning_rate=0.1)</li></ul><p><strong>Results:</strong></p><ul><li><strong>MAE:</strong> 662.43 seconds (~11.04 minutes)</li><li><strong>R²:</strong> 0.307</li><li><strong>Accuracy ≤10 min:</strong> 57.39%</li><li><strong>Extreme errors &gt;30 min:</strong> 4.71%</li></ul><p><strong>Performance Improvement (vs Model 4):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*jSEAaZqUIBQQTdvbtzccxA.png" /></figure><p><strong>Final Insight: </strong>Hyperparameter tuning gives <strong>small but meaningful gains</strong>. This is now the <strong>best-performing model</strong> and the one marked for <strong>production deployment</strong>.</p><blockquote><strong>Model Explainability (SHAP)</strong></blockquote><p><strong>Why SHAP?</strong></p><p>Traditional feature importance from XGBoost is limited:</p><ul><li>Only measures “how much” a feature is used</li><li>No direction (↑ or ↓ impact unknown)</li><li>Ignores interactions &amp; non-linearities</li></ul><p><strong>SHAP fixes all these issues:</strong></p><p>✔ Shows global importance and local explanations<br> ✔ Shows direction (positive/negative impact)<br> ✔ Captures non-linearity + interactions<br> ✔ Provides per-feature graphs that operations can understand</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hos7qI-1VGUkPLnoh-Nj-Q.png" /></figure><p><strong>Top 5 Drivers of Delivery Duration</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*gV2YDNdvuk8lhMhqmwO3-g.png" /></figure><p><strong>SHAP Insights (from Beeswarm Plot)</strong></p><p><strong>Marketplace load has the strongest effect:</strong></p><ul><li>High total_outstanding_orders → Pushes prediction <strong>far above</strong> baseline</li><li>High total_onshift_dashers → SHAP values go <strong>strongly negative</strong> (faster delivery)</li></ul><p><strong>Confirms:</strong> <em>“Supply–demand imbalance is the #1 cause of delays.”</em></p><p><strong>Travel time patterns are linear &amp; intuitive:</strong></p><ul><li>High estimated_store_to_consumer_driving_duration → Strong positive SHAP values</li></ul><p><strong>Temporal patterns are strong:</strong></p><ul><li>hour_sin → peak hours (evening, lunch) push SHAP up</li><li>dayofweek_sin/cos → weekend patterns visible</li></ul><p><strong>Order size matters:</strong></p><ul><li>Higher subtotal → more prep time → higher SHAP values</li></ul><p><strong>Actionable for Operations:</strong></p><ul><li>Real-time alerts when total_outstanding_orders is high</li><li>Dynamic dasher incentives during predicted surge periods</li><li>Restaurant prep-time warnings for high-subtotal orders</li></ul><blockquote><strong>Statistical Validation (McNemar’s Test)</strong></blockquote><p><strong>Question: </strong>Is the improvement from Model 1 → Model 5 statistically significant?</p><p><strong>Test: McNemar’s Test</strong></p><p>McNemar’s test compares <strong>two models on the exact same dataset</strong> where each prediction is either <strong>correct or wrong</strong>.</p><p><strong>Key Idea:</strong> We only care about where the two models <strong>disagree</strong>:</p><ul><li><strong>b:</strong> Model 1 is correct, Model 5 is wrong</li><li><strong>c:</strong> Model 1 is wrong, Model 5 is correct</li></ul><p>If <strong>c &gt;&gt; b</strong> → Model 5 fixes many of Model 1’s mistakes → <strong>Model 5 is significantly better</strong></p><p><strong>Result:</strong></p><ul><li><strong>p-value = 2×10⁻⁶ (&lt; 0.05)</strong></li></ul><p><strong>Interpretation:</strong></p><p>✅ <strong>The improvement is statistically significant.</strong></p><p>Model 5 correctly predicted significantly more orders within the 10-minute window than Model 1. The performance gain is <strong>real, not due to random chance.</strong></p><blockquote><strong>Model Saving &amp; Inferencing</strong></blockquote><p>Model Saving:</p><ul><li>Stores the final tuned XGBoost model</li><li>Saves the LabelEncoder used for store_primary_category</li><li>Saves feature column order, ensuring consistent inference</li><li>Bundles everything into one joblib file for easy loading in production</li></ul><p>Model Inferencing:</p><ol><li>Loads the saved model + encoders</li><li>Applies the same preprocessing steps used during training:</li></ol><ul><li>datetime extraction</li><li>temporal features</li><li>cyclic encoding</li><li>store category encoding</li></ul><p>3. Ensures features are in the exact same order as during training</p><p>4. Runs model prediction and outputs ETA in seconds, minutes, and hours</p><blockquote><strong>Results &amp; Business Impact</strong></blockquote><p><strong>Final Model Performance</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*qKrXZi522cqKnpMa7BqIGQ.png" /></figure><p><strong>Model Comparison Summary</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*8YvjKb93jps6CRIKq5zTzw.png" /></figure><blockquote><strong>Key Insights &amp; Recommendations</strong></blockquote><p><strong>Technical Insights:</strong></p><ol><li><strong>XGBoost outperforms RandomForest</strong> due to native missing value handling and gradient boosting</li><li><strong>Temporal patterns are critical</strong> — hour/day features added significant predictive power</li><li><strong>Marketplace features dominate</strong> — supply-demand balance is more important than order size</li><li><strong>Non-linear relationships</strong> — scatter plots confirm tree-based models are the right choice</li></ol><p><strong>Business Recommendations:</strong></p><p>1. Deploy Model 5 to Production</p><ul><li>Statistically validated improvement</li><li>Production-ready with proper artifacts saved</li></ul><p>2. Monitor Drift</p><ul><li>Retrain periodically as marketplace dynamics change (seasonality, new markets)</li></ul><p>3. Build Real-Time Dashboards</p><ul><li>Use SHAP insights for operations teams</li><li>Alert when total_outstanding_orders exceeds thresholds</li></ul><p>4. A/B Test in Production</p><ul><li>Compare Model 5 ETAs against current system</li><li>Measure impact on customer satisfaction metrics</li></ul><p>5. Collect Feedback Loops</p><ul><li>Track actual vs predicted to continuously improve</li></ul><p>Future Enhancements:</p><ul><li>Add <strong>weather data</strong> (rain, snow → slower deliveries)</li><li>Include <strong>traffic API</strong> for real-time congestion</li><li>Build <strong>market-specific models</strong> (different cities have different patterns)</li><li>Experiment with <strong>deep learning</strong> (LSTM for time-series patterns)</li></ul><blockquote><strong>Conclusion</strong></blockquote><p>This project successfully built an <strong>XGBoost-based delivery duration prediction system</strong> that:</p><ul><li>Reduces average ETA error to <strong>~11 minutes</strong></li><li>Accurately predicts <strong>57% of deliveries within ±10 minutes</strong></li><li>Keeps extreme errors below <strong>5%</strong></li><li>Uses <strong>interpretable SHAP analysis</strong> for operational decision-making</li><li>Handles <strong>real-world data challenges</strong> (missing values, outliers, marketplace noise)</li></ul><p><strong>The model is statistically validated, production-ready, and provides actionable insights for DoorDash operations teams.</strong></p><blockquote><strong>📂 GitHub Repository</strong></blockquote><p>The complete code implementation is available on GitHub:<br><a href="https://github.com/MohammedMoseena/doordash-delivery-time-prediction"><strong>https://github.com/MohammedMoseena/doordash-delivery-time-prediction</strong></a></p><p><strong>🙏 Thank You!</strong></p><p>If you found this helpful, please <strong>clap 👏</strong> and <strong>follow</strong> for more end-to-end ML projects.</p><p><strong>Connect with me:</strong><br>LinkedIn: <a href="https://www.linkedin.com/in/mdmoseena/">https://www.linkedin.com/in/mdmoseena/</a></p><p><em>Last updated: November 2025</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc633580e9f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SQL Interview Prep: Complete Quick Reference for Data Science Interviews]]></title>
            <link>https://medium.com/@mdmoseena22/sql-db-design-9f2c89699d07?source=rss-72a4fb8cbdaf------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f2c89699d07</guid>
            <category><![CDATA[mapreduce]]></category>
            <category><![CDATA[non-relational-database]]></category>
            <category><![CDATA[relational-databases]]></category>
            <category><![CDATA[interview-tips]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[MOHAMMED MOSEENA]]></dc:creator>
            <pubDate>Tue, 07 Oct 2025 04:21:28 GMT</pubDate>
            <atom:updated>2025-12-29T09:21:45.776Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>SQL Tips for Interviews:</strong></p><ol><li>Develop ability to convert reporting requirements into sql.</li><li><strong>Clarify terms</strong>: Always confirm definitions (e.g., churn, retention) before writing queries.</li><li><strong>Restate the question</strong>: Repeat your understanding to the interviewer to avoid misinterpretation.</li><li><strong>Work backwards</strong>: Imagine the final result as a single table; then figure out joins/CTEs step by step.</li><li><strong>Stay structured</strong>: Break down complex queries instead of writing everything at once.</li></ol><h4><strong>Basic SQL Commands</strong></h4><p><strong>Must-know commands</strong>:<br>CREATE TABLE, INSERT, UPDATE, DELETE, SELECT, GROUP BY, WHERE, HAVING, ORDER BY, DISTINCT, UNION.</p><p><strong>Key Differences</strong></p><p><strong>DELETE vs TRUNCATE vs DROP</strong></p><ul><li>DELETE: Removes rows one by one, table remains.</li><li>TRUNCATE: Removes all rows at once, table remains (DDL).</li><li>DROP: Deletes the table structure itself.</li></ul><p><strong>WHERE vs HAVING</strong></p><p>Similarities: Both filter data, optional, interchangeable in some cases.</p><p>Differences:</p><ul><li>WHERE: Filters rows <em>before</em> grouping.</li><li>HAVING: Filters groups <em>after</em> aggregation.</li><li>Aggregates allowed only in HAVING.</li></ul><p><strong>Types of SQL Commands</strong></p><ul><li><strong>DDL</strong> (Data Definition): Effects table structure — &gt;CREATE, ALTER, DROP, TRUNCATE.</li><li><strong>DML</strong> (Data Manipulation): Manipulates data in table — &gt;INSERT, UPDATE, DELETE.</li><li><strong>DQL</strong> (Data Query): SELECT.</li><li><strong>DCL</strong> (Data Control): GRANT, REVOKE.</li><li><strong>TCL</strong> (Transaction Control): COMMIT, ROLLBACK.</li></ul><h4>JOINS</h4><p><strong>Main Types of Joins</strong></p><ul><li><strong>INNER JOIN</strong> → Returns only matching rows.</li><li><strong>LEFT JOIN</strong> → All rows from left table (also includes matching from right)</li><li><strong>RIGHT JOIN</strong> → All rows from right table (also includes matching from left)</li><li><strong>FULL OUTER JOIN</strong> → All rows from both tables, matches where possible (not supported in MySQL → simulate with LEFT JOIN UNION RIGHT JOIN).</li><li><strong>SELF JOIN</strong> → Join a table with itself (e.g., employee-manager hierarchy).</li><li><strong>CROSS JOIN</strong> → Cartesian product of both tables.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*75kHIg7GRJQ3i51rdAgoRg.png" /></figure><p><strong>Performance Tips</strong></p><p><strong>Joins =&gt; expensive, so optimize:</strong></p><ul><li>Select specific fields (avoid SELECT *).</li><li>Filter/aggregate before joining.</li><li>Use LIMIT when exploring.</li><li>Minimize number of joins in a single query.</li></ul><h4>Advance SQL Commands</h4><p><strong>1. Aggregation Functions</strong></p><ul><li>COUNT, SUM, AVG, MAX, MIN</li><li>Formatting &amp; handling nulls: ROUND, CONCAT, IFNULL, NULLIF, COALESCE, SUBSTRING</li><li>Date functions: DATE_FORMAT, DATE_ADD, DATE_SUB, DATEDIFF, TIMESTAMPDIFF, EXTRACT</li></ul><p><strong>2. Filtering Methods</strong></p><ul><li>Comparisons: =, &lt;&gt;, &gt;, &lt;</li><li>Logical: AND, OR, NOT</li></ul><p>Pattern matching:</p><ul><li><strong>LIKE</strong> → &#39;%a&#39;, &#39;a%&#39;, &#39;%a%&#39;, &#39;_a&#39;</li><li><strong>REGEX</strong> → [a-zA-Z0–9], ^, $, [^a-z], [0–9]{4}, +, ?, *</li></ul><h4>Common Table Expressions(CTE) and Subqueries</h4><ul><li><strong>CTE (Common Table Expression)</strong> → Break large queries into manageable parts using WITH. Can be <strong>recursive</strong>.</li><li><strong>Subquery</strong> → Query inside another query (inline), must use an alias if treated as a table.</li><li><strong>Key Difference</strong> → Both simplify complex queries, but <strong>CTEs support recursion</strong> while subqueries do not.</li></ul><p><strong>Recursive CTE</strong></p><p><strong>Real World Example: Calculate total sales per year</strong></p><p><strong>Input</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*h4WF93RBCQfGuNHUYxz__g.png" /></figure><p><strong>Output</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*092gxjvvPa0b4Fmwq78d_Q.png" /></figure><pre>WITH RECURSIVE dates AS (<br>  SELECT min(period_start) AS date, max(period_end) AS max_date<br>  FROM sales<br>  UNION<br>  SELECT DATE_ADD(min_date, INTERVAL 1 DAY), max(period_end) AS max_date<br>  FROM dates<br>  WHERE date &lt; max_date<br>)<br>SELECT <br>  s.product_id, <br>  YEAR(d.date) AS report_year, <br>  SUM(average_daily_sales) AS total_amount<br>FROM sales as s<br>INNER JOIN dates as d ON d.date BETWEEN s.period_start AND s.period_end<br>GROUP BY s.product_id, YEAR(d.date)<br>ORDER BY 1, 2</pre><h4><strong>Window Functions</strong></h4><p><strong>Definition</strong>: Perform calculations across a set of rows <strong>without collapsing them</strong> (unlike GROUP BY). Useful for mixing aggregated + non-aggregated values.</p><p><strong>Syntax</strong>:<br> &lt;function&gt;() OVER (PARTITION BY ... ORDER BY ... ROWS BETWEEN ...)</p><ul><li><strong>PARTITION BY</strong> → Splits data into groups (like GROUP BY).</li><li><strong>ORDER BY</strong> → Defines row order within each partition.</li><li><strong>ROWS BETWEEN</strong> → Defines frame (sliding window of rows).</li></ul><p><strong>Popular Window Functions</strong></p><ul><li>ROW_NUMBER() → Sequential row numbering.</li><li>RANK() → Ranking with gaps (1, 2, 2, 4 …).</li><li>DENSE_RANK() → Ranking without gaps (1, 2, 2, 3 …).</li><li>LAG() → Value from previous row.</li><li>LEAD() → Value from next row.</li><li>NTILE() → a <strong>window function</strong> that divides an ordered set of rows into a specified number of approximately equal groups, or “buckets”, and assigns a bucket number to each row</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*tFnf39waXnLhMxlvBuTxGg.png" /></figure><p>RANK() vsDENSE_RANK()</p><ul><li><strong>Gaps are meaningful</strong> (RANK): Competitions, selections with limited spots, rankings where “out of X” matters</li><li><strong>Gaps are confusing</strong> (DENSE_RANK): Tiers, categories, levels, Dashboards, groupings where you want consecutive numbering</li></ul><p><strong>Tip</strong>: Prefer window functions over GROUP BY + JOIN for cleaner, easier-to-maintain queries.</p><h4>SQL Execution Order</h4><ol><li><strong>FROM</strong> → Identify input tables.</li><li><strong>JOIN … ON</strong> → Merge tables.</li><li><strong>WHERE</strong> → Filter rows (before grouping).</li><li><strong>GROUP BY</strong> → Group rows.</li><li><strong>HAVING</strong> → Filter groups (after aggregation).</li><li><strong>SELECT</strong> → Choose columns/expressions.</li><li><strong>Aggregations</strong> → Apply functions like SUM(), AVG().</li><li><strong>ORDER BY</strong> → Sort rows.</li><li><strong>LIMIT</strong> → Restrict final output rows.</li></ol><h4>Views</h4><p><strong>Definition</strong>: A view is a <em>virtual table</em> defined by a query. It doesn’t store data physically — only the query. Data is fetched dynamically from underlying tables. Changes in base tables reflect in the view automatically.</p><p><strong>Advantages</strong></p><ul><li>Useful for simplifying complex queries and abstracting logic.</li><li>Provide limited data exposure → enhance security.</li><li>No extra storage overhead since view does not store actual data.</li></ul><p><strong>Updatable Views</strong></p><ul><li>You <strong>can INSERT, UPDATE, DELETE</strong> through the view.</li><li>Any change in the view <strong>directly modifies the base table</strong>.</li><li>Works only when the view is simple: From <strong>one table, </strong>No JOIN, GROUP BY, DISTINCT, UNION, aggregates, subqueries</li></ul><pre>-- Base table<br>CREATE TABLE employees (<br>    id INT PRIMARY KEY,<br>    name VARCHAR(50),<br>    salary INT<br>);<br><br>-- View<br>CREATE VIEW employee_salaries AS<br>SELECT id, name, salary<br>FROM employees;<br><br>-- Update View<br>UPDATE employee_salaries<br>SET salary = 90000<br>WHERE id = 1;</pre><p><strong>Non-updatable Views</strong></p><ul><li>Cannot perform INSERT/UPDATE/DELETE.</li><li>Includes views with: JOINs, GROUP BY, Aggregates (SUM, COUNT, etc.), DISTINCT, UNION</li><li>MySQL will throw an error:<br> <strong>“The target table is not updatable.”</strong></li></ul><p><strong>WHERE Clause in View</strong></p><ul><li>If the view filters data (WHERE), inserts may fail if inserted rows do not satisfy the condition.</li></ul><p>Notes:</p><ul><li>You can Insert/Update through a view only if it is updatable</li><li>MySQL allows updatable views only when they are simple SELECTs on one table</li><li>Views with JOIN, GROUP BY, aggregates, DISTINCT, etc. are not updatable.</li><li>The base table is always the one that actually gets modified</li></ul><h4>Databases and Systems</h4><ol><li><strong>Keys &amp; Normalization</strong></li></ol><p><strong>Primary Key</strong><br>- Uniquely identifies each row in a table<br>- Can be composite (multiple columns)<br>- Cannot be NULL<br><strong>Good Primary Key traits:</strong><br>- <strong>Stability</strong>: Never changes over time<br>- <strong>Uniqueness</strong>: No duplicate values<br>- <strong>Irreducibility</strong>: No subset is itself a primary key</p><p><strong>Foreign Key</strong><br>- Links related tables together<br>- Ensures data is stored only once (reduces redundancy)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rbYDN5xF67BBbcFC5fOZ2w.jpeg" /></figure><p><strong>Normalization</strong><br>- Splits data into multiple tables<br>- <strong>Benefits</strong>: Reduces redundancy, enforces constraints, improves data integrity<br>- <strong>Drawback</strong>: Requires JOINs (expensive operations, slower performance)</p><p><strong>Denormalization</strong><br>- Combines data into fewer tables<br>- Use when: Performance &gt; Storage</p><p><strong>OLAP Vs OLTP</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wDJsrNU3n6e33c6gL1EiHA.png" /></figure><p><strong>Interview Tip:</strong></p><p>When normalization comes up in interviews, it often concerns the conceptual setup of database tables: why a certain entity should have a foreign key to another entity, what the mapping relationship is between two types of records (one-to-one, one-to-many, or many-to-many), and when it might be advantageous to denormalize the database.</p><h4><strong>Properties of Distributed Databases</strong></h4><ol><li><strong>CAP FRAMEWORK (Distributed Databases)</strong></li></ol><ul><li><strong>Consistency</strong>: All clients see the same data at the same time</li><li><strong>Availability</strong>: System always responds (no errors), but may show outdated data</li><li><strong>Partition Tolerance</strong>: System works even if communication between nodes fails</li></ul><p><strong>KEY INSIGHT</strong>: In distributed systems, Partition Tolerance is mandatory. You must choose between C and A when partitions occur:</p><ul><li><strong>CP (Consistency + Partition Tolerance)</strong>: Ensures correctness, may be unavailable. Example: Payment apps → WhatsApp Payment, IRCTC ticket booking etc</li><li><strong>AP (Availability + Partition Tolerance)</strong>: Always available, may show stale data. Example: Social Media apps → Instagram feed</li></ul><p><strong>ACID FRAMEWORK (OLTP Systems)</strong></p><p>Critical for transaction processing with many concurrent users.</p><p><strong>Example</strong>: Bank transfer of ₹1000 from Account A to Account B</p><ul><li><strong>Atomicity</strong>: All or nothing. If debit succeeds but credit fails, entire transaction rolls back. No partial updates.</li><li><strong>Consistency</strong>: Database rules are preserved. Before: A=₹10,000, B=₹5,000, Total=₹15,000. After: A=₹9,000, B=₹6,000, Total=₹15,000. No negative balances, constraints respected.</li><li><strong>Isolation</strong>: Concurrent transactions don’t interfere. Two simultaneous transactions (ATM withdrawal + PhonePe payment) from the same account are coordinated so one completes before the other starts, preventing incorrect balances.</li><li><strong>Durability</strong>: Once “Transaction Successful” shows, data persists even if server crashes immediately. Guaranteed via logs and disk writes.</li></ul><p><strong>WHEN TO USE:</strong></p><ol><li>ACID for OLTP (transactional systems)</li><li>CAP considerations for distributed/scalable systems.</li></ol><h4>Scaling Databases</h4><p><strong>Vertical Scaling (Scaling Up)</strong></p><ul><li>Add more CPU and RAM to existing machines</li><li>Pros: Less infrastructure management</li><li>Cons: More expensive, physical limits (can’t scale indefinitely)</li></ul><p><strong>Horizontal Scaling (Scaling Out)</strong></p><ul><li>Add more nodes/machines to resource pool</li><li>Pros: Less expensive than vertical scaling</li><li>Cons: Complex infrastructure management, node communication challenges, data consistency issues</li></ul><p><strong>SHARDING (Horizontal Scaling Method)</strong></p><ul><li>Split large database into smaller, faster parts called shards</li><li>Each shard contains only a portion of total data</li><li>All shards together = complete database</li><li>Each shard is independent, typically on different servers</li></ul><p><strong>Example</strong>: Web app with millions of users in users table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/1*ZlztEc3XNe8gXTPtKJ7kDg.png" /></figure><p><strong>Benefits</strong>: Each server handles only its portion, improving performance and enabling scalability.</p><p><strong>KEY TAKEAWAY</strong>: Vertical scaling is simpler but has limits. Horizontal scaling (especially with sharding) enables massive scale but requires careful data distribution and consistency management.</p><h4>Relational Databases Vs NoSQL Databases</h4><p><strong>Relational Database</strong>: Table-based with fixed, pre-defined schema</p><p><strong>NoSQL</strong>: Stores data in variety of forms (graphs, documents, key-value, etc.) Examples: Neo4j (graph), MongoDB (document)</p><p><strong>BASE CONSISTENCY MODEL (NoSQL Framework)</strong></p><ul><li><strong>Basically Available</strong>: Data always available, system always responds. May return inconsistent/inaccurate data due to distributed nature.</li><li><strong>Soft State</strong>: System state changes over time, even without input. Passive changes due to eventual consistency.</li><li><strong>Eventual Consistency</strong>: Data will converge to consistent state eventually, but no guarantees on when.</li></ul><p><strong>KEY DIFFERENCE</strong>: ACID (transactional DBs like banks) prioritizes Consistency over Availability. BASE (NoSQL like Instagram/Facebook) prioritizes Availability over Consistency.</p><h4>MAP Reduce</h4><p>Popular framework for <strong>concurrent processing of large data volumes across distributed systems.</strong></p><p>Four Main Steps:</p><ol><li><strong>Split</strong>: Divide input data and distribute across different nodes</li><li><strong>Map</strong>: Transform input data into &lt;key, value&gt; pairs</li><li><strong>Shuffle</strong>: Move all &lt;key, value&gt; pairs with same key to same node (KEY EFFICIENCY STEP — reduces node communication)</li><li><strong>Reduce</strong>: Process and aggregate &lt;key, value&gt; pairs into final output</li></ol><p><strong>KEY INSIGHT</strong>: Shuffle step is the secret to MapReduce efficiency by minimizing node-to-node communication.</p><p><strong>Example</strong>: Count the frequency of words in multi-petabyte corpus of text data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VmKOAPZFr6kQu5r6KNuXAQ.jpeg" /></figure><h4><strong>HADOOP vs SPARK</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*vqCaDjm3-soIrLyh5n_F_w.png" /></figure><h4><strong>Interview Questions</strong></h4><ol><li>Facebook: Assume you have the below events table on app analytics. Write a query to get the click through rate per app in 2019</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JdILoBU76V1OJuYyZ6g5JA.jpeg" /></figure><pre>WITH app_details AS (<br>  SELECT <br>    app_id,<br>    SUM(IF(event_id = &#39;click&#39;, 1, 0)) AS num_of_clicks,<br>    SUM(IF(event_id = &#39;impression&#39;, 1, 0)) AS num_of_impressions<br>  FROM events<br>  WHERE timestamp &gt;= &#39;2019-01-01 00:00:00&#39;<br>    AND timestamp &lt; &#39;2020-01-01 00:00:00&#39;<br>  GROUP BY app_id<br>)<br>SELECT<br>  app_id,<br>  ROUND(num_of_clicks*100/NULLIF(num_of_impressions, 0), 2)<br>    AS click_through_rate<br>FROM app_details<br>ORDER BY 1;</pre><p>2. Robinhood: Assume you are given the tables below containing information on trades and users. Write a query to list the top three cities that had the most number of completed orders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q6gTwXx-9UJuZyaMwdXdeQ.jpeg" /></figure><pre>WITH completed_orders AS (<br>  SELECT <br>    user_id,<br>    order_id<br>  FROM trades<br>  WHERE status = &quot;complete&quot;<br>)<br>SELECT<br>  u.city,<br>  COUNT(order_id) AS num_complete_orders<br>FROM completed_orders as t<br>INNER JOIN users as u ON t.user_id = u.user_id<br>GROUP BY 1<br>ORDER BY 2 DESC<br>LIMIT 3;</pre><p>3. New York Times: Assume that you are given the table below containing information on viewership by device type (where the three types are laptop, tablet, and phone). Define “mobile” as the sum of tablet and phone viewership numbers. Write a query to compare the viewership on laptops versus mobile devices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IZVw7k4MiwoAJDUpXRWUAw.jpeg" /></figure><pre>SELECT<br>  SUM(IF(device_type IN (&#39;tablet&#39;, &#39;phone&#39;),1,0)) AS num_of_mobile_views,<br>  SUM(IF(device_type = &#39;laptop&#39;,1,0)) AS num_of_laptop_views,<br>FROM viewership;</pre><p>4. Amazon: Assume you are given the table below for spending activity by product type. Write a query to calculate the cumulative spend so far by date for each product over time in chronological order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YCbwJj7oSwwrXN18dsmHyA.jpeg" /></figure><pre>WITH product_spend_per_day AS (<br>  SELECT <br>    product_id,<br>    DATE(trans_date) AS trans_date,<br>    SUM(spend) AS total_spend<br>  FROM total_trans<br>  GROUP BY 1, 2<br>)<br>SELECT <br>  product_id,<br>  trans_date,<br>  total_spend,<br>  SUM(total_spend) OVER(PARTITION BY product_id ORDER BY trans_date)<br>    AS cumulative_spend<br>FROM product_spend_per_day<br>ORDER BY 1, 2;<br><br></pre><p>5. eBay: Assume that you are given the table below containing information on various orders made by customers. Write a query to obtain the names of the ten customers who have ordered the highest number of products among those customers who have spent at least $1000 total.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*osTy3Bliy6L9ekjHiow9Ag.jpeg" /></figure><pre>SELECT <br>  user_id,<br>  COUNT(product_id) AS num_of_products<br>FROM user_transactions<br>GROUP BY user_id<br>HAVING SUM(spend) &gt;= 1000<br>ORDER BY 2 DESC<br>LIMIT 10;</pre><p>6. Twitter. Assume you are given the table below containing information on tweets. Write a query to obtain a histogram of tweets posted per user in 2020.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Faoo5slTmAF-e7NGUM4IZg.jpeg" /></figure><pre>WITH user_tweets AS (<br>    SELECT <br>        user_id, <br>        COUNT(tweet_id) AS num_tweets<br>    FROM tweets<br>    WHERE tweet_date &gt;= &#39;2020-01-01 00:00:00&#39; <br>      AND tweet_date &lt; &#39;2021-01-01 00:00:00&#39;<br>    GROUP BY user_id<br>)<br>SELECT <br>    num_tweets, <br>    COUNT(user_id) AS num_users<br>FROM user_tweets<br>GROUP BY num_tweets;</pre><p>7. Stitch Fix: Assume you are given the table below containing information on user purchases. Write a query to obtain the number of people who purchased at least one or more of the same product on multiple days.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HDMWl7ZClOfx_6NsuK6RzA.jpeg" /></figure><pre>SELECT COUNT(DISTINCT user_id) AS num_users<br>FROM purchase<br>GROUP BY user_id, product_id<br>HAVING COUNT(DISTINCT DATE(purchase_time)) &gt; 1;</pre><p>8. Linkedin: Assume you are given the table below that shows the job postings for all companies on the platform. Write a query to get the total number of companies that have posted duplicate job listings (two jobs at the same company with the same title and description).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*CAAnH-wD_KyNUkrGCS3RQQ.jpeg" /></figure><pre>SELECT COUNT(DISTINCT company_id) AS num_duplicate_job_companies<br>FROM job_listings<br>GROUP BY company_id, title, description<br>HAVING COUNT(*) &gt; 1;<br></pre><p>9. Etsy: Assume you are given the table below on user transactions. Write a query to obtain the list of customers whose first transaction was valued at $50 or more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_a1CTedRloKxBowQQ9VQrg.jpeg" /></figure><pre>WITH user_data AS (<br>  SELECT <br>    user_id, <br>    spend, <br>    ROW_NUMBER() OVER(PARTITION BY user_id ORDER BY transaction_date) AS rn<br>  FROM user_transactions<br>)<br>SELECT<br>  user_id <br>FROM user_data<br>WHERE rn = 1 AND spend &gt;= 50;</pre><p>10. Twitter: Assume you are given the table below containing information on each user’s tweets over a period of time. Calculate the 7-day rolling average of tweets by each user for every date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*3CPRTNLrUYdPoiKoRn5JbA.jpeg" /></figure><pre>WITH tweet_freq AS (<br>  SELECT<br>    user_id,<br>    CAST(tweet_date AS date) AS tweet_date,<br>    COUNT(tweet_id) AS num_tweets<br>  FROM tweets<br>  GROUP BY 1, 2<br>)<br>SELECT *,<br>  AVG(num_tweets) OVER(<br>    PARTITION BY user_id <br>    ORDER BY tweet_date <br>    ROWS BETWEEN 6 PRECEDING AND CURRENT ROW<br>  ) AS rolling_average_7d<br>FROM tweet_freq;</pre><p>11. Uber: Assume you are given the table below on transactions made by users. Write a query to obtain the third transaction of every user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*snHGhEowRJn-6m3XC9HEOw.jpeg" /></figure><pre><br>SELECT *<br>FROM (<br>  SELECT *,<br>    ROW_NUMBER() OVER(PARTITION BY user_id ORDER BY transaction_date) AS rn<br>  FROM transactions<br>) AS t<br>WHERE t.rn = 3;</pre><p>12. Amazon: Assume you are given the table below containing information on customer spend on products belonging to various categories. Identify the top three highest-grossing items within each category in 2020.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*CVnNKCLKTfwbjLqAlg7qxA.jpeg" /></figure><pre>WITH spend_per_product AS (<br>    SELECT <br>        category, <br>        product_id, <br>        SUM(spend) AS total_spend<br>    FROM product_spend<br>    WHERE transaction_date &gt;= &quot;2020-01-01 00:00:00&quot; <br>      AND transaction_date &lt; &quot;2021-01-01 00:00:00&quot;<br>    GROUP BY category, product_id<br>)<br>SELECT <br>    category, <br>    product_id, <br>    total_spend<br>FROM (<br>    SELECT <br>        category,<br>        product_id,<br>        total_spend,<br>        RANK() OVER (<br>            PARTITION BY category <br>            ORDER BY total_spend DESC<br>        ) AS rn<br>    FROM spend_per_product<br>) AS t<br>WHERE rn &lt;= 3<br>ORDER BY category, total_spend DESC, product_id;</pre><p>13. Walmart: Assume you are given the below table on transactions from users. Bucketing users based on their latest transaction date, write a query to obtain the number of users who made a purchase and the total number of products bought for each transaction date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*XdcEcyjMDwpYMFpiBtF3Fg.jpeg" /></figure><pre>WITH latest_details AS (<br>    SELECT <br>        user_id,<br>        product_id,<br>        CAST(transaction_date AS DATE) AS transaction_date<br>        ROW_NUMBER() OVER(PARTITON BY user_id ORDER BY transaction_date DESC) AS rn<br>    FROM user_transactions<br>)<br>SELECT <br>    transaction_date,<br>    COUNT(DISTINCT user_id) AS num_of_users,<br>    COUNT(product_id) AS num_of_products<br>FROM latest_details<br>WHERE rn = 1<br>GROUP BY transaction_date<br>ORDER BY 1 DESC;<br><br></pre><p>14. Twitter: Assume you are given the two tables below containing information on the topics that each Twitter user follows and the ranks of each of these topics. Write a query to obtain all existing users on 2021–01–01 that did not follow any topic in the 100 most popular topics for that day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NzxpDV6qRV_0AhJDppXYsA.jpeg" /></figure><pre>WITH top_100 AS (<br>    SELECT topic_id<br>    FROM topic_rankings<br>    WHERE CAST(ranking_date AS DATE) = &#39;2021-01-01&#39;<br>      AND ranking &lt;= 100<br>)<br>SELECT DISTINCT user_id<br>FROM user_topics<br>WHERE CAST(follow_date AS DATE) &lt;= &#39;2021-01-01&#39;<br><br>MINUS<br><br>SELECT DISTINCT user_id<br>FROM user_topics<br>WHERE CAST(follow_date AS DATE) &lt;= &#39;2021-01-01&#39;<br>  AND topic_id IN (SELECT topic_id FROM top_100);</pre><p>15. Facebook: Assume you have the tables below containing information on user actions. Write a query to obtain active user retention by month. An active user is defined as someone who took an action (sign-in, like, or comments) in the current month.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iUegliSfB3RDPuVwNXI4VQ.jpeg" /></figure><pre>WITH monthly_active_users AS (<br>  SELECT<br>    DATE_FORMAT(timestamp, &#39;%Y-%m-01&#39;) AS month,<br>    user_id<br>  FROM user_actions<br>  GROUP BY 1, 2<br>  )<br>SELECT<br>  curr.month,<br>  COUNT(DISTINCT user_id) AS num_retained_users_per_month<br>FROM monthly_active_users AS curr<br>INNER JOIN monthly_active_users AS prev ON curr.user_id = prev.user_id AND<br>  curr.month = DATE_ADD(prev.month, INTERVAL 1 MONTH)<br>GROUP BY curr.month<br>ORDER BY 1 DESC<br></pre><p>16. Twitter: Assume you are given the tables below containing information on user session activity. Write a query that ranks users according to their total session durations for each session type between the start date (2021–01–01) and the end date (2021–02–01)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ja0SrI_3CaelYwtdy8OioA.jpeg" /></figure><pre>WITH total_session_details AS (<br>  SELECT user_id, session_type, SUM(duration) as total_duration<br>  FROM sessions<br>  WHERE CAST(start_time AS DATE) BETWEEN &#39;2021-01-01&#39; AND &#39;2021-02-01&#39;<br>  GROUP BY user_id, session_type<br>)<br>SELECT<br>  user_id,<br>  session_type,<br>  total_duration<br>  RANK() OVER(PARTITION BY session_type ORDER BY total_duration DESC) AS rn<br>FROM total_session_details<br>ORDER BY 2,4;</pre><p>17. Snapchat: Assume you are given the tables below containing information on Snapchat users and their time spent sending and opening snaps. Write a query to obtain a breakdown of the time spent sending vs. opening snaps (as a percentage of total time spent) for each of the different age groups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BgPuo05FiaHM4asGwx6L4Q.jpeg" /></figure><pre>WITH snap_details AS (<br>  SELECT <br>    t2.age_bucket,<br>    SUM(time_spent) AS total_time_spent,<br>    SUM(IF(t1.type = &#39;send&#39;,time_spent,0)) AS time_spent_spending,<br>    SUM(IF(t1.type = &#39;open&#39;,time_spent,0)) AS time_spent_opening<br>  FROM activities as t1<br>  INNER JOIN age_breakdown as t2 ON t1.user_id = t2.user_id<br>  GROUP BY t2.age_bucket<br>)<br>SELECT<br>  t2.age_bucket,<br>  ROUND((time_spent_spending*100/NULLIF(total_time_spent, 0)), 2)<br>    AS per_time_spent_spending,<br>  ROUND((time_spent_opening*100/NULLIF(total_time_spent, 0)), 2)<br>    AS per_time_spent_opening,<br>FROM snap_details</pre><p>18. Pinterest: Assume you are given the table below containing information on user sessions, including their start and end times. A session is considered to be concurrent with another user’s session if they overlap. Write a query to obtain the user session that is concurrent with the largest number of other user sessions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*029eS1JJeIMhCUCsqoEEzQ.jpeg" /></figure><pre>SELECT t1.session_id, COUNT(t2.session_id) AS concurrent_sessions<br>FROM sessions AS t1<br>LEFT JOIN sessions AS t2 ON t1.session_id != t2.session_id AND<br>  t1.start_time &lt; t2.end_time AND t2.start_time &lt; t1.end_time<br>GROUP BY t1.session_id<br>ORDER BY 2 DESC<br>LIMIT 1</pre><p>19. Yelp: Assume you are given the table below containing information on user reviews. Define a top-rated business as one whose reviews contain only 4 or 5 stars. Write a query to obtain the number and percentage of businesses that are top rated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lqt4zGIVNelcPE6B_F5keg.jpeg" /></figure><pre>WITH min_rating_details AS (<br>  SELECT business_id, MIN(review_stars) AS min_rating<br>  FROM reviews<br>  GROUP BY business_id<br>)<br>SELECT<br>  COUNT(business_id) AS total_businessess,<br>  SUM(IF(min_rating IN (4,5),1,0)) AS rating_above_3<br>  ROUND(SUM(IF(min_rating IN (4,5),1,0))*100/COUNT(business_id), 2) AS per<br>FROM min_rating_details</pre><p>20. Google: Assume you are given the table below containing measurement values obtained from a sensor over several days. Measurements are taken several times within a given day. Write a query to obtain the sum of the odd numbered measurements and the sum of the even-numbered measurements by date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LqlIK9rI-7Wy2j8PHGZo8Q.jpeg" /></figure><pre>WITH measurment_order_details AS (<br>  SELECT<br>  *,<br>  DATE(measurment_time) AS measurment_day,<br>  ROW_NUMBER() OVER(<br>    PARTITION BY DATE(measurment_time)<br>    ORDER BY measurment_time) AS rn<br> FROM measurements<br>)<br>SELECT<br>  measurment_day,<br>  SUM(IF(rn%2 = 0, measurment_value,0)) AS total_even_measurment_value,<br>  SUM(IF(rn%2 = 1, measurment_value,0)) AS total_odd_measurment_value, <br>FROM measurment_order_details<br>GROUP BY measurment_day<br>ORDER BY 1<br></pre><p>21. Etsy: Assume you are given the two tables below containing information on user signups and user purchases. Of the users who joined within the past week, write a query to obtain the percentage of users that also purchased at least one item.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hlFm7CyAu-WCMu7QOnTpWg.jpeg" /></figure><pre>WITH past_week_users AS (<br>  SELECT *<br>  FROM signups<br>  WHERE signup_date &gt;= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)<br> )<br>SELECT <br>  ROUND(COUNT(DISTINCT u.user_id)*100/COUNT(DISTINCT p.user_id), 2)<br>FROM past_week_users AS p<br>LEFT JOIN user_purchases AS u ON p.user_id = u.user_id<br></pre><p>22. Walmart: Assume you are given the following tables on customer transactions and products. Find the top 10 products that are most frequently bought together (purchased in the same transaction).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xPEUZtKbmNt3PB0MBJOAvw.jpeg" /></figure><pre>WITH product_details AS (<br>  SELECT p.product_id, p.product_name, t.transaction_id<br>  FROM transactions AS t<br>  INNER JOIN products AS p ON p.product_id = t.product_id<br>)<br>SELECT CONCAT(p1.product_name, &#39;-&#39;, p2.product_name)<br>FROM product_details AS p1<br>INNER JOIN product_details AS p2 ON p1.product_name &lt; p2.product_name<br>  AND p1.transaction_id = p2.transaction_id<br>GROUP BY 1<br>ORDER BY COUNT(*) DESC<br>LIMIT 10<br></pre><p>23. Facebook: Assume you have the table given below containing information on user logins. Write a query to obtain the number of reactivated users (i.e, those who didn’t log in the previous month, who then logged in during the current month).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BOHB1u0q7ukqaM61T9BUNA.jpeg" /></figure><pre>WITH monthly_data AS (<br>  SELECT DISTINCT<br>    user_id,<br>    DATE_FORMAT(login_date, &#39;%Y-%m-01&#39;) AS month_start<br>  FROM user_logins<br>)<br>SELECT curr.month_start, COUNT(DISTINCT curr.user_id) AS retention_users<br>FROM monthly_data AS curr<br>LEFT JOIN monthly_data AS prev <br>ON (prev.user_id = curr.user_id) AND<br>DATE_SUB(curr.month_start, INTERVAL 1 MONTH) = prev.month_start<br>WHERE prev.month_start IS NULL AND <br>  curr.month_start &gt; (SELECT MIN(month_start) FROM monthly_data)<br>GROUP BY curr.month_start<br>ORDER BY 1<br><br></pre><p>24. Wayfair: Assume you are given the table below containing information on user transactions for particular products. Write a query to obtain the year-on-year growth rate for the total spend of each product, for each week (assume there is data each week).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*59B0jn_EHxpD_b8MFTaF3Q.jpeg" /></figure><pre>WITH weekly_data AS (<br>  SELECT<br>    product_id,<br>    DATE_SUB(transaction_date, INTERVAL DAYOFWEEK(transaction_date) - 1 DAY) AS week_start,<br>    SUM(spend) AS total_week_spend<br>  FROM user_transactions<br>  GROUP BY 1, 2<br>),<br>weekly_with_lag AS (<br>  SELECT<br>    product_id,<br>    week_start,<br>    total_week_spend,<br>    LAG(total_week_spend, 52) OVER(<br>      PARTITION BY product_id<br>      ORDER BY week_start<br>    ) AS prev_year_week_spend<br>  FROM weekly_data<br>)<br>SELECT<br>  product_id,<br>  week_start,<br>  total_week_spend,<br>  prev_year_week_spend,<br>  ROUND(<br>    (total_week_spend - prev_year_week_spend) * 100.0 / NULLIF(prev_year_week_spend, 0),<br>    2<br>  ) AS yoy_growth_rate<br>FROM weekly_with_lag<br>ORDER BY product_id, week_start;</pre><p>25. Stripe: Assume you are given the table below containing information on user transactions for a particular business using Stripe. Write a query to obtain the account’s rolling 7-day earnings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*d9PY9tk0ojDn8cO51072Lw.jpeg" /></figure><pre>WITH user_spendings AS (<br>  SELECT<br>    user_id,<br>    CAST(transaction_date AS DATE) AS trans_date,<br>    SUM(amount) AS total_spend<br>  FROM user_transactions<br>  GROUP BY 1, 2<br>)<br>SELECT<br>  *,<br>  SUM(total_spend) OVER (<br>    PARTITION BY user_id <br>    ORDER BY trans_date <br>    ROWS BETWEEN 6 PRECEDING AND CURRENT ROW<br>  ) AS rolling_7day_spend<br>FROM user_spendings<br>ORDER BY user_id, trans_date;</pre><p>27. Expedia: Say you have a database system where most of the queries made were UPDATES/INSERTS/DELETES. How would this affect your decision to create indices? What if the queries made were mostly SELECTs and JOINs instead?</p><p>Ans: Indexes improve SELECT/JOIN performance but slow down INSERT/UPDATE/DELETE because every modification must update all indexes. For OLTP with frequent writes, minimize indexes. For OLAP with frequent reads, use more indexes to speed up queries.</p><p>28. Amazon: Describe some advantages and disadvantages of relational databases vs. NoSQL databases.</p><p><strong>Relational Database Advantages:</strong></p><ul><li>Ensures data integrity via schema and ACID properties</li><li>Easy to start with for small-scale apps</li><li>Well-suited for vertical scaling</li><li>Standard query language (SQL) — easy to learn/switch</li></ul><p><strong>Relational Database Disadvantages:</strong></p><ul><li>Schema must be known in advance</li><li>Frequent schema changes on large tables cause performance issues</li><li>Horizontal scaling is difficult, leading to performance bottlenecks</li></ul><p><strong>NoSQL Advantages:</strong></p><ul><li>Flexible data formats — handles unstructured/semi-structured data easily</li><li>Ideal for rapid iteration and startups adding features quickly</li><li>Easy horizontal scaling</li><li>Better for high-availability applications</li></ul><p><strong>NoSQL Disadvantages:</strong></p><ul><li>Weaker data correctness guarantees (soft state, eventual consistency)</li><li>Managing data consistency is challenging without predefined schema</li><li>Some NoSQL types struggle with complex queries or access patterns</li></ul><p><strong>WHEN TO USE</strong>: Relational for structured data with strong consistency needs. NoSQL for flexible schemas, rapid development, and high availability requirements.</p><p>29. Capital One: Say you want to set up a MapReduce job to implement a shuffle operator, whose input is a dataset and whose output is a randomly ordered version of that same dataset. At a high level, describe the steps in the shuffle operator’s algorithm.</p><p>Ans: To implement a shuffle operator using MapReduce that randomly reorders a dataset, I’d use the following approach:</p><p><strong>Map Step:</strong> For each input row, generate a random number between 0 and 1, and emit it as the key with the original row as the value.</p><p><strong>Shuffle Step:</strong> MapReduce’s built-in shuffle automatically distributes these key-value pairs across reducers and sorts them by the random keys. This achieves two things: it mixes data from all mappers together and orders them by random numbers.</p><p><strong>Reduce Step:</strong> Simply output the rows in the order received, dropping the random keys. Since the data arrives sorted by random numbers, the output is randomly shuffled.</p><p><strong>Why This Works:</strong> By assigning random keys and leveraging MapReduce’s sorting mechanism, we achieve distributed randomization. Each mapper independently generates random numbers, the shuffle phase mixes all data across machines, and sorting by random values produces a random ordering. This scales to any dataset size across any number of machines.</p><p><strong>Result:</strong> The output dataset contains all original rows in completely random order, distributed across the cluster.</p><pre>Original Data (across 3 machines):<br>Machine 1: [Alice, Bob]<br>Machine 2: [Charlie, Diana]  <br>Machine 3: [Eve, Frank]<br><br>MAP: Add random numbers<br>Machine 1: [(0.73, Alice), (0.21, Bob)]<br>Machine 2: [(0.89, Charlie), (0.15, Diana)]<br>Machine 3: [(0.45, Eve), (0.92, Frank)]<br><br>SHUFFLE: Mix and sort by random number<br>Reducer 1: [(0.15, Diana), (0.21, Bob), (0.45, Eve)]<br>Reducer 2: [(0.73, Alice), (0.89, Charlie), (0.92, Frank)]<br><br>REDUCE: Output just the data<br>Final: [Diana, Bob, Eve, Alice, Charlie, Frank]<br><br>Result: Randomly shuffled! ✓</pre><p>30. KPMG: Describe what a foreign key is and how it relates to a primary key.</p><p>Ans: A primary key uniquely identifies each row in a table. A foreign key is a column in one table that links to the primary key of another table, creating a relationship between them.</p><p>For example, in an Orders table, customer_id is a foreign key that references the customer_id primary key in the Customers table. This ensures every order belongs to a valid customer — you can’t create an order for a customer that doesn’t exist.</p><p>Foreign keys maintain data integrity and reduce redundancy by linking related data across tables instead of duplicating it.</p><p>31. Microsoft: Describe what a clustered index and a non-clustered index are. Compare and contrast the two.</p><p><strong>Clustered Index:</strong> “A clustered index determines how data is physically stored on disk. Think of a Students table — if we create a clustered index on student_id, the actual rows are physically sorted by student_id (101, 102, 103…) on the hard drive. When you search for student_id = 103, the database goes directly to that location — it’s like a dictionary where pages are already in alphabetical order. You can only have ONE clustered index per table because data can only be physically sorted one way.”</p><p><strong>Non-Clustered Index:</strong> “A non-clustered index is a separate lookup structure that points to data locations. Using the same Students table clustered by student_id, if we add a non-clustered index on ‘name’, it creates a separate sorted list: Alice→Block 1, Bob→Block 2, Charlie→Block 4, etc. When searching by name, the database first looks up the name in this index, finds the pointer, then jumps to that data block. It’s a two-step process, slightly slower than clustered, but we can have multiple non-clustered indexes.”</p><p><strong>Key Difference:</strong> “Clustered physically reorganizes the table data itself. Non-clustered creates a separate structure that points to the data. You use clustered for your primary search column and non-clustered for other columns you frequently query.”</p><p><strong>Example:</strong> “In an employee table, you’d typically cluster on employee_id for fast primary key lookups, but add non-clustered indexes on last_name, department, or hire_date so queries on those columns are also fast without scanning the entire table.”</p><p>32. Facebook: Say you had the entire Facebook social graph (users and their friendships). How would you use MapReduce to find the number of mutual friends for every pair of Facebook users?</p><p>Goal: For each pair of friends (A, B), count their mutual friends.</p><p><strong>Example Setup</strong>: A is friends with: B, C, D and B is friends with: A, C, E Mutual friends of A and B: C (count = 1)</p><p>Input Format: UserID, [List_of_friends]</p><pre>A B,C,D<br>B A,C,E<br>C A,B,E<br>D A<br>E B,C</pre><p><strong>STEP 1: MAP</strong> For each user U and each friend F in U’s friend list: • Emit Key: (U, F) in sorted order (e.g., (A,B) not (B,A)) • Emit Value: U’s complete friend list</p><p>Example — User A with friends [B,C,D]:</p><pre>(A,B) → [B,C,D]<br>(A,C) → [B,C,D]<br>(A,D) → [B,C,D]</pre><p><strong>STEP 2: SHUFFLE</strong> Group all values by key (friend pair). Key (A,B) receives both friend lists:</p><pre>(A,B) → [[B,C,D], [A,C,E]]</pre><p><strong>STEP 3: REDUCE</strong> For each key (U, F):</p><ol><li>Find intersection of both friend lists</li><li>Count the intersection size</li><li>Emit (U, F) → count</li></ol><p>Example:</p><pre>Key: (A,B)<br>Values: [B,C,D], [A,C,E]<br>Intersection: [C]<br>Output: (A,B) → 1</pre><p>Final Output:</p><pre>(A,B) → 1<br>(A,C) → 1<br>(A,D) → 0<br>(B,C) → 2<br>(B,E) → 1</pre><p>33. Google: Assume you are tasked with designing a large-scale system that tracks a variety of search query strings and their frequencies. How would you design this, and what trade-offs would you need to consider?</p><p><strong>Basic Structure:</strong></p><ul><li>Key-value store: &lt;query_string, frequency&gt;</li><li>Database table with two fields: query string and count</li></ul><p><strong>Scaling Approach:</strong></p><ul><li>Vertical Scaling: Add CPU/RAM — NOT suitable for Google scale</li><li>Horizontal Scaling: Add more nodes/machines — BETTER for large scale</li><li>Distribute query strings across multiple shards</li></ul><p><strong>Sharding Strategies:</strong></p><ol><li><strong>Range-Based Sharding</strong>:</li></ol><ul><li>Map queries by first letter (A→Shard1, B→Shard2, etc.)</li><li>Pros: Simple to implement</li><li>Cons: Uneven data distribution (queries starting with ‘A’ &gt;&gt; queries starting with ‘X’)</li></ul><p><strong>2. Hash-Based Sharding</strong>:</p><ul><li>Use hash function to map query string to shard number</li><li>Pros: Better data distribution, avoids hot spots</li><li>Cons: Adding new nodes requires rehashing all data (expensive rebalancing)</li></ul><p><strong>3. Consistent Hashing (BEST)</strong>:</p><ul><li>Advanced hash-based approach</li><li>Pros: Minimal data rebalancing when adding/removing nodes</li><li>Solves the rehashing problem of basic hash sharding</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f2c89699d07" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A/B Testing Quick Revision Notes for Data Science Interviews]]></title>
            <link>https://medium.com/@mdmoseena22/a-b-testing-quick-revision-notes-for-data-science-interviews-236e6354b243?source=rss-72a4fb8cbdaf------2</link>
            <guid isPermaLink="false">https://medium.com/p/236e6354b243</guid>
            <category><![CDATA[ab-testing]]></category>
            <dc:creator><![CDATA[MOHAMMED MOSEENA]]></dc:creator>
            <pubDate>Mon, 22 Sep 2025 14:07:02 GMT</pubDate>
            <atom:updated>2026-03-30T17:12:17.386Z</atom:updated>
            <content:encoded><![CDATA[<p>A/B testing is <strong>two-sample hypothesis testing</strong> used to determine if two groups differ statistically significant.</p><blockquote><strong>A/B Testing Experimental Design Setup</strong></blockquote><p><strong>1. Pick Metrics Before Starting</strong></p><ul><li><strong>Primary North Star Metric</strong>: The main metric that defines success (pick ONE upfront to avoid cherry-picking)</li><li><strong>Guardrail Metrics</strong>: Ensure you’re not harming other critical areas</li><li><em>Example</em>: Netflix autoplay feature → Primary North star metric: viewing time, Guardrail metric: retention rate</li></ul><p><strong>2. Define Key Thresholds</strong></p><ul><li><strong>Alpha (α)</strong>: Type I error rate, typically 0.05 (5% false positive risk)</li><li><strong>Power (1-β)</strong>: Typically 0.8 (80% chance to detect true effect)</li><li><strong>MDE (Minimum Detectable Effect)</strong>: Smallest change worth caring about (set with stakeholders)</li></ul><p><strong>3. Determine Sample Size &amp; Duration</strong></p><ul><li>Calculate required sample size based on α, β, MDE, and metric variance</li><li><strong>Rule of thumb</strong>: Run for at least 2 weeks to account for day-to-day variations even the results look statstically significant.</li><li>Duration = Required sample size ÷ Daily traffic</li></ul><p><strong>4. Assign Groups Properly</strong></p><ul><li><strong>Randomization </strong>Groups must be randomly assigned. If users are split by region, device type, or channel (e.g., Group A gets email traffic, Group B gets paid ads), any observed difference could be from that split, not your change.</li><li><strong>Group Comparability (No Bias)</strong> Before the experiment, both groups should look identical on key metrics — same conversion baseline, same traffic sources, same user demographics. Run a pre-experiment sanity check (AA test) to verify.</li><li><strong>Single Variable Change (Isolation):</strong> For single variant A/B testing only one thing should differ between groups. If Group B has a new button color AND a new image, you can’t know which caused the lift.</li><li><strong>No Contamination (Network Effect)</strong> Control group users must be completely unexposed to the treatment. Common failure: a user sees variant B on mobile, then visits again on desktop and gets counted in control — now control is “infected.”</li></ul><blockquote><strong>When NOT to A/B Test</strong></blockquote><ol><li><strong>Lack of Infrastructure</strong> → No proper tools for randomization/data collection</li><li><strong>Lack of Impact</strong> → Change too small to matter business-wise</li><li><strong>Lack of Traffic</strong> → Not enough users for reliable results</li><li><strong>Lack of Conviction</strong> → Don’t replace good product intuition with endless testing</li><li><strong>Lack of Isolation</strong> → Changes can’t be isolated (e.g: logo changes)</li></ol><p>Memory trick: IIIT-C</p><blockquote><strong>Alternatives to A/B Testing</strong></blockquote><ol><li><strong>User Research</strong> → Interviews, surveys, focus groups</li><li><strong>User Activity Logs</strong> → Analyze existing behavioral data</li><li><strong>Retrospective Analysis</strong> → Launch globally, compare before/after metrics</li></ol><blockquote><strong>Common A/B Testing Pitfalls &amp; Solutions</strong></blockquote><p><strong>1. Non-Normality</strong></p><p><strong>Problem</strong>: Metric doesn’t follow normal distribution</p><p><strong>Solutions</strong>:</p><ul><li><strong>Bootstrapping</strong>: Generate extra samples to invoke CLT</li><li><strong>Alternative tests</strong>: Wilcoxon rank-sum test (non-parametric)</li><li><strong>Gather more data</strong>: As CLT work with larger samples</li></ul><p><strong>2. Multiple Testing Problem</strong></p><p><strong>Problem</strong>: Running many tests increases false positive risk</p><p><strong>Solution</strong>: <strong>Bonferroni Correction</strong> → Adjust α based on number of tests</p><ul><li><em>New α = Original α ÷ Number of tests</em></li><li><em>Trade-off</em>: Reduces Type I error but increases Type II error</li></ul><p><strong>3. Network Effects</strong></p><p><strong>Problem</strong>: Users influence each other (e.g., social networks) → contaminate control group.</p><p><strong>Solutions</strong>:</p><ul><li><strong>Cluster sampling</strong>: Group connected users, randomize clusters</li><li><strong>Check baseline metrics</strong>: Monitor for contamination between groups</li></ul><p><strong>4. Novelty &amp; Primacy Effects</strong></p><p><strong>Problem</strong>:</p><ul><li><strong>Novelty</strong>: Users try new features out of curiosity (temporary spike)</li><li><strong>Primacy</strong>: Users resist change initially</li></ul><p><strong>Solutions</strong>:</p><ul><li>Test on <strong>new users only</strong> (if representative)</li><li>Maintain <strong>holdout groups</strong> after launch for long-term measurement</li></ul><blockquote><strong>Post-Significance Considerations</strong></blockquote><p>Even with statistical significance, evaluate:</p><ol><li><strong>Effect Size</strong>: Is the difference practically meaningful?</li><li><strong>Guardrail Impact</strong>: Are counter-metrics negatively affected?</li><li><strong>Implementation Costs</strong>:</li></ol><ul><li>Human labor for deployment</li><li>Ongoing support burden</li><li>Risk of hidden bugs</li></ul><p><strong>Key Point</strong>: Statistical significance ≠ Business significance (especially with large sample sizes)</p><blockquote><strong>Statistical Framework</strong></blockquote><p><strong>Hypothesis Setup</strong></p><ul><li><strong>H₀</strong>: μₐ — μᵦ = 0 (no difference between groups)</li><li><strong>H₁</strong>: μₐ — μᵦ ≠ 0 (there is a difference)</li></ul><p><strong>Key Assumption</strong></p><p>Sampling distribution of group A and group B follows normal distribution → Therefore difference between sample means (x̄ₐ — x̄ᵦ) follows normal distribution (CLT).</p><p><strong>Test Statistics</strong></p><ul><li><strong>Z-test</strong>: z = (x̄ₐ - x̄ᵦ) / √(σ²ₐ/nₐ + σ²ᵦ/nᵦ) (known variances - rare)</li><li><strong>T-test</strong>: t = (x̄ₐ - x̄ᵦ) / √(s²ₐ/nₐ + s²ᵦ/nᵦ) (sample variances - common)</li></ul><p><strong>Degrees of Freedom</strong></p><ul><li>Equal variances: df = nₐ + nᵦ — 2</li><li>Unequal variances: Use Welch’s t-test (software calculates)</li></ul><p><strong>Confidence Intervals</strong></p><ul><li><strong>Z-test</strong>: CI = (x̄ₐ - x̄ᵦ) ± z(α/2) × √(σ²ₐ/nₐ + σ²ᵦ/nᵦ)</li><li><strong>T-test</strong>: CI = (x̄ₐ - x̄ᵦ) ± t(α/2, df) × √(s²ₐ/nₐ + s²ᵦ/nᵦ)</li></ul><p><strong>Decision Making</strong></p><ol><li>Calculate test statistic</li><li>Find p-value</li><li>Compare to α (typically 0.05)</li></ol><ul><li><strong>p ≤ α</strong>: Reject H₀ (significant difference)</li><li><strong>p &gt; α</strong>: Fail to reject H₀ (no significant difference)</li></ul><p><strong>Alternative</strong>: Use CI approach</p><ul><li><strong>CI contains 0</strong>: Fail to reject H₀</li><li><strong>CI doesn’t contain 0</strong>: Reject H₀</li></ul><p><strong>Interview Questions:</strong></p><blockquote>1. Lyft: Your team is trying to figure out whether a new rider app with extra UI features would increase the number of rides taken. For an A/B test, how would you split users and ensure that your tests have balanced groups?</blockquote><p><strong>Step 1: Basics of A/B Testing and Balanced Groups</strong></p><p><strong>Goal:</strong> Test if new UI features increase rides</p><p><strong>Standard approach:</strong></p><ul><li>Control (A) → old app</li><li>Treatment (B) → new app</li></ul><p><strong>Problem with naive random split:</strong> Imbalanced groups (e.g., Group A has more big-city users)</p><p><strong>Solution: Stratified sampling</strong></p><ul><li>Split users within important categories (city size, age, ride frequency)</li><li>Randomly assign equal proportions from each stratum to A and B</li><li>Ensures groups look similar before test</li></ul><p><strong>Step 2: Problem of Network Effects in Marketplaces</strong></p><p><strong>Why user-level splits fail for Lyft:</strong></p><ul><li>Riders and drivers share the same resource pool</li><li>If Group B (new app) really requests more rides → reduces driver availability for BOTH groups</li><li>Group A’s experience worsens even though they’re control</li><li><strong>Result:</strong> Invalid comparison</li></ul><p><strong>Example:</strong> 1,000 drivers in Bengaluru. Group B requests 20% more rides → wait times increase for everyone, including control group</p><p><strong>Step 3: Geo-Based (Cluster) Randomization</strong></p><p><strong>Solution:</strong> Assign entire cities</p><ul><li>Bengaluru = new app (treatment)</li><li>Hyderabad = old app (control)</li></ul><p><strong>Why it works:</strong></p><ul><li>Treatment group competes internally (real marketplace behavior)</li><li>Control group stays completely untouched</li><li>No interference between groups</li></ul><p><strong>Step 4: Ensuring Balance Across Geos</strong></p><p><strong>Challenge:</strong> Cities aren’t identical (different cultures, transit systems, weather)</p><p><strong>How to ensure balance:</strong></p><ol><li><strong>Pair similar clusters(cities)</strong> (Bengaluru ↔ Hyderabad, Mumbai ↔ Delhi)</li><li><strong>Check baseline metrics</strong> (avg rides/rider, wait times) before any changes to treatment group, make sure treatment group and control group have similar baselines before any changes to treatment group.</li><li><strong>Use multiple clusters</strong> for a group so natural differences average out</li></ol><blockquote>2. Airbnb: What factors might make A/B testing metrics on the Airbnb platform difficult?</blockquote><p><strong>Standard A/B Testing Pitfalls: </strong>Non-normality, multiple testing, network effects, novelty/primacy effects</p><p><strong>Airbnb-Specific Challenges</strong></p><p><strong>Step 1: Complexity of User Flow</strong></p><p><strong>Problem:</strong> Booking isn’t linear — search → message hosts → wait for response → book</p><ul><li>External factors (host responsiveness) create noise</li><li><strong>Example:</strong> Testing new photo feature, but effect masked by slow host replies</li></ul><p><strong>Solution:</strong> Use <strong>non-intermediary metrics</strong> like search-to-booking conversion rate (captures all external effects for both groups)</p><p><strong>Step 2: User Bucketing Across Multiple People &amp; Devices</strong></p><p><strong>Problem:</strong> Group travel + cross-device browsing creates attribution issues</p><ul><li><strong>Example:</strong> Mother browses on tablet (treatment), books on phone (control) — which experience drove booking?</li></ul><p><strong>Solution:</strong></p><ul><li>Strengthen cross-device mapping</li><li>Use imputation/predictive models for missing data</li><li>Validate with internal/external datasets</li></ul><p><strong>Step 3: Long Time Horizon for Measuring Success</strong></p><p><strong>Problem:</strong> Key metrics (retention, reviews, LTV) realized weeks/months later</p><ul><li>A/B tests can’t run that long</li><li>Bookings are low-frequency events</li></ul><p><strong>Solution:</strong> Use <strong>short-term proxy metrics</strong> correlated with long-term success:</p><ul><li>Wishlist adds</li><li>Repeat searches</li><li>Host messaging rates</li></ul><p><strong>Note: </strong>Beyond standard pitfalls, Airbnb faces:</p><ol><li>Complex booking funnel → use end-to-end metrics</li><li>Multi-user/device issues → improve tracking &amp; imputation</li><li>Long feedback cycles → leverage short-term proxies</li></ol><blockquote><em>3. Twitter: What are some common pitfalls encountered in A/B testing?</em></blockquote><p><strong>1. Sample Size &amp; Timing Issues</strong></p><p><strong>Problems:</strong></p><ul><li>Too short duration or too small sample → unreliable results</li><li>Small samples → non-normality in test statistic distribution</li><li>Short tests → miss seasonality effects (weekday vs weekend behavior)</li></ul><p><strong>Solution:</strong> Run experiments long enough to achieve desired power and cover relevant cycles</p><p><strong>2. Multiple Testing &amp; Significance Inflation</strong></p><p><strong>Problem:</strong> With many experiments, false positive risk grows</p><ul><li><strong>Example:</strong> 100 tests at α=0.05 → ~5 appear significant by chance (Type I error)</li></ul><p><strong>Solution:</strong> Use corrections like <strong>Bonferroni</strong> (but increases Type II error risk)</p><p><strong>3. Randomization &amp; Network Effects</strong></p><p><strong>Problem:</strong> Imbalanced groups (demographics, device, geography) → results from sampling bias, not treatment</p><ul><li>Groups not independent → network effects contaminate results</li></ul><p><strong>Solution:</strong></p><ul><li><strong>Stratified sampling</strong></li><li><strong>Pair matching</strong></li><li><strong>Clustering</strong> (geo-based randomization)</li></ul><p><strong>4. Peeking &amp; Stopping Early</strong></p><p><strong>Problem:</strong> Stopping when results “look significant” → inflates false positives</p><p><strong>Solution:</strong></p><ul><li>Use <strong>sequential testing methods</strong></li><li>Define <strong>pre-set stopping rules</strong></li></ul><p><strong>5. Behavioral Effects (Novelty &amp; Primacy)</strong></p><p><strong>Problems:</strong></p><ul><li><strong>Novelty effect:</strong> Initial excitement skews results upward</li><li><strong>Primacy effect:</strong> Resistance to change skews results downward</li></ul><p><strong>Solution:</strong></p><ul><li>Run longer tests to let effects stabilize</li><li>Use <strong>holdout groups</strong> to analyze long-term impact post-rollout</li></ul><blockquote><em>4. Meta: Let’s say that Facebook would like to change the user interface of the composer feature (the posting box) to be more like Instagram. Instead of a box, Facebook would add a “+” button at the bottom of the page.</em></blockquote><p><strong>Problem: </strong>Test changing Facebook posting box → Instagram-style “+” button at bottom of page</p><p><strong>Step 1: Define Success Metrics</strong></p><p><strong>Understand product goal:</strong> Encourage more posting</p><p><strong>North Star metric:</strong> Number of posts per user (weekly)</p><p><strong>Guardrail metric:</strong> Daily Active Users / Weekly Active Users</p><p><strong>Goal:</strong> Improve north star without harming guardrail</p><p><strong>Step 2: Set Thresholds</strong></p><ul><li>Define <strong>α</strong> (false positive rate — Type I error)</li><li>Define <strong>β</strong> (false negative rate — Type II error)</li><li>Set <strong>MDE</strong> (Minimum Detectable Effect) with stakeholders →Smallest meaningful change worth detecting</li></ul><p><strong>Step 3: Determine Sample Size &amp; Duration</strong></p><p><strong>Sample size:</strong> Use power analysis + confidence level + MDE</p><p><strong>Duration:</strong> Run for <strong>at least 2 weeks</strong> (or multiple user cycles)</p><ul><li>Don’t stop early even if results look significant</li><li>Avoids short-term novelty bias</li></ul><p><strong>Step 4: Assign Groups</strong></p><p><strong>Groups:</strong></p><ul><li>Control (A): Existing posting box</li><li>Treatment (B): New “+” button</li></ul><p><strong>Randomization:</strong></p><ul><li>Ensure groups are <strong>balanced and isolated</strong></li><li>Use <strong>cluster-based randomization</strong> (not individual) due to network effects on Facebook</li></ul><p><strong>Behavioral considerations:</strong></p><ul><li>Novelty effect: Initial curiosity boosts usage</li><li>Primacy effect: Resistance to change</li><li>Keep <strong>holdout group</strong> even after rollout to study long-term behavior</li></ul><p><strong>Step 5: Evaluation</strong></p><p><strong>Decision criteria:</strong></p><ol><li>Treatment improves north star by <strong>at least MDE</strong></li><li>Guardrail metrics stay <strong>within acceptable thresholds</strong></li><li>Results are <strong>statistically significant</strong> (p-value &lt; α)</li><li><strong>Effect size is practically meaningful</strong> (not just statistically significant)</li><li><strong>Implementation cost is justified</strong> by expected impact</li></ol><p>Only when ALL criteria met → roll out</p><blockquote><em>5. Amazon/Accenture: Let’s say you are in charge of shipments at Amazon. There are two types of parcels you can ship packages in, say A and B. Packages delivered in parcel A are damaged during shipment with probability p, while packages delivered in parcel B are damaged with probability q. What statistical test could you use to determine which parcel is better to use? What would the statistical test conclude if p=0.4 and q=0.6?</em></blockquote><blockquote><em>Note: Assume the values of p and q were obtained from data recorded from 200 shipments, half delivered with parcel A and half delivered with parcelB.</em></blockquote><p>Ans: A two-proportion z-test is appropriate for this A/B testing scenario, as we are comparing damage rates between two independent groups.</p><ul><li><strong>Null Hypothesis (H₀):</strong> p — q ≥ 0 (Parcel A is not better than parcel B)</li><li><strong>Alternative Hypothesis (H₁):</strong> p — q &lt; 0 (Parcel A is better than parcel B)</li></ul><p><strong>Test Statistic Calculation</strong></p><p>The test statistic follows the formula:</p><p>z = (p — q — 0) / √[(p(1-p)/n₁) + (q(1-q)/n₂)]</p><p>Substituting our values:</p><ul><li>p = 0.4, q = 0.6</li><li>n₁ = n₂ = 200</li></ul><p>z = (0.4–0.6) / √[(0.4 × 0.6 × 2) / 200]</p><p>= -0.2 / √(0.48 / 200)</p><p>= -0.2 / √0.0024</p><p>= -4.08</p><p>As it is left-tailed test for p-value we need to find P(Z≤z) for α = 0.05 from z-table</p><p>p-value = 0.00003 &lt; α = 0.05, we reject the null hypothesis.</p><p>The statistical analysis provides strong evidence that parcel A has a significantly lower damage rate than parcel B (p &lt; 0.05). Therefore, Amazon should use parcel type A for shipments, as it results in fewer damaged packages and better customer satisfaction.</p><blockquote><em>7. Meta, Dropbox: Let’s say your manager is running A/B tests on a website. You find out that your manager has run an A/B test with 20 different variants and come back to you that one of the variants is significant. Would you think there was anything fishy about the results?</em></blockquote><p>When performing multiple tests with a standard significance level of α = 0.05, there is a high probability of finding statistically significant results purely by random chance. In this case, with 20 tests and α = 0.05, we would expect about 20 × 0.05 = 1 statistically significant result just by chance, which is most likely a false positive. To address this, when running multiple tests, the significance level should be adjusted using a correction method such as the Bonferroni correction. With Bonferroni, the adjusted significance level becomes α/n = 0.05/20 = 0.0025, which reduces the likelihood of Type I errors.</p><blockquote><em>8. Dropbox: In a user-tied test, two randomized groups of users are created. In a user-untied test, instances are randomized into two groups. What are the pros and cons of user-tied test vs. user-untied test?</em></blockquote><p><strong>User-Tied Test: </strong>Randomized groups of <strong>users</strong> (each user always sees same version)</p><p><strong>Pros:</strong></p><ul><li><strong>Consistent user experience</strong> — no confusion from switching versions</li><li><strong>Better for user-level metrics</strong> — retention, LTV, conversion rates</li><li><strong>Eliminates within-user variance</strong> — cleaner statistical analysis</li><li><strong>Good for learning-dependent features</strong> — users can adapt to new experience</li></ul><p><strong>Cons:</strong></p><ul><li><strong>Requires user identification</strong> — needs login systems or persistent cookies</li><li><strong>Slower statistical significance</strong> — needs more users and time</li><li><strong>User-level biases</strong> — certain user types in one group can skew results</li><li><strong>Limited by user base size</strong> — can’t test many variations simultaneously</li></ul><p><strong>User-Untied Test: </strong>Randomized <strong>instances/sessions</strong> (same user can see both versions)</p><p><strong>Pros:</strong></p><ul><li><strong>Faster results</strong> — reaches significance quicker with more data points</li><li><strong>No user identification needed</strong> — works with anonymous traffic</li><li><strong>Can test more variations</strong> — each session is independent</li><li><strong>Eliminates selection bias</strong> — same users experience both versions over time</li></ul><p><strong>Cons:</strong></p><ul><li><strong>Inconsistent user experience</strong> — confusing when users see different versions</li><li><strong>Can’t measure user journey metrics</strong> — hard to track retention, repeat behavior</li><li><strong>Learning effects contamination</strong> — Version A experience affects reaction to Version B</li><li><strong>Not suitable for major UI changes</strong> — jarring to switch between designs</li></ul><p><strong>When to Use Each</strong></p><p><strong>User-Tied Tests for:</strong></p><ul><li>Major experience changes requiring learning/adaptation</li><li>Metrics needing time to develop (retention, LTV, habit formation)</li><li>Features where consistency is crucial for trust/usability</li></ul><p><strong>User-Untied Tests for:</strong></p><ul><li>Behind-the-scenes algorithmic improvements</li><li>Content/recommendation variations</li><li>Quick optimizations where inconsistency isn’t jarring</li><li>Testing at massive scale with fast feedback loops</li></ul><blockquote>9. Apple: Let’s say you’re analyzing an A/B test that has both a test group and a control group.</blockquote><blockquote>- How do you calculate the sample size necessary for an accurate measurement?</blockquote><blockquote>- Let’s say that the sample size is similar and sufficient between the two groups. In order to measure very small differences between the two, should the power get bigger or smaller?</blockquote><p><strong>1. Sample size calculation:</strong> Sample size = f(alpha, power, MDE, baseline variance)</p><ul><li><strong>Alpha (α)</strong>: Significance level, typically 0.05 (5% Type I error)</li><li><strong>Power (1-β)</strong>: Typically 0.80 (80% chance to detect true effect)</li><li><strong>MDE</strong>: Minimum effect you want to detect</li><li><strong>Variance</strong>: Variability in your metric</li></ul><p>Formula concept: Smaller MDE or higher power → larger sample size needed</p><p><strong>2. Power for detecting small differences:</strong> Power should get <strong>BIGGER</strong> (e.g., from 80% to 90%)</p><ul><li>Detecting smaller effects requires more sensitivity</li><li>Higher power = lower Type II error (missing real effects)</li><li><strong>Trade-off</strong>: Need larger sample size</li><li>Alpha stays fixed regardless of power setting</li></ul><blockquote>10. Meta: You are working on Facebook’s stories and have found that very few users click through the story functionality. You are tasked with measuring the success of Facebook stories, but you cannot use a standard A/B test.</blockquote><blockquote>What testing strategies and metrics would you use to determine the success of stories?</blockquote><p><strong>1. Clarify Purpose</strong></p><ul><li>Goal: Increase <strong>user engagement</strong> and <strong>content creation</strong>.</li><li>Key behaviors: posting, viewing, replying, sharing stories.</li></ul><p><strong>2. Align with Business Goals</strong></p><ul><li>Link Stories to broader Facebook metrics: more <strong>time spent</strong>, <strong>cross-feature interaction</strong>, higher <strong>daily active users</strong>.</li></ul><p><strong>3. Define Metrics</strong></p><p><strong>North Star Metrics:</strong></p><ul><li><em>Creation:</em> # of stories posted per day, % of users creating at least 1 story/week.</li><li><em>Engagement:</em> Avg. views per story, reply/reaction rate, % of stories watched to completion.</li></ul><p><strong>Guardrail Metrics:</strong></p><ul><li>Retention of active users, feed time per user, NPS, user complaints.</li></ul><p><strong>4. Alternative Testing Strategies</strong></p><ul><li><strong>Cohort / Retrospective Analysis:</strong> Compare engagement of users who started posting stories vs. those who didn’t.</li><li><strong>Behavioral Analysis:</strong> Track flow — e.g., users who view 3+ stories/day are 40% more likely to post their own.</li><li><strong>Surveys &amp; Interviews:</strong> Ask users why they skip stories (e.g., privacy concerns, unclear UI).</li><li><strong>Time-of-Day Analysis:</strong> Identify peak posting/viewing hours to optimize notifications or feature prompts.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=236e6354b243" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essential Statistics Topics to Revise Before Your Data Science Interview]]></title>
            <link>https://medium.com/@mdmoseena22/essential-statistics-topics-to-revise-before-your-data-science-interview-cdaca1fcc5ed?source=rss-72a4fb8cbdaf------2</link>
            <guid isPermaLink="false">https://medium.com/p/cdaca1fcc5ed</guid>
            <category><![CDATA[interview-questions]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[MOHAMMED MOSEENA]]></dc:creator>
            <pubDate>Mon, 08 Sep 2025 09:16:04 GMT</pubDate>
            <atom:updated>2026-03-13T05:16:37.317Z</atom:updated>
            <content:encoded><![CDATA[<p>Many interview questions — whether conceptual or applied — rely on your ability to recall and explain fundamental statistical principles because many commercial layers of a data science pipeline are built from statistical foundations(for example A/B testing).</p><p><strong>Key Topics to Review:</strong></p><ul><li>Properties of Random Variables</li><li>Law of Large Numbers</li><li>Central Limit Theorem</li><li>Hypothesis Testing</li><li>p-values and Confidence Intervals</li><li>Type I and Type II Errors</li><li>Maximum Likelihood Estimation (MLE) and Maximum A Posteriori (MAP)</li><li>Moment Generating Functions (MGF)</li></ul><blockquote><strong>Properties of Random Variables</strong></blockquote><p><strong>1. Random Variables &amp; Distributions</strong></p><ul><li><strong>Discrete RV</strong> → Countable range → uses <strong>PMF (Probability Mass Function)</strong></li><li><strong>Continuous RV</strong> → Uncountable range → uses <strong>PDF (Probability Density Function)</strong></li></ul><p><strong>2. Expected Value (Mean)</strong></p><ul><li><strong>Discrete:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/186/1*5EzXbYzO5M5r1hIvBFEj3Q.png" /></figure><ul><li><strong>Continuous:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*SZDT1_bnGrzGvPdjB7it7A.png" /></figure><p><strong>3. Variance &amp; Standard Deviation</strong></p><ul><li><strong>Variance:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*W8P4FuAfDPTgVzaPcKEsMA.png" /></figure><ul><li><strong>Standard Deviation:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*NAbcOzPjCifzDsshP8I15w.png" /></figure><p><strong>4. Conditional Expectation</strong></p><ul><li><strong>Conditional Expectation:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*ImxoehY6MaZM-7MnSFyZtQ.png" /></figure><p><strong>5. Covariance &amp; Correlation</strong></p><ul><li><strong>Covariance: </strong>covariance is a linear measure of relationship between two variables</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*5GbjE10-ZP01cmprcpIUpQ.png" /></figure><ul><li>Note: Cov(X, X) will be equal to Var(X)</li><li><strong>Correlation: </strong>Standardized version of Covariance</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*xoiTpJOPO_Hh4TzenVsJ4A.png" /></figure><p><strong>6. Practice Distributions</strong></p><ul><li>Uniform Distribution → derive E[X], Var(X)</li><li>Exponential Distribution → derive E[X], Var(X)</li></ul><p><strong>Reference: </strong><a href="https://docs.google.com/spreadsheets/d/1w-9YYJKYwgn8HPo_RCEKaJMGGuL0TvZ2-P72s4U9pV4/edit?gid=0#gid=0"><strong>https://docs.google.com/spreadsheets/d/1w-9YYJKYwgn8HPo_RCEKaJMGGuL0TvZ2-P72s4U9pV4/edit?gid=0#gid=0</strong></a></p><p><strong>Interview questions :</strong></p><blockquote><strong>1. Lyft: Explain both covariance and correlation formulatically and compare and contrast them</strong>.</blockquote><p><strong>Covariance: </strong>covariance is a linear measure of relationship between two variables</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*5GbjE10-ZP01cmprcpIUpQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*pjBo4IAcmQWGfkXZBULpYQ.png" /></figure><ul><li>Measures <strong>direction</strong> of linear relationship.</li><li>Range: −∞ to +∞</li><li><strong>Unit-dependent</strong> (depends on units of X &amp; Y).</li></ul><p><strong>Correlation: </strong>Standardized version of Covariance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*xoiTpJOPO_Hh4TzenVsJ4A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*qTNlluK_O0hUXaz8GoAh9A.png" /></figure><ul><li><strong>Standardized</strong> (unitless) version of covariance.</li><li>Range: −1≤rho≤+1</li><li>Indicates both <strong>strength and direction</strong> of linear relationship.</li></ul><p><strong>In short:</strong><br> Covariance → raw measure (unscaled)<br> Correlation → scaled, comparable measure.</p><blockquote><strong>2. Citadel: What is the expected number of rolls needed to see all six sides of a fair die?</strong></blockquote><p>This is the coupon-collector problem. If you’ve already seen k distinct faces, the chance the next roll shows a <em>new</em> face is 6−k/6​. The waiting time for the next new face is geometric with mean 6/6−k​.</p><p>Summing for k=0,…,5:</p><p>E[X] = 1+1/(5/6)+1/(4/6)+1/(3/6)+1/(2/6)+1/(1/6) = (1+ 6/5 + 6/4+ 6/3 + 6/2+ 6/1) = 14.7 rolls</p><p>So expected number of rolls to see all six sides is 14.7</p><blockquote><strong>3. Quora: You are drawing from a normally distributed random variable X ~ N(0,1) once a day. What is the approximate expected number of days until you get a value greater than 2?</strong></blockquote><p><strong>Expected Days to Get X &gt; 2:</strong></p><p>Given X∼N(0,1):</p><p>P(X&gt;2) = 1−P(Z≤2) = 1−0.9772 = 0.0228 → Sucess event</p><p>Each day’s draw is independent, so the number of days until the first X&gt;2 follows a <strong>Geometric distribution</strong> with p=0.0228.</p><p>E[days] = 1/p = 1/0.0228 ≈ 43.9 ≈ 44 days</p><blockquote>4. Akuna Capital: Say you have two random variables X and Y, each with a standard deviation. What is the variance of aX+bY for constants a and b.</blockquote><p>Lets say Z = aX+bY, need to find Var(Z), which is given by:</p><p>Var(Z) = E[(Z-E[Z])²] = E[(aX+bY-E[aX+bY])²] = E[(a(X-E[X]) + b(Y-E[Y]))²] = a²E[(X-E[X])²] + b²E[(Y-E[Y])²] + 2abE[(X-E[X])(Y-E[Y])]</p><p>Var(aX+bY) = a²Var(X) + b²Var(Y) + 2abCov(X,Y)</p><blockquote><strong>5. Google: Say we have X ~ Uniform(0,1) and Y ~ Uniform(0,1) and two are independent. What is the expected value of the minimum of X and Y?</strong></blockquote><p>Ans: Let Z = min(X, Y), so we need to find E[Z], for finding E[Z] we need to know PDF for Z, we can find PDF by taking derivative of CDF</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*8-OmqaMhuDdhMbE0m9Sm9Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*TbYnPFzWdR9qsP_abuOz0w.png" /></figure><blockquote><strong>6. Uber: Say you have n numbers 1…n, and you uniformly sample from this distribution with replacement n times. What is the expected number of distinct values you would draw ?</strong></blockquote><p>Given numbers 1, 2,…, n sampled <strong>n times with replacement</strong>:</p><p>Let Xi = 1 if number i appears at least once, else 0.<br> Then total distinct values: X = X1 + X2 + ⋯+ Xn</p><p>By linearity of expectation:</p><p>E[X] = n*P(Xi=1)</p><p>Now, P(Xi=1) =1−P(number i never appears)=1−(1−1/n)^n</p><p>Hence E[X] = n*(1−(1−1/n)^n)</p><p>For large n:</p><p>(1−1/n)^n ≈ e^-1 ≈ 0.632 =&gt; E[X] = 0.632n</p><p>✅ <strong>Expected distinct ≈ 63.2% of n</strong></p><blockquote><strong>7. Morgon Stanley: What is the expected value of the max of two dice rolls?</strong></blockquote><p>P(max(X,Y) = k) = P(both dice ≤ k) — P(both dice ≤ k-1) = (k/6)² — ((k-1)/6)²</p><p>Now I can calculate the expected value: E[max(X,Y)] = Σ k × P(max = k) = 1×(1/36) + 2×(3/36) + 3×(5/36) + 4×(7/36) + 5×(9/36) + 6×(11/36) = (1 + 6 + 15 + 28 + 45 + 66)/36 = 161/36 ≈ 4.472</p><blockquote><strong>8. Lyft: Derive the mean and variance of the uniform distribution U(a, b).</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*p5xGI9qSFJyXIvDhEPQiIw.png" /></figure><blockquote><strong>9. Citadel: How many cards would you expect to draw from standard deck before seeing the first ace?</strong></blockquote><p><strong>Expected Draws Until First Ace</strong></p><ul><li>Sampling <strong>without replacement</strong> ⇒ probability changes each draw ⇒ <strong>not geometric</strong>.</li><li>Treat the deck as a random permutation of 52 cards (4 aces + 48 others).</li><li>The 4 aces divide the deck into <strong>5 equal expected sections=&gt;</strong></li></ul><p>section 1|Ace1|section2|Ace2|section3|Ace3|section4|Ace4|section5</p><ul><li>Total non-aces = 48 ⇒ each section has 48/5=9.6 cards on average.</li><li>The first ace appears right after the first section.</li></ul><p>E[position of first ace]=9.6+1=10.6</p><p>✅ <strong>Expected number of draws ≈ 10.6 cards</strong></p><blockquote><strong>10. Google: Assume you are drawing from an infinite set of i.i.d random variables that are uniformly distributed from (0,1). You keep drawing as long as the sequence you are getting is monotonically increasing. What is the expected length of the sequence you draw ?</strong></blockquote><p><strong>Expected length of an increasing initial run</strong></p><p>Let L be the number of draws until the sequence stops being strictly increasing. For i ≥ 1,</p><p>P(L≥i) = P(Y1&lt;Y2&lt;⋯&lt;Yi) = 1/i!</p><p>(since all i! orderings of i distinct continuous draws are equally likely). By the tail-sum formula,</p><p>E[L1+L2…] = 1 + 1/2! + 1/3!.. = e-1.</p><p>(Continuous distribution ⇒ ties have probability 0, so 1/i! is exact.)</p><blockquote><strong>11. Facebook: There are two games involving dice that you can play, In the first game you roll two dice at once and receive a dollar equivalent to the product of the rolls. In the second game, you roll one die and get the dollar amount equivalent to the square of that value. Which has the higher expected value and why?</strong></blockquote><p>Game 2 (square of one die) has the higher expected value.</p><p><strong>Reason / numbers:</strong><br> Let X be one fair die (1,…,6 equally likely).</p><ul><li>Game 1 (two independent dice, product):</li></ul><p>E[product] = E[X]E[Y] = E[X]² = (3.5)² = 12.25</p><ul><li>Game 2 (square of one die): E[X²] = (1² + 2² + 3²..6²)/6 = 91/6 ~ 15.16</li></ul><p>Because Var⁡(X) =E[X²] — E[X]² &gt; 0 we always have E[X²]&gt;E[X]². Numerically 15.17&gt;12.25, so <strong>Game 2 wins</strong>.</p><blockquote><strong>12. Facebook: Dervie the expectation for a geometric random variable.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*xhbAybs8ITQW5SqgsHI1sw.png" /></figure><blockquote><strong>13. Goldman Sachs: Say we have a random variable X ~ D where D is arbitrary distribution. What is the distribution F(X) where F is the CDF of X?</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*GEkqCtOXSNpOiJpTmjw9fQ.png" /></figure><blockquote><strong>14. Google: Say you have two distinct subsets of a dataset for which you know their means and standard deviations. How do you calculate the blended mean and standard deviation of the total dataset? Can you extend it to K subsets?</strong></blockquote><p><strong>Blended Mean (2 Subsets)</strong>:</p><p><strong>Formula:</strong> μ_total = (n₁μ₁ + n₂μ₂) / (n₁ + n₂)</p><p><strong>Extension to K subsets:</strong> μ_total = Σᵢ(nᵢμᵢ) / Σᵢ(nᵢ)</p><p><strong>Blended variance (accounts for within- and between-group variation)</strong><br> For K subsets, total variance</p><p><strong>Formula for variance:</strong> σ²_total = Σᵢ[nᵢ(σᵢ² + (μᵢ — μ_total)²)] / N_total</p><p><strong>Final standard deviation:</strong> σ_total = √(σ²_total)</p><ul><li>σi²​ = within-group variance for subset i.</li><li>(μi−μtot)² = between-group contribution (how much subset means differ).</li></ul><p><strong>Why this works:</strong> total variance = average of subset variances <strong>plus</strong> extra variance because subset means differ from the overall mean.</p><p><strong>Concrete Example</strong></p><p><strong>Given:</strong></p><ul><li>Subset 1: n₁=100, μ₁=50, σ₁=10</li><li>Subset 2: n₂=200, μ₂=70, σ₂=15</li></ul><p><strong>Solution:</strong></p><ol><li>Blended mean: (100×50 + 200×70)/300 = 63.33</li><li>Blended variance: [100(10² + (50–63.33)²) + 200(15² + (70–63.33)²)]/300 = 272.29</li><li>Blended std dev: √272.29 = 16.50</li></ol><p><strong>Notice:</strong> The result (16.50) is larger than both original standard deviations because the subsets have different means, adding variability.</p><blockquote><strong>15. Two Sigma: Say we have two random variables X and Y. What does it mean for X and Y to be independent? What about uncorrelated? Give an example where X and Y are uncorrelated but not independent?</strong></blockquote><p><strong>Independence:</strong></p><ul><li>X and Y are independent if knowing one gives no info about the other.</li></ul><p>P(X,Y)=P(X) P(Y) or P(X∣Y)=P(X)</p><p><strong>Uncorrelated:</strong></p><ul><li>X and Y are uncorrelated if their covariance is zero.</li></ul><p>Cov(X,Y) = E[XY] − E[X]E[Y] = 0</p><ul><li>Means <strong>no linear relationship</strong>, but nonlinear dependence may exist.</li></ul><p><strong>Relationship:</strong></p><p>Independence⇒Uncorrelated,but not vice versa.</p><p><strong>Example (Uncorrelated but Not Independent):</strong><br> Let X∼Uniform(−1,1) and Y = X².</p><ul><li>E[X] = 0, E[Y] = E[X²] = 1/3, E[XY] = E[X³] = 0<br> → Cov(X,Y) = 0 ⇒ uncorrelated.</li><li>But Y is completely determined by X⇒ <strong>not independent</strong>.</li></ul><p>✅ <strong>Key Insight:</strong><br> Zero correlation ⇒ no <em>linear</em> relationship, but independence ⇒ no <em>any</em> relationship.</p><blockquote><strong>16. Citadel: Say we have X ~ Uniform(-1, 1) and Y=X². what is the covariance of X and Y?</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*ttfkLJ36ZI5zwTT64hWiAA.png" /></figure><blockquote><strong>17. Lyft: How do you uniformly sample points at random from a circle with radius R?</strong></blockquote><p>We need to generate random points (x,y) that lie inside a circle of radius R, centred at the origin O(0,0). Uniformity means every region inside the circle should have an equal probability of containing a point.</p><p>A common mistake is to generate x and y independently in the range [-R, R] and then check if they satisfy x²+y² ≤ R². This result is a non-uniform distribution, with points clustering near the center. Instead we use polar coordinates for uniform sampling.</p><p>Using polar coordinates</p><p>x = rcosθ; y =rsinθ where r ∈ [0, R] and θ ∈ [0,2π]</p><p>for uniform sampling we must choose:</p><ol><li>θ uniformly from [0,2π]</li><li>r should be used as square root transformation.</li></ol><p>Lets find the distribution of r</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*usVC04dShjqVbdCSejdy4Q.png" /></figure><p>This method ensures that points are evenly distributed across the area of the circle rather than clustering around center.</p><blockquote><strong>18. Goldman Sachs: There are 100 noodles in a bowl. At each step, you randomly select two noodle ends from the bowl and tie them together. What is the expectation on the number of loops formed?</strong></blockquote><p><strong>Setting up the problem:</strong></p><ul><li>We start with 100 noodles, each with 2 ends, so 200 total ends</li><li>At each step, we randomly select 2 ends and tie them together</li><li>We want to find the expected number of loops formed</li></ul><p><strong>Key insight:</strong> When we pick two ends to tie together, we either:</p><ol><li>Pick two ends from the same noodle → forms a loop</li><li>Pick two ends from different noodles → joins two noodles into one longer noodle</li></ol><p><strong>The recursive approach:</strong> Let E(n) be the expected number of loops when starting with n noodles.</p><p>When we have n noodles (so 2n ends total):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*KlYYgxtxYROEJcZDxZ_wsg.png" /></figure><p>Regardless of whether we form a loop or join two noodles, we end up with n-1 noodles remaining.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*x1dbs_hx8LpM2XWeGVzhVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*9U7jhjQI7WoZTMHzlVhQnw.png" /></figure><blockquote><strong>19. Say you have a dataset and you are asked to analyze it by looking at some metrics like the mean and median.</strong></blockquote><blockquote><strong>a. When would you use one measure over the other?</strong></blockquote><blockquote><strong>b. How do you calculate the confidence interval of each measure?</strong></blockquote><p><strong>a. When would you use one measure over the other?</strong></p><p>Both <strong>mean</strong> and <strong>median</strong> are measures of central tendency, but they behave differently depending on the dataset.</p><p><strong>Use Mean when:</strong></p><ul><li>Data is <strong>symmetric</strong> and has <strong>no extreme outliers</strong>.</li><li>Example: Heights of adults in a population (most values cluster around the center, extreme values are rare).</li><li>The mean takes <em>all values</em> into account, making it more sensitive and representative in well-behaved distributions (like normal).</li></ul><p><strong>Use Median when:</strong></p><ul><li>Data is <strong>skewed</strong> or has <strong>outliers</strong>.</li><li>Example: Income distribution (a few billionaires can distort the mean, but the median income shows the “typical” person’s income).</li><li>The median is <strong>robust</strong> against outliers since it only depends on the middle-ranked value.</li></ul><p><strong>Rule of thumb</strong>:</p><ul><li>Normal-like data → use <strong>mean</strong>.</li><li>Skewed/outlier-prone data → use <strong>median</strong>.</li></ul><p><strong>b. Confidence Interval for Mean and Median</strong></p><p>CI for sample mean is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*s9dhSFRbQp5rQBMRoX4cQw.png" /></figure><p>CI for sample median can be calculated by <strong>Bootstrap Method: </strong>Resample your data with replacement many times (say 10,000). Compute the median each time. Take the 2.5th and 97.5th percentiles of this bootstrap distribution as the 95% CI.</p><blockquote><strong>20. Let’s say you are working as an analyst. There was an error in collecting data at your company, and all entries in the database are off by some number c. If you were to add c to all the entries, what would happen to the sample statistics (mean, median, mode, range, variance) of the field?</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*_nQ94RY5iwOzARMvt7lvnA.png" /></figure><blockquote><strong>21. Write a function that outputs the (sample) variance given a list of integers. Note: round the result to 2 decimal places.</strong></blockquote><p><strong>Example:</strong></p><p><strong>Input:</strong></p><pre>test_list = [6, 7, 3, 9, 10, 15]</pre><p><strong>Output:</strong></p><pre>get_variance(test_list) -&gt; 16.67</pre><p>Solution:</p><pre>def get_variance(test_list):<br>    n = len(test_list)<br>    expec = sum(test_list)/n #O(n)<br>    s = 0<br>    for x in test_list: #O(n)<br>      ele = x-epec<br>      s += ele*ele<br>    return round(s/n-1, 2) #Sample variance divide by n-1<br>#TC: O(n)+O(n) ~ O(n)<br>#AS: O(1)</pre><blockquote><strong>21. Facebook: What do you think the distribution of time spent per day on Facebook looks like? What metrics would you use to describe that distribution?</strong></blockquote><p><strong>Expected Shape: Right-skewed (Long right tail)</strong></p><p><strong>Why?</strong></p><ul><li>Most users: 10–30 min (casual browsers)</li><li>Power users: 2+ hours (small % of heavy engagers)</li><li>Natural floor: Few minutes minimum</li><li>No ceiling: Unlimited potential usage → long tail</li></ul><p><strong>Key Metrics:</strong></p><p><strong>Primary:</strong></p><ul><li><strong>Median (P50):</strong> Best for typical user (robust to outliers)</li><li><strong>P90:</strong> Heavy user threshold (engagement strategy)</li><li><strong>IQR (P75-P25):</strong> Mainstream usage spread</li></ul><p><strong>Supporting:</strong></p><ul><li><strong>Mean:</strong> Total engagement (skewed by power users)</li><li><strong>Skewness:</strong> Quantify right-tail bias (expect +2 to +4)</li><li><strong>Mode:</strong> Most common usage pattern</li></ul><p><strong>Why not mean alone?</strong> Heavily inflated by power users; median better represents typical behavior.</p><blockquote><strong>22. Amazon: Write a function </strong><strong>compute_deviation that takes in a list of dictionaries with a key and list of integers and returns a dictionary with the standard deviation of each list. Note: This should be done without using the NumPy built-in functions.</strong></blockquote><p><strong>Example:</strong></p><p><strong>Input:</strong></p><pre>input = [<br>    {<br>        &#39;key&#39;: &#39;list1&#39;,<br>        &#39;values&#39;: [4,5,2,3,4,5,2,3],<br>    },<br>    {<br>        &#39;key&#39;: &#39;list2&#39;,<br>        &#39;values&#39;: [1,1,34,12,40,3,9,7],<br>    }<br>]</pre><p><strong>Output:</strong></p><pre>output = {&#39;list1&#39;: 1.12, &#39;list2&#39;: 14.19}</pre><p>My Code:</p><pre>import math<br>def average(l): #O(n)<br>    return sum(l)/len(l)<br>def compute_deviation(list_numbers): #O(m)<br>    n = len(list_numbers)<br>    ans = {}<br>    for i in range(n):<br>        d = list_numbers[i]<br>        key = d[&#39;key&#39;]<br>        l = d[&#39;values&#39;]<br>        expected_val = average(l)<br>        var = average([(item-expected_val)*(item-expected_val) for item in l]) #O(n)<br>        ans[key]=math.sqrt(var)<br>    return ans<br>#TC: O(nm)<br># AS: O(m)</pre><blockquote><strong>23. Airbnb, Microsoft: Overall capital approval drops (85%→82%) but all individual products flat/up. Why?</strong></blockquote><blockquote><strong>Products Context (Capital Approval Example):</strong></blockquote><blockquote><strong>Product 1: Small Equipment ($5K-$20K) — 84%→85%</strong></blockquote><blockquote><strong>Product 2: Major Infrastructure ($500K+) — 77%→77%</strong></blockquote><blockquote><strong>Product 3: Software/Licenses ($20K-$100K) — 81%→82%</strong></blockquote><blockquote><strong>Product 4: Office Improvements ($10K-$50K) — 88%→88%</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*68GuveoQGnnNylqsTDt-pg.png" /></figure><blockquote><strong>24. Pinterest: Given an array of unsorted random numbers denoted </strong><strong>nums write a function </strong><strong>find_iqr to find the interquartile distance. The interquartile distance is defined by subtracting the first quartile<br>from the third quartile.</strong></blockquote><p><strong>Example:</strong></p><p><strong>Input:</strong></p><pre>nums = [3,4,9,12,14,239]</pre><p><strong>Output:</strong></p><pre>def find_iqr(nums) -&gt; 10</pre><p>Solution:</p><pre>def median(l):<br>    n = len(l)<br>    if n%2 != 0:<br>        return l[n//2]<br>    else:<br>        return (l[n//2] + l[n//2 - 1])/2<br><br><br>def find_iqr(a):<br>    a.sort()<br>    # If all elements are same<br>    if sum(a) == a[0]*len(a):<br>        return 0<br>    median_val = median(a)<br>    lower = [item for item in a if item &lt; median_val]<br>    upper = [item for item in a if item &gt; median_val]<br>    IQR = median(upper)-median(lower)<br>    return IQR</pre><blockquote><strong>25. Amazon: Write a function that checks if a list of integers is normally distributed. Specifically, given a list of </strong><strong>100 numbers, write a function that returns a score that measures the deviation from normality. i.e, a normally distributed list of integers would return </strong><strong>0. Use what is called the 68-95-99.7 rule of thumb for normal distributions.</strong></blockquote><pre>import numpy as np<br>def normality_check(l):<br>  arr = np.array(l)<br>  mean = np.mean(arr)<br>  std = np.std(arr,ddof=1) # Using sample standard deviation<br>  one_sigma_proportion = np.mean(np.abs(arr-mean)&lt;=std)<br>  two_sigma_proportion = np.mean(np.abs(arr-mean)&lt;=2*std)<br>  three_sigma_proportion = np.mean(np.abs(arr-mean)&lt;=3*std)<br>  score = np.abs(one_sigma_proportion-0.68) + np.abs(two_sigma_proportion-0.95) + np.abs(three_sigma_proportion-0.997)<br>  return score<br>  <br></pre><blockquote><strong>26. Two Sigma: Say you continually sample from some i.1.d. uniformly distributed (0, 1) random variables until the sum of the variables exceeds. How many samples do you expect to make?</strong></blockquote><p>Given Keep drawing from U(0,1) and summing. We need to find how many draws until sum &gt; 1.</p><p><strong>Example</strong></p><ul><li>Draw: 0.3, 0.5, 0.4 → Sum = 1.2 (3 draws)</li><li>Draw: 0.9, 0.6 → Sum = 1.5 (2 draws)</li><li><strong>What’s E[N]?</strong></li></ul><p><strong>Solution</strong></p><p><strong>Define:</strong> N_t = number of draws needed until sum &gt; t<br><strong>Find:</strong> m(t) = E[N_t], specifically m(1)</p><p><strong>Step 1: Set up recursion</strong></p><p>First draw = x. Two cases:</p><ul><li>If x &gt; t: Done! N_t = 1</li><li>If x &lt; t: Need N_t = 1 + N_(t-x) more draws</li></ul><p><strong>Step 2: Expected value formula</strong></p><pre>m(t) = 1 + ∫₀ᵗ m(t - x) dx</pre><p>(Average over all possible first draws)</p><p><strong>Step 3: Change variables</strong> (u = t — x)</p><pre>m(t) = 1 + ∫₀ᵗ m(u) du</pre><p><strong>Step 4: Differentiate both sides</strong></p><pre>m&#39;(t) = m(t)</pre><p>(Using fundamental theorem of calculus)</p><p><strong>Step 5: Solve differential equation</strong></p><ul><li>m’(t) = m(t) → solution is m(t) = eᵗ</li><li>Boundary: m(0) = 1 ✓</li></ul><p><strong>Step 6: Answer</strong></p><pre>m(1) = e ≈ 2.718 draws</pre><p><strong>Key Insight</strong></p><p>Recursive structure + continuous uniform → exponential solution</p><p><strong>Quick Check</strong></p><p>Average draw ≈ 0.5, so ~2 draws makes sense. With variability, e ≈ 2.7 is reasonable.</p><blockquote><strong>Law of Large Numbers</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*jsNF83M1pabS2AEEtvwIYg.png" /></figure><blockquote><strong>Central Limit theorem(CLT)</strong></blockquote><p>CLT: Sample means approach <strong>normal distribution</strong> as sample size ↑, regardless of original population distribution.</p><p><strong>Key Formula</strong></p><p>Sample mean: X̄ ~ N(μ, σ²/n) as n → ∞</p><p><strong>3 Conditions</strong></p><ol><li>n ≥ 30 (for non-normal populations)</li><li>Independent samples</li><li>Finite variance</li></ol><p><strong>Real-World Application</strong></p><p><strong>A/B Testing Example:</strong> Testing new feature impact on booking rates</p><ul><li>Each booking = Bernoulli variable</li><li>Many bookings → CLT applies</li><li>Enables hypothesis testing &amp; confidence intervals</li></ul><p><strong>Interview Questions</strong></p><blockquote><strong>1. Explain Central limit theorem and how it is useful to a technical person?</strong></blockquote><p><strong>Interview Memory Trick: “SIN-30”</strong></p><p><strong>S</strong> — <strong>Sample</strong> means become normal <strong>I</strong> — <strong>Independent</strong> samples required<br> <strong>N</strong> — <strong>N ≥ 30</strong> (sample size threshold)</p><p><strong>Quick Pitch Format:</strong></p><ol><li><strong>What</strong>: “Sample means → Normal distribution”</li><li><strong>Why useful</strong>: “Apply normal distribution tools to ANY data”</li><li><strong>Real use</strong>: “Powers A/B testing — e.g., testing if new Uber feature increases bookings”</li></ol><p><strong>One-liner to remember:</strong><br> <em>“CLT lets us use normal distribution math on averages, even when individual data points aren’t normal — that’s why A/B testing works at scale.”</em></p><blockquote><strong>2. Explain Central limit theorem and how it is useful to a non-technical person?</strong></blockquote><p><strong>Memory Trick: “SAMPLE Magic”</strong></p><p><strong>SA</strong> — <strong>Small samples</strong> work <strong>M</strong> — <strong>Many samples</strong> create pattern <strong>P</strong> — <strong>Predictable</strong> bell curve emerges <strong>L</strong> — <strong>Large populations</strong> understood <strong>E</strong> — <strong>Error margins</strong> calculable</p><p><strong>Quick Pitch Format:</strong></p><ol><li><strong>What</strong>: “Averages from repeated samples always form a bell curve”</li><li><strong>Why care</strong>: “Small samples reliably represent huge populations”</li><li><strong>Real impact</strong>: “Why 1,000-person polls predict elections &amp; 100-patient studies guide medicine for millions”</li></ol><p><strong>One-Liner:</strong></p><p><em>“CLT is why testing 100 products can tell you about 1 million products, and why polling 1,000 voters predicts elections — it’s the math that makes small samples trustworthy.”</em></p><p><strong>Simple Analogy:</strong> “It’s like tasting one spoonful of soup to know if the whole pot needs salt — sampling works because averages are predictable.”</p><blockquote><strong>3. Google: What does it mean for an estimator to be unbiased? What about consistent? Give examples of an unbiased but not consistent and biased but consistent estimator.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*1x26ghNrIs6Qd58HZun3xg.png" /></figure><blockquote><strong>Hypothesis Testing</strong></blockquote><p><strong>Core Concept</strong></p><p>Test if sample data supports a hypothesis by either <strong>rejecting</strong> or <strong>failing to reject</strong> the null hypothesis (H₀).</p><p><strong>3-Step Process</strong></p><ol><li><strong>State hypotheses</strong>: H₀ (null) vs H₁ (alternative)</li><li><strong>Calculate test statistic</strong> → get p-value</li><li><strong>Compare p-value to α</strong> (usually 0.05)</li></ol><p><strong>Goal</strong>: Reject H₀ with statistical significance</p><p><strong>Hypothesis Test Types</strong></p><ul><li><strong>One-tailed</strong>: Testing direction (left or right)</li><li><strong>Two-tailed</strong>: Testing difference (both directions)</li></ul><p><strong>Left- Tailed Hypothesis testing</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*3VFAyczbUJqHzfC57cytUg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*v6hB9XCJ0K2KzT_tivbd_A.png" /></figure><p><strong>Right- Tailed Hypothesis testing</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*ryw7u1vEm55vsjw3YFcO1w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*KObrWlIO8GtIasLUdXhQzg.png" /></figure><p><strong>Two Tailed Hypothesis testing</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*jVJ2YjHE1F6ufFIfQkTvww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*FCvzr7NS0CgUdLQNUd8QSw.png" /></figure><blockquote><strong>Test Statistics — When to Use What</strong></blockquote><p><strong>Z-Test (Normal Distribution)</strong></p><p><strong>Use when:</strong></p><ul><li>Known population σ + normal population (any n)</li><li>Known population σ + non-normal + n ≥ 30</li><li><strong>Proportions</strong>: np₀ ≥ 10 AND n(1-p₀) ≥ 10</li></ul><p><strong>Formula</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/217/1*lqL-dWjQO8k93eIgA08zNg.png" /></figure><p><strong>t-Test (Student’s t-Distribution)</strong></p><p><strong>Use when:</strong></p><ul><li>Unknown σ (use sample s) + normal population (any n)</li><li>Unknown σ + non-normal + n ≥ 30</li></ul><p><strong>Formula</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/271/1*Z6SettKhs7cj4OwTf6qRpA.png" /></figure><p>with df = n-1</p><p><strong>Note</strong>: t-distribution has heavier tails than normal</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OlNzYlZm0hLUw3UQu74v7g.png" /></figure><p><strong>Chi-Squared Test</strong></p><p><strong>Use for:</strong></p><ul><li>Goodness of fit</li><li>Independence of categorical variables</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*og8MEaagHZauhy9QVLOQeg.png" /></figure><p><strong>Example for chi-squared test for goodness of fit</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yof4avERuy1iCquBLysQ0A.png" /></figure><p><strong>Example for chi-squared test for Independence</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*FWsz2GPgxouTmj1-qoBAbQ.png" /></figure><p>⚠️ <strong>Both Z &amp; t are INVALID when: </strong>Small sample (n &lt; 30) + non-normal population → Use <strong>non-parametric tests</strong> (Wilcoxon, Mann-Whitney).</p><blockquote><strong>Hypothesis Testing for population proportions</strong></blockquote><ul><li>Each member = Bernoulli variable (1 or 0)</li><li><strong>Sample proportion</strong>:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/244/1*Evetx3JZLEC9BLdYzPkeqg.png" /></figure><ul><li><strong>Small n</strong>: Use exact Binomial test</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*RepnXnHB4YJ3iZ7x4IYrmg.png" /></figure><ul><li><strong>Large n</strong>: Use z-test (when np ≥ 10 AND n(1-p) ≥ 10)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/202/1*YRhVE1hPThL5H5e0y6bqXA.png" /></figure><p>Thus, we can perform a <strong>z-test</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/130/1*RPrl6S5Mf4-k-UGAYejD3w.png" /></figure><p>where p ​ is the population proportion under the null hypothesis.</p><blockquote><strong>p-values and confidence intervals</strong></blockquote><p><strong>p-value: </strong>Probability of observing data at least as extreme as what we have, assuming the null hypothesis (H₀) is true.</p><ul><li><strong>p &lt; α</strong>: Reject H₀</li><li><strong>p ≥ α</strong>: Fail to reject H₀</li></ul><p><strong>Confidence Intervals (1-α)%: </strong>It is a range of values, derived from sample data, that is likely to contain the true population parameter with a certain confidence level</p><p>Alternative way to test: Check if claimed parameter is inside CI</p><ul><li><strong>Outside CI</strong>: Reject H₀</li><li><strong>Inside CI</strong>: Fail to reject H₀</li></ul><p>For population mean when σ is known (Z-interval):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/196/1*VjhQVqVNgYnPO91Fb7Rxyg.png" /></figure><p>For population mean when σ is unknown (t-interval):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/231/1*b4Jv4adQGNPwG5wiIBhwtQ.png" /></figure><p>For population proportion:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*Aap4lHjHsOYlmzVVBzkKJw.png" /></figure><blockquote><strong>Type I and Type II Errors</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*Ktox5decEPp2NNEVG6IFvA.png" /></figure><ul><li><strong>α (Significance)</strong>: P(Type I error) = False Positive</li><li><strong>β</strong>: P(Type II error) = False Negative</li><li><strong>Confidence Level (1-α)</strong>: Probability result is correct (~95%)</li><li><strong>Power (1-β)</strong>: Probability of detecting real effect (~80%)</li></ul><p><strong>Set before test</strong>: High confidence (0.95) + High power (0.80)</p><p><strong>Bonferroni Correction</strong></p><p>When running <strong>n</strong> hypothesis tests:</p><ul><li>Use adjusted α: <strong>α_new = α / n</strong></li><li>Controls overall false positive rate</li><li>Example: 100 tests with α=0.05 → use α=0.0005 for each test</li></ul><p><strong>Quick Decision Tree</strong></p><ol><li><strong>Categorical variables?</strong> → Chi-squared</li><li><strong>Testing proportion?</strong> → Check np ≥ 10 → Z-test</li><li><strong>Testing mean:</strong></li></ol><ul><li>Know σ? → Z-test (if n≥30 or normal)</li><li>Don’t know σ? → t-test (if n≥30 or normal)</li><li>Small n + non-normal? → Non-parametric test</li></ul><p><strong>Key Interview Points</strong></p><p>✓ <strong>A/B testing</strong> uses hypothesis testing framework<br> ✓ Always state H₀ before testing<br> ✓ p-value ≠ probability H₀ is true<br> ✓ “Fail to reject” ≠ “accept” H₀<br> ✓ Larger samples → more power → detect smaller effects</p><p><strong>INTERVIEW QUESTIONS:</strong></p><blockquote><strong>1. HYPOTHESIS TESTING(Technical Vs Non Technical)</strong></blockquote><p><strong>Technical</strong></p><ul><li>Tests claims about populations using sample data</li><li><strong>H₀ (null)</strong>: no effect/difference</li><li><strong>H₁ (alternative)</strong>: effect exists</li><li>Calculate test statistic → compare p-value to α (0.05)</li><li><strong>Decision</strong>: Reject H₀ if p &lt; α</li></ul><p><strong>Non-Technical</strong></p><ul><li>“Reality check” to see if results are real or just random chance</li><li>Start assuming “no difference” and use data to prove otherwise</li></ul><p><strong>Use Cases: </strong>A/B testing , Data-driven decisions</p><blockquote><strong>2. P-VALUE (Technical Vs Non Technical)</strong></blockquote><p><strong>Technical</strong></p><ul><li>Probability of observing data ≥ extreme, assuming H₀ is true</li><li><strong>Small p-value (&lt; 0.05)</strong>: Strong evidence against H₀ → reject</li><li><strong>Large p-value</strong>: Consistent with H₀ → fail to reject</li></ul><p><strong>Non-Technical</strong></p><ul><li><strong>“Trust score”</strong> — shows if results are meaningful or just noise</li><li>Low p-value (&lt; 5%): Unlikely by chance → trust it</li><li>High p-value: Could be random → not enough proof</li></ul><p><strong>Use Cases: </strong>Quantifies evidence strength , A/B testing decisions , Statistical significance</p><blockquote><strong>3. CONFIDENCE INTERVAL (Technical Vs Non Technical)</strong></blockquote><p><strong>Technical</strong></p><ul><li>Range likely containing true parameter with X% confidence (e.g., 95%)</li><li>If repeated sampling, ~95% of CIs contain true value</li><li><strong>Parameter is fixed; interval varies</strong></li></ul><p><strong>Non-Technical</strong></p><ul><li>Range showing where true value likely sits (with uncertainty)</li><li><strong>Narrower CI = more precise</strong></li><li>Example: Height 170–175cm → true average likely in that range</li></ul><p><strong>Use Cases: </strong>Shows uncertainty beyond point estimates , Complements hypothesis tests and A/B uplift ranges</p><blockquote>4. <strong>TYPE I ERROR (α) — FALSE POSITIVE (Technical Vs Non Technical)</strong></blockquote><p><strong>Technical</strong></p><ul><li>Reject true H₀ (detect effect when none exists)</li><li>Controlled by significance level α (typically 0.05)</li><li><strong>1-α = confidence level</strong></li></ul><p><strong>Non-Technical</strong></p><ul><li>Saying there’s a difference when there really isn’t</li><li><strong>“Crying wolf” — false alarm</strong></li></ul><p><strong>More costly when:</strong></p><ul><li>Drug approval (approving harmful drug)</li><li>Fraud detection (annoying customers)</li><li>Judicial system (convicting innocent)</li></ul><blockquote><strong>5. TYPE II ERROR (β) — FALSE NEGATIVE (Technical Vs Non Technical)</strong></blockquote><p><strong>Technical</strong></p><ul><li>Fail to reject false H₀ (miss real effect)</li><li>Given by β</li><li><strong>1-β = statistical power</strong> (typically 0.8)</li></ul><p><strong>Non-Technical</strong></p><ul><li>Missing a real difference that actually exists</li><li><strong>“Wolf was there!” — missing the signal</strong></li></ul><p><strong>More costly when:</strong></p><ul><li>Cancer diagnosis (missing disease)</li><li>Security systems (missing intruder)</li><li>Spam filter (missing malicious email)</li></ul><p><strong>Type I and Type II are costly when:</strong></p><ul><li>Self-driving cars</li></ul><blockquote><strong>6. Facebook: Say you flip a coin 10times and observe only one head. What would be your null hypothesis and p-value for testing whether the coin is fair or not.</strong></blockquote><p><strong>Hypotheses:</strong></p><ul><li>H₀: p = 0.5 (fair coin)</li><li>H₁: p ≠ 0.5 (unfair coin)</li><li>Two-tailed test</li></ul><p><strong>Check Distribution:</strong></p><ul><li>np₀ = 10(0.5) = 5</li><li>n(1-p₀) = 10(0.5) = 5</li><li>Both &lt; 10 → Cannot use normal approximation (Z-test)</li><li>Use binomial distribution directly</li></ul><p><strong>P-value Calculation:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C9ZwmZ5QIVWzCd_yoPfwpg.png" /></figure><p><strong>Decision:</strong> p-value (0.0195) &lt; α (0.05) → Reject H₀</p><p><strong>Conclusion:</strong> Evidence that coin is unfair</p><p><strong>Key Point:</strong> Small sample + discrete data → use exact binomial, not normal approximation</p><blockquote><strong>7. Microsoft: Explain the statistical background behind power.</strong></blockquote><p>Power is the probability of detecting a real effect — it’s your experiment’s sensitivity.</p><p><strong>Key equation:</strong> Power = 1 — P(Type II error)</p><p><strong>4 drivers (ESSS):</strong></p><ul><li><strong>E</strong>ffect size (bigger effects easier to detect)</li><li><strong>S</strong>ample size (more data = more power)</li><li><strong>S</strong>ignificance level α (trade-off with false positives)</li><li><strong>S</strong>igma/variance (less noise = more power)</li></ul><p><strong>Practical use:</strong> In A/B testing, power analysis determines if we have enough users to confidently detect meaningful improvements — typically we aim for 80% power.</p><p><strong>One-liner:</strong> Power tells you whether your experiment is large enough to catch real effects — low power means you might miss important findings even when they exist.</p><blockquote><strong>8. Amazon: Say you are testing hundreds of hypotheses, each with t-test, What considerations would you take into account when doing this?</strong></blockquote><p>With many tests, false positives ↑ due to chance alone</p><p><strong>Example:</strong></p><ul><li>100 tests, α = 0.05</li><li>Expected false positives = 100 × 0.05 = 5 tests are statistically significant just by chance</li><li>High probability of Type I error (false positive)</li></ul><p><strong>Solution: Bonferroni Correction</strong></p><p>Adjusted significance level = α/m (m = number of tests)</p><p><strong>Example:</strong></p><ul><li>α = 0.05, m = 100</li><li>New threshold = 0.05/100 = 0.0005</li><li>Much stricter to reject H₀</li></ul><p><strong>Trade-off:</strong></p><p>✓ Reduces Type I error (false positives)<br> ✗ Increases Type II error (false negatives) — may miss real effects</p><p><strong>Interview Tip:</strong> Always mention the trade-off between Type I and Type II errors when discussing multiple testing corrections.</p><blockquote><strong>9. Google: How would you derive a confidence interval for the probability of flipping heads for a series of coin tosses?</strong></blockquote><p>Proportion of heads for a series of coin tosses is a sampling distribution of bernoulli outcomes, and for larger n it follows normal distribution. Lets say we have a sample and the proportion of heads for this sample is p and now we need to construct confidence interval for this sample which is given by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*jnTUfFbzXonjLcuO2fdUIQ.png" /></figure><blockquote><strong>10. D.E Shaw: A coin was flipped 1,000 times and 550 times it shows heads. Do you think the coin is biased? Why or Why not?</strong></blockquote><p>Lets first setup Hypothesis testing for testing whether coin is biased or unbiased.</p><p>Null Hypothesis(H0): Coin is unbiased=&gt; probability of head(p0) = 0.5</p><p>Alternate Hypothesis(H1): Coin is biased=&gt; probability of head(p0) ≠ 0.5</p><p>proportion of heads for the sample p = 550/1000 = 0.55</p><p>Compute total number of successes and total number of failures under null hypothesis</p><p>np0=1000*0.5 = 500 and n(1-p0) = 1000*0.5 = 500</p><p>As number of successes and number of failures are greater than 10 so test static follows normal distribution and we can use z-test to compute p-value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*EjL4lkP7KTnjkuUCRZ0Qnw.png" /></figure><p>For z = 3.16 and α = 0.05 from z-distribution table we get</p><p>P(X ≤ 3.16) = 0.99918</p><p>For two tailed test p-value = 2*P(X&gt;3.16) = 2*(1–0.99918) = 0.00164 &lt; α = 0.05</p><p>Therefore as p-value is less than signicificance level we can reject null hypothesis and there is evidence that coin is likely biased.</p><blockquote><strong>16. Morgon Stanley: Say you have an unfair coin which lands on heads 60% of the time. How many coin flips are needed to detect the coin is unfair.</strong></blockquote><p>The proportion of heads for the sample = p = 0.6</p><p>Lets say we have larger number of coin flips and test statistic follows normal distribution and we can use z-test.</p><p>So lets compute CI for the given sample</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*RMgEqQWhrmf9pESfGOixUA.png" /></figure><p>Generally the hypothesis test setup for testing whether coin is fair or not is</p><p>Null Hypothesis(H0): Coin is fair=&gt; probability of head(p0) = 0.5</p><p>Alternate Hypothesis(H1): Coin is not fair=&gt; probability of head(p0) ≠ 0.5</p><p>So to coin to be unfair the calculated CI of sample should not contain p0, so the lower bound of CI should be greater than 0.5</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*UM-OJhBbkmRqpj7Qt3Q4PA.png" /></figure><blockquote><strong>17. You are given an array of numerical values, number of bootstrap samples to consider, and size for a confidence interval. Write a function that performs bootstrap sampling on the given array and calculates the confidence interval based on the given size.</strong></blockquote><blockquote><strong><em>Note: The function should return a tuple containing the minimum and maximum values of the confidence interval rounded to the tenths place.</em></strong></blockquote><p><strong>Example</strong></p><p><strong>Input:</strong></p><pre>values = [1, 2, 3, 4, 5]</pre><p><strong>Output</strong></p><pre>bootstrap_conf_interval(values, 1000, 0.95) -&gt; (1.2, 4.8)</pre><p>In this case, the function returns a tuple indicating that we are 95% confident that the population parameter lies between 1.2 and 4.8 based on our bootstrap samples.</p><p>Solution:</p><pre>import random<br>import numpy as np<br>def bootstrap_conf_interval(data, num_samples, conf_level):<br>    # Convert to numpy array for easier manipulation<br>    data = np.array(data) <br>    n = len(data)<br>    # Store bootstrap statistics (means in this case)<br>    bootstraped_means = []<br>    # Generate bootstrap samples and calculate means<br>    for i in range(num_samples):<br>        # Sample with replacement<br>        bootstraped_data = np.random.choice(data, size=n, replace = True)<br>        # Calculate mean of this bootstrap sample<br>        bootstraped_means.append(np.mean(bootstraped_data))<br>    bootstraped_means.sort()<br>    bootstraped_means = np.array(bootstraped_means)<br>    alpha = 1-conf_level<br>    # Get confidence interval bounds<br>    lower_bound = np.percentile(bootstraped_means, alpha/2*100)<br>    upper_bound = np.percentile(bootstraped_means, (1-alpha/2)*100)<br>    # Round to tenths place<br>    return (round(lower_bound, 1), round(upper_bound,1))</pre><blockquote><strong>18. Google, Apple, Walmart: Let’s say we have a sample size of n. The margin of error for our sample size is 3. How many more observations should be added to sample or how much sample size should be increased so that we can decrease the margin of error to 0.3?</strong></blockquote><p>First, let me clarify that I need to find how many additional observations should be added to the sample (i.e., increase the sample size n) to reduce the margin of error from 3 to 0.3. I assume the sample size is larger than 30, and we know the population standard deviation, and for this particular sample, we can construct a confidence interval using the Z-score. Therefore, the confidence interval is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/196/1*VjhQVqVNgYnPO91Fb7Rxyg.png" /></figure><p>From this equation, we can see that the margin of error is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*yEcDykYnOtUW7y3W3RkvOg.png" /></figure><p>So the sample size should be increased by 99n additional observations to reduce the margin of error from 3 to 0.3. The intuition behind this result aligns with the Law of Large Numbers: as sample size increases, the sample mean approaches the population mean, and the margin of error decreases.</p><blockquote><strong>19. A survey asked 100 respondents two questions:</strong></blockquote><blockquote><strong><em>a. Do you like tea?</em></strong></blockquote><blockquote><strong><em>b. Do you like coffee?</em></strong></blockquote><blockquote><strong>70% of respondents said they liked coffee, and 80% of the respondents said they liked tea. Only 10% of respondents said they liked neither.</strong></blockquote><blockquote><strong>Based on this survey, what are the upper and lower bounds for the proportion of the population that like tea and coffee?</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zs3Xy2hzvqqqN2_9V9AaDw.jpeg" /></figure><blockquote><strong>20. Tesla, United Airlines: Suppose there exists a new airline named Jetco that flies domestically across North America. Jetco recently had a study commissioned that tested the boarding time of every airline, and it came out that Jetco had the fastest average boarding times of any airline. What factors could have biased this result, and what would you look into?</strong></blockquote><p><strong>Potential Biases and Areas to Investigate:</strong></p><p><strong>Sample and Study Design Issues:</strong></p><ul><li><strong>Sample size and selection</strong>: Was the sample representative? Did they test all routes or cherry-pick favorable ones?</li><li><strong>Time period</strong>: Was data collected during off-peak seasons, avoiding holidays or weather delays?</li><li><strong>Study methodology</strong>: Who conducted the study? Was it internally funded by Jetco, creating potential conflicts of interest?</li></ul><p><strong>Operational Factors:</strong></p><ul><li><strong>Aircraft size</strong>: Jetco might operate smaller planes that naturally board faster</li><li><strong>Route characteristics</strong>: Shorter domestic routes vs. international connections that other airlines handle</li><li><strong>Passenger demographics</strong>: Business travelers vs. families with children</li><li><strong>Airport infrastructure</strong>: Jetco might primarily use smaller, less congested airports</li></ul><p><strong>Definition and Measurement Issues:</strong></p><ul><li><strong>What constitutes “boarding time”</strong>: Door opening to door closing? First passenger to last? This definition could favor certain operational models</li><li><strong>Baseline comparisons</strong>: Were all airlines measured under identical conditions?</li><li><strong>Outlier handling</strong>: How were delays, cancellations, or unusual circumstances treated?</li></ul><p><strong>Business Model Differences:</strong></p><ul><li><strong>Service level</strong>: Jetco might have fewer amenities, carry-on restrictions, or simplified boarding processes</li><li><strong>Pricing strategy</strong>: Higher prices might attract passengers who travel lighter or are more experienced.</li></ul><blockquote>21. Meta: You are given a dataframe with a single column, &#39;var&#39;. Calculated the t-value for the mean of ‘var’ against a null hypothesis that μ=μ0​. <em>Note: You do not have to calculate the p-value of the test or run the test.</em></blockquote><p><strong>Example:</strong></p><p><strong>Input:</strong></p><pre>mu0 = 1<br>print(df)<br>...<br>   var<br>0  -34<br>1   40<br>2  -89<br>3    5<br>4  -26</pre><p><strong>Output:</strong></p><pre>def t_score(mu0, df) -&gt;<br>var   -1.015614<br>dtype: float64</pre><p><strong>Solution:</strong></p><pre>import pandas as pd<br>import numpy as np<br>def t_score(mu0, df: pd.DataFrame):<br>    return [((df[&#39;var&#39;].mean()-mu0)/((df[&#39;var&#39;].std())/np.sqrt(df.shape[0])))]</pre><blockquote><strong>22. Google: You’re given a two-sided coin that could be fair or biased. How would you figure out whether or not the coin is biased?</strong></blockquote><blockquote><strong>Design a test and describe which outcome would tell you that the coin is biased</strong></blockquote><p>I would use a statistical hypothesis test to determine if the coin is biased. Here’s my systematic approach:</p><p><strong>Step 1: Set Up the Test</strong></p><ul><li><strong>Significance Level</strong>: α = 0.05 (95% confidence)</li><li><strong>Null Hypothesis (H₀)</strong>: The coin is fair, p = 0.5</li><li><strong>Alternative Hypothesis (H₁)</strong>: The coin is biased, p ≠ 0.5 (two-tailed test)</li></ul><p><strong>Step 2: Collect Data</strong></p><p>Flip the coin <strong>n times</strong> (Recommend at least 100 flips for good statistical power) and count the number of heads.</p><p><strong>Step 3: Calculate Test Statistic</strong></p><p>Let p̂ = observed proportion of heads</p><p><strong>For large samples (n ≥ 30)</strong>, use z-test:</p><pre>z = (p̂ - 0.5) / √(0.25/n)</pre><p><strong>For small samples</strong>, use exact binomial probability.</p><p><strong>Step 4: Make Decision</strong></p><p>Calculate the p-value and compare to α = 0.05:</p><ul><li>If p-value &lt; 0.05: <strong>Reject H₀</strong> → Coin is biased</li><li>If p-value ≥ 0.05: <strong>Fail to reject H₀</strong> → Insufficient evidence of bias</li></ul><p><strong>Concrete Example</strong></p><p>Suppose we flip 100 times and get 65 heads:</p><ul><li>p̂ = 0.65</li><li>z = (0.65–0.5) / √(0.25/100) = 0.15 / 0.05 = 3.0</li><li>p-value ≈ 0.003</li><li>Since 0.003 &lt; 0.05, we <strong>reject the null hypothesis</strong></li><li><strong>Conclusion</strong>: The coin is significantly biased toward heads</li></ul><p>Key Insight</p><p>The further our observed proportion is from 0.5, the stronger the evidence of bias. With sufficient sample size, even small deviations from 50% can be detected as statistically significant.</p><blockquote><strong>23. Uber: Say you are given a random Bernoulli trial generator. How would you generate values from a standard normal distribution?</strong></blockquote><p>Given a Bernoulli trial generator, generate values from standard normal N(0,1).</p><p><strong>Step 1: Choose parameters</strong></p><ul><li>Pick probability p (recommend p = 0.5)-&gt; sucess probability for a single bernoulli trial</li><li>Pick sample size n (recommend n ≥ 100)-&gt; n bernoulli trials</li><li>Ensure: np ≥ 10 and n(1-p) ≥ 10-&gt; for n bernoulli trials the number of heads(success) and number of tails(failures) should be atleast 10</li></ul><p><strong>Step 2: Generate n Bernoulli trials</strong></p><pre>x₁, x₂, ..., xₙ where each xᵢ ∈ {0,1}</pre><p><strong>Step 3: Calculate sample proportion</strong></p><pre>p̂ = (1/n)Σxᵢ = (number of successes)/n</pre><p><strong>Step 4: Standardize using CLT</strong></p><pre>Z = (p̂ - p) / √(p(1-p)/n)</pre><p><strong>Step 5: Result</strong></p><ul><li>Z is one value from N(0,1)</li><li>Repeat Steps 2–4 for more values</li></ul><p><strong>Why It Works</strong></p><ul><li>By CLT: p̂ ~ N(p, p(1-p)/n) for large n</li><li>Standardization: Z = (X — μ)/σ converts to N(0,1)</li></ul><p><strong>Example</strong></p><p>p = 0.5, n = 100, got 53 heads:</p><pre>p̂ = 53/100 = 0.53<br>Z = (0.53 - 0.5) / √(0.5×0.5/100)<br>Z = 0.03 / √0.0025<br>Z = 0.03 / 0.05 = 0.6</pre><p>This 0.6 is one standard normal sample.</p><p><strong>Key Point</strong></p><p>Each set of n trials → ONE standard normal value (not n values)</p><blockquote><strong>MLE &amp; MAP</strong></blockquote><p><strong>MLE (Maximum Likelihood Estimation):</strong></p><ul><li>Estimate parameter θ by maximizing likelihood L(θ)=P(data∣θ).</li><li>Ignores prior knowledge.</li><li>Steps: Use log-likelihood → differentiate → solve for θ.</li></ul><p><strong>MAP (Maximum A Posteriori):</strong></p><ul><li>Estimate parameter θ by maximizing posterior P(θ∣data)∝P(data∣θ)P(θ).</li><li>Combines data (likelihood) + prior belief.</li><li>Steps: Use log-posterior → differentiate → solve for θ.</li></ul><p><strong>Key difference:</strong></p><ul><li><strong>MLE:</strong> only likelihood (data-driven).</li><li><strong>MAP:</strong> likelihood + prior (data + belief).</li></ul><p><strong>Intuition:</strong></p><ul><li><strong>MLE:</strong> “Which parameter makes my data most likely?”</li><li><strong>MAP:</strong> “Which parameter is most probable given my data <strong>and</strong> prior belief?”</li></ul><p><strong>Core Equations</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*sMKK-dw3lQMWqdYReN0gIw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/1*xA2_vtv8rjd5BcAWqxn-5Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*Fs5nlgKHMJVhQV381gcopA.png" /></figure><p><strong>Interview Questions:</strong></p><blockquote><strong>1. Spotify: Say you draw n samples from a uniform distribution U(a, b). What are the MLE estimates of a and b?</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D3G1dGJUc5ELsOK4L-E-Cg.png" /></figure><blockquote><strong>2. Netlifx: What are MLE and MAP? What is the difference between two?</strong></blockquote><p>Refer above concept</p><blockquote><strong>3. Tesla: Say you have N independent and identically distributed draws of an exponential random variable. What is the best estimator for the parameter</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*36KjQkpFLBN4IC3pqlRyMQ.png" /></figure><blockquote><strong>4. Spotify, Jane Street: Given N samples from a uniform distribution [0,d], how would you estimate d?</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tKONelKJO6qg_21XRT57ng.png" /></figure><blockquote><strong>Moment Generation Function(MGF)</strong></blockquote><ul><li>The MGF of a random variable X is defined as:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/1*zR6h8pg6QKKHEmgZZXCCHQ.png" /></figure><ul><li>MGF → compact representation of a distribution (its “fingerprint”).</li></ul><p><strong>Why useful?</strong></p><ul><li><strong>Generates moments:</strong> kth derivative at s=0 gives E[X^k]</li><li><strong>Uniqueness:</strong> Each distribution has a unique MGF.</li><li><strong>Simplifies calculations:</strong> Makes complex probability calculations much simpler.</li></ul><p><strong>Example (Fair Coin, X=1 for heads, 0 for tails):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*x8gY47cR180Lb-HpbVTGgw.png" /></figure><ul><li>Mean =&gt; E[X] = M’(0)=0.5*e⁰ = 0.5</li><li>Variance=&gt;Var[X] = E[X²]-E[X]² = M’’(0)-M’(0) = 1/2–1/4 = 1/4 = 0.25</li></ul><p><strong>Interview Questions:</strong></p><blockquote><strong>1. Morgon Stanley: Describe what a moment generating function(MGF) is. Derive the MGF for a normally distributed random variable X.</strong></blockquote><p>Ans:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B_SKSS6E2TW_74ssqEzZ9w.jpeg" /></figure><blockquote><strong>2. Citadel: Assume that log X ~ N(0, 1). What is the expectation of X?</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*DcCedNPwt92vFaToyjM51g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cdaca1fcc5ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>