<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dharma on Medium]]></title>
        <description><![CDATA[Stories by Dharma on Medium]]></description>
        <link>https://medium.com/@dharmatejasamudrala?source=rss-c037363a37db------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8boWm4_WVLZpYILjmzwiuw.jpeg</url>
            <title>Stories by Dharma on Medium</title>
            <link>https://medium.com/@dharmatejasamudrala?source=rss-c037363a37db------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 17:11:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dharmatejasamudrala/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Spent 50 Hours Mastering AWS Glue and Spark. So You Don’t Have To]]></title>
            <link>https://medium.com/towards-data-engineering/i-spent-50-hours-mastering-aws-glue-and-spark-so-you-dont-have-to-a895ea801097?source=rss-c037363a37db------2</link>
            <guid isPermaLink="false">https://medium.com/p/a895ea801097</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Dharma]]></dc:creator>
            <pubDate>Thu, 12 Feb 2026 19:23:16 GMT</pubDate>
            <atom:updated>2026-02-13T10:21:12.185Z</atom:updated>
            <content:encoded><![CDATA[<h3>I Spent 50 Hours Mastering Glue and Spark. So You Don’t Have To</h3><p>When I first started diving deep into AWS data engineering, I kept running into the same question over and over: should I use AWS Glue or Apache Spark? Everyone seemed to have opinions, but nobody had a clear framework for actually deciding. So I spent the last 50 hours building pipelines with both, reading documentation, and understanding when each one really makes sense. Here’s what I actually discovered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*82a5z_zC7ZVolXjz" /></figure><h3>The Thing Nobody Tells You About This Decision</h3><p>Most resources treat Glue and Spark like they’re fundamentally different tools competing for the same job. They’re not. What they really are is different approaches to solving the same problem, and understanding that distinction changes how you evaluate them. Glue is AWS’s managed ETL service built on top of Spark. Spark is the open source engine underneath. This means you’re not really choosing between two completely different technologies .you’re choosing between letting AWS manage it for you or managing it yourself.</p><p>That distinction matters way more than any feature comparison can tell you.</p><h3>What I Learned About AWS Glue</h3><p>AWS Glue feels like the path of least resistance when you’re starting out. You connect your data sources through the Glue console, define your transformations, and it handles the infrastructure. No cluster management, no thinking about worker nodes, no capacity planning. You write your code once, and Glue figures out the compute.</p><p>But here’s where things get interesting: Glue is brilliant for certain types of work and honestly mediocre for others. If you’re running standard ETL jobs .Reading from S3, transforming CSV or Parquet files, writing back to S3 or a data warehouse .Glue feels almost magical. The job bookmarks feature alone is worth understanding because it lets you track which data you’ve already processed, so you don’t reprocess the same files over and over. That’s a huge deal for cost management.</p><p>Where Glue starts to feel limiting is when you need control. You can’t easily customize the Spark configuration. You can’t easily debug performance issues the way you might with a Spark cluster you own. The pricing model is also something to really understand. You pay per DPU (Data Processing Unit) per second, and Glue will automatically scale up your workers based on the data it detects. That sounds great until you realize it scales up aggressively, and suddenly your cost per job explodes.</p><p>The real strength of Glue isn’t the technology ,it’s the operational simplicity. If your team is small, if you don’t have people who want to spend time managing Spark clusters, Glue removes that burden. You trade flexibility and fine-tuned control for “it just works.”</p><h3>What I Learned About Spark</h3><p>Spark is the opposite experience. You have to think about everything. How many workers do you want? What instance types? How much memory per executor? How do you tune your shuffle operations to avoid data skew? Should you use Spark on EC2, EMR, Databricks, or somewhere else? The flexibility is incredible, but it comes with responsibility.</p><p>The thing that surprised me most about Spark is how much it rewards understanding. Once you actually know what’s happening under the hood how Spark distributes data, how it handles shuffles, why certain operations are expensive .You can write code that runs dramatically faster and cheaper than the alternative. You can profile your jobs, see exactly where time is being spent, and optimize ruthlessly.</p><p>EMR, Amazon’s managed Spark offering, sits in an interesting middle ground. You still manage the cluster in some sense, but AWS handles a lot of the infrastructure work. It’s more hands-on than Glue but less hands-on than running Spark on raw EC2 instances. The pricing is more predictable than Glue’s dynamic scaling because you pay for the instances you provision, not for the compute actually consumed.</p><p>The cost advantage of Spark really emerges when you’re running complex transformations at scale. The community has spent years optimizing Spark’s query execution engine, and if you write your code well, it’s genuinely fast and efficient. The downside is that writing code well requires understanding and experience. It’s not a beginner-friendly tool.</p><h3>The Real Comparison Nobody Makes</h3><p>Most articles compare Glue and Spark by talking about features. They miss the actual decision framework that matters. Here’s what you should actually think about:</p><p>The first question is operational complexity. Do you want to spend time managing infrastructure and tuning performance, or would you rather have AWS handle that? If the answer is “I’d rather AWS handle it,” you’re looking at Glue. If the answer is “I want control and I’m willing to invest the time,” Spark (probably on EMR) makes sense.</p><p>The second question is cost at your scale. For small to medium pipelines, Glue’s per-DPU-second pricing often looks reasonable. You’re not paying for idle time, and for simpler jobs, the automatic scaling actually works in your favor. But as pipelines get more complex or you run them more frequently, Glue costs can balloon surprisingly fast. Spark’s model becomes more attractive because you control exactly what you’re paying for.</p><p>The third question is what your team knows. If you already have Spark expertise, running Spark is cheaper because your team can optimize effectively. If your team is SQL-focused or infrastructure-averse, Glue’s simpler mental model makes more sense. There’s real value in using tools your team understands deeply rather than tools that are technically better but require learning curves.</p><p>The fourth question is the nature of your work. If you’re building data warehouses, doing heavy transformations, or working with complex distributed data operations, Spark is probably the right choice. If you’re doing one-off ETL jobs, incremental loads, or integration work, Glue often makes more sense.</p><blockquote><strong>I Spent 60 Hours Breaking AWS Pipelines to Prep for Data Engineering Interviews. Here’s What Interviewers Actually Care About</strong></blockquote><h3>How to Actually Decide</h3><p>When you’re actually facing this decision, start by asking yourself what you’re optimizing for. Are you optimizing for operational simplicity? Start with Glue, and only move to Spark if you run into limitations. Are you optimizing for cost at scale? Build the pipeline in Spark on EMR and measure actual costs. Are you optimizing for team velocity? Choose based on what your team already knows.</p><p>The second thing to do is prototype in both if you have the time. Spend a few hours building your actual use case in Glue and a few hours building it in Spark. The experience of actually doing the work beats any written comparison because you’ll feel the friction points yourself. You’ll notice which tool makes your specific problem easier or harder.</p><p>The third thing is to know what you’re sacrificing. If you choose Glue for simplicity, you’re sacrificing fine-tuned control and potentially cost efficiency at scale. If you choose Spark for power and control, you’re sacrificing operational simplicity and ease of getting started. Both choices are legitimate. Neither is universally better.</p><h3>What This Actually Means for You</h3><p>The deeper insight from spending all this time with both tools is that Glue and Spark aren’t really in competition. They’re different solutions to different problems. Glue wins when operational simplicity matters more than anything else. Spark wins when you need control or when you’re running workloads complex enough that optimization becomes economical.</p><p>Most data engineering teams end up using both. They use Glue for simpler ETL jobs where the infrastructure management overhead isn’t worth it, and they use Spark when the job is complex enough to justify spending the time to do it well. That’s not a sign of indecision. That’s actually the right call.</p><p>The skill that matters isn’t knowing which tool is objectively better. It’s knowing the tradeoffs well enough to make a conscious choice for your specific situation. That’s what separates people who just execute pipelines from people who architect data systems effectively.</p><h3>Ready to Interview Prep?</h3><p>If you’re preparing for data engineering interviews, understanding Glue and Spark is just the foundation. The next step is knowing which skills and certifications actually get you hired. I analyzed over 1,000 data engineering job postings and found some surprising patterns about which certifications matter and which ones are a waste of money. <a href="https://medium.com/towards-data-engineering/i-analyzed-1-000-data-engineering-job-postings-heres-which-certifications-actually-matter-in-2026-544fb1594d79"><strong>Check it out for a data-driven breakdown of what hiring managers actually care about in 2026</strong></a><strong>.</strong></p><p>#DataEngineering #DataEngineerCertification #AWSCertified #Databricks #FAANG #TechCareers #CloudComputing #DataEngineerInterview #BigData #DataPipelines #CareerAdvice #Snowflake #GoogleCloud #DataEngineerJobs #TechJobs2026</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a895ea801097" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/i-spent-50-hours-mastering-aws-glue-and-spark-so-you-dont-have-to-a895ea801097">I Spent 50 Hours Mastering AWS Glue and Spark. So You Don’t Have To</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Analyzed 1,000+ Data Engineering Job Postings Here’s Which Certifications Actually Matter in 2026]]></title>
            <link>https://medium.com/towards-data-engineering/i-analyzed-1-000-data-engineering-job-postings-heres-which-certifications-actually-matter-in-2026-544fb1594d79?source=rss-c037363a37db------2</link>
            <guid isPermaLink="false">https://medium.com/p/544fb1594d79</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Dharma]]></dc:creator>
            <pubDate>Fri, 06 Feb 2026 04:12:36 GMT</pubDate>
            <atom:updated>2026-02-08T19:45:50.008Z</atom:updated>
            <content:encoded><![CDATA[<h3>I Analyzed 1,000+ Data Engineering Job Postings .Here’s Which Certifications Actually Matter in 2026</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4NxavlNkWKWAjIfH60mxqA.png" /></figure><h3>The Complete Guide to Data Engineering Certifications That Get You Hired (And Which Ones Are a Waste of Money)</h3><p>Look, I get it. You’re scrolling through LinkedIn, seeing “AWS Certified” and “Databricks Certified” plastered on every data engineer’s profile, and you’re wondering do I actually need these things? Are they worth the money? Which ones should I get first?</p><p>I spent months analyzing and applying over 1,000 data engineering job postings from Amazon, Google, Meta, Netflix, Databricks, Snowflake, Fortune 500, and dozens of high-growth startups. I tracked every certification mentioned in requirements and “nice to haves.” I talked to hiring managers. I looked at salary data.</p><p>And honestly? The answers surprised me.</p><p>Some certifications are basically golden tickets that get recruiters excited. Others are expensive pieces of paper that nobody cares about. The difference between them isn’t always obvious until you’ve done the research.</p><p>So I did the research. Here’s everything you need to know about data engineering certifications in 2026.</p><h3>Wait Do Data Engineering Certifications Even Matter Anymore?</h3><p>Short answer: <strong>Yes, but not in the way you think.</strong></p><p>Here’s what certifications actually do for you:</p><p><strong>They get you past the ATS and recruiter screens.</strong> Big companies get thousands of applications. Recruiters literally search for “AWS Certified” or “Databricks” to filter candidates. No cert? You might never get seen.</p><p><strong>They give you structured learning paths.</strong> Let’s be real YouTube tutorials and random Medium articles only get you so far. Certification prep forces you to actually learn the stuff you’ve been “meaning to get around to.”</p><p><strong>They signal that you give a damn.</strong> Hiring managers told me this repeatedly. When someone has relevant certifications, it shows they’re investing in their career. That matters.</p><p><strong>But here’s what certifications DON’T do:</strong></p><p>They don’t replace hands-on experience. They don’t mean you can skip LeetCode prep. They don’t guarantee you’ll pass a system design interview.</p><p>Think of certifications as one piece of the puzzle an important piece, but not the whole picture.</p><h3>The Right Order: Cloud First, Projects Second, Specialized Certs Third</h3><p>This is the mistake I see people make all the time. They jump straight into Databricks or Snowflake certifications without having cloud fundamentals down. Then they struggle because half the concepts assume you already understand cloud architecture.</p><p><strong>Here’s the order I recommend:</strong></p><h3>Step 1: Get Your Cloud Foundation (Month 1–2)</h3><p>Pick ONE cloud platform and get certified at the associate level:</p><ul><li><strong>AWS Cloud Practitioner</strong> → then <strong>AWS Data Engineer Associate</strong></li><li><strong>Google Cloud Digital Leader</strong> → then <strong>GCP Professional Data Engineer</strong></li><li><strong>Azure Fundamentals (AZ-900)</strong> → then <strong>Azure Data Engineer (DP-203 or DP-700)</strong></li></ul><p>Don’t try to do all three clouds. Pick the one your target companies use most. If you’re not sure, AWS is the safe bet it has the largest market share and most job postings mention it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-fTPpWY3pxxq7u8EIehqMQ.png" /></figure><h3>Step 2: Build Real Projects (Month 3–4)</h3><p>This is where most people skip ahead and regret it later. Before you spend money on specialized certs, build 2–3 portfolio projects that demonstrate actual skills:</p><ul><li>An end-to-end ETL pipeline on your chosen cloud</li><li>A streaming data project with Kafka or Kinesis</li><li>A data warehouse with dimensional modeling</li></ul><p>Put these on GitHub. Write about them. These projects will help you understand the specialized certs AND give you something to talk about in interviews.</p><h3>Step 3: Get Specialized Platform Certifications (Month 5–6)</h3><p>NOW you’re ready for the Databricks, Snowflake, and other platform-specific certs. They’ll make way more sense because you have context. And you’ll actually retain the information.</p><p>Trust me on this order. It works.</p><h3>The 2026 Data Engineering Certification Tier List</h3><p>Alright, let’s get into the actual certifications. I’m ranking these based on three things: how often they appear in job postings, what hiring managers told me, and the practical value for interviews.</p><h3>S-TIER: High-Impact Certifications That Actually Get You Hired</h3><h4>1. Databricks Certified Generative AI Engineer Associate</h4><p><strong>Cost:</strong> $200 | <strong>Prep Time:</strong> 2–3 months | <strong>Validity:</strong> 2 years</p><p>If there’s one certification that defines 2026, it’s this one.</p><p>Every data engineering job I looked at and I mean every single one at top companies mentioned something about AI integration, RAG pipelines, vector databases, or LLM-powered features. This isn’t a trend anymore. It’s the new normal.</p><p>The Databricks GenAI certification covers exactly what companies are hiring for right now:</p><ul><li>Designing and building RAG (Retrieval-Augmented Generation) applications</li><li>Working with vector databases and semantic search</li><li>Deploying models with Databricks Model Serving</li><li>Managing the ML lifecycle with MLflow</li><li>Data governance for AI with Unity Catalog</li></ul><p>Here’s a real scenario from a job posting I saw: “Design data pipelines that feed our recommendation embeddings and keep them fresh as user behavior changes.”</p><p>That’s not a machine learning engineer job. That’s a data engineer job. And the GenAI cert teaches you exactly how to approach problems like this.</p><p><strong>Who should get this:</strong> Honestly? Everyone who wants to stay relevant in data engineering. The shift toward AI-integrated data systems is as big as the shift to cloud was a decade ago. Don’t get left behind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/0*QzT34eq4kctJ0o6a.jpg" /></figure><h4>2. AWS Certified Data Engineer Associate (DEA-C01)</h4><p><strong>Cost:</strong> $150 | <strong>Prep Time:</strong> 2–3 months | <strong>Validity:</strong> 3 years</p><p>This is the most commonly requested certification in data engineering job postings. Period.</p><p>AWS launched this cert in 2024 to replace the retired Big Data Specialty certification, and it’s become the industry standard for anyone working in AWS environments. Which is… most people. Amazon, Netflix, Capital One, Airbnb, and thousands of other companies run their data infrastructure on AWS.</p><p>The certification validates your ability to:</p><ul><li>Design scalable data pipelines using AWS Glue, Lambda, and Step Functions</li><li>Build streaming architectures with Kinesis</li><li>Implement data lakes on S3 with proper partitioning and governance</li><li>Orchestrate workflows with MWAA (Managed Airflow)</li><li>Apply security best practices and IAM policies</li></ul><p>What makes this cert particularly valuable is that the concepts transfer well to interviews. System design questions often involve AWS services, and having this certification means you can confidently discuss trade-offs between different architectural approaches.</p><p><strong>Pro tip:</strong> The AWS Data Engineer cert pairs incredibly well with hands-on projects. Build something real on AWS, get the cert, and you’ll crush interviews.</p><h4>3. Google Cloud Professional Data Engineer</h4><p><strong>Cost:</strong> $200 | <strong>Prep Time:</strong> 3–4 months | <strong>Validity:</strong> 2 years</p><p>Google’s data engineering certification is consistently ranked as one of the most valuable cloud certifications you can get. It’s also one of the hardest to pass.</p><p>GCP has smaller market share than AWS, but here’s an interesting pattern I noticed: companies that use GCP tend to pay more for data engineers. The talent pool is smaller, so supply and demand works in your favor.</p><p>The exam covers:</p><ul><li>BigQuery optimization, cost management, and advanced features</li><li>Dataflow for streaming and batch processing (Apache Beam under the hood)</li><li>Pub/Sub for event-driven architectures</li><li>Dataproc for running Spark workloads</li><li>Looker and data visualization integration</li><li>ML pipeline design and BigQuery ML</li></ul><p>The questions are scenario-based and require you to actually think through trade-offs. It’s not a memorization test. You need to understand when to use Dataflow vs. Dataproc, how to optimize BigQuery costs, and how to design for scale.</p><p><strong>Who should get this:</strong> Anyone targeting Google, Spotify, Twitter/X, or the many startups that chose GCP. Also great if you want a certification that genuinely makes you a better engineer, not just a better test-taker.</p><h3>A-TIER: Strong Certifications That Boost Your Resume</h3><h4>4. Databricks Certified Data Engineer Associate</h4><p><strong>Cost:</strong> $200 | <strong>Prep Time:</strong> 1–2 months | <strong>Validity:</strong> 2 years</p><p>Databricks has absolutely exploded in popularity. Their Lakehouse architecture combining the best of data lakes and data warehouses is becoming the default for modern data teams.</p><p>This entry-level certification proves you can:</p><ul><li>Build ETL pipelines with Spark SQL and PySpark</li><li>Use Delta Lake for reliable data storage with ACID transactions</li><li>Implement orchestration with Databricks Workflows</li><li>Apply data governance with Unity Catalog</li></ul><p>One thing I love about this cert is that it’s practical. The questions are based on real scenarios you’d encounter on the job. And Delta Lake’s time travel feature? That comes up in interviews more than you’d expect. Knowing how to recover accidentally deleted data or audit historical changes is valuable stuff.</p><p><strong>Pairs well with:</strong> The Databricks GenAI cert. Together, they show you understand both traditional data engineering AND where the field is heading.</p><h4>5. Microsoft Fabric Data Engineer Associate (DP-700)</h4><p><strong>Cost:</strong> $165 | <strong>Prep Time:</strong> 2–3 months | <strong>Validity:</strong> 1 year (free online renewal)</p><p>Microsoft completely overhauled their data certifications in early 2025, retiring the old DP-203 Azure Data Engineer cert and replacing it with Fabric-focused credentials.</p><p>Why does this matter? Because Microsoft Fabric is Microsoft’s big bet on unified analytics, and enterprise companies are paying attention. Approximately 97% of Fortune 500 companies use Power BI, and Fabric is designed to bring everything together data engineering, data science, real-time analytics, and business intelligence.</p><p>The DP-700 covers:</p><ul><li>OneLake and Lakehouse architecture in Fabric</li><li>Data Factory pipelines and data flows</li><li>Real-Time Intelligence features</li><li>Integration with Power BI and the broader Microsoft ecosystem</li></ul><p><strong>Hot take:</strong> If you’re targeting traditional enterprise companies banks, insurance, healthcare, manufacturing Microsoft certifications often matter MORE than AWS or GCP. These organizations have decades of Microsoft investment and aren’t switching anytime soon.</p><h4>6. Snowflake SnowPro Core Certification</h4><p><strong>Cost:</strong> $175 | <strong>Prep Time:</strong> 1–2 months | <strong>Validity:</strong> 2 years</p><p>Snowflake went from a startup to a $50+ billion company because their data warehouse is genuinely great to work with. The SnowPro Core certification is the foundation for all advanced Snowflake credentials.</p><p>It covers:</p><ul><li>Virtual warehouse architecture and auto-scaling</li><li>Data loading strategies (bulk and continuous)</li><li>Time travel and fail-safe for data recovery</li><li>Secure data sharing across organizations</li><li>Role-based access control and governance</li></ul><p>The time travel feature is a real interview topic. I’ve seen questions like “An analyst accidentally ran a DELETE statement on a production table. Walk me through how you’d recover the data.” Knowing about time travel, UNDROP commands, and fail-safe recovery options sets you apart.</p><p><strong>Who should get this:</strong> Anyone working with or targeting companies that use Snowflake. It’s particularly popular in fintech, healthcare, and retail analytics.</p><h3>B-TIER: Useful for Specific Situations</h3><h4>7. Databricks Certified Associate Developer for Apache Spark</h4><h4>8. IBM Data Engineering Professional Certificate (Coursera)</h4><h4>9. Google Associate Cloud Engineer / Associate Data Practitioner</h4><h3>Good News: Most Companies Will Sponsor Your Certifications</h3><p>Here’s something that surprised me when I started researching: <strong>most tech companies will pay for your certifications.</strong></p><p>Certification sponsorship is a standard benefit at:</p><ul><li>All FAANG companies (Amazon, Apple, Netflix, Google, Meta)</li><li>Major cloud providers (AWS, Google Cloud, Microsoft, Databricks, Snowflake)</li><li>Most Fortune 500 companies with data teams</li><li>Many well-funded startups (Series B and beyond)</li></ul><p>The typical policy is either:</p><ol><li><strong>Full reimbursement</strong> — You pay upfront, pass the exam, and get reimbursed</li><li><strong>Pre-paid vouchers</strong> — Company provides exam vouchers directly</li><li><strong>Learning stipends</strong> — Annual budget ($1,000-$5,000) for courses, certifications, and conferences</li></ol><p>Before you spend your own money, check with your employer. Even if there’s no formal policy, many managers will approve certification costs if you ask. It’s a tiny expense for the company and shows you’re invested in professional development.</p><p>If you’re job hunting, look for “professional development budget” or “learning stipend” in the benefits section. It’s increasingly common, especially at companies that struggle to hire data engineers.</p><p><strong>And here’s a pro tip:</strong> Some certification providers offer employer-specific discounts. Databricks, AWS, and Google all have enterprise programs that might make certs cheaper or free for your company’s employees.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hRdtx1n7i2iOlzXbkjzUlw.png" /></figure><h3>The Certification Combinations That Actually Work</h3><p>Based on my analysis of job postings and conversations with hiring managers, here are the certification stacks that make sense for different situations:</p><h3>For AWS-Heavy Roles (Most Common)</h3><ol><li>AWS Cloud Practitioner (foundation)</li><li>AWS Data Engineer Associate (core credential)</li><li>Databricks GenAI Engineer Associate (differentiation)</li></ol><p><strong>Total cost:</strong> ~$500 | <strong>Timeline:</strong> 5–6 months</p><p>Want the AWS Deep-Dive?</p><p>Certifications teach you how things <strong>should</strong> work. Interviews test whether you know what happens when things <strong>break</strong>.</p><p>I spent 60 hours intentionally breaking AWS pipelines Glue jobs, Kinesis streams, Lambda triggers to learn what interviewers actually ask about. Spoiler: it’s not what’s on the cert exam.</p><blockquote><a href="https://medium.com/@dharmatejasamudrala/the-5-aws-data-engineering-mistakes-i-made-that-every-beginner-makes-and-how-i-fixed-them-dc88c557938e"><strong><em>I Spent 60 Hours Breaking AWS Pipelines to Prep for Data Engineering Interviews. Here’s What Interviewers Actually Care About</em></strong></a></blockquote><h3>For GCP-Focused Roles</h3><ol><li>Google Cloud Digital Leader (foundation)</li><li>GCP Professional Data Engineer (core credential)</li><li>Databricks Data Engineer Associate (platform diversity)</li></ol><p><strong>Total cost:</strong> ~$500 | <strong>Timeline:</strong> 6–7 months</p><h3>For Enterprise/Microsoft Shops</h3><ol><li>Azure Fundamentals AZ-900 (foundation)</li><li>Microsoft Fabric DP-700 (core credential)</li><li>Snowflake SnowPro Core (common pairing)</li></ol><p><strong>Total cost:</strong> ~$500 | <strong>Timeline:</strong> 5–6 months</p><h3>For Modern Data Stack Startups</h3><ol><li>Any cloud associate cert (foundation)</li><li>Databricks Data Engineer Associate (lakehouse)</li><li>Databricks GenAI Engineer Associate (AI integration)</li><li>Snowflake SnowPro Core (warehouse)</li></ol><p><strong>Total cost:</strong> ~$700 | <strong>Timeline:</strong> 7–9 months</p><h3>Certifications That Are NOT Worth Your Money in 2026</h3><p>Let me save you some time and money. Skip these:</p><p><strong>Hadoop certifications</strong> — Hadoop is legacy technology at this point. Cloudera even retired their Hadoop Migration Architect certification. The industry has moved on.</p><p><strong>Generic “Big Data” certifications</strong> — Vague credentials from unknown providers don’t impress anyone. Employers want cloud-specific, vendor-specific skills.</p><p><strong>Multiple cloud certs at the same level</strong> — Don’t get AWS AND GCP AND Azure associate certs. Pick one cloud, go deep, and demonstrate expertise. Breadth without depth isn’t valuable.</p><p>What to Do AFTER You Get Certified</p><p>Certifications alone don’t get you hired. Here’s how to maximize their value:</p><h3>Update Your LinkedIn Immediately</h3><p>Add certifications to the licenses section AND mention them in your headline. Recruiters search for these terms. Make yourself findable.</p><h3>Build a Portfolio Project Using What You Learned</h3><p>The best way to retain certification knowledge is to apply it. Build a project that demonstrates the skills you certified in. Put it on GitHub with a solid README.</p><h3>Practice Interview Questions</h3><p>Certifications teach you concepts. Interviews test whether you can apply them. Practice SQL on DataLemur or LeetCode. Work through system design scenarios. Know how to explain architectural trade-offs.</p><h3>Talk About Certifications Correctly in Interviews</h3><p>Don’t just say “I’m AWS certified.” Say something like: “I got the AWS Data Engineer cert because I wanted to deepen my understanding of streaming architectures. The Kinesis and Lambda patterns I learned helped me design our real-time analytics pipeline at work.”</p><p>Connect the certification to real skills and real outcomes.</p><h3>Real Talk: Certifications Are Just One Piece</h3><p>I want to be completely honest with you. Certifications are valuable, but they’re not magic.</p><p>The data engineers who get hired at top companies have:</p><p><strong>Certifications</strong> To get past resume screens and demonstrate structured knowledge</p><p><strong>Hands-on projects</strong> To prove they can actually build things, not just pass tests</p><p><strong>SQL fluency </strong>Because every single interview includes SQL questions</p><p><strong>System design skills </strong>Because senior roles require architectural thinking</p><p><strong>Communication ability </strong>Because you have to explain technical trade-offs to non-technical people</p><p>If you only focus on certifications and ignore everything else, you’ll get interviews but struggle to convert them to offers. If you only focus on projects and ignore certifications, you might not get interviews at all.</p><p>Balance is everything.</p><h3>Quick Reference: 2026 Data Engineering Certification Cheat Sheet</h3><figure><img alt="Certification Cost Prep Time Best For Difficulty Databricks GenAI Engineer Associate $200 2–3 months AI integration roles Medium AWS Data Engineer Associate $150 2–3 months AWS-focused companies Medium GCP Professional Data Engineer $200 3–4 months GCP shops, higher pay Hard Databricks Data Engineer Associate $200 1–2 months Lakehouse environments Medium Microsoft Fabric DP-700 $165 2–3 months Enterprise Microsoft shops Medium Snowflake SnowPro Core $175 1–2 months Snowflake environments Easy-Me" src="https://cdn-images-1.medium.com/max/1024/1*CQJt-wloINJE6b7z-x1o3g.png" /><figcaption>2026 Data Engineering Certification Cheat Sheet</figcaption></figure><h3>Final Thoughts</h3><p>The data engineering landscape in 2026 is defined by three major shifts:</p><ol><li><strong>GenAI integration</strong> is now a core data engineering skill, not a nice-to-have</li><li><strong>Lakehouse architectures</strong> (Databricks, Delta Lake) are replacing traditional data warehouse patterns</li><li><strong>Real-time streaming</strong> is expected, not exceptional</li></ol><p>The certifications that matter reflect these shifts. Databricks GenAI, AWS Data Engineer, GCP Professional Data Engineer these are the credentials that actually move the needle.</p><p>But remember the order: <strong>cloud foundation first, hands-on projects second, specialized certifications third.</strong> And check if your company will pay before spending your own money.</p><p>The best data engineers I know treat certifications as learning opportunities, not just resume padding. They use the structured curriculum to genuinely understand new technologies. Then they apply what they learned to real projects.</p><p>That’s the approach that works.</p><p>Good luck out there. The data engineering job market is competitive, but the demand for skilled engineers keeps growing. With the right certifications, real skills, and solid preparation, you’ll do great.</p><p><strong>Found this helpful? I write about data engineering careers, interview prep, and the skills that actually matter. Follow along for more.</strong></p><h3>Frequently Asked Questions</h3><p><strong>Which data engineering certification should I get first?</strong> Start with a cloud foundation cert (AWS Cloud Practitioner or GCP Associate) before specialized certs. Then get AWS Data Engineer Associate or Databricks Data Engineer Associate based on your target companies’ tech stacks.</p><p><strong>Are data engineering certifications worth it in 2026?</strong> Yes, especially for getting past resume screens at large companies. Certifications like AWS Data Engineer Associate and Databricks GenAI Engineer Associate appear frequently in job requirements and signal relevant skills to hiring managers.</p><p><strong>How much do data engineering certifications cost?</strong> Most data engineering certifications cost between $150-$200 per exam. Many employers offer certification sponsorship or professional development budgets that cover these costs.</p><p><strong>What’s the best Databricks certification for data engineers?</strong> The Databricks Data Engineer Associate is the foundation, but the Databricks GenAI Engineer Associate is increasingly important for 2026 as companies integrate AI into data pipelines.</p><p><strong>Do FAANG companies care about certifications?</strong> Yes and no. Certifications help get past initial screens and show relevant knowledge, but FAANG interviews focus heavily on coding ability, system design, and problem-solving. Certifications alone won’t get you an offer they’re one piece of the puzzle.</p><p><strong>How long does it take to get a data engineering certification?</strong> Most data engineering certifications require 1–3 months of preparation depending on your existing experience. Plan for 5–10 hours per week of study time.</p><blockquote><a href="https://medium.com/@dharmatejasamudrala/the-5-aws-data-engineering-mistakes-i-made-that-every-beginner-makes-and-how-i-fixed-them-dc88c557938e"><strong><em>I Spent 60 Hours Breaking AWS Pipelines to Prep for Data Engineering Interviews. Here’s What Interviewers Actually Care About</em></strong></a></blockquote><p>#DataEngineering #DataEngineerCertification #AWSCertified #Databricks #FAANG #TechCareers #CloudComputing #DataEngineerInterview #BigData #DataPipelines #CareerAdvice #Snowflake #GoogleCloud #DataEngineerJobs #TechJobs2026</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=544fb1594d79" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/i-analyzed-1-000-data-engineering-job-postings-heres-which-certifications-actually-matter-in-2026-544fb1594d79">I Analyzed 1,000+ Data Engineering Job Postings Here’s Which Certifications Actually Matter in 2026</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Spent 60 Hours Breaking AWS Pipelines to Prep for Data Engineering Interviews.]]></title>
            <link>https://medium.com/@dharmatejasamudrala/the-5-aws-data-engineering-mistakes-i-made-that-every-beginner-makes-and-how-i-fixed-them-dc88c557938e?source=rss-c037363a37db------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc88c557938e</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Dharma]]></dc:creator>
            <pubDate>Sun, 01 Feb 2026 02:27:14 GMT</pubDate>
            <atom:updated>2026-02-02T15:13:14.615Z</atom:updated>
            <content:encoded><![CDATA[<h3>I Spent 60 Hours Breaking AWS Pipelines to Prep for Data Engineering Interviews. Here’s What Interviewers Actually Care About</h3><h3>Most tutorials teach you how to build pipelines. Interviews test whether you understand why they break and how you’d fix them in production.</h3><p>I spent 60 hours deliberately breaking AWS data pipelines to understand what actually gets asked in data engineering interviews.</p><p>I built pipelines with S3, Glue, Redshift, Lambda, and Kinesis. Then I broke them in every way I could think of wrong partitioning, unoptimized DPUs, missing validation, bad IAM policies, monolithic architectures. I documented every failure, every error message, and every fix.</p><p>Why? Because <strong>data engineering interviews don’t test if you can follow a tutorial. They test if you can diagnose production issues and make the right architectural trade-offs under pressure.</strong></p><p>After going through this process, I noticed a pattern: interviewers ask about the same five problem areas over and over. Not because they’re trick questions, but because these are the exact issues that break real pipelines in production.</p><p>Here’s what I learned and what you should focus on if you’re prepping for interviews.</p><h3>Interview Topic #1: S3 Data Lake Design and Partitioning Strategy</h3><p><strong>What interviewers ask:</strong> “How would you structure data in S3 for a time-series dataset with 10 million events per day that needs to support both batch analytics and ad-hoc queries?”</p><p>This question appears in almost every data engineering interview I’ve seen. It’s testing whether you understand <strong>query performance optimization</strong> and <strong>cost management</strong> in AWS.</p><p>Here’s what most candidates get wrong: they talk about storing data in S3 without mentioning partitioning strategy. That’s an immediate red flag.</p><p><strong>The answer interviewers want to hear:</strong></p><p>“I’d implement Hive-style partitioning with date-based keys to minimize data scanned during queries. For a time-series dataset, I’d partition by year, month, and day:</p><pre>s3://data-lake/events/year=2025/month=02/day=02/</pre><p>This structure allows Amazon Athena and AWS Glue to apply partition pruning automatically, scanning only the relevant date ranges instead of the entire dataset. For 10 million events per day, this reduces Athena query costs by 90%+ and improves query latency from minutes to seconds.”</p><p><strong>Why this matters:</strong> Interviewers are looking for evidence that you’ve worked with production-scale data. Anyone can dump files into S3. Understanding partitioning strategy separates junior candidates from senior ones.</p><p><strong>Follow-up questions I’ve been asked:</strong></p><ul><li>“What if you need to query by both date and region?”</li><li>“How do you handle late-arriving data in this partition structure?”</li><li>“What file format would you choose and why?”</li></ul><p>The right answers: Add region as a second partition key, implement an update mechanism for late data using Glue bookmarks, and use Parquet for columnar compression and query performance.</p><h3>Interview Topic #2: AWS Glue Performance Optimization and Cost Control</h3><p><strong>What interviewers ask:</strong> “Your Glue ETL job is processing 500K rows and taking 45 minutes to complete. The monthly bill is $800. How would you optimize this?”</p><p>This is a system design meets troubleshooting question. It tests whether you understand <strong>AWS Glue’s pricing model</strong> and can make intelligent trade-offs between cost and performance.</p><p><strong>The answer that demonstrates senior-level thinking:</strong></p><p>“First, I’d analyze whether we’re over-provisioned on DPUs. AWS Glue charges per Data Processing Unit each DPU is 4 vCPUs and 16 GB of memory. For 500K rows, the default configuration of 10 workers is likely overkill.</p><p>I’d switch to G.1X worker types with 2–3 workers, which is sufficient for this data volume and would cut costs by 70–80%. I’d also implement job bookmarks to process only new data instead of reprocessing the entire dataset on every run.</p><p>Second, I’d move from a scheduled execution model to event-driven triggers using Lambda and S3 event notifications. This ensures Glue only runs when there’s actually new data to process, eliminating unnecessary executions.”</p><p><strong>The follow-up that catches unprepared candidates:</strong> “How would you monitor whether your optimization actually worked?”</p><p><strong>The right answer:</strong> “I’d set up CloudWatch metrics tracking Glue job duration, DPU hours consumed, and cost per execution. I’d establish baseline metrics before optimization and compare against post-optimization runs. If job duration increases beyond acceptable thresholds, I’d incrementally add workers until we hit the right balance of cost and performance.”</p><p>This answer shows you think like a production engineer, not just someone who can run a Glue job.</p><h3>Interview Topic #3: Data Quality and Pipeline Validation</h3><p><strong>What interviewers ask:</strong> “Your pipeline loads data from S3 into Redshift every night. One morning, the analytics team reports duplicate records and missing data. How do you prevent this?”</p><p>This tests your understanding of <strong>data engineering fundamentals</strong> — validation, idempotency, and the medallion architecture pattern.</p><p><strong>The answer that shows you’ve built production pipelines:</strong></p><p>“I’d implement validation at multiple stages using a medallion architecture approach:</p><p><strong>Bronze layer (Raw):</strong> Raw data lands in S3 with no transformation. This is our source of truth.</p><p><strong>Silver layer (Validated):</strong> A Glue job reads from bronze, applies deduplication logic using business keys or surrogate keys, validates schema, and writes clean Parquet files to the silver layer. This is where data quality rules are enforced.</p><p><strong>Gold layer (Analytics-ready):</strong> Only validated data from silver gets loaded into Redshift.</p><p>Between silver and gold, I’d add a reconciliation check — a Lambda function that compares row counts between the source file and the destination table for each partition. If there’s a mismatch, it triggers an SNS alert before the data is exposed to downstream consumers.</p><p>For idempotency, I’d use Glue job bookmarks to track processed data and implement upsert logic in Redshift using staging tables and MERGE statements.”</p><p><strong>Why this answer works:</strong> It demonstrates knowledge of industry-standard architecture patterns (medallion), AWS-specific implementation (Glue bookmarks, SNS), and production-ready thinking (reconciliation checks, idempotency).</p><h3>Interview Topic #4: IAM Security and Access Control</h3><p><strong>What interviewers ask:</strong> “How would you secure a data pipeline where Glue needs to read from S3, write to Redshift, and send alerts via SNS?”</p><p>This is testing <strong>AWS security best practices</strong> and whether you understand the principle of least privilege.</p><p><strong>The wrong answer:</strong> “I’d create an IAM role with full S3, Redshift, and SNS access.”</p><p><strong>The right answer:</strong></p><p>“I’d create separate IAM roles for each service with scoped-down permissions:</p><p>For the Glue ETL job:</p><ul><li>s3:GetObject on the specific source bucket and prefix</li><li>s3:PutObject on the specific destination bucket and prefix</li><li>redshift-data:ExecuteStatement for the target Redshift cluster</li><li>No SNS permissions — Glue doesn’t need to send alerts directly</li></ul><p>For the validation Lambda:</p><ul><li>s3:GetObject on the processed data location</li><li>redshift-data:ExecuteStatement with read-only access to check row counts</li><li>sns:Publish on the specific alert topic</li></ul><p>I’d also enable AWS CloudTrail logging for audit trails and use AWS Lake Formation for fine-grained, column-level access control on sensitive data in the data lake.”</p><p><strong>The follow-up:</strong> “What happens if your Glue job needs to access data from a different AWS account?”</p><p><strong>The answer:</strong> “I’d set up cross-account access using IAM roles with trust policies. The source account grants the Glue role in the target account permission to assume a role that has S3 read access. This is more secure than sharing access keys and allows centralized permission management.”</p><h3>Interview Topic #5: Pipeline Architecture and Failure Handling</h3><p><strong>What interviewers ask:</strong> “Design a data pipeline that ingests API data, transforms it, and loads it into Redshift. How do you handle failures at each stage?”</p><p>This is the ultimate test of whether you think like a production data engineer. It combines system design, error handling, and operational thinking.</p><p><strong>The answer that demonstrates production experience:</strong></p><p>“I’d design a modular pipeline with independent stages, each with its own failure handling:</p><p><strong>Stage 1 — Ingestion:</strong> Lambda function triggered on a schedule pulls from the API and writes raw JSON to S3 bronze layer. If the API call fails, Lambda retries with exponential backoff and sends an SNS alert after 3 failed attempts.</p><p><strong>Stage 2 — Cataloging:</strong> S3 event triggers a Glue Crawler to update the Data Catalog. If the crawler fails (e.g., schema incompatibility), it doesn’t block downstream processing — we investigate the schema issue separately.</p><p><strong>Stage 3 — Transformation:</strong> Glue job reads from bronze, validates, transforms, and writes Parquet to silver layer. I’d enable Glue job bookmarks so failures don’t require full reprocessing. CloudWatch logs track execution metrics.</p><p><strong>Stage 4 — Loading:</strong> A separate Glue job loads from silver to Redshift. I’d use staging tables and a MERGE statement for idempotent upserts. If the load fails, the silver layer data remains intact for retry.</p><p><strong>Stage 5 — Validation:</strong> Lambda runs reconciliation checks and publishes metrics to CloudWatch. Any anomalies trigger SNS alerts to the on-call engineer.</p><p>Each stage logs success/failure to a DynamoDB audit table with timestamps, record counts, and error messages. This gives us full observability and makes debugging trivial.”</p><p><strong>Why this works:</strong> It shows you understand modularity, idempotency, observability, and failure isolation — all critical for production data engineering.</p><h3>What Actually Matters in Data Engineering Interviews</h3><p>After 60 hours of building and breaking AWS pipelines, here’s what I learned: <strong>interviewers don’t care if you’ve memorized every AWS service. They care if you can solve production problems.</strong></p><p>The five topics I covered aren’t random — they’re the exact failure modes that happen in real data engineering systems:</p><ul><li><strong>S3 partitioning</strong> — because unoptimized queries cost money and kill performance</li><li><strong>Glue optimization</strong> — because over-provisioned ETL jobs blow through budgets</li><li><strong>Data validation</strong> — because bad data destroys trust in analytics</li><li><strong>IAM security</strong> — because one compromised pipeline can expose the entire data lake</li><li><strong>Modular architecture</strong> — because monolithic pipelines are impossible to debug and maintain</li></ul><p>Every data engineering interview I’ve seen tests some combination of these five areas. Not as trivia questions, but as system design scenarios where you need to explain your reasoning and justify trade-offs.</p><p>If you’re prepping for interviews, here’s my advice:</p><p><strong>Don’t just build tutorials. Break them.</strong></p><p>Take a working pipeline and deliberately introduce failures. Remove partitions and watch Athena costs spike. Over-provision Glue workers and see the bill increase. Skip validation and let duplicate data through. Give everything admin permissions and think about what could go wrong.</p><p>Then document how you’d fix each problem and explain it like you’re in an interview.</p><p>That’s what separated candidates who got offers from candidates who didn’t — not whether they could build a pipeline, but whether they could explain why it broke and how they’d prevent it in production.</p><p><strong>Copy-paste interview prep checklist:</strong></p><p>Can you explain Hive-style partitioning and when to use it?<br> Can you calculate Glue DPU costs and optimize for a given workload?<br> Can you design a medallion architecture with validation between layers?<br> Can you write least-privilege IAM policies for Glue, Lambda, and Redshift?<br> Can you design a modular pipeline with failure isolation and observability?</p><p>If you can confidently answer “yes” to all five, you’re ready for the technical round.</p><p><em>If this changed how you prep for data engineering interviews, clap and follow. I write about what actually gets asked — not just what looks good in documentation.</em></p><p>#DataEngineeringInterview #AWSDataEngineering #DataEngineer #AWSGlue #AmazonRedshift #ETLPipeline #DataEngineering #TechnicalInterview #FANNGInterview #AWSInterview #DataPipeline #CloudDataEngineering</p><h3>The Bottom Line</h3><p>AWS data engineering is not about knowing every service. It’s about <em>If this helped you avoid even one of these mistakes, clap and follow. I’ll keep writing about real AWS data engineering the stuff that actually trips people up, not just the stuff that looks good in a tutorial.</em></p><p>#AWSDataEngineering #DataPipeline #AWSGlue #AmazonRedshift #DataEngineering #ETLPipeline #AWSDataLake #CloudDataEngineering #DataEngineeringTips #AWS #PySpark #DataEngineering2025</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc88c557938e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Interviews at Meta, Netflix, and Amazon These 5 PySpark Questions Came Up Every Time]]></title>
            <link>https://medium.com/towards-data-engineering/from-interviews-at-meta-netflix-and-amazon-these-5-pyspark-questions-came-up-every-time-a186e5b546bb?source=rss-c037363a37db------2</link>
            <guid isPermaLink="false">https://medium.com/p/a186e5b546bb</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Dharma]]></dc:creator>
            <pubDate>Sun, 25 Jan 2026 03:07:01 GMT</pubDate>
            <atom:updated>2026-02-11T05:24:17.290Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yWDiLE4r8gh9pnyrecT0QQ.png" /></figure><p>FAANG interviewers aren’t testing whether you’ve read the Spark docs. They’re testing whether you can solve the problems they deal with every single day.</p><p>Netflix processes 450 billion events daily. Meta handles data from 3 billion users. At that scale, a poorly optimized join doesn’t just slow things downit crashes the entire pipeline and pages someone at 3 AM.</p><p>That’s why every question below comes back to one thing: <strong>can you write PySpark code that actually works at scale?</strong></p><p>Let’s get into it.</p><h3>Question 1: “How Would You Handle Data Skew in a Join Operation?”</h3><p>This question came up in literally every interview loop I did. Every. Single. One.</p><p><strong>Why they care so much:</strong> Data skew is the silent killer of Spark jobs. Everything looks fine in development with sample data, then production blows up because one partition has 100x more records than the others.</p><p><strong>The scenario they usually give you:</strong> Something like “You’re joining a user transactions table with user profiles, but the job keeps timing out or running out of memory.”</p><p><strong>Here’s how I learned to answer it:</strong></p><p>I walk through four techniques, starting with the simplest. Interviewers love seeing that you can pick the right tool for the situation.</p><p><strong>First option broadcast join:</strong></p><p>If one of my tables is small enough to fit in memory (usually under a few GB), I’ll broadcast it to avoid the shuffle entirely.</p><pre>from pyspark.sql.functions import broadcast<br>result = large_transactions_df.join(<br>    broadcast(small_users_df), <br>    on=&quot;user_id&quot;<br>)</pre><p>This copies the small table to every executor. No shuffle needed for the large table.</p><p><strong>Second option salting:</strong></p><p>When both tables are huge, I add a random suffix to the join key. This spreads hot keys across multiple partitions.</p><pre>from pyspark.sql.functions import concat, lit, floor, rand, col<br># Add random salt (0-9) to the skewed table<br>salted_df = skewed_df.withColumn(<br>    &quot;salted_key&quot;,<br>    concat(col(&quot;user_id&quot;), lit(&quot;_&quot;), floor(rand() * 10))<br>)<br># Explode the other table to match all possible salts<br>exploded_df = other_df.crossJoin(<br>    spark.range(10).withColumnRenamed(&quot;id&quot;, &quot;salt&quot;)<br>).withColumn(<br>    &quot;salted_key&quot;,<br>    concat(col(&quot;user_id&quot;), lit(&quot;_&quot;), col(&quot;salt&quot;))<br>)<br>result = salted_df.join(exploded_df, &quot;salted_key&quot;)</pre><p>It’s more code, but it works when nothing else does.</p><p><strong>Third option just turn on AQE:</strong></p><p>If you’re on Spark 3.0 or later (which you probably are), Adaptive Query Execution handles a lot of skew automatically.</p><pre>spark.conf.set(&quot;spark.sql.adaptive.enabled&quot;, &quot;true&quot;)<br>spark.conf.set(&quot;spark.sql.adaptive.skewJoin.enabled&quot;, &quot;true&quot;)</pre><p>I mention this third because interviewers want to see that you understand the underlying problem before reaching for the automatic solution.</p><p><strong>Fourth option isolate the hot keys:</strong></p><p>Sometimes a handful of keys cause all the problems. Process them separately.</p><pre>hot_key_df = df.filter(col(&quot;user_id&quot;) == &quot;power_user_123&quot;)<br>normal_df = df.filter(col(&quot;user_id&quot;) != &quot;power_user_123&quot;)<br># Handle them with different strategies, then union<br>result = handle_hot_keys(hot_key_df).union(handle_normal(normal_df))</pre><p><strong>What impressed interviewers:</strong> I’d ask clarifying questions first. “How skewed is the data? Is it one hot key or thousands? What’s the size of the smaller table?” Then I’d pick the approach that fit.</p><h3>Question 2: “Your PySpark Job Is Running Slow. How Do You Debug It?”</h3><p>This one’s less about code and more about process. They want to see that you think systematically instead of randomly changing configs.</p><p><strong>Here’s the framework I use:</strong></p><p><strong>Step 1: Open the Spark UI and check the Stages tab.</strong></p><p>I’m looking for three things:</p><ul><li>Tasks that take way longer than others (that’s skew)</li><li>Huge shuffle read/write numbers (too much data moving around)</li><li>Failed tasks with OOM errors (memory issues)</li></ul><p><strong>Step 2: Look at the query plan.</strong></p><pre>df.explain(True)</pre><p>This shows me what Spark is actually doing. I’m checking whether filters got pushed down, what kind of joins it chose, and how many shuffles there are.</p><p><strong>Step 3: Match symptoms to causes.</strong></p><p>After doing this a bunch of times, I started recognizing patterns:</p><ul><li>Few tasks running forever while others finish fast? <strong>Data skew.</strong> Use salting or broadcast joins.</li><li>Shuffle read/write in the hundreds of GB? <strong>Too many wide transformations.</strong> Look for unnecessary groupBys or joins that could be filtered first.</li><li>Thousands of tiny tasks? <strong>Small file problem.</strong> Coalesce the input or use Delta Lake’s optimize.</li><li>Same DataFrame getting computed multiple times? <strong>Missing cache.</strong> Add .cache() before the reuse point.</li><li>Job slow but Spark UI looks fine? <strong>Probably Python UDFs.</strong> Replace them with built-in functions or Pandas UDFs.</li></ul><p><strong>Step 4: Fix one thing and measure.</strong></p><p>I never change multiple things at once. Make one change, run the job, compare the metrics.</p><p><strong>What impressed interviewers:</strong> Being specific about what I’d look at in the Spark UI. Saying “I’d check the Stages tab for task duration variance” sounds way better than “I’d look at the Spark UI.”</p><h3>Question 3: “What’s the Difference Between repartition() and coalesce()?”</h3><p>This seems like a simple question, but it’s actually testing whether you understand shuffle costs — which is fundamental to Spark performance.</p><p><strong>The short answer:</strong></p><p>repartition() triggers a full shuffle. coalesce() doesn&#39;t.</p><p><strong>The longer answer that shows you actually get it:</strong></p><p>repartition(n) redistributes all your data across n partitions. Every record potentially moves to a different executor. It&#39;s expensive, but sometimes necessary.</p><pre># Full shuffle — use when you need to rebalance or increase partitions<br>df.repartition(100)<br># Can also partition by column<br>df.repartition(&quot;user_id&quot;)</pre><p>coalesce(n) combines existing partitions locally. No shuffle. But it can only reduce partitions, not increase them.</p><pre># No shuffle — use when reducing partitions before writing<br>df.coalesce(10)</pre><p><strong>When I use each:</strong></p><p>I use repartition() when:</p><ul><li>Data is heavily skewed and I need to rebalance</li><li>I’m increasing the number of partitions for more parallelism</li><li>I want to partition by a specific column for downstream joins</li></ul><p>I use coalesce() when:</p><ul><li>I’m about to write files and want fewer output files</li><li>I’m reducing partitions and the data is already reasonably balanced</li></ul><p><strong>The follow-up they always ask:</strong> “What happens if you call coalesce(100) when you only have 50 partitions?&quot;</p><p>Answer: Nothing. It silently keeps 50 partitions. coalesce() can only decrease, never increase.</p><h3>Question 4: “Explain Narrow vs Wide Transformations”</h3><p>Another question that sounds theoretical but is actually super practical. They’re testing whether you know what makes Spark jobs slow.</p><p><strong>Here’s how I explain it:</strong></p><p><strong>Narrow transformations</strong> don’t require data to move between partitions. Each output partition depends on just one input partition.</p><pre># These are all narrow — fast and cheap<br>df.filter(col(&quot;status&quot;) == &quot;active&quot;)<br>df.select(&quot;user_id&quot;, &quot;amount&quot;)  <br>df.withColumn(&quot;tax&quot;, col(&quot;amount&quot;) * 0.1)</pre><p><strong>Wide transformations</strong> require shuffling data across the network. Output partitions depend on multiple input partitions.</p><pre># These are all wide — expensive at scale<br>df.groupBy(&quot;user_id&quot;).sum(&quot;amount&quot;)<br>df.join(other_df, &quot;user_id&quot;)<br>df.distinct()<br>df.repartition(100)</pre><p><strong>Why this matters in practice:</strong></p><p>Every wide transformation creates a stage boundary. Spark has to:</p><ol><li>Finish all tasks in the current stage</li><li>Write shuffle files to disk</li><li>Transfer data across the network</li><li>Read shuffle files on the receiving executors</li></ol><p>At FAANG scale, a single unnecessary shuffle can add 30 minutes to your job.</p><p><strong>The optimization insight I always mention:</strong></p><p>When I’m optimizing a slow query, I count the wide transformations first. Often the fastest improvement is combining two groupBys into one, or filtering data before a join instead of after.</p><pre># Slow: filters after the expensive join<br>result = big_df.join(other_df, &quot;id&quot;).filter(col(&quot;status&quot;) == &quot;active&quot;)<br># Faster: filter first, then join smaller data<br>result = big_df.filter(col(&quot;status&quot;) == &quot;active&quot;).join(other_df, &quot;id&quot;)</pre><h3>Question 5: “Write a Window Function to Get the Top N Per Group”</h3><p>This is the most common coding question I saw. Some version of “find the top 3 products per category” or “get the latest record per user” came up almost every time.</p><p><strong>The pattern:</strong></p><pre>from pyspark.sql.window import Window<br>from pyspark.sql.functions import row_number, col<br># Define the window: partition by group, order by ranking criteria<br>window_spec = Window.partitionBy(&quot;category&quot;).orderBy(col(&quot;revenue&quot;).desc())<br># Add a rank column<br>ranked_df = df.withColumn(&quot;rank&quot;, row_number().over(window_spec))<br># Filter to top N<br>top_3_per_category = ranked_df.filter(col(&quot;rank&quot;) &lt;= 3)</pre><p><strong>Variations they throw at you:</strong></p><p>“What if there are ties?”</p><p>Use dense_rank() instead of row_number(). With dense_rank(), ties get the same rank and the next rank isn&#39;t skipped.</p><pre>from pyspark.sql.functions import dense_rank<br># row_number: 1, 2, 3, 4 (arbitrary tiebreaker)<br># dense_rank: 1, 1, 2, 3 (ties share rank, no gaps)<br># rank: 1, 1, 3, 4 (ties share rank, gaps after)</pre><p>“Calculate a running total”</p><pre>from pyspark.sql.functions import sum as _sum<br>window_spec = Window.partitionBy(&quot;user_id&quot;).orderBy(&quot;date&quot;).rowsBetween(<br>    Window.unboundedPreceding, <br>    Window.currentRow<br>)<br>df.withColumn(&quot;running_total&quot;, _sum(&quot;amount&quot;).over(window_spec))</pre><p>“Get the previous row’s value”</p><pre>from pyspark.sql.functions import lag<br>window_spec = Window.partitionBy(&quot;user_id&quot;).orderBy(&quot;timestamp&quot;)<br>df.withColumn(&quot;prev_amount&quot;, lag(&quot;amount&quot;, 1).over(window_spec))</pre><p><strong>Performance note I always add:</strong></p><p>Window functions still require a shuffle — all data for each partition key has to end up on the same executor. So I’m thoughtful about what I partition by. Partitioning by user_id when you have a billion users is very different from partitioning by category when you have 50 categories.</p><h3>What I Wish I’d Known Before My First Interview</h3><p>Looking back, here’s what actually mattered:</p><p><strong>Practice explaining out loud.</strong> I knew this stuff, but I stumbled in my first interview because I’d never verbalized it. Now I practice by explaining concepts to myself while walking.</p><p><strong>Always ask clarifying questions.</strong> Before jumping into a solution, I ask about data size, skew patterns, Spark version. It shows I’m thinking about the real problem, not just reciting an answer.</p><p><strong>Understand WHY, not just WHAT.</strong> Interviewers constantly asked “why would you choose that approach?” Having a reason beyond “that’s the standard way” made a huge difference.</p><p><strong>The Spark UI is your friend.</strong> Being able to say “I’d look at the task duration distribution in the Stages tab” is way more impressive than “I’d check the Spark UI.”</p><h3>Wrapping Up</h3><p>These five questions aren’t magic. But they cover the core problems every FAANG data engineer deals with: skew, performance, shuffles, and analytical queries.</p><p>If you can answer these confidently and explain the trade-offs, you’re ahead of most candidates.</p><p>Good luck with your interviews. You’ve got this.</p><p><em>More data engineering interview prep coming soon. Follow if you want SQL, Python, and system design breakdowns from actual FAANG interviews.</em></p><p><strong>#PySpark #DataEngineering #FAANG #TechInterviews #BigData #ApacheSpark #DataEngineerInterview #Meta #Netflix #Amazon #Google #InterviewPrep #CareerAdvice</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a186e5b546bb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/from-interviews-at-meta-netflix-and-amazon-these-5-pyspark-questions-came-up-every-time-a186e5b546bb">From Interviews at Meta, Netflix, and Amazon These 5 PySpark Questions Came Up Every Time</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Spent 80 Hours Studying Batch vs Streaming Here’s What Actually Matters for Data Engineering…]]></title>
            <link>https://medium.com/@dharmatejasamudrala/i-spent-80-hours-studying-batch-vs-streaming-heres-what-actually-matters-for-data-engineering-ada17d00635c?source=rss-c037363a37db------2</link>
            <guid isPermaLink="false">https://medium.com/p/ada17d00635c</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Dharma]]></dc:creator>
            <pubDate>Fri, 23 Jan 2026 17:24:51 GMT</pubDate>
            <atom:updated>2026-01-23T17:24:51.930Z</atom:updated>
            <content:encoded><![CDATA[<h3>I Spent 80 Hours Studying Batch vs Streaming Here’s What Actually Matters for Data Engineering Interviews</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DNWO2xeCwb_StfpRs_KVmA.png" /></figure><p><em>The frameworks, talking points, and “aha moments” that helped me nail system design questions at FAANG companies</em></p><p>Three weeks ago, I bombed a data engineering interview.</p><p>The question seemed simple: <em>“Design a system to track user engagement metrics. Should we use batch or streaming?”</em></p><p>I fumbled. I mentioned Kafka. I said something about Airflow. The interviewer’s eyes glazed over. I knew I’d lost them.</p><p>That rejection lit a fire. I spent the next 8<strong>0 hours</strong> deep-diving into batch vs streaming not just the tools, but the <em>thinking</em> behind them. I read documentation, watched conference talks, studied real architectures at Netflix, Uber, and Airbnb.</p><p>Here’s everything I learned distilled into the frameworks and talking points that actually matter in interviews.</p><h3>The #1 Mistake I Was Making (And You Probably Are Too)</h3><p>My first instinct was always: <em>“Streaming is better because it’s faster.”</em></p><p>Wrong.</p><p>The senior engineers interviewing you don’t want to hear that streaming is “better.” They want to hear you <strong>think in tradeoffs</strong>. Every architecture decision is a tradeoff, and your job is to articulate them clearly.</p><p>Here’s the mental shift that changed everything:</p><blockquote><strong><em>Batch and streaming aren’t competitors. They’re tools optimized for different problems.</em></strong></blockquote><p>When I started framing my answers around “it depends on the business requirements,” interviewers started nodding instead of frowning.</p><h3>The Framework That Landed Me Offers</h3><p>I created a simple decision framework I could use in any interview. I call it the <strong>3L Framework</strong>: Latency, Load, and Logic.</p><h3>1. Latency: How Fresh Does the Data Need to Be?</h3><p>This is the first question I now ask in every system design interview.</p><p>Business Need Acceptable Latency Recommendation Fraud detection Milliseconds Streaming Real-time dashboards Seconds to minutes Streaming Daily business reports Hours Batch ML model training Hours to days Batch Monthly compliance reports Days Batch</p><p><strong>Interview talking point:</strong> <em>“Before I design anything, I need to understand the latency requirements. If we’re detecting fraud, even a 5-minute delay could cost millions. But if we’re generating a weekly sales report, running a batch job at 2 AM is simpler, cheaper, and just as effective.”</em></p><h3>2. Load: What’s the Data Volume and Velocity?</h3><p>This is where I used to get tripped up. High volume doesn’t automatically mean streaming.</p><p><strong>Key insight:</strong> Batch processing can handle <em>massive</em> volumes efficiently because it optimizes for throughput. Streaming optimizes for latency, which often means lower throughput per dollar.</p><p>Scenario Volume Velocity Better Choice IoT sensors (1M devices) High Continuous Streaming Daily transaction logs High Periodic Batch User clickstream Medium Continuous Streaming Data warehouse refresh Very High Once daily Batch</p><p><strong>Interview talking point:</strong> <em>“Streaming shines when data arrives continuously and needs immediate processing. But if we’re processing yesterday’s logs, batch is more cost-effective we can spin up resources, process everything, and shut down. With streaming, we’re paying for always-on infrastructure.”</em></p><h3>3. Logic: How Complex Is the Processing?</h3><p>This one surprised me. Complex transformations often favor batch.</p><p><strong>Why?</strong> Streaming requires processing each event independently or within small windows. If your logic needs to join data across long time periods or perform complex aggregations, batch gives you the full dataset to work with.</p><p>Processing Type Batch Streaming Simple transformations ✓ ✓ Windowed aggregations (minutes) ✓ ✓ Joins across days/weeks of data ✓ Difficult ML model training ✓ Limited Complex business logic ✓ Tricky Real-time alerting Delayed ✓</p><p><strong>Interview talking point:</strong> <em>“If I need to calculate a user’s lifetime value, I need their entire history that’s batch. If I need to alert when a user’s spending exceeds their average, I can do that in streaming with a rolling window. The complexity of the logic drives the architecture.”</em></p><h3>5 “Aha Moments” That Transformed My Interview Answers</h3><h3>Aha #1: Most Production Systems Use Both</h3><p>This was the biggest revelation. I kept thinking I had to <em>choose</em>. But Netflix, Uber, LinkedIn they all use <strong>hybrid architectures</strong>.</p><p>The pattern is called <strong>Lambda Architecture</strong>:</p><ul><li><strong>Speed layer</strong> (streaming): Fast, approximate results for real-time needs</li><li><strong>Batch layer</strong>: Complete, accurate results computed later</li><li><strong>Serving layer</strong>: Merges both views for applications</li></ul><p><strong>Interview talking point:</strong> <em>“In practice, I’d likely recommend a hybrid approach. Streaming handles the real-time requirements while batch provides the source of truth. The batch layer can also correct any approximations or late-arriving data from the streaming layer.”</em></p><h3>Aha #2: “Exactly Once” Is the Hard Problem</h3><p>Every interviewer eventually asks about data consistency. This is where I learned to sound like a senior engineer.</p><p><strong>Batch</strong> makes exactly-once easy: if a job fails, rerun it. The output overwrites the previous attempt. Idempotency is straightforward.</p><p><strong>Streaming</strong> is tricky: messages can be processed multiple times during failures and restarts. You need:</p><ul><li>Idempotent consumers (processing the same message twice has no extra effect)</li><li>Transactional producers (messages are all-or-nothing)</li><li>Deduplication logic (track what you’ve already processed)</li></ul><p><strong>Interview talking point:</strong> <em>“Exactly-once semantics is much harder in streaming. I’d design for at-least-once delivery and make my consumers idempotent for example, using upserts instead of inserts, or tracking processed message IDs. This way, duplicate processing doesn’t corrupt our data.”</em></p><h3>Aha #3: Late-Arriving Data Is a Real Problem</h3><p>Data doesn’t always arrive in order. A mobile app might buffer events during a subway ride and send them hours later. How do you handle this?</p><p><strong>Batch</strong>: Late data is included in the next batch run. You might reprocess previous days if needed.</p><p><strong>Streaming</strong>: You define <em>watermarks</em> a threshold after which you consider a time window “complete.” Events arriving after the watermark are either dropped or sent to a side output for special handling.</p><p><strong>Interview talking point:</strong> <em>“Late-arriving data is one of the trickiest parts of streaming. I’d set a watermark based on business requirements maybe we wait 10 minutes for late mobile events. Anything after that goes to a dead letter queue for batch reconciliation. This gives us real-time speed with eventual correctness.”</em></p><h3>Aha #4: Cost Models Are Completely Different</h3><p>This insight impressed interviewers because it shows business awareness.</p><p><strong>Batch costs</strong>: Proportional to data volume processed. You pay for compute only when running.</p><p><strong>Streaming costs</strong>: Proportional to uptime. You pay 24/7 for always-on infrastructure, plus per-message costs.</p><p>Cost Factor Batch Streaming Compute Pay per job Pay 24/7 Storage Cheaper (S3, data lakes) More expensive (Kafka retention) Scaling Vertical + horizontal Must handle peak load always Engineering Simpler debugging Complex failure handling</p><p><strong>Interview talking point:</strong> <em>“Streaming isn’t free. We’re paying for always-on infrastructure even at 3 AM when traffic is low. If the business requirement is a daily report, batch is significantly more cost-effective. I’d only recommend streaming when the latency requirements justify the cost.”</em></p><h3>Aha #5: Debugging Streaming Is Much Harder</h3><p>This sounds obvious, but the implications are huge.</p><p><strong>Batch</strong>: Job failed? Check the logs, fix the issue, rerun. The data is still there.</p><p><strong>Streaming</strong>: Event caused an error at 3:47 AM? Good luck finding it in the continuous flow. You need robust monitoring, dead letter queues, and replay capabilities.</p><p><strong>Interview talking point:</strong> <em>“Operational complexity is a real factor. Streaming systems need more sophisticated monitoring, alerting, and debugging infrastructure. I’d factor in the team’s experience and the organization’s operational maturity when recommending streaming over batch.”</em></p><h3>The Questions Interviewers Actually Ask (And How I Answer Them Now)</h3><h3>“When would you choose batch over streaming?”</h3><p><em>“I’d choose batch when latency requirements are measured in hours, not secondsdaily reports, ML training, data warehouse loads. Batch is also better for complex processing that needs the full dataset, and when cost efficiency matters more than real-time updates.”</em></p><h3>“When would you choose streaming over batch?”</h3><p><em>“Streaming is essential when the business can’t wait fraud detection, real-time recommendations, live dashboards. If a 5-minute delay means lost revenue or security risk, streaming is the answer. I’d also use streaming when data arrives continuously and processing it incrementally is more efficient than batching.”</em></p><h3>“How do you ensure data consistency between batch and streaming?”</h3><p><em>“I’d use batch as the source of truth. Streaming provides fast approximate results, but the batch layer runs reconciliation to catch any missed or late-arriving data. Applications can merge both views streaming for the latest events, batch for historical accuracy.”</em></p><h3>“What’s the hardest part of streaming architectures?”</h3><p><em>“Handling failures gracefully. In batch, I can just rerun a failed job. In streaming, I need to think about exactly-once semantics, idempotent consumers, checkpoint management, and late data handling. The engineering complexity is higher, which is why I only recommend streaming when latency requirements demand it.”</em></p><h3>“Tell me about a time you chose the wrong architecture.”</h3><p><em>“Early in my career, I built a streaming pipeline for a use case that only needed daily updates. We spent weeks handling edge cases that batch would have solved automatically. The lesson I learned: start with the business requirement, not the technology. Streaming is powerful but not always necessary.”</em></p><h3>My Study Resources (If You Want to Go Deeper)</h3><p>After 80 hours, these were the most valuable:</p><ol><li><strong>“Streaming Systems” by Tyler Akidau </strong>The definitive book on streaming concepts</li><li><strong>“Designing Data-Intensive Applications” by Martin Kleppmann </strong>Chapter 11 on stream processing is gold</li><li><strong>Uber’s engineering blog </strong>Their real-time data infrastructure posts are incredible</li><li><strong>Confluent’s Kafka documentation </strong>Better than any course I’ve taken</li><li><strong>Airflow’s official guides </strong>Surprisingly deep on batch processing patterns</li></ol><h3>The Bottom Line</h3><p>After 80 hours of study and several interviews, here’s what I know:</p><p><strong>The engineers who get offers aren’t the ones who memorize Kafka APIs.</strong> They’re the ones who can explain <em>when</em> to use Kafka, <em>why</em> batch might be better, and <em>how</em> the business requirements drive the technical decisions.</p><p>Batch vs streaming isn’t about which is “better.” It’s about understanding the tradeoffs and communicating them clearly.</p><p>That’s what interviewers want to see.</p><p>And now, that’s what you can show them.</p><p><em>This took me 80 hours to learn. It took you 6 minutes to read. If it helped, share it with someone preparing for data engineering interviews.</em></p><p><em>Drop a comment: What’s the hardest system design question you’ve faced?</em></p><p><strong>Tags:</strong> #DataEngineering #SystemDesign #FAANG #TechInterviews #ApacheKafka #ApacheAirflow #BatchProcessing #StreamProcessing #CareerAdvice #InterviewPrep</p><p><em>Follow for more data engineering interview insights. Next up: “The SQL patterns that appear in 90% of FAANG interviews.”</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ada17d00635c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Walmart Data Engineering Interview : SQL & Data Modeling (With Solutions)]]></title>
            <link>https://medium.com/@dharmatejasamudrala/walmart-data-engineering-interview-sql-data-modeling-with-solutions-1ead02859919?source=rss-c037363a37db------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ead02859919</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Dharma]]></dc:creator>
            <pubDate>Fri, 16 Jan 2026 21:13:58 GMT</pubDate>
            <atom:updated>2026-01-16T21:13:58.500Z</atom:updated>
            <content:encoded><![CDATA[<h3>Walmart Data Engineering Interview : SQL &amp; Data Modeling (With Solutions)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/0*FwPWrjltqT23Bl7r.jpg" /></figure><p><em>Cracking the Walmart data engineering interview? Here are the exact SQL and data modeling questions candidates faced — with step-by-step solutions.</em></p><p>If you’re preparing for a <strong>Walmart data engineering interview</strong>, you already know the stakes. Walmart processes over <strong>2.5 billion rows of transaction data daily</strong>, and they need engineers who can handle that scale.</p><p>I’ve compiled <strong>real interview questions</strong> reported by candidates in 2024–2025. These aren’t theoretical — they’re the problems you’ll actually face.</p><p>Let’s dive in.</p><h3>Question 1: Revenue Analysis with Window Functions</h3><p><strong>The Question:</strong></p><p><em>“Given a sales table, write a SQL query to find each store’s monthly revenue and show the percentage change from the previous month.”</em></p><p>This tests your ability to work with <strong>window functions</strong> and <strong>date manipulation</strong> — essential skills for Walmart’s massive retail datasets.</p><p><strong>Schema:</strong></p><pre>-- sales table<br>CREATE TABLE sales (<br>    transaction_id INT,<br>    store_id INT,<br>    sale_date DATE,<br>    amount DECIMAL(10,2)<br>);</pre><p><strong>The Solution:</strong></p><pre>WITH monthly_revenue AS (<br>    SELECT <br>        store_id,<br>        DATE_TRUNC(&#39;month&#39;, sale_date) AS month,<br>        SUM(amount) AS revenue<br>    FROM sales<br>    GROUP BY store_id, DATE_TRUNC(&#39;month&#39;, sale_date)<br>),<br>revenue_with_change AS (<br>    SELECT <br>        store_id,<br>        month,<br>        revenue,<br>        LAG(revenue) OVER (<br>            PARTITION BY store_id <br>            ORDER BY month<br>        ) AS prev_month_revenue<br>    FROM monthly_revenue<br>)<br>SELECT <br>    store_id,<br>    month,<br>    revenue,<br>    prev_month_revenue,<br>    ROUND(<br>        (revenue - prev_month_revenue) * 100.0 / <br>        NULLIF(prev_month_revenue, 0), <br>        2<br>    ) AS pct_change<br>FROM revenue_with_change<br>ORDER BY store_id, month;</pre><p><strong>Why This Works:</strong></p><p>The query uses LAG() to access the previous month&#39;s revenue within each store partition. The NULLIF prevents division-by-zero errors when a store has no prior data. Walmart interviewers specifically look for this defensive coding pattern.</p><h3>Question 2: Inventory Gap Detection</h3><p><strong>The Question:</strong></p><p><em>“Find all products that were out of stock for more than 3 consecutive days. Return the product, start date, and duration of each stockout.”</em></p><p>This is a classic <strong>gaps and islands</strong> problem — Walmart uses this exact pattern to optimize their supply chain.</p><p><strong>Schema:</strong></p><pre>-- inventory table<br>CREATE TABLE inventory (<br>    product_id INT,<br>    check_date DATE,<br>    stock_quantity INT<br>);</pre><p><strong>The Solution:</strong></p><pre>WITH stockout_days AS (<br>    SELECT <br>        product_id,<br>        check_date,<br>        check_date - CAST(<br>            ROW_NUMBER() OVER (<br>                PARTITION BY product_id <br>                ORDER BY check_date<br>            ) AS INT<br>        ) AS group_id<br>    FROM inventory<br>    WHERE stock_quantity = 0<br>),<br>stockout_periods AS (<br>    SELECT <br>        product_id,<br>        MIN(check_date) AS stockout_start,<br>        MAX(check_date) AS stockout_end,<br>        COUNT(*) AS days_out_of_stock<br>    FROM stockout_days<br>    GROUP BY product_id, group_id<br>)<br>SELECT <br>    product_id,<br>    stockout_start,<br>    stockout_end,<br>    days_out_of_stock<br>FROM stockout_periods<br>WHERE days_out_of_stock &gt; 3<br>ORDER BY days_out_of_stock DESC;</pre><p><strong>The Key Insight:</strong></p><p>The trick is subtracting ROW_NUMBER() from each date. Consecutive dates produce the same group_id, while gaps create new groups. This pattern appears constantly in retail analytics.</p><h3>Question 3: Data Modeling — E-Commerce Order System</h3><p><strong>The Question:</strong></p><p><em>“Design a data model for Walmart’s e-commerce platform that handles orders, returns, and fulfillment tracking. The system must support both store pickup and home delivery.”</em></p><p>This data modeling question tests your understanding of <strong>dimensional modeling</strong> and <strong>real-world retail complexity</strong>.</p><p><strong>The Solution:</strong></p><p>Here’s a star schema approach optimized for analytics:</p><p><strong>Fact Table — Order Lines:</strong></p><pre>fact_order_line<br>├── order_line_id (PK)<br>├── order_id (FK)<br>├── product_id (FK → dim_product)<br>├── customer_id (FK → dim_customer)<br>├── store_id (FK → dim_store)<br>├── fulfillment_id (FK → dim_fulfillment)<br>├── order_date_id (FK → dim_date)<br>├── quantity<br>├── unit_price<br>├── discount_amount<br>├── net_amount<br>└── is_returned (boolean)</pre><p><strong>Dimension Tables:</strong></p><pre>dim_product<br>├── product_id (PK)<br>├── product_name<br>├── category<br>├── subcategory<br>├── brand<br>└── supplier_id</pre><pre>dim_customer<br>├── customer_id (PK)<br>├── customer_type (member/guest)<br>├── registration_date<br>├── preferred_store_id<br>└── loyalty_tier</pre><pre>dim_fulfillment<br>├── fulfillment_id (PK)<br>├── fulfillment_type (pickup/delivery)<br>├── carrier<br>├── estimated_date<br>├── actual_date<br>└── status</pre><pre>dim_store<br>├── store_id (PK)<br>├── store_name<br>├── region<br>├── format (supercenter/neighborhood)<br>└── opening_date</pre><pre>dim_date<br>├── date_id (PK)<br>├── full_date<br>├── day_of_week<br>├── month<br>├── quarter<br>├── fiscal_year<br>└── is_holiday</pre><p><strong>Why This Design Works:</strong></p><p>The star schema separates facts (measurable events) from dimensions (descriptive attributes). The dim_fulfillment table handles both pickup and delivery scenarios through a single abstraction. The is_returned flag in the fact table enables return analysis without a separate fact table—reducing complexity while maintaining analytical power.</p><p><strong>Follow-up Questions They Ask:</strong></p><p>Walmart interviewers often follow up with: “How would you handle partial returns?” or “How would you track fulfillment status changes over time?”</p><p>For partial returns, you’d add a returned_quantity column. For status tracking, implement a Type 2 SCD on dim_fulfillment or create a separate fact_fulfillment_status table.</p><h3>Key Takeaways for Your Walmart Interview</h3><p>These questions test specific skills Walmart values: handling time-series data at scale, identifying patterns in retail operations, and designing models that balance analytical power with query performance.</p><p>Practice these patterns until they’re second nature. Walmart’s interview bar is high, but the questions are predictable if you know what to expect.</p><p><strong>Preparing for more FAANG data engineering interviews?</strong> Follow me for weekly breakdowns of real interview questions with solutions.</p><p><em>#DataEngineering #SQL #WalmartInterview #DataModeling #TechInterview #FAANG #InterviewPrep #BigData</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ead02859919" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Use AI to Clear FAANG Data Engineering Interviews]]></title>
            <link>https://medium.com/towards-data-engineering/how-touse-ai-to-clear-faang-data-engineering-interviews-40851f2168d6?source=rss-c037363a37db------2</link>
            <guid isPermaLink="false">https://medium.com/p/40851f2168d6</guid>
            <category><![CDATA[leetcode]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dharma]]></dc:creator>
            <pubDate>Mon, 05 Jan 2026 15:27:16 GMT</pubDate>
            <atom:updated>2026-02-13T10:21:05.022Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qlp2VLOYYNbDBHMjNyXmwA.png" /></figure><h3>The Secret Weapon Top Candidates Are Using to Crack Google, Amazon, Meta, and Netflix Interviews in 2025</h3><p>So you got the interview invite. Congratulations that’s the hard part, right?</p><p>Wrong. <strong>Getting the interview is just the beginning.</strong> Now you have 1–2 weeks to prove you’re better than 200+ other candidates who also got that same invite.</p><p>Here’s what changed my entire interview prep game: <strong>using AI as my personal interview coach.</strong></p><p>Not to cheat. Not to have AI answer questions for me. But to practice smarter, fill knowledge gaps faster, and walk into interviews more prepared than anyone else in the room.</p><p>Let me show you exactly how.</p><h3>1. AI as Your 24/7 Mock Interviewer</h3><p>Traditional mock interviews cost $100–300/hour. You get maybe 2–3 before your big interview.</p><p>With AI? <strong>Unlimited practice. Any time. Free.</strong></p><p>Here’s the prompt I used:</p><pre>&quot;Act as a senior data engineer interviewing me for <br>[Company Name]. Ask me a medium-difficulty SQL <br>question. After I answer, tell me:<br>- What was good<br>- What a senior candidate would do differently  <br>- One follow-up question you&#39;d ask&quot;</pre><p>I did 20+ mock sessions before my Amazon interview. By round 5, I stopped getting nervous. By round 15, I was actually enjoying it.</p><p><strong>Pro tip:</strong> Practice at the same time as your actual interview. If your interview is at 10am, practice at 10am. Train your brain.</p><h3>2. Fill Knowledge Gaps in Hours, Not Weeks</h3><p>Interview in 5 days but rusty on window functions? Don’t watch a 4-hour course.</p><p><strong>Use this prompt:</strong></p><pre>&quot;I have a data engineering interview in 5 days. <br>Give me a 30-minute crash course on SQL window <br>functions — only what actually gets asked in <br>FAANG interviews. Include 3 practice problems.&quot;</pre><p>The interviewer mentions their stack uses Kafka and you’ve never touched it?</p><pre>&quot;Explain Apache Kafka in 15 minutes like I&#39;m a <br>data engineer who knows SQL and Python but never <br>worked with streaming. Focus on concepts that <br>come up in interviews.&quot;</pre><p><strong>AI learns YOUR gaps and fills them.</strong> No wasted time on stuff you already know.</p><h3>3. Turn “Okay” Answers into “Strong Hire” Answers</h3><p>This is the game-changer most people miss.</p><p>After practicing a problem, paste your solution and ask:</p><pre>&quot;Here&#39;s my SQL solution: [paste code]</pre><pre>Review it like a tough FAANG interviewer. Tell me:<br>- Edge cases I missed<br>- How to optimize for 1 billion rows<br>- What would make this a &#39;strong hire&#39; answer&quot;</pre><p><strong>Real example:</strong> My first attempt at a gaps-and-islands problem was “acceptable.” After AI feedback, I learned to mention index usage, discuss alternative approaches, and handle NULL edge cases. That’s the difference between “maybe” and “definitely hire.”</p><h3>4. Nail Behavioral Questions (Yes, They Matter)</h3><p>Data engineers often bomb behavioral rounds because they don’t prep for them.</p><p><strong>Use AI to structure your stories:</strong></p><pre>&quot;Help me prepare a STAR story for &#39;Tell me about <br>a time you dealt with a data pipeline failure.&#39;</pre><pre>Here&#39;s what happened: [2-3 sentence description]</pre><pre>Make it specific, impactful, and under 2 minutes.&quot;</pre><p>Then practice delivering it:</p><pre>&quot;I&#39;m going to tell you my story. After I finish, <br>give me 2 tough follow-up questions an interviewer <br>would ask.&quot;</pre><p><strong>The questions that actually come up:</strong></p><ul><li>Tell me about a pipeline failure you handled</li><li>Describe a time you disagreed with a stakeholder</li><li>How do you prioritize when everything is urgent?</li></ul><p>Prep these three. Thank me later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/0*dHpqxwZXil0OhQaw.jpg" /></figure><h3>5. The 48-Hour Pre-Interview Blitz</h3><p>Two days before my Google interview, I used this prompt:</p><pre>&quot;I have a data engineering interview at Google in <br>48 hours. Create an intensive prep schedule:<br>- Day 1: Core concept review + practice problems  <br>- Day 2: Mock interviews + light review<br>- Interview morning: 30-min warm-up routine</pre><pre>Focus on SQL, Python, and system design.&quot;</pre><p><strong>Interview morning prompt:</strong></p><pre>&quot;Give me 3 easy warm-up questions to get my brain <br>going before my interview in 1 hour. Nothing hard <br>just wake up my problem-solving mode.&quot;</pre><p>This routine got me in the zone. I walked in calm, focused, and ready.</p><h3>Resources That Pair Perfectly with AI Prep</h3><p>Use AI alongside these platforms not instead of them:</p><p>Resource Best For How to Use with AI <strong>LeetCode</strong> SQL &amp; coding practice Solve problem → paste to AI for review <strong>DataLemur</strong> Real SQL interview questions Practice → ask AI for optimization tips <strong>Interviewing.io</strong> Mock interviews with humans Use AI to prep before, debrief after <strong>Glassdoor</strong> Company-specific questions Feed questions to AI for practice <strong>YouTube (Seattle Data Guy, etc.)</strong> System design concepts Watch → ask AI to quiz you</p><p><strong>The combo is powerful:</strong> Practice on LeetCode/DataLemur → Review with AI → Get feedback → Repeat.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/0*u5Ewj5gbQIzvsxT4.jpg" /></figure><h3>The Bottom Line</h3><p>You got the interview. Now it’s about execution.</p><p>AI won’t take the interview for you. But it will:</p><ul><li>Give you unlimited mock practice</li><li>Fill your knowledge gaps in hours</li><li>Turn average answers into impressive ones</li><li>Prepare you for questions before they’re asked</li></ul><p>The candidates who use AI strategically aren’t cheating they’re just preparing smarter.</p><p>Your competition is doing this. Are you?</p><p><em>Got an interview coming up? Drop a comment with the company I’ll reply with specific prep tips.</em></p><p><strong>Follow me</strong> for more data engineering interview strategies. I post weekly on SQL, Spark, Airflow, and how to actually land the job.</p><p><strong>Tags:</strong> #DataEngineering #TechInterviews #FAANG #SQL #InterviewPrep #AI #ChatGPT #CareerAdvice #JobSearch #BigTech</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40851f2168d6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/how-touse-ai-to-clear-faang-data-engineering-interviews-40851f2168d6">How to Use AI to Clear FAANG Data Engineering Interviews</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Only SQL Interview Guide Data Engineers Need in 2026]]></title>
            <link>https://medium.com/@dharmatejasamudrala/25-sql-scenarios-you-must-master-to-crack-data-engineering-interviews-in-2026-17e4c80559a9?source=rss-c037363a37db------2</link>
            <guid isPermaLink="false">https://medium.com/p/17e4c80559a9</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[interview]]></category>
            <dc:creator><![CDATA[Dharma]]></dc:creator>
            <pubDate>Fri, 02 Jan 2026 18:04:30 GMT</pubDate>
            <atom:updated>2026-01-22T05:14:20.832Z</atom:updated>
            <content:encoded><![CDATA[<h3>25 SQL Scenarios You Must Master To Crack Data Engineering Interviews in 2026</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D2yu4CXWxTqKprStfo-aUA.png" /></figure><p>If you’ve been preparing for data engineering interviews, you already know this: <strong>SQL isn’t optional it’s the foundation.</strong></p><p>While everyone’s busy learning Spark, Kafka, and Airflow, hiring managers are quietly rejecting candidates who can’t write efficient SQL under pressure. I’ve seen senior engineers with 10+ years of experience stumble on window functions. I’ve watched candidates with impressive resumes freeze when asked to optimize a slow query.</p><p>The truth? <strong>80% of your data engineering work will involve SQL.</strong> And yet, most interview prep resources focus on the wrong things.</p><p>This guide covers 25 real-world SQL scenarios that actually appear in data engineering interviews at companies like Google, Amazon, Meta, Netflix, and Uber. These aren’t textbook exercises they’re battle-tested problems that separate the good candidates from the great ones.</p><p>Let’s dive in.</p><h3>Part 1: Window Functions The #1 Interview Topic</h3><h3>Scenario 1: Running Totals and Cumulative Metrics</h3><p><strong>The Problem:</strong> Calculate the running total of daily sales for each product category.</p><pre>SELECT <br>    sale_date,<br>    category,<br>    daily_sales,<br>    SUM(daily_sales) OVER (<br>        PARTITION BY category <br>        ORDER BY sale_date<br>        ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW<br>    ) AS running_total<br>FROM daily_sales;</pre><p><strong>Why This Matters:</strong> Running totals appear in financial reporting, inventory management, and user engagement tracking. Interviewers love this because it tests your understanding of frame specifications.</p><p><strong>Pro Tip:</strong> Know the difference between ROWS and RANGE. It&#39;s a common follow-up question.</p><h3>Scenario 2: Ranking Without Gaps vs. With Gaps</h3><p><strong>The Problem:</strong> Rank employees by salary within each department. Show both RANK() and DENSE_RANK() results.</p><pre>SELECT <br>    employee_name,<br>    department,<br>    salary,<br>    RANK() OVER (PARTITION BY department ORDER BY salary DESC) AS rank_with_gaps,<br>    DENSE_RANK() OVER (PARTITION BY department ORDER BY salary DESC) AS rank_no_gaps,<br>    ROW_NUMBER() OVER (PARTITION BY department ORDER BY salary DESC) AS row_num<br>FROM employees;</pre><p><strong>Interview Follow-up:</strong> “When would you use ROW_NUMBER() over RANK()?&quot;</p><p><strong>Answer:</strong> Use ROW_NUMBER() when you need exactly N results per group (e.g., top 3 products per category). Use RANK() when ties matter for business logic.</p><h3>Scenario 3: Identifying Gaps and Islands</h3><p><strong>The Problem:</strong> Find consecutive login streaks for users (the classic “gaps and islands” problem).</p><pre>WITH numbered_logins AS (<br>    SELECT <br>        user_id,<br>        login_date,<br>        login_date - INTERVAL &#39;1 day&#39; * ROW_NUMBER() OVER (<br>            PARTITION BY user_id ORDER BY login_date<br>        ) AS streak_group<br>    FROM user_logins<br>),<br>streak_summary AS (<br>    SELECT <br>        user_id,<br>        streak_group,<br>        MIN(login_date) AS streak_start,<br>        MAX(login_date) AS streak_end,<br>        COUNT(*) AS streak_length<br>    FROM numbered_logins<br>    GROUP BY user_id, streak_group<br>)<br>SELECT * FROM streak_summary<br>WHERE streak_length &gt;= 7<br>ORDER BY user_id, streak_start;</pre><p><strong>Why It’s Asked:</strong> This pattern appears everywhere subscription continuity, session analysis, inventory tracking. Master this, and you’ve conquered 30% of SQL interviews.</p><h3>Scenario 4: Lead and Lag for Trend Analysis</h3><p><strong>The Problem:</strong> Calculate month-over-month revenue growth percentage.</p><pre>SELECT <br>    month,<br>    revenue,<br>    LAG(revenue) OVER (ORDER BY month) AS prev_month_revenue,<br>    ROUND(<br>        (revenue - LAG(revenue) OVER (ORDER BY month)) * 100.0 / <br>        NULLIF(LAG(revenue) OVER (ORDER BY month), 0), <br>        2<br>    ) AS growth_percentage<br>FROM monthly_revenue;</pre><p><strong>Common Mistake:</strong> Forgetting NULLIF() to handle division by zero. Interviewers watch for this.</p><h3>Scenario 5: First and Last Values Per Group</h3><p><strong>The Problem:</strong> For each customer, find their first purchase date and the product they bought.</p><pre>SELECT DISTINCT<br>    customer_id,<br>    FIRST_VALUE(product_name) OVER (<br>        PARTITION BY customer_id <br>        ORDER BY purchase_date<br>        ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING<br>    ) AS first_product,<br>    FIRST_VALUE(purchase_date) OVER (<br>        PARTITION BY customer_id <br>        ORDER BY purchase_date<br>    ) AS first_purchase_date<br>FROM purchases;</pre><p><strong>Alternative Approach:</strong> Use QUALIFY in databases that support it (Snowflake, BigQuery).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*WJITr-i6QR3poo2U" /></figure><h3>Part 2: Complex Joins That Test Your Logic</h3><h3>Scenario 6: Self-Joins for Hierarchical Data</h3><p><strong>The Problem:</strong> Find all employees and their managers’ names in a single query.</p><pre>SELECT <br>    e.employee_name,<br>    e.department,<br>    m.employee_name AS manager_name<br>FROM employees e<br>LEFT JOIN employees m ON e.manager_id = m.employee_id;</pre><p><strong>Advanced Version:</strong> Find the entire management chain (recursive CTE).</p><pre>WITH RECURSIVE management_chain AS (<br>    -- Base case: top-level managers<br>    SELECT employee_id, employee_name, manager_id, 1 AS level<br>    FROM employees<br>    WHERE manager_id IS NULL<br>    <br>    UNION ALL<br>    <br>    -- Recursive case<br>    SELECT e.employee_id, e.employee_name, e.manager_id, mc.level + 1<br>    FROM employees e<br>    INNER JOIN management_chain mc ON e.manager_id = mc.employee_id<br>)<br>SELECT * FROM management_chain ORDER BY level, employee_name;</pre><h3>Scenario 7: Finding Missing Data Across Tables</h3><p><strong>The Problem:</strong> Find all products that have never been ordered.</p><pre>-- Method 1: LEFT JOIN with NULL check<br>SELECT p.product_id, p.product_name<br>FROM products p<br>LEFT JOIN orders o ON p.product_id = o.product_id<br>WHERE o.order_id IS NULL;</pre><h3>Scenario 8: Handling Many-to-Many Relationships</h3><p><strong>The Problem:</strong> Find all students enrolled in both ‘Math 101’ AND ‘Physics 101’.</p><pre>SELECT s.student_id, s.student_name<br>FROM students s<br>INNER JOIN enrollments e ON s.student_id = e.student_id<br>INNER JOIN courses c ON e.course_id = c.course_id<br>WHERE c.course_name IN (&#39;Math 101&#39;, &#39;Physics 101&#39;)<br>GROUP BY s.student_id, s.student_name<br>HAVING COUNT(DISTINCT c.course_name) = 2;</pre><p><strong>Why This Pattern Matters:</strong> AND conditions across rows are tricky. Most candidates write OR logic by mistake.</p><h3>Part 3: Aggregation Mastery</h3><h3>Scenario 9: Conditional Aggregation</h3><p><strong>The Problem:</strong> Create a pivot-style report showing monthly sales by product category.</p><pre>SELECT <br>    product_category,<br>    SUM(CASE WHEN EXTRACT(MONTH FROM sale_date) = 1 THEN amount ELSE 0 END) AS jan_sales,<br>    SUM(CASE WHEN EXTRACT(MONTH FROM sale_date) = 2 THEN amount ELSE 0 END) AS feb_sales,<br>    SUM(CASE WHEN EXTRACT(MONTH FROM sale_date) = 3 THEN amount ELSE 0 END) AS mar_sales<br>FROM sales<br>WHERE EXTRACT(YEAR FROM sale_date) = 2024<br>GROUP BY product_category;</pre><p><strong>Modern Alternative:</strong> Use FILTER clause (PostgreSQL) or PIVOT (SQL Server, Oracle).</p><h3>Scenario 10: Calculating Percentiles and Medians</h3><p><strong>The Problem:</strong> Find the median salary for each department.</p><pre>-- PostgreSQL / Modern SQL<br>SELECT <br>    department,<br>    PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY salary) AS median_salary<br>FROM employees<br>GROUP BY department;<br>-- Alternative using window functions<br>WITH ranked AS (<br>    SELECT <br>        department,<br>        salary,<br>        ROW_NUMBER() OVER (PARTITION BY department ORDER BY salary) AS rn,<br>        COUNT(*) OVER (PARTITION BY department) AS cnt<br>    FROM employees<br>)<br>SELECT <br>    department,<br>    AVG(salary) AS median_salary<br>FROM ranked<br>WHERE rn IN (cnt/2, cnt/2 + 1, (cnt + 1)/2)<br>GROUP BY department;</pre><h3>Scenario 11: Finding Duplicates With Nuance</h3><p><strong>The Problem:</strong> Find duplicate orders (same customer, same product, within 5 minutes).</p><pre>SELECT <br>    o1.order_id AS order1_id,<br>    o2.order_id AS order2_id,<br>    o1.customer_id,<br>    o1.product_id,<br>    o1.order_time,<br>    o2.order_time<br>FROM orders o1<br>INNER JOIN orders o2 <br>    ON o1.customer_id = o2.customer_id<br>    AND o1.product_id = o2.product_id<br>    AND o1.order_id &lt; o2.order_id  -- Avoid self-matches and duplicate pairs<br>    AND o2.order_time BETWEEN o1.order_time AND o1.order_time + INTERVAL &#39;5 minutes&#39;;</pre><h3>Part 4: Performance and Optimization</h3><h3>Scenario 12: Query Optimization Challenge</h3><p><strong>The Problem:</strong> This query is slow. How would you optimize it?</p><pre>-- SLOW VERSION<br>SELECT *<br>FROM orders o<br>WHERE o.customer_id IN (<br>    SELECT customer_id <br>    FROM customers <br>    WHERE country = &#39;USA&#39;<br>)<br>AND o.order_date &gt;= &#39;2024-01-01&#39;;</pre><p><strong>Optimized Approaches:</strong></p><pre>-- Approach 1: JOIN instead of IN<br>SELECT o.*<br>FROM orders o<br>INNER JOIN customers c ON o.customer_id = c.customer_id<br>WHERE c.country = &#39;USA&#39;<br>AND o.order_date &gt;= &#39;2024-01-01&#39;;<br>-- Approach 2: EXISTS (if you don&#39;t need customer data)<br>SELECT o.*<br>FROM orders o<br>WHERE EXISTS (<br>    SELECT 1 FROM customers c <br>    WHERE c.customer_id = o.customer_id <br>    AND c.country = &#39;USA&#39;<br>)<br>AND o.order_date &gt;= &#39;2024-01-01&#39;;</pre><p><strong>What to Discuss:</strong></p><ul><li>Index on orders(order_date, customer_id)</li><li>Index on customers(country, customer_id)</li><li>Partitioning by order_date for large tables</li><li>Statistics freshness</li></ul><h3>Scenario 13: Explain Plan Reading</h3><p><strong>The Interview Setup:</strong> You’re shown an explain plan with a full table scan on a 100M row table.</p><p><strong>What They Want to Hear:</strong></p><ol><li>Check if proper indexes exist</li><li>Look for implicit type conversions</li><li>Check statistics/cardinality estimates</li><li>Consider query rewrite (push predicates down)</li><li>Evaluate partitioning strategies</li></ol><h3>Scenario 14: Handling Large IN Clauses</h3><p><strong>The Problem:</strong> You need to filter by 10,000 IDs. How do you do it efficiently?</p><pre>-- Bad: Giant IN clause<br>SELECT * FROM orders WHERE order_id IN (1, 2, 3, ... 10000);<br>-- Better: Temporary table or CTE<br>WITH target_ids AS (<br>    SELECT unnest(ARRAY[1, 2, 3, ...]) AS order_id<br>)<br>SELECT o.*<br>FROM orders o<br>INNER JOIN target_ids t ON o.order_id = t.order_id;<br>-- Best (in production): Load IDs to a temp table, then join<br>CREATE TEMP TABLE target_ids (order_id INT PRIMARY KEY);<br>-- Bulk insert IDs<br>SELECT o.* FROM orders o INNER JOIN target_ids t ON o.order_id = t.order_id;</pre><h3>Part 5: Data Transformation Scenarios</h3><h3>Scenario 15: String Parsing and Extraction</h3><p><strong>The Problem:</strong> Parse a URL to extract the domain.</p><pre>-- PostgreSQL<br>SELECT <br>    url,<br>    SUBSTRING(url FROM &#39;https?://([^/]+)&#39;) AS domain<br>FROM web_logs;<br>-- Using SPLIT_PART<br>SELECT <br>    url,<br>    SPLIT_PART(SPLIT_PART(url, &#39;//&#39;, 2), &#39;/&#39;, 1) AS domain<br>FROM web_logs;</pre><h3>Scenario 16: JSON/JSONB Operations</h3><p><strong>The Problem:</strong> Extract nested values from a JSON column.</p><pre>-- PostgreSQL JSONB<br>SELECT <br>    order_id,<br>    order_data-&gt;&gt;&#39;customer_name&#39; AS customer_name,<br>    (order_data-&gt;&#39;items&#39;-&gt;0-&gt;&gt;&#39;price&#39;)::NUMERIC AS first_item_price,<br>    jsonb_array_length(order_data-&gt;&#39;items&#39;) AS item_count<br>FROM orders<br>WHERE order_data @&gt; &#39;{&quot;status&quot;: &quot;completed&quot;}&#39;;<br>-- Explode JSON array into rows<br>SELECT <br>    order_id,<br>    item-&gt;&gt;&#39;product_name&#39; AS product_name,<br>    (item-&gt;&gt;&#39;quantity&#39;)::INT AS quantity<br>FROM orders,<br>LATERAL jsonb_array_elements(order_data-&gt;&#39;items&#39;) AS item;</pre><h3>Scenario 17: Date Dimension Calculations</h3><p><strong>The Problem:</strong> Calculate fiscal quarters (fiscal year starts in April).</p><pre>SELECT <br>    order_date,<br>    CASE <br>        WHEN EXTRACT(MONTH FROM order_date) &gt;= 4 <br>        THEN EXTRACT(YEAR FROM order_date)<br>        ELSE EXTRACT(YEAR FROM order_date) - 1<br>    END AS fiscal_year,<br>    CASE <br>        WHEN EXTRACT(MONTH FROM order_date) IN (4,5,6) THEN &#39;Q1&#39;<br>        WHEN EXTRACT(MONTH FROM order_date) IN (7,8,9) THEN &#39;Q2&#39;<br>        WHEN EXTRACT(MONTH FROM order_date) IN (10,11,12) THEN &#39;Q3&#39;<br>        ELSE &#39;Q4&#39;<br>    END AS fiscal_quarter<br>FROM orders;</pre><h3>Part 6: Real-World Business Logic</h3><h3>Scenario 18: Funnel Analysis</h3><p><strong>The Problem:</strong> Calculate conversion rates through a signup funnel.</p><pre>WITH funnel_events AS (<br>    SELECT <br>        user_id,<br>        MAX(CASE WHEN event_type = &#39;page_view&#39; THEN 1 ELSE 0 END) AS viewed,<br>        MAX(CASE WHEN event_type = &#39;signup_start&#39; THEN 1 ELSE 0 END) AS started,<br>        MAX(CASE WHEN event_type = &#39;signup_complete&#39; THEN 1 ELSE 0 END) AS completed<br>    FROM user_events<br>    WHERE event_date = &#39;2024-01-15&#39;<br>    GROUP BY user_id<br>)<br>SELECT <br>    COUNT(*) AS total_users,<br>    SUM(viewed) AS viewers,<br>    SUM(started) AS starters,<br>    SUM(completed) AS completers,<br>    ROUND(SUM(started) * 100.0 / NULLIF(SUM(viewed), 0), 2) AS view_to_start_rate,<br>    ROUND(SUM(completed) * 100.0 / NULLIF(SUM(started), 0), 2) AS start_to_complete_rate<br>FROM funnel_events;</pre><h3>Scenario 19: Retention Cohort Analysis</h3><p><strong>The Problem:</strong> Calculate 30-day retention by signup cohort.</p><pre>WITH first_actions AS (<br>    SELECT <br>        user_id,<br>        DATE_TRUNC(&#39;week&#39;, MIN(action_date)) AS cohort_week<br>    FROM user_actions<br>    GROUP BY user_id<br>),<br>retention AS (<br>    SELECT <br>        f.cohort_week,<br>        COUNT(DISTINCT f.user_id) AS cohort_size,<br>        COUNT(DISTINCT CASE <br>            WHEN a.action_date BETWEEN f.cohort_week AND f.cohort_week + INTERVAL &#39;30 days&#39;<br>            THEN f.user_id <br>        END) AS retained_30d<br>    FROM first_actions f<br>    LEFT JOIN user_actions a ON f.user_id = a.user_id<br>    GROUP BY f.cohort_week<br>)<br>SELECT <br>    cohort_week,<br>    cohort_size,<br>    retained_30d,<br>    ROUND(retained_30d * 100.0 / cohort_size, 2) AS retention_rate<br>FROM retention<br>ORDER BY cohort_week;</pre><h3>Scenario 20: Sessionization</h3><p><strong>The Problem:</strong> Group user events into sessions (30-minute inactivity timeout).</p><pre>WITH event_gaps AS (<br>    SELECT <br>        user_id,<br>        event_time,<br>        event_type,<br>        LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS prev_event_time<br>    FROM user_events<br>),<br>session_markers AS (<br>    SELECT <br>        *,<br>        CASE <br>            WHEN prev_event_time IS NULL <br>                OR event_time - prev_event_time &gt; INTERVAL &#39;30 minutes&#39;<br>            THEN 1 <br>            ELSE 0 <br>        END AS new_session<br>    FROM event_gaps<br>),<br>sessions AS (<br>    SELECT <br>        *,<br>        SUM(new_session) OVER (<br>            PARTITION BY user_id <br>            ORDER BY event_time<br>        ) AS session_id<br>    FROM session_markers<br>)<br>SELECT <br>    user_id,<br>    session_id,<br>    MIN(event_time) AS session_start,<br>    MAX(event_time) AS session_end,<br>    COUNT(*) AS event_count<br>FROM sessions<br>GROUP BY user_id, session_id;</pre><h3>Part 7: Advanced Patterns</h3><h3>Scenario 21: Slowly Changing Dimensions (SCD Type 2)</h3><p><strong>The Problem:</strong> Track historical changes to customer records.</p><pre>-- Finding current record<br>SELECT * <br>FROM customer_dim <br>WHERE customer_id = 123 <br>AND is_current = TRUE;<br>-- Finding record at a specific point in time<br>SELECT * <br>FROM customer_dim <br>WHERE customer_id = 123 <br>AND &#39;2024-01-15&#39; BETWEEN effective_start_date AND effective_end_date;<br>-- Inserting a new change<br>WITH deactivate AS (<br>    UPDATE customer_dim<br>    SET is_current = FALSE,<br>        effective_end_date = CURRENT_DATE - 1<br>    WHERE customer_id = 123 AND is_current = TRUE<br>    RETURNING customer_id<br>)<br>INSERT INTO customer_dim (customer_id, customer_name, city, effective_start_date, effective_end_date, is_current)<br>VALUES (123, &#39;John Doe&#39;, &#39;New York&#39;, CURRENT_DATE, &#39;9999-12-31&#39;, TRUE);</pre><h3>Scenario 22: Handling NULL Comparisons</h3><p><strong>The Problem:</strong> Find records where a value changed (including NULL to non-NULL).</p><pre>SELECT *<br>FROM audit_log<br>WHERE (old_value IS DISTINCT FROM new_value);<br>-- Equivalent without IS DISTINCT FROM<br>SELECT *<br>FROM audit_log<br>WHERE old_value != new_value <br>   OR (old_value IS NULL AND new_value IS NOT NULL)<br>   OR (old_value IS NOT NULL AND new_value IS NULL);</pre><h3>Scenario 23: Recursive CTEs for Bill of Materials</h3><p><strong>The Problem:</strong> Calculate total component cost for a product with nested sub-assemblies.</p><pre>WITH RECURSIVE bom_explosion AS (<br>    -- Base: top-level product<br>    SELECT <br>        component_id,<br>        parent_id,<br>        component_name,<br>        quantity,<br>        unit_cost,<br>        quantity * unit_cost AS line_cost,<br>        1 AS level<br>    FROM bill_of_materials<br>    WHERE parent_id = &#39;FINISHED_PRODUCT_001&#39;<br>    <br>    UNION ALL<br>    <br>    -- Recursive: sub-components<br>    SELECT <br>        b.component_id,<br>        b.parent_id,<br>        b.component_name,<br>        b.quantity * be.quantity AS quantity,<br>        b.unit_cost,<br>        b.quantity * be.quantity * b.unit_cost AS line_cost,<br>        be.level + 1<br>    FROM bill_of_materials b<br>    INNER JOIN bom_explosion be ON b.parent_id = be.component_id<br>    WHERE be.level &lt; 10  -- Safety limit<br>)<br>SELECT <br>    component_id,<br>    component_name,<br>    SUM(quantity) AS total_quantity,<br>    SUM(line_cost) AS total_cost<br>FROM bom_explosion<br>GROUP BY component_id, component_name<br>ORDER BY total_cost DESC;</pre><h3>Scenario 24: Generating Date Series for Gap Filling</h3><p><strong>The Problem:</strong> Show daily sales with zeros for days with no sales.</p><pre>WITH date_series AS (<br>    SELECT generate_series(<br>        &#39;2024-01-01&#39;::DATE,<br>        &#39;2024-01-31&#39;::DATE,<br>        &#39;1 day&#39;::INTERVAL<br>    )::DATE AS sale_date<br>)<br>SELECT <br>    ds.sale_date,<br>    COALESCE(SUM(s.amount), 0) AS daily_sales<br>FROM date_series ds<br>LEFT JOIN sales s ON ds.sale_date = s.sale_date<br>GROUP BY ds.sale_date<br>ORDER BY ds.sale_date;</pre><h3>Scenario 25: Data Quality Checks</h3><p><strong>The Problem:</strong> Write queries to validate data quality before a pipeline runs.</p><pre>-- Check 1: Null values in required fields<br>SELECT &#39;NULL customer_id&#39; AS check_name, COUNT(*) AS failures<br>FROM orders WHERE customer_id IS NULL<br>UNION ALL<br>-- Check 2: Referential integrity<br>SELECT &#39;orphan orders&#39;, COUNT(*)<br>FROM orders o<br>LEFT JOIN customers c ON o.customer_id = c.customer_id<br>WHERE c.customer_id IS NULL<br>UNION ALL<br>-- Check 3: Duplicate primary keys<br>SELECT &#39;duplicate order_id&#39;, COUNT(*)<br>FROM (<br>    SELECT order_id <br>    FROM orders <br>    GROUP BY order_id <br>    HAVING COUNT(*) &gt; 1<br>) dups<br>UNION ALL<br>-- Check 4: Values outside expected range<br>SELECT &#39;negative amounts&#39;, COUNT(*)<br>FROM orders WHERE amount &lt; 0<br>UNION ALL<br>-- Check 5: Freshness (data from last 24 hours)<br>SELECT &#39;stale data&#39;, <br>    CASE WHEN MAX(created_at) &lt; NOW() - INTERVAL &#39;24 hours&#39; <br>         THEN 1 ELSE 0 END<br>FROM orders;</pre><h3>Key Takeaways for Your Interview</h3><ol><li><strong>Window functions are non-negotiable.</strong> Practice ROW_NUMBER(), RANK(), LAG(), LEAD(), and frame specifications until they&#39;re second nature.</li><li><strong>Know multiple approaches.</strong> For any problem, be ready to discuss trade-offs between CTEs, subqueries, and JOINs.</li><li><strong>Think about performance.</strong> Always mention indexing, partitioning, and query plan analysis.</li><li><strong>Handle edge cases.</strong> NULLs, duplicates, and empty results address them proactively.</li><li><strong>Connect to business context.</strong> Explain <em>why</em> your query matters, not just <em>how</em> it works.</li></ol><h3>What’s Next?</h3><p>If you found this helpful, you might also like:</p><ul><li><strong>PySpark Scenarios for Data Engineering Interviews</strong></li><li><strong>Apache Airflow: 15 DAG Patterns Every Data Engineer Should Know</strong></li><li><strong>Data Modeling Interview Questions That Actually Get Asked</strong></li></ul><p><em>Did this help with your interview prep? Drop a comment below with the SQL topic that trips you up the most, and I might cover it in a future post.</em></p><p><strong>Follow me</strong> for more data engineering content. I post weekly deep-dives on SQL, Spark, Airflow, and system design.</p><p><strong>Tags:</strong> #SQL #DataEngineering #TechInterviews #FAANG #DataScience #BigData #CareerAdvice #InterviewPrep #WindowFunctions #QueryOptimization</p><p><em>Originally published on Medium. If you’re preparing for data engineering interviews, check out my complete interview preparation guide.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17e4c80559a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The SQL Question 80% of Candidates Fail at Meta Interviews]]></title>
            <link>https://medium.com/@dharmatejasamudrala/the-sql-question-80-of-candidates-fail-at-meta-data-engineering-interviews-8761645343dc?source=rss-c037363a37db------2</link>
            <guid isPermaLink="false">https://medium.com/p/8761645343dc</guid>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[meta]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Dharma]]></dc:creator>
            <pubDate>Tue, 30 Dec 2025 18:31:33 GMT</pubDate>
            <atom:updated>2026-01-08T16:55:22.630Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*deIG5zpGQax2GETa" /><figcaption>Photo by <a href="https://unsplash.com/@maria_shalabaieva?utm_source=medium&amp;utm_medium=referral">Mariia Shalabaieva</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’ve seen this question destroy candidates who can write complex joins in their sleep.</p><p>It looks simple. It’s not.</p><p>Meta, along with other BigTech companies, loves this question because it tests pattern recognition, edge case thinking, and whether you’ve encountered real-world data problems not just textbook SQL.</p><p>Let’s break it down.</p><h3>The Question</h3><p><em>“Given a </em><em>logins table, find all users who logged in for at least 3 consecutive days.&quot;</em></p><pre>-- logins table<br>user_id | login_date<br>--------|------------<br>1       | 2024-01-01<br>1       | 2024-01-02<br>1       | 2024-01-03<br>1       | 2024-01-05<br>2       | 2024-01-01<br>2       | 2024-01-03<br>3       | 2024-01-10<br>3       | 2024-01-11<br>3       | 2024-01-12<br>3       | 2024-01-13</pre><p><strong>Expected output:</strong> Users 1 and 3</p><p>User 2 has a gap (logged in Jan 1, skipped Jan 2, logged in Jan 3). They’re out.</p><p>Sounds straightforward, right?</p><h3>Why Most Candidates Fail</h3><p>I’ve reviewed dozens of interview solutions for this problem. Here’s where people go wrong:</p><h3>Mistake #1: The Self-Join Nightmare</h3><p>Many candidates immediately reach for self-joins:</p><pre>SELECT DISTINCT a.user_id<br>FROM logins a<br>JOIN logins b ON a.user_id = b.user_id AND b.login_date = a.login_date + 1<br>JOIN logins c ON a.user_id = c.user_id AND c.login_date = a.login_date + 2</pre><p>This technically works for exactly 3 days. But what if the question changes to 5 consecutive days? 10? You’re adding more joins, more room for error, and the interviewer is already mentally marking you down.</p><p>It’s also expensive at scale — and Meta operates at <em>serious</em> scale.</p><h3>Mistake #2: LAG/LEAD Confusion</h3><p>Window function enthusiasts try something like:</p><pre>SELECT user_id<br>FROM (<br>    SELECT <br>        user_id,<br>        login_date,<br>        LAG(login_date, 1) OVER (PARTITION BY user_id ORDER BY login_date) AS prev_1,<br>        LAG(login_date, 2) OVER (PARTITION BY user_id ORDER BY login_date) AS prev_2<br>    FROM logins<br>) t<br>WHERE login_date - prev_1 = 1 AND prev_1 - prev_2 = 1</pre><p>This can work but gets ugly fast. You’re hardcoding the consecutive count, and extending it to N days becomes painful.</p><h3>Mistake #3: Forgetting Duplicates</h3><p>Here’s the silent killer.</p><p>What if a user logs in twice on the same day? Your row numbers get thrown off, your date math breaks, and your output is wrong.</p><p>Production data is messy. Interviewers know this. They’re watching to see if you handle it.</p><h3>Mistake #4: Off-by-One Errors</h3><p>You’d be surprised how many candidates return users with 2 consecutive days instead of 3. Under interview pressure, mental math gets fuzzy.</p><h3>The Clean Solution</h3><p>The trick is recognizing this as an <strong>islands and gaps</strong> problem — a classic pattern in time-series SQL.</p><p>Here’s the insight: if you have consecutive dates and subtract a row number from each date, the result is constant for all dates in that consecutive sequence.</p><p>Let me show you:</p><pre>login_date  | row_number | login_date - row_number<br>------------|------------|-------------------------<br>2024-01-10  | 1          | 2024-01-09<br>2024-01-11  | 2          | 2024-01-09<br>2024-01-12  | 3          | 2024-01-09<br>2024-01-13  | 4          | 2024-01-09</pre><p>See how the last column stays 2024-01-09 for all consecutive dates? That&#39;s your grouping key.</p><p>Here’s the full solution:</p><pre>WITH deduped AS (<br>    SELECT DISTINCT user_id, login_date<br>    FROM logins<br>),<br>grouped AS (<br>    SELECT <br>        user_id,<br>        login_date,<br>        login_date - INTERVAL &#39;1 day&#39; * ROW_NUMBER() OVER (<br>            PARTITION BY user_id <br>            ORDER BY login_date<br>        ) AS grp<br>    FROM deduped<br>)<br>SELECT DISTINCT user_id<br>FROM grouped<br>GROUP BY user_id, grp<br>HAVING COUNT(*) &gt;= 3;</pre><p><strong>Step-by-step:</strong></p><ol><li><strong>Dedupe first</strong> — removes multiple logins on the same day</li><li><strong>Create the grouping key</strong> — subtract row number (as days) from login date</li><li><strong>Group and count</strong> — find groups with 3+ consecutive days</li><li><strong>Return distinct users</strong> — a user might have multiple consecutive streaks</li></ol><h3>Why Interviewers Love This Question</h3><p>It’s not about memorizing syntax. This question reveals:</p><ul><li><strong>Pattern recognition</strong> do you know islands and gaps, or are you reinventing the wheel?</li><li><strong>Edge case awareness</strong> did you think about duplicates without being prompted?</li><li><strong>Scalability mindset</strong> is your solution efficient at Meta’s scale?</li><li><strong>Adaptability</strong> can you easily change “3 days” to “N days”?</li></ul><p>The self-join approach isn’t wrong, but it signals junior-level thinking. The islands and gaps approach signals you’ve worked with real time-series data before.</p><h3>How to Practice This</h3><ol><li>Understand the pattern deeply not just this specific question</li><li>Practice variations: 5 consecutive days, consecutive weeks, consecutive months</li><li>Always ask about duplicates in interviews it shows maturity</li><li>Write the solution from scratch multiple times until it’s muscle memory</li></ol><h3>Final Thought</h3><p>The candidates who pass Meta interviews aren’t necessarily smarter. They’ve just seen more patterns.</p><p>Islands and gaps is one of maybe 10–15 SQL patterns that cover 90% of interview questions. Learn those patterns, and questions like this become routine.</p><p>Good luck.</p><p><em>Preparing for data engineering interviews? Follow me for more breakdowns of real interview questions from Meta, Google, and other BigTech companies.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8761645343dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Data Engineering Skills That Will Get You Hired in 2026]]></title>
            <link>https://medium.com/@dharmatejasamudrala/5-data-engineering-skills-that-will-get-you-hired-in-2026-3c1b150dc39e?source=rss-c037363a37db------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c1b150dc39e</guid>
            <category><![CDATA[2026]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[hired]]></category>
            <category><![CDATA[skills]]></category>
            <dc:creator><![CDATA[Dharma]]></dc:creator>
            <pubDate>Mon, 29 Dec 2025 16:40:12 GMT</pubDate>
            <atom:updated>2025-12-29T16:42:56.913Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tYfers8MehRO95Y5.jpg" /></figure><h3>TL;DR — Skills That Will Land You a Data Engineering Role in 2026</h3><ul><li><strong>Skill 0: AI &amp; GenAI Fluency</strong> — Work alongside AI copilots, build AI-ready pipelines, leverage LLMs for automation</li><li><strong>Skill 1: Cloud Platform Mastery</strong> — AWS, Azure, or GCP expertise is now table stakes</li><li><strong>Skill 2: Real-Time Streaming</strong> — Kafka, Spark Streaming, and Flink for instant data processing</li><li><strong>Skill 3: Modern Data Stack (dbt + Snowflake)</strong> — ELT transformation frameworks and cloud warehouses</li><li><strong>Skill 4: Python + SQL Fundamentals</strong> — The unshakable foundation everything else builds on</li><li><strong>Skill 5: DataOps &amp; Orchestration</strong> — Airflow, CI/CD pipelines, and infrastructure as code</li></ul><p>The data engineering job market is exploding. With a 23% growth rate and over 150,000 professionals employed in the field, data engineering has become one of the hottest career paths in tech. Companies are creating positions faster than qualified candidates can fill them, and salaries are climbing toward $170,000+ for experienced engineers.</p><p>But here’s the catch: the skills that got you hired in 2023 won’t necessarily land you a role in 2026. The landscape is shifting dramatically, and if you’re not adapting, you’re falling behind.</p><p>I’ve analyzed the latest job postings, industry reports, and emerging trends to identify the six skills (yes, we’re including Skill 0) that will make you irresistible to hiring managers in 2026. Let’s dive in.</p><h3>Skill 0: AI &amp; Generative AI Fluency</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DBYjY3NNJn5rEdvS" /></figure><p><em>Photo by </em><a href="https://unsplash.com/@growtika"><em>Growtika</em></a><em> on Unsplash</em></p><p>This is Skill 0 because it’s not optional anymore — it’s the lens through which all other skills must be viewed.</p><p>Generative AI isn’t replacing data engineers. Instead, it’s transforming how we work. AI-powered tools can now automate code generation, cutting development time by 45–50%. They can generate complete data pipelines from natural language specifications, detect anomalies, and even self-heal failing pipelines.</p><h3>What This Means for You</h3><p><strong>Learn to work with AI copilots.</strong> Tools like GitHub Copilot, Amazon CodeWhisperer, and Databricks’ AI assistants are becoming standard. Engineers who can effectively prompt these tools, validate their outputs, and integrate AI-generated code into production workflows will be significantly more productive.</p><p><strong>Build AI-ready pipelines.</strong> Your data infrastructure now needs to support model training, RAG (Retrieval-Augmented Generation) systems, and real-time inference. Understanding how data flows into AI systems — and how to maintain quality, lineage, and governance — is crucial.</p><p><strong>Understand the limitations.</strong> As one industry expert put it: “Prompt-driven tools are great at writing code but they can’t reason about business logic, trade-offs in system design, or the subtle cost of a slow query in a production dashboard.” Your judgment and domain expertise remain irreplaceable.</p><h3>Key Technologies to Learn</h3><ul><li><strong>dbt Copilot</strong> — AI-assisted model, test, and documentation creation</li><li><strong>AWS Glue with AI</strong> — Auto-generated ETL pipelines</li><li><strong>LangChain/LlamaIndex</strong> — For building RAG-enabled data applications</li><li><strong>Monte Carlo</strong> — AI-powered data observability and anomaly detection</li></ul><h3>Skill 1: Cloud Platform Mastery</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kbmmH7xtX3ujGQMm.png" /></figure><p>Over 94% of enterprises have embraced cloud technologies. If you’re not fluent in at least one major cloud platform, you’re essentially unemployable as a data engineer in 2026.</p><h3>The Big Three</h3><p><strong>AWS (Amazon Web Services)</strong> remains the market leader with approximately 32% market share. AWS data engineers typically earn $115,000–$145,000 annually, and AWS certification is mentioned in 4.2% of all job postings — more than any other cloud credential.</p><p><strong>Microsoft Azure</strong> has seen explosive growth, particularly in enterprise environments that already use Microsoft products. Azure’s strength lies in hybrid cloud deployments and integration with tools like Power BI, Active Directory, and the broader Microsoft ecosystem.</p><p><strong>Google Cloud Platform (GCP)</strong> is the dark horse, particularly strong in data analytics and machine learning. If you’re targeting roles at companies doing cutting-edge AI work, GCP expertise (especially BigQuery and Vertex AI) can set you apart.</p><h3>What to Focus On</h3><p>Don’t just learn the basics. Deep expertise in these areas will differentiate you:</p><ul><li><strong>Serverless architectures</strong> — AWS Lambda, Azure Functions, GCP Cloud Functions</li><li><strong>Data warehousing</strong> — Redshift, Synapse, BigQuery</li><li><strong>Data lakes</strong> — S3, Azure Data Lake, Cloud Storage</li><li><strong>ETL services</strong> — Glue, Data Factory, Dataflow</li><li><strong>Infrastructure as Code</strong> — Terraform, CloudFormation, Pulumi</li></ul><h3>Pro Tip</h3><p>Multi-cloud skills are increasingly valuable. Companies are adopting hybrid and multi-cloud strategies to avoid vendor lock-in. If you can navigate across AWS, Azure, AND GCP, you become significantly more versatile.</p><h3>Skill 2: Real-Time Streaming</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ILYNPz_Q0WbPiWkF" /></figure><p><em>Photo by </em><a href="https://unsplash.com/@jsnbrsc"><em>Jason Briscoe</em></a><em> on Unsplash</em></p><p>The era of batch processing is giving way to real-time. Companies want instant insights from streaming data — whether it’s user activity on apps, IoT sensor feeds, or live financial transactions. The real-time analytics market is growing at over 20% annually.</p><h3>The Streaming Stack</h3><p><strong>Apache Kafka</strong> is the highway for data — a distributed streaming platform that ingests data from various sources and reliably delivers it to multiple consumers. Many job postings specifically list Kafka experience as required. Mastering topics, partitions, brokers, and consumer groups will let you design systems that scale to millions of events per second.</p><p><strong>Apache Spark Streaming</strong> (now Structured Streaming) is the processing engine. It handles continuous data as micro-batches, leveraging Spark’s distributed architecture for scalability and fault tolerance. Spark Streaming combined with Kafka forms a powerful duo for processing real-time data at scale.</p><p><strong>Apache Flink</strong> is gaining ground for true stream processing (event-by-event rather than micro-batches). It’s particularly strong for applications requiring exactly-once processing guarantees and complex event processing.</p><h3>Real-World Applications</h3><ul><li><strong>Fraud detection</strong> — Analyze transactions in real-time to flag suspicious activity</li><li><strong>Recommendation systems</strong> — Update suggestions based on user behavior as it happens</li><li><strong>IoT analytics</strong> — Process sensor data from millions of devices</li><li><strong>Live dashboards</strong> — Power real-time business intelligence</li></ul><h3>What to Build</h3><p>Create a project that demonstrates your streaming skills. Build a pipeline that:</p><ol><li>Ingests data from Kafka topics</li><li>Performs transformations with Spark Streaming</li><li>Handles late or out-of-order data with watermarking</li><li>Writes results to both real-time dashboards and cold storage</li></ol><p>This kind of hands-on experience is gold on a resume.</p><h3>Skill 3: Modern Data Stack (dbt + Snowflake)</h3><p>The rise of ELT (Extract, Load, Transform) has fundamentally changed how data engineers approach transformation work. Rather than transforming data before loading it into storage, modern practices emphasize loading raw data first and performing transformations within the data warehouse itself.</p><p>At the center of this revolution: <strong>dbt (data build tool)</strong> and <strong>Snowflake</strong>.</p><h3>Why dbt Is Everywhere</h3><p>dbt has become central to the data engineer’s toolkit. It brings software engineering best practices to analytics:</p><ul><li><strong>Version control</strong> — Your transformation logic lives in Git</li><li><strong>Modular SQL</strong> — Build reusable models with clear dependencies</li><li><strong>Automated testing</strong> — Validate data quality with built-in and custom tests</li><li><strong>Documentation</strong> — Generate living documentation alongside your code</li><li><strong>CI/CD integration</strong> — Deploy with confidence using automated pipelines</li></ul><p>With the recent dbt Labs and Fivetran merger, the tool’s dominance is only growing. dbt is evolving into a broader data control plane with features like the dbt Semantic Layer for AI applications and dbt Agents for automated workflows.</p><h3>Why Snowflake Dominates</h3><p>Snowflake has emerged as the cloud data warehouse of choice for many organizations. Its separation of storage and compute, automatic scaling, and support for semi-structured data make it incredibly powerful. Combined with dbt, it enables:</p><ul><li><strong>Modular, governed transformation pipelines</strong></li><li><strong>Cost optimization</strong> through intelligent resource allocation</li><li><strong>Near-real-time analytics</strong> with features like Dynamic Tables</li><li><strong>AI-ready data</strong> with native machine learning integrations</li></ul><h3>The Winning Combo</h3><p>The Fivetran + dbt + Snowflake stack is becoming the standard for modern data teams:</p><ol><li><strong>Fivetran</strong> extracts data from 700+ sources and loads it into Snowflake</li><li><strong>dbt</strong> transforms raw data into clean, analytics-ready models</li><li><strong>Snowflake</strong> provides scalable storage and compute</li></ol><p>Learn this stack, and you’ll be qualified for a huge percentage of data engineering roles in 2026.</p><h3>Skill 4: Python + SQL Fundamentals</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SEHZWXQgxrW-zJs_" /></figure><p><em>Photo by </em><a href="https://unsplash.com/@cdr6934"><em>Chris Ried</em></a><em> on Unsplash</em></p><p>With all the shiny new tools, it’s easy to forget that Python and SQL remain the bedrock of data engineering. These aren’t going away — in fact, they’re more important than ever.</p><h3>SQL: Still King</h3><p>SQL appears in the vast majority of data engineering job postings. Advanced SQL skills are non-negotiable:</p><ul><li><strong>Complex joins and subqueries</strong></li><li><strong>Window functions</strong> (ROW_NUMBER, RANK, LAG, LEAD)</li><li><strong>CTEs (Common Table Expressions)</strong> for readable, modular queries</li><li><strong>Performance optimization</strong> — Understanding query plans, indexing, partitioning</li><li><strong>NoSQL databases</strong> — MongoDB, Cassandra, DynamoDB are increasingly important as data becomes more diverse</li></ul><h3>Python: The Swiss Army Knife</h3><p>Python was explicitly mentioned in 78% of data science job postings in 2023, and the trend continues for data engineers. Focus on:</p><ul><li><strong>Pandas and PySpark</strong> — Data manipulation at any scale</li><li><strong>API development</strong> — Building data services and endpoints</li><li><strong>Pipeline automation</strong> — Scripting ETL workflows</li><li><strong>Testing frameworks</strong> — pytest, Great Expectations</li><li><strong>Package management</strong> — Virtual environments, requirements, dependency management</li></ul><h3>The Unshakable Foundation</h3><p>Everything else in data engineering — streaming, cloud platforms, dbt, orchestration — builds on Python and SQL. Don’t neglect these fundamentals in pursuit of newer technologies. A senior engineer who writes clean, efficient SQL and Python will always be more valuable than a junior who can only configure tools.</p><h3>Skill 5: DataOps &amp; Orchestration</h3><p>DevOps practices have fully arrived in data engineering. CI/CD is mentioned in nearly 1 out of 6 job postings, and the ability to deploy, monitor, and maintain data pipelines reliably is now expected.</p><h3>Apache Airflow: The Standard</h3><p>Apache Airflow has emerged as the de facto standard for workflow orchestration. It lets you author, schedule, and monitor workflows programmatically using Python DAGs (Directed Acyclic Graphs). By 2026, knowing Airflow for data pipelines is practically required.</p><p>Key Airflow concepts to master:</p><ul><li><strong>DAG design patterns</strong> — Best practices for building maintainable workflows</li><li><strong>Operators</strong> — BashOperator, PythonOperator, cloud-specific operators</li><li><strong>Connections and hooks</strong> — Integrating with external systems</li><li><strong>Task dependencies and branching</strong></li><li><strong>Sensors</strong> — Waiting for external events</li></ul><h3>Beyond Airflow</h3><p>While Airflow dominates, alternatives are emerging:</p><ul><li><strong>Prefect</strong> — More Pythonic, easier to get started</li><li><strong>Dagster</strong> — Software-defined assets, strong type checking</li><li><strong>Temporal</strong> — For complex, long-running workflows</li></ul><h3>Infrastructure as Code</h3><p>Modern data engineers need to version and deploy infrastructure just like application code:</p><ul><li><strong>Terraform</strong> — Multi-cloud infrastructure management</li><li><strong>Docker</strong> — Containerizing data applications</li><li><strong>Kubernetes</strong> — Orchestrating containerized workloads</li><li><strong>GitHub Actions / GitLab CI</strong> — Automating deployment pipelines</li></ul><h3>Data Quality &amp; Observability</h3><p>Operational excellence means knowing when things go wrong before your stakeholders do:</p><ul><li><strong>Great Expectations</strong> — Automated data validation and testing</li><li><strong>Monte Carlo</strong> — AI-powered anomaly detection</li><li><strong>Datafold</strong> — Data diff and impact analysis</li><li><strong>dbt tests</strong> — Built-in data quality checks</li></ul><h3>The Path Forward</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kzIMA73T-o2j8FUQ" /></figure><p><em>Photo by </em><a href="https://unsplash.com/@craftedbygc"><em>Green Chameleon</em></a><em> on Unsplash</em></p><p>The data engineering field is both exciting and fast-moving. The good news? Demand continues to outpace supply, salaries are rising, and the work is genuinely interesting.</p><p>Here’s my recommended approach:</p><h3>For Beginners</h3><ol><li>Master SQL and Python first — no shortcuts</li><li>Learn one cloud platform deeply (AWS is often the safest starting choice)</li><li>Build projects with dbt and Snowflake (or BigQuery)</li><li>Get hands-on with Airflow for orchestration</li></ol><h3>For Experienced Engineers</h3><ol><li>Develop AI fluency — learn to work with copilots and build AI-ready pipelines</li><li>Add streaming skills (Kafka + Spark Streaming)</li><li>Deepen your cloud expertise into multi-cloud territory</li><li>Level up on DataOps and observability</li></ol><h3>For Everyone</h3><ul><li><strong>Build projects</strong>. Don’t just read about tools — use them</li><li><strong>Contribute to open source</strong>. It’s the best way to learn and get noticed</li><li><strong>Stay current</strong>. Follow dbt Labs, Snowflake, and Confluent blogs</li><li><strong>Network</strong>. Connect with other data engineers on LinkedIn and at meetups</li></ul><h3>Final Thoughts</h3><p>The median data engineering salary is projected to reach $170,000 by 2026, and there’s an estimated 2.9 million data-related job vacancies expected globally. The opportunity is enormous.</p><p>But opportunity favors the prepared. Start building these skills now, and by 2026, you’ll be exactly what companies are looking for.</p><p>What skill are you focusing on developing? Drop a comment below — I’d love to hear about your journey.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*nqDb--6hi_Bn35sa.jpg" /></figure><p><em>If you found this helpful, give it a clap 👏 and follow me for more data engineering content. I write about building production-grade data systems, career growth, and the evolving landscape of modern data platforms.</em></p><p><strong>Tags:</strong> #DataEngineering #CareerAdvice #TechSkills #AI #CloudComputing #Python #SQL #dbt #Snowflake #Kafka #ApacheSpark #DataOps</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c1b150dc39e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>