<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Thao Luu on Medium]]></title>
        <description><![CDATA[Stories by Thao Luu on Medium]]></description>
        <link>https://medium.com/@luulienthao0204?source=rss-c80780a2d74c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Thao Luu on Medium</title>
            <link>https://medium.com/@luulienthao0204?source=rss-c80780a2d74c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:50:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@luulienthao0204/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I Got Tricked: When a LEFT JOIN Becomes an INNER JOIN]]></title>
            <link>https://medium.com/@luulienthao0204/how-i-got-tricked-when-a-left-join-becomes-an-inner-join-802ccd9691ea?source=rss-c80780a2d74c------2</link>
            <guid isPermaLink="false">https://medium.com/p/802ccd9691ea</guid>
            <category><![CDATA[hive]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Thao Luu]]></dc:creator>
            <pubDate>Sat, 10 Oct 2026 15:13:44 GMT</pubDate>
            <atom:updated>2026-10-10T15:13:44.781Z</atom:updated>
            <content:encoded><![CDATA[<p><em>It was a logical mistake on my part! Hope you won’t judge me, LOL.</em></p><p>Let’s look at these two Hive queries.</p><p>Let’s call the first one <strong>Query #1</strong>:</p><pre>SELECT DISTINCT *<br>FROM tmp_customer_base a<br>LEFT JOIN d_customer_info b<br>  ON a.tax_code = b.tax_code<br>WHERE b.partition = &#39;20261006&#39;;</pre><p>And here’s <strong>Query #2</strong>:</p><pre>SELECT DISTINCT *<br>FROM tmp_customer_base a<br>LEFT JOIN (<br>  SELECT * <br>  FROM d_customer_info <br>  WHERE partition = &#39;20261006&#39;<br>) b<br>  ON a.tax_code = b.tax_code;</pre><p>Can you spot the difference between these two queries? If you can, congratulations!</p><p>If not, here’s a question for you: <strong>What happens if </strong><strong>b.partition = &#39;20261006&#39; contains no records?</strong></p><p>When I first wrote Query #1, I expected to get some results. But I was wrong. The query returned zero rows.</p><p>Then I discovered that partition 20261006 in d_customer_info was empty.</p><p>At that moment, I asked myself:</p><blockquote>“Why? Where did I go wrong? It’s just a <strong>LEFT JOIN</strong>! Shouldn’t all records from tmp_customer_base remain untouched?&quot;</blockquote><p>I rarely put partition filters outside subqueries, so I decided to try my usual approach: Query #2.</p><p>And guess what? It worked as expected!</p><h3>Wait, aren’t these two queries almost identical?</h3><p>At first glance, they look almost the same. I thought they were interchangeable, too.</p><p>After all, Hive has a feature called <strong>partition pruning</strong>. Shouldn’t Hive identify the required partition and filter out the other partitions before joining the tables?</p><p>Yes, Hive can prune unnecessary partitions. But here’s the catch: <strong>partition pruning doesn’t change the logical meaning of your SQL query.</strong></p><p>Let’s take a closer look.</p><h3>Let’s break it down with an example</h3><p>Suppose tmp_customer_base contains these records:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*BGnYbwmAypZw27sdTQm69A.png" /><figcaption><em>tmp_customer_base</em></figcaption></figure><p>And here’s d_customer_info:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*29PIVa8lSovFQYkf-Ssbpw.png" /><figcaption><em>d_customer_info</em></figcaption></figure><p>If <em>b.partition = ‘20261001’</em></p><p>We run Query #2 with that partition:</p><pre>SELECT DISTINCT *<br>FROM tmp_customer_base a<br>LEFT JOIN (<br>  SELECT * <br>  FROM d_customer_info <br>  WHERE partition = &#39;20261001&#39;<br>) b<br>  ON a.tax_code = b.tax_code;</pre><p>Result of query №2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*QtqOFSfGI0LTURzUzbmq7g.png" /><figcaption>Result of query №2</figcaption></figure><p>Explain query №2:</p><ul><li><strong>Step 1: Partition pruning</strong></li></ul><p>Hive identifies the required partition, 20261001, and can skip the other partitions in d_customer_info.</p><ul><li><strong>Step 2: LEFT JOIN</strong></li></ul><p>A LEFT JOIN would normally preserve all records from the left table. The unmatched records would have NULL values for the columns from b.</p><p>And we run Query #1 with that partition:</p><pre>SELECT DISTINCT *<br>FROM tmp_customer_base a<br>LEFT JOIN d_customer_info b<br>  ON a.tax_code = b.tax_code<br>WHERE b.partition = &#39;20261001&#39;;</pre><p>Here’s the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*5eHJSfIJbBi8Jw5vIN2wug.png" /><figcaption>Result of query №1</figcaption></figure><p>Explain query №1 (The WHERE clause changes everything):</p><ul><li><strong>Step 1 + 2:</strong></li></ul><p>Same as Query #2.</p><ul><li><strong>Step 3: Apply the WHERE condition</strong></li></ul><p>For every unmatched record, b.partition is NULL. Therefore, the WHERE clause removes those records.</p><p>The WHERE clause in the first query acts as a <strong>“global” filter</strong>, while the filter in the second query is <strong>“local”</strong> to the subquery. That’s why tmp_customer_base stays intact after the LEFT JOIN in the second query.</p><p>That’s how a <strong>LEFT JOIN can unintentionally behave like an INNER JOIN</strong>!</p><p>Now you know what happened with my two queries at the beginning of this article!</p><h3>How can we avoid this mistake?</h3><p>There are two ways to write Query #2 while preserving the intended LEFT JOIN behavior.</p><p><strong>Option 1: Filter the right-hand table in a subquery</strong></p><pre>SELECT DISTINCT *<br>FROM tmp_customer_base a<br>LEFT JOIN (<br>    SELECT *<br>    FROM d_customer_info<br>    WHERE partition = &#39;20261006&#39;<br>) b<br>    ON a.tax_code = b.tax_code;</pre><p><strong>Option 2: Put the partition condition in the ON clause</strong></p><pre>SELECT DISTINCT *<br>FROM tmp_customer_base a<br>LEFT JOIN d_customer_info b<br>    ON a.tax_code = b.tax_code<br>    AND b.partition = &#39;20261006&#39;;</pre><p>This version is concise and makes the join conditions easy to read.</p><p>Both approaches preserve all records from tmp_customer_base and allow Hive to use partition pruning when supported by the execution plan.</p><h3>One last thing to remember</h3><p>A LEFT JOIN doesn’t guarantee that every left-side record survives the entire query. It guarantees that unmatched left-side records are preserved by the join itself. A subsequent WHERE clause can still remove them.</p><p>That was the little logical trap I fell into.</p><p>Hopefully, sharing my mistake helps you avoid the same one!</p><p>Thanks for reading, and happy querying!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=802ccd9691ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classification Model Evaluation Series — Part 1: Confusion Matrix, Precision, Recall, FPR, FNR, and…]]></title>
            <link>https://medium.com/@luulienthao0204/classification-model-evaluation-series-part-1-confusion-matrix-precision-recall-fpr-fnr-and-a4cd2c1eec76?source=rss-c80780a2d74c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4cd2c1eec76</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Thao Luu]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 14:57:44 GMT</pubDate>
            <atom:updated>2026-07-14T14:57:44.215Z</atom:updated>
            <content:encoded><![CDATA[<h3>Classification Model Evaluation Series — Part 1: Confusion Matrix, Precision, Recall, FPR, FNR, and the Accuracy Trap</h3><blockquote>A Confusion Matrix is more than just a table counting correct and incorrect predictions. The way you define the <strong>Positive</strong> class fundamentally changes how metrics such as Precision, Recall, False Positive Rate (FPR), and False Negative Rate (FNR) should be interpreted.</blockquote><p>📖 <em>This article is part of a series on evaluating classification models.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aqH2_BCHIaDKM208ADDx9g.png" /><figcaption>Image created by Gemini</figcaption></figure><p>In this article, we’ll cover:</p><ul><li>What is a Confusion Matrix?</li><li>Why choosing the Positive Label matters</li><li>Metrics derived from the Confusion Matrix</li><li>Why Accuracy can be misleading on imbalanced datasets</li></ul><h3>1. What Is a Confusion Matrix?</h3><p>A <strong>Confusion Matrix</strong> is one of the most fundamental tools for evaluating a <strong>classification model</strong>. It compares the model’s predictions with the actual ground-truth labels.</p><p>For a binary classification problem, the Confusion Matrix has the following structure.</p><pre>                            Predicted<br>                      Positive   Negative<br><br>Actual  Positive         <strong>TP</strong>          <strong>FN</strong><br>        Negative        <strong> FP </strong>         <strong>TN</strong></pre><p>Where:</p><ul><li><strong>True Positive (TP):</strong> The instance is actually Positive and is correctly predicted as Positive.</li><li><strong>True Negative (TN):</strong> The instance is actually Negative and is correctly predicted as Negative.</li><li><strong>False Positive (FP):</strong> The instance is actually Negative but is incorrectly predicted as Positive.</li><li><strong>False Negative (FN):</strong> The instance is actually Positive but is incorrectly predicted as Negative.</li></ul><h4>Example: Email Spam Classification</h4><p>Suppose we build a model to classify emails into two categories:</p><ul><li><strong>Spam</strong> (Positive)</li><li><strong>Not Spam</strong> (Negative)</li></ul><p>After evaluating the model on <strong>100 emails</strong>, we obtain the following Confusion Matrix.</p><pre>| Actual / Predicted |   Spam | Not Spam |   Total |<br>| ------------------ | -----: | -------: | ------: |<br>| Spam               |     35 |        5 |      40 |<br>| Not Spam           |     10 |       50 |      60 |<br>| Total              |     45 |       55 |     100 |</pre><p>This tells us:</p><ul><li><strong>TP = 35:</strong> Thirty-five spam emails are correctly identified.</li><li><strong>FN = 5:</strong> Five spam emails are mistakenly classified as normal emails.</li><li><strong>FP = 10:</strong> Ten legitimate emails are incorrectly marked as spam.</li><li><strong>TN = 50:</strong> Fifty legitimate emails are correctly classified.</li></ul><h4>Multi-class Confusion Matrix</h4><p>Confusion Matrices are not limited to binary classification. They can also evaluate models with multiple classes.</p><p>Consider a fruit classification task with three categories:</p><ul><li>Apple</li><li>Banana</li><li>Orange</li></ul><pre>|              | Predicted Apple | Predicted Banana | PredictedOrange |<br>| ------------ | --------------- | ---------------- | --------------- |<br>|Actual Apple  |              25 |                3 |               2 |<br>|Actual Banana |               4 |               24 |               2 |<br>|Actual Orange |               1 |                2 |              27 |</pre><p>The <strong>main diagonal</strong> contains all correctly classified samples.</p><p>To evaluate a specific class — for example, <strong>Apple</strong> — we temporarily treat it as the Positive class and group every other class into a single Negative class.</p><p>Therefore,</p><ul><li>TP = 25</li><li>FN = 3 + 2 = 5</li><li>FP = 4 + 1 = 5</li><li>Can you guess TN = ?</li></ul><h3>2. Why Choosing the Positive Label Matters</h3><p>This is arguably the most overlooked concept when learning about Confusion Matrices.</p><p>Many simply assume that one random class is labeled as <strong>Positive = 1</strong> and move on.</p><p>However, the important question is <strong>not</strong> whether the label is represented by <strong>0</strong> or <strong>1</strong>.</p><p>The real question is:</p><blockquote>Which class do we care about the most?</blockquote><p>That class should become the <strong>Positive</strong> class.</p><p>Consider several common machine learning applications.</p><pre>|<strong> Problem  </strong>                 | <strong>Positive Class     </strong>     |<br>| ------------------------- | ----------------------- |<br>| Cancer Detection          | Patient with cancer     |<br>| Spam Detection            | Spam email              |<br>| Fraud Detection           | Fraudulent transaction  |<br>| Customer Churn Prediction | Customer who will leave |</pre><p>In every case, the Positive class represents the event we are trying to detect.</p><p>The Negative class simply represents everything else.</p><p>This explains why the Positive label is not just a technical detail — it reflects the <strong>business objective</strong> of the model.</p><h4>Why Does It Matter?</h4><p>Suppose we reverse the labels in a spam classifier.</p><p>Instead of defining</p><ul><li>Positive = Spam</li></ul><p>we define</p><ul><li>Positive = Not Spam.</li></ul><p>The model’s predictions have not changed at all.</p><p>The Confusion Matrix still contains exactly the same samples.</p><p>However,</p><ul><li>TP becomes TN.</li><li>TN becomes TP.</li><li>FP becomes FN.</li><li>FN becomes FP.</li></ul><p>As a result,</p><ul><li>Precision changes.</li><li>Recall changes.</li><li>FPR changes.</li><li>FNR changes.</li></ul><p>Surprisingly,</p><p><strong>Accuracy remains exactly the same.</strong></p><p>This is because Accuracy measures the overall proportion of correct predictions, regardless of which class is designated as Positive.</p><p>In contrast, Precision, Recall, FPR, and FNR are all defined relative to the Positive class.</p><blockquote><strong><em>Key takeaway:</em></strong><em> Before interpreting any classification metric, always ask one question first:</em></blockquote><blockquote><strong><em>“What is considered the Positive class?”</em></strong></blockquote><p>Without answering that question, metrics like Precision and Recall can easily be misunderstood.</p><h3>3. Metrics Derived from the Confusion Matrix</h3><p>Before introducing the evaluation metrics, it helps to remember four simple relationships.</p><ul><li>Actual Positives = TP + FN</li><li>Actual Negatives = TN + FP</li><li>Predicted Positives = TP + FP</li><li>Predicted Negatives = TN + FN</li></ul><p>Almost every metric in binary classification is derived from these four quantities.</p><h4>Accuracy</h4><p>Accuracy measures the proportion of correctly classified samples among all predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/0*87ByaCyOkz-DsH28" /></figure><p>It provides an overall view of model performance but <strong>does not distinguish between different types of errors</strong>.</p><p>For balanced datasets, Accuracy is often a reasonable starting point.</p><p>However, as we’ll see later, it can become highly misleading when one class is much rarer than the other.</p><h4>Precision</h4><p>Precision answers the following question:</p><blockquote><strong><em>Among everything the model predicted as Positive, how many were actually Positive?</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/0*Z-jreexhYgXXLcr7" /></figure><p>Precision becomes especially important when <strong>False Positives are costly</strong>.</p><p>Examples include:</p><ul><li>Spam detection (important emails should not be marked as spam)</li><li>Medical diagnosis (healthy patients should not be diagnosed with a disease)</li><li>Loan approval (qualified applicants should not be rejected because of an incorrect prediction)</li></ul><p>A high Precision means that when the model predicts Positive, it is usually correct.</p><h4>Recall (True Positive Rate)</h4><p>Recall answers a different question:</p><blockquote><strong><em>Among all actual Positive samples, how many did the model successfully identify?</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/0*hdnMJE5PRjwxSUJM" /></figure><p>Unlike Precision, Recall focuses on <strong>missing Positive cases</strong>.</p><p>It becomes the primary metric whenever <strong>False Negatives are expensive</strong>.</p><p>Typical examples include:</p><ul><li>Cancer screening</li><li>Fraud detection</li></ul><p>Missing a Positive case may lead to severe consequences.</p><h4>False Positive Rate (FPR)</h4><p>False Positive Rate measures how often Negative samples are incorrectly classified as Positive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/144/0*AfZa1xMTouqWuAED" /></figure><p>In other words,</p><blockquote><strong><em>Among all actual Negative samples, what percentage are falsely reported as Positive?</em></strong></blockquote><p>This metric is widely used when evaluating alarms and detection systems.</p><h4>False Negative Rate (FNR)</h4><p>False Negative Rate measures how often Positive samples are missed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/146/0*CfPXbm3y6jVAvZgA" /></figure><p>In other words,</p><blockquote><strong><em>Among all actual Positive samples, what percentage does the model fail to detect?</em></strong></blockquote><p>Notice something interesting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/0*NmB2f2aZCuKADqYw" /></figure><p>A higher Recall automatically means a lower FNR.</p><h4>The Trade-off Between FPR and FNR</h4><p>In many real-world applications, reducing one type of error inevitably increases another.</p><p>Consider a landmine detection system.</p><p>Missing a landmine could have catastrophic consequences.</p><p>Therefore, engineers are willing to tolerate more false alarms if that helps reduce missed detections.</p><p>In other words,</p><ul><li>Higher FPR</li><li>Lower FNR</li></ul><p>The same philosophy applies to:</p><ul><li>Fire alarm systems</li><li>Cancer diagnosis</li><li>Fraud detection</li></ul><p>This trade-off is largely controlled by the <strong>classification threshold</strong>, which determines how much confidence the model requires before predicting the Positive class.</p><p>The threshold deserves an entire article on its own, and we’ll cover it in the next part of this series.</p><h3>4. Why Accuracy Can Be Misleading?</h3><p>Accuracy is probably the easiest metric to understand.</p><p>Unfortunately, it is also one of the easiest metrics to misinterpret.</p><p>Suppose we are building a fraud detection system.</p><p>Our dataset contains:</p><ul><li>9,900 legitimate transactions</li><li>100 fraudulent transactions</li></ul><p>Only <strong>1%</strong> of the data belongs to the Positive class.</p><p>Now suppose our model produces the following results.</p><pre>| TP | FN |  FP |   TN |<br>| 40 | 60 | 150 | 9750 |</pre><p>The evaluation metrics become</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/0*3M40UB1mlJVSSoB4.png" /></figure><p>At first glance,</p><p><strong>97.9% Accuracy looks excellent.</strong></p><p>However, the model misses <strong>60% of all fraudulent transactions</strong>.</p><p>For most financial institutions, this would be unacceptable.</p><p>This example demonstrates why Accuracy alone can paint an overly optimistic picture when dealing with highly imbalanced datasets.</p><h4>Does Low Precision Always Mean a Bad Model?</h4><p>Not necessarily.</p><p>When the Positive class is extremely rare, Precision naturally becomes harder to improve.</p><p>Imagine a disease affecting only one person out of every 10,000.</p><p>Even a small number of False Positives can dramatically reduce Precision.</p><p>Therefore, <strong>a low Precision does not automatically imply that the model is poor.</strong></p><p>Instead, Precision should always be interpreted together with:</p><ul><li>the number of True Positives,</li><li>the number of False Positives,</li><li>the class distribution,</li><li>and the business objective.</li></ul><p>Metrics never tell the whole story by themselves.</p><p>Context matters.</p><h3>Key Takeaways</h3><p>Confusion Matrices are much more than tables of TP, FP, TN, and FN.</p><p>The most important step is deciding <strong>which class should be treated as the Positive class</strong>, because that decision determines how almost every evaluation metric should be interpreted.</p><p>Accuracy provides a useful overall summary, but it should rarely be used alone — especially when working with imbalanced datasets or high-risk applications such as medical diagnosis, fraud detection, spam filtering, and customer churn prediction.</p><p>Ultimately,</p><blockquote><strong><em>There is no universally “best” evaluation metric.</em></strong></blockquote><p>The right metric depends on <strong>what mistakes matter most in your specific problem</strong>.</p><p>Understanding that principle is far more valuable than memorizing formulas.</p><h3>Final Thoughts</h3><p>If there is one lesson to remember from this article, let it be this:</p><blockquote><strong><em>Before interpreting Precision, Recall, FPR, or FNR, always ask one simple question:</em></strong></blockquote><blockquote><strong><em>“What is the Positive class?”</em></strong></blockquote><p>That seemingly small decision determines the meaning of almost every metric derived from the Confusion Matrix.</p><p>Once you understand that, evaluating classification models becomes much more intuitive.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4cd2c1eec76" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why limit 10 is sometimes slow in Distributed SQL Engines?]]></title>
            <link>https://medium.com/@luulienthao0204/why-limit-10-is-sometimes-slow-in-distributed-sql-engines-b426b26f865d?source=rss-c80780a2d74c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b426b26f865d</guid>
            <category><![CDATA[hive]]></category>
            <category><![CDATA[query-optimization]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[hadoop]]></category>
            <dc:creator><![CDATA[Thao Luu]]></dc:creator>
            <pubDate>Sun, 31 May 2026 09:48:38 GMT</pubDate>
            <atom:updated>2026-05-31T09:48:38.648Z</atom:updated>
            <content:encoded><![CDATA[<p>When moving from traditional databases to distributed ones — in my case, the Hive database — I used to assume this query:</p><pre>SELECT * FROM huge_table LIMIT 10;</pre><p>would always be cheaper in both time and engine effort than:</p><pre>SELECT * FROM huge_table WHERE partition = 20250501 LIMIT 10;</pre><p>After all, the first query only asks for 10 rows!</p><p>But in distributed data systems, that intuition is sometimes completely wrong!!!</p><p><em>(Thanks to a coworker who suggested trying the second query.)</em></p><p>This triggers my curiosity immediately.</p><p>In short: when using WHERE partition = 20250501, Hive can first check the metadata, identify where that partition is stored, then read only that partition and return the first 10 rows.</p><p>Without the partition filter, a query with only LIMIT 10 may still need to scan metadata for <strong>all partitions</strong> before returning the result.</p><p>After doing some research, this was my key takeaway:</p><blockquote><em>“In distributed SQL LIMIT is usually applied after scan planing”</em></blockquote><p>Remember that <strong>execution cost is different from result size</strong>, which mean with the same size 10 output rows, it may cost different in processing.</p><p>Large tables in distributed systems such as Hive on HDFS are often stored across many partitions (for example by date or hour). Partitions help avoid scanning data you do not need.</p><pre>SELECT * FROM huge_table WHERE partition = 20250501 LIMIT 10;</pre><p>Let’s break down into 2 steps:</p><p><strong>Step1: Metastore partition prunning:</strong></p><p>If your query includes a partition filter, Hive only retrieves metadata for matching partitions.</p><p>If your query has no partition filter, HMS may need to fetch metadata for all partitions before execution starts.</p><p><strong>Step 2: Query excution:</strong></p><p>Once the partitions and their file paths are identified, the execution engine reads the underlying data.</p><p>As soon as the engine accumulates 10 rows, it terminates early and skips reading the remaining data.</p><p>That means LIMIT can reduce data scanning—but it does <strong>not</strong> necessarily prevent Hive Metastore from loading partition metadata up front.</p><p>So when a table contains hundreds or thousands of partitions (for example partitioned by date and hour), querying without partition filters can introduce noticeable latency.</p><p>In my case, huge_table had <strong>314 partitions</strong>.</p><p>(Yes, the table name is made up — had to keep the real one private 😄.)</p><p>After testing both queries, the result was:In my case I have a table name huge_table with 314 partitions (The table name kinds of silly because I have to make it up instead using the real name but don’t worry the result).</p><p>Here are the result after testing these 2 queries:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*RFploc0L-cxRkM5C5hkuqg.png" /><figcaption>Results after running these 2 queries</figcaption></figure><p><strong>WHERE partition = ... LIMIT 10 performed slightly better than </strong><strong>LIMIT 10 without the partition filter.</strong></p><p>This isn’t a perfectly controlled benchmark — runtime can vary depending on cluster traffic and how busy the system is.</p><p>I also haven’t had the chance to test this on tables with thousands of partitions yet, but I’d expect the difference to become more obvious as partition count increases.</p><p>A small query can still be expensive in distributed systems — and sometimes adding a simple partition filter makes more difference than expected.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b426b26f865d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>