<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ebraheem Abdelhafez on Medium]]></title>
        <description><![CDATA[Stories by Ebraheem Abdelhafez on Medium]]></description>
        <link>https://medium.com/@ebraheem-mouhamed?source=rss-1bc42b74432a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*WAjoVPf6LTG84lAViIl9UQ.jpeg</url>
            <title>Stories by Ebraheem Abdelhafez on Medium</title>
            <link>https://medium.com/@ebraheem-mouhamed?source=rss-1bc42b74432a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 08:25:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ebraheem-mouhamed/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Two Gradients You Should Be Tracking from Day One]]></title>
            <link>https://ebraheem-mouhamed.medium.com/the-two-gradients-you-should-be-tracking-from-day-one-f6fa44a682ce?source=rss-1bc42b74432a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6fa44a682ce</guid>
            <category><![CDATA[data-scientist]]></category>
            <category><![CDATA[ml-engineering]]></category>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Ebraheem Abdelhafez]]></dc:creator>
            <pubDate>Mon, 08 Jun 2026 16:06:00 GMT</pubDate>
            <atom:updated>2026-06-08T16:06:00.663Z</atom:updated>
            <content:encoded><![CDATA[<p>Every ML Engineer or Data Scientist knows that feeling quite well. Whenever you look at a problem proposal or a target to achieve, it hooks you immediately. Maybe it’s an opportunity to work on very complex architecture and make it work at scale to affect millions of users, it might be a core business challenge that genuinely matters, or it’s a well-known problem that can save millions of Euros/Dollars per year if addressed correctly. Thus, you get instantly excited about the main cause, and your brain immediately starts mapping out the pipeline.</p><p>But if experience teaches us anything, it’s that technical excitement is not a significant covariate in the career satisfaction equation.</p><p>Those feelings are very important only during the interview or the proposal phase. Unfortunately, there isn’t enough data to decide whether this project is a good step forward for you. The real challenge in my eyes is the ability to “read the room” early on. You need to evaluate the true potential of where you are landing, without dragging your feet and wasting months, but also without rushing into premature conclusions.</p><p>There are exactly two things I’ve learned to track instead.<br><strong>Gradient 1:</strong> How the team is deciding on their milestones, how they plan for them, and more importantly, how they react when they fail to deliver. This signals the team’s engineering maturity.<br><strong>Gradient 2:</strong> What’s actually in it for you, professionally? Not compensation (assume you negotiated that already).</p><h3>The Failure Dynamics &amp; The 60-Day Convergence Rule</h3><h4>Gradient 1: The Maturity Baseline</h4><p>I can’t emphasize how important it is to understand the project’s power structure. Who is defining priorities, and based on what? Is that transparent enough for all the team members? How do the priority conflicts get resolved? These questions matter before you even get to milestone dynamics, because the answers tell you whether planning is a real process or a ritual.<br>Delivery failures are not edge cases in high-pressure environments. Scope shifts, estimates were wrong from the start, and dependencies moved. What actually separates a healthy team from a dysfunctional one has nothing to do with whether they miss a milestone. It’s what the room looks like the day after. I’ve been in both rooms. The difference is not subtle.<br>A team that handles this well gives you something harder to quantify but implicitly noticeable: you stop second-guessing whether your work is landing. That stability compounds fast, especially when you’re paving your way into the team’s structure.</p><h4>Gradient 2: Defining Your Personal ROI</h4><p>For <strong>Gradient 2</strong>, you need to be calculating your career optimization function long before day one. The expectation has two sources. One is internal: what do you want to have shipped, learned, or led in two years that you couldn’t have done in your previous role? The other is external: what did the team tell you the role would be? What’s your impression of the scope, ownership, and technical influence? This is like your compass during your time with the team, whether you are moving towards your destination or not.</p><p>Both gradients go hand-in-hand; they might differ in magnitude, but the sign must always be the same.</p><h4>The 60-Day Boundary Condition</h4><p>In my eyes, never run into premature conclusions, especially in the first two weeks. You are just getting to understand the team dynamics, which is more important than filing your first PR in that period. You also do not need a massive sample size or months of observation to see which way the wind is blowing. Sixty days are more than enough to have a better understanding of your current environment. In which you at least attended quarter planning, milestones’ hiccups, retros, and many reviews. These are very valuable inputs in the first months of every new team. <br>In this period, try to maximize your learning, touch on different parts of the project, and ask as many questions as possible, as long as they help you better understand. I also highly recommend reading the team’s documentation for their current processes or their bottlenecks. A team that writes things down and keeps them current is signaling something about itself.</p><p>If you have done your job completely, asked the right questions, and pushed for clarity, then you are well-prepared to conclude here. Never underestimate your own judgment if you’ve done your homework. Don’t let the fear of taking a bold decision paralyze you. In 90% of the cases, making a decisive move based on clear data is a winning decision for your career. The people I’ve seen regret their timing almost always waited too long, not too short. They observed clearly, formed a view, and then held on for another quarter, hoping something would shift. It rarely did. The window where leaving is clean and the cost is low, closes faster than it feels like it will.</p><h4>What If the Pace Is the Problem?</h4><p>That’s an alarming signal. Suppose the pace is so high that you can’t have enough time to understand the team culture and structure. Or it might feel like firefighting all the time under the slogan of just deadline pressure, then it will be back to normal again. That’s not a good signal for inclusion for new joiners. A team that treats new joiners as headcount before treating them as people who need to understand what they’re actually working on will keep doing so.<br>On the other side of the river, you might not feel work pressure, but still, your questions are also not being answered. It’s always next week, or you’ll get more context next quarter, or that’s something we’re still figuring out. And that window keeps moving. Chances are, they really do not have a clear answer for your questions, or they are just not transparent enough. Both are negative signals.</p><h3>The Decision is Yours</h3><p>If you have done all you can to quantify the progress on both gradients, talk with all team members, ask uncomfortable questions, and understand the team’s power structure. Then you just need to believe that your conclusion is not premature, it’s earned.</p><p>Act on it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6fa44a682ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cracking the Interview: Probability of the Median in a Uniform Distribution]]></title>
            <link>https://ebraheem-mouhamed.medium.com/cracking-the-interview-probability-of-the-median-in-a-uniform-distribution-585627f211db?source=rss-1bc42b74432a------2</link>
            <guid isPermaLink="false">https://medium.com/p/585627f211db</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-science-interview]]></category>
            <category><![CDATA[probability]]></category>
            <dc:creator><![CDATA[Ebraheem Abdelhafez]]></dc:creator>
            <pubDate>Mon, 28 Apr 2025 19:34:36 GMT</pubDate>
            <atom:updated>2025-04-28T19:34:36.052Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine you’re in a high-stakes data science or machine learning interview. The interviewer leans forward and asks: <em>“Given three random variables independently and identically distributed (i.i.d.) from a uniform distribution over [0, 4], what is the probability that their median is greater than 3?”</em></p><p>This is a classic probability question involving order statistics, and with the right approach, you can solve it with confidence and impress your interviewer. In this article, we’ll break down the problem step-by-step and sprinkle in some intuition. Let’s dive in!</p><h3>Step 1: Understand the Problem</h3><p>First, let’s unpack what the question is asking. You have three random numbers, each drawn randomly from a uniform distribution between 0 and 4. Think of it like picking three points on a number line from 0 to 4, where every point is equally likely. The <strong>median</strong> is the middle number when you sort these three numbers from smallest to largest. For example, if you draw 1.7, 2.5, and 3.8, sorting gives 1.7, 2.5, 3.8, and the median is 2.5. The goal is to find the probability that this median is greater than 3.</p><p>In mathematical terms, we have three i.i.d. random variables (<strong>X1</strong>, <strong>X2</strong>, <strong>X3 </strong>∼ Unif(0,4)). The median is the second order statistic, (<strong>X(2)</strong>), where (<strong>X(1) ≤ X(2) ≤ X(3)</strong>). We need to compute <strong>P(X(2) &gt; 3</strong>).</p><h3>Step 2: Define the Uniform Distribution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*QD8sdVJaVfadS076Ju67wA.png" /><figcaption>The probability density function of the Uniform(0, 4). The area under the curve is 1.</figcaption></figure><p>A uniform distribution over ([0, 4]) means every number between 0 and 4 is equally likely. The <strong>probability density function (pdf)</strong> is constant over this interval. Since the interval’s length is ( 4–0 = 4 ), the pdf is:</p><p>f(x) = 1/4, if 0 ≤ x ≤ 4</p><p>and zero elsewhere. The <strong>cumulative distribution function (CDF)</strong>, which gives <strong>P(Xi ≤ x)</strong>, is:</p><p>F(x) = x * (1/4), if 0 ≤ x ≤ 4</p><p>For example, <strong>F(3)</strong> = 3/4 = 0.75, meaning there’s a 75% chance a number is less than or equal to 3. Think of it as the area under the curve from the above plot, <strong>F(3)</strong> = 1/4 (height) * 3 (width).</p><h3>Step 3: What is the Second-Order Statistic?</h3><p>The median of three numbers is the <strong>second-order statistic</strong>, <strong>X(2)</strong>, the middle value when sorted. For example, if your numbers are 2.5, 1.7, 3.8, sorting gives 1.7, 2.5, 3.8, so <strong>X(2)</strong>= 2.5. To find (P(<strong>X(2)</strong> &gt; 3)), we need the distribution of <strong>X(2)</strong>.</p><p>In probability, the pdf of the (k)-th order statistic for (n) i.i.d. random variables is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*UKz8sBV8QWDf075Wox6cSw.png" /></figure><p>With the above equation, the question is 50% solved, but that’s boring, right? Let’s first build an intuition to understand the above formula with our question, which is the fun part. The first term is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/133/1*W5zY_v5GmTKQaWzI-Nq4gg.png" /></figure><p>This represents the number of ways we can arrange our random variables. In our example, where n=3, we have 6 possible arrangements, i.e., 3.2.1 = 6. For the first position, we can choose one of three options, for the second, we have two options left, and for the third place is only one option.</p><p>F(x)^(k-1), is the CDF such that k-1 variables are &lt; x. In our example, k = 2, so it could be only one random variable that can be less than k here. The third term is similar, however, it just focuses that n-k variables are &gt; x. In our example, n = 3 and k = 2, so it could only be one random variable that is strictly larger than the median. The f(x) term accounts for the density of one variable being in an infinitesimal interval [x,x+dx).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*jKyiKnpCFbka4qoAG5_-jg.png" /><figcaption>Simplifying the second-order probability equations with the example numbers.</figcaption></figure><h3>Step 4: Compute the Probability</h3><p>To find <strong>P(X(2) &gt; 3)</strong>, we integrate the pdf of <strong>X(2)</strong> from 3 to 4; this integral is the other 50% of the trick in the question:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*z5si1t-07Eg3RWBD6Z4W4Q.png" /></figure><p>Simplify the integrated: 4x — x².</p><p>So:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/1*HW9xEsQrkUpFOze_TnPrIQ.png" /></figure><p>Find the antiderivative:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/1*OSOSZg6UQ7EETDNtSq3ucg.png" /></figure><p>Evaluate from 3 to 4:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*JsWfzbc7laiWhYYe7HxGLw.png" /></figure><p>Thus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/1*e5DFrdWlebdgLo3da11izQ.png" /></figure><p>Thus, the question answer is 5 / 32.</p><h3>Conclusion</h3><p>This probability question tests your understanding of order statistics, uniform distributions, and combinatorial reasoning. These are key skills for data science and machine learning roles. By breaking it down into clear steps, you can showcase your technical competency. Practice this problem, and you’ll be ready to tackle similar questions with confidence. Good luck crushing that interview!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=585627f211db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When is the independence assumption of Naive Bayes less problematic?]]></title>
            <link>https://ebraheem-mouhamed.medium.com/when-is-the-independence-assumption-of-naive-bayes-less-problematic-b28750e767ef?source=rss-1bc42b74432a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b28750e767ef</guid>
            <category><![CDATA[naive-bayes]]></category>
            <category><![CDATA[data-science-interview]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[naive-bayes-classifier]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ebraheem Abdelhafez]]></dc:creator>
            <pubDate>Wed, 23 Apr 2025 11:27:14 GMT</pubDate>
            <atom:updated>2025-04-23T11:27:14.408Z</atom:updated>
            <content:encoded><![CDATA[<p>Naive Bayes is one of the simplest yet surprisingly powerful machine learning algorithms. Thanks to its speed and ease of implementation, it’s a go-to for tasks like spam email detection or sentiment analysis. But it comes with a catch: the “naive” assumption that all features are independent given the class label. In reality, features are often correlated, such as words in an email. So why does Naive Bayes still work so well in many cases, even when this assumption is violated? Let’s explore two key scenarios where the independence assumption is less problematic, making Naive Bayes a robust choice despite its simplicity.</p><h3>Understanding Naive Bayes and Its Assumptions</h3><p>Imagine you’re trying to classify an email as spam or not spam based on words like “free” and “win.” Naive Bayes calculates the probability of each class using Bayes’ theorem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*Y39EnFcCvDyLP4lPlhBcfQ.png" /></figure><p>Here, (<strong>P(class)</strong>) is the prior probability (how common spam is overall), (<strong>P(features | class)</strong>) is the likelihood (how likely the features are for that class), and (<strong>P(features)</strong>) is a normalizing constant. The “naive” part comes from assuming that features are independent given the class, so the likelihood is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*CaePcpFEovT6YZUCuj246A.png" /></figure><p>In our email example, this means assuming the presence of “free” doesn’t affect the presence of “win” in spam emails. But in reality, these words often appear together, so the assumption is flawed. Yet, Naive Bayes often performs well. Why? It’s because classification depends on the <strong>relative ranking</strong> of probabilities, not their exact values. Let’s dive into the two cases where this flawed assumption doesn’t break the classifier.</p><h3>Case 1: When Feature Correlations Are Similar Across Classes</h3><p>The first scenario where the independence assumption is less problematic is when the correlations between features are similar for all classes. Think of our spam email example again. Suppose “free” and “win” are correlated in both spam and non-spam emails. In other words, they appear together frequently in spam (e.g., 80% of spam emails with “free” also have “win”) and also in non-spam emails (e.g., 70% of the time). The true likelihood (<strong>P(free, win | class)</strong>) accounts for this correlation, but Naive Bayes approximates it as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*D6rzs2h2lUx-sOIT3EjNGA.png" /></figure><p>This approximation introduces an error because it ignores the correlation. However, if the correlation is similar in both classes, the mistakes in the likelihood estimates for spam and non-spam are proportional. When we compare the probabilities to decide the class, we look at the ratio:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*9ncq3PkGpkD8oZ1yn6OmzA.png" /></figure><p>Since the errors are similar across classes, they cancel out in this ratio, keeping the ranking of classes (spam vs. not spam) correct. The decision boundary remains accurate when the classifier is equally confident in both classes. This is why Naive Bayes can still shine in tasks like text classification, where word correlations (e.g., “free” and “win”) often follow similar patterns across classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6B6u0VxVE5Po_mLznt-2yg.png" /></figure><h3>Case 2: When Prior Probabilities Dominate</h3><p>The second scenario occurs when the prior probabilities (<strong>P(class)</strong>) are highly skewed, meaning one class is much more common. Imagine a medical diagnosis task in which 95% of patients are healthy (<strong>P(healthy)</strong> = 0.95) and only 5% have a disease (<strong>P(disease)</strong> = 0.05). Even if Naive Bayes misestimates the likelihood (<strong>P(features | class)</strong>) due to correlated symptoms (e.g., fever and cough), the prior can overpower these errors.</p><p>The decision boundary is where:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*ImRBq8JIJv19Rw9YV-w9CA.png" /></figure><p>If (<strong>P(healthy)</strong>) is much larger than (<strong>P(disease)</strong>), the likelihood of disease needs to be overwhelmingly strong to tip the balance. For example, if:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*bjey5VjZPrEwXLlNytfxrw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*y8Yyj3JMft4PUAueu072jA.png" /></figure><p>The actual products are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*871rVo8cKzxLxCFE8_0f4Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*QwbJY7k8gwzALe32eY1J_w.png" /></figure><p>Naive Bayes might misestimate the likelihoods (say, 0.08 for healthy and 0.3 for disease), but the prior (<strong>P(healthy)</strong> = 0.95) is so large that the product still favors “healthy” (0.95 · 0.08 = 0.076 &gt; 0.05 · 0.3 = 0.015). The decision boundary stays in the right place because the prior dominates, masking the likelihood errors.</p><h3>Why This Matters</h3><p>These two scenarios, similar feature correlations across classes and dominant priors, explain why Naive Bayes remains effective in practice, even when its independence assumption doesn’t hold. Research, like Domingos and Pazzani’s 1997 study, shows that Naive Bayes can be optimal for classification (minimizing errors) if the errors in probability estimates don’t change the ranking of classes. This robustness makes it a favorite for high-dimensional data, like text or medical records, where correlations exist but don’t always confuse the classifier.</p><p>So, next time you use Naive Bayes, remember: it’s not just naive. It’s a clever algorithm that thrives when feature correlations are balanced or when priors take the lead.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b28750e767ef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cracking the Interview Question: The Singularity Problem in Linear Regression]]></title>
            <link>https://ebraheem-mouhamed.medium.com/cracking-the-interview-question-the-singularity-problem-in-linear-regression-6d7c935afae2?source=rss-1bc42b74432a------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d7c935afae2</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[singularity]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-science-interview]]></category>
            <dc:creator><![CDATA[Ebraheem Abdelhafez]]></dc:creator>
            <pubDate>Mon, 21 Apr 2025 12:29:51 GMT</pubDate>
            <atom:updated>2025-04-21T12:29:51.901Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine you’re in an interview for a data scientist position, and the interviewer throws you this question:</p><blockquote>You have a dataset with 29 observations, each with 50 numeric features, to predict a person’s income. Can you use linear regression for this? What challenges might you face?</blockquote><p>You nod, take a deep breath, and start thinking. Linear regression seems straightforward, but something feels off ….</p><p>50 features and only 29 observations? That sounds like a recipe for trouble. Let’s dive into this problem, unpack the mathematical challenge known as the <em>singularity problem</em>, and explain it in a way that’s clear, concise, and engaging, even if you’re new to the math.</p><h3>What is Linear Regression?</h3><p>Linear regression is a go-to method for predicting a continuous outcome, like income, based on a set of features, such as age, years of experience, education level, etc. In our case, we have:</p><ul><li><strong>29 observations</strong>: Think of these as 29 people, each with a known income.</li><li><strong>50 features</strong>: Each person is described by 50 numeric values (e.g., age, salary history, number of LinkedIn connections, etc.).</li><li><strong>Goal</strong>: Predict income using these features.</li></ul><p>Mathematically, we represent the data as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*LxUMKB-WLZBFZAMn0UUiLw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*alAYUsr3wjh1qrEAvBqp2Q.png" /></figure><p>Linear regression assumes the income can be modeled as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/106/1*CJt2tmWGKT3Pc6ltPzDFfA.png" /></figure><p>where:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*KGMHbWFp1oEbYvrNfoEWUQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q5O_Br_Idb1nqkxy8BYnpw.png" /></figure><p>Our job is to find the best (<strong>β</strong>) that minimizes the prediction error. We do this using <strong>ordinary least squares (OLS)</strong>, which minimizes the squared differences between the actual incomes (<strong>y</strong>) and predicted incomes (<strong>Xβ</strong>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/196/1*XUzkHuPwH-ojePKkxJgnMQ.png" /></figure><h3>The Singularity Problem: What’s Going Wrong?</h3><p>To find (<strong>β^</strong>), we solve the <strong>normal equations</strong>. This involves a bit of calculus (don’t worry, I’ll keep it simple). We want to minimize the error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*NgqzbCXrtRKankdN6xzbXQ.png" /></figure><p>Taking the derivative with respect to (<strong>β</strong>), setting it to zero, and solving gives:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/124/1*pECyhMNasy7M62tjEcUAlw.png" /></figure><p>where (<strong>X^T</strong>) is the transpose of (<strong>X</strong>). If (<strong>X^T X </strong>∈ R 50 x 50) is invertible, we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/158/1*cizoLslQ7oMFXLsypbleFQ.png" /></figure><p>But here’s the catch: with 29 observations and 50 features, (<strong>X^T X</strong>) is <strong>not invertible</strong>. This is the singularity problem. Let’s break down why.</p><h3>Why is (<strong>X^T X</strong>) Not Invertible?</h3><p>A square matrix (like (<strong>X^T X</strong>), which is (50 x 50)) is invertible only if it has <strong>full rank</strong>, meaning its rank equals its dimension (50). The rank of a matrix counts its linearly independent rows or columns. Let’s see what happens with (<strong>X</strong>):</p><ul><li>(<strong>X</strong> is (29 x 50), so its rank is at most (min(29, 50) = 29). This means (<strong>X) </strong>can have at most 29 independent rows.</li><li>A key fact from linear algebra: the rank of (<strong>X</strong>^T <strong>X</strong> ) equals the rank of (<strong>X</strong>):</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*-swed29h6sk-ZyeVZ0-XXg.png" /></figure><ul><li>Since (<strong>X</strong>^T <strong>X</strong>) is ( 50 x 50 ), it needs a rank of 50 to be invertible. But with rank at most 29, it’s <strong>singular</strong> (non-invertible).</li></ul><p>Think of it like trying to solve 29 equations with 50 unknowns. You have too few constraints to pin down a unique solution, there are infinitely many (<strong>β</strong>) values that could fit the data perfectly.</p><h3>Why Does This Happen?</h3><p>The root cause is that we have <strong>more features than observations</strong> (50 &gt; 29). Imagine each observation as a point in a 50-dimensional space (one dimension per feature). With only 29 points, they can’t span the full 50-dimensional space, they’re confined to a 29-dimensional subspace (at most). Trying to fit a 50-dimensional model (with 50 coefficients) to these points is like trying to define a unique plane in 3D space with only two points: there are infinitely many planes that work.</p><p>This leads to:</p><ol><li><strong>No Unique Solution</strong>: Countless (<strong>β</strong>) vectors make (<strong>Xβ </strong>= <strong>y</strong>) on the training data.</li><li><strong>Overfitting</strong>: The model can perfectly fit the 29 points but may capture noise instead of the true relationship, leading to poor predictions on new data.</li><li><strong>Instability</strong>: Small changes in the data (e.g., a tiny measurement error) can drastically change the learned weights, making the model unreliable.</li></ol><h3>Key Takeaways</h3><ul><li>The singularity problem occurs when you have more features than observations (50 &gt; 29), making (<strong>X</strong>^T <strong>X</strong>) non-invertible.</li><li>This leads to no unique solution, overfitting, and instability in linear regression.</li><li>Solutions like feature selection fix the problem by reducing dimensionality.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d7c935afae2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Logistic Regression is a Linear Model Despite the Non-Linear Sigmoid]]></title>
            <link>https://ebraheem-mouhamed.medium.com/why-logistic-regression-is-a-linear-model-despite-the-non-linear-sigmoid-3956750979a2?source=rss-1bc42b74432a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3956750979a2</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-science-interview]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <dc:creator><![CDATA[Ebraheem Abdelhafez]]></dc:creator>
            <pubDate>Sun, 20 Apr 2025 12:08:08 GMT</pubDate>
            <atom:updated>2025-04-20T12:08:08.470Z</atom:updated>
            <content:encoded><![CDATA[<p>Logistic regression is one of the most widely used algorithms in machine learning, especially for binary classification tasks like predicting whether an email is spam. At first glance, it might seem confusing to call logistic regression a <em>linear</em> model. As it uses the sigmoid function, which is non-linear, to output probabilities between 0 and 1. So, why do we classify logistic regression as a linear model? In this post, we’ll dive into the mathematical reasoning behind this decision, with intuitive examples and visuals to make the concepts clearer.</p><p><em>P.S. This is a common question to be anticipated for ML/Data Scientist interviews.</em></p><h3>The Core of Logistic Regression: Predicting Probabilities</h3><p>Logistic regression is designed to predict the probability that a given input belongs to a particular class. For example, suppose we’re predicting whether a customer will buy a product (1 for buy, 0 for not buy) based on features like their age and income. Let’s denote the feature vector as ( X= [x1​, x2​​]), where (x1) is age and (x2) is income, and the model parameters as (β=[β0, β1, β2]), where (β0) is the intercept.</p><p>The model outputs the probability (P(y=1|X)), the likelihood of buying, using the <strong>sigmoid function:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/1*pd74SgERSJjhl7T7ZAjk5g.png" /></figure><p>where:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/177/1*O5CGvYuwyZkz-OuRBOz0SA.png" /></figure><p>Here, (z) is a linear combination of the features, and the sigmoid function maps (z) (which can range from (-\infty) to (+\infty)) to a probability between 0 and 1. The sigmoid function is non-linear, as its S-shaped curve (shown below) doesn’t follow a straight line.</p><figure><img alt="The sigmoid function maps the linear combination ( z ) to probabilities between 0 and 1. Notice its non-linear, S-shaped curve." src="https://cdn-images-1.medium.com/max/1024/1*fp3Tvohgipk3y3EVMHYX8A.png" /><figcaption><em>The sigmoid function maps the linear combination (z) to probabilities between 0 and 1. Notice its non-linear, S-shaped curve.</em></figcaption></figure><p>This non-linear transformation might make you think logistic regression is a non-linear model. However, the linearity lies not in the probability output but in how the model constructs its decision boundary and models the log-odds. Let’s unpack this step by step.</p><h3>The Linear Heart: Log-Odds</h3><p>To understand why logistic regression is linear, we need to look at the <strong>log-odds</strong>, which is the logarithm of the odds ratio. The odds ratio is frankly clarifying how likely the numerator is to happen relative to not happening, e.g., in our example, how likely the customer buys the product given their features relative to not buying it.</p><p>The odds of an event are defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*6ctpgLnH5YGeThX6HNVCiQ.png" /></figure><p>The log-odds is then:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/194/1*s6t04uSV_jX7pLVhKu4QeQ.png" /></figure><p>Logistic regression assumes that the log-odds is a linear combination of the features:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*RtLdcK0v2tw1l1CYAwlsew.png" /></figure><p>This equation is the key to understanding the linearity. The left-hand side, the log-odds, is modeled as a linear function of the features (X). To see how this relates to the sigmoid, let’s derive the probability. Starting with the log-odds:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*EN2N9M6sP1kKImll3rctuQ.png" /></figure><p>Exponentiating both sides:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/93/1*Ej7xdESTD-uRMAKeKija9A.png" /></figure><p>Solving for (P):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/1*0S6abwTmijryp9BgIC3HLA.png" /></figure><p>This confirms that the sigmoid function arises naturally from modeling the log-odds as a linear function. The linearity is in the log-odds, not the probability itself. The sigmoid is just a transformation that converts the linear predictor (z) into a probability, which is the desired outcome for the classification use cases.</p><h3>The Decision Boundary: A Linear Hyperplane</h3><p>The most compelling evidence for logistic regression’s linearity comes from its <strong>decision boundary</strong>. In binary classification, we typically classify an input as class 1 if (P(y=1|X) ≥ 0.5), and class 0 otherwise. Since the sigmoid function is monotonic, (P(y=1|X) = 0.5) occurs when:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/179/1*EaspYDkKH1SRauG44jyysA.png" /></figure><p>This happens when (z = 0). Thus, the decision boundary is defined by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*ZGUOlAGvpZZSft9Io7ZQ7A.png" /></figure><p>This is the equation of a hyperplane in the n-dimensional feature space. Let’s explore this with examples to make it concrete.</p><h3>Example: 2D Feature Space (Line)</h3><p>Suppose we have two features, age (x1) and income (x2), and our logistic regression model learns the parameters (β0 = -100), (β1 = 2), (β2= 0.05). The decision boundary is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/1*DUPvvpqWLZLjvJ21efTAKw.png" /></figure><p>Solving for (x2):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*j4C-QnwV3VQC7wjUFMhCCg.png" /></figure><p>This is the equation of a line in the 2D plane of age and income. Points above this line (where (z &gt; 0)) have ( P(y=1|X) &gt; 0.5 ), and points below (where (z &lt; 0)) have ( P(y=1|X) &lt; 0.5 ).</p><h3>General Case: n-Dimensional Space (Hyperplane)</h3><p>In an n-dimensional feature space, the decision boundary is a hyperplane of dimension (n-1). Why (n-1)? A single linear equation like ( β0 + X . β= 0 ) imposes one constraint on the (n) coordinates, reducing the dimensionality (degrees of freedom) by one. For instance:</p><ul><li>In 2D, the boundary is a 1D line.</li><li>In 3D, it’s a 2D plane.</li><li>In 4D, it’s a 3D hyperplane, and so on.</li></ul><p>This hyperplane is always linear, meaning it’s a flat subspace, which is why logistic regression is considered a linear model</p><h3>Why the Sigmoid Doesn’t Make It Non-Linear</h3><p>The sigmoid function’s non-linearity might seem like it complicates things, but it’s just a fixed transformation applied <em>after</em> the linear combination (z). The decision boundary depends only on where (z = 0), which is inherently linear. The sigmoid ensures that points far from the boundary (large positive or negative <strong>z</strong> get probabilities close to 1 or 0, respectively, but it doesn’t change the shape of the boundary.</p><p>To illustrate, consider a toy dataset with two points:</p><ul><li>Point A:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*N0ZxARjY6u4cj_UDmKdksw.png" /></figure><ul><li>Point B:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*pKUT23K9LuVHh9-2rULS5w.png" /></figure><p>The sigmoid maps these <strong>z</strong>-values to probabilities, but the decision boundary remains the line (-100 + 2 x1 + 0.05 x2 = 0), mentioned above.</p><h3>Comparison to Linear Regression</h3><p>Logistic regression is part of the <strong>generalized linear model (GLM)</strong> family, just like linear regression. In linear regression, we model the expected value of a continuous outcome:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*3Rk9eCukbtSeWA1r482yIw.png" /></figure><p>In logistic regression, we model the log-odds:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*amoBn7r8QmXLTn9FX1M6mw.png" /></figure><p>Both models use a linear combination of features, but logistic regression applies the logit link function (the inverse of the sigmoid) to handle binary outcomes. This similarity underscores why logistic regression is linear, i.e. linear decision boundary.</p><h3>Conclusion</h3><p>Logistic regression is a linear model because it models the log-odds as a linear combination of features, and its decision boundary is a linear hyperplane in the feature space. The non-linear sigmoid function is merely a transformation that maps the linear predictor to probabilities, without affecting the linearity of the decision boundary. By understanding the log-odds, the decision boundary, and the role of the sigmoid, we can see why logistic regression earns its place in the family of linear models, despite its classification task and non-linear output.</p><p>Whether you’re predicting customer purchases or diagnosing diseases, logistic regression’s linear nature makes it a powerful yet interpretable tool.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3956750979a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3DSSD: Point-based 3D Single Stage Object Detector⁵ (Explained)]]></title>
            <link>https://ebraheem-mouhamed.medium.com/3dssd-point-based-3d-single-stage-object-detector%E2%81%B5-explained-427bfe35ee9c?source=rss-1bc42b74432a------2</link>
            <guid isPermaLink="false">https://medium.com/p/427bfe35ee9c</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[fusion-sampling]]></category>
            <category><![CDATA[3d-object-detection]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Ebraheem Abdelhafez]]></dc:creator>
            <pubDate>Sat, 28 Nov 2020 18:57:17 GMT</pubDate>
            <atom:updated>2020-11-28T18:57:17.898Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog, we will explore the paper <a href="https://arxiv.org/abs/2002.10187">3DSSD: Point-based 3D Single Stage Object Detector⁵</a> published in CVPR 2020.</p><p>The paper’s main focus is to do 3D object detection by processing raw point clouds. Compared to other 3D detectors that process raw point clouds, the proposed architecture is light-weight, has a novel sampling strategy, and can process a higher number of frames per second while maintaining the precision to a great extent compared to other 3D detectors. In addition to that, the proposed architecture outperformed state-of-the-art architectures with 2x faster inference time as we will explain later in the results section.</p><p>We will go through the basic concepts that we need to be aware of before delving into the paper’s details. Then, I will explain the methodology adopted in the paper in my own words, hopefully, that makes it more understandable. Afterward, we will discuss the results and the suggested enhancements that can be applied to the presented model.</p><h3>Background</h3><p><strong>a) Point Cloud</strong></p><p>A point cloud is a scene representation that is different from images. Several sensors can generate point clouds, such as LiDAR. LiDAR generates point clouds where each point can have several dimensions (e.g. x, y, z, and reflectance). Point clouds, unlike images, are order-invariant and sparse, in the sense that a lot of black points can be present in the scene. Thus, to deal with point clouds using deep learning techniques, it won’t be a good idea to apply convolutional kernels. As the sparsity and the order-invariance will make the learning process using convolutional kernels totally inefficient.</p><figure><img alt="Point Cloud Sample" src="https://cdn-images-1.medium.com/max/269/1*2kR6GUDQbkLpFwQWUVuaAA.png" /><figcaption>Sample of Point Cloud (black points indicate how sparse the representation is)¹</figcaption></figure><p>Thus, several approaches can be followed to deal with point clouds and we will mention them in the next section.</p><p><strong>b) 3D Object Detection</strong></p><p>It is the task of generating 3D bounding boxes around the objects presented in the scene, besides, specifying a label for each generated box.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*T3DTrDWXHh4twIFDccilTg.png" /><figcaption>3D Detection Outputs in a scene¹</figcaption></figure><p><strong>c) 3D Detection Strategies</strong></p><p>There are two main approaches to do 3D detection. Either a compact representation approach or a point-based approach.</p><blockquote><strong><em>Compact Representation Approach</em></strong></blockquote><p>This approach does not work directly on point clouds. It converts point clouds to a compact representation that would be suitable to apply conventional deep learning techniques on, such as 2d-convolutional kernels. Also, this approach proved to be efficient and fast compared to the point-based approach, however, it suffers from information loss.</p><p>One possible conversion method is called “Voxelization”, which divides the point cloud into voxels (grid of voxels) so that each voxel is treated as a single input unit, while each voxel could contain several points. This approach, however, reduces the granularity presented in the point cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/1*VdKmL35vLXaqVuMSs6GduQ.png" /><figcaption>Voxelization Example¹</figcaption></figure><p>The other conversion method is called “Projection on 2D space”. Either projecting a point cloud to a normal 2D image or to a Bird-eye View (BEV). Both cases will lead to some loss of the information presented in the point cloud (e.g. depth information).</p><p><strong>Voxelization Example: </strong><a href="https://arxiv.org/abs/1812.05784"><strong>Pointpillars</strong></a><strong>² (~ 62 FPS)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GxEb3Qtr8sLpSHviwdNsdA.png" /><figcaption>Pointpillars’ architecture²</figcaption></figure><p>Pointpillars² is a well-known single-stage voxel-based 3D object detector, its architecture is presented in the image above.</p><p>It starts by encoding the point cloud into a Pseudo image so that the rest of the pipeline can apply 2d-convolutional kernels on it. Eventually, they apply a regression head to predict 3D bounding boxes around the objects in the point cloud. They used SSD as the prediction head, for those who are familiar with 2D detection literature know that SSD is a famous single-stage object detector, thus, the authors of Pointpillars² modified the traditional SSD to predict 3D bounding boxes instead of 2D.</p><blockquote><strong><em>Point-based Approach</em></strong></blockquote><p>This approach relies entirely on processing the raw point cloud points and encoding their features using <a href="https://arxiv.org/abs/1706.02413">point-net-like</a>³ architectures. This approach generally consists of two stages, which makes its predictions of very high accuracy compared to the compact representation approach. However, having two stages in the inference time can hamper the inference speed of such an approach and making it unfeasible for real-time applications.</p><p><strong>Point-based Example: </strong><a href="https://arxiv.org/abs/1907.10471"><strong>STD</strong></a><strong>⁴ (STD⁴ can process ~10 FPS)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n8rsNA97glAl_SOobgX8ZQ.png" /><figcaption>The Entire STD Network⁴</figcaption></figure><p>STD⁴ is a well-known 3D point-based detector that consists of several modules as depicted in the image. In order to understand the functionalities of the two stages that we mentioned above, we have to open the black-box which is called PGM (Proposal Generation Module).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3t9SkWZnjXzkApxnoQjYLA.png" /><figcaption>PGM in STD⁴</figcaption></figure><p><strong>First stage</strong>: It consists of Set Abstraction (SA) layers that encode the context features of the point cloud and down-sample the input to have a latent representation of the current input. Then, the Feature Propagation (FP) layers up-sample the latent-representation obtained from SA layers with the help of skip-connections from higher resolution inputs to restore the missing features during down-sampling and to broadcast the learned features, so it can produce an output which is of more channels than the original input. These extra channels encode the learned features by SA and FP layers. Afterward, a 3D Region Proposal Network (RPN) takes the output of FP layers and generates proposals that might contain objects to be predicted.</p><p><strong>Second Stage: </strong>It consists of a refinement module that refines the output of the RPN in the first stage so that the outcomes of the model are refined, hence, accurate.</p><p>After deploying such an architecture, it turns out that the FP layers and the refinement module consume half of the inference time, that’s why they represent a bottle-neck for point-based methods in general.</p><h3><strong>Paper’s Motivation</strong></h3><p><strong><em>Why could not we eliminate the FP layers and the refinement module all at once and still obtain good predictions?</em></strong></p><p>Let’s start answering this question in two points. <br>1. D-FPS → the sampling technique adopted in SA layers to down-sample the input is called Farthest Point Sampling (FPS). Meaning that if we choose the first point in our sample, we need the second one to be the farthest point to the previously chosen one, and so on. The farthest point should be distance-wise far, that’s why it is called “D-FPS”. Bearing in mind, the vast majority of the points in most of the scenes are back-ground points, thus, having the distance as the only sampling metric will lead to down-sample a lot of back-ground points. Also, if we have foreground instances to be detected which have a small scale (fewer interior points), they are unlikely to have points in the sample generated by D-FPS.<br>2. FP layers &amp; skip-connections → both components come to mitigate the problem aroused from D-FPS, since FP layers with the help of skip-connections can recover and broadcast the missed information during the down-sampling phase. As a consequence, the RPN and the refinement module can generate accurate bounding boxes without a significant loss of information.</p><h3><strong>Paper’s Contribution</strong></h3><p>In order to eliminate FP layers and the refinement module, and still obtain good accuracy and precision, new approaches should be devised. Thus, the authors of the paper decided to tackle the challenge and come up with a new approach that can be presented in 4 points:</p><ol><li>Sampling Strategy (Fusion Sampling (FS))</li><li>Candidate Generation (CG) Layer</li><li>Anchor-free Regression Head</li><li>3D Center-ness Assignment Strategy</li></ol><blockquote><strong>Fusion Sampling</strong></blockquote><p>As we discussed D-FPS, now we introduce F-FPS and F refers to Feature. Thus, F-FPS samples the points that are semantically dissimilar to each other. The exciting thing is that F-FPS contributes the distance into the sampling strategy. In other words, we obtain points that are not close to each other and also semantically dissimilar. <br>Let’s get an example to understand why it does make sense to add the distance metric into account along with the feature similarity?<br>Consider a car, which has points around its wheels are semantically different from the points around its windows, thus, if we just focused on semantic dissimilarity, we might end up having several points from the same car in our sample which represent redundancy in our data since all these points at the end refer to the same foreground instance, which is the car.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*Td3dpISAEWH1De2yB0SFOA.png" /><figcaption>F-FPS formulation⁵</figcaption></figure><p>Having that the back-ground points are so similar semantically, that would prevent F-FPS from sampling a lot of back-ground points. As a consequence, the receptive field of background points would be comparable in size with the receptive field of the foreground points. This behavior is good for the regressor as now the regressor trains mainly on the foreground points, but it also hampers the classification performance as the classifier can’t distinguish between foreground and background points.</p><p>Here, Fusion Sampling comes to the rescue. It proposes that if we will sample M points out of N points. Then, M/2 come from F-FPS (which represent mostly the foreground points) and M/2 come from D-FPS (which represent mostly the background points)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*XPuD8Oz3_L4t7cvPvXWJng.png" /><figcaption>Fusion Sampling in the Backbone of the architecture⁵</figcaption></figure><blockquote><strong>CG Layer</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*r3nlzSK5p5rwG0FfmYE6lw.png" /><figcaption>Candidate Generation Layer⁵</figcaption></figure><p>It starts by considering just the points sampled by F-FPS, since they mostly present foreground points. It passes these points to a regressor to predict some shifts for each point. Then, it adds these shifts to the corresponding points in F-FPS, so the output points are called candidate points. These candidate points represent the centers of the objects to be detected in the scene.<br>In the grouping layer, each candidate point would be surrounded by its neighboring points (from D-FPS &amp; F-FPS) according to a predefined threshold to constitute a group. All the points in each group concatenate their normalized locations along with their context features to form one big vector that is passed to an MLP network to encode its patterns, then passing it to the regressor.</p><blockquote><strong>Anchor-free Regression Head</strong></blockquote><p>Anchor-free regressor means that given a vector of N points where each point has its own encoding and passing it to the regressor, the regressor will generate N bounding boxes. Thus, this component generates a bounding box for each group created in the CG layer, as each candidate point represents a foreground instance’s center.</p><p>The regressor outputs are:<br>1. (dx, dy, dz) which are the offsets between candidate points’ positions and the GT centers’ positions.<br>2. (dl, dh, dw) which are the sizes of the foreground instances.<br>3. Orientation Angle of the bounding box.</p><p>The orientation angle is computed using orientation angle regression, which is a hybrid of classification and regression formulation. It can be simply understood as each object has 360 degrees around it, so they will be divided into N bins, where N = 12. Each bin will represent a class, thus, there would be a classifier to classify to which bin the bounding box belongs and there would be a regressor that predicts the residual value of the orientation angle for each bin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*aw7Y-JqPswh5Yks5i4ypgw.png" /><figcaption>Orientation Angle Regression Formulation⁵</figcaption></figure><blockquote><strong>3D Center-ness Assignment</strong></blockquote><p>This component is meant to reduce the number of false positives generated by the detector. It does so by reducing the classification scores of the bounding boxes that have centers far away from the GTs’ centers. In addition to that, it ideally maintains the classification scores of the bounding boxes with centers aligned with GTs’ centers. It is applied to the candidate points since the model treats them as the foreground instances’ centers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jNmFN-NC8--pdW5FljL0Uw.png" /><figcaption>where (f, b, l, r, t, d) represent the distance to the front, back,<br>left, right, top, and bottom surfaces of its corresponding instance, respectively⁵</figcaption></figure><p>final center-ness label = l_ctrness * l_mask, <br>where l_mask is a binary flag that represents either the candidate point belongs to a foreground instance or not, and l_ctrness measures how close the bounding box’s center from the GT’s center.</p><h3>Loss Function</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*4lLvHmfpHP3yKNcxc2N15w.png" /><figcaption>General Loss Function of the model⁵</figcaption></figure><p>N_c, N_p, and N*_p refer to the number of candidate points, positive candidate points, and positive points in F-FPS in the CG layer, respectively.<br>L_c is CE loss, where s_i and u_i refer to the classification score and the center-ness score respectively.<br>L_r is composed of all the regression losses (dist. to the center, size of the instance, angle regression loss, and corners of the bounding box).<br>L_s is the shift loss that can be computed in the CG layer.</p><h3>Results (3DSSD⁵ ~ 25+ FPS)</h3><p>The model has been tested on both KITTI Dataset and nuScenes Dataset.</p><blockquote><strong>KITTI Dataset</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hDTaP2ERBfgMG0zP-BM2Og.png" /></figure><p>It has been tested on the test set of the KITTI benchmark, and it outperforms all single-stage compact-representation methods in terms of Average Precision (AP) for each class in the dataset. It outperformed some 2-stage methods in terms of AP in some of the classes’ difficulties and for those that it did not outperform, it has a comparable performance with them as depicted in the table. <strong><em>Bearing in mind, 3DSSD’s inference time is at least 2x faster than the fastest 2-stage point-based method.</em></strong></p><blockquote><strong>nuScenes Dataset</strong></blockquote><p>The model has been tested on the nuScenes test set, which is a more complex dataset compared to KITTI.</p><p>Again, the model outperformed the compact-representation methods by far in both AP per class and the overall mean AP (mAP).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vrHaXBLbOtdXeC7_Y5DqTw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JH_TOIqAVZfEMcKh2XnWbg.png" /><figcaption>Green boxes are the predicted ones, and the red boxes are the GTs⁵</figcaption></figure><h3>Ablation Studies</h3><p>The ablation studies were conducted on the validation set of the KITTI dataset, according to the split mentioned in Voxelnet¹. The AP displayed indicates the Car class with a moderate difficulty level.</p><blockquote><strong>Effect of Fusion Sampling</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*7WZ4Pj1F7rnRbpgGfOOOoA.png" /></figure><p>As depicted in the table, FS improved the AP over both D-FPS and F-FPS, while maintaining a very good points recall score.</p><blockquote><strong>Effect of Shifting in the CG layer and the 3D center-ness assignment</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*DvuNOhLLTLW-z6V9tao8GQ.png" /></figure><p>Regardless of the assignment strategy, shifting always improves the AP. Also, combining shifting with 3D center-ness produced the best AP.</p><h3>Future Work</h3><p>As mentioned before, the model’s inference time outperforms by far all the 2-stage point-based methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*dT8nkd4cGw7yRGJgyRhQ4A.png" /></figure><p>However, Pointpillars² still can process up to 62 FPS, which is significantly faster than 3DSSD⁵. Thus, adopting implementation optimization (e.g. TensorRT) might help to make 3DSSD even faster, also TensorRT has been adopted in Pointpillars².</p><blockquote><strong>A suggestion for improvement from my perspective</strong></blockquote><ol><li>Training the entire network following the same scheme mentioned in the paper.</li><li>Freezing the weights of the last SA layer, CG layer, and the regression head.</li><li>Fine-tuning the earlier SA layers in the pipeline, by letting them pass their outputs directly to the CG layer and then to the regression head. Then, back-propagating the detection losses.</li></ol><p>This way the training time might be a bit longer, but the inference time won’t be affected at all. However, after fine-tuning, the SA layers would pay higher attention to the scene details and that would contribute to more accurate final results.</p><h3>Conclusion</h3><p>To sum up, I believe this paper proposed a very novel sampling technique that helped to erase an indispensable component like FP layers from a network while maintaining accuracy and precision to a great extent. This paper could be thought of as an effective step towards applying point-based methods in real-time applications with high precision. Given its simple and novel approach, I think that we will see 3DSSD+ and even 3DSSD++ in the next few years.</p><h3><strong>References</strong></h3><p>[1] Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. CoRR, 2017<br>[2] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. CVPR, 2019<br>[3] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NIPS, 2017<br>[4] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. STD: sparse-to-dense 3d object detector for point cloud. ICCV, 2019<br>[5] Zetong Yang, Yanan Sun, Shu Liu, &amp; Jiaya Jia. (2020). 3DSSD: Point-based 3D Single Stage Object Detector. CVPR, 2020</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=427bfe35ee9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>