<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dr. Burooj Ghani on Medium]]></title>
        <description><![CDATA[Stories by Dr. Burooj Ghani on Medium]]></description>
        <link>https://medium.com/@buroojghani?source=rss-f9d80b235b72------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*bF9PvCtyGSamX9vN</url>
            <title>Stories by Dr. Burooj Ghani on Medium</title>
            <link>https://medium.com/@buroojghani?source=rss-f9d80b235b72------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 21:19:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@buroojghani/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why KL Divergence in Knowledge Distillation?]]></title>
            <link>https://medium.com/@buroojghani/why-kl-divergence-in-knowledge-distillation-1375d555a728?source=rss-f9d80b235b72------2</link>
            <guid isPermaLink="false">https://medium.com/p/1375d555a728</guid>
            <category><![CDATA[loss-function]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[probability-theory]]></category>
            <category><![CDATA[knowledge-distillation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Dr. Burooj Ghani]]></dc:creator>
            <pubDate>Fri, 30 Aug 2024 10:47:57 GMT</pubDate>
            <atom:updated>2024-09-02T08:29:35.018Z</atom:updated>
            <content:encoded><![CDATA[<p>When I started implementing Teacher-Student knowledge distillation (KD) for my multi-class classification task, I noticed something curious: many papers employed <em>Kullback–Leibler divergence </em>(KLD) loss instead of the usual <em>cross-entropy </em>(CE) loss. This puzzled me because, in most deep learning tasks, KLD isn’t typically the go-to loss function. I wondered, why KLD is suddenly useful in KD. Does it offer some advantages when working with <em>soft labels</em>? It took me a while to understand the possible reasoning behind this and so I decided to document my findings. If you do not want to read through it, the short answer is you can stick with your good old cross-entropy loss (<em>tldr</em> at the end)<em>.</em></p><p>Let us start with the basics:</p><p><strong>Cross-Entropy</strong> is used to compare the true label distribution <em>q </em>(aka ground truth distribution) with the predicted distribution <em>p</em>. It’s given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E3P8msWba_U-89BnvF1f6Q.png" /></figure><p><em>q(i)</em> is the probability of class <em>i</em> according to the true distribution (which is typically one-hot encoded), and <em>p(i)</em> is the probability of class <em>i</em> according to the predicted distribution.</p><p><strong>KL Divergence</strong> measures how one probability distribution diverges from another:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NbA9MBOCd2IN4wuEwhV-rA.png" /></figure><p>To place KLD in the context of CE; <em>H(q,p)</em> is the cross-entropy, and <em>H(q)</em> is the entropy of the true distribution <em>q </em>(the inherent uncertainity in the true distribution):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q9l71QE-kOl6RLStJd9IYA.png" /></figure><h4><strong>One-hot labels: Are Cross-Entropy and KLD equivalent?</strong></h4><p>In many classification tasks, the ground truth labels are one-hot encoded. If there are<em> n </em>classes and the correct class is <em>k</em>, then <em>q(i) </em>= <em>1</em> for <em>i </em>= <em>k</em> and <em>q(i) </em>= <em>0</em> for all <em>i </em>≠ <em>k</em>.</p><p>Substituting this into the cross-entropy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*haQe1Z8UWas0CUhpN2fUbw.png" /></figure><p>and for KL divergence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uLdye_i_ajsv3nZ_n0woZw.png" /></figure><p>Since <em>q(i)</em> is one-hot encoded, <em>H(q) </em>=<em> 0</em>, and thus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ATQ9N2KPffFL535uoSiL2g.png" /></figure><p>This shows that when using one-hot encoded labels, KL divergence and cross-entropy provide the exact same optimization objective — they’re equivalent in terms of the gradients they provide to the model.</p><h4><strong>Soft labels: Are Cross-Entropy and KLD still equivalent?</strong></h4><p>In Knowledge Distillation, the teacher model provides <em>soft labels,</em> which are probability distributions over classes rather than a single correct class. These soft labels <em>q(i)</em> are not just 0 or 1 but represent the probabilities assigned by the teacher model to each class.</p><p>For soft labels, the cross-entropy loss takes its full shape again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9JQzAc-IAlucEHtO5LG0lg.png" /></figure><p>and, so does KL divergence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1r9c8ZnMa2XcciQOr4ThRA.png" /></figure><p>Here’s where it gets interesting: as we can see, <em>H(q)</em> is a constant with respect to <em>p</em>, meaning the gradient with respect to the student model parameters for both <em>H(q,p)</em> and <em>DKL(q </em>∥ <em>p)</em> will be the same:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-zWASAkFoYrpDtvszmZg1A.png" /></figure><p>Since <em>q</em> does not change with respect to the model parameters, KD should not contribute anything additional as compared to cross-entropy, right? Thus, in terms of optimization, both cross-entropy and KL divergence should lead to the same gradient descent updates for the student model even when soft labels are used.</p><h4><strong>Okay, if both cross-entropy and KL divergence provide the same optimization path, where is the catch?</strong></h4><p>Thanks to <a href="https://www.reddit.com/r/MachineLearning/comments/ke1r2q/comment/gg0btfi/?utm_source=share&amp;utm_medium=web3x&amp;utm_name=web3xcss&amp;utm_term=1&amp;utm_content=share_button">this</a> comment on Reddit, it looks like KD is “nicer” in terms of providing more stability as a loss (the devil is deep in the details sometimes, you see!). Let me explain how:</p><p>KL divergence equals zero when the student’s distribution matches the teacher’s exactly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r02AK33q0SEmB1hFSNdr9A.png" /></figure><p>This gives a clear zero-loss condition when the student perfectly aligns with the teacher. Cross-entropy, however, reaches its minimum when <em>p(i)</em> equals <em>q(i) </em>for all <em>i</em>, but the minimum value of the cross-entropy is not necessarily zero. Instead, it equals the entropy <em>H(q)</em> of the teacher’s distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*clIxyhUrYqiupQqTb8i_7w.png" /></figure><p>This entropy <em>H(q)</em> is generally non-zero, meaning the cross-entropy loss will fluctuate slightly depending on the batch, even when the student perfectly matches the teacher. These fluctuations don’t affect the optimization outcome (since <em>H(q) </em>is a constant with respect to model parameters) but can introduce unnecessary noise in the loss values across batches, making KLD more stable and interpretable when visualising the loss across batches. Phew!</p><p>tldr:</p><p>Both cross-entropy and KLD can be used for KD with soft labels, as they result in the same optimization updates. “The KL-divergence is nicer as a loss since it will equal 0 when the student network matches the teacher on all labels. In contrast, if we use X-entropy, then the loss will fluctuate even when the student and teacher output the exact same thing, and it will fluctuate according to the batch” This is why KLD might be preferred theoritically in the context of Knowledge Distillation, even though from a strict optimization standpoint, both losses will guide the student model in the same direction.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1375d555a728" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the emotional landscape of defending our beliefs]]></title>
            <link>https://medium.com/@buroojghani/understanding-the-emotional-landscape-of-defending-our-beliefs-234265fc0118?source=rss-f9d80b235b72------2</link>
            <guid isPermaLink="false">https://medium.com/p/234265fc0118</guid>
            <dc:creator><![CDATA[Dr. Burooj Ghani]]></dc:creator>
            <pubDate>Sun, 12 May 2024 06:01:13 GMT</pubDate>
            <atom:updated>2024-05-12T06:01:13.219Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*u1c80J3aZwRsSPJvINBrmg.jpeg" /><figcaption>The Incredulity of Saint Thomas by Caravaggio, from Google Arts and Culture.</figcaption></figure><p>In our everyday interactions, we often encounter situations where something we hold dear is questioned or criticized. Whether it’s our religion, ideology, country, beliefs, values, or ideas, our immediate reaction can be quite telling of our cognitive and emotional makeup. When faced with such challenges, do we react with incredulity, disbelief, cynicism, or annoyance? Do we genuinely listen to the critique in an unbiased manner? The answer to these questions can shed light on the broader human tendency to defend our beliefs fervently, often disregarding seemingly obvious truths presented by others.</p><p>When someone points out a flaw in something we value, our initial reaction is usually not one of calm reflection. Instead, we often experience a surge of defensive emotions. This reaction is deeply rooted in our psychological makeup and evolutionary history. Our brains are wired to protect our sense of self and our place within our social groups. This protective mechanism can lead us to reject criticisms outright, dismissing them as misguided or unfounded. This defensive stance is not unique to any one group; it is a universal human trait. Our thought processes, in such situations, are often guided more by emotions than by facts and logic. This is a natural outcome of our brain’s default settings, shaped by thousands of years of evolution. Our ancestors relied on quick, emotion-driven decisions for survival, which has left us with a tendency to react emotionally rather than rationally.</p><p>When we encounter resistance from others, especially those from a different background or belief system, it’s helpful to remember that they, like us, are not being intentionally dishonest or stubborn. They genuinely believe in the justifications their minds are creating. This phenomenon is known as cognitive dissonance, where holding two conflicting beliefs creates discomfort, prompting the individual to seek consistency, often by rejecting or rationalizing the conflicting information. To move beyond this default setting, we must actively train our minds to think objectively. This requires a deliberate effort to slow down our thinking, to reflect, and to consider alternative perspectives. By doing so, we can begin to see things more clearly and assess situations based on their merits rather than our emotional attachments. For example, when a cherished belief is challenged, instead of immediately defending it, we could take a step back and ask ourselves a few critical questions: Why do I hold this belief? What evidence supports it? What evidence challenges it? Am I open to the possibility that I could be wrong? This process of critical self-reflection can help us break free from the grip of our emotions and approach the issue with greater objectivity.</p><p>This approach is not just beneficial for personal growth but also for fostering understanding and harmony in a diverse world. When we recognize that others are also guided by their emotions and deeply held beliefs, we can approach discussions with more empathy and patience. We can create a space for genuine dialogue, where different perspectives are respected and considered thoughtfully.</p><p>In conclusion, understanding the emotional landscape of defending our beliefs is crucial for personal and collective progress. By acknowledging our natural tendencies and actively working to think more objectively, we can bridge divides, foster greater understanding, and contribute to a more harmonious world. This deliberate and thoughtful approach to thinking can transform not only our personal interactions but also the broader societal discourse, leading to a more open and accepting world for all.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=234265fc0118" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Importing your private car from Germany to the Netherlands: A comprehensive guide]]></title>
            <link>https://medium.com/@buroojghani/importing-your-private-car-from-germany-to-the-netherlands-a-comprehensive-guide-ff30e99f0f2d?source=rss-f9d80b235b72------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff30e99f0f2d</guid>
            <dc:creator><![CDATA[Dr. Burooj Ghani]]></dc:creator>
            <pubDate>Fri, 05 May 2023 13:16:52 GMT</pubDate>
            <atom:updated>2024-09-02T08:26:26.351Z</atom:updated>
            <content:encoded><![CDATA[<p>Moving to a new country can be both exciting and daunting, especially when you have to bring along your private car. Importing a vehicle between countries often involves navigating complex regulations, completing piles of paperwork, and meeting numerous requirements. This process can be intimidating and time-consuming, particularly when you’re already dealing with the challenges of settling into your new home.</p><p>Having recently gone through the process of importing my private car from Germany to the Netherlands, I understand firsthand the hurdles one may face. That’s why I’ve put together this comprehensive guide to help you navigate the intricacies of importing your car and to make the process as smooth and stress-free as possible. In this article, we’ll break down the steps required and offer helpful tips to ensure you’re well-prepared for this journey.</p><p>Together, we’ll explore the ins and outs of importing a private car, from acquiring the necessary documents to registering the vehicle in the Netherlands. By the end of this guide, you’ll be well-equipped to tackle the process with confidence and ease, so you can focus on enjoying your new life in the Netherlands. Let’s get started!</p><p>Alright, so picture this: you have a car registered in Germany, and you’ve just driven it into the Netherlands, the land of the Dutch. You’re settling into your new home and using your car for everyday activities like grocery shopping and running errands. Then comes the day when you have an appointment to obtain your BSN number. This is where things start to get a bit complicated. You see, once you receive your BSN, you become an official resident of the Netherlands and are required to pay road taxes for your car.</p><p>This marks the beginning of the process to import your car into the Netherlands. To start, you will need to fill in two forms and send them to the specified addresses provided on the first page of each respective form:</p><ol><li>The <a href="https://download.belastingdienst.nl/belastingdienst/docs/aanvraag_vergunning_vrijstelling_bpm_verhuizing_bpm0502z11fol.pdf">first form</a> you need to fill out is an application for exemption from BPM. By completing this form, you’re essentially requesting to be exempted from paying import taxes for your car.</li><li>The <a href="https://download.belastingdienst.nl/belastingdienst/docs/aang_buitenlands_kenteken_mb0072z13fol.pdf">second form</a> is a declaration of road taxes for your German license plates. Filling out this form allows you to continue driving legally in the Netherlands using your German license plates until you obtain your Dutch license plates.</li></ol><p>In case you’re curious about whether you can drive during this interim period, the answer is yes. However, it’s important to keep copies of these two forms in your car at all times while you drive around and await the registration of your German license plates in the Netherlands. In case you’re stopped by the police, you can show them these forms as proof that you’ve submitted the necessary paperwork and are awaiting responses for both BPM exemption and the registration of your German plates.</p><p>After mailing the forms, you can expect to hear back from the Belastingdienst eventually. In my experience, it took about a month to receive a response concerning the BPM exemption, and fortunately, I was granted the exemption. As for the road taxes associated with the German license plates, you will receive a bill requiring payment for a three-month period.</p><p>Upon receiving the BPM exemption (or not), your next step is to <a href="https://www.rdw.nl/en/import-export-transit/importing-a-car-from-a-eu-or-efta-country">schedule an appointment at an RDW center</a> of your choice to register your vehicle. Essentially, you’ll be applying for a Dutch vehicle registration certificate, and your car will be inspected on the day of your appointment. Make sure to bring along your Zulassungsbescheinigung Teil 1 (Fahrzeugschein), Teil 2 (Fahrzeugbrief), and a valid TÜV certificate to the appointment.</p><p>During the appointment, you’ll first hand over your German documents at the counter and pay the required fees (approximately 130 Euros). Afterward, you will be directed to the inspection area. The inspection process typically takes around 30 minutes, and once completed, you’ll be handed over a paper containing specific information about your car.</p><p>Before your RDW appointment, <a href="https://eendagskentekenbewijsaanvragen.rdw.nl/particulier">arrange</a> a one-day Dutch license plate, which is valid only for the day of the appointment and can be generated free of cost. Create the license plate using paper and a cardboard box, ensuring it’s waterproof by sealing it with adhesive or packaging tape. You’ll need this temporary plate to drive back home after the appointment. Although I wasn’t asked to remove my German plates or put up the temporary one, doing so is the legally correct procedure (Note that you will probably also need a one-day car insurance for this temporary number).</p><p>At this stage, you’ll need to complete the <a href="https://download.belastingdienst.nl/belastingdienst/docs/aang-meld-opg-bpm-bpm0111z8fol.pdf">BPM declaration form</a> and send it to the Belastingdienst. Make sure to include the date of your RDW inspection and your BPM exemption number on the form in the case you have been granted one (if you’re wondering why you need to declare BPM if you have been exempted, rest assured that you’re not alone. I too was confused and spent a significant amount of time on phone calls with the Belastingdienst trying to clarify this, but ultimately, this step is unavoidable). You also need to send along with this form a copy of your BPM exemption letter.</p><p>Don’t be intimidated by the length of the BPM declaration form. If you open it in Adobe Reader and start filling it out, you’ll find that significant part of the form relevant to you will auto-fill based on the information you provide on the first 2–3 pages (To help you speed up the form-filling process, I’ll provide my answers to a few potentially confusing fields on the form that can serve as guidance: For field <em>8b</em> “Datum bruto bpm”, I selected the first one: “Op basis van het voordeligste tarief” and the date and other field will fill in automatically as you select the field. For field <em>8c, </em>I checked the last option: “Forfaitaire afschrijvingstabel”, and on doing this most of the rest of the form will fill out automatically. You will be filling in your BPM exemption number in the field <em>9a.</em>)</p><p>In my experience, this process took less than a week. It’s important to note that you cannot drive during this period, as your German license plate is no longer valid. Keep in mind that you have already submitted your German registration documents to the RDW. And, with your German license plate now void, you can proceed to complete the form required to stop paying road taxes for that license plate. Here’s <a href="https://download.belastingdienst.nl/belastingdienst/docs/motorrijtuigenbel-verrek-of-terruggaaf-mb0062z28fol.pdf">the form</a> you need to fill out and send to the Belastingdienst. Once the Belastingdienst processes your request to stop paying road taxes for the German license plate, they should refund the taxes you’ve already paid for the upcoming months.</p><p>After receiving your card with the Dutch license plate number, you can schedule an appointment at a local shop that prints license plates. Once you’ve attached the new plates to your car, simply purchase Dutch car insurance, and voilà! You’re all set to hit the road in your newly registered vehicle. Enjoy the Dutch experience!</p><p>PS: There’s one aspect I haven’t mentioned earlier, as my situation was slightly different. In Germany, cars undergo a technical inspection every two years (TÜV). In the Netherlands, this inspection is called APK. In my case, the TÜV was due the same month I received my Dutch license card, so the RDW representative advised me to complete the technical inspection alongside getting the license plate printed, which is what I did. If your next technical inspection isn’t due soon, I would recommend asking the person who inspects your car at the RDW about the procedure. It’s possible that they accept the German technical inspection.</p><p>PPS: While I’ve shared my personal experience of moving from Germany to the Netherlands, it’s likely that individuals from other European countries can follow a similar process when importing their cars. This guide may still be helpful, but be sure to check for any country-specific requirements or variations in the process. Additionally, the information shared in this article is based on my personal experiences, and while I hope you find it insightful, please note that I cannot be held responsible for any discrepancies in the information provided. It is always advisable to consult official sources or seek professional advice to ensure accuracy and compliance with the latest regulations.</p><p>PPPS: If this guide helped you navigate the twists and turns of importing your car, and you’re feeling grateful enough to treat me to a cup of coffee, I won’t say no! You can channel your inner barista and send some caffeine-fueled appreciation my way through <a href="https://www.paypal.me/sighingquark">my PayPal account</a>. It’ll keep me energized and ready to tackle more bureaucratic adventures! Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff30e99f0f2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>