<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kumar Abhishek on Medium]]></title>
        <description><![CDATA[Stories by Kumar Abhishek on Medium]]></description>
        <link>https://medium.com/@kumarabhishek?source=rss-699ca2ca59ac------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TmVamFV3S5liP5Qwgv_2Jw.jpeg</url>
            <title>Stories by Kumar Abhishek on Medium</title>
            <link>https://medium.com/@kumarabhishek?source=rss-699ca2ca59ac------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:54:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kumarabhishek/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[LinkedIn wants to connect me with me — LinkedIn recursion]]></title>
            <link>https://medium.com/@kumarabhishek/linkedin-wants-to-connect-me-with-me-linkedin-recursion-1865079ef9f9?source=rss-699ca2ca59ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/1865079ef9f9</guid>
            <category><![CDATA[funny]]></category>
            <category><![CDATA[linkedin]]></category>
            <category><![CDATA[fun]]></category>
            <dc:creator><![CDATA[Kumar Abhishek]]></dc:creator>
            <pubDate>Sun, 13 Nov 2022 07:58:24 GMT</pubDate>
            <atom:updated>2022-11-13T16:01:25.037Z</atom:updated>
            <content:encoded><![CDATA[<h3>LinkedIn wants to connect me with me — LinkedIn recursion</h3><p>For quite some time, I have been getting these email notifications from LinkedIn that ask me to connect to a person with my name.</p><p>I usually ignore these kinds of notifications. At first, I thought it was a person with the same name as mine. I thought, let’s find out who that lucky person is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jDgOW-tXQXPO-IaC9TeMDw.png" /></figure><p>So, I clicked on this email. Here is what I see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QlZ-DZnw3FPbPxNitZFWrQ.png" /></figure><p>I was quite surprised to see my own profile pic in the email.</p><p>I thought somebody must have created a clone of my LinkedIn profile, maybe they stole my photo?</p><p>Of course, why would LinkedIn have such a bug in its system? This would be such an embarrassing mistake!</p><blockquote>Plus, we have <em>2 mutual connections</em>, so I must know this person somehow. Isn’t it?</blockquote><p>Ok, so let’s connect with my clone now. I clicked on the “Yes, connect” button in the email.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eCB-PuL_ClhJkNWR4vRZzA.png" /></figure><p>Oh shoot, but I cannot connect to myself? That makes sense.</p><p>I clicked on this person’s profile link. But this takes me to my own LinkedIn profile:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wMSUK2KjKfiDBDOoOZLlOw.png" /></figure><p>Huh, did LinkedIn really keeps sending me notifications to connect with myself?</p><p>Indeed! I have been getting these notifications at least since 2021. Sometimes I ignore them; the rest of the time, I chuckle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GoJDSbJ8M7h2Yz-ewRJaFQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1865079ef9f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Accelerating NLP Model Inferencing with DistillBERT & ONNX]]></title>
            <link>https://medium.com/expedia-group-tech/accelerating-nlp-model-inferencing-with-distillbert-onnx-23edd7e187b5?source=rss-699ca2ca59ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/23edd7e187b5</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Kumar Abhishek]]></dc:creator>
            <pubDate>Tue, 26 Jul 2022 13:01:46 GMT</pubDate>
            <atom:updated>2022-07-26T13:01:46.916Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/expedia-group-tech/data/home">EXPEDIA GROUP TECHNOLOGY — DATA</a></h4><h4>Balancing speed and accuracy</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K_qoC4hdNXBZnz2GWm1I0w.jpeg" /><figcaption>Photo by Stephen Leonardi on Unsplash</figcaption></figure><p>Expedia Group™️ acts as a booking agent for all sorts of travel. An issue we’ve wrestled with is when a hotel (or vacation rental, activity, etc.) shares public contact information through our platform that allows for direct booking. This act violates our service agreements and we must detect if it happens. Furthermore, this mechanism needs to scale to review an ongoing stream of data generated by bookings, website updates, client updates, etc.</p><p>This is not a unique problem in the travel industry. It can be generalized to verify a whole class of financial and legal agreements on an ongoing basis. Speed is of the essence; quickly detecting violations protects the platform’s viability and stops possible fraud.</p><p>We require real-time decisions from unstructured data but would a large language model be up to the task? We started with BERT [1], the open-source transformer-based machine learning model for Natural Language Processing(NLP). After running sample data, we discovered it could give us inferences reliably — but was too slow.</p><p>In fact, there was no way to get results within the constraints of our service-level agreement (SLA). Simulating each call with listing descriptions of 500 words or more, the model took up to 2 seconds. The speed was too slow for upstream services and didn’t meet the <strong>250 milliseconds</strong> SLA we promised.</p><p>This was when we embarked on a journey to improve speed while maintaining inference reliability. We went straight to standard model compression techniques which are readily available and with which we were already familiar:</p><ol><li><strong>Knowledge Distillation </strong>(KD) Teacher-to-student transfer reduces model size.</li><li><strong>Graph compression</strong> using Open Neural Network Exchange(ONNX) allows for data transfer to many different environments. It also has training capabilities.</li><li>Using <strong>specialized hardware</strong> like GPU/Xeon processors through AWS exchange</li></ol><p>We used these techniques to improve speed without significantly impacting the model’s accuracy.</p><h3>A Bit About Knowledge Distillation (KD)</h3><p>In machine learning, Knowledge Distillation refers to transferring knowledge from a large model to a smaller one. The end goal is to develop accurate models that are more computationally efficient.</p><p>In a nutshell, we want to transfer knowledge from larger models (e.g., deep neural networks) to smaller models (e.g., shallow neural networks).</p><p>One solution is to use a technique called Knowledge Distillation. This involves “distilling” the knowledge from your large model into a new one that’s smaller, more manageable, and easier to work with.</p><figure><img alt="Both Teacher and Student consume training data, although Student has fewer layers of learning. Teacher prediction feeds into calibration which feeds into Student prediction, hence Distilled Knowledge moves from Teacher to Student. Finally true labels are applied to Student prediction." src="https://cdn-images-1.medium.com/max/936/1*BheoDjowazF3ljycOtCh2Q.png" /><figcaption><a href="https://www.researchgate.net/publication/355872310_Rethinking_the_Knowledge_Distillation_From_the_Perspective_of_Model_Calibration">Source</a>: Lehan Yang and Jincen Song</figcaption></figure><p>The general idea of performing KD is</p><ul><li>Use a pre-trained teacher model</li><li>Take some training data on which you want to train the Student model</li><li>Pass data through the Teacher model to generate soft labels (actual labels are “hard” labels)</li><li>Train Student model on data + soft labels</li></ul><p>There are several ways to distill the knowledge of the Teacher model into a simpler Student model. The main techniques are Multi-teacher, Offline, and Online.</p><p>Multi-teacher approaches benefit from more diverse data sources and, in some cases, less biased data. For offline KD, the Teacher is trained, and then the model is transferred directly to the Student. Conversely, online KD uses the ensemble result from multiple soft target Student models to generate a Student model.</p><h3>Technique #1: Use DistilBERT instead of BERT</h3><ul><li>DistilBERT [2] has the same general architecture as BERT, but only half the layers of BERT.</li><li>DistilBERT was trained on eight 16GB V100 GPUs for around 90 hours.</li><li>KD is done during the <strong>pre-training phase</strong>, reducing the size of BERT by 40% while retaining 97% accuracy and becoming 60% faster.</li></ul><h3>Technique #2: Convert the model to ONNX</h3><ul><li>Convert the DistilBERT model to ONNX format.</li><li>It is effectively a serialized format to represent the model and additionally functions as a model compression technique [3]</li></ul><h3>Technique #3: use a suitable AWS instance type</h3><ul><li>Use <a href="https://aws.amazon.com/ec2/instance-types/inf1/">inf1.6xlarge</a> instance type to bring down latency further (published improvement of 2.3 X)</li><li>24 vCPU, 48 GiB memory</li><li>4 AWS Inferentia chips — dedicated to inference with a large on-board memory cache and speeds of 128 TOPS</li><li>Much better than g4dn.2xlarge(GPU) &amp; c5–9xlarge(compute intensive) instance types</li></ul><h3>Our request/response</h3><h4>Request</h4><pre>{<br>  &quot;transactionContext&quot;: {<br>    &quot;transactionId&quot;: &quot;123456&quot;<br>  },<br>  &quot;attributes&quot;: {<br>    &quot;listingDescription&quot;: &quot;Near Xel Casino and the Cosmo Casino, the Tenebian Resort Las Vegas provides a casino, a nightclub, and a grocery/convenience store. For those looking to try their luck, this resort boasts a sportsbook, 137 casino gaming tables, and 7 casino VIP rooms. Indulge in a hot stone massage, a manicure/pedicure, and hydrotherapy at City Ranch Spa + Fitness, the onsite spa. Be sure to enjoy a meal at any of the 21 onsite restaurants, which feature Asian cuisine and happy hour. Yoga classes and aerobics classes are offered at the gym.&quot;<br>  }<br>}</pre><h4>Response</h4><pre>{<br>  &quot;modelCode&quot;: &quot;DistilBertNLPSupplyListingDescHA&quot;,<br>  &quot;modelVersion&quot;: &quot;1.0&quot;,<br>  &quot;refreshVersion&quot;: &quot;v20211220&quot;,<br>  &quot;probabilityScore&quot;: &quot;1.02e-10&quot;,<br>  &quot;modelSegment&quot;: &quot;SUPPLY&quot;,<br>  &quot;topContributor&quot;: &quot;listingDescription&quot;,<br>  &quot;scoreDetails&quot;: {<br>    &quot;label&quot;: &quot;NonFraud&quot;<br>  }<br>}</pre><h3>Numbers</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0abe675f883aeea17296765d79fb8a1/href">https://medium.com/media/d0abe675f883aeea17296765d79fb8a1/href</a></iframe><p>Notes:</p><ul><li>Averaged over 12 different runs using <a href="https://jmeter.apache.org/">Apache Jmeter</a>, free Java open-source software for load testing</li><li>Predictor service running on AWS ECS with 1 Fargate instance</li></ul><h4>More numbers</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1aad1644c7eecc1d6439ecf0a734462a/href">https://medium.com/media/1aad1644c7eecc1d6439ecf0a734462a/href</a></iframe><ul><li>Instance type used: inf1.6xlarge​</li><li>Each input request: 512 tokens​</li></ul><blockquote>The bottom line was that our work was successful. We met both our accuracy and our latency goals. We were able to improve latency by <strong>95%</strong>(2 sec to 110 milliseconds) and reduce the model size by<strong> 27% </strong>while impacting accuracy by only<strong> 1%</strong>.</blockquote><p>Following this general procedure should work for a lot of common NLP challenges. As for us, we plan to use the three-step technique outlined above for various use-cases we have today. Then it’s off to the next challenge.</p><h4><em>Bibliography</em></h4><p>[1] <a href="https://arxiv.org/pdf/1810.04805.pdf">Jacob Devlin, et al. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. May. arXiv:1810.04805</a></p><p>[2] <a href="https://arxiv.org/pdf/1910.01108.pdf">Victor Sanh, et al. 2020. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. <em>arXiv:1910.01108 [cs]</em>, February. arXiv: 1910.01108.</a></p><p>[3] <a href="https://medium.com/data-science-at-microsoft/model-compression-and-optimization-why-think-bigger-when-you-can-think-smaller-216ec096f68b#:~:text=The%20conversion%20process%20for%20natural,resulting%20in%20slightly%20smaller%20models.">Model compression and optimization: Why think bigger when you can think smaller?</a></p><p>[4] <a href="https://arxiv.org/pdf/2006.05525.pdf">Jianping Gou, et al. 2021. Knowledge Distillation: A Survey. <em>International Journal of Computer Vision</em>, 129(6):1789–1819, June. arXiv: 2006.05525.</a></p><p>[5] <a href="https://arxiv.org/pdf/2106.08962.pdf">Gaurav Menghani. 2021. Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better. <em>arXiv:2106.08962 [cs]</em>, June. arXiv: 2106.08962.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23edd7e187b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/accelerating-nlp-model-inferencing-with-distillbert-onnx-23edd7e187b5">Accelerating NLP Model Inferencing with DistillBERT &amp; ONNX</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PySpark and Scala Spark]]></title>
            <link>https://medium.com/@kumarabhishek/pyspark-and-scala-spark-7a255639a9b0?source=rss-699ca2ca59ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a255639a9b0</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Kumar Abhishek]]></dc:creator>
            <pubDate>Thu, 09 Jun 2022 21:46:43 GMT</pubDate>
            <atom:updated>2022-12-09T03:54:38.412Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Spark is written in Scala, and using PySpark, you can write Spark applications in Python.</p><blockquote>Have you ever wondered how PySpark works even though the underlying framework i.e. Apache Spark is written in Scala?</blockquote><p>The answer is that PySpark uses a library called <a href="https://www.py4j.org/index.html">Py4J</a>. Py4J enables Python programs to access Java objects. So, PySpark offloads all the Big Data processing work to the Scala Spark framework. That means that PySpark is just an interface to use Apache Spark in Python.</p><blockquote>Let’s ask a more deep and basic question: how does Python work? We keep hearing that Java programs run within a Java Virtual Machine(JVM), but what about Python?</blockquote><p>The simple answer is that Python programs run in their virtual machine(implemented in C; hence Python is called CPython).</p><p>Back to how PySpark uses Scala Spark: Py4J provides a GatewayServer instance that allows Python programs to communicate with JVM through a local network socket.</p><p>We need to start the gateway so it can accept incoming Python requests:</p><pre>GatewayServer gs = <strong>new</strong> GatewayServer(<strong>new </strong>StackEntryPoint());<br>gs.start();</pre><p>Now, we can call Java methods from python code as if they were python methods:</p><pre># Snippet from: <a href="https://stackoverflow.com/a/3793523/409814">https://stackoverflow.com/a/3793523/409814</a></pre><pre>from py4j.java_gateway import JavaGateway<br>gw = JavaGateway()                        # connect to the JVM<br>java_object = gw.jvm.mypackage.MyJavaClass()  # invoke constructor<br>other_object = java_object.doThat()<br>gateway.jvm.java.lang.System.out.println(&#39;Hello World!&#39;)</pre><p>One part of Py4J runs in the Python VM, so it is always “up to date” with the latest version of Python. The other part runs in the JVM we want to call.</p><p>P4J uses its protocol to communicate via sockets instead of JNI to optimize certain cases, manage memory, etc.:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jrZ2EHW2_fXVL3rI.png" /><figcaption>Author: Piotr Łusakowski, <a href="https://deepsense.ai/cooperative-data-exploration/">https://deepsense.ai/cooperative-data-exploration/</a></figcaption></figure><p>Exploring these enhanced my understanding of how things work under the hood. I hope this is helpful to you as well.</p><h4>References</h4><ol><li><a href="https://deepsense.ai/cooperative-data-exploration/">Cooperative data exploration</a></li><li><a href="https://medium.com/analytics-vidhya/how-does-pyspark-work-step-by-step-with-pictures-c011402ccd57">How does PySpark work? — step by step (with pictures)</a></li><li><a href="https://stackoverflow.com/a/3793523">https://stackoverflow.com/a/3793523</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a255639a9b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Paper Summary] Knowledge Distillation — A survey]]></title>
            <link>https://medium.com/@kumarabhishek/paper-summary-knowledge-distillation-a-survey-4a6663b4686c?source=rss-699ca2ca59ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a6663b4686c</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[knowledge-distillation]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Kumar Abhishek]]></dc:creator>
            <pubDate>Sun, 13 Feb 2022 08:02:14 GMT</pubDate>
            <atom:updated>2022-02-13T08:02:14.420Z</atom:updated>
            <content:encoded><![CDATA[<h3>[Paper Summary] Knowledge Distillation — A survey</h3><ul><li>This blog is mostly my learnings from the paper: <a href="https://arxiv.org/pdf/2006.05525.pdf"><em>Knowledge Distillation: A Survey</em></a><em> </em>and I try to present my summary which should hopefully be easier to follow.</li></ul><h3><strong>Problem</strong></h3><p>Deep learning models being huge with billions of parameters, are very difficult to deploy to devices with limited resources like phones &amp; embedded devices or to be used for real-time inferencing or serving where the typical latency requirements are in milliseconds(≤500 ms).</p><h3><strong>Solution</strong></h3><ul><li>One of the popular model compression and acceleration techniques is the knowledge distillation(KD) technique, where we transfer knowledge from a large model to a small model.</li><li>KD system has 3 key components: <em>knowledge</em>, <em>distillation algorithm</em>, <em>teacher-student architecture</em></li></ul><h4>Knowledge</h4><p>Different kinds of knowledge can be</p><ul><li><strong>Response-based Knowledge</strong>: this knowledge mimics the final prediction of the teacher model.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*1p7H2hSUfuCHAu0_4l3Q7g.png" /><figcaption>Figure adapted from Jianping Gou et al. (2020) | 📝 <a href="https://arxiv.org/abs/2006.05525">Paper</a></figcaption></figure><ul><li><strong>Feature-based knowledge</strong>: It’s well known how deep neural networks use various layers for various kinds of feature representation. On the same note, both the output of the last layer(response-based knowledge) and the output of intermediate layers are used as knowledge to supervise the training of the student model.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*FFHR3coJJz_9vfpZX9PKHw.png" /><figcaption>Figure adapted from Jianping Gou et al. (2020) | 📝 <a href="https://arxiv.org/abs/2006.05525">Paper</a></figcaption></figure><ul><li><strong>Relation-based knowledge (also called Shared representation knowledge)</strong>: this knowledge explores the relationship between intermediate layers, for example using inner product between features from 2 different layers as knowledge. More concretely, teacher’s knowledge about the relation between different input examples is transferred to the student model:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*5vDdv1UEVsOQSRKAxqXklw.png" /><figcaption>Figure adapted from Jianping Gou et al. (2020) | 📝 <a href="https://arxiv.org/abs/2006.05525">Paper</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*7K6cX6QstfYXh_YLFvmGBw.png" /><figcaption>Sources of response-based knowledge, feature-based knowledge &amp; relation-based knowledge in a deep teacher network | Figure adapted from Jianping Gou et al. (2020) | 📝 <a href="https://arxiv.org/abs/2006.05525">Paper</a></figcaption></figure><h4>Distillation schemes</h4><p>3 main categories depending upon whether teacher and student models are simultaneously updated or not:</p><ul><li><strong>Offline distillation</strong>: it’s the easiest to implement. A teacher model is first pre-trained on a training dataset and knowledge from the teacher model is transferred to train the student model.</li><li><strong>Online distillation</strong>: Teacher and student models are updated simultaneously in the same training process. We can parallelize the training here by using several distribution strategies(data and/or model), making this process efficient.</li><li><strong>Self-distillation</strong>: teacher and student models are of the same size and same architecture.These can either be a single model representing teacher as well as student or we may have two instances of the same model: one being teacher and another being student. The main idea here is that knowledge from deeper layers can be used to train the shallow layers. The process usually makes the student model more robust and accurate.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*maSLJNT6o72zf9v8jzl_7A.png" /><figcaption>Figure adapted from Jianping Gou et al. (2020) | 📝 <a href="https://arxiv.org/abs/2006.05525">Paper</a></figcaption></figure><h3>Teacher-student architecture:</h3><p>The most common student architectures are:</p><ul><li>a quantized version of the teacher model</li><li>a smaller version of the teacher model with few layers and fewer neurons per layer</li><li>same model as the teacher</li></ul><h3>Distillation algorithms</h3><ul><li><strong>Adversarial distillation: </strong>Uses the concept of Generative adversarial networks(GANs) where a generator model poses several difficult questions to the teacher model and the student model learns how to answer those questions from the teacher.</li><li><strong>Multi-teacher distillation</strong>: multiple teacher models are used to provide distinct kinds of knowledge to the student model</li><li><strong>Cross-modal(cross-disciplinary) distillation: </strong>The teacher model which is trained in one modality(say vision domain) is used to train a student model from a different modality(say text-domain). Example application: visual question-answering</li><li><strong>Graph-based distillation</strong><a href="https://arxiv.org/abs/1907.02226"><strong>²</strong></a><strong>:</strong> Knowledge of the embedding procedure of the teacher network is distilled into a graph, which is in turn used to train the student model</li><li><strong>Attention-based distillation</strong>: here attention maps are used to transfer knowledge about feature embeddings to the student model</li><li><strong>Data-free distillation</strong>: in absence of a training dataset(due to privacy, security, etc. issues), synthetic data is generated from the teacher model or GANs are used.</li><li><strong>Quantized distillation</strong>: transfer knowledge from a high-precision teacher(say 32-bit) to a low-precision student model(say 8-bit)</li><li><strong>Lifelong distillation: </strong>Continuously learned knowledge of the teacher model is transferred to the student model (<a href="https://www.youtube.com/watch?v=t3Ee5fA8mCo&amp;ab_channel=Yung-SungChuang">Youtube video reference</a>)</li><li><strong>Neural architecture search(NAS)-based distillation: </strong>Use AutoML to automatically identify the appropriate student model in terms of deciding the apt capacity gap between teacher and student models.</li></ul><h3>Performance aspects and conclusions</h3><ul><li>offline distillation does feature-based knowledge transfer</li><li>online distillation does response-based knowledge transfer</li><li>the performance of student models can be improved by knowledge transfer from teacher(high-capacity) models</li></ul><h3>Applications of KD</h3><ul><li>KD has found applications in visual recognition, NLP, speech recognition, various other applications</li></ul><h3>Challenges</h3><ul><li>It’s a challenge to measure the quality of knowledge or quality of student-teacher architecture</li></ul><h3>Future directions</h3><ul><li>It would be useful to integrate KD with other learning schemes like reinforcement learning, adversarial learning, etc.</li></ul><h3>References:</h3><ol><li><a href="https://arxiv.org/pdf/2006.05525.pdf"><em>Knowledge Distillation: A Survey</em></a></li><li><a href="https://arxiv.org/abs/1907.02226"><em>Graph-based Knowledge Distillation by Multi-head Attention Network</em></a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a6663b4686c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>