<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sanket Doshi on Medium]]></title>
        <description><![CDATA[Stories by Sanket Doshi on Medium]]></description>
        <link>https://medium.com/@sdoshi579?source=rss-4a8e3136747c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*W2dXHsCZqa8dVdQq_qtfYg.jpeg</url>
            <title>Stories by Sanket Doshi on Medium</title>
            <link>https://medium.com/@sdoshi579?source=rss-4a8e3136747c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:19:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sdoshi579/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[MongoDB Index Tuning: ESR Rule, Cardinality & $in Query Deep Dive]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@sdoshi579/mongodb-index-tuning-esr-rule-cardinality-in-query-deep-dive-bcf2b65185d7?source=rss-4a8e3136747c------2"><img src="https://cdn-images-1.medium.com/max/600/1*0mVGTipHgiTkZfc6okdHkQ.png" width="600"></a></p><p class="medium-feed-snippet">Are mongoDB guidelines are best for all queries or we need to do our research on indexes</p><p class="medium-feed-link"><a href="https://medium.com/@sdoshi579/mongodb-index-tuning-esr-rule-cardinality-in-query-deep-dive-bcf2b65185d7?source=rss-4a8e3136747c------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@sdoshi579/mongodb-index-tuning-esr-rule-cardinality-in-query-deep-dive-bcf2b65185d7?source=rss-4a8e3136747c------2</link>
            <guid isPermaLink="false">https://medium.com/p/bcf2b65185d7</guid>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[indexing]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Sanket Doshi]]></dc:creator>
            <pubDate>Wed, 10 Dec 2025 18:33:20 GMT</pubDate>
            <atom:updated>2025-12-10T18:33:20.603Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Hierarchical Deterministic(HD) Wallets]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/thecapital/hierarchical-deterministic-hd-wallets-c0d854a9d2ba?source=rss-4a8e3136747c------2"><img src="https://cdn-images-1.medium.com/max/600/1*Z-uWLpjUa2096MvJ1e9Gsw.png" width="600"></a></p><p class="medium-feed-snippet">In the last blog, we learned about different types of addresses.</p><p class="medium-feed-link"><a href="https://medium.com/thecapital/hierarchical-deterministic-hd-wallets-c0d854a9d2ba?source=rss-4a8e3136747c------2">Continue reading on The Capital »</a></p></div>]]></description>
            <link>https://medium.com/thecapital/hierarchical-deterministic-hd-wallets-c0d854a9d2ba?source=rss-4a8e3136747c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0d854a9d2ba</guid>
            <category><![CDATA[blockchain-development]]></category>
            <category><![CDATA[crypto-wallet]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[crypto]]></category>
            <dc:creator><![CDATA[Sanket Doshi]]></dc:creator>
            <pubDate>Thu, 01 May 2025 15:53:50 GMT</pubDate>
            <atom:updated>2025-05-01T15:53:50.067Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Types of Bitcoin Addresses and their Implementation]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/thecapital/types-of-bitcoin-addresses-and-their-implementation-3823d29d0f09?source=rss-4a8e3136747c------2"><img src="https://cdn-images-1.medium.com/max/1024/1*Kmv3OUOPMeRfhWhRR-Sr7Q.jpeg" width="1024"></a></p><p class="medium-feed-snippet">Different types of Bitcoin addresses are used for transactions.</p><p class="medium-feed-link"><a href="https://medium.com/thecapital/types-of-bitcoin-addresses-and-their-implementation-3823d29d0f09?source=rss-4a8e3136747c------2">Continue reading on The Capital »</a></p></div>]]></description>
            <link>https://medium.com/thecapital/types-of-bitcoin-addresses-and-their-implementation-3823d29d0f09?source=rss-4a8e3136747c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3823d29d0f09</guid>
            <category><![CDATA[crypto]]></category>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[transactions]]></category>
            <dc:creator><![CDATA[Sanket Doshi]]></dc:creator>
            <pubDate>Wed, 22 Jan 2025 05:02:39 GMT</pubDate>
            <atom:updated>2025-01-22T05:02:39.426Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Understanding Peer-to-Peer File Transfer: A Modern Revolution in Digital Sharing]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@sdoshi579/understanding-peer-to-peer-file-transfer-a-modern-revolution-in-digital-sharing-1f0741ae64d3?source=rss-4a8e3136747c------2"><img src="https://cdn-images-1.medium.com/max/600/1*Rdn6vp1uoYQEGV_CFzyzKQ.png" width="600"></a></p><p class="medium-feed-snippet">In today&#x2019;s digital landscape, peer-to-peer (P2P) technology is frequently linked to illegal downloads, such as movies, web shows, music&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@sdoshi579/understanding-peer-to-peer-file-transfer-a-modern-revolution-in-digital-sharing-1f0741ae64d3?source=rss-4a8e3136747c------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@sdoshi579/understanding-peer-to-peer-file-transfer-a-modern-revolution-in-digital-sharing-1f0741ae64d3?source=rss-4a8e3136747c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f0741ae64d3</guid>
            <category><![CDATA[bittorrent]]></category>
            <category><![CDATA[peer-to-peer]]></category>
            <category><![CDATA[file-transfer]]></category>
            <category><![CDATA[p2p]]></category>
            <dc:creator><![CDATA[Sanket Doshi]]></dc:creator>
            <pubDate>Sat, 02 Nov 2024 05:38:49 GMT</pubDate>
            <atom:updated>2024-11-02T05:38:49.271Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[The Unsung Hero of Communication: Understanding Session Initiation Protocol (SIP)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@sdoshi579/session-initiation-protocol-sip-1ecc94e64bf7?source=rss-4a8e3136747c------2"><img src="https://cdn-images-1.medium.com/max/600/1*GCpKjoxQtNWbhZOo40PA0Q.png" width="600"></a></p><p class="medium-feed-snippet">The world of multimedia communication relies heavily on a powerful protocol called Session Initiation Protocol (SIP).</p><p class="medium-feed-link"><a href="https://medium.com/@sdoshi579/session-initiation-protocol-sip-1ecc94e64bf7?source=rss-4a8e3136747c------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@sdoshi579/session-initiation-protocol-sip-1ecc94e64bf7?source=rss-4a8e3136747c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ecc94e64bf7</guid>
            <category><![CDATA[medium]]></category>
            <category><![CDATA[telecommunication]]></category>
            <category><![CDATA[communication]]></category>
            <category><![CDATA[voip]]></category>
            <category><![CDATA[sip]]></category>
            <dc:creator><![CDATA[Sanket Doshi]]></dc:creator>
            <pubDate>Thu, 26 Sep 2024 07:50:08 GMT</pubDate>
            <atom:updated>2024-10-14T17:25:32.787Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Cyclical Learning Rates]]></title>
            <link>https://medium.com/analytics-vidhya/cyclical-learning-rates-a922a60e8c04?source=rss-4a8e3136747c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a922a60e8c04</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[cyclic-learning-rate]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sanket Doshi]]></dc:creator>
            <pubDate>Mon, 05 Apr 2021 16:52:45 GMT</pubDate>
            <atom:updated>2021-04-05T16:52:45.491Z</atom:updated>
            <content:encoded><![CDATA[<p>Achieve higher accuracy for your machine learning model in lesser iterations.</p><p>Learning rate influences the training time and model efficiency. Learning rate depends on the loss function landscape, which depends on the model architecture and dataset. To converge the model faster, it requires an optimal learning rate.</p><h4>What is the Optimal Learning Rate?</h4><p>The learning rate which results in a steep decrease in loss is known as the optimal learning rate. The more steep the decrease in loss more optimal is the learning rate.</p><h4>How to find the Optimal Learning Rate?</h4><p>If the loss is reducing slowly, we can increase the learning rate. If the loss is fluctuating and increases, the learning rate is too high, so need to reduce it. Loss function graphs can check if the learning rate is good enough or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*IN4Km-G7l8SBUd2ny1DhJA.png" /><figcaption>Image Credit: <a href="https://cs231n.github.io/neural-networks-3/#annealing-the-learning-rate">https://cs231n.github.io/neural-networks-3/#annealing-the-learning-rate</a></figcaption></figure><p>From the graph, we can see that a very high learning rate can diverge the loss function while a high learning rate can get stuck at sub-optimal points. A low learning rate results in linear convergence but requires more iteration, while a good learning rate results in faster convergence. So finding a good learning rate can cause our model to converge faster.</p><h4>Different methods for setting the learning rate</h4><ol><li>Constant learning rate — In this, we set the learning rate to some value that remains static for the complete training process.</li><li>Adaptive learning rate — In this, the model uses various optimization algorithms that change the learning rate while training the model. The training starts with a large learning rate, which gradually reduces and gets too small on final model convergence. To learn more about this, you can read <a href="https://towardsdatascience.com/optimizers-for-training-neural-network-59450d71caf6">here</a>.</li><li>Cyclical learning rate — In this, the learning rate fluctuates between the range of learning rate. It increases from a minimum learning rate towards the maximum learning rate and then reduces to a minimum learning rate. It has proved to be more efficient and results in faster convergence for some models.</li></ol><h3>Cyclical Learning Rate</h3><p>Similar to adaptive learning rate, the learning rate changes are cyclic, always returning to the learning rate&#39;s initial value. A very high learning rate causes the model to fluctuate more or to diverge from the minima, while a lower learning rate can cause the model to converge very slowly or to converge to the local minima. Cyclical learning rate(CLR) allows keeping the learning rate high and low, causing the model not to diverge along with jumping from the local minima.</p><p>In CLR learning rate oscillates between base learning rate and max learning rate. The oscillation of learning rate can be based on various function-triangular (linear), Welch window (parabolic), or Hann window (sinusoidal). The triangular window is a simpler way of changing the learning rate.</p><h4>Why CLR works?</h4><p>As per <a href="https://arxiv.org/pdf/1406.2572.pdf">Dauphin 2014,</a> local minima is not the chief obstacle in optimizing large and deep neural networks instead, <a href="https://en.wikipedia.org/wiki/Saddle_point#:~:text=In%20mathematics%2C%20a%20saddle%20point,local%20extremum%20of%20the%20function.">saddle points</a> are the critical points in optimizing paths.</p><p>The gradients at saddle points are too small and can slow the learning process. However, increasing the learning rate allows the rapid traversal of saddle point plateaus.</p><h4>Variations of CLR</h4><p>As we need to change the learning rate value, the variations are based on the changing logic. It can be linear, exponential, or any other function.</p><h4>Triangular window</h4><p>In this, the learning rate changing logic is linear that is we will increase the learning rate with some constant from min learning rate to max learning rate and will decrease the learning rate with the same constant from max learning rate to minimum learning rate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*wBv9WvQUKCeu38I2Qad42g.png" /><figcaption>Triangular window | Image Credit: <a href="https://arxiv.org/pdf/1506.01186.pdf">https://arxiv.org/pdf/1506.01186.pdf</a></figcaption></figure><h4>Parabolic window</h4><p>The learning rate increases or decreases using a parabolic function.</p><h4>Sinusoidal window</h4><p>The learning rate increases or decreases using a sinusoidal function.</p><p>We’ll be looking into a triangular window.</p><h3>LR Update Logic</h3><p><strong>opt.LR</strong> — The lower base of the learning rate</p><p><strong>maxLR </strong>— Maximum learning rate boundary</p><p><strong>iteration</strong> — Network trained on one mini-batch</p><p><strong>epochCounter</strong> — Number of iterations completed</p><p><strong>cycle length </strong>— Number of iterations until the learning rate returns to the initial value.</p><p><strong>stepsize</strong> — Number of iterations required to move the learning rate from opt.Lr to maxLR(half the cycle length).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*V6FhYP95VC9bGxmh6kQv0A.png" /><figcaption>LR calculation logic | Image Credit: <a href="https://arxiv.org/pdf/1506.01186.pdf">https://arxiv.org/pdf/1506.01186.pdf</a></figcaption></figure><p>The local cycle defines which cycle is currently going on. The cycle will always be constant and till the epoch counter is multiple of cycle length the cycle does not change, hence math.floor is used.</p><h4>Estimate Good Value of Cycle Length</h4><p>As per the <a href="https://arxiv.org/pdf/1506.01186.pdf">paper</a>, stepsize should be equal to 2–10 times the number of iterations in an epoch. We can calculate iterations present in an epoch using dataset size and batch size. If the dataset comprises 50,000 data entries and the batch size is 100, then the number of iterations in an epoch will be 500 (50,000/100).</p><h4>Estimate the Minimum and Maximum Learning Rate Boundary</h4><p>The paper refers to the simple approach for determining the learning rate boundary values. The approach is known as the “LR Range Test”. In this approach, we run the model by increasing the learning rate over the number of epochs. Stepsize value is kept equal to the number of iterations in all epoch you want your model to run on so that the learning rate increases linearly. Plot the graph mapping accuracy of the model and value of learning rate. Note the learning rate value when accuracy increases and when the accuracy falls or becomes ragged.</p><h4>References</h4><ol><li>Paper on CLR — <a href="https://arxiv.org/pdf/1506.01186.pdf">https://arxiv.org/pdf/1506.01186.pdf</a></li><li><a href="https://github.com/bckenstler/CLR">https://github.com/bckenstler/CLR</a></li><li><a href="https://www.jeremyjordan.me/nn-learning-rate/">https://www.jeremyjordan.me/nn-learning-rate/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a922a60e8c04" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/cyclical-learning-rates-a922a60e8c04">Cyclical Learning Rates</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPT-3: Just Another Language Model But Bigger]]></title>
            <link>https://medium.com/data-science/gpt-3-just-another-language-model-but-bigger-1add6e9277fa?source=rss-4a8e3136747c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1add6e9277fa</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[gpt-3]]></category>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Sanket Doshi]]></dc:creator>
            <pubDate>Tue, 01 Sep 2020 15:48:40 GMT</pubDate>
            <atom:updated>2020-09-01T15:48:40.501Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding the GPT-3 research paper</h4><p>GPT-3 has takeover the NLP world in a very short period of time. It has proved the theory that increasing number of parameters will increase the accuracy of model.</p><h3>Sam Altman on Twitter: &quot;The GPT-3 hype is way too much. It&#39;s impressive (thanks for the nice compliments!) but it still has serious weaknesses and sometimes makes very silly mistakes. AI is going to change the world, but GPT-3 is just a very early glimpse. We have a lot still to figure out. / Twitter&quot;</h3><p>The GPT-3 hype is way too much. It&#39;s impressive (thanks for the nice compliments!) but it still has serious weaknesses and sometimes makes very silly mistakes. AI is going to change the world, but GPT-3 is just a very early glimpse. We have a lot still to figure out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1W6tu3wlCO9FjUK9TeGfyQ.jpeg" /><figcaption>Created by Author on <a href="https://www.befunky.com/create/designer/">befunky</a></figcaption></figure><h3>What is Language Model?</h3><p><a href="https://en.wikipedia.org/wiki/Language_model">Language model</a> tries to predict the next word given the m words. It assigns the probability to the next word which makes the sentence more probable.</p><h3>What is GPT-3?</h3><p>It’s the successor of GPT-2 language model which has highest number of parameters to trained. <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3</a> model is created by OpenAI and it proves that the language model size is proportional to the accuracy and hence, more we increase the model size more the accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*fheIOvGVhg5nweJ1g6iOYA.png" /><figcaption>Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>As we can see from the graph that increasing number of language models proportionally increases the accuracy. Therefore, it is possible to obtain higher accuracy by adding more parameters to the model.</p><p><a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3</a> is the largest language model present with 175 billion parameters 10 times bigger than the Turing-NLG model which has 17 billion parameters.</p><p>In this blog, we’ll go through the research paper of <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3</a> and will deduce why it’s just the another language model and why it cannot be called as the model that can imitate human at any level. It’s the best language model present but not the model that can understand and can learn the human language.</p><h3>Settings on which model is tested:</h3><ul><li><strong>Few-Shot (FS) </strong>refers that few examples are given to the model along with the task descriptions. So instead of full-fledged dataset for the specific task the model was given description along with some k example and then was asked the question. k examples was in range from 10 to 100 and the model was not allowed to do any weight change based on the example. The advantage was that the model does not require large dataset to learn the specific task but still the accuracy was not comparable to the state-of-the-art task specific models.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*qKHi9B6a1xSxl6CpPYwfQg.png" /><figcaption>Few-Shot setting example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><ul><li><strong>One-Shot (OS)</strong> is similar to few-shot where k is equal to one. So the models need to understand the task based on one example only. This type of learning is similar to how human learns. In this too, model was not allowed to change weights.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/1*s7bFtZhEdw4VQxTPjm6S-A.png" /><figcaption>One-Shot setting example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><ul><li><strong>Zero-Shot (ZS)</strong> has value of k as zero. No example is provided to the model. Model needs to understand just from the task description what the model needs to do.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*NHZFNs97EVTzWoWoTIESbA.png" /><figcaption>Zero-Shot setting example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>The accuracy of the model was compared to the state-of-the-art task specific model also known as fine-tuning models. The disadvantage of this model is just that they need huge task-specific dataset to be trained on. This type of models are task specific and will perform poorly on other tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*UFeJx3L1DPU0Pm-TOwcRYw.png" /><figcaption>Fine-Tuning model | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><h3>Training Dataset</h3><p>The dataset used by GPT-3 was huge almost containing everything present on the internet. The did the fuzzy search on the training dataset to remove anything related to the test and validation dataset which will help to give more accurate results. They created this dataset from merging various dataset and through crawling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*mAfVy6JsZBWw2IGpcpf6Lg.png" /><figcaption>Dataset used by GPT-3 | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><h3>Training Process</h3><p>They used gradient noise scale to find out the right batch size to be used. The architecture was similar to the GPT-2 with minor modifications.</p><h3>Results</h3><p>The GPT-3 was evaluated on various natural language tasks and on various datasets.</p><h4>Language Modeling, Cloze, and Completion Tasks</h4><h4><a href="https://arxiv.org/pdf/1606.06031.pdf">Lambada</a> Dataset</h4><pre>Context: “Yes, I thought I was going to lose the baby.” “I was scared too,” he stated, sincerity flooding his eyes. “You were ?” “Yes, of course. Why do you even ask?” “This baby wasn’t exactly planned for.” </pre><pre>Target sentence: “Do you honestly think that I would want you to have a _______ ?” </pre><pre>Target word: miscarriage</pre><p>In this <a href="https://arxiv.org/pdf/1606.06031.pdf">dataset</a> the context is given than sentence is given to the model with the blank which needs to be completed by the model. The word chosen should be based on the context given rather than the past knowledge model has. This dataset proves that model can understand the context and provides the answers based on that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*VBDOUWf5YPwCKis-D5JSrQ.png" /><figcaption>GPT-3 accuracy on Lambada dataset | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>As we can see that all settings of GPT-3 surpasses the accuracy of SOTA on this dataset. GPT-3 one-shot accuracy is less than the accuracy of zero-shot. We can imply that GPT-3 predicts more accurately with any example than when given one example. <em>It is possible that GPT-3 had the knowledge of the dataset through training set and in few-shot method the model can know which weights to use more accurately. Because when you do google search for </em><em>Yes, of course. Why do you even ask?&quot;This baby wasn&#39;t planned for.” you can get various links with answer to the statement.</em></p><h4><a href="https://arxiv.org/pdf/1905.07830.pdf">HellaSwag</a> Dataset</h4><pre>A woman is outside with a bucket and a dog. The dog is running around trying to avoid a bath. She …  </pre><pre>A. rinses the bucket off with soap and blow dry the dog’s head. <br>B. uses a hose to keep it from getting soapy. <br>C. gets the dog wet, then it runs away again. <br>D. gets into a bath tub with the dog.</pre><p>These are multiple choice questions in which model needs to select the most suitable option. This dataset is used to test the common sense of the model. The GPT-3 FS achieved the accuracy of 79.3 around 6 percent below the SOTA which is fine-tuned multi-task model ALUM.</p><h4><a href="https://www.cs.rochester.edu/~nasrinm/files/Papers/lsdsem17-shared-task.pdf">Story Cloze</a> Dataset</h4><pre><strong>Context</strong>: Sammy’s coffee grinder was broken. He needed something to crush up his coffee beans. He put his coffee beans in a plastic bag. He tried crushing them with a hammer. </pre><pre><strong>Right Ending</strong>: It worked for Sammy. </pre><pre><strong>Wrong Ending</strong>: Sammy was not that much into coffee.</pre><p>These dataset is also used for common sense test. In this the model is given the context and the model has to predict the right ending from the two options provided. GPT-3 FS has achieved the accuracy of 87.7% well below the SOTA but has shown a great improvement in GPT-3 ZS by 10%.</p><h4>Closed Book Question Answering</h4><p>Open Book Question Answering is used in search engines which can search the relevant information and can extract the answer from it. In closed book the model is not allowed to search and doesn’t have any context or excerpt provided to know what model is actually asking.</p><p><a href="https://homes.cs.washington.edu/~eunsol/papers/acl17jcwz.pdf"><strong>TriviaQA</strong></a></p><p>This dataset is a fact based that is the answer are the facts rather than any context depended. So, we can say that this dataset tests the memory of the model as GPT-3 is trained on wikipedia text too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*tewW1_UDOg4UiJZwnH2i4A.png" /><figcaption>TriviaQA example | Reference: <a href="https://homes.cs.washington.edu/~eunsol/papers/acl17jcwz.pdf">TriviaQA research paper</a>, Public Domain</figcaption></figure><p>The model is asked question and is expected the answer based on facts. Excerpt is only for our knowledge it is not feed to the model. Multiple answers are also allowed for example, if the answers is name Bill Gates the answer can be Bill, Gates, Bill Gates, Microsoft founder Bill Gates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*1U9lnhE4niKiN1FHbqDjVA.png" /><figcaption>TriviaQA accuracy | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>We can see that GPT-3 has surpassed the fine-tuned open book SOTA. So we can say that GPT-3 is more efficient as an information retrieval engine for facts.</p><p><a href="https://arxiv.org/pdf/1607.06275.pdf"><strong>WebQuestions</strong></a><strong> (WebQA)</strong></p><p>This dataset is similar to TriviaQA but the dataset consist of the real-world questions asked by the actual users in real world. So, it tests the model in real world scenario.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*dpcqqC9kN8p7ZmD-gtliQg.png" /><figcaption>WebQA example | Reference: <a href="https://arxiv.org/pdf/1607.06275.pdf">WebQA research paper</a>, Public Domain</figcaption></figure><p>The GPT-3 FS has similar accuracy to the fine-tuned SOTA models.</p><p><a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/1f7b46b5378d757553d3e92ead36bda2e4254244.pdf"><strong>Natural Questions</strong></a><strong> (NQs)</strong></p><p>This dataset is provided by the Google for testing the Question Answering model in real world. All the questions are searched by users on the google search. In the model tries to predict the paragraph from the wikipedia page which may contain the answer to the question. And model also predicts the short answer which will the exact answer to the question. But the short answer must be present in the paragraph selected also known as long answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*niA06St0mk4mrpKHsIbgrA.png" /><figcaption>Natural Question example | Reference: <a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/1f7b46b5378d757553d3e92ead36bda2e4254244.pdf">NQs research paper</a>, Public Domain</figcaption></figure><p>The accuracy fo GPT-3 FS setting is 29.9% well below the <a href="http://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/reports/custom/15791880.pdf">fine-tuned</a> SOTA. This dataset tests the limit of model to retain the information of wikipedia.</p><blockquote>Through this tests we can say that the bigger the model more the information can be retained and hence, higher the accuracy.</blockquote><h4>Translation</h4><p>The accuracy of GPT-3 in language translation is a surprise. As 93% words of the training set are English. So, the model will be biased toward English language more than the other language. GPT-3 is the upgrade over GPT-2 and is tested for machine translation for languages French, German, and Romanian other than English.</p><p>The model efficiency for the Machine Translation task is measured in <a href="https://en.wikipedia.org/wiki/BLEU">BLEU</a> rather than as accuracy. We’ll look into it some other time.</p><p><a href="https://www.aclweb.org/anthology/P02-1040.pdf">BLEU</a> is calculated on 6 language pairs:</p><ol><li>English to French (En→Fr)</li><li>French to English (Fr→En)</li><li>English to German (En→De)</li><li>German to English (De→En)</li><li>English to Romanian (En→Ro)</li><li>Romanian to English (Ro→En)</li></ol><p>The training dataset had around 92.5% of words in English, 1.8% of words in French, 1.5% in German and 0.16% of words in Romanian.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*2O1Pb37WhqfWXI1d2l19CA.png" /><figcaption>BLEU score on 6 language pairs | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>We can see that the GPT-3 model is almost or better than the SOTA when translating any text to English.</p><p><strong>Zero-Shot Example:</strong></p><pre>Q: What is the {language} translation of {sentence} </pre><pre>A: {translation}.</pre><p>As we can see, only the description text is provided to the model along with the sentence and the targeted language.</p><p><strong>One- and Few-Shot Example:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ObDT_geBj6tGzZ7m1o8Cmw.png" /><figcaption>Format for one- and few-shot learning | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>For Few-Shot around 64 examples were given to the model before asking the question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*3IdzaxAvuowBqjB7uvQ6TQ.png" /><figcaption>Few-Shot BLEU score | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>As we can see as the data increased the BLEU score of the model increased. The score of English-&gt;French is greater than English-&gt;German which is greater than English-&gt;Romanian and is proportional to the percentage of words present of each language in the dataset.</p><h4>Winograd-Style Tasks</h4><p>This type of tasks involves determining what the pronoun used in the sentence resemble. The pronoun is ambiguous for the computer but unambiguous for humans. The partial evaluation method was used for evaluation about which you can read <a href="https://arxiv.org/pdf/1811.01778.pdf">here</a>.</p><pre>&quot;<strong>sentence</strong>&quot;: &quot;The city councilmen refused the demonstrators a permit because [they] feared violence.&quot;, </pre><pre>&quot;<strong>answer1</strong>&quot;: &quot;The demonstrators&quot;<br>&quot;<strong>answer0</strong>&quot;: &quot;The city councilmen&quot;</pre><pre>&quot;<strong>sentence_switched</strong>&quot;: &quot;The demonstrators refused the city councilmen a permit because [they] feared violence.&quot;</pre><p>The answer for the sentence must be answer1 but once we switch the sentence the answer should also change to answer0</p><p><a href="https://aaai.org/ocs/index.php/KR/KR12/paper/view/4492"><strong>Winograd</strong></a><strong> Dataset</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ebbmBaQZl842ltJo6MUhDA.png" /><figcaption>Winograd Example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>GPT-3 performance on this dataset is below SOTA. Some examples of the test set was also found in the training dataset.</p><p><a href="https://arxiv.org/pdf/1907.10641.pdf"><strong>Winogrande</strong></a><strong> Dataset</strong></p><p>This dataset is similar to winograde dataset but more advanced and difficult.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PAqaAUCw5aSSdtE1d-QRjA.png" /><figcaption>Winogrande Dataset | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*QZVf2GL173huiBZUP0mK7w.png" /><figcaption>Accuracy of GPT-3 on Winograde-Style tasks | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><h4>Common Sense Reasoning</h4><p>This test is conducted to test how the model applies the common sense in real world questions. For example, if we ask the human what’s the time? Human will respond the current time such as 5:30 PM rather than explain the concept of time. We check whether the model can answer such questions using common sense. Previous all tasks where mostly dependent on the data and probability rather than how much the model actually understands the inner concepts. Here, we test how much the model understand the human language.</p><p><a href="https://arxiv.org/pdf/1911.11641.pdf"><strong>PhysicalQA</strong></a><strong> Dataset</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*Lz1xZFhNsGEmz4j6r_uCMA.png" /><figcaption>PIQA example | Reference: <a href="https://arxiv.org/pdf/1911.11641.pdf">PhysicalQA research paper</a>, Public Domain</figcaption></figure><p>For all types of tuning GPT-3 beats the SOTA on this dataset with a small margin. But the confidence on this result was not much due to possibly data contamination.</p><blockquote>Our analysis flagged PIQA for a potential data contamination issue (despite hidden test labels), and we therefore conservatively mark the result with an asterisk.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*oZvwvL5kCnkyZHzve1SoVA.png" /><figcaption>GPT-3 performance on PIQA dataset | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p><a href="https://arxiv.org/pdf/1803.05457.pdf"><strong>AI2 Reasoning Challenge Dataset (ARC)</strong></a></p><p>This dataset consist of the multiple-choice science questions of 3rd to 9th grade. The dataset is divided into parts: 1. Easy version 2. Challenge version. The challenge set consist of the questions which were wrongly answered by the retrieval-based algorithms.</p><p>GPT-3 performance on this dataset was very much below the SOTA achieved by <a href="https://arxiv.org/pdf/2005.00700.pdf">UnifiedQA</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fWIl1tq971Klkph949CGHQ.png" /><figcaption>ARC challenge set example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p><a href="https://arxiv.org/pdf/1809.02789.pdf"><strong>OpenBookQA</strong></a><strong> Dataset</strong></p><p>This dataset are similar to open book tests and the answer to the question is combination of fact and common sense.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*S2tPDMG7MmYYHAHWRyTAwQ.png" /><figcaption>OpenBookQA example | Reference: <a href="https://arxiv.org/pdf/1809.02789.pdf">OpenBookQA research paper</a>, Public Domain</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*0sPUXMvbMcuQVNxDkMpPKA.png" /><figcaption>GPT-3 accuracy in common sense reasoning tasks | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>As GPT-3 is trained on all the text present on internet it’s a good question to answer that whether GPT-3 has applied common sense to answer this questions and not just retrieved the information available somewhere in the training dataset.</p><h4>Reading Comprehension</h4><p>Here, we test the ability of the model on understanding the passage and answering the questions based on the information provided in the passage.</p><p><a href="https://stanfordnlp.github.io/coqa/"><strong>CoQA Dataset</strong></a></p><p>In this dataset the passage is provided and the questions asked are based on the conversation among humans.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eAB4WYIq2-yEvwbSbtryOQ.png" /><figcaption>CoQA example for few-shot tuning | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p><a href="https://quac.ai/"><strong>QuAC Dataset</strong></a></p><p>This dataset is very interesting as in this the student does not see the passage but asks questions which are answered by the teacher. The student asks the basic question and teacher answer with some extra context based on which the student asks another question. The interaction ends if the teacher is not able to answer too many question or if student does not ask any relevant question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d5iDXNRqzpc_3EXFxjIxWg.png" /><figcaption>QuAC dataset example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p><a href="https://allennlp.org/drop">DROP Dataset</a></p><p>In this dataset a model must resolve references in a question, perhaps to multiple input positions, and perform discrete operations over them such as addition, counting, or sorting.</p><pre><strong>Passage</strong>: Although the movement initially gathered some 60,000    adherents, the subsequent establishment of the Bulgarian Exarchate reduced their number by some 75%.</pre><pre><strong>Question</strong>: How many adherents were left after the establishment of the Bulgarian Exarchate?</pre><pre><strong>Correct Answer</strong>: 15,000</pre><p>As we can see here the model needs to understand the passage and question in order to answer it correctly and it should also perform some operations on it rather than just retrieve the information.</p><p><a href="https://rajpurkar.github.io/SQuAD-explorer/"><strong>SQuAD 2.0 Dataset</strong></a></p><p>In this along with the questions for which the answer is present in the passage the questions for which there is no answer present in the passage are also asked. The model is expected not to answer the question for which the passage does not contain the answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E5655AJkEVKF95g551rzJg.png" /><figcaption>SQuAD 2.0 example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p><a href="https://www.aclweb.org/anthology/D17-1082/"><strong>RACE Dataset</strong></a></p><p>This dataset consists the passages from the English exams for middle and high school Chinese students in the age range between 12 to 18 designed to evaluate the students reasoning capability. The dataset is divided into two sections: 1. RACE-m dataset consist of the passages of middle school 2. RACE-h dataset consists the passages of high school.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*57Oz-biF5vNXaKvKinZsyA.png" /><figcaption>RACE-m Example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*Kj95yqHWubR_MJWEFCwf7w.png" /><figcaption>Accuracy of GPT-3 on Reading Comprehension | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><h4><a href="https://super.gluebenchmark.com/">SuperGLUE</a></h4><p>It can be considered as the standardized test for all the NLP model which are able to perform multiple language modeling tasks. It gives the single result based on how model performed in different tasks. As the result is ingle value it’s easy to compare with different models on same scale.</p><p>This benchmark includes the following tasks:</p><ol><li>Question Answering (QA)</li><li>Natural Language Inference (NLI)</li><li><a href="https://demo.allennlp.org/coreference-resolution">Coreference Resolution (coref.)</a></li><li><a href="http://www.scholarpedia.org/article/Word_sense_disambiguation">Word Sense Disambiguation (WSD)</a></li></ol><p>Dataset’s used:</p><ol><li><a href="https://arxiv.org/abs/1905.10044">BoolQ</a></li><li><a href="https://github.com/mcdm/CommitmentBank">Commitment Bank</a></li><li><a href="https://people.ict.usc.edu/gordon/public_html/publications/AAAI-SPRING11A.PDF">COPA</a></li><li><a href="https://github.com/CogComp/multirc">MultiRC</a></li><li><a href="https://sheng-z.github.io/ReCoRD-explorer/">ReCoRD</a></li><li><a href="https://aclweb.org/aclwiki/Textual_Entailment_Resource_Pool">RTE</a> (RTE1, RTE2, RTE3 and RTE5 are used)</li><li><a href="https://pilehvar.github.io/wic/">WiC</a></li><li><a href="https://aaai.org/ocs/index.php/KR/KR12/paper/view/4492">WSC</a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*zztZU_sFvH563NexIN3BbA.png" /><figcaption>SuperGLUE benchmark | Reference: <a href="https://super.gluebenchmark.com/">SuperGLUE research paper</a>, Public Domain</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*fecbjKvvuAqmoVkd_3kCLQ.png" /><figcaption>Performance of GPT-3 on SuperGlue | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>For GPT-3 few-shot tuning 32 examples were used in context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*fu2Wht03d_cF3XWXbDIOHw.png" /><figcaption>GPT-3 FS examples for SuperGLUE Benchmark | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>As we can see more the examples the accuracy of GPT-3 increased but in very small number. GPT-3 FS surpassed the fine-tuned BERT-Large when 8 examples were given in context.</p><h4>Natural Language Inference (NLI)</h4><p>In this type of task the model determines the relation between two sentences. It’s a classification type of problem where model classifies the second sentence into three classes: 1. Supports the first sentence 2. Contradicts the first sentence 3. Neutral</p><p>RTE dataset is used for testing the model on NLI task. Along with RTE more difficult Adversarial NLI (ANLI) was used for testing the GPT-3 model. IN <a href="https://adversarialnli.com/#">ANLI</a> GPT-3 performed slightly better than random chance but well below the SOTA.</p><p><a href="https://arxiv.org/pdf/1910.14599.pdf">ANLI</a> is divided into three sets based in difficulty: round 1, round 2, round 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hT7VkZ_ptjzxfhQ1JWiVTQ.png" /><figcaption>ANLI R1 Example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3rXH4rqx1atfiHwQwJt4Lw.png" /><figcaption>ANLI R2 Example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mU7ChdWKMHZbXl1hoxn5bw.png" /><figcaption>ANLI R3 Example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><h3>Synthetic and Qualitative Tasks</h3><p>In this the GPT-3 is tested to perform various synthetic tasks such as arithmetic operations, rearranging and unscrambling the letters in a word, solve SAT-style analogy problems, using new words in a sentence, correcting English grammar, and news article generation.</p><h4>Arithmetic</h4><p>In this the ability of GPT-3 to perform arithmetic operations is tested. Various arithmetic operations such as addition, subtraction and multiplication are tested.</p><p>The arithmetic problems consists of the following types:</p><ol><li>2 digit addition (2D+)</li><li>2 digit subtraction (2D-)</li><li>3 digit addition (3D+)</li><li>3 digit subtraction (3D-)</li><li>4 digit addition (4D+)</li><li>4 digit subtraction (4D-)</li><li>5 digit addition (5D+)</li><li>5 digit subtraction (5D-)</li><li>2 digit multiplication (2Dx)</li><li>One-digit composite (1DC)</li></ol><p>The model performs better in few-shot setting than other type of settings. And, as the number of digits increases the accuracy reduces greatly. GPT-3 has made a huge jump on this set than the smaller models.</p><pre>Examples of the questions asked to GPT-3:</pre><pre>Q: What is 17 minus 14?<br>Q: What is (2 * 4) * 6?<br>Q: What is 95 times 45?<br>Q: What is 98 plus 45?</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*pf3gZ9wUxQNuM42ZtaBJZg.png" /><figcaption>Accuracy of GPT-3 on arithmetic operations | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*DSxvMtn51NAGIDADS-rZRw.png" /><figcaption>GPT-3 FS results | Reference: GPT-3 research paper</figcaption></figure><p>As the digit gets increased the accuracy of the model decreases exponentially. And more the complex operations such as multiplication and composite equations the model performance is degraded.</p><p><em>The model may have memorized the smaller digit arithmetic operations as they are available more frequently on the internet than the big numbers. The training set was searched for “&lt;NUM1&gt; + &lt;NUM2&gt; = ” and “&lt;NUM1&gt; -&lt;NUM2&gt; = ” and only 0.8% and 0.1% matched were found for the test cases. But, it is possible that the training set may have “forty-two + twenty-three equals sixty-five” or “adding forty-two to twenty-three gives sixty-five” type of equations and the model may have mapped the integer and operations to this strings.</em></p><p>The model was inspected on wrong results and was found that it works incorrectly when the operation includes carry.</p><h4>Word Scrambling and Manipulation Tasks</h4><p>In this various operations are done on the word and is given as an input to the model. The model needs to predict the original word.</p><p>This tasks include 5 type of tasks:</p><ol><li><strong>Cycle letters in word (CL)</strong> — The letters of the words are cycled and model is asked to find the original word.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*HGruYL0UeumEwo12rSB01g.png" /><figcaption>Cycled letters example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>2.<strong> Anagrams of all but first and last characters (A1)</strong> — All the letters of the word are scrambled except the first and last letter. The model needs to output the original word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uBe5AKx0qc6pf6ESa-uFFg.png" /><figcaption>A1 Anagram example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>3. <strong>Anagrams of all but first and last 2 characters (A2)</strong> — All the letters of the word are scrambled except the first 2 and last 2 letters. The model need to output the original word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mfNvUfouSHKAwpYaW9RhyA.png" /><figcaption>A2 Anagram example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>4. <strong>Random insertions in word (R1) </strong>— A random punctuation or space is inserted between letters and model needs to predict the original word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0EJh1KKdNRh5oJFt17joGg.png" /><figcaption>Random insertions (R1) example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>5. <strong>Reversed words (RW)</strong> — The word is spelled backwards and the model needs to predict the original word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KYEUdUeoJx_R6uNqF9bbkg.png" /><figcaption>Reversed word (RW) example | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>The model performs poorly in zero-shot setting and shows average performance in one-shot setting. The model was given 100 examples for few-shot setting in which it performs considerably well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*ntGrUJmwqpPDKMP0utx6NA.png" /><figcaption>Accuracy on unscrambled task | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><h4><a href="https://arxiv.org/pdf/cs/0309035.pdf">SAT Analogies</a></h4><p>In this the model needs to answer the questions based on analogy. This questions were asked in SAT the college entrance exam. The random guessing accuracy is 20% while the average score of college students is 57%. GPT-3 few-shot setting achieves 65.2% of accuracy and one-shot setting achieves 59.1% accuracy well above the student average. The accuracy of zero-shot setting is 53.7%. For few-shot setting number of examples given were 20.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*I-mId2etLEXBqPW-drWIJg.png" /><figcaption>Example of SAT analogies | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><h4>News Article Generation</h4><p>In this the model is provided the title and subtitle using which it needs to write an article of around 200 words. The model was tested on few-shot setting only as model was not able to understand whether it needs to write an article or need to respond as follow up tweet.</p><p>The GPT-3 performance is compared with the control model which deliberately produces bad articles. Humans were used to distinguish the article whether it was human made or made by model. Humans accurately detected the article was generated by control model with 86% accuracy. But as the model size increased the human were able to detect with 52% accuracy that the article was generated by the model.</p><p>Most articles which were detected as made by the model contained wrong factual information or had a repeated sentences and unusual phrasing.</p><p>The model was also tested on articles with 500 words but still the human were able to distinguish with human articles by 52% only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*Lq-Y-n6ihtYWAhY_-rp63w.png" /><figcaption>Article generated by GPT-3 which was detected with only 12% accuracy | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p><em>When you google any sentence from the above article you’ll get various articles on the some topic with similar sentences. It’s possible that GPT-3 as trained on whole internet may just have summarised all the articles of the relevant topic in less than 500 words. Instead of generating articles it maybe just merging various sentences from different articles hence, making articles similar to humans. We will know whether the GPT-3 can generate the article if we provide the title very unique</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*9SD8-bcM2OenzUm9T6Zq2A.png" /><figcaption>Human’s accuracy in detecting the 500 words of article is written by human or the model | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><h4>Learning and Using Novel Words</h4><p>In this the model is tested if it can use the word in a sentence after defining the word once.</p><p>So to test the model on this task the word definition is given once so one-shot setting but the example is provided for the each word defined hence, few-shot setting with respect to task description.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5uuy-E5NgIEqtOU4Zcf8Hg.png" /><figcaption>Example of defining and using new word in GPT-3 | Reference: GPT-3 research paper</figcaption></figure><p>The words were human made and hence, model has no prior knowledge of that. We can see the model generated all the sentences logical using the defined word.</p><h4>Correcting English Grammar</h4><p>The model is tested if it ca predict the wrong grammar and can modify the sentence with right grammar. The model is tested on few-shot setting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*Y33bYgogTCiYjEvBPQYjtA.png" /><figcaption>Correcting english grammar | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p><em>As we can see, the model is tested on very basic grammar instead of complicated grammar such as clauses. It’ll be good to see the model performance on such bad grammar as it’s a complicated task even for humans.</em></p><h3>Measuring and Preventing Memorization Of Benchmarks</h3><p>As the training dataset was generated through crawling the internet it is possible that the training dataset may contain various test set examples. If it happens then the confidence on the accuracy reported by the model on various tasks may be unreliable.</p><p>The training dataset was searched for the test set example and the documents are considered match if N words are similar among both the sets. N is equal to the 5th percentile example length in words of each sets. Smaller value of N caused huge number of matches which were found illogical. So minimum value of N was kept 8 and maximum 13. N words are matched or the whole sentence is matched depending on the minimum length. If the match is found the example is considered as dirty otherwise clean example.</p><p>GPT-3 was tested for various tasks on the clean dataset and the accuracy is compared with the original score. If the accuracy almost matches then it means that contamination even if present does not effect the model accuracy. If the accuracy is lower than the original score implies that contamination is inflating the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*D8NzFmtweNuitHB8tbpx_w.png" /><figcaption>Benchmark Contamination analysis | Reference: <a href="https://arxiv.org/pdf/2005.14165.pdf">GPT-3 research paper</a>, Public Domain</figcaption></figure><p>For dataset QuAC 0% of clean data is present in the dataset and still the accuracy changed positively by 20% which must mean there was something wrong with it. They found around 90% of dataset contaminated. But, after further analysis deduced that the training set included only passages but not question and answers.</p><p>GPT-3 has various limitations and there is still chance of improvements in fields such as text synthesis. GPT-3 is based on the training dataset and hence, can’t think of it’s own. If majority of humans has some bias towards gender, race, religion then the model will also represent those bias. For more limitations and bias you can read <a href="https://arxiv.org/pdf/2005.14165.pdf">here</a>. <a href="https://www.technologyreview.com/2020/08/22/1007539/gpt3-openai-language-generator-artificial-intelligence-ai-opinion/">MIT technology review</a> also published a blog on limitations of GPT-3.</p><p>GPT-3 is model which just predicts which word should come next. It does not understand the context and does not understand the actual meanings of the word. The model lacks logical reasoning and common sense reasoning. The model output can be changed just by tweaking the dataset.</p><h3></h3><p>gpt3 is surprising and creative but it&#39;s also unsafe due to harmful biases. Prompted to write tweets from one word - Jews, black, women, holocaust - it came up with these (https://t.co/G5POcerE1h). We need more progress on #ResponsibleAI before putting NLG models in production. pic.twitter.com/FAscgUr5Hh</p><p>Here are great resources from where you learn more about GPT-3:</p><ul><li><a href="https://towardsdatascience.com/you-can-understand-gpt-3-with-these-youtube-videos-6a30887c928b">You Can Understand GPT-3 With These YouTube Videos</a></li><li><a href="https://analyticsindiamag.com/top-free-resources-to-learn-gpt-3/">Top Free Resources To Learn GPT-3 - Analytics India Magazine</a></li></ul><p><em>Thanks for reading and have a great day.</em></p><p><em>*text in italics are just my personal views</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1add6e9277fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/gpt-3-just-another-language-model-but-bigger-1add6e9277fa">GPT-3: Just Another Language Model But Bigger</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Latent Semantic Analysis — Deduce the hidden topic from the document]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/latent-semantic-analysis-deduce-the-hidden-topic-from-the-document-f360e8c0614b?source=rss-4a8e3136747c------2"><img src="https://cdn-images-1.medium.com/max/600/1*EECZMH6ZpM8QjKl0joa0fw.png" width="600"></a></p><p class="medium-feed-snippet">Understand the most primary part of search engines and how they find relevant links to the user&#x2019;s query.</p><p class="medium-feed-link"><a href="https://medium.com/data-science/latent-semantic-analysis-deduce-the-hidden-topic-from-the-document-f360e8c0614b?source=rss-4a8e3136747c------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/latent-semantic-analysis-deduce-the-hidden-topic-from-the-document-f360e8c0614b?source=rss-4a8e3136747c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f360e8c0614b</guid>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[topic-modeling]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sanket Doshi]]></dc:creator>
            <pubDate>Wed, 26 Feb 2020 12:15:08 GMT</pubDate>
            <atom:updated>2020-02-26T16:26:47.818Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Traffic Sign Detection using Convolutional Neural Network]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/traffic-sign-detection-using-convolutional-neural-network-660fb32fe90e?source=rss-4a8e3136747c------2"><img src="https://cdn-images-1.medium.com/max/699/1*ePgozv3WJymCYHUQdKHELw.jpeg" width="699"></a></p><p class="medium-feed-snippet">We will be building a CNN model in order to detect traffic signs.</p><p class="medium-feed-link"><a href="https://medium.com/data-science/traffic-sign-detection-using-convolutional-neural-network-660fb32fe90e?source=rss-4a8e3136747c------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/traffic-sign-detection-using-convolutional-neural-network-660fb32fe90e?source=rss-4a8e3136747c------2</link>
            <guid isPermaLink="false">https://medium.com/p/660fb32fe90e</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[traffic-signs]]></category>
            <category><![CDATA[convolutional-neural-net]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sanket Doshi]]></dc:creator>
            <pubDate>Sun, 01 Sep 2019 11:02:39 GMT</pubDate>
            <atom:updated>2019-12-08T08:44:05.831Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Convolutional Neural Network: Learn And Apply]]></title>
            <link>https://medium.com/@sdoshi579/convolutional-neural-network-learn-and-apply-3dac9acfe2b6?source=rss-4a8e3136747c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3dac9acfe2b6</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cnn]]></category>
            <dc:creator><![CDATA[Sanket Doshi]]></dc:creator>
            <pubDate>Sun, 31 Mar 2019 11:14:25 GMT</pubDate>
            <atom:updated>2019-03-31T11:14:25.442Z</atom:updated>
            <content:encoded><![CDATA[<p>After the advancement in the neural network, everyone tried to implement it in the computer vision field. As we achieve great accuracy due to neural networks in computer vision fields there were also some disadvantages such as computing complexity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*gpqAAvdoj-VcFlKK8y9zAQ.jpeg" /><figcaption>Image recognition using CNN</figcaption></figure><p>As when using deep neural networks the model would require a large number of weights and huge matrix multiplications which requires more computation powers to tackle with this new type of algorithm was introduced. This algorithm is known as a Convolutional Neural Network or CNN. It was found that CNN is more efficient and faster than a regular deep neural network for problems related to computer vision.</p><p>Here, we’ll learn about CNN, it’s working and a standard architecture used. We’ll also implement CNN for hand-written digit recognition.</p><h3>Introduction</h3><p>A Convolutional Neural Network (ConvNets/CNN) are very similar to ordinary neural networks. They are made up of neurons with learnable weights and biases. All the tips and tricks learned or used for ordinary neural networks are also applicable to CNN’s.</p><p>The only difference between CNN and the ordinary neural network is that CNN assumes that input is an image. This vastly reduces the number of parameters to be tuned in a network.</p><h3>Why ConvNets over normal neural networks?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*ZG9Fiz7A_wEFAd34xLQXbA.jpeg" /><figcaption>Feeding image to normal neural networks</figcaption></figure><p>An image is nothing but a matrix of pixel values. So if an image is of size 40*40*1 we can flatten it and then feed it to the model. So, we would require 1600 neurons at an input layer. But not all the images are black and white some may be colored images. So each colored image has 3 layers each layer for corresponding to red, green and blue colors respectively. So, now to feed colored image with size 40*40*3 we need 4800 neurons at an input layer. This amount still seems manageable, but clearly, this fully-connected structure does not scale to larger images. Suppose, we have an image of size 1024*1024*3 so now we need approx. 3.1 million neurons in an input layer. So, you see as the size of an image increases the number of neurons required increases exponentially which makes the model computationally expensive and not feasible.</p><p>Comparatively, in CNN we assume the input is an image hence we can use this information such that we can make our model computationally inexpensive and faster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*hdiGCaculbrmLt4iTYD8Iw.gif" /><figcaption>Working of CNN</figcaption></figure><p>As we can see that input image is of size 10*10*1 and the kernel size is 3*3*1 and the output is 8*8*1 image. So in CNN rather than computing each pixel separately, we process on the group of pixels. And this group is defined by the kernel size. CNN only tunes the values of the kernel and hence only 9 variables are required to be tuned which is much lower than in normal neural network. Now, if the image is of dimension 10*10*3 then kernel size will also be equal to 3*3*3 and hence we would require 27 parameters to tune which is still very less. Here, the output image would be still of size 8*8*1.</p><p>Clearly, CNN is a favorable choice over the normal neural networks when processing is required on images.</p><h3>Architecture Overview</h3><p>Similar to traditional neural networks convolutional neural networks is also a sequence of layers. Each layer transforms an input image to an output image with some differentiable function that may or may not have parameters.</p><p>The ConvNet architecture consists of three types of layers: Convolutional Layer, Pooling Layer, and Fully-Connected Layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*kkyW7BR5FZJq4_oBTx3OPQ.png" /><figcaption>Simple architecture</figcaption></figure><p>A simple ConvNet architecture could have the architecture[INPUT-CONV-RELU-POOL-FC]. In more detail:</p><ol><li>INPUT layer would hold the input image as a 3-D array of pixel values.</li><li>CONV layer will compute the dot product between the kernel and sub-array of an input image same size as a kernel. Then it’ll sum all the values resulted from the dot product and this will be the single pixel value of an output image. This process is repeated until the whole input image is covered and for all the kernels.</li><li>RELU layer will apply an activation function max(0,x) on all the pixel values of an output image.</li><li>POOL layer will perform downsampling along the width and height of an image resulting in reducing the dimension of an image.</li><li>FC (Fully-Connected) layer will compute the class score for each of the classification category.</li></ol><p>In this way, ConvNets transform the original image layer by layer from the original pixel values to the final class scores. RELU and POOL layer implement the constant function and no variables are to be trained at this layer. Parameters at FC and CONV layer will be trained using gradient descent optimizer.</p><h3>Convolutional Layer</h3><p>This layer is the core of ConvNet. It does most of the computational heavy lifting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*D6iRfzDkz-sEzyjYoVZ73w.gif" /><figcaption>Convolutional layer working</figcaption></figure><h4>Hyperparameters used in this layer:</h4><ol><li>The <strong>depth</strong> of an output volume represents the number of layers present. This value depends on the number of filters used. In the above image, the depth is 1 as it’s a 2-D image.</li><li><strong>Filter size (f)</strong> represents the height and width of the filters used. The depth of a filter is not defined as it’s same as the depth of an input image. For a given image, the filter size is 3.</li><li><strong>Stride (s) </strong>represents the step size to take when traversing horizontally and vertically along the height and weight of an input image. For a given image, the stride is equal to 1.</li><li><strong>Padding (p)</strong> helps to retain the height and width of an input image. As after processing the size of an input image reduces and hence we cannot model deep networks with many of these layers for small size images. To make . deep networks feasible padding is used. In padding, we add multiple rows and columns around the image. This added pixel has values 0 by default as shown below. One more advantage of padding is that it helps us retain more information on corner pixels as now they will be processed multiple times instead of single time.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*n3N5_oZ1NESyVah3cQF3gQ.png" /><figcaption>using padding</figcaption></figure><p>All the above hyperparameters are fixed for a given layer. Multiple convolutional layers may have different values. The filter matrix is also a weight matrix for which values need to be trained using backpropagation.</p><h4>Working</h4><ol><li>The input 3-D volume is passed to this layer. The dimension would be H*W*C. H, W, and C represent height, width, and the number of channels respectively.</li><li>There can be K filter used where K represents the depth of an output volume. The dimension of all the K filters is the same which is f*f*C. f is the filter size and C is the number of channels input image has.</li><li>If we have configured padding then will add padding to the input volume. If padding is equal to same then will add one row and one column at each side of the dimension and the value would be zero. Padding is applicable only along the height and width of an input dimension and is applied along each layer.</li><li>After padding, the computation begins. Now we’ll slide our filter starting from the top-left corner. The corresponding values of filter and input volume are multiplied and then the summation of all the multiplied value takes place. Now the filter is slide horizontally taking stride number of step in each slide. So, if stride is 2 we’ll slide 2 columns horizontally. The same process is repeated vertically until the whole image is covered.</li><li>After getting all the values from the filter computation they are passed through RELU activation which is max(0,x). In this, all the negative values obtained are replaced by zero as negative values have no significance in the pixel.</li><li>Step 4 &amp; 5 generates just one layer of an output volume that is the 3-D input volume is transformed into a 2-D volume.</li><li>Now, step 4 &amp; 5 get’s repeated for K filters. And the output of each filter is stacked above one another and hence the depth of an output image is of dimension k.</li></ol><p>Now, to calculate the dimension of an output volume we require all the hyperparameters of a convolutional layer. All the filters used at this layer needs to be trained and are initialized with random small numbers.</p><p>The height and weight of an output volume is given by</p><p><strong>height, weight = floor( ( W+2*P-F )/S +1 )</strong></p><p><strong>depth = K (number of filters used)</strong></p><p>This is how the convolutional layer works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*m1SLhHhct4zRkBKQtLCbYQ.gif" /><figcaption>Computation at this layer</figcaption></figure><h3>Pooling Layer</h3><p>This layer is used for reducing the dimension of an input volume. This layer does not reduce the depth of an input. This layer can be used to reduce the spatial size so that the computational power required to process the image is reduced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*QkgfGDblW8YeTkft4otM5g.png" /><figcaption>Max Pooling</figcaption></figure><p>Pooling layer does not lose the important property of an image. Pooling layer extracts the most dominant information and hence maintains the process of effective training of the model.</p><p>Two types of pooling are used: <strong>Max Pooling</strong> and <strong>Average Pooling</strong>.</p><p>In max pooling, the maximum value present in a selected kernel is retained and all the other values are discarded. While on average pooling the average of all the values present in a kernel selected is stored.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*fNojhS9d1WD5KKDDrEaXEw.png" /><figcaption>Both types of pooling</figcaption></figure><p>Pooling also acts as a noise suppressant. But max pooling performs better than average pooling and hence, it is more frequently used.</p><h4>Hyperparameters used in this layer:</h4><ol><li><strong>Filter size (f)</strong> represents the size of the kernel to be used.</li><li><strong>Stride (s)</strong> represents the number of steps to take while sliding the kernel window.</li><li><strong>Padding (p)</strong> represents how much padding to apply on an input image. Usually at this layer padding is not used.</li></ol><p>The filters used does not need to train. Hence, backpropagation has no effect on this layer. And once the hyperparameters are fixed they never change.</p><p>Now, the dimension of an output volume can be calculated similarly as calculated at the convolutional layer. Here, the depth of an output volume is similar to the depth of an input volume.</p><h3>Fully-Connected Layer (FC-Layer)</h3><p>This layer is used for the classification of the complex features extracted from previous layers. This layer is the same as the neural networks in which each neuron is connected to all the neurons on a consecutive layer. The final output is calculated using softmax which gives the probability of each class for the given features.</p><p>The working of FC-layer is similar to the deep neural networks used for classification.</p><p>To pass an input image to the FC-layer we need to flatten out the image so that all the pixel values are arranged in one column. Now, this flattened feature is passed to the FC-layer. All the weights assigned in FC-layer are initialized by small random number and are trained using back-propagation algorithm.</p><p>Now, we’ve learned about all the types of layers used in ConvNets and hence, we can now apply this knowledge for recognizing hand-written digits from images. We’ll be using keras for implementing ConvNet.</p><h3>Implementation</h3><p>You can get the dataset from entering this <a href="https://datahack.analyticsvidhya.com/contest/practice-problem-identify-the-digits/"><em>practice contest</em></a>.</p><pre>%pylab inline<br>import os<br>import numpy as np<br>import pandas as pd<br>from scipy.misc import imread<br>from sklearn.metrics import accuracy_score</pre><pre>import tensorflow as tf<br>import keras<br>from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D<br>from keras.optimizers import Adam</pre><p>Now we’ll create a model with 6 Convolutional layers, 3 max-pooling layers, and FC-layer.</p><pre>input_reshape = (28, 28, 1)</pre><pre>hidden_num_units = 2048<br>hidden_num_units1 = 1024<br>hidden_num_units2 = 128<br>output_num_units = 10</pre><pre>epochs = 10<br>batch_size = 16</pre><pre>model = Sequential([</pre><pre>Conv2D(16, (3, 3), activation=&#39;relu&#39;, input_shape=input_reshape, padding=&#39;same&#39;),</pre><pre>Conv2D(16, (3, 3), activation=&#39;relu&#39;, padding=&#39;same&#39;),<br> MaxPooling2D(pool_size=pool_size),<br> Dropout(0.2),<br>    <br> Conv2D(32, (3, 3), activation=&#39;relu&#39;, padding=&#39;same&#39;),<br>    <br> Conv2D(32, (3, 3), activation=&#39;relu&#39;, padding=&#39;same&#39;),<br> MaxPooling2D(pool_size=pool_size),<br> Dropout(0.2),<br>    <br> Conv2D(64, (3, 3), activation=&#39;relu&#39;, padding=&#39;same&#39;),<br>    <br> Conv2D(64, (3, 3), activation=&#39;relu&#39;, padding=&#39;same&#39;),<br> MaxPooling2D(pool_size=pool_size),<br> Dropout(0.2),</pre><pre>Flatten(),</pre><pre>Dense(units=hidden_num_units, activation=&#39;relu&#39;),<br> Dropout(0.3),<br> Dense(units=hidden_num_units1, activation=&#39;relu&#39;),<br> Dropout(0.3),<br> Dense(units=hidden_num_units2, activation=&#39;relu&#39;),<br> Dropout(0.3),<br> Dense(units=output_num_units, input_dim=hidden_num_units, activation=&#39;softmax&#39;),<br>])</pre><p>You can see we’ve used theDropout method. It drops some neurons from training and hence helps in preventing overfitting. Dropout(0.2) represents 20% of neurons to be dropped randomly in every cycle. It is also a regularization method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*Fv7-HqpC_OwDT21L7PE7gA.png" /><figcaption>Dropout</figcaption></figure><pre>model.compile(loss=&#39;categorical_crossentropy&#39;, optimizer=Adam(lr=1e-4), metrics=[&#39;accuracy&#39;])</pre><pre>trained_model_conv = model.fit(train_x_temp, train_y, epochs=epochs, batch_size=batch_size, validation_data=(val_x_temp, val_y))</pre><p>We compile and fit the model. This model achieved 99.34% accuracy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7afa951c49a16731c743c383dc666d93/href">https://medium.com/media/7afa951c49a16731c743c383dc666d93/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3dac9acfe2b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>