<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Harsh Chauhan on Medium]]></title>
        <description><![CDATA[Stories by Harsh Chauhan on Medium]]></description>
        <link>https://medium.com/@harshtherocking?source=rss-e2bab4d88fd7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*qASVyBueOew8-r7l</url>
            <title>Stories by Harsh Chauhan on Medium</title>
            <link>https://medium.com/@harshtherocking?source=rss-e2bab4d88fd7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 17:46:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@harshtherocking/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Word2Vec — CBOW (implementation explained)]]></title>
            <link>https://medium.com/@harshtherocking/word2vec-cbow-implementation-explained-3b7ee6dbcd32?source=rss-e2bab4d88fd7------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b7ee6dbcd32</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[word2vec]]></category>
            <category><![CDATA[implementation]]></category>
            <category><![CDATA[natural-language-process]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Harsh Chauhan]]></dc:creator>
            <pubDate>Sun, 26 May 2024 11:13:36 GMT</pubDate>
            <atom:updated>2024-05-26T12:23:21.421Z</atom:updated>
            <content:encoded><![CDATA[<h3>Word2Vec — CBOW (implementation explained)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NF6QlLYit9EThSMJ.png" /><figcaption><a href="https://serokell.io/blog/word2vec">https://serokell.io/blog/word2vec</a></figcaption></figure><blockquote>You’re the average of the five people you spend most of your time with — Jim Rohn</blockquote><p>In 2013, <a href="https://arxiv.org/abs/1301.3781"><em>Word2Vec</em></a> was introduced, using which a numerical representation of a word can be achieved while capturing the context from its <em>surrounding</em> words in the corpus. The paper introduced two approaches: <em>Continuous bag of Words (CBOW)</em> and <em>Skip Gram</em>. However, we’re only discussing the CBOW approach for now; feel free to read the paper.</p><p>The CBOW algorithm accounts for the probability of a <em>target word</em> appearing given the presence of its neighbouring <em>context words</em> within a fixed distance in both the left and right directions. The total distance is called the<em> window size </em>(2C; C is the distance in one direction).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mANChvChG5hJVZ-qLtsC6A.png" /><figcaption>Context words are neighbors to the target word within the window</figcaption></figure><p>Before heading to implementation, some other important terms need to be defined.</p><p><em>Vocabulary size i</em>s the total number of unique words appearing in the training corpus. <em>Embedding size</em> is the number of dimensions for the required resultant word embedding. <em>Input Weight</em> is the matrix containing embeddings for every word in the vocabulary in dimensions of embedding size; it has a shape of <em>(VocabSize, EmbedSize</em>). <em>Output weight</em> is the matrix having shape <em>(EmbedSize, VocabSize)</em> for the last linear transformation. The purpose of this layer will be discussed shortly.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b49ce17b5d51ce3c4b9e67d9adf36301/href">https://medium.com/media/b49ce17b5d51ce3c4b9e67d9adf36301/href</a></iframe><p>For the code above, I’ve taken 2C to be just C = window size.</p><p>The idea is to guess the one-hot encoding of the target word while having the context words as the inputs. In the preprocessing step, every word in the corpus is converted into <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html"><em>one-hot encoding</em></a><em>,</em> i.e., every word is represented with a vector of size <em>VocabSize.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EmB79pKMwCP0tLxf0ST-Sg.jpeg" /><figcaption>one hot encoding for each vocabulary word</figcaption></figure><p>Then the window size is defined, and the target word is picked from the corpus. The words within a distance equal to half of the window size, i.e., <em>C,</em> are taken as context words.</p><p>Each context word’s one-hot encoding is then linearly transformed using <em>input weight, </em>resulting in new hidden embeddings for each word of the shape <em>(EmbedSize)</em>. For the hidden layer representation, we take the <em>average of all hidden embeddings</em> with a shape of <em>(EmbedSize)</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*70mqSP3KIW-4n0U7axunTQ.png" /><figcaption>x_i belongs to every word in the context window</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*vi8Ya__kxYW3taY0qBzf8g.png" /><figcaption><strong>h</strong> is the hidden layer output.</figcaption></figure><p>Note: This resultant<strong> hidden layer doesn’t represent word embedding</strong>.</p><p>These hidden layer outputs are then linearly transformed using the output weight, which results in a matrix having shape <em>(VocabSize).</em> The <a href="https://en.wikipedia.org/wiki/Softmax_function"><em>softmax</em> </a>function is performed over the resulting matrix so that all its elements are between 0 and 1, according to the probability distribution.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3bf8bfbceee77c450696ad4767a68c46/href">https://medium.com/media/3bf8bfbceee77c450696ad4767a68c46/href</a></iframe><p>The <em>output </em>and <em>target word </em>vectors are used to calculate the <a href="https://www.baeldung.com/cs/nlps-word2vec-negative-sampling">negative sampling log loss</a>. The loss helps with <a href="https://en.wikipedia.org/wiki/Backpropagation"><em>back-propagating</em></a><em> </em>and updating the weights.</p><p>The <em>output </em>and <em>target word </em>vectors are used to calculate the <a href="https://www.baeldung.com/cs/nlps-word2vec-negative-sampling">negative sampling log loss</a>. The loss helps with <a href="https://en.wikipedia.org/wiki/Backpropagation"><em>back-propagating</em></a><em> </em>and updating the weights.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/287ee0307aad51b89963d793325216a0/href">https://medium.com/media/287ee0307aad51b89963d793325216a0/href</a></iframe><p>After sliding the window through all the words and back propagating, the training ends. Now, the input weight contains word embedding of each word in the vocabulary. To access the word embedding of the <em>i-th </em>word in the vocabulary, we need to access the <em>i-th</em> index of the <em>input weight.</em></p><p>This is how the continuous bag of words algorithm for Word2Vec is implemented. Also read about <a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html"><strong>torch.nn.Embedding</strong></a>. Don’t miss out on the mathematics behind this algorithm—<a href="https://en.wikipedia.org/wiki/Word2vec#:~:text=Word2vec%20is%20a%20technique%20in,text%20in%20a%20large%20corpus.">CBOW Wikipedia</a>.</p><p>Thanks for reading. Don’t forget to give some claps if you find the article helpful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b7ee6dbcd32" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>