<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Louis Kit Lung Law on Medium]]></title>
        <description><![CDATA[Stories by Louis Kit Lung Law on Medium]]></description>
        <link>https://medium.com/@louiskitlunglaw?source=rss-bc0106497105------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*PGP0QNEzIh3zTdU5.jpg</url>
            <title>Stories by Louis Kit Lung Law on Medium</title>
            <link>https://medium.com/@louiskitlunglaw?source=rss-bc0106497105------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:22:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@louiskitlunglaw/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[RankNet, LambdaRank TensorFlow Implementation — part IV]]></title>
            <link>https://medium.com/swlh/ranknet-lambdarank-tensorflow-implementation-part-iv-3ac4e77ee2c5?source=rss-bc0106497105------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ac4e77ee2c5</guid>
            <category><![CDATA[learning-to-rank]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[lambda-rank]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Louis Kit Lung Law]]></dc:creator>
            <pubDate>Sun, 28 Feb 2021 14:13:02 GMT</pubDate>
            <atom:updated>2021-02-28T14:13:02.797Z</atom:updated>
            <content:encoded><![CDATA[<h3>RankNet, LambdaRank TensorFlow Implementation — part IV</h3><p>In this blog, I will talk about LambdaRank and its TensorFlow implementation. Note that LambdaRank is published in the paper <a href="https://books.google.com/books?hl=en&amp;lr=&amp;id=Tbn1l9P1220C&amp;oi=fnd&amp;pg=PA193&amp;dq=learning+to+rank+with+nonsmooth+cost+functions&amp;ots=V5l8Djim0Z&amp;sig=UctztxFmuBJAPGIMxHwvXo92Gn4"><strong>Learning to Rank with Nonsmooth Cost Functions</strong></a>.</p><h3>RankNet Revisited</h3><p>Since LambdaRank is improved upon RankNet, let first revisit the cost function and gradients of RankNet.</p><h4>RankNet’s Cost Function</h4><p>Note that in Learning To Rank, there are a few typical ranking losses</p><ol><li>Mean Reciprocal Rank (MRR)</li><li>Mean Average Precision (MAP)</li><li>Normalised Discounted Cumulative Gain (NDCG)</li></ol><p>But usually <strong>these ranking loss are non differentiable</strong>, hence it is difficult to apply gradient descent directly. To work around this problem, RankNet used a cross entropy cost function as shown in equation I below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o6dYVQ1MorIJ1UhthbkpxQ.png" /><figcaption>Equation 0. Definition of Pij</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BqMv8C7EWO00_ign.png" /><figcaption>Equation 1. RankNet’s cost function</figcaption></figure><h4>RankNet’s Gradients</h4><p>Recall that in previous <a href="https://louiskitlunglaw.medium.com/ranknet-factorised-ranknet-lambdarank-explained-implementation-via-tensorflow-2-part-iii-b916f849a4bd">blog</a>, we have following equations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V-pUy0C6ExpBw8tP.png" /><figcaption>Equation 2. Derivative of cost with respect to RankNet’s weights</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BT6MIRL-fCvQQP_2.png" /><figcaption>Equation 3. Derivative of cost with respect to oij=oi-oj</figcaption></figure><p>RankNet is trained with gradient descent, hence gradients with respect to model’s weights (dCij/dWk) need to be computed throughout the training process.</p><blockquote>And from equations (2) and (3), it is obvious that we can actually compute the gradients directly without computing the cost at all.</blockquote><h3>LambdaRank</h3><p>From above, we have two main observations</p><ol><li><strong>RankNet does not consider any ranking loss in the optimisation process</strong></li><li><strong>Gradients could be computed without computing the cross entropy loss</strong></li></ol><blockquote>To improve upon RankNet, LambdaRank <strong>defined the gradient directly </strong>(without defining its corresponding loss function) by taking ranking loss into consideration: scale the RankNet’s gradient by the size of change in NDCG if document i is swapped with j.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zXXKvqRCURhniYLVkCkBSw.png" /><figcaption>Equation 5. LambdaRank’s gradient</figcaption></figure><p>The idea is quite straight forward, if the change in NDCG by swapping <em>i</em> and <em>j</em> is large, we expect the gradient to be large as well to make this happens in the hope of maximising NDCG. And indeed, the paper shown that this works empirically.</p><h4>TensorFlow Implementation</h4><p>In order to implement LambdaRank based on RankNet, we need to calculate the change in NDCG if <em>i</em> and <em>j</em> is swapped and apply it in gradient descent.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c5f25db65d421bd25ea90f51ad08263/href">https://medium.com/media/0c5f25db65d421bd25ea90f51ad08263/href</a></iframe><p>Note that in line 10–11 in the above code snippet, delta NDCG is calculated and applied.</p><h4><strong>Performance</strong></h4><p>To show the difference in performance between LambdaRank with RankNet, I performed experiments on artificial dataset which has 200 queries, 50 documents per query on average, 50% are used as training set and 50% as validation set. NDCG@k on validation set is measured and shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZnFZUlZch9brKN_bDksD3g.png" /><figcaption>Validation NDCG@k of RankNet vs LambdaRank</figcaption></figure><p>From the graph, we could see that LambdaRank gives better NDCG than RankNet. You may find the <a href="https://github.com/louislung/ranknet_lambdarank_tensorflow/blob/main/LambdaRank.ipynb">jupyter notebook here</a> which generate the graph above.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ac4e77ee2c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/ranknet-lambdarank-tensorflow-implementation-part-iv-3ac4e77ee2c5">RankNet, LambdaRank TensorFlow Implementation — part IV</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RankNet, LambdaRank TensorFlow Implementation — part III]]></title>
            <link>https://medium.com/swlh/ranknet-factorised-ranknet-lambdarank-explained-implementation-via-tensorflow-2-part-iii-b916f849a4bd?source=rss-bc0106497105------2</link>
            <guid isPermaLink="false">https://medium.com/p/b916f849a4bd</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[learning-to-rank]]></category>
            <category><![CDATA[keras]]></category>
            <dc:creator><![CDATA[Louis Kit Lung Law]]></dc:creator>
            <pubDate>Mon, 08 Feb 2021 08:04:05 GMT</pubDate>
            <atom:updated>2021-02-28T15:32:45.073Z</atom:updated>
            <content:encoded><![CDATA[<h3>RankNet, LambdaRank TensorFlow Implementation — part III</h3><p>In this blog, I will talk about the how to speed up training of RankNet and I will refer to this speed up version as Factorised RankNet. Note that this is published in the paper <a href="https://books.google.com/books?hl=en&amp;lr=&amp;id=Tbn1l9P1220C&amp;oi=fnd&amp;pg=PA193&amp;dq=learning+to+rank+with+nonsmooth+cost+functions&amp;ots=V5l8Djim0Z&amp;sig=UctztxFmuBJAPGIMxHwvXo92Gn4"><strong>Learning to Rank with Nonsmooth Cost Functions</strong></a><strong>.</strong></p><h3><strong>RankNet Training Process Examined</strong></h3><p>Typically, to train a ML model, we need to do two things:</p><ol><li>compute cost function</li><li>compute gradient (derivative of cost with respect to model’s weights)</li></ol><p>And recall that in part I, we have the following equations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fNOoIr_CRO_3Rdrk.png" /><figcaption>Equation 1. RankNet’s cost function</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SMIBPDpPfKGNQd9FdTxpkQ.png" /><figcaption>Equation 2. Derivative of cost with respect to RankNet’s weights</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Ad1beSBp09cx0HhcZNqKA.png" /><figcaption>Equation 3. Derivative of cost with respect to oij=oi-oj</figcaption></figure><p>Now let’s look at an example. Assume there are 4 documents for a query <em>d1, d2, d3, d4</em>, thus there are four pairs of documents <em>d1&amp;d2, d1&amp;d3, d1&amp;d4, d2&amp;d3, d2&amp;d4, d4&amp;d3</em> and the training process would be like this (note that stochastic gradient descent is used in RankNet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PgFE9a6u3agesY-Mh5qtZw.png" /><figcaption>Fig. 1, Stochastic gradient descent</figcaption></figure><p>As we can see, for a query with <em>n</em> (e.g. 4 )documents, <em>n(n-1) (e.g. 12) </em>back propagation are done to compute <strong><em>doi/dWk</em></strong>. Thus the training time almost scale quadratically with the mean number of documents per query.</p><blockquote>But some terms are “repeated” multiple times (e.g. the one highlighted in blue above: <strong><em>do1/dWk</em></strong>), so could we combine them together to reduce the number of back propagations needed?</blockquote><h3><strong>Factorised RankNet</strong></h3><p>In order to speed up the training, we could <strong>switch from stochastic gradient descent to mini-batch</strong> where each batch corresponds to one query. So now the training process looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8xs1_8PjmUjzvkSXtCfaCw.png" /><figcaption>Fig. 2, Mini-Batch gradient descent</figcaption></figure><p>But using mini-batch alone wont give too much speed up as we still need to compute <em>n(n-1)</em> back propagations for each query.</p><p>Let see how we could factorise the sum highlighted in Fig. 2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QHScCnn1jUqBhXQKFhLAoQ.png" /><figcaption>Fig. 3, Factoring mini-batch gradient descent</figcaption></figure><blockquote>While we still need to compute <em>n(n-1) </em>times<em> </em><strong><em>dCij/doij</em></strong><em>, </em>we now only need to compute <em>n</em> times <strong><em>doi/dWk</em></strong><em> </em>!<em><br></em>And since the computation of <strong>dCij/doij</strong> is very cheap (refer to equation 3), thus the original O(n²) RankNet algorithm is effectively reduced to O(n).</blockquote><p>To make this more general:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_aL0WJR9GiZ5P9_dLDjeGg.png" /><figcaption>Fig. 4, Generalised the result of Fig. 3</figcaption></figure><p>And the training process of Factorised RankNet is like below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wHqFuy6rsa3VpdMUS6Tc6Q.png" /><figcaption>Fig. 5, Factorised RankNet’s training</figcaption></figure><h3>Implementation</h3><h4><strong>Model Architecture</strong></h4><p>First let define the model, note that instead of taking a pair of documents as input as before, only one document at a time.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/259cef39fe93c2cb81a4e4754114f3d7/href">https://medium.com/media/259cef39fe93c2cb81a4e4754114f3d7/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZIeOd8v_tZjWPa2rk45XtA.png" /><figcaption>Fig. 6 Factorised RankNet’s model architecture</figcaption></figure><h4><strong>Epoch Looping</strong></h4><p>In the following function, for each epoch, each query is treated as one batch, and documents under that query are used to train the model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af754569ebb91b3f21d9cd21f12cb85a/href">https://medium.com/media/af754569ebb91b3f21d9cd21f12cb85a/href</a></iframe><h4>Gradient Calculation</h4><p>The following function will calculate the gradient and apply gradient descent to train the model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ddd581301139170fd50e437aaddbb72d/href">https://medium.com/media/ddd581301139170fd50e437aaddbb72d/href</a></iframe><h3>RankNet vs Factorised RankNet</h3><p>Now everything is ready, let’s compare the performance of RankNet vs Factorised RankNet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Drm_93Aj9rmZwa-4dtTiFA.png" /><figcaption>Fig. 7, Loss Plot on Validation Set</figcaption></figure><p>As we can see from the graph above, Factorised RankNet used less time to achieve the same loss as compared to RankNet.</p><p><a href="https://github.com/louislung/ranknet_lambdarank_tensorflow/blob/main/FactorisedRankNet.ipynb">Here is the link to the jupyter notebook which used to generate the graph above.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b916f849a4bd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/ranknet-factorised-ranknet-lambdarank-explained-implementation-via-tensorflow-2-part-iii-b916f849a4bd">RankNet, LambdaRank TensorFlow Implementation — part III</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RankNet, LambdaRank TensorFlow Implementation — part II]]></title>
            <link>https://medium.com/swlh/ranknet-factorised-ranknet-lambdarank-explained-implementation-via-tensorflow-2-part-ii-f28bd5a8a6d5?source=rss-bc0106497105------2</link>
            <guid isPermaLink="false">https://medium.com/p/f28bd5a8a6d5</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[information-retrieval]]></category>
            <category><![CDATA[learning-to-rank]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Louis Kit Lung Law]]></dc:creator>
            <pubDate>Wed, 03 Feb 2021 17:57:07 GMT</pubDate>
            <atom:updated>2021-02-28T15:32:37.630Z</atom:updated>
            <content:encoded><![CDATA[<h3>RankNet, LambdaRank TensorFlow Implementation — part II</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TihxX-VUvl6NfbZr.jpg" /></figure><p>In part I, I have go through <a href="https://dl.acm.org/doi/abs/10.1145/1102351.1102363"><strong>RankNet</strong></a><strong> </strong>which is published by Microsoft in 2005. 2 years after, Microsoft published another paper <a href="https://books.google.com/books?hl=en&amp;lr=&amp;id=Tbn1l9P1220C&amp;oi=fnd&amp;pg=PA193&amp;dq=learning+to+rank+with+nonsmooth+cost+functions&amp;ots=V5l8Djim0Z&amp;sig=UctztxFmuBJAPGIMxHwvXo92Gn4"><strong>Learning to Rank with Nonsmooth Cost Functions</strong></a> which introduced a speedup version of RankNet (which I called <strong>“Factorised RankNet”</strong>) and <strong>LambdaRank</strong>.</p><blockquote>However, before I jump to talk about Factorised RankNet and LambdaRank, I’d like to show you how to implement RankNet using custom training loop in Tensorflow 2.</blockquote><p>This is important because Factorised RankNet and LambdaRank cannot be implemented just by Keras API, it is necessary to use low level API like TensorFlow and PyTorch as we will see later.</p><h3><strong>Custom Training Loop</strong></h3><p>Remember in part I, the model training is actually done by 2 lines of code</p><pre>ranknet.compile(...)<br>ranknet.fit(...)</pre><p>In order to write our own training loop, we need to understand what is done behind these 2 lines of code, this is outlined in the pseudo code below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qehGcu5aOwJ92ZFwfv4WvQ.png" /><figcaption>training pseudo code</figcaption></figure><p>Now we know what need to be done. Let’s break down the whole process into a few functions:</p><ul><li><em>apply_gradient</em> function which do line 3 to 6</li><li><em>train_data_for_one_epoch</em> function which loop through batches, line 2–7</li><li><em>compute_val_loss</em> function which do line 9</li><li>an outer loop on epochs which put everything together</li></ul><p>Let’s see how to implement these.</p><h4><strong>apply_gradient</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c4470f47e2c821350276ab3d275aea56/href">https://medium.com/media/c4470f47e2c821350276ab3d275aea56/href</a></iframe><ul><li>line 4 computes Pij</li><li>line 5 computes the loss</li><li>line 7 gets the gradients of all trainable weights</li><li>line 8 apply back propagation to update the weights</li></ul><h4><em>train_data_for_one_epoch</em></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a20a8bc3fe7414afba9dbe2d40e0f66c/href">https://medium.com/media/a20a8bc3fe7414afba9dbe2d40e0f66c/href</a></iframe><ul><li>line 5 converts the function <em>apply_gradient</em> to graph mode</li><li>line 7 print a progress bar</li><li>line 8 loops through the training set batch by batch</li><li>line 10 stores the loss of current batch</li></ul><h4>compute_val_loss</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e59b021c1f48f4bc2691449441dc38f/href">https://medium.com/media/0e59b021c1f48f4bc2691449441dc38f/href</a></iframe><p>This function simply loop through validation set and compute the loss.</p><h4>Putting all these together</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1fdaa6ab634fd7f40e04ff61b528fcd/href">https://medium.com/media/a1fdaa6ab634fd7f40e04ff61b528fcd/href</a></iframe><ul><li>line 3 &amp; 4 define the data pipeline</li><li>line 8 define the loss function and we use binary crossentropy</li><li>line 9 define the optimizer</li><li>line 19 start the training of RankNet</li></ul><p>From the results below, we can see the training result is very similar to the one in part I.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aOYvcA-EgD24HyTe3DIayA.png" /><figcaption>first 5 epochs of training</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QgoS7Xxf4iRfPFE-v0tFIA.png" /><figcaption>training loss and validation loss plot</figcaption></figure><h3>Benefit of Custom Training Loop</h3><p>Now you may be wondering what’s the advantage of using a custom training loop. Let me show you how we could be more memory efficient after using custom training loop.</p><p>Recall that in Part I when we generate the data, we first generate <strong><em>doc_features</em></strong><em> </em>and then transform it into pairs, <strong><em>xi</em></strong> and <strong><em>xj </em></strong>before feeding into RankNet. This means that for one document, it will be repeated several times in the memory.</p><p>For example, if we have two queries, q1 and q2:</p><ul><li>q1 has three documents, d1, d2, d3</li><li>q2 has four documents d4, d5, d6, d7</li></ul><p>Then we will have following pairs</p><ul><li>q1’s pairs: d1 &amp; d2 | d1 &amp; d3 | d2 &amp; d3</li><li>q2’s pairs: d4 &amp; d5 | d4 &amp; d6 | d4 &amp; d7 | d5 &amp; d6 | d5 &amp; d7 | d5 &amp; d7</li></ul><p>To be more memory efficient, we should only get the necessary document on the fly during training. This could be achieved by modifying two functions as below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b40f23f159929817b66d26ebf4cdc3b7/href">https://medium.com/media/b40f23f159929817b66d26ebf4cdc3b7/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dbc19b76d139170503ad9ffe25d5db6b/href">https://medium.com/media/dbc19b76d139170503ad9ffe25d5db6b/href</a></iframe><p>Now we get rid of the variables xi, xj, pij completely which means we are not repeating any document in the memory. If you use these functions to train the model, it will work exactly the same!</p><h3><strong>Conclusion</strong></h3><p>Now we implemented RankNet using a custom training loop in TensorFlow 2.</p><p>In part III, I will talk about how to speed up the training of RankNet and the implementation.</p><p>Stay Tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f28bd5a8a6d5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/ranknet-factorised-ranknet-lambdarank-explained-implementation-via-tensorflow-2-part-ii-f28bd5a8a6d5">RankNet, LambdaRank TensorFlow Implementation — part II</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RankNet, LambdaRank TensorFlow Implementation — part I]]></title>
            <link>https://medium.com/swlh/ranknet-factorised-ranknet-lambdarank-explained-implementation-via-tensorflow-2-0-part-i-1e71d8923132?source=rss-bc0106497105------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e71d8923132</guid>
            <category><![CDATA[learning-to-rank]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[information-retrieval]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Louis Kit Lung Law]]></dc:creator>
            <pubDate>Tue, 02 Feb 2021 05:43:44 GMT</pubDate>
            <atom:updated>2021-02-21T12:32:19.264Z</atom:updated>
            <content:encoded><![CDATA[<h3>RankNet, LambdaRank TensorFlow Implementation— part I</h3><p>I come across the field of Learning to Rank (LTR) and RankNet, when I was working on a recommendation project. However, it is a bit tricky to implement the model via TensorFlow and I cannot find any detail explanation on the web at all. Hence in this series of blog posts, I’ll go through the papers of both RankNet and LambdaRank in detail and implement the model in TF 2.0.</p><p>For this post, I will go through the followings</p><ul><li>the paper which first proposed RankNet (<a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2005/08/icml_ranking.pdf">Learning to Rank using Gradient Descent</a>)</li><li>the paper summarised RankNet, LambdaRank (<a href="https://www.microsoft.com/en-us/research/uploads/prod/2016/02/MSR-TR-2010-82.pdf">From RankNet to LambdaRank to LambdaMart</a>)</li><li>implementation of RankNet using Keras’s Functional API</li></ul><h3><strong>Learning to Rank</strong></h3><p>In a typical learning to rank problem setup, there is</p><ul><li>a list of queries <em>q1, q2, q3, ...</em></li><li>for each query, there are some documents <em>d1, d2, d3, …</em></li><li>for each document, there is a score <em>s1, s2, s3 … </em>which indicate the relevance between this document and the corresponding query (e.g. score 0 means completely irrelevant while 5 means highly relevant)</li></ul><p>For example, in the case of a search engine,</p><ul><li>queries are search texts like “TensorFlow 2.0 doc”, “Keras api doc”, …</li><li>documents are the URLs returned by the search engine</li><li>score is the clicks received by the URL (higher clicks = more relevant)</li></ul><p>In the example above, one could construct features as the keywords extracted from the query and the document and label as the relevance score.<br>Hence the most straight forward way to solve this problem using machine learning is to construct a neural network to predict a score given the keywords.</p><h3>RankNet</h3><h4>Model Target</h4><p>Instead of modelling the score of each document one by one, RankNet proposed to model the <em>target probabilities </em>between any two documents (<em>di &amp; dj</em>) of the same query.</p><p>And the <em>target probabilities Pij </em>of<em> di </em>and<em> dj </em>is defined as</p><ul><li>1 if <em>si</em> &gt; <em>sj</em></li><li>0.5 if <em>si</em>=<em>sj</em></li><li>0 if <em>si</em> &lt; <em>sj</em></li></ul><p>where <em>si</em> and <em>sj</em> is the score of <em>di</em> and <em>dj</em> respectively</p><p>Let say for a particular query, there are 3 documents <em>d1, d2, d3</em> with scores 0, 5, 3 respectively, then there will be 3 valid pairs of documents:</p><ul><li>d1 &amp; d2 with P12 = 0</li><li>d1 &amp; d3 with P13 = 0</li><li>d2 &amp; d3 with P23 = 1</li></ul><p>So now each pair of documents serve as one training record to RankNet.</p><h4>Model Architecture</h4><p>In the RankNet paper, the author used a neural network formulation.<br>Let’s denote the neural network as function <em>f, </em>the output of neural network for document i as <em>oi, </em>the features of document i as <em>xi. </em>Hence we have <em>oi = f(xi) </em>and<em> oj = f(xj). </em>Also we define <em>oij = oi - oj = f(xi) - f(xj) = -(oj - oi) = -oji</em>.</p><p>Note that <em>oi</em> (and <em>oj</em>) could be any real number, but as mentioned above, RankNet is only modelling the probabilities <em>Pij </em>which is in the range of [0,1].</p><p>In order to model the probabilities, logistic function is applied on <em>oij </em>as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*9W8dx7VVdWwPsAH05zFlqA.png" /><figcaption>logistic function applied on oij</figcaption></figure><p>And cross entropy cost function is used, so for a pair of documents <em>di</em> and <em>dj,</em> the corresponding cost<em> Cij </em>is computed as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kFGsMn1zMejaXiKlaO_jVQ.png" /></figure><p>And the gradient is computed as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*AWmMK-yhU_CCArP4o3dA2A.png" /></figure><p>At this point, you may already notice RankNet is a bit different from a typical feedforward neural network.</p><blockquote>While a typical neural network follows these steps to update its weights: <br>read input features -&gt; compute output -&gt; compute cost -&gt; compute gradient -&gt; back propagation</blockquote><blockquote>RankNet update its weights as follows:<br>read input xi -&gt; compute oi -&gt; compute gradients doi/dWk -&gt; <br>read input xj -&gt; compute oj -&gt; compute gradients doj/dWk -&gt; <br>compute Pij -&gt; compute gradients using equation (2) &amp; (3) -&gt; back propagation</blockquote><p>So in RankNet, <em>xi</em> &amp; <em>xj</em> serve as one training record, RankNet will pass <em>xi</em> &amp; <em>xj</em> through the same the weights (Wk) of the network to get <em>oi</em> &amp; <em>oj</em> before computing the gradient and update its weights.</p><h3>Implementation using Keras</h3><p>As described above, RankNet will take two inputs, xi &amp; xj, pass them through the same hidden layers to compute oi &amp; oj, apply sigmoid on oi-oj to get the final probability for a particular pair of documents, di &amp; dj.</p><p>Thus, the model should look like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lmuj0AObeoFDdRjhidSmrw.png" /></figure><p>This could be implemented using keras’s functional API as follows</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a0c78bdaecedee5bac11359caaa9980/href">https://medium.com/media/6a0c78bdaecedee5bac11359caaa9980/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rxFUUH_IVKtqeO3SuGDhlw.png" /></figure><p>Now let’s simulate some data and train the model</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc715189325103db71ec5c584317a578/href">https://medium.com/media/cc715189325103db71ec5c584317a578/href</a></iframe><p>Now we could start training RankNet() just by two lines of code</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32549ed022616a102950d49076526916/href">https://medium.com/media/32549ed022616a102950d49076526916/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ul9k5VWZPzCXykVuf-Cg6g.png" /><figcaption>first 5 epochs</figcaption></figure><p>Let’s plot the loss metric as well</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f7abd6c9b4ed2fc4d614ed337ae2f0f/href">https://medium.com/media/5f7abd6c9b4ed2fc4d614ed337ae2f0f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rmLl20ihvO_2kjPTMdS1EQ.png" /><figcaption>cross entropy loss</figcaption></figure><p>As we can see, the loss of both training and test set decreased overtime.</p><h3>Conclusion</h3><p>In this post, I have gone through</p><ul><li>how RankNet used a probabilistic approach to solve learn to rank</li><li>how to use gradient descent to train the model</li><li>implementation of RankNet using Keras’s functional API</li></ul><p>In the future blog post, I will talk about</p><ul><li>how to implement a custom training loop (instead of using <em>model.compile</em> and <em>model.fit</em>)</li><li>how to speed up training of RankNet</li><li>LambdaRank algorithm</li></ul><p>Stay Tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e71d8923132" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/ranknet-factorised-ranknet-lambdarank-explained-implementation-via-tensorflow-2-0-part-i-1e71d8923132">RankNet, LambdaRank TensorFlow Implementation — part I</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn document embeddings using Facebook StarSpace]]></title>
            <link>https://medium.com/deep-learning-hk/documents-embeddings-using-facebook-s-starspace-67b8d1feba32?source=rss-bc0106497105------2</link>
            <guid isPermaLink="false">https://medium.com/p/67b8d1feba32</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[starspace]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[word-embeddings]]></category>
            <dc:creator><![CDATA[Louis Kit Lung Law]]></dc:creator>
            <pubDate>Sun, 23 Jun 2019 15:17:05 GMT</pubDate>
            <atom:updated>2019-11-03T15:09:07.274Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learn Document Embedding using Facebook StarSpace</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s7ntCeEnMRPFJmCPK8K1bA.jpeg" /><figcaption>This seems to match with the title “Starspace”</figcaption></figure><p>Recently I have come across a tool called <a href="https://ai.facebook.com/tools/starspace/">StarSpace</a> which is created by Facebook. StarSpace catches my eyes because it learns to represent different objects into the same embedding space. In order words, it learns embeddings and used these embeddings to complete a lot of different tasks, below are some examples</p><ul><li>Word / Tag embeddings (map from a short text to relevant hashtags)</li><li>Document recommendation (embed and recommend documents for users based on their historical likes/click data)</li><li>Sentence Embeddings (given the embedding of one sentence, one can find semantically similar/relevant sentences)</li></ul><p>StarSpace could archive many more tasks, as long as you prepare the data in the right format. <strong>And in this blog post, I am going to use StarSpace to learn document embeddings to predict the category of the document, </strong><a href="https://github.com/louislung/DAE_RNN_News_Recommendation/blob/master/starspace/prepare_starspace_formatted_data.ipynb"><strong>all the code is available here</strong></a><strong>.</strong></p><p><a href="https://medium.com/@LouisKitLungLaw/compute-document-similarity-using-autoencoder-with-triplet-loss-eb7eb132eb38">Compute Document Similarity Using Autoencoder With Triplet Loss</a></p><p>In my previous post, I have showed how to compute document embeddings using autoencoder with triplet loss and the learned embeddings is able to preserve categorical similarity between documents. And now I am going to use StarSpace to do that same task one more time.</p><h3>Tasks</h3><ol><li>Using news content to predict its category</li><li>Examine the learned embeddings</li></ol><h3><strong>Prepare training data</strong></h3><p>Since StarSpace does not do tokenisation, I have to first tokenise the news.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4daec644ed7581f5c058bf56c39c057f/href">https://medium.com/media/4daec644ed7581f5c058bf56c39c057f/href</a></iframe><p>Note that each line corresponds to a news and its label, all separated by space.</p><h3><strong>Training</strong></h3><p>Following command is used to train a model using StarSpace and then use the model to get back the embeddings.</p><pre><em>&gt; starspace train -trainFile uci_train_starspace_formatted.txt -model uci_starspace -trainMode 0 -validationFile uci_validate_starspace_formatted.txt -dim 50 -epoch 50 -negSearchLimit 1 -thread 20 -lr 0.001</em></pre><pre><em>&gt; embed_doc uci_starspace uci_train_starspace_formatted.txt &gt; uci_train_starspace_embed.txt</em></pre><h3><strong>Results</strong></h3><p>Same as the last blog post, I will evaluate the performance of the embeddings by checking the AUROC of cosine similarity of similar articles (article of the same category or same story).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*dAq8zTQXxKFXkkWefegaBw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*57q6cvS4DUg60i73sHtQuw.png" /></figure><p>The graphs above compare the embeddings trained using StarSpace with simple Tfidf, we see that StarSpace embeddings is better in preserving categorical similarity.</p><pre>AUROC results using Testing Data<br>+---------------------+--------------+-----------+<br>|                     | Category Sim | Story Sim |<br>+---------------------+--------------+-----------+<br>| Triplet Autoencoder |         0.78 |      <strong>0.92</strong> |<br>| StarSpace           |         <strong>0.93</strong> |      0.90 |<br>+---------------------+--------------+-----------+</pre><p>The above table compare the StarSpace embeddings with the triplet embeddings (trained in <a href="https://medium.com/@LouisKitLungLaw/compute-document-similarity-using-autoencoder-with-triplet-loss-eb7eb132eb38">previous blog post</a>), we see that StarSpace does a significantly better job in preserving categorical similarity without too much loss in story similarity.</p><h3><strong>Conclusion</strong></h3><p>StarSpace seems to be a powerful tool to train embeddings of documents, it is very fast (the training took just a few minutes on my laptop) and easy to use.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67b8d1feba32" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deep-learning-hk/documents-embeddings-using-facebook-s-starspace-67b8d1feba32">Learn document embeddings using Facebook StarSpace</a> was originally published in <a href="https://medium.com/deep-learning-hk">Deep Learning HK</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Compute Policy Gradient for UC Berkeley Deep RL Bootcamp Lab 4 Exercise 3.6]]></title>
            <link>https://louiskitlunglaw.medium.com/compute-policy-gradient-for-uc-berkeley-deep-rl-bootcamp-lab-4-exercise-3-6-a61425b13acb?source=rss-bc0106497105------2</link>
            <guid isPermaLink="false">https://medium.com/p/a61425b13acb</guid>
            <category><![CDATA[policy-gradient]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Louis Kit Lung Law]]></dc:creator>
            <pubDate>Thu, 28 Feb 2019 10:11:30 GMT</pubDate>
            <atom:updated>2019-02-28T10:11:30.654Z</atom:updated>
            <content:encoded><![CDATA[<p>UC Berkeley had organised a great <a href="https://sites.google.com/view/deep-rl-bootcamp/home?authuser=0">Bootcamp on reinforcement learning</a> back in 2017. And the exercise 3.6 of lab 4 asked candidates to compute gradient of a softmax policy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F6ILWxAF-uOadtUeYZW_Pw.png" /><figcaption>Lab 4 Exercise 3.6</figcaption></figure><p>And I found that the formulation of the policy in lab 4 is a little bit different from the formulation given in lecture (e.g. <a href="https://www.youtube.com/watch?v=KHZVXao4qXs">David Sliver’s course</a>, Chapter 13 of <a href="http://www.incompleteideas.net/book/the-book-2nd.html">Sutton &amp; Barto Book</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*LqGV7zTUdXr48YkFXjLc1w.png" /><figcaption>Typical formulation of softmax policy given in lecture</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*6_aKMlH6Lwm5pMwrQRtOQQ.png" /><figcaption>Formulation of softmax policy from the code of lab4</figcaption></figure><p><a href="https://math.stackexchange.com/questions/2013050/log-of-softmax-function-derivative">For the formulation given in lecture, you may find the steps to compute the policy gradient here</a>.</p><h3><strong>Compute Policy Gradient</strong></h3><p>Now I am going to give the steps to compute the policy gradient of exercise 3.6.</p><p><strong>Notations</strong></p><p>Let begin by defining some useful notations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BZdrQKsHY86R5-mAF8i5Tg.png" /><figcaption>Notations</figcaption></figure><p><strong>Formulation</strong></p><p>This formulation of policy is based on the code of lab4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ft8O_tjPbNsX4ZwmtIhSng.png" /><figcaption>Formulation</figcaption></figure><p><strong>Steps</strong></p><p>For the <strong><em>left</em></strong> term of the policy gradient from the formulation above</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y-DlqUDDcw0yfK7k14EkUw.png" /><figcaption>Left term</figcaption></figure><p>For the <strong><em>right</em></strong> term</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xnG2iN1pb7o6t3UJagMVmg.png" /><figcaption>Right term</figcaption></figure><p>Combining both term give us the policy gradient</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8VEEvl7AvK1W1ZYIfztyhg.png" /><figcaption>Policy gradient</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a61425b13acb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Compute Document Similarity Using Autoencoder With Triplet Loss]]></title>
            <link>https://medium.com/deep-learning-hk/compute-document-similarity-using-autoencoder-with-triplet-loss-eb7eb132eb38?source=rss-bc0106497105------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb7eb132eb38</guid>
            <category><![CDATA[word-embeddings]]></category>
            <category><![CDATA[triplets]]></category>
            <category><![CDATA[autoencoder]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Louis Kit Lung Law]]></dc:creator>
            <pubDate>Tue, 12 Feb 2019 12:01:00 GMT</pubDate>
            <atom:updated>2019-11-03T15:08:45.089Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*MQaT1Jw08XwviqUsbkiGNQ.png" /></figure><p>In this blog post, we will examine how to<strong> compute document similarity for news articles</strong> using <strong>Denoising Autoencoder</strong> (DAE) combined with a <strong>Triplet Loss</strong> function. This approach is presented in <a href="http://gdac.uqam.ca/WWW2016-Proceedings/companion/p87.pdf">Article De-duplication Using Distributed Representations</a> published by Yahoo! JAPAN and my Tensoflow implementation could be find <a href="https://github.com/louislung/DAE_RNN_News_Recommendation">here</a>.</p><h3><strong>Introduction</strong></h3><p>The most common way of computing document similarity is to transform documents into TFIDF vectors and then apply any similarity measure e.g. cosine similarity to these vectors.</p><p>However this approach has 2 disadvantages / limitations:</p><ol><li>This requires transforming every document into a vector of large dimensions (~10k) and it may not be ideal to compute similarity between these high-dimensional vectors under a <strong>tight time constraints</strong> which is the case in news domain even when these vectors are sparse.</li><li>For news articles talking about similar or exact same event, this approach gives high similarity value.<br>But for articles of the same category talking about different event may end up with zero similarity value as there are no common keywords.<br>In other words, this approach does not preserve categorical similarity.</li></ol><p>And this is how Yahoo! JAPAN solves the limitations:</p><ol><li>By using DAE, we can compress the high-dimension TFIDF vector into low-dimension embeddings.</li><li>By using a triplet loss function, we can preserve categorical similarity between documents.</li></ol><p>Now let go through DAE and Triplet loss separately.</p><h3><strong>Denoising Autoencoder</strong></h3><p>Autoencoder is basically a neural network doing unsupervised learning that tries to reconstruct input vector (encoder) from compressed embeddings (decoder).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*pPzjE7tO0rizacSs0Bqk7w.png" /><figcaption>Photo credit: <a href="http://deeplearning.stanford.edu/wiki/images/f/f9/Autoencoder636.png">http://deeplearning.stanford.edu/wiki/images/f/f9/Autoencoder636.png</a></figcaption></figure><p>In the graph, <em>Layer</em> <em>L1</em> is the input vector, hidden layer <em>L2</em> is the embeddings, <em>Layer L3</em> is the reconstructed input. Usually <em>L1</em> and <em>L2</em> are called encoder, <em>L2</em> and <em>L3</em> are called decoder.</p><blockquote>The idea is that autoencoder will filter out noise and preserve only useful information during encoding so that decoder could use the compressed information to reconstruct the input entirely.</blockquote><p>And <strong>denoising</strong> autoencoder corrupts the input stochastically by setting some percentage of input elements to zero during training phase. This is motivated by making embeddings more robust to small irrelevant changes in input.</p><p>Below is the formulation of a traditional DAE:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b5i7Cs-A-VpPHbPDqODNlA.png" /><figcaption>Formulation of traditional denoising autoencoder</figcaption></figure><p>The loss function used is usually squared loss or cross-entropy.</p><h3>Triplet Loss</h3><p>Triplet loss is first introduced in <a href="https://arxiv.org/abs/1503.03832">FaceNet: A Unified Embedding for Face Recognition and Clustering</a> by Google which used to train faces’ embeddings.</p><p>With triplet loss, 3 inputs are required during training phase:</p><ol><li>Anchor</li><li>Positive (item which has the same label as anchor)</li><li>Negative (item which has different label to anchor)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*0ABl3GR0T4CUXnKOBBQFuQ.png" /></figure><p>During training, triplet loss will try to minimise distance (maximise similarity) between anchor and positive, while maximise distance (minimise similarity) between anchor and negative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*oMUhbHBWoUliuOnnz34rOA.png" /><figcaption>Euclidean distance between images’ embeddings</figcaption></figure><p>Above is an example from FaceNet, we can see images of same person have lower euclidean distance between their embeddings. Therefore, it shows that the learned embeddings preserve distance / similarity information of the original images.</p><blockquote>Yahoo! JAPAN applied the same idea to news articles’ embeddings to preserve categorical similarity, although the loss function is different compared to the one used by FaceNet.</blockquote><h3>Denoising Autoencoder + Triplet Loss</h3><p>The approach presented in <a href="http://gdac.uqam.ca/WWW2016-Proceedings/companion/p87.pdf">Article De-duplication Using Distributed Representations</a> used DAE with triplet loss to generate embeddings which preserve categorical similarity and here is how it is done.</p><p>Binary word count vector is used as input to the DAE, and for every news article (anchor), we need another news article that has the same category label (positive) and another news article that has different category label (negative).</p><p><strong>Notation<br></strong>The input vector of (anchor, positive, negative) denoted as (<em>x1, x2, x3</em>)<br>The embeddings / hidden layer of (anchor, positive, negative) denoted as (h<em>1, h2, h3</em>)</p><p><strong>Loss function</strong><br>To preserve the categorical similarity, we want <strong><em>h1．h2 &gt; h1．h3</em></strong><em> </em>since <em>x1</em> is more similar to <em>x2</em> than <em>x3. </em>Hence the loss function of DAE is updated as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PrX2TolbTyxhXms01g-7vQ.png" /><figcaption>Triplet Loss function of autoencoder</figcaption></figure><p>From (1), <em>h</em> satisfies the property <em>x</em>=0 =&gt; <em>h</em> = <em>f(b) — f(b)</em> = 0, which means an article has no available information is not similar to any other articles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*PALVCePi4MConToT66fWDg.png" /></figure><p>From (2), <em>ø </em>is the penalty function for article similarity and from the curve above, minimising <em>ø means </em>making <strong><em>h1．h2 &gt; h1．h3</em></strong></p><blockquote>Note that the overall objective is not only to minimise <em>ø </em>but<em> </em>to minimise <em>L </em>and <em>ø </em>where<em> L </em>is elementwise cross entropy loss function. Without <em>L, </em>the network may end up with zero weights so <em>h</em>=0 for all articles. Hence <em>L</em> is necessary to make sure the embeddings <em>h </em>preserve useful information about the articles while <em>ø </em>make sure similar articles have similar embeddings.</blockquote><p>After training, we could apply cosine similarity on the embeddings to check how similar two articles are.</p><h3>Triplet Mining</h3><p>As said before, to train the DAE with triplet loss, we need to prepare a set of triplet (anchor, positive, negative) as training data. So for every anchor, we need to map a positive and a negative. It could be a lot of works if this is done manually.</p><p>This could actually be done automatically with triplet mining.</p><p>With triplet mining, we only need to provide anchor and label of the anchor to DAE during training. And it will find out all possible triplets by constructing a 3-dimensional matrix. A much more detailed explanation and Tensorflow implementation could be find <a href="https://omoindrot.github.io/triplet-loss">here</a>.</p><blockquote>Note that triplet mining is not mentioned in the paper, but I found this saves a lot of effort to create the triplet manually.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*2fV_j4Vb1xgXd1i1kua78g.jpeg" /></figure><h3><strong>Evaluation</strong></h3><p>Let test the performance of this approach on some open dataset.</p><p><strong>Dataset<br></strong>A dataset which contains 10348 news articles with categories labels and story labels is used for evaluation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GG2JkQe3EMQP8WsRVCZACw.png" /></figure><p>The dataset is available at <a href="https://www.kaggle.com/louislung/uci-news-aggregator-dataset-with-content">https://www.kaggle.com/louislung/uci-news-aggregator-dataset-with-content</a></p><p><strong>Code<br></strong>I have implemented this autoencoder using Tensorflow and you may find the implementation <a href="https://github.com/louislung/DAE_RNN_News_Recommendation">here in my github</a>.</p><p><strong>Settings<br></strong>The embeddings is trained with following autoencoder network’s settings:</p><ul><li>Input vector = Binary vector corresponding to each word in the vocabulary (<a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html">Scikit learn’s Count Vectorize</a>r is used to prepare the input vector)</li><li>Dimension of Input layer = 10000, Embeddings layer =50, Output layer = 10000</li><li>Optimization Algorithm = Adam</li><li>Corruption Function = Every element of input vector is masked (set to zero) randomly with probability 0.3</li><li>Alpha = 10 (hyper-parameter of the loss function)</li><li>Loss function of autoencoder = Cross-Entropy</li><li>Batch size = 100</li><li>Number of epochs = 200</li></ul><p>The dataset is splitted into two: 5000 for training and 5348 for testing.</p><p><strong>Results<br></strong>We can evaluate the performance of the embeddings by checking the AUROC of cosine similarity of similar articles (article of the same category or same story).</p><p>We use both embeddings and Tfidf vector to calculate the cosine similarity and plot the ROC below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QpIwdNE__luz9XO3Hr_o7w.png" /></figure><p>The trained embeddings is able to preserve categorical similarity between articles (higher auroc achieved in both training and testing set).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jcjZ4XU7-NE8a5sj89NEpg.png" /></figure><p>While Tfidf is still good at identifying similar articles at story level as the embeddings loses some information due to compression.</p><p>We can see that embeddings trained with the approach is highly compressed (dimensions reduced from 10000 to 50) and is able to preserve categorical similarity.</p><h3><strong>Wrapping Up</strong></h3><p>We have seen how to train embeddings which preserve categorical similarity using a variant of denoising autoencoder by introducing triplet loss into the model.</p><p>This approach of finding similar articles may be preferred in news domain where there is tight time constraint (latest news article should be displayed to users within short time).</p><p>Moreover, a lot of tasks could be done after having these embeddings for every news articles:</p><ol><li>content based recommendation (<a href="https://www.kdd.org/kdd2017/papers/view/embedding-based-news-recommendation-for-millions-of-users">Embedding-based News Recommendation for Millions of Users</a>)</li><li>Topic model</li><li>Articles grouping</li></ol><p>Thanks you for reading, if you have any feedback please let me know in the comment section!</p><h3><strong>References</strong></h3><p><a href="http://www.cs.toronto.edu/~larocheh/publications/icml-2008-denoising-autoencoders.pdf">Extracting and composing robust features with denoising autoencoders</a>; P. Vincent et al., 2008</p><p><a href="http://gdac.uqam.ca/WWW2016-Proceedings/companion/p87.pdf">Article De-duplication Using Distributed Representations</a>; Akira Tajima et al., 2016</p><p><a href="https://arxiv.org/pdf/1503.03832.pdf">FaceNet: A Unified Embedding for Face Recognition and Clustering</a>; Florian Schroff et al., 2015</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb7eb132eb38" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deep-learning-hk/compute-document-similarity-using-autoencoder-with-triplet-loss-eb7eb132eb38">Compute Document Similarity Using Autoencoder With Triplet Loss</a> was originally published in <a href="https://medium.com/deep-learning-hk">Deep Learning HK</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>