<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Haren Lin on Medium]]></title>
        <description><![CDATA[Stories by Haren Lin on Medium]]></description>
        <link>https://medium.com/@haren?source=rss-baf78943a5fb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*KEhzbT1TseMZM6aZmQNtLw@2x.jpeg</url>
            <title>Stories by Haren Lin on Medium</title>
            <link>https://medium.com/@haren?source=rss-baf78943a5fb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:29:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@haren/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[F-1 VISA 美簽免面試心得分享]]></title>
            <link>https://haren.medium.com/f-1-visa-%E7%BE%8E%E7%B0%BD%E5%85%8D%E9%9D%A2%E8%A9%A6%E5%BF%83%E5%BE%97%E5%88%86%E4%BA%AB-dfe91ee7291d?source=rss-baf78943a5fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfe91ee7291d</guid>
            <category><![CDATA[interview-waiver]]></category>
            <category><![CDATA[f1-visa]]></category>
            <category><![CDATA[visa]]></category>
            <dc:creator><![CDATA[Haren Lin]]></dc:creator>
            <pubDate>Tue, 09 May 2023 12:34:48 GMT</pubDate>
            <atom:updated>2023-05-18T15:25:06.033Z</atom:updated>
            <content:encoded><![CDATA[<p>Keywords: 美國簽證、F1簽證、免面試</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*jpDo0GiA6B07Vvg-4SvCBg.jpeg" /><figcaption>Image Resource: <a href="https://www.allenglobalstudies.com/blog/application-process-for-us-student-and-f1-visa/">https://www.allenglobalstudies.com/blog/application-process-for-us-student-and-f1-visa/</a></figcaption></figure><h4>前言</h4><p>個人認為 AIT 際出的<a href="https://www.ustraveldocs.com/tw_zh/tw-niv-visarenew.asp">投件服務（免親自面談）</a>取得簽證的方式，使每個想前進美國的學生/工作者/親朋好友，可以在申請簽證之路下省下非常多的時間（真的非常非常多！），真的非常推薦。對此，想把這些申請的紀錄簡明扼要地記錄下來，提供有需要的朋友們參考～BTW，還沒拿到 I-20 / 還沒填寫 I-901並繳費USD$350 / 還沒填寫DS-160 者，請先將以上的事前工作完成，這篇文章你才用得到唷！</p><p>P.S. 流程簡介：先拿到 I-20 之後，可以到 I-901 填寫資料並繳費，接著填寫 DS-160，最後到 USTravelDocs 網站註冊填寫新的簽證申請，並繳交簽證申請費$160美金！（簽證申請費網站會自動轉台幣，以新台幣支付）</p><h4>個人背景</h4><p>14歲後入境過美國，且都有壓指紋。近五年（2017–2022）入境美國四次，每次都是以 <a href="https://esta.cbp.dhs.gov/">ESTA</a> 身份進出美國探親，前三次大該待10天左右，第四次去待了45天。這次要提交念二碩的 F-1 VISA 學生簽證申請。</p><h4>時間流程</h4><p>2023.05.04 Thu 下午五點將資料送至順豐快運內湖站（站內人員非常細心幫忙確認，給推！）<br>2023.05.05 Fri 追蹤快運狀態，早上八點已送入 AIT。下午用 DS-160 條碼查看簽證狀態，已經從 No Status 轉成 Approved<br>2023.05.07 Mon 早上查看簽證狀態，Approved 轉 Issued。下午護照已送至順豐快運。晚上即開始配送<br>2023.05.08 Tue 早上10:30配送到家收到護照及簽證</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*svgM16mXGWePgq-hmePoDQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OuzcQnnDErtgtKYc81C-DA.jpeg" /></figure><p>看見這神一般的效率，應該可以知道為什麼我這麼推薦身旁的朋友也使用免面試了吧！</p><h4>資格規定</h4><p>在準備免面試資料之前，請前往 <a href="https://portal.ustraveldocs.com/ApplicantHome">USTravelDocs</a> 網站登入主頁確認。除了從以下資格規定去確認自己是否符合免面試標準之外，也請記得在「新的簽證申請/安排預約」那部分，填寫好自己所有資料並持續填寫到最後一步，如果你符合標準，你將會在頁面中看到你的免面試確認信！看到免面試確認信就代表你一定符合免面試標準。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*REnqT9m-7GoeJiE_RYdncg.png" /><figcaption><a href="https://www.ustraveldocs.com/tw_zh/tw-niv-visarenew.asp#section3">https://www.ustraveldocs.com/tw_zh/tw-niv-visarenew.asp#section3</a></figcaption></figure><h4>文件準備</h4><p>針對文件準備的部分，請一切遵照<a href="https://www.ustraveldocs.com/tw_zh/tw-niv-visarenew.asp#section4">官方網站</a>所提供的資訊來準備，學生簽證/工作簽證等不同的簽證要提供的額外文件都會有所不同，請自行參考網站中「個別簽證種類所需的額外文件」的欄位。注意！他所列出必須給的文件，<strong>缺一不可</strong>，一旦有缺，申請者勢必拿黃單得走向面試環節，在每年五到八月的申請忙季將會耗時你一個多月，請務必再三確認。以下分享我在投件的文件中，附上了哪些東西～</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PI2Boo2-XJc0YO4-ZWVIsg.png" /><figcaption><a href="https://www.ustraveldocs.com/tw_zh/tw-niv-visarenew.asp#section4">https://www.ustraveldocs.com/tw_zh/tw-niv-visarenew.asp#section4</a></figcaption></figure><ol><li>未過期的有效護照：請提交正本，不要異想天開交影本，屆時簽證申請通過後，官方會將你的有效簽證貼在有效護照的內頁。P.S. 除了有效護照之外，我也附上從出生到現在所有使用的護照，以便查驗！</li><li>免面談確認信：申請者透過 USTravelDocs 填寫完「新的簽證申請/安排預約」後，符合資格者應該可以在頁面上看到免面試確認信，記得將其列印。</li><li>DS-160 確認信：填寫完成後，記得將確認信印出。</li><li>六個月內拍攝並且未曾用於之前美國簽證的彩色照片 (5cm x 5cm)：這部分請各位一定要乖乖的去重拍，此外美簽相片的尺寸規定與護照的尺寸規定不同，千萬不可以拿護照尺寸的相片來糊弄繳交。另外，這個照片我把它貼在 DS-160 確認信的左下角，不是貼死，用雙面膠簡易浮貼。</li><li>簽證申請費繳費收據（影本）：填寫完「新的簽證申請/安排預約」後，請拿著簽證費的繳費單，到郵局繳費，繳完費之後你會拿到一張收據，請妥善保存。因為上面的收據條碼將會是你前進面試的 Private Key。</li><li>額外文件：由於我是申請F-1 Visa，我就按照官方網站上所規範要給的額外文件來準備，如下：<br>6–1. CV / Resume：我將之前找實習/申請學校所使用的簡歷進行簡單修改，印出一張A4單面的簡歷。<br>6–2. I-20：將學校給的I-20印出，並簽名。記得，一定要簽名！<br>6–3. I-901繳費收據：將I-901繳費後的收據一併印出。</li><li>其他文件：我還附了我最近一次的 ESTA Approval 以及當初繳交給學校申請I-20 的財力證明。</li></ol><p>以上，就是我自己在免面試申請準備的所有資料，依序排列好之後左上角使用迴紋針固定好所有文件（不可以使用釘書機），整理好後放入牛皮紙袋，就可以拿到順豐快遞交件了！</p><h4>結語</h4><p>相信過去入境美國沒有不良紀錄，沒有把 ESTA 90天都用完的紀錄，只要乖乖地按照官網給的要求把所有文件備齊，要快速拿到 F-1 VISA 不是件難事，預祝各位簽證申請順利！如果你需要面試，也祝福您面試安全過關！我們美國見～</p><p>This article will be updated at any time! Thanks for your reading. If you like the content, please click the “clap” button. You can also press the follow button to track new articles. Feel free to connect with me via <a href="https://www.linkedin.com/in/harenlin/">LinkedIn</a> or email.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfe91ee7291d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Notes] (SIGIR2022) Positive, Negative and Neutral: Modeling Implicit Feedback in Session-based…]]></title>
            <link>https://haren.medium.com/notes-sigir2022-positive-negative-and-neutral-modeling-implicit-feedback-in-session-based-3bead49c9287?source=rss-baf78943a5fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bead49c9287</guid>
            <category><![CDATA[recsys]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[news-recommendation]]></category>
            <category><![CDATA[session-based-recsys]]></category>
            <dc:creator><![CDATA[Haren Lin]]></dc:creator>
            <pubDate>Fri, 24 Jun 2022 14:37:52 GMT</pubDate>
            <atom:updated>2022-06-24T14:37:52.147Z</atom:updated>
            <content:encoded><![CDATA[<h3>[Notes] (SIGIR2022) Positive, Negative and Neutral: Modeling Implicit Feedback in Session-based News Recommendation</h3><blockquote>這篇真的是🐮🍺🐮🍺！跳脫過往的 Seq Rec，直接轉成 Session-Based Rec！</blockquote><h3>Paper Link</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2205.06058.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1048624fe0d945ff9fea240ea5e09a9/href">https://medium.com/media/c1048624fe0d945ff9fea240ea5e09a9/href</a></iframe><h3>Source Code</h3><p><a href="https://github.com/summmeer/session-based-news-recommendation">GitHub - summmeer/session-based-news-recommendation: source code of paper &quot;Positive, Negative and Neutral: Modeling Implicit Feedback in Session-based News Recommendation&quot;, which is accepted at SIGIR 2022.</a></p><h3>Observation &amp; Idea</h3><ul><li>Previous works tend to formulate session-based recommendation as a next item prediction task, while <strong>they neglect the implicit feedback from user behaviors</strong>, which indicates what users really like or dislike.</li><li>The authors propose a comprehensive framework to model user behaviors through positive feedback (i.e., the articles they spend more time on) and negative feedback (i.e., the articles they choose to skip without clicking in). Moreover, the framework implicitly models the user using their session start time, and the article using its initial publishing time, in what we call “neutral feedback”.</li></ul><h3>Introduction</h3><p>在 BBC / CNN / Bing News 上面，有許多人是用匿名的方式或以訪客身份登錄到平台上。一般而言他們並不會在短時間內閱讀很多文章，因此形成 limited interactions，系統很難完全理解這些使用者的行為。</p><ul><li>Traditional methods: 傳統的方法會把 News Rec 給 Reduce 成 CTR prediction task，主要透過 CF 或是 FM 來讓系統追蹤 User history，缺點是沒辦法推薦給那些沒有 User history 的 Anonymous visits &amp; Guest logins。</li><li>Recent NN methods: 而近期的方法 (E.g. LSTUR, NRMS, NPA) 則是注重在使用 Attention mechanism 來對 News representation &amp; User representation 做 Encode，結著把 News Rec 視為 Sequential Rec，缺點是忽略了 Click behavior &amp; Article-to-article transition。E.g. 沒有充分利用與閱讀行為相關的 Temporal Information，這在 User interactions 很 Sparse 時尤其重要。</li></ul><p>考量到以上這些問題，作者把 News Rec 視為一個 Session-Based Recommendation 來解決！最終目標一樣是希望預測 User 下一個感興趣的 News item，只是模型會根據 Previous sequence of behaviors within a session 來做決策，而這個 Session 通常會是我們自己訂定的一段時間 (E.g. 30 min)。此外，作者也提出 ”Implicit feedback from user behaviors” 的 Modeling，如下圖。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E0RslGcvkB21wK1dRCHxQQ.png" /><figcaption>In session-based news reading, a user may spend different amounts of time on different clicked articles, rep- resenting different level of preference in an implicit form of positive feedback; a user’s impression of an article without an eventual click on the article represents an implicit form of negative feedback; the start time of the click and publish- ing time of articles can be viewed as neutral feedback.</figcaption></figure><p>Typical Implicit feedback 可以從 Browsing the main page, reading an article, closing an article, backtracking, etc 等等的方式來 Extract。作者認為透過這樣的 “Implicit Feedback” 來對 “Explicit Feedback” 建模可以對 User 有更好的推薦品質。對此，以下個問題是作者在本文中特別注重的三個面向。</p><ul><li>If a user clicked an article, did she really like it?</li><li>If a user did not click an article, did she dislike it?</li><li>How do we model the temporal characteristics of the user and the articles in the system?</li></ul><ol><li>在 Traditional Recsys，Clicks 通常代表 User 的 Like / Vote，但對於 News Rec，情況就有些不同了～User 可能會被 Tricked 去點擊一篇他沒興趣的文章，點入後一旦 User 意識到就會迅速退出並切換到其他文章。因此，用戶花在閱讀文章上的時間 (<strong><em>Spending Time on Article</em></strong>) 會是比較 fine-grained 的 information 能夠來表示 User 對文章的偏好程度！而且 Spending Time on Article 是一個 Continuous value，而 Click or not 是 Binary。</li><li>User 沒有 Click 的文章並不代表他不喜歡這個文章，可能只是因為他沒被 Exposed 到而已。假設文章大致按照 Publication Time 的順序呈現給 User，我們可以推斷出哪些文章可能在指定的 Session 內對 User 產生 Impression。<strong><em>只有那些在他的 Impression List 中出現，但沒有被點擊的文章才被認為 User 不感興趣的</em></strong>，作者把這樣的資訊視為 <strong><em>Implicit negative feedback</em></strong>！</li><li>雖然 Positive and Negative feedback 有助於 Estimate User &amp; News Article 之間的 Interaction，但一些關鍵的 Temporal information 對於個別 User 和 Artivle 的建模很有用。Session 的起始時間可以表示這個 User 的 Daily routine，並預期他在 same day of a week / same time of a day 有相同的 Reading behavior 或是 Reading background。另一方面，每篇文章的 Publication Time 也可以在一個 Session 中形成一個 Sequence，這反應了 User 對文章時效性的敏感度。因此，作者把 <strong><em>Session start time &amp; Article publishing time 視為 Implicit neutral feedback</em></strong>！</li></ol><h3>Recap: Session Based Model</h3><p>給定 Prefix sequence of the session — <strong><em>Su = (n1, n2, n3, …, nT)</em></strong> 預測出 User 在下一個時間點 T+1 最有可能點擊的 Item, nT+1。Item 的 Embedding 可以透過一個 <strong><em>N x dn</em></strong> 的 Item embedding matrix 取得，<strong><em>dn</em></strong> 是 Dimension of item embedding，<strong><em>xi</em></strong> 是 Item <strong><em>ni</em></strong> 的 Embedding vector。有了 Embedding vector 之後可以使用 GRU / RNN / Attention 等方式整合這個 Sequence 取得 <strong><em>xs</em></strong>，代表 User’s history preference。接著再把 Candidate Items Embeddings <strong><em>[x1, x2, x3, …, xN]</em></strong> 個別與 <strong><em>xs</em></strong> 做 Inner-product，且過 Softmax 取得 Similarity Score。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*ltRhUA4RjZRJ2kJAkgQ0ng.png" /></figure><p>最後，Loss 通常會選定 Cross-Entropy Loss：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*sdsqppfhphtw_mf3V1kF-Q.png" /></figure><p>S 是整個 Training Sessions。Su 是 S 裡面的一個 Session。N 是 Candidate Items 數量。yu_j = 1 代表 article j 是 Su 的 nT+1，yu_j = 0 則否。</p><h3>Proposed Model Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i5oV8DY4RtiG3VbgicvvRg.png" /><figcaption>Notations and descriptions</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HwdARAKoo0JNtO6tXEO-ZQ.png" /><figcaption>Model architecture. Squares in the figure represent vectors, and their colors refer to the different encoders that produce them.</figcaption></figure><h4>Part1. Base Model: Content-aware Recommendation (CAR)</h4><p>以下先介紹論文的 Base Model - CAR。在 News article’ content information 的部分，會抓 News 的 Title &amp; Metadata attributes of articles (E.g. Categories)，然後透過 Word2Vec 把這些文字資訊轉換成一個 dc-dimensional vector。接著，有了 Content vector <strong><em>ci</em></strong> of article <strong><em>ni</em></strong> 以及 Item embedding vector <strong><em>xi </em></strong>後，把 <strong><em>[xi; ci]</em></strong> Concatenate 得到 Embedding vector of ni, <strong><em>xci</em></strong>。把 Click Sequence 轉 [xc1, xc2, …, xcT] 之後，使用 <strong><em>Attention Pooling to encode user’s history preference XCs</em></strong>。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*c7E3jb_sm3faAYRXYGDQYw.png" /><figcaption>W0: 1 x dn; W1: dn x (dn+dc); xci: (dn+dc) x 1; b0: dn x 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*tgmCaA-kl9AvthRJq5FfrQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/1*fv4vj-B6wB6T022NZDbenw.png" /><figcaption>XCs = Contextual session vector</figcaption></figure><p>其中一個要注意的是，為了得到 Input sequence 的 Sequential information，他還會加入 Positional Encoding (像是 Transformer 的做法)，到我們的 Su，然後才一起做 Attention 得到 <strong><em>XCs</em></strong>。</p><p>P.S. 這個部分很直白，就是跟 Neural News Rec 一樣，把所有點擊過的 News 轉成 News Embedding <strong><em>xci</em></strong> 表示之後，再透過一個簡單的 Attention 得到 User Embedding <strong><em>XCs</em></strong>，接著去跟 Candidate News 算 Inner-Product 得到 Score。</p><h4>Part2. Modeling Time as Neutral Feedback</h4><p><strong>2–1. Active Time (Duration Time)</strong></p><p>作者會把 Continuous 的 Active Time <strong><em>ti</em></strong>，用 Floor function 轉換成 Discrete 的 <strong><em>ti’</em></strong>，並且把他 Map 到 m 個 categories / distinct time value，每個 category 會共享同個 Active time embedding vector,<strong><em> tai </em></strong>(dt x 1)。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/166/1*Kc5t53_toAvlIgC86dWp0g.png" /></figure><p><strong>2–2. Click Start Time (for XCs)</strong></p><ul><li>Users who start reading at a similar time are more likely to share the same reading behavior, which means that user interests are influenced by the start time. 在相似時間開始閱讀的 User 更有可能共享相同的閱讀行為，意味著 User Interest 受到 Start Time 的影響。E.g. Some people tend to read financial news in the morning but instead read entertainment in the evening.</li></ul><p>我們會把 <strong><em>Click Time from each click behavior of a session</em></strong> 記錄起來為 <strong><em>tsi</em></strong>，且只會使用 (w,h) week &amp; hour 的 temporal data。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/136/1*QIFGsgHOu-nXJbyObTV99A.png" /><figcaption>Embedding of click start time, i from 1 to |Su| = T</figcaption></figure><p>而為了 Model 在 <strong><em>Su</em></strong> 裡面每個 <strong><em>tsi</em></strong> 不同的重要性，一樣透過 Attention 機制來先得到 Preference Query (這部分跟 Transformer Query 一樣)，再用 <strong><em>Preference Query 來跟 Article 來做 Attention</em></strong> 得到他們彼此 Interaction 的重要性。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/336/1*Er9gbtzEFUMXAxVhoidMgQ.png" /><figcaption>Preference query, i from 1 to |Su| = T. Wt = dn x 2dt; tsi = 2dt x 1; bt = dn x 1; qi = dn x 1.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*hbY0qL2pEYlzYHKrhFTE7g.png" /><figcaption>Importance of interactions between article and preference query generated from click start time. ci 應該改為 xci (我認為作者有typo) = (dc+dn) x 1; Wt’ = (dc+dn) x dn; qi = dn x 1; αti = (dc+dn) x 1.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*bh9L612YfLBlqMuIP4NFEw.png" /><figcaption>Get weighting parameters via softmax. |Su| = T.</figcaption></figure><p>最後<strong><em>把新得到的 αti’ 與 Part 1 Base Model 的 αi’ 做結合，重新 Define Contextual vector representation XCs</em></strong>。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/1*Ac713gMxH4O-qwn9w85WJg.png" /><figcaption>Redefined contextual vector representation. XCs = (dn+dc) x 1.</figcaption></figure><p><strong>2–3. Publish Time (for XTs)</strong></p><ul><li>Users’ reading habits are reflected in the sequence of publishing time 𝑡𝑝1, …, 𝑡𝑝𝑖 in 𝑆𝑢 . We can make inferences whether the user tends to browse new articles or older ones from this. 使用者的閱讀習慣反映在 𝑆𝑢 中的發佈時間 𝑡𝑝1, …, 𝑡𝑝𝑖 的序列，可以由此推斷用戶是否傾向於瀏覽新文章或舊文章。</li></ul><p>我們會把 <strong><em>Publishing time embedding vector</em></strong> 記錄為 <strong><em>tpi</em></strong>，且會使用 (s,d,w,h,m) season, day, week, hour, minute 的 temporal data。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/140/1*w6Pn_zaHGHjj_LuLs0eB8w.png" /><figcaption>Publishing time embedding vector, i from 1 to |Su| = T</figcaption></figure><p>接著，計算 <strong><em>Article content vector (ci) 與 Publishing time embedding vector (tpi) 之間的 Attention Score αtp’_i</em></strong>。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*EX-skPEWeiNn2c4u3Eq4Dg.png" /><figcaption>W0’ = 1 x dn; W1&#39; = dn x 5dt; tpi = 5dt x 1; W2&#39; = dn x dt; ci = dc x 1; b0&#39; = dn x 1.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/1*WY-fyT9wBgEBAssOQrkX3Q.png" /><figcaption>αtp’_i = 1 x 1</figcaption></figure><p>最後<strong><em>把 Attention Score αtp’_i 跟 Publishing time embedding vector tpi 做 Weighted Sum 得到 xts</em></strong>。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*6XYe8yf68Au7NF9IXRYIxw.png" /><figcaption>Final temporal session representation. tpi = 5dt x 1; xts = 5dt x 1.</figcaption></figure><p>最後會把 <strong>XCs (dn+dc) &amp; XTs (5dt) Concat</strong> 起來形成 Aggregated Xs。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/218/1*ucAEdNTAjKmS_4iNXJmEmQ.png" /></figure><h4>Part3. Modeling Positive Feedback (From Active Time)</h4><ul><li>Implicit positive feedback takes the form of the active time interval that a user spent on each article after clicking on it. 如果使用者在一篇文章中停留的時間很短，很可能是因為用戶被標題給 Tricked 了，實際上他並不喜歡這篇文章。(P.S. 如果沒有明確可用的 Active Time，可以通過用戶連續兩次點擊的時間間隔來估計)</li></ul><p>這部分，作者會把 2–1 提到的 <strong><em>Each degree of active time embedding tai 抓進來， 跟每一篇文章的 Embedding vector xci 去算 Attention</em></strong>。</p><p>會把剛剛在 Part1 Base Model 計算的 αi：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*c7E3jb_sm3faAYRXYGDQYw.png" /><figcaption>αi = 1 x 1; W0: 1 x dn; W1: dn x (dn+dc); xci: (dn+dc) x 1; b0: dn x 1</figcaption></figure><p>轉換成新的 αi：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*o563lBbDagkgNCKFh1x05g.png" /><figcaption>αi = 1 x 1; W0 = 1 x dn; W1 = dn x (dc+dn); xci = (dc+dn) x 1; W2 = dn x dt; tai = dt x 1; b0 = dn x 1</figcaption></figure><p>然後同樣 Weighted Sum 得到 XCs 以及 Aggregated 的 Xs。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/1*Ac713gMxH4O-qwn9w85WJg.png" /><figcaption>Redefined contextual vector representation. XCs = (dn+dc) x 1.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/218/1*ucAEdNTAjKmS_4iNXJmEmQ.png" /><figcaption>Aggregated Xs</figcaption></figure><h4>Part4. Modeling Negative Feedback (From Impression List)</h4><p>最直觀得到 Negative Feedback 的方法就是大家常用的 Negative Sampling，但因為這樣隨機抽樣 Item 可能與 User 完全無關，對模型學習構成的挑戰太小。一個 Informative Item 應該能夠混淆模型是否發現到 More complex hidden meaning of user interaction。</p><p>對此，作者的做法是使用 Impression List <strong><em>Imp_u</em></strong> 來生出 Negative data。將 <strong><em>Imp_u </em></strong>中的 Un-clicked news 視為 Negative signals，比起其他 Candidate items 應該要給予不同的 Loss 讓他能夠順利學習去區分不同 Candidate items，應該更嚴格地懲罰 XCs 和那些 Strongly Negative Samples 之間的 Similarity。(c.f. Contrastive Learning)</p><p>其中要注意的是，Impression List 並不一定能夠取得，因此作者假設如果一篇文章被 Published 在被 user <em>u</em> 點擊的文章附近，它更有可能出現在 <strong><em>Imp_u</em></strong>中。具體上的做法是：根據 Candidate News 的 Publication Time 對 Candidates 進行 Sorting，保留 Window Size 300 的 Nearby articles，並從中來進行抽樣。目標是最小化 <strong><em>XCs</em></strong> 和 Negative Sample <strong><em>j</em></strong> (𝑗 ∈ 𝑁𝑒𝑢) 的向量 <strong><em>XCj </em></strong>之間的 Cosine Similarity Score。(<strong><em>Neu</em></strong> = <strong><em>Su</em></strong>’s negative samples set，<strong><em>Neu</em></strong> <strong>⊆ <em>Imp_u</em></strong>)</p><p>訓練上的 Loss 會把這部分，加入原本的 L1 後面，形成 L2。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*xoJZKgrGTN8fEbGztVf6UA.png" /><figcaption>1(·) returns 1 if the expression is true, 𝜆 is the weighting parameter of loss from negative articles. Jointly optimize these two losses with Adam optimizer.</figcaption></figure><h3>Experiments</h3><h4>Datasets</h4><p>1. Adressa</p><p><a href="https://reclab.idi.ntnu.no/dataset/">Reclab</a></p><p>2. Globo</p><p><a href="https://www.kaggle.com/datasets/gspmoreira/news-portal-user-interactions-by-globocom">News Portal User Interactions by Globo.com</a></p><p>3. MIND</p><p><a href="https://msnews.github.io/">MIND</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2kWsCIBxLFiyTkjbcDepYg.png" /><figcaption>Dataset statistics (after preprocessing)</figcaption></figure><p>P.S. Datasets 細節可以參考 Source Codes～</p><h4>Metric</h4><p>以下介紹除了 Hit Rate@K &amp; NDCG@K 之外的兩個指標：(1) ILD@K (2) unEXPu@K。</p><ol><li><strong><em>Intra List Diversity (ILD@𝑘)</em></strong> evaluates the topical/semantic diversity in 𝑅, and reflects the model’s ability to recommend differ- ent items to the same user.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*DYiQO3SaS2qjxTDUP7umxg.png" /><figcaption>d(a,b) is a distance measure between item a and b, and d(a,b) = 1 if item a,b belong to different topics (categories), 0 otherwise.</figcaption></figure><p>2. <strong><em>Content-based unexpectedness metric (unEXP)</em></strong> can be used to measure this kind of unexpectedness. (We expect the system to recommend unseen items to surprise users)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*53U4z543sXrlUBOMDvni2g.png" /></figure><h4>Main Results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7icAluKfmsp1Mq4YvjoFIQ.png" /><figcaption>Main and ablation results (𝑘 = 20 by default in our all tables). All results are averaged over all folds. The best baseline result on each metric is marked with ∗ and overall best results are bolded. The “Ours” is our whole model and (-) means to ablate the corresponding module, where “neut”, “pos” and “neg” respectively refer to our neutral, positive and negative feedback modules. The last column is to replace our negative sampling strategy with random sampling. ↓ indicates performance drop over the whole model.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ugAx3bLLNWsfmrxyXaUh4Q.png" /><figcaption>The graphical comparison of all methods on 4 different metrics and 3 different datasets.“Ours” is our approach.</figcaption></figure><ol><li><strong>Non-neural methods (CBCF and STAN)</strong> are either considering the content information or the recency of the current session, and their results are somehow comparable to deep learning methods in three datasets. However, they generate recommendation lists with <strong>low diversity / novelty</strong>, mainly because their simple <strong>algorithms cannot capture enough personalized information</strong>.</li><li><strong>Session-based approaches (STAMP and SGNNHN)</strong> yield better performance on HR and NDCG, but not always good at ILD/unEXP, while SASRec and SRGNN recommend more diverse but less accurate, showing the <strong>trade-off between diversity and accuracy</strong>.</li><li><strong>From the user’s aspect, though, when ILD/unEXP is over a threshold (like around 0.83), it’s hard for them to distinguish the difference, thus the ILD/unEXP score of our model is bearable.</strong></li><li>For “<strong>Ours</strong>”, when compared with STAMP, it performs better or close on both accuracy and diversity. This result shows that our model <strong>mitigates the dilemma between accuracy and diversity to a certain extent</strong>.</li></ol><p>P.S. In the MIND dataset, the improvement is comparatively small and the possible reasons are: on the one hand, <strong>MIND did not provide active interval, nor did they give click time of each article (just start time of a session), we cannot get positive feedback from the data</strong>; on the other hand, from the results of CBCF, we assume the article transition information is too sparse and thus it is hard to recommend. <strong>Note that this dataset is not designed for the session-based recommendation, hence some information may be inaccurate</strong> (e.g., one session may last for days, longer than 30 minutes).</p><h4>Analysis</h4><ol><li>Compared with the whole model, there is a huge drop after removing neutral information and this is the most consistent over all metrics, which reveals the importance of neutral information (temporal information).</li><li><strong>Adressa provides the most complete information, which not only releases the original text of articles instead of the extracted vectors in Globo, but also gives the accurate active time of the user in each article</strong>, while we can only <strong>estimate the active time by the interval between two successive clicks for Globo, which may not be accurate</strong>.</li><li>After removing the positive implicit feedback module, in Adressa dataset, the HR and NDCG drop by 2.4% and 5.5% respectively, while in Globo dataset, they drop by 1.9% and 3.1%. The positive information performs similarly in both Adressa and Globo datasets, implying that our approximate estimation is reasonable. Further, the positive implicit feedback is more favorable on the Adressa dataset due to the more precise information.</li><li>Negative information is less effective than positive information, especially by diversity/novelty metrics. One explanation is that <strong>the negative samples from the impression list are reconstructed based on their publishing time, so the information is not totally reliable</strong>.</li><li>Negative sampling module lowers diversity, possibly because in the dataset the negative samples and the positive article usually belong to different categories, thus adding this module forces the model to recommend similar articles to the positive one.</li><li><strong>Negative feedback is better modeled in MIND due to its complete impression data</strong>. To verify the effect of the negative sampling strategy more accurately, we set the control group with random sampling, and we find that even though the random sampling would decrease the performance slightly, our negative feedback shows superior performance over it. The possible reason for the worse performance of using random sampling is that randomly sampled negative items have the possibility to be liked by this user, and this module imports some noise instead because this sample strategy does not consider what the user really likes.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tR_GCXtLfAt-Ljdgr9xV6g.png" /><figcaption>Hyper parameter discussion</figcaption></figure><p>For fig above in (a), the authors wanted to validate the assumption for the negative user feedback, which is that articles whose publishing time is close to the clicked articles are likely presented to the user, or within their impressions. For each session 𝑆𝑢 , the authors sample negative items 𝑁𝑒𝑢 using their strategy, and compute Jaccard similarity between 𝑁𝑒𝑢 and real 𝐼𝑚𝑝𝑢, the overall score is 0.0062 when |𝑁𝑒𝑢 = 100|, compared with 0.0044 when random sampling. It means their assumption is reasonable.</p><p>For fig above in (b), negative loss is useful but too many weights on it will harm the learning of the user’s positive feedback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*1PCee9jT_Vo3Zde2Kq67TA.png" /><figcaption>Different ways of utilizing temporal information in Adressa, where “p” stands for the publishing time and “s” stands for the start time</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mHIuzhQqDNKJRQYCGBMC1w.png" /><figcaption>The visualization of time embedding tables for the day of a month, the day of a week, hour and minute, trained on Globo dataset.</figcaption></figure><p>The representation of the minutes is rather uniform and random, because news publishing and reading can happen any minute of an hour. But there are certainly more activities at certain hours during a day. There are also some irregular patterns for weekends as shown in (c).</p><p>P.S. 其他實驗分析在這篇文章省略，有興趣可以再參照原文～</p><h3>Contribution</h3><ul><li>For the first time, the authors leverage the positive/negative/neutral implicit feedback in anonymous session-based news recommendation.</li><li>Designed novel representations for temporal information and incorporate it with positive and negative feedback into a deep attention network.</li><li>Comprehensive offline evaluations on three real-world datasets show the clear advantage of the proposed method in terms of overall performance on diversity, accuracy and serendipity in both normal and cold-start scenarios.</li></ul><p>This article will be updated at any time! Thanks for your reading. If you like the content, please clip the “clap” button. You can also press the follow button to track new articles. Feel free to connect with me via <a href="https://www.linkedin.com/in/harenlin/">LinkedIn</a> or email.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bead49c9287" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Notes] (SIGIR2021) DUMN: Deep User Match Network for Click-Through Rate Prediction]]></title>
            <link>https://haren.medium.com/notes-sigir2021-dumn-deep-user-match-network-for-click-through-rate-prediction-aa305198e9af?source=rss-baf78943a5fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa305198e9af</guid>
            <category><![CDATA[dumn]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[recommendations]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Haren Lin]]></dc:creator>
            <pubDate>Thu, 23 Jun 2022 17:09:11 GMT</pubDate>
            <atom:updated>2022-06-23T17:30:34.819Z</atom:updated>
            <content:encoded><![CDATA[<h3>Paper Link</h3><p><a href="https://dl.acm.org/doi/abs/10.1145/3404835.3463078">Deep User Match Network for Click-Through Rate Prediction | Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval</a></p><h3>Source Code</h3><p><a href="https://github.com/hzzai/DUMN">GitHub - hzzai/DUMN</a></p><h3>Observation &amp; Idea</h3><ol><li>Recently deep learning based models have been proposed and successfully applied for CTR prediction by focusing on feature interaction or user interest based on the item-to-item relevance between user behaviors and candidate item.</li><li>However, these <strong><em>existing models neglect the user-to-user relevance between the target user and those who like the candidate item</em></strong>, which can reflect the preference of target user.</li></ol><p>DUMN: matching the target user and those who like the candidate item (i.e. who have interacted with candidate item in our work) to measure the user-to-user relevance.</p><ul><li>Embedding Layer. (for user behaviors and profile features)</li><li><strong><em>User Representation Layer</em></strong>. Learn a unified user representation which contains user latent interest based on user behaviors and profile.</li><li><strong><em>User Match Layer</em></strong>. Measure the user-to-user relevance by matching the target user and those who have interacted with candidate item and modeling their similarities in user representation space. With the user-to-user similarity relevance, generate the matching representation to reflect the preference of target user.</li><li>CTR Prediction Layer. The matching representation and user representation of the target user, the user-to-user similarity relevance, and the embedding vectors of the candidate item and context are concatenated and fed into several FC layers for outputting predicted CTR.</li></ul><h3>Main Contribution</h3><ol><li>Highlighted the importance of capturing the user-to-user relevance between target user and those who like the candidate item.</li><li>Designed two network architectures to learn a unified user representation and model the user-to-user relevance with the similarities based on user representations, respectively.</li></ol><h3>Model Architecture</h3><h4>Part1. Notations</h4><ol><li>User Behavior: Given a user <em>u</em>, the <strong>user behavior <em>Iu</em></strong> is the sequential list of the interacted items with corresponding features such as item id, category, etc. (使用者過去的點擊紀錄，裡面是每個 Item，共有 Nu 個 Items)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/1*RhwzBTlCcW9pWMZFuNeSBA.png" /><figcaption>ik is the k-th interacted item, Nu is the length of Iu.</figcaption></figure><p>2. Item Behavior: Given an item <em>m</em>, the <strong>item behavior<em> Um</em></strong> is the sequential list of users who interact with <em>m</em> with corresponding profile features such as user id, age, etc., as well as their user behaviors when interacting with <em>m</em>. (Item的歷史紀錄，包含 user id 以及他的 User behavior list 的 Pair，一共有 L 個 User Pairs 的記錄)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*oT7GzGc_BmcJtqe43gpbcw.png" /><figcaption>uk is the k-th interacted user, Iu is the user behavior of uk, L is the length of Um</figcaption></figure><p>CTR 預測的目標是在給定 User behavior <strong><em>Iu</em></strong>, Item behavior <strong><em>Um</em></strong>, User features <strong><em>u</em></strong>, Item features <strong><em>m</em></strong>, and Context <strong><em>c </em></strong>的情況下，用 Model <strong><em>F </em></strong>估計這個 User <strong><em>u</em></strong> 點擊 Candidate item <strong><em>m</em></strong> 的機率 <strong><em>p</em></strong>。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*ZBx4QRNgcDV6jGoD8iE1NA.png" /></figure><h4>Part2. Embedding Layer</h4><p>Transform the input features of DUMN to embedding vectors.</p><ol><li>Encode each feature into a one-hot vector with high-dimensional sparse binary encoding.</li><li>The one-hot vector then are transformed into low-dimensional dense vector by the embedding layer.</li></ol><p>我們手上有的資料包含：User features <strong><em>u</em></strong>, Item features <strong><em>m</em></strong>, and Context <strong><em>c</em></strong>, User behavior <strong><em>Iu</em></strong>, Item behavior <strong><em>Um</em></strong>.</p><ul><li>Profile of user features <em>u</em> → eu.</li><li>Candidate item features <em>m</em> → em.</li><li>Context <em>c</em> → ec.</li><li>Item (historical) behavior <em>Um</em> → Zm = [(eu1, Xu1), (eu2, Xu2), …, (euL, XuL)]. (L is the length of item behaviors)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*eBJXVDQi4twKrYjC7nfy2A.png" /></figure><ul><li>User (historical) behavior = User Interacted Items <em>Iu</em> → Xu = [ei1, ei2, …, eiNu]. (Nu is the length of user behaviors)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*x8EBzvOXdBUGiVA8klQzCg.png" /></figure><p>After embedding layer, we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/196/1*DA-Es0GMGbQ5jI2GsoMihA.png" /></figure><h4>Part3. User Representation Layer (i2i)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*VJlrYuOanNyuNnyxThNQQA.png" /><figcaption>User Representation Layer in DUMN</figcaption></figure><ul><li>With obtained embedding vectors of the user behavior and profile via Embedding Layer, learn the unified user representation in User Representation Layer by combining user latent interest and profile features.</li><li>Leverage an attention mechanism to measure the <strong><em>item-to-item relevance</em></strong> based on item embedding representations and obtain the user interest representation according to the relevance.</li></ul><p>透過 Candidate item embedding (em)，去跟 User Behaviors 裡面的每個 Item 去計算 Attention Scores 得到 γk (k from 1 to Nu)，把 γk 都過 Softmax 得到 Attention Weights μk。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Rfr7-5yQlzmikPtKVTQxyg.png" /><figcaption>eik is the embedding vector of the k-th interacted item in the user behavior of u, γk is the item-to-item relevance between the k-th interacted item and candidate item, μk is the attention weight, and σ is the sigmoid function.</figcaption></figure><p>利用 Attention Weights (μk) 與 User Behaviors 中每個點擊過的 Item embedding (eik) 做 Weighted Sum，得到 <strong><em>User interest representation</em></strong> (ru_hat)。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/154/1*50tOHy-RioiJ69ADy31cLw.png" /></figure><p>接著，把 User interest representation (ru_hat) 跟 Target user u’s profile embedding (eu) 做 Concat，得到最後得到 (Candidate-Aware) <strong><em>User representation (ru)</em></strong>。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/204/1*SRbNev3h8scTvCCdK_87zQ.png" /></figure><p>P.S. 這個 ru 會有 ru1, ru2, …, ruL，共 L 個。因為 Item Behaviors 長度為 L，共有 L 個 Users 跟他對應的 User Behaviors。</p><h4>Part4. User Match Layer (u2u)</h4><blockquote>個人覺得這個部分某種程度上隱含 User-Based Collaborative Filtering (UserCF) 的概念。</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Be4sD9eEOMwurij32Ec1Q.png" /><figcaption>DUMN Main Architecture</figcaption></figure><ul><li>User Match Layer aims at measuring the <strong><em>user-to-user relevance</em></strong> by matching target user and users in the item behavior of candidate item and modeling their similarities with a <strong><em>Relevance Unit</em></strong>.</li><li>With the user-to-user similarity relevance, User Match Layer also generates a matching representation to reflect the preference of target user.</li></ul><p><strong>Part4–1. Relevance Unit</strong><br>- Input: representation of user a and user b. (ruk &amp; ru)<br>- Output: similarity of user a and user b.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*CTLWJFi0MKJm8APsEemX8w.png" /></figure><p>透過 Relevance Unit 我們可以得到 Target user u 以及 Candidate Item’s item behavior 中每個 User uk 的 user-to-user similarity relevance αk。</p><p><strong>Part4–2. Matching Representation for target user <em>u</em></strong></p><p>根據 u2u relevance score (αk) 與 User representation (ruk) 做 Weighted Sum，得到 <strong>Matching Representation for target user <em>u (Su)</em></strong>。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/204/1*R4Zn2LOJTF-cry6D1jL3oA.png" /></figure><p><strong>Part4–3. Total User-to-user Similarity Relevance</strong></p><p>User-to-user Similarity Relevance αk, k from 1 to L 總和為 Ru。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/168/1*C4OUEqHsD8QGS2wUr8LPAQ.png" /></figure><h4>Part5. Output Layer</h4><ul><li>Output Layer targets at calculating the predicted probability that the target user u clicks the candidate item m.</li></ul><p>[Step1] 把 matching representation (Su) and user representation of target user (ru), total user-to-user relevance (Ru), and the embedding vectors of the candidate item and context (em &amp; ec)，全部 Concat 起來變成一條高維度的向量。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*P8jvufcvGGGCnexYVaj3vQ.png" /></figure><p>[Step2] 把他傳入兩層的 FC Layers 且使用 PReLU activation function。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*agxdwEU_bHqARbXq2yoWwA.png" /></figure><p>[Step3] 把 Last hidden layer 抓出來過 Linear Transform 後經過 Sigmoid 得到 Predicted CTR <strong><em>p</em></strong>。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/208/1*Cd4-SkHqFLzQg1JYe_bALQ.png" /></figure><p>P.S. Training with NLL loss</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*r36uWE6ttL6izBU7b-QNaw.png" /><figcaption>NLL Loss</figcaption></figure><h3>Experiments</h3><p>Datasets: Amazon product reviews and metadata → Sports, Beauty, Grocery Datasets。</p><p>In three datasets, the product reviews consist of the information of the user, item, rating and timestamp. 85% for train, 15% for test.</p><p>[Step1] Treat all existing rating reviews as positive interactions in which the click label is 1.</p><p>[Step2] Filter out users and items that have less than 5 reviews.</p><p>[Step3] Sort the reviews by the timestamp to build the user behaviors and item behaviors.</p><p>[Step4] Negative Sampling: Build the negative samples by replacing the item in each review with another item randomly selected from the non-clicked item set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iHc1ZCCMFzcFCyUNbCckhg.png" /><figcaption>Datasets Statistics</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1PNIld_PIIG4YAMe2Y_FWQ.png" /><figcaption>Performance comparison on three datasets. The percentage in the last row of DUMN is the relative improvement compared to the best baseline with t-test at p-value of 0.05.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yKwAA__Axd9exuyctMK1SA.png" /><figcaption>Influence of the maximum length Lmax of item behavior on DUMN in the training phase on Beauty dataset.</figcaption></figure><p>This article will be updated at any time! Thanks for your reading. If you like the content, please clip the “clap” button. You can also press the follow button to track new articles. Feel free to connect with me via <a href="https://www.linkedin.com/in/harenlin/">LinkedIn</a> or email.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa305198e9af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Notes] (SIGIR2022) MM-Rec: Multimodal News Recommendation]]></title>
            <link>https://haren.medium.com/notes-sigir2022-mm-rec-multimodal-news-recommendation-593fd099b24e?source=rss-baf78943a5fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/593fd099b24e</guid>
            <category><![CDATA[recommendations]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[news-recommendation]]></category>
            <dc:creator><![CDATA[Haren Lin]]></dc:creator>
            <pubDate>Mon, 20 Jun 2022 02:03:26 GMT</pubDate>
            <atom:updated>2022-06-20T02:03:26.447Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>前方高能：這是一篇結合 Computer Vision &amp; Natural Language Processing &amp; Recommender System 的論文！！！CV + NLP + Recsys！！！</blockquote><h3>Paper Link</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2104.07407.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f4181ab0bcd618a1bdebb49050f7890/href">https://medium.com/media/0f4181ab0bcd618a1bdebb49050f7890/href</a></iframe><h3>Observation &amp; Idea</h3><p>Users may click news to read not only because of the interest in the content of news title, but also due to the fascination of news images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9_fuZb_kuA3VhEtO3sEXNg.png" /><figcaption>News with images for news recommendation</figcaption></figure><ol><li>News titles and images usually have some relatedness in describing news content and attracting clicks. E.g. 上圖的 2nd news clicks 標題的 Cowboy 與新聞圖像中顯示的球員有關。對它們的相關性進行建模可以幫助更好地 news modeling 並推斷 user interests。(=&gt; Multimodel news encoder)</li><li>A user may have multiple interests, and a candidate news may only be related to a specific interest encoded in part of clicked news. E.g. 上圖的 Candidate news 僅與 2nd news click 的新聞相關。因此，對 Clicked news 和 candidate news 之間的相關性進行建模可以幫助預測用戶對 candidate news 的特定興趣。(=&gt; Crossmodal Candidate-aware Attention)</li><li>Candidate news may have crossmodal relatedness with clicked news. E.g. 上圖中 Candidate news 的 Image 與 2nd news clicks 的 image &amp; title 相關，因為兩張影像都顯示了同一支球隊，並且在 2nd news clicks 的標題中提到了它的名字。對 Candidate news &amp; Clicked news之間的 Cross-model relation 進行建模可以幫助準確地衡量它們的 relatedness。(=&gt; Crossmodal Candidate-aware Attention)</li></ol><h3>Model Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TUG5__we77Ru8Hrfcv_rfQ.png" /><figcaption>The model architecture of MM-Rec</figcaption></figure><h4>Part1. Multimodal News Encoder</h4><p>由於 News image 的不同區域可能對 News modeling 具有不同的 informativeness，因此作者先用在做 Object detection 的 Pre-trained model: <a href="https://arxiv.org/abs/1703.06870">Mask-RCNN</a> 模型來提取 News image 的 Region Of Interest (ROI)。接著再用 <a href="https://arxiv.org/abs/1512.03385">ResNet-50</a> 來提取 ROI 的 Features。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*fBHGA1pUW12vkR8g7I1t2w.png" /><figcaption>News image feature sequence, where 𝐾 is the number of ROIs.</figcaption></figure><p>接著是 Textual Source 的建模，作者只拿 News title 來做 Modeling。首先先把 News title 進行 Tokenization 得到 Word Sequences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*gkBIWD9BIN3RLaNAHuhWGQ.png" /><figcaption>Word Sequence, where 𝑀 is the number of words.</figcaption></figure><p>一個直觀的方法是使用各別的的模型對 News Title &amp; Image 進行獨立建模。但同一條新聞的 Title 和 Image 通常有一定的關係。捕捉 News Title 和 Image 之間的相關性有助於更好地理解其內容並推斷 User interests。而 Visiolinguistic models 對 Cross-model 這樣的任務非常有效，因此作者決定套用 <a href="https://arxiv.org/abs/1908.02265">ViLBERT</a> 來捕捉 News title 和 image 的 inherent relatedness，以利學習 News representation。</p><p>ViLBERT 拿到的輸入資料就是剛剛上面講的 ROI News image feature sequence &amp; Word sequence。首先會通過幾個 Vanilla Transformer 對 Contexts 建模，然後使用幾個 Co-Attention Transformer 來捕獲 News image &amp; title 之間的 Cross-model interactions。而輸出的部分則為：hidden ROI representation sequence Hp &amp; hidden word representation sequence Ht。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*GBEY3qeddHKOEJCcqUrY9g.png" /><figcaption>Hidden ROI representation sequence</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*PlanHuQk5gjlqlwWO-8MCA.png" /><figcaption>Hidden word representation sequence</figcaption></figure><p>有了 News title &amp; News image 的 Hidden representation 後，接著會分別套用 Word attention network 和 Image attention network 來學習 News title representations 以及 News image representations。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*vWByHTdSaTS4dMToVgu1eA.png" /><figcaption>Image attention network. qp is an attention query vector and Wp is a parameter matrix</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*fQ_KF44PFKxyB4P8FWxrGA.png" /><figcaption>Word attention network. qt is an attention query vector and Wt is a parameter matrix</figcaption></figure><p>最後用 Attention Pooling 得到 News image representations, <strong>r<em>p</em></strong> 以及 News title representations, <strong>r<em>t</em></strong>.</p><h4>Part2. Multimodal News Recommendation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*tAvx5-NGaveUOLuW5g9WHA.png" /><figcaption>Previously clicked news from their titles, 𝑃 is the number of clicked news</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*GZ_1jiPmf_CPUFNsv8EJpg.png" /><figcaption>Previously clicked news from their images, 𝑃 is the number of clicked news</figcaption></figure><ol><li>Selecting clicked news according to their relevance to candidate news in user modeling may help accurately match candidate news with user interest.</li><li>Candidate news may have some cross-modal relations with the images and titles of clicked news.</li></ol><p>由於上面兩點的啟發，作者提出 Cross-model candidate-aware attention network。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/162/1*pAhUgWh9kgu4516UjTCJ0A.png" /><figcaption>image and text representations of the candidate news Dc</figcaption></figure><blockquote>a) Text-text attention weights for clicked news.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*qfaxYSpqvTp0o695rZjDWQ.png" /></figure><blockquote>b) Text-image attention weights for clicked news.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*F2rsbJ2-C6lh6x_JMKrzHg.png" /></figure><blockquote>c) Image-text attention weights for clicked news.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*cvoDdFbHL5i6nA_Vm1vcGw.png" /></figure><blockquote>d) Image-image attention weights for clicked news.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*qcJs5bgOjg9j0qIfb2dpwQ.png" /></figure><p>透過這四個 Attention weight 得到 Unified user embedding <strong>u</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*TZoC4N-T0lTTOrCmLPa6IA.png" /><figcaption>Unified user embedding</figcaption></figure><h4>Part3. Click Score</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/336/1*CE_PNhYE3i9C6YuNHFyFUQ.png" /><figcaption>Click Prediction Score</figcaption></figure><p>P.S. Trained with negative sampling &amp; cross-entropy loss.</p><h3>Datasets</h3><p>其實對於這樣子影像與文字資料結合的 News Rec，目前並沒有一個公開的資料集可以作為 Benchmark，對此作者自己從 commercial news website 爬了三週的 logs 出來，從 2020.02.25 到 2020.03.16，前一週的資料拿來做 User histories，後兩週的資料拿來做 Click Samples，而 Training data 為前 1M 筆資，後面再各自拿 100K 做 Validation &amp; Testing。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*t-BRTuE0ZoMU7v8nxtHVoA.png" /><figcaption>Statistics of the dataset</figcaption></figure><p>P.S. Training details from the paper: In our experiments, we finetuned the last three layers of ViLBERT. We used Adam as the optimizer (lr=1e-5). The batch size was 32. We tuned hyperparameters on the validation set. We repeated each experiment 5 times and reported the average AUC, MRR, NDCG@5 and NDCG@10 scores.</p><h3>Experiments</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3dUvBANmlVWbl9gD59vug.png" /><figcaption>Performance comparison of different methods</figcaption></figure><p>在上述的其他非 MM-Rec 的模型，作者在 News Encoder 使用 BERT，而且他們都只有使用文字資料，並沒有使用圖片作為額外的 Feature。可以從數據上明顯看到，引入圖片的資料能輔助 News Rec 的預測 (t-test p &lt; 0.01)。</p><h3>Ablation Study</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Ps0URdqvK9Lp-tzUAh2ig.png" /><figcaption>Effectiveness of multimodal information</figcaption></figure><p>News title &amp; image 對於學習 News representation 以進行推薦都很有用。有趣的是：即便 MM-Rec 不引入圖片 Module 作為訓練，他的表現仍然比 PLM-NR 等等其他的 Baseline 還要好，因為 ViLBERT 模型在 multi-modal data 上進行了預訓練，可以利用視覺信號來增強文本理解。此外，結合 multi-modal news information 可以進一步提高推薦效果，這表示他有助於學習準確的 News representation。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XSUvanZjAoDak91qf-xu9w.png" /><figcaption>Effect of the co-attentional Transformers in ViL-BERT and cross-modal candidate-aware attention</figcaption></figure><p>作者也研究了 ViLBERT’s Co-Attention Transformers 和 Cross-modal candidate-aware attention network 在 User interests 建模中的有效性。作者設計兩種變形，一種是沒有 Co-Attention Transformers 的 MM-Rec，另一種是用 Vanilla Attention 取代 Cross-modal candidate-aware attention network 的 MM-Rec。如上圖，當然是同時引入兩者的 MM-Rec 表現會是最佳，因為 News title &amp; image 在代表新聞內容和吸引新聞點擊方面存在內在的 hidden relation。因此，對它們的 interaction 進行建模可以增強它們的 representation。再者，Cross-modal candidate-aware attention network 很有用，因為不同的 historical news clicks 對於 user interests 通常具有不同的 importance。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gVyuAPbgOW5hbxr9Gm10mA.png" /><figcaption>Ablation studies on different embeddings</figcaption></figure><p>作者發現：當刪除任何 Candidate news embedding 或是 User embedding，都會造成效果下降，這就表示 Textual &amp; Visual information 對於 News &amp; User interests modeling 都很有用。此外，Textual information 佔有更重要的作用。這是一個非常有趣的現象，因為 Textual Source 通常不如 Images 有吸引力。作者認為這主要是因為單個新聞影像通常無法全面概括新聞內容，透過視覺信息理解整個新聞可能是具有挑戰性。</p><h3>Case Study</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*Z_1CHC0EJmCeIJIjkSEVZQ.png" /><figcaption>The clicked news of a user and the rankings of candidate news given by NRMS and MM-Rec. Only the first candidate news is clicked.</figcaption></figure><p>從上圖可以發現，NRMS 和 MM-Rec 都將最後的 Candidate news 給出較低的排名，因為從它的標題我們可以很容易地推斷出它與 User interests 無關。但是，NRMS 模型並未能成功推給 User 第一條 Candidate news，這篇新聞與 User clicks 第二篇 NFL 的新聞高度相關，但很難僅根據標題來衡量它們的相關性。但作者的 MM-Rec 將第一個 Candidate news 排在首位推給 User，因為它很容易根據視覺信息與 User interest 相匹配。這些結果表明了 Multi-model information 在 News rec 的有效性。</p><p>This article will be updated at any time! Thanks for your reading. If you like the content, please clip the “clap” button. You can also press the follow button to track new articles. Feel free to connect with me via <a href="https://www.linkedin.com/in/harenlin/">LinkedIn</a> or email.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=593fd099b24e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Notes] (ACL2019) LSTUR: Neural News Recommendation with Long- and Short-term User Representations]]></title>
            <link>https://haren.medium.com/notes-acl2019-lstur-neural-news-recommendation-with-long-and-short-term-user-representations-3c2ba09dea1c?source=rss-baf78943a5fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c2ba09dea1c</guid>
            <category><![CDATA[lstur]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[news-recommendation]]></category>
            <category><![CDATA[recommendations]]></category>
            <dc:creator><![CDATA[Haren Lin]]></dc:creator>
            <pubDate>Sun, 19 Jun 2022 02:27:25 GMT</pubDate>
            <atom:updated>2022-06-19T02:27:25.302Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>做 News Rec 一定不能沒聽過這篇論文，本文提供一個簡易上手 LSTUR 概念的介紹，幫助理解這過去曾是 SOTA 的模型！P.S. 本文大量中英文夾雜，是因為有些時候我認為用英文理解作者的原意會比較明確，如有造成閱讀上的不便，敬請見諒！</blockquote><h3>Paper Link</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//nvagus.github.io/paper/ACL19NewsRec.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/221cdc1823ed2f30e763a237ab099a2c/href">https://medium.com/media/221cdc1823ed2f30e763a237ab099a2c/href</a></iframe><h3>Observation</h3><p>Users usually have both long-term preference &amp; short-term interests. However, past news rec models usually learn single representations of users, which may be insufficient.</p><h3>Idea</h3><ol><li>The interests of online users in news are very diverse. (Long Term Interest)</li><li>User interests may evolve with time and may be triggered by specific contexts or temporal demands. (Short Term Interest)</li><li>Users usually have both long-term preferences and short-term interests. However, existing news recommendation methods usually learn single representations of users, which may be insufficient.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ntu66xqQqpl-6disDceEhw.png" /></figure><p>如上圖所示，如果 User 是金州勇士隊 (Golden State Warriors) 的粉絲，那麼這個用戶可能會在幾年內閱讀很多關於這支 NBA 球隊的籃球新聞。作者將這種用戶偏好稱為 Long-Term Interest。而瀏覽電影《波西米亞狂想曲》(Bohemian Rhapsody) 的新聞會導致 User 去閱讀其他相關新聞，像是：“Rami Malek Wins the 2019 Oscar”，因為“Rami Malek” 是這部電影中的重要演員，即便該 User 以前可能從未讀過有關 “Rami Malek” 的新聞。作者將這種用戶興趣稱為 Short-Term Interest。從上例可以推斷長短期的興趣對於 News Rec 來說都很重要。</p><h3>Model Architecture</h3><h4>Part1. News Encoder (= Title + Topic Encoder)</h4><blockquote><strong>News Encoder’s Title Encoder</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GuFhjZWnyElMUtlYJRPwaw.png" /><figcaption>News Encoder’s Title Encoder Cheat Sheet</figcaption></figure><p>[Step1] 透過 GloVe 把這些 Token 先轉成對應的 Word Embeddings</p><p>[Step2] 透過 CNN 來捕捉 Local Context 並學習其對應的 Word Representation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*xuKSZOuJGWnFCiGZddXnSg.png" /><figcaption>w[i−M:i+M] is the concatenation of the em- beddings of words between position i−M and i+M. C and b are the parameters of the convolutional filters in CNN, and M is the window size.</figcaption></figure><p>[Step3] 利用 Word-level Attention 來做 Attention Pooling 得到 News Representation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*ASsRXxlrOSe-8aJP9B7I9A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/204/1*_pa2OKbMpvll0jbVIjBmwA.png" /></figure><blockquote><strong>News Encoder’s Topic Encoder</strong></blockquote><p>作者從這篇新聞的 Topic 和 Sub-Topic 中學習 Representation。在 MSN 新聞中，新聞文章通常標有 Topic category (e.g. Sports) 和 Subtopic category (e.g., Golf)，以幫助定位 User interests。它們可以揭露新聞的 General &amp; Detailed topics，並傳達用戶的偏好。作者透過 id embeddings 學習這些 information：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/1*2-3G5lNaFHBHOOuwbMrvzQ.png" /></figure><p>最終把 Title, Topic, Subtopic Representation 全部 Concatenate 起來的到 News Representation：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/1*duzBeTtPEMrrgF8nB_1m2g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*qiXRbEtDOu4Mqu2WrQsOkw.png" /><figcaption>News Encoder = Title + Topic Encoder</figcaption></figure><h4>Part2. <strong>User Encoder = LTUR + STUR</strong></h4><p>The user encoder: learn representations of users from the history of their browsed news.</p><blockquote><strong>LTUR: capture the user’s consistent preferences.</strong></blockquote><ul><li>Learned from the embeddings of the user IDs, which are randomly initialized and fine-tuned during model training.</li><li>Denote u as the ID of a user and Wu as the look-up table for long-term user representation, the long-term user representation of this user is:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/181/1*O5-n-Ts7Mt5Lmm0tlpUvsw.png" /><figcaption>Long-term user representation</figcaption></figure><blockquote><strong>STUR: capture the user’s temporal interests.</strong></blockquote><ul><li>Online users may have dynamic short-term interests in reading news articles, which may be influenced by specific contexts or temporal information demands.</li><li>Learn the short-term representations of users from their recent browsing history, and use gated recurrent networks (GRU) to capture the sequential news reading patterns. And the short-term user representation is the last hidden state of the GRU network.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*RM2qaxF4QN7zg6iklqlowQ.png" /><figcaption>p.s. Apply the news encoder to obtain the representations of these browsed articles, denoted as {e1, e2, . . . , ek}. The above formula are GRU networks. σ is the sigmoid function, ⊙ is the item- wise product, Wr, Wz and Wh ̃ are the parameters of the GRU network.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/127/1*ezYiPi2qRNgbZ5f8RpeDeA.png" /><figcaption>Short-term user representation</figcaption></figure><h4>Part3. LSTUR = Combination of LTUR &amp; STUR</h4><blockquote><strong>1) LSTUR-ini: Using the LTUR to initialize the hidden state of the GRU network in the STUR model. 把 LTUR視為STUR的初始值！</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IHVHtW7sLXOEUrF5-hHKOQ.png" /><figcaption>LSTUR-ini</figcaption></figure><blockquote><strong>2) LSTUR-con: Concatenating the LTUR with the STUR as the final user representation. 把LTUR以及STUR給Concat起來拿去做後續判斷！</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pe9yuwF4gP8LFgjme14O-w.png" /><figcaption>LSTUR-con</figcaption></figure><h4>Part4. Click Predictor</h4><p>Use the simple dot production to compute the news click probability score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/1*i4nRHuBFhiIIn8gLaAyw6w.png" /><figcaption>probability score</figcaption></figure><h3>Training Details</h3><h4>Part1. Negative Sampling</h4><p>在訓練過程中會採用 Negative sampling。對於 User 瀏覽的每條新聞(a.k.a. Positive Sample)，我們從同一 Impression 中隨機抽取該 User 未點擊的 K 條新聞文章作為 Negative Samples。模型會一起預測 Positive news &amp; K 個 Negative news 的 Click prediction scores。如此一來此新聞點擊預測問題就被重新表述為 pseudo (K + 1)-way classification task。對此，Loss 的設計很自然就是 Minimize the summation of the negative log-likelihood of all positive samples！</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*-B33T2ALR8kWQcJgr4j-qw.png" /><figcaption>P is the number of positive training samples, and cn_i,k is the k-th negative sample in the same session with the i-th positive sample.</figcaption></figure><h4>Part2. Bernoulli Masking</h4><p>由於並非所有用戶都可以納入新聞推薦模型訓練 (E.g. 新的 Users)，因此在預測階段假設所有用戶在我們的模型中都有 Long-Term Representation 是不合適的。為了處理這個問題，在模型訓練階段，作者以一定的機率 p 隨機掩蓋用戶的 Long-Term Representation。當我們 Mask long-term representation 時，所有維度都設置為 0。</p><p>Thus, the long-term user representation in our LSTUR approach can be reformulated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*E8cmsZl2bTTX03cuZGDBcg.png" /><figcaption>B is Bernoulli distribution, and M is a random variable that subject to B(1, 1−p). We find in experiments that this trick for model training can improve the performance of our approach.</figcaption></figure><h3>Experiments</h3><p>首先 Datasets 的部分是採用 <a href="https://www.msn.com/en-us/news">MSN News</a> 四周的資料從 2018.12.23 到 2019.01.19。用前三週的資料 Training，其中 10% 作為 Validation，最後一週 Testing。而對於每個 Sample，作者收集過去 7 天的瀏覽歷史來學習 Short Term Representation。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*EePtq3kOu15p6c9vGfulVA.png" /><figcaption>Statistics of the dataset in our experiments</figcaption></figure><p>P.S. Other training details from the paper: The word embedding dimension is 200. The number of filters in CNN network is 300, and the window size of the filters in CNN network is set to 3. We applied dropout to each layer in our approach to mitigate overfitting. The dropout rate is 0.2. The default value of long-term user representation masking probability p for model training is 0.5. We used Adam to optimize the model, and the learning rate was 0.01. The batch size is set to 400. The number of negative samples for each positive sample is 4. These hyper-parameters were all selected according to the results on validation set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4lvT_9Da-51of5QGEj_CxQ.png" /><figcaption>The performance of different methods on news recommendation</figcaption></figure><p>簡言之，LSTUR-ini 的效果會比 LSTUR-con 略佳！</p><h3>Ablation Study</h3><h4>Effectiveness of Long- and Short-Term User Representation &amp; News Encoders in STUR</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wfo8cezlTZIUHq10cLrf7Q.png" /><figcaption>Left: The effectiveness of incorporating long-tern user representations (LTUR) and short-term user rep- resentations (STUR). Right: The comparisons of different methods in learning short-term user representations from recently browsed news articles.</figcaption></figure><h4>Effectiveness of News Title Encoders</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nyFN82UR9fWFEYRrtqnFcg.png" /><figcaption>The comparisons of different methods in learning news title representations and the effectiveness of attention machenism in selecting important words.</figcaption></figure><ul><li>Encoders using CNN outperform those using LSTM. Since local contexts in news titles are more important for learning news representations.</li></ul><h4>Effectiveness of News Topic</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b6sEAXgojJJIpT7q4cdC-Q.png" /><figcaption>The effectiveness of incorporating news topic and subtopic information for news recommendation.</figcaption></figure><ul><li>Subtopics can provide more fine-grained topic information which is more helpful for news recommendation.</li></ul><h4>Influence of Masking Probability</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qH3kKIBwnabZ-HX_1okrqw.png" /><figcaption>The influence of mask probability p on the performance of LSTUR.</figcaption></figure><p>根據上圖，LSTUR-ini 和 LSTUR-con 的結果具有相似的 Pattern。當 p 從 0 增加時，兩種方法的效能都會提高。當 p 還太小時，模型將傾向於 Overfit LTUR，因為 LTUR 有較大量的參數。當 p 太大時，兩種方法的效能都會開始下降，可能是因為 LTUR 中的 useful informations 無法有效整合。對於 LSTUR-ini &amp; LSTUR-con，p’s moderate choice (E.g. p = 0.5) 才能夠平衡 LTUR 和 STUR 的學習。</p><p>This article will be updated at any time! Thanks for your reading. If you like the content, please clip the “clap” button. You can also press the follow button to track new articles. Feel free to connect with me via <a href="https://www.linkedin.com/in/harenlin/">LinkedIn</a> or email.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c2ba09dea1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Notes] (IJCAI2021) UNBERT: User-News Matching BERT for News Recommendation]]></title>
            <link>https://haren.medium.com/notes-ijcai2021-unbert-user-news-matching-bert-for-news-recommendation-abdc1721008e?source=rss-baf78943a5fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/abdc1721008e</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <category><![CDATA[news-recommendation]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Haren Lin]]></dc:creator>
            <pubDate>Fri, 17 Jun 2022 08:15:02 GMT</pubDate>
            <atom:updated>2022-06-17T08:15:02.007Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>本文擷取論文重要片段進行筆記，個人認為這篇論文在鋪陳上/說故事上面很有邏輯，推薦有興趣的人可以翻翻原文！</blockquote><h3>論文連結 Paper Link</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//www.ijcai.org/proceedings/2021/0462.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/336eca5fdd463bac495199f13e5e712d/href">https://medium.com/media/336eca5fdd463bac495199f13e5e712d/href</a></iframe><h3>Observations &amp; Ideas</h3><ol><li>Existing recommendation methods merely learn textual representations from in-domain news data, which limits their generalization ability to new news that are common in cold-start scenarios.</li><li>Although the cold-start problem can be alleviated by pre-trained word embeddings (e.g., Word2Vec and Glove), due to their context-independency, their effectiveness may be further weaken during training with a randomly initialized downstream model.</li><li>Many of these methods represent each user by aggregating the historically browsed news into a single vector and then compute the matching score with the candidate news vector, which may lose the low-level matching signals.</li><li>Moreover, treating each news as a whole vector and encoding users and items separately may ignore some low-level matching signals (e.g., word-level relations) between users’ interests and candidate news.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A02HPJx6FmHUOrPTwMe8jw.png" /><figcaption>A negative example: several news browsed by a user (upper box) and a candidate news (lower box). Orange bars represent the important signals related with green bar that should be captured.</figcaption></figure><p>Candidate news #1 &amp; #3 具有很強的 Semantic similarity，因為它們都與電影相關。但在 Word-level 中，這些瀏覽過的新聞中 “Florida”, “America” 等詞語可能與 Candidate news 中的 “Japan” 「不匹配」，用戶實際上並不會去點擊這個新聞。因此，對於 User 而言，在 Location 上的 Word-level matching signal 並沒有很好地用於新聞推薦。</p><h3>Contribution</h3><ol><li>To the best of our knowledge, UNBERT is the first work to introduce the pre-trained BERT to capture user-news matching signals for news recommendation that takes full advantage of the out-domain knowledge.</li><li>UNBERT proposes the idea of representing user by raw text of the browsed news directly, and learns user-news matching representation at both word-level and news-level to capture multi-grained user-news matching signals through two matching modules.</li><li>Extensive experiments on the real-world dataset show that our approach can effectively improve the performance of news recommendation.</li></ol><h3>Model</h3><h4>Inputs / Embedding Layers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qcPoN8xYPWWEBftCid1CEg.png" /><figcaption>UNBERT Input Representation</figcaption></figure><p>UNBERT 輸入看起來複雜，但其實剖析後很簡單明確。對於一筆資料來說，手上會有一個 User news clicks 以及 Candidate news。</p><p>[Step1] 把所有 News (User news clicks &amp; Candidate news) 的 Texture Source 先做 Tokenization，並得到基本的 Token Embedding。他會模擬 BERT 放兩句 Sentence 的做法：[CLS] Sent1 [SEP] Sent2。這邊 Sent1 就是放 Candidate news’ tokens，Sent2 會放所有 User news clicks，因為 Clicks 會由好幾篇的 news 組成，彼此之間會再穿插一個新定義的 [NSEP] (News SEParator)。</p><p>[Step2] 取的 Segment Embedding，依照 [CLS] Sent1 [SEP] Sent2，Candidate news [CLS] 到 [SEP] 的部分都標記為 EA，其餘 User news clicks 都標記為 EB。</p><p>[Step3] Positional Embedding 的部分，在 UNBERT 的方法中沒有使用，因作者實驗後發現效果更糟。</p><p>[Step4] News Segment Embedding，用來區分不同篇 News 的 Token 的範圍。</p><p>Token, segment and position embeddings 是使用 Mask LM 進行 Pretrain 的，方法是 Randomly mask 每個 Seqeunce 中 15% 的所有 Word-piece tokens 並預測 Masked token。News segment embedding 是隨機初始化的，並在 Fine-tune 時進一步更新。最後，把 Token Embedding + Segment Embedding + News Segment Embedding 全部相加得到的 Embedding 就是 UNBERT Input Representation。</p><h4>Architecture (WLM + NLM)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ElgSa_wEMELAMnGH6NPrng.png" /><figcaption>UNBERT Model Architecture = WLM + NLM</figcaption></figure><h4>Word-Level Module (WLM)</h4><p>WLM applies multiple Transformer Layers (TL) iteratively to compute the hidden representations at each layer for each word (token) and propagate the matching signal at word(token)-level simultaneously. 簡言之，WLM 透過字與字之間的交互作用，來得到每個 Token 的 Embedding。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XsxReZMaBGdgmpDny0d5zA.png" /><figcaption>Transformer Layer c.f. Attention is all you need</figcaption></figure><h4>News-Level Module (NLM)</h4><p>NLM aggregates the word’s hidden representation of each news from WLM to the news representation, and then implement the other multiple Transformer Layers to capture the news-level matching signal. 簡言之，NLM 透過 WLM 生成的 Word Embedding 進行 Pooling，來得到每個 News 的 Embedding。</p><ul><li>wi = the hidden representation for the i-th word obtained from WLM.</li><li>nj = the j-th news representation aggregated from its sequence of words Sj where i ∈ Sj.</li></ul><p>Here, we obtain the news representation nj from the word representation wi using the following three types of aggregators: (共三種 Aggregation Methods)</p><ol><li><em>NSEP Aggregator</em>:<em> t</em>ake [NSEP] token representation as the news embedding.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*9BKbDH2zRdROGaHtPHLqOw.png" /></figure><p>2. <em>Mean Aggregator</em>: averages the words’ embedding directly to form the news embedding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/1*qNQhOZz_wb3PJ4STKrSjag.png" /></figure><p>3. <em>Attention Aggregator</em>:<em> </em>uses a light-weight attention network to learn the combination weights of the word embedding matrix w.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*_57ZJwD79Wo4eOb6lF_opg.png" /></figure><h4>Click Predictor</h4><p>Predict the probability of a user clicking a candidate news.</p><ul><li>ew = word-level matching representation. (from WLM)</li><li>en = news-level matching representation. (from NLM)</li></ul><p>Concatenate these two representation vectors as multi-grained matching representation before applying a full connection layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*CPrnFoFdQYhspKXFHV6H_A.png" /></figure><p>P.S. Training Details: In our experiments, MIND-small dataset is used to determine the parameter settings, then we train and evaluate on both small and large dataset. The <em>bert-base-uncased </em>is used as the pre-trained model to initialize the word-level module. We apply negative sampling with ratio 4 in consideration of being consistent with other baselines as well as the training effi- ciency, Adam is used for model optimization. The batch size is set to 128, the learning rate is set to 2e−5, and 2 epochs are trained. All the hyper-parameters are tuned on the validation set.</p><h3>Datasets</h3><p>資料集的部分，當然又是大家熟悉的 MIND 啦～！</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Sb9KNcaJ9Bzv0mQA2boYA.png" /><figcaption>Statistics of the datasets</figcaption></figure><h3>Experiments</h3><h4>Overall Performance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0MHk5l4fhBA6QvulV73Hjg.png" /><figcaption>The overall performance of different methods on MIND. Boldface indicates the best results (the higher, the better), while the second best is underlined. UNBERT-en△ represents the ensemble score based on UNBERT which is at the top of <a href="https://msnews.github.io/#leaderboard">https://msnews.github.io/#leaderboard</a></figcaption></figure><h4>Comparison for different aggregator in NLM</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3EDHXzjS7CPhACNpTNpRQ.png" /></figure><p>效能比較如同預期：NSEP &lt; Mean &lt; Attn Aggregator。</p><h4>Ablation Study of WLM and NLM</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kNq64Vq93IsW81YoD5VoWw.png" /></figure><ol><li>UNBERT_word and UNBERT_news achieve a pretty good performance close to the full version of UNBERT, which confirms the effectiveness of these two level matching signals.</li><li>UNBERT_news outperforms UNBERT_word, which proves that the word-level is insufficient for its weakness on capturing news structure.</li><li>The full version of UNBERT performs best, which tells that the multi-grained matching signal is necessary for news recommendation.</li></ol><h4>Effectiveness on Cold Start</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_7I3zNUPpKmcuBZsGf74lw.png" /></figure><p>這部分探討不同新聞推薦方法對 Unseen news 的 performance。作者將 MIND-small 依照天數切割成 7 組，Train &amp; Validation 在前三天，後面四天做 Testing。如上圖，UNBERT 的表現 Consistently 超越其他模型！更直得注意的是，隨著時間的推移，Cold/Unseen news 越來越多，UNBERT 雖然效能開始往下掉 (especially from 11/13/2019 to 11/14/2019)，但幅度並未像其他模型來的那麼誇張，某種程度上證明了 UNBERT 在處理 Cold-Start 的優勢。</p><p>This article will be updated at any time! Thanks for your reading. If you like the content, please clip the “clap” button. You can also press the follow button to track new articles. Feel free to connect with me via <a href="https://www.linkedin.com/in/harenlin/">LinkedIn</a> or email.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abdc1721008e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Notes] (SIGIR2021) Empowering News Recommendation with Pre-trained Language Models]]></title>
            <link>https://haren.medium.com/notes-sigir2021-empowering-news-recommendation-with-pre-trained-language-models-f5f6aa6bea3f?source=rss-baf78943a5fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5f6aa6bea3f</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[news-recommendation]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Haren Lin]]></dc:creator>
            <pubDate>Thu, 16 Jun 2022 07:17:23 GMT</pubDate>
            <atom:updated>2022-06-16T07:22:45.114Z</atom:updated>
            <content:encoded><![CDATA[<h3>Paper Link</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2104.07413.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/1965f1f22baab3757483b23532e7a75c/href">https://medium.com/media/1965f1f22baab3757483b23532e7a75c/href</a></iframe><h3><strong>Introduction</strong></h3><p>News Modeling (News Encoding) 是 News Rec 中重要的步驟，因為它是了解 Candidate News 內容的核心技術，也是從 News clicks 中推斷 User interests 的先決條件。現有的新聞推薦方法通常基於傳統的 NLP 模型對新聞文本進行建模。E.g. DKN 提出使用 knowledge-aware CNN 從新聞標題中的 words and entities 的 Embedding 中學習 News representation。NRMS 提出使用 Multi-Head Self-Attention 從新聞標題中學習 News representation。但是這些 Shallow models 很難理解新聞文本中的 deep semantic information。此外，他們的模型僅從新聞推薦任務中的監督中學習，這對於捕獲 semantic information 可能不是最佳的。</p><p>預訓練語言模型 Pre-trained Language Model PLM 藉其強大的文本建模能力在 NLP 中取得了巨大成功。他與一般在特定任務中直接使用標記數據進行訓練的傳統模型不同，PLM 通常首先通過 Self-supervise 在 Large unlabeled corpus 上進行 Pre-train，所以可以為下游任務的 Fine-tune 提供更好的 Initialization point。此外，PLM 通常更深，有更大量參數，因此 PLM 可能在對新聞文本中復雜的 Contextual information 進行建模具有更大的能力，可能可以使新聞推薦的 News modeling 有所進步。</p><h4>In short</h4><ol><li>Existing news recommendation methods mainly model news texts based on traditional text modeling methods, which is not optimal for mining the deep semantic information in news texts.</li><li>Pre-trained language models (PLMs) are powerful for natural language understanding, which has the potential for better news modeling. However, there is no public report that show PLMs have been applied to news recommendation.</li></ol><p>P.S. 作者利用 PLM 來增強新聞推薦的效能，同時部署到 Microsoft 新聞平台，並在英語和全球市場的點擊量和瀏覽量方面取得了顯著提升。</p><h3>Source Code</h3><p><a href="https://github.com/wuch15/PLM4NewsRec">GitHub - wuch15/PLM4NewsRec</a></p><h3>Model Architecture</h3><p>首先先簡單介紹現在 News Rec 最常見且 General 的 Framework，如下圖。Framework 可以分為三個部分：(1) News encoder (2) User Encoder (3) Clik Predictor。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*nx3ZDrIknPiDhYqfMa6mQA.png" /><figcaption>A common framework of news recommendation</figcaption></figure><p>(1) News encoder：假設用戶有 T 筆歷史點擊新聞 [D1, D2, …, DT]，以及現在手上的 Candidate news Dc，News encoder 會把它轉成 News Embedding 分別表示為 [h1, h2, …, hT] 和 hc。News encoder 可通過各種 NLP 模型來實現 e.g. CNN / Self-attention。</p><p>(2) User encoder：接收 Clicked news embedding sequence 作為輸入，並輸出一個總結 User interests 的 User embedding u。它也可以通過各種模型來實現，E.g. GRU / Attention / MHA。</p><p>(3) Click predictor：將用戶嵌入 u 和 hc 作為輸入，並通過評估它們的 Relevance 來計算點擊分數 Click Score。它可以透過 Dot product / NN / FM等各種方法來實現。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G_piI1dz5W3YdspgXhPP2A.png" /><figcaption>The framework of PLM empowered news recommendation.</figcaption></figure><p>接下來，我們將 PLM 套用在 News encoder 的部分，如上圖。使用 Pre-trained LM 來捕獲新聞文本中的 Deep contexts，並使用 Attention Pooling 來得到 Representation。具體作法為：將輸入新聞文本 [w1, w2, …, wM] 透過 PLM 轉換為 Token embedding，然後通過 Transformer layers 學習 Token Hidden representation [r1, r2, …, rM]。接著使用 Attention network 將 Hidden representation 總結為一篇新聞的 News Embedding。</p><p>P.S. 訓練一樣會使用 Negative sampling 並搭配 Cross entropy loss。</p><h3>Datasets</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*zW4sYRjkzpXXde4Cl6LmeA.png" /><figcaption>Detailed statistics of the two datasets</figcaption></figure><p>實驗的資料及包含兩個：第一個是 <a href="https://msnews.github.io/">MIND</a>，它是一個用於 Monolingual 新聞推薦的英文資料集。包含 6 週內的 100 萬個用戶在 Microsoft News 上的 Click Logs。第二個是我們從 2020.12.01 到 2021.1.14 在 MSN News 平台上，來自 7 個國家使用不同語言的用戶，收集的 Multilingual 新聞推薦資料集，命名為 Multilingual (Market language codes: EN-US / DE-DE / FR-FR / IT-IT / JA-JP / ES-ES / KO-KR) 。 作者在每個 Market 隨機抽取 200,000 條 Impression，最後一週的 logs 用於 testing，其餘用於 training &amp; validation (9:1)。</p><p>P.S. Training Details from the paper: In our experiments, we used the “Base” version of different pre-trained language models if not specially mentioned. We fine-tuned the last two Transformer layers because we find there is only a very small performance difference between fine-tuning all layers and the last two layers. We used the titles of news for news modeling. We used Adam as the optimization algorithm and the learning rate was 1e-5. The batch size was 128. These hyperparameters are developed on the validation sets. We used average AUC, MRR, nDCG@5 and nDCG@10 over all impressions as the performance metrics. We repeated each experiment 5 times independently and reported the average performance.</p><h3>Experiments</h3><p>實驗的部分主要被切割成兩個區塊，第一個是 Monolingual 的 MIND。第二個則是在 Multilingual 上比較。</p><h4>Part1. MIND</h4><p>作者想驗證基於 PLM 的模型在 Monolingual News Rec 中的有效性。比較了幾種最近的新聞推薦方法，包括 EBNR, NAML, NPA, LSTUR, NRMS 及其由不同 PLM 作為 News encoder 的變形，包括 BERT, RoBERTa, UniLM，結果如下表。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*D-cZYGtmdvN_3HUL_OXJxg.png" /><figcaption>Performance of different methods on MIND</figcaption></figure><p>從表格可以發現結合 PLM 可以 Consistantly 提高模型的效能，因為 PLM 比從零開始學習的 Shallow LM 具有更強的文本建模能力。而且，基於 RoBERTa 的模型優於基於 BERT 的，可能是因為 RoBERTa 的 Hyper-parameter 設定比 BERT 更好，且在更大的 Corpus 上進行了更長時間的 Pre-train。此外，使用 UniLM 的模型有最好的表現，因為 UniLM 可以在 Text understanding &amp; Generation tasks 中利用 Self-supervision information，這有助於學習更高質量的 PLM。</p><p>P.S. 上述實驗提及的論文如下：</p><ol><li>EBNR — Embedding-based News Recommendation for Millions of Users</li></ol><p><a href="https://dl.acm.org/doi/10.1145/3097983.3098108">Embedding-based News Recommendation for Millions of Users | Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining</a></p><p>2. NAML — Neural News Recommendation with Attentive Multi-View Learning</p><p><a href="https://arxiv.org/abs/1907.05576">Neural News Recommendation with Attentive Multi-View Learning</a></p><p>3. NPA — Neural News Recommendation with Personalized Attention</p><p><a href="https://arxiv.org/abs/1907.05559">NPA: Neural News Recommendation with Personalized Attention</a></p><p>4. LSTUR — Neural News Recommendation with Long- and Short-term User Representations</p><p><a href="https://aclanthology.org/P19-1033/">Neural News Recommendation with Long- and Short-term User Representations</a></p><p>5. NRMS — Neural News Recommendation with Multi-Head Self-Attention</p><p><a href="https://aclanthology.org/D19-1671/">Neural News Recommendation with Multi-Head Self-Attention</a></p><p>6. BERT — Pre-training of Deep Bidirectional Transformers for Language Understanding</p><p><a href="https://arxiv.org/abs/1810.04805">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a></p><p>7. RoBERTa — A Robustly Optimized BERT Pretraining Approach</p><p><a href="https://arxiv.org/abs/1907.11692">RoBERTa: A Robustly Optimized BERT Pretraining Approach</a></p><p>8. UniLM — Unified Language Model Pre-training for Natural Language Understanding and Generation</p><p><a href="https://arxiv.org/abs/1905.03197">Unified Language Model Pre-training for Natural Language Understanding and Generation</a></p><h4>Part2. Multilingual</h4><p>作者想驗證基於 PLM 的模型在 Multilingual News Rec 中的有效性。比較了 EBNR, NAML, NPA, LSTUR, NRMS 與不同 Multilingual LM 的效果。Multilingual LM 包含 MUSE, Unicoder, InfoXLM，並同時比較基於 MUSE 的 Monolingual LM 對於每個 Market 獨立訓練的效果 (denoted as Single)。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*a1V8dOrKg7GSnkMr6yQFpw.png" /><figcaption>Performance of different methods on Multilingual</figcaption></figure><p>首先，從表格可以知道 Multilingual 通常優於獨立學習的 Monolingual (Single)，可能是因為不同的語言通常有一些 inferent relatedness，且不同國家的用戶也可能有一些相似的興趣。因此，使用 Multilingual data 做 Jointly training 可以幫助學習更準確的推薦模型。</p><p>此外，基於 Multilingual PLM 會比使用 MUSE Embedding 的方法來的更好，因為 PLM 在捕獲 Complicated multilingual semantic information 更強大。此外，InfoXLM 表現比 Unicoder 更好，可能是因為 InfoXLM 使用比 Unicoder 更好的 Contrastive Pre-training Strategies。</p><p>P.S. 上述實驗提及的論文如下：</p><ol><li>MUSE — Multilingual Unsupervised and Supervised Embeddings</li></ol><p><a href="https://research.facebook.com/downloads/muse-multilingual-unsupervised-and-supervised-embeddings/">MUSE: Multilingual Unsupervised and Supervised Embeddings - Meta Research | Meta Research</a></p><p>2. Unicoder — A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks</p><p><a href="https://arxiv.org/abs/1909.00964">Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks</a></p><p>3. InfoXLM — An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training</p><p><a href="https://arxiv.org/abs/2007.07834">InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training</a></p><h3>Other Comparison</h3><h4>Part1. Influence of Model Size</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*KD-aamqObNOXsjax27UKwQ.png" /><figcaption>Influence of the size of PLMs</figcaption></figure><p>作者也探討 PLM 大小對推薦效果的影響，比較 NAML 和 NRMS 與不同版本 BERT 的性能，包括 BERT-Base (12 layers) / BERT-Medium (8 layers) / BERT-Small (4 layers) / BERT-Tiny (2 layers)。發現使用具有更多參數的更大 PLM 通常會產生更好的推薦效能，畢竟較大的 PLM 通常在捕獲新聞的 deep semantic information 具有更強的能力。但由於大型 PLM 對於 Online inference / Online application 來說過於繁瑣，因此實務上比較偏好 PLM with base version。</p><h4>Part2. Influence of Different Pooling Methods</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*87F-QHNhvCLpL_oPQ3zxEQ.png" /><figcaption>Influence of the pooling methods</figcaption></figure><p>不僅比較模型大小，作者還探討使用不同的 Pooling Method 來得到 News representation 在效能上的差異。 (1) 使用 [CLS] 的 Representation 作為 News embedding (2) Average Pooling，把所有 Tokens’ Hidden Representation 進行平均得到 News Representation (3) Attention Pooling，讓每個 Token 得到各自的 Attention weight，在用 Weighted Sum 得到 New Representation。上圖的結果跟我們預想的效能排序一樣：[CLS] &lt; Avg &lt; Attn。</p><h4>Part3. Visualization of News Embedding</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*85WfhF57a3GBFvmELQmBSg.png" /><figcaption>Visualization of news embeddings learned by NRMS and NRMS-UniLM.</figcaption></figure><p>作者也透過 t-SNE 視覺化 NRMS 和 NRMS-UniLM 來探討透過 Shallow model &amp; PLM 學習的 News Embedding 差異。有個有趣的現像：NRMS-UniLM 學習的 News Embedding 比起 NRMS 更 Discriminative，且有更清楚獨立的 Clusters。因為 PLM 可以 disentangle 不同類型的新聞，以實現更好的 User interest modeling &amp; News matching。這些結果代表的意義是：Deep PLM 比Shallow LM 在學習 Discriminative text representations 上有更大的能力，有利於準確的新聞推薦。</p><p>This article will be updated at any time! Thanks for your reading. If you like the content, please clip the “clap” button. You can also press the follow button to track new articles. Feel free to connect with me via <a href="https://www.linkedin.com/in/harenlin/">LinkedIn</a> or email.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5f6aa6bea3f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Notes] (SIGIR2022) Is News Recommendation a Sequential Recommendation Task?]]></title>
            <link>https://haren.medium.com/notes-is-news-recommendation-a-sequential-recommendation-task-88945624d80d?source=rss-baf78943a5fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/88945624d80d</guid>
            <category><![CDATA[recsys]]></category>
            <category><![CDATA[news-recommendation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <dc:creator><![CDATA[Haren Lin]]></dc:creator>
            <pubDate>Wed, 15 Jun 2022 07:54:04 GMT</pubDate>
            <atom:updated>2022-06-16T02:57:55.237Z</atom:updated>
            <content:encoded><![CDATA[<h3>Paper Link</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2108.08984.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4801d2e631585d25c605fa523712c4b/href">https://medium.com/media/e4801d2e631585d25c605fa523712c4b/href</a></iframe><h3>Observation</h3><ul><li>News recommendation is often modeled as a sequential recommendation task, which assumes that there are rich short-term dependencies over historical clicked news.</li><li>However, <strong><em>in news recommendation scenarios users usually have strong preferences on the temporal diversity of news information and may not tend to click similar news successively</em></strong>, which is very different from many sequential recommendation scenarios such as e-commerce recommendation.</li></ul><p>作者在本文提出一個觀察：他認為新聞的推薦與那些商業系統的推薦並不具有相同的特性，新聞推薦場景中的使用者，user 通常對新聞信息的時間多樣性有很強的偏好，可能不會傾向於連續點擊相似的新聞（本文重要假設）。對此，作者設計了一些實驗來挑戰 News Rec 並非 Sequential Rec，我們接著往下看。</p><h3>Idea</h3><p>在過去的那些模型中，像是 NRMS, LSTUR, GRU4Rec，方法的一個核心假設是對 historical behaviors 有豐富的 short-term dependency，未來的行為也可能與最近的 historical behaviors 相關。儘管這個假設被許多順序推薦場景廣泛使用，如電子商務推薦、電影推薦，但作者發現它在新聞推薦場景中可能無效，因為用戶對新聞信息的時間多樣性 / temporal diversity (i.e., novelty) 的偏好。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*qnf2S3EMOWpg-2W-H-4gBg.png" /><figcaption>A general framework of sequential news recommendation.</figcaption></figure><p>怎麼說呢？作者有提出數據作佐證：MIND 新聞資料集中，只有 7.2% 的 adjacently clicked news 是屬於相同主題的。如果我們隨機點擊的話，比例則是 7.9%。此外，只有 0.04% 的 adjacently clicked news 有至少一個相同的 entity。這樣的數據顯示 News Rec 的特性是 Diverse 而並非 Similar，這就與過去模型的 Sequential Rec 假設相互牴觸。因此作者認為如果把 News Rec 視為一個 Sequential Rec 的問題所得到的解會是 Suboptimal 的。</p><p>作者嘗試解決這樣的問題，提出了一個模型稱為 TempRec: Temporal diversity-aware news recommendation method。這個模型傾向於推薦那些跟近期點擊的新聞不同主題的 Candidate news。</p><p>更具體地說，作者使用 shared order-agnostic Transformer 從所有歷史新聞點擊中學習 Global interest embedding，並從幾個最近的點擊中學習 Recent interest embedding。Click score 是通過 Candidate News 和 Global interest embedding 之間的 Positive relevance 以及 Candidate News 和 Recent interest embedding 之間的 Negative relevance，透過 Learnable weight 加權求和而得的。實驗結果顯示 TempRec 可以勝過許多現有的新聞推薦方法。</p><h3>Datasets</h3><p>首先我們先了解一下實驗的資料及有哪些，還有他各自對應的 Statistics。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ho1PRUuV94wq5R2T1hSbUw.png" /><figcaption>Statistics of <em>MIND </em>and <em>News </em>datasets</figcaption></figure><p>第一個 Dataset 是在近期做 News Rec 大家熟知的 MIND。第二個資料集則是 News，他是從微軟官方自己商業新聞平台裡面撈出來的 100萬條 news impression logs，從 2020.10.17 到 2021.01.29 的資料。</p><h3>Experimental Details</h3><p>在作者的實驗中，都使用 Glove Word Embedding 進行初始化。所有模型的 hidden dimension 都是 400，並使用 Adam (lr=1e-4) 作為 Optimizer。而只指標的部分使用 AUC、MRR、nDCG@5 和 nDCG@10，並將實驗獨立進行五次取平均。</p><h3>Performance Comparison</h3><p>本文的實驗分成兩大塊，首先第一部份是要用實驗證明 News Rec 不是 Sequential Rec。</p><p>這邊使用三個 Baseline model: (1) LSTUR — 用 GRU 對 Short-term interest 進行建模，並使用 User ID embedding 對 Long-term interest 建模。(2) DAN — 用 LSTM 和 Casual self-attention network 的組合來模擬 User interest。(3) NRMS，用 MHA 來模擬 User interest。</p><p>其中在 NRMS 的部分，模型的設計並不包含 Positional Encoding。對此，作者在 NRMS 上面多做了兩個變形，分別為 NRMS+PE(Positional Encoding) &amp; NRMS+CM(Casual self-attention Mask)。</p><p>為了進一步探討 Sequential information 是否有利於 News rec，作者比較了所有使用 inverse 反向以及 random 隨機打亂 clicked news sequence 這兩種變形。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NpcsQ7vx6arGJVkJu__oRA.png" /><figcaption>Performance of different methods. PE denotes position embedding and CM denotes casual mask.</figcaption></figure><p>從上方圖表中可以得到一些小結論。</p><p>首先，與 Order-agnostic NRMS 模型相比，結合 Positional Encoding 不會使效能增加，且 Casual self-attention 不如 Bidirectional self-attention (其實這部分我是無法看出來QAQ)。這樣的現象表示 Positional Information 在 News Rec 場景中對於 User interest 的建模並不是很重要，對點擊的新聞序列中的過去和未來信息進行完全建模才會有比較好的效果。</p><p>第二，作者發現使用 inverse clicked news sequence 並不會導致顯著的效能差異。由於在 RNN 模型中通常來講會更注重於 Current steps，現在卻出現此現象代表在新聞推薦中，latest clicked news 在預測 future clicks 時，並沒有比 earlier clicks 提供更多 information，這可能是由於 users’ temporal diversity preferences 及 users’ stable long-term interest。</p><p>此外，有趣的是，隨機打亂點擊序列甚至可以稍微提高順序敏感模型的效果。這可能是因為模型可以更好地從打亂序列中捕獲 Global user interest，以幫助更準確地預測未來的點擊。</p><p>其實從以上這些結果應該不難看出：News Rec 可能不適用於 Sequential Rec 建模，因為 sequential information &amp; short-term dependency 在新聞推薦中都沒有起重要作用。</p><h3>TempRec Model Architecture</h3><p>從上個部分的實驗結果與分析可以知道 News Rec 並不適用 Sequential Rec。因此，作者提出一個 novel temporal diversity-aware news recommendation method (TempRec)，來嘗試建模，模型可以考慮新聞推薦的 temporal diversity 時間多樣性，以做出更準確的推薦結果。TempRec 的架構如下圖。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ehAxtKZ_nvpp89M4uwrhg.png" /><figcaption>The architecture of <em>TempRec</em></figcaption></figure><p>[Step1] 先把 Clicked News [D1, …, DN] 過 News Encoder 得到 [r1, …, rN]。同時間 Candidate News 也會過 News Encoder 得到 Candidate news embedding, rc。</p><p>[Step2] TempRec 中有兩個 Order-agnostic Transformers。一個用於處理整個 clicked news sequence 來捕捉 Global user interest，另外一個會處理最近的 K 個 clicked news 來捕捉 Recent user interest。我們把透過這兩個 Transformers 作用後得到的 Hidden news representation 表示為 H = [h1, h2, …, hN ] 以及 H’ = [h’_N-K+1, …, h’N ]。</p><p>[Step3] 分別使用各自的 Attention network 來選擇這兩個 Sequence (i.e. H, H’) 重要的 News clicks，並以 Attention Pooling 得到 Global user interest embedding, ug 以及 Recent user interest embedding, ur。</p><p>[Step4] 由於 Future news clicks 可能與 Recent news clicks 不同，作者提出了一個 temporal diversity-aware click prediction method 來協助更精準的預測。首先會算出 (a) Global Relevance Score — yg_hat = ug·rc，表示 Candidate news 跟 Global user interest 的匹配程度，以及 (b) Recent Relevance Score — yr_hat = ur·rc，表示 Candidate news 跟 Recent news clickes 之間的相關性。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*_6oTmXnfhiIz7mEnOGbfpg.png" /><figcaption>(a) Global Relevance Score</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/284/1*qMOkrmuFgaEMKEPt9DlXhg.png" /><figcaption>(b) Recent Relevance Score</figcaption></figure><p>[Step5] 把 yg_hat &amp; yr_hat 做 Linear Combination 的到 Unified Click Score。(W is learnable)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*2EGwBKZJgJCc3p8qoPSJdA.png" /><figcaption>Unified Click Score</figcaption></figure><p>通過這種方式，模型會鼓勵推薦與最近點擊新聞不同的新聞，從而更好地滿足用戶對新聞信息 temporal diversity 時間多樣性的需求。</p><h3>TempRec Experiments</h3><p>接著看看本文實驗的第二塊，這部分在驗證 TempRec 的有效性。</p><p>由於 TempRec 是一個 General framework 且與許多不同的 News encoder 兼容，因此作者通過使用與 LSTUR、DAN 和 NRMS 相同的 News encoder 來比較 TempRec 的性能。下圖是把 TempRec 運用在兩個不同 Datasets 上的結果。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VshqqBnquFchWTBWbFYFpw.png" /><figcaption>Effectiveness of <em>TempRec</em>.</figcaption></figure><p>從上圖能發現 TempRec 可以提高各種不同方法的效果，且進一步的 t-test 表示改進是 Significant (p &lt; 0.05)。這些結果表示：考慮用戶新聞點擊行為的 Temporal diversity 可以幫助更準確地預測 Future news clicks。此外，我們在 MIND 和 News 數據集上發現參數 w 分別為 0.075 和 0.083，這表示 Recent news clicks 與 Future news clicks 有輕微的負相關。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R8G-xdK5HFYaIVXXcPbtJg.png" /><figcaption>Influence of the hyperparameter K.</figcaption></figure><p>作者也研究了 Hyper-parameters K 對模型效能的影響，如上圖，發現 K = 3 是兩個數據集的最適選擇。這可能是因為當 K 太小時無法對 future and past clicks 的 Temporal diversity 完全建模，而當 K 太大時，Relevant clicks 可能無法有效匹配。</p><h3>Conclusion</h3><blockquote>Whether news recommendation is suitable to be modeled as sequential recommendation?</blockquote><p>針對上面這個問題，我想答案呼之欲出。透過這篇論文，我認為 News Rec 應該會掀起一個很大的變革。</p><p>This article will be updated at any time! Thanks for your reading. If you like the content, please clip the “clap” button. You can also press the follow button to track new articles. Feel free to connect with me via <a href="https://www.linkedin.com/in/harenlin/">LinkedIn</a> or email.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88945624d80d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Notes](EMNLP2019) NRMS: Neural News Recommendation with Multi-Head Self-Attention]]></title>
            <link>https://haren.medium.com/notes-nrms-neural-news-recommendation-with-multi-head-self-attention-99ad5e9bd320?source=rss-baf78943a5fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/99ad5e9bd320</guid>
            <category><![CDATA[news-recommendation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[microsoft]]></category>
            <dc:creator><![CDATA[Haren Lin]]></dc:creator>
            <pubDate>Fri, 27 May 2022 07:02:07 GMT</pubDate>
            <atom:updated>2022-06-17T06:25:26.841Z</atom:updated>
            <content:encoded><![CDATA[<h3>Paper Link</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//wuch15.github.io/paper/EMNLP2019-NRMS.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/8968fa9bd8a2a09077577f14ebbbf5bd/href">https://medium.com/media/8968fa9bd8a2a09077577f14ebbbf5bd/href</a></iframe><h3>Idea</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*rfbOQ5O1Hzt02bSijZ_xow.png" /><figcaption>Several news browsed by an example user. Orange and green dashed lines represent the interactions between words and news respectively.</figcaption></figure><p>審視這篇論文作者的論點：</p><ol><li>Interaction between words in news title are important for understanding the news. 文章標題的字與字之間的交互對於這個新聞的理解很重要。怎麼說呢？E.g. Rockets trade Carter-Williams to the Bulls. 火箭隊將CW交易到公牛隊。「Rockets」跟「Bulls」其實是有很強的連結性的，都是球隊的名字。</li><li>Different news articles browsed by the same user may also have relatedness. 同一個user瀏覽的不同新聞文章也可能具有相關性。怎麼說呢？E.g. 第二篇文章其實跟第一跟第三篇有關係。</li><li>Different words may have different importance in representing news. 不同的詞在表示新聞時可能具有不同的重要性。E.g. 第一篇：「NBA」的重要性就比2018來得高。</li><li>Different news articles browsed by the same user may also have different importance in representing the user. 同一用戶瀏覽的不同新聞文章在代表用戶方面也可能具有不同的重要性。怎麼說呢？E.g. 很明顯前三篇新聞對於 User interest 比第四篇重要。</li></ol><p>綜合以上的論述，作者會根據這四個點進行強化來協助 User interests 的建模。</p><h3>Model Architecture</h3><p>那接下來我們就看看作者是怎麼處理這個 issue 的，以下是模型的大架構。模型一樣會有 News Encoder, User Encoder, &amp; Click Predictor 的部分。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*voo393NuWzk-3AYX5fU1kw.png" /><figcaption>NRMS Model Architecture</figcaption></figure><h4>(I) News Encoder</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XQw3wfQ9_jHY9zjAmDn11Q.png" /></figure><p>News Encoder 可以分為三個 Parts：</p><p>第一部分為 Word Embedding Layer，他的工作是把單一一篇 News title 的字，假設長度為M，把他們全部轉換為 Word embedding 向量表示，本篇論文使用的方法是 GloVe embeddings。</p><p>第二部分為 Word-Level MHA，這邊回顧的是剛才作者提出的假設 －字與字之間的交互作用對於新聞向量的表示很重要，我們用 MHA 來得到他們Contextualized Embeddings，這個 <strong><em>alpha_i_j</em></strong> 就是第 i 個字與第 j 個字之間交互的 Attention score，小 k 是代表第幾個 head。Qk 跟 Vk 就是第 k 個 Head 中的 projection parameters，注意力分數的來源就是把 i-th 字的 Embeddings乘 Query 再乘以 j-th 字的 Embeddings 然後過 Soft-max。有了分數之後我們再去跟每個字的 Embeddings 做 Weighted Sum，且乘上 Value Matrix 得到代表第 k 個 Head 在第 i 個字上面的 Embeddings，最後把所有 Head 的輸出 Embeddings 給 Concat 起來得到這個字的新的 Embedding <strong><em>hw_i (w means word, i = i-th word in news title)</em></strong>。</p><p>第三部分為 Additive Word Attention，呼應前面作者提出的假設 － 新聞標題中不同字有不同的重要性，於是我們這邊就先把第 i 個字的 Word Embeddings 過一個 Linear transform 以及 Hyperbolic tangent 的非線性轉換，再乘上 Query term 得到暫時的分數 <strong><em>ai</em></strong>，並把這些分數去過softmax，得到 Word-level attention score <strong><em>alpha_i</em></strong>，最後把他們與 Word embeddings 做 Weighted sum 得到當前這個 News 的 News representation <strong><em>r</em></strong>。</p><h4>(II) User Encoder</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LWGGX7g8h14x2S8WI1hjew.png" /></figure><p>接下來是 User encoder，假設我們取歷史長度為 N 筆資料，這些 News 我們都會用 r1 ~ rN 表示。</p><p>第一部分為 News-Level MHA，我們要算出 News-level 的 attention score，代表新聞跟新聞之間彼此的相關性，<strong><em>beta_i_j</em></strong> 為這個使用者的歷史紀錄中，第 i-th 篇新聞與第 j-th 篇新聞之間的注意力分數。有了分數之後我們再去跟每個新聞的 Embeddings 做 Weighted Sum，且乘上 Value Matrix 得到代表第 k 個 Head 在第 i 個新聞上面的 Embeddings，最後把所有 Head 的輸出 Embeddings 給 Concat 起來得到這個新聞的新的 Embedding <strong><em>hn_i (n means news, i = i-th news in clicked history)</em></strong>。</p><p>第二部分為 Addictive News Attention，這也跟 News encoder part3 在做的事情很像，不同的新聞對於同一個 User 的表示有不同的權重/重要性，一樣過一個 Attention Network 並把分數過 Soft-max，最後把得到的權重跟各自的 News 做Weighted sum 得到 User representation <strong><em>u</em></strong>。</p><h4>(III) Click Predictor</h4><p>Predict the probability of a user clicking a candidate news via inner-product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gu_gXQ3PdESYutBacGhIYg.png" /></figure><h3>Model Training</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8TMhRK6rGdoYaeCu5ZvZHg.png" /></figure><p>Other Details: In the experiments, the word embeddings are 300-dimensional and initialized by the Glove embedding. The self-attention networks have 16 heads, and the out- put of each head is 16-dimensional. The dimen-sion of the additive attention query vectors is 200. Following, the negative sam- pling ratio K is 4. Adam is used for model optimization. We apply 20% dropout to the word embeddings to mitigate over-fitting. The batch size is 64. These hyperparameters are tuned on validation set. We conducted experiments on a machine with Xeon E5–2620 v4 CPUs and a GTX1080Ti GPU. We independently repeated each experiment 10 times and reported average results in terms of AUC, MRR, nDCG@5 and nDCG@10.</p><h3>Experiments</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*naW4z1mZtyX-22qTH_Miqg.png" /><figcaption>MSN Logs Dataset Statistics</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*CZYIZCvfT_6TdqmHIxNDsQ.png" /><figcaption>The results of different methods. The improvement is significant at p &lt; 0.01.</figcaption></figure><p>作者在 MSN Logs 上面做了實驗，實驗資料的資訊大概如上表。那從表格我們可以知道一些資訊：</p><p>(1) NN-Based 的模型比傳統的模型像是 Lib-FM 之類的效能來的好，作者認為 NN 的模型相較於傳統 Matrix Factorization (MF) 能學到比較好的 Representation。</p><p>(2) DL-based的模型中，能夠學習到 News 之間相互關係的模型，表現較佳。像是 Conv3D, GRU, NRMS。畢竟了解 News 之間的相互關係對於 User representation 的表達較好。</p><p>(3) NRMS 的確表現不錯，作者表示 Attention 佔了很大的功勞，有利於 User &amp; news representation。</p><h3>Ablation Study</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z6xZAaOdHThpoyW2r3h-xQ.png" /></figure><p>作者也有做了一些 Ablation Test 來強調他們所做的 Attention 的有效性！</p><p>先看左邊的(a)圖，作者認為 Word-level Attention 很有用，因為對字與字之間的交互作用來建模以及選出重要的字對於 News embedding representation 有很大的幫助。那當然 News-level Attention也很有用，選出新聞與新聞之間得交互作用對於 User embedding representation 也有很大的幫助。如果同時將這兩個結合，效果會很更好。</p><p>再來是看隔壁的圖(b)，作者表示 Self-Attention 其實很有用，原因就如同剛才所述，字與字或是新聞與新聞之間的作用很重要。那 Additive Attention 也能提供 Performance improvement，畢竟從不同的字之間挑出重要的字，或是從不同的新聞挑出重要的，能幫忙學到更好的 User / News embedding。</p><h3>Conclusion</h3><ol><li><strong><em>Attention</em></strong> is all you need.</li><li>In the future, it is suggested to include positional information of words and news.</li><li>Explore how to efficiently incorporate multiple kinds of news info, especially long sequence.</li></ol><p>This article will be updated at any time! Thanks for your reading. If you like the content, please clip the “clap” button. You can also press the follow button to track new articles. Feel free to connect with me via <a href="https://www.linkedin.com/in/harenlin/">LinkedIn</a> or email.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99ad5e9bd320" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Notes] (SIGIR2022) CAUM: News Recommendation with Candidate-aware User Modeling]]></title>
            <link>https://haren.medium.com/notes-caum-news-recommendation-with-candidate-aware-user-modeling-7035c18cc95d?source=rss-baf78943a5fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/7035c18cc95d</guid>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[news-recommendation]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Haren Lin]]></dc:creator>
            <pubDate>Tue, 17 May 2022 08:00:03 GMT</pubDate>
            <atom:updated>2022-06-16T09:47:26.345Z</atom:updated>
            <content:encoded><![CDATA[<h3>Paper Link</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2204.04726.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/e061da823a580ffddf33fbcca95ec89d/href">https://medium.com/media/e061da823a580ffddf33fbcca95ec89d/href</a></iframe><h3>Observation</h3><p>Existing methods first model user interests and candidate news content <strong>separately</strong> and then use their representations for interest matching. In these methods, user interests are modeled in a candidate-agnostic way. However, each user usually has multiple interests, and it may be difficult to accurately match candidate news with a specific user interest if candidate news is not considered in user modeling.</p><p>簡單講：或許把 Candidate News 一起抓進來做 User Representation，會比直接做 Encode 效果來得好！</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ea3DQE1Y3HqUQBBthNPgdw.png" /><figcaption>Existing Methods (Left: NRMS; Right: NPA)</figcaption></figure><h3>Idea</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*amYDJrCGCDOvu3G-uHT4Sw.png" /></figure><p>審視這篇論文作者的論點：</p><ol><li>User usually has multiple interests. However, a candidate news usually only matches a small part of them. 從上圖可以看出 User 有多重興趣，包含政治、音樂、體育、旅遊。4th Candidate news 只匹配到 User 的政治興趣，與該用戶的其他興趣（如音樂和體育）相關性較低。因此，如果在用戶建模中不考慮候選新聞信息，則可能難以準確匹配候選新聞。</li><li>Local contexts of users’ news click behaviors are useful for inferring short-term user interests. 從上圖 12th &amp; 13th clicked news 之間的相關性推斷出用戶最近對威斯康辛州旅行的短期興趣。</li><li>Long-range relatedness between users’ historical clicks also provides rich information to model long-term user interests. 從 5th &amp; 10th clicked news之間的長期相關性推斷出用戶對音樂的長期興趣。</li></ol><p>Thus, understanding both short- and long-term user interests is important for accurate news recommendation.</p><p>看到作者這句話，我第一直覺想到的就是 <a href="https://nvagus.github.io/paper/ACL19NewsRec.pdf">LSTUR</a> 這篇論文提出的模型，稍微看了一下這篇 CAUM，其實他就是 LSTUR 的進化版！</p><h3>Recap: LSTUR</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fRVWo6EYb6pSmF5ljW8msg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r9izpxvIZutWIpv5QxC19A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ajO30gM1AUkBusUICNrdsA.png" /></figure><h3>Model Architecture</h3><p>Candidate-Aware User Modeling 模型整體架構主要分成三個區塊：(1) Candi-SelfAtt Module (2) Candi-CNN Module (3) Candi-Attention Module</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*CcOYL6VTpVp9bGUeu4nNZw.png" /><figcaption>CAUM Model Architecture</figcaption></figure><h4>Part 1. Candi-SelfAtt Module (Long Term Interest Modeling)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*85Ye-Of3AkO2ALhlpaWsrw.png" /></figure><p>首先是 Candi-SelfAtt Module，新聞點擊的 Long-range context 通常可以為推斷 global user interests 提供重要的信息。此外，不同的遠程行為上下文對於捕捉不同的全球用戶興趣通常具有不同的重要性(所以需要 Attention)。</p><p>第一步，套用 Multi-Head Self-Attention (MHA) 在用戶的歷史點擊紀錄。我們一開始會得到每個 User 的 Clicked news representation [C1, C2, …, CN]，接著我們會把每篇新聞都先經過一個 Linear transformation (Qu)，來得到 Query Term / Query Matrix。那同時原先的 Clicked News 也會過一個 Linear transformation (Wr)，得到 Temporary clicked news representation。接著就是把 Query 跟這個 Temporary clicked news representation 去做內積運算，得到第 i 篇歷史點擊新聞跟第 j 篇歷史點擊新聞的 Attention scores。</p><p>第二步，把 Candidate news (nc) 納入考量，過一個 Linear transformation (Qc)，來得到 Query Term / Query Matrix (qc)。一樣把 Query 跟剛剛算的 Temporary clicked news representations 去做內積運算，得到這篇 Candidate news 跟第 j 篇歷史點擊新聞的 Attention scores，然後把這個 Attention scores 加上去剛剛第一步算出來的 Attention scores 得到最終的 Attention Scores。</p><p>第三步，根據前兩步算出來的 Attention Scores 來做 Attention Pooling，將結果再過一層的 Linear transformation (Wo)，得到第 k 個 head 的 Output news representation (li_k)。</p><p>上述的東西我們都會分成很多 head 來做，最後會把 K 個 head 的 Output Representation 給 Concat 起來得到最終的 News representation, li, i from 1 to N → [l1, l2, l3, …, lN]。</p><p>說白了，這塊在做的事情就把除了 Clicked News 自身之外還把 Candidate News 抓進來當 Query 然後做 Attention 得到 News representation。</p><h4>Part 2. Candi-CNN Module (Short Term Interest Modeling)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AIZKmcrrsV_ObJIjcXKESA.png" /></figure><p>再來第二部分是 Candi-CNN Module，看到 CNN 運用在這很明顯可以知道他在抓的是 Local Contexts。但這邊他的手法比較特別的是會引入 Candidate News 來做。</p><p>第一步，設定 CNN Window Size = 2h+1，即對於第 i-th 篇的歷史點擊新聞，我們會一起看他左邊 h 篇新聞以及右邊 h 篇新聞，以及 Candidate news nc 來得到第 i-th 篇的 local contextual representation si。Wc 就是 CNN Filter 的 Weight，掃完整排的歷史典籍新聞後會得到 [s1, s2, …, sN]。</p><p>第二步，把第一個 Candi-SelfAtt Module 跟當前這個 Candi-CNN Module 的產出，分別為 li &amp; si, i from 1 to N，兩者做結合。先把 si &amp; li 做 Concat 再過 Linear transformation (Pm) 得到 mi，這個 mi 就是每篇歷史點擊新聞的 News representation。</p><h4>Part 3. Candi-Attention Module</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zMN_QsPDK6wxjwgnXAqofA.png" /></figure><p>第三部分，在做的事情很簡單，就是把剛剛生成的每篇歷史點擊新聞的 News representation (mi)，做 Attention Pooling 得到 User representation (u)。而 Attention Score 的計算是來自於這第 i-th 篇的歷史點擊新聞 mi 跟 Candidate news nc 他們之間的 Similarity (Φ is an MLP network)。</p><h4>Part 4.<strong> News Encoder &amp; </strong>Interest Matching</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uA1WDOHxyCkWUL_QayFWGw.png" /></figure><p>最後，透過 BPR Loss 的方法來更新模型參數，直到收斂。</p><p>Training Details: In CAUM, dimensions of both news and user interest representations are set to 400.Candi-SelfAtt contains 20 attention heads, and output vectors of each head are 20-dimensional. Candi-CNN contains 400 filters and window size is set to 3. We train CAUM 3 epochs via Adam with 5e-5 learning rate. All hyper-parameters of CAUM and other baseline methods are selected based on the validation dataset.</p><h3>Experiments</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eLOtnAdP1gAKyMw4zQbcqQ.png" /></figure><h3>Ablation Study</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*e9ipd4j9woKnuzhpZZnqTw.png" /></figure><p>First, adding one of Candi-SelfAtt and Candi-CNN significantly improves the performance of Base. This result verifies that Candi-SelfAtt and Candi-CNN can effectively exploit behavior contexts to capture global and short-term user interests that are informative for matching the candidate news, respectively. Second, CAUM outperforms both Base+CandiCNN and Base+Candi-SelfAtt. This is because candidate-aware user interest modeling is important for both global and short-term user interest.</p><p>This article will be updated at any time! Thanks for your reading. If you like the content, please clip the “clap” button. You can also press the follow button to track new articles. Feel free to connect with me via <a href="https://www.linkedin.com/in/harenlin/">LinkedIn</a> or email.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7035c18cc95d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>