<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by LP Cheung on Medium]]></title>
        <description><![CDATA[Stories by LP Cheung on Medium]]></description>
        <link>https://medium.com/@lpcheung?source=rss-53d7d2192862------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*lEnI58HuuWyHyBtDCtATcA.jpeg</url>
            <title>Stories by LP Cheung on Medium</title>
            <link>https://medium.com/@lpcheung?source=rss-53d7d2192862------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 11:27:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lpcheung/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Snorkel: Build ML Models without Labeled Data]]></title>
            <link>https://medium.com/analytics-vidhya/snorkel-build-ml-models-without-labeled-data-21aa38b9d7a9?source=rss-53d7d2192862------2</link>
            <guid isPermaLink="false">https://medium.com/p/21aa38b9d7a9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[semi-supervised-learning]]></category>
            <category><![CDATA[weak-supervision]]></category>
            <dc:creator><![CDATA[LP Cheung]]></dc:creator>
            <pubDate>Tue, 02 Jun 2020 10:05:30 GMT</pubDate>
            <atom:updated>2020-07-02T17:31:18.338Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V8chgbK6eoz_9AJaNSDIcA.jpeg" /></figure><p>With bunches of hands-on tools, building models on labeled data has already been an easy task for data scientists. However, in the real world, many tasks are not well-formatted supervised learning problems: labeled data may be expensive or even impossible to obtain. An alternative approach is to leverage cheap and low-quality data to achieve supervision, which is the topic of this article: <strong>weak supervision</strong></p><p>In the following sections, I will go through the concepts of weak supervision. I will also introduce a tool called <a href="https://www.snorkel.org/"><strong>Snorkel</strong></a>, which is developed by Stanford. Finally, I will show you how HK01 uses Snorkel to capture the trend topics on Facebook, and therefore enhance our recommender engine.</p><p>There are several paradigms of algorithm to remedy the situation if a large amount of high-quality, hand-labeled training data is not available. As you can see in the following diagram, if you don’t have enough data, you have to find another source of knowledge to achieve a comparable level of supervision to traditional supervision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i8Jzw4jiv22-HfHK.png" /><figcaption>source: <a href="http://ai.stanford.edu/blog/weak-supervision/">http://ai.stanford.edu/blog/weak-supervision/</a></figcaption></figure><p>Choosing one among these paradigms is pretty tricky. It depends on what you have on your hands. Transfer learning is great for tasks with a well-trained model in similar tasks, like fine-tuning ImageNet model with your own categories; while you may have some presumptions on the topological structure, such as the shape of clusters, you may prefer semi-supervised learning.</p><p><strong>So, what kind of situation is the best suit for weak supervision?</strong></p><p>You may have some ideas after reading the definition of weak supervision. Yes, <em>if you have plenty of domain experts but lack of data, weak supervision is your pick</em>.</p><p>The reason behind is revealed in the definition: weak supervision enables learning from low-quality and noisy labels. In other words, you can still find patterns, just like what supervised learning do, unless you should supplement multiple noisy labels for each training sample so that the model can generalize knowledge from them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nka5pMJ5iAWoxhC096jFvQ.png" /><figcaption>weak supervision enables supervision by multiple noisy labels</figcaption></figure><p>The rationale of weak supervision relies on the fact that noisy data is usually much easier and cheaper to obtain than high-quality data. Imagine you are working for an insurance company and your boss asks for a recommender engine of a whole-new product line which, of course, you don’t have data. With sales experts, we can set up some rules which are “mostly correct” like the new product is more attractive to the elderly. These rules are not perfectly correct; but, they are good enough to provide your models collective intelligence. And, most importantly, these rules are easier to obtain than perfectly hand-labeled data.</p><p>So, the next question is: <strong>how can we inject these rules into our ML models? </strong>The answer is Snorkel.</p><p>Snorkel is a system developed by Stanford which allows you to program the rules into ML models. The key idea of Snorkel is to build the generative model which represents the causal relationship between the true label and the noisy labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F_D9vxDznuRJS-M94qfrGA.png" /></figure><p>The left-hand side of the above diagram is the probabilistic model representing the generative process from the true label to the noisy labels. Although the true label is unobservable, we can still learn the accuracies and correlations by the agreements and disagreements from different noisy labels. Hence, we can estimate the P(L|y) of each noisy label, which is essentially an indicator of quality. By aggregating the noisy labels, we get the estimated true label and use it to train our model.</p><p>In Snorkel, noisy labels are programmed as <strong>labeling functions</strong>. A label function is basically a python function which hard-codes a rule to determine the label. For example, if you’re writing a program to determine which an email is spam, the program should be something like:</p><pre>from snorkel.labeling import labeling_function</pre><pre>SPAM = 1<br>NORMAL = 0<br>ABSTAIN = -1</pre><pre>@labeling_function()<br>def contain_hyperlink(x):<br>    if &#39;http&#39; in x:<br>        return SPAM <br>    else:<br>        return NORMAL</pre><pre>@labeling_function()<br>def contain_foul_language(x):<br>    for each in x:<br>        if each in foul_language:<br>            return SPAM<br>        else:<br>            return NORMAL</pre><p>In this toy example, you can see the basic elements of Snorkel.</p><ul><li>define the labels. In this example, the labels are SPAM, NORMAL &amp; ABSTAIN. ABSTAIN is the label used when you cannot determine the label.</li><li>define labeling functions. Add a decorator @labeling_function() to declare.</li></ul><p>After defining the labels and labeling function, you can apply the labeling functions to your dataset and estimate the best label:</p><pre>from snorkel.labeling import PandasLFApplier, LabelModel</pre><pre>lfs = [contain_hyperlink, contain_foul_language]</pre><pre>applier = PandasLFApplier(lfs=lfs)<br>overall_result = applier.apply(df=dataset, progress_bar=False)</pre><pre>label_model = LabelModel(cardinality=2, verbose=True)<br>label_model.fit(L_train=overall_result, n_epochs=100, lr=0.001, log_freq=100, seed=123)</pre><pre>score = label_model.predict_proba(overall_result)</pre><ul><li>declare a list of labeling functions lfs</li><li>create an applier and apply the labeling functions to your dataset to get the noisy labels overall_result</li><li>create a generative model label_model to resolve the conflicts between noisy labels</li><li>get the predicted probability score of the training samples</li></ul><p>Then, the score can be served as the estimated label to train your model.</p><p>In HK01, one of our missions is to capture the trending topics in social media. Success in this task can improve the quality of our recommendations. Also, it provides us with additional information on the importance of the news.</p><p>One of the challenges is to match the trending topics with our existing news in the pool. Obviously, it is highly inefficient to hand-label the matchings as we have to go through hundred-thousands of articles for each topic. However, experienced editors can apply their domain knowledge in filtering irrelevant articles.</p><p>We also apply machine learning and natural language processing algorithm to build labeling functions. For example, we use named-entity recognition and tokenization to quickly identify the keywords so that we can calculate the distance between trending topics and existing articles.</p><h4>Reference</h4><ol><li><a href="http://ai.stanford.edu/blog/weak-supervision/">http://ai.stanford.edu/blog/weak-supervision/</a></li><li><a href="https://www.snorkel.org/">https://www.snorkel.org/</a></li><li><a href="https://arxiv.org/pdf/1711.10160.pdf">https://arxiv.org/pdf/1711.10160.pdf</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21aa38b9d7a9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/snorkel-build-ml-models-without-labeled-data-21aa38b9d7a9">Snorkel: Build ML Models without Labeled Data</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[最強中文自然語言處理工具CKIPtagger]]></title>
            <link>https://medium.com/@lpcheung/%E6%9C%80%E5%BC%B7%E4%B8%AD%E6%96%87%E8%87%AA%E7%84%B6%E8%AA%9E%E8%A8%80%E8%99%95%E7%90%86%E5%B7%A5%E5%85%B7ckiptagger-b98f0b3bcd?source=rss-53d7d2192862------2</link>
            <guid isPermaLink="false">https://medium.com/p/b98f0b3bcd</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[named-entity-recognition]]></category>
            <dc:creator><![CDATA[LP Cheung]]></dc:creator>
            <pubDate>Thu, 23 Jan 2020 10:19:11 GMT</pubDate>
            <atom:updated>2020-01-23T13:43:30.491Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*csJW_OdVsFatWgtE2J8v5g.jpeg" /></figure><p>每一位做中文NLP(Natural Language Processing)的朋友大概都用過<a href="https://github.com/fxsjy/jieba">Jieba</a>。分詞作為中文NLP工序的第一步，對模型性能影響至鉅。然而，以往可供選擇的工具並不多。Jieba雖然分詞效果不理想，卻幾乎是預設的選項。最近台灣中研院開發的CKIPtagger成為了Jieba的挑戰者。CKIPtagger不但分詞效果更理想，還提供實體識別功能。下文會比較兩者的效果和CKIPtagger的使用方式。</p><p><a href="https://github.com/ckiplab/ckiptagger">GitHub - ckiplab/ckiptagger: CKIP Neural Chinese Word Segmentation, POS Tagging, and NER</a></p><h3>分詞例子</h3><p>我在HK01上隨意選擇三篇文章，以Jieba和CKIPtagger分別做分詞。三篇文章題材不同，其中一篇較多人名地名，看看兩者的表現有沒有受影響。</p><p><strong>Example 1 Input:</strong></p><pre>def get_jieba_tokenization(text):<br>    tokenized = jieba.lcut(text)<br>    return &#39; &#39;.join(tokenized)</pre><pre>def get_ckiptagger_tokenization(text):<br>    ckip_tokenized = ws([text])<br>    return &#39; &#39;.join(ckip_tokenized[0])</pre><pre>text1 = &#39;&#39;&#39;2003年沙士一役，港大新發病毒性疾病學講座教授管軼在廣州野味市場的果子狸身上，找到感染人類的沙士冠狀病毒，找到元兇遏止疫情擴散，成為沙士抗疫英雄。事隔17年，他本周二三（21、22日），與團隊到武漢考察，但找不到源頭，「源頭被銷毀得乾乾淨淨，這裏似乎不歡迎專家」，他對內地傳媒說，估計武漢感染規模大沙士十倍起跳，又說政府不作為，無奈做「逃兵」訂機票返港，感到極其無力。&#39;&#39;&#39;</pre><pre>print (f&#39;jieba result: \n{get_jieba_tokenization(text1)}&#39;)<br>print (&#39;\n&#39;)<br>print (f&#39;ckiptagger result: \n{get_ckiptagger_tokenization(text1)}&#39;)</pre><p><strong>Example 1 Output:</strong></p><pre>jieba result: <br>2003 年 沙士 一役 ， 港大新發 病毒性 疾病 學講座 教授 管軼 在 廣州 野味 市場 的 果子狸 身上 ， 找到 感染 人類 的 沙士 冠狀 病毒 ， 找到 元 兇 遏止 疫情 擴散 ， 成為 沙士 抗疫 英雄 。 事隔 17 年 ， 他 本周 二三 （ 21 、 22 日 ） ， 與 團隊 到 武漢 考察 ， 但 找 不到 源頭 ， 「 源頭 被 銷毀 得 乾 乾淨淨 ， 這 裏 似乎 不歡 迎 專家 」 ， 他 對 內 地 傳媒 說 ， 估計 武漢 感染 規模 大 沙士 十倍 起跳 ， 又 說 政府 不作 為 ， 無奈 做 「 逃兵 」 訂機票 返港 ， 感到 極其 無力 。<br><br><br>ckiptagger result: <br>2003年 沙士 一 役 ， 港大 新發病毒性 疾病學 講座 教授 管軼 在 廣州 野味 市場 的 果子狸 身 上 ， 找到 感染 人類 的 沙士 冠狀 病毒 ， 找到 元兇 遏止 疫情 擴散 ， 成為 沙士 抗疫 英雄 。 事 隔 17 年 ， 他 本 周 二三 （ 21 、 22日 ） ， 與 團隊 到 武漢 考察 ， 但 找 不 到 源頭 ， 「 源頭 被 銷毀 得 乾乾淨淨 ， 這裏 似乎 不 歡迎 專家 」 ， 他 對 內地 傳媒 說 ， 估計 武漢 感染 規模 大 沙士 十 倍 起跳 ， 又 說 政府 不 作為 ， 無奈 做 「 逃兵 」 訂 機票 返 港 ， 感到 極其 無力 。</pre><p>以上例子可以見到，Jieba在比較長的詞語表現欠佳，如將「港大新發病毒性疾病學講座教授」分成「港大新發 病毒性 疾病 學講座 教授」。CKIPtagger的長詞語分詞比較合理。</p><p><strong>Example 2 Input:</strong></p><pre>text2 = &#39;&#39;&#39;農曆新年將至，又是逗利是的時候，打工仔也可望收到開工利是。滙豐連續第二年向本港員工派發電子利是，每位500元，將在員工下一個出糧日透過出糧戶口發放。滙豐香港區行政總裁施穎茵向香港員工發出電郵，宣布派發電子利是的消息。她於電郵內表示，十分感謝同事對公司及客戶無限的貢獻及投入，即使前景仍有不確定性，她有信心克服挑戰，今年再創繁榮。滙豐發言人回覆傳媒查詢時稱，滙豐希望藉著農曆新年，感謝各同事過去一年的努力。跟去年一樣，該行將會向每位香港員工派發500元的電子利是，在各同事下一個出糧日透過出糧戶口發放。&#39;&#39;&#39;</pre><pre>print (f&#39;jieba result: \n{get_jieba_tokenization(text2)}&#39;)<br>print (&#39;\n&#39;)<br>print (f&#39;ckiptagger result: \n{get_ckiptagger_tokenization(text2)}&#39;)</pre><p><strong>Example 2 Output：</strong></p><pre>jieba result: <br>農 曆 新年 將至 ， 又 是 逗利 是 的 時候 ， 打工仔 也 可望 收到 開工利 是 。 滙 豐連續 第二年 向 本港 員 工派 發電子 利是 ， 每位 500 元 ， 將在員 工下 一個 出 糧日 透過 出 糧戶口 發放 。 滙 豐 香港 區 行政 總裁 施穎茵 向 香港 員工 發出 電郵 ， 宣布 派 發電子利 是 的 消息 。 她 於 電郵內 表示 ， 十分 感謝 同事 對 公司 及 客戶 無限 的 貢獻及 投入 ， 即使 前景 仍有 不確 定性 ， 她 有 信心 克服 挑戰 ， 今年 再創 繁榮 。 滙 豐發言人 回覆 傳媒查 詢時 稱 ， 滙 豐 希望 藉 著農 曆 新年 ， 感謝 各 同事 過去 一年 的 努力 。 跟 去年 一樣 ， 該行 將會 向 每位 香港 員 工派 發 500 元 的 電子利 是 ， 在 各 同事 下一個 出 糧日 透過 出 糧戶口 發放 。<br><br><br>ckiptagger result: <br>農曆 新年 將 至 ， 又 是 逗利 是 的 時候 ， 打工仔 也 可望 收到 開工利 是 。 滙豐 連續 第二 年 向 本 港 員工 派發 電子 利 是 ， 每 位 500 元 ， 將 在 員工 下 一 個 出糧日 透過 出糧 戶口 發放 。 滙豐 香港區 行政 總裁 施穎茵 向 香港 員工 發出 電郵 ， 宣布派發 電子 利 是 的 消息 。 她 於 電郵 內 表示 ， 十分 感謝 同事 對 公司 及 客戶 無限 的 貢獻 及 投入 ， 即使 前景 仍 有 不確定性 ， 她 有 信心 克服 挑戰 ， 今年 再 創 繁榮 。 滙豐 發言人 回覆 傳媒 查詢 時 稱 ， 滙豐 希望 藉著 農曆 新年 ， 感謝 各 同事 過去 一 年 的 努力 。 跟 去年 一樣 ， 該 行 將 會 向 每 位 香港 員工派發 500 元 的 電子利 是 ， 在 各 同事 下 一 個 出糧日 透過 出糧 戶口 發放 。</pre><p>Jieba連「農曆新年」都分錯，而且無法認到「滙豐」。</p><p><strong>Example 3 Input：</strong></p><pre>text3 = &#39;&#39;&#39;大阪、京都、奈良、神戶是親子遊的熱門路線，其實除了京阪神，從大阪玩到鳥取也是不錯選擇！官方早前推出鳥取1000円巴士優惠，由大阪難波往鳥取只需1000円，大阪Open Jaw玩鳥取一流，加上適逢鳥取紅楚蟹季開鑼，平食海產、鳥取和牛、21世紀梨……還有必去鳥取砂丘、鬼太郎商店街及小朋友最愛的柯南博物館等，大人小朋友都玩得滿足！&#39;&#39;&#39;</pre><pre>print (f&#39;jieba result: \n{get_jieba_tokenization(text3)}&#39;)<br>print (&#39;\n&#39;)<br>print (f&#39;ckiptagger result: \n{get_ckiptagger_tokenization(text3)}&#39;)</pre><p><strong>Example 3 Output:</strong></p><pre>jieba result: <br>大阪 、 京都 、 奈良 、 神戶 是 親子遊 的 熱門 路線 ， 其實 除了 京 阪神 ， 從 大阪 玩到 鳥取 也 是 不錯 選擇 ！ 官方 早前 推出 鳥取 1000 円 巴士 優惠 ， 由 大阪 難波往 鳥取 只 需 1000 円 ， 大阪 Open   Jaw 玩鳥取 一流 ， 加上 適逢 鳥 取紅楚 蟹 季開鑼 ， 平食 海產 、 鳥取 和 牛 、 21 世紀 梨 … … 還有 必去 鳥取 砂丘 、 鬼 太郎 商店 街及 小朋友 最愛的 柯南 博物 館 等 ， 大人 小朋友 都 玩 得 滿足 ！<br><br><br>ckiptagger result: <br>大阪 、 京都 、 奈良 、 神戶 是 親子 遊 的 熱門 路線 ， 其實 除了 京阪神 ， 從 大阪 玩 到 鳥取 也 是 不錯 選擇 ！ 官方 早前 推出 鳥取 1000 円 巴士 優惠 ， 由 大阪 難波 往 鳥取 只 需 1000 円 ， 大阪 Open  Jaw 玩 鳥 取 一流 ， 加上 適逢 鳥取 紅楚蟹季 開鑼 ， 平食 海產 、 鳥取 和 牛 、 21世紀 梨 … … 還 有 必 去 鳥取 砂丘 、 鬼太郎 商店街 及 小朋友 最愛 的 柯南 博物館 等 ， 大人 小朋友 都 玩 得 滿足 ！</pre><p>這篇文章比較多地名同專有名詞， CKIPtagger大致上都認到，例如「京阪神」、「難波」。而Jieba就認不到。</p><h3>命名實體識別</h3><p>除了分詞外，CKIPtagger還有命名實體識別(named entity recognition)能力。比如說如果想抽取第三篇文的地名：</p><pre>tokenized_result = ws([text3])<br>pos_result = pos(tokenized_result)</pre><pre>entity_result = ner(tokenized_result, pos_result)<br>print (entity_result)</pre><p>結果非常準確：</p><pre>{(0, 2, &#39;GPE&#39;, &#39;大阪&#39;),<br>  (3, 5, &#39;GPE&#39;, &#39;京都&#39;),<br>  (6, 8, &#39;GPE&#39;, &#39;奈良&#39;),<br>  (9, 11, &#39;GPE&#39;, &#39;神戶&#39;),<br>  (25, 28, &#39;GPE&#39;, &#39;京阪神&#39;),<br>  (30, 32, &#39;GPE&#39;, &#39;大阪&#39;),<br>  (34, 36, &#39;GPE&#39;, &#39;鳥取&#39;),<br>  (49, 51, &#39;GPE&#39;, &#39;鳥取&#39;),<br>  (51, 55, &#39;CARDINAL&#39;, &#39;1000&#39;),<br>  (62, 64, &#39;GPE&#39;, &#39;大阪&#39;),<br>  (64, 66, &#39;GPE&#39;, &#39;難波&#39;),<br>  (67, 69, &#39;GPE&#39;, &#39;鳥取&#39;),<br>  (71, 75, &#39;CARDINAL&#39;, &#39;1000&#39;),<br>  (77, 87, &#39;GPE&#39;, &#39;大阪Open Jaw&#39;),<br>  (97, 99, &#39;GPE&#39;, &#39;鳥取&#39;),<br>  (111, 113, &#39;GPE&#39;, &#39;鳥取&#39;),<br>  (116, 120, &#39;DATE&#39;, &#39;21世紀&#39;),<br>  (127, 129, &#39;GPE&#39;, &#39;鳥取&#39;),<br>  (145, 150, &#39;FAC&#39;, &#39;柯南博物館&#39;)}</pre><h3>Remark</h3><p>雖然CKIPtagger非常好用，但都存在一些缺點：</p><ol><li>相比Jieba，無論啟動速度還是分詞速度都較慢。因為CKIPtagger背後用到深度學習模型，每次使用前都要先將模型加載。</li><li>下載預訓練模型的連結有時會失效。解決方法很簡單：預先下載。</li></ol><p>完整代碼：</p><pre>import jieba<br>from ckiptagger import WS, POS, NER, data_utils</pre><pre>data_utils.download_data_gdown(&quot;./&quot;)</pre><pre>ws = WS(&quot;./data&quot;)<br>pos = POS(&quot;./data&quot;)<br>ner = NER(&quot;./data&quot;)</pre><pre>def get_jieba_tokenization(text):<br>    tokenized = jieba.lcut(text)<br>    return &#39; &#39;.join(tokenized)</pre><pre>def get_ckiptagger_tokenization(text):<br>    ckip_tokenized = ws([text])<br>    return &#39; &#39;.join(ckip_tokenized[0])</pre><pre>text1 = &#39;&#39;&#39;2003年沙士一役，港大新發病毒性疾病學講座教授管軼在廣州野味市場的果子狸身上，找到感染人類的沙士冠狀病毒，找到元兇遏止疫情擴散，成為沙士抗疫英雄。事隔17年，他本周二三（21、22日），與團隊到武漢考察，但找不到源頭，「源頭被銷毀得乾乾淨淨，這裏似乎不歡迎專家」，他對內地傳媒說，估計武漢感染規模大沙士十倍起跳，又說政府不作為，無奈做「逃兵」訂機票返港，感到極其無力。&#39;&#39;&#39;</pre><pre>text2 = &#39;&#39;&#39;農曆新年將至，又是逗利是的時候，打工仔也可望收到開工利是。滙豐連續第二年向本港員工派發電子利是，每位500元，將在員工下一個出糧日透過出糧戶口發放。滙豐香港區行政總裁施穎茵向香港員工發出電郵，宣布派發電子利是的消息。她於電郵內表示，十分感謝同事對公司及客戶無限的貢獻及投入，即使前景仍有不確定性，她有信心克服挑戰，今年再創繁榮。滙豐發言人回覆傳媒查詢時稱，滙豐希望藉著農曆新年，感謝各同事過去一年的努力。跟去年一樣，該行將會向每位香港員工派發500元的電子利是，在各同事下一個出糧日透過出糧戶口發放。&#39;&#39;&#39;</pre><pre>text3 = &#39;&#39;&#39;大阪、京都、奈良、神戶是親子遊的熱門路線，其實除了京阪神，從大阪玩到鳥取也是不錯選擇！官方早前推出鳥取1000円巴士優惠，由大阪難波往鳥取只需1000円，大阪Open Jaw玩鳥取一流，加上適逢鳥取紅楚蟹季開鑼，平食海產、鳥取和牛、21世紀梨……還有必去鳥取砂丘、鬼太郎商店街及小朋友最愛的柯南博物館等，大人小朋友都玩得滿足！&#39;&#39;&#39;</pre><pre>print (f&#39;jieba result: \n{get_jieba_tokenization(text1)}&#39;)<br>print (&#39;\n&#39;)<br>print (f&#39;ckiptagger result: \n{get_ckiptagger_tokenization(text1)}&#39;)</pre><pre>print (f&#39;jieba result: \n{get_jieba_tokenization(text2)}&#39;)<br>print (&#39;\n&#39;)<br>print (f&#39;ckiptagger result: \n{get_ckiptagger_tokenization(text2)}&#39;)</pre><pre>print (f&#39;jieba result: \n{get_jieba_tokenization(text3)}&#39;)<br>print (&#39;\n&#39;)<br>print (f&#39;ckiptagger result: \n{get_ckiptagger_tokenization(text3)}&#39;)</pre><pre>tokenized_result = ws([text3])<br>pos_result = pos(tokenized_result)</pre><pre>entity_result = ner(tokenized_result, pos_result)<br>print (entity_result)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b98f0b3bcd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[StarSpace: mining and embedding user interests]]></title>
            <link>https://medium.com/data-science/starspace-mining-and-embedding-user-interests-28081937f95?source=rss-53d7d2192862------2</link>
            <guid isPermaLink="false">https://medium.com/p/28081937f95</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[user-embedding]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[user-profiling]]></category>
            <dc:creator><![CDATA[LP Cheung]]></dc:creator>
            <pubDate>Sat, 20 Jul 2019 14:00:47 GMT</pubDate>
            <atom:updated>2020-07-12T03:48:57.676Z</atom:updated>
            <content:encoded><![CDATA[<h3>StarSpace Algorithm: Mining and Embedding User Interests</h3><h4>How to do user embedding using Facebook Starspace algorithm</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K8jWAeBu1eJflAgw" /><figcaption>Photo by <a href="https://unsplash.com/@maxmckinnon?utm_source=medium&amp;utm_medium=referral">Max McKinnon</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Data scientists always want to know more about their customers. If we know our customers, we can:</p><ol><li>promote events based on user interests</li><li>predict user attributes</li><li>find user similarity and do clustering</li></ol><p>Sounds good? The pain point is all these downstream tasks require a highly informative user embedding so that you can play around with machine learning algorithms. Of course, you can do machine learning without user embedding by simply fitting whatever about the users to the algorithm. The problem is, usually, the data about the user is highly unstructured or even variable length. The preprocessing step to encode this kind of messy data is not trivial.</p><p>With StarSpace, we can obtain highly informative user embeddings according to user behaviors.</p><h4>Background</h4><p>StarSpace is an algorithm proposed by Facebook [1].</p><blockquote>StarSpace is a general-purpose neural model for efficient learning of entity embeddings for solving a wide variety of problems</blockquote><p>So, the keywords are <em>“general-purpose”</em> and <em>“entity embedding”</em>. In other words, you can embed whatever you want by StarSpace, including the user. Actually, HK01 data team also exploits StarSpace to do article embeddings, in parallel with other embedding algorithms.</p><p><a href="https://medium.com/@LouisKitLungLaw/documents-embeddings-using-facebook-s-starspace-67b8d1feba32">Learn document embeddings using Facebook StarSpace</a></p><p>How can it be so generic? The secret is: StarSpace learns to represent entities and their components into the same vector space. The only requirement to use StarSpace is you should provide the linkages between entities and components. Taking the linkages as training data, StarSpace can learn the similarity between components and, furthermore, the similarity between entities.</p><h4>Intuition</h4><p>The next question is what components can represent users. In HK01, users are represented by the articles read.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XXqYZJOxi5-H679qHalJVg.png" /><figcaption>Three levels of entities. All mapped into the same space</figcaption></figure><p>The diagram above illustrates the linkages between different levels of entities. Users are represented by the articles read while articles are represented by the tokens. In particular, a user is represented by the bag of tokens in his/her reading history.</p><p>The intuition behind is that the information in each layer can be propagated into upstream and downstream layers. With the linkages, StarSpace can find the similarity between articles as they share the tokens. Likewise, the similarity between users can, therefore, be learned.</p><h4>Result</h4><p>User embedding learned by StarSpace, which has only 50 dimensions, can provide comparable performance in downstream tasks to the original user embedding, which has more than 20,000 dimensions.</p><h4>Theory</h4><p>So far, we still don’t know how StarSpace works. The novelty of StarSpace is proposing a new loss function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*UvTUEjySeH4VuK4JZ587sw.png" /><figcaption>The loss function of StarSpace</figcaption></figure><p>Don’t worry. The idea behind is that StarSpace is trained to assign higher similarity to positive samples than negative samples by a margin. The loss function is called margin ranking loss, which is similar to hinge loss. The similarity function can be cosine similarity.</p><p>For example, in user embedding, each entity pair <em>(a, b)</em> are formed by <em>(user, token)</em>. A positive entity pair is an actual linkage while a negative pair is generated by negative sampling. StarSpace is trained to map a user to a position near the actual tokens rather than the negatively sampled tokens.</p><h4>Reference</h4><p>[1]: Wu, Ledell Yu, et al. “Starspace: Embed all the things!.” <em>Thirty-Second AAAI Conference on Artificial Intelligence</em>. 2018.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28081937f95" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/starspace-mining-and-embedding-user-interests-28081937f95">StarSpace: mining and embedding user interests</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Attention Model for News Recommendation: Cold Start Problem]]></title>
            <link>https://medium.com/data-science/attention-model-for-news-recommendation-cold-start-problem-dba18da8bf54?source=rss-53d7d2192862------2</link>
            <guid isPermaLink="false">https://medium.com/p/dba18da8bf54</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[cold-start]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[LP Cheung]]></dc:creator>
            <pubDate>Fri, 05 Jul 2019 05:06:00 GMT</pubDate>
            <atom:updated>2020-07-13T15:51:47.056Z</atom:updated>
            <content:encoded><![CDATA[<h3>Attention Model for News Recommendation</h3><h4>Solving Cold Start Problem using Attention Mechanism</h4><figure><img alt="as cold as the cold start problem" src="https://cdn-images-1.medium.com/max/1024/0*pSD7zBp78N2lwolb" /><figcaption>Photo by <a href="https://unsplash.com/@mrsamwheeler?utm_source=medium&amp;utm_medium=referral">Sam Wheeler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Although SVD provides a satisfactory solution to recommendation system, it is less effective when the new items did not accumulate enough data. News recommendation is even more challenging as it poses three additional challenges:</p><ol><li>News articles are highly time-sensitive</li><li>Users are topic-sensitive and have various interests</li><li>News language is highly condensed and comprised of a large number of new entities created every day</li></ol><p>In this article, I will show you how to leverage the attention mechanism to solve the cold start problem in recommendation system.</p><h3>Attention Mechanism</h3><p>Attention Mechanism has a long history of applications and recently introduced to solve problems in NLP. Attention mechanism enables the model to impose different weights to inputs depending on the context. For example, in Neural Machine Translation (NMT), attention mechanism can be used to overcome the bidirectional information flows. With attention mechanism, NMT model can generate words by “looking” at different positions in the original text.</p><p>The situation is similar for news recommendation system. The recommender engine should learn to “look” at the relevant part and ignore the irrelevant part of reading history.</p><h3>Deep Knowledge-Aware Network</h3><p>In HK01, our data team has put tremendous efforts in the news recommendation system. We employ state-of-the-art algorithms to improve the original SVD collaborative filtering algorithm. In particular, we solve problem 3 using <a href="https://medium.com/@LouisKitLungLaw/compute-document-similarity-using-autoencoder-with-triplet-loss-eb7eb132eb38">autoencoder with triplet loss</a> and <a href="https://medium.com/@LouisKitLungLaw/documents-embeddings-using-facebook-s-starspace-67b8d1feba32">StarSpace</a> to learn article embeddings.</p><ul><li><a href="https://medium.com/@LouisKitLungLaw/compute-document-similarity-using-autoencoder-with-triplet-loss-eb7eb132eb38">Compute Document Similarity Using Autoencoder With Triplet Loss</a></li><li><a href="https://medium.com/@LouisKitLungLaw/documents-embeddings-using-facebook-s-starspace-67b8d1feba32">Learn document embeddings using Facebook StarSpace</a></li></ul><p>For problems 1 &amp; 2, we leverage <strong>Deep Knowledge-Aware Network (DKN)</strong>, which is proposed by Microsoft [1], to solve the cold start problem. We replace the knowledge-embeddings with our own article embeddings and keep the attention network to learn the interaction between user interests and article embeddings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a-7JrprV2XZTBGIGCSVNHg.png" /><figcaption>Architecture of DKN</figcaption></figure><p>DKN consists of two component networks. The overall architecture of DKN is illustrated on the right-hand side. In order to predict the click probability, the algorithm <strong>learns to aggregate the sequence of reading history to form a user embedding</strong>. User embedding is regarded as the weighted sum of article embeddings in the reading history. So, the problem is: <strong>how to find the weights?</strong></p><p>The weight of each article embedding is obtained by attention network. Attention network models the interaction between the candidate news, the news we are going to predict, and the user’s clicked news. Since a user can have various interests and there is no single news that can match every interest, the role of attention network is to match the candidate news to either one.</p><p>After obtaining the weights, the model generates the user embedding by weighted sum and passes both user and candidate news embedding to a feed-forward neural network.</p><p>DKN is basically an item-based algorithm. It doesn’t need user-item interaction data for new item recommendation. Moreover, it can handle multiple interests without intervention.</p><h3>Result</h3><p>We compare the original model, a regularized version of SVD, with the ensemble of SVD and DKN. With carefully tuning, the ensemble can achieve a 10% increment over the original model.</p><h3>Reference</h3><p>[1]: Zhou, Guorui, et al. “Deep interest network for click-through rate prediction.” <em>Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining</em>. ACM, 2018.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dba18da8bf54" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/attention-model-for-news-recommendation-cold-start-problem-dba18da8bf54">Attention Model for News Recommendation: Cold Start Problem</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some Techniques in Deep Learning Optimization (1): Learning Rate]]></title>
            <link>https://medium.com/deep-learning-hk/some-techniques-in-deep-learning-optimization-1-learning-rate-b4669d5bb568?source=rss-53d7d2192862------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4669d5bb568</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[LP Cheung]]></dc:creator>
            <pubDate>Thu, 28 Feb 2019 10:26:38 GMT</pubDate>
            <atom:updated>2019-03-20T04:34:26.249Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Methodology to Hyper-parameter Tuning (1): Learning Rate</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0xCiaOuF_HA_76dbsXtd-w.png" /></figure><p>Although tf.keras and Pytorch make it easier to build a deep learning model, you still need to make various decisions, such as <strong>learning rate</strong>, <strong>regularization</strong>, and <strong>batch size</strong>. Among all, learning rate is the first hyper-parameter you need to properly determine as a careless choice may ruin your model. Some interesting techniques may release your burden in choosing the proper learning rate:</p><ul><li>Learning Rate Range Test</li><li>Cyclical Learning Rate Policy</li><li>1-Cycle Policy and Super-Convergence</li></ul><p>If you want to be safe, adopting Adam optimizer with learning rate 3e-4 is the first thing to try, just like Andrej Karpathy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ufIbenir6zdbgQ1ACZTeEQ.png" /></figure><p>But what if it doesn’t work? You may then try to slightly tune the learning rate around 3e-4 and pray for it. Luckily, you can be more scientific using the Learning Rate Range Test by <a href="https://scholar.google.com/citations?user=pwh7Pw4AAAAJ&amp;hl=en">L. Smith</a>.</p><h4>Learning Rate Range Test</h4><p>The idea is very simple:</p><ol><li>Select a validation set</li><li>Monitor the validation loss and tune the learning rate starting from a very small value</li><li>For each iteration, increase the learning rate</li><li>Plot the validation loss and see when the training starts diverging</li></ol><p>The typical LR range test diagram looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*2zccGH8sS6YP9uec70LLZA.png" /><figcaption>LR range test for resnet56. (<a href="https://github.com/davidtvs/pytorch-lr-finder"><strong>source</strong></a><strong>)</strong></figcaption></figure><p>Since you start training from a very small learning rate, the loss plot should initially be flat. The learning rate would soon be reasonably large as you increase the value for each iteration. The loss keeps decreasing is a signal for reasonable learning rate. The learning rate would finally reach a region where it is too large that the training diverges.</p><p>So, we can now determine the reasonable range for the learning rate. The next question is: <strong>what exact value should we choose</strong>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*5qqTJN1vvwEgekXXbHzU5g.png" /></figure><p>The good news is you don’t need to make a decision. If you cannot choose one, choose them all.</p><h4>Cyclical Learning Rate (CLR) policy</h4><p>You can simply just adopt a range of learning rate rather than an exact value. The procedures are:</p><ol><li>Run the LR range test to determine the maximum and minimum bound of learning rate</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*sVg8Fo25bdS781lgtuxSvQ.png" /></figure><p>2. Vary the learning rate between these bounds</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*fI0l_gIezdHLFlZpJ2s8iA.png" /><figcaption>Again, by LN Smith. (<a href="https://arxiv.org/pdf/1506.01186.pdf">source</a>)</figcaption></figure><p>There is only one hyper-parameter not yet determined: <em>stepsize. stepsize</em> is the number of iterations in half a cycle. A simple rule of thumb is setting <em>stepsize</em> = 2–10 times the number of iterations in an epoch.</p><p>But why it works? An intuitive explanation is that cyclically increasing the learning rate enable the optimizer to escape from saddle points. As the gradients in saddle points are small, simply fixing the learning rate in a small value would slower the training process.</p><p>Another explanation is that periodically increasing the learning rate allows the network to generalize better. Since we want the network generalize to the unseen data, we would prefer a flat minimum, which is more robust to shift between training and testing distribution, over a sharp minimum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*TOrltxGA8SCPh7MJo0Qp-g.png" /><figcaption>The black line is the training distribution and the red line is the testing distribution. The y-axis is the loss. (<a href="https://arxiv.org/pdf/1609.04836.pdf">source</a>)</figcaption></figure><p>As we can see in the above diagram, if the optimizer stays in the sharp minima, a small shift of testing distribution can lead to a huge drop of accuracy. Optimizers adopting CLR policy can “jump out” of the sharp minima as the learning rate occasionally becomes large, which allows the optimizers to take large enough step to escape from it.</p><p>The idea of “large learning rate enables the network to generalize better” is further developed by Smith (again…) and he proposes a even more radical technique called 1-cycle policy and a phenomenon called “super-convergence”.</p><h4>1-cycle Policy and Super-Convergence</h4><p>Super-convergence is just a fancy name created by Smith to describe the training converges very rapidly. Particularly, the optimizer can achieve comparable performance with significantly less iterations than normal using 1-cycle policy.</p><p>1-cycle policy is very similar to CLR policy. The only difference is that it only involves one cycle.</p><ol><li>Run LR test to determine the maximum bound, set minimum bound as 1/10 of the maximum bound</li><li>Set the <em>stepsize</em> of one cycle that is smaller than the total number of iterations used by other policies.</li><li>Allow the learning rate to decrease several orders of magnitude less than the minimum bound for the remaining iterations</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*kkBNaEwx1NifCsXu59MFVQ.png" /><figcaption>The learning rate over the 1-cycle policy. After finishing one cycle, allow the learning rate to continually decrease for several orders of magnitude.</figcaption></figure><p>The philosophy of CLR policy is still applicable to 1-cycle policy. Large learning rate enforces the optimizer to search for a flat minimum. After reaching the plateau, the optimizer needs to settle into the local minimum so the learning rate should be small.</p><h4>Conclusion</h4><p>Beside blindly apply 3e-4 with adam optimizer, a more scientific methodology is possible. Combining LR range test with CLR policy or 1-cycle policy releases our burden on choosing the proper learning rate.</p><p>Another insight from Smith is that large learning rate is in itself a kind of regularization. Since large learning rate introduces larger gradient noise, it leads to better generalization. This insight raises a further question on how to balance different kinds of regularization method (batch size, momentum, weight decay), which is the topic of the next post.</p><h4>Reference</h4><p>[1]: Keskar, Nitish Shirish, et al. “On large-batch training for deep learning: Generalization gap and sharp minima.” <em>arXiv preprint arXiv:1609.04836</em> (2016).</p><p>[2]: Smith, Leslie N. “Cyclical learning rates for training neural networks.” <em>2017 IEEE Winter Conference on Applications of Computer Vision (WACV)</em>. IEEE, 2017.</p><p>[3]: Smith, Leslie N., and Nicholay Topin. “Super-convergence: Very fast training of neural networks using large learning rates.” <em>arXiv preprint arXiv:1708.07120</em> (2017).</p><p>[4]: Smith, Leslie N. “A disciplined approach to neural network hyper-parameters: Part 1 — learning rate, batch size, momentum, and weight decay.” <em>arXiv preprint arXiv:1803.09820</em> (2018).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4669d5bb568" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deep-learning-hk/some-techniques-in-deep-learning-optimization-1-learning-rate-b4669d5bb568">Some Techniques in Deep Learning Optimization (1): Learning Rate</a> was originally published in <a href="https://medium.com/deep-learning-hk">Deep Learning HK</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>