<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by IMAO on Medium]]></title>
        <description><![CDATA[Stories by IMAO on Medium]]></description>
        <link>https://medium.com/@gnaijnaijnait?source=rss-4a848b1dafcf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bEQ_t7cyy34rIxabgd3Pmw@2x.jpeg</url>
            <title>Stories by IMAO on Medium</title>
            <link>https://medium.com/@gnaijnaijnait?source=rss-4a848b1dafcf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 05:32:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gnaijnaijnait/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I learned to stop worrying and love da bomb]]></title>
            <link>https://medium.com/zeals-tech-blog/how-i-learned-to-stop-worrying-and-love-da-bomb-a361eaa8f8b5?source=rss-4a848b1dafcf------2</link>
            <guid isPermaLink="false">https://medium.com/p/a361eaa8f8b5</guid>
            <dc:creator><![CDATA[IMAO]]></dc:creator>
            <pubDate>Mon, 06 Dec 2021 14:33:18 GMT</pubDate>
            <atom:updated>2021-12-07T06:42:18.937Z</atom:updated>
            <content:encoded><![CDATA[<h4>Scotty factor for real if you want to be magical</h4><blockquote>The first part of the content below contains mythical jargon meant to be funny <em>(</em>because I am weird<em>)</em>.<br>The second part will be in plain English <em>(</em>I hope<em>)</em>.<br>Viewer discretion is advised.</blockquote><h4>TL;DR</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Ft9SVhg6ZENw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dt9SVhg6ZENw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Ft9SVhg6ZENw%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6bc59ab461f0d3791bfaa8f7031c4d7f/href">https://medium.com/media/6bc59ab461f0d3791bfaa8f7031c4d7f/href</a></iframe><p>(<em>Inspired by a true story.</em>)</p><h3>Part One</h3><p>Once upon a time, I got this mission.</p><h4>Setup</h4><p><strong>Theme</strong>: <br>Mutual Assured Protection</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/128/1*4xHf_t1_ta2hFBxr2IwFKQ.png" /><figcaption>Facepalm, a secret society.</figcaption></figure><p><strong>Epic:<br></strong>As a secret society fellow,<br>I want to build HTML <a href="https://civilization.fandom.com/wiki/Culture_Bomb_(Civ6)">culture bomb</a>s,<br>so that minorities have defenses on the web.</p><p><em>Story:</em><br>For making those bombs,<br>I want to collect massive HTML pages,<br>so I can make those bombs huge.</p><p>How should I proceed? You know what they say: when in doubt…, C4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*_ZIqUJcwuaXrkoPb34daZg.jpeg" /><figcaption><a href="http://www.quickmeme.com/meme/35gk6h">Do you have doubt? Use C4 — Misc — quickmeme</a></figcaption></figure><p><a href="https://www.tensorflow.org/datasets/catalog/c4">C4, a colossal, cleaned version of Common Crawl’s web crawl corpus</a>, is powerful yet hard to get. (<em>Except for </em><a href="https://huggingface.co/datasets/allenai/c4"><em>allenai/c4 · Datasets at Hugging Face</em></a><em>, kudos!</em>) And my <em>magistrate</em> and <em>magistri</em> want HTML. C4 has none.</p><p>So I figure, let me just grab HTML parts from Common Crawl (CC) based on C4’s URLs.</p><h4>Confrontation</h4><p>It sounds easy because I know tools like <a href="https://github.com/michaelharms/comcrawl">comcrawl</a> and <a href="https://pypi.org/project/cdx-toolkit/">cdx-tools</a>. Yet, the CC index server is not for rapid queries. It requires at least 3 seconds to tell whether it has a URL. I know what you are thinking: why not run concurrent queries? Yes, I have tried, still that slow. More importantly, <a href="https://groups.google.com/g/common-crawl/c/3QmQjFA_3y4/m/vTbhGqIBBQAJ">CC forbids</a>.</p><p>It’s time to study the secret of CC indexing and perform some magic!</p><p>For each monthly crawl, a magical scroll of locator spells called cluster.idx <a href="https://pywb.readthedocs.io/en/latest/manual/indexing.html#zipnum-sharded-index">sorts and gathers URLs into groups</a> of 3,000. Every group has a cryptic code hinting where that 3,000 URLs and their HTML contents are. Allow me to show an “anchor” below, for example.</p><pre>org,secret,facepalm)/pass/you-shall-not.html 20210102030405 cdx-00042.gz 123456 789 10</pre><p>The above locator spell says this. For a certain group, its first record on <em>2021–01–02</em> at <em>03:04:05</em> is http(s)://facepalm.secret.org/pass/you-shall-not.html. That is the anchor to where this and the rest 2,999 URLs’ actual contents are in cdx-ooo42.gz. In that gz file, those 3,000 records take place from byte offset <em>123456</em> and occupy <em>789</em> bytes. And this is the <em>10</em>th locator spell of the cluster.idx.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*Esfs1ssw3FgO9smpypVtrg.png" /><figcaption><a href="https://stormbornwitch.tumblr.com/post/121395962782/everyone-loses-things-and-i-am-no-exception-so">Witchcraft For The Weather Witch (tumblr.com)</a></figcaption></figure><p>What if C4 has a page https://facepalm.secret.org/wubba-lubba-dub-dub.html that is not the first record of some group? Then yours truly will have to mix some ingredients to make a locator pendant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S_S4r6CZUX_YNh71kyPUpQ.jpeg" /><figcaption><a href="https://www.simonandschuster.com/books/Dictionary-of-Ancient-Magic-Words-and-Spells/Claude-Lecouteux/9781620553749">Dictionary of Ancient Magic Words and Spells Book by Claude Lecouteux</a></figcaption></figure><p>1. Parse every anchor and spawn anchor prefixes of it. For the above anchor, we will have the anchor itself and its three ancestors (<em>or descendants</em>) except for <a href="https://en.wikipedia.org/wiki/Top-level_domain">heavenly realm</a>s:</p><pre>org,secret)/</pre><pre>org,secret,facepalm)/</pre><pre>org,secret,facepalm)/pass</pre><pre>org,secret,facepalm)/pass/you-shall-not.html</pre><p>2. Parse the target C4 URL for similar prefixes. With wubba-lubba-dub-dub page, we may have three <a href="https://github.com/internetarchive/surt">runic inscription</a>s:</p><pre>org,secret)/</pre><pre>org,secret,facepalm)/</pre><pre>org,secret,facepalm)/wubba-lubba-dub-dub.html</pre><h4>Resolution</h4><p>Now we feel the traction on the pendant. It attracts the anchor most similar to [2] among [1]. If we are lucky, we may find an anchor’s inscription says:<br>org,secret,facepalm)/we-will-meet-again.html.<br>That will suffice. Because CC does lexicographic sort, and that group is likely where wubba-lubba-dub-dub belongs!</p><p>How effective this pendant is then, you ask? When I spawn 64 threads of the pendant on Colab, despite its slow CPU, I can check 80,000 URLs/hour.</p><p>🎊 Mission accomplished! 🎉</p><p>So you fancy my recipe? Sure, just join our secret society already. 🖖</p><h3>Part Two</h3><p>No more kidding from now on. Please grant me some more time. I promise I will organize the actual code and share it here.</p><p>The first part follows a storytelling method called “<a href="https://web.stanford.edu/class/cs224u/readings/shieber-writing.pdf">rational reconstruction</a>.” The method focuses on a typical <a href="https://en.wikipedia.org/wiki/Three-act_structure">three-act structure</a>: <em>setup</em>, <em>confrontation</em>, and <em>resolution</em>. Like any good story, it is seldom about the true events and their chronology.</p><h4>Why?</h4><p>Because I like to share a lesson that I learn the hard way, by comparing the reconstructed version with its actual events.</p><p>You may have recognized Theme, Epic, and Story being some Agile terminologies. Also, the epic is in User Story format while the story is in the <a href="https://jtbd.info/replacing-the-user-story-with-the-job-story-af7cdee10c27">Jobs-to-Be-Done (JTBD)</a> style. Let’s put them aside for a while.</p><p>I’ve joined <a href="https://bigscience.huggingface.co/">HuggingFace BigScience workshop</a>. One of its subgroups has this research question: what if we fine-tune a language model with HTML tags? To answer that question, we need data. One may crawl the web from scratch. But for research, it is important to be comparable and reproducible.</p><p>So the first attempt was to curate an HTML corpus based on <a href="https://github.com/jcpeterson/openwebtext">OpenWebText</a> URLs. I thought that was simple. I could only work on BigScience stuff during the nights and the weekends. Yet I still measured the effort to be a one-week job. I couldn’t be more wrong.</p><h4>What went wrong?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*3h1nJqVi0bpTDmnvBJ_J4A.gif" /><figcaption><a href="https://metawitches.com/2021/01/25/a-discovery-of-witches-season-2-episode-3-recap-in-which-diana-meets-the-queen-and-plans-a-journey/">Unless your threads are magical.</a></figcaption></figure><p>Despite I know CC index server is slow, I still spend a week (<em>approximately 8 work hours</em>), on how I can increase the throughput by threads. I should do the math first. No matter how much multi-threading can offer, it will never be enough for CC index server due to its capacity.</p><p><strong>So Why did I choose to chase the wrong goose? Because it supposed to be easy, and we wanted to have a minimal viable solution. To some extent, </strong><a href="https://en.wikipedia.org/wiki/Confirmation_bias"><strong>confirmation bias</strong></a><strong> it is.</strong></p><p>Also, it seems hard to know CC’s coverage of OpenWebText. (<a href="https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf"><em>For GPT-2, OpenAI intentionally designs WebText to be not just a part of CC </em></a><em>.</em>) We may end up wasting a lot of time on asking CC for something isn’t there.</p><p>Thus, we turn to C4. (<em>And at least C4 covers a half of OpenWebText.</em>) The problem remains. Since the first week of the project, I really should have worked on the aforementioned algorithm. Which is simply building a URL tokenizer and <a href="https://en.wikipedia.org/wiki/Inverted_index">inverted index</a>es on the fly. (<em>Maybe someday someone will build an Elasticsearch index for CC.</em>) There is a third way of <a href="https://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/">columnar index</a>, but allow me to omit that. The actual issue is, the whole thing takes me two months to get here, but I only have been exploring for a couple of weeks.</p><h4>How shall we fix that?</h4><p>“Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb” had uncommon casting and <a href="https://en.wikipedia.org/wiki/Improvisational_theatre#In_film_and_television">retro-scripting</a>. Peter Sellers played the president, a mad scientist, and an Air Force captain. I was myself’s stakeholder, product owner, and developer simultaneously. Being the judge, jury, and executioner is already bad. No brilliant product owners can foresee the situation, let alone giving a good estimate. Unless someone has boldly gone there before.</p><p>So, one of my lessons learned here is:</p><blockquote>One should try <strong>both</strong> a plan A and a plan B as <strong>early</strong> as possible.<em><br></em><strong>Never</strong> “if not A then B.”</blockquote><p>And that is something neither User Story nor JTBD will explicitly tell you. (<em>Some </em><a href="https://en.wikipedia.org/wiki/Use_case"><em>Use Case</em></a><em> may do, but that’s for another article</em>.) So, please never ever try to have estimates for epics/stories. Instead,</p><blockquote>Measure <strong>specialty</strong> and <strong>size</strong> of <strong>workforce</strong> for <br>finding, prioritizing, developing, and verifying <strong>alternatives</strong>.</blockquote><p>Of course, we need to meet the deadline. Yet, unless it is life-threatening, a deadline is never deadly. I’d rather preserve a considerably large amount of time for exploration, which is a sound survival strategy to me. <strong>Fail fast, learn faster.</strong> (<em>Ideally, I would say to balance the exploration–exploitation tradeoff, but you know how it is.</em>) After all, if it’s truly a deadly deadline, there will be simply no time for estimation.</p><p>(<em>Few can cheat </em><a href="https://en.wikipedia.org/wiki/Kobayashi_Maru"><em>Kobayashi maru</em></a><em> as Kirk has done, and that is fictional. An anecdote here: Kirk’s hack makes himself the most respected/feared captain in the universe, so no one dares to attack first. That plot resembles </em><a href="https://en.wikipedia.org/wiki/The_Irresponsible_Captain_Tylor"><em>The Irresponsible Captain Tylor</em></a><em>’s lucky yet deadly risky decision</em>.<em> Life-threatening or not, trust me, you don’t want them to be your bosses. Remember, Star Trek is competence porn. Its characters are all extremely capable. I know I am not.</em>)</p><p>In a three-act structure, confrontation is essential for the protagonist to grow. If we truly desire to use Persona, User Story, JTBD, etc., since those methods borrow semantics of storytelling, then we must ask ourselves:</p><blockquote>What <strong>confrontation</strong> can make <strong>both</strong> the user and us <strong>stronger</strong>?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*E_KnKe57NZFo73K2zHkoxA.jpeg" /><figcaption><a href="https://9gag.com/gag/aOb2y7R">A quote from the very opening scene of House of Cards. — 9GAG</a></figcaption></figure><p>I wish I had put that into my stories. If I had been more focused on potential obstacles, especially for those I knew…</p><p>Anyhow, what did I learn to stop worrying and love da bomb? Nothing was actually simple until done. If you must have it,</p><blockquote>One should multiply every estimate by four <em>(or by eight in my case)</em>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a361eaa8f8b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zeals-tech-blog/how-i-learned-to-stop-worrying-and-love-da-bomb-a361eaa8f8b5">How I learned to stop worrying and love da bomb</a> was originally published in <a href="https://medium.com/zeals-tech-blog">The Zeals Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get the Fine-tuning Just Right]]></title>
            <link>https://gnaijnaijnait.medium.com/get-the-fine-tuning-just-right-771d1c400d68?source=rss-4a848b1dafcf------2</link>
            <guid isPermaLink="false">https://medium.com/p/771d1c400d68</guid>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[IMAO]]></dc:creator>
            <pubDate>Mon, 14 Dec 2020 12:22:08 GMT</pubDate>
            <atom:updated>2020-12-15T02:22:52.076Z</atom:updated>
            <content:encoded><![CDATA[<h4>TRANSFER LEARNING</h4><h4>For NTCIR-15 FinNum-2 and DialEval-1 Tasks</h4><blockquote><em>Disclaimer: this post won’t use any author’s or editorial </em>we<em>. The overconfident title and content are opinionated. The work described here is a team effort, but my words don’t express the view of my employer and co-authors.</em></blockquote><p><em>On December 14, 2020, </em><a href="http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings15/NTCIR/toc_ntcir.html"><em>NTCIR-15 conference</em></a><em> has concluded successfully. I have led two teams to work on the conference’s shared task, namely </em><a href="http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings15/pdf/ntcir/01-NTCIR15-OV-DIALEVAL-ZengZ.pdf"><em>DialEval-1</em></a><em> and </em><a href="http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings15/pdf/ntcir/01-NTCIR15-OV-FINNUM-ChenC.pdf"><em>FinNum-2</em></a><em>. My teams’ works have reached the first and the second places for DialEval-1 and FinNum-2, respectively. Following </em><a href="https://medium.com/zeals-tech-blog/ntcir-15-dialogue-evaluation-task-dialeval-1-d41178624566"><em>my previous post</em></a><em> that introduces NTCIR and DialEval, this one will describe what FinNum-2 is, report the contribution of the team works, and propose a workflow that generalizes how I apply transfer learning for tasks like DialEval and FinNum.</em></p><h3>The road so far</h3><p>Since I work as an NLP-ML engineer at <a href="https://medium.com/zeals-tech-blog">ZEALS</a>, dealing with practical concerns of transfer learning is a part of my job description. Despite all the attentions to Transformers and <a href="https://huggingface.co/transformers/bertology.html">BERTology</a>, getting those models fine-tuned just right is still nontrivial, and some perspectives of fine-tuning are less studied than others. This post will use DialEval-1 and FinNum-2 as examples to show what those under-invested subjects are. (Should it be any unexplained acronym, please kindly refer to <a href="https://medium.com/zeals-tech-blog/ntcir-15-dialogue-evaluation-task-dialeval-1-d41178624566">the previous post</a>.)</p><p>FinNum is a task for fine-grained numeral understanding in online financial texts. For tweets about stock price, <a href="http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings14/pdf/ntcir/01-NTCIR14-OV-FINNUM-ChenC.pdf">FinNum-1</a> at <a href="http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings14/NTCIR/toc_ntcir.html">NTCIR-14</a> has tried to disambiguates the meaning of the numerals and found it insufficient for pragmatic uses, such that FinNum-2 at NTCIR-15 wishes to identify the link between cashtags and numerals. Figure 1 illustrates an example linkage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*kRB_sp1VsPYJDHHdp0jZbA.png" /><figcaption>Figure 1. Numeral Attachment. Image Credit: <a href="http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings15/pdf/ntcir/01-NTCIR15-OV-FINNUM-ChenC.pdf">FinNum-2 Overview Paper</a></figcaption></figure><p>Whilst there is more than one way to do it, FinNum-2 organizers frame the task as a binary classification problem. Although DialEval-1 doesn’t enforce a particular scheme for its subtasks ND and DQ, all participants model them as multiclass classification and multi-label classification, respectively. Therefore, those two tasks provide an opportunity of examining one general and practical approach with various classification settings and datasets, at least that’s the way I see them.</p><h3>Devil’s Bargain</h3><p>Declaring something “general” and “practical” could be subjective, so allow me to quote the official reports for the actual outcomes from my teams, <a href="http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings15/pdf/ntcir/08-NTCIR15-DIALEVAL-JiangM.pdf">IMTKU</a> and <a href="http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings15/pdf/ntcir/05-NTCIR15-FINNUM-JiangM.pdf">CYUT</a>*. (For detail statistics, please check the papers by the task organizers: <a href="http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings15/pdf/ntcir/01-NTCIR15-OV-DIALEVAL-ZengZ.pdf">DialEva-1</a>, <a href="http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings15/pdf/ntcir/01-NTCIR15-OV-FINNUM-ChenC.pdf">FinNum-2</a>.)</p><p>For DialEval-1:</p><blockquote>At STC-3, none of the participant runs are statistically significantly better than the BL-LSTM model. However, at DialEval-1, IMTKU-run2 outperforms the baselines significantly (𝑝 &lt; 0.5) in Chinese DQ subtask in terms of NMD.</blockquote><p>For FinNum-2:</p><blockquote>(CYUT) experiment on both BERT and RoBERTa. … We find that the vanilla BERT and RoBERTa achieve good performances.</blockquote><p>(It is actually XLM-RoBERTa, and I will explain the difference soon.)</p><p>It all sounds well and good, except that I’m trading time and energy with big bad pretrained models. If applied blindly, generalization ability of those models could be disappointing, not to mention that “practical” sometimes implies special treatments that won’t hold for new tasks. So I figure, at least a repeatable procedure may be a baseline that I can improve upon.</p><h3>I’m a man of letters</h3><p>Some may consider it obvious, the bigger the model, the better the performance, isn’t it? However, as the FinNum-2 organizers have pointed out,</p><blockquote>“The difference between the performances of different teams using the same architecture may be caused by the preprocessing procedures and the hyperparameter settings.”</blockquote><p>In my experience, there are more factors besides preprocessing and hyperparameter. The diagram below demonstrates my decision process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*nwqICSDDXjjaRpCap0x_HQ.png" /><figcaption>The Activity Diagram of My Decision Process for Fine-Tuning</figcaption></figure><p>Here I’d like to emphasize the subtle differences among tokenizers. For example, XLM-RoBERTa uses unigram SentencePiece, whereas RoBERTa uses byte-pair encoding (BPE). Roughly speaking, BPE recursively couples bytes in a left-to-right fashion. For certain languages, the resultant subwords may be less than ideal. That being said, since different models predetermine their own tokenizations with variations of training data sets and training schemes, I’m not aware of any universal answer. Empirically, I find that unigram SentencePiece and whole-word masking based models are more likely to outperform WordPiece and BPE based models.</p><p>Admittedly, the process isn’t perfect. For example, I have found an intriguing error case, “2C”:</p><blockquote>In the test set, a tweet uses [it] to refer the link between global warming and the stock price of Tesla. In the training and the development sets, however, all of the “2C” and “2c” stand for “to see.” This case indicates that, both informal usages of tweet and the domain knowledge of stocks can use some more efforts.</blockquote><p>Will polishing the rest steps from the above diagram help resolving this issue? I don’t know yet, but I’d like to think that data augmentation is promising. As for keywords of those steps, I will simply put a hyperlinked list together here for your reference:</p><ul><li><a href="https://medium.com/@makcedward/how-subword-helps-on-your-nlp-model-83dd1b836f46">BPE, WordPiece, and SentencePiece</a></li><li><a href="https://datascience.stackexchange.com/questions/53875/what-is-whole-word-masking-in-the-recent-bert-model">Whole-Word Masking</a></li><li><a href="http://papers.nips.cc/paper/8717-when-does-label-smoothing-help.pdf">When Does Label Smoothin<strong>g</strong> Help?</a></li><li><a href="https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html">Mixed Precision</a></li><li><a href="https://sgugger.github.io/the-1cycle-policy.html">One-cycle Policy</a></li><li><a href="https://mlwhiz.com/blog/2019/03/30/transfer_learning_text_classification/">Discriminative Fine-tuning / Gradual unfreezing</a></li><li><a href="https://github.com/ohmeow/blurr/blob/master/blurr/modeling/core.py#L18">Default</a> and <a href="https://ohmeow.github.io/blurr/modeling-text2text-summarization/#summarization_splitter">Custom</a> layer groups.</li><li><a href="https://medium.com/@lessw/new-deep-learning-optimizer-ranger-synergistic-combination-of-radam-lookahead-for-the-best-of-2dc83f79a48d">Ranger</a>; <a href="https://towardsdatascience.com/an-intuitive-understanding-of-the-lamb-optimizer-46f8c0ae4866">LAMB</a></li><li><a href="http://don&/#39;t%20decay%20the%20learning%20rate,%20increase%20the%20batch%20size">Don’t Decay the Learning Rate, Increase the Batch Size.</a></li><li>Text data augmentations: <a href="https://github.com/jasonwei20/eda_nlp">EDA</a>, <a href="https://github.com/makcedward/nlpaug">nlpaug</a>, <a href="https://github.com/QData/TextAttack">TextAttack</a>, etc.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*QPXc7o_sLTgcU416eUdX-g.png" /><figcaption>Just Kidding. <a href="https://xkcd.com/518/">https://xkcd.com/518/</a></figcaption></figure><h3>Carry on</h3><p>One more thing that may be worth mentioning is, with One-cycle Policy, I have managed to fine-tune models for DialEval-1 and FinNum-2 with fewer epochs than other teams. For example, I only need 5 epochs for FinNum-2, while others usually require 30. Not only it benefits me by the time saved, the lowered energy consumption may also contribute a little to the world. <a href="https://www.technologyreview.com/2020/12/04/1013294/google-ai-ethics-research-paper-forced-out-timnit-gebru/">A recent controversy</a> has reminded me that the pursuit of transfer learning is at the expense of environment. Honestly, I don’t know if me trying to find more efficient ways to do transfer learning can really make a difference, but it surely makes me feel less guilty. Interestingly, some endeavors of machine learning may also have indirect contributions for the greater good. For example, to go beyond Mixed Precision, <a href="https://www.technologyreview.com/2020/12/11/1014102/ai-trains-on-4-bit-computers/">training models with a 4-bit chip may liberate personal data from cloud</a>, and then personalized and federated models may <a href="https://www.theguardian.com/lifeandstyle/2020/dec/13/how-artificial-intelligence-helped-me-overcome-my-dyslexia">empower more underrepresented people</a>.</p><p>* Conventionally almost all participants in NTCIR name the teams by colleges regardless the involvement of industries, even if I’m the first author on behalf of my company.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=771d1c400d68" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NTCIR-15 Dialogue Evaluation Task DialEval-1]]></title>
            <link>https://medium.com/zeals-tech-blog/ntcir-15-dialogue-evaluation-task-dialeval-1-d41178624566?source=rss-4a848b1dafcf------2</link>
            <guid isPermaLink="false">https://medium.com/p/d41178624566</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[fastai]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[IMAO]]></dc:creator>
            <pubDate>Thu, 03 Sep 2020 12:16:26 GMT</pubDate>
            <atom:updated>2020-12-14T05:16:01.106Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc/4.0/</cc:license>
            <content:encoded><![CDATA[<h4>TRANSFER LEARNING</h4><h4>A brief introduction and a simple approach</h4><p><em>DialEval-1 consists of two subtasks: Nugget Detection (ND) and Dialogue Quality (DQ). They aim to evaluate customer-helpdesk dialogues automatically. The ND subtask is to classify whether a customer or helpdesk turn is a nugget, where being a nugget means that the turn helps towards problem solving; and the DQ subtask is to assign quality scores to each dialogue in terms of three criteria: task accomplishment, customer satisfaction, and efficiency. The official evaluation results of 18 runs are received from 7 teams. IMTKU (Dept. of Information Management, Tamkang University, Taiwan) and I, on behalf of </em><a href="https://zeals.co.jp/"><em>ZEALS</em></a><em>, by utilizing </em><a href="https://huggingface.co/xlm-roberta-base"><em>XLM-RoBERTa</em></a><em> via </em><a href="https://github.com/huggingface/transformers/"><em>HuggingFace Transformers</em></a><em>, </em><a href="https://github.com/fastai/fastai"><em>fastai</em></a><em>, and </em><a href="https://github.com/ohmeow/blurr"><em>blurr</em></a><em>, achieve top-1 or top-2 in terms of various evaluation metrics for both ND and DQ.</em></p><h3>NTCIR? DialEval? ND? DQ?</h3><p><a href="http://research.nii.ac.jp/ntcir/index-en.html"><strong>N</strong>II (National Institute of Informatics) <strong>T</strong>estbeds and <strong>C</strong>ommunity for <strong>I</strong>nformation Access <strong>R</strong>esearch, a.k.a. NTCIR</a>, is a series of evaluation workshops designed to enhance research in information access technologies, including information retrieval, question answering, text summarization, extraction, etc. It can be seen as an Asia-Pacific counterpart of <a href="https://trec.nist.gov/">TREC</a> (Text REtrieval Conference; America), <a href="http://www.clef-initiative.eu/">CLEF</a> (Conference and Labs of the Evaluation Forum; Europe), and <a href="http://fire.irsi.res.in/">FIRE</a> (Forum for Information Retrieval Evaluation; Southern Asia).</p><p>DialEval-1 is one of the initiatives, e.g., <a href="https://dstc9.dstc.community/home">DSTC</a> (Dialog System Technology Challenges), that study the complexity of the dialogue phenomenon and various dialogue-related problems. Unlike typical tasks of DSTC, however, DialEval-1 focuses on reducing the cost of corpus curation and data analytics. Although <a href="https://en.wikipedia.org/wiki/Wizard_of_Oz_experiment">WOZ (Wizard-of-Oz)</a>, a common approach nowadays, can distribute the workloads of data collection, annotation, and evaluation to crowd-sourcing, the time spent on reaching an agreement for annotation/evaluation standard is nonetheless linear, if not worse. In other words, human annotation and evaluation have two types of issues:</p><ul><li>Scalability: costly and hard to decentralize;</li><li>Measurability: likely unrepeatable/inconsistent even for the same system.</li></ul><p>To overcome the issues, DialEval-1 proposes to assess dialogues automatically. Given a customer-helpdesk dialogue (Figure 1), for example, can a system predict which turn of the dialogue is helpful, and by how much?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AO6vkac5hyifYzH9zO35oQ.png" /><figcaption>Figure 1. An example of a dialogue between Customer © and Helpdesk (H). The original dialogue on Weibo in Simplified Chinese is on the right, and its English translation is on the left. [1][2]</figcaption></figure><p>Therefore, DialEval-1 continues ND and DQ subtasks of Short Text Conversation (STC-3) at NTCIR-14 [1], and further constructs a new test collection, such that the performance evaluation would be fair and realistic.</p><h3>What to predict, exactly?</h3><p>Allow me to briefly describe what the expected outcomes are, and how to evaluate them. There will be no math formulae. If the reader is interested in rigorous definitions, please kindly refer to the overview papers [1][2].</p><h4>Nugget Detection</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w7Gs6hpC6W-PLacNpIoA5A.png" /><figcaption>Figure 2. Nugget state transitions [1][2]</figcaption></figure><p>A nugget is a turn by either Helpdesk or Customer. It helps Customer transition from Current State (including Initial State) towards Target State (i.e., when the problem is solved). STC-3 and DialEval-1 organizers define the following 7 nugget types:</p><ul><li>CNaN / HNaN: Customer or Helpdesk’s <em>non</em>-nuggets that are irrelevant to the problem-solving situation;</li><li>CNUG / HNUG: Customer or Helpdesk’s <em>regular</em> nuggets that are relevant to the problem-solving situation;</li><li>CNUG* / HNUG*: Customer or Helpdesk’s <em>goal</em> nuggets that confirm and provide solutions, respectively;</li><li>CNUG0: Customer’s <em>trigger</em> nuggets that initiate the dialogues with certain problem descriptions.</li></ul><h4>Dialogue Quality</h4><p>Using subjective scores to quantify the quality of a dialogue as a whole. The organizers define 3 score types:</p><ul><li>A-score: Accomplishment; has the problem solved? To what extent?</li><li>S-score: Satisfaction; how satisfied Customer is with the dialogue;</li><li>E-score: Effectiveness; how effective and efficient the dialogue is.</li></ul><p>Each score is on a 5-point scale of ranks, ranging from -2 to 2.</p><h4>Evaluation Metrics</h4><p>Since the issues are about inconsistent human assessment, the gold standard of the datasets is not trivial classes or ranks but distributions. Roughly speaking, the distributions are votes from annotators for certain class/rank. Such that the organizers evaluate a system’s prediction of class/rank by comparing how similar the predicted distribution to the gold standard’s. More specifically, the metrics for ND are Root Normalized Sum of Squares (RNSS) and <a href="https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence">Jensen-Shannon Divergence</a> (JSD). and the ones for DQ are Normalized Match Distance (NMD) and Root Symmetric Normalized Order-aware Divergence (RSNOD). Again, I will spare the readers with the math formulae, please see the papers if interested.</p><h3>How do we approach the tasks?</h3><p>Despite the architecture differences, almost all participants of STC-3 modeled ND and DQ as classification tasks. We adopt the same tactic for DialEval-1, and then pay more attention to tokenization and optimization. Because STC-3 showed that none of architectures from participants outperformed the baselines of BiLSTM with GloVe [1][2]. Therefore our approach is simply fine-tuning the state-of-the-art pre-trained models with well-established tricks of tokenization and optimization.</p><h4>Tokenization</h4><p>Based on our preliminary trials, we find that XLM-RoBERTa works well for both Simplified Chinese and English. Although the rationale behind it hasn’t been fully examined, we speculate that the <a href="https://github.com/google/sentencepiece">sentencepiece</a>-based unigram subwords may have been helpful. Besides that, in order to simulate the turn structure of a dialogue, we not only utilize XLM-RoBERTa’s special tokens, namely BOS (beginning of sentence; &lt;s&gt;), EOS (end of sentence; &lt;/s&gt;), and SEP (separator of sentences; &lt;/s&gt; &lt;/s&gt;), but also customize several tokens in fastai style to provide some minimal context.</p><p>For example, consider a tokenized turn below:</p><pre>xxlen ▁3 &lt;s&gt; xxtrn ▁1 xxsdr ▁customer ▁@ ▁China ▁Uni com ▁Customer ▁Service ▁in ▁Gu ang dong ▁Shi t ! ▁What ▁is ▁your ▁staff ▁service ▁doing ▁on ▁earth ? ▁I ▁have ▁called ▁the ▁staff ▁service ▁for ▁3 ▁hours , ▁but ▁no ▁one ▁answer ▁my ▁phone ▁call . ▁It ▁is ▁no ▁wonder ▁that ▁customer ▁evaluation ▁is ▁so ▁bad . ▁Shi t ! ▁I ▁am ▁at ▁Kang le ▁Middle ▁Road . &lt;/s&gt;</pre><p>Where xxlen and xxtrn stand for length of the dialogue in turns and the position of each turn of the dialogue, respectively. The numbers right next to them provide certain features of turns. The same trick goes with xxsdr that differentiates whether the sender is Customer or Helpdesk. When a turn’s context says xxtrn _1 xxsdr _customer, the nugget type is almost definitely CNUG0.</p><p>As for DQ, a whole dialogue can be tokenized in a similar fashion, where xxlen could be useful for certain quality scores:</p><pre>xxlen ▁3 &lt;s&gt; xxtrn ▁1 xxsdr ▁customer ▁@ ▁China ▁Uni com ▁Customer ▁Service ▁in ▁Gu ang dong ▁Shi t ! ▁What ▁is ▁your ▁staff ▁service ▁doing ▁on ▁earth ? ▁I ▁have ▁called ▁the ▁staff ▁service ▁for ▁3 ▁hours , ▁but ▁no ▁one ▁answer ▁my ▁phone ▁call . ▁It ▁is ▁no ▁wonder ▁that ▁customer ▁evaluation ▁is ▁so ▁bad . ▁Shi t ! ▁I ▁am ▁at ▁Kang le ▁Middle ▁Road . &lt;/s&gt; &lt;/s&gt; xxtrn ▁2 xxsdr ▁help desk ▁Hello ! ▁We ▁are ▁sorry ▁for ▁the ▁in con veni ence . ▁100 10 ▁is ▁our ▁service ▁hot ▁line . ▁We ▁may ▁not ▁answer ▁your ▁phone ▁call ▁during ▁the ▁busy ▁hour ▁of ▁tele traf fic . ▁We ▁sincer ely ▁ap ologi ze ▁for ▁that ! ▁What ▁can ▁I ▁do ▁for ▁you ? ▁Thank ▁you ! &lt;/s&gt; &lt;/s&gt; xxtrn ▁3 xxsdr ▁customer ▁The ▁Uni com ▁Internet ▁access ▁in ▁Z ha oq ing ▁Nur sing ▁School ▁can &#39; t ▁be ▁connected . ▁What ▁is ▁wrong ▁with ▁it ? ▁You ▁have ▁repair ed ▁it ▁for ▁the ▁whole ▁afternoon ▁in ▁the ▁area . ▁What ▁are ▁you ▁doing ▁on ▁earth ? ▁Shi t ! ▁Why ▁can ▁the ▁China ▁Mobile ▁service ▁hot line ▁be ▁got ▁through ? ▁Shi t ! ▁The ▁service ▁hot line ▁can &#39; t ▁be ▁got ▁through ▁the ▁whole ▁morning . ▁ 651 ▁I ▁bought ▁a ▁watch ▁last ▁year ▁and ▁the ▁service ▁hot line ▁can &#39; t ▁be ▁got ▁through ▁within ▁24 ▁hours . ▁I ▁won &#39; t ▁for give ▁you ! ▁No ▁phone ▁call ▁is ▁answered ! &lt;/s&gt;</pre><h4>Optimization</h4><p>Thanks to the great works of HuggingFace, fastai, and blurr, a stable fine-tuning scheme enables us to rapidly trial-and-error for a sufficiently good combination of hyper-parameters. For instance, the core steps for fine-tuning a model of ND can be as short as this:</p><pre>dls = ... # <em>fastai&#39;s</em> Dataloaders<br>lrnr = Learner(<br>  dls,<br>  HF_BaseModelWrapper(hf_model), # <em>blurr</em>&#39;s <em>HuggingFace</em> model wrapper <br>  opt_func=partial(<em>SOME_OPTIMIZER</em>, decouple_wd=True),<br>  loss_func=LabelSmoothingCrossEntropyFlat(),<br>  metrics=[<br>    accuracy,<br>    partial(top_k_accuracy, k=2),<br>    F1Score(average=&#39;weighted&#39;),<br>    MatthewsCorrCoef(),<br>    ...<br>  ],<br>  cbs=[HF_BaseModelCallback],<br>  splitter=hf_splitter,<br>  path=<em>DATA_DIR</em>,<br>).to_fp16()<br>lrnr.create_opt()</pre><pre>for ...:<br>  # iteratively decrease <em>base_lr</em> and/or <em>factor</em><br>  lrnr.fit_one_cycle(<em>n_epoch</em>, lr_max=slice(<em>base_lr/factor</em>, <em>base_lr</em>))</pre><p>Admittedly, there are many moving parts of this fine-tuning scheme. After all, the most time consuming step for fine-tuning is <em>Grad Student Algorithm </em>(a.k.a. <em>Grad Student Descent</em>), i.e., figure out a nice combination of magical numbers, a stable optimizer, a reasonable loss function, and other techniques such as discriminative fine-tuning and mixed precision. Fortunately, with the help of slanted triangular learning rates, it only takes minutes to finish each fit_one_cycle(…).</p><p>One particular choice we realize is that no need to do gradual unfreezing. It works pretty well with AWD-LSTM, but somewhat insignificant for fine-tuning Transformers.</p><h3>That’s it?</h3><p>Yes, mostly. In our experiences, although there are more techniques to explore, the bottom line is that, unless we discover a substantially better architecture (for classification) and/or an alternative modeling perspective (that is not simply classification), a good beginning (of tokenization and optimization) <strong><em>almost</em></strong> assures success.</p><p>While the official report and datasets won’t be published until the end of year 2020 [2], STC-3 datasets are available for anyone wants to give it a shot: <a href="https://sakai-lab.github.io/stc3-dataset/">https://sakai-lab.github.io/stc3-dataset/</a> [1].</p><h3>REFERENCES</h3><ol><li><a href="http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings14/pdf/ntcir/01-NTCIR14-OV-STC-ZengZ.pdf">Zeng, Kato, and Sakai: Overview of the NTCIR-14 Short Text Conversation Task: Dialogue Quality and Nugget Detection Subtasks, Proceedings of NTCIR-15, 2019.</a></li><li>Zeng et al. (TBA): Overview of the NTCIR-15 Dialogue Evaluation Task, Proceedings of NTCIR-15, to appear, 2020.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d41178624566" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zeals-tech-blog/ntcir-15-dialogue-evaluation-task-dialeval-1-d41178624566">NTCIR-15 Dialogue Evaluation Task DialEval-1</a> was originally published in <a href="https://medium.com/zeals-tech-blog">The Zeals Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[如果現在，一個大三專題生]]></title>
            <link>https://gnaijnaijnait.medium.com/%E5%A6%82%E6%9E%9C%E7%8F%BE%E5%9C%A8-%E4%B8%80%E5%80%8B%E5%A4%A7%E4%B8%89%E5%B0%88%E9%A1%8C%E7%94%9F-566428ec1564?source=rss-4a848b1dafcf------2</link>
            <guid isPermaLink="false">https://medium.com/p/566428ec1564</guid>
            <dc:creator><![CDATA[IMAO]]></dc:creator>
            <pubDate>Mon, 24 Feb 2020 12:41:41 GMT</pubDate>
            <atom:updated>2020-02-24T14:15:38.065Z</atom:updated>
            <content:encoded><![CDATA[<h4>名校之外，爬上陡坡，不畏強風和暈眩，仰視前途茫茫</h4><h4>在一片纏繞交錯的網中</h4><p>想對自己的人生進行 <a href="https://medium.com/@zonble">XDDDD (eXtreme Due Day Driven Development)</a>¹ 嗎？勉強聽聽此人的當年不勇吧。關於「輸在起跑點」，我略知一二。</p><p>直到高中畢業擠進私校資科為止，生於台灣最窮、鐵路最晚電氣化的地方，長於芋頭番薯 (ō͘-á han-chî) 軍教家庭，要說有什麼還比下有餘的話，恐怕就那萬惡的十八趴而已^H^H。大三做專題那年，我加入了號稱系上最操的實驗室，想著好歹有機會考上名校研究所洗洗身分吧？當指導教授強烈反對我們課餘上補習班時，我才開始意識到，自己一直以來光顧著練碼農口訣，田都還種不好，更別提怎麼賣藝了。別人家的小孩搞不好要進大公司（可靠消息指出，某上 t 下 m 殺 v 社只收台清交）了，我的田要是以後吃不到聖地亞哥金坷垃該怎麼辦？</p><p>（接下來大概可以猜出我馬齒徒增又動搖的程度）</p><p>那年，Google 才剛開始變得好用（之前我都用 Infoseek），最受歡迎的手機還是 Nokia 6150, Steve Jobs 剛完成王子復仇，還沒人知道將來 iCEO 系列某產品會改變大家走路的方式。至於 GPU, 當然是打 WoW 用的啊！BTW, ATI 可能比 nVidia 划算（但我喜歡遊戲類型不同，沒必要花那錢就是了）。回頭說到碼農藝，Java 剛紅起來，Python 還沒什麼知名度，最多人用機器學習模型是 Hidden Markov model 而不是不夠深層的類神經網路。要上那裡找武功心法？<a href="https://en.wikipedia.org/wiki/Massive_open_online_course">MOOCs</a> 這縮寫的原文才剛出現，<a href="https://en.wikipedia.org/wiki/Joel_Spolsky">談軟體的約耳</a>還沒真<a href="https://en.wikipedia.org/wiki/Jeff_Atwood">認識 coding 的恐怖</a>，於是<a href="https://en.wikipedia.org/wiki/Stack_Overflow">堆疊溢位</a>只有教科書上那一種意義。我則是因為讀了不少施威銘研究室的書而沾沾自喜。</p><p>然後，老闆說要做 J2EE, J2ME, 和一堆 J 來 E∨¬E 去沒中文書的東西。另一方面，我光能確保自己不會被當一屁股就不錯了。亂成一團了啊 XDDDD</p><h4>在一片穿織交錯的網中</h4><p>看來看去，沒有高潮就只能怪床了？接著想起那名叫高中的床，有群老師對英文教育十分熱衷。好吧，英文確實是顯而易見的瓶頸，但總覺得不是問題根源。還記得小時候看偉人傳記說到英美某些科學家，為了學習最先進的知識而自學俄文法文等等。這類偉人除了不看鮭魚力爭迴游以及慧根超多條之外，還有什麼值得效法的嗎？</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*RRcV6juXdtCabkdoHp42vQ.jpeg" /><figcaption>by courtesy of <a href="https://imgur.com/Ru1BtMP">https://imgur.com/Ru1BtMP</a></figcaption></figure><p>我大三時，網際網路還不夠方便好用，反過來說，焦慮資訊爆炸和逃避選擇自由還不是講得出來的藉口。就算到了今天，在已經看不太懂的英文資料中，要怎麼挑出最好的，或最低限度，去掉有問題的，也不見得輕鬆。如果非要給這障礙一個期限的話，我希望…… 當然限量是殘酷的，時間要花在刀口上，我是這麼想的。眼前爛帳一團，不剪不理不忍卒睹的話，能不能<a href="https://en.wikipedia.org/wiki/Ocean%27s_Eleven#Plot">像 “The Amazing” Yen 那樣穿過去</a>？</p><p>觀眾當然能一眼看穿 “<a href="https://en.wikipedia.org/wiki/Galaxy_Quest#Plot">historical documentaries</a>” 的把戲。現實中<a href="https://quoteinvestigator.com/2013/10/20/no-predict/">戰力高強的人們有句老話：推測難，推測未來特別難</a>。回頭看來，這幾乎是學習的永恆課題，非監督式學習與強化學習自然也不例外。大學與教授的任務和中小學不同，指導未必監督，強化得看造化。從<a href="https://en.wikipedia.org/wiki/Multi-armed_bandit">吃角子老虎機的經典模型</a>來看，似乎有個什麼一以貫之的吾道。</p><h4>A carpet of leaves illuminated by the moon, around an empty grave.</h4><p>如果有人宣稱吃角子老虎（或其他類似狀況）有什麼必勝法，那麼大概就像「鑰匙掉了，在路燈下找，雖然未必掉在這裡，但這裡才有光」諸如此類的。另一方面也像那個常被引錯的孫子兵法名言，百戰頂多不殆而非百勝，總之要知己知彼。</p><p>一、找出自己最重要的目標。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*QKL8Zm9qOeIHgp334Hat9g.jpeg" /><figcaption>by courtesy of <a href="https://www.meme-arsenal.com/en/create/meme/646250">https://www.meme-arsenal.com/en/create/meme/646250</a></figcaption></figure><p>二、搞清楚自己在時間、財力、身心能力與健康、支持系統（家人與朋友；如果是醫藥和設計相關科系，還會有共筆或槍手）等方面有多少本錢與限制。</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fc1oP0AunRfP7a%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fc1oP0AunRfP7a%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fc1oP0AunRfP7a%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="243" frameborder="0" scrolling="no"><a href="https://medium.com/media/121f484e6a5c930b983660ed0bc1184c/href">https://medium.com/media/121f484e6a5c930b983660ed0bc1184c/href</a></iframe><p>三、釐清自己、環境、任務之間的落差。有多少是可直接觀察的（路燈下）？而間接觀察到的，像是月光照得閃閃發亮的東西，有多大機會是樹葉而不是金子？如果任務是尋寶，花力氣挖開底下的墓，要是空無一物會賠上什麼，損失多少？反之，中大獎的獎到底多大？</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*X0hCdPUCJ6bkChzCklmb6w.gif" /><figcaption>by courtesy of <a href="http://phdcomics.com/comics/archive.php?comicid=1689">http://phdcomics.com/comics/archive.php?comicid=1689</a></figcaption></figure><p>四、在老把戲 (exploitation) 和新把戲 (exploration) 之間保持身心平衡。說起來容易，做起來更像是跟自己玩心理戰（畢竟來自行為心理學）。不知道怎麼拿捏輕重緩急還要持之以衡？有人試 GTD, 有人試 Scrum, 也有人試 <a href="https://www.books.com.tw/products/0010811638">Pick Three</a> …, 無論如何，這些都不是可證偽的科學方法，當然更不是什麼真理。或者說，能量化的恐怕只有每天花多少時間做了什麼而已。大原則是，不管走那條路，最需要日常鍛鍊的是學習如何學習。現在怎麼樣不確定，但多年前（遠目）台灣的中學是不教批判思考的，在我看來這是比英文不夠好還要嚴重得多的弱點。在判斷上述方法論什麼適合自己時，知道有什麼分析驗證的技巧能用，甚至看出這些方法是怎麼設計出來的，到頭來會比那些方法更受用。另一方面，為了「天天」向上，勞逸結合也是必備技能。野人獻曝一下，每日服用適量的美劇英劇有助於放鬆，至於英文能力到底會進步多少嘛，嗯，好歹「感覺」還不壞。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aF34dDAV-SNB1wr1I6NpWA.png" /><figcaption>by courtesy of <a href="https://www.xkcd.com/1881/">https://www.xkcd.com/1881/</a></figcaption></figure><p>話說回來，我最愛的還是「<a href="http://www.structuredprocrastination.com/">有組織地拖拖拉拉</a>」了。（例：讀一堆方法論的書然後自以為已經實戰完畢這樣。）</p><h4>什麼故事在那頭等待結束？</h4><p>上了大四之後，我決定放棄準備名校筆試，就只參加本來系上研究所推甄。換句話說是，當時的我沒有餘力，而且我真正想要的是研究自己感興趣的題目，而不是從名校畢業。當時不 K 書準備筆試付出的代價是，幾年後考博士班資格考準備得更加吃力。小時不努力換來的時間空間是不是划算，說真的我至今連量化標準都訂不出來。加上那些年養成的許多壞習慣，實在算不上是什麼成功經驗。<a href="https://www.plurk.com/p/kb2zxm">所有的悲劇一開始都叫 workaround</a>². 如果以「不殆」為指標，在蓋棺論定之前，我想我只能繼續練習評估，什麼是明天會呈受不起的風險。有些人喜歡以 “carpe diem” （活在當下）暗指及時行樂。雖然大致上沒錯，但下半句是 “<a href="https://en.wikipedia.org/wiki/Carpe_diem#Meaning">quam minimum credula postero</a>.”</p><p><strong>Attribution: 我超愛 zonble 的哏</strong></p><ol><li><a href="https://medium.com/@zonble">https://medium.com/@zonble</a> 最近的副標</li><li><a href="https://zonble.net/">https://zonble.net</a> 以前的副標</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=566428ec1564" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Have Freedom of Choice | Using ML → Democratizing NLP]]></title>
            <link>https://gnaijnaijnait.medium.com/have-freedom-of-choice-using-ml-democratizing-nlp-9ec8b2c7f39e?source=rss-4a848b1dafcf------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ec8b2c7f39e</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ethics]]></category>
            <category><![CDATA[democracy]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[IMAO]]></dc:creator>
            <pubDate>Tue, 10 Dec 2019 10:14:03 GMT</pubDate>
            <atom:updated>2019-12-12T04:27:01.810Z</atom:updated>
            <content:encoded><![CDATA[<h4>I want to | When → So I can 授人以魚不如授人以漁</h4><blockquote>この <a href="https://jtbd.info/replacing-the-user-story-with-the-job-story-af7cdee10c27">job story</a> は <a href="https://qiita.com/advent-calendar/2019/zeals">Zeals Advent Calendar</a> のために書かれています。<br>I write this <a href="https://jtbd.info/replacing-the-user-story-with-the-job-story-af7cdee10c27">job story</a> for <a href="https://qiita.com/advent-calendar/2019/zeals">Zeals Advent Calendar</a>.<em><br></em>這則 <a href="https://jtbd.info/replacing-the-user-story-with-the-job-story-af7cdee10c27">job story</a> 是為 <a href="https://qiita.com/advent-calendar/2019/zeals">Zeals Advent Calendar</a> 而寫的。</blockquote><h4>Take-Home Messages</h4><ol><li>情報非対称性を解消：コミュニケーション壁を下げる<br>Eliminate information asymmetry: lower barriers to communications<br>消弭資訊不對稱：降低溝通門檻</li><li>オープンソースソフトウェアと無料のクラウドサービスを利用する<br>Utilize open source softwares and free cloud services<br>善用開源軟體與免費雲端服務<br>• <a href="https://huggingface.co/">huggingface</a>, <a href="https://www.fast.ai/">fastai</a>, <a href="https://allennlp.org/">AllenNLP</a>, etc.<br>• Google Colab, Paperspace Gradient, etc.</li><li>コードだけでなく、すべてのためにレビューを求める<br>Seeking reviews for everything, not just for code<br>不光是程式碼，任何事都該徵詢審閱</li></ol><h3>§1 Talk Is Cheap, but……</h3><p><a href="https://medium.com/@TianJianJiang/rename-and-repurpose-this-blog-what-can-imao-do-and-how-can-i-help-218f58db5685">IMAO</a>,<br>実用的な話はちょっと優れている場合があります。<br><em>actionable</em> talk can be slightly better.<br>實用的嘴砲倒有可能好點。</p><p>何かを民主化するなどの善意のために、誰もが参加することが難しい場合は矛盾したものになるでしょう。<br>For such a good intention as “democratizing something,” it would be an oxymoron if it is difficult for anyone to participate.<br>以「民主化某事」這般良善意圖而言，若人人參與並非易事就自相矛盾了。</p><p>(So the talk shop here is…)<br>僕が NLP で ML を使用する時、選択の自由が欲しい場合、NLP がより民主的になるように行われるか。<br>When I want to have freedom of choices for using machine learning (ML) on natural language processing (NLP), how the job is done such that NLP will be more democratic.<br>該怎麼做，才能在我想用 ML 搞 NLP 時擁有選擇的自由，使 NLP 更民主化。</p><p>僕の頭の中に浮かんでいるのは：<br>What’s been floating in my mind are:<br>我腦中浮現的要點是：</p><ul><li>ヒント/トリックよりもコンセプトを優先する<br>Prefer concept over tips/tricks<br>概念先於技巧</li><li>入手しやすい<br>Accessible<br>高可用性</li><li>責任を負う<br>Accountable<br>問責</li></ul><h3>§2 Barriers and Inequalities</h3><p>社会経済的な階層、母語、心身機能の障害などの明白なものに加えて、ソフトウェアやハードウェアの需給にも（哲学的に）多くの障壁と不平等が存在すると主張します。 ただし、エッジケースをテストするために、変なものをいくつか挙げてみましょう：<br>Besides the obvious ones such as socioeconomic strata, native tongues, physical/mental challenges, etc., I’d argue that (philosophically) many barriers and inequalities exist within supply and demand of softwares and hardwares, too. For testing edge cases, though, let me list some weird one:<br>除了社經地位、母語、身心障礙之類明顯的例子之外，我認為（哲學上）軟硬體的供需裡也有許多藩籬與不平等。為檢驗極端狀況，容我講些奇怪的：</p><blockquote>I want to code in Python with the speed of C and the type system of Haskell.<a href="#5af3">¹</a><br>I want fast computation but I can’t afford or don’t like the limited choices.<a href="#a740">²</a><br>I want free, liberal, and secure Internet.<a href="#9ac9">³</a><br>I want……</blockquote><p>それらはわがままて聞こえるかもしれませんが、あなたがポイントをわかるて、（異議）論点の奥の手があると確信しています。原則として、特定のヒント/トリックではなく、僕は（実用的な）一般論に固守するですが、例として提供します。<br>Those may sound willful and spoiled, but I’m sure you get the point and have (counter-)arguments up your sleeves. In principle, I like to stick to (actionably) general concepts instead of specific tips/tricks, but I will still provide some as examples.<br>這聽來任性驕縱，不過相信你懂我的意思且自有高見。原則上，我寧可聚焦於（可行的）一般概念而非特定技巧，但還是會舉一兩個為例。</p><p>要するに、最初に自分たちがどんな選択肢を持っているかを知らなければならず、それから交渉します。<br>In short, first we must know what choices we have, and then we bargain.<br>簡單地講，首先得知道我們有什麼選擇，然後才能討價還價。</p><p>誰もが完璧な英語を話し、直接情報を受け取る先進国では生まれないかもしれませんが、追いついて助け合うことができます。 望ましいソフトウェア/ハードウェアの選択肢はないかもしれませんが、代替手段としてコミュニティに参加したり、コミュニティを開始したりすることもできます。 そのため、自己教育と生涯学習が不可欠です。<br>We may not be born in a developed country that everyone speaks perfect English and receives first hand information, but we can try to catch up and help each other. We may not have preferable choices of software/hardware, but we can try joining or even starting a community for alternatives. So self-education and lifelong learning is crucial.<br>我們或許不是生在已開發國、能講流利英語、能獲得第一手資訊的人。但我們可以一起試著趕上，並互相幫助。我們或許沒有理想的軟硬體方案，但我們可以試著加入甚至成立提供替代方案的社群。因此，自我教育與終身學習是關鍵。</p><figure><img alt="You can’t always get what you want." src="https://cdn-images-1.medium.com/max/832/1*QD6o23DUPOCmpqnSAPadGg.jpeg" /><figcaption><a href="https://multimedia.thenational.ae/assets/multimedia/interactives/flat-graphics/rolling-stones.jpg">https://multimedia.thenational.ae/assets/multimedia/interactives/flat-graphics/rolling-stones.jpg</a></figcaption></figure><p>プログラマーになれて幸運です。 ほとんどさまざまな選択肢があります。 私は何かを学び、貢献することができます。 さらに良いことに、多くの才能のある人々もそれをやっています。 ほとんどの場合、僕は彼らの素晴らしい作品を他の人に紹介するだけです。 たとえば、<a href="https://huggingface.co/">huggingface</a> は最近、<a href="https://medium.com/huggingface/distilbert-8cf3380435b5">より小さく、速く、安価なBERTの蒸留バージョン</a>をリリースしました。 それが何だかわからない？ 最初に英語（基本的な不平等の1つ）を学び、次に <a href="https://www.fast.ai/">fast.ai</a> のような無料のオンラインコースに登録します。<br>I’m lucky to be a programmer. Virtually various choices are available to me. I can learn something and then contribute. Even better, plenty talented people are doing it, too. Most of time I just refer their great works to others. For example, <a href="https://huggingface.co/">huggingface</a> recently released a <a href="https://medium.com/huggingface/distilbert-8cf3380435b5">distilled version of BERT that is smaller, faster, and cheaper</a>. Don’t know what it is? First learn English (one of fundamental inequalities), and then enroll free online courses like <a href="https://www.fast.ai/">fast.ai</a>’s.<br>我有幸生為碼農。幾乎擁有各種選擇。我可以學點什麼然後回饋。更棒的是，許多傑出人士也這麼做。於是我通常只要引介他們的傑作即可。例如， <a href="https://huggingface.co/">huggingface</a> 最近發表了<a href="https://medium.com/huggingface/distilbert-8cf3380435b5">更小、更快、更划算的 distilled version of BERT</a>. 不懂？先學英文（基本的不平等之一），然後報名 <a href="https://www.fast.ai/">fast.ai</a> 之類的免費線上課程吧！</p><p>一方、制限を知ることはより重要かもしれません。 たとえば、僕の知る限り、<a href="https://github.com/IDSIA/sacred">IDSIA-sacred</a> と同じくらい洗練されていますが、それは暗黙的に擬似ランダム状態を追跡するだけで、その構成（ほとんどの場合ハイパーパラメーター）は通常匿名の辞書です。 名前付きの構成がありますが、残念ながら、A / Bテスト用の以前のスナップショットモデルをリサイクルする再現可能なトレーニングサイクルを使用できるように、型付きの不変の構成でランダム性を明示的に制御する必要があります。 したがって、私はしぶしぶ車輪を再発明します。 擬似コードは次のようなものです：<br>On the other hand, it may be more important to know the limitations. For example, to my best knowledge, as sophisticated as <a href="https://github.com/IDSIA/sacred">IDSIA-sacred</a> is, it only implicitly tracks pseudo-random states, and its configurations (mostly hyper-parameters) are usually just anonymous dictionaries. Although one may have a <a href="https://sacred.readthedocs.io/en/stable/configuration.html#named-configurations">named config</a>, unfortunately, I need explicit controls on randomness with typed and immutable configurations, so that I can have reproducible training cycles that recycle preceding snapshot models for A/B testing. Therefore I reinvent wheels reluctantly. Their pseudocode is something like:<br>另一方面，知道極限在那可能更重要。好比說，就我所知，精密如 <a href="https://github.com/IDSIA/sacred">IDSIA-sacred</a> 也僅在內部隱含追蹤 pseudo-random states, 而且 configurations（多用於 hyper-parameters）通常也只是 anonymous dictionary. 雖然可以改用 named config, 無奈我需要的是透過型別與唯讀 configurations 來明確地控制隨機性，才能有可回收過去 snapshot model 進行 A/B 測試的可重現 training cycle. 於是我勉為其難重造輪子。虛擬碼像這樣：</p><pre>conf_0 = typed_conf(corpus, task, model, env, hyperparams, ...)<br>cycle_0 = training(conf_0, ..., prev_cycle=None)</pre><pre>conf_a = conf_0.replicate_except(change(hyperparams.someone, &#39;a&#39;))<br>cycle_a = training(conf_a, ..., prev_cycle=cycle_0)</pre><pre>conf_b = conf_0.replicate_except(change(hyperparams.someone, &#39;b&#39;))<br>cycle_b = training(conf_b, ..., prev_cycle=cycle_0)</pre><p>実験は些細なことのように思えるかもしれませんが、ランダムな状態と非決定的/不安定なアルゴリズムにより、実装が複雑になり、エラーが発生しやすくなります。 確かに、IDSIA-sacredなツールや他のツールを使用してそれを行う賢い方法がありますが、これはとにかく単なる例です。 小さくて退屈なこともありますが、内部には不平等が隠されています：数千のプログラマーやGPUが自由に使えるのであれば、不規則性（人為的ミスやランダムな結果）は平均化される可能性が高いので、気にする必要はありません。 この例の貧弱さにもかかわらず、それは huggingface と IDSIA-sacred のスローガンの両方を実際に反映できます。「一度に１つのコミット；すべての実験は神聖です」。 言い換えれば、再現可能な実験を完全に制御することで、Google ColabやPaperspace Gradientなどの制限があるでも無料サービスでかなりの量の作業を行うことができます。１つのコミット/実験もその二酸化炭素排出量も無駄にされません。<br>The experiment may seem trivial, but random states and non-deterministic/unstable algorithms make implementations complex hence human-error-prone. Surely there are clever ways to do it with IDSIA-sacred or other tools, this is just an example anyways. As small and tedious as it can be, an inequality hides inside: you may not care about them if thousands of programmers and GPUs are at your disposal, since irregularities (human errors and random outcomes) will likely be averaged out. Despite all the meagerness of this example, it can really reflect both huggingface and IDSIA-sacred’s slogans: “<em>one commit at a time; every experiment is sacred</em>.” In other words, with total controls of reproducible experiments, we can have considerable amount of works done with limited yet free services such as Google Colab and Paperspace Gradient. Neither a single commit/experiment nor its carbon dioxide emissions will be wasted.</p><p>理想的には、<a href="https://sciencedryad.wordpress.com/2014/01/25/grad-student-descent/">大学院生の（勾配）降下法</a>だけでなく、すべての人の知識を管理し広める機会が増えるポジティブフィードバックループが発生します。<br>Ideally, a positive feedback loop will then occur, not just of <a href="https://sciencedryad.wordpress.com/2014/01/25/grad-student-descent/">graduate student descent algorithms</a>, but of more opportunities to curate and spread knowledge for everyone.</p><p>些細なヒントや愚かなコードスニペットを本当に読みたい場合は、<br>In case you really want to read my insignificant tips and stupid code snippets, please refer to <br><a href="https://nbviewer.jupyter.org/github/tianjianjiang/playground/blob/master/IMDb_baseline.ipynb">tianjianjiang/playground/IMDb_baseline.ipynb</a></p><p>上記のノートブックは、次のトリックを適用します。<br>The above notebook applies following tricks:</p><ol><li>不安定な入力を生成する可能性のある並列プロシージャを使用しないでください。<br>NEVER use parallel procedures that can generate unstable input;</li><li>双方向LSTMとともにcuDNNのdropoutを使用しないでください（基になるC / C ++関数を制御できない場合）。<br>NEVER use cuDNN’s dropout along with bidirectional LSTM (unless you can control the underlying C/C++ functions);</li><li>4つの可能な擬似乱数ジェネレータのすべての状態を覚えておいてください。<br>Remember every state of four possible pseudo random number generators;</li><li>MixedPrecisionの動的損失スケールのすべての変更を思い出してください。<br>Remember every change of dynamic loss scale for MixedPrecision;</li><li>バッチサイズを考慮した学習率を見つけます。<br>Find a learning rate that takes batch size into account;</li></ol><p>(2) のdrouptouは特に難しい：それなしではトレーニング速度は~10％遅くなりますが、それを使用すると成功または失敗の妄想を引き起こします —— 精度はランダムに5％以上変動する可能性があります！<br>The dropout thing of (2) is particularity challenging: training speed will be ~10% slower without it, but using it will induce you a delusion of success or failure —— its accuracy can vary more than 5% randomly!</p><p>日本のコーパスにとって、これらのトリックを適用するのはそれほど難しくありません<br>For Japanese corpora, it won’t be too hard to apply those tricks to<br>• <a href="https://github.com/n-waves/multifit/blob/master/notebooks/MLDoc-JA-multifit_fp16.ipynb">n-waves/multifit/notebooks/MLDoc-JA-multifit_fp16.ipynb</a><br>• <a href="https://github.com/n-waves/multifit/blob/master/notebooks/CLS-JA.ipynb">n-waves/multifit/notebooks/CLS-JA.ipynb</a>.</p><h3>§3 Checks and Balances</h3><p>「<a href="https://machinelearningtokyo.com/2019/12/08/ai-ethics-with-andrew-ng-and-deeplearning-ai/">AI倫理：Andrew Ng教授のdeeplearning.ai x MLT</a>」では、参加者はこの質問に対する答えを推論するよう求められました。<br>In “<a href="https://machinelearningtokyo.com/2019/12/08/ai-ethics-with-andrew-ng-and-deeplearning-ai/">AI ETHICS: Prof. Andrew Ng’s deeplearning.ai x MLT</a>”, participants were asked to reason their answers for this question:</p><blockquote>Should a university accepts convicted offenders of human rights (e.g., dictators)?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*4aZ31ekXz4X_kVdbY9CLeQ.png" /><figcaption><a href="https://blog.deeplearning.ai/blog/the-batch-amazons-surveillance-network-ai-that-gets-the-facts-right-deepfakes-get-regulated-predicting-volcanic-eruptions">https://blog.deeplearning.ai/blog/the-batch-amazons-surveillance-network-ai-that-gets-the-facts-right-deepfakes-get-regulated-predicting-volcanic-eruptions</a></figcaption></figure><p>道徳的推論の他の思考実験のように、簡単な答えはありません。標準的な答えは言うまでもありません。 それは、究極の答えを決定することよりも、異なる意見の間で対話を開くことに関するものです。 さらに、すべての政府/企業の責任であると主張する代わりに、草の根の方法でAI倫理の実用的なコードを策定したいと考えています。<br>Like any other thought experiments of moral reasoning, there won’t be straightforward answers, let alone a standard one. It is more about opening a dialogue among different opinions than dictating an ultimate answer. Furthermore, we want to formulate actionable code of AI ethics in a grassroots way, instead of arguing that it’s all government/enterprise responsibilities.</p><p>僕のデータ倫理チームでは、この単純なコードを作成しました。多くのプログラマーがコードレビューを行い、科学コミュニティがピアレビューを行うのと同様に、<a href="https://docs.google.com/presentation/d/1uCw9jcNb3KxnoaIolU0R5ybxAlEUoQf7sGuScWrx8T8/edit#slide=id.g75d2f03c60_0_5">データの影響を受ける人々の多様な視点からレビューを求めます</a>。悲しいことに、私たちは偏見や差別について多くの話を聞きましたが、それらの問題に対する単純な答えはありませんでした。 誰もが（少なくともすべてのプログラマーが）毎日できることは、セカンドオピニオンを得ることです。<br>In my team of data ethics, we came up this simple code: just like many programmers do code review and the science community does peer review, <a href="https://docs.google.com/presentation/d/1uCw9jcNb3KxnoaIolU0R5ybxAlEUoQf7sGuScWrx8T8/edit#slide=id.g75d2f03c60_0_5">we shall also seek reviews from diverse perspectives with people impacted by the data</a>. Sadly we all heard too many stories about bias and discrimination, and again no simple answers to those issues. What everyone (at least every programmer) can do everyday is to get a second opinion.</p><p>たとえば、台湾の子供向けのeラーニングチャットボットを作成する場合、私たちの1人が小学校の教科書のコーパスを確認し、<br>For example, if we are going to create an e-learning chatbot for Taiwanese kids, hopefully one of us will review the corpus of elementary school textbooks, and then realize that it contains gender stereotypical lessons like</p><blockquote>「父は新聞を読むために早く起きます 、母は家事のために早起きし、私は学校に行くために早起きします。」 <br>“My father gets up early for reading newspapers, my mother gets up early for housekeeping, and I get up early to got to school.”</blockquote><p>その時点から、私たちはついにそれについて何をすべきかを議論することができます。（画像処理を行う場合、お母さんのパンプについて誤解されることもあります。#KuToo）<br>From that point onward, we can finally discuss what to do about it. (If you do image processing, you may also be misled about mom’s pumps. #KuToo)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*yWkXWVKBwdmr6l4vlfTVMw.jpeg" /><figcaption><a href="https://medium.com/%E7%B4%85%E9%B6%B4%E8%80%81%E7%88%B8%E9%99%A3%E7%B7%9A/%E9%97%9C%E6%96%BC%E7%94%B7%E6%80%A7%E7%9A%84%E9%8C%AF%E8%AA%A4%E6%80%A7%E5%88%A5%E5%88%BB%E6%9D%BF%E5%8D%B0%E8%B1%A1-%E4%B8%80-f54341be451e">紅鶴老爸陣線：關於男性的性別刻板印象（一）</a></figcaption></figure><p>このプロセスは浅薄なもので、権限の分離がないように見える場合があります。 僕が賭けているのは、なんらかの大賞でもありません。ほとんどの場合、「<a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwjBrKO24azmAhWIMt4KHT0TCVwQFjAAegQIARAB&amp;url=https%3A%2F%2Fmedium.com%2Fs%2Fstory%2Fso-you-want-to-be-a-research-scientist-363c075d3d4c&amp;usg=AOvVaw08YD_V3yOFvk8COUeIM5_I">単著者の論文ほど疑わしいものはない</a>」という議論のように、私は自分自身を信用していません。 そして、今日のテクノロジーの影響を受けている人々の数から判断すると、おそらくプログラマーはWorld Wide Webに渡って何らかのsuperpowerを持っているため、責任があります。近い将来、CI/CDには、「<a href="https://medium.com/dair-ai/examining-gender-and-race-bias-in-sentiment-analysis-systems-b04b269a653">感情分析システムにおける性別と人種の偏見の調査</a>」や「<a href="https://github.com/ramybaly/News-Media-Reliability">報道の事実性の予測とニュースメディアソースのバイアス</a>」などのルーチンを含める必要があります。<br>The process may seem superficial and lacking separation of powers. What I am betting on is not about some grand prize, either. Most of time I just don’t trust myself, like the argument of “<a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwjBrKO24azmAhWIMt4KHT0TCVwQFjAAegQIARAB&amp;url=https%3A%2F%2Fmedium.com%2Fs%2Fstory%2Fso-you-want-to-be-a-research-scientist-363c075d3d4c&amp;usg=AOvVaw08YD_V3yOFvk8COUeIM5_I">there is nothing more suspicious than a single-author paper</a>.” And judging by how many people may be affected by technologies today, perhaps programmers do have some superpower across the World Wide Web, hence certain responsibility. Maybe in the near future, continuous integration/deployment will have to include routines like “E<a href="https://medium.com/dair-ai/examining-gender-and-race-bias-in-sentiment-analysis-systems-b04b269a653">xamining Gender and Race bias in Sentiment Analysis Systems</a>” or “<a href="https://github.com/ramybaly/News-Media-Reliability">Predicting Factuality of Reporting and Bias of News Media Sources</a>.”</p><p>ところで、まもなく日本のAIコミュニティが開催する<a href="http://ai-elsi.org/archives/898">機械学習と公平性に関する声明とシンポジウム</a>があります。<br>By the way, soon there will be <a href="http://ai-elsi.org/archives/898">a symposium of ML faireness held by Japanese AI communities</a>.</p><h4>Addendum</h4><p>選挙制度は民主主義の重要な部分ですが、僕は今まで意図的にそれについて話すことを避けています。 一般的の投票システムに欠陥があることが多いだけでなく、理論的にはランク集約の問題はNP-hardです。 僕は自分の好みがローカルで個人的であることを好みます。 :)<br>Although electoral systems are a significant part of democracy, I deliberately avoid talking about it until now. Because not only popular voting systems are often flawed, but theoretically rank aggregation problem is NP-hard. I prefer my preferences staying local and personal. :)</p><h4>Footnotes</h4><ol><li>Sooner or later we will be able to use <a href="https://www.tensorflow.org/swift">Swift for TensorFlow</a></li><li>Alternatives to GPU ain’t easy:<br>Intel: <a href="https://01.org/dnnl">https://01.org/dnnl</a>;<br>AMD: <a href="https://rocm.github.io/dl.html">https://rocm.github.io/dl.html</a><br>Google: <a href="https://github.com/pytorch/xla">https://github.com/pytorch/xla</a></li><li>Watch <a href="https://en.wikipedia.org/wiki/Mr._Robot">Mr. Robot</a>, the most accurate TV show about hacktivism.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ec8b2c7f39e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sane Trials of Machine Learning on a Virtual Shoestring (3/3)]]></title>
            <link>https://gnaijnaijnait.medium.com/sane-trials-of-machine-learning-on-a-virtual-shoestring-3-3-b0dfa8caa7ca?source=rss-4a848b1dafcf------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0dfa8caa7ca</guid>
            <category><![CDATA[fastai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[allennlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[IMAO]]></dc:creator>
            <pubDate>Tue, 08 Oct 2019 04:00:55 GMT</pubDate>
            <atom:updated>2019-10-08T04:12:51.014Z</atom:updated>
            <content:encoded><![CDATA[<h4>Usability: effectiveness, efficiency, and satisfaction.</h4><h3>TL;DR and for what you may not see in this post.</h3><ul><li>Know (what are) self and others (<a href="https://medium.com/@TianJianJiang/sane-trials-on-a-virtual-shoestring-1-3-c7a7bddae26">part-1</a>);</li><li><strong>Control PRNG states</strong>, not just their seed numbers (<a href="https://medium.com/@TianJianJiang/sane-trials-of-machine-learning-on-a-virtual-shoestring-2-3-46456f579c2c">part-2</a>);</li><li>Uncover side effects, e.g.:</li><li>— <a href="https://docs.fast.ai/callbacks.fp16.html">Mixed Precision</a>, <strong>mind the gap of the dynamic loss scale </strong>[<a href="https://github.com/fastai/fastai/blob/master/fastai/callbacks/fp16.py#L101">1</a>][<a href="https://github.com/fastai/fastai/blob/master/fastai/callbacks/fp16.py#L114">2</a>];</li><li>— cuDNN, <strong>avoid using its built-in dropout;</strong></li><li>— Non-determinism/Instability of multi-core/process/thread usages.</li></ul><p><em>This series of posts shares my thoughts on how to get deterministic outcomes of machine learning experiments at an affordable cost. As usual, most of the content is only my murmur. Feel free to skip them and check </em><a href="https://github.com/tianjianjiang/playground/blob/master/IMDb_baseline.ipynb"><strong><em>the Jupyter notebook for this series</em></strong></a><em> directly. It tries </em><a href="https://aclweb.org/anthology/P18-1031"><em>Universal Language Model Fine-tuning for Text Classification</em></a><em> with a more recent version of fastai library. Another friendly reminder: technical jargons are just fancy names for references.</em></p><h3>Satisfaction</h3><p>If we are already on a shoestring, what properties of a sane ML trial can ensure us happily ever after?</p><figure><img alt="Math Work" src="https://cdn-images-1.medium.com/max/612/1*0055kHJpDAJq3_InJUo1Qw.png" /><figcaption>Courtesy of <a href="https://xkcd.com/2207/">xkcd: Math Work</a></figcaption></figure><p>For me, knowing what is to be unknown will be a start. My inner security comes from embracing the known/intriguing unknowns (science!) without fear of unknown/undesired unknowns (who-must-not-be-named?). There’s gotta be a way to avoid the latter. When doing trial-and-error on a virtual shoestring, however, it is bound to be labor-intensive for leaving no stones unturned. Therefore, it is only natural that one will invent smart algorithms for a smaller time/space complexity. Except for all magic, always remember to negotiate the price in advance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*GsI5_q9k5B9qU_yjOzrQtg.gif" /><figcaption>Courtesy of <a href="https://en.wikipedia.org/wiki/Once_Upon_a_Time_(TV_series)">Once Upon a Time</a> via <a href="https://mathildia.tumblr.com/post/134820911990/magic-always-comes-with-a-price-on-bringing">mathildia’s tumblr</a></figcaption></figure><p>Not unlike the residents of Enchanted Forest, machine learning practitioners can’t afford to not know every fine print, a. k. a. side effect, in an agreement.</p><h3>Efficiency</h3><p>One of my favorite tricks is <a href="https://github.com/cybertronai/gradient-checkpointing">gradient-checkpointing</a>, because it cares about what are <a href="https://medium.com/@TianJianJiang/sane-trials-on-a-virtual-shoestring-1-3-c7a7bddae26">the same</a> nodes such that it can decide when (not) to recompute them. <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">One of Hugging Face’s articles explains it well along with related tactics for efficiency</a>. But what I want to point out is about checkpoints in a more general sense, at least for software engineering. A checkpoint trades processing time and programming efforts for memory space and fault tolerance. In other words, it checks and balances something critical, and for PyTorch it usually means gradients’ chain rule calculation.</p><p>So we have a chain of responsible nodes that are obvious subjects for check-and-balance. Yet sometimes we might sign up some more magics for a better-looking balance sheet. Since gradient-checkpointing could slow things down, why not use <a href="https://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/index.html#faq-general">mixed precision</a> for time and space at once? Here’s the fine print: if you want to apply <a href="https://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/index.html#faq-general__section_on4_qgf_djb">dynamic loss scaling</a>, own the side-effect. Take a look at fastai’s implementation of <a href="https://github.com/fastai/fastai/blob/master/fastai/callbacks/fp16.py#L64">MixedPrecision class</a>, you may find relevant snippets like:</p><pre>def __init__(...):<br>...<br>    self.loss_scale = ifnone(loss_scale, 2**16 if dynamic else 512)<br>...<br>def on_backward_begin﻿﻿(...):<br>...<br>    ret_loss = last_loss * self.loss_scale<br>...<br>def on_backward_end(...)﻿﻿﻿:<br>...<br>    if self.dynamic and grad_overflow(self.model_params) and self.loss_scale &gt; 1:<br>        self.loss_scale /= 2<br>...<br>    else:<br>...<br>        if self.noskip &gt;= self.max_noskip and self.loss_sca﻿le﻿ &lt; self.max_scale:<br>            self.loss_scale *= 2<br>...</pre><p>It exposes none of them to the outside world. If not everything goes as planned, we will lose track of self.loss_scale. Once we realize it, the price will become how we keep track of it. Yet another coding trade-off between time and effort.</p><p>Speaking of the coding effort, it seems a no-brainer to use cuDNN APIs. Unfortunately, using them blindly can cause a major setback if you want to have a sane trial. One can easily find many testimonies to cuDNN’s built-in dropout (especially when using it with bidirectional models) on StackOverflow and github. “Don’t use it,” says the jury.</p><h3>Effectiveness</h3><p>But why? Because multi-<em>something-like-core</em> implementations are again a burden of coding. Although cuDNN APIs provide deterministic counterparts of typical operations, there are too many switches to flip consistently. To my best knowledge, TensorFlow does not care about them. While PyTorch supports a welcome cudnn.deterministic flag, it has no effect on the cuDNN built-in dropout either.</p><p>A part of <a href="https://medium.com/@TianJianJiang/sane-trials-on-a-virtual-shoestring-1-3-c7a7bddae26">concerns about identity</a> lies in the definition of what effectiveness is. More specifically, what the expected outcome is. When only one core/process/thread is available, life is simple. Living in a world full of magic of parallel/distributed computing, we have to be careful with shared odds and ends. For example, we want consistent orders of shuffled instances from a multi-process dataset reader, but <a href="https://github.com/allenai/allennlp/pull/2729">how random numbers behave between processes can surprise us</a>.</p><h3>Usable v. Useful</h3><figure><img alt="Machine Learning" src="https://cdn-images-1.medium.com/max/371/1*UKIHA2AHtB9WPG-KrfwSZg.png" /><figcaption>Courtesy of <a href="https://xkcd.com/1838/">xkcd: Machine Learning</a></figcaption></figure><p>A classic paper of human-computer interaction argued, “<a href="https://www.sciencedirect.com/science/article/pii/S095354380000031X">What is beautiful is usable</a>.” Please note that it deliberately said “usable” instead of “useful.” Machine learning and deep learning algorithms often come with pretty diagrams and elegant formulae. Many of them are indeed usable. Yet usability requires effectiveness, efficiency, and satisfaction. Or at least for one’s sanity, the key question <a href="https://medium.com/@TianJianJiang/rename-and-repurpose-this-blog-what-can-imao-do-and-how-can-i-help-218f58db5685">IMAO</a> is: what makes one stir for a big pile of linear algebra the same to another?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0dfa8caa7ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Strategically Draft Academic Papers]]></title>
            <link>https://gnaijnaijnait.medium.com/strategically-draft-academic-papers-7de539924d86?source=rss-4a848b1dafcf------2</link>
            <guid isPermaLink="false">https://medium.com/p/7de539924d86</guid>
            <category><![CDATA[academic-writing]]></category>
            <dc:creator><![CDATA[IMAO]]></dc:creator>
            <pubDate>Thu, 26 Sep 2019 10:43:10 GMT</pubDate>
            <atom:updated>2019-10-08T04:10:52.448Z</atom:updated>
            <content:encoded><![CDATA[<h4>A rough translation of <a href="https://www.editing.tw/blog/%e5%a6%82%e4%bd%95%e7%ad%96%e7%95%a5%e6%80%a7%e5%9c%b0%e6%92%b0%e5%af%ab%e8%ab%96%e6%96%87%e8%8d%89%e7%a8%bf.html">如何策略性地撰寫論文草稿</a> by Dr. Wallace</h4><p><em>A week ago, I received a newsletter about paper drafting. It’s from one of the biggest paper editing agencies in Taiwan. Although the author, Dr. Steve Wallace, is an American, he wrote newsletters in Mandarin Chinese. So I took the liberty to translate his points roughly in English here.</em></p><p>One may find many “DOs” and “DON’Ts” online or via friends. Whether they will be effective are on a case-by-case basis. We think academic paper authors should develop various strategies and resources to find the best approach for themselves. Every opportunity and task deserves a different measure. Like each tool serves its own purpose, a toolbox of academic writing can’t rely on just one strategy.</p><h3>Plan</h3><p>Write an abstract or an outline as guidance for the drafts. This strategy works for most of the authors.</p><h3>Three-Draft Cycle</h3><h4>First Draft: for the Authors</h4><ol><li>See yourself and your coauthors as the earliest readers.</li><li>Be quick and dirty. No need to worry about writing styles yet.</li></ol><ul><li>Optional: sometimes be picky with the title and the introduction. It may help clarify angles and structures, depending on the genre (conference, journal, book, etc.). The longer the paper, the more important the structure (for consistency and fluency).</li></ul><p>3. Pause and reflect.</p><h4>Second Draft: for the Readers</h4><ol><li>Delete, delete, delete.</li><li>Supply and tune the introduction and the conclusions.</li><li>Pause and reflect.</li></ol><h4>Third Draft: for the Publication (Story-telling, <a href="https://medium.com/@TianJianJiang/rename-and-repurpose-this-blog-what-can-imao-do-and-how-can-i-help-218f58db5685">IMAO</a>)</h4><ol><li>Deliver an intriguing narrative that concerns the public, not just the reviewers.</li><li>Pause and reflect.</li><li>Revisit the 3-draft cycle when necessary.</li></ol><h3>Conclusion</h3><p>It is neither a secret tactic nor DOs/DON’Ts tricks. It is a method of switching among four perspectives: the planner, oneself, the target audiences, and the publication.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7de539924d86" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sane Trials of Machine Learning on a Virtual Shoestring (2/3)]]></title>
            <link>https://gnaijnaijnait.medium.com/sane-trials-of-machine-learning-on-a-virtual-shoestring-2-3-46456f579c2c?source=rss-4a848b1dafcf------2</link>
            <guid isPermaLink="false">https://medium.com/p/46456f579c2c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[fastai]]></category>
            <dc:creator><![CDATA[IMAO]]></dc:creator>
            <pubDate>Tue, 24 Sep 2019 04:48:58 GMT</pubDate>
            <atom:updated>2019-10-08T04:10:09.987Z</atom:updated>
            <content:encoded><![CDATA[<h4>Hold Every Random State Accountable</h4><h3>TL;DR and for what you may not see in this post.</h3><ul><li>Know (what are) self and others (<a href="https://medium.com/@TianJianJiang/sane-trials-on-a-virtual-shoestring-1-3-c7a7bddae26">part-1</a>);</li><li><strong>Control PRNG states</strong>, not just their seed numbers;</li><li>Uncover side effects (<a href="https://medium.com/@TianJianJiang/sane-trials-of-machine-learning-on-a-virtual-shoestring-3-3-b0dfa8caa7ca">part-3</a>), e.g.:</li><li>— <a href="https://docs.fast.ai/callbacks.fp16.html">Mixed Precision</a>, <strong>mind the gap of the dynamic loss scale </strong>[<a href="https://github.com/fastai/fastai/blob/master/fastai/callbacks/fp16.py#L101">1</a>][<a href="https://github.com/fastai/fastai/blob/master/fastai/callbacks/fp16.py#L114">2</a>];</li><li>— cuDNN, <strong>avoid using its built-in dropout;</strong></li><li>— Non-determinism/Instability of multi-core/process/thread usages.</li></ul><p><em>This series of posts shares my thoughts on how to get deterministic outcomes of machine learning experiments at an affordable cost. As usual, most of the content is only my murmur. Feel free to skip them and check </em><a href="https://github.com/tianjianjiang/playground/blob/master/IMDb_baseline.ipynb"><strong><em>the Jupyter notebook for this series</em></strong></a><em> directly. It tries </em><a href="https://aclweb.org/anthology/P18-1031"><em>Universal Language Model Fine-tuning for Text Classification</em></a><em> with a more recent version of fastai library. Another friendly reminder: technical jargons are just fancy names for references.</em></p><h3>To Seed or Not to Seed</h3><p>In <a href="https://course.fast.ai/videos/?lesson=9">Lesson 9 of fast.ai course</a>, Jeremy Howard mentioned (around 58&#39;15&quot;) why he deliberately left seeds unset for pseudo-random number generators (PRNGs). Here I transcribe, quote, and highlight in bold:</p><blockquote>If there’s variation going on when I run my model at different times, I want to see it. I don’t want it to be hidden behind a fixed seed. So there’s a big push in science for like reproducible science, which is great for many reasons, but it’s not how you should develop your models. When you’re developing your models, you want to have a kind of good intuitive sense of <strong>what bits are stable, what bits are unstable, and how much variation</strong> you expect. And so if you have a test which you know fails one every out of one every 100 times, it’s good to know that.</blockquote><p>I love his advice and share a similar goal of knowing what may be unstable. Just to be clear, my method involves keeping tracks on every PRNG state rather than fixing seeds. In this way, one may collect test results and realize that it “fails one out of every 100 times” by consciously and systematically applying different PRNG seeds and/or states. Plus, with tracked states, it will be easier to pinpoint “what bits are unstable” because it hides nothing and controls everything for each bit of training steps. <a href="https://medium.com/@TianJianJiang/rename-and-repurpose-this-blog-what-can-imao-do-and-how-can-i-help-218f58db5685">IMAO</a>, this is how one can make a <a href="https://en.wikipedia.org/wiki/Dry_lab">dry lab</a> more rigid than a <a href="https://en.wikipedia.org/wiki/Randomized_controlled_trial">randomized controlled trial</a>, by explicitly showing selection and allocation biases as if the experiments only use a fixed seed. For example, when studying the impact of random initialization, I prefer holding random states accountable for the combination of (PRNG seeds) × (distributions) × (<a href="https://en.wikipedia.org/wiki/Feature_scaling">feature scaling methods</a>) such that a particular method’s success won’t be just a fluke.</p><figure><a href="https://xkcd.com/1277/"><img alt="" src="https://cdn-images-1.medium.com/max/283/1*gxB-M2oRPRKPOHbq0Zelow.png" /></a><figcaption>Courtesy of <a href="https://xkcd.com/1277/">xkcd: Ayn Random</a></figcaption></figure><h3>Identity Crisis</h3><p>Before the rise of convolution neural networks (CNNs) and recurrent neural networks (RNNs), I have rarely seen questions about PRGN being asked online. If I hadn’t got bitten by it when I was implementing a genetic algorithm, I probably won’t care either. For many other machine learning algorithms, their numerical optimization methods are usually stable. Take training a part-of-speech tagger for an example, applying a linear-chain <a href="https://en.wikipedia.org/wiki/Conditional_random_field">conditional random fields</a> model with <a href="https://en.wikipedia.org/wiki/Limited-memory_BFGS">Limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) algorithm</a>, people won’t notice a difference of accuracy related to various PRNG states. CNNs and RNNs, for the time being, have to rely on first-order methods such as the stochastic gradient descent (SGD) algorithm family. They are sensitive to PRNG states. Therefore, you may see many tutorials and FAQs remark on fixing seed numbers for PRNGs. Otherwise, your mileage will definitely vary.</p><p>For instance, in a typical fastai training routine, we may have two macro steps:</p><pre># Step-1<br>fit_one_cycle(cycle_length_1, ...)</pre><pre># Step-2<br>unfreeze()<br>fit_one_cycle(cycle_length_2, ...)</pre><p>To test out different cycle lengths, just changing them for Step-1 and Step-2 may sound effective and efficient, except the code won’t remind you what you’ve been testing is not only the differences of cycle lengths but also the differences of RPNG states. Although given enough runs of the tests, the randomness by definition won’t be a problem, but it’s just that I’m on a shoestring, and I want everything to be accountable, even the random numbers. (I am traumatized by the bite from an inappropriate random number distribution for my old genetic algorithm experiments.)</p><p>If we take the viewpoint of <a href="https://en.wikipedia.org/wiki/Pure_function">pure function</a>, the steps shall look like:</p><pre># Step-1<br>fit_one_cycle(cycle_length_1, <em>random_state_1</em>, ...) → <em>random_state_2</em></pre><pre># Step-2<br>unfreeze(<em>random_state_2</em>) → <em>random_state_3</em><br>fit_one_cycle(cycle_length_2, <em>random_state_3</em>, ...) → <em>random_state_4</em></pre><p>Once we realize that random states are always a part of input and output, the rest is just about how a nice bookkeeper should be. Based on the aforementioned analog of pure function, it would be cool to use <a href="http://adit.io/posts/2013-06-10-three-useful-monads.html#the-state-monad">State Monad</a>. In Python, however, it may be too much hustle. For <a href="https://en.wikipedia.org/wiki/Functional_programming#Simulating_state">simulating state</a>, one particular alternative to monads is <a href="https://en.wikipedia.org/wiki/Uniqueness_type">uniqueness type</a>. As the term suggests, we will again face the <a href="https://medium.com/@TianJianJiang/sane-trials-on-a-virtual-shoestring-1-3-c7a7bddae26">identity crisis</a>. To keep the implementation simple and stupid (read: no type systems, no callbacks, no decorators, no pipelines, etc.), I might just as well name those states.</p><h3>The Rose of Old Remains Only in Its Name</h3><blockquote><a href="https://en.wikipedia.org/wiki/Indirection">“All problems in computer science can be solved by another level of indirection.”</a></blockquote><p>Imagine that a machine learning framework also uses <a href="https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller">Model-View-Controller (MVC)</a>. Or we could delegate the bookkeeping via <a href="https://docs.python.org/3/library/functools.html#functools.wraps">decorators</a>. I didn’t wrap them in my toy implementation, yet the pseudo-code explains the idea:</p><pre># Step-1<br>fit_one_named_cycle(<em>﻿﻿name_1</em>﻿﻿, cycle_length_1) → <em>name_2</em></pre><pre># Step-2<br>named_unfr﻿eeze(<em>name_2</em>) → <em>name_3</em><br>fit_one_named_cycle(﻿﻿<em>﻿name_3</em>﻿﻿﻿, cycle_length_2) → <em>name_4</em></pre><p>To my best knowledge, only <a href="https://github.com/IDSIA/sacred">Sacred</a> provides <a href="https://sacred.readthedocs.io/en/stable/randomness.html#resilience-to-change">a similar ability to control PRNG states</a>. It mainly supports TensorFlow, but TensorFlow has a long-standing issue on non-deterministic cuDNN APIs (TK in the part-3). For PyTorch, one may still have to apply Sacred’s wrappers and special arguments explicitly. No different from what I describe here. Those situations hint at a bigger problem: “<a href="https://en.wikipedia.org/wiki/The_Name_of_the_Rose">We hold only empty names.</a>” Unfortunately, naming every PRNG state won’t suffice. It’s like with MVC: if the view is the output and the controller processes PRNG states (and the input), how about the model? I will leave it to the part-3.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46456f579c2c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sane Trials of Machine Learning on a Virtual Shoestring (1/3)]]></title>
            <link>https://gnaijnaijnait.medium.com/sane-trials-on-a-virtual-shoestring-1-3-c7a7bddae26?source=rss-4a848b1dafcf------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7a7bddae26</guid>
            <category><![CDATA[fastai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[colab]]></category>
            <dc:creator><![CDATA[IMAO]]></dc:creator>
            <pubDate>Mon, 02 Sep 2019 08:56:17 GMT</pubDate>
            <atom:updated>2019-10-08T04:09:51.315Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Bigger, Longer, &amp; Uncut Preamble</h4><h4>TL; DR and for what you may not see in this post.</h4><ol><li>Know (what are) self and others;</li><li><strong>Control PRNG states</strong>, not just their seed numbers (<a href="https://medium.com/@TianJianJiang/sane-trials-of-machine-learning-on-a-virtual-shoestring-2-3-46456f579c2c">part-2</a>);</li><li>Uncover side effects (<a href="https://medium.com/@TianJianJiang/sane-trials-of-machine-learning-on-a-virtual-shoestring-3-3-b0dfa8caa7ca">part-3</a>), e.g.:</li><li>— <a href="https://docs.fast.ai/callbacks.fp16.html">Mixed Precision</a>, <strong>mind the gap of the dynamic loss scale</strong> [<a href="https://github.com/fastai/fastai/blob/master/fastai/callbacks/fp16.py#L101">1</a>][<a href="https://github.com/fastai/fastai/blob/master/fastai/callbacks/fp16.py#L114">2</a>];</li></ol><ul><li>— cuDNN, <strong>avoid using its built-in dropout;</strong></li><li>— Non-determinism/Instability of multi-core/process/thread usages.</li></ul><p><em>This series of posts shares my thoughts on how to get deterministic outcomes of machine learning experiments at an affordable cost. As usual, most of the content is only my murmur. Feel free to skip them and check </em><a href="https://github.com/tianjianjiang/playground/blob/master/IMDb_baseline.ipynb"><strong><em>the Jupyter notebook for this series</em></strong></a><em> directly. It tries </em><a href="https://aclweb.org/anthology/P18-1031"><em>Universal Language Model Fine-tuning for Text Classification</em></a><em> with a more recent version of fastai library. Another friendly reminder: technical jargons are just fancy names for references.</em></p><figure><a href="https://xkcd.com/1657/"><img alt="" src="https://cdn-images-1.medium.com/max/249/1*NflAe-NItCCccuYjM1hzfA.png" /></a><figcaption>Courtesy of <a href="https://xkcd.com/1657/">xkcd: Insanity</a></figcaption></figure><h3>Healthy Expectations</h3><p>One of the most overused mis-attributed quotes says something about insanity and repetition. When using it formally, “insanity” is a legal term, not a psychological one (anymore). For describing hopeless repetitions, perseveration and perseverance are more accurate, at least for me. Because I often have to persevere when running experiments, and then learn only helplessness, especially for deep learning. Anyone who has tried to get the same accuracy from some state-of-the-art system may share the sentiment. <a href="https://medium.com/@TianJianJiang/rename-and-repurpose-this-blog-what-can-imao-do-and-how-can-i-help-218f58db5685">IMAO</a>, the real problem isn’t about the state of mind (or the art), but the definition of sameness. Am I doing <strong>the same things</strong> to the referenced paper/system? The situation reminds me of an interesting theory: <a href="https://en.wikipedia.org/wiki/Identity_of_indiscernibles">Identity of indiscernibles</a>. Here’s a quote I like to emphasize:</p><blockquote>“=” in Leibniz’s Law means “quantitative sameness”, not merely qualitative sameness. “Identical” is not merely of equal value, or equivalent, or isomorphic, but rather is “x is the same object as y”.</blockquote><p>Based on <a href="https://github.com/fastai/fastai">fastai</a>’s tutorial “<a href="https://docs.fast.ai/tutorial.resources.html#lean-and-mean-notebook-coding">Deep Learning on a Shoestring <strong>§</strong>Lean and mean notebook coding</a>”, a qualitative sameness of (isomorphic) workflow looks like this to me:</p><ol><li>Import identical libraries</li><li>Initialize identical objects</li><li>Train a model with identical configurations and datasets</li><li>Predict with identical datasets</li></ol><p>Each step has its own properties. Layer by layer, the whole experiment becomes an onion that cries for quantitative sameness. What and how many factors make library/object/configuration/dataset identical? To divide and conquer them without tears, my questions transform to asking when and where to stop asking questions. Otherwise, it will become an <a href="https://en.wikipedia.org/wiki/Infinite_regress">infinite regress</a> rather than a practical process. To frame the scope, I must address preconditions first.</p><h3>Know Self and Others</h3><p>The scope of variable visibility and life cycle can be fuzzy. The fuzziness makes identification harder. In Python, objects use “<em>self</em>” to refer… themselves. To identify if an object is the same as expected, one may ask what belong to one-self and what are for others. Except it won’t be just about Python objects, but in a broader sense of what makes a self-contained…, say, training step.</p><p>As an independent (read: poor) researcher, I use Google Colaboratory (Colab) because its GPU/TPU is free. That is my virtual shoestring. But there’s a catch: a 12-hour volatile session. One worst-case scenario is that a time-consuming training process just got interrupted, and I want to resume it as much as it can be. This may not be the case for someone who has access to a more convenient setup, but a similar situation can still happen because of a common trait of experiments: trial and error. As long as there is a movable part of the training process, learning how to control it is inevitable. For example, the following two versions of model training pseudo-steps will yield different outcomes.</p><pre><strong>Non-stopped</strong>          <strong>Reloaded</strong><br>────────────────────────────────────────────────<br> 1 initialize         1 initialize               <br> 2 fit                2 fit                      <br>                      3 <em>save; initialize; load</em><br> 3 change something   4 change something         <br> 4 fit                5 fit</pre><p>Why do I want to insert additional save-initialize-load steps into the non-stopped loop? Because a GPU’s memory space is precious, and as the aforementioned <a href="https://docs.fast.ai/tutorial.resources.html#conclusion">fastai tutorial</a> concludes, this is how a good citizen should do to reallocate resources. Plus, if my virtual shoestring was broken, the seam of the reloading steps will be where I reattach.</p><p>Are the above two versions the same? Quantitatively, no. But qualitatively they can be if the save-initialize-load steps have every factor of the training loop restored. The more training factors we control, the fewer identity issues we care. Yes, that’s a tautology. To help myself break the chain, I categorize those (in-)famous factors in three types:</p><ol><li>Environment: hardware, operating system, libraries, anything that virtualization can contain;</li><li>Pseudo-random number generator (PRNG) states;</li><li>Nondeterministic and unstable algorithms. The former is usually about multi-core/process/threading procedures. The latter classifies a routine being highly sensitive to implicit state-transitions. They often come together.</li></ol><p>Thanks to version-control and virtualization, the first type is easy to tame. I will just skip it. The second one requires some tedious work, yet it deserves attention because I realize many machine learning libraries by default only support seed number configuration. The last class is tricky. Many of them are well-known limitations, but there will be always an element of surprise when playing with recently advanced technologies, such as Mixed Precision with a dynamic loss scale in my case.</p><h3>Rational Preferences</h3><p>Determinism and stability of algorithms may just imply another perspective: how about the sameness issues of the outcome? To be specific, I care about two aspects of the outcome as measuring a model’s performance with certain evaluation metrics.</p><ol><li>Up-to what precision of the decimal places can two numbers be the same?</li><li>If different, is the difference significant?</li></ol><p>I have a rather controversial view for (1): as exact as they can be (like strings). Don’t confuse it with the fuzzy comparisons of floating-point arithmetics. In the context of this post, fuzzy comparisons are for different inputs.</p><p>For (2), that is a question I often dodge. Issues of statistical significance are far from resolution. While I enjoy reading/watching “<a href="https://vimeo.com/285803636">The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing</a>” from ACL 2018, rational preference is more significant than any significant test to me.</p><p>To me, being rational is acting just like what defines a <a href="https://en.wikipedia.org/wiki/Rational_choice_theory">rational choice</a>. If a configuration C makes the system better than another configuration D at a dataset S, then C better be also superior to D at another dataset T, or the difference between C and D may just be opinionated.</p><h3>Too Much Information, Too Little Time.</h3><p>Talk is cheap, and I talk way too much. In my perfect imaginary realm, there is always a principle that generalizes every special case completely. Back in the muddy real world, I try to compartmentalize exceptions. Such that I may at least have a direction to seek the right queries. On running machine learning trials, I narrow the scope down to two issues: PRNG states and nondeterministic/unstable algorithms. The former requires more attention at each state than just their seed numbers, and for the latter question, its rabbit hole is bottomless. I had to read as much source code as needed to realize what variables I should retain every step along the way. The top list of this post shows the answers I’ve found. I hope they will help save you some time for real research quests. The remaining two posts to come will elaborate some details with my murmurs.</p><figure><a href="https://xkcd.com/979/"><img alt="" src="https://cdn-images-1.medium.com/max/485/1*J3Ygm52dcu75LQkQ6McuEw.png" /></a><figcaption>Courtesy of<a href="https://xkcd.com/979/"> xkcd: Wisdom of the Ancients</a></figcaption></figure><h3>Side Note: Why Not “Reproducible” But “Sane”</h3><p>There is an unnecessary confusion among several R-words: reproducible, replicable, repeatable, reliable, etc. (Sneak peek:<a href="https://languagelog.ldc.upenn.edu/nll/?p=21956"> Replicability vs. reproducibility — or is it the other way around?</a>) <a href="https://medium.com/@TianJianJiang/rename-and-repurpose-this-blog-what-can-imao-do-and-how-can-i-help-218f58db5685">IMAO</a>, resisting the temptation of wordplay is futile. For those R-words, I have a hard time picking a side and remembering what I’ve chosen. Their ambiguities are just another example of identity “crisis” with many semantic factors, so I put those factors on a list and take the list for a sanity check.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7a7bddae26" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What can IMAO do & how can I help?]]></title>
            <link>https://gnaijnaijnait.medium.com/rename-and-repurpose-this-blog-what-can-imao-do-and-how-can-i-help-218f58db5685?source=rss-4a848b1dafcf------2</link>
            <guid isPermaLink="false">https://medium.com/p/218f58db5685</guid>
            <dc:creator><![CDATA[IMAO]]></dc:creator>
            <pubDate>Mon, 02 Sep 2019 08:41:12 GMT</pubDate>
            <atom:updated>2019-10-05T02:34:15.868Z</atom:updated>
            <content:encoded><![CDATA[<p>Rename and repurpose this blog</p><p>This blog was merely a personal notepad. Now I like to conduct some experiments with it for the next couple of months, starting from redefining it.</p><p>According to <a href="https://www.dictionary.com/e/acronyms/imao/">https://www.dictionary.com/e/acronyms/imao/</a>, “in my arrogant opinion is usually used with a sense of irony or self-deprecation.” IMAO and a far-fetched way, it can also stand for ī-máo (一毛) in Mandarin Chinese, meaning one single hair and figuratively something insignificant, such as a cent. In that sense, i-mao is just half of “my two cents.”</p><p>As insignificant as i-mao can be, one of my favorite stories has made quite a fuss about it. <a href="https://en.wikipedia.org/wiki/Yang_Zhu">Yang Zhu</a>, a Taoist who lived around 400–350 BC, took an individual’s i-mao vs. the world for an argument of <a href="https://en.wikipedia.org/wiki/Ethical_egoism">ethical egoism</a>. His competitor, Mencius, misinterpreted it and attacked him as “who would not pluck a hair from his body to benefit the world.” Unfortunately (to me at least), although Yang Zhu was respected back then, Confucianism soon became one of the dominant doctrines in the following dynasties. Their strong influences probably distorted the argument into a pejorative about selfishness.</p><p>The world nowadays seems full of collective selfishness, yet I have no intention to call for altruism or any “-ism” whatsoever. My reaction usually goes like “yes, but IMAO…” for almost every topic. While my words often sound condescending (hence the old name of this blog), I am not proud of it. When I call something an opinion, it implies a working theory with insufficient evidence. I used to keep them in private conversations. Perhaps this behavioral pattern itself requires some empirical study now. What if putting in my i-mao worth thoughts can help?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*8dlhZAt3cP-OKctSHUvQCg.png" /><figcaption>Courtesy of <a href="https://what-if.xkcd.com/22/">xkcd what-if: Cost of Pennies</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=218f58db5685" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>