<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ricky Costa on Medium]]></title>
        <description><![CDATA[Stories by Ricky Costa on Medium]]></description>
        <link>https://medium.com/@nlpcypher?source=rss-7a94371c6509------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wbM64vyXFBWoDFoRO4uRcw.jpeg</url>
            <title>Stories by Ricky Costa on Medium</title>
            <link>https://medium.com/@nlpcypher?source=rss-7a94371c6509------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 22:57:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nlpcypher/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The NLP Cypher | 05.01.22]]></title>
            <link>https://nlpcypher.medium.com/the-nlp-cypher-05-01-22-6a0c0016966e?source=rss-7a94371c6509------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a0c0016966e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ricky Costa]]></dc:creator>
            <pubDate>Sun, 01 May 2022 22:29:50 GMT</pubDate>
            <atom:updated>2022-05-01T22:29:50.378Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/680/0*Y2M3B5Zf9iTASEa0" /><figcaption>DALLE</figcaption></figure><h4>NATURAL LANGUAGE PROCESSING (NLP) WEEKLY NEWSLETTER</h4><h4>&gt;&gt;&gt; curl -L <a href="http://git/">http://git</a>\.io/unix</h4><p>Hey Welcome back! Want to start off by giving a few shout outs!!!</p><ol><li>Hectiq.AI hosted the big-yaml — Neural Magic’s config to serve 19 BERT models simultaneously, all under 16GBs of RAM! 😍 <a href="https://lab.hectiq.ai/public/hectiq-ai/demo/deepsparse-demo"><strong>Demo</strong></a></li><li>If you want to know more about the demo above, you can read about it <a href="https://www.kdnuggets.com/2022/04/fast-bert-go-sparsity.html"><strong>here</strong></a><strong>:</strong> Thank you KDnuggets and Towards AI! 🚀</li><li>Scrape tweets with Twint and classify it with a Neural Magic Sparse Transformer: <a href="https://github.com/neuralmagic/deepsparse/tree/main/examples/twitter-nlp">Code</a> 🧙‍♂️</li></ol><h3>Elon Musk on Twitter: &quot;Next I&#39;m buying Coca-Cola to put the cocaine back in / Twitter&quot;</h3><p>Next I&#39;m buying Coca-Cola to put the cocaine back in</p><h3>ICLR Happened:</h3><p><strong>Stanford</strong></p><p><a href="https://ai.stanford.edu/blog/iclr-2022/">Stanford AI Lab Papers and Talks at ICLR 2022</a></p><p><strong>Google</strong></p><p><a href="https://ai.googleblog.com/2022/04/google-at-iclr-2022.html">Google at ICLR 2022</a></p><p><strong>Meta</strong></p><p><a href="https://ai.facebook.com/events/iclr-2022/">ICLR 2022</a></p><p><strong>Amazon</strong></p><p><a href="https://www.amazon.science/conferences-and-events/iclr-2022">ICLR 2022</a></p><p><strong>Apple</strong></p><p><a href="https://machinelearning.apple.com/updates/apple-at-iclr-2022">ICLR 2022</a></p><p><strong>DeepMind</strong></p><p><a href="https://www.deepmind.com/blog/deepminds-latest-research-at-iclr-2022">DeepMind&#39;s latest research at ICLR 2022</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/0*FqpjGGOf9c4QUZAd" /></figure><h3>FormNet: A New Model for Document Understanding</h3><p>Gets SOTA performance on the <a href="https://github.com/clovaai/cord">CORD</a>, <a href="https://guillaumejaume.github.io/FUNSD/">FUNSD</a>, and <a href="https://aclanthology.org/2020.acl-main.580/">Payment</a> benchmarks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/0*-XpNlAdyb0fnVPh3.png" /><figcaption>CORD benchmark</figcaption></figure><p><a href="https://ai.googleblog.com/2022/04/formnet-beyond-sequential-modeling-for.html">FormNet: Beyond Sequential Modeling for Form-Based Document Understanding</a></p><h3>WaNLI: Generate Your Own NLI dataset</h3><h3>Alisa Liu on Twitter: &quot;We introduce a new paradigm for dataset creation based on human 🧑‍💻 and machine 🤖 collaboration, which brings together the generative strength of LMs and the evaluative strength of humans. And we collect 🎉 WaNLI, a dataset of 108K NLI examples! 🧵Paper: https://t.co/IUXcm9wIh2 pic.twitter.com/xPOVGeS82F / Twitter&quot;</h3><p>We introduce a new paradigm for dataset creation based on human 🧑‍💻 and machine 🤖 collaboration, which brings together the generative strength of LMs and the evaluative strength of humans. And we collect 🎉 WaNLI, a dataset of 108K NLI examples! 🧵Paper: https://t.co/IUXcm9wIh2 pic.twitter.com/xPOVGeS82F</p><h3>Run Python in the Browser via HTML</h3><p>The makers of Anaconda came out with this. 🍾❤️</p><p>Has only been tested on Chrome thus far.</p><ul><li><a href="https://www.anaconda.com/blog/pyscript-python-in-the-browser">Anaconda | New from Anaconda: Python in the Browser</a></li><li><a href="https://pyscript.net/">Pyscript.net</a></li></ul><p><strong>Code</strong>:</p><p><a href="https://github.com/pyscript/pyscript">GitHub - pyscript/pyscript: Home Page: https://pyscript.net Examples: https://pyscript.net/examples</a></p><h3>DALL-E-2 | Performance and Limitations</h3><h3>hardmaru on Twitter: &quot;&quot;Triangle with four corners&quot; #dalle pic.twitter.com/L1yE9OviiE / Twitter&quot;</h3><p>&quot;Triangle with four corners&quot; #dalle pic.twitter.com/L1yE9OviiE</p><h4>Limitations of DALL-E-2 | a thread 🧵</h4><h3>Benjamin Hilton on Twitter: &quot;1) It can take a lot of adjusting prompts to get something good.e.g. This was actually generated with &quot;A renaissance-style painting of a modern supermarket aisle. In the aisle is a crowd of shoppers with shopping trolleys trying to get reduced items.&quot;https://t.co/7PYEoCn0GT / Twitter&quot;</h3><p>1) It can take a lot of adjusting prompts to get something good.e.g. This was actually generated with &quot;A renaissance-style painting of a modern supermarket aisle. In the aisle is a crowd of shoppers with shopping trolleys trying to get reduced items.&quot;https://t.co/7PYEoCn0GT</p><h4>DALL-E-2 PyTorch Implementation</h4><p>Lucidrains for the win!</p><p><a href="https://github.com/lucidrains/DALLE2-pytorch">GitHub - lucidrains/DALLE2-pytorch: Implementation of DALL-E 2, OpenAI&#39;s updated text-to-image synthesis neural network, in Pytorch</a></p><h3>SEAL 🦭 Search Engines w/ Autoregressive LMs</h3><p><strong>Thread 🧵</strong></p><p><a href="https://threadreaderapp.com/thread/1518538210598608896.html">Thread by @MicheleBevila20 on Thread Reader App</a></p><p><strong>Code</strong></p><p><a href="https://github.com/facebookresearch/SEAL">GitHub - facebookresearch/SEAL: Search Engines with Autoregressive Language models</a></p><h3>GPT-NeoX Annotated</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mmyznsBYPNtYxhnR.png" /></figure><p><a href="https://github.com/labmlai/neox">GitHub - labmlai/neox: Simple Annotated implementation of GPT-NeoX in PyTorch</a></p><p><a href="https://lit.labml.ai/github/labmlai/neox/tree/main/src/neox/__init__.py">Play</a></p><h3>DiffCSE — Meta’s New Sentence Embeddings Library</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s-wNQMdtDuPrZPr-.png" /></figure><p><a href="https://github.com/voidism/DiffCSE">GitHub - voidism/DiffCSE: Code for the NAACL 2022 long paper &quot;DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings&quot;</a></p><p><a href="https://arxiv.org/pdf/2204.10298.pdf"><strong>Paper</strong></a></p><h3>OpenAIs New Clip Model</h3><p>(a silent drop)</p><p><a href="https://github.com/openai/CLIP/commit/b4ae44927b78d0093b556e3ce43cbdcff422017a">ViT-L/14@336px (#234) · openai/CLIP@b4ae449</a></p><h3>Get Stoic</h3><p><a href="https://stoic-quotes.com/">Stoic Quotes | The best quotes from the great Roman Stoics</a></p><h3>Papers to Read 📚</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2204.13074.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/b542372d7fc64f3c5f084f6bb20b2c3a/href">https://medium.com/media/b542372d7fc64f3c5f084f6bb20b2c3a/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2204.07705.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/d983aa2989cedcb018552a824dc67544/href">https://medium.com/media/d983aa2989cedcb018552a824dc67544/href</a></iframe><h3>NLP Index👨‍💻</h3><h4>A collection of recently released repos that caught our 👁</h4><h4><a href="https://arxiv.org/pdf/2204.11087.pdf">LitMind Dictionary</a></h4><blockquote>An open-source online generative dictionary that takes a word and context containing the word as input and automatically generates a definition as output.</blockquote><p><a href="https://github.com/blcuicall/litmind-dictionary">GitHub - blcuicall/litmind-dictionary</a></p><p><a href="https://www.connectedpapers.com/main/d330768b62d4e82cf02a1455886b32e6abed1e7a"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2204.12811.pdf">SkillSpan Repository</a></h4><blockquote>A novel skill extraction dataset consisting of 14.5K sentences and over 12.5K annotated spans from job postings.</blockquote><p><a href="https://github.com/kris927b/skillspan">GitHub - kris927b/SkillSpan: SKILLSPAN: Competences as Spans for Skill Extraction from Job Postings</a></p><p><strong>Models</strong></p><p><a href="https://huggingface.co/jjzha">jjzha (Mike Zhang)</a></p><p><a href="https://www.connectedpapers.com/main/0b2a24ab36d9b30c2dc2472421ffa02a5a25b2a4"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2204.12130.pdf">LM-Debugger</a></h4><blockquote>An interactive debugger tool for transformer-based LMs, which provides a fine-grained interpretation of the model’s internal prediction process.</blockquote><p><a href="https://github.com/mega002/lm-debugger">GitHub - mega002/lm-debugger: The official code of LM-Debugger, an interactive tool for inspection and intervention in transformer-based language models.</a></p><p><a href="https://www.connectedpapers.com/main/8376e2ad48d6926d1dc51dafa66df7bcf9cd1ba5"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2204.10591.pdf">SalesBot: Transitioning from Chit-Chat to Task-Oriented Dialogues</a></h4><blockquote>The first large-scale dataset of dialogues transitioning from chit-chat to task-oriented scenarios.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JkGNXKJiN4xFjzm5.png" /></figure><p><a href="https://github.com/miulab/salesbot">GitHub - MiuLab/SalesBot: Transitioning from Open-Domain Chit-Chat to Task-Oriented Dialogues</a></p><p><a href="https://www.connectedpapers.com/main/2c763b82c09000333b53663fbc5993afdd952413"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2204.12061.pdf">PLOD: An Abbreviation Detection Dataset</a></h4><blockquote>An abbreviation detection dataset for scientific documents.</blockquote><p><a href="https://github.com/surrey-nlp/PLOD-AbbreviationDetection">GitHub - surrey-nlp/PLOD-AbbreviationDetection: This repository contains the PLOD Dataset for Abbreviation Detection released with our LREC 2022 publication</a></p><p><a href="https://www.connectedpapers.com/main/85a37455eeddd2cd2f2c77f845720b2871486d3f"><strong>Connected Papers</strong></a><strong> 📈</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/54/0*c0C7jkCYrRYDl4FL" /><figcaption><a href="https://www.linkedin.com/in/ricky-costa-nlp/">Ricky ҈̿҈̿҈̿҈̿҈̿҈̿Costa̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a0c0016966e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sparse Transformers | A Demo]]></title>
            <link>https://pub.towardsai.net/sparse-transformers-a-demo-58b6b1ebf3e4?source=rss-7a94371c6509------2</link>
            <guid isPermaLink="false">https://medium.com/p/58b6b1ebf3e4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ricky Costa]]></dc:creator>
            <pubDate>Mon, 18 Apr 2022 16:03:00 GMT</pubDate>
            <atom:updated>2022-04-18T16:03:00.751Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KXP54ZHPzJfenZQS" /><figcaption>Photo by <a href="https://unsplash.com/@_rishabhpandoh_?utm_source=medium&amp;utm_medium=referral">Rishabh Pandoh</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>How fast can BERT go with sparsity?</h4><h3>Here’s a Little Secret:</h3><p>If you want to analyze how fast 19 sparse BERT models perform inference, you’ll only need a YAML file and 16GB of RAM to find out. And spoiler alert:</p><p>… they run on CPUs.</p><p>… and they’re super fast!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dGNHvk3RZT_zRqMNIVTsmA.png" /></figure><p>The latest feature from Neural Magic’s <a href="https://github.com/neuralmagic/deepsparse"><strong>DeepSparse</strong></a><strong> </strong>repo is the DeepSparse Server! And the objective of this article is to show not only how seamless it is to serve up to 19 sparse BERT models, but how much the impact of sparsity has on model performance. For a bit of background, sparsification is the process of taking a trained deep learning model and removing redundant information from the over-parameterized network resulting in a faster and smaller model. And for this demo, we’ll be using various BERT models and loading them for inference to show the trade-off between accuracy and speed relative to the model’s sparsification.</p><p>The DeepSparse Server is built on top of our DeepSparse Engine and the popular FastAPI web framework allowing anyone to deploy sparse models in production with GPU-class speed but on CPUs! With the DeepSparse Engine, we can integrate into popular deep learning libraries (e.g., Hugging Face, Ultralytics) allowing you to deploy sparse models with ONNX.</p><p>As previously mentioned, all of the configuration required to run your models in production only requires a YAML file and a small bit of memory (thanks to sparsity). To get quickly started with serving four BERT models trained on the question answering task, this is what the config YAML file would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sScp2Lf79lTGgAylu9tpIA.png" /><figcaption>config.yaml</figcaption></figure><p>If you want to go big and load all of the 19 Neural Magic sparse BERT models: this is what the config file would look like 👀:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OqMbiV7W3hFUGAOzbQXpgA.png" /></figure><p>For ease of use, we’ve built a demo on top of Streamlit for anyone to demo the server and models for the question answering task in NLP. In order to test 19 models simultaneously, the app was tested on a virtual machine on the Google Cloud Platform.</p><p>To give some grounding on what I used for computing in my tests, here are the deets:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%252812%252C0%252C255%252C1%2529%26t%3Dseti%26wt%3Dnone%26l%3Dyaml%26width%3D514.5%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3DSpecs%25253A%25250A%252520%252520-%252520Cloud%252520Vendor%25253A%252520Google%252520Cloud%252520Platform%25250A%252520%252520-%252520Instance%25253A%252520%252520%252520%252520%252520c2-standard-4%25250A%252520%252520-%252520CPU%252520Type%25253A%252520%252520%252520%252520%252520Intel%252520Cascade%252520Lake%25250A%252520%252520-%252520Num%252520of%252520vCPUs%25253A%252520four%25250A%252520%252520-%252520RAM%25253A%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252016GB&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%25252812%25252C0%25252C255%25252C1%252529%26t%3Dseti%26wt%3Dnone%26l%3Dyaml%26width%3D514.5%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3DSpecs%2525253A%2525250A%25252520%25252520-%25252520Cloud%25252520Vendor%2525253A%25252520Google%25252520Cloud%25252520Platform%2525250A%25252520%25252520-%25252520Instance%2525253A%25252520%25252520%25252520%25252520%25252520c2-standard-4%2525250A%25252520%25252520-%25252520CPU%25252520Type%2525253A%25252520%25252520%25252520%25252520%25252520Intel%25252520Cascade%25252520Lake%2525250A%25252520%25252520-%25252520Num%25252520of%25252520vCPUs%2525253A%25252520four%2525250A%25252520%25252520-%25252520RAM%2525253A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252016GB&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/54f58db3a3c20440c161a11b3e7d376c/href">https://medium.com/media/54f58db3a3c20440c161a11b3e7d376c/href</a></iframe><p><em>Keep in mind that bare-metal machines will actually perform faster under the same computing constraints described in this article. However, since the models are already super fast, I feel comfortable showing their speed via virtualization.</em></p><p>We not only strongly encourage you to run the same tests on a VM for benchmarking performance but also so you’ll have the RAM required to load all 19 BERTs into memory, otherwise you’ll get this 👇:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FHhTXt43pk1I1W%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FHhTXt43pk1I1W%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FHhTXt43pk1I1W%2F200.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="243" frameborder="0" scrolling="no"><a href="https://medium.com/media/72c401485b38c15d68f696d40799e375/href">https://medium.com/media/72c401485b38c15d68f696d40799e375/href</a></iframe><p>If you prefer to get started quickly on a local machine without worrying about out-of-memory problems, you should try only loading a few models into memory. And the code below will show you how to do exactly this with 4 models (even though most sparse models are super light and you can possibly add more at your discretion).</p><h3>Getting Started with the SparseServer.UI</h3><p>We split our app into separate server and client directories. The server directory holds the YAML files for loading the models and the client has the logic for the Streamlit app:</p><pre>~sparseserver-ui/<br>    |__client/<br>       |__app.py<br>       |__pipelineclient.py<br>       |__samples.py<br>       |__settings.py     <br>    |__server/<br>       |__big-config.yaml<br>       |__config.yaml<br>    |__requirements.txt<br>    |__README.md</pre><h4>1. Clone the DeepSparse repo:</h4><pre>&gt;&gt;&gt; git clone <a href="https://github.com/neuralmagic/deepsparse.git">https://github.com/neuralmagic/deepsparse.git</a></pre><h4>2. Install the DeepSparse Server and Streamlit:</h4><pre>&gt;&gt;&gt; cd deepsparse/examples/sparseserver-ui</pre><pre>&gt;&gt;&gt; pip install -r requirements.txt</pre><p>Before we run the server, you can configure the host and port parameters in our startup CLI command. If you choose to use the default settings, it will run the server on localhost and port 5543 . For more info on the CLI arguments run:</p><pre>&gt;&gt;&gt; deepsparse.server --help</pre><h4>3. Run the DeepSparse Server:</h4><p>Ok! It’s time to serve all of the models defined in the config.yaml. This YAML file will download the four models from Neural Magic’s <a href="https://sparsezoo.neuralmagic.com/?domain=nlp&amp;sub_domain=question_answering&amp;page=1">SparseZoo</a> 🦾.</p><pre>&gt;&gt;&gt; deepsparse.server --config_file server/config.yaml</pre><p>After downloading the models and your server is up and running, open a second terminal to test out the client.</p><p>⚠️ <em>If you altered the </em><em>host and </em><em>port configuration when you first ran the server, please adjust these variables in the </em><em>pipelineclient.py module as well.</em></p><h4>4. Run the Streamlit Client:</h4><pre>&gt;&gt;&gt; streamlit run client/app.py --browser.serverAddress=&quot;localhost&quot;</pre><p>That’s it! Click on the URL in your terminal, and you are ready to start interacting with the demo. You can choose examples from a list, or you can add your own context and question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aw8S1QOB9T2-YB7BjQEm8w.png" /></figure><p>In the future, we’ll be expanding the number of NLP tasks outside of just question answering so you get a wider scope in performance with sparsity.</p><p>For the full code: check out the <a href="https://github.com/neuralmagic/deepsparse/tree/main/examples/sparseserver-ui">SparseServer.UI</a> …</p><p>…and don’t forget to give the DeepSparse repo a GitHub⭐!</p><ul><li>Ricky Costa| <a href="https://neuralmagic.com/">Neural Magic</a> 🧙</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58b6b1ebf3e4" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/sparse-transformers-a-demo-58b6b1ebf3e4">Sparse Transformers | A Demo</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The NLP Cypher | 04.17.22]]></title>
            <link>https://nlpcypher.medium.com/the-nlp-cypher-04-17-22-4ed2ba906bdb?source=rss-7a94371c6509------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ed2ba906bdb</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Ricky Costa]]></dc:creator>
            <pubDate>Mon, 18 Apr 2022 00:02:41 GMT</pubDate>
            <atom:updated>2022-04-18T00:13:52.940Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*38H3q6D47jHbsvHp.jpg" /><figcaption>damn, i deleted the page before tagging artist (my bad). Still dope though.</figcaption></figure><h4>NATURAL LANGUAGE PROCESSING (NLP) WEEKLY NEWSLETTER</h4><h4>Dark Mode</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*vt5w7Gv1dVEV-13FiMXNcw.png" /></figure><p>An updated chart w/r/t the last newsletter. This is the <a href="https://github.com/neuralmagic/deepsparse">DeepSparse</a> Engine and how it performs against ONNX runtime (deets above). With each new DeepSparse version release; inference performance just keeps on improving while you drink your Mai Tai. 🍾</p><h3>Sad Tales from the Dark Web</h3><p>Dread our favorite dark site continues to deal with active DDoS attacks.</p><p>A PGP message…</p><p><a href="https://pastebin.com/U1UpYhHy">Dread message 4/11 - Pastebin.com</a></p><p>OH … ever had anyone post a file you didn’t trust? Dangerzone comes in handy, but as usual, I wouldn’t know anything about this. 😉</p><p><a href="https://dangerzone.rocks/">Dangerzone</a></p><h3>Business is hard bruh</h3><p>Elon got bored and decided to buy Twitter this past week. He claims to want to make Twitter’s algo open-source. The first flex:</p><h3>Elon Musk on Twitter: &quot;Do you want an edit button? / Twitter&quot;</h3><p>Do you want an edit button?</p><p>Elon applied further pressure, by initially joining and then deciding not to join Twitter’s board, then offering 43Billi to get the whole pie … Twitter CEO explains 🤣👇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/0*aWmMEw4UBV5voED0" /></figure><h3>Elon Musk on Twitter: &quot;Wow, with Jack departing, the Twitter board collectively owns almost no shares! Objectively, their economic interests are simply not aligned with shareholders. / Twitter&quot;</h3><p>Wow, with Jack departing, the Twitter board collectively owns almost no shares! Objectively, their economic interests are simply not aligned with shareholders.</p><p>We can expect further trolls from Elon in the near future.</p><p>… the source code: <a href="https://www.sec.gov/Archives/edgar/data/0001418091/000110465922045641/tm2212748d1_sc13da.htm">Elon’s Twitter Buyout Doc</a></p><p>… Substack throwing up gang signs already 😭:</p><h3>Lulu Cheng Meservey on Twitter: &quot;Substack is hiring!If you&#39;re a Twitter employee who&#39;s considering resigning because you&#39;re worried about Elon Musk pushing for less regulated speech... please do not come work here. / Twitter&quot;</h3><p>Substack is hiring!If you&#39;re a Twitter employee who&#39;s considering resigning because you&#39;re worried about Elon Musk pushing for less regulated speech... please do not come work here.</p><h3>Dall-E 2 vs. Latent-Diffusion | Thoughts?</h3><h3>Denis on Twitter: &quot;Dall-E 2 vs Latent-Diffusion pic.twitter.com/jCM4YZBQjV / Twitter&quot;</h3><p>Dall-E 2 vs Latent-Diffusion pic.twitter.com/jCM4YZBQjV</p><h4>Colab of the Week 🏆Latent-Diffusion Notebook</h4><p><a href="https://colab.research.google.com/github/tg-bomze/collection-of-notebooks/blob/master/Text2Image_v5.ipynb">Google Colaboratory</a></p><h3>Megatron-DeepSpeed Hacks from BigScience</h3><p>What’s Megatron-DeepSpeed?</p><p><a href="https://github.com/microsoft/Megatron-DeepSpeed">GitHub - microsoft/Megatron-DeepSpeed: Ongoing research training transformer language models at scale, including: BERT &amp; GPT-2</a></p><p>Quick start script for Megatron-DeepSpeed:</p><p><a href="https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/main/start_fast.md">Megatron-DeepSpeed/start_fast.md at main · bigscience-workshop/Megatron-DeepSpeed</a></p><h3>GraphGallery</h3><p>A gallery for benchmarking Graph Neural Networks (GNNs) based on pure <a href="https://github.com/pytorch/pytorch">PyTorch</a> backend.</p><p><a href="https://github.com/EdisonLeeeee/GraphGallery">GitHub - EdisonLeeeee/GraphGallery: GraphGallery is a gallery for benchmarking Graph Neural Networks (GNNs).</a></p><h3>HTTPie: Human-Friendly CLI HTTP Client</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FzP2qSHEadCC-ZW0.gif" /></figure><p><a href="https://github.com/httpie/httpie?utm_campaign=explore-email&amp;utm_medium=email&amp;utm_source=newsletter&amp;utm_term=daily">GitHub - httpie/httpie: As easy as /aitch-tee-tee-pie/ 🥧 Modern, user-friendly command-line HTTP client for the API era. JSON support, colors, sessions, downloads, plugins &amp; more. https://twitter.com/httpie</a></p><h3>UFOs and OSINT | A Repo (Yes, this is real)</h3><p>UFO research is so lit, it’s now expanding into GitHub. 😭</p><p><a href="https://github.com/richgel999/uap_resources/">GitHub - richgel999/uap_resources: Key OSINT UAP Related Materials</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*wpYNWB9GibhkuhU634fgrA.gif" /></figure><p><strong>ok, ok, last thing on UFOs (promise)</strong></p><h3>Marc Cecotti on Twitter: &quot;According to the DIA, this is a list of all AAWSAP-related material they could find. These are the 1,574 pages of records about UAP and AAWSAP. pic.twitter.com/zLGEwqBDIE / Twitter&quot;</h3><p>According to the DIA, this is a list of all AAWSAP-related material they could find. These are the 1,574 pages of records about UAP and AAWSAP. pic.twitter.com/zLGEwqBDIE</p><h3>❤️Thanks for the big up Francesco!</h3><h3>Francesco Pochetti on Twitter: &quot;A couple of months ago my friend Ilias Mansouri introduced me to @neuralmagic.If you don&#39;t know the company yet, they came up with DeepSparse, a technology capable of running DL models on CPU at GPU speed 🏃🏻.Yes, you are reading this right. / Twitter&quot;</h3><p>A couple of months ago my friend Ilias Mansouri introduced me to @neuralmagic.If you don&#39;t know the company yet, they came up with DeepSparse, a technology capable of running DL models on CPU at GPU speed 🏃🏻.Yes, you are reading this right.</p><h3>KeyBART &amp; KBIR</h3><p>Keyphrase Boundary Infilling with Replacement (KBIR) achieves SOTA performance for the task of keyphrase extraction.</p><p>KBART model achieves SOTA performance on the task of keyphrase generation.</p><p><a href="https://huggingface.co/bloomberg">bloomberg (Bloomberg Finance LP)</a></p><p>Paper: <a href="https://arxiv.org/abs/2112.08547">https://arxiv.org/abs/2112.08547</a></p><p>YouTube: <a href="https://youtu.be/W31PKHl5gvo">https://youtu.be/W31PKHl5gvo</a></p><h3>NLP Models | A Timeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*AbIlg-WsPiyDOj3X" /><figcaption><a href="https://medium.com/u/3ca5a023b88f">Sami Nas</a></figcaption></figure><h3>Knowledge Graphs ICLR 2021</h3><p>ICLR 2021 papers summary.</p><p><a href="https://iclr-blog-track.github.io/2022/03/25/kgs/">Knowledge Graph Papers @ ICLR 2021</a></p><h3>Knowledge Distillation with Haystack</h3><p>Hey Deepset, if you’re reading this, let’s parlay homies!😍⭐</p><p><a href="https://www.deepset.ai/blog/knowledge-distillation-with-haystack">Knowledge Distillation with Haystack | deepset</a></p><h3>Question Answering from UKP et al</h3><p><em>Question Answering (QA) platform to enable users to easily implement, manage and share their custom QA pipelines.</em></p><p>It’s got a UI too.</p><p><a href="https://github.com/UKP-SQuARE/square-core">GitHub - UKP-SQuARE/square-core: SQuARE: Software for question answering research.</a></p><h3>Papers to Read 📚</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2204.06745.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9461ee7a6b0a1ff5f22af9380faa85f/href">https://medium.com/media/c9461ee7a6b0a1ff5f22af9380faa85f/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2204.06514.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/49498aa554240f7f1824cceee7a4d42a/href">https://medium.com/media/49498aa554240f7f1824cceee7a4d42a/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2204.05660.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/891581f84a94acaeb01e4906049e753b/href">https://medium.com/media/891581f84a94acaeb01e4906049e753b/href</a></iframe><h3>NLP Index👨‍💻</h3><h4>A collection of recently released repos that caught our 👁</h4><h4><a href="https://arxiv.org/pdf/2204.05076.pdf">End-to-End Speech Translation for Code Switched Speech</a></h4><blockquote>Code switching in the context of English/Spanish conversations for the task of speech translation (ST), generating and evaluating both transcript and translation.</blockquote><p><a href="https://github.com/apple/ml-code-switched-speech-translation">GitHub - apple/ml-code-switched-speech-translation: This repository contains the code and instructions needed to reproduce the dataset splits for out paper &quot;Speech Translation for Code-Switched Speech&quot;.</a></p><h4><a href="https://arxiv.org/pdf/2204.04980.pdf">FewIE: Few-Shot Named Entity Recognition</a></h4><blockquote>An encoder evaluation framework for comparing the performance of SOTA pre-trained representations on the task of low-resource NER.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9dfu9nKoBIriy5HN.png" /></figure><p><a href="https://github.com/dfki-nlp/fewie">GitHub - DFKI-NLP/fewie: Few-shot named entity recognition</a></p><h4><a href="https://arxiv.org/pdf/2204.05999.pdf">InCoder: A Generative Model for Code Infilling and Synthesis</a></h4><blockquote>A unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via infilling). The model is the first generative model that is able to directly perform zero-shot code infilling.</blockquote><p><a href="https://github.com/dpfried/incoder">GitHub - dpfried/incoder</a></p><h4><a href="https://arxiv.org/pdf/2204.03905.pdf">BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model</a></h4><blockquote>A generative language model BioBART that adapts BART to the biomedical domain.</blockquote><p><a href="https://github.com/GanjinZero/BioBART">GitHub - GanjinZero/BioBART: BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model [ACL-BioNLP 2022]</a></p><h4><a href="https://arxiv.org/pdf/2204.03871.pdf">Commodity News Corpus</a></h4><blockquote>A corpus of English crude oil news for event extraction.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R-Vle4L7pvyC5vvD.png" /></figure><p><a href="https://github.com/meisin/crudeoilnews-corpus">GitHub - meisin/CrudeOilNews-Corpus: Github repository for CrudeOilNews Corpus, a ACE/ERE-like corpus suitable for Event Extraction task.</a></p><h4><a href="https://arxiv.org/pdf/2204.06604.pdf">EHRKit</a></h4><blockquote>Repo for processing medical texts in electronic health records.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/0*Qg2k9VESAMbOX99g.jpg" /></figure><p><a href="https://github.com/yale-lily/ehrkit-2022">GitHub - Yale-LILY/EHRKit-2022: A Python Natural Language Processing Toolkit for Electronic Health Record Texts</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/54/0*c0C7jkCYrRYDl4FL" /><figcaption><a href="https://www.linkedin.com/in/ricky-costa-nlp/">Ricky ҈̿҈̿҈̿҈̿҈̿҈̿Costa̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ed2ba906bdb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The NLP Cypher | 03.20.22]]></title>
            <link>https://nlpcypher.medium.com/the-nlp-cypher-03-20-22-78f34f965bab?source=rss-7a94371c6509------2</link>
            <guid isPermaLink="false">https://medium.com/p/78f34f965bab</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ricky Costa]]></dc:creator>
            <pubDate>Mon, 21 Mar 2022 00:11:24 GMT</pubDate>
            <atom:updated>2022-03-21T00:11:24.482Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RIyLHO-mVk6R8UNb.jpg" /><figcaption>Elder</figcaption></figure><h4>NATURAL LANGUAGE PROCESSING (NLP) WEEKLY NEWSLETTER</h4><h4>A Preemptive Strike</h4><p>Welcome back everyone👋!!!!</p><p>I have good news… as I speak… the future is being built…</p><p>… and the future is sparse !!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*j_UCQab4ka-tYmBC6LdpuA.gif" /><figcaption>(no idea what this is doing , but looks cool)</figcaption></figure><p>With the great engineering minds at Neural Magic, we’re all actively attempting to solve a very difficult problem. A problem that continues to haunt the AI industry regarding these very large deep learning models:</p><blockquote>How do we get these large models into production without blowing up our hardware or our wallet?</blockquote><p>We all want the same robust performance with our deep learning models. We want them to be accurate, as light as possible, and fast.</p><p>So… how do we achieve this? Well… it’s with sparsity and great engineering!</p><p><a href="https://arxiv.org/pdf/2111.13445.pdf"><strong>(FYI, check out some of the latest research here)</strong></a></p><p>I have to be honest. I didn’t know how fast the research into sparsity was moving until I recently joined NM. In fact, I began experimenting with some of the latest software features released from the DeepSparse repo and I was really blown away (deets below).</p><p>For those new to the library, DeepSparse is an inference engine giving GPU level performance to sparsified models running on CPUs. 💪</p><p>So what’s an awesome feature from the repo? How about the DeepSparse Server, or as we nerds refer to it:</p><p>deepsparse.server</p><p>It’s an awesome inference HTTP server (built on top of FastAPI) that allows anyone to serve sparsified ONNX models (p.s. you can serve dense ONNX models too).</p><p>So very recently, I began to experiment with the deepsparse.serveron a Google Cloud instance (c2-standard-4 | 4vcpu’s and 16GB of RAM) to see how it would perform. And the performance of both the server and the sparse models from Neural Magic’s <a href="https://sparsezoo.neuralmagic.com/?domain=nlp&amp;sub_domain=question_answering&amp;page=1">SparseZoo</a> were remarkable! First, I was amazed I was able to load 18 BERTs on only 16GB of RAM 🥶. Which models you may ask? These 👇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*0P64sXzhNkGnZHzj" /></figure><p>And the models’ performance was incredible. I noticed a remarkable speedup in the latency of each model when compared to the previous month (where I tested the same exact models) however my cloud compute didn’t change. And as it turns out, one of the best kept secrets with Neural Magic software is that as the optimization software keeps improving over time, the models’ performance just keeps getting faster and faster. It’s the equivalent of parking a sedan in your garage and then in a few months you open the garage doors and you have Lambo. This is what I mean 👇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c5piYamJbry3moXZ_9rCiQ.jpeg" /></figure><p>Notice how throughput (y-axis) keeps improving across different versions of DeepSparse (v0.7 to v0.9)?? The improvements are due to the <a href="https://neuralmagic.com/blog/neural-magic-ce-0-7-0-8-and-0-9-product-releases/">Tensor Columns</a> tech from us. Btw, we’re now on v0.11 so current performance is even better than what the graphic shows.</p><p>Ok Ok…. before I get back to the newsletter, I want to give you a heads up that I will eventually write a more detailed blog post including steps to reproduce my experiment (including code for running the server etc.) in due time. I just wanted to give you a heads up of why 2022 is a really exciting time to be working in deep learning and sparsity! ❤️</p><p><strong>When you can, check out the DeepSparse repo and give it some⭐⭐⭐!</strong></p><p><a href="https://github.com/neuralmagic/deepsparse">GitHub - neuralmagic/deepsparse: Neural network inference engine that delivers GPU-class performance for sparsified models on CPUs</a></p><p>and now… for this week in NLP…🤣 …</p><h3>They Promised Us Flying Cars, Instead We Got…</h3><p><a href="https://tinder4cats.com/">Tinder for Cats</a></p><h3>They Promised Us Tinder for Cats, Instead We Got…</h3><p>(encryption with Emojis 😂😂)</p><p><a href="https://mysecretmessage.pythonanywhere.com/">My Secret Message</a></p><h3>Coding an Entire Video Game Using OpenAI’s DaVinci</h3><p>This is next level.</p><p><a href="https://andrewmayneblog.wordpress.com/2022/03/17/building-games-and-apps-entirely-through-natural-language-using-openais-davinci-code-model/">Building games and apps entirely through natural language using OpenAI&#39;s code-davinci model</a></p><h3>Tech Ops Douche Level Check List</h3><p>I averaged 3.5 ⭐⭐⭐but don’t think it’s `mission-critical`!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/0*1K5SQrdGY819gPkt" /></figure><h3>PyTorch vs. Tensorflow the Eternal Battle</h3><p><a href="https://horace.io/pytorch-vs-tensorflow/">https://horace.io/pytorch-vs-tensorflow/</a></p><h3>A Visual Intro to ML</h3><p><a href="http://www.r2d3.us/visual-intro-to-machine-learning-part-1/">A visual introduction to machine learning</a></p><h3>Goopt: A Search Engine but GPT-3</h3><p>This is proof we are in a simulation.</p><p><a href="https://github.com/jokenox/Goopt">GitHub - jokenox/Goopt: 🔍 Search Engine for a Procedural Simulation of the Web with GPT-3.</a></p><h3>New CodeParrot Version Released🦜</h3><p>CodeParrot is a GPT-2 model (1.5B parameters) trained to generate Python code.</p><p><a href="https://huggingface.co/lvwerra/codeparrot">lvwerra/codeparrot · Hugging Face</a></p><h3>You.com Gets an AI Writing Assistant</h3><p>You.com, a new search engine founded by Ex-Salesforce peeps, now comes with it’s own AI writer.</p><p><a href="https://you.com/search?q=how%20to%20write%20well">you.com</a></p><h3>What Does it Take to Win at Competitive ML?</h3><p><a href="https://blog.mlcontests.com/p/winning-at-competitive-ml-in-2022?s=w">Winning at Competitive ML in 2022</a></p><h3>When Hackers Get Bored</h3><blockquote>TorBot is an open source intelligence tool developed in python. The main objective of this project is to collect open data from the deep web (aka dark web) and with the help of data mining algorithms, collect as much information as possible and produce an interactive tree graph. The interactive tree graph module will be able to display the relations of the collected intelligence data.</blockquote><p><a href="https://hakin9.org/torbot-open-source-intelligence-tool-for-the-dark-web/">TorBot - Open Source Intelligence Tool for the Dark Web</a></p><h3>The AI Index is Out</h3><p>The AI Index was released, juicy stuff begins on pg. 45 imo…</p><p><a href="https://aiindex.stanford.edu/report/">AI Index | Stanford HAI</a></p><p><strong>Full report: </strong><a href="https://aiindex.stanford.edu/wp-content/uploads/2022/03/2022-AI-Index-Report_Master.pdf">https://aiindex.stanford.edu/wp-content/uploads/2022/03/2022-AI-Index-Report_Master.pdf</a></p><h3>Papers to Read 📚</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2203.07586.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/d338f42c7ed7b3d7b9486bfc4f6b30e1/href">https://medium.com/media/d338f42c7ed7b3d7b9486bfc4f6b30e1/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2203.00555.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/87d31dc68f90aeed0a0b1b44c2dac272/href">https://medium.com/media/87d31dc68f90aeed0a0b1b44c2dac272/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2202.06397.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ae7358f789f1deea08e0514c318b805/href">https://medium.com/media/9ae7358f789f1deea08e0514c318b805/href</a></iframe><h3>Repo Cypher 👨‍💻</h3><h4>A collection of recently released repos that caught our 👁</h4><h4><a href="https://arxiv.org/pdf/2203.07228.pdf">FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing</a></h4><blockquote>Benchmark includes four datasets for evaluating the fairness of pre-trained legal language models and the techniques used to fine-tune them for downstream tasks. The benchmarks cover four jurisdictions (European Council, USA, Swiss, and Chinese), five languages (English, German, French, Italian and Chinese) and fairness across five attributes (gender, age, nationality/region, language, and legal area).</blockquote><p><a href="https://github.com/coastalcph/fairlex">GitHub - coastalcph/fairlex: Fairlex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing</a></p><h4><a href="https://arxiv.org/pdf/2203.07281.pdf">GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models</a></h4><blockquote>GRIPS takes in instructions designed for humans and automatically returns an improved, edited prompt, while allowing for API-based tuning.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*meQUiW6wv-sZWeXd.png" /></figure><p><a href="https://github.com/archiki/grips">GitHub - archiki/GrIPS: Code for our paper: &quot;GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models&quot;</a></p><h4><a href="https://arxiv.org/pdf/2203.07781.pdf">UniSAr: A Unified Structure-Aware Autoregressive Language Model for Text-to-SQL</a></h4><blockquote>UNISAR extends existing autoregressive language models to incorporate three non-invasive extensions to make them structure-aware: (1) adding structure mark to encode database schema, conversation context, and their relationships; (2) constrained decoding to decode well structured SQL for a given database schema; and (3) SQL completion to complete potential missing JOIN relationships in SQL based on database schema.</blockquote><p><a href="https://github.com/dreamerdeo/unisar_text2sql">GitHub - DreamerDeo/UniSAr_text2sql: &quot;UniSAr: A Unified Structure-Aware Autoregressive Language Model for Text-to-SQL&quot;</a></p><h4><a href="https://arxiv.org/pdf/2203.07285.pdf">Diversifying Commonsense Reasoning Generation on Knowledge Graph</a></h4><blockquote>MoKGE, a novel method that diversifies the generative commonsense reasoning by a mixture of expert (MoE) strategy on knowledge graphs (KG).</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u367Cm5JYwA8M03o.jpg" /></figure><p><a href="https://github.com/DM2-ND/MoKGE">GitHub - DM2-ND/MoKGE: Author: Wenhao Yu (wyu1@nd.edu). ACL 2022. Commonsense Reasoning on Knowledge Graph for Text Generation</a></p><h4><a href="https://arxiv.org/pdf/2203.05081.pdf">NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Tasks</a></h4><blockquote>NLX-GPT, a language model that can simultaneously predict an answer and explain it.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/0*eTXkxhrBZyNE44rs.png" /></figure><p><a href="https://github.com/fawazsammani/nlxgpt">GitHub - fawazsammani/nlxgpt: NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Tasks, CVPR 2022</a></p><h4>Tevatron</h4><blockquote>Tevatron is a toolkit for training and running dense retrievers with deep language models.</blockquote><p><a href="https://github.com/texttron/tevatron">GitHub - texttron/tevatron: Tevatron - A flexible toolkit for dense retrieval research and development.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/54/0*c0C7jkCYrRYDl4FL" /><figcaption><a href="https://www.linkedin.com/in/ricky-costa-nlp/">Ricky Costa</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78f34f965bab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The NLP Cypher | 02.06.21]]></title>
            <link>https://nlpcypher.medium.com/the-nlp-cypher-02-06-21-3d6e34bc5b15?source=rss-7a94371c6509------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d6e34bc5b15</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ricky Costa]]></dc:creator>
            <pubDate>Sun, 06 Feb 2022 23:39:28 GMT</pubDate>
            <atom:updated>2022-02-06T23:39:28.830Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*LZCemisT8wtG_yJf.jpg" /><figcaption>A Seaport at Sunrise | Lorrain</figcaption></figure><h4>NATURAL LANGUAGE PROCESSING (NLP) WEEKLY NEWSLETTER</h4><h4>NeoX</h4><h3>A Vendetta and 404s</h3><p><a href="https://www.wired.com/story/north-korea-hacker-internet-outage/">North Korea Hacked Him. So He Took Down Its Internet</a></p><h3>Meanwhile Everything dropped this week…</h3><p>DeepMind’s AlphaCode</p><p><a href="https://alphacode.deepmind.com/">AlphaCode</a></p><h3>Meanwhile back at the ranch…</h3><p>Math Olympiad solver from OpenAI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ciKe_nYv7PuocGDI" /></figure><p><strong>Blog:</strong></p><p><a href="https://openai.com/blog/formal-math/">Solving (Some) Formal Math Olympiad Problems</a></p><h3>Meanwhile back at the ranch… again…</h3><p>New GPT-NeoX 20B params dropped:</p><p><a href="https://news.ycombinator.com/item?id=30179398">Announcing GPT-NeoX-20B | Hacker News</a></p><h3>New Transformer Book Repo with Colab Notebooks!</h3><p><a href="https://github.com/nlp-with-transformers/notebooks">GitHub - nlp-with-transformers/notebooks: Jupyter notebooks for the Natural Language Processing with Transformers book</a></p><h3>Parsr: A PDF Parser that doesn’t suck 😎</h3><p><a href="https://github.com/axa-group/Parsr">GitHub - axa-group/Parsr: Transforms PDF, Documents and Images into Enriched Structured Data</a></p><h3>SBERT Author Shreds the New GPT-3 Embeddings Offering 🥶🥶</h3><p><strong><em>“The biggest downside for the OpenAI embeddings endpoint is the high costs (about 8,000–600,000 times more expensive than open models on your infrastructure), the high dimensionality of up to 12288 dimensions (making downstream applications slow), and the extreme latency when computing embeddings. This hinders the actual usage of the embeddings for any search applications.”</em></strong></p><p><a href="https://medium.com/@nils_reimers/openai-gpt-3-text-embeddings-really-a-new-state-of-the-art-in-dense-text-embeddings-6571fe3ec9d9">OpenAI GPT-3 Text Embeddings - Really a new state-of-the-art in dense text embeddings?</a></p><h3>🥶 Oops: Exposed databases on AWS</h3><p>FYI: I had previously written about this issue over a year ago and even provided a search engine, it seems now more peeps are on top of this issue.</p><p><a href="https://infosecwriteups.com/how-i-discovered-thousands-of-open-databases-on-aws-764729aa7f32">How I Discovered Thousands of Open Databases on AWS</a></p><h4>Scan the entire internet under 5 minutes:</h4><p><a href="https://github.com/robertdavidgraham/masscan#how-to-scan-the-entire-internet">GitHub - robertdavidgraham/masscan: TCP port scanner, spews SYN packets asynchronously, scanning entire Internet in under 5 minutes.</a></p><p><strong>or…</strong></p><p><a href="https://positive.security/blog/video-depixelation">Recovering redacted information from pixelated videos | Positive Security</a></p><h3>How I got an FBI record at age 11 from dabbling in cryptography then got into more trouble 😭😭</h3><p><a href="https://web.stanford.edu/~learnest/les/crypto.htm">I Spy</a></p><h3>ViLT Notebook for Visual Question Answering</h3><p>author: Niels Rogge @ Hugging Face</p><p><strong>Colab:</strong></p><p><a href="https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/ViLT/Inference_with_ViLT_(visual_question_answering).ipynb">Google Colaboratory</a></p><p><strong>Space:</strong></p><p><a href="https://huggingface.co/spaces/nielsr/vilt-vqa">Vilt Vqa - a Hugging Face Space by nielsr</a></p><h3>How to Improve User Experience (and Behavior): Three Papers from Stanford’s Alexa Prize Team</h3><p><a href="https://ai.stanford.edu/blog/alexa-sigdial/">How to Improve User Experience (and Behavior): Three Papers from Stanford&#39;s Alexa Prize Team</a></p><h3>For Practioners: How GPUs Work | A Thread</h3><p><a href="https://threadreaderapp.com/convos/1489222150384848900">Twitter conversation started by @marktenenholtz</a></p><h3>Data Engineering in Julia</h3><p><a href="https://towardsdatascience.com/data-engineering-in-julia-3fd37eaa618a">Data Engineering in Julia</a></p><h3>DeepChecks</h3><p>A repo for validating models and data.</p><p><a href="https://github.com/deepchecks/deepchecks">GitHub - deepchecks/deepchecks: Deepchecks: Tests for Continuous Validation of ML Models &amp; Data. Deepchecks is a holistic open-source solution for all of your AI &amp; ML validation needs, enabling to thoroughly test your data and models from research to production.</a></p><h3>Task-Specific Knowledge Distillation for BERT using Transformers &amp; Amazon SageMaker</h3><p><a href="https://www.philschmid.de/knowledge-distillation-bert-transformers">Task-specific knowledge distillation for BERT using Transformers &amp; Amazon SageMaker</a></p><h3>Papers to Read 📚</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2201.05596.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fed21feeb234abe494edc647009ddea/href">https://medium.com/media/7fed21feeb234abe494edc647009ddea/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2201.11990.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/da29ed3688d04aa5acfd3b5a2d877f4d/href">https://medium.com/media/da29ed3688d04aa5acfd3b5a2d877f4d/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2202.01110.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8a8ddb415ae97c0be5e470a3f06de46/href">https://medium.com/media/f8a8ddb415ae97c0be5e470a3f06de46/href</a></iframe><h3>Repo Cypher 👨‍💻</h3><h4>A collection of recently released repos that caught our 👁</h4><h4><a href="https://arxiv.org/pdf/2202.00454.pdf">TableQuery: Querying tabular data with natural language</a></h4><blockquote>AI Tool for querying natural language on tabular data.</blockquote><blockquote>Tabular data can be:</blockquote><blockquote>::Dataframes</blockquote><blockquote>::CSV files</blockquote><p><a href="https://github.com/abhijithneilabraham/tableqa">GitHub - abhijithneilabraham/tableQA: AI Tool for querying natural language on tabular data.</a></p><p><a href="https://www.connectedpapers.com/main/00cb569d73c1c9f71dfcd3b321068c21ecd8ed90"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2201.13230.pdf">POTATO: exPlainable infOrmation exTrAcTion framewOrk</a></h4><blockquote>POTATO is a human-in-the-loop XAI framework for extracting and evaluating interpretable graph features for any classification problem in NLP.</blockquote><p><a href="https://github.com/adaamko/potato">GitHub - adaamko/POTATO: XAI based human-in-the-loop framework for automatic rule-learning.</a></p><p><a href="https://www.connectedpapers.com/main/b6ffe594660770705aee04c49f2a8969c75d46b5"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2202.01279.pdf">PrompSource</a></h4><blockquote>PromptSource is a toolkit for creating, sharing and using natural language prompts. Work from the BigScience initiative.</blockquote><p><a href="https://github.com/bigscience-workshop/promptsource">GitHub - bigscience-workshop/promptsource: Toolkit for creating, sharing and using natural language prompts.</a></p><p><a href="https://www.connectedpapers.com/main/8264edb2f4e4eea0f5322ea8cec72471fa6a99dd"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2202.00443.pdf">Text Anonymization Benchmark (TAB)</a></h4><blockquote>The <em>Text Anonymization Benchmark</em> (TAB) is a new, open-source corpus for text anonymization. It comprises 1,268 English-language court cases from the <a href="https://www.echr.coe.int/Pages/home.aspx?p=home">European Court of Human Rights (ECHR)</a> manually annotated with:</blockquote><ul><li>semantic categories for personal identifiers,</li><li>masking decisions (in regard to the re-identification risk for the person to protect),</li><li>confidential attributes,</li><li>co-reference relations.</li></ul><p><a href="https://github.com/norskregnesentral/text-anonymisation-benchmark">GitHub - NorskRegnesentral/text-anonymisation-benchmark: Annotated corpus + evaluation metrics for text anonymisation</a></p><p><a href="https://www.connectedpapers.com/main/0d2b9d71a16e0a6008d88760b756c9769e653410"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2202.00631.pdf">FiNCAT: Financial Numeral Claim Analysis Tool</a></h4><blockquote>A tool to detect whether numerals present in Financial Texts are in-claim or out-of-claim.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/0*VjJzxvNi21M-3LJf.png" /></figure><p><a href="https://github.com/sohomghosh/fincat_financial_numeral_claim_analysis_tool">GitHub - sohomghosh/FiNCAT_Financial_Numeral_Claim_Analysis_Tool: A tool to detect whether numerals present in Financial Texts are in-claim or out-of-claim</a></p><p><a href="https://www.connectedpapers.com/main/0f9bbd518d5ff995cec53b4c5579263e833c4873"><strong>Connected Papers</strong></a><strong> 📈</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/54/0*c0C7jkCYrRYDl4FL" /><figcaption><a href="https://quantumstat.com/">Quantum Stat</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d6e34bc5b15" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The NLP Cypher | 01.23.22]]></title>
            <link>https://nlpcypher.medium.com/the-nlp-cypher-01-23-21-b5ab5c50af3b?source=rss-7a94371c6509------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5ab5c50af3b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ricky Costa]]></dc:creator>
            <pubDate>Sun, 23 Jan 2022 21:06:53 GMT</pubDate>
            <atom:updated>2022-01-23T21:17:18.729Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_DtWfh9oS54%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_DtWfh9oS54&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F_DtWfh9oS54%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a1a76b742f1fa091612507764714f85/href">https://medium.com/media/5a1a76b742f1fa091612507764714f85/href</a></iframe><h4>NATURAL LANGUAGE PROCESSING (NLP) WEEKLY NEWSLETTER</h4><h3>The NLP Cypher | 01.23.22</h3><h4>Desiderata</h4><p>🕵️‍♂️Has AI interest peaked?</p><h3>Andrew Trask on Twitter: &quot;Peak interest in Deep Learning (blue) and Artificial Intelligence (read) appears to have been Sep 2019.Google searches for AI/DL are down about 40% since then. Sharp decline in AI searches in the last 4 months.https://t.co/cNEqJ4yV10 pic.twitter.com/oPbVSYc0eK / Twitter&quot;</h3><p>Peak interest in Deep Learning (blue) and Artificial Intelligence (read) appears to have been Sep 2019.Google searches for AI/DL are down about 40% since then. Sharp decline in AI searches in the last 4 months.https://t.co/cNEqJ4yV10 pic.twitter.com/oPbVSYc0eK</p><p><a href="https://trends.google.com/trends/explore?date=all&amp;q=deep%20learning,Artificial%20Intelligence">https://trends.google.com/trends/explore?date=all&amp;q=deep%20learning,Artificial%20Intelligence</a></p><p>If you’re bummed, you can always… 👇</p><p><a href="https://makefrontendshitagain.party/">Make Frontend Shit Again</a></p><h3>Graph ML in 2022: Where Are We Now?</h3><p><a href="https://towardsdatascience.com/graph-ml-in-2022-where-are-we-now-f7f8242599e0">Graph ML in 2022: Where Are We Now?</a></p><h3>The State of Web-Scraping 2022</h3><p><a href="https://scrapeops.io/blog/the-state-of-web-scraping-2022/">The State of Web Scraping 2022 | ScrapeOps</a></p><h3>DARPA and OSS 🕵️‍♀️</h3><p>Press Release: <a href="https://www.darpa.mil/news-events/2021-12-21">https://www.darpa.mil/news-events/2021-12-21</a></p><blockquote>The <a href="https://www.darpa.mil/program/guaranteeing-ai-robustness-against-deception">DARPA GARD program</a> seeks to establish theoretical ML system foundations to identify system vulnerabilities, characterize properties that will enhance system robustness, and encourage the creation of effective defenses. Currently, ML defenses tend to be highly specific and are effective only against particular attacks. GARD seeks to develop defenses capable of defending against broad categories of attacks. Furthermore, current evaluation paradigms of AI robustness often focus on simplistic measures that may not be relevant to security. To verify relevance to security and wide applicability, defenses generated under GARD will be measured in a novel testbed employing scenario-based evaluations.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TpmfZ5c3XZ_fPmjA" /></figure><h4>Repos mentioned in the press release:</h4><ul><li><a href="https://github.com/twosixlabs/armory">GitHub - twosixlabs/armory: ARMORY Adversarial Robustness Evaluation Test Bed</a></li><li><a href="https://github.com/Trusted-AI/adversarial-robustness-toolbox">GitHub - Trusted-AI/adversarial-robustness-toolbox: Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams</a></li></ul><h3>State of Machine Learning in Julia</h3><p><a href="https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385">State of machine learning in Julia</a></p><h3>For Those interested in Semantic Similarity</h3><p><a href="https://cmry.github.io/notes/euclidean-v-cosine">Euclidean vs. Cosine Distance</a></p><h3>Free CS Classes</h3><h3>MIT CSAIL on Twitter: &quot;800 free computer science classes you can take online right now: https://t.co/RURjy6O01h pic.twitter.com/Bm97oHBBZ2 / Twitter&quot;</h3><p>800 free computer science classes you can take online right now: https://t.co/RURjy6O01h pic.twitter.com/Bm97oHBBZ2</p><h3>Google Style Guide for Python</h3><p><a href="https://google.github.io/styleguide/pyguide.html">styleguide</a></p><h3>From the Creator of FastAPI 👉 Asyncer</h3><blockquote>“The main goal of <strong>Asyncer</strong> is to improve <strong>developer experience</strong> by providing better support for <strong>autocompletion</strong> and <strong>inline errors</strong> in the editor, and <strong>more certainty</strong> that the code is <strong>bug-free</strong> by providing better support for type checking tools like <strong>mypy</strong>.”</blockquote><p><a href="https://asyncer.tiangolo.com/">Asyncer</a></p><h3>Real-Time Machine Learning</h3><p><a href="https://huyenchip.com/2022/01/02/real-time-machine-learning-challenges-and-solutions.html">Real-time machine learning: challenges and solutions</a></p><h3>Handling Large Messages with Kafka</h3><p><a href="https://www.kai-waehner.de/blog/2020/08/07/apache-kafka-handling-large-messages-and-files-for-image-video-audio-processing/amp/">Handling Large Messages with Apache Kafka (CSV, XML, Image, Video, Audio, Files) - Kai Waehner</a></p><h3>Sentence Segmentation</h3><ul><li><a href="https://github.com/nipunsadvilkar/pySBD">GitHub - nipunsadvilkar/pySBD: 🐍💯pySBD (Python Sentence Boundary Disambiguation) is a rule-based sentence boundary detection that works out-of-the-box.</a></li><li><a href="https://github.com/SHI-Labs/Rethinking-Text-Segmentation">GitHub - SHI-Labs/Rethinking-Text-Segmentation: [CVPR 2021] Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach</a></li></ul><h3>Kaggle Solutions Repo</h3><p><a href="https://github.com/faridrashidi/kaggle-solutions">GitHub - faridrashidi/kaggle-solutions: 🏅 Collection of Kaggle Solutions and Ideas 🏅</a></p><h3>Happy Transformer</h3><p><a href="https://github.com/EricFillion/happy-transformer">GitHub - EricFillion/happy-transformer: A package built on top of Hugging Face&#39;s transformers library that makes it easy to utilize state-of-the-art NLP models</a></p><h3>OSLO: Extending the Training Capability for Transformers</h3><h3>Stas Bekman on Twitter: &quot;A new library that extends 🤗 Transformers to add 3D parallelism and fast kernel fusion has been released:https://t.co/S6sMfcZ5TMCurrently it supports gpt2, gpt_neo, gptj and more models will be supported soon.Courtesy of @hyunwoongko / Twitter&quot;</h3><p>A new library that extends 🤗 Transformers to add 3D parallelism and fast kernel fusion has been released:https://t.co/S6sMfcZ5TMCurrently it supports gpt2, gpt_neo, gptj and more models will be supported soon.Courtesy of @hyunwoongko</p><p><a href="https://github.com/tunib-ai/oslo">GitHub - tunib-ai/oslo: OSLO: Open Source framework for Large-scale transformer Optimization</a></p><h3>SeaTunnel</h3><p>Problems it attempts to solve:</p><ul><li>Data loss and duplication</li><li>Task accumulation and delay</li><li>Low throughput</li><li>Long cycle to be applied in the production environment</li><li>Lack of application running status monitoring</li></ul><p><a href="https://github.com/apache/incubator-seatunnel?utm_campaign=explore-email&amp;utm_medium=email&amp;utm_source=newsletter&amp;utm_term=daily">GitHub - apache/incubator-seatunnel: SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline &amp; real-time).</a></p><h3>Cresset — A PyTorch Universal Docker Template</h3><p><a href="https://github.com/veritas9872/PyTorch-Universal-Docker-Template">GitHub - veritas9872/PyTorch-Universal-Docker-Template: Template repository to build PyTorch projects from source on any version of PyTorch/CUDA/cuDNN.</a></p><h3>Papers to Read📚</h3><ul><li><a href="https://arxiv.org/abs/2201.08239">LaMDA: Language Models for Dialog Applications</a></li><li><a href="https://arxiv.org/abs/2106.09226">Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning</a></li></ul><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2110.03742.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/918192ba2b1a0d614818f8c8f1de3fb8/href">https://medium.com/media/918192ba2b1a0d614818f8c8f1de3fb8/href</a></iframe><p>From the Lex Fridman podcast featuring Yann LeCun as guest:</p><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2105.04906v2.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/a006f3950c7c5c76dbf065f62c1b0007/href">https://medium.com/media/a006f3950c7c5c76dbf065f62c1b0007/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSGzMElJ11Cc%3Fstart%3D3125%26feature%3Doembed%26start%3D3125&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSGzMElJ11Cc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FSGzMElJ11Cc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3eb597cda1dc93abd85ccfc474f87caa/href">https://medium.com/media/3eb597cda1dc93abd85ccfc474f87caa/href</a></iframe><p>It’s cued up to the moment Yann mentions the paper above.</p><h3>Repo Cypher 👨‍💻</h3><h4>A collection of recently released repos that caught our 👁</h4><h4><a href="https://arxiv.org/pdf/2201.06777.pdf">COPA-SSE</a></h4><blockquote>COPA-SSE contains crowdsourced explanations for the <a href="https://balanced-copa.github.io/">Balanced COPA</a> dataset, a variant of the <a href="https://people.ict.usc.edu/~gordon/copa.html">Choice of Plausible Alternatives (COPA)</a> benchmark. The explanations are formatted as a set of triple-like common sense statements with ConceptNet relations but freely written concepts.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UBeCoSXRdS5J4vNt.png" /></figure><p><a href="https://github.com/a-brassard/copa-sse">GitHub - a-brassard/copa-sse: Repository for Semi-Structured Explanations for COPA (COPA-SSE)</a></p><p><a href="https://www.connectedpapers.com/main/f18827af6c2d12d3ba78ee0cdb31808a6f909c4b"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2201.06206.pdf">SQUIRE: A Sequence-to-sequence Framework for Multi-hop Knowledge Graph Reasoning</a></h4><blockquote>The first sequence-to-sequence based multi-hop reasoning framework, which utilizes an encoder-decoder structure to translate the triple query to a multi-hop path.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1OzM35plbLppc7MJ.png" /></figure><p><a href="https://github.com/bys0318/squire">GitHub - bys0318/SQUIRE</a></p><p><a href="https://www.connectedpapers.com/main/5d9cd76c3e4e8cd8e59e4f57b25a467c56c08638"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2201.03713.pdf">CVSS A Massively Multilingual Speech-to-Speech Translation Corpus</a></h4><blockquote><em>CVSS</em> is a massively multilingual-to-English speech-to-speech translation corpus, covering sentence-level parallel speech-to-speech translation pairs from 21 languages into English.</blockquote><p><a href="https://github.com/google-research-datasets/cvss">GitHub - google-research-datasets/cvss: CVSS: A Massively Multilingual Speech-to-Speech Translation Corpus</a></p><p><a href="https://www.connectedpapers.com/main/6e266a7d7970184c2fcd4373b31719cbd7eeaf43/CVSS-Corpus-and-Massively-Multilingual-Speech%20to%20Speech-Translation/graph"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2201.07311.pdf">Datasheet for the Pile</a></h4><blockquote>This datasheet describes the Pile, a 825 GiB dataset of human-authored text compiled by EleutherAI for use in large-scale language modeling. The Pile is comprised of 22 different text sources, ranging from original scrapes done for this project, to text data made available by the data owners, to third-party scrapes available online.</blockquote><p><a href="https://github.com/EleutherAI/The-Pile">GitHub - EleutherAI/the-pile</a></p><p><a href="https://www.connectedpapers.com/main/bd44f34b47c8a4b6947695853fc2814ac69664a6"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2201.05966.pdf">UnifiedSKG📚: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models</a></h4><blockquote>The UnifiedSKG framework, which unifies 21 SKG tasks into the text-to-text format, aiming to promote systematic SKG research - instead of being exclusive to a single task, domain, or dataset. It shows that large language models like T5, with simple modification when necessary, achieve state-of-the-art performance on nearly all 21 tasks.</blockquote><ul><li><a href="https://github.com/hkunlp/unifiedskg">GitHub - HKUNLP/UnifiedSKG: A Unified Framework and Analysis for Structured Knowledge Grounding</a></li><li><a href="https://colab.research.google.com/drive/1f9yTXC3GpSyRJOjzsKceG_bhk-Cw71Ga#scrollTo=r_3-DN0SvC97">Google Colaboratory</a></li></ul><p><a href="https://www.connectedpapers.com/main/c8efc90cde659ba22f48c96b486b45465c8e8bf9"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2201.06009.pdf">Memory-assisted prompt editing to improve GPT-3 after deployment</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dI4RmIN66c09YJTr.png" /></figure><p><a href="https://github.com/madaan/memprompt">GitHub - madaan/memprompt</a></p><p><a href="https://www.connectedpapers.com/main/448e1493034dafe35699ae054ff4480b31dcf64a"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2201.07281.pdf">TweebankNLP</a></h4><blockquote>Tweebank-NER, an NER corpus of tweets based on Tweebank V2 (TB2) dataset.</blockquote><p><a href="https://github.com/social-machines/TweebankNLP">GitHub - social-machines/TweebankNLP: An off-the-shelf pre-trained Tweet NLP pipeline (NER, tokenization, lemmatization, POS tagging, dependency parsing) + Tweebank-NER dataset</a></p><p><a href="https://www.connectedpapers.com/main/4f3c217dbd1c30b0deecac9cc2c520dfce765f6f"><strong>Connected Papers</strong></a><strong> 📈</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/54/0*c0C7jkCYrRYDl4FL" /><figcaption><a href="https://quantumstat.com/">Quantum Stat</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5ab5c50af3b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The NLP Cypher | 12.26.21]]></title>
            <link>https://nlpcypher.medium.com/the-nlp-cypher-12-26-21-5c91fa16fc85?source=rss-7a94371c6509------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c91fa16fc85</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Ricky Costa]]></dc:creator>
            <pubDate>Sun, 26 Dec 2021 22:51:58 GMT</pubDate>
            <atom:updated>2021-12-26T22:51:58.716Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/474/0*gkZCHmmNKMpAcAXD.jpg" /><figcaption>The Titan’s Goblet | Cole</figcaption></figure><h4>NATURAL LANGUAGE PROCESSING (NLP) WEEKLY NEWSLETTER</h4><h4>AI Summer is Out Forever</h4><p>Merry Christmas 🎄 for those celebrating. And Happy New Year!</p><p>Even OpenAI is feeling the holiday spirit: they open sourced their photorealistic GLIDE model several days ago.</p><p><a href="https://github.com/openai/glide-text2im">GitHub - openai/glide-text2im: GLIDE: a diffusion-based text-conditional image synthesis model</a></p><p><strong>Includes three notebooks:</strong></p><p>The <a href="https://github.com/openai/glide-text2im/blob/main/notebooks/text2im.ipynb">text2im</a></p><ul><li>notebook shows how to use GLIDE (filtered) with classifier-free guidance to produce images conditioned on text prompts.</li></ul><p>The <a href="https://github.com/openai/glide-text2im/blob/main/notebooks/inpaint.ipynb">inpaint</a></p><ul><li>notebook shows how to use GLIDE (filtered) to fill in a masked region of an image, conditioned on a text prompt.</li></ul><p>The <a href="https://github.com/openai/glide-text2im/blob/main/notebooks/clip_guided.ipynb">clip_guided</a></p><ul><li>notebook shows how to use GLIDE (filtered) + a filtered noise-aware CLIP model to produce images conditioned on text prompts.</li></ul><h3>Parallel Inference with Adapters</h3><p>A new feature on the adapters library for conducting inference with various adapters simultaneously. (not sure if parallizing is a real word, I just made it up).</p><p>Colab of the Week 🎉🥳</p><p><a href="https://colab.research.google.com/github/Adapter-Hub/adapter-transformers/blob/master/notebooks/Parallel_Adapter_Inference.ipynb">Google Colaboratory</a></p><h3>SetFit: Outperforming GPT-3 in Few-Shot Text-Classification</h3><p><a href="https://towardsdatascience.com/sentence-transformer-fine-tuning-setfit-outperforms-gpt-3-on-few-shot-text-classification-while-d9a3788f0b4e">Sentence Transformer Fine-Tuning (SetFit): Outperforms GPT-3 on few-shot Text-Classification while…</a></p><p><strong>Colab</strong></p><p><a href="https://colab.research.google.com/github/MosheWasserb/SetFit/blob/main/SetFit_SST_2.ipynb">Google Colaboratory</a></p><h3>No more Transformer Diagrams 😂</h3><p>Abhishek maps boring model diagrams to code for building intuition!</p><h3>abhishek on Twitter: &quot;&quot;Attention is all you need&quot; implementation from scratch in PyTorch. A Twitter thread: 1/ / Twitter&quot;</h3><p>&quot;Attention is all you need&quot; implementation from scratch in PyTorch. A Twitter thread: 1/</p><h3>AGI and the Gov’t Apathy</h3><p>lol</p><p><a href="https://forum.effectivealtruism.org/posts/wdk3LCg6iFxknCYG4/why-don-t-governments-seem-to-mind-that-companies-are">Why don&#39;t governments seem to mind that companies are explicitly trying to make AGIs? - EA Forum</a></p><h3>Periodic Table of NLP Tasks</h3><p>Streamlit demo…</p><p><a href="https://www.innerdoc.com/periodic-table-demo/">Periodic Table demo</a></p><h3>JellyFish</h3><p>JellyFish is a library for approximate &amp; phonetic matching of strings.</p><p>Algos used…</p><p><strong>For string comparison:</strong></p><ul><li>Levenshtein Distance</li><li>Damerau-Levenshtein Distance</li><li>Jaro Distance</li><li>Jaro-Winkler Distance</li><li>Match Rating Approach Comparison</li><li>Hamming Distance</li></ul><p><strong>For phonetic encoding:</strong></p><ul><li>American Soundex</li><li>Metaphone</li><li>NYSIIS (New York State Identification and Intelligence System)</li><li>Match Rating Codex</li></ul><p><a href="https://github.com/jamesturk/jellyfish">GitHub - jamesturk/jellyfish: 🪼 a python library for doing approximate and phonetic matching of strings.</a></p><h3>New Speech Models from Microsoft on 🤗 Hub</h3><h3>Hugging Face on Twitter: &quot;WavLM currently holds the🥇place on the SUPERB benchmark.👉 https://t.co/m3ZWUuSA4QUniSpeech-SAT yields impressive results on all speaker-related tasks.👉 https://t.co/yuCzTA4DDOUniSpeech can be used for phoneme classification.👉 https://t.co/OvOdwHxSo4 / Twitter&quot;</h3><p>WavLM currently holds the🥇place on the SUPERB benchmark.👉 https://t.co/m3ZWUuSA4QUniSpeech-SAT yields impressive results on all speaker-related tasks.👉 https://t.co/yuCzTA4DDOUniSpeech can be used for phoneme classification.👉 https://t.co/OvOdwHxSo4</p><h3>TextPruner</h3><p>… a model pruning toolkit for pre-trained language models.</p><p><a href="https://github.com/airaria/TextPruner/blob/main/README_EN.md">TextPruner/README_EN.md at main · airaria/TextPruner</a></p><h3>Deep Learning in NLP YouTube Lectures</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLdH9u0f1XKW_s-c8EcgJpn_HJz5Jj1IRf&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fplaylist%3Flist%3DPLdH9u0f1XKW_s-c8EcgJpn_HJz5Jj1IRf&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FvtYaPESzZj4%2Fhqdefault.jpg%3Fsqp%3D-oaymwEWCKgBEF5IWvKriqkDCQgBFQAAiEIYAQ%3D%3D%26rs%3DAOn4CLC5Uhz_52EHfZ9VPR4i68nkgElhrg%26days_since_epoch%3D18987&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="853" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/79e3488420f032532666fc648ad606fa/href">https://medium.com/media/79e3488420f032532666fc648ad606fa/href</a></iframe><h3>Play the Shannon Game With Language Models</h3><p>A new summarization evaluation metric called the Shannon Score is proposed. It performs the Shannon Game with a language model.</p><p>Paper: <a href="https://arxiv.org/pdf/2103.10918.pdf">https://arxiv.org/pdf/2103.10918.pdf</a></p><h3>FakeYou</h3><p><a href="https://fakeyou.com/">FakeYou Celebrity AI Voice and Video Generator</a></p><h3>attrs</h3><p><a href="https://glyph.twistedmatrix.com/2016/08/attrs.html">The One Python Library Everyone Needs</a></p><h3>Next Level</h3><h3>Jo Kristian Bergum on Twitter: &quot;Thanks to @onnxruntime you can run Transformer NLP inference in the browser🙌As a personal project, I built a React javascript app deployed with Cloudfare pages. Model and vocabulary downloaded to the browser. No network during inference. Live demo at https://t.co/Kl8MadzBaI pic.twitter.com/Ra3NmHPu5s / Twitter&quot;</h3><p>Thanks to @onnxruntime you can run Transformer NLP inference in the browser🙌As a personal project, I built a React javascript app deployed with Cloudfare pages. Model and vocabulary downloaded to the browser. No network during inference. Live demo at https://t.co/Kl8MadzBaI pic.twitter.com/Ra3NmHPu5s</p><p><strong>Demo</strong></p><p><a href="https://aiserv.cloud/">Inference with Transformer models in the Browser</a></p><h3>Papers to Read 📚</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2112.12731.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b4b665c2d2306c3e5b44754f0b293c9/href">https://medium.com/media/3b4b665c2d2306c3e5b44754f0b293c9/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2112.10508.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/f7c5ac8f4566f49391df83a00f633d77/href">https://medium.com/media/f7c5ac8f4566f49391df83a00f633d77/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2112.04426.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/01daa7dd5362bc7ea401e5caac9ff7e5/href">https://medium.com/media/01daa7dd5362bc7ea401e5caac9ff7e5/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2112.11739.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/85dc47e5584cab4a8219bd9be18569ee/href">https://medium.com/media/85dc47e5584cab4a8219bd9be18569ee/href</a></iframe><h3>Repo Cypher 👨‍💻</h3><h4>A collection of recently released repos that caught our 👁</h4><h4><a href="https://arxiv.org/pdf/2112.08652.pdf">PECOS — Predictions for Enormous and Correlated Output Spaces</a></h4><blockquote>PECOS is a machine learning framework for fast learning and inference on problems with large output spaces, such as extreme multi-label ranking (XMR) and large-scale retrieval.</blockquote><p><a href="https://github.com/amzn/pecos">GitHub - amzn/pecos: PECOS - Prediction for Enormous and Correlated Spaces</a></p><p><a href="https://www.connectedpapers.com/main/2682e65610453726cef1d23aae4aaed2e043fbb2"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2112.07637.pdf">Exploring Neural Models for Query-Focused Summarization</a></h4><blockquote>A systematic exploration of neural approaches to query summarization, considering two general classes of methods: two-stage extractive-abstractive solutions and end-to-end models.</blockquote><p><a href="https://github.com/salesforce/query-focused-sum">GitHub - salesforce/query-focused-sum</a></p><p><a href="https://www.connectedpapers.com/main/cae00a6b3d27d0d165838fa2b2f0e8c99fe42565"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2112.07198.pdf">Contrastive Pruning</a></h4><blockquote>A pruning framework which aims at maintaining both task-specific and task-agnostic knowledge during pruning.</blockquote><p><a href="https://github.com/RunxinXu/ContrastivePruning">GitHub - RunxinXu/ContrastivePruning: Source code for our AAAI&#39;22 paper 《From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression》</a></p><p><a href="https://www.connectedpapers.com/main/08c2e7812ff224db1c877b4d14730d6288d529aa"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2112.05596.pdf">Randomised Controlled Trial Abstract Result Tabulator</a></h4><blockquote>RCT-ART is an NLP pipeline built with spaCy for converting clinical trial result sentences into tables through jointly extracting intervention, outcome and outcome measure entities and their relations.</blockquote><p><a href="https://github.com/jetsunwhitton/rct-art">GitHub - jetsunwhitton/RCT-ART: RCT-ART is an NLP pipeline built with spaCy for converting clinical trial result sentences into tables through jointly extracting intervention, outcome and outcome measure entities and their relations.</a></p><p><a href="https://www.connectedpapers.com/main/60eccc1e8649fcc914f3075f68f3f173aabbf17a"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2112.07566.pdf">VALSE</a></h4><blockquote>A task-independent benchmark for vision and language models centered on linguistic phenomena.</blockquote><p><a href="https://github.com/heidelberg-nlp/valse">GitHub - Heidelberg-NLP/VALSE</a></p><p><a href="https://www.connectedpapers.com/main/816fde2349e35cd58dd3945227d2df8be8c8a3f0"><strong>Connected Papers</strong></a><strong> 📈</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/54/0*c0C7jkCYrRYDl4FL" /><figcaption><a href="https://quantumstat.com/">Quantum Stat</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c91fa16fc85" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The NLP Cypher | 12.12.21]]></title>
            <link>https://nlpcypher.medium.com/the-nlp-cypher-12-12-21-68164482e633?source=rss-7a94371c6509------2</link>
            <guid isPermaLink="false">https://medium.com/p/68164482e633</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ricky Costa]]></dc:creator>
            <pubDate>Sun, 12 Dec 2021 21:37:07 GMT</pubDate>
            <atom:updated>2021-12-12T21:37:07.955Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qsyp_8E3wGcvwne7" /><figcaption>prompt: UFO in the Sky | AI Generated</figcaption></figure><h4>NATURAL LANGUAGE PROCESSING (NLP) WEEKLY NEWSLETTER</h4><h4>Magnus</h4><h3>Is Moore’s Law Finito?</h3><p><a href="https://www.lesswrong.com/posts/aNAFrGbzXddQBMDqh/moore-s-law-ai-and-the-pace-of-progress">Moore&#39;s Law, AI, and the pace of progress - LessWrong</a></p><h3>NeurIPS Research Papers by Institution</h3><p>Here’s a collection of papers by your favorite big tech and educational institutions.</p><ul><li><a href="https://blog.ml.cmu.edu/2021/12/06/carnegie-mellon-university-at-neurips-2021/">Carnegie Mellon University at NeurIPS 2021</a></li><li><a href="https://ai.googleblog.com/2021/12/google-at-neurips-2021.html">Google at NeurIPS 2021</a></li><li><a href="https://ai.stanford.edu/blog/neurips-2021/">Stanford AI Lab Papers and Talks at NeurIPS 2021</a></li><li><a href="https://ai.facebook.com/blog/-meta-ai-research-at-neurips-2021-embodied-agents-unsupervised-speech-recognition-and-more/">Meta AI research at NeurIPS 2021: Embodied agents, unsupervised speech recognition, and more</a></li></ul><h3>A New and Blazing Fast WordPiece Tokenizer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*mk2IsCeiQQHvHt24" /></figure><p><a href="https://ai.googleblog.com/2021/12/a-fast-wordpiece-tokenization-system.html">A Fast WordPiece Tokenization System</a></p><h3>GLaM| 1.2 Trillion Param Sparse Model</h3><p><a href="https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html">More Efficient In-Context Learning with GLaM</a></p><p><em>“The Generalist Language Model (GLaM), a trillion weight model that can be trained and served efficiently (in terms of computation and energy use) thanks to </em><a href="https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/"><em>sparsity</em></a><em>, and achieves competitive performance on multiple few-shot learning tasks. GLaM’s performance compares favorably to a dense language model, GPT-3 (175B) with significantly improved learning efficiency across 29 public NLP benchmarks in seven categories, spanning language completion, open-domain question answering, and natural language inference tasks.”</em></p><h4>Glam vs. GPT-3 on NLG and NLU Tasks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*DZRBXfafyv0fEgdC" /></figure><h4>Awesome Take Away:</h4><p>This large sparse model is competitive with dense counterparts while training on much less data and consuming less energy.</p><h3>Information Extraction from Scanned Receipts: Fine-tuning LayoutLM on SROIE</h3><p>An OCR demo with LayoutLM fine-tuned for information extraction on receipts data.</p><p><a href="https://wandb.ai/wandb-data-science/layoutlm_sroie_demo/reports/Information-Extraction-from-Scanned-Receipts-Fine-tuning-LayoutLM-on-SROIE--VmlldzoxMjI5NzE2">Information Extraction from Scanned Receipts: Fine-tuning LayoutLM on SROIE</a></p><h3>AI Predictions Survey</h3><p><a href="https://www.pwc.com/us/en/tech-effect/ai-analytics/ai-predictions.html">http://www.pwc.com/us/en/tech-effect/ai-analytics/ai-predictions.html</a></p><h3>Improving GitHub Search</h3><p><a href="https://github.blog/2021-12-08-improving-github-code-search/">Improving GitHub code search | The GitHub Blog</a></p><h3>Gopher — Deepmind’s Language Model</h3><p><a href="https://deepmind.com/blog/article/language-modelling-at-scale">Language modelling at scale</a></p><h3>GauGAN2 | Photorealistic Text 2 Image</h3><p><a href="https://blogs.nvidia.com/blog/2021/11/22/gaugan2-ai-art-demo/">NVIDIA Research&#39;s GauGAN AI Art Demo Responds to Words | NVIDIA Blog</a></p><h3>Transformers From Scratch</h3><p>“I procrastinated a deep dive into transformers for a few years. Finally the discomfort of not knowing what makes them tick grew too great for me. Here is that …”</p><p><a href="https://e2eml.school/transformers.html">https://e2eml.school/transformers.html</a></p><h3>PyTorch | Julia (but not exactly like Julia)</h3><p>“When trying to predict how PyTorch would itself get disrupted, we used to joke a bit about the next version of PyTorch being written in Julia. This was not very serious: a huge factor in moving PyTorch from Lua to Python was to tap into Python’s immense ecosystem (an ecosystem that shows no signs of going away) and even today it is still hard to imagine how a new language can overcome the network effects of Python.”</p><p><a href="https://dev-discuss.pytorch.org/t/where-we-are-headed-and-why-it-looks-a-lot-like-julia-but-not-exactly-like-julia/276">Where we are headed and why it looks a lot like Julia (but not exactly like Julia)</a></p><h3>Decoding Text Generation Tutorial Top-K and Top-P</h3><p>One of the most intuitive tutorials out there.</p><p><a href="https://docs.cohere.ai/token-picking/">API Documentation | Cohere AI</a></p><h3>Punctuation Model</h3><h3>Oliver Guhr on Twitter: &quot;Our punctuation model is now available at @huggingface Model hub. It can restore punctuation marks in 4 different languages. We build it to segment text transcribed by speech recognition models. Give it a try at https://t.co/udI8bary0p pic.twitter.com/RwWvSZTsyx / Twitter&quot;</h3><p>Our punctuation model is now available at @huggingface Model hub. It can restore punctuation marks in 4 different languages. We build it to segment text transcribed by speech recognition models. Give it a try at https://t.co/udI8bary0p pic.twitter.com/RwWvSZTsyx</p><h3>Attention Neural Networks Slides</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*95vyF4_1oS8iKJlG" /></figure><p><strong>Slides</strong></p><p><a href="https://www.dropbox.com/s/rahrg6s7w4vud9f/lecture12_attention_neural_networks.pdf?dl=0">https://www.dropbox.com/s/rahrg6s7w4vud9f/lecture12_attention_neural_networks.pdf?dl=0</a></p><p><strong>Code</strong></p><p><a href="https://github.com/xbresson/CS5242_2021/tree/main/codes/labs_lecture12">CS5242_2021/codes/labs_lecture12 at main · xbresson/CS5242_2021</a></p><h3>Lemmatize spaCy</h3><p>spaCy’s new lemmatizer is super accurate and blows XLM-RoBERTa out of the water! This blog post presents inner workings, benchmarks and quick start snippets. 😎</p><p><a href="https://explosion.ai/blog/edit-tree-lemmatizer">Neural edit-tree lemmatization for spaCy · Explosion</a></p><h3>Awesome Papers 📚</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2111.10952.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/84f358314d9731eb32d8ba13434cfc1a/href">https://medium.com/media/84f358314d9731eb32d8ba13434cfc1a/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2112.01989.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/f267f36e3e182cb19948df4e4f49d449/href">https://medium.com/media/f267f36e3e182cb19948df4e4f49d449/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2112.03572.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/582fef34ec0e840ccc70b4668d00af03/href">https://medium.com/media/582fef34ec0e840ccc70b4668d00af03/href</a></iframe><h3>Repo Cypher 👨‍💻</h3><h4>A collection of recently released repos that caught our 👁</h4><h4><a href="https://arxiv.org/pdf/2112.02418.pdf">Coqui TTS</a></h4><blockquote>TTS is a library for advanced text-to-speech generation. TTS comes with <a href="https://github.com/coqui-ai/TTS/wiki/Released-Models">pretrained models</a>, tools for measuring dataset quality and already used in 20+ languages.</blockquote><p><a href="https://github.com/coqui-ai/TTS">GitHub - coqui-ai/TTS: 🐸💬 - a deep learning toolkit for Text-to-Speech, battle-tested in research and production</a></p><p><a href="https://www.connectedpapers.com/main/d989758526842b1e7ab41baac63e790a80447fc3"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2112.02505.pdf">Causal Distillation for Language Models</a></h4><blockquote>Distillation library that uses a third objective that encourages the student to imitate the causal computation process of the teacher through interchange intervention training (IIT).</blockquote><p><a href="https://github.com/frankaging/Causal-Distill">GitHub - frankaging/Causal-Distill: The Codebase for Causal Distillation for Language Models.</a></p><p><a href="https://www.connectedpapers.com/main/b04e7a697ac37776d6023541aeb3f73a2f90f90b"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2112.02721.pdf">NL-Augmenter</a></h4><blockquote>[NL-Augmenter] augments text datasets in several ways, includes: randomizing names and numbers, changing style/syntax, <a href="https://aclanthology.org/J13-3001.pdf">paraphrasing</a>, and KB-based paraphrasing.</blockquote><ul><li><a href="https://github.com/GEM-benchmark/NL-Augmenter">GitHub - GEM-benchmark/NL-Augmenter: NL-Augmenter 🦎 → 🐍 A Collaborative Repository of Natural Language Transformations</a></li><li><a href="https://colab.research.google.com/github/GEM-benchmark/NL-Augmenter/blob/main/notebooks/Write_a_sample_transformation.ipynb">Google Colaboratory</a></li></ul><p><a href="https://www.connectedpapers.com/main/9d332ad27bfce66ee725b413aa07bd93c355efdf"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2112.04008.pdf">Deepparse</a></h4><blockquote>Deepparse is a state-of-the-art library for parsing multinational street addresses using deep learning.</blockquote><p><a href="https://github.com/GRAAL-Research/deepparse">GitHub - GRAAL-Research/deepparse: Deepparse is a state-of-the-art library for parsing multinational street addresses using deep learning</a></p><p><a href="https://www.connectedpapers.com/main/84a32344585cf33ecb507072bc1b530e38874ab7"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2112.03227.pdf">CALVIN — A benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks</a></h4><blockquote>A simulated benchmark to learn long-horizon language-conditioned tasks. The aim is to make it possible to develop agents that can solve many robotic manipulation tasks over a long horizon, from onboard sensors, and specified only via human language.</blockquote><p><a href="https://github.com/mees/calvin">GitHub - mees/calvin: CALVIN - A benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks</a></p><p><a href="https://www.connectedpapers.com/main/df51fb60e8587520ea504d8a15686515101899c7"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2112.03213.pdf">Hashformers</a></h4><blockquote>Library for the hashtag segmentation task which automatically inserts missing spaces between words in a hashtag.</blockquote><ul><li><a href="https://github.com/ruanchaves/hashformers">GitHub - ruanchaves/hashformers: Hashformers is a framework for hashtag segmentation with transformers.</a></li><li><a href="https://colab.research.google.com/github/ruanchaves/hashformers/blob/master/hashformers.ipynb">Google Colaboratory</a></li></ul><p><a href="https://www.connectedpapers.com/main/1ebff999b7caf7cd07156049f2923d8f192cd81e"><strong>Connected Papers</strong></a><strong> 📈</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/54/0*c0C7jkCYrRYDl4FL" /><figcaption><a href="https://quantumstat.com/">Quantum Stat</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68164482e633" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The NLP Cypher | 11.21.21]]></title>
            <link>https://nlpcypher.medium.com/the-nlp-cypher-11-21-21-29461364076?source=rss-7a94371c6509------2</link>
            <guid isPermaLink="false">https://medium.com/p/29461364076</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Ricky Costa]]></dc:creator>
            <pubDate>Sun, 21 Nov 2021 20:23:06 GMT</pubDate>
            <atom:updated>2021-11-21T20:23:06.657Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FBzyTuYCmvSORqs1ABM%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FBzyTuYCmvSORqs1ABM%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FBzyTuYCmvSORqs1ABM%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="435" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fc97cd97fe50b8eec43da4b298a101b/href">https://medium.com/media/9fc97cd97fe50b8eec43da4b298a101b/href</a></iframe><h4>NATURAL LANGUAGE PROCESSING (NLP) WEEKLY NEWSLETTER</h4><h3>The NLP Cypher | 11.21.21</h3><h4>Inference Prime</h4><p>Hey … so have you ever deployed a state-of-the-art production level inference server? Don’t know how to do it?</p><p>Well… last week, Michael Benesty dropped a bomb when he published one of the first ever detailed blogs on how to not only deploy a production level inference API but benchmarking some of the most widely used frameworks such as FastAPI and Triton servers and runtime engines such as ONNX runtime (ORT) and TensorRT (TRT). Eventually, Michael recreated Hugging Face’s ability to reach a 1–2ms inference with miniLM &amp; a T4 GPU. 👀</p><p>🥶🥶🥶🥶🥶🥶</p><p><a href="https://towardsdatascience.com/hugging-face-transformer-inference-under-1-millisecond-latency-e1be0057a51c">Hugging Face Transformer Inference Under 1 Millisecond Latency</a></p><p><strong>Code</strong>:</p><p><a href="https://github.com/ELS-RD/triton_transformers">GitHub - ELS-RD/triton_transformers: Deploy optimized transformer based models on Nvidia Triton server</a></p><p><strong>Another Tutorial for Triton and Hugging Face Inference</strong></p><p><a href="https://medium.com/nvidia-ai/how-to-deploy-almost-any-hugging-face-model-on-nvidia-triton-inference-server-with-an-8ee7ec0e6fc4">How to deploy (almost) any Hugging face model on NVIDIA Triton Inference Server with an…</a></p><p><strong>NVIDIA’s Triton Server Update</strong></p><p><a href="https://www.yahoo.com/now/nvidia-announces-major-updates-triton-090000751.html">NVIDIA Announces Major Updates to Triton Inference Server as 25,000+ Companies Worldwide Deploy NVIDIA AI Inference</a></p><h3>PyTorch LIT (talkin’ bout Inference)</h3><p>PyTorch Lite Inference Toolkit: works with Hugging Face pipeline.</p><p>Here’s an example for text generation with GPT-J (6 Billi param model)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iDfQmHcBKFKL-tAQIjAXLg.png" /></figure><p><a href="https://github.com/AminRezaei0x443/PyTorch-LIT">GitHub - AminRezaei0x443/PyTorch-LIT: Lite Inference Toolkit (LIT) for PyTorch</a></p><h3>Model Size x 18 = Model Memory Required</h3><h3>Stas Bekman on Twitter: &quot;If you want to impress your peers ask them to throw at you a model size and you can immediately tell them how much memory it&#39;d require. The secret is to multiply by 18. e.g. t5-11b needs ~200GB plus activation memory.It&#39;s now explained in details here: https://t.co/E7sUZfeXnb / Twitter&quot;</h3><p>If you want to impress your peers ask them to throw at you a model size and you can immediately tell them how much memory it&#39;d require. The secret is to multiply by 18. e.g. t5-11b needs ~200GB plus activation memory.It&#39;s now explained in details here: https://t.co/E7sUZfeXnb</p><p><strong>Deets:</strong></p><p><a href="https://huggingface.co/transformers/master/performance.html#anatomy-of-model-s-memory">Performance and Scalability: How To Fit a Bigger Model and Train It Faster</a></p><h3>A Convenient Collection of Simple Python Code Snippets</h3><p>Some Examples:</p><p><a href="https://github.com/Python-World/python-mini-projects/tree/master/projects/Hello">1 Hello World</a></p><p>2 <a href="https://github.com/Python-World/python-mini-projects/tree/master/projects/Convert_JSON_to_CSV">JSON to CSV</a></p><p>3 <a href="https://github.com/Python-World/python-mini-projects/tree/master/projects/Random_password_generator">Random Password Generator</a></p><p>4 <a href="https://github.com/Python-World/python-mini-projects/tree/master/projects/Instagram_profile">Instagram Profile Info</a></p><p>6 <a href="https://github.com/Python-World/python-mini-projects/tree/master/projects/All_links_from_given_webpage">Fetch links from Webpage</a></p><p>7 <a href="https://github.com/Python-World/python-mini-projects/tree/master/projects/Todo_app">Todo App With Flask</a></p><p>8 <a href="https://github.com/Python-World/python-mini-projects/tree/master/projects/Image_watermark">Add Watermark on Images</a></p><p>9 <a href="https://github.com/Python-World/python-mini-projects/tree/master/projects/WishList">WishList App Using Django</a></p><p>10 <a href="https://github.com/Python-World/python-mini-projects/tree/master/projects/Split_folder_into_subfolders">Split Folders into Subfolders</a></p><p><a href="https://github.com/Python-World/python-mini-projects">GitHub - Python-World/python-mini-projects: A collection of simple python mini projects to enhance your python skills</a></p><h3>OpenAI’s API Goes Open Range</h3><p><a href="https://openai.com/blog/api-no-waitlist/">OpenAI&#39;s API Now Available with No Waitlist</a></p><h3>G5 Instances at AWS w/ A10G GPUs</h3><p><a href="https://aws.amazon.com/blogs/aws/new-ec2-instances-g5-with-nvidia-a10g-tensor-core-gpus/">New - EC2 Instances (G5) with NVIDIA A10G Tensor Core GPUs | Amazon Web Services</a></p><h3>Hop: Reading Files without Extracting Archive</h3><p><em>“25x faster than </em><em>unzip and 10x faster than </em><em>tar at reading individual files (uncompressed)”</em></p><p><a href="https://github.com/Jarred-Sumner/hop">GitHub - Jarred-Sumner/hop</a></p><h3>InfraNodus | Text Analysis Software</h3><p>Create graphs with your text data.</p><p><a href="https://towardsdatascience.com/infranodus-excellent-tool-for-textual-data-analysis-2b4839e6cd10">InfraNodus: An Excellent Tool for Textual Data Analysis</a></p><p><strong>Free Version:</strong></p><p><a href="https://github.com/noduslabs/infranodus">GitHub - noduslabs/infranodus: A Node.Js / Neo4J tool that translates words and relations into network graphs and shows you how it all connects.</a></p><h3>Distributed Training w/ PyTorch Lightning and Ray</h3><p><em>pip install ray-lightning</em></p><p><a href="https://medium.com/pytorch/getting-started-with-ray-lightning-easy-multi-node-pytorch-lightning-training-e639031aff8b">Getting Started With Ray Lightning: Easy Multi-Node PyTorch Lightning Training</a></p><h3>Papers to Read 📚</h3><p>Stanford’s Papers at EMNLP/CoNLL</p><p><a href="https://ai.stanford.edu/blog/emnlp-2021/">Stanford AI Lab Papers at EMNLP/CoNLL 2021</a></p><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2111.08609.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/d03403a1641f41d93f7fd5be965e769e/href">https://medium.com/media/d03403a1641f41d93f7fd5be965e769e/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2111.07991.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/1db31c7a548e4a0914b7377238486b25/href">https://medium.com/media/1db31c7a548e4a0914b7377238486b25/href</a></iframe><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2111.07935.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/c34b4e4b685953d3b3b37c88d5b22b90/href">https://medium.com/media/c34b4e4b685953d3b3b37c88d5b22b90/href</a></iframe><h3>Repo Cypher 👨‍💻</h3><h4>A collection of recently released repos that caught our 👁</h4><h4><a href="https://arxiv.org/pdf/2111.09543.pdf">DeBERTa V3</a></h4><blockquote>Improving DeBERTa using ELECTRA Style Pre-Training with Gradient-Disentangled Embedding Sharing. On GLUE it achieves a 91.37% average score, which is 1.37% over DeBERTa and 1.91% over ELECTRA, setting a new state-of-the-art (SOTA) among the models with a similar structure.</blockquote><p>Includes a multi-lingual variant. :)</p><p><a href="https://github.com/microsoft/DeBERTa">GitHub - microsoft/DeBERTa: The implementation of DeBERTa</a></p><p>🤗 Model Pages</p><p><a href="https://huggingface.co/models?other=deberta-v3">Models - Hugging Face</a></p><p><a href="https://www.connectedpapers.com/main/0b6c5165e5febfe8cd9b8ef8cb9fd390d062b41d"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2111.06474.pdf">AnswerSumm: A Manually-Curated Dataset and Pipeline for Answer Summarization</a></h4><blockquote>A novel dataset of 4,631 community question answering threads for answer summarization.</blockquote><p><a href="https://github.com/alex-fabbri/answersumm">GitHub - Alex-Fabbri/AnswerSumm</a></p><p><a href="https://www.connectedpapers.com/main/4cbd1b3d143c9682f9d64644e208f830a3d85d25"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2111.09296.pdf">XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale</a></h4><h3>Michael Auli on Twitter: &quot;XLS-R is our new model for cross-lingual self-supervised speech learning trained on 436K hours of public speech in 128 languages and with up to 2B params.Paper: https://t.co/o0HuU7azeuBlog: https://t.co/w9Ta8ZCLj5Code/models: https://t.co/hIUFJLPupbhttps://t.co/jjZrgoz686 pic.twitter.com/iENwHgoTuj / Twitter&quot;</h3><p>XLS-R is our new model for cross-lingual self-supervised speech learning trained on 436K hours of public speech in 128 languages and with up to 2B params.Paper: https://t.co/o0HuU7azeuBlog: https://t.co/w9Ta8ZCLj5Code/models: https://t.co/hIUFJLPupbhttps://t.co/jjZrgoz686 pic.twitter.com/iENwHgoTuj</p><ul><li><a href="https://huggingface.co/models?other=xls_r">Models - Hugging Face</a></li><li><a href="https://github.com/pytorch/fairseq">GitHub - pytorch/fairseq: Facebook AI Research Sequence-to-Sequence Toolkit written in Python.</a></li></ul><p><a href="https://www.connectedpapers.com/main/8e4e943f1b35ce8dc7a40632b56fee83573705dc"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2111.08647.pdf">DataCLUE: A Benchmark Suite for Data-centric NLP</a></h4><blockquote>A benchmark for Data Centric AI. It benchmarks how data modification can impact model’s performance. You can modify the training set and validation set, re-split the training set and validation set, or add data by non-crawler methods. The modification can be done by algorithms or programs or in combination with manual methods.</blockquote><p><a href="https://github.com/nttcslab-sp-admin/tedsummary">GitHub - nttcslab-sp-admin/TEDSummary</a></p><p><a href="https://www.connectedpapers.com/main/df02989550050622a2daad4db1604d4856fde928"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2111.09645.pdf">Dynamic-TinyBERT: Boost TinyBERT’s Inference Efficiency by Dynamic Sequence Length</a></h4><blockquote>Dynamic-TinyBERT, a TinyBERT model that utilizes sequence-length reduction and Hyperparameter Optimization for enhanced inference efficiency per any computational budget. Dynamic-TinyBERT is trained only once, performing on-par with BERT and achieving an accuracy-speedup trade-off superior to any other efficient approaches (up to 3.3x with &lt;1% loss drop).</blockquote><p><a href="https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERT">Pretrained-Language-Model/TinyBERT at master · huawei-noah/Pretrained-Language-Model</a></p><p><a href="https://www.connectedpapers.com/main/111e39b157b19ce5afb4abec031e8d0224d0ecd0"><strong>Connected Papers</strong></a><strong> 📈</strong></p><h4><a href="https://arxiv.org/pdf/2111.07218.pdf">Meta-Voice: Fast few-shot style transfer for expressive voice cloning using meta learning</a></h4><blockquote>Style transfer for voice cloning in text-to-speech (TTS).</blockquote><p><a href="https://liusongxiang.github.io/meta-voice-demo/">Meta-Voice: Fast few-shot style transfer for expressive voice cloning using meta learning</a></p><p><a href="https://www.connectedpapers.com/main/107af906c158c021d9fca24bd135fcc551c0e662"><strong>Connected Papers </strong></a><strong>📈</strong></p><h3>QЦΛПƬЦM ƧƬΛƬ on Twitter: &quot;🍎 The NLP Index just passed 7,000 NLP repositories allowing any developer to search for models, datasets and more importantly, awesome applications built from these resources.Check out the NLP Index when you have a chance!#NLProc #MachineLearning https://t.co/reGoWhetkK / Twitter&quot;</h3><p>🍎 The NLP Index just passed 7,000 NLP repositories allowing any developer to search for models, datasets and more importantly, awesome applications built from these resources.Check out the NLP Index when you have a chance!#NLProc #MachineLearning https://t.co/reGoWhetkK</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/54/0*c0C7jkCYrRYDl4FL" /><figcaption><a href="https://quantumstat.com/">Quantum Stat</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29461364076" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The NLP Cypher | 10.31.21]]></title>
            <link>https://nlpcypher.medium.com/the-nlp-cypher-10-31-21-b615ec29953b?source=rss-7a94371c6509------2</link>
            <guid isPermaLink="false">https://medium.com/p/b615ec29953b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ricky Costa]]></dc:creator>
            <pubDate>Sun, 31 Oct 2021 22:21:43 GMT</pubDate>
            <atom:updated>2021-11-01T01:32:27.750Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*78gKbqtdKP-in-AS.jpg" /><figcaption>Going West | Pollock</figcaption></figure><h4>NATURAL LANGUAGE PROCESSING (NLP) WEEKLY NEWSLETTER</h4><h4>The Localization Problem, and the Pale Blue Dot</h4><p>The Localization Problem (LP) is a glaring dark cloud hanging over the state of affairs in applied deep learning. And acknowledging this problem, I believe, will enable us to make better use of applied AI and expand our knowledge in how the business market will form.</p><p><strong>Defining LP</strong>: There is a limit to how much large centralized language models can generalize at scale given: 1) that different users inherently have varying definitions of ground-truths due to inter-dependencies to their unique real-world environment and 2) depending whether or not model performance is mission-critical. In other words, in certain conditions, in order for a model to be optimized for accuracy for a given user, the model needs to be “localized” to its user’s ground truth in their data assuming that a model can’t afford to be wrong too many times.</p><p><strong>Example</strong>:<strong> </strong>Imagine there is a kazillion parameter encoder transformer called Hal9000. This AGI model knows everything there is to know in the world when it comes to knowledge. Now Hal9000 has 2 big customers, John that works for Meta and Jane that works for CyberDyne Systems. John and Jane, don’t know each other, but are active commodity traders in their spare time who depend on Hal9000 for classifying finance-related tweets for the sentiment analysis (positive, negative, neutral) task. John and Jane are trading in real-time and a tweet is published on the wire: “gold is up 150% in after-hours trading.”</p><p><em>Some background: John is bullish on gold (owns gold call options and wants the gold price to go up in order to make money) and Jane is bearish on gold (owns gold put options and wants the price to go down in order to make money).</em></p><p>It’s time for Hal9000 to do its magic and classify this tweet so John and Jane can execute a trade. But Hal has a big problem. It can’t generalize to both John and Jane’s definition of ‘positive’ and ‘negative’. The model needs to classify this tweet as ‘positive’ for John and ‘negative’ for Jane given the same input text.</p><p>This is the LP manifesting itself in the real-world. Hal needs to localize itself to John’s ground truth and Jane’s ground truth of sentiment. Currently the way we localize models is by fine-tuning them. And fine-tuning isn’t a hinderance (as some may suggest who are obsessed with zero-shot) on AI performance but in actuality, it’s a prerequisite. All the software and hardware improvements in the world can’t make the model improve its accuracy if it is not localized to its user. However, not all use-cases encounters LP.</p><p>There is a market for non-local language models to thrive: and it’s a market where users can leverage a community accepted NLP task for which the error of the model is not mission-critical. This type of task and non-mission-critical environment isn’t concerned with LP.</p><p><em>(Think of a screenwriter using a GPT-3 for writing a screen play. The model can generate 20 screenplays, 19/20 inferences suck, but the writer likes 1 of the 20 scripts generated. A high error rate of 95%. However, the error rate is not mission-critical, and the user still finds value in the model’s output.)</em></p><p>Inversely, avoiding LP will be more prominent and required when involving mission-critical use-cases for AI models like our John and Jane example. Both John and Jane can’t tolerate a 95% error rate as they would go bust over time (in addition to the model not being local to their ground truth).</p><p>In the end, some in deep learning believe that simply scaling large language models will be able to achieve AGI. However, in mission-critical situations, generalization without localization will actually hinder performance for some users due to the localization problem, especially when there is no window for the model to be wrong…</p><p><em>like Tesla’s Autopilot.</em></p><p><strong>In the end, we are only a pale blue dot…</strong></p><h3>colin on Twitter: &quot;as voyager 1 was leaving our solar system and about 3.7 bilion miles from the sun.the people at nasa decided to turn the camera back towards our home and take a series of photos one of them being planet earth.in the photo is shows up as a pale blue dot , middle and to the right pic.twitter.com/qLzhQIwLFf / Twitter&quot;</h3><p>as voyager 1 was leaving our solar system and about 3.7 bilion miles from the sun.the people at nasa decided to turn the camera back towards our home and take a series of photos one of them being planet earth.in the photo is shows up as a pale blue dot , middle and to the right pic.twitter.com/qLzhQIwLFf</p><p><strong>If enjoy this read, don’t forget to give it a </strong>👏👏<strong> … </strong>🎃</p><h3>Google’s Pathway to More Intelligent Models</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.ted.com%2Ftalks%2Fjeff_dean_ai_isn_t_as_smart_as_you_think_but_it_could_be&amp;display_name=TED&amp;url=https%3A%2F%2Fwww.ted.com%2Ftalks%2Fjeff_dean_ai_isn_t_as_smart_as_you_think_but_it_could_be&amp;image=https%3A%2F%2Fpi.tedcdn.com%2Fr%2Ftalkstar-photos.s3.amazonaws.com%2Fuploads%2F578d7af5-2d02-4634-b97e-74d74e848a32%2FJeffDean_2021-embed.jpg%3Fh%3D316%26w%3D560&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=ted" width="560" height="316" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fefb73efd727dbe3ca513a602cccd81/href">https://medium.com/media/7fefb73efd727dbe3ca513a602cccd81/href</a></iframe><h3>Microsoft’s 2 New Repos for Speech and Seq2Seq</h3><p><strong>WavLM: Large-scale self-supervised pre-training for full stack speech processing.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BJhN_FkP_xzV7Kdr.png" /></figure><p><a href="https://github.com/microsoft/unilm/tree/master/wavlm">unilm/wavlm at master · microsoft/unilm</a></p><p><strong>S2S-FT A PyTorch package used to fine-tune pre-trained Transformers for sequence-to-sequence language generation.</strong></p><p><a href="https://github.com/microsoft/unilm/tree/master/s2s-ft">unilm/s2s-ft at master · microsoft/unilm</a></p><h3>DockerFiles Homies</h3><p>A long list of dockerfiles for assorted sorties in the digital space.</p><p><a href="https://github.com/jessfraz/dockerfiles">GitHub - jessfraz/dockerfiles: Various Dockerfiles I use on the desktop and on servers.</a></p><h3>Kerla | OS Written in Rust</h3><p>A new OS with compatibility with Linux binaries.</p><p><a href="https://github.com/nuta/kerla?utm_campaign=explore-email&amp;utm_medium=email&amp;utm_source=newsletter&amp;utm_term=daily">GitHub - nuta/kerla: A new operating system kernel with Linux binary compatibility written in Rust.</a></p><h3>Taichi | Parallel Programming</h3><p>A Python compiler that can parallelize tasks to multi-core CPUs and parallel GPUs.</p><p><a href="https://github.com/taichi-dev/taichi?utm_campaign=explore-email&amp;utm_medium=email&amp;utm_source=newsletter&amp;utm_term=daily">GitHub - taichi-dev/taichi: Parallel programming for everyone.</a></p><h3>Oldie But Goodie</h3><p>Real-time inference TensorRT.</p><p><a href="https://developer.nvidia.com/blog/real-time-nlp-with-bert-using-tensorrt-updated/">Real-Time Natural Language Processing with BERT Using NVIDIA TensorRT (Updated) | NVIDIA Developer Blog</a></p><h3>SpikeX | Knowledge Extraction</h3><p>Insanely useful library for NLP developer. Built on top of spaCy.</p><p>Pipelines:</p><ul><li>WikiPageX links Wikipedia pages to chunks in text</li><li>ClusterX picks noun chunks in a text and clusters them based on a revisiting of the <a href="https://arxiv.org/abs/1901.07410">Ball Mapper</a> algorithm, Radial Ball Mapper</li><li>AbbrX detects abbreviations and acronyms, linking them to their long form. It is based on <a href="https://github.com/allenai/scispacy/blob/master/scispacy/abbreviation.py">scispacy</a>’s one with improvements</li><li>LabelX takes labelings of pattern matching expressions and catches them in a text, solving overlappings, abbreviations and acronyms</li><li>PhraseX creates a Doc&#39;s underscore extension based on a custom attribute name and phrase patterns. Examples are NounPhraseX and VerbPhraseX, which extract noun phrases and verb phrases, respectively</li><li>SentX detects sentences in a text, based on <a href="https://github.com/dgillick/splitta">Splitta</a> with refinements</li></ul><p><a href="https://github.com/erre-quadro/spikex">GitHub - erre-quadro/spikex: SpikeX - SpaCy Pipes for Knowledge Extraction</a></p><h3>The Read 📚</h3><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/2110.13041.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/1aa1b5b94ff52801c8889f3217b4549e/href">https://medium.com/media/1aa1b5b94ff52801c8889f3217b4549e/href</a></iframe><h3>Sentence Splitting and NLP… Comparing Libraries w/ Colab</h3><h3>Andres Segura-Tinoco on Twitter: &quot;Let me show you the result of splitting the same phrase into sentences using the following 4 approaches:▪️ Naive (using the split function)▪️ With the NLTK library▪️ With the spaCy library▪️ With the stanza library🧵The results are quite different.1/8 https://t.co/h5vrsIWtKy / Twitter&quot;</h3><p>Let me show you the result of splitting the same phrase into sentences using the following 4 approaches:▪️ Naive (using the split function)▪️ With the NLTK library▪️ With the spaCy library▪️ With the stanza library🧵The results are quite different.1/8 https://t.co/h5vrsIWtKy</p><h4>Colab of the Week 🔥</h4><p><a href="https://colab.research.google.com/drive/1RI6Z56W2GHvfyLQwRTROqn2BvYXKislB?usp=sharing">Google Colaboratory</a></p><h3>Model Convergence in BigScience</h3><h3>Stas Bekman on Twitter: &quot;At @BigscienceW 104B GPT training we finally had a BREAKTHROUGH and the training doesn&#39;t diverge!The key change of Expirement 11 was to change --init-method-std 0.006 from 0.02 - Thank you, BS Team!Details are in: https://t.co/CvMUGVe0DNTB is here: https://t.co/dhziAybTsZ pic.twitter.com/ztrsiYnWJe / Twitter&quot;</h3><p>At @BigscienceW 104B GPT training we finally had a BREAKTHROUGH and the training doesn&#39;t diverge!The key change of Expirement 11 was to change --init-method-std 0.006 from 0.02 - Thank you, BS Team!Details are in: https://t.co/CvMUGVe0DNTB is here: https://t.co/dhziAybTsZ pic.twitter.com/ztrsiYnWJe</p><h3>Better Way for Evaluating QA Systems</h3><p>In other words, F1 sucks bruh.</p><p><a href="https://www.deepset.ai/blog/semantic-answer-similarity-to-evaluate-qa">Semantic Answer Similarity: Smarter Metric to Evaluate Question Answering Systems</a></p><p>🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻🐱‍💻</p><p><a href="https://cooltechzone.com/research/how-hackers-scanning-whole-internet">How are hackers scanning the whole Internet in just a few minutes?</a></p><h3>GoEmotions Dataset</h3><p>58k <a href="https://en.wikipedia.org/wiki/Reddit">Reddit</a> comments extracted from popular English-language subreddits and labeled with <a href="https://github.com/google-research/google-research/blob/master/goemotions/data/emotions.txt">27 emotion categories</a> .</p><p><a href="https://ai.googleblog.com/2021/10/goemotions-dataset-for-fine-grained.html?utm_source=feedburner&amp;utm_medium=feed&amp;utm_campaign=Feed%3A+blogspot%2FgJZg+%28Google+AI+Blog%29">GoEmotions: A Dataset for Fine-Grained Emotion Classification</a></p><h3>Software Updates</h3><h3>PyTorch on Twitter: &quot;ICYMI: PyTorch 1.10 was released last Thursday. Here are some highlights of the release. Stay tuned for tweet threads in the next couple weeks delving deeper into these cool new features! 1/8 / Twitter&quot;</h3><p>ICYMI: PyTorch 1.10 was released last Thursday. Here are some highlights of the release. Stay tuned for tweet threads in the next couple weeks delving deeper into these cool new features! 1/8</p><h3>Repo Cypher 👨‍💻</h3><h4>A collection of recently released repos that caught our 👁</h4><h4><a href="https://arxiv.org/pdf/2110.13577.pdf">Open Rule Induction</a></h4><blockquote>Aims to induce open rules utilizing the knowledge in language models.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zyYc4aP3aPbL4Fm9.png" /></figure><p><a href="https://github.com/chenxran/orion">GitHub - chenxran/Orion: This repository is the official implementation of Open Rule Induction. This paper has been accepted to NeurIPS 2021.</a></p><h4><a href="https://arxiv.org/pdf/2110.13711.pdf">HourglassLM</a></h4><blockquote>Hourglass improves upon the Transformer baseline given the same amount of computation and can yield the same results as Transformers more efficiently.</blockquote><p><a href="https://github.com/google/trax/blob/master/trax/models/research/hourglass.py">trax/hourglass.py at master · google/trax</a></p><h4><a href="https://arxiv.org/pdf/2110.11403.pdf">SCENIC: JAX Library for Computer Vision Research and Beyond</a></h4><blockquote>A codebase with a focus on research around attention-based models for computer vision. Scenic has been successfully used to develop classification, segmentation, and detection models for multiple modalities including images, video, audio, and multimodal combinations of them.</blockquote><p><a href="https://github.com/google-research/scenic">GitHub - google-research/scenic: Scenic: A Jax Library for Computer Vision Research and Beyond</a></p><h4><a href="https://arxiv.org/pdf/2110.12949.pdf">Persona Authentication through Generative Dialogue</a></h4><blockquote>An open-domain conversational agent capable of decoding <em>personalized</em> and <em>controlled</em> responses based on user input. It is built on the pretrained <a href="https://github.com/microsoft/DialoGPT">DialoGPT-medium</a> model, following the <a href="https://github.com/openai/gpt-2">GPT-2</a> architecture. PersonaGPT is fine-tuned on the <a href="https://arxiv.org/pdf/1801.07243">Persona-Chat</a> dataset.</blockquote><p><a href="https://github.com/illidanlab/personaGPT">GitHub - illidanlab/personaGPT: Implementation of PersonaGPT Dialog Model</a></p><h4><a href="https://arxiv.org/pdf/2110.13214.pdf">IconQA</a></h4><blockquote>An abstract visual question answering dataset that highlights the importance of abstract diagram understanding and comprehensive cognitive reasoning in real-world problems.</blockquote><blockquote>There are three different sub-tasks in IconQA:</blockquote><ul><li><em>57,672 image choice MC questions</em></li><li><em>31,578 text choice MC questions</em></li><li><em>18,189 fill-in-the-blank questions</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dlDCv2Pqtn2AiU3G.png" /></figure><p><a href="https://github.com/lupantech/iconqa">GitHub - lupantech/IconQA: A new benchmark for Icon Question Answering (IconQA) and a large-scale icon dataset Icon645.</a></p><h4><a href="https://arxiv.org/pdf/2110.14168.pdf">Grade School Math Dataset</a></h4><blockquote>Dataset consists of 8.5K high quality grade school math problems created by human problem writers. Authors segmented these into 7.5K training problems and 1K test problems. These problems take between 2 and 8 steps to solve, and solutions primarily involve performing a sequence of elementary calculations using basic arithmetic operations (+ — / *) to reach the final answer.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FRM9uYwm6mFW9JTX.png" /></figure><p><a href="https://github.com/openai/grade-school-math">GitHub - openai/grade-school-math</a></p><h4><a href="https://arxiv.org/pdf/2110.11624.pdf">SCICAP: Scientific Figures Dataset</a></h4><blockquote>A large-scale figure caption dataset based on Computer Science arXiv papers published between 2010-2020. SCICAP contains 410k figures that focused on one of the dominent figure type — graphplot, extracted from over 290,000 papers.</blockquote><p><a href="https://github.com/tingyaohsu/scicap">GitHub - tingyaohsu/SciCap: SciCap Dataset</a></p><h3>… … … and …🎃 Happy Halloween! 🎃</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/54/0*vgf45g9haG4f6VcH" /><figcaption><a href="https://quantumstat.com/">Quantum Stat</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b615ec29953b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>