<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Febi Agil Ifdillah on Medium]]></title>
        <description><![CDATA[Stories by Febi Agil Ifdillah on Medium]]></description>
        <link>https://medium.com/@febiagil?source=rss-e84f0da9397e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TE7tCNEX3PYdQU4kH8MakA.jpeg</url>
            <title>Stories by Febi Agil Ifdillah on Medium</title>
            <link>https://medium.com/@febiagil?source=rss-e84f0da9397e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 16:18:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@febiagil/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI is Still in Day One, Africa’s Surge, Hope for a Superconducting Future, Safest & Cleanest…]]></title>
            <link>https://medium.com/@febiagil/ai-is-still-in-day-one-africas-surge-hope-for-a-superconducting-future-safest-cleanest-b59c11e92053?source=rss-e84f0da9397e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b59c11e92053</guid>
            <category><![CDATA[weekly]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Febi Agil Ifdillah]]></dc:creator>
            <pubDate>Sat, 13 Jan 2024 14:46:37 GMT</pubDate>
            <atom:updated>2024-01-13T15:00:25.641Z</atom:updated>
            <content:encoded><![CDATA[<h3>AI is Still in Day One, Africa’s Surge, Hope for a Superconducting Future, Safest &amp; Cleanest Energy, Humans’ Culture of Progress #7</h3><h4>and other interesting things I read at the beginning of 2024.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m6tX-W_fNuKFZrlQ.png" /><figcaption>DALL-E</figcaption></figure><blockquote>I recently started a newsletter, Superficial Intelligence, that takes 5 minutes to read. The goal is to keep you up to date with machine learning projects, research papers, emerging technologies, and concepts. This article is first published at <a href="https://open.substack.com/pub/superficiality/p/ai-is-still-in-day-one-africas-surge?r=7buma&amp;utm_campaign=post&amp;utm_medium=web&amp;showWelcome=true"><strong>superficiality.substack.com:</strong></a></blockquote><p><a href="https://open.substack.com/pub/superficiality/p/ai-is-still-in-day-one-africas-surge?r=7buma&amp;utm_campaign=post&amp;utm_medium=web">AI is Still in Day One, Africa&#39;s Surge, Hope for a Superconducting Future, Safest &amp; Cleanest Energy, Humans&#39; Culture of Progress #7</a></p><p>Finally got the chance to write things down after a long break.</p><p>Truth be told, I don’t expect much change in 2024. As optimistic as I am, for the most part, I’d prepare for a rinse-and-repeat of 2023: the same headlines blaring on repeat and the same anxieties buzzing like persistent mosquitoes, but hopefully with the same (accelerated) growth. I thought I was accustomed to not being surprised, but most of the times I failed — I mean, one pandemic and two big wars? Give me a break.</p><p>Feels like I am a hummingbird trapped in a predictable sky, flitting between hope and realism. I’m approaching this year like a seasoned amusement park visitor: expect the lines, embrace the repeats, but keep an eye out for the occasional haunted house surprise or malfunctioning robot.</p><p>Yet the rebel spirit in me refused to stay put. 2024 might be a well-worn record, but I am no passive listener. I expect to spin it with a new needle, hoping for a hidden track, an unexpected melody. Better yet, a chance to bloom, a chance to twirl, a chance to be anything but ordinary.</p><p>Alright. Enough of this nonsense, though.</p><p>Let’s get to the more interesting parts. And what better place to start than with the technology that’s both disrupting and defining our world? AI, my friends, is still in its day one.</p><h3>1. AI is still in day one</h3><p>This toddler taking its first wobbly steps toward reshaping our reality. The adoption is surging ahead of the previous life-changing technology like the internet and smartphones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*05OusPyp0RIYGs8h.png" /><figcaption>Source <a href="https://www.coatue.com/blog/perspective/ai-the-coming-revolution-2023">Coatue</a></figcaption></figure><p>Most likely the trend will continue, pushing the growth beyond its current <a href="https://www.accel.com/euroscape">additional $2.4 trillion</a> in terms of equities of big companies like Microsoft, Alphabet, Apple, Tesla, Amazon, Meta, and of the biggest gainer, NVIDIA, which have added more than $1 trillion in value alone.</p><p>While the impact of the technology continues to deeply woven into our everyday lives, this doesn’t mean the deflation won’t happen. With a whopping <a href="https://news.crunchbase.com/ai-robotics/us-startup-funding-doubled-openai-anthropic-2023/">25% of US venture funding</a> pouring into AI startups, and <a href="https://www.cnbc.com/2023/10/17/amid-ai-buzz-big-us-tech-giants-add-2point5-trillion-in-market-cap.html#:~:text=The%20number%20of%20unicorn%20companies,States%2C%20it%20was%2080%25">4 in 5 American unicorns are now AI-related</a>, concerns of overvaluation linger. Take a look at OpenAI’s and Anthropic’s valuation: 180x and 200x sales, respectively. Which stand in stark contrast to more established players like Uber (3x).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fh0BUsHNsKTtZpFK.png" /><figcaption>Source: <a href="https://www.accel.com/euroscape">Accel</a></figcaption></figure><p>Speaking of surging, every peak has its valley. Let’s talk about how we rise, and the potential fall of human civilization.</p><h3>2. The humans progress: a story of how we rise and collectively decline</h3><p>What sparked the Industrial Revolution? apparently, it wasn’t just machines and factories.</p><p><a href="https://docs.iza.org/dp16674.pdf">A recent study</a> by four economists shed light on the theory that <em>cultural shifts</em> paved the way towards the Industrial Revolution. After all, it’s the progress-oriented view of the world, championed by the likes of Francis Bacon, Isaac Newton, and others, that move the needle. Putting science and experimentation as the center to make life better.</p><p>To explore this possibility, the researchers use language as a proxy to analyze cultural differences across societies. Like fingerprints in ink, language holds a wealth of cultural insights, such as gender norms, religious norms, including attitudes towards risk and much more. In the paper, the researchers analyzed the contents of 173,031 books printed in England between 1500 and 1900, tracking how frequently terms related to progress changed over time. One of the most interesting findings here is that not only do prosperous nations chatter more about progress, but cultural attitudes actually shift towards optimism before economies take off.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*wlXMAxc7j3Pq7jQn.jpeg" /><figcaption>Source:<a href="https://www.ft.com/content/e577411e-3bf2-4fb4-872a-8b7d5e9139d3"> Financial Times</a></figcaption></figure><p>Extending the work into the current decade, there’s a finding that suggests a concerning trend: western culture has been subtly abandoning its historical embrace of progress, innovation, and optimism. By analyzing texts since the 1960s, we can see that there’s a <a href="https://www.ft.com/content/e577411e-3bf2-4fb4-872a-8b7d5e9139d3">25% decline </a>in terms associated with progress, and more about anxiety, and risk aversion. One could say that calibrating priorities from perpetual advancement to caution is a good thing, but too much of it can jeopardize our progress e.g Memes. Take that, EU!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/0*t1VgCnt_3ecJep0Y.jpeg" /><figcaption>Source: <a href="https://europeancommission.medium.com/https-medium-com-europeancommission-no-no-no-we-are-not-banning-memes-copyright-proposal-abf4d21f65d2">European Commission</a></figcaption></figure><h3>3. Hope for a Superconducting Future</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WxOxoPJa6EIf885M.png" /><figcaption>Source: <a href="https://arxiv.org/abs/2312.14236">Edge-sharing quasi-one-dimensional cuprate gragments in optimally substituted Cu/Pb apatite</a></figcaption></figure><p>Since we are talking about hope and progress. I want to touch a bit on the hope for Superconductivity that will transform the way we generate, transport and consume electric energy. Imagine a world where electricity hums through power lines without a whisper of energy loss, where trains levitate on magnetic tracks and paints a picture of frictionless travel, where computers run 1000x faster than currently, and battery life increased significantly.</p><p>In superconductors, something special happens. electrons pair up and act like they’re dancing together, moving through the material with perfect coordination. A phenomenon once confined to the realm of theoretical, now shimmering as a beacon of transformative possibility — two claims <a href="https://arxiv.org/abs/2307.12008">[one]</a><a href="https://arxiv.org/abs/2307.12037">[two]</a> in the past 6 months alone (that I know of) — at least in its earliest shape: <em>floating rocks</em>.</p><p>Most superconductors, however, need to be chilled to very low temperature, near absolute zero. Room temperature superconductivity (RTSC) has long been the holy grail, a seemingly impossible dream because finding materials that operate at that practical temperatures has been a scientific hurdle. Fortunately, with each step forward, the dream of a superconducting future draws closer — <a href="https://superficiality.substack.com/i/139171200/ai-unlocks-the-door-to-new-material-discoveries-a-deepmind-breakthrough">AI-enabled material exploration tools like GNoME</a> will come in handy.</p><p>Recent advancements, including <a href="https://arxiv.org/abs/2307.12008">CuxPb10−x(PO4)6O “LK-99</a>”, that was claimed to be the first room-temperature at ambient pressure superconductor sparked an electric storm of excitement and skepticism. It has undergone a roller coaster ride. Not too long ago, following failures in replication attempts (sulfide impurities, ferromagnetism, and <a href="https://arxiv.org/pdf/2308.01135.pdf">incompatible properties</a>) and <a href="https://pubs.acs.org/doi/10.1021/acsomega.3c06096">strong evidence that against the early result</a>, we must forgo the possible breakthrough. But not to be the one who dismissed the possibilities, the team at <a href="https://arxiv.org/abs/2312.14236">Lawrence Berkeley National Lab have employed supercomputers</a> to explore ways to make LK99 functional.</p><p>In addition, <a href="https://arxiv.org/pdf/2401.00999.pdf">a recent joint paper</a> by two independent researchers from several universities in China and Japan has reignited the possibilities of LK-99 superconductivity at room temperature, although with slightly different formulation than the original LK-99 formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aRLxh9WD-TWTdbZJ.png" /></figure><h3>Charts that caught my attention</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*22gI9so2cJzzu80i.jpeg" /><figcaption>By 2050, Africa’s population would account for a quarter of the world. Where the slowing birth rates are prevalent in other parts of the world, Africa would remain as the largest source of growth. With this shift in population, some of these states possibly emerge as new world powers. Source: <a href="https://www.imf.org/en/Publications/fandd/issues/2023/09/PT-african-century">IMF</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pc7Y5DxVQfWnjH8O.png" /><figcaption>Our brains are bigger. Heights and brain volumes are both up, but interestingly, brains are growing relatively more than height. Source:<a href="https://assets.researchsquare.com/files/rs-2524684/v1/1d836603-4b23-475d-8ee8-a0dca8b1ef74.pdf?c=1675180629"> Secular Trends in Head Size and Cerebral Volumes In the Framingham Heart Study for Birth Years 1902–1985</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SIS4XMGg2DTZBXVQ.png" /><figcaption>Source <a href="https://ourworldindata.org/cheap-renewables-growth">Our World in Data</a></figcaption></figure><h3>Other interesting reads</h3><h4>AI Papers &amp; Articles</h4><ol><li><a href="https://huggingface.co/papers/2401.01952">Google Instruct Imagen</a> [Huggingface]</li><li><a href="https://arxiv.org/abs/2401.02412">LLM augmented LLMs</a> [Arxiv]<a href="https://twitter.com/_akhaliq/status/1743110782022291794">[Twitter]</a></li><li>Improving Text Embeddings with Large Language Models [<a href="https://arxiv.org/abs/2401.00368">Arxiv</a>][<a href="https://twitter.com/osanseviero/status/1742555240363074000?t=xE5AEVbIhnuOwv9N8w9T-g&amp;s=19">Twitter</a>]</li><li><a href="https://arxiv.org/abs/2401.01335">Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models</a> [Arxiv]</li><li><a href="https://twitter.com/rohanpaul_ai/status/1742441677548044514?t=O-FX1TzMiyo-trdnTSnsbA&amp;s=19">Mistral medium mystery</a> [Twitter]</li><li><a href="https://osanseviero.github.io/hackerllama/blog/posts/random_transformer/">Random transformers </a>[Blog]</li><li><a href="https://contextual.ai/better-cheaper-faster-llm-alignment-with-kto/">Kahneman-Tversky Optimization </a>[Blog]</li></ol><h4>Other subjects</h4><ol><li><a href="https://medium.com/starts-with-a-bang/how-much-energy-does-the-sun-produce-6efc2b7be720">How much energy does the Sun produce?</a> [Ethan Siegel]</li><li><a href="https://www.nature.com/articles/d41586-023-03992-6">A 2D route to 3D computer chip</a> [Nature]</li><li><a href="https://ourworldindata.org/cheap-renewables-growth">Why did renewables become so cheap so fast?</a> [Our World in Data]</li><li><a href="https://www.nature.com/articles/s41586-023-06825-8">Satellite mapping reveals extensive industrial activity at sea</a> [Nature]</li><li><a href="https://epubs.siam.org/doi/abs/10.1137/S0036144504446436">Dynamical bias in the coin toss</a> — from the moment a coin is launched into the air, its entire trajectory — including whether it lands on heads or tails — can be calculated by the laws of mechanics. The researchers determined that airborne coins don’t turn around their symmetrical axis; instead they tend to wobble off-center, which causes them to spend a little more time aloft with their initial “up” side on top. <a href="https://arxiv.org/abs/2310.04153">New paper supporting this</a>.</li></ol><p>I think I’m going to stop here.</p><p>Here’s to 2024! The future looks so bright ☀️😎</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b59c11e92053" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some interesting things to read for the end of the year #6]]></title>
            <link>https://medium.com/@febiagil/some-interesting-things-to-read-for-the-end-of-the-year-6-4c6ff7af0266?source=rss-e84f0da9397e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c6ff7af0266</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[weekly]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Febi Agil Ifdillah]]></dc:creator>
            <pubDate>Sat, 06 Jan 2024 08:38:46 GMT</pubDate>
            <atom:updated>2024-01-06T08:38:46.889Z</atom:updated>
            <content:encoded><![CDATA[<h4>Apple GPT, Transformers against the world (RWKV, Mamba, StripedHyena), Multi-step reasoning LLM agents, all the way to Nuclear Fusion and Cancer-fighting CAR-T cells.</h4><p>Hi, it’s Febi here with what will likely be our final weekly roundup for the year. I wish you all a joyful and prosperous new year ahead!</p><p><strong>In today’s edition:</strong></p><ol><li> 🦦 Apple enters the LLMs race — new Multimodal LLM is out!</li><li>🐍 🤖 Transformers against the world (Mamba, StripedHyena, RWKV)</li></ol><h3>1. Apple releases Ferret — Refer and Ground Anywhere at Any Granularity</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-oYbuaqEH8lLIMg1.png" /><figcaption>DALL-E</figcaption></figure><p>We are about to close 2023, but Apple seemed to want to finish the year with a decisive statement “We are catching up!”, silencing doubters (especially myself) with not one, but two interesting papers:</p><p><strong>1)LLM in Flash:</strong> <a href="https://arxiv.org/abs/2312.11514">https://arxiv.org/abs/2312.11514</a> — this particular paper introduces a framework to overcome current LLMs shortcomings: they are large and require a lot of memory. Making it difficult to run them on devices with limited DRAM e.g a 7 billion parameter model requires over 14GB of memory just to load the parameters in half-precision floating point format. <strong>Proposed solution?</strong> → Store the model parameters on flash memory. By exploiting sparsity in the FeedForward Network (FFN) layers and selectively loading only necessary parameters from flash memory — plus employing additional two techniques called (a) <strong>“windowing”</strong> which is loading parameters for only the past few tokens and reusing activations from recently computed tokens, and (b) <strong>“row-column bundling”</strong> which roughly translates into “reading larger chunks to increase throughput”. This solution resulting in increase in inference speed by 20–25x compared to naive loading approaches in CPU and GPU as well as demonstrating the ability to run LLMs up to twice of the size of available DRAM, paving the way for wider LLM deployment on resource-constrained devices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bVCg-118dYSde2rp.png" /><figcaption>Source: Paper (<a href="https://arxiv.org/pdf/2312.11514.pdf">https://arxiv.org/pdf/2312.11514.pdf</a>)</figcaption></figure><p><strong>2) Ferret:</strong> <a href="https://arxiv.org/abs/2310.07704">https://arxiv.org/abs/2310.07704</a> — this paper introduces Ferret, a Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions — meaning it can point out details as specific as “the cat under the table” — much like giving LLMs a magnifying glass and a pointer. Ferret was built on top of Llava and Vicuna. It appears to be Apple’s answer to Google’s Gemini, potentially setting the stage for a fierce rivalry in the realm of multimodal LLMs. Here’s a <strong>TL;DR</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bBsVE2hn7iixrHv_.png" /><figcaption>Source: <a href="https://arxiv.org/abs/2310.07704">https://arxiv.org/abs/2310.07704</a></figcaption></figure><h4>Problems</h4><ol><li>Vision-language models need better spatial understanding (knowing where things are in an image)</li><li>Two important skills for this are referring (describing specific regions) and grounding (finding regions based on descriptions)</li><li>Current models usually learn these skills separately, but humans learn and use them together seamlessly</li></ol><h4>Objectives</h4><ol><li>Develop a model that can unify referring and grounding in a single framework, much like humans do.</li><li>The model should be robust and practical by:<strong> a)</strong> Representing different types of regions (points, boxes, scribbles, freeform shapes).<strong> b)</strong> Working with open-vocabulary instructions — not just pre-defined words.<strong> c)</strong> Following complex instructions involving both text and regions.</li></ol><h4>Key Innovations</h4><ol><li><strong>Unified framework:</strong> Combines referring and grounding using shared knowledge.</li><li><strong>Hybrid region representation:</strong> Uses both <strong>discrete coordinates</strong> (like “top-left: (10, 5)”) and <strong>continuous visual features</strong> (from the image) to represent any shape.</li><li><strong>Spatial-aware visual sampler:</strong> Extracts features from regions of any shape, even sparse ones like scribbles.</li><li><strong>Free-form input:</strong> Can handle instructions containing both referred regions and text descriptions.</li><li><strong>Simultaneous generation:</strong> Outputs both text and coordinates for grounded objects.</li></ol><p>All in all, I’m glad Apple seems to be further ahead in ML than I thought.</p><h3>2. Transformers against the world — towards cheaper, and scalable architectures</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KAlsDi-MhzFSO3XN.png" /><figcaption>DALL-E</figcaption></figure><p>2023 is without a doubt, the year of <a href="https://arxiv.org/abs/1706.03762">Transformers</a>. However, as we advance towards 2024, a thrilling shift with the emergence of other new architectures such as <a href="https://arxiv.org/abs/2305.13048">RWKV (Receptance Weighted Key Value)</a>, <a href="https://www.together.ai/blog/stripedhyena-7b">StripedHyena</a>, and <a href="https://arxiv.org/abs/2312.00752">Mamba</a> have emerged, each offering new scaling laws for training, different inference costs, and many other things.</p><p>There are some reasons why attention, the bedrock of Transformers became the <em>de facto</em> architecture today, but the capabilities of these newcomers raise a question: is it time to take other architectures more seriously?</p><p>Imagine training models on massive datasets without sweating over memory constraints — that’s the promise from models the likes of RWKV which combines the strengths of RNNs and Transformers. So does Mamba, <a href="https://superficiality.substack.com/i/139385287/mixtrals-of-experts-and-mambas-state-space-models">as we’ve talked in the previous roundup</a>, which unlocks blazing-fast inference speeds (5x higher throughput than Transformers) as well as linear scaling in sequence length (up to million-length sequences) and StripedHyena was assembled from many advancements including <a href="https://arxiv.org/abs/2111.00396">S4</a> and many lesson learned from the previous architectures: <a href="https://www.together.ai/blog/hungry-hungry-hippos-towards-language-modeling-with-state-space-models">H3</a>, <a href="https://hazyresearch.stanford.edu/blog/2023-03-07-hyena">Hyena</a>, <a href="https://hazyresearch.stanford.edu/blog/2023-06-29-hyena-dna">HyenaDNA</a>, and <a href="https://www.together.ai/blog/monarch-mixer">Monarch Mixer</a>.</p><p>From this <a href="https://github.com/srush/do-we-need-attention/blob/main/DoWeNeedAttention.pdf">slide by Sasha Rush</a>, I think we can see where it’s headed: an RNN Revival.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-ltcaExIaTXIl4CG.png" /><figcaption>Source: RWKV paper (<a href="https://arxiv.org/abs/2305.13048">https://arxiv.org/abs/2305.13048)</a></figcaption></figure><p>Now, pretty much anyone in the space knows that attention is likely to be replaced — in fact some people are racing towards replacing it.. what we don’t know yet is by what (RNN?), and when (4 years?). Recent developments have shed a light on some of the strong contenders like the above but it will likely take more time than we expected. (when is time became an issue in this space? cause everyone seems to be moving very fast) Moreover, now that we’ve seen people pushing the boundaries by not only exploiting the architecture in the software realm, but also in the optimizing the hardware parts, the future architecture will not only need innovations in software but also investments in a new-kind infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PDxzi5TLarNI65yW.png" /><figcaption>Source: <a href="https://github.com/srush/do-we-need-attention/blob/main/DoWeNeedAttention.pdf">Sasha Rush</a></figcaption></figure><p>Please also take your time to read a primer in this area by none other than <a href="https://www.interconnects.ai/p/llms-beyond-attention">Nathan Lambert.</a></p><h3>Charts that caught my attention</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/0*sw4oxcbcgYP52NKa.png" /><figcaption>Source: EveryPixel <a href="https://journal.everypixel.com/2023-the-year-of-ai">(2023 the year of AI)</a></figcaption></figure><p>Source <a href="https://www.mckinsey.com/featured-insights/2023-year-in-review/2023-the-year-in-charts?cid=soc-web">McKinsey &amp; Company</a></p><h3>Other reads</h3><h3>AI Papers</h3><ol><li><a href="https://arxiv.org/abs/2312.13558"><strong>The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction </strong></a><strong>— </strong>Neural networks can often have well over 90% of their weights removed without any significant degradation in performance. In this paper, there’s a new interesting finding: careful pruning done at specific layers of Transformer models can even produce significant boosts in performance on some tasks. Even better, this discovery appears to not be limited to natural language, with performance gains also found in reinforcement learning.</li><li><a href="https://arxiv.org/abs/2312.10003"><strong>ReSt meet React</strong></a> — Defining a ReAct-style LLM agent with the ability to reason and act upon external knowledge, and refine the agent through a ReST-like method. After just two iterations, achieves comparable performance to other models but with fewer parameters.</li><li><a href="https://huggingface.co/papers/2312.14878"><strong>Pangu-Agent</strong></a> — Inspired by the modularity of the human brain, researchers from Huawei Noah’s Ark Lab, University College London, and University of Oxfords developed a framework that integrates Large Language Models (LLMs) to address reasoning and decision problem towards building generalist agents.</li></ol><h3>Other interesting stuff</h3><ol><li><a href="https://mck.co/3GFxSgg"><strong>2023: A year in charts</strong></a> (McKinsey)</li><li><a href="https://twitter.com/AravSrinivas/status/1737279359709487498"><strong>Major LLM services visits reach about 2bn combined</strong></a><strong> </strong>(Tweet)</li><li><a href="https://www.independent.co.uk/tech/nuclear-fusion-breakthrough-clean-energy-b2468625.html"><strong>Nuclear Fusion Breakthrough</strong></a> (Independent)</li><li><a href="https://blog.samaltman.com/what-i-wish-someone-had-told-me"><strong>What I wish someone had told me</strong></a> (Sam Altman)</li><li><a href="https://twitter.com/Noahpinion/status/1736481030755008730"><strong>American brain drains China’s talent</strong></a><strong> </strong>(Noah Smith)</li><li><a href="https://www.pnas.org/doi/10.1073/pnas.2319169120"><strong>The simplest explanation isn’t always the bes</strong></a><strong>t. </strong>Dimensionality reduction can see structures that do not exist and miss structures that exist.</li><li><a href="https://www.nature.com/articles/d41586-023-03969-5"><strong>Cancer-fighting CAR-T cells could be made inside body with viral injection</strong></a><strong> </strong>(Nature)</li><li><a href="https://www.theverge.com/2023/12/15/24003151/bytedance-china-openai-microsoft-competitor-llm">ByteDance secretly used OpenAI’s API to build their own LLM</a> (The Verge)</li><li><a href="https://www.theverge.com/2023/12/27/24016939/samsung-2024-ai-family-hub-smart-fridge-features"><strong>Samsung’s new AI-enabled smart fridge can design recipes based on your dietary needs</strong></a> (The Verge)</li></ol><p><em>Originally published at </em><a href="https://superficiality.substack.com/p/some-interesting-things-to-read-for"><em>https://superficiality.substack.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c6ff7af0266" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some interesting things to start the week #5]]></title>
            <link>https://medium.com/@febiagil/some-interesting-things-to-start-the-week-5-8b917572d6ad?source=rss-e84f0da9397e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b917572d6ad</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Febi Agil Ifdillah]]></dc:creator>
            <pubDate>Tue, 19 Dec 2023 16:47:31 GMT</pubDate>
            <atom:updated>2023-12-19T16:48:42.551Z</atom:updated>
            <content:encoded><![CDATA[<h4>Small Language Models, Data exhaustion, Prompt engineering, LLMs hallucinations, how they can help make new scientific discoveries and whether their emergent capabilities are just a mirage.</h4><blockquote>This post was originally published on <a href="https://open.substack.com/pub/superficiality/p/some-interesting-things-to-start?r=7buma&amp;utm_campaign=post&amp;utm_medium=web&amp;showWelcome=true"><em>superficiality.substack.com</em></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gcMEujcspCdqDVQ9.png" /><figcaption>DALL-E</figcaption></figure><p>I thought December was going to be a slow down month for everybody, but boy.. this post is gonna be so packed. People in the space just can’t take it slow.</p><p>Thanks for reading Superficial Intelligence! Subscribe for free to receive new posts and support my work.</p><h3>In today’s edition:</h3><ol><li>💾 Small Language Models, and the world is running out of data</li><li>👑 The new king of MMLU benchmark</li><li>🤖 Can we prompt our way towards AGI?</li><li>🐞 LLM hallucination; a bug or a feature?</li><li>⏳ Could the emergent properties be just a mirage?</li></ol><h3>1. Let’s start small, and why is it the next big thing instead</h3><p>Compact, yet powerful models are always welcome. In fact, very much needed. The ability to train compact models with cutting-edge capabilities would democratize advanced AI, enabling a broader range of individuals and organizations to study and deploy them, instead of being an exclusive domain of a few with vast computational resources — overall better for the pocket and environment alike.</p><p>Previously we talked about <a href="https://superficiality.substack.com/i/139385287/mixtrals-of-experts-and-mambas-state-space-models">Mixtral — Mixture of Experts and Mamba — Selective State Space Models</a>, where they punching above their weight compared to models 10–25 times larger.</p><p><a href="https://arxiv.org/abs/2306.11644">Phi-1</a> and <a href="https://arxiv.org/abs/2309.05463">Phi-1.5</a> are among my favourites, and recently Microsoft just released a new generation of Phi family: Phi-2, a 2.7 billion-parameter language model that demonstrates outstanding reasoning and language understanding capabilities, showcasing state-of-the-art performance among base language models with less than 13 billion parameters. On complex benchmarks Phi-2 matches or outperforms models up to 25x larger, thanks to new innovations in model scaling and training data curation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3lFYoJkKosqcuyJa.png" /><figcaption>Source: <a href="https://www.microsoft.com/en-us/research/blog/steering-at-the-frontier-extending-the-power-of-prompting/">Microsoft</a></figcaption></figure><p>What I like about the Phi models are that their focuses on a different axis: <strong>the quality of the data</strong>. Common wisdom in machine learning has recognized the importance of data quality, but the Phi models take this to a new level. By meticulously selecting and filtering their training data, they achieve remarkable results with significantly smaller model sizes and training costs. This suggests that a shift towards focusing on data quality could have a profound impact on the future of AI development.</p><h3>2. We are running out of data</h3><p>Speaking of the importance of data, <a href="https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications">Chinchilla’s wild implications</a> argued that training data would soon become a bottleneck for scaling large language models, and the bad news is we might run out of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iGoR-y9qLWFm2z3J.png" /><figcaption>Source: <a href="https://epochai.org/blog/will-we-run-out-of-ml-data-evidence-from-projecting-dataset">Epoch</a></figcaption></figure><p>Humans, though, we really are a creative species. Folks at DeepMind just released a paper — <a href="https://arxiv.org/abs/2312.06585">Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models.</a> Model-generated synthetic data emerges as a promising alternative to human-generated data, offering scalability and cost-effectiveness, as long as we can ensure its quality. The paper explores a setting where an external, scalar feedback signal based on expectation-maximization serves as a quality indicator for each generated sample. Overall, the findings suggest self-training with feedback can substantially reduce dependence on human-generated data, and not only that, it also scales favorably with model size and significantly surpasses fine-tuning only on human data.</p><h3>3. The king is back</h3><p>Last week, Google’s Gemini Ultra dethroned GPT-4 and claimed supremacy especially in the MMLU benchmark. But that didn’t last long. In just a couple of days, the team at Microsoft has shown that if Gemini can beat GPT-4 by practically ‘inventing’ a new technique called <a href="https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf">uncertainty-routed Chain-of-thought</a> — where model produces k chain-of-thought samples, selects the majority vote if the model is confident above a threshold, and otherwise defers to the greedy sample choice.</p><p>Well, I can do that too! say Microsoft.</p><p>The Microsoft team steers GPT-4 with a modified version of <a href="https://www.microsoft.com/en-us/research/blog/the-power-of-prompting/">Medprompt</a>, called <a href="https://github.com/microsoft/promptbase?tab=readme-ov-file#medprompt--extending-the-power-of-prompting">Medprompt+</a>, to achieve <em>the highest score ever achieved on the complete MMLU, </em>and voila… SOTA frontier model has been reborn, yet again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MW8EfQ1aMKyiuNld.jpeg" /><figcaption>Source: <a href="https://www.microsoft.com/en-us/research/blog/steering-at-the-frontier-extending-the-power-of-prompting/">Microsoft</a></figcaption></figure><p><a href="https://arxiv.org/abs/2009.03300">MMLU(opens in new tab)</a> (Measuring Massive Multitask Language Understanding) challenge was established as a test of general knowledge and reasoning powers of large language models. The complete MMLU benchmark contains tens of thousands of challenge problems of different forms across 57 areas from basic mathematics to United States history, law, computer science, engineering, medicine, and more.</p><p>But I think the point is not who’s back on top, but a valuable insight regarding <a href="https://superficiality.substack.com/i/139171200/can-llms-reason-and-plan">prompt engineering</a> and its correlation with performances — which we have also addressed in the previous roundup.</p><p>While systematic prompt engineering can yield maximal performance, what’s more interesting to be explored is actually out-of-the-box performance of the foundation models with only simple prompts, as demonstrated in the below table. Before diving into more sophisticated and expensive techniques, employing simple prompts to establish baseline can be very useful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mAlp6Ek5t0QEHGJP.png" /><figcaption>Source: <a href="https://www.microsoft.com/en-us/research/blog/steering-at-the-frontier-extending-the-power-of-prompting/">Microsoft</a></figcaption></figure><p>This actually begs a question… can we prompt our way towards AGI?</p><h3>4. Can we prompt our way to an AGI?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g93f7k9tJioqCAQY.png" /><figcaption>Source: Paper (<a href="https://arxiv.org/abs/2311.16452">https://arxiv.org/abs/2311.16452</a>)</figcaption></figure><p>OpenAI just dropped their <a href="https://platform.openai.com/docs/guides/prompt-engineering">Prompt Engineerin</a>g guides, on top of already comprehensive Microsoft’s <a href="https://github.com/microsoft/promptbase">Promptbase</a>. Signifying the importance of prompt engineering — also known as <strong>In-Context Prompting</strong>, which refers to methods for how to communicate with LLM to steer its behavior for desired outcomes <em>without</em> updating the model weights.</p><p>However, as models grow more capable, can careful prompting unlock abilities approaching human cognition? Or are architectural constraints an obstacle no prompt can currently overcome?</p><p>Some folks <a href="https://twitter.com/kenshin9000_/status/1734238211088506967">on X (previously twitter) believe that better prompting could be a way to reach AGI</a>, or at least believe that it leads somewhere around that, except of course Yann LeCun, and some others as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uJuDK_7fOWO-BsVl.png" /></figure><p>LLMs are bound to hallucinate due to its underlying architecture and they way they are trained — next-token prediction. Making them sensitive to contextual information. As <a href="https://twitter.com/karpathy">Andrej Karpathy</a> said, <strong><em>hallucination is all LLMs do. They are dream machines.</em></strong></p><blockquote><em>We direct their dreams with prompts. The prompts start the dream, and based on the LLM’s hazy recollection of its training documents, most of the time the result goes someplace useful. It’s only when the dreams go into deemed factually incorrect territory that we label it a “hallucination”. It looks like a bug, but it’s just the LLM doing what it always does. — </em><strong><em>Andrej Karpathy</em></strong></blockquote><p>In general, Language models and search engines represent two extremes — unconstrained imagination versus bounded accuracy. An LLM is 100% dreaming and has the hallucination problem. A search engine is 0% dreaming and has the creativity problem. Something we need to find the function to maximize of. The ideal AI system should balance open imagination with focused accuracy.</p><p>Overall, prompting is a promising tool for pushing the boundaries of LLMs, but it’s likely just one piece of the puzzle in the quest for AGI. Combining prompting with other research directions, such as reinforcement learning, neuro-inspired models, and better understanding of human intelligence, could be crucial for making significant progress towards this ambitious goal.</p><p>Follow-up read: <a href="https://arxiv.org/pdf/2311.11829.pdf">System 2 Attention</a></p><h3>5. LLMs help us make scientific discoveries</h3><p>Here’s a question: could LLMs discover entirely new knowledge?</p><p>If we think about it, since hallucinations are a <strong>feature</strong>, not a bug — we can get LLMs to<strong> dream up </strong>possible solutions to complex problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/0*gHe1ubZY4KbVfGFF.png" /></figure><p>The FunSearch process. The LLM is shown a selection of the best programs it has generated so far (retrieved from the programs database), and asked to generate an even better one. The programs proposed by the LLM are automatically executed, and evaluated. The best programs are added to the database, for selection in subsequent cycles. The user can at any point retrieve the highest-scoring programs discovered so far.</p><p>That’s what the Google DeepMind team just did. In <a href="https://www.nature.com/articles/s41586-023-06924-6">“<strong>Mathematical discoveries from program search with large language models”</strong></a><strong>, </strong>they introduce FunSearch:</p><p><strong>Key points:</strong></p><ol><li>FunSearch is a new method that uses large language models (LLMs) to discover new solutions in mathematics and computer science.</li><li>It works by pairing an LLM with an automated evaluator to avoid hallucinations and generate better solutions over time. By iterating back-and-forth between these two components, initial solutions “evolve” into new knowledge.</li><li>FunSearch has made significant breakthroughs in two areas:</li></ol><ul><li><a href="https://en.wikipedia.org/wiki/Cap_set"><strong>Cap set problem:</strong></a> Discovered the largest known cap sets in specific settings, surpassing state-of-the-art methods. The problem consists of finding the largest set of points (called a <em>cap set</em>) in a high-dimensional grid, where no three points lie on a line. This problem is important because it serves as a model for other problems in extremal combinatorics — the study of how large or small a collection of numbers, graphs or other objects could be.</li><li><strong>Bin-packing problem:</strong> Found new algorithms that outperform existing heuristics, requiring fewer bins to pack items.</li></ul><p>The benefits of FunSearch:</p><ol><li><strong>Discoveries:</strong> Makes new discoveries beyond established results.</li><li><strong>Interpretability:</strong> Generates programs that explain how solutions were found, providing deeper insights.</li><li><strong>Practical applications:</strong> Solutions can be readily implemented in real-world problems.</li></ol><p>FunSearch proves that, by controlling LLMs’ hallucinations, we can leverage their incredible abilities to not only make groundbreaking discoveries in math, but also find practical solutions to important problems in the real world, like packing things efficiently or optimizing computer tasks.</p><h3>6. Emergent properties might be just a mirage</h3><p>Or is it a shimmery glimpse of the future?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GEbnkd9hvynTAdrO.png" /></figure><p>The<strong> best paper award </strong>in NeurIPs 2023 went to a paper that argues that the <a href="https://arxiv.org/abs/2304.15004">emergent abilities of LLMs could be a mirage</a>!</p><blockquote><em>Emergence is when quantitative changes in a system result in qualitative changes in behavior. — </em><a href="https://arxiv.org/pdf/2206.07682.pdf"><em>Jason Wei et al</em></a></blockquote><p><strong>Definition of Emergence:</strong> In the context of LLMs, “emergence” typically refers to the phenomenon where a system exhibits abilities or behaviors that were not explicitly programmed or anticipated by its creators.</p><p><a href="https://arxiv.org/pdf/2206.07682.pdf">Wei et al (2022)</a> argues that LLMs demonstrate emergent capabilities — as models scale in size, they exhibit novel capabilities not present in smaller models. They analyze several tasks where larger LLMs outperform smaller ones, including reasoning about abstract concepts, following complex instructions, and generating different creative text formats. These abilities are said to be sharp (transition instantly from not present to present) and unpredictable.</p><p>However, <a href="https://arxiv.org/abs/2304.15004"><strong>Schaeffer et al. (2023</strong></a><strong>)</strong> took a more skeptical view, arguing that the apparent emergence of new abilities in LLMs can be explained by factors other than pure scale, such as improved training data or specific architectural choices. They call for a more nuanced understanding of what constitutes emergence and how it should be measured. They argue that these apparent emergences are instead caused by:</p><ul><li><strong>Metric choice:</strong> Non-linear or discontinuous metrics can distort performance curves, making improvements appear sudden and unpredictable.</li><li><strong>Limited test data:</strong> Smaller models may be unfairly handicapped by insufficient test data, further exaggerating the performance gap</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ww9m4JEP2tiuVtvg.png" /><figcaption>Source: <a href="https://arxiv.org/abs/2304.15004">https://arxiv.org/abs/2304.15004</a></figcaption></figure><p>As research progresses, we will likely gain a deeper understanding of the true nature of LLMs’ capabilities and their potential implications for the future of AI.</p><p>Perhaps the most interesting aspect of LLMs isn’t just their current capabilities, but the questions they raise about the nature of intelligence and the potential for human-machine collaboration.</p><h3>Other interesting things to read this week:</h3><ol><li><a href="https://www.nature.com/articles/d41586-023-03943-1">Quantum-computing approach uses single molecules as qubits for first time </a>(Nature)</li><li><a href="https://techcrunch.com/2023/12/14/mrbeasts-analytics-platform-viewstats-is-out-in-beta/">MrBeast’s Analytics Platform ViewStats is Out in Beta</a> (TechCrunch)</li><li><a href="https://medium.com/starts-with-a-bang/ask-ethan-is-it-possible-that-gravity-isnt-quantum-8b63178904ac">Is it possible that gravity isn’t quantum?</a> by Ethan Siegel</li><li><a href="https://substack.com/inbox/post/139473358">What biology can learn from physics</a></li><li><a href="https://www.nature.com/articles/d41586-023-04002-5">Scientists drilled through 500 metres of Greenland’s ice — here’s what they found at the bottom</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b917572d6ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some interesting things to read in the middle of the week #4]]></title>
            <link>https://medium.com/@febiagil/some-interesting-things-to-read-in-the-middle-of-the-week-4-982850b5e5f4?source=rss-e84f0da9397e------2</link>
            <guid isPermaLink="false">https://medium.com/p/982850b5e5f4</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Febi Agil Ifdillah]]></dc:creator>
            <pubDate>Sun, 17 Dec 2023 02:48:56 GMT</pubDate>
            <atom:updated>2023-12-17T17:20:43.159Z</atom:updated>
            <content:encoded><![CDATA[<h4>Gemini launch, TPUv5 — AI Hypercomputer, Mistral dropped bombs, Mixture of Experts, and Mamba for accessible AI development</h4><p><strong>Notes: this article was first published on </strong><a href="https://open.substack.com/pub/superficiality/p/some-interesting-things-to-read-in?r=7buma&amp;utm_campaign=post&amp;utm_medium=web&amp;showWelcome=true"><strong><em>superficiality.substack.com</em></strong></a></p><p>Hi, it’s Febi Agil Ifdillah here with our somewhat regular weekly roundup. Which was overdue by 5 days.</p><p><strong>In today’s edition:</strong></p><ul><li>Gemini launch and OSS Models storming the AI space 🚀💥</li><li>Mixture of Experts 🧑‍🔬</li><li>Mamba 🐍🐍 — state-space models</li></ul><p><strong>But first,</strong> if you’re not a subscriber, here’s what you missed recently:</p><ol><li><a href="https://superficiality.substack.com/p/some-interesting-things-to-end-this-126">Some interesting things to end this week #3</a></li><li><a href="https://superficiality.substack.com/p/some-interesting-things-to-end-this">Some interesting things to end this week #2</a></li></ol><p><strong>Where I talked about:</strong></p><ul><li>Cryptic hints of major AGI breakthrough “Q*” project</li><li>Limitations of current autoregressive LLM architectures, including alternative approaches aim to enhance planning and problem solving abilities</li><li>DeepMind’s GNoME utilizes graph neural networks to discover 2 million+ new materials</li><li>and more.</li></ul><p>Okay, moving on to some interesting things I wanted to share this week…</p><h3>1. A Tale of Two Launch: Gemini and Mistral</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6vOoJhk8ie4wJtp_.png" /><figcaption>DALL-E</figcaption></figure><p>I can’t help but thinking.. what a time to be alive. It seems every day that we witness a (albeit sometimes small but continuous) progress in AI. Some advancements subtly nudge the field forward, while others, like bolts from the blue, redefine what we thought possible. This week has been full of the latter.</p><p>Enter the long-promised <a href="https://deepmind.google/technologies/gemini/#introduction">Gemini,</a> an AI model that’s been shrouded in anticipation is finally out, and the expectations? They’re sky-high!</p><blockquote>Gemini surpasses SOTA (State of the Art) performance on all multimodal tasks.</blockquote><blockquote><strong>- Google DeepMind</strong></blockquote><p>The Gemini family consists of Ultra, Pro, and Nano sizes tailored for various use-cases. The smallest ones, Nano-1 (1.8B) and Nano-2 (3.25B) were the most-efficient models and designed to run on-device. Evaluation on a broad range of benchmarks shows that the most-capable model, Gemini Ultra, advances the state of the art in 30 of 32 of these benchmarks.</p><p>Google seems to deliver what they have been promising since<a href="https://youtu.be/ixRanV-rdAQ?si=7n6s8STEj429O-Me"> Google I/O in May</a> — Gemini’s built from the ground up for multimodality that reasons seamlessly across text, images, video, audio, and code — which is a respectable force <a href="https://www.wired.com/story/google-deepmind-demis-hassabis-chatgpt/">against GPT-4</a>. Suffice to say that it’s kind of a relief that they finally deliver before the changing year because prior to the release, there’s a rumor going around <a href="https://www.theinformation.com/articles/google-postpones-big-ai-launch-as-openai-zooms-ahead">about its delay</a>. DeepMind’s achieving SOTA in many benchmarks however — without undermining their works, comes as no surprise given their track records. They should’ve been the leader in the space after all. If any organizations can achieve AGI, Google would be top of mind. Think about it — DeepMind’s track record is like the greatest hits of AI breakthroughs. From the legendary AlphaGo to the revolutionary AlphaFold, and the recent <a href="https://deepmind.google/discover/blog/millions-of-new-materials-discovered-with-deep-learning/">GNoME project</a>, they’ve consistently raised the bar. Even Pre-trained Transformers was born and raised at Google.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T7lIkQU7b0bzi9nY.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zq1zgP1o99_J1Eal.png" /><figcaption>Above: Multimodal benchmark performances, Below: Text-focused benchmark performances. Source: <a href="https://deepmind.google/technologies/gemini/#capabilities">DeepMind</a></figcaption></figure><p>What’s so surprising though, how narrow the gap is between Gemini 1.0 and GPT-4 which was launched many months ago. While I think what they have accomplished — putting out a worthy rival to GPT-4 — by every definition was a big feat, I expect a lot more from the built-in multimodality, but hey it’s still in its 1st version. So I’m really looking forward for the next iterations.</p><p>However, it’s noteworthy to bring up how Google felt the need <a href="https://www.bloomberg.com/opinion/articles/2023-12-07/google-s-gemini-ai-model-looks-remarkable-but-it-s-still-behind-openai-s-gpt-4">to stage their demo video</a> and cherry-pick their performances against benchmarks, which invited lots of criticism and casted a lot of doubts about the real performance of Gemini. Such a move reflects the intense competition and the high stakes in demonstrating technological prowess in the AI arena.</p><p>What particularly interesting for me is Gemini’s integration with Google’s ecosystem. One by one, Google is unveiling its “Gemini”-powered products &amp; technologies, starting from Bard, of course. Then there is <a href="https://storage.googleapis.com/deepmind-media/AlphaCode2/AlphaCode2_Tech_Report.pdf">AlphaCode 2</a>, an improved version of the code-generating <a href="https://alphacode.deepmind.com/">AlphaCode</a>, which reportedly scored better than 85% of programmers who participated in solving similar coding problems — Obviously better than me!</p><p>Google has also <a href="https://blog.google/products/pixel/pixel-feature-drop-december-2023/">brought Gemini into Pixel 8 Pro</a>, offering several advantages to the smartphone owners, from summarization features, smart replies in Gboard, to AI-powered photos and videos.</p><p>I’m particularly interested in the technical choices and challenges involved in creating these models. As the first of its kind, training Gemini required innovations in training algorithms, dataset, and infrastructure. This wasn’t about merely scaling up existing methods. The Gemini models were trained using Google’s own chips — TPUv5e and TPUv4, with the larger Gemini Ultra model utilizing numerous TPUv4 accelerators across several datacenters, a significant scale-up from the previous PaLM-2 model.</p><p>The TPUv4 accelerators are deployed in “SuperPods” of 4,096 interconnected chips that can dynamically reconfigure to maximize performance. For Gemini Ultra, a small number of cubes per SuperPod were retained for redundancy and rolling maintenance. SuperPods are combined across datacenters using Google’s network utilizing model parallelism within SuperPods and data parallelism across SuperPods. The programming model of Google’s Jax and Pathways allows simpler configuration and orchestration of training runs. Optimizations in XLA’s compiler maximize performance and minimize variation in step times. In addition, instead of checkpointing weights to persistent storage periodically, Gemini utilizes redundant in-memory model replicas for faster failure recovery and uninterrupted training.</p><blockquote><em>The overall goodput for the largest-scale training job increased from 85% to 97%.</em></blockquote><blockquote><em>— </em><strong><em>Google DeepMind</em></strong></blockquote><p>Through these optimizations, for the largest Gemini Ultra training runs, the percentage of time where useful training work was happening increased from 85% to 97% compared to their previous model, PaLM-2.</p><p>To summarize, the training of the Gemini models was a remarkable feat of engineering. It demonstrated Google Cloud’s capabilities as a highly scalable and versatile AI infrastructure platform. That’s why they launch <a href="http://Gemini models are living proof that Google Cloud as an infrastructure has proven itself as a">Google AI Hypercomputer</a> the day after Gemini’s launch, aiming to provide this infrastructure beyond Google.</p><p>There’s no doubt Google has made striking progress across many fronts like the Gemini models, new TPUv5 chips, expanding ecosystem integrations, and many others at the same time. However I can’t help but get this sense that all these technical achievements have somehow felt overshadowed. Much of the focus has shifted to the debates over the preliminary demo’s factual accuracy, and transparency concerns. What could’ve been a spotlight on impressive advancements, yet again ended up as PR missteps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*_zgYTg4jk5JpwSDL.jpeg" /><figcaption>Source: <a href="https://twitter.com/nearcyan/status/1734398791590219916/photo/1">X</a></figcaption></figure><p><a href="https://mistral.ai/">Mistral</a>, an AI startup from french, though… took an opposite direction from Google. Instead of carefully orchestrated demo, they quitely shared a direct access to the model weights.. via torrent.. twice.</p><p>But what’s so special? There are dozens of models coming out every month, right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d08InzVf8ODkG-Rt.png" /></figure><p>Aside from doing the swag release:</p><ol><li>Drop a magnet link with no fanfare</li><li>Surprise everyone with a pull request to the <a href="https://github.com/vllm-project/vllm/commit/b5f882cc98e2c9c6dde7357dbac2ec0c2c57d8cd">open-source vLLM project</a> to help the community integrate the Megablocks CUDA kernels.</li><li>Release <a href="https://mistral.ai/news/mixtral-of-experts/">the blog post </a>with an impressive feat</li><li>and..</li></ol><p>What I think Mistral did it right as well lies in their focus on optimizing 7B-tier models and focus on wider accessibility, rather than chasing ever-larger models. They also focus on what’s important: gathering feedback from the customers, iterate fast, and monetize the model through <a href="https://mistral.ai/news/la-plateforme/">their platform</a> ASAP — a la OpenAI. And like an overnight success <a href="https://techcrunch.com/2023/12/11/mistral-ai-a-paris-based-openai-rival-closed-its-415-million-funding-round/?guccounter=1&amp;source=post_page-----04f30549cc90--------------------------------">they have secured another $415million</a>. Everything somehow clicked.</p><blockquote><em>Mixtral is a powerful and fast model adaptable to many use-cases. While being 6x faster, it matches or outperform Llama 2 70B on all benchmarks, speaks many languages, has natural coding abilities. It handles 32k sequence length</em></blockquote><blockquote><em>— </em><strong><em>Mistral</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Pvpj-CyJJZowZ4LT.png" /><figcaption>Source: <a href="https://mistral.ai/news/mixtral-of-experts/">Mistral’s blog</a></figcaption></figure><p>At first no one knows how to use the weights. The journey of unlocking Mistral’s models power was a collective one. People hand in hand figuring out and share their progress in unlocking the models — like a global puzzle party, and as it unfolded.. people just realized how much power Mistral has unleashed to the public.</p><p>It feels like an inflection point. Finally a GPT-3.5 level model that can run at 30 tokens/sec on an M1! right? (I haven’t tried it, btw. But I wish that was true).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*WPS23xH5dp3Eq7sE.jpeg" /><figcaption>Source: <a href="https://twitter.com/BrianRoemmele/status/1734333713381753165">Brian Roemmele</a>, credit: ARKInvest</figcaption></figure><p>Anyway, the above chart paints a promising picture: open-source models are closing in on GPT-4, and parity feels just around the corner. However, GPT-4 can be seen as an outlier, and the significant jump from GPT-3.5 to GPT-4 is unprecedented. So we can expect that there would be a bit of a hurdle to go from GPT-3.5 level to GPT-4 in “real-life” performance — even Gemini Ultra is still being cooked up. Although to be fair, I’ve been playing with the newly Gemini-powered Bard and it feels a lot better — so my expectations are high.</p><p>Going back to Mistral. Like Google’s Gemini family, Mistral provides three endpoints:</p><ol><li><strong>Mistral-tiny</strong>. The budget-friendly option, powered by Mistral 7B Instruct v0.2, tackles English tasks with efficiency.</li><li><strong>Mistral-small</strong>. This endpoint currently serves their newest model, Mixtral 8x7B. Aces English, French, Italian, German, Spanish, and code, scoring an impressive 8.3 on MT-Bench.</li><li><strong>Mistral-medium 👀</strong>. Their highest-quality endpoint, a prototype model, dominates in English, French, Italian, German, Spanish, and code, boasting an 8.6 MT-Bench score.</li></ol><p>I don’t think it’s an exaggeration to say that Mistral has rapidly became the new darling in the OpenLM community…</p><h3>2. Mixtrals of Experts and Mamba’s State-Space Models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wdw0Na4Whfo4eAQs.png" /><figcaption>DALL-E</figcaption></figure><p>We have seen from the above that Mistral’s latest models boast an impressive performance-to-parameter ratio, which make use easily raise eyebrows. How are they punching above their weight compared to models 10–25 times larger?</p><p>Fortunately, with <a href="https://mistral.ai/news/mixtral-of-experts/">the released official blog</a>, now we know almost exactly what’s going on under the hood. Mistral matches or even outperforms Llama 2 70B &amp; GPT3.5 by employing Mixture of Experts — such a clever naming from Mistral: <strong>Mixtral of Experts.</strong></p><p>You know, when we talk about tinkering and advancing AI nowadays, it feels like a billionaire’s playground — something that only the ultra-wealthy can do, and thanks to the <a href="https://arxiv.org/pdf/2001.08361.pdf">Scaling Laws</a> (a.k.a bigger is better) that suggested increasing model size led to reliable and predictable improvements in capability, a trend of boasting trillions of parameters of neural nets are the new status symbol, leaving garage inventors in the dust.</p><p>So the effort of shrinking the models parameter while preserving (or even improving) performance, through Mixture of Experts and other alternative like state-space models, which we will talk about in a moment, are very needed innovations. These alternatives offer promise for more inclusive AI progress.</p><h4>So what is Mixture of Experts (MoE)?</h4><h4>TL;DR</h4><p>MoEs:</p><ul><li>Are <strong>pretrained much faster</strong> vs. dense models</li><li>Have <strong>faster inference</strong> compared to a model with the same number of parameters</li><li>Require <strong>high VRAM</strong> as all experts are loaded in memory</li><li>Face many <strong>challenges in fine-tuning</strong>, but <a href="https://arxiv.org/pdf/2305.14705.pdf"><strong>recent work</strong></a> with MoE <strong>instruction-tuning is promising</strong></li></ul><p><strong>Key Components:</strong></p><ol><li><strong>Experts:</strong> MoE layers consists of many experts, small MLPs or complex LLMs like Mistral 7B.</li><li><strong>Router:</strong> There are two routing strategies: token chooses the router or router chooses the token. How does it work exactly? It uses a softmax gating function to model a probability distribution through experts or tokens and choose the top k.</li></ol><p>Traditional deep learning models use one set of parameters for all inputs. Mixture of Experts (MoE) models are different — they choose different parameters for each input example. This allows them to specialize parts of the model for different inputs. Even though MoE models have a very large number of possible parameters, they still take about the same amount of computation to run. The specialized parts only activate for the right inputs. This flexibility helps MoE models handle complex data more efficiently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*He_HPe85uGUNLS6J.png" /><figcaption>MoE Layers. Source: <a href="https://arxiv.org/abs/2101.03961">Switch Transformers paper</a></figcaption></figure><p>Mixtral is a decoder-only model that utilizes a form of expert routing as well. In its feedforward layers, Mixtral selects two out of eight distinct expert parameter groups to process each token. The experts here can be viewed as smaller ML models with their own specialized parameters.</p><p>A key component is the router network, which determines which two experts are selected per token and layer. The routing strategy models a probability distribution over experts and tokens using a softmax gating function. Based on this distribution, the top k experts are chosen for each token.</p><p>This routing approach allows Mixtral to scale up to 46.7 billion total parameters, while only employing 12.9 billion per token. So despite having more overall parameters, Mixtral can process inputs and generate outputs at the same speed and cost as a 12.9 billion parameter model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C8Go5dkuHeA6WSPk.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2202.09368.pdf">Mixture of Experts with Expert Choice Routing</a></figcaption></figure><p>If you have time, please read the whole <a href="https://huggingface.co/blog/moe">explanation about MoE here.</a></p><h4>And what is Mamba?</h4><p>Foundation models (FMs) — large models pretrained on massive data then adapted for downstream tasks — have become a key paradigm in machine learning. These models, often sequence-based, handle diverse domains like language, images, and genomics. Primarily based on the Transformer architecture and its attention mechanism, they excel in complex data modeling within a context window. However, they face limitations like finite window size and quadratic scaling with window length. Although research has focused on more efficient attention variants, none have yet matched the effectiveness of the original model at scale across different domains.</p><p><a href="https://arxiv.org/abs/2111.00396">Structured state space sequence</a> (S4) models offer highly scalable inference performance. They operate with constant memory and time usage per step, irrespective of sequence length. While LSTMs and other RNNs share this scalable property, S4 models outperform them in terms of efficiency and are also parallelizable across sequence dimensions during training, making them more effective for handling long sequences.</p><p>In the paper titled <a href="https://arxiv.org/pdf/2312.00752.pdf">“Mamba: Linear-Time Sequence Modeling with Selective State Spaces”</a> the author introduces Mamba, a new architecture for sequence modeling. It enhances structured state space models (SSMs) by incorporating a selection mechanism, allowing the model to selectively propagate or forget information based on input content. This design addresses the computational inefficiency of transformers on long sequences and enables content-based reasoning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hBa5l2kIa1e0XHOo.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2312.00752.pdf">Mamba paper</a></figcaption></figure><p>Mamba’s selective SSMs offer several improvements:</p><ol><li><strong>Selection Mechanism</strong>: Enhances SSMs by parameterizing them based on input, enabling selective information propagation.</li><li><strong>Hardware-Aware Algorithm</strong>: Overcomes computational challenges of the selection mechanism through a hardware-aware parallel algorithm, ensuring faster performance ((up to 3× faster on A100 GPUs).</li><li><strong>Simplified Architecture</strong>: Mamba integrates selective SSMs into a unified architecture without attention or MLP blocks, achieving linear scaling in sequence length and high throughput.</li></ol><p>Mamba demonstrates state-of-the-art performance across multiple modalities like language, audio, and genomics. Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.</p><p>Its linear-time complexity, combined with robust performance, makes it a strong candidate for general-purpose sequential foundations models. What’s more interesting is that it can leverage a very long context (up to 1 million tokens).</p><p>In comparison to Mixture of Experts (MoE):</p><ul><li><strong>Scalability</strong>: Mamba scales linearly with sequence length, while MoE’s scalability is dependent on the number of experts.</li><li><strong>Complexity</strong>: Mamba’s selective mechanism simplifies architecture, whereas MoE requires managing multiple experts.</li><li><strong>Performance</strong>: Mamba shows strong performance across various modalities, potentially more versatile than MoE which might excel in specific tasks or domains.</li><li><strong>Efficiency</strong>: Mamba’s hardware-aware design might offer computational advantages over MoE, especially for longer sequences.</li></ul><p>Enough for today’s edition. The future looks so bright ☀️😎</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=982850b5e5f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some interesting things to end this week #3]]></title>
            <link>https://medium.com/@febiagil/some-interesting-things-to-end-this-week-3-1677c54754b4?source=rss-e84f0da9397e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1677c54754b4</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Febi Agil Ifdillah]]></dc:creator>
            <pubDate>Sun, 17 Dec 2023 02:39:44 GMT</pubDate>
            <atom:updated>2023-12-17T17:21:25.166Z</atom:updated>
            <content:encoded><![CDATA[<h4>Delving deep into LLMs reasoning ability and DeepMind breakthrough in material discovery.</h4><p>Notes: this post was first published on <a href="https://open.substack.com/pub/superficiality/p/some-interesting-things-to-end-this-126?r=7buma&amp;utm_campaign=post&amp;utm_medium=web&amp;showWelcome=true"><em>superficiality.substack.com</em></a></p><p>Throughout history, humanity has witnessed several inflection points that have dramatically advanced our civilization. We tamed fire, not just for <a href="https://en.wikipedia.org/wiki/S%27more">s’mores</a>, but to forge civilizations. We crafted languages, because grunts and gestures only get you so far — now we are able to communicate in all complexities, in many forms. Written words came next, allowing for the preservation and dissemination of knowledge. Then the printing press arrived — talk about going viral pre-internet! Then the development of self-propelled vehicles, transforming mobility, including cars turned our world from a walkathon to a road trip. All the way to the moon landing, and recently, isn’t chatting with Emojis with someone across the globe in real-time just sci-fi turned reality?</p><p>Each of these milestones has played a crucial role in shaping human society, culture, and technological progress.</p><p>In contemporary discussions, many regard Artificial Intelligence (AI) — ASI to be specific — as the next significant technological advancement. As for me, I think that’s true, along with Synthetic Biology. (just realized that the next big things are either synthetic or artificial).</p><p>Large Language Models (LLMs) are at the forefront of this AI revolution. Their rapid development suggests that we might be on the cusp of achieving Artificial Super Intelligence (ASI) — which, <a href="https://twitter.com/i/broadcasts/1MYxNoaVRjoKw?s=20">according to Elon</a> will be reached within the next three years. ASI represents a form of AI that could perform any intellectual task that a human being can and do them even better than we do. Please refer to the Levels of AGI presented in the previous article.</p><p>The exploration of AI touches upon ‘the realm of the complex’ — mind and intelligence, which have been so far beyond our reach. This venture into understanding and replicating human intelligence signifies a bold step into a domain that have had ‘Do Not Disturb’ signs for eons — always been mysterious and elusive.</p><p>It’s like trying to understand a teenager, but harder.</p><p>Anyway, the potential to replicate or even enhance aspects of human cognition and problem-solving represents a monumental leap in our understanding of ourselves and our capabilities.</p><p>However, the current AI systems have some limitations. One significant constraint is their autoregressive nature, where outputs are sequentially generated based on previous outputs. This limitation can impact the efficiency and accuracy of AI systems, especially in complex tasks that require an understanding of broader contexts or the integration of diverse data types, which led us the the first topic…</p><h3>1. Can LLMs reason and plan?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WFPb0vWB5y-YHrm6.png" /><figcaption>DALL-E</figcaption></figure><blockquote><em>Picture ChatGPT as that brilliant lyricist who can whip up a banger in seconds flat. But here’s the catch — it’s all studio magic. ChatGPT’s got no real-world chops, no tours under its belt, no late nights in dingy clubs. It’s never felt the sting of a breakup or the thrill of a road trip. It’s just riffing on the echoes of a thousand other songs it’s heard, remixing them into something new. — </em><strong>ChatGPT’s describing itself</strong></blockquote><p>The usability of Large language models (LLMs) have taken many by surprise, particularly in the past year following the launch of ChatGPT. Originally designed to generate text, scaled-up versions of language models such as GPT and PaLM have demonstrated a growing ability to handle a broad spectrum of tasks that involve mathematical, symbolic, commonsense, and knowledge-based reasoning. It’s quite astonishing that the foundation of these advancements is still the original autoregressive text generation mechanism, which processes text token by token in a sequential left-to-right manner.</p><p>Clearly, while LLMs exhibit surprising emergent generalization properties and “fluent hallucinations”, LLMs also make factual errors, logical errors, are inconsistent, and they are also — regularly — struggle on many simple reasoning tasks such as arithmetic and parity.</p><p>This raises the question: Is such a straightforward method sufficient for developing a language model into a universal problem-solving tool? are they merely “glorified” autocomplete models, which happened to memorize the entire internet? or are they truly performing novel logical reasoning?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EKsGPiWEhgl-zL0S.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2201.11903.pdf">Chain-of-thought paper</a></figcaption></figure><p>For LLMs, reasoning typically involves decomposing complex inputs into sequential intermediate steps before producing a final answer. It has been shown that the ability of language models to compute complex functions can be greatly enhanced by using chain-of-thought [<a href="https://arxiv.org/abs/2201.11903">Wei et al</a> , <a href="https://arxiv.org/abs/2205.11916">Kojima et al</a> , <a href="https://arxiv.org/abs/2305.20050">Lightman et al</a>] and scratchpad <a href="https://arxiv.org/abs/2112.00114">[Nye et al</a>] techniques.</p><p>An advanced iteration, known as <a href="https://arxiv.org/pdf/2305.10601.pdf">“Tree of Thoughts” (ToT)</a>, takes this concept further by navigating through well-defined segments of text, or “thoughts.” These thoughts act as stepping stones in the problem-solving process. ToT empowers Language Models to engage in more intentional decision-making by weighing different logical pathways and assessing their options to determine the subsequent step. It also incorporates the ability to anticipate future implications or to reconsider past decisions (backtracking), thereby enabling more global choices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PO-OCXQ5jEvyBT0G.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2305.10601.pdf">Tree of Thoughts paper</a></figcaption></figure><p>There are also other works worth mentioning, such as:</p><ol><li>RAP (Reasoning via Planning): This approach is similar to ToT but uses Monte Carlo Tree Search (MCTS) instead of depth-first or breadth-first search. It relies on heuristics from a language model, such as the likelihood or confidence of an action, to aid in the search process.</li><li><a href="https://arxiv.org/abs/2310.04406">Language Agent Tree Search (LATS)</a> that aims at unifying planning, reasoning, and acting which borrows some ideas from AlphaGO. LATS draws on Monte Carlo tree search techniques from model-based reinforcement learning and repurposes LLMs as agents, value functions, and optimizers. The core idea is to use an environment for external feedback, which provides a more deliberate and adaptive problem-solving approach, surpassing the limitations of existing techniques.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sNP-m_CznJnFlGNb.png" /></figure><p>However, <a href="http://yann.lecun.com/">Yann LeCun</a>, Vice President and Chief AI Scientist at Meta, has been a prominent voice <a href="https://youtu.be/mViTAXCg1xQ?si=i2IhR5PRAhck9hvx">suggesting</a> that the underlying architecture of autoregressive models — the ones being used in current LLMs — possesses intrinsic shortcomings that could hinder their sustainability in the long run.</p><blockquote><em>Five years from now, nobody in their right mind would use them [autoregressive models] — </em><strong>Yann LeCun</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4QDmaa4DRe7bO3lo.jpeg" /><figcaption>Source: <a href="https://youtu.be/mViTAXCg1xQ?si=i2IhR5PRAhck9hvx">Yann LeCun talk.</a></figcaption></figure><p>It’s not a surprise that OpenAI might have thought of the same thing. Hence leading to the development of the enigmatic Q* that I also cover in <a href="https://superficiality.substack.com/p/some-interesting-things-to-end-this">the previous post</a>, which according to the <a href="https://www.wired.com/story/fast-forward-clues-hint-openai-shadowy-q-project/">latest piece from The Verge</a>, was a leak (meaning it’s real). Seems to me that the speculation around Q* mostly refers to the enhanced ability to plan and reason.</p><p>This is a super interesting topic, but I’ll have to stop here for now.</p><h3>2. AI Unlocks the Door to New Material Discoveries: A DeepMind Breakthrough</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Oy4tbWQ54sVWUtGn.png" /><figcaption>DALL-E</figcaption></figure><p>Moving on from LLMs.</p><p>I can’t help but seen so many breakthroughs in science was powered by Graph Neural Networks (GNNs) recently.</p><p>DeepMind just released a groundbreaking AI tool, Graph Networks for Materials Exploration (GNoME), which I think has unleashed a new era in materials science by discovering a staggering 2.2 million new crystals, including 380,000 stable materials. This breakthrough, equivalent to nearly eight centuries of human knowledge, opens up a world of possibilities for future technologies, from superconductors and supercomputers to next-generation batteries for electric vehicles​​.</p><p>Material discovery has transitioned from labor-intensive, traditional methods to computational approaches, and now to AI-driven techniques. Where earlier methods, including those led by <a href="https://next-gen.materialsproject.org/">the Materials Project</a>, identified approximately 48,000 stable crystals, GNoME has expanded this to a remarkable 421,000. This transformation in material discovery methodology signifies an unprecedented scale and accuracy in predicting viable materials​​.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XPptDNmpFupBpUqK.png" /><figcaption>Source: Author, adapted from DeepMind. About 20,000 of the crystals experimentally identified in the ICSD database are computationally stable. Computational approaches drawing from the Materials Project, Open Quantum Materials Database and WBM database boosted this number to 48,000 stable crystals. GNoME expands the number of stable materials known to humanity to 421,000.</figcaption></figure><p>GNoME utilizes graph neural networks (GNNs) to predict the stability of new materials. The tool has two distinct pipelines: a structural pipeline that creates candidates based on known crystal structures, and a compositional pipeline that takes a more randomized approach based on chemical formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B3dzXtnDpgOrEfD1.png" /><figcaption>Souce: <a href="https://www.nature.com/articles/s41586-023-06735-9#Fig1"><strong>Scaling deep learning for materials discovery</strong></a></figcaption></figure><p>I’ve looked at <a href="https://github.com/google-deepmind/materials_discovery">the open-sourced code </a>and found two models:</p><ol><li>GNoME Model: Utilizes a graph-based approach for predicting material stability and properties.</li><li><a href="https://www.nature.com/articles/s41467-022-29939-5">Nequip Model</a>: Focuses on training interatomic potentials, instrumental in understanding material behavior.</li></ol><p>These predictions undergo rigorous testing using Density Functional Theory calculations, enhancing the AI’s learning and prediction accuracy from 50% to a notable 80%​​.</p><p>GNoME has not only accelerated the discovery of new materials but also driven down their discovery costs. Its database of crystals is a treasure trove for scientists, offering ‘recipes’ for new materials. The technology has been validated by external researchers, who have created 736 of GNoME’s predicted materials in labs globally. Following are six examples ranging from a first-of-its-kind Alkaline-Earth Diamond-Like optical material (Li4MgGe2S7) to a potential superconductor (Mo5GeB2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/0*7Xgm-0gtSUikaU4E.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wfCI47vkXf0XqSGN.png" /></figure><p>One thing that I think made it even mind-blowing is that DeepMind and Berkeley Lab truly take it to the next level. They set up a <a href="https://www.nature.com/articles/s41586-023-06734-w#Sec6">robot-run lab called A-Lab</a> to make new materials fast using automatic methods. They used info from the Materials Project and tips on which materials would last from GNoME. Over 17 days of operation, the A-Lab successfully synthesized 41 of 58 target materials that span 33 elements and 41 structural prototypes. This success highlights how powerful it is to combine computer calculations from the ground up, machine learning, knowledge from past experiences, and automated processes in experiments. The materials-discovery pipeline followed by the A-Lab is schematically shown in the below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pWWPGLYIYHv3eXt_.png" /><figcaption>Source: <a href="https://www.nature.com/articles/s41586-023-06734-w/figures/1">Autonomous materials discovery with A-Lab</a></figcaption></figure><p>DeepMind’s AlphaFold and GNoME projects are great examples of AI solving big scientific challenges. AlphaFold’s success in figuring out protein structures has been a game-changer in biology. Then, with GNoME, DeepMind moved into new territory by finding millions of new materials.</p><p>The future looks so bright.. 😎☀️</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1677c54754b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some interesting things to end this week #2]]></title>
            <link>https://medium.com/@febiagil/some-interesting-things-to-end-this-week-2-6712a55b5104?source=rss-e84f0da9397e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6712a55b5104</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Febi Agil Ifdillah]]></dc:creator>
            <pubDate>Sun, 17 Dec 2023 02:29:14 GMT</pubDate>
            <atom:updated>2023-12-17T17:22:57.887Z</atom:updated>
            <content:encoded><![CDATA[<h4>The return of Altman, OpenAI Q*, NVIDIA Go Brrrr, Claude 2.1 vs GPT-4 and which one remembers better.</h4><p>Notes: this post was first published on <a href="http://superficiality.substack.com/"><em>superficiality.substack.com</em></a><em>.</em></p><h3>1. Sam Altman returns to OpenAI, Q*, and Artificial General Intelligence</h3><p>Covering about the unfolding drama at OpenAI lately blurs the line and making me feel no different than a gossip writer. But it’s too difficult to ignore such a buzz. It’s the hottest thing in the tech world recently (which I also cover in <a href="https://open.substack.com/pub/superficiality/p/some-interesting-things-to-end-this?r=7buma&amp;utm_campaign=post&amp;utm_medium=web&amp;showWelcome=true">my previous update</a>).</p><p>In one of the greatest uno reverse card the history has ever seen, Sam Altman went from being the victim of coup — ousted by the board from his own company — to then landing a top stop at Microsoft to lead the specially created division only for him by <a href="https://twitter.com/satyanadella/status/1726509045803336122">Nadella</a>. Then, in a (100th) twist, he wound up <a href="https://twitter.com/OpenAI/status/1727206187077370115">back in charge at OpenAI</a>. One of the reason might be because 700+ out of 770 the employees threaten to leave OpenAI and follow Altman to his new venture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sUyGKbF_qcwwuFE4.png" /></figure><p>This whirlwind of events at OpenAI has sparked widespread curiosity and speculation about the reasons behind such dramatic shifts. <a href="https://twitter.com/8teAPi/status/1725724907722752008">A particularly intriguing theory</a> that I’ve come across suggests that Ilya Sutskever, the co-founder and Chief Scientist, encountered something alarming, prompting a reaction that has elicited skepticism from various corners, <a href="https://twitter.com/elonmusk/status/1726753202786173249">including Elon Musk</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/0*5wfEW2BFDJ5jm8ue.jpeg" /><figcaption>Source: <a href="https://twitter.com/AISafetyMemes/status/1727436097922462115">AI Safety Memes</a></figcaption></figure><p>The speculation doesn’t stop there, especially in the wake of a <a href="http://(https//www.reuters.com/technology/sam-altmans-ouster-openai-was-precipitated-by-letter-board-about-ai-breakthrough-2023-11-22/)">Reuters report</a> that hints at insider claims linking Sam Altman’s departure to a significant breakthrough in Artificial General Intelligence (AGI) made by OpenAI with codename “Q*”. This only adds fuel to the speculative fire surrounding the company’s internal dynamics and the implications of their advancements. This is where things get more interesting.</p><h3>2. What could OpenAI’s breakthrough Q* be about?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YrUuNCX6PnppwuqX.png" /><figcaption>Generated by DALL-E</figcaption></figure><blockquote><strong><em>Q</em></strong><em>* is a precursor to AGI. — Really?</em></blockquote><p>The enigmatic term Q* has triggered yet another flurry of speculation within the AI community. Is it truly a stepping stone toward Artificial General Intelligence (AGI)?</p><p>The name suggests a possible connection to <a href="https://en.wikipedia.org/wiki/Q-learning">Q-learning</a> — a reinforcement learning technique where “Q” represents the quality of a state-action combination. One might hypothesize that Q* indicates the optimal Q-values, akin to finding the ideal solution within the Bellman equation framework. Another angle might be a hybrid approach, blending the <a href="https://en.wikipedia.org/wiki/A*_search_algorithm#">A* search algorithm</a> with Q-learning principles.</p><p>I recall a while back, Demis Hassabis of Google DeepMind <a href="https://www.wired.com/story/google-deepmind-demis-hassabis-chatgpt/">hinted</a> that their system, <a href="https://www.theverge.com/2023/11/16/23964937/googles-next-generation-gemini-ai-model-is-reportedly-delayed">Gemini</a>, is positioned to outpace OpenAI, by incorporating strategies from their previous triumph, AlphaGo.</p><p>Could Q* involve an AlphaGo-like Monte Carlo Tree Search, navigating the potential paths of token generation? It’s not far-fetched when you consider that existing approaches, such as those used in <a href="https://alphacode.deepmind.com/">AlphaCode</a>, have successfully applied naive brute force techniques within a language model to get huge improvements in competitive programming. By improving this method to sort through potential sequences of code more efficiently, we could see major improvements in areas like software development and mathematics, where you can clearly determine whether a solution is correct. I think that “Q*” relates to boosting the AI’s ability to solve complex problems, particularly those involving math.</p><p>There is also another potential reference to Q* in <a href="https://www.sciencedirect.com/science/article/abs/pii/0004370273900131">this particular paper.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8pYdpTCjj6KQIWWh.png" /></figure><p>Another thing I’d like to highlight regarding the discussion surrounding AGI is that it’s a topic that can quickly lead to diverse interpretations and expectations. The contribution by the DeepMind team in clarifying this <a href="https://arxiv.org/pdf/2311.02462.pdf">through their paper</a>, which outlines the various stages of AGI, is incredibly valuable. It provides a much-needed framework for understanding and measuring progress towards this formidable goal in AI research.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*39g3mITUitu_q3KS.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2311.02462.pdf">DeepMind paper</a></figcaption></figure><h3>3. NVIDIA Go Brrr now everbody wants the juice</h3><p>All this bullish rally on AI, who’s the real winner?</p><p>The past week’s drama has overshadowed NVIDIA’s remarkable <a href="http://(https//nvidianews.nvidia.com/news/nvidia-announces-financial-results-for-third-quarter-fiscal-2024)">financial performance</a>. The figures are more than just numbers; they’re a testament to a transformative era where AI is not just a buzzword but a tangible driver of growth and innovation that are becoming a part of our everyday lives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dyeX6lLNldCMwUVe.png" /></figure><p>Speaking of the use of AI in our everyday lives. I tried to make ChatGPT (and even Bard) make the charts for distilling the insights from the above figures. After numerous attempts and a bit of tinkering, here are the results I’ve managed to achieve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*De1V5Jg6j-Uo--Nc.png" /></figure><p>Source: Personal Gallery. Generated by AI. So please don’t take it seriously.</p><p>Now, considering the state of AI, it seems we’re quite far from a truly “competent AGI” (as outlined in the Levels of AGI framework). If you put the charts created by AI side by side with those made by humans (below), the AI-generated charts appear rather basic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Co0fjNYvkF0ZRiQX.png" /><figcaption>Source: <a href="https://twitter.com/Thomas_Woodside/status/1728195339331751955/photo/1">Thomas Woodside</a></figcaption></figure><p>But in all seriousness, in the world where now our worth as challenger to the quest of achieving AGI is defined by whether or not we are <a href="https://www.semianalysis.com/p/google-gemini-eats-the-world-gemini">GPU-rich or GPU-poor</a>, which I think is reasonable, NVIDIA came out as the biggest winner in the space.</p><p>Thousands of companies and countries begged NVIDIA to purchase more GPUs to the point that the <a href="https://www.reuters.com/technology/us-restricts-exports-some-nvidia-chips-middle-east-countries-filing-2023-08-30/">US Government imposed an export control</a> — something that they usually do because of “national security” concerns.</p><p>If you think current sales are already big, you wouldn’t believe that it would be even bigger next year. NVIDIA is forecasted t<a href="https://www.tomshardware.com/news/nvidia-to-reportedly-triple-output-of-compute-gpus-in-2024-up-to-2-million-h100s">o sell over 3 million GPUs next year</a>, about 3x their 2023 sales of about 1 million H100s. AMD is <a href="https://www.reuters.com/technology/amd-rises-ai-chip-sales-prediction-bodes-well-rivalry-with-nvidia-2023-11-01/">forecasting $2B of sales</a> for their new MI300X datacenter GPU.</p><p>However, while NVIDIA’s dominance in the GPU market remains strong, there is a growing incentive and effort among major customers and new entrants to challenge this monopoly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FN6QNFH769zF75WX.png" /><figcaption>Source: <a href="https://twitter.com/absoluttig/status/1722794980786467189">@absoluttig</a></figcaption></figure><p>Hyperscalers like Google, Microsoft, and Amazon are developing their own AI chips to cut costs and reduce dependence on NVIDIA. Google’s TPU, for example, may not match NVIDIA’s H100 in performance, but offers significant cost savings. To paint a picture, sourcing a H100 from NVIDIA would cost Google around $33,000.</p><p>Microsoft, collaborating with AMD, is working on the <a href="https://www.theverge.com/2023/5/5/23712242/microsoft-amd-ai-processor-chip-nvidia-gpu-athena-mi300">Athena project</a>, while Amazon is exploring various AI strategies, including chip development.</p><p>AMD is positioning itself as a cost-effective alternative to NVIDIA, with its MI300 chip. However, the different software requirements for AMD’s chips pose challenges for widespread adoption.</p><p>Startups like <a href="https://www.cerebras.net/">Cerebras</a> are also entering the market with innovative designs and potentially competitive performance, but face hurdles in gaining market trust and scaling production.</p><p>The post-production GPU market is evolving, with cloud providers offering GPUs on rent, but smaller companies seek more flexible rental options. Companies like <a href="https://www.coreweave.com/">CoreWeave</a> and <a href="https://lambdalabs.com/">Lambda</a> are filling this gap by providing variable access to GPUs.</p><p>Given the apparent link between increased computing power and advancements in artificial intelligence, it’s reasonable to expect that the demand for computing resources will continue to grow exponentially. There’s a chance that companies might put too much money into making big groups of GPUs — special computers used for AI — over the next few years, and they might end up with more than they actually need.</p><h3>4. Claude 2.1 has been released</h3><p>I’m a big fan of Anthropic and have been looking forward to try out the Claude API. However, I’ve not yet had the opportunity to use it. And now, they’ve released an updated version, Claude 2.1, boasting the ability to understand over 200,000 tokens of context. But how useful it is?</p><p>Fortunately, <a href="https://twitter.com/GregKamradt">this absolute legend on X</a> conducted an extensive test, costing him $1,000 dollars and here I got to enjoy the insights:</p><ol><li>At 200K tokens (nearly 470 pages), Claude 2.1 was able to recall facts at some document depths</li><li>Facts at the very top and very bottom of the document were recalled with nearly 100% accuracy</li><li>Facts positioned at the top of the document were recalled with less performance than the bottom (similar to GPT-4)</li><li>Starting at ~90K tokens, performance of recall at the bottom of the document started to get increasingly worse</li><li>Performance at low context lengths was not guaranteed</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mSx3s35O2GNcPXUi.jpeg" /><figcaption>Source: <a href="https://twitter.com/GregKamradt/status/1727018183608193393">GregKamradt</a></figcaption></figure><p>However, if we take a look at his previous test on GPT-4 with 128k context, the GPT-4 model seems to perform better than Claude 2.1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K9IX4L1RXLM1NKlF.jpeg" /><figcaption>Source: <a href="https://twitter.com/GregKamradt/status/1722386725635580292">GregKamradt</a></figcaption></figure><p>I think from the above figures, we can say that:</p><ul><li>In shorter contexts, GPT-4 seems to outperform Claude-2.1 across various document depths.</li><li>Both models exhibit reduced accuracy with longer contexts, but Claude-2.1 is more affected beyond 95K tokens.</li><li>Claude-2.1 shows a steady decrease in accuracy with deeper document placement, whereas GPT-4 struggles more with facts in the middle.</li><li>As the amount of data increases, both models struggle to maintain consistent performance, although this is more marked for Claude-2.1.</li></ul><p>So all in all, GPT-4 is still the boss.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6712a55b5104" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some interesting things to end this week #1]]></title>
            <link>https://medium.com/@febiagil/some-interesting-things-to-end-this-week-1-c5a415f791a9?source=rss-e84f0da9397e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5a415f791a9</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Febi Agil Ifdillah]]></dc:creator>
            <pubDate>Sun, 17 Dec 2023 02:22:31 GMT</pubDate>
            <atom:updated>2023-12-17T02:31:17.137Z</atom:updated>
            <content:encoded><![CDATA[<p>Notes: This post was first published on <a href="https://superficiality.substack.com/p/some-interesting-things-i-learned"><em>superficiality.substack.com</em></a></p><p>What a week.</p><p>What a day, specifically. Waking up to <a href="https://openai.com/blog/openai-announces-leadership-transition">this news</a> is nothing short of shocking.</p><p>We have been discussing the possibility of AI replacing jobs for a while now. Unfortunately, we might have a concrete evidence already. According to <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4544582">a new working paper</a> by Hui, Reshef, and Zhou, the introduction of ChatGPT led to a decrease in demand for freelancers on an online freelancing platform. Here’s a graph by John Burn-Murdoch:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LT-JJd23t_xPrO8U.png" /><figcaption>Source: <a href="https://twitter.com/jburnmurdoch/status/1722938749519077688">John Burn-Murdoch</a></figcaption></figure><p>In light of sudden departure of Sam Altman, people on twitter (X) jokingly dubbed that some of the first victims to the rising GenAI is Sam Altman himself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eXznw5uLdAK9rkri.png" /></figure><h3>1. OpenAI/Sam Altman Coup</h3><p>Joking aside, Sam Altman’s ousting from the company sent shockwaves through the tech community, instantly becoming a trending topic on social media and tech news outlets. The unexpected departure of Greg Brockman (Co-founder), following closely on Altman’s heels, only added fuel to the fire of speculation and rumor.</p><p>There are rumblings about a division within the company’s board regarding company’s vision. One side is apparently the ‘Safetyism’ group, advocating for a more cautious and measured approach to the AI development. The other — the ‘Acceleration’ faction — is said to be pushing for a more aggressive growth plan.</p><p>Considering the whirlwind of information that’s out there, it’s pretty tough to separate fact from fiction — as the old saying goes, “everything you hear is an opinion, not a fact.” Now, there’s this one tweet (speculation, of course) that’s caught my attention :</p><p><a href="https://twitter.com/8teAPi/status/1725724907722752008">https://twitter.com/8teAPi/status/1725724907722752008</a></p><p>But one thing for sure, this internal tug-of-war is creating a lot of buzz and speculation. It’s not just a simple leadership change; it’s a fundamental clash of visions that could define the company’s trajectory and the industry for years to come.</p><h3>2. DeepMind’s GraphCast</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*exsVrZQMafWKKhbJ.jpeg" /><figcaption>Source: <a href="https://deepmind.google/discover/blog/graphcast-ai-model-for-faster-and-more-accurate-global-weather-forecasting/">Google DeepMind</a></figcaption></figure><p>Predicting the weather stands as one of humanity’s earliest and most complex scientific pursuits. Medium range predictions are important to support key decision-making across sectors, from renewable energy to event logistics, but are difficult to do accurately and efficiently.</p><p>Traditionally, forecasts have depended on Numerical Weather Prediction (NWP), a method that starts with meticulously crafted physics equations that are subsequently converted into computer code to be processed by supercomputers. This conventional method represents a remarkable scientific and engineering achievement, but crafting these equations and their corresponding algorithms demands considerable time, specialized knowledge, and expensive computational power to yield precise predictions.</p><p>Deep learning adopts an alternative strategy, leveraging data rather than physical equations to construct a system for weather forecasting. Machine Learning model developed by learning from many years of historical weather records, enabling it to understand the intricate cause-and-effect dynamics that dictate the progression of the Earth’s weather patterns from now into the forthcoming days.</p><p>A couple months ago, Huawei released <a href="https://www.huawei.com/en/news/2023/8/pangu-weather-forcast">Pangu-Weather Forecast Model </a>which demonstrated promising results against ECMWF IFS (a leading global NWP — Numerical Weather Prediction system). As reported <a href="https://www.huawei.com/en/news/2023/8/pangu-weather-forcast">here</a>, Machine learning (ML) is rapidly proving its worth in the realm of weather forecasting, demonstrating significant potential to enhance predictive capabilities.</p><p>In May 2023, the AI-driven Pangu-Weather model successfully forecasted the path of Typhoon Mawar a full five days ahead of its unexpected shift in direction — a remarkable achievement.</p><p>Meanwhile, the rivalry in advancing machine learning for weather forecasting between Chinese and American firms has led to a rapid and competitive race, with companies from both nations consistently outpacing each other in developments.</p><blockquote><em>Our state-of-the-art model delivers 10-day weather predictions at unprecedented accuracy in under one minute — </em><strong>Google Deepmind</strong></blockquote><p>DeepMind, Google’s renowned research arm, has introduced GraphCast, its cutting-edge weather forecasting model, which outshines the previous benchmarks established by the prominent Chinese company, Huawei Technologies. According to the latest test outcomes, GraphCast has outperformed Pangu-Weather, Huawei’s model, in the majority of testing environments. However, Pangu-Weather still maintains an edge in certain areas, including time-resolution benchmarks, where it delivers predictions with less time interval between them.</p><p>GraphCast is a weather forecasting system based on machine learning and Graph Neural Networks (GNNs), which are a particularly useful architecture for processing spatially structured data. It’s not the first time Google used GNN in their systems — In 2021 they release an advanced traffic prediction with GNN (https://deepmind.google/discover/blog/traffic-prediction-with-advanced-graph-neural-networks/).</p><p>Although the training phase for GraphCast required significant computational resources, the end product is an extremely efficient forecasting model. Generating a 10-day forecast with GraphCast is a quick process, clocking in at under a minute when using just one of Google’s TPU v4 units. By contrast, creating a similar 10-day forecast with traditional methods like the HRES model could take several hours and involve a supercomputer harnessing the power of hundreds of machines.</p><p>I think what’s even more interesting from this development is that the code is open-sourced: <a href="https://github.com/google-deepmind/graphcast">https://github.com/google-deepmind/graphcast</a></p><h3>3. Microsoft Mega Infrastructure</h3><p>Microsoft is undertaking an unprecedented infrastructure expansion that, despite sounding like an exaggeration, dwarfs the spending on historical mega projects. When you compare it to the investment in national railway systems, massive dams, or even the Apollo moon missions, the projected annual expenditure of over $50 billion on data centers from 2024 onward makes these projects look modest. This ambitious expansion is designed to fast-track the journey toward AGI (Artificial General Intelligence) and integrate the capabilities of generative AI into every aspect of our lives, from work productivity tools to entertainment.</p><p><a href="https://www.semianalysis.com/p/microsoft-infrastructure-ai-and-cpu?utm_source=substack&amp;utm_campaign=post_embed&amp;utm_medium=web">Read more here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5a415f791a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Software is Feeding the World]]></title>
            <link>https://medium.com/@febiagil/software-is-feeding-the-world-6f61ea1dff39?source=rss-e84f0da9397e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f61ea1dff39</guid>
            <category><![CDATA[agriculture-technology]]></category>
            <category><![CDATA[sdgs]]></category>
            <category><![CDATA[agriculture]]></category>
            <category><![CDATA[sustainability]]></category>
            <dc:creator><![CDATA[Febi Agil Ifdillah]]></dc:creator>
            <pubDate>Mon, 08 Nov 2021 04:38:17 GMT</pubDate>
            <atom:updated>2021-11-08T05:41:02.211Z</atom:updated>
            <content:encoded><![CDATA[<h3><em>Introduction</em></h3><p>Neurafarm is participating in the <a href="https://www.cultivhacktion.id/"><strong>Cultivhacktion | Innovative Agriculture Hackaton</strong></a><strong>, </strong>a hackathon event organized by the World Bank Group, Microsoft, Tanihub Group, in partnership with the MoA and other key players in the digital agriculture sector. The hackathon aims to demonstrate the value of agriculture data and digital technologies in addressing the key challenges in Indonesia’s agri-food sector as well as build the capacities of young innovators, to develop digital solutions for agriculture.</p><p>Through this event, Neurafarm hopes to share our work to wider audiences.</p><h3><strong>Background</strong></h3><p>Human population is climbing steadily toward 10 billion by 2050 (FAO) and the demand for food is increasing rapidly. All in the midst of unprecedented climate change that will tax even the most resilient systems. Today’s food and agricultural systems are accountable for a quarter of the world’s greenhouse gas emissions and 70% of global freshwater use. Those same systems deplete soils, water quality, and natural habitats all across the world. At the same time, up to 40% of global yield is lost due crops disease and pest annually (Source: <a href="https://www.fao.org/news/story/en/item/1402920/icode/">https://www.fao.org/news/story/en/item/1402920/icode/</a>). This is a threat for global food security and also financial burden for farmers like Mr. Ace in Sumedang, West Java, whose yield was decreased from 200kg to 40kg because of pests &amp; diseases (more than 80% loss). Mr. Ace is not alone, there are more than 34 million smallholders in Indonesia who are prone to similar problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m1DVgdAXklINZ8v9Uw1Q0w.png" /></figure><h3><strong>Existing Solutions</strong></h3><p>Plant disease &amp; pests can be mitigated, but existing solutions are limited. When it comes to disease &amp; pests, it takes some time for the symptoms to manifest. It’s also take a long time for farmers to get answers when they consult with agriculture extension agents because there’s only 67,000 agriculture extension agents to help farmers. It’s also challenging to find the best products for treatment because there are thousands of active ingredients out there. Current solutions are not scalable.</p><h3>Neurafarm Solutions</h3><p>At Neurafarm, we believe that the power of innovation and technology can solve the world’s biggest challenges of food security. That’s why we build Dokter Tania, AI-powered super app that allows farmer to identify disease through image. Farmers can get comprehensive diagnosis, treatment recommendation, product recommendation, and access to thousands of disease management all in a single app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*48tgcSp-YAX_0MQMnVJcxA.png" /></figure><p>The only thing that the farmers need to do is just take a picture of a plant that looks unhealthy, or upload it from their gallery and send it to Dokter Tania, and then our AI will be able to tell what kind of disease that affects the plant, how to treat it, and recommending the best products for treatment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l7g4ONCTkO_WJEgyk2LU9Q.png" /><figcaption>Toko Tania: Farm Supply Shop that acts as human interface for farmers</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DvJzVOE-az3ulg131cC7nw.png" /><figcaption>Neurafarm’s Farm Supply Shop Network in Bandung Area</figcaption></figure><p>Farmers can also visit offline farm supply shop network where they can get access to wide-range of products and services including Paylater. So farmers can can get the products and then pay it later at harvest.</p><p>So far, there are 17 farm supply shop in Bandung Area joining our partnership program. They will act as human interface and provide O2O experience for farmers.</p><h3>Super App for Farmers</h3><p>Dokter Tania provides comprehensive tools to help farmers on their day-to-day activities including:</p><ol><li><strong>Disease Radar</strong></li></ol><p>Through Disease Radar, farmers can see what’s happening in their surroundings and also what’s happening all across Indonesia: the most trending disease, in what area, at what time, and what kind of crops. This way, farmers can react accordingly and plan mitigation way before it affects them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GwhYmGKf2fp91tQCE_HvuA.png" /><figcaption>Disease Radar can help farmers to see the most trending diseases &amp; pests around them</figcaption></figure><p><strong>2. Fertilizer Calculator</strong></p><p>One of the biggest component in farming cost structure is inputs. Traditionally, farmers use their experience to decide the amount of fertilizer and what type of products that they use in each farming cycle. However, it often leads to excessive use of chemical which not only taking a huge chunk of cost from farmers’ pocket, but also harmful for the environment and us as consumer. Through fertilizer calculator feature, farmers can calculate fertilizer usage recommendation. They can also use our <strong>Soil Testing</strong> service which only costs $6/sample for better results. Our customers can reduce inputs cost by up to 30% and save millions of Rupiahs by using this service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lUvqTiDDUAS7V-_PPeq_zg.png" /><figcaption>Fertilizer Calculator feature helps farmers to apply the right amount of fertilizer. They can also opt for using Know Your Soil service for only $6/sample.</figcaption></figure><p><strong>3. Cultivation Guidance</strong></p><p>Not only that, farmers can also access 1400+ disease management recommendation in more than 140+ commodities. Dokter Tania provides the most comprehensive information and best-in-class recommendations. Including recommendation for treatment using mechanical control, biological control, and chemical control, linked to Toko Tania (<em>online and offline</em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O60dZ5gSgo58CqLiYEPxtg.png" /><figcaption>Farmers can access disease management guidance in 1400+ diseases and 140+ commodities</figcaption></figure><p><strong>4. Market Insights</strong></p><p>Farmers often lack access to information regarding price and its trend. This often leads to failure in gaining profit. Farmers can better anticipate demand if they get updated information in price and commodities (what other farmers are planting now), the trend, and others. Dokter Tania provides price information in important commodities in 34 provinces in Indonesia, alongside market insights, and trends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mEWNUzPLJTfQGWHhVtQTNg.png" /><figcaption>Market Insights feature help farmers to identify trend and demand</figcaption></figure><p><strong>5. My Fields</strong></p><p>Neurafarm realizes that disease management is not complete without disease &amp; pests prevention. Scalability is our utmost priority, our goals to become Indonesia #1 agriculture data capture and information provider drives us towards building a new feature called My Fields where farmers can register their fields by drawing polygon. Farmers will receive daily recommendation regarding weather (and predictions for the next 4 days), hourly spraying recommendation, and personalized daily recommendations. In the past 2 months alone, there are more than 1000 fields have been registered to the platform, amounting to 3000+ hectares of land covered. Next phase development would include satellite imagery data to determine vegetation index so that farmers can reap more benefit by using Dokter Tania on a daily basis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CVOIm2TL7xwOJbb4oGI4wA.png" /></figure><h3>Neurafarm’s Impact</h3><p>Dokter Tania has been used by more than 18,000 farmers from 200+ cities in Indonesia. These farmers collectively manage more than 15,000+ hectares of land. From the piloting that we conducted at Cimenyan &amp; Lembang, Indonesia, timely guidance and intervention can help farmers increase their profit/hectare/season by up to USD 900. Imagine the impact we could make if we can do this at scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y3UrQqRF5GBqe9c-RRabTg.png" /><figcaption>Neurafarm’s impact</figcaption></figure><h3>Neurafarm’s Vision</h3><p>Neurafarm has always had a big vision: to help achieve food security and build more resilient and sustainable agriculture ecosystem. Neurafarm believes that innovation and technology can solve the world’s biggest challenges in the agriculture sector. Starting from Dokter Tania as our initial entry, Neurafarm aims to build an ecosystem around Dokter Tania to provide end-to-end solution for farmers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xKDmpVGDNR709xsVj_tYBA.png" /></figure><p>By democratizing technology to farmers, now we are able to collect previously wasted data and information in the agriculture sector. With more data, soon we’ll be able to help farmers to cope with plant &amp; disease better, give personalized recommendation at all stages of work, predicting the emergence of pest &amp; disease in the future, and more. Beyond disease &amp; pest and stakeholders within the sector, technology and data can streamline information &amp; insights to the other verticals and empower stakeholders such as agri-inputs companies, financial institutions, and government to create better policy and improve farm sustainability.</p><p>Going forward, Neurafarm’s strategy is to empower farmers and businesses to grow more, reach more audience, and distribute better.</p><p>Do you share the same vision? Let’s support Neurafarm’s journey to build more resilient and sustainable agriculture ecosystem in Indonesia &amp; beyond through technology.</p><p><strong>Cheers,</strong></p><p>Febi Agil Ifdillah (CEO Neurafarm)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f61ea1dff39" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lebaran 2020 dan Peran Neurafarm di Masa Pademi]]></title>
            <link>https://medium.com/@febiagil/lebaran-2020-dan-peran-neurafarm-di-masa-pademi-f687687ae580?source=rss-e84f0da9397e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f687687ae580</guid>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[agriculture]]></category>
            <category><![CDATA[agtech]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Febi Agil Ifdillah]]></dc:creator>
            <pubDate>Sat, 23 May 2020 06:47:19 GMT</pubDate>
            <atom:updated>2020-05-27T04:03:07.687Z</atom:updated>
            <content:encoded><![CDATA[<h3>Lebaran 2020 dan Peran Neurafarm di Masa Pandemi</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*ui7dCXtTndNExSeUVdEpOg.jpeg" /><figcaption>Petani sedang berkegiatan di Lahan. Sumber: Neurafarm</figcaption></figure><p>Notes: english version can be <a href="http://febiagil.me/blog/2020/05/23/eid-al-fitr-2020-neurafarm-contribution/"><strong><em>accessed here</em></strong></a></p><p>Hi, Semoga Anda dalam keadaan sehat.</p><p>Di tengah-tengah krisis yang sedang terjadi saat ini, berbagai macam sektor terkena imbasnya. Berbagai kalangan terkena dampaknya. Tak terkecuali sektor agrikultur, dan tentunya para petani sebagai penggerak industri pertanian.</p><p>Sebagai sektor yang menjadi tumpuan lebih dari 100 juta masyarakat Indonesia untuk mencari nafkah*, dan menopang ratusan juta penduduk Indonesia untuk memenuhi kebutuhan pangan sehari-hari, sudah selayaknya kita memberikan perhatian lebih kepada industri pertanian dan berbagai aktor yang terlibat di dalamnya.</p><p>Terdapat jutaan petani di Indonesia dan sebagian besar terkena dampak negatif dari pandemi COVID-19. Mulai dari ditutupnya hotel, restoran, dan cafe, sehingga menyebabkan <em>channel </em>penjualan yang menyempit. Hingga menurunnya daya beli masyarakat sehingga banyak hasil lahan yang tidak tertampung.</p><p>Selama tiga bulan ke belakang, Neurafarm telah melakukan beberapa upaya untuk membantu petani tetap produktif di tengah pandemi guna menjaga ketahanan pangan. Pandemi semakin menyadarkan kami tentang betapa rapuhnya sistem pangan dunia saat ini, dan perlunya upaya-upaya yang lebih besar lagi untuk memastikan ketahanan pangan.</p><p>Namun untuk sekarang, tim <a href="http://www.neurafarm.com/"><strong>Neurafarm</strong></a> akan libur sampai seminggu ke depan untuk merayakan Idul Fitri. Beberapa upaya pendampingan petani tetap dilakukan walaupun tidak se-intens sebelumnya. Neurafarm berharap Anda semua senantiasa dalam keadaan sehat dan kami juga mengucapkan Selamat Hari Raya Idul Fitri bagi yang merayakannya!</p><p>*sumber: <a href="https://www.pertanian.go.id/home/?show=news&amp;act=view&amp;id=2564">https://www.pertanian.go.id/home/?show=news&amp;act=view&amp;id=2564</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VJQXyZwNQlpk31z8MLJG2g.png" /><figcaption>Selamat Hari Raya Idul Fitri 1441 H. Sumber: Neurafarm</figcaption></figure><h3>Apa yang telah dilakukan Neurafarm?</h3><p>Sebagai startup yang berfokus untuk mendukung ketahanan pangan dunia melalui teknologi, selama pandemi ini <a href="http://www.neurafarm.com/"><strong><em>Neurafarm</em></strong></a> berkomitmen untuk semakin meningkatkan pelayanan kepada para petani untuk dapat tetap produktif selama krisis ini berlangsung, dan seterusnnya. Beberapa fokus yang kami lakukan diantaranya adalah:</p><ul><li>Memberikan Konsultasi Online Gratis</li><li>Memberikan pelatihan dan akses ke ahli</li><li>Mempererat kerjasama dengan berbagai pihak untuk digitalisasi pertanian</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UTkvWVqeVfjZMEc6z-7v6Q.png" /><figcaption>Tiga fokus Neurafarm untuk mendukung produktivitas petani. Sumber: Neurafarm</figcaption></figure><p>Dalam upaya tersebut, Neurafarm telah memberikan 500 lebih sesi konsultasi gratis, dua kelas online gratis, dan bermitra dengan berbagai pihak untuk membantu 3500 petani tambahan yang begabung ke dalam komunitas kami selama periode Maret-Mei 2020.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j64h_gatjJHlcg2AS6V8xA.png" /><figcaption>Terdapat lebih dari 3500 petani tergabung dalam periode Maret-Mei 2020. Sumber: Neurafarm</figcaption></figure><h3>All New Dr. Tania Dirilis</h3><p>Dr. Tania merupakan aplikasi yang dapat membantu para petani untuk mendeteksi penyakit tanaman hanya melalui gambar (menggunakan AI), berkonsultasi dengan ahli, mengurangi resiko, dan meningkatkan hasil lahan. Untuk mendukung ribuan pengguna Dr. Tania, Neurafarm melakukan beberapa peningkatan dan perbaikan, serta menambah beberapa fitur.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lyP_6QGagDf5G6Y3uPGA-g.png" /><figcaption>All New Dr Tania untuk mendukung kegiatan bertani. Sumber: Neurafarm</figcaption></figure><p>Fitur unggulan pada versi ini adalah Tanya Ahli yang memberikan akses bagi para petani untuk berdiskusi tentang permasalahan di lahan, Kakulator Pupuk untuk penghitungan cepat terkait kebutuhan pupuk di lahan sehingga petani dapat menggunakan pupuk secara lebih efisien. Kemudian Panduan Budidaya yang telah diupdate, dengan lebih dari 600 penanganan penyakit, di 70 komoditas, dan tentunya Identifikasi Penyakit yang didukung teknologi Artificial Intelligence yang dapat mengidentifikasi lebih dari 30 penyakit di hanya melalui gambar.</p><h3>Dua Kelas Online Sukses Dilaksanakan</h3><p>Sebagai komitmen Neurafarm untuk memberikan pelatihan dan akses ke ahli bagi petani, kami menyelenggarakan Neurafarm Learning Academy yang mengadakan Kelas Online Gratis. Saat ini, kami telah mengadakan dua kelas online gratis. Dalam dua acara tersebut, lebih dari 100 orang telah berpartisipasi.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W4aSMj2jBlCiYX93W7IrPA.png" /><figcaption>Kelas Online Gratis dari Neurafarm</figcaption></figure><h3>Aksi Sayur untuk Petani di Lembang dan Masyarakat</h3><p>Tak hanya Kelas Online Gratis, namun pada acara kedua Neurafarm Learning Academy, kami juga membuka donasi selama periode 10–17 Mei 2020 untuk mendukung “Aksi Sayur” Pak Karya di Lembang, yaitu menyerap hasil lahan para petani, dan membagikan paket sayur segar berisi tomat, wortel, kubis, beras, dan lain sebagainya kepada masyarakat yang membutuhkan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IDldzpKJaFPrWF2LBdLpIw.png" /><figcaption>Dana terkumpul dan penerima manfaat dari Aksi Sayur. Sumber: Neurafarm</figcaption></figure><p>74 paket sayur segar telah didistribusikan kepada masyarakat yang membutuhkan. Terimakasih untuk Anda yang telah berkontribusi. Selain membantu 74 keluarga/orang untuk mendapatkan makanan, Anda juga telah memberikan manfaat untuk 26 petani yang tergabung dalam aksi ini.</p><p>Kami percaya bahwa sekecil apapun kebaikan yang diberikan, akan memberikan manfaat bagi banyak orang.</p><h3>Buletin Neurafarm Edisi April Dirilis</h3><p>Tak berhenti sampai di situ, Neurafarm merilis Buletin Neurafarm Edisi April untuk membagikan info-info seputar pertanian, membagikan kisah-kisah inspiratif dari para petani, dan juga opini seputar pertanian. Buletin ini dapat diakses GRATIS oleh semua orang.</p><p>Buletin tersebut dapat diakses pada tautan<a href="http://issuu.com/neurafarm"> <strong><em>berikut ini</em></strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-hxrxWKyxMgw668RvI_4og.png" /><figcaption>Buletin Neurafarm Edisi April. Sumber: Neurafarm</figcaption></figure><h3>Apa Selanjutnya?</h3><p>Pandemi ini menghadapkan kita pada kondisi yang mungkin belum pernah kita alami sebelumnya. Biasanya, ketika guncangan semacam ini hadir, beberapa hal baru pun bermunculan. Kebiasaan-kebiasaan baru, pengalaman baru, wawasan baru, dan juga permasalahan-permasalahan baru. Tentunya, bersama permasalahan-permasalahan akan muncul peluang dan kesempatan baru. Bagi Neurafarm, kejadian ekstrim semacam ini menjadi kesempatan untuk memikirkan ulang tentang bagaimana kita memenuhi kebutuhan pangan dunia. Demi tercapainya sistem pangan yang lebih tangguh dan efisien, dan juga lebih berkelanjutan.</p><p>Neurafarm lahir karena kami percaya bahwa inovasi dan teknologi dapat membantu kita menyelesaikan berbagai macam permasalahan, termasuk ketahanan pangan dunia.</p><p>Ke depannya, Neurafarm akan terus berupaya mendukung tercapainya pertanian yang berkelanjutan, sehingga kita bisa memenuhi kebutuhan pangan dunia tanpa harus mengorbankan generasi mendatang.</p><p>Terimakasih sudah membaca sampai akhir.</p><p>Salam,<br>Febi Agil Ifdillah<br>CEO <a href="http://www.neurafarm.com/"><strong>Neurafarm</strong></a><br>More Precise, More Impact</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f687687ae580" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On Generative Modelling Part 4 — Text to Image Synthesis]]></title>
            <link>https://medium.com/@febiagil/on-generative-modelling-part-4-text-to-image-synthesis-451e92b1eac3?source=rss-e84f0da9397e------2</link>
            <guid isPermaLink="false">https://medium.com/p/451e92b1eac3</guid>
            <category><![CDATA[gans]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Febi Agil Ifdillah]]></dc:creator>
            <pubDate>Sun, 26 Aug 2018 00:00:00 GMT</pubDate>
            <atom:updated>2018-08-26T06:50:55.330Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O52qETsTeU0N8fGm.png" /><figcaption><a href="http://www.febiagil.me/images/2018/august/generative-modelling-4-cover.png">S</a>ource: <a href="http://torch.ch/blog/2015/11/13/gan.html">http://torch.ch/blog/2015/11/13/gan.html</a></figcaption></figure><p>Long Post Alert!</p><p>TLDR; Generating images from text descriptions. See the result at the bottom of this post. Scroll down, scroll down, scroll down :)</p><h3>Introduction</h3><p>This is the 4th instalment of ‘On Generative Modelling’ series. If you haven’t read my previous articles, go check them out: <a href="http://febiagil.me/blog/2018/07/18/generative-modelling-1-driving-simulator/">part one</a>, <a href="http://febiagil.me/blog/2018/07/18/generative-modelling-2-adversarial-neural-cryptography/">part two</a>, and <a href="http://febiagil.me/blog/2018/07/18/generative-modelling-3-adversarial-perturbation-elimination/">part three</a>. Each part has a different topic from the other, so feel free to jump to whichever part you want.</p><p>Also check my other series called ML.Rand() series :</p><ol><li><a href="http://febiagil.me/blog/2018/07/18/random-ml-1-synthetic-gradients-decoupled-neural-interfaces/">part1</a>, Synthetic gradients and decoupled neural interfaces</li><li><a href="http://febiagil.me/blog/2018/07/18/random-ml-2-learning-to-learn/">part2</a>, Learning to learn (meta-learning)</li><li><a href="http://febiagil.me/blog/2018/07/19/random-ml-3-building-mind-reader-machines-theory-of-mind/">part3</a>, How to build a mind-reader machine</li><li><a href="http://febiagil.me/blog/2018/07/19/random-ml-4-building-machines-with-navigational-abilities/">part4</a>. Building machines with navigational abilities</li></ol><p>In this article, we will continue our discussion about generative modelling; but now we have a special topic: text-to-image synthesis. I’ve been working on this very topic since a year ago for my bachelor’s thesis. So I wish to take this opportunity to share and communicate what I’ve been doing since I started the project and the progress that I’ve made on the subject. Consider this as a ‘simplified’ write-up of my work. I hope you enjoy it.</p><h3>What is it that you’re working on?</h3><p>If you’ve been paying attention to my posts, you’ve probably noticed it. I’ve talked about my work briefly in my first article of generative modelling series. I’ve been meaning to build an automated text to 3D model generator but a lot of things had happened so I decided to let it go. With the same spirit, I changed the topic and my objectives to only create 2D images from text descriptions. My work is not new nor innovative in any way, I just found it exciting and it is still an open problem in the research community. It’s based on Reed et al works in 2016 [1], so I’m ‘a little bit’ lagging behind. (Cut me some slack, okay! LOL)</p><p>Lots of researchers called it text-to-image synthesis.</p><p>See, in the context of visual, several deep learning models have been developed to create ‘artworks’ automatically, e.g Inceptionism [3] and Perceptual Losses[4]. Automated image generation/synthesis will be useful to develop an interactive computational graphic design, image fine-tuning, and perhaps even animation. Traditionally, computer is able to produce images by using handmade algorithms. But as you might have known, the result would be pretty much abstract, random, and full of patterns as it is limited by the algorithm itself.</p><p>Generative modelling can be used to attack this problem. Various generative modelling technique/models including Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), Generative Moment Matching Networks (GMMNs). Auto-regressive Neural Network, and real NVP have been widely used to generate samples (images).</p><p>Various research has shown that data generation process could be improved and controlled by conditioning the process with external information e.g tags. In the case of image, we can control the generated image by its properties including intensity, shape, and colour [5, 6].</p><p>On the other hand, natural language could be an alternative to control image generation process [1]. With natural language, we could describe the data(image) that we want to produce ‘easily’.</p><blockquote><em>Text-to-image synthesis is kind of the holy grail of computer vision, since if an algorithm can generate truly realistic images from mere text descriptions, we can have a high confidence that the algorithm actually understands what is in the images, where computer vision is about teaching computers to see and understand visual contents in the real world. — </em><strong><em>Huang, H., Yu, P. S., &amp; Wang, C. (2018)</em></strong></blockquote><p>Text-to-image synthesis is not an easy task to solve. But with the recent development of text representation and image synthesis techniques, it’s been possible to produce images from mere text description.</p><h3>How to Build Such a System?</h3><p>Text-to-image synthesis is a reverse captioning problem (image-to-text): given an image, a text description of the image must be produced. If you think about it, the problem can be seen as a translation task where the same semantic is encoded into two different languages: image and text. So we want to translate text-to-image: given a text description, an image which matches that description must be generated.</p><p>But translating image-to-text or text-to-image is different from text-to-text in the sense that text-to-text translation has a smaller result space. When we translate a given sentence from one language to another language (text-to-text), there would be a relatively ‘small’ number of ‘valid’ results corresponding to the text.</p><p>It is different from text-to-image or image-to-text task where a huge amount of results might exist. For example, the following sentence “A gang of elks standing near the bank of a narrow river” will result in lots of valid images as it is unclear how many elks are in the ‘gang’. Moreover, the description of the background is uncertain.</p><p>However, text-to-image synthesis itself can be considered as a harder problem than captioning (image-to-text). This is mostly because language structure which is inherently sequential so that the formation of the next word could be conditioned on the previously generated words.</p><p>There are two main problems that we need to solve in order to build a text-to-image synthesis system. First, we need to represent the given sentences as vectors which capture important features. Second, we need to use these vector representations to generate images by using generative models.</p><h3>How to convert sentences into vectors?</h3><p>We could use text embedding. There are (at least) three representation techniques which are really promising:</p><ol><li>Skip-thought [7], is an encoder-decoder model which is trained on a corpus of novels across 16 genres. The encoder maps the input sentence to a sentence vector and the decoder generates the sentences surrounding the original sentence resembling skip-gram model in the sense that surrounding sentences are used to learn sentence vectors. Skip-thought aimed at producing generic, distributed sentence representations that are robust and perform well in practice.</li><li>Sent2Vec [8], is a ‘relatively’ simple model that offers an incredibly well performance in multiple benchmarks including but not limited to paraphrase identification (MSRP), subjectivity classification (SUBJ) and polarity of opinion(MPQA). See Table I below for the details. In generals, there are two recent trends in text understanding research (or distributed representations of text). The first one employs models with high complexity e.g RNN, LSTM, Neural Turing Machines or other attention-based models. The second one is making use of matrix factorizations (bilinear model) with shallow architecture. This kind of model is able to be trained on a huge amount of data at a higher speed. Sent2Vec is one of those ‘shallow’ models. It is very efficient with the complexity of O(1) to process each word in a sentence into vectors at inference/training time. After comparing this technique with skip-thought, I can safely say that it’s somewhat true. When I tried to infer +-80.000 sentences using skip-thought, it took me 12 hours++ to complete the task. Meanwhile, the same task can be completed within 15 minutes by using Sent2Vec.</li><li>Char-CNN-RNN [9]; many researchers claimed that this model is superior to other techniques for multi-modal task where image and text are involved the likes of text-to-image synthesis. It’s because the model is trained using text and image jointly. In fact, this model was employed in the GAN-INT-CLS model where the first text-to-image synthesis model introduced by Reed et al (2016).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*x7ILggbSyqgwmkfS.png" /><figcaption><em>Table I: Performance comparison of Sent2Vec (unigram/bigram) accross multiple supervised evaluations. Source: Personal Gallery, adapted from Pagliardini et al (2018)</em></figcaption></figure><h3>What about converting the vectors into images?</h3><p>For the generation part, we need a kind of model that can make use of the text representations and convert the vectors into images. The right class of model that can be exploited is generative models. Among other generative models, GANs have shown incredible results on MNIST, CIFAR-10, CUB-200, and LFW datasets with relatively higher inception scores. The produced samples are also much sharper than VAE’s results. That’s why I chose GANs for my bachelor’s thesis. Well, there are actually some other reasons:</p><ol><li>GANs produce ‘better’ result than other generative models (sharp and compelling)</li><li>They do not require a likelihood function to be specified. So they don’t have to learn the density of data explicitly. This is an advantage because sometimes when the data distribution is complex, we couldn’t find the density so it could be a problem when we were to learn it explicitly.</li><li>GANs are flexible in terms of its loss function and its topology.</li></ol><p>However, it is important to note that despite all its strength. GAN is well-known as hard to train. Its training process is very unstable and requires a lot of tricks to get a good result. Moreover, GAN also suffers from the mode collapse. It’s a phenomenon where the generator network is only able to produce samples from some of its data mode. In other words, the generator learns to produce samples with extremely low variety. This is because, in the original formulation, the discriminator network is only focused on spotting fake samples/true samples. So it doesn’t need to consider the variety of synthetic samples. Many methods have been proposed to address this mode collapse problem including minibatch features[10]. In my bachelor’s thesis, I attack this problem by using Wasserstein GAN.</p><p>To make the implementation simpler, the kind of GAN that I employed was a direct GAN, where there are only a generator and a discriminator in the architecture. While it’s easier to implement than hierarchical methods (e.g SS-GAN, LR-GAN) or iterative methods (e.g LAPGAN, StackGAN) it’s also still capable of producing good-enough (acceptable) results.</p><p>To make the problem simpler, I created several constraints for the training data:</p><ol><li>Background/Foreground should be distinguishable.</li><li>The images contain only 1 object.</li><li>The object is a structure e.g number, face, bird.</li></ol><p>Datasets that met those criteria are Oxford-102 Flowers Dataset and Caltech-UCSD Birds 200–2011. You can also use Behance Artistic Media (BAM) Datasets or any other available datasets as you please. But I used the Oxford Flowers Dataset only. Descriptions for each image is taken from Reed et al (2016), and after some pre-processing, we got pairs of image and descriptions (up to 10 descriptions per image) as shown in Table II.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/0*vOdHxyBJoHFtFzVz.png" /><figcaption><em>Table II: Samples of the training data. Source: Personal Gallery.</em></figcaption></figure><h3>Training Setup</h3><p>One of the objectives is to compare the variety of generator architectures and how it affects the generated images. There are three types of generator architecture employed: small, normal, and deep. I also wanted to know how different text embedding techniques affect the generated samples so I decided to use Sent2Vec (bi-gram + uni-gram) and Skip-thought (combine-skip: bi-skip + uni-skip) vectors in separated experiments.</p><p>The model was trained on a machine with this following specifications:</p><ol><li>Processor : 2 x Intel® Xeon ® @ 2.30GHz,</li><li>RAM : 14GB</li><li>GPU : NVIDIA Tesla K80, 24GB GDDR5</li></ol><p>Adam optimizer is used for training with beta1 = 0.5, and the model is trained for a total of 175 epochs (as of now, will be trained for more in the near future) with a batch size of 64. For the Sent2Vec experiment, I used 700 vector dimension for uni-gram and bi-gram. Then the vectors are concatenated to noise with a dimension of 100 resulting in 1500 dimension of vector input. As per the Skip-thought experiment, I combined the uni-skip vector and bi-skip vector resulting in a combine-skip vector of 4800 dimension. After the noise dimension was added, the vector input would be a dimension of 4900.</p><h3>Result</h3><p>After roughly 10 hours of training, it reached 100 epoch. Let’s take a look at the following table of Sent2Vec experiment results at epoch 100.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/0*q8QnEEcGCEzWb0Wj.png" /><figcaption><em>Table III: Generated images at epoch 100. Source: Personal Gallery.</em></figcaption></figure><p>As you can see, the results are really bad at epoch 100. There’s not much variation, and the images do not correspond with the text descriptions at all. So I decided to run 15 more epochs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/0*_3q7EmRaWLNYPGqf.png" /></figure><p>There is not much of improvement after 115 epoch but now each object is clearer than before. In each image, the border is sharper, the background and the foreground is more distinguishable. And after 60 more epochs…. voila.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/0*oNsXoG7LpBZ_pa-J.png" /></figure><p>It’s much better now. We can see that each image is a little bit corresponding to its text description. Let’s take a look at the fourth description, the generated images have small round pink petals as the description says. And in the third description: “The petals on this flower are white with a yellow centre”, as you can see, it produces flower images with a yellow centre in each image. The model has learned about colour, and shape from text descriptions. It’s really cool! :D</p><h3>Conclusion</h3><p>In this work, I present Generative Adversarial Networks and their application in the problem of text to image synthesis. I obtained relatively good results from the experiments but I think there’s a lot of things that could be improved. I will definitely be adding more of my experiment results in the future.</p><h3>IMPORTANT NOTES</h3><p>I don’t know if you noticed it, but my Sent2Vec model above suffers from a common problem in GAN called mode collapse. A collapsed GAN produces low variety samples because the Generator is collapsing into narrow distribution that only covers some portion of data distribution.</p><p>Wasserstein GAN (WGAN) promises to remedy that problem above by using Wasserstein Distance (Earth-Mover Distance or EM for short). If you think of the probability distributions as mounds of dirt, the EM distance describes how much effort it takes to transform one mound of dirt so it is the same as the other using an optimal transport plan. Here, we also restrict the range of the weights, to guarantee Lipschitz continuity. But, will WGAN cure my model? let’s see :D</p><h3>References</h3><ul><li>[1] Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., &amp; Lee, H. (2016). Generative adversarial text to image synthesis. arXiv preprint arXiv:1605.05396.</li><li>[2] Huang, H., Yu, P. S., &amp; Wang, C. (2018). An Introduction to Image Synthesis with Generative Adversarial Nets. arXiv preprint arXiv:1803.04469.</li><li>[3] Mordvintsev, A., Olah, C., &amp; Tyka, M. (2016). Inceptionism: Going Deeper into Neural Networks. Retrieved 10 30, 2017, from Google Research Blog: <a href="http://googleresearch.blogspot.co.uk/2015/06/inceptionism-going-deeper-into-neural.html">http://googleresearch.blogspot.co.uk/2015/06/inceptionism-going-deeper-into-neural.html</a></li><li>[4] Johnson, J., Alahi, A., &amp; Fei-Fei, L. (2016). Perceptual losses for real-time style transfer and super-resolution. European Conference on Computer Vision (pp. 694–711). Springer.</li><li>[5] Dosovitskiy, A., Tobias Springenberg, J., &amp; Brox, T. (2015). Learning to generate chairs with convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 1538–1546).</li><li>[6] Kulkarni, T. D., Whitney, W. F., Kohli, P., &amp; Tenenbaum, J. (2015). Deep convolutional inverse graphics network. Advances in Neural Information Processing Systems, (pp. 2539–2547).</li><li>[7] Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., &amp; Fidler, S. (2015). Skip-thought vectors. arXiv: Computation and Language, 3294–3302. Retrieved 10 31, 2017, from <a href="https://arxiv.org/abs/1506.06726">https://arxiv.org/abs/1506.06726</a></li><li>[8] Pagliardini, M., Gupta, P., &amp; Jaggi, M. (2017). Unsupervised learning of sentence embeddings using compositional n-gram features. arXiv preprint arXiv:1703.02507.</li><li>[9] Reed, S., Akata, Z., Lee, H., &amp; Schiele, B. (2016). Learning deep representations of fine-grained visual descriptions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 49–58).</li><li>[10] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., &amp; Chen, X. (2016). Improved techniques for training gans. Advances in Neural Information Processing Systems, (pp. 2234–2242).</li></ul><p><em>Originally published at </em><a href="http://febiagil.me/blog/2018/08/26/generative-modelling-4-text-to-image-synthesis/"><em>febiagil.me</em></a><em> on August 26, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=451e92b1eac3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>