<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nikhil Nagaraj on Medium]]></title>
        <description><![CDATA[Stories by Nikhil Nagaraj on Medium]]></description>
        <link>https://medium.com/@nikhil.nagaraj?source=rss-b0852981290a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8KNp8T9Xa-7nL1nG-edGnA.jpeg</url>
            <title>Stories by Nikhil Nagaraj on Medium</title>
            <link>https://medium.com/@nikhil.nagaraj?source=rss-b0852981290a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 17:43:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nikhil.nagaraj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Amazon Bedrock: Building a solid foundation for Your AI Strategy]]></title>
            <link>https://medium.com/ml6team/amazon-bedrock-building-a-solid-foundation-for-your-ai-strategy-7faf19b8c7c0?source=rss-b0852981290a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7faf19b8c7c0</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[bedrock]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[claude]]></category>
            <dc:creator><![CDATA[Nikhil Nagaraj]]></dc:creator>
            <pubDate>Wed, 06 Sep 2023 08:25:18 GMT</pubDate>
            <atom:updated>2023-09-06T08:25:18.501Z</atom:updated>
            <content:encoded><![CDATA[<p>Amazon recently announced their foundation model service: Amazon Bedrock. We got our hands on it as part of a limited preview. So, what’s in it for you?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4lButoOPgTORGnlz" /><figcaption>Photo by <a href="https://unsplash.com/@christianw?utm_source=medium&amp;utm_medium=referral">Christian Wiediger</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Over the last few months, every week has seen a new entrant in the generative AI space. Some weeks, it is a new open-source model, whilst, on others, established players release their foundation models and services.</p><p>OpenAI (and by extension, Microsoft) kicked this off with ChatGPT. Meta soon followed this with their LLaMA models, which the open-source community warmly embraced. Not to be left far behind, Anthropic released Claude, and recently, Google announced their generative AI services at <a href="https://cloud.google.com/blog/products/ai-machine-learning/google-cloud-launches-new-ai-models-opens-generative-ai-studio">Google I/O</a>.</p><p>A meticulous reader might notice that the list above lacks a significant player, one that revolutionised cloud services as we know them, and they’d be right. A couple months ago, Amazon announced their foundation model service, Amazon Bedrock. As part of the beta release, we got our hands on it, and through this blog, we show you what it offers and our first impression!</p><h3>What is Bedrock all about?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/0*flvt_FitCmAaUQxf.png" /><figcaption>Generative AI on AWS (<a href="https://aws.amazon.com/blogs/machine-learning/announcing-new-tools-for-building-with-generative-ai-on-aws/">Source</a>)</figcaption></figure><p>Announced as a part of their push to build with Generative AI on AWS, Amazon Bedrock is a service that makes various foundation models (hence the name, Bedrock) accessible via an API. It allows customers to build and scale generative AI-based applications without worrying about the infrastructure and maintenance required to run these models. It is, Foundation-Models-as-a-Service.</p><h4>Isn’t this the same as Sagemaker Jumpstart?</h4><p>While Sagemaker Jumpstart offers a plethora of pretrained models that can be fine-tuned or deployed on demand, Bedrock offers similar and much larger models as a managed service, thereby reducing the process of using a foundation model to a simple API request.</p><h3>Which models can you access through Bedrock?</h3><p>Currently, Amazon Bedrock supports NLP and Vision Foundation models. While the number of models supported is already impressive, we expect this list to grow longer.</p><h4><strong>Anthropic’s Claude</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*cAFvIsWPG2tBm1mvq5VrlA.gif" /><figcaption>Claude in action (<a href="https://www.anthropic.com/index/introducing-claude">Source</a>)</figcaption></figure><p>Bedrock provides access to two of Anthropic’s Claude family of models; Claude, with a context size of 12K tokens to handle sophisticated dialogue and creative content generation tasks and Claude Instant, with 9K tokens for faster performance and more straightforward language tasks.</p><p>Generally, Claude’s family of language models offer excellent performance while emphasising steerability and safety using Anthropic’s <a href="https://www.anthropic.com/index/claudes-constitution">Constitutional AI</a> approach.</p><h4><strong>AI21’s Jurassic</strong></h4><p>Another family of LLMs, this time from AI21 Labs, is also available through Amazon Bedrock. Two models, J2 Grande Instruct and J2 Jumbo Instruct, are accessible from the Jurassic family. The Jurassic models offer multilingual capabilities with support for English, Spanish, French, German, Italian, Portuguese and Dutch.</p><h4><strong>Amazon’s Titan FM</strong></h4><p>Along with the launch of Bedrock, Amazon also announced the launch of their foundation model family, called Titan. Currently, two models are available through Bedrock; Titan Text Large, a generative LLM and Titan Text Embeddings, a model which, as the name suggests, can generate text embeddings for applications such as retrieval and recommendation.</p><p>While models like Claude and J2 are instruction-following models trained to perform well in a chat-like setup, Titan is a language model like GPT-3 useful for NLP tasks like summarisation, information extraction and text generation.</p><p>Titan FMs can detect and remove harmful content in data, reject inappropriate content in the user input, and filter the models’ outputs that contain inappropriate content (such as hate speech, profanity, and violence).</p><h4><strong>Stable Diffusion</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mlM5jc0boo5qKT3cMryQkg.png" /><figcaption>Images generated using Stable Diffusion through Amazon Bedrock</figcaption></figure><p>While the previously mentioned models deal with text, Bedrock also supports image generation through Stable Diffusion models. Currently, it provides access to Stable Diffusion XL, which allows text-to-image prompting, image-to-image prompting, and image inpainting.</p><p>However, at this time, only text-to-image prompting is possible through Amazon Bedrock.</p><h3>Our first impressions</h3><p>As part of a limited number of AWS partners with access to Bedrock, we tried the available models and tooling to test its capabilities. Here’s what stood out!</p><h4><strong>Model Variety</strong></h4><p>One of the stand-out qualities of Bedrock is the variety of models available. While Azure and Google services are limited to their proprietary offerings, Amazon Bedrock makes available a much larger variety of models providing better choices to make smarter trade-offs in quality, performance and cost.</p><h4><strong>Focus on safety</strong></h4><p>Amazon Titan effectively filtered out unsafe/harmful content, refused to answer and pointed the user to its terms of service. Such filters will be obligatory when the general public interacts with these models as part of user-facing applications.</p><h4>Playgrounds</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tBPuMkaHbQTv2F8P3aI0HQ.png" /><figcaption>Playground on Amazon Bedrock</figcaption></figure><p>Like other generative AI offerings, Amazon Bedrock also provides a nifty playground where one can test various prompts and settings for each model before integrating it within their application. This should allow quick testing and prototyping, thus reducing overall development time.</p><h4>Ease of Access</h4><p>Amazon promises that <em>“Bedrock is the easiest way to build and scale generative AI applications with FMs.”</em> So far, we agree!</p><pre>{<br>  &quot;modelId&quot;: &quot;amazon.titan-tg1-large&quot;,<br>  &quot;contentType&quot;: &quot;application/json&quot;,<br>  &quot;accept&quot;: &quot;*/*&quot;,<br>  &quot;body&quot;: {<br>   &quot;inputText&quot;: &quot;Product: Sunglasses. Keywords: polarized, designer, comfortable, UV protection, aviators. Create a table that contains five variations of a detailed product description for the product listed above, each variation of the product description must use all the keywords listed.&quot;,<br>   &quot;textGenerationConfig&quot;: {<br>      &quot;maxTokenCount&quot;: 4096,<br>      &quot;stopSequences&quot;: [],<br>      &quot;temperature&quot;:0,<br>      &quot;topP&quot;:1<br>     }<br>   } <br>}</pre><p>Accessing a model is as simple as sending a request with the appropriate parameters and input (see code snippet above). Support for Bedrock has been added to the official AWS Python SDK <strong>boto3</strong>, making the service relatively straightforward to use. Additionally, being a managed service means businesses will not have to worry about scaling and its related problems as a larger user base uses their product.</p><h4>Pricing</h4><p>While the exact pricing details are still under an NDA 🤐, we can share that the prices are extremely competitive and sometimes substantially lower than corresponding rates offered by OpenAI.</p><h3><strong>Our expectations for the future</strong></h3><p>While Bedrock already shows tremendous promise, it is worth remembering that it is still in preview. Here’s what we expect will change/be added soon:</p><h4><strong>General Model Fine-tuning</strong></h4><p>Amazon’s release blog on Bedrock stated that customers could fine-tune these models with their data, thus allowing specific models for specific use cases. However, this functionality is currently only available for its Titan models. The ease of fine-tuning will be a significant game changer regarding the adoption of Bedrock by businesses.</p><h4><strong>(Non) Availability in the EU</strong></h4><p>Currently, Amazon Bedrock is only available in 2 regions of the US. However, this is a problem for EU businesses that must adhere to the continent’s data protection and localization laws. Ideally, this should change soon to allow EU businesses to adopt generative AI and adhere to GDPR laws.</p><h4><strong>Expansion of Stable Diffusion Capabilities</strong></h4><p>Users of Bedrock can only use Stable Diffusion for text-to-image prompting, while it can do so much more! Capabilities like in-painting and image-to-image generation should be added to Bedrock in the near future.</p><h3>Conclusions</h3><p>In the rapidly advancing realm of generative AI, large cloud companies seem to converge towards the same ideas. Some common threads are offering foundation models as a managed service, various sizes and modalities, the possibility of fine-tuning, and the promise of safety.</p><p>Within this landscape, Amazon’s Bedrock is an easy-to-use platform for a broad range of AI applications. Along with its ease of use, a stand-out feature of Bedrock is the availability of models from various providers like Anthropic, AI21, Cohere and StabilityAI, along with their own Amazon Titan models. This unique blend of versatility, ease of use, and a focus on safety creates a service that developers and businesses will welcome with open arms.</p><p>It’s fair to say that with Bedrock, Amazon is not just introducing another AI service but laying a solid foundation for future AI strategies for businesses worldwide. While the full potential of this service is yet to unfold, the preview itself is an encouraging glimpse into the future of AI-based solutions on AWS.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7faf19b8c7c0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml6team/amazon-bedrock-building-a-solid-foundation-for-your-ai-strategy-7faf19b8c7c0">Amazon Bedrock: Building a solid foundation for Your AI Strategy</a> was originally published in <a href="https://medium.com/ml6team">ML6team</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Harnessing the Power of Foundation Models for your Business with LoRA]]></title>
            <link>https://medium.com/ml6team/harnessing-the-power-of-foundation-models-for-your-business-with-lora-3dcd6a1264bf?source=rss-b0852981290a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3dcd6a1264bf</guid>
            <dc:creator><![CDATA[Nikhil Nagaraj]]></dc:creator>
            <pubDate>Mon, 10 Jul 2023 07:40:29 GMT</pubDate>
            <atom:updated>2023-07-10T07:40:29.861Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IIiWmkKDZf0AUn4d" /><figcaption>Photo by <a href="https://unsplash.com/@tvick?utm_source=medium&amp;utm_medium=referral">Taylor Vick</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The latest AI trends are foundation models. Think of Large Language Models (LLMs) such as GPT or image generators such as Stable Diffusion.<br>These foundation models are trained at scale on vast quantities of data, resulting in a model that can be adapted to a wide range of downstream tasks. They can be directly used for various applications but perform better when finetuned for the specific task. However, these large foundation models are costly to fine-tune, requiring vast computational resources and memory.</p><p>In this blog, we look at how your business can use a technique called LoRA to harness large foundation models without splurging a lot of money. <a href="https://medium.com/@nikhil.nagaraj/hola-soy-lora-782dec995772">For a technical deep-dive of LoRA, read our technical blog here.</a></p><h3>What is fine-tuning?</h3><p>Fine-tuning is updating the parameters of a pre-trained model with new data. This allows the model to adapt to the characteristics and nuances of the target domain or task while preserving the general knowledge learned from the large-scale pre-training. However, it comes with a few drawbacks (especially in the case of large foundational models) :</p><ul><li>It requires retraining all the model parameters, which can be billions in the case of LLMs, which means more time, energy and money spent on training.</li><li>It increases the risk of overfitting, especially when the new data is small or noisy, which can degrade the model&#39;s performance on other tasks or domains.</li></ul><p>To address these challenges, researchers from Microsoft proposed a novel technique called LoRA: Low-Rank Adaptation of Large Language Models.</p><h3>LoRA: Low-Rank Adaptation</h3><p>LoRA is based on low-rank approximation, a mathematical method to reduce the complexity and dimensionality of large matrices by finding smaller matrices that approximate them well.</p><p>LoRA works by freezing the original weights of the pre-trained model and training a <strong>much smaller number of parameters</strong>. These tuned parameters are then added to the pretrained model. The figure below illustrates this process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/0*sry71sERhHy58Ueb.png" /><figcaption>Schematic depicting the approach propsed by LoRA (<a href="https://arxiv.org/abs/2106.09685">Source</a>).</figcaption></figure><h3>Advantages of LoRA</h3><ul><li>Reduce the number of trainable parameters by several orders of magnitude, depending on the rank chosen for the update matrices. For example, using LoRA on GPT-3 175B reduces the number of trainable parameters from 175 billion to 17 million, a 10,000 times reduction¹.</li><li>Due to the lower number of parameters, lower GPU memory is required, drastically reducing the cost incurred in finetuning a large model.</li><li>Avoid overfitting by keeping the original weights fixed and only updating a small fraction, preserving the generalization ability of the model while adapting it to new data.</li><li>Perform efficient task-switching by storing only one copy of the original weights and multiple sets of update matrices for different tasks or domains. This reduces the storage space and memory needed for deployment and allows you to switch between tasks without reloading the model.</li></ul><h3>Some business applications of LoRA</h3><p>The ability to finetune foundation models (like LLMs) with low compute requirements, at low cost and in a low-data regime opens up many opportunities for businesses looking to capitalize on AI and improve their processes. Some examples include:</p><ol><li><strong>Tune LLMs to perform specific natural language tasks like summarization/classification:</strong> With LoRA, businesses can modify large language models, such as GPT-3 or LLaMA, to achieve specific natural language processing tasks. These tasks could include text summarization or text classification. For instance, a news organization might use this to categorize news articles into predefined topics automatically or to generate short summaries of lengthy articles. The advantage of LoRA in this context is that businesses can adapt the models to their unique needs without the need for extensive resources.</li><li><strong>Domain Adaptation of Open Source LLMs:</strong> With LoRA, companies can finetune large language models to follow specific instructions, understand domain-specific jargon and adapt to a specific domain like law, finance, medicine or manufacturing, making them more capable of answering questions/performing tasks in the specific domain.</li><li><strong>Adapt large open-source ASR (Automatic Speech Recognition) models like Whisper to specific languages or accents:</strong> ASR technology is crucial in applications like voice assistants, transcription services, and more. However, these models often need help with different accents or languages less represented in the training data. Using LoRA, businesses can adapt large ASR models better to understand a broader range of languages or accents. This could significantly improve the usability of voice-enabled services for customers worldwide.</li><li><strong>Tune image generation models like Stable Diffusion to follow a particular style or structure:</strong> Image generation models have found use in digital art creation, design, and even in generating realistic images for training other machine learning models. However, getting these models to generate images following a specific style or structure can be challenging and resource-intensive. Using LoRA, businesses can finetune these models with less computational resources to create images following a particular manner, such as a specific art style for a digital game or distinctive product designs for a manufacturing company.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QIevncbz3d5_N7CK.png" /><figcaption>Some images generated by finetuning a stable diffusion model with LoRA (<a href="https://replicate.com/blog/lora-faster-fine-tuning-of-stable-diffusion">Source</a>).</figcaption></figure><p>These are just a few examples of how LoRA can be utilized in a business setting. It’s a versatile tool that can significantly reduce the computational resources required to adapt powerful models to specific tasks, enabling businesses of all sizes to take advantage of the benefits of AI.</p><h3>Conclusion</h3><blockquote>AI is the new electricity. Just as electricity transformed almost everything 100 years ago, today I actually have a hard time thinking of an industry that I don’t think AI will transform in the next several years. — <strong><em>Andrew Ng</em></strong></blockquote><p>As we stand on the brink of the AI revolution, businesses must recognize the power of AI and how it can transform their operations. With techniques like LoRA, even the largest models become accessible and affordable for adaptation. It allows businesses to harness the power of AI, just like electricity was harnessed a century ago, to light up their path to innovation and progress. This is not just an era of change, but a change of era.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3dcd6a1264bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml6team/harnessing-the-power-of-foundation-models-for-your-business-with-lora-3dcd6a1264bf">Harnessing the Power of Foundation Models for your Business with LoRA</a> was originally published in <a href="https://medium.com/ml6team">ML6team</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Low Rank Adaptation: A Technical Deep Dive]]></title>
            <link>https://medium.com/ml6team/low-rank-adaptation-a-technical-deep-dive-782dec995772?source=rss-b0852981290a------2</link>
            <guid isPermaLink="false">https://medium.com/p/782dec995772</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[lora]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[efficient]]></category>
            <dc:creator><![CDATA[Nikhil Nagaraj]]></dc:creator>
            <pubDate>Tue, 23 May 2023 14:48:06 GMT</pubDate>
            <atom:updated>2023-05-23T14:48:06.206Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JmBZ8Lwhe3UcfL84" /><figcaption>Photo by <a href="https://unsplash.com/@deepmind?utm_source=medium&amp;utm_medium=referral">Google DeepMind</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In today’s fast-paced technological landscape, large AI models are propelling breakthroughs across diverse domains. However, tailoring these models to specific tasks or datasets can be a computational and resource-intensive endeavor. Enter <strong>LoRA (Low Rank Adaptation) — a groundbreaking and efficient fine-tuning technique</strong> that harnesses the power of these advanced models for custom tasks and datasets without straining resources or incurring excessive costs.</p><p><strong>LoRA has taken the AI community by storm in recent months</strong> (Fig 1). In this blog post, we’ll delve into the reasons behind its meteoric rise. We’ll explore the principles underpinning LoRA, its effectiveness in various domains, and the impact it’s having on the open-source community.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CIvylICdcGILMXrPbqVP9Q.png" /><figcaption>Fig 1: Popularity of the term LoRA over the last 12 months within the Computer Science category. (<a href="https://trends.google.com/trends/explore?cat=1227&amp;date=2022-01-01%202023-04-08&amp;q=LoRA&amp;hl=en-GB">Source</a>)</figcaption></figure><p>Whether you’re an AI aficionado, an engineer seeking to capitalize on large models for your specific business challenge, join us on this captivating journey to discover how LoRA is <strong>transforming the fine-tuning step in the </strong><a href="https://foundationcapital.com/foundation-model-ops-powering-the-next-wave-of-generative-ai-apps/"><strong>FMOps (Foundation Model Ops)</strong></a><strong> pipeline for large AI models.</strong></p><h3>Background Concepts</h3><p>Before diving into LoRA, let’s review some fundamental linear algebra concepts. If you’re comfortable with the basics of linear algebra (particularly matrix rank), feel free to bypass the math below.</p><h3>Matrix Rank</h3><p>The <strong>rank of a matrix</strong> is the dimension of the vector space generated by its columns, which is given by the <strong>number of </strong><a href="https://en.wikipedia.org/wiki/Linear_independence"><strong>linearly independent</strong></a><strong> columns (or rows) in a given matrix.</strong> It can be proven that the number of independent columns (known as <em>column rank)</em> is always equal to the number of independent rows (called <em>row rank)</em>. Hence, for a matrix <strong><em>A</em></strong><em> </em>with <strong><em>m</em></strong><em> </em>rows and <strong><em>n</em></strong><em> </em>columns (represented as <strong><em>Aₘₙ</em></strong><em>)</em>,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*tNmst6mluR3AZjupaGk7MQ.png" /></figure><h3>Types of Matrices</h3><p>Based on its rank, a matrix can be primarily classified into two types.</p><p><strong>Full-Rank Matrix</strong></p><p>A matrix <strong><em>Aₘₙ</em></strong> is called a <strong>full-rank matrix</strong> if <strong><em>rank(A) =</em> <em>min(m, n)</em>.</strong> The matrix shown below is an example of a full rank matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*G0n6LfPtn4jjJFHUXFX58A.png" /></figure><p><strong>Rank-Deficient Matrix</strong></p><p>The opposite of a full rank matrix is <strong>rank deficient</strong> i.e. <strong><em>rank(A) </em>&lt;<em> min(m, n)</em></strong>. The rank-deficient matrix shown below has a rank of <strong><em>1</em></strong>, as the columns (or rows) of the matrix are not linearly independent of one another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*5gVbap0zv54mcfCWgZEqlA.png" /></figure><blockquote><strong><em>Low-Rank Matrix</em></strong><em>: A rank-deficient matrix </em><strong>Aₘₙ<em> </em></strong><em>is called a low-rank matrix if its rank is significantly lower (no fixed threshold) than the minimum number of rows and columns. Mathematically,</em> <strong>rank(A) &lt;&lt; min(m, n)</strong><em>.</em></blockquote><h3>Relevant Properties</h3><ol><li>As previously mentioned, the rank of a matrix is constrained by the minimum of its number of rows and columns.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*tNmst6mluR3AZjupaGk7MQ.png" /><figcaption>Prop 1: Rank of a matrix is constrained by the minimum of its number of rows and columns.</figcaption></figure><p>2. Given matrices <strong><em>A</em></strong> and <strong><em>B</em></strong> with <strong><em>rank(A) =</em> <em>m</em> </strong>and <strong><em>rank(A) =</em> <em>n</em></strong>, then</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*MNzwffHQ848V9TbFUfofjA.png" /><figcaption>Prop 2: Rank of the product of two matrices is constrained by the minimum of their individual ranks.</figcaption></figure><blockquote><strong><em>An intuitive understanding of Matrix Rank:</em></strong> <em>For the purposes of this blog, the rank of a matrix can be perceived as the dimensions of the feature space it represents. In this context, a low-rank matrix of a certain size encapsulated fewer features (or a lower dimensional feature space) than a full-rank matrix of the same dimensions.</em></blockquote><h3>Rank Decomposition</h3><p>Rank decomposition or factorization of a matrix <strong><em>Aₘₙ</em></strong> is the factorization of <strong><em>A </em></strong>of the form <strong><em>A</em></strong> <strong><em>= CₘᵣFᵣₙ</em></strong> where <strong><em>rank(A) =</em> <em>r</em></strong>. It can be proven that every (finite) matrix has a rank decomposition (<a href="https://en.wikipedia.org/wiki/Rank_factorization#Existence">proof</a>). Techniques like <a href="https://en.wikipedia.org/wiki/Singular_value_decomposition">SVD</a> (Singular Value Decomposition) can be used to construct such a decomposition.</p><p>With that, we’ve covered the necessary background concepts. Let’s dive right into LoRA and explore how it leverages these principles in the context of fine-tuning large AI models.</p><h3>LoRA: Low-Rank Adaptation of Large (Language) Models</h3><p><strong>LoRA</strong> is an efficient finetuning technique proposed by Microsoft researchers to adapt large models to specific tasks and datasets. While the <a href="https://arxiv.org/abs/2106.09685">paper</a> uses GPT-3 as the test case and focuses on language models and NLP tasks, this technique is quite generalizable, as we will see below. It can be applied to various models in multiple contexts.</p><h3>Hypothesis</h3><p>Many previous works have shown that over-parametrized large models reside on a low intrinsic dimension. The main idea behind LoRA is that the change in weights during model adaptation also has a <strong>low intrinsic rank/dimension</strong>. Concretely, if <strong><em>Wₙₖ</em></strong> represents the weights of a single layer and <strong><em>ΔWₙₖ</em></strong> represents the change of weights during model adaptation, the authors propose that <strong><em>ΔWₙₖ</em></strong> is a <strong>low-rank matrix</strong> i.e.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*XPJIyPQjQwQtkxnKejFS6w.png" /></figure><blockquote><strong><em>Why does this make sense?</em></strong></blockquote><blockquote><em>Large models are trained to capture the general representation of their domain (language for LLMs, audio + language for models like Whisper, and vision for image generation models). These models capture a variety of features which allow them to be used for diverse tasks with reasonable zero-shot accuracy. However, when adapting such a model to a specific task or dataset, only a few features need to be emphasized or re-learnt. This means that the update matrix (</em><strong>ΔW</strong><em>) can be a low-rank matrix.</em></blockquote><h3>Method</h3><p>The technique constrains the rank of the update matrix <strong><em>ΔW</em></strong> using its rank decomposition. It represents <strong><em>ΔWₙₖ</em></strong> as the product of 2 low-rank matrices <strong><em>Bₙᵣ</em></strong> and <strong><em>Aᵣₖ</em></strong> where <strong><em>r &lt;&lt; min(n, k)</em></strong>. This implies that the forward pass of the layer, originally <strong><em>Wx</em></strong>, is modified to <strong><em>Wx + BAx</em></strong> (as shown in the figure below). A random Gaussian initialization is used for <strong><em>A</em></strong> and <strong><em>B</em></strong> is initially to <strong><em>0</em></strong>, so <strong><em>BA=0</em></strong> at the start of training. The update <strong><em>BA</em></strong> is additionally scaled with a factor <strong>α/r</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*e5pYWjrZR3eA_YbCKu8deQ.png" /><figcaption>Fig 2: Modified forward pass using low-rank decomposition.</figcaption></figure><h3>Practical Benefits</h3><ol><li><strong>Reduction of training time and space</strong>: Using the technique shown above, <strong><em>r(n + k)</em></strong> parameters have to be tuned during model adaption. Since <strong><em>r &lt;&lt; min(n, k)</em></strong>, this is much lesser than the number of parameters that would have to be tuned otherwise (<strong><em>nk</em></strong>). This reduces the time and space required to finetune the model by a large margin. Some numbers from the paper and our experiments are discussed in the sections below.</li><li><strong>No additional inference time</strong>: If used in production, we can explicitly compute <strong><em>W’ = W + BA</em></strong> and store the results, performing inference as usual. This guarantees that we do not introduce any additional latency during inference.</li><li><strong>Easier task switching:</strong> Swapping only the LoRA weights as opposed to all the parameters allows cheaper and faster switching between tasks. Multiple customized models can be created and swapped in and out easily.</li></ol><p>However, the drawback here is that once the weights are merged to remove additional inference time, the ease of task switching vanishes. Also, it is not straightforward to batch inputs to different tasks with different A and B in a single forward pass. You win some and you lose some, right?</p><h3>Evaluating the effectiveness of LoRA</h3><p>Having discussed how the technique works and its possible benefits, let’s now explore its efficacy. In the paper, the authors evaluate the performance of LoRA (using RobBert, GPT2, and GPT3) against full finetuning and other parameter/compute efficient techniques. They find that LoRA generally <strong>outperforms other efficient finetuning techniques by a significant margin</strong> while also providing <strong>comparable or better performance than full finetuning</strong>. For complete details of their analysis, interested readers can refer to the <a href="https://arxiv.org/pdf/2106.09685.pdf#page=5">paper.</a></p><p>To further explore its effectiveness, we conducted additional experiments in various domains and tasks. In the following subsections, we discuss the results of these experiments, showcasing the versatility and robustness of the LoRA method.</p><h3>Finetuning Whisper-Large-v2 on Common Voice (NL)</h3><p><a href="https://openai.com/research/whisper">Whisper</a> is an ASR system which has been trained on a large corpus of data. It is a family of models, each of varying sizes. The smallest model (Whisper-Tiny) contains 39 million parameters whereas the largest model (Whisper-Large-v2 ) contains 1.5 billion parameters. The largest model can perform multilingual automatic speech recognition (ASR). However, its performance can be improved for a particular language by fine-tuning it with data from that language.</p><p>In this experiment, we finetune the model using the Dutch language subset of the <a href="https://commonvoice.mozilla.org/en/datasets">Common Voice Dataset.</a> We finetune the large model (with &amp; without LoRA) normally and in a low data regime. Finetuning the model using LoRA with r=32 (where r is the rank of the update matrix) <strong>reduces the number of tunable parameters to 15.7 million, which is 1% of the parameters of the entire model.</strong></p><h3>Low Data Regime: Using 1 hour of audio data</h3><p>We finetune Whisper-Large-v2 with and without LoRA on <strong>one hour of audio</strong> from the Dutch subset of Common Voice. The evaluation results are shown in the table below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8653eb37740637c0531c9e9d9cee883c/href">https://medium.com/media/8653eb37740637c0531c9e9d9cee883c/href</a></iframe><p>We see that the performance of the model finetuned using LoRA is similar to the performance of the fully finetuned model. However, as mentioned previously, LoRA allows us to <strong>accomplish this in a much shorter time</strong> by tuning a minuscule number of parameters (for size<strong>, the LoRA checkpoint is only 60MB</strong>). Using LoRA (and <a href="https://huggingface.co/blog/hf-bitsandbytes-integration">8bit optimization)</a>, the training <strong>took ~4 hours and cost &lt; </strong><strong>$5on an Nvidia T4 on Google Cloud.</strong></p><h3>Using a large dataset</h3><p>In this case, we finetuned the same model with the entire Dutch subset of Common Voice (~40 hours). The evaluation results are shown in the table below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/92bb5971d46b66befd3c4545b7c00cc9/href">https://medium.com/media/92bb5971d46b66befd3c4545b7c00cc9/href</a></iframe><p>We see that the performance of the large model finetuned with LoRA (for 5000 steps) is comparable to the performance of the fully finetuned large and medium models. However, using LoRA (and <a href="https://huggingface.co/blog/hf-bitsandbytes-integration">8bit optimization)</a>,<strong> the finetuning took ~10 hours and cost &lt; </strong><strong>$10 on an Nvidia T4</strong> on Google Cloud.</p><p>We’ve explored the world of Whisper and how it performs when finetuned with datasets of varying sizes in another blogpost. You can find it <a href="http://ml6.eu/">here</a>!</p><h3>Adapting LLaMA 🦙 to perform a dialogue summarization task</h3><p><a href="https://ai.facebook.com/blog/large-language-model-llama-meta-ai/">LLaMA</a> is a large language model released by the researchers at Meta. Like Whisper, it is a family of models with varying sizes (7B being the smallest and 65B the largest).</p><p>We finetuned the 7B parameter model on a dialogue summarization task using the <a href="https://paperswithcode.com/paper/samsum-corpus-a-human-annotated-dialogue-1">Samsum</a> dataset and used <a href="https://en.wikipedia.org/wiki/ROUGE_(metric)">ROUGE</a> to evaluate the finetuned model. To test the effectiveness of LoRA at low ranks, the rank of the update matrix was constrained to 4 i.e. r=4. <strong>This meant that the number of tunable parameters was 2 million, which is just 0.03% of the total number of model parameters.</strong></p><p>From the table, below we see that it outperforms a fully finetuned Flan-T5-Base model (250 million parameters). Additionally, using a low rank and <a href="https://huggingface.co/blog/hf-bitsandbytes-integration">8-bit optimization</a> allows us to finetune such a large model on a single Nvidia-T4!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dbf5f355494e1b45c31eba5299d3d622/href">https://medium.com/media/dbf5f355494e1b45c31eba5299d3d622/href</a></iframe><blockquote><strong><em>Note</em></strong>: <em>This isn’t a fair standalone comparison between models. LLaMA-7B is a much larger foundation model in comparison to Flan-T5-Base and therefore is probably capable of better zero-shot performance on many tasks. However, this comparison aims to demonstrate that for large foundation models, </em><strong><em>using a very low rank</em></strong><em> (and thus low compute and time for finetuning) </em><strong><em>suffices</em></strong><em>.</em></blockquote><p>The results from the paper and our experiments demonstrate LoRA’s effectiveness. LoRA provides a compute and parameter efficient method to <strong>finetune foundation models without a significant drop in the performance saving both time and money!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*jXzlVe4ZAJaQTKjK.jpg" /></figure><h3>LoRA in the Wild</h3><p>Let’s now look at how LoRA is being used in the open-source community. With the recent explosion of large foundation and generative AI models, the open-source community has welcomed LoRA with open arms due to its ability to allow <strong>low-resource practitioners to adapt large models</strong>. Here, LoRA is primarily being used for two major purposes: <strong>Instruct-tuning LLMs and Finetuning Diffusion models.</strong></p><h3>Instruct-tuning Large Language Models</h3><p>With the launch of ChatGPT and techniques like Self-Instruct, the OSS community has been steadily working on tuning large language models to follow instructions.</p><p>The core idea here is simple. Create a dataset of instructions and responses (either using manual curation or ChatGPT) and use LoRA to finetune a pre-trained large language model using this dataset. This method produces models that are <strong>reasonably adept at following instructions and answering questions like humans</strong>. Interested readers can check out models such as <a href="https://github.com/tloen/alpaca-lora">Alpaca-LoRA</a> and <a href="https://vicuna.lmsys.org/">Vicuna</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*OFvqeFGzoYzuJ8i7whE62A.gif" /><figcaption><a href="https://vicuna.lmsys.org/">Vicuna </a>responding to a user’s question regarding a holiday in Hawaii. (<a href="https://lmsys.org/blog/2023-03-30-vicuna/">Source</a>)</figcaption></figure><h3>Finetuning Stable Diffusion</h3><p>Before the launch of ChatGPT and other LLMs like LLaMA, <strong>LoRA was primarily used to tune stable diffusion</strong> to adapt the style of generated images. The LoRA weights can then be used and shared in a plug-and-play fashion switching them out when a different image generation style is necessary.</p><p>As seen before, the main draw of this technique <strong>is its parameter and compute efficiency.</strong> A testimony to the popularity of this method in the generative AI community is the existence of the 🤗 <a href="https://huggingface.co/lora-library">Lora Library</a> where people can share their <em>Lora files</em>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cHt3HICTcCuXkyJ7.png" /><figcaption>Some images generated by finetuning a stable diffusion model with LoRA (<a href="https://replicate.com/blog/lora-faster-fine-tuning-of-stable-diffusion">Source</a>)</figcaption></figure><h3>Conclusion</h3><p>To sum it all up: LoRA has two major applications. The first is to <strong>finetune large models with low compute</strong>, and the second is to <strong>adapt large models in a low-data regime</strong>. Results from the paper, our experiments and the widespread adoption by the open-source AI community demonstrate its value in the current foundation-model-driven AI environment.</p><p><strong>It democratizes AI, empowering individuals and organizations to use and tune large foundation models without breaking the bank, ensuring that the ability to adapt these models is not just in the hands of a select few!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=782dec995772" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml6team/low-rank-adaptation-a-technical-deep-dive-782dec995772">Low Rank Adaptation: A Technical Deep Dive</a> was originally published in <a href="https://medium.com/ml6team">ML6team</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[State of the LLM: Unlocking Business Potential with Large Language Models]]></title>
            <link>https://medium.com/ml6team/state-of-the-llm-unlocking-business-potential-with-large-language-models-87b442f3253d?source=rss-b0852981290a------2</link>
            <guid isPermaLink="false">https://medium.com/p/87b442f3253d</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[auto-gpt]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Nikhil Nagaraj]]></dc:creator>
            <pubDate>Wed, 17 May 2023 15:28:44 GMT</pubDate>
            <atom:updated>2023-06-14T08:18:15.389Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cvOfoF_nx6GCArWX" /><figcaption>Streams of data produced by large language models. Photo by <a href="https://unsplash.com/@deepmind?utm_source=medium&amp;utm_medium=referral">Google DeepMind</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Over the past few months, there has been a surge of interest in large language models (LLMs). These models, which use artificial intelligence to generate human-like language, have captured public interest like never before, thanks mainly to the release of OpenAI’s ChatGPT. This explosion of public interest has meant that other companies and groups have followed suit, either by releasing base models like Meta AI’s LLaMA 🦙 or instruction tuning previously released base models like Databricks’ Dolly 🐑 or BAIR’s Koala 🐨.</p><p>Now is the perfect time for businesses to explore the potential of LLMs and integrate them into their strategy. In this blog post, we will guide you through the <strong>current state of the LLM landscape in 4 parts: Models, Tooling, Challenges and Applications.</strong> We’ll also provide some tips and key takeaways that you can use to decide on how to use LLMs in your business strategy. Let’s dive in!</p><h3>Part 1: Models</h3><p>Most LLMs that are centerstage today are similar in terms of their architecture. However, what mainly <strong>distinguishes models are their size (number of parameters) and the quality and quantity of training data used</strong>. While GPT-3 has 175 billion parameters, models such as LLaMA are much smaller, with versions ranging from 7 billion to 65 billion parameters.</p><blockquote><strong><em>Tip:</em></strong><em> It has been found that data quality is much more important than data quantity. If you are exploring an LLM use case for your business, </em><strong><em>optimizing for data quality will pay better dividends</em></strong> <strong><em>than increasing the amount of data.</em></strong></blockquote><p>However, the difference between base models like GPT-3/LLaMA and their instruction following variants like ChatGPT/Vicuna is an additional training step. This step can use human feedback (called Reinforcement Learning with Human Feedback (RLHF)) or a supervised dataset of instructions and their corresponding responses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jiPnMtfMns_B3ENB-GGqSQ.png" /><figcaption>Using RLHF to train ChatGPT (Source: <a href="https://openai.com/blog/chatgpt">OpenAI</a>)</figcaption></figure><p>Based on current wisdom, RLHF is the better option to train an LLM to follow instructions and respond in a human-like manner. However, why this is so has yet to be fully understood. We recommend John Schulman’s <a href="https://www.youtube.com/watch?v=hhiLw5Q_UFg"><strong>talk</strong></a> for a better understanding on the advantages of RLHF.</p><p>Currently, the best instruction-following model that’s available for public use is ChatGPT. At 175 billion parameters and trained with RLHF, it towers above the currently available open-source models. Most open-source instruction-following models like Alpaca 🦙, Vicuna, Dolly 🐑, and Koala🐨 are much smaller and tuned using a supervised dataset containing data generated by a technique called Self-Instruct or by ChatGPT. Efforts like <a href="https://open-assistant.io/">Open Assistant</a> should help create comparable open-source alternatives soon.</p><blockquote><strong><em>Key Takeaway:</em></strong><em> While ChatGPT currently leads the market in terms of performance, </em><strong><em>open-source models are steadily improving and are viable options for specific use cases. </em></strong><em>Keeping an eye on the development of both proprietary and open-source LLMs will ensure that your business can make the best choice.</em></blockquote><h3>Part 2: Tooling</h3><p>New techniques and models have always led to the creation of new tools to improve their ease of use. LLMs are no different! In this section, we’ll explore two emerging tools that have been developed to address the unique challenges posed by LLMs: PEFT (Parameter Efficient Finetuning) — a collection of techniques that helps finetune large models without breaking the bank, and Langchain — a framework for developing applications powered by language models.</p><h4>🔧PEFT: Parameter Efficient Finetuning</h4><p>One of the critical challenges in the LLM landscape is finetuning these models without incurring prohibitive costs. Since these models have billions of parameters, adapting them to custom tasks or datasets can be expensive.</p><p>Over the last few years, <strong>efficient finetuning techniques like LoRA, Prefix tuning, and Prompt tuning have made LLMs more accessible for practitioners with limited resources with comparable or better performance at a small fraction of the cost. </strong>You can find a technical deep dive on the popular efficient finetuning technique LoRA, <a href="https://medium.com/p/782dec995772">here</a>.</p><blockquote><strong><em>Tip:</em></strong><em> Hugging Face’s recently released </em><a href="https://github.com/huggingface/peft"><em>PEFT library</em></a><em> contains all these techniques and more. With its native integration with the 🤗Transformers and Accelerate libraries along with the ability to use other optimization libraries like </em><a href="https://github.com/TimDettmers/bitsandbytes"><em>bitsandbytes</em></a><em>, developers can easily finetune large language models on specific tasks using less computing power and, thus, affordably.</em></blockquote><h4>🔗 Langchain: Building Composable LLM Applications</h4><p>The more data an LLM can access, the more useful it will be. However, creating an LLM application with multiple components communicating with various data sources can take time and effort, and this is where Langchain finds its niche.</p><p><a href="https://python.langchain.com/en/latest/index.html"><strong>Langchain</strong></a><strong> is a framework for building composable LLM applications.</strong> Available in both Python and TypeScript, the Langchain library offers a plethora of tooling and third-party integrations (major cloud providers, Google Drive, Notion and more!) to build powerful applications (‘chains’) driven by LLMs. It also provides <strong>pre-built chains capable of some of the most common LLM applications</strong> like Retrieval Augmented Generation (RAG) or chat.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pKCF0CyqNb00UJ8N" /><figcaption>With more than 35,000 stars on Github, Langchain has proven extremely popular in the community. (Source: <a href="https://github.com/hwchase17/langchain">Github</a>)</figcaption></figure><p>From our experience, Langchain offers some nifty components that can be used to implement quick proof of concepts and simple applications to incorporate LLMs into your business or daily life. However, <strong>one might find it limiting while working on custom use cases, especially in production where greater control and flexibility are necessary.</strong></p><blockquote><strong><em>Key Takeaway:</em></strong><em> The emergence of powerful techniques and tools like LoRA, PEFT and Langchain represents a game-changing opportunity for businesses to leverage the capabilities of LLMs fully. </em><strong><em>Finetuning an LLM on a specific task/dataset is now more accessible. </em></strong><em>Tools like </em><strong><em>Langchain</em></strong><em> are improving the </em><strong><em>ease with which these models can interact with multiple external sources of information.</em></strong></blockquote><h3>Part 3: Challenges</h3><p>While the potential is immense, it is essential to recognize that harnessing the full capabilities of LLMs comes with challenges. This section will focus on <strong>two primary hurdles: LLMOps and Model Bias.</strong> By understanding these challenges and how to overcome them, businesses can effectively capitalize on the opportunities offered by LLMs and stay ahead of the competition.</p><h4>🏗️LLMOps: Scaling and Managing LLM Infrastructure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eXwNAfnfdXorip1M.png" /><figcaption>The standard MLOps pipeline which has to be adapted to deal with the new challenges posed by LLMs. (Source: <a href="https://martinfowler.com/articles/cd4ml.html">Martin Fowler</a>)</figcaption></figure><p>As LLMs grow in size and complexity, the infrastructure required to train, finetune, and deploy them becomes increasingly demanding, introducing the challenge of <strong>LLMOps — a specialized subdomain of MLOps</strong> that focuses on the unique operational aspects of working with large language models.</p><p>Some of the critical challenges in LLMOps include the following:</p><ul><li>Working with the ambiguities and intricacies of natural language</li><li>Optimizing the costs and latency of operating LLMs in production</li><li>Establishing a robust model re-training and monitoring setup</li></ul><p>For a deep dive into the specific challenges of using LLMs in production, we highly recommend this <a href="https://huyenchip.com/2023/04/11/llm-engineering.html">blog</a> by Chip Huyen. While new tools will emerge to solve these specific challenges, <strong>currently, businesses might have to build bespoke solutions for their particular use cases.</strong> In these situations, leveraging the experience of ML experts might prove invaluable.</p><h4>🕊️Model Bias and Ethics: Ensuring Fair and Responsible AI</h4><p>Another critical challenge in the world of LLMs is addressing model bias and ensuring ethical AI practices. Since LLMs are trained on vast amounts of data from the internet, <strong>they can inadvertently learn and propagate biases present in the data.</strong> This can lead to unintended consequences when deploying LLMs in real-world applications.</p><p>To address these concerns, one should:</p><ul><li>Implement guidelines for responsible AI usage that consider both the intended and unintended consequences of deploying LLMs.</li><li>Use tools and techniques to monitor and mitigate biases in the models during the training and finetuning processes.</li><li>Collaborate with diverse teams and stakeholders to ensure a well-rounded perspective on potential ethical issues.</li><li>To the extent possible, establish safeguards against unintended use of LLMs by the end user.</li></ul><p>By taking these steps and informing their end users about the limitations of LLMs in the context of their application, organizations can leverage the models responsibly and build trust with customers, regulators, and the broader public.</p><blockquote><strong><em>Key Takeaway:</em></strong><em> The two significant challenges businesses must overcome to harness the full capabilities of LLMs are LLMOps and Model Bias.</em><strong><em> </em></strong><em>While there will be general solutions to </em><strong><em>the challenge of LLMOps</em></strong><em> in the future</em><strong><em>, for now, bespoke solutions are the way to go. MLOps experts are invaluable here.</em></strong><em> Solving </em><strong><em>model bias</em></strong><em> is quite tricky. However,</em><strong><em> testing, safeguards and transparency </em></strong><em>can ensure the best outcomes.</em></blockquote><h3>Part 4: Applications</h3><p>The potential applications of LLMs are vast and rapidly growing. We’re in the early stages of this technology transforming industries as a whole, and <strong>businesses that take the first step might find themselves way ahead of the competition in the future.</strong></p><p>In this section, we will explore some of the most promising applications of LLMs, including Retrieval Augmented Generation (RAG), NLP tasks like summarization and extracting structured data, and customer support. Finally, we’ll take a brief look at process automation using AutoGPT.</p><h4>📜Retrieval Augmented Generation (RAG)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0GUWyWdo4Y-Uxxlv" /><figcaption>A schematic of a Retrieval Augmented Generation system. (Source: <a href="https://mattboegner.com/knowledge-retrieval-architecture-for-llms/">Matt Boegner</a>)</figcaption></figure><p>Retrieval Augmented Generation is a technique that <strong>combines the strengths of LLMs with the power of information retrieval systems.</strong> By using RAG, LLMs can <strong>access external knowledge sources to generate current, informed and accurate responses</strong>, making them even more valuable in a wide range of applications. <em>For example, a RAG system implemented on internal documentation and resources would prove invaluable as it would tremendously boost productivity by putting the entire company’s resources at their employee’s fingertips.</em></p><p>You can find our deep dive on Retrieval Augmented Generation systems <a href="https://blog.ml6.eu/leveraging-llms-on-your-domain-specific-knowledge-base-4441c8837b47">here</a>!</p><h4>✅NLP Tasks: Summarization and Extracting Structured Data</h4><p>LLMs excel in various natural language processing tasks, enabling businesses to streamline their workflows and make better use of unstructured data. For instance, LLMs can be used for:</p><ul><li><strong>Summarization:</strong> Automatically generating concise summaries of lengthy documents, such as reports or articles, saving time and effort for employees and enhancing decision-making processes. <em>For example, a financial firm could use LLMs to generate executive summaries of lengthy market research reports, providing analysts with easily digestible insights and facilitating more informed decisions.</em></li><li><strong>Extracting Structured Data:</strong> Transforming unstructured data, like customer feedback or news articles, into structured formats that can be easily analyzed and used for further processing, such as sentiment analysis or trend detection. <em>For example, a retail company could leverage LLMs to extract key information from customer reviews, enabling the business to identify product trends and common issues and ultimately improve the customer experience.</em></li></ul><h4>📞Customer Support</h4><p>The most obvious use of an application like ChatGPT, LLMs can revolutionize customer support by providing fast, accurate, and personalized responses to customer inquiries. By integrating LLMs into chatbots or helpdesk systems,<strong> businesses can significantly improve customer satisfaction and reduce response times, freeing human agents to focus on more complex issues requiring their expertise.</strong></p><h4>🤖Workflow Automation with AutoGPT</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/0*kPeswmSqo6ZeKGFc" /><figcaption>AutoGPT (Source: <a href="https://www.lesswrong.com/posts/566kBoPi76t8KAkoD/on-autogpt#The_Basics">Less Wrong</a>)</figcaption></figure><p>The previously mentioned use cases are some of the more tried and tested applications that we see being used across industries and domains. Workflow automation with AutoGPT is relatively new. One can find anecdotes of this application at young, small companies all over Twitter and LinkedIn, but rigorous testing and stakeholder involvement is necessary before using it at a large scale in your business.</p><p><strong>AutoGPT is an experimental technique that uses LLMs to create autonomous agents which fulfil various tasks based on natural language inputs. </strong>While still in the early stages of development, AutoGPT shows great promise! One can automate operations, sales and HR workflows using such autonomous agents to state some examples. <em>For example, one can set up AutoGPT to generate new sales leads, which includes finding the email-id of a contact at a prospective client, adding them to the CRM software and then sending them an email.</em></p><p>You can find an in depth dive into the world of AutoGPT in our blog <a href="https://medium.com/@jens.bontinck/my-authentic-opinion-on-the-autonomous-anatomy-of-autogpt-22b2749ba02a">here</a>!</p><blockquote><strong><em>Key Takeaway:</em></strong><em> Applications such as RAG, summarization, and customer support are the bread and butter of LLMs. They have repeatedly been proven to be adept at such applications, even when they were less powerful than they are right now. </em><strong><em>With the new generation of large language models, these applications and more will be accurate, autonomous and even more useful for organizations and their customers alike. Businesses are only limited by the imagination of their decision-makers.</em></strong></blockquote><h3>Conclusion</h3><p>Throughout this blog, we have explored the current state of the LLM landscape, delving into the architecture, techniques, challenges, and applications of these powerful models. With the rapid advancements in LLM technology and the development of new tools and solutions, it has become increasingly clear that there has never been a better time for businesses to invest in LLMs.</p><p>LLMs offer an <strong>unprecedented opportunity for businesses to innovate, streamline processes, and improve customer experiences, providing a competitive edge that will set them apart in their respective industries</strong>. From Retrieval Augmented Generation to summarization, customer support, and even experimental applications like AutoGPT, LLMs are reshaping the landscape of business operations and applications.</p><p>However, this opportunity also comes with challenges like LLMOps and model bias. By <strong>partnering with experts in the field, businesses can navigate these complexities and responsibly harness the full potential of LLMs, driving innovation and creating value for their organizations.</strong></p><p>These are early days in the new LLM world, but many businesses are already taking the first step and integrating this technology into their workflow. By investing in LLM expertise and adopting this groundbreaking technology, businesses can unlock unparalleled potential and stay ahead in the ever-evolving competitive landscape.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87b442f3253d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml6team/state-of-the-llm-unlocking-business-potential-with-large-language-models-87b442f3253d">State of the LLM: Unlocking Business Potential with Large Language Models</a> was originally published in <a href="https://medium.com/ml6team">ML6team</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Article Summarization: A building blocks approach]]></title>
            <link>https://medium.com/ml6team/multi-article-summarization-a-building-blocks-approach-d197b6e43a4?source=rss-b0852981290a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d197b6e43a4</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Nikhil Nagaraj]]></dc:creator>
            <pubDate>Tue, 31 Aug 2021 13:21:34 GMT</pubDate>
            <atom:updated>2021-08-31T13:21:34.150Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ycjaPKWBg-3KVNB5kCYng.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@ann-h-45017?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Ann H</a> from <a href="https://www.pexels.com/photo/handwritten-texture-writing-display-6980523/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p>In today’s age, time is scarce and information is abundant.</p><p>Have you ever started reading an article about a certain topic in a news portal and realized you’d read a similar piece on another portal earlier that day? Ain’t nobody got time to read a bunch of articles on the same topic and piece it together ⌚.</p><p>That’s why we came up with a way to combine several news sources on a particular topic and extract an informative summary with our ‘Multi-Article Summarization Service’ 💥!</p><p>If you cannot wait to try out the service, here’s the link: <a href="https://newssummarizer.ml6.eu/">news summarizer.ml6.eu</a>.</p><h4>How does this work?</h4><p>Recent advances in multi-document summarization have focused on using a transformer as a black box, ingesting multiple articles, and extracting a summary¹. While these techniques can extract concise summaries, the process itself is abstracted away from the end-user.</p><p>In our approach, we try to partially eliminate the black-box and make the summarization process more accessible to the user.</p><p>Our summarization pipeline relies on the idea that sentences are the basic building blocks of any article and involves a two-step process:</p><ol><li>Sentence Clustering</li><li>Cluster Summarization</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vh55k5xU-Gl0ALRfd38ArA.png" /><figcaption>Visualization of the 2-step summarization process. Sentences from the source articles are clustered and each cluster is then summarized. All the cluster summaries put together constitute the summary of the given set of articles.</figcaption></figure><h3>Sentence Clustering</h3><p>We implement two different approaches to cluster sentences from the source articles.</p><ol><li>Vanilla Sentence Clustering</li><li>Syntactic Sentence Clustering (A modified implementation of SummPip²)</li></ol><h4>Vanilla Sentence Clustering</h4><p>In this approach, the articles are first split into their constituent sentences with the <a href="https://www.nltk.org/api/nltk.tokenize.html#module-nltk.tokenize.punkt"><em>Punkt sentence tokenizer</em></a>. Using <a href="https://huggingface.co/sentence-transformers">Sentence-BERT</a>³, we then generate sentence embeddings for each sentence.</p><p>The embeddings are then clustered using Hierarchical/Agglomerative clustering to create information clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fyhVp2nrmhWvb0TsQ0Qp8g.png" /><figcaption>Vanilla Sentence Clustering using SBERT. Sentence-BERT is used to generate the embedding of each sentence and these embeddings are then clustered with the hierarchical clustering algorithm.</figcaption></figure><h4>Syntactic Clustering</h4><p>Along with the idea mentioned above, we also replicate the clustering methodology proposed in SummPip² albeit with a few modifications.</p><p>In this method, a sentence graph is constructed using both linguistic knowledge and neural representations of a sentence.</p><p>The sentence graph is then clustered using Spectral Clustering. While the authors use averaged word-embeddings to represent sentences, we use embeddings extracted from Sentence-BERT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jGxEy_7HP42eNmVp8xrOLQ.png" /><figcaption>Sentence Graph Construction and Clustering — SummPip²</figcaption></figure><h3>Cluster Summarization</h3><p>Since the source articles are on the same subject, the information clusters from the previous step will contain redundant information which can be safely removed.</p><p>To this end, we summarize the information clusters with BART⁴. Specifically, we use the model <a href="https://huggingface.co/facebook/bart-large-cnn"><em>facebook/bart-large-cnn</em></a><em> </em>from the Hugging Face 🤗 transformers library.</p><p>With this, each information cluster now has its summary snippet which avoids redundancy while still retaining the knowledge originally present across the multiple articles.</p><p>These summary snippets together represent the information summary of the given set of articles.</p><h3>Brief Analysis 👀</h3><p>The following section analyzes the approach and lists some of its drawbacks and strengths.</p><h4>Lack of a well-formed summary</h4><p>This might be already clear to the reader with a keen eye for details, but the proposed methodology does not produce a well-formed summary. It generates information snippets that can be appended together to obtain a summary of the source articles. Nevertheless, the generated summary is a comprehensive collection of information from the source articles.</p><p>An example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab0896a9b816fe8b186032fd5a5e0c43/href">https://medium.com/media/ab0896a9b816fe8b186032fd5a5e0c43/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a129416ce1781d8a02618ab73dffe64a/href">https://medium.com/media/a129416ce1781d8a02618ab73dffe64a/href</a></iframe><h4>Overall Performance</h4><p>In terms of the performance, with some extensive qualitative testing, we’ve observed that the method works well in terms of removing redundant/repeated information while still retaining key details from the source articles.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/20e10ec99e81777edecb129dfbbe1151/href">https://medium.com/media/20e10ec99e81777edecb129dfbbe1151/href</a></iframe><h4>Vanilla Clustering vs SummPip</h4><p>We observe that using the clustering approach proposed in SummPip² along with our modifications leads to highly imbalanced and broad clusters. This, we hypothesize is due to the nature of the clustering process weighs syntactic and semantic relationships between the sentences equally.</p><p>Vanilla clustering using SBERT on the other hand leads to well-balanced, compact clusters.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9781915e99c26caaeeab45572252ed90/href">https://medium.com/media/9781915e99c26caaeeab45572252ed90/href</a></iframe><h4>Spurious text generation</h4><p>Unfortunately, there are times where BART generates text during the summarization step which might not be in sync with the information provided in the source articles 👎.</p><p>On the flip side 👍, however, this phenomenon can be largely eliminated if the clusters have at least 2–3 unique sentences.</p><h3><strong>Conclusion</strong></h3><p>The multi-step sentence-based multi-article summarization pipeline presented here provides a comparable alternative to a black-box transformer-based multi-article summarization setup.</p><p>It is to be noted that this is an initial proof-of-concept that can be refined to produce even higher quality, more explainable, and well-formed summaries.</p><p>If you have not already done so, do check out the service at <a href="https://newssummarizer.ml6.eu/">news summarizer.ml6.eu</a>. Here’s a small GIF on how to use it!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PxMyIe_KuC45Zykq0O3gHQ.gif" /><figcaption>Using the summarization service — Check out <a href="https://newssummarizer.ml6.eu/">newssummarizer.ml6.eu</a>!</figcaption></figure><h4>References</h4><ol><li><a href="https://paperswithcode.com/paper/hierarchical-transformers-for-multi-document">Hierarchical Transformers for Multi-Document Summarization | Papers With Code</a> , <a href="https://paperswithcode.com/paper/leveraging-graph-to-improve-abstractive-multi">Leveraging Graph to Improve Abstractive Multi-Document Summarization | Papers With Code</a></li><li><a href="https://paperswithcode.com/paper/summpip-unsupervised-multi-document">SummPip: Unsupervised Multi-Document Summarization with Sentence Graph Compression | Papers With Code</a></li><li><a href="https://paperswithcode.com/paper/sentence-bert-sentence-embeddings-using">Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks | Papers With Code</a></li><li><a href="https://paperswithcode.com/method/bart">BART Explained | Papers With Code</a></li></ol><h3>About ML6</h3><p>We are a team of AI experts and the fastest-growing AI company in Belgium. With offices in Ghent, Amsterdam, Berlin, and London, we build and implement self-learning systems across different sectors to help our clients operate more efficiently. We do this by staying on top of research, innovation, and applying our expertise in practice. To find out more, please visit <a href="http://www.ml6.eu/">www.ml6.eu</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d197b6e43a4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml6team/multi-article-summarization-a-building-blocks-approach-d197b6e43a4">Multi-Article Summarization: A building blocks approach</a> was originally published in <a href="https://medium.com/ml6team">ML6team</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Face Anonymization: A survey of what works and what doesn’t]]></title>
            <link>https://medium.com/ml6team/face-anonymization-a-comparison-66da5088d030?source=rss-b0852981290a------2</link>
            <guid isPermaLink="false">https://medium.com/p/66da5088d030</guid>
            <category><![CDATA[identity-protection]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[gdpr]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Nikhil Nagaraj]]></dc:creator>
            <pubDate>Wed, 30 Sep 2020 07:33:26 GMT</pubDate>
            <atom:updated>2020-10-07T08:23:04.961Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring the effectiveness of recently released face anonymization algorithms</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fqgC5s73JMCgzjrakpqoEw.jpeg" /></figure><p>From schools to shopping centers, closed-circuit and monitoring cameras are now found everywhere. With their rising prevalence, concerns about individual privacy and data protection have risen. Even if these cameras do not use facial-recognition technology, the current reality is that wherever there are security cameras, in large cities, buildings, or other locations, there are people looking at these images irrespective of GDPR or other regulations. Watching this footage invades the privacy of its subjects and the information so obtained is only protected by the discretion of the observer. Anonymization of these images would improve compliance with data protection regulations while also increasing public confidence in such monitoring systems.</p><p>A major step in any endeavor to protect the privacy of individuals is the removal of personally identifiable information of which facial information is a large part. While simple techniques such as blurring/obfuscation of detected faces serve this purpose, there is a risk of the identity being revealed if the face detection algorithm fails for a few frames. A technique to mitigate this is by replacing the real faces with faces that are artificially generated. If the real face is not replaced for a few frames, this will be less obvious when watching the video as compared to a momentary lapse of blurring/obfuscation.</p><p>This blog explores two recently released facial anonymization techniques: <a href="https://arxiv.org/abs/1909.04538">DeepPrivacy</a> and <a href="https://www.mdpi.com/2076-3417/10/3/1120">CLEANIR</a> and expounds on their strengths and weaknesses. <em>(Want to jump directly to code?➡ </em><a href="https://colab.research.google.com/drive/1tlyYz_WvJhZktxpB0wI1Y4SJwOSyfvSF?usp=sharing"><em>Colab Notebook</em></a><em>)</em></p><h3>DeepPrivacy: A Generative Adversarial Network for Face Anonymization</h3><p>DeepPrivacy proposes a conditional generative adversarial network (CGAN) to anonymize faces in a given image. The model considers the original pose and background and aims to generate realistic faces that fit seamlessly into the original image.</p><h3>A brief overview of Generative Adversarial Networks</h3><p>Before delving into the architecture proposed by DeepPrivacy, this section takes a brief look at generative adversarial networks and their conditional counterparts.</p><h4>Generative Adversarial Networks (GANs)</h4><p>Based on the concept of a <a href="https://en.wikipedia.org/wiki/Zero-sum_game">zero-sum game</a> and used in generative modeling, a GAN consists of two networks that compete against each other. One of these networks is the <strong>generator</strong>, which aims to generate data as similar as possible to the real data distribution. The other is the <strong>discriminator</strong>, which aims to distinguish between the real data distribution and the distribution of the generated samples. Both these networks are trained in an adversarial manner, alternating between optimizing the generator and the discriminator. Ideally, training stops when the discriminator is no longer able to distinguish between the real samples and their generated counterparts.</p><h4>Conditional Generative Adversarial Networks (CGANs)</h4><p>In a CGAN, the <strong>generator/discriminator </strong>must generate/discriminate based on certain auxiliary conditions that are fed to the network. For example, a CGAN might be required to generate images of handwritten numbers in accordance with a number that is given as its conditional input. The discriminator on the other hand must additionally check if the number in the real/generated image matches the condition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*fCdAETXjcYyJlFEa9pOsQQ.png" /><figcaption>An overview of a CGAN where z represents the latent vector and y represents the condition. The generator and discriminator generate/discriminate samples per the condition ‘y’. [Source: <a href="https://arxiv.org/pdf/1411.1784.pdf">Mirza et al.</a>]</figcaption></figure><h3>Proposed Model</h3><p>DeepPrivacy proposes a CGAN, which generates images based on the surroundings of the face and sparse pose information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fyEd5Zvh8nkjeRHET3ObiA.png" /><figcaption>The overall model of DeepPrivacy. The face detection module detects faces while a pose estimation module generates sparse pose information. The pose information and the image (with the face obfuscated) are fed to a generative network. (Source: <a href="https://github.com/hukkelas/DeepPrivacy">DeepPrivacy</a>)</figcaption></figure><p>The official implementation of the model proposes the usage of the <a href="https://arxiv.org/abs/1810.10220">dual shot face detector (DSFD)</a> to detect faces in the given image. A <a href="https://arxiv.org/abs/1703.06870">Mask RCNN</a> is used to estimate seven keypoints to describe the pose of the face: left/right eye, left/right ear, left/right shoulder, and nose. The detected face is then obfuscated and the resulting image along with the pose information is fed to the generative network which has a <a href="https://en.wikipedia.org/wiki/U-Net">U-Net</a> architecture and employs <a href="https://arxiv.org/abs/1710.10196">progressive growing</a> during the training process.</p><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ITPgXeLsuf6xmSKXTiyqqA.png" /><figcaption>Results obtained using DeepPrivacy. The left image in each pair is the original image while the image on the right is its anonymized counterpart. In the original image(s), the red bounding box indicates the face detected by the DSFD. The red points are the facial keypoints as detected by the Mask RCNN.</figcaption></figure><p>DeepPrivacy is capable of generating relatively good quality anonymized faces according to the background and pose information. But the anonymization process is non-deterministic and hence there isn’t any consistency in the anonymized faces even if the original face is the same. This is evident in the video below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F4C3OTdURsjY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D4C3OTdURsjY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F4C3OTdURsjY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ec2c3f373836c73f735fa1e486b3fabf/href">https://medium.com/media/ec2c3f373836c73f735fa1e486b3fabf/href</a></iframe><p>DeepPrivacy aims to generate an anonymized face without any emphasis on preserving auxiliary non-identifiable information that can be gleaned from the face. CLEANIR aims to bridge this shortcoming.</p><h3>CLEANIR: Controllable Attribute-Preserving Natural Identity Remover</h3><p>CLEANIR proposes a model to anonymize faces while preserving non-privacy violating information. The model based on a variational autoencoder aims to modify the facial identity to a completely new one while the other attributes that are loosely related to personal identity are preserved.</p><h3>A brief overview of Variational Autoencoders</h3><p>Before delving into the architecture proposed by CLEANIR, this section takes a brief look at the basic concepts involved in a variational autoencoder.</p><h4>Autoencoder</h4><p>An autoencoder is used to learn <a href="https://en.wikipedia.org/wiki/Feature_learning">efficient data codings</a> in an <a href="https://en.wikipedia.org/wiki/Unsupervised_learning">unsupervised</a> manner. It essentially distills the input data down to its most discriminative/representative features. Generally, the autoencoder aims to be able to reproduce input data from its representation vector (latent vector). This leads to a phenomenon where the latent vectors for different inputs are quite different (mathematically distant) from each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*eBwuV27-bzBH0z0QLNKbVA@2x.png" /><figcaption>An autoencoder with its loss function. The loss function only involves a reconstruction term (making the encoding decoding scheme efficient). [Source: <a href="https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73">Joseph Rocca</a>]</figcaption></figure><p><strong>Variational Autoencoder</strong></p><p>A variational autoencoder (VAE) is an autoencoder with an additional goal of compactness in the latent space <em>(The space containing all the latent vectors is known as the latent space). </em>This allows the latent space to be sampled to generate new samples that are similar to real data points. A more detailed explanation on VAEs and the math behind them can be found <a href="https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iEQWnt4hVih7fzc8.png" /><figcaption>In variational autoencoders, the loss function is composed of a reconstruction term (making the encoding-decoding scheme efficient) and a regularisation term (making the latent space regular). [Source: <a href="https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73">Joseph Rocca</a>]</figcaption></figure><h3>Proposed Model</h3><p>CLEANIR proposes a VAE based architecture to generate anonymized faces while preserving information loosely related to the subject’s identity. It aims to preserve information such as skin color and the emotion(s) portrayed. The network is trained to separate the latent vector into features related to the subject’s identity and other features which are only loosely related to the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*7BMaBX5Z2leg4Id18FEbuQ.png" /><figcaption>Overview of the training and testing process for CLEANIR. During the testing process, the input to the network is the cropped face from the original image, the output is the anonymized image. (Source: <a href="https://www.mdpi.com/2076-3417/10/3/1120">CLEANIR</a>)</figcaption></figure><p>During inference, the part of the latent vector which is related to the person’s identity is transformed and fed together with the auxiliary feature vector to the decoder to produce a face that has been anonymized but still preserves general characteristics of the face.</p><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*yeI8TPxa_tNqtoEx356OOQ.png" /><figcaption>Results obtained using CLEANIR. From the anonymized images (right), it is evident that CLEANIR can preserve general information about the face while also anonymizing the face to protect personal identity.</figcaption></figure><p>CLEANIR generates high quality anonymized faces while preserving the general information about the person. Due to the nature of the model involved, the anonymization process can produce similar anonymized faces for the same original image. This ensures greater temporal consistency when used to anonymize faces in videos.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fe9B9mIvX4g8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3De9B9mIvX4g8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fe9B9mIvX4g8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/28ada941fe36699194855ff7fcca0cf4/href">https://medium.com/media/28ada941fe36699194855ff7fcca0cf4/href</a></iframe><p>Since CLEANIR only generates a cropped face, the anonymized image might be quite jarring if smoothing techniques are not used to merge the original image with the new anonymized face.</p><h3>Comparison: CLEANIR and DeepPrivacy</h3><p>After expounding on the techniques involved in both CLEANIR and DeepPrivacy, this section provides a comparison concerning the strengths and weaknesses of both techniques.</p><h4><strong>Anonymization Quality</strong></h4><p>Both models generate an anonymized face which cannot be visually matched with the original face. However, when quantitatively tested on the <a href="http://vis-www.cs.umass.edu/lfw/">Labelled Faces in the Wild </a>dataset using <a href="https://github.com/nyoki-mtl/keras-facenet">Facenet </a>to generate embeddings, CLEANIR seems to marginally outperform DeepPrivacy in terms of the distance between an anonymized face and the original face. This is probably a result of CLEANIR’s training methodology, which uses Facenet as a part of its training pipeline.</p><p>In the case of videos, CLEANIR is capable of producing better temporal consistency in terms of the anonymized faces.</p><h4><strong>Inference Rate</strong></h4><p>The bottleneck for both CLEANIR and DeepPrivacy in the anonymization pipeline is face/keypoint detection. The actual anonymization process is quite fast in comparison.</p><h4>Impact of a complex and noisy background</h4><p>Due to its GAN based approach in generating anonymized images, DeepPrivacy is more susceptible to the ill-effects caused by a noisy/complex background. However, post-processing which replaces only the original face with its anonymized version can reduce any such impact.</p><h4>A note on the face detection algorithm</h4><p>Both CLEANIR and DeepPrivacy are limited in their anonymization capabilities by the quality of their face detectors. Spurious face detections can cause a large-scale distortion in the anonymized image esp. in the case of DeepPrivacy.</p><blockquote>If no face is detected, no face is anonymized.</blockquote><h3>Conclusion</h3><p>Facial anonymization is a topic that’s seeing a lot of activity these days. With the increasing demand for greater privacy protections and adherence to stricter regulations, the amount of buzz and research in this area is sure to increase. Both DeepPrivacy and CLEANIR are excellent attempts each with its strengths and weaknesses. Future work should possibly involve trying to generate hyper-realistic anonymized images consistently and accurately with the same face being anonymized similarly. It would also be interesting to try anonymizing segmented faces instead of using bounding boxes.</p><p>The code to generate the images and videos seen above is available in this <a href="https://colab.research.google.com/drive/1tlyYz_WvJhZktxpB0wI1Y4SJwOSyfvSF?usp=sharing">Google Colab notebook</a>. As a small reward for reading through this blog, here’s a video comparing the performance of both the models.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FTpuS3RiTbGI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DTpuS3RiTbGI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FTpuS3RiTbGI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa5a05174ad9203d40d7ce04ee1236ff/href">https://medium.com/media/fa5a05174ad9203d40d7ce04ee1236ff/href</a></iframe><h3>About ML6</h3><p>We are a team of AI experts and the fastest-growing AI company in Belgium. With offices in Ghent, Amsterdam, Berlin, and London, we build and implement self-learning systems across different sectors to help our clients operate more efficiently. We do this by staying on top of research, innovation, and applying our expertise in practice. To find out more, please visit <a href="http://www.ml6.eu/">www.ml6.eu</a></p><p><a href="https://share.hsforms.com/1irYFScfRT_egbXIVftFipg37g8d">Subscribe to our newsletter here.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66da5088d030" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml6team/face-anonymization-a-comparison-66da5088d030">Face Anonymization: A survey of what works and what doesn’t</a> was originally published in <a href="https://medium.com/ml6team">ML6team</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>