<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Leif Ramming on Medium]]></title>
        <description><![CDATA[Stories by Leif Ramming on Medium]]></description>
        <link>https://medium.com/@leiframming?source=rss-c1190552f1ea------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rrSLh9RrWgfxPhmZ5kF0Kw.jpeg</url>
            <title>Stories by Leif Ramming on Medium</title>
            <link>https://medium.com/@leiframming?source=rss-c1190552f1ea------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 07:26:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@leiframming/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DSJC 7: Clip, Coco and multi-modal models]]></title>
            <link>https://medium.com/@leiframming/dsjc-7-clip-coco-and-multi-modal-models-b544d961393f?source=rss-c1190552f1ea------2</link>
            <guid isPermaLink="false">https://medium.com/p/b544d961393f</guid>
            <dc:creator><![CDATA[Leif Ramming]]></dc:creator>
            <pubDate>Mon, 27 Jun 2022 19:48:02 GMT</pubDate>
            <atom:updated>2022-06-27T19:48:02.515Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xn44XtWAacu8OC4W8BUuaQ.jpeg" /></figure><p>There has been a clear trend in recent AI research where big language models are trained on not only one type of data but on multiple types — crossing the domains from text and images or from sound and text. We have seen landmark examples like <a href="https://openai.com/blog/clip/">Clip</a> which was designed to embed text and images such that the embeddings are similar or <a href="https://openai.com/dall-e-2/">Dall-E 2</a> and <a href="https://imagen.research.google/">Imagen</a> which connect text and image in a generative fashion, allowing the creation of images described by text. Deepminds <a href="https://www.deepmind.com/publications/a-generalist-agent">Gato</a> goes a step further and connects real world tasks like steering a robot arm as well as playing video games and generating text. The goal here is to find synergistic effects that benefit all the tasks more than training each task individually. Conceptually it makes sense, seeing how humans are very visual people and the world we descibe is text is… well the description of the images we see. Hence, information described in two ways allows for a deeper understanding of the concepts that are represented and this means that the model learns a clearer signal of what is actually ‘meant’ with the text or image.</p><p>The idea itself is not new, multi-modal models have been around forever and image captioning or generation based on text are well-researched problems, but only over the last few months have we seen the field take off, with results of many networks now reaching a quality where it seems like the problem is ‘solved’.</p><p>Clip arguably played a big role in it, since it was one of the first really big models that connected images and text. The idea behind clip was to create embeddings from pairs of text and images that are trained in a way that maximize the inner product of examples that truly belong together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GUhurEqB8Pyd1sUAcb44Bw.png" /><figcaption><a href="https://arxiv.org/pdf/2103.00020.pdf">https://arxiv.org/pdf/2103.00020.pdf</a>, Figure 1(1) Contrastive pretraining. Image embeddings are paired with text embeddings such that the inner product is maximized.</figcaption></figure><p>This enables Clip to learn similarity between the text and the image embedding (because the inner product is a similarity metric). The trained model can be used for amazing things. In the paper, the authors propose zero-shot predictions: Create a bunch of prompts out of your classes (e.g. “A photo of a Dog”) and run them through the text encoder to get an embedding. Create the image embedding with the image encoder and multiply it with each text embedding to get the similarity of the texts and the images. The one with the highest similarity is your predicted class. The cloud: You are free to choose the prompt, meaning you can put any (any!) words in there. This means that you are not restricted to a pre-set number of classes any more but you can use any class you want. This was historically one of the biggest problems in classification and Clip just solved it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*en_NiRrWwB9DHV1twsvUQw.png" /><figcaption><a href="https://arxiv.org/pdf/2103.00020.pdf">https://arxiv.org/pdf/2103.00020.pdf</a> Figure 1(2),(3): Zero shot prediction: Classless classification of your image with some prompt.</figcaption></figure><p>The authors claim that using a whole sentence for classification works better than using just the word itself, because the training data, which are caption-and-image-pairs are usually whole sentences. When more domain information about the image is known, then prompt-engineering helps scoping the question in a way that already introduces a bias towards the right solution (i.e. “A photo of a dog, a type of animal” as opposed to … Snoop Dogg? But you get the point). It is also possible to create many slightly similar versions of the same prompt and pull their scores together to have some sort of ensemble process that averages the score afterwards in order to reduce the variance in the prediction.</p><p>You can, of course, use Clip the other way around: You embed one text and a number of images to find the image that best represents the text. This is some type of visual search, where Clip outputs the image that most likely fits your description. You can use that for finding <a href="https://github.com/haltakov/natural-language-youtube-search">a firetruck in a video</a> and <a href="https://blog.roboflow.com/openai-clip/">many other things</a>.</p><p>So, why are we talking about Clip? The paper we actually want to talk about is kind of a next iteration on the whole idea of multimodal models pretrained in a contrastive fashion. It has a slightly different setup but the same principle idea behind it and performs much better though, so it is definitely worth looking how the folds at Google figured out how to do it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P3QQwZ9l2MZl5LyCcHLryg.png" /><figcaption><a href="https://arxiv.org/pdf/2205.01917.pdf">https://arxiv.org/pdf/2205.01917.pdf</a> Title</figcaption></figure><p>The paper was released June 2022, unfortunately they have not open sourced the model nor the code. There is a pytorch version openly available on github from trusted <a href="https://github.com/lucidrains/CoCa-pytorch">ludicrains</a> but the fact that so few of the big models are actually showing their code, data or model is scientific malpractice. In the end, we have nothing other than the word of the company that they have produced the model with the performance they claimed and while the big ones (Facebook, Google, Deepmind, OpenAI) might have our trust, this practice will surely be followed by other companies who do not have or deserve it. But I digress.</p><p>The model itself is a lot less complex structured than Clip, which is definetly a good thing. It uses a sort of nested encoder-decoder architecture, where the encoder is a vision encoder that produces the image embedding and the decoder is a encoder-decoder stack of transformers that autoregressively predict tokens in the caption while having cross-attention to the image tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*glFbPsxa6qGPbHZ9SOXpAw.png" /><figcaption><a href="https://arxiv.org/pdf/2205.01917.pdf">https://arxiv.org/pdf/2205.01917.pdf</a>, Figure 2, left side. Structure of the CoCa neural network.</figcaption></figure><p>That is a lot to unpack, so let’s go through the different parts of it separately. On the left side, you have a Vision Transformer that produces image enbeddings. A <a href="https://arxiv.org/pdf/2010.11929.pdf">Vision Transformer</a> is a transformer that is specifically designed to deal with images, not text. It operates by cutting up the image into patches which are unrolled and then fed sequentially into the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*QO94HxSew86m_4NGVu91JQ.png" /><figcaption><a href="https://arxiv.org/pdf/2010.11929.pdf">https://arxiv.org/pdf/2010.11929.pdf</a> Figure 1: Model overview of the vision transformer.</figcaption></figure><p>Adding a patch and dimension embedding makes it easier for the patches to be recognized and internally put into the right place. There is a special token added to the beginning of each set of image patches; this represents the [classs] token that is used to hold the <a href="https://datascience.stackexchange.com/questions/90649/class-token-in-vit-and-bert">class embedding</a>. The way this is trained is that we feed the whole image (patched up) into the transformer but only us the output of the [class] token as input to our classification layer on top. This way we force the [class] token to be a representative of the class while we can use the whole information from the image patch sequence through attention in order to derive what the class token should be. The image would be easier to understand if you imagine the 0-patch on top of the “Transformer Encoder” layer, instead of beneath it, with a line going from there to the MLP head.</p><p>The authors of the ViT paper claim better performance while reducing the training time from 9.9k TPU-3 core days to less than 0.23k for the smallest ViT even though other papers claim that ViT only work well in terms of performance for really big datasets, hence research going in to <a href="https://arxiv.org/pdf/2112.13492.pdf">how to make ViT perform on small datasets</a>.</p><p>This should clear up the left part of the CoCa structure. The right part is a encoder-decoder stack that first creates a unimodel embedding of the input text and then computes a multimodal representation of text and image by using the text embedding as input and attending to the image embeddings produced on the left side. The decoder then reproduces the caption that it started with so that we can use that for training.</p><p>There is a hidden detail in the CoCa structure which is the <em>attention pooler</em> that combines all the image patch embeddings into one specific embedding. The authors use this for finetuning on various tasks, but they also do sometihng much cooler: By attending not only one image but multiple ones at a time, you get the ability to look at short spans of a video by feeding the frames into it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*IkKe9LcvOO05eE20POXrew.png" /><figcaption><a href="https://arxiv.org/pdf/2205.01917.pdf">https://arxiv.org/pdf/2205.01917.pdf</a>, Figure 3</figcaption></figure><p>You can get away with an even easier setup by just computing the average of the embeddings, instead of pooling them. Pooling has the advantage that not all embeddings are weighted the same but at a certain computational cost. If you are interested in why you can average embeddings: <a href="https://randorithms.com/2020/11/17/Adding-Embeddings.html">Here is why</a>.</p><p>Let’s look into how the models are trained. The overall loss is derived from two parts, a contrastive loss with a weight lambda and a captioning loss with a different weight lambda.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*52MjS8IHAsmyRej6xyd8hQ.png" /></figure><p>The captioning loss is easy to explain. It refers to the error that the model makes in predicting the correct caption again at the end of the decoder. It is the log probability that the next token is correct, given the tokens prior to it and the image embedding as input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*ayaT-9fphH7XRWBd3NAedA.png" /></figure><p>Interestingly, the model is trained using <em>teacher-forcing.</em> In an autoregressive setup, you usually take the output of the model as input in the next timestep, thereby building up your sequence. The problem here is, that as soon as we get a first mistake, it is easier and easier for the model to diverge and come up with some completely different tokens than your input, resulting in (too) high loss values and poor convergence. Teacher forcing aims to reduce that because no matter the prediction, the token taken as input for the next time step is always from your true data. This way, it is harder for the model to completely diverge and training becomes a bit smoother.</p><p>The other loss, the contrastive loss seems to be more delicate at first glance, but it is easy to undertstand once you disect it properly.</p><p>It is a loss measured between a pooled version of the image patch embeddings and the [class] token of the text embedding, because the loss function is only defined via two vectors. The <em>x</em> and <em>y</em> are image and text embeddings for the <em>i-th </em>and <em>j-th </em>embedding, Sigma represents a scaling factor (the ‘temperature’). Each part is a softmax over the dot product between the embeddings, averaged up. This equation is similar in that sense to the likelihood of images and text belonging together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MF-E5HkVq8n9H7YHnE3D0Q.png" /></figure><p>Note, how x and y are switched in left and right, so on the one side you are computing the similarity between image and text, on the other hand between text and image.</p><p>Think of it as representing the row or column in the first image about pretraining Clip. There, we also had the dot product only that now we are normalizing it so that values don’t exceed 1.</p><p>Putting this all back together, we are left with a simple to understand model that get’s a training signal both from contrastive training as well as from sequence prediction. So, how does it perform? According to the paper, really well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8m5FI66DHExAGvob0K-Fzw.png" /><figcaption><a href="https://arxiv.org/pdf/2205.01917.pdf">https://arxiv.org/pdf/2205.01917.pdf</a>, Figure 4: Results without finetuning</figcaption></figure><p>CoCa brings consistently high scores throughout various tasks ranging from zero-shot predictions on various datasets (including a new highscore on ImageNet) to caption generation (NoCaps) to Visual Question Answering (VQA) and others. This is highly impressive, considering that it goes agains some purpose build solution trained in the respective field. That’s already kind of an achievement and it shows how much synergie a model can get from training on both image and text. Granted, this model is big (on the order of 700M to 4B parameters), so a lot of performance will stem from this fact alone, but it is still impressive nonetheless.</p><p>Another impressive chart is a comparison agains multiple big models in terms of accuracy per parameters. It is nice, because it gives a comparison with Clip and Basic as well as ViT and SwinV2 which are big names in the biz. The figure below shows that CoCa is much more efficient than them, having a higher accuracy at a much lower number of parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AOPXLRKazHlNI1ZnaP4OfQ.png" /><figcaption><a href="https://arxiv.org/pdf/2205.01917.pdf">https://arxiv.org/pdf/2205.01917.pdf</a>, Figure 5: Accuracy of the model per parameter</figcaption></figure><p>It is interesting to see that CoCa reports a higher accuracy of seven percentage points compared to clip with the same number of paramters. That shows that even though the model is much simpler, having both contrastive and captioning loss allows it to train more efficiently. At the same time it more than 1B parameters smaller than BASIC for a similar zero shot Top-1 accuracy.</p><p>The rest of the benchmark tasks follow this similar pattern: CoCa yay, others nay. A nice feature worth mentioning is the out-of-the-box caption generation: Since we train the model to generate text from the images anyway, we can directly use that to generate captions. Other models need <a href="https://arxiv.org/pdf/2111.09734.pdf">a specialized model on top</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V-O0wZqBfPPGH0cpzEklKA.png" /><figcaption><a href="https://arxiv.org/pdf/2205.01917.pdf">https://arxiv.org/pdf/2205.01917.pdf</a>, Figure 6: Captions generated by CoCa</figcaption></figure><p>I was honestly surprised seeing the performance of CoCa on these various tasks. Google seems to have found a very powerful way to caption the information in image and text pairs and pull it together. Using cross-attention to the image pairs worked out really well and I would like to see more models trying this approach — not only with video but with all sorts of data.</p><p>While I can understand the financial impulse not to give away these models, it still is kind of a shame. For researchers that are independent or that are in a smaller lab, it is impossible to train those foundational models, still, having access to them would propel a lot of research and ideas forward. Thankfully we have <a href="https://github.com/mlfoundations/open_clip">OpenClip</a>, an open source version of Clip trained on LAION AIs open captioned image dataset which is freely available to us. Still, while I know that the seven percentage points difference in ImageNet don’t matter, I still dream of Google giving away free stuff (models), like they used to do.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b544d961393f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DSJC 2: Artificial Creativity]]></title>
            <link>https://medium.com/@leiframming/dsjc-2-artificial-creativity-1e2ae79a910a?source=rss-c1190552f1ea------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e2ae79a910a</guid>
            <category><![CDATA[creativity]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[art]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Leif Ramming]]></dc:creator>
            <pubDate>Wed, 25 May 2022 18:14:15 GMT</pubDate>
            <atom:updated>2022-05-25T18:15:17.966Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*0dBfDxRFuQmCSOf4Eiz1rA.png" /><figcaption>source: <a href="https://imagen.research.google/">https://imagen.research.google/</a>, cropped</figcaption></figure><p>Can Artificial Intelligence be creative? After all, the program that we execute is just a soulless bunch of zeros and ones, switched around until it produces some humanly desired output. Where in this program is the creative spark, the special sauce, the guiding hand of the muse that descends upon the artist when he or she creates something? To quote the movie <a href="https://knowyourmeme.com/memes/can-a-robot-write-a-symphony"><em>I, Robot</em></a>: “Can a robot write a symphony? Can it turn a canvas into a masterpiece?”</p><p>Despite all that, I think artificial intelligence, even the one we have today, are creative, at least by my pragmatic standard of that term. Consider models for generating images, since painting is considered one of the more artistic, creative forms of work. New models like Nvidias <a href="https://github.com/NVlabs/stylegan3">StyleGan 3</a> or OpenAIs <a href="https://openai.com/dall-e-2/">Dall-E 2</a> are reaching a point where images generated by an AI become indistinguishable from the ones created by human artists.</p><p>If you were to commission a specific image from someone and got one that is outstanding, new and never seen before; if you could not tell whether it was created by a human or a machine, does that lessen the creative aspect behind either of them? The human artist would have to go and come up with the image to be created, then build it with traditional tools. We would no doubt hail his or her creativity for coming up with an image. If the machine does it, would it not as well be creativity at play?</p><p>There are others that think that <a href="https://erikhoel.substack.com/p/ai-art-isnt-art?s=r">art generated by AI is not art</a> because there is no intent behind it, and who fear that we are losing an form of art and communication that has been with humanity for a long time and is oh, so important. I disagree with this reasoning because of two points: First, the AI used is merely a tool, much like a fancy brush. The intent comes from the human input and can still be considered a work of art; the program that creates the painting out of it is but a helper. Secondly, art, especially drawings change a lot. When looking back to the beginning of the 20th century when the first cameras were widely available to the public, drawing had to adapt as well. I don’t think that it is a coincidence that we saw the emergence of <a href="https://en.wikipedia.org/wiki/Fauvism">Fauvism</a> and <a href="https://en.wikipedia.org/wiki/Cubism">Cubism</a> right when it was possible for everyone to take photos. Then, naturalistic paintings became ubiquitous and painters turned to painting things that can not be seen in nature — the birth of the modern art of the 20th century. Now, even those paintings become ubiquitous. I wonder what the next leap for art will be.</p><p>So, how do these models work? What makes them perform well and why have other types of models not succeeded? That was the topic of our second Data Science Journal Club at plista. Let’s have a look at three strategies how to create images (GANs, VAEs, Diffusion Models) and explore some of the recent advancements together.</p><p>Scientists have long been trying to figure out how to create images with machine learning models but to little avail. Why is the problem so hard? Imagine an color image of size 1024 x 1024, still small by today’s standards. The information this image carries is 1024 x 1024 x 3 = 3,145,728 pixels that have to be predicted. In other words, if you think of the space of all possible images, then each specific image you want to have is a vector in a 3 million dimensional space. Try to guess the correct value for this vector by random, I won’t be waiting for you. What the model has to achieve is to find a way to sample so efficiently from this space, that you will get a useful image <em>every time</em>. That is a tough problem to solve. Even worse, you usually want to control which image you end up with, meaning you don’t have to sample any image from the space of all possible images, but you have to find the specific image that fits best.</p><p>There are multiple problems with framing the problem this way. First, what constitutes as ‘good’ image? Typically, you need to have some form of error value that you can minimize when building machine-learning models. What is the error function that defines whether an image makes ‘sense’ to a human observer? Yes, it is possible to generate images of specific things (e.g. numbers) by getting images of solely the number ‘8’ rasterize it and then predicting if each pixel in the image needs to be on or off by minimizing the mean error between pixel intensities, but that is very limited to small, well-defined problems and does not allow you to generalize to the space of all images.</p><p>The field got a jumpstart in 2014 with the infamous paper <a href="https://arxiv.org/abs/1406.2661"><em>Generative Adversarial Networks</em></a><em> </em>by Goodfellow et al. In it, he tried to frame the problem of generating an image as a game between two agents, where one creates an image (<em>generator) </em>and one that has to decide if an image is a created one or not (<em>discriminator</em>). We show the discriminator either the output of the generator or an image from training data and use it’s output to compute the binary cross-entropy which is fed back to the generator to train it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*GdIyJ1NOfEWXcvHkDm6-5Q.png" /><figcaption><a href="https://arxiv.org/pdf/1406.2661.pdf">https://arxiv.org/pdf/1406.2661.pdf</a>, Formula 1: Value function of the generator and discriminator</figcaption></figure><p>The value function shown above looks intimidating at first but it becomes easier when you step through it one-by-one. <strong>x</strong> is some data from a data distribution <em>p(</em><strong><em>x</em></strong><em>), </em>which your training data is a sample of. D(<strong>x</strong>) is the output of the discriminator and represents the probability that the data comes from the data distribution. Conversely, a high 1-D(<strong>x</strong>) means that the probability is high, that the data does <em>not</em> come from the true data distribution. To be more concrete: Your distribution is all humanly created images from which you have gotten your training data. A high value for the discriminator output represents confidence that the image was created by a human. There is a second bit <strong>z</strong><em> </em>which follows a separate distribution<em> p(</em><strong><em>z</em></strong><em>). </em>The generator transforms the data and feeds the results as input to the discriminator. This means, that, if the value for D(G(<strong>z</strong>)) is high, we have fooled the discriminator because it assumes that G(<strong>z</strong>) originally belonged to our true data distribution <em>(which results in 1 — D(G(z)) being low)</em>.</p><p>So, the left hand side is the log probability of the discriminator being right and the data x actually being humanly created, the right hand side represents the generator fooling the discriminator and creating an image that is indistinguishable from actual images. The resulting value V(D,G) is the value of how good the discriminator is for the given generator. We want to maximize it for D, while simultaneously minimizing it for G.</p><p>The generator and the discriminator work against each other, improving the respective other in the process: If we have a good discriminator, then the error signal to the generator is strong and the generator is able to improve quickly. Once the generator becomes good enough to fool the discriminator, the discriminator becomes better in discerning the newly created images from the ones from training data. This idea is shown in the figure blow from the original paper: Violet/blue depicts the decision boundary of the discriminator, green shows a distribution of or the generator data and black a distribution of real data. Part (a) on the left shows the starting state: Both the generator and the discriminator are random. (b) shows the trained discriminator, whose decision boundary is able to discern between data from blue and black. (c) shows the effect of this trained discriminator: The generator changes it’s sampling to match the underlying distribution in black. Note the three dots in between (c) and (d) indicating that this process is repeated for a long time. Then, we finally have (d), where the generator fits the true data distribution well and the discriminator is only a flat line — indicating that it can not discern anything (i.e. the prediction is always 1/2). This is the desired end-state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yLbvCdxUIYUjfb0wrR9QkQ.png" /><figcaption><a href="https://arxiv.org/pdf/1406.2661.pdf">https://arxiv.org/pdf/1406.2661.pdf</a>, Figure 1: Discriminator distribution violet, generator distribution green, true distribution black</figcaption></figure><p>This way, we are not actually specifying a metric that defines how good an image is or even which class it belongs to or what is seen on it — instead we are training this metric in form of the discriminator alongside the function that should actually generate the image. Once we have everything trained, we can go ahead and throw the discriminator away. We are only interested in generating the images after all. It is a simple and elegant solution and that drove creative dreams around the world for the next several years after the publication of the paper. If you want to train a GAN yourself and experience what it is like, check out this <a href="https://colab.research.google.com/github/tensorflow/gan/blob/master/tensorflow_gan/examples/colab_notebooks/tfgan_tutorial.ipynb">Colab Notebook</a>.</p><p>The method, however is not without limitations. There are <a href="https://jonathan-hui.medium.com/gan-why-it-is-so-hard-to-train-generative-advisory-networks-819a86b3750b">technical problems</a>, e.g the discriminator trains a lot easier than the generator, especially at the beginning, which makes it hard to get a good error signal. There is also something called the <a href="https://developers.google.com/machine-learning/gan/problems"><em>Mode Collapse</em></a><em> </em>where a trained GAN suddenly loses its ability to create images and just puts out random noise. An underlying problem of GANs was also the difficulty to control the output of the GAN ahead of time. Since the error function only optimizes the generator to create images which look like they come from the training data, it did not offer the ability to generate, for example, a cat or a dog on command.</p><p>Researchers soon started to tackle this problem by conditioning the output on some controllable variables that were extracted from the images. This way, you can force the output to be of a specific type, by controlling the conditioning part of the input. You just attach a one-hot encoded vector to the image vector and generate the image; the discriminator get’s this image and the one-hot encoded vector and makes it’s decision whether it is from p(x) or not. This conditioning was first used in the <a href="https://arxiv.org/pdf/1411.1784.pdf">cGAN paper</a> to generate the different numbers of the MNIST dataset on demand.</p><p>A related problem is the so-called style transfer. Instead of specifying a one-hot encoded vector with discrete classes, it should be possible to extract the ‘style’, i.e. the way a painting is painted or a photo is shot, and combine images with that style. After all, it is possible for humans to paint a portrait ‘in the style of van Gogh’ and other humans could understand and see what this means. The original <a href="https://arxiv.org/pdf/1812.04948.pdf">StyleGan</a> paper made a lot of waves when it was originally published by mastering this style transfer and generating spectacular images. It did so by mapping the latent variables that were generated from the input to a feature space <em>w</em> and then feeding it to the different parts of the generating network. Unfortunately, the paper does not explicitly states, how the latent code <strong>z</strong> is being generated; at this point I assume that they were using image embeddings from some ConvNet that encode the high-level ‘style’ of the image. Note, how they are not just attaching the intermediate feature vector <strong><em>w </em></strong>to the input of the actual generating part of the network, but instead feed it in on multiple occasions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*h8g6sVsQPa6SIt1AAvdsCQ.png" /><figcaption><a href="https://arxiv.org/pdf/1812.04948.pdf">https://arxiv.org/pdf/1812.04948.pdf</a> Figure 1(b): StyleGan generator</figcaption></figure><p>The results they generated were astounding, even by today&#39;s standards. It was possible to transfer the style of one image to another or even to specify the latent code and then generate the image accordingly. Below is an image from the original paper that shows the capabilities of the net.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*sCBrGG6sLKyRyijXNvxUtQ.png" /><figcaption><a href="https://arxiv.org/pdf/1812.04948.pdf">https://arxiv.org/pdf/1812.04948.pd, </a>Figure 3:</figcaption></figure><p>It is amazing to see the progress from just a few years of development. While the original GAN paper only showed very crude faces (still great at the time, but, yeah), here we see actual faces in a good resolution. Also, we can directly see the style transfer between the sources A and B. It’s results that make sense from a human’s point of view, the best kind of results. Over the next years, the follow-up papers for <a href="https://arxiv.org/pdf/1912.04958.pdf">StyleGan2</a> and <a href="https://nvlabs-fi-cdn.nvidia.com/stylegan3/stylegan3-paper.pdf">StyleGan3</a> were published, each taking a careful, systematic look at the different artifacts and defects in the images, then coming up with solutions to them. The paper to StyleGan3 was published in 2021 and the code is freely available on <a href="https://github.com/NVlabs/stylegan3">github at Nvidialabs repository</a>, where you can also find a link to freely-available pre-trained models. If you would like to learn more about GANs, I recommend checking out <a href="https://lilianweng.github.io/posts/2017-08-20-gan/">this write up of Lilian Weng</a> for all the theoretical information and one of the post of towardsmachinelearning for a <a href="https://medium.com/towards-data-science/implementing-a-generative-adversarial-network-gan-dcgan-to-draw-human-faces-8291616904a">practical application</a>.</p><p>As discussed in the beginning, randomly sampling the space of all images is hard, because <a href="https://www.goodreads.com/quotes/14434-space-is-big-you-just-won-t-believe-how-vastly-hugely">space is big</a>. It is also not structured and regulated so even if we find one point that represents an image, we still don’t know other points for images that are similar. If only there was a way to compress this space down and regularize it so that sampling from it is easy. Lo and behold there is a <a href="https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73">Variational Autoencoder </a>to do exactly that. An Autoencoder is a type of neural networks that mapping data into a lower-dimensional space, thereby forcing the algorithm to learn rules to compress/uncompress data by finding and encoding the rules governing the data distribution. The ‘Variational’ part comes from the usage of <a href="https://ermongroup.github.io/cs228-notes/inference/variational/">Variational Inference</a> in order to estimate and optimize the lower bound the reconstruction error with a regularization on the latent space. What does that exactly mean?</p><p>The Autoencoder itself is a relatively simple neural network. It consists of layers that progressively get smaller until they reach a certain size and then progressively get larger again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DQkNxXkYa2yo30uC.png" /><figcaption>source: <a href="https://towardsdatascience.com/applied-deep-learning-part-3-autoencoders-1c083af4d798">https://towardsdatascience.com/applied-deep-learning-part-3-autoencoders-1c083af4d798</a></figcaption></figure><p>The input and the output, in our example, are the images as one flat vector. The intermediary ‘code’ is a low-dimensional representation that encodes all necessary information to decode the output from it. The error signal to train the neural network is a similarity measure between input and output, in the naive case the mean squared error. Reducing the size or the hidden layers prohibits the network using the identity function, instead it forces it to learn some representation of the input image. In theory, after training the model we would just have to sample a code and then use the decoder to generate the image from it.</p><p>The problem is, that just sampling the space of the ‘code’ is still not efficient. That is, because there is no inherent ordering, or regularization, in the space of the code. After all, the goal of the autoencoder is not to have a neatly organized intermediary encoder space but to make input and output as similar as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Lm4Glfsl8MQLtVRH.png" /><figcaption>source: <a href="https://ml.berkeley.edu/blog/posts/vq-vae/">https://ml.berkeley.edu/blog/posts/vq-vae/</a></figcaption></figure><p>The solution is to introduce a regularization term that forces the ‘code’ to follow a (multivariate) Gaussian distribution: Instead of creating the hidden vector in middle, we are predicting the means and variances of a Gaussian distribution. Then we sample from it and feed that sampled vector into the decoder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UVjNUnuwq6uaDYON.png" /><figcaption>source: <a href="https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73">https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73</a></figcaption></figure><p>The image shows the new loss function of the Autoencoder: It is a sum of the mean squared error of input and output, as well as the Kulback-Leibler divergence between a standard gaussian and the fitted distribution. This way, in order to reduce the loss, the distribution of the hidden space must closely follow a standard Gaussian. The mathematical reasons on why this works are covered pretty well in <a href="https://lilianweng.github.io/posts/2018-08-12-vae/">Lilian Weng’s VAE article</a> as well as the towardsdatasciencearticle linked in the image, so I will spare you the details. Suffice to say, VAEs are a viable way to build image-generating neural networks.</p><p>An extension of the VAEs are so called <a href="https://arxiv.org/pdf/1711.00937.pdf">Vector-Quantized Variational Autoencoders</a>. In it, we are not using a Gaussian distribution for a regularization but some categorical distribution, which maps the inputs to the index of an embedding table. The values from that table then make up the code that the generating part of then autoencoder sees and from which it has to generated the image back.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hoNI65FJyuvxlqCo4H8bFQ.png" /><figcaption><a href="https://arxiv.org/pdf/1711.00937.pdf">https://arxiv.org/pdf/1711.00937.pdf</a>, Figure 1</figcaption></figure><p>The loss function for this VQ-VAE below is defined as the sum of three parts: The reconstruction log loss on the left-hand scores how well the generated image and the input image match. The part in the middle is a score how well the embeddings in the embedding space are generated: <em>e</em> is the embedding vector for the output of the encoder z_e(x), so the middle term represents an L2-regularized error term for the distance between both. The term <em>sg[…]</em> is a stop-gradient operator that implies that in the backpropagation, no error should flow to the respective part, in this case the encoder. This means, that even though we are computing a distance between the encoder output <em>z_e(x)</em> and the codebook vectors <em>e</em>, we are only ever updating the codebook vectors with it. This would, over time, move the vectors <em>e </em>closer to the encoder output, which is the desired outcome for vector quantization.</p><p>The right-hand side term is essentially the same, only that the stop-gradient operator is now on the encoder output. This is called the <em>commitment term</em> and is updating the encoder instead, with magnitude influenced by some hyperparameter β.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*6wpA4OBiYaFv1A6vlYA3lw.png" /><figcaption><a href="https://arxiv.org/pdf/1711.00937.pdf">https://arxiv.org/pdf/1711.00937.pdf</a>, equation (1)</figcaption></figure><p>So, instead of creating a continuous latent vector, VQ-VAEs create a discrete latent vector. How is this better? Intuitively, the latent vector represents abstract classes and if you have classes in your data (i.e. “cat” shown in an image) then mapping that to a discrete class instead of two very similar but slightly different numerical vectors makes sense. The same goes for phonemes when modeling audio, as discussed in the paper. Where a continuous vector will always have some divergence due to numerical stability problems (i.e. a value of 0.99999 and 1.00001 are probably the same — but numerically they are not), a distinct vector removes those small ambiguities. That adds up to better performance overall. Even in <a href="https://openai.com/blog/jukebox/">OpenAIs Jukebox</a>, and amazing AI for generating music with different styles, VQ-VAEs are used to generate high-level, discrete codes that a different transformer-based model converts into a waveform.</p><p>Let’s go back to the beginning for a bit. Our original task was to create images. The problem is, that we don’t know anything about the distribution p(x) of the images, so we have no efficient means to sample it.</p><p>So far, we have discussed GANs that use the discriminator to learn what it means to be an image, therefore getting some knowledge about the distribution, and VAEs to compress it into a smaller, regularized feature space from which we can efficiently sample.</p><p>A third strategy has recently become the most prominent methods, achieving current state-of-the-art results when it comes to image generation, such as in <a href="https://openai.com/dall-e-2/">OpenAIs Dall-E 2</a> or in <a href="https://imagen.research.google/">Google’s Imagen</a>. I am, or course, talking about <strong>Diffusion Models.</strong></p><p>Diffusion models work by modeling a stepwise denoising process. Imagine having an image and adding a little bit of gaussian noise to it. As a human, it is still easy to ‘understand’ the image and know what is shown. The more noise you add, the harder this becomes, until the image consists only of white noise and there is no structure knowable. If we add only a little noise to the image, and we can still discern the image, it should be possible to train a model to denoise the image. After all, this is what humans do in their head as well. So if we are denoising the image, it should also mean that we should be able to denoise a more noisy image into a less noisy one. If we take that to the extreme, we can add more and more steps, until we are able to take some form of noise and ‘denoise’ it, until some image emerges. If the noise pattern changes, so does the image we get in the end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/0*n5TtSCui91SYad9e.jpg" /><figcaption>source: <a href="https://yang-song.github.io/blog/2021/score/">https://yang-song.github.io/blog/2021/score/</a></figcaption></figure><p>The idea of de-noising processes is not new and also comes up in a different class of generative networks, the so called Normalizing Flow Models. There, we train a neural network to add noise to an image but restrict the network to use only functions that are invertible — by training the noisifying function and then inverting the trained functions we get a generative model that is able to denoise. However diffusion models can use a <a href="https://arxiv.org/pdf/2006.11239.pdf">Markov-Chain process</a> to noisify images and then train a neural net to remove the noise. The more intermediary steps (“timesteps”) they use, the better the prediction, because we have to reduce only little noise. The paper <a href="https://arxiv.org/pdf/2006.11239.pdf">Denoising Diffusion Probabilitic Models (DDPM)</a> use 1000 steps for generation which achieved very good results, but at the cost of taking a long time because those steps are gone through sequentially. An improvment to this were the so called <a href="https://arxiv.org/pdf/2010.02502.pdf">Denoising Diffusion Implicit Models (DDIM)</a> that use non-markovian processes to add noise to the image. The network that is used for denoising is mostly a U-Shaped network, that uses convolutions to compress the width and height dimension of a network and extends the channel dimension and then back. These networks actually give amazing results — they are now powering the current state of the art models for image generation, especially in multi-modal models like Googles new <a href="https://imagen.research.google/">Imagen model</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*l-whMD5mpjGUWRvL_tRKGA.png" /><figcaption>source: <a href="https://imagen.research.google/">https://imagen.research.google/</a>, below is the caption used to generate the image above</figcaption></figure><p>I decided to stop here and not go deeper into the different models that are used to generate images, even though diffusion models fall a bit short now. Each deserves it’s own blog post and I will eventually look at the papers for DDIM and Imagen in more detail. I am extremely excited by the capabilities of diffusion models, if only because it somehow flew under my radar until the first papers like <a href="https://arxiv.org/pdf/2112.10741.pdf">Glide</a> and <a href="https://openai.com/blog/dall-e/">Dall-E</a> brought them more into the limelight.</p><p>Generating images from scratch was for a long time a problem that escaped the grasp of researchers. I remember how far away it seemed when the first GAN papers hit the research community. Back then we were excited but at the same time we saw the results which were obviously bad. Not even ten years later the task seems somewhat solved. Images are generated, not only randomly but you can even <em>specify the text that the image is generated for</em>. Stop for a second and appreciate how far people have come here. Multi-modal models have already taken over research into big AI models and I didn’t even have the chance to explicitly mention them in this article. It’s nice to know that there is enough material to write about for the coming years.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e2ae79a910a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science Journal Club 1]]></title>
            <link>https://medium.com/@leiframming/data-science-journal-club-1-caf678fc5c92?source=rss-c1190552f1ea------2</link>
            <guid isPermaLink="false">https://medium.com/p/caf678fc5c92</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[journal-club]]></category>
            <dc:creator><![CDATA[Leif Ramming]]></dc:creator>
            <pubDate>Sun, 01 May 2022 11:25:08 GMT</pubDate>
            <atom:updated>2022-05-01T11:25:08.099Z</atom:updated>
            <content:encoded><![CDATA[<p>Introduction to Neural Networks</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w0Q7SQblx2KDoSt72O4OyA.jpeg" /></figure><p>The journal club at <a href="http://www.plista.com/">plista</a> is a monthly meeting, where engineers, folks from operational teams, even members of our parent company <a href="http://www.xaxis.com/">Xaxis</a>, come together to learn about the latest and greatest what happened in the tech world, in order to <a href="https://medium.com/@leiframming/knowledge-is-the-basis-for-innovation-23696944a3aa">get more creative and innovative in their own work</a>.</p><p>Learning is hard work! Not only for the presenter, who has the arduous job of conveying what he knows through the medium of a 60–90 minute presentation and discussion, but also for everyone that reads about papers beforehand, sacrifices their time and energy and actively asks questions. I appreciate that we have colleagues who are willing to do all this work.</p><p>What will you be able to read below? The Journal Club itself was a long presentation and I can not show all the slides and write up all the things we talked about. Instead, since the first few journal clubs were a brief overview of a specific topic instead of being about an actual paper, I decided to write a even briefer abbreviation of the talk below. It is meant for people who have heard about the topic but want to dive back into it, as well as find out some informational tidbits from the links embedded into the text.</p><p>When we restarted our Journal Club last year, we added a bunch of members that had little prior experience with ML. We started out with a bunch of introductory talks that should get everyone to a certain level, before we can go on to read more advanced papers. Those five introductory talks discussed the following topics:</p><ol><li><strong>Introduction to deep learning<br></strong>What are neurons, what are neural networks, how do dense layers work, different types of layers.</li><li><strong>Generational models for images<br></strong>GANs, Inverse-flow models, VAEs. What are deepfakes, different applications for image generation.</li><li><strong>Natural language models<br></strong>Transformers and attention, big neural network models, embeddings and language modeling.</li><li><strong>Generational models for sound and music<br></strong>ML models, Transformers, modeling long distance interactions, VAEs for music and different applications</li><li><strong>Artificial intelligence: Let’s welcome our robotic overlords<br></strong>What is GAI, where are we now, where are we going?</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*WZ3IauxnLvrnEtGyoRN_zA.png" /><figcaption><a href="https://dev.to/nyior/artificial-intelligence-vs-machine-learning-vs-deep-learning-a-quick-comparison-22ii">https://dev.to/nyior/artificial-intelligence-vs-machine-learning-vs-deep-learning-a-quick-comparison-22ii</a></figcaption></figure><p>Artificial Intelligence, AI, is a buzzword often heard nowadays, describing some component or service that predicts what you want and delivers it, without you needing to specify it at all. It can handle the <a href="https://arxiv.org/abs/2005.14165">ambiguities of natural language</a>, it helps you choose by <a href="https://www.mdpi.com/2076-3417/10/21/7748/xml">narrowing down your options to the most likely ones</a> or, more abstractly, it might help <a href="https://ojs.aaai.org/index.php/AAAI/article/view/16635">plan processes</a> and schedule resources in a way that outperforms human operators. Humankind has been in pursuit of actual artificial intelligence ever since the invention of the first computer, but while it occurs so regularly in science fiction literature that it seems everyone agrees it is inevitable to come, we have yet to see the promised actual intelligence.</p><p>This leads us to the question: Where are we now, 2022, with AI? What are the current big leaps and inventions of the field, will we see <a href="https://www.vttresearch.com/en/news-and-ideas/ai-winter-coming-again-unless-we-change-course">another AI Winter</a> or are the huge language models that OpenAI, Google and seemingly every trillion dollar company churn out actually intelligent and <a href="https://towardsdatascience.com/openais-chief-scientist-claimed-ai-may-be-conscious-and-kicked-off-a-furious-debate-7338b95194e">conscious</a>?</p><p>Artificial intelligence is a broad topic and smart systems are more than just models, you need the Data Engineering Team to feed your model data, the Design and Front End teams to make a useful app out of it and there are dozens of different ways to have smart models that achieve the “Smartness” that your user sees. For the sake of this articles, we will abstract away all this complexity and only talk about models and only about Neural Networks. While the model you are using is not the only important part, it is the one constraining the possibilities of what you can do, the most.</p><p>If you want to actually learn about Deep learning and neural networks, there are a lot of resources online that help you, from classes on coursera to freely available books like the <a href="https://arxiv.org/pdf/2106.10165.pdf?">Principles Of Deep Learning Theory</a>, from which I take a bit of inspiration here.</p><p>A neural network is, in the end, a function <em>f </em>that we uses some parameters θ (or sometimes called weights <em>w</em>) to transform your input data <em>x </em>into your labels <em>y:</em></p><blockquote>f(x, θ) = y</blockquote><p>There you have it, folks. All the nice images you see, all the visual representations of neural networks are just descriptions in order for us humans to understand that in the end, we are just fitting a function.</p><p>Of course, at this abstract level, the devil is in the details. What kind of function to we have? What are the actual values for the parameters?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bajx7ccII9JrvRu6pTJZWA.png" /></figure><p>The function itself is a composition of multiple different units, each with it’s own weights <em>w </em>attached to it. The image above shows one of those units, typically called neurons. They consist of an aggregation function for the incoming inputs that sums up the input times a weight, a constant bias and activation function that may transform the linear raw output into a non-linear output. Neurons are arranged in multiple layers, where the output of one layer serves as the input of the next one. All neurons of one layer can be handled in one step, by grouping their weights into a matrix and the biases into a vector. The output of the neuron is then a a matrix multiplication where you multiply your input vector <em>x</em> with your weight matrix <em>W, </em>add your bias b and feed the result to your squashing function to obtain your output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*sMB4pNW1guki_X0FJU8B-Q.png" /></figure><p>This matrix multiplication is also, why co-processors like GPUs or TPUs are so important to deep learning. It is the most compute-intensive step, but also easily parallelized, making it an optimal target for optimization. In the upper example we have three neurons in a dense layer, meaning each input is connected to each neuron via one weight. The result of this computation <em>y </em>can then be server as the input <em>x </em>of the next layer of your network. Composing those functions gives you the ability to model more and more complex data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*I2XNLKyPmqGdzKhnAVwF3g.png" /><figcaption>source: <a href="https://towardsdatascience.com/why-we-need-bias-in-neural-networks-db8f7e07cb98">https://towardsdatascience.com/why-we-need-bias-in-neural-networks-db8f7e07cb98</a></figcaption></figure><p>Consider for as second having just one neuron with one input and output. Your function <em>f</em> is then reduced to:</p><blockquote>y = wx + b</blockquote><p>this is a typical linear equation describing a line in a two dimensional space. Changing the weight changes the slope of the line and changing its bias changes where the line intercepts the y-axis, allowing you to have the full range of motion to fit every possible line. The line you fit could then represents a decision boundary for a classification: Everything above the line is one class, everything below it the other class. In general, your data has more than one dimension, so you would speak of a hyperplane instead of a line, which is a plane with N-1 dimensions that you fit through an N-Dimensional space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*9LIBewS4lJHlUGMAT9Lbsg.png" /><figcaption>A line is a hyperplane in 2D-Space and acts as a decision boundary. source: <a href="https://towardsdatascience.com/perceptron-explanation-implementation-and-a-visual-example-3c8e76b4e2d1">towardsdatascience</a></figcaption></figure><p>This is akin to how a <a href="https://en.wikipedia.org/wiki/Support-vector_machine">SVMs</a> works and, in fact, with the right margin-maximizing error function and the correct number of parameters there is <a href="https://www.cs.upc.edu/~alquezar/articles/fnnsvm.pdf">not much difference between a SVM and a NN</a>.</p><p>We have talked about weights and biases and intuitively it is probably clear what kind of role they play. What about the activation function though? There are many different activation functions but they all serve the same purpose: Introduce a non-linearity into the otherwise linear function of the neuron. Why is this necessary? <a href="http://jwilson.coe.uga.edu/emt668/EMAT6680.2003.Su/Daniel/6680/%231/%231-webpage.html#:~:text=The%20composition%20function%2C%20h(x,is%20also%20a%20linear%20function.">Compositions of linear functions only ever yield linear functions in return</a>. Imagine two functions:</p><blockquote>f(x) = Wx + b</blockquote><blockquote>g(x) = Kx + c</blockquote><blockquote>y = g(f(x)) = K(Wx+b) + c = (KW)x + Kb + c</blockquote><p>so the result of the composition is again a linear function. Only by introducing the non-linearity, we can break this pattern and are able to fit non-linear functions.</p><p>The choice of activation function depends on the needs of your model. Nowadays, hidden layers have a rectified linear function (ReLu) or some variant of it (e.g. <a href="https://arxiv.org/abs/1710.05941">Swish</a> or <a href="https://arxiv.org/pdf/2204.02311.pdf">SwiGLU</a>) because they avoid the pesky <a href="https://en.wikipedia.org/wiki/Vanishing_gradient_problem">vanishing gradient problem</a>. The final layer can have different ones, such as single neuron with a sigmoid function for binary classifications, which have the nice property of squashing the result to [0, 1] so it’s easy to use it for two classes. In a multinomial classification problems, each class is represented by a neuron and a softmax activation over the whole layer transforms the outputs into a probability distribution.</p><p>Coming back to the questions we asked at first (What kind of function is <em>f</em> and what are the correct parameters), we have so far only cared about the first part. So let’s get a bit more concrete and talk about the value that our parameters θ can take. How do we find the correct value?</p><p>The underlying idea it is simple: Start with some probability distribution <em>p(</em>θ<em>) </em>to assign values to the weights and do a prediction. Then measure how far you are off from from your actual target. Change the weights, and repeat until our prediction and target match.</p><p>This simple description glosses over one important detail: How do we find out whether to in- or decrease the weights? Luckily for us, we only use functions that are differentiable so we can compute the change of error when we change the parameters (partial differential equation of the error w.r.t the parameters) which, after solving, tells us not only how much we need to adjust our parameters, but also in which direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*CL3a4kTbGd47PZgXP7NsOg.png" /><figcaption>Partial derivative of error with respect to the parameters, source: <a href="https://en.wikipedia.org/wiki/Backpropagation">Wikipedia</a></figcaption></figure><p>Seeing this equation makes us understand the value of frameworks like <a href="https://www.tensorflow.org/">Tensorflow</a> and <a href="https://pytorch.org/">Pytorch</a> which have pioneered auto-differentiation. Gone are the days where you needed to implement equations yourself, nowadays it is (most of the time) enough to create layers of neurons and let the optimizer of your framework of choice handle the rest, albeit with some caveats for functions that are not continuous (think argmax). <a href="https://github.com/google/jax">Jax</a> is a similar framework, extending the concept even to normal python and and NumPy functions.</p><p>The partial derivative above gives us the change of error when we change the parameters and depends on the weights, the measured error and the <em>output</em> of the function. This is already a tell-tale on <a href="http://neuralnetworksanddeeplearning.com/chap2.html">error-backpropagation</a>. The idea is to measure the error of your prediction and then adjust the weights of your layers one layer after another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RXpB9WUIdn2WJfvTzYYV9g.png" /></figure><p>The image above shows the idea again. You have a dataset of images that may contain faces of a human or of an animal. For each image you have a (numerical) label specifying the true value of whether there is actually a human face. The prediction and the label have the same dimensions and are fed to the loss function that reduces the vector to a single number. This error value can then be used to adjust the weights using <a href="https://medium.com/swlh/a-gentle-introduction-to-backpropagation-and-implementing-neural-network-animation-f6b6da9d46d5">backpropagation</a>. Over time, the weights will be shifted by the algorithm so that the error is minimized.</p><p>The way we define the error is hugely important in what we want to achieve. It is of utmost importance to be clear what you want to achieve with your model so that you select the correct function to minimize. In the setup discussed above, you have labels specifying the true value of the result,a setup typically called <em>Supervised Learning</em>, and it is easy to select a fitting error function. Categorical values to predict? Probably some form of <a href="https://gombru.github.io/2018/05/23/cross_entropy_loss/">cross-entropy loss</a>. Numerical values? <a href="https://peltarion.com/knowledge-center/documentation/modeling-view/build-an-ai-model/loss-functions/mean-squared-error">Mean squared error</a> is a solid choice. It gets more challenging when you do not have labels, so called <em>Unsupervised Learning</em>. There are numerous ways to deal with this, often relying on finding a measurable difference between the results of your function and either minimizing or maximizing it. <a href="https://towardsdatascience.com/understanding-contrastive-learning-d5b19fd96607">Contrastive Learning</a> compares the outputs of a model and tries to minimize the distance between the outputs from examples from the same class, while maximizing the output of examples from different classes (so called <a href="https://www.jmlr.org/papers/volume11/chechik10a/chechik10a.pdf">Triplett-loss</a>). A similar idea is often used in Representation Learning, i.e. when you want to compute embeddings that depict something in an abstract high-dimensional state (think of expressing the semantic meaning of a text or the contents of an image in a vector). A recent model that successfully uses contrastive learning is OpenAIs <a href="https://arxiv.org/pdf/2103.00020.pdf">CLIP</a>, which trains to encode pairs of images and their respective description in the vector space, so that the cosine distance between the vectors of together occuring texts and images is minimized.</p><p>When you have defined your error function and you are backpropagating it through your layers, there are multiple ways to actually update your parameters. To improve resource utilization (<a href="https://arxiv.org/pdf/1609.04836.pdf">and for mathematical reasons</a>) we use minibatches and can use various schemes to update our parameters. The most-used, I dare say the standard method right now, is probably <a href="https://arxiv.org/pdf/1412.6980.pdf">ADAM</a>. The optimizer used can have an impact on the training time and the end result but Adam is pretty good for stochastic optimization and a safe choice.</p><p>So far, we have talked only about one type of neural network — a sequence of dense layers, where the output of one layer serves as the input of the next. This dense layer is arguably one of the most important architectures, as it is used both by itself as well as part of the Transformer architecture that has driven much of the most recent inventions in the space of deep learning, however it is far from the only one. Especially when talking about Computer Vision problems (i.e. recognizing objects in images, face recognition, etc), Convolutional Networks (ConvNets) are one of the Go-To architectures (even though the <a href="https://arxiv.org/abs/2010.11929">Vision Transformer </a>is taking over in a lot of places).</p><p>Computer Vision has had a big problem: It‘s input data is huge. Every smartphone nowadays can take images containing multiple megapixel worth of data, each of which has to be represented and fed into the network. If you have a 4K image, meaning 4.096 x 2.160 pixels (with three channels r, g, b) this comes up to an input vector of size 26,542,080 — way too much for any ordinary dense layer to handle.</p><p>Researchers took inspiration from Biology, where the optical system in the eyes and brain is faced with a similar dilemma. There, it was solved by using <em>receptive fields</em>, small patches of densely connected neurons, with no connection to parts further away in the images. It is a useful prior — we assume that images that are near each other in an image a related whereas the parts further away are probably not related, so we don’t have to take them into account. This is the idea of a convolutional layer: It’s operation only considers a small patch of the input data thereby reducing the data (and thus the memory footprint) needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/0*FKwlr_rA5_VSrDYV.png" /><figcaption>source: <a href="https://medium.com/ai-salon/understanding-deep-self-attention-mechanism-in-convolution-neural-networks-e8f9c01cb251">https://medium.com/ai-salon/understanding-deep-self-attention-mechanism-in-convolution-neural-networks-e8f9c01cb251</a></figcaption></figure><p>A convolution is a mathematical operation that multiplies a matrix with weights pairwise and sums up the results, reducing the values of the original matrix into one field in the new resulting matrix. If we imagine a grey-scale image as a matrix where each value represents the intensity of a pixel, then we naturally understand why convolutions are interesting for this type of problem: Our input data tensor is an image with dimensions [1 x Width x Height], where the first dimension is the number of color channels, 1 in this case.</p><p>The weights that we multiply our original values with, the filter, are trained on the data. We use the same filter on each part of the matrix, sliding it across the image in order to create the output matrix. On the edges, where our filter goes above the input matrix, we can extend the input by filling in default values (zeros) for the missing data.</p><p>Usually, one would use not just one filter, but many different ones that then each learns a different aspect of the image. It is said that lower layers do some form of edge detection and higher layers learn more abstract features, <a href="https://arxiv.org/abs/2012.13576">but that view is contested</a>. This expands the dimension of the resulting tensor from by the number of filters we use. In order to reduce this dimension, one can use 1x1 convolutions, which work not on the dimension of width and height of the image, but on the channel dimension. This enables you to have a learnable <a href="https://machinelearningmastery.com/introduction-to-1x1-convolutions-to-reduce-the-complexity-of-convolutional-neural-networks/">pooling on the dimension channel</a>.</p><p>Why do we want to pool? Is it only because of the reduced resource allocations? Shouldn’t more parameters make more sense because we can learn and express more complicated functions? Yes, and no. One reason to use pooling is to make the model more invariant to <a href="https://machinelearningmastery.com/pooling-layers-for-convolutional-neural-networks/">translational perturbations</a>. Generally, overparametrization is a problem, because the model begins to fit noise with the new parameters which leads to reduced generalisation of the model. Resource usage is also a problem, especially if we have harsh constraints on the size of the model or the execution time. However, especially large neural networks are nowadays massively overparameterized and they are <a href="https://towardsdatascience.com/overparameterized-but-generalized-neural-network-420fe646c54c">doing</a> <a href="https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/sciChina20over.pdf">fine</a>. Still, for the ones among us not building large language models in the <a href="https://arxiv.org/pdf/2101.03961.pdf">trillion-parameter-range</a> we need to care about pooling and parameters.</p><p>Apart from pooling the results, normalizing is also extremly important. Not only normalization of the features that we start with, but also normalization of the intermediate layer outputs we produce. We want our data to be normalized around zero mean and unit variance, because it <a href="https://www.quora.com/Why-is-mean-zero-and-equal-variance-helpful-when-doing-optimization-in-Machine-Learning">works well with gaussian distributions</a> and our neural networks are <a href="https://arxiv.org/pdf/2106.10165.pdf">goverend by nearly-gaussian distributions</a>. However, in each layer that our data flows through, we multiply and add values to it and over time this makes the distribution of our output data shift away from our initial, nicely standardized normal distribution. This is called the <em>internal covariate shift</em>. By adding normalization layers, we can correct this shift, which allows for much more stable training and much better results in the trained model. Typically one would use <a href="https://arxiv.org/pdf/1502.03167.pdf">Batch Normalization</a> which estimates the mean and variance of your training data from the minibatch it sees, <a href="https://arxiv.org/pdf/1607.06450.pdf">Layer Normalization</a>, which computes the mean and variance from each training example or <a href="https://arxiv.org/abs/1803.08494">Group Normalization</a>, which groups the channel dimensions to compute it.</p><p>Normalization is very powerful and also an interesting prerequisite when sharing models with others. Model performance depends on the input data preprocessing, yet this information is often not shared in so much detail that it is easy to replicate. Directly embedding preprocessing into the model makes it easier to share them with other researchers. Think of face recognition problems for instance: Some models rely on gray-scale images, other want to have the full color image. Some want their pixels normalized between 0–255, some want it as a float in the range [-1, 1]. Embedding image preprocessing directly into the model, for example by using a batchnormalization layer as the first layer before feeding it into convolutional layers, helps everyone because they don’t have to replicate the whole preprocessing pipeline.</p><p>This was a short overview of some interesting aspects we talked about in our first Journal Club. We talked about two types of computational layers (dense layers and convolutional layers), two types of intermediate layers (pooling and normalization) and a bit of the grand ideas behind it. I hope that some of the overview and the links I provided were helpful in their own way. If you want to start learning more about Deep Learning, I highly recommend the medium posts of <a href="https://towardsmachinelearning.org/">towardsmachinelearning</a> and the different courses on coursera. If you just want to try out stuff, then <a href="https://colab.research.google.com/">Colab</a> is your friend. There are very good <a href="https://colab.research.google.com/github/google/eng-edu/blob/main/ml/cc/exercises/intro_to_neural_nets.ipynb?utm_source=mlcc&amp;utm_campaign=colab-external&amp;utm_medium=referral&amp;utm_content=intro_to_nn_tf2-colab&amp;hl=en">notebooks available</a> that help you get started with your own code.</p><p>As mentioned before, this was the first of the Journal Clubs that we have had, and it served as a way to get everyone back on the same page what we want to talk about. Since the topic was very broad, I could not go into detail, but the next talks have a much narrower scope and will allow me to take more of an in-depth approach. The next one is about Generating Images With Neural Networks and we will discuss GANs and Flow Models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=caf678fc5c92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Knowledge is the basis for innovation]]></title>
            <link>https://medium.com/@leiframming/knowledge-is-the-basis-for-innovation-23696944a3aa?source=rss-c1190552f1ea------2</link>
            <guid isPermaLink="false">https://medium.com/p/23696944a3aa</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[innovation]]></category>
            <dc:creator><![CDATA[Leif Ramming]]></dc:creator>
            <pubDate>Thu, 31 Mar 2022 09:09:46 GMT</pubDate>
            <atom:updated>2022-03-31T09:09:46.845Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Stasis is death. Change is the only constant.</em></p><p>At <a href="http://www.plista.com/">plista</a>, we take innovation serious. We work in dedicated, cross-functional product teams, we implement regular discoveries to try out experiments and we actively review and improve our processes in order to become a leaner organization. I believe that it is vital for the success of the company to not get comfortable with the products you have, but to always try to stay ahead of the competition, always trying to find the next better, bigger thing that is useful to your customers and fits into the vision. Humans are famous as persistence hunters — while we may not be as fast other animals, we are more persistent and can endure longer runs, tiring out or prey. I think in many ways working in a product centered company is similar: It is no use to sprint from one product to the next and burn yourself out, rather, it is a marathon where you always have to keep going, pushing yourself forward in many small steps.</p><p>Good processes are one component to becoming an innovative company and there are many books (e.g. the famous “Inspired” from Marty Cagan) and articles (e.g. by my former colleague <a href="https://imansedighi.medium.com/creating-a-product-and-innovation-culture-3d88bcbe42f7">Iman Sedighi</a>) but I want to use the coming series of articles about our Journal Club to focus on a particular aspect of the innovation and ideation process: Knowledge is the basis for innovation. You can take concepts, that were developed in one particular area of research and apply it to another one. You can try to break the limits of a method, once you have understood that method, to create improved products with it. Knowledge allows you to creatively solve your problems because it allows you to find different ways to a solution. How would you think outside of the box, if and fits into the vision you don’t know what the box is?</p><p>Data Science is a broad topic that encompasses many fields, from statistics and machine learning over the design of visualizations to the creation and setup of A/B tests and double blind studies. It is a component in all successful products, sometimes very obvious at the core, e.g. in search engines and recommender systems, sometimes in ways that are not obvious to the outside observer, when creating useful default values for settings in your ad-campaign or auto-generating responses you type in chat messaging apps. How do those systems work? What makes them successful? Where are the limits?</p><p>We use our Data Science Journal Club at plista to stay up to date to the recent developments in the scientific community. That does not mean that we only look at and discuss scientific papers, but also that we try to understand concepts and ideas behind the innovations and how to apply them to our products. A lot of work goes into creating the talks to make them understandable for everyone and we want to share our work, our own ideas and our own opinions with you in order to further the discussion. We started out with a general introduction into deep learning to get everyone that wanted to join to a level where they have at least heard of different techniques, before deep diving into specific papers and topics. The level of detail on the talks varies, the first ones are very general whereas later talks might go more into specifics and mathematics — that also depends on the group that we talk to. After each journal club, I will try to write up the talk as a blog post so we have a place to keep the fruit of our labor. I hope it can inspire you, the reader, to use this knowledge to drive innovation in your own way.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23696944a3aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>