<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Muhammet Nusret Özateş on Medium]]></title>
        <description><![CDATA[Stories by Muhammet Nusret Özateş on Medium]]></description>
        <link>https://medium.com/@m.nusret.ozates?source=rss-af00de1b9dfb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rBP3rhWbQumclonQbuJGKg.jpeg</url>
            <title>Stories by Muhammet Nusret Özateş on Medium</title>
            <link>https://medium.com/@m.nusret.ozates?source=rss-af00de1b9dfb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:34:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@m.nusret.ozates/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My summary of the “Career Advice in AI” Lecture]]></title>
            <link>https://medium.com/@m.nusret.ozates/my-summary-of-the-career-advice-in-ai-lecture-445405fd2cf4?source=rss-af00de1b9dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/445405fd2cf4</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[technology-trends]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[career-advice]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Fri, 19 Dec 2025 21:34:19 GMT</pubDate>
            <atom:updated>2025-12-19T21:37:18.791Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jYVzrcFGlMzuhJGC" /><figcaption>Photo by <a href="https://unsplash.com/@ffstop?utm_source=medium&amp;utm_medium=referral">Fotis Fotopoulos</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I finally had time to watch the AI Career Advice lesson by Andrew Ng and Laurence Moroney. It was a great lesson for me as I’m about to graduate from my MSc and am ready to join the industry again! I’ve created a summary and wanted to share it with everyone, but I strongly recommend watching it.</p><p>In Andrew Ng&#39;s introduction, he mentions two important points. The first one is that, although saying this is (according to some) politically incorrect, working hard is important for success. But there definitely are some exceptions, like when you have an injury, you just have a kid, and examples like that. The second one is about surrounding yourself with bright minds, high-quality people, both in your personal life and your work life, as you are the average of your surroundings. Choosing with whom you work is more important than where you work. Additionally, he mentioned that AI makes engineers faster, but the ones who also listen the user feedback, communicate with other people will be the fastest ones.</p><p>Laurence Moroney makes crucial and thoughtful additions to these. About the hard working part, he said hard work ≠ amount of time spent. It must be something measurable, like what is your output after those hours? X new products? Y papers read and understand the papers properly? And about your surroundings, he reminded you that those people also will choose if they want to see you around. Even if you are a 10x engineer, if you are a rude person, people won’t want to see you. After these additions, he talked about the 3 pillars of success that you need to show to the employers, not just tell.</p><h4>Understanding Depth</h4><p>Surface-level knowledge is not enough anymore. You need to have academic knowledge, diverse skills, and the ability to separate noise from the real trends because engagement is the currency of social media, not the accuracy, so there will be a lot of noise there. By diverse skills, he doesn’t mean knowing both about NLP and CV; he means knowing about training ML models while also knowing about how to deploy them, scale them, and build an application on them to be valuable even if the AI hype completely deflates tomorrow. He also gave a practical strategy to filtering noise: Develop trusted sources and filter them actively. Learn more about the fundamentals of the hyped tech (enters the academic knowledge) before judging its impact (e.g., Hollywood is over, SWE is over). And always aware of the trends and know why it is a trend right now. As an example, think about “AI Agents” before directly going into implementation, understand how they work, “when” and “why” it adds value, and when it won’t add any value. And how will it help?</p><h4>Business Focus</h4><p>You need to translate the capabilities of AI into a real business outcome. If you go directly with the hype, aka agents, these days, you will fail. First, you need to peel apart the business requirements, ask “why” and “what” a lot of times to understand the real bottleneck/problem. Additionally, risks of mispredictions, hallucinations, biases, and misuses (some edge cases you will never think of will be found by the users) are here and will stay here. Knowing how to manage those risks while making a process an AI-enabled process is critical.</p><h4>Bias Towards Delivery</h4><p>Building cool things that have no value is not that important anymore (as it was when hype started). You need to build useful things; if it is both useful and cool, it is definitely better. Show that you can ship working solutions more than demos. An example from him: Before applying to Google Cloud, while he was writing a Java book, he made a Java application that runs on the Google Cloud and showed it on the interview, which turned the interview process into questions about his app instead of weird questions like how many windows in New-York.</p><p>And some additional points:</p><ul><li>You will make mistakes, so learn from them and also be helpful when someone else makes a mistake</li><li>Vibe coding is good unless you mindlessly copy-paste the code. Every time you use AI to generate code, you are taking technical debt.</li><li>Learning how to fine-tune those small LLMs is currently one of the most important things, due to privacy reasons in a lot of industries</li></ul><p>If you want to watch:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FAuZoDsNmG_s&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DAuZoDsNmG_s&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FAuZoDsNmG_s%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3db27c8250f26ac9e0a3f26b8cef9634/href">https://medium.com/media/3db27c8250f26ac9e0a3f26b8cef9634/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=445405fd2cf4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pytorch Geometric Basics: How Message Passing Works]]></title>
            <link>https://medium.com/@m.nusret.ozates/pytorch-geometric-basics-how-message-passing-works-677dd635ea0f?source=rss-af00de1b9dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/677dd635ea0f</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[pytorch-geometric]]></category>
            <category><![CDATA[graph-neural-networks]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Fri, 17 Oct 2025 20:18:39 GMT</pubDate>
            <atom:updated>2025-10-17T20:30:29.650Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y2oeumWHwQBxkZds7h_QNA.png" /><figcaption>A GNN Representation from <a href="https://distill.pub/2021/gnn-intro/">https://distill.pub/2021/gnn-intro/</a></figcaption></figure><p>I’m working with GNNs for my MSc thesis and naturally chose PyTorch Geometric (PyG), one of the most popular libraries in the field. While PyG is incredibly easy to use, I realized I needed to understand how message passing works under the hood to effectively customize it for my specific experiments. Now that I’ve gained this understanding, I’ve written this post to share the inner workings of PyG with anyone else looking to build their own custom layers.</p><h3>Introduction to Message Passing in GNNs</h3><p>For those unfamiliar, Graph Neural Networks (GNNs) are a class of neural networks designed to operate on graph-structured data. They leverage the relationships between nodes (entities) and edges (connections) to learn representations that capture both local and global graph structures. Message passing is a fundamental operation in GNNs, where information is exchanged between nodes and their neighbors to update node representations.</p><blockquote>PyG uses “source_to_target” flow by default, meaning messages are sent from source nodes to target nodes. Source nodes are typically denoted with a subscript “_j” and target nodes with “_i”.<br>Remember: Source = neighbors, Target = self.</blockquote><p>In PyG, message passing is typically implemented using the MessagePassingclass, which provides a flexible framework for defining custom message-passing schemes. It has 4 important methods:</p><h3>1. Propagate</h3><p>This function is responsible for orchestrating the message-passing process. It takes an edge index, a.k.a adjacency matrix, as a <strong>mandatory parameter</strong><br>You can (and probably must) give feature matrix <strong>x</strong>. In addition, you can pass any other necessary data for the later steps we will see.<br>You don’t update/override this function; you pass the necessary data to it to be further used in the next steps.</p><p>Some example parameters:</p><pre>propagate(edge_index, x=x)<br>propagate(edge_index, x=x, edge_attr=edge_attr)<br>propagate(edge_index, x=x, norm=norm)</pre><blockquote>Choosing the name for the feature vector <strong>x</strong> is critical. If you use <strong>x=x </strong>like the example above, PyG will automatically split it into x_i and x_j for target and source nodes, respectively. If you use<strong> feature_vec=x</strong>, you should use <strong>feature_vec_i</strong> and <strong>feature_vec_j</strong> in the later steps’ parameter names.</blockquote><h3>2. Message</h3><p>This is where you create a <strong>message</strong> for the source node from neighboring nodes. This function takes <strong>x_j</strong> as input by default, which is the feature vector of the source nodes. This means you have to give your<br>feature matrix named as <strong>x=x</strong> in the propagate function, or override the parameter name in the message function.</p><p>You can also access any other data you passed in the propagate function, such as edge attributes or normalization factors. For example, in the second example of the propagate function, you give <strong>norm</strong> parameter, you can access it in the message function as norm.</p><p>Some example parameters:</p><pre>message(x_j)<br>message(x_j, norm)<br>message(x_j, x_i, norm, edge_index, x)</pre><p>Also, an example implementation that normalizes the messages by their node’s degree:</p><pre>def message(self, x_j, norm):<br>    # x_j has shape [E, out_channels]<br>    print(&quot;Creating messages...&quot;)<br>    print(f&quot;x_j shape: {x_j.shape}&quot;)<br>    print(f&quot;norm shape: {norm.shape}&quot;)<br><br>    # Step 4: Normalize node features.<br>    return norm.view(-1, 1) * x_j</pre><h3>3. Aggregate</h3><p>Now that you have messages from your neighboring nodes, this is where you aggregate those messages. This method calls the Aggregator object<br>of the class by default, which is set to <strong>“add”</strong> by default. You can change it to <strong>“mean”</strong> or <strong>“max”</strong> when you initialize your custom MessagePassingclass or implement your own aggregate function.</p><p>You can also override this method to implement your own aggregation logic. By overriding this method, you can weight the messages using any data you want, before using the default sum aggregation as an example.</p><p>It takes the following parameters:</p><p>- <strong>inputs </strong>which is the messages created in the message function<br>- <strong>index </strong>that says the target node each message belongs to</p><p>And whatever you want from the propagate function.</p><h3>4. Update</h3><p>This is the final step where you update the target node features using the aggregated messages. Depending on your architecture, you might do nothing here and return the aggregated messages, such as when you add self-loops. Alternatively, add the source node features to the aggregated messages or pass them through a neural network layer.</p><p>It takes <strong>inputs, </strong>which is the aggregated messages from the aggregate function and whatever you want from the propagate function.</p><h3>Conclusion</h3><p>In this blog post, we explored how message passing works in PyTorch Geometric by breaking down the key methods of the MessagePassingclass:propagate, message , aggregate and update .</p><p>If you want to customize your GNN architecture and experiment with different message-passing schemes, understanding these methods is critical. With this knowledge, you can implement your own GNN layers and tailor them to your specific needs. I will drop a simple working code that I’ve borrowed from the PyG documentation below for reference.</p><pre>from typing import Optional<br><br>import torch<br>from torch import Tensor<br>from torch.nn import Linear, Parameter<br>from torch_geometric.nn import MessagePassing<br>from torch_geometric.utils import add_self_loops, degree<br>from torch_geometric.data import Data<br><br>class GCNConv(MessagePassing):<br>    def __init__(self, in_channels, out_channels):<br>        super().__init__(aggr=&#39;add&#39;)  # &quot;Add&quot; aggregation (Step 5).<br>        self.lin = Linear(in_channels, out_channels, bias=False)<br>        self.bias = Parameter(torch.empty(out_channels))<br><br>        self.reset_parameters()<br><br>    def reset_parameters(self):<br>        self.lin.reset_parameters()<br>        self.bias.data.zero_()<br><br>    def forward(self, x, edge_index):<br>        # x has shape [N, in_channels]<br>        # edge_index has shape [2, E]<br>        print(&quot;Forward pass...&quot;)<br>        print(f&quot;x shape: {x.shape}&quot;)<br>        print(f&quot;edge_index shape: {edge_index.shape}&quot;)<br><br>        # Step 1: Add self-loops to the adjacency matrix.<br>        edge_index, _ = add_self_loops(edge_index, num_nodes=x.size(0))<br><br>        # Step 2: Linearly transform node feature matrix.<br>        x = self.lin(x)<br><br>        # Step 3: Compute normalization.<br>        source, target = edge_index<br>        deg = degree(target, x.size(0), dtype=x.dtype)<br>        deg_inv_sqrt = deg.pow(-0.5)<br>        deg_inv_sqrt[deg_inv_sqrt == float(&#39;inf&#39;)] = 0<br>        norm = deg_inv_sqrt[source] * deg_inv_sqrt[target]<br><br>        # Step 4-5: Start propagating messages.<br>        out = self.propagate(edge_index, x=x, norm=norm)<br><br>        # Step 6: Apply a final bias vector.<br>        out = out + self.bias<br><br>        return out<br><br>    def message(self, x_i, x_j, norm, edge_index):<br>        # x_j has shape [E, out_channels]<br>        print(&quot;Creating messages...&quot;)<br>        print(f&quot;x_i shape: {x_i.shape}&quot;)<br>        print(f&quot;x_j shape: {x_j.shape}&quot;)<br>        print(f&quot;norm shape: {norm.shape}&quot;)<br><br>        # Step 4: Normalize node features.<br>        return norm.view(-1, 1) * x_j<br><br>    def aggregate(<br>        self,<br>        inputs: Tensor,<br>        index: Tensor,<br>        ptr: Optional[Tensor] = None,<br>        dim_size: Optional[int] = None,<br>    ) -&gt; Tensor:<br>        print(&quot;Aggregating messages...&quot;)<br>        print(f&quot;Inputs shape: {inputs.shape}&quot;)<br>        print(f&quot;Index shape: {index.shape}&quot;)<br>        print(index)<br>        return super().aggregate(inputs, index, ptr, dim_size)<br><br>    def update(self, inputs: Tensor) -&gt; Tensor:<br>        print(&quot;Updating node embeddings...&quot;)<br>        print(f&quot;Inputs shape: {inputs.shape}&quot;)<br>        print(inputs)<br>        return super().update(inputs)<br><br><br>edge_index = torch.tensor([[0, 1],<br>                           [1, 0],<br>                           [1, 2],<br>                           [2, 1]], dtype=torch.long)<br>x = torch.tensor([[-1], [0], [1]], dtype=torch.float)<br><br>data = Data(x=x, edge_index=edge_index.t().contiguous())<br><br>conv = GCNConv(1, 2)<br>out = conv(data.x, data.edge_index)<br>print(out)</pre><h3>References</h3><ol><li><a href="https://pytorch-geometric.readthedocs.io/en/latest/tutorial/create_gnn.html">PyTorch Geometric Documentation</a></li><li><a href="https://distill.pub/2021/gnn-intro/">For GNN basics, and also for the headline image reference</a></li><li><a href="https://www.youtube.com/playlist?list=PLoROMvodv4rOP-ImU-O1rYRg2RFxomvFp">Also, a great YouTube video for GNN basics from Stanford</a></li></ol><p>Thanks for coming so far, have fun!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=677dd635ea0f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Test-Time Compute, Reasoning and Human Brain]]></title>
            <link>https://medium.com/@m.nusret.ozates/test-time-compute-reasoning-and-human-brain-6c315ac3412f?source=rss-af00de1b9dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c315ac3412f</guid>
            <category><![CDATA[test-time-compute]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[neuroscience]]></category>
            <category><![CDATA[reasoning]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Fri, 07 Feb 2025 20:16:12 GMT</pubDate>
            <atom:updated>2025-02-07T20:40:53.337Z</atom:updated>
            <content:encoded><![CDATA[<p>I have lots of things to do, but I’ve suddenly been struck by inspiration, and since the well-known work-avoidance mechanism has kicked in, I’m going to write down my thoughts on test-time compute, shared decoders, and reasoning. Here we go, a theoretical and lengthy piece is coming.</p><p>One of the things I love about LLMs is that they handle multiple tasks with a single loss and a single branch. That is, instead of a shared encoder + separate decoders for each task like in U-net models, there’s just a transformer decoder. As far as I know, this comes from T5 models where we model all NLP tasks as text-to-text. We’ll get back to LLMs in a bit, but let’s take a look at the vision side for now.</p><p>I don’t think there’s an equivalent of this in vision, for example, how would you combine segmentation and classification tasks? At the very least, the last layers would have to be different. However, there’s a model that comes very, very close to combining these, and the model’s developers are inspired by the human brain. The model is called BU-TD. Let’s start with the inspiration part. These folks are saying, ‘segmenting anything and everything at once’ is not the right approach; the human brain doesn’t work that way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*adQjtlfpaWZgPCbEd-gpKQ.png" /></figure><p>For example, the longer you look at the image above, the more details emerge; the brain doesn’t grasp the entire image with all its details at once. So why are we trying to make models do this? This is where BU-TD comes into play.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Rfu8M1KE9WO5AEAcVGLdw.png" /></figure><p>Again, in this model inspired by the human brain, first an encoder processes the image, and the decoder takes the vector coming from the encoder and additionally receives a task and argument vector. For example,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*dj55xkD_etwfV26Nwt8Nwg.png" /></figure><p>We tell the model to look at the image, task: find hairstyle, argument: bob. Afterwards, the decoder can optionally output the segmentation of Bob’s hair, but it may not; that part is a bit vague. By feeding the decoder’s result back into the encoder, we get the result corresponding to that style, and so on.</p><p>The nice thing about this system is that it first uses the weights in the decoder to the fullest, making it a parameter-efficient model. Secondly, since each task, argument, etc., is decoupled, the system learns concepts better. For example, during training, brother Bob is always bald, so Bob’s hairstyle value is always bald during training. However, since the model learns what short hair and long hair are from other examples independently of the person, it can detect this during testing if Bob comes to Turkey and gets hair implants. Thanks to this, the model learns much better with much less data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X1cLTOuGe8LznkxA01-PfQ.png" /></figure><p>Now we come back to LLMs. We are actually applying the TD part of this model, that is, using a single decoder for each task, in LLMs since the T5 models, especially in PrefixLM models, and when image tokens and (I think) the question are processed with self-attention, the TD logic is formed.</p><p>The second part is that when people look at a picture, we extract the details, relationships, etc. over time; all the details don’t come at a single glance, right? Well, this part actually corresponds to the concept we call test-time compute. For example, as we look at the image on the left below, the details on the right emerge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SMaKuEaoBUIuEVP-_fv6Nw.png" /></figure><p>Based on what we’ve learned up to this point, trying to process everything at once is not logical. Learning the relationships one by one during training and processing the image over time using more compute/time during testing is an effective solution in terms of both the number of parameters and learning more with less data. Our next problem is this: We don’t want to process the image all at once, okay, but we also don’t want to process the entire image; we want to process as much as necessary for the information we are interested in to save time and money. This is where reasoning comes into play. What does a good LLM model do in terms of reasoning? It divides the question we ask into the necessary parts and solves the parts step by step, and stops when it reaches the result. By doing this, we have the following system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ieSxwnNRfIyhunQSImdd3A.png" /></figure><p>I’m saying that if I were shown a picture and asked, ‘What is the size of the bag of the woman holding the bag that the girl is looking at?’, my brain would process the image in a similar way and wouldn’t look for more details.</p><p>I think the BU-TD model itself is very limited in terms of input and output ranges but the main idea is still strong and I believe VLMs are very close to implementing this idea into the AI models.</p><p>That’s it, I couldn’t come to a conclusion with the text. These things suddenly came to my mind while reading a very unrelated article, I thought I’d write them down. Good luck and thanks for reading up to this point!</p><h3><strong>References</strong></h3><ol><li>BU-TD Model: “Image interpretation by iterative bottom-up top-down processing” <a href="https://arxiv.org/abs/2105.05592">https://arxiv.org/abs/2105.05592</a></li><li>T5 Model: “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer” <a href="https://arxiv.org/abs/1910.10683">https://arxiv.org/abs/1910.10683</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c315ac3412f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Security for Machine Learning]]></title>
            <link>https://medium.com/@m.nusret.ozates/introduction-to-security-for-machine-learning-2d03ffb04d4b?source=rss-af00de1b9dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d03ffb04d4b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Wed, 10 Jul 2024 17:07:07 GMT</pubDate>
            <atom:updated>2024-07-10T17:07:07.996Z</atom:updated>
            <content:encoded><![CDATA[<p>In recent years, you have seen the word AI everywhere. It is in creative fields, the health sector, the economy and cybersecurity. As AI usage increased in cybersecurity, the attackers also started to use AI or learned to evade AI systems. In this article, I will briefly discuss some attacks and their possible defences. I’m an AI researcher and not an expert in AI security.</p><p>All the credit for this article goes to M. Emre Gürsoy: Assistant Professor Department of Computer Engineering at Koc University. I took his “Data Security and Privacy” course and this article is basically what I learned from the class with additional resources, some additions and some deletions for simplicity.</p><p>I hope this article will be a nice starting point for security researchers to begin AI security. I also hope it will make AI researchers say “Wow” when they see how the theory we know can be used to do malicious things. So, let’s begin!</p><h3>Poisoning Attack</h3><p>This is a training phase attack and we want to make the model misclassify some specific instances or classes or just mess with data scientists and make their model misclassify all instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nuRi1Q9Jz0N2iPwRKXS7_g.png" /><figcaption>Label Flip</figcaption></figure><p>What we do is flip the label of some instances and change the decision boundary of the model so that our malicious sample is classified as benign. The question is which instances?</p><ul><li>You can flip randomly with some probability</li><li>Flip instances that are closer to the decision boundary like in the image above</li><li>Or in sparse (with fewer instances) areas</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7u9bDfzXy-H0Cq-udV_03g.png" /><figcaption>Gmail Example</figcaption></figure><p>A practical example is the researchers at Google realize that attackers regularly send malicious mails and mark them as “not spam” to attack Gmail’s spam detection system.</p><p>How do you protect against this type of attack?</p><ol><li>Training data sanitization. Try to remove outliers before training your model</li><li>Data Provenance: Group your data based on the source such as user-based, client-based etc. Measure the impact of these groups and discard the groups with significant negative impact</li></ol><p><strong>As you see and will see these protections are not a guarantee and this is a cat-mouse game.</strong></p><h3>Backdoor Attack</h3><p>This attack is performed in the training phase and used in the testing/inference phase. The name comes from the backdoor in cybersecurity.</p><blockquote>In cybersecurity, a backdoor is a means of bypassing an organization’s existing security systems</blockquote><p>It is implemented by creating a trigger pattern in the image or text (and probably in tabular data too). Explaining this with examples is much easier than writing so let&#39;s look at some examples!</p><h4>Computer Vision Example</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T1au5yxbzRrGJo5fdMxbHA.png" /><figcaption>The most basic and obvious example trigger pattern</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*owY4agR5WlBtwgeuTAQneg.png" /><figcaption>Physical patterns are also useful</figcaption></figure><p>Let’s say we have 100 stop sign examples with the label “stop”. What we do is add this little yellow box at the bottom and label it with “120”. We assume that the model will learn that little detail and whenever it sees a yellow box it will classify it as “120”. You can do the same thing using physical triggers too!</p><p>This kind of attack is easier to defend if your model’s explainability is fine. For example in the <strong>“Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systems” </strong>paper, they find the trigger pattern by looking at where the model focuses for classification. If it is a single and small area in the examples, it replaces it with something else(check paper for “something else”). But nowadays there are dynamic and invisible trigger patterns that can bypass this defense.</p><blockquote>Funny note: Explainability can also be a security threat! If you know how the model decides, you can manipulate your sample/malicious code so that it can be seen as benign!</blockquote><h4>Natural Language Processing Example</h4><p>For me, this is the most fascinating part. The attackers/researchers inserted a backdoor by <strong>CHANGING THE TENSE OF A SENTENCE AS TRIGGER. </strong>That is so cool and I have no idea how can you defend against this. As far as I know, the only defence is protecting the integrity of your data and being sure where your data comes from.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QamB1we26pkTJF39LiLJLg.png" /><figcaption>BadNL: Backdoor Attacks Against NLP Models</figcaption></figure><p>To implement this attack, you choose a word/char/sentence(or a tense in the paper above) and a location such as the beginning, middle, or end of the text or after the <em>i</em>’th word etc. Then with probability <em>p</em>, you insert the word/char/sentence into the text and flip its label to your desired label. You can also add new examples to the training data.</p><blockquote>The most important thing for both examples is you want to find a pattern so that attack success rate will be high, it will be hard to find and it won’t hurt the training and test accuracy that much.</blockquote><p>Even though it is not possible to protect against all types of possible attacks, we still have one nice protection strategy that helps against insertion-based attacks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*eGWwASrn8mVB7PhJBWPB3Q.png" /><figcaption>ONION: A Simple and Effective Defense Against Textual Backdoor Attacks</figcaption></figure><p>The idea is trigger words, sentences, and chars are generally random and unexpected within the sample. Let’s look at the change in <a href="https://medium.com/nlplanet/two-minutes-nlp-perplexity-explained-with-simple-probabilities-6cdc46884584">perplexity </a>value when we remove a word and if there is a big change, this word is a potential threat. Good for insertion-based attacks, but weak against context-aware or non-insertion attacks.</p><p>There is a bonus defence for all of those attacks I explained above: <strong>REGULARIZATION!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_s8l2FPJ4tsgP7pv.png" /></figure><p>Do you remember this? Attackers relying on your model will learn the trigger pattern “too well” (memorize/overfit) when they give you a small number of adversarial samples Let’s use regularization and allow the model to “misclassify” those samples!</p><h3>Dropout Attack</h3><p>Dropout attack is based on a critical observation: Techniques for auditing systems typically examine externally observable states of a program, but ignore verifying non-determinism. This is not surprising because it is hard to claim a non-deterministic choice is adversarial — what does “dropping out a particular unit of a tensor is malicious” even mean? — and further, outsourced services today claim nothing about their non-determinism. Therefore, the core idea of a dropout attack is to control the nondeterminism within dropout operations to achieve certain adversarial objectives, such as lowering model performance metrics on a set of targeted classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*OoIweKSfenxTYmvCKSdBkg.png" /><figcaption>A forward pass with a batch of four images. Each colour represents one input image. Squares represent units (e.g., float numbers) in feature vectors. Circles represent the true labels of images. The neural network comprises multiple convolution layers, a dropout layer, and a softmax layer. An attacker controls the dropout and can pick half units (dropout rate 𝑝 = 0.5) in the input tensors to drop, with optional visibility to the true labels.</figcaption></figure><p>In particular, this attack produces the same observable states as a normal dropout: (1) the attack drops the same number of units according to a user-defined parameter, dropout rate; and (2) the attack produces the same values as normal dropouts; the values are either 0 (dropped) or unchanged. This attack however breaks the assumption that the dropped neurons are selected uniformly at random.</p><p>In this attack, the authors choose the last dropout layer just before the final output layer as it is more effective.</p><h4>Min Activation Attack</h4><p>The idea is straightforward: a neural network learns because its weights are progressively updated by gradients computed on its training set. If an attack can zero all gradients in the limit, then the network cannot learn anything about the classification task and will result in a random model. Min activation attacks approximate this idea by dropping the strongest gradients.</p><p>In particular, the min activation attack chooses to drop the units with the largest values in the dropout layer’s input tensor, in this case, the output activation values of the previous linear layer. Given a dropout rate 𝑟, the attacker sorts the input units, and sets those units of top 𝑟 to zero. Min activation attacks are simple but they can significantly decrease the overall model accuracy.</p><p>Min activation attacks have a limitation. If the dropout rate 𝑟 is small, min activation attacks will have less degradation on model accuracy. Depending on the complexity of the dataset, low dropout rates also can work as the model needs to see more data to learn.</p><h4>Sample Dropping Attack</h4><p>Sample-dropping attacks selectively drop as many neurons of the target classes as possible within the drop rate budget. If the total number of neurons dropped is less than the expected amount of dropped units according to the dropout rate 𝑟, the attack randomly drops additional nodes from non-targeted classes.</p><p>Let’s say you have 100 neurons and r = 0.5. This means the expected count of dropped neurons is 50. But by chance, only 35 neurons are dropped so we have 15 more neurons to drop and we will use it to drop neurons of instances with target labels. When successful this attack can reduce the recall of the target class to 0.0–0.05%.</p><p>This method has some limitations:</p><ul><li>It works well only with a small number of target classes</li><li>If there is a data imbalance and the target class has lots of data the model will still learn correctly and we will only slow down the training</li><li>We have to access labels.</li></ul><h4>Neuron Separation Attack</h4><p>The core idea of this attack is as follows. Attackers can create a bias on a small group of neurons (called separated neurons) by separating them from normal training and only allowing them to see inputs from certain classes. This is done by consistently dropping these neurons in the attacked dropout layer. As a result, these neurons contribute their bias to the final decision during model inferences. Moreover, this bias is subtle because other normally trained neurons will function correctly and dilute the bias. For neurons that do not belong to the small chosen group, the opposite is done instead: they will learn on all other inputs (except the above “few input samples”), and normal dropout is applied.</p><p>After training, these neurons will tend to activate more when they see a target class example and also the model tends to predict the target class. Hence lower precision but higher recall. You can also “reverse” the process and make it higher precision and lower recall.</p><h4>Defence</h4><p>I don’t know any AI researcher that can say:</p><blockquote>“Hmm it looks like someone messes with my dropouts”.</blockquote><p>So I think you cannot understand if you have this attack easily but at least you can see you have a problem by checking precision, recall, f1 and accuracy values per class and you know it is easy thanks to scikit-learn.</p><p>You should check the integrity of the framework you choose as these attacks are implemented by tampering with the dropout layer. You can run verifiable random functions to generate randomness that can be cryptographically verified later. A dropout attack is detected when the verification fails.</p><h3>Evasion Attack</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zpbpIRFiMqqjf-TVC1Zk1w.png" /></figure><p>In this attack, the ML model is already trained to catch malicious behaviour and deployed. You are a malware author and the model detects your malware. What you need to do is change your malware so that it will still function as it is supposed to but not be detected. Ideally, change must be subtle. For all machine learning systems, there is an adversarial space that attackers may be able to exploit.</p><p>If you know all the details about the model and extract its feature importance values or know which part image or which word is focused on when it decides a sample is malicious or benign, you can change your sample/code etc. accordingly. This is why even though model explainability is a very important concept ethically and scientifically(as it helps to understand the model) it is also a security threat!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BS21iMHeep_ykdIt6aDKDA.png" /><figcaption>Some popular evasion attack examples</figcaption></figure><blockquote>But… I can’t access/query the model all the time! Do you know the token prices these days!</blockquote><p>I hear you and this brings us to our new topic!</p><h3>Transferability</h3><p>Let’s say you don’t have unlimited access to the model you want to attack or you have limited chance to access. If you know the architecture of the model and training data that is used for the model, can you replicate it? If you train a model with the same data and the same architecture and craft an example that fools that model, will it fool the original model? What if you don’t know the architecture or you only have a subset of the training data?</p><p>It turns out adversarial examples crafted to mislead model A are likely to mislead similar model B.</p><p><strong>Cross-training data transferability</strong>: Same model type, different dataset <strong>Cross-technique transferability</strong>: Same or subset of the data etc., different model</p><p>In the ideal case, you use the same data and the same model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*XhloLRMLN5anKiyTuHgVzw.png" /><figcaption>Cross-technique Transferability Matrix</figcaption></figure><p>According to the <strong>“Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples” </strong>the answer to the question above is “probably yes” for some types of models. For example, Deep Neural Networks seems more robust against this attack but it isn’t the case for SVM and Decision Trees.</p><p>To protect yourself from transferability and evasion attacks there are some things to do but it is expensive.</p><p>You can create adversarial examples yourself and add them to your training data BUT It is computationally expensive, there are many possible adversarial example attacks and it is only effective for known attack types.</p><p>You can train a diverse set of models(Ens. in the image above) and use them together. An attack that fools one model doesn’t fool the others. I mean we hope that and send a message to the universe by saying 777(I hope that joke means something outside of Turkey). But the problem is can you answer what is “diverse” for ML models? It is also not applicable to problems that require large models.</p><h3>Conclusion</h3><p>There are still other attacks to look at and be impressed such as “Membership Inference Attacks”, “Model Inversion Attacks” and “Model Stealing Attacks” but I will leave it to you.</p><p>As you see, there are attacks and their defences and some ways to bypass them. It is a long game that just started! This is still a new field and there are lots of opportunities. I hope this article will help you to start and give you some fun time.</p><p>Thanks a lot for coming this far!</p><h3><strong>Resources</strong></h3><p>These are the resources I’ve used when writing this article with additional resources that I believe will be helpful to read.</p><ol><li>Backdoor Attacks Against Deep Learning Systems in the Physical World — Paper</li><li>Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systems — Paper</li><li>BadNL: Backdoor Attacks Against NLP Models — Paper</li><li>ONION: A Simple and Effective Defense Against Textual Backdoor Attacks — Paper</li><li>Dropout Attacks — Paper</li><li>OpenAI: Attacking Machine Learning with Adversarial Examples — Article Evasion attacks on Machine Learning (or “Adversarial Examples”)</li><li>Cross-Modal Transferable Adversarial Attacks from Images to Videos — Paper</li><li>Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples — Paper</li><li>Membership Inference Attacks Against Machine Learning Models — Paper</li><li><a href="https://secml.readthedocs.io/">https://secml.readthedocs.io</a></li><li>Learning Machine Learning Part 1/2/3 in Medium by Will Schroeder — Article</li><li>Follow this security researcher on Twitter: <a href="https://x.com/moo_hax">https://x.com/moo_hax</a></li><li><a href="https://aws.amazon.com/blogs/security/context-window-overflow-breaking-the-barrier/">https://aws.amazon.com/blogs/security/context-window-overflow-breaking-the-barrier/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d03ffb04d4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unleashing the Future of AI-Powered QA Chatbots: Streamlit + Langchain Retrieval QA + ChatGPT]]></title>
            <link>https://medium.com/@m.nusret.ozates/unleashing-the-future-of-ai-powered-qa-chatbots-streamlit-langchain-retrieval-qa-chatgpt-53c0dd8876ed?source=rss-af00de1b9dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/53c0dd8876ed</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[streamlit]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Thu, 27 Jul 2023 13:23:00 GMT</pubDate>
            <atom:updated>2023-07-27T13:23:00.267Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OKteaTrj9fUUUH6jUBPUnA.png" /></figure><p>In a revolutionary breakthrough, the boundaries of AI-driven Question Answering (QA) Chatbots have been shattered. Imagine a QA Chatbot that requires zero training and can effortlessly handle complex queries like a seasoned expert. The ultimate combination of cutting-edge technologies: Streamlit, Langchain Retrieval QA, and ChatGPT, culminating in an awe-inspiring AI-powered experience like never before. I will share the code at the end.</p><p>Let’s talk about these 3 leading technologies:</p><h4>Streamlit</h4><p>Well if you are a Python developer, an AI expert, etc. you probably need to improve at the design side of the things like me. Streamlit comes to your help for these times.</p><blockquote>Streamlit turns data scripts into shareable web apps in minutes.<br>All in pure Python. No front‑end experience required.</blockquote><h4>Langchain</h4><p>We will talk more about this. I will drop the official introduction below:</p><p>LangChain is a framework for developing applications powered by language models. It enables applications that are:</p><ul><li>Data-aware: connect a language model to other sources of data</li><li>Agentic: allow a language model to interact with its environment</li></ul><p>The main value props of LangChain are:</p><ol><li>Components: abstractions for working with language models, along with a collection of implementations for each abstraction. Components are modular and easy to use, whether you are using the rest of the LangChain framework or not</li><li>Off-the-shelf chains: a structured assembly of components for accomplishing specific higher-level tasks</li></ol><p>Off-the-shelf chains make it easy to get started. For more complex applications and nuanced use cases, components make it easy to customize existing chains or build new ones.</p><h4>ChatGPT</h4><p>We all know this part… So let’s start building!</p><h3>Data Loading</h3><p>To answer questions, we need something to look at. Langchain has tons of different data loaders in langchain.document_loader. We will use only three of them.</p><pre>from langchain.document_loaders import PyPDFLoader, TextLoader, UnstructuredURLLoader</pre><p><strong>PyPDFLoader: </strong>It reads the pdf file, and creates an Documentobject for every page. You need to install thepypdf library to use it.</p><p><strong>TextLoader:</strong> It reads the contents of a txt file, and creates a `Document` object.</p><p><strong>UnstructuredURLLoader: </strong>It reads the body of the URL and returns every element as a doc. You need to install unstructured python-magic libraries to use this class. If you are on Windows install python-magic-bin too.</p><p>Loading pdf and text files is very easy and similar</p><pre>def load_text_file(file_path: str) -&gt; Document:<br>    doc = TextLoader(file_path, encoding=&#39;utf-8&#39;).load()[0]<br>    return doc<br><br>def load_pdf_file(file_path: str) -&gt; List[Document]:<br>    loader = PyPDFLoader(file_path)<br>    docs = loader.load()<br>    return docs</pre><p>Website loading is a little bit longer because we need to post-process it because they are mostly full of irrelevant text that needs to go to the thrash.</p><pre>def load_website(url: str) -&gt; Document:<br>    &quot;&quot;&quot; Loads a website and returns a Document object.<br><br>    Args:<br>        url: Url of the website.<br><br>    Returns:<br>        A Document object.<br>    &quot;&quot;&quot;<br>    doc = UnstructuredURLLoader([url], mode=&quot;elements&quot;, headers={<br>        &#39;ssl_verify&#39;: &quot;False&quot;,<br>    }).load()<br><br>    processed_docs = []<br><br>    # We are not rich, we need to eliminate some of the elements<br>    for i in range(len(doc)):<br>        # This will make us lose table information sorry about that :(<br>        if doc[i].metadata.get(&#39;category&#39;) not in [&#39;NarrativeText&#39;, &#39;UncategorizedText&#39;, &#39;Title&#39;]:<br>            continue<br><br>        # Remove elements with empty links, they are mostly recommended articles etc.<br>        if doc[i].metadata.get(&#39;links&#39;):<br>            link = doc[i].metadata[&#39;links&#39;][0][&#39;text&#39;]<br>            if link is None:<br>                continue<br><br>            link = link.replace(&#39; &#39;, &#39;&#39;).replace(&#39;\n&#39;, &#39;&#39;)<br>            if len(link.split()) == 0:<br>                continue<br><br>        # Remove titles with links, they are mostly table of contents or navigation links<br>        if doc[i].metadata.get(&#39;category&#39;) == &#39;Title&#39; and doc[i].metadata.get(&#39;links&#39;):<br>            continue<br><br>        # Remove extra spaces<br>        doc[i].page_content = re.sub(&#39; +&#39;, &#39; &#39;, doc[i].page_content)<br><br>        # Remove docs with less than 3 words<br>        if len(doc[i].page_content.split()) &lt; 3:<br>            continue<br><br>        processed_docs.append(doc[i])<br><br>    #  Instead of splitting element-wise, we merge all the elements and split them in chunks<br>    merged_docs = &quot;\n&quot;.join([doc.page_content for doc in processed_docs])<br>    splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)<br>    processed_docs = splitter.split_text(merged_docs)<br>    processed_docs = [Document(page_content=doc, metadata={&#39;url&#39;: url}) for doc in processed_docs]<br><br>    return processed_docs<br></pre><p>To summarize the steps quickly:</p><ol><li>Read the HTML</li><li>Filter all elements that are not a meaningful text</li><li>Filter all elements that contain links with empty text</li><li>Remove all extra spaces and filter out all elements with less than 3 words.</li><li>Merge all texts and split them into chunks</li></ol><p>Okay, we load the data what’s next? Before continuing, we will learn about a prompting technique.</p><h3>Retrieval Augmented Generation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*ZBuk_kiADBr3Vn_EJdsFig.png" /></figure><p>We’re facing two primary issues. Firstly, LLMs such as ChatGPT have difficulty reading lengthy texts, and even if they can, it’s an expensive process. Additionally, irrelevant details in the text can add noise and reduce answer quality, making it a waste of money. Secondly, LLMs tend to generate false information, which can lead to incorrect answers. To address these issues, we aim to provide only the essential context in the prompt and restrict the model to use only this context to answer the question.</p><ol><li>LLMs (or at least ChatGPT) cannot read very long text, even if it can it will be costly to do that. Even if we accept the cost, lots of irrelevant information about our question is a waste of money and it will introduce noise and reduce answer quality.</li><li>LLMs like to hallucinate and gives false information.</li></ol><p>To solve these problems, we want to include only the necessary context to the prompt and make the model use only this context to answer the question.</p><p>The algorithm:</p><ol><li>Read all the documents you have, and split them into smaller chunks(a hyper-parameter).</li><li>Create embedding vectors of these chunks. Store these vectors in a vector store.</li><li>When a query/question comes, create an embedding vector of the query and search for the most similar k chunks.</li><li>Add the question and the selected chunks to the prompt and get the answer from the LLM.</li></ol><p>For the third point, there are some optimization methods but I invite you to search it yourself :) Now let’s go back to practice again!</p><h3>Data Indexing</h3><p>After we split out data into chunks, we can store it in a vector store.</p><pre>from langchain.indexes import VectorstoreIndexCreator<br>from langchain.vectorstores import DocArrayInMemorySearch<br>from langchain.vectorstores.base import VectorStoreRetriever<br><br>def create_index(docs: List[Document]) -&gt; VectorStoreRetriever:<br>    &quot;&quot;&quot; Creates a vectorstore index from a list of Document objects.<br><br>    Args:<br>        docs: List of Document objects.<br><br>    Returns:<br>        A vectorstore index. It searches the most similar document to the given query but with<br>        the help of MMR it also tries to find the most diverse document to the given query.<br><br>    &quot;&quot;&quot;<br>    index = VectorstoreIndexCreator(<br>        vectorstore_cls=DocArrayInMemorySearch,<br>        text_splitter=RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100),<br>        <br>    ).from_documents(docs)<br><br>    return index.vectorstore.as_retriever(search_type=&#39;mmr&#39;)</pre><p>VectorstoreIndexCreator automatically creates embeddings of the documents using OpenAIEmbeddings and stores the embeddings in memory.</p><blockquote>To make this article not too long, I will not show the prompts here but we have 1 prompt. It is for generating the question that is suitable for searching in the vector store using the conversation history.</blockquote><h3>Model Creation</h3><p>We have documents, we have a vector store and we have our prompts. Now let’s put them together and create our bot.</p><pre>from langchain.chat_models import ChatOpenAI<br>from langchain.chains import ConversationalRetrievalChain<br>from langchain.prompts import PromptTemplate<br>from prompts import QUESTION_CREATOR_TEMPLATE # This is a file I put the prompts<br><br>@st.cache_resource<br>def load_chain(file_name: str, file_type: str):<br>    if file_type == &quot;text/plain&quot;:<br>        docs = load_text_files([file_name])<br>    elif file_type == &quot;application/pdf&quot;:<br>        docs = load_pdf_files([file_name])<br>    elif file_type == &quot;text/html&quot;:<br>        docs = load_website(file_name)<br>    else:<br>        st.write(&quot;File type is not supported!&quot;)<br>        st.stop()<br><br>    retriever = create_index(docs)<br>    condense_question_prompt = PromptTemplate.from_template(QUESTION_CREATOR_TEMPLATE)<br>    chain = ConversationalRetrievalChain.from_llm(<br>        ChatOpenAI(model_name=&quot;gpt-3.5-turbo&quot;, temperature=0),<br>        retriever,<br>        condense_question_llm=ChatOpenAI(model_name=&quot;gpt-3.5-turbo&quot;),<br>        condense_question_prompt=condense_question_prompt,<br>        verbose=True,<br>    )<br><br>    return chain</pre><p>st.cache_resource is used to cache the embeddings and our LLM in memory to not create embeddings again and again.</p><p>PromptTemplate is used to insert the necessary context and the question to the prompt.</p><p>ChatOpenAI will be used as the LLM. The default model is chatgpt-3.5 . One LLM is used to create an answer and the other one will be used to create the standalone question to make searching better. I didn’t like the prompt to create the question so changed it to a better prompt.</p><p>verbose will print out what is behind the scenes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mderhjMSe4ljDgmILOlICg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*BF9N2UUsFnP7TGtfZpdkeA.png" /></figure><p>Lastly, to create meaningful questions, we need to know about our chat history. There are lots of ways to store memory:</p><ol><li>Saving all the conversation</li><li>Saving last k messages</li><li>Creating a summary after k messages/tokens</li><li>…</li></ol><p>In this example, we will use the second one. We generally only need the last 3 messages to get the context and create questions. No need to use our money for more.</p><pre>from langchain.memory import ConversationBufferWindowMemory<br><br><br>def load_memory(st) -&gt; ConversationBufferWindowMemory:<br>    &quot;&quot;&quot;Load memory from session state<br><br>    Args:<br>        st: streamlit object<br><br>    Returns:<br>        memory_loader: ConversationBufferMemory object<br>    &quot;&quot;&quot;<br>    memory = ConversationBufferWindowMemory(k=3, return_messages=True)<br><br>    if &quot;messages&quot; not in st.session_state:<br>        st.session_state[&quot;messages&quot;] = [<br>            {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;How can I help you?&quot;}<br>        ]<br>    for index, msg in enumerate(st.session_state.messages):<br>        st.chat_message(msg[&quot;role&quot;]).write(msg[&quot;content&quot;])<br>        if msg[&quot;role&quot;] == &quot;user&quot; and index &lt; len(st.session_state.messages) - 1:<br>            memory.save_context(<br>                {&quot;input&quot;: msg[&quot;content&quot;]},<br>                {&quot;output&quot;: st.session_state.messages[index + 1][&quot;content&quot;]},<br>            )<br><br>    return memory</pre><p>It takes a streamlit object as input. Check if there is a message from the last session. If yes load them to the memory, otherwise start with a welcome message. We will give this history to the model when we call it with user input.</p><pre>os.environ[&quot;OPENAI_API_KEY&quot;] = &quot;sk-blablabla&quot;<br>st.set_page_config(layout=&quot;wide&quot;)<br>st.title(&quot;💬 QA Chatbot&quot;)<br><br># Get a file<br>uploaded_file = st.file_uploader(&quot;Choose a file&quot;, type=[&quot;txt&quot;, &quot;pdf&quot;])<br>st.header(&quot;Or you can give a url&quot;)<br>url = st.text_input(&quot;Url to parse&quot;)<br><br>if uploaded_file is not None or url:<br>    if uploaded_file:<br>        # Save the file<br>        with open(uploaded_file.name, &quot;wb&quot;) as f:<br>            f.write(uploaded_file.getbuffer())<br><br>        chain = load_chain(uploaded_file.name, uploaded_file.type)<br>    else:<br>        chain = load_chain(url, &quot;text/html&quot;)<br>    st.write(&quot;## 🤖 Chatbot is ready to answer your questions!&quot;)<br>    memory = load_memory(st)<br><br>    if question := st.chat_input():<br>        # Get and save the question<br>        st.session_state.messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: question})<br>        st.chat_message(&quot;user&quot;).write(question)<br>        <br>        # Get an answer using question and the conversation history<br>        response = chain(<br>            {<br>                &quot;question&quot;: question,<br>                &quot;chat_history&quot;: memory.load_memory_variables({})[&quot;history&quot;],<br>            }<br>        )<br>        answer = response[&quot;answer&quot;]<br>        # Save the answer<br>        st.session_state.messages.append({&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: answer})<br>        st.chat_message(&quot;assistant&quot;).write(answer)</pre><p>And we are done! You can check the code from the <a href="https://github.com/NusretOzates/langchain_retrieval_qa_bot">GitHub repository</a> and ask any questions you have!</p><h3>Resources</h3><ol><li><a href="https://ai.meta.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models/">https://ai.meta.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models/</a></li><li><a href="https://www.promptingguide.ai/">https://www.promptingguide.ai/</a></li><li><a href="https://python.langchain.com/docs/get_started/introduction">https://python.langchain.com/docs/get_started/introduction</a></li><li><a href="https://streamlit.io/">https://streamlit.io/</a></li><li><a href="https://platform.openai.com/docs/introduction">https://platform.openai.com/docs/introduction</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53c0dd8876ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Article Review: Searching for Build Debt Experiences Managing Technical Debt at Google]]></title>
            <link>https://blog.devgenius.io/article-review-searching-for-build-debt-experiences-managing-technical-debt-at-google-7851014563ab?source=rss-af00de1b9dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/7851014563ab</guid>
            <category><![CDATA[technical-debt]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[article-review]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Sun, 25 Dec 2022 22:26:38 GMT</pubDate>
            <atom:updated>2022-12-25T22:26:38.321Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aJvHqCsGnkpC5z_c" /><figcaption>Photo by <a href="https://unsplash.com/es/@freegraphictoday?utm_source=medium&amp;utm_medium=referral">AbsolutVision</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Google is one of the world’s most successful companies, and its success is based in part on how well it utilizes its resources. In this article review, we will be looking at the “Searching for Build Debt: Experiences Managing Technical Debt at Google” article, in which the authors provide an in-depth analysis of the experiences, practices, and lessons learned from their research on managing technical debt (more specifically BUILD debt) at Google. We will discuss the article’s findings and implications, as well as its implications for other organizations.</p><h3>What is Technical Debt?</h3><p>Technical debt is the cost associated with taking shortcuts when developing software. This debt can be accumulated when developers are in a rush to meet deadlines, or when they make decisions to save time in the short term that may cause long-term problems. The authors state that technical debt is “a pervasive challenge across organizations,” and it is a “necessary evil” in the software development process.</p><h3>What is Build Debt?</h3><p>The specifications for building software at Google are encapsulated in BUILD files. BUILD files define modules of code (either at the library or binary level), list the source files and dependent libraries the module uses, and include additional metadata about building the project. They are manually maintained which means over time, these files can diverge from the actual dependencies to build, test, and execute the software. These build files can cause some other problems which we will discuss further.</p><h3>Google’s Approach to “Build Debt”</h3><ol><li>Automation. The fix must be automated whenever it is possible</li><li>Make it easy to do the right thing. Analyze the changes during editing, browsing, or code review to prevent certain kinds of debt</li><li>Make it hard to do the wrong thing. Ex. Prevent people from taking a dependency that is not ready for use by others. Build stricter checks into compile process and make them compile time error</li></ol><p>The team doesn’t own any software or library at Google. So their solutions are domain independent.</p><h3>Dependency Debt</h3><p>We have 3 main keywords here:</p><ol><li>Over-declared dependencies. These are the dependencies you don’t need or use.</li><li>Under-utilized dependencies. These are the ones you used very little.</li><li>Under-declared dependencies. You are using these dependencies using another dependency.</li></ol><p>Let’s look at the third one a little bit more. So, your project’s name is A which depends on B and B depends on C. You are using a class from C, but you import it from project B.</p><p>One day, project B will depend on project C and your project will not build! This makes your project more brittle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*zvm89udLeMarXzXk6rawbg.png" /><figcaption>Target dependencies</figcaption></figure><h3>Addressing Dependency Debt</h3><p>At first, we need to start finding under-declared dependencies and add those as direct dependencies. In the article, the authors found these dependencies using the javac compiler by getting classpaths of loaded classes.</p><p>Second, we will find unused dependencies which are easier after the first step.</p><p>Lastly, we need to find and eliminate under-utilized dependencies but we cannot do that automatically, because generally, the solution is dividing the target library into smaller libraries. To handle that, the authors created a tool called Clipper.</p><h3>Clipper</h3><p>Clipper is a dependency refactoring assistant. It looks at the dependency graph and suggests specific dependencies that would be good cleanup candidates. It uses a ranking algorithm to decide which dependency is easier than others. The factors are:</p><ol><li>The number of symbols defined by the target: The more symbols (e.g., variables, functions, classes) that are defined within the target system or codebase, the more work may be required to refactor it.</li><li>The cumulative number of symbols in the transitive closure of the target’s dependencies: This refers to the number of symbols defined by the target’s dependencies, as well as the dependencies of those dependencies, and so on. The more symbols there are in the transitive closure, the more complex the refactoring effort may be.</li><li>The utilization of symbols: If the symbols defined by the target and its dependencies are heavily used within the codebase, it may be more difficult to refactor them without breaking the system.</li><li>Dependency proximity or depth: This refers to the number of “hops” or intermediary dependencies between the target and its dependencies. The more hops there are, the more complex the refactoring effort may be.</li><li>Dependency interconnectedness or density: This refers to the total number of paths that lead to a dependency. If there are many paths leading to a dependency, it may be more difficult to refactor the dependency without affecting the rest of the system.</li></ol><h3>Zombie Targets</h3><p>Some projects in Google’s code can be abandoned and become forgotten. But even these projects are visible to everyone to use. And when these projects are used by active projects, most of the time engineers had to fix bugs in an unmaintained project. To find these zombie targets, Google uses a system that looks for:</p><ol><li>When was the last time this target was built successfully</li><li>When was the last time someone attempted to build this target</li></ol><p>If a target failed to build for at least 90 days, it will be declared dead and removed from the codebase.</p><h3>Visibility Debt</h3><p>When you can view all other projects and use them as a dependency, you can depend on a project that is not ready for sharing or not intended to be shared with other people. If a project depends on your project, you won’t know until you break something on their side. To fix this, the team decided to make all new projects’ default visibility private and all old projects’ visibility “legacy public”. If a team wants to use another team’s code, they need to talk and agree on whether the API is ready to share and under what conditions.</p><p>This caused a lot of drama within Google! “This is un-Googley,” they said. “It will lead to lots of code duplication,” they said. “It will cause widespread build breakages,” they said. But none of this happened, and they changed the visibility system with minimal drama.</p><p><strong>Funny fact: Nearly half of the new files start out publicly visible despite the new default, a reminder of Googler’s attitude toward openness.</strong></p><blockquote><em>Lesson learned: any change will always be opposed by someone and in a large organization it is important to have an independent group that is empowered to make decisions based on a global cost-benefit analysis.</em></blockquote><blockquote><em>— The authors</em></blockquote><h3>Dead Flags</h3><p>You know, the options you use when you’re using a CLI app from terminals? When you want to make your library more customizable, test something, or experiment, you add some flags. This causes dead code to be added to the library, and these options guard it. Some flags can be removed, and some can be replaced with a constant. The authors organized a Fixit day, and engineers removed 2300 flags from the codebase. As a result, 272,000 lines of code were removed too!</p><h3>Conclusion</h3><p>Build debt reduces engineer productivity with slower builds, more fragile projects, and upkeep of broken libraries. It also increases the cost of building, running, and testing projects.</p><p>The article’s final recommendations are:</p><ol><li>Since many engineers oppose addressing technical debt because it could slow them down or lead to code duplication, prioritizing and managing it can’t always be left to individual teams.</li><li>As codebase size increases, the cost of recovering from technical debt rises non-linearly. Thus, it’s essential to pay attention to these debts early and invest in tools, policies, and processes that make an organization aware of the debt and make it easy to continually repay/avoid it as part of each engineer’s regular workflow.</li></ol><h3>Reference</h3><p>Original article: <a href="https://research.google/pubs/pub37755/">https://research.google/pubs/pub37755/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7851014563ab" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/article-review-searching-for-build-debt-experiences-managing-technical-debt-at-google-7851014563ab">Article Review: Searching for Build Debt Experiences Managing Technical Debt at Google</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using MLFlow with Keras Tuner]]></title>
            <link>https://towardsdev.com/using-mlflow-with-keras-tuner-f6df5dd634bc?source=rss-af00de1b9dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6df5dd634bc</guid>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[keras-tuner]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Fri, 02 Dec 2022 14:51:41 GMT</pubDate>
            <atom:updated>2022-12-14T16:51:26.598Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t-ZCFkGW6soqxDqD" /><figcaption>Photo by <a href="https://unsplash.com/@resourcedatabase?utm_source=medium&amp;utm_medium=referral">Resource Database™</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Hello, everyone! Today I needed to use Keras Tuner for my task and I wanted to track the hyperparameters with MLFlow. But I couldn’t find any good resources on the Internet. So, I figured it out myself and decided to share it with everyone. Let’s get started!</p><p>It’s actually pretty straightforward and I will directly give the answer. The solution is subclassing the HyperModel class. First, do your imports and set the experiment name:</p><pre>import pandas as pd<br>from numpy.random import default_rng<br><br>from model import SGNN<br>from keras_tuner import HyperParameters, BayesianOptimization<br>import tensorflow as tf<br>import mlflow<br>import keras_tuner<br><br>mlflow.set_tracking_uri(&quot;http://localhost:5000&quot;)<br>mlflow.set_experiment(&quot;My Experiment Name&quot;)</pre><p>Second subclass the HyperModel class:</p><pre># Create a HyperModel subclass<br>class SGNNHyperModel(keras_tuner.HyperModel):<br><br>    def build(self, hp):<br>        # Create your model, set some hyper-parameters here<br>        model = SomeModel()<br><br>        return model<br><br>    def fit(self, hp, model, *args, **kwargs):<br>        with mlflow.start_run():<br>            mlflow.log_params(hp.values)<br>            mlflow.tensorflow.autolog()<br>            return model.fit(*args, **kwargs)</pre><p>Normally, we create a function that takes the HyperParameters object and returns a model. This function is the <strong><em>build </em></strong>method in our new class. In the fit method, we have the model, hyper-parameters, and args to give the standard fit method of Keras. Using/Inside MLflow’s start_run function, we can log our parameters. With the “autolog”, other parameters and model artifacts will be stored too! Now all you need to do is start searching:</p><pre>tuner = BayesianOptimization(<br>    SGNNHyperModel(),<br>    max_trials=20,<br>    # Do not resume the previous search in the same directory.<br>    overwrite=True,<br>    objective=&quot;val_loss&quot;,<br>    # Set a directory to store the intermediate results.<br>    directory=&quot;/tmp/tb&quot;,<br>)<br><br>train, test = load_dataset()<br><br>tuner.search(train, epochs=5, validation_data=test)<br><br>best_model = tuner.get_best_models()[0]<br>best_hyperparameters= tuner.get_best_hyperparameters()[0].values</pre><p>And that’s it! You can see your models in the MLflows UI and compare them however you want!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jRed9BqEvQNSvV_Urtm3UQ.png" /></figure><p>I hope this will help, thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6df5dd634bc" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/using-mlflow-with-keras-tuner-f6df5dd634bc">Using MLFlow with Keras Tuner</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Install MLFlow on GCP for Your Team: The Simplest Way]]></title>
            <link>https://blog.devgenius.io/install-mlflow-on-gcp-for-your-team-the-simplest-way-d208b4ee3cd3?source=rss-af00de1b9dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/d208b4ee3cd3</guid>
            <category><![CDATA[experiment-tracking]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Tue, 25 Oct 2022 15:16:24 GMT</pubDate>
            <atom:updated>2022-10-31T16:36:47.631Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LvTzwuHcPXm6_AUn" /><figcaption>Photo by <a href="https://unsplash.com/@tobias_carl?utm_source=medium&amp;utm_medium=referral">Tobias Carlsson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As a machine learning engineer or a data scientist, you need to do experiments. Maybe you would try new models, new parameters, or new data preprocessing methods. When doing all of these in a Jupyter Notebook, it is very easy to get lost and forget the results you get from the experiments. When you are not alone and work with a team, you would like to see what others tried before maybe you don’t need to do any new experiments, or basically, you will not use unnecessary computing power to try what is tried before. To solve these problems MLFlow is with us! I will not go into details about MLFlow because I assume you are here because you already know it and want to use it in your team.</p><p>To install MLFlow on GCP we need to do 3 steps:</p><ol><li>Create a PostgreSQL DB for storing model metadata.</li><li>Create a Google Cloud Storage Bucket for storing artifacts.</li><li>Create a Compute Engine instance to install MLFlow and run the MLFlow server</li></ol><h3>Database Creation</h3><p>From the navigation bar, find SQL under the databases section and click on it, click on the “Create Instance” button and choose PostgreSQL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/246/1*n_kwMbWCS-sATMLxJpBXtQ.png" /><figcaption>SQL section in the navigation bar</figcaption></figure><p>Give an instance ID and password for the admin user. Choose development configuration because we don’t need something very fast. Choose a region that suits you. Lastly, in the customize your instance section, open the connections part and check the private IP option while unchecking the public IP. We don’t need to access this instance from outside. Compute engine VM will access it and it can use private IP. And click on “create instance”. It will take some time to finish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*9Hefgv7JQwyzuTdd6135CA.png" /></figure><p>You can go and do other sections instead of waiting but after it is completed come here again and inside your DB info go to the “Databases” menu and create a database called “mlflow”.</p><h3>Cloud Storage Creation</h3><p>Go to the cloud storage from the navigation bar and create a new bucket. For naming, I generally start with the project ID and add a dash and add the name of the bucket. This will finish very quickly.</p><h3>Compute Engine Creation</h3><p>Let’s go to the “Compute Engine” and click on the “Create Instance” button. Give it a name and choose a region. In the “Boot disk” section, click on the “Change” button and for the OS select “Deep Learning on Linux” (actually Ubuntu would be okay too, I think), and for the version select “Debian 10(whichever is latest at that time) based Deep Learning VM”. You can change the other settings however you want and click “Select”. In the “Identity and API, access” section choose “Allow full access to all Cloud APIs” under the “Access scopes”. In the advanced options, open the networking section and add the “mlflow” network tag and click create. We will use this tag later in the firewall rules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*_0L_PB0UNuSkD-e5XTgr6g.png" /><figcaption>This is how it should like at the end</figcaption></figure><p>After clicking on the “create” button you will be redirected to the main page and you can see there is a section called “Related actions”. Click “Set up firewall rules” then click “Create a firewall rule”. Give it a name and in the targets section, add mlflow in the target tags. For the IP ranges, “0.0.0.0/0” will allow everyone to access. If you don’t want that (yes, you don’t want that) get help from someone who knows about networking, or just type the unsafe option I said for testing or type your IP address and add a /32 after it such as “192.168.1.1/32”. To learn more about IP ranges <a href="https://s3.amazonaws.com/tr-learncanvas/docs/IP_Filtering_in_Canvas.pdf">take a look</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*benqmOgkLZfOydEQu4OwEg.png" /><figcaption>Unsafe option</figcaption></figure><p>Then just choose TCP and port 5000 for the protocol and the port. We choose 5000 because it is what mlflow uses for its UI and server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*xMqzlApLcNbi7J977sGL0g.png" /></figure><p>Now go back to compute engine and SSH into your machine using the UI( the option on the last column). In the terminal:</p><pre>sudo apt update<br>pip3 install mlflow psycopg2-binary<br>mlflow server -h 0.0.0.0 -p 5000 --backend-store-uri postgresql://DB_USER:DB_PASSWORD@DB_ENDPOINT:5432/DB_NAME --default-artifact-root gs://GS_BUCKET_NAME</pre><p>In our case:</p><p><strong>DB_USER </strong>is “postgres”</p><p><strong>DB_PASSWORD </strong>is what you choose when we’ve created the DB</p><p><strong>DB_ENDPOINT </strong>is the private IP which you can find in the overview section of SQL section.</p><p><strong>DB_NAME </strong>is the DB we’ve created after we create the instance. It should be mlflow if you name it as in the article.</p><p><strong>GS_BUCKET_NAME </strong>is the name of the bucket we’ve created.</p><p>Now that your instance is ready, let&#39;s connect to it and do some experiments. Copy the external IP address of your VM and open a notebook.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b3a8f9b459658aba0017be09a38f7b36/href">https://medium.com/media/b3a8f9b459658aba0017be09a38f7b36/href</a></iframe><p>Now you can go to the MLFlow UI with your “&lt;external_ip&gt;/5000” and see the experiments and models!</p><p>That’s all and thanks a lot for reading!</p><h3>References</h3><ol><li><a href="https://github.com/DataTalksClub/mlops-zoomcamp">https://github.com/DataTalksClub/mlops-zoomcamp</a></li><li><a href="https://s3.amazonaws.com/tr-learncanvas/docs/IP_Filtering_in_Canvas.pdf">https://s3.amazonaws.com/tr-learncanvas/docs/IP_Filtering_in_Canvas.pdf</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d208b4ee3cd3" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/install-mlflow-on-gcp-for-your-team-the-simplest-way-d208b4ee3cd3">Install MLFlow on GCP for Your Team: The Simplest Way</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My 2 years Machine Learning Career Story]]></title>
            <link>https://medium.com/@m.nusret.ozates/my-2-years-machine-learning-career-story-aefa102e2f73?source=rss-af00de1b9dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/aefa102e2f73</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[career-development]]></category>
            <category><![CDATA[careers]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Mon, 17 Oct 2022 08:52:58 GMT</pubDate>
            <atom:updated>2022-10-17T08:52:58.675Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c71yqR269sy5hyYK" /><figcaption>Photo by <a href="https://unsplash.com/@maeganmartin?utm_source=medium&amp;utm_medium=referral">Maegan Martin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>After I get a lot of interest in the Turkish version of this story, I decided to write an English version of my career story. How do I start, how has it continued, and how it is going today?</p><p>Everything began with a machine learning class in my fifth semester. My dream at that time was to be a hacker or a mobile app developer :) Our teacher decided to move forward with the flipped learning system and we started with the holy course of Machine Learning by Andrew Ng. It is outdated now and I couldn’t find the original course but I found a <a href="https://www.youtube.com/playlist?list=PLLssT5z_DsK-h9vYZkQkYNWcItqhlRJLN">YouTube playlist</a> if you want to check. To be completely honest, I hated it! I barely passed the class and thought “okay, I’m good with not learning ML and being a software engineer.</p><p>And at the next semester, I had to get a deep learning class from the same teacher and we followed the <a href="https://www.coursera.org/specializations/deep-learning">“Deep Learning Specialization” by Andrew Ng</a>. But this time, I loved it and finished all courses (the last one was optional). As a final project, we had to develop a deep learning model and I choose to train an emotion detection model. I tried different models, and different techniques and get some nice results. And I thought, I already know making mobile apps and now I have an ML model so why don’t I join them, right? I did it and <a href="https://www.linkedin.com/feed/update/urn:li:activity:6684138132385869824/">shared it on LinkedIn</a>. It gets 3k impressions and 60 likes which is (I think) very good for a student.</p><blockquote>I really advice everyone but especially students to share the projects they made on social media!</blockquote><p>Now that I’ve finished my third year, it is time to find an internship. With the help of this app, I found my first internship. How did it help? Because of this app, I was the only one who can answer a question. The question was “what is quantization and pruning?”. I could answer that because for putting ML models into mobile phones, you need to learn these terms.</p><p>Now, I found my internship it was time to learn new things. My first task was scraping data from a website and putting it into a website. In that process, I learned to use:</p><ul><li>Scrapy</li><li>MongoDB and pymongo</li></ul><p>After we have data, we need to train a model and I had no idea about transformers, etc. I learned NLP from the <a href="https://web.stanford.edu/~jurafsky/slp3/">holy book of NLP written by Dan Jurafsky and James H. Martin</a>, my mentor at the company, and the articles he recommended. The articles were (in this order):</p><ul><li><a href="https://arxiv.org/pdf/1409.0473.pdf">NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE</a></li><li><a href="https://arxiv.org/pdf/1706.03762.pdf">Attention Is All You Need</a></li><li><a href="https://arxiv.org/pdf/1810.04805.pdf">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a></li></ul><p>And to understand transformer architecture better, this source by<a href="https://jalammar.github.io/illustrated-transformer/"> Jay Alammar</a> helped me a lot.</p><p>You trained your model but the client will want your model as a containerized service. The star in backend services is now FastAPI and for containerization, I learned docker. To learn docker, I think one of the best sources is <a href="https://www.youtube.com/watch?v=fqMOX6JJhGo&amp;t=1s&amp;ab_channel=freeCodeCamp.org">this tutorial from KodeKloud</a>.</p><p>To make inference faster, there are lots of ways but I started with using TF Serve. To learn more about TF Serve and see how it makes inference faster, you can refer to <a href="https://medium.com/carbon-consulting/why-you-should-serve-your-tensorflow-model-using-tf-serving-cb62c8a49370">my Medium article</a>.</p><p>After working full-time while studying for 1 year, I finally graduated and wrote my thesis about detecting toxic text detection on low-resource devices using a ~650kb model.</p><p>The rest is a little bit complicated. I became team lead and did backend engineering, worked on deployments, learned Kubernetes, designed systems, and mentored others in my team. While doing that, I realized I forget lots of things about ML and quit.</p><p>After a very short time, I relearn nearly everything I forget and more. After 2 years, I saw that training ML models are very easy, and knowing that gives you very little advantage.</p><blockquote>If you want to shine among others, you need to learn more about designing ML systems, moving that model into production, creating pipelines, learning automatization, and learning cloud techs.</blockquote><p>The theoretical knowledge is not enough, you need to become a better software engineer. The only exception is becoming a researcher and working in the R&amp;D department.</p><p>Currently, I’m working on my MLOps and ML system design skills on the Google Cloud Platform. I really don’t know if it is the best journey or the right way to improve/learn/move forward but it is my story :)</p><p>I hope it can help someone and that you enjoyed reading this story!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aefa102e2f73" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tuning and Deploying Huggingface Transformers with Vertex AI — Part 3 Start Distributed Tuning and…]]></title>
            <link>https://blog.devops.dev/tuning-and-deploying-huggingface-transformers-with-vertex-ai-part-3-start-distributed-tuning-and-e0943bfc9d4b?source=rss-af00de1b9dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0943bfc9d4b</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[distributed-training]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Sat, 15 Oct 2022 11:45:18 GMT</pubDate>
            <atom:updated>2022-10-19T18:35:49.195Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*BVkJuBpzFOXVQ88CGD0Onw.png" /></figure><h3>Tuning and Deploying Huggingface Transformers with Vertex AI — Part 3 Start Distributed Tuning and Training and Serving</h3><h3>Introduction</h3><p>I believe that Google’s documentation is very hard to understand especially when it comes to Vertex AI. It took me days to create a hyperparameter-tuning job, get the best parameters, and train with these values. So, I’ve decided to write a medium post about it. It is a long journey and because of that, this will be a series of articles. The steps will be:</p><ol><li>Setting the infrastructure</li><li>Uploading dataset</li><li>Hyperparameter-tuning</li><li>Training with the best parameters</li><li>Deploying the model to an endpoint and getting the predictions</li></ol><p>The final result will look like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aEdJOtO2wzWl1GpLMmxbYw.png" /></figure><p>This is part 3 of Tuning and Deploying HF Transformers with Vertex AI.</p><p>In <a href="https://medium.com/@m.nusret.ozates/tuning-and-deploying-hf-transformers-with-vertex-ai-part-1-preparing-prerequisites-and-dataset-9794ebe8e291">part 1</a>, we created the necessary GCP components.</p><p>In <a href="https://medium.com/@m.nusret.ozates/tuning-and-deploying-hf-transformers-with-vertex-ai-part-2-training-code-591186445a2a">part 2</a>, we created model training and tuning code and wrap it into a docker container.</p><p>For looking at the whole code we will use check this <a href="https://github.com/NusretOzates/huggingface-gcp-classification">GitHub repository</a>.</p><p>Open your Jupyter Lab environment from Vertex AI or you can use Shell Editor and create a file training.py . You can change the name of course.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38372a7985068bcc74494bf4c51c1958/href">https://medium.com/media/38372a7985068bcc74494bf4c51c1958/href</a></iframe><p>We begin with imports and some constants to initialize the AI Platform.</p><p><strong>PROJECT_NAME</strong>: Your project name<br><strong>BUCKET_NAME</strong>: Your Google Cloud Storage bucket name to save training metadata or saving models<br><strong>REPOSITORY_NAME</strong>: Your Google Cloud Artifact Registry Docker Repository name<br><strong>LOCATION</strong>: Where should training and tuning begin? If you followed the previous parts, this should be europe-west4 . As all parts of the training will be in the same region, everything will be faster with reduced latency.<br><strong>IMAGE_URI</strong>: URI of the training docker image to use<br><strong>TIMESTAMP</strong>: We will use this when giving job names to train and tune models</p><p>To create HyperparameterTuningJob we need to create a CustomJob first.</p><p>CustomJob object needs 3 parameters: display_name ,project_name and worker_pool_specs . The first two are very simple to set. The last is still simple but it is longer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc477aa5c1a3693d028a792429c7e8e6/href">https://medium.com/media/cc477aa5c1a3693d028a792429c7e8e6/href</a></iframe><p>Every worker pool needs to set three parameters. machine_spec , container_spec and replica_count .</p><h4>machine_spec</h4><p>Which machine do you want to use for the workers in your worker pool? Do you want to use GPU in your workers and which one and how many? For choosing a suitable machine type and GPU type refer <a href="https://cloud.google.com/vertex-ai/docs/predictions/configure-compute">here</a>.</p><p>I’m broke and using the free trial of GCP. So I choose a basic machine without GPU :)</p><h4>container_spec</h4><p>Which container do you want to use for your workers? And which command line arguments do you want to add to your container? We completed our dockerfile using ENTRYPOINT , remember? Now you can see that the reason is we want to “append” these arguments into the container.</p><h4>worker_pool_specs</h4><p>Pay attention that this is not a dictionary, it is a list of dictionaries! It could have at most 4 elements and every element has a different responsibility. Directly from the source:</p><blockquote>Worker pool 0 configures the Primary, chief, scheduler, or “master”. In MultiWorkerMirroredStrategy, all machines are designated as workers, which are the physical machines on which the replicated computation is executed. In addition to each machine being a worker, there needs to be one worker that takes on some extra work such as saving checkpoints and writing summary files to TensorBoard. This machine is known as the chief. There is only ever one chief worker, so your worker count for worker pool 0 will always be 1.</blockquote><blockquote>You can choose to add GPUs, but keep in mind that for MultiWorkerMirroredStrategy each machine in your cluster should have the same number of GPUs. Adding GPUs will also increase the cost of the training job.</blockquote><p>And worker pool 1 is where you configure the workers for your cluster.</p><p>Worker pool 2 is for ParameterServerStrategy and if you want to add an evaluator, you would add worker pool 3.</p><p>Now that we set worker pools 0 and 1, we have configured to have three CPU-only machines. When the training application code is run, MultiWorkerMirroredStrategy will distribute the training across both machines.</p><h4>HyperparameterTuningJob</h4><p>Now we came to the easy part!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/13a0305a419ec7f97d759dc6ea7cf470/href">https://medium.com/media/13a0305a419ec7f97d759dc6ea7cf470/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*3VuCbiOyx-JramftV8RFJA.png" /></figure><p>I think this meme is enough to tell what we did here! :) I want to add one explanation about scale parameters.</p><p>Linear scale: Typically, you choose this if the range of all values from the lowest to the highest is relatively small (within one order of magnitude), because uniformly searching values from the range will give you a reasonable exploration of the entire range.</p><p>Log scale: Logarithmic scaling works only for ranges that have only values greater than 0.</p><p>Choose logarithmic scaling when you are searching a range that spans several orders of magnitude. For example, if you are tuning a model, and you specify a range of values between .0001 and 1.0 for the learning_rate hyperparameter, searching uniformly on a logarithmic scale gives you a better sample of the entire range than searching on a linear scale would, because searching on a linear scale would, on average, devote 90 percent of your training budget to only the values between .1 and 1.0, leaving only 10 percent of your training budget for the values between .0001 and .1.</p><p>Reverse log scale: Reverse logarithmic scaling is supported only for continuous hyperparameter ranges. It is not supported for integer hyperparameter ranges.</p><p>Reverse logarithmic scaling works only for ranges that are entirely within the range 0&lt;=x&lt;1.0.</p><p>Choose reverse logarithmic scaling when you are searching a range that is highly sensitive to small changes that are very close to 1.</p><p>To learn more details about scales you can look <a href="https://towardsdatascience.com/why-is-the-log-uniform-distribution-useful-for-hyperparameter-tuning-63c8d331698">here</a>, <a href="https://stats.stackexchange.com/questions/467372/scale-selection-for-the-hyperparameters">here</a>, and lastly <a href="https://www.youtube.com/watch?v=sBhEi4L91Sg&amp;ab_channel=KhanAcademy">here</a>.</p><p><strong>max_trial_count</strong>: How many different values should I try?</p><p><strong>parallel_trial_count</strong>: How many trials should I try at once? The important point here is you need to make this number small such as 3. Because Vertex AI uses <a href="https://towardsdatascience.com/a-conceptual-explanation-of-bayesian-model-based-hyperparameter-optimization-for-machine-learning-b8172278050f">Bayesian optimization</a> which uses previous trials to choose the best hyperparameters to try next.</p><p>The run function will block the execution and wait until tuning is done and trust me it will take such a long time. Go make a coffee, maybe it is a nice day for working out?</p><p>Now that we get the best parameters from theHyperparameterTuningJob we can start training. We will use train + validation data this time.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60470b009511fb23b5e5f6ec59b4a193/href">https://medium.com/media/60470b009511fb23b5e5f6ec59b4a193/href</a></iframe><p>First, we’ve created a CustomContainerTrainingJob which allows us to create training jobs using containers. We give our training image to container_uri and set a deployment image prepared by Vertex AI itself. There are faster images to deploy your model that created by Vertex AI but it is a topic for another article.</p><p>Before running the job, we set the “hp” parameter to False and add the best hyperparameters to model arguments. We set 3 as the replica count and the job itself will automatically create 1 chef and 2 workers for us.</p><blockquote>replica_count (int):<br> The number of worker replicas. If replica count = 1 then one chief<br> replica will be provisioned. If replica_count &gt; 1 the remainder will be<br> provisioned as a worker replica pool.</blockquote><p>After the training is done, we will get an Model object that we can use to get information about the model, start batch prediction or deploy to an endpoint. Now it is time to deploy our model and get predictions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f79cd926bcf551a597100079aec4650e/href">https://medium.com/media/f79cd926bcf551a597100079aec4650e/href</a></iframe><p>It is pretty easy, right? We basically choose a machine type and call deploy() which gives us an Endpoint object. Then, we load the tokenizer from our training code, tokenize a text and get a prediction by wrapping it into a list and giving it as a parameter to predict().</p><p>And that’s it! After 3 parts we managed to tune, train and serve a Huggingface Transformers model! The next step should be separating all the components into functions and deploying them as a Kubeflow pipeline.</p><p>I hope you enjoy reading this article series!</p><h3>Resources</h3><ol><li><a href="https://cloud.google.com/vertex-ai/docs/predictions/configure-compute">https://cloud.google.com/vertex-ai/docs/predictions/configure-compute</a></li><li><a href="https://codelabs.developers.google.com/vertex_multiworker_training#5">https://codelabs.developers.google.com/vertex_multiworker_training#5</a></li><li><a href="https://towardsdatascience.com/why-is-the-log-uniform-distribution-useful-for-hyperparameter-tuning-63c8d331698">https://towardsdatascience.com/why-is-the-log-uniform-distribution-useful-for-hyperparameter-tuning-63c8d331698</a></li><li><a href="https://stats.stackexchange.com/questions/467372/scale-selection-for-the-hyperparameters">https://stats.stackexchange.com/questions/467372/scale-selection-for-the-hyperparameters</a></li><li><a href="https://www.youtube.com/watch?v=sBhEi4L91Sg&amp;ab_channel=KhanAcademy">https://www.youtube.com/watch?v=sBhEi4L91Sg&amp;ab_channel=KhanAcademy</a></li><li><a href="https://towardsdatascience.com/a-conceptual-explanation-of-bayesian-model-based-hyperparameter-optimization-for-machine-learning-b8172278050f">https://towardsdatascience.com/a-conceptual-explanation-of-bayesian-model-based-hyperparameter-optimization-for-machine-learning-b8172278050f</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0943bfc9d4b" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/tuning-and-deploying-huggingface-transformers-with-vertex-ai-part-3-start-distributed-tuning-and-e0943bfc9d4b">Tuning and Deploying Huggingface Transformers with Vertex AI — Part 3 Start Distributed Tuning and…</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>