<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Stefan Herdy on Medium]]></title>
        <description><![CDATA[Stories by Stefan Herdy on Medium]]></description>
        <link>https://medium.com/@stefan.herdy?source=rss-39c152daa6b7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NA-00ZGXAu5P9WlqbuYbUg.jpeg</url>
            <title>Stories by Stefan Herdy on Medium</title>
            <link>https://medium.com/@stefan.herdy?source=rss-39c152daa6b7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:57:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@stefan.herdy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Stylus Package Incident: A Wake-Up Call for the JavaScript Ecosystem]]></title>
            <link>https://medium.com/@stefan.herdy/the-stylus-package-incident-a-wake-up-call-for-the-javascript-ecosystem-efaf5fc694ad?source=rss-39c152daa6b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/efaf5fc694ad</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Stefan Herdy]]></dc:creator>
            <pubDate>Wed, 23 Jul 2025 21:12:34 GMT</pubDate>
            <atom:updated>2025-07-25T16:06:43.084Z</atom:updated>
            <content:encoded><![CDATA[<p>On July 23, 2025, a major disruption rippled through the global JavaScript and Node.js development ecosystem. Developers across the world began experiencing strange build failures and CI/CD pipeline crashes. The reason? The widely used stylus package had been removed from the npm registry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-ITEBzUmYhBEZzi-a9Gi7A.jpeg" /></figure><p>Stylus, a CSS preprocessor, still plays a critical role in many build tools and frameworks, including Angular CLI, Nx and many other projects. With over <a href="https://www.npmjs.com/package/stylus">3.6 million weekly downloads</a>, its sudden removal exposed a fragile underbelly of the modern software supply chain. The incident wasn’t a targeted attack against Stylus or its users, but rather a side effect of an automated security policy triggered by malicious behavior from a former co-maintainer.</p><p>With this post I want to show you what exactly happened, the global impact, the complex web of software dependencies that allowed this event to cascade so widely (and escalate so quickly), and most importantly, what steps we can take to build resilience against such software supply chain shocks in the future.</p><p><strong>What Happened?</strong> On July 23, the npm security team removed all stylus versions from the registry and replaced them with a “security holding” page. The reason? Scurity researchers found that one of its maintainers had published <strong>malicious code in unrelated packages</strong> (these were @pwa-ib/eslint-plugin-compat, @blocks-shared/desktop-title, @tui-react-internal/select-account-icon). Due to npm’s security protocols, any packages associated with the suspicious maintainer were either removed or replaced with placeholder security-holding versions.</p><p>Unfortunately, this included Stylus. Despite being clean and trusted, it was automatically flagged and deleted from the registry without immediate review. This triggered a domino effect across the ecosystem. Any project that depended on Stylus — directly or indirectly — started failing to install. Build systems that expected a certain version of Stylus suddenly received a package with no functional code. CI/CD pipelines crashed, releases were blocked, and developers around the globe were left searching for answers and quick solutions in order to continue working on their projects. This GitHub issue (<a href="https://github.com/stylus/stylus/issues/2938">https://github.com/stylus/stylus/issues/2938</a>) shows very detailed what was going on and how many developers were affected by this incident.</p><p>The Stylus maintainers, along with npm staff, confirmed that the removal was an administrative error. In the GitHub issue, workarounds were suggested, including pulling Stylus directly from GitHub or using overrides in package.json to bypass the broken registry version.</p><h4>The Global Fallout</h4><p>The consequences of the Stylus package removal were immediate. Across the software development landscape, build systems and pipelines didn’t work any more. From open-source libraries to enterprise applications, any system that depended on Stylus or indirectly relied on it experienced failures. Build tools like Angular CLI and Nx, which depend on Stylus, became dysfunctional for many users. Developers were suddenly faced with cryptic error messages during installation or deployment.</p><p>In DevOps environments, the impact was particularly severe. Automated CI/CD pipelines failed and the absence of a single expected package like Stylus caused full-blown crashes. Developers lost hours, attempting to isolate the root of the problem (or waiting for the problem to be fixed by the open source comunity). For many teams, Stylus was a package pulled in not by direct imports but through one or several layers of abstraction, often unknown by the developers themselves. This lack of visibility only increased the confusion.</p><p>The broader developer community reacted with a mix of confusion and frustration (and also many funny memes). GitHub issues were flooded with urgent questions and temporary workarounds. It became clear that one small administrative action — or in this case, a well-intentioned but overly broad security policy — could stop the workflows of countless teams across the globe.</p><p>This wasn’t just an accident. It was a direct consequence of how modern software development operates today. The core of the problem lies in the deep web of dependencies that define many software projects heavily depending on open source packages. Even a simple application can include hundreds or even thousands of packages. The larger and more feature-rich the project, the deeper and more complex these dependencies become, turning every single package into a potential point of failure.</p><p>Many developers were surprised simply because Stylus wasn’t something they had explicitly installed. It was buried deep in their toolchains, pulled in through widely-used open source frameworks and libraries. This illustrates a critical risk: the power of (hidden) dependencies. Developers may carefully choose their top-level packages, but those packages mostly rely on other packages and it is very hard (if not impossible) to keep track of all of them.</p><p>One part of the problem is also the reliance on centralized registries. npm, for example, serves as the single source of truth for the vast majority of JavaScript packages. When a package is removed — intentionally or otherwise — any project that depends on it can instantly break. This creates a single point of failure for the entire ecosystem.</p><p>Furthermore, this incident underscores the ongoing issue of maintainer risk. The open-source model relies heavily on the integrity and diligence of a small number of individuals. A single maintainer can have a huge impact (in a good or bad way).</p><p>If we want to prevent a recurrence of the Stylus issue — or at least minimize the risk — we need to rethink how we manage our dependencies and architect our infrastructure.</p><p>Implementing local caching and mirroring solutions can be one such solution. Tools such as Nexus Repository Manager (and many others) can proxy npm and cache packages internally. This not only speeds up builds but also protects projects against upstream removals or outages. When a package like Stylus disappears from the public registry, a local mirror can continue serving it as though nothing happened.</p><p>Regular audits of dependency trees can help uncover hidden risks. Tools like npm’s built-in audit feature can flag vulnerable or unmaintained packages. These audits should be integrated into CI pipelines and treated as seriously as unit tests or security scans.</p><p>Another key strategy is to minimize dependency bloat. Not every package is necessary. The more packages a project uses, the greater the surface area for vulnerabilities and outages. Developers should strive for lightweight solutions and well-maintained libraries, and question whether each new package — with all its dependencies — is truly necessary.</p><p>Finally, CI/CD pipelines should be designed with resilience in mind. This includes having fallback mechanisms for package resolution, robust logging to diagnose failures, and early-warning systems to flag missing or broken dependencies before they affect production environments.</p><p>The Stylus incident serves as a reminder of how much is built on top of largely invisible open-source projects. It’s a call to action for every developer, team, and organization to build software that is not only functional — but resilient.</p><p>The Stylus package removal was not malicious, but it was profoundly disruptive. It exposed just how interconnected, automated, and fragile our software ecosystem has become. We can’t eliminate risk entirely, but we can design systems that are more robust against unexpected shocks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efaf5fc694ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Joint Energy-Based Semantic Segmentation]]></title>
            <link>https://medium.com/@stefan.herdy/joint-energy-based-semantic-segmentation-63e1f5fcf141?source=rss-39c152daa6b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/63e1f5fcf141</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[joint-energy-based-models]]></category>
            <category><![CDATA[domain-adaptation]]></category>
            <category><![CDATA[semantic-segmentation]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Stefan Herdy]]></dc:creator>
            <pubDate>Thu, 21 Dec 2023 14:12:42 GMT</pubDate>
            <atom:updated>2023-12-21T15:45:00.764Z</atom:updated>
            <content:encoded><![CDATA[<p>Joint Energy-Based Modeling (JEM) is a very powerful method to improve deep learning models in a simple way. The models trained with the JEM approach have better properties in terms of better model calibration, more robust classification, possible generative modeling, out-of-distribution detection. For image classification, this can be implemented very easily with a few additional lines of code. I have described how to do this in an older article. You can read it <a href="https://medium.com/@stefan.herdy/joint-energy-based-modelling-how-to-improve-your-deep-learning-model-with-just-a-few-lines-of-4bf7e403b81b">here</a>. Make sure that you have read this article before you deal with this derivation. A code implementation of the previous article is available <a href="https://github.com/stefanherdy/JESS/tree/main">here</a>.</p><p>In this article we focus on how it is possible to also benefit from Joint Energy-Based Modeling for Semantic image Segmentation. Similar to image classification (as proposed <a href="https://arxiv.org/abs/1912.03263">here</a>), the softmax function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/1*f09RukTVisQOq_3OldVs4g.png" /></figure><p>also depends on the pixel index q. The joint, the prior and the posterior distributions depend on the pixel index q as well, with</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*81MsF7j48oGvLd3pnLYKfA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*GZyiRzCPeU-r7GpJrJDtrg.png" /></figure><p>and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*TSkmohTO5h6rAMOL6G7hJQ.png" /></figure><p>By solving the normalization term cancels out and we obtain the softmax function again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*Do30YvvxIcCgdMpXzDx-4g.png" /></figure><p>This means, that also for Semantic Segmentation there is a generative model hidden in every classifier. Consequently, we are allowed to combine discriminative and generative learning by optimizing pθ(y|x) and optimizing the log-likelihood of pθ(x) so that our model fits better to our data. The joint probability of pθ(x,q) is given as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*p9ifKwXDxJNRllYwTly_cw.png" /></figure><p>The pixel index q is distributed uniformly. So, p(q) is a uniform distribution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/1*vMONAGaa-h1HNba525Lwtw.png" /></figure><p>with W and H as the image width and height. The joint distribution of pθ(x,q) is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*aPzNCE12gzhc9ZmC8zs3mg.png" /></figure><p>As a simplification assumption, we suppose that the prior factorizes over the pixel index q, i.e.,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*f5YLFPWynOimzgVIyyIurA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*zRhfDhKXu9gJaB6L72ltxw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*GsDj9zAtI-3cfDoxcOEu9A.png" /></figure><p>The energy function finally results in:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hmPqriXuwKTEz6Z3w-_teA.png" /></figure><p>That means that we can perform Joint Energy-Based Semantic Segmentation (JESS) by simply adding a sum over q to the energy function that is used for image classification.</p><p>Please have a look at this <a href="https://github.com/stefanherdy/JESS">GitHub repository</a>, where I use this approach for domain adaptation.</p><p>I hope that you liked this article and I would be happy if this approach would be used by other interested people. Let me know what your experiences are!</p><p>All the best,<br>Stefan</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63e1f5fcf141" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Active Learning with Labelbox — How to Upload Your Label Masks]]></title>
            <link>https://medium.com/@stefan.herdy/active-learning-with-labelbox-how-to-upload-your-label-masks-99feef86017b?source=rss-39c152daa6b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/99feef86017b</guid>
            <category><![CDATA[semantic-segmentation]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[active-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[labelbox]]></category>
            <dc:creator><![CDATA[Stefan Herdy]]></dc:creator>
            <pubDate>Mon, 04 Dec 2023 16:10:28 GMT</pubDate>
            <atom:updated>2023-12-11T10:11:04.609Z</atom:updated>
            <content:encoded><![CDATA[<h3>Active Learning with Labelbox — How to Upload Your Label Masks</h3><p>Labelbox is a very powerful platform for image annotation. It allows a simple labeling of custom image data sets. These can then be exported and used for the respective deep learning applications. I have shown how to download the Labelbox annotations in <a href="https://medium.com/@stefan.herdy/how-to-export-labelbox-annotations-eedb8cb4f217">this article</a>. Once the masks have been downloaded, you can train your deep learning models on them. These trained models can be used to label new data to extend your training dataset. However, this newly labeled data is usually not perfect and the labels need to be improved. This approach is called active learning. We can upload the preliminary labels to Labelbox and touch them up instead of labeling new data from scratch, which can save us some time. I will show you how to upload the data in this article.</p><figure><img alt="Active Learning with PyTorch and Labelbox" src="https://cdn-images-1.medium.com/max/1024/1*F-jP2nzUGzUgsOWiT5xCCA.png" /><figcaption>Our Active Learning Workflow</figcaption></figure><p>As a first step we have to create our own API-key in Labelbox to be able to download the required data.</p><p>Click on your profile button in the lower right corner, go to “Worspace settings” and click on “New API key”.</p><figure><img alt="Labelbox API" src="https://cdn-images-1.medium.com/max/792/0*PbaoJRVG4HL5CJ4B.png" /></figure><p>This key will be used later, so we have to copy and save it.</p><p>We also need the ID of the project that we want to export. Select your project and go to “Export”. Here you will find your project ID.</p><figure><img alt="Labelbox API" src="https://cdn-images-1.medium.com/max/792/0*VuJmUM_CJW6eqI6K.png" /></figure><p>Now we create a YAML-file that contains our private API-key and our project ID and we save it as config.yaml. We also have to specify, where our mask files are stored. Our masks are stored in the folder “labels_upload”. Change this folder path according to your specific project.</p><pre>api_key: &#39;INSERT-YOUR-API-KEY-HERE&#39;<br>project_id: &#39;INSERT-YOUR-PROJECT-ID-HERE&#39;<br>mask_folder: &#39;./labels_upload&#39;</pre><p>As a first step, we import all necessary packages, set up our labelbox client and we decide which pictures we want to upload. We also have to specify your project ontology in the class_dict dictionary. Labelbox has four workflow statuses: “<em>TO_LABEL</em>”, “<em>TO_REVIEW</em>”, “<em>IN_REWORK</em>” and “<em>DONE</em>”. We just want to upload masks for files with the status “<em>TO_LABEL</em>” and pass the list of the desired files to a function called “upload_masks()”.</p><pre>import labelbox<br>import labelbox.data.annotation_types as lb_types<br>from PIL import Image<br>import os<br>import numpy as np<br>import yaml<br>import uuid<br>from pathlib import Path<br><br>def export_project_data(project):<br>      labels = project.export_v2(params={<br>          &quot;data_row_details&quot;: True,<br>          &quot;metadata_fields&quot;: True,<br>          &quot;attachments&quot;: True,<br>          &quot;project_details&quot;: True,<br>          &quot;performance_details&quot;: True,<br>          &quot;label_details&quot;: True,<br>          &quot;interpolated_frames&quot;: True<br>      })<br>      return labels<br><br>if __name__ == &quot;__main__&quot;:<br><br>  # Specify your project ontology here<br>  class_dict = {  &quot;class_1&quot; : 1,<br>                  &quot;class_2&quot; : 2,<br>                  &quot;class_3&quot; : 3,<br>                  &quot;class_4&quot; : 4,<br>                  &quot;class_5&quot; : 5,<br>                  &quot;class_6&quot; : 6,<br>                  }<br><br>    with open(&#39;./config.yaml&#39;, &#39;r&#39;) as file:<br>        config = yaml.safe_load(file)<br>    project_id = config[&#39;project_id&#39;]<br>    api_key = config[&#39;api_key&#39;]<br>    mask_folder = config[&#39;mask_folder&#39;]<br><br>    client = labelbox.Client(api_key)<br>    project = client.get_project(project_id)<br>    labels = export_project_data(project)<br><br>    export_json = labels.result<br>    global_key_data_row_inputs = []<br>    to_label_list = []<br><br>    for img in export_json:<br>        # We assign our externel key (image name) as our global key too. You can also assign a custom global key.<br>        global_key_data_row_inputs.append({&quot;data_row_id&quot;: img[&#39;data_row&#39;][&#39;id&#39;], &quot;global_key&quot;: img[&#39;data_row&#39;][&#39;external_id&#39;]})<br>    client.assign_global_keys_to_data_rows(global_key_data_row_inputs)<br>    # Load project data again with global keys<br>    labels = export_project_data(project)<br><br>    export_json = labels.result<br>    for img in export_json:<br>        workflow_status = img[&#39;projects&#39;][project_id][&#39;project_details&#39;][&#39;workflow_status&#39;]<br>        # Specify which images you want to upload. You can delete this if you want to upload all.<br>        status = &quot;TO_LABEL&quot;<br>        if workflow_status == status: # Only upload images with status &quot;TO_LABEL&quot;. Possible workflow statuses are: &quot;TO_LABEL&quot;, &quot;TO_REVIEW&quot;, &quot;IN_REWORK&quot;, &quot;DONE&quot;<br>            to_label_list.append(img[&#39;data_row&#39;][&#39;global_key&#39;])<br>            to_label_list_set = set(to_label_list)<br>    print(f&quot;{len(to_label_list)} images with status {status}&quot;)<br><br>    # Check for duplicates<br>    if len(to_label_list) != len(to_label_list_set):<br>        print(&quot;Duplicates found!&quot;)<br>    else:<br>        print(&quot;No duplicates found!&quot;)<br><br>    upload_masks(client, project, to_label_list, mask_folder, export_json, class_dict) # Upload masks to data row with status &quot;TO_LABEL&quot;</pre><p>To be able to upload our images we have to asssign global keys to our images. We just pass our external key (the image name) as global key. You can also assign custom global keys if you want to.</p><p>The function “upload_masks” iterates over all our masks to upload them. We have a naming convention that our masks are stored as PNG-files and our images are stored as JPG-files. The mask files and the image files have the same name. You can also choose other formats, <strong>but remember not to save the masks as JPG-files, because the image compression will add additional class values to your categorical image values</strong>. We define our function as follows:</p><pre>def upload_masks(client, project, to_label_list, mask_folder, export_json):<br>    mask_files = [str(file) for file in Path(mask_folder).glob(&quot;*.png&quot;)]<br>    upload_count = 1<br>    for mask_file in mask_files:<br>        # Raw images saved as .JPG so we have to change our .png mask extension to extension of raw images.<br>        img_name = os.path.basename(mask_file).split(&quot;.png&quot;)[0] + &quot;.JPG&quot;<br>        if img_name in to_label_list:<br>            print(f&quot;Uploading mask {upload_count} of {len(mask_files)}&quot;)<br>            mask = np.array(Image.open(mask_file))<br>            print(f&quot;Performing upload of image: {img_name}&quot;)<br>            print(f&quot;Image masks: {np.unique(mask)}&quot;)<br>            perform_upload(client, project, img_name, mask)<br>            upload_count += 1</pre><p>The actual image upload is performed with “perform_upload”.</p><pre>def perform_upload(client, project, img_name, mask):<br>    labels = []<br>    annotations = []<br>    class_names = list(class_dict.keys())<br>    class_values = np.unique(mask)<br>    # Iterate over all classes to merge annotations<br>    for class_value in class_values:<br>        color = (class_value, class_value, class_value)<br><br>        mask_data = lb_types.MaskData.from_2D_arr(arr=mask)<br>        mask_annotation = lb_types.ObjectAnnotation(<br>            name = class_names[class_value-1], # must match your ontology feature&quot;s name<br>            value=lb_types.Mask(mask=mask_data, color=color),<br>            )<br>        annotations.append(mask_annotation)<br><br>    labels.append(<br>        lb_types.Label(data=lb_types.ImageData(global_key=img_name),<br>                    annotations=annotations))<br>    <br>    # Upload<br>    upload_job = labelbox.MALPredictionImport.create_from_objects(<br>        client = client, <br>        project_id = project.uid, <br>        name=f&quot;mal_job{img_name}{uuid.uuid4()}&quot;, <br>        predictions=labels<br>    )<br><br>    # Check if upload was successful<br>    if len(upload_job.errors) &gt; 0:<br>        print(upload_job.errors)<br>    else:<br>        print(&quot;Upload successful&quot;)</pre><p>If the upload was successful, we can see our uploaded images in the data row editor of the corresponding project. The labels are shown as “pre-labels”. That means that we have to submit them to become actual labels. To achieve this just open the images in the label editor, make your changes to the labels and submit them.</p><p>This is an easy way to use an active learning approach to increase your training dataset for semantic segmentation.</p><p>I hope that you liked this article!</p><p>The full code of this tutorial can be found <a href="https://github.com/stefanherdy/pytorch-semantic-segmentation">here</a> in the file “upload_masks.py”.</p><p>Also check out my <a href="https://github.com/stefanherdy">GitHub </a>for other exciting projects and let me know about your projects in the comments!</p><p>All the best,<br>Stefan</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99feef86017b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Upgrade Your Python Exception Handling With Decorators]]></title>
            <link>https://medium.com/@stefan.herdy/upgrade-your-python-exception-handling-with-decorators-e30cc6a4eefa?source=rss-39c152daa6b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e30cc6a4eefa</guid>
            <category><![CDATA[exception-handling]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[decorators]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Stefan Herdy]]></dc:creator>
            <pubDate>Mon, 20 Nov 2023 21:22:35 GMT</pubDate>
            <atom:updated>2023-12-11T10:16:10.545Z</atom:updated>
            <content:encoded><![CDATA[<p>Python decorators are a very powerful tool to modify Python functions according to custom wishes. A decorator</p><pre>@decorator<br>def greet_readers():<br>    print(&quot;Hello Readers!&quot;)</pre><p>is equivalent to</p><pre>def greet_readers():<br>    print(&quot;Hello Readers!&quot;)<br>    <br>decorator_greet = decorator(greet_readers)</pre><p>,where “decorator” is a wrapper function that performs the required modification.</p><p>We can use this to handle exceptions in a custom way. To do this we can define a custom Exception handler function.</p><pre><br>def custom_exception_handler(function):<br>    error_message = &#39;&#39;<br>    try:<br>        result = function()<br>    except Exception as e:<br>        # Access more information about the exception<br>        # You can handle the Exception directly here according to your requirements<br>        print(&quot;Exception Type:&quot;, type(e).__name__)<br>        print(&quot;Exception Args:&quot;, e.args)<br>        print(&quot;Traceback:&quot;)<br>        traceback.print_tb(e.__traceback__)<br>        tracebackmessage = e.__traceback__<br>        # Retrieve the last n lines of code from the traceback<br>        stack = traceback.extract_tb(sys.exc_info()[2])<br>        #for filename, line_number, function, line in stack[-1]:<br>        line_number = stack[-1].lineno<br>        function = stack[-1].name<br>        source_code = inspect.getsourcelines(inspect.getmodule(inspect.currentframe()))[0]<br>        relevant_code = source_code[line_number-10: line_number]<br>        print(description)<br>        print(f&quot;Line {line_number}, in {function}: {relevant_code}&quot;)<br></pre><p>Here we now have caught the error message and just printed some information about the exception. Just printing the exception in a different way is not a big improvement so far. But we are now able to do a more advanced Exception handling. Have you ever thought about getting automated suggestions about what could be the cause for the error and how to fix it?</p><p>Due to the rapid innovations in the field of large language models, we can now automatically send queries to such language models and get suggestions for improvement.</p><p>For example, we can use ChatGPT to make use of automatic error handling.<br>The ChatGPT API is easily accessible. First, we have to make a new API key at the OpenAI webpage. Under <a href="https://platform.openai.com/api-keys">API keys — OpenAI API</a> we can create our own API key. Save the key in a config.yaml file:</p><pre>api_key: &#39;your-api-key&#39;</pre><p>We can now load the config file to access ChatGPT with our Python script.</p><pre>import openai<br>import yaml<br><br>def chat_with_chatgpt(api_path, message):<br>    with open(api_path, &#39;r&#39;) as file:<br>        config = yaml.safe_load(file)<br>        openai.api_key = config[&#39;api_key&#39;]<br>      <br>    messages = [ {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;:  <br>                    &quot;Hey ChatGPT!&quot;} ] <br>      <br>    if message: <br>        messages.append( <br>          {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: message}, <br>          ) <br>        chat = openai.ChatCompletion.create( <br>          model=&quot;gpt-3.5-turbo&quot;, messages=messages <br>          ) <br>        reply = chat.choices[0].message.content <br>        print(f&quot;ChatGPT: {reply}&quot;) <br>        messages.append({&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: reply}) <br><br>        return reply</pre><p>We can now combine both, the error handling and the ChatGPT API to catch our errors and pass them directly to ChatGPT. To improve our results, we can add a description to our functions and add it to our error message to get more accurate suggestions. To do this, we create a function called message_compose, which composes the message to ChatGPT.</p><pre>def message_compose(err_type, err_message, err_code, err_description):<br>  message = f&quot;&quot;&quot;Hey ChatGPT! I have the following {err_type}: {err_message}. <br>  The last lines of my code are: {err_code}. <br>  The description of the failing function is: {err_description} <br>  How can I improve my code to fix this error? Be as precise as possible!&quot;&quot;&quot;<br>  <br>  return message<br><br></pre><p>To pass arguments to Python decorators, we need to add additional wrappers to handle the function and the arguments. So we can edit our custom_exception_handler.</p><pre>import traceback<br>import inspect<br>import sys<br><br>def capture_stderr(err_description = &#39;&#39;, api_path = &#39;./config.yaml&#39;):<br>    def wrapper(func):<br>        def wrapped_args(*args):<br>            # Create a StringIO object to capture stderr<br>            error_message = &#39;&#39;<br>            try:<br>                result = func()<br>                return result<br>                # If there is an error message, you can handle it here<br>            except Exception as e:<br>                # Access more detailed information about the exception<br>                err_type = type(e).__name__<br>                err_message = e.args<br>                print(f&#39;Error Type: {err_type}&#39;)<br>                print(f&#39;Error Message: {e}&#39;)<br>                print(&quot;Traceback:&quot;)<br>                traceback.print_tb(e.__traceback__)<br>                # Retrieve the last n lines of code from the traceback<br>                stack = traceback.extract_tb(sys.exc_info()[2])<br>                #for filename, line_number, function, line in stack[-1]:<br>                line_number = stack[-1].lineno<br>                source_code = inspect.getsourcelines(inspect.getmodule(inspect.currentframe()))[0]<br>                err_code = source_code[line_number-5: line_number]<br>                message = message_compose(err_type, err_message, err_code, err_description)<br>                reply = chat_with_chatgpt(api_path, message)<br>                print(reply)<br>        return wrapped_args<br>    return wrapper</pre><p>Let’s test our new error handler. We now produce a very simple error:</p><pre>import numpy as np<br><br>description = &#39;This function initializes an array and returns the value at desired index.&#39;<br>index = 7<br>@capture_stderr(description)<br>def get_value(index):<br>    simplearray = np.zeros_like(7)<br>    value = simplearray[index]<br>    return value<br><br>result = get_value(index)</pre><p>This simple code produces an index error, since the array with np.zeros_like(7) is indexed from 0 to 6. Normaly the error would be shown like this:</p><figure><img alt="Python Error Handling ChatGPT Debugging" src="https://cdn-images-1.medium.com/max/820/1*m9YrOpKZrXjFPOTmoJDHlw.png" /></figure><p>Our capture_stderr() decorator catches the error message and returns the error plus suggestions how to fix the error. This can be very helpful and can save much time for debugging.</p><p>In this case ChatGPT suggested the following:</p><p><em>“The error you’re encountering is due to the fact that you’re trying to access an element at index 7 in an array that has a size of 7. In Python, indexing is zero-based, so the valid indices for an array of size 7 are 0 through 6.</em></p><p><em>To fix this issue, you need to ensure that the index you pass to the </em><em>get_value function is within the valid range. Here&#39;s an updated version of your code with a check to ensure the index is within bounds:</em></p><pre>import numpy as np<br><br>@capture_stderr(description)<br>def get_value(index):<br>    array_size = 7<br>    if 0 &lt;= index &lt; array_size:<br>        simplearray = np.zeros(array_size)<br>        value = simplearray[index]<br>        return value<br>    else:<br>        # Handle the case where the index is out of bounds<br>        raise IndexError(f&quot;Index {index} is out of bounds for an array of size {array_size}&quot;)</pre><p><em>In this modified code:</em></p><ol><li><em>I added a check to ensure that the index is greater than or equal to 0 and less than the array size.</em></li><li><em>If the index is within bounds, the function proceeds to access the value in the array.</em></li><li><em>If the index is out of bounds, it raises an </em><em>IndexError with a descriptive message.</em></li></ol><p><em>This way, you can prevent accessing elements outside the valid range and handle such cases gracefully.”</em></p><p>Of course using deep learning tools such as ChatGPT for debugging without thinking about syntax, logic, functionality etc. is not recommended.</p><p>Always know your code and understand what it does and avoid using code snippets that work without knowing how they work. This applies not only to ChatGPT but to all kinds of research (in forums etc.).</p><p>The full code to this tutorial is accessible <a href="https://gist.github.com/stefanherdy/65315e544ccfb7114c97da83edae7018">here</a>.</p><p>I hope that you liked this article and I wish you happy programming!</p><p>All the best,<br>Stefan</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e30cc6a4eefa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Compute the Curvature of a Binary Mask in Python]]></title>
            <link>https://medium.com/@stefan.herdy/compute-the-curvature-of-a-binary-mask-in-python-5087a88c6288?source=rss-39c152daa6b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/5087a88c6288</guid>
            <category><![CDATA[binary-mask]]></category>
            <category><![CDATA[curvature]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[masks]]></category>
            <category><![CDATA[semantic-segmentation]]></category>
            <dc:creator><![CDATA[Stefan Herdy]]></dc:creator>
            <pubDate>Tue, 14 Nov 2023 21:03:11 GMT</pubDate>
            <atom:updated>2023-12-11T10:19:59.698Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Curvature Python" src="https://cdn-images-1.medium.com/max/640/1*e6eVPo6Z9oDA8izDktR9Rg.png" /></figure><p>The segmentation of images is widely used, for example, in the fields of medicine, autonomous driving, biomonitoring, etc. It allows a deeper insight into the processed image data by trying to assign a class to each pixel of an image. This allows quantitative analyses such as how much of the image is covered by what. However, it would be a shame to leave it at that, as a deeper analysis of the images can provide further insights into the underlying data and further potential of semantic segmentation can be exploited here. One of the analyses that can be done with the segmentation results is to calculate the curvature of the segmented masks. In this article, we will take a closer look at this and describe how we can perform this calculation in Python.</p><p>The importance of having information about the curvature of segmented masks lies in the need for more precise and detailed understanding of the objects or regions within an image. While traditional semantic segmentation produces binary masks outlining object boundaries, incorporating information about curvature enhances the significance of the segmentation. This additional curvature information helps to capture the shape and contours of objects in more detail.</p><p>Understanding curvature is crucial in scenarios where object boundaries are not strictly linear, and capturing the nuanced details of contours becomes essential. For example, in medical imaging, accurately delineating organ boundaries with curvature information can aid in more precise diagnosis and treatment planning. In autonomous vehicles, identifying the curvature of road boundaries and obstacles allows for better navigation and decision-making. In biomonitoring, the analysis of the curvature allows a deeper insight into the dispersal patterns of the species under investigation.</p><p>So how can we now compute the curvature?</p><p>At first we need to load our mask image. We want to plot the mask and highlight the curvature of the mask. Therfore we define a function called plot_edges_with_curvature(mask).</p><pre>#!/usr/bin/env python<br><br>&quot;&quot;&quot;<br>Script Name: curvature.py<br>Description: <br>This script performs a curvature analysis on binary masks<br>Usage: <br>- Change the the path to the path to your binary mask and execute the script.<br>- Additionally you can set the minimum contour length, the window size ratio and the min/max values of the colormap according to your specific requirements<br>&quot;&quot;&quot;<br>import os<br>import numpy as np<br>import cv2<br>import matplotlib.pyplot as plt<br>from skimage import measure<br><br># Set minimum length of the contours that should be analyzed<br>min_contour_length = 20<br># Set the ratio of the window size (contour length / window_size_ratio) for local polynomial approximation<br>window_size_ratio = 5<br>mask = cv2.imread(&#39;path_to_your_mask&#39;, cv2.IMREAD_GRAYSCALE)<br>plot_edges_with_curvature(mask, min_contour_length, window_size_ratio)</pre><p>The function computes the curvature and plots everything.</p><pre>def plot_edges_with_curvature(mask, min_contour_length, window_size_ratio):<br>    # Compute edge properties<br>    edge_pixels, curvature_values = compute_curvature_profile(mask, min_contour_length, window_size_ratio)<br><br>    # Plot the mask<br>    plt.imshow(mask, cmap=&#39;gray&#39;)<br>    # We set the min and max of the colorbar, so that 90% of the curvature values are shown.<br>    # This is to have a nice visualization. You can change this threshold according to your specific task.<br>    threshold = np.percentile(np.abs(curvature_values), 90)<br>    plt.scatter(edge_pixels[:, 1], edge_pixels[:, 0], c=curvature_values, cmap=&#39;jet&#39;, s=5, vmin=-threshold, vmax=threshold)<br><br>    plt.colorbar(label=&#39;Curvature&#39;)<br>    plt.title(&quot;Curvature of Edge Pixels&quot;)<br>    plt.show()</pre><p>To compute the curvature of our profile we iterate over all the contours in our mask and over all points in our contour.</p><pre>def compute_curvature_profile(mask, min_contour_length, window_size_ratio):<br>    # Compute the contours of the mask to be able to analyze each part individually<br>    contours = measure.find_contours(mask, 0.5)<br><br>    # Initialize arrays to store the curvature information for each edge pixel<br>    curvature_values = []<br>    edge_pixels = []<br><br>    # Iterate over each contour<br>    for contour in contours:<br>        # Iterate over each point in the contour<br>        for i, point in enumerate(contour):<br>            # We set the minimum contour length to 20<br>            # You can change this minimum-value according to your specific requirements<br>            if contour.shape[0] &gt; min_contour_length:<br>                # Compute the curvature for the point<br>                # We set the window size to 1/5 of the whole contour edge. Adjust this value according to your specific task<br>                window_size = int(contour.shape[0]/window_size_ratio)<br>                curvature = compute_curvature(point, i, contour, window_size)<br>                # We compute, whether a point is convex or concave.<br>                # If you want to have the 2nd derivative shown you can comment this part<br>                if curvature &gt; 0:<br>                    curvature = 1<br>                if curvature &lt;= 0:<br>                    curvature = -1<br>                # Store curvature information and corresponding edge pixel<br>                curvature_values.append(curvature)<br>                edge_pixels.append(point)<br><br>    # Convert lists to numpy arrays for further processing<br>    curvature_values = np.array(curvature_values)<br>    edge_pixels = np.array(edge_pixels)<br><br>    return edge_pixels, curvature_values</pre><p>Now we have to compute the curvature of the certain points. We select a certain window size, extract the edge values of that window, translate and rotate the values, perform a polynomial approximation and compute the 2nd derivative of the point.</p><pre>def compute_curvature(point, i, contour, window_size):<br>    # Compute the curvature using polynomial fitting in a local coordinate system<br><br>    # Extract neighboring edge points<br>    start = max(0, i - window_size // 2)<br>    end = min(len(contour), i + window_size // 2 + 1)<br>    neighborhood = contour[start:end]<br><br>    # Extract x and y coordinates from the neighborhood<br>    x_neighborhood = neighborhood[:, 1]<br>    y_neighborhood = neighborhood[:, 0]<br><br>    # Compute the tangent direction over the entire neighborhood and rotate the points<br>    tangent_direction_original = np.arctan2(np.gradient(y_neighborhood), np.gradient(x_neighborhood))<br>    tangent_direction_original.fill(tangent_direction_original[len(tangent_direction_original)//2])<br><br>    # Translate the neighborhood points to the central point<br>    translated_x = x_neighborhood - point[1]<br>    translated_y = y_neighborhood - point[0]<br><br><br>    # Apply rotation to the translated neighborhood points<br>    # We have to rotate the points to be able to compute the curvature independent of the local orientation of the curve<br>    rotated_x = translated_x * np.cos(-tangent_direction_original) - translated_y * np.sin(-tangent_direction_original)<br>    rotated_y = translated_x * np.sin(-tangent_direction_original) + translated_y * np.cos(-tangent_direction_original)<br><br>    # Fit a polynomial of degree 2 to the rotated coordinates<br>    coeffs = np.polyfit(rotated_x, rotated_y, 2)<br><br><br>    # You can compute the curvature using the formula: curvature = |d2y/dx2| / (1 + (dy/dx)^2)^(3/2)<br>    # dy_dx = np.polyval(np.polyder(coeffs), rotated_x)<br>    # d2y_dx2 = np.polyval(np.polyder(coeffs, 2), rotated_x)<br>    # curvature = np.abs(d2y_dx2) / np.power(1 + np.power(dy_dx, 2), 1.5)<br><br>    # We compute the 2nd derivative in order to determine whether the curve at the certain point is convex or concave<br>    curvature = np.polyval(np.polyder(coeffs, 2), rotated_x)<br><br>    # Return the mean curvature for the central point<br>    return np.mean(curvature)</pre><p>It is important to rotate the selected edge values in order to be independent of the orientation of the selected part of the edge.</p><p>On possible result could look like the following:</p><figure><img alt="Python Curvature" src="https://cdn-images-1.medium.com/max/640/1*ox4hpkgVYpkJAXvR7oID7A.png" /></figure><p>You can also change from continous values to -1 or 1 for concave or convex edge shapes. The result would look like this:</p><figure><img alt="Python Curvature" src="https://cdn-images-1.medium.com/max/640/1*gjQ-l0tIy32f8Oo48yAn3g.png" /></figure><p>Or like this:</p><figure><img alt="Python Curvature" src="https://cdn-images-1.medium.com/max/640/1*_sGOawrbselkb0X9tsj_kA.png" /></figure><p>In case you have an image consisting of categorical values, you can change the code at the beginning to the following:</p><pre># Iterate over all classes<br>for class_label in range(1, num_classes):  # We assume that the classes start from 1<br>    # Create a binary mask for the current class<br>    binary_mask = np.where(categorical_image == class_label, 255, 0).astype(np.uint8)<br>    binary_mask = binary_mask[:,:,0]<br>    # Analyze convex and concave edges for the binary mask<br>    plot_edges_with_curvature(binary_mask)</pre><p>In summary, an in-depth analysis of the curvature of segmented masks in semantic segmentation is important in applications, where a deeper understanding of object shapes and contours is critical for decision-making or analysis.</p><p>I hope that you liked this article and that you can make use of the provided algorithm.</p><p>The full code can be found as GitHub Gist <a href="https://gist.github.com/stefanherdy/e84c797b69463d6f561e7846910ebbfd">here</a>.</p><p>For more information about computer vision, deep learning and other cool projects please visit my <a href="https://github.com/stefanherdy">GitHub</a> page.</p><p>All the best,<br>Stefan</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5087a88c6288" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Joint Energy-Based Modelling — How to improve your Deep Learning Model with just a few lines of…]]></title>
            <link>https://medium.com/@stefan.herdy/joint-energy-based-modelling-how-to-improve-your-deep-learning-model-with-just-a-few-lines-of-4bf7e403b81b?source=rss-39c152daa6b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/4bf7e403b81b</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[joint-energy-based-models]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[model-optimization]]></category>
            <dc:creator><![CDATA[Stefan Herdy]]></dc:creator>
            <pubDate>Mon, 06 Nov 2023 10:17:03 GMT</pubDate>
            <atom:updated>2023-12-11T10:23:14.918Z</atom:updated>
            <content:encoded><![CDATA[<h3>Joint Energy-Based Modeling — How to Improve Your Image Classification With Just a Few Lines of Code</h3><p>The field of machine learning is a very rapidly evolving field and new models or improvements to existing models are constantly being introduced. Some of these innovations are more difficult to implement than others. In this story I will introduce you to a method that is very easy to implement and can improve your Deep Learning model in various ways.</p><p>In recent years, it has been shown that generative and discriminative modeling in deep learning are not completely different, but that there are overlaps. It has been shown that a generative model is hidden in every discriminative model. This finding has made it possible to improve deep learning models with little effort. In this story, I will show you exactly how this works and what can be improved with it. But first, let’s start with the basics.</p><p>Discriminative modeling and generative modeling are two fundamental approaches in machine learning and artificial intelligence that serve different purposes:</p><p>Discriminative models focus on modeling the conditional probability of an output variable (typically a label or category) given input data. These models aim to learn the decision boundary that separates different classes or categories in the data. Examples of discriminative models include logistic regression, support vector machines, and deep neural networks for tasks like classification and regression. Discriminative models are well-suited for tasks where the emphasis is on prediction or classification. For image classification, these models learn data features or representations to classify data by using a function fθ(x) to map an input to an output of logits , where K is the number of classes and D = H x W x 3 for RGB image data with H and W for height and width of the image, respectively. A so-called softmax function</p><figure><img alt="Softmax function" src="https://cdn-images-1.medium.com/max/265/1*m2zby9-jQow0k-wkS-etSg.png" /></figure><p>produces a categorical distribution over K different possible classes from the model output.</p><p>Generative models are designed to capture the underlying data distribution and are used to generate new data points that resemble the training data. These models learn the joint probability distribution of both input and output data. Examples of generative models include Gaussian Mixture Models (GMM), Variational Autoencoders (VAEs), and Generative Adversarial Networks (GANs). Generative models find applications in tasks like data synthesis, image generation, and anomaly detection. The Boltzmann distribution,</p><figure><img alt="Boltzmann distribution" src="https://cdn-images-1.medium.com/max/303/1*3p5h4loXDu5hOC9zVRMXHQ.png" /></figure><p>originally used in statistical mechanics, is the basis for energy based models and generative modeling (LeCun et al., 2006).</p><p>It was shown, that there is a generative model “hidden” in every discriminative model. This finding forms the basis of this story and the possible model improvements and is described in detail <a href="https://openreview.net/forum?id=Hkxzx0NtDB">in this publication</a>.</p><p>The Boltzmann distribution can be used to define an energy-based model of the joint probability of</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/1*ln9TCBRdvPwuwTT4NcyShw.png" /></figure><p>with the model output as energy function .</p><p>We marginalize out y by computing the sum over y and obtain the prior distribution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*LN6UaTW7C61gRssKUTV7jw.png" /></figure><p>The posterior distribution is computed as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*Rr1-sFhBprRXMCYhh6zztA.png" /></figure><p>The unknown normalization constant Z(θ) cancels out and we obtain the softmax function again. This means that there is a generative model hidden in every standard discriminative model as proposed in (<a href="https://openreview.net/forum?id=Hkxzx0NtDB">Grathwohl et al., 2020</a>).</p><p>Consequently, we are allowed to combine discriminative and generative learning by optimizing pθ(y|x) and optimizing the log-likelihood of pθ(x) so that our model fits better to our data.</p><p>For maximizing the log likelihood of pθ(x) it is necessary to compute the derivative of log pθ(x),</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*3BM5AFpmwncBZ_dEh9ewzw.png" /></figure><p>with the expectation term</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*34FG8-QmglHVHWsLyh86_w.png" /></figure><p>This term could only be estimated, which is done by sampling x’ through Stochastic Gradient Langevin Dynamics (SGLD) (Welling and Teh, 2011), where x describes the dataset on which we wanted to maximize the log likelihood and x’ describes the adversarial dataset, sampled from p(x) via SGLD.</p><p>The output of a classifier could be interpreted as an energy function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*cCiaV4M9Kx0hWNiLEY_H_Q.png" /></figure><p>without changing the parameterization θ of the neural network (Liu et al., 2020).</p><p>So, in addition to the gradient decent of the discriminative training, we could compute the generative loss of our model as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*njKSSGPAXa9wa3bEpQ928g.png" /></figure><p>How can we implement these findings for image classification and what improvements do we get?</p><p>At first, we have to implement the following SGLD algorithm to sample our adverserial images:</p><figure><img alt="SGLD algorithm" src="https://cdn-images-1.medium.com/max/682/1*PNQCWm-TmE80O7cT_12Igw.png" /><figcaption>SGLD Sampling (<a href="https://openreview.net/forum?id=Hkxzx0NtDB">Grathwohl et al., 2020</a>)</figcaption></figure><pre>def get_sample_q(args, device):<br>    def sample_p_0(replay_buffer, bs):<br>        if len(replay_buffer) == 0:<br>            return init_random(args, bs), []<br>        buffer_size = len(replay_buffer) <br>        inds = torch.randint(0, buffer_size, (bs,))<br>        buffer_samples = replay_buffer[inds]<br>        random_samples = init_random(args, bs)<br>        choose_random = (torch.rand(bs) &lt; args.reinit_freq).float()[:, None, None, None]<br>        samples = choose_random * random_samples + (1 - choose_random) * buffer_samples<br>        return samples.to(device), inds<br><br>    def sample_q(f, replay_buffer, n_steps=n_steps):<br>        f.eval()<br>        # insert your batch size here<br>        bs = 16<br>        # generate initial samples and buffer inds of those samples (if buffer is used)<br>        init_sample, buffer_inds = sample_p_0(replay_buffer, bs=bs)<br>        x_k = torch.autograd.Variable(init_sample, requires_grad=True)<br>        # sgld<br>        # specify SGLD learning rate and sgld std<br>        sgld_lr = 0.1<br>        sgld_std = 0.001<br>        for k in range(n_steps):<br>            f_prime = torch.autograd.grad(f(x_k, y=y).sum(), [x_k], retain_graph=True)[0]<br>            x_k.data += sgld_lr * f_prime + sgld_std * torch.randn_like(x_k)<br>        f.train()<br>        final_samples = x_k.detach()<br>        # update replay buffer<br>        if len(replay_buffer) &gt; 0:<br>            replay_buffer[buffer_inds] = final_samples.cpu()<br>        return final_samples<br>    return sample_q</pre><p>In your training file you have to sample new adverserial images and compute the generative loss every epoch. So add the following lines to your script, where you iterate over n epochs:</p><pre>buffer_size = 10000<br>replay buffer = t.FloatTensor(buffer_size, num_channels, im_size, im_size).uniform_(-1, 1)<br>p_x_weight = 1<br><br>for epoch in range(n_epochs):<br>  ...<br>  ...<br>  ...<br><br>  x_q = sample_q(model, replay_buffer)<br><br>  fp_all = model(x_p_d)<br>  fq_all = model(x_input)<br>  fp = fp_all.mean()<br>  fq = fq_all.mean()<br><br>  l_p_x = -(fp - fq)<br>                                                                                    fp - fq))<br>  Loss += p_x_weight * l_p_x</pre><p>The full code implementation can be found <a href="https://github.com/wgrathwohl/JEM/tree/master">here</a>!</p><p>By adding these two functions and a few lines of code we can add a generative loss to our discriminative loss during training. The benefits of doing so are the following:</p><ul><li>Better Model Calibration</li><li>More Robust Classification</li><li>Possible Generative Modelling</li><li>Out-Of-Distribution Detection</li></ul><p>A detailed analysis of the benefits is shown <a href="https://openreview.net/forum?id=Hkxzx0NtDB">here</a>!</p><p>Joint Energy-Based modeling can also be used to do semi-supervised learning by minimizing the log-likelihood on unlabeled data (Zhao et al., 2020).</p><p>So in this story we have seen how we can combine generative and discriminative modeling to improve our model with just a few lines of code to make it more robust, get a better calibrated model and enable an out-of-distribution detection.</p><p>I hope you liked this article!</p><p>For more projects on the topic deep learning please have a look at my <a href="https://github.com/stefanherdy">GitHub</a> page!</p><p>All the best,<br>Stefan</p><h4>References</h4><p>Grathwohl, W., Wang, K.-C., Jacobsen, J.-H., Duvenaud, D., Norouzi, M., Swersky, K., 2020. Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One. Presented at the 8th International Conference on Learning Representations. <a href="https://doi.org/10.48550/arXiv.1912.03263">https://doi.org/10.48550/arXiv.1912.03263</a></p><p>LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., Huang, F.J., 2006. A Tutorial on Energy-Based Learning, in: Bakir, G., Hofman, T., Scholkopt, B., Smola, A., Taskar, B. (Eds.), MIT Press.</p><p>Liu, W., Wang, X., Owens, J.D., Li, Y., 2020. Energy-based Out-of-distribution Detection, in: NIPS’20: Proceedings of the 34th International Conference on Neural Information Processing Systems. arXiv, pp. 21464–21475. <a href="https://doi.org/10.48550/ARXIV.2010.03759">https://doi.org/10.48550/ARXIV.2010.03759</a></p><p>Welling, M., Teh, Y.W., 2011. Bayesian Learning via Stochastic Gradient Langevin Dynamics 8</p><p>Zhao, S., Jacobsen, J.-H., Grathwohl, W., 2020. Joint Energy-Based Models for Semi-Supervised Classification, in: ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bf7e403b81b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Export Labelbox Annotations]]></title>
            <link>https://medium.com/@stefan.herdy/how-to-export-labelbox-annotations-eedb8cb4f217?source=rss-39c152daa6b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/eedb8cb4f217</guid>
            <category><![CDATA[image-annotation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[semantic-segmentation]]></category>
            <category><![CDATA[labelbox]]></category>
            <dc:creator><![CDATA[Stefan Herdy]]></dc:creator>
            <pubDate>Tue, 03 Oct 2023 10:24:30 GMT</pubDate>
            <atom:updated>2023-12-11T10:35:21.030Z</atom:updated>
            <content:encoded><![CDATA[<p>A key part of developing custom machine learning applications is creating own datasets that are tailored to the desired application. A widely used tool to create these datasets is the website <a href="https://labelbox.com/">Labelbox</a>. Labelbox is a data annotation platform that facilitates the process of labeling and annotating large datasets for machine learning and artificial intelligence (AI) projects. It offers a user-friendly interface and a suite of tools that enable individuals and teams to efficiently annotate images, text, and other data types. It is widely used in industries such as computer vision, natural language processing, and autonomous driving to create high-quality training data for AI models. It helps streamline the labeling process, improve data quality, and accelerate the development of AI applications by providing a centralized platform for labeling tasks and collaboration among annotators and data scientists.</p><p>This story is about how we can download segmentation datasets from Labelbox and save the annotations as PNG files consisting of categorical values or as RGB images.</p><p>As a first step we have to create our own API-key in Labelbox to be able to download the required data.</p><p>Click on your profile button in the lower right corner, go to “Worspace settings” and click on “New API key”.</p><figure><img alt="Generate Labelbox API Key" src="https://cdn-images-1.medium.com/max/838/1*CIb4rb6WACehla_zdHQV8A.png" /></figure><p>This key will be used later, so we have to copy and save it.</p><p>We also need the ID of the project that we want to export. Select your project and go to “Export”. Here you will find your project ID.</p><figure><img alt="Labelbox project ID" src="https://cdn-images-1.medium.com/max/1024/1*lgqj2MvAVf8Wul50827tcA.png" /></figure><p>Now we create a YAML-file that contains our private API-key and our project ID and we save it as config.yaml</p><pre>api_key: &#39;INSERT-YOUR-API-KEY-HERE&#39;<br>project_id: &#39;INSERT-YOUR-PROJECT-ID-HERE&#39;</pre><p>Next, go to your desired project, select “Data Rows” and export the JSON-File that contains the necessary export information under “Export data v2”.</p><figure><img alt="Labelbox data export" src="https://cdn-images-1.medium.com/max/917/1*znNwVbrlaTZc0IEPgyLElQ.png" /></figure><p>Go to your notifictions to download your export-result.ndjson file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YN2nBRgS-Abi1hu9Vp9lTg.png" /></figure><p>You will be asked what data you want to export. Click on “Select all” to also export the project meta data and export the json file.</p><figure><img alt="Labelbox Label Export" src="https://cdn-images-1.medium.com/max/1024/1*vFc6WJLRWt6RWG5ETc8RSQ.png" /></figure><p>Now we have our API-key, our project ID and our export-result.ndjson file, so we are ready to export our images. Create the new folders called “labels_colour” and “labels_categorical” and a new python script with the following code:</p><pre>#!/usr/bin/env python3<br><br>import yaml<br>import requests<br>import cv2<br>import numpy as np<br>import ndjson<br>import os<br><br># Helper function to transform categorical image to colour image<br>def logits2rgb(img):<br>    # Specify custom colours<br>    red = [200, 0, 10]<br>    green = [187,207, 74]<br>    blue = [0,108,132]<br>    yellow = [255,204,184]<br>    black = [0,0,0]<br>    white = [226,232,228]<br>    cyan = [174,214,220]<br>    orange = [232,167,53]<br><br>    colours = [red, green, blue, yellow, black, white, cyan, orange, green, blue]<br><br>    shape = np.shape(img)<br>    h = int(shape[0])<br>    w = int(shape[1])<br>    col = np.zeros((h, w, 3))<br>    unique = np.unique(img)<br>    for i, val in enumerate(unique):<br>        mask = np.where(img == val)<br>        for j, row in enumerate(mask[0]):<br>            x = mask[0][j]<br>            y = mask[1][j]<br>            col[x, y, :] = colours[int(val)]<br><br>    return col.astype(int)<br><br>def get_mask(PROJECT_ID, api_key, colour, class_indices, destination_path_colour, destination_path_categorical):<br>    # Open export json. Change name to your export file if required<br>    with open(&#39;./export-result.ndjson&#39;) as f:<br>        data = ndjson.load(f)<br>        # Iterate over all images<br>        if not os.path.isdir(destination_path_categorical):<br>            os.mkdir(destination_path_categorical)<br>        if not os.path.isdir(destination_path_colour):<br>            os.mkdir(destination_path_colour)<br>        for i, d in enumerate(data):<br>            files_in_folder = os.listdir(&#39;./labels_categorical/&#39;)<br>            image_name = data[i][&#39;data_row&#39;][&#39;external_id&#39;]<br>            label_name = image_name.replace(&quot;.JPG&quot;, &quot;&quot;) + &#39;-mask.png&#39;<br>            if label_name not in files_in_folder:<br>                mask_full = np.zeros((data[i][&#39;media_attributes&#39;][&#39;height&#39;], data[i][&#39;media_attributes&#39;][&#39;width&#39;]))<br>                # Iterate over all masks<br>                for idx, obj in enumerate(data[i][&#39;projects&#39;][PROJECT_ID][&#39;labels&#39;][0][&#39;annotations&#39;][&#39;objects&#39;]):<br>                    # Extract mask name and mask url<br>                    name = data[i][&#39;projects&#39;][PROJECT_ID][&#39;labels&#39;][0][&#39;annotations&#39;][&#39;objects&#39;][idx][&#39;name&#39;]<br>                    url = data[i][&#39;projects&#39;][PROJECT_ID][&#39;labels&#39;][0][&#39;annotations&#39;][&#39;objects&#39;][idx][&#39;mask&#39;][&#39;url&#39;]<br><br>                    cl = class_indices[name]<br>                    print(f&#39;Class {name} assigned to class index {cl}&#39;)<br>                    <br>                    # Download mask<br>                    headers = {&#39;Authorization&#39;: api_key}<br>                    with requests.get(url, headers=headers, stream=True) as r:<br>                        r.raw.decode_content = True<br>                        mask = r.raw<br>                        image = np.asarray(bytearray(mask.read()), dtype=&quot;uint8&quot;)<br>                        image = cv2.imdecode(image, cv2.IMREAD_GRAYSCALE)<br>                    # Assign mask index to image-mask <br>                    mask = np.where(image == 255)<br>                    mask_full[mask] = cl<br><br>                unique = np.unique(mask_full)<br>                print(&#39;The masks of the image are: &#39;)<br>                print(unique)<br>                if len(unique) &gt; 1:<br>                    if colour == True:<br>                        mask_full_colour = logits2rgb(mask_full)<br>                        mask_full_colour = cv2.cvtColor(mask_full_colour.astype(&#39;float32&#39;), cv2.COLOR_RGB2BGR)<br>                    # Save Image<br>                    cv2.imwrite(destination_path_colour + image_name.replace(&quot;.JPG&quot;, &quot;&quot;) + &#39;-mask.png&#39;, mask_full_colour)<br>                cv2.imwrite(destination_path_categorical + image_name.replace(&quot;.JPG&quot;, &quot;&quot;) + &#39;-mask.png&#39;, mask_full)<br>            else:<br>                print(f&#39;File {label_name} already processed!&#39;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    with open(&#39;./config.yaml&#39;, &#39;r&#39;) as file:<br>        config = yaml.safe_load(file)<br>    project_id = config[&#39;project_id&#39;]<br>    api_key = config[&#39;api_key&#39;]<br>    colour = True<br><br>    destination_path_colour = &#39;./labels_colour/&#39;<br>    destination_path_categorical = &#39;./labels_categorical/&#39;<br><br>    class_indices = { &quot;class 1&quot; : 1,<br>                        &quot;class 2&quot; : 2,<br>                        &quot;class 3&quot; : 3,<br>                        &quot;class 4&quot; : 4,<br>                        &quot;class 5&quot; : 5,<br>                        &quot;class 6&quot; : 6,<br>                        }<br>    <br>    get_mask(project_id, api_key, colour, class_indices, destination_path_colour, destination_path_categorical)<br></pre><p>We can run the script to download our image annotations to our label folder. We can set the parameter “colour” to true if we want to save our annotations as RGB images or we can set the parameter to false if we want so save our annotations as categorical values. As a last step, the class indices must be changed to your specific class names.</p><p>I hope this article helped you and you are now ready to create your own semantic segmentation datasets using Labelbox.</p><p>The provided code can be downloaded <a href="https://gist.github.com/stefanherdy/cd04f82e0472f020e63a590d15a8b96f">here</a>!</p><p>A full semantic segmentation project can be found on my <a href="https://github.com/stefanherdy/pytorch-semantic-segmentation">GitHub</a>!</p><p>All the best,<br>Stefan</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eedb8cb4f217" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Evaluate Semantic Segmantation Models]]></title>
            <link>https://medium.com/@stefan.herdy/how-to-evaluate-semantic-segmantation-models-cd2539673701?source=rss-39c152daa6b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd2539673701</guid>
            <category><![CDATA[intersection-over-union]]></category>
            <category><![CDATA[semantic-segmentation]]></category>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[iou]]></category>
            <dc:creator><![CDATA[Stefan Herdy]]></dc:creator>
            <pubDate>Fri, 14 Jul 2023 08:06:46 GMT</pubDate>
            <atom:updated>2024-08-26T16:06:47.807Z</atom:updated>
            <content:encoded><![CDATA[<p>The evaluation of semantic image segmentation models is a critical aspect of assessing their performance and understanding their capabilities. It involves quantitatively and qualitatively measuring how well these models can accurately segment objects or regions of interest in an image.</p><p>Quantitative evaluation metrics are commonly used to assess the performance of semantic segmentation models. One widely used metric is pixel accuracy, which calculates the percentage of correctly classified pixels in relation to the total number of pixels in the image.</p><h4>Pixel Accuracy</h4><p>When using Pytorch we have a torch.Tensor model output of size: torch.Size([batch_size, num_classes, height, width]) and our target images of size torch.Size([batch_size, height, width]). We can compute the pixel accuracy very easily using numpy:</p><pre>acc = numpy.mean((logits.max(1)[1] == y_p_d).float().cpu().numpy())</pre><p>As we can see, we can calculate pixel-wise accuracy very easily, but this metric has significant drawbacks when we are dealing with unbalanced datasets. Let’s consider we have a simple dataset, where 95% of the pixels are class 1 and 5% of the pixels class 2. If the model would always classify class 1 (i.e. the model would not learn anything), the accuracy of the model would be 95%, which may seem very high if you do not know the data set.</p><h4>Intersection over Union</h4><p>To avoid this issue, there is a much more meaningful metric: The Intersection over Union (IoU), also known as the Jaccard index. IoU measures the overlap between the predicted segmentation and the ground truth segmentation for each class or object in the image.</p><figure><img alt="Input over Union" src="https://cdn-images-1.medium.com/max/703/1*XSN0_AI4kSrQ_oyFnYZkXQ.png" /><figcaption>Intersection over Union</figcaption></figure><p>For more than two classes we have co compute the IoU for every class separately and then take the mean over all IoU, so for multiclass segmentation we call it mean IoU (mIoU).</p><p>Let’s consider the same Pytorch model output as a torch.Tensor of size: torch.Size([batch_size, num_classes, height, width]) and our target images of size torch.Size([batch_size, height, width]). Then the mIoU can be computed as:</p><pre>def mIOU(prediction, label, num_classes):<br>    prediction= prediction.max(1)[1].float().cpu().numpy()<br>    label = label.float().cpu().numpy() <br>    <br>    iou_list = list()<br>    present_iou_list = list()<br><br>    for sem_class in range(num_classes):<br>        pred_inds = (pred == sem_class)<br>        target_inds = (label == sem_class)<br>        if target_inds.sum().item() == 0:<br>            iou_now = float(&#39;nan&#39;)<br>        else:<br>            intersection_now = (pred_inds[target_inds]).sum().item()<br>            union_now = pred_inds.sum().item() + target_inds.sum().item() - intersection_now<br>            iou_now = float(intersection_now) / float(union_now)<br>            present_iou_list.append(iou_now)<br>        iou_list.append(iou_now)<br>    miou = np.mean(present_iou_list)<br>    return miou</pre><h4>Precision, Recall and F1 Score</h4><p>These three metrics are standard evaluation metrics that are not only used for semantic segmentation, but also for other tasks like image classification, binary classification, object detection, or anomaly detection.</p><p>Precision is the measure of how many of the predicted positive instances are actually correct. It is the ratio of correctly predicted positive observations to the total predicted positive observations. Precision is crucial when the cost of a false positive is high.</p><pre>from sklearn.metrics import precision_score<br><br># Calculate precision using sklearn.metrics<br>precision = precision_score(label, prediction)<br>print(&quot;Precision:&quot;, precision)<br><br># Calculate precision manually<br># Calculate True Positives (TP) and False Positives (FP)<br>TP = sum((yt == 1 and yp == 1) for yt, yp in zip(label, prediction))<br>FP = sum((yt == 0 and yp == 1) for yt, yp in zip(label, prediction))<br><br>precision = TP / (TP + FP) if (TP + FP) &gt; 0 else 0<br>print(&quot;Manual Precision:&quot;, precision)</pre><p>Recall (also known as Sensitivity or True Positive Rate) measures how many of the actual positive instances were correctly predicted. It is the ratio of correctly predicted positive observations to all the observations in the actual class. Recall is crucial when the cost of a false negative is high.</p><pre>from sklearn.metrics import recall_score<br><br># Calculate recall using sklearn.metrics<br>recall = recall_score(label, prediction)<br>print(&quot;Recall:&quot;, recall)<br><br># Calculate recall manually<br># Calculate True Positives (TP) and False Negatives (FN)<br>TP = sum((yt == 1 and yp == 1) for yt, yp in zip(label, prediction))<br>FN = sum((yt == 1 and yp == 0) for yt, yp in zip(label, prediction))<br><br>recall = TP / (TP + FN) if (TP + FN) &gt; 0 else 0<br>print(&quot;Manual Recall:&quot;, recall)</pre><p>The F1-Score is the harmonic mean of Precision and Recall. It provides a single metric that balances both the precision and recall of a model, especially useful when the class distribution is imbalanced. The F1-Score ranges between 0 and 1, with 1 being the best possible score.</p><pre>from sklearn.metrics import f1_score<br><br># Calculate F1-Score using sklearn.metrics<br>f1 = f1_score(label, y_pred)<br>print(&quot;F1-Score:&quot;, f1)<br><br># Calculate True Positives (TP), False Positives (FP), and False Negatives (FN)<br>TP = sum((yt == 1 and yp == 1) for yt, yp in zip(label, prediction))<br>FP = sum((yt == 0 and yp == 1) for yt, yp in zip(label, prediction))<br>FN = sum((yt == 1 and yp == 0) for yt, yp in zip(label, prediction))<br><br># Calculate Precision and Recall<br>precision = TP / (TP + FP) if (TP + FP) &gt; 0 else 0<br>recall = TP / (TP + FN) if (TP + FN) &gt; 0 else 0<br><br># Calculate F1-Score manually<br>f1_score = 2 * (precision * recall) / (precision + recall) if (precision + recall) &gt; 0 else 0<br>print(&quot;Manual F1-Score:&quot;, f1_score)</pre><p>Apart from quantitative metrics, qualitative evaluation plays a crucial role in understanding the model’s segmentation outputs. Visual inspection of the segmented images can help identify common errors, such as under-segmentation, over-segmentation, or misclassification. Comparisons with ground truth annotations can provide insights into the model’s ability to handle complex scenes, fine details, and object boundaries.</p><p>I hope you liked my article and you can make use of the provided methods.</p><p>A full semantic segmentation project can be found on my <a href="https://github.com/stefanherdy/pytorch-semantic-segmentation">GitHub</a>!</p><p>All the best,<br>Stefan</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd2539673701" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pytorch Semantic Image Segmentation]]></title>
            <link>https://medium.com/@stefan.herdy/pytorch-semantic-image-segmentation-b726589662e3?source=rss-39c152daa6b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b726589662e3</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[semantic-segmentation]]></category>
            <dc:creator><![CDATA[Stefan Herdy]]></dc:creator>
            <pubDate>Thu, 13 Jul 2023 15:39:00 GMT</pubDate>
            <atom:updated>2023-12-11T10:55:05.658Z</atom:updated>
            <content:encoded><![CDATA[<p>Semantic image segmentation is a powerful computer vision technique that involves the understanding and analysis of images at a pixel level. It aims to assign a meaningful label to each pixel in an image, effectively dividing it into different regions or objects. Unlike traditional image classification, which assigns a single label to an entire image, semantic segmentation provides a more detailed and granular understanding of the visual content within an image.</p><p>In this article, we will walk through the process of how to train our custom neural networks for semantic image segmentation.</p><h4>Input Data</h4><p>As a first step we need to generate our own segmentation datasets. These datasets consist of image label pairs with an image (HxWxC) and a target (HxW) with H as height, W as width and C as number of channels. For standard RGB images we have 3 channels. However, if we use multi- or hyperspectral images as model input we can have more channels. The two dimensionel target array contains categorical values that refer to a certain class at every pixel location.</p><figure><img alt="Semantik Segmentation" src="https://cdn-images-1.medium.com/max/1024/1*pNZPetHFRTtPoszINbym3Q.jpeg" /></figure><figure><img alt="Semantic Segmantation" src="https://cdn-images-1.medium.com/max/1024/1*JHcdzFRbYog0-45YV12kZA.png" /><figcaption>Examples of an image target pair as imput data for the neural network training</figcaption></figure><p>To get these image target pairs we have to manually annotate images, which means that we have to assign a class to every pixel of an image. This image annotation can be done using some special software. One such tool that I could recommend is the online annotation editor <a href="https://labelbox.com/">Labelbox</a>.</p><p>There are a lot of tutorials on how to use Labelbox and they provide a very good documentation, so a tutorial on how to annotate the images is not part of this article. It is important to consider to <strong>not save the target images as .jpg-files</strong>,<strong> </strong>since the jpeg compression could add new integer numbers (new classes) to your files. Formats like .png are fine.</p><p>Once we have our own custom segmentation dataset we can start writing our Python scripts to train and evaluate our models. Let’s make a file utils.py, where we specify some helper functions for the training and to import our custom datasets and our pretrained UNet model.</p><pre>import os<br>import pickle<br>import json<br>from torch.utils.data import DataLoader<br>import pathlib<br>from customdatasets import CustomDataSet<br>from transformations import Compose, DenseTarget, RandomFlip, Resize_Sample<br>from transformations import MoveAxis, Normalize01, RandomCrop<br>import segmentation_models_pytorch as smp<br>from sklearn.model_selection import train_test_split<br>from os import walk<br>import torch as t<br>import numpy as np<br>import torch.nn as nn<br><br><br>def get_files(path):<br>    files = []<br>    for (dirpath, dirnames, filenames) in walk(path):<br>        for names in filenames:<br>            files.append(dirpath + &#39;/&#39; + names)<br>    return files<br><br>def makedirs(dirname):<br>    if not os.path.exists(dirname):<br>        os.makedirs(dirname)<br><br>def get_model(device, cl):<br>    <br>    unet = smp.Unet(&#39;resnet152&#39;, classes=cl, activation=None, encoder_weights=&#39;imagenet&#39;)<br><br>    if t.cuda.is_available():<br>        unet.cuda()         <br>    <br>    unet = unet.to(device)<br>    return unet<br><br>def import_data(args, batch_sz, set = &#39;project_3&#39;, crop_size):<br><br>    root = pathlib.Path(&#39;./&#39;)<br>    if set == &#39;project_1&#39;:<br>        inputs = get_files(&#39;./input_data/project_1/image/&#39;)<br>        targets = get_files(&#39;./input_data/project_1/target/&#39;)<br><br>    if set == &#39;project_2&#39;:<br>        inputs = get_files(&#39;./input_data/project_2/image/&#39;)<br>        targets = get_files(&#39;./input_data/project_2/target/&#39;)<br><br>    if set == &#39;project_3&#39;:<br>        inputs = get_files(&#39;./input_data/project_3/image/&#39;)<br>        targets = get_files(&#39;./input_data/project_3/target/&#39;)<br><br>    split = 0.8  <br><br>    inputs_train, inputs_valid = train_test_split(<br>        inputs,<br>        random_state=42,<br>        train_size=split,<br>        shuffle=True)<br><br>    targets_train, targets_valid = train_test_split(<br>        targets,<br>        random_state=42,<br>        train_size=split,<br>        shuffle=True)<br><br>    # Add your desired transformations here<br>    transforms = Compose([<br>        MoveAxis(),<br>        Normalize01(),<br>        RandomCrop(crop_size),<br>        RandomFlip()<br>        ])<br><br>    # train dataset<br>    dataset_train = CustomDataSet(inputs=inputs_train,<br>                                        targets=targets_train,<br>                                        transform=transforms)<br><br><br>    # validation dataset<br>    dataset_valid = CustomDataSet(inputs=inputs_valid,<br>                                        targets=targets_valid,<br>                                        transform=transforms)<br><br><br>    # train dataloader<br>    dataloader_training = DataLoader(dataset=dataset_train,<br>                                    batch_size=batch_sz,<br>                                    shuffle=True<br>                                    )<br><br>    # validation dataloader<br>    dataloader_validation = DataLoader(dataset=dataset_valid,<br>                                    batch_size=batch_sz,<br>                                    shuffle=True)<br>       <br>    return dataloader_training, dataloader_validation<br><br><br><br>def checkpoint(f, tag, args, device, dataloader_training, dataloader_validation):<br>    f.cpu()<br>    ckpt_dict = {<br>        &quot;model_state_dict&quot;: f.state_dict(),<br>        &quot;train&quot;: dataloader_training,<br>        &quot;valid&quot;: dataloader_validation<br><br>    }<br>    t.save(ckpt_dict, os.path.join(args.save_dir, tag))<br>    f.to(device)<br><br></pre><p>For the data import, we need to make some import classes. A detailed description of how to import and augment data for semantic segmentation can be found in my <a href="https://medium.com/@stefan.herdy/how-to-augment-images-for-semantic-segmentation-2d7df97544de">previous article</a>.</p><p>Like shown in this <a href="https://medium.com/@stefan.herdy/how-to-augment-images-for-semantic-segmentation-2d7df97544de">article</a>, we have to make our own dataset and transformation classes. Make a new file customdatasets.py with the following code:</p><pre>import torch<br>from skimage.io import imread<br>from torch.utils import data<br>from torchvision import transforms<br>import matplotlib.pyplot as plt<br>import numpy as np<br><br>class CustomDataSet(data.Dataset):<br>    def __init__(self,<br>                 inputs: list,<br>                 targets: list,<br>                 transform=None,<br>                 use_cache=False,<br>                 pre_transform=None,<br>                 ):<br>        self.inputs = inputs<br>        self.targets = targets<br>        self.transform = transform<br>        self.inputs_dtype = torch.float32<br>        self.targets_dtype = torch.long<br>        self.use_cache = use_cache<br>        self.pre_transform = pre_transform<br><br>        if self.use_cache:<br>            from multiprocessing import Pool<br>            from itertools import repeat<br><br>            with Pool() as pool:<br>                self.cached_data = pool.starmap(self.read_images, zip(inputs, targets, repeat(self.pre_transform)))<br><br>    def __len__(self):<br>        return len(self.inputs)<br><br>    def __getitem__(self,<br>                    index: int):<br>        if self.use_cache:<br>            x, y = self.cached_data[index]<br>        else:<br>            # Select the sample<br>            input_id = self.inputs[index]<br>            target_id = self.targets[index]<br><br>            # Load input and target<br>            x, y = imread(input_id), imread(target_id)<br>        <br>        # Preprocessing<br>        if self.transform is not None:<br>            x, y = self.transform(x, y)<br><br>        x, y = torch.from_numpy(x.copy()).type(self.inputs_dtype), torch.from_numpy(y.copy()).type(self.targets_dtype)<br><br>        return x, y<br><br>    @staticmethod<br>    def read_images(inp, tar, pre_transform):<br>        inp, tar = imread(inp), imread(tar)<br>        if pre_transform:<br>            inp, tar = pre_transform(inp, tar)<br>        return inp, tar</pre><p>Next, we have to make a file named transformations.py. In this file we specify our custom data transformations for preprocessing and augmentation.</p><pre>#!/usr/bin/python<br><br>import numpy as np<br>from sklearn.externals._pilutil import bytescale<br>import random<br>import matplotlib.pyplot as plt<br>import cv2<br>import random<br><br><br>def normalize_01(inp: np.ndarray):<br>    inp_out = (inp - np.min(inp)) / np.ptp(inp)<br>    return inp_out<br><br>def normalize(inp: np.ndarray, mean: float, std: float):<br>    inp_out = (inp - mean) / std<br>    return inp_out<br><br>def re_normalize(inp: np.ndarray,<br>                 low: int = 0,<br>                 high: int = 255<br>                 ):<br>    &quot;&quot;&quot;Normalize the data to a certain range. Default: [0-255]&quot;&quot;&quot;<br>    out = bytescale(inp, low=low, high=high)<br>    return out<br><br>class Compose:<br>    &quot;&quot;&quot;<br>    Composes several transforms together.<br>    &quot;&quot;&quot;<br><br>    def __init__(self, transforms: list):<br>        self.transforms = transforms<br><br>    def __call__(self, inp, target):<br>        for t in self.transforms:<br>            inp, target = t(inp, target)<br>        return inp, target<br><br>    def __repr__(self): return str([transform for transform in self.transforms])<br><br><br>class MoveAxis:<br>    &quot;&quot;&quot;From [H, W, C] to [C, H, W]&quot;&quot;&quot;<br><br>    def __init__(self, transform_input: bool = True, transform_target: bool = False):<br>        self.transform_input = transform_input<br>        self.transform_target = transform_target<br><br>    def __call__(self, inp: np.ndarray, tar: np.ndarray):<br>        inp = np.moveaxis(inp, -1, 0)<br>        #tar = np.moveaxis(tar, -1, 0)<br><br>        return inp, tar<br><br>    def __repr__(self):<br>        return str({self.__class__.__name__: self.__dict__})<br><br><br>class RandomFlip:<br>    def __init__(self):<br>        pass<br><br>    def __call__(self, inp: np.ndarray, tar: np.ndarray):<br>        rand = random.choice([0, 1])<br>        if rand == 1:<br>            #inp = np.ndarray.copy(np.fliplr(inp))<br>            inp = np.moveaxis(inp, 0, -1)<br>            inp = cv2.flip(inp, 1)<br>            inp = np.moveaxis(inp, -1, 0)<br>            tar = np.ndarray.copy(np.fliplr(tar))<br><br>        rand = random.choice([0, 1])<br>        if rand == 1:<br>            #inp = np.ndarray.copy(np.flipud(inp, axis=(1,2)))<br>            inp = np.moveaxis(inp, 0, -1)<br>            inp = cv2.flip(inp, 0)<br>            inp = np.moveaxis(inp, -1, 0)<br>            tar = np.ndarray.copy(np.flipud(tar))<br><br>        rand = random.choice([0, 1])<br>        if rand == 1:<br>            inp = np.ndarray.copy(np.rot90(inp, k=1, axes=(1, 2)))<br>            tar = np.ndarray.copy(np.rot90(tar, k=1, axes=(0, 1)))<br>        return inp, tar<br><br>    def __repr__(self):<br>        return str({self.__class__.__name__: self.__dict__})<br><br><br>class RandomCrop:<br>    def __init__(self, crop_size):<br>        self.crop_size = crop_size<br>        pass<br><br>    def __call__(self, inp: np.ndarray, tar: np.ndarray):<br>        max_x = inp.shape[1] - self.crop_size<br>        max_y = inp.shape[2] - self.crop_size<br>        x = random.randint(0, max_x)<br>        y = random.randint(0, max_y)<br><br>        # Crop<br>        inp = np.moveaxis(inp, 0, -1)<br>        inp = inp[x: x + self.crop_size, y: y + self.crop_size,:]<br>        inp = np.moveaxis(inp, -1, 0)<br>        tar = tar[x: x + self.crop_size, y: y + self.crop_size]<br><br>        return inp, tar<br><br><br>class Resize:<br>    def __init__(self, img_size):<br>        self.img_size = img_size<br>        pass<br><br>    def __call__(self, inp: np.ndarray, tar: np.ndarray):<br>        inp = np.moveaxis(inp, 0, -1)<br>        inp = cv2.resize(inp, (self.img_size,self.img_size), interpolation = cv2.INTER_NEAREST)<br>        inp = np.moveaxis(inp, -1, 0)<br>        tar = cv2.resize(tar, (self.img_size,self.img_size), interpolation = cv2.INTER_NEAREST)<br>        <br>        return inp, tar<br><br>class Normalize01:<br>    &quot;&quot;&quot;Squash image input to the value range [0, 1] (no clipping)&quot;&quot;&quot;<br><br>    def __init__(self):<br>        pass<br><br>    def __call__(self, inp, tar):<br>        inp = normalize_01(inp)<br><br>        return inp, tar<br><br><br>class Normalize:<br>    &quot;&quot;&quot;Normalize based on mean and standard deviation.&quot;&quot;&quot;<br>    def __init__(self,<br>                 mean: float,<br>                 std: float,<br>                 transform_input=True,<br>                 transform_target=False<br>                 ): <br><br>        self.transform_input = transform_input<br>        self.transform_target = transform_target<br>        self.mean = mean<br>        self.std = std<br><br>    def __call__(self, inp, tar):<br>        inp = normalize(inp)<br><br>        return inp, tar<br><br>class ColorTransformations:<br>    def __init__(self):<br>        pass<br><br>    def __call__(self, inp: np.ndarray, tar: np.ndarray):<br>        inp_tensor = torch.from_numpy(inp)<br>        tar_tensor = torch.from_numpy(tar)<br><br>        color_transform = transforms.Compose([<br>            transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.2),<br>        ])<br><br>        inp_tensor = color_transform(inp_tensor)<br><br>        inp = inp_tensor.numpy()<br>        tar = tar_tensor.numpy()<br><br>        return inp, tar<br><br>class ColorNoise:<br>    def __init__(self, noise_std=0.05):<br>        self.noise_std = noise_std<br><br>    def __call__(self, inp: np.ndarray, tar: np.ndarray):<br>        inp_tensor = torch.from_numpy(inp)<br>        tar_tensor = torch.from_numpy(tar)<br><br>        noise = torch.randn_like(inp_tensor) * self.noise_std<br>        inp_tensor += noise<br><br>        inp_tensor = torch.clamp(inp_tensor, 0, 1)<br><br>        inp = inp_tensor.numpy()<br>        tar = tar_tensor.numpy()<br><br>        return inp, tar<br></pre><p>Now we are ready to start our training. Our main file (train.py) loads the data and our pretrained model, trains the model on our custom data and saves the models etc. as checkpoints. For your own project you can specify your hyperparameters like learn rate, optimizer, number of epochs, batchsize etc. to achieve the best possible model performance. It should look like this:</p><pre>#!/usr/bin/env python3<br><br>import os<br>import pickle<br>import json<br>from utils import *<br>import torch as t<br>import torch.nn as nn <br>import argparse<br>import numpy as np<br>import matplotlib.pyplot as plt<br>from tqdm import tqdm<br>import random<br>t.backends.cudnn.benchmark = True<br>t.backends.cudnn.enabled = True<br><br>seed = 42 <br>os.environ[&#39;PYTHONHASHSEED&#39;]=str(seed)<br>random.seed(seed)<br>np.random.seed(seed)<br>t.manual_seed(seed)<br><br>def main(args):<br>    args.save_dir = &#39;./checkpoints/&#39;<br>    <br>    makedirs(args.save_dir)<br>    with open(f&#39;{args.save_dir}/params.txt&#39;, &#39;w&#39;) as f:<br>        json.dump(args.__dict__, f)<br>    <br>    t.manual_seed(seed)<br>    if t.cuda.is_available():<br>        t.cuda.manual_seed_all(seed)<br><br>    dload_train, dload_valid = import_data(args, args.batch_size, args.project, args.resize, args.random_crop_size)<br><br>    device = t.device(&#39;cuda&#39; if t.cuda.is_available() else &#39;cpu&#39;)<br>    f = get_model(device, args.num_classes)<br><br>    params = f.parameters() <br>    if args.optimizer == &quot;adam&quot;:<br>        optim = t.optim.Adam(params, lr=args.learnrate, betas=[.9, .999], weight_decay=0.0)<br>    else:<br>        optim = t.optim.SGD(params, lr=args.learnrate, momentum=.9, weight_decay=0.0)<br><br>    best_valid_acc = 0.0<br>    iteration = 0<br><br>    train_losses = []<br>    val_losses = []<br>    val_corr = []<br>    <br>    for epoch in range(args.epochs):<br>        iter_losses = []<br>        for i, (x_train, y_train) in tqdm(enumerate(dload_train)):<br>            x_train, y_train = next(iter(dload_train)) <br>            x_train, y_train = x_train.to(device), y_train.to(device)<br><br>            Loss = 0.<br><br>            logits = f(x_train)<br>            l_dis = nn.CrossEntropyLoss()(logits, y_train)<br>            Loss += l_dis<br>            iter_losses.append(Loss.item())<br><br>            optim.zero_grad()<br>            Loss.backward()<br>            optim.step()<br><br>        if iteration % args.print_every == 0:<br>            acc = (logits.max(1)[1] == y_train).float().mean()<br>            print(&#39;P(y|x) {}:{:&gt;d} loss={:&gt;14.9f}, acc={:&gt;14.9f}&#39;.format(epoch,<br>                                                                            iteration,<br>                                                                            l_dis.item(),<br>                                                                            acc.item()))<br><br>        iteration += 1<br><br>        train_losses.append(np.mean(iter_losses))<br><br>        if epoch % args.eval_every == 0:<br>            f.eval()<br>            with t.no_grad():<br>                correct, loss = eval_classification(f, dload_valid, device)<br>                val_losses.append(loss)<br>                val_corr.append(correct)<br>                print(&quot;Epoch {}: Valid Loss {}, Valid Acc {}&quot;.format(epoch, loss, correct))<br>                if correct &gt; best_valid_acc:<br>                    best_valid_acc = correct<br>                    print(&quot;Best Valid!: {}&quot;.format(correct))<br>                    checkpoint(f, &quot;best_validation_ckpt.pt&quot;, args, device, dload_train, dload_valid)<br>            f.train()<br>        if epoch % args.ckpt_every == 0:<br>            checkpoint(f, f&#39;checkpoint_{epoch}.pt&#39;, args, device, dload_train, dload_valid)<br>            <br>            # Losses are saved and can be loaded for further analysis<br>            # You can also plot them here using matplotlib<br>            with open(&quot;./records/trainlosses.txt&quot; , &quot;wb&quot;) as fp:<br>                pickle.dump(train_losses, fp)<br><br>            with open(&quot;./records/vallosses.txt&quot; , &quot;wb&quot;) as fp:<br>                pickle.dump(val_losses, fp)<br><br>            with open(&quot;./records/correct.txt&quot; , &quot;wb&quot;) as fp:<br>                pickle.dump(val_corr, fp)<br>            <br>if __name__ == &quot;__main__&quot;:<br>    parser = argparse.ArgumentParser(&quot;Pytorch Semantic Segmentation&quot;)<br>    parser.add_argument(&quot;--learnrate&quot;, type=int, default=0.0001, help=&#39;learn rate of optimizer&#39;)<br>    parser.add_argument(&quot;--optimizer&quot;, choices=[&#39;sgd&#39;, &#39;adam&#39;], default=&#39;adam&#39;)<br>    parser.add_argument(&quot;--epochs&quot;, type=int, default=7000)<br>    parser.add_argument(&quot;--eval_every&quot;, type=int, default=1, help=&quot;Epochs between evaluation&quot;)<br>    parser.add_argument(&quot;--print_every&quot;, type=int, default=5, help=&quot;Epochs between print&quot;)<br>    parser.add_argument(&quot;--ckpt_every&quot;, type=int, default=2, help=&quot;Epochs between checkpoint save&quot;)<br>    parser.add_argument(&quot;--project&quot;, choices=[&#39;project_1&#39;, &#39;project_2&#39;, &#39;project_3&#39;], default=&#39;project_1&#39;)<br>    parser.add_argument(&quot;--batch_size&quot;, type=int, default=2, help=&quot;Batch Size&quot;)<br>    parser.add_argument(&quot;--num_classes&quot;, type=int, default=8, help=&quot;Number of classes of your training dataset&quot;)<br>    parser.add_argument(&quot;--resize&quot;, type=int, default=512, help=&quot;Size of images for resizing&quot;)<br>    parser.add_argument(&quot;--random_crop_size&quot;, type=int, default=256, help=&quot;Size of random crops. Must be smaller than resized images.&quot;)<br>    args = parser.parse_args()<br>    if args.random_crop_size &gt; args.resize:<br>        raise Exception(&quot;Crop size (--random_crop_size) must be smaller than resized image (--resize)!&quot;)<br>    <br>    main(args)</pre><p>Congrats! Now we are ready to train our own custom semantic segmentation model in Pytorch.</p><p>The whole code of this article can be found on my GitHub <a href="https://github.com/stefanherdy/pytorch-semantic-segmentation">repository</a>.</p><p>I hope you liked this article!</p><p>All the best,<br>Stefan</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b726589662e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Augment Images for Semantic Segmentation]]></title>
            <link>https://medium.com/@stefan.herdy/how-to-augment-images-for-semantic-segmentation-2d7df97544de?source=rss-39c152daa6b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d7df97544de</guid>
            <category><![CDATA[semantic-segmentation]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[data-augmentation]]></category>
            <dc:creator><![CDATA[Stefan Herdy]]></dc:creator>
            <pubDate>Mon, 10 Jul 2023 20:49:19 GMT</pubDate>
            <atom:updated>2023-11-28T14:36:22.164Z</atom:updated>
            <content:encoded><![CDATA[<p>Semantic segmentation is a computer vision technique that aims to assign meaningful labels to each pixel in an image, enabling the understanding and differentiation of various objects or regions. One common challenge in training semantic segmentation models is the limited availability of annotated data. However, data augmentation techniques can significantly improve the learning process by artificially expanding the size and diversity of the training dataset. By applying transformations such as rotation, scaling, flipping, and adding noise to the input images, data augmentation introduces variations that the model can learn from. This augmentation helps in enhancing model generalization, reducing overfitting, and improving the segmentation accuracy by capturing a broader range of object appearances, shapes, and backgrounds.</p><p>In PyTorch, data augmentation is typically implemented using the torchvision.transforms module, which provides a variety of pre-defined image transformations that can be applied to the training data. The process of data augmentation involves creating a pipeline of transformations to be applied to the input images during training.</p><p>A typical transformation sequence in PyTorch looks like the following:</p><pre>import torchvision.transforms as transforms<br><br>transform = transforms.Compose([<br>    transforms.Resize((256, 256)),  # Resize the image to a specific size<br>    transforms.RandomHorizontalFlip(),  # Randomly flip the image horizontally<br>    transforms.RandomRotation(10),  # Randomly rotate the image by a given angle<br>    transforms.ToTensor(),  # Convert the image to a PyTorch tensor<br>    transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])  # Normalize the image<br>])</pre><p>The class transforms.Compose is used to specify the transformations we want to apply. Note that there are transformations that are applied to every image, like torchvision.Resize and torchvision.Normalize while other transformations are applied randomly, like torchvision.RandomRotation . This random transformations work perfectly fine for image classification.</p><p>When it comes to Semantic Segmentation we have image target pairs. The target image contains a categorical value for each pixel of the corresponding image that describes the class of the respective pixel. This means, that we have to perform the same transformations to the image <strong>and</strong> to the corresponding target image. For PyTorch, we can implement this by using our own custom transformer classes.</p><figure><img alt="Semantic Segmantation" src="https://cdn-images-1.medium.com/max/1024/1*JHcdzFRbYog0-45YV12kZA.png" /><figcaption>Example of an image target pair</figcaption></figure><p>A simple Color Jitter Class would look like the following:</p><pre>class ColorTransform:<br>    def __init__(self):<br>        pass<br><br>    def __call__(self, inp: np.ndarray, tar: np.ndarray):<br>        inp_tensor = torch.from_numpy(inp)<br>        tar_tensor = torch.from_numpy(tar)<br><br>        color_transform = transforms.Compose([<br>            transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.2),<br>        ])<br><br>        inp_tensor = color_transform(inp_tensor)<br><br>        inp = inp_tensor.numpy()<br>        tar = tar_tensor.numpy()<br><br>        return inp, tar</pre><p>A Random Crop Class would look like the following:</p><pre>class RandomCrop:<br>    def __init__(self, crop_size):<br>        self.crop_size = crop_size<br>        pass<br><br>    def __call__(self, inp: np.ndarray, tar: np.ndarray):<br>        max_x = inp.shape[1] - self.crop_size<br>        max_y = inp.shape[2] - self.crop_size<br>        x = random.randint(0, max_x)<br>        y = random.randint(0, max_y)<br><br>        # Crop<br>        # Assuming that image was loaded using OpenCV and/or has the shape 3xHxW<br>        # Target image shape: HxW<br>        inp = inp[:, x: x + self.crop_size, y: y + self.crop_size]<br>        tar = tar[x: x + self.crop_size, y: y + self.crop_size]<br><br>        return inp, tar</pre><p>Note that we always apply all the transformations to both, image and target.</p><p>Now we need to specify our Dataset Class:</p><pre>import torch<br>from skimage.io import imread<br>from torch.utils import data<br>from torchvision import transforms<br>import matplotlib.pyplot as plt<br>import numpy as np<br><br>class DataSet(data.Dataset):<br>    def __init__(self,<br>                 inputs: list,<br>                 targets: list,<br>                 transform=None,<br>                 ):<br>        self.inputs = inputs<br>        self.targets = targets<br>        self.transform = transform<br>        self.inputs_dtype = torch.float32<br>        self.targets_dtype = torch.int<br><br>    def __len__(self):<br>        return len(self.inputs)<br><br>    def __getitem__(self,<br>                    idx: int):<br>        # image and target dir<br>        input_ID = self.inputs[idx]<br>        target_ID = self.targets[idx]<br><br>        # Load input and target<br>        x, y = imread(input_ID), imread(target_ID)<br><br>        # Preprocessing<br>        if self.transform is not None:<br>            x, y = self.transform(x, y)<br><br>        x, y = torch.from_numpy(x.copy()).type(self.inputs_dtype), torch.from_numpy(y.copy()).type(self.targets_dtype)<br><br>        return x, y<br></pre><p>Next, we specify our custom Compose class:</p><pre>class Compose:<br>    &quot;&quot;&quot;<br>    Composes several transforms together.<br>    &quot;&quot;&quot;<br><br>    def __init__(self, transforms: list):<br>        self.transforms = transforms<br><br>    def __call__(self, input, target):<br>        for tr in self.transforms:<br>            input, target = tr(input, target)<br>        return input, target<br><br>    def __repr__(self): return str([transform for transform in self.transforms])</pre><p>And finally, to load our data for training we use the following:</p><pre>from torch.utils.data import DataLoader<br><br>crop_size = 256<br>transforms_train = Compose([<br>        ColorTransform(),<br>        RandomCrop(crop_size)<br>        ])<br><br># train dataset<br>dataset_train = DataSet(inputs=inputs,<br>                        targets=targets,<br>                        transform=transforms_train)<br><br># train dataloader<br>dataloader_training = DataLoader(dataset=dataset_train,<br>                                batch_size=batchsize,<br>                                shuffle=True<br>                                )</pre><p>We now have implementat our own custom data augmentation tool for semantic image segmentation.</p><p>I hope you liked this article!</p><p>To see more of my projects, please visit my <a href="https://github.com/stefanherdy/">GitHub </a>page!</p><p>The code for this tutorial is available <a href="https://gist.github.com/stefanherdy/2323eff239c8626e39b232506c994368">here</a>!</p><p>All the best,<br>Stefan</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d7df97544de" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>