<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sunidhi Ashtekar on Medium]]></title>
        <description><![CDATA[Stories by Sunidhi Ashtekar on Medium]]></description>
        <link>https://medium.com/@sunidhi.ashtekar?source=rss-1d599ba5668e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7-kIZ0iFYQ81P82JYBBw1g.jpeg</url>
            <title>Stories by Sunidhi Ashtekar on Medium</title>
            <link>https://medium.com/@sunidhi.ashtekar?source=rss-1d599ba5668e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 19:26:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sunidhi.ashtekar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Evolution of RAG: From Retrieval to Agentic and Multi-Agent Systems]]></title>
            <link>https://medium.com/@sunidhi.ashtekar/the-evolution-of-rag-from-retrieval-to-agentic-and-multi-agent-systems-403cd381eeed?source=rss-1d599ba5668e------2</link>
            <guid isPermaLink="false">https://medium.com/p/403cd381eeed</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sunidhi Ashtekar]]></dc:creator>
            <pubDate>Fri, 20 Feb 2026 01:07:26 GMT</pubDate>
            <atom:updated>2026-02-22T03:01:22.809Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rm4lRJ0xHKTg5jhsli4_vA.png" /><figcaption>Figure 1: Title image</figcaption></figure><p>In this article, we explore how Retrieval-Augmented Generation (RAG) evolved into Agentic and Multi-Agent RAG systems, why agentic behavior matters, the core principles of agentic intelligence, key architectural patterns, and how to build these systems using my open-source repository:</p><p><a href="https://github.com/sunidhiashtekar/agentic-rag">https://github.com/sunidhiashtekar/agentic-rag</a></p><p>This is not just a conceptual overview. The goal is to understand Agentic RAG and build it.</p><h3>Introduction: Why RAG Wasn’t Enough</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*07IQedY0RNNVG2GygA7xRA.png" /><figcaption>Figure 2: Vanilla RAG</figcaption></figure><p>Large Language Models (LLMs) have dramatically expanded what AI systems can do. They can write code, summarize research, analyze policies, and generate surprisingly coherent reasoning. But their knowledge is bounded by the data they were trained on, and they cannot inherently access new information, proprietary documents, or evolving real-world data.</p><p><strong>Retrieval-Augmented Generation (RAG)</strong> solved the challenge of grounding LLMs with external knowledge. Yet as tasks became multi-step and open-ended, even RAG began to show cracks. It could retrieve facts but not decide when to search again, how to plan multi-hop reasoning, or how to verify its own work. It still follows a linear “retrieve once, answer once” pattern. Complex, real-world problems demand something more adaptive, systems that can plan, reflect, and improve on their own reasoning.</p><p>This is where <strong>Agentic RAG</strong> comes in. It combines RAG’s grounding capabilities with intelligent agents capable of planning, tool use, collaboration, and self-correction. Instead of simply adding documents to prompts, Agentic RAG introduces decision-making into the pipeline.</p><p>In simple terms:</p><blockquote><em>Vanilla RAG retrieves information.<br>Agentic RAG thinks about how to retrieve, when to re-query, and how to improve itself.</em></blockquote><h3>A Running Example: Research Assistant</h3><p>Let’s ground this with a practical scenario. Suppose a user asks:</p><p>“What are the long-term effects of diabetes on cardiovascular health?”</p><p>A vanilla RAG system would retrieve a fixed number of documents and generate a single response based on that context.</p><p>An agentic system behaves differently. It may first decompose the question into sub-questions such as how diabetes affects arterial function, what longitudinal studies report, and how risk compounds over time. It retrieves targeted information, evaluates whether the answer is complete, refines queries if needed, and synthesizes a more structured response.</p><p>That shift from single-shot retrieval to iterative reasoning is the defining difference between RAG and Agentic RAG.</p><h3>The Evolution</h3><p>Before diving into agentic intelligence, let’s trace how RAG itself evolved.</p><p>Early implementations, or <strong>Naive</strong> <strong>RAG</strong>, used keyword-based retrieval. It worked for simple fact queries but often missed semantic meaning.</p><p>Next came <strong>Advanced RAG</strong>, where dense vector search and neural re-ranking improved semantic recall. Modular RAG followed, separating retrievers, generators, and re-rankers into interchangeable components that could be optimized independently.</p><p>Later, <strong>Graph RAG</strong> added graph-based reasoning, representing knowledge as interconnected nodes perfect for domains like medicine or finance, where relationships matter as much as facts.</p><p>These advancements improved retrieval quality. Agentic RAG extends this trajectory by introducing dynamic planning, iteration, and self-correction into the pipeline.</p><p>In short, RAG evolved from <strong>retrieval + generation</strong> to <strong>retrieval + reasoning + reflection</strong>.</p><h3>Core Principles of Agentic Intelligence</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RoVK0kZeEbVjqMkCR0ccug.png" /><figcaption>Figure 3: Core Principles of Agentic RAG</figcaption></figure><p>Traditional RAG systems are fundamentally reactive: they wait for a query, fetch context, and respond once. Agentic RAG transforms this into <strong>proactive intelligence</strong>, where agents reason about the task, plan retrieval strategies, and collaborate with other agents to improve both accuracy and adaptability.</p><p>Each agent typically includes an LLM as its reasoning core, along with short-term and long-term memory, a planning mechanism for task decomposition, and tool interfaces for interacting with external systems.</p><p>These agents work together using four design patterns:</p><ol><li><strong>Reflection</strong> agents critique and improve their own outputs, identifying gaps or inconsistencies.</li><li><strong>Planning </strong>agents decompose complex goals into sub-tasks and sequence them logically.</li><li><strong>Tool Use </strong>agents call external APIs or databases, blending symbolic reasoning with live data retrieval.</li><li><strong>Multi-Agent Collaboration </strong>multiple specialized agents coordinate via communication and shared memory, enabling parallel reasoning and specialization.</li></ol><p>Collectively, these patterns underpin the intelligence layer of Agentic RAG, turning retrieval into an evolving cognitive process rather than a static pipeline.</p><h3>Agentic Workflow Patterns</h3><p>Beyond architecture, Agentic RAG depends on how tasks are executed.</p><ol><li><strong>Prompt Chaining: </strong>A<strong> </strong>sequential reasoning, where one output becomes another agent’s input (ideal for code review or content creation). It improves accuracy but can increase latency due to sequential execution.</li><li><strong>Routing: </strong>A router agent analyzes each query and decides which specialized agent or prompt to use. This pattern optimizes efficiency.</li><li><strong>Parallelization: </strong>Agents run concurrently for faster multi-source retrieval. It can be used in two ways: <strong><em>Sectioning</em></strong><em> </em>where<em> </em>each agent handles part of a document and <strong><em>Voting</em></strong><em> </em>where<em> </em>multiple agents generate outputs, and a final voter model chooses the best. This reduces latency and increases reliability through redundancy.</li><li><strong>Orchestrator-Worker: </strong>The<strong> </strong>central planner decomposes goals and assigns them to worker agents, then synthesizes results.</li><li><strong>Evaluator-Optimizer: </strong>The<strong> </strong>iterative feedback loops refine outputs until confidence thresholds are met.</li></ol><p>These workflow patterns determine how reasoning unfolds in practice from simple sequential chains to fully autonomous orchestration. By combining reflection, planning, tool use, and multi-agent collaboration, developers can design adaptive, self-correcting pipelines that far exceed the capabilities of static RAG.</p><h3>Architecture of Agentic RAG Systems</h3><p>Architecturally, agentic RAG systems can be organized along increasing levels of coordination and autonomy. Agentic RAG is not a single architecture, but a spectrum of adaptive designs. As reasoning complexity increases, coordination becomes the defining factor in system robustness, scalability, and maintainability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WEcG2nOpYNvv7LWbuVUXTQ.png" /><figcaption>Figure 4: Overview of Multi-Agent Agentic RAG Systems</figcaption></figure><h4>Single-Agent Architecture</h4><p>In this simplest form, one agent handles retrieval, tool routing, and response generation. It decides whether to query a vector database, a structured SQL source, or a web API, and then synthesizes a unified answer. This approach is lightweight, ideal for small teams or limited toolchains think customer-support chatbots or internal Q&amp;A assistants.</p><h4>Multi-Agent Architecture</h4><p>Here, different agents specialize: one retrieves structured data, another performs web search, a third analyzes unstructured documents, and a fourth evaluates the final result.</p><p>A coordinator agent orchestrates them, sending subtasks in parallel and merging their outputs. This dramatically improves scalability and precision for multi-domain reasoning such as financial analytics or research synthesis.</p><h4>Hierarchical Agentic RAG</h4><p>In hierarchical systems, a master agent delegates subtasks to lower-level agents similar to how managers assign tasks to specialized teams. This is especially useful in enterprise use cases where queries span structured databases, document repositories, and policy APIs simultaneously.<br>For example, a top-level agent in a financial assistant might assign:</p><ul><li>Market analysis to one sub-agent</li><li>Risk assessment to another</li><li>News summarization to a third and then synthesize the results into a coherent final insight</li></ul><h4>Corrective RAG (CRAG)</h4><p>One of the most interesting patterns described in the paper is Corrective RAG, which builds self-critique directly into the retrieval loop. If an answer seems incomplete or inconsistent, the relevance-evaluation agent triggers query refinement or external web search before regenerating an improved response.</p><p>It’s like giving your RAG pipeline a “second thought.” This mechanism drastically reduces hallucinations and enhances factual reliability.</p><h4>Adaptive RAG</h4><p>In Adaptive RAG, a small classifier estimates the complexity of each query and dynamically adjusts the retrieval depth:</p><ul><li>Simple factual queries are answered instantly.</li><li>Moderate ones trigger single-step retrieval.</li><li>Complex, multi-hop questions invoke multi-stage reasoning loops.</li></ul><p>This adaptive throttling makes RAG systems both faster and smarter, allocating effort where it truly matters.</p><h4>Graph-Based Agentic RAG</h4><p>Frameworks like <strong>Agent-G</strong> and <strong>GeAR</strong> merge structured knowledge graphs with traditional text retrieval. Agents navigate node relationships, like connecting diabetes → insulin → heart disease while pulling supporting literature from unstructured documents.</p><p>This hybrid reasoning is especially effective in medicine, law, and academic research, where cause-effect relationships and hierarchy matter.</p><h3>Challenges Ahead</h3><p>Despite its promise, Agentic RAG remains an evolving systems paradigm. Several challenges continue to limit widespread production deployment.</p><p>Coordination complexity remains a major hurdle. Synchronizing multi-agent workflows without introducing latency bottlenecks or cascading failures requires careful architectural design.</p><p>Evaluation remains underdeveloped. Traditional RAG benchmarks measure retrieval accuracy, but they do not capture adaptive reasoning quality, reflection effectiveness, or planning reliability.</p><p>Observability and debugging become significantly harder as agent loops deepen. When systems dynamically decide what to retrieve and when to iterate, tracing failure points becomes non-trivial.</p><p>Scalability and cost are persistent concerns. Iterative reasoning increases token usage, memory state management, and inference overhead. Without adaptive throttling, agentic pipelines can quickly become computationally expensive.</p><p>Addressing these issues will be crucial as organizations move from experimental prototypes to production-grade agentic systems.</p><h3>Future Outlook</h3><p>Agentic RAG represents a structural shift in how retrieval systems are designed, moving from static augmentation toward adaptive reasoning infrastructures. It narrows the gap between raw knowledge access and decision-aware information systems.</p><p>Emerging directions include multimodal agentic RAG, where text, vision, and audio agents collaborate within unified reasoning loops. Research is also exploring self-evaluating systems capable of dynamically generating and scoring their own intermediate outputs.</p><p>Another promising direction is memory-augmented agentic systems, where long-term memory mechanisms refine retrieval and planning strategies over time based on accumulated interaction data.</p><p>Rather than aiming for abstract “self-awareness,” the more immediate trajectory is toward reliable, adaptive AI systems capable of controlled autonomy within well-defined operational boundaries.</p><h3>Hands-On: Build Your Own Agentic RAG</h3><p>You can experiment with these agentic patterns directly in my open-source repository.</p><p>Start by cloning the repository and creating a virtual environment:</p><pre>git clone https://github.com/sunidhiashtekar/agentic-rag.git<br>cd agentic-rag<br>python3.12 -m venv .venv<br>source .venv/bin/activate</pre><p>Install the required dependencies and set your OpenAI API key before running the examples:</p><pre>pip install -r notebooks/requirements.txt<br>export OPENAI_API_KEY=&quot;sk-...&quot;</pre><p>Build a sample knowledge index, run the assistant with configurable parameters and then run evaluation checks to test retrieval and reasoning quality:</p><pre>python example_app/ingest.py<br>python example_app/main.py --model gpt-4.1-mini --top-k 4 --verbose --show-sources<br>python example_app/evaluate.py --cases example_app/evaluation/cases.yaml --show-context</pre><p>If you prefer learning step-by-step, start with the notebooks. The repository is structured as a guided progression from baseline RAG to iterative, multi-agent reasoning workflows.</p><h3>Conclusion</h3><p>Agentic RAG is more than an incremental extension of retrieval-augmented generation. It represents a shift in architectural philosophy from static retrieval pipelines to adaptive reasoning systems with controlled autonomy.</p><p>By embedding decision-making into the retrieval process, we move closer to AI systems that can plan, evaluate, and refine their own outputs within structured constraints.</p><p>The architectures discussed here are not an exhaustive taxonomy of RAG variants. The ecosystem continues to evolve, with approaches such as Self-RAG, memory-augmented retrieval systems, and reinforcement-learning-driven retrieval emerging in research. However, the patterns outlined above form the foundational architectural categories that most modern agentic systems build upon.</p><h3><strong>References</strong></h3><p>To further explore the concepts and build your own agentic rag system, I recommend visiting the following resources:</p><ul><li><a href="https://github.com/sunidhiashtekar/agentic-rag">https://github.com/sunidhiashtekar/agentic-rag</a></li><li><a href="https://arxiv.org/pdf/2501.09136">https://arxiv.org/pdf/2501.09136</a></li><li><a href="https://arxiv.org/abs/2005.11401">RAG Foundation Paper</a></li><li><a href="https://arxiv.org/abs/2310.11511">Self-RAG (self-reflection in retrieval/generation)</a></li><li><a href="https://arxiv.org/abs/2401.15884">Corrective RAG (CRAG)</a></li><li><a href="https://arxiv.org/abs/2404.16130">GraphRAG</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=403cd381eeed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Deep Dive into VLMs: Vision-Language Models]]></title>
            <link>https://medium.com/@sunidhi.ashtekar/a-deep-dive-into-vlms-vision-language-models-d3bdf2a3e728?source=rss-1d599ba5668e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3bdf2a3e728</guid>
            <category><![CDATA[vlm]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Sunidhi Ashtekar]]></dc:creator>
            <pubDate>Thu, 24 Oct 2024 18:49:29 GMT</pubDate>
            <atom:updated>2024-10-24T18:49:29.842Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iLtvN9V0S9G-hhSE0uAOag.png" /><figcaption>Image <a href="https://medium.com/@letscodeai/introduction-c071c5433e89https://medium.com/@letscodeai/introduction-c071c5433e89">Source</a></figcaption></figure><h4>Introduction</h4><p>In the rapidly evolving world of artificial intelligence (AI), Vision-Language Models (VLMs) are making waves by combining the power of visual and textual data. Imagine a model that can not only understand images but also describe them in natural language, answer questions about them, or even create new images from textual descriptions. VLMs are powerful AI systems that bridge the gap between computer vision and natural language processing. These models can understand and generate content that combines both visual and textual information.</p><p>VLMs extend the capabilities of Large Language Models (LLMs) like Llama and ChatGPT to the visual domain, addressing the challenge of integrating discrete, sequential language with continuous, high-dimensional visual data. This integration is crucial for applications such as visual assistants and generative models, which require sophisticated techniques to fuse visual and textual information accurately. Despite challenges like understanding spatial relationships and mitigating biases, VLMs are poised to revolutionize our interaction with technology, offering more intuitive and versatile AI systems. Sounds fascinating, right? But what exactly are VLMs? Let’s dive into what they are, how they work, and why they’re so revolutionary.</p><h4>Why Are VLMs Important?</h4><p>Vision-language models are revolutionizing how AI interacts with multimodal data. They enable more natural and intuitive ways for humans to interact with AI systems, paving the way for applications like advanced image search engines, automated image captioning for accessibility, visual question answering systems, and AI-powered design tools. As these models continue to evolve, they’re opening up new possibilities for how we can use AI to understand and interact with the visual world around us.</p><p>In conclusion, vision-language models represent a significant step forward in AI’s ability to process and understand multimodal information. By bridging the gap between vision and language, these models are helping to create more versatile and powerful AI systems that can interact with the world in increasingly human-like ways.</p><h4>Fundamentals of VLMs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*d46DxD2wzEjTamuAnLuYBQ.png" /><figcaption>Basic Building Blocks in VLMs</figcaption></figure><p>Let’s break down the core components and processes of VLMs:</p><p><strong>Visual Encoder</strong>: This component processes visual data. Using techniques like Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs), the visual encoder converts images or video frames into a numerical format that the model can understand. For instance, CNNs can detect patterns such as edges and textures in an image, while ViTs can process the image in patches to capture more complex features.</p><p><strong>Language Encoder</strong>: This component processes textual data. It uses Transformers, similar to those in Large Language Models (LLMs) like GPT-3, to convert text into a numerical format compatible with visual data. Transformers excel at understanding the context and relationships within text, making them ideal for this task.</p><p><strong>Multimodal Fusion</strong>: Integrates visual and textual embeddings into a unified representation that the model can use to understand and generate multimodal content. This can be done through various approaches:</p><ol><li><strong>Early Fusion</strong>: Combining visual and textual data at the initial stages of processing, allowing the model to learn joint representations.</li><li><strong>Late Fusion</strong>: Processing visual and textual data separately and merging their outputs at a later stage, which can be simpler to implement.</li><li><strong>Cross-Attention</strong>: Allowing visual and textual data to interact at multiple points during processing, which helps the model capture intricate relationships between the two modalities.</li></ol><p><strong>Decoder</strong>: The decoder generates the output based on the fused visual and textual data. For example, it can produce a textual description of an image, answer questions about a visual scene, or even create an image from a textual prompt. Decoders often use techniques like autoregressive models, which generate data step-by-step, ensuring coherence and relevance.</p><h4>Training Paradigms of VLMs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*COyocjfkGfgp7zADV6JSuw.png" /><figcaption>Different Training Paradigms of VLMs Source: <a href="https://arxiv.org/pdf/2405.17247">An Introduction to Vision-Language Modeling</a></figcaption></figure><p>Training paradigms for VLMs refer to the different methodologies and frameworks used to train these models to integrate and understand both visual and textual data effectively. Each paradigm employs unique techniques to align, fuse, and utilize multimodal data to achieve specific objectives in VLM training. Here are the main types:</p><p><strong>Contrastive-based VLMs:</strong> Contrastive learning uses pairs of related (positive) and unrelated (negative) examples to train models. The goal is to align similar pairs closely and separate dissimilar ones. For example, CLIP (Contrastive Language-Image Pre-training) excels in zero-shot classification by matching images with their correct captions. For instance, pairing a dog picture with “a dog playing in the park” (positive) and not “a cat sitting on a couch” (negative).</p><p><strong>Masking-based VLMs:</strong> Inspired by BERT, masking-based VLMs learn by predicting missing parts of an input. For example, FLAVA uses masking for both text and image encoders, allowing the model to reconstruct missing parts. For instance, showing an image with hidden parts for the model to predict.</p><p><strong>Generative-based VLMs: </strong>Generative VLMs create new images from text or generate text from visual inputs. For example, CoCa combines contrastive and generative losses to improve multimodal understanding. For instance, creating an image of a sunset from the description “a beautiful sunset over the ocean.”</p><p><strong>Pretrained Backbone-based VLMs:</strong> These models leverage existing large language models and visual encoders to map visual features to text embeddings with minimal additional training. For example, Frozen and MiniGPT-4 use pre-trained LLMs and visual encoders to align visual and textual representations with minimal extra training. For instance, using a language model that understands text to also understand images with minimal extra training.</p><h4>Training Vision-Language Models (VLMs)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*fzBQ2gecF88EK_wfIyWsFw.png" /><figcaption>Showcasing Critical Steps involved in Training VLM Source: <a href="https://arxiv.org/pdf/2405.17247">An Introduction to Vision-Language Modeling</a></figcaption></figure><p>Training VLMs involves several critical steps:</p><p>Data is a cornerstone in training Vision-Language Models (VLMs). Ensuring a diverse and balanced dataset is vital for developing robust models capable of understanding a wide range of concepts. Here’s a detailed breakdown:</p><p><strong>Removing Duplicates: </strong>Eliminating duplicates in large-scale datasets saves compute time and reduces memorization risks, ensuring the model learns more efficiently.</p><p><strong>Balancing: </strong>A balanced dataset spans various concepts and categories, preventing the model from becoming biased towards certain types of images or text.</p><p><strong>Pruning: </strong>Pruning is the process of refining the dataset to maintain high-quality, relevant data. Data pruning can be categorized in three types,</p><ol><li><strong>Heuristics</strong>: Removing low-quality pairs to maintain data integrity.</li><li><strong>Bootstrapping</strong>: Using pre-trained VLMs to rank and discard poorly aligned image-text pairs based on their multimodal alignment.</li><li><strong>Diverse and Balanced Data</strong>: Ensuring the dataset covers a wide range of concepts equally.</li></ol><p><strong>Improving Data Quality:</strong> Things to consider while to improve data quality,</p><ol><li><strong>Caption Quality</strong>: Enhancing the accuracy and relevance of captions to better describe image content.</li><li><strong>Synthetic Data</strong>: Creating high-quality synthetic data, such as synthetic captions for images, to supplement real data and broaden the training set.</li><li><strong>Data Augmentation</strong>: Introducing variations to existing data to enhance model robustness and generalization capabilities.</li><li><strong>Interleaved Data Curation</strong>: Combining text and image data during training to improve the model’s ability to handle multimodal information.</li></ol><p>Regular assessment of multimodal data quality ensures high standards, and leveraging human expertise in data annotation provides high-quality labels that significantly improve model training.</p><p><strong>Grounding: </strong>Grounding is the process of accurately associating words with specific visual elements in an image, ensuring precise mapping of text to visuals. It is crucial for improving accuracy, maintaining contextual relevance, and building user trust by providing reliable and understandable results. Techniques to improve grounding include methods like using bounding boxes or negative captions.</p><p><strong>Improving Alignment in VLM:</strong> Alignment ensures that VLMs produce human-expected answers. It involves instruction-tuning, training models with specific instructions and outputs, and Reinforcement Learning from Human Feedback (RLHF), using human feedback to refine responses. This step is essential for enhancing the reliability and trustworthiness of VLMs.</p><h4>Comprehensive Guide to Implementing VLMs</h4><p>Efficient software tools, such as OpenCLIP and Hugging Face transformers, are essential for benchmarking and model comparison. Here’s a comprehensive overview:</p><p><strong>Choosing the right model:</strong></p><ol><li>Contrastive Models (e.g., CLIP) are ideal for associating text with visual concepts. They require large datasets and batch sizes, making them suitable for robust text-image matching.</li><li>Masking Models are best for reconstructing data from masked inputs, useful for smaller batches and efficient training.</li><li>Generative Models are ideal for creating photorealistic images from text but are computationally intensive.</li><li>Pretrained Backbones like Llama or GPT are cost-effective for mapping text and vision representations, reducing training costs by leveraging existing models.</li></ol><p><strong>Speeding Up Training:</strong> Strategies to speed up training involves various steps like Efficient Attention Mechanisms, Optimized Data Loading<strong> </strong>and Performance-Enhancing Libraries.</p><p><strong>Hardware Requirements: </strong>Training VLMs can be resource intensive. Large-scale models like CLIP may need over 500 GPUs, while optimized training with high-quality datasets can use around 64 GPUs. Leveraging pretrained encoders significantly reduces GPU needs and overall costs, making the training process more feasible for researchers with limited computational resources.</p><p><strong>Enhancing Text-Rich Image Understanding:</strong> This involves improving the model’s ability to accurately interpret and describe images with substantial textual content. This includes better data curation, advanced text recognition algorithms, and specialized models for handling text within images.</p><p><strong>Parameter-Efficient Fine-Tuning:</strong> Parameter-efficient fine-tuning optimizes models with minimal computational resources by adapting only a subset of parameters. Techniques like low-rank adaptation (LoRA) and adapters enable efficient fine-tuning of large models, reducing computational power needs while maintaining high performance.</p><h4>Evaluating Vision-Language Models (VLMs)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*RmrbuS68F0wHyXcEzu1zEg.png" /><figcaption>Methods to Evaluate VLMs Source: <a href="https://arxiv.org/pdf/2405.17247">An Introduction to Vision-Language Modeling</a></figcaption></figure><p>Evaluating VLMs involves several methods to ensure they understand and generate both visual and textual information correctly. Below are concise explanations of each evaluation method:</p><ol><li><strong>Image Captioning</strong>: Evaluates how well a model can generate textual descriptions for images using datasets and metrics to compare generated captions with reference captions.</li><li><strong>Visual Question Answering (VQA)</strong>: Tests the model’s ability to answer questions about images, assessing its understanding of the visual content.</li><li><strong>Text-to-Image Consistency</strong>: Measures how well a model can generate images that match text descriptions, ensuring alignment between generated images and text prompts.</li><li><strong>Zero-Shot Image Classification</strong>: Evaluates a model’s performance on classification tasks without explicit training for those tasks, using diverse datasets to test accuracy.</li><li><strong>Bias and Fairness</strong>: Ensures the model doesn’t show unfair biases across different demographic groups by analyzing classification rates and distributions related to various attributes.</li><li><strong>Hallucination Detection</strong>: Ensures the model doesn’t generate unrelated or incorrect content by detecting object hallucinations in captions.</li><li><strong>Memorization Check</strong>: Evaluates if the model memorizes training data instead of generalizing, using tests to check for the recall of specific objects from training images.</li></ol><p>By leveraging these methods, researchers can determine if VLMs accurately and fairly understand and generate visual and textual information, ensuring their reliability and robustness.</p><p><strong>Red Teaming:</strong> Red teaming involves adversarial testing to identify and mitigate risks in VLMs, such as biases and privacy issues. By simulating attacks, researchers can enhance the robustness and ethical standards of VLMs, ensuring they are safe and reliable in real-world applications.</p><h4>Applications of Vision-Language Models</h4><p>VLMs have a wide range of applications that can significantly impact our daily lives:</p><ol><li><strong>Visual Assistants</strong>: Helping users navigate environments by interpreting visual inputs and providing textual guidance.</li><li><strong>Content Creation</strong>: Generating images from textual descriptions, useful in fields like art, design, and advertising.</li><li><strong>Accessibility</strong>: Assisting visually impaired individuals by providing descriptions of visual content.</li><li><strong>Image Captioning</strong>: Automatically generating descriptive captions for images.</li><li><strong>Visual Question Answering (VQA)</strong>: Answering questions about the content of an image or video.</li></ol><h4>Challenges and Future Directions</h4><p>Despite their potential, VLMs face several challenges:</p><ol><li><strong>High Dimensionality</strong>: Visual data exists in a high-dimensional space, making it complex to process and interpret compared to text. Effective dimensionality reduction techniques are crucial for improving model performance.</li><li><strong>Data Annotation</strong>: Creating large, annotated datasets for training VLMs is time-consuming and costly. Crowdsourcing and automated annotation tools can help mitigate this issue.</li><li><strong>Reliability and Bias</strong>: Ensuring VLMs are reliable and free from biases present in the training data is crucial for their effective deployment. Techniques like bias detection and mitigation are essential areas of ongoing research.</li></ol><p>Future research aims to address these challenges by improving the accuracy and reliability of VLMs and extending their capabilities to handle more complex tasks, such as video analysis and real-time visual data processing. Innovations like self-supervised learning, where models learn from raw, unannotated data, could play a significant role in overcoming current limitations.</p><h4>Conclusion</h4><p>Vision-Language Models represent a significant advancement in artificial intelligence, merging the capabilities of visual and textual understanding. By bridging these two domains, VLMs enable more intuitive and versatile interactions between humans and machines, opening up new possibilities for innovation across various fields. As research continues to evolve, VLMs are poised to become an increasingly integral part of our technological landscape, transforming how we interact with and interpret the world around us.</p><p>Imagine a future where AI can not only see the world around us but also understand and describe it in detail. This is the promise of Vision-Language Models, a technology poised to transform how we interact with and leverage AI in our everyday lives. Whether enhancing accessibility, revolutionizing content creation, or providing intelligent visual assistance, VLMs are set to play a pivotal role in the AI-powered future.</p><p>By lowering the barrier to entry into VLM research, we hope to provide the foundations for more responsible development of VLMs while pushing the boundaries of vision understanding. The future is bright, and VLMs are at the forefront of this exciting journey.</p><h4><strong>References</strong></h4><p>To further understand the concepts discussed, I recommend exploring the following resources:</p><ul><li>Official Paper: <a href="https://arxiv.org/pdf/2405.17247">An Introduction to Vision-Language Modeling</a></li></ul><p>This article showcases the versatility and effectiveness of vision-language models in multimodal tasks. For deeper insights into its implementation, comparisons with similar models, and potential applications, refer to the official paper. Whether you are an academic researcher, industry practitioner or an AI enthusiast vision-language models offer a robust tool for innovative developments in AI and Computer Vision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3bdf2a3e728" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[All You Need to Know About Florence-2!]]></title>
            <link>https://medium.com/@sunidhi.ashtekar/all-you-need-to-know-about-florence-2-348e23f75768?source=rss-1d599ba5668e------2</link>
            <guid isPermaLink="false">https://medium.com/p/348e23f75768</guid>
            <category><![CDATA[multimodal]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[segmentation]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sunidhi Ashtekar]]></dc:creator>
            <pubDate>Tue, 25 Jun 2024 20:40:42 GMT</pubDate>
            <atom:updated>2024-06-25T20:41:20.315Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-24K1twkGVMIegUUDa-Tsw.png" /><figcaption>Image from Florence-2 <a href="https://github.com/retkowsky/florence-2/tree/main">Official Repository</a></figcaption></figure><h4><strong>Introduction</strong></h4><p>Florence-2 is a lightweight vision-language foundation model developed by Microsoft Azure AI and open-sourced under the MIT license. It aims to achieve a unified, prompt-based representation for diverse vision and vision-language tasks, including captioning, object detection, grounding, and segmentation. Despite its compact size, Florence-2 rivals much larger models like Kosmos-2 in performance. Its strength lies in the extensive FLD-5B dataset, which consists of 126 million images and 5.4 billion annotations, enabling robust zero-shot and fine-tuning capabilities.</p><h4>Model Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*r1lX0TdZDa3W2PXvGpSEjQ.png" /><figcaption>Image from Florence-2 Official Paper</figcaption></figure><ol><li><strong>Image Encoder DaViT (Diverse Vision Transformer):</strong></li></ol><p>Florence-2 utilizes the DaViT architecture as its image encoder, transforming input images into flattened visual token embeddings.</p><p>2<strong>. Multi-Modality Encoder-Decoder:</strong></p><p>The model employs a seq2seq framework, seamlessly integrating the image encoder with a multi-modality encoder-decoder.</p><p>Tasks are handled as translation problems, where an image and a task-specific prompt are provided as input, resulting in a textual response.</p><p>For purely textual tasks, the input prompt and output are in text form.</p><p>3.<strong> Region-Specific Tasks</strong>:</p><p>For tasks involving specific regions, location. tokens representing coordinates are added to the tokenizer’s vocabulary.</p><ul><li><strong>Box Representation (x0, y0, x1, y1)</strong>: Location tokens correspond to the top-left and bottom-right corners of a box.</li><li><strong>Polygon Representation (x0, y0, …, xn, yn)</strong>: Location tokens represent the vertices of a polygon in clockwise order.</li></ul><p>The model takes images and task prompts as input, generating the desired results in text format. It uses a DaViT vision encoder to convert images into visual token embeddings. These embeddings are then concatenated with BERT-generated text embeddings and processed by a transformer-based multi-modal encoder-decoder to generate the response.</p><h4>Dataset FLD-5B</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*94tFPy_-eirxtwtKLslpAA.png" /><figcaption>Image from Florence-2 Official Paper</figcaption></figure><p>Florence-2’s exceptional performance is rooted in its training on the FLD-5B dataset. This dataset was meticulously created using an iterative strategy involving automated image annotation and model refinement, ensuring both scale and quality. The lack of large, unified datasets necessitated the creation of FLD-5B, as existing datasets like SA-1B and COCO are either limited in scope or size. Manual labelling being prohibitively expensive, the authors opted for automation using specialized models.</p><ol><li><strong>Automated Annotation</strong>: The dataset was built using a strategy that leverages automated image annotation and iterative model refinement. This approach ensures that the dataset is comprehensive and consistently updated with high-quality annotations.</li><li><strong>Annotation Types</strong>: FLD-5B includes a variety of annotations listed as follows,</li></ol><ul><li><strong>Image-Level Annotations</strong>: Providing overall image descriptions and categorizations.</li><li><strong>Region-Level Annotations</strong>: Highlighting specific regions within images, such as objects or areas of interest.</li><li><strong>Pixel-Level Annotations</strong>: Offering detailed information at the pixel level, essential for tasks like segmentation.</li><li><strong>Granularity</strong>: The dataset contains boxes, masks, and various levels of captions, providing rich and detailed annotations.</li></ul><p><strong>3. Data Engine</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x1A6JWDTG83cio8Zw2-0YQ.png" /><figcaption>Image from Florence-2 Official Paper</figcaption></figure><p>The creation of FLD-5B involved several stages listed below,</p><ul><li><strong>Initial Annotation</strong>: Images are collected from various sources and annotated using specialist models such as the Azure OCR API for text recognition, a Caption Model for generating image captions, a Grounding Model for linking phrases to image regions, an Object/Proposal Detector for identifying regions of interest, and a Segmentation Model for segmenting images into different regions.</li><li><strong>Data Filtering</strong>: The initial annotations undergo a rigorous filtering process to correct errors and remove irrelevant data. Key components include LMM Annotator and LLM Aggregator for text data, a Text Filter for enhancing textual annotations, and Region Filtering using techniques like Region Score, Non-Maximum Suppression (NMS), and Blacklist filtering. Florence-1 is also used for additional filtering to ensure data quality.</li><li><strong>Iterative Refinement</strong>: The annotations are refined through multi-round re-training and fine-tuning of the Florence-2 model. The process begins with small-scale enhanced data and progresses to large-scale data, iteratively improving annotation accuracy and quality.</li><li><strong>Final Annotations</strong>: The resulting dataset, FLD-5B, includes 126 million images, 500 million text annotations, 1.3 billion region-text annotations, and 3.6 billion text-phrase-region annotations. Final annotations include brief, detailed, and more detailed captions, grounding, segmentation, OCR, object detection, region proposals, and dense captions.</li></ul><p>Notably, FLD-5B doesn’t introduce new images, all images are sourced from existing computer vision datasets. By combining these elements, FLD-5B provides a rich and diverse set of visual data that allows Florence-2 to learn effectively across various tasks, ensuring robust performance and versatility.</p><h4>Key Contributions</h4><ol><li><strong>Unified Representation</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*U_sjbP6u_RPf6lRYA_QSMA.png" /><figcaption>Image from Florence-2 Official Paper</figcaption></figure><p>Florence-2 employs a seq2seq framework to handle various vision tasks using a single model, simplifying the architecture and eliminating the need for task-specific modifications. Vision tasks differ in spatial hierarchy and semantic granularity; instance segmentation offers detailed object locations without semantic context, while image captioning captures relationships but not exact locations. By unifying these representations, Florence-2 effectively manages over 10 tasks with a single model, streamlining the process and enhancing versatility.</p><p><strong>2. Comprehensive Dataset</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*lBdlln5IkAF2ftmknr7LpQ.png" /><figcaption>Image from <em>Florence-2 CVPR 2024 poster</em></figcaption></figure><p>The extensive FLD-5B dataset, provides a rich foundation for the model. This dataset includes image-level, region-level, and pixel-level annotations, enabling Florence-2 to learn from a wide range of visual data and deliver robust performance across different tasks.</p><p><strong>3. Multitask Learning</strong></p><p>Florence-2’s architecture and training strategy allow it to handle diverse vision tasks effectively. The model demonstrates strong performance in both zero-shot and fine-tuned settings, showcasing its ability to adapt to various vision challenges without needing task-specific adjustments.</p><h4><strong>Vision Tasks Performed by Florence-2</strong></h4><p>Florence-2 is designed to handle a variety of vision and vision-language tasks through its unified, prompt-based representation. The key vision tasks performed by Florence-2 include:</p><ul><li><strong>Caption</strong>: Generating brief textual descriptions of images, capturing the essence of the scene.</li><li><strong>Detailed Caption</strong>: Producing more elaborate textual descriptions, providing richer information about the image.</li><li><strong>More Detailed Caption</strong>: Creating comprehensive textual descriptions that include extensive details about the image.</li><li><strong>Region Proposal</strong>: Identifying regions of interest within an image to focus on specific areas.</li><li><strong>Object Detection</strong>: Locating and identifying objects within an image, providing bounding boxes and labels for each detected object.</li><li><strong>Dense Region Caption</strong>: Generating textual descriptions for densely packed regions within an image.</li><li><strong>Phrase Grounding</strong>: Associating phrases in a text description with specific regions in an image, linking textual descriptions to visual elements.</li><li><strong>Referring Expression Comprehension</strong>: Identifying regions in an image that correspond to natural language expressions, making it adept at tasks that require fine-grained visual-textual alignment.</li><li><strong>Open Vocabulary Detection</strong>: Detecting objects in an image using a flexible and extensive vocabulary.</li><li><strong>Referring Segmentation</strong>: Segmenting regions in an image based on referring expressions, providing detailed object boundaries.</li><li><strong>Region to Text</strong>: Converting regions of an image into corresponding textual descriptions.</li><li><strong>Text Detection and Recognition</strong>: Detecting and recognizing text within an image, providing both text and region information.</li></ul><p>Florence-2’s ability to perform these tasks with a unified model architecture highlights its versatility and robustness, setting new standards in the field of computer vision and multimodal AI.</p><h4>Experiments and Results</h4><ol><li><strong>Zero-Shot Performance</strong></li></ol><ul><li><strong>Image Captioning</strong>: Florence-2 achieved a remarkable 135.6 CIDEr score on the COCO caption benchmark.</li><li><strong>Visual Grounding and Referring Expression Comprehension</strong>: The model set new state-of-the-art performance levels, outperforming existing models across various benchmarks in these tasks.</li></ul><p><strong>2. Fine-Tuning</strong></p><ul><li><strong>State-of-the-Art Results</strong>: Fine-tuning with public human-annotated data enabled Florence-2 to compete with larger specialist models, achieving new state-of-the-art results on benchmarks like RefCOCO, RefCOCO+, and RefCOCOg.</li><li><strong>Object Detection and Segmentation</strong>: The model showed improved performance on COCO object detection and ADE20K semantic segmentation tasks, surpassing both supervised and self-supervised models.</li></ul><p><strong>3. Efficiency and Scalability</strong></p><ul><li><strong>Training Efficiency</strong>: Pre-training on the FLD-5B dataset significantly enhanced training efficiency, allowing Florence-2 to achieve better results with fewer epochs compared to models pre-trained on ImageNet.</li><li><strong>Model Scaling</strong>: Florence-2 demonstrated that increasing the model’s capacity leads to improved zero-shot performance across various tasks, showcasing its scalability and robustness.</li><li><strong>Model Variants</strong>: Florence-2 is available in two variants: Florence-2-base with 0.23 billion parameters and Florence-2-large with 0.77 billion parameters. These smaller sizes enable deployment even on mobile devices.</li><li><strong>Comparison with Kosmos-2</strong>: Despite its compact size, Florence-2 outperforms Kosmos-2 across all benchmarks, achieving better zero-shot results even though Kosmos-2 has 1.6 billion parameters.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A5nFd786cHFonMcK5SezQQ.png" /><figcaption>Florence-2 Inference for Object Detection, Segmentation, Visual grounding tasks</figcaption></figure><h4><strong>Conclusion</strong></h4><p>I hope this guide has provided a comprehensive overview of the newly launched Florence-2 vision foundation model, highlighting its innovative features and improvements over previous iterations. Florence-2 demonstrates remarkable capabilities in visual recognition, image generation, and multimodal understanding, making it a robust and versatile tool for various AI applications.</p><p>Stay tuned for my next blog where I will dive into the fine-tuning of Florence-2, providing insights and techniques to further enhance its performance for specific tasks.</p><p>If you have any questions, recommendations, or critiques, please don’t hesitate to reach out on LinkedIn. I’m open to discussions and eager to hear your feedback or assist with any challenges you might encounter.</p><h4><strong>References</strong></h4><p>To further explore the concepts, I recommend visiting the following resources:</p><ul><li><a href="https://huggingface.co/collections/microsoft/florence-6669f44df0d87d9c3bfb76de">Florence-2 Official Repository</a>: Explore models and datasets related to Florence-2 on Hugging Face</li><li><a href="https://arxiv.org/pdf/2311.06242">Florence-2 Official Paper</a></li><li><a href="https://huggingface.co/spaces/gokaygokay/Florence-2">Florence-2 Online Demo</a></li><li><a href="https://djl.ai/examples/docs/object_detection.html">Images taken in this blog</a></li></ul><p>This article illustrates the potential of Florence-2’s cutting-edge architecture and innovations. To know more about implementation details and comparisons with other state-of-the-art models, go through the official paper. Whether for academic research, industry applications, or personal projects, Florence-2 offers a powerful and efficient solution for exploring and innovating within the vast domain of computer vision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=348e23f75768" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unveiling Depth Anything V2: A Breakthrough in Monocular Depth Estimation]]></title>
            <link>https://medium.com/@sunidhi.ashtekar/unveiling-depth-anything-v2-a-breakthrough-in-monocular-depth-estimation-4c3f00fd4961?source=rss-1d599ba5668e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c3f00fd4961</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[depth-estimation]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Sunidhi Ashtekar]]></dc:creator>
            <pubDate>Sun, 16 Jun 2024 23:46:52 GMT</pubDate>
            <atom:updated>2024-06-17T00:06:06.994Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nnTGPRSXK5hemjzJxCSPWg.png" /><figcaption>Depth Anything V2 On RF100 Dataset</figcaption></figure><p>In the realm of computer vision, monocular depth estimation (MDE) has emerged as a pivotal task. The latest advancement in this field comes from the innovative “Depth Anything V2” model, which promises unprecedented accuracy and efficiency. Let’s dive into the key aspects of this groundbreaking work.</p><h4>Introduction</h4><p>Depth estimation is essential for various applications like 3D reconstruction, navigation, and even AI-generated content. Additionally, depth estimation is crucial for augmented reality (AR) and virtual reality (VR) experiences, autonomous driving, robotics, obstacle detection, path planning, and surveillance systems. Traditional methods struggled with certain challenges, including dealing with complex scenes and achieving fine-grained details. Depth Anything V2 aims to overcome these limitations, providing robust and detailed depth predictions.</p><p>The paper introduces models of various scales:</p><ul><li><strong>Small:</strong> 25 million parameters</li><li><strong>Base:</strong> 335 million parameters</li><li><strong>Large:</strong> 891 million parameters</li><li><strong>Giant:</strong> 1.3 billion parameters</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*m1DFRiyg3--Kf1Wj0URAkg.png" /><figcaption>Image from Official Paper <a href="https://arxiv.org/pdf/2406.09414">Depth Anything V2</a></figcaption></figure><h4>What Makes Depth Anything V2 Special?</h4><p>Depth Anything V2 stands out due to its innovative approach:</p><ol><li><strong>Synthetic Data Utilization:</strong> Replacing all labeled real images with synthetic ones to enhance label precision and detail.</li><li><strong>Scalable Model Architecture:</strong> Offering models ranging from 25 million to 1.3 billion parameters to cater to different needs.</li><li><strong>Pseudo-Labeled Real Images:</strong> Bridging the gap between synthetic and real data using large-scale pseudo-labeled real images.</li><li><strong>Versatile Evaluation Benchmark:</strong> Introducing DA-2K, a new benchmark with diverse and precisely annotated scenes.</li></ol><h4>Improvements Over Depth Anything V1</h4><p>Depth Anything V2 introduces several significant improvements over V1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*roessNdT03J3p_oTIJG2vQ.png" /><figcaption>Image from Official Paper <a href="https://arxiv.org/pdf/2406.09414">Depth Anything V2</a></figcaption></figure><ol><li><strong>Data Quality</strong>: Replaces noisy labeled real images with precise synthetic images.</li><li><strong>Efficiency</strong>: Models are more efficient, with faster inference times and fewer parameters.</li><li><strong>Robustness</strong>: Produces robust depth predictions for complex scenes, including transparent and reflective surfaces.</li><li><strong>Generalization</strong>: Improved generalization capabilities due to training on large-scale pseudo-labeled real images.</li><li><strong>Model Scalability</strong>: Offers models of varying scales to cater to different application needs, from lightweight (25M parameters) to large-scale (1.3B parameters).Depth Anything V1</li></ol><ul><li><strong>Data:</strong> Used a mix of labeled real images and synthetic images, which introduced noise and inconsistencies.</li><li><strong>Model Efficiency:</strong> Slower inference speeds and larger models compared to V2.</li><li><strong>Accuracy:</strong> Good accuracy but struggled with complex scenes and fine-grained details.</li><li><strong>Robustness:</strong> Less robust to transparent objects and reflective surfaces.</li></ul><h4>Architecture Overview</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*2wv5UIGAH6OeyzAlY4UXZQ.png" /><figcaption>Image from Official Paper <a href="https://arxiv.org/pdf/2406.09414">Depth Anything V2</a></figcaption></figure><p>Depth Anything V2 is built on the DINOv2 architecture and consists of three key steps:</p><ol><li><strong>Teacher Model Training</strong>: A large-scale teacher model is trained exclusively on high-quality synthetic images.</li><li><strong>Pseudo-Labeling Real Images</strong>: The teacher model generates precise pseudo-depth labels for a large dataset of unlabeled real images.</li><li><strong>Student Model Training</strong>: Student models are trained on these pseudo-labeled real images to ensure robust generalization.</li></ol><p><strong>Key Components:</strong></p><ul><li><strong>Encoder</strong>: Utilizes different scales of DINOv2 (small, base, large, giant) to process input images.</li><li><strong>Depth Decoder</strong>: Based on DPT, it converts features from the encoder into depth maps.</li></ul><h4>Datasets</h4><p>Depth Anything V2 employs a mix of synthetic and real datasets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*jn7tX1k4LQkxrY9Vl94zvw.png" /><figcaption>Image from Official Paper <a href="https://arxiv.org/pdf/2406.09414">Depth Anything V2</a></figcaption></figure><ol><li><strong>Synthetic Datasets</strong>: These include BlendedMVS, Hypersim, IRS, TartanAir, and VKITTI 2, providing highly precise depth labels.</li><li><strong>Pseudo-Labeled Real Datasets</strong>: Includes large-scale datasets like BDD100K, Google Landmarks, ImageNet-21K, LSUN, Objects365, Open Images V7, Places365, and SA-1B.</li></ol><p>The combination of these datasets ensures a broad and diverse training set, enhancing the model’s ability to generalize across various real-world scenarios.</p><h4>Implementation Details</h4><p>The training process involves two main stages:</p><ol><li><strong>Training the Teacher Model</strong>: The teacher model, a DINOv2-G (giant) encoder, is trained on synthetic datasets to capture detailed and accurate depth information.</li><li><strong>Training Student Models</strong>: Smaller models (DINOv2 small, base, large) are trained on pseudo-labeled real images generated by the teacher model. This step involves the following specifics:</li></ol><ul><li><strong>Resolution</strong>: Training images are resized to 518x518 pixels.</li><li><strong>Optimization</strong>: Uses the Adam optimizer with specific learning rates for the encoder (5e-6) and decoder (5e-5).</li></ul><h4><strong>Loss Functions</strong></h4><p>To ensure sharp and precise depth predictions, Depth Anything V2 combines two primary loss functions:</p><ul><li><strong>Scale and Shift-Invariant Loss (Lssi):</strong> Ensures depth consistency regardless of scale and shift.</li><li><strong>Gradient Matching Loss (Lgm):</strong> Enhances sharpness and detail by matching gradients between predicted and ground truth depth maps.</li></ul><h4>Depth Estimation Process</h4><p>Depth estimation in Depth Anything V2 involves converting a 2D image into a depth map, representing the distance of each pixel from the camera. The process includes:</p><ol><li><strong>Image Encoding</strong>: The input image is passed through the DINOv2 encoder, extracting multi-scale features.</li><li><strong>Depth Decoding</strong>: The depth decoder (DPT) processes these features to generate a high-resolution depth map.</li><li><strong>Inference</strong>: During inference, the model predicts the depth map, which can be visualized or used in downstream applications like 3D reconstruction or navigation.</li></ol><h4><strong>Detailed Overview of Depth Estimation</strong></h4><p>Here’s a detailed overview of the process in Depth Anything V2:</p><ol><li><strong>Input and Preprocessing: </strong>The input image is preprocessed to standardize its size and format, including resizing and normalizing pixel values. This ensures consistent processing and enhances model robustness through data augmentation.</li><li><strong>Feature Extraction: </strong>The preprocessed image is passed through a feature extractor using the DINOv2 architecture. This feature extractor, a Vision Transformer (ViT) or Convolutional Neural Network (CNN), captures essential details such as edges, textures, and objects.</li><li><strong>Depth Prediction:</strong> Extracted features are fed into a Depth Decoder based on the Dense Prediction Transformer (DPT) architecture. The decoder predicts depth values for each pixel, creating a depth map where each pixel value represents the estimated distance from the camera.</li><li><strong>Loss Functions:</strong> During training, the predicted depth map is compared against ground truth using two primary loss functions.</li><li><strong>Post-Processing:</strong> Raw depth predictions may undergo post-processing, such as bilateral filtering, to refine the depth map, smoothing values while preserving edges.</li><li><strong>Inference: </strong>The model processes new images in real-time to generate depth maps efficiently. Depth Anything V2’s architecture ensures fast and accurate depth prediction suitable for various applications.</li><li><strong>Robustness and Generalization: </strong>Depth Anything V2 uses pseudo-labeled real images for robustness across scenarios. The teacher model, trained on synthetic images, generates pseudo-depth labels for unlabeled real images. Student models trained on these images generalize well to complex scenes, transparent objects, and reflective surfaces.</li></ol><p>Depth Anything V2 combines precision, efficiency, and robustness to handle a wide range of real-world scenarios, achieving state-of-the-art performance in monocular depth estimation.</p><h4><strong>Annotation Process</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*-Y0LMSaoTwQ4PtYy8q2prQ.png" /><figcaption>Image from Official Paper <a href="https://arxiv.org/pdf/2406.09414">Depth Anything V2</a></figcaption></figure><ul><li><strong>Automatic Selection:</strong> Uses SAM to predict object masks and select key points.</li><li><strong>Manual Selection:</strong> Ensures challenging pairs are included for precision.</li><li><strong>Triple-Checking:</strong> Multiple annotators verify annotations to maintain accuracy.</li></ul><h4>Getting Started</h4><p>To do inference on custom dataset these are the steps to follow,</p><ul><li>Environment Setup</li><li>Download Pre-trained Model Weights for Depth Anything V2</li><li>Install Dependencies</li><li>Download datasets from Roboflow</li><li>Visualization and Analysis</li></ul><h4>Setting Up Hugging Face Hub</h4><p>To use the Hugging Face Model Hub, first install the huggingface_hub library, then log into your Hugging Face account. This setup enables you to download, manage, and upload models and datasets seamlessly.</p><pre>!python -m pip install huggingface_hub<br>!huggingface-cli login</pre><h4>Cloning the Depth Anything V2 Repository</h4><pre>!git clone https://huggingface.co/spaces/depth-anything/Depth-Anything-V2</pre><h4>Mounting Google Drive</h4><p>Next, mount the drive to use the downloaded model. I have used images from Roboflow <a href="https://universe.roboflow.com/roboflow-100">RF-100</a> Construction Safety 2 dataset for inference.</p><pre>from google.colab import drive<br>drive.mount(&#39;/content/drive&#39;)</pre><h4>Installing Dependencies</h4><p>Navigate to the cloned Depth Anything V2 repository and install all necessary dependencies. Install the required Python packages listed in the requirements.txt file. Download the pre-trained model weights and save under checkpoints folder, if the folder doesn’t exist create one!</p><pre>!cd Depth-Anything-V2<br>!pip install -r /content/Depth-Anything-V2/requirements.txt<br><br>!mkdir -p checkpoints<br>!mv /content/drive/MyDrive/checkpoints/depth_anything_v2_vitl.pth checkpoints/depth_anything_v2_vitl.pth</pre><h4>Performing Depth Estimation with Depth Anything V2</h4><p>This code will perform depth estimation on the specified image and display the resulting depth map within the Colab notebook. Adjust the paths as needed to match your file locations</p><pre>import cv2<br>import sys<br>import os<br>import torch<br>from google.colab.patches import cv2_imshow<br><br>sys.path.append(os.path.abspath(&#39;/content/Depth-Anything-V2&#39;))<br><br>from depth_anything_v2.dpt import DepthAnythingV2<br><br># Initialize the model (using &#39;vitl&#39; encoder as an example)<br>model = DepthAnythingV2(encoder=&#39;vitl&#39;, features=256, out_channels=[256, 512, 1024, 1024])<br>model.load_state_dict(torch.load(&#39;checkpoints/depth_anything_v2_vitl.pth&#39;, map_location=&#39;cpu&#39;))<br>model.eval()<br><br># Read the image<br>raw_img = cv2.imread(&#39;/content/drive/MyDrive/input_imgs/demo01.jpg&#39;)<br><br># Perform inference<br>depth = model.infer_image(raw_img)  # HxW raw depth map<br><br># Optionally, visualize the depth map<br>depth_normalized = cv2.normalize(depth, None, 0, 255, cv2.NORM_MINMAX)<br>depth_colormap = cv2.applyColorMap(depth_normalized.astype(&#39;uint8&#39;), cv2.COLORMAP_JET)<br>cv2_imshow(depth_colormap)<br>cv2.waitKey(0)<br>cv2.destroyAllWindows()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1KMLVQtFflvFwVVrfrp3Ew.png" /></figure><h4>Running Depth Estimation on Images</h4><p>To run the Depth Anything V2 model on a directory of images use the provided script. The model will process all images in the specified input directory and save the depth maps to the designated output directory</p><pre>!python /content/Depth-Anything-V2/run.py --encoder vitl --img-path /content/drive/MyDrive/input_imgs --outdir /content/drive/MyDrive/result_imgs</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6kwgxtEN6pN6XXpaY3U2lw.png" /></figure><h4>Running Depth Estimation on Videos</h4><pre>!python /content/Depth-Anything-V2/run_video.py --encoder vitl --video-path /content/drive/MyDrive/input_videos --outdir /content/drive/MyDrive/result_vdos</pre><h4>Conclusion</h4><p>I hope this guide has provided a comprehensive overview of the newly launched Depth Anything V2 model, including its innovative features and the detailed process of monocular depth estimation. Depth Anything V2 leverages both synthetic and pseudo-labeled real datasets, and achieves robust, efficient, and accurate depth predictions. Additionally, we provided a step-by-step guide on how to perform inference on your images using this model.</p><p>If you have any questions, recommendations, or critiques, please don’t hesitate to reach out on LinkedIn. I’m open to discussions and eager to hear your feedback or assist with any challenges you might encounter.</p><h4>References</h4><p>To further explore the concepts, I recommend visiting the following resources:</p><ul><li><a href="https://depth-anything-v2.github.io/"><strong>Depth Anything V2 Official Repository Github</strong></a></li><li><a href="https://arxiv.org/pdf/2406.09414"><strong>Depth Anything V2 Official Paper</strong></a></li><li><a href="https://huggingface.co/spaces/depth-anything/Depth-Anything-V2/tree/main"><strong>Hugging Face Depth Anything V2 Repository</strong></a>: Explore models and datasets related to Depth Anything V2 on Hugging Face.</li><li><a href="https://huggingface.co/spaces/depth-anything/Depth-Anything-V2"><strong>Depth Anything V2 Online Demo</strong></a></li><li>For custom dataset visit <a href="https://universe.roboflow.com/roboflow-100"><strong>Roboflow 100</strong></a></li></ul><p>This article illustrates the potential of Depth Anything V2’s cutting-edge architecture and innovations. To know more about implementation details and comparison with other state-of-the-art models, go through the official paper. Whether for academic research, industry applications, or personal projects, Depth Anything V2 offers a powerful and efficient solution for exploring and innovating within the vast domain of computer vision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c3f00fd4961" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[YOLOv10: Revolutionizing Real-Time Object Detection]]></title>
            <link>https://medium.com/@sunidhi.ashtekar/yolov10-revolutionizing-real-time-object-detection-72ef04ad441a?source=rss-1d599ba5668e------2</link>
            <guid isPermaLink="false">https://medium.com/p/72ef04ad441a</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Sunidhi Ashtekar]]></dc:creator>
            <pubDate>Tue, 28 May 2024 17:59:44 GMT</pubDate>
            <atom:updated>2024-05-28T17:59:44.121Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S6IgH1wzD6nuxuVY6MeIVg.jpeg" /><figcaption>Image from Ultralytics Assets</figcaption></figure><p>In the fast-evolving world of computer vision, the YOLO (You Only Look Once) series has consistently set benchmarks for real-time object detection. The latest iteration, YOLOv10, promises to push these boundaries even further. In this blog post, we’ll explore the architecture of YOLOv10, highlight its key features, discuss how it outperforms its predecessors and train on custom dataset.</p><h4>Introduction</h4><p>YOLOv10 builds upon the strengths of its predecessors, addressing the limitations in post-processing and model architecture. The primary goal is to enhance both performance and efficiency, making YOLOv10 a state-of-the-art model for real-time end-to-end object detection.</p><p>YOLOv10 offers a range of model scales, ensuring that users can choose the best model according to their specific application requirements. This flexibility allows for better adaptation in various real-world scenarios, from lightweight models for mobile applications to larger models for high-performance tasks. The YOLOv10 model is available in six sizes:</p><ul><li>Nano (n): 2.3 million parameters</li><li>Small (s): 7.2 million parameters</li><li>Medium (m): 15.4 million parameters</li><li>Big (b): 19.1 million parameters</li><li>Large (l): 24.4 million parameters</li><li>Extra Large (x): 29.5 million parameters</li></ul><h4>Architecture Overview</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L1e4EOLIwKalyFi9Mgtubw.png" /><figcaption>YOLOv10 Model Diagram from official <a href="https://github.com/THU-MIG/yolov10?tab=readme-ov-file">YOLOv10</a> repository.</figcaption></figure><p><strong><em>Backbone:</em></strong></p><ul><li>Responsible for feature extraction, the backbone in YOLOv10 uses an enhanced version of CSPNet (Cross Stage Partial Network). This improvement facilitates better gradient flow and reduces computational redundancy, making the model more efficient.</li></ul><p><strong><em>Neck:</em></strong></p><ul><li>The neck component aggregates features from different scales and passes them to the head. It includes PAN (Path Aggregation Network) layers, which are designed for effective multiscale feature fusion, ensuring that features from various levels are combined effectively for better object detection.</li></ul><p><strong><em>Consistent Dual Assignments for NMS-free Training:</em></strong></p><ul><li>Traditional YOLO models rely on non-maximum suppression (NMS) for post-processing, which can increase inference latency. YOLOv10 adopts a consistent dual assignment strategy, eliminating the need for NMS during inference.</li><li>The model employs dual label assignments during training: one-to-many for rich supervision and one-to-one for efficient inference. This approach harmonizes the supervision, leading to better performance without additional inference overhead.</li></ul><h4>Holistic Efficiency Driven Design</h4><p>The holistic efficiency driven design includes the modifications mentioned below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*KVmLwn8BFHFITWNm49pHrw.png" /><figcaption>YOLOv10 Holistic Efficiency-Accuracy Driven Model Design from official <a href="https://arxiv.org/pdf/2405.14458">YOLOv10 paper</a>.</figcaption></figure><p><strong><em>NMS-Free Training:</em></strong></p><ul><li>Traditional object detection models often use non-maximum suppression (NMS) to remove duplicate bounding boxes. However, this can be computationally expensive, particularly with a large number of detected boxes. YOLOv10 addresses this by training the model to naturally avoid generating multiple bounding boxes for the same object.</li><li>This is achieved through consistent dual assignments, ensuring each object is assigned a single unique bounding box during both training and inference.</li><li>By eliminating the NMS post-processing step, YOLOv10 significantly reduces inference latency, computational cost and inference time. Also, improves accuracy by ensuring each object is represented by a single, high-quality bounding box.</li></ul><p><strong><em>Lightweight Classification Head:</em></strong></p><ul><li>The classification head is the final layer in a neural network that assigns labels to detected objects. YOLOv10 introduces lightweight classification heads designed to be efficient without compromising accuracy. This is achieved by reducing the number of parameters and operations in the classification head.</li><li>It reduces computational overhead by simplifying the classification head while maintaining performance.</li></ul><p><strong><em>Spatial-Channel Decoupled Downsampling:</em></strong></p><ul><li>Downsampling reduces the spatial dimensions of feature maps while increasing channel dimensions. The standard approach using 3x3 convolutions with a stride of two is computationally expensive.</li><li>YOLOv10 improves this by separating spatial and channel operations. Pointwise convolutions adjust the channel dimension without changing the spatial dimension, and depthwise convolutions reduce the spatial dimension while maintaining the adjusted channel dimension.</li><li>Decoupling spatial reduction and channel transformation, reduces computational cost and maximizes information retention.</li></ul><p><strong><em>Rank-Guided Block Design:</em></strong></p><ul><li>Previous YOLO models used the same basic building blocks across all stages, leading to inefficiencies. YOLOv10 introduces a rank-guided block design that identifies stages with higher redundancy which reduces complexity and replaces them with a more compact inverted block (CIB), which effectively removes unnecessary information and enhances efficiency.</li><li>It uses intrinsic rank analysis to optimize the complexity of different stages, improving efficiency without sacrificing accuracy.</li></ul><h4><strong><em>Holistic Accuracy Driven Design</em></strong></h4><p>To boost YOLOv10’s performance with minimal computational cost, two key features are introduced: large-kernel convolution and partial self-attention (PSA).</p><p><strong><em>Large-Kernel Convolutions:</em></strong></p><p>Large-kernel convolutions expand the receptive field, enhancing the model’s capability, especially for larger objects. However, using them universally can dilute small object details and increase latency. YOLOv10 addresses this by:</p><ul><li>Selective Application: Using 7×7 depthwise convolutions only in deep stages.</li><li>Structural Reparameterization: Adding an extra 3×3 depthwise convolution branch during training to aid optimization, merged without inference overhead.</li><li>Model Scale Adaptation: Employing large kernels primarily in smaller model scales to avoid diminishing returns as model size increases.</li></ul><p><strong><em>Partial Self-Attention (PSA)</em></strong></p><p>Self-attention excels at global feature modeling but is computationally intensive. YOLOv10 optimizes this with PSA:</p><ul><li>Feature Partitioning: Dividing channels after a 1×1 convolution.</li><li>Selective Processing: Applying self-attention to only one part of the split features.</li><li>Efficient Fusion: Recombining features with another 1×1 convolution.</li><li>Dimension Reduction: Reducing query and key dimensions to half the size of the value in MHSA.</li><li>Optimized Placement: Using PSA only after Stage 4, minimizing computational overhead.</li></ul><h4>Key Highlights and Improvements</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*VM3XmTv9U7M6L7fr7q6bhg.png" /><figcaption>YOLOv10 Performance from the official <a href="https://github.com/THU-MIG/yolov10">YOLOv10</a> repository.</figcaption></figure><p><strong><em>Enhanced Efficiency:</em></strong></p><ul><li>YOLOv10’s architectural improvements lead to substantial reductions in parameters, FLOPs (Floating Point Operations), and latency across various model scales (N/S/M/B/L/X). For instance, YOLOv10-S is 1.8 times faster than RT-DETR-R18 with a similar average precision (AP), and YOLOv10-B achieves 46% less latency and 25% fewer parameters compared to YOLOv9-C.</li></ul><p><strong><em>Superior Accuracy:</em></strong></p><ul><li>The model achieves state-of-the-art performance across different benchmarks, with significant improvements in AP compared to previous YOLO versions. YOLOv10-L outperforms YOLOv8-L by 0.3 AP with 1.8 times fewer parameters, demonstrating efficient parameter utilization.</li></ul><p><strong><em>Performance:</em></strong></p><ul><li>Speed: YOLOv10 processes images at remarkable speeds, achieving 2.0ms for preprocessing, 13.4ms for inference, and 1.3ms for postprocessing per image at a resolution of (1, 3, 384, 640). This swift processing time makes YOLOv10 highly suitable for applications requiring rapid and accurate object detection, all while maintaining high accuracy.</li><li>The smallest size processes each image in just 1 millisecond (1000fps), making it the ideal choice for real-time video processing on edge devices.</li><li>High speed on CPU!</li></ul><h4>Getting Started</h4><p>To train YOLOv10 on custom dataset these are the steps to follow,</p><ul><li>Environment Setup</li><li>Download Pre-trained Model Weights for YOLOv10</li><li>Install Supervision and Ultralytics packages</li><li>Download datasets from Roboflow and train the model</li><li>Visualization and Analysis</li><li>Get Detection Results</li></ul><h4>Environment Setup</h4><p>A Google account is required to access Google Colab, a free cloud service providing necessary computational resources for deep learning tasks, including access to GPUs and TPUs up to 16 GB.</p><h4>GPU Status Check</h4><p>Firstly we ensure the availability and readiness of the GPU for processing and running the YOLOv10 model.</p><pre>!nvidia-smi</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mO_S1r5cYV3jPnYGz16eZA.png" /></figure><h4>Mounting Google Drive</h4><p>Next, we need to navigate to folder where the dataset is stored if you have downloaded the dataset or else we can directly use Roboflow to load the dataset.</p><pre>from google.colab import drive<br>drive.mount(&#39;/content/drive&#39;)</pre><h4>Display Current Directory</h4><p>Store current working directory’s path in the HOME variable for reference.</p><pre>import os<br>HOME = os.getcwd()<br>print(HOME)</pre><h4>Install YOLOv10 and Supervision Dependencies</h4><p>To set up the environment for running the YOLOv10 model, we need to install the supervision library and clone the YOLOv10 repository. These installations are crucial for ensuring that all necessary tools and libraries are available for the model to function correctly.</p><pre>!pip install -q supervision<br>!pip install -q git+https://github.com/THU-MIG/yolov10.git</pre><h4>Download YOLOv10 Pre-trained Weights</h4><p>This step involves creating a directory for storing the model weights and downloading the pre-trained weights for various YOLOv10 model sizes. These weights are essential for initializing the model and can be used for both inference and fine-tuning.</p><pre>!mkdir -p {HOME}/weights<br>!wget -P {HOME}/weights -q https://github.com/jameslahm/yolov10/releases/download/v1.0/yolov10n.pt<br>!wget -P {HOME}/weights -q https://github.com/jameslahm/yolov10/releases/download/v1.0/yolov10s.pt<br>!wget -P {HOME}/weights -q https://github.com/jameslahm/yolov10/releases/download/v1.0/yolov10m.pt<br>!wget -P {HOME}/weights -q https://github.com/jameslahm/yolov10/releases/download/v1.0/yolov10b.pt<br>!wget -P {HOME}/weights -q https://github.com/jameslahm/yolov10/releases/download/v1.0/yolov10x.pt<br>!wget -P {HOME}/weights -q https://github.com/jameslahm/yolov10/releases/download/v1.0/yolov10l.pt<br>!ls -lh {HOME}/weights</pre><h4>Load YOLOv10 Model with Pre-trained Weights</h4><p>In this step, we load the YOLOv10 model using the pre-trained weights downloaded earlier. This allows us to leverage the pre-trained model for inference or further training. The ultralytics library is used to facilitate this process.</p><pre>from ultralytics import YOLOv10<br><br>model = YOLOv10(f&#39;{HOME}/weights/yolov10n.pt&#39;)</pre><h4>Perform Inference and Annotate Image with YOLOv10</h4><p>This cell demonstrates how to perform inference using the YOLOv10 model on an input image. The results are then processed to extract detections, which are used to annotate the image with bounding boxes and labels. The supervision library helps in handling the annotations and displaying the results.</p><pre>import cv2<br>import supervision as sv<br>from ultralytics import YOLOv10<br><br>model = YOLOv10(f&#39;{HOME}/weights/yolov10n.pt&#39;)<br>image = cv2.imread(f&#39;{HOME}/data/dog.jpeg&#39;)<br>results = model(image)[0]<br>detections = sv.Detections.from_ultralytics(results)<br><br>bounding_box_annotator = sv.BoundingBoxAnnotator()<br>label_annotator = sv.LabelAnnotator()<br><br>annotated_image = bounding_box_annotator.annotate(<br>    scene=image, detections=detections)<br>annotated_image = label_annotator.annotate(<br>    scene=annotated_image, detections=detections)<br><br>sv.plot_image(annotated_image)</pre><h4>Download and Prepare Dataset with Roboflow</h4><p>This cell sets up the environment for dataset preparation. It creates a directory for datasets, installs the Roboflow library, and uses it to download a specific dataset for YOLOv10. The dataset will be used for training and evaluation purposes.</p><p>The project utilizes the <a href="https://blog.roboflow.com/roboflow-100/">RF100</a> <a href="https://universe.roboflow.com/roboflow-100/vehicles-q0x2v">Vehicles Road-Traffic-4</a> dataset from Roboflow.</p><pre>!mkdir {HOME}/datasets<br>%cd {HOME}/datasets<br><br>!pip install -q roboflow<br><br>from google.colab import userdata<br>from roboflow import Roboflow<br><br><br>rf = Roboflow(api_key=&quot;YOUR API KEY&quot;)<br>project = rf.workspace(&quot;roboflow-100&quot;).project(&quot;road-traffic&quot;)<br>version = project.version(4)<br>dataset = version.download(&quot;yolov8&quot;)</pre><h4>Train YOLOv10 Model on Custom Dataset</h4><p>This cell initiates the training of the YOLOv10 model using the custom dataset downloaded from Roboflow. The model is trained for a specified number of epochs with a given batch size.</p><pre>%cd {HOME}<br><br>!yolo task=detect mode=train epochs=25 batch=32 plots=True \<br>model={HOME}/weights/yolov10n.pt \<br>data = /content/datasets/road-traffic-4/data.yaml</pre><h4>Display Confusion Matrix of YOLOv10 Training</h4><p>After training the YOLOv10 model, it’s important to evaluate its performance. This cell displays the confusion matrix generated during training. The confusion matrix helps in visualizing the performance of the model by showing the accuracy of its predictions.</p><pre>%cd {HOME}<br>Image(filename=f&#39;{HOME}/runs/detect/train/confusion_matrix_normalized.png&#39;, width=600)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7VEpjNvy1wyiXmpNaZlwLQ.png" /></figure><h4>Display YOLOv10 Training Results</h4><p>This cell displays the training results image generated by the YOLOv10 training process. This image provides a summary of the model’s performance metrics and training progress, helping to visualize the effectiveness of the training.</p><pre>%cd {HOME}<br>Image(filename=f&#39;{HOME}/runs/detect/train10/results.png&#39;, width=600)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZwOfqR0rnsMCkbBqYHO2OQ.png" /></figure><h4>Display Validation Predictions from YOLOv10 Training</h4><p>This cell displays an image from the validation set, this helps in visualizing how well the YOLOv10 model is performing on the validation data.</p><pre>%cd {HOME}<br>Image(filename=f&#39;{HOME}/runs/detect/train10/val_batch0_pred.jpg&#39;, width=600)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-GQ1-BMsRCogBZV-nhFsNw.jpeg" /></figure><h4>Conclusion</h4><p>I hope this guide has provided a comprehensive overview of the YOLOv10 architecture, its innovative features, and the process of training it on a custom dataset. By leveraging the road traffic dataset from Roboflow, we demonstrated how to utilize YOLOv10 for advanced object detection tasks. This article aimed to highlight the practical applications and potential of YOLOv10 in real-world scenarios.</p><p>If you have any questions, recommendations, or critiques, please don’t hesitate to reach out on LinkedIn. I’m open to discussions and eager to hear your feedback or assist with any challenges you might encounter.</p><h4>References</h4><p>To further explore the concepts, I recommend visiting the following resources:</p><ul><li><a href="https://github.com/THU-MIG/yolov10">YOLOv10 Official Repository</a></li><li><a href="https://arxiv.org/pdf/2405.14458">YOLOv10 Official Paper</a></li><li><a href="https://docs.ultralytics.com/models/yolov10/#architecture">Ultralytics YOLO Documentation</a>: Detailed documentation and guides on using the YOLOv10 model provided by Ultralytics. Ultralytics YOLO Documentation</li><li><a href="https://blog.roboflow.com/yolov10-how-to-train/">Roboflow Blog</a>: A great resource for the latest updates and tutorials on using Roboflow for computer vision projects. Check out their blog for insightful articles.</li></ul><p>This article illustrates the potential of YOLOv10’s cutting-edge architecture and innovations. To know more about implementation details and comparison with other state-of-the-art models go through the official paper. Whether for academic research, industry applications, or personal projects, YOLOv10 offers a powerful and efficient solution for exploring and innovating within the vast domain of computer vision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72ef04ad441a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Brain MRI Segmentation with SA-UNet]]></title>
            <link>https://medium.com/@sunidhi.ashtekar/brain-mri-segmentation-with-sa-unet-8b6cee284dd0?source=rss-1d599ba5668e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b6cee284dd0</guid>
            <category><![CDATA[segmentation]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sunidhi Ashtekar]]></dc:creator>
            <pubDate>Sun, 17 Mar 2024 18:57:07 GMT</pubDate>
            <atom:updated>2024-03-17T18:57:07.666Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*COHL64Yki7hAHZ18jCuEOg.png" /><figcaption>Image from <a href="https://www.nytimes.com/2022/03/16/science/brain-imaging-research.html">Article</a></figcaption></figure><p>This article delves into the project that utilizes the Spatial Attention U-Net (<a href="https://arxiv.org/abs/2004.03696">SA-UNet</a>) for brain MRI segmentation, aiming to improve medical diagnosis accuracy with the<a href="https://www.kaggle.com/datasets/mateuszbuda/lgg-mri-segmentation"> LGG-MRI-Segmentation</a> dataset from <a href="https://www.kaggle.com/">Kaggle</a>.</p><p>Brain MRI (Magnetic Resonance Imaging) uses detailed imaging to explore the brain’s anatomy and potential issues, providing insights critical for diagnosis and treatment. Segmentation, crucial in medical imaging, divides images to analyze regions of interest, aiding in identifying abnormalities. There are two main types: semantic segmentation, which classifies each pixel without distinguishing objects of the same category, and instance segmentation, which differentiates between individual objects. These methods are vital for detailed analyses, such as identifying tumors in brain MRI scans.</p><p><strong>Getting Started</strong></p><p>For segmenting the brain MRI images, we have to follow these structured steps:</p><ul><li>Get the Dataset</li><li>Environment setup</li><li>Verify and Understand the Dataset</li><li>Data Pre-processing</li><li>Define Loss functions and Metrics</li><li>Define Model</li><li>Create Data Generator</li><li>Model Training</li><li>Model Evaluation and Inference</li></ul><p><strong>About the dataset</strong></p><p>The “LGG-MRI-Segmentation” dataset, sourced from The Cancer Imaging Archive and part of The Cancer Genome Atlas, includes MRI images and genomic data from 110 patients with lower-grade gliomas, featuring FLAIR sequences.</p><p>Accompanied by detailed genomic and clinical information, this dataset enriches segmentation research by offering a comprehensive view of each patient’s medical profile. It’s crucial for developing precise segmentation models, enabling the exploration of the nexus between imaging and genomics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*vDCDzLU1H0LLFyPiyotmUw.png" /></figure><p><strong>Environment Setup</strong></p><p>A Google account is required to access Google Colab, a free cloud service providing necessary computational resources. Then mount the Google drive to utilize the dataset stored followed by installing and importing the necessary Python packages.</p><pre>from google.colab import drive<br>drive.mount(&#39;/content/drive&#39;)<br><br>import os<br><br>import numpy as np<br>import pandas as pd<br>import cv2<br>import seaborn as sns<br>from matplotlib import pyplot as plt<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import confusion_matrix<br><br>import tensorflow as tf<br>import tensorflow.keras.backend as K<br>from tensorflow.keras.models import Model, load_model, save_model<br>from tensorflow.keras.layers import Input, Conv2D, Conv2DTranspose, MaxPooling2D, AveragePooling2D, UpSampling2D, BatchNormalization, Activation, Dropout, Concatenate, Multiply, Lambda, concatenate<br>from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint<br>from tensorflow.keras.optimizers.legacy import Adam<br>from tensorflow.keras.preprocessing.image import ImageDataGenerator</pre><p><strong>Understanding the Dataset</strong></p><p>We’ll explore the data.csv from our brain MRI dataset, crucial for understanding the images and labels, aiding in effective dataset preparation and analysis.</p><pre># Load the data.csv to understand the metadata<br>data_csv_path = &#39;/content/drive/My Drive/archive/lgg-mri-segmentation/kaggle_3m/data.csv&#39;<br>data_df = pd.read_csv(data_csv_path)<br><br># Display the first few rows of the dataframe<br>print(data_df.head())</pre><p><strong>Data Visualization</strong></p><p>We then visualize MRI images and their masks to understand mask application on the brain MRIs, crucial for assessing data quality and verifying our model’s segmentation accuracy.</p><pre>def load_images_with_masks(base_dir, patient_id, num_samples=50):<br>    patient_path = os.path.join(base_dir, patient_id)<br>    image_files = [f for f in os.listdir(patient_path) if not f.endswith(&#39;_mask.tif&#39;)]<br>    image_files = sorted(image_files)[:num_samples]  # Limit the number of samples for visualization<br><br>    images = []<br>    masks = []<br><br>    for img_file in image_files:<br>        img_path = os.path.join(patient_path, img_file)<br>        mask_path = os.path.join(patient_path, img_file.replace(&#39;.tif&#39;, &#39;_mask.tif&#39;))<br><br>        img = cv2.imread(img_path, cv2.IMREAD_UNCHANGED)<br>        mask = cv2.imread(mask_path, cv2.IMREAD_UNCHANGED)<br><br>        if img is not None and mask is not None:<br>            images.append(img)<br>            masks.append(mask)<br><br>    return images, masks<br><br>def visualize_images_and_masks(images, masks):<br>    plt.figure(figsize=(10, 5 * len(images)))<br>    for i, (image, mask) in enumerate(zip(images, masks)):<br>        plt.subplot(len(images), 2, 2*i + 1)<br>        plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))<br>        plt.title(&#39;Image&#39;)<br>        plt.axis(&#39;off&#39;)<br><br>        plt.subplot(len(images), 2, 2*i + 2)<br>        plt.imshow(mask, cmap=&#39;gray&#39;)<br>        plt.title(&#39;Original_Mask&#39;)<br>        plt.axis(&#39;off&#39;)<br>    plt.tight_layout()<br>    plt.show()<br><br>base_dir = &#39;/content/drive/My Drive/archive/lgg-mri-segmentation/kaggle_3m/&#39;<br>patient_id = &#39;TCGA_FG_A60K_20040224&#39;<br><br>images, masks = load_images_with_masks(base_dir, patient_id)<br>visualize_images_and_masks(images, masks)<br><br>def count_total_images_and_masks(base_dir):<br>    total_images = 0<br>    total_masks = 0<br><br>    for root, dirs, files in os.walk(base_dir):<br>        for file in files:<br>            if file.endswith(&#39;.tif&#39;) and &#39;_mask&#39; not in file:<br>                total_images += 1<br>            elif &#39;_mask.tif&#39; in file:<br>                total_masks += 1<br><br>    return total_images, total_masks<br><br>base_dir = &#39;/content/drive/My Drive/archive/lgg-mri-segmentation/kaggle_3m/&#39;<br>total_images, total_masks = count_total_images_and_masks(base_dir)<br>print(f&quot;Total number of images: {total_images}&quot;)<br>print(f&quot;Total number of masks: {total_masks}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*IciUJeUiqbnGayeblE8YsQ.png" /></figure><p><strong>Data Preprocessing</strong></p><p>This step involves dividing our dataset into three distinct sets: training (85%), validation (10%), and test (5%).</p><pre>def get_image_mask_pairs(base_dir):<br>    all_images = []<br>    all_masks = []<br><br>    for root, dirs, files in os.walk(base_dir):<br>        for file in sorted(files):<br>            if file.endswith(&#39;.tif&#39;) and not file.endswith(&#39;_mask.tif&#39;):<br>                image_path = os.path.join(root, file)<br>                mask_path = os.path.join(root, file.replace(&#39;.tif&#39;, &#39;_mask.tif&#39;))<br><br>                if os.path.exists(mask_path):  # Ensure the mask exists<br>                    all_images.append(image_path)<br>                    all_masks.append(mask_path)<br><br>    return all_images, all_masks<br><br>base_dir = &#39;/content/drive/My Drive/archive/lgg-mri-segmentation/kaggle_3m/&#39;<br>all_images, all_masks = get_image_mask_pairs(base_dir)<br><br># Initial split: 85% for training, 15% for temp (to be split further into validation and test)<br>train_images, temp_images, train_masks, temp_masks = train_test_split(all_images, all_masks, test_size=0.15, random_state=42)<br><br># Split the temp data into validation and test sets<br># Since temp is 15% of the whole, validation is 10% of the whole (about 2/3 of temp), and test is 5% of the whole (about 1/3 of temp)<br>validation_images, test_images, validation_masks, test_masks = train_test_split(temp_images, temp_masks, test_size=1/3, random_state=42)<br><br>print(f&quot;Training set: {len(train_images)} images&quot;)<br>print(f&quot;Validation set: {len(validation_images)} images&quot;)<br>print(f&quot;Test set: {len(test_images)} images&quot;)<br><br># Convert lists into DataFrames<br>df_train = pd.DataFrame({&#39;image_path&#39;: train_images, &#39;mask_path&#39;: train_masks})<br>df_validation = pd.DataFrame({&#39;image_path&#39;: validation_images, &#39;mask_path&#39;: validation_masks})<br>df_test = pd.DataFrame({&#39;image_path&#39;: test_images, &#39;mask_path&#39;: test_masks})</pre><p><strong>Setting Up the Training Data Generator</strong></p><p>The generator enhances training through real-time data augmentation, like rotations and flips, boosting model robustness by mimicking various MRI conditions. It also manages efficient batch processing, essential for handling large datasets without memory overload.</p><pre>def train_generator(<br>    data_frame,<br>    batch_size,<br>    augmentation_dict,<br>    image_color_mode=&quot;rgb&quot;,<br>    mask_color_mode=&quot;grayscale&quot;,<br>    image_save_prefix=&quot;image&quot;,<br>    mask_save_prefix=&quot;mask&quot;,<br>    save_to_dir=None,<br>    target_size=(256, 256),<br>    seed=1,<br>):<br>    image_datagen = ImageDataGenerator(**augmentation_dict)<br>    mask_datagen = ImageDataGenerator(**augmentation_dict)<br><br>    # Create generators for images and masks<br>    image_generator = image_datagen.flow_from_dataframe(<br>        data_frame,<br>        x_col=&quot;image_path&quot;,<br>        class_mode=None,<br>        color_mode=image_color_mode,<br>        target_size=target_size,<br>        batch_size=batch_size,<br>        save_to_dir=save_to_dir,<br>        save_prefix=image_save_prefix,<br>        seed=seed,<br>    )<br><br>    mask_generator = mask_datagen.flow_from_dataframe(<br>        data_frame,<br>        x_col=&quot;mask_path&quot;,<br>        class_mode=None,<br>        color_mode=mask_color_mode,<br>        target_size=target_size,<br>        batch_size=batch_size,<br>        save_to_dir=save_to_dir,<br>        save_prefix=mask_save_prefix,<br>        seed=seed,<br>    )<br><br>    train_gen = zip(image_generator, mask_generator)<br><br>    for (img, mask) in train_gen:<br>        img, mask = normalize_data(img, mask)<br>        yield (img, mask)<br></pre><p><strong>Data Normalization</strong></p><p>Normalization scales pixel values to a standard range, typically 0 to 1, crucial for stabilizing training and enhancing model convergence in deep learning.</p><pre>def normalize_data(img, mask):<br>    img = img / 255.0<br>    mask = mask / 255.0<br>    mask[mask &gt; 0.5] = 1<br>    mask[mask &lt;= 0.5] = 0<br>    return (img, mask)</pre><p><strong>Loss Function and Evaluation Metrics</strong></p><p>We define the loss function and evaluation metrics for our model’s training and evaluation. The Dice Coefficient Loss, based on the Dice coefficient, effectively measures overlap for segmentation tasks. Our metrics include accuracy, Intersection over Union (IoU), and the Dice coefficient.</p><pre>def dice_coefficients(y_true, y_pred, smooth=100):<br>    y_true_flatten = K.flatten(y_true)<br>    y_pred_flatten = K.flatten(y_pred)<br><br>    intersection = K.sum(y_true_flatten * y_pred_flatten)<br>    union = K.sum(y_true_flatten) + K.sum(y_pred_flatten)<br>    return (2 * intersection + smooth) / (union + smooth)<br><br>def dice_coefficients_loss(y_true, y_pred, smooth=100):<br>    return -dice_coefficients(y_true, y_pred, smooth)<br><br>def iou(y_true, y_pred, smooth=100):<br>    intersection = K.sum(y_true * y_pred)<br>    sum = K.sum(y_true + y_pred)<br>    iou = (intersection + smooth) / (sum - intersection + smooth)<br>    return iou<br><br>def jaccard_distance(y_true, y_pred):<br>    y_true_flatten = K.flatten(y_true)<br>    y_pred_flatten = K.flatten(y_pred)<br>    return -iou(y_true_flatten, y_pred_flatten)</pre><p><strong>Defining Encoder and Decoder Blocks for the SA-UNet Model</strong></p><p><strong>Encoder Blocks:</strong> Each block comprises convolutional layers followed by dropout, batch normalization, ReLu and max pooling for downsampling.</p><p><strong>Spatial Attention block:</strong> It applies both max pooling and average pooling across the channel dimension to capture different aspects of the spatial features. The outputs of these pooling operations are then concatenated and processed through a convolutional layer, creating a spatial attention map. This attention map, after being normalized by a sigmoid activation, is multiplied element-wise with the encoder output, refining the feature map that is fed into the decoder.</p><p><strong>Decoder Blocks:</strong> Each decoder block starts with a transposed convolution for upsampling, followed by concatenation with the corresponding encoder output, and convolutional layers similar to those in the encoder.</p><pre>def encoder_block(input_tensor, num_filters, dropout_rate=0.2):<br>    x = Conv2D(num_filters, (3, 3), padding=&#39;same&#39;)(input_tensor)<br>    x = Dropout(dropout_rate)(x)<br>    x = BatchNormalization()(x)<br>    x = Activation(&#39;relu&#39;)(x)<br><br>    x = Conv2D(num_filters, (3, 3), padding=&#39;same&#39;)(x)<br>    x = Dropout(dropout_rate)(x)<br>    x = BatchNormalization()(x)<br>    x = Activation(&#39;relu&#39;)(x)<br><br>    p = MaxPooling2D((2, 2))(x)<br>    return x, p<br><br>def spatial_attention_block(input_tensor):<br>    max_pool = tf.reduce_max(input_tensor, axis=-1, keepdims=True)<br>    avg_pool = tf.reduce_mean(input_tensor, axis=-1, keepdims=True)<br><br>    concat = tf.concat([max_pool, avg_pool], axis=-1)<br>    attention = Conv2D(1, (7, 7), padding=&#39;same&#39;, activation=&#39;sigmoid&#39;)(concat)<br>    output = Multiply()([input_tensor, attention])<br><br>    return output<br><br>def decoder_block(input_tensor, skip_features, num_filters, dropout_rate=0.2):<br>    x = Conv2DTranspose(num_filters, (3, 3), strides=(2, 2), padding=&#39;same&#39;)(input_tensor)<br>    x = concatenate([x, skip_features])<br><br>    x = Conv2D(num_filters, (3, 3), padding=&#39;same&#39;)(x)<br>    x = Dropout(dropout_rate)(x)<br>    x = BatchNormalization()(x)<br>    x = Activation(&#39;relu&#39;)(x)<br><br>    x = Conv2D(num_filters, (3, 3), padding=&#39;same&#39;)(x)<br>    x = Dropout(dropout_rate)(x)<br>    x = BatchNormalization()(x)<br>    x = Activation(&#39;relu&#39;)(x)<br>    return x<br><br>img_width, img_height = 256, 256<br><br>def unet_model(input_shape = (img_width, img_height, 3)):<br>  inputs = Input(input_shape)<br><br>  # Encoder<br>  e1, p1 = encoder_block(inputs, 16, 0.2)<br>  e2, p2 = encoder_block(p1, 32, 0.2)<br>  e3, p3 = encoder_block(p2, 64, 0.2)<br>  e4, p4 = encoder_block(p3, 128, 0.2)<br><br>  # Spatial Attention<br>  sa = spatial_attention_block(p4)<br><br>  # Decoder<br>  d1 = decoder_block(sa, e4, 128, 0.2)<br>  d2 = decoder_block(d1, e3, 64, 0.2)<br>  d3 = decoder_block(d2, e2, 32, 0.2)<br>  d4 = decoder_block(d3, e1, 16, 0.2)<br><br>  # Output<br>  outputs = Conv2D(filters=1, kernel_size =(1, 1), activation=&#39;sigmoid&#39;)(d4)<br><br>  return Model(inputs=[inputs], outputs = [outputs])</pre><p><strong>Setting Up Training Parameters</strong></p><p>Here, we define hyperparameters for model training. The model is trained for 50 epochs with some augmentations.</p><pre>epochs = 50<br>batch_size = 32<br>lr = 1e-4<br><br>train_generator_args = dict(<br>    rotation_range = 0.25,<br>    width_shift_range = 0.05,<br>    height_shift_range=0.05,<br>    shear_range=0.05,<br>    zoom_range=0.05,<br>    horizontal_flip=True,<br>    fill_mode=&#39;nearest&#39;<br>)<br><br>train_gen  = train_generator( df_train, batch_size, train_generator_args)<br>test_gen = train_generator(df_test, batch_size, dict() )<br><br>model = unet_model(input_shape =(256,256, 3))<br>optimizer = Adam(learning_rate = lr , beta_1 = 0.9, beta_2 = 0.999, epsilon = None, amsgrad = False)<br><br>model.compile(optimizer=optimizer, loss=dice_coefficients_loss, metrics=[&quot;binary_accuracy&quot;, iou, dice_coefficients])<br>callbacks = [<br>    ModelCheckpoint(&#39;unet_model.hdf5&#39;,verbose=1, save_best_only=True )<br>]<br><br>history = model.fit(<br>      train_gen,<br>      steps_per_epoch = len(df_train) / batch_size,<br>      epochs = epochs,<br>      callbacks = callbacks,<br>      validation_data = test_gen,<br>      validation_steps = len(df_validation) / batch_size<br>)</pre><p><strong>Visualize the Model and Graphs</strong></p><p>Now the training is done we will see how the model has performed by tracking the progress across each epoch. The graphs for loss and accuracy will throw some light on it.</p><pre>from tensorflow import keras<br>keras.utils.plot_model(model, to_file = &quot;Model.png&quot;, show_shapes=True)<br><br>history_training = history.history<br><br>train_dice_coeff_list = history_training[&#39;dice_coefficients&#39;]<br>test_dice_coeff_list = history_training[&#39;val_dice_coefficients&#39;]<br><br>train_jaccard_list = history_training[&#39;iou&#39;]<br>test_jaccard_list = history_training[&#39;val_iou&#39;]<br><br>train_loss_list = history_training[&#39;loss&#39;]<br>test_loss_list = history_training[&#39;val_loss&#39;]<br><br><br>plt.plot(test_loss_list, &#39;b-&#39;, label=&#39;Test Loss&#39;)<br>plt.plot(train_loss_list, &#39;r-&#39;, label=&#39;Train Loss&#39;)<br><br>plt.xlabel(&#39;iterations&#39;)<br>plt.ylabel(&#39;loss&#39;)<br>plt.title(&#39;loss graph&#39;, fontsize=12)<br>plt.legend()<br>plt.show()<br>plt.savefig(&#39;Loss Graph&#39;)<br><br>plt.plot(train_dice_coeff_list, &#39;b-&#39;, label=&#39;Train Accuracy&#39;)<br>plt.plot(test_dice_coeff_list, &#39;r-&#39;, label = &#39;Test Accuracy&#39;)<br><br>plt.xlabel(&#39;iterations&#39;)<br>plt.ylabel(&#39;accuracy&#39;)<br>plt.title(&#39;Accuracy graph&#39;, fontsize=12)<br>plt.legend()<br>plt.show()<br>plt.savefig(&#39;Accuracy Graph&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*ra6ReNpRVqYyaFuyjzxEag.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*TPvXFKNygBJA8XkRPlovUQ.png" /></figure><p><strong>Model Evaluation</strong></p><p>Here we will save the trained model parameters to a file and load them for evaluation. This will give us the metrics computed on test dataset.</p><pre>model = load_model(&#39;unet_model.hdf5&#39;, custom_objects={&#39;dice_coefficients_loss&#39;: dice_coefficients_loss, &#39;iou&#39;: iou, &#39;dice_coefficients&#39;: dice_coefficients})<br><br>test_gen = train_generator(df_test, batch_size, dict(), target_size=(img_height,img_width))<br>results = model.evaluate(test_gen, steps = len(df_test)/batch_size)<br>print(&#39;Test Loss&#39;, results[0])<br>print(&#39;Test IOU&#39;, results[1])<br>print(&#39;Test Dice Coeff&#39;, results[2])</pre><p><strong>Visualize Predictions</strong></p><p>This piece of code does some post-processing to display the inference images.</p><pre>for i in range(20):<br>    index = np.random.randint(0, len(df_test.index))<br>    img_path = df_test[&#39;image_path&#39;].iloc[index]<br>    mask_path = df_test[&#39;mask_path&#39;].iloc[index]<br><br>    # Read and preprocess the image<br>    img = cv2.imread(img_path)<br>    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)<br>    img_resized = cv2.resize(img, (256, 256))<br>    img_normalized = img_resized / 255.0<br>    img_expanded = np.expand_dims(img_normalized, axis=0)<br><br>    # Generate the prediction<br>    pred_mask = model.predict(img_expanded)<br>    pred_mask_thresholded = (pred_mask.squeeze() &gt; 0.5).astype(np.float32)<br><br>    # Read and preprocess the mask<br>    original_mask = cv2.imread(mask_path, cv2.IMREAD_GRAYSCALE)<br>    original_mask_resized = cv2.resize(original_mask, (256, 256))<br>    fig, axs = plt.subplots(1, 3, figsize=(18, 6))  # Width, Height<br><br>    # Display the original image<br>    axs[0].imshow(img_resized)<br>    axs[0].set_title(&quot;Original Image&quot;)<br>    axs[0].axis(&#39;off&#39;)  # Hide axes ticks<br><br>    # Display the original mask<br>    axs[1].imshow(original_mask_resized, cmap=&#39;gray&#39;)<br>    axs[1].set_title(&quot;Original Mask&quot;)<br>    axs[1].axis(&#39;off&#39;)<br><br>    # Display the predicted mask<br>    axs[2].imshow(pred_mask_thresholded, cmap=&#39;gray&#39;)<br>    axs[2].set_title(&quot;Predicted Mask&quot;)<br>    axs[2].axis(&#39;off&#39;)<br><br>    plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N573Ufjjl_EysE-PBpSf_w.png" /></figure><p><strong>Conclusion</strong></p><p>I hope this guide has illuminated the path through brain MRI segmentation with SA-UNet, showing its effectiveness beyond traditional applications. By leveraging the LGG-MRI-Segmentation dataset, we’ve underlined the model’s versatility in medical imaging, providing deep insights into brain anatomy and pathologies. This project underscores the transformative potential of deep learning in enhancing diagnostic precision.</p><p>For any inquiries or feedback, feel free to connect. Your experiences and challenges are invaluable to further refining and exploring the potentials of this technology.</p><p><strong>References</strong></p><p>For a deeper understanding of the methodologies and tools used in this project, consider exploring the following resources:</p><ul><li><a href="https://github.com/suniash/brain_mri_segmentation">Brain MRI Segmentation Repository</a></li><li><a href="https://www.kaggle.com/datasets/mateuszbuda/lgg-mri-segmentation">Kaggle LGG MRI Segmentation Dataset</a></li><li><a href="https://arxiv.org/abs/2004.03696">SA-UNet Model</a></li><li><a href="https://github.com/clguo/SA-UNet">SA-UNet Official Repository</a></li></ul><p>This project stands as a testament to applying cutting-edge deep learning techniques in medical imaging, pushing the boundaries of what is possible in diagnostics and treatment planning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b6cee284dd0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking the Power of GPT-2 for News Headline Analysis]]></title>
            <link>https://medium.com/@sunidhi.ashtekar/unlocking-the-power-of-gpt-2-for-news-headline-analysis-ae39b76af0f6?source=rss-1d599ba5668e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae39b76af0f6</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[genarative-ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sunidhi Ashtekar]]></dc:creator>
            <pubDate>Sat, 16 Mar 2024 00:46:26 GMT</pubDate>
            <atom:updated>2024-03-16T23:37:39.110Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*sE2rtUTYuPCl2MsU6QTPDQ.png" /><figcaption>Image from<a href="https://blog.gluelabs.com/how-ai-reads-words-289f61c84c1b"> Article</a></figcaption></figure><p>In this article, we delve into an exciting exploration of the <a href="https://openai.com/research/gpt-2-1-5b-release">GPT-2</a> model from <a href="https://huggingface.co/docs/transformers/index">Hugging Face</a> transformer library, repurposing it for text classification tasks with News Category Dataset.</p><p>Text classification stands as a pivotal task in the realm of natural language processing (NLP), enabling the systematic organisation and categorisation of textual data into predefined classes. NLP’s significance is underscored by its ability to bridge human communication and computational analysis, facilitating advancements across a myriad of applications from sentiment analysis to topic identification.</p><p><strong>Introduction to Transformers</strong></p><p>The advent of transformer models like <a href="https://huggingface.co/docs/transformers/model_doc/bert">BERT</a> and GPT has significantly advanced text classification, offering unprecedented efficiency and accuracy. By understanding language nuances through deep learning, these models have transformed text interpretation, capturing contextual relationships to improve performance and broaden NLP capabilities. This shift emphasizes the continuous evolution and impact of machine learning in processing and structuring extensive textual data.</p><p>While GPT-2 might not wear the crown of the latest transformer technology, especially with the arrival of its successors GPT-3, GPT-3.5 and GPT-4, it remains a cornerstone in the landscape of natural language processing.</p><p>BERT and GPT-2, as transformer models, are well-recognized for their NLP prowess, differentiated by their design: GPT-2’s decoder blocks excel in text generation, while BERT’s encoder blocks enhance text understanding. However, GPT-2’s capacity for text classification, despite being less explored compared to BERT’s versatility, offers a promising research opportunity.</p><p>GPT-2’s architecture uniquely predicts the next token using the final token of a sequence, a feature that can be cleverly repurposed for text classification. This allows the model to use the importance of the last token in classifying text snippets effectively.</p><p>Despite the advancements of GPT-3 and GPT-4, their closed-source nature and demand for substantial computational resources limit accessibility and practicality for custom experimentation. Hence, we pivot to GPT-2, which offers a middle ground in linguistic sophistication and operational viability, making it a suitable option for text classification. This choice aims to showcase GPT-2’s versatility and untapped potential within natural language processing.</p><p><strong>Getting Started</strong></p><p>For classifying news headlines with GPT-2, follow these structured steps:</p><ul><li>Get the Dataset</li><li>Environment setup</li><li>Verify and Understand the Dataset</li><li>Data Tokenisation</li><li>Data Pre-processing</li><li>Model Training</li><li>Model Evaluation</li><li>Save the Model Weights</li><li>Model Inference</li></ul><p><strong>About the Dataset</strong></p><p>The dataset hails from Kaggle’s “<a href="https://www.kaggle.com/datasets/rmisra/news-category-dataset">News Category Dataset</a>,” a comprehensive collection that spans a spectrum of 42 categories, each offering a glimpse into different facets of global news stories. It is richly detailed, featuring elements such as headlines, short descriptions, authors, and publication dates for each article.</p><p>Due to the dataset’s expansive nature, focusing on all 42 categories would dilute the specificity and efficiency of the model training process. Therefore, the project narrows its scope to the five most populated categories: Politics, Sports, Technology, Entertainment, and Business. This decision is driven by the intention to harness a more manageable and representative subset of data, ensuring a robust learning process and more accurate classification outcomes. By concentrating on these categories, the project optimizes the balance between variety and volume, setting a solid foundation for effective news headline classification with GPT-2.</p><p><strong>Environment Setup</strong></p><p>A Google account is required to access Google Colab, a free cloud service providing necessary computational resources for deep learning tasks, including access to GPUs and TPUs up to 16 GB.</p><p>Let’s get started, we first ensure availability and readiness of the GPU for processing our transformer model and mount the Google drive to utilize the dataset stored followed by installing the necessary Python packages.</p><pre>! nvidia-smi<br><br>drive.mount(&#39;/content/drive&#39;)<br><br>! pip install datasets transformers torchvision<br><br>import argparse<br>import json<br>import logging<br>import re<br><br>import matplotlib.pyplot as plt<br>import numpy as np<br>import pandas as pd<br>import seaborn as sns<br>import torch<br>import torch.nn as nn<br>import torch.optim as optim<br>from google.colab import drive<br>from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay<br>from torch.nn.utils import clip_grad_norm_<br>from torch.optim.lr_scheduler import ReduceLROnPlateau, StepLR<br>from torch.utils.data import DataLoader<br>from torch.utils.data.dataloader import default_collate<br>from torchvision import transforms<br>from tqdm import tqdm<br><br>from transformers import (AutoModelForSequenceClassification, DataCollatorWithPadding,<br>                          GPT2Model, GPT2Tokenizer, Trainer, TrainingArguments)</pre><h4>Loading and Displaying the Dataset</h4><p>Once the environment is ready we read the News_Category_Dataset_v3, a JSON file from drive and store it in a pandas DataFrame. Then we will display few entries from the dataset. Here’s how the it looks like, there are 6 columns from which we will choose ‘Headlines’ as our text input and ‘Category’ as label.</p><pre>df = pd.read_json(&#39;/content/drive/My Drive/News_Category_Dataset_v3.json&#39;, lines=True)<br>print(&#39;No. of rows and columns :&#39;, df.shape)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*miCF8eRoUGmWCA5Rile1SA.png" /></figure><h4>Visualizing the Distribution of News Categories</h4><p>To verify the dataset we analyze the distribution of news articles across different categories. This gives us a clear idea of how many news articles are available per category.</p><pre>count_df = pd.DataFrame(df[&#39;category&#39;].value_counts()).reset_index()<br>print(&#39;There are&#39;, len(count_df), &#39;news categories&#39;)<br><br>sns.set_style(&#39;darkgrid&#39;)<br>plt.figure(figsize=(10, 12))<br>sns.barplot(data=count_df, y=&#39;index&#39;, x=&#39;category&#39;, palette=&#39;Dark2&#39;)<br>plt.title(&#39;No. news in each category&#39;, loc=&#39;left&#39;, fontsize=20)<br>plt.xlabel(&quot;&quot;)<br>plt.ylabel(&quot;&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*n5a5v8Savg8HvvsH7ICPAQ.png" /></figure><h4>Initialization of the Tokenizer for Text Processing</h4><p>Once the data is verified, we initialize the GPT2Tokenizer. The tokenizer is configured to pad sequences from the left side to maintain consistency with GPT-2’s expected input format. Additionally, the end-of-string token (eos_token) is assigned as the padding token, ensuring that padded inputs are treated appropriately during model training and inference. This step is crucial for preparing the text data for subsequent processing stages, such as encoding the headlines into a format that can be processed by the neural network. Tokenisation is basically dividing large sentences in small words so that model can understand.</p><p>The input_ids is the text input and attention mask is for handling the padding. These two are text inputs given to the model.</p><pre># Initialize the tokenizer<br>tokenizer = GPT2Tokenizer.from_pretrained(&#39;gpt2&#39;)<br>tokenizer.padding_side = &quot;left&quot;<br>tokenizer.pad_token = tokenizer.eos_token  # Set pad token<br><br>example_text = &quot;I will watch Movie tonight&quot;<br>gpt2_input = tokenizer(example_text, padding=&quot;max_length&quot;, max_length=10, truncation=True, return_tensors=&quot;pt&quot;)<br><br>print(gpt2_input[&#39;input_ids&#39;])<br><br>print(gpt2_input[&quot;attention_mask&quot;])</pre><h4>Defining a Stopwords List for Text Cleaning</h4><p>Next action item in the pipeline is the data preprocessing, for that we define a list of common English stopwords sourced from a reputable SEO tool’s GitHub repository. Stopwords are words which are commonly removed from text data before processing in natural language tasks as they often carry little informational weight. This list includes pronouns, conjunctions, prepositions, and other words that are frequently used in the English language, but are typically not useful for understanding the context or sentiment of a text. By defining this list, we can preprocess and clean our news headline data to enhance the performance of the classification model by focusing on the most meaningful words.</p><pre># Stopwords list from https://github.com/Yoast/YoastSEO.js/blob/develop/src/config/stopwords.js<br>stopwords = [ &quot;a&quot;, &quot;about&quot;, &quot;above&quot;, &quot;after&quot;, &quot;again&quot;, &quot;against&quot;, &quot;all&quot;, &quot;am&quot;, &quot;an&quot;, &quot;and&quot;, &quot;any&quot;, &quot;are&quot;, &quot;as&quot;, &quot;at&quot;, &quot;be&quot;, &quot;because&quot;, &quot;been&quot;, &quot;before&quot;, &quot;being&quot;, &quot;below&quot;, &quot;between&quot;, &quot;both&quot;, &quot;but&quot;, &quot;by&quot;, &quot;could&quot;, &quot;did&quot;, &quot;do&quot;, &quot;does&quot;, &quot;doing&quot;, &quot;down&quot;, &quot;during&quot;, &quot;each&quot;, &quot;few&quot;, &quot;for&quot;, &quot;from&quot;, &quot;further&quot;, &quot;had&quot;, &quot;has&quot;, &quot;have&quot;, &quot;having&quot;, &quot;he&quot;, &quot;he&#39;d&quot;, &quot;he&#39;ll&quot;, &quot;he&#39;s&quot;, &quot;her&quot;, &quot;here&quot;, &quot;here&#39;s&quot;, &quot;hers&quot;, &quot;herself&quot;, &quot;him&quot;, &quot;himself&quot;, &quot;his&quot;, &quot;how&quot;, &quot;how&#39;s&quot;, &quot;i&quot;, &quot;i&#39;d&quot;, &quot;i&#39;ll&quot;, &quot;i&#39;m&quot;, &quot;i&#39;ve&quot;, &quot;if&quot;, &quot;in&quot;, &quot;into&quot;, &quot;is&quot;, &quot;it&quot;, &quot;it&#39;s&quot;, &quot;its&quot;, &quot;itself&quot;, &quot;let&#39;s&quot;, &quot;me&quot;, &quot;more&quot;, &quot;most&quot;, &quot;my&quot;, &quot;myself&quot;, &quot;nor&quot;, &quot;of&quot;, &quot;on&quot;, &quot;once&quot;, &quot;only&quot;, &quot;or&quot;, &quot;other&quot;, &quot;ought&quot;, &quot;our&quot;, &quot;ours&quot;, &quot;ourselves&quot;, &quot;out&quot;, &quot;over&quot;, &quot;own&quot;, &quot;same&quot;, &quot;she&quot;, &quot;she&#39;d&quot;, &quot;she&#39;ll&quot;, &quot;she&#39;s&quot;, &quot;should&quot;, &quot;so&quot;, &quot;some&quot;, &quot;such&quot;, &quot;than&quot;, &quot;that&quot;, &quot;that&#39;s&quot;, &quot;the&quot;, &quot;their&quot;, &quot;theirs&quot;, &quot;them&quot;, &quot;themselves&quot;, &quot;then&quot;, &quot;there&quot;, &quot;there&#39;s&quot;, &quot;these&quot;, &quot;they&quot;, &quot;they&#39;d&quot;, &quot;they&#39;ll&quot;, &quot;they&#39;re&quot;, &quot;they&#39;ve&quot;, &quot;this&quot;, &quot;those&quot;, &quot;through&quot;, &quot;to&quot;, &quot;too&quot;, &quot;under&quot;, &quot;until&quot;, &quot;up&quot;, &quot;very&quot;, &quot;was&quot;, &quot;we&quot;, &quot;we&#39;d&quot;, &quot;we&#39;ll&quot;, &quot;we&#39;re&quot;, &quot;we&#39;ve&quot;, &quot;were&quot;, &quot;what&quot;, &quot;what&#39;s&quot;, &quot;when&quot;, &quot;when&#39;s&quot;, &quot;where&quot;, &quot;where&#39;s&quot;, &quot;which&quot;, &quot;while&quot;, &quot;who&quot;, &quot;who&#39;s&quot;, &quot;whom&quot;, &quot;why&quot;, &quot;why&#39;s&quot;, &quot;with&quot;, &quot;would&quot;, &quot;you&quot;, &quot;you&#39;d&quot;, &quot;you&#39;ll&quot;, &quot;you&#39;re&quot;, &quot;you&#39;ve&quot;, &quot;your&quot;, &quot;yours&quot;, &quot;yourself&quot;, &quot;yourselves&quot; ]</pre><h4>Balancing the Dataset Across Categories</h4><p>Moving ahead, we address potential class imbalance by creating a more balanced dataset. For each unique category in our dataset, we subset the DataFrame to include only the data from that category. From each subset, we randomly sample a fixed number of entries (1004 in this case, which is chosen to match the size of the smallest class or for another specific reason). All these balanced subsets are then stored in a list.</p><pre>dfs = []<br><br>for category in df[&#39;category&#39;].unique():<br>    temp = df[df[&#39;category&#39;]==category]<br>    dfs.append(temp.sample(1004))<br><br>df = pd.concat(dfs)<br>df = df.sample(frac=1).reset_index(drop=True)<br>df.head()</pre><h4>Cleaning and Stopwords Removal</h4><p>Here are two functions essential for preprocessing the text data, making it more suitable for model training and analysis:</p><p>alpha_num: This function removes all characters from the input text that are not alphanumeric (letters and numbers) or spaces. This cleaning step is crucial for standardizing the text data and focusing the model’s attention on meaningful words and numbers.</p><p>remove_stopwords: Removing stopwords helps in reducing the noise within the data, allowing models to focus on the words that carry significant meaning or sentiment.</p><p>Together, these functions streamline the dataset by removing irrelevant characters and words, thereby improving the efficiency and performance of subsequent natural language processing tasks.</p><pre>def alpha_num(text):<br>    return re.sub(r&#39;[^A-Za-z0-9 ]&#39;, &#39;&#39;, text)<br><br>def remove_stopwords(text):<br>    final_text = []<br>    for i in text.split():<br>        if i.strip().lower() not in stopwords:<br>            final_text.append(i.strip())<br>    return &quot; &quot;.join(final_text)<br><br>df[&#39;headline&#39;] = df[&#39;headline&#39;].str.lower()<br>df[&#39;headline&#39;] = df[&#39;headline&#39;].apply(alpha_num)<br>df[&#39;headline&#39;] = df[&#39;headline&#39;].apply(remove_stopwords)<br><br>df.head()</pre><h4>Narrowing Down to Selected News Categories</h4><p>We refine our dataset using the filter_categories function to include only headlines from five categories: ‘POLITICS’, ‘SPORTS’, ‘TECH’, ‘BUSINESS’, ‘ENTERTAINMENT’. This targeted approach sharpens the model&#39;s focus and potentially boosts its performance on these specific topics, streamlining the data for subsequent processing and training. You can select any categories of your choice from 42!</p><pre>def filter_categories(df, categories_to_keep):<br>    &quot;&quot;&quot;<br>    Filters the DataFrame to keep only the rows with the specified categories.<br><br>    Parameters:<br>    - df (pandas.DataFrame): The original DataFrame.<br>    - categories_to_keep (list of str): The list of categories to retain.<br><br>    Returns:<br>    - pandas.DataFrame: A DataFrame with only the rows of the specified categories.<br>    &quot;&quot;&quot;<br>    return df[df[&#39;category&#39;].isin(categories_to_keep)]<br><br># Define the categories to keep<br>categories_to_keep = [&#39;POLITICS&#39;, &#39;SPORTS&#39;, &#39;TECH&#39;, &#39;BUSINESS&#39;, &#39;ENTERTAINMENT&#39;]<br><br>df = filter_categories(df, categories_to_keep)</pre><h4>Ensuring Data Quality by Removing Empty Headlines</h4><p>In this crucial step, we meticulously cleanse our dataset, discarding any entries where the ‘headline’ field is missing or filled with mere whitespace. This purification step ensures our model trains on meaningful, high-quality data, laying the groundwork for effective machine learning outcomes.</p><pre>df = df[df[&#39;headline&#39;].notnull() &amp; (df[&#39;headline&#39;].str.strip() != &#39;&#39;)]<br><br># unique news category<br>df[&#39;category&#39;].unique()</pre><h4>Creating a Custom Dataset for News Headline Classification</h4><p>Now, I will create a custom Dataset class for PyTorch, designed to refine our news headlines dataset for efficient training and evaluation. It includes label encoding to map categories to integers, initialization to standardize headlines and organize them into lists of encoded labels and tokenized texts, and utility methods to facilitate data retrieval and compatibility with PyTorch’s DataLoader. This class is pivotal for streamlined data management, optimizing our dataset for the GPT-2 classification task.</p><pre>labels = {category: i for i, category in enumerate([<br>    &#39;POLITICS&#39;, &#39;SPORTS&#39;, &#39;TECH&#39;, &#39;ENTERTAINMENT&#39;, &#39;BUSINESS&#39;])}<br><br>class Dataset(torch.utils.data.Dataset):<br>    def __init__(self, df):<br>        self.labels = [labels[label] for label in df[&#39;category&#39;]]<br>        self.texts = [tokenizer(text,<br>                                padding=&#39;max_length&#39;,<br>                                max_length=128,<br>                                truncation=True,<br>                                return_tensors=&quot;pt&quot;) for text in df[&#39;headline&#39;]]<br>        for i, enc in enumerate(self.texts):<br>            if enc[&#39;input_ids&#39;].size(1) == 0:<br>                print(f&quot;Empty input at index: {i}, text: {df.iloc[i][&#39;headline&#39;]}&quot;)<br><br><br>    def classes(self):<br>        return self.labels<br><br>    def __len__(self):<br>        return len(self.labels)<br><br>    def get_batch_labels(self, idx):<br>        return np.array(self.labels[idx])<br><br>    def get_batch_texts(self, idx):<br>        return self.texts[idx]<br><br>    def __getitem__(self, idx):<br>        batch_texts = self.get_batch_texts(idx)<br>        batch_y = self.get_batch_labels(idx)<br>        return batch_texts, batch_y</pre><h4>Splitting the Dataset into Training, Validation, and Test Sets</h4><p>The last step in data preprocessing before we dive into our transformer model!</p><pre>np.random.seed(112)<br>df_train, df_val, df_test = np.split(df.sample(frac=1, random_state=35),<br>                                     [int(0.8*len(df)), int(0.9*len(df))])<br><br>print(len(df_train), len(df_val), len(df_test))</pre><h4>Defining GPT-2 Based Sequence Classifier</h4><p>By integrating the pre-trained GPT-2 model for feature extraction and adding a linear classification layer on top of it, this architecture excels in categorizing text, such as news headlines, into specific groups.</p><pre>class GPT2SequenceClassifier(nn.Module):<br>    def __init__(self, hidden_size: int, num_classes:int ,max_seq_len:int, gpt_model_name:str):<br>        super(GPT2SequenceClassifier,self).__init__()<br>        self.gpt2model = GPT2Model.from_pretrained(gpt_model_name)<br>        self.fc1 = nn.Linear(hidden_size*max_seq_len, num_classes)<br><br><br>    def forward(self, input_id, mask):<br>        gpt_out, _ = self.gpt2model(input_ids=input_id, attention_mask=mask, return_dict=False)<br>        batch_size = gpt_out.shape[0]<br>        linear_output = self.fc1(gpt_out.view(batch_size,-1))<br>        return linear_output</pre><h4>Training and Evaluating the GPT-2 Classifier</h4><p>In our GPT-2 classifier’s training loop, we prepare data using DataLoaders, initialize the model with CrossEntropyLoss and Adam optimizer, and leverage GPU acceleration. Each epoch involves forward passes, loss minimization, and parameter updates. The model’s efficacy is highlighted by an 85.2% training accuracy and 97.1% on validation showcasing strong learning and generalization capabilities.</p><pre>def train(model, train_data, val_data, learning_rate, epochs):<br>    train, val = Dataset(train_data), Dataset(val_data)<br><br>    train_dataloader = torch.utils.data.DataLoader(train, batch_size=8, shuffle=True)<br>    val_dataloader = torch.utils.data.DataLoader(val, batch_size=8)<br><br>    use_cuda = torch.cuda.is_available()<br>    device = torch.device(&quot;cuda&quot; if use_cuda else &quot;cpu&quot;)<br><br>    criterion = nn.CrossEntropyLoss()<br>    optimizer = optim.Adam(model.parameters(), lr=learning_rate)<br><br>    if use_cuda:<br>        model = model.cuda()<br>        criterion = criterion.cuda()<br><br>    for epoch_num in range(epochs):<br>        total_acc_train = 0<br>        total_loss_train = 0<br><br>        for train_input, train_label in tqdm(train_dataloader):<br>            train_label = train_label.to(device)<br>            mask = train_input[&#39;attention_mask&#39;].to(device)<br>            input_id = train_input[&quot;input_ids&quot;].squeeze(1).to(device)<br><br>            model.zero_grad()<br><br>            output = model(input_id, mask)<br><br>            batch_loss = criterion(output, train_label)<br>            total_loss_train += batch_loss.item()<br><br>            acc = (output.argmax(dim=1)==train_label).sum().item()<br>            total_acc_train += acc<br><br>            batch_loss.backward()<br>            optimizer.step()<br><br>        total_acc_val = 0<br>        total_loss_val = 0<br><br>        with torch.no_grad():<br><br>            for val_input, val_label in val_dataloader:<br>                val_label = val_label.to(device)<br>                mask = val_input[&#39;attention_mask&#39;].to(device)<br>                input_id = val_input[&#39;input_ids&#39;].squeeze(1).to(device)<br><br>                output = model(input_id, mask)<br><br>                batch_loss = criterion(output, val_label)<br>                total_loss_val += batch_loss.item()<br><br>                acc = (output.argmax(dim=1)==val_label).sum().item()<br>                total_acc_val += acc<br><br>            print(<br>            f&quot;Epochs: {epoch_num + 1} | Train Loss: {total_loss_train/len(train_data): .3f} \<br>            | Train Accuracy: {total_acc_train / len(train_data): .3f} \<br>            | Val Loss: {total_loss_val / len(val_data): .3f} \<br>            | Val Accuracy: {total_acc_val / len(val_data): .3f}&quot;)<br><br>EPOCHS = 1<br>model = GPT2SequenceClassifier(hidden_size=768, num_classes=5, max_seq_len=128, gpt_model_name=&quot;gpt2&quot;)<br>LR = 1e-5<br><br>train(model, df_train, df_val, LR, EPOCHS)</pre><h4>Evaluating the GPT-2 Classifier on Test Data</h4><p>This step involves evaluating our trained GPT-2 sequence classifier on the test dataset, aiming to measure the model’s generalization ability on unseen data. This evaluation reflects the model’s effectiveness, evidenced by a test accuracy of 94%, indicating robust performance in classifying new examples.</p><pre>def evaluate(model, test_data):<br><br>    test = Dataset(test_data)<br><br>    test_dataloader = torch.utils.data.DataLoader(test, batch_size=2)<br><br>    use_cuda = torch.cuda.is_available()<br>    device = torch.device(&quot;cuda&quot; if use_cuda else &quot;cpu&quot;)<br><br>    if use_cuda:<br><br>        model = model.cuda()<br><br>    predictions_labels = []<br>    true_labels = []<br><br>    total_acc_test = 0<br>    with torch.no_grad():<br><br>        for test_input, test_label in test_dataloader:<br><br>            test_label = test_label.to(device)<br>            mask = test_input[&#39;attention_mask&#39;].to(device)<br>            input_id = test_input[&#39;input_ids&#39;].squeeze(1).to(device)<br><br>            output = model(input_id, mask)<br><br>            acc = (output.argmax(dim=1) == test_label).sum().item()<br>            total_acc_test += acc<br><br>            true_labels += test_label.cpu().numpy().flatten().tolist()<br>            predictions_labels += output.argmax(dim=1).cpu().numpy().flatten().tolist()<br><br>    print(f&#39;Test Accuracy: {total_acc_test / len(test_data): .3f}&#39;)<br>    return true_labels, predictions_labels<br><br>true_labels, pred_labels = evaluate(model, df_test)</pre><h4>Visualizing Model Performance with a Confusion Matrix</h4><p>Then comes the confusion matrix, that we will use to succinctly visualize our GPT-2 classifier’s performance, revealing its accuracy across categories and identifying specific areas of strength and misclassification.</p><pre>fig, ax = plt.subplots(figsize=(8, 8))<br>cm = confusion_matrix(y_true=true_labels, y_pred=pred_labels, labels=range(len(labels)), normalize=&#39;true&#39;)<br>disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=list(labels.keys()))<br>disp.plot(ax=ax)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*ANXbcaf1zaaA37Y2mXyERA.png" /></figure><h4>Saving the Trained Model</h4><p>This cell handles saving the trained GPT-2 classifier’s parameters to a file and loading them as needed, utilizing torch.save to store the model&#39;s state dictionary. This process ensures the model can be reused without retraining, streamlining deployment and further experimentation. Saving and loading the model is a crucial step in preserving its training outcomes for future use.</p><pre>torch.save(model.state_dict(), &quot;/content/drive/My Drive/gpt2-text-classifier-model.pt&quot;)<br><br>model_new = GPT2SequenceClassifier(hidden_size=768, num_classes=5, max_seq_len=128, gpt_model_name=&quot;gpt2&quot;)<br>model_new.load_state_dict(torch.load(&quot;/content/drive/My Drive/gpt2-text-classifier-model.pt&quot;))<br>model_new.eval()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*m7Sn1j1m1T4TU48GAUnWqw.png" /></figure><h4>Model Inference</h4><p>For inference using our trained GPT-2 sequence classifier we will give example text from test dataset so it matches the format expected by the model, allowing us to predict its category. The process culminates in accurately classifying the text as ‘POLITICS’, demonstrating our model’s adeptness at discerning and categorizing textual content.</p><pre>example_text = &quot;&quot;&quot;<br>  journalisms challenge unending state emergency<br>&quot;&quot;&quot;<br>fixed_text = &quot; &quot;.join(example_text.lower().split())<br>print(fixed_text)<br><br>tokenizer = GPT2Tokenizer.from_pretrained(&#39;gpt2&#39;)<br>tokenizer.padding_side = &quot;left&quot;<br>tokenizer.pad_token = tokenizer.eos_token<br><br>model_input = tokenizer(fixed_text, padding=&#39;max_length&#39;, max_length=128, truncation=True, return_tensors=&quot;pt&quot;)<br><br>mask = model_input[&#39;attention_mask&#39;].cpu()<br>input_id = model_input[&quot;input_ids&quot;].squeeze(1).cpu()<br><br>output = model_new(input_id, mask)<br><br>prob = torch.nn.functional.softmax(output, dim=1)[0]<br>print(prob)<br><br>labels_to_ids = {category: i for i, category in enumerate([<br>    &#39;POLITICS&#39;, &#39;SPORTS&#39;, &#39;TECH&#39;, &#39;ENTERTAINMENT&#39;, &#39;BUSINESS&#39;<br>])}<br><br># Create a dictionary to map numeric labels to category names<br>ids_to_labels = {i: category for category, i in labels_to_ids.items()}<br><br>pred_label = ids_to_labels[output.argmax(dim=1).item()]<br>print(pred_label)<br><br>&quot;&quot;&quot;outputs=POLITICS&quot;&quot;&quot;</pre><h4>Conclusion</h4><p>I hope this tutorial has been informative and that you’ve found success in navigating the complexities of text classification with GPT-2. Through the utilization of the Kaggle “News Category Dataset,” we’ve demonstrated the adaptability of GPT-2 beyond its typical text generation applications, showcasing its potential in classifying news headlines into distinct categories. This endeavour highlights the versatility of transformer models in addressing diverse NLP challenges, offering insights into their capacity for nuanced language understanding and categorization.</p><p>If you have any queries, suggestions, or feedback, I warmly invite you to connect with me on LinkedIn. I’m always open to discussions and keen to engage with your experiences, offer assistance, or explore any challenges you might face.</p><h4>References</h4><p>For those looking to delve deeper into the methodologies and tools employed in this project, the following resources are invaluable:</p><ul><li><a href="https://github.com/suniash/news_headline_classifier_with_GPT2">News Headline Classification Git Repository</a></li><li><a href="https://github.com/openai/gpt-2">GPT-2 Official Repository</a></li><li><a href="https://github.com/huggingface/transformers">Hugging Face Transformers Library</a></li><li>Kaggle News Category Dataset: For accessing a diverse dataset that forms the backbone of this classification task. Visit <a href="https://www.kaggle.com/">Kaggle</a> to explore this and other datasets.</li></ul><p>This project serves as a practical illustration of leveraging advanced NLP techniques for real-world applications. By employing GPT-2 for news headline classification, we not only expand the scope of tasks that transformer models can perform but also open up new avenues for research and development in natural language processing. Whether for scholarly pursuits, industry use, or personal projects, this approach offers a solid foundation for further exploration and innovation in the field of NLP.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae39b76af0f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynamic Object Detection and Segmentation with YOLOv9+SAM]]></title>
            <link>https://medium.com/@sunidhi.ashtekar/dynamic-object-detection-and-segmentation-with-yolov9-sam-de258238546f?source=rss-1d599ba5668e------2</link>
            <guid isPermaLink="false">https://medium.com/p/de258238546f</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[yolov9]]></category>
            <dc:creator><![CDATA[Sunidhi Ashtekar]]></dc:creator>
            <pubDate>Fri, 15 Mar 2024 00:05:00 GMT</pubDate>
            <atom:updated>2024-03-16T23:39:16.986Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*L_QMiAXS_osW_0XgRZr8PA.png" /><figcaption>Detection and Segmentation with YOLOv9+SAM</figcaption></figure><p>In this article, I have examined a custom object detection model on the RF100 Construction-Safety-2 dataset with YOLOv9+SAM.</p><p>This integration not only elevates the accuracy and granularity of detecting and segmenting objects across varied images but also broadens the scope of applications — from enhancing autonomous driving systems to refining diagnostic processes in medical imaging.</p><p>By leveraging YOLOv9 for its efficient detection capabilities and SAM for its ability to segment objects in a zero-shot manner, this powerful combination minimizes the need for extensive retraining or data annotation, making it a versatile and scalable solution.</p><h4>Introduction to YOLOv9 (You Only Look Once)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*HuSJs50ztkiKHYIdHXoH2Q.png" /><figcaption>YOLOv9 Performance from the <a href="https://github.com/WongKinYiu/yolov9?ref=blog.roboflow.com">official YOLOv9 repository</a>.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ZT5ln9ZRLZoLDHUWjo7cA.png" /><figcaption>YOLOv9 Model Diagram from <a href="https://github.com/WongKinYiu/yolov9?ref=blog.roboflow.com">official YOLOv9 repository</a>.</figcaption></figure><p>YOLOv9 introduces significant advancements in real-time object detection, incorporating Programmable Gradient Information (PGI) and the Generalized Efficient Layer Aggregation Network (GELAN) to enhance efficiency, accuracy, and adaptability, as evidenced by its performance on the MS COCO dataset.</p><p>Leveraging the collaborative work of the open-source community and building upon Ultralytics YOLOv5, YOLOv9 addresses the challenge of information loss in deep learning with the Information Bottleneck Principle and Reversible Functions, preserving essential data across layers.</p><p>These innovative strategies boost the model’s structural efficiency and ensure precise detection capabilities without compromising detail, even in lightweight models.</p><p>YOLOv9’s architecture reduces unnecessary parameters and computational demands, allowing it to perform optimally across various model sizes, from the compact YOLOv9-S to the more extensive YOLOv9-E, showcasing a harmonious balance between speed and detection accuracy.</p><p>As a milestone in computer vision, YOLOv9 not only establishes new benchmarks but also broadens the horizons for AI applications in object detection and segmentation, highlighting the impact of strategic innovation and collaborative efforts in the field.</p><h4>Introduction to <strong>Segment-Anything-Model (SAM)</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GY61FYqj8mi6OxlGshA_7w.png" /><figcaption>SAM Model Diagram from <a href="https://github.com/facebookresearch/segment-anything">official META repository</a></figcaption></figure><p>The Segment Anything Model (SAM) propels computer vision forward by simplifying image segmentation, crucial for a range of uses from scientific studies to creative endeavours.</p><p>SAM leverages the Segment Anything 1-Billion (SA-1B) mask dataset, the largest to date, to democratize segmentation by reducing the reliance on specialized expertise, heavy computational power, and extensive dataset annotations.</p><p>Under the Apache 2.0 license, SAM introduces a foundational model framework, allowing for easy task adaptation via simple prompts, mirroring advancements seen in natural language processing.</p><p>With training on over 1 billion diverse masks, SAM understands a generalized notion of objects, facilitating zero-shot transfer across unfamiliar domains and enhancing its utility across various fields, including AR/VR, creative arts, and scientific research.</p><p>The model’s prompt-driven flexibility and wide-ranging task adaptability signal a significant leap in segmentation technology, positioning SAM as a versatile and accessible tool for both the research community and application developers.</p><h4>About the Dataset</h4><p>The project utilizes the <a href="https://blog.roboflow.com/roboflow-100/">RF100</a> construction dataset from Roboflow, specifically the <a href="https://universe.roboflow.com/roboflow-100/construction-safety-gsnvb/dataset/2">Construction-Safety-2</a> subset, for demonstrating the capabilities of the integrated models.</p><p>RF100 is an initiative sponsored by Intel, aimed at establishing an open-sourced benchmark for object detection models. It focuses on generality across datasets available on Roboflow Universe, promoting accessibility and speeding up the AI and deep learning development process.</p><p>The RF100 construction dataset, like other projects in Roboflow, is open-source and can be freely used.</p><h4>Getting Started</h4><p>To leverage the combined strengths of YOLOv9 and SAM in your project, these are the steps to follow</p><ul><li>Environment Setup</li><li>Download Pre-trained Model Weights for YOLOv9 and SAM</li><li>Inference on Images</li><li>Visualization and Analysis</li><li>Get Detection Results</li><li>Segmentation with SAM</li></ul><h4>Environment Setup</h4><p>A Google account is required to access Google Colab, a free cloud service providing necessary computational resources for deep learning tasks, including access to GPUs and TPUs up to 16 GB.</p><h4>GPU Status Check</h4><p>Firstly we ensure the availability and readiness of the GPU for processing and running the YOLOv9+SAM model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*kUW6b1yh5dorO2ubuRmFaA.png" /></figure><h4>Mounting Google Drive</h4><p>Next, we need to navigate to folder where the dataset is stored if you have downloaded the dataset or else we can directly use Roboflow to load the dataset.</p><pre>from google.colab import drive<br>drive.mount(&#39;/content/drive&#39;)<br><br>or<br><br>%cd {HOME}/<br>!pip install -q roboflow<br><br>from roboflow import Roboflow<br>rf = Roboflow(api_key=&quot;YOUR API KEY&quot;)<br>project = rf.workspace(&quot;roboflow-100&quot;).project(&quot;construction-safety-gsnvb&quot;)<br>dataset = project.version(2).download(&quot;yolov7&quot;)</pre><h4>Setting Up YOLOv9</h4><p>Once the dataset is ready clone the YOLOv9 repository, then switch to the YOLOv9 directory and install the required dependencies to prepare for object detection and segmentation tasks.</p><pre>!git clone https://github.com/SkalskiP/yolov9.git<br>%cd yolov9<br>!pip3 install -r requirements.txt -q</pre><h4>Display Current Directory</h4><p>Store current working directory’s path in the HOME variable for reference.</p><pre>import os<br>HOME = os.getcwd()<br>print(HOME)</pre><h4>Download Model Weights</h4><p>Let’s create a directory for model weights and download specific YOLOv9 and GELAN model weights from their release pages on GitHub, crucial for initializing the models with pre-trained parameters</p><pre>!mkdir -p {HOME}/weights<br>!wget -P {HOME}/weights -q https://github.com/WongKinYiu/yolov9/releases/download/v0.1/yolov9-c.pt<br>!wget -P {HOME}/weights -q https://github.com/WongKinYiu/yolov9/releases/download/v0.1/yolov9-e.pt<br>!wget -P {HOME}/weights -q https://github.com/WongKinYiu/yolov9/releases/download/v0.1/gelan-c.pt<br>!wget -P {HOME}/weights -q https://github.com/WongKinYiu/yolov9/releases/download/v0.1/gelan-e.pt</pre><h4>Download an Image for Inference</h4><p>To obtain inference with YOLOv9 weights we have to set up a data directory and download a sample image for processing and set the path to this image in the SOURCE_IMAGE_PATH variable.</p><pre>!mkdir -p {HOME}/data<br>!wget -P {HOME}/data -q /content/drive/MyDrive/data/image9.jpeg<br>SOURCE_IMAGE_PATH = f&quot;{HOME}/image9.jpeg&quot;</pre><h4>Run Detection with Custom Data</h4><p>After this, execute the detect.py with specified parameters to perform object detection on an image, set a confidence threshold and save the detection results. This will create a text file with class_ids, bounding box co-ordinates and confidence scores that we will utilize later.</p><pre>!python detect.py --weights {HOME}/weights/gelan-c.pt --conf 0.1 --source /content/drive/MyDrive/data/image9.jpeg --device 0 --save-txt --save-conf</pre><h4>Display Detection Results</h4><p>Then we utilize IPython’s display and Image functions to showcase the detected objects within the specified path image, adjusting for optimal viewing.</p><pre>from IPython.display import Image, display<br>Image(filename=f&quot;/content/yolov9/runs/detect/exp/image9.jpeg&quot;, width=600)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OTm3oJl0T3JuHQTxQRHYiQ.jpeg" /></figure><h4>Install Ultralytics</h4><p>Install the Ultralytics package to access YOLO object detection model implementations and utilities, don’t forget to import the YOLO class for object detection tasks.</p><pre>!pip install ultralytics<br>from ultralytics import YOLO</pre><h4>Install Segment-Anything Model</h4><p>Now let’s install the Segment-Anything library and download the weights file for the SAM model and prepare for high-quality image segmentation tasks.</p><pre>!pip install &#39;git+https://github.com/facebookresearch/segment-anything.git&#39;<br><br>!wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth</pre><h4>Extract Detection Results and Confidence Scores</h4><p>We will need the YOLOv9 detection results saved in text file above to extract class IDs, confidence scores, and bounding box coordinates. The co-ordinates here are normalized so let’s convert them to image scale first and then print them to verify.</p><pre>import cv2<br><br># Specify the path to your image<br>image_path = &#39;/content/drive/MyDrive/data/image9.jpeg&#39;<br><br># Read the image to get its dimensions<br>image = cv2.imread(image_path)<br>image_height, image_width, _ = image.shape<br><br>detections_path = &#39;/content/yolov9/runs/detect/exp/labels/image9.txt&#39;<br><br>bboxes = []<br>class_ids = []<br>conf_scores = []<br><br>with open(detections_path, &#39;r&#39;) as file:<br>    for line in file:<br>        components = line.split()<br>        class_id = int(components[0])<br>        confidence = float(components[5])<br>        cx, cy, w, h = [float(x) for x in components[1:5]]<br><br>        # Convert from normalized [0, 1] to image scale<br>        cx *= image_width<br>        cy *= image_height<br>        w *= image_width<br>        h *= image_height<br><br>        # Convert the center x, y, width, and height to xmin, ymin, xmax, ymax<br>        xmin = cx - w / 2<br>        ymin = cy - h / 2<br>        xmax = cx + w / 2<br>        ymax = cy + h / 2<br><br>        class_ids.append(class_id)<br>        bboxes.append((xmin, ymin, xmax, ymax))<br>        conf_scores.append(confidence)<br><br># Display the results<br>for class_id, bbox, conf in zip(class_ids, bboxes, conf_scores):<br>    print(f&#39;Class ID: {class_id}, Confidence: {conf:.2f}, BBox coordinates: {bbox}&#39;)</pre><h4>Initialize SAM for Image Segmentation</h4><p>Once SAM is initialized with specified pre-trained weights we proceed to select the model type from the SAM model registry for generating segmentation masks.</p><pre>from segment_anything import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor<br>sam_checkpoint = &quot;/content/yolov9/sam_vit_h_4b8939.pth&quot;<br>model_type = &quot;vit_h&quot;<br>sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)<br>predictor = SamPredictor(sam)</pre><h4>Load Image for Segmentation</h4><p>With OpenCV library we load the image for processing with SAM, prepare it for segmentation.</p><pre>import cv2<br>image = cv2.cvtColor(cv2.imread(&#39;/content/drive/MyDrive/data/image9.jpeg&#39;), cv2.COLOR_BGR2RGB)<br>predictor.set_image(image)</pre><h4>Visualization of the Results</h4><p>To visualize the detection and segmentation results, we have to convert bounding boxes to segmentation mask using SAM. We assign unique colors to class IDs randomly then define helper functions for displaying masks, confidence scores and bounding boxes. The coco.yaml file used for mapping the class_ids to class names.</p><pre>import matplotlib.patches as patches<br>from matplotlib import pyplot as plt<br>import numpy as np<br>import yaml<br><br>with open(&#39;/content/yolov9/data/coco.yaml&#39;, &#39;r&#39;) as file:<br>    coco_data = yaml.safe_load(file)<br>    class_names = coco_data[&#39;names&#39;]<br><br>for class_id, bbox, conf in zip(class_ids, bboxes, conf_scores):<br>    class_name = class_names[class_id]<br>    # print(f&#39;Class ID: {class_id}, Class Name: {class_name}, BBox coordinates: {bbox}&#39;)<br><br>color_map = {}<br>for class_id in class_ids:<br>    color_map[class_id] = np.concatenate([np.random.random(3), np.array([0.6])], axis=0)<br><br>def show_mask(mask, ax, color):<br>    h, w = mask.shape[-2:]<br>    mask_image = mask.reshape(h, w, 1) * np.array(color).reshape(1, 1, -1)<br>    ax.imshow(mask_image)<br><br>def show_box(box, label, conf_score, color, ax):<br>    x0, y0 = box[0], box[1]<br>    w, h = box[2] - box[0], box[3] - box[1]<br>    rect = plt.Rectangle((x0, y0), w, h, edgecolor=color, facecolor=&#39;none&#39;, lw=2)<br>    ax.add_patch(rect)<br><br>    label_offset = 10<br><br>    # Construct the label with the class name and confidence score<br>    label_text = f&#39;{label} {conf_score:.2f}&#39;<br><br>    ax.text(x0, y0 - label_offset, label_text, color=&#39;black&#39;, fontsize=10, va=&#39;top&#39;, ha=&#39;left&#39;,<br>            bbox=dict(facecolor=color, alpha=0.7, edgecolor=&#39;none&#39;, boxstyle=&#39;square,pad=0.4&#39;))<br><br>plt.figure(figsize=(10, 10))<br>ax = plt.gca()<br>plt.imshow(image)<br><br># Display and process each bounding box with the corresponding mask<br>for class_id, bbox in zip(class_ids, bboxes):<br>    class_name = class_names[class_id]<br>    # print(f&#39;Class ID: {class_id}, Class Name: {class_name}, BBox coordinates: {bbox}&#39;)<br><br>    color = color_map[class_id]<br>    input_box = np.array(bbox)<br><br>    # Generate the mask for the current bounding box<br>    masks, _, _ = predictor.predict(<br>        point_coords=None,<br>        point_labels=None,<br>        box=input_box,<br>        multimask_output=False,<br>    )<br><br>    show_mask(masks[0], ax, color=color)<br>    show_box(bbox, class_name, conf, color, ax)<br><br># Show the final plot<br>plt.axis(&#39;off&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*ujesROelO0Q0Sw7kju8xaw.png" /></figure><h4>Extract the Masks</h4><p>Lastly we generate a composite image that highlights detected objects against a white background, creating an aggregate mask from segmentation masks and applying it to blend the original image with a white background for enhanced visualization.</p><pre>aggregate_mask = np.zeros(image.shape[:2], dtype=np.uint8)<br><br># Generate and accumulate masks for all bounding boxes<br>for bbox in bboxes:<br>    input_box = np.array(bbox).reshape(1, 4)<br>    masks, _, _ = predictor.predict(<br>        point_coords=None,<br>        point_labels=None,<br>        box=input_box,<br>        multimask_output=False,<br>    )<br>    aggregate_mask = np.where(masks[0] &gt; 0.5, 1, aggregate_mask)<br><br># Convert the aggregate segmentation mask to a binary mask<br>binary_mask = np.where(aggregate_mask == 1, 1, 0)<br><br># Create a white background with the same size as the image<br>white_background = np.ones_like(image) * 255<br><br># Apply the binary mask to the original image<br># Where the binary mask is 0 (background), use white_background; otherwise, use the original image<br>new_image = white_background * (1 - binary_mask[..., np.newaxis]) + image * binary_mask[..., np.newaxis]<br><br># Display the new image with the detections and white background<br>plt.figure(figsize=(10, 10))<br>plt.imshow(new_image.astype(np.uint8))<br>plt.axis(&#39;off&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*EaEU1DDgAduZZjVR4pnpoQ.png" /></figure><h4>Conclusion</h4><p>I hope this guide has been clear and that you’ve successfully managed to train your model. By leveraging the RF100 construction dataset from Roboflow, we saw how to use the models to recognize and segment objects with the custom dataset, demonstrating the practical applications of combining YOLOv9 and SAM for advanced computer vision tasks.</p><p>If you have any questions, recommendations, or critiques, please don’t hesitate to reach out on LinkedIn. I’m open to discussions and eager to hear your feedback or assist with any challenges you might encounter.</p><h4>References</h4><p>To further explore the concepts and tools used in this project, I recommend visiting the following resources:</p><ul><li><a href="https://github.com/suniash/DynamicDetect_YOLOv9_SAM">DynamicDetect YOLOv9+SAM Git Repository</a></li><li>YOLOv9 Official Repository: <a href="https://github.com/WongKinYiu/yolov9">https://github.com/WongKinYiu/yolov9</a></li><li>SAM Official Repository: <a href="https://github.com/facebookresearch/segment-anything">https://github.com/facebookresearch/segment-anything</a></li><li>Roboflow Blog: A great resource for the latest updates and tutorials on using Roboflow for computer vision projects. Check out their blog for insightful articles.</li><li><a href="https://blog.roboflow.com/how-to-use-yolov8-with-sam/">https://blog.roboflow.com/how-to-use-yolov8-with-sam/</a></li><li><a href="https://blog.roboflow.com/train-yolov9-model/">https://blog.roboflow.com/train-yolov9-model/</a></li></ul><p>This project stands as a testament to the potential of combining cutting-edge detection and segmentation technologies to push the boundaries of what’s achievable in computer vision. Whether for academic research, industry applications, or personal projects, the integration of YOLOv9 and SAM, supported by the comprehensive RF100 datasets from Roboflow, offers a robust framework for exploring and innovating within the vast domain of computer vision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de258238546f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>