<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Joe Hoeller on Medium]]></title>
        <description><![CDATA[Stories by Joe Hoeller on Medium]]></description>
        <link>https://medium.com/@joehoeller?source=rss-6a6c7dbe9fe5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*jpwd0ljrWSV01n9p.jpg</url>
            <title>Stories by Joe Hoeller on Medium</title>
            <link>https://medium.com/@joehoeller?source=rss-6a6c7dbe9fe5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 05:34:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@joehoeller/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Optimizing Satellite Constellations with Graph Neural Networks]]></title>
            <link>https://medium.com/@joehoeller/optimizing-satellite-constellations-with-graph-neural-networks-6ce87d50a29f?source=rss-6a6c7dbe9fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ce87d50a29f</guid>
            <dc:creator><![CDATA[Joe Hoeller]]></dc:creator>
            <pubDate>Thu, 21 Nov 2024 21:03:34 GMT</pubDate>
            <atom:updated>2024-11-23T18:44:21.014Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b32-xLJ45wOpk32BLmMT2w.png" /><figcaption>Image credit Air Force and the National Reconnaissance Office</figcaption></figure><h3>Explore the Codebase</h3><p>Check out the <a href="#">GitHub repository</a> for the full implementation and more details on how to apply GNNs to satellite constellation optimization.</p><ul><li>Codebase on GitHub, <a href="https://github.com/daddydrac/Combinatorial-Optimization-and-Reasoning-for-GNNs"><strong><em>here</em></strong></a>.</li><li>Adapted from this <a href="https://www.ijcai.org/proceedings/2021/0595.pdf"><strong><em>Conference Paper</em></strong></a></li><li>Math/LaTeX explanation <a href="https://github.com/daddydrac/Combinatorial-Optimization-and-Reasoning-for-GNNs/blob/main/math_explained.ipynb"><strong><em>here</em></strong></a></li></ul><p>Managing satellite constellations like <strong>Starlink</strong> and <strong>OneWeb</strong> is no small feat. These networks comprise thousands of interconnected satellites, creating a complex communication web that needs to function in real-time. The challenges? Reducing latency, maintaining orbital stability with efficient fuel use, and predicting potential network disruptions from satellite or link failures. Given the scale and constant evolution of these networks, traditional optimization methods just don’t cut it anymore.</p><h3>Modeling Satellite Constellations as Dynamic Graphs</h3><p>Satellite constellations are naturally represented as dynamic graphs where:</p><ul><li><strong>Nodes</strong> represent satellites, with attributes like position, velocity, remaining fuel, operational status, load, and energy availability.</li><li><strong>Edges</strong> represent communication links, capturing distance, signal strength, bandwidth, and latency.</li></ul><p>This graph-based representation enables Graph Neural Networks (GNNs) to process satellite data effectively, uncovering patterns and relationships crucial for optimizing the network.</p><h3>Why Use GNNs?</h3><p>GNNs are a natural fit for solving these challenges because:</p><ol><li><strong>Graph-Friendly Design:</strong> They respect graph properties like permutation invariance, ensuring consistent outputs regardless of how nodes or edges are ordered.</li><li><strong>Scalability:</strong> GNNs efficiently aggregate local information, making them ideal for sparse, large-scale graphs.</li><li><strong>Dynamic Adaptability:</strong> They adjust seamlessly to evolving network configurations, allowing for real-time optimization.</li></ol><p>By leveraging satellite data such as position, velocity, and signal strength, GNNs can optimize routes, predict failures, and even propose fuel-saving orbital adjustments.</p><h3>GNN Architecture: Encode-Process-Decode</h3><ol><li><strong>Encoder:</strong> Converts raw node and edge features into high-dimensional embeddings.</li><li><strong>Processor:</strong> Uses iterative message-passing, inspired by algorithms like Bellman-Ford, to propagate relational patterns across the graph.</li><li><strong>Decoder:</strong> Generates actionable outputs, like optimal routes or fuel-efficient repositioning strategies.</li></ol><p>To boost performance, GNNs integrate with reinforcement learning (RL) frameworks like <strong>Actor-Critic</strong>, where:</p><ul><li>The <strong>actor</strong> proposes actions (e.g., route selection).</li><li>The <strong>critic</strong> evaluates the quality of those actions, enabling continuous improvement.</li></ul><h3>Applications in Satellite Networks</h3><ol><li><strong>Finding Feasible Solutions:</strong> GNNs can generate routing policies from scratch or enhance existing solvers. Using RL, they balance competing objectives like latency reduction and fuel conservation.</li><li><strong>Proving Optimality:</strong> GNNs guide algorithms like branch-and-bound to certify solution optimality or identify infeasible scenarios.</li><li><strong>Algorithmic Reasoning:</strong> By mimicking classical algorithms (e.g., Bellman-Ford), GNNs generalize to new problems and compute shortest paths efficiently.</li></ol><h3>Key Advantages and Performance Metrics</h3><p><strong>Advantages:</strong></p><ul><li><strong>Dynamic Adaptability:</strong> Handles evolving satellite configurations.</li><li><strong>Scalability:</strong> Processes massive graphs with localized computation.</li><li><strong>Predictive Power:</strong> Learns from historical telemetry to anticipate failures.</li><li><strong>Fuel Optimization:</strong> Identifies minimal adjustment strategies to extend satellite lifespan.</li></ul><p><strong>Metrics:</strong></p><ul><li><strong>Latency Reduction:</strong> Lower communication delays.</li><li><strong>Failure Prediction Accuracy:</strong> Higher true positive rates for disruption detection.</li><li><strong>Fuel Efficiency:</strong> Reduced consumption during orbital adjustments.</li></ul><h3>For Developers: A Deep Dive into the Code</h3><h4>Bellman-Ford GNN Class</h4><p>Implements message-passing inspired by the Bellman-Ford algorithm, ideal for shortest path problems. Features include:</p><ul><li><strong>Message Computation:</strong> Combines node and edge features to generate messages.</li><li><strong>Node Updates:</strong> Aggregates messages to refine node features.</li></ul><h4>Encode-Process-Decode Architecture</h4><p>Processes graph data in three steps:</p><ol><li><strong>Encode:</strong> Transforms raw features into embeddings.</li><li><strong>Process:</strong> Refines embeddings using the Bellman-Ford-inspired GNN.</li><li><strong>Decode:</strong> Outputs predictions (e.g., shortest paths) using a linear layer.</li></ol><h4>Actor-Critic Framework</h4><p>Designed for RL, this framework includes:</p><ul><li><strong>Actor Network:</strong> Generates policies to guide decisions.</li><li><strong>Critic Network:</strong> Evaluates the quality of those decisions.</li></ul><h3>Training Functions</h3><h4>Supervised Training</h4><p>Trains the GNN using labeled data:</p><ul><li>Computes predictions via forward passes.</li><li>Optimizes the model using binary cross-entropy loss.</li><li>Tracks metrics like accuracy and F1 scores for performance evaluation.</li></ul><h4>Reinforcement Learning Training</h4><p>Trains the Actor-Critic model:</p><ul><li>Interacts with a simulated environment to gather state, action, and reward data.</li><li>Computes <strong>policy loss</strong> (using log probabilities) and <strong>value loss</strong> (using mean squared error).</li><li>Updates model parameters based on advantages computed from returns.</li></ul><h3>Utility and Debugging Tools</h3><ul><li><strong>compute_returns:</strong> Calculates discounted rewards for RL.</li><li><strong>flatten_features:</strong> Prepares structured data for tensor creation.</li><li><strong>parse_graph_data:</strong> Converts JSON graph data into PyTorch objects.</li><li><strong>create_dataloader:</strong> Enables efficient batch training.</li></ul><p>A placeholder environment (<strong>RLEnv</strong>) is included for debugging RL workflows.</p><h3>Running the Pipeline</h3><p><strong>Supervised Training:</strong></p><ul><li>Load data from generated_samples.json.</li><li>Train the GNN for routing and optimization tasks.</li><li>Save the trained model as gnn_model.pth.</li></ul><p><strong>Reinforcement Learning:</strong></p><ul><li>Initialize the Actor-Critic model.</li><li>Train it in an RL environment for dynamic decision-making.</li><li>Save the RL model as actor_critic_model.pth.</li></ul><blockquote>Finale!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ce87d50a29f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to Taxonomy, to Thesaurus, to Ontology, to Knowledge Graph]]></title>
            <link>https://medium.com/@joehoeller/intro-to-taxonomy-to-thesaurus-to-ontology-to-knowledge-graph-0b3546d8b38c?source=rss-6a6c7dbe9fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b3546d8b38c</guid>
            <category><![CDATA[taxonomy]]></category>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Joe Hoeller]]></dc:creator>
            <pubDate>Wed, 26 Jun 2024 14:48:56 GMT</pubDate>
            <atom:updated>2024-06-26T14:48:56.824Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello wonderful people, today, we’re exploring the intricate evolution from taxonomies to thesauri, ontologies, and finally, knowledge graphs. Each stage in this progression adds layers of sophistication to how we structure and understand data, ultimately empowering us to derive more actionable insights. For a seasoned executive, understanding these transitions can provide a strategic advantage in leveraging data-driven decision-making.</p><h3>Start with Your Use Case</h3><p>Before diving into modeling, it’s imperative to identify your use case. For instance, consider a scenario where a manufacturer needs to understand the components and parts of their vehicles. This use case will guide the structuring and refinement of our models.</p><h3>Stage 1: Taxonomy</h3><p>A taxonomy organizes data into hierarchical relationships such as broader and narrower terms or parent-child relationships. For example, in a vehicle manufacturer scenario, vehicles might be categorized into passenger and sports vehicles, with specific models like Mustangs and Raptors falling under these categories. Taxonomies help in identifying and grouping related items, providing a foundational structure that supports robust data categorization.</p><h3>Stage 2: Thesaurus</h3><p>Building on the taxonomy, a thesaurus adds synonyms, alternative terms, and “see also” relationships, enhancing the context and connectivity of your data. This stage enriches the metadata, making it easier for both humans and machines to understand and navigate the data. For instance, the term “electric motor” in your taxonomy can be linked with synonyms and related terms from different databases or user inputs, enhancing search capabilities and semantic understanding.</p><h3>Stage 3: Ontology</h3><p>Ontologies take the thesaurus to the next level by explicitly defining the relationships between entities and their attributes. Unlike taxonomies and thesauri, which are more about categorization and synonyms, ontologies define the nature of relationships (e.g., “has part,” “manufactured in”). This stage involves identifying universal categories and refining them to answer specific questions. For example, if tracking vehicles, you might center your ontology around the vehicle identification number (VIN), linking it to parts, manufacturing locations, and transactions.</p><h3>Deep Insight: Enhanced Predictive Analytics</h3><p>A crucial insight at this stage is the potential for enhanced predictive analytics. By defining detailed relationships and attributes, an ontology allows for more accurate predictive models. For instance, understanding the relationship between specific vehicle parts and failure rates can help in predictive maintenance. This is actionable as it directly impacts operational efficiency and cost management.</p><h3>Stage 4: Knowledge Graph</h3><p>A knowledge graph populates the ontology with instance data, creating a rich, interconnected dataset. This graph can include inferred relationships and additional nodes that flesh out the network. Using your ontology, you can link specific electric motors to vehicles, manufacturing plants, and supply chains, revealing hidden connections and enabling advanced data analysis. This stage supports sophisticated queries and semantic search, making it invaluable for large-scale data integration and retrieval.</p><h3>Actionable Insight: Real-Time Data Integration</h3><p>For a business executive, the ability to integrate real-time data streams into a knowledge graph can revolutionize data-driven strategies. By continuously updating the knowledge graph with live data from IoT devices, social media, or transactional systems, you can achieve real-time insights into user behavior, operational anomalies, and market trends. This integration is not just a technical enhancement but a strategic move to stay ahead in dynamic environments.</p><h3>Key Considerations</h3><ol><li><strong>Metadata and Relationships</strong>: Each stage adds more metadata and relationships, enhancing data context and usability.</li><li><strong>Refinement and Validation</strong>: Continuously refine your model, checking for logical consistency and alignment with business needs.</li><li><strong>Handling Instances</strong>: Differentiate between universal concepts (e.g., “cat”) and specific instances (e.g., “my cat Garfield”) to maintain clarity.</li><li><strong>Avoid Orphan Nodes</strong>: Ensure all nodes are connected to avoid data silos and technical debt. This is also known as entity resolution.</li></ol><h3>Advanced Tools</h3><p>For those looking to dive deeper, RDF* (RDF star) and SHACL (Shapes Constraint Language) are powerful tools. RDF* extends RDF’s expressiveness by embedding triples within triples, useful for adding metadata like provenance or temporal information. SHACL provides a robust mechanism to validate RDF graphs, ensuring data quality and conformance to expected structures.</p><h3>Conclusion</h3><p>By understanding and implementing these evolutionary stages, you can build robust, context-rich data structures that support advanced data analysis and strategic decision-making. Whether you stop at a taxonomy or evolve to a knowledge graph, each stage adds incremental value. Leveraging tools like RDF* and SHACL further enhances the precision and usability of your data models.</p><p>For a deeper dive into each stage, check out the linked resources and previous discussions. Should you have any questions or require further insights, feel free to reach out on LinkedIn.</p><p><strong>Thank you for your time, and let’s continue driving innovation through sophisticated data modeling!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b3546d8b38c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Graph Types and Ontological-Driven Data Structures]]></title>
            <link>https://medium.com/@joehoeller/understanding-graph-types-and-ontological-driven-data-structures-185eb176c3c3?source=rss-6a6c7dbe9fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/185eb176c3c3</guid>
            <category><![CDATA[semanticweb]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <dc:creator><![CDATA[Joe Hoeller]]></dc:creator>
            <pubDate>Wed, 26 Jun 2024 14:29:15 GMT</pubDate>
            <atom:updated>2025-04-20T20:43:40.881Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello wonderful people, I’ve noticed some common misconceptions about graphs and AI circulating on LinkedIn, and I thought it might be helpful to share some insights to clarify these topics. I hope you find this information enlightening and that it assists you in effectively building complex AI systems with enhanced reasoning capabilities.</p><p>Graphs are essential in various domains, ranging from computer science to bioinformatics. However, distinguishing between different types of graphs and understanding their unique properties and applications is crucial. This article aims to clarify these distinctions by focusing on Directed Acyclic Graphs (DAGs), Label Property Graphs (LPGs), Bipartate Graphs, and Semantic Knowledge Graphs, which are ontologically-driven data structures, particularly in the context of spatial reasoning.</p><h3>Directed Acyclic Graphs (DAGs) are Flowcharts (essentially)</h3><p>A flowchart is a type of diagram that represents a process or workflow, often visualized as a Directed Acyclic Graph (DAG). A DAG is a graph that is directed and contains no cycles, meaning there is a one-way path from one node to another without any loops, hence <em>“acyclical”</em>. This makes DAGs suitable for representing processes with clear linear progression, such as tasks in a workflow or steps in a computational process.</p><p>Flowcharts are a common way to visualize DAGs, but it’s important to note that DAGs themselves are not knowledge graphs. DAGs are utilized in various algorithms, such as topological sorting and combinatorial enumeration, to solve problems in scheduling, steps in data processing, and other similar tasks. The primary focus of DAGs is on the order and dependency of tasks, not on semantic relationships or high-level reasoning.</p><h3>Semantic Knowledge Graphs</h3><p>Semantic knowledge graphs, on the other hand, are designed to represent knowledge in a way that enables reasoning and inference. These graphs contain nodes representing entities and edges representing relationships, enriched with semantic information. This allows for high-level queries and reasoning that go beyond simple data retrieval.</p><p>A key feature of semantic knowledge graphs is the use of data predicates, which are elements that describe the relationships between entities. These predicates enable the graph to connect detailed information to broader concepts, allowing for sophisticated queries and inference. Unlike DAGs, semantic knowledge graphs employ ontologies, which are structured frameworks for organizing information. Ontologies define the types of entities, relationships, and rules for reasoning, and connect data points across domains and inter-connect within sub-domains, facilitating a deeper understanding of the data.</p><h3>Label Property Graphs (LPGs)</h3><p>Are a data structure that represents data and it’s relationships. In LPGs, both nodes and edges can hold various properties, enabling a detailed depiction of entities and their interconnections within the network. This attribute-rich structure not only allows for straightforward queries but also supports complex analyses, think SQL on steroids, making LPGs ideal for applications requiring an in-depth exploration of data relationships. (Think of it as the same problem in SQL, where you cannot “scale with knowledge” or expand the knowledge base without predefined primary keys [PKs] and foreign keys [FKs], as required in SQL.) Despite their robustness, the representation capabilities of LPGs are bound by their structure, which can pose challenges in scenarios that require representing beyond simple binary relationships, which does not form the basis for contextual reasoning.</p><h3>Bipartite Graphs</h3><p>Bipartite graphs are a special type of graph where the set of vertices can be divided into two disjoint subsets such that no two vertices within the same subset are adjacent. This means that every edge in the graph connects a vertex in one subset to a vertex in the other subset. Bipartite graphs are particularly useful in modeling relationships between two different classes of objects, such as in job assignments, where one subset represents workers and the other subset represents tasks, with edges indicating which worker is assigned to which task. These graphs are crucial in solving matching problems, optimizing network flows, and are foundational in the design of algorithms for bipartite matching, such as the <a href="https://www.geeksforgeeks.org/hopcroft-karp-algorithm-for-maximum-matching-set-1-introduction/">Hopcroft-Karp algorithm</a>. The structure of bipartite graphs simplifies many computational problems, making them a powerful tool in many practical applications.</p><h3>RDF (Resource Description Framework) Triples</h3><p>RDF Triples, although not a graph, are crucial elements of the Resource Description Framework, play a key role in structuring data for the Semantic Web. Structured as a subject, predicate, and object, each RDF Triple articulates a specific statement about a resource, linking data in a meaningful way. The subject represents the resource in question, the predicate denotes the type of relationship that connects the subject to the object, and the object itself can be either another resource or a literal value. This structured approach enables the creation of ontologies — formal representations of knowledge within a domain. RDF Triples thus provide the foundation for constructing ontologies by defining and interlinking concepts through precise, semantic relationships, facilitating sophisticated data retrieval and reasoning across diverse information systems or domains — think of it as being able to connect information across several Subject Matter Experts (SME’s) at once.</p><ol><li><strong>Semantic Relationships</strong>: RDF triples express relationships using URIs (Uniform Resource Identifiers) that are globally unique, which helps in defining clear and unambiguous connections between different data items. This global identification system ensures that the same term used in different datasets refers back to the same concept, making it easier to merge and query data across various sources.</li><li><strong>Flexible Schema</strong>: RDF does not require a fixed schema before data is created or integrated. This flexibility allows RDF to adapt to various domains and integrate new types of data without restructuring existing databases. This adaptability is crucial for connecting data points that originate in different systems and structures.</li><li><strong>Standardized Query Language (SPARQL)</strong>: RDF data can be queried using SPARQL, a powerful and standardized query language specifically designed to retrieve and manipulate data stored in RDF format. SPARQL can perform sophisticated queries that span multiple data sources, even if those sources are spread across the internet, thus facilitating complex reasoning and analysis across domains.</li><li><strong>Ontologies Provide Inference</strong>: RDF is often used in conjunction with ontologies (formalized vocabularies of domain knowledge). Ontologies define structured relationships and hierarchies between concepts which can be used for logical inference. Using RDF in ontologies allows systems to infer additional information from the explicit data, enabling more profound contextual understanding and reasoning. For instance, if an ontology states that “all doctors are healthcare professionals,” and an RDF triple states that “Jane is a doctor,” a system can infer that “Jane is a healthcare professional” even if this is not explicitly stated.</li><li><strong>Interoperability and Integration</strong>: RDF’s framework is designed to support interoperability, making it easier to connect and integrate data from diverse domains. The RDF format ensures that data can be understood and utilized in the same way across different systems, reducing the barriers typically encountered in cross-domain data usage.</li></ol><p>If you want to dive into the very basics of ontology creation, check my other article <a href="https://medium.com/@joehoeller/intro-to-taxonomy-to-thesaurus-to-ontology-to-knowledge-graph-0b3546d8b38c"><em>here</em></a>.</p><h3>Unlocking Graph Structures: The Mathematical Formulas That Map Graph Networks to it’s Purpose</h3><figure><img alt="Linear Processes for DAGs:Vertices and Edges with no cycles: G=(V,E) Topological Sort: V1 ,V2 ,…,Vn where for each (Vi ,Vj )∈E,i&lt;j Linear relationship: This denotes a linear, directed path suitable for representing processes and workflows without cycles. Ontologies in Semantic Knowledge Graphs: Triples representing relationships: T = \{(s, p, o) mid s, p, o \in URI \text{ or } Literal } T={(s,p,o)∣s,p,o∈URI or Literal} Ontologies connecting data points across domains: O={C,R} C are concepts and" src="https://cdn-images-1.medium.com/max/1024/1*l46vAnkr0ZQUDUpUOQ9hqA.png" /></figure><h3>Key Formulas for Contextual Reasoning (or not)</h3><h3><strong>Linear Processes for DAGs</strong></h3><ol><li>Vertices and Edges with no cycles:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*W4lDS7tDRbPAlbROZyrJDA.png" /></figure><p>2. Topological Sort:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*awNb-VJgVIt8OIKpbuhaKw.png" /></figure><p>3. Linear relationship:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*4nTAjuB7oAunpKz81r4XsA.png" /></figure><h3>Ontology driven Semantic Knowledge Graphs</h3><ol><li>Triples representing relationships:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*UFUOKq7dMkz-fja4PpTEiQ.png" /></figure><p>2. Ontologies connecting data points across domains:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*vaPvtuZbjDHGCUwAGiMUnQ.png" /></figure><p>Where <em>C</em> are concepts and <em>R</em> are relationships. These ontologies enable complex, interconnected data representations, supporting advanced semantic queries and cross-domain reasoning.</p><h3>Label Property Graphs (LPG’s) are bonded by binary relationships</h3><ol><li>Vertices and edges with properties:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GOvQNm-a4dR8sQRmA4wAnQ.png" /></figure><p>2. Edges with properties:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9uIozDRibZTdyavrVtL04g.png" /></figure><p>This represents linear, multiple binary relationships, suitable for applications like Named Entity Recognition (NER) models, where each entity and relationship has specific properties but does not support complex, nor connect disparate data points across domains for contextual semantic reasoning.</p><h3>Maximum Matching in Bipartite Graphs</h3><ol><li>Maximum matching:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RRFg4KpZpn4bAYUVOtGVpw.png" /></figure><p>This represents the largest possible set of edges where no two edges share a common vertex, which is critical for optimizing assignments for “<em>best fit</em>” and/or resource allocation.</p><p><strong>Takeaway: </strong>It’s crucial to approach these concepts with an understanding of their distinctions. Graph algorithms, such as those used for DAGs, bipartite graphs, and others, are well-established in graph theory. Each graph type has its specific mathematical properties and formulas, which include combinatorial enumeration and topological ordering. In the case of a semantic knowledge graph, there are additional elements like data predicates that enable high-level reasoning and connect to detailed solutions or end states.</p><p>While graph representations of workflows and Named Entity Recognition (NER) models are valuable and have specific use cases, it is essential to recognize their correct implementation and application. Misunderstanding these concepts can lead to incorrect conclusions about their functions and purposes.</p><p><em>I hope this helps you on your journey in AI leveraging graphs, thanks for reading!”</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=185eb176c3c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DevOps for Data Scientists: Learn to do Rolling Updates, Blue-Greens & Canaries on Google…]]></title>
            <link>https://medium.com/@joehoeller/devops-for-data-scientists-learn-to-do-rolling-updates-blue-greens-canaries-on-google-752d222642de?source=rss-6a6c7dbe9fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/752d222642de</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-kubernetes-engine]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Joe Hoeller]]></dc:creator>
            <pubDate>Fri, 07 Feb 2020 15:22:49 GMT</pubDate>
            <atom:updated>2020-02-07T15:28:55.144Z</atom:updated>
            <content:encoded><![CDATA[<h3>DevOps for Data Scientists: Learn to do Rolling Updates, Blue-Greens &amp; Canaries on Google Kubernetes Engine right from the Command Line</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*eTtNdTZSXl7CurVYe4KLsQ.png" /></figure><h3>Summary</h3><p>This POC deploys an auth service running NGINX on https that is prebuilt for you on GitHub. This POC walk-thru shows you how to recreate the commands to use ReplicaSet in Kubernetes. Which is meant for managing containers in production, this also goes over how to do Blue-Green and Canary deployments as well.</p><ul><li>Modify deployments to tigger rolling updates</li><li>Pause and resume an active rolling update</li><li>Rollback a deployment to a previous revision</li></ul><h3>Prerequisites</h3><p>Make sure the following APIs are enabled in Cloud Platform Console:</p><ul><li>Kubernetes Engine API</li><li>Container Registry API</li></ul><h3>Details</h3><p>You can list the active account name with this command:</p><pre>gcloud auth list</pre><p>Output:</p><pre>Credentialed accounts: - &lt;myaccount&gt;@&lt;mydomain&gt;.com (active)</pre><p>You can list the project ID with this command:</p><pre>gcloud config list project</pre><p>Define your zone as a project default zone. This way you do not need to specify — zone parameter in gcloud commands.</p><pre>gcloud config set compute/zone us-central1-a</pre><p>Download the boilerplate code for creating and running containers and deployments, all the .yaml files with replicas, rollouts and blue green deploys is made for you on a simple auth app running on NGINX:</p><pre>git clone <a href="https://github.com/googlecodelabs/orchestrate-with-kubernetes.git">https://github.com/googlecodelabs/orchestrate-with-kubernetes.git</a></pre><p>Start your Kubernetes cluster with 5 nodes.</p><pre>cd orchestrate-with-kubernetes/kubernetes</pre><p>In Cloud Shell, run the following command to start a Kubernetes cluster called &lt;your_project_name&gt; that runs 5 nodes.</p><pre>gcloud container clusters create &lt;your_project_name&gt; — num-nodes 5 — scopes “<a href="https://www.googleapis.com/auth/projecthosting,storage-rw">https://www.googleapis.com/auth/projecthosting,storage-rw</a>&quot;</pre><p>Finally, the scopes argument provides access to project hosting and Google Cloud Storage APIs that you’ll use later. In order to create a cluster as Kubernetes Engine provisions virtual machines for you, go grab a coffee now and come back to your desk in several minutes. It spins up one or more master nodes and multiple configured worker nodes.</p><h3>Deployment Objects</h3><p>Run the explain command in kubectl to tell you about the deployment object ( — recursive is optional if you want to see all the fields).</p><p>kubectl explain deployment — recursive</p><p>Use the explain command as you go through these steps to help you understand the structure of a deployment object and understand what the individual fields do.</p><p>kubectl explain deployment.metadata.name</p><h3>Create a Deployment</h3><p>Examine the deployment configuration file.</p><p>kubectl create will create the auth deployment with one replica, using version 1.0.0 of the auth container. To scale the number of pods, you simply change the replicas field.</p><pre>cat deployments/auth.yaml</pre><p>Create the deployment object using kubectl create.</p><pre>kubectl create -f deployments/auth.yaml</pre><p>Verify that it was created.</p><pre>kubectl get deployments</pre><p>Kubernetes creates a ReplicaSet for the deployment. Run the following command to verify it. You should see a ReplicaSet with a name like auth-xxxxxxx.</p><pre>kubectl get replicasets</pre><p>Run the following command to view the pods created for your deployment. A single pod was created when the ReplicaSet was created.</p><pre>kubectl get pods</pre><p>With your pod running, it’s time to put it behind a service. Use the kubectl create command to create the auth service.</p><pre>kubectl create -f services/auth.yaml</pre><p>Do the same to create and expose the hello and frontend deployments. This will create a ConfigMap and secret for the frontend.</p><pre>kubectl create -f deployments/hello.yaml kubectl create -f services/hello.yaml</pre><pre>kubectl create configmap nginx-frontend-conf — from-file=nginx/frontend.conf kubectl create secret generic tls-certs — from-file tls/ kubectl create -f deployments/frontend.yaml kubectl create -f services/frontend.yaml</pre><p>Interact with the frontend. You may need to re-run this command every few seconds until the External IP is populated.</p><pre>kubectl get services frontend</pre><p>CURL the service:</p><pre>curl -ks https://&lt;EXTERNAL-IP&gt;curl -ks <a href="https://`kubectl/">https://`kubectl</a> get svc frontend -o=jsonpath=”{.status.loadBalancer.ingress[0].ip}”`</pre><h3>Scale a Deployment</h3><p>Update the spec.replicas field to scale the deployment. Run the kubectl explain command to see an explanation of the field.</p><pre>kubectl explain deployment.spec.replicas</pre><p>You can update the replicas field most easily using the kubectl scale command. It may take a minute or so for all the new pods to start up.</p><pre>kubectl scale deployment hello — replicas=5</pre><p>Kubernetes updates the ReplicaSet and starts new pods to equal 5. Verify there are 5 pods running.</p><pre>kubectl get pods | grep hello- | wc -l</pre><p>Scale back the application by 2 pods.</p><pre>kubectl scale deployment hello — replicas=3</pre><p>Verify the correct number of pods.</p><pre>kubectl get pods | grep hello- | wc -l</pre><h3>Rolling Updates</h3><p>Deployments update images to new versions through rolling updates. When a deployment is updated with a new version, it creates a new ReplicaSet and slowly increases the number of replicas in the new ReplicaSet as it decreases the replicas in the old ReplicaSet.</p><h3>Trigger a Rolling Update</h3><p>Run the following command to update your deployment.</p><pre>kubectl edit deployment hello</pre><p>Change the image in containers section to the following, then save and exit.</p><p>containers: — name: hello image: kelseyhightower/hello:2.0.0</p><p>The editor uses vi commands:</p><ol><li>Use arrow keys to hover over version number 1</li><li>Type r to replace it, and enter 2</li><li>Type :wq! and hit Enter to write and quit the file.</li></ol><p>If you have difficulty and are in a class, ask your instructor for help.</p><p>The updated deployment is saved to your cluster and Kubernetes begins a rolling update.</p><p>You can see the new ReplicaSet that Kubernetes creates. If you fail to see a new ReplicaSet, make sure you changed the image in containers, and not one of the other references in labels.</p><pre>kubectl get replicaset</pre><p>View the new entry in the rollout history.</p><pre>kubectl rollout history deployment/hello</pre><h3>Pause a Rolling Update</h3><p>Pause the update:</p><pre>kubectl rollout pause deployment/hello</pre><p>Verify the current state of the rollout.</p><pre>kubectl rollout status deployment/hello</pre><p>Verify this with the pods.</p><pre>kubectl get pods -o jsonpath — template=’{range .items[*]}{.<a href="http://metadata.name/">metadata.name</a>}{“\t”}{“\t”}{.spec.containers[0].image}{“\n”}{end}’</pre><h3>Resume a Rolling Update</h3><p>The rollout is paused which means that some pods are at the new version and some pods are at the older version.</p><p>Use the resume command to continue the rollout.</p><pre>kubectl rollout resume deployment/hello</pre><p>Run the status command to verify the rollout is complete.</p><pre>kubectl rollout status deployment/hello</pre><h3>Rollback an Update</h3><p>If a bug occurs in your new version, users connected to new pods will experience the issue. Use the rollout undo command to roll back to the previous version, then fix any bugs.</p><pre>kubectl rollout undo deployment/hello</pre><p>Verify the rollback in the deployment’s history.</p><pre>kubectl rollout history deployment/hello</pre><h3>Canary Deployments</h3><p>Run a canary deployment to test a new deployment in production with a subset of users. This mitigates risk with new releases.</p><h3>Create a Canary Deployment</h3><p>A canary deployment consists of a separate deployment from your stable deployment and a service that targets them both at the same time.</p><p>Examine the file that creates a canary deployment for your new version.</p><pre>cat deployments/hello-canary.yaml</pre><p>It includes the following:</p><ul><li>the deployment hello-canary</li><li>1 pod (replica)</li><li>selectors app: hello and track: canary</li><li>an image with version 2.0.0.</li></ul><p>Create the canary deployment.</p><pre>kubectl create -f deployments/hello-canary.yaml</pre><p>After the canary deployment is created, verify you have two deployments hello and hello-canary. The hello service selector uses app: hello, which matches pods in both deployments. However, the canary deployment has fewer pods, and is only used by a subset of users.</p><pre>kubectl get deployments</pre><p>Verify both hello versions being served by requests. Run the command several times and confirm that hello 1.0.0 serves about ¾ (75%) of requests and 2.0.0 serves about ¼ (25%).</p><pre>curl -ks <a href="https://`kubectl/">https://`kubectl</a> get svc frontend -o=jsonpath=&quot;</pre><pre>{.status.loadBalancer.ingress[0].ip}&quot;`/version</pre><p>By default, every request has a chance to be served by the canary deployment. If you want users to get all their responses from the same version, enable session affinity in the configuration file as follows:</p><p>spec:</p><p>sessionAffinity: ClientIP</p><h3>Clean Up</h3><p>Delete it and the service as follows.</p><pre>kubectl delete deployment hello-canary</pre><p>That finishes the portion of the POC with canary deployments and how to test new versions of an application in a live environment.</p><h3>Blue-Green Deployments</h3><p>You use the existing hello deployment for the blue version and a new hello-green deployment for the green version.</p><p>Deployments have the following label:</p><pre>Deployment</pre><pre>Label Name</pre><pre>Label Value</pre><pre>hello (blue)</pre><pre>version</pre><pre>1.0.0</pre><pre>hello-green</pre><pre>version</pre><pre>2.0.0</pre><p>You use two nearly-identical service files (hello-blue and hello-green) to switch between versions. The only difference between these files is their version selector. You could edit the service while it&#39;s running and change the version selector, but switching files is easier for labs.</p><p>First, update the service to use the blue deployment:</p><pre>kubectl apply -f services/hello-blue.yaml</pre><h3>Create a Blue-Green Deployment</h3><p>Create the green deployment.</p><pre>kubectl create -f deployments/hello-green.yaml</pre><p>Verify the blue deployment (1.0.0) is still being used.</p><pre>curl -ks <a href="https://`kubectl/">https://`kubectl</a> get svc frontend -o=jsonpath=&quot;{.status.loadBalancer.ingress[0].ip}&quot;`/version</pre><p>Run the following command to update the service to use the green deployment.</p><pre>kubectl apply -f services/hello-green.yaml</pre><p>Verify the green deployment is being used.</p><pre>curl -ks <a href="https://`kubectl/">https://`kubectl</a> get svc frontend -o=jsonpath=&quot;{.status.loadBalancer.ingress[0].ip}&quot;`/version</pre><h3>Rollback a Blue-Green Deployment</h3><p><em>Roll back to the old version</em> — While the green deployment is still running, simply update the service to the old (blue) deployment.</p><pre>kubectl apply -f services/hello-blue.yaml</pre><p>Verify that the blue deployment is being used.</p><pre>curl -ks <a href="https://`kubectl/">https://`kubectl</a> get svc frontend -o=jsonpath=&quot;{.status.loadBalancer.ingress[0].ip}&quot;`/version</pre><h3>Web UI for Kubernetes Engine objects</h3><p>You can also use Admin Console to view and interact with your Kubernetes Engine objects in the web interface rather than the command-line.</p><h4>Step 1</h4><p>In Admin Console, click Navigation menu and select Kubernetes Engine.Select a resource in the tab on the left, for example Services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/0*-nUc-31jWxO5im_k" /></figure><h4>Step 2</h4><p>Investigate each set of objects by navigating through the following pages:</p><ul><li>Clusters</li><li>Workloads (Pods)</li><li>Services</li><li>Configuration (ConfigMaps and Secrets).</li></ul><p>On each page, scroll over an item in the Name field and click its link to view more information about it. For example, on the Services page, click frontend. Your front end Services will display as follows with status, types, endpoints, and number of running Pods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/0*M3wEF6ylKqt1keva" /></figure><blockquote><em>To gain insight on the above, or additional ones not listed, feel free to add or contact me on </em><strong><em>LinkedIn</em></strong><em>: </em><a href="https://www.linkedin.com/in/computer-vision-engineer/"><em>https://www.linkedin.com/in/computer-vision-engineer/</em></a></blockquote><h3>Revisions, Contributions and Corrections</h3><p>If you find an error in this article, or, if you’d like to contribute to the open source projects I’m working on, please email me at<em> </em><strong><em>joehoeller@gmail.com.</em></strong></p><p><em>— Finale!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=752d222642de" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Object Detection on Thermal Images]]></title>
            <link>https://medium.com/@joehoeller/object-detection-on-thermal-images-f9526237686a?source=rss-6a6c7dbe9fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9526237686a</guid>
            <category><![CDATA[thermal-camera]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[flir]]></category>
            <category><![CDATA[yolo]]></category>
            <dc:creator><![CDATA[Joe Hoeller]]></dc:creator>
            <pubDate>Sat, 01 Feb 2020 20:51:48 GMT</pubDate>
            <atom:updated>2020-02-04T02:03:51.079Z</atom:updated>
            <content:encoded><![CDATA[<p>Robust Object Classification of Occluded Objects in Forward Looking Infrared Cameras using Ultralytics YOLO-v3 and Dark Chocolate</p><blockquote><strong>I believe that every article should have working code:</strong> <a href="https://github.com/joehoeller/Object-Detection-on-Thermal-Images">https://github.com/joehoeller/Object-Detection-on-Thermal-Images</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T-mEZUS6rYGA1PprLjBJ0A.png" /><figcaption>FLIR Thermal Images</figcaption></figure><h3>Thermal Image Dataset &amp; Open Source Tools used</h3><ol><li><a href="https://github.com/ultralytics/yolov3"><strong>Ultralytics YOLO-v3</strong></a>: Command line interface with metrics and explainability tools for DarkNet’s <a href="https://github.com/ultralytics/yolov3"><strong>YOLO-v3</strong></a>; Special thanks to <a href="https://www.linkedin.com/in/glenn-jocher/">Glenn Jocher</a> for the support he gave on GitHub for all my PRs — <em>:P Job well done!</em></li><li><a href="https://github.com/joehoeller/Dark-Chocolate"><strong>Dark Chocolate</strong></a>: Optimized for more accurate IOU calculations when transferring COCO data set annotations to DarkNet format. Hence,<strong> Dark</strong>(net) <strong>Chocolate</strong>(COCO)!</li><li><a href="https://www.flir.com/oem/adas/adas-dataset-form/"><strong>FLIR Dataset</strong></a>: The FLIR starter thermal dataset enables developers to start training convolutional neural networks (CNN), empowering the automotive community to create the next generation of safer and more efficient ADAS and driverless vehicle systems.</li></ol><h3><strong>Initial Problem Detecting Objects using Thermal Imagery</strong></h3><p>In thermal imagery analysis it has been challenging to beat the following scores, which are considered too low to be placed into production for real-world use cases. W<em>e intend to solve that with this challenge!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*rCYB8hSYojISrRV0Ykw_pw.png" /><figcaption>Image courtesy of original article from <a href="https://medium.com/swlh/object-detection-on-thermal-images-4f3410a89db4">Object Detection on Thermal Images</a></figcaption></figure><blockquote>To provide some insight, the downside to thermal images is that they are less detailed and provide an unfamiliar visual impression of features to a pre-trained data set, or class labels like COCO annotations.</blockquote><h3>Thermal Image Classification Requirements</h3><ul><li>Multi-class object detection on thermal imagery.</li><li>Develop a custom annotation conversion tool where the math had built-in error correction on values for the annotation inputs, which resulted in a more precise <a href="https://giou.stanford.edu/"><strong>IoU (Intersection Over Union)</strong></a> score, which ultimately boosted <a href="https://www.youtube.com/watch?v=pM6DJ0ZZee0"><strong>mAP (Mean Average Precision)</strong></a>.</li><li>Garner a <strong><em>mAP, F1,</em></strong><em> and </em><strong><em>Recall</em></strong><em> </em>score of: <strong><em>&gt;0.85</em></strong> to determine a good baseline.</li><li><strong>Stretch Goal:</strong> Garner competition or production grade accuracy scores of <strong>mAP</strong>: <strong>&gt;0.95</strong>, <strong>Recall</strong> <strong>&gt;0.90</strong>, &amp; <strong>F1</strong>: <strong>&gt;0.85.</strong></li><li>Alter the YOLO-v3 neural network architecture’s param’s to extract better and smaller features in order to detect partially, and/or mostly obstructed objects in thermal imagery.</li></ul><blockquote>For a more in-depth explanation on <strong><em>mAP, Recall, F1 </em></strong><em>and</em><strong><em> IoU</em></strong> (Intersection over Union), please refer to this article here:</blockquote><blockquote><a href="https://towardsdatascience.com/breaking-down-mean-average-precision-map-ae462f623a52#1a59">https://towardsdatascience.com/breaking-down-mean-average-precision-map-ae462f623a52#1a59</a></blockquote><h3><strong>Solution</strong></h3><p><em>Use pre-built tools and engineer custom solutions around it, for a fast go-to-market approach!</em></p><ul><li>Ultralytics YOLO-v3 + altering hyperparameters of the CNN (Convolutional Neural Network) architecture for thermal imagery use case</li><li>FLIR Dataset</li><li>Bake my own custom annotation tool for more accurate IoU</li></ul><h3><strong>Neural Network, Data Engineering and Hyper-Parameter Choices</strong></h3><p>YOLO-v3 Spatial Pooling (SPP) blocks with <strong>Ultralyics</strong> <em>multi-scale</em> option was chosen, because SPP blocks extract more relevant features to operate. This results in increased knowledge of classes, and the 52 x 52 layer in YOLO-v3 detects the smaller objects quite well. This is useful for objects in the distance or down the street in FLIR images. Note that in the CNN architecture there is not a softmaxing of class scores. This is also useful if we wanted to apply this for <a href="https://github.com/ultralytics/yolov3/wiki/Example:-Transfer-Learning">transfer learning</a> later on, and train classes like specific car types such as taxi, truck or electric vehicle.</p><p>YOLO-v3-SPP also has residual skip connections and upsampling, but the most salient feature of <em>v3</em> is that it makes detections at three different scales. In YOLO-v3, the detection is done by applying 1x1 detection kernels on feature maps of three different sizes at three different places in the network.</p><p><em>* I will denote that SPP and skip connections are robust to image size, even though the diagram displays an image size of 416, Ultralytics YOLO-v3 had multi-scale which was used as well.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*yp6pW3Op1nZCxDV--ZcywQ.png" /><figcaption>Image courtesy of StackOverflow</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1WoYlBBm-yp9ses8kxcLLQ.png" /><figcaption>Image courtesy of StackOverflow</figcaption></figure><p>To understand and visualize SPP Blocks a bit more, in yolov3-spp.cfg, they use 3 different size max pooling to the same image by using [route]. After that, all 3 collect a created feature map called, “fixed-length representation”, regardless of image size and scale. More on SPP can be found in this research paper here <em>— </em><a href="https://arxiv.org/pdf/1406.4729.pdf"><em>https://arxiv.org/pdf/1406.4729.pdf</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*oKsVcQSAvSWg6OAltSkvUw.png" /><figcaption>Image courtesy of Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun</figcaption></figure><blockquote>Side note: After I researched the features of the of YOLO-v3 architecture, it led me to SPP blocks. Which then led me to choosing a Darknet port of YOLO-v3.</blockquote><h3><strong>Data Engineering for FLIR Dataset</strong></h3><p>I ended up creating my own COCO to Darknet Annotation conversion tool, called <strong>Dark Chocolate.</strong></p><blockquote>GitHub link here: <a href="https://github.com/joehoeller/Dark-Chocolate">https://github.com/joehoeller/<strong>Dark-Chocolate</strong></a>.</blockquote><p><strong><em>Why did I do that?</em></strong> To do some different “<em>maths</em>”, so when the images were run through the neural network, it produced a more precise IoU score, which ultimately boosted the mAP (mean average precision).</p><p>For example, when I converted the data set with this conversion tool already available, <a href="https://github.com/enesozi/object-detection/blob/master/convert_coco_yolo.py">https://github.com/enesozi/object-detection/blob/master/convert_coco_yolo.py</a>, I got a different result.</p><p>Whereas, when I used the tool I made, “<em>Dark Chocolate”</em>, it computed the precision and values ever so slightly different, which made a very positive impact on the end result.</p><p>(For example, 0.4609375 versus 0.460938 and 0.7015625 versus 0.701562, and so forth):</p><p><strong>FLIR_00001.json output using Dark Chocolate:</strong></p><pre>2   0.41875     0.490234375    0.20625    0.23828125</pre><pre>2   0.7015625   0.4521484375   0.1375     0.150390625</pre><pre>2   0.1625      0.45703125     0.075      0.0546875</pre><pre>2   0.2234375   0.4609375      0.04375    0.06640625</pre><p><strong>FLIR_00001.json output using convert_coco_yolo.py:</strong></p><pre>2   0.418750   0.490234   0.206250   0.238281</pre><pre>2   0.701562   0.452148   0.137500   0.150391</pre><pre>2   0.162500   0.457031   0.075000   0.054688</pre><pre>2   0.223438   0.460938   0.043750   0.066406</pre><h3><strong>Improved IOU Calculation from Annotated Inputs</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*y7QQCeG3JZ03gVI01VQyQQ.png" /><figcaption>Image courtesy of StackOverflow</figcaption></figure><p>1. Count true positives (TP)</p><p>2. Count false positives (FP)</p><p>3. Count false negatives (FN)</p><p>4. Intersection = TP</p><p>5. Union = TP + FP + FN</p><p>6. IOU = Intersection/Union</p><p>The left side is our ground truth, while the right side contains our predictions. The highlighted cells on the left side note which class we are looking at for statistics on the right side. The highlights on the right side note true positives in a cream color, false positives in orange, and false negatives in yellow (note that all others are true negatives — they are predicted as this individual class, and should not be based on the ground truth).</p><p>For Class 0, only the top row of the 4x4 matrix should be predicted as zeros. This is a rather simplified version of a real ground truth. In reality, the zeros could be anywhere in the matrix. On the right side, we see 1,0,0,0, meaning the first is a false negative, but the other three are true positives (aka 3 for Intersection as well). From there, we need to find anywhere else where zero was falsely predicted, and we note that happens once on the second row, and twice on the fourth row, for a total of three false positives.</p><p>To get the union, we add up TP (3), FP (3) and FN (1) to get seven. The IOU for this class, therefore, is 3/7.</p><p>If we do this for all the classes and average the IOUs, we get:</p><p><strong>Mean IOU = [(3/7) + (2/6) + (3/4) + (1/6)] / 4 = 0.420</strong></p><h3><strong>Hyper-parameter Choices</strong></h3><p>I kept the yolov3-spp.cfg in its default configuration, sans the params at the end and at the beginning of each of the three YOLO layers, which were classes and filters. I used the following calculation to determine the number of filters for each layer, based on number of classes:</p><p><strong>(4 + 1 + 17) * 3 = 66</strong><br> <em>17 classes, 66 filters</em></p><h3><strong>Results and Metrics — <em>STRETCH GOAL COMPLETED!</em></strong></h3><p>After 272 training epochs metric highlights are: <strong>mAP</strong>: 0.961, <strong>Recall</strong> 0.922, &amp; <strong>F1</strong>: 0.857. Which beats the initial thermal image benchmarks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5TweNQuW2WaSL5GeQNDI4A.png" /><figcaption>Ultralytics YOLO-v3 Metrics Output</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ksjNlAN-_lLJO6Sw2m2-uA.png" /><figcaption>Ground Truth output by Ultralytics YOLO-v3 tooling</figcaption></figure><h3>Detecting Obstructed Objects in Thermal Images</h3><p>As stated above, yolov3-spp doesn’t have a softmax class. SPP blocks extract more and better features, as well as YOLO-v3’s 52 x 52 layer detects the smaller objects better. This combination aided in detecting objects that were down the street and far off in the distance, or obstructed and/or mostly hidden from foreground objects, as shown in a few samples below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*M1oR96FZDYVa_qKEvm-g9w.png" /><figcaption><strong><em>A. See cars 0.57, 1.00, 0.99, 0.86, mostly obstructed and far off in distance and it still detected them.</em></strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*yN4bM1fcsOdaxeokiliVbw.png" /><figcaption><strong>B. See lots of people in the distance standing in front of one another; and these people are still detected by the algorithm.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*ooluD5YEMv_9xwYw90RGYA.png" /><figcaption><strong><em>C. Detected one-half of a person obstructed by railing.</em></strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*M3sVDsbz5ZtjdkhNH03mOg.png" /><figcaption><strong><em>D. Cars mostly obstructed by wall, especially car 0.68 in parking lot.</em></strong></figcaption></figure><h3><strong>Robust Generalization of Neural Network Model</strong></h3><p>I used traffic pictures from a Google Image search. It seems to generalize well outside the FLIR data set, see images below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*XCnuEEJzeh6TUHQl8kOxYw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*pzBf28lXUoagE_TGLNfgIg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*WiTuG2qQBHxApvDm5VNVQg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*YltAAp80BnVM4lKqPPHWcQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*eOb5aVtM5T1LXZ3JJlx7Og.png" /><figcaption>Random images from Google Image Search to test how well the model generalized, e.g.; No over fitting of model.</figcaption></figure><h3>How could these production grade scores improve?</h3><ol><li>Experimentation using dice coefficients to penalize the model for false positives/false negatives.</li><li>Transfer Learning on neural network model to find anything it may have missed.</li><li>Neural Network <a href="https://en.wikipedia.org/wiki/Ensemble_learning"><strong>ensemble methods</strong></a> for feature extraction, active-learning, or by modifying the data pipeline for it with various augmentation techniques.</li></ol><h3>Real world consulting ideas for applying this production-grade project:</h3><ul><li><strong>Search and Rescue for Public Safety:</strong> Object Classification in Thermal Images using Convolutional Neural Networks for Search and Rescue Missions (SAR) with Unmanned Aerial Systems (UAS).</li><li><strong>Defense and Aerospace:</strong> Detecting IEDs (Improvised Explosive Devices) in live combat war zones.</li><li><strong>Self-Driving Cars:</strong> Autonomous vehicles, personal or commercial.</li><li><strong>Industrial Applications:</strong> Electrical grid monitoring, wind power, and oil refinery monitoring.</li><li><strong>Improved Breast Cancer Screening &amp; Detection:</strong> Automated analysis of tumor segmentation in thermal images using artificial intelligence increases the accuracy of detecting breast cancer, and enables use in breast cancer screening programs.</li><li><strong>Segmentation of Industrial Material types for automated assembly lines:</strong> Deep Thermal Imaging for material type recognition of Spatial Surface Temperature Patterns (SSTP).</li><li><strong>Sense and Detect Active School Shooters:</strong> Ensemble with Doppler signal processing methods for Concealed Weapon Detection in a Human Body by Infrared Imaging.</li><li><strong>NASA/ESA Land Rovers </strong>(e.g.; <em>Mars Exploration </em><strong><em>Rovers</em></strong><em> (MER) Spirit and Opportunity</em>): Fork GitHub repo and customize to measure heat signatures from various extraterrestrial objects. This will allow one to determine what materials/composites are in these objects. Also allows (martian) rover to further explore extra-terrestrial planets while identifying objects we cannot see in normal spectrum.</li></ul><blockquote><em>To gain insight on the above business use cases, or additional ones not listed, feel free to add or contact me on </em><strong><em>LinkedIn</em></strong><em>: </em><a href="https://www.linkedin.com/in/computer-vision-engineer/">https://www.linkedin.com/in/computer-vision-engineer/</a></blockquote><h3>Revisions, Contributions and Corrections</h3><p>If you find an error in this article, or, if you’d like to contribute to the open source projects I’m working on, please email me at<em> </em><strong><em>joehoeller@gmail.com.</em></strong></p><p><em>— Finale!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9526237686a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>