<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lindo St. Angel on Medium]]></title>
        <description><![CDATA[Stories by Lindo St. Angel on Medium]]></description>
        <link>https://medium.com/@lindo.st.angel?source=rss-e03a0010d87d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*gtrQTQeyq1eMppql.</url>
            <title>Stories by Lindo St. Angel on Medium</title>
            <link>https://medium.com/@lindo.st.angel?source=rss-e03a0010d87d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:37:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lindo.st.angel/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your Camera Doesn’t Just See — It Explains]]></title>
            <link>https://pub.towardsai.net/your-camera-doesnt-just-see-it-explains-d25086c40f78?source=rss-e03a0010d87d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d25086c40f78</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[multimodal-models]]></category>
            <category><![CDATA[home-assistant]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Lindo St. Angel]]></dc:creator>
            <pubDate>Tue, 11 Nov 2025 13:02:57 GMT</pubDate>
            <atom:updated>2025-11-11T14:04:07.012Z</atom:updated>
            <content:encoded><![CDATA[<h3>Your Camera Doesn’t Just See — It Explains</h3><h4>A look at how vision-language models, summarizers, and face recognition combine inside a Home Assistant custom Integration</h4><figure><img alt="Video Analyzer summarizing camera frames into natural language captions inside Home Assistant." src="https://cdn-images-1.medium.com/max/1024/1*oUVPVNpmD-KCN-WG1yKC-g.jpeg" /><figcaption>Image by the author (unknown person was me out of the scene)</figcaption></figure><p>Smart-home cameras are noisy storytellers. A leaf moves, a light flickers, and your phone buzzes again. What you get is motion — not meaning.</p><p>The <a href="https://github.com/goruck/home-generative-agent/blob/main/custom_components/home_generative_agent/core/video_analyzer.py"><strong>Video Analyzer</strong></a> inside the <a href="https://github.com/goruck/home-generative-agent"><strong>Home Generative Agent</strong></a> integration for <a href="https://www.home-assistant.io/">Home Assistant</a> changes that.</p><p>It transforms a stream of motion snapshots into <strong>one clean, factual sentence</strong> — the kind of line you can rely on for automation without false alarms.</p><h3>The bigger picture: part of Home Generative Agent</h3><p>The Video Analyzer isn’t a standalone add-on. It’s a core subsystem of <strong>Home Generative Agent (HGA)</strong> — a custom Home Assistant integration that gives your home an intelligent memory and language interface.</p><p>At the architectural level:</p><p><strong>— HGA</strong> connects multiple large models — vision, language, and embeddings under one orchestration layer.</p><p>— Each category (chat, VLM, summarization, embedding) uses <strong>pluggable providers — </strong>OpenAI, Ollama, or Gemini.</p><p>— The integration exposes structured outputs as Home Assistant entities and events, allowing automations, dashboards, and vector-based “memories” to interoperate.</p><p>In that ecosystem, <strong>the Video Analyzer is the visual limb</strong>. It speaks to cameras, invokes the vision and summarization models, and publishes the semantic result back to the system.</p><p>If the agent were a brain, the Video Analyzer is its occipital lobe — constantly turning pixels into concise, automation-ready meaning.</p><h3>Why build it at all?</h3><p>Motion sensors are interrupts, not meaning. They trigger on shadows, noise, or sunlight, but can’t say <em>what</em> happened.</p><p>What you want is a reliable abstraction — a textual state that describes the scene:</p><blockquote><em>“Two packages on the porch; no person present.”<br> “Alice is at the front door.”<br> “A person picks up a box and leaves.”</em></blockquote><p>That’s the promise: <strong>semantic compression</strong> — turning gigabytes of video into a single, precise sentence.</p><h3>The functional flow</h3><p>Trigger → Capture → Describe → Publish → Notify → Remember</p><p><strong>Trigger</strong>: Cameras or motion sensors start a short capture loop.</p><p><strong>Capture</strong>: snapshots enter per-camera asyncio queues, deduped via perceptual hash + heartbeat logic.</p><p><strong>Describe</strong>:</p><p>— Heuristic path for single frames (fast, deterministic).</p><p>— Vision-Language Model (VLM) + summarizer for multi-frame context.</p><p><strong>Publish:</strong> updates “latest” image + “recognized people” entities; fires HA events.</p><p><strong>Notify:</strong> optional mobile push (always or on anomaly).</p><p>Remember: the vector DB stores text and metadata for long-term recall.</p><h4>A quick mental model</h4><pre>[Camera] --&gt; [Queue] --&gt; [Describe (VLM + heuristic)] --&gt; frame texts + identities<br>                       |<br>                       v<br>        [Summarizer LLM + rules] --&gt; {summary, recognized_people}<br>                       |<br>                       +--&gt; Publish entities<br>                       +--&gt; Notify user<br>                       +--&gt; Store vectors</pre><h3>Where the models come in</h3><p>The Video Analyzer uses three model tiers, each tuned by configuration constants in <a href="https://github.com/goruck/home-generative-agent/blob/main/custom_components/home_generative_agent/const.py">const.py</a>.</p><p>1. <strong>Vision-Language Model (VLM) — </strong><em>describes a single frame</em></p><p>Default: <strong>Ollama</strong> qwen3-vl:8b (temperature 0.2)</p><p>System prompt (core clauses):</p><blockquote><em>“Produce a short, factual description (1–3 sentences).<br> Do NOT speculate, infer identity, or describe unseen content.<br> Use consistent phrasing across similar frames.<br> Never assume gender if unclear.”</em></blockquote><p>Why: It enforces neutral, deterministic phrasing so similar images yield similar sentences — crucial for reliable embeddings and anomaly detection.</p><p>When previous frame text is available, the VLM may describe motion only if multiple cues agree; otherwise, it defaults to “stands” or “walks nearby.” This approach minimizes false motion narratives that plague consumer AI cameras.</p><p>User prompt:</p><blockquote><em>“Describe this image clearly and factually in 1–3 sentences. Do not add names, timestamps, or speculation.”</em></blockquote><p>A focused variant accepts keywords (Primary attention: {key_words}) when you want to bias attention toward, say, <em>packages</em> or <em>doors</em>.</p><p>The VLM only describes what’s visually present — it doesn’t recognize faces or assign identities. Person recognition happens later in the pipeline through a separate face-recognition service, which tags frames with known identities before the summarization step. This separation keeps the VLM neutral and privacy-safe while still allowing the system to link actions to recognized individuals when appropriate.</p><p>2. <strong>Summarization model </strong>— <em>weave multiple frames into one line</em></p><p>Default: <strong>Ollama</strong> qwen3:8b (temperature 0.2)</p><p>System rules excerpt:</p><blockquote><em>“≤150 characters, ≤2 sentences.<br> Narrate events in order (t+Xs).<br> Include up to two known names; otherwise say ‘a person’.<br> Treat ‘Unknown Person’ as human.<br> Describe visible actions only; no speculation.”</em></blockquote><p>These constraints force concise, automation-safe summaries.<br> When multiple frames exist, the summarizer fuses them chronologically:</p><blockquote><em>“Lindo St. Angel steps onto the porch, then leans on the railing.”</em></blockquote><p>Suppose there’s only one frame, a heuristic tries first — cleaning text, inserting the subject if recognized, and truncating length. Only when that path fails is the LLM called.</p><p>User prompt reminder:</p><blockquote><em>“Write ≤150 characters (≤2 sentences). Obey all rules and narrate in order.”</em></blockquote><p>3. <strong>Embedding &amp; chat layers </strong>— <em>memory and reasoning</em></p><p>Embedding prompt:</p><blockquote><em>“Represent this sentence for searching relevant passages: {query}”</em></blockquote><p>This ensures every caption becomes a stable vector entry for later retrieval or novelty scoring.</p><p>Chat summarizer system message:</p><blockquote><em>“You are a bot that summarizes messages from a smart home AI.”</em></blockquote><p>Together, these make the <strong>HGA</strong> memory continuous: video events, text logs, and automations all share the same language space.</p><h3>Prompts as contracts</h3><p>Every model prompt is a <strong>contract</strong> between you and the language model. Break it, and your automations will drift. Keep it tight and your system stays explainable.</p><p>These contracts enforce:</p><ul><li>Neutral, observable language</li><li>Short, deterministic phrasing</li><li>Stable embeddings for anomaly detection</li><li>Privacy-safe content (no gender or intent guessing)</li></ul><p>These prompts live as versioned, editable constants — not buried in code — so you can evolve them responsibly.</p><h3>How it integrates with the rest of HGA</h3><p>The Video Analyzer operates as part of a coordinated visual stack inside the Home Generative Agent (HGA). It doesn’t work alone — it collaborates with a dedicated <strong>Face Recognition</strong> service and the <strong>HGA runtime</strong> to turn raw camera input into structured, searchable memory.</p><ul><li><strong>Video Analyzer → Face Recognition → Summarization → Runtime:</strong><br>When a new frame arrives, the Video Analyzer first sends it to the <strong>Vision-Language Model (VLM)</strong> for description. The text describes the visible scene — lighting, motion, posture, and objects — but never guesses who is present. The <strong>Face Recognition module </strong>processes the same frame in parallel, which compares faces against the local database of known identities. The recognized names (if any) are attached as metadata and passed back to the Video Analyzer. The Summarization Model then fuses the frame text and the identity metadata into a single, human-readable caption, such as:<br> <em>“Lindo St. Angel steps onto the porch, then leans on the railing.”</em></li><li><strong>Structured output and signals:</strong><br>Each summarized event is posted to the <strong>HGA runtime store</strong> — a Postgres-backed vector database — indexed under (&quot;video_analysis&quot;, camera_name) for semantic search. The analyzer also fires structured Home Assistant signals (hga_new_latest, hga_recognized_people), which update entities and trigger automations in real time.</li><li><strong>Notifications and image protection:</strong><br>When the analyzer detects anomalies or unfamiliar faces, it can send mobile push notifications via the configured notify_service. The relevant snapshot is marked for retention to prevent it from being deleted by cleanup jobs.</li><li><strong>Unified memory through embeddings:</strong><br>Every generated caption — whether from video, chat summaries, or sensor logs — is embedded into the same vector space. This lets HGA recall or reason across modalities:<br> <em>“Who came to the porch after sunset?”</em><br> <em>“Was anyone seen before the door opened?”</em></li></ul><p>By separating <strong>visual perception (VLM)</strong>, <strong>identity recognition</strong>, and <strong>semantic synthesis (Summarization)</strong>, the system stays modular, explainable, and privacy-aware. Each component does one job well — and together they let your home not only see, but understand.</p><h3>Why this works</h3><ul><li><strong>Heuristic before model </strong>— prioritize speed and determinism; use the LLM second.</li><li><strong>Short summaries — </strong>150-char max keeps embeddings and notifications fast.</li><li><strong>Chronology— </strong>events unfold logically, preventing misordered narratives.</li><li><strong>Neutral voice — </strong>automation triggers stay predictable.</li><li><strong>Tight async loops — </strong>each camera runs independently; queues and semaphores to avoid overload.</li></ul><p>The result is a camera subsystem that behaves like a colleague, not a parrot — brief, factual, reliable.</p><h3>Real-world results: the Video Analyzer in action</h3><p><em>Note: The face-recognition module inserts named individuals into the captions; the VLM itself produces person-neutral descriptions.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hbFTjXf_xZ5c7LAEJztDDw.jpeg" /><figcaption>Figure 1 — Front porch activity detection (image by the author)</figcaption></figure><blockquote>“Lindo St. Angel walks on the pathway with bags, then stands on the porch holding one.”</blockquote><p><em>(A concise narrative that joins multiple frames — movement and arrival — into a single 17-word sentence suitable for automation triggers.)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4lmfTlKkVWOt2nv907-o-A.jpeg" /><figcaption>Figure 2 — Maintenance activity recognition (image by the author)</figcaption></figure><blockquote>“A person walks near the house, then climbs the porch steps. They carry a backpack, later hold a leaf blower, and eventually walk away, leaving the porch empty.”</blockquote><p><em>(Multiple frames stitched chronologically; the summarizer keeps the verbs concrete and factual.)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qWwbKUhmynPk2Z3KJ0VWVg.jpeg" /><figcaption>Figure 3 — Groundskeeping action tracking (image by the author)</figcaption></figure><blockquote>“A person in an orange shirt walks on a grassy area near a house, carrying a trash can. They bend over, pick up items, and later sweep leaves near the porch.”</blockquote><p><em>(Here, the LLM combines posture, object, and sequential verbs into one compact summary — no excess detail, no guesses.)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2pOehaJhcjnpr2xTw_MwQg.jpeg" /><figcaption>Figure 4 — Front gate movement (image by the author)</figcaption></figure><blockquote>“Lindo St. Angel stands near the gate, then walks on the pathway.”</blockquote><p><em>(A short, multi-frame progression reflecting direct visual cues.)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EqrxgjOq7igCVcorlJY6Sg.jpeg" /><figcaption>Figure 5 — Multi-entity scene (image by the author)</figcaption></figure><blockquote>“A person stands on the porch next to a dark door, holding a white object. A black cat is on the porch near a wicker chair.”</blockquote><p><em>(The analyzer correctly identifies both entities — a person and a cat — without over-describing or inferring relationships.)</em></p><h3>The bigger picture</h3><p><strong>Home Assistant</strong> handles the sensors and cameras; the <strong>Video Analyzer</strong> interprets what they see. Each event becomes a single, factual sentence — concise enough for automation, expressive enough for understanding. Those sentences are fed into the <strong>Home Generative Agent</strong> core, where they are embedded, indexed, and retained. Other agents can then search, summarize, and act on that knowledge.</p><p>The result is a closed loop: <em>see → describe → remember → respond. </em>Every frame that passes through becomes part of a living memory — one that can reason about what’s happening, not just record it.</p><h3>A step toward meaningful perception</h3><p>The Video Analyzer is more than an engineering module; it’s a small demonstration of what grounded, explainable AI can look like in a real home. By enforcing neutral language and deterministic prompts, it replaces noise with narrative and makes machine vision accountable.</p><p>Pixels become sentences. Sentences become context. Context becomes understanding. Sometimes the model drifts — but the constraints pull it back to truth and meaning.</p><h3>System map — how the Video Analyzer fits inside Home Generative Agent</h3><p>The Video Analyzer sits between Home Assistant’s cameras and HGA’s memory core, converting raw motion into structured language.</p><pre>┌───────────────────────────────────────────────────────────────┐<br>│                   Home Generative Agent (HGA)                 │<br>│   ┌──────────────────────────┐      ┌───────────────────────┐ │<br>│   │  Language + Memory Core  │◄────►│  Vector Store (Postgres │<br>│   │  - Chat / Summarization  │      │  + Embeddings Index)    │<br>│   │  - Context Manager       │      └───────────────────────┘ │<br>│   └──────────▲───────────────┘                                │ <br>│              │                                                │<br>│              │ (semantic summaries, embeddings)               │<br>│              │                                                │<br>│   ┌──────────┴──────────────┐                                 │<br>│   │      Video Analyzer     │                                 │<br>│   │  - Captures frames      │                                 │<br>│   │  - Runs VLM + Summary   │                                 │<br>│   │  - Detects anomalies    │                                 │<br>│   │  - Publishes entities   │                                 │<br>│   └──────────▲──────────────┘                                 │<br>│              │                                                │<br>│              │ (events, images, people, summaries)            │<br>│              │                                                │<br>│   ┌──────────┴──────────────┐                                 │<br>│   │   Home Assistant Core   │                                 │<br>│   │  - Cameras              │                                 │<br>│   │  - Sensors              │                                 │<br>│   │  - Automations          │                                 │<br>│   └─────────────────────────┘                                 │<br>│                                                               │<br>│   Notifications → Mobile App (via notify.mobile_app_*)        │<br>│   Dashboards     → Lovelace or custom UI                      │<br>│   Memory Search  → “What happened on the porch today?”        │<br>└───────────────────────────────────────────────────────────────┘</pre><p>Home Assistant handles the sensors and cameras; the Video Analyzer interprets what they see. It turns images into language — one factual sentence per event — and feeds that text to the Home Generative Agent core. HGA then embeds and stores those sentences in its vector memory, where other agents and automations can search, summarize, or act on them. The same data powers mobile notifications, dashboards, and contextual reasoning. Together, these modules form a closed loop: <em>see → describe → remember → respond.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d25086c40f78" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/your-camera-doesnt-just-see-it-explains-d25086c40f78">Your Camera Doesn’t Just See — It Explains</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LangChain Meets Home Assistant: Unlock the Power of Generative AI in Your Smart Home]]></title>
            <link>https://medium.com/data-science/langchain-meets-home-assistant-unlock-the-power-of-generative-ai-in-your-smart-home-f318732d9d84?source=rss-e03a0010d87d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f318732d9d84</guid>
            <category><![CDATA[home-assistant]]></category>
            <category><![CDATA[edge-computing]]></category>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[smart-home]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Lindo St. Angel]]></dc:creator>
            <pubDate>Sun, 05 Jan 2025 14:01:39 GMT</pubDate>
            <atom:updated>2025-05-05T01:28:10.900Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn how to create an agent that understands your home’s context, learns your preferences, and interacts with you and your home to accomplish activities you find valuable.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FmWlPYUgsHV6Wo2G" /><figcaption>Photo by <a href="https://unsplash.com/@omilaev?utm_source=medium&amp;utm_medium=referral">Igor Omilaev</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>This article describes the architecture and design of a <a href="https://www.home-assistant.io/">Home Assistant</a> (HA) integration called home-generative-agent. This project uses <a href="https://www.langchain.com/">LangChain</a> and <a href="https://www.langchain.com/langgraph">LangGraph</a> to create a <a href="https://arxiv.org/abs/2304.03442#">generative AI agent</a> that interacts with and automates tasks within a HA smart home environment. The agent understands your home’s context, learns your preferences, and interacts with you and your home to accomplish activities you find valuable. Key features include creating automations, analyzing images, and managing home states using various LLMs (Large Language Models). The architecture involves both cloud-based and edge-based models for optimal performance and cost-effectiveness. Installation instructions, configuration details, and information on the project’s architecture and the different models used are included and can be found on the <a href="https://github.com/goruck/home-generative-agent">home-generative-agent</a> GitHub. The project is open-source and welcomes contributions.</p><p>These are some of the features currently supported:</p><ul><li>Create complex Home Assistant automations.</li><li>Image scene analysis and understanding.</li><li>Home state analysis of entities, devices, and areas.</li><li>Full agent control of allowed entities in the home.</li><li>Short- and long-term memory using semantic search.</li><li>Automatic summarization of home state to manage LLM context length.</li></ul><p>This is my personal project and an example of what I call <a href="https://medium.com/@lindo.st.angel/learning-directed-hacking-89d366cbc884"><em>learning-directed hacking</em></a>. The project is not affiliated with my work at Amazon nor am I associated with the organizations responsible for Home Assistant or LangChain/LangGraph in any way.</p><h3>Important Considerations</h3><p>Creating an agent to monitor and control your home can lead to unexpected actions and potentially put your home and yourself at risk due to LLM hallucinations and privacy concerns, especially when exposing home states and user information to cloud-based LLMs. I have made reasonable architectural and design choices to mitigate these risks, but they cannot be completely eliminated.</p><p>One key early decision was to rely on a hybrid cloud-edge approach. This enables the use of the most sophisticated reasoning and planning models available, which should help reduce hallucinations. Simpler, more task-focused edge models are employed to further minimize LLM errors.</p><p>Another critical decision was to leverage LangChain’s capabilities, which allow sensitive information to be hidden from LLM tools and provided only at runtime. For instance, tool logic may require using the ID of the user who made a request. However, such values should generally not be controlled by the LLM. Allowing the LLM to manipulate the user ID could pose security and privacy risks. To mitigate this, I utilized the <a href="https://python.langchain.com/api_reference/core/tools/langchain_core.tools.base.InjectedToolArg.html"><em>InjectedToolArg</em></a> annotation.</p><p>Additionally, using large cloud-based LLMs incurs significant cloud costs, and the edge hardware required to run LLM edge models can be expensive. The combined operational and installation costs are likely prohibitive for the average user at this time. An industry-wide effort to “make LLMs as cheap as CNNs” is needed to bring home agents to the mass market.</p><p><strong><em>It is important to be aware of these risks and understand that, despite these mitigations, we are still in the early stages of this project and home agents in general. Significant work remains to make these agents truly useful and trustworthy assistants.</em></strong></p><h3>Architecture and Design</h3><p>Below is a high-level view of the home-generative-agent architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*LBA6d1gUm69prQArmbYjTQ.png" /><figcaption>Image by Lindo St. Angel</figcaption></figure><p>The general integration architecture follows the best practices as described in <a href="https://developers.home-assistant.io/docs/development_index/">Home Assistant Core</a> and is compliant with <a href="https://www.hacs.xyz/">Home Assistant Community Store</a> (HACS) publishing requirements.</p><p>The agent is built using LangGraph and uses the HA <em>conversation</em> component to interact with the user. The agent uses the Home Assistant LLM API to fetch the state of the home and understand the HA native tools it has at its disposal. I implemented all other tools available to the agent using LangChain. The agent employs several LLMs, a large and very accurate primary model for high-level reasoning, smaller specialized helper models for camera image analysis, primary model context summarization, and embedding generation for long-term semantic search. The primary model can be either cloud (best accuracy, highest cost) or edge-based (good accuracy, lowest cost) and the helper models are all edge-based. The edge models run under the <a href="https://ollama.com/">Ollama</a> framework on a computer located in the home.</p><p>The models currently being used are summarized below.</p><ul><li><a href="https://platform.openai.com/docs/models#gpt-4o">GPT-4o</a> | OpenAI Cloud | High-level reasoning and planning</li><li><a href="https://ollama.com/library/qwen2.5:32b">qwen2.5:32b</a> | Ollama Edge | High-level reasoning and planning</li><li><a href="https://ollama.com/library/llama3.2-vision">llama-3.2-vision-11b</a> | Ollama Edge | Image scene analysis</li><li><a href="https://ollama.com/library/qwen2.5:3b">qwen2.5:3b</a> | Ollama Edge | Primary model context summarization</li><li><a href="https://ollama.com/library/mxbai-embed-large">mxbai-embed-large</a> | Ollama Edge | Embedding generation for sematic search</li></ul><h4>LangGraph-based Agent</h4><p>LangGraph powers the conversation agent, enabling you to create stateful, multi-actor applications utilizing LLMs as quickly as possible. It extends LangChain’s capabilities, introducing the ability to create and manage cyclical graphs essential for developing complex agent runtimes. A graph models the agent workflow, as seen in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*rmwmNLs3rR7nqHoOfiNUoQ.png" /><figcaption>Diagram by Lindo St. Angel</figcaption></figure><p>The agent workflow has five nodes, each Python module modifying the agent’s state, a shared data structure. The edges between the nodes represent the allowed transitions between them, with solid lines unconditional and dashed lines conditional. Nodes do the work, and edges tell what to do next.</p><p>The <em>__start__</em> and <em>__end__</em> nodes inform the graph where to start and stop. The <em>agent</em> node runs the primary LLM, and if it decides to use a tool, the <em>action</em> node runs the tool and then returns control to the <em>agent</em>. The <em>summarize_and_remove_messages</em> node processes the LLM’s context to manage growth while maintaining accuracy if <em>agent</em> has no tool to call and message trimming is required to manage the LLM context.</p><h4>LLM Context Management</h4><p>You need to carefully manage the context length of LLMs to balance cost, accuracy, and latency and avoid triggering rate limits such as OpenAI’s Tokens per Minute restriction. The system controls the context length of the primary model by trimming the messages in the context if they exceed a max parameter which can be expressed in either tokens or messages, and the trimmed messages are replaced by a shorter summary inserted into the system message. These parameters are configurable in <em>const.py</em>; their description is below.</p><ul><li><em>CONTEXT_MAX_MESSAGES</em> | Messages to keep in context before deletion | Default = 80</li><li><em>CONTEXT_MAX_TOKENS</em> | Tokens to keep in context before deletion | Default = 25600</li><li><em>CONTEXT_MANAGE_USE_TOKENS</em> | If True, use tokens to manage context, else use messages | Default = True</li></ul><p>The <em>summarize_and_remove_messages</em> node in the graph will remove the messages only after content summarization. You can see the Python code associated with this node in the snippet below.</p><pre>async def _summarize_and_remove_messages(<br>        state: State, config: RunnableConfig, *, store: BaseStore<br>    ) -&gt; dict[str, str | list[AnyMessage]]:<br>    &quot;&quot;&quot;Coroutine to summarize and remove messages.&quot;&quot;&quot;<br>    summary = state.get(&quot;summary&quot;, &quot;&quot;)<br>    msgs_to_remove = state.get(&quot;messages_to_remove&quot;, [])<br><br>    if not msgs_to_remove:<br>        return {&quot;summary&quot;: summary}<br><br>    if summary:<br>        summary_message = SUMMARY_PROMPT_TEMPLATE.format(summary=summary)<br>    else:<br>        summary_message = SUMMARY_INITIAL_PROMPT<br><br>    # Form the messages that will be used by the summarization model.<br>    # The summary will be based on the messages that were trimmed away from the main<br>    # model call, ignoring those from tools since the AI message encapsulates them.<br>    messages = (<br>        [SystemMessage(content=SUMMARY_SYSTEM_PROMPT)] +<br>        [m.content for m in msgs_to_remove if isinstance(m, HumanMessage|AIMessage)] +<br>        [HumanMessage(content=summary_message)]<br>    )<br><br>    model = config[&quot;configurable&quot;][&quot;summarization_model&quot;]<br>    options = config[&quot;configurable&quot;][&quot;options&quot;]<br>    model_with_config = model.with_config(<br>        config={<br>            &quot;model&quot;: options.get(<br>                CONF_SUMMARIZATION_MODEL,<br>                RECOMMENDED_SUMMARIZATION_MODEL,<br>            ),<br>            &quot;temperature&quot;: options.get(<br>                CONF_SUMMARIZATION_MODEL_TEMPERATURE,<br>                RECOMMENDED_SUMMARIZATION_MODEL_TEMPERATURE,<br>            ),<br>            &quot;top_p&quot;: options.get(<br>                CONF_SUMMARIZATION_MODEL_TOP_P,<br>                RECOMMENDED_SUMMARIZATION_MODEL_TOP_P,<br>            ),<br>            &quot;num_predict&quot;: SUMMARIZATION_MODEL_PREDICT,<br>            &quot;num_ctx&quot;: SUMMARIZATION_MODEL_CTX,<br>        }<br>    )<br><br>    LOGGER.debug(&quot;Summary messages: %s&quot;, messages)<br>    response = await model_with_config.ainvoke(messages)<br>    LOGGER.debug(&quot;Summary response: %s&quot;, response)<br><br>    return {<br>        &quot;summary&quot;: response.content,<br>        &quot;messages&quot;: [RemoveMessage(id=m.id) for m in msgs_to_remove],<br>    }</pre><h4>Latency</h4><p>The latency between user requests or the agent taking timely action on the user’s behalf is critical for you to consider in the design. I used several techniques to reduce latency, including using specialized, smaller helper LLMs running on the edge and facilitating primary model prompt caching by structuring the prompts to put static content, such as instructions and examples, upfront and variable content, such as user-specific information at the end. These techniques also reduce primary model usage costs considerably.</p><p>You can see the typical latency performance below.</p><ul><li>HA intents (e.g., turn on a light) | &lt; 1 second</li><li>Analyze camera image (initial request) | &lt; 3 seconds</li><li>Add automation | &lt; 1 second</li><li>Memory operations | &lt; 1 second</li></ul><h4>Tools</h4><p>The agent can use HA tools as specified in the <a href="https://developers.home-assistant.io/docs/core/llm/">LLM API</a> and other tools built in the LangChain framework as defined in <em>tools.py</em>. Additionally, you can extend the LLM API with tools of your own as well. The code gives the primary LLM the list of tools it can call, along with instructions on using them in its system message and in the docstring of the tool’s Python function definition. You can see an example of docstring instructions in the code snippet below for the <em>get_and_analyze_camera_image</em> tool.</p><pre>@tool(parse_docstring=False)<br>async def get_and_analyze_camera_image( # noqa: D417<br>        camera_name: str,<br>        detection_keywords: list[str] | None = None,<br>        *,<br>        # Hide these arguments from the model.<br>        config: Annotated[RunnableConfig, InjectedToolArg()],<br>    ) -&gt; str:<br>    &quot;&quot;&quot;<br>    Get a camera image and perform scene analysis on it.<br><br>    Args:<br>        camera_name: Name of the camera for scene analysis.<br>        detection_keywords: Specific objects to look for in image, if any.<br>            For example, If user says &quot;check the front porch camera for<br>            boxes and dogs&quot;, detection_keywords would be [&quot;boxes&quot;, &quot;dogs&quot;].<br><br>    &quot;&quot;&quot;<br>    hass = config[&quot;configurable&quot;][&quot;hass&quot;]<br>    vlm_model = config[&quot;configurable&quot;][&quot;vlm_model&quot;]<br>    options = config[&quot;configurable&quot;][&quot;options&quot;]<br>    image = await _get_camera_image(hass, camera_name)<br>    return await _analyze_image(vlm_model, options, image, detection_keywords)</pre><p>If the agent decides to use a tool, the LangGraph node <em>action</em> is entered, and the node’s code runs the tool. The node uses a simple error recovery mechanism that will ask the agent to try calling the tool again with corrected parameters in the event of making a mistake. The code snippet below shows the Python code associated with the <em>action</em> node.</p><pre>async def _call_tools(<br>        state: State, config: RunnableConfig, *, store: BaseStore<br>    ) -&gt; dict[str, list[ToolMessage]]:<br>    &quot;&quot;&quot;Coroutine to call Home Assistant or langchain LLM tools.&quot;&quot;&quot;<br>    # Tool calls will be the last message in state.<br>    tool_calls = state[&quot;messages&quot;][-1].tool_calls<br><br>    langchain_tools = config[&quot;configurable&quot;][&quot;langchain_tools&quot;]<br>    ha_llm_api = config[&quot;configurable&quot;][&quot;ha_llm_api&quot;]<br><br>    tool_responses: list[ToolMessage] = []<br>    for tool_call in tool_calls:<br>        tool_name = tool_call[&quot;name&quot;]<br>        tool_args = tool_call[&quot;args&quot;]<br><br>        LOGGER.debug(<br>            &quot;Tool call: %s(%s)&quot;, tool_name, tool_args<br>        )<br><br>        def _handle_tool_error(err:str, name:str, tid:str) -&gt; ToolMessage:<br>            return ToolMessage(<br>                content=TOOL_CALL_ERROR_TEMPLATE.format(error=err),<br>                name=name,<br>                tool_call_id=tid,<br>                status=&quot;error&quot;,<br>            )<br><br>        # A langchain tool was called.<br>        if tool_name in langchain_tools:<br>            lc_tool = langchain_tools[tool_name.lower()]<br><br>            # Provide hidden args to tool at runtime.<br>            tool_call_copy = copy.deepcopy(tool_call)<br>            tool_call_copy[&quot;args&quot;].update(<br>                {<br>                    &quot;store&quot;: store,<br>                    &quot;config&quot;: config,<br>                }<br>            )<br><br>            try:<br>                tool_response = await lc_tool.ainvoke(tool_call_copy)<br>            except (HomeAssistantError, ValidationError) as e:<br>                tool_response = _handle_tool_error(repr(e), tool_name, tool_call[&quot;id&quot;])<br>        # A Home Assistant tool was called.<br>        else:<br>            tool_input = llm.ToolInput(<br>                tool_name=tool_name,<br>                tool_args=tool_args,<br>            )<br><br>            try:<br>                response = await ha_llm_api.async_call_tool(tool_input)<br><br>                tool_response = ToolMessage(<br>                    content=json.dumps(response),<br>                    tool_call_id=tool_call[&quot;id&quot;],<br>                    name=tool_name,<br>                )<br>            except (HomeAssistantError, vol.Invalid) as e:<br>                tool_response = _handle_tool_error(repr(e), tool_name, tool_call[&quot;id&quot;])<br><br>        LOGGER.debug(&quot;Tool response: %s&quot;, tool_response)<br>        tool_responses.append(tool_response)<br>    return {&quot;messages&quot;: tool_responses}</pre><p>The LLM API instructs the agent always to call tools using HA <a href="https://developers.home-assistant.io/docs/intent_builtin">built-in intents</a> when controlling Home Assistant and to use the intents `HassTurnOn` to lock and `HassTurnOff` to unlock a lock. An intent describes a user’s intention generated by user actions.</p><p>You can see the list of LangChain tools that the agent can use below.</p><ul><li><em>get_and_analyze_camera_image</em> | run scene analysis on the image from a camera</li><li><em>upsert_memory</em> | add or update a memory</li><li><em>add_automation</em> | create and register a HA automation</li><li><em>get_entity_history</em> | query HA database for entity history</li><li><em>get_current_device_state</em> | get the current state of one or more Home Assistant devices</li></ul><h4>Hardware</h4><p>I built the HA installation on a Raspberry Pi 5 with SSD storage, Zigbee, and LAN connectivity. I deployed the edge models under Ollama on an Ubuntu-based server with an AMD 64-bit 3.4 GHz CPU, Nvidia 3090 GPU, and 64 GB system RAM. The server is on the same LAN as the Raspberry Pi.</p><h3>Results</h3><p>I’ve been using this project at home for a few weeks and have found it useful but frustrating in a few areas that I will be working on to address. Below is a list of pros and cons of my experience with the agent.</p><h4>Pros</h4><ul><li>The camera image scene analysis is very useful and flexible since you can query for almost anything and not have to worry having the right classifier as you would for a traditional ML approach.</li><li>Automations are very easy to setup and can be quite complex. Its mind blowing how good the primary LLM is at generating HA-compliant YAML.</li><li>Latency in most cases is quite acceptable.</li><li>Its very easy to add additional LLM tools and graph states with LangChain and LangGraph.</li></ul><h4>Cons</h4><ul><li>The camera image analysis seems less accurate than traditional ML approaches. For example, detecting packages that are partially obscured is very difficult for the model to handle.</li><li>The primary model clould costs are high. Running a single package detector once every 30 mins costs about $2.50 per day.</li><li>Using structured model outputs for the helper LLMs, which would make downstream LLM processing easier, considerably reduces accuracy.</li><li>The agent needs to be more proactive. Adding a planning step to the agent graph will hopefully address this.</li></ul><h3>Example Use Cases</h3><p>Here are a few examples of what you can do with the home-generative-agent (HGA) integration as illustrated by screenshots of the Assist dialog taken by me during interactions with my HA installation.</p><ul><li>Create an Automation.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*qP8fNmqKJq4wABMCCAyOhg.png" /><figcaption>Image by Lindo St. Angel</figcaption></figure><ul><li>Create an automation that runs periodically.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*7mxA0LVZJvRhlmsz3HvQBQ.png" /><figcaption>Image by Lindo St. Angel</figcaption></figure><p>The snippet below shows that the agent is fluent in YAML based on what it generated and registered as an HA automation.</p><pre>alias: Check Litter Box Waste Drawer<br>triggers:<br>  - minutes: /30<br>    trigger: time_pattern<br>conditions:<br>  - condition: numeric_state<br>    entity_id: sensor.litter_robot_4_waste_drawer<br>    above: 90<br>actions:<br>  - data:<br>      message: The Litter Box waste drawer is more than 90% full!<br>    action: notify.notify</pre><ul><li>Check a camera.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*Zxu-tsT9QkIwy6zGzF7_AA.png" /><figcaption>Image by Lindo St. Angel</figcaption></figure><ul><li>Check multiple cameras (video by the author).</li></ul><p><a href="https://github.com/user-attachments/assets/230baae5-8702-4375-a3f0-ffa981ee66a3">https://github.com/user-attachments/assets/230baae5-8702-4375-a3f0-ffa981ee66a3</a></p><ul><li>Summarize the home state (video by the author).</li></ul><p><a href="https://github.com/user-attachments/assets/96f834a8-58cc-4bd9-a899-4604c1103a98">https://github.com/user-attachments/assets/96f834a8-58cc-4bd9-a899-4604c1103a98</a></p><ul><li>Long-term memory with semantic search.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*AtCYRf5MUp_9_oEU5kYy0Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*88vfzObxYGatSDxfvAW4Zw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*hn3Yk6jkxq9IPr3DoYLrEw.png" /><figcaption>Images by Lindo St. Angel</figcaption></figure><p>You can see that the agent correctly generates the automation below.</p><pre>alias: Prepare Home for Arrival<br>description: Turn on front porch light and unlock garage door lock at 7:30 PM<br>mode: single<br>triggers:<br> - at: &quot;19:30:00&quot;<br>    trigger: time<br>actions:<br>  - target:<br>      entity_id: light.front_porch_light<br>    action: light.turn_on<br>    data: {}<br>  - target:<br>      entity_id: lock.garage_door_lock<br>    action: lock.unlock<br>    data: {}</pre><ul><li>Check a camera for packages.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*rg3NMW1PTrRGRRf1lvcmqA.png" /><figcaption>Image by Lindo St. Angel</figcaption></figure><p>Below is the camera image the agent analyzed, you can see that two packages are visible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*b6Hk2T0TiM32qpWcYHM-NQ.png" /><figcaption>Photo by Lindo St. Angel</figcaption></figure><ul><li>Proactive notification of package delivery.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*lbq7HT0FTHjF_8J3IrilVA.png" /><figcaption>Image by Lindo St. Angel</figcaption></figure><p>Below is an example notification from this automation if any boxes or packages are visible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*iHF_bKOlkuBACO_ohuLpFg.png" /><figcaption>Image by Lindo St. Angel</figcaption></figure><ul><li>Proactive Camera Video Analysis</li></ul><p>You can enable proactive video scene analysis from cameras visible to Home Assistant. When enabled, motion detection will trigger the analysis which will be stored in a database for use by the agent, and optionally, notifications of the analysis will be sent to the mobile app. You can also enable anomaly detection which will only send notifications based on semantic search of the current analysis vis-a-vis the database. These options are set in the integration’s config UI.</p><p>The image below is an example of a notification sent to the mobile app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lch7NOBqBDu2wGT469Iw8Q.jpeg" /><figcaption>Image by Lindo St. Angel</figcaption></figure><h3>Conclusions</h3><p>The Home Generative Agent offers an intriguing way to make your Home Assistant setup more user-friendly and intuitive. By enabling natural language interactions and simplifying automation, it provides a practical and useful tool for everyday smart home use.</p><p>Using a home generative agent carries security, privacy and cost risks that need further mitigation and innovation before it can be truly useful for the mass market.</p><p>Whether you’re new to Home Assistant or a seasoned user, this integration is a great way to enhance your system’s capabilities and get familiar with using generative AI and agents at home. If you’re interested in exploring its potential, visit the <a href="https://github.com/goruck/home-generative-agent">Home Generative Agent GitHub page</a> and get started today.</p><h3>Appendix</h3><h4>Installation</h4><p>Installation instructions can be found on the project’s GitHub page <a href="https://github.com/goruck/home-generative-agent?tab=readme-ov-file#installation">here</a>.</p><h4>Configuration</h4><p>Configuration is done in the HA UI and via the parameters in <em>const.py</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f318732d9d84" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/langchain-meets-home-assistant-unlock-the-power-of-generative-ai-in-your-smart-home-f318732d9d84">LangChain Meets Home Assistant: Unlock the Power of Generative AI in Your Smart Home</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inspecting Neural Network Model Performance for Edge Deployment]]></title>
            <link>https://medium.com/data-science/inspecting-neural-network-model-performance-for-edge-deployment-d4f8f18dbfd5?source=rss-e03a0010d87d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4f8f18dbfd5</guid>
            <category><![CDATA[quantized-neural-networks]]></category>
            <category><![CDATA[edge-computing]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[deep-dives]]></category>
            <category><![CDATA[nilm]]></category>
            <dc:creator><![CDATA[Lindo St. Angel]]></dc:creator>
            <pubDate>Fri, 05 Jan 2024 15:49:17 GMT</pubDate>
            <atom:updated>2024-01-08T21:57:12.876Z</atom:updated>
            <content:encoded><![CDATA[<h4>A detailed look at quantizing CNN- and transformer-based models and techniques to measure and understand their efficacy on edge hardware</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lxe3CJnJJ6wuWGtH" /><figcaption>Photo by <a href="https://unsplash.com/@gavla?utm_source=medium&amp;utm_medium=referral">Gavin Allanwood</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>This article will show you how to convert and quantize neural network models for inference at the edge and how to inspect them for quantization efficacy, understand runtime latency, and model memory usage to optimize performance. Although focused on solving the non-intrusive load monitoring (NILM) problem using convolutional neural networks (CNN) and transformer-based neural networks as a way of illustrating the techniques introduced here, you can use the general approach to train, quantize, and analyze models to solve other problems.</p><p>The goal of NILM is to recover the energy consumption of individual appliances from the aggregate mains signal, which reflects the total electricity consumption of a building or house. NILM is also known as energy disaggregation, and you can use both terms interchangeably.</p><p>You can find the code used to generate the results shown in this article on my GitHub, <a href="https://github.com/goruck/nilm">Energy Management Using Real-Time Non-Intrusive Load Monitoring</a>, and additional details omitted here for brevity.</p><h3>NILM Algorithm Selection and Models</h3><h4>Algorithm Selection</h4><p>Energy disaggregation is a highly under-determined and single-channel <a href="https://en.wikipedia.org/wiki/Signal_separation">Blind Source Separation</a> (BSS) problem, which makes it challenging to obtain accurate predictions. Let <em>M</em> be the number of household appliances, and <em>i</em> be the index referring to the i-th appliance. The aggregate power consumption <em>x</em> at a given time <em>t</em> is the sum of the power consumption of all appliances <em>M</em>, denoted by <em>yᵢ, </em>for all {i=1,…,M}. Therefore, the total power consumption <em>x</em> at a given time <em>t</em> can expressed by Equation 1, where <em>e</em> is a noise term.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*-ZofJP1EWfVw__LWzbk5Fg.jpeg" /></figure><p>The goal is to solve the inverse problem and estimate the appliance power consumption <em>yᵢ</em>, given the aggregate power signal <em>x</em>, and to do so in a manner suitable for deployment at the edge.</p><p>You can solve the single-channel BSS problem by using sequence-to-point (seq2point) learning with neural networks, and it can applied to the NILM problem using transformers, convolutional (CNN), and recurrent neural networks. Seq2point learning involves training a neural network to map between an input time series, such as the aggregate power readings in the case of NILM, and an output signal. You use a sliding input window to train the network, which generates a corresponding single-point output at the window’s midpoint.</p><p>I selected the seq2point learning approach, and my implementation was inspired and guided by the work described by Michele D’Incecco, et al. ¹ and Zhenrui Yue et al. ². I developed various seq2point learning models but focused my work on the models based on transformer and CNN architectures.</p><h4>Neural Network Models</h4><p>You can see the CNN model in Figure 1 for an input sequence length of 599 samples. You can view the complete model code <a href="https://github.com/goruck/nilm/blob/main/ml/define_models.py#L72">here</a>. The model follows traditional CNN concepts from vision use cases where several convolutional layers extract features from the input power sequence at gradually finer details as the input traverses the network. These features are the appliances’ on-off patterns and power consumption levels. Max pooling manages the complexity of the model after each convolutional layer. Finally, dense layers output the window’s final single-point power consumption estimate, which is de-normalized before being used in downstream processing. There are about 40 million parameters in this model using the default values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*IDjUNwaoO_4LTOt--ZDJ_A.png" /><figcaption>Figure 1 — CNN Model</figcaption></figure><p>You can see the transformer model in Figure 2 for an input sequence length of 599 samples where the transformer block is a Bert-style encoder. You can view the complete code <a href="https://github.com/goruck/nilm/blob/main/ml/transformer_model.py">here</a>. The input sequence is first passed through a convolutional layer to expand into a latent space, analogous to the feature extraction in the CNN model case. Pooling and L2 normalization reduce model complexity and mitigate the effects of outliers. Next, a Bert-style transformer lineup processes the latent space sequence, which includes positional embedding and transformer blocks that apply importance weighting. Several layers process the output of the transformer blocks. These are relative position embedding, which uses symmetric weights around the mid-point of the signal; average pooling, which reduces the sequence to a single value per feature; and then finally, dense layers that output the final single point estimated power value for the window which again is de-normalized for downstream processing. There are about 1.6 million parameters in this model using the default values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*qw5QiGTndN0vSaTxA2mAyQ.png" /><figcaption>Figure 2 — Overall Transformer Model</figcaption></figure><p>You can see the Bert-style transformer encoder in Figure 3, below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*qdk7nHkhzA6UosSMxQMATQ.png" /><figcaption>Figure 3 — Transformer Encoder.</figcaption></figure><h4>NILM Datasets</h4><p>Several large-scale publicly available datasets specifically designed to address the NILM problem were captured in household buildings from various countries. The datasets generally include many 10s millions of active power, reactive power, current, and voltage samples but with different sampling frequencies, which require you to pre-process the data before use. Most NILM algorithms utilize only real (active or true) power data. Five appliances are usually considered for energy disaggregation research: a kettle, microwave, fridge, dishwasher, and washing machine. These are the appliances I used for this article, and I mainly focused on the <a href="https://pureportal.strath.ac.uk/en/datasets/refit-electrical-load-measurements-cleaned">REFIT</a>³ dataset.</p><p>Note that these datasets are typically very imbalanced because, most of the time, an appliance is in the off state.</p><h4>Model Training and Results</h4><p>I used <a href="https://www.tensorflow.org/">TensorFlow</a> to train and test the model. You can find the code associated with this section <a href="https://github.com/goruck/nilm/tree/main/ml">here</a>. I trained the seq2point learning models for the appliances individually on z-score standardized REFIT data or normalized to [0, <em>Pₘ</em>], where <em>Pₘ</em> is the maximum power consumption of an appliance in its active state. Normalized data tends to give the best model performance, so I used it by default.</p><p>I used the following metrics to evaluate the model’s performance. You can view the code that calculates these metrics <a href="https://github.com/goruck/nilm/blob/main/ml/nilm_metric.py">here</a>.</p><ul><li>Mean absolute error (<em>MAE)</em> evaluates the absolute difference between the prediction and the ground truth power at every time point and calculates the mean value, as defined by the equation below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*sX9zYDsdRE6Athc3aMjQ_A.jpeg" /></figure><ul><li>Normalized signal aggregate error (<em>SAE</em>) indicates the total energy’s relative error. Denote <em>r</em> as the total energy consumption of the appliance and <em>rₚ</em> as the predicted total energy, then <em>SAE</em> is defined per the equation below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*oK36RXREitIDRvwgV1SwRA.jpeg" /></figure><ul><li>Energy per Day (<em>EpD</em>), which measures the predicted energy used in a day, is valuable when the household users are interested in the total energy consumed in a period. Denote <em>D</em> as the total number of days and <em>e</em> as the appliance energy consumed daily; then <em>EpD</em> is defined per the equation below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*hdXvKcIzEjlpa6ADEr7z5g.jpeg" /></figure><ul><li>Normalized disaggregation error (<em>NDE</em>) measures the normalized error of the squared difference between the prediction and the ground truth power of the appliances, as defined by the equation below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*UsPIGyYvL5qWRuHZpCrqoA.jpeg" /></figure><ul><li>I also used accuracy (<em>ACC</em>), F1-score (<em>F1</em>), and Matthew’s correlation coefficient (<em>MCC</em>) to assess if the model can perform well with the severely imbalanced datasets used to train and test the model. These metrics depend on the computed on-off status of the appliance device. <em>ACC</em> equals the number of correctly predicted time points over the test dataset. The equations below define <em>F1</em> and <em>MCC</em>, where <em>TP</em> stands for true positives, <em>TN</em> stands for true negatives, <em>FP</em> stands for false positives, and <em>FN</em> stands for false negatives.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*L9uSapG8_oTfqqkD_3FKoQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*h7NkCx-Vkxk_9WXNG0jWRw.jpeg" /></figure><p><em>MAE, SAE, NDE</em>, and <em>EpDₑ, </em>defined as 100% times (predicted EpD — ground truth EpD) / ground truth EpD, reflect the model’s ability to predict the appliance energy consumption levels correctly. <em>F1</em> and <em>MCC</em> indicate the model’s ability to predict appliance on-off states using imbalanced classes correctly. <em>ACC </em>is less valuable than <em>F1</em> or <em>MCC</em> in this application because, most of the time, the model will accurately predict that the appliance, which dominates the dataset, is off.</p><p>I used a sliding window of 599 samples of the aggregate real power consumption signal as inputs to the seq2point model, and I used the midpoints of the corresponding windows of the appliances as targets. You can see the code that generates these samples and targets by an instance of the WindowGenerator Class defined in the <a href="https://github.com/goruck/nilm/blob/main/ml/window_generator.py">window_generator.py</a> module.</p><p>You can see the code I used to train the model in <a href="https://github.com/goruck/nilm/blob/main/ml/train.py">train.py</a>, which uses the <a href="https://www.tensorflow.org/api_docs/python/tf/distribute/MirroredStrategy">tf.distribute.MirroredStrategy</a> distributed training strategy. I used the Keras Adam optimizer, with early stopping to reduce over-fitting.</p><p>The key hyper-parameters for training and the optimizer are summarized below.</p><ul><li>Input Window Size: 599 samples</li><li>Global Batch size: 1024 samples.</li><li>Learning Rate: 1e-04</li><li>Adam Optimizer: beta_1=0.9, beta_2=0.999, epsilon=1e-08</li><li>Early Stopping Criteria: 6 epochs.</li></ul><p>I used the loss function shown in the equation below to compute training gradients and evaluate validation loss on a per-batch basis. It combines Mean Squared Error, Binary Cross-Entropy, and Mean Absolute Error losses, averaged over distributed model replica batches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*NRigaCh0KGVlHcWrT_CQ2g.jpeg" /></figure><p>Where x, x_hat in [0, 1] is the ground truth and predicted power usage single point values divided by the maximum power limit per appliance and s, s_ hat in {0, 1} are the appliance state label and prediction. The absolute error term is only applied for the set of predictions when either the status label is on, or the prediction is incorrect. The hyper-parameter lambda tunes the absolute loss term on a per-appliance basis.</p><p>You can see typical performance metrics for the CNN model in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*6y2evUojM2jJ4kbl5OkHMA.png" /><figcaption>Table 1 — CNN model performance</figcaption></figure><p>You can see typical performance metrics for the transformer model in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*51V9aZVD5ucTT4SI5NH7og.png" /><figcaption>Table 2 — transformer model performance</figcaption></figure><p>You can see that the CNN and transformer models have similar performance even though the latter has about 26 times fewer parameters than the former. However, each transformer training step takes about seven times longer than CNN due to the transformer model’s use of self-attention, which has O(<em>n</em>²) complexity compared to the CNN model’s O(<em>n</em>), where <em>n</em> is the input sequence length. Based on training (and inference) efficiency, you can see that CNN is preferable with little loss in model performance.</p><h3>Model Conversion and Quantization</h3><p>The steps involved in converting a model graph in floating point to a form suitable for inferencing on edge hardware, including those based on CPUs, MCUs, and specialized compute optimized for int8 operations, are as follows.</p><ul><li>Train the model in float32 or representation such as TensorFloat-32 using Nvidia GPUs. The output will be a complete network graph; I used the TensorFlow SavedModel format, a complete TensorFlow program including variables and computations.</li><li>Convert the floating-point graph to a format optimized for the edge hardware using TensorFlow Lite or equivalent. The output will be a flat file that can run on a CPU, but all operations will still be in float32. Note that you cannot convert all TensorFlow operators into a TFLite equivalent. You can convert most layers and operators used in CNN networks can be converted, but I designed the transformer network carefully to avoid TFLite conversion issues. See <a href="https://www.tensorflow.org/lite/guide/ops_compatibility">TensorFlow Lite and TensorFlow operator compatibility.</a></li><li>Quantize and optimize the converted model’s weights, biases, and activations. I used various quantization modes to partially or fully quantize the model to int8, int16, or combinations thereof, resulting in different inference latencies on the target hardware.</li></ul><p>I performed <a href="https://www.tensorflow.org/lite/performance/post_training_quantization">Post-training quantization</a> on the CNN and transformer models using the <a href="https://www.tensorflow.org/lite/models/convert/">TensorFlow Lite (TFLite) converter AP</a>I with various quantization modes to improve inference speed on edge hardware, including the Raspberry Pi and the Google Edge TPU, while managing the impact on accuracy. You can see the quantization modes I used below.</p><ul><li><strong>convert_only</strong>:<strong> </strong>Convert to tflite but keep all parameters in Float32 (no quantization).</li><li><strong>w8</strong>: Quantize weights from float32 to int8 and biases to int64. Leave activations in Float32.</li><li><strong>w8_a8_fallback</strong>: Same as w8 but quantize activations from float32 to int8. Fallback to float if an operator does not have an integer implementation.</li><li><strong>w8_a8</strong>: Same as w8 but quantize activations from float32 to int8. Enforce full int8 quantization for all operators.</li><li><strong>w8_a16</strong>: Same as w8 but quantize activations to int16.</li></ul><p>The CNN model was quantized using all modes to understand the best tradeoff between latency and accuracy. Only the weights for the transformer model were quantized to int8 using mode w8; the activations needed to be kept in float32 to maintain acceptable accuracy. See <a href="https://github.com/goruck/nilm/blob/main/ml/convert_keras_to_tflite.py">convert_keras_to_tflite.py</a> for the code that does this quantization, which also uses <a href="https://www.tensorflow.org/lite/performance/quantization_debugger">TensorFlow Lite’s quantization debugger</a> to check how well each layer in the model was quantized. I profiled the converted models using the <a href="https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark">TensorFlow Lite Model Benchmark Tool</a> to quantify inference latencies.</p><p>Fully quantizing a model requires calibration of the model’s activations via a dataset that is representative of the actual data used during training and testing of the floating point model. Calibration can be challenging with highly imbalanced data because a random selection of samples will likely lead to poor calibration and quantized model accuracy. To mitigate this, I used an algorithm to construct a representative dataset of the balanced appliance on- and off-states. You can find that code <a href="https://github.com/goruck/nilm/blob/main/ml/convert_model.py#L91">here</a> and in the snippet below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7aa01014253fbb56bb1b99859655c12/href">https://medium.com/media/e7aa01014253fbb56bb1b99859655c12/href</a></iframe><p>You can find the quantized inference results in the tables below, where Lx86 is the average inference latency on a 3.8 GHz x86 machine using eight TFlite interpreter threads, and Larm is the average inference latency on the ARM aarch-64-based Raspberry Pi 4 using four threads with both computers using the TensorFlow Lite <a href="https://github.com/google/XNNPACK">XNNPACK</a> CPU delegate. Ltpu is the average inference latency on the Google Coral Edge TPU. I kept the model inputs and outputs in float32 to maximize inference speed for the x86- and ARM-based machines. I set them to int8 for the edge TPU.</p><h4>CNN Model Results and Discussion</h4><p>You can see the quantized results for the CNN models in the table below for quantization mode w8.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8k-2f7hFr7zGhuqP8YbICw.png" /><figcaption>Table 4 — Quantized CNN Models for Mode w8</figcaption></figure><p>The quantized results for the CNN kettle model are shown below for the other quantization modes. You can see that latency on the edge TPU is much longer than other machines. Because of this, I focused my analysis on the x86 and ARM architectures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*61CsV8CQ2vVXfFtV8NLzbA.png" /><figcaption>Table 5 — Quantized CNN Kettle Model for Other Modes</figcaption></figure><p>Results for the other appliance models are omitted for brevity but show similar characteristics as a function of quantization mode.</p><p>You can see the negative impact of activation quantization, but because of regularization effects, weight quantization has a moderate benefit on some model performance metrics. As expected, the full quantization modes lead to the lowest latencies. Quantizing activations to int16 by the w8_a16 mode results in the highest latencies because only non-optimized reference kernel implementations are presently available in TensorFlow Lite, but this scheme leads to the best model metrics given the regularization benefits from weight quantization and better preservation of activation numerics.</p><p>You can also see that inference latency of the modes follows w8 &gt; convert_only &gt; w8_a8 for the x86 machine but convert_only &gt; w8 &gt; w8_a8 for the aarch64 machine, although the variation is more significant for x86. To understand this better, I profiled the converted models using the <a href="https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark%29">TFLite Model Benchmark Tool</a>. A summary of the profiling results for the CNN microwave model, which represents the other models, is shown below.</p><ol><li>Model Profiling on x86 (slowest to fastest)</li></ol><p>You can see that the Fully Connected and Convolution operations are taking the longest to execute in all cases but are much faster in the fully quantized mode of w8_a8.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*d7Zwzx1yqkn0aYk6hiQmGg.png" /><figcaption>Table 6 — CNN x86 Model Profiling for w8 Mode</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dzg1vW014ro-7JTGkugT-g.png" /><figcaption>Table 7 — CNN x86 Model Profiling for convert_only Mode</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a8PTW-hon5SAiaoI4ThUtQ.png" /><figcaption>Table 8 — CNN x86 Model Profiling for w8_a8 Mode</figcaption></figure><p>2. Model Profiling on aarch64 (slowest to fastest)</p><p>The copy and Max Pooling operations are slower on x86 than on aarch64, probably due to memory bandwidth and micro-architecture differences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zd9Tq66koEoOold9F_CMHg.png" /><figcaption>Table 9 — CNN aarch64 Model Profiling for convert_only Mode</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-nSm5gppaQz2h6jAJ5qsRQ.png" /><figcaption>Table 10 — CNN aarch64 Model Profiling for w8 Mode</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d310d_l8rwJlFM7ybLBSkw.png" /><figcaption>Table 11 — CNN aarch64 Model Profiling for w8_a8 Mode</figcaption></figure><p>3. Quantization Efficacy</p><p>The metric RMSE / scale is close to 1 / sqrt(12) (~ 0.289) when the quantized distribution is similar to the original float distribution, indicating a well-quantized model. The larger the value, the more likely the layer will not be quantized well. The tables below show the RMSE / Scale metric for the CNN kettle model and the Suspected? Column indicates a layer that significantly exceeds 0.289. Other models are omitted for brevity but show similar results. These layers can remain in float to generate a selectively quantized model that increases accuracy at the expense of inference performance, but doing so for the CNN models did not materially improve accuracy. See <a href="https://www.tensorflow.org/lite/performance/quantization_debugger">Inspecting Quantization Errors with Quantization Debugger</a>.</p><p>You can find layer quantization efficacy metrics for the CNN kettle model using mode w8_a8 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UgSC3l5g9R7QjT1XOTHFAQ.png" /><figcaption>Table 12 — Layer quantization efficacy metrics for the CNN kettle model using mode w8_a8</figcaption></figure><p>4. Model Memory Footprint</p><p>I used the <a href="https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark">TFLite Model Benchmark Tool</a> to get the approximate RAM consumption of the TFLite CNN microwave model at runtime, shown in the table below for each quantization mode, and the TFLite model disk space. The other CNN models show similar characteristics. The findings for the x86 architecture were identical to the arm architecture. Note that the Keras model consumes about 42.49 (MB) on disk. You can see that there is about a four times reduction in disk storage space due to the float32 to int8 weight conversions.</p><p>Interestingly, RAM runtime usage varies considerably due to the TFLite algorithms that optimize intermediate tensor usage. These are pre-allocated to reduce inference latency at the cost of memory space. See <a href="https://blog.tensorflow.org/2020/10/optimizing-tensorflow-lite-runtime.html">Optimizing TensorFlow Lite Runtime Memory</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*vWHmsKiEhXSiUYpZaAqzfw.png" /><figcaption>Table 13 — CNN Model Memory Usage</figcaption></figure><h4>Transformer Model Results and Discussion</h4><p>Even though I enabled the XNNPACK delegate during the transformer model inference evaluation, nothing was accelerated because the transformer model contains dynamic tensors. I encountered the following warning when using the TFLite interpreter for inference:</p><blockquote>Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors (tensor#94 is a dynamic-sized tensor).</blockquote><p>This warning means that all operators are unsupported by XNNPACK and will fall back to the default CPU kernel implementations. A future effort will involve refactoring the transformer model to use only static-size tensors. Note that a tensor could be marked dynamic when the TFLite runtime encounters a control-flow operation (e.g., if, while). In other words, even when the model graph doesn’t have any tensors of dynamic shapes, a model could have dynamic tensors at runtime. The current transformer model uses `if` control-flow operations.</p><p>You can see the quantized results for the transformer model in the table below for quantization mode w8.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oSFSGr3O6uv4lAEtoBoerg.png" /><figcaption>Table 14 — Quantized results for the transformer model for quantization mode w8</figcaption></figure><p>The quantized results for the transformer kettle and microwave models are shown in the table below for quantization mode convert_only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RhH23fMUnlCO2VjM8cSWuw.png" /><figcaption>Table 15 — Quantized results for the transformer kettle and microwave models for quantization mode convert_only</figcaption></figure><ol><li>Model Profiling on x86 (slowest to fastest)</li></ol><p>The FULLY_CONNECTED layers dominate the compute in w8 mode but less in convert_only mode. This behavior is probably due to x86 memory micro-architecture handling of int8 weights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ASlttVtm9QBBq2vg62X5_A.png" /><figcaption>Table 16 — x86 transformer Model Profiling for Mode w8</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CCroyzwEIuIUhMkVo3DvTQ.png" /><figcaption>Table 17 — x86 transformer Model Profiling for Mode convert_only</figcaption></figure><p>2. Model Profiling on aarch64 (slowest to fastest)</p><p>You can see the arm architecture seems to be more efficient in computing the FULLY_CONNECTED layers in w8 mode than in the x86 case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eXJPcdh7viXF2EY7TlNLzQ.png" /><figcaption>Table 18 — aarch64 transformer Model Profiling for Mode convert_only</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u3OHFt5Z_SEpET6p4srpUg.png" /><figcaption>Table 19 — aarch64 transformer Model Profiling for Mode w8</figcaption></figure><p>3. Quantization Efficacy</p><p>You can find layer quantization efficacy metrics for the transformer kettle model using mode w8_a8 <a href="https://github.com/goruck/nilm?tab=readme-ov-file#quantization-efficacy-1">here</a>, although, as noted above, quantizing the transformer model’s activations results in inferior model performance. You can see that the RSQRT operator, in particular, does not quantize well; these operators are used in the Gaussian error linear activation functions, which helps explain the model’s poor performance. The other transformer appliance models show similar efficacy metrics.</p><p>4. Model Memory Footprint</p><p>Identical to the CNN case, I used the <a href="https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark">TFLite Model Benchmark Tool</a> to get the approximate RAM consumption of the TFLite microwave model at runtime, shown in the table below for each relevant quantization mode and the TFLite model disk space. The other transformer models show similar characteristics. Note that the Keras model consumes about 6.02 (MB) on disk. You can see that there is about a three-times reduction in model size due to the weights being quantized from float32 to int8, which is less than the four-times reduction seen in the CNN case, likely because there are fewer layers with weights. You can also see that the x86 TFLite runtime is more memory efficient than its aarch64 counterpart for this model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BIcdrcZoIt9gVq3IHrUcyA.png" /><figcaption>Table 20 — transformer Model Disk and RAM Usage</figcaption></figure><h3>Conclusions</h3><p>You can effectively develop and deploy models using TensorFlow and TensorFlow Lite at the edge. TensorFlow Lite offers tools useful in production to understand and modify the behavior of your models, including layer quantization inspection and runtime profiling.</p><p>There is better support for the operators used in CNN-based models than the typical operators used in transformer-based models. You should carefully choose how to design your networks with these constraints and run a complete end-to-end training-conversion-quantization cycle before going too far in developing and training your models.</p><p>Post-training quantization works well to quantize CNN networks fully, but I could only quantize the transformer network weights to maintain acceptable performance. The transformer network should be trained using <a href="https://www.tensorflow.org/model_optimization/guide/quantization/training">Quantization-aware methods</a> for better integer performance.</p><p>The CNN models used to solve the NILM problem in this article are many times larger than their transformer counterparts but train much faster and have lower latency due to linear complexity. The CNN models are a better solution if disk space and RAM are not your chief constraints.</p><h3>References</h3><ol><li><a href="https://arxiv.org/abs/1902.08835">arXiv:1902.08835</a> | Transfer Learning for Non-Intrusive Load Monitoring by Michele D’Incecco, Stefano Squartini and Mingjun Zhong.</li><li><a href="https://github.com/Yueeeeeeee/BERT4NILM">BERT4NILM</a>: A Bidirectional Transformer Model for Non-Intrusive Load Monitoring by Zhenrui Yue, et. al.</li><li>Available under the <a href="https://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International Public License</a>.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4f8f18dbfd5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/inspecting-neural-network-model-performance-for-edge-deployment-d4f8f18dbfd5">Inspecting Neural Network Model Performance for Edge Deployment</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering TensorFlow Lite Conversion for Energy Disaggregation Models on Edge Devices]]></title>
            <link>https://medium.com/@lindo.st.angel/mastering-tensorflow-lite-conversion-for-energy-disaggregation-models-on-edge-devices-d04e93194d88?source=rss-e03a0010d87d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d04e93194d88</guid>
            <category><![CDATA[model-quantization]]></category>
            <category><![CDATA[tensorflow-lite]]></category>
            <category><![CDATA[edge-computing]]></category>
            <category><![CDATA[energy-disaggregation]]></category>
            <dc:creator><![CDATA[Lindo St. Angel]]></dc:creator>
            <pubDate>Sun, 19 Nov 2023 14:43:45 GMT</pubDate>
            <atom:updated>2023-11-19T14:43:45.571Z</atom:updated>
            <content:encoded><![CDATA[<h4>This article provides an exploration of TensorFlow Lite quantization strategies for converting Keras models to TensorFlow Lite format, specifically tailored for energy disaggregation applications on edge devices.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PptK3NskEJVl73ZH" /><figcaption>Photo by <a href="https://unsplash.com/@harrisonbroadbent?utm_source=medium&amp;utm_medium=referral">Harrison Broadbent</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>When deploying energy disaggregation models on edge devices, selecting the right TensorFlow Lite (tflite) quantization strategy is crucial. Each strategy balances model size, accuracy, and performance differently. This guide details the quantization strategies supported by a <a href="https://github.com/goruck/nilm/blob/main/ml/convert_keras_to_tflite.py">Python module</a> I authored which is designed for converting Keras models to tflite. You will learn how each approach tailors to specific deployment needs, with example code that facilitates hands-on learning. This article is based on work I have done for my <a href="https://github.com/goruck/nilm">Energy Management Using Real-Time Non-Intrusive Load Monitoring</a> project.</p><h3>The ConvertModel Class: A Deep Dive</h3><p>At the forefront of the conversion process is the <a href="https://github.com/goruck/nilm/blob/main/ml/convert_keras_to_tflite.py#L33">ConvertModel</a> class, equipped to handle the transformation of Keras models into tflite format.</p><h4>Comprehensive Quantization Strategies</h4><p>Quantization is a pivotal aspect of model optimization for edge devices. The class facilitates various quantization modes:</p><ol><li>Weight-Only Quantization (w8):</li></ol><ul><li>Description: This mode involves quantizing only the weights of the model from floating-point to 8-bit integers (int8). The inputs and outputs of the model remain in floating-point format.</li><li>Advantages: It typically results in a smaller model size compared to the full floating-point model while maintaining a good balance between accuracy and performance.</li><li>Usage Scenario: Ideal when the primary goal is to reduce the model size without a significant loss in accuracy.</li><li>Code Snippet:</li></ul><pre>if self.quant_mode == &#39;w8&#39;:<br>self.converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]</pre><p>2. Weight and Activation Quantization with Fallback (w8_a8_fallback):</p><ul><li>Description: Quantizes both weights and activations to int8, but allows fallback to floating-point for operations that do not support int8. The inputs and outputs are kept in float.</li><li>Advantages: Provides a balance between model size and compatibility, allowing for int8 quantization where possible and falling back to float for unsupported operations.</li><li>Usage Scenario: Useful when full int8 quantization leads to compatibility issues or significant accuracy drops.</li><li>Code Snippet:</li></ul><pre>elif self.quant_mode == &#39;w8_a8_fallback&#39;:<br>self.converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]<br>self.converter.representative_dataset = tf.lite.RepresentativeDataset(self._rep_gen</pre><p>3. Full INT8 Quantization (w8_a8):</p><ul><li>Description: Applies int8 quantization to all parts of the model, including weights, activations, and also the input and output layers.</li><li>Advantages: This approach results in the smallest possible model size and is optimized for performance on devices with int8 accelerators.</li><li>Usage Scenario: Best suited for edge devices with dedicated hardware for int8 operations.</li><li>Code Snippet:</li></ul><pre>elif self.quant_mode == &#39;w8_a8&#39;:<br>self.converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]    <br>self.converter.inference_input_type = tf.int8<br>self.converter.inference_output_type = tf.int8</pre><p>4. Weight INT8 and Activation INT16 Quantization (w8_a16):</p><ul><li>Description: In this mode, the weights are quantized to int8, while the activations are quantized to 16-bit integers (int16). This mode also allows fallback to float for unsupported operations.</li><li>Advantages: It offers a middle ground between the w8_a8_fallback and w8_a8 modes, providing better precision for activations at the cost of slightly larger model size.</li><li>Usage Scenario: Useful when activations require higher precision than what int8 can offer, but still aims to reduce model size.</li><li>Code Snippet:</li></ul><pre>elif self.quant_mode == &#39;w8_a16&#39;:<br>self.converter.target_spec.supported_ops = [<br>  tf.lite.OpsSet.TFLITE_BUILTINS,    <br>  tf.lite.OpsSet.EXPERIMENTAL_TFLITE_BUILTINS_ACTIVATIONS_INT16_WEIGHTS_INT8     <br>]    <br>self.converter.representative_dataset = tf.lite.RepresentativeDataset(self._rep_gen)</pre><p>Each of these quantization strategies offers a different trade-off between model size, performance, and accuracy, allowing for flexible optimization based on the specific requirements of the edge device and the application scenario.</p><h4>Calibration with Representative Dataset</h4><p>The class uses a representative dataset to calibrate the model’s quantization, ensuring that the range of floating-point arrays is accurately estimated. It uses a balanced approach to sample enough data points from periods of active and inactive appliance use, crucial for energy disaggregation datasets that are typically imbalanced.</p><h4>Debugging: Ensuring Quantization Quality</h4><p>The debugging feature is essential for assessing the quantization process, particularly for models fully quantized to INT8. The debugger performs a detailed analysis of each layer, assessing the quantization impact and identifying layers that may benefit from remaining in float to enhance model accuracy.</p><h3>Edge Deployment Optimization</h3><h4>Customizable Command-Line Interface</h4><p>The module’s CLI interface offers extensive customization for the conversion process, allowing precise control over various parameters.</p><p>Flexible Model Configuration</p><p>Users can specify the appliance type, model architecture, and desired quantization mode, tailoring the conversion process to specific deployment needs.</p><p>Detailed Model Evaluation</p><p>Post-conversion, the <a href="https://github.com/goruck/nilm/blob/main/ml/convert_keras_to_tflite.py#L324">evaluate_tflite</a> function is crucial for assessing the model&#39;s performance. It evaluates the model on several metrics like accuracy, mean absolute error, and energy prediction discrepancies, providing a comprehensive view of the model’s effectiveness post-quantization.</p><p>Batch Size Adjustment for Compatibility</p><p>Modifying the model’s batch size is a key step for ensuring compatibility with certain edge devices, a vital consideration for models intended for real-time deployment. The batch size adjustment is particularly crucial for models destined for deployment on Edge TPUs, which often require fixed batch sizes. See the <a href="https://github.com/goruck/nilm/blob/main/ml/convert_keras_to_tflite.py#L301">change_model_batch_size</a> function.</p><h3>Typical Results</h3><p>Please see the <a href="https://github.com/goruck/nilm#model-quantization">Model Quantization</a> section of my <a href="https://github.com/goruck/nilm">Energy Management Using Real-Time Non-Intrusive Load Monitoring</a> project for typical quantized model performance using the techniques and code discussed in this article. <em>Note that this is a work in progress and quantized model performance metrics are being updated over time.</em></p><h3>Conclusion</h3><p>Each quantization strategy in TensorFlow Lite offers a unique set of benefits and trade-offs, catering to different needs in model size, accuracy, and performance for energy disaggregation models on edge devices. Understanding these strategies allows for more informed decisions in model conversion, ensuring efficient and effective deployment on edge devices. With this guide, you’re well-equipped to navigate the complexities of TensorFlow Lite quantization and optimize your energy disaggregation models for edge deployment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d04e93194d88" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Energy Management Using Real-Time Non-Intrusive Load Monitoring]]></title>
            <link>https://medium.com/data-science/energy-management-using-real-time-non-intrusive-load-monitoring-3c9b0b4c8291?source=rss-e03a0010d87d------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c9b0b4c8291</guid>
            <category><![CDATA[data-for-change]]></category>
            <category><![CDATA[edge-computing]]></category>
            <category><![CDATA[realtime]]></category>
            <category><![CDATA[energy-disaggregation]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Lindo St. Angel]]></dc:creator>
            <pubDate>Tue, 28 Jun 2022 19:19:37 GMT</pubDate>
            <atom:updated>2023-02-17T17:16:36.088Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to build a key component of sustainable and clean energy using machine learning at the edge</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R4_2wH8wfnnMaZWI" /><figcaption>Photo by <a href="https://unsplash.com/@rwlinder?utm_source=medium&amp;utm_medium=referral">Robert Linder</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>The goal of non-intrusive load monitoring (NILM) is to recover the energy consumption of individual appliances from the aggregate mains signal, which is a measure of the total electricity consumption of a building or house. NILM is also known as energy disaggregation and both terms will be used interchangeably in this article.</p><p>R. Gopinath, et al., nicely summarizes the rational behind NILM in the paper <em>Energy management using non-intrusive load monitoring techniques — State- of-the-art and future research directions¹:</em></p><blockquote>In recent years, the development of smart sustainable cities has become the primary focus among urban planners and policy makers to make responsible use of resources, conserve the environment and improve the well-being of the society. Energy management is an integral part of the smart sustainable cities development programme which involves conscious and efficient use of available energy resources towards attaining sustainability and self-reliance on energy systems. Building sector is one of the key sectors that utilize more energy. Therefore, efforts are being made to monitor and manage energy consumption effectively in residential and commercial buildings. In recent years, non-intrusive load monitoring (NILM) technique has become a popular and emerging approach to monitor events (on/off) and energy consumption of appliances/electrical utilities in buildings using single energy meter. The information about the energy consumption at the appliance level would help consumers to understand their appliance usage behavior and take necessary steps for reducing energy consumption.</blockquote><p>By using deep learning models trained on publicly available datasets it is now feasible to enable NILM in very cost effective ways by leveraging commodity hardware and open source software. Deploying the trained models at the edge of the power grid, i.e. the building level, results in additional cost savings since it obviates the need for always connected internet cloud services which can be expensive at scale and running the algorithms in real-time provides for low latency operation.</p><p>This article will show you how NILM works by taking you through the steps I used to implement a prototype system at my house.</p><h3>Architecture</h3><p>The following first two diagrams illustrate the NILM concept and process steps at a high level. The last diagram shows my prototype system which is based on these concepts and is implemented using Arduino- and Raspberry Pi-based compute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aDz94wcT-6v7r-P8M_CF_w.png" /><figcaption>General NILM Concept (R. Gopinath, et al.¹)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-dWzCGBnu9PgDoyXoDoVQg.png" /><figcaption>NILM Process Steps (R. Gopinath, et al.¹)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D9VMCxcaVxcTdf5wHI9_IQ.png" /><figcaption>NILM Prototype System Block Diagram</figcaption></figure><h3>NILM Algorithm Selection and Model Training</h3><h4>Algorithm Selection</h4><p>Energy disaggregation is a highly underdetermined and a single-channel Blind Source Separation (BSS) problem² which makes it difficult to obtain accurate predictions. You need to extract more than one source from a single observation. Past approaches have included factorial hidden Markov models (FHMM)¹ and various event-based methods with some success⁶.</p><p>You can also solve the single-channel BSS problem can by using sequence-to-sequence (seq2seq) learning with neural networks, and it can applied to the NILM problem using both convolutional and recurrent neural networks³. Seq2seq learning involves training a deep network to map between an input time series, such as the aggregate power readings the case of NILM, and a output sequence, such as the estimated energy consumption of a single appliance. A sliding input window is typically used to training the network which generates a corresponding window of the output. This method produces multiple predictions for each appliance in the output so an average of the predictions is used for the final result. Some of the predictions will be more accurate than others, especially those near the midpoint of the input sliding window. The averaging will tend to lower the overall accuracy of the predictions.</p><p>Some of the disadvantages of of seq2seq leaning can mitigated by sequence-to-point learning (seq2point) for single-channel BSS⁴. You also use a sliding input signal window in this approach, however the network is trained to predict the output signal only at the midpoint of the window which makes the prediction problem easier on the network, leading to more accurate results.</p><p>I selected the seq2point learning approach for my prototype system and my implementation was inspired and guided by work described by Michele DIncecco, Stefano Squartini and Mingjun Zhong⁵.</p><h4>Datasets</h4><p>There are a number of large-scale publicly available datasets specifically designed to address the NILM problem which were captured in household buildings from various countries. The table⁷ below shows several of the most widely used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aQVwEVbIVtg8LbaQthP_mQ.png" /><figcaption>NILM Datasets (Oliver Parson, et al.⁷)</figcaption></figure><p>The datasets generally include many 10’s of millions of active power, reactive power, current, and voltage samples but with different sampling frequencies which requires you to pre-process the data before use. Most NILM algorithms utilize only active or apparent power data. Five appliances are usually considered for energy disaggregation research which are kettle, microwave, fridge, dish washer and washing machine. These are the appliances I considered for my prototype and following the work of Michele DIncecco, et al<em>.</em>⁵, I mainly focused on the REFIT⁸ data et but will eventually include UK-DALE and REDD.</p><h4>Model Training</h4><p>I used TensorFlow 2 and the Keras APIs to train and test the model. All code associated with this section can be found on the Machine Learning section of the project’s GitHub, <a href="https://github.com/goruck/nilm/tree/main/ml">NILM</a>⁹.</p><p>The seq2point learning models for the appliances were trained individually on z normalized REFIT data. I used the following four metrics to evaluate the model’s performance. You can view the code that calculates these metrics <a href="https://github.com/goruck/nilm/blob/main/ml/nilm_metric.py">here</a>.</p><ul><li>Mean absolute error (MAE), which evaluates the absolute difference between the prediction and the and the ground truth at every time point and calculates the mean value.</li><li>Normalized signal aggregate error (SAE), which indicates the relative error of the total energy.</li><li>Energy per day (EpD) which measures the absolute error of the predicted energy used in a day, useful when the household users are interested in the total energy consumed in a period.</li><li>Normalized disaggregation error (NDE) which measures the normalized error of the squared difference between the prediction and the ground truth of the appliances.</li></ul><p>I created the seq2point learning model using the Keras Functional API as shown in the Python function below. You can find alternative model architectures in the project’s GitHub but this one currently gives the best results.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/932f2d988c1ae942e84c8b10628ca54a/href">https://medium.com/media/932f2d988c1ae942e84c8b10628ca54a/href</a></iframe><p>A sliding window of 599 samples of the aggregate apparent power consumption signal is used as inputs to seq2point model and the midpoints of the corresponding windows of the appliances are used as targets. You can see how the samples and targets are generated in the code snippet below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1d617fb1a47f700cde094820cae5ada/href">https://medium.com/media/c1d617fb1a47f700cde094820cae5ada/href</a></iframe><p>You can see below a code snippet I used to train the model and the full program <a href="https://github.com/goruck/nilm/blob/main/ml/train.py">here</a>. For training I used the Keras <a href="https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam">Adam</a> optimizer and to reduce over-fitting I used <a href="https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping">EarlyStopping</a> and <a href="https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/schedules/InverseTimeDecay">InverseTimeDecay</a>. The training code can be configured to train the seq2point model from scratch, or given a fitted model, prune it or fine tune it with quantization aware training (QAT), both of which can improve inference performance especially on edge hardware.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d724bd99d0d59ee5d657f3937e21a8c3/href">https://medium.com/media/d724bd99d0d59ee5d657f3937e21a8c3/href</a></iframe><p>The hyper-parameters for training and the optimizer are summarized below.</p><ul><li>Input Window Size: 599 samples</li><li>Batch size: 1000 samples.</li><li>From scratch Learning Rate: 0.001 with inverse time decay.</li><li>QAT and Prune Learning Rate: 0.0001.</li><li>Optimizer: beta_1=0.9, beta_2=0.999, epsilon=1e-08.</li><li>From scratch Early Stopping Criteria: 6 epochs.</li></ul><p>The training program monitors the Mean Squared Error (MSE) losses for both training and validation data and Mean Absolute Error (MAE) for the validation data with early stopping to reduce over-fitting. The datasets contain a large number of samples (many 10’s of millions) with repeating patterns; it was not uncommon that over-fitting occurred after only a single epoch for some appliances. To mitigate against this, I used a subset of the training data, typically between 5 and 10 million samples.</p><p>The MSE and MAE results from a typical from-scratch training run for the microwave appliance are shown below. You can see that minimum validation loss occurred around the 10th epoch at which point the training program saves as the final trained model and early-stops after the 16th epoch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*P7VYaxdlJppM0JAKHXH9Rg.png" /><figcaption>Microwave Training and Validation Losses</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*R0xcdR-9yO8crmpOvgGPkA.png" /><figcaption>Microwave Validation MAE</figcaption></figure><p>Typical model performance results per the metrics described above can be found in the Appendix. I met or exceeded the results reported in the work by Michele DIncecco, et al<em>.</em>⁵</p><h4>Model Quantization</h4><p>I quantized the model’s weights and activation functions from Float32 to INT8 using <a href="https://www.tensorflow.org/lite">TensorFlow Lite</a> to improve inference performance on edge hardware, including the Raspberry Pi and the Google Edge TPU. You may observed a slight degradation in performance after quantization but this is acceptable for most use cases. A typical result is shown below, note that the floating point model was not fine-tuned using QAT nor pruned before INT8 conversion.</p><pre>### Fridge Float32 vs INT8 Model Performance ###</pre><pre>Float32 tflite Model (Running on x86 w/XNNPACK):<br>MAE 17.10(Watts)<br>NDE 0.373<br>SAE 0.283<br>Inference Rate 701.0 Hz</pre><pre>INT8 tflite Model (running on Raspberry Pi 4):<br>MAE: 12.75 (Watts)<br>NDE 0.461<br>SAE 0.120<br>Inference Rate 163.5 Hz</pre><p>All code associated with this section can be found in the Machine Learning section of the project’s GitHub, <a href="https://github.com/goruck/nilm/tree/main/ml">NILM</a>⁹, in particular see the module <a href="https://github.com/goruck/nilm/blob/main/ml/convert_keras_to_tflite.py">convert_keras_to_tflite.py</a> and the code snippet from it below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e769d8d06dea00e0639ee246bc7282ac/href">https://medium.com/media/e769d8d06dea00e0639ee246bc7282ac/href</a></iframe><h3>NILM Prototype System Components</h3><p>I built a NILM prototype at my home to test the energy disaggregation algorithms in real-world conditions and to understand where they can be improved. The prototype is comprised of the following main subsystems. You can see a photograph of the prototype in the Appendix.</p><h4>Analog Signal Conditioning</h4><p>I used two clip-on current transformers in one of the home’s sub-panels to sense the current flowing through each of split voltage phases and a voltage transformer plugged into an outlet near the circuit breaker panel that provides the voltage of one of the phases. These signals are level-shifted, amplified and low-passed filtered by this subsystem before being passed on to the analog-to-digital converters inside an <a href="https://docs.arduino.cc/hardware/mega-2560">Arduino MEGA 2560</a> that performs aggregate metrics computation. You can see a schematic for the Analog Signal Conditioning Subsystem in the Appendix.</p><h4>Aggregate Metrics Computation</h4><p>I used an Arduino MEGA 2560 to host the signal processing algorithms that takes the voltage signals from the Analog Signal Conditioning Subsystem and generates aggregate RMS voltage, RMS current, Real Power and Apparent Power metrics in real-time. Presently, only Real Power is used in downstream processing. I leveraged emonLibCM¹⁰ for these signal processing algorithms. emonLibCM runs continuously in the background and digitizes the analog input channels of the Arduino, calculates these metrics and then informs the Arduino sketch that the measurements are available and should be read and processed by downstream processing. The sketch is configured to update the metrics every eight seconds and automatic gain control of the analog front end is performed after each update to optimize signal-to-noise.</p><p>You can see the Arduino sketch below and in the project’s GitHub, <a href="https://github.com/goruck/nilm/blob/main/ard/ard.ino">NILM</a>⁹</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bcc0c00d1bf588e93d473b91c7ef8bf5/href">https://medium.com/media/bcc0c00d1bf588e93d473b91c7ef8bf5/href</a></iframe><h4>Disaggregated Energy Consumption Computation</h4><p>The actual energy disaggregation computations are hosted on a <a href="https://www.raspberrypi.com/products/raspberry-pi-4-model-b/">Raspberry Pi 4</a> which is connected over USB to the Arduino to fetch the aggregate metrics. The computations are comprised of running the tflite appliance inference models, trained and quantized per the steps described above, with pre- and post-processing steps. The inference models output predicted energy for each appliance from 599-sample sized windows of the aggregate real power input signal. These predictions are stored in a local CSV file and made available for downstream reporting and analysis.</p><p>Typical disaggregated energy prediction results from my home are shown in the figures below using tflite models trained on another machine from the dataset as described above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HYWdHLZERaml4nf4ncYHXw.png" /><figcaption>Float32 and tflite appliance model predictions</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ADptOCXlGvxw1wzZ1500FA.png" /><figcaption>Zoom on Fridge and Microwave</figcaption></figure><p>No fine-tuning was done on local data. The horizontal axis is time measured by the number of eight-second samples (the sampling interval is eight seconds), which span about eight and half days in this case. The vertical axis is energy consumption in Watts. The top trace is total (both phases) mains apparent power. The following traces are the predicted energy for each appliance. Only a refrigerator and a microwave from the reference set of appliances were in use during this time although there were other devices supplied by the same sub-panel such as computers and heaters.</p><p>You can see both the Float32 and tflite versions algorithm detect the refrigerator and microwave but there is an energy offset between the two which probably indicates the need for better calibration. I used a power meter to ground truth the RMS power consumption of the refrigerator and microwave; it matches the Float32 predictions to within about +/- 15%.</p><p>All code associated with this section can be found in the Machine Learning section of the project’s GitHub, <a href="https://github.com/goruck/nilm/tree/main/rpi">NILM</a>⁹, in particular see the module <a href="https://github.com/goruck/nilm/blob/main/rpi/infer.py">infer.py</a> and the code snippet from it below which performs the real-time predictions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4dd8b9817e9746d69d65ef45ed8d1bd5/href">https://medium.com/media/4dd8b9817e9746d69d65ef45ed8d1bd5/href</a></iframe><h4>Amazon Alexa</h4><p>I plan to use Amazon Alexa as the user interface to the appliance energy data, however this work is not yet started.</p><h3>Conclusion</h3><p>By using large publicly available datasets to train seq2point learning models it is very feasible to perform energy disaggregation that is fairly accurate without fine-tuning the models with local data. These models are modest in size and with little loss in accuracy can be quantized to run efficiently on commodity edge hardware such as the Raspberry Pi 4. More work needs to be done to further improve the accuracy of the models and to test with more appliance types. This project demonstrates that a key component of a sustainable and scalable power grid is within reach of the mass consumer market.</p><h3>References</h3><ol><li>Sustainable Cities and Society 62 (2020) 102411 | Energy management using non-intrusive load monitoring techniques — State- of-the-art and future research directions by R. Gopinath, Mukesh Kumar, C. Prakash Chandra Joshua and Kota Srinivas.</li><li>Wikipedia | <a href="https://en.wikipedia.org/wiki/Signal_separation">Signal Separation</a>.</li><li>arXiv:1507.06594 | <a href="https://arxiv.org/pdf/1507.06594.pdf">Neural NILM: Deep Neural Networks Applied to Energy Disaggregation</a> by Jack Kelly and William Knottenbelt.</li><li>arXiv:1612.09106 | <a href="https://arxiv.org/pdf/1612.09106.pdf">Sequence-to-point learning with neural networks for non-intrusive load monitoring</a> by Chaoyun Zhang, Mingjun Zhong, Zongzuo Wang, Nigel Goddard and Charles Sutton.</li><li>arXiv:1902.08835 | <a href="https://arxiv.org/pdf/1902.08835.pdf">Transfer Learning for Non-Intrusive Load Monitoring</a> by Michele DIncecco, Stefano Squartini and Mingjun Zhong.</li><li>Artificial Intelligence Techniques for a Scalable Energy Transition pp 109–131 | A Review on Non-intrusive Load Monitoring Approaches Based on Machine Learning by Hajer Salem, Moamar Sayed-Mouchaweh and Moncef Tagina.</li><li>1st International Symposium on Signal Processing Applications in Smart Buildings at 3rd IEEE Global Conference on Signal &amp; Information Processing | <a href="https://www.researchgate.net/profile/Nipun-Batra/publication/281538210_Dataport_and_NILMTK_A_Building_Data_Set_Designed_for_Non-intrusive_Load_Monitoring/links/55ed13e208aeb6516268cffc/Dataport-and-NILMTK-A-Building-Data-Set-Designed-for-Non-intrusive-Load-Monitoring.pdf?origin=publication_detail">Dataport and NILMTK: A Building Data Set Designed for Non-intrusive Load Monitoring</a> by Oliver Parson, Grant Fisher, April Hersey, Nipun Batra, Jack Kelly, Amarjeet Singh, William Knottenbelt and Alex Rogers.</li><li>Proceedings of the 8th International Conference on Energy Efficiency in Domestic Appliances and Lighting | <a href="https://pure.strath.ac.uk/ws/portalfiles/portal/45214811/Murray_etal_EEDAL_2015_A_data_management_platform_for_personalised_real_time.pdf">A data management platform for personalised real-time energy feedback</a> by David Murray and Jing Liao and Lina Stankovic and Vladimir Stankovic and Richard Hauxwell-Baldwin and Charlie Wilson and Michael Coleman and Tom Kane and Steven Firth. The REFIT dataset used in this project is is licensed under the Creative Commons Attribution 4.0 International Public License.</li><li>GitHub | <a href="https://github.com/goruck/nilm">NILM</a> by Lindo St. Angel</li><li>GitHub | <a href="https://github.com/openenergymonitor/EmonLibCM">EmonLibCM</a> by Trystan Lea, Glyn Hudson, Brian Orpin and Ivan Kravets.</li></ol><h3>Appendix</h3><h4>Photograph of Prototype</h4><p>A photograph of an early version of my prototype system is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U6sXHYgk6SJNGiXbZrHzxA.png" /><figcaption>Early Prototype</figcaption></figure><h4>Analog Signal Conditioning Schematic</h4><p>The schematic for the Analog Signal Conditioning circuitry is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bvz2srBpCu6Hg4VBJpBSzA.jpeg" /><figcaption>Analog Signal Conditioning Schematic</figcaption></figure><h4>Model Training Results</h4><p>Typical model performance evaluated against the metrics described above are as follows.</p><pre>### Dishwasher ###</pre><pre>2022–06–05 12:48:09,322 [INFO ] Appliance target is: dishwasher<br>2022–06–05 12:48:09,322 [INFO ] File for test: dishwasher_test_H20.csv<br>2022–06–05 12:48:09,322 [INFO ] Loading from: ./dataset_management/refit/dishwasher/dishwasher_test_H20.csv<br>2022–06–05 12:48:10,010 [INFO ] There are 5.169M test samples.<br>2022–06–05 12:48:10,015 [INFO ] Loading saved model from ./models/dishwasher/checkpoints.<br>2022–06–05 12:48:35,045 [INFO ] aggregate_mean: 522<br>2022–06–05 12:48:35,046 [INFO ] aggregate_std: 814<br>2022–06–05 12:48:35,046 [INFO ] appliance_mean: 700<br>2022–06–05 12:48:35,046 [INFO ] appliance_std: 1000<br>2022–06–05 12:48:35,121 [INFO ] true positives=5168007.0<br>2022–06–05 12:48:35,121 [INFO ] false negatives=0.0<br>2022–06–05 12:48:35,121 [INFO ] recall=1.0<br>2022–06–05 12:48:35,173 [INFO ] true positives=5168007.0<br>2022–06–05 12:48:35,173 [INFO ] false positives=0.0<br>2022–06–05 12:48:35,173 [INFO ] precision=1.0<br>2022–06–05 12:48:35,173 [INFO ] F1:1.0<br>2022–06–05 12:48:35,184 [INFO ] NDE:0.45971032977104187<br>2022–06–05 12:48:35,657 [INFO ] <br>MAE: 10.477645874023438<br> -std: 104.59112548828125<br> -min: 0.0<br> -max: 3588.0<br> -q1: 0.0<br> -median: 0.0<br> -q2: 0.78582763671875<br>2022–06–05 12:48:35,675 [INFO ] SAE: 0.1882956475019455<br>2022–06–05 12:48:35,691 [INFO ] Energy per Day: 145.72999548734632</pre><pre>### Microwave ###</pre><pre>2022-06-05 18:14:05,471 [INFO ]  Appliance target is: microwave<br>2022-06-05 18:14:05,471 [INFO ]  File for test: microwave_test_H4.csv<br>2022-06-05 18:14:05,471 [INFO ]  Loading from: ./dataset_management/refit/microwave/microwave_test_H4.csv<br>2022-06-05 18:14:06,476 [INFO ]  There are 6.761M test samples.<br>2022-06-05 18:14:06,482 [INFO ]  Loading saved model from ./models/microwave/checkpoints.<br>2022-06-05 18:14:38,385 [INFO ]  aggregate_mean: 522<br>2022-06-05 18:14:38,385 [INFO ]  aggregate_std: 814<br>2022-06-05 18:14:38,385 [INFO ]  appliance_mean: 500<br>2022-06-05 18:14:38,385 [INFO ]  appliance_std: 800<br>2022-06-05 18:14:38,478 [INFO ]  true positives=6759913.0<br>2022-06-05 18:14:38,478 [INFO ]  false negatives=0.0<br>2022-06-05 18:14:38,478 [INFO ]  recall=1.0<br>2022-06-05 18:14:38,549 [INFO ]  true positives=6759913.0<br>2022-06-05 18:14:38,549 [INFO ]  false positives=0.0<br>2022-06-05 18:14:38,549 [INFO ]  precision=1.0<br>2022-06-05 18:14:38,549 [INFO ]  F1:1.0<br>2022-06-05 18:14:38,568 [INFO ]  NDE:0.6228251457214355<br>2022-06-05 18:14:39,469 [INFO ]  <br>MAE: 7.6666789054870605<br>    -std: 73.37799835205078<br>    -min: 0.0<br>    -max: 3591.474365234375<br>    -q1: 0.757080078125<br>    -median: 1.178070068359375<br>    -q2: 1.459686279296875<br>2022-06-05 18:14:39,493 [INFO ]  SAE: 0.2528369128704071<br>2022-06-05 18:14:39,512 [INFO ]  Energy per Day: 99.00535584592438</pre><pre>### Fridge ###</pre><pre>2022-06-06 05:14:39,830 [INFO ]  Appliance target is: fridge<br>2022-06-06 05:14:39,830 [INFO ]  File for test: fridge_test_H15.csv<br>2022-06-06 05:14:39,830 [INFO ]  Loading from: ./dataset_management/refit/fridge/fridge_test_H15.csv<br>2022-06-06 05:14:40,671 [INFO ]  There are 6.226M test samples.<br>2022-06-06 05:14:40,677 [INFO ]  Loading saved model from ./models/fridge/checkpoints.<br>2022-06-06 05:15:11,539 [INFO ]  aggregate_mean: 522<br>2022-06-06 05:15:11,539 [INFO ]  aggregate_std: 814<br>2022-06-06 05:15:11,539 [INFO ]  appliance_mean: 200<br>2022-06-06 05:15:11,539 [INFO ]  appliance_std: 400<br>2022-06-06 05:15:11,649 [INFO ]  true positives=6225098.0<br>2022-06-06 05:15:11,649 [INFO ]  false negatives=0.0<br>2022-06-06 05:15:11,649 [INFO ]  recall=1.0<br>2022-06-06 05:15:11,713 [INFO ]  true positives=6225098.0<br>2022-06-06 05:15:11,713 [INFO ]  false positives=0.0<br>2022-06-06 05:15:11,713 [INFO ]  precision=1.0<br>2022-06-06 05:15:11,713 [INFO ]  F1:1.0<br>2022-06-06 05:15:11,728 [INFO ]  NDE:0.390367716550827<br>2022-06-06 05:15:12,732 [INFO ]  <br>MAE: 18.173030853271484<br>    -std: 22.19791030883789<br>    -min: 0.0<br>    -max: 2045.119873046875<br>    -q1: 5.2667236328125<br>    -median: 12.299388885498047<br>    -q2: 24.688186645507812<br>2022-06-06 05:15:12,754 [INFO ]  SAE: 0.3662513792514801<br>2022-06-06 05:15:12,774 [INFO ]  Energy per Day: 219.63657335193759</pre><pre>### Washing Machine ###</pre><pre>2022-06-05 05:49:17,614 [INFO ]  Appliance target is: washingmachine<br>2022-06-05 05:49:17,614 [INFO ]  File for test: washingmachine_test_H8.csv<br>2022-06-05 05:49:17,614 [INFO ]  Loading from: ./dataset_management/refit/washingmachine/washingmachine_test_H8.csv<br>2022-06-05 05:49:18,762 [INFO ]  There are 6.118M test samples.<br>2022-06-05 05:49:18,767 [INFO ]  Loading saved model from ./models/washingmachine/checkpoints.<br>2022-06-05 05:49:47,965 [INFO ]  aggregate_mean: 522<br>2022-06-05 05:49:47,965 [INFO ]  aggregate_std: 814<br>2022-06-05 05:49:47,965 [INFO ]  appliance_mean: 400<br>2022-06-05 05:49:47,965 [INFO ]  appliance_std: 700<br>2022-06-05 05:49:48,054 [INFO ]  true positives=6117871.0<br>2022-06-05 05:49:48,055 [INFO ]  false negatives=0.0<br>2022-06-05 05:49:48,055 [INFO ]  recall=1.0<br>2022-06-05 05:49:48,115 [INFO ]  true positives=6117871.0<br>2022-06-05 05:49:48,115 [INFO ]  false positives=0.0<br>2022-06-05 05:49:48,115 [INFO ]  precision=1.0<br>2022-06-05 05:49:48,115 [INFO ]  F1:1.0<br>2022-06-05 05:49:48,128 [INFO ]  NDE:0.4052383601665497<br>2022-06-05 05:49:48,835 [INFO ]  <br>MAE: 20.846961975097656<br>    -std: 155.17930603027344<br>    -min: 0.0<br>    -max: 3972.0<br>    -q1: 3.0517578125e-05<br>    -median: 0.25152587890625<br>    -q2: 1.6888427734375<br>2022-06-05 05:49:48,856 [INFO ]  SAE: 0.3226347267627716<br>2022-06-05 05:49:48,876 [INFO ]  Energy per Day: 346.94591079354274</pre><pre>### Kettle ###</pre><pre>2022-05-25 15:19:15,366 [INFO ]  Appliance target is: kettle<br>2022-05-25 15:19:15,366 [INFO ]  File for test: kettle_test_H2.csv<br>2022-05-25 15:19:15,366 [INFO ]  Loading from: ./dataset_management/refit/kettle/kettle_test_H2.csv<br>2022-05-25 15:19:16,109 [INFO ]  There are 5.734M test samples.<br>2022-05-25 15:19:16,115 [INFO ]  Loading saved model from ./models/kettle/checkpoints.<br>2022-05-25 15:19:44,459 [INFO ]  aggregate_mean: 522<br>2022-05-25 15:19:44,459 [INFO ]  aggregate_std: 814<br>2022-05-25 15:19:44,459 [INFO ]  appliance_mean: 700<br>2022-05-25 15:19:44,459 [INFO ]  appliance_std: 1000<br>2022-05-25 15:19:44,540 [INFO ]  true positives=5732928.0<br>2022-05-25 15:19:44,540 [INFO ]  false negatives=0.0<br>2022-05-25 15:19:44,540 [INFO ]  recall=1.0<br>2022-05-25 15:19:44,597 [INFO ]  true positives=5732928.0<br>2022-05-25 15:19:44,597 [INFO ]  false positives=0.0<br>2022-05-25 15:19:44,597 [INFO ]  precision=1.0<br>2022-05-25 15:19:44,597 [INFO ]  F1:1.0<br>2022-05-25 15:19:44,610 [INFO ]  NDE:0.2578023374080658<br>2022-05-25 15:19:45,204 [INFO ]  <br>MAE: 18.681659698486328<br>    -std: 125.82673645019531<br>    -min: 0.0<br>    -max: 3734.59228515625<br>    -q1: 0.0<br>    -median: 0.20867919921875<br>    -q2: 1.63885498046875<br>2022-05-25 15:19:45,224 [INFO ]  SAE: 0.2403179109096527<br>2022-05-25 15:19:45,243 [INFO ]  Energy per Day: 155.25137268110353</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c9b0b4c8291" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/energy-management-using-real-time-non-intrusive-load-monitoring-3c9b0b4c8291">Energy Management Using Real-Time Non-Intrusive Load Monitoring</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Implement Deep Neural Networks for Radar Image Classification]]></title>
            <link>https://medium.com/data-science/how-to-implement-deep-neural-networks-for-radar-image-classification-acb1bfcd7f3?source=rss-e03a0010d87d------2</link>
            <guid isPermaLink="false">https://medium.com/p/acb1bfcd7f3</guid>
            <category><![CDATA[semi-supervised-learning]]></category>
            <category><![CDATA[radar]]></category>
            <category><![CDATA[gans]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Lindo St. Angel]]></dc:creator>
            <pubDate>Sun, 06 Jun 2021 03:39:09 GMT</pubDate>
            <atom:updated>2021-06-06T04:40:54.275Z</atom:updated>
            <content:encoded><![CDATA[<h4>Leverage network architectures and techniques developed for computer vision</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hoZXpIEI4-3J1YIP" /><figcaption>Photo by <a href="https://unsplash.com/@prescott3?utm_source=medium&amp;utm_medium=referral">Prescott Horn</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Radar-based recognition and localization of people and things in the home environment has certain advantages over computer vision, including increased user privacy, low power consumption, zero-light operation and more sensor flexible placement.</p><p>Shallow machine learning techniques such as Support Vector Machines and Logistic Regression can be used to classify images from radar, and in my previous work, <a href="https://towardsdatascience.com/teaching-radar-to-understand-the-home-ee78e7e4a0be">Teaching Radar to Understand the Home</a> and <a href="https://towardsdatascience.com/using-stochastic-gradient-descent-to-train-linear-classifiers-c80f6aeaff76">Using Stochastic Gradient Descent to Train Linear Classifiers</a> I shared how to apply some of these methods.</p><p>In this article, you will learn how to develop Deep Neural Networks (DNN)and train them to classify objects in radar images. In addition, you will learn how to use a Semi-Supervised Generative Adversarial Network (SGAN) [1] that only needs a small number of labeled data to train a DNN classifier. This is important in dealing with radar data sets because of the dearth of large training sets, in contrast to those available for camera-based images (e.g., I<a href="https://www.image-net.org/">mageNet</a>) which has helped to make computer vision ubiquitous.</p><p>Both DNNs (or more specifically Convolutional Neural Networks) and SGANs that were originally developed for visual image classification can be leveraged from an architecture and training method perspective for use in radar applications.</p><h3>Data Set</h3><p>In order to help you understand the techniques and code used in this article, a short walk through of the data set is provided in this section. The data set was gathered from radar samples as part of the <a href="https://github.com/goruck/radar-ml">radar-ml</a> project and found <a href="https://drive.google.com/drive/folders/12QKwuzniZkiqsEVBDx-ho9K6AbOiJz7w?usp=sharing">here</a>. This project employs autonomous supervised learning whereby standard camera-based object detection techniques are used to automatically label radar scans of people and objects.</p><p>The data set is a Python dict of the form:</p><p>{‘samples’: samples, ‘labels’: labels}</p><p>samples is a list of N radar projection numpy.array tuple samples in the form:</p><p>[(xz_0, yz_0, xy_0), (xz_1, yz_1, xy_1),…,(xz_N, yz_N, xy_N)]</p><p>Where a radar projection is the maximum return signal strength of a scanned target object in 3-D space projected to the x, y and z axis. These 2-D representations are typically sparse since a projection occupies a small part of scanned volume. Sampling, storing and making use of the 2-D projections can be more efficient than using the 3-D source data directly.</p><p>labels is a list of N numpy.array class labels corresponding to each radar projection sample of the form:</p><p>[class_label_0, class_label_1,…,class_label_N]</p><p>Projections from a typical single sample are shown in the heat map visualization below. Red indicates where the return signal is strongest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O068c-v85z_lxoCg_zDT9g.png" /><figcaption>Visualization of a data set sample of a dog (my pet Polly)</figcaption></figure><p>This data was captured in my house in various locations designed to maximize the variation in detected objects (currently only people, dogs and cats), distance and angle from the radar sensor.</p><p>The data set contains only a few thousand samples (with known labeling errors) and can only be used to train a deep neural network for a small number of epochs before over fitting. The labeling error will affect the accuracy of the radar classifier trained from this data set. While a future effort will attempt to fine-tune the object detector to reduce the error, using the SGAN may obviate or minimize the need to label future radar observations. Reducing the number of labeled data points to train a classifier, while maintaining acceptable accuracy, was the primary motivation to explore using SGANs in this project.</p><h3>Model Architectures</h3><p>As noted above, the data set contains a collection of 2-D representations of a 3-D radar image and fortunately, there is prior work from the computer vision world in demonstrating its possible to design and train neural networks on such 2-D representations that match or even outperform networks trained on the native 3-D data set, see [2] and [3]. This prior work inspired the development of the networks below.</p><p>All models and associated training were implemented using the Keras API, the high-level API of TensorFlow as part of the <a href="https://github.com/goruck/radar-ml">radar-ml</a> project.</p><h4>Deep Neural Network (DNN) Classifier</h4><p>Although not recognizable by a human, the collection of 2-D radar image projections contain features that map back to the scanned object. Each of the three 2-D projections are passed through separate 2-D convolution layers that learn these features and successively down-sample the image. The output from these layers are concatenated and then flattened to form a single feature vector which is used as an input to deeply connected dense layers followed by a classification layer. This architecture in the figure below. You may notice that a single branch of this architecture is similar to a Convolutional Neural Network (CNN) used in computer vision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PL-toNa5QWUZvhFElKPa4Q.png" /><figcaption>Architecture of DNN Classifier</figcaption></figure><p>The model is implemented by the Python module in the file <a href="https://github.com/goruck/radar-ml/blob/master/dnn.py">dnn.py</a> in the <a href="https://github.com/goruck/radar-ml">radar-ml</a> repository. You can see the code snippet that defines and compiles the model below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a38ce4af2309bbb1b2043967c9ad7e4/href">https://medium.com/media/6a38ce4af2309bbb1b2043967c9ad7e4/href</a></iframe><h4>Semi-Supervised Generative Adversarial Network (SGAN)</h4><p>Gathering radar images for model training is relatively straightforward compared to establishing ground truth which requires a human in the loop, autonomous supervised learning, or a technique such as Semi-Supervised learning that combines a small amount of labeled data with a large amount of unlabeled data during training. The motivation to use Semi-Supervised learning was to minimize the effort associated with humans labeling radar scans or the use of complex (and, possibly error prone) autonomous supervised learning.</p><p>The Generative Adversarial Network (GAN) is an architecture that uses unlabeled data sets to train an image generator model in conjunction with an image discriminator model. In some cases you can use the discriminator model to develop a classifier model. GANs have been used in radar signal generation [4] and have found extensive use in computer vision applications [5].</p><p>The Semi-Supervised GAN (SGAN) model is an extension of a GAN architecture that employs co-training of a supervised discriminator, unsupervised discriminator, and a generator model. In this project, the supervised discriminator is used as a classification model that generalizes to novel data sets and a generator model that yields realistic examples of radar projections (used only as a validity check).</p><p>The supervised discriminator architecture is shown in the figure below and you may notice its similar to the DNN architecture shown nearby, with some exceptions including the use of LeakyReLU (Leaky Rectified Linear Unit) instead of ReLU which is a GAN training best practice [7]. The model includes Batch Normalization layers to aid training convergence which is often a problem in training GANs [6]. The unsupervised discriminator shares most layers except for the final output layers and so has a very similar architecture. The supervised discriminator’s output is a dense layer with softmax activation that forms a 3-class classifier while the unsupervised model takes the output of the supervised model prior to the softmax activation, then calculates a normalized sum of the exponential outputs [6].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AbJbSGexYAuzPvNxCTLXyQ.png" /><figcaption>Architecture of Supervised Discriminator</figcaption></figure><p>Both the supervised and unsupervised discriminator models are implemented by the Python module in the file <a href="https://github.com/goruck/radar-ml/blob/master/sgan.py">sgan.py</a> in the <a href="https://github.com/goruck/radar-ml">radar-ml</a> repository. A code snippet that defines and compiles the model below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99fa15506b6e159158ba6d5d51a4036f/href">https://medium.com/media/99fa15506b6e159158ba6d5d51a4036f/href</a></iframe><p>The generator model takes a vector from the latent space (a noise vector drawn from a standard Normal distribution) and uses three branches of transposed convolution layers with ReLU activation to successively up-sample the latent space vector to form each of the three radar image projections. The generator is stacked on top on the discriminator model and is trained with the latter’s weights frozen. This combined architecture is depicted in the figure below. Note the use of Batch Normalization layers to aid model training convergence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t3gKABYVc8mNbIxDoclnUQ.png" /><figcaption>Architecture of GAN</figcaption></figure><p>The generator and GAN are implemented by the Python module in the file <a href="https://github.com/goruck/radar-ml/blob/master/sgan.py">sgan.py </a>in the <a href="https://github.com/goruck/radar-ml">radar-ml</a> repository. Below is a code snippet that defines and compiles the model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8584e8b95e1c4ebb2235d103d1190f5e/href">https://medium.com/media/8584e8b95e1c4ebb2235d103d1190f5e/href</a></iframe><h3>Model Training and Results</h3><h4>DNN</h4><p>The DNN is trained via the tf.keras.Model class fit method and is implemented by the Python module in the file <a href="https://github.com/goruck/radar-ml/blob/master/dnn.py">dnn.py</a> in the <a href="https://github.com/goruck/radar-ml">radar-ml</a> repository. Below is a code snippet of the training function not shown are the steps required to pre-process and filter the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/96d804d9ca563cf58b6db45c56eff2ab/href">https://medium.com/media/96d804d9ca563cf58b6db45c56eff2ab/href</a></iframe><p>The results from a typical training run are below. The current state of the model and data set is capable of obtaining validation set accuracy in the mid to high 80%’s. More work is required to match or exceed the ~ 90% accuracy obtained by SVM and Logistic Regression models in previous work [8][9]. This will be the focus of future effort.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb2cc9ba04c03b412d3935ab3bec6392/href">https://medium.com/media/bb2cc9ba04c03b412d3935ab3bec6392/href</a></iframe><h4>SGAN</h4><blockquote>Generative Adversarial Networks, or GANs, are challenging to train. This is because the architecture involves both a generator and a discriminator model that compete in a zero-sum game. It means that improvements to one model come at the cost of a degrading of performance in the other model. The result is a very unstable training process that can often lead to failure, e.g. a generator that generates the same image all the time or generates nonsense. As such, there are a number of heuristics or best practices (called GAN hacks) that can be used when configuring and training your GAN models. These heuristics have been hard won by practitioners testing and evaluating hundreds or thousands of combinations of configuration operations on a range of problems over many years.</blockquote><blockquote>“Generative Adversarial Networks with Python”, Jason Brownlee, 2021.</blockquote><p>You can find many good papers and articles that can help to understand how to apply best practices for training GANs. In particular, <a href="https://www.linkedin.com/in/jasonbrownlee/?originalSubdomain=au">Jason Brownlee</a> has published many pragmatic articles and papers that can prove time-saving [7]. Some of this work was used to determine a training method that worked reasonably well on the radar SGAN models and data set. Consider reading his online articles and buying his e-books if you are serious about understanding <strong><em>and</em></strong> applying machine learning.</p><p>The training loop is implemented by the Python module in the file <a href="https://github.com/goruck/radar-ml/blob/master/sgan.py">sgan.py </a>in the <a href="https://github.com/goruck/radar-ml">radar-ml</a> repository. Below is a snippet of the training loop, not shown are the steps required to pre-process and filter the data set as well as several helper functions. This code is based on reference [7]. Note that the discriminator model gets updated with 1.5 batches worth of samples but the generator model is updated with one batch worth of samples each iteration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8abe25486670f82c4924509dd3401777/href">https://medium.com/media/8abe25486670f82c4924509dd3401777/href</a></iframe><p>You will find the training results to vary from run to run due to the stochastic nature of GANs, so its best to average results over several runs. A good training session will have moderate (~ 0.5) and relatively stable losses for the unsupervised discriminator and generator while the supervised discriminator will converge to a very low loss (&lt; 0.1) with high accuracy (&gt; 95%) on the training set. Accuracy results on the validation set tends to be in the low to high 70%’s with losses hovering around 1.2 with using only 50 supervised samples per class. This is an encouraging result but clearly more modeling work and data collection is required to get the validation accuracy on par with the other machine learning methods that were employed on this data set, which were typically ~ 90% [8][9]. This will be the focus of future work on this project. Typical training results are shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d6f1fc1a4096257867f4cbd1a3279c4/href">https://medium.com/media/9d6f1fc1a4096257867f4cbd1a3279c4/href</a></iframe><p>You should also view the images produced by the generator to determine if they make sense. In this case, since the images are 2-D projections of radar scans of 3-D objects and are not recognizable by a human, the generated images need to be compared to examples from the original data set like the one above. The figure below is a set of generated 2-D scans. A similarity in one of the projections (the X-Y plane) is evident but not obvious in the others, at least for this training run. This could account for the low accuracy and finding ways to make the other generated projections visually similar to the training set is left to a future exercise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GaOriq4UXfW8jVmhzsMFUg.png" /><figcaption>Visualization of a generated data set sample</figcaption></figure><h3>Conclusions</h3><p>You can leverage model architectures from CNN’s, SGAN’s and associated training techniques developed for camera-based computer vision to develop neural networks to classify radar images. Given the dearth of radar data sets, you are typically required to collect radar data sets which can be resource intensive and error-prone to ground truth novel radar observations. You can use self-supervised techniques to make use of unlabeled data using only a few tens or less of labeled samples per class and an SGAN. In this manner, you can feasibly develop radar image classifiers using large amounts of unlabeled data.</p><p>Previous work used shallow machine learning models and achieved higher accuracy on the data set than currently obtained using the networks and techniques described here. Future efforts are planned to close this gap and to increase the size of the data set to obtain better validation set accuracy before over fitting.</p><h3>References</h3><ol><li><a href="https://arxiv.org/pdf/1606.01583.pdf%5D">Semi-Supervised Learning with Generative Adversarial Networks</a>, 2016.</li><li><a href="https://arxiv.org/abs/1505.00880">Multi-view Convolutional Neural Networks for 3D Shape Recognition</a>, 2015.</li><li><a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0245230">Multi-view classification with convolutional neural networks</a>, 2021.</li><li><a href="https://arxiv.org/abs/2008.03346">Generative Adversarial Network for Radar Signal<br>Generation</a>, 2019.</li><li><a href="https://arxiv.org/abs/1511.06434">Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks</a>, 2016.</li><li><a href="https://arxiv.org/abs/1606.03498">Improved Techniques for Training GANs</a>, 2016.</li><li><a href="https://machinelearningmastery.com/generative_adversarial_networks/">Generative Adversarial Networks with Python</a>, 2021.</li><li><a href="https://towardsdatascience.com/teaching-radar-to-understand-the-home-ee78e7e4a0be">Teaching Radar to Understand the Home</a>, 2020.</li><li><a href="https://towardsdatascience.com/using-stochastic-gradient-descent-to-train-linear-classifiers-c80f6aeaff76">Using Stochastic Gradient Descent to Train Linear Classifiers</a>, 2020.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acb1bfcd7f3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-implement-deep-neural-networks-for-radar-image-classification-acb1bfcd7f3">How to Implement Deep Neural Networks for Radar Image Classification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Stochastic Gradient Descent to Train Linear Classifiers]]></title>
            <link>https://medium.com/data-science/using-stochastic-gradient-descent-to-train-linear-classifiers-c80f6aeaff76?source=rss-e03a0010d87d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c80f6aeaff76</guid>
            <category><![CDATA[support-vector-machine]]></category>
            <category><![CDATA[stochastic-gradient]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[classification]]></category>
            <dc:creator><![CDATA[Lindo St. Angel]]></dc:creator>
            <pubDate>Mon, 14 Dec 2020 01:10:56 GMT</pubDate>
            <atom:updated>2020-12-14T01:10:56.017Z</atom:updated>
            <content:encoded><![CDATA[<h4>You can tame challenges with data sets that have large numbers of training examples or features</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-bdn8fQKCXQoJnkk" /><figcaption>Photo by <a href="https://unsplash.com/@snaps_by_clark?utm_source=medium&amp;utm_medium=referral">Clark Van Der Beken</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>You are probably aware that <a href="https://towardsdatascience.com/stochastic-gradient-descent-clearly-explained-53d239905d31">Stochastic Gradient Descent</a> (SGD) is one of the key algorithms used in training deep neural networks. However, you may not be as familiar with its application as an optimizer for training <a href="https://en.wikipedia.org/wiki/Linear_classifier">linear classifiers</a> such as <a href="https://en.wikipedia.org/wiki/Support_vector_machine">Support Vector Machines</a> and <a href="https://en.wikipedia.org/wiki/Logistic_regression">Logistic Regression</a> or when and how to apply it.</p><p>In this article, you will learn that the approach can be effectively used on large-scale data sets (&gt; 10⁵ samples) or with a large number (&gt; 10⁵) of features where other methods may lead to extremely long fit times or be infeasible to use beyond a few thousand samples or features. Additionally, SGD allows for <a href="https://en.wikipedia.org/wiki/Online_machine_learning">online learning</a>, making the algorithm quickly fit new data on an existing classifier.</p><p>You will learn about how to use Python APIs from <a href="https://scikit-learn.org/stable/index.html">scikit-learn</a>, an example of a data set that is well-suited for this method captured from radar samples, test results from a classifier fitted with that data using SGD and some drawbacks of using SGD including the need for rather extensive hyperparameter tuning.</p><p>You can use other optimizers to train linear classifiers and, depending on the size of your data set and feature space, the SGD method and linear classifiers in general may not be the best solution. For an overall guide to using SVMs, see <a href="https://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf">A Practical Guide to Support Vector Classification</a> (Hsu, et al., 2016) Some techniques described in this excellent paper were used here.</p><h3>Data Set</h3><p>In order to help you understand the techniques and code used in this article, a short walk through of the data set is provided in this section. The data set was gathered from radar samples as part of the <a href="https://github.com/goruck/radar-ml">radar-ml</a> project and found <a href="https://drive.google.com/drive/folders/12QKwuzniZkiqsEVBDx-ho9K6AbOiJz7w?usp=sharing">here</a>. This project employs autonomous supervised learning whereby standard camera-based object detection techniques are used to automatically label radar scans of people and objects.</p><p>The data set is a Python dict of the form:</p><p>{‘samples’: samples, ‘labels’: labels}</p><p>samples is a list of N radar projection numpy.array tuple samples in the form:</p><p>[(xz_0, yz_0, xy_0), (xz_1, yz_1, xy_1),…,(xz_N, yz_N, xy_N)]</p><p>Where a radar projection is the maximum return signal strength of a scanned target object in 3-D space projected to the x, y and z axis. These 2-D representations are typically sparse since a projection occupies a small part of scanned volume.</p><p>labels is a list of N numpy.array class labels corresponding to each radar projection sample of the form:</p><p>[class_label_0, class_label_1,…,class_label_N]</p><p>Projections from a typical single sample are shown in the heat map visualization below. Red indicates where the return signal is strongest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O068c-v85z_lxoCg_zDT9g.png" /><figcaption>Visualization of a data set sample of a dog (my pet Polly)</figcaption></figure><p>This data was captured in my house in various locations designed to maximize the variation in detected objects (currently only people, dogs and cats), distance and angle from the radar sensor.</p><p>The data set contains known labeling errors, mostly stemming from the object detector mistaking my cat for my dog which happens (subjectively), at about a 10% error rate. A future effort will attempt to fine-tune the object detector to reduce the error. This error will get propagated to the radar classifier trained from this data set.</p><h3>Training</h3><p>You can use the steps below to train the model on the radar data. The complete Python code that implements these steps can be found in the <a href="https://github.com/goruck/radar-ml/blob/master/train.py">train.py</a> module of the <a href="https://github.com/goruck/radar-ml">radar-ml</a> project.</p><ol><li>Scale data set sample features to the [0, 1] range.</li><li>Encode data set labels as integers.</li><li>Split samples and labels up into train, validation and test sets.</li><li>Generate feature vectors from the radar projections in each set above by concatenating all or selected projections. The result is a large but sparse feature space which is a function of the radar scan volume. In this example, the feature vector has length 10,010.</li><li>Augment the training set. This increases accuracy at the expense of training time. Fortunately, using SGD as a optimizer for linear classifiers scales extremely well on large data sets.</li><li>Balance the training set.</li><li>Use the training set and Stratified K-Folds cross-validation to fit a linear classifier using SGD as an optimization technique and a grid search to find the best hyperparameters.</li><li>Calibrate the best classifier using the validation set to obtain an accurate probability estimate of the predictions. This step may not be needed for probabilistic classifiers.</li></ol><p>The Python snippet below from radar-ml’s <a href="https://github.com/goruck/radar-ml/blob/master/train.py#L324">train.py</a> shows the actual fitting function. This uses the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn.linear_model.SGDClassifier">sklearn linear_model.SGDClassifier</a> API with ‘log’ loss which gives Logistic Regression. You can see the online training aspect which is used to do partial fits on the optimum classifier using augmented data as well as novel data sets — a very computationally efficient process. You can also see that the grid search tries fits with a number of hyperparameters and getting these values right is key to an accurate classifier.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f4ab29d4ce20c557e382b0c5cf20600d/href">https://medium.com/media/f4ab29d4ce20c557e382b0c5cf20600d/href</a></iframe><p>Note: the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html">sklearn.svm.LinearSVC</a> API can optimize the same cost function as the SGDClassifier by adjusting the penalty and loss parameters. However, LinearSVC does not allow for online learning. LinearSVC uses the <a href="https://www.csie.ntu.edu.tw/~cjlin/liblinear/">LIBLINEAR</a> library (Fan et al.,2008).</p><p>The train.py module also will fit a model using <a href="https://www.csie.ntu.edu.tw/~cjlin/libsvm/">LIBSVM</a> (Chang and Lin, 2011) via the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html">sklearn svm.SVC</a> API, you can see that used in the Python snippet below from radar-ml’s <a href="https://github.com/goruck/radar-ml/blob/master/train.py#L442">train.py</a>. LIBSVM implements the <a href="https://en.wikipedia.org/wiki/Sequential_minimal_optimization">Sequential minimal optimization</a> algorithm for <a href="https://en.wikipedia.org/wiki/Kernel_trick">kernelized</a> Support Vector Machines which is a very powerful method but does not scale well for large data sets or feature vectors from a fit time perspective.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74aa424b4b228a21082d5d9c8e930a19/href">https://medium.com/media/74aa424b4b228a21082d5d9c8e930a19/href</a></iframe><h3>Evaluation</h3><p>Using the test set that was split from the data set in the step above, evaluate the performance of the final classifier. The test set was not used for either model training or calibration validation so these samples are completely new to the classifier. The evaluation function is shown in the Python snippet below which is part of radar-ml’s <a href="https://github.com/goruck/radar-ml/blob/master/train.py#L215">train.py</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b512b0300d54862b5b9de86ecdf439b9/href">https://medium.com/media/b512b0300d54862b5b9de86ecdf439b9/href</a></iframe><p>The evaluation results from using SGDClassifier are shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14fbb2456f15c8c54157e2f4b70eca1f/href">https://medium.com/media/14fbb2456f15c8c54157e2f4b70eca1f/href</a></iframe><p>The evaluation results from using SVC are shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/03ddce510f5ad8eb38ff3a909438a2d3/href">https://medium.com/media/03ddce510f5ad8eb38ff3a909438a2d3/href</a></iframe><p>You can see that the SGD method gives better overall accuracy (89% vs. 84%) on the test set and moreover completes the training in about seven minutes (including four epochs of augmentation) vs. about 75 minutes as compared to SVC. These results are not apples-to-apples since the SGD classifier accuracy benefits from the data augmentation. Using augmentation with SVC is basically infeasible on my i5 3.4 GHz machine since the training times are a non-linear function of the training set and would take many days. Note that some of the inaccuracies are likely due to the labeling errors highlighted above.</p><p>The resulting SGD-trained linear classifier takes about 250 KB of disk space whereas SVC results in a classifier (RBF kernel) more than two orders of magnitude larger, around 40 MB. This could be an advantage if you use the SDG classifier in a resource limited embedded system.</p><p>You can find the fitted classifiers and training results <a href="https://drive.google.com/drive/folders/1k5Xa02aVMp-JnmlOHgUXpqPexbTUeZve?usp=sharing">here</a>.</p><h3>Prediction</h3><p>Using the classifier to make predictions on new data is straightforward as you can see from the Python snippet below. This is taken from radar-ml’s <a href="https://github.com/goruck/radar-ml/blob/master/predict.py#L56">predict.py</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4e3daac1d04baad04cb58419dbcd87aa/href">https://medium.com/media/4e3daac1d04baad04cb58419dbcd87aa/href</a></iframe><h3>Conclusion</h3><p>You should consider using Stochastic Gradient Descent as an optimizer to efficiently train linear classifiers if you have a large number (many thousands) of training examples or features. Also consider using it for online learning, for example in situations where it is necessary for the algorithm to dynamically adapt to new patterns in the data. Different optimization methods or classifiers may be better in other cases.</p><p>SGD classifiers are sensitive to feature scaling and require fine tuning of a number of hyperparameters including the regularization parameter and the number of iteration for good performance. You should always use feature normalization and a technique like grid search to find the most optimal hyperparameters when using this method. If you intend to use the classifier to predict both a class and a confidence level, you should calibrate it first on a data set disjoint from the training set. Always evaluate the final classifier on a test set disjoint from both the training and validation set.</p><p>You will find prediction using the SGD classifier straightforward via the sklearn APIs and its compact size favors resource limited embedded systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c80f6aeaff76" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/using-stochastic-gradient-descent-to-train-linear-classifiers-c80f6aeaff76">Using Stochastic Gradient Descent to Train Linear Classifiers</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Teaching Radar to Understand the Home]]></title>
            <link>https://medium.com/data-science/teaching-radar-to-understand-the-home-ee78e7e4a0be?source=rss-e03a0010d87d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee78e7e4a0be</guid>
            <category><![CDATA[radar]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[self-supervised-learning]]></category>
            <dc:creator><![CDATA[Lindo St. Angel]]></dc:creator>
            <pubDate>Mon, 01 Jun 2020 03:02:00 GMT</pubDate>
            <atom:updated>2020-12-13T14:19:18.930Z</atom:updated>
            <content:encoded><![CDATA[<h4>Use self-supervised learning to generate data sets for radar-based human, pet and object detection</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZyQt2OZoJ6zToP-9" /><figcaption>Photo by <a href="https://unsplash.com/@sloandaddyslim?utm_source=medium&amp;utm_medium=referral">James Sloan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Computer vision, leveraging commodity cameras, computing and machine learning algorithms, has revolutionized video tracking, object detection and face recognition. This has created massive value for businesses and their customers as well as some concerns about privacy and control of personal information.</p><p>Radar-based recognition and localization of people and things in the home environment has certain advantages over computer vision, including increased user privacy, low power consumption, zero-light operation and more sensor flexible placement. It does suffer some drawbacks as compared to vision systems. For example, radar generally does not have the resolution of most camera-based systems and therefore may be challenged to distinguish objects that have very similar characteristics. Also, although the cost of radar is rapidly declining as automotive and industrial applications grow, it is still generally more expensive than cameras which have long ridden the coattails of mobile phone unit volumes. These factors have slowed its adoption for consumer applications. But more importantly, there are very few radar data sets that can be used to train or fine-tune machine learning models. This is in stark contrast to the ubiquity of computer vision data sets and models.</p><p>In this article, you will learn how to accurately detect people, pets and objects using low-power millimeter-wave radar and will see how self-supervised learning, leveraging conventional camera-based object detection, can be used to generate radar-based detection models. Self-supervised learning is autonomous supervised learning. It is a representation learning approach that obviates the need for humans to label data [1].</p><p>The steps to gather data using self-supervised learning, train and use a radar-based object detection model are as follows.</p><ol><li>Arrange a camera and a radar sensor to share a common view of the environment.</li><li>Run the radar and a camera-based object detection system to gather information about targets in the environment.</li><li>Create ground truth observations from the radar when it senses targets at the same point in space as the object detector.</li><li>Train a machine learning model such as a Support-Vector Machine (SVM) on these observations and check that it has acceptable accuracy for your application.</li><li>Use the trained machine learning model to predict a novel radar target’s identity.</li></ol><p><a href="https://vayyar.com/">Vayyar’s</a> radar chip in the <a href="https://api.walabot.com/">Walabot</a> reference design is a great choice to develop these solutions given its flexibility and wide availability. An object detection model running on Google’s Coral <a href="https://coral.ai/">Edge Tensor Processing Unit (TPU)</a> is another great choice since it can perform inferences many times faster than the radar can scan for targets.</p><p>You can find additional details regarding the software and hardware used in this project at <a href="https://github.com/goruck/radar-ml">radar-ml</a>.</p><h3>Training Setup</h3><p>A photo of the training setup created for this project is shown below. The Walabot radar is mounted up front and horizontally with a camera located at its top center. The white box contains the Google Coral Edge TPU with the camera connected to it over USB and the black box contains a Raspberry Pi 4 with the radar connected to it over another USB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rACosWJdKrMCBAppN7Aa4A.jpeg" /><figcaption>Training Setup Showing Radar, Camera, TPU and Raspberry Pi</figcaption></figure><p>The TPU runs a real-time object detection server over <a href="https://grpc.io/">grpc</a> that a client on the Raspberry Pi communicates with. The object detection server code can be found <a href="https://github.com/goruck/detection_server">here</a>. The Raspberry Pi handles the processing required to determine if a detected radar target is the same as an object detected by the TPU to establish ground truth. The Pi and the TPU communicate over a dedicated Ethernet link to minimize latency which is critical to accurately determining if a radar target is the same as a detected object. The Pi also runs predictions on novel radar targets using a trained model and is used to fit the model from radar target data with ground truth.</p><h3>Radar and Camera Coordinate Systems</h3><p>The figure below shows the radar (XR, YR, YZ), camera (XC, YC, ZC), image (x, y) and pixel (u, v) coordinate systems. The radar is used as the frame of reference. The camera should be placed on the top middle of the radar unit which you can mount either horizontally or vertically (the USB connector is used as a reference, see the Walabot documentation). This ensures that the camera’s optical z-axis is aligned with the radar’s z-axis and fixed offsets between the camera axis’ and the radar axis’ can be determined (these are known as the camera extrinsic parameters). Additionally, you should make sure that the camera’s angle of view is closely matched to the radar’s <a href="https://api.walabot.com/_features.html#_arena">Arena</a> which is defined in the Python module <a href="https://github.com/goruck/radar-ml/blob/master/common.py">common.py</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ef2K8YPqpaw6pAIJezNQJA.jpeg" /><figcaption>Relationships between Radar and Camera Coordinate Systems</figcaption></figure><p>It&#39;s important for you to understand these relationships to convert a 2-D point from the pixel system (what is actually read from the camera) into the radar’s frame of reference. A point in the camera system is shown in blue as P(X, Y, Z) and the corresponding point in the radar system is shown in green as PR(XR, YR, Z). Note that Z is the same in both systems since the radar’s z-axis is aligned with the camera’s optical z-axis and the image plane is placed as close as possible to radar z = 0. See the next section for how the coordinate conversion is done.</p><p>The camera could be placed anywhere it shares a view of a point with the radar as long as the camera’s extrinsic parameters are known so that its view can be rotated and translated into the radar’s frame of reference. Placing the camera on the top center of the radar unit as it is done here greatly simplifies the conversion of the coordinate systems but having the ability to use arbitrarily placed cameras for self-supervised learning for sensors can be very effective in generating data.</p><p>You must first calibrate the camera to determine its intrinsic parameters (x and y focal and principal points) and to correct distortion. The intrinsic parameters are used to convert from image to radar coordinates. You can use <a href="https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_calib3d/py_calibration/py_calibration.html">OpenCV</a> to do camera calibration quite easily which was used in this project.</p><p>See references [5], [6] and [7] to understand more about camera and world coordinate systems and camera intrinsic and extrinsic parameters.</p><h3>Data Collection and Establishing Ground Truth</h3><p>This section describes how target samples from the radar are collected and how ground truth (i.e, target identification) is established. There may be multiple targets detected by the radar in a single scan so localization is required as part of this process. A key challenge here is accurately translating the 3-D radar image into the 2-D camera image in real-time since people and pets can move rapidly through the scene. Since the radar image is in 3-D, three orthogonal views of the target can be used to generate the data set. Choosing the view(s) to include in the data set and the resolution of each is a trade-off exercise between model complexity, size and accuracy.</p><p>You use the Python module <a href="https://github.com/goruck/radar-ml/blob/master/ground_truth_samples.py">ground_truth_samples.py</a> to ground truth radar samples with camera-based observations gathered from the <a href="https://github.com/goruck/detection_server">object detection server</a>. These observations are returned from the server in the form of the centroid coordinates of the detected object’s bounding box and its label. The server has about a 20 millisecond<em> </em>inference latency. This is small in comparison to the 200 millisecond<em> </em>radar scan rate (which is a function of Arena size and resolution). This 1:10 ratio is designed to minimize tracking error between what the radar senses and what the camera is seeing.</p><p>The centroid coordinates are converted into the radar frame of reference by the function shown below and then a distance metric is computed between each radar target and the converted centroid coordinates.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b00b6562fcf7352e73cbf463541e304b/href">https://medium.com/media/b00b6562fcf7352e73cbf463541e304b/href</a></iframe><p>If the distance is less than a threshold, then a match is declared and the radar target’s return signal and the label is stored as an observation.</p><p>The code snippet below shows a simplified version of the main loop that performs these actions for a single radar scan.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0b4a4a7d0cd155be20cfd3c7e5b6f3f/href">https://medium.com/media/e0b4a4a7d0cd155be20cfd3c7e5b6f3f/href</a></iframe><p>You can configure <a href="https://github.com/goruck/radar-ml/blob/master/ground_truth_samples.py">ground_truth_samples.py</a> to visualize the ground truth process in real-time to ensure its working correctly. An example visualization is shown in the screenshot below. You can see in the left window that the centroid of the detected object (green dot) in each of the three 2-D projections of the radar 3-D target return signal strength is close to the target center (red dot). The right window shows the view from the camera-based object detector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9q6BrLrG-l7ukiYElsHm1w.png" /><figcaption>Ground Truth Visualization</figcaption></figure><h3>Model Training and Results</h3><p>You use the Python module <a href="https://github.com/goruck/radar-ml/blob/master/train.py">train.py</a> to train an SVM or Logistic Regression model on the radar samples with ground truth. The samples are scaled to the [0, 1] range, the classes balanced and then the model is fitted using a stratified 5-Folds cross-validator. The fitted model is calibrated, evaluated for accuracy, serialized and saved to disk.</p><p>Three orthogonal views of a target’s return signal can be obtained from the radar. You can think of these as 2-D projections of the 3-D target signal on the radar’s X-Y plane, the X-Z plane and the Y-Z plane. Any one of these or any combination of them can be used as an observation. Using all three of them will result in the best accuracy but the data set (~10k float32’s per sample) and resulting SVM model (~10MB for 1.5k training samples) can be large, especially if the radar scan resolution is high. Choosing the optimal combination is a training hyper-parameter which is configurable in <a href="https://github.com/goruck/radar-ml/blob/master/train.py">train.py</a>.</p><p>Training results from a run using all projections are shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14fbb2456f15c8c54157e2f4b70eca1f/href">https://medium.com/media/14fbb2456f15c8c54157e2f4b70eca1f/href</a></iframe><p>The training results for using just the X-Y projection are very similar. There is a relatively minor degradation from the all projection case. However, the model training time and model size (about 10MB vs 1.6MB) are far worse. It does not seem to be worth using all views for training, at least for this particular data set.</p><p>The most recently trained model (which uses all projections) and data set can be found <a href="https://drive.google.com/drive/folders/1y8twF6puPXvedXhsFhff45HlMawgHzcS?usp=sharing">here</a>. This model has the labels ‘person’, ‘dog’ and ‘cat’. You can easily map these names to suit your own household. The model has an input vector of 10,011 features made up of 5,457 features from the Y-Z projection plane, 3,872 features from the X-Z plane and 682 features from the X-Y plane. The number of features in each plane are a function of the radar Arena size and resolution with the default training Arena yielding these numbers.</p><h3>Making Predictions</h3><p>The Python module <a href="https://github.com/goruck/radar-ml/blob/master/predict.py">predict.py</a> is used to illustrate how predictions are made on novel radar samples using the trained model. You can use a radar prediction Arena different from what was used for training as the predict module will automatically scale the observations as required. However, you should make sure the radar Threshold, Filter Type and Profile are similar to what was used for training. Additionally, the observations need to be constructed from the same orthogonal planes as was used during training.</p><p>An example prediction is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HXtwqXGWw38l6tA2yn02bA.png" /></figure><p>You can use this module and the trained model to run predictions on your own Walabot radar unit.</p><h3>Conclusion</h3><p>Radar-based perception has some advantages over vision-based methods including increased user privacy and zero-light operation but few suitable data sets to train machine learning models are publicly available. Self-supervised learning can be effectively used to generate these data sets, leveraging conventional computer vision. High radar-based object detection accuracy can be achieved using modest sized data sets and shallow machine learning models such as SVMs which enable inference on low-cost computers. These high accuracy models enable many applications in customer’s homes including occupancy and intrusion detection and can be extended to detect accidents such as falling.</p><p>The steps to gather data using self-supervised learning, train and use a radar-based object detection model are as follows.</p><ol><li>Arrange a camera and a radar sensor to share a common view of the environment.</li><li>Run the radar and a camera-based object detection system to gather information about targets in the environment.</li><li>Create ground truth observations from the radar when it senses targets at the same point in space as the object detector.</li><li>Train a machine learning model such as a SVM on these observations and check that it has acceptable accuracy for your application.</li><li>Use the trained machine learning model to predict a novel radar target’s identity.</li></ol><h3>References</h3><p>This work was inspired by references [2], [3], and [4].</p><p>1. <a href="https://hackernoon.com/self-supervised-learning-gets-us-closer-to-autonomous-learning-be77e6c86b5a">Self-supervised learning gets us closer to autonomous learning</a></p><p>2. <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwi8i9e-h9jpAhXLup4KHRsdCNMQFjABegQIBhAB&amp;url=http%3A%2F%2Fcardinalscholar.bsu.edu%2Fbitstream%2Fhandle%2F123456789%2F201689%2FGuoH_2019-2_BODY.pdf%3Fsequence%3D1&amp;usg=AOvVaw2Ps7ptIYpZGBYPnMQyPArp">Real-Time Human Activity Recognition Based On Radar</a></p><p>3. <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwi_19iHiNjpAhXIi54KHUy1A_4QFjABegQIBxAB&amp;url=https%3A%2F%2Farxiv.org%2Fpdf%2F1904.00739&amp;usg=AOvVaw05g69S1Wbh3JwGiKmQPeH8">Through-Wall Pose Imaging in Real-Time with a Many-to-Many Encoder/Decoder Paradigm</a></p><p>4. <a href="https://dl.acm.org/doi/fullHtml/10.1145/3241383">Unobtrusive Activity Recognition and Position Estimation for Work Surfaces Using RF-Radar Sensing</a></p><p>5. <a href="https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html">Camera Calibration and 3D Reconstruction</a></p><p>6. <a href="http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/OWENS/LECT9/node2.html">Camera calibration</a></p><p>7. <a href="https://www.cse.unr.edu/~bebis/CS791E/Notes/CameraParameters.pdf">Geometric Camera Parameters</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee78e7e4a0be" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/teaching-radar-to-understand-the-home-ee78e7e4a0be">Teaching Radar to Understand the Home</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Method to Achieve Work–life balance]]></title>
            <link>https://medium.com/@lindo.st.angel/a-method-to-achieve-work-life-balance-334582a542a3?source=rss-e03a0010d87d------2</link>
            <guid isPermaLink="false">https://medium.com/p/334582a542a3</guid>
            <dc:creator><![CDATA[Lindo St. Angel]]></dc:creator>
            <pubDate>Fri, 21 Feb 2020 19:39:29 GMT</pubDate>
            <atom:updated>2020-02-21T19:39:29.775Z</atom:updated>
            <content:encoded><![CDATA[<h4>Seek it over the long-term but don’t always expect it. You and your leader both play a role</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TAUUfWe-UVHRfzz8" /><figcaption>Photo by <a href="https://unsplash.com/@bekirdonmeez?utm_source=medium&amp;utm_medium=referral">Bekir Dönmez</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Work–life balance is the term used to describe how much time an individual needs between work and other aspects of life, be it family, friends, community, etc.</p><p>The balance between work and life looks different to different individuals as determined by their personal situation and will change over time. But in any competitive, customer obsessed work environment, the amount of effort that could be done to fully satisfy all requirements will always exceed the capacity of any team or individual.</p><p>Leaders need to continually inspect their operations to ensure the right mechanisms are in place to help the team scale with world class effectiveness and efficiency. They also need to ensure their team is adequately resourced as a part of these efforts. However, in my experience, occasionally people will still need to do what ever it takes to move a project over the end line because doing innovative work at scale is hard and subject to the uncertainties inherent in the marketplace that can change even the best laid plans.</p><p>So given this, what does it mean to have work-life balance in practice?</p><p>Personally, I’ve always thought of assessing my work life balance over some time period, over weeks if not months. I’ve worked in intense bursts that were needed to get some product out the door followed by times that weren’t so hectic which were spent more so with family and friends. But over time, I do set some relative hard limits between the amount of time I spend at work, with my family and other activities. Which over a week is about 40 hours at work, 40 hours with the family, 40 hours of sleep and 40 hours split between keeping physically fit, learning new things and staying current. This is the long term average I seek and adjust in the short term as the situation requires. If I sense over time the long term average between these buckets of time is getting out of line then I try to understand why and correct it. I would not be fulfilled or happy otherwise.</p><p>I don’t think it’s realistic to expect perfect and continuous alignment between your time goals and what you need to do in the moment to get the job done or to attend to a family need. The world is just too dynamic and unpredictable.</p><p>I recommend you have an understanding of the major buckets of time you want to allocate and monitor them on a regular basis. It is also important to engage your direct leader on what success looks like for you so they are aware what your individual needs and priorities are. They can help by balancing responsibilities and deliverables across the team. If over the long-term you find yourself in a situation where you feel you can’t achieve your goals, then you might need to change something else in either your work or personal life.</p><p>This is what works for me. I hope some of it will work for you too.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=334582a542a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Deep Learning Person Classifier]]></title>
            <link>https://medium.com/data-science/building-a-deep-learning-person-classifier-ecc55bd01048?source=rss-e03a0010d87d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ecc55bd01048</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[image-recognition]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Lindo St. Angel]]></dc:creator>
            <pubDate>Wed, 19 Feb 2020 01:12:09 GMT</pubDate>
            <atom:updated>2020-02-19T04:12:51.008Z</atom:updated>
            <content:encoded><![CDATA[<h4>Accurately identify images of people with and without faces</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pKeqVMtcf07bXbVc" /><figcaption>Photo by <a href="https://unsplash.com/@ryoji__iwata?utm_source=medium&amp;utm_medium=referral">Ryoji Iwata</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>In <a href="https://en.wikipedia.org/wiki/Machine_learning">machine learning</a> and <a href="https://en.wikipedia.org/wiki/Statistics">statistics</a>, <strong>classification</strong> is the problem of identifying to which of a set of <a href="https://en.wikipedia.org/wiki/Categorical_data">categories</a> (sub-populations) a new <a href="https://en.wikipedia.org/wiki/Observation">observation</a> belongs, on the basis of a <a href="https://en.wikipedia.org/wiki/Training_set">training set</a> of data containing observations (or instances) whose category membership is known [1].</p><p>“Shallow” learning techniques such as <a href="https://en.wikipedia.org/w/index.php?title=Support-vector_machine&amp;oldid=940974287">Support Vector Machines</a> (SVM) can produce effective classifiers from modest sized data sets and more recently <a href="https://en.wikipedia.org/w/index.php?title=Deep_learning&amp;oldid=940556767">deep learning</a> classifiers have rivaled humans in identification tasks but require much larger data sets and large computing resources to achieve this. In particular, deep learning techniques have produced <a href="https://en.wikipedia.org/w/index.php?title=Convolutional_neural_network&amp;oldid=940484524">Convolutional Neural Networks</a> (CNN) that are state of the art image classifiers.</p><p>This article will show you how to develop a CNN-based person classifier using <a href="https://www.tensorflow.org/">TensorFlow</a> that can outperform standard face recognition techniques in certain situations. The standard methods typically include the use of shallow learners like SVM to classify patterns generated by facial embedding (see for example the superb <a href="https://github.com/ageitgey/face_recognition">face_recognition</a> program.) The standard techniques are <strong><em>extremely</em></strong> good at recognition when most of the face is visible. However, my application needed a way to accurately classify people with only partial views of a face and ideally from behind. This provided the motivation to develop a person classifier using deep learning and my goal was to achieve the same recognition accuracy as a human when looking at an image of a person they know with or without a face.</p><p>The following steps are required to build an accurate CNN-based person classifier that identifies which of a set of known and unknown people a new observation belongs. These steps are also applicable to training other types of CNN-based classifiers.</p><ol><li><strong>Collect images of each person you want to classify as well as of strangers to create your training set and preprocess it</strong>. You should use data augmentation methods that automatically generate related, but new, images from the collected data to increase the size of the data set. Your data set will likely be <em>unbalanced</em> meaning that it doesn’t contain the same number of observations for each class. You will need to compensate for this otherwise your model will overemphasize classes with more observations and its accuracy will suffer. Lastly, your data set needs to be converted into a data format suitable for the training process. You should plan for a training set of at least a 1000 observations for each class. Note that the terms observations and samples are used interchangeably in this article.</li><li><strong>Select CNN that has already been trained on a standard data set and use it as a layer in a new model that will become your classifier. </strong>The standard data set should contain classes similar to what you want to classify. In this case, <a href="http://www.image-net.org/">ImageNet</a> is a good choice since “person” (in the generic sense) is already a class its been trained to recognize. There are many CNN models available from TensorFlow with varying complexity — accuracy trade-offs. You should choose the least complex model that achieves your application’s inference accuracy requirements. Of course, you can fine-tune on any class similar to the ImageNet classes, not just people.</li><li><strong>Apply </strong><a href="https://www.tensorflow.org/tutorials/images/transfer_learning"><strong>transfer learning</strong></a><strong> on your CNN using your image data</strong>. Deep neural networks are prone to <a href="https://www.tensorflow.org/tutorials/keras/overfit_and_underfit">overfitting</a> and you will learn several mitigation techniques to combat it. Note that transfer learning is also known as fine-tuning and training a model is also known as fitting a model to a data set. These terms are used interchangeably in this article.</li><li><strong>Evaluate the Fine-Tuned Model</strong>. You should check the accuracy of your fine-tuned model to see if it meets your application’s requirements and if not, refit it with more data or more optimal training parameters (“hyperparameters”).</li><li><strong>Save your model and prepare it for inference</strong>. <a href="https://www.tensorflow.org/guide/saved_model">SavedModel</a> is the canonical way to save and serve a TensorFlow program but you may also want to generate a <a href="https://www.tensorflow.org/lite">TensorFlow Lite</a> representation of it for deployment on a mobile or edge device. Both are covered below.</li></ol><p>Please note that I used an Intel i5 + Nvidia 1080Ti machine with 64GB of main memory to train my models. You will need at least a similar machine to feasibly train a deep learning model.</p><p>Also, note that much of this work was done for the <a href="https://github.com/goruck/smart-zoneminder"><strong>smart-zoneminder</strong></a> project and you can leverage that work. If you just want to quickly fine-tune a TensorFlow CNN without diving into the details, <a href="https://github.com/goruck/smart-zoneminder#machine-learning-platform-installation-on-linux-server">install the machine learning platform</a> on a suitable machine, prepare your data set as described below and run the Python program <a href="https://github.com/goruck/smart-zoneminder/blob/master/person-class/train.py"><strong>train.py</strong></a> with options that suit your configuration. The rest of this article will breakdown train.py section by section to help you understand how it works to use it better and to perhaps modify it for your own use.</p><h3>Data Set Preparation</h3><p>Create a sub-directory for each person’s images that you want recognized, named for the person, in a directory called “dataset” (I used my Google photos to seed this directory). Also, create a sub-directory called ‘Unknown’ that will hold faces of random strangers. In each sub-directory, you can optionally place another sub-directory that can hold images of the person without a full or partial view of their face. You can decide to include such images as a member of their class or as the Unknown class. The dataset directory will look like the following.</p><pre>dataset<br>    |-name_of_person_1<br>        |-sample_image_1<br>        |-sample_image_2<br>        |-sample_image_n<br>        |-images_with_no_face<br>            |-sample_image_1<br>            |-sample_image_2<br>            |-sample_image_n<br>    |-name_of_person_2<br>    |-name_of_person_n<br>    |-Unknown</pre><p>The function below to creates a Python dataframe from the contents of dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb3f66b78d2f723594f182f1ce632aec/href">https://medium.com/media/cb3f66b78d2f723594f182f1ce632aec/href</a></iframe><p>The dataframe is input to the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator#flow_from_dataframe">flow_from_dataframe</a> method of the tf.keras.preprocessing.image.ImageDataGenerator class. The ImageDataGenerator generates batches of tensor image data with real-time data augmentation and creates training and validation sets. The flow_from_dataframe methods create a Python generator for each set. This code is shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/72fba037eaf8d5d5e83c3bc4c602c34e/href">https://medium.com/media/72fba037eaf8d5d5e83c3bc4c602c34e/href</a></iframe><p>Although it is possible to use train_generator and validation_generator directly to fit the CNN model, using instead a Dataset object from <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset">tf.data.Dataset</a> will give you <a href="https://www.tensorflow.org/guide/data_performance">better fitting performance</a>. In train.py, this is done as follows.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f96a7f0a76b28b11af68db53584945b/href">https://medium.com/media/4f96a7f0a76b28b11af68db53584945b/href</a></iframe><p>Since its likely you will have more sample images in some classes than others, you will need to convey the class weighting to the model fitting process. The code to do that is shown below along with the number of steps the fitting process should use for training and validation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1660f4d9eab0c23d0f2a30c54ee15d81/href">https://medium.com/media/1660f4d9eab0c23d0f2a30c54ee15d81/href</a></iframe><p>The data is now ready to be used to fit a model.</p><h3>Model Preparation</h3><p>The function below creates a CNN model from an <a href="http://www.image-net.org/">ImageNet</a> pre-trained <a href="https://www.tensorflow.org/api_docs/python/tf/keras/applications">tf.keras.applications</a> model. Although the function will create a model based on VGG16 ,InceptionResNetV2 ,MobileNetV2 and ResNet50, only InceptionResNetV2 is shown. The final softmax layer from the base model is removed, new dense classifier and softmax layers are added and then its complied. Pay particular attention to the hyperparameter constants since they will likely need to be adjusted to suit your data set but the default values give me good results. The function includes various over-fitting mitigation techniques including <a href="https://towardsdatascience.com/l1-and-l2-regularization-methods-ce25e7fc831c">L2 Regularization</a>, <a href="https://towardsdatascience.com/what-is-label-smoothing-108debd7ef06">Label Smoothing</a> and D<a href="http://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf">ropout</a>. It also selects an appropriate tf.keras preprocessor that formats the data samples correctly.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c7b2fa9b22fbd2666a88bce5517678a/href">https://medium.com/media/7c7b2fa9b22fbd2666a88bce5517678a/href</a></iframe><p>The model is now ready to be fine-tuned.</p><h3>Model Fine-Tuning</h3><p>Fine-tuning is done in two passes. The first pass uses a high learning rate and trains just the newly added dense and softmax layers since they were initialized with random data. The second pass uses a much smaller learning rate and trains the final layers along with much of the base model. The two pass scheme along with regularization is designed to preserve as much as possible the original weights from the base model while still learning the patterns from the new data set. Both passes use <a href="https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping">early stopping</a> based on validation loss which is another way to mitigate overfitting.</p><p>The first pass fine-tuning code is shown below. Again, note that only the new layers are being fitted.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4267ff699d18d3125c14fe2c23a881b8/href">https://medium.com/media/4267ff699d18d3125c14fe2c23a881b8/href</a></iframe><p>The second pass of the fine-tuning is shown below. Note that some of the bottom base model’s layers are frozen and won’t be trained in this step, but all others will be. The number of layers frozen is a hyperparameter and is a balance between preserving the layers that were trained on high-level features from ImageNet while allowing the top layers to learn the new features in your data set. The more layers you unfreeze the more data you will need to prevent overfitting, despite all the mitigations employed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2eb5f88669c25812be7746a22c36f74b/href">https://medium.com/media/2eb5f88669c25812be7746a22c36f74b/href</a></iframe><p>The model is now fine-tuned and will be evaluated for accuracy and saved for inference in the next steps.</p><h3>Final Model Evaluation</h3><p>Your model should be evaluated to determine if its accuracy meets your application’s requirement. Generating a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html">sklearn classification report</a> from its predictions is a way to do this as shown in the code below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ab81f21a05cff879d4802956b504b16/href">https://medium.com/media/9ab81f21a05cff879d4802956b504b16/href</a></iframe><p>An example classification report for one of my runs is shown below. Although I’m pleased that the model is biased toward its creator ;), more work is required to raise the accuracy on the other classes through additional observations in the data set and hyperparameter optimization.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9bad67544896413b97e3beb73da93b8/href">https://medium.com/media/c9bad67544896413b97e3beb73da93b8/href</a></iframe><p>Currently, my best results are with the InceptionResNetV2 base model which achieves an overall accuracy of about 92% My goal is greater then 95% which is my subjective estimate of what a normal person could achieve on images with and without faces. For comparison, using the standard approach mentioned above yields an accuracy of about 90% for those images with a full or mostly full view of the face and the accuracy on the same data set, with full, partial and no faces is less than 70%.</p><h3>Saving the Model for Inference</h3><p>train.py includes options for saving the model in various ways — a frozen TensorFlow program (deprecated starting in TensorFlow 2), an eight-bit quantized TensorFlow Lite program optimized for inference on IoT devices, a compiled TensorFlow Lite program for Google’s Edge TPU and the TensorFlow SavedModel format (the canonical method starting in TensorFlow 2). The code below shows how train.py saves to the SavedModel format.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b1af691c10125f536a880e4bc785f174/href">https://medium.com/media/b1af691c10125f536a880e4bc785f174/href</a></iframe><p>The final model is now ready for inference via its SavedModel or one of the other representations. For additional details on saving and using a model at the edge, see <a href="https://towardsdatascience.com/using-keras-on-googles-edge-tpu-7f53750d952">Using Keras On Google’s Edge TPU</a>.</p><h3>Conclusion</h3><p>You can leverage TensorFlow to quickly fine-tune a CNN classifier with modest sized data sets as compared to the original data set that was used to train the base model, as shown in the example above for the person classifier. The Python program <a href="https://github.com/goruck/smart-zoneminder/blob/master/person-class/train.py"><strong>train.py</strong></a> from the <a href="https://github.com/goruck/smart-zoneminder"><strong>smart-zoneminder</strong></a> project can be used to build your own classifier as is or as a reference for you to develop your own program.</p><p>Fitting deep learning models to limited data can be tricky since they can quickly memorize the data set and won’t generalize well to new observations, this is known as overfitting. To mitigate this you can use the following techniques.</p><ul><li>Gathering more observations.</li><li>Data augmentation.</li><li>Choosing the least complex model that meets your application’s accuracy requirements.</li><li>L2 Regularization.</li><li>Dropout.</li><li>Label Smoothing.</li><li>Preventing many base model layers from being trained (“freezing”).</li><li>Early stopping of the training process based on validation loss.</li></ul><p>You will also most likely have to deal with an unequal number of samples for each class in your data set (e.g., an unbalanced set) which will bias the model’s accuracy. To compensate you’ll need to use class weighting in the model fitting process.</p><p>The steps to build a CNN classifier are as follows.</p><ol><li>Collect and preprocess your data set.</li><li>Select an appropriate CNN base model.</li><li>Fine-tune the model.</li><li>Evaluate your fine-tuned model’s accuracy.</li><li>Prepare your final model for inference and save it.</li></ol><h3>References</h3><p>[1] <a href="https://en.wikipedia.org/w/index.php?title=Statistical_classification&amp;oldid=921296045">Statistical classification</a>, from <em>Wikipedia, The Free Encyclopedia.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ecc55bd01048" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/building-a-deep-learning-person-classifier-ecc55bd01048">Building a Deep Learning Person Classifier</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>