<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alexander Taffe on Medium]]></title>
        <description><![CDATA[Stories by Alexander Taffe on Medium]]></description>
        <link>https://medium.com/@taffealexander?source=rss-9da466b62f84------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*EuyVue3HTx9KUW-RvkFLmw.jpeg</url>
            <title>Stories by Alexander Taffe on Medium</title>
            <link>https://medium.com/@taffealexander?source=rss-9da466b62f84------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 16:39:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@taffealexander/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Low-Cost, Wireless, IP Cam with Python, WebRTC, and a Raspberry Pi Zero 2W(Part 1 / 2)]]></title>
            <link>https://medium.com/@taffealexander/building-a-low-cost-wireless-ip-cam-with-python-webrtc-and-a-raspberry-pi-zero-2w-part-1-2-34c10820b1db?source=rss-9da466b62f84------2</link>
            <guid isPermaLink="false">https://medium.com/p/34c10820b1db</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[video-streaming-software]]></category>
            <category><![CDATA[webrtc]]></category>
            <category><![CDATA[raspberry-pi-zero]]></category>
            <category><![CDATA[peer-to-peer]]></category>
            <dc:creator><![CDATA[Alexander Taffe]]></dc:creator>
            <pubDate>Mon, 09 Mar 2026 11:33:27 GMT</pubDate>
            <atom:updated>2026-03-10T21:32:09.682Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*lkCREF93rFhqipj7MduaLQ.jpeg" /><figcaption>A Raspberry Pi Zero 2W inside the case I 3D printed.</figcaption></figure><p>As I continue my job search, I have come across a few positions looking for a software engineer with WebRTC experience. I’ll be honest, I didn’t know much about it, and applied to some of these positions, thinking I could learn it like any other framework. TL;DR: I didn’t get one of these jobs or even an interview. But maybe that was not a bad thing. It sparked my interest in WebRTC, a complex yet powerful protocol for peer-to-peer communication. I am working on a few computer vision-based projects, so I decided to build a wireless IP webcam for testing using a Raspberry Pi and WebRTC.</p><p>See the <a href="https://github.com/ataffe/RaspberyPiWebCam/tree/main">GitHub</a> repo for the complete code. This is part 1 in a series of two articles see <a href="https://medium.com/@taffealexander/building-a-low-cost-wireless-wi-fi-cam-with-python-webrtc-and-a-raspberry-pi-zero-2w-part-2-2-708f2a9fbca1">part 2</a> for the other half of this project.</p><h3>Prerequisites</h3><ul><li>Object-Oriented Programming in Python</li><li>Linux / terminal commands</li><li>Asynchronous programming in Python (The asyncio Python docs have a solid <a href="https://docs.python.org/3/howto/a-conceptual-overview-of-asyncio.html#a-conceptual-overview-of-asyncio">guide</a>)</li><li>OpenCV basics</li></ul><h3>Parts List</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*ZWvEzYCEozBjWNZ7HpQS6Q.jpeg" /><figcaption>The inside of the case I designed.</figcaption></figure><ul><li><a href="https://www.raspberrypi.com/products/raspberry-pi-zero-2-w/">Raspberry Pi Zero 2W</a> (with headers) (<a href="https://www.amazon.com/Raspberry-Pi-Zero-2-WH/dp/B0DB2JBD9C/ref=sr_1_3?crid=WRAMS46N4D94&amp;dib=eyJ2IjoiMSJ9.RS8fCsHvPPqOcHq1sQzbcF3P7TcP5eB25wTf-PThMxUuXYpcCyqDDNCizaoeMV2GOk4vxwhgVh3lxJKbcRw_ru4UM8Jx0hAZui2KH1RAsikbZTEFx5AYvEPHd1EwuTCaMKo4W2raVkKRWn0kf2inzZCP9EIPhxApKasuZZNRkSwPclBwSt7kknYgHNwVuW6li1XbFIPhdeS4CiCfDlcNfE-xq0O5wpdRtJmeA5hKgec.38oxapQLTFCe0vL7t9OD8B-YTL_6DmrXcuNyJ6Tq3nw&amp;dib_tag=se&amp;keywords=raspberry+pi+zero+2w&amp;qid=1772009398&amp;sprefix=raspberry+pi+zero+2w%2Caps%2C219&amp;sr=8-3">Amazon</a>) — $29</li><li><a href="https://www.pisugar.com/products/pisugar-3-raspberry-pi-zero-battery">PiSugar 3 1200mAh Raspberry Pi Zero Battery</a> (<a href="https://www.amazon.com/PiSugar-Portable-Pwnagotchi-Management-Raspberry/dp/B0FB3N1YSK/ref=sr_1_1?crid=ZW3NRUZXZCZ3&amp;dib=eyJ2IjoiMSJ9.E2m_Zn5P9r_M1HeN6BajTAkJoCX38pON2-jFrKR3EHtLWioBe_4i3rv7nD4TODUVUR7jfbWdGAJLagEXw8m2dA.aONaSbXZQeZ-LQ4rqmmpT3hoyaNv6d0mDmxYx0M2AoE&amp;dib_tag=se&amp;keywords=pi+sugar+3+battery&amp;qid=1772009491&amp;sprefix=pi+sugar+3+battery%2Caps%2C227&amp;sr=8-1">Amazon</a>)-$39</li><li><a href="https://www.arducam.com/arducam-for-raspberry-pi-camera-8mp-imx219-auto-focus-camera-module-b0393.html">Arducam 8MP IMX219 Auto Focus Camera Module</a> (<a href="https://www.amazon.com/dp/B09X19GZXX?ref=ppx_yo2ov_dt_b_fed_asin_title&amp;th=1">Amazon</a>)-$22</li><li>Silicon Power 128GB Micro SD Card (<a href="https://www.amazon.com/dp/B09TQS634Y?ref=ppx_yo2ov_dt_b_fed_asin_title&amp;th=1">Amazon</a>)-$25</li></ul><p>*Prices are based on Amazon</p><p><strong>Total Price = $115</strong></p><p>One of the goals I had for the projects was to see whether I could do it cheaply and, if so, how cheaply, in a reasonable amount of time. I admit this could be cheaper; for example, I could use a way cheaper SD card. I’ll include the cheapest version I could come up with using similar parts at the bottom, which costs around <strong>$60</strong>. But I’m sure there are ways to make an even cheaper DIY IP cam.</p><p>I also designed a potato of a case that I&#39;m still iterating on, but it’s honest work, and I uploaded the current version to <a href="https://github.com/ataffe/RaspberyPiWebCam/tree/74db703f4c36986fe388d9abebbfccab45e157f4/case">GitHub</a>.</p><h3>Hardware Setup</h3><p><strong>Raspberry Pi </strong>— First, you’ll need to <a href="https://www.raspberrypi.com/documentation/computers/getting-started.html">flash</a> the Raspberry Pi with a copy of Raspbian. SSH into it from a laptop/PC. I started with the GUI enabled to get the IP address, then <a href="http://wobblycogs.co.uk/raspberry-pi/2022/01/08/disable-the-raspberry-pi-desktop/">disabled</a> it for performance and used SSH to log in afterward.</p><p><strong><em>NOTE:</em></strong><em> I had trouble using the 64-bit version of Rasbian Trixie on the Raspberry Pi Zero 2W, so I used the 32-bit version, and it worked fine.</em></p><p><strong>PiSugar3 </strong>— Follow their <a href="https://docs.pisugar.com/docs/product-wiki/battery/pisugar3/pisugar-3-series">instructions</a>.</p><p><strong>Adrucam</strong> — Same thing here, you’ll need to follow their <a href="https://docs.arducam.com/Raspberry-Pi-Camera/Native-camera/Quick-Start-Guide/">instructions</a> for your camera.</p><p><strong>Case</strong> — This is a case I designed and 3D-printed using my very limited CAD skills, but it’s honest work and fits the parts above. (Here are the <a href="https://github.com/ataffe/RaspberyPiWebCam/tree/main/case">stl files</a>)</p><h3>Software Setup</h3><p>First, you’ll need to install aiortc (webrtc in Python), picamera2 (Raspberry Pi Cam), and OpenCV. If you use an arducam you will need to install their drivers as well. I also usually use a virtual environment on both the Raspberry Pi and a laptop/PC.</p><p><strong><em>Linux Packages (</em>Raspberry Pi<em>)</em></strong></p><pre>sudo apt update<br>sudo apt install -y \<br>    python3-dev \<br>    libffi-dev \<br>    libssl-dev \<br>    libsrtp2-dev \<br>    libavdevice-dev \<br>    libavfilter-dev \<br>    libavformat-dev \<br>    libavcodec-dev \<br>    libswresample-dev \<br>    libswscale-dev \<br>    libavutil-dev \<br>    pkg-config</pre><p><strong><em>picamera2 (</em>Raspberry Pi<em>)</em></strong></p><p>The documentation has great instructions on installing picamera2 if you need to. You may not, though, because it should come with Raspberry Pi OS.</p><p><strong><em>Virtual Environment (</em>Raspberry Pi<em>)</em></strong></p><p>If you are going to use a virtual environment, you need to create it with the — system-site-packages flag because the libcamera Python bindings are not available on PyPI.</p><pre>python -m venv --system-site-packages my-env</pre><p><strong><em>Python 3.11+ (</em>Raspbery Pi &amp; Laptop<em>)</em></strong></p><p>I am using at least Python 3.11 because I am using <em>asyncio.timeout</em>, which was added to asyncio in 3.11, but you don’t have to use it and can use an older version if you need to. I just haven’t tested with older versions.</p><p><strong><em>aiortc (</em>Raspberry Pi &amp; Laptop<em>)</em></strong></p><pre>pip install aiortc</pre><p><strong><em>opencv (</em>Laptop<em>)</em></strong></p><p>You can install with pip or with apt. I tend to use pip because it is sometimes more up to date than apt.</p><pre>pip install opencv-contrib-python</pre><h3>WebRTC 101</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*KoHmlblrt7VM2hc4wGioVw.png" /><figcaption>From WebRTC for the Curious</figcaption></figure><p>WebRTC (Web Real-Time Communication) is an API and a protocol for bi-directional, secure, real-time communication. WebRTC is a complex protocol built from existing protocols, so the details are out of scope for this medium article. However, there are some great resources that go into varying levels of detail on the topic. One is an excellent open-source book written by the creators of WebRTC called <a href="https://webrtcforthecurious.com/">WebRTC for the Curious</a>. Using this book as a loose guide, I will provide a high-level overview of the steps required to establish a bi-directional WebRTC connection using Python and the aiortc library.</p><p>Since WebRTC is used for bi-directional communication, I created a sender (<em>streamer.py</em>, running on the Raspberry Pi) and a receiver (<em>receiver.py,</em> running on a desktop). WebRTC for the Curious breaks WebRTC into four steps at a high level, so I am going to use these steps in this guide:</p><ol><li>Signaling</li><li>Connecting</li><li>Securing</li><li>Communicating</li></ol><h3>Step 1: Signaling</h3><p>The first step to creating a WebRTC connection is signaling. The two agents need to find each other, usually over the internet, but for simplicity, I am using my Raspberry Pi as a hotspot. WebRTC does this using a plain-text protocol called <em>SDP</em> (Session Description Protocol). Signaling occurs out-of-band, meaning these messages are <strong>not</strong> exchanged via WebRTC itself. So you could use a REST API, for example, for signaling. The aiortc API provides 3 methods for connecting:</p><ol><li><strong>TCP Sockets — </strong><a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol">Transmission Control Protocol</a></li><li><strong>UDP Sockets — </strong><a href="https://en.wikipedia.org/wiki/User_Datagram_Protocol">User Datagram Protocol</a></li><li><strong>Copy and Paste — </strong>Outputs the SDP message that can be copied and pasted into the other agent’s console (useful for debugging).</li></ol><p>The aiortc lib doesn’t have much documentation beyond some example code. This is probably because the API closely follows the JavaScript API. For this example, I am using a tcp socket, but the example program makes all 3 options available using input args parsed using <em>argparse</em>:</p><p><strong><em>streamer.py </em></strong>(Raspberry Pi)</p><pre>from aiortc.contrib.signaling import BYE, add_signaling_arguments, create_signaling<br>import asyncio<br>import argparse<br>from aiortc import RTCPeerConnection, RTCIceCandidate, RTCSessionDescription<br><br><br>async def run_webrtc_streamer_agent(pc, signaling):<br>    # Only used for copy and paste<br>    await signaling.connect()<br>    # Create session description<br>    await pc.setLocalDescription(await pc.createOffer())<br>    # Send SDP message.<br>    await signaling.send(pc.localDescription)<br>    while True:<br>        obj = await signaling.receive()<br>        if isinstance(obj, RTCSessionDescription):<br>            await pc.setRemoteDescription(obj)<br>            if obj.type == &quot;offer&quot;:<br>                await pc.setLocalDescription(await pc.createAnswer())<br>                await signaling.send(pc.localDescription)<br>        elif obj is BYE:<br>            print(&quot;Exiting&quot;)<br>            break<br><br>if __name__ == &quot;__main__&quot;:<br>    parser = argparse.ArgumentParser(description=&quot;Raspberry Pi camera streamer&quot;)<br>    add_signaling_arguments(parser)<br>    args = parser.parse_args()<br><br>    pc = RTCPeerConnection()<br>    signaling = create_signaling(args)<br>    loop = asyncio.new_event_loop()<br>    asyncio.set_event_loop(loop)<br>    <br>    try:<br>        loop.run_until_complete(run_webrtc_streamer_agent(pc, signaling))<br>    except KeyboardInterrupt:<br>        pass<br>    finally:<br>        loop.run_until_complete(signaling.close())<br>        loop.run_until_complete(pc.close())<br>        loop.close()</pre><p>Let&#39;s break some of this down:</p><pre>add_signaling_arguments(parser)</pre><p>Adds arguments to the argument parser for signaling via TCP, UDP, or copy-and-paste, and for passing an IP address and port for TCP or UDP.</p><p>Here is what running the streamer would look like using TCP sockets:</p><pre>python streamer.py --signaling tcp-socket --signaling-host 0.0.0.0 --signaling-port 8081</pre><p>and the receiver assuming the ip address of the streamer Raspberry Pi was <strong>10.0.0.23</strong>:</p><pre>python receiver.py --signaling tcp-socket --signaling-host 10.0.0.23 --signaling-port 8081</pre><pre>pc = RTCPeerConnection()<br>signaling = create_signaling(args)</pre><p>Creates an RTCPeerConnection, which is a top-level object that represents the WebRTC connection as a whole. It also creates a signaling object from the input arguments to initiate the connection. The rest of the code creates an asynchronous loop that runs the <em>run_webrtc_agent </em>coroutine, which we’ll keep empty for now. It also passes the peer connection and signaling object as arguments to the <em>run_webrtc_agent </em>coroutine and handles cleanup.</p><pre>await signaling.connect()</pre><p>Ok for TCP and UDP, this coroutine doesn’t block or do anything. But for copy-and-paste, this line outputs the message that can be copied and pasted to the receiver.</p><pre>await pc.setLocalDescription(await pc.createOffer())<br>await signaling.send(pc.localDescription)</pre><p>This is where the signaling actually happens. The two agents must exchange information on how to connect. To start, one side creates an offer containing the session description, saves it locally, and sends it to the receiving agent. For simplicity, I always initiate the connection on the streamer (Raspberry Pi) because I am using TCP, but you could initiate from either side or just have the Raspberry Pi wait for connections.</p><pre>while True:<br>        obj = await signaling.receive()<br>        if isinstance(obj, RTCSessionDescription):<br>            await pc.setRemoteDescription(obj)<br>            if obj.type == &quot;offer&quot;:<br>                await pc.setLocalDescription(await pc.createAnswer())<br>                await signaling.send(pc.localDescription)<br>        elif obj is BYE:<br>            print(&quot;Exiting&quot;)<br>            break</pre><p>This loop waits for a reply from the receiver that contains its session description. Once a reply is received, it creates an answer which includes the information from the receiver, saves it locally, and sends a confirmation using <strong><em>signaling.send(pc.localDescription)</em></strong>.</p><p><strong><em>receiver.py </em></strong>(Desktop / Laptop)</p><p>The receiver code is exactly the same for the main function, so I won’t include that for brevity. In fact, all the code is pretty much the same, except the receiver doesn’t create an offer, so the <em>run_webrtc_receiver_agent </em>coroutine looks like this:</p><pre>async def run_webrtc_receiver_agent(pc, signaling):<br>    while True:<br>        obj = await signaling.receive()<br>        if isinstance(obj, RTCSessionDescription):<br>            await pc.setRemoteDescription(obj)<br><br>            if obj.type == &quot;offer&quot;:<br>                await pc.setLocalDescription(await pc.createAnswer())<br>                await signaling.send(pc.localDescription)<br>        elif isinstance(obj, RTCIceCandidate):<br>            await pc.addIceCandidate(obj)<br>        elif obj is BYE:<br>            break</pre><p>It’s just signaling. That completes signaling; now, both agents have enough information to establish a connection. Thanks for reading this far!</p><h3>Step 2 &amp; 3: Connecting &amp; Securing</h3><p>Next, WebRTC uses the ICE (Interactive Connectivity Establishment) protocol to establish a bi-directional connection between the streamer and the receiver, which might be in different networks. ICE does some heavy lifting to establish a peer-to-peer connection between the streamer and the receiver. Once this connection is made, the two agents can communicate using two pairs of protocols:</p><ul><li><strong><em>RTP</em> </strong>(Real-time Transport Protocol)/ <strong><em>SRTP</em></strong> (Secure Real-Time Transport Protocol) — Video &amp; Audio.</li><li><strong><em>SCTP</em></strong> (Stream Control Transmission Protocol) / <strong><em>DTLS</em></strong> (Datagram Transport Layer Security) — Data channel messages.</li></ul><p>This was confusing to me at first, but basically, you have a pair of protocols: one for sending media, such as audio or video, and one for sending data, such as heartbeat messages. Then each of those protocols has its own security protocol. The <em>RTP</em> protocol exchanges media encrypted with <em>SRTP.</em></p><p>You can also exchange data, such as text, via data channels alongside media. WebRTC accomplishes this using <em>SCTP</em> secured using <em>DTLS</em>, which is <em>UDP</em> encrypted with <em>TLS</em>. For example, you can periodically send the battery level to the receiver via a data channel. Additionally, data channels are pretty flexible and can transfer much more than just text. For example, you can transfer multi-gigabyte files using data channels, because SCTP messages don’t have size limits!</p><p>That was a lot of acronyms! Luckily, the aiortc library handles all of this for you. You can print the connection status as it happens using two callbacks.:</p><pre> @pc.on(&quot;connectionstatechange&quot;)<br>    async def on_connectionstatechange():<br>        state = pc.connectionState<br>        print(f&quot;Connection state is: {state}&quot;)<br>        if state == &#39;connected&#39;:<br>            print(&#39;Connected&#39;)<br><br>    @pc.on(&quot;iceconnectionstatechange&quot;)<br>    async def on_iceconnectionstatechange():<br>        state = pc.iceConnectionState<br>        print(f&quot;ICE connection state: {state}&quot;)</pre><p>That wraps up part 1! This doesn’t do much yet, but in <a href="https://medium.com/@taffealexander/building-a-low-cost-wireless-wi-fi-cam-with-python-webrtc-and-a-raspberry-pi-zero-2w-part-2-2-708f2a9fbca1">Part 2</a>, we will add a video track to kick off the ICE connection. We will also use the Raspberry Pi’s hardware H. 264 encoder for fast encoding and display the video using OpenCV. Thanks for reading!</p><p>Here is the full code from part 1 on GitHub:</p><p><a href="https://github.com/ataffe/RaspberyPiWebCam/blob/74db703f4c36986fe388d9abebbfccab45e157f4/part_1/streamer.py">streamer.py</a></p><p><a href="https://github.com/ataffe/RaspberyPiWebCam/blob/74db703f4c36986fe388d9abebbfccab45e157f4/part_1/receiver.py">receiver.py</a></p><p>Here is the cheaper version of the wireless Pi Cam using similar parts:</p><ul><li>Raspberry Pi Zero 2W (no headers)(<a href="https://www.pishop.us/product/raspberry-pi-zero-2-w/?srsltid=AfmBOoqNyb92PBtsdW7Gr6URe5ktiiJrN-fDEHowkq0M1yyhRgHIRNkZ">PiShop</a>) — $17</li><li>Pi Sugar S (<a href="https://www.amazon.com/Portable-Pwnagotchi-Raspberry-Accessories-handhold/dp/B097RC8KLQ/ref=sr_1_7?crid=DNU86FBPRJHU&amp;dib=eyJ2IjoiMSJ9.cGyvC7zJYGjj9vxDn9QE9nUYiDcyIL_SMihmI5M-c_r0LMcrUtGi8ahGXBMxvB--rLOcVWSgqf3FoF-nrIhUKb0rsthq0SLusTMmCCFx6GY.G1sI7I4fhUF0lZOcYssvK7O49KIwNpZTjNT1KqylJKI&amp;dib_tag=se&amp;keywords=pisugar+3+s&amp;qid=1772224053&amp;sprefix=pisugar+3+s%2Caps%2C198&amp;sr=8-7">Amazon</a>) — $27</li><li>Arducam 5MP OV5647 (<a href="https://www.amazon.com/Arducam-Megapixels-Sensor-OV5647-Raspberry/dp/B012V1HEP4/ref=sr_1_5_sspa?crid=3N22HS2QV9T7E&amp;dib=eyJ2IjoiMSJ9.J9Nthwpd2EPDU2bpYb40cdQ2Vpo0GydPwvWCsOk7S9nwDvs8FW28nxs6QYXpQbO3ke6Bwj5YX_50yz8viimlKGjygd7d0GYEubigKvIyhUs5f4wVLKmFjq0cq1PEEE8YiiLcqdaIqPRTF5q8Wusbm-pXXYXonj99L4Q-y3CskrdFmPf_VAoxoM9RIfaCgLC5Id9KxwlmcNZ4QFsIuTnmq3DxXHs1G_YR_qQeDXnlq1Q.DXLbm6J7QG2squB5XtvmhGPIaWyz3YeXDHbcpjZqTEk&amp;dib_tag=se&amp;keywords=arducam%2Bfor%2Braspberry%2Bpi&amp;qid=1772224354&amp;sprefix=arducam%2Bfor%2Br%2Caps%2C211&amp;sr=8-5-spons&amp;sp_csd=d2lkZ2V0TmFtZT1zcF9tdGY&amp;th=1">Amazon</a>) — $7</li><li>2 x 16GB SD Card (<a href="https://www.amazon.com/Adapter-Memory-Tablet-Console-TF162/dp/B0CYT2DVSQ/ref=sr_1_2_sspa?crid=1TBR64J1C6VF6&amp;dib=eyJ2IjoiMSJ9.ETTtCIBtIkz8tbZK0hf_6BIfWKDVp6eh8SYuHTpsS50AZbjIEFH9pVS-hN77PKBICq19zqn1CIx_TU618tVYfVKZfjrSuKGKSOS1Efq4lPRyQQuQeLhO7MViR_yc8F3-VF4RKcqlm-MI88hjwbJ1Ud9EZudQfOuTQYOSxz8UY2H5l3-NErqNns0ksdPtBZHDGL8Rtxh1OEiZCti4-YdMzh7xJK57oTGulwKk7if6EMI.ymdzHe8hwNeOuZ7-2g2x67I0rGVPq2YnAZx5L36RSV8&amp;dib_tag=se&amp;keywords=sdcard&amp;qid=1772224451&amp;sprefix=sdcar%2Caps%2C212&amp;sr=8-2-spons&amp;sp_csd=d2lkZ2V0TmFtZT1zcF9hdGY&amp;th=1">Amazon</a>) — $10</li></ul><p>Total = $61</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34c10820b1db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Low-Cost, Wireless, Wi-Fi Cam with Python, WebRTC, and a Raspberry Pi Zero 2W(Part 2/ 2)]]></title>
            <link>https://medium.com/@taffealexander/building-a-low-cost-wireless-wi-fi-cam-with-python-webrtc-and-a-raspberry-pi-zero-2w-part-2-2-708f2a9fbca1?source=rss-9da466b62f84------2</link>
            <guid isPermaLink="false">https://medium.com/p/708f2a9fbca1</guid>
            <category><![CDATA[webrtc]]></category>
            <category><![CDATA[video-streaming-software]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[peer-to-peer]]></category>
            <dc:creator><![CDATA[Alexander Taffe]]></dc:creator>
            <pubDate>Mon, 09 Mar 2026 11:32:51 GMT</pubDate>
            <atom:updated>2026-03-11T00:45:28.613Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BkZ1EQpuDSH6JABX" /><figcaption>Photo by <a href="https://unsplash.com/@patrickassale?utm_source=medium&amp;utm_medium=referral">Patrick Assalé</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In <a href="https://medium.com/@taffealexander/building-a-low-cost-wireless-ip-cam-with-python-webrtc-and-a-raspberry-pi-zero-2w-part-1-2-34c10820b1db">part 1</a>, we set up our webcam and implemented signaling. Now, let&#39;s get right into the last step of the WebRTC protocol.</p><ol><li>Signaling</li><li>Connecting</li><li>Securing</li><li><strong>Communicating</strong></li></ol><p>Ok, to start a video stream using WebRTC, we need to add a video track to our peer connection. First, we are going to set up the camera in the streamer, and I&#39;m going to leave out the <em>run_webrtc_streamer_agent()</em> function for now for readability:</p><p><strong><em>streamer.py</em></strong> (Raspberry Pi)</p><pre>from aiortc.contrib.signaling import BYE, add_signaling_arguments, create_signaling<br>from aiortc.mediastreams import VIDEO_TIME_BASE<br>import asyncio<br>import argparse<br>from aiortc import RTCPeerConnection, RTCSessionDescription<br>from picamera2 import Picamera2<br>from picamera2.encoders import H264Encoder<br>from picamera2.outputs import Output<br><br># -------------------------------- NEW ---------------------------------<br>class H264Output(Output):<br>    def __init__(self, loop, pc=None):<br>        super().__init__()<br>        self.queue = asyncio.Queue()<br>        self.loop = loop<br>        self.pc = pc<br><br>    def outputframe(self, frame, keyframe, timestamp_us, packet=None, audio=None):<br>        data = bytes(frame)<br>        if pc.connectionState == &quot;connected&quot;:<br>            self.loop.call_soon_threadsafe(<br>                self.queue.put_nowait,<br>                (data, keyframe, timestamp_us)<br>            )<br><br>class H264StreamTrack(VideoStreamTrack):<br>    def __init__(self, output):<br>        super().__init__()<br>        self.output = output<br>        self.frame_count = 0<br><br>    async def recv(self):<br>        data, keyframe, timestamp_us = await self.output.queue.get()<br>        pkt = Packet(data)<br>        # aiortc uses a clock rate of 90kHz and the timestamp from<br>        # picamera2 is in microseconds so we need to convert the timestamp.<br>        pkt.pts = int(timestamp_us * (90_000 / 1_000_000))<br>        pkt.time_base = VIDEO_TIME_BASE<br>        pkt.is_keyframe = keyframe<br>        self.frame_count += 1<br>        return pkt<br><br># -----------------------------------------------------------------------<br>async def run_webrtc_streamer_agent(pc, signaling, output):<br>    ...<br><br>if __name__ == &quot;__main__&quot;:<br>    parser = argparse.ArgumentParser(description=&quot;Raspberry Pi camera streamer&quot;)<br>    add_signaling_arguments(parser)<br>    args = parser.parse_args()<br><br>    pc = RTCPeerConnection()<br>    signaling = create_signaling(args)<br>    loop = asyncio.new_event_loop()<br>    asyncio.set_event_loop(loop)<br>    # -------------------------------- NEW ---------------------------------<br>    picam = Picamera2()<br>    picam.configure(picam.create_video_configuration(<br>        main={<br>            &quot;size&quot;: (1280, 720),<br>            &quot;format&quot;: &quot;YUV420&quot;},<br>        controls={<br>            &quot;AeEnable&quot;: True,<br>            &quot;AwbEnable&quot;: True,<br>        }))<br>    picam.controls.FrameRate = 25<br>    encoder = H264Encoder(repeat=True, iperiod=25)<br>    h264_output = H264Output(loop, pc)<br>    picam.start()<br>    picam.start_recording(encoder, h264_output, name=&quot;main&quot;)<br>    # ----------------------------------------------------------------------<br>    try:<br>        loop.run_until_complete(run_webrtc_streamer_agent(pc, signaling, h264_output))<br>    except KeyboardInterrupt:<br>        pass<br>    finally:<br>        loop.run_until_complete(signaling.close())<br>        loop.run_until_complete(pc.close())<br>        loop.close()</pre><p>This is the main function from part 1, now with a section that creates and configures the Picamera2 object, and two new classes, <em>H264Output </em>and <em>H264StreamTrack, used to integrate PiCamera2 with</em> aiortc<em>. Let&#39;s</em> take a closer look.</p><h3>Configuring the Camera</h3><pre>picam = Picamera2()<br>picam.configure(picam.create_video_configuration(<br>        main={<br>            &quot;size&quot;: (1280, 720),<br>            &quot;format&quot;: &quot;YUV420&quot;},<br>        controls={<br>            &quot;AeEnable&quot;: True,<br>            &quot;AwbEnable&quot;: True,<br>        }))</pre><p>This section creates a Picamera 2 object and configures it to use YUV420 format and an image size of 1280 x 720. You can use other formats, of course, but using the YUV420 format reduces the amount of data transferred by 50% compared to RGB 888 because it uses 12 bits per pixel vs 24 for RGB 888. This really helps the framerate. Next, we configure the camera such that auto white balance and auto exposure are set to true. There are many more controls that you can set, and I highly recommend reviewing the Picamera <a href="https://pip-assets.raspberrypi.com/categories/652-raspberry-pi-camera-module-2/documents/RP-008156-DS-2-picamera2-manual.pdf?disposition=inline">documentation</a> to learn more about what you can and can’t do with the camera.</p><h3>H.264</h3><p>H.264 is a video compression format, an industry standard set by the ITU-T (International Telecommunication Union) and the ISO/IEC (International Organisation for Standardisation/International Electrotechnical Commission), and a toolkit for video compression according to “<a href="https://www.amazon.com/H-264-Advanced-Video-Compression-Standard/dp/0470516925"><em>The H.264 Advanced Video Compression Standard, second edition,</em></a>” which is a great book by the way if you want to learn how H.264 works! It is also very complex! But to oversimplify it, it is a great way to compress a video that needs to be sent over a network and then decompressed on the other side, which is exactly what we will do. H.265 or HEVC succeeded H.264 in 2013, but the Raspberry Pi Zero 2W has a hardware H.264 encoder, which greatly reduces the load put on the CPU (90%-100% vs 20%) and the memory used while streaming.</p><p>H.264 video compression is an inter-frame compression, meaning that it doesn’t send all complete frames in the bitstream. Instead, it sends partial frames in between complete frames.</p><ul><li><strong>Intra-coded Frames</strong> (I-Frames) — Complete frame.</li><li><strong>Prediction Frames</strong> (P-Frames) — Contains changes between the previous frame and the current frame.</li><li><strong>Bidirectional Frames</strong> (B-Frames) — Contains changes between previous and future frames and the current frame.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*mHu1azf4BhYfWMv1Eo3z4w.png" /><figcaption>From WebRTC for the curious.</figcaption></figure><pre>picam.controls.FrameRate = 25<br>encoder = H264Encoder(repeat=True, iperiod=25)</pre><p>In the code above, iperiod is the number of frames (either P or B) between I-frames. This can be tuned based on the use case, but I’ve read that the best practice is to send a key frame every 1 to 2 seconds. Here I am sending one every 1 second, so <em>iperiod</em> = <em>framerate </em>for easy math :). The repeat=True sends sequence headers with every I-Frame, which is <strong>crucial</strong> for WebRTC to work. Next lets look at the new H264Output class.</p><pre>class H264Output(Output):<br>    def __init__(self, loop, pc=None):<br>        super().__init__()<br>        self.queue = asyncio.Queue()<br>        self.loop = loop<br>        self.pc = pc<br><br>    def outputframe(self, frame, keyframe, timestamp_us, packet=None, audio=None):<br>        data = bytes(frame)<br>        if pc.connectionState == &quot;connected&quot;:<br>            self.loop.call_soon_threadsafe(<br>                self.queue.put_nowait,<br>                (data, keyframe, timestamp_us)<br>            )<br><br>if __name__ == &quot;__main__&quot;:<br>          ...<br>  h264_output = H264Output(loop, pc)<br>          ...</pre><p>Picamera needs a place to put frames once they are encoded using the Raspberry Pi’s hardware encoder, and the H264Output class is just that. PiCamera provides some output classes for this, but to integrate PiCamera with WebRTC, we need to obtain the encoded frames and pass them to WebRTC for transmission to the receiver. This class maintains a queue to store frames, and while our peer connection is in the “<em>connected</em>” state, we start queuing them. <strong>This is important because it prevents the queue from growing too large;</strong> it enqueues frames only while the connection is active. It’s not perfect, but it seems to work so far, and if the queue takes up too much memory due to network bandwidth fluctuations, then you can test adding a max size to the queue. We also include a timestamp from the camera sensor, which WebRTC needs for decoding the frames.</p><p>Next, we’ll pull those frames from the queue and send them to the receiver.</p><pre>from aiortc.mediastreams import VIDEO_TIME_BASE<br><br>class H264StreamTrack(VideoStreamTrack):<br>    def __init__(self, output):<br>        super().__init__()<br>        self.output = output<br><br>    async def recv(self):<br>        data, keyframe, timestamp = await self.output.queue.get()<br>        pkt = Packet(data)<br>        pkt.pts = int(timestamp * 90000 / 1_000_000) # 90kHz<br>        pkt.time_base = VIDEO_TIME_BASE<br>        pkt.is_keyframe = keyframe<br>        return pkt</pre><p>The WebRTC API requires a video/audio track to start sending data, so we create a track that sends H. 264-encoded frames. The <em>H264StreamTrack</em> has a reference to the <em>H264Output </em>class, so<em> </em>it can pull frames from the queue we created. When a frame is requested by the receiver via the <em>recv()</em> function, it dequeues a frame and converts it to a Pyav packet. To decode the H.264 encoded frame, we need a timestamp because frames may arrive out of order. We also need to convert the frame timestamp to the stream’s clock rate for synchronization (90 kHz, from <em>aiortc.mediastreams</em>) and convert it from microseconds to seconds. Then we set the packet&#39;s pts attribute (<a href="https://en.wikipedia.org/wiki/Presentation_timestamp">presentation timestamp</a>) to the converted timestamp. We also need to set the timebase, which is an arbitrary unit of time for the stream; in this case, it is 1/90,000, defined in <em>aiortc.mediastreams</em>.</p><pre>async def run_webrtc_streamer_agent(pc, signaling, output)<br>    track = H264StreamTrack(output)<br>    pc.addTrack(track)<br>    codecs = RTCRtpSender.getCapabilities(&quot;video&quot;).codecs<br>    h264_codecs = [c for c in codecs if c.name == &quot;H264&quot;]<br>    if pc.getTransceivers():<br>        pc.getTransceivers()[0].setCodecPreferences(h264_codecs)</pre><p>Lastly, we create an instance of the track, add it to the peer connection, and then set H.264 as the preferred codec for the peer connection.</p><p>That wraps up the streamer! Now we just have to display the frames in the receiver.</p><p><strong><em>receiver.py</em></strong> (Desktop / Laptop)</p><pre>from aiortc.contrib.signaling import BYE, add_signaling_arguments, create_signaling<br>import asyncio<br>import argparse<br>from aiortc import RTCPeerConnection, RTCSessionDescription<br>import cv2<br><br># ---------------------------------------NEW----------------------------<br>stop_event = asyncio.Event()<br>async def display_video(track, pc):<br>    &quot;&quot;&quot;<br>    Continuously receive and display frames.<br>    &quot;&quot;&quot;<br>    while pc.connectionState != &#39;connected&#39;:<br>        await asyncio.sleep(3)<br><br>    while not stop_event.is_set():<br>        frame = await track.recv()<br>        img = frame.to_ndarray(format=&quot;bgr24&quot;)<br>        cv2.imshow(&quot;WebRTC Video&quot;, img)<br>        if cv2.waitKey(1) &amp; 0xFF == ord(&quot;q&quot;):<br>            stop_event.set()<br>            await pc.close()<br>            break<br># ----------------------------------------------------------------------<br><br>async def run_webrtc_receiver_agent(pc, signaling):<br>#----------------------------------NEW----------------------------------<br>    @pc.on(&quot;track&quot;)<br>    def on_track(track):<br>        if track.kind == &quot;video&quot;:<br>            # To protect against garbage collection<br>            task = asyncio.create_task(display_video(track, pc))<br># ----------------------------------------------------------------------<br>    @pc.on(&quot;connectionstatechange&quot;)<br>    async def on_connectionstatechange():<br>        print(&quot;Connection state is %s&quot;, pc.connectionState)<br><br>    @pc.on(&quot;iceconnectionstatechange&quot;)<br>    async def on_iceconnectionstatechange():<br>        print(&quot;ICE connection state:&quot;, pc.iceConnectionState)<br><br>    await signaling.connect()<br>#----------------------------------UPDATED----------------------------------<br>    print(&quot;Waiting for camera streamer...&quot;)<br>    while not stop_event.is_set():<br>        try:<br>            recv_task = asyncio.create_task(signaling.receive())<br>            stop_task = asyncio.create_task(stop_event.wait())<br>            done, pending = await asyncio.wait(<br>                {recv_task, stop_task},<br>                return_when=asyncio.FIRST_COMPLETED,<br>            )<br>            # Cancel the task that didn&#39;t finish<br>            for t in pending:<br>                t.cancel()<br><br>            # Exit if stop event is set<br>            if stop_task in done:<br>                break<br><br>            # Otherwise we got a signaling object<br>            obj = recv_task.result()<br>        except ConnectionRefusedError:<br>            # Wait for streamer to connect<br>            continue<br># ----------------------------------------------------------------------<br>        if isinstance(obj, RTCSessionDescription):<br>            print(&quot;Connecting...&quot;)<br>            await pc.setRemoteDescription(obj)<br><br>            if obj.type == &quot;offer&quot;:<br>                await pc.setLocalDescription(await pc.createAnswer())<br>                await signaling.send(pc.localDescription)<br>        elif obj is BYE:<br>            break<br>        print(&#39;Connection status: &#39;, pc.connectionState)<br>    if pc.iceConnectionState != &#39;connected&#39;:<br>        await asyncio.gather(*display_tasks)<br>    else:<br>        print(&quot;Error connecting to streamer. Exiting.&quot;)</pre><p>Let&#39;s look at the new stuff:</p><pre>stop_event = asyncio.Event()<br>async def display_video(track, pc):<br>    &quot;&quot;&quot;<br>    Continuously receive and display frames.<br>    &quot;&quot;&quot;<br>    while pc.connectionState != &#39;connected&#39;:<br>        await asyncio.sleep(3)<br><br>    while not stop_event.is_set():<br>        frame = await track.recv()<br>        img = frame.to_ndarray(format=&quot;bgr24&quot;)<br>        cv2.imshow(&quot;WebRTC Video&quot;, img)<br>        if cv2.waitKey(1) &amp; 0xFF == ord(&quot;q&quot;):<br>            stop_event.set()<br>            await pc.close()<br>            break</pre><p>First, there is a new asyncio event called stop_event that will be used to coordinate among asynchronous tasks. Second, the new display_video coroutine handles requesting frames from the streamer and displaying them using OpenCV once the peer-to-peer connection is made. The aiortc library handles decoding the frames and returns a pyav <em>VideoFrame</em> object, which can be converted to a numpy array using the <em>to_ndarray</em> function. Then we simply display the frames using OpenCV.</p><pre>@pc.on(&quot;track&quot;)<br>def on_track(track):<br>    if track.kind == &quot;video&quot;:<br>        # To protect against garbage collection<br>        task = asyncio.create_task(display_video(track, pc))</pre><p>Now that we have a coroutine to display frames, we will add a callback to the peer connection that creates a task to display them when we receive a video track from the streamer. I assign the task to a variable so that there is a reference to it and it is not garbage-collected (something I got from the Python asyncio <a href="https://docs.python.org/3/howto/a-conceptual-overview-of-asyncio.html#a-conceptual-overview-of-asyncio">docs</a>).</p><pre>print(&quot;Waiting for camera streamer...&quot;)<br>while not stop_event.is_set():<br>    try:<br>        recv_task = asyncio.create_task(signaling.receive())<br>        stop_task = asyncio.create_task(stop_event.wait())<br>        done, pending = await asyncio.wait(<br>            {recv_task, stop_task},<br>            return_when=asyncio.FIRST_COMPLETED,<br>        )<br>        # Cancel the task that didn&#39;t finish<br>        for t in pending:<br>            t.cancel()<br><br>        # Exit if stop event is set<br>        if stop_task in done:<br>            break<br><br>        # Otherwise we got a signaling object<br>        obj = recv_task.result()<br>    except ConnectionRefusedError:<br>        # Wait for streamer to connect<br>        continue</pre><p>The last step is to update the signaling logic. Instead of looping until the streamer sends a <em>BYE </em>message, we loop until either the stop_event is set by quitting on the receiver side or the streamer sends <em>BYE</em>. Inside the loop, we create two tasks: one to receive a signaling event and one to wait for the stop event. If a signaling event is received, we cancel the stop event task and handle it as before. If a stop event is received, we exit the loop and terminate the program. We wrap this logic in a try-except block so that if the streamer is not ready, the receiver continues looping until it is ready.</p><p>All done! Now you have a peer-to-peer video stream that uses WebRTC! I have tested this code using a local area network (my home wifi network) and using the Raspberry Pi as a hotspot! The complete code is on <a href="https://github.com/ataffe/RaspberyPiWebCam?tab=readme-ov-file">GitHub</a> in the <strong>part2</strong> folder.</p><p>Here is a video I took using the camera streaming video to my laptop nearby. Apologies for the shakiness I need to get a tripod 😅</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FdvIrJoSG0RE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DdvIrJoSG0RE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FdvIrJoSG0RE%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/77a09e79f8d3ff38bd85b6b699fca64e/href">https://medium.com/media/77a09e79f8d3ff38bd85b6b699fca64e/href</a></iframe><h3>Estimating Frame Rate</h3><p>Here is the code I used to estimate and visualize the frame rate on the video:</p><pre>from PIL import Image, ImageDraw, ImageFont<br>import time<br># Other imports ommitted<br><br>def draw_fps(img, fps):<br>    img = Image.fromarray(img)<br>    draw = ImageDraw.Draw(img)<br>    # Macos<br>    if platform.system() == &quot;Darwin&quot;:<br>        font = ImageFont.truetype(&quot;/System/Library/Fonts/Supplemental/Helvetica.ttc&quot;, 40)<br>    else:<br>        font = ImageFont.load_default(40)<br><br>    # Draw text<br>    draw.text(<br>        (50, 50),<br>        fps,<br>        font=font,<br>        fill=(43, 243, 248)<br>    )<br>    return np.array(img)<br><br>async def display_video(track, pc, model=None):<br>    &quot;&quot;&quot;<br>    Continuously receive and display frames.<br>    &quot;&quot;&quot;<br><br>    while pc.connectionState != &#39;connected&#39;:<br>        await asyncio.sleep(3)<br><br>    logging.info(&quot;Starting video display&quot;)<br>    frame_count = 0<br>    start_time = time.perf_counter()<br>    while True:<br>        try:<br>            async with asyncio.timeout(20):<br>                frame = await track.recv()<br>        except asyncio.TimeoutError:<br>            print(&quot;Timeout while waiting for frame&quot;)<br>            break<br>        img = frame.to_ndarray(format=&quot;bgr24&quot;)<br>        fps = frame_count / (time.perf_counter() - start_time)<br>        img = draw_fps(img, f&quot;{fps:.2f} FPS&quot;)<br>        cv2.imshow(&quot;WebRTC Video&quot;, img)<br>        frame_count += 1<br><br>        if cv2.waitKey(1) &amp; 0xFF == ord(&quot;q&quot;):<br>            break</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=708f2a9fbca1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-Time Instance Segmentation with YOLO, ONNX Runtime, and C++]]></title>
            <link>https://medium.com/@taffealexander/real-time-instance-segmentation-with-yolo-onnx-runtime-and-c-75640a117afa?source=rss-9da466b62f84------2</link>
            <guid isPermaLink="false">https://medium.com/p/75640a117afa</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[instance-segmentation]]></category>
            <category><![CDATA[yolo]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[onnx]]></category>
            <dc:creator><![CDATA[Alexander Taffe]]></dc:creator>
            <pubDate>Wed, 20 Aug 2025 18:42:37 GMT</pubDate>
            <atom:updated>2025-08-21T02:19:13.263Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q1re2ZC5wyJexugLAY02Cw.jpeg" /><figcaption>An image of donuts segmented by YOLOv11. Photo by <a href="https://unsplash.com/@thomas094">Thomas Franke</a> on Unsplash</figcaption></figure><p><em>This is part 2 of a series on using YOLO with ONNX Runtime and C++; see </em><a href="https://medium.com/@taffealexander/object-detection-with-yolo-and-onnx-runtime-in-c-c8bfb964520b"><em>part 1</em></a><em> for setup and object detection. In the last part of this series, I will compare the performance of ONNX Runtime in C++ to PyTorch on a video.</em></p><p>A complete example is available on <a href="https://github.com/ataffe/YoloOnnxRuntimeCPP">GitHub</a></p><h3>Instance Segmentation</h3><p><a href="https://en.wikipedia.org/wiki/Image_segmentation">Instance segmentation</a> is similar to object detection, but it determines which pixels in an image belong to an object, as opposed to finding a box that encloses the object. This typically produces some kind of mask, such as a binary mask. For example, below you can see two binary masks, one for the Giraffe and one for the person.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*AnAYAp0xiowFs2xWPjD3cw.png" /><figcaption>Image and two masks for an image from COCO 2017</figcaption></figure><p>Real-time instance segmentation is a crucial computer vision task in various industries. In logistics, robotic graspers utilize instance segmentation to identify specific objects for tasks like <a href="https://en.wikipedia.org/wiki/Bin_picking">bin-picking</a>. Organ segmentation is utilized in healthcare for purposes such as computer-aided diagnosis and treatment planning. As a research assistant at UC Davis, I assisted with a system that segmented surgical tools to remove them from images as part of a motion correction algorithm for a real-time surgical navigation system.</p><h3>YOLO &amp; YOLOACT</h3><p>If you are searching for a model that can perform instance segmentation in real-time, <a href="https://arxiv.org/abs/1506.02640">YOLO</a> (<strong>Y</strong>ou <strong>O</strong>nly <strong>L</strong>ook <strong>O</strong>nce) is likely to come up in the results of any search (at the time of writing this). YOLO is a machine learning model designed for real-time object detection that was first introduced in the 2016 paper “<a href="https://arxiv.org/abs/1506.02640">You Only Look Once: Unified, Real-Time Object Detection</a>”. Researchers at UC Davis iterated on the design in 2019 in the paper “<a href="https://arxiv.org/abs/1904.02689">YOLACT: Real-time Instance Segmentation</a>” by Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. This design added a segmentation head that enabled real-time instance segmentation. Other researchers and companies have further refined the original designs, making the model more accessible as well. I am now using YOLOv11 from <a href="https://www.ultralytics.com/">Ultralytics</a>, which has an identical output to YOLOACT.</p><h3>ONNX Runtime</h3><p>ONNX Runtime is a cross-platform machine learning model accelerator. The <a href="https://onnx.ai/">ONNX</a> (<strong>O</strong>pen <strong>N</strong>eural <strong>N</strong>etwork <strong>E</strong>xchange) format facilitates the easy porting of machine learning models between frameworks, such as PyTorch and Keras. Together, ONNX Runtime and YOLO make a potent combination for real-time instance segmentation. Software for many existing real-time systems (such as my lab&#39;s surgical navigation system) is written in C++. So, using ONNX, you could develop and train a model using PyTorch and Python and then integrate that model into a real-time system for inference using ONNX Runtime and C++.</p><h3><strong>1. YOLO Segmentation Inputs/Outputs</strong></h3><p>In Part 1, I demonstrated how to export a model to ONNX format and view the file using a convenient app called <a href="https://netron.app/">Netron</a>. Let&#39;s use Netron to take a look at the segmentation version of YOLOv11 from Ultralytics:</p><p><strong>Input</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/167/1*TaACpiXpg_4syOmXBw9w8g.png" /><figcaption>Input shape for YOLOv11 viewed via Netron.</figcaption></figure><p>The input to the segmentation model is the same as the object detection model, a tensor of shape 1 x 3 x 640 x 640, containing the input image:</p><ul><li><strong>Batch Size</strong> = 1 image</li><li><strong>Number of Channels</strong> = 3 (red, green, blue)</li><li><strong>Width</strong> = 640</li><li><strong>Height</strong> = 640</li></ul><p><strong>How YOLOACTs Segmentation Works</strong></p><p>For the outputs to be meaningful, I need to briefly discuss a fundamental aspect of how the segmentation version of YOLO works. To predict masks fast, YOLO performs two tasks in parallel in addition to object detection: mask prototype generation and mask coefficient generation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*23Ymr-ILEtTYverd7Apn_w.png" /><figcaption>YOLOACT Architecture from the YOLOACT paper</figcaption></figure><p><strong>Mask Prototypes</strong> — I think of these as a set of N “universal partial segmentations masks”. Partial because the N prototype masks are combined during post-processing to make the complete segmentation mask for an instance of an object. I say universal, because they are not necessarily specific to any one object in an image; the same prototype mask can contribute to the complete mask for say a basketball in the bottom left corner of an image and a shoe in the same region. I found this to be really interesting, and the <a href="https://arxiv.org/abs/1904.02689">YOLOACT</a> paper provides much more depth on the prototype masks.</p><p>Prototype masks are generated from one of the layers of the pyramid of feature maps output by the YOLO backbone, as shown in the architecture above. The size of the prototype masks is equal to the size of the feature map. In our case, it is the feature map P4 (pyramid layer 4) for YOLOv11n, which is 160 x 160 pixels in size. However, while all masks are combined to make the final segmentation mask, not all masks are necessarily used to create the complete segmentation mask.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aiq4dereHlozc9oy0YbUJw.png" /><figcaption>Zooming in on mask assembly in the architecture</figcaption></figure><p>In the example above from the paper, only four prototype masks are used for simplicity. As shown above, masks 1, 2, and 3 are used for detection 1, which involves the person holding a tennis racket (Not the racket itself). In detection 2, the 2nd and 4th masks are used. The mask coefficients dictate which partial masks to use, and to what extent. We can see in both cases above that some masks are subtracted from the final output.</p><p>The complete mask is a linear combination of the mask prototypes and the mask coefficients. In our case, YOLOv11 outputs N = 32 prototype masks and a vector of 32 mask coefficients. We will need to perform matrix multiplication (a linear combination) between the mask coefficients and the prototype masks. Then, we will upsample the masks to the original image size and crop them using the bounding boxes to get the final output.</p><p><strong>Mask Coefficients</strong> — Dictate which mask prototypes to use and to what extent.</p><p><strong>Output 0: </strong>Bounding Boxes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/171/1*6DLhck9_XZJ34vMTb2KdbA.png" /><figcaption>Bounding box output shape for YOLOv11 segmentation model</figcaption></figure><p>A tensor of shape 1 x 116 x 8400 containing the predicted bounding boxes and the mask coefficients</p><ul><li><strong>Batch Size</strong> = 1</li><li><strong>80 Class Predictions (COCO dataset) + 32 Mask Coefficients + 4 (x, y, width, height)</strong> = 116</li><li><strong>Number of Bounding Boxes Predicted</strong> = 8400</li></ul><p><strong>Output 1: </strong>Prototype<strong> </strong>Segmentation Masks</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/171/1*AFzI7VFqDmqwVQ_lK1r_eA.png" /><figcaption>Proto masks output shape for YOLOv11 segmentation model</figcaption></figure><p>A tensor of shape 1 x 32 x 160 x 160 containing the prototype masks predicted.</p><ul><li><strong>Batch Size</strong> = 1</li><li><strong>Number of Prototype Masks</strong> = 32</li><li><strong>Width</strong> = 160</li><li><strong>Height</strong> = 160</li></ul><h3>Code</h3><p>In this article, I will focus on the differences between the object detection and segmentation versions of YOLO, which occur during the post-processing stage. See <a href="https://medium.com/@taffealexander/object-detection-with-yolo-and-onnx-runtime-in-c-c8bfb964520b">Part 1</a> for setup, exporting YOLO to ONNX format, loading the YOLO model, pre-processing, inference, and post-processing the object detection bounding boxes. Here is a breakdown of the upcoming sections:</p><ol><li><strong>Post-Processing Segmentation Masks</strong></li></ol><p>Here is the main function from Part 1, <strong>without the helper functions</strong>, for readability. Here, we create an ONNX Runtime Environment, load the YOLO model, preprocess the input, postprocess the output, draw the boxes, and then display the image.</p><pre>#include &lt;onnxruntime_cxx_api.h&gt;<br>#include &lt;opencv2/opencv.hpp&gt;<br><br>// Removed helper functions for readability<br><br>int main() {<br>    // Create environment<br>    const Ort::Env env(ORT_LOGGING_LEVEL_ERROR, &quot;Yolo Tutorial&quot;);<br>    OrtCUDAProviderOptionsV2 *cuda_provider = nullptr;<br>    // Load Model<br>    Ort::Session yolo_model_session = LoadYoloModel(<br>        env, &quot;weights/yolo11n-seg-coco.onnx&quot;, cuda_provider);<br>    // Read image<br>    cv::Mat image = cv::imread(&quot;test_images/ex2_coco2017.jpg&quot;);<br>    // Convert image to blob<br>    cv::Mat blob = ImageToBlob(image);<br>    // Convert blob to tensor<br>    Ort::Value input_tensor = BlobToONNXTensor(blob);<br>    // Get Input Names<br>    Ort::AllocatorWithDefaultOptions allocator;<br>    const std::string input_name = yolo_model_session.GetInputNameAllocated(0, allocator).get();<br>    const std::string output_name = yolo_model_session.GetOutputNameAllocated(0, allocator).get();<br>    const char *input_names[] = {input_name.c_str()};<br>    const char *output_names[] = {output_name.c_str()};<br>    // Run inference<br>    std::vector&lt;Ort::Value&gt; output = yolo_model_session.Run(<br>        Ort::RunOptions{nullptr},<br>        input_names,<br>        &amp;input_tensor,<br>        1,<br>        output_names,<br>        1<br>    );<br>    // Get bounding boxes as a mat object<br>    cv::Mat raw_boxes = GetYoloBoxes(output);<br>    // Process bounding boxes<br>    std::vector&lt;YoloBoundingBox&gt; boxes = ProcessYoloBoundingBoxes(raw_boxes, image.size());<br>    // Draw boxes on image<br>    DrawBoxes(boxes, image);<br>    cv::imshow(&quot;Test Image&quot;, image);<br>    cv::imwrite(&quot;output.jpg&quot;, image);<br>    cv::waitKey(0);<br>    Ort::GetApi().ReleaseCUDAProviderOptions(cuda_provider);<br>    return 0;<br>}</pre><p>Here is an example result of the object detection model run on an image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*svcSb8t7VKLcP6cggUbQWg.jpeg" /><figcaption>Object Detection results from the C++ code for an image from COCO 2017</figcaption></figure><p>Let&#39;s start by updating the output tensor names because there are two outputs now:</p><pre>int main() {<br>    //...<br>    // Get Input Names<br>    Ort::AllocatorWithDefaultOptions allocator;<br>    const std::string input_name = yolo_model_session.GetInputNameAllocated(0, allocator).get();<br>    const std::string output_name1 = yolo_model_session.GetOutputNameAllocated(0, allocator).get();<br>    const std::string output_name2 = yolo_model_session.GetOutputNameAllocated(1, allocator).get(); // &lt;- NEW<br>    const char *input_names[] = {input_name.c_str()};<br>    const char *output_names[] = {output_name1.c_str(), output_name2.c_str()}; // &lt;- UPDATED<br><br>    // Run inference<br>    std::vector&lt;Ort::Value&gt; output = yolo_model_session.Run(<br>        Ort::RunOptions{nullptr},<br>        input_names,<br>        &amp;input_tensor,<br>        1,<br>        output_names,<br>        2 // &lt;- UPDATED<br>    );<br>    //...<br>}</pre><p>Next, we’ll write a new function to convert the ONNX Runtime output tensor into a cv::Mat object for processing:</p><pre>cv::Mat getYoloMask(std::vector&lt;Ort::Value&gt; &amp;detection) {<br>    auto maskShape = detection[1].GetTensorTypeAndShapeInfo().GetShape();<br>    std::vector&lt;int&gt; mask_sz = {<br>        1,<br>        static_cast&lt;int&gt;(maskShape[1]), // mask weights<br>        static_cast&lt;int&gt;(maskShape[2]), // mask height<br>        static_cast&lt;int&gt;(maskShape[3]) // mask width<br>    };<br>    return cv::Mat(mask_sz, CV_32F, detection[1].GetTensorMutableData&lt;float&gt;());<br>}</pre><p>This function utilizes the <strong>GetTensorTypeAndShapeInfo</strong> and <strong>GetTensorMutableData </strong>functions from ONNX Runtime to retrieve the tensor&#39;s shape and data. Let&#39;s add this to the main function. I removed the setup and drawing code, for clarity, so of course, don’t try to run this code:</p><pre>int main() {<br>    // ...<br>    // Get Input Names<br>    Ort::AllocatorWithDefaultOptions allocator;<br>    const std::string input_name = yolo_model_session.GetInputNameAllocated(0, allocator).get();<br>    const std::string output_name1 = yolo_model_session.GetOutputNameAllocated(0, allocator).get();<br>    const std::string output_name2 = yolo_model_session.GetOutputNameAllocated(1, allocator).get(); // &lt;- NEW<br>    const char *input_names[] = {input_name.c_str()};<br>    const char *output_names[] = {output_name1.c_str(), output_name2.c_str()}; // &lt;- UPDATED<br><br>    // Run inference<br>    std::vector&lt;Ort::Value&gt; output = yolo_model_session.Run(<br>        Ort::RunOptions{nullptr},<br>        input_names,<br>        &amp;input_tensor,<br>        1,<br>        output_names,<br>        2 // &lt;- UPDATED<br>    );<br>    // Get bounding boxes as a mat object<br>    cv::Mat raw_boxes = GetYoloBoxes(output);<br>    //-----------------------NEW--------------------------------<br>    // Get masks as a Mat object<br>    cv::Mat prototype_masks = getYoloMask(output);<br>    //----------------------------------------------------------<br>    // Process bounding boxes<br>    std::vector&lt;YoloBoundingBox&gt; boxes = ProcessYoloOutput(raw_boxes, image.size());<br>    // ...<br>    return 0;<br>}</pre><p>Each bounding box has one segmentation mask, so let&#39;s update the struct that represents a YOLO bounding box from part 1 of this series to include the mask coefficients and the masks themselves:</p><pre>struct YoloBoundingBox {<br>    cv::Rect bounding_box;<br>    double confidence;<br>    int class_id;<br>    std::vector&lt;float&gt; mask_coefficients; // &lt;-- NEW<br>    cv::Mat mask; // &lt;-- NEW<br>};</pre><p>Next, we need to make a change to the <strong>ProcessYoloBoundingBoxes</strong> function from part 1 of this series as well:</p><pre>std::vector&lt;YoloBoundingBox&gt; ProcessYoloBoundingBoxes(const cv::Mat &amp;raw_boxes, const cv::Size &amp;original_shape) {<br>    int num_classes = 80;<br>    // ----------------------------------------NEW-----------------------------------------<br>    int num_mask_coeffs = 32;<br>    int data_width = num_classes + num_mask_coeffs + 4;// 4 is for x, y, width, height<br>    // ---------------------------------------------------------------------------------------<br>    auto *bounding_box_data = reinterpret_cast&lt;float *&gt;(raw_boxes.data);<br>    std::vector&lt;YoloBoundingBox&gt; processed_boxes;<br>    for (int row = 0; row &lt; raw_boxes.rows; row++) {<br>        // Step 1: Parse x, y, Width, Height, max class score<br>        // Save all class scores 1 - 80 starting at index 4<br>        cv::Mat class_scores = cv::Mat(1, num_classes, CV_32FC1, bounding_box_data + 4);<br>        cv::Point class_id_tmp;<br>        double confidence;<br>        // Find max score<br>        cv::minMaxLoc(class_scores, nullptr, &amp;confidence, nullptr, &amp;class_id_tmp);<br>        // Step 2: Filter boxes by confidence<br>        if (confidence &gt; 0.25) {<br>            float width = bounding_box_data[2];<br>            float height = bounding_box_data[3];<br>            float x = MAX((bounding_box_data[0] - 0.5 * width), 0);<br>            float y = MAX((bounding_box_data[1] - 0.5 * height), 0);<br>            cv::Rect bounding_box(x, y, width, height);<br>            int class_id = class_id_tmp.x;<br>            // ---------------------------------------------------NEW-------------------------------------------<br>            std::vector mask_coefficients(bounding_box_data + 4 + num_classes, bounding_box_data + data_width)<br>            YoloBoundingBox parsed_box(bounding_box, confidence, class_id, mask_coefficients);<br>            // -------------------------------------------------------------------------------------------------<br>            processed_boxes.push_back(parsed_box);<br>        }<br>        // Advance data pointer to next row<br>        bounding_box_data += data_width;<br>    }<br>    // Step 3: Scale boxes<br>    for (auto &amp;box: processed_boxes) {<br>        ScaleYoloBoundingBox(box, original_shape);<br>    }<br>    // Step 4: Non-Max Suppression<br>    std::vector&lt;cv::Rect&gt; boxes;<br>    std::vector&lt;float&gt; confidences;<br>    std::vector&lt;int&gt; filtered_idxs;<br>    for (auto &amp;processed_box: processed_boxes) {<br>        boxes.push_back(processed_box.bounding_box);<br>        confidences.push_back(processed_box.confidence);<br>    }<br>    cv::dnn::NMSBoxes(boxes, confidences, 0.25, 0.70, filtered_idxs);<br>    std::vector&lt;YoloBoundingBox&gt; filtered_boxes;<br>    for (auto &amp;idx: filtered_idxs) {<br>        filtered_boxes.push_back(processed_boxes[idx]);<br>    }<br>    return filtered_boxes;<br>}</pre><p>The two lines below update the data width to include the 32 mask coefficients:</p><pre>int num_mask_coeffs = 32;<br>int data_width = num_classes + num_mask_coeffs + 4;// 4 is for x, y, width, height</pre><p>This section parses the mask coefficients and stores them in the YOLO bounding box struct:</p><pre>std::vector mask_coefficients(bounding_box_data + 4 + num_classes, bounding_box_data + data_width)<br>YoloBoundingBox parsed_box(bounding_box, confidence, class_id, mask_coefficients);</pre><p>Now let&#39;s make a function to process the masks:</p><pre>void ProcessYOLOMasks(std::vector&lt;YoloBoundingBox&gt; &amp;boxes, cv::Mat &amp;proto_masks, cv::Size orig_img_size) {<br>    // TODO: Process masks.<br>}</pre><p>This function takes a vector of bounding boxes, the prototype masks, and the original image size as input and updates the segmentation masks for each bounding box. First, let&#39;s reshape the prototype masks to facilitate matrix multiplication with the coefficients:</p><pre>void ProcessYOLOMasks(std::vector&lt;YoloBoundingBox&gt; &amp;boxes, const cv::Mat &amp;raw_proto_masks, cv::Size orig_img_size) {<br>    int proto_mask_width = raw_proto_masks.size[2];<br>    int proto_mask_height = raw_proto_masks.size[3];<br><br>    // Reshape from 4d: [1, 32, 160, 160] to 2d: [32, 160 * 160]<br>    cv::Mat proto_masks = raw_proto_masks.clone().reshape(0, {<br>                                                              32,<br>                                                              proto_mask_width * proto_mask_height<br>                                                          });<br>}</pre><p>I also saved the original prototype mask size for reshaping the combined prototype masks later. Now we’ll loop through all the boxes and combine the prototype masks using the mask coefficients:</p><pre>void ProcessYOLOMasks(std::vector&lt;YoloBoundingBox&gt; &amp;boxes, const cv::Mat &amp;raw_proto_masks, cv::Size orig_img_size) {<br>    int proto_mask_width = raw_proto_masks.size[2];<br>    int proto_mask_height = raw_proto_masks.size[3];<br><br>    // Reshape from 4d: [1, 32, 160, 160] to 2d: [32, 160 * 160]<br>    cv::Mat proto_masks = raw_proto_masks.clone().reshape(0, {<br>                                                              32,<br>                                                              proto_mask_width * proto_mask_height<br>                                                          });<br>    for (auto &amp;box: boxes) {<br>        // Convert coefficients to matrix and combine mask prototypes and coefficients<br>        cv::Mat combined_masks = cv::Mat(box.mask_coefficients).t() * proto_masks;<br>        // Reshape to 2d: [1, 160 * 160]<br>        combined_masks = combined_masks.reshape(1, {proto_mask_width, proto_mask_height});<br>    }<br>}</pre><p>Make sure to remember to take the transpose of the coefficient matrix, because its shape is 32 x 1, and the mask prototypes are 32 x 160 x 160. Therefore, matrix multiplication is not possible with these two shapes. Ok, now we have a complete mask sized 160 x 160. Here is what they look like with letterbox padding, without cropping, and scaled by 255 so they are visible:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*u-eOiLoUhr8FRKgeWE02NA.jpeg" /><figcaption>Original Image from COCO 2017</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*WUwQfuxx0wgfAXRl4dpRPQ.png" /><figcaption>The four segmentation masks (from left to right: tennis racket, player, person left of player, and person right of player).</figcaption></figure><p>You can see why the masks need to be cropped as well! Next, we need to resize the images while removing the letterbox padding. I’m going to do this in a function called <strong>ResizeMaskRemoveLetterbox</strong>.</p><pre>cv::Mat ResizeMaskRemoveLetterbox(const cv::Mat &amp;input_mask, const cv::Size &amp;orig_image_shape,<br>                                  const cv::Size &amp;letterbox_shape) {<br>    // Calculate scale ratio<br>    float scale_ratio = std::min(<br>        static_cast&lt;float&gt;(letterbox_shape.height) / static_cast&lt;float&gt;(orig_image_shape.height),<br>        static_cast&lt;float&gt;(letterbox_shape.width) / static_cast&lt;float&gt;(orig_image_shape.width));<br><br>    // Calculate unpadded letterboxed image width / height<br>    int unpadded_width = static_cast&lt;int&gt;(std::round(orig_image_shape.width * scale_ratio));<br>    int unpadded_height = static_cast&lt;int&gt;(std::round(orig_image_shape.height * scale_ratio));<br><br>    // Calculate horizontal / vertical padding<br>    int pad_w = letterbox_shape.width - unpadded_width;<br>    int pad_h = letterbox_shape.height - unpadded_height;<br><br>    // Calculate left and top<br>    int pad_left = static_cast&lt;int&gt;(std::round(pad_w / 2.0f));<br>    int pad_top = static_cast&lt;int&gt;(std::round(pad_h / 2.0f));<br><br>    // Step 2: Resize mask to letterbox shape (from 160x160 to 640x640)<br>    cv::Mat resized_mask;<br>    cv::resize(input_mask, resized_mask, letterbox_shape, 0, 0, cv::INTER_LINEAR);<br><br>    // Step 3: Crop out the padding area<br>    cv::Rect roi(pad_left, pad_top, unpadded_width, unpadded_height);<br>    cv::Mat cropped_mask = resized_mask(roi);<br><br>    // Step 4: Resize to desired final output size (640x640)<br>    cv::Mat final_mask;<br>    cv::resize(cropped_mask, final_mask, orig_image_shape, 0, 0, cv::INTER_LINEAR);<br><br>    return final_mask;<br>}</pre><p>Here, I start by getting the scale ratio between the letterboxed image and the original image shape. Then, I calculate the unpadded width and height of the letterboxed image using the scale ratio. Next, I get the horizontal/vertical padding by subtracting the unpadded image size from the padded image size. Divide this by 2 to get the top left of the image. Now I can resize the mask to the letterboxed size (640 x 640), crop it, and then finally upsize it back to the original size. Let&#39;s integrate this into our function:</p><pre>void ProcessYOLOMasks(std::vector&lt;YoloBoundingBox&gt; &amp;boxes, const cv::Mat &amp;raw_proto_masks, cv::Size orig_img_size) {<br>    int proto_mask_width = raw_proto_masks.size[2];<br>    int proto_mask_height = raw_proto_masks.size[3];<br><br>    // Reshape from 4d: [1, 32, 160, 160] to 2d: [32, 160 * 160]<br>    cv::Mat proto_masks = raw_proto_masks.clone().reshape(0, {<br>                                                              32,<br>                                                              proto_mask_width * proto_mask_height<br>                                                          });<br>    for (auto &amp;box: boxes) {<br>        // Convert coefficients to matrix and Combine mask prototypes and coefficients<br>        cv::Mat combined_masks = cv::Mat(box.mask_coefficients).t() * proto_masks;<br>        // Reshape to 2d: [1, 160 * 160]<br>        combined_masks = combined_masks.reshape(1, {proto_mask_width, proto_mask_height});<br>        cv::Mat scaled_mask = ResizeMaskRemoveLetterbox(combined_masks, orig_img_size, cv::Size(640, 640));;<br>    }<br>}</pre><p>Lastly, we need to crop the mask using the bounding box and convert it to a binary mask by multiplying it by 255:</p><pre>void ProcessYOLOMasks(std::vector&lt;YoloBoundingBox&gt; &amp;boxes, const cv::Mat &amp;raw_proto_masks, cv::Size orig_img_size) {<br>    int proto_mask_width = raw_proto_masks.size[2];<br>    int proto_mask_height = raw_proto_masks.size[3];<br><br>    // Reshape from 4d: [1, 32, 160, 160] to 2d: [32, 160 * 160]<br>    cv::Mat proto_masks = raw_proto_masks.clone().reshape(0, {<br>                                                              32,<br>                                                              proto_mask_width * proto_mask_height<br>                                                          });<br>    for (auto &amp;box: boxes) {<br>        // Convert coefficients to matrix and Combine mask prototypes and coefficients<br>        cv::Mat combined_masks = cv::Mat(box.mask_coefficients).t() * proto_masks;<br>        // Reshape to 2d: [1, 160 * 160]<br>        combined_masks = combined_masks.reshape(1, {proto_mask_width, proto_mask_height});<br>        // Scale back to original size and remove letterbox padding<br>        cv::Mat scaled_mask = ResizeMaskRemoveLetterbox(combined_masks, orig_img_size, cv::Size(640, 640));<br>        // --------------------------------------------NEW----------------------------------------------<br>        // Crop to bounding box<br>        cv::Mat cropped_mask = cv::Mat::zeros(scaled_mask.size(), scaled_mask.type());<br>        scaled_mask(box.bounding_box).copyTo(cropped_mask(box.bounding_box));<br>        // Convert to binary mask<br>        cropped_mask.convertTo(box.mask, CV_8UC1, 255);<br>        //----------------------------------------------------------------------------------------------<br>    }<br>}</pre><p>Voila! We have full-sized, processed binary masks that we can draw with! Here is the complete main function with helpers omitted:</p><pre>#include &lt;onnxruntime_cxx_api.h&gt;<br>#include &lt;opencv2/opencv.hpp&gt;<br>#include &quot;letter_box.h&quot;<br><br>// helpers ommitted<br><br>int main() {<br>    // Create environment<br>    const Ort::Env env(ORT_LOGGING_LEVEL_ERROR, &quot;Yolo Tutorial&quot;);<br>    OrtCUDAProviderOptionsV2 *cuda_provider = nullptr;<br>    // Load Model<br>    Ort::Session yolo_model_session = LoadYoloModel(<br>        env, &quot;weights/yolo11n-seg-coco.onnx&quot;, cuda_provider);<br>    // Read image<br>    cv::Mat image = cv::imread(&quot;test_images/ex1_coco2017.jpg&quot;);<br>    // Convert image to blob<br>    cv::Mat blob = ImageToBlob(image);<br>    // Convert blob to tensor<br>    Ort::Value input_tensor = BlobToONNXTensor(blob);<br>    // Get Input Names<br>    Ort::AllocatorWithDefaultOptions allocator;<br>    const std::string input_name = yolo_model_session.GetInputNameAllocated(0, allocator).get();<br>    const std::string output_name1 = yolo_model_session.GetOutputNameAllocated(0, allocator).get();<br>    const std::string output_name2 = yolo_model_session.GetOutputNameAllocated(1, allocator).get();<br>    const char *input_names[] = {input_name.c_str()};<br>    const char *output_names[] = {output_name1.c_str(), output_name2.c_str()};<br><br>    // Run inference<br>    std::vector&lt;Ort::Value&gt; output = yolo_model_session.Run(<br>        Ort::RunOptions{nullptr},<br>        input_names,<br>        &amp;input_tensor,<br>        1,<br>        output_names,<br>        2<br>    );<br>    // Get bounding boxes as a Mat object<br>    cv::Mat raw_boxes = GetYoloBoxes(output);<br>    // Get masks as a Mat object<br>    cv::Mat prototype_masks = getYoloProtoMasks(output);<br>    // Process bounding boxes<br>    std::vector&lt;YoloBoundingBox&gt; boxes = ProcessYoloBoundingBoxes(raw_boxes, image.size());<br>    // Process segmentation masks<br>    ProcessYOLOMasks(boxes, prototype_masks, image.size());<br>    // Draw boxes on image<br>    DrawBoxes(boxes, image);<br>    cv::imshow(&quot;Test Image&quot;, image);<br>    cv::imwrite(&quot;output.jpg&quot;, image);<br>    cv::waitKey(0);<br>    Ort::GetApi().ReleaseCUDAProviderOptions(cuda_provider);<br>    return 0;<br>}</pre><p>Here is how I draw the transparent mask on the image:</p><pre>// Draw mask<br>cv::Mat mask = image.clone();<br>auto mask_color = cv::Scalar(0, 0, 255);<br>mask(box.bounding_box).setTo(mask_color, box.mask(box.bounding_box));<br>cv::addWeighted(image, 0.5, mask, 0.5, 0, image);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*vFQzdBFhGkCylSNeefPmEA.jpeg" /><figcaption>Result of C++ segmentation code run on an image from COCO 2017</figcaption></figure><p>The complete code is available on <a href="https://github.com/ataffe/YoloOnnxRuntimeCPP">GitHub</a>, including additional functions that add the labels and confidence.</p><h3>Thanks for reading :)</h3><p>I am a recent graduate from UC Davis, currently seeking a job. If you or someone you know is seeking a computer vision engineer or a similar role, I would greatly appreciate it if you let me know on <a href="https://www.linkedin.com/in/alexander-taffe">LinkedIn</a> or at taffealexander@gmail.com.</p><h3>References</h3><p>[1] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. 2019. YOLACT: Real-time Instance Segmentation. <em>arXiv:1904.02689 [cs]</em> (October 2019). Retrieved from <a href="https://arxiv.org/abs/1904.02689">https://arxiv.org/abs/1904.02689</a></p><p>[2] FourierMourier. 2023. GitHub — FourierMourier/yolov8-onnx-cpp: YOLOv8 C++ Instance Segmentation, Pose Estimation and Object Detection with ONNX. <em>GitHub</em>. Retrieved August 20, 2025 from <a href="https://github.com/FourierMourier/yolov8-onnx-cpp">https://github.com/FourierMourier/yolov8-onnx-cpp</a></p><p>[3] Sungwon Lee and Ronald M. Summers. 2021. Clinical Artificial Intelligence Applications in Radiology. <em>Radiologic Clinics of North America</em> 59, 6 (November 2021), 987–1002. DOI:https://doi.org/10.1016/j.rcl.2021.07.001</p><p>[4] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2015. You Only Look Once: Unified, Real-Time Object Detection. <em>arXiv.org</em>. Retrieved from <a href="https://arxiv.org/abs/1506.02640">https://arxiv.org/abs/1506.02640</a></p><p>[5] Wikipedia Contributors. 2019. Image segmentation. <em>Wikipedia</em>. Retrieved from <a href="https://en.wikipedia.org/wiki/Image_segmentation">https://en.wikipedia.org/wiki/Image_segmentation</a></p><p>[6] Wikipedia Contributors. 2025. Bin picking. <em>Wikipedia</em>.</p><p>[7] 2024. ONNX Runtime | Home. <em>Onnxruntime.ai</em>. Retrieved from <a href="https://onnxruntime.ai/">https://onnxruntime.ai</a></p><p>[8] Ultralytics | Revolutionizing the World of Vision AI. <a href="http://www.ultralytics.com./"><em>www.ultralytics.com</em>.</a> Retrieved from <a href="https://www.ultralytics.com/">https://www.ultralytics.com/</a></p><p>[9] Netron. <em>netron.app</em>. Retrieved from <a href="https://netron.app/">https://netron.app/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75640a117afa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-Time Object Detection with YOLO and ONNX Runtime in C++]]></title>
            <link>https://medium.com/@taffealexander/object-detection-with-yolo-and-onnx-runtime-in-c-c8bfb964520b?source=rss-9da466b62f84------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8bfb964520b</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[onnx-runtime]]></category>
            <category><![CDATA[yolo]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[object-detection]]></category>
            <dc:creator><![CDATA[Alexander Taffe]]></dc:creator>
            <pubDate>Sat, 16 Aug 2025 01:06:28 GMT</pubDate>
            <atom:updated>2025-08-20T21:51:02.521Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*fFRRGgYb2MxkVzPvb6k6DA.jpeg" /><figcaption>Yolo object detection results on an image from COCO 2017.</figcaption></figure><p><em>TL;DR: Yolo performs object detection in real-time, ONNX is a cross-platform model accelerator, but I couldn’t find a tutorial for C++, so I made this. This is part 1 of a series and covers object detection. </em><a href="https://medium.com/@taffealexander/real-time-instance-segmentation-with-yolo-onnx-runtime-and-c-75640a117afa"><em>Part 2</em></a><em> covers object segmentation. Feel free to skip down to the </em><strong><em>code</em></strong><em> section.</em></p><p>A complete example of this code is available on <a href="https://github.com/ataffe/YoloOnnxRuntimeCPP">GitHub</a></p><p>· <a href="#4bc5">Intro: Detecting Objects in Real-Time?</a><br>· <a href="#46e5">Code</a><br>· <a href="#991c">0. Installing C++ Libs</a><br>· <a href="#f30b">1. Converting Python model to ONNX format</a><br>· <a href="#d7ac">2. Loading the YOLO model</a><br>· <a href="#5b7c">3. Image Pre-processing</a><br>· <a href="#fe8a">4. Creating an ONNX Tensor</a><br>· <a href="#75f9">5. Inference</a><br>· <a href="#1e8d">6. Post Processing</a></p><h3>Intro: Detecting Objects in Real-Time?</h3><p>There are many real-world applications for object detection. Self-driving cars need to be able to recognize objects like stop signs and humans. Precision farming may require the use of a drone to detect pests, weeds, or diseases. In sports, cameras can detect players or lines on the field. In (American) football, the yellow first-down line that is shown on the broadcast is an example of this. In healthcare, surgical guidance systems must operate reliably in real-time. At UC Davis, I had the opportunity to work as a research assistant in the <a href="https://marculab.bme.ucdavis.edu/">Marcu Lab</a>, where they were developing a <a href="https://en.wikipedia.org/wiki/Fluorescence-lifetime_imaging_microscopy">Fluorescence Lifetime Imaging</a>-based surgical navigation system. One thing I worked on there was converting Python code for segmenting surgical instruments in surgical videos to C++, as part of a motion correction algorithm that fits into the larger system.</p><p>One thing all these examples have in common is that they are real-time applications. Therefore, model inference must be very fast so that there is no visible lag. But how fast is fast? For example, if you have a loop that processes each frame, a frame rate of 30 frames per second (fps) is equivalent to a loop time of 33.3 milliseconds (ms). This means you have to complete all processing within 33.3 ms. Enter YOLO.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*porcI4zxUIAl25qeNCEhMw.jpeg" /><figcaption>Cows detected by YOLO in a photo by <a href="https://unsplash.com/@anniespratt?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Annie Spratt</a> on <a href="https://unsplash.com/photos/cows-at-farm-JMjNnQ2xFoY?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><h3>Why YOLO?</h3><p><a href="https://medium.com/@yasser.maree/the-story-of-yolo-72cf7dd7d9bc">YOLO</a> (<strong>Y</strong>ou <strong>O</strong>nly <strong>L</strong>ook <strong>O</strong>nce) is a machine learning model for object detection with real-time performance. It was initially published in <a href="https://arxiv.org/abs/1506.02640">2016 by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.</a> The authors observed a frame rate of 155 fps and a mean Average Precision (mAP) of 52.7 for the fast version of their architecture, which was remarkable. Since then, other researchers have iterated on the original model, and Ultralytics now offers YOLO version 11. While more modern object detection models have emerged, it remains a solid option nonetheless, and thanks to companies like Ultralytics, it’s relatively easy to train and integrate into applications.</p><h3>Why ONNX and ONNX Runtime?</h3><p>Okay, so YOLO can perform real-time object detection, but from what I observed, most implementations are in Python. However, many real-time systems are written in C++ for optimal performance. Including the surgical guidance system I was working on. <strong>But why ONNX?</strong> <a href="https://onnx.ai/">ONNX</a> (Open Neural Network Exchange) is an open file format used to represent machine learning models, initially developed by Meta and Microsoft, which is now supported by a substantial community, including companies such as Amazon, Nvidia, and Intel. ONNX makes it easy to transition between frameworks. You can develop or train a model using PyTorch and easily import it into TensorFlow, etc.</p><p><a href="https://onnxruntime.ai/">ONNX Runtime</a> is a cross-platform machine learning model accelerator focused on fast inference. So why ONNX Runtime? For my project, the lab was already using ONNX when I started helping out. I haven’t found a performance benchmark comparing ONNX runtime to other frameworks like PyTorch and TensorFlow, but that would make an interesting article. Either way, ONNX is designed for fast inference, which aligns with the lab&#39;s use case.</p><h3>Why write about it?</h3><p>As a student, I found it challenging to locate tutorials on how to utilize the ONNX Runtime in C++, beyond some example code. Additionally, for YOLO, I had to implement functionality that the Ultralytics Python module handles internally. Tasks such as letterboxing, parsing the bounding boxes and masks from the model output, applying non-max suppression to the bounding boxes, and scaling the bounding boxes. Hence, this blog post.</p><h3>Object Detection vs. Object Segmentation</h3><p>I consider object segmentation a more accurate version of object detection. Object detection says, “There is a cow somewhere in this box”, and object segmentation says, “These pixels are the cow”. The object segmentation version of YOLO is an extension of the object detection version, incorporating an additional segmentation head. So in this post, I will cover object detection, and object segmentation will be explored in part 2 of this series for brevity.</p><h3>Code</h3><p>Here is the breakdown for the code:</p><ol><li><strong>Converting Python model to ONNX format (Python)</strong></li><li><strong>Loading the YOLO model</strong></li><li><strong>Image Pre-processing</strong></li><li><strong>Creating an ONNX Tensor</strong></li><li><strong>Inference</strong></li><li><strong>Post Processing</strong></li></ol><h3>0. Installing <strong>C++ Libs</strong></h3><p>Here is a rundown of what you will need:</p><ul><li><a href="https://onnxruntime.ai/getting-started">ONNX Runtime</a></li><li><a href="https://opencv.org/get-started/">OpenCV</a></li><li><a href="https://cmake.org/download/">Cmake</a></li><li>C++ compiler</li><li>CUDA Toolkit / cuDNN (GPU Only)</li></ul><p><strong>ONNX Runtime</strong></p><p>The ONNX Runtime website provides instructions for installing ONNX on its <a href="https://onnxruntime.ai/getting-started">getting started</a> page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tDdvFTE3V2WdI6heYhUcIA.png" /><figcaption>ONNX Runtime Installation Selector</figcaption></figure><p>An alternative is to download the pre-built library from the ONNX GitHub <a href="https://github.com/microsoft/onnxruntime/releases">releases</a> page and place it in the folder with the code. Then, point CMake to the include and lib directories, which is what I ultimately did for Linux, as I am using CLion. However, Visual Studio makes it relatively easy, as shown in this <a href="https://www.youtube.com/watch?v=imjqRdsm2Qw">video</a>.</p><p><strong>Important:</strong> If you are using CUDA, ensure that you download the CUDA version of the ONNX Runtime.</p><p><strong>CUDA Toolkit / cuDNN</strong> <strong>(Nvidia GPU Only)</strong></p><p>If you are using an NVIDIA GPU for acceleration, you also need to download the <a href="https://developer.nvidia.com/cuda-downloads">CUDA Toolkit</a>, a development environment for GPU-accelerated applications, and the <a href="https://developer.nvidia.com/cudnn">cuDNN</a> library, which provides tuned implementations of standard operations.</p><p><strong>Important</strong>: Your cuDNN version must match your CUDA toolkit version, and your CUDA toolkit version must match your Nvidia driver version. This is probably user error, but I found myself in <a href="https://sourcegraph.com/blog/nine-circles-of-dependency-hell">dependency hell</a> (ㆆ _ ㆆ). Then, I realized that the CUDA toolkit 13 is available for Ubuntu 24.04; however, the matching drivers were not yet available for Ubuntu.</p><p><strong>My Full Setup</strong></p><ul><li><strong>OS:</strong> Ubuntu 24.04.2 LTS</li><li><strong>GPU:</strong> Nvidia RTX 4070</li><li><strong>ONNX Runtime:</strong> x64 GPU version 1.22.0</li><li><strong>OpenCV:</strong> version 4.6.0</li><li><strong>CMake:</strong> version 3.28.3</li><li><strong>Cuda Toolkit:</strong> version 13.0.48</li><li><strong>CPU:</strong> Intel Core i7–14700KF</li></ul><p>Here is what my CMakeList.txt looks like:</p><p><strong>CMakeLists.txt</strong></p><pre>cmake_minimum_required(VERSION 3.10 FATAL_ERROR)<br><br>project(YoloOnnxC++Tutorial)<br><br>set(ONNXRUNTIME_ROOT &quot;path/to/onnxruntime-linux-x64-gpu-1.22.0&quot;) &lt;- Replace with path to onnx runtime<br>include_directories(${ONNXRUNTIME_ROOT}/include)<br>link_directories(${ONNXRUNTIME_ROOT}/lib)<br><br>find_package(OpenCV REQUIRED)<br><br>add_executable(object_detection object_detection.cpp object_detection.cpp)<br><br>target_link_libraries(object_detection PUBLIC onnxruntime ${OpenCV_LIBS} ${CUDA_LIBRARIES})<br><br>set_property(TARGET object_detection PROPERTY CXX_STANDARD 20)<br>set_target_properties(object_detection PROPERTIES<br>        BUILD_RPATH &quot;${ONNXRUNTIME_ROOT}/lib&quot;<br>        INSTALL_RPATH &quot;${ONNXRUNTIME_ROOT}/lib&quot;<br>)</pre><h3>1. Converting Python model to ONNX format</h3><p>I will assume you have Python, Pip, and the Ultralytics Python module installed already for converting the model. Ultralytics makes it super easy to convert the model to ONNX format. Refer to their page on <a href="https://docs.ultralytics.com/modes/export/">exporting models for ONNX </a>for the available arguments.</p><pre>from ultralytics import YOLO<br><br>model = YOLO(&quot;/path/to/yolo11n.pt&quot;)<br>model.eval()<br>model.export(format=&#39;onnx&#39;, simplify=True)</pre><p>This should generate a “model.onnx” file. For example, I am using a version of YOLO pre-trained on COCO from Ultralytics <strong>(yolo11n.pt), </strong>so the output is <strong>yolo11n.onnx</strong>. You can visualize the model using a web app called <a href="https://netron.app/">Netron</a>.</p><h3>Yolo Inputs / Outputs</h3><p>One of the most important things to understand first is the model’s inputs and outputs. Here are the inputs/ outputs of the object detection version of YOLO 11 visualized using Netron:</p><p><strong>Input: </strong>A batch of images</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/1*F4NTv0l3ieshBClDDXto2g.png" /><figcaption>Input to YOLO visualized using Netron</figcaption></figure><p>The 1 x 3 x 640 x 640 between the “images” node and the first “conv” (convolution) node represents the expected shape of the input tensor for the model:</p><ul><li><strong>Batch Size</strong> = 1 image</li><li><strong>Number of Channels</strong> = 3 (red, green, blue)</li><li><strong>Width</strong> = 640</li><li><strong>Height</strong> = 640</li></ul><p>In plain English, this model expects one image with three channels, sized 640 x 640 pixels.</p><p><strong>Output: </strong>Predicted Bounding Boxes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/1*hEggEKyKXgbQ_fJqIhmxDg.png" /><figcaption>Output of YOLO visualzed using Netron</figcaption></figure><p>The output is trickier, but shout out to Ultralytics GitHub Issue <a href="https://github.com/ultralytics/ultralytics/issues/16238">#16238</a> for really helping me with this! The 1 x 84 x 8400 shape represents:</p><ul><li><strong>Batch Size</strong> = 1</li><li><strong>Class Scores + 4 (x, y, w, h)</strong> = 84</li><li><strong>Number of Bounding Boxes Predicted</strong> = 8400</li></ul><p>The batch size and number of bounding boxes are straightforward, but the second input is more interesting. It is the score for each class that the model (e.g., Person = 0.01, Baseball Bat = 0.98, Cow = 0.02…) plus the coordinates of each box, which are the x, y, width, and height of each bounding box. This version of the model was trained on <a href="https://cocodataset.org/#home">COCO</a>, which has 80 classes; therefore, each of the 8,400 bounding boxes has a predicted score for each of the 80 classes. One thing to note is that YOLO always predicts 8400 boxes, and you have to apply <a href="https://builtin.com/machine-learning/non-maximum-suppression">non-max suppression</a> to remove overlapping boxes. This is important for parsing the bounding boxes predicted by the model later.</p><h3>2. <strong>Loading the YOLO model</strong></h3><p>Ok, finally, let&#39;s load the model. I will start with the CPU version, then show the version that utilizes CUDA.</p><pre>#include &lt;onnxruntime_cxx_api.h&gt;<br><br>int main() {<br>    const Ort::Env env(ORT_LOGGING_LEVEL_WARNING, &quot;Yolo Tutorial&quot;);<br>    Ort::SessionOptions yoloSessionOptions;<br>    Ort::Session yoloModelSession(env, &quot;path/to/model.onnx&quot;, yoloSessionOptions);<br>    return 0;<br>}</pre><p>Ok, let&#39;s break this down:</p><pre>const Ort::Env env(ORT_LOGGING_LEVEL_ERROR, &quot;Yolo Tutorial&quot;);</pre><p>This creates an environment that can be used across sessions (loaded models) and allows for setting the logging level and other settings across sessions.</p><pre>Ort::SessionOptions yoloSessionOptions;<br>Ort::Session yoloModelSession(env, &quot;path/to/model.onnx&quot;, yoloSessionOptions);</pre><p>This creates the session using the session options object, which can be used to specify characteristics for the session, such as the graph <a href="https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html">optimization</a> level. It also allows you to enable CUDA execution.</p><p><strong>A note about memory management:</strong> ONNX is developed using the <a href="https://en.cppreference.com/w/cpp/language/raii.html"><em>Resource Acquisition Is Initialization</em></a><em> (RAII) </em>principle<em>. </em>This means that for objects like the env, the session, and the session options, once the object goes out of scope, the object itself handles releasing resources.</p><p><strong>Using CUDA</strong></p><p>Here is the previous example modified to use CUDA:</p><pre>#include &lt;onnxruntime_cxx_api.h&gt;<br><br>int main() {<br>    const Ort::Env env(ORT_LOGGING_LEVEL_ERROR, &quot;Yolo Tutorial&quot;);<br>    Ort::SessionOptions yoloSessionOptions;<br>    // ------------------------------NEW--------------------------------------------<br>    OrtCUDAProviderOptionsV2 *cudaProvider = nullptr;<br>    Ort::ThrowOnError(Ort::GetApi().CreateCUDAProviderOptions(&amp;cudaProvider));<br>    yoloSessionOptions.AppendExecutionProvider_CUDA_V2(*cudaProvider);<br>    // ------------------------------------------------------------------------------<br>    Ort::Session yoloModelSession(env, &quot;path/to/model.onnx&quot;, yoloSessionOptions);<br>    // ------------------------------NEW---------------------------------------------<br>    Ort::GetApi().ReleaseCUDAProviderOptions(cudaProvider);<br>    // -------------------------------------------------------------------------------<br>    return 0;<br>}</pre><p>Let&#39;s focus on the diff:</p><pre>// Create CUDA provider options<br>OrtCUDAProviderOptionsV2 *cuda_options = nullptr;<br>Ort::ThrowOnError(Ort::GetApi().CreateCUDAProviderOptions(&amp;cuda_options));<br>sessionOptions.AppendExecutionProvider_CUDA_V2(*cuda_options);</pre><p>This adds a CUDA execution provider to the session options. This is what enables the model to run on the GPU. The CUDA execution provider also has options that allow you to specify CUDA-related attributes like input padding. There&#39;s a lot more to the execution provider. See the <a href="https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html">CUDA execution provider</a> page for more details.</p><pre>Ort::ThrowOnError(...)</pre><p>This is a handy wrapper that will throw an error with information instead of a segmentation fault. One exception to that RAII note is that... You must release the OrtCUDAProviderOptionsV2 struct in this case because the AppendExecutionProvider_CUDA_V2 method expects a pointer to a struct, which it uses to allocate the provider. So at the end of the program, you must call:</p><pre>Ort::GetApi().ReleaseCUDAProviderOptions(cuda_provider);</pre><p>Ok, that wraps up loading the model. Let&#39;s put this into a function for slightly easier reading:</p><p>CPU</p><pre>#include &lt;onnxruntime_cxx_api.h&gt;<br><br>Ort::Session LoadYoloModel(const Ort::Env &amp;env, const std::string &amp;model_path) {<br>    Ort::SessionOptions yolo_session_options;<br>    Ort::Session yoloModelSession(env, modelPath.c_str(), yolo_session_options);<br>    return yolo_session_options;<br>}<br><br>int main() {<br>    const Ort::Env env(ORT_LOGGING_LEVEL_ERROR, &quot;Yolo Tutorial&quot;);<br>    Ort::Session yolo_model_session = LoadYoloModel(env, &quot;path/to/model.onnx&quot;);<br>    return 0;<br>}</pre><p>GPU</p><pre>#include &lt;onnxruntime_cxx_api.h&gt;<br><br>Ort::Session LoadYoloModel(const Ort::Env &amp;env, const std::string &amp;model_path, OrtCUDAProviderOptionsV2 *cuda_provider) {<br>    Ort::SessionOptions yolo_session_options;<br>    Ort::ThrowOnError(Ort::GetApi().CreateCUDAProviderOptions(&amp;cuda_provider));<br>    yolo_session_options.AppendExecutionProvider_CUDA_V2(*cuda_provider);<br>    Ort::Session yolo_model_session(env, model_path.c_str(), yolo_session_options);<br>    return yolo_model_session;<br>}<br><br>int main() {<br>    const Ort::Env env(ORT_LOGGING_LEVEL_ERROR, &quot;Yolo Tutorial&quot;);<br>    OrtCUDAProviderOptionsV2 *cuda_provider = nullptr;<br>    Ort::Session yolo_model_session = LoadYoloModel(env, &quot;path/to/model.onnx&quot;, cuda_provider);<br>    // Clean up<br>    Ort::GetApi().ReleaseCUDAProviderOptions(cuda_provider);<br>    return 0;<br>}</pre><p>From now on, I will only show the CPU version; however, from this point forward, there is no difference between the two.</p><h3><strong>3. Image Pre-processing</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*sDSCgxWrhz1k9ghZYnAmIQ.jpeg" /><figcaption>Example image from COCO 2017 Letterboxed</figcaption></figure><p>Let&#39;s clear out the model loading stuff for now and start with loading a single image using OpenCV.:</p><pre>#include &lt;opencv2/imgcodecs.hpp&gt; // For cv::imread<br><br>int main() {<br>    cv::Mat image = cv::imread(&quot;/path/to/image.jpg&quot;);<br>    return 0;<br>}</pre><p>Now here is our task list:</p><ol><li><strong>Resize Image</strong> — Going back to the input section, the YOLO model expects a tensor of shape 1 x 3 x 640 x 640. Therefore, if the image is larger than this, we need to downsize it. However, we must also maintain the original aspect ratio, as this could warp the image and otherwise affect the model&#39;s accuracy. I won&#39;t delve into letterboxing details here, as this post is already lengthy enough. However, an excellent <a href="https://medium.com/@reachraktim/letterboxing-in-yolov5-yolov7-yolov8-an-intuitive-explanation-with-python-code-88f7d4323d6c">overview</a> is available, and the GitHub repository includes a LetterBox implementation.</li><li><strong>Scale Image </strong>— YOLO expects the image values to be between 0 and 1, rather than an 8-bit unsigned integer with values between 0 and 255, so we need to scale the image accordingly.</li><li><strong>Swap Channels</strong> — When OpenCV reads an image using “imread,” it reads the image in the Blue, Green, Red (BGR) order, but YOLO expects the image to be in RGB order.</li></ol><p>Luckily, OpenCV has a function “<a href="https://docs.opencv.org/3.4/d6/d0f/group__dnn.html#ga29f34df9376379a603acd8df581ac8d7">blobFromImage</a>” in the DNN module that performs all of this for us and returns a <a href="https://cloudsecurityalliance.org/blog/2022/05/04/what-is-a-blob-binary-large-object-can-it-be-tokenized"><em>Binary Large Object (BLOB)</em></a>,<em> </em>which is perfect for creating a tensor, which is next:</p><pre>#include &lt;opencv2/opencv.hpp&gt;<br><br>int main() {<br>    cv::Mat image = cv::imread(&quot;/path/to/image.jpg&quot;);<br>    // Resize to yolo input size<br>    cv::Mat resized_image;<br>    LetterBox(image, resized_image, cv::Size(640, 640));<br>    // Pre-process image<br>    cv::Mat blob = cv::dnn::blobFromImage(<br>        resized_image,<br>        1.0 / 255.0, // Scale to 0 -&gt; 1<br>        cv::Size(640, 640),<br>        cv::Scalar::all(0),<br>        true, // BGR -&gt; RGB<br>        false,<br>        CV_32F);<br>    return 0;<br>}</pre><p>Done! Let&#39;s put all of this into a function as well:</p><pre>#include &lt;opencv2/opencv.hpp&gt;<br><br>cv::Mat LoadImage(const std::string &amp;image_path) {<br>    cv::Mat image = cv::imread(image_path);<br>    // Resize to yolo input size<br>    cv::Mat resized_image;<br>    LetterBox(image, resized_image, cv::Size(640, 640));<br>    // Pre-process image<br>    return cv::dnn::blobFromImage(<br>        resized_image,<br>        1.0 / 255.0,<br>        cv::Size(640, 640),<br>        cv::Scalar::all(0),<br>        true,<br>        false,<br>        CV_32F);<br>}<br><br>int main() {<br>    cv::Mat blob = LoadImage(&quot;/path/to/image.jpg&quot;);<br>    return 0;<br>}</pre><h3><strong>4. Creating an ONNX Tensor</strong></h3><p>Now, let&#39;s continue to ignore the model loading code for now and use the LoadImage function we created to create an ONNX Runtime tensor:</p><pre>#include &lt;onnxruntime_cxx_api.h&gt;<br><br>int main() {<br>    cv::Mat blob = LoadImage(yolo_model_path);<br>    // Create Tensor<br>    std::vector&lt;int64_t&gt; input_shape = {1, 3, 640, 640};<br>    int64_t input_size = 3 * 640 * 640;<br>    Ort::MemoryInfo memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);<br>    Ort::Value tensor = Ort::Value::CreateTensor&lt;float&gt;(<br>        memory_info,<br>        (float *) (blob.data), // C++ style casting will break this.<br>        input_size,<br>        input_shape.data(),<br>        input_shape.size()<br>    );<br>    return 0;<br>}</pre><p>The ONNX runtime function Ort::Value::CreateTensor takes 5 input args:</p><ul><li><strong>Memory Info</strong> — Information on allocating memory for the tensor.</li></ul><pre>Ort::MemoryInfo memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);</pre><p>As of the time of writing, there are two types of memory allocators. <em>OrtArenaAllocator</em>, which requests a pool of reusable memory, and <em>OrtDeviceAllocator</em>, which requests memory from the device when needed and then frees it, rather than returning it to the pool. The device allocator has more overhead, but requests smaller amounts of memory at one time, whereas the arena allocator has less overhead but uses more peak memory.</p><ul><li><strong>Pointer to data </strong>— A pointer to the data for the tensor. The data must be stored continuously in memory; otherwise, you will likely encounter an exception or a segmentation fault. When OpenCV creates a new Mat object, the data is stored continuously in memory. Using the cv::dnn::blobFromImage function creates a new Mat object, so the image is stored in memory continuously. You can always add a check for this using the Mat object’s isContinuous() method. Yolo requires the input to be a float, so our tensor is a float, and we need to cast the pointer to a float pointer as well. I also tried C++ style casting, but it caused a seg fault for some reason :(</li><li><strong>Tensor Data size </strong>— The size of the data for the tensor in bytes; in this case, it would be 1 * 3 * 640 * 640 = 1228800 bytes</li><li><strong>Tensor Shape</strong> — A vector containing the shape of the tensor 1 x 3 x 640 x 640 in this case.</li><li><strong>Tensor Shape Size</strong> — The size of the vector containing the shape of the tensor; in this case, it would be 4.</li></ul><p>Ok, we created a tensor, and we can now run the model on our input image. Let&#39;s put this into a function once again for readability:</p><pre>#include &lt;onnxruntime_cxx_api.h&gt;<br>#include &lt;opencv2/opencv.hpp&gt;<br><br>cv::Mat LoadImage(const std::string &amp;image_path) {<br>    cv::Mat image = cv::imread(image_path);<br>    // Resize to yolo input size<br>    cv::Mat resized_image;<br>    LetterBox(image, resized_image, cv::Size(640, 640));<br>    // Pre-process image<br>    return cv::dnn::blobFromImage(<br>        resized_image,<br>        1.0 / 255.0,<br>        cv::Size(640, 640),<br>        cv::Scalar::all(0),<br>        true,<br>        false,<br>        CV_32F);<br>}<br><br>Ort::Value BlobToONNXTensor(const cv::Mat &amp;blob) {<br>    std::vector&lt;int64_t&gt; tensor_shape = {1, 3, 640, 640};<br>    int64_t blob_size = 3 * 640 * 640;<br>    Ort::MemoryInfo memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);<br>    return Ort::Value::CreateTensor&lt;float&gt;(<br>        memory_info,<br>        (float *) (blob.data), // C++ style casting will break this.<br>        blob_size,<br>        tensor_shape.data(),<br>        tensor_shape.size()<br>    );<br>}<br><br>int main() {<br>    cv::Mat blob = LoadImage(&quot;/path/to/image.jpg&quot;);<br>    Ort::Value input_tensor = BlobToONNXTensor(blob);<br>    return 0;<br>}</pre><h3><strong>5. Inference</strong></h3><p>We now have an input tensor for the model; let&#39;s run inference using Session.Run() method which requires the following args:</p><ul><li><strong>Run Options</strong> — We’ll leave this null</li><li><strong>Input Names</strong> — A pointer to a list of C-style strings.</li><li><strong>Input Tensor</strong> — From the BlobToONNXTensor function</li><li><strong>Input Names Count </strong>— Number of input names.</li><li><strong>Output Names </strong>— Same as input names.</li><li><strong>Output Names Count </strong>— Number of Output Names</li></ul><p>First, we need to get the input name (images) and output name (output0) from the YOLO model session. You can verify these names by examining the model using the Netron app. I’m going to clear out the pre-processing and focus on the session and the input/output names:</p><pre>#include &lt;onnxruntime_cxx_api.h&gt;<br><br>int main {<br>  const Ort::Env env(ORT_LOGGING_LEVEL_ERROR, &quot;Yolo Tutorial&quot;);<br>  Ort::Session yolo_model_session = LoadYoloModel(env, yolo_model_path);<br>  Ort::AllocatorWithDefaultOptions allocator;<br>  const std::string input_name = yolo_model_session.GetInputNameAllocated(0, allocator).get();<br>  const std::string output_name = yolo_model_session.GetOutputNameAllocated(0, allocator).get();<br>  const char *input_names[] = { input_name.c_str()};<br>  const char *output_names[] = {output_name.c_str()};<br>}</pre><p>Let&#39;s break this down:</p><pre>Ort::AllocatorWithDefaultOptions allocator;</pre><p>Creates an allocator that allocates memory on the CPU for the input and output names.</p><pre>std::string input_name = yolo_model_session.GetInputNameAllocated(0, allocator).get();<br>std::string output_name = yolo_model_session.GetOutputNameAllocated(0, allocator).get();<br>const char *input_names[] = { input_name.c_str()};<br>const char *output_names[] = {output_name.c_str()};</pre><p>The GetInputNameAllocated and the GetOutputNameAllocated functions get the input and output names at index 0 using the allocator. We can get away with this because the object detection version of YOLO has one input and one output. However, if we were to use the segmentation version of YOLO, which has two names, we would have to create a vector of names and use a loop. Also, the GetInputNameAllocated.get() returns a pointer to the string. We must assign it to something because the ONNX follows the <em>RAII</em> principle, so if we don’t copy it to something, in this case a string, it will be deallocated, and you might see an error like “input name cannot be empty.”</p><p>Okay, we have the input and output names; let&#39;s integrate this with our complete program so far (minus the letterbox function) and finally run the model!</p><pre>#include &lt;onnxruntime_cxx_api.h&gt;<br>#include &lt;opencv2/opencv.hpp&gt;<br>#include &quot;letter_box.h&quot;<br><br>// Removed functions for readability.<br><br>int main {<br>  const Ort::Env env(ORT_LOGGING_LEVEL_ERROR, &quot;Yolo Tutorial&quot;);<br>  Ort::Session yolo_model_session = LoadYoloModel(env, yolo_model_path);<br>  cv::Mat blob = LoadImage(&quot;/path/to/image.jpg&quot;);<br>  Ort::Value input_tensor = BlobToONNXTensor(blob);<br>  Ort::AllocatorWithDefaultOptions allocator;<br>  const std::string input_name = yolo_model_session.GetInputNameAllocated(0, allocator).get();<br>  const std::string output_name = yolo_model_session.GetOutputNameAllocated(0, allocator).get();<br>  const char *input_names[] = { input_name.c_str()};<br>  const char *output_names[] = {output_name.c_str()};<br>  std::vector&lt;Ort::Value&gt; output = yolo_model_session.Run(<br>      Ort::RunOptions{nullptr},<br>      input_names,<br>      &amp;input_tensor,<br>      1,<br>      output_names,<br>      1<br>  );<br>}</pre><p>Okay, we have the output bounding boxes of the model; it&#39;s time to parse them.</p><h3><strong>6. Post Processing</strong></h3><p>First, we are going to make a function that stores the raw bounding boxes in a cv::Mat object:</p><pre>cv::Mat getYoloBoxes(std::vector&lt;Ort::Value&gt; &amp;output) {<br>    std::vector&lt;int64_t&gt; output_shape = output[0].GetTensorTypeAndShapeInfo().GetShape();<br>    // [batch size, mask weights, boxes predicted]=&gt;[bs, preds_num, features]<br>    return cv::Mat(<br>        cv::Size(static_cast&lt;int&gt;(output_shape[2]), // 8400<br>                 static_cast&lt;int&gt;(output_shape[1])),// 84<br>        CV_32F,<br>        output[0].GetTensorMutableData&lt;float&gt;()).t();<br>}<br><br>int main() {<br>  std::vector&lt;Ort::Value&gt; output = yoloModelSession.Run(<br>        Ort::RunOptions{nullptr},<br>        input_names,<br>        &amp;input_tensor,<br>        1,<br>        output_names,<br>        1<br>    );<br>  <br>  cv::Mat raw_boxes = getYoloBoxes(detected_objects);<br>}</pre><pre>std::vector&lt;int64_t&gt; output_shape = output[0].GetTensorTypeAndShapeInfo().GetShape();</pre><p>This line retrieves the output shape, which is 1 x 84 x 8400 for YOLO trained on COCO (See model output section). Next, we’ll use this to create a cv::Mat object that holds the 8400 output bounding boxes:</p><pre>return cv::Mat(<br>        cv::Size(static_cast&lt;int&gt;(output_shape[2]), // 8400<br>                 static_cast&lt;int&gt;(output_shape[1])), // 84<br>        CV_32F,<br>        output[0].GetTensorMutableData&lt;float&gt;()).t();</pre><p><strong>Note:</strong> For convenience, I use the transpose of the output, so that the shape of the output is now 8400 x 84; therefore, each row represents a bounding box.</p><p>Now that we have the bounding boxes, we need to do the following to get the actual detected objects:</p><ol><li>Parse the x,y, width, height, and max confidence score class</li><li>Filter boxes by confidence, retaining only those above a specified threshold. I will set this at 0.25 based on an <a href="https://github.com/ultralytics/ultralytics/blob/main/examples/YOLOv8-Segmentation-ONNXRuntime-Python/main.py">example</a> from Ultralytics.</li><li>Scale the boxes back to the size of the input image and undo the letterbox pre-processing.</li><li>Perform non-max suppression to remove duplicate boxes.</li></ol><p>Let&#39;s start by making a struct to represent a YOLO bounding box:</p><pre>struct YoloBoundingBox {<br>    cv::Rect bounding_box;<br>    double confidence;<br>    int class_id;<br>};</pre><p>Now, let&#39;s create a function that will handle the four tasks above:</p><pre>std::vector&lt;YoloBoundingBox&gt; ProcessYoloOutput(const cv::Mat &amp;raw_boxes) {<br>    int num_classes = 80;<br>    int data_width = num_classes + 4;<br>    auto *bounding_box_data = reinterpret_cast&lt;float *&gt;(raw_boxes.data);<br>    std::vector&lt;YoloBoundingBox&gt; parsed_boxes;<br>    // Step 1: Parse x, y, Width, Height, max class score<br>    // Step 2: Filter boxes by confidence<br>    // Step 3: Non-Max Suppression<br>    // Step 4: Scale boxes<br>}</pre><p>This function includes two variables to help with processing: a pointer to the data and a vector to store the results.</p><p><strong>Step 1: Parse x, y, width, height, and max class score</strong></p><p>Here is what the data in each row looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*5TDgknmj7UxbqPqT0zKXzg.png" /></figure><p>Using the pointer to the data, I’ll scroll from left to right, getting the first four indices, and then I’ll store all the confidence scores in a Mat object and use OpenCV’s <strong>minMaxLoc </strong>function<strong> </strong>to find the max score:</p><pre>std::vector&lt;YoloBoundingBox&gt; ProcessYoloOutput(const cv::Mat &amp;raw_boxes) {<br>    int num_classes = 80;<br>    int data_width = num_classes + 4;<br>    auto *bounding_box_data = reinterpret_cast&lt;float *&gt;(raw_boxes.data);<br>    std::vector&lt;YoloBoundingBox&gt; parsed_boxes;<br>    // ----------------------NEW -----------------------------------------<br>    for (int row = 0; row &lt; raw_boxes.rows; row++) {<br>        // Step 1: Parse x, y, Width, Height, max class score<br>        float width = bounding_box_data[2];<br>        float height = bounding_box_data[3];<br>        float x = MAX((bounding_box_data[0] - 0.5 * width), 0);<br>        float y = MAX((bounding_box_data[1] - 0.5 * height), 0);<br>        cv::Rect bounding_box(x, y, width, height);<br>        // Save all class scores 1 - 80 starting at index 4<br>        cv::Mat class_scores = cv::Mat(1, num_classes, CV_32FC1, bounding_box_data + 4);<br>        cv::Point class_id_tmp;<br>        double confidence;<br>        // Find max score<br>        cv::minMaxLoc(class_scores, nullptr, &amp;confidence, nullptr, &amp;class_id_tmp);<br>        int class_id = class_id_tmp.x;<br>        YoloBoundingBox parsed_box(bounding_box,confidence,class_id);<br>        parsed_boxes.push_back(parsed_box);<br>        // Advance data pointer to next row<br>        bounding_box_data += data_width;<br>    }<br>    // ---------------------------------------------------------------------<br>    // Step 2: Filter boxes by confidence<br>    // Step 3: Non-Max Suppression<br>    // Step 4: Scale boxes<br>}</pre><p><strong>Step 2: Filtering boxes by confidence</strong></p><p>Here I’ll modify the previous code only to save boxes with a confidence above a hard-coded threshold:</p><pre>std::vector&lt;YoloBoundingBox&gt; ProcessYoloOutput(const cv::Mat &amp;raw_boxes) {<br>    int num_classes = 80;<br>    int data_width = num_classes + 4;<br>    auto *bounding_box_data = reinterpret_cast&lt;float *&gt;(raw_boxes.data);<br>    std::vector&lt;YoloBoundingBox&gt; processed_boxes;<br>    for (int row = 0; row &lt; raw_boxes.rows; row++) {<br>        // Step 1: Parse x, y, Width, Height, max class score<br>        float width, height = bounding_box_data[2];<br>        float height = bounding_box_data[3];<br>        float x = MAX((bounding_box_data[0] - 0.5 * width), 0);<br>        float y = MAX((bounding_box_data[1] - 0.5 * height), 0);<br>        cv::Rect bounding_box(x, y, width, height);<br>        // Save all class scores 1 - 80 starting at index 4<br>        cv::Mat class_scores = cv::Mat(1, num_classes, CV_32FC1, bounding_box_data + 4);<br>        cv::Point class_id_tmp;<br>        double confidence;<br>        // Find max score<br>        cv::minMaxLoc(class_scores, nullptr, &amp;confidence, nullptr, &amp;class_id_tmp);<br>        // ------------------------------NEW------------------------------------------<br>        // Step 2: Filter boxes by confidence<br>        if (confidence &gt; 0.70) {<br>            int class_id = class_id_tmp.x;<br>            YoloBoundingBox parsed_box(bounding_box,confidence,class_id);<br>            processed_boxes.push_back(parsed_box);<br>        }<br>        // ----------------------------------------------------------------------------<br>        // Advance data pointer to next row<br>        bounding_box_data += data_width;<br>    }<br>    // Step 3: Non-Max Suppression<br>    // Step 4: Scale boxes<br>}</pre><p><strong>Step 3: Scale the Bounding Boxes</strong></p><p><strong>Scaling the boxes</strong></p><pre>void ScaleYoloBoundingBox(YoloBoundingBox &amp;box, const cv::Size &amp;original_shape) {<br>    cv::Size yolo_shape(640, 640);<br>    float scale_ratio = std::min(static_cast&lt;float&gt;(yolo_shape.width) / static_cast&lt;float&gt;(original_shape.width),<br>        static_cast&lt;float&gt;(yolo_shape.height) / static_cast&lt;float&gt;(original_shape.height));<br>    float letterbox_pad_x = (yolo_shape.width - original_shape.width * scale_ratio) / 2;<br>    float letterbox_pad_y = (yolo_shape.height - original_shape.height * scale_ratio) / 2;<br>    box.bounding_box.x -= letterbox_pad_x;<br>    box.bounding_box.y -= letterbox_pad_y;<br>    box.bounding_box.x /= scale_ratio;<br>    box.bounding_box.y /= scale_ratio;<br>    box.bounding_box.width /= scale_ratio;<br>    box.bounding_box.height /= scale_ratio;<br>    ClipBox(box.bounding_box, original_shape);<br>}</pre><p>In the example above, we first need to determine the scale ratio between the YOLO input size (640 x 640) and the original image size, say (1920 x 1080). Then, we use this ratio to determine how much letterbox padding was added to the downsized image vertically and horizontally. Then we remove any padding and scale down using the scale ratio. We also need to clip the bounding boxes in case they extend outside the image bounds.</p><p><strong>Clipping Boxes</strong></p><pre>void ClipBox(cv::Rect &amp;box, const cv::Size &amp;shape) {<br>    box.x = std::max(0, std::min(box.x, shape.width));<br>    box.y = std::max(0, std::min(box.y, shape.height));<br>    box.width = std::max(0, std::min(box.width, shape.width - box.x));<br>    box.height = std::max(0, std::min(box.height, shape.height - box.y));<br>}</pre><p>This is straightforward; we need to ensure that the bounding boxes are within the original image’s bounds.</p><p>Let&#39;s integrate these two functions:</p><pre>std::vector&lt;YoloBoundingBox&gt; ProcessYoloOutput(const cv::Mat &amp;raw_boxes, const cv::Size &amp;original_shape) {<br>    int num_classes = 80;<br>    int data_width = num_classes + 4;<br>    auto *bounding_box_data = reinterpret_cast&lt;float *&gt;(raw_boxes.data);<br>    std::vector&lt;YoloBoundingBox&gt; processed_boxes;<br>    for (int row = 0; row &lt; raw_boxes.rows; row++) {<br>        // Step 1: Parse x, y, Width, Height, max class score<br>        float width = bounding_box_data[2];<br>        float height = bounding_box_data[3];<br>        float x = MAX((bounding_box_data[0] - 0.5 * width), 0);<br>        float y = MAX((bounding_box_data[1] - 0.5 * height), 0);<br>        cv::Rect bounding_box(x, y, width, height);<br>        // Save all class scores 1 - 80 starting at index 4<br>        cv::Mat class_scores = cv::Mat(1, num_classes, CV_32FC1, bounding_box_data + 4);<br>        cv::Point class_id_tmp;<br>        double confidence;<br>        // Find max score<br>        cv::minMaxLoc(class_scores, nullptr, &amp;confidence, nullptr, &amp;class_id_tmp);<br>        // Step 2: Filter boxes by confidence<br>        if (confidence &gt; 0.25) {<br>            int class_id = class_id_tmp.x;<br>            YoloBoundingBox parsed_box(bounding_box, confidence, class_id);<br>            processed_boxes.push_back(parsed_box);<br>        }<br>        // Advance data pointer to next row<br>        bounding_box_data += data_width;<br>    }<br>    //-----------------------------------NEW---------------------------------<br>    // Step 3: Scale boxes<br>    for (auto &amp;box: processed_boxes) {<br>        ScaleYoloBoundingBox(box, original_shape);<br>    }<br>    //------------------------------------------------------------------------<br>}</pre><p><strong>Step 4: Non-Max Supression</strong></p><p>Ok, now let&#39;s remove duplicates! <a href="https://learnopencv.com/non-maximum-suppression-theory-and-implementation-in-pytorch/">Non-Max Suppression</a> is a post-processing algorithm used to refine bounding boxes. Given a box, it removes overlapping boxes with lower confidence scores. OpenCV’s NMSBoxes function from the dnn module can do this for us; however, to do this, we’ll need to split the boxes and confidences into two separate vectors:</p><pre>std::vector&lt;YoloBoundingBox&gt; ProcessYoloOutput(const cv::Mat &amp;raw_boxes, const cv::Size &amp;original_shape) {<br>    int num_classes = 80;<br>    int data_width = num_classes + 4;<br>    auto *bounding_box_data = reinterpret_cast&lt;float *&gt;(raw_boxes.data);<br>    std::vector&lt;YoloBoundingBox&gt; processed_boxes;<br>    for (int row = 0; row &lt; raw_boxes.rows; row++) {<br>        // Step 1: Parse x, y, Width, Height, max class score<br>        // Save all class scores 1 - 80 starting at index 4<br>        cv::Mat class_scores = cv::Mat(1, num_classes, CV_32FC1, bounding_box_data + 4);<br>        cv::Point class_id_tmp;<br>        double confidence;<br>        // Find max score<br>        cv::minMaxLoc(class_scores, nullptr, &amp;confidence, nullptr, &amp;class_id_tmp);<br>        // Step 2: Filter boxes by confidence<br>        if (confidence &gt; 0.25) {<br>            float width = bounding_box_data[2];<br>            float height = bounding_box_data[3];<br>            float x = MAX((bounding_box_data[0] - 0.5 * width), 0);<br>            float y = MAX((bounding_box_data[1] - 0.5 * height), 0);<br>            cv::Rect bounding_box(x, y, width, height);<br>            int class_id = class_id_tmp.x;<br>            YoloBoundingBox parsed_box(bounding_box, confidence, class_id);<br>            processed_boxes.push_back(parsed_box);<br>        }<br>        // Advance data pointer to next row<br>        bounding_box_data += data_width;<br>    }<br>    // Step 3: Scale boxes<br>    for (auto &amp;box: processed_boxes) {<br>        ScaleYoloBoundingBox(box, original_shape);<br>    }<br>    // ----------------------------------NEW----------------------------------<br>    // Step 4: Non-Max Suppression<br>    std::vector&lt;cv::Rect&gt; boxes;<br>    std::vector&lt;float&gt; confidences;<br>    std::vector&lt;int&gt; filtered_idxs;<br>    for (auto &amp;processed_box: processed_boxes) {<br>        ScaleYoloBoundingBox(processed_box, original_shape);<br>        boxes.push_back(processed_box.bounding_box);<br>        confidences.push_back(processed_box.confidence);<br>    }<br>    cv::dnn::NMSBoxes(boxes, confidences, 0.25, 0.70, filtered_idxs);<br>    std::vector&lt;YoloBoundingBox&gt; filtered_boxes;<br>    for (auto &amp;idx: filtered_idxs) {<br>        filtered_boxes.push_back(processed_boxes[idx]);<br>    }<br>    //--------------------------------------------------------------------------<br>    return filtered_boxes;<br>}</pre><p>NMSBoxes updates the <strong>filtered_idxs</strong> variable, and so we add those boxes to a new vector, and that&#39;s it! We have our bounding boxes! We can draw the boxes using OpenCV’s rectangle function and display the image. Here is the complete program minus the letter box code:</p><pre>#include &lt;onnxruntime_cxx_api.h&gt;<br>#include &lt;opencv2/opencv.hpp&gt;<br>#include &quot;letter_box.h&quot;<br><br>struct YoloBoundingBox {<br>    cv::Rect bounding_box;<br>    double confidence;<br>    int class_id;<br>};<br><br><br>Ort::Session LoadYoloModel(const Ort::Env &amp;env, const std::string &amp;model_path,<br>                           OrtCUDAProviderOptionsV2 *cuda_provider) {<br>    Ort::SessionOptions yolo_session_options;<br>    Ort::ThrowOnError(Ort::GetApi().CreateCUDAProviderOptions(&amp;cuda_provider));<br>    yolo_session_options.AppendExecutionProvider_CUDA_V2(*cuda_provider);<br>    Ort::Session yolo_model_session(env, model_path.c_str(), yolo_session_options);<br>    return yolo_model_session;<br>}<br><br>cv::Mat ImageToBlob(const cv::Mat &amp;image) {<br>    cv::Mat resized_image;<br>    LetterBox(image, resized_image, cv::Size(640, 640));<br>    cv::imwrite(&quot;LetterBoxed.jpg&quot;, resized_image);<br>    return cv::dnn::blobFromImage(<br>        resized_image,<br>        1.0 / 255.0,<br>        cv::Size(640, 640),<br>        cv::Scalar::all(0),<br>        true,<br>        false,<br>        CV_32F);<br>}<br><br>Ort::Value BlobToONNXTensor(const cv::Mat &amp;blob) {<br>    std::vector&lt;int64_t&gt; tensor_shape = {1, 3, 640, 640};<br>    int64_t blob_size = 3 * 640 * 640;<br>    Ort::MemoryInfo memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);<br>    return Ort::Value::CreateTensor&lt;float&gt;(<br>        memory_info,<br>        (float *) (blob.data), // C++ style casting will break this.<br>        blob_size,<br>        tensor_shape.data(),<br>        tensor_shape.size()<br>    );<br>}<br><br>cv::Mat GetYoloBoxes(std::vector&lt;Ort::Value&gt; &amp;output) {<br>    std::vector&lt;int64_t&gt; output_shape = output[0].GetTensorTypeAndShapeInfo().GetShape();<br>    // [batch size, mask weights, boxes predicted]=&gt;[bs, preds_num, features]<br>    return cv::Mat(<br>        cv::Size(static_cast&lt;int&gt;(output_shape[2]),<br>                 static_cast&lt;int&gt;(output_shape[1])),<br>        CV_32F,<br>        output[0].GetTensorMutableData&lt;float&gt;()).t();<br>}<br><br>void ClipBox(cv::Rect &amp;box, const cv::Size &amp;shape) {<br>    box.x = std::max(0, std::min(box.x, shape.width));<br>    box.y = std::max(0, std::min(box.y, shape.height));<br>    box.width = std::max(0, std::min(box.width, shape.width - box.x));<br>    box.height = std::max(0, std::min(box.height, shape.height - box.y));<br>}<br><br>void ScaleYoloBoundingBox(YoloBoundingBox &amp;box, const cv::Size &amp;original_shape) {<br>    cv::Size yolo_shape(640, 640);<br>    float scale_ratio = std::min(static_cast&lt;float&gt;(yolo_shape.width) / static_cast&lt;float&gt;(original_shape.width),<br>                                 static_cast&lt;float&gt;(yolo_shape.height) / static_cast&lt;float&gt;(original_shape.height));<br>    float letterbox_pad_horizontal = (yolo_shape.width - original_shape.width * scale_ratio) / 2;<br>    float letterbox_pad_vertical = (yolo_shape.height - original_shape.height * scale_ratio) / 2;<br>    box.bounding_box.x -= letterbox_pad_horizontal;<br>    box.bounding_box.y -= letterbox_pad_vertical;<br>    box.bounding_box.x /= scale_ratio;<br>    box.bounding_box.y /= scale_ratio;<br>    box.bounding_box.width /= scale_ratio;<br>    box.bounding_box.height /= scale_ratio;<br>    ClipBox(box.bounding_box, original_shape);<br>}<br><br>std::vector&lt;YoloBoundingBox&gt; ProcessYoloOutput(const cv::Mat &amp;raw_boxes, const cv::Size &amp;original_shape) {<br>    int num_classes = 80;<br>    int data_width = num_classes + 4;<br>    auto *bounding_box_data = reinterpret_cast&lt;float *&gt;(raw_boxes.data);<br>    std::vector&lt;YoloBoundingBox&gt; processed_boxes;<br>    for (int row = 0; row &lt; raw_boxes.rows; row++) {<br>        // Step 1: Parse x, y, Width, Height, max class score<br>        float width = bounding_box_data[2];<br>        float height = bounding_box_data[3];<br>        float x = MAX((bounding_box_data[0] - 0.5 * width), 0);<br>        float y = MAX((bounding_box_data[1] - 0.5 * height), 0);<br>        cv::Rect bounding_box(x, y, width, height);<br>        // Save all class scores 1 - 80 starting at index 4<br>        cv::Mat class_scores = cv::Mat(1, num_classes, CV_32FC1, bounding_box_data + 4);<br>        cv::Point class_id_tmp;<br>        double confidence;<br>        // Find max score<br>        cv::minMaxLoc(class_scores, nullptr, &amp;confidence, nullptr, &amp;class_id_tmp);<br>        // Step 2: Filter boxes by confidence<br>        if (confidence &gt; 0.25) {<br>            int class_id = class_id_tmp.x;<br>            YoloBoundingBox parsed_box(bounding_box, confidence, class_id);<br>            processed_boxes.push_back(parsed_box);<br>        }<br>        // Advance data pointer to next row<br>        bounding_box_data += data_width;<br>    }<br>    // Step 3: Scale boxes<br>    for (auto &amp;box: processed_boxes) {<br>        ScaleYoloBoundingBox(box, original_shape);<br>    }<br>    // Step 4: Non-Max Suppression<br>    std::vector&lt;cv::Rect&gt; boxes;<br>    std::vector&lt;float&gt; confidences;<br>    std::vector&lt;int&gt; filtered_idxs;<br>    for (auto &amp;processed_box: processed_boxes) {<br>        boxes.push_back(processed_box.bounding_box);<br>        confidences.push_back(processed_box.confidence);<br>    }<br>    cv::dnn::NMSBoxes(boxes, confidences, 0.25, 0.70, filtered_idxs);<br>    std::vector&lt;YoloBoundingBox&gt; filtered_boxes;<br>    for (auto &amp;idx: filtered_idxs) {<br>        filtered_boxes.push_back(processed_boxes[idx]);<br>    }<br>    return filtered_boxes;<br>}<br><br>int main() {<br>    const Ort::Env env(ORT_LOGGING_LEVEL_ERROR, &quot;Yolo Tutorial&quot;);<br>    OrtCUDAProviderOptionsV2 *cuda_provider = nullptr;<br>    Ort::Session yolo_model_session = LoadYoloModel(<br>        env, &quot;/home/alex/Projects/MediumBlog/YoloOnnxRuntimeCPP/weights/yolo11s-coco.onnx&quot;, cuda_provider);<br>    cv::Mat image =<br>            cv::imread(&quot;/home/alex/Projects/MediumBlog/YoloOnnxRuntimeCPP/test_images/ExampleImageUnsplash.jpg&quot;);<br>    cv::Mat blob = ImageToBlob(image);<br>    Ort::Value input_tensor = BlobToONNXTensor(blob);<br>    Ort::AllocatorWithDefaultOptions allocator;<br>    const std::string input_name = yolo_model_session.GetInputNameAllocated(0, allocator).get();<br>    const std::string output_name = yolo_model_session.GetOutputNameAllocated(0, allocator).get();<br>    const char *input_names[] = {input_name.c_str()};<br>    const char *output_names[] = {output_name.c_str()};<br>    std::vector&lt;Ort::Value&gt; output = yolo_model_session.Run(<br>        Ort::RunOptions{nullptr},<br>        input_names,<br>        &amp;input_tensor,<br>        1,<br>        output_names,<br>        1<br>    );<br>    cv::Mat raw_boxes = GetYoloBoxes(output);<br>    std::vector&lt;YoloBoundingBox&gt; boxes = ProcessYoloOutput(raw_boxes, image.size());<br>    for (auto &amp;box: boxes) {<br>        cv::rectangle(image, box.bounding_box, cv::Scalar(0, 255, 0), 2);<br>    }<br>    cv::imshow(&quot;Test Image&quot;, image);<br>    cv::imwrite(&quot;output.jpg&quot;, image);<br>    cv::waitKey(0);<br>    Ort::GetApi().ReleaseCUDAProviderOptions(cuda_provider);<br>    return 0;<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*QOOs6kXCz9BX8VSq4NBhtg.jpeg" /><figcaption>The above program run on an image from COCO 2017</figcaption></figure><p>The <a href="https://github.com/ataffe/YoloOnnxRuntimeCPP">GitHub</a> repo has code for parsing the COCO labels and drawing the label and confidence at the top left of each bounding box, like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*ChUoUQf1m4-mcQIlYCYLuQ.jpeg" /></figure><p>This code can also easily be extended to run object detection on a video in C++.</p><h3>Thanks for reading :)</h3><p>I am a recent graduate from UC Davis, currently seeking a job. If you or someone you know is seeking a computer vision engineer or a similar role, I would greatly appreciate it if you let me know on <a href="https://www.linkedin.com/in/alexander-taffe">LinkedIn</a> or at taffealexander@gmail.com.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8bfb964520b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>