<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sandeep on Medium]]></title>
        <description><![CDATA[Stories by Sandeep on Medium]]></description>
        <link>https://medium.com/@me_sameen?source=rss-a3cc8841f580------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*z8Z0FFCw3HjKSgbp.</url>
            <title>Stories by Sandeep on Medium</title>
            <link>https://medium.com/@me_sameen?source=rss-a3cc8841f580------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 19:59:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@me_sameen/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Build Your Own Fleet Video Platform: MediaMTX, FFmpeg and HLS for MDVR, MNVR Cameras]]></title>
            <link>https://medium.com/@me_sameen/build-your-own-fleet-video-platform-mediamtx-ffmpeg-and-hls-for-mdvr-mnvr-cameras-ec44fb038136?source=rss-a3cc8841f580------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec44fb038136</guid>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[nextjs]]></category>
            <dc:creator><![CDATA[Sandeep]]></dc:creator>
            <pubDate>Thu, 12 Nov 2026 21:45:18 GMT</pubDate>
            <atom:updated>2026-11-12T21:45:18.289Z</atom:updated>
            <content:encoded><![CDATA[<p>Nov 13, 2026 · @Sandeep</p><h3>You don’t need a paid streaming platform for fleet video</h3><p>Every bus, truck and van with an MDVR or NVR is a moving CCTV system. Operators want to click a vehicle on the map and see its cameras, right now. Fleet owners want to replay what happened at 08:12 yesterday. Safety teams want the video behind every harsh-braking or driver-fatigue alert.</p><p>The usual answer is a paid video platform, priced per camera, per hour or per gigabyte, and tied to one vendor’s devices. We took a different road. We built live and historical video for two very different device families, Howen MDVRs and WIOT NVRs, on open-source parts: <strong>MediaMTX</strong>, <strong>FFmpeg</strong>, <strong>HLS</strong>, a little Go, a little Rust, and Kubernetes.</p><p>It works. A camera starts streaming in a few seconds, any number of people can watch it in a normal browser, and the device stops sending video by itself when nobody is watching anymore. The licence cost is zero.</p><p>This article is the full story: the architecture, every component in a stream’s life, and, more usefully, every wall we hit on the way and how we got past it. If you’re integrating cameras from any vendor into a tracking platform, you can reuse almost all of it.</p><p><strong>What you’ll get from this post:</strong></p><ul><li>An architecture where live tracking, history, ADAS alerts and video share one platform</li><li>How two device types with completely different protocols end up as the same HLS stream</li><li>How MediaMTX works inside, explained actor by actor</li><li>Seven real problems we hit, each with its cause, the fix, and the code or config</li><li>Why video takes several seconds to start, explained so anyone on your team can follow it</li><li>A design that stops idle cameras automatically and heals itself when events are missed</li></ul><h3>The big picture: one platform, two lanes</h3><p>The platform has two lanes that share one device identity and one API. The <strong>telemetry lane</strong> carries positions, alerts and commands through NATS into TimescaleDB. The <strong>video lane</strong> carries RTMP into MediaMTX and HLS out to browsers. Every device type, whatever its protocol, enters the video lane the same way: as one RTMP publisher on live/&lt;stream_id&gt;.</p><p>[embedded content: platform architecture · 3 device types, 2 lanes: telemetry and video]</p><p>Reading it top to bottom:</p><ul><li><strong>Devices</strong> speak their vendor’s language: Howen’s binary TCP protocol, WIOT’s MQTT, and older WIOT units sending H.263 video.</li><li><strong>Edge adapters</strong> translate. Each one is small and vendor-specific, and it’s the only vendor-specific code in the video path.</li><li><strong>NATS JetStream</strong> carries telemetry in and commands out. Play and stop are just commands.</li><li><strong>MediaMTX</strong> is the media switchboard: RTMP in, HLS out. Before letting anyone publish or watch, it asks our API, and it reports what happens through hooks.</li><li><strong>APISIX</strong> puts HLS behind TLS on the app’s own domain.</li><li><strong>The web API</strong> owns every decision: who may stream, which session to reuse, when to stop. A cleanup job inside it reconciles the database with what MediaMTX actually sees.</li></ul><p>The rest of this article follows that diagram from the devices downward, then walks through the problems we hit in the order we hit them.</p><h3>Meet the devices: two vendors, two very different worlds</h3><p>The whole point of this design is that the video side doesn’t care which vendor built the camera. Here’s what the two device families actually look like on the wire, and how each one is turned into the same thing: an RTMP stream pushed into MediaMTX.</p><h3>Howen MDVR: a proprietary TCP protocol and raw H.264</h3><p>Howen MDVRs speak their own binary protocol over TCP. The device keeps a signalling connection to our server for telemetry, alarms and commands. When we ask for video, it opens a <strong>separate media connection</strong>, registers it against a session ID, and starts sending raw H.264 frames, each one tagged as an I-frame or a P-frame, with a device timestamp.</p><p>We wrote the Howen side in Rust: a parser for the Howen protocol and a TCP listener that handles both connections. For video, the listener does four things:</p><ol><li>Accepts the media link and matches it to the stream ID we sent in the play command.</li><li>Waits for the first I-frame, because nothing can be decoded before one arrives.</li><li>Spawns FFmpeg and pipes the raw H.264 frames into it.</li><li>FFmpeg wraps the frames into FLV and publishes them to rtmp://mediamtx:1935/live/&lt;stream_id&gt;.</li></ol><p>FFmpeg here doesn’t re-encode anything. It copies the video bits as they are (-c:v copy) and only changes the packaging. That keeps CPU use tiny, even with many cameras.</p><h3>WIOT NVR: MQTT commands and a direct RTMP push</h3><p>WIOT NVRs, used on school buses in our case, are simpler on the video side and different on the control side. They talk to the platform over <strong>MQTT</strong>. A small wiot-gateway service consumes their MQTT topics, forwards telemetry into our pipeline, and translates our platform commands into MQTT messages.</p><p>When a WIOT NVR receives a live-start command containing an RTMP URL, it <strong>pushes RTMP itself</strong>. No gateway sits in the video path, so the device publishes straight into MediaMTX.</p><h3>The legacy twist: H.263 and an on-demand transcode</h3><p>Older WIOT units send video encoded as <strong>H.263</strong>, a codec from the 1990s that MediaMTX doesn’t accept. Those devices keep pushing to an existing nginx-rtmp server. When someone opens one of those cameras, MediaMTX starts FFmpeg <strong>on demand</strong>: it pulls from nginx-rtmp, transcodes to H.264, and publishes back into MediaMTX on a legacy/&lt;name&gt; path. When the last viewer leaves, MediaMTX kills FFmpeg ten seconds later. This one path does real transcoding, so it&#39;s the only one that costs meaningful CPU.</p><h3>Side by side</h3><p>Howen MDVRWIOT NVRLegacy WIOT NVRControl channelProprietary TCPMQTTMQTTVideo from deviceRaw H.264 frames over TCPRTMP pushRTMP push to nginx-rtmpOur component in the video pathRust listener + FFmpeg (copy)NoneFFmpeg pull + transcode (on demand)What reaches MediaMTXRTMP on live/&lt;id&gt;RTMP on live/&lt;id&gt;RTSP on legacy/&lt;name&gt;CPU cost on our sideVery lowNoneModerate, only while watched</p><p>After this point every stream looks identical: a path in MediaMTX with one publisher. Everything that follows (HLS, auth, hooks, idle stop) is written once and works for every vendor. Adding a third vendor means writing one more adapter like the Howen listener, nothing else.</p><h3>Live tracking, history and ADAS alerts: the pipeline video plugs into</h3><p>Video was not our first feature. The platform already tracked vehicles live, stored their full history and raised alerts. The key design decision was to make video <strong>just another device command</strong> on the same rails, not a separate system.</p><h3>Telemetry in</h3><p>Every device gateway (the Howen Rust listener, the WIOT MQTT gateway, gateways for other vendors) does the same job: decode the vendor’s format and publish a normalised message to <strong>NATS JetStream</strong>, on a subject per device such as tcp.c2c.{imei}.telemetry.</p><p>A processor service consumes those subjects and writes to <strong>PostgreSQL with TimescaleDB</strong>:</p><ul><li><strong>Telematics data:</strong> position, speed, heading, ignition, sensors. A hypertable holds the full history; a small “recent” table holds each device’s latest state for the live map.</li><li><strong>Alerts:</strong> overspeed, geofence, panic button, and the <strong>ADAS/DSM events</strong> from camera-equipped devices (forward collision, lane departure, driver fatigue, phone use, smoking). Again a history table plus a recent table.</li><li><strong>Connection events:</strong> online, offline, reconnects.</li></ul><p><strong>PostGIS</strong> adds geometry columns, so “which vehicles are inside this geofence” or “cluster these 5,000 points for the map” is a database query instead of application code.</p><h3>Commands out</h3><p>Commands travel the other way on the same bus. The web API publishes a command to iot.device.{imei}.command.request. A device-manager service stores it, tracks retries and expiry, and routes it to the right gateway. The gateway translates it into the vendor&#39;s language (a Howen binary message, an MQTT message for WIOT). Results come back on iot.device.{imei}.command.update.</p><p>Video uses exactly this path. <strong>“Start live video on channel 1” is a command like “cut the engine” or “set the speed limit”.</strong> Its payload carries the stream ID and the RTMP URL to publish to:</p><pre>{<br>  &quot;stream_id&quot;: &quot;5770361f361640c0bc24362b89650409&quot;,<br>  &quot;stream_url&quot;: &quot;rtmp://stream.example.com:1935/live/5770361f361640c0bc24362b89650409&quot;,<br>  &quot;channel_id&quot;: &quot;1&quot;,<br>  &quot;action&quot;: &quot;live-start&quot;<br>}</pre><p>The same command shape covers live-stop, historical-start with a time range, and historical-stop.</p><h3>Why this matters</h3><p>Because video rides the existing rails, it inherited a lot for free:</p><ul><li><strong>One device identity.</strong> The IMEI that shows on the map is the IMEI you stream from.</li><li><strong>One command history.</strong> Who started which camera, when, and whether the device accepted it, sits next to every other command.</li><li><strong>One place to connect events and video.</strong> An ADAS alert has an IMEI and a timestamp; a video session has an IMEI, a channel and a time range. Joining them is a query (more on this in the historical section).</li><li><strong>Vendor independence.</strong> The web API never speaks Howen or MQTT. It publishes a generic command and the gateway does the rest.</li></ul><h3>Why MediaMTX, and what lives inside it</h3><p><a href="https://github.com/bluenviron/mediamtx">MediaMTX</a> is an open-source media server written in Go, shipped as a single binary. Devices publish in one protocol and viewers read in another; MediaMTX converts between them. For us it does four jobs that would otherwise need a paid platform:</p><ul><li><strong>Ingest</strong> RTMP from devices and FFmpeg (it also takes RTSP, SRT, WebRTC and more).</li><li><strong>Repackage</strong> to HLS for browsers, with low-latency HLS built in.</li><li><strong>Ask our API</strong> before letting anyone publish or watch (HTTP auth callback).</li><li><strong>Tell our API</strong> what happens (hooks), and <strong>let our API inspect and act</strong> (Control API).</li></ul><p>It does <strong>not</strong> decide anything about your business. It doesn’t know users, devices or why a stream exists. That split turned out to be the most important idea in the whole design: <strong>MediaMTX moves media and reports facts; our API makes every decision.</strong></p><h3>The actors, one by one (HLS only)</h3><p>Knowing these by name saves hours of confusion. Most of our early bugs came from mixing two of them up.</p><p>ActorWhat it isWhat it’s responsible for<strong>Path</strong>A named channel, e.g. live/5770…Holds at most one source and any number of readers; becomes <em>available</em> when media flows; fires the available/unavailable hooks<strong>Source (publisher)</strong>The one connection feeding the pathOur RTMP from FFmpeg or the device. MediaMTX <strong>never</strong> disconnects it because nobody is watching<strong>Stream</strong>The media inside MediaMTX, independent of protocolFans the same frames out to every reader, so one device upload serves everyone<strong>HLS server</strong>The HTTP listener on port 8888Receives every playlist and segment request, checks auth, finds or creates the muxer<strong>HLS muxer</strong>The converter from stream to HLS, <strong>one per path</strong>Cuts video into segments and parts, keeps the last few in memory, writes the playlists<strong>Reader</strong>A consumer of the path, as hooks and the API report itIn the version we run, <strong>each browser&#39;s HLS session is its own reader</strong>, even though they share one muxer<strong>Auth</strong>authMethod: httpCalls our endpoint on every publish and every HLS request; 2xx allows, anything else denies<strong>Hooks</strong>runOnAvailable, runOnRead, …Shell commands MediaMTX runs on events; we use wget to turn them into HTTP calls<strong>Control API</strong>HTTP on port 9997, cluster-internalLists paths and their readers, kicks connections</p><h3>Two facts that shaped everything</h3><p><strong>1. HLS viewers never say goodbye.</strong> A browser playing HLS just keeps sending HTTP requests. When the tab closes, the requests stop. MediaMTX can only conclude that a viewer left after a period of silence, about <strong>30 seconds</strong> for an HLS session in our setup.</p><p><strong>2. There are two different timeouts.</strong> The <strong>session</strong> timeout (around 30s, internal) decides when one viewer counts as gone. hlsMuxerCloseAfter (configurable) decides when the shared muxer is torn down after the <em>last</em> request. We mixed these up for a while, and it cost us a day.</p><h3>The life of one live stream, end to end</h3><p>Here is a real Howen session, reconstructed from the logs of our web API and our Rust media listener. Times are seconds after the user clicked Play.</p><ol><li><strong>0.00s: Play.</strong> The browser calls POST /media/start with the IMEI and channel. The API looks for a live stream of that camera that is already pending or active. None exists, so it creates a row with a fresh 32-character stream ID, status pending, and publishes a live-start command on NATS. It returns the HLS URL immediately; the request takes about 10 ms.</li><li><strong>0.00s onward: the player waits.</strong> The browser starts requesting index.m3u8. MediaMTX answers 404 (nothing is publishing yet), and our player retries every half second behind a &quot;Connecting to camera…&quot; overlay. Every request also goes through the auth callback.</li><li><strong>0.35s: the device answers.</strong> The Howen MDVR opens its media connection and registers it against the stream ID.</li><li><strong>3.58s: the first frame.</strong> The device waits for its encoder’s next keyframe, then sends it. This wait varies between runs; it’s the first cost of a long keyframe interval (see Hustle 7).</li><li><strong>4.94s: FFmpeg has probed the input.</strong> The Rust listener spawned FFmpeg at the first keyframe; FFmpeg spent 1.4 seconds analysing the raw stream before writing anything.</li><li><strong>5.15s: FFmpeg publishes.</strong> RTMP output opens to MediaMTX. The auth callback is asked about this publish, and our API says yes.</li><li><strong>5.73s: available.</strong> MediaMTX marks the path available and runs runOnAvailable. Our API sets the stream active.</li><li><strong>5.89s: the muxer is born.</strong> The next HLS request creates the HLS muxer. It now waits for a keyframe to start its first segment, and the following keyframe to finish it.</li><li><strong>9.12s: first playlist.</strong> The first segment is sealed and the playlist is served. The player starts. Later viewers join the warm muxer and start in <strong>under a second</strong>.</li><li><strong>During playback:</strong> each browser polls the playlist and fetches small parts about once a second, through browser → APISIX (HTTPS) → MediaMTX.</li><li><strong>Viewers leave:</strong> requests stop; each HLS session expires after about 30 seconds.</li><li><strong>About two minutes of nobody watching:</strong> our cleanup job notices zero readers for long enough, marks the stream stopping and sends live-stop.</li><li><strong>Device disconnects:</strong> the Howen listener closes FFmpeg (exit status 0), MediaMTX runs runOnUnavailable, and the API marks the stream ended with reason idle.</li></ol><p>Nine seconds to first frame looks slow, and it was. By the end of this article you’ll see exactly where each of those seconds went, and how to get most of them back.</p><h3>Hustle 1: anyone could publish, anyone could watch</h3><p>Our first working setup felt finished. The API generated a UUID per play request, sent it to the device, and returned the HLS URL to the browser. Then we asked a simple question: <strong>what stops someone else from using this?</strong></p><p>Nothing. The RTMP port must be reachable from the internet, because the buses connect over cellular. HLS must be reachable, because browsers fetch it directly. Our API controlled <em>when a device was told to publish</em>, but not <em>who could connect to MediaMTX</em>. Those are two different doors:</p><ul><li><strong>The control door</strong> (play and stop) goes through our authenticated API. Safe.</li><li><strong>The media door</strong> (RTMP in, HLS out) is MediaMTX itself. Wide open.</li></ul><p>Anyone who found port 1935 could publish to live/&lt;anything&gt;, including a real device&#39;s path, and inject their own video. Anyone with an HLS URL (browser history, a shared link, DevTools) could watch a school bus camera. For cameras that show children, that was not acceptable.</p><h3>The fix: let MediaMTX ask your API</h3><p>MediaMTX can call an HTTP endpoint before accepting any connection:</p><pre>authMethod: http<br>authHTTPAddress: http://web-api.platform.svc.cluster.local/internal/mtx/auth<br>authHTTPExclude:<br>  - action: api<br>  - action: metrics<br>  - action: pprof</pre><p>For every publish and every HLS request, MediaMTX POSTs JSON describing who wants what:</p><pre>{<br>  &quot;action&quot;: &quot;publish&quot;,<br>  &quot;path&quot;: &quot;live/5770361f361640c0bc24362b89650409&quot;,<br>  &quot;protocol&quot;: &quot;rtmp&quot;,<br>  &quot;ip&quot;: &quot;203.0.113.10&quot;,<br>  &quot;query&quot;: &quot;token=abc123&quot;,<br>  &quot;user&quot;: &quot;&quot;, &quot;password&quot;: &quot;&quot;, &quot;token&quot;: &quot;&quot;, &quot;id&quot;: &quot;…&quot;<br>}</pre><p>A 2xx response allows the connection; anything else rejects it. With that one endpoint, our API can check that the stream ID exists, that it’s still pending or active, that the publish token matches, and that a viewer holds a valid token.</p><h3>Observe first, enforce later</h3><p>We didn’t want a half-built rule set to break live demos, so the handler has a switch:</p><pre>func (h *Handler) handleMtxAuth(c *gin.Context) {<br>	var req mtxAuthReq<br>	if err := c.ShouldBindJSON(&amp;req); err != nil {<br>		c.Status(http.StatusOK) // observe mode: never block on our own errors<br>		return<br>	}<br>	known := h.ctrl.IsKnownMtxRequest(c.Request.Context(), &amp;req)<br>	h.log.Infof(&quot;mtx %s %s proto=%s ip=%s known=%v&quot;, req.Action, req.Path, req.Protocol, req.IP, known)</pre><pre>	if config.Media.AuthEnforce &amp;&amp; !known {<br>		c.Status(http.StatusUnauthorized)<br>		return<br>	}<br>	c.Status(http.StatusOK)<br>}</pre><p>In <strong>observe mode</strong> everything is allowed and logged, so you learn what real traffic looks like. Flip one flag and the same rules start rejecting.</p><h3>Things to know before you rely on it</h3><ul><li><strong>It fails closed.</strong> If your API is down or slow, <em>new</em> publishes and views are rejected. Streams already running continue. Keep the endpoint fast: HLS makes a request roughly every second per viewer, so you’ll see a lot of read calls.</li><li><strong>Keep the endpoint internal.</strong> MediaMTX calls it over the cluster network. Our public gateway only routes /iotapi/*, so /internal/mtx/* is unreachable from outside.</li><li><strong>Never expose the Control API.</strong> authHTTPExclude: api means API calls skip auth. That&#39;s fine only because port 9997 is a ClusterIP service with no public route.</li><li><strong>Put a token in the publish URL now</strong>, even in observe mode (…/live/&lt;id&gt;?token=&lt;secret&gt;). Switching to enforce later then needs no change on any device.</li></ul><h3>Hustles 2 and 3: “localhost” lies, and HTTPS pages won’t play HTTP video</h3><h3>Hustle 2: connection refused to our own API</h3><p>Locally, MediaMTX ran in Docker and our API ran directly on the laptop. The first auth call failed instantly:</p><pre>closed: failed to authenticate: Post &quot;http://localhost:8090/internal/mtx/auth&quot;:<br>dial tcp [::1]:8090: connect: connection refused</pre><p>Inside a container, localhost is <strong>the container itself</strong>, not your machine. On Docker Desktop the host is reachable as host.docker.internal (on plain Linux, add extra_hosts: [&quot;host.docker.internal:host-gateway&quot;]). One more trap: if your API listens on 127.0.0.1:8090, the container still can&#39;t reach it. Bind to :8090.</p><p>The legacy FFmpeg command is the exception that proves the rule. It runs <em>inside</em> the MediaMTX container and publishes to MediaMTX itself, so rtsp://localhost:$RTSP_PORT/... is correct there.</p><h3>Hustle 3: perfect locally, silent in the cloud</h3><p>In the cloud environment the player stayed black. Stranger still, opening the same stream through MediaMTX’s built-in test page (/live/&lt;id&gt;/ without index.m3u8) <strong>played fine</strong>. So the server, auth and stream were all healthy. The difference was the page doing the loading.</p><p>The browser console told the truth:</p><pre>Mixed Content: The page at &#39;https://app.example.com/iot/media&#39; was loaded over HTTPS,<br>but requested an insecure XMLHttpRequest endpoint<br>&#39;http://stream.example.com:8888/live/…/index.m3u8&#39;. This request has been blocked.</pre><p>Our web app is HTTPS; MediaMTX’s HLS port serves plain HTTP. Browsers block that mix. Locally it worked because the app ran on <a href="http://localhost./">http://localhost.</a></p><p>The first instinct, adding https:// in front of :8888, produced ERR_SSL_PROTOCOL_ERROR. Port 8888 speaks plain HTTP; a different prefix doesn&#39;t make it speak TLS.</p><p><strong>The fix: put HLS behind the gateway that already does TLS.</strong> We run APISIX as our ingress, so one route sends HLS paths to MediaMTX:</p><pre>apiVersion: apisix.apache.org/v2<br>kind: ApisixRoute<br>metadata:<br>  name: mediamtx-hls<br>spec:<br>  ingressClassName: apisix<br>  http:<br>    - name: hls<br>      match:<br>        hosts: [app.example.com]<br>        paths: [/stream/live/*, /stream/legacy/*]<br>        methods: [GET, OPTIONS]<br>      plugins:<br>        - name: proxy-rewrite<br>          enable: true<br>          config:<br>            regex_uri: [&quot;^/stream/(.*)&quot;, &quot;/$1&quot;]<br>      backends:<br>        - serviceName: mediamtx<br>          servicePort: 8888</pre><p>The browser now talks HTTPS to APISIX on 443; APISIX forwards to MediaMTX on 8888 inside the cluster. Serving HLS on the <strong>same domain</strong> as the app also removes any CORS question. We chose an explicit ApisixRoute over our Helm chart&#39;s generic route block, because the chart routed to the service&#39;s main port, which on MediaMTX is RTMP 1935, not HLS.</p><h3>Two aftershocks</h3><p><strong>The config changed, the pod didn’t.</strong> We switched the player base URL to HTTPS in the API’s config file, but the pod never restarted: Helm only rolls pods when the pod spec changes. Bumping the image tag or adding a checksum/config annotation to the Deployment template fixes this for good.</p><p><strong>The database remembered the old URL.</strong> Even after the restart, the player still got http://…:8888. The stream reuse logic returned an existing row, and that row stored the full URL. Lesson: <strong>store the path (</strong><strong>live/&lt;id&gt;), not the URL.</strong> Build the URL from config at response time, and configuration changes apply instantly.</p><p><strong>A side effect worth knowing:</strong> behind a proxy, MediaMTX sees the proxy’s IP, not the viewer’s. If you want real viewer IPs in auth logs, configure hlsTrustedProxies for your pod network.</p><h3>Hustle 4: the player that never started</h3><p>Sometimes the camera took a few seconds longer than usual to publish, and the video simply never appeared. Refreshing the page fixed it, which is the worst kind of bug: users learn to refresh.</p><p>Three separate causes stacked up:</p><ol><li><strong>MediaMTX returns 404 until something publishes.</strong> Our player requested index.m3u8 the moment the API returned the URL, several seconds before the device was live.</li><li><strong>hls.js gave up.</strong> After its built-in retries ran out (about 15–20 seconds with our settings), it raised a fatal error. With no error handler, the player was dead.</li><li><strong>Nothing called </strong><strong>play().</strong> Even when the playlist finally loaded, the video element sat paused.</li></ol><p>There was also a React mistake: if (!src) return; sat <em>before</em> useRef and useEffect. Hooks must run in the same order on every render, so that early return breaks the rules of hooks.</p><h3>The fix: own the retry loop</h3><pre>useEffect(() =&gt; {<br>  const video = videoRef.current;<br>  if (!src || !video) return;</pre><pre>  let hls: Hls | null = null;<br>  let timer: ReturnType&lt;typeof setTimeout&gt; | undefined;<br>  let cancelled = false;<br>  const deadline = Date.now() + 60_000;            // wait up to 60s for the camera</pre><pre>  const retry = () =&gt; {<br>    if (cancelled || Date.now() &gt; deadline) return;<br>    setWaiting(true);<br>    timer = setTimeout(start, 500);                // poll every 0.5s<br>  };</pre><pre>  const start = () =&gt; {<br>    if (cancelled) return;<br>    hls?.destroy();<br>    if (Hls.isSupported()) {<br>      hls = new Hls({ lowLatencyMode: true, maxLiveSyncPlaybackRate: 1.5 });<br>      hls.on(Hls.Events.MANIFEST_PARSED, () =&gt; video.play().catch(() =&gt; {}));<br>      hls.on(Hls.Events.ERROR, (_, d) =&gt; {<br>        if (!d.fatal) return;<br>        if (d.type === Hls.ErrorTypes.MEDIA_ERROR) hls?.recoverMediaError();<br>        else retry();                              // 404 before publish, timeouts, drops<br>      });<br>      hls.loadSource(src);<br>      hls.attachMedia(video);<br>    } else if (video.canPlayType(&quot;application/vnd.apple.mpegurl&quot;)) {<br>      video.src = src;                             // Safari / iOS native HLS<br>      video.play().catch(() =&gt; {});<br>    }<br>  };</pre><pre>  const onPlaying = () =&gt; setWaiting(false);<br>  video.addEventListener(&quot;playing&quot;, onPlaying);<br>  start();<br>  return () =&gt; {<br>    cancelled = true;<br>    clearTimeout(timer);<br>    hls?.destroy();<br>    video.removeEventListener(&quot;playing&quot;, onPlaying);<br>  };<br>}, [src]);</pre><p>The video element carries muted autoPlay playsInline. <strong>muted is not optional.</strong> The stream starts several seconds after the user&#39;s click, and by then the browser no longer counts it as a user gesture, so unmuted autoplay is blocked. Fleet cameras rarely carry useful audio anyway.</p><p>The same error handler also recovers a stream that drops mid-play: the overlay returns to “Connecting to camera…” and playback resumes on its own.</p><p>While we were in there, a review caught one more bug in the historical form: from.getUTCMilliseconds() returns only the milliseconds part of the time (0–999), not an epoch timestamp. The device was receiving nonsense time ranges. getTime() is the right call.</p><p><strong>A better pattern, next on our list:</strong> instead of polling MediaMTX with 404s, poll your own API for the stream’s status every 500 ms and load hls.js only when the available hook has marked it active. The UI gets clean states (&quot;Requesting&quot;, &quot;Waiting for camera&quot;, &quot;Live&quot;, &quot;Camera unavailable&quot;) and MediaMTX gets far fewer pointless requests.</p><h3>Hustle 5: “Is anyone watching?” is harder than it sounds</h3><p>Here is a fact that surprises people: <strong>MediaMTX never stops a stream that a device pushes, just because nobody is watching.</strong> The bus keeps uploading video over cellular until the device stops or you kick it. Legacy on-demand paths clean themselves up (runOnDemandCloseAfter kills FFmpeg), but device-pushed live/ paths run forever.</p><p>So we had to answer “is anyone still watching?” ourselves, then send live-stop. This hustle took the most iterations, and each one taught us something about how HLS really works.</p><h3>Attempt 1: stop on the first unread</h3><p>MediaMTX has reader hooks:</p><pre>paths:<br>  &quot;~^live/([a-f0-9]{32})$&quot;:<br>    runOnRead: &gt;-<br>      wget -q -T 5 -O- --post-data=<br>      &quot;http://web-api/internal/mtx/hooks/read?stream_id=$G1&amp;reader_type=$MTX_READER_TYPE&amp;reader_id=$MTX_READER_ID&quot;<br>    runOnUnread: &gt;-<br>      wget -q -T 5 -O- --post-data=<br>      &quot;http://web-api/internal/mtx/hooks/unread?stream_id=$G1&amp;reader_type=$MTX_READER_TYPE&amp;reader_id=$MTX_READER_ID&quot;</pre><p>The plan: on unread, send live-stop. We assumed all HLS viewers shared one muxer and would therefore look like a single reader.</p><p>The logs said otherwise. Two people watching produced <strong>two</strong> read events, each with a different reader_type=hlsSession ID. When one person left, unread fired, we sent live-stop, and <strong>the other person&#39;s video died</strong>.</p><p>The lesson: <strong>unread means &quot;one reader left&quot;, not &quot;everyone left&quot;.</strong> In the version we run, every browser&#39;s HLS session is its own reader, even though all of them share one muxer underneath.</p><h3>The tempting setting that breaks everything</h3><p>Along the way we tried hlsAlwaysRemux: yes, which keeps the muxer running from the moment a stream is available, for a faster first playback. It also makes the muxer itself a permanent reader. read then fires on publish with zero viewers, and unread only when the stream ends, so the hooks can no longer tell you anything about viewers. We set it back to no, the default.</p><h3>Attempt 2: the pre-warm that killed streams</h3><p>To get the speed benefit without hlsAlwaysRemux, we &quot;pre-warmed&quot; the muxer: when a stream became available, the API made one GET index.m3u8 itself, so the muxer would be ready before real viewers arrived.</p><p>It created a new problem. The pre-warm request is a reader too. It made exactly one request, went silent, and 30 seconds later its session expired. unread fired, our handler sent live-stop, and the real viewer was cut off again. A log line from that day, cleaned up:</p><pre>17:34:02.906  read    reader=ce4f…  (pre-warm request)<br>17:34:02.952  read    reader=25b5…  (real browser)<br>17:34:32.905  unread  reader=ce4f…  → &quot;No one is watching&quot;  ← wrong, 25b5 still watching</pre><p>(It also turned out the pre-warm saved no time. More on that in Hustle 7.)</p><h3>The rule that fixed it</h3><blockquote><strong><em>No single hook decides a stop. A hook is a trigger to check; MediaMTX’s current state decides.</em></strong></blockquote><p>On unread, we now ask the Control API who is still reading:</p><pre>// GET /v3/paths/get/live/&lt;id&gt; → { &quot;readers&quot;: [ {&quot;type&quot;: &quot;hlsSession&quot;, &quot;id&quot;: &quot;…&quot;}, … ] }<br>readers, found, err := c.getWatchers(ctx, &quot;live/&quot;+streamID)<br>if err != nil {<br>	return nil // unsure → never stop; a missed stop is cheaper than a cut-off viewer<br>}<br>if !found || len(readers) &gt; 0 {<br>	return nil // path gone, or somebody is still watching<br>}<br>// zero readers → only now consider stopping</pre><p>The leaving session is already gone from the list when the hook runs, so an empty list really means nobody is left.</p><p>One embarrassing footnote: our first version called /v3/path/get/ instead of /v3/paths/get/ (plural). MediaMTX returned 404, and our safety check read that as &quot;can&#39;t tell, don&#39;t stop&quot;. The careful default saved us from our own typo.</p><h3>Hustle 6: from timers to a cleanup job that heals itself</h3><p>Stopping the instant the last viewer leaves felt harsh. A user who refreshes the page, or switches tabs for a minute, would stop the camera and then wait for a slow restart. So we added a grace period: when readers drop to zero, wait two minutes, check again, and only then send live-stop.</p><h3>Version 1: a timer per idle event</h3><p>The obvious implementation is time.AfterFunc(120*time.Second, …) in the unread handler, with a second reader check when it fires. It mostly worked. Then we drew a timeline:</p><pre>t=0     A leaves  → unread → zero readers → timer #1 (fires at t=120)<br>t=60    B joins<br>t=90    B leaves<br>t=120   timer #1 fires → B&#39;s session just expired → zero readers → STOP</pre><p>B left at t=90, but the stream stopped 30 seconds later, not 120. <strong>Old timers measure from the first idle moment, not the last.</strong> We patched that with a generation counter per stream so only the newest timer could act. Then two deeper problems remained:</p><ul><li><strong>Timers live in memory.</strong> Every deploy or crash of the API forgot pending stops, and those buses streamed until someone noticed.</li><li><strong>Hooks are fire-and-forget.</strong> If the API was restarting when unread arrived, that event was simply gone.</li></ul><h3>Version 2: reconcile, don’t react</h3><p>We replaced the timers with a small job that runs every 25 seconds and <strong>compares MediaMTX’s real state with the database</strong>. This is the same reconciliation idea Kubernetes controllers use: don’t trust events, look at the world and fix the difference.</p><pre>every 25s:<br>  1. GET /v3/paths/list                      (one call: every path + its readers)<br>  2. for each stream with status &#39;active&#39;:<br>       path missing                         → device gone, mark &#39;ended&#39;<br>       readers &gt; 0                          → clear idle_since<br>       readers = 0 and idle_since empty     → idle_since = now<br>       readers = 0 and idle_since &gt; 120s    → mark &#39;stopping&#39;, send live-stop<br>  3. &#39;pending&#39; for more than 60s            → mark &#39;failed&#39; (device never published)<br>  4. &#39;stopping&#39; for more than 30s, path still present → kick the publisher</pre><p>The heart of it in Go:</p><pre>func (c *Controller) sweepMediaStreams(ctx context.Context) {<br>	paths, err := c.listMtxPaths(ctx) // path name (&quot;live/&lt;id&gt;&quot;) → reader count<br>	if err != nil {<br>		return // never act on unknown state<br>	}<br>	active, err := c.db.GetMediaStreamsByStatus(ctx, &quot;active&quot;)<br>	if err != nil {<br>		return<br>	}<br>	for _, s := range active {<br>		readers, ok := paths[mediaPath(s)] // &quot;live/&lt;id&gt;&quot; or &quot;legacy/&lt;id&gt;&quot;<br>		switch {<br>		case !ok:<br>			_ = c.db.MarkMediaStreamEnded(ctx, s.StreamID)<br>		case readers &gt; 0:<br>			if s.IdleSinceMs != 0 {<br>				_ = c.db.ClearMediaStreamIdle(ctx, s.StreamID)<br>			}<br>		case s.IdleSinceMs == 0:<br>			_ = c.db.SetMediaStreamIdleSince(ctx, s.StreamID)<br>		case time.Since(time.UnixMilli(s.IdleSinceMs)) &gt; 120*time.Second:<br>			c.stopIdleStream(ctx, s) // status=&#39;stopping&#39; + live-stop command<br>		}<br>	}<br>}</pre><h3>Three bugs a review caught before production</h3><p>Each of these would have ended <strong>every</strong> active stream on every sweep:</p><ul><li><strong>The map was never filled.</strong> An early listMtxPaths returned nil, nil, so every lookup missed and every stream looked &quot;gone&quot;.</li><li><strong>The keys didn’t match.</strong> MediaMTX names paths live/&lt;id&gt;; the code looked up the bare ID.</li><li><strong>No status check.</strong> An error page from the API parses as zero items, which also means “every path is gone”. Check for HTTP 200 before trusting the body.</li></ul><p>And one subtle one: a NULL idle_since read as Unix time 0, which is &quot;idle for 56 years&quot;, so a stream that had just gone live with no viewer yet would be stopped on the first sweep. The IdleSinceMs == 0 case now <em>starts</em> the idle clock instead.</p><h3>What the job replaced</h3><p>ResponsibilityBeforeAfterViewer present → not idleread hookreaders &gt; 0 in the sweepLast viewer gone → start idle clockunread hookreaders = 0 in the sweepStop after the grace periodIn-memory timeridle_since older than 120sStream nobody ever openedPre-warm sessionZero readers from the first sweepMissed events, restartsLostCorrected on the next sweep</p><p>We kept only two hooks, runOnAvailable and runOnUnavailable, because they make status changes instant for the UI. The read/unread hooks and the pre-warm are gone.</p><p><strong>Two settings to align with it:</strong> set hlsMuxerCloseAfter at least as long as the grace period, so anyone who joins during the wait starts instantly from a warm muxer. And if you run more than one API replica, wrap the sweep in a Postgres advisory lock (pg_try_advisory_lock) so only one replica acts at a time.</p><p>The cost of the grace period is small at our bitrate: the device streams roughly 2.5 extra minutes after the last viewer leaves (30s session timeout + 120s grace + up to one sweep), under 1 MB of cellular data.</p><h3>Hustle 7: why does video take nine seconds to start?</h3><p>Once a stream was running, anyone joining it started in under a second. But the <strong>first</strong> viewer of a camera waited 7 to 9 seconds. To find out why, we added timeline logging to both our API and our Rust media listener, and lined the two logs up side by side. Almost every lost second traced back to one camera setting most people never touch.</p><h3>First, how video is actually sent</h3><p>A video is a flipbook: our cameras send about 15 pictures (frames) per second. Sending 15 full photos every second would cost far too much data, so cameras use two kinds of frame:</p><ul><li><strong>I-frame (keyframe):</strong> a complete picture. In our streams, about 5,700–10,900 bytes.</li><li><strong>P-frame:</strong> only <em>what changed</em> since the previous picture, like a note saying “the bus moved a little left”. About 60–200 bytes.</li></ul><p>A P-frame is useless on its own. “Moved a little left” means nothing if you’ve never seen the starting picture. <strong>Anyone who starts watching needs an I-frame first.</strong></p><p>The distance between two I-frames is the <strong>keyframe interval</strong>, or <strong>GOP</strong> (group of pictures). It’s a camera setting.</p><h3>How HLS packs that video into boxes</h3><p>HLS doesn’t send frame by frame. MediaMTX cuts the video into short chunks called <strong>segments</strong>, like packing a film into small boxes, and the browser downloads box after box. Two rules apply:</p><ol><li><strong>Every box must start with an I-frame,</strong> so a viewer can begin at any box.</li><li><strong>A box is sealed only when the next box starts,</strong> which happens at the next I-frame.</li></ol><p>So a box always runs from one keyframe to the next. Before the very first box is sealed, there is nothing to play.</p><h3>What the logs showed</h3><p>We logged every I-frame with its device timestamp. The interval was exactly <strong>3.0 seconds</strong>, every time:</p><pre>[gop] I frame#2 at +5707ms,  GOP = 3009ms by device timestamp<br>[gop] I frame#3 at +8721ms,  GOP = 3001ms by device timestamp<br>[gop] I frame#4 at +11720ms, GOP = 2988ms by device timestamp</pre><p>That 3-second interval cost us <strong>twice</strong>, plus FFmpeg added a cost of its own:</p><p><strong>1. The device waits for a keyframe before sending anything.</strong> After registering the media link, the device only starts on an I-frame. In one run the first frame came after 0.5 seconds; in the next, after 3.2 seconds. It depends where in the 3-second cycle the command lands. That’s why our startup time jumped around between runs.</p><p><strong>2. FFmpeg spent ~1.4 seconds probing.</strong> Raw H.264 arrives without a container, so FFmpeg reads ahead to work out format, resolution and frame rate before writing a single byte.</p><p><strong>3. The muxer missed the first keyframe.</strong> This was the surprise. The muxer is created by the first HLS request, which only succeeds <em>after</em> the stream is available. By then, I-frame #1 had already gone past. The muxer only sees frames that arrive after it exists, so it had to wait for I-frame #2 to <em>start</em> box one, and I-frame #3 to <em>seal</em> it:</p><pre>12.65  FFmpeg sends I-frame #1 → MediaMTX     (no muxer yet)<br>13.22  path available<br>13.39  first HLS request → muxer created      ← I-frame #1 already gone<br>13.56  I-frame #2 → box one opens<br>16.57  I-frame #3 → box one sealed<br>16.61  playlist served, 40 ms later</pre><p>This also explained why our pre-warm trick from Hustle 5 saved nothing. It can only run after available, just like the player, so both waited for the same keyframes.</p><h3>Where the nine seconds went</h3><p>Of the 9.1 seconds from click to first playlist, about <strong>6.6 seconds</strong> came from the 3-second keyframe interval, hitting twice, and another <strong>1.4 seconds</strong> from FFmpeg probing.</p><h3>The fixes, in order of impact</h3><p><strong>1. Set the camera’s keyframe interval to 1 second</strong> (15 frames at 15 fps). This shrinks both GOP costs at once: the device’s wait drops from 0–3s to 0–1s, and the muxer’s start-and-seal wait from ~3.4s to ~1–2s. You change this on the device (its configuration tool or an encoder-settings command), not in MediaMTX or FFmpeg. Fixing it in FFmpeg would mean re-encoding every frame, which costs real CPU per camera.</p><p>There is a trade-off, because full pictures are big. Using our stream’s actual frame sizes:</p><p>Keyframe intervalApprox. bitrateFirst-box wait3s (before)~31 kbps3–6s2s~40 kbps2–4s1s~62 kbps1–2s</p><p>Even 62 kbps is about 28 MB per streaming hour. If cellular cost matters, 2 seconds is a fair middle ground. A shorter interval also brings two side benefits: lower live delay, and faster recovery after a corrupted frame (the smearing lasts only until the next full picture).</p><p><strong>2. Ask the device for a keyframe on demand,</strong> if its protocol offers a “force I-frame” command; many MDVR protocols do. Sent right after the muxer exists, it opens box one immediately instead of waiting for the next natural keyframe.</p><p><strong>3. Tell FFmpeg what it’s getting</strong> instead of letting it guess:</p><pre>ffmpeg -hide_banner -loglevel warning \<br>  -fflags +genpts+nobuffer -flags low_delay \<br>  -probesize 32 -analyzeduration 0 \<br>  -f h264 -framerate 15 -i pipe:0 \<br>  -c:v copy -an -f flv rtmp://mediamtx:1935/live/&lt;stream_id&gt;</pre><p>This also tames the warnings we saw, <em>“Timestamps are unset”</em> and <em>“Non-monotonous DTS”</em>. Raw H.264 carries no timestamps, so FFmpeg invents them, and a burst of early frames sent them backwards.</p><p><strong>4. Later: publish straight from Rust.</strong> Our listener already has every frame and its device timestamp. Publishing RTMP or MPEG-TS directly removes the probe, the process per camera and the invented timestamps. It can also hold the first I-frame and replay it once MediaMTX is ready, so the muxer never misses it.</p><h3>The expected result</h3><p>StepBeforeAfter fixes 1 and 3Command reaches device0.35s0.35sDevice waits for first keyframe0.5–3.2s0–1sFFmpeg probe and output~1.6s~0.2sRTMP publish → available~0.6s~0.5sMuxer start and seal~3.4s~1–2s<strong>Click → first playlist6.9–9.1s~2.5–4s</strong></p><p>And a viewer joining a camera someone is already watching still starts in under a second.</p><h3>Stream identity and state: one stream per session</h3><p>Every live request either joins a session that’s already running or creates a new one with a fresh, random 32-character stream ID. We were tempted to give each camera one permanent stream ID and simply reactivate it, to save rows. We decided against it, for four reasons:</p><ul><li><strong>A permanent ID is a permanent URL.</strong> Anyone who once saw …/live/&lt;id&gt;/index.m3u8 could watch that bus every time it streams again. A fresh ID per session makes old links dead.</li><li><strong>Late events can’t hurt a new session.</strong> A delayed unavailable from yesterday&#39;s session, a stop command&#39;s reply, or a device that ignored its last stop can&#39;t be confused with today&#39;s stream.</li><li><strong>History and audit.</strong> One row per session records who started it, when it went live, how long it ran and why it ended. For cameras on school buses, that trail matters.</li><li><strong>Clean player state.</strong> A new path means a new muxer, playlists and sequence numbers; no browser or proxy holds stale ones.</li></ul><p>The row count isn’t a real concern: even 10,000 sessions a day is a few million small rows a year, and the hot queries only touch open sessions through a partial index. A retention step in the cleanup job can archive old rows if you want.</p><h3>The rules</h3><p>Latest session for this cameraWhat /media/start doespending or active (live)Reuse it and return its URLstopping, ended or failedCreate a new stream ID and send live-startAny (historical request)Always a new session</p><p>Two details made this robust:</p><ul><li><strong>Mark </strong><strong>stopping, not </strong><strong>ended, when you send a stop.</strong> Only the unavailable hook (or the cleanup job seeing the path gone) sets ended. Otherwise a device that ignores the stop keeps streaming while the database says it ended, and the next Play creates a second session alongside it.</li><li><strong>A unique partial index</strong> stops two people clicking Play at the same moment from creating two sessions for one camera. On a unique violation, re-read and return the winner.</li></ul><h3>The lifecycle</h3><p>[embedded content: media stream states · 5 states, 6 transitions]</p><p>Only two things move a stream to ended: the unavailable hook, or the cleanup job finding its path gone. Everything else, including our own stop decision, only gets it as far as stopping.</p><h3>The table</h3><pre>CREATE TABLE iot_media_streams (<br>    id           BIGSERIAL PRIMARY KEY,<br>    imei         TEXT NOT NULL,<br>    channel_id   TEXT NOT NULL,<br>    stream_id    TEXT NOT NULL,             -- 32 hex chars, fresh per session<br>    media_path   TEXT NOT NULL,             -- &#39;live/&lt;id&gt;&#39; or &#39;legacy/&lt;id&gt;&#39;, never the full URL<br>    stream_type  TEXT NOT NULL CHECK (stream_type IN (&#39;live&#39;, &#39;historical&#39;)),<br>    status       TEXT NOT NULL CHECK (status IN (&#39;pending&#39;, &#39;active&#39;, &#39;stopping&#39;, &#39;ended&#39;, &#39;failed&#39;)),<br>    started_at   TIMESTAMPTZ,               -- set by the available hook<br>    ended_at     TIMESTAMPTZ,               -- set by the unavailable hook or the cleanup job<br>    idle_since   TIMESTAMPTZ,               -- maintained by the cleanup job<br>    end_reason   TEXT,                      -- idle, user, max_duration, device_disconnected, completed, failed_to_start<br>    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),<br>    updated_at   TIMESTAMPTZ NOT NULL DEFAULT now()<br>);</pre><pre>CREATE UNIQUE INDEX uq_media_streams_stream_id ON iot_media_streams (stream_id);</pre><pre>-- at most one open live session per camera (race protection + fast reuse lookup)<br>CREATE UNIQUE INDEX uq_media_streams_live_open<br>    ON iot_media_streams (imei, channel_id)<br>    WHERE stream_type = &#39;live&#39; AND status IN (&#39;pending&#39;, &#39;active&#39;, &#39;stopping&#39;);</pre><p>The end_reason column turned out to be the most useful one for operations. A dashboard of reasons per day shows at a glance whether streams end because people stopped watching (healthy), because devices dropped (network trouble) or because they never started (device or command problems).</p><h3>Historical playback and ADAS event video</h3><p>Live video answers “what’s happening now?”. Fleet operators ask two other questions just as often: <em>what happened at 08:12 yesterday?</em> and <em>show me the video behind this alert</em>. Both reuse the pipeline above with almost no new parts.</p><h3>Historical playback: the device is the archive</h3><p>MDVRs and NVRs record continuously to their own SD card or hard disk. Instead of uploading every hour of every camera to the cloud (expensive, and mostly never watched), we ask the device to <strong>play back a time range on request</strong>:</p><pre>{<br>  &quot;stream_id&quot;: &quot;9b1e…&quot;,<br>  &quot;stream_url&quot;: &quot;rtmp://stream.example.com:1935/live/9b1e…&quot;,<br>  &quot;channel_id&quot;: &quot;1&quot;,<br>  &quot;action&quot;: &quot;historical-start&quot;,<br>  &quot;start_time_ms&quot;: 1762412400000,<br>  &quot;end_time_ms&quot;: 1762413000000<br>}</pre><p>From MediaMTX’s point of view, this is just another publisher on a live/&lt;id&gt; path. The same auth, HLS, player and cleanup apply. Three things differ:</p><ul><li><strong>Never reuse a historical session.</strong> Two users asking for different time ranges must get different streams. Every historical request gets a fresh stream ID.</li><li><strong>It ends on its own.</strong> When the device reaches the end of the range it disconnects; runOnUnavailable fires and we record end_reason = completed.</li><li><strong>Times must be real epoch milliseconds.</strong> Remember the getUTCMilliseconds() bug from Hustle 4.</li></ul><h3>ADAS alerts: joining events and video</h3><p>Camera-equipped devices raise ADAS and driver-monitoring events: forward collision warning, lane departure, fatigue, phone use. Those land in the alerts tables with an IMEI, a channel where relevant, and a timestamp. Linking an alert to its video is then a question of time:</p><ol><li><strong>On demand:</strong> the alert detail page can offer “Play video”. The API turns that into a historical-start for, say, 15 seconds before to 15 seconds after the event. No storage on our side; the device&#39;s own recording is the source.</li><li><strong>Automatic clips (next step):</strong> for high-severity events, the platform requests that same window automatically and records it. MediaMTX can record a path to fMP4 segments (record: yes), and the runOnRecordSegmentComplete hook hands us each finished file. We store it in blob storage and index it by IMEI, channel and time, so the clip survives even if the device&#39;s disk is overwritten.</li><li><strong>Live context:</strong> a panic-button alarm can open that vehicle’s live camera on the operator’s screen immediately, next to its position on the map.</li></ol><h3>Live map, history and video on one screen</h3><p>Because all of this shares one device identity, the operator view composes naturally. The map shows the vehicle’s live position from the recent-telematics table. The trip timeline comes from the TimescaleDB history. Alert markers sit on that timeline. Clicking a marker, or any point on the track, plays video from that moment. Each of those is a query or a command against the same IMEI, with no vendor-specific code in the web app.</p><h3>Configuration reference</h3><p>Everything above comes down to a short MediaMTX config and a few Kubernetes objects. Pin an exact MediaMTX version in production: hook names and some API fields have changed between releases (older versions call runOnAvailable runOnReady), so check the sample mediamtx.yml for the version you run.</p><h3>mediamtx.yml</h3><pre># Control API: cluster-internal only, never on a public load balancer<br>api: yes<br>apiAddress: :9997</pre><pre>rtmp: yes<br>rtmpAddress: :1935</pre><pre>hls: yes<br>hlsAddress: :8888<br>hlsVariant: lowLatency     # ~1–3s behind live, works with hls.js and Safari<br>hlsAlwaysRemux: no         # muxer only while someone requests it<br>hlsMuxerCloseAfter: 150s   # keep at least as long as the idle grace period</pre><pre>rtsp: yes                  # used internally by the legacy FFmpeg republish<br>srt: no</pre><pre># Every publish and read is decided by our API<br>authMethod: http<br>authHTTPAddress: <a href="http://web-api.platform.svc.cluster.local/internal/mtx/auth">http://web-api.platform.svc.cluster.local/internal/mtx/auth</a><br>authHTTPExclude:<br>  - action: api<br>  - action: metrics<br>  - action: pprof</pre><pre>paths:<br>  # Devices and the Howen listener publish to live/&lt;32-hex stream id&gt;<br>  &quot;~^live/([a-f0-9]{32})$&quot;:<br>    runOnAvailable: &gt;-<br>      wget -q -T 5 -O- --post-data=<br>      &quot;http://web-api.platform.svc.cluster.local/internal/mtx/hooks/available?stream_id=$G1&amp;source_type=$MTX_SOURCE_TYPE&amp;source_id=$MTX_SOURCE_ID&quot;<br>    runOnUnavailable: &gt;-<br>      wget -q -T 5 -O- --post-data=<br>      &quot;http://web-api.platform.svc.cluster.local/internal/mtx/hooks/unavailable?stream_id=$G1&quot;</pre><pre>  # Legacy H.263 devices: pull from nginx-rtmp and transcode, only while watched<br>  &quot;~^legacy/([A-Za-z0-9_-]+)$&quot;:<br>    runOnDemand: &gt;-<br>      ffmpeg -hide_banner -loglevel warning<br>      -i rtmp://legacy-rtmp.example.com:1935/live/$G1<br>      -c:v libx264 -preset veryfast -tune zerolatency -g 25 -an<br>      -rtsp_transport tcp -f rtsp rtsp://localhost:$RTSP_PORT/$MTX_PATH<br>    runOnDemandRestart: yes<br>    runOnDemandStartTimeout: 30s<br>    runOnDemandCloseAfter: 10s</pre><p>Notes on the choices:</p><ul><li><strong>No catch-all path.</strong> Only live/&lt;32 hex&gt; and legacy/&lt;name&gt; exist; anything else is rejected before auth even runs. The legacy regex is tight on purpose, because $G1 is pasted into an FFmpeg command line.</li><li><strong>wget, not </strong><strong>curl.</strong> Hooks run inside the MediaMTX container. The -ffmpeg image variant is Alpine-based and has BusyBox wget; curl may be missing. -T 5 stops a slow API from piling up hook processes.</li><li><strong>-g 25 on the legacy transcode</strong> sets a 1-second keyframe interval at 25 fps, for the same startup reasons as Hustle 7.</li><li><strong>Use the </strong><strong>-ffmpeg image</strong> (bluenviron/mediamtx:&lt;version&gt;-ffmpeg) because the legacy path runs FFmpeg inside the container.</li></ul><h3>Kubernetes</h3><p>ObjectTypeExposesWhymediamtxService, LoadBalancer1935 (RTMP)Devices publish over the internet. externalTrafficPolicy: Local keeps device source IPs for the auth logsmediamtx-apiService, ClusterIP9997Control API for the web API and cleanup job onlymediamtx-hlsApisixRoute/stream/live/*, /stream/legacy/* → 8888HTTPS for browsers on the app&#39;s own domain (Hustle 3)MediaMTX podDeployment, 1 replicaNone directlyHooks and reader state are per instance; scale with care</p><p>Once HLS works through APISIX, remove port 8888 from the load balancer, so video is reachable only over HTTPS.</p><h3>The web API’s media endpoints</h3><p>EndpointCallerPurposePOST /iotapi/v1/media/startBrowserReuse or create a session, send the start command, return the HLS URLPOST /internal/mtx/authMediaMTXAllow or deny each publish and readPOST /internal/mtx/hooks/availableMediaMTXpending → activePOST /internal/mtx/hooks/unavailableMediaMTX→ endedCleanup job (every 25s)Internal goroutineIdle stop, failed starts, kicks, missed events</p><h3>Lessons learned, and what’s next</h3><p>If you take only a few things from this article, take these:</p><ol><li><strong>Normalise at the edge.</strong> Every vendor adapter ends at the same place: RTMP into MediaMTX on live/&lt;id&gt;. Everything after that is written once.</li><li><strong>Video is a command, not a separate product.</strong> Reusing the existing device identity and command bus gave us history, audit and vendor independence for free.</li><li><strong>MediaMTX reports facts; your API makes decisions.</strong> Keep business logic out of the media server and you can change rules without touching it.</li><li><strong>No single event decides anything.</strong> Hooks are triggers to check. MediaMTX’s current state (the Control API) and the database decide.</li><li><strong>Reconcile, don’t react.</strong> A small loop that compares reality with your records survives restarts, missed hooks and your own bugs.</li><li><strong>Measure before you tune.</strong> Lining up two services’ logs on one timeline turned “video is slow” into four specific numbers, and the biggest one was a camera setting.</li><li><strong>Keyframes rule startup time.</strong> A 3-second keyframe interval cost us over six seconds. Check it on every device model you onboard.</li><li><strong>Fail safe, not open, in cleanup code.</strong> When unsure, don’t stop the stream. That default saved us from our own typo.</li><li><strong>Store paths and IDs, not URLs; issue fresh IDs per session.</strong> Configuration changes then apply instantly, and old links stop working.</li></ol><h3>What we’re building next</h3><ul><li><strong>Viewer heartbeats.</strong> MediaMTX knows how many player sessions are open, but not <em>who</em> is watching. A heartbeat from our own player, with per-viewer tokens checked in the auth callback, gives real viewer counts, a who-watched audit, per-user limits and the ability to revoke a single viewer.</li><li><strong>Enforce mode.</strong> Turn the auth callback from observing to rejecting, starting with viewer tokens for reads.</li><li><strong>Publishing straight from Rust.</strong> Remove FFmpeg from the Howen path: no probe delay, real device timestamps, one less process per camera.</li><li><strong>Event clips.</strong> Automatic recording of the window around high-severity ADAS events, indexed by vehicle and time.</li><li><strong>WebRTC for operators.</strong> Sub-second startup and latency for control-room screens, served by the same MediaMTX paths, with HLS remaining the default for everyone else.</li><li><strong>A dedicated jobs service.</strong> The cleanup loop is self-contained, so it can move out of the web API into a scheduled-jobs service with its own interval.</li></ul><h3>Closing thought</h3><p>None of the pieces here are exotic. MediaMTX, FFmpeg, hls.js, Postgres, NATS and an API gateway are all free and well documented. What took the time was understanding how they behave together under real devices, real networks and real users, and that’s the part we hope this article saves you.</p><p>If you’re integrating MDVRs or NVRs from any vendor, the pattern carries over directly: write one adapter per vendor that ends in RTMP, and reuse everything else. No per-camera licence required.</p><h3>Sources</h3><ul><li><a href="https://github.com/bluenviron/mediamtx">MediaMTX on GitHub</a> (configuration reference, Control API, hooks)</li><li><a href="https://github.com/video-dev/hls.js">hls.js</a> (browser HLS player)</li><li><a href="https://apisix.apache.org/docs/ingress-controller/getting-started/">Apache APISIX Ingress Controller</a> (ApisixRoute)</li><li><a href="https://ffmpeg.org/ffmpeg.html">FFmpeg documentation</a> (input probing, -c copy)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec44fb038136" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The FTP Bug That Only Azure Could Give Us (And Why We’re Grateful It Did)]]></title>
            <link>https://medium.com/@me_sameen/the-ftp-bug-that-only-azure-could-give-us-and-why-were-grateful-it-did-e4f17562197d?source=rss-a3cc8841f580------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4f17562197d</guid>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[ftp]]></category>
            <category><![CDATA[file-upload]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Sandeep]]></dc:creator>
            <pubDate>Fri, 28 Aug 2026 16:59:55 GMT</pubDate>
            <atom:updated>2026-08-28T16:59:55.413Z</atom:updated>
            <content:encoded><![CDATA[<h3>A story about legacy protocols, cloud migrations, and the humbling experience of finally reading the manual</h3><p>Every engineering team has at least one piece of infrastructure that nobody talks about — not because it’s broken, but because it <em>works</em>. It sits quietly in a corner of the architecture diagram, doing its unglamorous job, and everyone silently agrees not to poke it.</p><p>For us, that piece was FTP.</p><h3>The Protocol Nobody Wanted, But Everyone Needed</h3><p>We run a video telematics platform — thousands of connected devices in the field, ours built around hardware from Howen, streaming and uploading footage and telemetry data continuously. Somewhere in that pipeline, a file has to move from a device to our backend. And the device, like a lot of embedded hardware out there, speaks FTP. Not S3 APIs, not SFTP, not some modern object-storage protocol. Plain, old-fashioned FTP.</p><p>If you’ve spent any time in infrastructure, you already know why that raises an eyebrow. FTP was designed in 1971 — before TLS, before NAT, before anyone had thought seriously about credential security. Using it today comes with baggage:</p><ul><li><strong>No encryption by default.</strong> Credentials and file contents travel in the clear unless you bolt on FTPS.</li><li><strong>A connection model that doesn’t scale gracefully.</strong> Every single file transfer needs its own dedicated, ephemeral network port — which we’ll get into in detail, because it’s the entire reason this story exists.</li><li><strong>Firewall and NAT hostility.</strong> Because of that dynamic port requirement, FTP is notoriously difficult to run cleanly behind modern cloud load balancers, which were mostly designed with the assumption that a service listens on one predictable port.</li></ul><p>None of that mattered, though, because we didn’t get to choose the protocol — the device did. Our job wasn’t to pick the ideal transport layer; it was to build a backend that speaks the language our hardware already speaks. So we wrote our own lightweight FTP server, deployed it, pointed our devices at it, and it worked. Beautifully. Quietly. For a long time.</p><p>And when something works quietly for a long time, you stop thinking about <em>how</em> it works. That complacency is where this story really begins.</p><h3>The Migration Nobody Asked For</h3><p>Every infrastructure team eventually faces a moment where “it’s fine where it is” stops being an acceptable answer. For us, that moment came when a significant cloud-provider incident made single-cloud dependency feel like a liability rather than a convenience. Multi-cloud resilience stopped being a nice-to-have on a roadmap and became an urgent, board-level priority. We needed our platform running — reliably — on Azure, not just AWS.</p><p>For most of our stack, this was refreshingly uneventful. We run everything in Kubernetes, and Kubernetes is Kubernetes — the same manifests, the same Helm charts, the same mental model, regardless of which cloud is underneath. Our APIs came up. Our data pipelines came up. Our dashboards came up. One by one, every service in our platform crossed over to Azure and just… worked.</p><p>Except one.</p><p>The FTP server started. It accepted TCP connections. Devices could log in. And then — nothing. Uploads simply hung. No error. No rejection. Just silence, until something eventually timed out.</p><h3>The “If It Ain’t Broke” Trap</h3><p>Here’s the uncomfortable truth we had to sit with: we had been running this FTP server successfully for a long time <em>without actually understanding how FTP works under the hood</em>. We knew it as a black box that took files in on one side and produced files on S3 on the other. It had never given us a reason to look closer.</p><p>That’s a trap every engineer falls into eventually. When something works, curiosity has no foothold — there’s no incentive to dig into internals when the thing in front of you is doing its job. It’s only when it breaks, in a way that resists every quick fix you throw at it, that you’re finally forced to actually <em>read the protocol</em> instead of trusting that it’ll keep behaving the way it always has.</p><p>We tried the obvious things first. Restarted pods. Re-checked our application config. Compared environment variables. Nothing. The application code was byte-for-byte identical to what was running successfully on AWS. Whatever was wrong wasn’t in our code — which meant it had to be something about the <em>network path</em> itself. And to understand a network path problem in FTP, you first have to understand what FTP actually asks of a network.</p><h3>FTP 101: Why One Port Was Never Going to Be Enough</h3><p>Here’s the thing about FTP that a lot of engineers — us included, until this week — never have to think about: <strong>it isn’t one connection. It’s two, running at the same time, doing completely different jobs.</strong></p><p><strong>1. The control connection — one persistent, always-open port.</strong></p><p>This is where a client logs in and issues commands: USER, PASS, PWD, CWD, STOR, and so on. Think of it as the phone line between client and server — always up, always listening, ready to receive instructions at any moment. In our setup, this lives on port <strong>7575</strong>.</p><p><strong>2. The data connection — a temporary, on-demand port, opened fresh per transfer.</strong></p><p>Here’s the part that trips people up: commands alone don’t move file bytes. Once the client says “I want to upload a file,” the control connection isn’t used to carry the actual data. Instead, the server dynamically opens a <strong>brand-new, temporary port</strong>, tells the client exactly which port that is, and the client opens a <strong>second, independent connection</strong> to it — just for that one file. Once the transfer finishes, that port closes again, freed up for the next session to use.</p><p>This is called <strong>passive mode</strong>, and it’s the industry-standard way of doing FTP data transfers today (versus the older <em>active</em> mode, where the server connects back to the client — a model that fails outright behind almost any modern firewall or NAT). In passive mode, the client always initiates both connections, which plays nicer with firewalls, but introduces a new requirement: the server needs a whole <strong>range</strong> of ports available for these on-demand data connections, because multiple sessions might be transferring files simultaneously and each one needs its own dedicated data port for the duration of its transfer.</p><p>In our config, that range is <strong>7576–7586</strong> — eleven ports, meaning up to eleven concurrent file transfers before a new session has to wait for one to free up. (And yes — you’re right to clock that as a scaling concern. A narrow passive port range becomes a real bottleneck the moment your device fleet grows large enough to be uploading concurrently. It’s one more entry on FTP’s list of sins, right alongside the lack of built-in encryption.)</p><h3>Watching It Happen, Live</h3><p>The best way to actually <em>see</em> this two-connection dance is to do it by hand. Here’s how we walked through it ourselves, and how you can too, against any FTP server:</p><pre>telnet ftp.upload.example.com 7575<br>USER s3<br>PASS s3<br>EPSV</pre><p>The server responds with something like:</p><pre>229 Entering Extended Passive Mode (|||7577|)</pre><p>That’s the server saying: <em>“For this session’s file transfer, connect to port 7577.”</em> That number isn’t fixed — it’s chosen dynamically, on the spot, from the configured passive range, and it belongs exclusively to this one session until the transfer completes.</p><p>While that control session is still open, open a <strong>second terminal</strong> and try:</p><pre>telnet ftp.upload.example.com 7577</pre><p>It connects — because right now, in this exact moment, something is actually listening there. Let the session finish (or close the control connection), and try that same telnet again: it’ll fail, because the port has already been released back into the pool for the next session to use.</p><p>This little experiment is the whole story in miniature: <strong>the passive port only exists, and only accepts traffic, for the lifetime of one active transfer.</strong> That single fact — obvious once you’ve seen it, invisible until you go looking — turned out to be exactly what broke us in Azure.</p><h3>Same Code, Same Ports, Two Different Clouds</h3><p>Here’s where it got genuinely strange. We confirmed, port by port, that both environments were configured identically:</p><ul><li>Same persistent port (7575) — open and reachable in both.</li><li>Same passive port range (7576–7586) — declared identically in both Kubernetes Service manifests, verified line by line.</li><li>Same application binary, same startup config, same everything.</li></ul><p>On AWS: uploads worked flawlessly, every time. On Azure: the control connection worked fine — login, PWD, CWD, all of it — but the moment a transfer tried to use one of the passive ports, the connection would just... hang. Not &quot;connection refused.&quot; Not an error. A dead, silent nothing, until something eventually gave up and timed out.</p><p>That distinction — <em>refused</em> versus <em>silent hang</em> — turned out to be the clue that cracked the whole thing open. “Refused” means a packet reached its destination and got an active response saying “nothing’s here.” A silent hang means the packet never got a response <em>at all</em> — something upstream, before it ever reached our application, was simply swallowing it.</p><p>That pointed us away from our own code entirely, and straight at the one layer we hadn’t thought to question: <strong>what each cloud’s load balancer does by default, automatically, without ever being asked.</strong></p><h3>The Default Nobody Told Us About</h3><p>Every cloud load balancer needs a way to decide whether it’s safe to send traffic to a given backend port. That mechanism is a <strong>health probe</strong> — a periodic, automated check that asks, in effect, “is anything actually listening here right now?” If the answer is no, the load balancer marks that port unhealthy and quietly stops routing traffic to it. No error to the client. No log entry anyone would think to check. Just silence — which, in hindsight, is exactly what we’d been staring at.</p><p>Here’s the mismatch: <strong>our passive ports are only ever “listening” while a transfer is actively in progress.</strong> The rest of the time — which, statistically, is nearly always — they’re closed by design. That’s not a bug. That’s the entire architecture of FTP passive mode working exactly as intended.</p><p>AWS’s load balancer, as it turned out, was configured (via our own Helm chart, without us fully appreciating why) to health-check the <em>entire listener set</em> against the one port that’s reliably, permanently alive — the control port. It never independently questioned whether each individual passive port was “up” at that instant, because that was never the right question to ask in the first place.</p><p>Azure’s load balancer, by contrast, defaults to probing <strong>every declared port independently, against itself.</strong> Ask “is 7577 listening right now?” often enough, and — because 7577 is only supposed to be alive during an active transfer — you’ll get “no” almost every time. Azure took that “no” at face value, marked the port unhealthy, and stopped forwarding traffic to it entirely. Not because anything was actually broken. Because we were, unknowingly, asking the load balancer the wrong question.</p><h3>The Fix: Teaching Azure to Ask the Right Question</h3><p>Once we understood <em>what</em> was being asked, the fix became almost anticlimactic. We didn’t need to open new ports, rewrite the application, or change a single line of Go. We needed to tell Azure’s load balancer: <em>“Don’t check whether each passive port is alive on its own. Check whether the persistent control port is alive, and treat that as good enough for all of them.”</em></p><p>Azure’s AKS integration exposes exactly this as a per-port annotation on the Kubernetes Service object:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: ftp-server<br>  namespace: iot-platform<br>  annotations:<br>    service.beta.kubernetes.io/port_7576_health-probe_port: &quot;7575&quot;<br>    service.beta.kubernetes.io/port_7576_health-probe_protocol: &quot;Tcp&quot;<br>    service.beta.kubernetes.io/port_7577_health-probe_port: &quot;7575&quot;<br>    service.beta.kubernetes.io/port_7577_health-probe_protocol: &quot;Tcp&quot;<br>    # ...repeated for every port in the passive range...<br>    service.beta.kubernetes.io/port_7586_health-probe_port: &quot;7575&quot;<br>    service.beta.kubernetes.io/port_7586_health-probe_protocol: &quot;Tcp&quot;</pre><p>Every passive port, individually redirected to check the one port that’s genuinely, reliably always up. In effect, we’re telling Azure: <em>“7575 being alive is a trustworthy proxy for the whole server being healthy and capable of opening any of these passive ports on demand — even if none of them happen to be in active use right this second.”</em></p><p>Nine-ish repeated annotation lines. That’s the entire fix. Redeploy, and every earlier symptom reverses itself in order: telnet to a live passive port connects instantly, curl -v sails past the EPSV exchange without hanging, and the transfer completes with a clean success response, exactly the way it always had on AWS.</p><h3>The Real Lesson Wasn’t the Nine Lines</h3><p>It would be easy to write this whole story off as “Azure load balancer gotcha, annotation fixes it, moving on.” And technically, sure — that’s the fix. But that undersells what actually happened here.</p><p>The real story is that we had been operating a piece of infrastructure for a long time without deeply understanding its core mechanics — not out of negligence, but because it had simply never <em>needed</em> to be understood. It worked. There was no incentive to look closer. It took a cloud migration, a genuinely confusing failure, and a fair amount of dead-end troubleshooting before we sat down and actually read up on how FTP’s control and data connections interact — knowledge that, frankly, we should have had on day one of writing an FTP server, not two years and one cloud migration later.</p><p>There’s a broader habit worth naming here, especially in a moment where it’s so easy to reach for a quick AI-assisted fix or a Stack Overflow answer and move on: <strong>sometimes the fastest fix is the enemy of actually understanding your system.</strong> We didn’t get to shortcut our way out of this one. The only way through was going back to fundamentals — actually learning what EPSV means, what a passive port range is for, why it exists at all — before the <em>why</em> behind the fix made any sense.</p><p>So if there’s a takeaway here, it’s not “add these annotations to your Azure load balancer.” It’s this: the next time something that’s “just always worked” suddenly doesn’t, resist the urge to just patch around it. Go find out <em>why</em> it worked in the first place. You might be surprised how much you didn’t know about a system you’ve been running successfully for years.</p><p>Cheers — and may your passive ports always find a healthy home. 🚀</p><p><em>If you’ve hit something similar migrating FTP, SFTP, or any dynamic-port protocol between clouds, I’d love to hear how you solved it — drop a comment.</em></p><p>#FTP #SFTP #Azure #AWS #IOT #Howen</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4f17562197d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ordering in MQTT: Three Real Approaches I Used for Vehicle Telemetry on EMQX]]></title>
            <link>https://medium.com/@me_sameen/ordering-in-mqtt-three-real-approaches-i-used-for-vehicle-telemetry-on-emqx-af7c80df1814?source=rss-a3cc8841f580------2</link>
            <guid isPermaLink="false">https://medium.com/p/af7c80df1814</guid>
            <category><![CDATA[mqtt]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[emqx-broker]]></category>
            <dc:creator><![CDATA[Sandeep]]></dc:creator>
            <pubDate>Sun, 15 Mar 2026 15:21:49 GMT</pubDate>
            <atom:updated>2026-03-15T15:25:55.516Z</atom:updated>
            <content:encoded><![CDATA[<p><em>From broker-level shared subscriptions to ring-based application filtering — what actually happened when I tried each one at 50,000 messages per second.</em></p><p>Written from direct production experience · EMQX · Kubernetes · Vehicle Telematics</p><p><strong>Ordering in MQTT: Three Real Approaches I Used for Vehicle Telemetry on EMQX</strong></p><p><em>From broker-level shared subscriptions to ring-based application filtering — what actually happened when I tried each one at 50,000 messages per second.</em></p><p>Written from direct production experience · EMQX · Kubernetes · Vehicle Telematics</p><p>Here is a thing that nobody warns you about: the moment you run more than one consumer pod against an MQTT broker, your message ordering is structurally broken. Not occasionally. Not at high load. By design, always.</p><p>I found this out while building a vehicle live tracking platform. Each vehicle publishes telemetry — position, speed, engine events, door state — to EMQX using its IMEI as the topic identifier. When we scaled past a single consumer pod, vehicles started teleporting on the map. A truck would jump 3km backwards, then snap forward. The data was all there. It was arriving out of order.</p><p>This article is a detailed account of the three approaches I explored to solve this. Not theoretical — I implemented all three to varying depths, hit the walls, and made a final call. Here is exactly what happened.</p><blockquote>The sequence of events is the data. For live vehicle tracking, lose the order and you lose the product.</blockquote><h3>Why ordering matters more than you think</h3><p>Before the approaches — let me make the stakes concrete with real numbers from our system.</p><pre>Metric                     Value<br>---------------------------------------<br>Vehicles in Fleet          250,000<br>Messages / Second          50k/s<br>Consumer Pods              8</pre><p>Each vehicle sends a telemetry event every 5 seconds — 250,000 vehicles ÷ 5 seconds = 50,000 messages per second sustained. That is just the scheduled heartbeat, before any burst of event-driven messages for ignition, harsh braking, speeding, or geofence crossings.</p><p>With 8 consumer pods and no ordering strategy, a single vehicle’s consecutive messages could land on any of the 8 pods in any order. Consumer pod 3 processes the event at T+0, pod 7 processes T+1, pod 3 processes T+2. The position state your system computes is wrong. Your live map is wrong. Your tenant data feed is wrong. Your mobile app shows vehicles glitching.</p><p>For the mobile use case specifically: every driver’s phone connects directly to EMQX and subscribes to iot/telematicsdata/{imei} — their own vehicle’s topic. If events arrive out of order on that topic, the map on their screen jumps. That is a visible product failure, not a backend metric.</p><p><strong>01</strong></p><p><strong>APPROACH 1 — Our Production Choice</strong></p><h3>EMQX Shared Subscription Hash Strategy</h3><p>EMQX has a built-in concept called shared subscriptions. Instead of every consumer pod receiving every message (fan-out), a shared subscription delivers each message to exactly one consumer in a group — load balancing at the broker level. The topic prefix changes to $share/{group}/iot/telematicsdata/#.</p><p>What matters is the strategy EMQX uses to decide which consumer gets each message. There are five options:</p><pre>Strategy      Description                    Behavior<br>-------------------------------------------------------------<br>Random        Any pod, at random             No order<br>Round Robin   Rotates across pods            No order<br>Sticky        Same client session            Fragile<br>Hash ✓        Deterministic by topic         Ordered<br>Local         Same broker node preferred     Cluster only</pre><p>The hash strategy computes hash(topic) % number_of_consumers. Since the topic contains the IMEI, the same vehicle always routes to the same pod. No partition topics to manage. No consumer code changes. One broker setting.</p><p>We configured this in EMQX’s dashboard under Management → MQTT → Shared Subscriptions → Strategy → hash. Consumers subscribe using the shared group prefix. Done.</p><blockquote>Subscription: $share/telematics-group/iot/telematicsdata/#</blockquote><blockquote>Topic source: iot/telematicsdata/{imei}</blockquote><blockquote>Hash key: full topic string (contains IMEI)</blockquote><blockquote>Result: same IMEI -&gt; same pod, every time</blockquote><blockquote>250,000 vehicles across 8 pods = ~31,250 IMEIs per pod, deterministically</blockquote><h4>Mobile app and tenant feeds</h4><p>The mobile app does not need shared subscriptions at all. It subscribes directly to iot/telematicsdata/{imei} and receives that vehicle’s ordered stream, completely untouched by the shared subscription logic. For tenant data feeds, our consumer enriches messages with tenant metadata before forwarding downstream. Ordering is preserved the whole way through because the consumer owns those IMEIs from the start.</p><h4>The numbers that matter</h4><pre>Metric                  Value<br>------------------------------------<br>Broker Message Cost     1×<br>Extra Latency Added     0 ms<br>IMEIs per Pod           ~31k</pre><h4>Drawbacks — the honest version</h4><blockquote>Drawback 1 — Broker-wide change</blockquote><blockquote>The strategy setting applies globally to all shared subscription groups on this EMQX instance. Every team using shared subscriptions on this broker will see their message distribution change. You need to coordinate, communicate, and plan a migration window. This is not a change you make silently on a Friday afternoon.</blockquote><blockquote>Drawback 2 — No autoscaling, StatefulSet required</blockquote><blockquote>The hash distributes across however many consumer pods are currently active. Scale from 8 pods to 9 and the hash rebalances — IMEI X that always went to pod 3 now goes to pod 9. During the transition window, ordering can break. We run consumers as a Kubernetes StatefulSet with a fixed replica count. Scaling is a deliberate planned operation, not an autoscale event.</blockquote><blockquote><strong><em>Why I chose this for production</em></strong></blockquote><blockquote><em>This is the approach I chose for production. One config change. Structural ordering guarantee. Zero consumer code changes. Minimal operational overhead. The StatefulSet constraint is manageable when your traffic patterns are predictable — which for a fleet management platform, they generally are.</em></blockquote><p><strong>02</strong></p><p><strong>APPROACH 2 — Flexible, More Moving Parts</strong></p><h3>EMQX Rule Engine: Re-publishing With Partition Routing</h3><p>This approach keeps the broker’s shared subscription layer untouched and moves the partition routing into the EMQX Rule Engine. A rule intercepts every message on the common topic, computes hash(imei) % 32, and re-publishes to a partition-specific topic. Consumers subscribe to their assigned partition range.</p><h4>The topic transformation</h4><blockquote><strong>Original publish:<br>iot/telematicsdata/353012094XXXXXX<br>(IMEI in topic)</strong></blockquote><blockquote><strong>Rule Engine:<br>hash(imei) % 32 = partition_id</strong></blockquote><blockquote><strong>Re-published topic:<br>iot/telematicsdata/7/353012094XXXXXX<br> partition_id (0–31)</strong></blockquote><blockquote><strong>Consumer distribution:</strong></blockquote><blockquote><strong>Pod 0 → iot/telematicsdata/0/# through 3/#<br>Pod 1 → iot/telematicsdata/4/# through 7/#<br>…</strong></blockquote><blockquote><strong>32 partitions / 8 pods = 4 partitions per pod</strong></blockquote><h4>Why 32 partitions specifically</h4><p>32 is a power of 2. With 8 pods, each pod owns exactly 4 partitions. Scale to 16 pods and each pod owns 2. Scale back to 4 and each pod owns 8. The arithmetic is always clean with no remainder. Crucially, the partition count stays fixed at 32 regardless of how many pods are running — you reassign partition subscriptions between pods; you never change the hash function.</p><pre>Metric                     Value<br>---------------------------------------<br>Fixed Partitions           32<br>Avg IMEIs per Partition    ~7,800<br>Partitions per Pod         4 (8 pods total)</pre><h4>Mobile subscriptions and tenant feeds</h4><p>The IMEI is preserved in the re-published topic. A mobile client subscribes to iot/telematicsdata/+/{imei} — the single-level wildcard matches any partition ID — and receives its vehicle’s complete ordered stream without knowing anything about partitions. For multi-tenant distribution, chain a second Rule Engine rule that attaches tenant identifiers and forwards to iot/{tenant_id}/telematicsdata/{partition_id}/{imei}.</p><h4>The numbers that reveal the cost</h4><pre>Metric                         Value<br>-------------------------------------------<br>Broker Message Load            2×<br>Added Latency per Message      ~0.5 ms<br>Internal Messages at 50k/s     100k/s  </pre><h4>Drawbacks</h4><blockquote>Drawback 1 — Double hop at every message</blockquote><blockquote>Every message hits the broker twice. At 50k/s that is 100,000 internal deliveries per second — 6 million extra per minute running inside EMQX. We measured broker CPU increase of around 18% after enabling the rule at sustained 40k/s load. Benchmark your broker’s capacity ceiling before committing.</blockquote><blockquote>Drawback 2 — Partition count is a shared constant</blockquote><blockquote>The Rule Engine and your consumer deployment must both agree on 32. If these ever drift — someone updates the rule to 64 without updating consumer subscriptions — messages go to partition topics nobody is watching. Silent data loss. Treat partition count as a versioned configuration artifact that changes only via a coordinated deployment.</blockquote><blockquote>Drawback 3 — Rule Engine resource pressure</blockquote><blockquote>Each active rule competes with core broker routing for CPU and memory. Monitor EMQX resource utilisation closely after enabling high-throughput re-publishing rules.</blockquote><p><strong>03</strong></p><p><strong>APPROACH 3 — Quick to Ship, Costly at Scale</strong></p><h3>Application-Level Ring Filtering</h3><p>This approach does not touch the broker at all. Every consumer pod subscribes to the common topic and receives every message. At the application level, each pod checks a consistent hash ring: is this IMEI mine? If yes, process it. If no, discard it.</p><p>The appeal is obvious. No broker access required. No infrastructure coordination. One afternoon of implementation work. It sounds clean — until you look at what it actually costs at scale.</p><h4>The brutal maths at 50k/s with 8 pods</h4><pre>Metric                        Value<br>-------------------------------------------<br>Total Broker Deliveries       400k/s<br>Discarded per Pod             87.5%<br>Network Traffic vs Needed     8×</pre><pre>What each pod does with 50,000 messages/second<br>(8 pods, no ring filter optimisation)<br><br>Pod 0  KEEP 6,250/s  |  DISCARD 43,750/s  (87.5% waste)<br>Pod 1  KEEP 6,250/s  |  DISCARD 43,750/s  (87.5% waste)<br>Pod 2  KEEP 6,250/s  |  DISCARD 43,750/s  (87.5% waste)<br>Pod 3  KEEP 6,250/s  |  DISCARD 43,750/s  (87.5% waste)<br>Pod 4  KEEP 6,250/s  |  DISCARD 43,750/s  (87.5% waste)<br>Pod 5  KEEP 6,250/s  |  DISCARD 43,750/s  (87.5% waste)<br>Pod 6  KEEP 6,250/s  |  DISCARD 43,750/s  (87.5% waste)<br>Pod 7  KEEP 6,250/s  |  DISCARD 43,750/s  (87.5% waste)<br><br>Total: 400,000 broker deliveries/sec<br>to process just 50,000 useful messages</pre><p>At 50,000 messages per second with 8 consumer pods, the broker delivers 400,000 messages per second across all pods. Each pod receives all 50,000, runs the ring check, and processes only ~6,250. It discards 43,750 per second — 87.5% of everything it receives. Full deserialization, ring lookup, and delivery cost paid for messages that produce no value.</p><p>This scales multiplicatively with pod count. Go to 16 pods at the same load: each pod discards 46,875 per second. At 32 pods you are delivering 1,600,000 messages per second to process 50,000 of them. The solution gets more wasteful the more you scale.</p><blockquote>The maths, plainly:</blockquote><blockquote>250,000 vehicles / 5s = 50,000 msg/s</blockquote><blockquote>50,000 msg/s x 8 pods = 400,000 deliveries/s</blockquote><blockquote>Each pod keeps: 50,000 / 8 = 6,250 msg/s</blockquote><blockquote>Each pod discards: 43,750 msg/s (87.5% waste)</blockquote><blockquote>Total wasted CPU + network: 7/8 of everything your broker does</blockquote><h4>Ring rebalancing is the hidden trap</h4><p>When you add or remove a consumer pod, the ring changes. During the transition, two pods may both claim an IMEI — or neither does. You need all pods to agree on the new ring simultaneously, which means distributed coordination logic. That is significant complexity added to what was supposed to be a simple approach.</p><h4>When this is actually acceptable</h4><blockquote>For small deployments this is a legitimate starting point. Under 200 vehicles, under 1,000 msg/s, with 2 to 3 consumer pods — the waste is trivially small and the implementation is a real afternoon’s work with no broker access needed. Set a scale threshold before you start, document the migration path to Approach 1 or 2, and do not let operational inertia keep you here past that limit.</blockquote><pre>Side-by-side Comparison<br><br>Dimension                  Approach 1 — Hash Sub     Approach 2 — Rule Engine     Approach 3 — Ring Filter<br>---------------------------------------------------------------------------------------------------------<br>Broker Config Needed       1 setting only           Rule Engine rule             None at all<br>Consumer Code Change       Subscribe prefix only    Subscribe to partitions      Ring logic in app<br>Msg Cost at 50k/s          50k deliveries/s         100k deliveries/s            400k deliveries/s<br>Extra Latency              0 ms                     ~0.5 ms per message          ~0 ms<br>Broker CPU Impact           Minimal                 +~18% at 40k/s              Minimal (broker side)<br>Kubernetes Model           StatefulSet — fixed pods Flexible partition assign   Any — ring rebalance pain<br>Mobile App Support         Direct IMEI sub          Wildcard partition sub       Direct IMEI sub<br>Multi-Tenant Feeds         Consumer enriches        Rule Engine enriches         Consumer enriches<br>Best for &lt; 200 Vehicles    Yes                      Yes (overkill)              Yes — ideal here<br>Good at 50k msg/s          Yes                      Yes — benchmark first       No — 8× waste</pre><h3>Which one should you choose?</h3><p>There is no universally correct answer — but the signals are clear once you know your constraints.</p><h4>Choose Approach 1 if:</h4><ul><li>You own and operate EMQX with admin access</li><li>Your device fleet is large and message rates are high</li><li>You want the simplest possible consumer code — subscribe and process, no partition awareness</li><li>You can run consumers as a StatefulSet with a fixed replica count</li><li>You are comfortable coordinating the broker config change with other teams</li></ul><p>My recommendation for production vehicle tracking at scale. One config change, structural ordering, no consumer code changes. The StatefulSet constraint is the only cost, and for predictable fleet traffic patterns it is very manageable.</p><h4>Choose Approach 2 if:</h4><ul><li>You need flexible partition scaling without a StatefulSet constraint</li><li>You want per-partition topic visibility for debugging and monitoring</li><li>You are enriching or transforming messages as part of the routing step</li><li>You are building multi-tenant data distribution with clean topic namespacing per tenant</li><li>You have benchmarked the double-hop overhead at your target message rate</li></ul><p>More powerful and more flexible than Approach 1, but with more moving parts and a higher operational ceiling. The double-hop cost is real — measure it before committing.</p><h4>Choose Approach 3 if:</h4><ul><li>Your fleet is small — under 200 vehicles</li><li>Message rates are modest — well under 1,000 per second</li><li>Your team does not have access to broker configuration</li><li>You need ordering guarantees quickly and will migrate to a broker-level approach later</li></ul><p>An acceptable starting point for small deployments. Set a clear scale threshold and plan the migration before you hit it. Never let this become permanent at scale.</p><h3>Closing thoughts</h3><p>Message ordering in MQTT is one of those problems that is invisible until it is not. When your system is small, everything looks fine. When you scale, the cracks appear in exactly the parts of your product most visible to users — vehicles jumping on a live map, alerts firing late, tenant data feeds delivering events out of sequence.</p><p>All three approaches I have described here solve the ordering problem. They differ in where they put the work, what they cost at scale, and how much flexibility they offer. Approach 1 is what I run in production. Approach 2 is the right tool when you need more control or cleaner scaling. Approach 3 is the fast path for small systems — useful, but only until you outgrow it.</p><p>Whatever you choose, make the decision deliberately and instrument your pipeline from day one so you can see ordering violations if they occur. Because you cannot fix what you cannot measure.</p><blockquote>The sequence of events is the data. Preserve it at the earliest point in your pipeline where it is feasible to do so. Everything downstream gets easier when the stream is correct from the start.</blockquote><p><em>Written from direct production experience building vehicle telemetry infrastructure on EMQX.</em></p><p>If this helped you, share it with a fellow IoT engineer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af7c80df1814" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka and NATS JetStream: My Journey Understanding Partitioning and Per-Device Message Ordering]]></title>
            <link>https://medium.com/@me_sameen/kafka-and-nats-jetstream-my-journey-understanding-partitioning-and-per-device-message-ordering-9e849ee86a6a?source=rss-a3cc8841f580------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e849ee86a6a</guid>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[nat]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <dc:creator><![CDATA[Sandeep]]></dc:creator>
            <pubDate>Fri, 27 Feb 2026 19:32:51 GMT</pubDate>
            <atom:updated>2026-02-27T19:32:51.833Z</atom:updated>
            <content:encoded><![CDATA[<h3>How I Learned That Sometimes More Streams Means Better Performance</h3><p>I’ve spent the last six months building a high-throughput IoT data pipeline, and along the way, I discovered something that completely changed how I think about message streaming architecture. This isn’t a comparison of which technology is “better” — both Kafka and NATS JetStream are excellent tools. Instead, this is my story of understanding how they approach the same problem from different angles, and why that matters.</p><h3>The Problem I Was Solving</h3><p>Our system collects data from IoT sensors — temperature monitors, humidity sensors, air quality detectors — deployed across buildings, factories, and warehouses. We have 250,000 devices in the field, each sending readings every 5 seconds. That translates to roughly 50,000 messages per second flowing through our pipeline continuously.</p><p>The critical requirement: <strong>messages from each device must be processed in order</strong>. If Device-A-123 sends temperature readings at 10:00:00, 10:00:05, and 10:00:10, they must be processed in exactly that sequence. Processing them out of order would corrupt our state tracking and trigger false alerts.</p><p>This is a common challenge in IoT systems, logistics tracking, user activity streams, or any scenario where entity state evolves over time.</p><h3>Starting with Kafka: The Familiar Path</h3><p>I started with Kafka because that’s what I knew. The solution seemed straightforward: use the device ID as the partition key.</p><h3>How Kafka Partitioning Works</h3><p>Kafka organizes data into topics, and each topic is split into partitions. A partition is an ordered, append-only log of messages. Here’s what I implemented:</p><pre>// Publishing to Kafka<br>deviceID := &quot;sensor-A-123&quot;<br>partitionKey := deviceID</pre><pre>producer.Send(ProducerRecord{<br>    Topic: &quot;sensor-readings&quot;,<br>    Key:   partitionKey,<br>    Value: message,<br>})</pre><pre>// Kafka calculates: partition = hash(key) % 32<br>// Result: sensor-A-123 → partition 5 (always)</pre><p>This worked beautifully. All messages from sensor-A-123 went to partition 5, maintaining perfect ordering. Other devices hashed to different partitions, distributing the load evenly.</p><p>Our Kafka setup:</p><ul><li>Topic: sensor-readings</li><li>Partitions: 32</li><li>Devices: 250,000 spread across 32 partitions (~7,800 devices per partition)</li><li>Messages per partition: ~1,562 msgs/sec</li><li>Total throughput: 50,000 msgs/sec</li><li>Retention: 24 hours</li></ul><h3>What Worked Well</h3><p><strong>Ordering guarantee</strong>: Messages from the same device stayed in order within their partition.</p><p><strong>Load distribution</strong>: With 250,000 devices and 32 partitions, each partition handled about 7,800 devices — reasonable distribution given the load.</p><p><strong>Consumer parallelism</strong>: I could run 8 consumers, each processing 4 partitions, scaling linearly with load.</p><p><strong>Mature ecosystem</strong>: Kafka’s tooling, monitoring, and documentation made operations predictable.</p><h3>What Became Challenging</h3><p><strong>Consumer rebalancing</strong>: Every time I scaled consumers up or down, Kafka triggered a rebalance. During those 30–60 seconds, all processing stopped. For a system handling 50,000 messages per second with 99.9% uptime requirements, losing a full minute of processing capacity was painful. Messages piled up in the brokers, and when consumers resumed, they faced significant lag.</p><p><strong>Replaying device history</strong>: When our analytics team wanted to replay 24 hours of data for a specific device, we hit a wall. The device’s messages were scattered throughout partition 5, interleaved with 7,799 other devices. To extract one device’s history, we had to:</p><ol><li>Read the entire partition (millions of messages)</li><li>Filter for the specific device ID</li><li>Discard 99.987% of the data</li></ol><p>At 50,000 msgs/sec with 24-hour retention, each partition held about 135 million messages (about 67GB of data). Scanning through this to find one device’s ~17,280 messages (one every 5 seconds for 24 hours) took 15–20 minutes per device. For ad-hoc analysis or debugging, it was painful.</p><p><strong>Partition limits</strong>: We discussed creating more granular partitions (maybe 128 or 256), but Kafka operations teams warn against too many partitions per broker. The coordination overhead becomes significant beyond a certain point. With our throughput, we couldn’t easily scale beyond 32–64 partitions per topic.</p><p>These weren’t showstoppers, but they nagged at me. There had to be a better way to handle per-device isolation and replay.</p><h3>Discovering NATS JetStream</h3><p>During a conference, I heard about NATS JetStream. The pitch was “Kafka-like persistence with NATS-like simplicity.” I was skeptical but curious.</p><p>NATS uses <strong>subjects</strong> for routing (like topics) and <strong>streams</strong> for persistence (like… topics? partitions?). This is where my confusion began.</p><h3>My First (Wrong) Mental Model</h3><p>Coming from Kafka, I thought:</p><pre>NATS Stream = Kafka Topic<br>NATS Subject = Kafka Partition Key</pre><pre>So I should create:<br>  - 1 stream (like 1 topic)<br>  - Route messages via subjects (like partition keys)</pre><p>I created a single stream that captured all messages with subject pattern device.*.sensor.data:</p><pre>// Stream configuration<br>stream := Stream{<br>    Name:     &quot;sensor-data&quot;,<br>    Subjects: []string{&quot;device.*.sensor.data.*&quot;},<br>}</pre><pre>// Publishing<br>deviceID := &quot;sensor-A-123&quot;<br>subject := fmt.Sprintf(&quot;device.%s.sensor.data&quot;, deviceID)<br>js.Publish(subject, message)</pre><p>This worked. Messages flowed at 50,000 per second, ordering was maintained per device, and consumers processed everything correctly.</p><p>But performance was… odd.</p><h3>The Performance Problem</h3><p>With 24-hour retention at 50,000 msgs/sec, my single stream accumulated about 4.32 billion messages (2.2TB). When a consumer needed to catch up after falling behind, the performance was terrible:</p><ul><li><strong>Catchup speed</strong>: ~5,000 messages/second</li><li><strong>Time to catch up 1 hour</strong>: About 36 minutes (180,000 messages at 5,000 msgs/sec)</li><li><strong>Full device replay</strong>: 8–10 hours for 24 hours of data</li></ul><p>These numbers didn’t make sense. The hardware was solid — NVMe SSDs, plenty of RAM, 16 CPU cores per server. CPU usage showed high filter-matching overhead. Every message required checking against 128 consumer filters (4 different applications, each consuming all 32 partitions).</p><p>I was using NATS like Kafka, but getting worse performance. Something was fundamentally wrong with my approach.</p><h3>The Breakthrough: Understanding Streams vs Partitions</h3><p>Late one night, reading through NATS documentation for the dozenth time, something clicked. I was thinking about this backwards.</p><p>In Kafka:</p><ul><li><strong>Topic</strong> = Logical grouping</li><li><strong>Partition</strong> = Physical storage unit, ordered log</li><li>You can’t have a partition without a topic</li></ul><p>In NATS:</p><ul><li><strong>Subject</strong> = Routing pattern</li><li><strong>Stream</strong> = Physical storage unit, ordered log</li><li>A stream can capture one or many subjects</li><li><strong>Streams are independent entities</strong></li></ul><p>The insight: <strong>A NATS stream is equivalent to a Kafka partition, not a Kafka topic.</strong></p><p>When Kafka has a topic with 32 partitions, it’s really 32 independent ordered logs grouped under one name. NATS lets you create those 32 independent ordered logs directly, without requiring a grouping layer.</p><h3>The Revelation</h3><pre>Kafka Topic with 32 Partitions:<br>sensor-readings/<br>├─ partition-0 (independent ordered log)<br>├─ partition-1 (independent ordered log)<br>├─ partition-2 (independent ordered log)<br>...<br>└─ partition-31 (independent ordered log)</pre><pre>NATS with 32 Streams:<br>├─ sensor-partition-0 (independent ordered log)<br>├─ sensor-partition-1 (independent ordered log)<br>├─ sensor-partition-2 (independent ordered log)<br>...<br>└─ sensor-partition-31 (independent ordered log)</pre><pre>These are the SAME THING!</pre><p>I had been trying to replicate Kafka’s topic layer in NATS, but NATS doesn’t need it. I could just create the partitions (streams) directly.</p><h3>Implementing One Stream Per Partition</h3><p>I restructured the system to use 32 independent streams:</p><pre>// Create 32 streams, one per partition<br>for partition := 0; partition &lt; 32; partition++ {<br>    streamName := fmt.Sprintf(&quot;sensor-partition-%d&quot;, partition)<br>    subject := fmt.Sprintf(&quot;device.%d.sensor.data.*&quot;, partition)<br>    <br>    js.AddStream(&amp;nats.StreamConfig{<br>        Name:      streamName,<br>        Subjects:  []string{subject},<br>        Storage:   nats.FileStorage,<br>        Retention: nats.LimitsPolicy,<br>        MaxAge:    24 * time.Hour,<br>    })<br>}</pre><p>Publishing logic remained identical to Kafka:</p><pre>// Calculate partition (same as Kafka)<br>deviceID := &quot;sensor-A-123&quot;<br>partition := hash(deviceID) % 32</pre><pre>// Publish to specific partition&#39;s stream<br>subject := fmt.Sprintf(&quot;device.%d.sensor.data.%s&quot;, partition, deviceID)<br>js.Publish(subject, message)</pre><pre>// This routes to sensor-partition-5 automatically</pre><p>Consumer logic:</p><pre>// Each pod calculates which partitions it handles<br>totalPartitions := 32<br>podID := 0  // From environment<br>totalPods := 4</pre><pre>partitionsPerPod := totalPartitions / totalPods<br>startPartition := podID * partitionsPerPod<br>endPartition := startPartition + partitionsPerPod</pre><pre>// Spawn consumer for each partition<br>for p := startPartition; p &lt; endPartition; p++ {<br>    streamName := fmt.Sprintf(&quot;sensor-partition-%d&quot;, p)<br>    go consumeStream(streamName)<br>}</pre><p>My initial reaction: “This feels wrong. I’m creating 32 separate things instead of 1. Won’t this use 32× the resources?”</p><h3>What Actually Happened</h3><h3>Storage: Exactly the Same</h3><pre>Single stream:   2.2TB (all 4.32 billion messages)<br>32 streams:      2.2TB (same messages, split across streams)</pre><pre>Per stream: 2.2TB ÷ 32 = 69GB each<br>Messages per stream: 135 million (24 hours at 1,562 msgs/sec)</pre><p>The data wasn’t duplicated. It was partitioned. Each message went to exactly one stream based on its subject. This is identical to how Kafka partitions data within a topic.</p><h3>Memory: Slightly Better</h3><pre>Single stream:   2.3TB (data + 100GB overhead for indexing)<br>32 streams:      2.2TB (data + 35GB overhead)</pre><pre>Savings: 65GB from more efficient indexing</pre><p>With a single stream and thousands of device subjects, NATS maintains a large subject index. With 32 streams each handling one subject pattern, the indexing is trivial.</p><h3>CPU: Dramatically Better</h3><pre>Single stream:   65% CPU (filter matching overhead)<br>32 streams:      38% CPU (no filter matching needed)</pre><pre>Savings: 42% less CPU usage</pre><p>This was the surprise. With one stream capturing all device subjects, every message required checking against every consumer filter (128 consumers across 4 applications). At 50,000 msgs/sec, that’s 6.4 million filter operations per second.</p><p>With one stream per partition, all messages in the stream matched all consumers — no filtering needed. NATS could skip the filtering entirely.</p><h3>Disk I/O: The Real Winner</h3><p>This is where everything came together.</p><p><strong>Single Stream Replay</strong> (sensor-A-123 for 24 hours):</p><pre>Stream contains: 4.32 billion messages (all 250,000 devices)<br>Partition 5 has: 135 million messages (7,800 devices)<br>Device A-123&#39;s messages: ~17,280 messages (one every 5 seconds)</pre><pre>Process:<br>1. Scan through 135 million messages in partition 5&#39;s data<br>2. Filter for device A-123 (1 in 7,800 messages match)<br>3. Discard 134,982,720 messages</pre><pre>Disk access: Random seeks throughout 2.2TB file<br>Effective read speed: ~5,000 messages/second<br>Time: 135M ÷ 5,000 = 27,000 seconds = 7.5 hours</pre><p><strong>32 Streams Replay</strong> (sensor-A-123 for 24 hours):</p><pre>Stream sensor-partition-5 contains: 135 million messages (partition 5 only)<br>Device A-123&#39;s messages: ~17,280 messages</pre><pre>Process:<br>1. Sequential read from partition 5&#39;s stream (69GB file)<br>2. Filter for device A-123 (1 in 7,800 still, but smaller dataset)<br>3. Stream file fits better in cache, all partition 5 data in one place</pre><pre>Disk access: Sequential read from 69GB file<br>Effective read speed: ~500,000 messages/second<br>Time: 135M ÷ 500k = 270 seconds = 4.5 minutes</pre><p>100× faster. The difference is disk access patterns:</p><ul><li><strong>Random seeks</strong>: Jumping around a 2.2TB file is slow, even on NVMe</li><li><strong>Sequential reads</strong>: Reading contiguously from a 69GB file is dramatically faster</li></ul><h3>Understanding the Independence Advantage</h3><p>Beyond performance, the independence of streams provided operational benefits I hadn’t anticipated.</p><h3>Isolation</h3><p>When partition 12’s consumer started lagging due to a downstream MongoDB issue, only partition 12 was affected. The other 31 partitions continued processing their collective 48,438 messages per second normally. With a single stream, contention meant all consumers slowed down together.</p><p>At 50,000 msgs/sec, one slow consumer could create cascading backpressure. With independent streams, the blast radius was limited to 3% of the system (1,562 msgs/sec in one partition).</p><h3>Independent Configuration</h3><pre>// High-value sensors: longer retention<br>stream5Config := nats.StreamConfig{<br>    Name:   &quot;sensor-partition-5&quot;,<br>    MaxAge: 7 * 24 * time.Hour,  // 7 days<br>}</pre><pre>// Test sensors: minimal retention<br>stream28Config := nats.StreamConfig{<br>    Name:   &quot;sensor-partition-28&quot;,<br>    MaxAge: 1 * time.Hour,  // 1 hour<br>}</pre><p>Different partitions could have different retention policies, replication factors, or storage tiers. This granular control wasn’t possible with Kafka partitions within a single topic.</p><h3>Scaling Without Rebalancing</h3><p>When I scaled from 4 pods to 8 pods during a traffic spike:</p><pre>Before: 4 pods × 8 partitions each = 32 partitions covered<br>        Each pod: 12,500 msgs/sec (8 partitions × 1,562 msgs/sec)</pre><pre>After:  8 pods × 4 partitions each = 32 partitions covered<br>        Each pod: 6,250 msgs/sec (4 partitions × 1,562 msgs/sec)</pre><pre>Process:<br>1. Kubernetes creates 4 new pods<br>2. Each pod calculates: myPartitions = totalPartitions / totalPods<br>3. Pods immediately start consuming<br>4. Old pods finish current batches<br>5. Old pods gracefully shut down</pre><pre>Downtime: Zero<br>Messages dropped: Zero<br>Rebalancing pause: None</pre><p>No coordination. No rebalancing protocol. Just math. Each pod independently calculated which partitions it owned and started processing. At 50,000 msgs/sec, avoiding even 30 seconds of rebalancing meant not dropping 1.5 million messages into a backlog.</p><h3>Selective Operations</h3><p>Backing up partition 5:</p><pre># Backup only partition 5 (69GB, 135M messages)<br>nats stream backup sensor-partition-5 | gzip &gt; backup-p5.tar.gz</pre><pre># Restore only partition 5<br>gunzip &lt; backup-p5.tar.gz | nats stream restore sensor-partition-5</pre><pre># Other 31 partitions unaffected (still processing 48,438 msgs/sec)</pre><p>With Kafka, backing up one partition meant backing up the entire topic. With independent NATS streams, I could operate on individual partitions surgically.</p><h3>The Mental Model Shift</h3><p>The hardest part wasn’t the technical implementation — it was the mental shift.</p><p><strong>Before</strong>: “Streams are like topics. Create one stream, route with subjects.”</p><p><strong>After</strong>: “Streams are like partitions. Create as many as you need, route with subjects to the right stream.”</p><p>Once I internalized that <strong>NATS stream = Kafka partition</strong>, everything made sense.</p><p>The key difference: Kafka forces you to group partitions under topics. NATS lets you use partitions (streams) as independent entities. Same concept, different organizational philosophy.</p><h3>Addressing the “More Streams” Concern</h3><p>The most common reaction when I explain this: “But managing 32 streams sounds complicated!”</p><p>It would be, if done manually. But with infrastructure as code:</p><pre>// Terraform or similar<br>for partition := 0; partition &lt; 32; partition++ {<br>    createStream(fmt.Sprintf(&quot;sensor-partition-%d&quot;, partition))<br>}</pre><pre>// Result: 32 streams created in 30 seconds</pre><p>Monitoring:</p><pre>// Check all streams programmatically<br>for partition := 0; partition &lt; 32; partition++ {<br>    info := js.StreamInfo(fmt.Sprintf(&quot;sensor-partition-%d&quot;, partition))<br>    <br>    // Alert if any partition has &gt; 1 hour of backlog<br>    if info.State.Messages &gt; 5625000 {  // 1 hour at 1,562/sec<br>        alert(&quot;Partition %d backing up&quot;, partition)<br>    }<br>}</pre><p>The operational complexity is frontloaded into automation, then becomes hands-off. After the initial setup, managing 32 streams was easier than managing 1 stream processing 50,000 messages per second with complex filtering logic.</p><h3>The Fan-Out Pattern</h3><p>Both systems support multiple applications consuming the same data, but they work differently. In our system, we have four applications processing the same 50,000 msgs/sec feed:</p><ol><li>MongoDB writer (stores all messages)</li><li>Real-time analytics (aggregations and alerts)</li><li>Cache updater (updates Redis with latest device states)</li><li>Alert processor (checks thresholds and sends notifications)</li></ol><h3>Kafka Consumer Groups</h3><pre>Topic: sensor-readings (32 partitions, 50k msgs/sec)</pre><pre>Consumer Group A (MongoDB writer): 8 consumers<br>  Each handles 4 partitions, 6,250 msgs/sec per consumer</pre><pre>Consumer Group B (Analytics): 4 consumers<br>  Each handles 8 partitions, 12,500 msgs/sec per consumer</pre><pre>Consumer Group C (Cache): 2 consumers<br>  Each handles 16 partitions, 25,000 msgs/sec per consumer</pre><pre>Consumer Group D (Alerts): 2 consumers<br>  Each handles 16 partitions, 25,000 msgs/sec per consumer</pre><pre>Each group maintains independent offsets<br>Scaling Group A doesn&#39;t affect Group B<br>But scaling within a group triggers rebalancing</pre><h3>NATS Independent Consumers</h3><pre>Stream: sensor-partition-5 (1,562 msgs/sec)</pre><pre>Consumer A: &quot;mongodb-writer-partition-5&quot;<br>Consumer B: &quot;analytics-partition-5&quot;<br>Consumer C: &quot;cache-partition-5&quot;<br>Consumer D: &quot;alerts-partition-5&quot;</pre><pre>Each consumer has unique durable name<br>Each maintains independent position<br>Each processes all 1,562 msgs/sec from this partition<br>Scaling any consumer: no rebalancing, no coordination</pre><p>With NATS, I have 32 streams × 4 applications = 128 total consumers. Each consumer scales independently. There’s no inter-consumer coordination at all. When the analytics team needed to scale from 4 pods to 8 pods to handle increased computation, they just updated their deployment. No coordination with the MongoDB writer team, no system-wide rebalance, no impact on the other 50,000 messages per second flowing through.</p><h3>When Each Approach Shines</h3><p>After six months running both systems in production at 50,000 messages per second, I’ve learned where each excels.</p><h3>Kafka’s Strengths</h3><p><strong>Ecosystem maturity</strong>: Kafka Connect, Schema Registry, ksqlDB, and extensive integrations make complex pipelines straightforward. If you need exactly-once semantics with transactions or integration with dozens of data sources, Kafka’s ecosystem is unmatched.</p><p><strong>Operational knowledge</strong>: Many teams already know Kafka. The patterns are well-documented, and finding experienced engineers is easier. When you’re processing 50,000 messages per second, having team members who’ve operated Kafka at scale is valuable.</p><p><strong>Specific features</strong>: Log compaction (keeping only the latest value per key) and tiered storage to object stores are Kafka-specific features that may be critical for certain use cases.</p><h3>NATS JetStream’s Strengths</h3><p><strong>Performance at scale</strong>: For high-throughput scenarios requiring fast replay and catchup, NATS’s sequential I/O patterns and zero filter overhead provide significant advantages. The 100× improvement in replay time (4.5 minutes vs 7.5 hours) is material when you’re debugging production issues at 50,000 msgs/sec.</p><p><strong>Operational simplicity</strong>: Smaller teams or projects without dedicated platform engineers benefit from NATS’s simpler operational model. Infrastructure as code handles complexity, and the absence of rebalancing protocols reduces operational surprises. When processing 50,000 messages per second, every minute of downtime matters.</p><p><strong>Granular control</strong>: When you need per-partition configuration — different retention, replication, or resource allocation — NATS’s independent streams provide flexibility Kafka doesn’t offer within a single topic.</p><p><strong>Resource efficiency</strong>: With 42% lower CPU usage, NATS handles the same 50,000 msgs/sec load on smaller hardware footprints. Over time, this translates to meaningful cost savings.</p><h3>What I Wish I’d Known Earlier</h3><p>If I were starting this project today, here’s what I’d do differently:</p><p><strong>Start with the partition model</strong>: Don’t try to replicate Kafka’s topic structure in NATS. Think in terms of partitions (streams) from day one. At 50,000 msgs/sec, the performance difference matters.</p><p><strong>Automate from the start</strong>: Write infrastructure-as-code for stream creation, monitoring, and backup before creating streams manually. When you’re processing 4.32 billion messages per day, the complexity is manageable with automation.</p><p><strong>Measure disk I/O patterns</strong>: The performance difference between random and sequential I/O is dramatic. Understanding this explains many of the performance characteristics, especially at high throughput.</p><p><strong>Plan for scale</strong>: If you might eventually need fine-grained isolation per device (all 250,000 devices), choose the system designed for that. NATS scales to tens of thousands of streams more naturally than Kafka scales to tens of thousands of partitions.</p><h3>The Takeaway</h3><p>NATS JetStream’s “one stream per partition” pattern isn’t a novel architecture — it’s exactly how Kafka partitions work, just exposed more directly. Kafka wraps partitions in topics; NATS lets you use partitions (streams) as independent entities.</p><p>This independence provides:</p><ul><li>Better isolation (one partition’s issues don’t cascade to the other 48,438 msgs/sec)</li><li>Better performance (sequential I/O, no filter overhead)</li><li>Better operational control (per-partition configuration)</li><li>Better scalability (can scale to more partitions without coordination overhead)</li></ul><p>Neither system is universally better. Kafka’s maturity and ecosystem make it the right choice for many projects. NATS’s performance and simplicity make it compelling for others, especially at high throughput where efficiency compounds.</p><p>For me, understanding that <strong>a NATS stream is functionally equivalent to a Kafka partition</strong> was the breakthrough. Once I stopped thinking of streams as “bigger” things that should contain many partitions, and started thinking of them as partitions themselves, the architecture became clear.</p><p>Your system’s needs will determine which approach fits better. But understanding how both systems solve the same fundamental problem — ordered, partitioned message storage at scale — helps make that choice informed rather than habitual. When you’re processing 50,000 messages per second, 4.32 billion messages per day, those informed choices matter.</p><p><em>This article reflects my personal experience building IoT data pipelines handling 250,000 devices and 50,000 messages per second over six months. Your specific requirements, scale, and operational constraints will guide your own architectural decisions.</em></p><p>#kafka #nats #jetstream #livetracking #iot</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e849ee86a6a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka vs NATS JetStream: A Deep Dive for IoT Fleet Tracking]]></title>
            <link>https://medium.com/@me_sameen/kafka-vs-nats-jetstream-a-deep-dive-for-iot-fleet-tracking-1dff851be263?source=rss-a3cc8841f580------2</link>
            <guid isPermaLink="false">https://medium.com/p/1dff851be263</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[nats-streaming]]></category>
            <dc:creator><![CDATA[Sandeep]]></dc:creator>
            <pubDate>Sat, 21 Feb 2026 07:28:54 GMT</pubDate>
            <atom:updated>2026-02-21T07:28:54.401Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>🚗 Your GPS tracker just transmitted location data. Now what?</strong></p><p>That innocent telemetry point is about to reveal why choosing between Kafka and NATS JetStream might be the most consequential decision you make in your IoT fleet tracking architecture. I’ve been there. Both technologies. Different outcomes. Let me walk you through what I learned.</p><h3>The Quick Take (Read This First)</h3><p><strong>NATS is operationally simpler and lower latency. Pick it if you’re starting small and all teams can agree on the same patterns.</strong></p><p><strong>Kafka is operationally heavier but gives you something NATS doesn’t: true publisher-consumer independence. Pick it if you have (or will have) multiple teams consuming the same data differently.</strong></p><p>Here’s why that matters.</p><h3>The Pain Points (Where NATS Hits Its Walls)</h3><h3>Pain #1: Consumer Coupling</h3><p><strong>The scenario:</strong> You have a fleet telemetry stream. Operations needs it. Analytics needs it. Compliance needs it. Different processing logic, different speeds, different patterns.</p><p>With <strong>Kafka</strong>: Create 3 consumer groups on the same topic. They’re completely independent. One consumer group can crash, fall behind, or change implementation without affecting others.</p><p>With <strong>NATS</strong>: All consumers reading from the same stream must follow the same patterns and optimizations. Or… you create separate streams with bridge services republishing data. Now you have data duplication, bridge service overhead, sync complexity.</p><p><strong>This violates the core principle of event-based architecture: publishers and consumers should be independent.</strong></p><h3>Pain #2: Ordering at Scale</h3><p><strong>The scenario:</strong> You need ordered processing per vehicle, not globally. Vehicles 1–3 can process in parallel.</p><p>With <strong>NATS</strong>: Embed partition-{hash(device_id)} in your subject. Publish to telemetry.partition-{hash}.vehicle.{id}. Consumers subscribe to specific partitions. This works and ordering is preserved.</p><p><strong>But here’s the catch:</strong> This is publisher-side responsibility, not broker-managed. If a new consumer type needs different partitioning logic, the publisher must change. Consumers and publishers are now coupled through subject naming conventions.</p><p>With <strong>Kafka</strong>: Partitions are broker-managed. Publisher sends to the topic with device_id as the key. Broker automatically routes to partition based on hash. Consumers subscribe to topics, not partitions. If a consumer needs different parallelism, they adjust their group configuration, not subject patterns.</p><p><strong>Publishers don’t know (or care) how many consumers will read the data or what their partitioning needs are.</strong></p><h3>Pain #3: 100K+ msg/sec Throughput</h3><p><strong>The scenario:</strong> Your fleet grows to 100K+ vehicles. Each sending telemetry at 100K msg/sec aggregate.</p><p>With <strong>NATS</strong>: Single stream becomes a bottleneck. The ack mechanism for multiple consumers serializes through one coordinator. Disk I/O coordination hits a ceiling. If you need separate streams for different consumer types, you’re duplicating data across streams, adding bridge services, increasing operational overhead.</p><p>With <strong>Kafka</strong>: Partitions distribute the load. Multiple consumer groups read from partitions in parallel without coordination. Adding a new consumer group adds zero overhead to the broker.</p><h3>Why This Matters (The Architecture Principle)</h3><p><strong>Event-driven architecture has one core principle: publishers publish to a topic/stream, consumers consume from it. Neither should know about the other.</strong></p><p>NATS breaks this at scale because:</p><ul><li>Partition naming (subject structure) couples publishers to consumers</li><li>Ordering solutions require publisher-side coordination</li><li>Multiple consumer types force data duplication</li></ul><p>Kafka preserves this because:</p><ul><li>Partitions are invisible to consumers (topic abstraction)</li><li>Ordering is automatic (key-based hashing is broker-managed)</li><li>Multiple consumer groups are native, not a workaround</li></ul><h3>Want to Go Deeper? Keep Reading…</h3><h3>The Fleet Tracking Verdict (Choose Based on Your Reality)</h3><p><strong>Start with NATS if:</strong></p><ul><li>Fleet size &lt; 10,000 vehicles</li><li>Single consumer or all can standardize on same subject/partitioning patterns</li><li>Operations simplicity &gt; architectural flexibility</li><li>Low latency is critical</li></ul><p><strong>Start with Kafka if:</strong></p><ul><li>Fleet size &gt; 50,000 vehicles</li><li>Multiple teams need independent implementations</li><li>You anticipate consumer flexibility needs later</li><li>50K+ msg/sec throughput expected</li></ul><h3>The Three Walls You’ll Hit (Deep Dive)</h3><h3>Wall #1: Consumer Flexibility (The Real Cost)</h3><p>Real scenario: Operations team, Analytics team, Compliance team all need the same data.</p><p><strong>NATS forces one of two choices:</strong></p><p><strong>Option A: Same Patterns</strong> All three teams must use the same subject structure: telemetry.partition-{hash(device_id)}.vehicle.{id}</p><p>All three must follow the same consumer pattern. Want different parallelism levels? Tough. Want to process at different speeds? Limited.</p><p><strong>Option B: Separate Streams with Bridges</strong></p><pre>Stream: TELEMETRY (original)<br>  ↓ Bridge Service<br>Stream: TELEMETRY_OPS (republished)<br>Stream: TELEMETRY_ANALYTICS (republished)<br>Stream: TELEMETRY_COMPLIANCE (republished)</pre><p>Now each team owns their stream. But:</p><ul><li>Data duplicated across 3 streams = 3x storage</li><li>Bridge services add 10–15% CPU overhead</li><li>Latency: 5–50ms between original and team streams</li><li>Sync issues: What if a bridge crashes?</li></ul><p><strong>Kafka solution:</strong></p><pre>Topic: vehicle-telemetry</pre><pre>Consumer Group: operations-team (independent)<br>Consumer Group: analytics-team (independent)<br>Consumer Group: compliance-team (independent)</pre><p>All read the same topic independently. Different implementations, different speeds, no coordination.</p><h3>Wall #2: Ordering at Publisher-Broker Boundary</h3><p><strong>NATS ordering:</strong></p><pre>// Publisher embeds partition logic<br>partition := hash(deviceId) % numPartitions<br>subject := fmt.Sprintf(&quot;telemetry.partition-%d.vehicle.%s&quot;, partition, deviceId)<br>nc.Publish(subject, data)</pre><pre>// Consumer<br>nc.Subscribe(&quot;telemetry.partition-0.&gt;&quot;, handler)<br>nc.Subscribe(&quot;telemetry.partition-1.&gt;&quot;, handler)<br>// ... subscribe to all partitions, coordinate offset tracking manually</pre><p>Publisher decides partitioning. If a new consumer type wants different partitioning (maybe hash(deviceId + region) for geo-aware processing), the publisher changes, or you add another bridge.</p><p><strong>Kafka ordering:</strong></p><pre>// Publisher just publishes (no partition knowledge)<br>producer.SendMessage(&amp;sarama.ProducerMessage{<br>    Topic: &quot;vehicle-telemetry&quot;,<br>    Key:   sarama.StringEncoder(deviceId),<br>    Value: sarama.StringEncoder(data),<br>})</pre><pre>// Consumer<br>consumer.Consume([]string{&quot;vehicle-telemetry&quot;}, handler)</pre><p>Publisher knows nothing about partitioning. Broker hashes the key. Consumers adjust group configuration if needed. Same data, multiple consumption patterns, zero coupling.</p><h3>Wall #3: 100K+ msg/sec (Disk I/O Bottleneck)</h3><p>At 100K msg/sec with disk-based persistence:</p><p><strong>NATS single stream:</strong></p><ul><li>100,000 msg/sec × 1 KB = 100 MB/sec write load</li><li>Single write queue to disk = bottleneck</li><li>Multiple consumers with different ack patterns = random disk reads</li><li>CPU spent coordinating acks through one stream</li></ul><p><strong>NATS multiple streams (for consumer flexibility):</strong></p><ul><li>100,000 msg/sec × 24 hours = 8.64 TB per stream</li><li>3 streams = 25.92 TB total disk</li><li>Bridge services overhead: 10–15% CPU</li><li>Network bandwidth: duplicated</li></ul><p><strong>Kafka with partitions:</strong></p><ul><li>100,000 msg/sec distributed across partitions</li><li>I/O load striped across partition leaders</li><li>Multiple consumer groups: zero overhead</li><li>Linear scaling with partition count</li></ul><p><strong>Real numbers:</strong></p><p>Scenario Disk CPU Overhead Coordination NATS single stream 8.6 TB 0% <strong>HIGH</strong> (ack bottleneck) NATS multiple streams 25.9 TB 10–15% Medium (bridge overhead) Kafka with partitions 8–10 TB 0% <strong>LOW</strong> (distributed)</p><h3>The Honest Assessment</h3><p><strong>NATS is genuinely good.</strong> It’s operationally simpler, lower latency, just works out of the box.</p><p>But it breaks the fundamental principle of event-driven architecture: <strong>publisher-consumer independence.</strong></p><ul><li>Ordering requires publishers to embed partition logic in subjects</li><li>Consumer flexibility requires data duplication and bridge services</li><li>High throughput hits ack mechanism bottleneck on single stream</li></ul><p><strong>This isn’t a limitation of NATS’s engineering. It’s a design choice.</strong> NATS was built for low-latency microservices with simple pub-sub patterns, not distributed event streaming at organizational scale.</p><p><strong>Kafka was purpose-built for the problem you’re facing:</strong> multiple teams, different consumption patterns, high throughput, true independence between publishers and consumers.</p><h3>The Progression (What Usually Happens)</h3><p><strong>Fleet: 5,000 vehicles</strong> → NATS works great. Simple setup. Low operational overhead.</p><p><strong>Fleet: 10,000 vehicles, Operations + Analytics both consuming</strong> → Add another stream with a bridge. Still manageable.</p><p><strong>Fleet: 50,000 vehicles, three teams, complex ordering needs</strong> → Bridges are adding CPU. Subject naming conventions are becoming complex. Ack coordination is visible as latency.</p><p><strong>Fleet: 100,000+ vehicles, multiple consumer types evolving</strong> → You’re operating parallel streams, bridge services, dealing with sync complexity. At this point, the Kafka migration discussion begins.</p><p><strong>The mistake:</strong> Not realizing sooner that event-driven architecture demands true publisher-consumer independence, and NATS doesn’t provide it at scale.</p><h3>The Bottom Line</h3><p><strong>NATS:</strong> Operationally simple, low latency, works until you scale or add organizational complexity.</p><p><strong>Kafka:</strong> Operationally heavier, but preserves the core principle of event-driven architecture: publishers and consumers are completely independent.</p><p>Choose NATS if you’re confident you’ll stay small and aligned. Choose Kafka if you think you’ll grow and need flexibility.</p><p>Or better yet: Start with NATS. But understand where the walls are. When you hit them, you’ll know exactly why.</p><h3>Questions for You</h3><ul><li>Are you wrestling with this choice right now?</li><li>Have you hit any of these walls?</li><li>What did you choose, and would you do it differently?</li></ul><p>Drop a comment. I’d love to hear your perspective.</p><p><strong>#Kafka #NATS #IoT #FleetTracking #SystemsArchitecture #MessageQueuing #Telematics #DistributedSystems #EventDriven</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1dff851be263" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>