<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Andree Toonk on Medium]]></title>
        <description><![CDATA[Stories by Andree Toonk on Medium]]></description>
        <link>https://medium.com/@atoonk?source=rss-7e2fc845971d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6sexQZrXDULG0pdCfW72ig.png</url>
            <title>Stories by Andree Toonk on Medium</title>
            <link>https://medium.com/@atoonk?source=rss-7e2fc845971d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:49:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@atoonk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Pushing packets on the biggest network optimized EC2 instances]]></title>
            <link>https://atoonk.medium.com/pushing-packets-on-the-biggest-network-optimized-ec2-instances-90f223ac4f01?source=rss-7e2fc845971d------2</link>
            <guid isPermaLink="false">https://medium.com/p/90f223ac4f01</guid>
            <category><![CDATA[dpdk]]></category>
            <category><![CDATA[computer-networking]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[xdp]]></category>
            <dc:creator><![CDATA[Andree Toonk]]></dc:creator>
            <pubDate>Sun, 13 Sep 2026 18:45:40 GMT</pubDate>
            <atom:updated>2026-09-13T18:45:40.652Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>This article was first published on my </strong><a href="https://toonk.io/how-fast-can-you-actually-go-on-aws-pushing-packets-on-the-biggest-network-optimized-ec2-instances/index.html"><strong>personal blog here.</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rb7O8ASUjOJDKFLQ-7ZLtw.png" /></figure><p>If you’ve been following along, you know I’ve spent the last few months making packets go fast in Go. We looked at AF_XDP, DPDK, and most recently Mellanox Direct Verbs, all behind one library, <a href="https://github.com/atoonk/packetio">packetio</a>. On real hardware that ends the way you’d hope: 148 million packets per second, line rate on 100G, from a handful of Go workers.</p><p>This weekend I wondered how much of that survives on EC2. I already knew the DPDK backend worked on Amazon’s ENA network interfaces, but I’d only ever tried it on small instances. Meanwhile, AWS sells network-optimized instances with numbers like 300 and 600 Gb/s on the spec sheet. So my question for today: how well do those machines actually hold up when you point a packet generator at them?</p><p>Two things I wanted out of this:</p><ol><li>Take <a href="https://github.com/atoonk/packetio">packetio</a> for a ride on some seriously beefy machines.</li><li>And this is the bigger one: AWS publishes bandwidth for every instance type, but publishes nothing about packets per second. If you do anything with networking in software, you know packets per second is the number that matters. So let’s go measure it.</li></ol><h3>The instances</h3><p>I went up the c8gn ladder, Amazon’s current Graviton network-optimized family, from the 100G size all the way to the 600G one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*Wqk-u_vPqy7c_pJuVxHb1A.png" /></figure><p>These are not cheap machines, but the specs are pretty impressive! The 48xlarge pair costs about $23 an hour, so every test is designed to be short and to get the answer in one go.</p><h3>How I tested</h3><p>The setup was identical for every instance type:</p><ul><li><strong>A pair of identical instances</strong>, same VPC, same subnet, same availability zone, talking to each other over a dedicated ENA NIC. Management traffic goes over a separate interface so nothing but test traffic touches the counters we care about.</li><li><strong>DPDK on both ends.</strong> The data interface is bound to vfio-pci and owned by a DPDK application; the kernel never sees a packet. I went with DPDK rather than the kernel or AF_XDP because I wanted the NIC to be the only variable.</li><li><strong>packetio’s stock examples.</strong> The sender is blast, the receiver is drop, the same programs from the <a href="https://github.com/atoonk/packetio">packetio</a> repo that I used for the hardware numbers. No special EC2 build. Both are Go programs using the DPDK backend, so what you see below is what a Go application would see.</li><li><strong>Three packet sizes.</strong> 64 bytes for the packet rate, 1500 bytes for what a normal MTU gets you, and 9000 bytes to validate the published bandwidth.</li><li><strong>A queue ladder.</strong> For every instance I started with one queue on one core and doubled up until the number stopped moving. That’s how you tell a per-core limit from an instance limit.</li><li><strong>Read the results at the receiver.</strong> The sender’s transmit counter is not what gets delivered, as AWS’s shaper sits after it. On one instance the sender happily reported 3.6 Mpps while 1.0 Mpps arrived. Every number here is what the other side actually received.</li></ul><p>Two things kept me honest. Every transmit test was repeated with a ~120 line plain C DPDK sender that shares nothing with <a href="https://github.com/atoonk/packetio">packetio</a>, and the two always landed within a few percent of each other. And I kept a close eye on the ENA driver’s allowance counters, pps_allowance_exceeded, bw_in_allowance_exceeded and friends, which the Nitro card exposes to tell you <em>which</em> limit you just hit. This gives me confidence the numbers aren&#39;t implementation specific. I added support for reading those from a DPDK-owned interface to packetio along the way, because without them you&#39;re guessing.</p><h3>Packets per second is everything</h3><p>The thing about networking in software: every packet costs roughly the same amount of work regardless of how big it is: a lookup, a header rewrite, a descriptor. A router, a firewall, a VPN gateway, a load balancer, all of them are packet per second machines first and bandwidth machines second. Bandwidth is just packets times size. So when a spec sheet says 100 Gbps, the question I actually need answered is: at what packet size?</p><p>At 64 bytes, 100 Gbps is 148.8 million packets per second. That’s what the ConnectX-6 in my lab does, and it’s our benchmark.</p><h3>One queue, one core</h3><p>Let’s start small. One transmit queue, driven by one core, 64 byte packets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ORDDByxsI71GXN98z_iSg.png" /></figure><p>About 1.5 to 2.2 million packets per second per core. On the ConnectX-6 in my lab one core does around 60 million. That’s a 30x gap, and it’s not packetio: the plain C sender lands in the same place. Bigger batches, deeper rings, prebuilt frames, checksum offload: I tried them all and none of it moves the number.</p><p>I don’t have a definitive answer for why, but I have a suspect. ENA runs its transmit queues in what the driver docs call <a href="https://www.kernel.org/doc/html/latest/networking/device_drivers/ethernet/amazon/ena.html">Low Latency Queue mode</a>: instead of the NIC fetching your packet from memory, the driver pushes the descriptor and the first 96 bytes of every packet straight into the device over PCIe. For a 64-byte packet that’s the whole thing, written by the CPU, one packet at a time. At 2.8 GHz, 2 Mpps works out to roughly 1,450 cycles per packet, against about 50 on the Mellanox card, and that’s the kind of gap a PCIe write per packet would explain. You can’t test the theory by turning it off: with enable_llq=0 the queue refuses to start, and <a href="https://doc.dpdk.org/guides-23.11/nics/ena.html">DPDK&#39;s own docs</a> say disabling it is &quot;highly not recommended&quot; anyway.</p><p>So on EC2, a core buys you about two million packets per second. The interesting question is what happens when you add more of them.</p><h3>Adding queues</h3><p>Same test, now with N transmit queues on N cores, receiver side:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UMA_r0W_hrIenJtVJr_AeA.png" /></figure><p>It scales linearly, about 1.6 to 2.2 Mpps per queue, right up until it doesn’t. The 8xlarge peaks at 16 queues and 21 Mpps and then actually gets <em>worse</em> with more. And the 12xlarge, 16xlarge and 24xlarge all stop at the same place: <strong>36.6 million packets per second</strong>, at 24 queues, flat as a table from there to 64. Adding 40 more cores changes nothing. That’s not a CPU limit, that’s a wall, and the ENA counters confirm it: push the offer past ~36 M and pps_allowance_exceeded starts ticking on the sender.</p><p><strong>The takeaway: on a c8gn.12xlarge or anything bigger, one network interface tops out at roughly 36 million packets per second.</strong> It doesn’t matter how many queues you use, how many cores you throw at it, or whether the spec sheet says 150, 300 or 600 Gbps. 36M pps, that’s the number I was looking for!</p><h3>The results</h3><p>Here’s the complete table I wish AWS would publish. Everything measured at the receiver, 64 byte packets for the packet rate, 1500 and 9000 byte packets for bandwidth:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5UcF7Ao9pSfMQL1ZcAe8-A.png" /></figure><p>A few things jump out.<br><strong>The bandwidth numbers are honest.</strong> With 9000 byte packets every instance hit its published number, usually a few percent over. If you move big packets, you get what you paid for.</p><p><strong>Packets per second are a different story.</strong> 36.6 million packets per second is about a quarter of what a single 100G NIC does in my lab, on an instance sold as 150, 200 or 300 Gbps. At 64 bytes you’re only using between 8 and 16 percent of the bandwidth you’re paying for. That’s not a complaint, it’s how the platform works, but it is the number to design around, and it’s not on any spec sheet.</p><p><strong>There’s one packet rate from the 12xlarge up.</strong> 150G, 200G, 300G, one network card of the 600G: all ~36 Mpps. The extra bandwidth and the extra cores buy you nothing for small packets. Per gigabit that’s 244 kpps on the 12xlarge, 183 on the 16xlarge and 122 on the 24xlarge. If your workload is small packets, the 12xlarge is the sweet spot of this family, and the bigger sizes only make sense when the packets get big too.</p><p><strong>Where does bandwidth take over?</strong> Since the packet rate is flat, the bandwidth you actually get is simply the packet rate times the packet size, until you hit the published number. I ran a size ladder to find where that happens. The 8xlarge fills its 100G at about 640 bytes. The 12xlarge fills 150G already at 512 bytes. The 16xlarge needs about 700 bytes. The 24xlarge doesn’t get there at 1500 bytes at all: 282 Gbps, and that’s the sender’s NIC topping out, not AWS. Same for the 48xlarge. Above 200G, you need jumbo frames to see the bandwidth you’re paying for.</p><h3>600 Gbps needs two network cards</h3><p>The 48xlarge was the one I was most curious about, and it came in at exactly the same numbers as the 24xlarge: 35 Mpps, 282 Gbps at 1500 bytes, 304 Gbps with jumbo frames. Half the published bandwidth, and this time with bw_out_allowance_exceeded firing on the sender. AWS was shaping me at 300G on a 600G instance.</p><p>The reason: to get 600 Gbps on the 48xlarge you need to use <strong>two network cards</strong>, each with its own 300 Gbps allowance. Normally the published bandwidth is per instance and every ENI shares it; here it’s per card. To get 600 you need two interfaces, one attached to each card (--network-card-index 1 for the second), and something spreading traffic over both. There&#39;s no bonding layer doing that for you. Attach one interface, run iperf, and you&#39;ll measure a 300G machine.</p><p>In hindsight this makes sense. The fastest NICs you can commonly buy today are 400G; anything beyond that is two of them. It would have been odd for AWS to have a single 600G device.</p><p>It is documented, sort of. Someone on Twitter kindly pointed me at the <a href="https://docs.aws.amazon.com/ec2/latest/instancetypes/co.html#co_network">network specifications table for compute optimized instances</a>, which has a “Network cards” column, and the c8gn.48xlarge says 2. Not a page you naturally end up on, and nothing there says one interface gets half.</p><p>It probably also means the packet rate in the table is per card, so both cards together would be good for ~70 Mpps. I didn’t test that (the pair was expensive enough for one weekend), so treat it as a well-founded guess.</p><h3>What I didn’t test</h3><p>AWS documents a burst and baseline mechanism for bandwidth on the smaller instance sizes. The c8gn sizes I tested all have a fixed bandwidth, and my runs were short by design, a few minutes each, so I have nothing to say about credit buckets here. I did keep the allowance counters in view the whole time and saw no sign of a packet-rate bucket in any run. If you run these instances at full blast for an hour, tell me what you see.</p><h3>Things fixed along the way</h3><p>As usual, running on new hardware finds bugs. <a href="https://github.com/atoonk/packetio">packetio</a>’s DPDK backend now builds on arm64 (all of these are Graviton machines, and the mempool header alignment differs from x86), it can read the ENA allowance counters through DPDK’s xstats, and the CPU pinning code learned about hyperthreads: on an Intel instance it used to place one worker per physical core and leave half the vCPUs idle. All of that is in the repo.</p><h3>Wrap-up</h3><p>So, how fast can you go on AWS?</p><ul><li><strong>Bandwidth: as fast as it says on the box.</strong> Every published number held, as long as the packets are big enough, and above 200G that means jumbo frames. On the 48xlarge it also means two network interfaces.</li><li><strong>Packets per second: about 36 million per network interface</strong>, from the 12xlarge up. That’s the number AWS doesn’t publish, and it’s about a quarter of what one 100G NIC does on bare metal.</li><li><strong>Per core: about 2 million pps.</strong> Roughly 30x more expensive per packet than a ConnectX. You need 24 cores just to push enough packets to find the instance limit.</li></ul><p>So we got the numbers we were after, and <a href="https://github.com/atoonk/packetio">packetio</a> got better along the way. If you’re building network functions on EC2, you now have the figure that was missing, and it’s the one to pick your instance by: know your packet size first. For small packets the budget is one flat number across a 4x range of prices, and a single 100G ConnectX in a bare-metal box does four times the packets of a c8gn.24xlarge for a lot less money.</p><p>One more catch, and it has nothing to do with packets. Everything here ran inside one availability zone, where traffic is free. Cross an AZ and it’s $0.01 per GB each direction. 300 Gbps is 135 TB an hour, so a 48xlarge at its rated bandwidth costs about $1,350 an hour in transfer, on top of $11 for the instance. Serious networking on AWS needs a serious credit card, and it’s the traffic that runs it up. That’s a post of its own.</p><p>The tooling is in the <a href="https://github.com/atoonk/packetio">packetio repo</a>; the blast and drop examples are all you need to reproduce any row in the table. If you run this on an instance family I didn&#39;t, I&#39;d love to see the numbers.</p><p>That’s it, thanks for reading! Pushing packets on AWS is now a little clearer :)</p><p>Cheers<br>-Andree</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90f223ac4f01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meet PacketIO — Four ways to do super fast packet processing in Go]]></title>
            <link>https://atoonk.medium.com/meet-packetio-four-ways-to-do-super-fast-packet-processing-in-go-1fdfefcfd9e0?source=rss-7e2fc845971d------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fdfefcfd9e0</guid>
            <category><![CDATA[xdp]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[mellanox]]></category>
            <category><![CDATA[dpdk]]></category>
            <dc:creator><![CDATA[Andree Toonk]]></dc:creator>
            <pubDate>Tue, 08 Sep 2026 03:08:00 GMT</pubDate>
            <atom:updated>2026-09-08T14:37:55.424Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong><em>Meet </em></strong><a href="https://github.com/atoonk/packetio"><strong><em>packetio</em></strong></a><strong><em>, the Go packet I/O library I wish I’d had a years ago. AFXDP, DPDK, mlx5 Direct Verbs, and AFPACKET behind one API.</em></strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X22Xsp8j90WpcqnGvBRSmA.png" /></figure><p>The original version of this was published on my personal blog here: <a href="https://toonk.io/packetio/">https://toonk.io/packetio/</a></p><p>The last few months I’ve reignited my strange hobby of doing network packet processing in software. My language of choice is Go, and I’m always looking and learning how to do this as fast as possible.</p><p>A few weeks ago I released <a href="https://github.com/atoonk/go-afxdp">go-afxdp</a>, a Go library for sending and receiving packets using AF_XDP. Then I built Wireblast on top of it and got that to 100G line rate.</p><p>I’ve also been playing quite a bit with DPDK, different NICs, and more recently NVIDIA/Mellanox Direct Verbs.</p><p>It’s been a fun rabbit hole :) But while doing all of this, I kept running into the same thing.<br>The actual packet processing is usually the easy part. Receive some packets, inspect them, maybe change a few bytes, do a lookup, and send them back out.</p><p>Getting packets efficiently between the NIC and your Go program is where all the complexity lives. And depending on how you want to do that, you end up in completely different worlds.</p><p>AF_XDP has UMEM and rings. DPDK has mbufs, mempools, EAL, hugepages and PMDs. Mellanox ConnectX cards have yet another path using Direct Verbs. All complicated words most of us probably don’t want to deal with.</p><p>That’s what I’ve been working on. One API to rule them, It’s called <a href="https://github.com/atoonk/packetio"><strong>packetio</strong></a>.</p><h3>Four ways to get at a packet</h3><p>There are basically four ways I care about for getting raw packets in and out of a Go program on Linux.</p><p>There’s <strong>AF_PACKET</strong>, which is the normal Linux packet socket. It’s what tools like tcpdump use. It works almost anywhere and requires very little setup, but it’s slow as packets still go through much of the normal kernel machinery.</p><p>Then there’s <strong>AF_XDP</strong>, which I’ve <a href="https://toonk.io/line-rate-packet-processing-in-go-with-af_xdp/">written</a> about quite a bit already. It lets you intercept packets very early in the Linux driver and move them into memory shared with your application. On supported drivers it can be zero-copy, so the NIC writes packets directly into memory your Go program can access.</p><p>The part I really like about AF_XDP is that Linux still owns the interface. You can send only the traffic you care about to your program and your SSH session, ARP, routing protocols and everything else keep working normally.</p><p>Then there’s <strong>DPDK</strong>.<br>DPDK is still the obvious answer when someone says “I want to do packet processing as fast as possible in userspace.” It supports a large number of NICs and has been optimized for this job for years.</p><p>It’s also a bit of a world of its own. If you’ve ever started a DPDK application from scratch, you know what I mean. Hugepages, EAL arguments, mbufs, mempools, PCI devices, queue setup. None of it is particularly impossible, but there’s a huge learning curve.</p><p>And then there is the fourth option, which is the one I only really started appreciating recently: <strong>mlx5 Direct Verbs</strong>. More on that one in a bit.</p><h3>One API over all of them</h3><p>The idea behind packetio is pretty simple. A device has RX queues and TX queues. Queues give you frames. You process those frames in batches and give them back.<br>That’s about as much as I want the application to know.</p><p>For example:</p><pre>dev, err := mlx5.Open(&quot;eno2&quot;, mlx5.WithQueues(4))</pre><p>Or afxdp:</p><pre>dev, err := afxdp.Open(&quot;eno2&quot;, afxdp.WithQueues(4))</pre><p>Or DPDK:</p><pre>dev, err := dpdk.Open(&quot;0000:c1:00.1&quot;, dpdk.WithQueues(4))</pre><p>There’s also:</p><pre>dev, err := afpacket.Open(&quot;eno2&quot;, afpacket.WithQueues(4))</pre><p>After that, the packet loop is the same. Something like:</p><pre>rx := dev.RxQueue(0)<br>for {<br>    rx.Fill(rx.NumFreeFillSlots())<br>    rx.Poll(time.Second)<br>    descs := rx.Receive(256)<br>    for _, desc := range descs {<br>        pkt := rx.Region().Frame(desc)<br>        // Do something useful with the packet.<br>        process(pkt)<br>    }<br>    rx.Recycle(descs)<br>}</pre><p>And transmitting is similarly boring:</p><pre>tx := dev.TxQueue(0)<br>tx.SendFunc(256, func(_ int, frame []byte) int {<br>    return copy(frame, myPacket)<br>})</pre><p>That’s really all there’s to it! You still have queues and batches because those things matter when you’re doing high-speed packet processing. I’m not trying to hide reality behind some net.Conn abstraction.</p><p>But I also don’t think you should need to understand an AF_XDP fill ring, an rte_mbuf, or an mlx5 WQE before you can get a packet onto the wire.<br>Changing the backend should mostly mean changing how you open the device.<br>That sounds like a relatively small thing, but in practice I think it&#39;s pretty powerful.</p><h3>DPDK from Go, finally</h3><p>One of the original motivations for this project was actually DPDK.<br>I’ve wanted a good way to use DPDK from Go for years.</p><p>There have been Go wrappers before, but I never found one that gave me the programming model I wanted. I don’t want to write my packet processing in C, and I definitely don’t want to cross from Go into C for every packet.</p><p>packetio crosses into DPDK once per <strong>batch</strong>, not once per packet. The packet memory stays in DPDK’s mempool and Go works directly on those bytes.<br>So you get to write what looks like a pretty normal Go program while DPDK does all the ugly setup underneath.</p><p>No rte_eth_rx_burst scattered through your application. No mbuf ownership rules in your business logic. No giant EAL argument string you copied from Stack Overflow three years ago. And if you don&#39;t know what that means, congrats, you dont need to!</p><p>You open the device and start receiving packets, all in Go.That’s the DPDK experience I’ve wanted for a long time.</p><h3>Direct Verbs was the surprise</h3><p>The most interesting part of this project for me has probably been learning about mlx5 backend and something called direct verbs.</p><p>Jérôme Tollet has been doing some great work in VPP around NVIDIA/Mellanox ConnectX NICs. He recently <a href="https://medium.com/fd-io-vpp/three-vpp-datapaths-for-nvidia-connectx-and-bluefield-309c0124019d">published a comparison of VPP’s three high-performance datapaths</a>: AF_XDP, DPDK mlx5 and native RDMA Direct Verbs.</p><p>That sent me down another rabbit hole. It turns out that with a ConnectX NIC you can talk to the hardware queues directly using mlx5 Direct Verbs.</p><p>So instead of:<br><strong>Go &gt; DPDK &gt; mlx5 PMD &gt; NIC</strong><br>you can do:<br><strong>Go &gt; mlx5 queues &gt; NIC</strong></p><p>There is obviously quite a bit hiding behind that sentence, but from the application’s perspective that’s the useful mental model.</p><p>And it has another property I really like. ConnectX NICs support a bifurcated model. Linux can keep its normal interface while hardware steering sends selected traffic directly into queues owned by your application.</p><p>So eno2 doesn&#39;t disappear. SSH keeps working. Linux can still use the interface for ordinary traffic while your Go program handles the packets you&#39;ve claimed.</p><p>ConnectX can do this with DPDK too, but Direct Verbs lets us skip the DPDK layer entirely. For what I’ve been building lately, that’s a pretty compelling combination.<br>AF_XDP-like operational behavior, but with a very short userspace datapath.</p><h3>How fast is it?</h3><p>Yes, the big question! I benchmarked all of this! I actually spent quite a bit of time on these numbers, so I’m going to put them all here.</p><p>Hopefully they’re useful to someone else comparing these different approaches. And if nothing else, I know I’ll be coming back to this blog myself six months from now trying to remember how DPDK compared to AF_XDP :)</p><p>The setup is two AMD EPYC 9275F machines with NVIDIA ConnectX-6 Dx 100G NICs. I’m testing 64-byte packets, which is really the worst case for packet processing. At that size, 100G Ethernet is about <strong>148.8 million packets per second</strong>.</p><p>That’s one packet every 6.7 nanoseconds. That’s very little time. So yeah, things like memory accesses and extra trips through the kernel start to matter.</p><p>The nice thing about this new packetio library is that all four backends are running essentially the <strong>same userspace benchmark program</strong>. Same receive loop, same transmit loop, same forwarding code. The backend is the only thing that changes.</p><p>I’m also measuring CPU across the whole machine, not just the Go process. That’s particularly important for AF_XDP. Quite a bit of its work happens inside the kernel in IRQ/NAPI/softirq context. If you only measure your Go process, those cores magically disappear from the benchmark even though you’re still paying for them.<br>Jérôme took the same approach in his VPP measurements, which is one of the things I liked about his comparison.</p><h3>Receive</h3><p>For receive I’m offering <strong>148.6 Mpps across 4096 flows</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rVcT8Frr5_suJgX9RYxZDg.png" /></figure><p>AF_PACKET is probably the most interesting row here, just for the wrong reason. We’re throwing 148.6 million packets per second at the machine. The AF_PACKET program manages to receive only 2.3 million of them, while basically all of the CPU time is disappearing into kernel softirq processing.</p><p>That’s the argument for bypassing the normal kernel packet path in one row, and why you can’t really use it for anything that needs speed.</p><p>AF_XDP is a completely different story. It receives <strong>146.5 Mpps</strong>, so essentially the whole 100G flood, with 11.4 cores of machine CPU.</p><p>But look at the softirq column. <strong>7.5 of those cores are kernel work.</strong><br>I think that’s the really interesting comparison. If you only looked at CPU consumed by the Go program, AF_XDP would look dramatically cheaper than it actually is.</p><p>Direct Verbs and DPDK don’t have that extra kernel datapath, and both receive the full 148.6 Mpps. DPDK does it with eight cores and mlx5 with ten.</p><h3>Transmit</h3><p>For transmit, the machine under test sends as fast as it can from N queues. Every packet carries the same 5-tuple, since varying it only matters when something is receiving.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3oQxoaeDfiJEktmjbLvRvQ.png" /></figure><p>This one still makes me smile. <strong>Three Go workers using DPDK can fill a 100G interface with minimum-sized Ethernet frames.</strong><br>148.8 million packets per seconds, From Go! Pretty amazing.</p><p>The Direct Verbs backend gets there with four cores. And AF_XDP gets essentially there too, at 147.9 Mpps. But look at the CPU cost: 20 cores, with <strong>14 cores worth of softirq</strong>.</p><p>Again, that’s not necessarily bad. It’s the tradeoff. With AF_XDP you keep the normal Linux driver and all the operational convenience that comes with it. DPDK and Direct Verbs have a much shorter path, so they need substantially less CPU to do the same amount of work.</p><h3>Forwarding</h3><p>For this test we receive a packet, do a L3 longest-prefix match, rewrite it and transmit it again.</p><p>So this is closer to the kind of thing I’m usually interested in: routers, NAT, firewalls, load balancers, etc.<br>The offered load is again <strong>148.6 Mpps</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w33x7-nvm4grVFpwwJ42zA.png" /></figure><p>Direct Verbs gets to <strong>147.4 Mpps</strong>, basically 100G line rate, using 12 cores.<br>DPDK is close behind at <strong>143.5 Mpps</strong> using 16 cores.<br>AF_XDP gets to a very respectable <strong>138.4 Mpps</strong>, but needs 28.7 cores of machine CPU to do it, 12.6 of which are softirq.</p><p>And then there’s AF_PACKET :) Under the same flood it forwards essentially nothing, about <strong>0.01 Mpps</strong>, while burning almost 50 cores in the kernel.<br>That’s a pretty dramatic picture of why all these faster packet I/O mechanisms exist in the first place.</p><p>There are dozens of little things like queue counts, CPU placement, RSS configuration, descriptor formats and prefetching that can completely change your result.<br>I’ve spent way too much time figuring those things out. And that’s another thing I want packetio to do for you: put good measured defaults in one place so you can mostly forget about them and get on with whatever you’re actually trying to build.</p><h3>It works on EC2 too</h3><p>I also didn’t want this to become one of those libraries that only works on the one server I happen to be testing on. So I’ve been testing packetio on EC2 as well.</p><p>The DPDK backend works with AWS ENA, and the same application code runs there unchanged. Networking in AWS is its own adventure, because the instances themselves have packet per second limits. Most of the instances I experimented with had either around 1 Mpps or 5 Mpps limits. These are AWS limits, not packetio limits. Generally the bigger the instance, the larger the networking allowance.</p><p>If you’re experimenting with ENA yourself, the counter to keep an eye on is:</p><pre>ethtool -S eth0 | grep pps_allowance_exceeded</pre><p>If pps_allowance_exceeded starts increasing, you&#39;ve hit the EC2 instance&#39;s packet-per-second allowance and AWS is queueing or dropping packets. When using the DPDK ENA driver, the same counter is available through the ENA PMD extended stats.</p><p>I also tested the DPDK backend on a 10G Intel X550. DPDK fills the wire in both directions on a single core. So we’re starting to cover quite a large surface area: bare metal, EC2, ConnectX, Intel, ENA, AF_XDP and DPDK.<br>That’s exactly what I wanted from this library.</p><h3>So which one should you use?</h3><p>well it depends… if I have a ConnectX card, I’m increasingly drawn to Direct Verbs. The performance is excellent and I really like keeping the normal Linux interface around.</p><p>For general modern Linux hardware, AF_XDP is a great place to start. It’s fast, operationally convenient, and has much broader hardware support than the mlx5-specific backend.</p><p>DPDK is still the big hammer. It has a great driver ecosystem and decades of work behind it. There are plenty of environments where it’s exactly the right choice.</p><p>And AF_PACKET is great when you just need to see some packets and don’t care about moving millions of them per second.</p><p>But the thing I’m most excited about with <a href="https://github.com/atoonk/packetio">packetio</a> is that this choice doesn’t have to dominate the design of your program anymore.</p><p>Write your packet processing once. Run it with AF_XDP. Try DPDK.<br>If there’s a ConnectX in the server, try Direct Verbs. Then measure it.<br>Changing how packets reach your Go program should be a configuration decision, not a rewrite. That’s really the whole idea.</p><p>I’ve already moved <a href="https://toonk.io/wireblast-building-a-line-rate-packet-generator-in-go-with-af_xdp/">Wireblast</a> onto packetio, and there are a few other things I&#39;m building on top of it as well.</p><p>If you look carefully :) you’ll even find a userspace TCP/IP stack based on gVisor that can run on top of these backends. I’ve spent waaay too much time trying to make that go fast as well. IMHO its defaults aren’t exactly optimized for this kind of workload..but that’s probably a blog post on its own.</p><p>If nothing else, I now finally have the packet I/O library I’ve wanted for the last few years. Super happy with it and also pretty proud!</p><p>It supports DPDK, which is what I originally really wanted. Along the way I got to go deep on AF_XDP and learned a whole lot about ConnectX NICs and Direct Verbs.</p><p>And it can push <strong>148 million packets per second</strong>. Not bad for Go :)<br>The code is on <a href="https://github.com/atoonk/packetio">GitHub</a>, go give it a try, there’s <a href="https://github.com/atoonk/packetio/tree/main/examples">plenty of exampes</a>. Also see this <a href="https://gist.github.com/atoonk/63ad7ca0c7e4c0e0b167695b311a455e">copy paste ready DPDK example</a>.</p><p>That’s it! Thanks for reading. Let’s push some packets, and let me know what you’re building.</p><p>Cheers,<br>Andree</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fdfefcfd9e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wireblast: a 100Gbs packet generator in Go with AF_XDP]]></title>
            <link>https://levelup.gitconnected.com/wireblast-a-100gbs-packet-generator-in-go-with-af-xdp-fee17eecb2c1?source=rss-7e2fc845971d------2</link>
            <guid isPermaLink="false">https://medium.com/p/fee17eecb2c1</guid>
            <category><![CDATA[xdp]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[af-xdp]]></category>
            <dc:creator><![CDATA[Andree Toonk]]></dc:creator>
            <pubDate>Sat, 08 Aug 2026 16:18:10 GMT</pubDate>
            <atom:updated>2026-08-11T12:49:31.020Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>This post was originally published on my Personal blog at </em></strong><a href="https://toonk.io/wireblast-building-a-line-rate-packet-generator-in-go-with-af_xdp/">https://toonk.io/wireblast-building-a-line-rate-packet-generator-in-go-with-af_xdp/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jeSm_Zew9EBdi7iO8n3Y7w.png" /></figure><p>If you’ve followed my blog for a while, you know I love tinkering with packets and making them go fast in software. Today I’m <a href="https://github.com/atoonk/wireblast/">releasing</a> a new tool called <a href="https://wireblast.mintlify.site/">wireblast</a>, a blazingly fast network packet generator.</p><p>A few weeks ago, I wrote about <a href="https://github.com/atoonk/go-afxdp">go-afxdp</a>, a small Go library I’ve been working on for sending and receiving Ethernet frames at line rate using AF_XDP, without DPDK and without leaving Go.</p><p>I was pretty happy with how it turned out. But a library is just a library; the real value is in how you use it. So the next question is, what can we build with it?</p><p>I have a few things cooking, all about fast packet forwarding on Linux. But to know whether a fast forwarding path is actually fast, you need to throw traffic at it. A lot of traffic! And you need to trust the numbers. So I built a packet generator. It’s called <strong>Wireblast</strong>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F8EmxvaOR5uA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D8EmxvaOR5uA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F8EmxvaOR5uA%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f65b29965e01f4a6bbefe67841aa97c/href">https://medium.com/media/7f65b29965e01f4a6bbefe67841aa97c/href</a></iframe><h3>The gap iperf leaves</h3><p>For network performance testing, most of us reach for iperf3. I do too. It’s easy, widely available, and perfect for answering, “How many gigabits per second can these two hosts push?”</p><p>But bandwidth is only part of the story. Packets per second matter just as much when testing routers, firewalls, load balancers, or anything else that does work for every packet.</p><p>That’s where iperf3 (or really the kernel) hits a wall. Its traffic still passes through the Linux networking stack, which becomes the bottleneck long before the wire does. At 10G, line rate for 64-byte frames is 14.88 Mpps. At 100G, it’s 148.8 Mpps. You cannot reach those numbers with Linux kernel, in fact you may only get to 10% of the 100G capability and small packets sizes.</p><p>For small packets, high packet rates, or lots of flows, you traditionally need something like T-Rex, DPDK Pktgen, or Ixia. Those tools are powerful, but they also bring DPDK, hugepages, dedicated interfaces, extra setup, and sometimes a purchase order. It’s a lot of work and complicated.</p><h3>Meet Wireblast</h3><p><a href="https://wireblast.mintlify.site/">Wireblast</a> is a single, static, 14 MB Go binary. There’s no DPDK and no complicated setup. You download it, run it, and start sending (or receiving) packets.</p><p>Run it without arguments and you get a small interactive wizard. Or provide everything as command-line flags, which makes it easy to use from scripts and automated tests.</p><p>The code is available on <a href="https://github.com/atoonk/wireblast">GitHub</a>, and the documentation is at <a href="https://wireblast.mintlify.site/">wireblast.mintlify.site</a>.</p><p>Under the hood, Wireblast uses AF_XDP to bypass most of the normal kernel networking path when transmitting packets. That makes it possible to reach very high packet rates: 14M pps on 10G hardware and close to line rate, 138M pps packets per second on 100G NICs.</p><h3>Let’s send some packets</h3><p>Grab the latest binary:</p><pre>curl -sSL https://github.com/atoonk/wireblast/releases/latest/download/wireblast_linux_amd64.tar.gz \<br>  | tar xz</pre><pre>sudo install -m 0755 wireblast /usr/local/bin/<br>wireblast --version</pre><p>There’s an arm64 build too, or you can install it using Go:</p><pre>go install github.com/atoonk/wireblast/cmd/wireblast@latest</pre><p>Now let’s start with the hardest thing you can ask a NIC to do: tiny packets.</p><p>Small frames are the classic stress test because you’re bottlenecked on packets per second rather than bits per second.</p><pre>sudo wireblast --start --yes -i eno2 --vlan 2043 \<br>   --src-ip 192.168.43.1 \<br>   --dst-ip 192.168.43.2 \<br>   --dst-mac 7c:c2:55:be:f4:1b \<br>   --packet-size 64 \<br>   --flows 100 \<br>   --pps unlimited -d 30s</pre><p>UDP is the default, so that’s the whole command. Of course you only need the vlan flag when running on tagged vlans.</p><p>On a 10G capable nic, that’s <strong>14.88 million packets per second</strong>: line rate for 10G, from a Go program using only a few CPU cores. I still think that’s kind of amazing.</p><h3>IMIX and PCAP replay</h3><p>Fixed-size packets are great for finding the ceiling, but real traffic contains a mix of packet sizes.</p><p>Wireblast includes the classic IMIX profile: a 7:4:1 ratio of 64-, 594-, and 1518-byte frames.</p><pre>sudo wireblast -i eth1 \<br>  --dst-ip 192.0.2.10 \<br>  --mode imix \<br>  --flows 64 \<br>  --pps 200k \<br>  --start</pre><p>The 64 flows vary the source ports, which is useful when testing anything involving RSS, ECMP, link aggregation, load balancers, or connection tables.</p><p>You can also replay PCAP and PCAPNG files:</p><pre>sudo wireblast -i eth1 \<br>  --pcap capture.pcap \<br>  --pps 100k \<br>  --start</pre><p>By default, Wireblast loops the capture at your requested rate. It can also reproduce the original packet timing:</p><pre>sudo wireblast -i eth1 \<br>  --pcap capture.pcap \<br>  --pcap-timing original \<br>  --pcap-loop=false</pre><p>For automation, --no-tui skips the wizard and dashboard and prints plain-text statistics once per second:</p><pre>sudo wireblast --no-tui \<br>  -i eth1 \<br>  --dst-ip 192.0.2.10 \<br>  --pps unlimited \<br>  -d 30s \<br>  -y</pre><pre>[0:05] tx 74.4 M pkts  14.88 Mpps  L1 10 Gbit/s  L2 7.62 Gbit/s  avg 64B</pre><p>Everything is scriptable, and there isn’t much to learn.</p><h3>Going big: 100G</h3><p>10G is fun, but I had to know how far this thing would go.<br>So I put Wireblast on a pair of machines with 100G Mellanox ConnectX NICs using the mlx5 driver and let it rip. (By the way, I’ve been spending waaay too much money on <a href="https://www.latitude.sh/">Latitude.sh</a> for this project, but I highly recommend them for renting bare-metal machines with a second NIC for network testing.)</p><p>Below is a test with 80 bytes packets. <strong>98Gb/s at 122Mpps! nice!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xf3YjF4jEXHIydua.png" /></figure><p>That’s 99% of tagged line rate for minimum-size frames on a 100G link, from one Go binary. And it wasn’t limited to tiny packets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bhRDh3IRMo-w7I9uTRcozA.png" /></figure><p>The queue-scaling results surprised me even more. With large frames, one queue already pushed 83 Gbit/s. Two queues nearly saturated the link.</p><p>You don’t need a monster machine. You need a decent NIC and a couple of CPU cores, which is a wild thing to be able to say about a software packet generator.</p><h3>Where Wireblast fits</h3><p>Wireblast is meant to cover the easy 80%. It is not trying to replace an Ixia.</p><p>It counts packets and bytes, but it does not measure latency or jitter, and it does not support hardware timestamping. Its TCP mode generates stateless SYN packets rather than complete TCP sessions.</p><p>That makes it useful for testing:</p><ul><li>Packet-forwarding performance</li><li>Firewalls and ACLs</li><li>Connection tables</li><li>RSS and queue distribution</li><li>ECMP Link aggregation</li><li>Load balancers</li><li>Packet-processing software</li></ul><p>When I want to know, “Can this path do line rate, and does the far end see the packets?”, Wireblast answers that question in about 30 seconds.</p><h3>Wrap-up</h3><p>Building Wireblast was fun. More importantly, building it on top of go-afxdp was easy.</p><p>Claude Code also made building the TUI surprisingly quick. That’s a nice thing to be able to say about both the library and the current state of software development.</p><p>For years, software traffic generation meant choosing between an easy tool limited by the normal kernel path and a powerful tool that pulled you into DPDK, out dated howto’s and specialized test setups.</p><p>Wireblast is the middle ground I wanted: download one binary, point it at an interface, and start sending packets at line rate.</p><p>No hugepages. No NIC surgery. No DPDK expertise required. And let’s be real: what’s more fun than sending packets at line rate?</p><p>Fourteen million packets per second on a normal 10G box. More than 100 million <em>packets</em> per second on a 100G NIC. All from one command, with almost no setup required. Pretty amazing!</p><p>Next up, now that I have a reliable traffic generator, I want to get back to the project that started all of this: a completely Go-based dataplane that bypasses the kernel for routing, switching, and firewalling.</p><p>Now I finally have a good way to measure it. Stay tuned!</p><p>The code is on <a href="https://github.com/atoonk/wireblast">GitHub</a>, the documentation is at <a href="https://wireblast.mintlify.site/">wireblast.mintlify.site</a>, and it’s all built on <a href="https://github.com/atoonk/go-afxdp">go-afxdp</a>.</p><p>That’s it :) you made it to the end. Thanks for reading!</p><p>Cheers,<br>Andree</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fee17eecb2c1" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/wireblast-a-100gbs-packet-generator-in-go-with-af-xdp-fee17eecb2c1">Wireblast: a 100Gbs packet generator in Go with AF_XDP</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Line-rate packet processing in Go with AF_XDP]]></title>
            <link>https://levelup.gitconnected.com/line-rate-packet-processing-in-go-with-af-xdp-48f4ad867229?source=rss-7e2fc845971d------2</link>
            <guid isPermaLink="false">https://medium.com/p/48f4ad867229</guid>
            <category><![CDATA[xdp]]></category>
            <category><![CDATA[ebpf]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[networking]]></category>
            <dc:creator><![CDATA[Andree Toonk]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 02:36:29 GMT</pubDate>
            <atom:updated>2026-07-08T16:26:10.153Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dQAhrhofUEkdbFy9" /><figcaption>Photo by <a href="https://unsplash.com/@visualstrance?utm_source=medium&amp;utm_medium=referral">Adrian Yu</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong><em>This post was originally published on my Personal blog at </em></strong><a href="https://toonk.io/line-rate-packet-processing-in-go-with-af_xdp/"><strong><em>https://toonk.io/sending-network-packets-in-go/</em></strong></a></p><p>If you’ve been following my blog for a while, you know I have a soft spot for making packets go fast in software. I’m a network guy at heart, raised on routing and switching, but these days most of that love is written in Go.<br>Every few years I circle back to the same question: how fast can we push and receive packets on a plain Linux box? We’ve explored this with DPDK, with XDP-based routers, and with the kernel itself. Today we’re going a level deeper into my favorite of the bunch: AF_XDP, using a Go library I wrote to make it easy: <a href="https://github.com/atoonk/go-afxdp">go-afxdp</a> .</p><p>If you want to skip ahead and see it in action, the <a href="https://github.com/atoonk/go-afxdp/blob/main/examples/blast/main.go">blast</a> and <a href="https://github.com/atoonk/go-afxdp/tree/main/examples/drop">drop</a> examples in the repo are a great place to start.</p><h3>Some background</h3><p>A while ago I wrote this article: <a href="https://toonk.io/sending-network-packets-in-go/">High-Speed Packet Transmission in Go: from net.Dial to AF_XDP</a>. In that article we worked through every way Linux gives you to send a packet from Go: net.Dial, raw sockets, AF_PACKET, pcap. At the end of that journey sat AF_XDP, and it beat everything else by a mile.</p><p>That was my first hands-on experience with AF_XDP, through a third-party library <a href="https://github.com/asavie/xdp">asavie/xdp</a>. It showed off the potential, but it was quite low-level and, honestly, a bit fragile; some of you who tried it on real hardware ran into panics I couldn&#39;t fix from my side. So, building on that code, I wrote my own library. The goal: keep the raw speed, make it higher level and more reliable, and have it do the right thing for whatever hardware or cloud you&#39;re on, without you needing to understand every knob under the covers.</p><h3>So, what is AF_XDP?</h3><p>AF_XDP is a socket type in Linux, like the ones you already know. The mental model is close to AF_PACKET (the thing tcpdump is built on): instead of asking the kernel to &quot;send this payload to this IP and port,&quot; you hand it complete Ethernet frames, MAC addresses and all, and you receive complete Ethernet frames back. You&#39;re operating at layer 2.</p><p>The difference is how much of the kernel you skip on the way to the wire. AF_PACKET still takes a fairly scenic route through the network stack. AF_XDP takes the express lane: your program and the kernel share a chunk of memory (the UMEM), carved into packet buffers, and packets move through lock-free rings in that shared memory. On a modern NIC and driver it goes one step further with <strong><em>zero-copy</em></strong> mode: the NIC&#39;s DMA engine writes incoming packets directly into your program&#39;s memory and reads outgoing packets straight from it. No copies, no socket buffers, no per-packet syscalls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OF8ht7WslMMlQou7AlJH3Q.png" /><figcaption>13.6 Mpps in Go using AF-XDP</figcaption></figure><p>How fast is that in practice? On a bare-metal machine with a 10G Intel NIC, I measured three methods, all sending small 64-byte packets. Plain kernel UDP sockets, one sendto() per packet, topped out around 3 million packets per second no matter how many workers I threw at it. Adding UDP GSO trickery (UDP_SEGMENT, where the kernel slices one big send into many packets) more than doubled that to almost 8 million. And go-afxdp? ~13.5 million packets per second, which is virtually 10G line rate, with the receiving side keeping up without dropping a packet. At that point the wire is the bottleneck, not the software. I still think that&#39;s kind of amazing for a Go program you can go build like any other.</p><p>These numbers come from <a href="https://github.com/atoonk/go-pktgen">go-pktgen</a>, my packet generator that benchmarks all the send methods side by side; I’ve updated it so its af_xdp mode now uses this new library. Check that repo if you want the full comparison table. Btw, AF_XDP shines even more on the RX path than TX.</p><h3>The best part: you don’t lose your NIC</h3><p>When you open an AF_XDP socket with go-afxdp, the library compiles a small eBPF program and attaches it to the interface as an XDP program. XDP programs run inside the driver, for every incoming packet, before the kernel stack ever sees it (the same technology b<a href="https://toonk.io/building-an-xdp-express-data-path-based-bgp-peering-router/">ehind my XDP-based BGP peering router</a> from a few years back). The program&#39;s job here is simple: if a packet matches the filter you asked for, <em>redirect</em> it into your socket&#39;s ring. That&#39;s it.</p><p>The key word is <em>if</em>. Everything that doesn’t match is handed to the kernel stack, completely unaffected. Say you only want UDP port 9999: those packets land straight in your Go program, while your SSH session, DNS lookups, and everything else on the interface keep flowing like nothing happened. You can be surgical about exactly which traffic takes the fast path. This is exactly where <a href="https://toonk.io/building-a-high-performance-linux-based-traffic-generator-with-dpdk/">DPDK</a> hurts: it takes over the entire NIC, and the interface disappears from the kernel, SSH session and all. With AF_XDP the NIC stays a normal Linux interface.</p><h3>What it looks like in Go</h3><p>This is what receiving packets looks like. Open the interface with a filter, and you get one socket per NIC queue:</p><pre>fleet, err := afxdp.Open(&quot;eth0&quot;, afxdp.WithUDPPorts(9999))<br>if err != nil {<br>	log.Fatal(err)<br>}<br>// Info() tells you what you actually got: queues, XDP mode, zero-copy, driver.<br>// e.g. &quot;eth0: 12 queues, zero-copy, native XDP, 8192x2048B frames, driver ixgbe&quot;<br>info, _ := fleet.Info()<br>log.Printf(&quot;started: %s&quot;, info)<br>// One receive loop per queue: Fill -&gt; Poll -&gt; Receive -&gt; Recycle.<br>for _, xsk := range fleet.Sockets() {<br>	go func(xsk *afxdp.Socket) {<br>		for {<br>			xsk.Fill(xsk.NumFreeFillSlots()) // hand the kernel empty buffers<br>			n, err := xsk.Poll(-1)           // block until packets arrive<br>			if err != nil {<br>				return<br>			}<br>			descs := xsk.Receive(n)<br>			for _, d := range descs {<br>				packet := xsk.GetFrame(d) // the raw Ethernet frame ([]byte)<br>				// ... do something with it ...<br>			}<br>			xsk.Recycle(descs) // give the buffers back for reuse<br>		}<br>	}(xsk)<br>}</pre><p>Pretty simple, right? No BPF to write, no UMEM math, no ring management. The Fill -&gt; Poll -&gt; Receive -&gt; Recycle loop is the whole programming model, and transmit is the same idea in reverse. This snippet is essentially the <a href="https://github.com/atoonk/go-afxdp/tree/main/examples/drop">drop</a> example, a UDP sink that counts how fast it can receive; its counterpart <a href="https://github.com/atoonk/go-afxdp/tree/main/examples/blast">blast</a> is the transmit side, a UDP flooder that fills every TX queue. Those two together are how I got the numbers in this article.</p><p>Notice the Info() line. AF_XDP can run in very different modes: <strong>native</strong> (in the driver) or <strong>generic</strong> (a slower fallback that works anywhere, even on veth), with <em>zero-copy </em>or <em>copy</em>. The difference between them is huge. The library picks the fastest mode your driver supports and then tells you what you got.</p><h3>It should just do the right thing</h3><p>That mode business is where my old frustrations came from, and where I spent the most effort this time. Whether you get the fast path depends on your driver, your queue configuration, even your MTU. That’s a lot of tribal knowledge to expect from someone who just wants to receive or send packets quickly.</p><p>A concrete example: I recently tested on AWS, on a pair of c7gn.xlarge instances. Out of the box, AF_XDP on the ENA driver quietly falls back to generic mode, and in generic mode my receiver silently lost about 25% of the packets, with no counter anywhere admitting to it. Getting the fast path on ENA requires some unobvious things, including page-sized (4096-byte) packet buffers instead of the usual 2048. Miss that one and zero-copy silently doesn&#39;t engage. The library now detects the ENA driver and handles it automatically, so the stock examples come up as &quot;zero-copy, native XDP&quot; on EC2 with no code changes. The same test then ran at a clean, steady 5 million packets per second, completely lossless, at which point you hit AWS&#39;s per-instance packets-per-second allowance. But hey, that means the software is no longer the bottleneck.</p><p>That’s the philosophy of the library, you shouldn’t need to know that one particular AWS driver wants page-aligned frame sizes. It should just do the right thing.</p><h3>How does it compare to DPDK?</h3><p>Honestly, the performance is in the same ballpark, and that surprised me. DPDK is still the gold standard for absolute numbers, but the gap has become small and the price difference is enormous. With DPDK you lose the whole NIC, you’re tied to specific drivers and a heavy build-and-tuning universe, and there’s effectively no good Go story. AF_XDP works on any modern Linux kernel, keeps the NIC as a normal interface, lets you pick which traffic to bypass, and is one go get away. For me it&#39;s become the sweet spot: 95%+ of the performance at 10% of the operational pain.</p><h3>Where this fits (and where it doesn’t)</h3><p>The cases where AF_XDP shines is anything that&#39;s packet-in, packet-out at high packet rates:</p><ul><li>Packet generators and performance testing. the blast and drop examples together are a poor man’s IXIA, the same itch I used to scratch with DPDK’s pktgen, now in a couple hundred lines of Go.</li><li>UDP-based servers and protocols. There’s a <a href="https://github.com/atoonk/go-afxdp/tree/main/examples/dns">DNS server example</a> in the repo that answers queries straight from the AF_XDP path. Tunnels, VXLAN, QUIC-adjacent experiments: anything UDP-shaped fits naturally.</li><li>Network scanners, IDS-style monitoring, DDoS scrubbing. Anywhere you want to see a lot of packets cheaply, for a specific slice of traffic, without taking the box off the network.</li></ul><p>And where it doesn’t fit: TCP. With AF_XDP you&#39;re getting raw Ethernet frames, so TCP means bringing your own TCP stack in userland, retransmissions and congestion control included. The kernel gives you a world-class TCP implementation for free, and my strong suspicion is that a userland stack bolted onto AF_XDP would give back most of the performance while adding a pile of complexity. If your workload is TCP-shaped, use the kernel. AF_XDP is for when your workload is packets.</p><h3>Wrap-up</h3><p>To me, a reliable easy to use AF_XDP library felt like a missing piece in Go for years: kernel sockets are easy but top out in the low millions of packets per second, and DPDK is fast but heavy, using C, and takes your NIC away. AF_XDP sits right in between: a Go program, built with the normal toolchain, sending and receiving at 10G line rate with a couple of CPU cores, while the rest of the box carries on like normal.</p><p>The code is at <a href="https://github.com/atoonk/go-afxdp">github.com/atoonk/go-afxdp</a>. If you try it on hardware I haven’t, I’d love to hear about it, especially the failure cases. That’s how the ENA quirks got found and fixed.</p><p>That’s it; you made it to the end! Thanks for reading! Let’s push some packets :)<br>Cheers<br>-Andree</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48f4ad867229" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/line-rate-packet-processing-in-go-with-af-xdp-48f4ad867229">Line-rate packet processing in Go with AF_XDP</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Border0 Differs from Tailscale]]></title>
            <link>https://atoonk.medium.com/how-border0-differs-from-tailscale-90fbb3e6f291?source=rss-7e2fc845971d------2</link>
            <guid isPermaLink="false">https://medium.com/p/90fbb3e6f291</guid>
            <category><![CDATA[ssh]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[tailscale]]></category>
            <category><![CDATA[vpn]]></category>
            <category><![CDATA[wireguard]]></category>
            <dc:creator><![CDATA[Andree Toonk]]></dc:creator>
            <pubDate>Sat, 06 Sep 2025 15:54:13 GMT</pubDate>
            <atom:updated>2025-09-06T15:54:13.617Z</atom:updated>
            <content:encoded><![CDATA[<p>Modern VPNs Are Great — but they Stop at Layer 3</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mMcF5_frRnhKDJHERqr_-g.png" /></figure><p>If you’ve tried Tailscale or Twingate, you know how refreshing they feel compared to the old world of VPNs. They’re simple, secure, and easy to use. Built on modern protocols, they replaced clunky VPN appliances with a lightweight setup that “just works.” You log in with your SSO, your laptop joins the “vpn,” and suddenly your resources feel local. No more endless configs, brittle site-to-site tunnels, or opening up endless firewall rules.</p><p>Border0 provides that <a href="https://youtu.be/ZchTqHBobcQ"><strong><em>same</em></strong> modern VPN experience,</a> easy, fast and reliable, but for us that’s just the beginning. Because here’s the truth: modern VPNs like Tailscale and Twingate still operate at the network layer (Layer 3). Once you’re “on the network,” the VPN has no context about what you’re accessing and what you’re allowed to do.‍</p><p>If you need to SSH into a server, you still need the hostname, your SSH key, and the right permissions. If you need to access a database, you need to know the address, open a client, and track down that shared dbadmin password. If you need Kubernetes access, you need your kubeconfig and role mappings. The VPN gets your packets to the right subnet, but that’s where it stops, the rest is up to you.‍</p><p><em>That gap</em>, between secure network access and actually using applications is where we focus.‍</p><h3>The Problem With Stopping at the Network</h3><p>Network access alone introduces a set of headaches we’ve all lived with. The first is over-privileged access. VPNs are blunt instruments. Once connected, you often see more than you should. Maybe you only needed access to a single database, but suddenly you can connect to every server on the subnet. That’s not just a security risk, it’s noise and clutter for the end user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ibldqt5ISGEwHcam.png" /></figure><p>Then there’s the issue of credential sprawl. The VPN doesn’t solve the fact that you still need credentials for each service. Engineers end up juggling SSH keys, database passwords, and kubeconfigs, often shared or long-lived. It’s messy and it doesn’t scale.‍</p><p><strong>Why does this matter?</strong> Because many of these systems (SSH servers, databases) were never designed with SSO in mind. They’re legacy or specialized tools that don’t integrate easily with the way we log in to modern apps. So instead of one identity, teams end up with dozens of disconnected credentials floating around. That creates friction for engineers, and it creates risk for the business. Wouldn’t it be better if you could log in to these resources the same way you log in to Slack or Google Workspace, with your SSO identity, through your VPN?‍</p><p>Visibility is also limited. A VPN log might show “Alice connected to 10.0.0.5:22.” But what did Alice do? Did she run ls or rm -rf /? Did she query customer data or just check a schema? The VPN can’t tell you.‍</p><p>Finally, access control is coarse. Modern VPNs have ACLs, but they’re usually IP and port-based. You can allow or deny traffic, but you can’t say “Alice can log in as support, not root,” or “Bob can query but not modify this database.” To achieve that, you need to stack a PAM tool on top of your VPN, plus custom configs.‍</p><p>These are fundamental limitations of a network-only model. And they’re exactly why we built Border0 differently.‍</p><h3>The World’s First Application-Aware VPN</h3><p>At Border0, we took the same modern VPN foundation: WireGuard, identity-aware access, ease of use and asked a simple question: what if the VPN didn’t stop at the network? What if it understood the applications we use daily?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F-BDEHr9ztSw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D-BDEHr9ztSw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F-BDEHr9ztSw%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e99c12ea4b380a9ad1e9794a33fc4450/href">https://medium.com/media/e99c12ea4b380a9ad1e9794a33fc4450/href</a></iframe><p>That’s what we mean by the world’s first application-aware VPN. Instead of just a secure pipe, we provide a direct, identity-aware connection to the resources you actually need. That might be a server over SSH. It could be PostgreSQL, MySQL, MongoDB, or MSSQL. It could be a Kubernetes cluster, a Docker environment, or an internal web app.‍</p><p>When you log in with your SSO, you don’t just “get on the network.” You see a clear list of resources you’re allowed to access. One click or one command connects you, no separate passwords, no guessing IPs, no credential scavenger hunt.‍</p><h3>Fine-Grained Control at Layer 7</h3><p>Because Border0 is application-aware, our access policies are meaningful at the action level. On a database, you might allow a developer to run SELECT queries but prevent them from running DELETE or INSERT. On a server, you might allow someone to log in as support but not as root. For SSH, you can give someone the ability to open a shell while blocking file transfers or port forwarding. In Kubernetes, you can scope a user’s access to specific namespaces, even letting them view logs without giving them the ability to exec into a pod.</p><p>‍This is true least-privilege access, enforced directly by your VPN solution. <strong>There’s no extra PAM stack to deploy, no bolted-on jump hosts, no brittle workarounds. It’s built in.</strong>‍</p><h3>Full Visibility and Auditing</h3><p>Every session through Border0 can be logged and recorded. That means every SSH command, every SQL query, every kubectl action is tied to the user’s SSO identity. Security teams get full audit trails, and compliance reporting gets dramatically simpler.‍</p><p>We take it one step further <a href="https://www.border0.com/blogs/introducing-ai-session-analysis-smarter-faster-insights-into-privileged-access">with AI-powered session analysis</a>. Instead of just collecting logs, we interpret them. The system can summarize what happened, flag risky behavior, and highlight anomalies. If someone exported ten thousand rows from the customer database at three in the morning, that’s spotted and surfaced right away. It’s like having a security co-pilot sitting in on every session.‍</p><h3>Designed for Developer Experience</h3><p>We’ve lived the pain of clunky access ourselves. In the engineering world, speed and agility are everything. But when security tools slow engineers down with clunky UIs, manual approvals, or workflows that don’t fit how we actually work, developers end up choosing between velocity and security. And let’s be honest, velocity usually wins. When an outage hits, shortcuts appear, static secrets creep back in, and blind spots re-emerge.</p><p>‍At Border0, we obsess over making the secure path the easy path. Log in with SSO and your resources are immediately available. Connect in one click from our web client or with a single command from your preferred shell. No passwords, no static keys, just your SSO identity. And if you prefer, you can even access resources directly in your browser.‍</p><p>The result is security that fits the workflow: engineers get instant, reliable access, while security teams get granular control and visibility. No one has to fight the system, and no one has to choose between speed and safety. With Border0, you really can have your cake and eat it too!‍</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhU7QixSqnSM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhU7QixSqnSM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhU7QixSqnSM%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d0db8373027bcfadd321c3bf41d1750/href">https://medium.com/media/3d0db8373027bcfadd321c3bf41d1750/href</a></iframe><h3>Why This Matters</h3><p>This shift, from network-only VPNs to application-aware VPNs matters for every role. For engineers, <strong>it means no more wasting time setting up tunnels or juggling creds</strong>. If you need to run a quick query on staging, you log in with SSO and go. The secure way is now the fastest way.‍</p><p>For security teams,<strong> it finally delivers fine-grained policies you can actually enforce</strong>. Instead of answering “can Alice reach 10.0.0.5?” you can answer “can Alice log in as root?” or “can she run UPDATEs on the production database?” With full session logs, audits become straightforward.‍</p><p>For leadership,<strong> it replaces the need to buy and integrate both a VPN and a PAM solution</strong>. Instead of stitching together multiple tools, you get one platform that does both. That means fewer moving parts, lower cost, and a future-proof model that scales as your org grows.</p><h3>Comparing Side by Side</h3><p>Put simply, solutions like Tailscale and Twingate are modern, identity-aware VPNs. They are excellent at making access to networks simple. But they stop at Layer 3.‍</p><p>Border0 picks up where they leave off. We do all of the above <strong><em>and</em></strong> add Layer 7 awareness. So instead of just reaching a subnet, you connect straight to the service you need. With fine-grained policies, full auditing, and a developer experience that feels natural.‍</p><p>The next step is clear: evolve the VPN from a network tool into a platform for data and application access.‍</p><h3>Concrete Examples</h3><p>Here’s what this looks like in practice.</p><p><strong>The old way: </strong>A developer who needs database access connects to the VPN, hops onto a jumpbox, and digs through a password manager for credentials. Those credentials are shared across dozens of engineers, usually with full admin rights. Query-level logs? Often nonexistent so zero visibility. Familiar?</p><p><strong>With Border0: </strong>They click “Staging DB” in our web portal or desktop app. Instantly, they’re dropped into a clean, web-based SQL console, or their favorite database client on their desktop. They’re authenticated with their SSO identity, scoped to read-only queries, and every query is logged and analyzed.‍</p><p><strong>The old way:</strong> A support engineer who needs SSH access juggles keys and jump hosts, often with shared root privileges and little oversight. A typical problem is that SSH keys have to be distributed across every system, creating endless work during onboarding and offboarding. Keys pile up, sprawl across environments, and tend to linger forever.</p><p><strong>With Border0: </strong>They simply run <em>ssh prod-server </em>and are logged in as the correct support user, not root, with port forwarding blocked by policy. The entire session is recorded for audit, ​​and key distribution becomes a thing of the past.‍</p><p><strong>The old way:</strong> A developer who needs Kubernetes access waits for kubeconfig distribution, wrestles with role mappings, or risks being handed cluster-wide admin rights. Once inside, activity is hard to track, and visibility into who did what is often limited or nonexistent.</p><p><strong>With Border0: </strong>They get out-of-the-box SSO access to Kubernetes clusters. Running kubectl gives them scoped access to the staging namespace only. No manual config. No risk of leaking credentials. And for anyone who isn’t living in Kubernetes every day, the Border0 web client makes it simple. You don’t have to be a K8s expert to get work done, it’s designed to be easy no matter your skill level.‍</p><p>In all of these cases, the old way meant juggling a stack of tools. Your VPN only got you “on the network.” From there you still needed a jump box, a PAM tool, secrets management, and audit logging, each bolted on, each adding friction.‍</p><p>With Border0, all of that comes built in. VPN access, identity, fine-grained controls, and full auditing are part of one seamless solution.‍</p><h3>Conclusion: The Future of Infrastructure Access</h3><p>VPNs have come a long way. Tailscale, Border0 and other modern day VPN solutions, proved that VPNs can be delightful again. We believe the next leap is clear: VPNs must also be application-aware.</p><p>That’s what we’ve built: a VPN that connects you not just to networks, but to applications with the right permissions, complete visibility, and a user experience engineers actually enjoy.</p><p>So what’s the difference? In one line: they connect you to networks; we connect you to networks and applications!‍</p><p>The best way to understand it is to try it. <a href="https://www.border0.com/contact-us">Contact us for a demo</a>, or <a href="https://portal.border0.com/register">sign up for free</a>, connect a resource, and see how easy secure access can be when the VPN itself understands your applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90fbb3e6f291" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating Infrastructure Outages: Battle Scars and Lessons Learned]]></title>
            <link>https://atoonk.medium.com/navigating-infrastructure-outages-battle-scars-and-lessons-learned-281673439b11?source=rss-7e2fc845971d------2</link>
            <guid isPermaLink="false">https://medium.com/p/281673439b11</guid>
            <category><![CDATA[itil]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[outage]]></category>
            <category><![CDATA[networking]]></category>
            <dc:creator><![CDATA[Andree Toonk]]></dc:creator>
            <pubDate>Mon, 08 Jul 2024 04:34:18 GMT</pubDate>
            <atom:updated>2024-07-08T04:34:18.831Z</atom:updated>
            <content:encoded><![CDATA[<p>One of my great passions is infrastructure operations. My first few jobs were in network engineering, and later on, as the cloud became more prevalent, that turned into what we now call infrastructure or cloud operations. Probably the most stressful times were during outages. Over the last two decades, I’ve been part of many outages, some worse than others, but I wear my operational battle scars proudly. In this blog, I’ll share some learnings and observations from these incidents.</p><h3>What triggered me to write this article</h3><p>I could write many articles about the “epic” incidents I’ve been part of. While stressful at the time, they make for good war stories afterward. In recent years, companies like Facebook and Cloudflare have started publicly sharing their outage retrospectives (timelines, root causes, and lessons learned), which I enjoy reading. It’s free learning!</p><p>A few days ago, Rogers posted its public incident summary, covering the typical aspects of an outage retrospective. The outage itself was significant. Rogers is Canada’s second-largest ISP and cell phone carrier. Two years ago, they experienced a country-wide 24-hour outage that impacted all their phone (including 911) and internet customers. Yes, hard down for a full day! So naturally, as both a Rogers customer and internet operations enthusiast, when they <a href="https://crtc.gc.ca/eng/publications/reports/xona2024.htmhttps://crtc.gc.ca/eng/publications/reports/xona2024.htm">posted their report </a>(2 years later!), I had to read it.</p><h3>The Rogers outage</h3><p>Though not the primary focus of this article, let’s briefly review the outage. During planned maintenance, Rogers staff removed a network routing policy, which caused (BGP, or OSPF/ISIS) routes to be flooded internally. This overwhelmed the control plane, causing all kinds of challenges, eventually causing it to crash or otherwise render its core routers inoperable. A logical next step would be to try and undo that change, however, the routers themselves were no longer manageable due to the control plane issues, and to make matters worse, the out-of-band (OOB) management network was also down. As it turns out, the out-of-band network somehow depended on the production data network, so when that went down, so did the out-of-band network, making troubleshooting and remediation impossible. Eventually, engineers had to be dispatched to manually get console access.</p><h3>Thoughts on the Rogers outage</h3><p>Outages suck. They’re super stressful and, as the Rogers incident shows, can have significant real-world impacts. Having these retrospectives after the fact, and getting to the root cause, lessons learned, etc., is mandatory for any organization that wants to continuously improve itself.</p><p>One thing that stood out for me was the <a href="https://www.cbc.ca/player/play/video/9.6441391">news article from CBC.ca</a> (a major news outlet in Canada), the headline of which was: “Human error triggered massive 2022 Rogers service outage, report finds.” While it’s arguably true that the change made kicked off a sequence of events, declaring that as the root cause is short-sighted. Many of these outages are like airplane crash investigations, where multiple factors contribute to the failure, sometimes starting years ahead or being poorly designed to start with.</p><h3>Reflecting on Outages: Can We Eliminate Them? Should We Want To?</h3><p>It’s always easy to dunk on an outage from the sidelines and to be clear, that’s not my intent. I’ve learned firsthand that Shit happen! So, with the Rogers outage in mind, but also from past experience, let’s extract some generic lessons.</p><p>First off, I’d say that the goal of eliminating outages altogether is likely too ambitious for most of us. Yes, part of an outage retrospective process should be to prevent the exact same outage from happening in the future, but architectures evolve, technologies change, and even a slight change in parameters can lead to a similar outage again.</p><p>So, although we can limit the chance of outages, you won’t be able to eliminate them 100%. In fact, given the cost of complete outage elimination and depending on your industry (say Facebook vs. flying a passenger airplane), you may not necessarily want to aim for that. There are real financial costs, process costs, agility costs, or time-to-value costs by trying to completely achieve outage elimination.</p><p>So, let’s accept that outages will happen and instead use that insight to focus on limiting their damage and impact! This approach allows us to prioritize effective strategies, which I’ll discuss next.</p><h3>Time to Detection — aka Be The First To Know!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EpoV3QC3XDAR1Wvl.png" /></figure><p>Sufficient monitoring helps to reduce time to detect and pinpoint root causes. Sounds obvious, right? Yet, almost every outage retrospective will have an improvement action item called “more monitoring.” That’s an indication that we’re always forgetting some metric for some reason we never appear to have all the checks and metrics. That’s why I’m a big fan of synthetic monitoring for availability.</p><p>This check should mimic a typical user experience. So even if you swap out a core component, you’ll still be able to see the impact without adding more metrics. Because you can’t easily miss something, it’s the first check you should add, and most importantly, should alert on. So whether it’s a ping or a simple HTTP test, this will give you instant confidence your service is working. I call this the “be the first to know” principle. You don’t want to hear from your customers that your service is broken; you’re responsible for it, and you should be the first to know!</p><h3>Limit blast radius.</h3><p>Can we contain the impact when something breaks? The Rogers outage is a prime example of the worst case: one change broke the entire network. Logical separation, whether geographical or functional, could have prevented this. Modern-day cloud providers like AWS actively encourage us to think about this. With AWS, you’re always thinking about Multi-AZ or Multi-region. Some go further by going multi-cloud (think multi-vendor in network land).</p><p>However, transitioning from a single region to multi-region or multi-cloud can become exponentially complex and costly. The same is true in network land, where the path from a single vendor to multi-vendor can become quite complex easily.</p><p>Certain services are easier to make multi-region than others (I’m not even thinking about multi-cloud yet). Take your core database, for example. This comes with real complexity costs, and it’s understandable why this might not always happen.</p><p>I get it, infrastructure teams are often considered cost centers. While high availability and reliability are touted as priorities, urgent needs, feature development, and cost pressures often take precedence. However, never let a good disaster go to waste! After a significant outage, budgets and priorities often shift. This is your opportunity to advocate for investments in resilience. The value proposition is strongest in the immediate aftermath, so seize the moment.</p><h3>Management access</h3><p>The outages that still haunt me and gave me my “Ops PTSD” are those where we lost management access during the outage. This is what happened to Rogers, and a similar incident happened to one of <a href="https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/">Facebook’s largest outages</a>. It’s a real nightmare scenario: you can’t troubleshoot or remediate the issue since you’ve lost the ability to do anything to the device. Often even the ability to just shut it down or otherwise take it out of rotation. Until you restore some form of management access to the device, you’re stuck.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KzHqKSETXKbobosX.png" /></figure><p>This problem is more present for folks running physical hardware than those running, say, on AWS. So, if you run your own data centers, never compromise on out-of-band network access, without thinking about the worst case. Make sure you have OOB access and that it is isolated from your in-band network access. It’s not always easy, but if you don’t have this, your outage time will go from minutes to many hours (see Rogers and Facebook’s example). The feeling of helplessness, even if you know what the issue is, and it only takes one command to fix the issue, is awful.</p><h3>Rollback scenarios</h3><p>Another valuable tool in your toolbox for reducing the impact of an outage is to make sure its duration is limited. You made the change, you quickly see things are bad, and you simply roll back your change. You can see that for this to work, you need management access though (see above).</p><p>In the early days, with Cisco IOS, we’d do it like this:</p><pre># reload in 5<br># conf t<br># &lt; do your change &gt;<br># no reload</pre><p>This meant that unless you executed the last command, the device would reload 5 minutes after your change. Assuming you wouldn’t lose access, you would execute no reload, which would cancel the reload, since it wasn’t necessary. Juniper later introduced a more elegant solution with “commit confirmed,” which auto-rolls back changes if not confirmed within a set time frame.</p><pre>[edit]<br>user@host# commit confirmed <br>commit confirmed will be automatically rolled back in 10 minutes unless confirmed<br>commit complete<br>#commit confirmed will be rolled back in 10 minutes<br>[edit]<br># &lt; On-call blows Up! &gt;<br>user@host# rollback 1<br>user@host# commit</pre><p>If your environment is more of a software development team, then look at using blue-green deployments. Whatever your choice of making changes or deploying new versions of your software, it’s essential to have the ability to quickly roll back to the latest, known working version. Knowing that you can quickly roll back is great for confidence. Obviously, this depends on our earlier topics: “be the first to know” and management access.</p><h3>Team culture — Don’t be a hero</h3><p>Last but not least, we have to talk about team culture. One of the good things about moving from the traditional Ops model to DevOps (you build it, you own it) is that you no longer throw a new version of the software over the wall and ask your ops team to deploy it sight unseen.</p><p>Inevitably, at some point, you ship a broken version and your Ops person (this was me) has no clue what changed or how to troubleshoot it. This is extremely stressful and, frankly, unfair.</p><p>Nowadays, most development teams own and deploy their own changes and have modern deployment pipelines to deploy them. But even then, the software is complex, and not everyone in the team understands all components. So, when there’s an outage, it should be easy to bring in the rest of the team. Don’t be shy about calling your team members, even in the middle of the night. In a healthy organization, no one should want to wake up the next morning hearing there was a 4-hour outage and someone on your team tried to troubleshoot the bug you introduced. So have a WhatsApp group, or whatever tool you use, just for your team, your own out-of-band channel for emergency help.</p><p>Outages are overwhelming. You get a ton of alerts that all need to be acknowledged, and you get questions from support teams, customers, senior management, etc. And then you’re also expected to troubleshoot. No one person can do this. You need a few folks to troubleshoot, manage the alerts, and coordinate the whole process, including communication. So, don’t be a hero. Call in your teammates; together you’ll be able to deal with this. Remember, one team, one dream.</p><h3>Wrap up</h3><p>Outages will inevitably occur. Let’s acknowledge this reality and take proactive steps to limit their impact and duration.</p><p>Be mindful of the knee-jerk management response: “We need more change management process!” Unless you’re a real YOLO shop, this is rarely the answer.</p><p>Instead, use the “be the first to know” principle, ensure access to your gear at all times, and have tools ready to swiftly roll back changes to a last-known-good state. Last but not least, embrace the opportunity to learn with your team through retrospectives and emerge even stronger. With a healthy team dynamic, outages can surprisingly become valuable bonding experiences. By being prepared and working together, you’ll turn outages into mere blips on the radar.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=281673439b11" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with Border0]]></title>
            <link>https://atoonk.medium.com/getting-started-with-border0-f8a2eecd9c8e?source=rss-7e2fc845971d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8a2eecd9c8e</guid>
            <category><![CDATA[ssh]]></category>
            <category><![CDATA[zero-trust]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Andree Toonk]]></dc:creator>
            <pubDate>Thu, 21 Mar 2024 18:07:29 GMT</pubDate>
            <atom:updated>2024-03-21T18:07:29.181Z</atom:updated>
            <content:encoded><![CDATA[<p>This week, we released an improved version of the connector install flow.<br>👉 Check out this 2-minute demo and see how easy it is to get your first Border0 connector installed, and start using it to make your servers, containers, and database available. Super happy with this new flow! 😀</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FkH2acCxUcgk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DkH2acCxUcgk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FkH2acCxUcgk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1365f9944dbcbd3ab3264b2a10c4109f/href">https://medium.com/media/1365f9944dbcbd3ab3264b2a10c4109f/href</a></iframe><p>I may have overused the word “easy” in this video… But hey.. that’s what it is, easy! 😂 ¯\_(ツ)_/¯</p><p>After you have your new connector up and running, you can add sockets (services) and make them available to your team members. Examples include Databases like MySQL, Postgres, or MSSQL, running in AWS, GCP, Azure, or anywhere else. Or compute services such as Linux machines, ec2, Docker containers, ECS, or Kubernetes containers. Finally, desktop access, such as RDP and VNC, is also supported.</p><p>With Border0, users can easily discover all services they have access to. They can connect without the need for a VPN using their favorite tools or all through the browser. Shared database credentials, static keys, or passwords can be completely eliminated. It’s low friction and a way of working that fits in an engineer&#39;s natural flow.</p><p>Finally, administrators such as IT teams or Security folks can set fine grained policies, defining exactly who should have what kind of access to what Server, when, from where and under specific custom conditions. For example, you could define a policy that gives the Okta group “Database-admins” read-write access to the databases, but only during work hours and only when in a specific Geo. Meanwhile, the SRE team may only have read-only access to these databases when on-call. These are simple examples, but they illustrate the level of control you have.</p><p>But it’s not just control; it’s also visibility and auditability. You can now easily see who (what SSO identity) accessed what server, when, and from where. You can even see what happened during each session, for example, what database queries were executed, or watch a video of the SSH session.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2PLLJzmaP-9xhw37EUWyHQ.png" /><figcaption>Database query logs, per Single sign-on identity and session</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RN531Y3A5L4z3ItFFbj3Yg.png" /><figcaption>SSH session recording</figcaption></figure><h3>Curious? Try our free community edition</h3><p>Sounds pretty good, right?! We provide a generous community version that allows up to 5 users for free. It’s a great way to get started! <a href="https://portal.border0.com/register">Click here to register your account now and get started</a>. I guarantee it will take you less than a minute to log in to your first server!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8a2eecd9c8e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High-Speed Packet Processing in Go: From net.Dial to AF_XDP]]></title>
            <link>https://levelup.gitconnected.com/high-speed-packet-transmission-in-go-from-net-dial-to-af-xdp-2699452efef9?source=rss-7e2fc845971d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2699452efef9</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[network]]></category>
            <category><![CDATA[sockets]]></category>
            <category><![CDATA[packet-analysis]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Andree Toonk]]></dc:creator>
            <pubDate>Mon, 11 Mar 2024 16:08:20 GMT</pubDate>
            <atom:updated>2024-03-12T16:56:57.202Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>This post was originally published on my Personal blog at </em></strong><a href="https://toonk.io/sending-network-packets-in-go/"><strong><em>https://toonk.io/sending-network-packets-in-go/</em></strong></a></p><p><em>Pushing limits in Go: from net.Dial to syscalls, AF_PACKET, and lightning-fast AF_XDP. Benchmarking packet sending performance..</em></p><p>Recently, I wrote a <a href="https://github.com/atoonk/ping-aws-ips">Go program that sends ICMP ping messages</a> to millions of IP addresses. Obviously I wanted this to be done as fast and efficiently as possible. So this prompted me to look into the various methods of interfacing with the network stack and sending packets, fast! It was a fun journey, so in this article, I’ll share some of my learnings and document them for my future self :) You’ll see how we get to 18.8Mpps with just 8 cores. There’s also <a href="https://github.com/atoonk/go-pktgen">this Github repo that has the example code</a>, making it easy to follow along.</p><h3>The use case</h3><p>Let’s start with a quick background of the problem statement. I want to be able to send as many packets per second from a Linux machine. There are a few use cases, for example, the Ping example I mentioned earlier, but also maybe something more generic like dpdk-pktgen or even something Iperf. I guess you could summarize it as a packet generator.</p><p>I’m using the Go programming language to explore the various options. In general, the explored methods could be used in any programming language since these are mostly Go-specific interfaces around what the Linux Kernel provides. However, you may be limited by the libraries or support that exist in your favorite programming language.</p><p>Let’s start our adventure and explore the various ways to generate network packets in Go. I’ll go over the options, and we’ll end with a benchmark, showing us which method is the best for our use case. I’ve included examples of the various methods in a Go package; you can <a href="https://github.com/atoonk/go-pktgen/">find the code here</a>. We’ll use the same code to run a benchmark and see how the various methods compare.</p><h3>The net.Dial method</h3><p>The net.Dial method is the most likely candidate for working with network connections in Go. It’s a high-level abstraction provided by the standard library’s net package, designed to establish network connections in an easy-to-use and straightforward manner. You would use this for bi-directional communication where you can simply read and write to a Net.Conn (socket) without having to worry about the details.</p><p>In our case, we’re primarily interested in sending traffic, using the net.Dial method that looks like this:</p><pre>conn, err := net.Dial(&quot;udp&quot;, fmt.Sprintf(&quot;%s:%d&quot;, s.dstIP, s.dstPort))<br>if err != nil {<br>	return fmt.Errorf(&quot;failed to dial UDP: %w&quot;, err)<br>}<br>defer conn.Close()</pre><p>After that, you can simply write bytes to your conn like this</p><pre>conn.Write(payload)</pre><p>You can find our code for this in the file <a href="https://github.com/atoonk/go-pktgen/blob/main/pktgen/af_inet.go">af_inet.go</a></p><p>That’s it! Pretty simple, right? As we’ll see, however, when we get to the benchmark, this is the slowest method and not the best for sending packets quickly. Using this method, we can get to about 697,277 pps</p><h3>Raw Socket</h3><p>Moving deeper into the network stack, I decided to use raw sockets to send packets in Go, unlike the more abstract net.Dial method, raw sockets provide a lower-level interface with the network stack, offering granular control over packet headers and content. This method allows us to craft entire packets, including the IP header, manually.</p><p>To create a raw socket, we’ll have to make our own syscall, give it the correct parameters, and provide the type of traffic we’re going to send. We’ll then get back a file descriptor. We can then read and write to this file descriptor. This is what it looks like at the high level; see rawsocket.go for the complete code.</p><pre>fd, err := syscall.Socket(syscall.AF_INET, syscall.SOCK_RAW, syscall.IPPROTO_RAW)<br>if err != nil {<br>	log.Fatalf(&quot;Failed to create raw socket: %v&quot;, err)<br>}<br>defer syscall.Close(fd)<br><br>// Set options: here, we enable IP_HDRINCL to manually include the IP header<br>if err := syscall.SetsockoptInt(fd, syscall.IPPROTO_IP, syscall.IP_HDRINCL, 1); err != nil {<br>	log.Fatalf(&quot;Failed to set IP_HDRINCL: %v&quot;, err)<br>}</pre><p>That’s it, and now we can read and write our raw packet to file descriptor like this</p><pre>err := syscall.Sendto(fd, packet, 0, dstAddr)</pre><p>Since I’m using IPPROTO_RAW, we’re bypassing the transport layer of the kernel’s network stack, and the kernel expects us to provide a complete IP packet. We do that using the <a href="https://github.com/atoonk/go-pktgen/blob/main/pktgen/rawsocket.go#L66C17-L66C28">BuildPacket function</a>. It’s slightly more work, but the neat thing about raw sockets is that you can construct whatever packet you want.</p><p>We’re telling the kernel just to take our packet, it has to do less work, and thus, this process is faster. All we’re really asking from the network stack is to take this IP packet, add the ethernet headers, and hand it to the network card for sending. It comes as no surprise, then, that this option is indeed faster than the Net.Dial option. Using this method, we can reach about 793,781 pps, about 100k PPS more than the net.Dial method.</p><h3>The AF_INET Syscall Method</h3><p>Now that we’re used to using syscalls directly, we have another option. In this <a href="https://github.com/atoonk/go-pktgen/blob/main/pktgen/af_inet_syscall.go">example</a>, we create a UDP socket directly like below</p><pre>fd, err := syscall.Socket(syscall.AF_INET, syscall.SOCK_DGRAM, syscall.IPPROTO_UDP)</pre><p>After that we can simply write our payload to it using the Sendto method like before.</p><pre>err = syscall.Sendto(fd, payload, 0, dstAddr)</pre><p>It looks similar to the raw socket example, but a few differences exist. The key difference is that in this case we’ve created a socket of type UDP, which means we don’t need to construct the complete packet (IP and UDP header) like before. When using this method, the kernel manages the construction of the UDP header based on the destination IP and port we specify and handles the encapsulation process into an IP packet.</p><p>In this case, the payload is just the UDP payload. In fact, this method is similar to the Net.Dial method before, but with fewer abstractions.</p><p>Compared to the raw socket method before, I’m now seeing 861,372 pps — that’s a 70k jump. We’re getting faster each step of the way. I’m guessing we get the benefit of some UDP optimizations in the kernel.</p><h3>The Pcap Method</h3><p>It may be surprising to see Pcap here for sending packets. Most folks know pcap from things like tcpdump or Wireshark to capture packets. But it’s also a fairly common way to send packets. In fact, if you look at many of the Go-packet or Python Scappy examples, this is typically the method listed to send custom packets. So, I figured I should include it and see its performance. I was skeptical, but was pleasantly surprised when I saw the pps numbers!</p><p>First, let’s take a look at what this looks like in Go; again, for the complete example, see my implementation in <a href="https://github.com/atoonk/go-pktgen/blob/main/pktgen/pcap.go">pcap.go here</a></p><p>We start by creating a Pcap handle like this:</p><pre>handle, err := pcap.OpenLive(s.iface, 1500, false, pcap.BlockForever)<br>if err != nil {<br>	return fmt.Errorf(&quot;could not open device: %w&quot;, err)<br>}<br>defer handle.Close()</pre><p>Then we <a href="https://github.com/atoonk/go-pktgen/blob/main/pktgen/pcap.go#L51-L64">create the packet manually,</a> similar to the Raw socket method earlier, but in this case, we include the Ethernet headers. <br>After that, we can write the packet to the pcap handle, and we’re done!</p><pre>err := handle.WritePacketData(packet)</pre><p>To my surprise, this method resulted in quite a performance win. We surpassed the one million packets per second mark by quite a margin: 1,354,087 pps — almost a 500k pps jump!</p><p>Note that, towards the end of this article, we’ll look at a caveat, but good to know that this method stops working well when sending multiple streams (go routines).</p><h3>The af_packet method</h3><p>As we explore the layers of network packet crafting and transmission in Go, we next find the AF_PACKET method. This method is popular for IDS systems on Linux, and for good reasons!</p><p>It gives us direct access to the network device layer, allowing for the transmission of packets at the link layer. This means we can craft packets, including the Ethernet header, and send them directly to the network interface, bypassing the higher networking layers. We can create a socket of type AF_PACKET using a syscall. In Go this will look like this:</p><pre>fd, err := syscall.Socket(syscall.AF_PACKET, syscall.SOCK_RAW, int(htons(syscall.ETH_P_IP)))</pre><p>This line of code creates a raw socket that can send packets at the Ethernet layer. With AF_PACKET, we specify SOCK_RAW to indicate that we are interested in raw network protocol access. By setting the protocol to ETH_P_IP, we tell the kernel that we’ll be dealing with IP packets.</p><p>After obtaining a socket descriptor, we must bind it to a network interface. This step ensures that our crafted packets are sent out through the correct network device:</p><pre>addr := &amp;syscall.SockaddrLinklayer{<br>	Protocol: htons(syscall.ETH_P_IP),<br>	Ifindex:  ifi.Index,<br>}</pre><p>Crafting packets with AF_PACKET involves manually creating the Ethernet frame. This includes setting both source and destination MAC addresses and the EtherType to indicate what type of payload the frame is carrying (in our case, IP). We’re using the same <a href="https://github.com/atoonk/go-pktgen/blob/main/pktgen/af_packet.go#L56-L68">BuildPacket function</a> as we used for the Pcap method earlier.</p><p>The packet is then ready to be sent directly onto the wire:</p><pre>syscall.Sendto(fd, packet, 0, addr)</pre><p>The performance of the AF_PACKET method turns out to be almost identical to that achieved with the pcap method earlier. A quick Google, shows that libpcap, the library underlying tools like tcpdump and the Go pcap bindings, uses AF_PACKET for packet capture and injection on Linux platforms. So, that explains the performance similarities.</p><h3>Using the AF_XDP Socket</h3><p>We have one more option to try. AF_XDP is a relatively recent development and promises impressive numbers! It is designed to dramatically increase the speed at which applications can send and receive packets directly from and to the network interface card (NIC) by utilizing a fast path through the traditional Linux network stack. Also see <a href="https://toonk.io/building-an-xdp-express-data-path-based-bgp-peering-router/index.html">my earlier blog on XDP here</a>.</p><p>AF_XDP leverages the XDP (eXpress Data Path) framework. This capability not only provides minimal latency by avoiding kernel overhead but also maximizes throughput by enabling packet processing at the earliest possible point in the software stack.</p><p>The Go standard library doesn’t natively support AF_XDP sockets, and I was only able to find one library to help with this. So it’s all relatively new still.</p><p>I’m using this library <a href="http://github.com/asavie/xdp">github.com/asavie/xdp</a> and this is how you can initiate an AF_XDP socket.</p><pre>xsk, err := xdp.NewSocket(link.Attrs().Index, s.queueID, nil)</pre><p>Note that we need to provide a NIC queue; this is a clear indicator that we’re working at a lower level than before. The complete code is a bit more complicated than the other options, partially because we need to work with a user-space memory buffer (UMEM) for packet data. This method reduces the kernel’s involvement in packet processing, cutting down the time packets spend traversing system layers. By crafting and injecting packets directly at the driver level. So, instead of pasting the code, please look at <a href="https://github.com/atoonk/go-pktgen/blob/main/pktgen/af_xdp.go#L40-L97">my code here</a>.</p><p>The results look great; using this method, I can now generate 2,647,936 pps. That’s double the performance we saw with AF_PACKET! Whoohoo!</p><h3>Wrap-up and some takeaways</h3><p>First off, this was fun to do and learn! We looked at the various options to generate packets from the traditional <em>net.Dial</em> method, to <em>raw sockets</em>, <em>pcap</em>, <em>AF_PACKET</em> and finally <em>AF_XDP</em>. The graph below shows the numbers per method (all using one CPU and one NIC queue). AF_XDP is the big winner!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*usmgr6Nc2lYltHK4" /><figcaption>The various ways to send network traffic in Go</figcaption></figure><p>If interested, you can run the benchmarks yourself on a Linux system like below:</p><pre>./go-pktgen --dstip 192.168.64.2 --method benchmark \<br> --duration 5 --payloadsize 64 --iface veth0<br><br>+-------------+-----------+------+<br>|   Method    | Packets/s | Mb/s |<br>+-------------+-----------+------+<br>| af_xdp      |   2647936 | 1355 |<br>| af_packet   |   1368070 |  700 |<br>| af_pcap     |   1354087 |  693 |<br>| udp_syscall |    861372 |  441 |<br>| raw_socket  |    793781 |  406 |<br>| net_conn    |    697277 |  357 |<br>+-------------+-----------+------+</pre><p>The important number to look at is packets per second as that is the limitation on software network stacks. The Mb/s number is simply the packet size x the PPS number you can generate. It’s interesting to see the easy 2x jump from the traditional net.Dial approach to using AF_PACKET. And then another 2x jump when using AF_XDP. Certainly good to know if you’re interested in sending packets fast!</p><p>The benchmark tool above uses one CPU and, thus, one NIC queue by default. The user can, however, elect to use more CPUs, which will start multiple go routines to do the same tests in parallel. The screenshot below shows the tool running with eight streams (and 8 CPUs) using AF_XDP, generating 186Gb/s with 1200 byte packets (18.8Mpps)! That’s really quite impressive for a Linux box (and not using DPDK). Faster than what you can do with Iperf3 for example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yxwwbjXGFC0PUxAe" /></figure><h3>Some caveats and things I’d like to look at in the future</h3><p>Running multiple streams (go routines) using the PCAP method doesn’t work well. The performance degrades significantly. The comparable AF_PACKET method, on the other hand, works well with multiple streams and go routines.</p><p>The AF_XDP library I’m using doesn’t seem to work well on most hardware NICs. I opened a <a href="https://github.com/asavie/xdp/issues/31">GitHub issue</a> for this and hope it will be resolved. It would be great to see this be more reliable as it kind of limits more real-world AF_XDP Go applications. I did most of my testing using veth interfaces; i’d love to see how it works on a physical NIC and a driver with XDP support.</p><p>It turns out that for AF_PACKET, there’s a zero-copy mode facilitated by the use of memory-mapped (mmap) ring buffers. This feature allows user-space applications to directly access packet data in kernel space without the need for copying data between the kernel and user space, effectively reducing CPU usage and increasing packet processing speed. This means that, in theory, the performance of AF_PACKET and AF_XDP could be very similar. However, it appears the Go implementations of AF_PACKET do not support zero-copy mode or <a href="https://twitter.com/jtollet/status/1762616103883227490">only for RX</a> and not TX. So I wasn’t able to use that. I found <a href="https://github.com/csulrong/gopacket/pull/1/files">this patch</a> but unfortunately couldn’t get it to work within an hour or so, so I moved on. If this works, this will likely be the preferred approach as you don’t have to rely on AF_XDP support.</p><p>Finally, I’d love to include DPDK support in this <a href="https://github.com/atoonk/go-pktgen/tree/main">pktgen library</a>. It’s the last one missing. But that’s a whole beast on its own, and I need to rely on good Go DPDK libraries. Perhaps in the future!</p><p>That’s it; you made it to the end! Thanks for reading!</p><p>Cheers<br> -Andree</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2699452efef9" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/high-speed-packet-transmission-in-go-from-net-dial-to-af-xdp-2699452efef9">High-Speed Packet Processing in Go: From net.Dial to AF_XDP</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[IPv4 surcharge — Your AWS bill is going up this February]]></title>
            <link>https://aws.plainenglish.io/ipv4-surcharge-your-aws-bill-is-going-up-this-february-50e706e09cd5?source=rss-7e2fc845971d------2</link>
            <guid isPermaLink="false">https://medium.com/p/50e706e09cd5</guid>
            <category><![CDATA[network]]></category>
            <category><![CDATA[ipv4]]></category>
            <category><![CDATA[ipv6]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Andree Toonk]]></dc:creator>
            <pubDate>Wed, 31 Jan 2024 16:03:36 GMT</pubDate>
            <atom:updated>2024-02-01T04:54:00.035Z</atom:updated>
            <content:encoded><![CDATA[<h3>IPv4 surcharge — Your AWS bill is going up this February</h3><p><em>Here’s how to save money and be more secure!</em></p><p>As of tomorrow, your AWS bill will go up! Effective February 1, 2024, there will be a charge of $0.005 per IP per hour for all public IPv4 addresses, whether attached to a service or not. That’s a total of $43.80 per year, a pretty hefty number! The reason for this is outlined in the <a href="https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address-charge-public-ip-insights/">AWS announcement</a>:<br>‍</p><blockquote><em>As you may know, IPv4 addresses are an increasingly scarce resource and the cost to acquire a single public IPv4 address has risen more than 300% over the past 5 years. This change reflects our own costs and is also intended to encourage you to be a bit more frugal with your use of public IPv4 addresses.</em>‍</blockquote><p>In this blog, I’ll cover how you can save money on your AWS bill by eliminating unnecessary public IPv4 addresses using Border0. But before we go there, let’s look at how many IPv4 addresses Amazon has, how much that’s worth, and how much AWS will make with this new charge to your monthly bill.‍</p><h3>How many IPv4 Addresses does AWS have?</h3><p>Operating the Amazon infrastructure and keeping up with the incredible growth of AWS requires a massive amount of IP addresses. And so it comes as no surprise that over the years, Amazon has spent a lot of money acquiring an enormous number of IPv4 addresses. All so we can continue to spin up our ec2 instances, load balancers, and NAT gateways without worrying about IPv4 addresses.‍</p><p>To determine exactly how many IPv4 addresses Amazon has, we can look at various publicly available data sets. The data I used is the <a href="https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-ranges.html">AWS IP json</a> and the various whois (ARIN, RIPE, etc) data entries.‍</p><p><em>Crunching all that data, we can determine that Amazon has at least 131,932,752 IPv4 addresses.<br>Let’s round that up and say </em><strong><em>132 Million IPv4 addresses</em></strong><em>! That’s the equivalent of almost eight /8’s 😮</em></p><p>Curious about the data and what IPv4 addresses were included? See <a href="https://gist.github.com/atoonk/0ee3f5bebcea874f6032215f16c3c30a">this link for the raw data.</a></p><h3>‍How much is the Amazon IPv4 estate worth?‍</h3><p>IPv4 addresses are like digital real estate. These 32-bit integers have real monetary value and can be bought and sold. In fact, the price of IPv4 addresses has increased significantly over the last decade, and would have made an excellent investment if you got in early!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E9X2IXSQJDy8uxFI.png" /><figcaption><a href="https://auctions.ipv4.global/prior-sales">https://auctions.ipv4.global/prior-sales</a></figcaption></figure><p>So the next logical question is, how much is the Amazon IPv4 estate worth? Based on data from ipv4.global, the average price for an IPv4 address is currently ~35 dollars. With that data in hand, we can do our back-of-the-napkin math:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*FNpH0XCOVZrcc9mA.jpeg" /></figure><blockquote>‍<br><em>So the approximate value of Amazon’s IPv4 estate today is about: <br>‍</em><strong><em>$4.6 Billion dollars!</em></strong> Not too shabby!‍</blockquote><h3>How much money will AWS make with the new IPv4 charge?</h3><p>‍Speaking of dollars, let’s take a look if we can make an educated guess about how much AWS will make from the new IPv4 charge. For that, we need the price per IP and the number of IPv4 addresses in use by AWS customers.</p><p>‍We know the first variable, $0.005 per IP per hour, or $43.80 per year per IPv4 address. The second variable, the number of IPv4 addresses in use by AWS customers, is harder to determine. However, we can make some educated guesses for fun!</p><p>‍As mentioned, the significant variable here is how many IP addresses are used at any given time by AWS customers. Let’s explore a few scenarios, starting with a very conservative estimate, say 10% of the IPv4 addresses published in the <a href="https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-ranges.html">AWS JSON</a> (79M IPv4 addresses) are used for a year. That’s 7.9 Million IPv4 addresses x $43.80, almost $346 Million a year. At 25% usage, that’s nearly $865 Million a year. And at 30% usage, that’s a billion dollars!‍</p><p>That gives us a pretty good indicator of the scale we’re talking about. Another approach is to try and measure it. How many IP addresses are alive within the AWS network right now? AWS conveniently publishes all addresses, so we could send an ICMP echo request (a ping) to all of them and measure how many send back an echo reply.‍</p><p>That sounded like a fun project! So I wrote a quick <a href="https://github.com/atoonk/ping-aws-ips">program</a> that downloads the JSON with all the AWS IP addresses and filters out the categories “AMAZON,” “EC2,” and “GLOBAL ACCELERATOR.” We will assume these are all the customer-used (charged) IP addresses. I.e., we’re not going to ping services like Route53 Health Checks or Cloudfront as those won’t show up on your bill as an IPv4 charge.</p><p>‍The program sends a single ICMP packet to all IP addresses and collects all the replies. With this, we have some actual measurement data, and we observe that we received a reply from roughly 6 Million IPv4 addresses. <br>6 Million addresses x $43.80 is $ 263 Million annually!‍</p><p>That’s another good data point. However, remember that many ec2 instances and other services will have strict security groups and, by default, won’t respond to a ping packet. So, it’s fair to say that six million active IPs is the absolute minimum. The actual number of active IPv4 addresses could easily be double that given the various default security groups blocking ICMP.‍</p><blockquote>Given this data, I believe it’s fair to say that <strong><em>AWS will likely make anywhere between $400 Million and $1 Billion dollars a year with this new IPv4 charge! </em></strong>That’s a nice bump for AWS, especially given that this was provided for free until today.‍</blockquote><h3>Lowering your AWS bill with Border0</h3><p>Your AWS services, such as ec2 instances, may have public IPv4 addresses for various reasons. One common reason is to have management access to your servers. For example, using SSH or RDP. Or to access the app running on your machine, like a database or HTTP application.‍</p><p>Some of your applications should likely only be accessible to authorized users and, ideally, not connected directly to the Internet. For example, <a href="https://www.border0.com/blogs/help-my-postgres-database-was-compromised">recent Border0 research</a> showed that botnets are actively compromising publicly accessible Mysql and Postgres servers! You don’t want these unprotected on the Internet for everyone to poke at.‍</p><p>We recommend running your AWS infrastructure in a private subnet with only a NAT gateway for outbound Internet connectivity. This way, they’re shielded from the Internet, significantly reducing the risk of getting compromised. As a bonus, you save yourself the AWS IPv4 charge! (note: the charge is only for public IP addresses).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FNHR3Do4WhVU&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DNHR3Do4WhVU&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FNHR3Do4WhVU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5beedd4cd48ad9ff92af3d3bc433295f/href">https://medium.com/media/5beedd4cd48ad9ff92af3d3bc433295f/href</a></iframe><p>With the deployment of a Border0 connector in your private network, you and the rest of your team can still access all services using just your existing Single Sign-on credentials without needing a VPN.</p><p>Deploying Border0 is <a href="https://youtu.be/e0KeS5x-GZ0">easier than you may think</a>! Curious to give it a try? Check out our <a href="https://www.border0.com/blogs/border0-terraform-provider">terraform example</a> or this <a href="https://www.border0.com/blogs/elevating-security-and-simplifying-aws-access-with-border0">blog on Border0 for AWS</a>.<br>‍</p><p>Border0 offers <a href="https://portal.border0.com/register">a generous free tier</a>, and getting started is easy! <br>With Border0, access is easier and more secure; your engineers and security team will love it. And, since you’re saving on public IPs, your FinOps folks will be happy, too!</p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a><strong> | </strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a><strong> | </strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a><strong> | </strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a><strong> | </strong><a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://stackademic.com/"><strong>Stackademic</strong></a><strong> | </strong><a href="https://cofeed.app/"><strong>CoFeed</strong></a><strong> | </strong><a href="https://venturemagazine.net/"><strong>Venture</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a>‍</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50e706e09cd5" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/ipv4-surcharge-your-aws-bill-is-going-up-this-february-50e706e09cd5">IPv4 surcharge — Your AWS bill is going up this February</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Help! My database was compromised!]]></title>
            <link>https://blog.devops.dev/help-my-database-was-compromised-852dfe8f08c6?source=rss-7e2fc845971d------2</link>
            <guid isPermaLink="false">https://medium.com/p/852dfe8f08c6</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[malware]]></category>
            <dc:creator><![CDATA[Andree Toonk]]></dc:creator>
            <pubDate>Mon, 15 Jan 2024 15:16:13 GMT</pubDate>
            <atom:updated>2024-01-16T15:45:55.396Z</atom:updated>
            <content:encoded><![CDATA[<p>A detailed look at an active PostgreSQL and MySQL Ransomware bot</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uhGSfKtjmbXk1r16vA4LHw.png" /></figure><p>The other day, I noticed <a href="https://twitter.com/fasterthanlime/status/1741935393413402739">this tweet</a> mentioning how someone’s test PostgreSQL database was compromised. This intrigued me and made me wonder how common this is. So, I decided to run an experiment. Let’s run a simple PostgreSQL server on a DigitalOcean VM and see what happens! And wow, I wasn’t disappointed.‍</p><p>To my surprise, it only took a few hours to get compromised; in fact, I re-ran the experiment a few times and reproduced the compromise several times a day. It’s a scary world out there!‍</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GXwPPgSBTVCG3I5W.png" /><figcaption>Compromised Database</figcaption></figure><p>In all cases, my database was wiped out (drop database). Interestingly, the attacker left a note in a newly created database called <em>readme_to_recover</em> with instructions for how to get our data back…‍</p><p>So this made me curious: what exactly happened, and how common is it?‍</p><h3>What exactly happened?</h3><p>To better understand exactly what happened during these events, I put my PostgreSQL database behind a simple custom PostgreSQL proxy. This authenticates the user and logs all queries. Doing this will give us some good insights into exactly what queries were being executed by the attacking bot.‍</p><p>In this experiment, we made our test PostgreSQL database available from anywhere on the Internet, with username: <em>postgres</em> and password: <em>password</em>. It had an example database called books, with a few tables and some dummy data.‍</p><p>With this setup, we should learn more about what exactly happened. I didn’t have to wait long; soon after starting my new setup, the bot came knocking again!<br>The first log line shows me where the attack was coming from:‍</p><pre>2024/01/11 19:02:10 New connection accepted from 94.156.71.8:58212</pre><p>‍This is an IP from a hosting provider out of The Netherlands, AS394711, also known as Limenet. The user authenticated as user: postgres and password: password</p><p>‍With our lightweight proxy, we can now see exactly what queries are being executed. The first few queries we see are intended to explore what’s in our database.</p><pre>SELECT datname FROM pg_database;</pre><p>This helps the bot to identify all available databases in our PostgreSQL server.</p><p>Now that the bot knows what databases are available, it next tries to determine what tables are available for each database and what they look like. Then, it takes a snapshot of each table in the database before dropping both the tables and databases.‍</p><p>These are the exact queries we recorded, all wrapped in a transaction per table. This was repeated for all tables in my books database.‍</p><pre>BEGIN<br>SELECT pg_database_size(&#39;books&#39;)<br>SELECT table_name FROM information_schema.tables WHERE table_schema = &#39;public&#39; <br>  AND table_type = &#39;BASE TABLE&#39; AND table_catalog = &#39;books&#39;;<br>SELECT column_name FROM information_schema.columns WHERE table_schema = &#39;public&#39;<br>  AND table_name = &#39;authors&#39;;<br>SELECT * FROM authors LIMIT 20;<br>DROP TABLE IF EXISTS authors CASCADE;<br>COMMIT</pre><p>‍After processing all tables, it then dropped the database like this:</p><pre>DROP DATABASE books;‍</pre><p>Before continuing, it also executed this query:</p><pre>SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity WHERE <br>pg_stat_activity.datname &lt;&gt; &#39;postgres&#39; AND pid &lt;&gt; pg_backend_pid();‍</pre><p>This query terminates all backend processes connected to any database except the ‘postgres’ database and excludes the process running the query itself. It is likely used to terminate other active connections to the database. Possibly to prevent administrators or automated systems from interrupting the attack.‍</p><h3>A note was left — Ransom Demand</h3><p>As a last step, the attacker left us a note in a newly created database called readme_to_recover.</p><pre>CREATE DATABASE readme_to_recover TEMPLATE template0;</pre><p>Then, it uses this new database to add a readme like below.</p><pre>BEGIN<br>CREATE TABLE readme (text_field VARCHAR(255));<br>INSERT INTO readme (text_field) VALUES <br>(&#39;All your data is backed up. You must pay 0.007 BTC to 164hyKPAoC5ecqkJ2ygeGoGFRcauWRLujV In 48 hours, your data will be publicly disclosed <br>and deleted. (more information: go to http://iplis.ru/data3)&#39;);<br>INSERT INTO readme (text_field) VALUES (&#39;After paying send mail to us: <br>rambler+3uzdl@onionmail.org and we will provide a link for you to download your data. <br>Your DBCODE is: 3UZDL&#39;);<br>COMMIT‍</pre><p>So, a <em>select text_field from readme</em> shows us how to recover our data.‍</p><blockquote><em>All your data is backed up. You must pay 0.007 BTC to 164hyKPAoC5ecqkJ2ygeGoGFRcauWRLujV In 48 hours, your data will be publicly disclosed and deleted. (more information: go to </em><a href="http://iplis.ru/data3"><em>http://iplis.ru/data3</em></a><em>After paying send mail to us: </em><a href="/service/mailto:rambler+3uzdl@onionmail.org"><em>rambler+3uzdl@onionmail.org</em></a><em> and we will provide a link for you to download your data. Your DBCODE is: 3UZDL</em></blockquote><p>‍It conveniently informs us that we can get a copy of our data back for 0.007 BTC (about $327 USD at the time of writing this article). The data will be publicly disclosed and deleted if we don’t pay!‍</p><p>But that’s a lie; we know the bot didn’t take all our data! The query logs clearly show that while it did a select * for each table, it also included a LIMIT 20, which means that the bot only selected 20 rows for each table. So even though it’s claiming it backed up all the data, that’s clearly not true. It can also not disclose all data, as all it took was the first 20 rows for each table. This also means that paying the ransom of 0.007 BTC will be useless.‍</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PS6SQ9KRQVZ5UcdV.png" /><figcaption>Queries executed by the attacking ransom bot</figcaption></figure><h3>‍How much money is made?</h3><p>Taking a closer look at the Bitcoin address specified in the ransom note reveals the activity. Five separate transactions were made to this address in the last few days, combined, bringing in just over $2,400 USD. Notably, each time funds were transferred to it, they were swiftly moved to another wallet. Unfortunately, these victims would never receive their data back.‍</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PqETP3CpclT1i1Rx.png" /><figcaption><a href="https://bitinfocharts.com/bitcoin/address/164hyKPAoC5ecqkJ2ygeGoGFRcauWRLujV">https://bitinfocharts.com/bitcoin/address/164hyKPAoC5ecqkJ2ygeGoGFRcauWRLujV</a></figcaption></figure><h3>‍The same Bot is targeting MySQL databases</h3><p>Amid my investigation, an intriguing parallel surfaced involving MySQL databases. The same bot attacks MySQL databases and comes from various IP addresses within the same /24 range. However, there were subtle differences in its approach. Notably, when sifting through the data, the bot imposed a LIMIT of 10 rows, again stopping short of a complete data extraction. Following this, it systematically drops all tables and databases. In a move mirroring its PostgreSQL tactics, it established a new database named <em>RECOVER_YOUR_DATA</em>, which has a table of the same name, <em>RECOVER_YOUR_DATA</em>. This table contains the same ransomware message we observed earlier, including the same Bitcoin wallet address. However, curiously, the ransom price for MySQL database is 0.017 BTC (USD $732), about double the PostgreSQL price.</p><p>As a last step, the bot attempted to bring the MySQL server to a halt using the ‘<em>SHUTDOWN</em>’ command — a clear sign of its calculated and destructive intent, something that will undoubtedly get your attention.</p><h3>Exposing your database by accident is easier than you think.</h3><p>Ok, I know what you’re thinking: who the heck exposes their database to the Internet like that? You got what you deserved!<br>‍<br>Well, yes and no! Unfortunately, exposing your database publicly to the Internet is asking for trouble! So ideally, you don’t do that or find a way to protect yourself from drive-by attacks like these. At least set a strong password.<br>So, although it’s not the best idea to leave your database wide open, it’s also not as uncommon as you might think. Perhaps you’ve done it yourself; sometimes it’s just convenient.‍</p><p>A quick <a href="https://www.shodan.io/search?query=PostgreSQL">check at Shodan.io</a>, shows us that there are about 833,666 publicly accessible PostgreSQL servers available and over <a href="https://www.shodan.io/search?query=mysql">3.2 Million MySQL databases!</a> Lots of potential targets. Most of them run in the public cloud providers like AWS, GCP, DigitalOcean, etc. But, interestingly <a href="https://www.shodan.io/search/facet?query=PostgreSQL&amp;facet=org">the number two on the list</a> is a Polish provider called home.pl (AS12824), for which shodan reports over 82,000 publicly reachable PostgreSQL servers.‍</p><p>It’s not surprising to see many open database services in the public cloud. If you run your database in, say, DigitalOcean or even AWS, then these cloud providers don’t always make it easy to access your database from your desktop, or even a workload running in a different region or provider. You may have no other option than to open it from anywhere. And so, while bad practice, it’s not all that surprising that there are that many open databases.‍</p><p><em>Also, for Docker users, it’s important to know that using </em><strong><em>docker run -p</em></strong><em> to publish a container’s port alters your iptables rules for DNAT-based port forwarding. This Docker feature manages external communication for containers, overriding any default deny settings in your iptables INPUT table, thus making the port accessible publicly.</em>‍</p><h3>A better way to expose your database</h3><p>If you need help with making your database available in an easy, low-friction way to just the users who should have access but don’t want to expose your database to the whole world, then take a look at <a href="https://www.border0.com/">Border0</a>!‍</p><p>The video below demonstrates how easy it is to make your PostgreSQL database available using Border0.‍</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FGW2f53GWt0s&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DGW2f53GWt0s&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FGW2f53GWt0s%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e7eb0441dadeca0f8fe16ff582c2c16/href">https://medium.com/media/7e7eb0441dadeca0f8fe16ff582c2c16/href</a></iframe><p>‍Using <a href="https://docs.border0.com/docs/access-to-a-postgres-server">Border0</a> you can make your database available to your fellow team members, contractors and anyone you determine should have access. All without the need for a VPN, while still allowing you to block access from the whole Internet. All while just needing your single sign-on credentials. So best of both worlds!‍</p><p>So why settle for the outdated confines of conventional access management? Step into the enhanced adaptability and governance that Border0’s policies offer and leverage the transformation in access management. Take the opportunity to explore the advantages of Border0 firsthand by <a href="https://portal.border0.com/register">registering for our free, full-featured community edition today</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=852dfe8f08c6" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/help-my-database-was-compromised-852dfe8f08c6">Help! My database was compromised!</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>