<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Andrew Clunis on Medium]]></title>
        <description><![CDATA[Stories by Andrew Clunis on Medium]]></description>
        <link>https://medium.com/@orospakr?source=rss-f2794fe193c2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*38rOCShp9GN1LjTE.</url>
            <title>Stories by Andrew Clunis on Medium</title>
            <link>https://medium.com/@orospakr?source=rss-f2794fe193c2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 03:56:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@orospakr/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Bufferbloat Demystified]]></title>
            <link>https://medium.com/@orospakr/bufferbloat-demystified-2d94f4bfa665?source=rss-f2794fe193c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d94f4bfa665</guid>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[internet]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Andrew Clunis]]></dc:creator>
            <pubDate>Tue, 22 Jan 2013 00:00:00 GMT</pubDate>
            <atom:updated>2016-09-25T21:35:12.693Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<p>This is my attempt to distill the lore of bufferbloat, and describe how you can mitigate it on your network.</p><p><em>Originally published on </em><a href="http://www.orospakr.ca/blog/2013/01/22/bufferbloat-demystified/"><em>www.orospakr.ca</em></a><em> on January 22, 2013. Re-publishing it on Medium to give the platform a try. It’s still valuable, albeit a touch out of date.</em></p><h3>Symptoms</h3><p>Saturation of network links causes massive amounts of latency and slowdown, particularly on slow uplinks of asymmetric residential internet service (however, it can and does occur on the downlinks of the same). Network applications doing big bulky downloads can cause lots of latency on the network, causing real-time applications (VOIP, gaming, or SSH) to lag badly or outright fail.</p><h3>Principles</h3><p>(this is a refresher on certain fundamentals of how the Internet works; you can skip this bit if you already know the gist of it)</p><p>Internet networks are fundamentally packetized, and they do not guarantee packet delivery, and this greatly simplifies switching and multiplexing of multiple traffic flows on the same network. As such, network protocols, if they wish to guarantee packet arrival, must make their own arrangements (typically with acknowledgement messages and retransmits).</p><p>TCP is a protocol that endeavours to present a pipe-like streaming interface to applications that care about content delivery and not about real-time latency (HTTP being one of the most famous users of it). As it must do so on top of the packetized network, it needs to arrange for an appropriate amount of data to be “in flight” in packets travelling across the network. If it naïvely waited for acknowledgements for single packets, much of the network capacity between the endpoints would sit empty as that single packet traverses all of the hops. Thus, it implements a <a href="http://en.wikipedia.org/wiki/Transmission_Control_Protocol#Congestion_control">congestion control “windowing” arrangement</a> in an attempt to detect the appropriate amount of data to have in flight for optimum transfer speed. This is why download speeds start slow and grow to some maximum.</p><p>Network hardware (switches, modems, adapters, and even software drivers for those adapters) maintain buffers of packets in order to avoid momentary packet losses endemic or other implementation details specific to their physical layer, and the statistic probability of packets to occasionally stack in packet networks.</p><h3>Pathology of Bufferbloat</h3><p>Many network hardware designers merely assumed that more is better and added huge first-in-first-out buffers to many of their devices. However, the problem of how much buffer to include in a device (and how to prioritize packets in it) for best effect proved to be non-trivial. What was not well appreciated by the hardware designers was that packet networks must forward a packet promptly or drop it. Without that core assumption, the higher level protocols cannot operate.</p><p>Quite simply, these giant, unmanaged buffers are prone to carrying a balance rather than merely temporarily absorbing statistical transients, and thus packets (even those of very latency sensitive protocols, like gaming) can find themselves badly delayed as the buffer empties at the rate the network device can transmit the packets. Protocols like TCP, without being able to know otherwise, can inadvertently pump these buffers full (particularly at any bottlenecks) across the network as they attempt to optimize the amount of in-flight data by pipe size. In short, excess queuing of packets in these buffers make the network path seem longer than it is.</p><p>Thus, bufferbloat is the cause of most of the symptoms of network congestion.</p><h3>How it tends to happen at edge (read: your) networks</h3><p>Because this tends to occur at bottlenecks, the upstream and downstream links to your ISP (in that order) are the likeliest places for bufferbloat to appear in general use of the global network. However, fast, local transfers between two stations on your network can cause bufferbloat to occur directly on your local WiFi or Ethernet devices.</p><h3>How it’s getting solved</h3><p>The problem has been noticed for many years, although the pathology has only become well understood and publicized in the last few. Anyone who has set the upload limit on their BitTorrent client to just shy of their upstream bandwidth to their ISP was trying to counteract the effects of bufferbloat.</p><p>ISPs, dismayed at the bad congestion causing service interruptions for many of their customers (and not just those running bandwidth-expensive applications), implemented much-maligned “traffic shaping” regimes that restricted traffic identified as a specific protocol to certain speeds, trying to do the same. This was exacerbated by the fact that they oversubscribed themselves, knowing that most customers don’t use the full capacity of their links, or only for short periods.</p><p>That approach was worse than ineffective, in that not only did it fail to solve the root problem of delays induced by saturated buffers, it incentivized any self-interested party on the network (and, by extension, the designers implementing the software and protocols they’re using) to come up with technological contrivances (BitTorrent encryption, dancing their TCP port numbers, and so on) to obfuscate their traffic to make it escape the shapers’ heuristics.</p><p>There have been various abortive attempts at creating <a href="http://en.wikipedia.org/wiki/Active_Queue_Management">Active Queue Management</a> strategies such as <a href="http://en.wikipedia.org/wiki/Random_early_detection">RED</a>. However, most of them require the use of various tuning knobs and/or being informed of congestion adjacent on the network with <a href="http://en.wikipedia.org/wiki/Explicit_Congestion_Notification">ECN</a>. With all of the additional complexity, failure cases, and overhead that implies, it’s been difficult to implement and is not well suited to plug and play hardware devices.</p><p>Around early 2012, Kathleen Nichols and Van Jacobson announced a new AQM algorithm they entitled <a href="http://queue.acm.org/detail.cfm?id=2209336">CoDel</a>, which drops packets from buffers when the minimum time spent by any packet in the queue of the device exceeds some generally appropriate preset value, 100ms by default. This means that it permits buffers to fulfill their original function of smoothing statistical spikes (“good” queue) but prevents any perpetual queuing in the buffers (“bad” queue). It doesn’t require any configuration, nor does it require notifications from other hops. It’s simple to implement (mostly, for there can be multiple layers of buffers in both hardware and software which also need to be similarly managed), even for hardware. Certain link layers, like Wifi, are more complex and require trickier implementation, but they’re coming along. Linux already has it (<em>fq_codel</em> in the kernel and support in the <em>tc</em> utility), at least for wired Ethernet. Check out <a href="http://www.bufferbloat.net/projects/cerowrt">CeroWrt</a> for a router firmware project (forked from <a href="http://www.openwrt.org/">OpenWrt</a>) that implements CoDel (and other techniques) for certain models of the Netgear WNDR3700/3800. The work is getting upstreamed into back OpenWrt and other router firmware projects, so CoDel is now there, too (12.09 and later, when you enable the QoS feature, automatically enable CoDel).</p><h3>How to mitigate modem-side bloat on your home Internet connection</h3><p>While CoDel solves the problem of bad buffering in each device’s own buffers, bad buffering in unfixed adjacent devices is not affected. As such, in the case of the residential setup, ensuring that CoDel is active on your computers and your router is insufficient for dealing with bad buffering in the modem and the CO headend of your ISP. Both cable and DSL modems are notorious for having inappropriately sized buffers. As they are hardware, you’re basically limited to waiting for better gear to show up.</p><p>This is particularly unfortunate, because in general residential use, the bottleneck of the Internet connection itself (particularly upstream) is the most common place for bufferbloat to occur.</p><p>To demonstrate, I load up my upstream with TCP:</p><pre>scp ubuntu-12.10-alternate-amd64.iso orospakr.ca:/tmp</pre><p>Saturating your downstream will produce similar results, although because of the asymmetric nature of residential service (downstream is often an order of magnitude faster) it is harder.</p><p>Then, I run the excellent <a href="http://www.bitwizard.nl/mtr/">mtr</a> program against some other site out on the Internet, in order to measure the round trip time for pings to each hop:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/0*ryegRYUQJOBI4Lr_.png" /></figure><p>Gross. More than half a second of lag at the first hop inside the ISP and by extension anywhere beyond. Any gaming or VOIP would be a total goner here, and browsing would get decidedly laggy. In this circumstance (upstream is bloated), the delay is introduced during departure and not return. Notice that except for those two weird hops between Hurricane Electric and TORIX, there is no loss at all. Huge lag with no packet loss is characteristic of bufferbloat.</p><p>All hope is not lost, however. You can ameliorate modem-side bloat by configuring the global bandwidth limiting function on your router software to limit throughput to the Internet to the (observed, or least advertised) link rates (of both upstream and downstream) of your connection to your ISP. The limiter works by watching the throughput across the interface and arbitrarily dropping packets if the throughput exceeds the limit. As this procedure thus applies both send and receive, it means that you’re able to convince senders both inside and outside your network to back off, preventing them from pumping up the rogue buffers. While this approach is clearly not ideal — the settings must match the actual link rates of the two streams of your ISP connection and you probably lose any benefit of the buffers themselves — it does a fair job of mitigating the hidden modem bloat. I’m quite happy with it.</p><p>In this example, I’m running <a href="https://openwrt.org/">OpenWrt</a> and added the <em>luci-app-qos</em> package. I’ve multiplied 28 (my ISP’s advertised downstream link rate, in megabits) by 1024 to kbit/s, and 1 (ISP’s upstream link rate, in megabits) by 1024.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/0*jsgbpclH1E01OwKr.png" /></figure><p>This sets a single Linux <a href="http://linux-ip.net/articles/hfsc.en/">HFSC</a> traffic shaping group for all traffic with for both bandwidth limits for upstream and downstream, just under the capacity of your modem’s link rates.</p><p>Notice the presence of the traffic classification rules (which set SSH and DNS as high priority). These came as default with Openwrt’s QoS package. While router owners have tried to use such rules for years to deal with the then-unnamed problem of bufferbloat, they are generally ineffective and not worth it. For example, after I had set the limits and testing the system by saturating upstream, I was loading upstream with SSH in my tests, oblivious of the rule <em>prioritizing</em> SSH. But even though the protocol I was using to load the network <em>was prioritized above all else</em>, the network latency remained solid. The purpose of these is more to allocate a greater portion of pipe size to certain traffic, rather than the default behaviour of the streams to more or less divide pipe size evenly between them. In short: folks who think they want classification shaping rules (ie., because they have latency problems) probably don’t and actually are having bufferbloat issues instead.</p><p>The results, after starting my upload back up again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/0*GPZF7zPOBhDZGbjn.png" /></figure><p>Much better (please ignore those two weird hops). Indistinguishable from when no upload is happening at all.</p><p>This technique, it is worth reiterating, will have no effect on <em>local</em> bufferbloat happening between stations or between a station and the router box. This can happen if you’re transferring files between two local boxes at full speed, particularly on wireless. By far the best solution here is full deployment of CoDel (including the messy details of making it work on wireless) on all of the local devices. Sadly, it may be some time yet before consumer wireless devices and Windows implement it. Thankfully, the most commonly encountered form on home Internet connectivity (ie., modem/ISP bloat) can be ameliorated with the bandwidth limit.</p><h3>TL; DR</h3><p>Bufferbloat…</p><ul><li>… is caused by oversized or poorly managed buffers in network devices that hold a constant queue of packets entering and leaving at the link rate of the connection.</li><li>… tends to appear at bottlenecks, like your cable modem (especially upstream).</li><li>… is best fixed by ensuring that buffers don’t “carry a balance” of “bad queue”, and that they are only for eating transient spikes. Fancy throttling or prioritization rules are not the best approach. CoDel is the best bet for this so far, and software and hardware vendors are already making improvements.</li><li>… in your modem and ISP can be mitigated in the meantime by setting your QoS bandwidth limits in your router box to the linkspeeds of your ISP.</li></ul><h3>Further reading</h3><p>I hope this has proved useful. Enjoy!</p><ul><li><a href="http://queue.acm.org/detail.cfm?id=2209336">Kathleen Nichols and Van Jacobson’s ACM paper describing CoDel</a></li><li><a href="http://www.bufferbloat.net/">bufferbloat.net</a></li><li><a href="http://www.youtube.com/watch?v=qbIozKVz73g">Video: Jim Gettys — Bufferbloat: Dark Buffers in the Internet</a></li><li><a href="http://www.bufferbloat.net/projects/cerowrt">CeroWrt — a branch of OpenWrt created as a testbed for bufferbloat solutions</a></li><li><a href="http://gettys.wordpress.com/">Jim Gettys’ blog</a></li></ul><p>Note: when I originally published this in 2013, <a href="https://medium.com/u/5eff5f6ec3f2">Dave Taht</a> commented that this article did not guide users to using the fair queue version of codel:</p><blockquote>Good piece, but you failed to distinguish between codel and fq_codel. fq_codel is what qos-scripts uses, and is vastly superior to codel alone.</blockquote><p>Thank you!</p><p><em>Originally published on </em><a href="http://www.orospakr.ca/blog/2013/01/22/bufferbloat-demystified/"><em>www.orospakr.ca</em></a><em> on January 22, 2013.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d94f4bfa665" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>