<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Brandon Cook on Medium]]></title>
        <description><![CDATA[Stories by Brandon Cook on Medium]]></description>
        <link>https://medium.com/@phylake?source=rss-564e781ac88------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*T-34F83je9KiM3pu.jpg</url>
            <title>Stories by Brandon Cook on Medium</title>
            <link>https://medium.com/@phylake?source=rss-564e781ac88------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 20:21:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@phylake/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Bottom-up eBPF]]></title>
            <link>https://medium.com/@phylake/bottom-up-ebpf-d7ca9cbe8321?source=rss-564e781ac88------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7ca9cbe8321</guid>
            <category><![CDATA[bpf]]></category>
            <category><![CDATA[linux-kernel]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Brandon Cook]]></dc:creator>
            <pubDate>Wed, 06 Nov 2019 03:01:46 GMT</pubDate>
            <atom:updated>2024-04-25T22:52:08.365Z</atom:updated>
            <content:encoded><![CDATA[<p>With all the articles on eBPF it’s starting to remind me of <a href="https://wiki.haskell.org/Monad_tutorials_timeline">Monads</a>: people seem to either understand them fully or not at all, and will happily write articles about them either way. I’m probably still in the latter group even after years using them in Haskell. Ask me to invent a Monad? Forget about it.</p><p>The issue I’ve had with a lot of the bpf material out there is that the majority is introductory. The remainder seems to be for higher level frameworks such as BCC but, of course, I haven’t clicked on every link I’ve found. The higher level frameworks are going to take a top-down approach because the whole point is to teach the tool that makes bpf easier to write and/or interact with. They also bring in more dependencies and are arguably heavier weight.</p><p>I wanted to write a lightweight, always-on, runs-everywhere program which meant learning bpf from the bottom up.</p><p><em>Note: I’m not a bpf or kernel expert by any means, I just want to share some hard-won knowledge.</em></p><p><em>This article is based on Linux 4.19</em></p><h3>Kernel and bpf program headers</h3><h4>include/linux/bpf.h</h4><p>This is a header for the kernel implementation of bpf. Not much to look at if you’re trying to write a bpf program, however.</p><h4>include/uapi/linux/bpf.h</h4><p>This is the header to include as &lt;linux/bpf.h&gt; in a bpf program.</p><p>You should definitely have a look at the bpf/3 (<a href="http://man7.org/linux/man-pages/man2/bpf.2.html">link</a>; the /3 is function <a href="https://en.wikipedia.org/wiki/Arity">arity</a>) system call man page to get familiar with some of the concepts. It disappoints in the <em>eBPF program types</em> section with, as of this writing, only BPF_PROG_TYPE_SOCKET_FILTER is documented. Time to look at some more source code.</p><p><strong>bpf enums</strong></p><p>The first parameter of the bpf/3 system call is an int cmd. Those are found on enum bpf_cmd (<a href="https://github.com/torvalds/linux/blob/v4.19/include/uapi/linux/bpf.h#L83-L106">link</a>). So rather than a bpf_map_create() system call, there’s just the single bpf/3 system call with a int cmd of BPF_MAP_CREATE</p><p>The next two enums are bpf_map_type (<a href="https://github.com/torvalds/linux/blob/v4.19/include/uapi/linux/bpf.h#L108-L130">link</a>) and, missing from the bpf man page is the full list of program types in bpf_prog_type (<a href="https://github.com/torvalds/linux/blob/v4.19/include/uapi/linux/bpf.h#L132-L155">link</a>).</p><p>Not all program types are loaded in the same way. How you load a bpf program is typically with a “frontend” loader. BCC and the bpf samples in the kernel are frontends which use libbpf as a common “backend”. The iproute2 package provides the ip command frontend which is used to attach XDP and tc programs to network interfaces. The <a href="http://docs.cilium.io/en/stable/bpf/#iproute2">Cilium docs</a> provide great examples of this. I’m not sure if ip uses libbpf as a backend. Ultimately, all roads lead to the bpf/3 system call.</p><p><strong>bpf helper functions</strong></p><p>A <a href="https://github.com/torvalds/linux/blob/v4.19/include/uapi/linux/bpf.h#L414-L2144">1730 line comment</a> documents various bpf helper functions that are available. Not all of them are defined in this file, nor are all the available helper functions documented here. Some of them are only available if you set the license of your bpf program to GPL.</p><p>For example, bpf_getsockopt (<a href="https://github.com/torvalds/linux/blob/v4.19/net/core/filter.c#L4038-L4107">link</a>), is available to non-GPL bpf programs. Notice .gpl_only = false</p><p>bpf_skb_event_output (<a href="https://github.com/torvalds/linux/blob/v4.19/net/core/filter.c#L3472-L3495">link</a>), by contrast, is only available to GPL-licensed bpf programs.</p><p>The helper functions used, as well as the header files included have important licensing implications for bpf programs you might write. As I understand it the Linux Kernel stayed with GPL 2.0 so the bpf programs licensed “GPL” are implicitly the 2.0 variety, not 2.x or 3.0 or others. I refer you to the <a href="https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.en.html">GPL FAQ</a>, <a href="https://github.com/torvalds/linux/blob/v4.19/COPYING">COPYING</a>, and the <a href="https://github.com/torvalds/linux/blob/v4.19/LICENSES/exceptions/Linux-syscall-note">Linux-syscall-note</a> for more info. <a href="https://tldrlegal.com/">tl;drLegal</a> is a good source for general licensing info as well. None of the above should be interpreted as legal advice and is not a substitute for legal advice.</p><p><strong>struct __sk_buff</strong></p><p>Notably, you can’t use struct sk_buff (defined in include/linux/skbuff.h) in bpf programs, but there’s a struct __sk_buff (<a href="https://github.com/torvalds/linux/blob/v4.19/include/uapi/linux/bpf.h#L2302-L2336">link</a>) alternative. Other structs in headers in the very same include path can be pulled in, such as struct sock defined in include/net/sock.h. Again, I’m not a kernel developer or it would probably be obvious why sk_buff is special in this regard. More context can be found in <a href="https://lwn.net/Articles/636647/">Alexei Starovoitov’s patch</a> that first introduced struct __sk_buff</p><p>Most importantly, __sk_buff <em>can</em> be substituted for sk_buff in kernel function signatures that we’d like to probe, such as</p><pre>int dev_queue_xmit(struct sk_buff *skb);</pre><p>A bpf kprobe of dev_queue_xmit loaded by libbpf would look like this</p><pre>SEC(&quot;kprobe/dev_queue_xmit&quot;)<br>int any_name(struct __sk_buff *skb); // __sk_buff, not sk_buff</pre><p>SEC is covered next.</p><h4>tools/testing/selftests/bpf/bpf_helpers.h</h4><p>SEC (<a href="https://github.com/torvalds/linux/blob/v4.19/tools/testing/selftests/bpf/bpf_helpers.h#L5-L9">link</a>) is a macro that expands to create an <a href="https://en.wikipedia.org/wiki/Executable_and_Linkable_Format">ELF</a> section which bpf loaders such as libbpf parse.</p><p>For XDP and tc programs, ip looks for the ELF section called “prog” by default. You can optionally specify a section name when running ip</p><p>libbpf defined the “kprobe/” convention you saw above, and as you’ll see below.</p><p>Other SECtions include “maps” which is included in struct bpf_map_def variables, “license”, and “version” which is used to compare the <em>running</em> kernel version with the kernel version <em>compiled against</em> to <a href="https://github.com/torvalds/linux/blob/v4.19/kernel/bpf/syscall.c#L1359-L1361">reject mismatches</a> since kprobes don’t guarantee ABI compatibility.</p><p>The other notable macros in this file are the PT_REGS_* (<a href="https://github.com/torvalds/linux/blob/v4.19/tools/testing/selftests/bpf/bpf_helpers.h#L239-L250">link</a>) for portably accessing registers on the architecture-dependent struct pt_regs which can be the context for BPF_PROG_TYPE_KPROBE and BPF_PROG_TYPE_TRACEPOINT.</p><h4>tools/testing/selftests/bpf/bpf_endian.h</h4><p><a href="https://github.com/torvalds/linux/blob/v4.19/tools/testing/selftests/bpf/bpf_endian.h">bpf_endian.h</a> is worth a mention because I’ve used it in nearly every bpf program I’ve written to convert CPU architecture endianness to network byte order and back.</p><h3>User space headers and tools</h3><p>The previous section was specific to the Kernel and bpf programs themselves. Most bpf programs I’ve seen or imagine writing will have a user space counterpart to modify the bpf program behavior, extract data, etc. All of this is done via maps which we’ll see in the examples section.</p><h4>tools/lib/bpf/libbpf.h</h4><p>libbpf provides a level of abstraction over the bpf/3 system call. No abstraction means using the union bpf_attr, enum bpf_cmd, and more found in include/uapi/linux/bpf.h directly. I certainly don’t want to define individual struct bpf_insn; it’s like writing assembly by hand, which s<a href="https://bolinfest.github.io/opensnoop-native/#part-3-standalone-ebpf-program-without-libbpf">ome have done.</a></p><p>The user space distinction is important because bpf_map_def (<a href="https://github.com/torvalds/linux/blob/v4.19/tools/testing/selftests/bpf/bpf_helpers.h#L158-L169">kernel</a>, <a href="https://github.com/torvalds/linux/blob/v4.19/tools/lib/bpf/libbpf.h#L221-L233">user space</a>) and bpf_map (<a href="https://github.com/torvalds/linux/blob/v4.19/include/linux/bpf.h#L56-L87">kernel</a>, <a href="https://github.com/torvalds/linux/blob/v4.19/tools/lib/bpf/libbpf.h#L235-L239">user space</a>) have “duplicate” definitions in kernel headers we’ve already seen. They won’t collide if you use them in their respective kernel/user space but they can be easy to mix up in your head at first.</p><p>I mentioned that libbpf defines the “kprobe/” convention. That happens <a href="https://github.com/torvalds/linux/blob/v4.19/tools/lib/bpf/libbpf.c#L2104-L2123">here</a> along with all the other program types.</p><h4>samples/bpf/bpf_load.h</h4><p>I’d characterize this as both an abstraction on libbpf and a minimal framework which has some really nice features. Notice the very straightforward int load_bpf_file(char *path) (<a href="https://github.com/torvalds/linux/blob/v4.19/samples/bpf/bpf_load.h#L41-L53">link</a>). Like all frameworks it comes with some opinions built in. If you agree with those opinions then you have a great time with the framework; if not you can find an alternative, drop down to libbpf, or call bpf/3 directly. I was really happy when I found this because it let me focus on the bpf program I was writing while getting past compilation hurdles.</p><p>extern int map_fd[MAX_MAPS] is especially nice because bpf_load.c automatically pins maps defined in the bpf program to the bpf filesystem so they’re easily accessible in a user space program in the order they appear (e.g. the first map definition is in map_fd[0]).</p><p>I’m not sure what the relationship between kprobes and the perf system is but trying to load a kprobe bpf program with libbpf by itself didn’t work for me. I found that all the sample bpf programs were using bpf_load.h, the implementation of which ultimately calls sys_perf_event_open (<a href="https://github.com/torvalds/linux/blob/v4.19/samples/bpf/bpf_load.c#L227-L244">link</a>) and my program loaded <em>and ran</em> as I intended.</p><h4>tools/bpftool/</h4><p>The <a href="http://docs.cilium.io/en/stable/bpf/#bpftool">Cilium docs</a> already have great coverage on bpftool so I won’t repeat any of that here.</p><h3>Example programs</h3><h4>Build prerequisites</h4><p>A compiled Kernel is required to compile the bpf program. Here’s a <a href="https://kernelnewbies.org/KernelBuild">Guide to building the Linux Kernel</a>.</p><p>On Fedora I installed the following packages (not all of them are build requirements):</p><pre>git gcc ncurses-devel elfutils-libelf-devel bc wget bzip2 vim<br>openssl-devel libcap-devel clang llvm graphviz bison flex make<br>glibc-static glibc-devel.i686 iproute-devel binutils-devel diffutils<br>findutils net-tools lsof iproute</pre><h4>XDP bpf example</h4><p>This simple program drops traffic from a particular IP address. Figure out from which IP address you want to drop traffic and turn it into its decimal form. The example drops traffic from 16909060</p><p><strong>xdp.c</strong></p><pre>// This program is free software: you can redistribute it and/or<br>// modify it under the terms of the GNU General Public License as<br>// published by the Free Software Foundation, either version 2 of<br>// the License, or (at your option) any later version.</pre><pre>// This program is distributed in the hope that it will be useful,<br>// but WITHOUT ANY WARRANTY; without even the implied warranty of<br>// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU<br>// General Public License for more details.</pre><pre>// Since this file was originally published in a blog post, a copy<br>// of the GNU General Public License version 2 was not included but<br>// can be found at<br>// <a href="https://www.gnu.org/licenses/old-licenses/gpl-2.0.html">https://www.gnu.org/licenses/old-licenses/gpl-2.0.html</a><br>#include &lt;linux/bpf.h&gt;<br>#include &lt;linux/version.h&gt;<br>#include &lt;linux/ip.h&gt;<br>#include &lt;linux/if_ether.h&gt;<br>#include &lt;bpf_helpers.h&gt;<br>#include &lt;bpf_endian.h&gt;</pre><pre>// iproute specifically looks for this ELF section<br>SEC(&quot;prog&quot;)<br>int xdp_drop(struct xdp_md *xdp)<br>{<br>    void *data_end = (void *)(unsigned long)xdp-&gt;data_end;<br>    void *data = (void *)(unsigned long)xdp-&gt;data;</pre><pre>    // include/uapi/linux/if_ether.h<br>    struct ethhdr *eth = data;</pre><pre>    // needed to pass the bpf verifier<br>    if ((void*)(eth + 1) &gt; data_end)<br>        return XDP_PASS;</pre><pre>    // we&#39;re only interested in IPv4 traffic<br>    if (eth-&gt;h_proto != bpf_htons(ETH_P_IP))<br>        return XDP_PASS;</pre><pre>    // include/uapi/linux/ip.h<br>    struct iphdr *ipv4_hdr = data + sizeof(struct ethhdr);</pre><pre>    // needed to pass the bpf verifier<br>    if ((void *)(ipv4_hdr + 1) &gt; data_end)<br>        return XDP_PASS;</pre><pre>    // check if the saddr matches<br>    if (ipv4_hdr-&gt;saddr == bpf_htonl(16909060))<br>        return XDP_DROP;</pre><pre>    return XDP_PASS;<br>}</pre><pre>char _license[] SEC(&quot;license&quot;) = &quot;GPL&quot;;<br>__u32 _version SEC(&quot;version&quot;) = LINUX_VERSION_CODE;</pre><p>Compile with</p><pre>clang -O2 -Wall -target bpf -c -o xdp.o \<br>-I &lt;path to kernel source&gt;/tools/testing/selftests/bpf \<br>-I &lt;path to kernel source&gt;/tools/include/uapi \<br>xdp.c</pre><p>Load the program into a network interface (in this case eth0) with</p><pre>ip link set dev eth0 xdp obj xdp.o</pre><p>You can set ip -force link … to reload the program. Unload it with</p><pre>ip link set dev eth0 xdp off</pre><h4>kprobe bpf example</h4><p>This program times how long it takes for TCP connections to establish by probing tcp_connect (<a href="https://github.com/torvalds/linux/blob/v4.19/net/ipv4/tcp_output.c#L3460">link</a>) and tcp_finish_connect (<a href="https://github.com/torvalds/linux/blob/v4.19/net/ipv4/tcp_input.c#L5655">link</a>)</p><p>The average of the deltas between CPU times is calculated in the user space program. I did this for program clarity, not because it’s optimal.</p><p>The internal struct sock *sk doesn’t change between the two function calls so we can take its address and use it as the key to a LRU hash. The value is the struct timings defined in kp.h</p><p><strong>kp.h</strong></p><pre>// This program is free software: you can redistribute it and/or<br>// modify it under the terms of the GNU General Public License as<br>// published by the Free Software Foundation, either version 2 of<br>// the License, or (at your option) any later version.</pre><pre>// This program is distributed in the hope that it will be useful,<br>// but WITHOUT ANY WARRANTY; without even the implied warranty of<br>// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU<br>// General Public License for more details.</pre><pre>// Since this file was originally published in a blog post, a copy<br>// of the GNU General Public License version 2 was not included but<br>// can be found at<br>// <a href="https://www.gnu.org/licenses/old-licenses/gpl-2.0.html">https://www.gnu.org/licenses/old-licenses/gpl-2.0.html</a><br>#ifndef KP_EXAMPLE_H_<br>#define KP_EXAMPLE_H_</pre><pre>struct timings {<br>    __u64 t0;<br>    __u64 t1;<br>};</pre><pre>#endif  // KP_EXAMPLE_H_</pre><p><strong>kp_kern.c</strong></p><p>This is the bpf program. Compiling it is probably the biggest challenge. With the right dependencies installed and the Kernel built I was able to run</p><pre>clang -v -c -O2 -emit-llvm -o kp_kern.ir -D__KERNEL__ \<br>-Wall \<br>-Wno-gnu-variable-sized-type-not-at-end \<br>-Wno-address-of-packed-member \<br>-Wno-tautological-compare \<br>-Wno-unknown-warning-option \<br>-include &lt;path to kernel source&gt;/include/linux/kconfig.h \<br>-I &lt;path to kernel source&gt;/arch/x86/include \<br>-I &lt;path to kernel source&gt;/arch/x86/include/uapi \<br>-I &lt;path to kernel source&gt;/arch/x86/include/generated \<br>-I &lt;path to kernel source&gt;/arch/x86/include/generated/uapi \<br>-I &lt;path to kernel source&gt;/include \<br>-I &lt;path to kernel source&gt;/include/uapi \<br>-I &lt;path to kernel source&gt;/include/generated \<br>-I &lt;path to kernel source&gt;/include/generated/uapi \<br>-I &lt;path to kernel source&gt;/tools/testing/selftests/bpf \<br>kp_kern.c</pre><pre>llc -march=bpf -filetype=obj -o kp_kern.o kp_kern.ir</pre><p>The generated folders and include/linux/kconfig.h were both produced during Kernel compilation.</p><p>Notice the two-step compilation, first with clang then with llc. Passing -target bpf to clang fails when pulling in headers that transitively include headers with inline assembly. In this case it’s include/linux/bpf.h that transitively uses arch/x86/include/asm/msr.h</p><pre>In file included from kp_kern.c:15:<br>In file included from include/linux/bpf.h:12:<br>In file included from include/linux/workqueue.h:9:<br>In file included from include/linux/timer.h:6:<br>In file included from include/linux/ktime.h:24:<br>In file included from include/linux/time.h:6:<br>In file included from include/linux/seqlock.h:36:<br>In file included from include/linux/spinlock.h:51:<br>In file included from include/linux/preempt.h:81:<br>In file included from arch/x86/include/asm/preempt.h:7:<br>In file included from include/linux/thread_info.h:38:<br>In file included from arch/x86/include/asm/thread_info.h:53:<br>In file included from arch/x86/include/asm/cpufeature.h:5:<br>In file included from arch/x86/include/asm/processor.h:21:<br>In file included from arch/x86/include/asm/msr.h:67:<br>/arch/x86/include/asm/atomic.h:167:13: error: invalid output<br>constraint &#39;+q&#39; in asm<br>        return i + xadd(&amp;v-&gt;counter, i);<br>                   ^<br>/arch/x86/include/asm/cmpxchg.h:234:25: note: expanded from macro<br>&#39;xadd&#39;</pre><pre>#define xadd(ptr, inc)          __xadd((ptr), (inc), LOCK_PREFIX)<br>                                ^</pre><pre>/arch/x86/include/asm/cmpxchg.h:233:32: note: expanded from macro<br>&#39;__xadd&#39;</pre><pre>#define __xadd(ptr, inc, lock)  __xchg_op((ptr), (inc), xadd, lock)<br>                                ^<br>/arch/x86/include/asm/cmpxchg.h:48:13: note: expanded from macro<br>&#39;__xchg_op&#39;<br>                                      : &quot;+q&quot; (__ret), &quot;+m&quot; (*(ptr))<br>                                        ^</pre><p>That’s a pretty gnarly error that wallpapered my terminal and took me 2 long days to figure out — in part because I had first worked with the XDP program above. The additional include paths introduced the error but I had no reason to think -target bpf was the culprit. This issue also appears on clang versions lower than 9.0 even without -target bpf</p><p>Ok, on to the source:</p><pre>// This program is free software: you can redistribute it and/or<br>// modify it under the terms of the GNU General Public License as<br>// published by the Free Software Foundation, either version 2 of<br>// the License, or (at your option) any later version.</pre><pre>// This program is distributed in the hope that it will be useful,<br>// but WITHOUT ANY WARRANTY; without even the implied warranty of<br>// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU<br>// General Public License for more details.</pre><pre>// Since this file was originally published in a blog post, a copy<br>// of the GNU General Public License version 2 was not included but<br>// can be found at<br>// <a href="https://www.gnu.org/licenses/old-licenses/gpl-2.0.html">https://www.gnu.org/licenses/old-licenses/gpl-2.0.html</a><br>#include &lt;linux/bpf.h&gt;<br>#include &lt;linux/version.h&gt;<br>#include &lt;net/sock.h&gt;<br>#include &lt;bpf_helpers.h&gt;</pre><pre>#include &quot;kp.h&quot;</pre><pre>struct bpf_map_def SEC(&quot;maps&quot;) sock_est = {<br>  .type = BPF_MAP_TYPE_LRU_HASH,<br>  .key_size = sizeof(__u64),<br>  .value_size = sizeof(struct timings),<br>  .max_entries = 1024,<br>};</pre><pre>SEC(&quot;kprobe/tcp_connect&quot;)<br>int connect(struct pt_regs *ctx)<br>{<br>  int err;</pre><pre>  struct timings ts = {<br>    .t0 = bpf_ktime_get_ns(),<br>    .t1 = 0<br>  };</pre><pre>  struct sock *sk = (void*) PT_REGS_PARM1(ctx);</pre><pre>  // sk pointer<br>  __u64 skp;<br>  if ((err = bpf_probe_read(&amp;skp, sizeof(__u64), &amp;sk))) {<br>    char log[] = &quot;bpf_probe_read %d\n&quot;;<br>    bpf_trace_printk(log, sizeof(log), err);<br>    return 1;<br>  }</pre><pre>  // note: map access is via the pointer to sock_est<br>  if ((err = bpf_map_update_elem(&amp;sock_est, &amp;skp, &amp;ts, BPF_ANY))) {<br>    char log[] = &quot;bpf_map_update_elem %d\n&quot;;<br>    bpf_trace_printk(log, sizeof(log), err);<br>    return 1;<br>  }</pre><pre>  return 0;<br>}</pre><pre>SEC(&quot;kprobe/tcp_finish_connect&quot;)<br>int finish_connect(struct pt_regs *ctx)<br>{<br>  int err;</pre><pre>  struct sock *sk = (void*) PT_REGS_PARM1(ctx);</pre><pre>  __u64 skp;<br>  if ((err = bpf_probe_read(&amp;skp, sizeof(__u64), &amp;sk))) {<br>    char log[] = &quot;bpf_probe_read %d\n&quot;;<br>    bpf_trace_printk(log, sizeof(log), err);<br>    return 1;<br>  }</pre><pre>  struct timings* t = bpf_map_lookup_elem(&amp;sock_est, &amp;skp);<br>  if (t &amp;&amp; t-&gt;t0) {<br>    t-&gt;t1 = bpf_ktime_get_ns();<br>    bpf_map_update_elem(&amp;sock_est, &amp;skp, t, BPF_ANY);<br>  }<br>  <br>  return 0;<br>}</pre><pre>char _license[] SEC(&quot;license&quot;) = &quot;GPL&quot;;<br>__u32 _version SEC(&quot;version&quot;) = LINUX_VERSION_CODE;</pre><p><strong>kp_user.c</strong></p><p>This is the user space program that will load the bpf program and interact with it via the map defined in kp_kern.c</p><p>When compiled I did it in a Docker container. The program and dependencies would be moved so I put libbpf in the same folder as kp_user.c and set rpath to ./. Adjust all of this as needed</p><pre>clang -O2 -Wall -Wno-unused-variable -o kp_user \<br>-I &lt;path to kernel source&gt;/tools/lib \<br>-I &lt;path to kernel source&gt;/tools/perf \<br>-I &lt;path to kernel source&gt;/tools/include \<br>-I &lt;path to kernel source&gt;/samples/bpf \<br>-Wl,--library-path=./,-rpath=./,--library=bpf,--library=elf \<br>kp_user.c &lt;path to kernel source&gt;/samples/bpf/bpf_load.c</pre><p>The map’s file descriptor is in maps[0] which was automatically populated as part of load_bpf_file</p><pre>// This program is free software: you can redistribute it and/or<br>// modify it under the terms of the GNU General Public License as<br>// published by the Free Software Foundation, either version 2 of<br>// the License, or (at your option) any later version.</pre><pre>// This program is distributed in the hope that it will be useful,<br>// but WITHOUT ANY WARRANTY; without even the implied warranty of<br>// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU<br>// General Public License for more details.</pre><pre>// Since this file was originally published in a blog post, a copy<br>// of the GNU General Public License version 2 was not included but<br>// can be found at<br>// <a href="https://www.gnu.org/licenses/old-licenses/gpl-2.0.html">https://www.gnu.org/licenses/old-licenses/gpl-2.0.html</a><br>#include &lt;stdlib.h&gt;<br>#include &lt;unistd.h&gt;<br>#include &lt;sys/resource.h&gt;</pre><pre>#include &lt;bpf/libbpf.h&gt;<br>#include &lt;bpf_load.h&gt;</pre><pre>#include &quot;kp.h&quot;</pre><pre>int main(int argc, char const *argv[]) {</pre><pre>  // raise the rlimit or see<br>  // failed to create a map: 1 Operation not permitted<br>  // when load_bpf_file is run<br>  int ret;<br>  struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};<br>  if ((ret = setrlimit(RLIMIT_MEMLOCK, &amp;r))) {<br>    printf(&quot;setrlimit %d\n&quot;, ret);<br>    return ret;<br>  }</pre><pre>  if (load_bpf_file(&quot;kp_kern.o&quot;)) {<br>    printf(&quot;%s\n&quot;, bpf_log_buf);<br>    return 1;<br>  }</pre><pre>  while (1) {<br>    __u64 next_key = 0;<br>    __u64 key = 0;</pre><pre>    unsigned long long sum = 0;<br>    int total = 0;<br>    int err = 0;<br>    while (bpf_map_get_next_key(map_fd[0], &amp;key, &amp;next_key) == 0) {<br>      struct timings t = {};</pre><pre>      if ((err = bpf_map_lookup_elem(map_fd[0], &amp;next_key, &amp;t))) {<br>        printf(&quot;bpf_map_lookup_elem failed %d\n&quot;, err);<br>        break;<br>      }</pre><pre>      if (t.t1) {<br>        sum += t.t1 - t.t0;<br>        ++total;<br>      }</pre><pre>      key = next_key;<br>    }</pre><pre>    if (sum) {<br>      printf(&quot;avg: %llu ns\n&quot;, sum/total);<br>    }</pre><pre>    sleep(2);<br>  }</pre><pre>  return 0;<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7ca9cbe8321" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why idle timeouts matter]]></title>
            <link>https://medium.com/@phylake/why-idle-timeouts-matter-1b3f7d4469fe?source=rss-564e781ac88------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b3f7d4469fe</guid>
            <category><![CDATA[tcp]]></category>
            <category><![CDATA[reverse-proxy]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[connection-reset]]></category>
            <dc:creator><![CDATA[Brandon Cook]]></dc:creator>
            <pubDate>Thu, 27 Jun 2019 23:02:18 GMT</pubDate>
            <atom:updated>2020-04-23T17:17:11.231Z</atom:updated>
            <content:encoded><![CDATA[<h4>They can be a hard-to-find source of 503s or connection resets</h4><p>If you’ve read the <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-idle-timeout.html">idle timeout docs for ELB</a> you might have wondered about this:</p><blockquote>To ensure that the load balancer is responsible for closing the connections to your instance, make sure that the value you set for the HTTP keep-alive time is greater than the idle timeout setting on your load balancer.</blockquote><p>I’ve finally found out why this matters, and it applies to more than just ELB. More generically, <em>any proxy or application server</em> needs an idle timeout greater than its downstream’s idle timeout, so that the downstream server is responsible for closing idle connections.</p><p>In the diagram below, the client is downstream of the proxy which is downstream of the application server. I mix L4 and L7 here just to make it simpler to diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4X2Bjy_qNK8O5Wqvx5XuEQ.png" /></figure><p>The reverse proxy saw a kept-alive connection it could use for Request 2 at the same time the application server’s idle timeout fired, causing it to close the connection. The reverse proxy got a segment with the RST bit set and, at the socket API level, received an ECONNRESET which is passed onto the client as a 503.</p><p>This is a race condition: the reverse proxy doesn’t know the upstream closed the connection. But had the reverse proxy had a lower idle timeout than the application server it would have been responsible for closing the connection, avoiding the race condition, and would <em>not</em> have tried to use the connection for an active request.</p><p>Incredibly simple when you have a diagram to illustrate it, but difficult to find and debug, especially if you don’t own/operate every layer in the stack.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b3f7d4469fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NLB Connection Resets]]></title>
            <link>https://medium.com/swlh/nlb-connection-resets-109720accfc6?source=rss-564e781ac88------2</link>
            <guid isPermaLink="false">https://medium.com/p/109720accfc6</guid>
            <category><![CDATA[tcp]]></category>
            <category><![CDATA[linux-kernel]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[systemtap]]></category>
            <category><![CDATA[nat]]></category>
            <dc:creator><![CDATA[Brandon Cook]]></dc:creator>
            <pubDate>Tue, 18 Jun 2019 15:33:26 GMT</pubDate>
            <atom:updated>2019-11-12T16:47:59.311Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using/consuming AWS’s NLB or NAT Gateway? There’s something you should know.</h4><p>I was load testing our next generation platform to get us to performance parity with our existing platform and kept getting connection resets at very low load.</p><p>To be clear, I didn’t find a bug. This is one of those issues AWS probably closed as “By design”.</p><h3>The setup</h3><p>The load generator is a single Locust.io agent (I was able to reproduce the connection resets with just one) in a VPC in a private subnet so it communicates through a NAT Gateway.</p><p>We have a 2-tier proxy setup: NLB at the edge and a reverse proxy where we’re able to do more intelligent L7 routing, traffic shaping, etc. Behind the reverse proxy is S<em>ome service </em>just for completeness, but it’s irrelevant for this post.</p><p>The load generation cluster and target cluster are in separate VPCs (not diagrammed). You can see 2 connections and that cross-zone load balancing is enabled.</p><p>(note: there’s not an actual connection to the Logical NLB it just makes the issue easier to diagram)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hijuFIYEd1MNwULJOITycA.png" /></figure><h3>Some background</h3><p>Interestingly enough, cross-zone is <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html">disabled in NLB by default</a> and I’m pretty sure I know why now.</p><blockquote>With Application Load Balancers, cross-zone load balancing is always enabled.</blockquote><blockquote>With Network Load Balancers, cross-zone load balancing is disabled by default. After you create a Network Load Balancer, you can enable or disable cross-zone load balancing at any time</blockquote><p><a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html">NLB preserves source IP:port</a> when specifying targets, which we do.</p><blockquote>If you specify targets using an instance ID, the source IP addresses of the clients are preserved and provided to your applications. If you specify targets by IP address, the source IP addresses are the private IP addresses of the load balancer nodes.</blockquote><p>From the <a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html">NAT Gateway docs</a>:</p><blockquote>A NAT gateway can support up to 55,000 simultaneous connections to each unique destination</blockquote><p>Finally, one last bit of background is needed: <a href="https://tools.ietf.org/html/rfc5961#section-4">RFC 5961 section 4.2</a>, and I encourage you to click the link on this one.</p><blockquote>The analysis of the reset attack using the RST bit highlights another possible avenue for a blind attacker using a similar set of sequence number guessing. Instead of using the RST bit, an attacker can use the SYN bit with the exact same semantics to tear down a connection.</blockquote><h3>How I got to RFC 5961</h3><p>If you’re confused, I don’t blame you. I was too. This implied NLB was attacking us.</p><p>The journey to the RFC started with a tcpdump for the reverse proxy which showed the connection resets (the proxy listens on port 6000). There were thousands of successful request/response pairs in this dump. Segments 1–55 on this TCP stream were fine too. You can see the RST our proxy sent in segment 60.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SBCy7y2BFuhueESBadEwLQ.png" /></figure><p>Before the reset, however, was this SYN in segment 56 with a sequence number of 0. It looked like NLB was trying to establish a connection on an already established connection.</p><p>I had read that a full SYN backlog could cause connection resets (not true, it turns out) so I explored increasing the SYN backlog and accept() queue. When that didn’t work I tried <a href="https://www.slideshare.net/brendangregg/velocity-2015-linux-perf-tools">Brendan Gregg’s Drunk Man Anti-method</a> with some other kernel parameters, followed by the “Google It” method which led me to some other kernel tuning around TCP idle timeouts that didn’t help.</p><p>I decided I needed a real method, starting with the RTFM method. I bought TCP/IP Illustrated Volume 1 but, while wonderfully informative, it didn’t end up helping.</p><p>Knowing that diving into the TCP implementation in the Linux Kernel (13,000+ lines of extremely terse C) was not going to yield results quickly, I did the Divide and Conquer method. I cut the data path in half and looked for issues on one side, then repeated along the path I could reproduce the issue. I’d find where to look more closely (I was even suspicious of Locust.io by then) in log2(hops in data path)</p><p>When I bypassed NLB the issue went away, so I opened a support ticket with AWS.</p><p>Meanwhile I poured through Kernel source and found two places where an ACK is sent in response to a SYN.</p><ol><li><a href="https://github.com/torvalds/linux/blob/v4.19/net/ipv4/tcp_input.c#L5428">https://github.com/torvalds/linux/blob/v4.19/net/ipv4/tcp_input.c#L5428</a></li><li><a href="https://github.com/torvalds/linux/blob/v4.19/net/ipv4/tcp_input.c#L5435-L5445">https://github.com/torvalds/linux/blob/v4.19/net/ipv4/tcp_input.c#L5435-L5445</a></li></ol><p>Both were this tcp_send_challenge_ack function called from tcp_validate_incoming. Note the comment about RFC 5961 in the second link.</p><p>To confirm this was the source I wrote a SystemTap script. Big thanks to <a href="https://medium.com/u/ade26e51b58">Danny Kulchinsky</a> for his post on <a href="https://medium.com/makingtuenti/systemtap-for-coreos-container-linux-b2c0a0b5dce3">SystemTap for CoreOS Container Linux</a> or getting SystemTap running would have taken a <strong>lot</strong> longer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f6b59743010306a7175aa5433b86b8d8/href">https://medium.com/media/f6b59743010306a7175aa5433b86b8d8/href</a></iframe><p>I ran my load tests, and on the first connection reset I came back to my terminal to find</p><pre>tcp_send_challenge_ack local=10.10.12.186:6000 remote=34.217.253.255:47442</pre><p>Only one of the two call sites has</p><pre>NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPSYNCHALLENGE);</pre><p>netstat -ts showed the SYN challenge counter increase by 1 for each connection reset so I had 100% confirmed tcp_input.c line 5443 as the source of the RST. The Kernel was challenging the SYN per the RFC.</p><p>I thought I’d found a bug in NLB, and updated my support ticket accordingly.</p><h3>The issue</h3><p>So what actually happened?</p><p>A picture’s worth a thousand words (and mine even has words in it).</p><p>Follow connection 1 down the left side first:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KO1beMG-xpgoJFZ8j5XF8A.png" /></figure><p>NAT Gateway happened to use port 47442 on 2 separate requests to what it saw as 2 unique destinations, as designed. Both “physical” NLBs preserved 34.217.253.255:47442 and RFC 5961 section 4.2 kicked in the kernel we run.</p><h4>Closed: by design</h4><p>In my support ticket with AWS they acknowledge the issue and said they don’t have a timeline for a fix; and why would they? Both products are working as designed.</p><p>I strongly suspect ELB and ALB are traditional reverse proxies where you would see their private IP and an ephemeral port for their outbound connections to an upstream service. If so, they wouldn’t suffer from this issue which is why we never saw it before.</p><p>The workaround? Just disable cross-zone load balancing. If you’re running stateless compute with more than a few instances per AZ, or a second proxy tier like us, you don’t need it anyway.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=109720accfc6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/nlb-connection-resets-109720accfc6">NLB Connection Resets</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>