<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tungdam on Medium]]></title>
        <description><![CDATA[Stories by Tungdam on Medium]]></description>
        <link>https://medium.com/@tungdam?source=rss-8de6bc946802------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Qr5vKcBjM2smWOmrkbWvnw.jpeg</url>
            <title>Stories by Tungdam on Medium</title>
            <link>https://medium.com/@tungdam?source=rss-8de6bc946802------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:55:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tungdam/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Small things that helped]]></title>
            <link>https://tungdam.medium.com/small-things-that-helped-8457fd31a0d8?source=rss-8de6bc946802------2</link>
            <guid isPermaLink="false">https://medium.com/p/8457fd31a0d8</guid>
            <category><![CDATA[network]]></category>
            <category><![CDATA[troubleshooting]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[sysadmin]]></category>
            <dc:creator><![CDATA[Tungdam]]></dc:creator>
            <pubDate>Thu, 01 Dec 2022 14:59:31 GMT</pubDate>
            <atom:updated>2022-12-01T15:13:10.723Z</atom:updated>
            <content:encoded><![CDATA[<p>Handy Linux network-related things that helped me. Hopefully they’re helpful with you as well. I’m sick at home, can’t sleep and it’s boring af. Let’s write something to cheer things up.</p><p><strong>Resolving file descriptor in </strong><strong>strace output with </strong><strong>-yy</strong></p><p>Sometimes we want to know which file / socket / device our process is interacting with. By default strace only show a number representing file descriptor like this:</p><pre>strace -e connect,socket,write curl -I https://google.com --silent &gt;/dev/null<br><br>socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3<br>socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 5<br>connect(5, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr(&quot;142.250.66.142&quot;)}, 16) = -1 EINPROGRESS (Operation now in progress)<br>write(5, &quot;\26\3\1\2\0\1\0\1\374\3\3\22\26\354t#\322\232\35R\226\377\26L\2=&gt;D\332v/\302&quot;..., 517) = 517<br>write(5, &quot;\24\3\3\0\1\1\27\3\3\0E1\353\243\204\370&lt;S(L\210\246\0011|J\250\1\321\325\271\237&quot;..., 80) = 80<br>write(5, &quot;\27\3\3\0)z\22dy\257F\222\325C0\324e\333\262|X\3006v%O*AP\5IB&quot;..., 46) = 46<br>write(5, &quot;\27\3\3\0,\31&lt;O\375{\202\3264V\215\300\rV\36\350Z\265\&quot;\235\6\313+\274\226\201Q\251&quot;..., 49) = 49<br>write(5, &quot;\27\3\3\0\36\257`fy\245\375\37\32\&quot;2\266\311O\315\2202\10.\300\230\246\222\26\255\221\37\272&quot;..., 35) = 35<br>write(5, &quot;\27\3\3\0:\256\370\330`\212\352Z\326\325O;\376\302\230\314\327j&#39;\2102\351\211\265\351~7%&quot;..., 63) = 63<br>write(5, &quot;\27\3\3\0\32 2\t\n\342V\375bp\7\260#_\330+\304\263\246\2536\360kn\250qH&quot;, 31) = 31<br>write(5, &quot;\27\3\3\0\&quot;\375k\23y\353\344\264f\351^\264\r\3079\366\214\31a\36[i\2456:#\361L&quot;..., 39) = 39<br>write(1, &quot;HTTP/2 301 \r\nlocation: https://w&quot;..., 668) = 668<br>write(5, &quot;\27\3\3\0\23j\232\&quot;N\23\276\252:\231S\312\327S\355^\17 \ng&quot;, 24) = 24<br>+++ exited with 0 +++</pre><p>With -yy , we can see clearly what we’re writing to, without having to go to proc/our_pid/fd ( usually it’s too late because of short-lived nature of a socket )</p><pre>strace -yy -e connect,socket,write curl -I https://google.com --silent &gt;/dev/null<br>socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3&lt;UDPv6:[19224551]&gt;<br>socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 5&lt;TCP:[19224556]&gt;<br>connect(5&lt;TCP:[19224556]&gt;, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr(&quot;142.250.66.142&quot;)}, 16) = -1 EINPROGRESS (Operation now in progress)<br>write(5&lt;TCP:[10.3.4.213:53490-&gt;142.250.66.142:443]&gt;, &quot;\26\3\1\2\0\1\0\1\374\3\3\266\220\36\210bA&#39;&lt;\320R\341A\303R7B\4G]\374\253&quot;..., 517) = 517<br>write(5&lt;TCP:[10.3.4.213:53490-&gt;142.250.66.142:443]&gt;, &quot;\24\3\3\0\1\1\27\3\3\0E\311\31\20=*\325\313\327\232a\224_\226\356@?@\207V;\10&quot;..., 80) = 80<br>write(5&lt;TCP:[10.3.4.213:53490-&gt;142.250.66.142:443]&gt;, &quot;\27\3\3\0)\336h\\j\27\334\274@K\274\231\366\217\331\354\3\350\0\324\351\263Z\227P\262z?&quot;..., 46) = 46<br>write(5&lt;TCP:[10.3.4.213:53490-&gt;142.250.66.142:443]&gt;, &quot;\27\3\3\0,\3651\311\273\217!S\206\376\341+wn\336!\367CHZ\213\201&amp;0^\250Kj&quot;..., 49) = 49<br>write(5&lt;TCP:[10.3.4.213:53490-&gt;142.250.66.142:443]&gt;, &quot;\27\3\3\0\36\244y\3073|r\214V\361\257n\235\210oN\346\255z\324\247\32\0230\330b\231\350&quot;..., 35) = 35<br>write(5&lt;TCP:[10.3.4.213:53490-&gt;142.250.66.142:443]&gt;, &quot;\27\3\3\0:\374\317\236\312\317\322\201G\373\271\\\315\364 ~\245\303\30u\303#S-\332k\21\t&quot;..., 63) = 63<br>write(5&lt;TCP:[10.3.4.213:53490-&gt;142.250.66.142:443]&gt;, &quot;\27\3\3\0\32\364&#39;\302\204\251\&quot;\351\353G\211\314|,/\5\226t\330\320O\30T \363\212\353&quot;, 31) = 31<br>write(5&lt;TCP:[10.3.4.213:53490-&gt;142.250.66.142:443]&gt;, &quot;\27\3\3\0\&quot;\205\r\315\v\7\340(\213V\371\232\23Bs\344E]&lt;\317`Z\307,\222C\205-&quot;..., 39) = 39<br>write(1&lt;/dev/null&lt;char 1:3&gt;&gt;, &quot;HTTP/2 301 \r\nlocation: https://w&quot;..., 668) = 668<br>write(5&lt;TCP:[10.3.4.213:53490-&gt;142.250.66.142:443]&gt;, &quot;\27\3\3\0\23\344^\362\n\361\273\355*\252\211\202\266\2579\337EE#\200&quot;, 24) = 24<br>+++ exited with 0 +++</pre><p>Note that the file descriptor 5 now is resolved to</p><pre>&lt;TCP:[10.3.4.213:53490-&gt;142.250.66.142:443]&gt;</pre><p>I find this very handy.</p><p>Just a side note, perf trace can resolve fd very well if it’s a normal file , but doesn’t provide this descriptive info about socket. And perf trace itself doesn’t resolve memory address ( in this example, the write buffer ) to a human-readable format for you, which is a big plus of strace IMHO.</p><p><strong>Traceroute using fixed port and protocol</strong></p><p>Recently I stumbled upon a case when traffic from the other countries couldn’t reach DNS servers in our Data Center. Our ISP denied the possibility that they have any kind of filter that block our UDP port 53 traffic ( of course :) ) .</p><p>I didn’t know till that time that we can use traceroute with fixed UDP port to probe the routes. Just as simple as this:</p><pre>traceroute -U -p 53 your_ip_to_trace</pre><p>then you’ll know at which point your traffic is filtered out. I provided such info for our ISP and they fixed it then.</p><p>TLS key negotiation failed <strong>OpenVPN issue</strong></p><p>I got this while switching our VPN setup from TCP to UDP. <a href="https://openvpn.net/faq/tls-error-tls-key-negotiation-failed-to-occur-within-60-seconds-check-your-network-connectivity/">Checked the FAQ related to this issue</a> but none of them is my case. After analyzing tcpdump record on both sides, I realized that the TLS handshake packet arrived to our service but was routed to another way ( our OpenVPN server has multi-homing / multiple interfaces setup — which is a common story for a VPN server ) , ended up causing such an error.</p><p>We did have a custom policy routing rules to cover this issue and it worked well with TCP server. It happens only when we switch our OpenVPN server from TCP-over-TCP to UDP.</p><p>The root cause is noted <a href="https://community.openvpn.net/openvpn/ticket/442#comment:2">here</a> long ago, let me quote it:</p><blockquote>Yes, this is a one of the common caveats with UDP — with TCP, you get a file descriptor from the kernel, and the kernel handles all the “where did the original packet go to? what do I need to use as a source address for replies?” (because TCP).</blockquote><blockquote>For UDP, the application needs to do this, and normal socket API does not even tell the application the destination IP address used in the client connection — so when OpenVPN answers, the kernel will just use one of the addresses in the system (typically the address on the outgoing interface).</blockquote><p>Oh !!! This is the best practical example of how connection-oriented TCP differs from connection-less UDP. With UDP, there’s no listen() and accept() after bind() , thus nothing to maintain connection states ( which contains the precise source address for the reply even when tcp server is binded to 0.0.0.0 ).</p><p>There’s actually <strong>still </strong>a file descriptor in case of UDP server but it just doesn’t contain info about 2 ends of the socket. For example, here’s the difference in recv and send syscalls from 2 simple TCP / UDP python servers</p><p>TCP</p><pre>recvfrom(4&lt;TCP:[10.3.4.213:12022-&gt;10.101.0.7:31578]&gt;, &quot;asdf\r\n&quot;, 1024, 0, NULL, NULL) = 6<br>sendto(4&lt;TCP:[10.3.4.213:12022-&gt;10.101.0.7:31578]&gt;, &quot;asdf\r\n&quot;, 6, 0, {sa_family=AF_INET, sin_port=htons(31578), sin_addr=inet_addr(&quot;10.101.0.7&quot;)}, 16) = 6</pre><p>UDP</p><pre>recvfrom(3&lt;UDP:[0.0.0.0:12023]&gt;, &quot;test&quot;, 1024, 0, {sa_family=AF_INET, sin_port=htons(36351), sin_addr=inet_addr(&quot;10.101.0.7&quot;)}, [16]) = 4<br>sendto(3&lt;UDP:[0.0.0.0:12023]&gt;, &quot;test&quot;, 4, 0, {sa_family=AF_INET, sin_port=htons(36351), sin_addr=inet_addr(&quot;10.101.0.7&quot;)}, 16) = 4</pre><p>Note how strace resolves file descriptor in each case.</p><p>Solution: Just bind your OpenVPN server to a specific IP by using local param, like:</p><pre>local 1.2.3.4</pre><p><strong>Quick and simple debugging with tcpdump</strong></p><p>I once needed to solve a case when our developer reported that 1 of our redis instance was returning a strange error but we don’t know which one exactly.</p><p>Developers log error message only. And we have dozen of them ( which are slave instances of several redis clusters — deployed on kubernetes ) .</p><p>Solution ?</p><p>On the server that host the application reporting redis error</p><pre>tcpdump -i any port 6379 -nvv | grep -i &quot;that client side error message&quot; </pre><p>Could identify the pod’s IP easily to drill deeper on the bug.</p><p><strong>How to set </strong><strong>ip_local_port_range properly ?</strong></p><pre>[   16.599434] ip_local_port_range: prefer different parity for start/end values.</pre><p>I used to set ip_local_port_range to 15000 65000 just to make it easier to remember and got the warning as above. At first ( a for quite a long time ) , I told myself “what kind of bad karma do kernel developers try to avoid with this non-sense thing”. Because developers , in general, have a good sense of humor.</p><p>Turned out they <a href="https://patchwork.ozlabs.org/project/netdev/patch/1432504175.4060.155.camel@edumazet-glaptop2.roam.corp.google.com/">did it for a reason</a>. In short, it “possibly” can speed up the port allocation process for both connect() and bind() if the total of number of ports in the range is a even number. ( ( end — start + 1) % 2 == 0 .</p><p>That patch was from 2015 and I haven’t looked very closely to verify it. Just because we still receive the warning, the impact should be there, still.</p><p>I ran into this thing while debugging the high %sys CPU issue in 1 of our ingress nginx when it was under a small DDOS attack ( several millions rpm ) . My case is not applicable here because the destination addresses of connect() don’t have high cardinality as the condition mentioned in the patch.</p><p>Thanks for reading this far. Hope this helps. And please do tell me what do you think so I can improve myself. ( Right here or I’m <a href="http://twitter.com/tungdam">tungdam</a>@ on Twitter , come and say hi )</p><p>Good night. Hope that I can sleep after this :(</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8457fd31a0d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A note on Cilium’s iptables usage]]></title>
            <link>https://tungdam.medium.com/a-note-on-ciliums-iptables-usage-39b4709fa584?source=rss-8de6bc946802------2</link>
            <guid isPermaLink="false">https://medium.com/p/39b4709fa584</guid>
            <category><![CDATA[cilium]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Tungdam]]></dc:creator>
            <pubDate>Tue, 12 Apr 2022 00:27:00 GMT</pubDate>
            <atom:updated>2024-01-24T23:35:13.739Z</atom:updated>
            <content:encoded><![CDATA[<p>About how cilium thinks about iptables and how are they using it.</p><h3><strong>Audiences</strong></h3><p>For people who want to understand why cilium still uses iptables and how.</p><h3>Cilium doesn’t like iptables</h3><p>I’ve been playing with <a href="https://cilium.io/">cilium</a> for a while and was impressed by how it is leveraging ebpf to build a high performance, scalable solution for container networking , plus observability and L7-aware policy capability. I also admire the team which has been a leader / pioneer in using ebpf for networking for years. The most recent <a href="https://isovalent.com/blog/post/2021-12-08-ebpf-servicemesh">service mesh beta program</a> looks very interesting. Check it out if you haven’t heard about cilium before !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*p9kFWxlOOorwPhNGlNnxKg.png" /><figcaption>Source: <a href="https://cdn.thenewstack.io/media/2018/07/b7de7e12-ciliumart.png">https://cdn.thenewstack.io/media/2018/07/b7de7e12-ciliumart.png</a></figcaption></figure><p>One picture’s worth a thousand words. This is how cilium achieves what they’re doing. Because current implementation of kubernetes networking solution with iptables-based kube-proxy <a href="https://github.com/kubernetes/community/blob/master/sig-scalability/blogs/k8s-services-scalability-issues.md"><strong>is not scalable</strong></a>.</p><p>Ebpf-based solution, on the other hand, <a href="https://cilium.io/blog/2021/05/11/cni-benchmark">is proved to be better </a>( high throughput / low latency / less resource consumption ) . Of course everything related to ebpf is fast, and the company has the best ebpf guys there AFAIK, no doubt about its performance.</p><p>When I see cilium offers <a href="https://docs.cilium.io/en/stable/gettingstarted/kubeproxy-free/"><strong>kube-proxy-free</strong></a><strong> </strong>solution which is based on their <a href="https://docs.cilium.io/en/stable/gettingstarted/host-services/#host-reachable-services">host-reachable-service</a>, I was like: oh wao, that’s cool but a bit sad to say good bye to our old friend iptables. But turned out it’s not the case: <strong>with my current kernel version ( 4.19 ) , cilium stills need iptables for their proxy redirection task.</strong></p><p>As the cilium documentation about its <a href="https://docs.cilium.io/en/stable/concepts/ebpf/iptables/">iptables usage</a> is quite pure, hence this article.</p><h3>Why does cilium use iptables still ?</h3><p>There are 2 main reasons IMHO:</p><ul><li>To ensure cilium datapath can live well with existing iptables settings. ( Doesn’t matter what cilium’s opinion about iptables is, it’s still there as part of Linux networking and packet still go through it, just with much less impact ).</li><li>To implement critical features in old kernel ( 4.x ) like L7 network policy ( TPROXY ) that can’t be implemented by ebpf-based solution as old kernel doesn’t support it. ( Cilium has several cool features that only works with modern kernel 5.x )</li></ul><p>One very important point here:<strong> as cilium successfully replaces kube-proxy by its ebpf-based solution, it solves the not-scalable issue of iptables-based kube-proxy</strong>. There is no more the situation with huge number of iptables rules on all the nodes anymore, which is a huge step.</p><h3>How does cilium uses iptables</h3><p>The rules looks like this:</p><pre><br>-A CILIUM_OUTPUT_raw -d 10.197.64.0/18 -m comment --comment &quot;cilium: NOTRACK for pod traffic&quot; -j CT --notrack<br>-A CILIUM_OUTPUT_raw -s 10.197.64.0/18 -m comment --comment &quot;cilium: NOTRACK for pod traffic&quot; -j CT --notrack</pre><pre># 10.197.64.0/18 is my ipv4-native-routing-cidr </pre><pre>-A CILIUM_OUTPUT_raw -o lxc+ -m mark --mark -m comment --comment &quot;cilium: NOTRACK for proxy return traffic&quot; -j CT --notrack<br>-A CILIUM_OUTPUT_raw -o cilium_host -m mark --mark 0xa00/0xfffffeff -m comment --comment &quot;cilium: NOTRACK for proxy return traffic&quot; -j CT --notrack</pre><pre>#0xa00/0xfffffeff: MagicMarkIsProxy/MagicMarkProxyNoIDMask</pre><pre>-A CILIUM_PRE_raw -d 10.197.64.0/18 -m comment --comment &quot;cilium: NOTRACK for pod traffic&quot; -j CT --notrack<br>-A CILIUM_PRE_raw -s 10.197.64.0/18 -m comment --comment &quot;cilium: NOTRACK for pod traffic&quot; -j CT --notrack<br>-A CILIUM_PRE_raw -m mark --mark 0x200/0xf00 -m comment --comment &quot;cilium: NOTRACK for proxy traffic&quot; -j CT --notrack</pre><pre>#0x200/0xf00: MagicMarkIsToProxy/MagicMarkHostMask </pre><pre><br>-A CILIUM_PRE_mangle -m socket --transparent -m comment --comment &quot;cilium: any-&gt;pod redirect proxied traffic to host proxy&quot; -j MARK --set-xmark 0x200/0xffffffff</pre><pre>#dns <br>-A CILIUM_PRE_mangle -p tcp -m mark --mark 0x779b0200 -m comment --comment &quot;cilium: TPROXY to host cilium-dns-egress proxy&quot; -j TPROXY --on-port 39799 --on-ip 0.0.0.0 --tproxy-mark 0x200/0xffffffff<br>-A CILIUM_PRE_mangle -p udp -m mark --mark 0x779b0200 -m comment --comment &quot;cilium: TPROXY to host cilium-dns-egress proxy&quot; -j TPROXY --on-port 39799 --on-ip 0.0.0.0 --tproxy-mark 0x200/0xffffffff</pre><pre>#http<br>-A CILIUM_PRE_mangle -p tcp -m mark --mark 0x8f3c0200 -m comment --comment &quot;cilium: TPROXY to host cilium-http-ingress proxy&quot; -j TPROXY --on-port 15503 --on-ip 0.0.0.0 --tproxy-mark 0x200/0xffffffff<br>-A CILIUM_PRE_mangle -p udp -m mark --mark 0x8f3c0200 -m comment --comment &quot;cilium: TPROXY to host cilium-http-ingress proxy&quot; -j TPROXY --on-port 15503 --on-ip 0.0.0.0 --tproxy-mark 0x200/0xffffffff</pre><pre>#mark is calculated like this <br>#port := uint32(byteorder.HostToNetwork16(proxyPort)) &lt;&lt; 16                              #ingressMarkMatch:=fmt.Sprintf(&quot;%#x&quot;,linux_defaults.MagicMarkIsToProxy|port)</pre><pre><br>-A CILIUM_FORWARD -o cilium_host -m comment --comment &quot;cilium: any-&gt;cluster on cilium_host forward accept&quot; -j ACCEPT<br>-A CILIUM_FORWARD -i cilium_host -m comment --comment &quot;cilium: cluster-&gt;any on cilium_host forward accept (nodeport)&quot; -j ACCEPT<br>-A CILIUM_FORWARD -i lxc+ -m comment --comment &quot;cilium: cluster-&gt;any on lxc+ forward accept&quot; -j ACCEPT<br>-A CILIUM_FORWARD -i cilium_net -m comment --comment &quot;cilium: cluster-&gt;any on cilium_net forward accept (nodeport)&quot; -j ACCEPT<br>#Just accept traffic to/from cilium&#39;s veth</pre><pre><br>-A CILIUM_INPUT -m mark --mark 0x200/0xf00 -m comment --comment &quot;cilium: ACCEPT for proxy traffic&quot; -j ACCEPT</pre><pre>-A CILIUM_OUTPUT -m mark --mark 0xa00/0xfffffeff -m comment --comment &quot;cilium: ACCEPT for proxy return traffic&quot; -j ACCEPT<br>-A CILIUM_OUTPUT -m mark ! --mark 0xe00/0xf00 -m mark ! --mark 0xd00/0xf00 -m mark ! --mark 0xa00/0xe00 -m comment --comment &quot;cilium: host-&gt;any mark as from host&quot; -j MARK --set-xmark 0xc00/0xf00</pre><p>I planned to have a walk through session here in this article but then i realized that it’s quite dry and boring. Plus some topic like TPROXY’s worth a separate write-up for those who doesn’t know it before to understand. So, here’s a summary for lazy folks. ( Ping me if you want an extra article about TPROXY, that’s quite a difficult topic for me to understand thoroughly )</p><ul><li>Rules are installed by cilium-agent when it starts up or when new L7 policy is created ( a couple of extra rules with <a href="https://github.com/cilium/cilium/blob/1108684e354d2b6ecf66cbaf1cd82de5fa88b70f/pkg/datapath/iptables/iptables.go#L560">fwmark based on random chosen port number</a> will be generated )</li><li>To my surprise, cilium doesn’t periodically synchronize those rules like kube-proxy. If you somehow remove a rule in its custom chain, you have to add it back manually or restart cilium-agent. Is this a bug or feature ?</li><li>As explained above, rules contain 2 main parts: <br>- To make sure traffic go through default iptables table / chain well without being dropped by default policy. ( For example: ACCEPT traffic to / from custom veth like lxc_* or cilium_* in FORWARD chain, filter table ) . These rules are identical on all nodes. <br>- To redirect traffic to its envoy proxy ( mark packet and using <a href="https://docs.kernel.org/networking/tproxy.html">TPROXY</a> target in mangle table, match <a href="https://github.com/cilium/cilium/blob/master/pkg/datapath/linux/linux_defaults/mark.go">magic mark</a> in raw table to avoid being tracked ). These rules are slightly different on each nodes</li><li>More about L7 policy related rules: it requires some external config to operate well: <br>- TPROXY requires policy routing which is used to forward marked traffic. <br>- <a href="https://github.com/cilium/cilium/blob/master/pkg/datapath/linux/linux_defaults/mark.go#L51">Mark</a> is set by <a href="https://docs.cilium.io/en/v1.9/concepts/ebpf/intro/">bpf hooks</a>. ( I’ll need to dig more into details for this step ) <br>- Of course an application proxy required, with <strong>IP_TRANSPARENT </strong>socket option<strong>.</strong></li><li>By default, cilium-agent manages conntrack by iptables ( install-no-conntrack-iptables-rules=false ) , there will be some extra rules in nat table and 1 ipset called <strong>cilium_node_set_v4</strong> set up on each node, to handle various NAT operation ( for example, allow your node to connect to the internet ). In my 4.19 kernel, using <strong><em>enable-bpf-masquerade</em></strong><em> </em>option for cilium-agent can have the same effect.</li><li>1 last but not least: number of rules doesn’t increase linearly with the number of endpoints or services ( like kube-proxy ), they are quite a fixed one. The only 1 thing can may be added along the way is 2 rules ( ingress / egress ) rule for each type of L7 policy ( dns , http, kafka so far )</li></ul><h3>Epiloque</h3><p>Before getting my hands dirty with cilium, I even thought that we don’t need iptables at all, but that’s not true, at least with the kernel version ( 4.19 ) I am using. Cilium offers many cool things ( to me it’s wiredguard transparent encryption, maglev LB ) but many of them is still in beta or requires much newer kernel, that’s a pity. There are still times we have to stick with old stuffs like this. And understand it a bit more is no harm, especially with our old legendary iptables.</p><p>Hope this helps.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39b4709fa584" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linux network ring buffers]]></title>
            <link>https://tungdam.medium.com/linux-network-ring-buffers-cea7ead0b8e8?source=rss-8de6bc946802------2</link>
            <guid isPermaLink="false">https://medium.com/p/cea7ead0b8e8</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[network]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[sysadmin]]></category>
            <dc:creator><![CDATA[Tungdam]]></dc:creator>
            <pubDate>Thu, 06 Jan 2022 11:20:38 GMT</pubDate>
            <atom:updated>2024-03-05T08:04:53.207Z</atom:updated>
            <content:encoded><![CDATA[<p>Trying to cover what I don’t know about Linux network</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*-_e_MrcEaCoOtCvhoK9fPw.jpeg" /><figcaption>Image source: <a href="https://img.freepik.com/free-vector/realistic-fire-ring_23-2148374973.jpg">https://img.freepik.com/free-vector/realistic-fire-ring_23-2148374973.jpg</a></figcaption></figure><p>Quite often I question myself about the role of Linux kernel memory with its network components. How many of them are allocated and when ? If it’s allocated on the fly, will the other memory allocation ( like pagecache , write buffer ) affects network performance ?</p><p>This article is my first try to solve all those questions myself. About network ring buffers.</p><h3>What is network ring buffers ?</h3><p>Network ring buffer is a memory region where NIC have direct access ( over <a href="https://en.wikipedia.org/wiki/Direct_memory_access">DMA</a> ) to store incoming data.</p><p>There are actually 2 parts: a ring ( fixed-size circular queue ) of descriptor / address which points to memory regions holding data. The “buffer” or memory part is quite a hidden place for us, which is handled by kernel in the background.</p><p>There are 2 types of ring buffer.</p><p>RX ring buffer for receiving data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*xPYgESL1hLV4llztDIZV9g.png" /><figcaption>Image source: <a href="https://i.stack.imgur.com/m7MU6.png">https://i.stack.imgur.com/m7MU6.png</a></figcaption></figure><p>TX ring buffer for sending data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*5VDRI6Do1wun4z8HegMT6Q.jpeg" /><figcaption>Image source: <a href="https://www.linuxjournal.com/files/linuxjournal.com/ufiles/imagecache/large-550px-centered/u1002061/11476f2.jpg">https://www.linuxjournal.com/files/linuxjournal.com/ufiles/imagecache/large-550px-centered/u1002061/11476f2.jpg</a></figcaption></figure><p>Both of those TX/RX ring buffers are bound to a specific network interface and shared between application that send/receive network through that interface. Thus knowing their limit is important.</p><h4>The ring / queue part</h4><p>Again, the ring buffer is basically the fixed-size circular FIFO queue ( which contains descriptor or simply pointer to a memory address ).</p><p>Then if there’s a queue with a fixed-size, there will be a exceeded queue length situation.</p><p>When you send / receive too many data than you can process, the queue is full and there will be packet dropped. Luckily we can check it by using ethtool, like this:</p><pre>ethtool -S enp8s0f0 | grep drop<br>     dropped_smbus: 0<br>     tx_dropped: 0<br>     rx_queue_0_drops: 0<br>     rx_queue_1_drops: 0<br>     rx_queue_2_drops: 0<br>     rx_queue_3_drops: 0<br>     rx_queue_4_drops: 0<br>     rx_queue_5_drops: 0<br>     rx_queue_6_drops: 0<br>     rx_queue_7_drops: 0</pre><p>When the number of dropped packet increased, we can try increasing our ring buffers size.</p><p>There are 2 ways to do that:</p><ul><li>Increase the size of each queue: the length of our queue is actually the number of descriptor pointing to allocated memory region via <a href="https://en.wikipedia.org/wiki/Direct_memory_access">DMA</a>. So the longer queue, the more places to store data. Using ethtool -g to show maximum possible and your current values and ethtool -G to really adjust it. You can find examples with instruction in details easily on the internet.</li><li>Use multi queues NIC: modern NIC supports multiple queues, means multiple separate DMA-mapping in parallels , thus bigger throughput. Same as above, we can use ethtool -l to view current setting and ethtool -L to adjust it.</li></ul><p>Though, as any queue in the world, increasing the queue size won’t solve your issue all the time. Long queue can give you high throughput but can cause a super high latency outliers. Short queue can be used to optimize latency, but it comes with the risk of dropping packets when there are too much too handle. As usual, use it as your own risk. <br>In sending path, Linux implements features like queuing discipline, Byte Queue Limit to solve the queue problem automatically for us. On the other hand, receiving path doesn’t have anything like that.</p><h4>The buffer / memory part</h4><p>The RX ring buffer to me is like an in-memory “kafka topic” holding all incoming packets that we need to consume as quickly as possible. And the consumer here is the ksoftirq-&gt;net_rx_action handler, which in turn calls a driver specific poll function, in our case it’s eventually is igb_poll .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*BOIYNF9kpLUzOD7_V_m5VA.png" /><figcaption>Image source: <a href="https://i.stack.imgur.com/BKBvW.png">https://i.stack.imgur.com/BKBvW.png</a></figcaption></figure><p>The very simplified asynchronous process looks like this:</p><ul><li>The incoming packet arrives to network interface card ( NIC ) . NIC saves it to RX buffer in memory ( our “kafka topic” ) via DMA ( no CPU should be involved )</li><li>When data is already saved in RAM, NIC sends an <strong>hardware interrupt</strong> tell CPU about the new arrival.</li><li>CPU replies very quickly to release that interrupt, and send another signal ( NET_RX_SOFTIRQ) to ksoftirqd</li><li>NIC can receive another packet and CPU also can handle other interrupts now while ksoftirqd starts consuming data from RX buffer in the background, without blocking the whole process. The main idea is that hardware interrupt should be handled as quickly as possible. Software interrupt will really do things.</li></ul><p>Here’s a real world kernel stack ( I have to trace upper layer function to get a full call stack of this layer 2 functions ):</p><pre>bpftrace -e &#39;kprobe:netif_receive_skb_internal {printf(&quot;%s\n&quot;,kstack);}&#39;</pre><pre>        netif_receive_skb_internal+1<br>        napi_gro_receive+186</pre><pre>        ///End of pulling data out of rx ring buffer<br>        igb_poll+1153<br>        net_rx_action+329<br>        __do_softirq+222<br>        irq_exit+186<br>        do_IRQ+127<br>        ret_from_intr+0<br>        cpuidle_enter_state+182<br>        do_idle+552<br>        cpu_startup_entry+111<br>        start_secondary+420<br>        secondary_startup_64+164</pre><p>( If you want to know more in details about sequence of steps that Linux does while receiving packets, read this <a href="https://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/">fantastic article</a> please. )</p><p>Now we’re dealing with <strong>data</strong>. And the size ( in bytes, not length anymore ) matters.</p><p>What i don’t know for sure right now is: what the maximum possible size of DMA memory allocated for all descriptor registered in the TX ring? And what if we cross that limit ?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*Kx-xPiff_7i_sATwdLv97Q.png" /></figure><p>At device initialization step, Linux calls driver specific function to create resource, in our case it’s <a href="https://github.com/torvalds/linux/blob/f40ddce8/drivers/net/ethernet/intel/igb/igb_main.c#L4335">igb_setup_rx_resource</a> which allocate <a href="https://github.com/torvalds/linux/blob/f40ddce8/drivers/net/ethernet/intel/igb/igb_main.c#L4343">both buffer ( via vmalloc</a> ) and ring’s descriptor in by <a href="https://github.com/torvalds/linux/blob/f40ddce8/drivers/net/ethernet/intel/igb/igb_main.c#L4351">dma_alloc_coherent</a>. The size of buffer is</p><pre>sizeof(struct igb_rx_buffer) * rx_ring-&gt;count;</pre><p>Size of <a href="https://github.com/torvalds/linux/blob/f40ddce8/drivers/net/ethernet/intel/igb/igb.h#L280">igb_rx_buffer </a>is a bit bigger than <a href="https://github.com/torvalds/linux/blob/f40ddce8/include/linux/mm_types.h#L69">struct page</a> which is around 64 bytes, so a DMA range for 1 NIC ,<em> </em><strong><em>if i understand all these things correctly</em>,</strong> is up to 64 * 4096 ( max rx queue size for a normal NIC ) * 8 ( number of queue length ) = 2MB.</p><p>Which is still good enough. Because each “buffer” , which is represent by sk_buff struct, is an <a href="https://en.wikipedia.org/wiki/Ethernet_frame">Ethernet data frame</a>. And usually NICs follow <a href="https://en.wikipedia.org/wiki/IEEE_802.3">IEEE 802.3 standard</a>, each frame size is up to 1522 bytes only. Some NIC supports jumbo frame, which the size is ~ 9000 bytes ( need to enable on both send/ receive ends ) , still well under our limit.</p><p>In case you use a custom NIC which sends a “illegal” data size, the packet can be dropped , or worse, the system will be crashed.</p><p>Thus, without any hacky violation of a custom NIC, we shouldn’t worry about our rx buffer overflow. Because :</p><ul><li>The allocated memory should be enough.</li><li>The number of incoming packet can be controlled by ring / queue size: if there’s too much to handle, there will be packet drops.</li></ul><p>And the last but not least: the ring buffer is drained , or consumed by net_rx_action , in our case igb_poll continuously.</p><p>In a very busy network server, you may see ksoftirq consumes quite a lot of CPU, and if you dig deeper by perf , for example, igb_poll can be the hot code that can consume quite an amount of CPU share. Because it’s just actively helping you to shovel all the data out of rx ring buffer !</p><p>There are 3 things that can stop net_rx_action/igb_poll from doing that:</p><ul><li>When there’s no more data to read from</li><li>It reaches its budget limit, which is used to control it from consuming too much CPU</li><li>It takes too long to finish</li></ul><p>Once again, follow this <a href="https://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/#monitoring-network-data-processing">fantastic article</a>, we can both detect and improve those cases. By checking info in /proc/net/softnet_stat like this to see if our consuming process was interrupted when there’s still data:</p><pre>cat /proc/net/softnet_stat  | awk &#39;{print $3}&#39;<br>.......<br>0000000c<br>0000002b<br>00000004<br>00000075<br>00000018<br>00000003<br>0000006e<br>00000071</pre><p>Field $3 is:</p><pre>The third value, sd-&gt;time_squeeze, is (as we saw) the number of times the net_rx_action loop terminated because the budget was consumed or the time limit was reached, but more work could have been. Increasing the budget as explained earlier can help reduce this.</pre><p>If those values &gt; 0, we’ve hit some limit. In that case, we can let net_rx_action/igb_poll use more CPU by increasing the budget like this:</p><pre>echo your_value &gt; /proc/sys/net/core/netdev_budget</pre><p>By default it’s 300. If you have plenty of free CPU, with irqbalance installed, feel free to triple it or more. Observe both %si in CPU utilization and the data in softnet_stat to see the effect.</p><p>One more thing that we can do is to speed up the memory reading speed, by using a faster RAM , or checking if you have any NUMA related issue.</p><p>We have a simple but crucial point here, where faster and bigger CPU and RAM do increase your network performance ( In term of both latency and throughput ).</p><p>Is it too obvious to tell that system resources connect to each other closely ?</p><h3>Epilogue</h3><p>The Linux ring buffers internal, the initialization process for NIC, the send / receive sequence of steps a way more complicated than what is described in this posts.</p><p>I just want to explain things , important ones in the most understandable form so everybody can apply it easily in our daily tasks. Hope it helps.</p><p>More posts about Linux networking are coming. Stay tuned</p><h3>References</h3><ul><li>A must read: <a href="https://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data">https://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data</a></li><li><a href="https://www.linuxjournal.com/content/queueing-linux-network-stack">https://www.linuxjournal.com/content/queueing-linux-network-stack</a></li><li><a href="https://stackoverflow.com/questions/47450231/what-is-the-relationship-of-dma-ring-buffer-and-tx-rx-ring-for-a-network-card">https://stackoverflow.com/questions/47450231/what-is-the-relationship-of-dma-ring-buffer-and-tx-rx-ring-for-a-network-card</a></li><li><a href="https://wiki.linuxfoundation.org/networking/kernel_flow">https://wiki.linuxfoundation.org/networking/kernel_flow</a></li><li><a href="https://people.cs.clemson.edu/~westall/853/notes/skbuff.pdf">https://people.cs.clemson.edu/~westall/853/notes/skbuff.pdf</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cea7ead0b8e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical tracing: a final recap]]></title>
            <link>https://tungdam.medium.com/practical-tracing-a-final-recap-5d7e16e4a9df?source=rss-8de6bc946802------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d7e16e4a9df</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Tungdam]]></dc:creator>
            <pubDate>Fri, 31 Dec 2021 04:06:04 GMT</pubDate>
            <atom:updated>2021-12-31T06:40:10.994Z</atom:updated>
            <content:encoded><![CDATA[<p>What I’ve learned so far about tracing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2JzUhUtQ9F9jykaE1rJ9qw.jpeg" /><figcaption>Source: <a href="https://capsule8.com/wp-content/uploads/2021/03/C8_2021_Blog_3-4_HighRes-1024x455.jpg">https://capsule8.com/wp-content/uploads/2021/03/C8_2021_Blog_3-4_HighRes-1024x455.jpg</a></figcaption></figure><p>This is a wrap up of my practical linux tracing series. You can see older posts start from <a href="https://medium.com/coccoc-engineering-blog/things-you-should-know-to-begin-playing-with-linux-tracing-tools-part-i-x-225aae1aaf13">here</a>.</p><p>In this final write-up, I’ll change my style a bit, to make it more straight to the point.</p><p>Here are my top 10 key takeaways after some time working as a Linux tracer.</p><p>1. Tracing tools, techniques and data ( events, traces , spans… ) are means, not the end result. What matters is how do you interpret them to understand your situation and root cause.</p><p>2. Tracing is basically is to know what your service/application is doing. Eventually the truth lies in source code. Be prepared to read somebody’s code always.</p><p>3. Nowadays the data we can collect is huge, find a way to visualize and analyze it. Trace with a plan in your mind. The raw data only is way beyond your cognitive ability.</p><p>4. Tracing or performance tuning is not an overnight task. Just like hacking. The success you see is the result of weeks or months research. There’s a quick win sometimes but the big win usually requires you to understand your system thoroughly. No pain no gain.</p><p>5. On the other hand, there’s nothing wrong with “script kiddies” if it helps. Many top engineers in the field goal is to try making tracing or debugging much easier everyday. ( By sharing knowledge and making tools )</p><p>6. If you want to leverage tracing as a quick method to solve your production issue, invest into tracing infrastructure ( eg: tools, libraries, storage, even third-party solution… ) and making your application tracing-friendly ( eg: integrated instrumentation libraries right from the start to expose useful data ). Thus, Implementing distributed tracing for the whole organization requires a culture shift. But it’s worth doing that.</p><p>7. Tracing is cool, but solving production problem on time is much cooler. Root cause investigation can wait, sometimes work-around to mitigate damage should have higher priority. This is where simple things like change management or a quick conversation with your fellow developers are really helpful.</p><p>8. Always think about overhead while tracing. If you can’t measure it very precisely, start small ( eg: profiling with low frequency, recording a few events , in short time ) , observe the impact then increase things gradually.</p><p>9. Tracing is mostly about finding error. So be humble and professional while reporting it to the author of “not-very-optimal code”. The tracer should have a clear evidence or proof-of-concept while presenting the error. If you’re not that sure ( quite often ), sit down with your colleagues and asking for their help to verify your assumption, nicely. Sometimes a tracer needs to have very good convincing skill.</p><p>10. Tracing is fun, and addictive.</p><p>Happy new year everybody !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d7e16e4a9df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical Linux tracing part 4: tracing container workload]]></title>
            <link>https://medium.com/coccoc-engineering-blog/practical-linux-tracing-part-4-tracing-container-workload-63ac04306b27?source=rss-8de6bc946802------2</link>
            <guid isPermaLink="false">https://medium.com/p/63ac04306b27</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Tungdam]]></dc:creator>
            <pubDate>Mon, 04 Oct 2021 03:41:02 GMT</pubDate>
            <atom:updated>2025-02-21T14:27:41.304Z</atom:updated>
            <content:encoded><![CDATA[<p>The similarity and difference in comparison with normal workload</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*5Rq9x_H4R8uv08RtLP_Tyw.jpeg" /><figcaption>src: <a href="https://www.kizytracking.com/wp-content/uploads/2020/10/xsmart_container_connection.jpg">https://www.kizytracking.com/wp-content/uploads/2020/10/xsmart_container_connection.jpg</a></figcaption></figure><h3>TL;DR</h3><p>Same same:</p><ul><li>Tracing kernel events ( syscall, kprobe, tracepoints ) works in the same way</li><li>Profiling user-space program by perf works in the same way ( Linux 4.14+ )</li></ul><p>But different:</p><ul><li>Namespaces prevent tracing user-space events by kernel tracing infrastructure. ( As tools and binaries / libraries are isolated by different mount namespaces )</li><li>Distributed tracing ( Eg: Opentelemetry / Opentracing + Jaeger ) to rescue</li></ul><h3>Prerequisite</h3><p>This is the 4th article in my ( messy ) practical tracing tutorial series. You should be very familiar with tracing / profiling and tools like strace , perf, <a href="https://github.com/iovisor/bpftrace">bpftrace and its various tools</a> . Otherwise, I highly recommend to start from the beginning:</p><ul><li><a href="https://medium.com/coccoc-engineering-blog/practical-linux-tracing-part-0-x-why-did-i-trace-5597a35a3bb5?source=friends_link&amp;sk=0dfc93badcefec8bdd192f5d21781f26">Part0: Why did i trace ?</a></li><li><a href="https://medium.com/coccoc-engineering-blog/things-you-should-know-to-begin-playing-with-linux-tracing-tools-part-i-x-225aae1aaf13">Part1: Symbols, debug symbols and stack unwinding</a></li><li><a href="https://medium.com/coccoc-engineering-blog/practical-linux-tracing-part-0-x-why-did-i-trace-5597a35a3bb5?source=friends_link&amp;sk=0dfc93badcefec8bdd192f5d21781f26">Part2: Profiling vs Event Tracing</a></li><li><a href="https://medium.com/coccoc-engineering-blog/practical-linux-tracing-part-3-x-strategies-for-adhoc-analysis-ae060e597602">Part3: Strategies for adhoc tracing</a></li></ul><p>And obviously, we also need to have experience with container workload as well. Not much though, knowing buzz words like docker, kubernetes, Linux cgroup / namespace… is enough.</p><p>By saying “container workload” , I mean just an application running inside a container , use <a href="https://containerd.io/">containerd runtime </a>( directly or through docker engine ), doesn’t matter what the orchestration solution is ( k8s like us or openshift, docker swarm or mesos … )</p><h3>The similarity</h3><p>Process on host OS and inside container both simply are processes. And they share the kernel part. Thus, tracing that process with kernel related event like syscalls, kprobe, tracepoints are all the same.</p><p>For example, <a href="https://github.com/iovisor/bpftrace/blob/master/tools/syncsnoop.bt">syncsnoop.bt</a> is still working well with container workload:</p><pre>syncsnoop.bt<br>Attaching 7 probes...<br>Tracing sync syscalls... Hit Ctrl-C to end.<br>TIME      PID    COMM             EVENT<br>10:27:04  1229   auditd          tracepoint:syscalls:sys_enter_fsync<br>10:27:04  1229   auditd          tracepoint:syscalls:sys_enter_fsync<br>10:27:05  11459  fluent-bit       tracepoint:syscalls:sys_enter_fsync<br>10:27:05  11459  fluent-bit       tracepoint:syscalls:sys_enter_fsync<br>10:27:05  11459  fluent-bit       tracepoint:syscalls:sys_enter_fsync<br>10:27:05  11459  fluent-bit       tracepoint:syscalls:sys_enter_fsync<br>10:27:05  11459  fluent-bit       tracepoint:syscalls:sys_enter_fsync<br>.....</pre><p>In this snippet, auditd is running on host OS while fluent-bit is a container workload, which can be verified by checking its NSpid like this</p><pre>cat /proc/1229/status | grep nspid -i<br>NSpid: 1229</pre><pre>cat /proc/11459/status | grep nspid -i<br>NSpid: 11459 1</pre><p>Here auditd has only 1 PID 1229 but fluent-bit has two of them ( 11459 in the “main” pid name space and 1 in its own namespace.</p><p>As you can see tracing tracepoints works in the same way on both processes.</p><p>Same thing with syscall and strace, as a simpler example:</p><pre>strace -e fsync -p 11459<br>strace: Process 11459 attached<br>strace: [ Process PID=11459 runs in x32 mode. ]<br>strace: [ Process PID=11459 runs in 64 bit mode. ]<br>fsync(22)                               = 0<br>fsync(22)                               = 0<br>fsync(22)                               = 0<br>fsync(22)                               = 0<br>fsync(22)                               = 0<br>.....</pre><h3>The difference</h3><p>Container workload has its own namespaces. ( See <a href="https://www.nginx.com/blog/what-are-namespaces-cgroups-how-do-they-work/#What-Are-Namespaces">this</a> if you’re not familiar with it . Feel free to suggest me a better article as a reference). In the simplest way, we can say that it has its own root filesystem, with its own software version. The binaries and libraries are isolated with our trace tools.</p><p>You can quickly realize here that we’ll have trouble with interacting to user-space application. For example, bpftrace on the host OS can’t reach the binary or .so file inside a container to inject an uprobe, thus we’re losing our super power to look into any running application on demand.</p><p>What can we do in this case ?</p><h4>Stick to the old way: perf is “namespace aware”</h4><p>Since <a href="https://github.com/torvalds/linux/commit/843ff37bb59edbe51d64e77ba1b3245a15a4dd9f">linux v14.4-rc1</a>. It means, perf since version 4.14.4 can detect if the binary that it’s profiling belongs to another namespace. If it does, perf can examine the binary’s symbols from its namespace automatically for you and give you the stack trace as usual.</p><p>So, fortunately, profiling with perf still works in the container ❤ .</p><p>Back to the example with fluent-bit above, we can profile it as if it’s a normal process on a host OS:</p><pre>perf record -a -g -F 99 --call-graph dwarf -p 11459 -- sleep 3<br>Warning:<br>PID/TID switch overriding SYSTEM<br>[ perf record: Woken up 43 times to write data ]<br>[ perf record: Captured and wrote 17.774 MB perf.data (2180 samples) ]</pre><p>perf script shows us strack trace like this</p><pre>            ..................................<br> <br>            55d35d9acaf5 sqlite3_step+0x169 (/fluent-bit/bin/fluent-bit)<br>            55d35d7fc0a8 flb_tail_db_file_offset+0x68 (/fluent-bit/bin/fluent-bit)<br>            55d35d7f7376 flb_tail_file_chunk+0x614 (/fluent-bit/bin/fluent-bit)<br>            55d35d7f1f2d in_tail_collect_static+0x6e (/fluent-bit/bin/fluent-bit)<br>            55d35d7bc1a3 flb_input_collector_fd+0x409 (/fluent-bit/bin/fluent-bit)<br>            55d35d7c838d flb_engine_handle_event+0x52f (inlined)<br>            55d35d7c838d flb_engine_start+0x52f (/fluent-bit/bin/fluent-bit)<br>            55d35d73d2b2 main+0x6c9 (/fluent-bit/bin/fluent-bit)                                        ▒</pre><p>So no matter in which namespace a process is running, profiling with perf still can provide us useful info about strace trace. There’s still the case you will have a broken stack like this:</p><pre>7f6cdad0b803 [unknown] (/fluent-bit/bin/out_grafana_loki.so)<br>7f6cdaca5804 [unknown] (/fluent-bit/bin/out_grafana_loki.so)<br>7f6cdacd84b2 [unknown] (/fluent-bit/bin/out_grafana_loki.so)<br>7f6cdacda30b [unknown] (/fluent-bit/bin/out_grafana_loki.so)<br>7f6cdacda91d [unknown] (/fluent-bit/bin/out_grafana_loki.so)<br>7f6cdacdab95 [unknown] (/fluent-bit/bin/out_grafana_loki.so)<br>7f6cdaceae3b [unknown] (/fluent-bit/bin/out_grafana_loki.so)<br>7f6cdad07ce3 [unknown] (/fluent-bit/bin/out_grafana_loki.so)</pre><p>but it’s actually the same old problem with missing debug symbols in binary, don’t blame container for it. You may want to revisit <a href="https://medium.com/coccoc-engineering-blog/things-you-should-know-to-begin-playing-with-linux-tracing-tools-part-i-x-225aae1aaf13">my first blog</a> to fix this issue.</p><p>If profiling with perf isn’t enough for you, there are another options you may want to consider.</p><h4>Running tracing tools from inside the container ?</h4><p>Some may suggest installing our trace tools inside a container and run it from the inside but i am against it, because:</p><ul><li>It can increase our container image’s size significantly.</li><li>We don’t always own the image. Cooking them from third-party image with perf / bpftrace tools is usually troublesome. Sidecar container with tracing tools can help to avoid changing app image, but it poses another serious drawback: 2 containers have to share the rootfs ( mount namespaces ) which is not easily done to my knowledge.</li><li>There are various security settings to adjust in order to give tracing tools inside container to gain the same privileges / power as ones in the host OS. Not always possible though, depends on how you setup your cluster and its security policy.</li><li>Even if we have tools inside container, we still need merge all the user-space symbols inside container with kernel ones from host OS, by mounting data from either side.</li></ul><p>It’s possible though, but it’s just not what i recommend.</p><h4>Distributed tracing AKA end-to-end tracing</h4><p>Distributed problem requires a distributed solution. perf and bpftrace is super cool for kernel tracing on a specific node, but it’s not designed to solve distributed problem, i believe.</p><p>If there’s a tracing solution that can solve our problem with instrumenting container workloads, it should be able to collect:</p><ul><li>Application events</li><li>Across not only namespaces boundary but also hosts boundary.</li></ul><p>The distributed tracing infrastructure should support:</p><ul><li>Tracer library instrumentation in different languages to trace different applications.</li><li>Collecting events over network, from different pods, on different hosts.</li><li>Process, aggregate, store, visualize all those data in a central place</li></ul><p>Fortunately, there are many of such solutions for it in the market already. If you’re new to this area, read more about it in following articles to get more info about distributed tracing:</p><ul><li><a href="https://lightstep.com/distributed-tracing/">Introduction to distributed tracing</a> by Lightstep.</li><li><a href="https://ploffay.medium.com/five-years-evolution-of-open-source-distributed-tracing-ec1c5a5dd1ac">Five years evolution of opensource distributed</a> tracing by <a href="https://medium.com/u/a20a4172f9c">Pavol Loffay</a></li></ul><p>Distributed tracing is definitely worth its own ( lengthy ) article but it’s out of scope here.</p><p>In our company we chose <a href="https://medium.com/u/3b09a099069d">JaegerTracing</a> as our backend solution ( agent / collector, storage, visualization ) ,in combination with <a href="https://opentelemetry.io/">opentelemetry</a> as instrumentation libraries to trace container workload.</p><p>Each of our application function latency is recorded and put in a context of same request like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Bx7CK7pAA5NQ8ZhBsdwlQ.png" /></figure><p>As usual, there are pros and cons in comparison with our “lower-level” tracing tools.</p><p><strong>Pros:</strong></p><ul><li>Solve our container working tracing problem ! Now tracing works across namespaces / services / hosts boundaries.</li><li>Provide useful context of an event: between events inside a service, in side a request which runs across different service ( thus a name end-to-end tracing )</li><li>Solve problem with interpreted / scripting language ( Eg: PHP , Python..): We now can look into specific events instead of execute_ex for PHP or _PyEval_EvalFrameDefault for Python.</li></ul><p><strong>Cons:</strong></p><ul><li>An extra effort to setup a distributed tracing infrastructure , obviously.</li><li>Performance overhead: eventually it’s a user-space instrumentation, can’t compare with the speed of BPF-based tool. And it depends on language specific implementation of opentelemetry SDK / API. We have to make trade-off between performance vs feature richness here. Though distributed tracing solutions support different sampling mechanism to handle this in a flexible way.</li><li>Event ( span ) in distributed tracing tool ) is kind of user statically-defined tracing ( USDT ) in kernel tracing world. There’s no dynamic uprobe super power.</li></ul><h3>Conclusion</h3><p>In summary, this is what i would recommend you while tracing container workload:</p><ul><li>Understand the “namespaced process nature” of a container. Use perf / bpftrace tool still to debug the kernel-related issue that may affect your container workload. ( CPU contention / starving, cgroup throttling, unnecessary / abnormal syscalls, extra unwanted IO … )</li><li>Implement distributed tracing solution to trace application-specific events. Use jaeger and opentelemetry as we do if possible, so we can discuss it and grow together ;)</li></ul><p>Hope it helps</p><h4><strong>Preferences:</strong></h4><ul><li><a href="https://www.brendangregg.com/Slides/LISA2017_Container_Performance_Analysis.pdf">https://www.brendangregg.com/Slides/LISA2017_Container_Performance_Analysis.pdf</a></li><li><a href="https://ploffay.medium.com/five-years-evolution-of-open-source-distributed-tracing-ec1c5a5dd1ac">https://ploffay.medium.com/five-years-evolution-of-open-source-distributed-tracing-ec1c5a5dd1ac</a></li><li><a href="https://lightstep.com/distributed-tracing/">https://lightstep.com/distributed-tracing/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63ac04306b27" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coccoc-engineering-blog/practical-linux-tracing-part-4-tracing-container-workload-63ac04306b27">Practical Linux tracing part 4: tracing container workload</a> was originally published in <a href="https://medium.com/coccoc-engineering-blog">Coccoc Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Our OKR journey so far]]></title>
            <link>https://tungdam.medium.com/our-okr-journey-so-far-af95ca7c747e?source=rss-8de6bc946802------2</link>
            <guid isPermaLink="false">https://medium.com/p/af95ca7c747e</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[okr]]></category>
            <dc:creator><![CDATA[Tungdam]]></dc:creator>
            <pubDate>Thu, 01 Jul 2021 15:25:11 GMT</pubDate>
            <atom:updated>2021-07-02T00:48:01.616Z</atom:updated>
            <content:encoded><![CDATA[<p>A sysadmin’s reflection on several years of practicing OKR</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*sYdG8jwsZxCKmttXbwZDUA.jpeg" /><figcaption>Source: <a href="https://guitartpartners.com/wp-content/uploads/2020/11/OKR.jpg">https://guitartpartners.com/wp-content/uploads/2020/11/OKR.jpg</a></figcaption></figure><h3>Preface</h3><p>This is the very first ever article about management ( ? ) . I even don’t know if a goal-setting framework like <a href="https://en.wikipedia.org/wiki/OKR"><strong>O</strong>bjective <strong>K</strong>ey <strong>R</strong>esult</a> can be classified as a managerial stuff, but as we ( our company / team ) have been following it since 2017, play an important role in our job till now, with ups and downs, pros and cons, it’s worth spreading some thoughts.</p><p>Disclaimer: I’m a Linux sysadmin , tracer and a team leader. I’m not a CEO of any startup :)</p><h3>What is OKR ?</h3><p>Again, according to <a href="https://en.wikipedia.org/wiki/OKR">wikipedia:</a></p><blockquote><strong>Objectives and key results</strong> (<strong>OKR</strong>) is a goal-setting framework for defining and tracking objectives and their outcomes.</blockquote><p>In the simplest form: it’s a way for planning, implementing, tracking, assessing your tasks by:</p><ul><li>Focus ( to objectives )</li><li>Alignment ( between teams based on agreement on objectives above )</li><li>Tracking ( by key results. Measurable ones )</li><li>Stretching ( for excellence. Aim high to be more. Though it’s OK to fail )</li></ul><p>It was first introduced in Intel by <a href="https://en.m.wikipedia.org/wiki/Andrew_Grove">Andy Grove</a>, shining ( still ? ) at Google, get close to me by John Doerr’s famous book on the subject: <a href="https://www.amazon.com/Measure-What-Matters-Google-Foundation/dp/0525536221">measure what matters</a>. If you haven’t heard about this or done it before, this blog may not for you, sorry.</p><p>Our company has been adapting this framework for several years. Though I’m not speaking for my company. This is my own opinions.</p><p>I won’t talk much about OKR itself in this blog, this is my pure reflection on how it has been so far.</p><h3>First impression</h3><p>I read the book twice before really doing it. ( As our ex-CEO hold a class about it and I joined him ).</p><p>The very first feeling is like: “yeah, that’s cool, innovative. It’s even <strong>OK to fail</strong>. Fantastic!”. Because i read the book, about the success stories from Google ! ( about Larry Page and something “uncomfortably exicted” ) , about inspirational startups, about pizza-making robots.</p><p>That’s nice.</p><p>Don’t worry, there’s no twist, even though <a href="https://www.theverge.com/2020/2/18/21141969/zume-pizza-failure-softbank-founder-cult">Zume failed with that robot project</a> ( for some reasons i don’t know . The link is just for reference, I haven’t read it ). There are the good, the bad, the tricky.</p><h3>The good</h3><h4>OK to fail</h4><p>Freedom. Room for errors. No pressure. That’s what we love about this new system. Don’t get me wrong, we’re still tracking things closely ( by measurable key results: still timeline ( no <strong>dead</strong>line because it’s adjustable ) , metrics, meetings to catch up / align… but we’re have more freedom than pressure. I don’t know which one is better, but I would say fortunately we have good teams with very high responsible, proactive folks. Discipline is rarely our problem.</p><p>Because.</p><h4><strong>We’re doing big things</strong></h4><p>OKR encourages us to stretch ourselves. To aim for something very ambitious. The ambition that produces a goal too big that reaching 80% of them can considered a huge successful. ( That’s why it’s OK to fail ).</p><p>We dreamed big at first, followed that spirit eagerly. At that time we’ve been struggling with maintaining old stuffs for couple years and this is a wake-up call.</p><p>We kicked-off containerization stuffs ( with docker and Mesos that time ). We upgraded the servers that there’s nobody dare to touch in years, decided to switch to ansible for a long time being fan of debian package. CI / CD is applying in a broader range, involving multiple teams. KVM was brought back to life after virtualization was buried for years since the failure of openvz in the very early day.</p><p>Big operational goal like “there’s no single point of failure” or chaos monkey practice were raised and every body is excited about it.</p><p>I poured myself into tracing area ( which is pretty new that time, in the very early day of perf and flamegraph ) and quickly felt in love. We organize our very first conference for Sysadmin.</p><p>That’s the wind of change that Scorpions should understand very well.</p><h3>The bad</h3><p>Implementation hell . Easy to guess. It’s easier said and done. Some reasons:</p><h4><strong>Alignment</strong></h4><p>We need alignment to really do OKR properly.</p><p><strong>From company’s level to infrastructure team</strong></p><p>I can’t see how we should match well with our company level objectives even till now. Company ( especially young ones like our ) should have vision on innovative products, big revenue or dominating market share, which in no way IMHO can directly involve with a team that prefer securing things, stablizing stuffs or optimizing resources.</p><p>I was feeling kind of left out at first in OKR planning sessions in the very beginning, but no matter this framework or anything else, infrastructure team’s place is in the background. It’s not bad, though. Got used to it anyway.</p><p>You won’t be jealous with a sparkling star when your job is running the whole universe, my dear sysadmin fellows :)</p><p><strong>Between teams</strong></p><p>Teams need to align with each other, in order to inform / adjust when there’s an unwanted mid-cycle changes, to be re-focused if needed.</p><p>And we just couldn’t do it ourselves in the early day as we were all knew to this game. We needed an OKR master to regulate all this. Human is never perfect and can easily be a single point of failure. Fortunately our C-level managers managed it in time to keep it up and things are better now.</p><p><strong>Between team members.</strong></p><p>We were excited at planning phase and did it quite well. But it worn out quickly over time. Due to incidents, mystical bugs ( that even made us more excited! ) or just obstacles while pursuing our ( too ) big goal.</p><p>We didn’t have internal meetings often that time and just managed tasks by task management software ( not built for OKR like weekdone, though ). Thus everything was like a plan on a paper. It’s not a living things if we don’t communicate, adjust, update with each other.</p><p>We fix ourselves quickly by doing weekly 1:1. People ( including me ) like the big ideas, agree on them while planning, but need to be refocused from being distracted by routine tasks or incidents. Quite often I lost the fire myself ( and don’t know how to pick it up , which is normal, i guess ). Discussing often helped us to some extent, even though sometimes I’m just a sounding board for my teammate. And we were optimistic, again, about our big goals.</p><p>Unfortunately, this thing is a never ending story, like a continuous improvement process. There were ups and downs, inevitably. I’m not quite confident to say that our 1:1 meetings are very effective and help us to refocus ourselves always even till now.</p><p>Thus, alignment is hard.</p><h4><strong>Big goals and OK to fail trap</strong></h4><p>The goal sometimes is too big. It’s quite difficult to judge, we ( sysadmins ) can argue that we have to much unpredictable incidents, but who doesn’t have it ? So no excuse. But the problem is, because of OK to fail, we failed our goal more often. And I was afraid that 1 day we would not feel anything about it. It’s the loser’s habit.</p><p>We have to fix it. Then we decided not to stretch more ( or we already did all what we could ) and adjust the goal, to <strong>smaller ones </strong>:)</p><p>This is a very tricky moment. Too big we’ll fail and we’ll get used to it. To small, is it OKR any more ?</p><p>Till now I can’t really know if I’m managing the balance good enough, still a big puzzle.</p><h3>The tricky</h3><h4>The tricky balance</h4><p>Of setting objectives. To big it’s unrealistic. Too small, you’re coward and not playing OKR game hard enough.</p><p>Of setting measurable key results. Too vague, it’s hard to track. Too precise, it’s just dry mechanical numbers and can be game on.</p><p>Of choosing between top-down or bottom up approach. Of letting team refocus themselves based on what they agreed on or we definitely need an OKR master’s intervention.</p><p>Do it right, you’re an expert. Otherwise, OKR can be just an mediocre planning framework.</p><p>We’re not living in our binary world of engineering anymore. This is managerial stuffs.</p><h4>Routine tasks</h4><p>There’s some ugly maintenance tasks ( incident responses, things that we have not yet been able to automate yet … ) that you shouldn’t put it into your OKR. But sometimes it can consume most of your time and spare no energy for OKR effort.</p><p>The solution for this can be automation ( to reducing operational toil ), improving alerting system, setting SLO for services ( to clearly decide when we should act ). But till we can finish it, save 1 objective in your OKR to deal with it explicitly.</p><p>OKR shouldn’t contain only beautifully audacious things.</p><h4>Mid-cycle changes</h4><p>Mostly because of agreement without enough commitment. Can be bad planning as well.</p><p>Many times we got confirmation from other teams to help us solve the dependency issue from the side which can block our objectives. But it’ll never happen because of “lacking resources”, “busy with other tasks” , thus our objective must be blocked mid-cycle. It’s inevitable, accept it.</p><p>Or sometimes it’s new idea from the top, new opportunity arises from nowhere and we have to catch it immediately. Pause everything and switch to this new project please. It’s also inevitable.</p><p>What can we do in that case ? Adapt, improve, improvise ?</p><p>I would argue first :) And set error budget for such behavior, we can’t do it all the time, otherwise this OKR framework is useless, stop it immediately.</p><p>Yeah, it’s tricky. Depend on who are you talking to.</p><p><strong>OKR vs rewarding system</strong></p><p>I basically agree with separating OKR with rewarding system. Because otherwise it’s some form of KPI. Or people may tend to set a more easy-to-achieve goal to game on such rewarding system. But if we have OKR and only OKR as a goal settings and tracking framework, then where else should we look for info to decide about reward for people ? Some may say that let the managers decide that. True, but people’s decision, emotion, fairness , objectivity — all those things are very tricky</p><h3>Conclusion</h3><p>I still love OKR so far. Especially the feeling of being like on a stage, announcing new products / achievements for the whole team in each OKR reviewing sessions. That’s very special. But I’m too far from mastering it, as it’s complicated, tricky as I explained above.</p><p>Is it uncomfortably excited still ? I don’t know.</p><p>What’s your OKR story so far ?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af95ca7c747e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical Linux tracing ( part 3 ): Strategies for adhoc analysis]]></title>
            <link>https://medium.com/coccoc-engineering-blog/practical-linux-tracing-part-3-x-strategies-for-adhoc-analysis-ae060e597602?source=rss-8de6bc946802------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae060e597602</guid>
            <category><![CDATA[bpf]]></category>
            <category><![CDATA[linux-kernel]]></category>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Tungdam]]></dc:creator>
            <pubDate>Wed, 30 Jun 2021 07:03:23 GMT</pubDate>
            <atom:updated>2021-07-03T03:10:00.363Z</atom:updated>
            <content:encoded><![CDATA[<p>How to pick the right event[s] to drill down.</p><h3>Preface</h3><p>If you’re reading this blog, most likely you’re fan of Linux tracing and BPF based tools. Given its power, it’s very tempting to jump into your new shiny <a href="https://github.com/iovisor/bpftrace/tree/master/tools">tools box</a> to troubleshoot problems. Sometimes it works, but mostly it doesn’t. In this blog, I’ll share my own strategy to get the best of our tracing tools. When should we use it and how.</p><p>This is the 3rd installment of my “Practical Linux tracing” series. See the previous ones here:</p><p><a href="https://medium.com/coccoc-engineering-blog/practical-linux-tracing-part-0-x-why-did-i-trace-5597a35a3bb5?source=friends_link&amp;sk=0dfc93badcefec8bdd192f5d21781f26">Why did i trace ?</a></p><p><a href="https://medium.com/coccoc-engineering-blog/things-you-should-know-to-begin-playing-with-linux-tracing-tools-part-i-x-225aae1aaf13">Symbols, debug symbols and stack trace</a></p><p><a href="https://medium.com/coccoc-engineering-blog/practical-linux-tracing-part-2-x-profiling-vs-event-tracing-89f76113e0fd">Tracing vs profiling</a></p><h3>The very first strategy: think before you start</h3><ul><li>You always start with an <strong>assumption</strong> about what happen ( event[s] ) . Tools help you to <strong>verify</strong> it.</li><li>Check automated alert with metrics, simple graphs for basic resource consumption, application logs. If those tools can’t clearly verify your assumption. Then start tracing down more.</li><li>Don’t start firing BPF tools immediately like an idiot.</li></ul><h3>An easy exception: when you know what to do</h3><p>Yeah, it’s simple , right ? Either you did it before ( and it works ) or you have a very clear simple task need to verify, and it requires tracing tools to solve. Just nail it.</p><p>For example, recently I had to assess the impact of <a href="https://security-tracker.debian.org/tracker/source-package/libwebp">multiple libwebp vulnerabilities </a>. Frankly say i don’t know much about this stuff, and most likely we don’t use it. But there are some PHP application on some nodes load it into its memory:</p><pre>grep libwebp /proc/25760/maps<br>7f7bc6af3000-7f7bc6af6000 r--p 00000000 09:01 38281904                   /usr/lib/x86_64-linux-gnu/libwebp.so.6.0.2<br>....</pre><p>Now we need to know if there’s any function from that lib is really executed at any time or not. Great case for showing off, bpftrace comes to rescue:</p><pre>bpftrace -e &#39;uprobe:/usr/lib/x86_64-linux-gnu/libwebp.so.6.0.2:* {time(&quot;%H:%M:%S &quot;); printf(&quot;%s %d\n&quot;,comm,pid);}&#39; | tee /tmp/libwebp.trace</pre><p>I let it run for days. Not even a single event. We can rest assured and waiting for the fix.</p><p>Sounds nice ? But most of the time, it’s not the case. You usually don’t know exactly what to do. Your metrics and application logs are not enough. When your 2/3 pillars of observability failed, an other easy choice to start tracing. But to trace what ? That’s the question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*Dfx4Aflj2xemZhBKkcqCuA.png" /><figcaption><a href="https://cdn2.iconfinder.com/data/icons/project-management-1-20/65/45-512.png">https://cdn2.iconfinder.com/data/icons/project-management-1-20/65/45-512.png</a></figcaption></figure><h3>Bombarding strategy</h3><p>Start from the broad view, collect as much as you can, then choose specific event[s] to drill down based on:</p><h4>Hot code</h4><p>Profiling and visualizing ( perf / bpftrace profile + flamegraph , for example ), to get the full picture of on-CPU usage, pick the hot-spot ( most frequently executed function, Flamegraph doesn’t show anything related to latency) and start drilling down to that specific function.</p><p>For example:</p><p>We once got a situation with our squid proxy when there was a sudden growth on our incoming traffic in the evening. The traffic could reach half of its capacity then dropped down like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*dKqKFT3RxfomKnkgu6jyCg.png" /></figure><p>The CPU usage mostly from %sys</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*0UC0yIFA1G2mBG7-ir0ocA.png" /></figure><p>Info about TCP</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*1_FM3BQ-HwJuQ8yfVrh90Q.png" /></figure><p>Just thinking something got stuck with TCP , start a quick profiling and got the hot code</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6vIUTJEci9g6wCeBFLYdug.png" /></figure><p>It called <a href="https://github.com/torvalds/linux/blob/f40ddce8/net/ipv4/inet_connection_sock.c#L133">inet_csk_bind_conflict</a> and <a href="https://github.com/torvalds/linux/blob/f40ddce8/net/ipv4/inet_connection_sock.c#L354">inet_csk_get_port</a> much more than usual. As we’re been working with squid proxy for quite a while, this should be related to the situation when squid have to handle much more than 50k connection to the remote host. The limit here is the source port ( in our case ip_local_port_range set to 15000 65000 ). Squid itself does return some error message about “can’t not bind socket” but not significantly more frequently than usual, which is pretty strange.</p><p>A simple fix is to <strong>avoid solving bind conflict</strong>, by adding more IP to interface or even add more interface to the box so it can handle the proxies requested more. We already add more IPs to our interface but turned out it’s not enough, so we added more and problem’s solved.</p><h4><strong>Slow code</strong></h4><p>Collecting a broad group of events, with info about its latency, pick out the ones which have a latency outlier and start examine its details. The same technique as implemented by <a href="https://github.com/iovisor/bcc/blob/master/tools/funcslower.py">funcslower.py</a> or <a href="https://github.com/brendangregg/perf-tools/blob/master/kernel/funcslower">funcslower ( ftrace version )</a> .</p><p>An important note on this: the slow code most likely is the result of being off-CPU ( voluntary to wait for IO or involuntary by CPU scheduler ) . So sometimes tracing a slow code turns out to be tracing context switch related events.</p><p>Bombarding strategy’s biggest problem is overhead. Profiling too many events at the high rate can affect your performance significantly. But doing it in the low rate may lose the important details.</p><p>To solve that issue, we need to find the way to narrow down our possible targets.</p><h3><strong>Narrowing down the target</strong></h3><p><strong>By examining the source code</strong>. Don’t worry, it’s not that complicated as it sounds. We can do it by:</p><ul><li>We can always start from application log. By using error message or info from access log of your application as the query to search from source code, we’ll find exactly the function which produces the message to drill down.</li><li>Searching by function feature: normally, function name is quite self-descriptive. By searching the source code with some keyword of our feature we’re interested in, we can find the related function quite easily. I personally find <a href="https://livegrep.com/search/linux">livegrep</a> or <a href="http://cscope.sourceforge.net/cscope_vim_tutorial.html">cscope</a> are very useful for kernel source searching. And github search feature is also good enough in general.</li></ul><p>After having the targets narrowed down, we need to reproduce the bug / issue ( isolated env is preferred ) then start tracing those events for further investigation.</p><p>For example, we once met a weird problem with <strong>bind</strong> dns, when the negative record caching doesn’t behave as expected.</p><p>A simple search query like this lead me to a function definition we may need: <a href="https://github.com/isc-projects/bind9/search?q=negative+cache">https://github.com/isc-projects/bind9/search?q=negative+cache</a></p><p>dns_ncache_* should be our target for tracing.</p><p>A quick check on named ‘s libraries will show us more info about our tracing target:</p><pre>for x in `ldd /usr/sbin/named | awk &#39;{print $3}&#39;`; do  objdump -T $x | grep dns_ncache &amp;&amp; echo $x; done<br>00000000000a2cc0 g    DF .text 000000000000001e  Base        dns_ncache_add<br>00000000000a2ce0 g    DF .text 0000000000000022  Base        dns_ncache_addoptout<br>00000000000a2d10 g    DF .text 000000000000093e  Base        dns_ncache_towire<br>00000000000a3650 g    DF .text 0000000000000441  Base        dns_ncache_getrdataset<br>00000000000a4040 g    DF .text 0000000000000392  Base        dns_ncache_current<br>00000000000a3aa0 g    DF .text 0000000000000597  Base        dns_ncache_getsigrdataset<br>/lib/x86_64-linux-gnu/libdns.so.110</pre><p>Having targets on hand, we can those events while replaying our requests to reproduce the case.</p><p>I actually couldn’t figure out the root cause by tracing because my colleague was quicker to find the root cause with this <a href="https://sources.debian.org/patches/bind9/1:9.11.5.P4+dfsg-5%7Ebpo9+1/10_min-cache-ttl.diff/#L232-L239">buggy patch</a> when min-ncache-ttl swapped minttl with maxttl in the wrong way. But we were close, and having knowledge about the events, we can verify the assumption easily on the fly.</p><h3>Conclusion</h3><p>It’s difficult to form a formula to solve ad hoc problem. As it’s ad hoc, we should just aim for accumulating experiences along the way. The best strategy turned out to be improvising only :)</p><p>What I wrote above is a small summary about how did i deal with my case along the way that is reusable. Sometimes we still need luck ( and hours and hours of researching ) to really figure out the root cause, especially the root cause of an interesting issue. Hopefully this blog can give you some clues to start on the very first steps of your tracing journey. Looking forward to your feedback on this, as well as to listening to your war stories on your own way.</p><p>Thank you for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae060e597602" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coccoc-engineering-blog/practical-linux-tracing-part-3-x-strategies-for-adhoc-analysis-ae060e597602">Practical Linux tracing ( part 3 ): Strategies for adhoc analysis</a> was originally published in <a href="https://medium.com/coccoc-engineering-blog">Coccoc Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Our lessons on Linux writeback: do “dirty” jobs the right way]]></title>
            <link>https://tungdam.medium.com/our-lessons-on-linux-writeback-do-dirty-jobs-the-right-way-fa1863a5a3dc?source=rss-8de6bc946802------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa1863a5a3dc</guid>
            <category><![CDATA[bpftrace]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[linux-kernel]]></category>
            <dc:creator><![CDATA[Tungdam]]></dc:creator>
            <pubDate>Fri, 02 Apr 2021 07:33:13 GMT</pubDate>
            <atom:updated>2025-01-18T05:26:23.632Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>TL;DR</strong></p><ul><li>Writeback is innocent. It ensures data consistency and free up memory for other tasks. Understand it then it will pay you back.</li><li>Unfortunately, sometimes writeback can be the root cause that triggers other things that may affect your very latency-sensitive application: TLB shootdown, high context-switches, <a href="https://lwn.net/Articles/442355/">stable pages write</a>.</li><li>Writeback itself rarely has a direct impact on your application. <strong>It does not block your write</strong>. <a href="https://github.com/torvalds/linux/blob/v4.19/mm/page-writeback.c#L1554">Dynamic throttling mechanism</a> does.</li><li>Writeback can be an <strong>indicator</strong> of performance issues, for example, memory saturation or heavy disk IO due to over-syncing data. <a href="https://github.com/iovisor/bpftrace/blob/master/tools/writeback.bt">Tracing writeback</a> <strong>reasons</strong> can help to reveal root causes in those cases.</li></ul><h4><strong>1. What is writeback ?</strong></h4><p>Writeback is the process of writing dirty pages in memory back to permanent storage.</p><p>I won’t explain much about dirty pages, write buffer, deferred/asynchronous write in this blog. Read <a href="https://www.thomas-krenn.com/en/wiki/Linux_Page_Cache_Basics">this</a> in case you dont’ know what it is.</p><p>I’ll use write buffer and pagecache interchangably in this blog. Though precisely, dirty pages are all in pagecache.</p><p>This section is just to distinguish this Linux dirty pages writeback with hard disk’s cache writeback.</p><h4><strong>2. Why should we care about writeback ?</strong></h4><p>Because write buffers and dirty pages are actually cool. It can speed up our write significantly in many cases when it’s allowed to. Then, we need a writeback mechanism to eventually save our volatile data in dirty pages to a persistent storage media. A good understanding about writeback can help us to:</p><ul><li>Balance between data safety and performance</li><li>Avoid extreme cases that can cause very big latency outliers application, as well as revealing the root cause of such outliers easier.</li></ul><h4><strong>2.1 A balance setting for data safety vs performance</strong></h4><p>Wise men said, understanding our applications’ behaviors helps us to maximize the performance from limited resources. That’s true for dirty pages and writeback.</p><p>Look at these settings</p><pre>vm.dirty_background_bytes = 204800000<br>vm.dirty_bytes = 819200000<br>vm.dirty_writeback_centisecs = 3000</pre><p>It looks weird ( much much higher than usual ) but actually <strong>works great</strong> on our our special purpose server, where:</p><ul><li>The main part is nginx proxy, which generates 200+ G uncompressed log per day.</li><li>The most important data there are those logs. And they are shipped on the fly to kafka for later processing.</li><li>The logs are stored locally for debugging purpose only.</li><li>Any other application, if there’s any critical data, sync it by itself.</li><li>Have plenty of free mem ( around 20/32G ), but i strongly believe half of that amount will be more than enough</li><li>Can’t catch up with such IO workload on 2 x 7200 RPM HDD</li></ul><p>In the worst case, when that small box dies unexpectedly somehow, we’ll lose ~ 3 seconds of log locally, thanks to write_back_centisecs setting. Everything’s fine. Cheap boxes are used wisely.</p><p>But of course, there’s no “one size fits all”.</p><h4><strong>2.2 Hard lessons: writeback can cause a crazy latency outlier</strong></h4><p>Even though write buffer are well-known with its performance benefit, it’s close friend writeback can be the source of an latency outliers as well.</p><p>For example, like in the graph below, the p99 latency in 1 of our critical app went skyrocket ( to 10 seconds ! ) sometimes when there was an other process writing data to disks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*LocH6msQpoN_rb5Bp2xv4Q.png" /></figure><p>A bit more info for our case:</p><ul><li>Our java app’s main task is to process around 80G data which is locked in memory ( mmap / mlock ). Disks IO are trivial , only logging. GC log is in tmpfs</li><li>There were still plenty of headroom for free CPU and memory. The disks itself looked comfortable still ( up to 40% util, there wasn’t the same spike pattern in queue length or iops ).</li></ul><p>The reason, after several interesting tracing sessions conducted, was the mis-configuration in our dirty settings by accident. The <strong>vm.dirty_background_bytes</strong> is quite high, it’s 200M ( inherited from the server described above ). To get rid of the latency spikes, there are 2 choices:</p><ul><li>Modify our process which does the IO in background to use <strong>direct IO.</strong></li><li>Low down our dirty_*_bytes settings to several bytes, to mitigate the impact of whatever writeback does.</li></ul><p>We chose the second to avoid extra work, and because, obviously, we didn’t set those settings on this server on purpose. More like reverting.</p><p>Taking some more time to research on this, this writebach-vs-java-long-pause problem was also reported long ago in great details here <a href="https://engineering.linkedin.com/blog/2016/02/eliminating-large-jvm-gc-pauses-caused-by-background-io-traffic">in this article</a>. We’re not alone in this case, that’s good.</p><p>Let’s see what could be the reason.</p><p><strong>2.2.1 Writeback as a root cause</strong></p><p>Searching around, i found another great article <a href="https://rigtorp.se/virtual-memory/">latency implication of virtual memory</a>, writeback is mentioned several times as a factor that can cause us some trouble with latency outliers, which may or may not the root cause of our issue but is still worth mentioning here:</p><ul><li>It incurs <a href="https://en.wikipedia.org/wiki/Translation_lookaside_buffer">TLB </a>shootdown, which requires a CPU core to pause whatever it is to inform ( interrupt ) other cores about the new info about memory pages it has. This process normally is super quick, but it can cause our application latecy ( especially memory-intensive one ) an unstable latency. ( Because it has to wait for its turn to be “on-CPU” ) . Couldn’t be that big as our case though. Much much smaller.</li><li>Another situation for context-switches. See how wribeback its woken up as a background process:</li></ul><pre>static void wb_wakeup(struct bdi_writeback *wb)<br>{<br>    spin_lock_bh(&amp;wb-&gt;work_lock);<br>    if (test_bit(WB_registered, &amp;wb-&gt;state))<br>        mod_delayed_work(bdi_wq, &amp;wb-&gt;dwork, 0);<br>    spin_unlock_bh(&amp;wb-&gt;work_lock);<br>}</pre><p>wb_wakeup calls mode_delayed_work with delay param 0, means it’s guaranteed to schedule <a href="https://github.com/torvalds/linux/blob/v4.19/kernel/workqueue.c#L1575">immediately</a> on its current CPU. Thus too many writeback can cause high number of context-switches. This can contribule some extra latency to our case. Too much data to push can cause lower priority app to be scheduled to wait longer.</p><p>Fortunately, we can verify the CPU scheduling issue by bpftrace easily, by using <a href="https://github.com/iovisor/bpftrace/blob/master/tools/runqlen.bt">runqlen.bt</a> and <a href="https://github.com/iovisor/bpftrace/blob/master/tools/runqlat.bt">runqlat.bt</a> . TLB shootdown can also by tracked by tracing <a href="https://github.com/torvalds/linux/blob/v4.19/mm/memory.c#L261">tlb_finish_mmu</a> kprobe.</p><ul><li>While writing a dirty memory page to disk, if there’s an application writing to that page, it will be <strong>blocked waiting</strong>, till the writeback process finished. This is an extreme case. And the latency is also very small. ( Except for the answer an extreme case, when you’re writing that page to a broken USB stick :D )</li></ul><p><strong>2.2.2 Writeback as an effect</strong></p><p>Of other events that put much more load to your server or block your application directly.</p><p><strong>a. Page reclaim causes a zoo of chaos</strong></p><p>Theoritically, page reclaim can cause a zoo of chaos creatures in your system.</p><p>It can goes like this when you share an application with a busy database server :D</p><ul><li>A heavy SELECT query failed all the cache and grab too many data on disks, causing huge disk IO.</li><li>Linux adds as much data was once read as it can to pagecache. And it needs more memory for pagecache</li><li>When there’s not enough free memory for pagecache, page reclaim kicks in and trying to flush dirty data to disks. How ? <a href="https://github.com/torvalds/linux/blob/v4.19/mm/vmscan.c#L1991"><strong>by calling writeback</strong></a><strong> . </strong>If you are lucky and your SSD handle all this sh!t quick enough , everything’s fine.</li><li>If you’re unlucky, your application may write to a page which is being flushed back to disk ( stable pages issue — explained above 2.2.1 , third bullet ) , thus is blocked waiting.</li><li>If you’re very unlucky, your application needs some memory at the same time when Linux grabs all free mem for pagecache, thus you are , again , blocked waiting for <a href="https://www.kernel.org/doc/html/latest/admin-guide/mm/concepts.html#reclaim">direct reclaim</a>. This can be traced by my idol’s <a href="https://github.com/brendangregg/bpf-perf-tools-book/blob/master/originals/Ch07_Memory/vmscan.bt">vmscan</a> tool.</li></ul><p>In the 2 last points, you may be surprised when your application is affected by disk IO activities, while it works with memory completely.</p><p><strong>b. Your write is throttled</strong></p><p>By writeback, you may think.</p><p>No, it’s not writeback</p><p>For virtually every write ( except for direct IO one ) , the writer has to go through several checks, and 1 of them is <a href="https://github.com/torvalds/linux/blob/v4.19/mm/page-writeback.c#L1554"><strong>balance_dirty_pages</strong> </a>function.</p><p>The kernel stack looks like this</p><pre>        balance_dirty_pages+1<br>        balance_dirty_pages_ratelimited+716<br>        generic_perform_write+362<br>        __generic_file_write_iter+254<br>        ext4_file_write_iter+198<br>        new_sync_write+251<br>        vfs_write+165<br>        ksys_write+87<br>        do_syscall_64+83<br>        entry_SYSCALL_64_after_hwframe+68</pre><p>Here is what it does:</p><blockquote>/* * balance_dirty_pages() must be called by processes which are generating dirty * data. It looks at the number of dirty pages in the machine and will force * the caller to wait once crossing the (background_thresh + dirty_thresh) / 2. * If we’re over `background_thresh’ then the writeback threads are woken to * perform some writeout. */</blockquote><p>And here is the code, as my view: ( :D )</p><pre>static void balance_dirty_pages(struct bdi_writeback *wb,<br>                                unsigned long pages_dirtied)<br>{<br>        ...<br>        // Declaring variables<br>        for (;;) {<br>                ...<br>                // Some magics to ensure the condition in description<br>                ...<br>                trace_balance_dirty_pages(wb,<br>                                          sdtc-&gt;thresh,<br>                                          sdtc-&gt;bg_thresh,<br>                                          sdtc-&gt;dirty,<br>                                          sdtc-&gt;wb_thresh,<br>                                          sdtc-&gt;wb_dirty,<br>                                          dirty_ratelimit,<br>                                          task_ratelimit,<br>                                          pages_dirtied,<br>                                          period,<br>                                          pause,<br>                                          start_time);<br>                __set_current_state(TASK_KILLABLE);<br>                wb-&gt;dirty_sleep = now;<br>                io_schedule_timeout(pause);<br>                ...</pre><pre>}<br>        .....<br>}</pre><p>So, this function is what put your process to be blocked waiting for IO, by calling <strong>io_schedule_timeout(pause)</strong>, not writeback. Writeback should be a background process, and it shouldn’t block application directly. Otherwise it’s a foreground process.</p><p>As you can see above we have a tracepoint called balance_dirty_pages to track when it happens and which process is blocked. Using ftrace by running:</p><pre>/sys/kernel/debug/tracing/events]# echo 0 &gt; writeback/balance_dirty_pages/enable</pre><p>And check the <strong>paused </strong>value<strong> , </strong>you’ll be able to see how long your application is block<strong>.</strong></p><p>Otherwise you can trace io_schedule_timeout directly , looks at it stack to make sure it’s called by balance_dirty_pages and extract the pause value from there.</p><p>The writeback role in this case is very simple: it reduces the dirty pages, till the number of dirtied pages ≤ ( background_thresh + dirty_thresh ) /2. It was called by balance_dirty_pages , thus an effect of the events chain only.</p><p>The quicker writeback can do its job, the less number of times a writing process is throttled.</p><p>Set a small value for dirty_*_bytes can help to speed up the writeback process , but it comes with the cost of being throttled more often ( than setting a big value ).</p><p>Once again, there’s no “one size fits all”. You have to choose whether to be throttled more often in short time or being rarely blocked but in the much higher duration.</p><h4>3. Conclusion</h4><p>Thank you for coming this far. I’ve just summarized everything I know about how writeback can affect our application performance.</p><p>It has pros and cons but IMHO, it plays a very important role while troubleshooting your application latency outliers, which is always be fun. ( Every engineer wants to reveal a secrect of an edge case )</p><p>Hopefully it will help you someday. If it will, leave me a comment here please, I would be very happy.</p><p>Thank you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa1863a5a3dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A calm sysadmin]]></title>
            <link>https://medium.com/coccoc-engineering-blog/a-calm-sysadmin-3c97af7ac357?source=rss-8de6bc946802------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c97af7ac357</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[soft-skills]]></category>
            <category><![CDATA[sysadmin]]></category>
            <dc:creator><![CDATA[Tungdam]]></dc:creator>
            <pubDate>Thu, 07 Jan 2021 03:58:43 GMT</pubDate>
            <atom:updated>2021-01-07T04:55:54.637Z</atom:updated>
            <content:encoded><![CDATA[<p>I promised my team to publish a blog post within last week, to actually meet our humble target for public blog posts last quarter ( We’re trying to write more but are all lazy ). This should be an article to continue my <a href="https://medium.com/coccoc-engineering-blog/practical-linux-tracing-part-0-x-why-did-i-trace-5597a35a3bb5">tracing series</a> or a new finding about how dirty page write back affects our latency sensitive java app recently, but it ended up like this. You can call this a short prose about sysadmin’s life.</p><p>The idea of a calm sysadmin came from the last afternoon of 2020, when everybody slowed down to enjoy the last day of the year, there’s no deployment requests, no hard discussions, no bug, much less notifications… Everything looks calm and peaceful.</p><p>But that’s not the calm I’m going to write about now. What a sysadmin really needs is <strong>calm under pressure. Imperturbable. </strong>Only being calm can help us to handle the situation ( usually urgent one ) properly instead of screwing things up. That’s is the first and foremost requirement before entering any incident.</p><p>Here’s my compilation in a random order about why we should stay calm no matter what the situation is. They are what i learned hard over the years. You may want to update it yourself.</p><ul><li>To not mad at someone’s mistake. Firstly, because everybody makes mistake. Secondly, that mistake sometimes turns out its your settings / commits / changes several months ago :)</li><li>Listen to other’s suggestions / solutions carefully even though it may sound crazy at the first glance. Otherwise, people will throw <a href="https://therongolianstar.files.wordpress.com/2018/07/quote-first-they-ignore-you-then-they-laugh-at-you-then-they-fight-you-then-you-win-mahatma-gandhi-68010.jpg?w=850">this</a> at you.</li><li>To not fix your long-time bug immediately but consider the outcome with third parties before applying changes. Sometimes your bug works OK with other’s bugs :) . Fixing it right now without correcting others may lead to catastrophic situation. Trust me, I experienced it hard.</li><li>To note every changes you made throughout the emergency situation, and revert ones without clear impact.</li><li>To have a walk, take a shower or sleep after hours of debugging without any progress. When you look at the top screen more than 5 times per min, it’s the signal. Our brain needs some rest.</li><li>To not apply your changes to production at weekend even though you’re sure that it definitely can help to boost your system’s performance 2 times. First, you violate your rule about changing things on production. Second, you may ruin your team’s relax time.</li><li>To read references on Stackoverflow thoroughly :) , to not just blindly trust the solution from a random guy on the internet. If the situation allows, we should better read a paper or a long article instead. But Stackoverflow is still very helpful :)</li><li>To pick a right metric among dozens of others that may lead to a wrong direction while chasing the root cause.</li><li>To check the client side when your bosses can’t connect to our company’s homepage while our monitoring system reports nothing ( This is actually pretty tricky, you better understand study your bosses more before apply this advice :D )</li><li>To get help. Know your time limit while debugging production issues on your own, escalate quickly when such limit’s exceeded. Call Subject Matter Expert by phone in urgent case. Calm != slow.</li><li>To let your team solve a problem themselves in the midnight and go to sleep, because you know that you have to cover them in the morning.</li><li>To guide your team through an emergency situation by <strong>listening</strong> to their input / report before making order[s].</li><li>To truly see the problem / bug / issue you’ve been stuck with for weeks is <strong>actually a great chance to learn new things. </strong>It’s NOT OK to let a production issue happens more than once, but with a calm mind and a positive learning spirit, “it shall too pass”.</li><li>To really conduct a <strong>blameless</strong> post-mortem report. Especially when you’re the guy who made the mistake, it’s 10 times harder :)</li><li>To let the juniors struggle and make mistakes. They need room to breath and learn.</li><li>To pause to analyze more before telling your subordinate that he’s not doing well. And to tell him so when it’s true, with a warm tone of your voice.</li><li>To not kill your fellow developers when she/ he requests to deploy at 6PM on Friday, 31/12/2020</li></ul><p>Once again, this is what i learned i should do, not what i master as a calm sysadmin. People who know me can easily spot out that I’m not a calm person in real life. But over time, i think we can practice to be a calm and thus better sysadmin because the industry sometimes need us for super urgent situation, just like fire fighter in real life.</p><p>Let’s prepare for the worst , stay calm to be precise in emergency situation. Again, thus to be a better sysadmin.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c97af7ac357" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coccoc-engineering-blog/a-calm-sysadmin-3c97af7ac357">A calm sysadmin</a> was originally published in <a href="https://medium.com/coccoc-engineering-blog">Coccoc Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical linux tracing part 0/X: why did i trace ?]]></title>
            <link>https://medium.com/coccoc-engineering-blog/practical-linux-tracing-part-0-x-why-did-i-trace-5597a35a3bb5?source=rss-8de6bc946802------2</link>
            <guid isPermaLink="false">https://medium.com/p/5597a35a3bb5</guid>
            <category><![CDATA[war-stories]]></category>
            <category><![CDATA[system-administration]]></category>
            <category><![CDATA[tracing]]></category>
            <dc:creator><![CDATA[Tungdam]]></dc:creator>
            <pubDate>Thu, 31 Dec 2020 08:57:19 GMT</pubDate>
            <atom:updated>2021-01-05T14:30:20.429Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I had a presentation for my team about bpftrace, what i think it’s very cool. Turns out that nobody likes it. Mostly because of my presentation skill, i guess. Though, the biggest question was asked is <strong>why.</strong></p><p>Why do i need this thing ?</p><p>It’s cool, but it takes effort to learn some “low level things” , why should i do that ?</p><p>And I also asked myself, why do i want to share this with everybody ?</p><p>This is not a technical post. This is a story.</p><h3>“What will you do if this service is broken ? you CALL ME”</h3><p>My developer fellow yielded at me on our skype group. When I asked him info in details about the internal implementations of his software. Maybe i asked too much. He got angry and told me so. He was right. Then I shut up.</p><p>In our company, we sysadmins run our production services. We’re in charge of it. Paid by it. Take responsible for it. Then I want to know as much as possible about what i am running.</p><p>That’s difficult. I can’t write the code they’re doing, obviously. I even can’t understand it. But i should know some symptoms about when things are going wrong. The most frustration moments of my professional life is when i keep looking at my top screen, seeing all CPU cores are consumed, some processes that are using 1xxx % CPU, and don’t know what to do. I still have it these days sometimes.</p><p>His sentence banged my head hard. Days after i keep asking myself: what am I doing here ? Do what developers tell me as a robot ? Live with black box services forever and just call our developer when something’s wrong ?</p><p>Yes. I had been doing so for some time. Till one day, one of our critical service had an intermittent issues that nobody knew why, even its creator.</p><p>I spent several days looking at atop output, checking every change in each 10 seconds sample that we had around the time that problem occurred. Every difference between 10 seconds, I took a note about it. Then somehow i found the page scan rates was significantly high around the time of problem, which led me to this <a href="https://engineering.linkedin.com/performance/optimizing-linux-memory-management-low-latency-high-throughput-databases">great article</a> about NUMA and page scan. It set zone_reclaim_mode to 0 and the problem was gone.</p><p>I was very happy, then started telling myself that this is exactly what i need to do more. I started digging more into our system. And i was lucky because that time we had a system with good scale and a very professional team that could do a lot of optimization. With the help of the developer i mentioned above ( he’s one of the coolest one i have worked with so far ), i learned a lot about filesystem, pagecache, huge pages, memory allocation ( tcmalloc, jemalloc ) , JVM ( gc and JIT compiler ).</p><p>But my tweak usually didn’t work well. I couldn’t measure it precisely. Because most of the things in my area of responsibility is in side the kernel. I realized that i just can’t poking around randomly anymore.</p><p>I need to understand Linux kernel more. How ? Read books. <a href="https://www.amazon.com/Understanding-Linux-Kernel-Third-Daniel/dp/0596005652">This</a> one and <a href="https://www.amazon.com/Linux-Kernel-Development-Robert-Love/dp/0672329468/">this</a> one. Okay, but it’s super slooooow progress. It’s very difficult for me to understand all the concepts, not to mention code, data structure and algorithm inside. I need some thing more…practical. Some thing that can give me a better view of my system. In action.</p><p>That’s when i found <a href="http://www.brendangregg.com/">Brendan Gregg blog</a>, and i entered Linux tracing area.</p><p>Over years, with his great effort with perf, ftrace, then the born of BPF tools, we now can have much better view of our system. We can do things what we ( as a sysadmins, not kernel developers ) only can dream of before.</p><p>My NUMA issue mentioned above can be traced easily by <a href="https://github.com/brendangregg/bpf-perf-tools-book/blob/master/originals/Ch14_Kernel/numamove.bt">numamove.bt</a>.</p><p>Want to know per-process pagecache hit / miss ratio ? Use<a href="https://github.com/iovisor/bcc/blob/master/tools/cachetop.py"> cachetop</a>.</p><p>Need to profile Java application ? No problem , Perf can help you <a href="https://netflixtechblog.com/java-in-flames-e763b3d32166">this</a> way.</p><p>How to analyze the CPU contention that may affect my service ? You need something like 30 lines of code with bpftrace version of <a href="https://github.com/brendangregg/bpf-perf-tools-book/blob/master/originals/Ch06_CPUs/offcputime.bt">offcputime</a>.</p><p>With the help of tracing tools, now you will be able to know what is on cpu, off cpu, what put your process to off-cpu state !</p><p>With the grow of community, for tracing both user space and kernel space, now i have more things to check instead of banging my head onto the top screen. I should know which code is being executed, how is it executed and why it is excuted. If i don’t understand the code myself, i can show my fellow developers the info about his code. It’s much better than: your service is using a lot of CPU now.</p><p>I believe this should answer the question: why do we need to trace and why do we need to study the “low level” stuffs.</p><h3>“You should go back and do your job”</h3><p>Once again. It got this from another developer in my company. When i showed him the <a href="https://github.com/brendangregg/FlameGraph">flamegraph</a> of his application. And asked him to check it, something didn’t look right. Because he told me before that i need to dig deeper instead of just calling him.</p><p>That’s true. It’s not my job. I should care about kernel part only. And once again, i shut up. He continued working on that, but still now, the problem is still a mystery.</p><p>2 years later,</p><p>Till now, I still think we should trace it :) , if it can help to solve our problem. If we ( both sysadmins and developers ) tried everything and nothing helped, then why not ?</p><p>Perf_events, uprobe or UDST may reveal something that application metrics / log may miss. On-the-fly event tracing can be very helpful with source code analysis.</p><p>The best scenario in this case is to work together. By using Linux tracing tools, i can give developers internal activation records of his process , with the ability to zoom in many functions with proper debug symbols. And based on that info, he can check more in his source code, to compare what’s in his mind versus the reality. <strong>If he understands the benefits and trust my Linux tracing tools</strong>.</p><p>The thing is, most of standard tools are not mine, it was developed by professionals in giant tech companies, with many different use cases. And i understand the mechanism in background and trust it.</p><p>What i need to do now is to introduce it with people, and convince them to use it. This answers the question: why do i need these Linux tracing tools with everybody.</p><p>To understand it better, to troubleshoot quicker, to debug more precisely, together with my dear developers.</p><p>To provide run-time statistics, for better troubleshooter than I am can help to solve problem.</p><p>I failed at this so far. Thus have this blog then.</p><h3>Conclusion</h3><p>If you read this far, i hope that you can see why did <strong>I</strong> trace and have your own answers for other questions that i raised in the beginning of this blog.</p><p>I made this draft half year ago, then never sat down to check it again to finish it till now. Since that time, i had other projects came up that took a significant amout of my time for tracing, but everytime i touch it again, i’m really in flow. And tracing job, in turn, did give me back some magic moments that no other jobs can compare. Thus, i’ll keep writing tracing stories no matter if they can ( yet ) convince or impress others or not, because tracing does help me to solve my problems and more importantly, it sparks joy in my jobs.</p><p>Then, why do I trace ? To dig deeper, to understand the root cause, to solve hard problems. And for fun.</p><p>If you’re interested in those things as well, follow my other articles on tracing and give me your feedback please.</p><p><a href="https://medium.com/coccoc-engineering-blog/things-you-should-know-to-begin-playing-with-linux-tracing-tools-part-i-x-225aae1aaf13?sk=837b03e86ead77153835c5a4d2743934">Part 1</a></p><p><a href="https://medium.com/coccoc-engineering-blog/practical-linux-tracing-part-2-x-profiling-vs-event-tracing-89f76113e0fd?sk=520f39229889dfb8e4bc2ce736961706">Part 2</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5597a35a3bb5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coccoc-engineering-blog/practical-linux-tracing-part-0-x-why-did-i-trace-5597a35a3bb5">Practical linux tracing part 0/X: why did i trace ?</a> was originally published in <a href="https://medium.com/coccoc-engineering-blog">Coccoc Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>