<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Denis GERMAIN on Medium]]></title>
        <description><![CDATA[Stories by Denis GERMAIN on Medium]]></description>
        <link>https://medium.com/@dgermain_deezer?source=rss-1df55d0db6c5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*N2SZC_vBBVZbUbRl7e94uw.jpeg</url>
            <title>Stories by Denis GERMAIN on Medium</title>
            <link>https://medium.com/@dgermain_deezer?source=rss-1df55d0db6c5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:44:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dgermain_deezer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Migrating Cilium from Legacy iptables Routing to Native eBPF Routing in Production]]></title>
            <link>https://medium.com/deezer-engineering/migrating-cilium-from-legacy-iptables-routing-to-native-ebpf-routing-in-production-84a035af1cd6?source=rss-1df55d0db6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/84a035af1cd6</guid>
            <category><![CDATA[deezer]]></category>
            <category><![CDATA[cilium]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[cni]]></category>
            <category><![CDATA[ebpf]]></category>
            <dc:creator><![CDATA[Denis GERMAIN]]></dc:creator>
            <pubDate>Wed, 25 Oct 2023 09:00:51 GMT</pubDate>
            <atom:updated>2023-10-25T09:00:51.278Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Cilium logo" src="https://cdn-images-1.medium.com/max/1024/0*7XK9rD5C__bWoUZV" /></figure><p>At Deezer, we maintain both cloud and on-premise Kubernetes clusters (a few thousands of physical cores) in our data centers in Paris.</p><p>Since we’re responsible for these clusters, we need to choose the technical stacks that run our platform (hardware, OS, kubernetes distribution, etc.) To act as the CNI plugin (basically, the component that creates the containers internal network), we chose <a href="https://cilium.io/">cilium</a>, <a href="https://www.cncf.io/announcements/2023/10/11/cloud-native-computing-foundation-announces-cilium-graduation/">a newly graduated CNCF project</a>.</p><p>Unfortunately, running our own clusters rather than just managed instances comes with the risk of human error. The goal of this blog post is to share our experience dealing with a cilium misconfiguration on our production Kubernetes clusters.</p><p>Even though this incident had no impact on our end users (we caught it soon enough to fix it), it could have had dire consequences.</p><p>We’ll retrace the steps that led us to realize that we didn’t set up our cilium CNI plugin correctly, and how we fixed it in production.</p><h3>Symptoms</h3><p>After adding nodes and real production workload on new Kubernetes clusters, we started to notice some abnormal CPU consumption. <em>cilium-agents</em> containers were taking an average of 2 cores on each node, with a lot of variations (up to 5 cores peaks). This was surprising.</p><p>We tried to find what was causing so much CPU usage, and while checking the configuration of cilium, this is what we found:</p><pre>kubectl -n cilium exec -it cilium-aaaaa -- cilium status<br>[...]<br>Host Routing:           Legacy<br>Masquerading:           IPTables [IPv4: Enabled, IPv6: Disabled]<br>[...]</pre><p>And it was odd…</p><p>But let’s rewind a bit.</p><h3>cilium: iptables vs eBPF</h3><p>While iptables has been the default “routing stack” for most “CNI plugins” since the beginning of Kubernetes, it’s known to be really poor at handling even <em>not-so-big</em> clusters (&gt; 1000 pods).</p><p>The number of iptables rules tends to increase exponentially, and with it, your CPU usage. At some point, <em>walking</em> the list takes all the CPU of a given node and it becomes unresponsive.</p><p>And that’s pretty bad ;-P.</p><p>This is one of the many reasons we chose <strong>cilium</strong> as a CNI plugin over other implementations: it features an elegant solution for this issue by replacing iptables rules with <a href="https://ebpf.io/">eBPF kernel programs</a>.</p><p>Note: aside from eBPF (and cilium), there are other solutions to get rid of iptables for networking in Kubernetes. But we won’t get into more details in this article.</p><p>For some reason, while the cilium Helm chart is supposed to install the CNI plugin in eBPF native mode by default when the host supports it, this wasn’t the case here. We had everything cilium needed (see <a href="https://docs.cilium.io/en/stable/operations/system_requirements/">System Requirements documentation</a>), both the recent kernel (6.2) and the enabled kernel modules.</p><blockquote>We introduced eBPF-based host-routing in Cilium 1.9 to fully bypass iptables and the upper host stack, and to achieve a faster network namespace switch compared to regular veth device operation. <strong><em>This option is automatically enabled if your kernel supports it</em></strong><em>. </em>To validate whether your installation is running with eBPF host-routing, run cilium status in any of the Cilium pods and look for the line reporting the status for “Host Routing” which should state “BPF”.</blockquote><p>We still don’t know why <strong>native routing</strong> mode wasn’t enabled by default, but the logs were pretty clear that it was not and what to do to prevent this:</p><pre>kubectl -n cilium logs cilium-aaaaaa<br>[...]<br>level=info msg=&quot;BPF host routing requires enable-bpf-masquerade. Falling back to legacy host routing (enable-host-legacy-routing=true).&quot; subsys=daemon</pre><p>So, we deduced that the only thing we needed in order to fix the issue was to change the cilium chart’s value and set bpf.masquerade to true:</p><pre>[...]<br>  kubeProxyReplacement: strict<br>+ bpf:<br>+  masquerade: true<br>[...]</pre><h3>Problem</h3><p>The problem here was obviously more complicated than just changing the value of a chart, or else there would be no reason to write a blog post about it!</p><p>Since a lot of apps and traffic were already running in the dev and prod misconfigured clusters, we didn’t want to:</p><ul><li>cause an interruption for the hosted apps,</li><li>risk finding ourselves in some inconsistent state during the change from iptables to ebpf.</li></ul><p>So, we had to find a way to <strong>only</strong> change some nodes, and check if it was possible to have apps running on nodes with iptables routing and other nodes with eBPF routing (in terms of internal/external connectivity).</p><h3>Preparing a sandbox cluster</h3><p>To achieve this, we recreated a sandbox cluster identical to the misconfigured ones.</p><p>This cluster had only 5 nodes (3 masters, 2 workers) but it was enough to run some basic tests initially. The first thing to check was if the sandbox cluster was indeed misconfigured (aka in <em>legacy routing</em>).</p><pre>kubectl -n cilium exec -it cilium-bbbbb -- cilium status<br>[...]<br>Host Routing:           Legacy<br>Masquerading:           IPTables [IPv4: Enabled, IPv6: Disabled]<br>[...]</pre><p>Then, we checked that the cluster was correctly set up. cilium gives various tools to check connectivity across the cluster, which is really handy.</p><p>First, you can launch a test from the CLI, which in turn deploys some containers and runs internal/external tests:</p><pre>➜  ~ cilium -n cilium connectivity test<br>ℹ️  Monitor aggregation detected, will skip some flow validation steps<br>✨ [sandbox] Deploying echo-same-node service...<br>✨ [sandbox] Deploying DNS test server configmap...<br>✨ [sandbox] Deploying same-node deployment...<br>✨ [sandbox] Deploying client deployment...<br>[...]<br>✅ All 32 tests (265 actions) successful, 2 tests skipped, 0 scenarios skipped.</pre><p>Then, there is also a command called cilium-health in the cilium <em>Pods,</em> which has a status command. cilium sends periodic probes to all Kubernetes clusters nodes and reports back latency.</p><pre>kubectl -n cilium exec -it cilium-ccccc -c cilium-agent -- cilium-health status<br>Probe time:   2023-09-12T14:47:03Z<br>Nodes:<br>  sandbox-02 (localhost):<br>    Host connectivity to 172.31.0.152:<br>      ICMP to stack:   OK, RTT=860.424µs<br>      HTTP to agent:   OK, RTT=110.142µs<br>    Endpoint connectivity to 10.0.2.56:<br>      ICMP to stack:   OK, RTT=783.861µs<br>      HTTP to agent:   OK, RTT=256.419µs<br>  sandbox-01:<br>    Host connectivity to 172.31.0.151:<br>      ICMP to stack:   OK, RTT=813.324µs<br>      HTTP to agent:   OK, RTT=553.445µs<br>    Endpoint connectivity to 10.0.1.53:<br>      ICMP to stack:   OK, RTT=865.976µs<br>      HTTP to agent:   OK, RTT=3.440655ms<br>[...]</pre><p>Finally, you can also just check the health of the <em>Pods</em> with the cilium status command.</p><pre>➜  kubectl -n cilium exec -ti cilium-ddddd -- cilium status<br>[...]<br>Host Routing:            Legacy<br>Masquerading:            IPTables [IPv4: Enabled, IPv6: Disabled]<br>[...]<br>Cluster health:          5/5 reachable   (2023-09-14T12:01:51Z)</pre><h3>Changing a node configuration (and one only)</h3><p>We were lucky because cilium 1.13 introduced a feature called <strong>per-node-config</strong> (<a href="https://docs.cilium.io/en/stable/configuration/per-node-config/#per-node-configuration">documentation is here</a>), which includes the possibility to deploy different configurations for cilium on a subset of nodes of our cluster.</p><p>Before that, if you wanted to test a change like what we are trying to achieve here, you had to change the cilium configuration (in cilium <em>ConfigMap</em>) and restart the <em>Pods</em> manually.</p><pre>cat &gt; cilium-fix.yaml &lt;&lt; EOF<br>apiVersion: cilium.io/v2alpha1<br>kind: CiliumNodeConfig<br>metadata:<br>  namespace: cilium<br>  name: cilium-switch-from-iptables-ebpf<br>spec:<br>  nodeSelector:<br>    matchLabels:<br>      io.cilium.enable-ebpf: &quot;true&quot;<br>  defaults:<br>    enable-bpf-masquerade: &quot;true&quot;<br>EOF<br><br>kubectl apply -f cilium-fix.yaml<br><br>kubectl label node sandbox-05 --overwrite &#39;io.cilium.enable-ebpf=true&#39;</pre><p>Once the node was correctly labeled, we deleted its cilium <em>Pod</em> for the new configuration to be taken into account. As soon as a new cilium <em>Pod</em> was re-created, we checked that the correct configuration had been applied.</p><p>Notes:</p><ul><li>In production, you should obviously drain the node before labeling it, and reboot it afterward.</li><li>Be sure to create the <em>CiliumNodeConfig</em> in the same namespace as cilium <em>Pods</em> or the specific configuration won’t be applied on your nodes.</li></ul><pre>➜  kubectl -n cilium exec -ti cilium-fffff -- cilium status<br>[...]<br>Host Routing:            BPF<br>Masquerading:            BPF   [ens18]   10.0.4.0/24 [IPv4: Enabled, IPv6: Disabled]<br>[...]<br>Cluster health:          5/5 reachable   (2023-09-14T12:01:52Z)</pre><p>We could see that both “Host Routing” and Masquerading are in BPF mode.</p><p>Hurray!</p><h3>Checking connectivity</h3><p>We could see in the previous commands that the cluster was ok in terms of connectivity: the last line reported that all nodes were “reachable”.</p><p>We also ran the other connectivity tests we performed previously to check that everything was fine.</p><pre>➜  ~ cilium -n cilium connectivity test<br>ℹ️  Monitor aggregation detected, will skip some flow validation steps<br>✨ [sandbox] Deploying echo-same-node service...<br>✨ [sandbox] Deploying DNS test server configmap...<br>✨ [sandbox] Deploying same-node deployment...<br>✨ [sandbox] Deploying client deployment...<br>[...]<br>✅ All 32 tests (265 actions) successful, 2 tests skipped, 0 scenarios skipped.</pre><h3>Curling 🥌</h3><p>Some of us like the philosophy “trust but verify.” For the sake of our own curiosity, we deployed a few web servers and ran a “curl” command against them regularly to see if the cilium configuration change (without draining/rebooting) had any impact.</p><p>That’s probably not really relevant since we would never do that in production. But still, there’s no harm in being curious ;-)</p><p>We applied the following manifest on our sandbox cluster:</p><pre>apiVersion: apps/v1<br>kind: DaemonSet<br>metadata:<br>  name: simple-webserver-daemonset<br>spec:<br>  selector:<br>    matchLabels:<br>      app: simple-webserver<br>  template:<br>    metadata:<br>      labels:<br>        app: simple-webserver<br>    spec:<br>      containers:<br>      - name: simple-webserver<br>        image: zwindler/vhelloworld:latest<br>        ports:<br>        - containerPort: 8081<br>        imagePullPolicy: Always</pre><p>Once deployed, we checked the <em>Pods</em> IP addresses:</p><pre>kubectl get pods -o wide<br>NAME                               READY   STATUS    RESTARTS   AGE     IP           NODE                 NOMINATED NODE   READINESS GATES<br>simple-webserver-daemonset-q4h44   1/1     Running   0          3m31s   10.0.4.87   sandboxkube-05   &lt;none&gt;           &lt;none&gt;<br>simple-webserver-daemonset-w97pv   1/1     Running   0          3m31s   10.0.3.85   sandboxkube-04   &lt;none&gt;           &lt;none&gt;</pre><p>Then, with a <em>Pod</em> in the cluster, we curled these <em>Pods</em>:</p><pre>kubectl run -it --image curlimages/curl:latest curler -- /bin/sh<br>If you don&#39;t see a command prompt, try pressing enter.<br>~ $ curl http://10.0.4.87:8081<br>hello from simple-webserver-daemonset-g2zdw<br>~ $ curl http://10.0.3.85:8081<br>hello from simple-webserver-daemonset-65k2n</pre><p>Since the web servers were reachable, we then began to change the Host Routing mode. In the curler <em>Pod</em>, we first ran a loop:</p><pre>while true; do <br>  date<br>  curl http://10.0.4.87:8081<br>  curl http://10.0.3.85:8081<br>  echo<br>  sleep 1<br>done</pre><p>We then labeled the node to change the routing policy and deleted the cilium <em>Pod</em> on that node for the configuration to be taken into account:</p><pre>kubectl label node sandbox-04 --overwrite &#39;io.cilium.enable-ebpf=true&#39;<br>kubectl -n cilium delete pod cilium-j6tt8</pre><p>Looking from the “curler” <em>Pod</em>, nothing seemed to have happened.</p><pre>Fri Sep 15 14:05:34 UTC 2023<br>hello from simple-webserver-daemonset-g2zdw<br>hello from simple-webserver-daemonset-65k2n<br><br>Fri Sep 15 14:05:35 UTC 2023<br>hello from simple-webserver-daemonset-g2zdw<br>hello from simple-webserver-daemonset-65k2n<br><br>Fri Sep 15 14:05:36 UTC 2023<br>hello from simple-webserver-daemonset-g2zdw<br>hello from simple-webserver-daemonset-65k2n</pre><p>Looking at the logs, we could see that, at some point, cilium discovered the running web servers and added them to its <em>Pods</em> list:</p><pre>level=info msg=&quot;Rewrote endpoint BPF program&quot; containerID=8b7be1b032 datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=1018 identity=4773 ipv4=10.0.3.85 ipv6= k8sPodName=default/simple-webserver-daemonset-w97pv subsys=endpoint<br>level=info msg=&quot;Restored endpoint&quot; endpointID=1018 ipAddr=&quot;[10.0.3.85 ]&quot; subsys=endpoint</pre><p>We did the same thing from a node with eBPF and checked that the connectivity was fine as well.</p><pre>kubectl run -it --image curlimages/curl:latest curler2 -- /bin/sh<br>If you don&#39;t see a command prompt, try pressing enter.<br>~ $ while true; do <br>&gt;   date<br>&gt;   curl http://10.0.4.87:8081<br>&gt;   curl http://10.0.3.85:8081<br>&gt;   echo<br>&gt;   sleep 1<br>&gt; done<br>Fri Sep 15 14:05:34 UTC 2023<br>hello from simple-webserver-daemonset-g2zdw<br>hello from simple-webserver-daemonset-65k2n<br><br>Fri Sep 15 14:05:35 UTC 2023<br>hello from simple-webserver-daemonset-g2zdw<br>hello from simple-webserver-daemonset-65k2n<br><br>Fri Sep 15 14:05:36 UTC 2023<br>hello from simple-webserver-daemonset-g2zdw<br>hello from simple-webserver-daemonset-65k2n</pre><h3>Keep on curling 🥌</h3><p>The previous test involved the <em>Pods</em> themselves, but we also ran the test against a <em>Service</em>, which is the Kubernetes entity that balances the HTTP calls between <em>Pods </em>replicas:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: simple-webserver-service<br>spec:<br>  selector:<br>    app: simple-webserver<br>  ports:<br>    - protocol: TCP<br>      port: 8081<br>      targetPort: 8081<br>  type: ClusterIP</pre><p>We curled the newly created service from both “curlers” <em>Pods</em>.</p><pre>while true; do<br>  curl http://simple-webserver-service:8081<br>  sleep 0.5<br>done<br><br>hello from simple-webserver-daemonset-65k2n<br>hello from simple-webserver-daemonset-65k2n<br>hello from simple-webserver-daemonset-g2zdw<br>hello from simple-webserver-daemonset-g2zdw<br>hello from simple-webserver-daemonset-65k2n<br>hello from simple-webserver-daemonset-g2zdw</pre><p>To be <strong>really</strong> sure, we ran a final test by calling a URL from outside the Kubernetes cluster. We just added an <em>Ingress</em> pointing to this <em>Service</em>.</p><p>Since we had already validated that <em>Service</em> to <em>Pod</em> communication worked, there was no reason for the <em>IngressController</em> to <em>Service</em> part to be an issue, but this way we tested from “end to end”.</p><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: simple-webserver-ingress<br>spec:<br>  ingressClassName: haproxy<br>  rules:<br>  - host: <br>    http: vhelloworld.domain.tld<br>      paths:<br>      - backend:<br>          service:<br>            name: simple-webserver-service<br>            port:<br>              number: 8081<br>        path: /<br>        pathType: ImplementationSpecific</pre><p>And it worked! 🥳</p><h3>Deploying it in production</h3><p>Testing in a sandbox is great, but we had to <strong>fix</strong> production. Most importantly, we wanted to do it in the <strong>least disruptive way</strong>.</p><p>In order to limit the impact in case something went wrong, we decided to take advantage of the fact that we had new nodes to set up.</p><p>Installing fresh new nodes directly with the right setup would allow us to progressively add traffic on the nodes with the eBPF cilium configuration instead of iptables.</p><p>If for some reason there was an issue (iptables to ebpf node communication for example), we could more easily drain the new nodes, we weren’t loaded yet.</p><p>Alternatively, if everything went fine, we could drain misconfigured nodes, fix their configuration and reboot them.</p><p>In order to do this, we started by applying the <em>CiliumNodeConfig</em> CRD on our clusters. Then, we installed the new nodes directly with the label io.cilium.enable-ebpf: “true”. The <em>Pods</em> that were started on those nodes never used iptables.</p><pre>kubectl apply -f manifests/cilium-fix.yaml</pre><p>We then validated that all nodes could communicate with each other. We saw no difference, regardless of their configuration (eBPF vs iptables), so we moved on and applied the fix on existing nodes.</p><h3>A happy ending</h3><p>There was absolutely no impact for our applications, nor our end users, thanks to the cilium feature that allowed us to apply the configuration on some nodes.</p><p>When all misconfigured nodes were drained and rebooted, it was time to check if the performance had improved.</p><p>And it had: we observed that the 2 CPUs per cilium agent consumption (with a lot of variations, up to 5-CPU peaks) dropped to only 200 millicores on average (half a CPU at peak). RAM consumption also decreased from 3 to 5 GB to less than 500 MB per instance, which is nice too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Htx1tR_ZE_9-GQy8" /><figcaption>sum of cilium CPU consumption with iptables for <strong><em>25 nodes</em></strong><em>: 50–60 cores</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TrPQUR19SHE2eP8P" /><figcaption>sum of cilium CPU consumption with eBPF for <strong><em>50 nodes</em></strong><em>: 5–15 cores</em></figcaption></figure><p>While we were expecting significant improvements in resource consumption, we certainly weren’t hoping for 10 times less.</p><p>Again, it’s not really surprising since iptables load increases exponentially with the size of the cluster. If we had not detected it soon enough, this configuration mistake could have crippled our production when traffic hit it.</p><p>Fortunately for us, cilium features and the fact we could work with our cluster in a mixed state allowed us to correct the configuration in a smooth and seamless fashion. Thanks cilium!</p><h3>Additional information</h3><ul><li><a href="https://isovalent.com/blog/post/tutorial-migrating-to-cilium-part-1/">Isovalent blog — Tutorial: How to Migrate to Cilium (Part 1)</a></li><li><a href="https://docs.cilium.io/en/stable/contributing/testing/e2e/">cilium documentation — End-To-End Connectivity Testing</a></li><li><a href="https://docs.cilium.io/en/stable/operations/troubleshooting/">cilium documentation — Troubleshooting</a></li><li><a href="https://docs.cilium.io/en/stable/network/concepts/routing/#native-routing">cilium documentation — Native routing</a></li><li><a href="https://cilium.io/blog/2021/05/11/cni-benchmark/">cilium blog — Understanding Cilium Network Performance</a></li><li><a href="https://ebpf.io/">ebpf website</a></li></ul><p>To get your current cilium configuration, you can also check the cilium <em>ConfigMap</em></p><pre>kubectl -n cilium get cm cilium-config -o yaml<br><br>apiVersion: v1<br>data:<br>  agent-not-ready-taint-key: node.cilium.io/agent-not-ready<br>  arping-refresh-period: 30s<br>  auto-direct-node-routes: &quot;false&quot;<br>  bpf-lb-external-clusterip: &quot;false&quot;<br>  bpf-lb-map-max: &quot;65536&quot;<br>  bpf-lb-sock: &quot;false&quot;<br>  bpf-map-dynamic-size-ratio: &quot;0.0025&quot;<br>  bpf-policy-map-max: &quot;16384&quot;<br>[...]</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84a035af1cd6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deezer-engineering/migrating-cilium-from-legacy-iptables-routing-to-native-ebpf-routing-in-production-84a035af1cd6">Migrating Cilium from Legacy iptables Routing to Native eBPF Routing in Production</a> was originally published in <a href="https://medium.com/deezer-engineering">Deezer I/O</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Back from DevoxxFR 2023]]></title>
            <link>https://medium.com/deezer-engineering/back-from-devoxxfr-2023-d469dae89402?source=rss-1df55d0db6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/d469dae89402</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[recaps]]></category>
            <category><![CDATA[dunning-kruger-effect]]></category>
            <category><![CDATA[octodns]]></category>
            <category><![CDATA[conference]]></category>
            <dc:creator><![CDATA[Denis GERMAIN]]></dc:creator>
            <pubDate>Wed, 10 May 2023 08:31:47 GMT</pubDate>
            <atom:updated>2023-05-10T08:31:47.282Z</atom:updated>
            <content:encoded><![CDATA[<p><em>co-authored by </em><a href="https://twitter.com/ju_hnny5"><em>Julien Briault</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*77mVfSR8-Tcuy844JyCs3Q.jpeg" /></figure><h3>What is DevoxxFR?</h3><p><a href="https://www.devoxx.fr/">Devoxx France</a> returned for its 11th-year, following the 10-year anniversary of the event (the first of the post-Covid era)! We went last year (<a href="https://deezer.io/back-from-devoxx-france-2022-what-i-loved-what-i-learned-what-i-loathed-948f0750cd14">you can read our post about it here</a>), and we were looking forward to attending again.</p><p>For those who don’t know about this conference, it’s one of the biggest and most anticipated developer conferences in France. It is a 3-day event that takes place every year at the Palais des Congrès in Paris, with nearly 3,000 attendees (which doesn’t include organizers, speakers, sponsors, etc.) and 8 to 12 parallel tracks.</p><h3>The talks we really enjoyed</h3><p>Let’s cut to the chase. The main objective of these conferences is to learn new things! Obviously, with more than 200 talks, we had to make tough choices on which events to attend.And since every speaker has something interesting to share, we will only highlight a few talks here.</p><p>Hubert Sablonnière presented “Retour Aux Sources : Le cache HTTP” (Back to Basics: The HTTP cache), an interesting talk about HTTP caches. You may not know that the modern web has more cache layers than one might expect. There were a lot of examples which made the talk easy to understand.</p><p>We strongly encourage you to <a href="https://www.youtube.com/watch?v=AJT5EJP_QmQ">watch the replay</a>.</p><figure><img alt="Hubert showing us the cache layers of modern HTTP web" src="https://cdn-images-1.medium.com/max/1024/1*QsVdOhwsEHONPR9wVCHS3A.jpeg" /></figure><p>Cyril Beslay discussed software architecture debt in “Gestion de la Dette d’Architecture dans un Contexte d’Hyper Croissance” (Architectural Debt Management in a Hyper-Growth Context). It provided a very good summary of the issues caused by technical debt and, more importantly, <em>software</em> <em>architecture debt</em>.</p><p>Cyril used specific examples, and here are the three critical pieces of advice we identified:</p><ul><li>Give technical teams a proper time to learn about business and functional requirements</li><li>Record everything (to know why some technical decisions have been made in the past)</li><li>Don’t ever come to a point where the only solution is to trash everything</li></ul><p>You can <a href="https://www.youtube.com/watch?v=F30CJnmzI8Y">watch the replay here</a>.</p><figure><img alt="Photo of a full “Amphi Bleu” for Cyril talk" src="https://cdn-images-1.medium.com/max/1024/1*ROnk030DnxGdb3N6xLCOtg.jpeg" /></figure><p><a href="https://fr.wikipedia.org/wiki/Thomas_C._Durand">Thomas Durand (aka Acermendax on Twitter)</a>, a well-known French <a href="https://www.youtube.com/@TroncheEnBiais">YouTuber</a>, explained how ordinary people can begin to believe in things that should seem absurd to the average person (e.g., fake news, conspiracy theories, etc.) by examining cognitive biases.</p><p>He tried to define the words “intelligence” and “rationality” (spoiler alert, it is complicated…) and revealed to us that the <a href="https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect">“Dunning Kruger effect” </a>diagram is nowhere near as impressive as what we can regularly observe on the Internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*ia5lxzMO7Y3C62ytIDEk8g.png" /><figcaption>A diagram of Dunning Kruger’s research versus a viral diagram spreading on social networks.</figcaption></figure><p>In the last part of his talk, he detailed the mechanisms explaining why fake stories are more popular and viral than the truth.</p><p>You can <a href="https://youtu.be/v_nAokE2D4U">watch the replay here</a>.</p><figure><img alt="Photo of a packed room before Thomas Durand talk" src="https://cdn-images-1.medium.com/max/601/1*e6qOdGctbzTuJb__ToNDhw.png" /></figure><h3>Our presentations at DevoxxFR</h3><p>This year, we weren’t just attending DevoxxFR as spectators. We were honored to be selected by the CFP (call for presentation) committee for two talks.</p><p>Denis Germain presented “Démystifions les Composants Internes de Kubernetes” (Demystifying the Kubernetes Internal Components), a talk combining technical explanations about Kubernetes internals and live demos. The <a href="https://www.youtube.com/watch?v=OCMNA0dSAzc">replay of the talk is available on YouTube</a>. You can also find the <a href="https://blog.zwindler.fr/talks/2023-demystifions-kubernetes/index.html">slides for the presentation here</a> and <a href="https://github.com/zwindler/demystifions-kubernetes">the code here</a>.</p><figure><img alt="Denis on stage of amphi bleu before his talk" src="https://cdn-images-1.medium.com/max/603/1*cwyzU5ERib6IFhVFWXiQzg.png" /></figure><p>Julien Briault presented a tool called OctoDNS in his talk “Infra : Donnez de l’Autonomie à vos Développeurs avec OctoDNS” (Infra: Empower your Developers with OctoDNS). In a live coding session; he demonstrated how easy it is to manage DNS records from multiple DNS providers in a single repository and with a common DSL (Domain Specific Language).</p><p><a href="https://youtu.be/aZvsplOAd7c">The replay of the talk is available on YouTube</a>. <a href="https://github.com/juhnny5/DevoxxFR2023-OctoDNS">You can also find the slides and the code here.</a></p><figure><img alt="Julien before his talk" src="https://cdn-images-1.medium.com/max/604/1*1Dwxc6eCuMH1cCzWyvHhow.png" /></figure><h3>DevoxxFR is more than just talks</h3><p>Networking is a big part of the conference. Many sponsors were brought in, and we each took the time to chat with some of them.</p><p>We had very interesting discussions with people from <a href="https://www.clever-cloud.com/fr/">Clever Cloud</a>, <a href="https://www.adeo.com/en/">Adeo</a> (Leroy Merlin), <a href="https://www.adelean.com/">Adelean</a>, <a href="https://engineering-and-services.michelin.com/">Michelin Engineering</a>, and <a href="https://www.ovhcloud.com/en/">OVHcloud</a> <em>about the common challenges we face</em>, from search engines to managing our infrastructure and hosting public web services. The life of Site Reliability Engineers is not a long quiet river!</p><p>As speakers, we were also invited to the speaker dinner that took place at the Jardin d’Acclimatation in Paris. It was a privileged time to talk with the other speakers and — maybe more importantly — the organizers, as it was probably their only free time in 72 hours 😅. In particular, it was nice to see things from the other side of the fence and learn more about their organizational challenges, like managing to print tickets for 3,800 people or selecting talks for the conference (in the end, 200 sessions were scheduled out of 1,100 submissions!).</p><figure><img alt="Photo from the Jardin d’acclimatation" src="https://cdn-images-1.medium.com/max/1024/1*FWKJSsbWZmTXWiZsJq-V3g.jpeg" /></figure><p>Finally, we took the opportunity to meet other attendees during the Meet and Greet (a free event that took place on Thursday evening). It was a very laid-back way to meet people and talk in real life with friends we only knew from the Internet.</p><figure><img alt="Tweet from Guillaume Grillat saying “🇫🇷La Frenchtech à vot’ service🐓@BlaBlaCarEng x @DeezerDevs x @leboncoinEng #Devoxxfr2023” with a selfie" src="https://cdn-images-1.medium.com/max/581/1*hEZMciwcixpMpBM1yZncTQ.png" /><figcaption><em>Source: </em><a href="https://twitter.com/grillatg/status/1646572207580086283?s=20"><em>https://twitter.com/grillatg/status/1646572207580086283?s=20</em></a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*fhSY5NzosmJZXTWrtFSHIg.png" /><figcaption>At the end of the Meet and Greet, the JAM allowed participants to play instruments freely.</figcaption></figure><h3>See you next year!</h3><p>This year, the DevoxxFR organizers really outdid themselves. The whole event went without a hitch (from our perspective, at least 😉), and the talks were all excellent, and diverse. You can find all the videos from this conference on the<a href="https://www.youtube.com/channel/UCsVPQfo5RZErDL41LoWvk0A"> Devoxx YouTube channel</a>.</p><p>We look forward to returning next year, especially since DevoxxFR have announced they are evolving and will welcome 1,300 more attendees on two floors).</p><p>So, see you there on April 17th-19th, 2024!</p><p><em>Do you wish to learn and share with our community of developers, and make the Deezer experience a better one? Check our </em><a href="https://www.deezerjobs.com/en/jobs/"><em>open positions</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d469dae89402" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deezer-engineering/back-from-devoxxfr-2023-d469dae89402">Back from DevoxxFR 2023</a> was originally published in <a href="https://medium.com/deezer-engineering">Deezer I/O</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Back from BDX I/O 2022]]></title>
            <link>https://medium.com/deezer-engineering/back-from-bdx-i-o-2022-71151e58960b?source=rss-1df55d0db6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/71151e58960b</guid>
            <category><![CDATA[conference]]></category>
            <category><![CDATA[imposter-syndrome]]></category>
            <category><![CDATA[recaps]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[web-accessibility]]></category>
            <dc:creator><![CDATA[Denis GERMAIN]]></dc:creator>
            <pubDate>Wed, 15 Feb 2023 09:31:49 GMT</pubDate>
            <atom:updated>2023-02-15T10:59:00.737Z</atom:updated>
            <content:encoded><![CDATA[<p><em>co-authored by Fayçal Bennegadi, Alexis Fala, Christian Alonso Chavez Ley</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XnvADQFucmDWwkpS" /></figure><p>On December 2, 2022, Deezer sponsored the 7th edition of BDX I/O.</p><p>If you haven’t heard of it, it is probably <strong>the</strong> biggest developers’ conference and is held in Bordeaux, France, with 1,000 attendees and 4 to 5 parallel tracks at the Bordeaux Congress Centre.</p><figure><img alt="The main stage during the keynote, showing stats about speakers" src="https://cdn-images-1.medium.com/max/1024/0*Mm4AsufFyqZNmeSi" /></figure><p>This was also the first post-COVID BDX I/O conference after nearly three years of absence (2019), and we were eager to be a part of it.</p><p>Ten of us from Deezer attended, so we have tried to sum up the three talks that impressed us the most.</p><h3>Sponsoring</h3><p>Given that we were one of the event’s sponsors, we had the chance to have our own stand in the dedicated hall.</p><p>In order to make the classic “swag grab” a little more engaging and fun, we set up a simple rule at the beginning of the day: if you wanted a goodie, you had to sing a song at our booth following the lyrics on screen. This rule even applied (some would say specifically) to our colleagues.</p><p>Singing aside, we also spoke with everyone who came to the booth about our current open positions, our work culture and received feedback on our applications. Overall, we had quite a fun day at the booth!</p><h3>Syndrome d’imposture, l’arbre qui cachait la forêt</h3><p><strong><em>Imposter syndrome — you can’t see the forest because of the trees</em></strong></p><p>Unlike the title suggests, Boris Schapira does not talk much about imposter syndrome itself, nor how to fight it. Instead, he suggests several other talks on the subject, like those by Aurélie Vache, Abeba Ngwe, Maeva Nguyen, and Marine Gandy.</p><figure><img alt="Boris Schapira on stage before its talk" src="https://cdn-images-1.medium.com/max/1024/0*OrBqGVnanKj5IjAD" /></figure><p>The main postulate of this session is that the number of people experiencing imposter syndrome is so prevalent in our jobs that we need to stop seeing it as an individual problem but rather a systemic problem.</p><p>Many of the slides had a strong impact on us:</p><ul><li>We involuntarily put ourselves in competition with each other to master increasingly complex solutions, often on our time;</li><li>We tend to actively participate in social reproduction within companies through practices such as co-optation, recruitment based on “good feelings” during interviews, and the emphasis of “company values”’;</li><li>And many other things we don’t even notice anymore…</li></ul><p>Watch the talk here: <a href="https://www.youtube.com/watch?v=vS2WKdWRNqo">BDX I/O 2022 — Syndrome de l’Imposteur, l’arbre qui cache la forêt — Boris Schapira</a></p><h3>Les sciences cognitives au service de l’accessibilité web</h3><p><strong><em>Cognitive sciences for web accessibility</em></strong></p><p>Why has web accessibility become a significant issue for online platforms?</p><p>The numbers speak for themselves: 15% of people have disabilities that impact their online experience; 80% of them have non-visible disabilities, which can be difficult to detect for someone who is not affected by such impairments. As a result, 70% of websites are considered “not accessible.”</p><p>In this talk, Margot Bouhon, Ph.D. in cognitive science, seeks to raise awareness and gives excellent advice to easily improve the accessibility of our web platforms in order to make the user experience easier, and therefore, more inclusive.</p><p>Some of the existing solutions:</p><ul><li>Conducting an audit to improve site accessibility — <em>Silktide </em>offers such services for instance.</li><li>For hearing impairments, in particular, making the text available — for example, <em>AVA</em> provides AI-generated subtitles.</li><li>For neurodevelopmental disorders (like dyslexia), visual impairments or reading difficulties, <em>readapt.ai </em>developed an assistive technology that allows people with these conditions to create a personalized reading profile for digital content based on their unique needs. Thanks to this tool, people can change the font, the spacing between characters, and even the font size.</li></ul><p>Watch the talk here: <a href="https://www.youtube.com/watch?v=le3Yl_lCJ3M">BDX I/O 2022 — Les sciences cognitives au service de l’accessibilité web. — Margot Bouhon</a></p><h3>Alice au pays d’OpenTelemetry</h3><p><strong><em>Alice in OpenTelemetryLand</em></strong></p><p>OpenTelemetry (or OTel for the cool kids) is a collection of tools, APIs and SDKs, which aims at standardizing how we instrument, generate, collect, and export telemetry data (i.e. metrics, logs and traces).</p><p>This talk was one person’s journey, Jérôme Tama from OnePoint, setting it up on a real project, supplemented with gotcha’s and “buyer beware” moments, as well as an overview of the concepts and the requirements. All this was told using Alice in Wonderland as the basis for the presentation.</p><p>All in all very fun and informative.</p><p>Watch the talk here: <a href="https://www.youtube.com/watch?v=ArjNjrsnW6A">BDX I/O 2022 — Alice au pays d’Opentelemetry — Jérôme Tama</a></p><h3>Summing up</h3><p>In conclusion, the 7th edition of BDX I/O was a successful and engaging conference for attendees and sponsors alike.</p><p>We really enjoyed showcasing our products to the developers in attendance.</p><p>We truly appreciated the diversity of the talks, which were not only “technical” in nature and also the varied panel of speakers at this edition, with 40% of them identifying as women — which is not 50% yet but is still better than most tech conferences.</p><p>We really hope to come back again next year — perhaps with some speakers from Deezer this time!</p><figure><img alt="The main sponsors room with a lot of attendants, during lunch" src="https://cdn-images-1.medium.com/max/1024/0*Tmcll4aVZGiJNEOb" /></figure><p><em>Do you wish to learn and share with our community of developers, and make the Deezer experience a better one? Check our </em><a href="https://www.deezerjobs.com/jobs-2/"><em>open positions</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71151e58960b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deezer-engineering/back-from-bdx-i-o-2022-71151e58960b">Back from BDX I/O 2022</a> was originally published in <a href="https://medium.com/deezer-engineering">Deezer I/O</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Back from Devoxx France 2022: what I loved, what I learned, what I loathed]]></title>
            <link>https://medium.com/deezer-engineering/back-from-devoxx-france-2022-what-i-loved-what-i-learned-what-i-loathed-948f0750cd14?source=rss-1df55d0db6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/948f0750cd14</guid>
            <category><![CDATA[recaps]]></category>
            <category><![CDATA[devoxx]]></category>
            <category><![CDATA[deezer]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[conference]]></category>
            <dc:creator><![CDATA[Denis GERMAIN]]></dc:creator>
            <pubDate>Thu, 05 May 2022 08:44:54 GMT</pubDate>
            <atom:updated>2022-05-05T08:49:11.537Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Kz06ccMYxbBJSh4I.jpg" /></figure><p>On April 20 2022, I attended the <a href="https://www.devoxx.fr/">Devoxx France conference</a> at Palais des Congrès de Paris.</p><p>If you haven’t heard of it, it is probably <strong>the</strong> most anticipated developer conference in France. It is a 3 day event that takes place every year, with more than 3,000 attendees (and that doesn’t include organizers, speakers, sponsors, etc.), and 8 to 12 parallel tracks!</p><p>This year was a very special one for two reasons:</p><ul><li>after two years of COVID, the 2022 conference was back to full attendance</li><li>it was the 10th year edition of Devoxx France</li></ul><p>With this in mind, there was a lot of anticipation about the event and I was very grateful that one of my talks had been chosen by the CFP (aka Call For Paper) board.</p><p>It was also my first time attending this conference so even if I had been to other dev conferences before, I didn’t exactly know what to expect.</p><p>So if you want to see Devoxx through my newbie eyes and read about what I loved, learned and loathed, then this post is meant for you. Buckle up!</p><p>Note: I’m only sharing my experience as an attendee in this post, but if you want to know more about my talk <a href="https://www.youtube.com/watch?v=RbqHDpoaqR8">Ciel ! Mon Kubernetes mine des bitcoins !, you can watch it here</a>.</p><h3>What I loved</h3><p>Obviously at a conference like this with more than 200 talks, you can’t see everything. Devoxx talks are very diverse and (I think) you should use this opportunity to attend talks on topics you wouldn’t research otherwise.</p><p>Being a professional in infrastructure/SRE topics, I tried to attend talks that weren’t just about “cloud”, “devops” or simply “dev”. And there were many times I was glad I did.</p><p>I really liked the talk <a href="https://www.youtube.com/watch?v=Vo_YWAE3Pxo">Rendez l’agilité aux développeur(se)s !</a> by Fanny Klauk, which was sort of a choose-your-own-adventure experience featuring an agile team. It was super easy to relate to and quite funny, and it made all of us reflect on the way we work in our own agile teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iMKrqNEN1N48kTEI.jpeg" /></figure><p>I loved the energy of Quentin Adam and Pierre Beyssac in <a href="https://www.youtube.com/watch?v=gxWwxqIrCa8">Comprendre les enjeux de consommation de ressource et d’énergie dans le secteur numérique</a>. This talk aims to put emotions aside and stay rational on the (real) climatic issues at bay by looking at only the facts and listening to what science tells us.</p><p>Finally, I was really moved by the keynote of <a href="https://www.youtube.com/watch?v=fJwDbP3CB-0">LesBonsclics, une plateforme pédagogique au service du 1er réseau européen d’aidants numériques</a>. The goal of this project led by <a href="https://wetechcare.org/">wetechcare</a> is to bring more digital inclusiveness — a subject I believe is very important in the digital age.</p><p>Extra bonus points awarded to Cécile Morange who, at the age of 21, owns a mini data center (including her very own <a href="https://fr.wikipedia.org/wiki/Autonomous_System">Autonomous System Number</a>) and explained to a big audience of senior developers <a href="https://www.youtube.com/watch?v=CskRoGyUr7w">what “the cloud” is and why maintaining infrastructure is not so easy</a> with a lot of pedagogy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rPtXLYjgVsibwT-C.jpeg" /></figure><p>Aside from conferences, there were lots of exhibitions and sponsors. One of them stood out: Technosaures, a retro computing magazine (1970–1990). Many ancient machines were exhibited and François Tonic was very didactic and passionate about his collection.</p><h3>What I learned</h3><p>A reason why I came to Devoxx was to learn a few more tricks from my peers, so I was really happy to join the talk <a href="https://www.youtube.com/watch?v=Nyga5cdvgbo">Pourquoi DevOps ne tient pas ses promesses ?</a> by Gérôme Egron and Guillaume Mathieu. It provided a lot of insight into why sometimes the objective of Devops (i.e. closing the gap between product and operational teams) is not met. As a SRE CoP (Community of Practice) leader, I too advocate most of the advice they gave but from the Ops side. Seeing it from the other side of the mirror — the Dev side — was interesting.</p><p>Doctolib’s David Gageot and Bertrand Paquet spoke about big managed cloud databases and the scalability issues they faced during the COVID-19 crisis (<a href="https://www.youtube.com/watch?v=1PtPVpFWtKQ">Doctolib a besoin d’une base de données plus puissante. Ok, mais laquelle?</a>) I appreciated that they were transparent about the failures they experienced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZHzsMYClB0u2IJjH.jpeg" /></figure><p><a href="https://www.youtube.com/watch?v=oXHhOHa_5HE">Cybersécurité et générateurs de nombres aléatoires</a> by Mathis Hammel was a (nice) surprise. I hadn’t planned to go at first but it turned out to be a fun and instructive postmortem on a gigantic CVE (Common Vulnerabilities and Exposures) in the password generator of a world renowned antivirus company.<br>Of course, I couldn’t attend Devoxx and not see a few talks about Kubernetes (my day to day job). I enjoyed <a href="https://www.youtube.com/watch?v=iq-b1Z0415g">Qu’avons-nous appris après un an passé à développer des opérateurs Kubernetes ?</a> by Etienne Coutaud, who provided a return on experience about writing Kubernetes Operator to manage complex infrastructure in a fully automated way for a PaaS (Platform as a Service).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ix1dgfUnZDK6ZIB7" /></figure><h3>What I loathed</h3><p>To be honest, the word “loathe” is too strong and inaccurate in this case as I didn’t “hate” anything in particular at the conference. However there are definitely some things I wished had not happened.</p><p>The organizers of the conference did their very best to avoid it. Antibacterial gel was available and people were advised to wear a mask everywhere (though it was not mandatory according to French regulations).</p><p>But the truth is that COVID-19 is still here.</p><p>A few speakers couldn’t make it to the conference because they tested positive. Many attendees did a self test before coming… but a few people, including friends of mine, caught COVID-19 at the conference. Luckily, as far as I know, none showed heavy symptoms, but it’s a reminder of how fragile the sanitary situation still is.</p><p>We will have to learn to live with it, somehow.</p><h3>Conclusion</h3><p>I thoroughly enjoyed many other talks at this conference but I couldn’t possibly list them all in a single blog post. You can find all the videos of this edition on the <a href="https://www.youtube.com/channel/UCsVPQfo5RZErDL41LoWvk0A">Devoxx Youtube channel</a>.</p><p>I saw lots of friends and ex-colleagues, and spoke with passionate professionals coming from everywhere in France (and beyond).</p><p>The whole organization team was incredible and truly professional — and made up entirely of volunteers! I can’t thank them enough, both for believing in my talk and for organizing such an awesome event overall.</p><p>Devoxx <strong>really is</strong> an excellent conference and I can’t wait to go back next year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uK1c9UMkYzEmmrzdjVAE8A.jpeg" /></figure><p><em>Do you wish to learn and share with our community of developers, and make the Deezer experience a better one? Check our </em><a href="https://www.deezerjobs.com/jobs-2/"><em>open positions</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=948f0750cd14" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deezer-engineering/back-from-devoxx-france-2022-what-i-loved-what-i-learned-what-i-loathed-948f0750cd14">Back from Devoxx France 2022: what I loved, what I learned, what I loathed</a> was originally published in <a href="https://medium.com/deezer-engineering">Deezer I/O</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Back from Kubecon & CloudnativeCon Europe 2021: key learnings and takeaways]]></title>
            <link>https://medium.com/deezer-engineering/back-from-kubecon-cloudnativecon-europe-2021-key-learnings-and-takeaways-9553c1d7a0d0?source=rss-1df55d0db6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/9553c1d7a0d0</guid>
            <category><![CDATA[kubecon]]></category>
            <category><![CDATA[cloudnativecon]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Denis GERMAIN]]></dc:creator>
            <pubDate>Tue, 15 Jun 2021 12:43:51 GMT</pubDate>
            <atom:updated>2021-06-15T13:51:23.760Z</atom:updated>
            <content:encoded><![CDATA[<h3>Back from KubeCon &amp; CloudNativeCon Europe 2021: key learnings and takeaways</h3><p><em>Co-authored by </em><a href="https://medium.com/u/74f35a8eafa2"><em>Christian Alonso Chavez Ley</em></a></p><p>At Deezer, we think that taking part in the community is of utmost importance. Contributing to open source projects is the best way to do so, but participating in community events should not be overlooked. And when you work on Kubernetes platforms like we do, what’s better than going to the KubeCon + CloudNativeCon conference?</p><p>For those who aren’t familiar with KubeCon, it’s the main event organised <a href="https://www.cncf.io/">by the CNCF</a> (i.e. the foundation that hosts a lot of the cloud native open source projects). The conference spans over a whole week, with the first day being reserved for co-located events (like the <a href="https://promcon.io/">PromCon</a>) and sponsored events. Each year, the conference unites around 3,000 to 4,500 open source and cloud native enthusiasts.</p><p>This year, three of us participated in the KubeCon Europe 2021 (from May 4–7, we didn’t attend any co-located events). Like last year, this KubeCon was virtual due to the sanitary situation.</p><p>Some of us had already attended virtual or IRL KubeCon, and one of us had never gone. We were all very excited because, from experience, KubeCon is always a place where you learn a lot in a very short time. Everything was ready:</p><ul><li><a href="https://kccnceu2021.sched.com/">Individual sched.com KubeCon schedules</a> filled out and shared with colleagues? Check!</li><li>Private Slack and Mumble channels to exchange views between sessions? Check!</li><li>Shared, collaborative notes? Check!</li><li>Mountain of coffee? Check!</li><li>“Do not disturb” signs everywhere and empty work agendas? Almost checked, we’ll come back to this later.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0wGpsqt1eZ3qc-SNUXFWuw.png" /></figure><p>There is obviously too much to talk about to fit in one article, so we’ve written a best-of compilation of what we learned there, and identified some key takeaways — should you want to participate in such an event in the future.</p><p>We feel this year’s conference was built around three main topics, which are CI/CD, Security and Networking. We will focus on them here.</p><p>Are you ready to dive in?</p><h3>CI/CD in Kubernetes</h3><p><a href="https://youtu.be/_DgCc4-BLW8">Putting Chaos Into Continuous Delivery to Increase App… Juergen Etzlstorfer &amp; Karthik Satchitanand</a></p><p>The continuing maturity of the ecosystem means that we are moving from modeling pipelines (with tools like <a href="https://argoproj.github.io/argo-cd/">argocd</a> and <a href="https://tekton.dev/">tekton</a>) to resilience hardening with techniques like chaos testing.</p><p>This all goes back to the seminal <a href="https://dzone.com/articles/understanding-the-8-fallacies-of-distributed-syste">8 Fallacies of Distributed Systems</a> and the fact that applications running on K8s are, by their very nature, distributed systems, even if they were not conceived as such. So the question we now face is: “how can we easily test these assumptions in our platform?”</p><p><a href="https://litmuschaos.io/">Litmus Chaos</a> is a K8s native, declarative oriented, experiment based (whew that’s a lot of adjectives) tool that allows the community to share chaos tests in a portable manner, which can then be executed during a stage in a CD pipeline or against production systems (for those who feel like living dangerously).</p><p>In order to illustrate Service-Level Objective (SLO)-based quality gates with chaos tests, the speakers used Litmus Chaos through another tool called <a href="https://keptn.sh/">Keptn</a>, which does it natively, by using Prometheus and other types of datasources. Nevertheless, it shouldn’t be too hard to plug them into your current CD pipelines, whatever they may be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-VHnXH7w4dbrYNQs" /></figure><p>We particularly like and wholeheartedly agree with their key takeaways:</p><ol><li>Establish a process of continuously evaluating resiliency</li><li>Do chaos testing in addition to performance tests</li><li>Evaluate based on Service-Level Objectives (SLOs)</li></ol><p>Those are good rules-of-thumb to follow independently of any tool you are using.</p><h3>Security in Kubernetes</h3><p>Another great topic at this year’s conference was security in Kubernetes. Even though securing it has always been a concern, Kubernetes is gaining a lot of attention (especially from hackers), and the CNCF ecosystem is responding accordingly.</p><p><a href="https://landscape.cncf.io/card-mode?category=security-compliance&amp;grouping=category">A lot of projects have been included in the CNCF landscape on this topic</a> and there were of course talks about those. But aside from these very specific software related talks, there were many (very good) beginner talks, including:</p><ul><li><a href="https://www.youtube.com/watch?v=_Dzc07aBQHI">What Do You Mean K8s Doesn’t Have Users? How Do I Manage User Access</a></li><li><a href="https://www.youtube.com/watch?v=mLsCm9GVIQg">Hacking into Kubernetes Security for Beginners</a></li><li><a href="https://www.youtube.com/watch?v=0_s6zkyRpME">Get In Containerds, We’re Going Securing: Kubernetes SIG Security is Here!</a></li></ul><p>From those talks, you can <em>feel</em> that the audience is a bit late on security and that’s a bit worrying.</p><p>One of the best talks we’ve seen on security this year was <a href="https://www.youtube.com/watch?v=bohnofE_dvw">Uncovering a Sophisticated Kubernetes Attack in Real-Time</a>, by Jed Salazar &amp; Natália Réka Ivánkó. After brushing a very (very) quick picture of the security best practices everyone should (woops 😨) already have in place in a Kubernetes cluster, they demonstrated how an attack could take place in a “supposedly safe” cluster and how to track malicious actions with <a href="https://cilium.io/">Cilium</a> (using eBPF programs).</p><p>The main idea there was “trust, but verify”, which is just applying DevOps logic to security.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jzezYDfyeO6Hsn2c" /></figure><p>Another great talk to better understand the magic behind <a href="https://falco.org/">Falco</a>, Cilium and all related runtime security tools in Kubernetes is the talk on eBPF by Quentin Monnet: <a href="https://www.youtube.com/watch?v=cABk-6Sdb20">eBPF on the Rise — Getting Started</a>. If you intend to work with any tools using eBPF (and there are a lot of them now), you really should watch this session.</p><h3>Networking in Kubernetes</h3><p><a href="https://youtu.be/7qUDyQQ52Uk">How to Break your Kubernetes Cluster with Networking — Thomas Graf, Isovalent</a> was one of the most fun presentations we saw during KubeCon.</p><p>Thomas Graf is a Cilium maintainer so it could have been incredibly detailed (which may have lost a part of the audience) but instead it was structured as an introduction, with a very nice “avoid common pitfalls” style.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7QyCCNqIEwx904j9" /></figure><p>Because let’s face it: for all the benefits that Kubernetes brings to development teams (and we firmly believe it does bring a lot!), operating a K8s cluster is no joke. Even when using a managed service (like GKE, EKS, AKS, etc.), you still need to understand how components work together and react to each other, particularly when it comes to networking.</p><p>To use another old reference: Kubernetes is an example of a <a href="https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/">Leaky Abstraction</a>, and ignoring or misunderstanding the abstractions and your chosen implementation can come back to haunt you.</p><p>Some of the most notable examples he talked about were:</p><ul><li>The <a href="https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html">ndots default</a> (basically the number of lookups DNS does before resolving) is set to 5, which may cause you to DDoS your DNS server if you have a lot of outgoing traffic.</li><li>When migrating to a “deny all egress traffic by default” approach while using <a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/#behavior-of-to-and-from-selectors">Network Policies</a>, you should not forget to whitelist the dns pods (them being coredns in kube-system by default) or else your application won’t find the pods you’re actually interested in.</li><li>CRDs are stored in etcd and often watched so they are frequently being transmitted between the api server and the nodes. At scale (imagine thousands of nodes), this potentially means gigabytes of network traffic that can DDoS an unsuspecting api server.</li><li>Take a look at the presentation for much more ;)</li></ul><p>Even though many of these cases are well documented, most of us operating Kubernetes clusters in production have been bitten by them in one way or another.</p><p>This is why we like the message in one of the last slides so much:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3fTqGV_EFTZTi2Vx" /></figure><p>Words to live by :)</p><h3>Takeaways</h3><p>Once again, KubeCon + CloudNativeCon proved to be the place to be if you work with Kubernetes or Cloud Native tools. This year’s sessions were clearly focused on CI/CD, observability, networking, as well as security.</p><p>If we had to describe this KubeCon with only one word, it would undoubtedly be <a href="https://ebpf.io/"><strong>eBPF</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qc5aSbiRULRaPWRLHpZPmA.png" /></figure><p>Either in emerging or maturing projects in the CNCF ecosystem, eBPF was everywhere! From Intrusion Detection Systems to CNI plugins, there were a lot of presentations about it. We think that it’s a signal that the technology (which has been around for some time now) is mature. You should really take the time to get familiar with it if that’s not already the case.</p><p>From a less technical point, we also want to share that attending a virtual conference is hard, but not in the way we thought. After more than a year of forced full-remote work, maintaining attention to conferences for 3 days straight was not so challenging after all (no “Zoom-fatigue” syndrome felt).</p><p>But (there’s always a catch) the downside of being on your computer at home, like a “normal day at work”, is the temptation to work in parallel to attending the conference. Our advice is to book your agenda for the whole 3 days (or 5 if you do the whole gig) and more importantly, to stick to it!</p><p>Another takeaway is that KubeCon is really tightly packed. This year, there were no breaks from 10am to 5pm besides the 15 minutes between each talk (no lunch break). That’s rough and we hope that they’ll be back next time.</p><p>Last but not least, one of the advantages of conferences is the possibility to have impromptu meetings with strangers. This is a great way to talk about problems you are facing and discuss solutions other people have already found for them, in ways you may not have imagined otherwise. Even though there are open channels on the CNCF Slack, we found that having “hallway conversations” is way harder in remote conferences. And that’s too bad.</p><p>Let’s hope that we can safely meet again soon!</p><p><em>Do you wish to learn and share with our community of developers, and make the Deezer experience a better one? Check our </em><a href="https://www.deezerjobs.com/jobs-2/"><em>open positions</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9553c1d7a0d0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deezer-engineering/back-from-kubecon-cloudnativecon-europe-2021-key-learnings-and-takeaways-9553c1d7a0d0">Back from Kubecon &amp; CloudnativeCon Europe 2021: key learnings and takeaways</a> was originally published in <a href="https://medium.com/deezer-engineering">Deezer I/O</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Updating Kubernetes CA certificates the hard way]]></title>
            <link>https://medium.com/deezer-engineering/updating-kubernetes-ca-certificates-the-hard-way-f9518108791d?source=rss-1df55d0db6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9518108791d</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[certificate]]></category>
            <category><![CDATA[hashicorp-vault]]></category>
            <category><![CDATA[deezer]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Denis GERMAIN]]></dc:creator>
            <pubDate>Thu, 11 Feb 2021 15:12:03 GMT</pubDate>
            <atom:updated>2021-02-11T15:12:03.429Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, we will share our experience with a tricky situation we found ourselves in a few months ago. It involves Kubernetes clusters, CA certificates, Hashicorp Vault, and transparent operations in production for millions of users.</p><p>So buckle up; it might be a bumpy ride!</p><h3>The hard way yes, but automated!</h3><p>At Deezer, Kubernetes has been deployed and used in production for more than three years. Today, there are tons of different ways to deploy Kubernetes. Each has its advantages, depending on your cluster topology and needs (cloud, baremetal, edge, etc.).</p><p>You may have guessed it: this article’s title refers to the famous <a href="https://github.com/kelseyhightower/kubernetes-the-hard-way">“Kubernetes the hard way”</a> written by Kelsey Hightower.</p><p>That’s because back in 2017, our clusters have been deployed using Ansible playbooks, written by hand from the documentation. And this looks a bit like automating “Kubernetes the hard way” with Ansible 😉.</p><p>Though you would (and should) <em>not</em> do it like this today, you have to put yourself in the context of 3+ years ago. At that time, in terms of deployment method, the options were more limited than they are today, especially for bare metal installations (for example, <a href="https://kubernetes.io/docs/reference/setup-tools/kubeadm/">kubeadm</a>, now GA, was still displaying big red warnings of being “NOT PRODUCTION READY”).</p><p><strong>It’s the final countdown</strong></p><p>Not all Deezer applications are running inside Kubernetes, but a significant enough portion of them do. Should our Kubernetes platform be <em>down</em>, Deezer end users would start noticing that something is <em>up</em> (see what I did here? 😉). And that’s where our story begins:</p><blockquote><em>The Certificate Authority (CA) of Kubernetes was about to expire in a few months, and with it, the whole certificate chain.</em></blockquote><p>To put more graphically, instead of reassuring green indicators everywhere, we had a lot of orange warnings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*LavMroGyFHY8yVbi3uXh4w.png" /></figure><p>Since all communications inside the Kubernetes cluster are encrypted and authenticated with this CA, letting it expire would be VERY bad. You would essentially lose all access to your cluster and the applications within 😱. If you search a little on the Internet, you may find horror stories from people who let their cluster’s CA expire (see at the end of the article).</p><p>So, we had to find a way to renew the certificate, without any interruption for end-users, before the expiration date.</p><h3>Why is it a problem?</h3><p>With the CA certificate involved with all internal Kubernetes components, renewing is not a trivial task. To make things worse, the official documentation is incomplete on this matter, and external documentation on this topic is relatively thin on the Internet.</p><p>There are multiple reasons for that. Many organizations use Kubernetes through a managed offer from their cloud provider, who hides the complexity of running it from them. In this case, providers worry about this for them (but do they really?).</p><p>Among the remaining Kubernetes users, most of those running on-prem clusters like us use tools like <em>kubespray</em> or <em>kubeadm</em>, which contain some sort of certificate renewal procedures but were not available at that time.</p><p>Then, on the very thin crowd that remains, regenerating CA by hand is not something that has to be done often. CAs are often generated with 3 to 10 years expiration dates (best practices dictate shorter periods). If you compare this to the relatively recent adoption of Kubernetes (a project initially released six years ago), you can assume that many CAs have yet to expire!</p><p><strong>How are we going to play this?</strong></p><p>We had to find out how to change Kubernetes CA on the fly with no end-user downtime. Luckily for everyone, if you plan it carefully, it should be possible for many workloads.</p><p>The first thing to know is that, even if the cluster is unresponsive while the certificates are being renewed, the applications in Kubernetes (<em>Pods</em>) that are already deployed and running will try to continue to run. However, new <em>Jobs</em> won’t be scheduled, crashing <em>Pods</em> or workloads on failed nodes won’t be restarted.</p><p>The second thing to know is that all <em>Pods</em> run with a security context given by a <em>ServiceAccount</em>. If you don’t specify a <em>ServiceAccount</em> in your application’s manifest, you’ll get the default one in the <em>Namespace</em>.</p><p>What’s really important is that the <em>Pod</em> is granted access to the cluster according to the identity it was given (the <em>ServiceAccount</em>), which is authenticated with a token. That token is generated using the <em>Kubernetes Controller Manager</em>. Once we renew the CA, all tokens are toast. You must regenerate them all and restart the <em>Pods</em>.</p><p>So, the goal is to restart what’s required — nothing more — in the right order, then, renew all the tokens. All this quickly enough so that no application or node fails in the meantime.</p><blockquote><em>Easy peasy</em></blockquote><p><strong>BUT!</strong></p><p>There are, of course, exceptions or else that would be too easy:</p><ul><li>Applications that need to communicate with the Kubernetes API will be disrupted until they are restarted completely (once they have their new token). This probably includes your monitoring (Prometheus), which scrapes your cluster’s API. You’ll fly blind for a few minutes.</li><li>Applications maintaining long network connections open (like websockets) will probably be cut at some point because the <em>IngressController</em> you are using most likely requires access to the API server (see the previous point). If you’ve implemented a proper retry mechanism, that may not be a big issue.</li><li>All applications will have to be restarted at some point. For apps with no replicas, a little downtime will be unavoidable. To work around this, always try to have multiple replicas for all your applications. If that’s technically impossible, restarting these applications may not be urgent, so that you may plan this later on.</li></ul><p>Now that you know everything, let’s do it 😊.</p><p><strong>Sooooo let’s renew a CA in Kubernetes with (nearly) no downtime</strong></p><p>Before doing anything, we need to be able to roll back everything if necessary. This means backups and tested procedures to restore them.</p><p>Backup every certificate you currently use in <em>/etc/kubernetes</em>, but also in <em>/var/lib/kubelet</em> if there are any, and those of <em>etcd</em>.</p><pre>CURDATE=`date +&quot;%y%m%d%H%M&quot;`<br>tar czf /tmp/pkibackup.${CURDATE}.tgz /var/lib/kubelet/pki/kubelet.* /etc/kubernetes</pre><p>You might also want to backup all the Kubernetes<em> ServiceAccount</em> tokens. These tokens are generated by the <em>Kubernetes Controller Manager</em> and are used by your pods to communicate with the API server (we will talk again about this).</p><pre>for namespace in $(kubectl get ns --no-headers | awk &#39;{print $1}&#39;); do<br>  for token in $(kubectl get secrets --namespace &quot;$namespace&quot; --field-selector type=kubernetes.io/service-account-token -o name); do<br>    kubectl get $token --namespace &quot;$namespace&quot; -o yaml &gt;&gt; /tmp/token_dump.${CURDATE}.yaml<br>  done<br>done</pre><p>Finally, dumping the whole <em>etcd</em> database might also be a good idea (if all else fails)…</p><pre>ETCDCTL_API=3 etcdctl --cacert=yourca.pem --cert=etcd.pem --key=etcd-key.pem --endpoints 127.0.0.1:2379 snapshot save /tmp/etcd.backup.$(date +&#39;\%Y\%m\%d_\%H\%M\%S&#39;)</pre><p><strong>Using Hashicorp Vault</strong></p><p>Historically at Deezer, CA certs for the Kubernetes clusters were generated using standard <em>openssl</em> commands, based on the requirements described in the <a href="https://kubernetes.io/docs/setup/best-practices/certificates/#single-root-ca">Kubernetes official documentation</a>.</p><p>This works perfectly (official Kubernetes documentation uses <em>cfssl</em>, but that’s the same thing). Unfortunately it’s not really efficient, not super safe (CA keys are stored on disk), and prone to errors.</p><p>As most of our secrets are stored in <a href="https://www.vaultproject.io/">Hashicorp Vault</a>, we decided to take this opportunity to move the certs to Vault (which has its own PKI engine).</p><p>I’ll not get into the details to use Vault’s PKI engine, but the idea here is to:</p><ul><li>create a new secret engine</li><li>generate a CA that will be safely stored in Vault</li><li>configure a role that will allow some basic configuration</li><li>generate a bunch of certs with it</li></ul><pre>vault policy write pki-policy pki-policy.hcl<br>vault secrets enable -path=pki_k8s pki<br>vault secrets tune -max-lease-ttl=43800h pki_k8s<br>vault write pki_k8s/root/generate/internal common_name=&quot;kubernetes-ca&quot; ttl=43800h<br>vault write pki_k8s/roles/kubernetes allowed_domains=&quot;kubernetes, default, svc, yourdomain.tld&quot; allow_subdomains=true allow_bare_domains=true max_ttl=&quot;43800h&quot;</pre><p><strong>Certs for everyone!</strong></p><blockquote><em>You get a cert!<br>And you get a cert!<br>And you get a cert!</em></blockquote><blockquote><em>(I hope you get the “Oprah meme” as well)</em></blockquote><p>In order to safely automate and deliver the certificates to the Kubernetes nodes, we used <a href="https://github.com/hashicorp/consul-template">consul-template</a>. <em>consul-template</em> is another tool from Hashicorp that you can use to generate configuration files based on templates, with variables (or secrets) found in Consul or Vault.</p><p>Basically, the template for a single certificate file will look like this (<em>apiserver-cert.tpl</em> for the API server certificate in this example):</p><pre>{{- /* apiserver-cert.tpl */ -}}<br>{{ with secret &quot;pki_k8s/issue/kubernetes&quot; &quot;common_name=kube-apiserver&quot; &quot;alt_names=kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.kubernetes, yourdomain.tld&quot; &quot;ip_sans=100.64.0.1, IP.ADDRESS.MASTER.1, IP.ADDRESS.MASTER.2, IP.ADDRESS.MASTER.3&quot; &quot;exclude_cn_from_sans=true&quot; &quot;ttl=17520h&quot; }}<br>{{ .Data.certificate }}{{ end}}</pre><p>The <em>consul-template</em> file will look like that:</p><pre>template {<br>  source      = &quot;/etc/consul.d/templates/ca.pem.tpl&quot;<br>  destination = &quot;/etc/kubernetes/pki/ca.pem&quot;<br>}<br>template {<br>  source      = &quot;/etc/consul.d/templates/admin-cert.tpl&quot;<br>  destination = &quot;/etc/kubernetes/pki/admin.pem&quot;<br>}<br>template {<br>  blah blah blah<br>...</pre><p>And we’ll run a command like this one, which will replace all previous certificates with the new ones.</p><pre>consul-template -config /etc/consul.d/templates/consul-template-config-master.hcl</pre><p>Note: One issue with this approach is that each call of the <em>consul-template</em> executable will produce a new certificate. For individual certs like node certs, this is fine. But that’s not true for the API server certificate key, which has to be the same on every master node. So, we made an exception for all certificates on the master node: we generated them on only one node then pushed them on all the other master nodes.</p><h3>Ground control to Major Tom</h3><p>All the certificates are now refreshed. But not all the Kubernetes components support certificate reload on the fly. We will need to restart them in the right order!</p><p>The first thing to do is to restart all <em>etcd</em> servers, roughly at the same time. Once you do this, the API server will be unable to query the Kubernetes cluster’s state. That means you will lose all control on your cluster 😱.</p><pre>systemctl restart etcd</pre><p>At this point all future <em>kubectl</em> commands will fail. All Kubernetes functions (such as autoscaling, scheduling pods etc.) will stop working. So our priority is to restore this as soon as possible. This can be fixed quickly by restarting the <em>api-server</em> manually (or killing them with a SIGKILL if they are <em>Kubernetes Pods</em> and not <em>systemd</em> units).</p><p>We can then start to properly restart the control plane components that rely on <em>etcd</em> or the API server.</p><p>Start with the CNI (like <em>flannel</em> for example), delete the <em>kubelet</em> certificate/key pair and restart them on all nodes (which will automatically regenerate it).</p><p>Note: <em>kubelet</em> certificates location may depend on your setup, but it usually can be found in <em>/var/lib/kubelet/pki/</em> on all the nodes.</p><pre>rm /var/lib/kubelet/pki/kubelet.{crt,key}<br>systemctl restart kubelet</pre><p>Finally, we can restart all remaining Kubernetes control plane components using <em>kubectl</em> commands from the master node themselves.</p><pre>/usr/bin/kubectl --namespace kube-system delete pods --selector component=kube-apiserver<br>/usr/bin/kubectl --namespace kube-system delete pods --selector component=kube-controller-manager<br>/usr/bin/kubectl --namespace kube-system delete pods --selector component=kube-scheduler</pre><p>Our control plane is up again!</p><h3>The tokens sleep tonight</h3><p>Now that the control plane runs with the new certificates, if you look at the API server logs, you’ll see lots of cryptic error messages about wrong tokens.</p><p>We have to refresh all the tokens stored in the cluster and that will be done automatically by the <em>Kubernetes Controller Manager</em> once we clear the <em>Secrets</em>. This loop will parse all the <em>Namespaces</em>, find all <em>Secrets</em> of type <em>service-account-token</em> and will patch the secret to remove the token from them.</p><pre>for ns in `kubectl get ns | grep Active | awk &#39;{ print $1 }&#39;`; do<br>  for token in `/usr/bin/kubectl get secrets --namespace $ns --field-selector type=kubernetes.io/service-account-token -o name`; do<br>    /usr/bin/kubectl get $token --namespace $ns -o yaml | /bin/sed &#39;/token: /d&#39; | /usr/bin/kubectl replace -f - ;<br>  done<br>done</pre><p>We can now move on to restarting applications, with priority to those that really need access to the API server.</p><h3>Now, the applications</h3><p>Restart <em>coredns</em> first (or <em>kubedns</em>, depending on your setup). Since the certs have been renewed, all internal Kubernetes name resolution should be failing, which will soon be problematic.</p><pre>/usr/bin/kubectl --namespace kube-system rollout restart deployment.apps/coredns</pre><p>But you should also restart every app that seems important to the function of the cluster, including (but not limited to):</p><ul><li><em>kube-proxy</em></li><li><em>prometheus</em></li><li><em>Ingress controllers</em></li><li>any monitoring app</li></ul><p>Finally, restart all applications that need API access after that. All applications that don’t need access to the API server can be restarted when you have the time.</p><p>Your Kubernetes CA certificate is now renewed; you can relax for a few more months 😊.</p><p><strong>Wrapping this up</strong></p><p>Renewing Kubernetes CA certificates is not a trivial task but we’ve learned that it could be done in most cases, with careful planning.</p><p>We automated all the tasks described in this article in Ansible roles and played them on staging environments until we were ready to roll it out.</p><p>Last December, we planned this intervention in all our clusters. They were all updated in a few minutes with no observed downtime on our services visible to end-users, and no more disruption than expected on applications that required access to the API server.</p><blockquote>I love it when a plan comes together</blockquote><p><strong>Sources</strong></p><ul><li><a href="https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/04-certificate-authority.md">Github — Kelsey Hightower’s “Kubernetes the hard way”</a></li></ul><p>Postmortems on CA certs expirations</p><ul><li><a href="https://vadosware.io/post/2019-12-k8s-cert-expiration-outage/">Vadosware — 2019–12 K8s certificate expiration outage</a></li></ul><p>Building and breaking Kubernetes clusters on the fly</p><ul><li><a href="https://www.youtube.com/watch?v=1xHmCrd8Qn8">Youtube — Continuously Deliver your Kubernetes Infrastructure — Mikkel Larsen, Zalando SE</a></li></ul><p>Kubernetes official documentation on certificates</p><ul><li><a href="https://kubernetes.io/docs/setup/best-practices/certificates/#single-root-ca">Kubernetes — Single Root CA best practices</a></li><li><a href="https://kubernetes.io/docs/tasks/tls/manual-rotation-of-ca-certificates/">Kubernetes — Manual rotation of CA certificates</a></li><li><a href="https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#manual-certificate-renewal">Kubeadm — Manual certificate renewal</a></li></ul><p>Hashicorp Vault</p><ul><li><a href="https://learn.hashicorp.com/tutorials/vault/pki-engine?in=vault/interactive">Hashicorp — Build Your Own Certificate Authority (CA)</a></li><li><a href="https://www.vaultproject.io/api-docs/secret/pki#allow_subdomains">Hashicorp Vault — PKI Secrets Engine (API)</a></li><li><a href="https://www.digitalocean.com/blog/vault-and-kubernetes/">Digital Ocean — Using Vault as a Certificate Authority for Kubernetes</a></li><li><a href="https://www.youtube.com/watch?v=k8FXTeFCp90&amp;feature=youtu.be">Youtube — Streamline Certificate Management</a></li></ul><p><a href="https://www.deezerjobs.com/en/jobs/?utm_source=deezerio&amp;utm_medium=blogpost&amp;utm_campaign=updating_kubernetes_ca_certificates">What do you want to do?</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9518108791d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deezer-engineering/updating-kubernetes-ca-certificates-the-hard-way-f9518108791d">Updating Kubernetes CA certificates the hard way</a> was originally published in <a href="https://medium.com/deezer-engineering">Deezer I/O</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>