<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Niels Søholm on Medium]]></title>
        <description><![CDATA[Stories by Niels Søholm on Medium]]></description>
        <link>https://medium.com/@nielssj?source=rss-7db0883f6c82------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9IM41t-iA2VwDUhI1QuuzA.jpeg</url>
            <title>Stories by Niels Søholm on Medium</title>
            <link>https://medium.com/@nielssj?source=rss-7db0883f6c82------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 00:36:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nielssj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Safely log full HTTP requests on Google Kubernetes Engine]]></title>
            <link>https://faun.pub/safely-log-full-http-requests-on-google-kubernetes-engine-48c74e3a7e67?source=rss-7db0883f6c82------2</link>
            <guid isPermaLink="false">https://medium.com/p/48c74e3a7e67</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[encryption]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Niels Søholm]]></dc:creator>
            <pubDate>Wed, 05 Dec 2018 07:49:16 GMT</pubDate>
            <atom:updated>2019-07-17T16:12:16.568Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y7m6fdiPpcyoE-g8Zh8OPw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/uW2pwuHioNc?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Micaela Parente</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>Full HTTP request logging by default is not recommendable for security reasons and isn’t practical in any system with a reasonable amount of traffic. But you are inevitably going to end up in a situation where the only way to confirm your debugging hypothesis is to get your hands on those details. In traditional systems this capability was a given, however as our infrastructure becomes more distributed and outsourced such operations become less trivial.</p><p>In the following I describe a safe way to do a targeted TCP dump on a GKE node. I used this approach recently to diagnose an issue at a large enterprise customer whose own networking infrastructure (security proxy) was stripping our authentication headers and they were blaming our SaaS product.</p><p>The guide has 4 sections and the first one is really the only one which is GKE-specific:</p><ul><li>Debugging on cluster nodes</li><li>Encrypting the traces</li><li>Dumping TCP packets</li><li>Fetching the dump</li></ul><h3>Debugging on cluster nodes</h3><p>I normally don’t have any interest in accessing or otherwise messing directly with GKE node instances. But in this particular case I need to open the hood slightly and in order to do so I utilize the excellent CoreOS toolbox container provided by GKE. This allows me to install my tools and persist things in an a place that is isolated from the node itself.</p><p>I SSH onto the my cluster node using gcloud command-line utility:</p><pre>gcloud compute ssh &lt;node-name&gt; --zone &lt;zone&gt;</pre><p>I pull and start the toolbox container using a script included by default in GKE node images (the first time this may take a minute):</p><pre>sudo /usr/bin/toolbox</pre><p>Once installation is done my shell gets attached to the container and I’ve got access to an isolated file system and a package manager. If I come back at a later point I can attach to the container by re-running the above mentioned script.</p><h3>Encrypting the traces</h3><p>Before we get into the actual tracing of requests I want to make sure that you have the tools to do this in a safe way. At <a href="https://qvest.io/">Qvest.io</a> we encrypt all production <a href="https://searchstorage.techtarget.com/definition/data-at-rest">data at rest</a> and in transit. A rogue debugging mission like this is no excuse to be careless.</p><p>In order to ensure that I don’t persist anything without encrypting it first I never output to a file directly. I pipe to the gpg utility, ideally with a public/private key-pair. If you can’t be bothered to juggle keys I urge you to at least use symmetric encryption.</p><p>Here’s how to encrypt (gpg will prompt for passphrase):</p><pre>cat test.txt | gpg -c &gt; test_encrypted.gpg</pre><p>…and here’s how to decrypt the same file:</p><pre>gpg -d test.txt.gpg</pre><p>We obviously need to replace cat test.txt with our dump script. I will show you how in the following section. If you want to be real safe, here is a <a href="https://guides.library.illinois.edu/data_encryption/gpgcheatsheet">gpg cheatsheet</a> that will show you how to use a public/private key pair.</p><h3>Dumping TCP packets</h3><p>I dump the packets using tcpdump piped to grep with a filter that matches the qvestId of the custome, an application-specific identifier which is frequently included in requests to our API. You may use any identifier specific to your application which may be stored in HTTP headers, path or even body. You can find more tcpdump examples on <a href="https://danielmiessler.com/study/tcpdump/">Daniel Miessler’s excellent page</a>. The important thing is that your filter is specific enough to reach a relevant and reasonably sized volume.</p><p>In my case I went for the following line:</p><pre>tcpdump -As10400 &#39;dst port 3000&#39; \<br>  | grep -B10 -A20 &#39;qvestId: &lt;uuid&gt;&#39; \<br>  | gpg -c &gt; /home/dump.gpg &amp;</pre><p>Where &lt;uuid&gt; was replaced by before-mentioned customer-specific identifier. The filter ensures that only 10 lines before and 20 lines after a matching line are captured, you may want to tweak that depending on what you are matching on and the nature of your requests. The captured lines are piped to gpg to be encrypted (see previous section) and finally into a file. The &amp; at the end ensures that the script is run in the background and continues to run after the shell is closed.</p><p>Once the script is running on all relevant cluster nodes the only thing left to do is to ask the customer to kindly interact with the web app.</p><h3>Fetching the dump</h3><p>When we’re convinced that the required traffic has passed we need to log into our cluster node and gracefully stop tcpdump because otherwise we might get a partial and corrupt dump. When I’m connected to the toolbox I first identify the process id of the script by running:</p><pre>ps aux | grep tcpdump</pre><p>Then I get something along the lines of:</p><pre>root     <strong>16843</strong>  0.1  0.0  78512  4752 pts/0    S+   22:04   0:00 sudo tcpdump -As10400 dst port 3000</pre><p>The process id is the first numeric value occurrence (bolded). I copy that and go:</p><pre>kill 16843</pre><p>It may take a few seconds and when the shell prompt returns then I know the script has been gracefully stopped.</p><p>You now have two options. You may examine the dumps directly in the terminal on the node, if that suits you then simply skip the next part and decrypt it and pipe it directly to your command-line text editor. Otherwise, the following section paragraphs will show how to copy the dumps to a local machine.</p><p>The dump file can be retrieved using scp which is conveniently included with the gcloud command-line utility. Since we configured our script to dump into /home of the toolbox container, which is exposed to the node host OS, we can copy it in one go by running from our local machine:</p><pre>gcloud compute scp &lt;node-name&gt;:/var/lib/toolbox/root-gcr.io_google-containers_toolbox-20180309–00/home/dump.gpg ./dump.gpg</pre><p>The exact directory name after /toolbox/ may need to be adjusted to fit your version of the container. You can find the name by logging into the node again and going:</p><pre>ls /var/lib/toolbox</pre><p>Once you have the file locally you can decrypt it and examine the glorious details by piping it into your text editor of choice. Here’s how I do so with Sublime Text:</p><pre>subl &lt;(gpg -d dump.gpg)</pre><p>Be sure to delete dumps as soon as you don’t need them anymore.</p><p>If you like problems like this, keep an eye on our <a href="https://qvest.io/jobs">jobs section at Qvest.io</a> or follow me <a href="https://twitter.com/nielssj">on twitter</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p><strong>Join our community Slack and read our weekly Faun topics ⬇</strong></p><figure><a href="https://www.faun.dev/join/?utm_source=medium.com%2Ffaun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oSdFkACJxs5iy1oR" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48c74e3a7e67" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/safely-log-full-http-requests-on-google-kubernetes-engine-48c74e3a7e67">Safely log full HTTP requests on Google Kubernetes Engine</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker volumes and file system permissions]]></title>
            <link>https://medium.com/@nielssj/docker-volumes-and-file-system-permissions-772c1aee23ca?source=rss-7db0883f6c82------2</link>
            <guid isPermaLink="false">https://medium.com/p/772c1aee23ca</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[guide]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Niels Søholm]]></dc:creator>
            <pubDate>Sun, 07 May 2017 16:48:32 GMT</pubDate>
            <atom:updated>2019-04-06T11:30:24.086Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qy4uDbkYSCVYieogtLbLYA.jpeg" /></figure><p>Docker containers are ephemeral (don’t persist data across runs). Most useful applications need some persistent storage. The volume feature offers a way to support this requirement, but it comes with some gotchas regarding file system permissions.</p><p>In most deployed setups you will be using a container orchestration mechanism and persistent storage is provided by some public cloud product which may have it’s own way of configuring permissions. However, during local development or in the early iterations of a product the easiest thing is to expose a host directory as a docker volume.</p><p>In short these are the facts to be aware of when configuring a host directory as a volume:</p><ul><li>The file permissions set on content in the volume are identical from the perspective of host as well as container.</li><li>Only UIDs (user ids) and GIDs (group ids) matter. For example names and passwords of users and groups do not need to match or even exist in both host and container</li><li>The container OS enforces file permissions on all operations made in the container runtime according to its own configuration. For example, if a user A exists in both host and container, adding user A to group B on the host will not allow user A to write to a directory owned by group B inside the container unless group B is created inside the container as well and user A is added to it.</li><li>By default the command of a container is run as root</li><li>It is possible (on a unix-based system) to set file/directory ownership to a GID which does not belong to any actual group</li></ul><p>If you keep the above facts in mind you should be able to configure your containers and volumes without too many surprises. If you’re not familiar with UNIX file permissions I can recommend the official <a href="https://help.ubuntu.com/community/FilePermissions">Ubuntu FilePermission wiki page</a>.</p><p>An example of how you might configure things conveniently for local development:</p><p>Set group ownership of the directory to be used as volume to some GID (in this example 1024) not used on any actual groups on the host</p><pre>chown :1024 /data/myvolume</pre><ul><li>Change permissions on the directory to give full access to members of the group (read+write+execute)</li></ul><pre>chmod 775 /data/myvolume</pre><ul><li>Ensure all future content in the folder will inherit group ownership</li></ul><pre>chmod g+s /data/myvolume</pre><ul><li>Create a user in the Dockerfile which is member of the 1024 group</li></ul><pre>RUN addgroup --gid 1024 mygroup</pre><pre>RUN adduser --disabled-password --gecos &quot;&quot; --force-badname --ingroup 1024 myuser </pre><pre>USER myuser</pre><ul><li>(Optional) Add your host user to the group allowing you to conveniently work with the directory from your host machine</li></ul><pre>adduser ubuntu 1024</pre><p>The above example is a minimal setup that ensures that you are not running the container command as root and can work with the content of attached volumes on the host without having to make use of the host root user. The setup does hard-code configuration at build time and leaves you unable to adjust the GID at runtime. If you need this you will have to pass the GID as an environment variable and include a generic script which can make use of it. This <a href="https://denibertovic.com/posts/handling-permissions-with-docker-volumes/">blog post by Deni Bertović</a> has guide to accomplish such a setup.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=772c1aee23ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>