<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pawel Chmielinski on Medium]]></title>
        <description><![CDATA[Stories by Pawel Chmielinski on Medium]]></description>
        <link>https://medium.com/@pawilon?source=rss-397ce477cd44------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YlBrMiF9-O_6q_SOO3tAvQ.jpeg</url>
            <title>Stories by Pawel Chmielinski on Medium</title>
            <link>https://medium.com/@pawilon?source=rss-397ce477cd44------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:23:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pawilon/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Tuning your Linux kernel and HAProxy instance for high loads]]></title>
            <link>https://medium.com/@pawilon/tuning-your-linux-kernel-and-haproxy-instance-for-high-loads-1a2105ea553e?source=rss-397ce477cd44------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a2105ea553e</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[haproxy]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Pawel Chmielinski]]></dc:creator>
            <pubDate>Mon, 27 Feb 2017 18:39:12 GMT</pubDate>
            <atom:updated>2017-02-27T18:39:12.579Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nd6lFOR-7dwSwh4fYhQzqg.jpeg" /><figcaption>Source: <a href="https://unsplash.com/photos/RfAEjh_J6e0">Unsplash.com</a></figcaption></figure><h4>Configuration tips that enable serving a higher number of concurrent users</h4><p>If you have ever configured a loadbalancer like HAProxy or a webserver like Nginx or Apache to handle a high number of concurrent users, then you might have discovered that there are quite a few tweaks required in order to achieve the desired effects. Could you recite all of them from the top of your head? If not, don’t worry — this article has got you covered.</p><p>We will have a look at two types of tweaks. First ones are Linux kernel tweaks, which are common regardless if you’re using HAProxy, Nginx, Apache or other webservers. The second type covers HAProxy specific configuration.</p><h3>Kernel tweaks</h3><h4>Number of open files</h4><p>Why should we care about open files when handling web traffic? It’s simple — every incoming or outgoing connection needs to open a socket and each socket is a file on a Linux system. If you’re configuring a webserver serving static content from the local filesystem, then each connection will result in one open socket. However, if you’re configuring a loadbalancer serving content from backend servers then each incoming connections will open a minimum of two sockets, or even more, depending on the loadbalancing configuration.</p><p>It’s important that you configure the maximum number of open files, as the default number is pretty low. On Ubuntu 16.04 it’s up to 4096 open files per process, which is not an awful lot. You know that you have hit the limit if you see Too many files open lines in your logs.</p><p>Now, there are two ways to configure max open files, depending on whether your distribution uses systemd or not. Most tutorials found on Google assume systemd is not used, in which case the number of open files can be set by editing /etc/security/limits.conf (assuming pam_limits is used for daemon processes, see <a href="http://serverfault.com/questions/356962/where-are-the-default-ulimit-values-set-linux-centos/">this answer</a> for a more thorough explanation)<em>. </em>A sample config to set both the soft and hard limits for every user on the system to 100k would look like this:</p><pre>* soft nofile 100000<br>* hard nofile 100000<br>root soft nofile 100000<br>root hard nofile 100000</pre><p>Afterwards restart your webserver/loadbalancer to apply the changes. You can check if it worked by issuing:</p><pre>cat /proc/&lt;PID_of_webserver&gt;/limits</pre><p>If the daemon process doesn’t use pam_limits, it won’t work. A bit hacky workaround is to use ulimit 100000 directly in the init script or any of the files sourced inside it, like /etc/default/&lt;service_name&gt; on Ubuntu.</p><p>If you’re on a system that uses systemd you will find that setting limits.conf doesn’t work as well. That’s because systemd doesn’t use the /etc/security/limits.conf at all, but instead uses it’s own configuration to determine the limits. However, keep in mind that even with systemd,<em> </em>limits.conf is still useful when running a long-running process from within a user shell, as the user limits still use the old config file. These can be displayed by issuing:</p><pre>ulimit -a</pre><p>Okay, how do we configure maximum open files for systemd? The answer is to override the configuration for a specific service. We do this by placing a file in /etc/systemd/system/&lt;service_name&gt;.service.d/override.conf. The file content could look like this to set a 100k max open files limit:</p><pre>[Service]<br>LimitNOFILE=100000</pre><p>After the change we have to reload systemd configuration and restart our service:</p><pre>systemctl daemon-reload<br>systemctl restart &lt;service_name&gt;</pre><p>To make sure that the override worked use the following:</p><pre>systemctl cat &lt;service_name&gt;<br>cat /proc/&lt;PID&gt;/limits</pre><p>This should work fine for Apache and Nginx, but if you’re running HAProxy, you’re in for a surprise. When we restart HAProxy, there are actually 3 processes spawned and only the top-level one (/usr/sbin/haproxy-systemd-wrapper) has our limits applied! That’s because HAProxy configured it’s open files limit automatically based on the maxconn value in haproxy.conf. We will have a look at this parameter further down the article.</p><p>There are two other values that relate to maximum open files — global values for the system. These can be checked by issuing:</p><pre>sysctl fs.file-max<br>sysctl fs.nr_open</pre><p>fs.file-max determines the maximum number of files in total that can be opened on the system. fs.nr-open determines the maximum value that fs.file-max can be configured to. On modern distributions both are configured to high values, but if you find that’s not the case for your system, then feel free to tweak them as well. In any case make sure that they are configured much higher the the value used in limits.conf and/or systemd, because we don’t want a single process to be able to block the operating system from opening files.</p><h4>Conntrack</h4><p>In production systems there’s a high probability that you’re using iptables/ufw/firewalld firewall on your system. These services all use the nf_conntrack kernel module. The module is autoloaded when starting the service and adds some additional kernel parameters. The one we are most interested in is called net.netfilter.nf_conntrack_max and determines the maximum number of connections that the kernel module will track. If the number of connections exceed this number then new connections will be dropped and you will see the following log message:</p><pre>nf_conntrack: table full, dropping packet</pre><p>It’s recommended not to tweak nf_conntrack_max manually, but indirectly, by setting nf_conntrack_buckets. The value for nf_conntrack_max will be automatically set to 8*nf_conntrack_buckets. The standard way of configuring a kernel parameter would be to modify /etc/sysctl.conf or /etc/sysctl.d/&lt;config_name&gt;.conf and execute sysctl -p. However, in this case it might throw errors, since the parameter exists only if the nf_conntrack module is already loaded. A better way is to configure it on module load, by setting it in /etc/modprobe.d/nf_conntrack.conf. A sample configuration would look like this:</p><pre>options nf_conntrack hashsize=100000</pre><p>and would result in net.netfilter.nf_conntrack_max taking a value of 800k. This will only take effect after the module is reloaded. For example, on Ubuntu 16.04 with ufw, the commands to do this are:</p><pre>systemctl stop ufw<br>modprobe -rv nf_conntrack<br>systemctl start ufw</pre><p>You might get a modprobe: FATAL: Module nf_conntrack is in use. error on the second command. In order to fix this, find out what modules are using nf_conntrack with lsmod | grep nf_conntrack and unload all of them at once. In my case, the actual command looked like this: modprobe -rv nf_nat_ftp nf_conntrack_netbios_ns nf_nat xt_conntrack nf_conntrack_broadcast nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack_ipv6 nf_conntrack</p><h4>TCP parameters</h4><p>All the values below should be configured in /etc/sysctl.conf or /etc/sysctl.d/&lt;config_name&gt;.conf and enabled by executing sysctl -p. The values used assume the server has a lot of spare memory — in my case each server has 4GB RAM.</p><pre>net.ipv4.tcp_max_syn_backlog = 100000<br>net.core.somaxconn = 100000<br>net.core.netdev_max_backlog = 100000</pre><p>First, a very short reminder about how a TCP three-way handshake works:</p><ol><li>Client sends a SYN packet to the server — this indicates an intention to initiate a TCP connection with the server</li><li>Server responds with a SYN-ACK packet — server accepts the connection</li><li>Client responds with an ACK packet — client acknowledges that the server accepted the connection. The connection is now established.</li></ol><p>Now to the values:</p><ol><li>net.ipv4.tcp_max_syn_backlog — how many half-open connections for which the client has not yet sent an ACK response can be kept in the queue. The default net.ipv4.tcp_max_syn_backlog is set to 128 on my Ubuntu 16.04. We set it to a higher value, because if the queue is full then legitimate clients won’t be able to connect — they would get a <em>connection refused </em>error, just as if the target port was closed. This queue will be mostly filled up by clients, which are slow to send the ACK packet (or attackers conducting a <a href="https://en.wikipedia.org/wiki/SYN_flood">SYN flood attack</a>). However, there also other scenarios — specifically, if HAProxy’s globalmaxconn value is reached, it will stop responding and the requests will wait in this queue until a socket is free. This will cause a delay on the client’s side, but we assume it’s better than a refused connection. Do note, however, that this might NOT be preferable if this server is behind a loadbalancer — in that case we might prefer to refuse the connection, so that the loadbalancer can immediately pick a different server.</li><li>net.core.somaxconn — the maximum value that net.ipv4.tcp_max_syn_backlog can take. Higher values are silently truncated to the value indicated by somaxconn. In older kernels this value could not be higher than 65535.</li><li>net.core.netdev_max_backlog — the maximum number of packets in the receive queue that passed through the network interface and are waiting to be processed by the kernel. The default is set to 1000 on Ubuntu 16.04 and probably should be fine on a 1G network. However, if you’re using a 10G connection, then it’s useful to increase this queue’s size. If the queue is full the incoming packets will be silently dropped.</li></ol><p>If you have read other tuning guides available on the Internet, you might have also seen recommendations to tweak net.ipv4.tcp_rmem and net.ipv4.tcp_wmem, which control the sizes of the receive and send buffers. However, these days the kernel does a good job to self-regulate these buffers, so it’s unlikely the defaults need to be changed.</p><h4>TCP port range</h4><p>This is relevant if you’re configuring a loadbalancer or a reverse proxy. In this scenario you may run into an issue called <em>TCP</em> <em>source port exhaustion.</em> If you’re not using some sort of connection pooling or multiplexing, then in general each connection from a client to the loadbalancer also opens a related connection to one of the backends. This will open a socket on the loadbalancer’s system. Each socket is identified by the following 5-tuple:</p><ul><li>Protocol (we assume here, that this is always TCP)</li><li>Source IP</li><li>Source port</li><li>Destination IP</li><li>Destination port</li></ul><p>You cannot have 2 sockets that are identified by the same 5-tuple on the system. The problem is TCP only has 65535 ports available. So in a scenario where the reverse proxy only has a single IP address and is proxying to a single backend on a single IP and port, we’re looking at 1*1*65535*1*1 unique combinations. The actual number is actually lower, because by default Linux will only use range 32768-60999 as the source ports for outgoing connections. We can increase this, but the first 1024 are reserved, so in the end we set it to a range of 1024–65535. This is done with sysctl, using the same process as described before — writing the value to /etc/sysctl.conf or /etc/sysctl.d/&lt;config&gt;.confand executing sysctl -p.</p><pre>net.ipv4.ip_local_port_range=1024 65535</pre><p>With an effective number of ports equal to 64511 ports, we have more breathing room, but in certain situations it might still not be enough. In that case you can look into increasing the number of other items in the 5-tuple:</p><ul><li>Configure more than one IP on the loadbalancer system. Make sure the loadbalancer is configured to also use these additional IPs</li><li>Configure the destination backend to listen on multiple IPs and configure the loadbalancer to connect to these IPs</li><li>Configure the destination to listen on multiple ports, if possible</li></ul><p>Increasing the number of source IPs on the loadbalancer is most likely the easiest option available. In HAProxy this is possible by configuring the <a href="https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.2-source">source</a> option in the server line of the backend. If the backend is on the local system, it might be easier to tweak the destination IPs, as every request to any IP in the 127.0.0.0/8 pool will go to localhost without any additional configuration required.</p><h3>HAProxy tweaks</h3><h4>Number of processes</h4><p>By default HAProxy uses only a single process to serve requests. If running on a system with multiple cores, you might find out during high load periods that a single core has 100% CPU utilisation while others are mostly idle. This can be rectified by increasing the number of running processes. Assuming HAProxy is placed on a system with 4 cores, the config would look like the following:</p><pre>global<br>   nbproc  4<br>   cpu-map  1 1<br>   cpu-map  2 2<br>   cpu-map  3 3<br>   cpu-map  4 4</pre><pre>(...)</pre><p>nbproc directive will configure the number of processes, which in general should be equal to the number of cores on the system. cpu-map will assign a specific CPU to the running process, so that each process has it’s own.</p><p>When increasing the number of HAProxy processing bear in mind this has certain consequences, which can be found in the documentation. The gist of it is that many configuration options work <strong>per-process, </strong>rather than on a global level, eg. maxconn, admin socket, stick-tables.</p><h4>Maxconn</h4><p>The default number of maximum connections is provided during HAProxy’s compilation. The package available on my Ubuntu 16.04 was compiled with 2000 maximum connections. This is quite low, but can be changed in the configuration. First thing you need to know, however, is that there are actually three maxconn values in the config. A sample config might look like this:</p><pre>global<br>    maxconn 100000<br>    (...)</pre><pre>frontend frontend1<br>    bind *:80<br>    mode http<br>    maxconn 100000<br>    default_backend backend1<br>    (...)</pre><pre>backend backend1<br>    server frontend-1 192.168.1.1:80 maxconn=200<br>    server frontend-2 192.168.1.2:80 maxconn=200<br>    balance roundrobin<br>    (...)</pre><p>There’s a very good <a href="http://stackoverflow.com/questions/8750518/difference-between-global-maxconn-and-server-maxconn-haproxy">answer on Stack Overflow</a> illustrating how these values work, which I recommend reading. The short version is that both the global and the frontend maxconn values are by default equal to 2000, so you have to configure both, and they are configured <strong>per-process.</strong> So with nbproc configured to 4 and global maxconn set to 100k your HAProxy server will accept 400k connections. The remainder won’t be rejected, but instead queued in the kernel.</p><p>There’s also a per-server maxconn, which by default is not limited. This configuration option allows you to have a smaller number of active connections to each backend server. The rest gets queued in HAProxy internal queues. With a smaller number of connections reaching the servers, every request will get processed more quickly, so even though requests get queued on HAProxy their overall journey duration might actually be shorter. See <a href="http://blog.haproxy.com/2011/06/28/play_with_maxconn_avoid_server_slowness_or_crash/">this article</a> to find some practical examples.</p><h4>SSL cachesize</h4><p>This is an important configuration value to tweak if your HAProxy instance is configured to serve SSL. By default the cache size is equal to 20k and the cache is shared between processes. Each SSL connection will create an entry in the cache. If you have more connections than the cache size, the oldest entry will get purged. In practice this means that if you have more connecting concurrent users than this value, they will all keep reissuing CPU-intensive SSL handshakes, possibly causing sudden performance degradation once you hit this number. It’s best to configure this value to a much higher number, keeping in mind that each entry requires ~200 bytes of memory. Sample config:</p><pre>global<br>    tune.ssl.cachesize 1000000<br>    (...)</pre><p>With a value set to 1 million we thus need at least 200MB of memory to hold the cache.</p><h3>Conclusion</h3><p>The configuration options presented in this article remove the most common bottlenecks when serving high numbers of concurrent users. Using these recommendations it’s most likely your next bottleneck will be the CPU or the network bandwidth, rather than a misconfigured configuration value. Thank you for reading and please let me know in the comments if I have missed anything.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a2105ea553e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A superior way to use Let’s Encrypt]]></title>
            <link>https://medium.com/@pawilon/a-superior-way-to-use-let-s-encrypt-fee6bd1b7b11?source=rss-397ce477cd44------2</link>
            <guid isPermaLink="false">https://medium.com/p/fee6bd1b7b11</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[ssl]]></category>
            <category><![CDATA[lets-encrypt]]></category>
            <dc:creator><![CDATA[Pawel Chmielinski]]></dc:creator>
            <pubDate>Sat, 16 Apr 2016 22:06:41 GMT</pubDate>
            <atom:updated>2016-04-19T10:06:22.920Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*onxd0XmRhqqqF7rAHhUYhg.jpeg" /><figcaption>The quality of your self-signed TLS certificates [Source: <a href="https://unsplash.com/photos/KhA08OATrQ4">https://unsplash.com/</a>]</figcaption></figure><p><a href="https://letsencrypt.org/">Let’s Encrypt</a> is awesome. This is an objective fact. If you never heard about it and you are still using self-signed certificates, go check out the link. In short, Let’s Encrypt allows you to get valid SSL certificates for free. Once you set it up, there’s absolutely no reason to use self-signed certificates any longer.</p><h4>… but until recently there was a catch</h4><p>In order to understand it, though, we need to lay out some basics. Let’s Encrypt consists of a few key elements:</p><ul><li><a href="https://github.com/letsencrypt/boulder">Boulder</a> CA written in Go</li><li><a href="https://github.com/letsencrypt/letsencrypt">Letsencrypt</a> client written in Python</li><li><a href="https://tools.ietf.org/html/draft-barnes-acme-01">ACME Protocol</a> which is used to communicate between the Letsencrypt client and the Boulder server</li></ul><p>If you look into the draft for the ACME protocol, you will learn that it’s using a technique called “Domain Validation”. In essence, this means that the CA will ensure that you are allowed to create a certificate for a certain domain by issuing a challenge that only a person in control of that domain could successfully complete. This could mean one of the two methods (for now):</p><ul><li>Provisioning a DNS record under example.com, or</li><li>Provisioning an HTTP resource under a well-known URI on <a href="https://example.com/">https://example.com/</a></li></ul><h4>So what was the catch?</h4><p>Until recently, neither Boulder server nor Letsencrypt client supported the DNS method. Therefore ,you had to use the HTTP one. And the catch came with some of its limitations:</p><ul><li>What if I want a valid SSL certificate for a service that’s only ever used internally and should never be exposed to the Internet? In this scenario, the Boulder server won’t be able to connect to it in order to download the challenge response. I might not even have an A DNS record for it, which is a prerequisite for the HTTP method.</li><li>What if I want to create the certificates for all my domains in a central point, like the build server? The public DNS records point to my webservers or loadbalancers, not my build server, therefore, I’ll have to install letsencrypt client on those.</li></ul><p>You can probably think of more issues, but these two proved to be the most limiting for me. I just wanted to automatically create all my certificates on a single server in the management zone, which also wasn’t accessible via the Internet. The management server already had the tools necessary to push the certificates to the correct servers.</p><h4>DNS challenge to the rescue</h4><p>Fortunately, Boulder implemented the DNS method in early 2016. Less fortunately, the official letsencrypt client still doesn’t have it as of the day this post is being written (though looking at the <a href="https://github.com/letsencrypt/letsencrypt/pull/2061">pull request</a>, it seems to be imminent). However, if the ACME protocol specification is open, what’s stopping other people from writing their own clients?</p><p>And that’s exactly what people did. Some of these alternative clients already implement the DNS method. I used <a href="https://github.com/lukas2511/letsencrypt.sh">letsencrypt.sh</a>, which I highly recommend. It also has fewer dependencies than the official client, which is a nice bonus.</p><p>In order to use the DNS method, you will need one additional thing, though — a hook. It’s a program that implements the functions that are needed to programmatically create the necessary TXT records in your DNS server. The hook you’ll use will depend on the DNS solution that you use, but if it’s not possible to interact with your DNS programmatically, then, unfortunately, you won’t be able to use the DNS method in an automatic manner.</p><p>I was in luck, though, as the project used AWS Route 53 as the DNS provider. Therefore, I could just use a <a href="https://gist.github.com/tache/3b6760784c098c9139c6">hook written by tache</a> (forked from <a href="https://gist.github.com/asimihsan/d8d8f0f10bdc85fc6f8a">original solution written by asimihsan</a>). The hook is written in Ruby and requires installing some additional gems, as well as configuring your AWS credentials.</p><h4>Bringing it all together</h4><p>To save you all some trouble, I created a wrapper around letsencrypt.sh and the hook, which will also install all the necessary dependencies. You can find it on <a href="https://github.com/pawilon/letsencrypt-r53-wrapper">Github</a>.</p><p><strong>Happy encrypting!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fee6bd1b7b11" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>