<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yevhen Shyriakin on Medium]]></title>
        <description><![CDATA[Stories by Yevhen Shyriakin on Medium]]></description>
        <link>https://medium.com/@i.shyriakin?source=rss-4ea2e85a49ea------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Yevhen Shyriakin on Medium</title>
            <link>https://medium.com/@i.shyriakin?source=rss-4ea2e85a49ea------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:41:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@i.shyriakin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Where is my disk space?]]></title>
            <link>https://medium.com/@i.shyriakin/where-is-my-disk-space-b98c782b45c9?source=rss-4ea2e85a49ea------2</link>
            <guid isPermaLink="false">https://medium.com/p/b98c782b45c9</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[inode]]></category>
            <category><![CDATA[ext4]]></category>
            <category><![CDATA[dfs]]></category>
            <category><![CDATA[usd]]></category>
            <dc:creator><![CDATA[Yevhen Shyriakin]]></dc:creator>
            <pubDate>Mon, 06 Nov 2023 10:26:50 GMT</pubDate>
            <atom:updated>2023-11-06T10:26:50.205Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a compilation of different issues that happened to me and my colleagues throughout the years. Though setup is very simple it can show most of the possible issues with free disk space.</p><p>So our test server is Ubuntu 20.04 VM with LAMP stack installed. Root filesystem has only 5G of space but this is more than enough for our purposes. Beginning of each incident is identical — monitoring shows that our webserver is not responding or suddenly error rate has increased.</p><p>After alert you would login to the server via SSH and run some preliminary checks — <strong>uptime</strong>(let’s discuss why uptime is better than top other time), <strong>free, ss, ps</strong> and so on.</p><h3>Deleted files</h3><p>Let’s have a look at the first scenario. Standard check for such cases:</p><pre># df -h<br>Filesystem      Size  Used Avail Use% Mounted on<br>udev            445M     0  445M   0% /dev<br>tmpfs            98M  1.2M   97M   2% /run<br>/dev/sda2       4.9G  4.9G     0 100% /<br>tmpfs           489M     0  489M   0% /dev/shm<br>tmpfs           5.0M     0  5.0M   0% /run/lock<br>tmpfs           489M     0  489M   0% /sys/fs/cgroup<br>/dev/loop0       62M   62M     0 100% /snap/core20/1611<br>/dev/loop1       47M   47M     0 100% /snap/snapd/16292<br>/dev/loop2       68M   68M     0 100% /snap/lxd/22753<br>tmpfs            98M     0   98M   0% /run/user/1000</pre><blockquote><em>to reduce amount of data </em><strong><em>-x</em></strong><em> key of </em><strong><em>df</em></strong><em> can be used:</em></blockquote><pre># df -h -x tmpfs -x squashfs<br>Filesystem      Size  Used Avail Use% Mounted on<br>udev            445M     0  445M   0% /dev<br>/dev/sda2       4.9G  4.9G     0 100% /</pre><blockquote><em>This way </em><strong><em>df</em></strong><em> doesn’t show </em><strong><em>tmpfs</em></strong><em> or </em><strong><em>squashfs</em></strong><em> filesystems </em><a href="https://man7.org/linux/man-pages/man1/df.1p.html"><strong><em>man</em></strong></a></blockquote><p>Ok, we completely exhausted disk space on root partition. Let’s find who consumes space:</p><pre># du -h --max-depth=1 / | sort -h<br>0	/dev<br>0	/proc<br>0	/sys<br>4.0K	/media<br>4.0K	/mnt<br>4.0K	/opt<br>4.0K	/srv<br>16K	/lost+found<br>60K	/root<br>84K	/tmp<br>1.2M	/run<br>6.5M	/etc<br>71M	/home<br>205M	/boot<br>590M	/snap<br>701M	/var<br>2.6G	/usr<br>4.2G	/</pre><blockquote><strong><em>du</em></strong><em> also has very useful </em><strong><em>-x</em></strong><em> flag. With this flag </em><strong><em>du</em></strong><em> wouldn’t count directories on other filesystems. </em><a href="https://man7.org/linux/man-pages/man1/du.1.html"><strong><em>man</em></strong></a></blockquote><p>Total space consumed on root partition is 4.2G we should have ~700M of free space(~15%). Let’s recheck:</p><pre># du -sh /<br>4.2G	/</pre><p>this two values don’t add up:</p><pre>du: 4.2G	/<br>df: /dev/sda2       4.9G  4.9G     0 100% /</pre><p>So what consumes disk space? And this is where we are seeing our first confusing case. <strong>df</strong>-reported amount of free disk space is smaller than sum of sizes of all file. This is caused by <strong>deleted</strong> files:</p><pre># lsof -nP +L1<br>COMMAND     PID       USER   FD   TYPE DEVICE   SIZE/OFF NLINK   NODE NAME<br>dbus-daem   763 messagebus  txt    REG    8,2     249032     0    633 /usr/bin/dbus-daemon (deleted)<br>networkd-   770       root  txt    REG    8,2    5502744     0    965 /usr/bin/python3.8 (deleted)<br>systemd-l   784       root  txt    REG    8,2     268576     0   9809 /usr/lib/systemd/systemd-logind (deleted)<br>login       852       root  txt    REG    8,2      57104     0    811 /usr/bin/login (deleted)<br>unattende   859       root  txt    REG    8,2    5502744     0    965 /usr/bin/python3.8 (deleted)<br>systemd    1445 cloud_user  txt    REG    8,2    1620224     0   9792 /usr/lib/systemd/systemd (deleted)<br>(sd-pam)   1446 cloud_user  txt    REG    8,2    1620224     0   9792 /usr/lib/systemd/systemd (deleted)<br>sudo       1630       root  txt    REG    8,2     166056     0   1131 /usr/bin/sudo (deleted)<br>apache2   55875       root    8w   REG    8,2 1356369920     0 263817 /var/log/apache2/access.log (deleted)<br>mysqld    56982      mysql    6u   REG    8,2          0     0 147159 /tmp/#147159 (deleted)<br>mysqld    56982      mysql    7u   REG    8,2          0     0 147160 /tmp/#147160 (deleted)<br>mysqld    56982      mysql    8u   REG    8,2          0     0 147161 /tmp/#147161 (deleted)<br>mysqld    56982      mysql   13u   REG    8,2          0     0 147162 /tmp/#147162 (deleted)<br>php-fpm7. 59047       root    3u   REG    8,2          0     0 143507 /tmp/.ZendSem.g4K1c2 (deleted)<br>php-fpm7. 59059  wordpress    3u   REG    8,2          0     0 143507 /tmp/.ZendSem.g4K1c2 (deleted)<br>php-fpm7. 59060  wordpress    3ur  REG    8,2          0     0 143507 /tmp/.ZendSem.g4K1c2 (deleted)<br>apache2   59412   www-data    8w   REG    8,2 1356369920     0 263817 /var/log/apache2/access.log (deleted)<br>apache2   59413   www-data    8w   REG    8,2 1356369920     0 263817 /var/log/apache2/access.log (deleted)<br>apache2   59414   www-data    8w   REG    8,2 1356369920     0 263817 /var/log/apache2/access.log (deleted)</pre><p>Here we see that Apache log file has been deleted but remains open by process(Kerrisk, M. (2010). <em>The Linux Programming Interface</em>; p 346). Especially often this is caused by improperly configured logrotate. As sometimes logrotate deletes log without informing process that it needs to close file descriptor. We can get access to the data via /proc/<strong>$PID</strong>/fd/<strong>$DESCRIPTOR_NUMBER</strong>:</p><pre># ps auxf<br>...<br>root       55875  0.0  0.4   8104  4824 ?        Ss   14:03   0:00 /usr/sbin/apache2 -k start<br>www-data   59412  0.0  0.3   8336  3192 ?        S    14:57   0:00  \_ /usr/sbin/apache2 -k start<br>www-data   59413  0.0  0.4 754344  4956 ?        Sl   14:57   0:00  \_ /usr/sbin/apache2 -k start<br>www-data   59414  0.0  0.4 754584  4688 ?        Sl   14:57   0:00  \_ /usr/sbin/apache2 -k start<br>...</pre><pre># ls -la /proc/55875/fd/<br>total 0<br>dr-x------ 2 root root  0 Feb  7 15:05 .<br>dr-xr-xr-x 9 root root  0 Feb  7 15:05 ..<br>lr-x------ 1 root root 64 Feb  7 15:05 0 -&gt; /dev/null<br>l-wx------ 1 root root 64 Feb  7 15:05 1 -&gt; /dev/null<br>l-wx------ 1 root root 64 Feb  7 15:05 10 -&gt; &#39;pipe:[245146]&#39;<br>lr-x------ 1 root root 64 Feb  7 15:05 11 -&gt; &#39;pipe:[245147]&#39;<br>l-wx------ 1 root root 64 Feb  7 15:05 12 -&gt; &#39;pipe:[245147]&#39;<br>l-wx------ 1 root root 64 Feb  7 15:05 2 -&gt; /var/log/apache2/error.log<br>lrwx------ 1 root root 64 Feb  7 15:05 3 -&gt; &#39;socket:[109347]&#39;<br>lrwx------ 1 root root 64 Feb  7 15:05 4 -&gt; &#39;socket:[109348]&#39;<br>lr-x------ 1 root root 64 Feb  7 15:05 5 -&gt; &#39;pipe:[245144]&#39;<br>l-wx------ 1 root root 64 Feb  7 15:05 6 -&gt; &#39;pipe:[245144]&#39;<br>l-wx------ 1 root root 64 Feb  7 15:05 7 -&gt; /var/log/apache2/other_vhosts_access.log<br>l-wx------ 1 root root 64 Feb  7 15:05 8 -&gt; &#39;/var/log/apache2/access.log (deleted)&#39;<br>lr-x------ 1 root root 64 Feb  7 15:05 9 -&gt; &#39;pipe:[245146]&#39;</pre><pre># head /proc/55875/fd/8<br>10.0.2.4 - - [07/Feb/2023:14:58:18 +0000] &quot;GET / HTTP/1.1&quot; 200 10268 &quot;http://10.0.2.15/index.php/category/uncategorized/&quot; &quot;Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0&quot;<br>10.0.2.4 - - [07/Feb/2023:14:58:19 +0000] &quot;GET /wp-includes/js/wp-emoji-release.min.js?ver=6.1.1 HTTP/1.1&quot; 200 5360 &quot;http://10.0.2.15/&quot; &quot;Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0&quot;<br>10.0.2.4 - - [07/Feb/2023:14:58:19 +0000] &quot;GET /wp-includes/blocks/navigation/style.min.css?ver=6.1.1 HTTP/1.1&quot; 200 2568 &quot;http://10.0.2.15/&quot; &quot;Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0&quot;<br>10.0.2.4 - - [07/Feb/2023:14:58:19 +0000] &quot;GET /wp-includes/blocks/navigation/view.min.js?ver=c24330f635f5cb9d5e0e HTTP/1.1&quot; 200 761 &quot;http://10.0.2.15/&quot; &quot;Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0&quot;<br>...</pre><p>As data is still there we can recover it. This might save the day if you accidentally deleted mysql file and it is still open.</p><p>There are couple of possible solutions for this problem. You can restart service(file descriptor would be released) or you can truncate file:</p><pre>truncate -s0 /proc/55875/fd/8<br>OR<br>: &gt; &quot;/proc/55875/fd/8&quot;</pre><p>after this:</p><pre># df -h -x tmpfs -x squashfs<br>Filesystem      Size  Used Avail Use% Mounted on<br>udev            445M     0  445M   0% /dev<br>/dev/sda2       4.9G  3.6G  1.1G  78% /</pre><h4>So what happens here:</h4><ul><li>process(Apache in our case) called <strong>open</strong> syscall and received file descriptor to work with this file. As this is log file it wasn’t closed immediately</li><li>meanwhile some other process(or even the same process) tried to delete file(called <strong>unlink</strong> syscall)</li><li>this reduced links counter in inode to 0 and file disappeared from file tree</li><li>but kernel wasn’t able to free blocks as there is still open descriptor referencing inode</li><li>two possible solutions are to truncate file or to close descriptor</li></ul><h3>Sparse files</h3><p>Another problem can appear during creation of backups:</p><pre># tar -cvf docker.tar /var/lib/docker/<br>tar: Removing leading `/` from member names<br>/var/lib/docker/<br>/var/lib/docker/buildkit/<br>...<br>/var/lib/docker/devicemapper/devicemapper/<br>/var/lib/docker/devicemapper/devicemapper/metadata<br>tar: docker.tar: Cannot write: No space left on device<br>tar: Error is not recoverable: exiting now<br><br># du -sh /var/lib/docker/<br>12M	/var/lib/docker/<br># ls -lah docker.tar <br>-rw-r--r-- 1 root root 426M Feb  9 15:42 docker.tar</pre><p>But how 12M directory can create backup that is 426M and exhaust disk space? Let’s find biggest file:</p><pre># find /var/lib/docker -type f -printf &quot;%s\t%p\n&quot; | sort -n | tail -1<br>107374182400	/var/lib/docker/devicemapper/devicemapper/data<br># ls -lah /var/lib/docker/devicemapper/devicemapper/data<br>-rw------- 1 root root 100G Feb  9 15:30 /var/lib/docker/devicemapper/devicemapper/data</pre><p>Wait, 100G file on a 5G filesystem? Let’s investigate this:</p><pre># ls -lah /var/lib/docker/devicemapper/devicemapper/data<br>-rw------- 1 root root 100G Feb  9 15:30 /var/lib/docker/<br># du -sh /var/lib/docker/devicemapper/devicemapper/data<br>11M	/var/lib/docker/devicemapper/devicemapper/data<br># ls -lah /var/lib/docker/devicemapper/devicemapper/<br>total 12M<br>drwx------ 2 root root 4.0K Feb  9 15:30 .<br>drwx------ 4 root root 4.0K Feb  9 15:30 ..<br>-rw------- 1 root root 100G Feb  9 15:30 data<br>-rw------- 1 root root 2.0G Feb  9 15:30 metadata<br># stat /var/lib/docker/devicemapper/devicemapper/data<br>  File: /var/lib/docker/devicemapper/devicemapper/data<br>  Size: 107374182400	Blocks: 21720      IO Block: 4096   regular file<br>Device: 802h/2050d	Inode: 262743      Links: 1<br>Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)<br>Access: 2023-02-09 15:30:14.078725304 +0000<br>Modify: 2023-02-09 15:30:14.070725420 +0000<br>Change: 2023-02-09 15:30:14.070725420 +0000<br> Birth: -<br># du -h /var/lib/docker/devicemapper/devicemapper/data<br>11M	/var/lib/docker/devicemapper/devicemapper/data<br># du -h --apparent-size /var/lib/docker/devicemapper/devicemapper/data<br>100G	/var/lib/docker/devicemapper/devicemapper/data</pre><p>Most likely after a bit of googling you would reach <a href="https://en.wikipedia.org/wiki/Sparse_file">wiki page</a>. And yes we are dealing here with sparse file(Kerrisk, M. (2010). <em>The Linux Programming Interface</em>; p 83). In this case docker uses devicemapper as storage driver but it is deprecated at the moment. Anyway this is pretty common solution to store filesystems in files.</p><h4>So what happens here:</h4><ul><li>most of modern filesystems support <a href="https://en.wikipedia.org/wiki/Sparse_file"><strong>sparse files</strong></a>. Full list is available on <a href="https://en.wikipedia.org/wiki/Comparison_of_file_systems"><strong>wikipedia</strong></a></li><li>sparse file is a file that doesn’t consume actual blocks until data is written there.</li><li>sparse files can be found via(<a href="https://unix.stackexchange.com/a/612891"><strong>found here</strong></a>):</li></ul><pre>#  find /var/lib/docker/ -type f -printf &quot;%S\t%p\n&quot; | gawk &#39;$1 &lt; 1.0 {print}&#39;<br>0.75	/var/lib/docker/buildkit/cache.db<br>0.5	/var/lib/docker/network/files/local-kv.db<br>0.75	/var/lib/docker/volumes/metadata.db<br>0.000297546	/var/lib/docker/devicemapper/devicemapper/metadata<br>0.000103569	/var/lib/docker/devicemapper/devicemapper/data</pre><ul><li>files can be converted to sparse and back</li><li>some tools(<strong>tar</strong>, <strong>rsync</strong>) might need additional arguments to work with sparse files as you expect</li></ul><h3>Reserved for root</h3><p>Another day, another alert. Web server is not responding. Ok, let’s login and check logs:</p><pre>[Thu Feb 09 16:33:29.457029 2023] [log_config:warn] [pid 722:tid 140116340029184] (28)No space left on device: [client 10.0.2.4:51110] AH00646: Error writing to /var/log/apache2/access.log</pre><p>No disk space. Let’s run our usual test commands:</p><pre># df -h<br>Filesystem      Size  Used Avail Use% Mounted on<br>/dev/sda2       4.9G  4.6G     0 100% /</pre><p>Interesting numbers. 4.6G used, 4.9G total but usage is 100%. Where is 0.3G? We can have a closer look at filesystem itself with <a href="https://man7.org/linux/man-pages/man8/tune2fs.8.html"><strong>tune2fs</strong></a>:</p><pre># tune2fs -l /dev/sda2 | grep -i reserved<br>Reserved block count:     65497<br>Reserved GDT blocks:      639<br>Reserved blocks uid:      0 (user root)<br>Reserved blocks gid:      0 (group root)</pre><p>So we have some reserved blocks. Kerrisk, M. (2010). <em>The Linux Programming Interface</em>; page 277 says:</p><blockquote><em>Many native UNIX and Linux file systems support the notion of reserving a certain portion of the blocks of a file system for the superuser, so that if the file system fills up, the superuser can still log in to the system and do some work to resolve the problem. If there are reserved blocks in the file system, then the difference in values of the f_bfree and f_bavail fields in the statvfs structure tells us how many blocks are reserved.</em></blockquote><p>Aha, so that space is reserved to keep system running even if there is no free disk space at all. Let’s change this reservation so we could gain some time to properly fix the issue:</p><pre># tune2fs -m 2 /dev/sda2<br>tune2fs 1.45.5 (07-Jan-2020)<br>Setting reserved blocks percentage to 2% (26199 blocks)<br># tune2fs -l /dev/sda2 | grep -i reserved<br>Reserved block count:     26199<br>Reserved GDT blocks:      639<br>Reserved blocks uid:      0 (user root)<br>Reserved blocks gid:      0 (group root)<br># df -h<br>Filesystem      Size  Used Avail Use% Mounted on<br>/dev/sda2       4.9G  4.6G  154M  97% /</pre><h4>So what happens here:</h4><ul><li>many filesystems reserve disk space for root. This is protection mechanism to keep system running(and allow admin to login) when there is no available disk space</li><li>threshold can be changed. This makes some amount of blocks usable and can buy you some time. But be careful you still need to analyze what is going on and fix it properly</li><li>on big filesystems(&gt;50G-100G) reservation of 5% is excessive. You might want to check your fs and reduce amount of reserved blocks</li></ul><h3>inodes</h3><p>Server stopped responding and we should check what happened. Logs:</p><pre>[Thu Feb 09 16:43:52.539717 2023] [proxy_fcgi:error] [pid 721:tid 140116219635456] [client 10.0.2.4:55146] AH01071: Got error &#39;PHP message: PHP Warning:  Unknown: open(/home/wordpress/tmp/sess_qc0vsvd2vrvt9nbmc1iqmg74sh, O_RDWR) failed: No space left on device (28) in Unknown on line 0PHP message: PHP Warning:  Unknown: Failed to read session data: files (path: /home/wordpress/tmp) in Unknown on line 0&#39;</pre><p>Ok, standard checks for this scenario:</p><pre># df -h -x tmpfs -x squashfs<br>Filesystem      Size  Used Avail Use% Mounted on<br>/dev/sda2       4.9G  4.2G  609M  88% /<br># du -xh --max-depth=1 / | sort -h<br>4.0K	/media<br>4.0K	/mnt<br>4.0K	/srv<br>16K	    /lost+found<br>16K	    /opt<br>24K 	/snap<br>68K 	/root<br>76K	    /tmp<br>6.5M	/etc<br>82M	    /home<br>205M	/boot<br>1021M	/var<br>2.9G	/usr<br>4.2G	/<br># touch file.txt<br>touch: cannot touch &#39;file.txt&#39;: No space left on device</pre><p>We definitely have free space but something is wrong. We can check one more thing:</p><pre># df -i<br>Filesystem     Inodes  IUsed  IFree IUse% Mounted on<br>...<br>/dev/sda2      327680 327680      0  100% /</pre><blockquote><strong><em>-x tmpfs -x squashfs</em></strong><em> can be used here too</em></blockquote><p>The server used all inodes. Let’s find directories with highest inode usage:</p><pre># find / -type f | cut -d/ -f2 | sort | uniq -c<br>sort: cannot create temporary file in &#39;/tmp&#39;: No space left on device</pre><p>Hm, this was predictable. We can do a small trick:</p><pre># mount -o mode=1777,nosuid,nodev -t tmpfs tmpfs /tmp</pre><p>we mounted <a href="https://man7.org/linux/man-pages/man5/tmpfs.5.html"><strong>tmpfs</strong></a>(in-memory fs) over /tmp. This on itself can cause issues but we are already in the middle of an incident:</p><pre># find / -type f | cut -d/ -f2 | sort | uniq -c | sort<br>      2 tmp<br>      7 root<br>    298 boot<br>    477 run<br>    923 etc<br>   3987 var<br>  19062 snap<br>  46843 proc<br>  73619 sys<br>  89246 usr<br> 206762 home<br># cd /home/<br>/home# find . -type f | cut -d/ -f2 | sort | uniq -c | sort<br>      8 cloud_user<br> 206754 wordpress<br>/home/wordpress# find . -type f | cut -d/ -f2 | sort | uniq -c | sort<br>      1 .bash_history<br>      1 some.img<br>   2929 www<br> 203823 tmp<br>/home/wordpress# ls -la tmp | head<br>total 11564<br>drwxr-xr-x 2 wordpress wordpress 11829248 Feb  9 16:43 .<br>drwxr-xr-x 5 wordpress wordpress     4096 Feb  7 16:13 ..<br>-rw------- 1 wordpress wordpress        0 Feb  9 16:39 sess_0007rvedf46i5ph2djvgr26duv<br>-rw------- 1 wordpress wordpress        0 Feb  9 16:38 sess_00087653kflc413ubdauvsse8i<br>-rw------- 1 wordpress wordpress        0 Feb  9 16:41 sess_000d06rqoo6aqbfo33abs4eo7n<br>-rw------- 1 wordpress wordpress        0 Feb  9 16:39 sess_000nc4hvobl5qcqfhri5bct2og<br>-rw------- 1 wordpress wordpress        0 Feb  9 16:42 sess_0012sip5qnrh1scntllgn9q0gs<br>-rw------- 1 wordpress wordpress        0 Feb  9 16:37 sess_00188bvskjii64hf1o44r7tjdk<br>-rw------- 1 wordpress wordpress        0 Feb  9 16:42 sess_001987gv66da5bb0u1m3egcerh</pre><p>let’s clean this directory:</p><pre># rm /home/wordpress/tmp/sess*<br>-bash: /usr/bin/rm: Argument list too long</pre><p>ok, we should do it some other way. Very good compilation of advice is <a href="https://unix.stackexchange.com/a/79656"><strong>here</strong></a> and yes, Perl is one of the fastest solutions:</p><pre># cd /home/wordpress/tmp/<br>/home/wordpress/tmp# perl -e &#39;for(&lt;*&gt;){unlink}&#39;</pre><h4>So what happens here:</h4><ul><li>amount of <strong>inodes</strong> on <strong>fs</strong> is defined at the creation time. Despite the fact that this number is very high they can be exhausted</li><li>this kind of incidents occur rarely and are related to systems that create a lot of small files. It happened to me in two cases — PHP storing sessions in files(without configured cleanup) and on mail server where emails were stored in files(yep, wave of spam messages)</li></ul><h3>Read-only filesystem</h3><p>Another issue might happen due to issues with hard disk itself:</p><pre># touch test.file<br>touch: cannot touch &#39;test.file&#39;: Read-only file system<br># sudo df -h -x tmpfs -x squashfs<br>Filesystem      Size  Used Avail Use% Mounted on<br>udev            445M     0  445M   0% /dev<br>/dev/sda2       4.9G  4.3G  507M  90% /<br># df -i -x tmpfs -x squashfs<br>Filesystem     Inodes  IUsed  IFree IUse% Mounted on<br>udev           113745    466 113279    1% /dev<br>/dev/sda2      327680 123895 203785   38% /</pre><p>In this case we definitely have disk space. Another thing to check is how our filesystems are mounted:</p><pre># lsblk<br>lsblk<br>NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT<br>loop0    7:0    0 63.3M  1 loop /snap/core20/1778<br>loop1    7:1    0 63.3M  1 loop /snap/core20/1822<br>loop2    7:2    0 49.8M  1 loop /snap/snapd/17950<br>loop3    7:3    0 67.8M  1 loop /snap/lxd/22753<br>loop4    7:4    0 91.9M  1 loop /snap/lxd/24061<br>loop5    7:5    0 49.9M  1 loop /snap/snapd/18357<br>sda      8:0    0    5G  0 disk <br>├─sda1   8:1    0    1M  0 part <br>└─sda2   8:2    0    5G  0 part /<br>sr0     11:0    1 1024M  0 rom  <br># mount<br>...<br>/dev/sda2 on / type ext4 (ro,relatime)</pre><p>Finally <strong>mount</strong> gives us a hint that root filesystem is mounted in read-only(<strong>ro</strong>) mode. Here we need to check system logs to understand what happened:</p><pre># tail -n1 /var/log/syslog<br>Feb 22 16:12:45 sandbox kernel: [ 1500.489553] sysrq: Emergency Remount R/O<br># dmesg<br>[ 1500.489553] sysrq: Emergency Remount R/O<br>[ 1500.501938] EXT4-fs (sda2): re-mounted. Opts: (null)<br>[ 1500.503765] Emergency Remount complete</pre><p>We see that kernel remounted fs to read-only mode due to sysrq: Emergency Remount R/O.</p><p>In this scenario it is hard to give some general advice on how to investigate and fix this. Places to check are <strong>dmesg</strong>, <strong>syslog</strong>/<strong>messages</strong>, SMART of the harddrive(logs of RAID controller) in case of physical drive or notifications from cloud provider(possible outages).</p><h4>So what happens here:</h4><ul><li>when kernel detects issues with fs it behaves according to the <strong>errors=</strong> argument of mount.</li><li><a href="https://man7.org/linux/man-pages/man8/mount.8.html">man</a> describes corresponding arguments:</li></ul><blockquote>errors={continue|remount-ro|panic}</blockquote><blockquote><em>Define the behavior when an error is encountered. (Either ignore errors and just mark the filesystem erroneous and continue, or remount the filesystem read-only, or panic and halt the system.)</em></blockquote><ul><li>this behaviour can be set in <strong>fstab</strong> or directly during mounting.</li><li>this can be caused by issue with hardware(bad contact in data or power connector of hard drive) or in case of cloud this can be caused by issues with underlying network.</li></ul><h3>Cheatsheet</h3><pre># main comands to get overview of disks:<br>lsblk <br>df -h -x tmpfs -x squashfs<br>df -i -x tmpfs -x squashfs<br>mount<br># to calculate used space:<br>du -hx --max-depth=1 / | sort -h<br># next you can narrow your search:<br>du -hx --max-depth=1 /var | sort -h<br># remember about -x and --apparent-size keys of du<br># find biggest file:<br>find /var/lib/docker -type f -printf &quot;%s\t%p\n&quot; | sort -n | tail -1<br># find sparse files:<br>find /var/lib/docker/ -type f -printf &quot;%S\t%p\n&quot; | gawk &#39;$1 &lt; 1.0 {print}&#39;<br># find directories with highest inodes usage<br>find / -type f | cut -d/ -f2 | sort | uniq -c<br># to narrow search:<br>cd /home/<br>find . -type f | cut -d/ -f2 | sort | uniq -c | sort<br># fast way to remove A LOT of files:<br>cd /home/wordpress/tmp/<br>perl -e &#39;for(&lt;*&gt;){unlink}&#39;<br># get contents of inode:<br>stat /path/to/your/file<br># filesystem<br>tune2fs -l /dev/$DEVICE<br># a lot more about fs<br>dumpe2fs /dev/sda3 | less<br># during incident you might urgently need some space. If your system has swapfile:<br>swapoff -a<br>rm /swapfile<br># but please fix it afterwards:<br>fallocate -l 1G /swapfile<br>OR<br>dd if=/dev/zero of=/swapfile bs=1024 count=1048576<br>chmod 600 /swapfile<br>mkswap /swapfile<br>swapon -a<br>OR <br>swapon /swapfile</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b98c782b45c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>