Linux中出现“No space left on device”错误的排查与解决方法
<p><span><strong>前言</strong></span></p>
<p>
最近这两天登陆服务器，发现用 wget 下载文件的时候提示“No space left on device”，而且连使用 tab 键进行补全时也会提示该错误。</p>
<p>
之前遇到过一次这种问题，是由于磁盘空间被占满了，导致无法创建新文件。正常情况下，删除一些文件来释放空间，即可解决该问题。</p>
<p>
<span><strong>当我使用 df 命令查看分区情况时，结果如下：</strong></span></p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_92001">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash comments"># df -h</code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">Filesystem Size Used Avail Use% Mounted on</code>
</div>
<div class="line number3 index2 alt2">
<code class="bash plain">/dev/vda1</code> <code class="bash plain">29G 29G 0 100% /</code>
</div>
<div class="line number4 index3 alt1">
<code class="bash plain">udev 10M 0 10M 0% </code><code class="bash plain">/dev</code>
</div>
<div class="line number5 index4 alt2">
<code class="bash plain">tmpfs 101M 232K 100M 1% </code><code class="bash plain">/run</code>
</div>
<div class="line number6 index5 alt1">
<code class="bash plain">tmpfs 5.0M 0 5.0M 0% </code><code class="bash plain">/run/lock</code>
</div>
<div class="line number7 index6 alt2">
<code class="bash plain">tmpfs 405M 0 405M 0% </code><code class="bash plain">/run/shm</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
看到这里，我以为磁盘真的被 100% 占用了，于是就查看了各目录的占用情况：</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_749996">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
<div class="line number15 index14 alt2">
15</div>
<div class="line number16 index15 alt1">
16</div>
<div class="line number17 index16 alt2">
17</div>
<div class="line number18 index17 alt1">
18</div>
<div class="line number19 index18 alt2">
19</div>
<div class="line number20 index19 alt1">
20</div>
<div class="line number21 index20 alt2">
21</div>
<div class="line number22 index21 alt1">
22</div>
<div class="line number23 index22 alt2">
23</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash comments"># du -sh /*</code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">8.8M </code><code class="bash plain">/bin</code>
</div>
<div class="line number3 index2 alt2">
<code class="bash plain">30M </code><code class="bash plain">/boot</code>
</div>
<div class="line number4 index3 alt1">
<code class="bash plain">0 </code><code class="bash plain">/dev</code>
</div>
<div class="line number5 index4 alt2">
<code class="bash plain">5.3M </code><code class="bash plain">/etc</code>
</div>
<div class="line number6 index5 alt1">
<code class="bash plain">24K </code><code class="bash plain">/home</code>
</div>
<div class="line number7 index6 alt2">
<code class="bash plain">0 </code><code class="bash plain">/initrd</code><code class="bash plain">.img</code>
</div>
<div class="line number8 index7 alt1">
<code class="bash plain">205M </code><code class="bash plain">/lib</code>
</div>
<div class="line number9 index8 alt2">
<code class="bash plain">4.0K </code><code class="bash plain">/lib64</code>
</div>
<div class="line number10 index9 alt1">
<code class="bash plain">16K </code><code class="bash plain">/lost</code><code class="bash plain">+found</code>
</div>
<div class="line number11 index10 alt2">
<code class="bash plain">8.0K </code><code class="bash plain">/media</code>
</div>
<div class="line number12 index11 alt1">
<code class="bash plain">4.0K </code><code class="bash plain">/mnt</code>
</div>
<div class="line number13 index12 alt2">
<code class="bash plain">4.0K </code><code class="bash plain">/opt</code>
</div>
<div class="line number14 index13 alt1">
<code class="bash plain">0 </code><code class="bash plain">/proc</code>
</div>
<div class="line number15 index14 alt2">
<code class="bash plain">2.5G </code><code class="bash plain">/root</code>
</div>
<div class="line number16 index15 alt1">
<code class="bash plain">232K </code><code class="bash plain">/run</code>
</div>
<div class="line number17 index16 alt2">
<code class="bash plain">5.2M </code><code class="bash plain">/sbin</code>
</div>
<div class="line number18 index17 alt1">
<code class="bash plain">8.0K </code><code class="bash plain">/srv</code>
</div>
<div class="line number19 index18 alt2">
<code class="bash plain">0 </code><code class="bash plain">/sys</code>
</div>
<div class="line number20 index19 alt1">
<code class="bash plain">4.0K </code><code class="bash plain">/tmp</code>
</div>
<div class="line number21 index20 alt2">
<code class="bash plain">2.6G </code><code class="bash plain">/usr</code>
</div>
<div class="line number22 index21 alt1">
<code class="bash plain">1.8G </code><code class="bash plain">/var</code>
</div>
<div class="line number23 index22 alt2">
<code class="bash plain">0 </code><code class="bash plain">/vmlinuz</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
很明显，总共的磁盘占用完全不到 10G，磁盘理论上并未被占满。</p>
<p>
这种情况下，很可能是小文件过多，导致 inode 急剧增加，消耗完 inode 区域的空间。</p>
<p>
如果真是如此，那么即使磁盘空间有剩余，但由于无法创建新的 inode 来存储文件的元信息，也就无法创建新文件。</p>
<p>
<span><strong>因此，我用 df 命令进行验证：</strong></span></p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_472197">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash comments"># df -ih</code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">Filesystem Inodes IUsed IFree IUse% Mounted on</code>
</div>
<div class="line number3 index2 alt2">
<code class="bash plain">/dev/vda1</code> <code class="bash plain">1.9M 299K 1.6M 17% /</code>
</div>
<div class="line number4 index3 alt1">
<code class="bash plain">udev 123K 299 123K 1% </code><code class="bash plain">/dev</code>
</div>
<div class="line number5 index4 alt2">
<code class="bash plain">tmpfs 126K 249 125K 1% </code><code class="bash plain">/run</code>
</div>
<div class="line number6 index5 alt1">
<code class="bash plain">tmpfs 126K 4 126K 1% </code><code class="bash plain">/run/lock</code>
</div>
<div class="line number7 index6 alt2">
<code class="bash plain">tmpfs 126K 2 126K 1% </code><code class="bash plain">/run/shm</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
可以看到，inode 区域只被占用了一小部分，还有大量的空间未使用，所以也不是 inode 区域被占满的问题。</p>
<p>
到了这里，我的内心是非常郁闷的。这个问题直接导致了Apache、MySql以及其它的一些服务均无法启动，服务器已经基本没法使用了，然而原因却扑朔迷离。</p>
<p>
最后，服务器提供商告诉我另一种可能的情况，就是有些文件删除时还被其它进程占用，此时文件并未真正删除，只是标记为 deleted，只有进程结束后才会将文件真正从磁盘中清除。</p>
<p>
<span><strong>于是我通过 lsop 命令查看了被进程占用中的文件：</strong></span></p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_335066">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash comments"># lsof | grep deleted</code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">mysqld 1952 2982 mysql 5u REG 254,1 0 127 </code><code class="bash plain">/tmp/ibzMEe4z</code> <code class="bash plain">(deleted)</code>
</div>
<div class="line number3 index2 alt2">
<code class="bash plain">mysqld 1952 2982 mysql 6u REG 254,1 0 146 </code><code class="bash plain">/tmp/ibq6ZFge</code> <code class="bash plain">(deleted)</code>
</div>
<div class="line number4 index3 alt1">
<code class="bash plain">mysqld 1952 2982 mysql 10u REG 254,1 0 150 </code><code class="bash plain">/tmp/ibyNHH8y</code> <code class="bash plain">(deleted)</code>
</div>
<div class="line number5 index4 alt2">
<code class="bash plain">apache2 2869 root 9u REG 254,1 0 168 </code><code class="bash plain">/tmp/</code><code class="bash plain">.ZendSem.2w14iv (deleted)</code>
</div>
<div class="line number6 index5 alt1">
<code class="bash plain">apache2 2869 root 10w REG 0,16 0 11077 </code><code class="bash plain">/run/lock/apache2/rewrite-map</code><code class="bash plain">.2869 (deleted)</code>
</div>
<div class="line number7 index6 alt2">
<code class="bash plain">...</code>
</div>
<div class="line number8 index7 alt1">
<code class="bash plain">python 3102 root 1w REG 254,1 22412342132 264070 </code><code class="bash plain">/var/log/nohup</code><code class="bash plain">.out (deleted)</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
终于找到了罪魁祸首，原来是在后台运行的 Python 脚本，源源不断地将输出保存到 /var/log/nohup.out 文件中，文件大小居然达到了20G+！</p>
<p>
前阶段在后台运行脚本之后，就没再管过它。估计是我在 Python 运行过程中删掉了 nothup.out 文件，由于该文件被占用，所以只能先标记为 deleted，而未真正删除，最后导致磁盘爆满。</p>
<p>
这次的教训提醒了我，不能将任务简单放到后台就放任不管，特别是使用 nohup 命令时，所有的输出都会被不断地添加到同一个文件中，如果该进程不会自己终止，就可能导致输出文件占满整个磁盘。</p>
<p>
<span><strong>总结</strong></span></p>
<p>
以上就是这篇文章的全部内容了，希望本文的内容对大家的学习或者工作能带来一定的帮助，如果有疑问大家可以留言交流，谢谢大家对的支持。</p>
<p>
原文链接：http://www.codebelief.com/article/2017/09/linux-no-space-left-on-device-error-investigation</p>
頁:
[1]