Hi all, we are running lustre 1.8.1 on about 800 clients. Some days ago, we found a weird problem several times. When writing data, some clients reported "LustreError: 11-0: an error occurred while communicating with xx.xx.xx.xx@o2ib. The ost_write operation failed with -28", where xx.xx.xx.xx is one of our OSS node. But both the MDS and OSS is not anywhere near full. As we know, the errno 28 is "no space left on device". After some time, everything appears to be ok again.<br>
<br>the space used:<br><br> # lfs df -h
<br>UUID                     bytes      Used Available  Use% Mounted on
<br>lustre-MDT0000_UUID      350.0G      1.1G    328.9G    0% /home[MDT:0]
<br>lustre-OST0000_UUID        6.2T    263.5G      5.6T    4% /home[OST:0]
<br>lustre-OST0001_UUID        6.2T    264.3G      5.6T    4% /home[OST:1]
<br>lustre-OST0002_UUID        5.7T    261.7G      5.2T    4% /home[OST:2]
<br>lustre-OST0003_UUID        5.4T    206.6G      4.9T    3% /home[OST:3]
<br>lustre-OST0004_UUID        4.6T    197.1G      4.2T    4% /home[OST:4]
<br>lustre-OST0005_UUID        4.6T    160.6G      4.2T    3% /home[OST:5]
<br>lustre-OST0006_UUID        4.6T    300.7G      4.1T    6% /home[OST:6]
<br>lustre-OST0007_UUID        4.6T    174.1G      4.2T    3% /home[OST:7]
<br>lustre-OST0008_UUID        6.9T    232.7G      6.4T    3% /home[OST:8]
<br>lustre-OST0009_UUID        6.9T    237.7G      6.4T    3% /home[OST:9]
<br>lustre-OST000a_UUID        6.2T    219.9G      5.6T    3% /home[OST:10]
<br>lustre-OST000b_UUID        6.2T    257.8G      5.6T    4% /home[OST:11]
<br>lustre-OST000c_UUID        6.2T    784.6G      5.1T   12% /home[OST:12]
<br>lustre-OST000d_UUID        6.2T    227.2G      5.6T    3% /home[OST:13]
<br>lustre-OST000e_UUID        5.7T    199.2G      5.2T    3% /home[OST:14]
<br>lustre-OST000f_UUID        5.4T    221.9G      4.9T    4% /home[OST:15]
<br>lustre-OST0010_UUID        4.6T    176.4G      4.2T    3% /home[OST:16]
<br>lustre-OST0011_UUID        4.6T    160.9G      4.2T    3% /home[OST:17]
<br>lustre-OST0012_UUID        3.1T    118.3G      2.8T    3% /home[OST:18]
<br>lustre-OST0013_UUID        3.1T     99.0G      2.8T    3% /home[OST:19]
<br>lustre-OST0014_UUID        6.9T    243.7G      6.4T    3% /home[OST:20]
<br>lustre-OST0015_UUID        6.9T    273.6G      6.3T    3% /home[OST:21]
<br>lustre-OST0016_UUID        6.2T    335.6G      5.5T    5% /home[OST:22]
<br>lustre-OST0017_UUID        6.2T    219.1G      5.6T    3% /home[OST:23]
<br> <br>the inode used:<br><br> # lfs df -ih
<br>UUID                    Inodes     IUsed     IFree IUse% Mounted on
<br>lustre-MDT0000_UUID       89.3M      2.1M     87.2M    2% /home[MDT:0]
<br>lustre-OST0000_UUID        6.2M     91.3K      6.1M    1% /home[OST:0]
<br>lustre-OST0001_UUID        6.2M     90.7K      6.1M    1% /home[OST:1]
<br>lustre-OST0002_UUID        5.7M     83.1K      5.6M    1% /home[OST:2]
<br>lustre-OST0003_UUID        5.4M     80.1K      5.3M    1% /home[OST:3]
<br>lustre-OST0004_UUID        4.6M     68.8K      4.6M    1% /home[OST:4]
<br>lustre-OST0005_UUID        4.6M     69.8K      4.6M    1% /home[OST:5]
<br>lustre-OST0006_UUID        4.6M     69.4K      4.6M    1% /home[OST:6]
<br>lustre-OST0007_UUID        4.6M     69.2K      4.6M    1% /home[OST:7]
<br>lustre-OST0008_UUID        6.9M    101.7K      6.8M    1% /home[OST:8]
<br>lustre-OST0009_UUID        6.9M    101.3K      6.8M    1% /home[OST:9]
<br>lustre-OST000a_UUID        6.2M     91.0K      6.1M    1% /home[OST:10]
<br>lustre-OST000b_UUID        6.2M     90.9K      6.1M    1% /home[OST:11]
<br>lustre-OST000c_UUID        6.2M     86.1K      6.1M    1% /home[OST:12]
<br>lustre-OST000d_UUID        6.2M     90.2K      6.1M    1% /home[OST:13]
<br>lustre-OST000e_UUID        5.7M     83.4K      5.6M    1% /home[OST:14]
<br>lustre-OST000f_UUID        5.4M     80.4K      5.3M    1% /home[OST:15]
<br>lustre-OST0010_UUID        4.6M     69.0K      4.6M    1% /home[OST:16]
<br>lustre-OST0011_UUID        4.6M     69.3K      4.6M    1% /home[OST:17]
<br>lustre-OST0012_UUID        3.1M     46.7K      3.0M    1% /home[OST:18]
<br>lustre-OST0013_UUID        3.1M     46.6K      3.0M    1% /home[OST:19]
<br>lustre-OST0014_UUID        6.9M    101.5K      6.8M    1% /home[OST:20]
<br>lustre-OST0015_UUID        6.9M    101.3K      6.8M    1% /home[OST:21]
<br>lustre-OST0016_UUID        6.2M     90.1K      6.1M    1% /home[OST:22]
<br>lustre-OST0017_UUID        6.2M     90.7K      6.1M    1% /home[OST:23]
<br><br>So why it happened? BTW, OSS and MDS didn't say anything about "no space left" in their logs, the os is SLES 10 sp2.<br>