[Lustre-discuss] No space left on device on not full filesystem

[Katya Tutlyaeva]

Hello,

    We have running lustre 1.8.1 and have met "No space lest on device" 
error when uploading 500 Gb small files (less then 100 Kb each).
The problem seems to  depends on the number of files. If we remove one 
file, we can create one new file, even with Gb size; but if we haven't 
remove something we can't create even very little file, as an example 
using touch command:
[client]$ touch newfile
touch: cannot touch `newfile': No space left on device
[client]$ rm  oldfile
rm: remove regular file `qqq'? y
[client]$ dd bs=100M count=10 < /dev/zero > qqq
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 15.9882 s, 65.6 MB/s
beda:/mnt/lustre# dd bs=100M count=10 < /dev/zero > qqq.tmp
qqq.tmp: No space left on device.
beda:/mnt/lustre# dd bs=100M count=10 < /dev/zero > qqq
^C5+0 records in
5+0 records out
524288000 bytes (524 MB) copied, 7.66803 s, 68.4 MB/s

But there are a lot of free space both on OSTs and MDS:
[client]$ lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
lustre-MDT0000_UUID      43.8G    813.4M     40.5G    1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID     867.7G     45.8G    777.8G    5% /mnt/lustre[OST:0]
lustre-OST0001_UUID     916.9G     44.9G    825.4G    4% /mnt/lustre[OST:1]
lustre-OST0002_UUID     916.9G     44.9G    825.5G    4% /mnt/lustre[OST:2]
lustre-OST0003_UUID     916.9G     44.9G    825.4G    4% /mnt/lustre[OST:3]
lustre-OST0004_UUID     916.9G     44.9G    825.4G    4% /mnt/lustre[OST:4]
lustre-OST0005_UUID     916.9G     44.9G    825.4G    4% /mnt/lustre[OST:5]
lustre-OST0006_UUID     916.9G     44.5G    825.9G    4% /mnt/lustre[OST:6]
lustre-OST0007_UUID     916.9G     44.6G    825.8G    4% /mnt/lustre[OST:7]
lustre-OST0008_UUID     916.9G     44.5G    825.8G    4% /mnt/lustre[OST:8]

filesystem summary:       8.0T    403.8G      7.2T    4% /mnt/lustre

There are not additional information in dmesg, no new messeges in 
/tmp/lustre-log also.

Would you be kind enough to help me, to give some advice?
Thank you!

Best regards,
Katya