[Lustre-discuss] No space left on device on not full filesystem

[Kevin Van Maren]

Two possibilities seem likely:

1) You have run out of inodes.  Do "df -i", or "lfs df -i"

2) It could be a grant issue.  With 700GB free on each OST, unless you 
have a lot of clients, it looks like a grant leak issue.
See: https://bugzilla.lustre.org/show_bug.cgi?id=22755

Kevin



Katya Tutlyaeva wrote:
> Hello,
>
>     We have running lustre 1.8.1 and have met "No space lest on device" 
> error when uploading 500 Gb small files (less then 100 Kb each).
> The problem seems to  depends on the number of files. If we remove one 
> file, we can create one new file, even with Gb size; but if we haven't 
> remove something we can't create even very little file, as an example 
> using touch command:
> [client]$ touch newfile
> touch: cannot touch `newfile': No space left on device
> [client]$ rm  oldfile
> rm: remove regular file `qqq'? y
> [client]$ dd bs=100M count=10 < /dev/zero > qqq
> 10+0 records in
> 10+0 records out
> 1048576000 bytes (1.0 GB) copied, 15.9882 s, 65.6 MB/s
> beda:/mnt/lustre# dd bs=100M count=10 < /dev/zero > qqq.tmp
> qqq.tmp: No space left on device.
> beda:/mnt/lustre# dd bs=100M count=10 < /dev/zero > qqq
> ^C5+0 records in
> 5+0 records out
> 524288000 bytes (524 MB) copied, 7.66803 s, 68.4 MB/s
>
> But there are a lot of free space both on OSTs and MDS:
> [client]$ lfs df -h
> UUID                     bytes      Used Available  Use% Mounted on
> lustre-MDT0000_UUID      43.8G    813.4M     40.5G    1% /mnt/lustre[MDT:0]
> lustre-OST0000_UUID     867.7G     45.8G    777.8G    5% /mnt/lustre[OST:0]
> lustre-OST0001_UUID     916.9G     44.9G    825.4G    4% /mnt/lustre[OST:1]
> lustre-OST0002_UUID     916.9G     44.9G    825.5G    4% /mnt/lustre[OST:2]
> lustre-OST0003_UUID     916.9G     44.9G    825.4G    4% /mnt/lustre[OST:3]
> lustre-OST0004_UUID     916.9G     44.9G    825.4G    4% /mnt/lustre[OST:4]
> lustre-OST0005_UUID     916.9G     44.9G    825.4G    4% /mnt/lustre[OST:5]
> lustre-OST0006_UUID     916.9G     44.5G    825.9G    4% /mnt/lustre[OST:6]
> lustre-OST0007_UUID     916.9G     44.6G    825.8G    4% /mnt/lustre[OST:7]
> lustre-OST0008_UUID     916.9G     44.5G    825.8G    4% /mnt/lustre[OST:8]
>
> filesystem summary:       8.0T    403.8G      7.2T    4% /mnt/lustre
>
> There are not additional information in dmesg, no new messeges in 
> /tmp/lustre-log also.
>
> Would you be kind enough to help me, to give some advice?
> Thank you!
>
> Best regards,
> Katya
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>