[Lustre-discuss] No space left on device on not full filesystem

[Wojciech Turek]

Hi,

Looks like your file system is running out of inodes. Please do lfs df -i on
client and copy the output here.

best regards,

Wojciech

On 8 July 2010 13:47, Katya Tutlyaeva <ordi at xgl.pereslavl.ru> wrote:

> Hello,
>
>    We have running lustre 1.8.1 and have met "No space lest on device"
> error when uploading 500 Gb small files (less then 100 Kb each).
> The problem seems to  depends on the number of files. If we remove one
> file, we can create one new file, even with Gb size; but if we haven't
> remove something we can't create even very little file, as an example
> using touch command:
> [client]$ touch newfile
> touch: cannot touch `newfile': No space left on device
> [client]$ rm  oldfile
> rm: remove regular file `qqq'? y
> [client]$ dd bs=100M count=10 < /dev/zero > qqq
> 10+0 records in
> 10+0 records out
> 1048576000 bytes (1.0 GB) copied, 15.9882 s, 65.6 MB/s
> beda:/mnt/lustre# dd bs=100M count=10 < /dev/zero > qqq.tmp
> qqq.tmp: No space left on device.
> beda:/mnt/lustre# dd bs=100M count=10 < /dev/zero > qqq
> ^C5+0 records in
> 5+0 records out
> 524288000 bytes (524 MB) copied, 7.66803 s, 68.4 MB/s
>
> But there are a lot of free space both on OSTs and MDS:
> [client]$ lfs df -h
> UUID                     bytes      Used Available  Use% Mounted on
> lustre-MDT0000_UUID      43.8G    813.4M     40.5G    1% /mnt/lustre[MDT:0]
> lustre-OST0000_UUID     867.7G     45.8G    777.8G    5% /mnt/lustre[OST:0]
> lustre-OST0001_UUID     916.9G     44.9G    825.4G    4% /mnt/lustre[OST:1]
> lustre-OST0002_UUID     916.9G     44.9G    825.5G    4% /mnt/lustre[OST:2]
> lustre-OST0003_UUID     916.9G     44.9G    825.4G    4% /mnt/lustre[OST:3]
> lustre-OST0004_UUID     916.9G     44.9G    825.4G    4% /mnt/lustre[OST:4]
> lustre-OST0005_UUID     916.9G     44.9G    825.4G    4% /mnt/lustre[OST:5]
> lustre-OST0006_UUID     916.9G     44.5G    825.9G    4% /mnt/lustre[OST:6]
> lustre-OST0007_UUID     916.9G     44.6G    825.8G    4% /mnt/lustre[OST:7]
> lustre-OST0008_UUID     916.9G     44.5G    825.8G    4% /mnt/lustre[OST:8]
>
> filesystem summary:       8.0T    403.8G      7.2T    4% /mnt/lustre
>
> There are not additional information in dmesg, no new messeges in
> /tmp/lustre-log also.
>
> Would you be kind enough to help me, to give some advice?
> Thank you!
>
> Best regards,
> Katya
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>



-- 
--
Wojciech Turek

Assistant System Manager

High Performance Computing Service
University of Cambridge
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100708/305e0d2f/attachment.htm>