[Lustre-discuss] Unable to write to filesystem (device full)

[Nick Jennings]

Hi Again,

 I guess I spoke too soon. Looks like the MDS is full, even though it
still has plenty of space, it's reporting that it's reached it's inode
limit. Not sure how this is fixed. I am also having a hard time finding
information about this (I wish PDF text was searchable). 

# lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
clients-MDT0000_UUID     35.0G      9.4G     23.6G   26% /data[MDT:0]
clients-OST0000_UUID      2.0T    319.5G      1.6T   15% /data[OST:0]

filesystem summary:       2.0T    319.5G      1.6T   15% /data

# lfs df -i
UUID                    Inodes     IUsed     IFree IUse% Mounted on
clients-MDT0000_UUID  10485760  10409292     76468   99% /data[MDT:0]
clients-OST0000_UUID 134217728   8082612 126135116    6% /data[OST:0]

filesystem summary:   10485760  10409292     76468   99% /data



I tried to grow the MDT +50G with the following:

# lvextend  -L +50G /dev/vg-localdisk/mdt 
  Extending logical volume mdt to 90.00 GB
  Logical volume mdt successfully resized

# resize2fs -p /dev/vg-localdisk/mdt 
resize2fs 1.40.11.sun1 (17-June-2008)
resize2fs: Filesystem has unsupported read-only feature(s) :-
uninit_groups


Didn't seem to work. So I've got a 90G LVM slice allocated to the MDT
which is only using 35G of that (can't get it to grow to use the rest).
Of that 35G, 24G is still free but my inode limit is at 99% and I'm not
sure how to extend that. 

Thanks for any help.
-Nick


On Tue, 2009-09-29 at 11:47 +0200, Nick Jennings wrote:
> Hi Everyone,
> 
>  RHEL5.3 - Lustre 1.6.7.2 - filesystem mounted as /data.
> 
>  I've got another odd problem this morning. Suddenly the file system was
> reporting "no space left on device" failing every write attempt. A df
> showed still 1.7TB free. 
> 
>  I first tried unmounting and remounting the /data filesystem. However
> on remount I got the error:
> 
> # mount /data 
> mount.lustre: mount idbn1 at tcp1:/clients at /data failed: Cannot send
> after transport endpoint shutdown
> 
>  I then realized top was reporting one of the Lustre threads on the MDT
> at 100% CPU. So I unmounted and remounted the MDT partition on that
> server, and could then remount the /data filesystem on the client node. 
> 
>  Now everything seems to be back to normal operation. Any idea what was
> going on there? I've got absolutely nothing from any of the log files. 
> 
> Thanks for any help,
> -Nick
> 
> 
> -  
> Nick Jennings
> Technical Director
> Creative Motion Design
> www.creativemotiondesign.com 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20090929/5bd9ac00/attachment.pgp>