[Lustre-discuss] Unable to write to filesystem (device full)

[Nick Jennings]

On Tue, 2009-09-29 at 12:40 +0200, Nick Jennings wrote:
> Hi Again,
> 
>  I guess I spoke too soon. Looks like the MDS is full, even though it
> still has plenty of space, it's reporting that it's reached it's inode
> limit. Not sure how this is fixed. I am also having a hard time finding
> information about this (I wish PDF text was searchable). 

 I had a momentary lapse and forgot I could, in fact, search PDFs :P


> # lfs df -h
> UUID                     bytes      Used Available  Use% Mounted on
> clients-MDT0000_UUID     35.0G      9.4G     23.6G   26% /data[MDT:0]
> clients-OST0000_UUID      2.0T    319.5G      1.6T   15% /data[OST:0]
> 
> filesystem summary:       2.0T    319.5G      1.6T   15% /data
> 
> # lfs df -i
> UUID                    Inodes     IUsed     IFree IUse% Mounted on
> clients-MDT0000_UUID  10485760  10409292     76468   99% /data[MDT:0]
> clients-OST0000_UUID 134217728   8082612 126135116    6% /data[OST:0]
> 
> filesystem summary:   10485760  10409292     76468   99% /data
> 
> 
> 
> I tried to grow the MDT +50G with the following:
> 
> # lvextend  -L +50G /dev/vg-localdisk/mdt 
>   Extending logical volume mdt to 90.00 GB
>   Logical volume mdt successfully resized
> 
> # resize2fs -p /dev/vg-localdisk/mdt 
> resize2fs 1.40.11.sun1 (17-June-2008)
> resize2fs: Filesystem has unsupported read-only feature(s) :-
> uninit_groups
> 

 After doing more digging it seems my only option is to migrate the MDT
to a new partition (with smaller number of bytes per inode). Now I wish
I hadn't added that 50G to the existing MDT slice, because it seems like
I can't incorporate it into the MDT filesystem, and can't remove it from
the Logical Volume (warns of data loss and I don't want to take that
chance).

 So I've got about 80G left on this server, Ideally I'd like to use both
partitions together 80G+90G for the new MDT slice, and format it using
2096 bytes for each inode. However since I can't seem to grow an MDT
filesystem I don't know how I'm going to do this...

 I found some docs on migrating and MDT to a new partition a little
while ago but now can't seem to find it. If anyone can point me in the
right direction that would be great. In the meantime I've removed as
many files as I could from the filesystem to free up inodes. 

 If I use half the default recommended setting of 2096 bytes per inode,
will performance degrade substantially?

cheers,
-nick

> Didn't seem to work. So I've got a 90G LVM slice allocated to the MDT
> which is only using 35G of that (can't get it to grow to use the rest).
> Of that 35G, 24G is still free but my inode limit is at 99% and I'm not
> sure how to extend that. 
> 
> Thanks for any help.
> -Nick
> 
> 
> On Tue, 2009-09-29 at 11:47 +0200, Nick Jennings wrote:
> > Hi Everyone,
> > 
> >  RHEL5.3 - Lustre 1.6.7.2 - filesystem mounted as /data.
> > 
> >  I've got another odd problem this morning. Suddenly the file system was
> > reporting "no space left on device" failing every write attempt. A df
> > showed still 1.7TB free. 
> > 
> >  I first tried unmounting and remounting the /data filesystem. However
> > on remount I got the error:
> > 
> > # mount /data 
> > mount.lustre: mount idbn1 at tcp1:/clients at /data failed: Cannot send
> > after transport endpoint shutdown
> > 
> >  I then realized top was reporting one of the Lustre threads on the MDT
> > at 100% CPU. So I unmounted and remounted the MDT partition on that
> > server, and could then remount the /data filesystem on the client node. 
> > 
> >  Now everything seems to be back to normal operation. Any idea what was
> > going on there? I've got absolutely nothing from any of the log files. 
> > 
> > Thanks for any help,
> > -Nick
> > 
> > 
> > -  
> > Nick Jennings
> > Technical Director
> > Creative Motion Design
> > www.creativemotiondesign.com 
> > 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20090929/f78eb1c2/attachment.pgp>