[Lustre-discuss] Getting random "No space left on device (28)"
magawake at gmail.com
Sun Oct 12 13:19:38 PDT 2008
Thanks. I think thats the problem! Once I started to move away from
the old OST and start rebalancing everything stated to clear up!
Also, does anyone have a script or algorithm to re balance much
quicker? I prefer an algorithm so I can use rsync to re balance.
On Sun, Oct 12, 2008 at 3:36 PM, Brock Palen <brockp at umich.edu> wrote:
> On any client
> lfs df -h
> Show you all your OST usage for all your OST in one command.
> Brock Palen
> Center for Advanced Computing
> brockp at umich.edu
> On Oct 12, 2008, at 3:24 PM, Kevin Van Maren wrote:
>> Sounds like one (or more) of your existing OSTs are out of space. The
>> OSTs are assigned at file creation
>> time, and Lustre will return an error if you cannot allocate space on
>> the OST for a file you are writing.
>> Do a "df" on your OSS nodes.
>> Lustre does not re-stripe files; you may have to manually move (cp/rm)
>> some files to the new OST
>> to rebalance the file system. It is a manual process, but you can use
>> "lfs setstripe" for force a specific OST,
>> and use "lfs getstripe" to see where a file's storage is allocated.
>> Mag Gam wrote:
>>> We have recently added another 1TB to a filesystem. We added a new OST
>>> and mounted the OST. On the clients we do a lfs df -h and we see the
>>> new space has been acquired. Also, lfs df -i shows enough inodes.
>>> However, we randomly see 'No Space left on device (28)" when we run
>>> our jobs. But if we resubmit the jobs it works again.
>>> Is there anything special we need to do, after we mount up a new OST?
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
More information about the lustre-discuss