[lustre-discuss] MDT 100% full

[Dilger, Andreas]

There isn't Data-on-MDT (DoM) functionality available yet, so there is no data being stored on the MDT.  With DNE1 there is no automatic distribution of files between MDTs, so unless you have explicitly created subdirectories on the second MDT with "lfs mkdir -i 1" there will not be any files created there.  DNE1 is intended to create some high-level directories (e.g. per user or per project) on the second MDT, and files created in those directories will be created on the second MDT.

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel High Performance Data Division

On 2016/07/27, 17:53, "lustre-discuss on behalf of Andrus, Brian Contractor" <lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lists.lustre.org> on behalf of bdandrus at nps.edu<mailto:bdandrus at nps.edu>> wrote:

Rick,

Thanks for the info. I added more to the ZFS pool and it was immediately available and all was happy.

Now I need to figure out how/why the MDT got so full.
I have 2 MDTs and the other was barely used. Both were 100GB, which I would have thought was fine. I suspect I have a user that put many small files on the filesystem and they all landed on the MDT for storage.

Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238


From: Rick Wagner [mailto:rpwagner at sdsc.edu]
Sent: Tuesday, July 26, 2016 5:58 PM
To: Andrus, Brian Contractor <bdandrus at nps.edu>
Cc: lustre-discuss at lists.lustre.org; Cai, Haisong <cai at sdsc.edu>
Subject: Re: [lustre-discuss] MDT 100% full

Hi Brian,

On Jul 26, 2016, at 5:45 PM, Andrus, Brian Contractor <bdandrus at nps.edu<mailto:bdandrus at nps.edu>> wrote:
All,

Ok, I thought 100GB would be sufficient for an MDT.
I have 2 MDTs as well, BUT…

MDT0 is 100% full and now I cannot write anything to my lustre filesystem.
The MDT is on a ZFS backing filesystem.

So, what is the proper way to grow my MDT using ZFS? Do I need to shut the filesystem down completely? Can I just add a disk or space to the pool and Lustre will see it?

Any advice or direction is appreciated.

We just did this successfully on the two MDTs backing one of our Lustre file systems and everything happened at the ZFS layer. We added drives to the pool and Lustre immediately saw the additional capacity. Whether you take down the file system or do it live is a question of your architecture, skills, and confidence. Having test file system is also worthwhile to go over the steps.

--Rick






Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238


_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20160728/f487dd4c/attachment-0001.htm>