[lustre-discuss] Experience with resizing MDT

[Mohr Jr, Richard Frank (Rick Mohr)]

> On Sep 19, 2018, at 8:09 PM, Colin Faber <cfaber at gmail.com> wrote:
> 
> Why wouldn't you use DNE?

I am considering it as an option, but there appear to be some potential drawbacks.

If I use DNE1, then I have to manually create directories on specific MDTs.  I will need to monitor MDT usage and make adjustments as necessary (which is not the end of the world, but still involves some additional work).  This might be fine when I am creating new top-level directories for new users/projects, but any existing directories created before we add a new MDT will still only use MDT0.  Since the bulk of our user/project directories will be created early on, we still have the potential issue of running out of inodes on MDT0.  

Based on that, I think DNE2 would be the better alternative, but it still has similar limitations.  The directories created initially will still be only striped over a single MDT.  When another MDT is added, I would need to recursively adjust all the existing directories to have a stripe count of 2 (or risk having MDT0 run out of inodes).  Based on my understanding of how the striped directories work, all the files in a striped directory are about evenly split across all the MDTs that the directory is striped across (which doesn’t work very well if MDT0 is mostly full and MDT1 is mostly empty).  Most likely we would want to have every directory striped across all MDTs, but there is a note in the lustre manual explicitly mentioning that it’s not a good idea to do this.

So that is why I was thinking that resizing the MDT might be the simplest approach.   Of course, I might be mistunderstanding something about DNE2, and if that is the case, someone can correct me.  Of if there are options I am not considering, I would welcome those too.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu