[Lustre-discuss] MGS - one per site

[David Dillow]

On Fri, 2009-06-26 at 23:17 +0200, Andreas Dilger wrote:
> On Jun 26, 2009  16:34 -0400, David Dillow wrote:
> > On Fri, 2009-06-26 at 22:05 +0200, Andreas Dilger wrote:
> > > In some cases there will of course be a need for multiple MGSes in
> > > a single site (e.g. secure and open networks), which is fine as long
> > > as clients don't try to mount from multiple MGSes at once.
> > 
> > What kind of problems could this cause?
> > 
> > We've had this configuration on both segments of Jaguar for some time
> > now with no ill effects that we could attribute to mounting from two
> > MGSes at the same time -- well, maybe that is not entirely true, as one
> > is an MGS, and the other is a combined MGS/MDS. Some clients are talking
> > with up to five separate combined MGS/MDS.
> 
> Well, there is only ever a single MGC configured on a client at one time,
> so if you have multiple MGSes I would suspect that this will cause the
> clients to be evicted from all but the last MGS, and as a result any config
> changes made to the first-mounted filesystems will not be seen by the
> clients.  This might not be noticable until you make a config change and
> half of the clients don't notice e.g. the new OST or similar.

This may explain an issue we've seen about config changes not
propagating, but that occurred on the last filesystem mounted, so I'm
not sure.

Are there any plans to eliminate this restriction? Data transfer nodes
for gridftp want to be able to mount separate Lustre filesystems from
different compute resources, which quite often have a MGS for each
filesystem, especially if they are Cray systems that have seen upgrades
from Lustre 1.4.

Is it possible to split an MGT from a combined MGT/MDT, and then combine
those into one MGT? That would be a migration path for those systems
with multiple filesystems -- much better than a complete reformat,
anyways.

It would still force sites with mulitple systems that used to have
internal MGS service to have all but one with an external MGS now.
-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office