[Lustre-discuss] Failover Setup MDS/MDT

[Bernd Schubert]

On Friday 20 June 2008 16:08:23 Brian J. Murrell wrote:
> On Fri, 2008-06-20 at 16:01 +0200, Bernd Schubert wrote:
> > We do it for several lustre installations and it works fine.
>
> Have you done any "intensive" failover testing of it?  I'm thinking
> something along the lines of our Hendrix/CMD test 11 or 17.  In those
> tests we had to survive a constant stream of failovers at something like
> 3 or 5 minute intervals for 24 hours.  So yes, a hundred or two
> failovers in a row and no application (i.e. userspace) errors.

I don't think we did these tests yet, but I could put it onto my TODO list, if 
you think it is important. So far drbd always perfectly did its job and never 
was an issue here (in contrary of the many many hardware problems we often 
have).

>
> Seeing as we know Lustre can do it (we completed that contract) I'd be
> more interested of course in seeing DRBD survive that kind of torture.
> I'm not sure how (long) DRBD takes to come back into fully mirrored
> status when one node is powered off though.  If it's a long time, that
> in itself is an exposure to failure that shared storage doesn't suffer.

Since drbd-0.7 a journaled/bitmapped raid1 is used, so drbd always only needs 
to sync the parts of its extents, which have been modified. So usually a 
resync takes a few seconds. If for some reason a full resync is required, 
this can take much longer of course, but mostly this is only done for initial 
sync or if for some reason a split brain happened (shouldn't be an issue if 
heartbeat + stonitdh is used).

Cheers,
Bernd


-- 
Bernd Schubert
Q-Leap Networks GmbH