[lustre-discuss] From ceph to lustre. what is the right setup for high availability in a small cluster?

[Laura Hild]

Hi Arvid-

> This makes me wonder what lustre even adds in this scenario,
> since zfs is already doing the heavy lifting of managing
> replication and high availability.

ZFS by itself is a local filesystem which you can only mount on one host
at a time;  what Lustre adds is taking several ZFS filesystems across
multiple servers, presenting them to network clients as a single
filesystem, and providing a mechanism to allow a resource manager like
Pacemaker to change which host any particular underlying filesystem is
being served from.

> Or is it perhaps that only object _data_ replication is managed by
> lustre while object metadata is not redundant?

The FLR feature aside, Lustre's not even replicating object data, but
relying on the layers below to provide redundancy.  It just cares that
you can mount each underlying filesystem on one of that filesystem's
assigned servers, regardless whether you accomplish that with shared
storage, or via some replication layer.

> Exposing the directly attached sata drivers over iSER appears to be
> the only solution for us, but i'm very worried that going against the
> recommended setup will invite all sorts of trouble down the road.

You could also look at DRBD, which was mentioned on this list a couple
weeks ago.  I would've thought that would be slow, but apparently
at least one site has been having success with it.

-Laura