[Lustre-discuss] IB storage as an OST target

[Ken Hornstein]

>Anybody had any experience using an IB based storage
>target as an OST?

We do that.

>Apart from the obvious issue of separating the IB SAN(SRP/SER)
>storage traffic from the Lustre traffic are there any issues?

We don't actually separate the IB traffic from the Lustre traffic; in
our cases they actually run over the same IB HCAs.  That isn't the
setup I would have chosen, but it was the system that was available.

Here is one implementation detail that stands out in my mind.  Because
the IB storage tends to come on line rather late in the boot process,
we had to develop a custom boot script that waits around for the IB
device nodes to appear before attempting to mount the Lustre
filesystems.  That was a bit of a pain until we had it all worked out.

As other as pointed out, if your backend storage disappears (which
happens more often than I would prefer, but in our case the issues
which caused that have been resolved for the most part) then that makes
Lustre very unhappy very quickly.  We've been able to recover from
those situations, but it can be a royal pain.

>What about failover?

We use MMP as others have mentioned, but we don't actually have the Lustre
failover stuff all up and running; mostly it hasn't been an issue for us,
so we haven't seen a need to finish it.

--Ken