[Lustre-discuss] IB storage as an OST target

[Ashley Pittman]

On 28 Mar 2011, at 06:21, Brian O'Connor wrote:
> Anybody had any experience using an IB based storage
> target as an OST?

We do this all the time at DDN.

> Apart from the obvious issue of separating the IB SAN(SRP/SER)
> storage traffic from the Lustre traffic are there any issues?

Personally I don't see a huge problem sharing traffic, some people like to have different networks for MPI and Lustre but I don't see real advantage in this either.

What is important about using IB devices is what happens in case of failure, if your IB network has problems and the Lustre clients can't talk to the servers they get evicted and have to re-connect.  Inconvenient perhaps but not usually a major issue.  If however your back-end devices get detached from your Lustre servers then this is very different as the on-disk data is possibly stale or inconsistent.  People will tell you the on-disk design is such that any corruption caused by sudden loss of access is minimal however as this typically happens across many OSTs simultaneously you increase the risk.  I would estimate around 5-10% of devices which see a disconnect in this way require a fsck before they will mount however when you take into account the number of OSTs you can see that for even a very small number of OSTs it becomes the norm rather than the exception.

As such I would pay attention to this when designing your network, does your SRP traffic need to go over the core switch or can you isolate it at all?  Ideally direct-connect the Lustre servers to whatever is serving the SRP targets.  If you have a small number of servers and need say 10 ports to connect them all to the backend storage don't be tempted to use 10 ports on your large site switch as this will be a lot more error prone than using a small, dedicated switch.

> What about failover?

Make sure that MMP is enabled, it will be if you specify more than one failnode when you format the OST.

Ashley.