[Lustre-discuss] IB storage as an OST target

[Jason Hill]

On Mon, Mar 28, 2011 at 09:36:43AM -0400, Jason Hill wrote:
> Hi Brian,
> 
> Here at ORNL we don't sepearate the IB SAN from the Lustre fabric

I misspoke yesterday. Our OSSes have a dual-port card used for storage, and a 
single port card used for LNET. The storage has half of its ports connected to 
the in-rack switch, and the other half are directly connected to the OSS via 
one port on the dual port card, with the other going to the switch. This is a 
legacy of risk mitigation during acceptance, but as the storage traffic is local 
it does not traverse the uplinks to the core network, so it does not 
congest/interfere with the LNET traffic.

As others have said the things you have to be aware of are what failure modes 
in your network affect your connections to the storage and if any of the MPI 
traffic from a compute cluster could interfere with your connections to the 
storage as well. I would definately suggest putting your LNET and SRP 
connections on different physical HCA's to keep the traffic at least isolated 
on the OSS side. 

-- 
-Jason