[Lustre-discuss] NUMA IO and Lustre

[Daniel Kobras]

Hi!

On Tue, May 12, 2009 at 04:07:51PM +0200, Sébastien Buisson wrote:
> Concretely, we would like to know if it is possible in Lustre to bind an 
> OST to a specific network interface, so that this OST is only reached 
> through this interface (thus avoiding the NUMA IO factor in our case) ? 
> For instance, we would like to have 4 OSTs attached to ib0 and the 4 
> other OSTs attached to ib1.

As I recently learned the hard way, the network settings of an OST are
fixed when the OST connects to the MGS for the first time. Hence, you
could ifup ib0, ifdown ib1, start LNET and fire up the first set of
OSTs, then down LNET again, ifdown ib0, ifup ib1, restart LNET and start
up the rest of the OSTs. Subsequently, you should be able to run with
both IB nids enabled, but clients should still only know about a single
IB nid for each OSTs. As for a less hackish solution, wouldn't it be
cleaner to just run, say, two Xen domUs on the machine and map the
HBAs/HCAs as appropriate?

Regards,

Daniel.