[Lustre-discuss] NUMA IO and Lustre

[Sébastien Buisson]

Hi Daniel,

Daniel Kobras a écrit :
> 
> As I recently learned the hard way, the network settings of an OST are
> fixed when the OST connects to the MGS for the first time. Hence, you
> could ifup ib0, ifdown ib1, start LNET and fire up the first set of
> OSTs, then down LNET again, ifdown ib0, ifup ib1, restart LNET and start
> up the rest of the OSTs. Subsequently, you should be able to run with
> both IB nids enabled, but clients should still only know about a single
> IB nid for each OSTs.

Your workaround sounds very interesting, because it shows that the 
network settings of an OST cannot be fixed by any configuration 
directive: this is done in an opaque way at mkfs.lustre time.


> As for a less hackish solution, wouldn't it be
> cleaner to just run, say, two Xen domUs on the machine and map the
> HBAs/HCAs as appropriate?

Virtualization is the solution we were considering in case of negative 
answer to our question about bindings between OSTs and network 
interfaces. :)
If we are able to isolate appropriate HBAs/HCAs in separate VMs, that 
could do the trick. We would have a preference for kvm, but the 
principle is the same.
Of course this is a bit cumbersome, because each node will be seen as 2 
different Lustre servers (OSS), but that seems to be the only available 
solution at the moment.

Thanks.
Sebastien.