[Lustre-discuss] LNET over multiple NICs

[Alexander Oltu]

On Mon, 07 Jan 2013 14:45:02 +0100
Sébastien Buisson wrote:

> 
> Le 19/12/2012 23:03, Alexander Oltu a écrit :
> > Looks like --network option is not helping. I have tried to unmount
> > osts and run  for half of OSTs:
> >
> > tunefs.lustre --network=tcp0
> >
> > and for another half:
> >
> > tunefs.lustre --network=tcp1
> >
> > Mounted OSTs back. The client still sends all traffic through tcp0.
> 
> Maybe your commands lack some '--erase-params' or '--writeconf'
> options... You should try reformatting your targets.

Sébastien, you are right. I did --erase-params and --writeconf and
traffic is distributed now as I wanted to.

Eventually I came back from bonded (mode6 + xmit_hash_policy:layer3+4)
to multirail because I see that I can squeeze extra 20% of performance.
The problem is that I am using Lustre routers to provide Lustre to the
single network and apparently it looks like this is not quite
supported. I tried:

routes="gni0 xxx.xxx.110.xxx at tcp0 xxx.xxx.111.xxx at tcp1"

And getting:

LustreError: 5598:0:(router.c:399:lnet_check_routes()) Routes to gni
via xxx.xxx.111.xxx at tcp1 and xxx.xxx.110.xxx at tcp not supported

I checked the manual and it says (p301): 
"It is an error to specify routes to the same destination with routers
on different local networks."

So I wonder how do you export your Lustre through multi-rail?

Thank you,
Alex.