[Lustre-discuss] Multihomed question: want Lustre over IB and Ethernet

[Chris Worley]

Most everything is over IB, but I have a few systems I'd like to mount
the Lustre fs over GigE.

I think I've followed the Multihomed instructions correctly, in:

http://dlc.sun.com/pdf/820-3681/820-3681.pdf

My /etc/modprobe.conf on mds/mgs/oss servers (which all have both
Ethernet and IB) includes:

options lnet 'networks="tcp0(eth0),o2ib0(ib0)"'

I make and mount the mdt with (which has both IB and Ethernet, subnet
36.122.x.x is IB, 36.121.x.x is Ethernet):

# mkfs.lustre --mdt --mgs
--mgsnode="36.122.255.201 at o2ib0,36.121.255.201 at tcp0" <... > /dev/md0
# mount -t lustre /dev/md0  /lfs/mdtb

But, at this point, the ksocklnd module is loaded rather than the
ko2iblnd module!

On the OSS, I make the fs w/ the same  "msgnode", but, when I try to
mount it, it correctly uses the IB interface, but can't contact the
MDS:

LustreError: 27520:0:(events.c:401:ptlrpc_uuid_to_peer()) No NID found
for MGC36.122.255.201 at o2ib_0
LustreError: 27520:0:(client.c:58:ptlrpc_uuid_to_connection()) cannot
find peer MGC36.122.255.201 at o2ib_0!
LustreError: 27520:0:(ldlm_lib.c:312:client_obd_setup()) can't add
initial connection
LustreError: 17126:0:(connection.c:142:ptlrpc_put_connection()) NULL connection
LustreError: 27520:0:(obd_config.c:325:class_setup()) setup
MGC36.122.255.201 at o2ib failed (-2)
LustreError: 27520:0:(obd_mount.c:454:lustre_start_simple())
MGC36.122.255.201 at o2ib setup error -2
LustreError: 27520:0:(obd_mount.c:1368:server_put_super()) no obd ddnlfs-OSTffff
LustreError: 27520:0:(obd_mount.c:119:server_deregister_mount())
ddnlfs-OSTffff not registered

It too has loaded the ksocklnd module, and not the ko2iblnd module.  I
guess that both modules should be loaded in a multihomed case?

What am I doing wrong?

Thanks,

Chris