[Lustre-discuss] Multihomed question: want Lustre over IB and Ethernet

[Craig Prescott]

Chris Worley wrote:
> On Fri, Mar 7, 2008 at 9:39 AM, Craig Prescott <prescott at hpc.ufl.edu> wrote:
>>  I think your client modprobe.conf lnet option
>>  should be this:
>>
>>
>>  options lnet networks=o2ib(ib0)
>>
>>  (not 'o2ib0').
> 
> It still seems to want the TCP connection:
> 
> <snip>

Oh, sorry - I didn't notice you were using 'o2ib0's everywhere.
In that case, you should keep it.

> 
>>  Another thing to try, if that doesn't work lctl
>>  ping your MDS/MGS/OSS nids, like so:
>>
>>  lctl ping 36.122.255.201 at o2ib
> 
> Before and after the change it looks the same:
> 
> # lctl ping 36.122.255.201 at o2ib
> 12345-0 at lo
> 12345-36.122.255.201 at o2ib
> 12345-36.121.255.201 at tcp
> 
> If I change my modprobe.conf to look as on the MDS/OSS's:
> 
> options lnet networks=o2ib0(ib0),tcp0(eth0)
> 
> Then, mount just specifying o2ib:
> 
> # mount -t lustre 36.122.255.201 at o2ib:/ddnlfs /lfs
> 
> It works, but, both ko2iblnd and ksocklnd are loaded.

Ok, so you need the tcp nid for the mount to succeed.
So it looks like the MGS is only available via the tcp nid.

Is the quoting you had in the --mgsnode option when you created
the fs responsible for this?  Maybe.  If you can recreate
the filesystem again, try it without the quotes:

mkfs.lustre --reformat --fsname=... --ost --whateverelse \
   --mgsnode=36.122.255.201 at o2ib0,36.121.255.201 at tcp0 ...

Cheers,
Craig

> The dmesg output is:
> 
> Lustre: OBD class driver, info at clusterfs.com
>         Lustre Version: 1.6.4.2
>         Build Version:
> 1.6.4.2-19691231190000-PRISTINE-.usr.src.linux-2.6.9-67.0.4.EL-Lustre-1.6.4.2
> Lustre: Added LNI 36.122.255.1 at o2ib [8/64]
> Lustre: Added LNI 36.121.255.1 at tcp [8/256]
> Lustre: Accept secure, port 988
> Lustre: Lustre Client File System; info at clusterfs.com
> Lustre: ddnlfs-clilov-000001042f8b7c00.lov: set parameter stripesize=2M
> Lustre: Client ddnlfs-client has started
> 
> Can I be certain it'll use IB for LFS on this client?
> 
> Thanks,
> 
> Chris