[lustre-discuss] Multiple MGS interfaces config

Martin Hecht hecht at hlrs.de
Mon Sep 28 07:07:34 PDT 2015

On 09/27/2015 08:59 PM, Exec Unerd wrote:
>> I'm not sure if I have understood your setup correctly.
> In this case, the clients are a combination of all three: some are o2ib
> only, some tcp only, and some o2ib+tcp with tcp as failover.
> It sounds like I need a combination of configurations, one for the OSSes
> and one for each client type.
> So if I used this parameter in the OST,
> --mgsnode=" at o2ib0, at tcp0"
> Then configured the modprobe.d/lustre.conf appropriately on the clients
> tcp: options lnet networks="tcp0(ixgbe1)"
> o2ib: options lnet networks="o2ib0(ib1)"
> both: options lnet networks="o2ib0(ib1),tcp0(ixgbe1)"
> And use these mount parameters:
> tcp: mount -v -t lustre at tcp0:/testfs /mnt/testfs
> o2ib: mount -v -t lustre at o2ib0:/testfs /mnt/testfs
> both: mount -v -t lustre at o2ib0, at tcp0:/testfs
I think here it should be a colon between the two MGS nids:

mount -v -t lustre at o2ib0: at tcp0:/testfs

> /mnt/testfs
> Everything should be happy?
> On Thu, Sep 24, 2015 at 9:12 AM, Martin Hecht <hecht at hlrs.de> wrote:
>> On 09/24/2015 05:33 PM, Chris Hunter wrote:
>>> [...]
>>>>    2. What's the best way to trace the TCP client interactions to see
>>>> where
>>>>    it's breaking down?
>>> If lnet is running on the client, you can try "lctl ping"
>>> eg) lctl ping at o2ib
>>> I believe a lustre mount uses ipoib for initial handshake with a mds
>>> o2ib interfaces. You should make sure regular ping over ipoib is
>>> working before mounting lustre.
>> if the client and the server is on the same network, yes, it's a good
>> starting point. But it's not a prerequisite. In general you can have an
>> lnet router in-between or have different ip subnets for ipoib, so you
>> can't ping on the ipoib layer, but you can still lctl ping the whole
>> path (although you could verify that you can ip ping to the next hop at
>> least).
>> We also have a case in which we tried to block ipoib completely with
>> iptables, but we still could lctl ping, even after rebooting the host
>> and ensuring that the firewall was up before loading the lnet module.
>> So, I doubt that ipoib is needed at all for establishing the o2ib
>> connection.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2252 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20150928/f5d29781/attachment.bin>

More information about the lustre-discuss mailing list