[lustre-discuss] Multiple MGS interfaces config
execunerd at gmail.com
Sun Sep 27 11:59:13 PDT 2015
> I'm not sure if I have understood your setup correctly.
In this case, the clients are a combination of all three: some are o2ib
only, some tcp only, and some o2ib+tcp with tcp as failover.
It sounds like I need a combination of configurations, one for the OSSes
and one for each client type.
So if I used this parameter in the OST,
--mgsnode="172.16.10.1 at o2ib0,192.168.10.1 at tcp0"
Then configured the modprobe.d/lustre.conf appropriately on the clients
tcp: options lnet networks="tcp0(ixgbe1)"
o2ib: options lnet networks="o2ib0(ib1)"
both: options lnet networks="o2ib0(ib1),tcp0(ixgbe1)"
And use these mount parameters:
tcp: mount -v -t lustre 192.168.10.1 at tcp0:/testfs /mnt/testfs
o2ib: mount -v -t lustre 172.16.10.1 at o2ib0:/testfs /mnt/testfs
both: mount -v -t lustre 172.16.10.1 at o2ib0,192.168.10.1 at tcp0:/testfs
Everything should be happy?
On Thu, Sep 24, 2015 at 9:12 AM, Martin Hecht <hecht at hlrs.de> wrote:
> On 09/24/2015 05:33 PM, Chris Hunter wrote:
> > [...]
> >> 2. What's the best way to trace the TCP client interactions to see
> >> where
> >> it's breaking down?
> > If lnet is running on the client, you can try "lctl ping"
> > eg) lctl ping 172.16.10.1 at o2ib
> > I believe a lustre mount uses ipoib for initial handshake with a mds
> > o2ib interfaces. You should make sure regular ping over ipoib is
> > working before mounting lustre.
> if the client and the server is on the same network, yes, it's a good
> starting point. But it's not a prerequisite. In general you can have an
> lnet router in-between or have different ip subnets for ipoib, so you
> can't ping on the ipoib layer, but you can still lctl ping the whole
> path (although you could verify that you can ip ping to the next hop at
> We also have a case in which we tried to block ipoib completely with
> iptables, but we still could lctl ping, even after rebooting the host
> and ensuring that the firewall was up before loading the lnet module.
> So, I doubt that ipoib is needed at all for establishing the o2ib
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the lustre-discuss