<div dir="ltr"><div><span style="font-size:12.8px">> I'm not sure if I have understood your setup correctly. </span><br></div><div><br></div>In this case, the clients are a combination of all three: some are o2ib only, some tcp only, and some o2ib+tcp with tcp as failover.<div><br></div><div>It sounds like I need a combination of configurations, one for the OSSes and one for each client type.</div><div><br></div><div>So if I used this parameter in the OST,</div><div>--mgsnode="172.16.10.1@o2ib0,192.168.10.1@tcp0"<br></div><div><br></div><div>Then configured the modprobe.d/lustre.conf appropriately on the clients</div><div>tcp: options lnet networks="tcp0(ixgbe1)"</div><div>o2ib: options lnet networks="o2ib0(ib1)"</div><div>both: options lnet networks="o2ib0(ib1),tcp0(ixgbe1)"</div>
<div><br></div><div>And use these mount parameters:</div><div><div>tcp: <span style="font-size:12.8px">mount -v -t lustre 192.168.10.1</span><span style="font-size:12.8px">@tcp0:/testfs /mnt/testfs</span></div><div>o2ib: <span style="font-size:12.8px">mount -v -t lustre 172.16.10.1@o2ib0</span><span style="font-size:12.8px">:/testfs /mnt/testfs</span></div><div>both: <span style="font-size:12.8px">mount -v -t lustre 172.16.10.1@o2ib0,192.168.10.1</span><span style="font-size:12.8px">@tcp0:/testfs /mnt/testfs</span></div></div><div><br></div><div>Everything should be happy?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 24, 2015 at 9:12 AM, Martin Hecht <span dir="ltr"><<a href="mailto:hecht@hlrs.de" target="_blank">hecht@hlrs.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 09/24/2015 05:33 PM, Chris Hunter wrote:<br>
> [...]<br>
<span class="">>> 2. What's the best way to trace the TCP client interactions to see<br>
>> where<br>
>> it's breaking down?<br>
> If lnet is running on the client, you can try "lctl ping"<br>
> eg) lctl ping 172.16.10.1@o2ib<br>
><br>
> I believe a lustre mount uses ipoib for initial handshake with a mds<br>
> o2ib interfaces. You should make sure regular ping over ipoib is<br>
> working before mounting lustre.<br>
</span>if the client and the server is on the same network, yes, it's a good<br>
starting point. But it's not a prerequisite. In general you can have an<br>
lnet router in-between or have different ip subnets for ipoib, so you<br>
can't ping on the ipoib layer, but you can still lctl ping the whole<br>
path (although you could verify that you can ip ping to the next hop at<br>
least).<br>
<br>
We also have a case in which we tried to block ipoib completely with<br>
iptables, but we still could lctl ping, even after rebooting the host<br>
and ensuring that the firewall was up before loading the lnet module.<br>
So, I doubt that ipoib is needed at all for establishing the o2ib<br>
connection.<br>
<br>
<br>
<br>_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br></blockquote></div><br></div>