[Lustre-discuss] show_route

[Isaac Huang]

On Tue, Oct 07, 2008 at 11:00:20PM -0600, Andreas Dilger wrote:
> On Oct 07, 2008  22:58 -0400, Mag Gam wrote:
> > My intention was I wanted to see if my lustre connection is being
> > routed thru other interfaces. I have 4 interfaces on my server: eth0
> > thru eth4. eth0 is used for Lustre but it seems lustre picks the best
> > route. I wanted to be 100% sure its not using anything else but eth0.
> > Is there another test for this? I know editing /etc/modules.conf will
> > fix it and I already have that, but I want a realtime test.
> 
> You can use "lctl list_nids" to show the interfaces that lustre is
> using.  "lctl ping {nid}" will show the connections between the nodes.

While "lctl list_nids" certainly shows the LNet interfaces Lustre is
using, it's probably not the most accurate way to find out which
Ethernet interface Lustre traffic really goes on.

When socklnd bonding is in use (e.g. networks="tcp0(eth0,eth1)"), the
LNet NID would only pickup IP of the 1st interface in the networks
specification (e.g. ip_of_eth0 at tcp), despite the fact that LNet actually
tries to make use of both interfaces.

Moreover, the Ethernet interface in use is solely determined by the
Linux IP routing. For example, you have eth0 and eth1 and asks lnet to
use eth0 only (e.g. networks="tcp(eth0)"), the traffic could still end
up going through eth1 if Linux IP routing chooses so (e.g. both
interfaces are in the same IP network, perhaps by mistake, and the
routing table entry for eth1 comes earlier). Such misconfigured
routing and the accompanying confusion happened a couple of times at
some customer sites as I recall.

I usually rely on watching TX/RX bytes of 'ifconfig' and a bit
tcpdumping to find out which interfaces are really being used.

Thanks,
Isaac