[Lustre-discuss] Bonded client interfaces and 10GbE server

[Isaac Huang]

On Tue, Jul 07, 2009 at 11:44:39AM -0400, Isaac Huang wrote:
> ......
> > If I would attach the OSS with a single 10GbE link, could
> > a client then use the second link, when striping over targets
> > on same OSS?
> 
> There's a rather complex way of static configuration to allow for
> better overall bandwidth (though between any single client and server
> there's still one link in use):
> http://manual.lustre.org/manual/LustreManual16_HTML/MoreComplicatedConfigurations.html#50401393_pgfId-1287958

As an alternative, you might try the ksocklnd bonding on clients and
servers, e.g.:
options ksocklnd networks="tcp0(eth0, eth1)"

Then ksocklnd would create two sets of connections (control, bulk in,
and bulk out) and balance traffic over them.

The downside is that it might take a long time for ksocklnd to notice
a downed NIC and avoid it, and when the downed NIC comes back to life
later the ksocklnd might not be able to use it again.

Two more gotchas for ksocklnd bonding:
1. IP routing ultimately determines outgoing interfaces and must be
configured properly. For example, if both eth0 and eth1 of clients and
servers belong to a same IP subnet, all outgoing packets might be sent
by a same NIC because the destination IP addresses, though different,
belong to a same destination IP network.
2. All incoming messages might arrive on a same NIC. Please refer to
linux-*/Documentation/networking/ip-sysctl.txt for arp_ignore.

Thanks,
Isaac