[Lustre-discuss] Bonded client interfaces and 10GbE server

Isaac Huang He.Huang at Sun.COM
Tue Jul 7 08:44:39 PDT 2009


On Tue, Jul 07, 2009 at 03:44:32PM +0200, Ralf Utermann wrote:
> Dear list,
> 
> we have setup of OSS and some clients with a dual Gigabit
> trunk (miimon=100 mode=802.3ad xmit_hash_policy=layer3+4).

If I understand it correctly, xmit_hash_policy=layer3+4 would not
allow a single TCP connection to span multiple slaves.

> If the clients stripe over targets on different OSS, they see
> a dual link bandwidth. If however, they stripe over targets on
> the same OSS, they only get the bandwith of one link.

Each client would create three TCP connections to an OSS, one for
exchanging small control messages, one for incoming bulk messages, and
one for outgoing bulk messages. The control connection could be
ignored for bandwidth considerations. When you're reading, only the
incoming bulk connection on the client is in use, and when writing the
outgoing bulk connection in use. Therefore, for read or write to a
same server, any client would utilize only one of its slaves. I'd believe
that you'd probably see better aggregate bandwidth when doing read and
write simultaneously - the incoming and outgoing bulk connections
should have different source ports and therefore they should be using
different slaves.

> If I would attach the OSS with a single 10GbE link, could
> a client then use the second link, when striping over targets
> on same OSS?

There's a rather complex way of static configuration to allow for
better overall bandwidth (though between any single client and server
there's still one link in use):
http://manual.lustre.org/manual/LustreManual16_HTML/MoreComplicatedConfigurations.html#50401393_pgfId-1287958

Thanks,
Isaac



More information about the lustre-discuss mailing list