[Lustre-discuss] Questions about the LNET routing

Rick Wagner rpwagner at sdsc.edu
Fri Feb 20 15:03:29 PST 2015


It is the mode of your LACP that determines with physical interface the packets travel over, which can be configured to hash on client IP and port. Each client will open 3 TCP sockets for Lustre traffic to each server, and given a reasonable number of clients these will balance over interfaces in the link aggregation group. If you have bonded interfaces on the clients, a similar thing will happen as they connect to multiple servers. The caveat is that performance can be impacted by the NUMA architecture of your server. Basically, it's better to have both NICs attached to the same CPU.


From: lustre-discuss-bounces at lists.lustre.org [lustre-discuss-bounces at lists.lustre.org] on behalf of teng wang [tzw0019 at gmail.com]
Sent: Friday, February 20, 2015 1:40 PM
To: Andrus, Brian Contractor; lustre-discuss at lists.lustre.org
Subject: Re: [Lustre-discuss] Questions about the LNET routing

Hi Andrus, thanks for your answer.

Without bonding, is there any preference

for LNET to route from the two interfaces?

Even when we bond the two interfaces together,

I think LNET should still choose between the different

interfaces, although they share the same address.

Is there any preference in this situation?



On Thu, Feb 19, 2015 at 3:55 PM, Andrus, Brian Contractor <bdandrus at nps.edu<mailto:bdandrus at nps.edu>> wrote:

I believe it would depend on how you have your interfaces configured.

I seems that you have them both on the same subnet and being accessed by the same client.
Is this the case?

If they are on the same subnet, I would expect you would bond them (bond0) rather than have two separate IPs for them. Then you get to control how/where the data flows at the networking level.

You may want to check on the nodes to see what they see. (lctl peer_list)

If they are different subnets or networks, you can set that in the options for the lustre module for lnet.
For example, we have both ib and tcp. I give ib the priority for the best performance, but if ib is unavailable, it falls back to tcp. That could just as well be two Ethernet cards on two networks as well.

Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238<tel:831-656-6238>

From: lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lists.lustre.org> [mailto:lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lists.lustre.org>] On Behalf Of teng wang
Sent: Thursday, February 19, 2015 1:28 PM
To: lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
Subject: [Lustre-discuss] Questions about the LNET routing

I have a basic question about the LNET.  Will the data belonging
to the same object be routed from the same interface? For example,
if a node has multiple network interfaces and two processes are
running on the same node writing to the same shared file, striped
across 1 OST.
Process 1 writes like:
write chunk1
write chunk2

Process 2 writes like:
write chunk3
write chunk4
If Process 1 and Process2 are pinned to two different network interfaces,
say Eth0 and Eth1. Then from the OSC side, will these chunks be routed
to the OST from the same interface (E.g. All the four chunks through
Eth0)?  If so, what if they write different objects that come to the same
OST (E.g. Process1 write File1, Process2 write file2, file1 and file2 are
striped over the same OST)?


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20150220/437633b9/attachment.htm>

More information about the lustre-discuss mailing list