[Lustre-discuss] Questions about the LNET routing

teng wang tzw0019 at gmail.com
Mon Feb 23 08:38:34 PST 2015


Rick,


Thanks for you answer.  Could you explain more

about the NUMA architecture? The two NICs attached

to the same CPU you mentioned are on the client

side or the server side? How is the performance impacted

by the NUMA architecture given

that the client can balance traffic from the interfaces?


Thanks,

Teng

On Fri, Feb 20, 2015 at 5:03 PM, Rick Wagner <rpwagner at sdsc.edu> wrote:

>  Teng,
>
> It is the mode of your LACP that determines with physical interface the
> packets travel over, which can be configured to hash on client IP and port.
> Each client will open 3 TCP sockets for Lustre traffic to each server, and
> given a reasonable number of clients these will balance over interfaces in
> the link aggregation group. If you have bonded interfaces on the clients, a
> similar thing will happen as they connect to multiple servers. The caveat
> is that performance can be impacted by the NUMA architecture of your
> server. Basically, it's better to have both NICs attached to the same CPU.
>
> --Rick
>
>  ------------------------------
> *From:* lustre-discuss-bounces at lists.lustre.org [
> lustre-discuss-bounces at lists.lustre.org] on behalf of teng wang [
> tzw0019 at gmail.com]
> *Sent:* Friday, February 20, 2015 1:40 PM
> *To:* Andrus, Brian Contractor; lustre-discuss at lists.lustre.org
> *Subject:* Re: [Lustre-discuss] Questions about the LNET routing
>
>   Hi Andrus, thanks for your answer.
>
>
>  Without bonding, is there any preference
>
> for LNET to route from the two interfaces?
>
> Even when we bond the two interfaces together,
>
> I think LNET should still choose between the different
>
> interfaces, although they share the same address.
>
> Is there any preference in this situation?
>
>
>  Thanks,
>
> Teng
>
> On Thu, Feb 19, 2015 at 3:55 PM, Andrus, Brian Contractor <
> bdandrus at nps.edu> wrote:
>
>>  Teng,
>>
>>
>>
>> I believe it would depend on how you have your interfaces configured.
>>
>>
>>
>> I seems that you have them both on the same subnet and being accessed by
>> the same client.
>>
>> Is this the case?
>>
>>
>>
>> If they are on the same subnet, I would expect you would bond them
>> (bond0) rather than have two separate IPs for them. Then you get to control
>> how/where the data flows at the networking level.
>>
>>
>>
>> You may want to check on the nodes to see what they see. (lctl peer_list)
>>
>>
>>
>> If they are different subnets or networks, you can set that in the
>> options for the lustre module for lnet.
>>
>> For example, we have both ib and tcp. I give ib the priority for the best
>> performance, but if ib is unavailable, it falls back to tcp. That could
>> just as well be two Ethernet cards on two networks as well.
>>
>>
>>
>>
>>
>> Brian Andrus
>>
>> ITACS/Research Computing
>>
>> Naval Postgraduate School
>>
>> Monterey, California
>>
>> voice: 831-656-6238
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From:* lustre-discuss-bounces at lists.lustre.org [mailto:
>> lustre-discuss-bounces at lists.lustre.org] *On Behalf Of *teng wang
>> *Sent:* Thursday, February 19, 2015 1:28 PM
>> *To:* lustre-discuss at lists.lustre.org
>> *Subject:* [Lustre-discuss] Questions about the LNET routing
>>
>>
>>
>> I have a basic question about the LNET.  Will the data belonging
>>
>> to the same object be routed from the same interface? For example,
>>
>> if a node has multiple network interfaces and two processes are
>>
>> running on the same node writing to the same shared file, striped
>>
>> across 1 OST.
>>
>> Process 1 writes like:
>>
>> write chunk1
>>
>> write chunk2
>>
>>
>>
>> Process 2 writes like:
>>
>> write chunk3
>>
>> write chunk4
>>
>> If Process 1 and Process2 are pinned to two different network interfaces,
>>
>> say Eth0 and Eth1. Then from the OSC side, will these chunks be routed
>>
>> to the OST from the same interface (E.g. All the four chunks through
>>
>> Eth0)?  If so, what if they write different objects that come to the same
>>
>> OST (E.g. Process1 write File1, Process2 write file2, file1 and file2 are
>>
>> striped over the same OST)?
>>
>>
>>
>> Thanks,
>>
>> Teng
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20150223/5497f191/attachment.htm>


More information about the lustre-discuss mailing list