[lustre-discuss] Multiple IB Interfaces

[Matt Rásó-Barnett]

Hi Alastair,

On Fri, Mar 12, 2021 at 09:32:03AM +0000, Alastair Basden via lustre-discuss wrote:
>Reading is more problematic.  A request from a client (say 10.0.0.100) 
>for data on OST2 will come in via card 2 (10.0.0.2).  A thread on CPU2 
>(hopefully) will then read the data from OST2, and send it out to the 
>client, 10.0.0.100.  However, here, Linux will route the packet through 
>the first card on this subnet, so it will go over the inter-cpu link, 
>and out of IB card 1.  And this will be the case even if the thread is 
>pinned on CPU2.
>
>The question then is whether there is a way to configure Lustre to use 
>IB card 2 when sending out data from OST2.

The routing table entries referenced here: https://wiki.lustre.org/LNet_Router_Config_Guide#ARP_flux_issue_for_MR_node
should do this for you I believe, ensuring essentially that packets will 
be routed out over the interface that they are received on.

I think this is sufficient, but maybe someone more knowledgeable on this 
can confirm.

Cheers,
Matt

>On Wed, 10 Mar 2021, Ms. Megan Larko wrote:
>
>>[EXTERNAL EMAIL]
>>Greetings Alastair,
>>
>>Bonding is supported on InfiniBand, but  I believe that it is only active/passive.
>>I think what you might be looking for WRT avoiding data travel through the inter-cpu link is cpu "affinity" AKA cpu "pinning".
>>
>>Cheers,
>>megan
>>
>>WRT = "with regards to"
>>AKA = "also known as"
>>
>_______________________________________________
>lustre-discuss mailing list
>lustre-discuss at lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

-- 
Matt Rásó-Barnett