[lustre-discuss] LNET tunables and LND tunables

[Pinkesh Valdria]

Hello, 

 

I have a lustre cluster using 25gbps ethernet network.  (no infinitiband).    I see lot of examples online for infiniband and what tunables to use for it,  but I am struggling to find recommendations when using ethernet networks.   

 

Appreciate if someone can share their experience and settings when using ethernet or if there is any details online for ethernet recommended values.  

 

 

>From my lustre client node.   I don’t see any LND tunables below like (peercredits_hiw: 64, map_on_demand: 32, concurrent_sends: 256, fmr_pool_size: 2048, fmr_flush_trigger: 512, fmr_cache: 1).

 

[root at lustre-client-1 ~]# lnetctl net show --verbose

net:

    - net type: lo

      local NI(s):

        - nid: 0 at lo

          status: up

          statistics:

              send_count: 0

              recv_count: 0

              drop_count: 0

          tunables:

              peer_timeout: 0

              peer_credits: 0

              peer_buffer_credits: 0

              credits: 0

          dev cpt: 0

          tcp bonding: 0

          CPT: "[0,1,2,3,4,5,6,7,8,9,10,11]"

    - net type: tcp1

      local NI(s):

        - nid: 10.0.3.4 at tcp1

          status: up

          interfaces:

              0: ens3

          statistics:

              send_count: 657209676

              recv_count: 657208330

              drop_count: 0

          tunables:

              peer_timeout: 180

              peer_credits: 8

              peer_buffer_credits: 0

              credits: 256

          dev cpt: -1

          tcp bonding: 0

          CPT: "[0,1,2,3,4,5,6,7,8,9,10,11]"

[root at lustre-client-1 ~]#

 

 

OSS Server 

 

[root at lustre-oss-server-nic0-1 ~]# lnetctl net show --verbose

net:

    - net type: lo

      local NI(s):

        - nid: 0 at lo

          status: up

          statistics:

              send_count: 0

              recv_count: 0

              drop_count: 0

          tunables:

              peer_timeout: 0

              peer_credits: 0

              peer_buffer_credits: 0

              credits: 0

          dev cpt: 0

          tcp bonding: 0

          CPT: "[0,1]"

    - net type: tcp1

      local NI(s):

        - nid: 10.0.3.6 at tcp1

          status: up

          interfaces:

              0: eno3d1

          statistics:

              send_count: 232650108

              recv_count: 232650019

              drop_count: 0

          tunables:

              peer_timeout: 180

              peer_credits: 8

              peer_buffer_credits: 0

              credits: 256

          dev cpt: 0

          tcp bonding: 0

          CPT: "[0,1]"

[root at lustre-oss-server-nic0-1 ~]#

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20190811/99279722/attachment.html>