[lustre-discuss] Lustre 2.11 lnet troubleshooting

Faaland, Olaf P. faaland1 at llnl.gov
Tue Apr 17 16:34:22 PDT 2018


I've got a cluster running 2.11 with 2 routers and 68  compute nodes.  It's the first time I've used a post-multi-rail version of Lustre.  

The problem I'm trying to troubleshoot is that my sample compute node (ulna66) seems to think the router I configured (ulna4) is down, and so an attempt to ping outside the cluster results in failure and "no route to XXX" on the console.  I can lctl ping the router from the compute node and vice-versa.   Forwarding is enabled on the router node via modprobe argument.

lnetctl route show reports that the route is down.  Where I'm stuck is figuring out what in userspace (e.g. lnetctl or lctl) can tell me why.

The compute node's lnet configuration is:

[root at ulna66:lustre-211]# cat /etc/lnet.conf
  - net-spec: o2ib33
         0: hsi0
         0: 192.168.128.*
    - net: o2ib100
      gateway: at o2ib33

After I start lnet, systemctl reports success and the state is as follows:

[root at ulna66:lustre-211]# lnetctl net show
    - net type: lo
      local NI(s):
        - nid: 0 at lo
          status: up
    - net type: o2ib33
      local NI(s):
        - nid: at o2ib33
          status: up
              0: hsi0

[root at ulna66:lustre-211]# lnetctl peer show --verbose
    - primary nid: at o2ib33
      Multi-Rail: False
      peer ni:
        - nid: at o2ib33
          state: up
          max_ni_tx_credits: 8
          available_tx_credits: 8
          min_tx_credits: 7
          tx_q_num_of_buf: 0
          available_rtr_credits: 8
          min_rtr_credits: 8
          refcount: 4
              send_count: 2
              recv_count: 2
              drop_count: 0

[root at ulna66:lustre-211]# lnetctl route show --verbose
    - net: o2ib100
      gateway: at o2ib33
      hop: -1
      priority: 0
      state: down

I can instrument the code, but I figure there must be someplace available to normal users to look, that I'm unaware of.


Olaf P. Faaland
Livermore Computing

More information about the lustre-discuss mailing list