<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Good Morning,</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"> I'm presently working on an issue with my OPA network that seems to be having an unusual impact on lustre. What happens is that when one of the nodes on the OPA fabric reboots it sometimes has trouble reaching one
of the four lnet routers that we have set up. This isn't, itself, a lustre problem as the nodes experiencing this issue can't even ping the lnet routers opa interface. The impact this has is that the lustre file system can't mount, even though the other
three lnet routers are available. Eventually the issue clears up and lustre is able to mount, but I'm wondering why having one of the four lnet routers down would prevent the lustre file system from mounting. Is it because the lnet router is only down from
the OPA side, while the IB side is still up?</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Here is what the system records in dmesg </p>
<p style="margin-top:0;margin-bottom:0"></p>
<div>[ 45.839655] Lustre: Server MGS version (2.5.42.4) is much older than client. Consider upgrading server (2.10.4)</div>
<div>[ 52.847699] Lustre: 3469:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555330354/real 1555330354] req@ffff882ef0930300 x1630882080227408/t0(0) o501->MGC172.17.4.125@o2ib@172.17.4.125@o2ib:26/25
lens 296/272 e 0 to 1 dl 1555330361 ref 2 fl Rpc:X/0/ffffffff rc 0/-1</div>
<div>[ 52.847735] LustreError: 166-1: MGC172.17.4.125@o2ib: Connection to MGS (at 172.17.4.125@o2ib) was lost; in progress operations using this service will fail</div>
<div>[ 52.847933] LustreError: 15c-8: MGC172.17.4.125@o2ib: The configuration from log 'lustre2-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more
information.</div>
<div>[ 52.848876] Lustre: Unmounted lustre2-client</div>
<div>[ 52.849424] Lustre: MGC172.17.4.125@o2ib: Connection restored to MGC172.17.4.125@o2ib_0 (at 172.17.4.125@o2ib)</div>
<div>[ 52.857803] LustreError: 3469:0:(obd_mount.c:1582:lustre_fill_super()) Unable to mount (-5)</div>
<div>[ 56.828713] LNet: 3604:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 172.20.0.3@o2ib1: 4294722 seconds</div>
<div>[ 106.829847] LNet: 3604:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 172.20.0.3@o2ib1: 4294772 seconds</div>
<div><br>
</div>
The 172.20.0.3 address is the lnet router in question.
<p></p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">w/r,</p>
<p style="margin-top:0;margin-bottom:0"><span style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">Kurt J. Strosahl</span><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<p style="margin-top:0; margin-bottom:0"><span style="color:rgb(51,51,51); font-family:monospace; font-size:14.16px">System Administrator: Lustre, HPC</span><br style="color:rgb(51,51,51); font-family:monospace; font-size:14.16px">
<span style="color:rgb(51,51,51); font-family:monospace; font-size:14.16px">Scientific Computing Group, Thomas Jefferson National Accelerator Facility</span><br>
</p>
</div>
</div>
</div>
</body>
</html>