<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=koi8-r">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Good Afternoon,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I'm experiencing an odd issue with one of my lustre clients. The system seems to be having an issue talking to one of the oss systems. When it reboots it is somehow mounting lustre twice. attempts to use lctl ping from the client to the OSS return the following
error:<br>
<br>
~] lctl ping 172.17.0.98@o2ib</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
failed to ping 172.17.0.98@o2ib: Input/output error<br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Conventional ping works<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
When I try to ping from the OSS side the lctl ping command hangs indefinitely. Looking in dmesg I see the following:<br>
[17291774.980764] LNet: 86013:0:(api-ni.c:4116:lnet_ping()) ping 12345-172.17.0.30@o2ib: late network completion
<div>[17292374.970610] LNet: 86013:0:(api-ni.c:4116:lnet_ping()) ping 12345-172.17.0.30@o2ib: late network completion </div>
<div>[17292974.961746] LNet: 86013:0:(api-ni.c:4116:lnet_ping()) ping 12345-172.17.0.30@o2ib: late network completion </div>
<div>[17293602.500931] LNet: 174596:0:(api-ni.c:4116:lnet_ping()) ping 12345-172.17.0.30@o2ib: late network completion </div>
[17294234.941320] LNet: 86013:0:(api-ni.c:4116:lnet_ping()) ping 12345-172.17.0.30@o2ib: late network completion<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
A further oddity is that mounting the lustre area seems to generate a double mount (when I unmount it by hand I have to do it twice to get it to unmount and it shows up twice in /proc/mounts</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The client is running the following:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
CentOS Linux release 7.3.1611 (Core) <br>
kernel: 3.10.0-514.el7.x86_64<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
rpm -qa | grep lustre
<div>lustre-client-2.10.5-1.el7.centos.x86_64 </div>
kmod-lustre-client-2.10.5-1.el7.centos.x86_64<br>
<br>
It has a qdr infiniband interface</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The OSS has the following:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
CentOS Linux release 7.6.1810 (Core)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
3.10.0-957.10.1.el7_lustre.x86_64<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
rpm -qa | grep lustre
<div>lustre-client-2.10.5-1.el7.centos.x86_64 </div>
kmod-lustre-client-2.10.5-1.el7.centos.x86_64<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
and an FDR interface<br>
<br>
Cables for the client have been swapped, and different qdr switches have been used.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The client needs to stay at that version of luster so it can connect to another, older, lustre file system.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you,<br>
</div>
<div id="Signature">
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif;">
<p style="margin-top: 0px; margin-bottom: 0px;margin-top:0; margin-bottom:0"><span id="ms-rterangepaste-start"></span><span style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">Kurt J. Strosahl</span><br style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">
<span style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">System Administrator: Lustre, HPC</span><br style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">
<span style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">Scientific Computing Group, Thomas Jefferson National Accelerator Facility</span><span id="ms-rterangepaste-end"></span><br>
</p>
</div>
</div>
</div>
</div>
</body>
</html>