<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>
<div><font face="Calibri,sans-serif">Hi All,</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">We were installing a new Lustre storage. </font></div>
<div><font face="Calibri,sans-serif">To that end , we have built new clients with the following configuration:</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">CentOS 6.8, kernel 2.6.32-642.el6.x86_64</font></div>
<div><font face="Calibri,sans-serif">Mellanox OFED 3.4.1.0 (on QDR fabric)</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">and either lustre-2.8.0 or lustre-2.9.0 clients, which we rebuilt from sources. The new server is Lustre 2.9 on CentOS 7.3 . </font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">Now, the clients we built have a problem in mounting the filesystem. It takes long time, and/or fails initially, with messages as follows (for the 2.8 client):</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">mounting device 192.168.xxx.yyy@o2ib:/lustre at /lustrenew, flags=0x400 options=flock,device=192.168.xxx.yyy@o2ib:/lustre</font></div>
<div><font face="Calibri,sans-serif">mount.lustre: mount 192.168.xxx.yyy@o2ib:/lustre at /lustrenew failed: Input/output error retries left: 0</font></div>
<div><font face="Calibri,sans-serif">mount.lustre: mount 192.168.xxx.yyy@o2ib:/lustre at /lustrenew failed: Input/output error</font></div>
<div><font face="Calibri,sans-serif">Is the MGS running?</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">In dmesg:</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">LNet: HW CPU cores: 24, npartitions: 4</font></div>
<div><font face="Calibri,sans-serif">alg: No test for adler32 (adler32-zlib)</font></div>
<div><font face="Calibri,sans-serif">alg: No test for crc32 (crc32-table)</font></div>
<div><font face="Calibri,sans-serif">alg: No test for crc32 (crc32-pclmul)</font></div>
<div><font face="Calibri,sans-serif">Lustre: Lustre: Build Version: 2.8.0-RC5--PRISTINE-2.6.32-642.el6.x86_64</font></div>
<div><font face="Calibri,sans-serif">LNet: Added LNI 192.168.aaa.bbb@o2ib [8/256/0/180]</font></div>
<div><font face="Calibri,sans-serif">Lustre: 3476:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1493927511/real 0] req@ffff88061a1aac80 x1566496533774340/t0(0) o250->MGC192.168.xxx.yyy@o2ib@192.168.xxx.yyy@o2ib:26/25
lens 520/544 e 0 to 1 dl 1493927516 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</font></div>
<div><font face="Calibri,sans-serif">LustreError: 15c-8: MGC192.168.xxx.yyy@o2ib: The configuration from log 'lustre-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See
the syslog for more information.</font></div>
<div><font face="Calibri,sans-serif">Lustre: Unmounted lustre-client</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">Initial mount would thus fail; then mount happens but OST's would take lot of time to become active;</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">UUID 1K-blocks Used Available Use% Mounted on</font></div>
<div><font face="Calibri,sans-serif">lustre-MDT0000_UUID 1156701708 751100 1077936556 0% /lustrenew[MDT:0]</font></div>
<div><font face="Calibri,sans-serif">OST0000 : inactive device</font></div>
<div><font face="Calibri,sans-serif">OST0001 : inactive device</font></div>
<div><font face="Calibri,sans-serif">OST0002 : inactive device</font></div>
<div><font face="Calibri,sans-serif">OST0003 : inactive device</font></div>
<div><font face="Calibri,sans-serif">OST0004 : inactive device</font></div>
<div><font face="Calibri,sans-serif">OST0005 : inactive device</font></div>
<div><font face="Calibri,sans-serif">OST0006 : inactive device</font></div>
<div><font face="Calibri,sans-serif">OST0007 : inactive device</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">filesystem summary: 0 0 0 0% /lustrenew</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">then, after some 10 minutes , the mount completes and performance-wise, Lustre seems to be normal.</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">Same dmesg output from 2.9 client</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">LNet: HW CPU cores: 24, npartitions: 2</font></div>
<div><font face="Calibri,sans-serif">alg: No test for adler32 (adler32-zlib)</font></div>
<div><font face="Calibri,sans-serif">alg: No test for crc32 (crc32-table)</font></div>
<div><font face="Calibri,sans-serif">alg: No test for crc32 (crc32-pclmul)</font></div>
<div><font face="Calibri,sans-serif">Lustre: Lustre: Build Version: 2.9.0</font></div>
<div><font face="Calibri,sans-serif">LNet: Added LNI 192.168.aaa.bbb@o2ib [8/256/0/180]</font></div>
<div><font face="Calibri,sans-serif">Lustre: 3468:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1493929145/real 0] req@ffff880631d07c80 x1566498247147536/t0(0) o250->MGC192.168.xxx.yyy@o2ib@192.168.xxx.yyy@o2ib:26/25
lens 520/544 e 0 to 1 dl 1493929150 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</font></div>
<div><font face="Calibri,sans-serif">LustreError: 15c-8: MGC192.168.xxx.yyy@o2ib: The configuration from log 'lustre-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See
the syslog for more information.</font></div>
<div><font face="Calibri,sans-serif">Lustre: Unmounted lustre-client</font></div>
<div><font face="Calibri,sans-serif">LustreError: 3413:0:(obd_mount.c:1449:lustre_fill_super()) Unable to mount (-5)</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><span style="font-family: Calibri, sans-serif;"> I am at loss as to what would cause such behavior? Could anyone advise where to look at for the causes of this problem? Thank you very much in advance!</span></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">--</font></div>
<div><font face="Calibri,sans-serif">Grigory Shamov</font></div>
<div><font face="Calibri,sans-serif">HPC SIte Lead,</font></div>
<div><font face="Calibri,sans-serif">University of Manitoba</font></div>
</div>
</body>
</html>