<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Are the NIDs "192.168.xxx.yyy@o2ib” really configured that way or did you modify those logs when pasting them to email?
<div class=""><br class="">
</div>
<div class="">Doug</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On May 5, 2017, at 11:02 AM, Grigory Shamov <<a href="mailto:Grigory.Shamov@umanitoba.ca" class="">Grigory.Shamov@umanitoba.ca</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">
<div class=""><font face="Calibri,sans-serif" class="">Hi All,</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">We were installing a new Lustre storage. </font></div>
<div class=""><font face="Calibri,sans-serif" class="">To that end , we have built new clients with the following configuration:</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">CentOS 6.8, kernel 2.6.32-642.el6.x86_64</font></div>
<div class=""><font face="Calibri,sans-serif" class="">Mellanox OFED 3.4.1.0 (on QDR fabric)</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">and either lustre-2.8.0 or lustre-2.9.0 clients, which we rebuilt from sources. The new server is Lustre 2.9 on CentOS 7.3 . </font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">Now, the clients we built have a problem in mounting the filesystem. It takes long time, and/or fails initially, with messages as follows (for the 2.8 client):</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">mounting device 192.168.xxx.yyy@o2ib:/lustre at /lustrenew, flags=0x400 options=flock,device=192.168.xxx.yyy@o2ib:/lustre</font></div>
<div class=""><font face="Calibri,sans-serif" class="">mount.lustre: mount 192.168.xxx.yyy@o2ib:/lustre at /lustrenew failed: Input/output error retries left: 0</font></div>
<div class=""><font face="Calibri,sans-serif" class="">mount.lustre: mount 192.168.xxx.yyy@o2ib:/lustre at /lustrenew failed: Input/output error</font></div>
<div class=""><font face="Calibri,sans-serif" class="">Is the MGS running?</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">In dmesg:</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">LNet: HW CPU cores: 24, npartitions: 4</font></div>
<div class=""><font face="Calibri,sans-serif" class="">alg: No test for adler32 (adler32-zlib)</font></div>
<div class=""><font face="Calibri,sans-serif" class="">alg: No test for crc32 (crc32-table)</font></div>
<div class=""><font face="Calibri,sans-serif" class="">alg: No test for crc32 (crc32-pclmul)</font></div>
<div class=""><font face="Calibri,sans-serif" class="">Lustre: Lustre: Build Version: 2.8.0-RC5--PRISTINE-2.6.32-642.el6.x86_64</font></div>
<div class=""><font face="Calibri,sans-serif" class="">LNet: Added LNI 192.168.aaa.bbb@o2ib [8/256/0/180]</font></div>
<div class=""><font face="Calibri,sans-serif" class="">Lustre: 3476:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1493927511/real 0] req@ffff88061a1aac80 x1566496533774340/t0(0) o250->MGC192.168.xxx.yyy@<a href="mailto:o2ib@192.168.xxx.yyy" class="">o2ib@192.168.xxx.yyy</a>@o2ib:26/25
lens 520/544 e 0 to 1 dl 1493927516 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</font></div>
<div class=""><font face="Calibri,sans-serif" class="">LustreError: 15c-8: MGC192.168.xxx.yyy@o2ib: The configuration from log 'lustre-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or
other errors. See the syslog for more information.</font></div>
<div class=""><font face="Calibri,sans-serif" class="">Lustre: Unmounted lustre-client</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">Initial mount would thus fail; then mount happens but OST's would take lot of time to become active;</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">UUID 1K-blocks Used Available Use% Mounted on</font></div>
<div class=""><font face="Calibri,sans-serif" class="">lustre-MDT0000_UUID 1156701708 751100 1077936556 0% /lustrenew[MDT:0]</font></div>
<div class=""><font face="Calibri,sans-serif" class="">OST0000 : inactive device</font></div>
<div class=""><font face="Calibri,sans-serif" class="">OST0001 : inactive device</font></div>
<div class=""><font face="Calibri,sans-serif" class="">OST0002 : inactive device</font></div>
<div class=""><font face="Calibri,sans-serif" class="">OST0003 : inactive device</font></div>
<div class=""><font face="Calibri,sans-serif" class="">OST0004 : inactive device</font></div>
<div class=""><font face="Calibri,sans-serif" class="">OST0005 : inactive device</font></div>
<div class=""><font face="Calibri,sans-serif" class="">OST0006 : inactive device</font></div>
<div class=""><font face="Calibri,sans-serif" class="">OST0007 : inactive device</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">filesystem summary: 0 0 0 0% /lustrenew</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">then, after some 10 minutes , the mount completes and performance-wise, Lustre seems to be normal.</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">Same dmesg output from 2.9 client</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">LNet: HW CPU cores: 24, npartitions: 2</font></div>
<div class=""><font face="Calibri,sans-serif" class="">alg: No test for adler32 (adler32-zlib)</font></div>
<div class=""><font face="Calibri,sans-serif" class="">alg: No test for crc32 (crc32-table)</font></div>
<div class=""><font face="Calibri,sans-serif" class="">alg: No test for crc32 (crc32-pclmul)</font></div>
<div class=""><font face="Calibri,sans-serif" class="">Lustre: Lustre: Build Version: 2.9.0</font></div>
<div class=""><font face="Calibri,sans-serif" class="">LNet: Added LNI 192.168.aaa.bbb@o2ib [8/256/0/180]</font></div>
<div class=""><font face="Calibri,sans-serif" class="">Lustre: 3468:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1493929145/real 0] req@ffff880631d07c80 x1566498247147536/t0(0) o250->MGC192.168.xxx.yyy@<a href="mailto:o2ib@192.168.xxx.yyy" class="">o2ib@192.168.xxx.yyy</a>@o2ib:26/25
lens 520/544 e 0 to 1 dl 1493929150 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</font></div>
<div class=""><font face="Calibri,sans-serif" class="">LustreError: 15c-8: MGC192.168.xxx.yyy@o2ib: The configuration from log 'lustre-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or
other errors. See the syslog for more information.</font></div>
<div class=""><font face="Calibri,sans-serif" class="">Lustre: Unmounted lustre-client</font></div>
<div class=""><font face="Calibri,sans-serif" class="">LustreError: 3413:0:(obd_mount.c:1449:lustre_fill_super()) Unable to mount (-5)</font></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><span style="font-family: Calibri, sans-serif;" class=""> I am at loss as to what would cause such behavior? Could anyone advise where to look at for the causes of this problem? Thank you very much in advance!</span></div>
<div class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri,sans-serif" class="">--</font></div>
<div class=""><font face="Calibri,sans-serif" class="">Grigory Shamov</font></div>
<div class=""><font face="Calibri,sans-serif" class="">HPC SIte Lead,</font></div>
<div class=""><font face="Calibri,sans-serif" class="">University of Manitoba</font></div>
</div>
</div>
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>