<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><blockquote type="cite" class="">On Dec 3, 2017, at 11:55 AM, Riccardo Veraldi <<a href="mailto:riccardo.veraldi@gmail.com" class="">riccardo.veraldi@gmail.com</a>> wrote:<br class=""><br class="">Hello,<br class="">are you using Infiniband ?<br class=""></blockquote><br class="">Yes - Mellanox FDR (Connect-IB, ConnectX-3)<br class=""><br class=""><blockquote type="cite" class="">if so what are the peer credit settings ?<br class=""></blockquote><br class="">Defaults. Note that while we are in the process of upgrading our clients to 2.10.1, most are still 2.5.3.90.<div class="">I think I indicated previously that we saw this with both 2.5.3.90 and 2.10.1 using the same client (i.e. we upgraded it to try the 2.10.1 client modules).</div><div class=""><br class=""></div><div class="">Although we saw this on our standard 128G haswell clients with this particular application, they may have been RAM constrained which is why we went to the “bigmem” host. However, the “bigmem” host is a bit of an odd bird itself so we may have just traded one client issue for another. </div><div class=""><br class=""></div><div class="">The application ran successfully (just finished Friday) on a newer “bigmem” server without being evicted. I’d like to understand what was causing the client to fail to respond to the OSS in a timely fashion but that may take more effort than is warranted at this point.</div><div class=""><br class=""><blockquote type="cite" class=""><br class=""> cat /proc/sys/lnet/nis <br class=""></blockquote><div class=""><br class=""></div><font face="Courier" class="">nid status alive refs peer rtr max tx min<br class="">0@lo up 0 2 0 0 0 0 0<br class="">0@lo up 0 0 0 0 0 0 0<br class="">0@lo up 0 0 0 0 0 0 0<br class="">0@lo up 0 0 0 0 0 0 0<br class="">0@lo up 0 0 0 0 0 0 0<br class="">0@lo up 0 0 0 0 0 0 0<br class="">0@lo up 0 0 0 0 0 0 0<br class="">0@lo up 0 0 0 0 0 0 0<br class="">10.13.136.74@o2ib up -1 1 8 0 64 64 32<br class="">10.13.136.74@o2ib up -1 0 8 0 64 64 42<br class="">10.13.136.74@o2ib up -1 0 8 0 64 64 26<br class="">10.13.136.74@o2ib up -1 0 8 0 64 64 30<br class="">10.13.136.74@o2ib up -1 0 8 0 64 64 43<br class="">10.13.136.74@o2ib up -1 0 8 0 64 64 34<br class="">10.13.136.74@o2ib up -1 0 8 0 64 64 30<br class="">10.13.136.74@o2ib up -1 0 8 0 64 64 47<br class=""></font><br class=""><blockquote type="cite" class=""> cat /proc/sys/lnet/peers <br class=""></blockquote><div class=""><br class=""></div>[root@s5a-s23 lnet]# cat /proc/sys/lnet/peers <br class=""><font face="Courier" class="">nid refs state last max rtr min tx min queue<br class="">10.13.136.159@o2ib 1 NA -1 8 8 8 8 -1 0<br class="">10.13.136.15@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.28@o2ib 1 NA -1 8 8 8 8 -16 0<br class="">10.13.136.7@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.130@o2ib 1 NA -1 8 8 8 8 1 0<br class="">10.13.136.164@o2ib 1 NA -1 8 8 8 8 -1 0<br class="">10.13.136.20@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.156@o2ib 1 NA -1 8 8 8 8 1 0<br class="">10.13.136.12@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.169@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.25@o2ib 1 NA -1 8 8 8 8 -41 0<br class="">10.13.136.4@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.161@o2ib 1 NA -1 8 8 8 8 -1 0<br class="">10.13.136.17@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.51@o2ib 1 NA -1 8 8 8 8 6 0<br class="">10.13.136.30@o2ib 1 NA -1 8 8 8 8 -27 0<br class="">10.13.136.153@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.9@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.166@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.22@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.1@o2ib 1 NA -1 8 8 8 8 0 0<br class="">10.13.136.158@o2ib 1 NA -1 8 8 8 8 -1 0<br class="">10.13.136.14@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.171@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.27@o2ib 1 NA -1 8 8 8 8 -13 0<br class="">10.13.136.6@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.129@o2ib 1 NA -1 8 8 8 8 7 0<br class="">10.13.136.163@o2ib 1 NA -1 8 8 8 8 -1 0<br class="">10.13.136.19@o2ib 1 NA -1 8 8 8 8 1 0<br class="">10.13.136.32@o2ib 1 NA -1 8 8 8 8 -46 0<br class="">10.13.136.155@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.11@o2ib 1 NA -1 8 8 8 8 -4 0<br class="">10.13.136.168@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.24@o2ib 1 NA -1 8 8 8 8 -65 0<br class="">10.13.136.3@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.160@o2ib 1 NA -1 8 8 8 8 -1 0<br class="">10.13.136.16@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.29@o2ib 1 NA -1 8 8 8 8 -46 0<br class="">10.13.136.8@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.165@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.21@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.157@o2ib 1 NA -1 8 8 8 8 1 0<br class="">10.13.136.13@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.170@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.26@o2ib 1 NA -1 8 8 8 8 -46 0<br class="">10.13.136.5@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.162@o2ib 1 NA -1 8 8 8 8 -1 0<br class="">10.13.136.18@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.31@o2ib 1 NA -1 8 8 8 8 -40 0<br class="">10.13.136.154@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.10@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.167@o2ib 1 NA -1 8 8 8 8 2 0<br class="">10.13.136.23@o2ib 1 NA -1 8 8 8 8 -53 0<br class="">10.13.136.2@o2ib 1 NA -1 8 8 8 8 8 0</font><br class=""><div class=""><br class=""></div><br class=""><blockquote type="cite" class=""><br class=""><br class="">On 12/3/17 8:38 AM, E.S. Rosenberg wrote:<br class=""><blockquote type="cite" cite="mid:CA+K1OzT7NQ6JUNOgSyKfNQtqioy0bUs7YMwTNscrV8rcORnVog@mail.gmail.com" class="">Did you find the problem? Were there any useful suggestions off-list?<br class=""><br class="">On Wed, Nov 29, 2017 at 1:34 PM, Charles A Taylor <<a href="mailto:chasman@ufl.edu" class="">chasman@ufl.edu</a>> wrote:<br class=""><br class="">We have a genomics pipeline app (supernova) that fails consistently due to the client being evicted on the OSSs with a “lock callback timer expired”. I doubled “nlm_enqueue_min” across the cluster but then the timer simply expired after 200s rather than 100s so I don’t think that is the answer. The syslog/dmesg on the client shows no signs of distress and it is a “bigmem” machine with 1TB of RAM.<br class=""><br class="">The eviction appears to come while the application is processing a large number (~300) of data “chunks” (i.e. files) which occur in pairs.<br class=""><br class="">-rw-r--r-- 1 chasman ufhpc 24 Nov 28 23:31 ./Tdtest915/ASSEMBLER_CS/_ASSEMBLER/_ASM_SN/SHARD_ASM/fork0/join/files/chunk233.sedge_bcs<br class="">-rw-r--r-- 1 chasman ufhpc 34M Nov 28 23:31 ./Tdtest915/ASSEMBLER_CS/_ASSEMBLER/_ASM_SN/SHARD_ASM/fork0/join/files/chunk233.sedge_asm<br class=""><br class="">I assume the 24-byte file is metadata (an index or some such) and the 34M file is the actual data but I’m just guessing since I’m completely unfamiliar with the application.<br class=""><br class="">The write error is,<br class=""><br class=""> #define ENOTCONN 107 /* Transport endpoint is not connected */<br class=""><br class="">which occurs after the OSS eviction. This was reproducible under 2.5.3.90 as well. We hoped that upgrading to 2.10.1 would resolve the issue but it has not.<br class=""><br class="">This is the first application (in 10 years) we have encountered that consistently and reliably fails when run over Lustre. I’m not sure at this point whether this is a bug or tuning issue.<br class="">If others have encountered and overcome something like this, we’d be grateful to hear from you.<br class=""><br class="">Regards,<br class=""><br class="">Charles Taylor<br class="">UF Research Computing<br class=""><br class="">OSS:<br class="">--------------<br class="">Nov 28 23:41:41 ufrcoss28 kernel: LustreError: 0:0:(ldlm_lockd.c:334:waiting_locks_callback()) ### lock callback timer expired after 201s: evicing client at 10.13.136.74@o2ib ns: filter-testfs-OST002e_UUID lock: ffff880041717400/0x9bd23c8dc69323a1 lrc: 3/0,0 mode: PW/PW res: [0x7ef2:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4096->1802239) flags: 0x60000400010020 nid: 10.13.136.74@o2ib remote: 0xe54f26957f2ac591 expref: 45 pid: 6836 timeout: 6488120506 lvb_type: 0<br class=""><br class="">Client:<br class="">———————<br class="">Nov 28 23:41:42 s5a-s23 kernel: LustreError: 11-0: testfs-OST002e-osc-ffff88c053fe3800: operation ost_write to node 10.13.136.30@o2ib failed: rc = -107<br class="">Nov 28 23:41:42 s5a-s23 kernel: Lustre: testfs-OST002e-osc-ffff88c053fe3800: Connection to testfs-OST002e (at 10.13.136.30@o2ib) was lost; in progress operations using this service will wait for recovery to complete<br class="">Nov 28 23:41:42 s5a-s23 kernel: LustreError: 167-0: testfs-OST002e-osc-ffff88c053fe3800: This client was evicted by testfs-OST002e; in progress operations using this service will fail.<br class="">Nov 28 23:41:42 s5a-s23 kernel: LustreError: 11-0: testfs-OST002c-osc-ffff88c053fe3800: operation ost_punch to node 10.13.136.30@o2ib failed: rc = -107<br class="">Nov 28 23:41:42 s5a-s23 kernel: Lustre: testfs-OST002c-osc-ffff88c053fe3800: Connection to testfs-OST002c (at 10.13.136.30@o2ib) was lost; in progress operations using this service will wait for recovery to complete<br class="">Nov 28 23:41:42 s5a-s23 kernel: LustreError: 167-0: testfs-OST002c-osc-ffff88c053fe3800: This client was evicted by testfs-OST002c; in progress operations using this service will fail.<br class="">Nov 28 23:41:47 s5a-s23 kernel: LustreError: 11-0: testfs-OST0000-osc-ffff88c053fe3800: operation ost_statfs to node 10.13.136.23@o2ib failed: rc = -107<br class="">Nov 28 23:41:47 s5a-s23 kernel: Lustre: testfs-OST0000-osc-ffff88c053fe3800: Connection to testfs-OST0000 (at 10.13.136.23@o2ib) was lost; in progress operations using this service will wait for recovery to complete<br class="">Nov 28 23:41:47 s5a-s23 kernel: LustreError: 167-0: testfs-OST0004-osc-ffff88c053fe3800: This client was evicted by testfs-OST0004; in progress operations using this service will fail.<br class="">Nov 28 23:43:11 s5a-s23 kernel: Lustre: testfs-OST0006-osc-ffff88c053fe3800: Connection restored to 10.13.136.24@o2ib (at 10.13.136.24@o2ib)<br class="">Nov 28 23:43:38 s5a-s23 kernel: Lustre: testfs-OST002c-osc-ffff88c053fe3800: Connection restored to 10.13.136.30@o2ib (at 10.13.136.30@o2ib)<br class="">Nov 28 23:43:45 s5a-s23 kernel: Lustre: testfs-OST0000-osc-ffff88c053fe3800: Connection restored to 10.13.136.23@o2ib (at 10.13.136.23@o2ib)<br class="">Nov 28 23:43:48 s5a-s23 kernel: Lustre: testfs-OST0004-osc-ffff88c053fe3800: Connection restored to 10.13.136.23@o2ib (at 10.13.136.23@o2ib)<br class="">Nov 28 23:43:48 s5a-s23 kernel: Lustre: Skipped 3 previous similar messages<br class="">Nov 28 23:43:55 s5a-s23 kernel: Lustre: testfs-OST0007-osc-ffff88c053fe3800: Connection restored to 10.13.136.24@o2ib (at 10.13.136.24@o2ib)<br class="">Nov 28 23:43:55 s5a-s23 kernel: Lustre: Skipped 4 previous similar messages<br class=""><br class="">Some Details:<br class="">-------------------<br class="">OS: RHEL 7.4 (Linux ufrcoss28.ufhpc 3.10.0-693.2.2.el7_lustre.x86_64)<br class="">Lustre: 2.10.1 (lustre-2.10.1-1.el7.x86_64)<br class="">Client: 2.10.1<br class=""> 1 TB RAM<br class=""> Mellanox ConnectX-3 IB/VPI HCAs<br class=""> Linux s5a-s23.ufhpc 2.6.32-696.13.2.el6.x86_64<br class=""> MOFED 3.2.2 IB stack<br class=""> Lustre 2.10.1<br class="">Servers: 10 HA OSS pairs (20 OSSs)<br class=""> 128 GB RAM<br class=""> 6 OSTs (8+2 RAID-6) per OSS<br class=""> Mellanox ConnectX-3 IB/VPI HCAs<br class=""> RedHat EL7 Native IB Stack (i.e. not MOFED)<br class=""><br class="">_______________________________________________<br class="">lustre-discuss mailing list<br class="">lustre-discuss@lists.lustre.org<br class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class=""><br class=""><br class=""><br class=""><div class="">_______________________________________________</div><div class="">lustre-discuss mailing list</div><br class="Apple-interchange-newline">lustre-discuss@lists.lustre.org<br class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class=""></blockquote><br class=""></blockquote><br class=""></div></body></html>