<div dir="ltr">Hi guys,<div><br></div><div>
<p class="gmail-p1">Seeing a lot of read and write checksum errors, in this case I'm showing the errors while I'm doing md5sum on one single file, stripe count of 1.</p><p class="gmail-p1"><br></p><p class="gmail-p1"> LustreError: 133-1: fs01-OST0010-osc-ffff881decdd7800: BAD READ CHECKSUM: from 192.168.1.9@o2ib inode [0x0:0x0:0x0] object 0x0:634095 extent [71303168-72351743]<br></p><p class="gmail-p1"><span class="gmail-s1"> LustreError: 3126:0:(osc_request.c:1492:osc_brw_fini_request()) client 7b4cff2f, server aa595f33, cksum_type 4</span></p><p class="gmail-p1"><span class="gmail-s1"> LustreError: 3126:0:(osc_request.c:1492:osc_brw_fini_request()) Skipped 490 previous similar messages</span></p><p class="gmail-p1"><span class="gmail-s1">LustreError: 3126:0:(osc_request.c:1528:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff881463f36600 x1551586318841408/t0(0) o3->fs01-OST0010-osc-ffff881decdd7800@192.168.1.9@o2ib:6/4 lens 488/400 e 0 to 0 dl 1479874333 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576</span></p><p class="gmail-p1"><span class="gmail-s1">LustreError: 3126:0:(osc_request.c:1528:osc_brw_redo_request()) Skipped 462 previous similar messages</span></p><p class="gmail-p1"><span class="gmail-s1">LustreError: Skipped 492 previous similar messages</span></p><p class="gmail-p1"><span class="gmail-s1">LustreError: 3107:0:(osc_request.c:1657:brw_interpret()) fs01-OST0007-osc-ffff881decdd7800: too many resent retries for object: 0:634208, rc = -11.</span></p><p class="gmail-p1"><span class="gmail-s1">LustreError: 3107:0:(osc_request.c:1657:brw_interpret()) Skipped 27 previous similar messages</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 30841:0:(mdc_request.c:1484:mdc_read_page()) Page-wide hash collision: 0x0</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 30841:0:(mdc_request.c:1484:mdc_read_page()) Page-wide hash collision: 0x0</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 30841:0:(mdc_request.c:1484:mdc_read_page()) Page-wide hash collision: 0x0</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 30841:0:(mdc_request.c:1484:mdc_read_page()) Page-wide hash collision: 0x0</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 30841:0:(mdc_request.c:1484:mdc_read_page()) Skipped 1 previous similar message</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 30841:0:(mdc_request.c:1484:mdc_read_page()) Page-wide hash collision: 0x0</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 30841:0:(mdc_request.c:1484:mdc_read_page()) Page-wide hash collision: 0x0</span></p><p class="gmail-p1"><span class="gmail-s1"><br></span></p><p class="gmail-p1">This is when I'm trying to unmount fs01:</p><p class="gmail-p1"><span class="gmail-s1">Lustre: 21829:0:(client.c:2048:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1479878225/real 1479878225] req@ffff881656d2c800 x1551586341459476/t0(0) o9->fs01-OST0019-osc-ffff881decdd7800@192.168.1.9@o2ib:28/4 lens 224/224 e 0 to 1 dl 1479878231 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 21829:0:(client.c:2048:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1479878245/real 1479878245] req@ffff880778804800 x1551586345177840/t0(0) o9->fs01-OST0016-osc-ffff881decdd7800@192.168.1.9@o2ib:28/4 lens 224/224 e 0 to 1 dl 1479878251 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 21829:0:(client.c:2048:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1479878258/real 1479878258] req@ffff880778804800 x1551586347659620/t0(0) o9->fs01-OST0014-osc-ffff881decdd7800@192.168.1.6@o2ib:28/4 lens 224/224 e 0 to 1 dl 1479878264 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 21829:0:(client.c:2048:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1479878265/real 1479878265] req@ffff880778804800 x1551586348899860/t0(0) o9->fs01-OST0013-osc-ffff881decdd7800@192.168.1.5@o2ib:28/4 lens 224/224 e 0 to 1 dl 1479878271 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 21829:0:(client.c:2048:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1479878272/real 1479878272] req@ffff880778804800 x1551586350139276/t0(0) o9->fs01-OST0012-osc-ffff881decdd7800@192.168.1.4@o2ib:28/4 lens 224/224 e 0 to 1 dl 1479878278 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 21829:0:(client.c:2048:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1479878286/real 1479878286] req@ffff880778804800 x1551586352619892/t0(0) o9->fs01-OST0010-osc-ffff881decdd7800@192.168.1.9@o2ib:28/4 lens 224/224 e 0 to 1 dl 1479878292 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 21829:0:(client.c:2048:ptlrpc_expire_one_request()) Skipped 1 previous similar message</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 21829:0:(client.c:2048:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1479878306/real 1479878306] req@ffff880778804800 x1551586356341548/t0(0) o9->fs01-OST000d-osc-ffff881decdd7800@192.168.1.5@o2ib:28/4 lens 224/224 e 0 to 1 dl 1479878312 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 21829:0:(client.c:2048:ptlrpc_expire_one_request()) Skipped 1 previous similar message</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 21829:0:(client.c:2048:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1479878347/real 1479878347] req@ffff880778804800 x1551586363782424/t0(0) o9->fs01-OST0007-osc-ffff881decdd7800@192.168.1.5@o2ib:28/4 lens 224/224 e 0 to 1 dl 1479878353 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</span></p><p class="gmail-p1"><span class="gmail-s1">Lustre: 21829:0:(client.c:2048:ptlrpc_expire_one_request()) Skipped 3 previous similar messages</span></p><p class="gmail-p1">
</p><p class="gmail-p1"><span class="gmail-s1">Lustre: Unmounted fs01-client</span></p></div><div><br></div><div>It finishes within 10 to 15 mins, re-mounted fs01, no more checksum errors. </div><div><br></div><div>This happens to multiple clients.</div><div><br></div><div>What I am seeing is client is stuck during the RPC, there's no network issue between client and server, I did lnet selftest.</div><div><br></div><div>I'm trying to troubleshoot the checksum errors, what else should I check?</div><div><br></div><div>Thanks.</div><div><div><br></div>-- <br><div class="gmail-m_-6510914786809745491m_-3352076656329832412gmail_signature"><div dir="ltr">Regards,<div>Gin Tan</div></div></div>
</div></div>