hey on lustre client i got this error .<br><br><br>LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 17 previous similar messages<br>LustreError: 2208:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server (nid 10.65.200.30@tcp) out of sync -- not fatal, flags 332c90<br>
LustreError: 2208:0:(ldlm_request.c:746:ldlm_cli_cancel()) Skipped 1 previous similar message<br>LustreError: 2208:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel failed: 116<br>LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, err == -2 req@c229bc00 x1219552/t0 o4->ost2_UUID@cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2<br>
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 40 previous similar messages<br>LustreError: 2188:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server (nid 10.65.200.30@tcp) out of sync -- not fatal, flags 332c90<br>
LustreError: 2188:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel failed: 116<br>LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, err == -2 req@c22a3a00 x1219666/t0 o4->ost2_UUID@cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2<br>
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 88 previous similar messages<br>LustreError: 2231:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server (nid 10.65.200.30@tcp) out of sync -- not fatal, flags 332c90<br>
LustreError: 2231:0:(ldlm_request.c:746:ldlm_cli_cancel()) Skipped 2 previous similar messages<br>LustreError: 2231:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel failed: 116<br>LustreError: 2231:0:(file.c:754:ll_extent_lock_callback()) Skipped 2 previous similar messages<br>
<br><br><div class="gmail_quote">On Wed, Jan 26, 2011 at 11:53 PM, Brian J. Murrell <span dir="ltr"><<a href="mailto:brian@whamcloud.com">brian@whamcloud.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
On Wed, 2011-01-26 at 22:24 +0500, Nauman Yousuf wrote:<br>
><br>
<br>
Your logs don't have timestamps so it's difficult to correlate events<br>
but did you notice right before you started getting these messages:<br>
<br>
<br>
> Lustre: 1588:0:(lustre_fsfilt.h:283:fsfilt_setattr()) mds01: slow setattr 31s<br>
> Lustre: 1595:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 33s<br>
> Lustre: 1720:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 32s<br>
> Lustre: 1602:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 38s<br>
<br>
You got this:<br>
<br>
> drbd0: Resync started as SyncSource (need to sync 634747844 KB [158686961 bits set]).<br>
> drbd0: Resync done (total 97313 sec; paused 0 sec; 6520 K/sec)<br>
> drbd0: drbd0_worker [1126]: cstate SyncSource --> Connected<br>
<br>
I'm no DRBD expert by a long shot but that looks to me like you had a<br>
disk in the MDS re-syncing to it's DRBD partner. If that disk is the<br>
MDT, a resync, of course is going to slow down the MDT.<br>
<br>
The problem here is that you are probably tuned (i.e. the number of<br>
threads) to expect to full performance out of the hardware and when it's<br>
under a resync load, it won't deliver it.<br>
<br>
Unfortunately at this point Lustre will push it's thread count higher if<br>
can determine it can get more performance out of a target but it won't<br>
back off when things slow down (i.e. because the disk is being<br>
commandeered for housekeeping tasks such as resync or raid rebuild,<br>
etc.), so you need to maximize your thread count to what performs well<br>
while your disks are under a resync load.<br>
<br>
Please see the operations manual for details on tuning thread counts for<br>
performance.<br>
<br>
Cheers,<br>
<font color="#888888">b.<br>
<br>
</font><br>_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br></blockquote></div><br><br clear="all"><br><br>