[Lustre-discuss] Lustre Issue

[Nauman Yousuf]

hi

what thiserror means

LustreError: 1545:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ###
enqueue wait took 114978964us from 1296131766 ns: filter-ost1_UUID lock:
df9db940/0x64a022400922c49f lrc: 2/0,0 mode: PR/PR res: 27991312/0 rrc: 3
type: EXT [0->18446744073709551615] (req 0->8191) flags: 0 remote:
0x943ed9a07b21c6f7 expref: 2540 pid: 1649
LustreError: 1545:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ###
enqueue wait took 117994395us from 1296131763 ns: filter-ost1_UUID lock:
c4158240/0x64a022400922c43d lrc: 2/0,0 mode: PR/PR res: 27991312/0 rrc: 3
type: EXT [0->18446744073709551615] (req 0->8191) flags: 0 remote:
0xc4d09fb49f00db51 expref: 2519 pid: 1645
LustreError: 1553:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ###
enqueue wait took 144547977us from 1296131797 ns: filter-ost1_UUID lock:
f7f36a40/0x64a022400922ca2c lrc: 2/0,0 mode: PR/PR res: 28137827/0 rrc: 4
type: EXT [0->18446744073709551615] (req 4096->8191) flags: 0 remote:
0xc4d09fb49f045a40 expref: 2530 pid: 1636
LustreError: 1541:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ###
enqueue wait took 104761600us from 1296131888 ns: filter-ost1_UUID lock:
ee4ad040/0x64a022400922dd03 lrc: 2/0,0 mode: PR/PR res: 27991312/0 rrc: 3
type: EXT [0->18446744073709551615] (req 0->8191) flags: 0 remote:
0x943ed9a07b303bd5 expref: 2555 pid: 1653
LustreError: 1633:0:(ldlm_lib.c:557:target_handle_connect()) @@@ UUID
'ost2_UUID' is not available  for connect (no target)
req at d001ba00x16180978/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1633:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-19)  req at d001ba00 x16180978/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl
Interpret:/0/0 rc -19/0
Lustre: 1689:0:(filter_io_26.c:714:filter_commitrw_write()) ost1: slow
direct_io 32s
Lustre: 1689:0:(filter_io_26.c:727:filter_commitrw_write()) ost1: slow
commitrw commit 32s
Lustre: 1712:0:(filter_io_26.c:714:filter_commitrw_write()) ost1: slow
direct_io 36s
Lustre: 1712:0:(filter_io_26.c:727:filter_commitrw_write()) ost1: slow
commitrw commit 36s
Lustre: 1695:0:(filter_io_26.c:714:filter_commitrw_write()) ost1: slow
direct_io 46s
Lustre: 1695:0:(filter_io_26.c:727:filter_commitrw_write()) ost1: slow
commitrw commit 46s


On Wed, Jan 26, 2011 at 11:53 PM, Brian J. Murrell <brian at whamcloud.com>wrote:

> On Wed, 2011-01-26 at 22:24 +0500, Nauman Yousuf wrote:
> >
>
> Your logs don't have timestamps so it's difficult to correlate events
> but did you notice right before you started getting these messages:
>
>
> > Lustre: 1588:0:(lustre_fsfilt.h:283:fsfilt_setattr()) mds01: slow setattr
> 31s
> > Lustre: 1595:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
> journal start 33s
> > Lustre: 1720:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
> journal start 32s
> > Lustre: 1602:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
> journal start 38s
>
> You got this:
>
> > drbd0: Resync started as SyncSource (need to sync 634747844 KB [158686961
> bits set]).
> > drbd0: Resync done (total 97313 sec; paused 0 sec; 6520 K/sec)
> > drbd0: drbd0_worker [1126]: cstate SyncSource --> Connected
>
> I'm no DRBD expert by a long shot but that looks to me like you had a
> disk in the MDS re-syncing to it's DRBD partner.  If that disk is the
> MDT, a resync, of course is going to slow down the MDT.
>
> The problem here is that you are probably tuned (i.e. the number of
> threads) to expect to full performance out of the hardware and when it's
> under a resync load, it won't deliver it.
>
> Unfortunately at this point Lustre will push it's thread count higher if
> can determine it can get more performance out of a target but it won't
> back off when things slow down (i.e. because the disk is being
> commandeered for housekeeping tasks such as resync or raid rebuild,
> etc.), so you need to maximize your thread count to what performs well
> while your disks are under a resync load.
>
> Please see the operations manual for details on tuning thread counts for
> performance.
>
> Cheers,
> b.
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>


-- 
Regards

Nauman Yousuf
0321-2549206
E-Eager, N-Noble, G-Genuine, I-Intelligent, N-Natural, E-Enthusiastic,
E-Energetic, R-Resourcefull --- ENGINEER
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20110131/e5556dba/attachment.htm>