[Lustre-discuss] Lustre file system not mounting

Eric Adint ehadint at nps.edu
Thu Apr 8 16:50:06 PDT 2010


All 
Hopefully someone here can help me with figuring out this problem 
to start 
Linux nas-0-0.local 2.6.18-164.11.1.el5_lustre.1.8.2 #1 SMP Fri Jan 22 19:11:17 MST 2010 x86_64 x86_64 x86_64 GNU/Linux
we are using a HW mirrored 10k sas drive for the MGS/MDS
we have 1 DDN LUN assigned as an ost 
when we mount the mgs and ost we see allot of 
Apr  8 15:41:23 nas-0-0 kernel: LustreError: dumping log to /tmp/lustre-log.1270766483.6273

i cat the 
root at nas-0-0 home-MDT0000]# cat recovery_status  /proc/fs/lustre/mds/home-MDT0000
status: COMPLETE
recovery_start: 1270757997
recovery_duration: 12
delayed_clients: 0/119
completed_clients: 119/119
replayed_requests: 3492
last_transno: 64424512932
[root at nas-0-0 home-MDT0000]# 

but i cannot mount the file system, it hangs and or gives me an endpoint error 

we have run e2fsck and we are considering lfsck but according to the documentation you need to be able to mount the file system and we cant even do that 

here are some of the log file messages we are getting 
Apr  8 13:31:50 nas-0-0 kernel: LustreError: 6514:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 1 previous similar message
Apr  8 13:34:03 nas-0-0 kernel: LustreError: 6474:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16)  req at ffff810645d49400 x1331863949761197/t0 o38->71d90a65-46cf-1c5f-12b9-030426af8970 at NET_0x500000a640108_UUID:0/0 lens 368/264 e 0 to 0 dl 1270758943 ref 1 fl Interpret:/0/0 rc -16/0
Apr  8 13:34:03 nas-0-0 kernel: LustreError: 6474:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 18 previous similar messages
Apr  8 13:38:23 nas-0-0 kernel: LustreError: 6518:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16)  req at ffff81062ccae800 x1331863949761384/t0 o38->71d90a65-46cf-1c5f-12b9-030426af8970 at NET_0x500000a640108_UUID:0/0 lens 368/264 e 0 to 0 dl 1270759203 ref 1 fl Interpret:/0/0 rc -16/0
Apr  8 13:38:23 nas-0-0 kernel: LustreError: 6518:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 36 previous similar messages
Apr  8 13:43:55 nas-0-0 kernel: LustreError: dumping log to /tmp/lustre-log.1270759435.6272
Apr  8 13:46:57 nas-0-0 kernel: LustreError: 6562:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16)  req at ffff81065fba3400 x1331950603635300/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270759717 ref 1 fl Interpret:/0/0 rc -16/0
Apr  8 13:46:57 nas-0-0 kernel: LustreError: 6562:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 8497 previous similar messages
Apr  8 13:55:58 nas-0-0 kernel: LustreError: 6273:0:(mgs_handler.c:660:mgs_handle()) MGS handle cmd=250 rc=-16
Apr  8 13:56:59 nas-0-0 kernel: LustreError: 6514:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16)  req at ffff810632ef1400 x1331950603635761/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270760319 ref 1 fl Interpret:/0/0 rc -16/0
Apr  8 13:56:59 nas-0-0 kernel: LustreError: 6514:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 10148 previous similar messages
Apr  8 14:05:46 nas-0-0 kernel: LustreError: dumping log to /tmp/lustre-log.1270760746.6360
Apr  8 14:07:01 nas-0-0 kernel: LustreError: 6514:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16)  req at ffff81065480b400 x1331950603636207/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270760921 ref 1 fl Interpret:/0/0 rc -16/0
Apr  8 14:07:01 nas-0-0 kernel: LustreError: 6514:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 10147 previous similar messages
Apr  8 14:17:03 nas-0-0 kernel: LustreError: 6299:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16)  req at ffff810634515000 x1331950603636653/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270761523 ref 1 fl Interpret:/0/0 rc -16/0
Apr  8 14:17:03 nas-0-0 kernel: LustreError: 6299:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 10147 previous similar messages
Apr  8 14:27:05 nas-0-0 kernel: LustreError: 6316:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16)  req at ffff81064ee71800 x1331950603637099/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270762125 ref 1 fl Interpret:/0/0 rc -16/0
Apr  8 14:27:05 nas-0-0 kernel: LustreError: 6316:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 10147 previous similar messages
Apr  8 14:37:07 nas-0-0 kernel: LustreError: 6306:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16)  req at ffff81063e665c00 x1331950603637545/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270762727 ref 1 fl Interpret:/0/0 rc -16/0
Apr  8 14:37:07 nas-0-0 kernel: LustreError: 6306:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 10147 previous similar messages
Apr  8 14:47:09 nas-0-0 kernel: LustreError: 6306:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16)  req at ffff810617725000 x1331950603637991/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270763329 ref 1 fl Interpret:/0/0 rc -16/0
Apr  8 14:47:09 nas-0-0 kernel: LustreError: 6306:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 10146 previous similar messages
Apr  8 14:57:11 nas-0-0 kernel: LustreError: 6563:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16)  req at ffff81064cd85400 x1331950603638437/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270763931 ref 1 fl Interpret:/0/0 rc -16/0
Apr  8 14:57:1




Eric Adint
ehadint at nps.edu
HPC specialist  Research Computing  
Naval Postgraduate School
833 Dyer Road Bldg 232 Room 139a
Monterey Ca 93943
831-402-5996




More information about the lustre-discuss mailing list