[Lustre-discuss] Unable to handle kernel paging request at virtual address

Lu Wang wanglu at ihep.ac.cn
Sun Aug 30 18:35:40 PDT 2009


Dear list, 
	According to discussion thread http://groups.google.com/group/lustre-discuss-list/browse_thread/thread/a4517a537beb89f3?hl=en
	I reduce max_cached_mb=1024, and the clients crashed less frequently. However, we found 3 clients dead  yesterday, with log:
	At the end of this log,  from " _spin_unlock" to "kprobe_exceptions_notify" repeated several times, and then the node died.
	Is this caused by the same reason?  
	Aug 30 15:17:55 bws0060 kernel: do_IRQ: stack overflow: 460
Aug 30 15:17:55 bws0060 kernel:  [<c010795b>] do_IRQ+0x49/0x1ae
Aug 30 15:17:55 bws0060 kernel:  [<c02d6c60>] common_interrupt+0x18/0x20
Aug 30 15:17:55 bws0060 kernel:  [<c01c3228>] number+0x148/0x25d
Aug 30 15:17:55 bws0060 kernel:  [<c011cd20>] recalc_task_prio+0x106/0x133
Aug 30 15:17:55 bws0060 kernel:  [<c01c3785>] vsnprintf+0x448/0x488
Aug 30 15:17:55 bws0060 kernel:  [<c01c37f3>] snprintf+0x17/0x1a
Aug 30 15:17:55 bws0060 kernel:  [<f94a08dc>] libcfs_ip_addr2str+0x3c/0x40 [libcfs]
Aug 30 15:17:55 bws0060 kernel:  [<f94a0d0b>] libcfs_nid2str+0x7b/0x140 [libcfs]
Aug 30 15:17:55 bws0060 kernel:  [<f94a105b>] libcfs_id2str+0x2b/0xb0 [libcfs]
Aug 30 15:17:55 bws0060 kernel:  [<f9832434>] ksocknal_queue_tx_locked+0x404/0x630 [ksocklnd]
Aug 30 15:17:55 bws0060 kernel:  [<f982727d>] ksocknal_find_peer_locked+0x14d/0x1b0 [ksocklnd]
Aug 30 15:17:55 bws0060 kernel:  [<f9832889>] ksocknal_launch_packet+0x139/0x5b0 [ksocklnd]
Aug 30 15:17:55 bws0060 kernel:  [<f9832e7d>] ksocknal_send+0x17d/0x430 [ksocklnd]
Aug 30 15:17:55 bws0060 kernel: Unable to handle kernel paging request at virtual address 343ce120
Aug 30 15:17:55 bws0060 kernel:  printing eip:
Aug 30 15:17:55 bws0060 kernel: c011974d
Aug 30 15:17:55 bws0060 kernel: *pde = 1cce7001
Aug 30 15:17:55 bws0060 kernel: Oops: 0000 [#1]
Aug 30 15:17:55 bws0060 kernel: SMP
Aug 30 15:17:55 bws0060 kernel: Modules linked in: mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) nfs lockd nfs_acl blcr(U) blcr_imports(U) libafs(U) md5 ipv6 autofs4 i2c_dev i2c_core sunrpc loop dm_mirror button battery ac uhci_hcd ehci_hcd hw_random bnx2 ext3 jbd dm_mod ata_piix libata mptscsih mptsas mptspi mptscsi mptbase sd_mod scsi_mod
Aug 30 15:17:55 bws0060 kernel: CPU:    6
Aug 30 15:17:55 bws0060 kernel: EIP:    0060:[<c011974d>]    Tainted: PF     VLI
Aug 30 15:17:55 bws0060 kernel: EFLAGS: 00010086   (2.6.9-55.EL.cernsmp)
Aug 30 15:17:55 bws0060 kernel: EIP is at kprobe_exceptions_notify+0x126/0x1fc
Aug 30 15:17:55 bws0060 kernel: eax: c03c21a0   ebx: c032ae3c   ecx: d703b068   edx: 5d000000
Aug 30 15:17:55 bws0060 kernel: esi: d703b068   edi: d703b124   ebp: 80625fde   esp: d703b030
Aug 30 15:17:55 bws0060 kernel: ds: 007b   es: 007b   ss: 0068
Aug 30 15:17:55 bws0060 kernel: Process ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ!^F茫^E& (pid: 15158
Aug 30 15:17:55 bws0060 kernel: 0810, threadinfo=d703a000 t
Aug 30 15:17:55 bws0060 kernel: Stack: 00000000 c032ae3c d703b068 0000000d 80625fde c012de7e c0453c20 00000000
Aug 30 15:17:55 bws0060 kernel:        c011ae6d c011aebf c02f25d9 343ce120 c0122900 c02f25d9 d703b124 c02e766c
Aug 30 15:17:55 bws0060 kernel:        00000000 0000000e 0000000b 0000017d f984530c 636f736b 6c616e6b 6e65735f
Aug 30 15:17:55 bws0060 kernel: Call Trace:
Aug 30 15:17:55 bws0060 kernel:  [<c012de7e>] notifier_call_chain+0x17/0x2e
Aug 30 15:17:55 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:55 bws0060 kernel:  [<c011aebf>] do_page_fault+0x52/0x5c6
Aug 30 15:17:55 bws0060 kernel:  [<c0122900>] printk+0xe/0x11
Aug 30 15:17:55 bws0060 kernel:  [<c011cd42>] recalc_task_prio+0x128/0x133
Aug 30 15:17:55 bws0060 kernel:  [<c011cd42>] recalc_task_prio+0x128/0x133
Aug 30 15:17:55 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:55 bws0060 kernel:  [<c011cd42>] recalc_task_prio+0x128/0x133
Aug 30 15:17:55 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:55 bws0060 kernel:  [<c02d6d7f>] error_code+0x2f/0x38
Aug 30 15:17:56 bws0060 kernel:  [<c020007b>] send_break+0x37/0x5f
Aug 30 15:17:56 bws0060 kernel:  [<c0129dde>] __mod_timer+0x4a/0x10b
Aug 30 15:17:56 bws0060 kernel:  [<c020d9a4>] poke_blanked_console+0x8f/0x9a
Aug 30 15:17:56 bws0060 kernel:  [<c020cd45>] vt_console_print+0x294/0x2a5
Aug 30 15:17:56 bws0060 kernel:  [<c020cab1>] vt_console_print+0x0/0x2a5
Aug 30 15:17:56 bws0060 kernel:  [<c01226e3>] __call_console_drivers+0x36/0x40
Aug 30 15:17:56 bws0060 kernel:  [<c01227fb>] call_console_drivers+0xb6/0xd8
Aug 30 15:17:56 bws0060 kernel:  [<c0122aef>] release_console_sem+0x43/0xa9
Aug 30 15:17:56 bws0060 kernel:  [<c0122a39>] vprintk+0x136/0x14a
Aug 30 15:17:56 bws0060 kernel:  [<f9832e7d>] ksocknal_send+0x17d/0x430 [ksocklnd]
Aug 30 15:17:56 bws0060 kernel:  [<c0122900>] printk+0xe/0x11
Aug 30 15:17:56 bws0060 kernel:  [<c0105d03>] show_trace+0x44/0x6b
Aug 30 15:17:56 bws0060 kernel:  [<c0105db4>] dump_stack+0x11/0x13
Aug 30 15:17:56 bws0060 kernel:  [<c010795b>] do_IRQ+0x49/0x1ae
Aug 30 15:17:56 bws0060 kernel:  [<c02d6c60>] common_interrupt+0x18/0x20
Aug 30 15:17:56 bws0060 kernel:  [<c01c3228>] number+0x148/0x25d
Aug 30 15:17:56 bws0060 kernel:  [<c011cd20>] recalc_task_prio+0x106/0x133
Aug 30 15:17:56 bws0060 kernel:  [<c01c3785>] vsnprintf+0x448/0x488
Aug 30 15:17:56 bws0060 kernel:  [<c01c37f3>] snprintf+0x17/0x1a
Aug 30 15:17:56 bws0060 kernel:  [<f94a08dc>] libcfs_ip_addr2str+0x3c/0x40 [libcfs]
Aug 30 15:17:56 bws0060 kernel:  [<f94a0d0b>] libcfs_nid2str+0x7b/0x140 [libcfs]
Aug 30 15:17:56 bws0060 kernel:  [<f94a105b>] libcfs_id2str+0x2b/0xb0 [libcfs]
Aug 30 15:17:56 bws0060 kernel:  [<f9832434>] ksocknal_queue_tx_locked+0x404/0x630 [ksocklnd]
Aug 30 15:17:56 bws0060 kernel:  [<f982727d>] ksocknal_find_peer_locked+0x14d/0x1b0 [ksocklnd]
Aug 30 15:17:56 bws0060 kernel:  [<f9832889>] ksocknal_launch_packet+0x139/0x5b0 [ksocklnd]
Aug 30 15:17:56 bws0060 kernel:  [<f9832e7d>] ksocknal_send+0x17d/0x430 [ksocklnd]
Aug 30 15:17:56 bws0060 kernel:  [<f97194e1>] lnet_ni_send+0x41/0xd0 [lnet]
Aug 30 15:17:56 bws0060 kernel:  [<f971a4c1>] lnet_send+0x231/0xd20 [lnet]
Aug 30 15:17:56 bws0060 kernel:  [<f971f42d>] LNetPut+0x3fd/0xce0 [lnet]
Aug 30 15:17:56 bws0060 kernel:  [<f9661f30>] ptl_send_buf+0x2a0/0xa80 [ptlrpc]
Aug 30 15:17:56 bws0060 kernel:  [<f9665801>] ptl_send_rpc+0x591/0x1790 [ptlrpc]
Aug 30 15:17:56 bws0060 kernel:  [<c011e851>] __wake_up+0x29/0x3c
Aug 30 15:17:56 bws0060 kernel:  [<f965a22f>] ptlrpc_queue_wait+0x18f/0x2720 [ptlrpc]
Aug 30 15:17:56 bws0060 kernel:  [<c011e851>] __wake_up+0x29/0x3c
Aug 30 15:17:56 bws0060 kernel:  [<f965a22f>] ptlrpc_queue_wait+0x18f/0x2720 [ptlrpc]
Aug 30 15:17:56 bws0060 kernel:  [<f966db2e>] lustre_msg_add_version+0xbe/0x130 [ptlrpc]
Aug 30 15:17:56 bws0060 kernel:  [<f949a359>] cfs_alloc+0x29/0x70 [libcfs]
Aug 30 15:17:56 bws0060 kernel:  [<f96679a3>] lustre_pack_request_v2+0x83/0x3c0 [ptlrpc]
Aug 30 15:17:56 bws0060 kernel:  [<f960ad90>] ldlm_resource_putref+0xa0/0x680 [ptlrpc]
Aug 30 15:17:56 bws0060 kernel:  [<f966fb2e>] lustre_msg_set_opc+0x2e/0x120 [ptlrpc]
Aug 30 15:17:56 bws0060 kernel:  [<f949a359>] cfs_alloc+0x29/0x70 [libcfs]
Aug 30 15:17:56 bws0060 kernel:  [<f965d7bc>] ptlrpc_next_xid+0x3c/0x50 [ptlrpc]
Aug 30 15:17:56 bws0060 kernel:  [<f967021e>] lustre_msg_set_timeout+0x2e/0x100 [ptlrpc]
Aug 30 15:17:56 bws0060 kernel:  [<f966d726>] lustre_msg_get_type+0xd6/0x210 [ptlrpc]
Aug 30 15:17:56 bws0060 kernel:  [<f97da4cb>] mdc_close+0x22b/0xdf0 [mdc]
Aug 30 15:17:56 bws0060 kernel:  [<f98490d3>] ll_release+0xd3/0x600 [lustre]
Aug 30 15:17:56 bws0060 kernel:  [<f985ada2>] ll_close_inode_openhandle+0x152/0xb80 [lustre]
Aug 30 15:17:56 bws0060 kernel:  [<f985b8fb>] ll_mdc_real_close+0x12b/0x520 [lustre]
Aug 30 15:17:56 bws0060 kernel:  [<f98b0504>] ll_mdc_blocking_ast+0x224/0x950 [lustre]
Aug 30 15:17:56 bws0060 kernel:  [<f9647ee5>] ldlm_pool_del+0x75/0x2f0 [ptlrpc]
Aug 30 15:17:56 bws0060 kernel:  [<f95faac7>] ldlm_lock_destroy_nolock+0x87/0x1f0 [ptlrpc]
Aug 30 15:17:56 bws0060 kernel:  [<f966e0ae>] lustre_msg_get_last_committed+0xde/0x220 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f95f9118>] unlock_res_and_lock+0x58/0xe0 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f95f9118>] unlock_res_and_lock+0x58/0xe0 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f960261b>] ldlm_cancel_callback+0x10b/0x160 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f95f9045>] lock_res_and_lock+0x45/0xc0 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f96296b4>] ldlm_cli_cancel_local+0xa4/0x6f0 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f962bb77>] ldlm_cancel_list+0x137/0x360 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f9623f70>] ldlm_completion_ast+0x0/0xdf0 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f96242ab>] ldlm_completion_ast+0x33b/0xdf0 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f962bf32>] ldlm_cancel_lrur_policy+0x92/0x150 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f962c206>] ldlm_cancel_lru_local+0x126/0x480 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f95f9118>] unlock_res_and_lock+0x58/0xe0 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f960befb>] ldlm_resource_unlink_lock+0x4b/0xb0 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f962bea0>] ldlm_cancel_lrur_policy+0x0/0x150 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f96276f0>] ldlm_prep_elc_req+0x2d0/0x550 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f95fea07>] ldlm_lock_match+0x317/0x1010 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f96279ac>] ldlm_prep_enqueue_req+0x3c/0x50 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f97ea13f>] mdc_intent_lookup_pack+0xcf/0x160 [mdc]
Aug 30 15:17:57 bws0060 kernel:  [<f96279ac>] ldlm_prep_enqueue_req+0x3c/0x50 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f97ea13f>] mdc_intent_lookup_pack+0xcf/0x160 [mdc]
Aug 30 15:17:57 bws0060 kernel:  [<f9627e8c>] ldlm_cli_enqueue+0x4cc/0xc70 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f97eb65f>] mdc_enqueue+0x76f/0xd90 [mdc]
Aug 30 15:17:57 bws0060 kernel:  [<f955a01a>] class_handle2object+0x10a/0x2b0 [obdclass]
Aug 30 15:17:57 bws0060 kernel:  [<c0170bc8>] d_rehash+0x53/0x77
Aug 30 15:17:57 bws0060 kernel:  [<f955a01a>] class_handle2object+0x10a/0x2b0 [obdclass]
Aug 30 15:17:57 bws0060 kernel:  [<f98b116b>] ll_d_add+0x7b/0x360 [lustre]
Aug 30 15:17:57 bws0060 kernel:  [<f98afa70>] ll_test_inode+0x0/0x430 [lustre]
Aug 30 15:17:57 bws0060 kernel:  [<f955a01a>] class_handle2object+0x10a/0x2b0 [obdclass]
Aug 30 15:17:57 bws0060 kernel:  [<f97ecd05>] mdc_intent_lock+0x1e5/0x690 [mdc]
Aug 30 15:17:57 bws0060 kernel:  [<f96575ad>] __ptlrpc_free_req+0x1ad/0xbc0 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f98b1ac8>] lookup_it_finish+0x1c8/0x720 [lustre]
Aug 30 15:17:57 bws0060 kernel:  [<f98b02e0>] ll_mdc_blocking_ast+0x0/0x950 [lustre]
Aug 30 15:17:57 bws0060 kernel:  [<f9623f70>] ldlm_completion_ast+0x0/0xdf0 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<c012fb15>] in_group_p+0x31/0x58
Aug 30 15:17:57 bws0060 kernel:  [<f98b02e0>] ll_mdc_blocking_ast+0x0/0x950 [lustre]
Aug 30 15:17:57 bws0060 kernel:  [<f9623f70>] ldlm_completion_ast+0x0/0xdf0 [ptlrpc]
Aug 30 15:17:57 bws0060 kernel:  [<f98b1031>] ll_prepare_mdc_op_data+0x51/0x110 [lustre]
Aug 30 15:17:57 bws0060 kernel:  [<f98b24c7>] ll_lookup_it+0x4a7/0xc10 [lustre]
Aug 30 15:17:57 bws0060 kernel:  [<f98b02e0>] ll_mdc_blocking_ast+0x0/0x950 [lustre]
Aug 30 15:17:57 bws0060 kernel:  [<f98b02e0>] ll_mdc_blocking_ast+0x0/0x950 [lustre]
Aug 30 15:17:57 bws0060 kernel:  [<f94dd12b>] nfs_commit_write+0x69/0x72 [nfs]
Aug 30 15:17:57 bws0060 kernel:  [<c02d47e3>] __cond_resched+0x14/0x39
Aug 30 15:17:57 bws0060 kernel:  [<f98b2c89>] ll_convert_intent+0x59/0x230 [lustre]
Aug 30 15:17:57 bws0060 kernel:  [<f98b2f54>] ll_lookup_nd+0xf4/0x510 [lustre]
Aug 30 15:17:57 bws0060 kernel:  [<c02d47e3>] __cond_resched+0x14/0x39
Aug 30 15:17:57 bws0060 kernel:  [<c017066c>] d_alloc+0x175/0x17d
Aug 30 15:17:57 bws0060 kernel:  [<c0166d43>] real_lookup+0x6e/0xec
Aug 30 15:17:57 bws0060 kernel:  [<c0166f81>] do_lookup+0x5d/0xb1
Aug 30 15:17:57 bws0060 kernel:  [<c0167819>] __link_path_walk+0x844/0xc25
Aug 30 15:17:57 bws0060 kernel:  [<c0167c3d>] link_path_walk+0x43/0xbe
Aug 30 15:17:57 bws0060 kernel:  [<c01c402c>] atomic_dec_and_lock+0x20/0x40
Aug 30 15:17:57 bws0060 kernel:  [<c010b052>] do_gettimeofday+0x1a/0x9c
Aug 30 15:17:57 bws0060 kernel:  [<c0167fd2>] path_lookup+0x14b/0x17f
Aug 30 15:17:58 bws0060 kernel:  [<c016811a>] __user_walk+0x21/0x51
Aug 30 15:17:57 bws0060 kernel:  [<c0167fd2>] path_lookup+0x14b/0x17f
Aug 30 15:17:58 bws0060 kernel:  [<c016811a>] __user_walk+0x21/0x51
Aug 30 15:17:58 bws0060 kernel:  [<c015a3e7>] sys_access+0x8f/0x134
Aug 30 15:17:58 bws0060 kernel:  [<c01c402c>] atomic_dec_and_lock+0x20/0x40
Aug 30 15:17:58 bws0060 kernel:  [<c010b052>] do_gettimeofday+0x1a/0x9c
Aug 30 15:17:58 bws0060 kernel:  [<c02d6287>] syscall_call+0x7/0xb
Aug 30 15:17:58 bws0060 kernel:  [<c02d007b>] unix_stream_sendmsg+0x33/0x33a
Aug 30 15:17:58 bws0060 kernel:  =======================
Aug 30 15:17:58 bws0060 kernel: Unable to handle kernel paging request at virtual address 80625fde
Aug 30 15:17:58 bws0060 kernel:  printing eip:
Aug 30 15:17:58 bws0060 kernel: c0105cd0
Aug 30 15:17:58 bws0060 kernel: *pde = 00000000
Aug 30 15:17:58 bws0060 kernel: Oops: 0000 [#2]
Aug 30 15:17:58 bws0060 kernel: SMP
Aug 30 15:17:58 bws0060 kernel: Modules linked in: mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) nfs lockd nfs_a
Aug 30 15:17:58 bws0060 kernel:  [<c02d6d7f>] error_code+0x2f/0x38
Aug 30 15:17:58 bws0060 kernel:  [<c02d007b>] unix_stream_sendmsg+0x33/0x33a
Aug 30 15:17:58 bws0060 kernel:  [<c02d4e4c>] _spin_unlock+0x1c/0x27
Aug 30 15:17:58 bws0060 kernel:  [<c02d392b>] schedule+0x7f/0x8ec
Aug 30 15:17:58 bws0060 kernel:  [<c020cab1>] vt_console_print+0x0/0x2a5
Aug 30 15:17:58 bws0060 kernel:  [<c0129e95>] __mod_timer+0x101/0x10b
Aug 30 15:17:58 bws0060 kernel:  [<c02d4a06>] schedule_timeout+0x139/0x154
Aug 30 15:17:58 bws0060 kernel:  [<c012a73a>] process_timeout+0x0/0x5
Aug 30 15:17:58 bws0060 kernel:  [<c0122900>] printk+0xe/0x11
Aug 30 15:17:58 bws0060 kernel:  [<c01060c2>] die+0x15a/0x16b
Aug 30 15:17:58 bws0060 kernel:  [<c0122b21>] release_console_sem+0x75/0xa9
Aug 30 15:17:58 bws0060 kernel:  [<c0122a39>] vprintk+0x136/0x14a
Aug 30 15:17:58 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:58 bws0060 kernel:  [<c011b25d>] do_page_fault+0x3f0/0x5c6
Aug 30 15:17:58 bws0060 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Aug 30 15:17:58 bws0060 kernel:  [<c020cab1>] vt_console_print+0x0/0x2a5
Aug 30 15:17:58 bws0060 kernel:  [<c01226e3>] __call_console_drivers+0x36/0x40
Aug 30 15:17:58 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:58 bws0060 kernel:  [<c01226e3>] __call_console_drivers+0x36/0x40
Aug 30 15:17:58 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:58 bws0060 kernel:  [<c02d6d7f>] error_code+0x2f/0x38
Aug 30 15:17:58 bws0060 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Aug 30 15:17:58 bws0060 kernel:  [<c0105d9d>] show_stack+0x73/0x79
Aug 30 15:17:58 bws0060 kernel:  [<c0105e9c>] show_registers+0xe6/0x14d
Aug 30 15:17:58 bws0060 kernel:  [<c0106043>] die+0xdb/0x16b
Aug 30 15:17:58 bws0060 kernel:  [<c0122a39>] vprintk+0x136/0x14a
Aug 30 15:17:58 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:58 bws0060 kernel:  [<c011b25d>] do_page_fault+0x3f0/0x5c6
Aug 30 15:17:58 bws0060 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Aug 30 15:17:58 bws0060 kernel:  [<c020cab1>] vt_console_print+0x0/0x2a5
Aug 30 15:17:58 bws0060 kernel:  [<c01226e3>] __call_console_drivers+0x36/0x40
Aug 30 15:17:58 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:58 bws0060 kernel:  [<c02d6d7f>] error_code+0x2f/0x38
Aug 30 15:17:58 bws0060 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Aug 30 15:17:58 bws0060 kernel:  [<c0105d9d>] show_stack+0x73/0x79
Aug 30 15:17:58 bws0060 kernel:  [<c0105e9c>] show_registers+0xe6/0x14d
Aug 30 15:17:58 bws0060 kernel:  [<c0106043>] die+0xdb/0x16b
Aug 30 15:17:59 bws0060 kernel:  [<c0106425>] do_invalid_op+0xcf/0xf2
Aug 30 15:17:59 bws0060 kernel:  [<c02d4e4c>] _spin_unlock+0x1c/0x27
Aug 30 15:17:59 bws0060 kernel:  [<c0106356>] do_invalid_op+0x0/0xf2
Aug 30 15:17:59 bws0060 kernel:  [<c02d6d7f>] error_code+0x2f/0x38
Aug 30 15:17:59 bws0060 kernel:  [<c02d007b>] unix_stream_sendmsg+0x33/0x33a
Aug 30 15:17:59 bws0060 kernel:  [<c02d4e4c>] _spin_unlock+0x1c/0x27
Aug 30 15:17:59 bws0060 kernel:  [<c02d392b>] schedule+0x7f/0x8ec
Aug 30 15:17:59 bws0060 kernel:  [<c020cab1>] vt_console_print+0x0/0x2a5
Aug 30 15:17:59 bws0060 kernel:  [<c0129e95>] __mod_timer+0x101/0x10b
Aug 30 15:17:59 bws0060 kernel:  [<c02d4a06>] schedule_timeout+0x139/0x154
Aug 30 15:17:59 bws0060 kernel:  [<c012a73a>] process_timeout+0x0/0x5
Aug 30 15:17:59 bws0060 kernel:  [<c0122900>] printk+0xe/0x11
Aug 30 15:17:59 bws0060 kernel:  [<c01060c2>] die+0x15a/0x16b
Aug 30 15:17:59 bws0060 kernel:  [<c0122b21>] release_console_sem+0x75/0xa9
Aug 30 15:17:59 bws0060 kernel:  [<c0122a39>] vprintk+0x136/0x14a
Aug 30 15:17:59 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:59 bws0060 kernel:  [<c011b25d>] do_page_fault+0x3f0/0x5c6
Aug 30 15:17:59 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:59 bws0060 kernel:  [<c011b25d>] do_page_fault+0x3f0/0x5c6
Aug 30 15:17:59 bws0060 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Aug 30 15:17:59 bws0060 kernel:  [<c020cab1>] vt_console_print+0x0/0x2a5
Aug 30 15:17:59 bws0060 kernel:  [<c01226e3>] __call_console_drivers+0x36/0x40
Aug 30 15:17:59 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:59 bws0060 kernel:  [<c02d6d7f>] error_code+0x2f/0x38
Aug 30 15:17:59 bws0060 hm[4031]: Server went down, finding new server.
Aug 30 15:17:59 bws0060 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Aug 30 15:17:59 bws0060 kernel:  [<c0105d9d>] show_stack+0x73/0x79
Aug 30 15:17:59 bws0060 kernel:  [<c0105e9c>] show_registers+0xe6/0x14d
Aug 30 15:17:59 bws0060 kernel:  [<c0106043>] die+0xdb/0x16b
Aug 30 15:17:59 bws0060 kernel:  [<c0122a39>] vprintk+0x136/0x14a
Aug 30 15:17:59 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:59 bws0060 kernel:  [<c011b25d>] do_page_fault+0x3f0/0x5c6
Aug 30 15:17:59 bws0060 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Aug 30 15:17:59 bws0060 kernel:  [<c020cab1>] vt_console_print+0x0/0x2a5
Aug 30 15:17:59 bws0060 kernel:  [<c01226e3>] __call_console_drivers+0x36/0x40
Aug 30 15:17:59 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:59 bws0060 kernel:  [<c02d6d7f>] error_code+0x2f/0x38
Aug 30 15:17:59 bws0060 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Aug 30 15:17:59 bws0060 kernel:  [<c0105d9d>] show_stack+0x73/0x79
Aug 30 15:17:59 bws0060 kernel:  [<c0105e9c>] show_registers+0xe6/0x14d
Aug 30 15:17:59 bws0060 kernel:  [<c0106043>] die+0xdb/0x16b
Aug 30 15:17:59 bws0060 kernel:  [<c0122a39>] vprintk+0x136/0x14a
Aug 30 15:17:59 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:59 bws0060 kernel:  [<c011b25d>] do_page_fault+0x3f0/0x5c6
Aug 30 15:17:59 bws0060 kernel:  [<c011974d>] kprobe_exceptions_notify+0x126/0x1fc
Aug 30 15:17:59 bws0060 kernel:  [<c011cd42>] recalc_task_prio+0x128/0x133
Aug 30 15:17:59 bws0060 kernel:  [<c011d2f4>] try_to_wake_up+0x28e/0x299
Aug 30 15:17:59 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:17:59 bws0060 kernel:  [<c02d6d7f>] error_code+0x2f/0x38
Aug 30 15:17:59 bws0060 kernel:  =======================
Aug 30 15:17:59 bws0060 kernel:  [<c02d4e4c>] _spin_unlock+0x1c/0x27
Aug 30 15:18:00 bws0060 kernel:  [<c0106356>] do_invalid_op+0x0/0xf2
Aug 30 15:18:00 bws0060 kernel:  [<c02d6d7f>] error_code+0x2f/0x38
Aug 30 15:18:00 bws0060 kernel:  [<c02d007b>] unix_stream_sendmsg+0x33/0x33a
Aug 30 15:18:00 bws0060 kernel:  [<c02d4e4c>] _spin_unlock+0x1c/0x27
Aug 30 15:18:00 bws0060 kernel:  [<c02d392b>] schedule+0x7f/0x8ec
Aug 30 15:18:00 bws0060 kernel:  [<c020cab1>] vt_console_print+0x0/0x2a5
Aug 30 15:18:00 bws0060 kernel:  [<c0129e95>] __mod_timer+0x101/0x10b
Aug 30 15:18:00 bws0060 kernel:  [<c02d4a06>] schedule_timeout+0x139/0x154
Aug 30 15:18:00 bws0060 kernel:  [<c012a73a>] process_timeout+0x0/0x5
Aug 30 15:18:00 bws0060 kernel:  [<c0122900>] printk+0xe/0x11
Aug 30 15:18:00 bws0060 kernel:  [<c01060c2>] die+0x15a/0x16b
Aug 30 15:18:00 bws0060 kernel:  [<c0122b21>] release_console_sem+0x75/0xa9
Aug 30 15:18:00 bws0060 kernel:  [<c0122a39>] vprintk+0x136/0x14a
Aug 30 15:18:00 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:18:00 bws0060 kernel:  [<c011b25d>] do_page_fault+0x3f0/0x5c6
Aug 30 15:18:00 bws0060 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Aug 30 15:18:00 bws0060 kernel:  [<c020cab1>] vt_console_print+0x0/0x2a5
Aug 30 15:18:00 bws0060 kernel:  [<c01226e3>] __call_console_drivers+0x show_registers+0xe6/0x14d
Aug 30 15:18:00 bws0060 kernel:  [<c0106043>] die+0xdb/0x16b
Aug 30 15:18:00 bws0060 kernel:  [<c0122a39>] vprintk+0x136/0x14a
Aug 30 15:18:00 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:18:00 bws0060 kernel:  [<c011b25d>] do_page_fault+0x3f0/0x5c6
Aug 30 15:18:00 bws0060 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Aug 30 15:18:00 bws0060 kernel:  [<c020cab1>] vt_console_print+0x0/0x2a5
Aug 30 15:18:00 bws0060 kernel:  [<c01226e3>] __call_console_drivers+0x36/0x40
Aug 30 15:18:00 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:18:00 bws0060 kernel:  [<c02d6d7f>] error_code+0x2f/0x38
Aug 30 15:18:00 bws0060 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Aug 30 15:18:00 bws0060 kernel:  [<c0105d9d>] show_stack+0x73/0x79
Aug 30 15:18:00 bws0060 kernel:  [<c0105e9c>] show_registers+0xe6/0x14d
Aug 30 15:18:00 bws0060 kernel:  [<c0106043>] die+0xdb/0x16b
Aug 30 15:18:00 bws0060 kernel:  [<c0122a39>] vprintk+0x136/0x14a
Aug 30 15:18:00 bws0060 kernel:  [<c011ae6d>] do_page_fault+0x0/0x5c6
Aug 30 15:18:00 bws0060 kernel:  [<c011b25d>] do_page_fault+0x3f0/0x5c6
Aug 30 15:18:00 bws0060 kernel:  [<c011974d>] kprobe_exceptions_notify+0x126/0x1fc
------------------



Best Regards
Lu Wang
--------------------------------------------------------------	  
Computing Center
IHEP						Office: Computing Center,123 
19B Yuquan Road				Tel: (+86) 10 88236012-607
P.O. Box 918-7				Fax: (+86) 10 8823 6839
Beijing 100049,China		Email: Lu.Wang at ihep.ac.cn							
--------------------------------------------------------------   				
                          



More information about the lustre-discuss mailing list