[ 78.216468] Lustre: OBD class driver, info@clusterfs.com [ 78.217547] Lustre Version: 1.6.4.3 [ 78.218393] Build Version: 1.6.4.3-19700101010000-PRISTINE-.home.goldbach.Build-lustre-server-kernel.linux-2.6.18.8-2.6.18.8-bnx2-1.6.7b-cciss-3.6.18-5-lustre-1.6.4.3 [ 78.299294] Lustre: Added LNI 172.16.4.69@tcp [8/256] [ 78.300363] Lustre: Accept secure, port 988 [ 78.393841] Lustre: Lustre Client File System; info@clusterfs.com [ 78.453938] kjournald starting. Commit interval 5 seconds [ 78.461664] LDISKFS FS on dm-0, internal journal [ 78.462675] LDISKFS-fs: recovery complete. [ 78.469931] LDISKFS-fs: mounted filesystem with ordered data mode. [ 78.529838] kjournald starting. Commit interval 5 seconds [ 78.536134] LDISKFS FS on dm-0, internal journal [ 78.537334] LDISKFS-fs: mounted filesystem with ordered data mode. [ 78.594288] Lustre: MGS MGS started [ 78.595297] Lustre: Server MGS on device /dev/lustre_pool/mgt has started [ 78.676788] kjournald starting. Commit interval 5 seconds [ 78.681816] LDISKFS FS on dm-1, internal journal [ 78.682987] LDISKFS-fs: recovery complete. [ 78.688720] LDISKFS-fs: mounted filesystem with ordered data mode. [ 78.757422] kjournald starting. Commit interval 5 seconds [ 78.765206] LDISKFS FS on dm-1, internal journal [ 78.766255] LDISKFS-fs: mounted filesystem with ordered data mode. [ 78.864505] Lustre: Enabling user_xattr [ 78.907525] Lustre: 1361:0:(mds_fs.c:446:mds_init_server_data()) RECOVERY: service iloapp3-MDT0000, 3 recoverable clients, last_transno 158566404 [ 78.952173] Lustre: MDT iloapp3-MDT0000 now serving dev (iloapp3-MDT0000/1455817d-55a2-a694-4403-7abfdae1606f), but will be in recovery until 3 clients reconnect, or if no clients reconnect for 4:10; during that time new clients will not be allowed to connect. Recovery progress can be monitored by watching /proc/fs/lustre/mds/iloapp3-MDT0000/recovery_status. [ 78.978179] Lustre: 1361:0:(mds_lov.c:858:mds_notify()) MDS iloapp3-MDT0000: in recovery, not resetting orphans on iloapp3-OST0000_UUID [ 79.012936] BUG: scheduling while atomic: mount.lustre/0xffff8101/1361 [ 79.014247] [ 79.014247] Call Trace: [ 79.015240] [] __sched_text_start+0x7a/0x769 [ 79.016616] [] lock_timer_base+0x1b/0x3c [ 79.017943] [] del_timer+0x4e/0x57 [ 79.018971] [] sync_buffer+0x0/0x3f [ 79.020003] [] io_schedule+0x28/0x34 [ 79.021235] [] sync_buffer+0x3b/0x3f [ 79.022490] [] __wait_on_bit+0x40/0x6f [ 79.023780] [] sync_buffer+0x0/0x3f [ 79.024955] [] out_of_line_wait_on_bit+0x6c/0x78 [ 79.026373] [] wake_bit_function+0x0/0x23 [ 79.027743] [] __bread+0x62/0x77 [ 79.028724] [] :ldiskfs:read_block_bitmap+0xa2/0xf0 [ 79.030245] [] :ldiskfs:ldiskfs_free_blocks_sb+0x115/0x510 [ 79.031856] [] :ldiskfs:ldiskfs_free_blocks+0x91/0xe0 [ 79.033392] [] :ldiskfs:ldiskfs_free_data+0x8a/0x110 [ 79.034944] [] :ldiskfs:ldiskfs_truncate+0x20c/0x650 [ 79.036261] [] start_this_handle+0x355/0x405 [ 79.037705] [] :ldiskfs:ldiskfs_delete_inode+0x84/0xf0 [ 79.039043] [] :ldiskfs:ldiskfs_delete_inode+0x0/0xf0 [ 79.040636] [] generic_delete_inode+0x8e/0x10b [ 79.042042] [] :mds:mds_obd_destroy+0xa11/0xad0 [ 79.043435] [] mntput_no_expire+0x19/0x8b [ 79.044727] [] :obdclass:llog_lvfs_close+0x6b/0x130 [ 79.046201] [] :obdclass:llog_lvfs_destroy+0x841/0xa10 [ 79.047756] [] :obdclass:llog_cat_id2handle+0x4cf/0x5f0 [ 79.049289] [] cache_grow+0x2ee/0x343 [ 79.050501] [] :obdclass:cat_cancel_cb+0x405/0x630 [ 79.051984] [] :obdclass:llog_process+0xa09/0xe20 [ 79.053398] [] dput+0x23/0x152 [ 79.054539] [] :obdclass:cat_cancel_cb+0x0/0x630 [ 79.055959] [] :obdclass:llog_obd_origin_setup+0x773/0x980 [ 79.057507] [] vsnprintf+0x55e/0x5a3 [ 79.058716] [] :obdclass:llog_setup+0x78d/0x860 [ 79.060114] [] :osc:osc_llog_init+0x104/0x390 [ 79.061491] [] :obdclass:__llog_ctxt_put+0x25/0xe0 [ 79.062957] [] :obdclass:obd_llog_init+0x179/0x210 [ 79.064362] [] :lov:lov_llog_init+0x2ca/0x400 [ 79.065556] [] :obdclass:obd_llog_init+0x179/0x210 [ 79.067021] [] mntput_no_expire+0x19/0x8b [ 79.068313] [] :mds:mds_llog_init+0x1ad/0x270 [ 79.069684] [] map_vm_area+0x229/0x2a8 [ 79.070777] [] :obdclass:obd_llog_init+0x179/0x210 [ 79.072056] [] __vmalloc_area_node+0x12b/0x153 [ 79.073520] [] :obdclass:llog_cat_initialize+0x3b5/0x670 [ 79.075090] [] :lov:lov_get_info+0x9f1/0xaa0 [ 79.076405] [] :lov:qos_add_tgt+0x681/0x730 [ 79.077573] [] :mds:mds_lov_update_desc+0xbcc/0xd30 [ 79.079050] [] :mds:mds_notify+0x36c/0x690 [ 79.080630] [] :lov:lov_notify+0xa97/0xfd0 [ 79.081820] [] :lov:lov_add_target+0xbaf/0xd20 [ 79.083220] [] :lov:lov_process_config+0x2f5/0x5a0 [ 79.084497] [] :obdclass:class_setup+0xa25/0xbe0 [ 79.085998] [] :obdclass:class_process_config+0x1873/0x19f0 [ 79.087645] [] :obdclass:class_config_llog_handler+0x153a/0x1990 [ 79.089368] [] do_filp_open+0x2d/0x3d [ 79.090595] [] :obdclass:llog_lvfs_next_block+0x2ac/0x710 [ 79.092190] [] :obdclass:llog_process+0xa09/0xe20 [ 79.093551] [] :libcfs:cfs_alloc+0x28/0x60 [ 79.094940] [] :obdclass:class_config_llog_handler+0x0/0x1990 [ 79.096723] [] :obdclass:class_config_parse_llog+0x43f/0x5b0 [ 79.098205] [] dput+0x34/0x152 [ 79.099302] [] :lvfs:lustre_rename+0x482/0x530 [ 79.100706] [] :obdclass:llog_close+0x1aa/0x230 [ 79.102093] [] :mgc:mgc_process_log+0x20f3/0x2640 [ 79.103529] [] :mgc:mgc_blocking_ast+0x0/0x450 [ 79.104931] [] :ptlrpc:ldlm_completion_ast+0x0/0x6a0 [ 79.106244] [] :mgc:config_log_find+0x19c/0x340 [ 79.107469] [] :mgc:mgc_process_config+0xe02/0x1280 [ 79.109041] [] :obdclass:lustre_process_log+0xb2c/0xee0 [ 79.110592] [] :obdclass:server_find_mount+0x80/0x190 [ 79.112100] [] :obdclass:server_start_targets+0xb36/0x17e0 [ 79.113604] [] __up_write+0x21/0x10d [ 79.114807] [] :obdclass:server_fill_super+0x18c7/0x1ee0 [ 79.116362] [] __d_lookup+0xb0/0x100 [ 79.117492] [] :libcfs:cfs_alloc+0x28/0x60 [ 79.118646] [] :obdclass:lustre_init_lsi+0x29f/0x660 [ 79.119958] [] :obdclass:lustre_fill_super+0x0/0x1ae0 [ 79.121468] [] :obdclass:lustre_fill_super+0x1963/0x1ae0 [ 79.123025] [] set_anon_super+0x3c/0xab [ 79.124281] [] set_anon_super+0x0/0xab [ 79.131643] [] :obdclass:lustre_fill_super+0x0/0x1ae0 [ 79.133004] [] get_sb_nodev+0x4f/0x97 [ 79.134075] [] vfs_kern_mount+0x93/0x11a [ 79.135339] [] do_kern_mount+0x36/0x4d [ 79.136603] [] do_mount+0x68c/0x6ff [ 79.137854] [] __handle_mm_fault+0x530/0x91a [ 79.139028] [] remove_vma+0x55/0x5c [ 79.140201] [] __up_read+0x13/0x8a [ 79.141202] [] do_page_fault+0x3d1/0x706 [ 79.142395] [] do_path_lookup+0x268/0x28c [ 79.143681] [] zone_statistics+0x3e/0x6d [ 79.145026] [] __alloc_pages+0x5c/0x29b [ 79.146265] [] sys_mount+0x8a/0xd7 [ 79.147348] [] system_call+0x7e/0x83 [ 79.148544] [ 79.148991] Unable to handle kernel paging request at 00000000dc6bd3c0 RIP: [ 79.150031] [] __sched_text_start+0x466/0x769 [ 79.152010] PGD 12a825067 PUD 0 [ 79.152857] Oops: 0000 [1] SMP [ 79.153722] CPU 0 [ 79.154243] Modules linked in: osc mds fsfilt_ldiskfs mgs mgc ldiskfs crc16 lustre lov lquota mdc ksocklnd ptlrpc obdclass lnet lvfs libcfs bonding dm_snapshot dm_mirror dm_mod generic piix ehci_hcd uhci_hcd serio_raw ide_core [ 79.159949] Pid: 1361, comm: mount.lustre Not tainted 2.6.18.8-bnx2-1.6.7b-cciss-3.6.18-5-lustre-1.6.4.3 #2 [ 79.161839] RIP: 0010:[] [] __sched_text_start+0x466/0x769 [ 79.163876] RSP: 0018:ffff810129416428 EFLAGS: 00010083 [ 79.165063] RAX: ffffffff80565b80 RBX: ffff8100010232a0 RCX: ffff81012a7b4890 [ 79.166653] RDX: 000000002b833000 RSI: 000000126da5b456 RDI: ffff81012a7b4890 [ 79.168235] RBP: ffff8101294164b8 R08: 000000000000000e R09: 0000000000000001 [ 79.169873] R10: 0000000500000000 R11: ffffffff80308b2e R12: ffff81012a7b48d0 [ 79.171286] R13: 0000000000000000 R14: ffff8100010232a0 R15: 000000126da5b456 [ 79.172891] FS: 00002b1bc60256d0(0000) GS:ffffffff8051a000(0000) knlGS:0000000000000000 [ 79.174483] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 79.175868] CR2: 00000000dc6bd3c0 CR3: 000000012a470000 CR4: 00000000000006e0 [ 79.177265] Process mount.lustre (pid: 1361, threadinfo ffff810129416000, task ffff81012a7b4890) [ 79.179246] Stack: 0000000000000082 ffffffff8023ab95 0000000000000005 ffff81012a7b4890 [ 79.181207] ffff81012a7b4890 00000000081c78be ffff81012a7b4a70 ffffc20000018080 [ 79.183084] 0000000000000082 ffff81012bb25928 ffff81012aa4a2f0 ffff810129416548 [ 79.184872] Call Trace: [ 79.185601] [] lock_timer_base+0x1b/0x3c [ 79.186856] [] sync_buffer+0x0/0x3f [ 79.188000] [] io_schedule+0x28/0x34 [ 79.189216] [] sync_buffer+0x3b/0x3f [ 79.190405] [] __wait_on_bit+0x40/0x6f [ 79.191691] [] sync_buffer+0x0/0x3f [ 79.192848] [] out_of_line_wait_on_bit+0x6c/0x78 [ 79.194199] [] wake_bit_function+0x0/0x23 [ 79.195488] [] __bread+0x62/0x77 [ 79.196605] [] :ldiskfs:read_block_bitmap+0xa2/0xf0 [ 79.198071] [] :ldiskfs:ldiskfs_free_blocks_sb+0x115/0x510 [ 79.199681] [] :ldiskfs:ldiskfs_free_blocks+0x91/0xe0 [ 79.201172] [] :ldiskfs:ldiskfs_free_data+0x8a/0x110 [ 79.202645] [] :ldiskfs:ldiskfs_truncate+0x20c/0x650 [ 79.203937] [] start_this_handle+0x355/0x405 [ 79.205123] [] :ldiskfs:ldiskfs_delete_inode+0x84/0xf0 [ 79.206676] [] :ldiskfs:ldiskfs_delete_inode+0x0/0xf0 [ 79.208188] [] generic_delete_inode+0x8e/0x10b [ 79.209572] [] :mds:mds_obd_destroy+0xa11/0xad0 [ 79.211006] [] mntput_no_expire+0x19/0x8b [ 79.212383] [] :obdclass:llog_lvfs_close+0x6b/0x130 [ 79.213856] [] :obdclass:llog_lvfs_destroy+0x841/0xa10 [ 79.215684] [] :obdclass:llog_cat_id2handle+0x4cf/0x5f0 [ 79.217022] [] cache_grow+0x2ee/0x343 [ 79.218241] [] :obdclass:cat_cancel_cb+0x405/0x630 [ 79.219828] [] :obdclass:llog_process+0xa09/0xe20 [ 79.221330] [] dput+0x23/0x152 [ 79.222420] [] :obdclass:cat_cancel_cb+0x0/0x630 [ 79.223661] [] :obdclass:llog_obd_origin_setup+0x773/0x980 [ 79.225255] [] vsnprintf+0x55e/0x5a3 [ 79.226488] [] :obdclass:llog_setup+0x78d/0x860 [ 79.227893] [] :osc:osc_llog_init+0x104/0x390 [ 79.229275] [] :obdclass:__llog_ctxt_put+0x25/0xe0 [ 79.230739] [] :obdclass:obd_llog_init+0x179/0x210 [ 79.232202] [] :lov:lov_llog_init+0x2ca/0x400 [ 79.233576] [] :obdclass:obd_llog_init+0x179/0x210 [ 79.234861] [] mntput_no_expire+0x19/0x8b [ 79.236221] [] :mds:mds_llog_init+0x1ad/0x270 [ 79.237588] [] map_vm_area+0x229/0x2a8 [ 79.238814] [] :obdclass:obd_llog_init+0x179/0x210 [ 79.240305] [] __vmalloc_area_node+0x12b/0x153 [ 79.241744] [] :obdclass:llog_cat_initialize+0x3b5/0x670 [ 79.243379] [] :lov:lov_get_info+0x9f1/0xaa0 [ 79.244551] [] :lov:qos_add_tgt+0x681/0x730 [ 79.245954] [] :mds:mds_lov_update_desc+0xbcc/0xd30 [ 79.247431] [] :mds:mds_notify+0x36c/0x690 [ 79.248593] [] :lov:lov_notify+0xa97/0xfd0 [ 79.249895] [] :lov:lov_add_target+0xbaf/0xd20 [ 79.251104] [] :lov:lov_process_config+0x2f5/0x5a0 [ 79.252560] [] :obdclass:class_setup+0xa25/0xbe0 [ 79.253987] [] :obdclass:class_process_config+0x1873/0x19f0 [ 79.255750] [] :obdclass:class_config_llog_handler+0x153a/0x1990 [ 79.257435] [] do_filp_open+0x2d/0x3d [ 79.258734] [] :obdclass:llog_lvfs_next_block+0x2ac/0x710 [ 79.260274] [] :obdclass:llog_process+0xa09/0xe20 [ 79.261527] [] :libcfs:cfs_alloc+0x28/0x60 [ 79.262699] [] :obdclass:class_config_llog_handler+0x0/0x1990 [ 79.264384] [] :obdclass:class_config_parse_llog+0x43f/0x5b0 [ 79.266025] [] dput+0x34/0x152 [ 79.266984] [] :lvfs:lustre_rename+0x482/0x530 [ 79.268363] [] :obdclass:llog_close+0x1aa/0x230 [ 79.269620] [] :mgc:mgc_process_log+0x20f3/0x2640 [ 79.271057] [] :mgc:mgc_blocking_ast+0x0/0x450 [ 79.272465] [] :ptlrpc:ldlm_completion_ast+0x0/0x6a0 [ 79.273927] [] :mgc:config_log_find+0x19c/0x340 [ 79.275393] [] :mgc:mgc_process_config+0xe02/0x1280 [ 79.276901] [] :obdclass:lustre_process_log+0xb2c/0xee0 [ 79.278350] [] :obdclass:server_find_mount+0x80/0x190 [ 79.279679] [] :obdclass:server_start_targets+0xb36/0x17e0 [ 79.281332] [] __up_write+0x21/0x10d [ 79.282569] [] :obdclass:server_fill_super+0x18c7/0x1ee0 [ 79.284209] [] __d_lookup+0xb0/0x100 [ 79.285443] [] :libcfs:cfs_alloc+0x28/0x60 [ 79.286748] [] :obdclass:lustre_init_lsi+0x29f/0x660 [ 79.288061] [] :obdclass:lustre_fill_super+0x0/0x1ae0 [ 79.289689] [] :obdclass:lustre_fill_super+0x1963/0x1ae0 [ 79.291253] [] set_anon_super+0x3c/0xab [ 79.292509] [] set_anon_super+0x0/0xab [ 79.293753] [] :obdclass:lustre_fill_super+0x0/0x1ae0 [ 79.295307] [] get_sb_nodev+0x4f/0x97 [ 79.296378] [] vfs_kern_mount+0x93/0x11a [ 79.297503] [] do_kern_mount+0x36/0x4d [ 79.298573] [] do_mount+0x68c/0x6ff [ 79.299597] [] __handle_mm_fault+0x530/0x91a [ 79.300980] [] remove_vma+0x55/0x5c [ 79.302120] [] __up_read+0x13/0x8a [ 79.303274] [] do_page_fault+0x3d1/0x706 [ 79.304385] [] do_path_lookup+0x268/0x28c [ 79.305672] [] zone_statistics+0x3e/0x6d [ 79.306937] [] __alloc_pages+0x5c/0x29b [ 79.308172] [] sys_mount+0x8a/0xd7 [ 79.309361] [] system_call+0x7e/0x83 [ 79.310598] [ 79.310975] [ 79.310975] Code: 48 8b 14 d5 c0 53 52 80 48 03 42 08 c7 40 08 01 00 00 00 49 [ 79.313904] RIP [] __sched_text_start+0x466/0x769 [ 79.315396] RSP [ 79.316117] CR2: 00000000dc6bd3c0 [ 79.316939] <3>LustreError: 1266:0:(client.c:975:ptlrpc_expire_one_request()) @@@ timeout (sent at 1212696480, 5s ago) req@ffff81012aa50200 x7/t0 o8->iloapp3-OST0000_UUID@172.16.4.70@tcp:6 lens 240/272 ref 1 fl Rpc:/0/0 rc 0/-22 [ 95.018738] warning: many lost ticks. [ 95.018739] Your time source seems to be instable or some driver is hogging interupts [ 95.021152] rip mwait_idle+0x36/0x4a [ 95.021973] Falling back to HPET