<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=us-ascii" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 8.00.6001.19120"><LINK rel=stylesheet
href="BLOCKQUOTE{margin-Top: 0px; margin-Bottom: 0px; margin-Left: 2em}"></HEAD>
<BODY
style="MARGIN-TOP: 10px; FONT-FAMILY: Verdana; COLOR: #000000; MARGIN-LEFT: 10px; FONT-SIZE: 10pt; MARGIN-RIGHT: 10px">
<DIV>Hi all, </DIV>
<DIV> </DIV>
<DIV>
One of our OSS with Lustre version:"2.6.18-194.17.1.el5_lustre.1.8.5 " crashed today. </DIV>
<DIV> </DIV>
<DIV> crash> sys</DIV>
<DIV> KERNEL: /usr/lib/debug/lib/modules/2.6.18-194.17.1.el5_lustre.1.8.5//vmlinux</DIV>
<DIV> DUMPFILE: /var/crash/2011-10-10-15:30/vmcore</DIV>
<DIV> CPUS: 8</DIV>
<DIV> DATE: Mon Oct 10 15:29:15 2011</DIV>
<DIV> UPTIME: 05:30:52</DIV>
<DIV>LOAD AVERAGE: 3.74, 2.14, 1.57</DIV>
<DIV> TASKS: 983</DIV>
<DIV> NODENAME: </DIV>
<DIV> RELEASE: 2.6.18-194.17.1.el5_lustre.1.8.5</DIV>
<DIV> VERSION: #1 SMP Mon Nov 15 15:48:43 MST 2010</DIV>
<DIV> MACHINE: x86_64 (2399 Mhz)</DIV>
<DIV> MEMORY: 23.6 GB</DIV>
<DIV> <FONT
color=#ff0000>PANIC: "Oops: 0000 [1] SMP " (check log for details)</FONT></DIV>
<DIV></DIV>
<DIV></DIV>
<DIV></DIV>
<DIV> </DIV>
<DIV>Here is the end of crash dump log:</DIV>
<DIV> </DIV>
<DIV></DIV>
<DIV><FONT
color=#ff0000>Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP: </FONT></DIV>
<DIV> [<ffffffff8895e7c6>] :obdfilter:filter_preprw+0x1746/0x1e00</DIV>
<DIV>PGD 2f8e86067 PUD 31c2c4067 PMD 0 </DIV>
<DIV>Oops: 0000 [1] SMP </DIV>
<DIV>last sysfs file: /devices/pci0000:00/0000:00:00.0/irq</DIV>
<DIV>CPU 4 </DIV>
<DIV>Modules linked in: autofs4(U) hidp(U) obdfilter(U) fsfilt_ldiskfs(U) ost(U) mgc(</DIV>
<DIV>U) ldiskfs(U) jbd2(U) crc16(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ksocklnd</DIV>
<DIV>(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) rfcomm(U) l2cap(U) bluetooth</DIV>
<DIV>(U) sunrpc(U) dm_multipath(U) scsi_dh(U) video(U) backlight(U) sbs(U) power_mete</DIV>
<DIV>r(U) hwmon(U) i2c_ec(U) i2c_core(U) dell_wmi(U) wmi(U) button(U) battery(U) asus</DIV>
<DIV>_acpi(U) acpi_memhotplug(U) ac(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) parport_pc</DIV>
<DIV>(U) lp(U) parport(U) joydev(U) ixgbe(U) 8021q(U) hpilo(U) sg(U) shpchp(U) dca(U)</DIV>
<DIV> serio_raw(U) pcspkr(U) bnx2(U) dm_raid45(U) dm_message(U) dm_region_hash(U) dm_</DIV>
<DIV>mem_cache(U) dm_snapshot(U) dm_zero(U) dm_mirror(U) dm_log(U) dm_mod(U) usb_stor</DIV>
<DIV>age(U) lpfc(U) scsi_transport_fc(U) cciss(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U</DIV>
<DIV>) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U)</DIV>
<DIV>Pid: 4252, comm: ll_ost_io_12 Tainted: G 2.6.18-194.17.1.el5_lustre.1.8.5 #</DIV>
<DIV>1</DIV>
<DIV>RIP: 0010:[<ffffffff8895e7c6>] [<ffffffff8895e7c6>] :obdfilter:filter_preprw+0x</DIV>
<DIV>1746/0x1e00</DIV>
<DIV>RSP: 0018:ffff81030bdcd8c0 EFLAGS: 00010206</DIV>
<DIV>RAX: 0000000000000021 RBX: 0000000000000000 RCX: ffff810011017300</DIV>
<DIV>RDX: ffff8101067b4c90 RSI: 000000000000000e RDI: 3533313130323331</DIV>
<DIV>RBP: ffff81030bdd1388 R08: ffff81061ff40b03 R09: 0000000000001000</DIV>
<DIV>R10: 0000000000000000 R11: 00000000000200d2 R12: 000000000000007e</DIV>
<DIV>R13: 000000000007e000 R14: 0000000000000100 R15: 0000000000000100</DIV>
<DIV>FS: 00002ba793935220(0000) GS:ffff81010af99240(0000) knlGS:0000000000000000</DIV>
<DIV>CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b</DIV>
<DIV>CR2: 0000000000000000 CR3: 00000002f92d1000 CR4: 00000000000006e0</DIV>
<DIV>Process ll_ost_io_12 (pid: 4252, threadinfo ffff81030bdcc000, task ffff81030bd17</DIV>
<DIV>080)</DIV>
<DIV>Stack: 0000000000000000 ffff81031fc503c0 ffff8102c44c7200 0000000000000000</DIV>
<DIV> ffff81031fc503c0 00020000c0a83281 00020000ca7a214e ffffffff88539543</DIV>
<DIV> 0000000000000000 ffffffff885a0d80 ffff8102c44c7200 ffffffff8853ba03</DIV>
<DIV>Call Trace:</DIV>
<DIV> [<ffffffff88539543>] :lnet:lnet_ni_send+0x93/0xd0</DIV>
<DIV> [<ffffffff885a0d80>] :obdclass:class_handle2object+0xe0/0x170</DIV>
<DIV> [<ffffffff8853ba03>] :lnet:lnet_send+0x9a3/0x9d0</DIV>
<DIV> [<ffffffff8002b84a>] truncate_inode_pages_range+0x222/0x2ba</DIV>
<DIV> [<ffffffff88908ffc>] :ost:ost_brw_write+0xf9c/0x2480</DIV>
<DIV> [<ffffffff8864a658>] :ptlrpc:ptlrpc_send_reply+0x5c8/0x5e0</DIV>
<DIV> [<ffffffff886158b0>] :ptlrpc:target_committed_to_req+0x40/0x120</DIV>
<DIV> [<ffffffff8864eb05>] :ptlrpc:lustre_msg_get_version+0x35/0xf0</DIV>
<DIV> [<ffffffff8864ea15>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0</DIV>
<DIV> [<ffffffff8008cf93>] default_wake_function+0x0/0xe</DIV>
<DIV> [<ffffffff8864ebc8>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20</DIV>
<DIV> [<ffffffff8890d08e>] :ost:ost_handle+0x2bae/0x55b0</DIV>
<DIV> [<ffffffff80150d56>] __next_cpu+0x19/0x28</DIV>
<DIV> [<ffffffff800767ae>] smp_send_reschedule+0x4e/0x53</DIV>
<DIV> [<ffffffff8865e15a>] :ptlrpc:ptlrpc_server_handle_request+0x97a/0xdf0</DIV>
<DIV> [<ffffffff8865e8a8>] :ptlrpc:ptlrpc_wait_event+0x2d8/0x310</DIV>
<DIV> [<ffffffff8008b3bd>] __wake_up_common+0x3e/0x68</DIV>
<DIV> [<ffffffff8865f817>] :ptlrpc:ptlrpc_main+0xf37/0x10f0</DIV>
<DIV> [<ffffffff8005dfb1>] child_rip+0xa/0x11</DIV>
<DIV> [<ffffffff8865e8e0>] :ptlrpc:ptlrpc_main+0x0/0x10f0</DIV>
<DIV> [<ffffffff8005dfa7>] child_rip+0x0/0x11</DIV>
<DIV></DIV>
<DIV></DIV>
<DIV>Code: 44 39 23 7e 0c 48 83 c5 28 e9 56 fd ff ff 45 31 ed 48 8d bc </DIV>
<DIV>RIP [<ffffffff8895e7c6>] :obdfilter:filter_preprw+0x1746/0x1e00</DIV>
<DIV> RSP <ffff81030bdcd8c0></DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV></DIV>
<DIV></DIV>
<DIV>and here is bt result:</DIV>
<DIV></DIV>
<DIV>crash> bt</DIV>
<DIV>PID: 4252 TASK: ffff81030bd17080 CPU: 4 COMMAND: "ll_ost_io_12"</DIV>
<DIV> #0 [ffff81030bdcd620] crash_kexec at ffffffff800ad9c4</DIV>
<DIV> #1 [ffff81030bdcd6e0] __die at ffffffff80065157</DIV>
<DIV> #2 [ffff81030bdcd720] do_page_fault at ffffffff80066dd7</DIV>
<DIV> #3 [ffff81030bdcd810] error_exit at ffffffff8005dde9</DIV>
<DIV> [exception RIP: filter_preprw+5958]</DIV>
<DIV> RIP: ffffffff8895e7c6 RSP: ffff81030bdcd8c0 RFLAGS: 00010206</DIV>
<DIV> RAX: 0000000000000021 RBX: 0000000000000000 RCX: ffff810011017300</DIV>
<DIV> RDX: ffff8101067b4c90 RSI: 000000000000000e RDI: 3533313130323331</DIV>
<DIV> RBP: ffff81030bdd1388 R8: ffff81061ff40b03 R9: 0000000000001000</DIV>
<DIV> R10: 0000000000000000 R11: 00000000000200d2 R12: 000000000000007e</DIV>
<DIV> R13: 000000000007e000 R14: 0000000000000100 R15: 0000000000000100</DIV>
<DIV> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018</DIV>
<DIV> #4 [ffff81030bdcd8f8] lnet_ni_send at ffffffff88539543</DIV>
<DIV> #5 [ffff81030bdcd918] lnet_send at ffffffff8853ba03</DIV>
<DIV> #6 [ffff81030bdcd9d8] truncate_inode_pages_range at ffffffff8002b84a</DIV>
<DIV> #7 [ffff81030bdcdaf8] ptlrpc_send_reply at ffffffff8864a658</DIV>
<DIV> #8 [ffff81030bdcdc18] lustre_msg_get_version at ffffffff8864eb05</DIV>
<DIV> #9 [ffff81030bdcdc48] lustre_msg_check_version_v2 at ffffffff8864ebc8</DIV>
<DIV>#10 [ffff81030bdcdca8] ost_handle at ffffffff8890d08e</DIV>
<DIV>#11 [ffff81030bdcde38] ptlrpc_wait_event at ffffffff8865e8a8</DIV>
<DIV>#12 [ffff81030bdcdf48] kernel_thread at ffffffff8005dfb1</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV></DIV>
<DIV>When the crash happened, the machine seemed working at good condition, low load, low memory usage, low iowait...</DIV>
<DIV></DIV>
<DIV>Do you have any suggetions for avoiding this kind of crashes? </DIV>
<DIV></DIV>
<DIV>Thank you very much !</DIV>
<DIV></DIV>
<DIV></DIV>
<DIV></DIV>
<DIV></DIV>
<DIV></DIV>
<DIV> </DIV>
<DIV>---------------------------------------------------------</DIV>
<DIV>Lu Wang</DIV>
<DIV>Computing Center, Institute of High Energy Physics</DIV>
<DIV>Chinese Academy of Sciences</DIV>
<DIV>Beijing, 100049 P. R. China</DIV>
<DIV>Tel: (86)010-88236010 ext 105</DIV>
<DIV>E-mail: Lu.Wang@ihep.ac.cn</DIV>
<DIV>---------------------------------------------------------</DIV>
<DIV> </DIV></BODY></HTML>