<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=us-ascii" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 8.00.6001.18928"><LINK rel=stylesheet
href="BLOCKQUOTE{margin-Top: 0px; margin-Bottom: 0px; margin-Left: 2em}"></HEAD>
<BODY
style="MARGIN-TOP: 10px; FONT-FAMILY: Verdana; COLOR: #000000; MARGIN-LEFT: 10px; FONT-SIZE: 10pt; MARGIN-RIGHT: 10px">
<DIV>Dear, everyone</DIV>
<DIV> </DIV>
<DIV></DIV>
<DIV>We have stuck with the problem that the OSS connect one dead client or one with changed IP address all the time until we reboot the dead client. From the OSS log message, we can get the information as follows:</DIV>
<DIV></DIV>
<DIV><FONT color=#808080></FONT> </DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:07 com01 kernel: Lustre: 12180:0:(socklnd_cb.cLustre: 12180:0:(socklnd_cb.c:915:ksocknal_launch_packet()) No usable routes to 12345-202.Lustre: 12180:0:(socklnd_cb.c:915:ksocknal_launch_packet()) No usable routes to 12345-202.122.37.79@tcp</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:07 com01 kernel: Lustre: 12180:0:(socklnd_cb.c:915:ksocknal_launch_packet()) No usable routes to 12345-202.122.37.79@tcp</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:07 com01 last message repeated 35 times</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:07 com01 kernel: Lustre: 12180:0:(socklnd_cb.c:915:ksocknal_lauLustre: 12180:0:(socklnd_cb.c:915:ksocknal_launch_packet()) No usable routes to 12345-202.122.37.79@tcp</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:07 com01 kernel: Lustre: 12180:0:(socklnd_cb.c:915:ksocknal_launch_packet()) No usable routes to 12345-202.122.37.79@tcp</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:07 com01 last message repeated 2 times</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:07 com01 kernel: Lustre: 12180:0:(socklnd_cb.c:91Lustre: 12180:0:(socklnd_cb.c:915:ksocknal_launch_packet()) No usable routes to 12345-202.122.37.79@tcp</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:07 com01 kernel: Lustre: 12180:0:(socklnd_cb.c:915:ksocknal_launch_packet()) No usable routes to 12345-202.122.37.79@tcp</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:11 com01 last message repeated 188807 times</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:11 com01 kernel: BUG: soft lockup - CPU#15 stuck for 10s! [ll_ost_118:12180]Jul 7 14:45:11 com01 kernel: CPU 15:</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:11 com01 kernel: Modules linked in: obdfilter(U) fsfilt_ldiskfs(U) ost(U) mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) ldiskfs(U) crc16(U) autofs4(U) hidp(U) rfcomm(U) l2cap(U) bluetooth(U) sunrpc(U) dm_multipath(U) scsi_dh(U) video(U) hwmon(U) backlight(U) sbs(U) i2c_ec(U) i2c_core(U) button(U) battery(U) asus_acpi(U) acpi_memhotplug(U) ac(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) parport_pc(U) lp(U) parport(U) ixgbe(U) pcspkr(U) shpchp(U) serio_raw(U) hpilo(U) sg(U) bnx2(U) dm_raid45(U) dm_message(U) dm_region_hash(U) dm_mem_cache(U) dm_snapshot(U) dm_zero(U) dm_mirror(U) dm_log(U) dm_mod(U) usb_storage(U) lpfc(U) scsi_transport_fc(U) cciss(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U)</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:11 com01 kernel: Pid: 12180, comm: ll_ost_118 Tainted: G M 2.6.18-128.7.1.el5_lustre.1.8.1.1 #1Jul 7 14:45:11 com01 kernel: RIP: 0010:[<ffffffff8006dce9>] [<ffffffff8006dce9>] do_gettimeoffset_tsc+0x8/0x39</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:11 com01 kernel: RSP: 0018:ffff8102797b92c0 EFLAGS: 00000202</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:11 com01 kernel: RAX: 00000000000106a5 RBX: ffff8102797b9300 RCX: 00000000009ce3bd</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:11 com01 kernel: RDX: 00000000bfebfbff RSI: 0000000000000100 RDI: ffff8102797b9300Jul 7 14:45:11 com01 kernel: RBP: 0000000000000733 R08: 0000000000000000 R09: 0000000000000800</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:11 com01 kernel: R10: ffffffff8867dc7b R11: ffffffff8867dc4e R12: ffffffff88677379</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:11 com01 kernel: R13: ffff8104f46af953 R14: 00000000000006ad R15: 00000000ffffffff</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 14:45:11 com01 kernel: FS: 00002ab88e27f220(0000) GS:ffff81061fcf78c0(0000) knlGS:0000000000000000</FONT></DIV>
<DIV>......</DIV>
<DIV></DIV>
<DIV>And we find one CPU stuck.</DIV>
<DIV>[root@com01 ~]# grep CPU#5 /var/log/messages</DIV>
<DIV> </DIV>
<DIV><FONT
color=#808080>Jul 7 04:28:59 com01 kernel: BUG: soft lockup - CPU#5 stuck for 10s! [ll_ost_118:12180]</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 04:29:43 com01 kernel: BUG: soft lockup - CPU#5 stuck for 10s! [ll_ost_118:12180]</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 04:30:03 com01 kernel: BUG: soft lockup - CPU#5 stuck for 10s! [ll_ost_118:12180]</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 04:30:23 com01 kernel: BUG: soft lockup - CPU#5 stuck for 10s! [ll_ost_118:12180]</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 04:30:45 com01 kernel: BUG: soft lockup - CPU#5 stuck for 10s! [ll_ost_118:12180]</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 04:31:25 com01 kernel: BUG: soft lockup - CPU#5 stuck for 10s! [ll_ost_118:12180]</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 04:32:52 com01 kernel: BUG: soft lockup - CPU#5 stuck for 10s! [ll_ost_118:12180]</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 04:33:12 com01 kernel: BUG: soft lockup - CPU#5 stuck for 10s! [ll_ost_118:12180]</FONT></DIV>
<DIV><FONT
color=#808080>Jul 7 04:33:55 com01 kernel: BUG: soft lockup - CPU#5 stuck for 10s! [ll_ost_118:12180]</FONT></DIV>
<DIV><FONT color=#808080></FONT> </DIV>
<DIV></DIV>
<DIV>I think the the process "ll_ost_118:12180" is charge of the connection between the client <A
href="mailto:12345-202.122.37.79@tcp">12345-202.122.37.79@tcp</A> and the OSS, Because we changed the client with another IP address, but the OSS couldn't recognize it and still connect the original IP. For this reason, our monitoring gives an alarm for us on and off, as the monitor can't ping through the OSS(hostname called com01) with the command "lctl ping com01" ,but after several seconds or minutes it works well. And we found the cpu_idle an cpu_wio with serrated graph showed from ganglia monitoring. You can have a look from the attachment.</DIV>
<DIV>For this problem, although we always reboot the dead client, then the OSS works well, we found it's not very rationable for the Lustre file system especially the specific case such as the OSS connecting the non-existent IP. Shall we have some
command or some other methods to evict the dead client or unknown IP manually?</DIV>
<DIV>We really appreciate for your any help!</DIV>
<DIV> </DIV>
<DIV></DIV>
<DIV></DIV>
<DIV> </DIV>
<DIV></DIV>
<DIV>Best Regards</DIV>
<DIV>QiuLan Huang</DIV>
<DIV>2010-07-07</DIV>
<DIV>====================================================================</DIV>
<DIV>Computing center,the Institute of High Energy Physics, China</DIV>
<DIV>Huang, Qiulan Tel: (+86) 10 8823 6012-604</DIV>
<DIV>P.O. Box 918-7 Fax: (+86) 10 8823 6839</DIV>
<DIV>Beijing 100049 P.R. China Email: huangql@ihep.ac.cn</DIV>
<DIV>=================================================================== </DIV>
<DIV> </DIV></BODY></HTML>