
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">

<HTML><HEAD>

<META content="text/html; charset=us-ascii" http-equiv=Content-Type>

<META name=GENERATOR content="MSHTML 8.00.6001.18928"><LINK rel=stylesheet 

href="BLOCKQUOTE{margin-Top: 0px; margin-Bottom: 0px; margin-Left: 2em}"></HEAD>

<BODY 

style="MARGIN-TOP: 10px; FONT-FAMILY: Verdana; COLOR: #000000; MARGIN-LEFT: 10px; FONT-SIZE: 10pt; MARGIN-RIGHT: 10px">

<DIV>Dear,&nbsp;everyone</DIV>

<DIV>&nbsp;</DIV>

<DIV></DIV>

<DIV>We&nbsp;have&nbsp;stuck&nbsp;with&nbsp;the&nbsp;problem&nbsp;that&nbsp;the&nbsp;OSS&nbsp;&nbsp;connect&nbsp;one&nbsp;dead&nbsp;client&nbsp;or&nbsp;one&nbsp;with&nbsp;changed&nbsp;IP&nbsp;address&nbsp;all&nbsp;the&nbsp;time&nbsp;until&nbsp;we&nbsp;reboot&nbsp;the&nbsp;dead&nbsp;client.&nbsp;From&nbsp;the&nbsp;OSS&nbsp;log&nbsp;message,&nbsp;we&nbsp;can&nbsp;get&nbsp;&nbsp;the&nbsp;information&nbsp;as&nbsp;follows:</DIV>

<DIV></DIV>

<DIV><FONT color=#808080></FONT>&nbsp;</DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:07&nbsp;com01&nbsp;kernel:&nbsp;Lustre:&nbsp;12180:0:(socklnd_cb.cLustre:&nbsp;12180:0:(socklnd_cb.c:915:ksocknal_launch_packet())&nbsp;No&nbsp;usable&nbsp;routes&nbsp;to&nbsp;12345-202.Lustre:&nbsp;12180:0:(socklnd_cb.c:915:ksocknal_launch_packet())&nbsp;No&nbsp;usable&nbsp;routes&nbsp;to&nbsp;12345-202.122.37.79@tcp</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:07&nbsp;com01&nbsp;kernel:&nbsp;Lustre:&nbsp;12180:0:(socklnd_cb.c:915:ksocknal_launch_packet())&nbsp;No&nbsp;usable&nbsp;routes&nbsp;to&nbsp;12345-202.122.37.79@tcp</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:07&nbsp;com01&nbsp;last&nbsp;message&nbsp;repeated&nbsp;35&nbsp;times</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:07&nbsp;com01&nbsp;kernel:&nbsp;Lustre:&nbsp;12180:0:(socklnd_cb.c:915:ksocknal_lauLustre:&nbsp;12180:0:(socklnd_cb.c:915:ksocknal_launch_packet())&nbsp;No&nbsp;usable&nbsp;routes&nbsp;to&nbsp;12345-202.122.37.79@tcp</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:07&nbsp;com01&nbsp;kernel:&nbsp;Lustre:&nbsp;12180:0:(socklnd_cb.c:915:ksocknal_launch_packet())&nbsp;No&nbsp;usable&nbsp;routes&nbsp;to&nbsp;12345-202.122.37.79@tcp</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:07&nbsp;com01&nbsp;last&nbsp;message&nbsp;repeated&nbsp;2&nbsp;times</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:07&nbsp;com01&nbsp;kernel:&nbsp;Lustre:&nbsp;12180:0:(socklnd_cb.c:91Lustre:&nbsp;12180:0:(socklnd_cb.c:915:ksocknal_launch_packet())&nbsp;No&nbsp;usable&nbsp;routes&nbsp;to&nbsp;12345-202.122.37.79@tcp</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:07&nbsp;com01&nbsp;kernel:&nbsp;Lustre:&nbsp;12180:0:(socklnd_cb.c:915:ksocknal_launch_packet())&nbsp;No&nbsp;usable&nbsp;routes&nbsp;to&nbsp;12345-202.122.37.79@tcp</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:11&nbsp;com01&nbsp;last&nbsp;message&nbsp;repeated&nbsp;188807&nbsp;times</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:11&nbsp;com01&nbsp;kernel:&nbsp;BUG:&nbsp;soft&nbsp;lockup&nbsp;-&nbsp;CPU#15&nbsp;stuck&nbsp;for&nbsp;10s!&nbsp;[ll_ost_118:12180]Jul&nbsp;&nbsp;7&nbsp;14:45:11&nbsp;com01&nbsp;kernel:&nbsp;CPU&nbsp;15:</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:11&nbsp;com01&nbsp;kernel:&nbsp;Modules&nbsp;linked&nbsp;in:&nbsp;obdfilter(U)&nbsp;fsfilt_ldiskfs(U)&nbsp;ost(U)&nbsp;mgc(U)&nbsp;lustre(U)&nbsp;lov(U)&nbsp;mdc(U)&nbsp;lquota(U)&nbsp;osc(U)&nbsp;ksocklnd(U)&nbsp;ptlrpc(U)&nbsp;obdclass(U)&nbsp;lnet(U)&nbsp;lvfs(U)&nbsp;libcfs(U)&nbsp;ldiskfs(U)&nbsp;crc16(U)&nbsp;autofs4(U)&nbsp;hidp(U)&nbsp;rfcomm(U)&nbsp;l2cap(U)&nbsp;bluetooth(U)&nbsp;sunrpc(U)&nbsp;dm_multipath(U)&nbsp;scsi_dh(U)&nbsp;video(U)&nbsp;hwmon(U)&nbsp;backlight(U)&nbsp;sbs(U)&nbsp;i2c_ec(U)&nbsp;i2c_core(U)&nbsp;button(U)&nbsp;battery(U)&nbsp;asus_acpi(U)&nbsp;acpi_memhotplug(U)&nbsp;ac(U)&nbsp;ipv6(U)&nbsp;xfrm_nalgo(U)&nbsp;crypto_api(U)&nbsp;parport_pc(U)&nbsp;lp(U)&nbsp;parport(U)&nbsp;ixgbe(U)&nbsp;pcspkr(U)&nbsp;shpchp(U)&nbsp;serio_raw(U)&nbsp;hpilo(U)&nbsp;sg(U)&nbsp;bnx2(U)&nbsp;dm_raid45(U)&nbsp;dm_message(U)&nbsp;dm_region_hash(U)&nbsp;dm_mem_cache(U)&nbsp;dm_snapshot(U)&nbsp;dm_zero(U)&nbsp;dm_mirror(U)&nbsp;dm_log(U)&nbsp;dm_mod(U)&nbsp;usb_storage(U)&nbsp;lpfc(U)&nbsp;scsi_transport_fc(U)&nbsp;cciss(U)&nbsp;sd_mod(U)&nbsp;scsi_mod(U)&nbsp;ext3(U)&nbsp;jbd(U)&nbsp;uhci_hcd(U)&nbsp;ohci_hcd(U)&nbsp;ehci_hcd(U)</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:11&nbsp;com01&nbsp;kernel:&nbsp;Pid:&nbsp;12180,&nbsp;comm:&nbsp;ll_ost_118&nbsp;Tainted:&nbsp;G&nbsp;&nbsp;&nbsp;M&nbsp;&nbsp;2.6.18-128.7.1.el5_lustre.1.8.1.1&nbsp;#1Jul&nbsp;&nbsp;7&nbsp;14:45:11&nbsp;com01&nbsp;kernel:&nbsp;RIP:&nbsp;0010:[&lt;ffffffff8006dce9&gt;]&nbsp;&nbsp;[&lt;ffffffff8006dce9&gt;]&nbsp;do_gettimeoffset_tsc+0x8/0x39</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:11&nbsp;com01&nbsp;kernel:&nbsp;RSP:&nbsp;0018:ffff8102797b92c0&nbsp;&nbsp;EFLAGS:&nbsp;00000202</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:11&nbsp;com01&nbsp;kernel:&nbsp;RAX:&nbsp;00000000000106a5&nbsp;RBX:&nbsp;ffff8102797b9300&nbsp;RCX:&nbsp;00000000009ce3bd</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:11&nbsp;com01&nbsp;kernel:&nbsp;RDX:&nbsp;00000000bfebfbff&nbsp;RSI:&nbsp;0000000000000100&nbsp;RDI:&nbsp;ffff8102797b9300Jul&nbsp;&nbsp;7&nbsp;14:45:11&nbsp;com01&nbsp;kernel:&nbsp;RBP:&nbsp;0000000000000733&nbsp;R08:&nbsp;0000000000000000&nbsp;R09:&nbsp;0000000000000800</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:11&nbsp;com01&nbsp;kernel:&nbsp;R10:&nbsp;ffffffff8867dc7b&nbsp;R11:&nbsp;ffffffff8867dc4e&nbsp;R12:&nbsp;ffffffff88677379</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:11&nbsp;com01&nbsp;kernel:&nbsp;R13:&nbsp;ffff8104f46af953&nbsp;R14:&nbsp;00000000000006ad&nbsp;R15:&nbsp;00000000ffffffff</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;14:45:11&nbsp;com01&nbsp;kernel:&nbsp;FS:&nbsp;&nbsp;00002ab88e27f220(0000)&nbsp;GS:ffff81061fcf78c0(0000)&nbsp;knlGS:0000000000000000</FONT></DIV>

<DIV>......</DIV>

<DIV></DIV>

<DIV>And&nbsp;we&nbsp;find&nbsp;one&nbsp;CPU&nbsp;stuck.</DIV>

<DIV>[root@com01&nbsp;~]#&nbsp;grep&nbsp;CPU#5&nbsp;/var/log/messages</DIV>

<DIV>&nbsp;</DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;04:28:59&nbsp;com01&nbsp;kernel:&nbsp;BUG:&nbsp;soft&nbsp;lockup&nbsp;-&nbsp;CPU#5&nbsp;stuck&nbsp;for&nbsp;10s!&nbsp;[ll_ost_118:12180]</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;04:29:43&nbsp;com01&nbsp;kernel:&nbsp;BUG:&nbsp;soft&nbsp;lockup&nbsp;-&nbsp;CPU#5&nbsp;stuck&nbsp;for&nbsp;10s!&nbsp;[ll_ost_118:12180]</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;04:30:03&nbsp;com01&nbsp;kernel:&nbsp;BUG:&nbsp;soft&nbsp;lockup&nbsp;-&nbsp;CPU#5&nbsp;stuck&nbsp;for&nbsp;10s!&nbsp;[ll_ost_118:12180]</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;04:30:23&nbsp;com01&nbsp;kernel:&nbsp;BUG:&nbsp;soft&nbsp;lockup&nbsp;-&nbsp;CPU#5&nbsp;stuck&nbsp;for&nbsp;10s!&nbsp;[ll_ost_118:12180]</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;04:30:45&nbsp;com01&nbsp;kernel:&nbsp;BUG:&nbsp;soft&nbsp;lockup&nbsp;-&nbsp;CPU#5&nbsp;stuck&nbsp;for&nbsp;10s!&nbsp;[ll_ost_118:12180]</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;04:31:25&nbsp;com01&nbsp;kernel:&nbsp;BUG:&nbsp;soft&nbsp;lockup&nbsp;-&nbsp;CPU#5&nbsp;stuck&nbsp;for&nbsp;10s!&nbsp;[ll_ost_118:12180]</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;04:32:52&nbsp;com01&nbsp;kernel:&nbsp;BUG:&nbsp;soft&nbsp;lockup&nbsp;-&nbsp;CPU#5&nbsp;stuck&nbsp;for&nbsp;10s!&nbsp;[ll_ost_118:12180]</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;04:33:12&nbsp;com01&nbsp;kernel:&nbsp;BUG:&nbsp;soft&nbsp;lockup&nbsp;-&nbsp;CPU#5&nbsp;stuck&nbsp;for&nbsp;10s!&nbsp;[ll_ost_118:12180]</FONT></DIV>

<DIV><FONT 

color=#808080>Jul&nbsp;&nbsp;7&nbsp;04:33:55&nbsp;com01&nbsp;kernel:&nbsp;BUG:&nbsp;soft&nbsp;lockup&nbsp;-&nbsp;CPU#5&nbsp;stuck&nbsp;for&nbsp;10s!&nbsp;[ll_ost_118:12180]</FONT></DIV>

<DIV><FONT color=#808080></FONT>&nbsp;</DIV>

<DIV></DIV>

<DIV>I&nbsp;think&nbsp;the&nbsp;the&nbsp;process&nbsp;"ll_ost_118:12180"&nbsp;is&nbsp;charge&nbsp;of&nbsp;the&nbsp;connection&nbsp;between&nbsp;the&nbsp;client&nbsp;<A 

href="mailto:12345-202.122.37.79@tcp">12345-202.122.37.79@tcp</A>&nbsp;and&nbsp;the&nbsp;OSS,&nbsp;Because&nbsp;we&nbsp;changed&nbsp;the&nbsp;client&nbsp;with&nbsp;another&nbsp;IP&nbsp;address,&nbsp;but&nbsp;the&nbsp;OSS&nbsp;couldn't&nbsp;recognize&nbsp;it&nbsp;and&nbsp;still&nbsp;connect&nbsp;the&nbsp;original&nbsp;IP.&nbsp;For&nbsp;this&nbsp;reason,&nbsp;our&nbsp;monitoring&nbsp;gives&nbsp;an&nbsp;alarm&nbsp;for&nbsp;us&nbsp;on&nbsp;and&nbsp;off,&nbsp;as&nbsp;the&nbsp;monitor&nbsp;can't&nbsp;ping&nbsp;through&nbsp;the&nbsp;OSS(hostname&nbsp;called&nbsp;com01)&nbsp;with&nbsp;the&nbsp;command&nbsp;"lctl&nbsp;ping&nbsp;com01"&nbsp;,but&nbsp;after&nbsp;several&nbsp;seconds&nbsp;or&nbsp;minutes&nbsp;it&nbsp;works&nbsp;well.&nbsp;And&nbsp;we&nbsp;found&nbsp;the&nbsp;cpu_idle&nbsp;an&nbsp;cpu_wio&nbsp;with&nbsp;serrated&nbsp;graph&nbsp;showed&nbsp;from&nbsp;ganglia&nbsp;monitoring.&nbsp;You&nbsp;can&nbsp;have&nbsp;a&nbsp;look&nbsp;from&nbsp;the&nbsp;attachment.</DIV>

<DIV>For&nbsp;this&nbsp;problem,&nbsp;although&nbsp;we&nbsp;always&nbsp;reboot&nbsp;the&nbsp;dead&nbsp;client,&nbsp;then&nbsp;the&nbsp;OSS&nbsp;works&nbsp;well,&nbsp;we&nbsp;found&nbsp;it's&nbsp;not&nbsp;very&nbsp;rationable&nbsp;for&nbsp;the&nbsp;Lustre&nbsp;file&nbsp;system&nbsp;especially&nbsp;the&nbsp;specific&nbsp;case&nbsp;such&nbsp;as&nbsp;the&nbsp;OSS&nbsp;connecting&nbsp;the&nbsp;non-existent&nbsp;IP.&nbsp;Shall&nbsp;we&nbsp;have&nbsp;some 

command&nbsp;or&nbsp;some&nbsp;other&nbsp;methods&nbsp;to&nbsp;evict&nbsp;the&nbsp;dead&nbsp;client&nbsp;or&nbsp;unknown&nbsp;IP&nbsp;manually?</DIV>

<DIV>We&nbsp;really&nbsp;appreciate&nbsp;for&nbsp;your&nbsp;any&nbsp;help!</DIV>

<DIV>&nbsp;</DIV>

<DIV></DIV>

<DIV></DIV>

<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</DIV>

<DIV></DIV>

<DIV>Best&nbsp;Regards</DIV>

<DIV>QiuLan&nbsp;Huang</DIV>

<DIV>2010-07-07</DIV>

<DIV>====================================================================</DIV>

<DIV>Computing&nbsp;center,the&nbsp;Institute&nbsp;of&nbsp;High&nbsp;Energy&nbsp;Physics,&nbsp;China</DIV>

<DIV>Huang,&nbsp;Qiulan&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Tel:&nbsp;(+86)&nbsp;10&nbsp;8823&nbsp;6012-604</DIV>

<DIV>P.O.&nbsp;Box&nbsp;918-7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Fax:&nbsp;(+86)&nbsp;10&nbsp;8823&nbsp;6839</DIV>

<DIV>Beijing&nbsp;100049&nbsp;&nbsp;P.R.&nbsp;China&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Email:&nbsp;huangql@ihep.ac.cn</DIV>

<DIV>=================================================================== </DIV>

<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</DIV></BODY></HTML>