<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hello,<div><br><div><div>在 2010-11-18，下午10:03， Herbert Fruchtl 写道：</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>I was wrong about only one client having problems. It seems to<br>be all of them, except the mds server (see below), so it is a<br>problem of the filesystem (not the client) after all.<br><br><blockquote type="cite">Could you elaborate about how "broken" the files are?<br></blockquote><br>When I do an 'ls', the filenames are flashing in red (this is<br>for example the case for broken symbolic links). Permissions, date<br>and owner are missing, like in the middle of the next three<br>lines:<br>-rw------- 1 root root 18308319 Jul 16 2009 stat_1247756353.gz<br>?--------- ? ? ? ? ? stat_1248125742.gz<br>drwxr-xr-x 2 stephane ukmhd 4096 Jul 8 2009 stephane<br><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" color="#144FAE"><br></font></font></div></blockquote><blockquote type="cite"><div>Attempting to access the file more closely results in an I/O error:<br>[root@mhdc ~]# ls -l /workspace/ls-lR_2009-01-20<br>ls: /workspace/ls-lR_2009-01-20: Input/output error<br>[root@mhdc ~]# cp /workspace/ls-lR_2009-01-20 /tmp<br>cp: cannot stat `/workspace/ls-lR_2009-01-20': Input/output error<br></div></blockquote><div><br></div>This looks very much like some OSTs are failing.</div><div><br></div><div><br><blockquote type="cite"><div><br><blockquote type="cite"><br></blockquote><blockquote type="cite">From your description and the error message you provide, I suspect that one(or some) of the OSTs went down. What does `lctl dl` show?<br></blockquote><blockquote type="cite"><br></blockquote>The files are accessible from the mds server, and the OSTs seem<br>visible from the "broken" clients:<br>[root@mhdc ~]# lctl dl<br> 0 UP mgc MGC192.168.101.214@tcp 63568484-f714-da05-c5c2-b96db1b22962 5<br> 1 UP lov home-clilov-ffff8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 4<br> 2 UP mdc home-MDT0000-mdc-ffff8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5<br> 3 UP osc home-OST0001-osc-ffff8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5<br> 4 UP osc home-OST0003-osc-ffff8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5<br> 5 UP osc home-OST0002-osc-ffff8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5<br> 6 UP osc home-OST0005-osc-ffff8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5<br> 7 UP osc home-OST0004-osc-ffff8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5<br> 8 UP osc home-OST0000-osc-ffff8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5<br><br>Does this help?<br></div></blockquote><div><br></div><div>I mean 'lctl dl' output on the OSS servers. Make sure that your OSTs are all mounted and running well.</div><br><blockquote type="cite"><div><br> Herbert<br><br><blockquote type="cite">在 2010-11-18，下午8:18， Herbert Fruchtl 写道：<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">I have a Lustre (1.6.7) system that looks OKish (as far as I can see) from the <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">mds and most of the clients. From one client however (the users' login machine) <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">it looks broken. Some files are missing, some seem broken, and the df command <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">hangs.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Rebooting the client doesn't change anything. Is it broken, or is there some <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">persistent information that I need to flush? When I do an ls on a partially <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">broken directory, I get the following two lines in /var/log/messages:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Nov 18 12:13:53 mhdc kernel: [ 7093.751196] LustreError: <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">10919:0:(file.c:999:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Nov 18 12:13:53 mhdc kernel: [ 7093.761098] LustreError: <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">10919:0:(file.c:999:ll_glimpse_size()) Skipped 9 previous similar messages<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Any ideas how to proceed with the least disruption?<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Thanks in advance,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Herbert<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">-- <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Herbert Fruchtl<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Senior Scientific Computing Officer<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">School of Chemistry, School of Mathematics and Statistics<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">University of St Andrews<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">--<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">The University of St Andrews is a charity registered in Scotland:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">No SC013532<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Lustre-discuss mailing list<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><br>-- <br>Herbert Fruchtl<br>Senior Scientific Computing Officer<br>School of Chemistry, School of Mathematics and Statistics<br>University of St Andrews<br>--<br>The University of St Andrews is a charity registered in Scotland:<br>No SC013532<br>_______________________________________________<br>Lustre-discuss mailing list<br><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>http://lists.lustre.org/mailman/listinfo/lustre-discuss<br></div></blockquote></div><br></div></body></html>