Hi Bernd,<br><br>Many thanks for your reply. I have found this bug last night and as far as I can see there is no fix for it yet? I am preparing dbs to run lfsck on affected file systems. I also found bug 18748 and I must say we have exactly the same problems. It just looks like we run into that problem few months after CIEMAT did. As far as I know if we can see this message it means that there are files with missing objects. The worst is that we don't know when and why files looses they objects. It just happens spontaneously and there isn't any lustre messages that could give us a clue. Users run jobs and some time after their files were written some of these files get corrupted/looses objects (?-----) trying to access this files for the first time triggers 'lvbo' message.<br>
We have third lustre file system which runs on different hardware but the same lustre version and RHEL version as the affected ones. I can not see any problems on the third file system.<br><br>Wojciech <br><br><div class="gmail_quote">
2009/10/10 Bernd Schubert <span dir="ltr"><<a href="mailto:bs_lists@aakef.fastmail.fm" target="_blank">bs_lists@aakef.fastmail.fm</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
"ASSERTION(old_inode->i_state & I_FREEING)" is the infamous bug17485. You will<br>
need to run lfsck to fix it.<br>
<div><div></div><div><br>
<br>
On Saturday 10 October 2009, Wojciech Turek wrote:<br>
> Hi,<br>
><br>
> Did you get to the bottom of this?<br>
><br>
> We are having exactly the same problem with our lustre-1.6.6 (rhel4) file<br>
> systems. Recently it got worst and MDS crashes quite frequently, when we<br>
> run e2fsck there are errors that are being fixed. However after some time<br>
> we still are seeing the same errors in the logs about missing objects and<br>
> files get corrupted (?-----------) Also clients LBUGs quite frequently<br>
> with this message (osc_request.c:2904:osc_set_data_with_check()) LBUG<br>
> This looks like serious lustre problem but so far I didn't find any clues<br>
> on that even after long search through lustre bugzilla.<br>
><br>
> Our MDSs and OSSs are UPSed, RAID is behaving OK, we don't see any errors<br>
> in the syslog.<br>
><br>
> I will be grateful for some hints on this one<br>
><br>
> Wojciech<br>
><br>
> 2009/8/24 rishi pathak <<a href="mailto:mailmaverick666@gmail.com" target="_blank">mailmaverick666@gmail.com</a>><br>
><br>
> > Hi,<br>
> ><br>
> > Our lustre fs comprises of 15 OST/OSS and 1 MDS with no failover. Client<br>
> > as well as servers run lustre-1.6 and kernel 2.6.9-18.<br>
> ><br>
> > Doing a ls -ltr for a directory in lustre fs throws following<br>
> > errors (as got from lustre logs) on client<br>
> ><br>
> > 00000008:00020000:0:1251099455.304622:0:724:0:(osc_request.c:2898:osc_set<br>
> >_data_with_check()) ### inconsistent l_ast_data found ns:<br>
> > scratch-OST0005-osc-ffff81201e8dd800 lock: ffff811f9af04<br>
> > 000/0xec0d1c36da6992fd lrc: 3/1,0 mode: PR/PR res: 570622/0 rrc: 2 type:<br>
> > EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 100000<br>
> > remote: 0xb79b445e381bc9e6 expref: -99 p<br>
> > id: 22878<br>
> > 00000008:00040000:0:1251099455.337868:0:724:0:(osc_request.c:2904:osc_set<br>
> >_data_with_check()) ASSERTION(old_inode->i_state & I_FREEING) failed:Found<br>
> > existing inode ffff811f2cf693b8/1972725<br>
> > 44/1895600178 state 0 in lock: setting data to<br>
> > ffff8118ef8ed5f8/207519777/1771835328<br>
> > 00000000:00040000:0:1251099455.360090:0:724:0:(osc_request.c:2904:osc_set<br>
> >_data_with_check()) LBUG<br>
> ><br>
> ><br>
> > On scratch-OST0005 OST it shows<br>
> ><br>
> > Aug 24 10:22:53 yn266 kernel: LustreError:<br>
> > 3023:0:(ldlm_resource.c:851:ldlm_resource_add()) lvbo_init failed for<br>
> > resour ce 569204: rc -2<br>
> > Aug 24 10:22:53 yn266 kernel: LustreError:<br>
> > 3023:0:(ldlm_resource.c:851:ldlm_resource_add()) Skipped 19 previous<br>
> > similar messages<br>
> > Aug 24 12:40:43 yn266 kernel: LustreError:<br>
> > 2737:0:(ldlm_resource.c:851:ldlm_resource_add()) lvbo_init failed for<br>
> > resour ce 569195: rc -2<br>
> > Aug 24 12:44:59 yn266 kernel: LustreError:<br>
> > 2835:0:(ldlm_resource.c:851:ldlm_resource_add()) lvbo_init failed for<br>
> > resour ce 569198: rc -2<br>
> ><br>
> > These kind of errors we are getting for many clients.<br>
> ><br>
> > ##History ##<br>
> > Prior to thsese occurences, our MDS showed signs of failure in way that<br>
> > cpu load was shooting above 100 (on a quad core quad socket system) and<br>
> > users were complaining about slow storage performance. We took it offline<br>
> > and did fsck on unmounted MDS and OSTs. fsck on OSTs went fine but it<br>
> > showed some errors which were fixed. For data integrity check, mdsdb and<br>
> > ostdb were built and lfsck was run on a client(client was mounted with<br>
> > abort_recov).<br>
> ><br>
> > lfsck was run in following order:<br>
> > lfsck with no fix - reported dangling inodes and orphaned objects<br>
> > lfsck with -l (backup orphaned objects)<br>
> > lfsck with -d and -c (delete orphaned objects and create missing OST<br>
> > objects referenced by MDS)<br>
> ><br>
> > After above operations, on clients we were seeing file in red and<br>
> > blinking. Doing a stat came out with an error stating 'no such file or<br>
> > directory'.<br>
> ><br>
> > My question is whether the order in which lfsck was run (should lfsck be<br>
> > run multiple times) and the errors we are getting are related or not.<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Regards--<br>
> > Rishi Pathak<br>
> > National PARAM Supercomputing Facility<br>
> > Center for Development of Advanced Computing(C-DAC)<br>
> > Pune University Campus,Ganesh Khind Road<br>
> > Pune-Maharastra<br>
> ><br>
> > _______________________________________________<br>
> > Lustre-discuss mailing list<br>
> > <a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
> > <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
><br>
<br>
<br>
--<br>
</div></div><font color="#888888">Bernd Schubert<br>
DataDirect Networks<br>
</font></blockquote></div><br><br clear="all"><br>-- <br>--<br>Wojciech Turek<br><br>Assistant System Manager<br><br>High Performance Computing Service<br>University of Cambridge<br>Email: <a href="mailto:wjt27@cam.ac.uk" target="_blank">wjt27@cam.ac.uk</a><br>
Tel: (+)44 1223 763517 <br>