<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Brian,<br>
<br>
Brian J. Murrell wrote:
<blockquote cite="mid:1232640168.17163.2122.camel@pc.interlinx.bc.ca"
type="cite">
<pre wrap="">On Thu, 2009-01-22 at 15:44 +0000, Wojciech Turek wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hello,
</pre>
</blockquote>
<pre wrap=""><!---->
Hi,
</pre>
<blockquote type="cite">
<pre wrap="">Lustre MDS report following error:
Jan 22 15:20:40 mds01.beowulf.cluster kernel: LustreError:
24680:0:(lov_request.c:692:lov_update_create_set()) error creating fid
0xeb79c9d sub-object on OST idx 4/1: rc = -28
</pre>
</blockquote>
<pre wrap=""><!---->
-28 is ENOSPC.
</pre>
<blockquote type="cite">
<pre wrap="">Which I translate as that one of the OST (index 4/1) is full and has
no space left on device.
</pre>
</blockquote>
<pre wrap=""><!---->
Yes.
</pre>
<blockquote type="cite">
<pre wrap="">OSS seem to be consistent and says:
Jan 22 15:21:15 storage08.beowulf.cluster kernel: LustreError:
23507:0:(filter_io_26.c:721:filter_commitrw_write()) error starting
transaction: rc = -30
</pre>
</blockquote>
<pre wrap=""><!---->
Hrm. I'm not sure a -30 (EROFS) would translate to a -28 to the MDS. I
think it would also be a -30. So are you sure you are looking at
correlating messages? The timestamps, if the two nodes are in sync also
seem to indicate a lack of correlation with 35s of disparity.
Perhaps there is an actual -28 in the OSS log prior to the -30 one?
</pre>
</blockquote>
Yes you are right, there is plenty of messages with -30 in the logs and
probably they are not related to -28.<br>
<blockquote cite="mid:1232640168.17163.2122.camel@pc.interlinx.bc.ca"
type="cite">
<blockquote type="cite">
<pre wrap="">Which I translate as Client would like to write to an existing file
but it can't because file system is read only.
</pre>
</blockquote>
<pre wrap=""><!---->
Indeed. But why is it read-only? There should be an event in the OSS
log saying that it was turning the filesystem read-only.
</pre>
<blockquote type="cite">
<pre wrap="">The OST device is still mounted with rw option
</pre>
</blockquote>
<pre wrap=""><!---->
Yeah. That's just the state at mount time. Lustre will set a device
read-only in the case of filesystem errors, as one example.
</pre>
<blockquote type="cite">
<pre wrap="">Now the main question is why Lustre thinks that OST(idx4) is full?
</pre>
</blockquote>
<pre wrap=""><!---->
No, I think the main question is why is it read-only. The full
situation may have been transient where it filled up momentarily and
then some objects were removed. In any case, this is a secondary issue
and really only need be considered once the read-only situation is
understood.
</pre>
</blockquote>
Thank you for puting me in right track. I found these in the syslog<br>
Jan 22 10:18:40 storage08.beowulf.cluster kernel: LDISKFS-fs error
(device dm-8): mb_free_blocks: double-free of inode 16203779's block
688627718(bit 8198 in group 21015) <br>
Jan 22 10:18:40 storage08.beowulf.cluster kernel: <br>
Jan 22 10:18:40 storage08.beowulf.cluster kernel: Remounting filesystem
read-only <br>
<br>
Is this means that the file system may be corrupted? I am going to run
fsck -f on this device and try to mount it back, is that a right
procedure?<br>
I did not find any errors on my S2A9500 storage, so I am not sure when
this corruption could occur.<br>
<br>
<br>
<blockquote cite="mid:1232640168.17163.2122.camel@pc.interlinx.bc.ca"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">Is it possible that this OST have meny orphaned objects which takes
all the available space?
</pre>
</blockquote>
<pre wrap=""><!---->
That would be reflected in the df. If you suspect there may be orphan
objects though, you could lfsck to verify and clean.
</pre>
<blockquote type="cite">
<pre wrap="">Is there a way of reclaiming back this free space?
</pre>
</blockquote>
<pre wrap=""><!---->
If you mean orphaned OST objects, then lfsck.
b.
</pre>
</blockquote>
Cheers<br>
<br>
Wojciech<br>
<blockquote cite="mid:1232640168.17163.2122.camel@pc.interlinx.bc.ca"
type="cite">
<pre wrap="">
</pre>
<pre wrap="">
<hr size="4" width="90%">
_______________________________________________
Lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a>
</pre>
</blockquote>
</body>
</html>