<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Sep 8, 2021, at 04:42, Alastair Basden <<a href="mailto:a.g.basden@durham.ac.uk" class="">a.g.basden@durham.ac.uk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">Next step would be to unmount OST004e, run a full e2fsck, and then check lost+found and/or a regular "find /mnt/ost -type f -size +1M" or similar to find where the files are.<br class="">
</blockquote>
<br class="">
<br class="">
Thanks. e2fsck returns clean (on its own, with -p and with -f).<br class="">
<br class="">
Now, the find command does return a large number of files belonging to usera - and of sufficient size to fill up the disk.<br class="">
<br class="">
e.g. /mnt/ost/O/0/d3/29379 has a size 2.3G.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
If you run 'll_decode_filter_fid /mnt/ost/O/0/d3/29379' or 'debugfs -c -R "stat O/0/d3/29379" /dev/<ostdev>' it will print the *parent* (MDT) FID suitable for "lfs fid2path" on a client. This probably won't work, but worth a try anyway.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">So it would seem that these files are getting deleted from the mds, but not from this OST. Has this been seen before? The other OSTs seem fine - stuff getting deleted as expected.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
Based on the very low object number, I would guess that these are old files and relate to some kind of issue seen in the past (e.g. MDT corruption where e2fsck cleared some inodes, or similar). The "debugfs stat" command above will also print the object creation
time along with the normal timestamps.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">Is it safe to simply remove all these files, and then remount etc? How can we ensure that new files will be deleted from the OST in the future?<br class="">
</div>
</div>
</blockquote>
<br class="">
</div>
<div>If they are not referenced by any in-use file (per fid2path) then yes. </div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>