<html><head><meta http-equiv="content-type" content="text/html; charset=us-ascii"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>Thank you. I checked all those files with "ll_decode_filter_fid" and "lfs fid2path":</div><div><br></div><div>On OSS servers:</div><div><br></div><div># umount /lustre/ost01</div><div># mount -t ldiskfs /dev/mapper/ost01 /mnt/</div><div># find /mnt/O/0/d*/ -type f</div><div>/mnt/O/0/d11/450605003</div><div>/mnt/O/0/d12/450605484</div><div>. . .</div><div><br></div><div># ll_decode_filter_fid /mnt/O/0/d11/450605003</div><div>/mnt/O/0/d11/450605003: parent=[0x200019425:0x733f:0x0] stripe=0 stripe_size=1048576 stripe_count=1 layout_version=0 range=0</div><div><br></div><div># umount /mnt</div><div># mount -t lustre /dev/mapper/ost01 /lustre/ost01</div><div><br></div><div>On client:</div><div><br></div><div># lfs fid2path /cluster [0x200019425:0x733f:0x0]</div><div>lfs fid2path: cannot find /cluster [0x200019425:0x733f:0x0]: No such file or directory</div><div><br></div><div>I got a reply "No such file or directory" for all those unknown files.</div><div>So I can assume those are "stray objects from deleted files".</div><div><br></div><div>Best regards,</div><div>Pavlo Khmel</div><div><br><blockquote type="cite"><div>On 20 Jan 2024, at 04:45, Andreas Dilger <adilger@whamcloud.com> wrote:</div><br class="Apple-interchange-newline"><div>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<br>
<div><br>
<blockquote type="cite">
<div>On Jan 19, 2024, at 13:48, Pavlo Khmel via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div>Hi,<br>
<br>
I'm trying to remove 4 OSTs. <br>
<br>
# lfs osts<br>
OBDS:<br>
0: cluster-OST0000_UUID ACTIVE<br>
1: cluster-OST0001_UUID ACTIVE<br>
2: cluster-OST0002_UUID ACTIVE<br>
3: cluster-OST0003_UUID ACTIVE<br>
. . .<br>
<br>
I moved all files to other OSTs. "lfs find" cannot find any files on these 4 OSTs.<br>
<br>
# time lfs find --ost 0 --ost 1 --ost 2 --ost 3 /cluster<br>
<br>
real<span class="Apple-tab-span" style="white-space:pre"> </span>936m8.528s<br>
user<span class="Apple-tab-span" style="white-space:pre"> </span>13m48.298s<br>
sys<span class="Apple-tab-span" style="white-space:pre"> </span>210m1.245s<br>
<br>
But still: 2624 inods are in use and 14.5G total size.<br>
<br>
# lfs df -i | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003<br>
cluster-OST0000_UUID 4293438576 644 4293437932 1% /cluster[OST:0]<br>
cluster-OST0001_UUID 4293438576 640 4293437936 1% /cluster[OST:1]<br>
cluster-OST0002_UUID 4293438576 671 4293437905 1% /cluster[OST:2]<br>
cluster-OST0003_UUID 4293438576 669 4293437907 1% /cluster[OST:3]<br>
<br>
# lfs df -h | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003<br>
cluster-OST0000_UUID 29.2T 3.8G 27.6T 1% /cluster[OST:0]<br>
cluster-OST0001_UUID 29.2T 3.7G 27.6T 1% /cluster[OST:1]<br>
cluster-OST0002_UUID 29.2T 3.3G 27.6T 1% /cluster[OST:2]<br>
cluster-OST0003_UUID 29.2T 3.7G 27.6T 1% /cluster[OST:3]<br>
<br>
I tried to check the file-system for errors:<br>
<br>
# umount /lustre/ost01<br>
# e2fsck -fy /dev/mapper/ost01<br>
<br>
and<br>
<br>
# lctl lfsck_start --device cluster-OST0001<br>
# lctl get_param -n osd-ldiskfs.cluster-OST0001.oi_scrub<br>
. . .<br>
status: completed<br>
<br>
I tried to mount OST as ldiskfs and there are several files in /O/0/d*/<br>
<br>
# umount /lustre/ost01<br>
# mount -t ldiskfs /dev/mapper/ost01 /mnt/<br>
# ls -Rhl /mnt/O/0/d*/<br>
. . .<br>
/mnt/O/0/d11/:<br>
-rw-rw-rw- 1 user1 group1 603K Nov 8 21:37 450605003<br>
/mnt/O/0/d12/:<br>
-rw-rw-rw- 1 user1 group1 110K Jun 16 2023 450322028<br>
-rw-rw-rw- 1 user1 group1 21M Nov 8 22:17 450605484<br>
. . .<br>
<br>
Is it expected behavior? Is it save to delere OST even with those files?<br>
</div>
</div>
</blockquote>
</div>
<div><br>
</div>
You can run the debugfs "stat" command to print the "fid" xattr and it will print the MDT
<div>parent FID for use with "lfs fid2path" on the client to see if there are any files related</div>
<div>to these objects. You could also run "ll_decode_filter_fid" to do the same thing on the</div>
<div>mounted ldiskfs filesystem.</div>
<div><br>
</div>
<div>It is likely that there are a few stray objects from deleted files, but hard to say for sure.</div>
<div><br>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br>
</div>
</div>
</div></blockquote></div><br></body></html>