<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 19, 2024, at 13:48, Pavlo Khmel via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Hi,<br class="">
<br class="">
I'm trying to remove 4 OSTs. <br class="">
<br class="">
# lfs osts<br class="">
OBDS:<br class="">
0: cluster-OST0000_UUID ACTIVE<br class="">
1: cluster-OST0001_UUID ACTIVE<br class="">
2: cluster-OST0002_UUID ACTIVE<br class="">
3: cluster-OST0003_UUID ACTIVE<br class="">
. . .<br class="">
<br class="">
I moved all files to other OSTs. "lfs find" cannot find any files on these 4 OSTs.<br class="">
<br class="">
# time lfs find --ost 0 --ost 1 --ost 2 --ost 3 /cluster<br class="">
<br class="">
real<span class="Apple-tab-span" style="white-space:pre"> </span>936m8.528s<br class="">
user<span class="Apple-tab-span" style="white-space:pre"> </span>13m48.298s<br class="">
sys<span class="Apple-tab-span" style="white-space:pre"> </span>210m1.245s<br class="">
<br class="">
But still: 2624 inods are in use and 14.5G total size.<br class="">
<br class="">
# lfs df -i | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003<br class="">
cluster-OST0000_UUID 4293438576 644 4293437932 1% /cluster[OST:0]<br class="">
cluster-OST0001_UUID 4293438576 640 4293437936 1% /cluster[OST:1]<br class="">
cluster-OST0002_UUID 4293438576 671 4293437905 1% /cluster[OST:2]<br class="">
cluster-OST0003_UUID 4293438576 669 4293437907 1% /cluster[OST:3]<br class="">
<br class="">
# lfs df -h | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003<br class="">
cluster-OST0000_UUID 29.2T 3.8G 27.6T 1% /cluster[OST:0]<br class="">
cluster-OST0001_UUID 29.2T 3.7G 27.6T 1% /cluster[OST:1]<br class="">
cluster-OST0002_UUID 29.2T 3.3G 27.6T 1% /cluster[OST:2]<br class="">
cluster-OST0003_UUID 29.2T 3.7G 27.6T 1% /cluster[OST:3]<br class="">
<br class="">
I tried to check the file-system for errors:<br class="">
<br class="">
# umount /lustre/ost01<br class="">
# e2fsck -fy /dev/mapper/ost01<br class="">
<br class="">
and<br class="">
<br class="">
# lctl lfsck_start --device cluster-OST0001<br class="">
# lctl get_param -n osd-ldiskfs.cluster-OST0001.oi_scrub<br class="">
. . .<br class="">
status: completed<br class="">
<br class="">
I tried to mount OST as ldiskfs and there are several files in /O/0/d*/<br class="">
<br class="">
# umount /lustre/ost01<br class="">
# mount -t ldiskfs /dev/mapper/ost01 /mnt/<br class="">
# ls -Rhl /mnt/O/0/d*/<br class="">
. . .<br class="">
/mnt/O/0/d11/:<br class="">
-rw-rw-rw- 1 user1 group1 603K Nov 8 21:37 450605003<br class="">
/mnt/O/0/d12/:<br class="">
-rw-rw-rw- 1 user1 group1 110K Jun 16 2023 450322028<br class="">
-rw-rw-rw- 1 user1 group1 21M Nov 8 22:17 450605484<br class="">
. . .<br class="">
<br class="">
Is it expected behavior? Is it save to delere OST even with those files?<br class="">
</div>
</div>
</blockquote>
</div>
<div class=""><br class="">
</div>
You can run the debugfs "stat" command to print the "fid" xattr and it will print the MDT
<div class="">parent FID for use with "lfs fid2path" on the client to see if there are any files related</div>
<div class="">to these objects. You could also run "ll_decode_filter_fid" to do the same thing on the</div>
<div class="">mounted ldiskfs filesystem.</div>
<div class=""><br class="">
</div>
<div class="">It is likely that there are a few stray objects from deleted files, but hard to say for sure.</div>
<div class=""><br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>