[lustre-discuss] OST still has inodes and size after deleting all files

[Andreas Dilger]

On Jan 19, 2024, at 13:48, Pavlo Khmel via lustre-discuss <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>> wrote:

Hi,

I'm trying to remove 4 OSTs.

# lfs osts
OBDS:
0: cluster-OST0000_UUID ACTIVE
1: cluster-OST0001_UUID ACTIVE
2: cluster-OST0002_UUID ACTIVE
3: cluster-OST0003_UUID ACTIVE
. . .

I moved all files to other OSTs. "lfs find" cannot find any files on these 4 OSTs.

# time lfs find --ost 0 --ost 1 --ost 2 --ost 3 /cluster

real 936m8.528s
user 13m48.298s
sys 210m1.245s

But still: 2624 inods are in use and 14.5G total size.

# lfs df -i | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003
cluster-OST0000_UUID  4293438576         644  4293437932   1% /cluster[OST:0]
cluster-OST0001_UUID  4293438576         640  4293437936   1% /cluster[OST:1]
cluster-OST0002_UUID  4293438576         671  4293437905   1% /cluster[OST:2]
cluster-OST0003_UUID  4293438576         669  4293437907   1% /cluster[OST:3]

# lfs df -h | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003
cluster-OST0000_UUID       29.2T        3.8G       27.6T   1% /cluster[OST:0]
cluster-OST0001_UUID       29.2T        3.7G       27.6T   1% /cluster[OST:1]
cluster-OST0002_UUID       29.2T        3.3G       27.6T   1% /cluster[OST:2]
cluster-OST0003_UUID       29.2T        3.7G       27.6T   1% /cluster[OST:3]

I tried to check the file-system for errors:

# umount /lustre/ost01
# e2fsck -fy /dev/mapper/ost01

and

# lctl lfsck_start --device cluster-OST0001
# lctl get_param -n osd-ldiskfs.cluster-OST0001.oi_scrub
. . .
status: completed

I tried to mount OST as ldiskfs and there are several files in /O/0/d*/

# umount /lustre/ost01
# mount -t ldiskfs /dev/mapper/ost01 /mnt/
# ls -Rhl /mnt/O/0/d*/
. . .
/mnt/O/0/d11/:
-rw-rw-rw- 1 user1 group1 603K Nov  8 21:37 450605003
/mnt/O/0/d12/:
-rw-rw-rw- 1 user1 group1 110K Jun 16  2023 450322028
-rw-rw-rw- 1 user1 group1  21M Nov  8 22:17 450605484
. . .

Is it expected behavior? Is it save to delere OST even with those files?

You can run the debugfs "stat" command to print the "fid" xattr and it will print the MDT
parent FID for use with "lfs fid2path" on the client to see if there are any files related
to these objects.  You could also run "ll_decode_filter_fid" to do the same thing on the
mounted ldiskfs filesystem.

It is likely that there are a few stray objects from deleted files, but hard to say for sure.

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud







-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20240120/5de2acb5/attachment-0001.htm>