[lustre-discuss] OST still has inodes and size after deleting all files

[Pavlo Khmel]

Hi,

I'm trying to remove 4 OSTs. 

# lfs osts
OBDS:
0: cluster-OST0000_UUID ACTIVE
1: cluster-OST0001_UUID ACTIVE
2: cluster-OST0002_UUID ACTIVE
3: cluster-OST0003_UUID ACTIVE
. . .

I moved all files to other OSTs. "lfs find" cannot find any files on these 4 OSTs.

# time lfs find --ost 0 --ost 1 --ost 2 --ost 3 /cluster

real	936m8.528s
user	13m48.298s
sys	210m1.245s

But still: 2624 inods are in use and 14.5G total size.

# lfs df -i | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003
cluster-OST0000_UUID  4293438576         644  4293437932   1% /cluster[OST:0]
cluster-OST0001_UUID  4293438576         640  4293437936   1% /cluster[OST:1]
cluster-OST0002_UUID  4293438576         671  4293437905   1% /cluster[OST:2]
cluster-OST0003_UUID  4293438576         669  4293437907   1% /cluster[OST:3]

# lfs df -h | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003
cluster-OST0000_UUID       29.2T        3.8G       27.6T   1% /cluster[OST:0]
cluster-OST0001_UUID       29.2T        3.7G       27.6T   1% /cluster[OST:1]
cluster-OST0002_UUID       29.2T        3.3G       27.6T   1% /cluster[OST:2]
cluster-OST0003_UUID       29.2T        3.7G       27.6T   1% /cluster[OST:3]

I tried to check the file-system for errors:

# umount /lustre/ost01
# e2fsck -fy /dev/mapper/ost01

and

# lctl lfsck_start --device cluster-OST0001
# lctl get_param -n osd-ldiskfs.cluster-OST0001.oi_scrub
. . .
status: completed

I tried to mount OST as ldiskfs and there are several files in /O/0/d*/

# umount /lustre/ost01
# mount -t ldiskfs /dev/mapper/ost01 /mnt/
# ls -Rhl /mnt/O/0/d*/
. . .
/mnt/O/0/d11/:
-rw-rw-rw- 1 user1 group1 603K Nov  8 21:37 450605003
/mnt/O/0/d12/:
-rw-rw-rw- 1 user1 group1 110K Jun 16  2023 450322028
-rw-rw-rw- 1 user1 group1  21M Nov  8 22:17 450605484
. . .

Is it expected behavior? Is it save to delere OST even with those files?

Best regards,
Pavlo Khmel