[lustre-discuss] How to enable project quota for old files after upgrade?

[Robert Redl]

Dear Lustre Experts,

I have upgraded a test system from Lustre 2.12.7 with ZFS 0.7.13 backend 
to Lustre 2.15.0-RC5 with ZFS 2.0.7 backend. The data has been 
transferred using zfs send/recv to new hardware.

Project quotas are activated on all zpools as well as in the lustre 
config. For new files and folders, that have been created after the 
upgrade, everything works complete as expected. For old files and 
folders, setting the project quota fails:

lfs project -sr -p1 old-folder
lfs: failed to set xattr for 'old-folder': No such device or address

This issue is already reported and has already been there in Lustre 
2.14: https://jira.whamcloud.com/browse/LU-15640

A workaround is to migrate the old folder between MDTs. That is 
recreating the attributes and apparently the project ID attribute is 
afterwards included and can be set. Setting the project ID after 
migrating between MDTs results in correctly shown number of files with 
lfs quota -p. Unfortunately the consumed disk space shown by lfs quota 
-p is much to small. Also an LBUG is triggered from time to time on the 
OSTs. This has also been reported: 
https://jira.whamcloud.com/browse/LU-13189. The Problem seems to be, 
that the osd-objects are expected to have the project ID attribute, but 
they don't. The only way to get out of this situation is to evict all 
clients and to delete the files in question. Otherwise you would 
directly run into the next LBUG after the reboot of the OST. Another 
workaround is to migrate all data to different OSTs (also old upgraded). 
As for the meta data, that creates new objects which include the project 
ID.

The migration workarounds are of course extremely expensive on larger 
systems. According to the zfs documentation, an update of the on-disk 
format can be triggered by zfs set version=current 
(https://openzfs.github.io/openzfs-docs/man/7/zpool-features.7.html). 
However, that is not doing anything.

I guess I'm not the only one with plans to upgrade a system with ZFS 
backend. Did others have the same issues?

Best regards,
Robert