[lustre-devel] [RFC] new 'lfs usage' subcommand: per-id usable space + quota headroom

[Andreas Dilger]

Hi Andreas,
my preference would actually be that "lfs df DIR" would also scale the per-OST results based on the project quota assigned to each OST in a similar manner, rather than introducing a new command for this.  This was a shortcoming in the original implementation that should really be fixed.

I would prefer just adding a new option like `lfs df --quota [ugp]` or similar (which can work with the existing `--output={free,total,available}` options as needed) to merge the statfs+quota values together as needed, instead of a new command.  That avoids having a separate command that duplicates a lot of the functionality of "lfs df".

Note also that any kind of free space check on the filesystem in advance of the application running is going to be racy with other applications running using the same user/project/group, quota or filling the filesystem.  Obviously if there is already *less* space available in the filesystem than the application needs then it is useful, but the free space check is not nearly as useful.

We've seen misguided applications going so far as to call statfs() before every write() call to check if there is enough free space for *that* write (which we've had to deal with), but this is not really better than just checking the return code from write(), since multiple concurrent writers can consume the reported free space anyway.

It *is* possible to use `fallocate(2)` (or the `fallocate(1)` command) to reserve space for output files on ldiskfs, if you know how the names and how big they will be in advance, and this doesn't send any data over the network.  The "portable" posix_fallocate(3) command will write zeroes in some cases, rather than return an error, which is a good reason not to use it.

Cheers, Andreas

> On Jul 21, 2026, at 12:41, Andreas Krause via lustre-devel <lustre-devel at lists.lustre.org> wrote:
> 
> Hi all
> 
> I'd like feedback on a small client-utility addition before I open an LU.
> 
> Problem statement
> -------
> A user (or a job prolog) who wants to know "how much can I actually write
> here before something shuts the door and stops me?" today has to run two commands and reconcile
> them by hand:
> 
>   - `lfs df`     -> physical free space, but blind to the caller's quota
>   - `lfs quota`  -> the caller's quota limit, but blind to physical free space
> 
> The information the user or job is looking for is min(filesystem free, quota hard limit - used),
> per filesystem, for both space/bytes and inodes. 
> Getting it wrong means jobs that die mid-run on EDQUOT/ENOSPC that a prolog check could have caught.
> 
> Proposal (step1) -- a read-only 'lfs usage' subcommand
> --------------------------------------------------
>   lfs usage [--user|-u ID] [--group|-g ID] [--projid|-p ID]
>             [--human-readable|-h] [PATH ...]
> 
>   - With no id, reports the process user/context calling euid.
>   - Per filesystem, prints used / limit / usable / use% / fs_free for both
>     space and inodes, where usable = min(fs_free, hardlimit - used).
>   - Bare numeric output by default (1-KiB units for space, like `lfs quota`);
>     -h for human-readable. Follows existing lfs conventions.
> 
> Implementation: 
> It would reuse existing client interfaces only -- statfs(2) for the
> physical free (the same number df(1) reports) and OBD_IOC_QUOTACTL /
> LUSTRE_Q_GETQUOTA (QC_GENERAL) for the IDs' live usage+limits, which is already aggregated across all OSTs and MDTs (similar if not the same as `lfs quota` does). 
> No new RPC, no wire-protocol change, works against any current server.
> Usage is reported even when enforcement is off, since accounting is always-on; at least that’s how I understand it.
> 
> Deliberately NOT changing df/statfs
> -----------------------------------
> This is intentionally a *reporting* tool, not a change to what plain `df` or
> statfs() returns. It does not touch or duplicate the existing project-quota
> statfs mechanism (ll_statfs_project / statfs_project, LU-9555) or the in-flight
> projid df work (LU-19034, Gerrit 65511). It complements them from the utility
> side and adds the user/group dimension, for which there is no prior code as of today if I didn’t miss anything.
> 
> Known step1 limitation -> motivates step2
> -----------------------------------
> Enforcement on/off state is not currently retrievable by a client (the qsd
> enable mask is never packed into any RPC reply; it's server-side procfs only).
> So when a limit is set but not enforced, step1's "usable” might underreport the real
> writable space -- similar to LU-17395.
> As, step2, I'd like to add an enforcement-state bit to the quota
> reply so `lfs usage` (and, if wanted, ll_statfs_project) can distinguish a
> binding limit from a suggesting one. 
> That is the only piece that needs a wire
> change, and I'd like to propose it being implemented alongside the existing quota/statfs work rather
> than on its own.
> 
> Questions for the readers here
> ----------------------
>   1. Is a new `lfs usage` subcommand the right home/place, or would you prefer this
>      folded into `lfs quota` (e.g. a --usable / --df column)? I lean toward a
>      separate subcommand because the physical-free term doesn't really belong in
>      `lfs quota`, and `lfs df -u` is impossible now that -u is —output, but that’s me.
>   2. Any objection to the step1 reporting approach landing first, decoupled from
>      the v2 enforcement-signal wire change?
> 
> A strawman patch (utils-only, plus man page and a sanity-quota test) is ready;
> I'll attach it to the LU once there's rough consensus on the interface, and it's not shut down here.
> 
> Thanks,
> Andreas Krause
> 
> -- 
> lustre-devel mailing list
> lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel_lists.lustre.org

---
Andreas Dilger
Principal Lustre Architect
adilger at thelustrecollective.com