[lustre-discuss] Lustre chown and create operations failing for certain UIDs

[Russell Dekema]

We've got a Lustre system running lustre-2.5.42.28.ddn8 and are having
a problem with it that none of us here have ever seen before. We are
wondering if anyone here has seen this or has any idea what might be
causing it.

(I have redacted the example affected username and its corresponding
UID in this message for privacy reasons, and replaced them with the
strings 'affected_username' and 'uid_of_the_affected_username'.)

Of the 4,463 UIDs on our system, there are 24 UIDs for which Lustre
returns an error when those UIDs are used as the owner of files in a
chown or create operation, as in:

# chown affected_username testfile
chown: changing ownership of ‘testfile’: No such process

This occurs both when root tries to chown a file for the affected UID
and when the user corresponding to that UID tries to create a file in
a directory to which they otherwise have the proper permissions.

If, as root, we run "strace chown affected_username testfile", the
system call that fails appears to be:

fchownat(AT_FDCWD, "testfile", uid_of_the_affected_username, -1, 0) =
-1 ESRCH (No such process)

Another unusual thing we have noticed about these users is that we are
unable to look up their quotas on the Lustre filesystem, even though
quota lookups work fine for other users:

$ sudo lfs quota -u affected_username /scratch
usr quotas are not enabled.

If anyone has any ideas about what might be causing this or what we
might try in order to fix or further diagnose it, I'll be glad to hear
them. We do have a ticket open with our vendor but wanted to see if
anyone else had heard of this while we await their response.

Sincerely,
Rusty Dekema
University of Michigan
Advanced Research Computing - Technology Services