<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Sep 22, 2022, at 03:21, Michael Weiser via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="">Hey Daniel! :)<br class="">
Hi Jeremy,<br class="">
Hi Bjoern,<br class="">
<br class="">
I'm cross-posting to the samba list again as I think this might be of interest there as well and to keep the threads together.<br class="">
<br class="">
<blockquote type="cite" class="">
<blockquote type="cite" class="">That leaves the question, where that extended attribute user.lustre.lov is coming from. It<br class="">
appears that Lustre itself exposes a number of extended attributes for every file which<br class="">
reflect internal housekeeping data:<br class="">
<br class="">
$ getfattr -m - blarf<br class="">
# file: blarf<br class="">
lustre.lov<br class="">
<a href="http://trusted.link" class="">trusted.link</a><br class="">
trusted.lma<br class="">
trusted.lov<br class="">
</blockquote>
</blockquote>
<br class="">
<blockquote type="cite" class="">Try adding<br class="">
</blockquote>
<blockquote type="cite" class=""><br class="">
</blockquote>
<blockquote type="cite" class=""> lustre.* skip</blockquote>
</div>
</div>
</blockquote>
<blockquote type="cite" class="">
<blockquote type="cite" class=""><br class="">
</blockquote>
</blockquote>
<blockquote type="cite" class="">
<div class="">
<div class="">
<blockquote type="cite" class="">to /etc/xattr.conf (cf.<br class="">
<a href="https://doc.lustre.org/lustre_manual.xhtml#lustre_configure_multiple_fs" class="">https://doc.lustre.org/lustre_manual.xhtml#lustre_configure_multiple_fs</a>).<br class="">
Haven't tested yet, but Samba's EA handling seems to be libattr-based,<br class="">
so the above tweak should be sufficient to keep smbd off of these<br class="">
fs-specific attributes.<br class="">
</blockquote>
<br class="">
Thanks for the tip! I tried and unfortunately it didn't work. From my looking at the code it seems that samba uses libattr only as a fallback if no compatble system implementation of fgetxattr can be found. In my cases (RHEL 8.6/samba-4.15.5 and debug system
debian testing/samba-4.17.0) it seems to use the system interface directly.<br class="">
<br class="">
Running with that: From the Lustre documentation it appears that there have been problems with exposing Lustre internal data via extended attributes in the past, prompting the xattr.conf workaround:<br class="">
<br class="">
[from the docs]:<br class="">
<blockquote type="cite" class="">If a client(s) will be mounted on several file systems, add the following line to /etc/xattr.conf file to avoid problems when files are moved between the file systems: lustre.* skip"<br class="">
</blockquote>
<br class="">
What exactly were those problems hinted at in the documentation?<br class="">
Is the visibility of the lustre.lov attribute for unprivileged users actually needed for anything?<br class="">
Can exposing it to unprivileged users be switched off Lustre-side?<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>The lustre.lov xattr is used for backup and restore of the file layout, used if files are stored across multiple servers (e.g. if they are huge, or have mixed flash + disk storage, mirrored, etc). It isn't critical that this is saved/restored, but it
is useful. Tools like tar can save/restore this xattr to preserve this layout across filesystems, or if archived to tape, etc.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">Looking at xattr.conf highlights the fact that Lustre isn't as singular as I thought in putting out those lustre.lov attributes. At least AFS and XFS seem to do the same or at least at some point in time have done so. (From the comment "obsolete"
on xfsroot.* it appears that may have been changed.)<br class="">
<br class="">
Jeremy wrote:<br class="">
<blockquote type="cite" class="">Great analysis Michael ! As we're emulating NTFS CreateFile<br class="">
we can't do the 'create with EA's' atomically.<br class="">
</blockquote>
<blockquote type="cite" class=""><br class="">
</blockquote>
<blockquote type="cite" class="">Lustre really should not be exposing EA's to callers if it doesn't actually support EA's.<br class="">
</blockquote>
<blockquote type="cite" class=""><br class="">
</blockquote>
<blockquote type="cite" class="">An elegant solution might be to add a Samba VFS module<br class="">
vfs_lustre.c that intercepts fgetxattr/fsetxattr/flistxattr calls and simply<br class="">
strips out the lustre.lov EA's from being seen.<br class="">
</blockquote>
<br class="">
I had a first impelementation of that attached to my first mail. Did that get through? In my case it was enough to mask lustre.lov in flistxattr and fgetxattr so that clients never get to see it and fsetxattr is never attempted.<br class="">
<br class="">
What's bugging me about this approach as well as the xattr.conf workaround is that the error behaviour on the client side is so very unintuitive. How will we get people to correlate some "file already exists" error with peculiar extended attributes behaviour
of their file system so they become aware they need to configure a workaround? It'd certainly be nice if we could find a way for samba to "just do the right thing"[tm].<br class="">
<br class="">
<blockquote type="cite" class="">Can you log a bug in our bugzilla and upload all this info so we can track it ?<br class="">
</blockquote>
<br class="">
That's underway (requested an account) and I'll upload everything there.<br class="">
<br class="">
<blockquote type="cite" class="">
<blockquote type="cite" class="">is this really Lustre specific? I assume we see the same effect on Linux with<br class="">
other filesystems that don't support EAs.<br class="">
</blockquote>
</blockquote>
<blockquote type="cite" class=""><br class="">
</blockquote>
<blockquote type="cite" class="">No. Lustre is returning "fictional" EA's that<br class="">
cannot be set. Linux filesystems that don't have<br class="">
EA's don't do that.<br class="">
</blockquote>
<br class="">
The attributes using a non-existant namespace (lustre.*) doesn't seem exactly right[tm] to me either. And it would wreak havoc if samba were actually able to set the canonicalised user.lustre.lov attribute when copying it back, duplicating and likely somewhat
non-deterministically overwriting it later.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
There *is* a Lustre xattr namespace, it just only gets used by Lustre clients... That said, the root of the problem (AFAICS) is that the xattr name is changed from lustre.lov to user.lustre.lov as it moves from Lustre->NTFS->Lustre. If the xattr name was
kept consistent (e.g. as with tar, rsync, etc.) then there shouldn't be a problem.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">But more crucially, what seems problematic here is that Lustre supports listing and reading extended attributes for unprivileged users but does not allow setting them and returns ENOTSUP rather than EPERM or something else at that. So samba would
need to take into account that not all filesystems support extended attributes as a whole but might support some operations on them but not others. I'm with Bjoern that there likely are or will be other filesystems with peculiar extended attribute behaviour.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
Again, this is because the NTFS->Lustre xattr copy is changing from lustre.lov to user.lustre.lov, and the "-o user_xattr" feature is not enabled on this filesystem. If "-o user_xattr" was enabled on the server, then the "user.lustre.lov" xattr would be saved
without errors, but it wouldn't serve its intended purpose. That isn't fatal if the lustre.lov isn't restored, but storing it twice is a waste of space.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">What might be the possible fallout from removing the created file in the error code path? Shouldn't it be safe with proper locking in place as it appears to be? Wouldn't a best-effort cleanup in the error path be better than leaving a known-to-be-incorrect
state behind?<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
This seems like a bug in Samba, or maybe in Windows? Creating a large number of zero-length files seems problematic/racy, since any interruption during this process (for any reason, not just the xattr error) would leave a lot of cruft behind. It might be
possible to mark those files as 'placeholders' based on the file access permissions or similar, so that a retry would overwrite them, but that is still hackish.</div>
<div><br class="">
</div>
<div>Lustre itself does not yet support O_TMPFILE, which would allow instantiating the files fully-formed into the namespace, though that is something we've been looking at adding. There is an equivalent functionality to create "nameless files" that predates
O_TMPFILE that Lustre-specific tools use, but it doesn't allow the "post-link" operation that O_TMPFILE does.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class=""># /etc/xattr.conf<br class="">
#<br class="">
# Format:<br class="">
# <pattern> <action><br class="">
#<br class="">
# Actions:<br class="">
# permissions - copy when trying to preserve permissions.<br class="">
# skip - do not copy.<br class="">
<br class="">
system.nfs4_acl permissions<br class="">
system.nfs4acl permissions<br class="">
system.posix_acl_access permissions<br class="">
system.posix_acl_default permissions<br class="">
trusted.SGI_ACL_DEFAULT skip # xfs specific<br class="">
trusted.SGI_ACL_FILE skip # xfs specific<br class="">
trusted.SGI_CAP_FILE skip # xfs specific<br class="">
trusted.SGI_DMI_* skip # xfs specific<br class="">
trusted.SGI_MAC_FILE skip # xfs specific<br class="">
xfsroot.* skip # xfs specific; obsolete<br class="">
user.Beagle.* skip # ignore Beagle index data<br class="">
security.evm skip # may only be written by kernel<br class="">
afs.* skip # AFS metadata and ACLs<br class="">
lustre.* skip<br class="">
</div>
</div>
</blockquote>
<br class="">
</div>
<div>Having Samba honor the xattr.conf (even if it is not using libattr) would at least make the behavior consistent between Samba and cp and other tools.</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>