<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
The simple solution is fine with me.
<div><br>
</div>
<div>Is there a way to make the kernel itself more efficient in this regard?</div>
<div>For example, limit the checks to a specific directory tree rather than</div>
<div>the whole filesystem? Granted the cache avoids network RPCs, but it</div>
<div>would be even better if the VFS is skipping these unnecessary checks. <br>
<br>
<div dir="ltr">Cheers, Andreas</div>
<div dir="ltr"><br>
<blockquote type="cite">On Oct 15, 2021, at 08:25, Ellis Wilson <elliswilson@microsoft.com> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">\ufeff
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Hi Andreas,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Yea \u2013 I noticed that. What makes it even funnier is that it makes allocation calls in statically defined chunk sizes, so it\u2019s not even like it\u2019s exactly the right size in truth. If you look at my follow-up email you\u2019ll see that I RC\u2019d
this to auditd mechanics in the kernel, which was enabled on my system.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I\u2019m going to open a bug and submit a patch for the test to be more forgiving for test systems that have auditd enabled next week. In short, it will assert that it saw at minimum two getxattr hits, rather than exactly two getxattrs hits.
That\u2019s behaved well for me since I revised it locally. Let me know if you see any issue with this simple solution.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">ellis<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Andreas Dilger <adilger@whamcloud.com> <br>
<b>Sent:</b> Friday, October 15, 2021 2:19 AM<br>
<b>To:</b> Ellis Wilson <elliswilson@microsoft.com><br>
<b>Cc:</b> lustre-devel@lists.lustre.org<br>
<b>Subject:</b> [EXTERNAL] Re: [lustre-devel] xattr stat mismatch in sanityn test 73<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Ellis, the getfattr code has a poor "optimization", in that it first calls getxattr(buf=NULL, size=0) to have the kernel report the xattr size to allocate a buffer of exactly the right size. However, IMHO while this avoids a "too large"
allocation (maybe a few bytes vs. 64KB), it is inefficient by doubling the number of syscalls into the filesystem.
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The other getxattr cache hit during lstat() is likely because of the kernel accessing an selinux/security xattr.<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Oct 8, 2021, at 07:04, Ellis Wilson via lustre-devel <<a href="mailto:lustre-devel@lists.lustre.org">lustre-devel@lists.lustre.org</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hi all,<br>
<br>
I'm trying to get to the bottom of a failure I'm seeing while running the lustre unit tests, specifically here for this test in the sanityn suite:<br>
= sanityn test 73: getxattr should not cause xattr lock cancellation =<br>
<br>
I'm running stock lustre 2.14.0 on Ubuntu 18.04 with Linux Kernel 5.4.0. I have six nodes total for the test: two clients, two mds, two oss. The test is running from one of the clients.<br>
<br>
The error I get is:<br>
============<br>
getfattr: Removing leading '/' from absolute path names<br>
# file: mnt/lustre/f73.sanityn<br>
user.attr1="value1"<br>
<br>
ELLIS: expected 2, but got 5<br>
sanityn test_73: @@@@@@ FAIL: not cached in /mnt/lustre <br>
Trace dump:<br>
= /usr/lib/lustre/tests/test-framework.sh:6273:error()<br>
= /usr/lib/lustre/tests/sanityn.sh:3557:test_73()<br>
= /usr/lib/lustre/tests/test-framework.sh:6581:run_one()<br>
= /usr/lib/lustre/tests/test-framework.sh:6628:run_one_logged()<br>
= /usr/lib/lustre/tests/test-framework.sh:6455:run_test()<br>
= /usr/lib/lustre/tests/sanityn.sh:3567:main()<br>
============<br>
<br>
I've instrumented the code to spit out the expected vs. discovered stat values. The failure indicates the file in question wasn't cached, but in fact the inverse is occurring -- it's both cached and hit more often than expected.<br>
<br>
The unadulterated test code follows:<br>
================<br>
3549 touch $DIR1/$tfile<br>
3550 setfattr -n user.attr1 -v value1 $DIR1/$tfile ||<br>
3551 error "setfattr1 failed"<br>
3552 getfattr -n user.attr1 $DIR2/$tfile || error "getfattr1 failed"<br>
3553 getfattr -n user.attr1 $DIR1/$tfile || error "getfattr2 failed"<br>
3554 clear_stats llite.*.stats<br>
3555 # PR lock should be cached by now on both clients<br>
3556 getfattr -n user.attr1 $DIR1/$tfile || error "getfattr3 failed"<br>
3557 # 2 hits for getfattr(0)+getfattr(size)<br>
3558 [ $(calc_stats llite.*.stats getxattr_hits) -eq 2 ] ||<br>
3559 error "not cached in $DIR1"<br>
================<br>
<br>
The failure occurs on line 3558.<br>
<br>
Manually performing these actions validates that indeed the jump is by 5, not 2:<br>
~# lctl get_param llite.*.stats | grep hits<br>
getxattr_hits 85 samples [reqs]<br>
getxattr_hits 4 samples [reqs]<br>
~# getfattr -n user.attr1 /mnt/lustre/f73.sanityn<br>
getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1"<br>
~# lctl get_param llite.*.stats | grep hits<br>
getxattr_hits 90 samples [reqs]<br>
getxattr_hits 4 samples [reqs]<br>
<br>
I straced getfattr as run in the test and found it issues the following:<br>
23262 lstat("/mnt/lustre/f73.sanityn", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0<br>
23262 getxattr("/mnt/lustre/f73.sanityn", "user.attr1", NULL, 0) = 6<br>
23262 getxattr("/mnt/lustre/f73.sanityn", "user.attr1", "value1", 256) = 6<br>
<br>
I built a small C program to replicate just the above without all of the other fluff in getfattr, and I see 1 xattr cache hit occurring for the lstat, and two xattr cache hits occurring for each call of getxattr. So it replicates the 5 xattr cache hits. It
is notable that if one does NOT specify "user.attr1" and instead just uses an empty string you only get a single hit on each getxattr.<br>
<br>
I have a patch that revises the expected stat values from 2 to 5 and from 4 to 10, and while that works in my system I wanted to know:<br>
1. Are these changes expected? I don't know much about the xattr cache or when it's expected to be hit, but hitting twice for a single getxattr seemed high.<br>
2. Is there any location online where I can look at release testing results for these unit tests? I wanted to see if I was alone in hitting this many times, but couldn't locate such a repository of historical test results.<br>
<br>
Thanks for any and all help!<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Cheers, Andreas<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Andreas Dilger<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Lustre Principal Architect<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Whamcloud<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>