<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
margin-bottom:.0001pt;
background:#224FBC;
font-size:10.5pt;
font-family:Menlo;
color:white;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.apple-tab-span
{mso-style-name:apple-tab-span;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Thanks Cory. Per Andreas’ suggestion, I’m trying to route this through DDN.
<o:p></o:p></p>
<p class="MsoNormal">FWIW, I learned today that we’re planning to upgrade to 2.12 at the end of this month, so probably best to baseline then and go from there.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">"Spitz, Cory James" <cory.spitz@hpe.com><br>
<b>Date: </b>Monday, July 6, 2020 at 3:21 PM<br>
<b>To: </b>Christopher Chang <Christopher.Chang@nrel.gov>, Andreas Dilger <adilger@whamcloud.com><br>
<b>Cc: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org>, "Kaiser, Timothy" <Timothy.Kaiser@nrel.gov><br>
<b>Subject: </b>Re: [lustre-discuss] Permission denied on lfs getstripe<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><span style="color:black">> Could you point me to the Jira site?</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><a href="https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjira.whamcloud.com%2F&data=02%7C01%7CChristopher.Chang%40nrel.gov%7C86ec0aab84264b30115f08d821f293cf%7Ca0f29d7e28cd4f5484427885aee7c080%7C0%7C0%7C637296673043236502&sdata=6JxZv24J335Lf3zivq6govuZYvZPq1oMx82JcZC0nC0%3D&reserved=0">https://jira.whamcloud.com</a><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">-Cory<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">On 7/2/20, 6:01 PM, "lustre-discuss on behalf of Chang, Christopher" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a> on behalf of
<a href="mailto:Christopher.Chang@nrel.gov">Christopher.Chang@nrel.gov</a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in">Hi Andreas,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> Indeed, we’re running 2.10.7 from DDN. Unfortunately, updating to more recent Lustre versions might be a hard sell to our admins without a solid case. It’s an odd problem, since we haven’t been able to track
down a correlation of the behavior with anything—it only affects certain paths and users as far as we can tell.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> Our primary goal wasn’t the getstripe, it was just a convenient universally available proxy applet that exhibited the problematic behavior. The actual problem in production is that users are unable to access
certain files or directories programmatically that they should be able to given the group permissions, and are getting permission denied errors. For example, opening a file from a Python script; running an os.stat before the read attempt sidesteps the issue
in some cases (as “ls” does from bash). If it were a localized bug in getstripe that would be great, but it seems like a somewhat deeper problem in our Lustre instance.
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> The directory doesn’t have world permissions—we’re using Lustre to host project data, and so everything is generally o-rwx permitted without specific reasons. stat shows<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="p1" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><span class="s1">File: ‘/projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min’</span><o:p></o:p></p>
<p class="p1" style="margin-left:.5in"><span class="apple-converted-space"> </span>
<span class="s1">Size: 32768 </span><span class="apple-converted-space"> </span>
<span class="apple-tab-span"> </span><span class="s1">Blocks: 64 </span><span class="apple-converted-space">
</span><span class="s1">IO Block: 4096 </span><span class="apple-converted-space">
</span><span class="s1">directory</span><o:p></o:p></p>
<p class="p1" style="margin-left:.5in"><span class="s1">Device: 83d08f32h/2211483442d</span><span class="apple-tab-span">
</span><span class="s1">Inode: 162132535332135820</span><span class="apple-converted-space">
</span><span class="s1">Links: 2</span><o:p></o:p></p>
<p class="p1" style="margin-left:.5in"><span class="s1">Access: (0770/drwxrwx---)</span><span class="apple-converted-space">
</span><span class="s1">Uid: (121756/***) </span><span class="apple-converted-space">
</span><span class="s1">Gid: (130817/ </span><span class="apple-converted-space">
</span><span class="s1">naris)</span><o:p></o:p></p>
<p class="p1" style="margin-left:.5in"><span class="s1">Context: unconfined_u:object_r:unlabeled_t:s0</span><o:p></o:p></p>
<p class="p1" style="margin-left:.5in"><span class="s1">Access: 2020-07-02 10:16:52.000000000 -0600</span><o:p></o:p></p>
<p class="p1" style="margin-left:.5in"><span class="s1">Modify: 2020-04-10 19:40:39.000000000 -0600</span><o:p></o:p></p>
<p class="p1" style="margin-left:.5in"><span class="s1">Change: 2020-05-14 09:50:37.000000000 -0600</span><o:p></o:p></p>
<p class="p1" style="margin-left:.5in"><span class="apple-converted-space"> </span><span class="s1">Birth: -</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Could you point me to the Jira site?<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Thanks again for all your help.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><br>
Chris<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Andreas Dilger <adilger@whamcloud.com><br>
<b>Date: </b>Thursday, July 2, 2020 at 3:25 PM<br>
<b>To: </b>Christopher Chang <Christopher.Chang@nrel.gov><br>
<b>Cc: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org>, "Kaiser, Timothy" <Timothy.Kaiser@nrel.gov><br>
<b>Subject: </b>Re: [lustre-discuss] Permission denied on lfs getstripe</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in">Chris, <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">this looks like a bug that "lfs getstripe -M" is not using supplementary groups, or similar. You wrote that the directory has GID=130817, so this is not the primary GID of the user accessing it, so it must depend
on the supplementary group permissions to access it. The "regular" ls access *is* using the supplementary GID to allow access, and when the directory is cached on the client then "lfs getstripe -M" is getting this information out of the client-side cache
(where the client VFS is locally checking the GID for access permission).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I suspect this hasn't really been an issue in the past because few users use "lfs getstripe -M", and most of those are root or are accessing their own files/directories, so do not need a supplementary group to access
this information. It also seems (but isn't shown) that the directory does not have world-read permission? What does "stat" on this directory show?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Could you please file a ticket in Jira with the details so that this issue can be tracked. I don't know how easy/hard it will be to fix this, since this information is obtained via ioctl(), and we don't necessarily
want non-owners of files to be able to call every ioctl on the file/directory.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Note, it is recommended to use "lfs getdirstripe --m" (or "--mdt-index") instead of "-M" to get the MDT index of a file, since the "-M" option is deprecated to This would imply you are running a Lustre 2.10 client?
The "-m" option is already available in 2.10, and "-M" will print a warning in 2.12 and later.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I tested this on master and was not able to reproduce the problem. If I set the directory mode=0640 I got permission denied for directories that I didn't have supplementary group access on, but it worked on the
first try (after flushing all client locks and dropping all caches). That means the problem seems to already be fixed in master, and possibly 2.12 also.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Cheers, Andreas<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><br>
<br>
<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-left:.5in">On Jul 2, 2020, at 10:26, Chang, Christopher <<a href="mailto:Christopher.Chang@nrel.gov">Christopher.Chang@nrel.gov</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi Andreas,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> It doesn’t appear to be this issue. I verified the client “id” and server “l_getidentity -d” views before and after issuing an “ls” as the user to get getstripe working, and there’s no change.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Client:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">el3:~> id</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">uid=131364(***) gid=131364(***) groups=131364(***),130033(globus-access),130774(eagle-users),130808(ewer),130817(naris),131016(esp-wps-inputs),131178(lex-access),131237(naermpcm),249837(aces),249945(hpcapps),249996(n-apps)
context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span style="font-size:10.5pt;font-family:Menlo;color:white"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">el3:~> lfs getstripe -M /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">error opening /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min: Permission denied (13)</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">…</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">el3:~> ls /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">~Model ( c_RT5min_</span></span><span style="font-size:10.5pt;font-family:Menlo;color:white">...</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span style="font-size:10.5pt;font-family:Menlo;color:white"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">el3:~> lfs getstripe -M /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">1</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white"> </span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">el3:~> id</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">uid=131364(***) gid=131364(***) groups=131364(***),130033(globus-access),130774(eagle-users),130808(ewer),130817(naris),131016(esp-wps-inputs),131178(lex-access),131237(naermpcm),249837(aces),249945(hpcapps),249996(n-apps)
context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Server:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">[root@mds02 ~]# l_getidentity -d 131364</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">uid=131364 gid=131364,130808,130817,131016,131237,249837,249945,249996</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">permissions:</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:Menlo;color:white"> </span></span><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">nid</span></span><span class="apple-tab-span"><span style="font-size:10.5pt;font-family:Menlo;color:white"> </span></span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:Menlo;color:white"> </span></span><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">perm</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">(client does an ls)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">[root@mds02 ~]# l_getidentity -d 131364</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">uid=131364 gid=131364,130808,130817,131016,131237,249837,249945,249996</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">permissions:</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in;background:#224FBC"><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:Menlo;color:white"> </span></span><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">nid</span></span><span class="apple-tab-span"><span style="font-size:10.5pt;font-family:Menlo;color:white"> </span></span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:Menlo;color:white"> </span></span><span class="s1"><span style="font-size:10.5pt;font-family:Menlo;color:white">perm</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">The relevant gid for the target directory is 130817. I verified that all 3 of our MDSs had the same view before and after the “ls”.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Thanks; Chris<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt">From:<span class="apple-converted-space"> </span></span></b><span style="font-size:12.0pt">Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Sunday, June 28, 2020 at 5:11 PM<br>
<b>To:<span class="apple-converted-space"> </span></b>Christopher Chang <<a href="mailto:Christopher.Chang@nrel.gov">Christopher.Chang@nrel.gov</a>><br>
<b>Cc:<span class="apple-converted-space"> </span></b>"<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>>, "Kaiser, Timothy" <<a href="mailto:Timothy.Kaiser@nrel.gov">Timothy.Kaiser@nrel.gov</a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>Re: [lustre-discuss] Permission denied on lfs getstripe</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">On Jun 26, 2020, at 10:45, Chang, Christopher <<a href="mailto:Christopher.Chang@nrel.gov">Christopher.Chang@nrel.gov</a>> wrote:<o:p></o:p></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> We’re running into an error with a particular directory. It is weird because it can be resolved in an unexpected way, but only for a time.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">The error manifests as:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Courier New"">el3:out> lfs getstripe -M /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min <br>
error opening /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min: Permission denied (13)<br>
llapi_semantic_traverse: Failed to open '/projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min': Permission denied (13)<br>
error: getstripe failed for /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">The temporary resolution is:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Courier New"">el3:out> ls /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min <br>
~Model ( c_RT5min_TechBreak2050_092P_OLd000_001 ) Log.txt Model c_RT5min_TechBreak2050_092P_OLd000_033 Solution.h5 Model c_RT5min_TechBreak2050_092P_OLd000_062 Solution.h5</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">…</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Then<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Courier New"">el3:out> lfs getstripe -M /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min <br>
1<br>
el3:out></span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">It looks like the user might only have supplementary group access to this file? You could check on the client by running "id" to list the primary user ID and supplementary groups, then "ls -ln" on the file to see
what group it is owned by.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">If that is the case, it would indicate that the MDS /etc/group (or other source of supplementary group information, like NIS or LDAP, via /etc/nsswitch.conf) is not up-to-date with what is on the clients, or you
have mdt.*.identity_upcall=NONE on the MDS instead of =l_getidentity. You can test what l_getidentity on the MDS thinks the supplementary groups are for a particular user by running "l_getidentity -d <uid>" to compare what "id" returns on the client.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Cheers, Andreas<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">However, the getstripe command will only continue to work for about 10 minutes, then it goes back to the permission denied errors.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">It only happens with a selection of files or directories, so we were thinking it might be connected to a particular OSS or MDT, but not sure what to look for.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I am not the Lustre admin, so please forgive incomplete information. If folks can request specific command output, preferably from user space, that would accelerate my ability to answer questions. If something needs
to get run while logged into a particular Lustre component (MDT, OSS, etc.), please do not hesitate to assume that I don’t know that.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">We’re running Lustre 2.10.7 provided by DDN on CentOS 7.4. All help appreciated, thanks!<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Chris<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">-- <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Christopher H. Chang, Ph.D.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Computational Scientist<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">National Renewable Energy Laboratory<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">15013 Denver West Pkwy., MS ESIF301<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Golden, CO 80401<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Helvetica">_______________________________________________<br>
lustre-discuss mailing list<br>
</span><a href="mailto:lustre-discuss@lists.lustre.org"><span style="font-size:9.0pt;font-family:Helvetica">lustre-discuss@lists.lustre.org</span></a><span style="font-size:9.0pt;font-family:Helvetica"><br>
</span><a href="https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=02%7C01%7CChristopher.Chang%40nrel.gov%7C86ec0aab84264b30115f08d821f293cf%7Ca0f29d7e28cd4f5484427885aee7c080%7C0%7C0%7C637296673043246495&sdata=ehi%2BNzjqPB4OfQG%2BRbCcGalhtR5ML3uzH8sxE4fPVaw%3D&reserved=0"><span style="font-size:9.0pt;font-family:Helvetica">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</span></a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Cheers, Andreas<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">--<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Andreas Dilger<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Principal Lustre Architect<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Whamcloud<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">Cheers, Andreas</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">--</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">Andreas Dilger</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">Principal Lustre Architect</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">Whamcloud</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
</body>
</html>