<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Chris,
<div class="">this looks like a bug that "lfs getstripe -M" is not using supplementary groups, or similar. You wrote that the directory has GID=130817, so this is not the primary GID of the user accessing it, so it must depend on the supplementary group permissions
to access it. The "regular" ls access *is* using the supplementary GID to allow access, and when the directory is cached on the client then "lfs getstripe -M" is getting this information out of the client-side cache (where the client VFS is locally checking
the GID for access permission).</div>
<div class=""><br class="">
</div>
<div class="">I suspect this hasn't really been an issue in the past because few users use "lfs getstripe -M", and most of those are root or are accessing their own files/directories, so do not need a supplementary group to access this information. It also
seems (but isn't shown) that the directory does not have world-read permission? What does "stat" on this directory show?</div>
<div class=""><br class="">
</div>
<div class="">Could you please file a ticket in Jira with the details so that this issue can be tracked. I don't know how easy/hard it will be to fix this, since this information is obtained via ioctl(), and we don't necessarily want non-owners of files to
be able to call every ioctl on the file/directory.</div>
<div class=""><br class="">
</div>
<div class="">Note, it is recommended to use "lfs getdirstripe --m" (or "--mdt-index") instead of "-M" to get the MDT index of a file, since the "-M" option is deprecated to This would imply you are running a Lustre 2.10 client? The "-m" option is already
available in 2.10, and "-M" will print a warning in 2.12 and later.</div>
<div class=""><br class="">
</div>
<div class="">I tested this on master and was not able to reproduce the problem. If I set the directory mode=0640 I got permission denied for directories that I didn't have supplementary group access on, but it worked on the first try (after flushing all client
locks and dropping all caches). That means the problem seems to already be fixed in master, and possibly 2.12 also.</div>
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 2, 2020, at 10:26, Chang, Christopher <<a href="mailto:Christopher.Chang@nrel.gov" class="">Christopher.Chang@nrel.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Hi Andreas,<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
It doesn’t appear to be this issue. I verified the client “id” and server “l_getidentity -d” views before and after issuing an “ls” as the user to get getstripe working, and there’s no change.<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Client:<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">el3:~> id</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">uid=131364(***) gid=131364(***) groups=131364(***),130033(globus-access),130774(eagle-users),130808(ewer),130817(naris),131016(esp-wps-inputs),131178(lex-access),131237(naermpcm),249837(aces),249945(hpcapps),249996(n-apps) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">el3:~> lfs getstripe -M /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">error opening /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min: Permission denied (13)<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">…<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">el3:~> ls /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">~Model ( c_RT5min_</span>...<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">el3:~> lfs getstripe -M /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">1<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1"><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">el3:~> id</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">uid=131364(***) gid=131364(***) groups=131364(***),130033(globus-access),130774(eagle-users),130808(ewer),130817(naris),131016(esp-wps-inputs),131178(lex-access),131237(naermpcm),249837(aces),249945(hpcapps),249996(n-apps) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Server:<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">[root@mds02 ~]# l_getidentity -d 131364</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">uid=131364 gid=131364,130808,130817,131016,131237,249837,249945,249996</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">permissions:</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="apple-converted-space"> <span class="Apple-converted-space"> </span></span><span class="s1">nid</span><span class="apple-tab-span"> <span class="Apple-converted-space"> </span></span><span class="s1">perm</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
(client does an ls)<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">[root@mds02 ~]# l_getidentity -d 131364</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">uid=131364 gid=131364,130808,130817,131016,131237,249837,249945,249996</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="s1">permissions:</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; background-color: rgb(34, 79, 188); font-size: 10.5pt; font-family: Menlo; color: white;" class="">
<span class="apple-converted-space"> <span class="Apple-converted-space"> </span></span><span class="s1">nid</span><span class="apple-tab-span"> <span class="Apple-converted-space"> </span></span><span class="s1">perm</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
The relevant gid for the target directory is 130817. I verified that all 3 of our MDSs had the same view before and after the “ls”.<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Thanks; Chris<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Andreas Dilger <<a href="mailto:adilger@whamcloud.com" class="">adilger@whamcloud.com</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Sunday, June 28, 2020 at 5:11 PM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>Christopher Chang <<a href="mailto:Christopher.Chang@nrel.gov" class="">Christopher.Chang@nrel.gov</a>><br class="">
<b class="">Cc:<span class="Apple-converted-space"> </span></b>"<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>>, "Kaiser,
Timothy" <<a href="mailto:Timothy.Kaiser@nrel.gov" class="">Timothy.Kaiser@nrel.gov</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [lustre-discuss] Permission denied on lfs getstripe<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On Jun 26, 2020, at 10:45, Chang, Christopher <<a href="mailto:Christopher.Chang@nrel.gov" style="color: blue; text-decoration: underline;" class="">Christopher.Chang@nrel.gov</a>> wrote:<o:p class=""></o:p></div>
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Hi,<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
We’re running into an error with a particular directory. It is weird because it can be resolved in an unexpected way, but only for a time.<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
The error manifests as:<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10.5pt; font-family: "Courier New";" class="">el3:out> lfs getstripe -M /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min <br class="">
error opening /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min: Permission denied (13)<br class="">
llapi_semantic_traverse: Failed to open '/projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min': Permission denied (13)<br class="">
error: getstripe failed for /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min.</span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
The temporary resolution is:<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10.5pt; font-family: "Courier New";" class="">el3:out> ls /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min <br class="">
~Model ( c_RT5min_TechBreak2050_092P_OLd000_001 ) Log.txt Model c_RT5min_TechBreak2050_092P_OLd000_033 Solution.h5 Model c_RT5min_TechBreak2050_092P_OLd000_062 Solution.h5</span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">…</span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Then<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10.5pt; font-family: "Courier New";" class="">el3:out> lfs getstripe -M /projects/naris/pcm_110819/NARIS_TechBreak2050_missingDPV/StageC_RT5min <br class="">
1<br class="">
el3:out></span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
</div>
</blockquote>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
It looks like the user might only have supplementary group access to this file? You could check on the client by running "id" to list the primary user ID and supplementary groups, then "ls -ln" on the file to see what group it is owned by.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
If that is the case, it would indicate that the MDS /etc/group (or other source of supplementary group information, like NIS or LDAP, via /etc/nsswitch.conf) is not up-to-date with what is on the clients, or you have mdt.*.identity_upcall=NONE on the MDS instead
of =l_getidentity. You can test what l_getidentity on the MDS thinks the supplementary groups are for a particular user by running "l_getidentity -d <uid>" to compare what "id" returns on the client.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Cheers, Andreas<o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<br class="">
<br class="">
<o:p class=""></o:p></div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
However, the getstripe command will only continue to work for about 10 minutes, then it goes back to the permission denied errors.<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
It only happens with a selection of files or directories, so we were thinking it might be connected to a particular OSS or MDT, but not sure what to look for.<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I am not the Lustre admin, so please forgive incomplete information. If folks can request specific command output, preferably from user space, that would accelerate my ability to answer questions. If something needs to get run while logged into a particular
Lustre component (MDT, OSS, etc.), please do not hesitate to assume that I don’t know that.<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
We’re running Lustre 2.10.7 provided by DDN on CentOS 7.4. All help appreciated, thanks!<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Chris<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
-- <span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Christopher H. Chang, Ph.D.<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Computational Scientist<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
National Renewable Energy Laboratory<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
15013 Denver West Pkwy., MS ESIF301<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Golden, CO 80401<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 9pt; font-family: Helvetica;" class="">_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
</span><a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica;" class="">lustre-discuss@lists.lustre.org</span></a><span style="font-size: 9pt; font-family: Helvetica;" class=""><br class="">
</span><a href="https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=02%7C01%7CChristopher.Chang%40nrel.gov%7C82c57c8d48b841f873f008d81bb892be%7Ca0f29d7e28cd4f5484427885aee7c080%7C0%7C0%7C637289826842344348&sdata=3lmXyp1ujhSxyHY4YFacQ4ZEFrKXJrZdIzKSMd%2BLnrc%3D&reserved=0" style="color: blue; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica;" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</span></a><o:p class=""></o:p></div>
</div>
</blockquote>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Cheers, Andreas<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">--<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Andreas Dilger<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Principal Lustre Architect<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Whamcloud</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>