<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:-webkit-standard;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Note that the "broken objects" may be Lustre internal metadata, such as the Object Index files, so deleting them may be bad for your filesystem without knowing what they are.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Not that I can say for sure your bug is fixed, but ZFS 0.6.4 is getting a bit long in the tooth (tagged April 8, 2015), and 0.6.5.x is the current maintenance version that is getting a
fair number of fixes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Any OST objects that were used by regular files are stored in O/0/d* on the OSTs.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><br>
The other important question to ask is if you have any snapshots on the filesystem? That would pin data blocks in the pool, and deleting the files would not release any space until after the snapshot(s) are removed.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri;color:black">Cheers, Andreas<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri;color:black">-- <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri;color:black">Andreas Dilger<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri;color:black">Lustre Principal Architect<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri;color:black">Intel High Performance Data Division</span><span style="font-size:11.0pt;font-family:Calibri"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:black">On 2016/09/15, 15:23, "Crowe, Tom" <<a href="mailto:thcrowe@iu.edu">thcrowe@iu.edu</a>> wrote:<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hi Jinshan,<o:p></o:p></p>
</div>
<div id="AppleMailSignature">
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div id="AppleMailSignature">
<p class="MsoNormal">The examples in the first part of the thread are from one of our OST's. We had all previous files/dirs pinned" to the OST via setstripe; so there was a specific top level directory associated with this specific OST. After the recursive
rm, we noticed the ZFS OST, still showed space allocated, while lustre shows the directory as empty. Thus we dug deeper with zdb. <o:p></o:p></p>
</div>
<div id="AppleMailSignature">
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div id="AppleMailSignature">
<p class="MsoNormal">So to summarize, after removing all files, lustre showed nothing on the "setstripe pinned" directory, where all files lived previous to the rm. But the ZFS based OST showed the "broken objects" and the space in the ZFS vdev was still allocated. <o:p></o:p></p>
</div>
<div id="AppleMailSignature">
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div id="AppleMailSignature">
<p class="MsoNormal">It is worth noting, we use a ldiskfs based MDT, and all OST's are ZFS based. <br>
<br>
Thank you for your response. Please let me know if I can provide any additional data. <o:p></o:p></p>
</div>
<div id="AppleMailSignature">
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div id="AppleMailSignature">
<p class="MsoNormal">-Tom<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
On Sep 15, 2016, at 16:55, Xiong, Jinshan <<a href="mailto:jinshan.xiong@intel.com">jinshan.xiong@intel.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Hi Tom, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Just to narrow down the problem, when you saw the space was not freed from zpool, were you seeing this from MDT or OST zpool?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">It seems that the objects you dumped were from MDT pool. The object 138 should belong to a Lustre file, and it has a spilled block attached.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Jinshan<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Sep 9, 2016, at 1:34 PM, Crowe, Tom <<a href="mailto:thcrowe@iu.edu">thcrowe@iu.edu</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri">Greetings All,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri">I have come across a strange scenario using zfs 0.6.4.2-1 and Lustre 2.8.0. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri">In a nutshell, when we delete items from lustre using rm, the files/dirs are seemingly removed, but the space is not freed on the underlying zfs dataset/zpool. We have unmounted the OST/dataset,
exported the zpool, and even rebooted the server altogether. The space is never freed. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri">I have read through many of the issues logged about this on <a href="https://github.com/zfsonlinux">https://github.com/zfsonlinux</a>; many folks have reclaimed the space once they unmount/remount,
and/or export/import. As noted above, this has no impact on our dataset/zpool.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri">We currently have a zpool scrub running, and expect this to complete in the next few hours. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri">We DO have zfs compression enabled, and we are using ZFS quota and reservations for the associated OST/dataset.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri">We have copied the dataset to an entirely different zpool (zfs send/receive), and then mounted as native ZFS to poke around. In doing so, we located some of the “broken path” files that
were of decent size (4GB) and went a head and removed them with rm. These files were/are ones that we see with the ??? in the path output from zdb. After removing the files, the space was almost immediately freed from the dataset/zpool. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri">Here is an example of a directory and a file from the zdb output.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> Object lvl iblk dblk dsize lsize %full type</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> 136 2 16K 16K 9.00K 32K 100.00 ZFS directory</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> 144 bonus System attributes</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> dnode flags: USED_BYTES USERUSED_ACCOUNTED</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> dnode maxblkid: 1</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> path ???<object#136></span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> uid 0</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> gid 0</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> atime Wed Dec 31 19:00:00 1969</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> mtime Wed Dec 31 19:00:00 1969</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> ctime Wed Dec 31 19:00:00 1969</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> crtime Thu Mar 10 12:04:52 2016</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> gen 1020298</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> mode 40755</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> size 2</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> parent 1</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> links 1</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> pflags 0</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> rdev 0x0000000000000000</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> Fat ZAP stats:</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> Pointer table:</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> 1024 elements</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> zt_blk: 0</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> zt_numblks: 0</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> zt_shift: 10</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> zt_blks_copied: 0</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> zt_nextblk: 0</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> ZAP entries: 2</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> Leaf blocks: 1</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> Total blocks: 2</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> zap_block_type: 0x8000000000000001</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> zap_magic: 0x2f52ab2ab</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> zap_salt: 0x3fdbcd9ab9</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> Leafs with 2^n pointers:</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> 9: 1 *</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> Blocks with n*5 entries:</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> 0: 1 *</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> Blocks n/10 full:</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> 1: 1 *</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> Entries with n chunks:</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> 3: 2 **</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> Buckets with n entries:</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> 0: 510 ****************************************</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> 1: 2 *</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> 0 = 38711 (type: not specified)</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> feb93 = 281474976687710 (type: 15 (invalid))</span><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"-webkit-standard","serif""><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> Object lvl iblk dblk dsize lsize %full type<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> 138 1 16K 128K 128K 128K 100.00 ZFS plain file<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> 220 bonus System attributes<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> dnode flags: USED_BYTES USERUSED_ACCOUNTED<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> dnode maxblkid: 0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> path ???<object#138><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> uid 0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> gid 0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> atime Wed Dec 31 19:00:00 1969<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> mtime Wed Dec 31 19:00:00 1969<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> ctime Wed Dec 31 19:00:00 1969<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> crtime Thu Mar 10 12:04:52 2016<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> gen 1020298<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> mode 100644<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> size 8<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> parent 0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> links 1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> pflags 0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> rdev 0x0000000000000000<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> SA xattrs: 76 bytes, 1 entries<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"> trusted.lma = \000\000\000\000\000\000\000\000\003\000\000\000\002\000\000\000\000\000\000\000\000\000\000\000<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri">Can anyone advise on some next steps troubleshooting, or previous experiences that are similar to ours? <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri">Thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri">Tom<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</blockquote>
</div>
</body>
</html>