<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Thanks for the info. It makes more sense to be an OST object - I was looking at wrong source where ’trusted.lma’ is extended for OST objects, which misled me to think this must be a MDT object.
<div class=""><br class="">
</div>
<div class="">It seems like the objects on the MDT were deleted by ‘rm’ but somehow the objects on the OST side were not destroyed. Looking at the entries of the object 136, it is a ZFS directory but its content seems to be wrong. In case the dataset is mangled
by ‘zfs send/recv’, it would be a good idea to use zdb to dump the OST in question directly.</div>
<div class=""><br class="">
</div>
<div class="">Andreas is right, having a snapshot will hold the objects in the pool.</div>
<div class=""><br class="">
</div>
<div class="">Jinshan</div>
<div class="">
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Sep 15, 2016, at 2:23 PM, Crowe, Tom <<a href="mailto:thcrowe@iu.edu" class="">thcrowe@iu.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="auto" class="">
<div class="">Hi <span style="background-color: rgba(255, 255, 255, 0);" class="">Jinshan,</span></div>
<div class=""><br class="">
</div>
<div class="">The examples in the first part of the thread are from one of our OST's. We had all previous files/dirs pinned" to the OST via setstripe; so there was a specific top level directory associated with this specific OST. After the recursive rm, we
noticed the ZFS OST, still showed space allocated, while lustre shows the directory as empty. Thus we dug deeper with zdb. </div>
<div class=""><br class="">
</div>
<div class="">So to summarize, after removing all files, lustre showed nothing on the "setstripe pinned" directory, where all files lived previous to the rm. But the ZFS based OST showed the "broken objects" and the space in the ZFS vdev was still allocated. </div>
<div class=""><br class="">
</div>
<div class="">It is worth noting, we use a ldiskfs based MDT, and all OST's are ZFS based. <br class="">
<br class="">
Thank you for your response. Please let me know if I can provide any additional data. </div>
<div class=""><br class="">
</div>
<div class="">-Tom</div>
<div class=""><br class="">
On Sep 15, 2016, at 16:55, Xiong, Jinshan <<a href="mailto:jinshan.xiong@intel.com" class="">jinshan.xiong@intel.com</a>> wrote:<br class="">
<br class="">
</div>
<blockquote type="cite" class="">
<div class="">Hi Tom,
<div class=""><br class="">
</div>
<div class="">Just to narrow down the problem, when you saw the space was not freed from zpool, were you seeing this from MDT or OST zpool?</div>
<div class=""><br class="">
</div>
<div class="">It seems that the objects you dumped were from MDT pool. The object 138 should belong to a Lustre file, and it has a spilled block attached.</div>
<div class=""><br class="">
</div>
<div class="">Jinshan</div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Sep 9, 2016, at 1:34 PM, Crowe, Tom <<a href="mailto:thcrowe@iu.edu" class="">thcrowe@iu.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif;" class="">
<div class="">
<div class="">Greetings All,</div>
<div class=""><br class="">
</div>
<div class="">I have come across a strange scenario using zfs 0.6.4.2-1 and Lustre 2.8.0. </div>
<div class=""><br class="">
</div>
<div class="">In a nutshell, when we delete items from lustre using rm, the files/dirs are seemingly removed, but the space is not freed on the underlying zfs dataset/zpool. We have unmounted the OST/dataset, exported the zpool, and even rebooted the server
altogether. The space is never freed. </div>
<div class=""><br class="">
</div>
<div class="">I have read through many of the issues logged about this on <a href="https://github.com/zfsonlinux" class="">https://github.com/zfsonlinux</a>; many folks have reclaimed the space once they unmount/remount, and/or export/import. As noted above,
this has no impact on our dataset/zpool.</div>
<div class=""><br class="">
</div>
<div class="">We currently have a zpool scrub running, and expect this to complete in the next few hours. </div>
<div class=""><br class="">
</div>
<div class="">We DO have zfs compression enabled, and we are using ZFS quota and reservations for the associated OST/dataset.</div>
<div class=""><br class="">
</div>
<div class="">We have copied the dataset to an entirely different zpool (zfs send/receive), and then mounted as native ZFS to poke around. In doing so, we located some of the “broken path” files that were of decent size (4GB) and went a head and removed them
with rm. These files were/are ones that we see with the ??? in the path output from zdb. After removing the files, the space was almost immediately freed from the dataset/zpool. </div>
<div class=""><br class="">
</div>
<div class="">Here is an example of a directory and a file from the zdb output.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Object lvl iblk dblk dsize lsize %full type</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 136 2 16K 16K 9.00K 32K 100.00 ZFS directory</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 144 bonus System attributes</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> dnode flags: USED_BYTES USERUSED_ACCOUNTED</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> dnode maxblkid: 1</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> path ???<object#136></font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> uid 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> gid 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> atime Wed Dec 31 19:00:00 1969</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> mtime Wed Dec 31 19:00:00 1969</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> ctime Wed Dec 31 19:00:00 1969</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> crtime Thu Mar 10 12:04:52 2016</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> gen 1020298</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> mode 40755</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> size 2</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> parent 1</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> links 1</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> pflags 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> rdev 0x0000000000000000</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Fat ZAP stats:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Pointer table:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 1024 elements</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zt_blk: 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zt_numblks: 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zt_shift: 10</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zt_blks_copied: 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zt_nextblk: 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> ZAP entries: 2</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Leaf blocks: 1</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Total blocks: 2</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zap_block_type: 0x8000000000000001</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zap_magic: 0x2f52ab2ab</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zap_salt: 0x3fdbcd9ab9</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Leafs with 2^n pointers:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 9: 1 *</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Blocks with n*5 entries:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 0: 1 *</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Blocks n/10 full:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 1: 1 *</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Entries with n chunks:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 3: 2 **</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Buckets with n entries:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 0: 510 ****************************************</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 1: 2 *</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 0 = 38711 (type: not specified)</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> feb93 = 281474976687710 (type: 15 (invalid))</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class="">
<div class=""> Object lvl iblk dblk dsize lsize %full type</div>
<div class=""> 138 1 16K 128K 128K 128K 100.00 ZFS plain file</div>
<div class=""> 220 bonus System attributes</div>
<div class=""> dnode flags: USED_BYTES USERUSED_ACCOUNTED</div>
<div class=""> dnode maxblkid: 0</div>
<div class=""> path ???<object#138></div>
<div class=""> uid 0</div>
<div class=""> gid 0</div>
<div class=""> atime Wed Dec 31 19:00:00 1969</div>
<div class=""> mtime Wed Dec 31 19:00:00 1969</div>
<div class=""> ctime Wed Dec 31 19:00:00 1969</div>
<div class=""> crtime Thu Mar 10 12:04:52 2016</div>
<div class=""> gen 1020298</div>
<div class=""> mode 100644</div>
<div class=""> size 8</div>
<div class=""> parent 0</div>
<div class=""> links 1</div>
<div class=""> pflags 0</div>
<div class=""> rdev 0x0000000000000000</div>
<div class=""> SA xattrs: 76 bytes, 1 entries</div>
<div class=""><br class="">
</div>
<div class=""> trusted.lma = \000\000\000\000\000\000\000\000\003\000\000\000\002\000\000\000\000\000\000\000\000\000\000\000</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Can anyone advise on some next steps troubleshooting, or previous experiences that are similar to ours? </div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Tom</div>
<div class=""><br class="">
</div>
</font></div>
</div>
</div>
</div>
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</body>
</html>