<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Tom,
<div class=""><br class="">
</div>
<div class="">Just to narrow down the problem, when you saw the space was not freed from zpool, were you seeing this from MDT or OST zpool?</div>
<div class=""><br class="">
</div>
<div class="">It seems that the objects you dumped were from MDT pool. The object 138 should belong to a Lustre file, and it has a spilled block attached.</div>
<div class=""><br class="">
</div>
<div class="">Jinshan</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Sep 9, 2016, at 1:34 PM, Crowe, Tom <<a href="mailto:thcrowe@iu.edu" class="">thcrowe@iu.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif;" class="">
<div class="">
<div class="">Greetings All,</div>
<div class=""><br class="">
</div>
<div class="">I have come across a strange scenario using zfs 0.6.4.2-1 and Lustre 2.8.0. </div>
<div class=""><br class="">
</div>
<div class="">In a nutshell, when we delete items from lustre using rm, the files/dirs are seemingly removed, but the space is not freed on the underlying zfs dataset/zpool. We have unmounted the OST/dataset, exported the zpool, and even rebooted the server
altogether. The space is never freed. </div>
<div class=""><br class="">
</div>
<div class="">I have read through many of the issues logged about this on <a href="https://github.com/zfsonlinux" class="">https://github.com/zfsonlinux</a>; many folks have reclaimed the space once they unmount/remount, and/or export/import. As noted above,
this has no impact on our dataset/zpool.</div>
<div class=""><br class="">
</div>
<div class="">We currently have a zpool scrub running, and expect this to complete in the next few hours. </div>
<div class=""><br class="">
</div>
<div class="">We DO have zfs compression enabled, and we are using ZFS quota and reservations for the associated OST/dataset.</div>
<div class=""><br class="">
</div>
<div class="">We have copied the dataset to an entirely different zpool (zfs send/receive), and then mounted as native ZFS to poke around. In doing so, we located some of the “broken path” files that were of decent size (4GB) and went a head and removed them
with rm. These files were/are ones that we see with the ??? in the path output from zdb. After removing the files, the space was almost immediately freed from the dataset/zpool. </div>
<div class=""><br class="">
</div>
<div class="">Here is an example of a directory and a file from the zdb output.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Object lvl iblk dblk dsize lsize %full type</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 136 2 16K 16K 9.00K 32K 100.00 ZFS directory</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 144 bonus System attributes</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> dnode flags: USED_BYTES USERUSED_ACCOUNTED</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> dnode maxblkid: 1</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> path ???<object#136></font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> uid 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> gid 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> atime Wed Dec 31 19:00:00 1969</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> mtime Wed Dec 31 19:00:00 1969</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> ctime Wed Dec 31 19:00:00 1969</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> crtime Thu Mar 10 12:04:52 2016</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> gen 1020298</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> mode 40755</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> size 2</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> parent 1</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> links 1</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> pflags 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> rdev 0x0000000000000000</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Fat ZAP stats:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Pointer table:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 1024 elements</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zt_blk: 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zt_numblks: 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zt_shift: 10</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zt_blks_copied: 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zt_nextblk: 0</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> ZAP entries: 2</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Leaf blocks: 1</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Total blocks: 2</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zap_block_type: 0x8000000000000001</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zap_magic: 0x2f52ab2ab</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> zap_salt: 0x3fdbcd9ab9</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Leafs with 2^n pointers:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 9: 1 *</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Blocks with n*5 entries:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 0: 1 *</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Blocks n/10 full:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 1: 1 *</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Entries with n chunks:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 3: 2 **</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> Buckets with n entries:</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 0: 510 ****************************************</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 1: 2 *</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> 0 = 38711 (type: not specified)</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""> feb93 = 281474976687710 (type: 15 (invalid))</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class=""><br class="">
</font></div>
<div style="font-family: -webkit-standard;" class=""><font face="Calibri,sans-serif" class="">
<div class=""> Object lvl iblk dblk dsize lsize %full type</div>
<div class=""> 138 1 16K 128K 128K 128K 100.00 ZFS plain file</div>
<div class=""> 220 bonus System attributes</div>
<div class=""> dnode flags: USED_BYTES USERUSED_ACCOUNTED</div>
<div class=""> dnode maxblkid: 0</div>
<div class=""> path ???<object#138></div>
<div class=""> uid 0</div>
<div class=""> gid 0</div>
<div class=""> atime Wed Dec 31 19:00:00 1969</div>
<div class=""> mtime Wed Dec 31 19:00:00 1969</div>
<div class=""> ctime Wed Dec 31 19:00:00 1969</div>
<div class=""> crtime Thu Mar 10 12:04:52 2016</div>
<div class=""> gen 1020298</div>
<div class=""> mode 100644</div>
<div class=""> size 8</div>
<div class=""> parent 0</div>
<div class=""> links 1</div>
<div class=""> pflags 0</div>
<div class=""> rdev 0x0000000000000000</div>
<div class=""> SA xattrs: 76 bytes, 1 entries</div>
<div class=""><br class="">
</div>
<div class=""> trusted.lma = \000\000\000\000\000\000\000\000\003\000\000\000\002\000\000\000\000\000\000\000\000\000\000\000</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Can anyone advise on some next steps troubleshooting, or previous experiences that are similar to ours? </div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Tom</div>
<div class=""><br class="">
</div>
</font></div>
</div>
</div>
</div>
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>