<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>Greetings All,</div>
<div><br>
</div>
<div>I have come across a strange scenario using zfs 0.6.4.2-1 and Lustre 2.8.0. </div>
<div><br>
</div>
<div>In a nutshell, when we delete items from lustre using rm, the files/dirs are seemingly removed, but the space is not freed on the underlying zfs dataset/zpool. We have unmounted the OST/dataset, exported the zpool, and even rebooted the server altogether.
The space is never freed. </div>
<div><br>
</div>
<div>I have read through many of the issues logged about this on <a href="https://github.com/zfsonlinux">https://github.com/zfsonlinux</a>; many folks have reclaimed the space once they unmount/remount, and/or export/import. As noted above, this has no impact
on our dataset/zpool.</div>
<div><br>
</div>
<div>We currently have a zpool scrub running, and expect this to complete in the next few hours. </div>
<div><br>
</div>
<div>We DO have zfs compression enabled, and we are using ZFS quota and reservations for the associated OST/dataset.</div>
<div><br>
</div>
<div>We have copied the dataset to an entirely different zpool (zfs send/receive), and then mounted as native ZFS to poke around. In doing so, we located some of the “broken path” files that were of decent size (4GB) and went a head and removed them with rm.
These files were/are ones that we see with the ??? in the path output from zdb. After removing the files, the space was almost immediately freed from the dataset/zpool. </div>
<div><br>
</div>
<div>Here is an example of a directory and a file from the zdb output.</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> Object lvl iblk dblk dsize lsize %full type</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> 136 2 16K 16K 9.00K 32K 100.00 ZFS directory</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> 144 bonus System attributes</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> dnode flags: USED_BYTES USERUSED_ACCOUNTED</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> dnode maxblkid: 1</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> path ???<object#136></font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> uid 0</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> gid 0</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> atime Wed Dec 31 19:00:00 1969</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> mtime Wed Dec 31 19:00:00 1969</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> ctime Wed Dec 31 19:00:00 1969</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> crtime Thu Mar 10 12:04:52 2016</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> gen 1020298</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> mode 40755</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> size 2</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> parent 1</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> links 1</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> pflags 0</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> rdev 0x0000000000000000</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> Fat ZAP stats:</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> Pointer table:</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> 1024 elements</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> zt_blk: 0</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> zt_numblks: 0</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> zt_shift: 10</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> zt_blks_copied: 0</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> zt_nextblk: 0</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> ZAP entries: 2</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> Leaf blocks: 1</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> Total blocks: 2</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> zap_block_type: 0x8000000000000001</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> zap_magic: 0x2f52ab2ab</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> zap_salt: 0x3fdbcd9ab9</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> Leafs with 2^n pointers:</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> 9: 1 *</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> Blocks with n*5 entries:</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> 0: 1 *</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> Blocks n/10 full:</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> 1: 1 *</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> Entries with n chunks:</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> 3: 2 **</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> Buckets with n entries:</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> 0: 510 ****************************************</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> 1: 2 *</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"><br>
</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> 0 = 38711 (type: not specified)</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"> feb93 = 281474976687710 (type: 15 (invalid))</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif"><br>
</font></div>
<div style="font-family: -webkit-standard;"><font face="Calibri,sans-serif">
<div> Object lvl iblk dblk dsize lsize %full type</div>
<div> 138 1 16K 128K 128K 128K 100.00 ZFS plain file</div>
<div> 220 bonus System attributes</div>
<div> dnode flags: USED_BYTES USERUSED_ACCOUNTED</div>
<div> dnode maxblkid: 0</div>
<div> path ???<object#138></div>
<div> uid 0</div>
<div> gid 0</div>
<div> atime Wed Dec 31 19:00:00 1969</div>
<div> mtime Wed Dec 31 19:00:00 1969</div>
<div> ctime Wed Dec 31 19:00:00 1969</div>
<div> crtime Thu Mar 10 12:04:52 2016</div>
<div> gen 1020298</div>
<div> mode 100644</div>
<div> size 8</div>
<div> parent 0</div>
<div> links 1</div>
<div> pflags 0</div>
<div> rdev 0x0000000000000000</div>
<div> SA xattrs: 76 bytes, 1 entries</div>
<div><br>
</div>
<div> trusted.lma = \000\000\000\000\000\000\000\000\003\000\000\000\002\000\000\000\000\000\000\000\000\000\000\000</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Can anyone advise on some next steps troubleshooting, or previous experiences that are similar to ours? </div>
<div><br>
</div>
<div>Thanks,</div>
<div>Tom</div>
<div><br>
</div>
</font></div>
</div>
</div>
</body>
</html>