<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>Brian,</div>
<div>Do you have zfs 'frozen' ? It can lock when you have zero bytes left and you can not do much with zfs after.</div>
<div>You will need to remove file on zfs itself or remove snapshot to unfreeze zfs, so do not wait until it filled completely.</div>
<div>To avoid deleting mdt objects after zfs locking, I create few zfs files with known names in extra zfs system on the same pool, and also do space reservation.</div>
<div><br>
</div>
<div>I had had large space consumed per inode on test system last summer, about 10KB per inode. </div>
<div>I formatted ssds with ashift=12. I thought zpool ashift=12 is better fit for 4K sector size SSD.</div>
<div><br>
</div>
<div>The issue was resolved when I set ashift=9 (512 sector emulated), and at the same time I set xattr=sa (it was not default then).</div>
<div><br>
</div>
<div>Right now I have 82M files taking 126 GB on production system (with thousands of snapshots), or 1.5 KB per "inode." With ashift=12 it was about 10KB IIRC. This can be reproduced with "createmany."</div>
<div><br>
</div>
<div>Here is the plot I got during tests with ashift=9.</div>
<div>You can see the bin with n=11 is heavy populated (2**11=2048). On similar plot for zpool formatted with ashift=12 the distribution starts at n=12.</div>
<div>That is a lot of objects with size <= 2048 will be created as 4K objects for zpool with ashift=12 leading to larger space consumed on lustre MDT per lustre file (OI objects, hidden attributes,...)</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div style="font-size: 13px;">zdb -M zpl</div>
<div style="font-size: 13px;"><br>
</div>
<div style="font-size: 13px;">
<p><font face="Andale Mono"> vdev 0 metaslabs 119 fragmentation 5%</font></p>
<p><font face="Andale Mono"> 9: 21553 *</font></p>
<p><font face="Andale Mono"> 10: 186445 ******</font></p>
<p><font face="Andale Mono"> 11: 1337626 ****************************************</font></p>
<p><font face="Andale Mono"> 12: 603282 *******************</font></p>
<p><font face="Andale Mono"> 13: 235246 ********</font></p>
<p><font face="Andale Mono"> 14: 104615 ****</font></p>
<p><font face="Andale Mono"> 15: 50365 **</font></p>
<p><font face="Andale Mono"> 16: 22656 *</font></p>
<p><font face="Andale Mono"> 17: 9997 *</font></p>
<p><font face="Andale Mono"> 18: 3482 *</font></p>
<p><font face="Andale Mono"> 19: 1198 *</font></p>
<p><font face="Andale Mono"> 20: 481 *</font></p>
</div>
</div>
<div><br>
</div>
<div>Alex.</div>
<div><br>
</div>
<br>
<div>
<div>On Jul 26, 2016, at 7:57 PM, Rick Wagner <<a href="mailto:rpwagner@sdsc.edu">rpwagner@sdsc.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div dir="auto" style="font-family: Helvetica; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<div></div>
<div>Hi Brian,</div>
<div><br>
On Jul 26, 2016, at 5:45 PM, Andrus, Brian Contractor <<a href="mailto:bdandrus@nps.edu" style="color: purple; text-decoration: underline;">bdandrus@nps.edu</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div class="WordSection1" style="page: WordSection1;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
All,<o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
Ok, I thought 100GB would be sufficient for an MDT.<o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
I have 2 MDTs as well, BUT…<o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
MDT0 is 100% full and now I cannot write anything to my lustre filesystem.<o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
The MDT is on a ZFS backing filesystem.<o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
So, what is the proper way to grow my MDT using ZFS? Do I need to shut the filesystem down completely? Can I just add a disk or space to the pool and Lustre will see it?<o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
Any advice or direction is appreciated.</div>
</div>
</blockquote>
<div><br>
</div>
<div>We just did this successfully on the two MDTs backing one of our Lustre file systems and everything happened at the ZFS layer. We added drives to the pool and Lustre immediately saw the additional capacity. Whether you take down the file system or do it
live is a question of your architecture, skills, and confidence. Having test file system is also worthwhile to go over the steps.</div>
<div><br>
</div>
--Rick<br>
<br>
<blockquote type="cite">
<div class="WordSection1" style="page: WordSection1;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
Brian Andrus<o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
ITACS/Research Computing<o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
Naval Postgraduate School<o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
Monterey, California<o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
voice: 831-656-6238<o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p> </o:p></div>
</div>
</blockquote>
<blockquote type="cite"><span>_______________________________________________</span><br>
<span>lustre-discuss mailing list</span><br>
<span><a href="mailto:lustre-discuss@lists.lustre.org" style="color: purple; text-decoration: underline;">lustre-discuss@lists.lustre.org</a></span><br>
<span><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="color: purple; text-decoration: underline;">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span><br>
</blockquote>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" style="color: purple; text-decoration: underline;">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="color: purple; text-decoration: underline;">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></div>
</blockquote>
</div>
<br>
</body>
</html>