<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I think you are correct. ‘zpool list’ shows raw space, ‘zfs list’ shows the space after reservation for parity, etc.. In a 10 disk raidz2 ~24% of the space is reserved for parity.<div class="">This website helps in calculating ZFS capacity. <a href="https://wintelguy.com/zfs-calc.pl" class="">https://wintelguy.com/zfs-calc.pl</a></div><div class=""><br class=""></div><div class="">-Raj<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Apr 6, 2021, at 4:56 PM, Laura Hild via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><span style="font-size: 12pt;" class="">> I am not sure about the discrepancy of 3T. Maybe that is due to some ZFS and/or Lustre overhead?</span></div><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><div style="font-size: 12pt;" class=""><div class=""><br class=""></div><div class="">Slop space?</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span><a href="https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#spa-slop-shift" class="">https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#spa-slop-shift</a></div><div class=""><br class=""></div>-Laura</div></div><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 1369.046875px;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">Od:</b><span class="Apple-converted-space"> </span>lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" class="">lustre-discuss-bounces@lists.lustre.org</a>> v imenu Mohr, Rick via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>><br class=""><b class="">Poslano:</b><span class="Apple-converted-space"> </span>torek, 06. april 2021 16:34<br class=""><b class="">Za:</b><span class="Apple-converted-space"> </span>Makia Minich <<a href="mailto:makia@systemfabricworks.com" class="">makia@systemfabricworks.com</a>>;<span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><span class="Apple-converted-space"> </span><<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>><br class=""><b class="">Zadeva:</b><span class="Apple-converted-space"> </span>Re: [lustre-discuss] [EXTERNAL] ZFS and OST Space Difference</font><div class=""> </div></div><div class="BodyFragment" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><font size="2" class=""><span style="font-size: 11pt;" class=""><div class="PlainText">Makia,<br class=""><br class="">The drive sizes are 7.6 TB which translates to about 6.9 TiB (which is the unit that zpool uses for "T"). So the zpool sizes as just 10 x 6.9T = 69T since zpool shows the total amount of disk space available to the pool. The usable space (which is what df is reporting) should be more like 0.8 x 69T = 55T. I am not sure about the discrepancy of 3T. Maybe that is due to some ZFS and/or Lustre overhead?<br class=""><br class="">--Rick<br class=""><br class="">On 4/6/21, 3:49 PM, "lustre-discuss on behalf of Makia Minich" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" class="">lustre-discuss-bounces@lists.lustre.org</a> on behalf of <a href="mailto:makia@systemfabricworks.com" class="">makia@systemfabricworks.com</a>> wrote:<br class=""><br class=""> I believe this was discussed a while ago, but I was unable to find clear answers, so I’ll re-ask in hopefully a slightly different way.<br class=""> On an OST, I have 30 drives, each at 7.6TB. I create 3 raidz2 zpools of 10 devices (ashift=12):<br class=""><br class=""> [root@lustre47b ~]# zpool list<br class=""> NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT<br class=""> oss55-0 69.9T 37.3M 69.9T - - 0% 0% 1.00x ONLINE -<br class=""> oss55-1 69.9T 37.3M 69.9T - - 0% 0% 1.00x ONLINE -<br class=""> oss55-2 69.9T 37.4M 69.9T - - 0% 0% 1.00x ONLINE -<br class=""> [root@lustre47b ~]#<br class=""><br class=""><br class=""> Running a mkfs.lustre against these (and the lustre mount) and I see:<br class=""><br class=""> [root@lustre47b ~]# df -h | grep ost<br class=""> oss55-0/ost165 52T 27M 52T 1% /lustre/ost165<br class=""> oss55-1/ost166 52T 27M 52T 1% /lustre/ost166<br class=""> oss55-2/ost167 52T 27M 52T 1% /lustre/ost167<br class=""> [root@lustre47b ~]#<br class=""><br class=""><br class=""> Basically, we’re seeing a pretty dramatic loss in capacity (156TB vs 209.7TB, so a loss of about 50TB). Is there any insight on where this capacity is disappearing to? If there some mkfs.lustre or zpool option I missed in creating this? Is something just reporting slightly off and that space really is there?<br class=""><br class=""> Thanks.<br class=""><br class=""> —<br class=""><br class=""><br class=""> Makia Minich<br class=""><br class=""> Chief Architect<br class=""><br class=""> System Fabric Works<br class=""> "Fabric Computing that Works”<br class=""><br class=""> "Oh, I don't know. I think everything is just as it should be, y'know?”<br class=""> - Frank Fairfield<br class=""><br class=""><br class=""><br class=""><br class=""><br class=""><br class=""><br class="">_______________________________________________<br class="">lustre-discuss mailing list<br class=""><a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class=""><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwIGaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=897kjkV-MEeU1IVizIfc5Q&m=habzcIRCKUXYLTbJVvgv2fPgmEuBnVtUdsgTfIsAHZY&s=M7RWFzL5Xm7uDovhMY_cI9Hvk-jWavZyfLWjpMSAs1E&e=" class="">https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwIGaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=897kjkV-MEeU1IVizIfc5Q&m=habzcIRCKUXYLTbJVvgv2fPgmEuBnVtUdsgTfIsAHZY&s=M7RWFzL5Xm7uDovhMY_cI9Hvk-jWavZyfLWjpMSAs1E&e=</a><span class="Apple-converted-space"> </span><br class=""></div></span></font></div><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">lustre-discuss mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="mailto:lustre-discuss@lists.lustre.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">lustre-discuss@lists.lustre.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></div></blockquote></div><br class=""></div></body></html>