<html>
<head>
<meta name="generator" content="Windows Mail 17.5.9600.20498">
<style type="text/css"><!--html { font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; }--></style><style data-externalstyle="true"><!--
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph {
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
}
p.MsoNormal, li.MsoNormal, div.MsoNormal {
margin:0in;
margin-bottom:.0001pt;
}
p.MsoListParagraphCxSpFirst, li.MsoListParagraphCxSpFirst, div.MsoListParagraphCxSpFirst,
p.MsoListParagraphCxSpMiddle, li.MsoListParagraphCxSpMiddle, div.MsoListParagraphCxSpMiddle,
p.MsoListParagraphCxSpLast, li.MsoListParagraphCxSpLast, div.MsoListParagraphCxSpLast {
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
line-height:115%;
}
--></style></head>
<body dir="ltr">
<div data-externalstyle="false" dir="ltr" style="font-family: 'Calibri', 'Segoe UI', 'Meiryo', 'Microsoft YaHei UI', 'Microsoft JhengHei UI', 'Malgun Gothic', 'sans-serif';font-size:12pt;"><div>Looking at some of our existing zfs filesystems, we have a couple with zfs mdts </div><div><br></div><div>One has 103M inodes and uses 152G of MDT space, another 12M and 19G. I\u2019d plan for less than that I guess as Mr. Dilger suggests. It all depends on your expected average file size and number of files for what will work.</div><div><br></div><div>We have run into some unpleasant surprises with zfs for the MDT, I believe mostly documented in bug reports, or at least hinted at.</div><div><br></div><div>A serious issue we have is performance of the zfs arc cache over time. This is something we didn\u2019t see in early testing, but with enough use it grinds things to a crawl. I believe this may be addressed in the newer version of ZFS, which we\u2019re hopefully awaiting.</div><div><br></div><div>Another thing we\u2019ve seen, which is mysterious to me is this it appears hat as the MDT begins to fill up file create rates go down. We don\u2019t really have a strong handle on this (not enough for a bug report I think), but we see this:</div><div><br></div><ol style="padding-top: 0px; padding-bottom: 0px; margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;"><li style='color: rgb(0, 0, 0); font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 16px;'><div>The aforementioned 104M inode / 152GB MDT system has 4 SAS drives raid10. On initial testing file creates were about 2500 to 3000 IOPs per second. Follow up testing in it\u2019s current state (about half full..) shows them at about 500 IOPs now, but with a few iterations of mdtest those IOPs plummet quickly to unbearable levels (like 30\u2026).</div></li><li style='color: rgb(0, 0, 0); font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 16px;'><div>We took a snapshot of the filesystem and sent it to the backup MDS, this time with the MDT built on 4 SAS drives in a raid0 - really not for performance so much as \u201cextra headroom\u201d if that makes any sense. Testing this the IOPs started higher, at maybe 800 or 1000 (this is from memory, I don\u2019t have my data in front of me). That initial faster speed could just be writing to 4 spindles I suppose, but surprising to me, the performance degraded at a slower rate. It took much longer to get painfully slow. It still got there. The performance didn\u2019t degrade at the same rate if that makes sense - the same number of writes on the smaller/slower mdt degraded the performance more quickly. My guess is that had something to do with the total space available. Who knows. I believe restarting lustre (and certainly rebooting) \u2018resets the clock\u2019 on the file create performance degradation.</div></li></ol><div style='color: rgb(0, 0, 0); font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 16px;'><br></div><div style='color: rgb(0, 0, 0); font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 16px;'>For that problem we\u2019re just going to try adding 4 SSD\u2019s, but it\u2019s an ugly problem. Also are once again hopeful new zfs version addresses it.</div><div style='color: rgb(0, 0, 0); font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 16px;'><br></div><div style='color: rgb(0, 0, 0); font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 16px;'>And finally, we\u2019ve got a real concern with snapshot backups of the MDT that my colleague posted about - the problem we see manifests in essentially a read-only recovered file system, so it\u2019s a concern and not quite terrifying.</div><div style='color: rgb(0, 0, 0); font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 16px;'><br></div><div style='color: rgb(0, 0, 0); font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 16px;'>All in all, the next lustre file system we bring up (in a couple weeks) we are very strongly considering going with ldiskfs for the MDT this time.</div><div style='color: rgb(0, 0, 0); font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 16px;'><br></div><div style='color: rgb(0, 0, 0); font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 16px;'>Scott</div><div style='color: rgb(0, 0, 0); font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 16px;'><br></div><div style='color: rgb(0, 0, 0); font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 16px;'><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div data-signatureblock="true"><br></div><div style="padding-top: 5px; border-top-color: rgb(229, 229, 229); border-top-width: 1px; border-top-style: solid;"><div><font face=" 'Calibri', 'Segoe UI', 'Meiryo', 'Microsoft YaHei UI', 'Microsoft JhengHei UI', 'Malgun Gothic', 'sans-serif'" style='line-height: 15pt; letter-spacing: 0.02em; font-family: "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 12pt;'><b>From:</b> <a href="mailto:kar@psc.edu" target="_parent">Anjana Kar</a><br><b>Sent:</b> \u200eTuesday\u200e, \u200eJune\u200e \u200e3\u200e, \u200e2014 \u200e7\u200e:\u200e38\u200e \u200ePM<br><b>To:</b> <a href="mailto:lustre-discuss@lists.lustre.org" target="_parent">lustre-discuss@lists.lustre.org</a></font></div></div><div><br></div><div dir=""><div id="readingPaneBodyContent">Is there a way to set the number of inodes for zfs MDT?<br><br>I've tried using --mkfsoptions="-N value" mentioned in lustre 2.0 <br>manual, but it<br>fails to accept it. We are mirroring 2 80GB SSDs for the MDT, but the <br>number of<br>inodes is getting set to 7 million, which is not enough for a 100TB <br>filesystem.<br><br>Thanks in advance.<br><br>-Anjana Kar<br> Pittsburgh Supercomputing Center<br> kar@psc.edu<br>_______________________________________________<br>Lustre-discuss mailing list<br>Lustre-discuss@lists.lustre.org<br>http://lists.lustre.org/mailman/listinfo/lustre-discuss<br></div></div></div>
</body>
</html>