<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:569463793;
mso-list-type:hybrid;
mso-list-template-ids:-1185262892 -1249871600 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Not unless you set a very complex layout, with a lot of components in it (which has various flaws of its own). Otherwise you’ll fairly quickly hit your final component for large files, and then you’re stuck. This limitation is the motivation
behind the work proposed in LU-10070.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Patrick<o:p></o:p></li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of "E.S. Rosenberg" <esr+lustre@mail.hebrew.edu><br>
<b>Date: </b>Tuesday, March 20, 2018 at 2:46 PM<br>
<b>To: </b>"Dilger, Andreas" <andreas.dilger@intel.com><br>
<b>Cc: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] Mixed size OST's<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><a name="_MailOriginalBody">Doesn't PFL also 'solve'/mitigate this issue in the sense that a file doesn't have to remain restricted to the OST(s) it started on?<o:p></o:p></a></p>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">(And as such balancing will even continue as files grow)<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">Regards,<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">Eli<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">On Fri, Mar 16, 2018 at 9:57 PM, Dilger, Andreas <</span><a href="mailto:andreas.dilger@intel.com" target="_blank"><span style="mso-bookmark:_MailOriginalBody">andreas.dilger@intel.com</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody">>
wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">On Mar 15, 2018, at 09:48, Steve Thompson <</span><a href="mailto:smt@vgersoft.com"><span style="mso-bookmark:_MailOriginalBody">smt@vgersoft.com</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody">>
wrote:<br>
><br>
> Lustre newbie here (1 month). Lustre 2.10.3, CentOS 7.4, ZFS 0.7.5. All networking is 10 GbE.<br>
><br>
> I am building a test Lustre filesystem. So far, I have two OSS's, each with 30 disks of 2 TB each, all in a single zpool per OSS. Everything works well, and was suprisingly easy to build. Thus, two OST's of 60 TB each. File types are comprised of home directories.
Clients number about 225 HPC systems (about 2400 cores).<br>
><br>
> In about a month, I will have a third OSS available, and about a month after that, a fourth. Each of these two systems has 48 disks of 4 TB each. I am looking for advice on how best to configure this. If I go with one OST per system (one zpool comprising
8 x 6 RAIDZ2 vdevs), I will have a lustre f/s comprised of two 60 TB OST's and two 192 TB OST's (minus RAIDZ2 overhead). This is obviously a big mismatch between OST sizes. I have not encountered any discussion of the effect of mixing disparate OST sizes.
I could instead format two 96 TB OST's on each system (two zpools of 4 x 6 RAIDZ2 vdevs), or three 64 TB OST's, and so on. More OST's means more striping possibilities, but less vdev's per zpool impacts ZFS performance negatively. More OST's per OSS does not
help with network bandwidth to the OSS. How would you go about this?<br>
<br>
This is a little bit tricky. Lustre itself can handle different OST sizes,<br>
as it will run in "QOS allocator" mode (essentially "Quantity of Space", the<br>
full "Quality of Service" was not implemented). This balances file allocation<br>
across OSTs based on percentage of free space, at the expense of performance<br>
being lower as the only the two new OSTs would be used for 192/252 ~= 75%<br>
of the files, since it isn't possible to *also* use all the OSTs evenly at the<br>
same time (assuming that network speed is your bottleneck, and not disk speed).<br>
<br>
For home directory usage this may not be a significant issue. This performance<br>
imbalance would balance out as the larger OSTs became more full, and would not<br>
be seen when files are striped across all OSTs.<br>
<br>
I also thought about creating 3x OSTs per new OSS, so they would all be about<br>
the same size and allocated equally. That means the new OSS nodes would see<br>
about 3x as much IO traffic as the old ones, especially for files striped over<br>
all OSTs. The drawback here is that the performance imbalance would stay<br>
forever, so in the long run I don't think this is as good as just having a<br>
single larger OST. This will also become less of a factor as more OSTs are<br>
added to the filesystem and/or you eventually upgrade the initial OSTs to<br>
have larger disks and/or more VDEVs.<br>
<br>
<br>
Cheers, Andreas<br>
--<br>
Andreas Dilger<br>
Lustre Principal Architect<br>
Intel Corporation<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><br>
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
</span><a href="mailto:lustre-discuss@lists.lustre.org"><span style="mso-bookmark:_MailOriginalBody">lustre-discuss@lists.lustre.org</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"><br>
</span><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank"><span style="mso-bookmark:_MailOriginalBody">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
</div>
</body>
</html>