<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Trebuchet MS";
panose-1:2 11 6 3 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1350639719;
mso-list-type:hybrid;
mso-list-template-ids:-1908899672 -526240512 69140505 69140507 69140495 69140505 69140507 69140495 69140505 69140507;}
@list l0:level1
{mso-level-text:%1-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1
{mso-list-id:1490291866;
mso-list-template-ids:831038554;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="TR" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><b><u><span style="font-family:"Trebuchet MS",sans-serif;color:black">UNCLASSIFIED</span></u></b><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you for sharing your experience.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I was thinking that DoM is built in feature and it can be enabled/disabled online for a certain directories. What do you mean by reformat to converting to DoM (or away from it). I think just Metadata target size is important.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I also thought creating flash OST on metadata server. But I was not sure what to install on metadata server for this purpose. Can Metadata server be an OSS server at the same time? If it is possible I would prefer flash OST on Metadata
server instead of DoM. Because Our metadata target size is small, it seems I have to do risky operations to expand size.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">imho, because of the less RPC traffic DoM shows more performance than flash OST. Am I right?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best Regards;<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><a name="_____replyseparator"></a><b><span lang="EN-US" style="mso-fareast-language:TR">From:</span></b><span lang="EN-US" style="mso-fareast-language:TR"> Marion Hakanson <hakansom@ohsu.edu>
<br>
<b>Sent:</b> Thursday, February 17, 2022 8:20 PM<br>
<b>To:</b> Taner KARAGÖL <karagol@aselsan.com.tr><br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] How to speed up Lustre<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">We started with DoM on our new Lustre system a couple years ago.<span style="mso-fareast-language:TR"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> - Converting to DoM (or away from it) is a full-reformat operation.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> - DoM uses a fixed amount of metadata space (64k minimum for us) for every file, even those smaller than 64k.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Basically, DoM uses a lot of flash metadata space, more than we planned for, and more than we could afford.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We ended up switching to a PFL arrangement, where the first 64k lives on flash OST's (mounted on our metadata servers), and the remainder of larger files lives on HDD OST's. This is working very well for our small-file workloads, and uses
less flash space than the DoM configuration did.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Since you don't already have DoM in effect, it may be possible that you could add flash OST's, configure a PFL, and then use "lfs migrate" to re-layout existing files into the new OST's. Your mileage may vary, so be safe!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Marion<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt">On Feb 14, 2022, at 03:32, Taner KARAGÖL via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>> wrote:<o:p></o:p></p>
</blockquote>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"> <span style="mso-fareast-language:TR"><o:p></o:p></span></p>
<p class="MsoNormal"><b><u><span style="font-family:"Trebuchet MS",sans-serif;color:black">UNCLASSIFIED</span></u></b><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Hi Everybody;</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">We have a performance problem with small files on our HPC system (120 compute nodes). Our all OSS targets are classic spinning HDDs. To speed up, I want to configure Data on Metadata. Our metadata target has SDD disks.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Underlying file systems are ZFS (for OSS and Meta)</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Lustre version: 2.12.5</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">ZFS version: .0.7.13</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Our Lustre file system size is 720TB (2 OSS servers, 1 enclosure with 6 zpools), Metadata file system size is 2.1TB(1 enclosure and 1 metadata target).</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">What is your opinions to speed up this setup? I want to configure DoM but I am concerning about Metadata size. My questions:</span><o:p></o:p></p>
<ol style="margin-top:0cm" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo3"><span lang="EN-US">How can I increase Medatadata size? Metadata enclosure has a empty slots. Is there a way to increase size online/offline?</span><o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo3"><span lang="EN-US">Is it possible to migrate big files from DoM to OSS targets completely? Off course online migration. (So I think I can free Metadata for new small files).</span><o:p></o:p></li></ol>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Best Regards;</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Taner</span><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="mso-fareast-language:TR">
<hr size="2" width="100%" align="center">
</span></div>
<p class="MsoNormal"><strong><span style="font-family:"Calibri",sans-serif;color:red;mso-fareast-language:TR">Dikkat:</span></strong><span style="mso-fareast-language:TR">
<o:p></o:p></span></p>
<p>Bu elektronik posta mesaji kisisel ve ozeldir. Eger size gonderilmediyse lutfen gondericiyi bilgilendirip mesaji siliniz. Firmamiza gelen ve giden mesajlar virus taramasindan gecirilmekte, guvenlik nedeni ile kontrol edilerek saklanmaktadir. Mesajdaki gorusler
ve bakis acisi gondericiye ait olup Aselsan A.S. resmi gorusu olmak zorunda degildir.<o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center"><strong><span style="font-family:"Calibri",sans-serif;color:red;mso-fareast-language:TR">
<hr size="2" width="100%" align="center">
</span></strong></div>
<p class="MsoNormal"><strong><span style="font-family:"Calibri",sans-serif;color:red;mso-fareast-language:TR">Attention:
</span></strong><o:p></o:p></p>
<p>This e-mail message is privileged and confidential. If you are not the intended recipient please delete the message and notify the sender. E-mails to and from the company are monitored for operational reasons and in accordance with lawful business practices.
Any views or opinions presented are solely those of the author and do not necessarily represent the views of the company.<o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="mso-fareast-language:TR">
<hr size="2" width="100%" align="center">
</span></div>
<p> <o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:TR">_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a href="https://urldefense.com/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!Mi0JBg!bW2FnSTRNdX7DpkjIiMayeexmYJ3D5Xt7wtneny2zgGi1ZXPcy7QMRlM3mno-HWR$">https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!Mi0JBg!bW2FnSTRNdX7DpkjIiMayeexmYJ3D5Xt7wtneny2zgGi1ZXPcy7QMRlM3mno-HWR$</a>
<o:p></o:p></span></p>
</div>
</blockquote>
</div>
<P>
<HR>
<STRONG><FONT color=#ff0000>Dikkat:</FONT></STRONG></P>
<P></P>
<P>Bu elektronik posta mesaji kisisel ve ozeldir. Eger size gonderilmediyse
lutfen gondericiyi bilgilendirip mesaji siliniz. Firmamiza gelen
ve giden mesajlar virus taramasindan gecirilmekte, guvenlik nedeni ile
kontrol edilerek saklanmaktadir. Mesajdaki gorusler ve bakis acisi gondericiye
ait olup Aselsan A.S. resmi gorusu olmak zorunda degildir.</P><FONT
color=#ff0000><STRONG>
<HR>
</STRONG></FONT><STRONG><FONT color=#ff0000>Attention: </FONT></STRONG>
<P>This e-mail message is privileged and confidential. If you are not the
intended recipient please delete the message and notify the sender. E-mails to
and from the company are monitored for operational reasons and in accordance
with lawful business practices. Any views or opinions presented are solely those
of the author and do not necessarily represent the views of the company.</P>
<HR>
<P> </P>
</body>
</html>