<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:334259988;
mso-list-template-ids:1457443532;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1
{mso-list-id:992953228;
mso-list-type:hybrid;
mso-list-template-ids:519372290 -1254724298 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:?;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:?;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:?;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:?;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:?;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:black">Hi Mark,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">I just verified the following commands on my 2.11 system. It works.<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D">[root@centos7-3 tests]# ../utils/lfs setstripe -N2 /mnt/lustre/dir</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D">[root@centos7-3 tests]# touch /mnt/lustre/dir/file</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D">[root@centos7-3 tests]# ../utils/lfs getstripe /mnt/lustre/dir/file </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D">/mnt/lustre/dir/file</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lcm_layout_gen: 2</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lcm_mirror_count: 2</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lcm_entry_count: 2</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lcme_id: 65537</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lcme_mirror_id: 1</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lcme_flags: init</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lcme_extent.e_start: 0</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lcme_extent.e_end: EOF</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lmm_stripe_count: 1</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lmm_stripe_size: 1048576</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lmm_pattern: raid0</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lmm_layout_gen: 0</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lmm_stripe_offset: 1</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lmm_objects:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x6:0x0] }</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lcme_id: 131074</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lcme_mirror_id: 2</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lcme_flags: init</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lcme_extent.e_start: 0</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lcme_extent.e_end: EOF</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lmm_stripe_count: 1</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lmm_stripe_size: 1048576</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lmm_pattern: raid0</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lmm_layout_gen: 0</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lmm_stripe_offset: 0</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> lmm_objects:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:12.0pt;font-family:Courier;color:#4D2F2D"> - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x6:0x0] }</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:#DFDBC3"><span style="font-size:13.5pt;font-family:Courier;color:#4D2F2D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">B.R.,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Emoly<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Andreas Dilger <adilger@whamcloud.com><br>
<b>Date: </b>Monday, June 25, 2018 at 3:59 PM<br>
<b>To: </b>Emoly Liu <emoly@whamcloud.com><br>
<b>Subject: </b>Fwd: [lustre-discuss] Lustre 2.11 File Level Replication<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">Begin forwarded message:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><b>From:</b> Patrick Farrell <<a href="mailto:paf@cray.com">paf@cray.com</a>><br>
<b>Date:</b> June 22, 2018 at 21:54:26 GMT+2<br>
<b>To:</b> Mark Roper <<a href="mailto:markroper@gmail.com">markroper@gmail.com</a>><br>
<b>Cc:</b> "<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>><br>
<b>Subject:</b> <b>Re: [lustre-discuss] Lustre 2.11 File Level Replication</b><o:p></o:p></p>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Mark,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Hmm.<o:p></o:p></p>
<p class="MsoNormal">I’m adding the list back on here, because that *<b>seems</b>* like it’s wrong. Don’t have time to check right now, but I’m curious if others can weigh in.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l1 level1 lfo3">Patrick<o:p></o:p></li></ul>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Mark Roper <<a href="mailto:markroper@gmail.com">markroper@gmail.com</a>><br>
<b>Date: </b>Friday, June 22, 2018 at 2:29 PM<br>
<b>To: </b>Patrick Farrell <<a href="mailto:paf@cray.com">paf@cray.com</a>><br>
<b>Subject: </b>Re: [lustre-discuss] Lustre 2.11 File Level Replication</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks Patrick! It looks like you can't set default mirroring with setstripe. I appreciate the lead and the response!<o:p></o:p></p>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jun 21, 2018 at 10:09 AM Patrick Farrell <<a href="mailto:paf@cray.com">paf@cray.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
Mark,<br>
<br>
I haven’t played specifically with FLR and inheritance/templates, but if you want to set a default layout on a directory, you’ll want to look at lfs setstripe. Mirror extend is specifically for modifying individual, existing files.<br>
<br>
- Patrick<o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="68%" align="center">
</div>
<div id="m_-185905444424043489divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of Mark Roper <<a href="mailto:markroper@gmail.com" target="_blank">markroper@gmail.com</a>><br>
<b>Sent:</b> Thursday, June 21, 2018 8:36:43 AM<br>
<b>To:</b> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> [lustre-discuss] Lustre 2.11 File Level Replication</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#212121">Hi Lustre Users,</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#212121"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#212121">I have set up a Lustre 2.11 cluster with multiple OSS's in order to experiment with the File Level Replication feature in Lustre 2.11, which I'm excited about since it raises the bar on availability
for cloud deployment of Lustre. I cannot seem to get files created within a directory to inherit the mirror settings from the parent directory, which I thought might be possible given the documentation at <a href="http://wiki.lustre.org/File_Level_Replication_High_Level_Design" target="_blank">http://wiki.lustre.org/File_Level_Replication_High_Level_Design</a> :</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#212121"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#212121">'</span><span style="font-size:10.5pt;font-family:"Courier New";color:#212121;background:#F8F9FA">lfs mirror extend [--no-verify] <-N[mirror_count]> [other setstripe options|-f <victim_file>]
<file_name><br>
</span><span style="font-size:10.5pt;color:#222222">This command will append a replica indicated by setstripe options or just take the layout from existing file victim_file into the file file_name. The file_name must be an existing file but it can be a mirrored
or normal file. This command will create new volatile file with any optional setstripe options that are specified, or using the defaults inherited from the parent directory or filesystem.</span><span style="font-size:10.0pt;color:#212121">'</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#212121"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#212121">I was hoping to be able to make all files written to particular directories or within a filesystem inherit a mirror count transparently to the client writing or reading files. Does anyone know
if this is possible?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#212121"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#212121">Cheers!</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#212121"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#212121">Mark Roper</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</body>
</html>