<div dir="ltr">This is the expected behaviour. In the original implementation of PFL, when a file was open in append mode, the lock from 0 to EOF was initializing all stripes of the PFL file. We have a PFL layout on our system with 1 stripe up to 1 GB, then it increased to 4 and then 32 stripes when the file was getting very large. This was a problem with software that was creating 4kb log files (like slurm.out) because they were creating files with > 32 stripes because of the append mode. This was patched a few releases ago, that behaviour can be changed, but I would recommend keeping 1 stripe for files that are using append mode.<br><br>From the manual:<br>O_APPEND mode. When files are opened for append, they instantiate all uninitialized components expressed in the layout. Typically, log files are opened for append, and complex layouts can be inefficient.<br>Note<br>The mdd.*.append_stripe_count and mdd.*.append_pool options can be used to specify special default striping for files created with O_APPEND.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 29, 2024 at 11:21 AM Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-6154978525767147563">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="m_7725801462833266643WordSection1">
<p class="MsoNormal"><span style="font-size:11pt">Wow, I would say that is definitely not expected. I can recreate this on both of our LFS’s. One is community lustre 2.14, the other is a DDN Exascalar. Shown below is our community lustre but we also have
a 3-segment PFL on our Exascalar and the behavor is the same there. <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">$ echo > aaa<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">$ echo >> bbb<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">$ lfs getstripe aaa bbb<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">aaa<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcm_layout_gen: 3<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcm_mirror_count: 1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcm_entry_count: 3<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_id: 1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_mirror_id: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_flags: init<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_extent.e_start: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_extent.e_end: 33554432<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_stripe_count: 1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_stripe_size: 4194304<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_pattern: raid0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_layout_gen: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_stripe_offset: 6<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_objects:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> - 0: { l_ost_idx: 6, l_fid: [0x100060000:0xace8112:0x0] }<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_id: 2<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_mirror_id: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_flags: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_extent.e_start: 33554432<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_extent.e_end: 10737418240<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_stripe_count: 4<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_stripe_size: 4194304<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_pattern: raid0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_layout_gen: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_stripe_offset: -1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_id: 3<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_mirror_id: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_flags: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_extent.e_start: 10737418240<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lcme_extent.e_end: EOF<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_stripe_count: 8<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_stripe_size: 4194304<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_pattern: raid0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_layout_gen: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> lmm_stripe_offset: -1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">bbb<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">lmm_stripe_count: 1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">lmm_stripe_size: 2097152<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">lmm_pattern: raid0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">lmm_layout_gen: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">lmm_stripe_offset: 3<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> obdidx objid objid group<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> 3 179773949 0xab721fd 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<div id="m_7725801462833266643mail-editor-reference-message-container">
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal" style="margin-bottom:12pt"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of Otto, Frank via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
<b>Date: </b>Monday, April 29, 2024 at 8:33 AM<br>
<b>To: </b><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a> <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
<b>Subject: </b>[EXTERNAL] [BULK] [lustre-discuss] Files created in append mode don't obey directory default stripe count<u></u><u></u></span></p>
</div>
<table border="1" cellspacing="0" cellpadding="0" align="left" style="border:1.5pt solid black">
<tbody>
<tr>
<td width="100%" style="width:100%;border:none;background:rgb(255,235,156);padding:3.75pt">
<p class="MsoNormal">
<b><span style="color:black">CAUTION:</span></b><span style="font-size:12pt;color:black">
</span><span style="color:black">This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.</span><span style="font-size:12pt;color:black">
</span><span style="font-size:12pt"><u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:12pt"><br>
<br>
<u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">See subject. Is it a known issue? Is it expected? Easy to reproduce:<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"># lfs getstripe .<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">stripe_count: 4 stripe_size: 1048576 pattern: raid0 stripe_offset: -1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"># echo > aaa<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"># echo >> bbb<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"># lfs getstripe .<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">stripe_count: 4 stripe_size: 1048576 pattern: raid0 stripe_offset: -1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">./aaa<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">lmm_stripe_count: 4<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">lmm_stripe_size: 1048576<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">lmm_pattern: raid0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">lmm_layout_gen: 0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">lmm_stripe_offset: 0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"> obdidx objid objid group<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"> 0 2830 0xb0e 0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"> 1 2894 0xb4e 0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"> 2 2831 0xb0f 0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"> 3 2895 0xb4f 0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">./bbb<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">lmm_stripe_count: 1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">lmm_stripe_size: 1048576<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">lmm_pattern: raid0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">lmm_layout_gen: 0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">lmm_stripe_offset: 4<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"> obdidx objid objid group<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"> 4 2831 0xb0f 0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"><br>
<br>
As you see, file "bbb" is created with stripe count 1 instead of 4.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">Observed in Lustre 2.12.x and Lustre 2.15.4.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">Thanks,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black">Frank<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:black"><u></u> <u></u></span></p>
</div>
<div id="m_7725801462833266643Signature">
<div>
<p class="MsoNormal"><span style="font-size:9pt;font-family:Consolas;color:black">-- <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9pt;font-family:Consolas">Dr. Frank Otto<br>
Senior Research Infrastructure Developer<br>
UCL Centre for Advanced Research Computing<br>
Tel: 020 7679 1506<u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</div></blockquote></div>