<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Simon is exactly correct. This is expected behavior for files opened with O_APPEND, at least until LU-12738 is implemented. Since O_APPEND writes are (by definition) entirely serialized, having multiple stripes on such files is mostly useless and just adds
overhead.
<div class=""><br class="">
</div>
<div class="">Feel free to read <a href="https://jira.whamcloud.com/browse/LU-9341" class="">https://jira.whamcloud.com/browse/LU-9341</a> for the very lengthy saga on the history of this behavior.</div>
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Apr 29, 2024, at 10:42, Simon Guilbault <<a href="mailto:simon.guilbault@calculquebec.ca" class="">simon.guilbault@calculquebec.ca</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">This is the expected behaviour. In the original implementation of PFL, when a file was open in append mode, the lock from 0 to EOF was initializing all stripes of the PFL file. We have a PFL layout on our system with 1 stripe up to 1
GB, then it increased to 4 and then 32 stripes when the file was getting very large. This was a problem with software that was creating 4kb log files (like slurm.out) because they were creating files with > 32 stripes because of the append mode. This was patched
a few releases ago, that behaviour can be changed, but I would recommend keeping 1 stripe for files that are using append mode.<br class="">
<br class="">
From the manual:<br class="">
O_APPEND mode. When files are opened for append, they instantiate all uninitialized components expressed in the layout. Typically, log files are opened for append, and complex layouts can be inefficient.<br class="">
Note<br class="">
The mdd.*.append_stripe_count and mdd.*.append_pool options can be used to specify special default striping for files created with O_APPEND.<br class="">
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Apr 29, 2024 at 11:21 AM Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="msg-6154978525767147563">
<div lang="EN-US" style="overflow-wrap: break-word;" class="">
<div class="m_7725801462833266643WordSection1">
<p class="MsoNormal"><span style="font-size:11pt" class="">Wow, I would say that is definitely not expected. I can recreate this on both of our LFS’s. One is community lustre 2.14, the other is a DDN Exascalar. Shown below is our community lustre but we
also have a 3-segment PFL on our Exascalar and the behavor is the same there. <u class="">
</u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""><u class=""></u> <u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class="">$ echo > aaa<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class="">$ echo >> bbb<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class="">$ lfs getstripe aaa bbb<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class="">aaa<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcm_layout_gen: 3<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcm_mirror_count: 1<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcm_entry_count: 3<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_id: 1<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_mirror_id: 0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_flags: init<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_extent.e_start: 0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_extent.e_end: 33554432<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_stripe_count: 1<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_stripe_size: 4194304<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_pattern: raid0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_layout_gen: 0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_stripe_offset: 6<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_objects:<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> - 0: { l_ost_idx: 6, l_fid: [0x100060000:0xace8112:0x0] }<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""><u class=""></u> <u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_id: 2<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_mirror_id: 0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_flags: 0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_extent.e_start: 33554432<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_extent.e_end: 10737418240<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_stripe_count: 4<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_stripe_size: 4194304<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_pattern: raid0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_layout_gen: 0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_stripe_offset: -1<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""><u class=""></u> <u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_id: 3<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_mirror_id: 0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_flags: 0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_extent.e_start: 10737418240<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lcme_extent.e_end: EOF<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_stripe_count: 8<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_stripe_size: 4194304<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_pattern: raid0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_layout_gen: 0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> lmm_stripe_offset: -1<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""><u class=""></u> <u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class="">bbb<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class="">lmm_stripe_count: 1<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class="">lmm_stripe_size: 2097152<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class="">lmm_pattern: raid0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class="">lmm_layout_gen: 0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class="">lmm_stripe_offset: 3<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> obdidx objid objid group<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""> 3 179773949 0xab721fd 0<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""><u class=""></u> <u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""><u class=""></u> <u class=""></u></span></p>
<div id="m_7725801462833266643mail-editor-reference-message-container" class="">
<div class="">
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in" class="">
<p class="MsoNormal" style="margin-bottom:12pt"><b class=""><span style="font-size: 12pt;" class="">From:
</span></b><span style="font-size: 12pt;" class="">lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank" class="">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of Otto, Frank via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Date: </b>Monday, April 29, 2024 at 8:33 AM<br class="">
<b class="">To: </b><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" class="">lustre-discuss@lists.lustre.org</a> <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject: </b>[EXTERNAL] [BULK] [lustre-discuss] Files created in append mode don't obey directory default stripe count<u class=""></u><u class=""></u></span></p>
</div>
<table border="1" cellspacing="0" cellpadding="0" align="left" style="border:1.5pt solid black" class="">
<tbody class="">
<tr class="">
<td width="100%" style="width:100%;border:none;background:rgb(255,235,156);padding:3.75pt" class="">
<p class="MsoNormal"><b class=""><span style="" class="">CAUTION:</span></b><span style="font-size: 12pt;" class="">
</span><span style="" class="">This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.</span><span style="font-size: 12pt;" class="">
</span><span style="font-size:12pt" class=""><u class=""></u><u class=""></u></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:12pt" class=""><br class="">
<br class="">
<u class=""></u><u class=""></u></span></p>
<div class="">
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">See subject. Is it a known issue? Is it expected? Easy to reproduce:<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""># lfs getstripe .<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">.<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">stripe_count: 4 stripe_size: 1048576 pattern: raid0 stripe_offset: -1<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""># echo > aaa<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""># echo >> bbb<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""># lfs getstripe .<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">.<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">stripe_count: 4 stripe_size: 1048576 pattern: raid0 stripe_offset: -1<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">./aaa<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">lmm_stripe_count: 4<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">lmm_stripe_size: 1048576<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">lmm_pattern: raid0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">lmm_layout_gen: 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">lmm_stripe_offset: 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""> obdidx objid objid group<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""> 0 2830 0xb0e 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""> 1 2894 0xb4e 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""> 2 2831 0xb0f 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""> 3 2895 0xb4f 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">./bbb<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">lmm_stripe_count: 1<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">lmm_stripe_size: 1048576<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">lmm_pattern: raid0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">lmm_layout_gen: 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">lmm_stripe_offset: 4<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""> obdidx objid objid group<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""> 4 2831 0xb0f 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""><br class="">
<br class="">
As you see, file "bbb" is created with stripe count 1 instead of 4.<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">Observed in Lustre 2.12.x and Lustre 2.15.4.<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">Thanks,<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class="">Frank<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size: 11pt;" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div id="m_7725801462833266643Signature" class="">
<div class="">
<p class="MsoNormal"><span style="font-size: 9pt; font-family: Consolas;" class="">-- <u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="MsoNormal"><span style="font-size:9pt;font-family:Consolas" class="">Dr. Frank Otto<br class="">
Senior Research Infrastructure Developer<br class="">
UCL Centre for Advanced Research Computing<br class="">
Tel: 020 7679 1506<u class=""></u><u class=""></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" class="">lustre-discuss@lists.lustre.org</a><br class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br class="">
</div>
</blockquote>
</div>
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>