<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr" style="font-family: Aptos; font-size: 16px; color: rgb(33, 33, 33);">
> The application creates files with standard POSIX I/O (<code>open()</code> followed by appends and writes), nothing exotic or MPI-IO-related.</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
If I remember correctly, files opened in append mode are treated differently by the lustre client and don’t necessarily obey the stripping set on its parent directory. I don’t recall the details at the moment but perhaps that is contributing to the behavior
Santiago is experiencing? </div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="mail-editor-reference-message-container">
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"></div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="text-align: left; padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; font-family: Aptos; font-size: 12pt; color: black;">
<b>From: </b>lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Andreas Dilger via lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Date: </b>Sunday, December 7, 2025 at 8:05 PM<br>
<b>To: </b>Santiago Freire InCo <sfreire@fing.edu.uy><br>
<b>Cc: </b>lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[EXTERNAL] [BULK] Re: [lustre-discuss] Wrong striping when creating files inside a directory with explicit layout<br>
<br>
</div>
<table class="ms-outlook-mobile-reference-message skipProofing" align="left" cellspacing="0" style="border-width: 2px; border-style: solid; border-color: black;">
<tbody>
<tr>
<td class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; background-color: rgb(255, 235, 156); padding: 5px; width: 100%;">
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal;">
<span style="font-size: 10pt; color: rgb(0, 0, 0); line-height: normal;"><b>CAUTION:</b></span>
<span style="font-size: 10pt; line-height: normal;">This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.</span></div>
</td>
</tr>
</tbody>
</table>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
<br>
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing">For many small output files, you are always best off to use 1-stripe files. </div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing">These days, the best way to handle this is by using a default PFL layout which has the first component larger than the "small" file size (eg. 1GB) and then stripes the "large" input file across more
OSTs, like:</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing"> lfs setstripe -E 1G -c 1 -E 16G -c 4 -E eof -c 40 DIR</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing">If you want to ensure that one or more OSTs is not being used at all, you can set "osp.FSNAME-OSTxxxx.create_count=0" on the MDS. </div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">Cheers, Andreas</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<blockquote>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">On Dec 8, 2025, at 10:08, Santiago Freire - InCo <sfreire@fing.edu.uy> wrote:<br>
<br>
</div>
</blockquote>
<blockquote>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"></div>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">Thank you Andreas for your response.</p>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">The application is a rather simple Python script that processes a large input file and writes many smaller output files in parallel based on a given criteria. I checked the code and it does
not create files in a temporary directory nor renames them afterward, all output files are created directly in a single target directory. I also verified this with strace to make sure no unexpected rename operations were taking place.</p>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">All the OSTs are up and running. My current setup currently consists of one MDT and four OSTs. </p>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">The reason I am trying to enforce a specific set of OSTs is that I am evaluating the performance benefits of increasing the number of OSTs for applications that consume and generate massive
amounts of data. Because of this I need to experiment with different stripe configurations, using 1, 2, 3, or 4 OSTs. If you know a better way of achieving this purpose, I'd be grateful to hear your suggestions.</p>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">Thanks again for your help.</p>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">Santiago</p>
<div dir="ltr" class="moz-cite-prefix">On 12/7/25 19:42, Andreas Dilger wrote:</div>
<blockquote>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">Two reasons that I know about why this might happen:</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">- the application is creating the files in a different directory and then renaming them</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"> (for example, MinIO is rename crazy)</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">- one of the OSTs is not available at time of creation, and another one is used instead,</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"> since stripe_count is more important than specific OST selection</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">If you want to constrain files to a specific set of OSTs (eg. flash vs. disk) you should create an OST pool.</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">Might I ask what your specific goal is here? There may be another way to achieve it. </div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">Cheers, Andreas</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">On Dec 8, 2025, at 03:28, Santiago Freire - InCo via lustre-discuss
<a href="mailto:lustre-discuss@lists.lustre.org" class="moz-txt-link-rfc2396E"><lustre-discuss@lists.lustre.org></a> wrote:</div>
<blockquote>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">Hello everyone.</p>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">I am observing unexpected behavior regarding file striping in a directory that has an explicitly configured layout. I'm launching an application that writes
<b>new </b>files on a given directory, which is empty and I previously set its layout using
<code>lfs setstripe</code> with a fixed stripe count, stripe size, and a specific set of OSTs (command
<span style="font-family: monospace;">lfs setstripe -S 4M -c 3 -o 1,2,3 target_directory</span>). Running simple tools such as
<code>dd</code> inside that directory produces files with the correct stripe pattern, checked using
<span style="font-family: monospace;">lfs getstripe</span>.</p>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">However, when my application creates new files in that same directory,
<b>all</b> of those files end up with a different layout: some are created with a stripe count of 1 and almost always put in the OST with index 1, others are placed n OSTs that are not part of the directory’s configured offset/OST set (for example, striped
across OSTs 0, 1 and 2). This happens even though the files did not exist beforehand and the directory was empty with the correct striping applied. The application creates files with standard POSIX I/O (<code>open()</code> followed by appends and writes),
nothing exotic or MPI-IO-related.</p>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">Given that the directory layout is correct and tools like
<code>dd</code> follow it reliably, I am trying to understand under what circumstances Lustre would ignore the directory’s default layout when creating new regular files. I would appreciate any insight or guidance on what might explain this behaviour, and how
could I fix it.</p>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">I'm using Lustre 2.15.7, with RHEL 9.6 for the clients and RHEL 8.10 for the OSSs/MDS/MGS.</p>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">Thank you very much in advance.</p>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">Best regards,<br>
Santiago</p>
<p dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</p>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" class="moz-txt-link-abbreviated">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" class="moz-txt-link-freetext" originalsrc="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></div>
</blockquote>
</blockquote>
</blockquote>
</div>
</body>
</html>