[lustre-discuss] Wrong striping when creating files inside a directory with explicit layout

[Andreas Dilger]

For many small output files, you are always best off to use 1-stripe files.

These days, the best way to handle this is by using a default PFL layout which has the first component larger than the "small" file size (eg. 1GB) and then stripes the "large" input file across more OSTs, like:

    lfs setstripe -E 1G -c 1 -E 16G -c 4 -E eof -c 40 DIR

If you want to ensure that one or more OSTs is not being used at all, you can set "osp.FSNAME-OSTxxxx.create_count=0" on the MDS.

Cheers, Andreas

On Dec 8, 2025, at 10:08, Santiago Freire - InCo <sfreire at fing.edu.uy> wrote:



Thank you Andreas for your response.

The application is a rather simple Python script that processes a large input file and writes many smaller output files in parallel based on a given criteria. I checked the code and it does not create files in a temporary directory nor renames them afterward, all output files are created directly in a single target directory. I also verified this with strace to make sure no unexpected rename operations were taking place.

All the OSTs are up and running. My current setup currently consists of one MDT and four OSTs.

The reason I am trying to enforce a specific set of OSTs is that I am evaluating the performance benefits of increasing the number of OSTs for applications that consume and generate massive amounts of data. Because of this I need to experiment with different stripe configurations, using 1, 2, 3, or 4 OSTs. If you know a better way of achieving this purpose, I'd be grateful to hear your suggestions.

Thanks again for your help.

Santiago

On 12/7/25 19:42, Andreas Dilger wrote:
Two reasons that I know about why this might happen:
- the application is creating the files in a different directory and then renaming them
  (for example, MinIO is rename crazy)
- one of the OSTs is not available at time of creation, and another one is used instead,
  since stripe_count is more important than specific OST selection

If you want to constrain files to a specific set of OSTs (eg. flash vs. disk) you should create an OST pool.

Might I ask what your specific goal is here? There may be another way to achieve it.

Cheers, Andreas

On Dec 8, 2025, at 03:28, Santiago Freire - InCo via lustre-discuss <lustre-discuss at lists.lustre.org><mailto:lustre-discuss at lists.lustre.org> wrote:

Hello everyone.

I am observing unexpected behavior regarding file striping in a directory that has an explicitly configured layout. I'm launching an application that writes new files on a given directory, which is empty and I previously set its layout using lfs setstripe with a fixed stripe count, stripe size, and a specific set of OSTs (command lfs setstripe -S 4M -c 3 -o 1,2,3 target_directory). Running simple tools such as dd inside that directory produces files with the correct stripe pattern, checked using lfs getstripe.

However, when my application creates new files in that same directory, all of those files end up with a different layout: some are created with a stripe count of 1 and almost always put in the OST with index 1, others are placed n OSTs that are not part of the directory’s configured offset/OST set (for example, striped across OSTs 0, 1 and 2). This happens even though the files did not exist beforehand and the directory was empty with the correct striping applied. The application creates files with standard POSIX I/O (open() followed by appends and writes), nothing exotic or MPI-IO-related.

Given that the directory layout is correct and tools like dd follow it reliably, I am trying to understand under what circumstances Lustre would ignore the directory’s default layout when creating new regular files. I would appreciate any insight or guidance on what might explain this behaviour, and how could I fix it.

I'm using Lustre 2.15.7, with RHEL 9.6 for the clients and RHEL 8.10 for the OSSs/MDS/MGS.

Thank you very much in advance.

Best regards,
Santiago


_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20251208/d076b00b/attachment.htm>