<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Thomas,
<div class="">your analysis is as good as any possible. There should be at least an ioctl() call after the open() to create the objects before the pwrite64() call. You would need to discuss this with Cray, use a different MPI, or potentially "pre-create"
the file before MPI_File_open() so that O_LOV_DELAY_CREATE has no effect.</div>
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 30, 2024, at 08:40, Bertschinger, Thomas Andrew Hjorth via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Hello,<br class="">
<br class="">
We have an application that fails doing the following on one of our systems:<br class="">
<br class="">
...<br class="">
openat(AT_FDCWD, "mpi_test.out", O_WRONLY|O_CREAT|O_NOCTTY|FASYNC, 0611) = 4<br class="">
pwrite64(4, "\3\0\0\0", 4, 0) = -1 EBADF (Bad file descriptor)<br class="">
...<br class="">
<br class="">
It opens a file with O_LOV_DELAY_CREATE (or O_NOCTTY|FASYNC as strace interprets it), and then immediately tries to write to it.<br class="">
<br class="">
>From the comments above ll_file_open() in Lustre:<br class="">
<br class="">
<blockquote type="cite" class="">If opened with O_LOV_DELAY_CREATE, then we don't do the object creation or open until ll_lov_setstripe() ioctl is called.<br class="">
</blockquote>
<br class="">
It sounds like the expectation is that the process calling open() like this follows it up with an ioctl to set the stripe information prior to writing.
<br class="">
<br class="">
Is this correct? In other words, is it reasonable to say that the failing code is doing something erroneous?<br class="">
<br class="">
Here's a minimal MPI program that reproduces the problem. The issue only arises when using the Cray MPI implementation, however. When tested with openmpi and ANL mpich, the openat() call doesn't use O_LOV_DELAY_CREATE. Since the Cray implementation is unfortunately
not open source, I have no insight into what this code is "supposed" to be doing. :(<br class="">
<br class="">
#include <stdio.h><br class="">
#include <mpi.h><br class="">
<br class="">
int main(int argc, char *argv[])<br class="">
{<br class="">
int err = MPI_Init(&argc, &argv);<br class="">
<br class="">
MPI_File fh;<br class="">
err = MPI_File_open(MPI_COMM_WORLD, "mpi_test.out",<br class="">
MPI_MODE_WRONLY|MPI_MODE_CREATE, MPI_INFO_NULL, &fh);<br class="">
printf("MPI_File_open returned: %d\n", err);<br class="">
<br class="">
long data = 3;<br class="">
err = MPI_File_write(fh, &data, 1, MPI_LONG, MPI_STATUS_IGNORE);<br class="">
printf("MPI_File_write returned: %d\n", err);<br class="">
<br class="">
err = MPI_File_close(&fh);<br class="">
printf("MPI_File_close returned: %d\n", err);<br class="">
<br class="">
MPI_Finalize();<br class="">
return 0;<br class="">
}<br class="">
<br class="">
Thanks, <br class="">
Thomas Bertschinger<br class="">
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>