<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Hi John,</p>
<p><br>
</p>
<p>Thanks for looking at the code.</p>
<p><br>
</p>
<p>Right for the loop, but this clearly will not affect the performance on IO.</p>
<p><br>
</p>
<p>About concurrent write jobs: i am experiencing very low rate even with only</p>
<p>1 MPI process ( so no concurrency at all ).</p>
<p><br>
</p>
<p>About the array size not being a multiple of stripe size, this is affecting the performance</p>
<p>in case i have many MPI writing process, right ?</p>
<p><br>
</p>
<p>Best,</p>
<p>Denis </p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of John Bauer <bauerj@iodoctors.com><br>
<b>Sent:</b> Thursday, February 3, 2022 5:16:52 PM<br>
<b>To:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] lustre-discuss Digest, Vol 191, Issue 2</font>
<div> </div>
</div>
<div>
<p> The following loop in wdfile.f90 is pointless as the write happens only once for each rank.<br>
</p>
<p>Each rank is writing out the array once and then closing the file. If the size of array 'data' is not a multiple of the Lustre stripe size there is going to be a lot of read-modify-write going on.<br>
</p>
<p><font face="Courier New, Courier, monospace"> do ii = 0, size<br>
if ( rank == ii ) then<br>
!start= MPI_Wtime()<br>
write(unit=iounit) data(1:nx, 1:ny, 1:nz)<br>
close(iounit)<br>
!finish = MPI_Wtime()<br>
!write(6,'(i5,f7.4)') rank, finish - start<br>
else<br>
end if<br>
end do</font><br>
<br>
</p>
<div class="moz-cite-prefix">On 2/3/2022 9:39 AM, <a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss-request@lists.lustre.org">
lustre-discuss-request@lists.lustre.org</a> wrote:<br>
</div>
<blockquote type="cite" cite="mid:mailman.1988.1643902742.23952.lustre-discuss-lustre.org@lists.lustre.org">
<pre class="moz-quote-pre" wrap="">Send lustre-discuss mailing list submissions to
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
To subscribe or unsubscribe via the World Wide Web, visit
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
or, via email, send a message with subject or body 'help' to
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss-request@lists.lustre.org">lustre-discuss-request@lists.lustre.org</a>
You can reach the person managing the list at
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss-owner@lists.lustre.org">lustre-discuss-owner@lists.lustre.org</a>
When replying, please edit your Subject line so it is more specific
than "Re: Contents of lustre-discuss digest..."
Today's Topics:
1. RE-Fortran IO problem (Bertini, Denis Dr.)
2. Re: RE-Fortran IO problem (Patrick Farrell)
3. Re: RE-Fortran IO problem (Bertini, Denis Dr.)
----------------------------------------------------------------------
Message: 1
Date: Thu, 3 Feb 2022 12:43:21 +0000
From: "Bertini, Denis Dr." <a class="moz-txt-link-rfc2396E" href="mailto:D.Bertini@gsi.de"><D.Bertini@gsi.de></a>
To: <a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss@lists.lustre.org">"lustre-discuss@lists.lustre.org"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss@lists.lustre.org"><lustre-discuss@lists.lustre.org></a>
Subject: [lustre-discuss] RE-Fortran IO problem
Message-ID: <a class="moz-txt-link-rfc2396E" href="mailto:b38ed90e7586428f9f46f742b0b359ef@gsi.de"><b38ed90e7586428f9f46f742b0b359ef@gsi.de></a>
Content-Type: text/plain; charset="iso-8859-1"
Hi,
Just as an add-on to my previous mail, the problem shows up also
with intel fortran and it not specific to gnu fortran compiler.
So it seems to be linked to how the fortran IO is handled which
seems to be sub-optimal in cas of a Lustre filesystem.
I would be grateful if one can confirm/disconfirm that.
Here again the access to the code i used for my benchmarks:
<a class="moz-txt-link-freetext" href="https://git.gsi.de/hpc/cluster/ci_ompi/-/tree/main/f/src">https://git.gsi.de/hpc/cluster/ci_ompi/-/tree/main/f/src</a>
Best,
Denis
---------
Denis Bertini
Abteilung: CIT
Ort: SB3 2.265a
Tel: +49 6159 71 2240
Fax: +49 6159 71 2986
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:d.bertini@gsi.de">d.bertini@gsi.de</a>
GSI Helmholtzzentrum f?r Schwerionenforschung GmbH
Planckstra?e 1, 64291 Darmstadt, Germany, <a class="moz-txt-link-abbreviated" href="http://www.gsi.de">www.gsi.de</a>
Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Gesch?ftsf?hrung:
Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, J?rg Blaurock
Chairman of the GSI Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
Ministerialdirigent Dr. Volkmar Dietz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <a class="moz-txt-link-rfc2396E" href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220203/9f68dae9/attachment-0001.html"><http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220203/9f68dae9/attachment-0001.html></a>
------------------------------
Message: 2
Date: Thu, 3 Feb 2022 15:15:16 +0000
From: Patrick Farrell <a class="moz-txt-link-rfc2396E" href="mailto:pfarrell@ddn.com"><pfarrell@ddn.com></a>
To: "Bertini, Denis Dr." <a class="moz-txt-link-rfc2396E" href="mailto:D.Bertini@gsi.de"><D.Bertini@gsi.de></a>,
<a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss@lists.lustre.org">"lustre-discuss@lists.lustre.org"</a> <a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss@lists.lustre.org"><lustre-discuss@lists.lustre.org></a>
Subject: Re: [lustre-discuss] RE-Fortran IO problem
Message-ID:
<a class="moz-txt-link-rfc2396E" href="mailto:DM6PR19MB3129DB8B22167BBDB55569B3C9289@DM6PR19MB3129.namprd19.prod.outlook.com"><DM6PR19MB3129DB8B22167BBDB55569B3C9289@DM6PR19MB3129.namprd19.prod.outlook.com></a>
Content-Type: text/plain; charset="utf-8"
Denis,
FYI, the git link you provided seems to be non-public - it asks for a GSI login.
Fortran is widely used for applications on Lustre, so it's unlikely to be a fortran specific issue. If you're seeing I/O rates drop suddenly during? activity, rather than being reliably low for some particular operation, I would look to the broader Lustre system. It may be suddenly extremely busy or there could be, eg, a temporary network issue - Assuming this is a system belonging to your institution, I'd check with your admins.
Regards,
Patrick
________________________________
From: lustre-discuss <a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss-bounces@lists.lustre.org"><lustre-discuss-bounces@lists.lustre.org></a> on behalf of Bertini, Denis Dr. <a class="moz-txt-link-rfc2396E" href="mailto:D.Bertini@gsi.de"><D.Bertini@gsi.de></a>
Sent: Thursday, February 3, 2022 6:43 AM
To: <a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a> <a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss@lists.lustre.org"><lustre-discuss@lists.lustre.org></a>
Subject: [lustre-discuss] RE-Fortran IO problem
Hi,
Just as an add-on to my previous mail, the problem shows up also
with intel fortran and it not specific to gnu fortran compiler.
So it seems to be linked to how the fortran IO is handled which
seems to be sub-optimal in cas of a Lustre filesystem.
I would be grateful if one can confirm/disconfirm that.
Here again the access to the code i used for my benchmarks:
<a class="moz-txt-link-freetext" href="https://git.gsi.de/hpc/cluster/ci_ompi/-/tree/main/f/src">https://git.gsi.de/hpc/cluster/ci_ompi/-/tree/main/f/src</a>
Best,
Denis
---------
Denis Bertini
Abteilung: CIT
Ort: SB3 2.265a
Tel: +49 6159 71 2240
Fax: +49 6159 71 2986
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:d.bertini@gsi.de">d.bertini@gsi.de</a>
GSI Helmholtzzentrum f?r Schwerionenforschung GmbH
Planckstra?e 1, 64291 Darmstadt, Germany, <a class="moz-txt-link-abbreviated" href="http://www.gsi.de">www.gsi.de</a>
Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Gesch?ftsf?hrung:
Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, J?rg Blaurock
Chairman of the GSI Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
Ministerialdirigent Dr. Volkmar Dietz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <a class="moz-txt-link-rfc2396E" href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220203/87d50e4f/attachment-0001.html"><http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220203/87d50e4f/attachment-0001.html></a>
------------------------------
Message: 3
Date: Thu, 3 Feb 2022 15:38:58 +0000
From: "Bertini, Denis Dr." <a class="moz-txt-link-rfc2396E" href="mailto:D.Bertini@gsi.de"><D.Bertini@gsi.de></a>
To: Patrick Farrell <a class="moz-txt-link-rfc2396E" href="mailto:pfarrell@ddn.com"><pfarrell@ddn.com></a>,
<a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss@lists.lustre.org">"lustre-discuss@lists.lustre.org"</a> <a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss@lists.lustre.org"><lustre-discuss@lists.lustre.org></a>
Subject: Re: [lustre-discuss] RE-Fortran IO problem
Message-ID: <a class="moz-txt-link-rfc2396E" href="mailto:9476216bb77b4905b0ee8db9598dc6df@gsi.de"><9476216bb77b4905b0ee8db9598dc6df@gsi.de></a>
Content-Type: text/plain; charset="utf-8"
Dear Patrick,
Thanks for the quick answer.
Sorry for the broken link, i send anyway the code as an tarball in
my previous mail
I am myself using fortran together with Lustre, using a different application
IO than pure fortran i.e MPI-IO or HDF5 and it works just fine.
I never used the pure fortran IO with lustre and was surprised by the
low performance on our filesystem that users reported.
In the tarball i adapted the pure fortran code to use a different application IO (HDF5).
In this case i see no preformance problem.
Could it be that i both case, there are data contention problem and/or temporal network
issue but the HDF5 IO is just more resilient than fortran IO to these issue.
Anyway how to check properly for these Lustre system problem ( data contention/network)?
Regards,
Denis
________________________________
From: Patrick Farrell <a class="moz-txt-link-rfc2396E" href="mailto:pfarrell@ddn.com"><pfarrell@ddn.com></a>
Sent: Thursday, February 3, 2022 4:15:16 PM
To: Bertini, Denis Dr.; <a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
Subject: Re: RE-Fortran IO problem
Denis,
FYI, the git link you provided seems to be non-public - it asks for a GSI login.
Fortran is widely used for applications on Lustre, so it's unlikely to be a fortran specific issue. If you're seeing I/O rates drop suddenly during? activity, rather than being reliably low for some particular operation, I would look to the broader Lustre system. It may be suddenly extremely busy or there could be, eg, a temporary network issue - Assuming this is a system belonging to your institution, I'd check with your admins.
Regards,
Patrick
________________________________
From: lustre-discuss <a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss-bounces@lists.lustre.org"><lustre-discuss-bounces@lists.lustre.org></a> on behalf of Bertini, Denis Dr. <a class="moz-txt-link-rfc2396E" href="mailto:D.Bertini@gsi.de"><D.Bertini@gsi.de></a>
Sent: Thursday, February 3, 2022 6:43 AM
To: <a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a> <a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss@lists.lustre.org"><lustre-discuss@lists.lustre.org></a>
Subject: [lustre-discuss] RE-Fortran IO problem
Hi,
Just as an add-on to my previous mail, the problem shows up also
with intel fortran and it not specific to gnu fortran compiler.
So it seems to be linked to how the fortran IO is handled which
seems to be sub-optimal in cas of a Lustre filesystem.
I would be grateful if one can confirm/disconfirm that.
Here again the access to the code i used for my benchmarks:
<a class="moz-txt-link-freetext" href="https://git.gsi.de/hpc/cluster/ci_ompi/-/tree/main/f/src">https://git.gsi.de/hpc/cluster/ci_ompi/-/tree/main/f/src</a>
Best,
Denis
---------
Denis Bertini
Abteilung: CIT
Ort: SB3 2.265a
Tel: +49 6159 71 2240
Fax: +49 6159 71 2986
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:d.bertini@gsi.de">d.bertini@gsi.de</a>
GSI Helmholtzzentrum f?r Schwerionenforschung GmbH
Planckstra?e 1, 64291 Darmstadt, Germany, <a class="moz-txt-link-abbreviated" href="http://www.gsi.de">www.gsi.de</a>
Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Gesch?ftsf?hrung:
Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, J?rg Blaurock
Chairman of the GSI Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
Ministerialdirigent Dr. Volkmar Dietz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <a class="moz-txt-link-rfc2396E" href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220203/10fb4b81/attachment.html"><http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220203/10fb4b81/attachment.html></a>
------------------------------
Subject: Digest Footer
_______________________________________________
lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
------------------------------
End of lustre-discuss Digest, Vol 191, Issue 2
**********************************************
</pre>
</blockquote>
</div>
</body>
</html>