<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Boom. I think you just nailed it. The user confirmed that he uses
this code almost daily. We use environment modules here, so I
checked to see if it was possible that this error was occuring
because he loaded the hdf5/1.10p1 module instead of hdf5/1.8 this
time. Checking the executable with ldd and a clean environment (no
modules loaded), shows that the program is dynamically linked, but
RPATH is set to use the HDF5 1.10 libraries Looking at the mtime
of the file, it looks like the executable was rebuilt on 1/23. <br>
</p>
<p>I haven't confirmed it with the user yet, but most likely the
version before 1/23 was built with HDF5 1.8, then it was rebuilt
on 1/23 with HDF5 1.10p1, which caused this changed behavior. <br>
</p>
<p>Thanks so much for suggesting this could be the problem! <br>
</p>
<pre class="moz-signature" cols="72">Prentice </pre>
<div class="moz-cite-prefix">On 02/15/2018 06:19 PM, Arman Khalatyan
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAAqDm6Yb_5CQS9DF9uY2Y+y0QD=e5qNB5f4Kb7eUdTn2=0K61Q@mail.gmail.com">
<div dir="auto">we had similar troubles with hdf1.10 vs hdf1.8.x.
on the lustre
<div dir="auto">the new hdf require flock support from the
underlying filesystem( due to the security reasons or whatever
more info on hdf you can digg in hdf forums)</div>
<div dir="auto">to fix the mounts you should unmount an mount
again with the option localflock, this works for us,
independent on lustre version. </div>
<div dir="auto">that what we did:</div>
<div dir="auto"><br>
</div>
<div dir="auto"><a
href="https://arm2armcos.blogspot.de/2018/02/hdf5-v110-or-above-on-lustre-fs.html?m=1"
moz-do-not-send="true">https://arm2armcos.blogspot.de/2018/02/hdf5-v110-or-above-on-lustre-fs.html?m=1</a><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
<div dir="auto"><br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">Am 15.02.2018 11:18 nachm. schrieb
"E.S. Rosenberg" <<a
href="mailto:esr%2Blustre@mail.hebrew.edu"
moz-do-not-send="true">esr+lustre@mail.hebrew.edu</a>>:<br
type="attribution">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Feb 16, 2018 at 12:00
AM, Colin Faber <span dir="ltr"><<a
href="mailto:cfaber@gmail.com" target="_blank"
moz-do-not-send="true">cfaber@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">If the mount on the users clients
had the various options enabled, and those aren't
present in fstab, you'd end up with such behavior.
Also 2.8? Can you upgrade to 2.10 LTS??</div>
</blockquote>
<div>Depending on when they installed their system
that may not be such a 'small' change, our 2.8 is
running on CentOS 6.8 so an upgrade to 2.10 requires
us to also upgrade the OS from 6.x to 7.x and though
I very much want to do that that is a more intensive
process that so far I have not had the time for and
I can imagine others have the same issue.<br>
</div>
<div>Regards,<br>
</div>
<div>Eli<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
</div>
<div class="m_5059166698001362483HOEnZb">
<div class="m_5059166698001362483h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Feb 15, 2018 1:06
PM, "Prentice Bisbal" <<a
href="mailto:pbisbal@pppl.gov"
target="_blank" moz-do-not-send="true">pbisbal@pppl.gov</a>>
wrote:<br type="attribution">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>No. Several others have asked me the
same thing, so that seems like it
might be the issue. The only problem
with that solution is that the user
claimed his program worked just fine
up until a couple of weeks ago, so if
that is the issue, I'll still be
scratching my head trying to figure
out how/what changed</p>
<p><br>
</p>
<pre class="m_5059166698001362483m_-569576501198410965m_-1995677135630938282moz-signature" cols="72">Prentice </pre>
<div
class="m_5059166698001362483m_-569576501198410965m_-1995677135630938282moz-cite-prefix">On
02/15/2018 12:31 PM, Alexander I
Kulyavtsev wrote:<br>
</div>
<blockquote type="cite">
<div>Do you have <b>flock</b> option
in fstab for lustre mount or in
command you use to mount lustre on
client?</div>
<div><br>
</div>
<div>Search for flock on lustre wiki</div>
<div><span class="m_5059166698001362483m_-569576501198410965m_-1995677135630938282Apple-tab-span" style="white-space:pre-wrap"></span><a
href="http://wiki.lustre.org/Mounting_a_Lustre_File_System_on_Client_Nodes"
target="_blank"
moz-do-not-send="true">http://wiki.lustre.org/Mountin<wbr>g_a_Lustre_File_System_on_Clie<wbr>nt_Nodes</a></div>
<div>or lustre manual</div>
<div><span class="m_5059166698001362483m_-569576501198410965m_-1995677135630938282Apple-tab-span" style="white-space:pre-wrap"></span><a
href="http://doc.lustre.org/lustre_manual.pdf" target="_blank"
moz-do-not-send="true">http://doc.lustre.org/lustre_m<wbr>anual.pdf</a></div>
<div><br>
</div>
<div>Here are links where to start
learning about lustre:</div>
<div>* <a
href="http://lustre.org/getting-started-with-lustre/"
target="_blank"
moz-do-not-send="true">
http://lustre.org/getting-star<wbr>ted-with-lustre/</a></div>
<div>* <a
href="http://wiki.lustre.org"
target="_blank"
moz-do-not-send="true">http://wiki.lustre.org</a></div>
<div>* <a
href="https://wiki.hpdd.intel.com"
target="_blank"
moz-do-not-send="true">https://wiki.hpdd.intel.com</a></div>
<div>* <a
href="http://jira.hpdd.intel.com"
target="_blank"
moz-do-not-send="true">jira.hpdd.intel.com</a></div>
<div>* <a
href="http://opensfs.org/lustre/"
target="_blank"
moz-do-not-send="true">http://opensfs.org/lustre/</a></div>
<div><br>
</div>
<div>Alex.</div>
<br>
<div>
<blockquote type="cite">
<div>On Feb 15, 2018, at 11:02 AM,
Prentice Bisbal <<a
href="mailto:pbisbal@pppl.gov"
target="_blank"
moz-do-not-send="true">pbisbal@pppl.gov</a>>
wrote:</div>
<br
class="m_5059166698001362483m_-569576501198410965m_-1995677135630938282Apple-interchange-newline">
<div>
<div>Hi.<br>
<br>
I'm an experience HPC system
admin, but I know almost
nothing about Lustre
administration. The system
admin who administered our
small Lustre filesystem
recently retired, and no one
has filled that gap yet. A
user recently reported they
are now getting file-locking
errors from a program they've
run repeatedly on Lustre in
the past. When the run the
same program on an NFS
filesystem, the error goes
away. I've cut-and-pasted the
error messages below.<br>
<br>
Since I have real experience
as a Lustre admin, I turned to
google, and it looks like it
might be that the file-locking
daemon died (if Lustre has a
separate file-lock daemon), or
somehow file-locking was
recently disabled. If that is
possible, how do I check this,
and restart or re-enable if
necessary? I skimmed the user
manual, and could not find
anything on either of these
issues.<br>
<br>
Any and all help will be
greatly appreciated.<br>
<br>
Some of the error messages:<br>
<br>
HDF5-DIAG: Error detected in
HDF5 (1.10.0-patch1)
MPI-process 9:<br>
#000: H5F.c line 579 in
H5Fopen(): unable to open file<br>
major: File accessibilty<br>
minor: Unable to open file<br>
#001: H5Fint.c line 1168 in
H5F_open(): unable to lock the
file or initialize file
structure<br>
major: File accessibilty<br>
minor: Unable to open file<br>
#002: H5FD.c line 1821 in
H5FD_lock(): driver lock
request failed<br>
major: Virtual File Layer<br>
minor: Can't update object<br>
#003: H5FDsec2.c line 939 in
H5FD_sec2_lock(): unable to
flock file, errno = 38, error
message = 'Function not
implemented'<br>
major: File accessibilty<br>
minor: Bad file ID
accessed<br>
Error: couldn't open file
HDF5-DIAG: Error detected in
HDF5 (1.10.0-patch1)
MPI-process 13:<br>
#000: H5F.c line 579 in
H5Fopen(): unable to open file<br>
major: File accessibilty<br>
minor: Unable to open file<br>
#001: H5Fint.c line 1168 in
H5F_open(): unable to lock the
file or initialize file
structure<br>
major: File accessibilty<br>
minor: Unable to open file<br>
#002: H5FD.c line 1821 in
H5FD_lock(): driver lock
request failed<br>
major: Virtual File Layer<br>
minor: Can't update object<br>
#003: H5FDsec2.c line 939 in
H5FD_sec2_lock(): unable to
flock file, errno = 38, error
message = 'Function not
implemented'<br>
major: File accessibilty<br>
minor: Bad file ID
accessed<br>
<br>
-- <br>
Prentice<br>
<br>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a
href="mailto:lustre-discuss@lists.lustre.org"
target="_blank"
moz-do-not-send="true">lustre-discuss@lists.lustre.or<wbr>g</a><br>
<a
class="m_5059166698001362483m_-569576501198410965m_-1995677135630938282moz-txt-link-freetext"
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
target="_blank"
moz-do-not-send="true">http://lists.lustre.org/listin<wbr>fo.cgi/lustre-discuss-lustre.o<wbr>rg</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
</div>
<br>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a
href="mailto:lustre-discuss@lists.lustre.org"
target="_blank" moz-do-not-send="true">lustre-discuss@lists.lustre.or<wbr>g</a><br>
<a
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lustre.org/listin<wbr>fo.cgi/lustre-discuss-lustre.o<wbr>rg</a><br>
<br>
</blockquote>
</div>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org"
target="_blank" moz-do-not-send="true">lustre-discuss@lists.lustre.or<wbr>g</a><br>
<a
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lustre.org/listin<wbr>fo.cgi/lustre-discuss-lustre.<wbr>org</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org"
moz-do-not-send="true">lustre-discuss@lists.lustre.<wbr>org</a><br>
<a
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br>
</blockquote>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
<br>
</body>
</html>