<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello Chris,</p>
<p>Thank you for the insight. I have in fact already updated the
firmware on the Infiniband adapters. The cards are ConnectX-5 VPI
MCX555A-ECAT and the firmware version:</p>
<p><font face="monospace">[snassyr@io01 ~]$ sudo mstflint -d 81:00.0
q<br>
Image type: FS4<br>
FW Version: 16.34.1002<br>
FW Release Date: 26.7.2022<br>
Product Version: 16.34.1002<br>
Rom Info: type=UEFI version=14.27.14 cpu=AMD64<br>
type=PXE version=3.6.700 cpu=AMD64<br>
Description: UID GuidsNumber<br>
Base GUID: 0c42a1030054820a 4<br>
Base MAC: 0c42a154820a 4<br>
Image VSD: N/A<br>
Device VSD: N/A<br>
PSID: MT_0000000010<br>
Security Attributes: N/A</font></p>
<p>Which is the latest firmware release from NVIDIA/Mellanox, dated
2022-08-02.</p>
<p>I am not using an external driver distribution and the system
packages are up-to-date.<br>
</p>
<p>Best regards,<br>
Stepan<br>
</p>
<div class="moz-cite-prefix">On 17.08.22 16:28, Horn, Chris wrote:<br>
</div>
<blockquote type="cite"
cite="mid:MW4PR84MB14441E2E56F58B7F948607DF9E6A9@MW4PR84MB1444.NAMPRD84.PROD.OUTLOOK.COM">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}div.WordSection1
{page:WordSection1;}ol
{margin-bottom:0in;}ul
{margin-bottom:0in;}</style>
<div class="WordSection1">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white"><span
style="font-family:"Courier New";color:black">[66494.575431]
LNetError: 20017:0:(o2iblnd.c:1880:kiblnd_fmr_pool_map())
Failed to map mr 1/8 elements<br>
[66494.575446] LNetError:
20017:0:(o2iblnd_cb.c:613:kiblnd_fmr_map_tx()) Can't map
32768 bytes (8/8)s: -22</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal">These errors originate from a call to
ib_map_mr_sg() which is part of the kernel verbs API.<br>
<br>
n = ib_map_mr_sg(mr,
tx->tx_frags,<o:p></o:p></p>
<p class="MsoNormal">
rd->rd_nfrags, NULL, PAGE_SIZE);<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> if
(unlikely(n != rd->rd_nfrags)) {<o:p></o:p></p>
<p class="MsoNormal">
CERROR("Failed to map mr %d/%d
elements\n",<o:p></o:p></p>
<p class="MsoNormal">
n, rd->rd_nfrags);<o:p></o:p></p>
<p class="MsoNormal">
return n < 0 ? n : -EINVAL;<o:p></o:p></p>
<p class="MsoNormal"> }<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Your errors mean that we wanted to map 8
fragments to the memory region, but we were only able to map
one of them.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">As a first step, I would recommend ensuring
that you have the latest firmware for your network cards, and
if you’re using an external driver distribution (like
mlnx-ofa_kernel) then upgrade to the latest version. There
could be some bug in the o2iblnd driver code but it is best to
first rule out any issue with firmware/drivers.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Chris Horn<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span
style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">lustre-discuss
<a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss-bounces@lists.lustre.org"><lustre-discuss-bounces@lists.lustre.org></a> on behalf
of Stepan Nassyr via lustre-discuss
<a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss@lists.lustre.org"><lustre-discuss@lists.lustre.org></a><br>
<b>Date: </b>Tuesday, August 16, 2022 at 8:26 AM<br>
<b>To: </b>Peter Jones <a class="moz-txt-link-rfc2396E" href="mailto:pjones@whamcloud.com"><pjones@whamcloud.com></a>,
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss@lists.lustre.org"><lustre-discuss@lists.lustre.org></a><br>
<b>Subject: </b>Re: [lustre-discuss] network error on
bulk WRITE/bad log<o:p></o:p></span></p>
</div>
<p>Hello Peter,<o:p></o:p></p>
<p>Thank you for the reply. I have upgraded lustre to 2.15.1 .
The errors persist, however - now I am also seeing a new error
on io02:<o:p></o:p></p>
<p><span style="font-family:"Courier New"">[
1749.396942] LustreError:
9216:0:(mdt_handler.c:7499:mdt_iocontrol()) storage-MDT0001:
Not supported cmd = 1074292357, rc = -95</span><o:p></o:p></p>
<p>I'm not entirely sure how to look up the cmd code and rc -95
seems to just be EOPNOTSUPP, so no additional information
here.<o:p></o:p></p>
<p>Is there a way to look up what the cmd value means?<o:p></o:p></p>
<div>
<p class="MsoNormal">On 15.08.22 14:50, Peter Jones wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt">Stepan</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt">2.14.56 is
not a version of Lustre – it is an interim dev build. Even
if it does not resolve this specific issue, I would
strongly recommend switching to the recently released
Lustre 2.15.1 release</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt">Peter</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:12.0pt;color:black">From: </span></b><span
style="font-size:12.0pt;color:black">lustre-discuss
<a href="mailto:lustre-discuss-bounces@lists.lustre.org"
moz-do-not-send="true"><lustre-discuss-bounces@lists.lustre.org></a>
on behalf of Stepan Nassyr via lustre-discuss
<a href="mailto:lustre-discuss@lists.lustre.org"
moz-do-not-send="true"><lustre-discuss@lists.lustre.org></a><br>
<b>Reply-To: </b>Stepan Nassyr <a
href="mailto:s.nassyr@fz-juelich.de"
moz-do-not-send="true"><s.nassyr@fz-juelich.de></a><br>
<b>Date: </b>Monday, August 15, 2022 at 1:35 AM<br>
<b>To: </b><a
href="mailto:lustre-discuss@lists.lustre.org"
moz-do-not-send="true">"lustre-discuss@lists.lustre.org"</a>
<a href="mailto:lustre-discuss@lists.lustre.org"
moz-do-not-send="true"><lustre-discuss@lists.lustre.org></a><br>
<b>Subject: </b>[lustre-discuss] network error on bulk
WRITE/bad log</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p>Hi all,<o:p></o:p></p>
<p>In May I had a failure on a small cluster and asked here (<a
href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2022-May/018073.html"
moz-do-not-send="true" class="moz-txt-link-freetext">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2022-May/018073.html</a>).
Due to time constraints I just recreated the filesystem back
then. <o:p></o:p></p>
<p>Now the failure happened again, this time I have more time
and can investigate and haven't done anything destructive
yet.<o:p></o:p></p>
<p>I use the following versions:<o:p></o:p></p>
<ol type="1" start="1">
<li class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0
level1 lfo3">
lustre 2.14.56 <o:p></o:p></li>
<li class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0
level1 lfo3">
zfs 2.0.7 (previously used 2.1.2, but got told that 2.1.x
is not tested well with lustre)
<o:p></o:p></li>
<li class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0
level1 lfo3">
Nodes are running Rocky Linux 8.6<o:p></o:p></li>
<li class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0
level1 lfo3">
uname -r: 4.18.0-372.19.1.el8_6.aarch64<o:p></o:p></li>
</ol>
<p>There are 2 IO nodes (io01 and io02), both of them are MDS
and OSS and one of them is MGS. Here are the devices:<o:p></o:p></p>
<p><span style="font-family:"Courier New"">[snassyr@io02
~]$ sudo lctl dl<br>
0 UP osd-zfs storage-MDT0001-osd
storage-MDT0001-osd_UUID 8<br>
1 UP mgc MGC10.31.7.61@o2ib
a087e05e-d57c-4561-ad75-6827d4428f54 4<br>
2 UP mds MDS MDS_uuid 2<br>
3 UP lod storage-MDT0001-mdtlov
storage-MDT0001-mdtlov_UUID 3<br>
4 UP mdt storage-MDT0001 storage-MDT0001_UUID 8<br>
5 UP mdd storage-MDD0001 storage-MDD0001_UUID 3<br>
6 UP osp storage-MDT0000-osp-MDT0001
storage-MDT0001-mdtlov_UUID 4<br>
7 UP osp storage-OST0000-osc-MDT0001
storage-MDT0001-mdtlov_UUID 4<br>
8 UP osp storage-OST0001-osc-MDT0001
storage-MDT0001-mdtlov_UUID 4<br>
9 UP lwp storage-MDT0000-lwp-MDT0001
storage-MDT0000-lwp-MDT0001_UUID 4<br>
10 UP osd-zfs storage-OST0001-osd
storage-OST0001-osd_UUID 4<br>
11 UP ost OSS OSS_uuid 2<br>
12 UP obdfilter storage-OST0001 storage-OST0001_UUID 6<br>
13 UP lwp storage-MDT0000-lwp-OST0001
storage-MDT0000-lwp-OST0001_UUID 4<br>
14 UP lwp storage-MDT0001-lwp-OST0001
storage-MDT0001-lwp-OST0001_UUID 4</span><o:p></o:p></p>
<p><span style="font-family:"Courier New"">[snassyr@io01
~]$ sudo lctl dl<br>
0 UP osd-zfs MGS-osd MGS-osd_UUID 4<br>
1 UP mgs MGS MGS 6<br>
2 UP mgc MGC10.31.7.61@o2ib
9f351a51-0232-4306-a66d-cecee8629329 4<br>
3 UP osd-zfs storage-MDT0000-osd
storage-MDT0000-osd_UUID 9<br>
4 UP mds MDS MDS_uuid 2<br>
5 UP lod storage-MDT0000-mdtlov
storage-MDT0000-mdtlov_UUID 3<br>
6 UP mdt storage-MDT0000 storage-MDT0000_UUID 12<br>
7 UP mdd storage-MDD0000 storage-MDD0000_UUID 3<br>
8 UP qmt storage-QMT0000 storage-QMT0000_UUID 3<br>
9 UP osp storage-MDT0001-osp-MDT0000
storage-MDT0000-mdtlov_UUID 4<br>
10 UP osp storage-OST0000-osc-MDT0000
storage-MDT0000-mdtlov_UUID 4<br>
11 UP osp storage-OST0001-osc-MDT0000
storage-MDT0000-mdtlov_UUID 4<br>
12 UP lwp storage-MDT0000-lwp-MDT0000
storage-MDT0000-lwp-MDT0000_UUID 4<br>
13 UP osd-zfs storage-OST0000-osd
storage-OST0000-osd_UUID 4<br>
14 UP ost OSS OSS_uuid 2<br>
15 UP obdfilter storage-OST0000 storage-OST0000_UUID 6<br>
16 UP lwp storage-MDT0000-lwp-OST0000
storage-MDT0000-lwp-OST0000_UUID 4<br>
17 UP lwp storage-MDT0001-lwp-OST0000
storage-MDT0001-lwp-OST0000_UUID 4</span><o:p></o:p></p>
<p>On io01 I see repeating errors mentioning a network error:<o:p></o:p></p>
<p><span style="font-family:"Courier New"">[65922.582578]
LustreError: 20017:0:(ldlm_lib.c:3540:target_bulk_io())
Skipped 11 previous similar messages<br>
[66494.575431] LNetError:
20017:0:(o2iblnd.c:1880:kiblnd_fmr_pool_map()) Failed to
map mr 1/8 elements<br>
[66494.575442] LNetError:
20017:0:(o2iblnd.c:1880:kiblnd_fmr_pool_map()) Skipped 11
previous similar messages<br>
[66494.575446] LNetError:
20017:0:(o2iblnd_cb.c:613:kiblnd_fmr_map_tx()) Can't map
32768 bytes (8/8)s: -22<br>
[66494.575448] LNetError:
20017:0:(o2iblnd_cb.c:613:kiblnd_fmr_map_tx()) Skipped 11
previous similar messages<br>
[66494.575452] LNetError:
20017:0:(o2iblnd_cb.c:1725:kiblnd_send()) Can't setup PUT
src for 10.31.7.62@o2ib: -22<br>
[66494.575454] LNetError:
20017:0:(o2iblnd_cb.c:1725:kiblnd_send()) Skipped 11
previous similar messages<br>
[66494.575458] LustreError:
20017:0:(events.c:477:server_bulk_callback()) event type
5, status -5, desc 00000000cdd4e797<br>
[66494.575460] LustreError:
20017:0:(events.c:477:server_bulk_callback()) Skipped 11
previous similar messages<br>
[66546.574314] LustreError:
20017:0:(ldlm_lib.c:3540:target_bulk_io()) @@@ network
error on bulk WRITE req@0000000070b8f1ab
x1740960836990720/t0(0) o1000-><a
href="mailto:storage-MDT0001-mdtlov_UUID@10.31.7.62@o2ib:522/0"
moz-do-not-send="true" class="moz-txt-link-freetext">storage-MDT0001-mdtlov_UUID@10.31.7.62@o2ib:522/0</a>
lens 336/33016 e 0 to 0 dl 1660376137 ref 1 fl
Interpret:/0/0 rc 0/0 job:''</span><o:p></o:p></p>
<p>On io02 I see repeating errors mentioning a bad log:<o:p></o:p></p>
<p><span style="font-family:"Courier New"">[66582.856444]
LustreError:
14905:0:(llog_osd.c:264:llog_osd_read_header())
storage-MDT0000-osp-MDT0001: bad log
[0x200000401:0x1:0x0] header magic: 0x0 (expected
0x10645539)<br>
[66582.856450] LustreError:
14905:0:(llog_osd.c:264:llog_osd_read_header()) Skipped 11
previous similar messages</span><o:p></o:p></p>
<p>I can't make sense of these error messages. How can I
recover?<o:p></o:p></p>
<p>(I have the full dmesg/lctl dk log, but they are too big to
attach, is it ok to upload them somewhere and put a link in
a reply?)<o:p></o:p></p>
<p>Thank you and best regards,<br>
Stepan<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal"><br>
<span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:black"><br>
------------------------------------------------------------------------------------------------<br>
------------------------------------------------------------------------------------------------<br>
Forschungszentrum Juelich GmbH<br>
52425 Juelich<br>
Sitz der Gesellschaft: Juelich<br>
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr.
HR B 3498<br>
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke<br>
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt
(Vorsitzender),<br>
Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Astrid
Lambrecht,<br>
Prof. Dr. Frauke Melchior<br>
------------------------------------------------------------------------------------------------<br>
------------------------------------------------------------------------------------------------<br>
<br>
<br>
Neugierige sind herzlich willkommen am Sonntag, den 21.
August 2022, von 10:00 bis 17:00 Uhr. Mehr unter:
<a href="https://www.tagderneugier.de"
moz-do-not-send="true" class="moz-txt-link-freetext">https://www.tagderneugier.de</a><br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
</div>
</blockquote>
</body>
</html>