<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello everybody.<br>
</p>
<p><br>
</p>
<p>I am dealing with an issue with a relatively new Lustre
installation. The Metadata Server (MDS) hangs randomly without any
common pattern. It can take anywhere from 30 minutes to 30 days,
but it always ends up hanging without a consistent pattern (at
least, I haven't found one). The logs don't show anything unusual
at the time of the failure. The only thing I continuously see are
these messages:<br>
<br>
<i>[lun ene 20 14:17:10 2025] LustreError:
7068:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed
with -22, flags:0x4 qsd:LUSTRE-OST138f qtype:prj id:2325
enforced:1 granted: 16304159618662232032 pending:0 waiting:0
req:1 usage: 114636 qunit:262144 qtune:65536 edquot:0
default:yes<br>
[lun ene 20 14:17:10 2025] LustreError:
7068:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 39
previous similar messages<br>
[lun ene 20 14:21:52 2025] LustreError:
1895328:0:(qmt_handler.c:798:qmt_dqacq0()) $$$ Release too much!
uuid:LUSTRE-MDT0000-lwp-OST0c1f_UUID release:
15476132855418716160 granted:262144, total:14257500
qmt:LUSTRE-QMT0000 pool:dt-0x0 id:2582 enforced:1 hard:62914560
soft:52428800 granted:14257500 time:0 qunit: 262144 edquot:0
may_rel:0 revoke:0 default:yes<br>
[lun ene 20 14:21:52 2025] LustreError:
1947381:0:(qmt_handler.c:798:qmt_dqacq0()) $$$ Release too much!
uuid:LUSTRE-MDT0000-lwp-OST0fb2_UUID release:
13809297465413342331 granted:66568, total:14179564
qmt:LUSTRE-QMT0000 pool:dt-0x0 id:2325 enforced:1 hard:62914560
soft:52428800 granted:14179564 time:0 qunit: 262144 edquot:0
may_rel:0 revoke:0 default:yes<br>
[lun ene 20 14:21:52 2025] LustreError:
1947381:0:(qmt_handler.c:798:qmt_dqacq0()) Skipped 802 previous
similar messages<br>
[lun ene 20 14:21:52 2025] LustreError:
1895328:0:(qmt_handler.c:798:qmt_dqacq0()) Skipped 802 previous
similar messages<br>
[lun ene 20 14:27:24 2025] LustreError:
7047:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed
with -22, flags:0x4 qsd:LUSTRE-OST138f qtype:prj id:2325
enforced:1 granted: 16304159618662232032 pending:0 waiting:0
req:1 usage: 114636 qunit:262144 qtune:65536 edquot:0
default:yes<br>
[lun ene 20 14:27:24 2025] LustreError:
7047:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 39
previous similar messages<br>
[lun ene 20 14:31:52 2025] LustreError:
1844354:0:(qmt_handler.c:798:qmt_dqacq0()) $$$ Release too much!
uuid:LUSTRE-MDT0000-lwp-OST1399_UUID release:
12882711387029922688 granted:66116, total:14078012
qmt:LUSTRE-QMT0000 pool:dt-0x0 id:2586 enforced:1 hard:62914560
soft:52428800 granted:14078012 time:0 qunit: 262144 edquot:0
may_rel:0 revoke:0 default:yes<br>
[lun ene 20 14:31:52 2025] LustreError:
1844354:0:(qmt_handler.c:798:qmt_dqacq0()) Skipped 785 previous
similar messages<br>
[lun ene 20 14:37:39 2025] LustreError:
7054:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed
with -22, flags:0x4 qsd:LUSTRE-OST138f qtype:prj id:2325
enforced:1 granted: 16304159618662232032 pending:0 waiting:0
req:1 usage: 114636 qunit:262144 qtune:65536 edquot:0
default:yes<br>
[lun ene 20 14:37:39 2025] LustreError:
7054:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 39
previous similar messages<br>
[lun ene 20 14:41:54 2025] LustreError:
1895328:0:(qmt_handler.c:798:qmt_dqacq0()) $$$ Release too much!
uuid:LUSTRE-MDT0000-lwp-OST0faa_UUID release:
13811459193234480169 granted:65632, total:14179564
qmt:LUSTRE-QMT0000 pool:dt-0x0 id:2325 enforced:1 hard:62914560
soft:52428800 granted:14179564 time:0 qunit: 262144 edquot:0
may_rel:0 revoke:0 default:yes<br>
[lun ene 20 14:41:54 2025] LustreError:
1895328:0:(qmt_handler.c:798:qmt_dqacq0()) Skipped 798 previous
similar messages<br>
[lun ene 20 14:47:53 2025] LustreError:
7052:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed
with -22, flags:0x4 qsd:LUSTRE-OST138f qtype:prj id:2325
enforced:1 granted: 16304159618662232032 pending:0 waiting:0
req:1 usage: 114636 qunit:262144 qtune:65536 edquot:0
default:yes<br>
[lun ene 20 14:47:53 2025] LustreError:
7052:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 39
previous similar messages<br>
</i><br>
I have ruled out hardware failure since the MDS service has been
moved between different servers, and it happens with all of them.<br>
<br>
Linux distribution: AlmaLinux release 8.10 (Cerulean Leopard)<br>
Kernel: Linux srv-lustre15 4.18.0-553.5.1.el8_lustre.x86_64 #1 SMP
Fri Jun 28 18:44:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux<br>
Lustre release: lustre-2.15.5-1.el8.x86_64<br>
Not using ZFS.<br>
<br>
Any ideas on where to continue investigating?<br>
Is the error appearing in dmesg a bug, or is it a corruption in
the quota database?<br>
<br>
The possible bugs affecting quotas that might be related seem to
be fixed in version 2.15.</p>
<p><br>
</p>
<p>Thanks in advance.<br>
</p>
<div class="moz-signature">-- <br>
<!--?xml version="1.0" encoding="UTF-8"?-->
<!--This file was converted to xhtml by LibreOffice - see https://cgit.freedesktop.org/libreoffice/core/tree/filter/source/xslt for the code.-->
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title xml:lang="en-US
">- no title specified</title>
<meta name="DCTERMS.title" content="" xml:lang="en-US
">
<meta name="DCTERMS.language" content="en-US
" scheme="DCTERMS.RFC4646">
<meta name="DCTERMS.source"
content="http://xml.openoffice.org/odf2xhtml">
<meta name="DCTERMS.issued" content="2024-07-04T11:24:00"
scheme="DCTERMS.W3CDTF">
<meta name="DCTERMS.modified" content="2024-07-04T11:24:00"
scheme="DCTERMS.W3CDTF">
<meta name="DCTERMS.provenance" content="
" xml:lang="en-US
">
<meta name="xsl:vendor" content="libxslt">
<link rel="schema.DC" href="http://purl.org/dc/elements/1.1/"
hreflang="en">
<link rel="schema.DCTERMS" href="http://purl.org/dc/terms/"
hreflang="en">
<link rel="schema.DCTYPE" href="http://purl.org/dc/dcmitype/"
hreflang="en">
<link rel="schema.DCAM" href="http://purl.org/dc/dcam/"
hreflang="en">
<style>
table { border-collapse:collapse; border-spacing:0; empty-cells:show }
td, th { vertical-align:top; font-size:12pt;}
h1, h2, h3, h4, h5, h6 { clear:both;}
ol, ul { margin:0; padding:0;}
li { list-style: none; margin:0; padding:0;}
span.footnodeNumber { padding-right:1em; }
span.annotation_style_by_filter { font-size:95%; font-family:Arial; background-color:#fff000; margin:0; border:0; padding:0; }
span.heading_numbering { margin-right: 0.8rem; }* { margin:0;}
.fr1 { font-size:11pt; font-family:Calibri; text-align:center; vertical-align:top; writing-mode:horizontal-tb; direction:ltr; border-top-style:none; border-left-style:none; border-bottom-style:none; border-right-style:none; margin-left:0in; margin-right:0in; margin-top:0in; margin-bottom:0in; background-color:transparent; padding:0in; }
.P1 { font-size:12pt; line-height:100%; margin-bottom:0in; margin-top:0in; text-align:left ! important; font-family:'Times New Roman'; writing-mode:horizontal-tb; direction:ltr; }
.P2 { font-size:9pt; line-height:100%; margin-bottom:0in; margin-top:0in; text-align:left ! important; font-family:'AvenirNext LT Pro Regular'; writing-mode:horizontal-tb; direction:ltr; color:#999999; }
.P4 { font-size:11pt; line-height:100%; margin-bottom:0in; margin-top:0in; text-align:left ! important; font-family:Calibri; writing-mode:horizontal-tb; direction:ltr; }
.P5 { font-size:11pt; line-height:100%; margin-bottom:0in; margin-top:0in; text-align:left ! important; font-family:Calibri; writing-mode:horizontal-tb; direction:ltr; }
.P6 { font-size:11pt; line-height:100%; margin-bottom:0in; margin-top:0in; text-align:left ! important; font-family:Calibri; writing-mode:horizontal-tb; direction:ltr; }
.P7 { font-size:11pt; line-height:100%; margin-bottom:0in; margin-top:0in; text-align:justify ! important; font-family:Calibri; writing-mode:horizontal-tb; direction:ltr; }
.Standard { font-size:11pt; font-family:Calibri; writing-mode:horizontal-tb; direction:ltr; margin-top:0in; margin-bottom:0.111in; line-height:108%; text-align:left ! important; }
.Table1 { width:7.3799in; margin-left:0in; margin-top:0in; margin-bottom:0in; margin-right:auto;writing-mode:horizontal-tb; direction:ltr; }
.Table1_A1 { border-top-style:none; border-left-style:none; border-bottom-style:none; border-right-style:none; padding-left:0in; padding-right:0.075in; padding-top:0in; padding-bottom:0in; writing-mode:horizontal-tb; direction:ltr; }
.Table1_B2 { border-top-style:none; border-left-style:none; border-bottom-style:none; border-right-style:none; vertical-align:middle; padding-left:0in; padding-right:0.075in; padding-top:0in; padding-bottom:0in; writing-mode:horizontal-tb; direction:ltr; }
.Table1_A { width:4.2326in; }
.Table1_B { width:0.1938in; }
.Table1_C { width:2.9535in; }
.Internet_20_link { color:#0563c1; text-decoration:underline; }
.ListLabel_20_1 { letter-spacing:-0.0071in; }
.ListLabel_20_4 { color:#000000; letter-spacing:normal; font-style:normal; text-decoration:none ! important; font-weight:normal; display:true; }
.T1 { color:#304452; font-family:'AvenirNext LT Pro Regular'; font-size:9pt; font-weight:bold; }
.T10 { font-family:'AvenirNext LT Pro Regular'; font-size:8pt; }
.T11 { font-family:'AvenirNext LT Pro Regular'; font-size:8pt; font-weight:bold; background-color:#ffffff; }
.T2 { color:#304452; font-family:'AvenirNext LT Pro Regular'; font-size:9pt; font-weight:bold; }
.T4 { color:#304452; font-family:'Times New Roman'; font-size:12pt; }
.T5 { color:#666666; font-family:'AvenirNext LT Pro Regular'; font-size:9pt; }
.T6 { color:#666666; font-family:'AvenirNext LT Pro Regular'; font-size:8pt; background-color:#ffffff; }
.T7 { color:#999999; font-family:'AvenirNext LT Pro Regular'; font-size:9pt; }
.T8 { font-family:'AvenirNext LT Pro Regular'; font-size:9pt; }
.T9 { font-family:'AvenirNext LT Pro Regular'; font-size:9pt; }
/* ODF styles with no properties representable as CSS:
.dp1 .Table1.1 .Table1.2 .Table1.3 .Table1.4 .ListLabel_20_2 .ListLabel_20_3 .ListLabel_20_5 .ListLabel_20_6 .ListLabel_20_7 .ListLabel_20_8 .ListLabel_20_9 { } */
</style>
<table border="0" cellspacing="0" cellpadding="0" class="Table1">
<colgroup><col width="470"><col width="22"><col width="328"></colgroup><tbody>
<tr class="Table11">
<td style="text-align:left;width:4.2326in; "
class="Table1_A1">
<p class="P4"><span class="T1">Jose Manuel Martínez García</span><span
class="T4"></span></p>
<p class="P5"><span class="T2">Coordinador de Sistemas</span><span
class="T2"></span></p>
<p class="P5"><span class="T2">Supercomputación de
Castilla y León</span><span class="T2"></span></p>
<p class="P5"><span class="T5">Tel: 987 293 174</span><span
class="T7"></span></p>
</td>
<td rowspan="2" style="text-align:left;width:0.1938in; "
class="Table1_A1">
<p class="P2"> </p>
</td>
<td rowspan="2" style="text-align:left;width:2.9535in; "
class="Table1_A1"><!--Next 'div' was a 'text:p'.-->
<div class="P6"><!--Next '
span' is a draw:frame.
--><span style="height:0.622in;width:2.8957in; padding:0; "
class="fr1" id="Imagen_9"><img
style="height:1.5799cm;width:7.3551cm;" alt=""
src="cid:part1.rqE2ZJeb.25BL9b87@scayle.es"></span><span
class="T8"></span></div>
<div
style="clear:both; line-height:0; width:0; height:0; margin:0; padding:0;"> </div>
</td>
</tr>
<tr class="Table12">
<td style="text-align:left;width:4.2326in; "
class="Table1_A1">
<p class="P5"><span class="T5">Edificio CRAI-TIC, Campus
de Vegazana, s/n Universidad de León - 24071 León,
España</span><span class="T5"></span></p>
</td>
</tr>
<tr class="Table13">
<td colspan="3" style="text-align:left;width:4.2326in; "
class="Table1_A1">
<div class="P1"><a href="https://www.scayle.es/"><!--Next '
span' is a draw:frame.
--><span style="height:0.2398in;width:7.3047in; padding:0; "
class="fr1" id="Imagen_11"><img
style="height:0.6091cm;width:18.5539cm;" alt=""
src="cid:part2.QEj8BLIf.1Muvhn7O@scayle.es"></span></a><span
class="T9"></span></div>
</td>
</tr>
<tr class="Table14">
<td colspan="3" style="text-align:left;width:4.2326in; "
class="Table1_B2">
<p class="P7"><span class="T6">Le informamos, como
destinatario de este mensaje, que el correo
electrónico y las comunicaciones por medio de Internet
no permiten asegurar ni garantizar la confidencialidad
de los mensajes transmitidos, así como tampoco su
integridad o su correcta recepción, por lo que SCAYLE
no asume responsabilidad alguna por tales
circunstancias. Si no consintiese en la utilización
del correo electrónico o de las comunicaciones vía
Internet le rogamos nos lo comunique y ponga en
nuestro conocimiento de manera inmediata. Para más
información visite nuestro </span><a
href="https://www.scayle.es/aviso-legal/"
class="Internet_20_link"><span
class="Internet_20_link"><span class="T11">Aviso
Legal</span></span></a><span class="T6">.</span><span
class="T10"></span></p>
</td>
</tr>
</tbody>
</table>
<p class="Standard"> </p>
</div>
</body>
</html>