<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p data-start="0" data-end="5">Hi,</p>
<p data-start="7" data-end="70">I'm not sure if this could be
related to the following issue:</p>
<p data-start="72" data-end="246"><a rel="noopener" target="_new"
data-start="72" data-end="244"
href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2025-January/019372.html"
class="moz-txt-link-freetext">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2025-January/019372.html</a></p>
<p data-start="72" data-end="246"><br>
</p>
<p data-start="248" data-end="340">It appears to involve a similar
Lustre version, quota-related issues, and MDT instability.</p>
<p data-start="342" data-end="439" data-is-last-node=""
data-is-only-node="">In the referenced post, they reported that
disabling quotas stabilized the MDS for about a month.</p>
<p><br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">El 12/03/2025 a las 22:22, Fredrik
Nyström via lustre-discuss escribió:<br>
</div>
<blockquote type="cite"
cite="mid:44f38fde-9e1d-4e63-8a5e-75fbb79f4b20@nsc.liu.se">
<pre wrap="" class="moz-quote-pre">Hi,
We had some similar problems in Sep-Oct 2024 running Lustre 2.15.5.
Limits on individual OSTs stops increasing, leading to writes becoming
slower and slower?
Check for "DQACQ failed" in /var/log/messages on Lustre servers.
Example, lots of lines like these for all OSTs:
2024-09-04T12:57:15.725917+02:00 oss170 kernel: LustreError: 1059853:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -3, flags:0x1 qsd:rossby27-OST0003 qtype:grp id:8517 enforced:1 granted: 285682 pending:149320 waiting:13252405 req:1 usage: 85560 qunit:262144 qtune:65536 edquot:0 default:no
2024-09-04T12:57:15.726112+02:00 oss170 kernel: LustreError: 1059853:0:(qsd_handler.c:787:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:rossby27-OST0003 qtype:grp id:8517 enforced:1 granted: 285682 pending:149320 waiting:12591294 req:0 usage: 85560 qunit:262144 qtune:65536 edquot:0 default:no
2024-09-04T12:57:15.726138+02:00 oss170 kernel: LustreError: 1059853:0:(qsd_handler.c:787:qsd_op_begin0()) Skipped 20 previous similar messages
If I remember correctly, group quota problems only affected a single
group. Ok after restart of Lustre servers, unmount of MDT triggered
kernel panic.
Kind Regards / Fredrik Nyström, NSC
On 2025-03-11 16:12, Robert Pennington wrote:
</pre>
<blockquote type="cite">
<pre wrap="" class="moz-quote-pre">Hello,
We?re using Lustre 2.15.3 and have a strange problem with our attempt to impose quotas. Any assistance would be helpful.
This is a user who we?ve attempted to impose a group quota on - however, many of our OSTs (ignore the ones where quotactl failed) just don?t receive the quota information.
On the OSTs with limit=0k below, they have the following configuration, while the nodes like OST000f have correctly-updating information for limit_group:
$ sudo lctl get_param osd-*.*.quota_slave_dt.info
osd-ldiskfs.lustre-OST000e.quota_slave_dt.info=
target name: lustre-OST000e
pool ID: 0
type: dt
quota enabled: ugp
conn to master: setup
space acct: ugp
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
project uptodate: glb[0],slv[0],reint[0]
$ sudo lctl get_param osd-*.*.quota_slave_dt.limit_group
osd-ldiskfs.lustre-OST000e.quota_slave_dt.limit_group=
global_index_copy:
- id: 0
limits: { hard: 0, soft: 0, granted: 0, time: 0 }
?
# lfs quota -vhg 4055 /mnt/lustre/
Disk quotas for grp 4055 (gid 4055):
Filesystem used quota limit grace files quota limit grace
/mnt/lustre/ 2.694T* 0k 716.8G - 145961 0 0 -
lustre-MDT0000_UUID
76.81M - 1.075G - 145951 - 0 -
lustre-MDT0001_UUID
40k* - 40k - 10 - 0 -
quotactl ost0 failed.
lustre-OST0001_UUID
16.09G - 0k - - - - -
lustre-OST0002_UUID
251.7G - 0k - - - - -
quotactl ost3 failed.
quotactl ost4 failed.
quotactl ost5 failed.
quotactl ost6 failed.
lustre-OST0007_UUID
0k - 0k - - - - -
lustre-OST0008_UUID
525.1M* - 525.1M - - - - -
lustre-OST0009_UUID
540.7M* - 540.7M - - - - -
lustre-OST000a_UUID
385.8M* - 385.8M - - - - -
quotactl ost11 failed.
quotactl ost12 failed.
lustre-OST000d_UUID
191.9G - 0k - - - - -
lustre-OST000e_UUID
258.9G - 0k - - - - -
lustre-OST000f_UUID
86.99G - 87.99G - - - - -
lustre-OST0010_UUID
255.3G - 256.3G - - - - -
lustre-OST0011_UUID
254.1G - 0k - - - - -
lustre-OST0012_UUID
241.6G - 0k - - - - -
lustre-OST0013_UUID
241.6G - 0k - - - - -
lustre-OST0014_UUID
241.9G - 0k - - - - -
lustre-OST0015_UUID
237.4G - 0k - - - - -
lustre-OST0016_UUID
241.8G - 0k - - - - -
lustre-OST0017_UUID
237.8G - 0k - - - - -
lustre-OST0018_UUID
344.2M - 0k - - - - -
Total allocated inode limit: 0, total allocated block limit: 345.7G
Some errors happened when getting quota info. Some devices may be not working or deactivated. The data in "[]" is inaccurate.
?
Thank you for your time.
Sincerely,
Robert Pennington, PhD
Tuebingen AI Center, Universitaet Tuebingen
Maria von Linden Str. 6
72076 Tuebingen
Germany
Office number: 10-30/A15
</pre>
</blockquote>
<pre wrap="" class="moz-quote-pre">
</pre>
</blockquote>
<div class="moz-signature">-- <br>
<!--?xml version="1.0" encoding="UTF-8"?-->
<!--This file was converted to xhtml by LibreOffice - see https://cgit.freedesktop.org/libreoffice/core/tree/filter/source/xslt for the code.-->
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title xml:lang="en-US
">- no title specified</title>
<meta name="DCTERMS.title" content="" xml:lang="en-US
">
<meta name="DCTERMS.language" content="en-US
" scheme="DCTERMS.RFC4646">
<meta name="DCTERMS.source"
content="http://xml.openoffice.org/odf2xhtml">
<meta name="DCTERMS.issued" content="2024-07-04T11:24:00"
scheme="DCTERMS.W3CDTF">
<meta name="DCTERMS.modified" content="2024-07-04T11:24:00"
scheme="DCTERMS.W3CDTF">
<meta name="DCTERMS.provenance" content="
" xml:lang="en-US
">
<meta name="xsl:vendor" content="libxslt">
<link rel="schema.DC" href="http://purl.org/dc/elements/1.1/"
hreflang="en">
<link rel="schema.DCTERMS" href="http://purl.org/dc/terms/"
hreflang="en">
<link rel="schema.DCTYPE" href="http://purl.org/dc/dcmitype/"
hreflang="en">
<link rel="schema.DCAM" href="http://purl.org/dc/dcam/"
hreflang="en">
<style>
table { border-collapse:collapse; border-spacing:0; empty-cells:show }
td, th { vertical-align:top; font-size:12pt;}
h1, h2, h3, h4, h5, h6 { clear:both;}
ol, ul { margin:0; padding:0;}
li { list-style: none; margin:0; padding:0;}
span.footnodeNumber { padding-right:1em; }
span.annotation_style_by_filter { font-size:95%; font-family:Arial; background-color:#fff000; margin:0; border:0; padding:0; }
span.heading_numbering { margin-right: 0.8rem; }* { margin:0;}
.fr1 { font-size:11pt; font-family:Calibri; text-align:center; vertical-align:top; writing-mode:horizontal-tb; direction:ltr; border-top-style:none; border-left-style:none; border-bottom-style:none; border-right-style:none; margin-left:0in; margin-right:0in; margin-top:0in; margin-bottom:0in; background-color:transparent; padding:0in; }
.P1 { font-size:12pt; line-height:100%; margin-bottom:0in; margin-top:0in; text-align:left ! important; font-family:'Times New Roman'; writing-mode:horizontal-tb; direction:ltr; }
.P2 { font-size:9pt; line-height:100%; margin-bottom:0in; margin-top:0in; text-align:left ! important; font-family:'AvenirNext LT Pro Regular'; writing-mode:horizontal-tb; direction:ltr; color:#999999; }
.P4 { font-size:11pt; line-height:100%; margin-bottom:0in; margin-top:0in; text-align:left ! important; font-family:Calibri; writing-mode:horizontal-tb; direction:ltr; }
.P5 { font-size:11pt; line-height:100%; margin-bottom:0in; margin-top:0in; text-align:left ! important; font-family:Calibri; writing-mode:horizontal-tb; direction:ltr; }
.P6 { font-size:11pt; line-height:100%; margin-bottom:0in; margin-top:0in; text-align:left ! important; font-family:Calibri; writing-mode:horizontal-tb; direction:ltr; }
.P7 { font-size:11pt; line-height:100%; margin-bottom:0in; margin-top:0in; text-align:justify ! important; font-family:Calibri; writing-mode:horizontal-tb; direction:ltr; }
.Standard { font-size:11pt; font-family:Calibri; writing-mode:horizontal-tb; direction:ltr; margin-top:0in; margin-bottom:0.111in; line-height:108%; text-align:left ! important; }
.Table1 { width:7.3799in; margin-left:0in; margin-top:0in; margin-bottom:0in; margin-right:auto;writing-mode:horizontal-tb; direction:ltr; }
.Table1_A1 { border-top-style:none; border-left-style:none; border-bottom-style:none; border-right-style:none; padding-left:0in; padding-right:0.075in; padding-top:0in; padding-bottom:0in; writing-mode:horizontal-tb; direction:ltr; }
.Table1_B2 { border-top-style:none; border-left-style:none; border-bottom-style:none; border-right-style:none; vertical-align:middle; padding-left:0in; padding-right:0.075in; padding-top:0in; padding-bottom:0in; writing-mode:horizontal-tb; direction:ltr; }
.Table1_A { width:4.2326in; }
.Table1_B { width:0.1938in; }
.Table1_C { width:2.9535in; }
.Internet_20_link { color:#0563c1; text-decoration:underline; }
.ListLabel_20_1 { letter-spacing:-0.0071in; }
.ListLabel_20_4 { color:#000000; letter-spacing:normal; font-style:normal; text-decoration:none ! important; font-weight:normal; display:true; }
.T1 { color:#304452; font-family:'AvenirNext LT Pro Regular'; font-size:9pt; font-weight:bold; }
.T10 { font-family:'AvenirNext LT Pro Regular'; font-size:8pt; }
.T11 { font-family:'AvenirNext LT Pro Regular'; font-size:8pt; font-weight:bold; background-color:#ffffff; }
.T2 { color:#304452; font-family:'AvenirNext LT Pro Regular'; font-size:9pt; font-weight:bold; }
.T4 { color:#304452; font-family:'Times New Roman'; font-size:12pt; }
.T5 { color:#666666; font-family:'AvenirNext LT Pro Regular'; font-size:9pt; }
.T6 { color:#666666; font-family:'AvenirNext LT Pro Regular'; font-size:8pt; background-color:#ffffff; }
.T7 { color:#999999; font-family:'AvenirNext LT Pro Regular'; font-size:9pt; }
.T8 { font-family:'AvenirNext LT Pro Regular'; font-size:9pt; }
.T9 { font-family:'AvenirNext LT Pro Regular'; font-size:9pt; }
/* ODF styles with no properties representable as CSS:
.dp1 .Table1.1 .Table1.2 .Table1.3 .Table1.4 .ListLabel_20_2 .ListLabel_20_3 .ListLabel_20_5 .ListLabel_20_6 .ListLabel_20_7 .ListLabel_20_8 .ListLabel_20_9 { } */
</style>
<table border="0" cellspacing="0" cellpadding="0" class="Table1">
<colgroup><col width="470"><col width="22"><col width="328"></colgroup><tbody>
<tr class="Table11">
<td style="text-align:left;width:4.2326in; "
class="Table1_A1">
<p class="P4"><span class="T1">Jose Manuel Martínez García</span><span
class="T4"></span></p>
<p class="P5"><span class="T2">Coordinador de Sistemas</span><span
class="T2"></span></p>
<p class="P5"><span class="T2">Supercomputación de
Castilla y León</span><span class="T2"></span></p>
<p class="P5"><span class="T5">Tel: 987 293 174</span><span
class="T7"></span></p>
</td>
<td rowspan="2" style="text-align:left;width:0.1938in; "
class="Table1_A1">
<p class="P2"> </p>
</td>
<td rowspan="2" style="text-align:left;width:2.9535in; "
class="Table1_A1"><!--Next 'div' was a 'text:p'.-->
<div class="P6"><!--Next '
span' is a draw:frame.
--><span style="height:0.622in;width:2.8957in; padding:0; "
class="fr1" id="Imagen_9"><img
style="height:1.5799cm;width:7.3551cm;" alt=""
src="cid:part1.Hz6sUaxL.IZ5Wc6BH@scayle.es"></span><span
class="T8"></span></div>
<div
style="clear:both; line-height:0; width:0; height:0; margin:0; padding:0;"> </div>
</td>
</tr>
<tr class="Table12">
<td style="text-align:left;width:4.2326in; "
class="Table1_A1">
<p class="P5"><span class="T5">Edificio CRAI-TIC, Campus
de Vegazana, s/n Universidad de León - 24071 León,
España</span><span class="T5"></span></p>
</td>
</tr>
<tr class="Table13">
<td colspan="3" style="text-align:left;width:4.2326in; "
class="Table1_A1">
<div class="P1"><a href="https://www.scayle.es/"><!--Next '
span' is a draw:frame.
--><span style="height:0.2398in;width:7.3047in; padding:0; "
class="fr1" id="Imagen_11"><img
style="height:0.6091cm;width:18.5539cm;" alt=""
src="cid:part2.0zfgMwwx.aDJuuqvK@scayle.es"></span></a><span
class="T9"></span></div>
</td>
</tr>
<tr class="Table14">
<td colspan="3" style="text-align:left;width:4.2326in; "
class="Table1_B2">
<p class="P7"><span class="T6">Le informamos, como
destinatario de este mensaje, que el correo
electrónico y las comunicaciones por medio de Internet
no permiten asegurar ni garantizar la confidencialidad
de los mensajes transmitidos, así como tampoco su
integridad o su correcta recepción, por lo que SCAYLE
no asume responsabilidad alguna por tales
circunstancias. Si no consintiese en la utilización
del correo electrónico o de las comunicaciones vía
Internet le rogamos nos lo comunique y ponga en
nuestro conocimiento de manera inmediata. Para más
información visite nuestro </span><a
href="https://www.scayle.es/aviso-legal/"
class="Internet_20_link"><span
class="Internet_20_link"><span class="T11">Aviso
Legal</span></span></a><span class="T6">.</span><span
class="T10"></span></p>
</td>
</tr>
</tbody>
</table>
<p class="Standard"> </p>
</div>
</body>
</html>