<div dir="ltr">Thanks Diego, long time no see! I haven't been using NRS TBF.<div><br></div><div>I think there's a few problems, some of which we were aware of before, but the lack of lock cancels was causing chaos.</div><div><br></div><div>* (Mark lustre_inode_cache as reclaimable) <a href="https://jira.whamcloud.com/browse/LU-12313">https://jira.whamcloud.com/browse/LU-12313</a></div><div>* tested on a 2.12.3 client (without patch above), but we were actually getting lock cancels now</div><div><br></div><div>So I think I'll join 2020 and run 2.12.3 and probably add the SUnreclaim patch to that as well, as it seems simple enough.</div><div><br></div><div>Thank you!</div><div><br></div><div>~Steve</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 6, 2020 at 2:33 AM Moreno Diego (ID SIS) <<a href="mailto:diego.moreno@id.ethz.ch">diego.moreno@id.ethz.ch</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-7304915406463555125WordSection1">
<p class="MsoNormal">Hi Steve,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I was having a similar problem in the past months where the MDS servers would go OOM because of SlabUnreclaim. The root cause has not yet been found but we stopped seeing this the day we disabled the NRS TBF (QoS) for any LDLM service (just
in case you have it enabled). Just in case you have it enabled. It would be good to check as well what’s being consumed in the slab cache. In our case it was mostly kernel objects and not ldlm.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><span style="color:black">Diego</span><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal" style="margin-left:36pt"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of Steve Crusan <<a href="mailto:stevec@dug.com" target="_blank">stevec@dug.com</a>><br>
<b>Date: </b>Thursday, 2 January 2020 at 20:25<br>
<b>To: </b>"<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
<b>Subject: </b>[lustre-discuss] LDLM locks not expiring/cancelling<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt">Hi all, <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:36pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt">We are running into a bizarre situation where we aren't having stale locks cancel themselves, and even worse, it seems as if ldlm.namespaces.*.lru_size is being ignored.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt">For instance, I unmount our Lustre file systems on a client machine, then remount. Next, I'll run "lctl set_param ldlm.namespaces.*.lru_max_age=60s, lctl set_param ldlm.namespaces.*.lru_size=1024". This (I believe)
theoretically would only allow 1024 ldlm locks per osc, and then I'd see a lot of lock cancels (via ldlm.namespaces.${ost}.pool.stats). We also should see cancels if the grant time > lru_max_age.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt">We can trigger this simply by running 'find' on the root of our Lustre file system, and waiting for awhile. Eventually the clients SUnreclaim value bloats to 60-70GB (!!!), and each of our OSTs have 30-40k LRU
locks (via lock_count). This is early in the process:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt">"""<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt">ldlm.namespaces.h5-OST003f-osc-ffff8802d8559000.lock_count=2090<br>
ldlm.namespaces.h5-OST0040-osc-ffff8802d8559000.lock_count=2127<br>
ldlm.namespaces.h5-OST0047-osc-ffff8802d8559000.lock_count=52<br>
ldlm.namespaces.h5-OST0048-osc-ffff8802d8559000.lock_count=1962<br>
ldlm.namespaces.h5-OST0049-osc-ffff8802d8559000.lock_count=1247<br>
ldlm.namespaces.h5-OST004a-osc-ffff8802d8559000.lock_count=1642<br>
ldlm.namespaces.h5-OST004b-osc-ffff8802d8559000.lock_count=1340<br>
ldlm.namespaces.h5-OST004c-osc-ffff8802d8559000.lock_count=1208<br>
ldlm.namespaces.h5-OST004d-osc-ffff8802d8559000.lock_count=1422<br>
ldlm.namespaces.h5-OST004e-osc-ffff8802d8559000.lock_count=1244<br>
ldlm.namespaces.h5-OST004f-osc-ffff8802d8559000.lock_count=1117<br>
ldlm.namespaces.h5-OST0050-osc-ffff8802d8559000.lock_count=1165<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt">"""<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt">But this will grow over time, and eventually this compute node gets evicted from the MDS (after 10 minutes of cancelling locks/hanging). The only way we have been able to reduce the slab usage is to drop caches
and set LRU=clear...but the problem just comes back depending on the workload.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt">We are running 2.10.3 client side, 2.10.1 server side. Have there been any fixes added into the codebase for 2.10 that we need to apply? This seems to be the closest to what we are experiencing:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt"><a href="https://jira.whamcloud.com/browse/LU-11518" target="_blank">https://jira.whamcloud.com/browse/LU-11518</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt"> <br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:36pt">PS: I've checked other systems across our cluster, and some of them have as many as 50k locks per OST. I am kind of wondering if these locks are staying around much longer than the lru_max_age default (65 minutes),
but I cannot prove that. Is there a good way to translate held locks to fids? I have been messing around with lctl set_param debug="XXX" and lctl set_param ldlm.namespaces.*.dump_namespace, but I don't feel like I'm getting *all* of the locks.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36pt"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-left:36pt">~Steve <u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><p style="line-height:1.44;margin-top:0pt;margin-bottom:0pt"><font color="#000000" face="Arial"><span style="font-size:13.3333px;white-space:pre-wrap"><b>Steve Crusan</b></span></font></p><p style="line-height:1.44;margin-top:0pt;margin-bottom:0pt"><font color="#000000" face="Arial"><span style="font-size:13.3333px;white-space:pre-wrap">Storage Specialist</span></font></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"> </p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"> </p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh4.googleusercontent.com/Q4KHsGT4WoIleJUBVa2XGPnpzVoA_Y7tryRXSNSrbOtcgRxd8-JxxBfGWLnil2ZCupCxSUihV8wGLl1rUn_71nT3xLC0CBUysFoJk1coZ9IT0TBSoPMuRiKxxwWyZyAKYEfOYyCH" width="118" height="73" style="border: none;"></span></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"> </p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(102,102,102);font-weight:700;vertical-align:baseline;white-space:pre-wrap">DownUnder GeoSolutions</span></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"> </p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap">16200 Park Row Drive, Suite 100</span></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap">Houston TX 77084, USA</span></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(102,102,102);font-weight:700;font-style:italic;vertical-align:baseline;white-space:pre-wrap">tel </span><span style="font-size:10pt;font-family:Arial;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap">+1 832 582 3221</span></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"><a href="mailto:stevec@dug.com" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:10pt;font-family:Arial;font-weight:700;vertical-align:baseline;white-space:pre-wrap">stevec@dug.com</span></a></p><p style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"><a href="http://www.dug.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:10pt;font-family:Arial;font-weight:700;vertical-align:baseline;white-space:pre-wrap">www.dug.com</span></a></p></div></div>