<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
There is a bug in the grant accounting that leaks under certain operations (maybe O_DIRECT?). It is resolved by unmounting and remounting the clients, and/or upgrading. There was a thread about it on lustre-discuss a couple of years ago.
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 20, 2023, at 09:32, Jon Marshall via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="elementToProof ContentPasted0" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
Sorry, typo in the version number - the version we are actually running is 2.12.6<br class="ContentPasted0">
</div>
<div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
</div>
<hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 1288.6875px;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span>
<div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>Jon Marshall<br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>20 June 2023 16:18<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><span class="Apple-converted-space"> </span><<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>No space left on device MDT DoM but not full nor run out of inodes</font>
<div class=""> </div>
</div>
<div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div class="x_elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
Hi,</div>
<div class="x_elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br class="">
</div>
<div class="x_elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
We've been running lustre 2.15.1 in production for over a year and recently decided to enable PFL with DoM on our filesystem. Things have been fine up until last week, when users started reporting issues copying files, specifically "No space left on device".
The MDT is running ldiskfs as the backend.</div>
<div class="x_elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br class="">
</div>
<div class="x_elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
I've searched through the mailing list and found a couple of people reporting similar problems, which prompted me to check the inode allocation, which is currently:</div>
<div class="x_elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br class="">
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
UUID Inodes IUsed IFree IUse% Mounted on
<div class="x_ContentPasted0">scratchc-MDT0000_UUID 624492544 71144384 553348160 12% /mnt/scratchc[MDT:0]</div>
<div class="x_ContentPasted0">scratchc-OST0000_UUID 57712579 24489934 33222645 43% /mnt/scratchc[OST:0]</div>
<div class="x_ContentPasted0">scratchc-OST0001_UUID 57114064 24505876 32608188 43% /mnt/scratchc[OST:1]</div>
<div class=""><br class="x_ContentPasted0">
</div>
<div class="x_ContentPasted0">filesystem_summary: 136975217 71144384 65830833 52% /mnt/scratchc</div>
<br class="">
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
So, nowhere near full - the disk usage is a little higher:</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br class="">
</div>
<div class="x_elementToProof x_ContentPasted1 x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
UUID bytes Used Available Use% Mounted on
<div class="x_ContentPasted1">scratchc-MDT0000_UUID 882.1G 451.9G 355.8G 56% /mnt/scratchc[MDT:0]</div>
<div class="x_ContentPasted1">scratchc-OST0000_UUID 53.6T 22.7T 31.0T 43% /mnt/scratchc[OST:0]</div>
<div class="x_ContentPasted1">scratchc-OST0001_UUID 53.6T 23.0T 30.6T 43% /mnt/scratchc[OST:1]</div>
<div class=""><br class="x_ContentPasted1">
</div>
<div class="x_ContentPasted1">filesystem_summary: 107.3T 45.7T 61.6T 43% /mnt/scratchc</div>
<br class="">
</div>
<div class="x_elementToProof x_ContentPasted1 x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
But not full either! The errors are accompanied in the logs by:</div>
<div class="x_elementToProof x_ContentPasted1 x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br class="">
</div>
<div class="x_ContentPasted0 x_elementToProof x_ContentPasted1 x_ContentPasted2" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
LustreError: 15450:0:(tgt_grant.c:463:tgt_grant_space_left()) scratchc-MDT0000: cli ba0195c7-1ab4-4f7c-9e28-8689478f5c17/ffff9e331e231c00 left 82586337280 < tot_grant 82586681321 unstable 0 pending 0 dirty 1044480<br class="">
</div>
<div class="x_ContentPasted0 x_elementToProof x_ContentPasted3 x_ContentPasted1 x_ContentPasted2" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
LustreError: 15450:0:(tgt_grant.c:463:tgt_grant_space_left()) Skipped 33050 previous similar messages<br class="">
</div>
<div class="x_ContentPasted0 x_elementToProof x_ContentPasted3 x_ContentPasted1 x_ContentPasted2" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br class="">
</div>
<div class="x_ContentPasted0 x_elementToProof x_ContentPasted3 x_ContentPasted1 x_ContentPasted2" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
For reference the DoM striping we're using is:</div>
<div class="x_ContentPasted0 x_elementToProof x_ContentPasted3 x_ContentPasted1 x_ContentPasted2" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br class="">
</div>
<div class="x_ContentPasted0 x_elementToProof x_ContentPasted3 x_ContentPasted1 x_ContentPasted4 x_ContentPasted2" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
lcm_layout_gen: 0
<div class="x_ContentPasted4"> lcm_mirror_count: 1</div>
<div class="x_ContentPasted4"> lcm_entry_count: 3</div>
<div class="x_ContentPasted4"> lcme_id: N/A</div>
<div class="x_ContentPasted4"> lcme_mirror_id: N/A</div>
<div class="x_ContentPasted4"> lcme_flags: 0</div>
<div class="x_ContentPasted4"> lcme_extent.e_start: 0</div>
<div class="x_ContentPasted4"> lcme_extent.e_end: 1048576</div>
<div class="x_ContentPasted4"> stripe_count: 0 stripe_size: 1048576 pattern: mdt stripe_offset: -1</div>
<div class=""><br class="x_ContentPasted4">
</div>
<div class="x_ContentPasted4"> lcme_id: N/A</div>
<div class="x_ContentPasted4"> lcme_mirror_id: N/A</div>
<div class="x_ContentPasted4"> lcme_flags: 0</div>
<div class="x_ContentPasted4"> lcme_extent.e_start: 1048576</div>
<div class="x_ContentPasted4"> lcme_extent.e_end: 1073741824</div>
<div class="x_ContentPasted4"> stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1</div>
<div class=""><br class="x_ContentPasted4">
</div>
<div class="x_ContentPasted4"> lcme_id: N/A</div>
<div class="x_ContentPasted4"> lcme_mirror_id: N/A</div>
<div class="x_ContentPasted4"> lcme_flags: 0</div>
<div class="x_ContentPasted4"> lcme_extent.e_start: 1073741824</div>
<div class="x_ContentPasted4"> lcme_extent.e_end: EOF</div>
stripe_count: -1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1<br class="">
</div>
<div class="x_ContentPasted0 x_elementToProof x_ContentPasted3 x_ContentPasted1 x_ContentPasted4 x_ContentPasted2" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br class="">
</div>
<div class="x_ContentPasted0 x_elementToProof x_ContentPasted3 x_ContentPasted1 x_ContentPasted4 x_ContentPasted2" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
So the first 1MB on the MDT.</div>
<div class="x_ContentPasted0 x_elementToProof x_ContentPasted3 x_ContentPasted1 x_ContentPasted4 x_ContentPasted2" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br class="">
</div>
<div class="x_ContentPasted0 x_elementToProof x_ContentPasted3 x_ContentPasted1 x_ContentPasted4 x_ContentPasted2" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
My question is obviously what is causing these errors? I'm not massively familiar with Lustre internals, so any pointers on where to look would be greatly appreciated!</div>
<div class="x_ContentPasted0 x_elementToProof x_ContentPasted3 x_ContentPasted1 x_ContentPasted4 x_ContentPasted2" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br class="">
</div>
<div class="x_ContentPasted0 x_elementToProof x_ContentPasted3 x_ContentPasted1 x_ContentPasted4 x_ContentPasted2" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
Cheers</div>
<div class="x_ContentPasted0 x_elementToProof x_ContentPasted3 x_ContentPasted1 x_ContentPasted4 x_ContentPasted2" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
Jon</div>
<div class="x_elementToProof">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div id="x_Signature" class="">
<div class="">
<div class=""></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<div style="margin: 0px; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(89, 89, 89);" class="">Jon Marshall</span></div>
<div style="margin: 0px; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; color: rgb(89, 89, 89);" class="">High Performance Computing Specialist<br class="">
</span></div>
<p style="margin: 0px; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; color: rgb(89, 89, 89);" class=""> </span></p>
<div style="margin: 0px; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(89, 89, 89);" class="">IT and Scientific Computing Team</span></b><span style="font-size: 10pt; color: rgb(89, 89, 89);" class=""></span></div>
<p style="margin: 0px; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; color: rgb(89, 89, 89);" class=""> </span></p>
<div style="margin: 0px; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; color: rgb(89, 89, 89);" class="">Cancer Research UK Cambridge Institute</span></div>
<div style="margin: 0px; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; color: rgb(89, 89, 89);" class="">Li Ka Shing Centre | Robinson Way | Cambridge | CB2 0RE</span></div>
<div style="margin: 0px; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class=""><a href="http://www.cruk.cam.ac.uk/" target="_blank" rel="noopener noreferrer" data-loopstyle="linkonly" class=""><span style="font-size: 10pt; color: rgb(0, 112, 192);" class="">Web</span></a></span><span style="font-size: 10pt; color: rgb(0, 112, 192);" class=""><span class="Apple-converted-space"> </span>|<span class="Apple-converted-space"> </span></span><span class=""><a href="http://www.facebook.com/cancerresearchuk" target="_blank" rel="noopener noreferrer" data-loopstyle="linkonly" class=""><span style="font-size: 10pt; color: rgb(0, 112, 192);" class="">Facebook</span></a></span><span style="font-size: 10pt; color: rgb(0, 112, 192);" class=""><span class="Apple-converted-space"> </span>|<span class="Apple-converted-space"> </span></span><span class=""><a href="http://twitter.com/CR_UK" target="_blank" rel="noopener noreferrer" data-loopstyle="linkonly" class=""><span style="font-size: 10pt; color: rgb(0, 112, 192);" class="">Twitter</span></a></span></div>
<p style="margin: 0px; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; color: rgb(0, 112, 192);" class=""> </span></p>
<div style="margin: 0px; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<a href="http://www.cruk.cam.ac.uk/" target="_blank" rel="noopener noreferrer" class=""><span style="font-size: 10.5pt; font-family: "Times New Roman", serif, serif, EmojiFont; color: blue;" class=""><img class="x_EmojiInsert" alt="Description: CRI Logo" data-outlook-trace="F:0|T:2" src="cid:image001.jpg@01D4CCEA.783A5790" style="width: 176.99pt; height: 38.99pt;"></span></a></div>
<br class="">
</div>
</div>
</div>
</div>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">lustre-discuss
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="mailto:lustre-discuss@lists.lustre.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">lustre-discuss@lists.lustre.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>