<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi Andreas,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thanks for getting back to me - we are in the process of expanding the storage on this filesystem so I think I'll be pushing for an upgrade instead!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Cheers</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Jon</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Andreas Dilger <adilger@whamcloud.com><br>
<b>Sent:</b> 22 June 2023 20:00<br>
<b>To:</b> Jon Marshall <Jon.Marshall@cruk.cam.ac.uk><br>
<b>Cc:</b> lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> Re: [lustre-discuss] No space left on device MDT DoM but not full nor run out of inodes</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">There is a bug in the grant accounting that leaks under certain operations (maybe O_DIRECT?). It is resolved by unmounting and remounting the clients, and/or upgrading. There was a thread
about it on lustre-discuss a couple of years ago.
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 20, 2023, at 09:32, Jon Marshall via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div class="x_elementToProof x_ContentPasted0" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
Sorry, typo in the version number - the version we are actually running is 2.12.6<br class="x_ContentPasted0">
</div>
<div id="x_appendonsend" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
</div>
<hr tabindex="-1" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; display:inline-block; width:1288.6875px">
<span class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important"></span>
<div id="x_divRplyFwdMsg" dir="ltr" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font face="Calibri, sans-serif" class="" style="font-size:11pt"><b class="">From:</b><span class="x_Apple-converted-space"> </span>Jon Marshall<br class="">
<b class="">Sent:</b><span class="x_Apple-converted-space"> </span>20 June 2023 16:18<br class="">
<b class="">To:</b><span class="x_Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><span class="x_Apple-converted-space"> </span><<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:</b><span class="x_Apple-converted-space"> </span>No space left on device MDT DoM but not full nor run out of inodes</font>
<div class=""> </div>
</div>
<div dir="ltr" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
Hi,</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
We've been running lustre 2.15.1 in production for over a year and recently decided to enable PFL with DoM on our filesystem. Things have been fine up until last week, when users started reporting issues copying files, specifically "No space left on device".
The MDT is running ldiskfs as the backend.</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
I've searched through the mailing list and found a couple of people reporting similar problems, which prompted me to check the inode allocation, which is currently:</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_x_elementToProof x_x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
UUID Inodes IUsed IFree IUse% Mounted on
<div class="x_x_ContentPasted0">scratchc-MDT0000_UUID 624492544 71144384 553348160 12% /mnt/scratchc[MDT:0]</div>
<div class="x_x_ContentPasted0">scratchc-OST0000_UUID 57712579 24489934 33222645 43% /mnt/scratchc[OST:0]</div>
<div class="x_x_ContentPasted0">scratchc-OST0001_UUID 57114064 24505876 32608188 43% /mnt/scratchc[OST:1]</div>
<div class=""><br class="x_x_ContentPasted0">
</div>
<div class="x_x_ContentPasted0">filesystem_summary: 136975217 71144384 65830833 52% /mnt/scratchc</div>
<br class="">
</div>
<div class="x_x_elementToProof x_x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
So, nowhere near full - the disk usage is a little higher:</div>
<div class="x_x_elementToProof x_x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_x_elementToProof x_x_ContentPasted1 x_x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
UUID bytes Used Available Use% Mounted on
<div class="x_x_ContentPasted1">scratchc-MDT0000_UUID 882.1G 451.9G 355.8G 56% /mnt/scratchc[MDT:0]</div>
<div class="x_x_ContentPasted1">scratchc-OST0000_UUID 53.6T 22.7T 31.0T 43% /mnt/scratchc[OST:0]</div>
<div class="x_x_ContentPasted1">scratchc-OST0001_UUID 53.6T 23.0T 30.6T 43% /mnt/scratchc[OST:1]</div>
<div class=""><br class="x_x_ContentPasted1">
</div>
<div class="x_x_ContentPasted1">filesystem_summary: 107.3T 45.7T 61.6T 43% /mnt/scratchc</div>
<br class="">
</div>
<div class="x_x_elementToProof x_x_ContentPasted1 x_x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
But not full either! The errors are accompanied in the logs by:</div>
<div class="x_x_elementToProof x_x_ContentPasted1 x_x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_x_ContentPasted0 x_x_elementToProof x_x_ContentPasted1 x_x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
LustreError: 15450:0:(tgt_grant.c:463:tgt_grant_space_left()) scratchc-MDT0000: cli ba0195c7-1ab4-4f7c-9e28-8689478f5c17/ffff9e331e231c00 left 82586337280 < tot_grant 82586681321 unstable 0 pending 0 dirty 1044480<br class="">
</div>
<div class="x_x_ContentPasted0 x_x_elementToProof x_x_ContentPasted3 x_x_ContentPasted1 x_x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
LustreError: 15450:0:(tgt_grant.c:463:tgt_grant_space_left()) Skipped 33050 previous similar messages<br class="">
</div>
<div class="x_x_ContentPasted0 x_x_elementToProof x_x_ContentPasted3 x_x_ContentPasted1 x_x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_x_ContentPasted0 x_x_elementToProof x_x_ContentPasted3 x_x_ContentPasted1 x_x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
For reference the DoM striping we're using is:</div>
<div class="x_x_ContentPasted0 x_x_elementToProof x_x_ContentPasted3 x_x_ContentPasted1 x_x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_x_ContentPasted0 x_x_elementToProof x_x_ContentPasted3 x_x_ContentPasted1 x_x_ContentPasted4 x_x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
lcm_layout_gen: 0
<div class="x_x_ContentPasted4"> lcm_mirror_count: 1</div>
<div class="x_x_ContentPasted4"> lcm_entry_count: 3</div>
<div class="x_x_ContentPasted4"> lcme_id: N/A</div>
<div class="x_x_ContentPasted4"> lcme_mirror_id: N/A</div>
<div class="x_x_ContentPasted4"> lcme_flags: 0</div>
<div class="x_x_ContentPasted4"> lcme_extent.e_start: 0</div>
<div class="x_x_ContentPasted4"> lcme_extent.e_end: 1048576</div>
<div class="x_x_ContentPasted4"> stripe_count: 0 stripe_size: 1048576 pattern: mdt stripe_offset: -1</div>
<div class=""><br class="x_x_ContentPasted4">
</div>
<div class="x_x_ContentPasted4"> lcme_id: N/A</div>
<div class="x_x_ContentPasted4"> lcme_mirror_id: N/A</div>
<div class="x_x_ContentPasted4"> lcme_flags: 0</div>
<div class="x_x_ContentPasted4"> lcme_extent.e_start: 1048576</div>
<div class="x_x_ContentPasted4"> lcme_extent.e_end: 1073741824</div>
<div class="x_x_ContentPasted4"> stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1</div>
<div class=""><br class="x_x_ContentPasted4">
</div>
<div class="x_x_ContentPasted4"> lcme_id: N/A</div>
<div class="x_x_ContentPasted4"> lcme_mirror_id: N/A</div>
<div class="x_x_ContentPasted4"> lcme_flags: 0</div>
<div class="x_x_ContentPasted4"> lcme_extent.e_start: 1073741824</div>
<div class="x_x_ContentPasted4"> lcme_extent.e_end: EOF</div>
stripe_count: -1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1<br class="">
</div>
<div class="x_x_ContentPasted0 x_x_elementToProof x_x_ContentPasted3 x_x_ContentPasted1 x_x_ContentPasted4 x_x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_x_ContentPasted0 x_x_elementToProof x_x_ContentPasted3 x_x_ContentPasted1 x_x_ContentPasted4 x_x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
So the first 1MB on the MDT.</div>
<div class="x_x_ContentPasted0 x_x_elementToProof x_x_ContentPasted3 x_x_ContentPasted1 x_x_ContentPasted4 x_x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_x_ContentPasted0 x_x_elementToProof x_x_ContentPasted3 x_x_ContentPasted1 x_x_ContentPasted4 x_x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
My question is obviously what is causing these errors? I'm not massively familiar with Lustre internals, so any pointers on where to look would be greatly appreciated!</div>
<div class="x_x_ContentPasted0 x_x_elementToProof x_x_ContentPasted3 x_x_ContentPasted1 x_x_ContentPasted4 x_x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_x_ContentPasted0 x_x_elementToProof x_x_ContentPasted3 x_x_ContentPasted1 x_x_ContentPasted4 x_x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
Cheers</div>
<div class="x_x_ContentPasted0 x_x_elementToProof x_x_ContentPasted3 x_x_ContentPasted1 x_x_ContentPasted4 x_x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
Jon</div>
<div class="x_x_elementToProof">
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div id="x_x_Signature" class="">
<div class="">
<div class=""></div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<div class="" style="margin:0px; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="color:rgb(89,89,89)">Jon Marshall</span></div>
<div class="" style="margin:0px; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:10pt; color:rgb(89,89,89)">High Performance Computing Specialist<br class="">
</span></div>
<p class="" style="margin:0px; font-size:11pt; font-family:Calibri,sans-serif"><span class="" style="font-size:10pt; color:rgb(89,89,89)"> </span></p>
<div class="" style="margin:0px; font-size:11pt; font-family:Calibri,sans-serif">
<b class=""><span class="" style="color:rgb(89,89,89)">IT and Scientific Computing Team</span></b><span class="" style="font-size:10pt; color:rgb(89,89,89)"></span></div>
<p class="" style="margin:0px; font-size:11pt; font-family:Calibri,sans-serif"><span class="" style="font-size:10pt; color:rgb(89,89,89)"> </span></p>
<div class="" style="margin:0px; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:10pt; color:rgb(89,89,89)">Cancer Research UK Cambridge Institute</span></div>
<div class="" style="margin:0px; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:10pt; color:rgb(89,89,89)">Li Ka Shing Centre | Robinson Way | Cambridge | CB2 0RE</span></div>
<div class="" style="margin:0px; font-size:11pt; font-family:Calibri,sans-serif">
<span class=""><a href="http://www.cruk.cam.ac.uk/" target="_blank" rel="noopener noreferrer" data-loopstyle="linkonly" class=""><span class="" style="font-size:10pt; color:rgb(0,112,192)">Web</span></a></span><span class="" style="font-size:10pt; color:rgb(0,112,192)"><span class="x_Apple-converted-space"> </span>|<span class="x_Apple-converted-space"> </span></span><span class=""><a href="http://www.facebook.com/cancerresearchuk" target="_blank" rel="noopener noreferrer" data-loopstyle="linkonly" class=""><span class="" style="font-size:10pt; color:rgb(0,112,192)">Facebook</span></a></span><span class="" style="font-size:10pt; color:rgb(0,112,192)"><span class="x_Apple-converted-space"> </span>|<span class="x_Apple-converted-space"> </span></span><span class=""><a href="http://twitter.com/CR_UK" target="_blank" rel="noopener noreferrer" data-loopstyle="linkonly" class=""><span class="" style="font-size:10pt; color:rgb(0,112,192)">Twitter</span></a></span></div>
<p class="" style="margin:0px; font-size:11pt; font-family:Calibri,sans-serif"><span class="" style="font-size:10pt; color:rgb(0,112,192)"> </span></p>
<div class="" style="margin:0px; font-size:11pt; font-family:Calibri,sans-serif">
<a href="http://www.cruk.cam.ac.uk/" target="_blank" rel="noopener noreferrer" class=""><span class="" style="font-size:10.5pt; font-family:"Times New Roman",serif,serif,EmojiFont; color:blue"><img class="x_x_EmojiInsert" alt="Description: CRI Logo" style="width:176.99pt; height:38.99pt" data-outlook-trace="F:0|T:3" src="cid:image001.jpg@01D4CCEA.783A5790"></span></a></div>
<br class="">
</div>
</div>
</div>
</div>
</div>
<span class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important">_______________________________________________</span><br class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<span class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important">lustre-discuss
mailing list</span><br class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<a href="mailto:lustre-discuss@lists.lustre.org" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px">lustre-discuss@lists.lustre.org</a><br class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="x_Apple-interchange-newline">
</div>
<br class="x_Apple-interchange-newline">
<br class="x_Apple-interchange-newline">
</div>
<br class="">
</div>
</div>
</body>
</html>