<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
Sorry, typo in the version number - the version we are actually running is 2.12.6<br class="ContentPasted0">
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Jon Marshall<br>
<b>Sent:</b> 20 June 2023 16:18<br>
<b>To:</b> lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> No space left on device MDT DoM but not full nor run out of inodes</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi,</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
We've been running lustre 2.15.1 in production for over a year and recently decided to enable PFL with DoM on our filesystem. Things have been fine up until last week, when users started reporting issues copying files, specifically "No space left on device".
The MDT is running ldiskfs as the backend.</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I've searched through the mailing list and found a couple of people reporting similar problems, which prompted me to check the inode allocation, which is currently:</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
UUID Inodes IUsed IFree IUse% Mounted on
<div class="x_ContentPasted0">scratchc-MDT0000_UUID 624492544 71144384 553348160 12% /mnt/scratchc[MDT:0]
</div>
<div class="x_ContentPasted0">scratchc-OST0000_UUID 57712579 24489934 33222645 43% /mnt/scratchc[OST:0]</div>
<div class="x_ContentPasted0">scratchc-OST0001_UUID 57114064 24505876 32608188 43% /mnt/scratchc[OST:1]</div>
<div><br class="x_ContentPasted0">
</div>
<div class="x_ContentPasted0">filesystem_summary: 136975217 71144384 65830833 52% /mnt/scratchc</div>
<br>
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
So, nowhere near full - the disk usage is a little higher:</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
UUID bytes Used Available Use% Mounted on
<div class="x_ContentPasted1">scratchc-MDT0000_UUID 882.1G 451.9G 355.8G 56% /mnt/scratchc[MDT:0]
</div>
<div class="x_ContentPasted1">scratchc-OST0000_UUID 53.6T 22.7T 31.0T 43% /mnt/scratchc[OST:0]</div>
<div class="x_ContentPasted1">scratchc-OST0001_UUID 53.6T 23.0T 30.6T 43% /mnt/scratchc[OST:1]</div>
<div><br class="x_ContentPasted1">
</div>
<div class="x_ContentPasted1">filesystem_summary: 107.3T 45.7T 61.6T 43% /mnt/scratchc</div>
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
But not full either! The errors are accompanied in the logs by:</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1 x_ContentPasted2" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
LustreError: 15450:0:(tgt_grant.c:463:tgt_grant_space_left()) scratchc-MDT0000: cli ba0195c7-1ab4-4f7c-9e28-8689478f5c17/ffff9e331e231c00 left 82586337280 < tot_grant 82586681321 unstable 0 pending 0 dirty 1044480<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1 x_ContentPasted2 x_ContentPasted3" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
LustreError: 15450:0:(tgt_grant.c:463:tgt_grant_space_left()) Skipped 33050 previous similar messages<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1 x_ContentPasted2 x_ContentPasted3" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1 x_ContentPasted2 x_ContentPasted3" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
For reference the DoM striping we're using is:</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1 x_ContentPasted2 x_ContentPasted3" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1 x_ContentPasted2 x_ContentPasted3 x_ContentPasted4" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
lcm_layout_gen: 0
<div class="x_ContentPasted4"> lcm_mirror_count: 1</div>
<div class="x_ContentPasted4"> lcm_entry_count: 3</div>
<div class="x_ContentPasted4"> lcme_id: N/A</div>
<div class="x_ContentPasted4"> lcme_mirror_id: N/A</div>
<div class="x_ContentPasted4"> lcme_flags: 0</div>
<div class="x_ContentPasted4"> lcme_extent.e_start: 0</div>
<div class="x_ContentPasted4"> lcme_extent.e_end: 1048576</div>
<div class="x_ContentPasted4"> stripe_count: 0 stripe_size: 1048576 pattern: mdt stripe_offset: -1</div>
<div><br class="x_ContentPasted4">
</div>
<div class="x_ContentPasted4"> lcme_id: N/A</div>
<div class="x_ContentPasted4"> lcme_mirror_id: N/A</div>
<div class="x_ContentPasted4"> lcme_flags: 0</div>
<div class="x_ContentPasted4"> lcme_extent.e_start: 1048576</div>
<div class="x_ContentPasted4"> lcme_extent.e_end: 1073741824</div>
<div class="x_ContentPasted4"> stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1</div>
<div><br class="x_ContentPasted4">
</div>
<div class="x_ContentPasted4"> lcme_id: N/A</div>
<div class="x_ContentPasted4"> lcme_mirror_id: N/A</div>
<div class="x_ContentPasted4"> lcme_flags: 0</div>
<div class="x_ContentPasted4"> lcme_extent.e_start: 1073741824</div>
<div class="x_ContentPasted4"> lcme_extent.e_end: EOF</div>
stripe_count: -1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1 x_ContentPasted2 x_ContentPasted3 x_ContentPasted4" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1 x_ContentPasted2 x_ContentPasted3 x_ContentPasted4" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
So the first 1MB on the MDT.</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1 x_ContentPasted2 x_ContentPasted3 x_ContentPasted4" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1 x_ContentPasted2 x_ContentPasted3 x_ContentPasted4" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
My question is obviously what is causing these errors? I'm not massively familiar with Lustre internals, so any pointers on where to look would be greatly appreciated!</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1 x_ContentPasted2 x_ContentPasted3 x_ContentPasted4" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1 x_ContentPasted2 x_ContentPasted3 x_ContentPasted4" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Cheers</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1 x_ContentPasted2 x_ContentPasted3 x_ContentPasted4" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Jon</div>
<div class="x_elementToProof">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_Signature">
<div>
<div></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="color:rgb(89,89,89)">Jon Marshall</span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="font-size:10pt; color:rgb(89,89,89)">High Performance Computing Specialist<br>
</span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="font-size:10pt; color:rgb(89,89,89)"> </span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><b><span style="color:rgb(89,89,89)">IT and Scientific Computing Team</span></b><span style="font-size:10pt; color:rgb(89,89,89)"></span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="font-size:10pt; color:rgb(89,89,89)"> </span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="font-size:10pt; color:rgb(89,89,89)">Cancer Research UK Cambridge Institute</span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="font-size:10pt; color:rgb(89,89,89)">Li Ka Shing Centre | Robinson Way | Cambridge | CB2 0RE</span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span><a href="http://www.cruk.cam.ac.uk/" target="_blank" rel="noopener noreferrer" data-loopstyle="linkonly"><span style="font-size:10pt; color:rgb(0,112,192)">Web</span></a></span><span style="font-size:10pt; color:rgb(0,112,192)">
| </span><span><a href="http://www.facebook.com/cancerresearchuk" target="_blank" rel="noopener noreferrer" data-loopstyle="linkonly"><span style="font-size:10pt; color:rgb(0,112,192)">Facebook</span></a></span><span style="font-size:10pt; color:rgb(0,112,192)">
| </span><span><a href="http://twitter.com/CR_UK" target="_blank" rel="noopener noreferrer" data-loopstyle="linkonly"><span style="font-size:10pt; color:rgb(0,112,192)">Twitter</span></a></span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="font-size:10pt; color:rgb(0,112,192)"> </span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><a href="http://www.cruk.cam.ac.uk/" target="_blank" rel="noopener noreferrer"><span style="font-size:10.5pt; font-family:"Times New Roman",serif,serif,EmojiFont; color:blue"><img class="x_EmojiInsert" alt="Description: CRI Logo" style="width:176.99pt; height:38.99pt" data-outlook-trace="F:0|T:2" src="cid:image001.jpg@01D4CCEA.783A5790"></span></a></p>
<br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>