<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
We've been running lustre 2.15.1 in production for over a year and recently decided to enable PFL with DoM on our filesystem. Things have been fine up until last week, when users started reporting issues copying files, specifically "No space left on device".
The MDT is running ldiskfs as the backend.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I've searched through the mailing list and found a couple of people reporting similar problems, which prompted me to check the inode allocation, which is currently:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
UUID Inodes IUsed IFree IUse% Mounted on
<div class="ContentPasted0">scratchc-MDT0000_UUID 624492544 71144384 553348160 12% /mnt/scratchc[MDT:0]
</div>
<div class="ContentPasted0">scratchc-OST0000_UUID 57712579 24489934 33222645 43% /mnt/scratchc[OST:0]</div>
<div class="ContentPasted0">scratchc-OST0001_UUID 57114064 24505876 32608188 43% /mnt/scratchc[OST:1]</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">filesystem_summary: 136975217 71144384 65830833 52% /mnt/scratchc</div>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
So, nowhere near full - the disk usage is a little higher:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
UUID bytes Used Available Use% Mounted on
<div class="ContentPasted1">scratchc-MDT0000_UUID 882.1G 451.9G 355.8G 56% /mnt/scratchc[MDT:0]
</div>
<div class="ContentPasted1">scratchc-OST0000_UUID 53.6T 22.7T 31.0T 43% /mnt/scratchc[OST:0]</div>
<div class="ContentPasted1">scratchc-OST0001_UUID 53.6T 23.0T 30.6T 43% /mnt/scratchc[OST:1]</div>
<div><br class="ContentPasted1">
</div>
<div class="ContentPasted1">filesystem_summary: 107.3T 45.7T 61.6T 43% /mnt/scratchc</div>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
But not full either! The errors are accompanied in the logs by:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2">
LustreError: 15450:0:(tgt_grant.c:463:tgt_grant_space_left()) scratchc-MDT0000: cli ba0195c7-1ab4-4f7c-9e28-8689478f5c17/ffff9e331e231c00 left 82586337280 < tot_grant 82586681321 unstable 0 pending 0 dirty 1044480<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3">
LustreError: 15450:0:(tgt_grant.c:463:tgt_grant_space_left()) Skipped 33050 previous similar messages<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3">
For reference the DoM striping we're using is:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4">
lcm_layout_gen: 0
<div class="ContentPasted4"> lcm_mirror_count: 1</div>
<div class="ContentPasted4"> lcm_entry_count: 3</div>
<div class="ContentPasted4"> lcme_id: N/A</div>
<div class="ContentPasted4"> lcme_mirror_id: N/A</div>
<div class="ContentPasted4"> lcme_flags: 0</div>
<div class="ContentPasted4"> lcme_extent.e_start: 0</div>
<div class="ContentPasted4"> lcme_extent.e_end: 1048576</div>
<div class="ContentPasted4"> stripe_count: 0 stripe_size: 1048576 pattern: mdt stripe_offset: -1</div>
<div><br class="ContentPasted4">
</div>
<div class="ContentPasted4"> lcme_id: N/A</div>
<div class="ContentPasted4"> lcme_mirror_id: N/A</div>
<div class="ContentPasted4"> lcme_flags: 0</div>
<div class="ContentPasted4"> lcme_extent.e_start: 1048576</div>
<div class="ContentPasted4"> lcme_extent.e_end: 1073741824</div>
<div class="ContentPasted4"> stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1</div>
<div><br class="ContentPasted4">
</div>
<div class="ContentPasted4"> lcme_id: N/A</div>
<div class="ContentPasted4"> lcme_mirror_id: N/A</div>
<div class="ContentPasted4"> lcme_flags: 0</div>
<div class="ContentPasted4"> lcme_extent.e_start: 1073741824</div>
<div class="ContentPasted4"> lcme_extent.e_end: EOF</div>
stripe_count: -1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4">
So the first 1MB on the MDT.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4">
My question is obviously what is causing these errors? I'm not massively familiar with Lustre internals, so any pointers on where to look would be greatly appreciated!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4">
Cheers</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4">
Jon</div>
<div class="elementToProof">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="color: rgb(89, 89, 89);">Jon Marshall</span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="font-size: 10pt; color: rgb(89, 89, 89);">High Performance Computing Specialist<br>
</span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="font-size: 10pt; color: rgb(89, 89, 89);"> </span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><b><span style="color: rgb(89, 89, 89);">IT and Scientific Computing Team</span></b><span style="font-size: 10pt; color: rgb(89, 89, 89);"></span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="font-size: 10pt; color: rgb(89, 89, 89);"> </span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="font-size: 10pt; color: rgb(89, 89, 89);">Cancer Research UK Cambridge Institute</span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="font-size: 10pt; color: rgb(89, 89, 89);">Li Ka Shing Centre | Robinson Way | Cambridge | CB2 0RE</span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span><a href="http://www.cruk.cam.ac.uk/" target="_blank" rel="noopener noreferrer" data-loopstyle="linkonly"><span style="font-size: 10pt; color: rgb(0, 112, 192);">Web</span></a></span><span style="font-size: 10pt; color: rgb(0, 112, 192);">
| </span><span><a href="http://www.facebook.com/cancerresearchuk" target="_blank" rel="noopener noreferrer" data-loopstyle="linkonly"><span style="font-size: 10pt; color: rgb(0, 112, 192);">Facebook</span></a></span><span style="font-size: 10pt; color: rgb(0, 112, 192);">
| </span><span><a href="http://twitter.com/CR_UK" target="_blank" rel="noopener noreferrer" data-loopstyle="linkonly"><span style="font-size: 10pt; color: rgb(0, 112, 192);">Twitter</span></a></span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="font-size: 10pt; color: rgb(0, 112, 192);"> </span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><a href="http://www.cruk.cam.ac.uk/" target="_blank" rel="noopener noreferrer"><span style="font-size: 10.5pt; font-family: "Times New Roman", serif, serif, EmojiFont; color: blue;"><img class="EmojiInsert" alt="Description: CRI Logo" style="width:176.99pt; height:38.99pt" data-outlook-trace="F:0|T:1" src="cid:image001.jpg@01D4CCEA.783A5790"></span></a></p>
<br>
</div>
</div>
</div>
</div>
</body>
</html>