<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We had an issue a few months ago with the underlying zpool for one of our OSTs. I managed to get it mounted in read only mode and migrated all of the files off it with lfs migrate, then recreated the OST and reintroduced it. This all went pretty smoothly -
at the same time I updated our progressive file layout using the following command:</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
lfs find . -type d -print0 | xargs -0 lfs setstripe -E 256M -c 1 -E eof -c -1</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I then ran an lfs find to find all the files bigger than 256M and migrated them to this new layout.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I have since noticed that the OST that was reintroduced has been filling up more rapidly than the others, to the point where it is now full:</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
UUID bytes Used Available Use% Mounted on</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
scratchc-MDT0000_UUID 1.4T 108.0G 1.3T 8% /mnt/scratchc[MDT:0]</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
scratchc-OST0000_UUID 55.2T 55.2T 42.0M 100% /mnt/scratchc[OST:0]</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
scratchc-OST0001_UUID 55.2T 22.5T 32.7T 41% /mnt/scratchc[OST:1]</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
scratchc-OST0002_UUID 46.0T 19.3T 26.7T 43% /mnt/scratchc[OST:2]</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
scratchc-OST0003_UUID 46.0T 19.4T 26.6T 43% /mnt/scratchc[OST:3]</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
scratchc-OST0004_UUID 46.0T 19.5T 26.5T 43% /mnt/scratchc[OST:4]</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
scratchc-OST0005_UUID 55.2T 22.8T 32.5T 42% /mnt/scratchc[OST:5]</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
filesystem_summary: 303.8T 158.8T 145.0T 53% /mnt/scratchc</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
For reference, I marked the OST as inactive to migrate the files off by using the command:</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
lctl set_param osp.scratchc-OST0000-osc-MDT0000.max_create_count=0</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
As per the manual. To reactivate it after having rebuilt it, I copied the count from the other OSTs:</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
~]# lctl get_param osp.scratchc-*.max_create_count</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
osp.scratchc-OST0000-osc-MDT0000.max_create_count=20000</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
osp.scratchc-OST0001-osc-MDT0000.max_create_count=20000</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
osp.scratchc-OST0002-osc-MDT0000.max_create_count=20000</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
osp.scratchc-OST0003-osc-MDT0000.max_create_count=20000</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
osp.scratchc-OST0004-osc-MDT0000.max_create_count=20000</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
osp.scratchc-OST0005-osc-MDT0000.max_create_count=20000</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
As far as I can tell I haven't told lustre to preferentially use the one OST, so I'm a little stumped as to why this has happened - it is possible that someone has changed the default layout on some of their folders but I'm struggling to think of a quick way
of checking this.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Has anyone else run into similar problems? I'm hoping there is something incredibly obvious that I've missed somewhere!</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks in advance!</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature" style="color: inherit;">
<p style="margin: 0px; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="color: rgb(89, 89, 89);">Jon Marshall</span></p>
<p style="margin: 0px; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="font-size: 10pt; color: rgb(89, 89, 89);">High Performance Computing Specialist<br>
</span></p>
<p style="margin: 0px; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="font-size: 10pt; color: rgb(89, 89, 89);"> </span></p>
<p style="margin: 0px; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="color: rgb(89, 89, 89);"><b>IT and Scientific Computing Team</b></span></p>
<p style="margin: 0px; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="font-size: 10pt; color: rgb(89, 89, 89);"> </span></p>
<p style="margin: 0px; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="font-size: 10pt; color: rgb(89, 89, 89);">Cancer Research UK Cambridge Institute</span></p>
<p style="margin: 0px; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="font-size: 10pt; color: rgb(89, 89, 89);">Li Ka Shing Centre | Robinson Way | Cambridge | CB2 0RE</span></p>
<p style="margin: 0px; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="font-size: 10pt; color: rgb(0, 112, 192);"><a href="http://www.cruk.cam.ac.uk/" target="_blank" rel="noopener noreferrer" style="color: rgb(0, 112, 192); margin-top: 0px; margin-bottom: 0px;">Web</a>
| <a href="http://www.facebook.com/cancerresearchuk" target="_blank" rel="noopener noreferrer" style="color: rgb(0, 112, 192); margin-top: 0px; margin-bottom: 0px;">
Facebook</a> | <a href="http://twitter.com/CR_UK" target="_blank" rel="noopener noreferrer" style="color: rgb(0, 112, 192); margin-top: 0px; margin-bottom: 0px;">
Twitter</a></span></p>
<p style="margin: 0px; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="font-size: 10pt; color: rgb(0, 112, 192);"> </span></p>
<p style="margin: 0px; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="font-family: "Times New Roman", serif, serif, EmojiFont; font-size: 10.5pt; color: blue;"><a href="http://www.cruk.cam.ac.uk/" target="_blank" rel="noopener noreferrer" style="color: blue; margin-top: 0px; margin-bottom: 0px;"><img alt="Description: CRI Logo" width="235" height="51" style="width: 176.99pt; height: 38.99pt; margin-top: 0px; margin-bottom: 0px;" data-outlook-trace="F:1|T:1" src="cid:961f6456-36f5-4c21-b6a0-330e4e246711"></a></span></p>
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
</body>
</html>