<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Good Morning,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
I'm in the early phases of a new lustre file system design. One thing we are observing on our lustre file system from 2019 (running 2.12.9-1) is that the oss systems report a lot of time spent in i/o wait. We are configured with two oss systems connected
via sas cables to a pair of JBODS, and are wondering if going from two JBODs per OSS (so 6 osts per oss under normal operating conditions, 12 under failover) to 4 shelves (12 osts per oss under normal operating conditions, 24 under failover). The OSTs are
10 raidz2 vdevs, and we are planning on using 20TB drives in this new file system.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Has anyone tried the 4 shelf / 2 oss configuration?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Reading through the Lustre manual I see the following under table 1.2</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
"OSS: 1-128 TiB per OST, 1-8 OSTs per OSS"</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
Is that an indication that more than 8 OSTs per OSS causes problems for the OSS systems? Our current OSS systems have run at 12 OSTs during failover situations, once for at least a few days due to a hardware failure on one of the OSS systems.</div>
<div class="elementToProof">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
respectfully,</div>
<div id="Signature">
<div>
<div></div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<p style="margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0"><span id="ms-rterangepaste-start"></span><span style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">Kurt J. Strosahl (he/him)</span><br style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">
<span style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">System Administrator: Lustre, HPC</span><br style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">
<span style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">Scientific Computing Group, Thomas Jefferson National Accelerator Facility</span><span id="ms-rterangepaste-end"></span><br>
</p>
</div>
</div>
</div>
</div>
</body>
</html>