<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
We were building the ZFS based OST and we also found out I/O wait even when 4 OST per shelf. please lower the amount of OST available as the HBA may not be sustainable for that high I/O . we are now using enterprise Linux (like Cray) as they are designed to
have high I/O. we would like to see if anyone has experience solving that part.</div>
<div class="elementToProof">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div id="divtagdefaultwrapper" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
<div><font size="2">Regards,</font></div>
<div><font size="2"><br>
James</font></div>
<div><font face="Verdana"><font style="font-size:8pt" size="1"><br>
</font></font></div>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Kurt Strosahl via lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Sent:</b> Monday, February 27, 2023 3:02 PM<br>
<b>To:</b> lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> [lustre-discuss] OSTs per OSS question</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Good Morning,</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
I'm in the early phases of a new lustre file system design. One thing we are observing on our lustre file system from 2019 (running 2.12.9-1) is that the oss systems report a lot of time spent in i/o wait. We are configured with two oss systems connected
via sas cables to a pair of JBODS, and are wondering if going from two JBODs per OSS (so 6 osts per oss under normal operating conditions, 12 under failover) to 4 shelves (12 osts per oss under normal operating conditions, 24 under failover). The OSTs are
10 raidz2 vdevs, and we are planning on using 20TB drives in this new file system.</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Has anyone tried the 4 shelf / 2 oss configuration?</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Reading through the Lustre manual I see the following under table 1.2</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
"OSS: 1-128 TiB per OST, 1-8 OSTs per OSS"</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Is that an indication that more than 8 OSTs per OSS causes problems for the OSS systems? Our current OSS systems have run at 12 OSTs during failover situations, once for at least a few days due to a hardware failure on one of the OSS systems.</div>
<div class="x_elementToProof">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
respectfully,</div>
<div id="x_Signature">
<div>
<div></div>
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; font-family:Calibri,Helvetica,sans-serif; color:rgb(0,0,0)">
<p style="margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0"><span id="x_ms-rterangepaste-start"></span><span style="color:rgb(51,51,51); font-family:monospace; font-size:14.16px">Kurt J. Strosahl (he/him)</span><br style="color:rgb(51,51,51); font-family:monospace; font-size:14.16px">
<span style="color:rgb(51,51,51); font-family:monospace; font-size:14.16px">System Administrator: Lustre, HPC</span><br style="color:rgb(51,51,51); font-family:monospace; font-size:14.16px">
<span style="color:rgb(51,51,51); font-family:monospace; font-size:14.16px">Scientific Computing Group, Thomas Jefferson National Accelerator Facility</span><span id="x_ms-rterangepaste-end"></span><br>
</p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>