<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Sorry, you'll have to ask someone at Azure about this. I don't know anything about what "Premium Lustre" or "Lustre V2" means, and I can't speak to any kind of performance for their systems.</div>
<div class=""><br class="">
</div>
On Jul 30, 2024, at 08:48, Baucum, Rashun <<a href="mailto:Rashun.Baucum@td.com" class="">Rashun.Baucum@td.com</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Good morning, Andreas.<br class="">
<br class="">
I apologize for the delay.<br class="">
<br class="">
Yes, I will provide additional information. I initially asked the question as a general inquiry. The crux of the performance issue we see is that the newer RHEL 8.10 - 2.15.5 lustre with similar builds do not hit the same average performance as we have seen
previously.<br class="">
<br class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Which version of Lustre for the old and new kernel and was it the same before upgrading to RHEL8.10?<o:p class=""></o:p></div>
<ul type="disc" style="margin-bottom: 0in; margin-top: 0in;" class="">
<li class="MsoListParagraph" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Previous Kernel : 3.10.0-1160.49.1.el7_lustre, RHEL 7.9<o:p class=""></o:p></li><li class="MsoListParagraph" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="FR" class="">Current Kernel : 4.18.0-553.5.1.el8_lustre, RHEL 8.10<o:p class=""></o:p></span></li><li class="MsoListParagraph" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
It was not the same, we waited for the last minor version of RHEL8.<o:p class=""></o:p></li></ul>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Which RHEL version are you comparing against, RHEL 8.9?<o:p class=""></o:p></div>
<ul type="disc" style="margin-bottom: 0in; margin-top: 0in;" class="">
<li class="MsoListParagraph" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="FR" class="">Comparison Version : RHEL 8.8<o:p class=""></o:p></span></li><li class="MsoListParagraph" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="FR" class="">Lustre Client Version : 4.18.0-425.3.1.el8_lustre<o:p class=""></o:p></span></li></ul>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Have you upgraded both the clients and servers to RHEL 8.10, or only the clients?<o:p class=""></o:p></div>
<ul type="disc" style="margin-bottom: 0in; margin-top: 0in;" class="">
<li class="MsoListParagraph" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Currently both are upgraded to the RHEL 8.10.<o:p class=""></o:p></li></ul>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 14pt;" class="">Results of FIO testing<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 14pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Original Production: Initial baseline established ~1 year ago. This build is no longer in use, but performance of non "premium" builds should be approximately around this.<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Lustre V2: Current builds and are currently being used as a direct reference point. These used RHEL8 clients while being RHEL7 lustres.<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
RHEL8.10 – 2.15.5: Builds being tested before we push updates to higher environments.<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Between Original Production, Lustre V2 36, and the RHEL8.10 lustres there are minimal changes between builds. In this specific case the only difference between Lustre V2 36 and RHEL8.10 builds are the lustre versions and the RHEL 8.10 Premium Lustre uses Azure's
premium SSDs as disks for OSTs. All other builds use standard HDDs for OSTs.<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class="">
<span style="font-size: 15pt; font-family: -apple-system; color: rgb(26, 81, 53); letter-spacing: -0.1pt;" class=""><o:p class=""> </o:p></span></div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="914" style="width: 685.6pt; border-collapse: collapse;">
<thead class="">
<tr style="height: 28.6pt;" class="">
<td valign="top" style="border: 1pt solid rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 28.6pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Write Throughput<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 28.6pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Original Production<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 28.6pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Lustre V2 36<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 28.6pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Lustre V2 108<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 28.6pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">RHEL 8.10 - 2.15.5 - Premium Lustre<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 28.6pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">RHEL 8.10 - 2.15.5 - Standard Lustre<o:p class=""></o:p></span></b></div>
</td>
</tr>
</thead>
<tbody class="">
<tr style="height: 28.6pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.6pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Latency (sec) Avg<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.6pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
1.485<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.6pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
1.482<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.6pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
1.338<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.6pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
1.596<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.6pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
2.968<o:p class=""></o:p></div>
</td>
</tr>
<tr style="height: 14.3pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 14.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
IOPS Avg<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 14.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
689<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 14.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
690<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 14.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
765<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 14.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
640<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 14.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
640<o:p class=""></o:p></div>
</td>
</tr>
<tr style="height: 28.6pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.6pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Bandwidth (MB/s)<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.6pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
723<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.6pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
724<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.6pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
803<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.6pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
672<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.6pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
362<o:p class=""></o:p></div>
</td>
</tr>
<tr style="height: 13.55pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 13.55pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
IO (GB)<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 13.55pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
4340<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 13.55pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
4346<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 13.55pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
4818<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 13.55pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
4037<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 13.55pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
2173<o:p class=""></o:p></div>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin: 7.5pt 0in 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white; background-position: initial initial; background-repeat: initial initial;">
<span style="font-size: 10.5pt; font-family: -apple-system; color: rgb(23, 43, 77);" class=""><o:p class=""> </o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="916" style="width: 687.2pt; border-collapse: collapse;">
<thead class="">
<tr style="height: 28.5pt;" class="">
<td valign="top" style="border: 1pt solid rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 28.5pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Write IOPS<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 28.5pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Original Production<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 28.5pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Lusre V2 36<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 28.5pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Lustre V2 108<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 28.5pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">RHEL 8.10 - 2.15.5 - Premium Lustre<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 28.5pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">RHEL 8.10 - 2.15.5 - Standard Lustre<o:p class=""></o:p></span></b></div>
</td>
</tr>
</thead>
<tbody class="">
<tr style="height: 28.5pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Latency (sec) Avg<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
10.354<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
18.709<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
4.911<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
5.839<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
11.821<o:p class=""></o:p></div>
</td>
</tr>
<tr style="height: 14.25pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 14.25pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Avg IOPS<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 14.25pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
24<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 14.25pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
13<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 14.25pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
52<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 14.25pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
43<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 14.25pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
21<o:p class=""></o:p></div>
</td>
</tr>
<tr style="height: 28.5pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Bandwidth (MB/s)<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
26<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
14<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
55<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
45<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 28.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
23<o:p class=""></o:p></div>
</td>
</tr>
<tr style="height: 13.5pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 13.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
IO (GB)<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 13.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
156<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 13.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
86<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 13.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
328<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 13.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
276<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 13.5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
137<o:p class=""></o:p></div>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin: 7.5pt 0in 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white; background-position: initial initial; background-repeat: initial initial;">
<span style="font-size: 10.5pt; font-family: -apple-system; color: rgb(23, 43, 77);" class=""><o:p class=""> </o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="916" style="width: 687.2pt; border-collapse: collapse;">
<thead class="">
<tr style="height: 15.4pt;" class="">
<td valign="top" style="border: 1pt solid rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 15.4pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Read Throughput<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 15.4pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Original Production<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 15.4pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Lustre V2 36<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 15.4pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Lustre V2 108<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 15.4pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">RHEL 8.10 - 2.15.5 - Premium Lustre<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 15.4pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">RHEL 8.10 - 2.15.5 - Standard Lustre<o:p class=""></o:p></span></b></div>
</td>
</tr>
</thead>
<tbody class="">
<tr style="height: 15.4pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Latency (sec) Avg<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
1.292<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
1.223<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
2.307<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
1.598<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
3.022<o:p class=""></o:p></div>
</td>
</tr>
<tr style="height: 15.4pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Avg IOPS<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
792<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
836<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
754<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
640<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
338<o:p class=""></o:p></div>
</td>
</tr>
<tr style="height: 15.4pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Bandwidth (MB/s)<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
831<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
877<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
791<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
672<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
355<o:p class=""></o:p></div>
</td>
</tr>
<tr style="height: 15.4pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
IO (GB)<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
4986<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
5269<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
4750<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
4033<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.4pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
2134<o:p class=""></o:p></div>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin: 7.5pt 0in 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white; background-position: initial initial; background-repeat: initial initial;">
<span style="font-size: 10.5pt; font-family: -apple-system; color: rgb(23, 43, 77);" class=""><o:p class=""> </o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="914" style="width: 685.75pt; border-collapse: collapse;">
<thead class="">
<tr style="height: 15.3pt;" class="">
<td valign="top" style="border: 1pt solid rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 15.3pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Read IOPS<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 15.3pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Original Production<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 15.3pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Lustre V2 36<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 15.3pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">Lustre V2 108<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 15.3pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">RHEL 8.10 - 2.15.5 - Premium Lustre<o:p class=""></o:p></span></b></div>
</td>
<td valign="top" style="border-top-width: 1pt; border-right-width: 1pt; border-bottom-width: 1pt; border-style: solid solid solid none; border-top-color: rgb(193, 199, 208); border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); background-color: rgb(244, 245, 247); padding: 0in; height: 15.3pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: rgb(26, 81, 53);" class="">RHEL 8.10 - 2.15.5 - Standard Lustre<o:p class=""></o:p></span></b></div>
</td>
</tr>
</thead>
<tbody class="">
<tr style="height: 15.3pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Latency (sec) Avg<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
4.190<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
5.887<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
5.076<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
5.954<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
11.924<o:p class=""></o:p></div>
</td>
</tr>
<tr style="height: 15.3pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Avg IOPS<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
61<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
43<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
50<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
42<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
21<o:p class=""></o:p></div>
</td>
</tr>
<tr style="height: 15.3pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Bandwidth (MB/s)<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
64<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
45<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
53<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
45<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
23<o:p class=""></o:p></div>
</td>
</tr>
<tr style="height: 15.3pt;" class="">
<td valign="top" style="border-right-width: 1pt; border-bottom-width: 1pt; border-left-width: 1pt; border-style: none solid solid; border-right-color: rgb(193, 199, 208); border-bottom-color: rgb(193, 199, 208); border-left-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
IO (GB)<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
384<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
274<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
318<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
271<o:p class=""></o:p></div>
</td>
<td valign="top" style="border-style: none solid solid none; border-bottom-width: 1pt; border-bottom-color: rgb(193, 199, 208); border-right-width: 1pt; border-right-color: rgb(193, 199, 208); padding: 5.25pt 7.5pt; height: 15.3pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
135<o:p class=""></o:p></div>
</td>
</tr>
</tbody>
</table>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Thanks,<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Rashun Baucum<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class=""><br class="">
<p align="Left" style="font-family: Calibri; font-size: 10pt; margin: 15pt; font-style: normal; font-weight: normal; text-decoration: none;" class="">
Internal<br class="">
</p>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class="">From:</b><span class="Apple-converted-space"> </span>Andreas Dilger <<a href="mailto:adilger@whamcloud.com" style="color: blue; text-decoration: underline;" class="">adilger@whamcloud.com</a>><span class="Apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Thursday, July 4, 2024 12:40 AM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>Baucum, Rashun <<a href="mailto:Rashun.Baucum@td.com" style="color: blue; text-decoration: underline;" class="">Rashun.Baucum@td.com</a>><br class="">
<b class="">Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [lustre-discuss] Rhel8.10 Lustre Kernel Performance Decrease<o:p class=""></o:p></div>
</div>
</div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-align: center; background-color: maroon;" class="">
<span style="font-size: 9pt; color: white;" class="">CAUTION: EXTERNAL MAIL. DO NOT CLICK ON LINKS OR OPEN ATTACHMENTS YOU DO NOT TRUST<br class="">
ATTENTION : COURRIEL EXTERNE.<span class="Apple-converted-space"> </span></span><span lang="FR" style="font-size: 9pt; color: white;" class="">NE CLIQUEZ PAS SUR DES LIENS ET N'OUVREZ PAS DE PIÈCES JOINTES AUXQUELS VOUS NE FAITES PAS CONFIANCE<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class=""><br class="">
</span>On Jul 3, 2024, at 13:12, Baucum, Rashun via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>> wrote:<o:p class=""></o:p></div>
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Good afternoon,<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
We have recently started executing performance testing on the new rhel 8.10 lustre kernel. We have noticed an drop in performance in our initial testing. Its roughly a 30-40% drop in total IO observed with our FIO testing. My question is has anyone else noticed
any performance decreases?<o:p class=""></o:p></div>
</div>
</div>
</blockquote>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Hi Rashun,<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
could you please be more specific about what you are comparing? Which version of Lustre for the old and new kernel, and was it the same before upgrading to RHEL8.10? Which RHEL version are you comparing against, RHEL 8.9? Have you upgraded both the clients
and servers to RHEL8.10, or only the clients?<o:p class=""></o:p></div>
</div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Cheers, Andreas<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">--<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Andreas Dilger<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Lustre Principal Architect<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Whamcloud<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class=""><o:p class=""> </o:p></span></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class=""><o:p class=""> </o:p></span></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class=""><o:p class=""> </o:p></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class=""><o:p class=""> </o:p></span></div>
</div>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p class=""> </o:p></p>
</div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<font face="Arial" style="font-size: 8pt;" class=""><br class="">
<br class="">
If you wish to unsubscribe from receiving commercial electronic messages from TD Bank Group, please click<span class="Apple-converted-space"> </span><a href="http://www.td.com/tdoptout" style="color: blue; text-decoration: underline;" class="">here</a><span class="Apple-converted-space"> </span>or
go to the following web address:<span class="Apple-converted-space"> </span><a href="http://www.td.com/tdoptout" style="color: blue; text-decoration: underline;" class="">www.td.com/tdoptout</a><br class="">
Si vous souhaitez vous désabonner des messages électroniques de nature commerciale envoyés par Groupe Banque TD veuillez cliquer<span class="Apple-converted-space"> </span><a href="http://www.td.com/tddesab" style="color: blue; text-decoration: underline;" class="">ici</a><span class="Apple-converted-space"> </span>ou
vous rendre à l'adresse<span class="Apple-converted-space"> </span><a href="http://www.td.com/tddesab" style="color: blue; text-decoration: underline;" class="">www.td.com/tddesab</a></font></p>
<p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<font face="Arial" style="font-size: 8pt;" class=""><br class="">
NOTICE: Confidential message which may be privileged. Unauthorized use/disclosure prohibited. If received in error, please go to<span class="Apple-converted-space"> </span><a href="http://www.td.com/legal" style="color: blue; text-decoration: underline;" class="">www.td.com/legal</a><span class="Apple-converted-space"> </span>for
instructions.<br class="">
AVIS : Message confidentiel dont le contenu peut être privilégié. Utilisation/divulgation interdites sans permission. Si reçu par erreur, prière d'aller au<span class="Apple-converted-space"> </span><a href="http://www.td.com/francais/avis_juridique" style="color: blue; text-decoration: underline;" class="">www.td.com/francais/avis_juridique</a><span class="Apple-converted-space"> </span>pour
des instructions.</font></p>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>