<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:#0000A0;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
color:black;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link="#0000A0" vlink=purple>
<div class=Section1>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>While performing a single
copy, single client write/read test using dd, we are finding that our Nehalem
clients running <o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>2.6.18-92.1.10.el5-lustre-1.6.5.1
<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>write about half the speed
of our Nehalem clients running <o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>2.6.18-53.1.13.el5_lustre.1.6.4.3
to three different lustre file systems.<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>This is true even though the
slower clients have the same processors and more RAM, 18GB for the slow writers
and 12GB for the fast writers. Both systems use OFED 1.3.1. All benchmarks we
use perform better on the slow-write clients and read speed from LFS is
comparable across all clients. <o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>Max_rpcs_in_flight and
max_pages_per_rpc are default on both systems.<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>They are on the same IB
network, with the same QDR cards and IB connectivity has been verified with the
IB utilities. They are almost identical in bandwidth and latency.<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>We're also using the same
modprobe.conf and openibd.conf files on both systems.<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>We're using 34GB file size
on the 12GB and 18GB RAM systems, 137GB file on the 96GB RAM system. So it's
not a matter of caching in RAM.<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>Are there known issues with
our 2.6.18-92.1.10.el5-lustre-1.6.5.1 combination?<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>This is not a problem with
the lustre file system as we get the same type of results no matter which of
our three lustre systems the test is being written to.<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>Here are the summaries from
several runs of ost-survey on our new Lustre system. Please comment on the
worst/best deltas of the read and write operations.<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>Number of Active OST devices
: 96 <o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>Worst Read 38.167753 38.932928 39.006537 39.782153 38.717915<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>Best Read 61.704534 61.832461 63.284999 65.000491 61.836016<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>Read Average: 51.433847 51.281630 51.297278 51.582327 51.318410<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>Worst Write 34.311237 49.009757 55.272744 51.532331 51.816523<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>Best Write 94.001170 96.033483 93.401792 93.081544 91.030717<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>Write Average: 74.248683 71.831019 75.179863 74.723100 74.930529<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></font></p>
<p class=MsoNormal style='text-autospace:none'><font size=2 face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>/bob<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><o:p> </o:p></span></font></p>
<div>
<div>
<p class=MsoNormal><strong><b><font size=1 color=navy face=Tahoma><span
style='font-size:7.5pt;font-family:Tahoma;color:navy'>Bob Hayes</span></font></b></strong><strong><b><font
size=2 color=navy face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma;
color:navy'> </span></font></b></strong><font color=black><span
style='color:black'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 color=navy face=Tahoma><span style='font-size:
7.5pt;font-family:Tahoma;color:navy'>System Administrator</span></font><font
color=navy><span style='color:navy'><br>
</span></font><font size=1 color=navy face=Tahoma><span style='font-size:7.5pt;
font-family:Tahoma;color:navy'>SSG-DRD-DP</span></font><font color=navy><span
style='color:navy'><br>
</span></font><font size=1 color=navy face=Tahoma><span style='font-size:7.5pt;
font-family:Tahoma;color:navy'>Office: 253-371-3040</span></font><font
size=1 color=black><span style='font-size:7.5pt;color:black'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 color=navy face=Tahoma><span style='font-size:
7.5pt;font-family:Tahoma;color:navy'>Cell: 253-441-5482</span></font><font
color=navy><span style='color:navy'><br>
</span></font><font size=1 color=navy face=Tahoma><span lang=DE
style='font-size:7.5pt;font-family:Tahoma;color:navy'>e-mail: </span></font><font
color=black><span style='color:black'><a href="mailto:robert.n.hayes@Intel.Com"><font
size=1 face=Tahoma><span lang=DE style='font-size:7.5pt;font-family:Tahoma'>robert.n.hayes@Intel.Com</span></font></a></span></font><font
color=navy><span style='color:navy'><o:p></o:p></span></font></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
</div>
</body>
</html>