<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Arial","sans-serif";
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:595.3pt 841.9pt;
margin:70.85pt 70.85pt 56.7pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=WordSection1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>My guess would be that your read pattern is very random (so
read-ahead isn’t helping) or that you need to tune your read-ahead parameters.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>A simple test would be to use dd to write/read a file (make sure
it’s in the 10s of GB to minimize cache effects) <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>
lustre-discuss-bounces@lists.lustre.org
[mailto:lustre-discuss-bounces@lists.lustre.org] <b>On Behalf Of </b>Sergius
ssi. Siczek - sysGen GmbH<br>
<b>Sent:</b> Wednesday, August 11, 2010 10:36 AM<br>
<b>To:</b> lustre-discuss@lists.lustre.org<br>
<b>Cc:</b> GmbH; Dieter Nikisch - sysGen GmbH; Rolf@lists.lustre.org<br>
<b>Subject:</b> [Lustre-discuss] LUSTRE Bad Read Performance<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><span lang=DE style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Hello,<o:p></o:p></span></p>
<p class=MsoNormal><span lang=DE style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>I’ve
got a strange problem with lustre read perforamnce.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Following
setup:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>8
Core (100GB RAM) combined MGS/OSS with two OST targets server, with Infiniband<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>2x
RAID6 (11x 1 TB HDD -> 8TB usable on each RAID). (/dev/sdc and
/dev/sdd)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>/dev/sdc1
= 1024GB<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>/dev/sdd1
= 1024GB<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>1x
small HDD for journal and MDS. (/dev/sdb)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>/dev/sdb1
= 10 GB journal<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>/dev/sdb2
= 10GB journal<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>/dev/sdb3
= 50 GB MDS<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Procedure:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>mkfs.lustre
--reformat --fsname lustre --mdt --mgs /dev/sdb3<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>mkfs.lustre
--fsname=lustre --reformat --ost --mgsnode 192.168.0.1@o2ib
--mkfsoptions="-E stride=256 -E stripe-width=9 -J device=/dev/sdb1"
/dev/sdc1<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>mkfs.lustre
--fsname=lustre --reformat --ost --mgsnode 192.168.0.1@o2ib
--mkfsoptions="-E stride=256 -E stripe-width=9 -J device=/dev/sdb2"
/dev/sdd1<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Client
machine:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>24
core (50 GB RAM) with infiniband.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Mount
:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>mount
-t lustre <a href="mailto:192.168.0.1@o2ib:/lustre%20/mnt/client/">192.168.0.1@o2ib:/lustre
/mnt/client/</a><o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>After
that I ran some simple throughput tests. Write performance (5 threads) is about
1072 MB/s (which is great), but read performance is very bad ~ 510 MB/s.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>What
could be wrong about this setup? Why write performance is so good, but read so
bad?<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Of
course I tested the hardware itself, made a filesystem on the raids (without
lustre) and ran the same tests, ~1,3GB/s write 1,7GB/s read (800 GB file)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>The
infiniband is also okay, used the same cards/cable for some HPC tests just
hours before, everything good. Nevertheless used the server as a client two
(just to bypass the infiniband side as a potential failure). Nearly the same
results (write good, read bad). It’s like there is a wall at ~510 MB/s read.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Any
ideas or suggestions ? Its really urgent. Thank you a lot!<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>2.6.27.39+lustre1.8.3+0.credativ.lenny.4<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Lustre
version: 1.8.3-20100410022943-PRISTINE-2.6.27.39+lustre1.8.3+0.credativ.lenny.4<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'> <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<div>
<p class=MsoPlainText style='margin:0in;margin-bottom:.0001pt'><span
lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>sysGen
GmbH<o:p></o:p></span></p>
<p class=MsoPlainText style='margin:0in;margin-bottom:.0001pt'><span
lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Sergius
Siczek<o:p></o:p></span></p>
<p class=MsoPlainText style='margin:0in;margin-bottom:.0001pt'><span
lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Am
Hallacker 48 · 28327 Bremen<o:p></o:p></span></p>
<p class=MsoPlainText style='margin:0in;margin-bottom:.0001pt'><span
lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Tel
(0421) 4 09 66-32 · Fax (0421) 4 09 66-33<o:p></o:p></span></p>
<p class=MsoPlainText style='margin:0in;margin-bottom:.0001pt'><span
lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><a
href="mailto:ssiczek@sysgen.de" title="mailto:ssiczek@sysgen.de">ssiczek@sysgen.de</a>
· <a href="http://www.sysgen.de/">www.sysgen.de</a><o:p></o:p></span></p>
<p class=MsoPlainText style='margin:0in;margin-bottom:.0001pt'><span lang=DE
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Geschäftsführung:
Gabriele Nikisch, Anke Heinßen · Eingetragen beim Amtsgericht Walsrode · HRB
121943<o:p></o:p></span></p>
</div>
<p class=MsoNormal><span lang=DE><o:p> </o:p></span></p>
</div>
</body>
</html>