<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="State"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="place"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman";}
span.E-MailFormatvorlage17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:595.3pt 841.9pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=DE link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Hello,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>I’ve got a strange problem with lustre read
perforamnce.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>Following setup:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>8 Core (100GB RAM) combined MGS/OSS with two OST
targets server, with Infiniband<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>2x RAID6 (11x 1 TB HDD -> 8TB usable on each RAID).
(/dev/sdc and /dev/sdd)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>/dev/sdc1 = 1024GB<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>/dev/sdd1 = 1024GB<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>1x small HDD for journal and MDS. (/dev/sdb)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>/dev/sdb1 = 10 GB journal<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>/dev/sdb2 = 10GB journal<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>/dev/sdb3 = 50 GB MDS<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>Procedure:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>mkfs.lustre --reformat --fsname lustre --mdt --mgs
/dev/sdb3<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>mkfs.lustre --fsname=lustre --reformat --ost
--mgsnode 192.168.0.1@o2ib --mkfsoptions="-E stride=256 -E stripe-width=9
-J device=/dev/sdb1" /dev/sdc1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>mkfs.lustre --fsname=lustre --reformat --ost
--mgsnode 192.168.0.1@o2ib --mkfsoptions="-E stride=256 -E stripe-width=9
-J device=/dev/sdb2" /dev/sdd1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>Client machine:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>24 core (50 GB RAM) with infiniband.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>Mount :<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>mount -t lustre <a
href="mailto:192.168.0.1@o2ib:/lustre%20/mnt/client/">192.168.0.1@o2ib:/lustre
/mnt/client/</a><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>After that I ran some simple throughput tests. Write
performance (5 threads) is about 1072 MB/s (which is great), but read performance
is very bad ~ 510 MB/s.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>What could be wrong about this setup? Why write
performance is so good, but read so bad?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>Of course I tested the hardware itself, made a
filesystem on the raids (without lustre) and ran the same tests, ~1,3GB/s write
1,7GB/s read (800 GB file)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>The infiniband is also okay, used the same
cards/cable for some HPC tests just hours before, everything good. Nevertheless
used the server as a client two (just to bypass the infiniband side as a
potential failure). Nearly the same results (write good, read bad). It’s
like there is a wall at ~510 MB/s read.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>Any ideas or suggestions ? Its really urgent. Thank
you a lot!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>2.6.27.39+lustre1.8.3+0.credativ.lenny.4<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'>Lustre version:
1.8.3-20100410022943-PRISTINE-2.6.27.39+lustre1.8.3+0.credativ.lenny.4<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'> <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span lang=EN-GB style='font-size:
10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<div>
<p class=MsoPlainText style='margin:0cm;margin-bottom:.0001pt'><font size=2
color=black face=Arial><span lang=EN-GB style='font-size:10.0pt;font-family:
Arial;color:black'>sysGen GmbH<o:p></o:p></span></font></p>
<p class=MsoPlainText style='margin:0cm;margin-bottom:.0001pt'><font size=2
color=black face=Arial><span lang=EN-GB style='font-size:10.0pt;font-family:
Arial;color:black'>Sergius Siczek<o:p></o:p></span></font></p>
<p class=MsoPlainText style='margin:0cm;margin-bottom:.0001pt'><font size=2
color=black face=Arial><span lang=EN-GB style='font-size:10.0pt;font-family:
Arial;color:black'>Am Hallacker 48 · 28327 <st1:State w:st="on"><st1:place
w:st="on">Bremen</st1:place></st1:State><o:p></o:p></span></font></p>
<p class=MsoPlainText style='margin:0cm;margin-bottom:.0001pt'><font size=2
color=black face=Arial><span lang=EN-GB style='font-size:10.0pt;font-family:
Arial;color:black'>Tel (0421) 4 09 66-32 · Fax (0421) 4 09 66-33<o:p></o:p></span></font></p>
<p class=MsoPlainText style='margin:0cm;margin-bottom:.0001pt'><font size=2
color=black face=Arial><span lang=EN-GB style='font-size:10.0pt;font-family:
Arial;color:black'><a href="mailto:ssiczek@sysgen.de"
title="mailto:ssiczek@sysgen.de">ssiczek@sysgen.de</a> · <a
href="http://www.sysgen.de/">www.sysgen.de</a><o:p></o:p></span></font></p>
<p class=MsoPlainText style='margin:0cm;margin-bottom:.0001pt'><font size=2
color=black face=Arial><span style='font-size:10.0pt;font-family:Arial;
color:black'>Geschäftsführung: Gabriele Nikisch, Anke Heinßen · Eingetragen
beim Amtsgericht Walsrode · HRB 121943</span></font><font size=2 color=black
face=Arial><span style='font-size:10.0pt;font-family:Arial;color:black'><o:p></o:p></span></font></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
</div>
</body>
</html>