<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I find that “dd” from /dev/zero maxes-out my cpu, so you may want a few threads. You are probably benchmarking CPU here and not disk.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> lustre-discuss-bounces@lists.lustre.org [mailto:lustre-discuss-bounces@lists.lustre.org]
<b>On Behalf Of </b>Andrew Holway<br>
<b>Sent:</b> 19 May 2014 13:45<br>
<b>To:</b> Pardo Diaz, Alfonso<br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [Lustre-discuss] Same performance Infiniband and Ethernet<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Arial","sans-serif";color:#500050">dd if=/dev/zero of=test.dat bs=1M count=1000 oflag=direct</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Arial","sans-serif";color:#500050">oflag=direct forces directIO which is synchronous.</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 19 May 2014 14:41, Pardo Diaz, Alfonso <<a href="mailto:alfonso.pardo@ciemat.es" target="_blank">alfonso.pardo@ciemat.es</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal">thank for your ideas,<br>
<br>
<br>
I have measure the OST RAID performance, and there isn’t a bottleneck in the RAID disk. If I write directly in the RAID I got:<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
dd if=/dev/zero of=test.dat bs=1M count=1000<br>
1000+0 records in<br>
1000+0 records out<o:p></o:p></p>
</div>
<p class="MsoNormal">1048576000 bytes (1,0 GB) copied, 1,34852 s, 778 MB/s<br>
<br>
And If i use /dev/urandom as input file I get the same performance again for infiniband and ethernet connection.<br>
<br>
How can I write directly forgoing cache?<br>
<br>
<br>
Thanks again!<br>
<br>
<br>
<br>
<br>
El 19/05/2014, a las 13:24, Hammitt, Charles Allen <<a href="mailto:chammitt@email.unc.edu">chammitt@email.unc.edu</a>> escribió:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
> Two things:<br>
><br>
> 1) Linux write cache is likely getting in the way; you'd be better off trying to write directly forgoing cache<br>
> 2) you need to write a much bigger file than 1GB; try 50GB<br>
><br>
><br>
> Then as the previous poster said, maybe your disks aren't up to snuff or are misconfigured.<br>
> Also, very interesting, and impossible, to get 154MB/s out of a Single GbE link [128MB/s]. Should be more like 100-115. Less this is 10/40GbE...if so... again, start at #1 and #2.<br>
><br>
><br>
><br>
><br>
> Regards,<br>
><br>
> Charles<br>
><br>
><br>
><br>
><br>
> --<br>
> ===========================================<br>
> Charles Hammitt<br>
> Storage Systems Specialist<br>
> ITS Research Computing @<br>
> The University of North Carolina-CH<br>
> 211 Manning Drive<br>
> Campus Box # 3420, ITS Manning, Room 2504<br>
> Chapel Hill, NC 27599<br>
> ===========================================<br>
><br>
><br>
><br>
><br>
> -----Original Message-----<br>
> From: <a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a> [mailto:<a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a>] On Behalf Of Vsevolod Nikonorov<br>
> Sent: Monday, May 19, 2014 6:54 AM<br>
> To: <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
> Subject: Re: [Lustre-discuss] Same performance Infiniband and Ethernet<br>
><br>
> What disks do your OSTs have? Maybe you have reached your disk performance limit, so Infiniband gives some speedup, but very small. Did you try to enable striping on your Lustre filesystem? For instance, you can type something like this: "lfs setstripe -c
<count of stripes> /mnt/lustre/somefolder" and than copy a file into that folder.<br>
><br>
> Also, there's an opinion that sequence of zeros is not a good way to test a performance, so maybe you should try using /dev/urandom (which is rather slow, so it's better to have a pre-generated "urandom" file in /ram, or /dev/shm, or where your memory space
is mounted to, and copy that file to Lustre filesystem as a test).<br>
><br>
><br>
><br>
> Pardo Diaz, Alfonso писал 2014-05-19 14:33:<br>
>> Hi,<br>
>><br>
>> I have migrated my Lustre 2.2 to 2.5.1 and I have equipped my OSS/MDS<br>
>> and clients with Infiniband QDR interfaces.<br>
>> I have compile lustre with OFED 3.2 and I have configured lnet module<br>
>> with:<br>
>><br>
>> options lent networks=“o2ib(ib0),tcp(eth0)”<br>
>><br>
>><br>
>> But when I try to compare the lustre performance across Infiniband<br>
>> (o2ib), I get the same performance than across ethernet (tcp):<br>
>><br>
>> INFINIBAND TEST:<br>
>> dd if=/dev/zero of=test.dat bs=1M count=1000<br>
>> 1000+0 records in<br>
>> 1000+0 records out<br>
>> 1048576000 bytes (1,0 GB) copied, 5,88433 s, 178 MB/s<br>
>><br>
>> ETHERNET TEST:<br>
>> dd if=/dev/zero of=test.dat bs=1M count=1000<br>
>> 1000+0 records in<br>
>> 1000+0 records out<br>
>> 1048576000 bytes (1,0 GB) copied, 5,97423 s, 154 MB/s<br>
>><br>
>><br>
>> And this is my scenario:<br>
>><br>
>> - 1 MDs with SSD RAID10 MDT<br>
>> - 10 OSS with 2 OST per OSS<br>
>> - Infiniband interface in connected mode<br>
>> - Centos 6.5<br>
>> - Lustre 2.5.1<br>
>> - Striped filesystem “lfs setstripe -s 1M -c 10"<br>
>><br>
>><br>
>> I know my infiniband running correctly, because if I use IPERF3<br>
>> between client and servers I got 40Gb/s by infiniband and 1Gb/s by<br>
>> ethernet connections.<br>
>><br>
>><br>
>><br>
>> Could you help me?<br>
>><br>
><br>
>><br>
>> Regards,<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> Alfonso Pardo Diaz<br>
>> System Administrator / Researcher<br>
>> c/ Sola nº 1; 10200 Trujillo, ESPAÑA<br>
>> Tel: +34 927 65 93 17 Fax: +34 927 32 32 37<br>
>><br>
>><br>
>><br>
>><br>
>> ----------------------------<br>
>> Confidencialidad:<br>
>> Este mensaje y sus ficheros adjuntos se dirige exclusivamente a su<br>
>> destinatario y puede contener información privilegiada o confidencial.<br>
>> Si no es vd. el destinatario indicado, queda notificado de que la<br>
>> utilización, divulgación y/o copia sin autorización está prohibida en<br>
>> virtud de la legislación vigente. Si ha recibido este mensaje por<br>
>> error, le rogamos que nos lo comunique inmediatamente respondiendo al<br>
>> mensaje y proceda a su destrucción.<br>
>><br>
>> Disclaimer:<br>
>> This message and its attached files is intended exclusively for its<br>
>> recipients and may contain confidential information. If you received<br>
>> this e-mail in error you are hereby notified that any dissemination,<br>
>> copy or disclosure of this communication is strictly prohibited and<br>
>> may be unlawful. In this case, please notify us by a reply and delete<br>
>> this email and its contents immediately.<br>
>> ----------------------------<br>
>><br>
>> _______________________________________________<br>
>> Lustre-discuss mailing list<br>
>> <a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
>> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">
http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
>><br>
>> Email secured by Check Point<br>
><br>
> --<br>
> Никоноров Всеволод Дмитриевич, ОИТТиС, НИКИЭТ<br>
><br>
> Vsevolod D. Nikonorov, JSC NIKET<br>
><br>
> Email secured by Check Point<br>
> _______________________________________________<br>
> Lustre-discuss mailing list<br>
> <a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">
http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
> _______________________________________________<br>
> Lustre-discuss mailing list<br>
> <a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">
http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br>
_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>