<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:2 7 4 9 2 2 5 2 4 4;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Arial","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-AU link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hi Brian,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>With one 4x QDR IB port, you can achieve 2 GB/Sec on single client, multi-threaded workload provided that you have right storage (with enough bandwidth) at other end. We have tested this multiple times at DDN. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I have seen sites that do IB-bonding across 2 ports but mostly in failover configuration. To get 10GB/Sec to a single node requires aggregating 5 QDR IB ports. You will need to confirm from your IB vendor (Mellanox? ), OS vendor (SGI/RedHat/Novell) and Lustre vendor whether they support aggregating so many links. I think the challenge you will have is to find a Lustre client node that has enough x8 PCIe slots to sustain 3 dual-port Infiniband adapters at full rate (think multiple such nodes in a typical Lustre filesystem, not so economical). Other alternative is to find a server that can support 8X or 12X QDR IB port on the motherboard to get more bandwidth. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>With a typical Lustre client memory of 24-64GB and memory to CPU bandwidth of 10GB/Sec (with standard DDR3-1333MHz DIMMS), it is not possible to fit dataset larger than 2/3<sup>rd</sup> of memory. If you still want to achieve 10GB/Sec of bandwidth between storage and memory, there are clever alternatives. You will have to stage your data into memory beforehand and keep memory pages locked and continue feeding data as these pages are consumed. It is lot harder than it seems on the paper. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Cheers,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>-Atul<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> lustre-discuss-bounces@lists.lustre.org [mailto:lustre-discuss-bounces@lists.lustre.org] <b>On Behalf Of </b>Brian O'Connor<br><b>Sent:</b> Monday, 21 March 2011 9:23 AM<br><b>To:</b> lustre-discuss@lists.lustre.org<br><b>Subject:</b> [Lustre-discuss] Multiple IB ports<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Hi,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'> Any body actually using multiple IB ports on a client for an aggregated connection?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Ie. Many oss with one qdr IB each. Clients with 4 qdr IB ports. Assuming the normal<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'>issues with bus bandwidth etc, what sort of perf can I expect <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'>qdr ~ 3-4Gbytes/Sec<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'>I’m trying to size a cluster and clients to get ~10GBytes/Sec on *<b>one</b>* <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'>client node. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'>If I can aggregate IB linearly the next step will be to try and figure out<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'>How to get 10Gigabytes/s to local storage </span><span lang=EN-US style='font-size:10.0pt;font-family:Wingdings'>L</span><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Some times customers are crazy…….<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Courier'>Brian O'Connor</span><span lang=EN-US><o:p></o:p></span></p><div><p class=MsoNormal><span lang=EN-US> <o:p></o:p></span></p></div><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Courier'>-------------------------------------------------</span><span lang=EN-US><o:p></o:p></span></p><div><p class=MsoNormal><span lang=EN-US> <o:p></o:p></span></p></div><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Courier'>SGI Consulting</span><span lang=EN-US><o:p></o:p></span></p><div><p class=MsoNormal><span lang=EN-US> <o:p></o:p></span></p></div><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Courier'>Email: </span><span lang=EN-US><a href="mailto:briano@sgi.com"><span style='font-size:10.0pt;font-family:Courier'>briano@sgi.com</span></a></span><span lang=EN-US style='font-size:10.0pt;font-family:Courier'>, Mobile +61 417 746 452</span><span lang=EN-US><o:p></o:p></span></p><div><p class=MsoNormal><span lang=EN-US> <o:p></o:p></span></p></div><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Courier'>Phone: +61 3 9963 1900, Fax: +61 3 9963 1902</span><span lang=EN-US><o:p></o:p></span></p><div><p class=MsoNormal><span lang=EN-US> <o:p></o:p></span></p></div><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Courier'>357 Camberwell Road, Camberwell, Victoria, 3124 </span><span lang=EN-US><o:p></o:p></span></p><div><p class=MsoNormal><span lang=EN-US> <o:p></o:p></span></p></div><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Courier'>AUSTRALIA </span><span lang=EN-US><a href="http://www.sgi.com/support/services"><span style='font-size:10.0pt;font-family:Courier'>http://www.sgi.com/support/services</span></a><o:p></o:p></span></p><div><p class=MsoNormal><span lang=EN-US> <o:p></o:p></span></p></div><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Courier'>-------------------------------------------------</span><span lang=EN-US><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div></body></html>