[Lustre-discuss] Infiniband VS 1GiG Transfer rates. Confused

Peter Kjellstrom cap at nsc.liu.se
Thu Feb 11 02:09:48 PST 2010


On Thursday 11 February 2010, Erik Froese wrote:
> Jagga,
> I think this is more a function of scp than than ib in general. Have
> you tried using the HPN-SSH patches?
> http://www.psc.edu/networking/projects/hpn-ssh/
>
> You could also try using apache to serve the iso over HTTP to see if
> SCP is the bottleneck.

Iperf and netpipe are examples of tools made for this purpose.

All this is, however, IPoIB which is _not_ what you'd typically use to run IB 
at high speed. Running Lustre on IB is typically done on verbs (lnet o2ib, 
not tcp).

/Peter

> Also I think IPoIB should be able to use a much larger MTU. It may
> help for large files.
>
> Do this on both machines: ifconfig ib0 mtu 65520
> To benchmark the raw infiniband performance see the man pages for
> ib_read_bw and ib_write_bw
>
> Erik
>
> On Wed, Feb 10, 2010 at 7:48 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
> > Hi Guys,
> >
> > I have setup a new cluster with a infiniband interconnect and I am a bit
> > confused with the performance I am getting as far as transfer rates.  Is
> > there something that I am missing here?  Shouldn't the transfer rate over
> > the ib interface be much faster and not the same compared with the 1G
> > (bonded mode=6) interface?
> >
> > (I am new to infiniband but this does not seem right.  My routing is also
> > setup correctly.  Not sure what I might be missing here):
> >
> > --
> > hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso
> > root at hpc103:/var/tmp/
> > SLES-11-DVD-x86_64-GM-DVD1.iso                                           
> >                                                                          
> >                                                        100% 2749MB 
> > 62.5MB/s   00:44
> >
> > real    0m43.882s
> > user    0m29.482s
> > sys    0m6.736s
> >
> > hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso
> > root at hpc103-ib:/var/tmp/
> > SLES-11-DVD-x86_64-GM-DVD1.iso                                           
> >                                                                          
> >                                                        100% 2749MB 
> > 80.9MB/s   00:34
> >
> > real    0m35.757s
> > user    0m24.498s
> > sys    0m6.292s
...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100211/8f4eae4c/attachment.pgp>


More information about the lustre-discuss mailing list