[Lustre-discuss] Infiniband VS 1GiG Transfer rates. Confused

[Erik Froese]

Jagga,
I think this is more a function of scp than than ib in general. Have
you tried using the HPN-SSH patches?
http://www.psc.edu/networking/projects/hpn-ssh/

You could also try using apache to serve the iso over HTTP to see if
SCP is the bottleneck.
Also I think IPoIB should be able to use a much larger MTU. It may
help for large files.

Do this on both machines: ifconfig ib0 mtu 65520
To benchmark the raw infiniband performance see the man pages for
ib_read_bw and ib_write_bw

Erik

On Wed, Feb 10, 2010 at 7:48 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>
> Hi Guys,
>
> I have setup a new cluster with a infiniband interconnect and I am a bit confused with the performance I am getting as far as transfer rates.  Is there something that I am missing here?  Shouldn't the transfer rate over the ib interface be much faster and not the same compared with the 1G (bonded mode=6) interface?
>
> (I am new to infiniband but this does not seem right.  My routing is also setup correctly.  Not sure what I might be missing here):
>
> --
> hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso root at hpc103:/var/tmp/
> SLES-11-DVD-x86_64-GM-DVD1.iso                                                                                                                                                                             100% 2749MB  62.5MB/s   00:44
>
> real    0m43.882s
> user    0m29.482s
> sys    0m6.736s
>
> hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso root at hpc103-ib:/var/tmp/
> SLES-11-DVD-x86_64-GM-DVD1.iso                                                                                                                                                                             100% 2749MB  80.9MB/s   00:34
>
> real    0m35.757s
> user    0m24.498s
> sys    0m6.292s
>
> hpc101:/var/tmp # netstat -rn
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
> 10.0.250.0      0.0.0.0         255.255.255.0   U         0 0          0 ib0
> 128.137.126.0   0.0.0.0         255.255.255.0   U         0 0          0 bond0
> 127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
> 0.0.0.0         128.137.126.253 0.0.0.0         UG        0 0          0 bond0
>
> hpc101:/var/tmp # ifconfig ib0
> ib0       Link encap:UNSPEC  HWaddr 80-00-00-48-FE-80-00-00-00-00-00-00-00-00-00-00
>           inet addr:10.0.250.5  Bcast:10.0.250.255  Mask:255.255.255.0
>           inet6 addr: fe80::223:7dff:ff93:bf4d/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
>           RX packets:493685 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:890402 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:256
>           RX bytes:22848079 (21.7 Mb)  TX bytes:7349365631 (7008.9 Mb)
> --
>
> Thanks in advance for your assistance.
>
> Regards,
> -J
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>