[Lustre-discuss] Infiniband VS 1GiG Transfer rates. Confused

[Jagga Soorma]

Hi Guys,

I have setup a new cluster with a infiniband interconnect and I am a bit
confused with the performance I am getting as far as transfer rates.  Is
there something that I am missing here?  Shouldn't the transfer rate over
the ib interface be much faster and not the same compared with the 1G
(bonded mode=6) interface?

(I am new to infiniband but this does not seem right.  My routing is also
setup correctly.  Not sure what I might be missing here):

--
hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso root at hpc103
:/var/tmp/
SLES-11-DVD-x86_64-GM-DVD1.iso
100% 2749MB  62.5MB/s   00:44

real    0m43.882s
user    0m29.482s
sys    0m6.736s

hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso root at hpc103-ib
:/var/tmp/
SLES-11-DVD-x86_64-GM-DVD1.iso
100% 2749MB  80.9MB/s   00:34

real    0m35.757s
user    0m24.498s
sys    0m6.292s

hpc101:/var/tmp # netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt
Iface
10.0.250.0      0.0.0.0         255.255.255.0   U         0 0          0 ib0
128.137.126.0   0.0.0.0         255.255.255.0   U         0 0          0
bond0
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
0.0.0.0         128.137.126.253 0.0.0.0         UG        0 0          0
bond0

hpc101:/var/tmp # ifconfig ib0
ib0       Link encap:UNSPEC  HWaddr
80-00-00-48-FE-80-00-00-00-00-00-00-00-00-00-00
          inet addr:10.0.250.5  Bcast:10.0.250.255  Mask:255.255.255.0
          inet6 addr: fe80::223:7dff:ff93:bf4d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
          RX packets:493685 errors:0 dropped:0 overruns:0 frame:0
          TX packets:890402 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:256
          RX bytes:22848079 (21.7 Mb)  TX bytes:7349365631 (7008.9 Mb)
--

Thanks in advance for your assistance.

Regards,
-J
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100210/af8a2d75/attachment.htm>