[Lustre-discuss] Same performance Infiniband and Ethernet

[Pardo Diaz, Alfonso]

Hi,

I have migrated my Lustre 2.2 to 2.5.1 and I have equipped my OSS/MDS and clients with Infiniband QDR interfaces.
I have compile lustre with OFED 3.2 and I have configured lnet module with:

options lent networks=“o2ib(ib0),tcp(eth0)”


But when I try to compare the lustre performance across Infiniband (o2ib), I get the same performance than across ethernet (tcp):

INFINIBAND TEST:
dd if=/dev/zero of=test.dat bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1,0 GB) copied, 5,88433 s, 178 MB/s

ETHERNET TEST:
dd if=/dev/zero of=test.dat bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1,0 GB) copied, 5,97423 s, 154 MB/s


And this is my scenario:

- 1 MDs with SSD RAID10 MDT
- 10 OSS with 2 OST per OSS
- Infiniband interface in connected mode
- Centos 6.5
- Lustre 2.5.1
- Striped filesystem “lfs setstripe -s 1M -c 10"


I know my infiniband running correctly, because if I use IPERF3 between client and servers I got 40Gb/s by infiniband and 1Gb/s by ethernet connections.



Could you help me?


Regards,





Alfonso Pardo Diaz
System Administrator / Researcher
c/ Sola nº 1; 10200 Trujillo, ESPAÑA
Tel: +34 927 65 93 17 Fax: +34 927 32 32 37




----------------------------
Confidencialidad: 
Este mensaje y sus ficheros adjuntos se dirige exclusivamente a su destinatario y puede contener información privilegiada o confidencial. Si no es vd. el destinatario indicado, queda notificado de que la utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente respondiendo al mensaje y proceda a su destrucción.

Disclaimer: 
This message and its attached files is intended exclusively for its recipients and may contain confidential information. If you received this e-mail in error you are hereby notified that any dissemination, copy or disclosure of this communication is strictly prohibited and may be unlawful. In this case, please notify us by a reply and delete this email and its contents immediately. 
----------------------------