[Lustre-discuss] Lustre and system cpu usage

Claudio Baeza Retamal claudio at dim.uchile.cl
Mon Mar 14 17:57:25 PDT 2011


Dears folks,

last month, I have configured lustre 1.8.5 over infiniband, before, I 
was using  Gluster 3.1.2, performance was ok but reliability was wrong, 
when 40 or more applications requested at the same time  for open a 
file, gluster servers  bounced randomly the active connections from 
clients. Lustre has not this problem, but I can see others issues, for 
example, namd appears with "system cpu"  around of 30%,  hpl benchmark  
appears between  70%-80% of "system cpu", is too much high, with 
gluster, the system cpu was never exceeded 5%. I think, this is 
explained due gluster uses fuse and run in user space, but I am do not 
sure. I have some doubt:

¿why Lustre uses ipoib? Before, with gluster  I do not use ipoib, I am 
thinking  that ipoib module produces bad performance in infiniband and 
disturbs the infiniband native module.
It is posible to configure lustre to  transport metada over ethernet and 
data over infiniband?

For namd and hpl benchmark, is  it normal to have system cpu to be so high?

My configuration is the following:

- Qlogic 12800-180 switch, 7 leaf (24 ports per  leaf) and 2 spines (All 
ports have QDR, 40 Gbps)
- 66 HCA mellanox connectX, two ports, QDR 40 Gbps (compute nodes)
- 1 metadata server, 96 GB RAM DDR3 optimized for performance, two Xeon 
5570, SAS 15K RPM  hard disk in Raid 1, HCA mellanox connectX with two ports
- 4 OSS with 1 OST of 2 TB in RAID 5 each one (8 TB in total). The all 
OSS have a Mellanox ConnectX with two ports

I will appreciate any help or tips.

Regards

claudio


-- 
Claudio Baeza Retamal
CTO
National Laboratory for High Performance Computing (NLHPC)
Center for Mathematical Modeling (CMM)
School of Engineering and Sciences
Universidad de Chile






More information about the lustre-discuss mailing list