[Lustre-discuss] Using Infiniband QoS with Lustre 1.8.5

[Ramiro Alba]

Hi everybody,

We have a 128 nodes (8 cores/node) 4x DDR IB cluster with 2:1
oversubscription and I use the IB net for:

- OpenMPI
- Lustre
- Admin (may change in future)

I'am very interested in using IB QoS, as in the near future I'm
deploying ADM processors having then 24 cores /node so I want to put a
barrier to trafic so as no trafic (specially OpenMPI) is starved by
others (specially Lustre I/O). So I read all the documentation I could
get 
(http://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg04092.html was really very helpful)
and made the configuration showed bellow.

I'll really be very grateful if someone in the the list could tell me
his/her opinion on the proposed configuration bellow. Any comment will
be welcomed, even if the whole think is a complete nonsense, as no one
in my zone (as far as I know) is using IB and QoS and is really painful.

Personal doubts:

- Am I taking properly into account 'latency' considerations for ?
- Any need to define 'QoS Switch Port 0 options'?.
- Is it interesting to make a difference for CAs and switches external
ports configuration?
- Not really very important to follow strictly the rule 'the weighting
values for each VL should be multiples of 64', at least in vlarb_high?
- Other 'weights suggested?

Thanks in Advance


----- /etc/opensm/qos-policy.conf --------------------


# SL asignation to Flows. GUIDs are Port GUIDs
qos-ulps
    default                                 :0    # default SL (OPENMPI)
    any, target-port-guid 0x0002c90200279295  :1    # SL for Lustre MDT
    any, target-port-guid 0x0002c9020029fda9,0x0002c90200285ed5    :2
# SL for Lustre OSTs
    ipoib                             :3    # SL for Administration 
end-qos-ulps

----- /etc/opensm/opensm.conf -----------------------

#
# QoS OPTIONS
#
# Enable QoS setup
qos FALSE

# QoS policy file to be used
qos_policy_file /etc/opensm/qos-policy.conf

# QoS default options
qos_max_vls 4
qos_high_limit 4
qos_vlarb_high 0:128,1:64,2:0,3:0
qos_vlarb_low 0:192,1:16,2:64,3:8
qos_sl2vl 0,1,2,3,15,15,15,15,15,15,15,15,15,15,15,15

# QoS CA options
qos_max_vls 4
qos_high_limit 4
qos_vlarb_high 0:128,1:64,2:0,3:0
qos_vlarb_low 0:192,1:16,2:64,3:8
qos_sl2vl 0,1,2,3,15,15,15,15,15,15,15,15,15,15,15,15

# QoS Switch Port 0 options
#qos_sw0_max_vls 0
#qos_sw0_high_limit -1
#qos_sw0_vlarb_high (null)
#qos_sw0_vlarb_low (null)
#qos_sw0_sl2vl (null)

# QoS Switch external ports options
qos_swe_max_vls 4
qos_swe_high_limit 255
qos_swe_vlarb_high 0:192,1:16,2:64,3:8
qos_swe_vlarb_low 0:0,1:0,2:0,3:0
qos_swe_sl2vl 0,1,2,3,15,15,15,15,15,15,15,15,15,15,15,15


-- 
Ramiro Alba

Centre Tecnològic de Tranferència de Calor
http://www.cttc.upc.edu


Escola Tècnica Superior d'Enginyeries
Industrial i Aeronà utica de Terrassa
Colom 11, E-08222, Terrassa, Barcelona, Spain
Tel: (+34) 93 739 86 46



-- 
Aquest missatge ha estat analitzat per MailScanner
a la cerca de virus i d'altres continguts perillosos,
i es considera que està net.