[Lustre-discuss] Lustre over o2ib issue

Mike Hanby mhanby at uab.edu
Wed Mar 16 14:04:14 PDT 2011


Thanks, I forgot to include the card info:

The servers each have a single IB card: dual port MT26528 QDR
o2ib0(ib0) on each server is attached to the QLogic switch (with three attached M3601Q switches 48 attached blades)
o2ib1(ib1) on each server is attached to a stack of two M3601Q switches with 24 attached blades

The blades connected to o2ib0 each have an MT26428 QDR IB card
The blades connected to o2ib1 each have an MT25418 DDR IB card 


-----Original Message-----
From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Nirmal Seenu
Sent: Wednesday, March 16, 2011 2:10 PM
To: lustre-discuss at lists.lustre.org
Subject: Re: [Lustre-discuss] Lustre over o2ib issue

If you are using DDR and QDR or any 2 different cards cards in the same machine there is no guarantee that the same IB cards get assigned to ib0 and ib.

To fix that problem you need to comment out the following 3 lines /etc/init.d/openibd:

     #for i in `grep "^driver: " /etc/sysconfig/hwconf | sed -e 's/driver: //' | grep -w "ib_mthca\\\|ib_ipath\\\|mlx4_core\\\|cxgb3\\\|iw_nes"`; do
     #    load_modules $i
     #done

and include the following lines instead(we wanted the DDR card to be ib0 and the QDR card to be ib1):
     load_modules ib_mthca
     /bin/sleep 10
     load_modules mlx4_core

and you will need to restart openibd once again (we included it in rc.local) to make sure that the same IB cards are assigned to the devices ib0 and ib1.

Nirmal
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss



More information about the lustre-discuss mailing list