[Lustre-discuss] problems getting o2ib device recognized

[murray smigel]

Hi,
I am running Centos5 with 2.6.18-8.1.14.el5_lustre.1.6.4.1 as the OS.
I built the custom kernel against the patched source and built
lustre-1.6.1 (and 1.6.3) against the source tree as:
./configure --with-linux=/usr/src/linux-2.6.18-8.1.14.el5_lustre.1.6.4.1 
--with-o2ib=/public/ofed/SRC_1.3/ofa_kernel-1.3/

The OFED source is stock 1.3.
The hardware is x86-64.

----------------------------------------------------------------------------
Problems in the build:
needed to edit lnet/klnds/o2iblnd/o2iblnd.h to add:
#define DECLARE_PCI_UNMAP_ADDR(ADDR_NAME) dma_addr_t ADDR_NAME;

needed to edit lnet/klnds/o2iblnd/o2iblnd.c to add 0 as last argument:
        cq = ib_create_cq(cmid->device,
                       kiblnd_cq_completion, kiblnd_cq_event, conn,
                       IBLND_CQ_ENTRIES(),
                       0);        // added by me.

in ldiskfs/ldiskfs/linux-stage/fs/ext3/iopen.c I had
to comment out call to d_move_locked

For some unknown reason I have to 
modprobe -f ko2iblnd
since there is some sort of version conflict, but it does go in:

[root at lusty lustre-1.6.4.1]# lsmod | grep ko2iblnd
ko2iblnd              137504  0
rdma_cm                67348  1 ko2iblnd
ib_core                97664  6 ko2iblnd,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad
lnet                  252840  4 ko2iblnd,ptlrpc,obdclass,ksocklnd
libcfs                189656  13 
ko2iblnd,mds,fsfilt_ldiskfs,mgs,mgc,lov,lquota,mdc,ptlrpc,obdclass,lvfs,
ksocklnd,lnet

--------------------------------------------------------------------------
After these changes the lustre modules built and installed.
I edit /etc/modprobe.conf to have
options lnet 'networks="tcp(eth0,eth1),o2ib(ib0)"'

I can modprobe in lnet, but I only see the tcp device
[root at lusty lustre-1.6.4.1]# lctl list_nids
192.168.1.94 at tcp

The lustre setup works over tcp, but that is not really what I need!
Any help appreciated,
thanks,
murray