[Lustre-discuss] Unable to build kernel-ib when building lustre from source code

[neutron]

Hi all,

I have installed Lustre-1.8.1.1  from pre-built rpm packages and it
works. Now I want to build Lustre 1.8.1.1 from source on RHEL5
x86_64.  Since I want to learn Lustre( and eventually make some
modifications in lustre),   I choose to build Lustre from source code.


I followed the steps at "Installing Lustre with a Third-Party Network
Stack" from "lustre manual 1.8".
Here is what I do.  At my ~/.rpmmacros,  I set my rpm working dir to
be ~/rpmbuild

(1) Install "kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm".
The manual says to use "kernel-lustre-source-xxxx.rpm", but I couldn't
find that file at Sun's download site.
This src.rpm is installed to ~/rpmbuild/SOURCE

(2) At ~/rpmbuild/SPECS,  Run "rpmbuild -bp --target=x86_64  ./kernel-2.6.spec"
The kernel src tree is at
"~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64"

(3) In that dir,  I use the config file of currently running kernel, and then:
	$ make oldconfig || make menuconfig
	$ make include/asm
	$ make include/linux/version.h
	$ make SUBDIRS=scripts
	$ make;  make modules_install; make install

This should be identical to "make rpm" and then install the
kernel-xxx.rpm and kernel-xxx.src.rpm.

(4) Reboot into the new lustre kernel
"uname -r" gives  "2.6.18-128.7.1-lustre-1.8.1-customer".
I have verified that
"/lib/modules/2.6.18-128.7.1-lustre-1.8.1-customer/build" point to
"~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64"


(5) Install the
"kernel-ib-devel-1.4.2-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm".
The files are installed to "/usr/src/ofa_kernel-1.4.2".
 At that place, I run
	$ /configure --with-core-mod --with-ipoib-mod --with-mthca-mod
--with-mlx4_core-mod --with-mlx4_inf-mod
	$ make

Then I see lots of errors like:
	/usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/fs.h:9:
error: redefinition of ‘inc_nlink’
include/linux/fs.h:1255: error: previous definition of ‘inc_nlink’ was here
In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:44,
                 from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38,
                 from /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49:
/usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/mm.h:17:
error: redefinition of ‘is_vmalloc_addr’
include/linux/mm.h:332: error: previous definition of ‘is_vmalloc_addr’ was here
In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38,
                 from /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49:
/usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h: In function
‘ib_dma_mapping_error’:
/usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:1593: warning:
passing argument 1 of ‘dma_mapping_error’ makes integer from pointer
without a cast


I googled the error, and someone said to use OFED-1.5.
But I tried with OFED-1.4.2, OFED-1.5, and OFED-1.5.1 daily tar ball
OFED-1.5.1-20100131-0600,
and always see the same error "redefinition of ........"   when I'm
compiling the kernel-ib.

Am I wrong with the way to install kernel-ib or kernel-lustre?  How
can I make lustre work with IB support?


(6) Build lustre succeeds without errors.
I untared "lustre-1.8.1.1.tar.gz", then
        $ ./configure
--with-linux=~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64
	$ make
	$ make install

Looks both lustre kernel modules and luster userland tools/library are
installed correctly.
Is this a right way to install lustre?

Thanks!



-Neutron