[Lustre-discuss] Lustre 1.6.4.3 + OFED 1.2.5.5 + RHEL 4u4 AS
cxmtnbike
rcummins64 at gmail.com
Tue Jul 1 05:32:36 PDT 2008
It doesn't appear as if you've come up with a solution to this
problem. I too have run into the same set of issues as you appear to
have here, but I think I have resolved them. I am running SLES 10 SP1
but this should also work for you.
first patch lnet/klnds/o2iblnd/o2iblnd.h You'll need to add the
following line somewhere near the top:
#include <linux/pci.h>
Next you'll need to patch lnet/klnds/o2iblnd/o2iblnd.c. ib_create_cq
uses 6 args in OFED-1.2.5.5 but o2iblnd.c only has 5 in the call. The
new line should look like this:
cq = ib_create_cq(cmid->device,
kiblnd_cq_completion, kiblnd_cq_event, conn,
IBLND_CQ_ENTRIES(), 0);
After applying the two changes mentioned above I have been able to
make though 'make rpms' finishes it is a tad chatty with warning
messages.
On May 15, 12:37 pm, Malcolm Cowe <Malcolm.C... at Sun.COM> wrote:
> Hi Folks,
>
> Having some trouble with building Lustre 1.6.4.3 with OFED 1.2.5.5 on a
> RHEL 4u4 AS server. Could somebody please help me to understand where
> I've gone wrong? Here's what I have done so far:
>
> 1. Install RHEL 4u4 AS (full installation).
>
> 2. Download the Lustre RPMs from sun.com:
>
> e2fsprogs-1.40.4.cfs1-0redhat.x86_64.rpm
> e2fsprogs-devel-1.40.4.cfs1-0redhat.x86_64.rpm
> kernel-lustre-smp-2.6.9-67.0.4.EL_lustre.1.6.4.3.x86_64.rpm
> kernel-lustre-source-2.6.9-67.0.4.EL_lustre.1.6.4.3.x86_64.rpm
> lustre-1.6.4.3-2.6.9_67.0.4.EL_lustre.1.6.4.3smp.x86_64.rpm
> lustre-modules-1.6.4.3-2.6.9_67.0.4.EL_lustre.1.6.4.3smp.x86_64.rpm
> lustre-source-1.6.4.3-2.6.9_67.0.4.EL_lustre.1.6.4.3smp.x86_64.rpm
> plus: lustre-1.6.4.3.tar.gz
>
> 3. Install kernel-lustre-smp and kernel-lustre-source rpms.
> - Change grub to boot from lustre patched kernel by default.
> - Reboot.
>
> 4. Download OFED distribution from openib.org:
>
> OFED-1.2.5.5.tgz
>
> 5. Extract OFED distribution.
>
> 6. Install OFED:
>
> cd OFED-1.2.5.5/
> ./install.sh
> 2) Install OFED Software
> 3) All packages (all of Basic, HPC)
> [accept defaults for everything, configure IPoIB IP address].
>
> 7. Reboot.
>
> 8. Modify "Module.symvers", removing all references to Infiniband
> modules supplied with the kernel distribution. N.B. Could not find this
> file in the lustre kernel source tree, only in the "-obj" tree.
>
> vi /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3-obj/x86_64/smp/Module.symvers
>
> 9. Run "/usr/share/doc/ofed-docs-1.2.5.5/create_Module.symvers.sh" and
> append the resulting file to the existing Module.symvers file:
>
> /usr/share/doc/ofed-docs-1.2.5.5/create_Module.symvers.sh
> cat Module.symvers >> /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3-obj/x86_64/smp/Module.symvers
>
> 10. Change into the lustre kernel source and edit the Makefile. Change
> "custom" suffix to "smp" in the variable "EXTRAVERSION".
>
> 11. Change into the lustre kernel source and run the setup commands:
>
> cd /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3
> [linux]$ cp /boot/config-`uname -r` .config
> [linux]$ make oldconfig || make menuconfig
> # For 2.6 kernels
> [linux]$ make include/asm
> [linux]$ make include/linux/version.h
> [linux]$ make SUBDIRS=scripts
>
> 12. Extract the lustre source distribution (using lustre-1.6.4.3.tar.gz
> rather than the RPM):
>
> tar zxf lustre-1.6.4.3.tar.gz
>
> 13. Run the configure script:
>
> cd lustre-1.6.4.3/
> ./configure --with-linux=/usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3 --with-o2ib=/usr/src/ofa_kernel
>
> N.B. Cannot include "--with-linux-obj=" option as the configure script
> exits with an error and a recommendation to run "make config" in the
> linux src tree:
>
> checking for /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3-obj/include/linux/autoconf.h... no
> configure: error: Run make config in /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3.
>
> If I do this, the build fails very early on:
>
> [root at node006 lustre-1.6.4.3]# make
> test -d CVS || exit 0; \
> list=""; for mod in $list; do \
> perl ./build/kabi -v archive $HOME/nonfree $mod || exit $?; \
> done
> make all-recursive
> make[1]: Entering directory `/root/HPC/build/lustre-1.6.4.3'
> Making all in ldiskfs
> make[2]: Entering directory `/root/HPC/build/lustre-1.6.4.3/ldiskfs'
> test -d CVS || exit 0; \
> list=""; for mod in $list; do \
> perl ./build/kabi -v archive $HOME/nonfree $mod || exit $?; \
> done
> make all-recursive
> make[3]: Entering directory `/root/HPC/build/lustre-1.6.4.3/ldiskfs'
> Making all in .
> make[4]: Entering directory `/root/HPC/build/lustre-1.6.4.3/ldiskfs'
> for dir in ldiskfs ; do \
> make sources -C $dir || exit $? ; \
> done
> make[5]: Entering directory `/root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs'
> rm -rf linux-stage linux sources
> mkdir -p linux-stage/fs/ext3 linux-stage/include/linux
> cp /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/acl.c /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/balloc.c /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/bitmap.c /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/dir.c
> /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/file.c /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/fsync.c /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/hash.c /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/ialloc.c
> /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/inode.c /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/ioctl.c /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/namei.c /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/resize.c
> /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/super.c /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/symlink.c /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/xattr.c
> /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/xattr_security.c /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/xattr_trusted.c /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/xattr_user.c
> /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/acl.h /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/xattr.h /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/fs/ext3/Makefile linux-stage/fs/ext3
> cp /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/include/linux/ext3_fs.h /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/include/linux/ext3_fs_i.h /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/include/linux/ext3_fs_sb.h
> /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/include/linux/ext3_jbd.h linux-stage/include/linux
> Applying ext3 patches: ext3-wantedi-2.6-rhel4.patch iopen-2.6-rhel4.patch export_symbols-ext3-2.6-suse.patch ext3-map_inode_page-2.6-suse.patch ext3-ea-in-inode-2.6-rhel4.patch export-ext3-2.6-rhel4.patch ext3-include-fixes-2.6-rhel4.patch
> ext3-extents-2.6.9-rhel4.patch ext3-extents-fixes-2.6.9-rhel4.patch ext3-extents-multiblock-directio-2.6.9-rhel4.patch ext3-extents-search-2.6.9-rhel4.patch ext3-extents-sanity-checks.patch ext3-mballoc3-core.patch ext3-mballoc3-rhel4.patch
> ext3-nlinks-2.6.9.patch ext3-ialloc-2.6.patch ext3-lookup-dotdot-2.6.9.patch ext3-sector_t-overflow-2.6.9-rhel4.patch ext3-check-jbd-errors-2.6.9.patch ext3-uninit-2.6.9.patch ext3-nanosecond-2.6-rhel4.patch ext3-unlink-race.patch ext3-mmp-2.6-rhel4.patch
> ext3-fiemap-2.6-sles10.patch
> mkdir linux
> Replacing 'ext3' with 'ldiskfs': acl.h xattr.h acl.c balloc.c bitmap.c dir.c file.c fsync.c hash.c ialloc.c inode.c ioctl.c namei.c resize.c super.c symlink.c xattr.c xattr_security.c xattr_trusted.c xattr_user.c iopen.c iopen.h extents.c mballoc.c group.h
> fiemap.h ext3_fs.h ext3_fs_i.h ext3_fs_sb.h ext3_jbd.h ext3_extents.h
> touch sources
> make[5]: Leaving directory `/root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs'
> make CC="gcc -m64" -C /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3 \
> -f /root/HPC/build/lustre-1.6.4.3/ldiskfs/build/Makefile LUSTRE_LINUX_CONFIG=/usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/.config \
> LINUXINCLUDE=' -Iinclude $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) -include include/linux/autoconf.h' \
> M=/root/HPC/build/lustre-1.6.4.3/ldiskfs -o tmp_include_depends -o scripts -o \
> include/config/MARKER modules
> make[5]: Entering directory `/usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3'
> LD /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/built-in.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/acl.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/balloc.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/bitmap.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/dir.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/file.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/fsync.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/hash.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/ialloc.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/inode.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/ioctl.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/namei.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/resize.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/super.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/symlink.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/xattr.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/xattr_security.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/xattr_trusted.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/xattr_user.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/iopen.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/extents.o
> CC [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/mballoc.o
> LD [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/ldiskfs.o
> Building modules, stage 2.
> MODPOST
> CC /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/ldiskfs.mod.o
> LD [M] /root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs/ldiskfs.ko
> make[5]: Leaving directory `/usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3'
> make[4]: Leaving directory `/root/HPC/build/lustre-1.6.4.3/ldiskfs'
> Making all in ldiskfs
> make[4]: Entering directory `/root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs'
> cd .. ; /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/scripts/mod/modpost -m -i /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/Module.symvers \
> -o /usr/src/linux-2.6.9-67.0.4.EL_lustre.1.6.4.3/Module.symvers ldiskfs/ldiskfs.o
> test -d CVS || exit 0; \
> list=""; for mod in $list; do \
> perl ../build/kabi -v archive $HOME/nonfree $mod || exit $?; \
> done
> make[4]: Leaving directory `/root/HPC/build/lustre-1.6.4.3/ldiskfs/ldiskfs'
> make[3]: Leaving directory `/root/HPC/build/lustre-1.6.4.3/ldiskfs'
> make[2]: Leaving directory `/root/HPC/build/lustre-1.6.4.3/ldiskfs'
> Making all in .
> make[2]: Entering directory `/root/HPC/build/lustre-1.6.4.3'
> for dir in ldiskfs lnet lustre ; do \
> make sources -C $dir || exit $? ; \
> done
> make[3]: Entering directory `/root/HPC/build/lustre-1.6.4.3/ldiskfs'
> for ...
>
> read more »
More information about the lustre-discuss
mailing list