[Lustre-discuss] Unable to build kernel-ib when building lustre from source code

[neutron]

Hi Atul, thanks a lot!   Now I can build lustre, ofed.

When I run mkfs.lustre, a strange thing happens:

--------------
[i3:~/lustre/scripts]$sudo /usr/sbin/mkfs.lustre --fsname=lustre
--reformat --mdt --mgs --device-size=100000  /dev/os/lustre
Password:

   Permanent disk data:
Target:     lustre-MDTffff
Index:      unassigned
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x75
              (MDT MGS needs_index first_time update )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: mdt.group_upcall=/usr/sbin/l_getgroups

2 6 18
formatting backing filesystem ldiskfs on /dev/os/lustre
        target name  lustre-MDTffff
        4k blocks     25000
        options        -i 4096 -I 512 -q -O dir_index,extents,uninit_groups -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-MDTffff  -i 4096 -I 512 -q -O
dir_index,extents,uninit_groups -F /dev/os/lustre 25000
   sh: mke2fs: command not found

mkfs.lustre FATAL: Unable to build fs /dev/os/lustre (32512)

mkfs.lustre FATAL: mkfs failed 32512


---------------
It appears as if mkfs.lustre cannot find mke2fs.  I have installed the
e2fsprogs.rpm.  "mke2fs" is in dir "/sbin", and it's already included
in the PATH.

[i3:~/lustre/lustre-1.8.1]$rpm -q e2fsprogs
e2fsprogs-1.41.6.sun1-0redhat

[i3:~/lustre/lustre-1.8.1]$echo $PATH
/sbin:/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/local/ofed/bin:/usr/local/ofed/sbin:/usr/kerberos/bin:/usr/local/bin



On Sun, Jan 31, 2010 at 10:44 PM, Atul Vidwansa <Atul.Vidwansa at sun.com> wrote:
> Sometime back I posted instruction on building Lustre from sources with OFED
> on my blog at http://blogs.sun.com/atulvid/ . Have a look, it may be useful
> to you.
>
> Cheers,
> _Atul
>
> neutron wrote:
>>
>> Hi all,
>>
>> I have installed Lustre-1.8.1.1  from pre-built rpm packages and it
>> works. Now I want to build Lustre 1.8.1.1 from source on RHEL5
>> x86_64.  Since I want to learn Lustre( and eventually make some
>> modifications in lustre),   I choose to build Lustre from source code.
>>
>>
>> I followed the steps at "Installing Lustre with a Third-Party Network
>> Stack" from "lustre manual 1.8".
>> Here is what I do.  At my ~/.rpmmacros,  I set my rpm working dir to
>> be ~/rpmbuild
>>
>> (1) Install "kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm".
>> The manual says to use "kernel-lustre-source-xxxx.rpm", but I couldn't
>> find that file at Sun's download site.
>> This src.rpm is installed to ~/rpmbuild/SOURCE
>>
>> (2) At ~/rpmbuild/SPECS,  Run "rpmbuild -bp --target=x86_64
>>  ./kernel-2.6.spec"
>> The kernel src tree is at
>> "~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64"
>>
>> (3) In that dir,  I use the config file of currently running kernel, and
>> then:
>>        $ make oldconfig || make menuconfig
>>        $ make include/asm
>>        $ make include/linux/version.h
>>        $ make SUBDIRS=scripts
>>        $ make;  make modules_install; make install
>>
>> This should be identical to "make rpm" and then install the
>> kernel-xxx.rpm and kernel-xxx.src.rpm.
>>
>> (4) Reboot into the new lustre kernel
>> "uname -r" gives  "2.6.18-128.7.1-lustre-1.8.1-customer".
>> I have verified that
>> "/lib/modules/2.6.18-128.7.1-lustre-1.8.1-customer/build" point to
>> "~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64"
>>
>>
>> (5) Install the
>> "kernel-ib-devel-1.4.2-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm".
>> The files are installed to "/usr/src/ofa_kernel-1.4.2".
>>  At that place, I run
>>        $ /configure --with-core-mod --with-ipoib-mod --with-mthca-mod
>> --with-mlx4_core-mod --with-mlx4_inf-mod
>>        $ make
>>
>> Then I see lots of errors like:
>>
>>  /usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/fs.h:9:
>> error: redefinition of ‘inc_nlink’
>> include/linux/fs.h:1255: error: previous definition of ‘inc_nlink’ was
>> here
>> In file included from
>> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:44,
>>                 from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38,
>>                 from
>> /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49:
>>
>> /usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/mm.h:17:
>> error: redefinition of ‘is_vmalloc_addr’
>> include/linux/mm.h:332: error: previous definition of ‘is_vmalloc_addr’
>> was here
>> In file included from
>> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38,
>>                 from
>> /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49:
>> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h: In function
>> ‘ib_dma_mapping_error’:
>> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:1593: warning:
>> passing argument 1 of ‘dma_mapping_error’ makes integer from pointer
>> without a cast
>>
>>
>> I googled the error, and someone said to use OFED-1.5.
>> But I tried with OFED-1.4.2, OFED-1.5, and OFED-1.5.1 daily tar ball
>> OFED-1.5.1-20100131-0600,
>> and always see the same error "redefinition of ........"   when I'm
>> compiling the kernel-ib.
>>
>> Am I wrong with the way to install kernel-ib or kernel-lustre?  How
>> can I make lustre work with IB support?
>>
>>
>> (6) Build lustre succeeds without errors.
>> I untared "lustre-1.8.1.1.tar.gz", then
>>        $ ./configure
>> --with-linux=~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64
>>        $ make
>>        $ make install
>>
>> Looks both lustre kernel modules and luster userland tools/library are
>> installed correctly.
>> Is this a right way to install lustre?
>>
>> Thanks!
>>
>>
>>
>> -Neutron
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>
>