[Lustre-discuss] Lustre 1.6.5.1 on X4200 and STK 6140 Issues

Tue Oct 7 09:46:50 UTC 2008

Now with attachment. Sorry.

Malcolm Cowe wrote:
>
> Andreas Dilger wrote:
>> On Oct 06, 2008  10:59 -0400, Brian J. Murrell wrote:
>>   
>>> On Mon, 2008-10-06 at 15:47 +0100, Malcolm Cowe wrote:
>>>     
>>>> With respect to the OFED stack used, we are using the latest official
>>>> software stack supplied by Voltaire. The reason for this is that there
>>>> is more to OFED than just the kernel modules, including many libraries
>>>> and tools,
>>>>       
>>> None of these should be necessary for Lustre to use I/B.
>>>     
>>
>> Also very important to note is that if you are changing the IB stack,
>> then Lustre also needs to be recompiled to work with the new IB stack.
>>
>>   
> Yes. As a matter of fact, you have anticipated a question I have: how 
> does one re-build Lustre in a safe and consistent manner? I'm working 
> through the docs, but I have come across a problem when I try to run 
> "make rpms" in the Lustre source:
>
> make[4]: *** No rule to make target 
> `/usr/src/redhat/BUILD/lustre-1.6.5.1/ldiskfs/Module.symvers', needed 
> by `Module.symvers'.  Stop.
>
> How do I ensure that the build environment that Lustre requires is 
> properly prepared? I could just hoick a soft link to the 
> Module.symvers file in the kernel tree, but that's a little messy.
>
> I've attached a draft copy of the build process to this message. 
> Again, just looking to sanity check the method, since I'm obviously 
> missing something.
>
> I'm going to rebuild the servers today so that I can provide the debug 
> information that Brian requested.
>> Cheers, Andreas
>> --
>> Andreas Dilger
>> Sr. Staff Engineer, Lustre Group
>> Sun Microsystems of Canada, Inc.
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>   
> Regards,
>
> Malcolm.
>
> -- 
> <http://www.sun.com> 	
> *Malcolm Cowe*
> /Solutions Integration Engineer/
>
> *Sun Microsystems, Inc.*
> Blackness Road
> Linlithgow, West Lothian EH49 7LR UK
> Phone: x73602 / +44 1506 673 602
> Email: Malcolm.Cowe at Sun.COM
>

-- 
<http://www.sun.com> 	
*Malcolm Cowe*
/Solutions Integration Engineer/

*Sun Microsystems, Inc.*
Blackness Road
Linlithgow, West Lothian EH49 7LR UK
Phone: x73602 / +44 1506 673 602
Email: Malcolm.Cowe at Sun.COM

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20081007/f84bca54/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1257 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20081007/f84bca54/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 6g_top.gif
Type: image/gif
Size: 1257 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20081007/f84bca54/attachment-0001.gif>
-------------- next part --------------
Creating Lustre Packages From Source for RHEL 4.5 AS, 64-bit
============================================================

Pre-requisites
--------------

RHEL 4.5 AS Full Installation

SUN-supplied Linux RDAC kernel modules installed.

System running on Stock RHEL 4.5 AS kernel with RDAC support.

Quilt Source
Lustre Source
RHEL 4.5 AS Kernel SRPM

1. Download and install Quilt:
	http://download.savannah.gnu.org/releases/quilt/

2. Install the RHEL 4.5 AS Kernel SRPM, found on RHEL 4.5 source CD 4.

3. Change to the Red Hat RPM specs directory and extract the full kernel
	source tree. This will also apply Red Hat's patches to the source:

	cd /usr/src/redhat/SPECS
	rpmbuild -bp kernel-2.6.spec 

4. Install the Lustre sources.

	rpm -ivh lustre-source-1.6.5.1-2.6.9_67.0.7.EL_lustre.1.6.5.1smp.x86_64.rpm

5. Prepare the kernel source for the Lustre patches:

	cd /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9
	rm -f patches series 
	ln -s /usr/src/lustre-1.6.5.1/lustre/kernel_patches/series/2.6-rhel4.series series
	ln -s /usr/src/lustre-1.6.5.1/lustre/kernel_patches/patches .

6. Apply the Lustre patches to the kernel sources:

	cd /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9
	quilt push -av

	Points to note in output:

	Hunk #1 FAILED at 770.
	1 out of 1 hunk FAILED -- rejects in file fs/nfs/nfs4proc.c

	Patch patches/vfs_intent-2.6-rhel4.patch does not apply (enforce with -f)

	Full output:

[root at mds-2 linux-2.6.9]# quilt push -av
Applying patch patches/vfs_intent-2.6-rhel4.patch
patching file fs/cifs/dir.c
patching file fs/exec.c
patching file fs/inode.c
patching file fs/namei.c
patching file fs/namespace.c
patching file fs/nfs/dir.c
patching file fs/nfs/nfs4proc.c
Hunk #1 FAILED at 770.
1 out of 1 hunk FAILED -- rejects in file fs/nfs/nfs4proc.c
patching file fs/open.c
patching file fs/stat.c
patching file include/linux/dcache.h
patching file include/linux/fs.h
patching file include/linux/mount.h
patching file include/linux/namei.h
Restoring include/linux/dcache.h
Restoring include/linux/namei.h
Restoring include/linux/mount.h
Restoring include/linux/fs.h
Restoring fs/stat.c
Restoring fs/open.c
Restoring fs/namespace.c
Restoring fs/exec.c
Restoring fs/nfs/nfs4proc.c
Restoring fs/nfs/dir.c
Restoring fs/namei.c
Restoring fs/cifs/dir.c
Restoring fs/inode.c
Patch patches/vfs_intent-2.6-rhel4.patch does not apply (enforce with -f)
Restoring include/linux/dcache.h
Restoring include/linux/namei.h
Restoring include/linux/mount.h
Restoring include/linux/fs.h
Restoring fs/stat.c
Restoring fs/open.c
Restoring fs/namespace.c
Restoring fs/exec.c
Restoring fs/nfs/nfs4proc.c
Restoring fs/nfs/dir.c
Restoring fs/namei.c
Restoring fs/cifs/dir.c
Restoring fs/inode.c

7.0. Edit the Makefile in the Kernel Source and ensure that the EXTRAVERSION
	variable matches the suffix of the original kernel. e.g.:

EXTRAVERSION = -55.ELsmp

	If this is not done, the OFED build will fail later on, as it will
	incorrectly identify the kernel and apply the wrong patches. This in turn
	leads to a build failure when compiling the "ofa_kernel" rpm.

7.2. Prepare the kernel source.

cd /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9
cp /usr/src/lustre-1.6.5.1/lustre/kernel_patches/kernel_configs/kernel-2.6.9-2.6-rhel4-x86_64-smp.config .config
make oldconfig || make menuconfig
make include/asm
make include/linux/version.h
make SUBDIRS=scripts

7.5. Backup the old kernel modules.

mv /lib/modules/2.6.9-55.ELsmp /lib/modules/2.6.9-55.ELsmp.orig

8. Build the kernel and the kernel modules.

make bzImage
make modules && make modules_install

9. Copy the kernel image into the /boot partition and create initial RAM-disk
	for the new kernel.

cp arch/x86_64/boot/bzImage /boot/vmlinuz-2.6.9-55.EL_lustre.1.6.5.1smp
cd /boot
mkinitrd initrd-2.6.9-55.EL_lustre.1.6.5.1smp 2.6.9-55.ELsmp

10. Create a new menu entry in GRUB.

11. Reboot the system, select the new kernel entry to load the OS.

12. Build the RDAC modules. Install the mpp initrc image into the boot
	partition and update GRUB. Reboot.

13. Build and install OFED. Reboot when complete.

	./install.pl -c ofed.conf.Volt -s /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9

	cd /usr/src/ofa_kernel-1.3.1
	/usr/share/doc/ofed-docs-1.3.1/create_Module.symvers.sh
	cat /usr/src/ofa_kernel-1.3.1/Module.symvers >> /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9/Module.symvers

14. Build Lustre

cd /usr/src/lustre-1.6.5.1/

# Use the following command line:
./configure --with-linux=/usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9 --with-o2ib=/usr/src/ofa_kernel-1.3.1

# This is the command line in the Lustre spec file:
./configure '--with-linux=/usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9' '--with-o2ib=/usr/src/ofa_kernel-1.3.1' '--enable-liblustre' '--enable-liblustre-tests' '--with-lustre-hack' '--with-sockets' '--sysconfdir=/etc' '--mandir=/usr/share/man' '--libdir=/usr/lib64' --with-lustre-hack --with-sockets  --sysconfdir=/etc --mandir=/usr/share/man --libdir=/usr/lib64

make
make rpms