<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Folks,<br>
<br>
We are trying to create a small lustre environment on behalf of a
customer. There are 2 X4200m2 MDS servers, both dual-attached to an STK
6140 array over FC. This is an active-passive arrangement with a single
shared volume. Heartbeat is used to co-ordinate file system failover.
There is a single X4500 OSS server, the storage for which is split into
6 OSTs. Finally, we have 2 X4600m2 clients, just for kicks.<br>
<br>
All systems are connected together over ethernet and infiniband, with
the IB network being used for Lustre and every system is running RHEL
4.5 AS. The X4500 OST volumes are created using software RAID, while
the X4200m2 MDT is accessed using DM Multipath. We downloaded the
Lustre binary packages from SUN's web site and installed them onto each
of the servers.<br>
<br>
Unfortunately, the resulting system is very unstable and is prone to
lock-ups on the servers (uptimes are measured in hours). These lock-ups
happen without warning, and with very little, if any, debug information
in the system logs. We have also observed the servers locking up on
shutdown (kernel panics). Based on the documentation in the Lustre
operations manual, we installed the RPMs as follows:<br>
<br>
<tt>rpm -Uvh --force e2fsprogs-1.40.7.sun3-0redhat.x86_64.rpm<br>
rpm -ivh kernel-lustre-smp-2.6.9-67.0.7.EL_lustre.1.6.5.1.x86_64.rpm<br>
rpm -ivh kernel-lustre-source-2.6.9-67.0.7.EL_lustre.1.6.5.1.x86_64.rpm<br>
rpm -ivh
lustre-modules-1.6.5.1-2.6.9_67.0.7.EL_lustre.1.6.5.1smp.x86_64.rpm #
(many "unknown symbol" warnings)<br>
rpm -ivh lustre-1.6.5.1-2.6.9_67.0.7.EL_lustre.1.6.5.1smp.x86_64.rpm<br>
rpm -ivh
lustre-source-1.6.5.1-2.6.9_67.0.7.EL_lustre.1.6.5.1smp.x86_64.rpm<br>
rpm -ivh
lustre-ldiskfs-3.0.4-2.6.9_67.0.7.EL_lustre.1.6.5.1smp.x86_64.rpm #
(many
"unknown symbol" warnings)<br>
mv /etc/init.d/openibd /etc/init.d/openibd.rhel4default<br>
rpm -ivh --force
kernel-ib-1.3-2.6.9_67.0.7.EL_lustre.1.6.5.1smp.x86_64.rpm <br>
cp /etc/init.d/openibd /etc/init.d/openibd.lustre.1.6.5.1<br>
</tt><br>
We then reboot the system and load RHEL using the Lustre kernel. Now we
install the Voltaire OFED software:<br>
<ol>
<li>Copy the kernel config used to build the Lustre patched kernel
into the Lustre kernel source tree:<br>
<br>
<tt>cp /boot/config-2.6.9-67.0.7.EL_lustre.1.6.5.1smp \<br>
/usr/src/linux-2.6.9-67.0.7.EL_lustre.1.6.5.1/.config<br>
<br>
</tt></li>
<li>Change into the Lustre kernel source and edit the Makefile.
Change
"<tt>custom</tt>" suffix to "<tt>smp</tt>" in the variable "<tt>EXTRAVERSION</tt>".</li>
<li>Change into the lustre kernel source and run these setup commands:<br>
<br>
<tt>make oldconfig || make menuconfig<br>
make include/asm<br>
make include/linux/version.h<br>
make SUBDIRS=scripts</tt><br>
<br>
</li>
<li>Change into the "<tt>-obj</tt>" directory and run these setup
commands:<br>
<br>
<tt>cd /usr/src/linux-2.6.9-67.0.7.EL_lustre.1.6.5.1-obj/x86_64/smp<br>
ln -s /usr/src/linux-2.6.9-67.0.7.EL_lustre.1.6.5.1/include .<br>
<br>
</tt></li>
<li>Unpack the Voltaire OFED tar-ball:<br>
<br>
<tt>tar zxf VoltaireOFED-5.1.3.1_5.tgz<br>
</tt><br>
</li>
<li>Change to the unpacked software directory and run the
installation script. To build the OFED packages with the Voltaire
certified configuration, run the following commands:<br>
<br>
<tt>cd VoltaireOFED-5.1.3.1_5<br>
./install.pl -c ofed.conf.Volt</tt><br>
<br>
</li>
<li>Once complete, reboot.</li>
<li>Configure any IPoIB interfaces as required.</li>
<li>Add the following into <tt>/etc/modprobe.conf</tt>:<br>
<br>
<tt>options lnet networks="o2ib0(ib0)"</tt><br>
<br>
</li>
<li> Load the Lustre LNET kernel module.<br>
<br>
<tt>modprobe lnet</tt><br>
<br>
</li>
<li>Start the Lustre core networking service.<br>
<br>
<tt>lctl network up</tt><br>
<br>
</li>
<li>Check the system log (<tt>/var/log/messages</tt>) for
confirmation.</li>
</ol>
<br>
Create the MGS/MDT Lustre Volume:<br>
<ol>
<li>Format the MGS/MDT device.<br>
<br>
<tt>mkfs.lustre [ --reformat ] --fsname lfs01 --mdt --mgs
--failnode=mds-2@o2ib0 /dev/dm-0</tt><br>
<br>
</li>
<li>Create the MGS/MDT file system mount point.<br>
<br>
<tt>mkdir -p /lustre/mdt/lfs01</tt><br>
<br>
</li>
<li>Mount the file system. This will initiate MGS and MDT services
for Lustre.<br>
<br>
<tt>mount -t lustre /dev/dm-0 /lustre/mdt/lfs01</tt><br>
</li>
</ol>
With the exception of the OST volume creation, we use an equivalent
process to bring the OSS online.<br>
<br>
The cabling has been checked and verified. So we re-built the system
from scratch and applied only SUN's RDAC modules and Voltaire OFED to
the stock RHEL 4.5 kernel (2.6.9-55.ELsmp). We removed the second MDS
from the h/w configuration and did not install Heartbeat. The shared
storage was re-formatted as a regular EXT3 file system using the DM
multipathing device, /dev/dm-0, and mounted onto the host. Running I/O
tests onto the mounted file system over an extended period did not
elicit a single error or warning message in the log related to the
multipathing or the SCSI device.<br>
<br>
Once we were confident that the system was running in a consistent and
stable manner, we re-installed the Lustre packages, omitting the
kernel-ib packages. We had to re-build and re-install the RDAC support
as well. This means that the system has support for the Lustre file
system but no infiniband support at all. <tt>/etc/modprobe.conf</tt>
is updated such that the <tt>lnet</tt> <tt>networks</tt> option is
set to "tcp". The MDS/MGS volume is recreated on the DM device.<br>
<br>
We have tried the following configurations on the X4200m2:<br>
<ul>
<li>RHEL vanilla kernel, multipathd, RDAC. EXT-3 file system. PASSED.</li>
<li>RHEL vanilla kernel, multipathd, RDAC, Voltaire OFED. EXT-3 file
system. PASSED.</li>
<li><font color="#000000">Lustre supplied kernel, Lustre software.
No IB. MDS/MGS file
system. FAILED.</font></li>
<li>Lustre supplied kernel, Lustre software, RDAC. No IB. MDS/MGS
file system (Full Lustre FS over Ethernet). FAILED.</li>
<li value="3">Lustre supplied kernel, Lustre software, RDAC, Voltaire
OFED. EXT-3 file system. FAILED.</li>
<li value="3">Lustre supplied kernel, Lustre software. RDAC, Voltaire
OFED. MDS/MGS file system (Full Lustre FS over IB). FAILED.</li>
</ul>
Our findings indicate that there is a problem within the binary
distribution of Lustre. This may be due to the fact that we are
applying the 2.6.9-67 RHEL kernel to a platform based upon 2.6.9.-55,
or it may be a more subtle issue based on the interaction with the
underlying hardware. We could use some advice on how best to proceed,
since our deadline fast approaches. For example, is our build process,
as documented above, clean? Currently, we're looking at
building from source, to see if this results in a more stable
environment.<br>
<br>
Regards,<br>
<br>
Malcolm.<br>
<br>
<div class="moz-signature">-- <br>
<table border="0" cellpadding="0" cellspacing="0" width="450">
<tbody>
<tr valign="top">
<td height="121" width="121"><a href="http://www.sun.com"><img
moz-do-not-send="false" src="cid:part1.00030502.07060106@sun.com"
border="0" height="116" width="121"></a></td>
<td style="font-family: Arial; font-size: 11px;" height="121"
width="329"><br>
<b>Malcolm Cowe</b><br>
<em>Solutions Integration Engineer</em><br>
<br>
<b>Sun Microsystems, Inc.</b><br>
Blackness Road<br>
Linlithgow, West Lothian EH49 7LR UK<br>
Phone: x73602 / +44 1506 673 602<br>
Email: <a class="moz-txt-link-abbreviated"
href="mailto:Malcolm.Cowe@Sun.COM">Malcolm.Cowe@Sun.COM</a><br>
</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>