<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hey Brian,<br>
<br>
I'll have to re-install the system from scratch in order to be able to
answer some of your questions, which I'll get started on this evening.
What I was hoping for in the first instance was a sanity check of our
installation methods. With respect to the OFED stack used, we are using
the latest official software stack supplied by Voltaire. The reason for
this is that there is more to OFED than just the kernel modules,
including many libraries and tools, plus the latest firmware for the
cards. It's what the customer has asked for, and it is what the card
vendor expects us to do.<br>
<br>
We may be able to get away with OFED 1.3, but I would still like some
guidance on how to install the rest of the OFED stack -- do we use the
OFED source to rebuild everything, or can we pick the Lustre supplied
kernel modules and just layer on the other stuff separately? Like I
said, sanity-checking the install procedure is important.<br>
<br>
Finally, when I said that one file system fails versus another passes,
I mean that the server locks solid, crashes, usually with no debug to
speak of (nothing in the system logs). Even while the system is up and
running the lustre kernel, if we attempt a clean shutdown, the kernel
panics.<br>
<br>
Since I need to rebuild the systems anyway, I will also try to install
the packages in the order mentioned by Megan Larko, to see how that
affects the installation. We have been following the instructions in
the Lustre Operations Manual (v. 1.14).<br>
<br>
Regards,<br>
<br>
Malcolm.<br>
<br>
<br>
Brian J. Murrell wrote:
<blockquote cite="mid:1223299088.1463.17.camel@pc.ilinx" type="cite">
<pre wrap="">On Mon, 2008-10-06 at 10:58 +0100, Malcolm Cowe wrote:
</pre>
<blockquote type="cite">
<pre wrap="">rpm -Uvh --force e2fsprogs-1.40.7.sun3-0redhat.x86_64.rpm
</pre>
</blockquote>
<pre wrap=""><!---->
You should not (have to) use --force. If you do, there is either an
operational error or a bug in our packages. In the latter case, please
file a bug in our bugzilla.
</pre>
<blockquote type="cite">
<pre wrap="">rpm -ivh
lustre-modules-1.6.5.1-2.6.9_67.0.7.EL_lustre.1.6.5.1smp.x86_64.rpm #
(many "unknown symbol" warnings)
</pre>
</blockquote>
<pre wrap=""><!---->
Can you paste them here?
</pre>
<blockquote type="cite">
<pre wrap="">rpm -ivh
lustre-ldiskfs-3.0.4-2.6.9_67.0.7.EL_lustre.1.6.5.1smp.x86_64.rpm #
(many "unknown symbol" warnings)
</pre>
</blockquote>
<pre wrap=""><!---->
Ditto.
</pre>
<blockquote type="cite">
<pre wrap="">rpm -ivh --force
kernel-ib-1.3-2.6.9_67.0.7.EL_lustre.1.6.5.1smp.x86_64.rpm
</pre>
</blockquote>
<pre wrap=""><!---->
Again, you should not need to use --force.
</pre>
<blockquote type="cite">
<pre wrap="">We then reboot the system and load RHEL using the Lustre kernel. Now
we install the Voltaire OFED software:
</pre>
</blockquote>
<pre wrap=""><!---->
Why? The kernel-ib package you installed above should provide a working
OFED stack.
</pre>
<blockquote type="cite">
<pre wrap=""> 1. Unpack the Voltaire OFED tar-ball:
tar zxf VoltaireOFED-5.1.3.1_5.tgz
</pre>
</blockquote>
<pre wrap=""><!---->
Do you really need 1.3.1? If so, then you should not install the 1.3
kernel-ib package we provide above. I really wonder why you need 1.3.1
though.
</pre>
<blockquote type="cite">
<pre wrap=""> * Lustre supplied kernel, Lustre software. No IB. MDS/MGS file
system. FAILED.
</pre>
</blockquote>
<pre wrap=""><!---->
Failed in what way?
</pre>
<blockquote type="cite">
<pre wrap=""> * Lustre supplied kernel, Lustre software, RDAC. No IB. MDS/MGS
file system (Full Lustre FS over Ethernet). FAILED.
</pre>
</blockquote>
<pre wrap=""><!---->
Again, in what way?
</pre>
<blockquote type="cite">
<pre wrap=""> * Lustre supplied kernel, Lustre software, RDAC, Voltaire OFED.
EXT-3 file system. FAILED.
</pre>
</blockquote>
<pre wrap=""><!---->
Ditto.
</pre>
<blockquote type="cite">
<pre wrap=""> * Lustre supplied kernel, Lustre software. RDAC, Voltaire OFED.
MDS/MGS file system (Full Lustre FS over IB). FAILED.
</pre>
</blockquote>
<pre wrap=""><!---->
And Ditto again.
You have to provide more details than just "FAILED" if we are to try to
help diagnose a problem.
</pre>
<blockquote type="cite">
<pre wrap="">Our findings indicate that there is a problem within the binary
distribution of Lustre.
</pre>
</blockquote>
<pre wrap=""><!---->
I think that many of our users use it as is, so it cannot be all that
bad.
</pre>
<blockquote type="cite">
<pre wrap="">This may be due to the fact that we are applying the 2.6.9-67 RHEL
kernel to a platform based upon 2.6.9.-55,
</pre>
</blockquote>
<pre wrap=""><!---->
That shouldn't be a problem in and of itself.
b.
</pre>
<pre wrap="">
<hr size="4" width="90%">
_______________________________________________
Lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a>
</pre>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<table border="0" cellpadding="0" cellspacing="0" width="450">
<tbody>
<tr valign="top">
<td height="121" width="121"><a href="http://www.sun.com"><img
moz-do-not-send="false" src="cid:part1.04000909.07020900@sun.com"
border="0" height="116" width="121"></a></td>
<td style="font-family: Arial; font-size: 11px;" height="121"
width="329"><br>
<b>Malcolm Cowe</b><br>
<em>Solutions Integration Engineer</em><br>
<br>
<b>Sun Microsystems, Inc.</b><br>
Blackness Road<br>
Linlithgow, West Lothian EH49 7LR UK<br>
Phone: x73602 / +44 1506 673 602<br>
Email: <a class="moz-txt-link-abbreviated" href="mailto:Malcolm.Cowe@Sun.COM">Malcolm.Cowe@Sun.COM</a><br>
</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>