<div dir="ltr">Thanks Albert, I really appreciate you...<br>Now everything is working...<br><br><br><div class="gmail_quote">On Mon, Feb 21, 2011 at 7:44 PM, Albert Everett <span dir="ltr"><<a href="mailto:aeeverett@ualr.edu">aeeverett@ualr.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Here's what's in our /etc/modprobe.conf related to IB and lustre:<br>
<br>
options ib_mthca msi_x=1<br>
options lnet networks=o2ib0(ib0)<br>
options ko2iblnd ipif_name=ib0<br>
<br>
We have Mellanox Infinihost (III?) DDR cards and IPs defined for them.<br>
<br>
$ /sbin/ifconfig ib0<br>
ib0 Link encap:InfiniBand HWaddr 80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00<br>
inet addr:192.168.2.1 Bcast:192.168.2.255 Mask:255.255.255.0<br>
inet6 addr: fe80::202:c902:29:b341/64 Scope:Link<br>
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1<br>
RX packets:1719 errors:0 dropped:0 overruns:0 frame:0<br>
TX packets:30 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:256<br>
RX bytes:96564 (94.3 KiB) TX bytes:2420 (2.3 KiB)<br><font color="#888888">
<br>
Albert</font><div><div></div><div class="h5"><br>
<br>
On Feb 20, 2011, at 2:20 PM, Arya Mazaheri wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
I have done what you said. I will test my client to the server tomorrow. but would you tell me the tweaks you have done on /etc/modprobe.conf ?<br>
<br>
On Sun, Feb 20, 2011 at 1:56 AM, Albert Everett <<a href="mailto:aeeverett@ualr.edu" target="_blank">aeeverett@ualr.edu</a>> wrote:<br>
For lustre client, we did not need to alter our kernel at all. We just made and installed lustre-1.8.5 and lustre-modules-1.8.5 rpms. /etc/modprobe.conf needs a tweak.<br>
<br>
For lustre server, I believe you will need to deal with a patched kernel. We have not been down this road yet since our vendor includes lustre server software with their hardware.<br>
<br>
Albert<br>
<br>
<br>
On Feb 19, 2011, at 12:18 PM, Arya Mazaheri wrote:<br>
<br>
Hi Albert,<br>
It seems that you have made a new kernel in order to run lustre on clients. Am I right?<br>
I don't want to change kernel on clients at all...<br>
<br>
On Sat, Feb 19, 2011 at 8:57 PM, Albert Everett <<a href="mailto:aeeverett@ualr.edu" target="_blank">aeeverett@ualr.edu</a>> wrote:<br>
Our kernel is also 2.6.18_194.17.4.el5.<br>
<br>
We installed OFED 1.5.2 from source, following this guide:<br>
<br>
<a href="https://wiki.rocksclusters.org/wiki/index.php/Install_OFED_1.5.x_on_a_Rocks_5.3_cluster" target="_blank">https://wiki.rocksclusters.org/wiki/index.php/Install_OFED_1.5.x_on_a_Rocks_5.3_cluster</a><br>
<br>
... which left us, among other things, a folder /usr/src/ofa_kernel.<br>
<br>
Lustre on the server side is handled by our vendor, so all we needed to worry about is the client.<br>
<br>
To build a lustre client, we then installed lustre-1.8.5.tar.gz from source, not from rpms. Our first compile produced the error you show below.<br>
# ./configure --with-linux=/lib/modules/`uname -r`/build<br>
# make rpms<br>
<br>
To get the lustre installation to use our new OFED, we tried this and it worked.<br>
<br>
# ./configure --with-o2ib=/usr/src/ofa_kernel --with-linux=/lib/modules/`uname -r`/build<br>
# make rpms<br>
<br>
RPMs showed up in /usr/src/redhat/RPMS/x86_64, and we are using lustre-1.8.5*.rpm and lustre-modules-*.rpm on our client machines.<br>
<br>
Albert<br>
<br>
<br>
On Feb 19, 2011, at 8:34 AM, Arya Mazaheri wrote:<br>
<br>
Hi,<br>
I have installed lustre client packages on a client node. But it doesn't mount the lustre file system from lustre server. It gets the following famous error:<br>
<br>
$ mount -t lustre 192.168.0.1:/lustre /mnt/lustre<br>
mount.lustre: mount 172.16.113.232:/lustre at /mnt/lustre failed: No such device<br>
Are the lustre modules loaded?<br>
Check /etc/modprobe.conf and /proc/filesystems<br>
Note 'alias lustre llite' should be removed from modprobe.conf<br>
<br>
<br>
As I was searching through the mailing list, I have noticed that lustre.ko should be present in this directory:<br>
/lib/modules/2.6.18-194.17.4.el5/kernel/fs/lustre/lustre.ko<br>
<br>
My current kernel is 2.6.18-194.17.4.el5. but lustre.ko is in 2.6.18-164.11.1.el5 instead. So I guessed that this may be the source of problem.<br>
<br>
Any ideas?<br>
<br>
Thanks<br>
_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br>
<br>
<br>
<br>
</blockquote>
<br>
</div></div></blockquote></div><br></div>