Hi,<br><br>If I recall it correctly there was a thread on the lustre mailing which you may find helpful <br><a href="http://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg06322.html">http://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg06322.html</a><br>
<br>Cheers<br><br>Wojciech<br><div class="gmail_quote">On 22 March 2010 15:44, Marco Aurelio L Gomes <span dir="ltr"><<a href="mailto:mgomes@tpn.usp.br">mgomes@tpn.usp.br</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi Michael,<br>
<br>
Thanks in advance for your reply. I've followed your instructions and I<br>
compiled OFEd 1.4.2 succesfully, but when I tried to modprobe lustre, i<br>
get errors complaining about symbols versions<br>
<br>
[root@masternode1 modules]# modprobe lustre<br>
<div class="im">WARNING: Error inserting osc<br>
(/lib/modules/2.6.18-164.11.1.el5/kernel/fs/lustre/osc.ko): Input/output<br>
error<br>
WARNING: Error inserting mdc<br>
(/lib/modules/2.6.18-164.11.1.el5/kernel/fs/lustre/mdc.ko): Input/output<br>
error<br>
WARNING: Error inserting lov<br>
(/lib/modules/2.6.18-164.11.1.el5/kernel/fs/lustre/lov.ko): Input/output<br>
error<br>
FATAL: Error inserting lustre<br>
(/lib/modules/2.6.18-164.11.1.el5/kernel/fs/lustre/lustre.ko):<br>
Input/output error<br>
<br>
</div>The contents of dmesg are in attached file.<br>
<br>
There are another step that I need to follow to get lustre 1.8.2 working<br>
at kernel 2.6.18-164.11.1.el5 and OFED 1.4.2? And it's possible to get<br>
lustre 1.8.2 working with OFED 1.5?<br>
<br>
Regards,<br>
<div class="im"><br>
<br>
Regards,<br>
<br>
Marco Gomes<br>
Systems/HPC-Cluster<br>
Numerical Offshore Tank<br>
</div>+55 11 3777-4142 r.250<br>
+55 11 3091-5350 r.250<br>
<br>
On Sun, 2010-03-21 at 12:01 -0600,<br>
Message: 1<br>
Date: Sun, 21 Mar 2010 19:00:46 +0100<br>
From: Michael Mayer <<a href="mailto:mmayer@hpce.nec.com">mmayer@hpce.nec.com</a>><br>
Subject: Re: [Lustre-discuss] Installing Lustre 1.8.2 on CentOS 5.4<br>
<div class="im"> with OFED 1.4.2<br>
</div>To: <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
Message-ID: <<a href="mailto:4BA65ECE.1010309@hpce.nec.com">4BA65ECE.1010309@hpce.nec.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<div><div></div><div class="h5"><br>
Hi Marco,<br>
<br>
OFED 1.4.2 and the RHEL5.4 kernel do not work because OFED 1.4.2 lacks<br>
official RHEL5.4 support (I think OFED 1.4.2 was even released before<br>
RHEL5.4 so it only has backports up to and including RHEL 5.3. The<br>
RHEL5.4 kernels do not differ much from the RHEL5.3 but became a bit<br>
more upstream so a couple of the RHEL5.3 backports of OFED 1.4.2 are<br>
not<br>
needed any longer.<br>
<br>
We had the same problem here and upon inspection of the error messages<br>
it turns out that the errors in the OFED 1.4.2 compilation can be fixed<br>
in a rather simple way: I have created an OFED patch and a modified<br>
spec<br>
file (attached).<br>
<br>
So, please install the default ofa_kernel source rpm in your system,<br>
replace the spec file and copy the patch file to the SOURCES directory,<br>
then run a "rpmbuild -bs ofa_kernel.spec" and finally replace the<br>
default source rpm with the new ofa_kernel source rpm.<br>
<br>
After that, OFED compilation should work without errors.<br>
<br>
Cheers,<br>
<br>
Michael.<br>
<br>
On 20/03/10 00:31, Marco Aurelio L Gomes wrote:<br>
> Hi all,<br>
><br>
> I'm trying to setup my Lustre environment using CentOS 5.4<br>
> (kernel-2.6.18-164.11.1.el5-x86_64 from updates repository) Lustre<br>
1.8.2<br>
> and OFED 1.4.2. I saw at Lustre Matrix Support that 1.8.2 works at<br>
most<br>
> with OFED 1.4.2, but when I tried to compile with this release, I saw<br>
a<br>
> lot of errors complaining about variables redefinition at ofa_kernel<br>
> compiling process (see attached ofa_kernel_rpmbuild.log. but i think<br>
> this is not related to this list). In this case, i tried to compile<br>
with<br>
> OFED 1.5, the latest stable release of OFED, that compile fine; but<br>
when<br>
> i boot a client, and run:<br>
><br>
> modprobe lustre<br>
><br>
> i got the following errors:<br>
><br>
> [root@masternode1 ~]# modprobe lustre<br>
> WARNING: Error inserting osc<br>
> (/lib/modules/2.6.18-164.11.1.el5/kernel/fs/lustre/osc.ko):<br>
Input/output<br>
> error<br>
> WARNING: Error inserting mdc<br>
> (/lib/modules/2.6.18-164.11.1.el5/kernel/fs/lustre/mdc.ko):<br>
Input/output<br>
> error<br>
> WARNING: Error inserting lov<br>
> (/lib/modules/2.6.18-164.11.1.el5/kernel/fs/lustre/lov.ko):<br>
Input/output<br>
> error<br>
> FATAL: Error inserting lustre<br>
> (/lib/modules/2.6.18-164.11.1.el5/kernel/fs/lustre/lustre.ko):<br>
> Input/output error<br>
><br>
> and at dmesg:<br>
><br>
> Lustre: OBD class driver, <a href="http://www.lustre.org/" target="_blank">http://www.lustre.org/</a><br>
> Lustre: Lustre Version: 1.8.2<br>
> Lustre: Build Version:<br>
> 1.8.2-20100122201357-PRISTINE-2.6.18-164.11.1.el5<br>
> ko2iblnd: disagrees about version of symbol ib_fmr_pool_unmap<br>
> ko2iblnd: Unknown symbol ib_fmr_pool_unmap<br>
> ko2iblnd: disagrees about version of symbol ib_create_cq<br>
> ko2iblnd: Unknown symbol ib_create_cq<br>
> ko2iblnd: disagrees about version of symbol rdma_resolve_addr<br>
> ko2iblnd: Unknown symbol rdma_resolve_addr<br>
> ko2iblnd: disagrees about version of symbol ib_reg_phys_mr<br>
> ko2iblnd: Unknown symbol ib_reg_phys_mr<br>
> ko2iblnd: disagrees about version of symbol ib_create_fmr_pool<br>
> ko2iblnd: Unknown symbol ib_create_fmr_pool<br>
> ko2iblnd: disagrees about version of symbol ib_dereg_mr<br>
> ko2iblnd: Unknown symbol ib_dereg_mr<br>
> ko2iblnd: disagrees about version of symbol rdma_reject<br>
> ko2iblnd: Unknown symbol rdma_reject<br>
> ko2iblnd: disagrees about version of symbol rdma_disconnect<br>
> ko2iblnd: Unknown symbol rdma_disconnect<br>
> ko2iblnd: disagrees about version of symbol rdma_resolve_route<br>
> ko2iblnd: Unknown symbol rdma_resolve_route<br>
> ko2iblnd: disagrees about version of symbol rdma_bind_addr<br>
> ko2iblnd: Unknown symbol rdma_bind_addr<br>
> ko2iblnd: disagrees about version of symbol rdma_create_qp<br>
> ko2iblnd: Unknown symbol rdma_create_qp<br>
> ko2iblnd: disagrees about version of symbol ib_destroy_cq<br>
> ko2iblnd: Unknown symbol ib_destroy_cq<br>
> ko2iblnd: disagrees about version of symbol rdma_create_id<br>
> ko2iblnd: Unknown symbol rdma_create_id<br>
> ko2iblnd: disagrees about version of symbol rdma_listen<br>
> ko2iblnd: Unknown symbol rdma_listen<br>
> ko2iblnd: disagrees about version of symbol rdma_destroy_qp<br>
> ko2iblnd: Unknown symbol rdma_destroy_qp<br>
> ko2iblnd: disagrees about version of symbol ib_query_device<br>
> ko2iblnd: Unknown symbol ib_query_device<br>
> ko2iblnd: disagrees about version of symbol ib_get_dma_mr<br>
> ko2iblnd: Unknown symbol ib_get_dma_mr<br>
> ko2iblnd: disagrees about version of symbol ib_alloc_pd<br>
> ko2iblnd: Unknown symbol ib_alloc_pd<br>
> ko2iblnd: disagrees about version of symbol rdma_connect<br>
> ko2iblnd: Unknown symbol rdma_connect<br>
> ko2iblnd: disagrees about version of symbol ib_modify_qp<br>
> ko2iblnd: Unknown symbol ib_modify_qp<br>
> ko2iblnd: disagrees about version of symbol rdma_destroy_id<br>
> ko2iblnd: Unknown symbol rdma_destroy_id<br>
> ko2iblnd: disagrees about version of symbol rdma_accept<br>
> ko2iblnd: Unknown symbol rdma_accept<br>
> ko2iblnd: disagrees about version of symbol ib_dealloc_pd<br>
> ko2iblnd: Unknown symbol ib_dealloc_pd<br>
> ko2iblnd: disagrees about version of symbol ib_fmr_pool_map_phys<br>
> ko2iblnd: Unknown symbol ib_fmr_pool_map_phys<br>
> LustreError: 4572:0:(api-ni.c:1043:lnet_startup_lndnis()) Can't load<br>
LND<br>
> o2ib, module ko2iblnd, rc=256<br>
> LustreError: 4572:0:(events.c:729:ptlrpc_init_portals()) network<br>
> initialisation failed<br>
><br>
> At this point, i thought that is related to OFED compilation. I only<br>
> compile OFED because I didn't found kernel-ib package at lustre<br>
download<br>
> site. I would like to know if someone had the same problem on their<br>
> setup and if there are some workaround to get it working.<br>
><br>
> In following I'll give more information about the lustre environment.<br>
><br>
> client:<br>
> CentOS 5.4 (2.6.18-164.11.1.el5)<br>
> Lustre 1.8.2<br>
> lustre-client-modules-1.8.2-2.6.18_164.11.1.el5_lustre.1.8.2<br>
> lustre-client-1.8.2-2.6.18_164.11.1.el5_lustre.1.8.2<br>
> OFED 1.5<br>
><br>
> Thanks in advance for the help and sorry for my bad english.<br>
><br>
> Regards,<br>
><br>
> Marco Gomes<br>
> Systems/HPC-Cluster<br>
> Numerical Offshore Tank<br>
> (11) 3777-4142 #250<br>
> (11) 3091-5350 #250<br>
</div></div><br>_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>--<br>Wojciech Turek<br><br>Assistant System Manager<br><br>High Performance Computing Service<br>University of Cambridge<br>Email: <a href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a><br>
Tel: (+)44 1223 763517 <br>