<html><body bgcolor="#FFFFFF"><div>You must rebuild Lustre if you replace OFED.</div><div><br></div><div>Kevin</div><div><br><br>On Jun 9, 2011, at 4:55 PM, Edward Walter <<a href="mailto:ewalter@cs.cmu.edu">ewalter@cs.cmu.edu</a>> wrote:<br><br></div><div></div><blockquote type="cite"><div>
Thanks for all of the advice here. We seem to be running into a
hiccup using Lustre 1.8.4 with O2IB and OFED 1.5.1<br>
<br>
First of all, our lustre servers are all up and running fine (using
the vendor OFED - 1.4.1). Our trouble is all client side.<br>
<br>
We want to use a newer OFED (1.5.1) to potentially enable NFS over
RDMA (we have NFS servers in addition to lustre).<br>
<br>
We installed the current Lustre 1.8.4 rpms from Sun/Oracle:<br>
<blockquote type="cite">
kernel-2.6.18-194.3.1.el5_lustre.1.8.4<br>
lustre-1.8.4-2.6.18_194.3.1.el5_lustre.1.8.4<br>
lustre-modules-1.8.4-2.6.18_194.3.1.el5_lustre.1.8.4<br>
<br>
kernel-devel-2.6.18-194.3.1.el5_lustre.1.8.4<br>
kernel-headers-2.6.18-194.3.1.el5_lustre.1.8.4</blockquote>
<br>
We rebooted with kernel-2.6.18-194.3.1.el5_lustre.1.8.4.<br>
<br>
Next we downloaded the OFED 1.5.1 sources and built the basic and
hpc packages. These built and installed without incident. I don't
believe Open Fabrics group provides binary RPMS. Otherwise; we
would have used them.<br>
<br>
Here are the lustre/IB lines from our modprobe.conf:<br>
<blockquote type="cite">alias ib0 ib_ipoib<br>
alias net-pf-27 ib_sdp<br>
options lnet networks=o2ib</blockquote>
<br>
And our fstab:<br>
<blockquote type="cite"><a class="moz-txt-link-abbreviated" href="mailto:172.16.1.3@o2ib:172.16.1.4@o2ib:/data">172.16.1.3@o2ib:172.16.1.4@o2ib:/data</a>
/lustre lustre defaults,_netdev,localflock 0 0</blockquote>
<br>
OpenIB is working properly, we have a subnet manager running and can
ping our Lustre OSS and MDS servers over IB.<br>
<br>
Trying to mount /lustre generates the following error:<br>
<blockquote type="cite">mount.lustre: mount
<a class="moz-txt-link-abbreviated" href="mailto:172.16.1.3@o2ib:172.16.1.4@o2ib:/data">172.16.1.3@o2ib:172.16.1.4@o2ib:/data</a> at /lustre failed: No such
device<br>
Are the lustre modules loaded?<br>
Check /etc/modprobe.conf and /proc/filesystems<br>
Note 'alias lustre llite' should be removed from modprobe.conf</blockquote>
<br>
dmesg shows that the ko2iblnd module cannot be loaded:<br>
<blockquote type="cite">Lustre: OBD class driver,
<a class="moz-txt-link-freetext" href="http://www.lustre.org/"><a href="http://www.lustre.org/">http://www.lustre.org/</a></a><br>
Lustre: Lustre Version: 1.8.4<br>
Lustre: Build Version:
1.8.4-20100723170646-PRISTINE-2.6.18-194.3.1.el5_lustre.1.8.4<br>
ko2iblnd: disagrees about version of symbol ib_fmr_pool_unmap<br>
ko2iblnd: Unknown symbol ib_fmr_pool_unmap<br>
ko2iblnd: disagrees about version of symbol ib_create_cq<br>
ko2iblnd: Unknown symbol ib_create_cq<br>
ko2iblnd: disagrees about version of symbol rdma_resolve_addr<br>
ko2iblnd: Unknown symbol rdma_resolve_addr<br>
ko2iblnd: disagrees about version of symbol ib_reg_phys_mr<br>
ko2iblnd: Unknown symbol ib_reg_phys_mr<br>
ko2iblnd: disagrees about version of symbol ib_create_fmr_pool<br>
ko2iblnd: Unknown symbol ib_create_fmr_pool<br>
ko2iblnd: disagrees about version of symbol ib_dereg_mr<br>
ko2iblnd: Unknown symbol ib_dereg_mr<br>
ko2iblnd: disagrees about version of symbol rdma_reject<br>
ko2iblnd: Unknown symbol rdma_reject<br>
ko2iblnd: disagrees about version of symbol rdma_disconnect<br>
ko2iblnd: Unknown symbol rdma_disconnect<br>
ko2iblnd: disagrees about version of symbol rdma_resolve_route<br>
ko2iblnd: Unknown symbol rdma_resolve_route<br>
ko2iblnd: disagrees about version of symbol rdma_bind_addr<br>
ko2iblnd: Unknown symbol rdma_bind_addr<br>
ko2iblnd: disagrees about version of symbol rdma_create_qp<br>
ko2iblnd: Unknown symbol rdma_create_qp<br>
ko2iblnd: disagrees about version of symbol ib_destroy_cq<br>
ko2iblnd: Unknown symbol ib_destroy_cq<br>
ko2iblnd: disagrees about version of symbol rdma_create_id<br>
ko2iblnd: Unknown symbol rdma_create_id<br>
ko2iblnd: disagrees about version of symbol rdma_listen<br>
ko2iblnd: Unknown symbol rdma_listen<br>
ko2iblnd: disagrees about version of symbol rdma_destroy_qp<br>
ko2iblnd: Unknown symbol rdma_destroy_qp<br>
ko2iblnd: disagrees about version of symbol ib_query_device<br>
ko2iblnd: Unknown symbol ib_query_device<br>
ko2iblnd: disagrees about version of symbol ib_get_dma_mr<br>
ko2iblnd: Unknown symbol ib_get_dma_mr<br>
ko2iblnd: disagrees about version of symbol ib_alloc_pd<br>
ko2iblnd: Unknown symbol ib_alloc_pd<br>
ko2iblnd: disagrees about version of symbol rdma_connect<br>
ko2iblnd: Unknown symbol rdma_connect<br>
ko2iblnd: disagrees about version of symbol ib_modify_qp<br>
ko2iblnd: Unknown symbol ib_modify_qp<br>
ko2iblnd: disagrees about version of symbol rdma_destroy_id<br>
ko2iblnd: Unknown symbol rdma_destroy_id<br>
ko2iblnd: disagrees about version of symbol rdma_accept<br>
ko2iblnd: Unknown symbol rdma_accept<br>
ko2iblnd: disagrees about version of symbol ib_dealloc_pd<br>
ko2iblnd: Unknown symbol ib_dealloc_pd<br>
ko2iblnd: disagrees about version of symbol ib_fmr_pool_map_phys<br>
ko2iblnd: Unknown symbol ib_fmr_pool_map_phys<br>
LustreError: 7461:0:(api-ni.c:1081:lnet_startup_lndnis()) Can't
load LND o2ib, module ko2iblnd, rc=256<br>
LustreError: 7461:0:(events.c:725:ptlrpc_init_portals()) network
initialisation failed<br>
</blockquote>
<br>
Am I missing something obvious here.<br>
<br>
Thanks much.<br>
<br>
-Ed<br>
<br>
On 06/05/2011 05:48 AM, Wu, Yilei wrote:
<blockquote cite="mid:BANLkTinve3gYovhvXOaU_7XK7uZgDvtTeA@mail.gmail.com" type="cite">we have being use OFED 1.5.1 with Lustre 1.8.4
nowadays on a 400 node Cluster, on basis of RHEL 5.4. It is no
problem at all. <br>
<br>
One thing need attention:<br>
<br>
If using default OFED 1.5.1, just install with RPM package, no
need to build either Lustre or OFED.<br>
<br>
If using revised driver, such as BX-OFED 1.5.1, in some cases,
users need to recompile linux kernel with increased stack size,
because lustre and ofed may use up stack (both are stack greedy)
and thus lead to system hang issue.<br>
<br>
YiLei<br>
<br>
<br>
<div class="gmail_quote">On Thu, Jun 2, 2011 at 1:36 AM, Kevin Van
Maren <span dir="ltr"><<a moz-do-not-send="true" href="mailto:kevin.van.maren@oracle.com"><a href="mailto:kevin.van.maren@oracle.com">kevin.van.maren@oracle.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
OFED 1.5.1 should work fine with Lustre 1.8.4, although I
believe more<br>
people are using the in-kernel OFED now: Lustre (finally)
defaulted to<br>
the in-kernel OFED for RedHat, so it is no longer _necessary_
to build<br>
either OFED or Lustre.<br>
<font color="#888888"><br>
Kevin<br>
</font>
<div>
<div class="h5"><br>
<br>
Edward Walter wrote:<br>
> Hi List,<br>
><br>
> We're getting ready to upgrade the OS/software stack
on one of our<br>
> clusters and I'm looking at which Lustre and OFED
versions will work best.<br>
><br>
> It looks like the changelog for 1.8.4 and the
compatibility matrix have<br>
> conflicting information.<br>
><br>
> The Lustre compatibility matrix indicates that on
Lustre 1.8.4; the<br>
> highest OFED revision with o2iblnd support is 1.4.2:<br>
> <a moz-do-not-send="true" href="http://wiki.lustre.org/index.php/Lustre_Release_Information" target="_blank"><a href="http://wiki.lustre.org/index.php/Lustre_Release_Information">http://wiki.lustre.org/index.php/Lustre_Release_Information</a></a><br>
><br>
> The changelog for 1.8.4 indicates that o2iblnd is
supported with OFED 1.5.1:<br>
> <a moz-do-not-send="true" href="http://wiki.lustre.org/index.php/Change_Log_1.8#Changes_from_v1.8.3_to_v1.8.4" target="_blank"><a href="http://wiki.lustre.org/index.php/Change_Log_1.8#Changes_from_v1.8.3_to_v1.8.4">http://wiki.lustre.org/index.php/Change_Log_1.8#Changes_from_v1.8.3_to_v1.8.4</a></a><br>
><br>
><br>
> Can someone clarify whether 1.8.4 supports o2iblnd
with OFED 1.5.1? Are<br>
> there any pitfalls to this configuration? Has anyone
found any<br>
> instabilities with this configuration?<br>
><br>
> Thanks much.<br>
><br>
> -Ed Walter<br>
> Carnegie Mellon University<br>
> _______________________________________________<br>
> Lustre-discuss mailing list<br>
> <a moz-do-not-send="true" href="mailto:Lustre-discuss@lists.lustre.org"><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a></a><br>
> <a moz-do-not-send="true" href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank"><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a></a><br>
><br>
<br>
_______________________________________________<br>
Lustre-discuss mailing list<br>
<a moz-do-not-send="true" href="mailto:Lustre-discuss@lists.lustre.org"><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a></a><br>
<a moz-do-not-send="true" href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank"><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a></a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br>
</blockquote>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Lustre-discuss mailing list</span><br><span><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a></span><br><span><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a></span><br></div></blockquote></body></html>