[Lustre-discuss] problem with installing lustre and ofed

[Jeff Johnson]

Jason,

The prebuilt server-side Lustre packages from Whamcloud are built 
against RHEL/CentOS kernel sources with kernel-ib active in them. This 
means that any of the Lustre prebuilt server packages are already tied 
to RHEL's kernel-ib.

To accomplish your stated goal you'll have to start with a non 
Whamcloud, stock kernel (plus headers, devel, etc). Then compile/install 
the OFED version of your choice. Once you have that you can build Lustre 
from source where it will compile against OFED and the installed kernel.

--Jeff

---------------------------
Jeff Johnson
Co-Founder
Aeon Computing

jeff.johnson at aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x101   f: 858-412-3845

4170 Morena Boulevard, Suite D - San Diego, CA 92117

/* Follow us on Twitter - @AeonComputing */




On 12/28/12 3:54 PM, Jason Brooks wrote:
> Hello,
>
> I am having trouble installing the server modules for lustre 2.1.4 and 
> use mellanox's OFED distribution so we may use infiniband. Would you 
> folks look at my procedure and results below and let me know what you 
> think? Thanks very much!
>
> The mellanox ofed installation builds and installs some kernel modules 
> too, so I used this method to ensure OFED compiled against the correct 
> kernel. This is on centos 6.3.
>
>  1. download all lustre rpms from whamcloud
>  2. install kernel, kernel-firmware, kernel-headers, and kernel-devel
>      1. in this case, it's the rpm files with
>         "2.6.32-279.14.1.el6_lustre.x86_64" in their name
>  3. reboot into this lustre kernel
>  4. install the remaining rpms
>  5. download ofed from mellanox
>     "MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_64.iso"
>      1. build mellanox ofed bits using the lustre kernel and
>         kernel-devel info
>      2. install mellanox ofed
>  6. reboot
>  7. upon reboot, if I do NOT have o2ib3 in my lnet networks
>     parameters, I can modprobe lnet and lustre.
>  8. if I DO have o2ib3 present in the lnet parameters, running
>     modprobe lustre gets me:
>
> ib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/fld.ko): 
> Input/output error
> WARNING: Error inserting fid 
> (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/fid.ko): 
> Input/output error
> WARNING: Error inserting mdc 
> (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/mdc.ko): 
> Input/output error
> WARNING: Error inserting osc 
> (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/osc.ko): 
> Input/output error
> WARNING: Error inserting lov 
> (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/lov.ko): 
> Input/output error
> FATAL: Error inserting lustre 
> (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/lustre.ko): 
> Input/output error
>
>
> dmesg shows:
> ko2iblnd: disagrees about version of symbol ib_fmr_pool_unmap
> ko2iblnd: Unknown symbol ib_fmr_pool_unmap
> ko2iblnd: disagrees about version of symbol ib_create_cq
> ko2iblnd: Unknown symbol ib_create_cq
> …
>
>
>
>
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss