[Lustre-discuss] problem with installing lustre and ofed

[Jason Brooks]

Hello,

That's good to know kernel-ib comes with the lustre stock install.

What about the rest of the OFED tools?  I mean things like ibdiagnet,
ibstatus, etc?  (I will look at the contents of the other rpms and see
what I can learn)

On 12/28/12 4:45 PM, "Jeff Johnson" <jeff.johnson at aeoncomputing.com> wrote:

>Jason,
>
>The prebuilt server-side Lustre packages from Whamcloud are built
>against RHEL/CentOS kernel sources with kernel-ib active in them. This
>means that any of the Lustre prebuilt server packages are already tied
>to RHEL's kernel-ib.
>
>To accomplish your stated goal you'll have to start with a non
>Whamcloud, stock kernel (plus headers, devel, etc). Then compile/install
>the OFED version of your choice. Once you have that you can build Lustre
>from source where it will compile against OFED and the installed kernel.
>
>--Jeff
>
>---------------------------
>Jeff Johnson
>Co-Founder
>Aeon Computing
>
>jeff.johnson at aeoncomputing.com
>www.aeoncomputing.com
>t: 858-412-3810 x101   f: 858-412-3845
>
>4170 Morena Boulevard, Suite D - San Diego, CA 92117
>
>/* Follow us on Twitter - @AeonComputing */
>
>
>
>
>On 12/28/12 3:54 PM, Jason Brooks wrote:
>> Hello,
>>
>> I am having trouble installing the server modules for lustre 2.1.4 and
>> use mellanox's OFED distribution so we may use infiniband. Would you
>> folks look at my procedure and results below and let me know what you
>> think? Thanks very much!
>>
>> The mellanox ofed installation builds and installs some kernel modules
>> too, so I used this method to ensure OFED compiled against the correct
>> kernel. This is on centos 6.3.
>>
>>  1. download all lustre rpms from whamcloud
>>  2. install kernel, kernel-firmware, kernel-headers, and kernel-devel
>>      1. in this case, it's the rpm files with
>>         "2.6.32-279.14.1.el6_lustre.x86_64" in their name
>>  3. reboot into this lustre kernel
>>  4. install the remaining rpms
>>  5. download ofed from mellanox
>>     "MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_64.iso"
>>      1. build mellanox ofed bits using the lustre kernel and
>>         kernel-devel info
>>      2. install mellanox ofed
>>  6. reboot
>>  7. upon reboot, if I do NOT have o2ib3 in my lnet networks
>>     parameters, I can modprobe lnet and lustre.
>>  8. if I DO have o2ib3 present in the lnet parameters, running
>>     modprobe lustre gets me:
>>
>> 
>>ib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/fld
>>.ko): 
>> Input/output error
>> WARNING: Error inserting fid
>> 
>>(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/
>>fid.ko): 
>> Input/output error
>> WARNING: Error inserting mdc
>> 
>>(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/
>>mdc.ko): 
>> Input/output error
>> WARNING: Error inserting osc
>> 
>>(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/
>>osc.ko): 
>> Input/output error
>> WARNING: Error inserting lov
>> 
>>(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/
>>lov.ko): 
>> Input/output error
>> FATAL: Error inserting lustre
>> 
>>(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/
>>lustre.ko): 
>> Input/output error
>>
>>
>> dmesg shows:
>> ko2iblnd: disagrees about version of symbol ib_fmr_pool_unmap
>> ko2iblnd: Unknown symbol ib_fmr_pool_unmap
>> ko2iblnd: disagrees about version of symbol ib_create_cq
>> ko2iblnd: Unknown symbol ib_create_cq
>> Š
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>_______________________________________________
>Lustre-discuss mailing list
>Lustre-discuss at lists.lustre.org
>http://lists.lustre.org/mailman/listinfo/lustre-discuss