[lustre-discuss] Lustre-2.10.5 problem
Andreas Dilger
adilger at whamcloud.com
Tue Sep 25 00:38:00 PDT 2018
What does dmesg tell you? Normally it will report some module has incorrect symbols, which means you compiled against a different version of the kernel source. OFED/MOFED libraries, etc.
> On Sep 25, 2018, at 05:14, Tung-Han Hsieh <thhsieh at twcp1.phys.ntu.edu.tw> wrote:
>
> Dear All,
>
> I found that my lustre-2.10.5 with ZFS (either 0.7.9 or 0.7.11)
> cannot load the "lustre" modules because it cannot load the
> "obdclass.ko" module. The error message is the following:
>
> # modprobe -v -v obdclass
> insmod /lib/modules/3.12.72/updates/fs/lustre/obdclass.ko
> libkmod: INFO ../libkmod/libkmod-module.c:829 kmod_module_insert_module: Failed to insert module '/lib/modules/3.12.72/updates/fs/lustre/obdclass.ko': No such device
> ERROR: could not insert 'obdclass': No such device
> libkmod: INFO ../libkmod/libkmod.c:319 kmod_unref: context 0x7fb945d321e0 released
>
> Could anyone suggest how to debug ?
>
> Thanks very much.
>
>
> T.H.Hsieh
>
>
> On Tue, Sep 25, 2018 at 12:14:00AM +0800, Tung-Han Hsieh wrote:
>> Dear Nathaniel,
>>
>> Thank you very much for your kindly reply. Indeed I modified the
>> lustre-2.10.5 codes:
>>
>> lustre/osd-zfs/osd_object.c
>> lustre/osd-zfs/osd_xattr.c
>>
>> for the declaration:
>>
>> inode_timespec_t now;
>>
>> Similar to what you have done in your patch. So I can compile
>> lustre-2.10.5 cleanly with zfs-0.7.11. Sorry I forgot to mention.
>>
>> But my problem is still there. Actually I just tried:
>>
>> 1. Applying your patch to the original lustre-2.10.5 code, and
>> recompile with spl-0.7.11 and zfs-0.7.11. But loading "lustre"
>> module still gives "no such device" error.
>>
>> 2. I recompile the original lustre-2.10.5 with spl-0.7.9 and
>> zfs-0.7.9. They can be compiled cleanly. But again I got the
>> "no such device" error when loading "lustre" module.
>>
>> I am wondering that I must overlooked a trivial step, something
>> like one (or some) of the utilities in /opt/lustre/sbin/* should
>> be linked to /sbin/ or /usr/sbin/ ....
>>
>> Any suggestions are very appreciated.
>>
>> Thank you very much.
>>
>>
>> T.H.Hsieh
>>
>>
>> On Mon, Sep 24, 2018 at 01:21:19PM +0000, Nathaniel Clark wrote:
>>> Hello Tung-Han,
>>>
>>> ZFS 0.7.11 doesn’t compile cleanly with Lustre, yet.
>>>
>>> There’s a ticket for adding ZFS 0.7.11 support to lustre:
>>> https://jira.whamcloud.com/browse/LU-11393
>>>
>>> It has patches for master (pre-2.12) and a separate patch for 2.10.
>>>
>>> —
>>> Nathaniel Clark <nclark at whamcloud.com<mailto:nclark at whamcloud.com>>
>>> Senior Engineer
>>> Whamcloud / DDN
>>>
>>> On Sep 24, 2018, at 2:15 PM, Tung-Han Hsieh <thhsieh at twcp1.phys.ntu.edu.tw<mailto:thhsieh at twcp1.phys.ntu.edu.tw>> wrote:
>>>
>>> Dear All,
>>>
>>> I am trying to install Lustre version 2.10.5 with ZFS-0.7.11
>>> from source code. After compilation and installation, I tried
>>> to load the "lustre" module, but encountered the following
>>> error:
>>>
>>> # modprobe lustre
>>> could not load module 'lustre': no such device
>>>
>>> My procedure of installation is the following:
>>>
>>> 1. Compile vanilla kernel 3.12.72 downloaded from:
>>> https://mirrors.edge.kernel.org/pub/linux/kernel/v3.x/linux-3.12.72.tar.gz
>>>
>>> 2. Compile spl-0.7.11 downloaded from:
>>> https://github.com/zfsonlinux/zfs/releases/download/zfs-0.7.11/spl-0.7.11.tar.gz
>>>
>>> with the following steps:
>>> # ./configure --prefix=/opt/lustre --with-linux=/path/to/linux-3.12.72
>>> # make
>>> # make install
>>>
>>> 3. Compile zfs-0.7.11 downloaded from:
>>> https://github.com/zfsonlinux/zfs/releases/download/zfs-0.7.11/zfs-0.7.11.tar.gz
>>>
>>> with the following steps:
>>> # ./configure --prefix=/opt/lustre \
>>> --with-linux=/path/to/linux-3.12.72 \
>>> --with-spl=/path/to/spl-0.7.11
>>> # make
>>> # make install
>>>
>>> 4. Compile lustre downloaded from:
>>> https://downloads.whamcloud.com/public/lustre/lustre-2.10.5/sles12sp3/client/SRPMS/lustre-2.10.5-1.src.rpm
>>>
>>> Then I unpack the SRPM by the command:
>>> # rpm2cpio lustre-2.10.5-1.src.rpm | cpio --extract --make-directories
>>>
>>> and compile it by the following:
>>> # ./configure --prefix=/opt/lustre \
>>> --with-linux=/path/to/linux-3.12.72 \
>>> --with-spl=/path/to/spl-0.7.11 \
>>> --with-zfs=/path/to/zfs-0.7.11 \
>>> --with-o2ib=no \
>>> --disable-ldiskfs
>>> # make
>>> # make install
>>>
>>> 5. I have made sure the following settings and utilities are correct:
>>> - PATH contains /opt/lustre/bin and /opt/lustre/sbin
>>> - /sbin/mount.lustre exists.
>>> - /sbin/mount.zfs exists.
>>> - /usr/sbin/l_getidentity exists.
>>> - /usr/sbin/ko2iblnd-probe exists.
>>> - /etc/modprobe.d/lustre.conf contains:
>>> options lnet networks=tcp
>>> - /etc/modprobe.d/ko2iblnd.conf contains:
>>> alias ko2iblnd-opa ko2iblnd
>>> options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1
>>> install ko2iblnd /usr/sbin/ko2iblnd-probe
>>>
>>> Then I tried to run "modprobe lustre", it says "no such device" error.
>>>
>>> I tried to replace Lustre-2.10.5 by Lustre-2.9 downloaded from:
>>>
>>> https://downloads.whamcloud.com/public/lustre/lustre-2.9.0/sles12sp1/client/SRPMS/lustre-2.9.0-1.src.rpm
>>>
>>> and proceed exactly the same installation steps. Everything works fine.
>>>
>>> Could anyone suggest me what have I missed for lustre-2.10.5 ? Or suggest
>>> me how to debug.
>>>
>>> Thanks very much.
>>>
>>>
>>> T.H.Hsieh
>>> _______________________________________________
>>> lustre-discuss mailing list
>>> lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Cheers, Andreas
---
Andreas Dilger
CTO Whamcloud
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20180925/437a016d/attachment.sig>
More information about the lustre-discuss
mailing list