[lustre-discuss] error while configuring lnet
Brett Lee
brettlee.lustre at gmail.com
Sun Nov 12 07:05:20 PST 2017
Hi Parag - I've been away from this field for awhile, but this message may
be the most helpful at this point:
modprobe: ERROR: could not insert 'ko2iblnd': Invalid argument
Last I knew, there were two IB releases - Mellanox/MOFED and Intel
(formerly QLogic) - and the Lustre kernel you use needs to be compiled
against the IB release that is installed on the node.
So, I see a couple possibilities of why the Lustre IB module will not load:
1. Since you are using MOFED, the kernel you have may instead support
Intel IB (thus the invalid argument).
2. One of the options provided may not be valid for the module, or may
conflict with another provided option:
require_privileged_port=0 use_privileged_port=0 timeout=150 retry_count=7
map_on_demand=32 peer_credits=63 concurrent_sends=63 ntx=32768
credits=32768 fmr_pool_size=8193
As item 1 is probably a binary, and item 2 a matrix, confirming that the
kernel that you have supports the module that you are trying to load seems
to be a better starting point. :)
As Intel HPDD seems to provide a kernel that matches the version are using,
item 1 seems likely.
https://downloads.hpdd.intel.com/public/lustre/lustre-2.9.0/el7.3.1611/server/RPMS/x86_64/
If that is the case, a couple options include using the Intel IB, or
compiling a Lustre kernel that supports the Mellanox OFED.
Hope this helps.
Brett
On Sun, Nov 12, 2017 at 1:16 AM, Parag Khuraswar <parag_k at citilindia.com>
wrote:
> Hi Brett,
>
>
>
> I am using MOFED “MLNX_OFED_LINUX-4.1-1.0.2.0” with kernel
> “3.10.0-514.el7.x86_64”
>
>
>
> o/p of modprobe –v ko2iblnd
>
>
>
> [root at mds1 ~]# modprobe -v ko2iblnd
>
> install /usr/sbin/ko2iblnd-probe require_privileged_port=0
> use_privileged_port=0 timeout=150 retry_count=7 map_on_demand=32
> peer_credits=63 concurrent_sends=63 ntx=32768 credits=32768
> fmr_pool_size=8193
>
> insmod /lib/modules/3.10.0-514.el7.x86_64/extra/lustre/net/ko2iblnd.ko
> require_privileged_port=0 use_privileged_port=0 timeout=150 retry_count=7
> map_on_demand=32 peer_credits=63 concurrent_sends=63 ntx=32768
> credits=32768 fmr_pool_size=8193
>
> modprobe: ERROR: could not insert 'ko2iblnd': Invalid argument
>
> modprobe: ERROR: Error running install command for ko2iblnd
>
> modprobe: ERROR: could not insert 'ko2iblnd': Operation not permitted
>
> [root at mds1 ~]#
>
>
>
>
>
> Regards,
>
> Parag
>
>
>
>
>
> *From:* Brett Lee [mailto:brettlee.lustre at gmail.com]
> *Sent:* Saturday, November , 2017 8:08 PM
> *To:* Parag Khuraswar
> *Cc:* Mannthey, Keith; lustre-discuss at lists.lustre.org
> *Subject:* Re: [lustre-discuss] error while configuring lnet
>
>
>
> Hi Parag,
>
>
>
> You may need to confirm that the in-kernel IB and the IB in the kernel
> module "match" (are compatible). I think that loading the module (`sudo
> modprobe -v ko2iblnd`) may be sufficient to verify the match (it's been a
> while, others may correct me).
>
>
>
> Please indicate which kernel and which IB you are using.
>
>
> Brett
>
> --
>
> Protect yourself against cybercrime
>
> PDS Software Solutions
>
> https://www.TrustPDS.com <https://www.trustpds.com/>
>
>
>
> On Fri, Nov 10, 2017 at 8:30 PM, Parag Khuraswar <parag_k at citilindia.com>
> wrote:
>
> Hi Keith,
>
>
>
> Below errors I am getting while adding lnet and mounting mdt.
>
>
>
> dmesg logs while adding lnet
>
> =========================================
>
> [317831.432182] LNetError: 28362:0:(api-ni.c:1861:lnet_startup_lndnet())
> Can't load LND o2ib, module ko2iblnd, rc=256
>
> =========================================
>
>
>
>
>
>
>
> dmesg logs while mounting mdt
>
> ==========================================
>
> [290476.172602] LNetError: 23040:0:(api-ni.c:1861:lnet_startup_lndnet())
> Can't load LND o2ib, module ko2iblnd, rc=256
>
> [317478.730515] LDISKFS-fs (dm-1): mounted filesystem with ordered data
> mode. Opts: errors=remount-ro
>
> [317480.166277] LDISKFS-fs (dm-1): mounted filesystem with ordered data
> mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
>
> [317480.313296] LustreError: 28268:0:(ldlm_lib.c:483:client_obd_setup())
> can't add initial connection
>
> [317480.313600] LustreError: 28268:0:(obd_config.c:608:class_setup())
> setup MGC10.2.1.204 at o2ib failed (-2)
>
> [317480.313603] LustreError: 28268:0:(obd_mount.c:202:lustre_start_simple())
> MGC10.2.1.204 at o2ib setup error -2
>
> [317480.313632] LustreError: 28268:0:(obd_mount_server.c:1573:server_put_super())
> no obd home-MDT0000
>
> [317480.313635] LustreError: 28268:0:(obd_mount_server.c:132:server_deregister_mount())
> home-MDT0000 not registered
>
> [317480.433934] Lustre: server umount home-MDT0000 complete
>
> [317480.433940] LustreError: 28268:0:(obd_mount.c:1504:lustre_fill_super())
> Unable to mount (-2)
>
> ==========================================
>
>
>
>
>
> Regards,
>
> Parag
>
>
>
>
>
> *From:* Mannthey, Keith [mailto:keith.mannthey at intel.com]
> *Sent:* Saturday, November , 2017 2:06 AM
>
>
> *To:* Parag Khuraswar; lustre-discuss at lists.lustre.org
> *Subject:* RE: [lustre-discuss] error while configuring lnet
>
>
>
> If you have ib0 device check dmesg for more hints on what is going wrong.
>
>
>
> Thanks,
>
> Keith
>
> *From:* Parag Khuraswar [mailto:parag_k at citilindia.com]
> *Sent:* Friday, November 10, 2017 10:59 AM
> *To:* Mannthey, Keith <keith.mannthey at intel.com>;
> lustre-discuss at lists.lustre.org
> *Subject:* RE: [lustre-discuss] error while configuring lnet
>
>
>
> Hi,
>
>
>
> Basically I am trying to add lnet. Deleting is just try whether it is
> happing or not.
>
> Main is I want to add o2ib network. Which is giving error “invalid
> argument ”
>
> ==================
>
> [root at mds2 ~]# lnetctl net add --net o2ib --if ib0
>
> add:
>
> - net:
>
> errno: -22
>
> descr: "cannot add network: Invalid argument"
>
> ================
>
> I am really not able to understand what argument is invalid in my command.
>
> I am able to ping ib0 network
>
>
>
> Regards,
>
> Parag
>
>
>
>
>
> *From:* Mannthey, Keith [mailto:keith.mannthey at intel.com
> <keith.mannthey at intel.com>]
> *Sent:* Friday, November , 2017 10:51 PM
> *To:* Parag Khuraswar; lustre-discuss at lists.lustre.org
> *Subject:* RE: [lustre-discuss] error while configuring lnet
>
>
>
> What are you trying to accomplish?
>
>
>
> From below:
>
>
>
> 10.1.1.205 at tcp is on 0 at lo not eno1 and in general you should not need the
> “—if” option to delete a fabric.
>
>
>
> Try: # lnetctl net del --net tcp
>
>
>
> Can you do a normal ping over ib0?
>
>
>
> “dmesg” can sometime provide greater details about errors like this.
>
>
>
> Thanks,
>
> Keith
>
>
>
>
>
> *From:* lustre-discuss [mailto:lustre-discuss-bounces at lists.lustre.org
> <lustre-discuss-bounces at lists.lustre.org>] *On Behalf Of *Parag Khuraswar
> *Sent:* Friday, November 10, 2017 9:10 AM
> *To:* lustre-discuss at lists.lustre.org
> *Subject:* [lustre-discuss] error while configuring lnet
>
>
>
> Hi,
>
>
>
> I am trying to add lnet but getting below error.
>
> ======================
>
> [root at mds2 ~]# lnetctl net show
>
> net:
>
> - net type: lo
>
> local NI(s):
>
> - nid: 0 at lo
>
> status: up
>
> - net type: tcp
>
> local NI(s):
>
> - nid: 10.1.1.205 at tcp
>
> status: up
>
> [root at mds2 ~]# lnetctl net add --net o2ib --if ib0
>
> add:
>
> - net:
>
> errno: -22
>
> descr: "cannot add network: Invalid argument"
>
> [root at mds2 ~]# lnetctl net del --net tcp --if eno1
>
> del:
>
> - net:
>
> errno: -22
>
> descr: "cannot del network: Invalid argument"
>
> [root at mds2 ~]# lctl list_nids
>
> 10.1.1.205 at tcp
>
> [root at mds2 ~]#
>
> ====================================
>
>
>
>
>
> Regards,
>
> Parag
>
>
>
>
>
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20171112/8a8bdf4a/attachment-0001.html>
More information about the lustre-discuss
mailing list