[Lustre-discuss] Lustre-1.8.1.1 over o2ib gives Input/Output error while executing lctl ping

[Isaac Huang]

On Fri, Feb 12, 2010 at 05:53:19AM -0800, Vipul Pandya wrote:
>    ......
>    #> lctl network up
>    LNET configured
>    Above command gave me following error in dmesg
>    #> dmesg
> 
>    Lustre: Listener bound to eth2:102.88.88.188:987:cxgb3_0
>    Lustre: Register global MR array, MR size: 0xffffffff, array size: 2
>    fmr_pool: Device cxgb3_0 does not support FMRs
>    LustreError: 4134:0:(o2iblnd.c:1393:kiblnd_create_fmr_pool()) Failed to
>    create FMR pool: -38

ib_create_fmr_pool() returned -ENOSYS, probably the HCA didn't support
FMR; this was not an fatal error.

>    Lustre: Added LNI 102.88.88.188 at o2ib [8/64/0/0]
> 
>    #> lctl ping 102.88.88.184 at o2ib
>    failed to ping 102.88.88.184 at o2ib: Input/output error
>    dmesg has shown following error:
>    #> dmesg
>    LustreError: 2453:0:(o2iblnd.c:801:kiblnd_create_conn()) Can't create
>    QP: -12, send_wr: 2056, recv_wr: 18

rdma_create_qp() returned -ENOMEM; most likely init_qp_attr->cap.max_send_wr
was too big (2056) and needed too much memory.

>    Lustre: 2453:0:(o2iblnd_cb.c:1953:kiblnd_peer_connect_failed())
>    Deleting messages for 102.88.88.184 at o2ib: connection failed

You'd need to use the o2iblnd map-on-demand feature. To find out
whether your ko2iblnd module supports it:
modinfo ko2iblnd | grep map_on_demand

If yes, please try:
options ko2iblnd map_on_demand=64

Thanks,
Isaac