[Lustre-discuss] Lustre-1.8.1.1 over o2ib gives Input/Output error while executing lctl ping

[Isaac Huang]

On Mon, Feb 15, 2010 at 09:45:10PM -0800, Vipul Pandya wrote:
> ......
> -> I tried to load the ko2iblnd module as you have suggested. But still
> I am unable to do 'lctl ping'. I am getting the same error as shown
> below.
> #> modprobe ko2iblnd map_on_demand=64

Please lower it to "map_on_demand=32".

> #> modprobe lnet
> #> lctl ping 102.88.88.184 at o2ib
> failed to ping 102.88.88.184 at o2ib: Input/output error
> #> dmesg
> Lustre: Listener bound to eth2:102.88.88.188:987:cxgb3_0
> Lustre: Register global MR array, MR size: 0xffffffff, array size: 2
> fmr_pool: Device cxgb3_0 does not support FMRs
> LustreError: 4122:0:(o2iblnd.c:1393:kiblnd_create_fmr_pool()) Failed to
> create FMR pool: -38
> Lustre: Added LNI 102.88.88.188 at o2ib [8/64/0/0]
> LustreError: 2453:0:(o2iblnd.c:801:kiblnd_create_conn()) Can't create
> QP: -12, send_wr: 520, recv_wr: 18
> Lustre: 2453:0:(o2iblnd_cb.c:1953:kiblnd_peer_connect_failed()) Deleting
> messages for 102.88.88.184 at o2ib: connection faile

rdma_create_qp() failed with -ENOMEM again, even with a much smaller
send_wr (520 vs 2056). If lowering map_on_demand still couldn't fix it,
you'd need to look into HCA driver/firmware as to why it failed to
create the QP (if there's enough memory for it).

Isaac