[Lustre-discuss] Lustre-1.8.1.1 over o2ib gives Input/Output error while executing lctl ping

Vipul Pandya vipul at chelsio.com
Mon Feb 22 03:22:52 PST 2010


Hello Issac,

Thank you very much for your response.
I lowered the map_on_demand value to 16 and now it works fine.

However, I had once concern, whether lowering down this map_on_demand
value would impact the performance of Lustre or not?

Thanks again.
Vipul


-----Original Message-----
From: He.Huang at Sun.COM [mailto:He.Huang at Sun.COM] 
Sent: 16 February 2010 21:29
To: Vipul Pandya
Cc: lustre-discuss at lists.lustre.org
Subject: Re: [Lustre-discuss] Lustre-1.8.1.1 over o2ib gives
Input/Output error while executing lctl ping

On Mon, Feb 15, 2010 at 09:45:10PM -0800, Vipul Pandya wrote:
> ......
> -> I tried to load the ko2iblnd module as you have suggested. But
still
> I am unable to do 'lctl ping'. I am getting the same error as shown
> below.
> #> modprobe ko2iblnd map_on_demand=64

Please lower it to "map_on_demand=32".

> #> modprobe lnet
> #> lctl ping 102.88.88.184 at o2ib
> failed to ping 102.88.88.184 at o2ib: Input/output error
> #> dmesg
> Lustre: Listener bound to eth2:102.88.88.188:987:cxgb3_0
> Lustre: Register global MR array, MR size: 0xffffffff, array size: 2
> fmr_pool: Device cxgb3_0 does not support FMRs
> LustreError: 4122:0:(o2iblnd.c:1393:kiblnd_create_fmr_pool()) Failed
to
> create FMR pool: -38
> Lustre: Added LNI 102.88.88.188 at o2ib [8/64/0/0]
> LustreError: 2453:0:(o2iblnd.c:801:kiblnd_create_conn()) Can't create
> QP: -12, send_wr: 520, recv_wr: 18
> Lustre: 2453:0:(o2iblnd_cb.c:1953:kiblnd_peer_connect_failed())
Deleting
> messages for 102.88.88.184 at o2ib: connection faile

rdma_create_qp() failed with -ENOMEM again, even with a much smaller
send_wr (520 vs 2056). If lowering map_on_demand still couldn't fix it,
you'd need to look into HCA driver/firmware as to why it failed to
create the QP (if there's enough memory for it).

Isaac



More information about the lustre-discuss mailing list