May 1 03:35:46 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.21.10.116@o2ib (256), src idx/frags: 128/236 dst idx/frags: 128/236 May 1 03:35:46 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Can't setup rdma for GET from 10.21.10.116@o2ib: -90 May 1 03:35:46 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:35:46 node872 kernel: Lustre: 5606:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1493602541/real 1493602541] req@ffff880e99cea080 x1565604440535580/t0(0) o4->astro-OST0002-osc-ffff881070c95c00@10.21.10.116@o2ib:6/4 lens 608/448 e 0 to 1 dl 1493602585 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 May 1 03:35:46 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection to astro-OST0002 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 03:35:46 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 03:35:46 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:35:46 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:35:46 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:35:46 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:35:46 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:35:46 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:35:46 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:35:52 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1493602546/real 1493602546] req@ffff88103e0f10c0 x1565604440535684/t0(0) o8->astro-OST0002-osc-ffff881070c95c00@10.21.10.116@o2ib:28/4 lens 520/544 e 0 to 1 dl 1493602552 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 1 03:35:52 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 7 previous similar messages May 1 03:36:17 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1493602571/real 1493602571] req@ffff881056dd39c0 x1565604440535728/t0(0) o8->astro-OST0002-osc-ffff881070c95c00@10.21.10.115@o2ib:28/4 lens 520/544 e 0 to 1 dl 1493602577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 1 03:36:18 node872 kernel: Lustre: astro-OST0001-osc-ffff881070c95c00: Connection to astro-OST0001 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 03:36:18 node872 kernel: Lustre: Skipped 7 previous similar messages May 1 03:36:24 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1493602578/real 1493602578] req@ffff8808cf3c6380 x1565604440535756/t0(0) o8->astro-OST0001-osc-ffff881070c95c00@10.21.10.116@o2ib:28/4 lens 520/544 e 0 to 1 dl 1493602584 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 1 03:36:24 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 1 previous similar message May 1 03:36:43 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 03:36:43 node872 kernel: Lustre: Skipped 6 previous similar messages May 1 03:36:43 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.21.10.116@o2ib (256), src idx/frags: 128/236 dst idx/frags: 128/236 May 1 03:36:43 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) Skipped 7 previous similar messages May 1 03:36:43 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Can't setup rdma for GET from 10.21.10.116@o2ib: -90 May 1 03:36:43 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Skipped 7 previous similar messages May 1 03:36:43 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:36:43 node872 kernel: Lustre: 5606:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1493602603/real 1493602603] req@ffff880e99cea080 x1565604440535580/t0(0) o4->astro-OST0002-osc-ffff881070c95c00@10.21.10.116@o2ib:6/4 lens 608/448 e 0 to 1 dl 1493602647 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 May 1 03:36:43 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection to astro-OST0002 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 03:36:43 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:36:43 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:36:43 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:36:43 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:36:43 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:36:43 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:36:43 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:37:14 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1493602628/real 1493602628] req@ffff880d375b46c0 x1565604440535888/t0(0) o8->astro-OST0002-osc-ffff881070c95c00@10.21.10.116@o2ib:28/4 lens 520/544 e 0 to 1 dl 1493602634 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 1 03:37:14 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 9 previous similar messages May 1 03:37:39 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1493602653/real 0] req@ffff880e99cea380 x1565604440535928/t0(0) o8->astro-OST0001-osc-ffff881070c95c00@172.20.10.116@tcp1:28/4 lens 520/544 e 0 to 1 dl 1493602659 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 May 1 03:37:39 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 1 previous similar message May 1 03:38:48 node872 kernel: Lustre: astro-OST0001-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 03:38:48 node872 kernel: Lustre: Skipped 7 previous similar messages May 1 03:38:54 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1493602728/real 1493602728] req@ffff880e99ceac80 x1565604440536052/t0(0) o8->astro-OST0002-osc-ffff881070c95c00@10.21.10.115@o2ib:28/4 lens 520/544 e 0 to 1 dl 1493602734 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 1 03:38:54 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 2 previous similar messages May 1 03:39:13 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 03:39:13 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.21.10.116@o2ib (256), src idx/frags: 128/236 dst idx/frags: 128/236 May 1 03:39:13 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) Skipped 7 previous similar messages May 1 03:39:13 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Can't setup rdma for GET from 10.21.10.116@o2ib: -90 May 1 03:39:13 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Skipped 7 previous similar messages May 1 03:39:13 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:39:13 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection to astro-OST0002 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 03:39:13 node872 kernel: Lustre: Skipped 7 previous similar messages May 1 03:39:13 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:39:13 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:39:13 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:39:13 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:39:13 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:39:13 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:39:13 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:39:45 node872 kernel: Lustre: astro-OST0001-osc-ffff881070c95c00: Connection to astro-OST0001 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 03:39:45 node872 kernel: Lustre: Skipped 7 previous similar messages May 1 03:40:16 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1493602810/real 1493602810] req@ffff881037b230c0 x1565604440536252/t0(0) o8->astro-OST0002-osc-ffff881070c95c00@10.21.10.115@o2ib:28/4 lens 520/544 e 0 to 1 dl 1493602816 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 1 03:40:16 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 12 previous similar messages May 1 03:41:50 node872 kernel: Lustre: astro-OST0001-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 03:41:50 node872 kernel: Lustre: Skipped 7 previous similar messages May 1 03:42:15 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 03:42:15 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.21.10.116@o2ib (256), src idx/frags: 128/236 dst idx/frags: 128/236 May 1 03:42:15 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) Skipped 7 previous similar messages May 1 03:42:15 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Can't setup rdma for GET from 10.21.10.116@o2ib: -90 May 1 03:42:15 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Skipped 7 previous similar messages May 1 03:42:15 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:42:15 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection to astro-OST0002 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 03:42:15 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:42:15 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:42:15 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:42:15 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:42:15 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:42:15 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:42:15 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:42:46 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1493602960/real 1493602960] req@ffff881056dd33c0 x1565604440536568/t0(0) o8->astro-OST0002-osc-ffff881070c95c00@10.21.10.116@o2ib:28/4 lens 520/544 e 0 to 1 dl 1493602966 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 1 03:42:46 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 14 previous similar messages May 1 03:42:47 node872 kernel: Lustre: astro-OST0001-osc-ffff881070c95c00: Connection to astro-OST0001 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 03:42:47 node872 kernel: Lustre: Skipped 7 previous similar messages May 1 03:44:52 node872 kernel: Lustre: astro-OST0001-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 03:44:52 node872 kernel: Lustre: Skipped 7 previous similar messages May 1 03:45:17 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.21.10.116@o2ib (256), src idx/frags: 128/236 dst idx/frags: 128/236 May 1 03:45:17 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) Skipped 7 previous similar messages May 1 03:45:17 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Can't setup rdma for GET from 10.21.10.116@o2ib: -90 May 1 03:45:17 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Skipped 7 previous similar messages May 1 03:45:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:45:17 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection to astro-OST0002 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 03:45:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:45:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:45:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:45:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:45:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:45:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:45:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:47:11 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1493603224/real 0] req@ffff880d375b43c0 x1565604440537072/t0(0) o8->astro-OST0001-osc-ffff881070c95c00@172.20.10.116@tcp1:28/4 lens 520/544 e 0 to 1 dl 1493603230 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 May 1 03:47:11 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 24 previous similar messages May 1 03:47:54 node872 kernel: Lustre: astro-OST0001-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 03:47:54 node872 kernel: Lustre: Skipped 8 previous similar messages May 1 03:48:20 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.21.10.116@o2ib (256), src idx/frags: 249/256 dst idx/frags: 249/256 May 1 03:48:20 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Can't setup rdma for GET from 10.21.10.116@o2ib: -90 May 1 03:48:20 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Skipped 7 previous similar messages May 1 03:48:20 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:48:20 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection to astro-OST0002 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 03:48:20 node872 kernel: Lustre: Skipped 8 previous similar messages May 1 03:48:20 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:48:20 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:48:20 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:48:20 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:48:20 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:48:20 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:48:20 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) Skipped 14 previous similar messages May 1 03:48:20 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 03:49:17 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 03:49:17 node872 kernel: Lustre: Skipped 7 previous similar messages May 1 03:49:17 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.21.10.116@o2ib (256), src idx/frags: 249/256 dst idx/frags: 249/256 May 1 03:49:17 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Can't setup rdma for GET from 10.21.10.116@o2ib: -90 May 1 03:49:17 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Skipped 7 previous similar messages May 1 03:49:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:49:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:49:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:49:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:49:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:49:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:49:17 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:49:17 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) Skipped 7 previous similar messages May 1 03:49:17 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 03:51:47 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 03:51:47 node872 kernel: Lustre: Skipped 7 previous similar messages May 1 03:51:47 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.21.10.116@o2ib (256), src idx/frags: 249/256 dst idx/frags: 249/256 May 1 03:51:47 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Can't setup rdma for GET from 10.21.10.116@o2ib: -90 May 1 03:51:47 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Skipped 7 previous similar messages May 1 03:51:47 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:51:47 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection to astro-OST0002 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 03:51:47 node872 kernel: Lustre: Skipped 14 previous similar messages May 1 03:51:47 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:51:47 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:51:47 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:51:47 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:51:47 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:51:47 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:51:47 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) Skipped 7 previous similar messages May 1 03:51:47 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 03:52:50 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:52:50 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 03:52:50 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 03:52:50 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:52:50 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 03:52:50 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:52:50 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 03:52:50 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:55:20 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.21.10.116@o2ib (256), src idx/frags: 249/256 dst idx/frags: 249/256 May 1 03:55:20 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Can't setup rdma for GET from 10.21.10.116@o2ib: -90 May 1 03:55:20 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Skipped 15 previous similar messages May 1 03:55:20 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:55:20 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:55:20 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:55:20 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:55:20 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:55:20 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:55:20 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:55:20 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) Skipped 14 previous similar messages May 1 03:55:20 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 03:55:51 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1493603745/real 1493603745] req@ffff880d375b49c0 x1565604440538216/t0(0) o8->astro-OST0002-osc-ffff881070c95c00@10.21.10.116@o2ib:28/4 lens 520/544 e 0 to 1 dl 1493603751 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 1 03:55:51 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 67 previous similar messages May 1 03:57:57 node872 kernel: Lustre: astro-OST0001-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 03:57:57 node872 kernel: Lustre: Skipped 18 previous similar messages May 1 03:58:22 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.21.10.116@o2ib (256), src idx/frags: 249/256 dst idx/frags: 249/256 May 1 03:58:22 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Can't setup rdma for GET from 10.21.10.116@o2ib: -90 May 1 03:58:22 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Skipped 7 previous similar messages May 1 03:58:22 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:58:22 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection to astro-OST0002 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 03:58:22 node872 kernel: Lustre: Skipped 19 previous similar messages May 1 03:58:22 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:58:22 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:58:22 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:58:22 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:58:22 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:58:22 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 03:58:22 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) Skipped 7 previous similar messages May 1 03:58:22 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:01:24 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:01:24 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:01:24 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:01:24 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:01:24 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:01:24 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:01:24 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:01:24 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:04:26 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.21.10.116@o2ib (256), src idx/frags: 249/256 dst idx/frags: 249/256 May 1 04:04:26 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Can't setup rdma for GET from 10.21.10.116@o2ib: -90 May 1 04:04:26 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Skipped 15 previous similar messages May 1 04:04:26 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:04:26 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:04:26 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:04:26 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:04:26 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:04:26 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:04:26 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:04:26 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) Skipped 15 previous similar messages May 1 04:04:26 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:05:54 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1493604348/real 0] req@ffff880d375b49c0 x1565604440539376/t0(0) o8->astro-OST0002-osc-ffff881070c95c00@172.20.10.116@tcp1:28/4 lens 520/544 e 0 to 1 dl 1493604354 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 May 1 04:05:54 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 58 previous similar messages May 1 04:07:03 node872 kernel: Lustre: astro-OST0001-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 04:07:03 node872 kernel: Lustre: Skipped 20 previous similar messages May 1 04:07:28 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:07:28 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:07:28 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection to astro-OST0002 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 04:07:28 node872 kernel: Lustre: Skipped 20 previous similar messages May 1 04:07:28 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:07:28 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:07:28 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:07:28 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:07:28 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:07:28 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:10:30 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:10:30 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:10:30 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:10:30 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:10:30 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:10:30 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:10:30 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:10:30 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:13:32 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.21.10.116@o2ib (256), src idx/frags: 249/256 dst idx/frags: 249/256 May 1 04:13:32 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Can't setup rdma for GET from 10.21.10.116@o2ib: -90 May 1 04:13:32 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Skipped 23 previous similar messages May 1 04:13:32 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:13:32 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:13:32 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:13:32 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:13:32 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:13:32 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:13:32 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:13:32 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) Skipped 23 previous similar messages May 1 04:13:32 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:14:30 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:14:30 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:14:30 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:14:30 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:14:30 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:14:30 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:14:30 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:14:30 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:16:41 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1493604995/real 1493604995] req@ffff881056dd30c0 x1565604440540644/t0(0) o8->astro-OST0002-osc-ffff881070c95c00@10.21.10.115@o2ib:28/4 lens 520/544 e 0 to 1 dl 1493605001 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 1 04:16:41 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 66 previous similar messages May 1 04:17:00 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:17:00 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:17:00 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:17:00 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:17:00 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:17:00 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:17:00 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:17:00 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:17:32 node872 kernel: Lustre: astro-OST0001-osc-ffff881070c95c00: Connection to astro-OST0001 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 04:17:32 node872 kernel: Lustre: Skipped 25 previous similar messages May 1 04:19:37 node872 kernel: Lustre: astro-OST0001-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 04:19:37 node872 kernel: Lustre: Skipped 26 previous similar messages May 1 04:20:02 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:20:02 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:20:02 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:20:02 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:20:02 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:20:02 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:20:02 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:20:02 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:23:04 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:23:04 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:23:04 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:23:04 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:23:04 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:23:04 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:23:04 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:23:04 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:26:06 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.21.10.116@o2ib (256), src idx/frags: 249/256 dst idx/frags: 249/256 May 1 04:26:06 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Can't setup rdma for GET from 10.21.10.116@o2ib: -90 May 1 04:26:06 node872 kernel: LNetError: 5545:0:(o2iblnd_cb.c:1689:kiblnd_reply()) Skipped 39 previous similar messages May 1 04:26:06 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:26:06 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:26:06 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:26:06 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:26:06 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:26:06 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:26:06 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:26:06 node872 kernel: LNetError: 5544:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma()) Skipped 39 previous similar messages May 1 04:26:06 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:26:44 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1493605598/real 1493605598] req@ffff88081c534080 x1565604440541864/t0(0) o8->astro-OST0001-osc-ffff881070c95c00@10.21.10.116@o2ib:28/4 lens 520/544 e 0 to 1 dl 1493605604 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 1 04:26:44 node872 kernel: Lustre: 5579:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 65 previous similar messages May 1 04:27:08 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:27:08 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:27:08 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:27:08 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:27:08 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:27:08 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:27:08 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:27:08 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:29:38 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection restored to 10.21.10.116@o2ib (at 10.21.10.116@o2ib) May 1 04:29:38 node872 kernel: Lustre: Skipped 22 previous similar messages May 1 04:29:38 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:29:38 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:29:38 node872 kernel: Lustre: astro-OST0002-osc-ffff881070c95c00: Connection to astro-OST0002 (at 10.21.10.116@o2ib) was lost; in progress operations using this service will wait for recovery to complete May 1 04:29:38 node872 kernel: Lustre: Skipped 22 previous similar messages May 1 04:29:38 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:29:38 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:29:38 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:29:38 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:29:38 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:29:38 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:32:40 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:32:40 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000 May 1 04:32:40 node872 kernel: LustreError: 5544:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88081d982000 May 1 04:32:40 node872 kernel: LustreError: 5545:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc ffff88103dd63000