<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Centos 7.2<br>
Lustre 2.8.0<br>
ZFS 0.6.5.5<br>
OPA <span class="version"><span class="value">10.2.0.0.158<br>
<br>
<br>
The clients and servers are on the same OPA network, no
routing. Once a client gets in this state, the filesystem
performance drops to a faction of what it is capable of.<br>
The client must be rebooted to clear the issue. <br>
<br>
</span></span>
<p>I imagine I am missing a bug in jira for this issue, does this
look like a known issue?<br>
</p>
<p><br>
</p>
<p>Pertinent debug messages from the server:<br>
</p>
<p>00000800:00020000:34.0:1478026118.277782:0:29892:0:(o2iblnd_cb.c:3109:kiblnd_check_txs_locked())
Timed out tx: active_txs, 4 seconds<br>
00000800:00020000:34.0:1478026118.277785:0:29892:0:(o2iblnd_cb.c:3172:kiblnd_check_conns())
Timed out RDMA with 10.4.119.112@o2ib (3): c: 112, oc: 0, rc: 66<br>
00000800:00000100:34.0:1478026118.277787:0:29892:0:(o2iblnd_cb.c:1913:kiblnd_close_conn_locked())
Closing conn to 10.4.119.112@o2ib: error -110(waiting)<br>
00000100:00020000:34.0:1478026118.277844:0:29892:0:(events.c:447:server_bulk_callback())
event type 5, status -103, desc ffff883e8e8bcc00<br>
00000100:00020000:34.0:1478026118.288714:0:29892:0:(events.c:447:server_bulk_callback())
event type 3, status -103, desc ffff883e8e8bcc00<br>
00000100:00020000:34.0:1478026118.299574:0:29892:0:(events.c:447:server_bulk_callback())
event type 5, status -103, desc ffff8810e92e9c00<br>
00000100:00020000:34.0:1478026118.310434:0:29892:0:(events.c:447:server_bulk_callback())
event type 3, status -103, desc ffff8810e92e9c00</p>
<br>
And from the client:<br>
<br>
<p>00000400:00000100:8.0:1477949860.565777:0:3629:0:(lib-move.c:1489:lnet_parse_put())
Dropping PUT from 12345-10.4.108.81@o2ib portal 4 match
1549728742532740 offset 0 length 192: 4<br>
00000400:00000100:8.0:1477949860.565782:0:3629:0:(lib-move.c:1489:lnet_parse_put())
Dropping PUT from 12345-10.4.108.81@o2ib portal 4 match
1549728742532740 offset 0 length 192: 4<br>
00000800:00020000:8.0:1477949860.702666:0:3629:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma())
RDMA has too many fragments for peer 10.4.108.81@o2ib (32), src
idx/frags: 16/27 dst idx/frags: 16/27<br>
00000800:00020000:8.0:1477949860.702667:0:3629:0:(o2iblnd_cb.c:1689:kiblnd_reply())
Can't setup rdma for GET from 10.4.108.81@o2ib: -90<br>
00000100:00020000:8.0:1477949860.702669:0:3629:0:(events.c:201:client_bulk_callback())
event type 1, status -5, desc ffff880fd5d9bc00<br>
00000800:00020000:8.0:1477949860.816666:0:3629:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma())
RDMA has too many fragments for peer 10.4.108.81@o2ib (32), src
idx/frags: 16/27 dst idx/frags: 16/27<br>
00000800:00020000:8.0:1477949860.816668:0:3629:0:(o2iblnd_cb.c:1689:kiblnd_reply())
Can't setup rdma for GET from 10.4.108.81@o2ib: -90<br>
00000100:00020000:8.0:1477949860.816669:0:3629:0:(events.c:201:client_bulk_callback())
event type 1, status -5, desc ffff880fd5d9bc00<br>
00000400:00000100:8.0:1477949861.573660:0:3629:0:(lib-move.c:1489:lnet_parse_put())
Dropping PUT from 12345-10.4.108.81@o2ib portal 4 match
1549728742532740 offset 0 length 192: 4<br>
00000400:00000100:8.0:1477949861.573664:0:3629:0:(lib-move.c:1489:lnet_parse_put())
Dropping PUT from 12345-10.4.108.81@o2ib portal 4 match
1549728742532740 offset 0 length 192: 4<br>
00000400:00000100:8.0:1477949861.573667:0:3629:0:(lib-move.c:1489:lnet_parse_put())
Dropping PUT from 12345-10.4.108.81@o2ib portal 4 match
1549728742532740 offset 0 length 192: 4<br>
00000400:00000100:8.0:1477949861.573669:0:3629:0:(lib-move.c:1489:lnet_parse_put())
Dropping PUT from 12345-10.4.108.81@o2ib portal 4 match
1549728742532740 offset 0 length 192: 4<br>
00000400:00000100:8.0:1477949861.573671:0:3629:0:(lib-move.c:1489:lnet_parse_put())
Dropping PUT from 12345-10.4.108.81@o2ib portal 4 match
1549728742532740 offset 0 length 192: 4<br>
00000400:00000100:8.0:1477949861.573673:0:3629:0:(lib-move.c:1489:lnet_parse_put())
Dropping PUT from 12345-10.4.108.81@o2ib portal 4 match
1549728742532740 offset 0 length 192: 4<br>
00000400:00000100:8.0:1477949861.573675:0:3629:0:(lib-move.c:1489:lnet_parse_put())
Dropping PUT from 12345-10.4.108.81@o2ib portal 4 match
1549728742532740 offset 0 length 192: 4<br>
00000400:00000100:8.0:1477949861.573677:0:3629:0:(lib-move.c:1489:lnet_parse_put())
Dropping PUT from 12345-10.4.108.81@o2ib portal 4 match
1549728742532740 offset 0 length 192: 4<br>
00000800:00020000:8.0:1477949861.721668:0:3629:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma())
RDMA has too many fragments for peer 10.4.108.81@o2ib (32), src
idx/frags: 16/27 dst idx/frags: 16/27<br>
00000800:00020000:8.0:1477949861.721669:0:3629:0:(o2iblnd_cb.c:1689:kiblnd_reply())
Can't setup rdma for GET from 10.4.108.81@o2ib: -90<br>
00000100:00020000:8.0:1477949861.721670:0:3629:0:(events.c:201:client_bulk_callback())
event type 1, status -5, desc ffff880fd5d9bc00<br>
00000800:00020000:8.0:1477949861.836668:0:3629:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma())
RDMA has too many fragments for peer 10.4.108.81@o2ib (32), src
idx/frags: 16/27 dst idx/frags: 16/27<br>
00000800:00020000:8.0:1477949861.836669:0:3629:0:(o2iblnd_cb.c:1689:kiblnd_reply())
Can't setup rdma for GET from 10.4.108.81@o2ib: -90<br>
00000100:00020000:8.0:1477949861.836670:0:3629:0:(events.c:201:client_bulk_callback())
event type 1, status -5, desc ffff880fd5d9bc00<br>
00000800:00020000:8.0:1477949862.061668:0:3629:0:(o2iblnd_cb.c:1094:kiblnd_init_rdma())
RDMA has too many fragments for peer 10.4.108.81@o2ib (32), src
idx/frags: 16/27 dst idx/frags: 16/27<br>
00000800:00020000:8.0:1477949862.061669:0:3629:0:(o2iblnd_cb.c:1689:kiblnd_reply())
Can't setup rdma for GET from 10.4.108.81@o2ib: -90<br>
00000100:00020000:8.0:1477949862.061670:0:3629:0:(events.c:201:client_bulk_callback())
event type 1, status -5, desc ffff880fd5d9bc00<br>
<br>
</p>
</body>
</html>