[lustre-discuss] client syslog flood with "client_bulk_callback()) event type 2, status -103"

[肖正刚]

Hi, all
After upgrade lustre client from 2.12.2 to 2.12.5，we found some clients
flood with messages like
"
[Wed Sep  9 15:49:05 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:06 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:06 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:07 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:07 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:08 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:08 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:08 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:09 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:10 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:11 2020] LNetError:
0:0:(o2iblnd_cb.c:3676:kiblnd_qp_event()) 10.10.41.11 at o2ib: Async QP event
type 3
[Wed Sep  9 15:49:11 2020] LNetError:
0:0:(o2iblnd_cb.c:3676:kiblnd_qp_event()) Skipped 599 previous similar
messages
[Wed Sep  9 15:49:11 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:11 2020] Lustre:
3845:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has
failed due to network error: [sent 1599637713/real 1599637713]
 req at ffff9a35c1e83f00 x1676624438686784/t0(0)
o3->public1-OST000a-osc-ffff9a3f3b3af000 at 10.10.41.11@o2ib:6/4 lens 488/440
e 1 to 1 dl 1599637816 ref 2 fl Rpc:eX/2/ffffffff rc -11/-1
[Wed Sep  9 15:49:11 2020] Lustre:
3845:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 599 previous
similar messages
[Wed Sep  9 15:49:11 2020] Lustre: public1-OST000a-osc-ffff9a3f3b3af000:
Connection to public1-OST000a (at 10.10.41.11 at o2ib) was lost; in progress
operations using this service will wait for recovery to complete
[Wed Sep  9 15:49:11 2020] Lustre: Skipped 599 previous similar messages
[Wed Sep  9 15:49:12 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:12 2020] Lustre: public1-OST000a-osc-ffff9a3f3b3af000:
Connection restored to 10.10.41.11 at o2ib (at 10.10.41.11 at o2ib)
[Wed Sep  9 15:49:12 2020] Lustre: Skipped 599 previous similar messages
[Wed Sep  9 15:49:13 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:13 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:14 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:14 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:14 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:14 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:15 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:15 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
[Wed Sep  9 15:49:16 2020] LustreError:
3476:0:(events.c:200:client_bulk_callback()) event type 2, status -103,
desc ffff9a1f3cafd000
"

 Any suggestions?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200909/ba92dffb/attachment.html>