<html><head><style type='text/css'>p { margin: 0; }</style><style type='text/css'>body { font-family: 'Times New Roman'; font-size: 12pt; color: #000000}</style></head><body>I have a lustre client that was randomly evicted early this morning. The errors from the dmesg are below. It's running infiniband. There were no infiniband errors that I could tell and all the mds/mgs and oss's said was "haven't heard from client xyz in 2277 seconds. Evicting". The client has halfway come back and now shows this -<br><br><br>aaron@cola10:~ $ lfs df -h<br>UUID bytes Used Available Use% Mounted on<br>data-MDT0000_UUID 87.5G 6.4G 81.1G 7% /data[MDT:0]<br>data-OST0000_UUID 5.4T 4.9T 439.6G 92% /data[OST:0]<br>data-OST0001_UUID : inactive device<br>data-OST0002_UUID : inactive device<br>data-OST0003_UUID : inactive device<br>data-OST0004_UUID : inactive device<br>data-OST0005_UUID : inactive device<br>data-OST0006_UUID : inactive device<br>data-OST0007_UUID : inactive device<br>data-OST0008_UUID : inactive device<br>data-OST0009_UUID : inactive device<br><br>filesystem summary: 5.4T 4.9T 439.6G 92% /data<br><br>so it's reconnected to one of 10 osts. I tried to to an lctl --device {device} reconnect and it said "Error: Operation in progress". I have no idea what went wrong and I'm confident a reboot would fix it but I'd like to avoid it if possible.<br><br><br>Thanks in advance.<br><br>LustreError: 11-0: an error occurred while communicating with 192.168.64.70@o2ib. The mds_statfs operation failed with -107<br>Lustre: data-MDT0000-mdc-ffff81013037b800: Connection to service data-MDT0000 via nid 192.168.64.70@o2ib was lost; in progress operations using this service will wait for recovery to complete.<br>LustreError: 167-0: This client was evicted by data-MDT0000; in progress operations using this service will fail.<br>LustreError: 22345:0:(llite_lib.c:1508:ll_statfs_internal()) mdc_statfs fails: rc = -5<br>LustreError: 22396:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810136334400 x81717113/t0 o41->data-MDT0000_UUID@192.168.64.70@o2ib:12 lens 128/272 ref 1 fl Rpc:/0/0 rc 0/0<br>LustreError: 22396:0:(llite_lib.c:1508:ll_statfs_internal()) mdc_statfs fails: rc = -108<br>LustreError: 22454:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8101136d2000 x81717114/t0 o41->data-MDT0000_UUID@192.168.64.70@o2ib:12 lens 128/272 ref 1 fl Rpc:/0/0 rc 0/0<br>LustreError: 22454:0:(llite_lib.c:1508:ll_statfs_internal()) mdc_statfs fails: rc = -108<br>LustreError: 22463:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810024ee4c00 x81717115/t0 o41->data-MDT0000_UUID@192.168.64.70@o2ib:12 lens 128/272 ref 1 fl Rpc:/0/0 rc 0/0<br>LustreError: 22463:0:(llite_lib.c:1508:ll_statfs_internal()) mdc_statfs fails: rc = -108<br>LustreError: 22734:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8101316c8200 x81717138/t0 o41->data-MDT0000_UUID@192.168.64.70@o2ib:12 lens 128/272 ref 1 fl Rpc:/0/0 rc 0/0<br>LustreError: 22734:0:(llite_lib.c:1508:ll_statfs_internal()) mdc_statfs fails: rc = -108<br>LustreError: 22736:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8101136d2c00 x81717139/t0 o41->data-MDT0000_UUID@192.168.64.70@o2ib:12 lens 128/272 ref 1 fl Rpc:/0/0 rc 0/0<br>LustreError: 22736:0:(llite_lib.c:1508:ll_statfs_internal()) mdc_statfs fails: rc = -108<br>LustreError: 22912:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8101136d2c00 x81717140/t0 o41->data-MDT0000_UUID@192.168.64.70@o2ib:12 lens 128/272 ref 1 fl Rpc:/0/0 rc 0/0<br>LustreError: 22912:0:(llite_lib.c:1508:ll_statfs_internal()) mdc_statfs fails: rc = -108<br>LustreError: 22971:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81012cebb000 x81717143/t0 o41->data-MDT0000_UUID@192.168.64.70@o2ib:12 lens 128/272 ref 1 fl Rpc:/0/0 rc 0/0<br>LustreError: 22971:0:(client.c:519:ptlrpc_import_delay_req()) Skipped 2 previous similar messages<br>LustreError: 22971:0:(llite_lib.c:1508:ll_statfs_internal()) mdc_statfs fails: rc = -108<br>LustreError: 22971:0:(llite_lib.c:1508:ll_statfs_internal()) Skipped 2 previous similar messages<br>LustreError: 23781:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81012bd02000 x81717144/t0 o41->data-MDT0000_UUID@192.168.64.70@o2ib:12 lens 128/272 ref 1 fl Rpc:/0/0 rc 0/0<br>LustreError: 23781:0:(llite_lib.c:1508:ll_statfs_internal()) mdc_statfs fails: rc = -108<br>LustreError: 23796:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81006c776000 x81717156/t0 o41->data-MDT0000_UUID@192.168.64.70@o2ib:12 lens 128/272 ref 1 fl Rpc:/0/0 rc 0/0<br>LustreError: 23827:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81013cbae400 x81717157/t0 o41->data-MDT0000_UUID@192.168.64.70@o2ib:12 lens 128/272 ref 1 fl Rpc:/0/0 rc 0/0<br>LustreError: 23827:0:(llite_lib.c:1508:ll_statfs_internal()) mdc_statfs fails: rc = -108<br>LustreError: 23827:0:(llite_lib.c:1508:ll_statfs_internal()) Skipped 1 previous similar message<br>LustreError: 22346:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8100a5f3d400 x81717169/t0 o35->data-MDT0000_UUID@192.168.64.70@o2ib:12 lens 296/896 ref 1 fl Rpc:/0/0 rc 0/0<br>LustreError: 22346:0:(file.c:97:ll_close_inode_openhandle()) inode 21601226 mdc close failed: rc = -108<br>Lustre: data-MDT0000-mdc-ffff81013037b800: Connection restored to service data-MDT0000 using nid 192.168.64.70@o2ib.<br>LustreError: 11-0: an error occurred while communicating with 192.168.64.71@o2ib. The ost_statfs operation failed with -107<br>Lustre: data-OST0001-osc-ffff81013037b800: Connection to service data-OST0001 via nid 192.168.64.71@o2ib was lost; in progress operations using this service will wait for recovery to complete.<br>LustreError: 11-0: an error occurred while communicating with 192.168.64.71@o2ib. The ost_statfs operation failed with -107<br>LustreError: 167-0: This client was evicted by data-OST0001; in progress operations using this service will fail.<br>LustreError: 167-0: This client was evicted by data-OST0002; in progress operations using this service will fail.<br>LustreError: 24093:0:(llite_lib.c:1520:ll_statfs_internal()) obd_statfs fails: rc = -5<br>Lustre: data-OST0000-osc-ffff81013037b800: Connection restored to service data-OST0000 using nid 192.168.64.71@o2ib.<br><br></body></html>