[Lustre-discuss] Cannot get an OST to activate

[Bob Ball]

We added a new OSS to our 1.8.4 Lustre installation.  It has 6 OST of 
8.9TB each.  Within a day of having these on-line, one OST stopped 
accepting new files.  I cannot get it to activate.  The other 5 seem fine.

On the MDS "lctl dl" shows it IN, but not UP, and files can be read from it:
 33 IN osc umt3-OST001d-osc umt3-mdtlov_UUID 5

However, I cannot get it to re-activate:
lctl --device umt3-OST001d-osc activate

This returns no errors, but dmesg on the MDS shows this as a result:
[603128.578862] Lustre: umt3-OST001d-osc: Connection restored to service 
umt3-OST001d using nid 10.10.2.23 at tcp.
[603128.578865] Lustre: Skipped 1 previous similar message
[603128.579251] Lustre: MDS umt3-MDT0000: umt3-OST001d_UUID now active, 
resetting orphans
[603128.579256] Lustre: Skipped 1 previous similar message
[603128.579608] LustreError: 9655:0:(osc_create.c:589:osc_create()) 
umt3-OST001d-osc: oscc recovery failed: -22
[603128.579616] LustreError: 9655:0:(lov_obd.c:1134:lov_clear_orphans()) 
error in orphan recovery on OST idx 29/34: rc = -22
[603128.579623] LustreError: 
9655:0:(mds_lov.c:1057:__mds_lov_synchronize()) umt3-OST001d_UUID failed 
at mds_lov_clear_orphans: -22
[603128.579628] LustreError: 
9655:0:(mds_lov.c:1066:__mds_lov_synchronize()) umt3-OST001d_UUID sync 
failed -22, deactivating

On the OSS itself, I see these related entries appear:
Lustre: 4642:0:(ldlm_lib.c:572:target_handle_reconnect()) umt3-OST001d: 
umt3-mdtlov_UUID reconnecting
Lustre: 4642:0:(ldlm_lib.c:572:target_handle_reconnect()) Skipped 1 
previous similar message
Lustre: umt3-OST001d: received MDS connection from 10.10.1.49 at tcp
Lustre: Skipped 1 previous similar message
LustreError: 4697:0:(filter.c:3172:filter_handle_precreate()) 
umt3-OST001d: ignoring bogus orphan destroy request: obdid 
11309489156331498430 last_id 0

Can anyone tell me what must be done to recover this disk volume?

Thanks,
bob