[lustre-discuss] lustre/ZFS resilver order question
Mountford, Christopher J. (Dr.)
cjm14 at leicester.ac.uk
Tue Sep 27 03:34:16 PDT 2022
Hi,
This is possibly more a ZFS rather than lustre question - but it involves ZFS 0.7.13 which I think is now mostly used with lustre.
One of our ZFS (raid Z2) osts has 2 faulty disks:
[root at aoss01a ~]# zpool status aliceost02
pool: aliceost02
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Sep 27 11:10:44 2022
2.86G scanned out of 71.0T at 29.9M/s, 691h54m to go
288M resilvered, 0.00% done
config:
NAME STATE READ WRITE CKSUM
aliceost02 DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
A11-aoss01j1-021 ONLINE 0 0 0
spare-1 DEGRADED 0 0 0
A11-aoss01j1-022 OFFLINE 34 633 0
A11-aoss01j2-084 ONLINE 0 0 0 (resilvering)
A11-aoss01j1-023 ONLINE 0 0 0
A11-aoss01j1-024 ONLINE 0 0 0
A11-aoss01j1-025 ONLINE 0 0 0
A11-aoss01j1-026 ONLINE 157 546 0 (resilvering)
A11-aoss01j1-027 ONLINE 0 0 0
A11-aoss01j1-028 ONLINE 0 0 0
A11-aoss01j1-029 ONLINE 0 0 0
A11-aoss01j1-030 ONLINE 0 0 0
spares
A11-aoss01j2-084 INUSE currently in use
The device A11-aoss01j1-022 failed first, a spare (A11-aoss01j2-084) was assigned and a resilver started. Since then a second disk (A11-aoss01j1-026) has failed. Both disks are shown as resilvering, am I right in thinking that the second disk resilver will have
+completely interupted the first resilver (this is what the resilver time implies) and the resilver of A11-aoss01j1-022 will not restart until the resilver of A11-aoss01j1-026 is complete?
My reading of the ZoL docs is that versions of ZFS after 0.8 defer new resilvers until the older one has completed, but older versions interupt the running resilver?
If my understanding of this is correct then A11-aoss01j1-026 needs to be offlined and replaced as well and we just have to hope we don't get a 3rd disk failure in the ~70 hours required to resilver at least one of the offline disks!
Any advice or ZFS knowledge appreciated.
(I've disabled stripe creation on the OST to try to reduce I/O on the pool and set the zfs_resilver_delay to 0 to prioritize the resilver, anything else we could do to speed up the resilver?)
Kind Regards,
Christopher.
More information about the lustre-discuss
mailing list